How to amdgpu on proxmox 7

[+DS_DV+]

Hello community,

id like to install amdgpu / amdgpu-pro drivers on proxmox to pass my gpu to lxc containers.

did anyone get that to work on proxmox 7?

can anyone help me out with steps to get drivers to work on proxmox?

with kindregards


PS: do i also have to install the drivers in the lxc container or can i skip that? (provided it is possible to get amdgpu onto proxmox7)

 

[Stefan_R]

'amdgpu' is included in our kernel, no need to install anything. passthrough to container is a rather tricky business, but it does not require seperate drivers in the container.

 

[+DS_DV+]

sadly i cannot confirm that.
But the amdgpu and amdgpu pro drivers can normaly be installed if you install "pve-headers" first via:

apt install pve-headers

the use inside the lxc for transcoding seems not to work without installing amdgpu drivers for the container.
ill do further testing.

 

[timonych]

sadly i cannot confirm that.
But the amdgpu and amdgpu pro drivers can normaly be installed if you install "pve-headers" first via:

apt install pve-headers

the use inside the lxc for transcoding seems not to work without installing amdgpu drivers for the container.
ill do further testing.

HI!

I have a success with AMDGPU Passthrough to LXC for Hardware transcoding.
I'm using Jellyfin for My Home Library and I'm using HW Transcoding with vaapi.

PreqReq: LXC Container Ubuntu 20.04

Spoiler: cpuinfo

root@Jellyfin:/# cpu-info
Packages:
0: AMD Athlon 200GE with
Cores:
0: 1 processor (0), AMD Zen
1: 1 processor (1), AMD Zen
2: 1 processor (2), AMD Zen
3: 1 processor (3), AMD Zen
Logical processors:
0: APIC ID 0x00000000
1: APIC ID 0x00000001
2: APIC ID 0x00000002
3: APIC ID 0x00000003
root@Jellyfin:/# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 23
model : 17
model name : AMD Athlon 200GE with Radeon Vega Graphics
stepping : 0
microcode : 0x8101016
cpu MHz : 3200.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd sev ibpb vmmcall sev_es fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_
model : 17
model name : AMD Athlon 200GE with Radeon Vega Graphics
stepping : 0
microcode : 0x8101016
cpu MHz : 3200.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd sev ibpb vmmcall sev_es fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
bugs : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips : 6388.15
TLB size : 2560 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate eff_freq_ro [13] [14]

processor : 3
vendor_id : AuthenticAMD
cpu family : 23
model : 17
model name : AMD Athlon 200GE with Radeon Vega Graphics
stepping : 0
microcode : 0x8101016
cpu MHz : 3200.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 2
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd sev ibpb vmmcall sev_es fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
bugs : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips : 6388.15
TLB size : 2560 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate eff_freq_ro [13] [14]

1. I have used this guide from emby
2. In lxc.conf I'm include this options to passthrough amdgpu

lxc.cgroup2.devices.allow: c 226:* rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file

3. install vainfo and mesa-va-drivers
4. use vainfo to check is GPU available

Spoiler: vainfo

root@Jellyfin:/# ls /dev/dri/
card0  renderD128
root@Jellyfin:/# ls -la /dev/dri
total 0
drwxr-xr-x 2 root root         80 Aug 28 08:10 .
drwxr-xr-x 9 root root        540 Aug 28 08:10 ..
crw-rw---- 1 root video  226,   0 Aug 28 08:06 card0
crw-rw---- 1 root render 226, 128 Aug 28 08:06 renderD128
root@Jellyfin:/# vainfo
error: can't connect to X server!
libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Mesa Gallium driver 21.0.3 for AMD Radeon(TM) Vega 3 Graphics (RAVEN, DRM 3.40.0, 5.11.22-3-pve, LLVM 12.0.0)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

5. Enable Vaapi for HW Transconding in Jellyfin and use correct path to your render

Spoiler: Screen

 

[+DS_DV+]

HI!

I have a success with AMDGPU Passthrough to LXC for Hardware transcoding.
I'm using Jellyfin for My Home Library and I'm using HW Transcoding with vaapi.

im using ubuntu 20.04 with jellyfin too

i had to install pve-headers with apt install to get amdgpu-pro to run.
i also installed amdgpu inside the contrainer and use amd AMF to hardware transcode.

##Proxmox commands

apt install pve-headers

./amdgpu-pro-install --no-32 --opencl=legacy,rocr -y
apt install amf-amdgpu-pro

Container.conf

lxc.cgroup2.devices.allow: c 226:* rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file

##LXC commands

./amdgpu-install --opencl=legacy --headless --no-dkms --no-32 -y
apt install amf-amdgpu-pro

 

[+DS_DV+]

@timonych
did you have any particular reason to use mesa / vaapi over amd AMF ?

did you try to use @Stefan_R s way without drivers inside the lxc ?

 

[timonych]

@timonych
did you have any particular reason to use mesa / vaapi over amd AMF ?

did you try to use @Stefan_R s way without drivers inside the lxc ?

No, there is no any particullar reason.

In my situation I think it's overhead to use amd proprietary driver to use HW Transcoding instead of Vaapi (It's fully supported also HEVC 10-bit)

Spoiler: Kimi no Na Wa

jellyfin 155 1 0 Aug28 ? 00:20:22 /usr/bin/jellyfin --webdir=/usr/share/jellyfin/web --restartpath=/usr/lib/jellyfin/restart.sh --ffmpeg=/usr/lib/jellyfin-ffmpeg/ffmpeg
jellyfin 105047 155 99 20:44 ? 00:00:32 /usr/lib/jellyfin-ffmpeg/ffmpeg -ss 00:50:51.000 -hwaccel vaapi -noaccurate_seek -hwaccel_output_format vaapi -vaapi_device /dev/dri/renderD128 -i file:/var/medialibrary/video/cartoons/Kimi no Na Wa/Kimi no Na wa.mkv -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_vaapi -b:v 19408766 -maxrate 19408766 -bufsize 38817532 -profile:v:0 high -level 41 -force_key_frames:0 expr:gte(t,3051+n_forced*3) -vf format=nv12|vaapi,hwupload,scale_vaapi=w=1920:h=1080:format=nv12 -start_at_zero -vsync -1 -codec:a:0 copy -strict -2 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 1017 -hls_segment_filename /var/medialibrary/video/transcodes/2169012a3ff85cfa1df7f9ba145e18c7%d.ts -hls_playlist_type vod -hls_list_size 0 -y /var/medialibrary/video/transcodes/2169012a3ff85cfa1df7f9ba145e18c7.m3u8

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L5@Main
Codec ID : V_MPEGH/ISO/HEVC
Duration : 1 h 46 min
Bit rate : 20.3 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 23.976 (24000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.102
Stream size : 15.1 GiB (87%)
Writing library : x265 2.5+4-b4a5bcfe29c7:[Windows][MSVC 1910][64 bit] 10bit
Encoding settings : cpuid=1173503 / frame-threads=3 / numa-pools=8 / wpp / no-pmode / no-pme / no-psnr / ssim / log-level=2 / csvfn=Main.csv / csv-log-level=2 / input-csp=1 / input-res=3840x2160 / interlace=0 / total-frames=0 / level-idc=0 / high-tier=1 / uhd-bd=0 / ref=4 / no-allow-non-conformance / no-repeat-headers / annexb / no-aud / no-hrd / info / hash=0 / no-temporal-layers / no-open-gop / min-keyint=1 / keyint=720 / bframes=8 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=40 / lookahead-slices=1 / scenecut=40 / no-intra-refresh / ctu=64 / min-cu-size=8 / no-rect / no-amp / max-tu-size=32 / tu-inter-depth=2 / tu-intra-depth=2 / limit-tu=4 / rdoq-level=2 / dynamic-rd=0.00 / ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-merge=4 / limit-refs=3 / limit-modes / me=3 / subme=3 / merange=57 / temporal-mvp / weightp / weightb / no-analyze-src-pics / deblock=0:0 / no-sao / no-sao-non-deblock / rd=4 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / b-intra / rdpenalty=2 / psy-rd=1.00 / psy-rdoq=0.00 / no-rd-refine / analysis-reuse-mode=0 / no-lossless / cbqpoffs=-4 / crqpoffs=-4 / rc=crf / crf=17.0 / qcomp=0.75 / qpstep=1 / stats-write=0 / stats-read=0 / ipratio=1.20 / pbratio=1.00 / aq-mode=1 / aq-strength=0.80 / cutree / zone-count=0 / no-strict-cbr / qg-size=16 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=0 / overscan=0 / videoformat=5 / range=0 / colorprim=1 / transfer=1 / colormatrix=1 / chromaloc=0 / display-window=0 / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / opt-qp-pps / opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / no-hdr / no-hdr-opt / no-dhdr10-opt / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / no-limit-sao / ctu-info=0
Language : Japanese
Default : Yes
Forced : No
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

 

[+DS_DV+]

No, there is no any particullar reason.

In my situation I think it's overhead to use amd proprietary driver to use HW Transcoding instead of Vaapi (It's fully supported also HEVC 10-bit)

thanks i finaly got my lcx jellyfin to run

vaapi also seems to be faster. Sadly most transcodes are super distored

 

[TheHellSite]

Hello,

I am also having trouble getting Jellyfin + VAAPI working in my Arch Linux LXC.

Things I did so far.

# Jellyfin LXC GPU passthrough

1. **Proxmox Host:** Find the GPU device number.

       ls -l /dev/dri

2. **Proxmox Host:** Add them to the LXC configuration file.

       nano /etc/pve/lxc/LXC_ID.conf
       add the lines below...
       
       lxc.cgroup2.devices.allow: c 226:0 rwm
       lxc.cgroup2.devices.allow: c 226:128 rwm
       lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
       lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file

3. **LXC Guest:** Start the LXC, install the latest Mesa drivers and reboot the LXC.

       pacman -Syyu mesa --noconfirm && reboot

4. **Jellyfin:** Enable VAAPI.

       Go to: Admin --> Server --> Dashboard --> Playback
       Hardware acceleration: VAAPI
       VA API Device: /dev/dri/renderD128

Sadly when I run "radeontop" inside the LXC it displays no activity.
This is while using a an android client to force Jellyfin to transcode a 1080p video to 480p.

 

[+DS_DV+]

Sadly when I run "radeontop" inside the LXC it displays no activity.
This is while using a an android client to force Jellyfin to transcode a 1080p video to 480p.

radeontop only displays properly in my proxmox host. inside the lxc it seems broken.

Also which android client do you use ? (in mine i cant set replay resolutions :/ )

 

[TheHellSite]

Android 11 latest Jellyfin from the Play Store.

Ahhh so it is a bug with the android client. (offical jellyfin from play store)
For me the playback also doesn't even start on android if I use anything below the actual file resolution and mbps.

then I will try antother client. Samsung TV.

So also there it seems that transcoding is not working.

 

[danluckner]

im using ubuntu 20.04 with jellyfin too

i had to install pve-headers with apt install to get amdgpu-pro to run.
i also installed amdgpu inside the contrainer and use amd AMF to hardware transcode.

##Proxmox commands

apt install pve-headers

./amdgpu-pro-install --no-32 --opencl=legacy,rocr -y
apt install amf-amdgpu-pro

Container.conf

lxc.cgroup2.devices.allow: c 226:* rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file

##LXC commands

./amdgpu-install --opencl=legacy --headless --no-dkms --no-32 -y
apt install amf-amdgpu-pro

Hello!

I have been trying to get jellyfin HW encoding on a LXC with Ubuntu 20.04 as well, but I don't get the same success as you.

I am stuck on finding the amdgpu-pro and the amf-amdgpu-pro.
Both on the host OS and on the LXC.
My "apt install pve-headers" returns the folling:

root@pve:~# apt install pve-headers
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
pve-headers is already the newest version (7.2-1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Would you mind sharing with more details the step-by-step how you achieved that?