PCIe Passthrough broken

Dudefoxlive

Member
Jun 15, 2021
17
0
6
23
So I am attempting to passthrough a GT 730 through to a QEMU VM. I followed the guide and got everything setup and its very weird. When the host boots I am able to start the VM but the graphics look broken. (Images) The drivers won't install and windows update driver says error 43. If you turn off the vm or shut it down the VM will not start back up. I installed a remote control program to make changes when no display is connected and windows never loads. I have tried everything and can't seem to figure out what the problem is. Any ideas?
 
can you post the config (qm config ID)
the version (pveversion -v)
and the output of 'dmesg' ?
 
can you post the config (qm config ID)
the version (pveversion -v)
and the output of 'dmesg' ?
QM Config
Code:
agent: 1
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
bios: ovmf
boot: order=sata0;net0;virtio0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: local:110/vm-110-disk-1.qcow2,size=128K
hostpci0: 0000:03:00,pcie=1,romfile=GT730BIOS.rom
machine: pc-q35-5.2
memory: 12288
name: Gaming
net0: e1000=42:EA:DE:AB:24:FF,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
sata0: local:iso/systemrescue-8.03-amd64.iso,media=cdrom,size=717M
scsihw: virtio-scsi-pci
smbios1: uuid=941a3c18-d3be-4cfe-a868-1f32eccd92c5
sockets: 1
usb0: host=046d:c52b
virtio0: local:110/vm-110-disk-0.qcow2,size=100G
vmgenid: a23b6631-9254-43db-a87a-121446e8201f

PVE Version
Code:
proxmox-ve: 6.4-1 (running kernel: 5.4.119-1-pve)
pve-manager: 6.4-8 (running version: 6.4-8/185e14db)
pve-kernel-5.4: 6.4-3
pve-kernel-helper: 6.4-3
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.9-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-6
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.2-4
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

dmesg
Code:
[41795.727926] device tap110i0 entered promiscuous mode
[41795.776711] fwbr110i0: port 1(fwln110i0) entered blocking state
[41795.776718] fwbr110i0: port 1(fwln110i0) entered disabled state
[41795.776785] device fwln110i0 entered promiscuous mode
[41795.776823] fwbr110i0: port 1(fwln110i0) entered blocking state
[41795.776826] fwbr110i0: port 1(fwln110i0) entered forwarding state
[41795.783406] vmbr0: port 11(fwpr110p0) entered blocking state
[41795.783412] vmbr0: port 11(fwpr110p0) entered disabled state
[41795.783482] device fwpr110p0 entered promiscuous mode
[41795.785800] vmbr0: port 11(fwpr110p0) entered blocking state
[41795.785805] vmbr0: port 11(fwpr110p0) entered forwarding state
[41795.851785] fwbr110i0: port 2(tap110i0) entered blocking state
[41795.851792] fwbr110i0: port 2(tap110i0) entered disabled state
[41795.851885] fwbr110i0: port 2(tap110i0) entered blocking state
[41795.851888] fwbr110i0: port 2(tap110i0) entered forwarding state
[41799.345393] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[41799.384441] vfio-pci 0000:03:00.1: enabling device (0000 -> 0002)
[41896.027547] usb 8-2: reset full-speed USB device number 2 using ohci-pci
[41967.976317] hrtimer: interrupt took 5193701 ns
[42028.207115] perf: interrupt took too long (5110 > 2500), lowering kernel.perf_event_max_sample_rate to 39000
[42029.795062] fwbr110i0: port 2(tap110i0) entered disabled state
[42029.830257] fwbr110i0: port 1(fwln110i0) entered disabled state
[42029.830392] vmbr0: port 11(fwpr110p0) entered disabled state
[42029.832455] device fwln110i0 left promiscuous mode
[42029.832463] fwbr110i0: port 1(fwln110i0) entered disabled state
[42029.854713] device fwpr110p0 left promiscuous mode
[42029.854720] vmbr0: port 11(fwpr110p0) entered disabled state
[42030.218371] usb 8-2: reset full-speed USB device number 2 using ohci-pci
[42030.424238] logitech-djreceiver 0003:046D:C52B.0008: hiddev0,hidraw0: USB HID v1.11 Device [Logitech USB Receiver] on usb-0000:00:12.0-2/input2
[42030.558871] input: Logitech K400 as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2:1.2/0003:046D:C52B.0008/0003:046D:404B.0009/input/input18
[42030.559185] logitech-hidpp-device 0003:046D:404B.0009: input,hidraw1: USB HID v1.11 Keyboard [Logitech K400] on usb-0000:00:12.0-2/input2:1
[42030.566755] input: Logitech M305 as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2:1.2/0003:046D:C52B.0008/0003:046D:101F.000A/input/input19
[42030.566873] logitech-hidpp-device 0003:046D:101F.000A: input,hidraw2: USB HID v1.11 Mouse [Logitech M305] on usb-0000:00:12.0-2/input2:2
[42035.210107] device tap110i0 entered promiscuous mode
[42035.262249] fwbr110i0: port 1(fwln110i0) entered blocking state
[42035.262256] fwbr110i0: port 1(fwln110i0) entered disabled state
[42035.262627] device fwln110i0 entered promiscuous mode
[42035.262664] fwbr110i0: port 1(fwln110i0) entered blocking state
[42035.262667] fwbr110i0: port 1(fwln110i0) entered forwarding state
[42035.269240] vmbr0: port 11(fwpr110p0) entered blocking state
[42035.269246] vmbr0: port 11(fwpr110p0) entered disabled state
[42035.269322] device fwpr110p0 entered promiscuous mode
[42035.272046] vmbr0: port 11(fwpr110p0) entered blocking state
[42035.272051] vmbr0: port 11(fwpr110p0) entered forwarding state
[42035.345816] fwbr110i0: port 2(tap110i0) entered blocking state
[42035.345823] fwbr110i0: port 2(tap110i0) entered disabled state
[42035.345922] fwbr110i0: port 2(tap110i0) entered blocking state
[42035.345925] fwbr110i0: port 2(tap110i0) entered forwarding state
[42038.897750] vfio-pci 0000:03:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
[42081.721167] fwbr110i0: port 2(tap110i0) entered disabled state
[42081.738068] fwbr110i0: port 1(fwln110i0) entered disabled state
[42081.738165] vmbr0: port 11(fwpr110p0) entered disabled state
[42081.740267] device fwln110i0 left promiscuous mode
[42081.740275] fwbr110i0: port 1(fwln110i0) entered disabled state
[42081.766271] device fwpr110p0 left promiscuous mode
[42081.766277] vmbr0: port 11(fwpr110p0) entered disabled state
[42082.145919] usb 8-2: reset full-speed USB device number 2 using ohci-pci
[42082.348390] logitech-djreceiver 0003:046D:C52B.000D: hiddev0,hidraw0: USB HID v1.11 Device [Logitech USB Receiver] on usb-0000:00:12.0-2/input2
[42082.478825] input: Logitech K400 as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2:1.2/0003:046D:C52B.000D/0003:046D:404B.000E/input/input20
[42082.479063] logitech-hidpp-device 0003:046D:404B.000E: input,hidraw1: USB HID v1.11 Keyboard [Logitech K400] on usb-0000:00:12.0-2/input2:1
[42082.488746] input: Logitech M305 as /devices/pci0000:00/0000:00:12.0/usb8/8-2/8-2:1.2/0003:046D:C52B.000D/0003:046D:101F.000F/input/input21
[42082.488901] logitech-hidpp-device 0003:046D:101F.000F: input,hidraw2: USB HID v1.11 Mouse [Logitech M305] on usb-0000:00:12.0-2/input2:2
 
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
remove that, it is not necessary, we already set everything you want. if you still want to overwrite the hv_vendor_id, you can do that in the 'cpu' config
Code:
qm set ID -cpu host,hv-vendor-id=something

[42038.897750] vfio-pci 0000:03:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff
seems the rom is not ok, try without the romfile or get a working version
 
Can you tell us which guide you followed? A lot of stuff is no longer required or often not needed. Please note that passthrough is a bit trial and error as it is not a common use case and some hardware will just not reset properly. Check that the GT 730 is in an isolated IOMMU group and use x-vga=1 or Primary GPU is enabled for Nvidia GPUs. If you dump a ROM file, make sure that no driver or BIOS touches it before saving it to file. If the GT 730 is the only GPU (and used during boot and startup of Proxmox), you will need some additional video= kernel parameter, which will prevent you from seeing startup messages and problem if they occur. A little more information about your motherboard and the IOMMU groups (using for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done) would help.
EDIT: fixed typos for future readers.
 
Last edited:
Can you tell us which guide you followed? A lot of stuff is no longer required or often not needed. Please note that passthrough is a bit trial and error as it is not a common use case and some hardware will just not reset properly. Check that the GT 730 is in an isolated IOMMU group and use x-vga=1 or Primary GPU is enabled for Nvidia GPUs. If you dump a ROM file, make sure that no driver or BIOS touches it before saving it to file. If the GT 730 is the only GPU (and used during boot and startup of Proxmox), you will need some additional video= kernel parameter, which will prevent you from seeing startup messages and problem if they occur. A little more information about your motherboard and the IOMMU groups (using or d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done) would help.
This was the guide I used. (https://pve.proxmox.com/wiki/PCI(e)_Passthrough)
Running the command you provided for the IOMMU groups provides
Code:
zsh: parse error near `do'
the Motherboard is an ASUS M5A99FX PRO R2.0 with an AMD FX 6300.
The system has 2 GPUs. an AMD 5450 and the GT730. the 5450 is the primary on boot as it is in the 1st slot.
 
Sorry, a character fell off during the copy and paste. Please use: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
EDIT: That guide does not mention -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off', where did you get that from?
 
Last edited:
Sorry, a character fell off during the copy and paste. Please use: for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
Code:
IOMMU group 0 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge [1002:5a14] (rev 02)
IOMMU group 10 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d] (rev 40)
IOMMU group 11 00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384] (rev 40)
IOMMU group 12 00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399]
IOMMU group 13 00:15.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a0]
IOMMU group 13 00:15.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1) [1002:43a1]
IOMMU group 13 00:15.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2) [1002:43a2]
IOMMU group 13 00:15.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 3) [1002:43a3]
IOMMU group 13 06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
IOMMU group 13 07:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
IOMMU group 13 08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
IOMMU group 13 09:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 09)
IOMMU group 14 00:16.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
IOMMU group 14 00:16.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
IOMMU group 15 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] [1002:68f9]
IOMMU group 15 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series] [1002:aa68]
IOMMU group 16 03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF108 [GeForce GT 730] [10de:0f02] (rev a1)
IOMMU group 16 03:00.1 Audio device [0403]: NVIDIA Corporation GF108 High Definition Audio Controller [10de:0bea] (rev a1)
IOMMU group 1 00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GFX port 0) [1002:5a16]
IOMMU group 2 00:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 0) [1002:5a18]
IOMMU group 3 00:0b.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD990 PCI to PCI bridge (PCI Express GFX2 port 0) [1002:5a1f]
IOMMU group 4 00:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP2 Port 0) [1002:5a1e]
IOMMU group 5 00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1002:4391] (rev 40)
IOMMU group 6 00:12.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
IOMMU group 6 00:12.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
IOMMU group 7 00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
IOMMU group 7 00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
IOMMU group 8 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385] (rev 42)
IOMMU group 9 00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) [1002:4383] (rev 40)
 
Group 16 for the GT 730 looks fine. Are you using the AMD GPU for booting and the Proxmox host?
You could try enabling All functions or change the hostpci to hostpci0: 0000:03,pcie=1,romfile=GT730BIOS.rom to make sure the other parts of the multi-function device get passthrough as well to the same VM. Did you have any luck with dumping the rom in such a way (untouched) that is starts with hex value AA55 or not using the romfile option?
EDIT: The M5A99FX PRO R2.0 is a rather old motherboard, and I'm not sure if the AM3+ socket era supports PCI passthrough and IOMMU isolation properly.
 
Last edited:
Group 16 for the GT 730 looks fine. Are you using the AMD GPU for booting and the Proxmox host?
You could try enabling All functions or change the hostpci to hostpci0: 0000:03,pcie=1,romfile=GT730BIOS.rom to make sure the other parts of the multi-function device get passthrough as well to the same VM. Did you have any luck with dumping the rom in such a way (untouched) that is starts with hex value AA55 or not using the romfile option?
EDIT: The M5A99FX PRO R2.0 is a rather old motherboard, and I'm not sure if the AM3+ socket era supports PCI passthrough and IOMMU isolation properly.
I am using the AMD GPU for booting proxmox. We have had All functions enabled since the beginning. I tried changing the line in the qm.conf file and the issue i run into is that everytime I start the VM it works 1 time. turn the vm off and back on and nothing. it just does nothing. the GPU appears to not be resetting. I dumped the rom using nvflash as trying to "echo rom >> file" reported input/output error (this is without having anything touch the rom.

if my motherboard/cpu does not support it properly is there any way to make it at least work?
 
All funtions enabled and your VM configuration file don't match. Please make sure to enable it and/or change to VM configuration file (remove the .00). I guess if dumping the ROM file does not work, then don't use the romfile option. Can you provide links to people on the internet who got a GT 730 working with passthrough? Can you provide links to people with AM3(+) systems to who got GPU passthrough to work? I don't want to be a pessimist, but sometimes it just does not work. Have you tried a Linux VM (Ubuntu 20.04 Live ISO)? At least them we might get informative error messages and/or people might have fixed drivers to work with Nvida PCI passthrough.
 
Group 16 for the GT 730 looks fine. Are you using the AMD GPU for booting and the Proxmox host?
You could try enabling All functions or change the hostpci to hostpci0: 0000:03,pcie=1,romfile=GT730BIOS.rom to make sure the other parts of the multi-function device get passthrough as well to the same VM. Did you have any luck with dumping the rom in such a way (untouched) that is starts with hex value AA55 or not using the romfile option?
EDIT: The M5A99FX PRO R2.0 is a rather old motherboard, and I'm not sure if the AM3+ socket era supports PCI passthrough and IOMMU isolation properly.
Just Re-exported the BIOS of the card and also reboot the host with no VMs starting. Using cmp I compared the new rom with the old rom and they are 100% identical. Any other ideas?
 
All funtions enabled and your VM configuration file don't match. Please make sure to enable it and/or change to VM configuration file (remove the .00). I guess if dumping the ROM file does not work, then don't use the romfile option. Can you provide links to people on the internet who got a GT 730 working with passthrough? Can you provide links to people with AM3(+) systems to who got GPU passthrough to work? I don't want to be a pessimist, but sometimes it just does not work. Have you tried a Linux VM (Ubuntu 20.04 Live ISO)? At least them we might get informative error messages and/or people might have fixed drivers to work with Nvida PCI passthrough.
adding hostpci0: 0000:03,pcie=1 just causes it to remove itself from the config. unless I did something wrong.

Config looks like
Code:
agent: 1
args: -cpu 'host,hv-vendor-id=linustorvaldswasright'
bios: ovmf
boot: order=sata2
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: local:110/vm-110-disk-1.qcow2,size=128K
hostpci0: 0000:03:00,pcie=1,x-vga=1,romfile=GT730BIOS.rom
machine: pc-q35-5.2
memory: 12288
name: Gaming
net0: e1000=42:EA:DE:AB:24:FF,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
sata2: local:iso/systemrescue-8.03-amd64.iso,media=cdrom,size=717M
scsihw: virtio-scsi-pci
smbios1: uuid=941a3c18-d3be-4cfe-a868-1f32eccd92c5
sockets: 1
usb0: host=046d:c52b
vga: none
virtio0: local:110/vm-110-disk-0.qcow2,size=100G
vmgenid: a23b6631-9254-43db-a87a-121446e8201f

attempting to load linux does that. Just as screwed up and would not load beyond that point.
 

Attachments

  • image0 (2).jpeg
    image0 (2).jpeg
    991.4 KB · Views: 18
You're right 0000:03:00 is already correct and enables All functions. Have you tried it without the args (which might interfere with the regular cpu and x-vga=1 Proxmox settings)? Are you sure the GT 730 does not have a hardware problem? Does it look normal if you boot (a LiveCD) from it without virtualization and without the AMD GPU? Did you fix the ROM image in the meantime? Sorry, I'm running out of ideas to check or try.
 
You're right 0000:03:00 is already correct and enables All functions. Have you tried it without the args (which might interfere with the regular cpu and x-vga=1 Proxmox settings)? Are you sure the GT 730 does not have a hardware problem? Does it look normal if you boot (a LiveCD) from it without virtualization and without the AMD GPU? Did you fix the ROM image in the meantime? Sorry, I'm running out of ideas to check or try.
Yes we have tried without the args. The nvidia gpu should not have a hardware problem. I had it working not to long ago. As for booting a live cd i cant do that right now. I reextrached the rom image from the card using nvflash as it is the only thing that seems to be able to extract it and compared it to the original image. It is an exact 100% match.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!