[SOLVED] Extremely slow upload speed without compression on a 10G nic

Merguez

New Member
Jul 15, 2024
13
1
3
EDIT:
Probably a hardware issue, I need to order new gear to check if that's the real issue. So it's [SOLVED] for now even though there's no solution.

Hi, I'm writing this after spending my weekend on testing stuff to fix this behaviour.
I installed pve on my homelab which is a 40 CPU Xeon Gold 6138 @2.00GHz with 128GB ECC RAM.
PVE has its own nvme to run from, and there are two zfs pool: one with SSD (2x2TB), one with HDDS (3x18TB)
10G nic is plugged in PCIe and bridged to vmbr0 using the UI.
Here are two scp commands, one without compression and one with compression - please note that the results are the same using python simplehttpserver, etc.

Bash:
╭─merguez@MBP ~
╰─$ scp root@192.168.0.10:1G .                                                                                                                                                                        255 ↵
1G                                                                                                                                                                          0%  255KB 254.7KB/s 1:08:36 ETA^C%
╭─merguez@MBP ~
╰─$ scp -C root@192.168.0.10:1G .                                                                                                                                                                       1 ↵
1G                                                                                                                                                                          5%   51MB  25.9MB/s   00:37 ETA^C%

The code above is from my laptop through a vpn (I'm outside now). But the results are the same with higher values with compression from my LAN and from my router which also has 10G nic.

I thought it was the disks, but compressing then sending also implies to read from the disk.

Right now, I have reinstalled proxmox completely, and the results are the same, I'm lost…

Any help is appreciated!
Thank you very much!
 
Last edited:
I though I've written it.
I get the full bandwidth with it, here's one through my vpn, (it's capped on my current laptop wifi, but it's still way higher than sending a file without compression).

Code:
$ iperf3-darwin -c 192.168.0.10
Connecting to host 192.168.0.10, port 5201
[  5] local 192.168.27.65 port 60125 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd          RTT
[  5]   0.00-1.00   sec  10.1 MBytes  84.3 Mbits/sec  1182   2.51 MBytes   255ms
[  5]   1.00-2.00   sec  5.72 MBytes  48.1 Mbits/sec  255    229 KBytes   33ms
[  5]   2.00-3.00   sec  7.86 MBytes  65.9 Mbits/sec    0    248 KBytes   27ms
[  5]   3.00-4.00   sec  8.59 MBytes  72.1 Mbits/sec    0    285 KBytes   30ms
[  5]   4.00-5.00   sec  9.19 MBytes  77.0 Mbits/sec    0    324 KBytes   42ms
[  5]   5.00-6.00   sec  7.99 MBytes  67.0 Mbits/sec    0    355 KBytes   38ms
[  5]   6.00-7.00   sec  8.70 MBytes  73.0 Mbits/sec    0    384 KBytes   25ms
[  5]   7.00-8.00   sec  11.4 MBytes  95.5 Mbits/sec    0    422 KBytes   33ms
[  5]   8.00-9.00   sec  11.5 MBytes  96.7 Mbits/sec    0    456 KBytes   32ms
[  5]   9.00-10.00  sec  10.3 MBytes  86.3 Mbits/sec    0    484 KBytes   50ms
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  91.3 MBytes  76.6 Mbits/sec  1437             sender
[  5]   0.00-10.02  sec  90.8 MBytes  76.0 Mbits/sec                  receiver






root@pve:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.27.65, port 60124
[  5] local 192.168.0.10 port 5201 connected to 192.168.27.65 port 60125
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  6.32 MBytes  53.0 Mbits/sec
[  5]   1.00-2.00   sec  9.07 MBytes  76.1 Mbits/sec
[  5]   2.00-3.00   sec  7.74 MBytes  64.9 Mbits/sec
[  5]   3.00-4.00   sec  8.63 MBytes  72.4 Mbits/sec
[  5]   4.00-5.00   sec  9.13 MBytes  76.6 Mbits/sec
[  5]   5.00-6.00   sec  7.92 MBytes  66.5 Mbits/sec
[  5]   6.00-7.00   sec  8.62 MBytes  72.3 Mbits/sec
[  5]   7.00-8.00   sec  11.2 MBytes  93.7 Mbits/sec
[  5]   8.00-9.00   sec  11.6 MBytes  97.4 Mbits/sec
[  5]   9.00-10.00  sec  10.2 MBytes  85.7 Mbits/sec
[  5]  10.00-10.02  sec   363 KBytes   127 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.02  sec  90.8 MBytes  76.0 Mbits/sec                  receiver
-----------------------------------------------------------

[EDIT] From my router I have the full 10G bandwidth (1100MB/s) but I can't access the iperf client on it right now
 
Last edited:
I'll need to wait for coming home tonight. But consider the results to be the same, except a much higher speed with compression. During my debug weekend, I never went higher than 700KB/s without compression on the lan without vpn.
My router hosts my vpn server (wireguard) and has a 8G wan with 8G subscription. My server is directly plugged in my router using an sfp dac cable and a 10G network card on both sides.
I use this VPN to stream gaming when I'm outside, I know it's an additional potential issue, but it's more or less the same as being directly on the lan.
 
I went through a lot of debugging, so I didn't post anything to let people give new ideas. Here is the output with details:


Code:
04:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
        DeviceName: Intel LAN i210AT #1
        Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection
        Flags: bus master, fast devsel, latency 0, IRQ 18, NUMA node 0, IOMMU group 32
        Memory at aa200000 (32-bit, non-prefetchable) [size=512K]
        I/O ports at 2000 [size=32]
        Memory at aa280000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number ac-1f-6b-ff-ff-ee-87-70
        Capabilities: [1a0] Transaction Processing Hints
        Kernel driver in use: igb
        Kernel modules: igb

05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
        DeviceName: Intel LAN i210AT #2
        Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection
        Flags: bus master, fast devsel, latency 0, IRQ 19, NUMA node 0, IOMMU group 33
        Memory at aa100000 (32-bit, non-prefetchable) [size=512K]
        I/O ports at 1000 [size=32]
        Memory at aa180000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number ac-1f-6b-ff-ff-ee-87-71
        Capabilities: [1a0] Transaction Processing Hints
        Kernel driver in use: igb
        Kernel modules: igb

67:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Network Connection (rev 01)
        Subsystem: Beijing Sinead Technology Co., Ltd. 82599 10 Gigabit Network Connection
        Flags: bus master, fast devsel, latency 0, IRQ 28, NUMA node 0, IOMMU group 6
        Memory at 388000000000 (64-bit, prefetchable) [size=512K]
        I/O ports at b000 [size=32]
        Memory at 388000080000 (64-bit, prefetchable) [size=16K]
        Expansion ROM at e0c00000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [e0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 98-b7-85-ff-ff-1e-ec-ef
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe
 
retry scp without vpn.
Here it is (directly on my LAN through wifi on a MacBook):

Code:
root@pve:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.0.103, port 63155
[  5] local 192.168.0.10 port 5201 connected to 192.168.0.103 port 63156
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  18.5 MBytes   155 Mbits/sec
[  5]   1.00-2.00   sec  19.6 MBytes   164 Mbits/sec
[  5]   2.00-3.00   sec  14.2 MBytes   119 Mbits/sec
[  5]   3.00-4.00   sec  14.9 MBytes   125 Mbits/sec
[  5]   4.00-5.00   sec  30.8 MBytes   258 Mbits/sec
[  5]   5.00-6.00   sec  38.8 MBytes   325 Mbits/sec
[  5]   6.00-7.00   sec  39.8 MBytes   334 Mbits/sec
[  5]   7.00-8.00   sec  38.5 MBytes   323 Mbits/sec
[  5]   8.00-9.00   sec  39.6 MBytes   333 Mbits/sec
[  5]   9.00-10.00  sec  38.7 MBytes   325 Mbits/sec
[  5]  10.00-10.03  sec  1008 KBytes   314 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.03  sec   295 MBytes   247 Mbits/sec                  receiver

Code:
$ iperf3-darwin -c 192.168.0.10                                                              255 ↵
Connecting to host 192.168.0.10, port 5201
[  5] local 192.168.0.103 port 63156 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd          RTT
[  5]   0.00-1.00   sec  20.8 MBytes   174 Mbits/sec    0   1.50 MBytes   77ms
[  5]   1.00-2.00   sec  19.9 MBytes   167 Mbits/sec    0   1.58 MBytes   101ms
[  5]   2.00-3.00   sec  14.2 MBytes   119 Mbits/sec    0   1.65 MBytes   113ms
[  5]   3.00-4.00   sec  14.8 MBytes   124 Mbits/sec    0   1.57 MBytes   76ms
[  5]   4.00-5.00   sec  30.5 MBytes   256 Mbits/sec    0   1.59 MBytes   53ms
[  5]   5.00-6.00   sec  38.9 MBytes   326 Mbits/sec    0   1.60 MBytes   37ms
[  5]   6.00-7.00   sec  40.2 MBytes   338 Mbits/sec    0   1.61 MBytes   34ms
[  5]   7.00-8.00   sec  38.1 MBytes   319 Mbits/sec    0   1.63 MBytes   45ms
[  5]   8.00-9.00   sec  39.8 MBytes   333 Mbits/sec    0   1.67 MBytes   40ms
[  5]   9.00-10.00  sec  38.9 MBytes   327 Mbits/sec    0   1.72 MBytes   46ms
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   296 MBytes   248 Mbits/sec    0             sender
[  5]   0.00-10.03  sec   295 MBytes   247 Mbits/sec                  receiver

iperf Done.

EDIT: And here is an iperf public server to show that my wifi is the bottleneck:

Code:
$ iperf3-darwin -c ping.online.net
Connecting to host ping.online.net, port 5201
[  7] local 192.168.0.103 port 63260 connected to 51.158.1.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd          RTT
[  7]   0.00-1.00   sec  21.3 MBytes   179 Mbits/sec    0   2.37 MBytes   177ms
[  7]   1.00-2.00   sec  19.8 MBytes   166 Mbits/sec    0   2.56 MBytes   108ms
[  7]   2.00-3.00   sec  17.8 MBytes   149 Mbits/sec    0   1.78 MBytes   134ms
[  7]   3.00-4.00   sec  21.2 MBytes   178 Mbits/sec    0   2.25 MBytes   127ms
[  7]   4.00-5.00   sec  19.9 MBytes   167 Mbits/sec    0   2.09 MBytes   79ms
[  7]   5.00-6.00   sec  19.2 MBytes   161 Mbits/sec    0   2.09 MBytes   90ms
[  7]   6.00-7.00   sec  19.1 MBytes   160 Mbits/sec    0   2.10 MBytes   91ms
[  7]   7.00-8.00   sec  19.7 MBytes   165 Mbits/sec    0   1.98 MBytes   78ms
[  7]   8.00-9.00   sec  19.7 MBytes   165 Mbits/sec    0   1.95 MBytes   82ms
[  7]   9.00-10.00  sec  19.1 MBytes   160 Mbits/sec    0   2.03 MBytes   116ms
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  7]   0.00-10.00  sec   197 MBytes   165 Mbits/sec    0             sender
[  7]   0.00-10.06  sec   195 MBytes   162 Mbits/sec                  receiver

iperf Done.
 
Last edited:
Oh indeed… I don't know iperf3 well, sorry for not putting this straightaway.

Code:
$ iperf3-darwin -Rc 192.168.0.10
Connecting to host 192.168.0.10, port 5201
Reverse mode, remote host 192.168.0.10 is sending
[  5] local 192.168.0.103 port 63685 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bitrate         Rwnd
[  5]   0.00-1.01   sec   235 KBytes  1.91 Mbits/sec   128 KBytes
[  5]   1.01-2.01   sec  63.6 KBytes   521 Kbits/sec   128 KBytes
[  5]   2.01-3.01   sec   362 KBytes  2.97 Mbits/sec   128 KBytes
[  5]   3.01-4.00   sec   191 KBytes  1.57 Mbits/sec   128 KBytes
[  5]   4.00-5.01   sec   255 KBytes  2.08 Mbits/sec   128 KBytes
[  5]   5.01-6.01   sec   337 KBytes  2.76 Mbits/sec   128 KBytes
[  5]   6.01-7.01   sec  63.6 KBytes   521 Kbits/sec   128 KBytes
[  5]   7.01-8.01   sec   127 KBytes  1.04 Mbits/sec   128 KBytes
[  5]   8.01-9.01   sec   156 KBytes  1.27 Mbits/sec   128 KBytes
[  5]   9.01-10.01  sec   137 KBytes  1.12 Mbits/sec   128 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  1.98 MBytes  1.66 Mbits/sec  222             sender
[  5]   0.00-10.01  sec  1.88 MBytes  1.58 Mbits/sec                  receiver

iperf Done.


root@pve:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.0.103, port 63684
[  5] local 192.168.0.10 port 5201 connected to 192.168.0.103 port 63685
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   297 KBytes  2.43 Mbits/sec   36   1.41 KBytes
[  5]   1.00-2.00   sec  76.4 KBytes   626 Kbits/sec   12   1.41 KBytes
[  5]   2.00-3.00   sec   393 KBytes  3.22 Mbits/sec   43   1.41 KBytes
[  5]   3.00-4.00   sec   154 KBytes  1.26 Mbits/sec   19   2.83 KBytes
[  5]   4.00-5.00   sec   242 KBytes  1.98 Mbits/sec   21   1.41 KBytes
[  5]   5.00-6.00   sec   399 KBytes  3.27 Mbits/sec   35   1.41 KBytes
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    8   2.83 KBytes
[  5]   7.00-8.00   sec   158 KBytes  1.30 Mbits/sec   17   1.41 KBytes
[  5]   8.00-9.00   sec   157 KBytes  1.29 Mbits/sec   18   2.83 KBytes
[  5]   9.00-10.00  sec   156 KBytes  1.27 Mbits/sec   13   1.41 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  1.98 MBytes  1.66 Mbits/sec  222             sender
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
 
I reinstalled yesterday, so the latest one:

uname -a:
Linux pve 6.8.8-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-2 (2024-06-24T09:00Z) x86_64 GNU/Linux

PVE 8.2.4
 
try with PVE 7.4 and its Kernel 5.15 , on spare disk , just to confirm if ixgbe problem, there was some reports since 6.2 kernel.
 
UPDATE:

On new install in 7.4.1 using kernel 5.15.102-1-pve, there is still the issue.

I tried doing the dkms compilation on ixgbe 5.20.10 as @Kingneutron tried in this post, but no more luck neither: https://forum.proxmox.com/threads/intel-x553-sfp-ixgbe-no-go-on-pve8.135129/

Code:
root@pve:~# uname -a
Linux pve 5.15.102-1-pve #1 SMP PVE 5.15.102-1 (2023-03-14T13:48Z) x86_64 GNU/Linux

Code:
$ iperf3-darwin -Rc 192.168.0.10
Connecting to host 192.168.0.10, port 5201
Reverse mode, remote host 192.168.0.10 is sending
[  5] local 192.168.0.103 port 56557 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bitrate         Rwnd
[  5]   0.00-1.00   sec  1.01 MBytes  8.44 Mbits/sec   192 KBytes
[  5]   1.00-2.00   sec   536 KBytes  4.39 Mbits/sec   192 KBytes
[  5]   2.00-3.00   sec   646 KBytes  5.29 Mbits/sec   192 KBytes
[  5]   3.00-4.00   sec   848 KBytes  6.95 Mbits/sec   190 KBytes
[  5]   4.00-5.00   sec   728 KBytes  5.97 Mbits/sec   192 KBytes
[  5]   5.00-6.00   sec   229 KBytes  1.88 Mbits/sec   190 KBytes
[  5]   6.00-7.01   sec   318 KBytes  2.59 Mbits/sec   192 KBytes
[  5]   7.01-8.00   sec   605 KBytes  4.98 Mbits/sec   192 KBytes
[  5]   8.00-9.00   sec   953 KBytes  7.80 Mbits/sec   192 KBytes
[  5]   9.00-10.00  sec   498 KBytes  4.07 Mbits/sec   192 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  6.66 MBytes  5.58 Mbits/sec  418             sender
[  5]   0.00-10.00  sec  6.24 MBytes  5.23 Mbits/sec                  receiver

iperf Done.

$ iperf3-darwin -c 192.168.0.10
Connecting to host 192.168.0.10, port 5201
[  5] local 192.168.0.103 port 56562 connected to 192.168.0.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd          RTT
[  5]   0.00-1.00   sec  95.3 MBytes   798 Mbits/sec    0   6.03 MBytes   13ms
[  5]   1.00-2.00   sec  94.9 MBytes   797 Mbits/sec    0   6.03 MBytes   26ms
[  5]   2.00-3.00   sec  95.8 MBytes   803 Mbits/sec    0   6.04 MBytes   25ms
[  5]   3.00-4.00   sec  96.3 MBytes   809 Mbits/sec    0   6.04 MBytes   32ms
[  5]   4.00-5.00   sec  96.8 MBytes   812 Mbits/sec    0   6.06 MBytes   52ms
[  5]   5.00-6.00   sec   103 MBytes   861 Mbits/sec    0   6.06 MBytes   21ms
[  5]   6.00-7.00   sec  98.4 MBytes   826 Mbits/sec    0   6.06 MBytes   33ms
[  5]   7.00-8.00   sec   102 MBytes   855 Mbits/sec    0   6.06 MBytes   28ms
[  5]   8.00-9.00   sec  99.7 MBytes   836 Mbits/sec    0   6.06 MBytes   39ms
[  5]   9.00-10.00  sec   101 MBytes   846 Mbits/sec    0   6.06 MBytes   24ms
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   983 MBytes   824 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   980 MBytes   821 Mbits/sec                  receiver

iperf Done.
 
I tried both, same results… I guess it's no longer a Proxmox issue.

Code:
$ iperf3-darwin -Rc 192.168.0.107                                                              1 ↵
Connecting to host 192.168.0.107, port 5201
Reverse mode, remote host 192.168.0.107 is sending
[  5] local 192.168.0.103 port 62371 connected to 192.168.0.107 port 5201
[ ID] Interval           Transfer     Bitrate         Rwnd
[  5]   0.00-1.01   sec   469 KBytes  3.83 Mbits/sec   129 KBytes
[  5]   1.01-2.00   sec   731 KBytes  6.01 Mbits/sec   129 KBytes
[  5]   2.00-3.01   sec   338 KBytes  2.76 Mbits/sec   129 KBytes
[  5]   3.01-4.01   sec  67.9 KBytes   556 Kbits/sec   129 KBytes
[  5]   4.01-5.01   sec   165 KBytes  1.36 Mbits/sec   129 KBytes
[  5]   5.01-6.00   sec   443 KBytes  3.64 Mbits/sec   129 KBytes
[  5]   6.00-7.01   sec   123 KBytes  1.00 Mbits/sec   129 KBytes
[  5]   7.01-8.01   sec   173 KBytes  1.41 Mbits/sec   128 KBytes
[  5]   8.01-9.01   sec  77.8 KBytes   637 Kbits/sec   129 KBytes
[  5]   9.01-10.00  sec   256 KBytes  2.11 Mbits/sec   129 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  2.88 MBytes  2.41 Mbits/sec  224             sender
[  5]   0.00-10.00  sec  2.78 MBytes  2.33 Mbits/sec                  receiver

iperf Done.

Also, I have a PCIE nvme bay for 2 SSD, and my NIC is on the very next port. Could this bring issues?
 
Last edited:
Indeed… I'll try another slot for the nic first and unplugging the one for my SSD as well before doing windows stuff.
 
Hello again, didn't have time yesterday.

In a windows vm with the pcie adapter passed through, it's the same issue : https://www.speedtest.net/result/16517676413.png
I didn't bother trying iperf or anything local test, as I should have approximately the same speed in both directions at this hour of the day.

I reboot the server and try to only leave the pcie nic plugged in.
 
  • Like
Reactions: _gabriel

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!