Freezes with Gigabyte GB-BRR7H-4800 (rev. 1.0)

fsc

Member
Oct 4, 2021
21
3
8
25
I have been experiencing random freezes for a while now with this computer. It is a perfect device for proxmox due to its high power and very low energy consumption.

Freeze-ups occur approximately every 1 to 2 days.

System:
  • Gigabyte GB-BRR7H-4800 (rev. 1.0)
  • 64 GB RAM: 2x Kingston KVR26S19D8/32 SO-DIMM DDR4 2666Mhz 32GB C
  • NVMe: Samsung 980 SSD 1TB PCIe 3.0 NVMe M.2
  • SSD: SanDisk 1TB

I've done:
  • memtest86 for 20h. No errors
  • I have put additional ventilation for the device to lower the temperature (no component rises above 40ºC).
  • Alternative Linux Kernels.
  • Change the installation drive from NVMe to SSD and vice versa.
  • Stress the CPU and RAM of several VMs to try to force the freeze. This does not cause freezes of any kind.
With these tests I can say that there are no problems with the hardware.

During some freezes I have been able to have a screen connected. The screen simply freezes and does not show any errors. I can't enter commands or interact


In syslog I see this the last time it happened (The logs on the different freezes do not seem to be related and each log is different):

Code:
Jul 29 05:17:01 pve CRON[715550]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 29 05:17:01 pve CRON[715551]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 29 05:17:01 pve CRON[715550]: pam_unix(cron:session): session closed for user root
Jul 29 06:06:47 pve systemd[1]: Starting Daily apt upgrade and clean activities...
Jul 29 06:06:47 pve systemd[1]: apt-daily-upgrade.service: Succeeded.
Jul 29 06:06:47 pve systemd[1]: Finished Daily apt upgrade and clean activities.
Jul 29 06:12:31 pve postfix/qmgr[1127]: 426884C083A: from=<root@pve.xxxxxxx.es>, size=1140, nrcpt=1 (queue active)
Jul 29 06:12:32 pve postfix/smtp[734720]: connect to gmail-smtp-in.l.google.com[2a00:1450:400c:c00::1a]:25: Network is unreachable
Jul 29 06:13:02 pve postfix/smtp[734720]: connect to gmail-smtp-in.l.google.com[173.194.76.27]:25: Connection timed out
Jul 29 06:13:32 pve postfix/smtp[734720]: connect to alt1.gmail-smtp-in.l.google.com[142.250.153.27]:25: Connection timed out
Jul 29 06:13:32 pve postfix/smtp[734720]: connect to alt1.gmail-smtp-in.l.google.com[2a00:1450:4013:c16::1a]:25: Network is unreachable
Jul 29 06:13:32 pve postfix/smtp[734720]: connect to alt2.gmail-smtp-in.l.google.com[2a00:1450:4025:c03::1a]:25: Network is unreachable
Jul 29 06:13:32 pve postfix/smtp[734720]: 426884C083A: to=<xxxxxx@gmail.com>, relay=none, delay=183295, delays=183234/0.01/61/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[2a00:1450:4025:c03::1a]:25: Network is unreachable)
Jul 29 06:17:01 pve CRON[736270]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 29 06:17:01 pve CRON[736271]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 29 06:17:01 pve CRON[736270]: pam_unix(cron:session): session closed for user root
Jul 29 06:25:01 pve CRON[739027]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 29 06:25:01 pve CRON[739028]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Jul 29 06:25:01 pve CRON[739027]: pam_unix(cron:session): session closed for user root
Jul 29 06:42:37 pve systemd: systemd-journald.service: Main process exited, code=killed, status=6/ABRT
Jul 29 06:46:06 pve systemd: systemd-journald.service: Failed with result 'watchdog'.
Jul 29 06:47:14 pve systemd: systemd-journald.service: Consumed 58.594s CPU time.
-- Reboot --
Jul 29 16:38:33 pve kernel: Linux version 5.15.30-2-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.15.30-3 (Fri, 22 Apr 2022 18:08:27 +0200) ()
Jul 29 16:38:33 pve kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.30-2-pve root=/dev/mapper/pve-root ro quiet

To "fix it" I press and hold the power button on the device.

I'm going crazy with this, I've been trying to fix it for weeks. I love Proxmox and really don't want to switch to xcp-ng or another platform.
 
Your PVE is upgraded to the latest version? Because looks like postfix can't connect to the internet so PVE also maybe won't show any available updates.
 
Yes, the system is up to date. I don't use postfix at all, it has the default configuration. I don't think that's the problem, the freeze comes half an hour after those attempts.

My versions:
Code:
proxmox-ve: 7.2-1 (running kernel: 5.15.30-2-pve)
pve-manager: 7.2-3 (running version: 7.2-3/c743d6c1)
pve-kernel-helper: 7.2-2
pve-kernel-5.15: 7.2-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-8
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-6
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.2-2
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.1.8-1
proxmox-backup-file-restore: 2.1.8-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-10
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-1
pve-ha-manager: 3.3-4
pve-i18n: 2.7-1
pve-qemu-kvm: 6.2.0-5
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1
 
My versions:
proxmox-ve: 7.2-1 (running kernel: 5.15.30-2-pve)
Do you have a subscription and is the enterprise repository so far behind, or was it never updated after installation? What is the output of apt update?

I'm going crazy with this, I've been trying to fix it for weeks. I love Proxmox and really don't want to switch to xcp-ng or another platform.
Do those other platforms run flawlessly on your system? Have you had no issues with other operating systems or an Ubuntu Live CD?
 
Last edited:
  • Like
Reactions: Neobin
Yes, the system is up to date.

It is not. You probably have not added the pve-no-subscription repository:

If you do not have an active subscription [1], you need to disable the pve-enterprise repository [2] and add the pve-no-subscription one [3].

Do not forget to reboot the host after the update. (Ever use apt full-upgrade or apt dist-upgrade; never apt upgrade with Proxmox products!)

I would also check, if the bios/UEFI and the SSD-firmware of both are up-to-date.

[1] https://proxmox.com/en/proxmox-ve/pricing
[2] https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_enterprise_repo
[3] https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_no_subscription_repo
 
Last edited:
It is not. You probably have not added the pve-no-subscription repository:

If you do not have an active subscription [1], you need to disable the pve-enterprise repository [2] and add the pve-no-subscription one [3].

Do not forget to reboot the host after the update. (Ever use apt full-upgrade or apt dist-upgrade; never apt upgrade with Proxmox products!)

I would also check, if the bios/UEFI and the SSD-firmware of both are up-to-date.

[1] https://proxmox.com/en/proxmox-ve/pricing
[2] https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_enterprise_repo
[3] https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_no_subscription_repo

Thanks, you are right, I forgot to add the pve-no-subscription repositories in this installation. In other previous test installations I did upgrade correctly with pve-no-subscription, although I had always done it with apt upgrade.

The BIOS is updated. The SSD firmware also has the latest version.

Now I have everything updated (I think):
Code:
proxmox-ve: 7.2-1 (running kernel: 5.15.39-2-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-7
pve-kernel-helper: 7.2-7
pve-kernel-5.15.39-2-pve: 5.15.39-2
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.5-1
proxmox-backup-file-restore: 2.2.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.5-1
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.5-pve1

I tried XCP-ng for a few hours and it did not freeze, although I may have to leave it for a few days, just to confirm. If it freezes again I will install the latest version of ubuntu to test.
 
Today there are two new freezes: in less than 12 hours:

Code:
Jul 29 23:31:12 pve systemd[1]: user@0.service: Succeeded.
Jul 29 23:31:12 pve systemd[1]: Stopped User Manager for UID 0.
Jul 29 23:31:12 pve systemd[1]: Stopping User Runtime Directory /run/user/0...
Jul 29 23:31:12 pve systemd[1]: run-user-0.mount: Succeeded.
Jul 29 23:31:12 pve systemd[1]: user-runtime-dir@0.service: Succeeded.
Jul 29 23:31:12 pve systemd[1]: Stopped User Runtime Directory /run/user/0.
Jul 29 23:31:12 pve systemd[1]: Removed slice User Slice of UID 0.
Jul 29 23:42:08 pve postfix/qmgr[1191]: 426884C083A: from=<root@pve.felipesanchezweb.es>, size=1140, nrcpt=1 (queue active)
Jul 29 23:42:09 pve postfix/smtp[79145]: connect to gmail-smtp-in.l.google.com[2a00:1450:400c:c06::1b]:25: Network is unreachable
Jul 29 23:42:39 pve postfix/smtp[79145]: connect to gmail-smtp-in.l.google.com[66.102.1.27]:25: Connection timed out
Jul 29 23:42:39 pve postfix/smtp[79145]: connect to alt1.gmail-smtp-in.l.google.com[2a00:1450:4013:c16::1b]:25: Network is unreachable
Jul 29 23:43:09 pve postfix/smtp[79145]: connect to alt1.gmail-smtp-in.l.google.com[142.250.153.26]:25: Connection timed out
Jul 29 23:43:09 pve postfix/smtp[79145]: connect to alt2.gmail-smtp-in.l.google.com[2a00:1450:4025:c03::1a]:25: Network is unreachable
Jul 29 23:43:09 pve postfix/smtp[79145]: 426884C083A: to=<felipesanchezcalzada@gmail.com>, relay=none, delay=246272, delays=246211/0.01/61/0, dsn=4.4.1, status=deferred (connect to alt2.gmail-smtp-in.l.google.com[2a00:1450:4025:c03::1a]:25: Network is unreachable)
-- Reboot --
Jul 30 15:32:42 pve kernel: Linux version 5.15.39-2-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.15.39-2 (Wed, 20 Jul 2022 17:22:19 +0200) ()
Jul 30 15:32:42 pve kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.39-2-pve root=/dev/mapper/pve-root ro quiet
Jul 30 15:32:42 pve kernel: KERNEL supported cpus:
Jul 30 15:32:42 pve kernel:   Intel GenuineIntel
Jul 30 15:32:42 pve kernel:   AMD AuthenticAMD
Jul 30 15:32:42 pve kernel:   Hygon HygonGenuine
Jul 30 15:32:42 pve kernel:   Centaur CentaurHauls
Jul 30 15:32:42 pve kernel:   zhaoxin   Shanghai

Code:
Jul 30 17:53:43 pve postfix/smtp[43081]: connect to alt1.gmail-smtp-in.l.google.com[142.250.153.27]:25: Connection timed out
Jul 30 17:54:13 pve postfix/smtp[43080]: connect to alt2.gmail-smtp-in.l.google.com[142.251.9.27]:25: Connection timed out
Jul 30 17:54:13 pve postfix/smtp[43081]: connect to alt2.gmail-smtp-in.l.google.com[142.251.9.27]:25: Connection timed out
Jul 30 17:54:43 pve postfix/smtp[43080]: connect to alt3.gmail-smtp-in.l.google.com[142.250.150.27]:25: Connection timed out
Jul 30 17:54:43 pve postfix/smtp[43081]: connect to alt3.gmail-smtp-in.l.google.com[142.250.150.27]:25: Connection timed out
Jul 30 17:55:13 pve postfix/smtp[43080]: connect to alt4.gmail-smtp-in.l.google.com[74.125.200.27]:25: Connection timed out
Jul 30 17:55:13 pve postfix/smtp[43080]: 426884C083A: to=<felipesanchezcalzada@gmail.com>, relay=none, delay=311796, delays=311645/0.01/151/0, dsn=4.4.1, status=deferred (connect to alt4.gmail-smtp-in.l.google.com[74.125.200.27]:25: Connection timed out)
Jul 30 17:55:13 pve postfix/smtp[43081]: connect to alt4.gmail-smtp-in.l.google.com[74.125.200.27]:25: Connection timed out
Jul 30 17:55:13 pve postfix/smtp[43081]: 91F084C089C: to=<felipesanchezcalzada@gmail.com>, relay=none, delay=90994, delays=90843/0.01/151/0, dsn=4.4.1, status=deferred (connect to alt4.gmail-smtp-in.l.google.com[74.125.200.27]:25: Connection timed out)
Jul 30 18:17:01 pve CRON[51425]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 30 18:17:01 pve CRON[51426]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 30 18:17:01 pve CRON[51425]: pam_unix(cron:session): session closed for user root
-- Reboot --
Jul 30 20:49:17 pve kernel: Linux version 5.15.39-2-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.15.39-2 (Wed, 20 Jul 2022 17:22:19 +0200) ()
Jul 30 20:49:17 pve kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.39-2-pve root=/dev/mapper/pve-root ro quiet
Jul 30 20:49:17 pve kernel: KERNEL supported cpus:
Jul 30 20:49:17 pve kernel:   Intel GenuineIntel
Jul 30 20:49:17 pve kernel:   AMD AuthenticAMD
Jul 30 20:49:17 pve kernel:   Hygon HygonGenuine
Jul 30 20:49:17 pve kernel:   Centaur CentaurHauls
Jul 30 20:49:17 pve kernel:   zhaoxin   Shanghai


I'm going to install Ubuntu on the same SSD that I now have Proxmox on and report the results... (Removing Proxmox)
 
I finally found out what was causing the problems. It seems to be the fault of a USB zigbee dongle that I use in home Assistant (SONOFF-ZB Zigbee 3.0 Dongle).
In my opinion it is still a Proxmox problem since it should not freeze under any circumstances due to a USB, and less a USB that is not damaged

However, I can bear that little problem because I love proxmox.
 
After a few days unfortunately it happened again... it seems that the hardware is just shit. I talked to Gigabyte and the only thing they say is that they don't support that operating system. DO NOT BUY THIS OR ANY GIGABYTE MINI PC, it's just poorly designed crap with a BIOS that will never update even though it obviously has serious bugs.
 
fsc, I think I might have a solution and testing it right now. Basic I've replaced the CMOS battery. I remember having a similar issue with an old intel nuc and just replacing the battery fixed the issue.

Will post later in the week to see how long I go without and glitches.
 
More to add, so I believe these CPU's don't like to stay at over 4GHz for a long time. The VRMs get really hot and apparently they are placed right under the interface controller and for some reason creates a broadcast storm when hot for too long.

Added thermal tape and heat sinks to the VRM and interface controller and having a outside fan cool the unit helps. Also setting the P-State to 2, which on my stress test limits the CPU to 3.1Ghz, help stabilize this system

1664718500913.png

Hopefully I can get this stable, I was planning to have this as a personal proxmox server.
 
Last edited:
More to add, so I believe these CPU's don't like to stay at over 4GHz for a long time. The VRMs get really hot and apparently they are placed right under the interface controller and for some reason creates a broadcast storm when hot for too long.

Added thermal tape and heat sinks to the VRM and interface controller and having a outside fan cool the unit helps. Also setting the P-State to 2, which on my stress test limits the CPU to 3.1Ghz, help stabilize this system

View attachment 41861

Hopefully I can get this stable, I was planning to have this as a personal proxmox server.
I see from your screenshot that you are using Windows. With windows everything works fine, the problem is to install any operating system other than windows.

I don't think the battery is a problem, it is a device that is only a few months old, regarding setting the P-Stats on 2 I have not tried it, I will try to sell the machine at a good price this week, if I don't get it maybe I will try your solutions.

Change the battery is a problem, they use one of those with wires attached instead of a normal one on the motherboard. All wrong gigabyte...
 
Last edited:
I'm having the same problem with the BRR7H 4800, although for me freezes happen several times a day. Was messing around for weeks trying to get it to run stable, but have pretty much given up at this point. Somehow it even makes my NAS completely freeze, when Proxmox freeze happens during copying files over to the NAS.
 
Last edited:
I actually had two of these, it seems like the VRMs on these are not cooled well enough or not at all. My first one was running for a month or two then noticed it crashing from time to time, to the point where it's almost unuable.

My second one was not used at the time, but when I noticed that VRMs would overheat, I actually had to put a heatsink and fan on the VRMs. Basically open the top portion of the case to have some airflow to the VRMs.

Another thing to note, is that the interface chip could be a bad chip. After doing the cooling mod on my second brix, the interface would act weird and some times crash the entire network. Must be creating a broadcast storm. I disabled the onboard nic and used a usb to ethernet adapter. So far its been stable.

To any new owner of a BRR7H 4800, either return it while you can or mod it and put a heatsink and fan on the VRMs then disable the onboard nic.
 
Hello,
Just want to add some feedback regarding this issue.
After multiple crash and test, (I even put a usb fan and cut the case), the issue seem to be related to power supply.
When adding usb devices, the system got more unstable, when I remove the usb, the system became more stable (I still got crash when I have heavy CPU load.
I am looking to change the power supply to see if it will solve the instability.
 
Hello,
I have the same problem with my Minisforum HM90.
I am trying to figure it out for weeks but it still keeps crashing.
The power supply is a good idea, I haven’t been tested that. I will give it a try.
 
Hola,
Solo quiero agregar algunos comentarios sobre este problema.
Después de múltiples fallas y pruebas (incluso puse un ventilador USB y corté la carcasa), el problema parece estar relacionado con la fuente de alimentación.
Al agregar dispositivos USB, el sistema se volvió más inestable, cuando eliminé el USB, el sistema se volvió más estable (todavía se bloquea cuando tengo una gran carga de CPU.
Estoy buscando cambiar la fuente de alimentación para ver si se soluciona la inestabilidad.
That would be weird, the system seems to randomly freeze even without any cpu usage, and when running windows pushing cpu and gpu to max it never crashes.

Anyway, I'm interested to know your results
 
I have the same issue with a fully upgraded no-subscription Proxmox 7.3 installation but with an ASRock Rack E3C242D4U mainboard, Intel Xeon E-2176G CPU, 4x Kingston 32 GB DDR4-2666 MHz ECC UDIMM modules, a Samsung SSD 970 EVO Plus 2TB connected via M.2 and 3x Seagate IronWolf 8TB HDD's (ST8000VN004-3CP101) connected to SATA_2 - SATA_4. The setup is powered by a new Corsair RM550x 80 Plus 550 Watt ATX power supply.

After 5-35 minutes the host system freezes with a single VM running a single-core, 2GB RAM Ubuntu 22.04 server providing a Pi-Hole installation to my network. Ballooning is turned off, CPU is set to host, NUMA is enabled, qemu-agent enabled and async-io set to native (all as recommended in other threads I have skimmed through). While I'm able to connect via IPMI I can't connect to the VM or the host either through the in-build KVM or via SSH. Even connecting a monitor and keyboard to the rig directly doesn't respond to any key strokes and the only option left is to reset/power-off the system. Without an installed or running VM I didn't notice a freeze yet so the freeze might be related to how Proxmox handles the VM.

While I do see a couple of random spikes in the summary, none of them gets even close to the maximum. The temperature of the rig is also somewhere between 35 and 42° C and therefore heat shouldn't be an issue. I was exchanging emails with ASRock the past few days in an attempt to spot the problem and they made me tear the whole rig apart multiple times and their conclusion is/was that it's not a problem with the mainboard and therefore out of their scope. I even installed Ubuntu 22.04 desktop bare-metal and didn't notice a freeze in a couple of hours, though I have to admit that I didn't do anything fancy here. My setup even "survived" a memtest86 running for 2 days.
---
edit: Ok, I seem to have found a setting in the BIOS that allowed the CPU to take in less power when not under load that turns out to be problematic in regards to Proxmox. Once I disabled this setting the system seems to run stable now.
 
Last edited:
Hi there,
@whitehatmiddleman: would you mind sharing pics of your mod (heatsink on vrm)?
not really clear to me which parts you are cooling, but as i am experiencing freezes as well, I would be interested to apply the mod. Is your BRIX still running stable?
@all: still wondering if this is related to proxmox or the hardware… did anyone try and verify if the freezes do not happen e.g. with windows as OS?

TIA
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!