Proxmox VE 8.0 released!

  • Like
Reactions: Neobin and dMopp
the_ma4rio said:
hello how do u fix the network problemsß
Click to expand...
Hi the_ma4rio, as I saw it was a kernel issue, I did revert the nuc that had the issues to use Proxmox 7.4, and have still not upgraded it to 8.X but am eager to test again as I see there is a new kernel 6.5 released and talked about, I did buy the dummy hdmi plugs from amazon that fake a display as a backup plan and well the day i do upgrade and leave 7.X and if the issue still is present, I will use the dummy plug on it. which seem to fix the issue... but a very bad workaround offcourse so hopefully the new kernel has solved the issues I got.

and offcourse will try to renaming /lib/firmware/i915/kbl_dmc_ver1_04.bin prior to using the dummy plug as talked about in the post above :)
https://forum.proxmox.com/threads/p...ues-with-hardware-transcoding-in-plex.132187/

In the the issue report here they do also mention adding kernelflag of i915.enable_dc=0 which might solve it.

Interestingly I stumbled upon this thread about passthrough today aswell and as the display issue might also be solved if blacklisting i915 as talked about in that thread as then it wont be allowed to load into the kernel.

I do use the passthrough of the integrated graphics myself which works verywell on a headless nuc into a vm then into a docker container.
 
Last edited:
Hi team,

We just finished upgrading to version 8....
We are running 3 node cluster with Ceph, we are using no-subscription repo on this cluster.

Syslog on all nodes has tons of the following

Code: 
Nov 18 19:38:40 pvelw11 ceph-crash[2163]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-11-17T17:42:59.187044Z_5d617bc9-8bbe-45f6-8f69-2b46318e0e39 as client.admin failed: 2023-11-18T19:38:39.986+0000 7f2d3d26f6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Nov 18 19:38:40 pvelw11 ceph-crash[2163]: 2023-11-18T19:38:39.994+0000 7f2d3d26f6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Nov 18 19:38:40 pvelw11 ceph-crash[2163]: 2023-11-18T19:38:39.994+0000 7f2d3d26f6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Nov 18 19:38:40 pvelw11 ceph-crash[2163]: 2023-11-18T19:38:39.994+0000 7f2d3d26f6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
Nov 18 19:38:40 pvelw11 ceph-crash[2163]: 2023-11-18T19:38:39.994+0000 7f2d3d26f6c0 -1 monclient: keyring not found
Nov 18 19:38:40 pvelw11 ceph-crash[2163]: [errno 13] RADOS permission denied (error connecting to the cluster)

The file is in place, dont see any permission issues (same as other cluster we have).

Code: 
root@pvelw11:/etc/pve/priv# ls -la
total 5
drwx------ 2 root www-data    0 Sep 22  2022 .
drwxr-xr-x 2 root www-data    0 Jan  1  1970 ..
drwx------ 2 root www-data    0 Sep 22  2022 acme
-rw------- 1 root www-data 1675 Nov 18 07:16 authkey.key
-rw------- 1 root www-data 1573 Nov 17 17:46 authorized_keys
drwx------ 2 root www-data    0 Sep 23  2022 ceph
-rw------- 1 root www-data  151 Sep 23  2022 ceph.client.admin.keyring
-rw------- 1 root www-data  228 Sep 23  2022 ceph.mon.keyring
-rw------- 1 root www-data 4500 Nov 17 17:46 known_hosts
drwx------ 2 root www-data    0 Sep 22  2022 lock
drwx------ 2 root www-data    0 Oct 19  2022 metricserver
-rw------- 1 root www-data 3243 Sep 22  2022 pve-root-ca.key
-rw------- 1 root www-data    3 Oct 19  2022 pve-root-ca.srl
drwx------ 2 root www-data    0 Sep 30  2022 storage
-rw------- 1 root www-data    2 Jul  2 16:10 tfa.cfg

Here's the content of the crash report

Code: 
root@pvelw11:~# cat /var/lib/ceph/crash/2023-11-17T17\:42\:59.187044Z_5d617bc9-8bbe-45f6-8f69-2b46318e0e39/meta
{
    "crash_id": "2023-11-17T17:42:59.187044Z_5d617bc9-8bbe-45f6-8f69-2b46318e0e39",
    "timestamp": "2023-11-17T17:42:59.187044Z",
    "process_name": "ceph-osd",
    "entity_name": "osd.5",
    "ceph_version": "17.2.6",
    "utsname_hostname": "pvelw11",
    "utsname_sysname": "Linux",
    "utsname_release": "5.15.131-1-pve",
    "utsname_version": "#1 SMP PVE 5.15.131-2 (2023-11-14T11:32Z)",
    "utsname_machine": "x86_64",
    "os_name": "Debian GNU/Linux 12 (bookworm)",
    "os_id": "12",
    "os_version_id": "12",
    "os_version": "12 (bookworm)",
    "assert_condition": "end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout",
    "assert_func": "int OSD::shutdown()",
    "assert_file": "./src/osd/OSD.cc",
    "assert_line": 4368,
    "assert_thread_name": "signal_handler",
    "assert_msg": "./src/osd/OSD.cc: In function 'int OSD::shutdown()' thread 7f1e97a32700 time 2023-11-17T17:42:59.177646+0000\n./src/osd/OSD.cc: 4368: FAILED ceph_assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)\n",
    "backtrace": [
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7f1e9b8ed140]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x55b20d071042]",
        "/usr/bin/ceph-osd(+0xc25186) [0x55b20d071186]",
        "(OSD::shutdown()+0x1364) [0x55b20d169764]",
        "(SignalHandler::entry()+0x648) [0x55b20d7f2dc8]",
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7) [0x7f1e9b8e1ea7]",
        "clone()"
    ]
}

Dont see any issues with Ceph status

Code: 
root@pvelw11:~# ceph -s
  cluster:
    id:     a447dbaf-a9ea-442f-a072-cb5b333afe73
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum pvelw13,pvelw12,pvelw11 (age 26h)
    mgr: pvelw13(active, since 26h), standbys: pvelw12, pvelw11
    osd: 6 osds: 6 up (since 26h), 6 in (since 13M)

  data:
    pools:   2 pools, 33 pgs
    objects: 462.67k objects, 1.7 TiB
    usage:   4.5 TiB used, 30 TiB / 35 TiB avail
    pgs:     33 active+clean

  io:
    client:   273 KiB/s rd, 23 MiB/s wr, 18 op/s rd, 3.10k op/s wr

Looking for assistance please.

Thanks
 
It seems that the message has basically existed for a long time and was also present in PVE 7. In PVE 8 the message has changed slightly.

Proxmox 7
Code: 
Nov 12 00:07:57 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:14.367104Z_5cb93760-31f1-42a9-8870-8952b645e1b4 as client.crash.prox1 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:57 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:14.367104Z_5cb93760-31f1-42a9-8870-8952b645e1b4 as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:57 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:14.367104Z_5cb93760-31f1-42a9-8870-8952b645e1b4 as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:57 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:21.854428Z_44f940d8-d0c8-4b3e-be51-89a02b92cdc8 as client.crash.prox1 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:58 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:21.854428Z_44f940d8-d0c8-4b3e-be51-89a02b92cdc8 as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:58 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:21.854428Z_44f940d8-d0c8-4b3e-be51-89a02b92cdc8 as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:58 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:17.703242Z_de30260e-ba98-4079-b077-2b2a88fde277 as client.crash.prox1 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:58 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:17.703242Z_de30260e-ba98-4079-b077-2b2a88fde277 as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:58 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:17.703242Z_de30260e-ba98-4079-b077-2b2a88fde277 as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:59 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:14.387004Z_0710b1c0-9d96-4053-bfbc-41f4480f4c04 as client.crash.prox1 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:59 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:14.387004Z_0710b1c0-9d96-4053-bfbc-41f4480f4c04 as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:59 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:14.387004Z_0710b1c0-9d96-4053-bfbc-41f4480f4c04 as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:59 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:13.510591Z_7442ea8e-e48d-4cdc-99e6-96ee82567cec as client.crash.prox1 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:07:59 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:13.510591Z_7442ea8e-e48d-4cdc-99e6-96ee82567cec as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:08:00 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:13.510591Z_7442ea8e-e48d-4cdc-99e6-96ee82567cec as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:08:00 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:18.056916Z_42ca60c2-43d4-4954-ae4c-750a93bc27ec as client.crash.prox1 failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:08:00 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:18.056916Z_42ca60c2-43d4-4954-ae4c-750a93bc27ec as client.crash failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Nov 12 00:08:00 prox1 ceph-crash[2547]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-05-28T12:47:18.056916Z_42ca60c2-43d4-4954-ae4c-750a93bc27ec as client.admin failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

After Upgrading to PVE 8
Code: 
2023-11-18T11:28:44.473021+01:00 prox1 ceph-crash[2376]: 2023-11-18T11:28:44.463+0100 7fabbd4a76c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
2023-11-18T11:28:44.473069+01:00 prox1 ceph-crash[2376]: 2023-11-18T11:28:44.463+0100 7fabbd4a76c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
2023-11-18T11:28:44.473101+01:00 prox1 ceph-crash[2376]: 2023-11-18T11:28:44.463+0100 7fabbd4a76c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.admin.keyring: (13) Permission denied
2023-11-18T11:28:44.473129+01:00 prox1 ceph-crash[2376]: 2023-11-18T11:28:44.463+0100 7fabbd4a76c0 -1 monclient: keyring not found
2023-11-18T11:28:44.473172+01:00 prox1 ceph-crash[2376]: [errno 13] RADOS permission denied (error connecting to the cluster)

//EDIT:

@hepo there is another Thread: https://forum.proxmox.com/threads/ceph-crash-problem.128423
 
Last edited:
  • Like
Reactions: hepo
Another issue we observe, VMs are becoming non-responsive (cannot ssh to them), the following messages are displayed on the console

1700410192141.png

I cannot reboot the VM cleanly as it lost connection to the storage...
 
  • Like
Reactions: dgk
Just noticed bunch of patches released in the non-sub repo, which I rushed in deploying.
The quickest/lamest way to detect non-responsive systems for me was to check if IP address is detected (as the qemu agent is not running).
After patching all VMs appear to be ok. Not sure if rebooting the nodes/cluster did the job.
I will continue monitoring.

Code: 
proxmox-ve: 8.0.2 (running kernel: 6.2.16-19-pve)
pve-manager: 8.0.9 (running version: 8.0.9/fd1a0ae1b385cdcd)
proxmox-kernel-helper: 8.0.5
pve-kernel-5.15: 7.4-8
pve-kernel-5.13: 7.1-9
pve-kernel-5.11: 7.0-10
proxmox-kernel-6.2.16-19-pve: 6.2.16-19
proxmox-kernel-6.2: 6.2.16-19
pve-kernel-5.15.131-1-pve: 5.15.131-2
pve-kernel-5.15.108-1-pve: 5.15.108-2
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph: 17.2.7-pve1
ceph-fuse: 17.2.7-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx6
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.1
libpve-access-control: 8.0.6
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.10
libpve-guest-common-perl: 5.0.5
libpve-http-server-perl: 5.0.5
libpve-rs-perl: 0.8.7
libpve-storage-perl: 8.0.4
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.4-1
proxmox-backup-file-restore: 3.0.4-1
proxmox-kernel-helper: 8.0.5
proxmox-mail-forward: 0.2.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.2
proxmox-widget-toolkit: 4.1.1
pve-cluster: 8.0.5
pve-container: 5.0.5
pve-docs: 8.0.5
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.0.7
pve-qemu-kvm: 8.1.2-2
pve-xtermjs: 5.3.0-2
qemu-server: 8.0.8
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.0-pve3
 
Last edited:
The issue continues (randomly)...
We have noticed that migrating the VM to different node fixes the problem i.e. VM is responsive again immediately (issue yesterday was resolved by the migration, not rebooting the cluster nodes, nor patches).
We have implemented detection mechanism to understand better when this is happening.

Need help with guidance how to troubleshoot this, please
 
So far, I haven't been able to find any such problems on a PVE / CEPH cluster with 8.0.4 and Quincy (only the syslog messages mentioned). Does it only affect this one VM? Can you paste the config of it?
 
Random VMs, it also looks like this is happening after backup (early morning), which I need to confirm once again

All VMs are configured in similar way, this VM was hanging this morning
Code: 
agent: 1,fstrim_cloned_disks=1
boot: order=scsi0;net0
cores: 32
cpu: x86-64-v2-AES
memory: 65536
name: prod-lws141-dbcl41
net0: virtio=46:8D:E1:6D:4F:D6,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: ceph:vm-4141-disk-0,discard=on,iothread=1,size=32G,ssd=1
scsi1: ceph:vm-4141-disk-1,discard=on,iothread=1,size=100G,ssd=1
scsi2: ceph:vm-4141-disk-2,discard=on,iothread=1,size=40G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=f50aafdb-6fa0-4a38-8fef-0e7d57c1d8a7
sockets: 1
vmgenid: b05f52c0-4203-43c7-a4be-f841fe7fdae5
 
In connection with CEPH, I have not been able to see any advantages with the "ssd" and "iothread" flags so far. Maybe you can disable these flags and see if it still occurs?

Do you have corresponding metrics from your CEPH cluster so that any abnormalities become visible?
 
virtio-scsi-single with iothreads was deemed better for our database servers long time ago when doing performance testing...
can definitely give it a try but need to understand how to reproduce the problem (e.g. target to particular vm)

sb-jw said:
Do you have corresponding metrics from your CEPH cluster so that any abnormalities become visible?
Click to expand...

Can you please expand what do you mean with this?
 
hepo said:
virtio-scsi-single with iothreads was deemed better for our database servers long time ago when doing performance testing...
Click to expand...
That's really funny, I tested it several times with FIO in different configurations and never achieved more performance or stability.
hepo said:
can definitely give it a try but need to understand how to reproduce the problem (e.g. target to particular vm)
Click to expand...
I would suspect that the flags clash with each other, possibly causing a hiccup that results in this problem, or that there is a bug in these flags that lead to such behavior.
Our starting point is quite similar, except that your problems are not visible on my infrastructure and I don't use these two flags. It's just a guess, but it could of course also be nonsense.
hepo said:
Can you please expand what do you mean with this?
Click to expand...
By this I mean something like IOPS, bandwidth, PG status, latency visualized in Grafana (or similar tool). For example, I query these parameters about every 10 seconds and have a pretty good picture of what's going on in my cluster. I would never want to live without metrics like that again.
 
Thanks for the response... I would love to understand what monitoring you have implemented, sounds really good.
We only collect standard proxmox metrics -> influx -> grafana...

This cluster is really really quiet, we use it as hot standby to out production environment, and also for testing new stuff (PVE 8).
I did found rados bench results stored somewhere, I will re-run the test to compare (although the initial test was performed on an empty cluster).
I was also thinking to switch to KRBD instead of librdb...
 
hepo said:
Thanks for the response... I would love to understand what monitoring you have implemented, sounds really good.
We only collect standard proxmox metrics -> influx -> grafana...
Click to expand...
We also use the metrics from PVE. With this we can create an overview to find out whether a VM is spinning or generating a high load.

For CEPH's metrics we use the Python script from https://pypi.org/project/collectd-ceph-storage/. Of course, a collectd runs on every node that has then activated the Graphite Connector.

However, you have to adapt the script a bit because it no longer fully runs under Python 3.11. We have also optimized the collection of metrics; the nodes themselves only send the OSD latencies to the Graphite-Go server. The other values such as cluster IOPS (ceph -w for example), pool utilization etc. are taken over by a VM “SNMP Collector”. There are several collectd instances set up on the SNMP Collector, one of which has also stored the Python script and then pulls the information, which can be queried once. This way we reduce the load on the nodes and still get our 10 second interval.

But keep in mind that the Graphite server should have enough storage space and IOPS. There should also be no VM on the CEPH storage so that you have metrics to track in the event of an error.
We have set up a dedicated server with PVE for this purpose, which does not run in a Cluster. He got 6x 240 GB Samsung SM863a from us that run in HW-RAID10 (PERC H730, it at least recognizes the SSDs, it's been running like this for a few years) with cache. The VM continuously writes to disk at 10 MiB/s.

I created a small graphic for you so that you can understand our structure.

1700509951144.png
 
  • Like
Reactions: hepo
I need to come back to this...
Did additional validation and testing as follows:
  • OSD bench is consistent, no issues to report
  • Rados bench shows slightly better results compared to the tests we keep record of 2 years ago
  • Did fio testing in the VM and compared to previous results we have - no issue
Conclusion is that there is no problem with Ceph.

We also managed to confirm that the issue is cased by the backup, we are using PBS, datastore on TrueNAS Core server over NFS.
Triggered two of-schedule backups that caused few random VMs to freeze, some databases (although still responding via ssh) were damaged.
We have stopped the backups at the moment to confirm stability over the next few days.
During backup it looks the VM loose their IO completely (migration to different host appears to unfreeze the IO)

1700642508438.png

Speaking of backup we also noticed that the "--mailnotification failure" is no longer regarded.
We are getting emails for successful backups, and started getting two emails for the same job as follows

1700643107008.png

I see many posts on this to which I will hop on, some examples:
https://forum.proxmox.com/threads/issues-during-backups-vms-blocked-and-corrupted.129152/
https://forum.proxmox.com/threads/guest-agent-fs-freeze-command-breaks-the-system-on-backup.69605/

Not convinced this is related to PVE8 upgrade, stopping the spam in this thread.

All the best!
 
Last edited:
  • Like
Reactions: sb-jw
FWIW, I could move the post chain between you and sb-jw out to a separate thread if you want?
Not only w.r.t. crowding the release thread but also because there's some valuable info in there that others might easier find if located in their own thread.
 
  • Like
Reactions: hepo
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom