PVE 5.2: netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.

micro

Renowned Member
Nov 28, 2014
60
21
73
After dist-upgrading one of my cluster nodes to PVE 5.2 the following messages started to show up in the syslog/dmesg:

grep length /var/log/syslog

May 26 06:58:18 node1 kernel: [ 30.711141] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 06:58:18 node1 kernel: [ 30.786683] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 06:58:18 node1 kernel: [ 30.858183] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 06:58:19 node1 kernel: [ 31.176130] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 06:58:22 node1 kernel: [ 33.979768] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 06:58:23 node1 kernel: [ 35.703723] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 06:58:23 node1 kernel: [ 35.853023] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 06:58:23 node1 kernel: [ 35.901700] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 06:58:23 node1 kernel: [ 35.950775] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 06:58:24 node1 kernel: [ 36.000079] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 07:00:03 node1 kernel: [ 135.447675] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 07:00:04 node1 kernel: [ 136.074886] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 07:00:04 node1 kernel: [ 136.600086] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 07:03:12 node1 kernel: [ 324.287308] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 07:03:13 node1 kernel: [ 324.807152] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 07:03:13 node1 kernel: [ 325.334061] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
May 26 07:03:14 node1 kernel: [ 325.863852] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.

I think this message has something to do with the new kernel version 4.15
 
Last edited:
  • Like
Reactions: chrone
Hi,

can you please send you network config?
 
Hi,

I'm afraid I may sound weird, but I don't feel comfortable sharing my prod config here (even sanitized). Moreover I consider it way too complicated with many interfaces, extra OVS options and mirror ports to be able to point you guys in the right direction.

I also believe it is not because of something specific to our network config, because I found that some other PVE user (with way more simple config than mine) also have such messages after upgrading to 5.2: please see https://forum.proxmox.com/threads/ovs-warnung-nach-update-auf-5-2.43935/

His network config is included, and the only similar with ours is that he is using OVS too. He is using bonds, we don't. I translated it with google translate and it looks like there aren't any clues why this is happening there as well.

Because of all the above I'm attempting to reproduce this message on a new PVE 5.2 setup in VM (nested) - so far without luck. I also tried to extract from our config only the OVS bridge in question with all the extra OVS parameters and member ports but the message doesn't appear in the VM test setup. I also attempted to use the same config as user pixelpeter (from the german forum thread above), but again - the message doesn't appear in the VM. Maybe it is because I'm attempting to reproduce it in a nested PVE setup (VM) and not in the original hardware.

In our original cluster node this message appears in the syslog/dmesg only when VM with interfaces attached to this OVS bridge is started (one message for each VM interface). So far it looks the this OVS bridge is working fine, except for this message which appeared after the PVE 5.2 upgrade (with kernel 4.15). I'm still in hold and not upgrading my other cluster nodes to 5.2 because I don't feel safe with this message. OVS is critical for this cluster and I cannot afford any mistakes or tests.
 
I'm seeing the same thing!


[ 12.708936] Netfilter messages via NETLINK v0.30.
[ 13.129192] audit: type=1400 audit(1527669328.323:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=3049 comm="apparmor_parser"
[ 13.147092] audit: type=1400 audit(1527669328.339:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=3048 comm="apparmor_parser"
[ 13.147095] audit: type=1400 audit(1527669328.339:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=3048 comm="apparmor_parser"
[ 13.147108] audit: type=1400 audit(1527669328.339:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=3048 comm="apparmor_parser"
[ 13.147111] audit: type=1400 audit(1527669328.339:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=3048 comm="apparmor_parser"
[ 13.238414] new mount options do not match the existing superblock, will be ignored
[ 13.277948] softdog: initialized. soft_noboot=0 soft_margin=60 sec soft_panic=0 (nowayout=0)
[ 13.591159] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[ 13.698288] openvswitch: Open vSwitch switching datapath
[ 14.349405] bnx2x 0000:05:00.0 enp5s0f0: using MSI-X IRQs: sp 128 fp[0] 130 ... fp[7] 137
[ 14.487613] bnx2x 0000:05:00.0 enp5s0f0: Warning: Unqualified SFP+ module detected, Port 0 from DIREKTRONIK part number P.8596.02
[ 14.491791] bnx2x 0000:05:00.0 enp5s0f0: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
[ 14.646591] device ovs-system entered promiscuous mode
[ 14.668139] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 14.800075] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 14.800331] device vmbr0 entered promiscuous mode
[ 14.932696] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 14.932746] device enp4s0f0 entered promiscuous mode
[ 14.933107] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 14.933136] device enp4s0f1 entered promiscuous mode
[ 14.935096] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 14.935274] device bond0 entered promiscuous mode
[ 15.055871] IPv6: ADDRCONF(NETDEV_UP): enp4s0f0: link is not ready
[ 15.173122] IPv6: ADDRCONF(NETDEV_UP): enp4s0f1: link is not ready
[ 15.213081] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 15.213269] device vlan956 entered promiscuous mode
[ 15.593355] audit: type=1400 audit(1527669330.787:7): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/bin/lxc-start" pid=4334 comm="apparmor_parser"
[ 15.599965] audit: type=1400 audit(1527669330.791:8): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default" pid=4339 comm="apparmor_parser"
[ 15.599978] audit: type=1400 audit(1527669330.791:9): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-cgns" pid=4339 comm="apparmor_parser"
[ 15.599980] audit: type=1400 audit(1527669330.791:10): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-with-mounting" pid=4339 comm="apparmor_parser"
[ 15.599982] audit: type=1400 audit(1527669330.791:11): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-container-default-with-nesting" pid=4339 comm="apparmor_parser"
[ 18.564039] tg3 0000:04:00.0 enp4s0f0: Link is up at 1000 Mbps, full duplex
[ 18.564041] tg3 0000:04:00.0 enp4s0f0: Flow control is off for TX and off for RX
[ 18.564043] tg3 0000:04:00.0 enp4s0f0: EEE is enabled
[ 18.564057] IPv6: ADDRCONF(NETDEV_CHANGE): enp4s0f0: link becomes ready
[ 18.864240] ip_set: protocol 6
[ 19.605451] ip6_tables: (C) 2000-2006 Netfilter Core Team
[ 20.017156] Ebtables v2.0 registered
[ 23.529119] device tap100i0 entered promiscuous mode
[ 23.545076] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 27.674195] device tap101i0 entered promiscuous mode
[ 27.691166] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 28.116425] device tap101i1 entered promiscuous mode
[ 28.133756] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 28.531660] device tap101i2 entered promiscuous mode
[ 28.548142] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 33.236752] sdf: sdf1
[ 33.239093] sdg: sdg1
[ 33.240809] sdh: sdh1
[ 33.753691] device tap103i0 entered promiscuous mode
[ 33.772559] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 34.233294] device tap103i1 entered promiscuous mode
[ 34.250903] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 38.250902] sdi: sdi1
[ 38.253285] sdj: sdj1
[ 38.255174] sdk: sdk1
[ 38.782393] device tap104i0 entered promiscuous mode
[ 38.802122] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 39.265538] device tap104i1 entered promiscuous mode
[ 39.283291] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 43.770636] device tap105i0 entered promiscuous mode
[ 43.816029] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 44.271615] device tap105i1 entered promiscuous mode
[ 44.289268] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 48.854765] device tap106i0 entered promiscuous mode
[ 48.873224] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 49.340283] device tap106i1 entered promiscuous mode
[ 49.361228] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 53.897752] device tap107i0 entered promiscuous mode
[ 53.915284] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 54.365203] device tap107i1 entered promiscuous mode
[ 54.386564] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 58.974764] device tap108i0 entered promiscuous mode
[ 58.992926] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
[ 59.444764] device tap108i1 entered promiscuous mode
[ 59.464060] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length.
 
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage part of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp4s0f0 inet manual

iface enp5s0f1 inet manual

iface enp5s0f2 inet manual

iface enp5s0f3 inet manual

iface enp5s0f4 inet manual

iface enp5s0f5 inet manual

iface enp5s0f6 inet manual

iface enp5s0f7 inet manual

iface idrac inet manual

iface enp4s0f1 inet manual

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto enp5s0f0
iface enp5s0f0 inet static
address 10.10.10.2
netmask 255.255.255.0

# Bond enp4s0f0 and enp4s0f1 together
allow-vmbr0 bond0
iface bond0 inet manual
ovs_bridge vmbr0
ovs_type OVSBond
ovs_bonds enp4s0f0 enp4s0f1
# Force the MTU of the physical interfaces to be jumbo-frame capable.
# This doesn't mean that any OVSIntPorts must be jumbo-capable.
# We cannot, however set up definitions for eth0 and eth1 directly due
# to what appear to be bugs in the initialization process.
pre-up ( ifconfig enp4s0f0 mtu 1500 && ifconfig enp4s0f1 mtu 1500 )
ovs_options bond_mode=balance-tcp lacp=active other_config:lacp-time=fast

# Bridge for our bond and vlan virtual interfaces (our VMs will
# also attach to this bridge)
auto vmbr0
allow-ovs vmbr0
iface vmbr0 inet manual
ovs_type OVSBridge
# NOTE: we MUST mention bond0, vlanx, and vlanx even though each
# of them lists ovs_bridge vmbr0! Not sure why it needs this
# kind of cross-referencing but it won't work without it!
ovs_ports bond0 vlan111 vlan222 vlan333 vlan1
#mtu 8996

allow-vmbr0 vlan111
iface vlan111 inet static
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_options tag=111
ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif
address xxx.xxx.xxx.xxx
netmask xxx.xxx.xxx.xxx
gateway xxx.xxx.xxx.xxx
#mtu 8996
 
Same problem, but works...(knock on the wood).

allow-auto ens1f0
iface ens1f0 inet manual

allow-auto ens1f1
iface ens1f1 inet manual

allow-vmbr0 bond0
iface bond0 inet manual
ovs_bridge vmbr0
ovs_type OVSBond
ovs_bonds ens1f0 ens1f1
ovs_options lacp=active bond-mode=balance-tcp

auto vmbr0
iface vmbr0 inet manual
ovs_type OVSBridge
ovs_ports bond0 nameif

allow-vmbr0 nameif
iface nameif inet static
address *
netmask *
gateway *
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_options tag=*
#node management
 
I will check what this exactly means and if this could a problem.
Will report if I found something.
 
  • Like
Reactions: micro
On a quick look, there might be a kernel commit between 4.13 and 4.15 that could have triggered it. Does the message disappear if you run the 4.13 kernel?
 
Hi,
I see the same...

my network-config:
Code:
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

auto eth2.3928
iface eth2.3928 inet manual

auto eth2.3927
iface eth2.3927 inet manual

iface eth3 inet manual

auto eth3.3928
iface eth3.3928 inet manual

auto eth3.3927
iface eth3.3927 inet manual

iface eth4 inet manual                                                                                                                  
iface eth5 inet manual
#                                                                                                                                                 
#pve-cluster + mgmt
#
# avlan1000
allow-cluster bond0 
iface bond0 inet manual                                                                                                                
        ovs_bonds eth0 eth1
        ovs_type OVSBond 
        ovs_bridge cluster  
        ovs_options bond_mode=active-backup 

auto cluster 
iface cluster inet manual 
        ovs_type OVSBridge
        ovs_ports bond0 avlan1000 vlan3926

allow-cluster avlan1000
iface avlan1000 inet static
        address  10.10.2.16
        netmask  255.255.255.0
        gateway  10.10.2.1
        ovs_type OVSIntPort
        ovs_bridge cluster
        ovs_options tag=1000

allow-cluster vlan3926
iface vlan3996 inet static
        address  10.10.3.16
        netmask  255.255.255.128
        ovs_type OVSIntPort
        ovs_bridge cluster
        ovs_options tag=3926

#
#vlans fuer VM
#
allow-vmbr0 bond1
iface bond1 inet manual
        ovs_bonds eth4 eth5
        ovs_type OVSBond
        ovs_bridge vmbr0
        ovs_options bond_mode=active-backup

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond1

#
# ceph public + cluster
#
allow-vmbr1 bond2
iface bond2 inet manual
        ovs_bonds eth2.3928 eth3.3928
        set-active-slave bond2 eth2.3928
        ovs_type OVSBond
        ovs_bridge vmbr1
        ovs_options bond_mode=active-backup

auto vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports bond2 vlan3928
        pre-up ( ifconfig eth2 mtu 9000 && ifconfig eth2.3928 mtu 9000 && ifconfig eth3 mtu 9000 && ifconfig eth3.3928 mtu 9000 )
        mtu 9000

allow-vmbr1 vlan3928
iface vlan3928 inet static
        address 10.20.2.16
        netmask 255.255.248.0
        ovs_type OVSIntPort
        ovs_bridge vmbr1
        pre-up ( ifconfig eth2 mtu 9000 && ifconfig eth2.3928 mtu 9000 && ifconfig eth3 mtu 9000 && ifconfig eth3.3928 mtu 9000 )
        mtu 9000

auto bond3
iface bond3 inet static
        bond-downdelay 200
        bond-updelay 200
        bond-miimon 100
        bond-mode active-backup
        bond-slaves eth3.3927 eth2.3927
        bond-primary eth3.3927
        pre-up ( ifconfig eth2 mtu 9000 && ifconfig eth3 mtu 9000 )
        address 10.10.3.136
        netmask 255.255.255.128
        mtu 9000
Udo
 
  • Like
Reactions: micro
Hi,
I see the same...

my network-config:
Code:
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

auto eth2.3928
iface eth2.3928 inet manual

auto eth2.3927
iface eth2.3927 inet manual

iface eth3 inet manual

auto eth3.3928
iface eth3.3928 inet manual

auto eth3.3927
iface eth3.3927 inet manual

iface eth4 inet manual                                                                                                                 
iface eth5 inet manual
#                                                                                                                                                
#pve-cluster + mgmt
#
# avlan1000
allow-cluster bond0
iface bond0 inet manual                                                                                                               
        ovs_bonds eth0 eth1
        ovs_type OVSBond
        ovs_bridge cluster 
        ovs_options bond_mode=active-backup

auto cluster
iface cluster inet manual
        ovs_type OVSBridge
        ovs_ports bond0 avlan1000 vlan3926

allow-cluster avlan1000
iface avlan1000 inet static
        address  10.10.2.16
        netmask  255.255.255.0
        gateway  10.10.2.1
        ovs_type OVSIntPort
        ovs_bridge cluster
        ovs_options tag=1000

allow-cluster vlan3926
iface vlan3996 inet static
        address  10.10.3.16
        netmask  255.255.255.128
        ovs_type OVSIntPort
        ovs_bridge cluster
        ovs_options tag=3926

#
#vlans fuer VM
#
allow-vmbr0 bond1
iface bond1 inet manual
        ovs_bonds eth4 eth5
        ovs_type OVSBond
        ovs_bridge vmbr0
        ovs_options bond_mode=active-backup

auto vmbr0
iface vmbr0 inet manual
        ovs_type OVSBridge
        ovs_ports bond1

#
# ceph public + cluster
#
allow-vmbr1 bond2
iface bond2 inet manual
        ovs_bonds eth2.3928 eth3.3928
        set-active-slave bond2 eth2.3928
        ovs_type OVSBond
        ovs_bridge vmbr1
        ovs_options bond_mode=active-backup

auto vmbr1
iface vmbr1 inet manual
        ovs_type OVSBridge
        ovs_ports bond2 vlan3928
        pre-up ( ifconfig eth2 mtu 9000 && ifconfig eth2.3928 mtu 9000 && ifconfig eth3 mtu 9000 && ifconfig eth3.3928 mtu 9000 )
        mtu 9000

allow-vmbr1 vlan3928
iface vlan3928 inet static
        address 10.20.2.16
        netmask 255.255.248.0
        ovs_type OVSIntPort
        ovs_bridge vmbr1
        pre-up ( ifconfig eth2 mtu 9000 && ifconfig eth2.3928 mtu 9000 && ifconfig eth3 mtu 9000 && ifconfig eth3.3928 mtu 9000 )
        mtu 9000

auto bond3
iface bond3 inet static
        bond-downdelay 200
        bond-updelay 200
        bond-miimon 100
        bond-mode active-backup
        bond-slaves eth3.3927 eth2.3927
        bond-primary eth3.3927
        pre-up ( ifconfig eth2 mtu 9000 && ifconfig eth3 mtu 9000 )
        address 10.10.3.136
        netmask 255.255.255.128
        mtu 9000
Udo
nice config!
 
As far I can say it is only a new warning in the kernel.[1]
I think the error was handled before by ovs and now we have it in the syslog.
So I would say it is harmless but I will check if we can get rid of this warning.

1.) linux kernel "git commit 6e237d099fac1f73a7b6d7287bb9191f29585a4e"
 
  • Like
Reactions: chrone and micro
Any news on this one? I see the same error in Proxmox 5.2-2 (upgraded from 4.4).

pve-manager/5.2-9/4b30e8f9 (running kernel: 4.15.18-5-pve)

IDK if it is related, but after an upgrade I get terrible migration speed (28-40MB/s) over 10Gbps with openvswitch. Before that it was 400-500MB/s.
And "migration: type=insecure" was replaced in datacenter.cfg (old parameter was "migration_unsecure: 1")
 
Any news on this one? I see the same error in Proxmox 5.2-2 (upgraded from 4.4).

pve-manager/5.2-9/4b30e8f9 (running kernel: 4.15.18-5-pve)

IDK if it is related, but after an upgrade I get terrible migration speed (28-40MB/s) over 10Gbps with openvswitch. Before that it was 400-500MB/s.
And "migration: type=insecure" was replaced in datacenter.cfg (old parameter was "migration_unsecure: 1")


Likewise, we've started to see this error on our nodes, and we're seeing fairly poor network performance too with 4.5.18-5 and pve-manager/5.2-9/4b30e8f9



root@pve-7:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet static
address 172.16.88.1
netmask 255.255.255.0
#Proxmox Private

allow-vmbr0 bond0
iface bond0 inet manual
ovs_bonds enp5s0f0 enp5s0f1
ovs_type OVSBond
ovs_bridge vmbr0
ovs_options lacp=active bond_mode=balance-tcp

auto vmbr0
iface vmbr0 inet manual
ovs_type OVSBridge
ovs_ports bond0 vlan603 vlan850 vlan855

allow-vmbr0 vlan603
iface vlan603 inet static
address x.x.x.
netmask m.m.m.m
gateway g.g.g.g
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_options tag=603

allow-vmbr0 vlan850
iface vlan850 inet static
address 172.16.50.247
netmask 255.255.255.0
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_options tag=850

allow-vmbr0 vlan855
iface vlan855 inet static
address y.y.y.y
netmask 255.255.255.192
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_options tag=855


running bonded 2 x 10Gig NICS in a Dell R610 ... rbd/ceph over vlan 850, public access on vlan603
 
also started seeing this in 5.2 & 5.3.

It *seems* as if the ovs-switchd is a reason systemd didn't want to restart, with those eerors (but could've been fun/troub;es with swapoff... I hate systemd)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!