Upgrade to 8.0 causes "NETDEV WATCHDOG: enp6s0 (igc): transmit queue 0 timed out"

BugVito

New Member
Jul 10, 2023
6
0
1
I have recently upgraded from proxmox 7.4 (kernel 6.2) to 8.0. I had no issues prior to the upgrade, and while not performing a speedtest, there are no issues.
When running speedtests from my main workstation (1.75Gbps pipe) my main pc loses connectivity (but the link is up) with the following being displayed in Proxmox:

Code:
Jul 10 08:27:17 proxmox kernel: ------------[ cut here ]------------
Jul 10 08:27:17 proxmox kernel: NETDEV WATCHDOG: enp6s0 (igc): transmit queue 0 timed out
Jul 10 08:27:17 proxmox kernel: WARNING: CPU: 9 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
Jul 10 08:27:17 proxmox kernel: Modules linked in: tcp_diag inet_diag nf_conntrack_netlink xt_nat xt_tcpudp xt_conntrack nft_chain_nat xt_MASQUERADE xfrm_user xfrm_algo xt_addrtype nft_compat overlay cmac nls_utf8 cifs cifs_arc4 rdma_cm iw_cm ib_cm ib_core cifs_md4 fscache netfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter scsi_transport_iscsi nf_tables nfnetlink_cttimeout bonding tls openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 softdog sunrpc nfnetlink_log nfnetlink binfmt_misc snd_hda_codec_hdmi snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus snd_soc_core snd_compress ac97_bus x86_pkg_temp_thermal snd_pcm_dmaengine intel_powerclamp i915 snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi
Jul 10 08:27:17 proxmox kernel:  kvm_intel drm_buddy ttm snd_hda_codec drm_display_helper kvm snd_hda_core crct10dif_pclmul polyval_clmulni cec polyval_generic snd_hwdep processor_thermal_device_pci ghash_clmulni_intel rc_core processor_thermal_device mei_pxp mei_hdcp sha512_ssse3 processor_thermal_rfim snd_pcm drm_kms_helper aesni_intel processor_thermal_mbox snd_timer processor_thermal_rapl cmdlinepart i2c_algo_bit intel_rapl_msr crypto_simd pmt_telemetry syscopyarea snd spi_nor mei_me pmt_class cryptd intel_rapl_common sysfillrect int3400_thermal intel_hid rapl intel_cstate pcspkr wmi_bmof soundcore mtd ee1004 sysimgblt mei int340x_thermal_zone intel_vsec igen6_edac acpi_pad sparse_keymap acpi_thermal_rel acpi_tad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap coretemp vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb
Jul 10 08:27:17 proxmox kernel:  uas usb_storage dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c nvme xhci_pci spi_intel_pci xhci_pci_renesas i2c_i801 crc32_pclmul intel_lpss_pci spi_intel i2c_smbus ahci nvme_core intel_lpss xhci_hcd igc libahci idma64 nvme_common video wmi pinctrl_tigerlake
Jul 10 08:27:17 proxmox kernel: CPU: 9 PID: 0 Comm: swapper/9 Tainted: P           O       6.2.16-3-pve #1
Jul 10 08:27:17 proxmox kernel: Hardware name: CWWK CW-MBX-AD12/MINIPC-G12, BIOS AD12R20230207 02/08/2023
Jul 10 08:27:17 proxmox kernel: RIP: 0010:dev_watchdog+0x23a/0x250
Jul 10 08:27:17 proxmox kernel: Code: 00 e9 2b ff ff ff 48 89 df c6 05 8a 6f 7d 01 01 e8 6b 08 f8 ff 44 89 f1 48 89 de 48 c7 c7 58 64 c0 ad 48 89 c2 e8 06 ab 30 ff <0f> 0b e9 1c ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00
Jul 10 08:27:17 proxmox kernel: RSP: 0018:ffff9d9c403a0e38 EFLAGS: 00010246
Jul 10 08:27:17 proxmox kernel: RAX: 0000000000000000 RBX: ffff8ce894426000 RCX: 0000000000000000
Jul 10 08:27:17 proxmox kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Jul 10 08:27:17 proxmox kernel: RBP: ffff9d9c403a0e68 R08: 0000000000000000 R09: 0000000000000000
Jul 10 08:27:17 proxmox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff8ce8944264c8
Jul 10 08:27:17 proxmox kernel: R13: ffff8ce89442641c R14: 0000000000000000 R15: 0000000000000000
Jul 10 08:27:17 proxmox kernel: FS:  0000000000000000(0000) GS:ffff8cf7ffa40000(0000) knlGS:0000000000000000
Jul 10 08:27:17 proxmox kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 10 08:27:17 proxmox kernel: CR2: 000000080123b128 CR3: 00000005f5010000 CR4: 0000000000752ee0
Jul 10 08:27:17 proxmox kernel: PKRU: 55555554
Jul 10 08:27:17 proxmox kernel: Call Trace:
Jul 10 08:27:17 proxmox kernel:  <IRQ>
Jul 10 08:27:17 proxmox kernel:  ? __pfx_dev_watchdog+0x10/0x10
Jul 10 08:27:17 proxmox kernel:  call_timer_fn+0x29/0x160
Jul 10 08:27:17 proxmox kernel:  ? __pfx_dev_watchdog+0x10/0x10
Jul 10 08:27:17 proxmox kernel:  __run_timers+0x259/0x310
Jul 10 08:27:17 proxmox kernel:  run_timer_softirq+0x1d/0x40
Jul 10 08:27:17 proxmox kernel:  __do_softirq+0xd6/0x346
Jul 10 08:27:17 proxmox kernel:  ? hrtimer_interrupt+0x11f/0x250
Jul 10 08:27:17 proxmox kernel:  __irq_exit_rcu+0xa2/0xd0
Jul 10 08:27:17 proxmox kernel:  irq_exit_rcu+0xe/0x20
Jul 10 08:27:17 proxmox kernel:  sysvec_apic_timer_interrupt+0x92/0xd0
Jul 10 08:27:17 proxmox kernel:  </IRQ>
Jul 10 08:27:17 proxmox kernel:  <TASK>
Jul 10 08:27:17 proxmox kernel:  asm_sysvec_apic_timer_interrupt+0x1b/0x20
Jul 10 08:27:17 proxmox kernel: RIP: 0010:cpuidle_enter_state+0xde/0x6f0
Jul 10 08:27:17 proxmox kernel: Code: 2a 17 53 e8 54 7e 4a ff 8b 53 04 49 89 c7 0f 1f 44 00 00 31 ff e8 82 86 49 ff 80 7d d0 00 0f 85 eb 00 00 00 fb 0f 1f 44 00 00 <45> 85 f6 0f 88 12 02 00 00 4d 63 ee 49 83 fd 09 0f 87 c7 04 00 00
Jul 10 08:27:17 proxmox kernel: RSP: 0018:ffff9d9c401b7e38 EFLAGS: 00000246
Jul 10 08:27:17 proxmox kernel: RAX: 0000000000000000 RBX: ffffbd9c3fc40600 RCX: 0000000000000000
Jul 10 08:27:17 proxmox kernel: RDX: 0000000000000009 RSI: 0000000000000000 RDI: 0000000000000000
Jul 10 08:27:17 proxmox kernel: RBP: ffff9d9c401b7e88 R08: 0000000000000000 R09: 0000000000000000
Jul 10 08:27:17 proxmox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffae6c33a0
Jul 10 08:27:17 proxmox kernel: R13: 0000000000000004 R14: 0000000000000004 R15: 00002a26a1046e6c
Jul 10 08:27:17 proxmox kernel:  ? cpuidle_enter_state+0xce/0x6f0
Jul 10 08:27:17 proxmox kernel:  cpuidle_enter+0x2e/0x50
Jul 10 08:27:17 proxmox kernel:  do_idle+0x216/0x2a0
Jul 10 08:27:17 proxmox kernel:  cpu_startup_entry+0x1d/0x20
Jul 10 08:27:17 proxmox kernel:  start_secondary+0x122/0x160
Jul 10 08:27:17 proxmox kernel:  secondary_startup_64_no_verify+0xe5/0xeb
Jul 10 08:27:17 proxmox kernel:  </TASK>
Jul 10 08:27:17 proxmox kernel: ---[ end trace 0000000000000000 ]---
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: Register Dump
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: Register Name   Value
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: CTRL            181c0641
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: STATUS          00680693
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: CTRL_EXT        100000c0
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: MDIC            1805dd01
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: ICR             00000081
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: RCTL            0440803a
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: RDH[0-3]        0000008d 000000a3 0000003d 0000006e
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: RDT[0-3]        0000008c 000000a2 0000003c 0000006d
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: RDBAL[0-3]      1153b000 059e0000 1153d000 11541000
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: TCTL            a503f0fa
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: TDBAL[0-3]      1447e000 11539000 11119000 099ad000
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: TDH[0-3]        0000008d 00000057 00000075 00000025
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: TDT[0-3]        000000de 00000057 00000075 00000027
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Jul 10 08:27:17 proxmox kernel: igc 0000:06:00.0 enp6s0: Reset adapter
Jul 10 08:27:27 proxmox kernel: igc 0000:06:00.0 enp6s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX

My setup is relatively simple:
Typical multi-nic ChangWang box with 6x i226
multiple ports in an OVS bridge includingthe workstation launching the test, going through OpnSense in transparent mode (VM with a NIC in the same bridge). The WAN is an i226 with passthrough, to OpnSense.

I noticed similar threads, but all were focused around the realtek drivers, and kernel 5.15 vs 6.2. In my case, I had kernel 6.2 with 7.4 as with 8.0, and with intel NICs (igc).
Any idea what could be the cause?
 
Did either of you find a solution? I am having the same issue after upgrading to 8.0. I have even installed a new 2-port intel NIC, and the problem has moved to the virtual interface on the new nic. The second port on the nic is passed through to a VM and stays up when the virtual one goes down.

It's driving me mad!
 
Did either of you find a solution? I am having the same issue after upgrading to 8.0. I have even installed a new 2-port intel NIC, and the problem has moved to the virtual interface on the new nic. The second port on the nic is passed through to a VM and stays up when the virtual one goes down.

It's driving me mad!
I went back to 7.4 with my productive Nodes.
I will setup this week a testnode.
 
Thanks - I can probably roll back to an earlier kernel, but not sure if there is a downgrade path to 7.4? I would rather find a solution if possible...

There seem to be quite a few bug reports on this, mostly related to realtek NICs. What is interesting is that having switched to an intel NIC I am seeing the same behaviour.

A few threads have suggested Updating grub (/etc/default/grub) with:

Code:
GRUB_CMDLINE_LINUX="pcie_aspm=off pcie_port_pm=off"

I have given this a go and will see if it makes a different. Have also split my VMs to use two different virtual adaptors on different ports - will see if I lose all virtual adaptors at the same time or if only one is affected if the problem persists...
 
At the moment my Testnode on V8 run fine.

Its a fresh install with updates, without any changes like Kernel.
I add this Node now to my V7.4 Cluster to test.
 
Last edited:
UPDATE:
Testnode went Offline
1690181474159.png

Not accessable via Network.
No Signal on Monitore.

Log between 23.07. and 24.07. reboot
Code:
Jul 24 08:50:37 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:50:37 N-4 pvestatd[836]: status update time (10.219 seconds)
Jul 24 08:50:47 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:50:48 N-4 pvestatd[836]: status update time (10.229 seconds)
Jul 24 08:50:58 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:50:58 N-4 pvestatd[836]: status update time (10.195 seconds)
Jul 24 08:51:08 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:51:08 N-4 pvestatd[836]: status update time (10.212 seconds)
Jul 24 08:51:10 N-4 pvescheduler[189323]: jobs: cfs-lock 'file-jobs_cfg' error: no quorum!
Jul 24 08:51:10 N-4 pvescheduler[189322]: replication: cfs-lock 'file-replication_cfg' error: no quorum!
Jul 24 08:51:18 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:51:18 N-4 pvestatd[836]: status update time (10.229 seconds)
Jul 24 08:51:28 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:51:28 N-4 pvestatd[836]: status update time (10.243 seconds)
Jul 24 08:51:32 N-4 kernel: net_ratelimit: 9 callbacks suppressed
Jul 24 08:51:32 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
Jul 24 08:51:32 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:51:32 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:51:32 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:51:32 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:51:32 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:51:32 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:51:32 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:51:32 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:51:32 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:51:39 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:51:39 N-4 pvestatd[836]: status update time (10.204 seconds)
Jul 24 08:51:49 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:51:49 N-4 pvestatd[836]: status update time (10.211 seconds)
Jul 24 08:51:59 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:51:59 N-4 pvestatd[836]: status update time (10.220 seconds)
Jul 24 08:52:09 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:52:09 N-4 pvestatd[836]: status update time (10.200 seconds)
Jul 24 08:52:10 N-4 pvescheduler[189476]: jobs: cfs-lock 'file-jobs_cfg' error: no quorum!
Jul 24 08:52:10 N-4 pvescheduler[189475]: replication: cfs-lock 'file-replication_cfg' error: no quorum!
Jul 24 08:52:20 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:52:20 N-4 pvestatd[836]: status update time (10.223 seconds)
Jul 24 08:52:28 N-4 kernel: net_ratelimit: 9 callbacks suppressed
Jul 24 08:52:28 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
Jul 24 08:52:28 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:52:28 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:52:28 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:52:28 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:52:28 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:52:28 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:52:28 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:52:28 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:52:28 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:52:30 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:52:30 N-4 pvestatd[836]: status update time (10.220 seconds)
Jul 24 08:52:40 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:52:40 N-4 pvestatd[836]: status update time (10.219 seconds)
Jul 24 08:52:50 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:52:50 N-4 pvestatd[836]: status update time (10.220 seconds)
Jul 24 08:53:00 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:53:00 N-4 pvestatd[836]: status update time (10.204 seconds)
Jul 24 08:53:10 N-4 pvescheduler[189625]: jobs: cfs-lock 'file-jobs_cfg' error: no quorum!
Jul 24 08:53:10 N-4 pvescheduler[189624]: replication: cfs-lock 'file-replication_cfg' error: no quorum!
Jul 24 08:53:10 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:53:11 N-4 pvestatd[836]: status update time (10.207 seconds)
Jul 24 08:53:21 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:53:21 N-4 pvestatd[836]: status update time (10.204 seconds)
Jul 24 08:53:24 N-4 kernel: net_ratelimit: 9 callbacks suppressed
Jul 24 08:53:24 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
Jul 24 08:53:24 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:53:24 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:53:24 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:53:24 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:53:24 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:53:24 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:53:24 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:53:24 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:53:24 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:53:31 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:53:31 N-4 pvestatd[836]: status update time (10.207 seconds)
Jul 24 08:53:37 N-4 kernel: fbcon: i915drmfb (fb0) is primary device
Jul 24 08:53:37 N-4 kernel: Console: switching to colour frame buffer device 240x67
Jul 24 08:53:37 N-4 kernel: i915 0000:00:02.0: [drm] fb0: i915drmfb frame buffer device
Jul 24 08:53:41 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:53:41 N-4 pvestatd[836]: status update time (10.239 seconds)
Jul 24 08:53:46 N-4 kernel: usb 1-3: new low-speed USB device number 4 using xhci_hcd
Jul 24 08:53:46 N-4 kernel: usb 1-3: New USB device found, idVendor=04b3, idProduct=3025, bcdDevice= 1.09
Jul 24 08:53:46 N-4 kernel: usb 1-3: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Jul 24 08:53:46 N-4 kernel: usb 1-3: Product: USB NetVista Full Width Keyboard.
Jul 24 08:53:46 N-4 kernel: usb 1-3: Manufacturer: LITE-ON Technology
Jul 24 08:53:46 N-4 kernel: hid: raw HID events driver (C) Jiri Kosina
Jul 24 08:53:46 N-4 kernel: usbcore: registered new interface driver usbhid
Jul 24 08:53:46 N-4 kernel: usbhid: USB HID core driver
Jul 24 08:53:46 N-4 kernel: usbcore: registered new interface driver usbkbd
Jul 24 08:53:46 N-4 kernel: input: LITE-ON Technology USB NetVista Full Width Keyboard. as /devices/pci0000:00/0000:00:15.0/usb1/1-3/1-3:1.0/0003:04B3:3025.0001/input/input12
Jul 24 08:53:46 N-4 kernel: hid-generic 0003:04B3:3025.0001: input,hidraw0: USB HID v1.10 Keyboard [LITE-ON Technology USB NetVista Full Width Keyboard.] on usb-0000:00:15.0-3/input0
Jul 24 08:53:46 N-4 systemd-logind[523]: Watching system buttons on /dev/input/event12 (LITE-ON Technology USB NetVista Full Width Keyboard.)
Jul 24 08:53:51 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:53:52 N-4 pvestatd[836]: status update time (10.246 seconds)
Jul 24 08:54:02 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:54:02 N-4 pvestatd[836]: status update time (10.226 seconds)
Jul 24 08:54:10 N-4 pvescheduler[189811]: jobs: cfs-lock 'file-jobs_cfg' error: no quorum!
Jul 24 08:54:10 N-4 pvescheduler[189810]: replication: cfs-lock 'file-replication_cfg' error: no quorum!
Jul 24 08:54:12 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:54:12 N-4 pvestatd[836]: status update time (10.225 seconds)
Jul 24 08:54:19 N-4 kernel: net_ratelimit: 9 callbacks suppressed
Jul 24 08:54:19 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_chipcmd_cond == 1 (loop: 100, delay: 100).
Jul 24 08:54:19 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:54:19 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:54:19 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:54:19 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:54:19 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:54:19 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_ephyar_cond == 1 (loop: 100, delay: 10).
Jul 24 08:54:19 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:54:19 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:54:19 N-4 kernel: r8169 0000:01:00.0 enp1s0: rtl_eriar_cond == 1 (loop: 100, delay: 100).
Jul 24 08:54:22 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:54:22 N-4 pvestatd[836]: status update time (10.219 seconds)
Jul 24 08:54:32 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:54:32 N-4 pvestatd[836]: status update time (10.245 seconds)
Jul 24 08:54:43 N-4 pvestatd[836]: storage 'Backup' is not online
Jul 24 08:54:43 N-4 pvestatd[836]: status update time (10.264 seconds)
Jul 24 08:54:47 N-4 systemd-logind[523]: Power key pressed short.
Jul 24 08:54:47 N-4 systemd-logind[523]: Powering off...
Jul 24 08:54:47 N-4 systemd-logind[523]: System is powering down.
-- Reboot --
Jul 24 08:55:26 N-4 kernel: Linux version 6.2.16-4-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-5 (2023-07-14T17:53Z) ()
Jul 24 08:55:26 N-4 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-4-pve root=/dev/mapper/pve-root ro quiet
Jul 24 08:55:26 N-4 kernel: KERNEL supported cpus:
Jul 24 08:55:26 N-4 kernel:   Intel GenuineIntel
Jul 24 08:55:26 N-4 kernel:   AMD AuthenticAMD
Jul 24 08:55:26 N-4 kernel:   Hygon HygonGenuine
Jul 24 08:55:26 N-4 kernel:   Centaur CentaurHauls
Jul 24 08:55:26 N-4 kernel:   zhaoxin   Shanghai

I found this Post: https://forum.proxmox.com/threads/networking-issues-pve8.129742/post-571159

With this Solution: https://medium.com/@pattapongj/how-...and-encountering-the-r8169-error-d2e322cc26ed

I installed the Package now and will write tomorrow.

At the Moment it looks good:
1690187456812.png

No Issues, no high ms pings.
 
Last edited:
@BugVito did you find a solution with proxmox ve 8.0.x?
For me corin.corvus's solution didn't work as i too have a i266-v nic using igc drivers. So no wrong driver issue here, like the r8169 / r8168 problem.

What i could observe, if i don't stop a transfere / speedtest shortly after i get
Code:
kernel: NETDEV WATCHDOG: enp6s0 (igc): transmit queue 0 timed out
kernel: WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
kernel: Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc nfnetlink_log nfnetlink binfmt_misc snd_hda_codec_hdmi snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus i915 snd_soc_core x86_pkg_temp_thermal intel_powerclamp snd_compress coretemp ac97_bus snd_pcm_dmaengine kvm_intel snd_hda_intel drm_buddy snd_intel_dspcfg snd_intel_sdw_acpi ttm snd_hda_codec drm_display_helper kvm cec snd_hda_core snd_hwdep rc_core intel_rapl_msr irqbypass snd_pcm crct10dif_pclmul drm_kms_helper polyval_generic snd_timer ghash_clmulni_intel processor_thermal_device_pci_legacy sha512_ssse3 processor_thermal_device aesni_intel i2c_algo_bit cmdlinepart crypto_simd
kernel:  syscopyarea mei_me cryptd spi_nor snd processor_thermal_rfim sysfillrect processor_thermal_mbox sysimgblt soundcore intel_cstate mtd processor_thermal_rapl intel_rapl_common pcspkr ee1004 mei wmi_bmof int340x_thermal_zone 8250_dw intel_soc_dts_iosf joydev input_leds acpi_pad acpi_tad mac_hid vhost_net vhost vhost_iotlb tap drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c simplefb hid_logitech_hidpp hid_logitech_dj hid_generic usbkbd usbmouse spi_pxa2xx_platform usbhid dw_dmac hid dw_dmac_core nvme i2c_i801 i2c_smbus crc32_pclmul xhci_pci xhci_pci_renesas spi_intel_pci spi_intel ahci libahci igc nvme_core nvme_common intel_lpss_pci xhci_hcd intel_lpss idma64 video wmi
kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: P           O       6.2.16-6-pve #1
kernel: Hardware name: Default string Default string/CW-N11, BIOS 5.19 10/11/2022
kernel: RIP: 0010:dev_watchdog+0x23a/0x250

my host freezes up and this is the last video output. (Stays for some minuts, after this it's a black screen)

Code:
pveproxy[47214]: worker exit
pveproxy[1811]: worker 47214 finished
pveproxy[1811]: starting 1 worker(s)
pveproxy[1811]: worker 52110 started
kernel: igc 0000:06:00.0 enp6s0: Register Dump
kernel: igc 0000:06:00.0 enp6s0: Register Name   Value
kernel: igc 0000:06:00.0 enp6s0: CTRL            181c0641
kernel: igc 0000:06:00.0 enp6s0: STATUS          00680683
kernel: igc 0000:06:00.0 enp6s0: CTRL_EXT        100000c0
kernel: igc 0000:06:00.0 enp6s0: MDIC            1805dd01
kernel: igc 0000:06:00.0 enp6s0: ICR             00000081
kernel: igc 0000:06:00.0 enp6s0: RCTL            0440803a
kernel: igc 0000:06:00.0 enp6s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
kernel: igc 0000:06:00.0 enp6s0: RDH[0-3]        000000a1 000000a2 00000022 000000a5
kernel: igc 0000:06:00.0 enp6s0: RDT[0-3]        000000a0 000000a1 00000021 000000a4
kernel: igc 0000:06:00.0 enp6s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
kernel: igc 0000:06:00.0 enp6s0: RDBAL[0-3]      0fddd000 0fde0000 0fde3000 0fde6000
kernel: igc 0000:06:00.0 enp6s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
kernel: igc 0000:06:00.0 enp6s0: TCTL            a503f0fa
kernel: igc 0000:06:00.0 enp6s0: TDBAL[0-3]      0fd93000 0fd98000 0fd9d000 0fdda000
kernel: igc 0000:06:00.0 enp6s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
kernel: igc 0000:06:00.0 enp6s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
kernel: igc 0000:06:00.0 enp6s0: TDH[0-3]        0000005a 000000d7 0000002d 00000091
kernel: igc 0000:06:00.0 enp6s0: TDT[0-3]        0000005a 000000d7 0000002d 00000091
kernel: igc 0000:06:00.0 enp6s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
kernel: igc 0000:06:00.0 enp6s0: Reset adapter
kernel: vmbr1: port 4(enp6s0) entered disabled state
kernel: igc 0000:06:00.0 enp6s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
kernel: vmbr1: port 4(enp6s0) entered blocking state
kernel: vmbr1: port 4(enp6s0) entered forwarding state
kernel: igc 0000:06:00.0 enp6s0: NIC Link is Down
kernel: vmbr1: port 4(enp6s0) entered disabled state
kernel: igc 0000:06:00.0 enp6s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
kernel: vmbr1: port 4(enp6s0) entered blocking state
kernel: vmbr1: port 4(enp6s0) entered forwarding state
under syslog i get two lines more
Code:
kernel: ------------[ cut here ]------------
kernel: kernel BUG at lib/dynamic_queue_limits.c:27!
 
Last edited:
@BugVito did you find a solution with proxmox ve 8.0.x?
For me corin.corvus's solution didn't work as i too have a i266-v nic using igc drivers. So no wrong driver issue here, like the r8169 / r8168 problem.

What i could observe, if i don't stop a transfere / speedtest shortly after i get
Code:
kernel: NETDEV WATCHDOG: enp6s0 (igc): transmit queue 0 timed out
kernel: WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x23a/0x250
kernel: Modules linked in: tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc nfnetlink_log nfnetlink binfmt_misc snd_hda_codec_hdmi snd_sof_pci_intel_icl snd_sof_intel_hda_common soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi soundwire_bus i915 snd_soc_core x86_pkg_temp_thermal intel_powerclamp snd_compress coretemp ac97_bus snd_pcm_dmaengine kvm_intel snd_hda_intel drm_buddy snd_intel_dspcfg snd_intel_sdw_acpi ttm snd_hda_codec drm_display_helper kvm cec snd_hda_core snd_hwdep rc_core intel_rapl_msr irqbypass snd_pcm crct10dif_pclmul drm_kms_helper polyval_generic snd_timer ghash_clmulni_intel processor_thermal_device_pci_legacy sha512_ssse3 processor_thermal_device aesni_intel i2c_algo_bit cmdlinepart crypto_simd
kernel:  syscopyarea mei_me cryptd spi_nor snd processor_thermal_rfim sysfillrect processor_thermal_mbox sysimgblt soundcore intel_cstate mtd processor_thermal_rapl intel_rapl_common pcspkr ee1004 mei wmi_bmof int340x_thermal_zone 8250_dw intel_soc_dts_iosf joydev input_leds acpi_pad acpi_tad mac_hid vhost_net vhost vhost_iotlb tap drm efi_pstore dmi_sysfs ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor raid6_pq libcrc32c simplefb hid_logitech_hidpp hid_logitech_dj hid_generic usbkbd usbmouse spi_pxa2xx_platform usbhid dw_dmac hid dw_dmac_core nvme i2c_i801 i2c_smbus crc32_pclmul xhci_pci xhci_pci_renesas spi_intel_pci spi_intel ahci libahci igc nvme_core nvme_common intel_lpss_pci xhci_hcd intel_lpss idma64 video wmi
kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: P           O       6.2.16-6-pve #1
kernel: Hardware name: Default string Default string/CW-N11, BIOS 5.19 10/11/2022
kernel: RIP: 0010:dev_watchdog+0x23a/0x250

my host freezes up and this is the last video output. (Stays for some minuts, after this it's a black screen)

Code:
pveproxy[47214]: worker exit
pveproxy[1811]: worker 47214 finished
pveproxy[1811]: starting 1 worker(s)
pveproxy[1811]: worker 52110 started
kernel: igc 0000:06:00.0 enp6s0: Register Dump
kernel: igc 0000:06:00.0 enp6s0: Register Name   Value
kernel: igc 0000:06:00.0 enp6s0: CTRL            181c0641
kernel: igc 0000:06:00.0 enp6s0: STATUS          00680683
kernel: igc 0000:06:00.0 enp6s0: CTRL_EXT        100000c0
kernel: igc 0000:06:00.0 enp6s0: MDIC            1805dd01
kernel: igc 0000:06:00.0 enp6s0: ICR             00000081
kernel: igc 0000:06:00.0 enp6s0: RCTL            0440803a
kernel: igc 0000:06:00.0 enp6s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
kernel: igc 0000:06:00.0 enp6s0: RDH[0-3]        000000a1 000000a2 00000022 000000a5
kernel: igc 0000:06:00.0 enp6s0: RDT[0-3]        000000a0 000000a1 00000021 000000a4
kernel: igc 0000:06:00.0 enp6s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
kernel: igc 0000:06:00.0 enp6s0: RDBAL[0-3]      0fddd000 0fde0000 0fde3000 0fde6000
kernel: igc 0000:06:00.0 enp6s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
kernel: igc 0000:06:00.0 enp6s0: TCTL            a503f0fa
kernel: igc 0000:06:00.0 enp6s0: TDBAL[0-3]      0fd93000 0fd98000 0fd9d000 0fdda000
kernel: igc 0000:06:00.0 enp6s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
kernel: igc 0000:06:00.0 enp6s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
kernel: igc 0000:06:00.0 enp6s0: TDH[0-3]        0000005a 000000d7 0000002d 00000091
kernel: igc 0000:06:00.0 enp6s0: TDT[0-3]        0000005a 000000d7 0000002d 00000091
kernel: igc 0000:06:00.0 enp6s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
kernel: igc 0000:06:00.0 enp6s0: Reset adapter
kernel: vmbr1: port 4(enp6s0) entered disabled state
kernel: igc 0000:06:00.0 enp6s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
kernel: vmbr1: port 4(enp6s0) entered blocking state
kernel: vmbr1: port 4(enp6s0) entered forwarding state
kernel: igc 0000:06:00.0 enp6s0: NIC Link is Down
kernel: vmbr1: port 4(enp6s0) entered disabled state
kernel: igc 0000:06:00.0 enp6s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
kernel: vmbr1: port 4(enp6s0) entered blocking state
kernel: vmbr1: port 4(enp6s0) entered forwarding state
under syslog i get two lines more
Code:
kernel: ------------[ cut here ]------------
kernel: kernel BUG at lib/dynamic_queue_limits.c:27!
My apologies, all replies were going straight to junk mail... I do not have a response or fix unfortunately... I did update the kernel yesterday to 6.2.16-8-pve but unfortunately I seemed to experience the same issue this morning when downloading at speeds above 1.1Gbps. As soon as I sustained a download of 1.2Gbps or higher I'd end up with the original issue.

I'll try to test some more however time is not quite on my side. I'll go through links mentioned above and if anything "fixes" the issue I'll report back.
 
Some comments above seem to relate to the realtek drivers again, but I believe some threads had answers for those. In my case, this is an intel driver for the i226 interfaces.
Reading some links, I tried the following on all interfaces:
Code:
ethtool -K <interface> tso off gso off
After a reboot this seems fine however I'd assume some performance degradation. This doesn't matter to me, my main goal is to fully use my internet pipe (1.8g) which I can still hit.
If the issue reappears I'll chime in again. This in my opinion is just a workaround and not a proper fix even if it works long term.
 
I tested yesterday with "Linux 6.2.16-10-pve" and no issues (ran speedtest 3 times in a row). However, this morning during the first test, the same issue surfaced.

Code:
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: Register Dump
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: Register Name   Value
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: CTRL            181c0641
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: STATUS          00680693
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: CTRL_EXT        100000c0
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: MDIC            1805dd01
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: ICR             00000081
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: RCTL            0440803a
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: RDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: RDH[0-3]        00000009 0000001b 000000ff 0000002a
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: RDT[0-3]        00000008 0000001a 000000fe 00000029
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: RDBAL[0-3]      07f99000 0e5ee000 0e5f1000 0e5f4000
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: RDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: TCTL            a503f0fa
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: TDBAL[0-3]      16984000 06f24000 04314000 0527d000
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: TDBAH[0-3]      00000001 00000001 00000001 00000001
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: TDLEN[0-3]      00001000 00001000 00001000 00001000
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: TDH[0-3]        000000ad 000000a9 00000068 000000b8
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: TDT[0-3]        000000fe 000000aa 00000069 000000b8
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
Sep 16 08:58:22 proxmox kernel: igc 0000:06:00.0 enp6s0: Reset adapter
Sep 16 08:58:26 proxmox kernel: igc 0000:06:00.0 enp6s0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX

In a way this is not critical for me, internally I have only low throughput traffic between systems. This is an issue when performing speedtests (throughput >1.25G). However this seems to point to an intel igc issue, potentially a driver issue (or driver vs kernel version issue).
 
This is an aging thread, however it seems that the issue lies with the NIC on the other side, not with proxmox nor the server on which promox is hosted in my case; My workstation is connected directly to the proxmox box, has an ASUS XG-C100C 10G, and an option appears to be at fault, causing a timeout on the proxmox box "transmit queue 0 timed out".
The option is "Recv Segment Coalescing". Once disabled, I appear to be able to repeatedly trigger a speed test without hanging, and without the above outputs on proxmox.
Unfortunately this appears to be a "known", yet obscure issue with this NIC.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!