Proxmox frozen randomly, any ideas?

Sandbo · Feb 27, 2021

Hi,

I have been using Proxmox for a year now, and it has been running great on that machine (rock solid).
Lately I put on a Windows Guest OS, but simultaneously true is I have not turned on that machine for a while as I relocated.

In the past couple days after turning it back on, I have encounter almost random freezing of the machine.
The loading is minimal at 1x LXC (ubuntu), 2xVMs (1 Windows, 1 CentOS as router) .
The LXC and VM CentOS use only 4 GB RAM each, and Windows takes 16 GB. The machine has 32 GB so I think it is safe to assume resource isn't a problem.

Harddisk space wise, the Linux ones uses only 64 GB, and I have 128 GB for Windows, over a 512 GB SSD.

Before the freeze, I seem to see this log before I have manually rebooted it.
Unfortunately it is headless so I cannot see the actual output, but WebGUI can't be accessed nor the server could be pinged.

The log seems to suggest troubles from my X550-T2 NIC, do you think it is the problem?

Code:

Feb 27 12:56:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 12:57:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 12:57:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 12:57:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 12:57:52 pve postfix/qmgr[2368]: AE6D6580875: from=<root@pve.chroniclescience.com>, size=26843, nrcpt=1 (queue active)
Feb 27 12:58:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 12:58:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 12:58:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 12:58:22 pve postfix/smtp[15453]: connect to outlook-com.olc.protection.outlook.com[104.47.55.33]:25: Connection timed out
Feb 27 12:58:52 pve postfix/smtp[15453]: connect to outlook-com.olc.protection.outlook.com[104.47.70.33]:25: Connection timed out
Feb 27 12:58:52 pve postfix/smtp[15453]: AE6D6580875: to=<sandbochang@outlook.com>, relay=none, delay=118232, delays=118171/0.01/60/0, dsn=4.4.1, status=deferred (connect to outlook-com.olc.protection.outlook.com[104.47.70.33]:$
Feb 27 12:59:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 12:59:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 12:59:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 12:59:03 pve kernel: [42687.415678] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:07 pve kernel: [42691.463511] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:11 pve kernel: [42695.511487] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:15 pve kernel: [42699.559839] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:19 pve kernel: [42703.611264] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:21 pve kernel: [42705.657956] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:25 pve kernel: [42709.708593] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:29 pve kernel: [42713.763309] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:33 pve kernel: [42717.815078] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:37 pve kernel: [42721.863253] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:41 pve kernel: [42725.911063] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:45 pve kernel: [42729.959236] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:50 pve kernel: [42734.007143] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:54 pve kernel: [42738.045321] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:58 pve kernel: [42742.096097] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:00:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 13:00:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 13:00:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 13:01:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 13:01:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 13:01:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 13:01:02 pve kernel: [42806.878683] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:06 pve kernel: [42810.917461] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:11 pve kernel: [42814.964764] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:15 pve kernel: [42819.008985] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:19 pve kernel: [42823.058165] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:23 pve kernel: [42827.110563] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:27 pve kernel: [42831.155421] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:31 pve kernel: [42835.205226] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:35 pve kernel: [42839.254467] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:39 pve kernel: [42843.303748] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:43 pve kernel: [42847.358350] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:47 pve kernel: [42851.404476] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:51 pve kernel: [42855.454103] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:55 pve kernel: [42859.502367] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:59 pve kernel: [42863.549995] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 13:02:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 13:02:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 13:02:03 pve kernel: [42867.590467] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:05 pve kernel: [42869.386522] ------------[ cut here ]------------
Feb 27 13:02:05 pve kernel: [42869.386524] NETDEV WATCHDOG: enp11s0 (igb): transmit queue 0 timed out
Feb 27 13:02:05 pve kernel: [42869.386538] WARNING: CPU: 15 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x264/0x270
Feb 27 13:02:05 pve kernel: [42869.386539] Modules linked in: tcp_diag inet_diag ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 xt_hl ip6t_rt ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp xt_addrtype xt_$
Feb 27 13:02:05 pve kernel: [42869.386575]  dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c i2c_piix4 ixgbe xfrm_algo mdio igb i2c_algo_bit dca xhci_pci ahci xhci_hcd libahci wmi gpio_amdpt gpio_generic
Feb 27 13:02:05 pve kernel: [42869.386584] CPU: 15 PID: 0 Comm: swapper/15 Tainted: P           O      5.4.98-1-pve #1
Feb 27 13:02:05 pve kernel: [42869.386585] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470 Gaming-ITX/ac, BIOS P3.40 07/16/2019
Feb 27 13:02:05 pve kernel: [42869.386587] RIP: 0010:dev_watchdog+0x264/0x270
Feb 27 13:02:05 pve kernel: [42869.386589] Code: 48 85 c0 75 e6 eb a0 4c 89 ef c6 05 7f e8 ef 00 01 e8 e0 b6 fa ff 89 d9 4c 89 ee 48 c7 c7 40 55 e3 9f 48 89 c2 e8 d6 51 15 00 <0f> 0b eb 82 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00$
Feb 27 13:02:05 pve kernel: [42869.386590] RSP: 0018:ffffaba800530e58 EFLAGS: 00010282
Feb 27 13:02:05 pve kernel: [42869.386591] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006
Feb 27 13:02:05 pve kernel: [42869.386592] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff955aaebd78c0
Feb 27 13:02:05 pve kernel: [42869.386592] RBP: ffffaba800530e88 R08: 0000000000000495 R09: 0000000000000004
Feb 27 13:02:05 pve kernel: [42869.386593] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000008
Feb 27 13:02:05 pve kernel: [42869.386594] R13: ffff955a9c120000 R14: ffff955a9c120480 R15: ffff955a9e107940
Feb 27 13:02:05 pve kernel: [42869.386595] FS:  0000000000000000(0000) GS:ffff955aaebc0000(0000) knlGS:0000000000000000
Feb 27 13:02:05 pve kernel: [42869.386595] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 27 13:02:05 pve kernel: [42869.386596] CR2: 000002d36b8b4b1e CR3: 00000007e4b60000 CR4: 00000000003406e0
Feb 27 13:02:05 pve kernel: [42869.386596] Call Trace:
Feb 27 13:02:05 pve kernel: [42869.386598]  <IRQ>
Feb 27 13:02:05 pve kernel: [42869.386600]  ? pfifo_fast_enqueue+0x160/0x160
Feb 27 13:02:05 pve kernel: [42869.386603]  call_timer_fn+0x32/0x130
Feb 27 13:02:05 pve kernel: [42869.386604]  run_timer_softirq+0x1a5/0x430
Feb 27 13:02:05 pve kernel: [42869.386606]  ? enqueue_hrtimer+0x3c/0x90
Feb 27 13:02:05 pve kernel: [42869.386607]  ? ktime_get+0x3c/0xa0
Feb 27 13:02:05 pve kernel: [42869.386609]  ? lapic_next_event+0x20/0x30
Feb 27 13:02:05 pve kernel: [42869.386611]  ? clockevents_program_event+0x93/0xf0
Feb 27 13:02:05 pve kernel: [42869.386614]  __do_softirq+0xdc/0x2d4
Feb 27 13:02:05 pve kernel: [42869.386616]  irq_exit+0xa9/0xb0
Feb 27 13:02:05 pve kernel: [42869.386617]  smp_apic_timer_interrupt+0x79/0x130
Feb 27 13:02:05 pve kernel: [42869.386619]  apic_timer_interrupt+0xf/0x20
Feb 27 13:02:05 pve kernel: [42869.386620]  </IRQ>
Feb 27 13:02:05 pve kernel: [42869.386622] RIP: 0010:cpuidle_enter_state+0xbd/0x450
Feb 27 13:02:05 pve kernel: [42869.386624] Code: ff e8 17 94 88 ff 80 7d c7 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 63 03 00 00 31 ff e8 6a 9b 8e ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 8d 02 00 00 49 63 cd 48 8b 75 d0 48$
Feb 27 13:02:05 pve kernel: [42869.386624] RSP: 0018:ffffaba80019fe48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Feb 27 13:02:05 pve kernel: [42869.386625] RAX: ffff955aaebeae00 RBX: ffffffffa0166940 RCX: 000000000000001f
Feb 27 13:02:05 pve kernel: [42869.386626] RDX: 000026fd4e8e1558 RSI: 0000000022a8daa1 RDI: 0000000000000000
Feb 27 13:02:05 pve kernel: [42869.386626] RBP: ffffaba80019fe88 R08: 0000000000000002 R09: 000000000002a680
Feb 27 13:02:05 pve kernel: [42869.386627] R10: 0000901a2d6d7eeb R11: ffff955aaebe9aa0 R12: ffff955a9ebff000
Feb 27 13:02:05 pve kernel: [42869.386626] RDX: 000026fd4e8e1558 RSI: 0000000022a8daa1 RDI: 0000000000000000
Feb 27 13:02:05 pve kernel: [42869.386626] RBP: ffffaba80019fe88 R08: 0000000000000002 R09: 000000000002a680
Feb 27 13:02:05 pve kernel: [42869.386627] R10: 0000901a2d6d7eeb R11: ffff955aaebe9aa0 R12: ffff955a9ebff000
Feb 27 13:02:05 pve kernel: [42869.386627] R13: 0000000000000002 R14: ffffffffa0166a18 R15: ffffffffa0166a00
Feb 27 13:02:05 pve kernel: [42869.386630]  ? cpuidle_enter_state+0x99/0x450
Feb 27 13:02:05 pve kernel: [42869.386631]  cpuidle_enter+0x2e/0x40
Feb 27 13:02:05 pve kernel: [42869.386633]  call_cpuidle+0x23/0x40
Feb 27 13:02:05 pve kernel: [42869.386634]  do_idle+0x22c/0x270
Feb 27 13:02:05 pve kernel: [42869.386635]  cpu_startup_entry+0x1d/0x20
Feb 27 13:02:05 pve kernel: [42869.386637]  start_secondary+0x166/0x1c0
Feb 27 13:02:05 pve kernel: [42869.386639]  secondary_startup_64+0xa4/0xb0
Feb 27 13:02:05 pve kernel: [42869.386641] ---[ end trace 0fc9293d87fbf079 ]---
Feb 27 13:02:05 pve kernel: [42869.386657] igb 0000:0b:00.0 enp11s0: Reset adapter
Feb 27 13:02:05 pve kernel: [42869.439085] vmbr0: port 1(enp11s0) entered disabled state
Feb 27 13:02:07 pve kernel: [42871.632955] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:09 pve kernel: [42873.039041] igb 0000:0b:00.0 enp11s0: igb: enp11s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Feb 27 13:02:09 pve kernel: [42873.146735] vmbr0: port 1(enp11s0) entered blocking state
Feb 27 13:02:09 pve kernel: [42873.146737] vmbr0: port 1(enp11s0) entered forwarding state
Feb 27 13:02:11 pve kernel: [42875.686052] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:15 pve kernel: [42879.729130] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:19 pve kernel: [42883.774845] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:23 pve kernel: [42887.826045] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:27 pve kernel: [42891.877783] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:31 pve kernel: [42895.926072] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Feb 27 13:03:38 p$

leesteken · Feb 27, 2021

The logs appear to indicate a network connection or network card issue (timing out) and even a network card driver crash.
I don't think it causes a freeze, just no more network connection to the machine. Try a different network card (and connect a screen and keyboard)?

Sandbo · Feb 27, 2021

avw said:
The logs appear to indicate a network connection or network card issue (timing out) and even a network card driver crash.
I don't think it causes a freeze, just no more network connection to the machine. Try a different network card (and connect a screen and keyboard)?

Thanks for the reply, it is so random that it just happened again - two hours ago before I have left my apartment it was working,
now I can't access any server on it. The loss of internet (the NIC shown in the message) should not disable the management: My Windows PC was configured with a fixed IP within the same subnet, and the Proxmox server has a third NIC for management only. This was done such that even when the router OS is dead I can still access the mangement WebGUI, but now when it does freeze even this dies.

I will probably open it up tonight and check if things have moved inside due to shipping, but otherwise I have zero clue. I can try to hook it up to a monitor, but it is a bit far away from the monitor and it has no GPU with it at the moment.

Sorry, I could actually access the router through internet and its WebGUI, this time it might just be a routing issue. I will report back if it does freeze again.

Search

Search

Proxmox frozen randomly, any ideas?

Sandbo

Well-Known Member

leesteken

Distinguished Member

Sandbo

Well-Known Member