Proxmox frozen randomly, any ideas?

Sandbo

Active Member
Jul 4, 2019
84
10
28
33
Hi,

I have been using Proxmox for a year now, and it has been running great on that machine (rock solid).
Lately I put on a Windows Guest OS, but simultaneously true is I have not turned on that machine for a while as I relocated.

In the past couple days after turning it back on, I have encounter almost random freezing of the machine.
The loading is minimal at 1x LXC (ubuntu), 2xVMs (1 Windows, 1 CentOS as router) .
The LXC and VM CentOS use only 4 GB RAM each, and Windows takes 16 GB. The machine has 32 GB so I think it is safe to assume resource isn't a problem.

Harddisk space wise, the Linux ones uses only 64 GB, and I have 128 GB for Windows, over a 512 GB SSD.

Before the freeze, I seem to see this log before I have manually rebooted it.
Unfortunately it is headless so I cannot see the actual output, but WebGUI can't be accessed nor the server could be pinged.

The log seems to suggest troubles from my X550-T2 NIC, do you think it is the problem?

Code:
Feb 27 12:56:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 12:57:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 12:57:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 12:57:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 12:57:52 pve postfix/qmgr[2368]: AE6D6580875: from=<root@pve.chroniclescience.com>, size=26843, nrcpt=1 (queue active)
Feb 27 12:58:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 12:58:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 12:58:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 12:58:22 pve postfix/smtp[15453]: connect to outlook-com.olc.protection.outlook.com[104.47.55.33]:25: Connection timed out
Feb 27 12:58:52 pve postfix/smtp[15453]: connect to outlook-com.olc.protection.outlook.com[104.47.70.33]:25: Connection timed out
Feb 27 12:58:52 pve postfix/smtp[15453]: AE6D6580875: to=<sandbochang@outlook.com>, relay=none, delay=118232, delays=118171/0.01/60/0, dsn=4.4.1, status=deferred (connect to outlook-com.olc.protection.outlook.com[104.47.70.33]:$
Feb 27 12:59:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 12:59:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 12:59:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 12:59:03 pve kernel: [42687.415678] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:07 pve kernel: [42691.463511] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:11 pve kernel: [42695.511487] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:15 pve kernel: [42699.559839] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:19 pve kernel: [42703.611264] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:21 pve kernel: [42705.657956] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:25 pve kernel: [42709.708593] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:29 pve kernel: [42713.763309] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:33 pve kernel: [42717.815078] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:37 pve kernel: [42721.863253] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:41 pve kernel: [42725.911063] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:45 pve kernel: [42729.959236] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:50 pve kernel: [42734.007143] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:54 pve kernel: [42738.045321] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 12:59:58 pve kernel: [42742.096097] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:00:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 13:00:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 13:00:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 13:01:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 13:01:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 13:01:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 13:01:02 pve kernel: [42806.878683] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:06 pve kernel: [42810.917461] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:11 pve kernel: [42814.964764] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:15 pve kernel: [42819.008985] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:19 pve kernel: [42823.058165] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:23 pve kernel: [42827.110563] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:27 pve kernel: [42831.155421] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:31 pve kernel: [42835.205226] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:35 pve kernel: [42839.254467] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:39 pve kernel: [42843.303748] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:43 pve kernel: [42847.358350] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:47 pve kernel: [42851.404476] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:51 pve kernel: [42855.454103] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:55 pve kernel: [42859.502367] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:01:59 pve kernel: [42863.549995] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:00 pve systemd[1]: Starting Proxmox VE replication runner...
Feb 27 13:02:01 pve systemd[1]: pvesr.service: Succeeded.
Feb 27 13:02:01 pve systemd[1]: Started Proxmox VE replication runner.
Feb 27 13:02:03 pve kernel: [42867.590467] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:05 pve kernel: [42869.386522] ------------[ cut here ]------------
Feb 27 13:02:05 pve kernel: [42869.386524] NETDEV WATCHDOG: enp11s0 (igb): transmit queue 0 timed out
Feb 27 13:02:05 pve kernel: [42869.386538] WARNING: CPU: 15 PID: 0 at net/sched/sch_generic.c:448 dev_watchdog+0x264/0x270
Feb 27 13:02:05 pve kernel: [42869.386539] Modules linked in: tcp_diag inet_diag ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 xt_hl ip6t_rt ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit xt_tcpudp xt_addrtype xt_$
Feb 27 13:02:05 pve kernel: [42869.386575]  dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c i2c_piix4 ixgbe xfrm_algo mdio igb i2c_algo_bit dca xhci_pci ahci xhci_hcd libahci wmi gpio_amdpt gpio_generic
Feb 27 13:02:05 pve kernel: [42869.386584] CPU: 15 PID: 0 Comm: swapper/15 Tainted: P           O      5.4.98-1-pve #1
Feb 27 13:02:05 pve kernel: [42869.386585] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470 Gaming-ITX/ac, BIOS P3.40 07/16/2019
Feb 27 13:02:05 pve kernel: [42869.386587] RIP: 0010:dev_watchdog+0x264/0x270
Feb 27 13:02:05 pve kernel: [42869.386589] Code: 48 85 c0 75 e6 eb a0 4c 89 ef c6 05 7f e8 ef 00 01 e8 e0 b6 fa ff 89 d9 4c 89 ee 48 c7 c7 40 55 e3 9f 48 89 c2 e8 d6 51 15 00 <0f> 0b eb 82 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00$
Feb 27 13:02:05 pve kernel: [42869.386590] RSP: 0018:ffffaba800530e58 EFLAGS: 00010282
Feb 27 13:02:05 pve kernel: [42869.386591] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006
Feb 27 13:02:05 pve kernel: [42869.386592] RDX: 0000000000000007 RSI: 0000000000000082 RDI: ffff955aaebd78c0
Feb 27 13:02:05 pve kernel: [42869.386592] RBP: ffffaba800530e88 R08: 0000000000000495 R09: 0000000000000004
Feb 27 13:02:05 pve kernel: [42869.386593] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000008
Feb 27 13:02:05 pve kernel: [42869.386594] R13: ffff955a9c120000 R14: ffff955a9c120480 R15: ffff955a9e107940
Feb 27 13:02:05 pve kernel: [42869.386595] FS:  0000000000000000(0000) GS:ffff955aaebc0000(0000) knlGS:0000000000000000
Feb 27 13:02:05 pve kernel: [42869.386595] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 27 13:02:05 pve kernel: [42869.386596] CR2: 000002d36b8b4b1e CR3: 00000007e4b60000 CR4: 00000000003406e0
Feb 27 13:02:05 pve kernel: [42869.386596] Call Trace:
Feb 27 13:02:05 pve kernel: [42869.386598]  <IRQ>
Feb 27 13:02:05 pve kernel: [42869.386600]  ? pfifo_fast_enqueue+0x160/0x160
Feb 27 13:02:05 pve kernel: [42869.386603]  call_timer_fn+0x32/0x130
Feb 27 13:02:05 pve kernel: [42869.386604]  run_timer_softirq+0x1a5/0x430
Feb 27 13:02:05 pve kernel: [42869.386606]  ? enqueue_hrtimer+0x3c/0x90
Feb 27 13:02:05 pve kernel: [42869.386607]  ? ktime_get+0x3c/0xa0
Feb 27 13:02:05 pve kernel: [42869.386609]  ? lapic_next_event+0x20/0x30
Feb 27 13:02:05 pve kernel: [42869.386611]  ? clockevents_program_event+0x93/0xf0
Feb 27 13:02:05 pve kernel: [42869.386614]  __do_softirq+0xdc/0x2d4
Feb 27 13:02:05 pve kernel: [42869.386616]  irq_exit+0xa9/0xb0
Feb 27 13:02:05 pve kernel: [42869.386617]  smp_apic_timer_interrupt+0x79/0x130
Feb 27 13:02:05 pve kernel: [42869.386619]  apic_timer_interrupt+0xf/0x20
Feb 27 13:02:05 pve kernel: [42869.386620]  </IRQ>
Feb 27 13:02:05 pve kernel: [42869.386622] RIP: 0010:cpuidle_enter_state+0xbd/0x450
Feb 27 13:02:05 pve kernel: [42869.386624] Code: ff e8 17 94 88 ff 80 7d c7 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 63 03 00 00 31 ff e8 6a 9b 8e ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 8d 02 00 00 49 63 cd 48 8b 75 d0 48$
Feb 27 13:02:05 pve kernel: [42869.386624] RSP: 0018:ffffaba80019fe48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
Feb 27 13:02:05 pve kernel: [42869.386625] RAX: ffff955aaebeae00 RBX: ffffffffa0166940 RCX: 000000000000001f
Feb 27 13:02:05 pve kernel: [42869.386626] RDX: 000026fd4e8e1558 RSI: 0000000022a8daa1 RDI: 0000000000000000
Feb 27 13:02:05 pve kernel: [42869.386626] RBP: ffffaba80019fe88 R08: 0000000000000002 R09: 000000000002a680
Feb 27 13:02:05 pve kernel: [42869.386627] R10: 0000901a2d6d7eeb R11: ffff955aaebe9aa0 R12: ffff955a9ebff000
Feb 27 13:02:05 pve kernel: [42869.386626] RDX: 000026fd4e8e1558 RSI: 0000000022a8daa1 RDI: 0000000000000000
Feb 27 13:02:05 pve kernel: [42869.386626] RBP: ffffaba80019fe88 R08: 0000000000000002 R09: 000000000002a680
Feb 27 13:02:05 pve kernel: [42869.386627] R10: 0000901a2d6d7eeb R11: ffff955aaebe9aa0 R12: ffff955a9ebff000
Feb 27 13:02:05 pve kernel: [42869.386627] R13: 0000000000000002 R14: ffffffffa0166a18 R15: ffffffffa0166a00
Feb 27 13:02:05 pve kernel: [42869.386630]  ? cpuidle_enter_state+0x99/0x450
Feb 27 13:02:05 pve kernel: [42869.386631]  cpuidle_enter+0x2e/0x40
Feb 27 13:02:05 pve kernel: [42869.386633]  call_cpuidle+0x23/0x40
Feb 27 13:02:05 pve kernel: [42869.386634]  do_idle+0x22c/0x270
Feb 27 13:02:05 pve kernel: [42869.386635]  cpu_startup_entry+0x1d/0x20
Feb 27 13:02:05 pve kernel: [42869.386637]  start_secondary+0x166/0x1c0
Feb 27 13:02:05 pve kernel: [42869.386639]  secondary_startup_64+0xa4/0xb0
Feb 27 13:02:05 pve kernel: [42869.386641] ---[ end trace 0fc9293d87fbf079 ]---
Feb 27 13:02:05 pve kernel: [42869.386657] igb 0000:0b:00.0 enp11s0: Reset adapter
Feb 27 13:02:05 pve kernel: [42869.439085] vmbr0: port 1(enp11s0) entered disabled state
Feb 27 13:02:07 pve kernel: [42871.632955] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:09 pve kernel: [42873.039041] igb 0000:0b:00.0 enp11s0: igb: enp11s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Feb 27 13:02:09 pve kernel: [42873.146735] vmbr0: port 1(enp11s0) entered blocking state
Feb 27 13:02:09 pve kernel: [42873.146737] vmbr0: port 1(enp11s0) entered forwarding state
Feb 27 13:02:11 pve kernel: [42875.686052] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:15 pve kernel: [42879.729130] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:19 pve kernel: [42883.774845] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:23 pve kernel: [42887.826045] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:27 pve kernel: [42891.877783] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
Feb 27 13:02:31 pve kernel: [42895.926072] ixgbe 0000:0c:00.1 enp12s0f1: VF Reset msg received from vf 0
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Feb 27 13:03:38 p$
 
The logs appear to indicate a network connection or network card issue (timing out) and even a network card driver crash.
I don't think it causes a freeze, just no more network connection to the machine. Try a different network card (and connect a screen and keyboard)?
 
The logs appear to indicate a network connection or network card issue (timing out) and even a network card driver crash.
I don't think it causes a freeze, just no more network connection to the machine. Try a different network card (and connect a screen and keyboard)?
Thanks for the reply, it is so random that it just happened again - two hours ago before I have left my apartment it was working,
now I can't access any server on it. The loss of internet (the NIC shown in the message) should not disable the management: My Windows PC was configured with a fixed IP within the same subnet, and the Proxmox server has a third NIC for management only. This was done such that even when the router OS is dead I can still access the mangement WebGUI, but now when it does freeze even this dies.

I will probably open it up tonight and check if things have moved inside due to shipping, but otherwise I have zero clue. I can try to hook it up to a monitor, but it is a bit far away from the monitor and it has no GPU with it at the moment.


Sorry, I could actually access the router through internet and its WebGUI, this time it might just be a routing issue. I will report back if it does freeze again.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!