Hi folks!
I've got a Proxmox host that is freezing pretty much on a daily basis, and I'm looking for advice on how to get started debugging it. I've tried looking at the logs and don't see an obvious answer, but I'm not sure I know what to look for.
I've got the host connected to a KVM and when it freezes the host and the VMs on it disappear from my network, but on the KVM I can see the login shell ("Welcome to the Proxmox Virtual Environment") but it's not responsive and I can't log in, even if I attach a physical keyboard. I also ran memtest to see if there was an issue with the ram and that passed without complaint.
I'm happy to provide more information about the machine, logs, etc. if that would help.
Some of the questions I have in mind are:
Update: Providing some additional information based on @esi_y 's comments. journalctl output is attached, and here are some more details on the hardware.
![Screenshot 2024-08-17 at 1.51.54 PM.png Screenshot 2024-08-17 at 1.51.54 PM.png](https://forum.proxmox.com/data/attachments/72/72891-f46e467c4742443f5571d62c3c115c51.jpg)
I've got a Proxmox host that is freezing pretty much on a daily basis, and I'm looking for advice on how to get started debugging it. I've tried looking at the logs and don't see an obvious answer, but I'm not sure I know what to look for.
I've got the host connected to a KVM and when it freezes the host and the VMs on it disappear from my network, but on the KVM I can see the login shell ("Welcome to the Proxmox Virtual Environment") but it's not responsive and I can't log in, even if I attach a physical keyboard. I also ran memtest to see if there was an issue with the ram and that passed without complaint.
I'm happy to provide more information about the machine, logs, etc. if that would help.
Some of the questions I have in mind are:
- What logs should I be looking at?
- What sorts of issues could cause the machine to freeze but not just completely crash?
- What would be the recommended debugging steps for isolating the problem?
Update: Providing some additional information based on @esi_y 's comments. journalctl output is attached, and here are some more details on the hardware.
![Screenshot 2024-08-17 at 1.51.54 PM.png Screenshot 2024-08-17 at 1.51.54 PM.png](https://forum.proxmox.com/data/attachments/72/72891-f46e467c4742443f5571d62c3c115c51.jpg)
Bash:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 43 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
BIOS Vendor ID: Advanced Micro Devices, Inc.
Model name: AMD Ryzen 9 3950X 16-Core Processor
BIOS Model name: AMD Ryzen 9 3950X 16-Core Processor Unknown CPU @ 3.5GHz
BIOS CPU family: 107
CPU family: 23
Model: 113
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 0
Frequency boost: enabled
CPU(s) scaling MHz: 75%
CPU max MHz: 4761.2300
CPU min MHz: 2200.0000
BogoMIPS: 6999.57
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse
2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid e
xtd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcn
t aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dno
wprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb c
at_l3 cdp_l3 hw_pstate ssbd mba ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed
adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm
_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vm
cb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl
umip rdpid overflow_recov succor smca sev sev_es
Virtualization features:
Virtualization: AMD-V
Caches (sum of all):
L1d: 512 KiB (16 instances)
L1i: 512 KiB (16 instances)
L2: 8 MiB (16 instances)
L3: 64 MiB (4 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-31
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Reg file data sampling: Not affected
Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection
Spec rstack overflow: Mitigation; Safe RET
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected
; BHI Not affected
Srbds: Not affected
Tsx async abort: Not affected
Bash:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdi 8:128 0 238.5G 0 disk
├─sdi1 8:129 0 1007K 0 part
├─sdi2 8:130 0 1G 0 part /boot/efi
└─sdi3 8:131 0 237.5G 0 part
├─pve-swap 252:0 0 8G 0 lvm [SWAP]
├─pve-root 252:1 0 69.4G 0 lvm /
├─pve-data_tmeta 252:2 0 1.4G 0 lvm
│ └─pve-data-tpool 252:4 0 141.2G 0 lvm
│ ├─pve-data 252:5 0 141.2G 1 lvm
│ ├─pve-vm--100--disk--0 252:6 0 4M 0 lvm
│ ├─pve-vm--100--disk--1 252:7 0 20G 0 lvm
│ └─pve-vm--101--disk--0 252:8 0 32G 0 lvm
└─pve-data_tdata 252:3 0 141.2G 0 lvm
└─pve-data-tpool 252:4 0 141.2G 0 lvm
├─pve-data 252:5 0 141.2G 1 lvm
├─pve-vm--100--disk--0 252:6 0 4M 0 lvm
├─pve-vm--100--disk--1 252:7 0 20G 0 lvm
└─pve-vm--101--disk--0 252:8 0 32G 0 lvm
sr0 11:0 1 1024M 0 rom
Bash:
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7
02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset USB 3.1 XHCI Controller
02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset SATA Controller
02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset Switch Upstream Port
03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
29:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2)
29:00.1 Audio device: NVIDIA Corporation GM107 High Definition Audio Controller [GeForce 940MX] (rev a1)
2a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
2b:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3224 PCI-Express Fusion-MPT SAS-3 (rev 01)
2c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
2d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
2d:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
2d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
2d:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
31:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
Attachments
Last edited: