[SOLVED] Automatic restart of Proxmox server

[Giova.]

A few days ago I installed Proxmox VE 8.0 on my computer where I created two virtual machines: Windows 10 and Windows Server 2022. I use the first one for application tests while in the second one I have created an FTP server.

Whenever I try to access the ftp server or to the computer's file in general from my file explorer the proxmox server reboots, shutting down all the active VM; In fact sometimes it even reboots doing nothing.

I get this error: Error: unable to read tail(got 0 byte).

I am a beginner in proxmox, I have not studied any guides.

Any help would be appreciated because I have no idea what the problem might be.

 

[LnxBil]

Is it a "real" reboot or is PVE crashing? Maybe provide some logs from /var/log/syslog around the time of the behavior.

 

[Giova.]

Yes, the reboot is really of the real server preceded, however, by a VM crash. How can I get the log in /var/log/syslog, I don't know how to do it

 

[LnxBil]

I suppose you're on Windows on your client? Use WinSCP to log into your machine, navigate to /var/log and copy the file syslog. Open it in a editor of your choice and copy lines around the reboot (e.g. on pastebin if it's really large).

 

[Giova.]

Yes, the computer with which I access the server is Windows 10. I saw the contents of the folder /var/log/ in the server shell with the command 'dir'.
The result is in the screenshot, but no syslog file is present.
I did not understand how WinSCP could help on my real computer

 

[leesteken]

Yes, the computer with which I access the server is Windows 10. I saw the contents of the folder /var/log/ in the server shell with the command 'dir'.
The result is in the screenshot, but no syslog file is present.

Run journalctl and scroll using the arrow keys on the keyboard to look in the system log. Or look at the Node > System > Syslog page of the Proxmox web GUI.

I did not understand how WinSCP could help on my real computer

Then you can copy-paste text instead of people here having to look at screenshots, which only show part of the information.

 

[LnxBil]

Run journalctl and scroll using the arrow keys on the keyboard to look in the system log.

Oh yeah ... no rsyslog anymore in Bookworm. Still not used to it. I hate it that more and more is done with stuff from the SystemD eco system. It's going to replace all.

 

[Giova.]

Here is the syslog. At 22:19:30 the server restarted for no reason

Sep 11 22:16:00 www kernel: fwbr101i0: port 2(tap101i0) entered disabled state
Sep 11 22:16:00 www kernel: fwbr101i0: port 1(fwln101i0) entered disabled state
Sep 11 22:16:00 www kernel: vmbr0: port 3(fwpr101p0) entered disabled state
Sep 11 22:16:00 www kernel: device fwln101i0 left promiscuous mode
Sep 11 22:16:00 www kernel: fwbr101i0: port 1(fwln101i0) entered disabled state
Sep 11 22:16:00 www kernel: device fwpr101p0 left promiscuous mode
Sep 11 22:16:00 www kernel: vmbr0: port 3(fwpr101p0) entered disabled state
Sep 11 22:16:01 www qmeventd[695]: read: Connection reset by peer
Sep 11 22:16:01 www pvedaemon[1043]: VM 101 qmp command failed - VM 101 not running
Sep 11 22:16:01 www qmeventd[6556]: Starting cleanup for 101
Sep 11 22:16:01 www qmeventd[6556]: Finished cleanup for 101
Sep 11 22:16:01 www pvedaemon[1043]: <root@pam> end task UPID:www:00001989:0002FE96:64FF757B:vncproxy:101:root@pam: OK
Sep 11 22:16:02 www systemd[1]: 101.scope: Deactivated successfully.
Sep 11 22:16:02 www systemd[1]: 101.scope: Consumed 4min 13.218s CPU time.
Sep 11 22:16:03 www pvedaemon[1042]: <root@pam> starting task UPID:www:0000199D:0003019C:64FF7583:vncproxy:100:root@pam:
Sep 11 22:16:03 www pvedaemon[6557]: starting vnc proxy UPID:www:0000199D:0003019C:64FF7583:vncproxy:100:root@pam:
Sep 11 22:17:01 www CRON[6693]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 22:17:01 www CRON[6694]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 22:17:01 www CRON[6693]: pam_unix(cron:session): session closed for user root
-- Reboot --
Sep 11 22:20:49 www kernel: Linux version 6.2.16-3-pve (tom@sbuild) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) ()
Sep 11 22:20:49 www kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-3-pve root=/dev/mapper/pve-root ro quiet
Sep 11 22:20:49 www kernel: KERNEL supported cpus:
Sep 11 22:20:49 www kernel:   Intel GenuineIntel
Sep 11 22:20:49 www kernel:   AMD AuthenticAMD
Sep 11 22:20:49 www kernel:   Hygon HygonGenuine
Sep 11 22:20:49 www kernel:   Centaur CentaurHauls
Sep 11 22:20:49 www kernel:   zhaoxin   Shanghai 
Sep 11 22:20:49 www kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Sep 11 22:20:49 www kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Sep 11 22:20:49 www kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Sep 11 22:20:49 www kernel: x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
Sep 11 22:20:49 www kernel: x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Sep 11 22:20:49 www kernel: signal: max sigframe size: 1776
Sep 11 22:20:49 www kernel: BIOS-provided physical RAM map:
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x0000000000000000-0x0000000000057fff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x0000000000058000-0x0000000000058fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x0000000000059000-0x000000000009efff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000c6059fff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000c605a000-0x00000000c6060fff] ACPI NVS
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000c6061000-0x00000000c64b4fff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000c64b5000-0x00000000c68d8fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000c68d9000-0x00000000d8697fff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d8698000-0x00000000d88a0fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d88a1000-0x00000000d88b7fff] ACPI data
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d88b8000-0x00000000d8df8fff] ACPI NVS
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d8df9000-0x00000000d9f58fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d9f59000-0x00000000d9ffefff] type 20
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d9fff000-0x00000000d9ffffff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000db000000-0x00000000df1fffff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000fed00000-0x00000000fed03fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x0000000100000000-0x000000021fdfffff] usable
Sep 11 22:20:49 www kernel: NX (Execute Disable) protection: active
Sep 11 22:20:49 www kernel: efi: EFI v2.31 by American Megatrends
Sep 11 22:20:49 www kernel: efi: ESRT=0xd9f57898 ACPI=0xd88a5000 ACPI 2.0=0xd88a5000 SMBIOS=0xf04c0
Sep 11 22:20:49 www kernel: efi: Remove mem145: MMIO range=[0xf8000000-0xfbffffff] (64MB) from e820 map
Sep 11 22:20:49 www kernel: e820: remove [mem 0xf8000000-0xfbffffff] reserved
Sep 11 22:20:49 www kernel: efi: Not removing mem146: MMIO range=[0xfec00000-0xfec00fff] (4KB) from e820 map
Sep 11 22:20:49 www kernel: efi: Not removing mem147: MMIO range=[0xfed00000-0xfed03fff] (16KB) from e820 map
Sep 11 22:20:49 www kernel: efi: Not removing mem148: MMIO range=[0xfed1c000-0xfed1ffff] (16KB) from e820 map
Sep 11 22:20:49 www kernel: efi: Not removing mem149: MMIO range=[0xfee00000-0xfee00fff] (4KB) from e820 map
Sep 11 22:20:49 www kernel: efi: Remove mem150: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 map
Sep 11 22:20:49 www kernel: e820: remove [mem 0xff000000-0xffffffff] reserved
Sep 11 22:20:49 www kernel: secureboot: Secure boot disabled
Sep 11 22:20:49 www kernel: SMBIOS 2.7 present.
Sep 11 22:20:49 www kernel: DMI: ASUS All Series/Z87-K, BIOS 1401 07/29/2014
Sep 11 22:20:49 www kernel: tsc: Fast TSC calibration using PIT
Sep 11 22:20:49 www kernel: tsc: Detected 4000.415 MHz processor
Sep 11 22:20:49 www kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Sep 11 22:20:49 www kernel: e820: remove [mem 0x000a0000-0x000fffff] usable
Sep 11 22:20:49 www kernel: last_pfn = 0x21fe00 max_arch_pfn = 0x400000000
Sep 11 22:20:49 www kernel: total RAM covered: 8110M
Sep 11 22:20:49 www kernel: Found optimal setting for mtrr clean up
Sep 11 22:20:49 www kernel:  gran_size: 64K     chunk_size: 128M     num_reg: 8      lose cover RAM: 0G
Sep 11 22:20:49 www kernel: x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT 
Sep 11 22:20:49 www kernel: e820: update [mem 0xdb000000-0xffffffff] usable ==> reserved
Sep 11 22:20:49 www kernel: last_pfn = 0xda000 max_arch_pfn = 0x400000000
Sep 11 22:20:49 www kernel: found SMP MP-table at [mem 0x000fd760-0x000fd76f]
Sep 11 22:20:49 www kernel: esrt: Reserving ESRT space from 0x00000000d9f57898 to 0x00000000d9f578d0.
Sep 11 22:20:49 www kernel: Using GB pages for direct mapping
Sep 11 22:20:49 www kernel: secureboot: Secure boot disabled

 

[leesteken]

Here is the syslog. At 22:19:30 the server restarted for no reason

Sep 11 22:16:00 www kernel: fwbr101i0: port 2(tap101i0) entered disabled state
Sep 11 22:16:00 www kernel: fwbr101i0: port 1(fwln101i0) entered disabled state
Sep 11 22:16:00 www kernel: vmbr0: port 3(fwpr101p0) entered disabled state
Sep 11 22:16:00 www kernel: device fwln101i0 left promiscuous mode
Sep 11 22:16:00 www kernel: fwbr101i0: port 1(fwln101i0) entered disabled state
Sep 11 22:16:00 www kernel: device fwpr101p0 left promiscuous mode
Sep 11 22:16:00 www kernel: vmbr0: port 3(fwpr101p0) entered disabled state
Sep 11 22:16:01 www qmeventd[695]: read: Connection reset by peer
Sep 11 22:16:01 www pvedaemon[1043]: VM 101 qmp command failed - VM 101 not running
Sep 11 22:16:01 www qmeventd[6556]: Starting cleanup for 101
Sep 11 22:16:01 www qmeventd[6556]: Finished cleanup for 101
Sep 11 22:16:01 www pvedaemon[1043]: <root@pam> end task UPID:www:00001989:0002FE96:64FF757B:vncproxy:101:root@pam: OK
Sep 11 22:16:02 www systemd[1]: 101.scope: Deactivated successfully.
Sep 11 22:16:02 www systemd[1]: 101.scope: Consumed 4min 13.218s CPU time.
Sep 11 22:16:03 www pvedaemon[1042]: <root@pam> starting task UPID:www:0000199D:0003019C:64FF7583:vncproxy:100:root@pam:
Sep 11 22:16:03 www pvedaemon[6557]: starting vnc proxy UPID:www:0000199D:0003019C:64FF7583:vncproxy:100:root@pam:
Sep 11 22:17:01 www CRON[6693]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 11 22:17:01 www CRON[6694]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Sep 11 22:17:01 www CRON[6693]: pam_unix(cron:session): session closed for user root
-- Reboot --
Sep 11 22:20:49 www kernel: Linux version 6.2.16-3-pve (tom@sbuild) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) ()
Sep 11 22:20:49 www kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-3-pve root=/dev/mapper/pve-root ro quiet
Sep 11 22:20:49 www kernel: KERNEL supported cpus:
Sep 11 22:20:49 www kernel:   Intel GenuineIntel
Sep 11 22:20:49 www kernel:   AMD AuthenticAMD
Sep 11 22:20:49 www kernel:   Hygon HygonGenuine
Sep 11 22:20:49 www kernel:   Centaur CentaurHauls
Sep 11 22:20:49 www kernel:   zhaoxin   Shanghai
Sep 11 22:20:49 www kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Sep 11 22:20:49 www kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Sep 11 22:20:49 www kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Sep 11 22:20:49 www kernel: x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
Sep 11 22:20:49 www kernel: x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Sep 11 22:20:49 www kernel: signal: max sigframe size: 1776
Sep 11 22:20:49 www kernel: BIOS-provided physical RAM map:
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x0000000000000000-0x0000000000057fff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x0000000000058000-0x0000000000058fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x0000000000059000-0x000000000009efff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x000000000009f000-0x000000000009ffff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000c6059fff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000c605a000-0x00000000c6060fff] ACPI NVS
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000c6061000-0x00000000c64b4fff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000c64b5000-0x00000000c68d8fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000c68d9000-0x00000000d8697fff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d8698000-0x00000000d88a0fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d88a1000-0x00000000d88b7fff] ACPI data
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d88b8000-0x00000000d8df8fff] ACPI NVS
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d8df9000-0x00000000d9f58fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d9f59000-0x00000000d9ffefff] type 20
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000d9fff000-0x00000000d9ffffff] usable
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000db000000-0x00000000df1fffff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000f8000000-0x00000000fbffffff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000fed00000-0x00000000fed03fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
Sep 11 22:20:49 www kernel: BIOS-e820: [mem 0x0000000100000000-0x000000021fdfffff] usable
Sep 11 22:20:49 www kernel: NX (Execute Disable) protection: active
Sep 11 22:20:49 www kernel: efi: EFI v2.31 by American Megatrends
Sep 11 22:20:49 www kernel: efi: ESRT=0xd9f57898 ACPI=0xd88a5000 ACPI 2.0=0xd88a5000 SMBIOS=0xf04c0
Sep 11 22:20:49 www kernel: efi: Remove mem145: MMIO range=[0xf8000000-0xfbffffff] (64MB) from e820 map
Sep 11 22:20:49 www kernel: e820: remove [mem 0xf8000000-0xfbffffff] reserved
Sep 11 22:20:49 www kernel: efi: Not removing mem146: MMIO range=[0xfec00000-0xfec00fff] (4KB) from e820 map
Sep 11 22:20:49 www kernel: efi: Not removing mem147: MMIO range=[0xfed00000-0xfed03fff] (16KB) from e820 map
Sep 11 22:20:49 www kernel: efi: Not removing mem148: MMIO range=[0xfed1c000-0xfed1ffff] (16KB) from e820 map
Sep 11 22:20:49 www kernel: efi: Not removing mem149: MMIO range=[0xfee00000-0xfee00fff] (4KB) from e820 map
Sep 11 22:20:49 www kernel: efi: Remove mem150: MMIO range=[0xff000000-0xffffffff] (16MB) from e820 map
Sep 11 22:20:49 www kernel: e820: remove [mem 0xff000000-0xffffffff] reserved
Sep 11 22:20:49 www kernel: secureboot: Secure boot disabled
Sep 11 22:20:49 www kernel: SMBIOS 2.7 present.
Sep 11 22:20:49 www kernel: DMI: ASUS All Series/Z87-K, BIOS 1401 07/29/2014
Sep 11 22:20:49 www kernel: tsc: Fast TSC calibration using PIT
Sep 11 22:20:49 www kernel: tsc: Detected 4000.415 MHz processor
Sep 11 22:20:49 www kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
Sep 11 22:20:49 www kernel: e820: remove [mem 0x000a0000-0x000fffff] usable
Sep 11 22:20:49 www kernel: last_pfn = 0x21fe00 max_arch_pfn = 0x400000000
Sep 11 22:20:49 www kernel: total RAM covered: 8110M
Sep 11 22:20:49 www kernel: Found optimal setting for mtrr clean up
Sep 11 22:20:49 www kernel:  gran_size: 64K     chunk_size: 128M     num_reg: 8      lose cover RAM: 0G
Sep 11 22:20:49 www kernel: x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT
Sep 11 22:20:49 www kernel: e820: update [mem 0xdb000000-0xffffffff] usable ==> reserved
Sep 11 22:20:49 www kernel: last_pfn = 0xda000 max_arch_pfn = 0x400000000
Sep 11 22:20:49 www kernel: found SMP MP-table at [mem 0x000fd760-0x000fd76f]
Sep 11 22:20:49 www kernel: esrt: Reserving ESRT space from 0x00000000d9f57898 to 0x00000000d9f578d0.
Sep 11 22:20:49 www kernel: Using GB pages for direct mapping
Sep 11 22:20:49 www kernel: secureboot: Secure boot disabled

That does not look like a graceful shutdown (no signs of VM/CTs being shutdown) and could well have been a power interruption or someone pressing the reset button.

 

[LnxBil]

That does not look like a graceful shutdown (no signs of VM/CTs being shutdown) and could well have been a power interruption or someone pressing the reset button.

That's what I think. The problem is now, further debugging into the real reason is going to involve a lot of stuff including another computer. AFAIK, we still have no instructions in how to get crashdump to work on PVE in the handbook, have we? Neither a netconsole, so we have currently no way in getting useful logs.

Maybe another approach: Have you configured PCIe passthrough in your VM? Is there anything special configured there that could potentially crash your machine? What about the configured guest RAM with respect to the RAM of your PVE host?

 

[Giova.]

I checked the settings of each VM and noticed that the sum of the RAM used by the VMs was greater than the actual RAM. After decreasing the amount of RAM for each VM, I thought I had solved the problem: I used the server throughout the morning without any problems.
But after a while the error reoccurred.
I did not activate the PCIe passthrough.
Unfortunately, with these sudden reboots, one of my Windows Server VMs no longer turned on, so I recreated it. I wouldn't want it to happen again

 

[LnxBil]

one of my Windows Server VMs no longer turned on, so I recreated it.

What an OS for a server ... I still don't get why people use MS software at all.

I wouldn't want it to happen again

Of coures. What storage does you use on your PVE host? Maybe you need even more ram dedicated for your PVE host. You can also install zram-tools for having compressed swap memory.

 

[Giova.]

In the PVE host I use a 230 gigabyte SSD and 8 gigabytes of RAM, which I can also increase to 12gb.
What advantages can Zram-tools give? I have never heard of them. How do I install it?
Anyway, I did a check on the computer and found the only component that could cause the reboot is either faulty RAM or the processor. Could it be?

 

[Giova.]

I solved the problem because on rechecking the pve host I found the ATX cable of the motherboard slightly disconnected. Sorry for creating this discussion unnecessarily. Anyway, thank you very much!

 

[LnxBil]

What advantages can Zram-tools give? I have never heard of them. How do I install it?

You can add swap to your system besides having disk space. Afterwards, an OOM event is less likely.