Proxmox Host Reboot unexpected

[Damonz]

Hi My Proxmox Host Server reboot unexpected on 17-Dec 2:0x am , What I can found is only its doing the backup job , then it reboot . Can anyone help ?

•Dec 17 02:00:03 cancer-anf-hk4 pvescheduler[153377]: INFO: starting new backup job: vzdump 101 --quiet 1 --mode snapshot --notes-template '{{guestname}}' --storage backup --mailnotification always --compress zstd
•Dec 17 02:09:05 cancer-anf-hk4 systemd-fsck[1310]: There are differences between boot sector and its backup.
•Dec 17 02:09:05 cancer-anf-hk4 systemd-fsck[1310]: This is mostly harmless. Differences: (offsetriginal/backup)
•Dec 17 02:09:06 cancer-anf-hk4 kernel: bond0: (slave ens6f0np0): Enslaving as a backup interface with an up link
•Dec 17 02:09:06 cancer-anf-hk4 kernel: bond0: (slave ens6f1np1): Enslaving as a backup interface with an up link
•Dec 17 02:09:06 cancer-anf-hk4 systemd[1]: Started dpkg-db-backup.timer - Daily dpkg database backup timer.
•Dec 17 02:09:10 cancer-anf-hk4 pvesh[1848]: removed left over backup lock from '101'!
•Dec 17 02:09:10 cancer-anf-hk4 pve-guests[1850]: removed left over backup lock from '101'!
•Dec 17 02:09:21 cancer-anf-hk4 nfsrahead[1916]: setting /mnt/pve/backup readahead to 128

 

[sb-jw]

Please post larger excerpts from your syslog and give more details about your setup.

 

[Damonz]

Please post larger excerpts from your syslog and give more details about your setup.

Hi I can only found the below journal , the server is reboot around Dec 17 2:06

Dec 17 02:00:03 machine1 pvescheduler[153376]: <root@pam> starting task UPID:machine1:00025721:2513D9E8:657DE5A3:vzdump:101:root@pam:
Dec 17 02:00:03 machine1 pvescheduler[153377]: INFO: starting new backup job: vzdump 101 --quiet 1 --mode snapshot --notes-template '{{guestname}}' --storage backup --mailnotification always --compress zstd
Dec 17 02:00:03 machine1 pvescheduler[153377]: INFO: Starting Backup of VM 101 (qemu)
Dec 17 02:05:59 machine1 pve-firewall[1789]: firewall update time (38.982 seconds)
Dec 17 02:09:04 machine1 kernel: Linux version 6.2.16-3-pve (tom@sbuild) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC PVE 6.2.16-3 (2023-06-17T05:58Z) ()
Dec 17 02:09:04 machine1 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.2.16-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on
Dec 17 02:09:04 machine1 kernel: KERNEL supported cpus:
Dec 17 02:09:04 machine1 kernel: Intel GenuineIntel
Dec 17 02:09:04 machine1 kernel: AMD AuthenticAMD
Dec 17 02:09:04 machine1 kernel: Hygon HygonGenuine
Dec 17 02:09:04 machine1 kernel: Centaur CentaurHauls
Dec 17 02:09:04 machine1 kernel: zhaoxin Shanghai

 

[sb-jw]

There must be more to find under /var/log/syslog*. At least you can't read much from the log.

and give more details about your setup.

And what about your setup?

 

[Damonz]

I am using the PVE 8.0.3 , by default it seems not have syslog, so I am hard to trouble shoot . If you got any idea to find more log, please advice . Many thanks.
For my system step , existing only one PVE host , 2VM on the Host.
Intel(R) Xeon(R) Gold 5317 CPU @ 3.00GHz x 2 , 512GB Memory.

 

[sb-jw]

Does the server always reboot after a backup or is it always just the same time?
Does it happen more often? If yes, how often?

Please post the configs of your two VMs here in code tags qm config VMID

 

[Damonz]

Does the server always reboot after a backup or is it always just the same time?
Does it happen more often? If yes, how often?

Please post the configs of your two VMs here in code tags qm config VMID

Only happen twice , last time is Oct 6 , also happen during the backup period . I am suspect if the insufficient backup storage cause this issue .

VM100:
qm config 100
agent: 1
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
description:
ide2: local:iso/ubuntu-22.04.2-live-server-amd64.iso,media=cdrom,size=1929660K
memory: 8000
meta: creation-qemu=8.0.2,ctime=1692946303
name: dtf2-pve-hk4
net0: virtio=0A:A3:C9:97:14:92,bridge=vmbr1
numa: 0
ostype: l26
parent: dockerinstall
scsi0: local-lvm:vm-100-disk-0,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=d3e2bdca-6374-4c99-a5db-9698b93af5b3
sockets: 1
vmgenid: 5266e176-c75a-49e5-a8a0-c29e6735d7bb
VM101
qm config 101
agent: 1
boot: order=scsi0;ide2;net0
cores: 12
cpu: host
description: %0Ascsi1%3A temp-data%3Avm-101-disk-0,backup=0,iothread=1,size=10T
ide2: local:iso/ubuntu-22.04.2-live-server-amd64.iso,media=cdrom,size=1929660K
memory: 512000
meta: creation-qemu=8.0.2,ctime=1691998157
name: cancer-pve-hk4
net0: virtio=EE:A9:25:87:35:5F,bridge=vmbr1
numa: 0
ostype: l26
scsi0: data2:vm-101-disk-0,iothread=1,size=400G
scsi1: temp-data:vm-101-disk-1,backup=0,iothread=1,size=10T
scsihw: virtio-scsi-single
smbios1: uuid=eea393bf-d979-424d-b208-6f216e5171ad
sockets: 2
vmgenid: ef7ba736-c2c9-4f6c-adc1-3d708ac52475

 

[sb-jw]

You have 520 GB of RAM allocated to the VMs and your host only has 512 GB. I think you're going out of memory and killing your node. You should leave the host with 4 - 8 GB of RAM and if ZFS is used, you should increase this significantly.