Proxmox self reboot sometimes

[riri1310]

Hi all,

I just noticed that proxmox restart all VM but can't find the reason could you please help me with the syslog below?

Feb 15 17:19:00 ns3128946 systemd[1]: Starting Proxmox VE replication runner...                                                                                             
Feb 15 17:19:00 ns3128946 systemd[1]: Started Proxmox VE replication runner.                                                                                                 
Feb 15 17:20:00 ns3128946 systemd[1]: Starting Proxmox VE replication runner...                                                                                             
Feb 15 17:20:00 ns3128946 systemd[1]: Started Proxmox VE replication runner.                                                                                                 
Feb 15 17:21:00 ns3128946 pve-firewall[2704]: firewall update time (6.873 seconds)                                                                                           
Feb 15 17:21:11 ns3128946 pve-firewall[2704]: firewall update time (9.220 seconds)                                                                                           
Feb 15 17:21:15 ns3128946 systemd[1]: Starting Proxmox VE replication runner...                                                                                             
Feb 15 17:21:29 ns3128946 pve-firewall[2704]: firewall update time (14.626 seconds)                                                                                         
Feb 15 17:21:33 ns3128946 pveproxy[24544]: proxy detected vanished client connection                                                                                         
Feb 15 17:21:54 ns3128946 pvestatd[2716]: command 'zpool list -o name -H rpool' failed: got timeout                                                                         
Feb 15 17:22:12 ns3128946 pve-firewall[2704]: firewall update time (42.569 seconds)                                                                                         
Feb 15 17:22:19 ns3128946 pvedaemon[4312]: authentication failure; rhost=5.42.199.51 user=root@pam msg=Échec d'authentification                                             
Feb 15 17:22:30 ns3128946 pvestatd[2716]: storage 'ftpbackup_ns3126' is not online                                                                                       
Feb 15 17:22:31 ns3128946 pve-ha-lrm[2996]: loop take too long (35 seconds)                                                                                                 
Feb 15 17:26:38 ns3128946 systemd-modules-load[1088]: Inserted module 'vhost_net'                                                                                           
Feb 15 17:26:38 ns3128946 systemd-sysctl[1098]: Couldn't write '0' to 'net/ipv6/conf/eno1/autoconf', ignoring: No such file or directory                                     
Feb 15 17:26:38 ns3128946 systemd-sysctl[1098]: Couldn't write '0' to 'net/ipv6/conf/eno1/accept_ra', ignoring: No such file or directory                                   
Feb 15 17:26:38 ns3128946 keyboard-setup.sh[1063]: Impossible d'ouvrir le fichier /tmp/tmpkbd.cdsV0Q                                                                         
Feb 15 17:26:38 ns3128946 systemd-sysctl[1098]: Couldn't write '0' to 'net/ipv6/conf/eno1/accept_ra_defrtr', ignoring: No such file or directory                             
Feb 15 17:26:38 ns3128946 systemd-sysctl[1098]: Couldn't write '0' to 'net/ipv6/conf/eno1/accept_ra_pinfo', ignoring: No such file or directory                             
Feb 15 17:26:38 ns3128946 systemd[1]: Starting Flush Journal to Persistent Storage...                                                                                       
Feb 15 17:26:38 ns3128946 systemd-udevd[1117]: Network interface NamePolicy= disabled on kernel command line, ignoring.                                                     
Feb 15 17:26:38 ns3128946 systemd-udevd[1117]: [/etc/systemd/network/50-public-interface.link:7] Unknown lvalue 'Name' in section 'Match'                                   
Feb 15 17:26:38 ns3128946 systemd[1]: Started udev Kernel Device Manager.                                                                                                   
Feb 15 17:26:38 ns3128946 systemd[1]: Started Flush Journal to Persistent Storage.                                                                                           
Feb 15 17:26:38 ns3128946 systemd-udevd[1143]: Process '/bin/mount -t fusectl fusectl /sys/fs/fuse/connections' failed with exit code 32.                                   
Feb 15 17:26:38 ns3128946 systemd[1]: Started udev Coldplug all Devices.                                                                                                     
Feb 15 17:26:38 ns3128946 systemd[1]: Starting udev Wait for Complete Device Initialization...                                                                               
Feb 15 17:26:38 ns3128946 systemd[1]: Found device /dev/ttyS0.                                                                                                               
Feb 15 17:26:38 ns3128946 systemd[1]: Found device /dev/ttyS1.                                                                                                               
Feb 15 17:26:38 ns3128946 systemd[1]: Found device /dev/zvol/rpool/swap.                                                                                                     
Feb 15 17:26:38 ns3128946 systemd[1]: Activating swap /dev/zvol/rpool/swap...                                                                                               
Feb 15 17:26:38 ns3128946 systemd[1]: Activated swap /dev/zvol/rpool/swap.                                                                                                   
Feb 15 17:26:38 ns3128946 systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch.                                                                       
Feb 15 17:26:38 ns3128946 systemd[1]: Reached target Swap.                                                                                                                   
Feb 15 17:26:38 ns3128946 systemd[1]: Started udev Wait for Complete Device Initialization.                                                                                 
Feb 15 17:26:38 ns3128946 systemd[1]: Starting Activation of LVM2 logical volumes...                                                                                         
Feb 15 17:26:38 ns3128946 systemd[1]: Started Activation of LVM2 logical volumes.                                                                                           
Feb 15 17:26:38 ns3128946 systemd[1]: Reached target Encrypted Volumes.                                                                                                     
Feb 15 17:26:38 ns3128946 systemd[1]: Starting Activation of LVM2 logical volumes...                                                                                         
Feb 15 17:26:38 ns3128946 systemd[1]: Reached target ZFS pool import target.                                                                                                 
Feb 15 17:26:38 ns3128946 systemd[1]: Starting Mount ZFS filesystems...                                                                                                     
Feb 15 17:26:38 ns3128946 systemd[1]: Started Activation of LVM2 logical volumes.                                                                                           
Feb 15 17:26:38 ns3128946 systemd[1]: Starting Activation of LVM2 logical volumes...                                                                                         
Feb 15 17:26:38 ns3128946 systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...                                             
Feb 15 17:26:38 ns3128946 systemd[1]: Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.                                                 
Feb 15 17:26:38 ns3128946 systemd[1]: Started Activation of LVM2 logical volumes.                                                                                           
Feb 15 17:26:38 ns3128946 systemd[1]: Started Mount ZFS filesystems.                                                                                                         
Feb 15 17:26:38 ns3128946 systemd[1]: Started Set the console keyboard layout.

It happen sometimes and after the reboot everything work as expected but it might be a problem somewhere...

Thanks!

 

[Moayad]

Hi,

The syslog provided doesn't include why the VMs got restarted... but I see these two lines in the syslog:

Feb 15 17:22:19 ns3128946 pvedaemon[4312]: authentication failure; rhost=5.42.199.51 user=root@pam msg=Échec d'authentification

Feb 15 17:22:30 ns3128946 pvestatd[2716]: storage 'ftpbackup_ns3126' is not online

Does the VMs disks stored on the `ftpbackup_ns3126` storage?

 

[riri1310]

Hi,

The syslog provided doesn't include why the VMs got restarted... but I see these two lines in the syslog:





Does the VMs disks stored on the `ftpbackup_ns3126` storage?

Hi

The VMs disks are not stored on the `ftpbackup_ns3126` storage which is only the distant FTP disk for vzdump backup.

Thanks for your help

 

[Moayad]

Can you provide us with the syslog with the time when the VMs got restarted? to do this you can use journalctl by issuing the following command:

journalctl --since "2023-02-15 00:00" --until "2023-02-15 01:00" > /tmp/Syslog.log

You may change the time on the above command.

 

[riri1310]

here is the gist of the syslog

https://gist.github.com/2803media/777ebc1e10a2a8f799365e6b58d158e7

 

[Dunuin]

Looks more like the whole server crashed and rebooted.

 

[Moayad]

Thank you for the Syslog!

From the syslog you provided I see this `Feb 15 17:21:54 ns3128946 pvestatd[2716]: command 'zpool list -o name -H rpool' failed: got timeout` then looks the whole server got restarted?

 

[riri1310]

Yes it seem that the server crash (all VM restarted) but how can I find why it crash? Thanks for your insights

 

[Moayad]

You have to see the Syslog before the server restarted

 

[riri1310]

Yes indeed

As I see in log the problem start here

Feb 15 17:21:00 ns3128946 pve-firewall[2704]: firewall update time (6.873 seconds)

Which generate this:

Feb 15 17:21:33 ns3128946 pveproxy[24544]: proxy detected vanished client connection
Feb 15 17:21:54 ns3128946 pvestatd[2716]: command 'zpool list -o name -H rpool' failed: got timeout
Feb 15 17:22:12 ns3128946 pve-firewall[2704]: firewall update time (42.569 seconds)
Feb 15 17:22:19 ns3128946 pvedaemon[4312]: authentication failure; rhost=5.42.199.51 user=root@pam msg=Échec d'authentification
Feb 15 17:22:30 ns3128946 pvestatd[2716]: storage 'ftpbackup_ns3128946' is not online
Feb 15 17:22:31 ns3128946 pve-ha-lrm[2996]: loop take too long (35 seconds)

But I can't find the reason

 

[proxnoci]

Did the firewall block access to some iscsi / NAS ..?
Starting firewalls WHILE there still are sessions from before the firewall was running may cause the connection to get blocked/interrupted.
So starting a firewall before any host services are started will not cause this, replacing the firewall is done with an atomic transaction.

 

[riri1310]

So as I understand the "auto" update of the firewall crash all existing connection and with that crash the whole server... Is it possible?

 

[proxnoci]

If a single pair of iptables-save / update / iptables-restore with all changes in one go i would not expect it (unless intentional to block traffic).
if rules are added/removed one at a time it may be possible, if a rules is removed... some traffic happens that needed it, (and times out) before another is added.
I didn't study the PVE firewall enough to be more specific.
pve-firewall does use iptables-save / iptables-restore to make a change atomic. So updating an existing firewall should be rather safe. it should only fail links that are not allowed.