a pve server rebooted - any insight?

jmckee

Renowned Member
Oct 19, 2013
15
0
66
Since:



Until:


Happened during the night. Hopefully this log is useful - it is a bit opaque to me.. Server rebooted and seems normal again today... I think. Any insight is much appreciated especially if hardware is failing. I like to get right no top of that :)

Feb 05 23:01:43 proxmox1 pmxcfs[1474]: [status] notice: received log
Feb 05 23:16:43 proxmox1 pmxcfs[1474]: [status] notice: received log
Feb 05 23:17:01 proxmox1 CRON[2768566]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Feb 05 23:17:01 proxmox1 CRON[2768567]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 05 23:17:01 proxmox1 CRON[2768566]: pam_unix(cron:session): session closed for user root
Feb 05 23:39:19 proxmox1 pmxcfs[1474]: [dcdb] notice: data verification successful
Feb 06 00:00:02 proxmox1 pvescheduler[2778104]: <root@pam> starting task UPID:proxmox1:002A6400:317D2960:65C1E701:vzdump:103:root@pam:
Feb 06 00:00:05 proxmox1 pvescheduler[2778112]: INFO: starting new backup job: vzdump 103 --storage backups --mailto redacted--mode snapshot --mailnotification failure --quiet 1 --compress zstd --node proxmox1
Feb 06 00:00:05 proxmox1 pvescheduler[2778112]: INFO: Starting Backup of VM 103 (qemu)
Feb 06 00:00:08 proxmox1 systemd[1]: Starting Rotate log files...
Feb 06 00:00:08 proxmox1 systemd[1]: Starting Daily man-db regeneration...
Feb 06 00:00:09 proxmox1 systemd[1]: Reloading PVE API Proxy Server.
Feb 06 00:00:31 proxmox1 pvestatd[1545]: status update time (14.376 seconds)
Feb 06 00:01:10 proxmox1 pvestatd[1545]: status update time (9.197 seconds)
Feb 06 00:01:12 proxmox1 systemd[1]: man-db.service: Succeeded.
Feb 06 00:01:12 proxmox1 systemd[1]: Finished Daily man-db regeneration.
Feb 06 00:01:14 proxmox1 pvestatd[1545]: got timeout
Feb 06 00:01:23 proxmox1 pvestatd[1545]: got timeout
Feb 06 00:01:28 proxmox1 pvestatd[1545]: status update time (7.485 seconds)
Feb 06 00:01:39 proxmox1 pveproxy[2778164]: send HUP to 1609
Feb 06 00:01:39 proxmox1 pveproxy[1609]: received signal HUP
Feb 06 00:01:39 proxmox1 systemd[1]: Reloaded PVE API Proxy Server.
Feb 06 00:01:39 proxmox1 pveproxy[1609]: server closing
Feb 06 00:01:39 proxmox1 pveproxy[1609]: server shutdown (restart)
Feb 06 00:01:39 proxmox1 systemd[1]: Reloading PVE SPICE Proxy Server.
Feb 06 00:01:40 proxmox1 spiceproxy[2778620]: send HUP to 1616
Feb 06 00:01:40 proxmox1 systemd[1]: Reloaded PVE SPICE Proxy Server.
Feb 06 00:01:40 proxmox1 systemd[1]: Stopping Proxmox VE firewall logger...
Feb 06 00:01:40 proxmox1 pvefw-logger[2459839]: received terminate request (signal)
Feb 06 00:01:40 proxmox1 pvefw-logger[2459839]: stopping pvefw logger
Feb 06 00:01:40 proxmox1 spiceproxy[1616]: received signal HUP
Feb 06 00:01:40 proxmox1 spiceproxy[1616]: server closing
Feb 06 00:01:40 proxmox1 spiceproxy[1616]: server shutdown (restart)
Feb 06 00:01:41 proxmox1 systemd[1]: pvefw-logger.service: Succeeded.
Feb 06 00:01:41 proxmox1 systemd[1]: Stopped Proxmox VE firewall logger.
Feb 06 00:01:41 proxmox1 systemd[1]: pvefw-logger.service: Consumed 6.351s CPU time.
Feb 06 00:01:41 proxmox1 systemd[1]: Starting Proxmox VE firewall logger...
Feb 06 00:01:41 proxmox1 systemd[1]: Started Proxmox VE firewall logger.
Feb 06 00:01:41 proxmox1 pvefw-logger[2778630]: starting pvefw logger
Feb 06 00:01:41 proxmox1 systemd[1]: logrotate.service: Succeeded.
Feb 06 00:01:41 proxmox1 systemd[1]: Finished Rotate log files.
Feb 06 00:01:41 proxmox1 spiceproxy[1616]: restarting server
Feb 06 00:01:42 proxmox1 spiceproxy[1616]: starting 1 worker(s)
Feb 06 00:01:42 proxmox1 spiceproxy[1616]: worker 2778640 started
Feb 06 00:01:42 proxmox1 pveproxy[1609]: restarting server
Feb 06 00:01:42 proxmox1 pveproxy[1609]: starting 3 worker(s)
Feb 06 00:01:42 proxmox1 pveproxy[1609]: worker 2778642 started
Feb 06 00:01:42 proxmox1 pveproxy[1609]: worker 2778643 started
Feb 06 00:01:42 proxmox1 pveproxy[1609]: worker 2778644 started
Feb 06 00:01:47 proxmox1 pveproxy[2692965]: worker exit
Feb 06 00:01:47 proxmox1 pveproxy[2752908]: worker exit
Feb 06 00:01:47 proxmox1 pveproxy[2741582]: worker exit
Feb 06 00:01:47 proxmox1 spiceproxy[2142143]: worker exit
Feb 06 00:01:51 proxmox1 spiceproxy[1616]: worker 2142143 finished
Feb 06 00:01:52 proxmox1 pveproxy[1609]: worker 2692965 finished
Feb 06 00:01:52 proxmox1 pveproxy[1609]: worker 2752908 finished
Feb 06 00:01:52 proxmox1 pveproxy[1609]: worker 2741582 finished
Feb 06 00:02:04 proxmox1 pvestatd[1545]: got timeout
Feb 06 00:02:07 proxmox1 pvestatd[1545]: status update time (5.557 seconds)
Feb 06 00:02:19 proxmox1 pvestatd[1545]: status update time (8.485 seconds)
Feb 06 00:02:42 proxmox1 pvestatd[1545]: status update time (10.772 seconds)
Feb 06 00:03:15 proxmox1 pvestatd[1545]: status update time (12.686 seconds)
Feb 06 00:03:40 proxmox1 pvestatd[1545]: status update time (5.004 seconds)
Feb 06 00:03:54 proxmox1 pvestatd[1545]: status update time (9.367 seconds)
Feb 06 00:04:18 proxmox1 pvestatd[1545]: got timeout
Feb 06 00:04:19 proxmox1 pvestatd[1545]: unable to activate storage 'newbacks' - directory '/mnt/pve/newbacks' does not exist or is unreachable
Feb 06 00:04:22 proxmox1 pvestatd[1545]: status update time (6.131 seconds)
Feb 06 00:04:54 proxmox1 pvestatd[1545]: status update time (29.250 seconds)
Feb 06 00:05:14 proxmox1 pvestatd[1545]: status update time (20.245 seconds)
Feb 06 00:05:17 proxmox1 pve-ha-crm[1590]: loop take too long (35 seconds)
Feb 06 00:05:22 proxmox1 pvestatd[1545]: status update time (7.508 seconds)
Feb 06 00:05:41 proxmox1 pvestatd[1545]: status update time (17.055 seconds)
Feb 06 00:05:47 proxmox1 watchdog-mux[688]: client watchdog expired - disable watchdog updates
Feb 06 00:05:49 proxmox1 watchdog-mux[688]: exit watchdog-mux with active connections
Feb 06 00:05:49 proxmox1 pve-ha-lrm[1622]: loop take too long (82 seconds)
Feb 06 00:05:50 proxmox1 pvestatd[1545]: status update time (9.364 seconds)
Feb 06 00:05:53 proxmox1 systemd[1]: watchdog-mux.service: Succeeded.
Feb 06 00:05:53 proxmox1 kernel: watchdog: watchdog0: watchdog did not stop!
-- Reboot --
 
Last edited:
Yes. I do have a separate cluster network on its own nic. I put some diagnostic output below - they seem to look good? My non-cluster traffic is on a different interface which is a bond of 2 separate nics. Is it possible that my system is trying to backup across my cluster network? Does that make sense? I wonder if my switch is failing..

root@proxmox1:~# pvecm status
Cluster information
-------------------
Name: SWVC
Config Version: 7
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Wed Feb 7 10:40:38 2024
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000001
Ring ID: 1.2b2
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.1.0.1 (local)
0x00000002 1 10.1.0.2
0x00000003 1 10.1.0.3

root@proxmox1:~# ping 10.1.0.2
PING 10.1.0.2 (10.1.0.2) 56(84) bytes of data.
64 bytes from 10.1.0.2: icmp_seq=1 ttl=64 time=0.221 ms
64 bytes from 10.1.0.2: icmp_seq=2 ttl=64 time=0.330 ms
64 bytes from 10.1.0.2: icmp_seq=3 ttl=64 time=0.202 ms
64 bytes from 10.1.0.2: icmp_seq=4 ttl=64 time=0.149 ms
64 bytes from 10.1.0.2: icmp_seq=5 ttl=64 time=0.180 ms
^C
--- 10.1.0.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4101ms
rtt min/avg/max/mdev = 0.149/0.216/0.330/0.061 ms
root@proxmox1:~# ping 10.1.0.3
PING 10.1.0.3 (10.1.0.3) 56(84) bytes of data.
64 bytes from 10.1.0.3: icmp_seq=1 ttl=64 time=0.260 ms
64 bytes from 10.1.0.3: icmp_seq=2 ttl=64 time=0.236 ms
64 bytes from 10.1.0.3: icmp_seq=3 ttl=64 time=0.280 ms
64 bytes from 10.1.0.3: icmp_seq=4 ttl=64 time=0.163 ms
64 bytes from 10.1.0.3: icmp_seq=5 ttl=64 time=0.165 ms
^C
--- 10.1.0.3 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4084ms
rtt min/avg/max/mdev = 0.163/0.220/0.280/0.048 ms
 
Yes. I do have a separate cluster network on its own nic. I put some diagnostic output below - they seem to look good? My non-cluster traffic is on a different interface which is a bond of 2 separate nics. Is it possible that my system is trying to backup across my cluster network? Does that make sense? I wonder if my switch is failing..

root@proxmox1:~# pvecm status
Cluster information
-------------------
Name: SWVC
Config Version: 7
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Wed Feb 7 10:40:38 2024
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000001
Ring ID: 1.2b2
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.1.0.1 (local)
0x00000002 1 10.1.0.2
0x00000003 1 10.1.0.3

root@proxmox1:~# ping 10.1.0.2
PING 10.1.0.2 (10.1.0.2) 56(84) bytes of data.
64 bytes from 10.1.0.2: icmp_seq=1 ttl=64 time=0.221 ms
64 bytes from 10.1.0.2: icmp_seq=2 ttl=64 time=0.330 ms
64 bytes from 10.1.0.2: icmp_seq=3 ttl=64 time=0.202 ms
64 bytes from 10.1.0.2: icmp_seq=4 ttl=64 time=0.149 ms
64 bytes from 10.1.0.2: icmp_seq=5 ttl=64 time=0.180 ms
^C
--- 10.1.0.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4101ms
rtt min/avg/max/mdev = 0.149/0.216/0.330/0.061 ms
root@proxmox1:~# ping 10.1.0.3
PING 10.1.0.3 (10.1.0.3) 56(84) bytes of data.
64 bytes from 10.1.0.3: icmp_seq=1 ttl=64 time=0.260 ms
64 bytes from 10.1.0.3: icmp_seq=2 ttl=64 time=0.236 ms
64 bytes from 10.1.0.3: icmp_seq=3 ttl=64 time=0.280 ms
64 bytes from 10.1.0.3: icmp_seq=4 ttl=64 time=0.163 ms
64 bytes from 10.1.0.3: icmp_seq=5 ttl=64 time=0.165 ms
^C
--- 10.1.0.3 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4084ms
rtt min/avg/max/mdev = 0.163/0.220/0.280/0.048 ms
Thanks for the outputs, they however only show that the cluster is currently healthy, that does not tell much about the time when the backups are ongoing. Please share your corosync config cat /etc/pve/corosync.conf as well as the network configuration for the hosts, cat /etc/network/interfaces. You could also monitor you network traffic during the backup runs, just to verify the traffic is routed as expected. Also, excluding the switch as point of failure would definitely be of interest. If possible, you might configure a failover network for corosync for maintenance on the other network without downtime, see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy
 
Hi. Thanks for your reply. The info is below. I have a feeling my config is ok but let me know if not. Yes I will observe closely during backup. Looking at my hosts file I think it should not be a problem. I don't have the cluster network addresses named at all. They are only referred to by ip and only in the corosync.conf
I think maybe it is my switch. I could use more redundancy.

root@proxmox1:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: proxmox1
nodeid: 1
quorum_votes: 1
ring0_addr: 10.1.0.1
}
node {
name: proxmox2
nodeid: 2
quorum_votes: 1
ring0_addr: 10.1.0.2
}
node {
name: proxmox3
nodeid: 3
quorum_votes: 1
ring0_addr: 10.1.0.3
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: SWVC
config_version: 7
interface {
bindnetaddr: 10.1.0.1
ringnumber: 0
}
ip_version: ipv4
secauth: on
version: 2
}
root@proxmox1:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

#onboard
auto eno1
iface eno1 inet static
address 10.1.0.1
netmask 255.255.255.0

#onboard
auto enp4s0
iface enp4s0 inet static
address 10.2.0.1
netmask 255.255.255.0

#mellanox
auto enp2s0
iface enp2s0 inet static
address 10.0.1.1
netmask 255.255.255.0

#pci card
iface enp1s0f0 inet manual

#pci card
iface enp1s0f1 inet manual

auto bond0
iface bond0 inet manual
slaves enp1s0f0 enp1s0f1
bond_miimon 100
bond_mode 802.3ad

auto vmbr0
iface vmbr0 inet static
address 192.168.1.21
netmask 255.255.255.0
gateway 192.168.1.1
bridge_ports bond0
bridge_stp off
bridge_fd 0

auto bond0.4088
iface bond0.4088 inet manual
vlan-raw-device bond0

auto vmbr4088
iface vmbr4088 inet manual
bridge_ports bond0.4088
bridge_stp off
bridge_fd 0

auto bond0.10
iface bond0.10 inet manual
vlan-raw-device bond0

auto vmbr10
iface vmbr10 inet manual
bridge_ports bond0.10
bridge_stp off
bridge_fd 0
 
Yes, this looks fine so far, so definitely worth to exclude the switch as bad actor here. Maybe you can find some relevant logs on the switch itself.
 
Happened again despite setting up a second, redundant cluster network. Also I replaced the single ethernet connection between switch and backup server (proxmox3) with a bonded connection.
Is it possible to tell anything from the log of the other 2 nodes? Although I see complaints about unreachable storage, it is always mounted whenever I check.
proxmox1 and proxmox2 nodes host containers and vms. proxmox3 functions as quorum and backup server.

New corosync.conf:

logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: proxmox1
nodeid: 1
quorum_votes: 1
ring0_addr: 10.1.0.1
ring1_addr: 10.2.0.1
}
node {
name: proxmox2
nodeid: 2
quorum_votes: 1
ring0_addr: 10.1.0.2
ring1_addr: 10.2.0.2
}
node {
name: proxmox3
nodeid: 3
quorum_votes: 1
ring0_addr: 10.1.0.3
ring1_addr: 10.2.0.3
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: SWVC
config_version: 8
ip_version: ipv4
secauth: off
version: 2
interface {
bindnetaddr: 10.1.0.0
ringnumber: 0
}
interface {
bindnetaddr: 10.2.0.0
ringnumber: 1
}
}

relevant logs:

proxmox 1 log ---> same as previously


Feb 12 00:00:03 proxmox2 pmxcfs[1514]: [status] notice: received log
Feb 12 00:00:44 proxmox2 systemd[1]: Starting Rotate log files...
Feb 12 00:00:44 proxmox2 systemd[1]: Starting Daily man-db regeneration...
Feb 12 00:00:44 proxmox2 systemd[1]: Reloading PVE API Proxy Server.
Feb 12 00:00:46 proxmox2 systemd[1]: man-db.service: Succeeded.
Feb 12 00:00:46 proxmox2 systemd[1]: Finished Daily man-db regeneration.
Feb 12 00:00:49 proxmox2 audit[3456233]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-902_</var/lib/lxc>" name="/run/systemd/unit-root/proc/" pid=3456233 comm="(ogrotate)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Feb 12 00:00:49 proxmox2 kernel: audit: type=1400 audit(1707724849.622:243): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-902_</var/lib/lxc>" name="/run/systemd/unit-root/proc/" pid=3456233 comm="(ogrotate)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
Feb 12 00:00:50 proxmox2 pveproxy[3456223]: send HUP to 1667
Feb 12 00:00:50 proxmox2 pveproxy[1667]: received signal HUP
Feb 12 00:00:50 proxmox2 systemd[1]: Reloaded PVE API Proxy Server.
Feb 12 00:00:50 proxmox2 pveproxy[1667]: server closing
Feb 12 00:00:50 proxmox2 systemd[1]: Reloading PVE SPICE Proxy Server.
Feb 12 00:00:50 proxmox2 pveproxy[1667]: server shutdown (restart)
Feb 12 00:00:51 proxmox2 spiceproxy[3456258]: send HUP to 1674
Feb 12 00:00:51 proxmox2 systemd[1]: Reloaded PVE SPICE Proxy Server.
Feb 12 00:00:51 proxmox2 systemd[1]: Stopping Proxmox VE firewall logger...
Feb 12 00:00:51 proxmox2 pvefw-logger[2869334]: received terminate request (signal)
Feb 12 00:00:51 proxmox2 pvefw-logger[2869334]: stopping pvefw logger
Feb 12 00:00:51 proxmox2 spiceproxy[1674]: received signal HUP
Feb 12 00:00:51 proxmox2 spiceproxy[1674]: server closing
Feb 12 00:00:51 proxmox2 systemd[1]: pvefw-logger.service: Succeeded.
Feb 12 00:00:51 proxmox2 systemd[1]: Stopped Proxmox VE firewall logger.
Feb 12 00:00:51 proxmox2 systemd[1]: pvefw-logger.service: Consumed 6.826s CPU time.
Feb 12 00:00:51 proxmox2 systemd[1]: Starting Proxmox VE firewall logger...
Feb 12 00:00:51 proxmox2 spiceproxy[1674]: server shutdown (restart)
Feb 12 00:00:51 proxmox2 pvefw-logger[3456288]: starting pvefw logger
Feb 12 00:00:51 proxmox2 systemd[1]: Started Proxmox VE firewall logger.
Feb 12 00:00:51 proxmox2 systemd[1]: logrotate.service: Succeeded.
Feb 12 00:00:51 proxmox2 systemd[1]: Finished Rotate log files.
Feb 12 00:00:51 proxmox2 pveproxy[1667]: restarting server
Feb 12 00:00:52 proxmox2 pveproxy[1667]: starting 3 worker(s)
Feb 12 00:00:52 proxmox2 pveproxy[1667]: worker 3456306 started
Feb 12 00:00:52 proxmox2 pveproxy[1667]: worker 3456308 started
Feb 12 00:00:52 proxmox2 pveproxy[1667]: worker 3456310 started
Feb 12 00:00:52 proxmox2 spiceproxy[1674]: restarting server
Feb 12 00:00:52 proxmox2 spiceproxy[1674]: starting 1 worker(s)
Feb 12 00:00:52 proxmox2 spiceproxy[1674]: worker 3456317 started
Feb 12 00:00:56 proxmox2 pveproxy[2285387]: worker exit
Feb 12 00:00:56 proxmox2 pveproxy[2285386]: worker exit
Feb 12 00:00:57 proxmox2 pveproxy[2285384]: worker exit
Feb 12 00:00:57 proxmox2 spiceproxy[2285383]: worker exit
Feb 12 00:01:01 proxmox2 pveproxy[1667]: worker 2285384 finished
Feb 12 00:01:01 proxmox2 pveproxy[1667]: worker 2285386 finished
Feb 12 00:01:01 proxmox2 pveproxy[1667]: worker 2285387 finished
Feb 12 00:01:02 proxmox2 spiceproxy[1674]: worker 2285383 finished
Feb 12 00:01:43 proxmox2 pvestatd[1610]: got timeout
Feb 12 00:02:43 proxmox2 pvestatd[1610]: got timeout
Feb 12 00:02:45 proxmox2 pvestatd[1610]: got timeout
Feb 12 00:02:45 proxmox2 pvestatd[1610]: unable to activate storage 'backups' - directory '/mnt/pve/backups' does not exist or is unreachable
Feb 12 00:02:54 proxmox2 pvestatd[1610]: got timeout
Feb 12 00:02:54 proxmox2 pvestatd[1610]: unable to activate storage 'backups' - directory '/mnt/pve/backups' does not exist or is unreachable
Feb 12 00:02:56 proxmox2 pvestatd[1610]: got timeout
Feb 12 00:03:03 proxmox2 pvestatd[1610]: got timeout
Feb 12 00:03:03 proxmox2 pvestatd[1610]: unable to activate storage 'backups' - directory '/mnt/pve/backups' does not exist or is unreachable
Feb 12 00:03:13 proxmox2 pvestatd[1610]: got timeout
Feb 12 00:03:13 proxmox2 pvestatd[1610]: unable to activate storage 'backups' - directory '/mnt/pve/backups' does not exist or is unreachable
Feb 12 00:03:15 proxmox2 pvestatd[1610]: got timeout
Feb 12 00:03:15 proxmox2 pvestatd[1610]: unable to activate storage 'newbacks' - directory '/mnt/pve/newbacks' does not exist or is unreachable
Feb 12 00:06:13 proxmox2 kernel: mlx4_en: enp2s0: Link Down
Feb 12 00:06:15 proxmox2 corosync[1559]: [KNET ] link: host: 1 link: 0 is down
Feb 12 00:06:15 proxmox2 corosync[1559]: [KNET ] link: host: 1 link: 1 is down
Feb 12 00:06:15 proxmox2 corosync[1559]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Feb 12 00:06:15 proxmox2 corosync[1559]: [KNET ] host: host: 1 has no active links
Feb 12 00:06:15 proxmox2 corosync[1559]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Feb 12 00:06:15 proxmox2 corosync[1559]: [KNET ] host: host: 1 has no active links
Feb 12 00:06:16 proxmox2 corosync[1559]: [TOTEM ] Token has not been received in 2737 ms
Feb 12 00:06:17 proxmox2 corosync[1559]: [TOTEM ] A processor failed, forming new configuration: token timed out (3650ms), waiting 4380ms for consensus.
Feb 12 00:06:21 proxmox2 corosync[1559]: [QUORUM] Sync members[2]: 2 3
Feb 12 00:06:21 proxmox2 corosync[1559]: [QUORUM] Sync left[1]: 1
Feb 12 00:06:21 proxmox2 corosync[1559]: [TOTEM ] A new membership (2.2ee) was formed. Members left: 1
Feb 12 00:06:21 proxmox2 corosync[1559]: [TOTEM ] Failed to receive the leave message. failed: 1



Feb 12 00:00:03 proxmox3 pmxcfs[1675]: [status] notice: received log
Feb 12 00:00:10 proxmox3 systemd[1]: Starting Rotate log files...
Feb 12 00:00:10 proxmox3 systemd[1]: Starting Daily man-db regeneration...
Feb 12 00:00:11 proxmox3 systemd[1]: Reloading PVE API Proxy Server.
Feb 12 00:00:16 proxmox3 systemd[1]: man-db.service: Succeeded.
Feb 12 00:00:16 proxmox3 systemd[1]: Finished Daily man-db regeneration.
Feb 12 00:00:18 proxmox3 pveproxy[723459]: send HUP to 1770
Feb 12 00:00:18 proxmox3 pveproxy[1770]: received signal HUP
Feb 12 00:00:18 proxmox3 pveproxy[1770]: server closing
Feb 12 00:00:18 proxmox3 pveproxy[1770]: server shutdown (restart)
Feb 12 00:00:18 proxmox3 systemd[1]: Reloaded PVE API Proxy Server.
Feb 12 00:00:18 proxmox3 systemd[1]: Reloading PVE SPICE Proxy Server.
Feb 12 00:00:19 proxmox3 spiceproxy[723531]: send HUP to 1777
Feb 12 00:00:19 proxmox3 spiceproxy[1777]: received signal HUP
Feb 12 00:00:19 proxmox3 spiceproxy[1777]: server closing
Feb 12 00:00:19 proxmox3 spiceproxy[1777]: server shutdown (restart)
Feb 12 00:00:19 proxmox3 systemd[1]: Reloaded PVE SPICE Proxy Server.
Feb 12 00:00:19 proxmox3 systemd[1]: Stopping Proxmox VE firewall logger...
Feb 12 00:00:19 proxmox3 pvefw-logger[338662]: received terminate request (signal)
Feb 12 00:00:19 proxmox3 pvefw-logger[338662]: stopping pvefw logger
Feb 12 00:00:19 proxmox3 systemd[1]: pvefw-logger.service: Succeeded.
Feb 12 00:00:19 proxmox3 systemd[1]: Stopped Proxmox VE firewall logger.
Feb 12 00:00:19 proxmox3 systemd[1]: pvefw-logger.service: Consumed 7.119s CPU time.
Feb 12 00:00:19 proxmox3 systemd[1]: Starting Proxmox VE firewall logger...
Feb 12 00:00:19 proxmox3 systemd[1]: Started Proxmox VE firewall logger.
Feb 12 00:00:19 proxmox3 pvefw-logger[723559]: starting pvefw logger
Feb 12 00:00:20 proxmox3 systemd[1]: logrotate.service: Succeeded.
Feb 12 00:00:20 proxmox3 systemd[1]: Finished Rotate log files.
Feb 12 00:00:20 proxmox3 spiceproxy[1777]: restarting server
Feb 12 00:00:20 proxmox3 spiceproxy[1777]: starting 1 worker(s)
Feb 12 00:00:20 proxmox3 spiceproxy[1777]: worker 723568 started
Feb 12 00:00:20 proxmox3 pveproxy[1770]: restarting server
Feb 12 00:00:20 proxmox3 pveproxy[1770]: starting 3 worker(s)
Feb 12 00:00:20 proxmox3 pveproxy[1770]: worker 723570 started
Feb 12 00:00:20 proxmox3 pveproxy[1770]: worker 723571 started
Feb 12 00:00:20 proxmox3 pveproxy[1770]: worker 723572 started
Feb 12 00:00:25 proxmox3 spiceproxy[338671]: worker exit
Feb 12 00:00:25 proxmox3 spiceproxy[1777]: worker 338671 finished
Feb 12 00:00:25 proxmox3 pveproxy[338682]: worker exit
Feb 12 00:00:25 proxmox3 pveproxy[338684]: worker exit
Feb 12 00:00:25 proxmox3 pveproxy[338683]: worker exit
Feb 12 00:00:25 proxmox3 pveproxy[1770]: worker 338682 finished
Feb 12 00:00:25 proxmox3 pveproxy[1770]: worker 338683 finished
Feb 12 00:00:25 proxmox3 pveproxy[1770]: worker 338684 finished
Feb 12 00:01:01 proxmox3 pvestatd[1712]: got timeout
Feb 12 00:02:42 proxmox3 pvestatd[1712]: got timeout
Feb 12 00:02:44 proxmox3 pvestatd[1712]: got timeout
Feb 12 00:03:01 proxmox3 pvestatd[1712]: got timeout
Feb 12 00:03:01 proxmox3 pvestatd[1712]: unable to activate storage 'backups' - directory '/mnt/pve/backups' does not exist or is unreachable
Feb 12 00:03:03 proxmox3 pvestatd[1712]: got timeout
Feb 12 00:06:15 proxmox3 corosync[29596]: [KNET ] link: host: 1 link: 0 is down
Feb 12 00:06:15 proxmox3 corosync[29596]: [KNET ] link: host: 1 link: 1 is down
Feb 12 00:06:15 proxmox3 corosync[29596]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Feb 12 00:06:15 proxmox3 corosync[29596]: [KNET ] host: host: 1 has no active links
Feb 12 00:06:15 proxmox3 corosync[29596]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Feb 12 00:06:15 proxmox3 corosync[29596]: [KNET ] host: host: 1 has no active links
Feb 12 00:06:16 proxmox3 corosync[29596]: [TOTEM ] Token has not been received in 2737 ms
Feb 12 00:06:17 proxmox3 corosync[29596]: [TOTEM ] A processor failed, forming new configuration: token timed out (3650ms), waiting 4380ms for consensus.
Feb 12 00:06:21 proxmox3 corosync[29596]: [QUORUM] Sync members[2]: 2 3
Feb 12 00:06:21 proxmox3 corosync[29596]: [QUORUM] Sync left[1]: 1
Feb 12 00:06:21 proxmox3 corosync[29596]: [TOTEM ] A new membership (2.2ee) was formed. Members left: 1
Feb 12 00:06:21 proxmox3 corosync[29596]: [TOTEM ] Failed to receive the leave message. failed: 1
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [dcdb] notice: members: 2/1514, 3/1675
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [dcdb] notice: starting data syncronisation
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [status] notice: members: 2/1514, 3/1675
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [status] notice: starting data syncronisation
Feb 12 00:06:21 proxmox3 corosync[29596]: [QUORUM] Members[2]: 2 3
Feb 12 00:06:21 proxmox3 corosync[29596]: [MAIN ] Completed service synchronization, ready to provide service.
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [dcdb] notice: received sync request (epoch 2/1514/0000001E)
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [status] notice: received sync request (epoch 2/1514/0000001C)
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [dcdb] notice: received all states
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [dcdb] notice: leader is 2/1514
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [dcdb] notice: synced members: 2/1514, 3/1675
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [dcdb] notice: all data is up to date
Feb 12 00:06:21 proxmox3 pmxcfs[1675]: [dcdb] notice: dfsm_deliver_queue: queue length 4
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!