Dear Members,
I have a random crash of one of node in cluster. This node is pve3
I have message in syslog, "Reached target Shutdown."
What causes it?
I read in forum, it can causes nfs storage under VM, I don't have nfs under vm.
After restart mon is down on this node.
What can be wrong?
--------------------------------------------------
root@pve3:~# ceph -s
cluster:
id: 58b7c533-09d5-4c82-8aa8-9ee4a5af696d
health: HEALTH_WARN
1/4 mons down, quorum pve1,pve2,pve4
1 slow ops, oldest one blocked for 8978 sec, mon.pve3 has slow ops
services:
mon: 4 daemons, quorum pve1,pve2,pve4 (age 5h), out of quorum: pve3
mgr: pve1(active, since 36h), standbys: pve2, pve4, pve3
osd: 26 osds: 26 up (since 5h), 26 in (since 29h)
data:
pools: 3 pools, 768 pgs
objects: 2.00M objects, 7.6 TiB
usage: 22 TiB used, 41 TiB / 63 TiB avail
pgs: 767 active+clean
1 active+clean+scrubbing+deep
io:
client: 546 KiB/s rd, 11 MiB/s wr, 81 op/s rd, 739 op/s wr
-------------------------------------------------------
root@pve3:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-1-pve)
pve-manager: 7.1-5 (running version: 7.1-5/6fe299a0)
pve-kernel-5.13: 7.1-4
pve-kernel-helper: 7.1-4
pve-kernel-5.11: 7.0-10
pve-kernel-5.13.19-1-pve: 5.13.19-2
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph: 16.2.6-pve2
ceph-fuse: 16.2.6-pve2
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.14-1
proxmox-backup-file-restore: 2.0.14-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.4-2
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-1
pve-qemu-kvm: 6.1.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-3
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3
--------------------------------------------------------
journalctl:
Nov 26 03:11:51 pve3 sudo[3791372]: pam_unix(sudo:session): session closed for user root
Nov 26 03:11:51 pve3 sshd[3791351]: Received disconnect from 172.26.73.37 port 35836:11: disconnected by user
Nov 26 03:11:51 pve3 sshd[3791351]: Disconnected from user nagios 172.26.73.37 port 35836
Nov 26 03:11:51 pve3 sshd[3791344]: pam_unix(sshd:session): session closed for user nagios
Nov 26 03:11:51 pve3 systemd[1]: session-2789.scope: Succeeded.
Nov 26 03:11:51 pve3 systemd-logind[1884]: Session 2789 logged out. Waiting for processes to exit.
Nov 26 03:11:51 pve3 systemd-logind[1884]: Removed session 2789.
Nov 26 03:11:51 pve3 sudo[3791374]: pam_unix(sudo:session): session closed for user root
Nov 26 03:11:51 pve3 sshd[3791359]: Received disconnect from 172.26.73.37 port 35838:11: disconnected by user
Nov 26 03:11:51 pve3 sshd[3791359]: Disconnected from user nagios 172.26.73.37 port 35838
Nov 26 03:11:51 pve3 sshd[3791346]: pam_unix(sshd:session): session closed for user nagios
Nov 26 03:11:51 pve3 systemd[1]: session-2790.scope: Succeeded.
Nov 26 03:11:51 pve3 systemd-logind[1884]: Session 2790 logged out. Waiting for processes to exit.
Nov 26 03:11:51 pve3 systemd-logind[1884]: Removed session 2790.
-- Boot 47a4c673bf8441c4beee3f275c0e5c94 --
Nov 26 03:17:22 pve3 kernel: Linux version 5.13.19-1-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld>
Nov 26 03:17:22 pve3 kernel: Command line: BOOT_IMAGE=/vmlinuz-5.13.19-1-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpoo>
Nov 26 03:17:22 pve3 kernel: KERNEL supported cpus:
Nov 26 03:17:22 pve3 kernel: Intel GenuineIntel
Nov 26 03:17:22 pve3 kernel: AMD AuthenticAMD
--------------------------------------------
syslog:
Nov 26 03:12:01 pve3 systemd[1]: Stopping User Manager for UID 113...
Nov 26 03:12:01 pve3 systemd[3789311]: Stopped target Main User Target.
Nov 26 03:12:01 pve3 systemd[3789311]: Stopped target Basic System.
Nov 26 03:12:01 pve3 systemd[3789311]: Stopped target Paths.
Nov 26 03:12:01 pve3 systemd[3789311]: Stopped target Sockets.
Nov 26 03:12:01 pve3 systemd[3789311]: Stopped target Timers.
Nov 26 03:12:01 pve3 systemd[3789311]: dirmngr.socket: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Closed GnuPG network certificate management daemon.
Nov 26 03:12:01 pve3 systemd[3789311]: gpg-agent-browser.socket: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Nov 26 03:12:01 pve3 systemd[3789311]: gpg-agent-extra.socket: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Nov 26 03:12:01 pve3 systemd[3789311]: gpg-agent-ssh.socket: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Nov 26 03:12:01 pve3 systemd[3789311]: gpg-agent.socket: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Closed GnuPG cryptographic agent and passphrase cache.
Nov 26 03:12:01 pve3 systemd[3789311]: Removed slice User Application Slice.
Nov 26 03:12:01 pve3 systemd[3789311]: Reached target Shutdown.
Nov 26 03:12:01 pve3 systemd[3789311]: systemd-exit.service: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Finished Exit the Session.
Nov 26 03:12:01 pve3 systemd[3789311]: Reached target Exit the Session.
Nov 26 03:12:01 pve3 systemd[1]: user@113.service: Succeeded.
Nov 26 03:12:01 pve3 systemd[1]: Stopped User Manager for UID 113.
Nov 26 03:12:01 pve3 systemd[1]: Stopping User Runtime Directory /run/user/113...
Nov 26 03:12:01 pve3 systemd[1]: run-user-113.mount: Succeeded.
Nov 26 03:12:01 pve3 systemd[1]: user-runtime-dir@113.service: Succeeded.
Nov 26 03:12:01 pve3 systemd[1]: Stopped User Runtime Directory /run/user/113.
Nov 26 03:12:01 pve3 systemd[1]: Removed slice User Slice of UID 113.
Nov 26 03:12:01 pve3 systemd[1]: user-113.slice: Consumed 1.351s CPU time.
Nov 26 03:12:02 pve3 pmxcfs[2624]: [status] notice: received log
Nov 26 03:17:23 pve3 systemd-modules-load[1560]: Inserted module 'iscsi_tcp'
Nov 26 03:17:23 pve3 systemd-modules-load[1560]: Inserted module 'ib_iser'
Nov 26 03:17:23 pve3 systemd-modules-load[1560]: Inserted module 'vhost_net'
Nov 26 03:17:23 pve3 systemd-udevd[1620]: Using default interface naming scheme 'v247'.
Nov 26 03:17:23 pve3 systemd-udevd[1620]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
-------------------------------
I attached full syslog file to post.
Thank you,
Gabor
I have a random crash of one of node in cluster. This node is pve3
I have message in syslog, "Reached target Shutdown."
What causes it?
I read in forum, it can causes nfs storage under VM, I don't have nfs under vm.
After restart mon is down on this node.
What can be wrong?
--------------------------------------------------
root@pve3:~# ceph -s
cluster:
id: 58b7c533-09d5-4c82-8aa8-9ee4a5af696d
health: HEALTH_WARN
1/4 mons down, quorum pve1,pve2,pve4
1 slow ops, oldest one blocked for 8978 sec, mon.pve3 has slow ops
services:
mon: 4 daemons, quorum pve1,pve2,pve4 (age 5h), out of quorum: pve3
mgr: pve1(active, since 36h), standbys: pve2, pve4, pve3
osd: 26 osds: 26 up (since 5h), 26 in (since 29h)
data:
pools: 3 pools, 768 pgs
objects: 2.00M objects, 7.6 TiB
usage: 22 TiB used, 41 TiB / 63 TiB avail
pgs: 767 active+clean
1 active+clean+scrubbing+deep
io:
client: 546 KiB/s rd, 11 MiB/s wr, 81 op/s rd, 739 op/s wr
-------------------------------------------------------
root@pve3:~# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-1-pve)
pve-manager: 7.1-5 (running version: 7.1-5/6fe299a0)
pve-kernel-5.13: 7.1-4
pve-kernel-helper: 7.1-4
pve-kernel-5.11: 7.0-10
pve-kernel-5.13.19-1-pve: 5.13.19-2
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph: 16.2.6-pve2
ceph-fuse: 16.2.6-pve2
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.14-1
proxmox-backup-file-restore: 2.0.14-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.4-2
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-1
pve-qemu-kvm: 6.1.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-3
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3
--------------------------------------------------------
journalctl:
Nov 26 03:11:51 pve3 sudo[3791372]: pam_unix(sudo:session): session closed for user root
Nov 26 03:11:51 pve3 sshd[3791351]: Received disconnect from 172.26.73.37 port 35836:11: disconnected by user
Nov 26 03:11:51 pve3 sshd[3791351]: Disconnected from user nagios 172.26.73.37 port 35836
Nov 26 03:11:51 pve3 sshd[3791344]: pam_unix(sshd:session): session closed for user nagios
Nov 26 03:11:51 pve3 systemd[1]: session-2789.scope: Succeeded.
Nov 26 03:11:51 pve3 systemd-logind[1884]: Session 2789 logged out. Waiting for processes to exit.
Nov 26 03:11:51 pve3 systemd-logind[1884]: Removed session 2789.
Nov 26 03:11:51 pve3 sudo[3791374]: pam_unix(sudo:session): session closed for user root
Nov 26 03:11:51 pve3 sshd[3791359]: Received disconnect from 172.26.73.37 port 35838:11: disconnected by user
Nov 26 03:11:51 pve3 sshd[3791359]: Disconnected from user nagios 172.26.73.37 port 35838
Nov 26 03:11:51 pve3 sshd[3791346]: pam_unix(sshd:session): session closed for user nagios
Nov 26 03:11:51 pve3 systemd[1]: session-2790.scope: Succeeded.
Nov 26 03:11:51 pve3 systemd-logind[1884]: Session 2790 logged out. Waiting for processes to exit.
Nov 26 03:11:51 pve3 systemd-logind[1884]: Removed session 2790.
-- Boot 47a4c673bf8441c4beee3f275c0e5c94 --
Nov 26 03:17:22 pve3 kernel: Linux version 5.13.19-1-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld>
Nov 26 03:17:22 pve3 kernel: Command line: BOOT_IMAGE=/vmlinuz-5.13.19-1-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpoo>
Nov 26 03:17:22 pve3 kernel: KERNEL supported cpus:
Nov 26 03:17:22 pve3 kernel: Intel GenuineIntel
Nov 26 03:17:22 pve3 kernel: AMD AuthenticAMD
--------------------------------------------
syslog:
Nov 26 03:12:01 pve3 systemd[1]: Stopping User Manager for UID 113...
Nov 26 03:12:01 pve3 systemd[3789311]: Stopped target Main User Target.
Nov 26 03:12:01 pve3 systemd[3789311]: Stopped target Basic System.
Nov 26 03:12:01 pve3 systemd[3789311]: Stopped target Paths.
Nov 26 03:12:01 pve3 systemd[3789311]: Stopped target Sockets.
Nov 26 03:12:01 pve3 systemd[3789311]: Stopped target Timers.
Nov 26 03:12:01 pve3 systemd[3789311]: dirmngr.socket: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Closed GnuPG network certificate management daemon.
Nov 26 03:12:01 pve3 systemd[3789311]: gpg-agent-browser.socket: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Nov 26 03:12:01 pve3 systemd[3789311]: gpg-agent-extra.socket: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Nov 26 03:12:01 pve3 systemd[3789311]: gpg-agent-ssh.socket: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Nov 26 03:12:01 pve3 systemd[3789311]: gpg-agent.socket: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Closed GnuPG cryptographic agent and passphrase cache.
Nov 26 03:12:01 pve3 systemd[3789311]: Removed slice User Application Slice.
Nov 26 03:12:01 pve3 systemd[3789311]: Reached target Shutdown.
Nov 26 03:12:01 pve3 systemd[3789311]: systemd-exit.service: Succeeded.
Nov 26 03:12:01 pve3 systemd[3789311]: Finished Exit the Session.
Nov 26 03:12:01 pve3 systemd[3789311]: Reached target Exit the Session.
Nov 26 03:12:01 pve3 systemd[1]: user@113.service: Succeeded.
Nov 26 03:12:01 pve3 systemd[1]: Stopped User Manager for UID 113.
Nov 26 03:12:01 pve3 systemd[1]: Stopping User Runtime Directory /run/user/113...
Nov 26 03:12:01 pve3 systemd[1]: run-user-113.mount: Succeeded.
Nov 26 03:12:01 pve3 systemd[1]: user-runtime-dir@113.service: Succeeded.
Nov 26 03:12:01 pve3 systemd[1]: Stopped User Runtime Directory /run/user/113.
Nov 26 03:12:01 pve3 systemd[1]: Removed slice User Slice of UID 113.
Nov 26 03:12:01 pve3 systemd[1]: user-113.slice: Consumed 1.351s CPU time.
Nov 26 03:12:02 pve3 pmxcfs[2624]: [status] notice: received log
Nov 26 03:17:23 pve3 systemd-modules-load[1560]: Inserted module 'iscsi_tcp'
Nov 26 03:17:23 pve3 systemd-modules-load[1560]: Inserted module 'ib_iser'
Nov 26 03:17:23 pve3 systemd-modules-load[1560]: Inserted module 'vhost_net'
Nov 26 03:17:23 pve3 systemd-udevd[1620]: Using default interface naming scheme 'v247'.
Nov 26 03:17:23 pve3 systemd-udevd[1620]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
-------------------------------
I attached full syslog file to post.
Thank you,
Gabor