All the nodes and VMs are suddenly showing unknown

[jegan]

Hello,

Server Details,

root@proxmox1:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"

root@proxmox1:~# pveversion -v
proxmox-ve: 6.2-2 (running kernel: 5.4.34-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.8.19-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-12
pve-cluster: 6.1-8
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-2
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-14
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve1


In proxmox all the node and VMs are suddenly showing unknown, Once I restart the " service pvestatd restart " it's working fine.

Again After few minute it's going unknow.

​

I guess there is problem with pvestat daemon,

 

[spirit]

it could block if you have an hanging storage.

can you send your /etc/pve/storage.cfg ?

 

[jegan]

hello,

Thanks for the reply,


root@proxmox1:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content vztmpl,iso,backup

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

dir: data
path /data
content iso,images
shared 1

dir: proxdata
path /proxdata
content iso,images
shared 0

zfspool: vmdata
pool zfsstorage
content rootdir,images
mountpoint /zfsstorage
sparse 0

pbs: backupserver
datastore backupserver
server 101.0.0.21
content backup
fingerprint ab:4:21:84:e3:ee
username root@pam

 

[jegan]

Hello,

Thanks,

I find the issue, backup server is not connecting properly so the it's not working. Once I remove the backup server and restart service.

service corosync restart
service pve-cluster restart
service pveproxy restart
service pvedaemon restart
service pvestatd restart

Now it's working.

 

[spirit]

Hello,

Thanks,

I find the issue, backup server is not connecting properly so the it's not working. Once I remove the backup server and restart service.

service corosync restart
service pve-cluster restart
service pveproxy restart
service pvedaemon restart
service pvestatd restart

Now it's working.

Hi, I think a fix for this has been applied recently, but not yet released
(ping timeout checkout, to avoid pvestatd hang)
https://git.proxmox.com/?p=pve-storage.git;a=commit;h=3de423680a5e9e8f8955ba11c1b52ec5a56f9d3b

 

[jegan]

hello,

ok Let me try.

 

[jegan]

hello,

I have modified the line as per patch and then restart all the service.

/usr/share/perl5/PVE/Storage/PBSPlugin.pm

But still I am getting same error.

sub activate_storage {
my ($class, $storeid, $scfg, $cache) = @_;

#run_client_cmd($scfg, $storeid, "status");
# a 'status' client command is to expensive here
# TODO: use a dummy ping API call to ensure the PBS API daemon is available for real
my $server = $scfg->{server};
my $port = $scfg->{port} // 8007;
PVE::Network::tcp_ping($server, $port, 2);

return 1;
}


Error from syslog

Oct 16 21:08:44 proxmox1 vzdump[5847]: VM 111 qmp command failed - VM 111 qmp command 'backup' failed - Node 'drive-scsi0' is busy: block device is in use by block job: mirror
Oct 16 21:08:44 proxmox1 vzdump[5847]: ERROR: Backup of VM 111 failed - VM 111 qmp command 'backup' failed - Node 'drive-scsi0' is busy: block device is in use by block job: mirror
Oct 16 21:08:44 proxmox1 vzdump[5847]: INFO: Starting Backup of VM 112 (qemu)
Oct 16 21:08:48 proxmox1 vzdump[5847]: INFO: Finished Backup of VM 112 (00:00:04)
Oct 16 21:08:48 proxmox1 vzdump[5847]: INFO: Starting Backup of VM 113 (qemu)
Oct 16 21:09:15 proxmox1 vzdump[5847]: INFO: Finished Backup of VM 113 (00:00:27)
Oct 16 21:09:15 proxmox1 vzdump[5847]: INFO: Backup job finished with errors
Oct 16 21:09:15 proxmox1 vzdump[5847]: job errors
Oct 16 21:09:15 proxmox1 vzdump[5718]: <root@pam> end task UPIDroxmox1:000016D7:0B69B52D:5F89BC7A:vzdump::root@pam: job errors

 

[velocity08]

hello,

I have modified the line as per patch and then restart all the service.

/usr/share/perl5/PVE/Storage/PBSPlugin.pm

But still I am getting same error.

sub activate_storage {
my ($class, $storeid, $scfg, $cache) = @_;

#run_client_cmd($scfg, $storeid, "status");
# a 'status' client command is to expensive here
# TODO: use a dummy ping API call to ensure the PBS API daemon is available for real
my $server = $scfg->{server};
my $port = $scfg->{port} // 8007;
PVE::Network::tcp_ping($server, $port, 2);

return 1;
}


Error from syslog

Oct 16 21:08:44 proxmox1 vzdump[5847]: VM 111 qmp command failed - VM 111 qmp command 'backup' failed - Node 'drive-scsi0' is busy: block device is in use by block job: mirror
Oct 16 21:08:44 proxmox1 vzdump[5847]: ERROR: Backup of VM 111 failed - VM 111 qmp command 'backup' failed - Node 'drive-scsi0' is busy: block device is in use by block job: mirror
Oct 16 21:08:44 proxmox1 vzdump[5847]: INFO: Starting Backup of VM 112 (qemu)
Oct 16 21:08:48 proxmox1 vzdump[5847]: INFO: Finished Backup of VM 112 (00:00:04)
Oct 16 21:08:48 proxmox1 vzdump[5847]: INFO: Starting Backup of VM 113 (qemu)
Oct 16 21:09:15 proxmox1 vzdump[5847]: INFO: Finished Backup of VM 113 (00:00:27)
Oct 16 21:09:15 proxmox1 vzdump[5847]: INFO: Backup job finished with errors
Oct 16 21:09:15 proxmox1 vzdump[5847]: job errors
Oct 16 21:09:15 proxmox1 vzdump[5718]: <root@pam> end task UPIDroxmox1:000016D7:0B69B52D:5F89BC7A:vzdump::root@pam: job errors

Hi @jegan

did this ever get resolved?

seeing the same error in our environment on 1 specific VM.

ERROR: Backup of VM 114 failed - Node 'drive-virtio0' is busy: block device is in use by block job: mirror

""Cheers
G