All the nodes and VMs are suddenly showing unknown

jegan

New Member
Jul 27, 2020
20
0
1
30
Hello,

Server Details,

root@proxmox1:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"

root@proxmox1:~# pveversion -v
proxmox-ve: 6.2-2 (running kernel: 5.4.34-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-6
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.8.19-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-12
pve-cluster: 6.1-8
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-2
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-14
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve1


In proxmox all the node and VMs are suddenly showing unknown, Once I restart the " service pvestatd restart " it's working fine.

Again After few minute it's going unknow.

proxmox_bug.jpg


I guess there is problem with pvestat daemon,
 
Last edited:
hello,

Thanks for the reply,


root@proxmox1:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content vztmpl,iso,backup

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

dir: data
path /data
content iso,images
shared 1

dir: proxdata
path /proxdata
content iso,images
shared 0

zfspool: vmdata
pool zfsstorage
content rootdir,images
mountpoint /zfsstorage
sparse 0

pbs: backupserver
datastore backupserver
server 101.0.0.21
content backup
fingerprint ab:4:21:84:e3:ee
username root@pam
 
Hello,

Thanks,

I find the issue, backup server is not connecting properly so the it's not working. Once I remove the backup server and restart service.

service corosync restart
service pve-cluster restart
service pveproxy restart
service pvedaemon restart
service pvestatd restart

Now it's working.
 
Hello,

Thanks,

I find the issue, backup server is not connecting properly so the it's not working. Once I remove the backup server and restart service.

service corosync restart
service pve-cluster restart
service pveproxy restart
service pvedaemon restart
service pvestatd restart

Now it's working.
Hi, I think a fix for this has been applied recently, but not yet released
(ping timeout checkout, to avoid pvestatd hang)
https://git.proxmox.com/?p=pve-storage.git;a=commit;h=3de423680a5e9e8f8955ba11c1b52ec5a56f9d3b
 
hello,

I have modified the line as per patch and then restart all the service.

/usr/share/perl5/PVE/Storage/PBSPlugin.pm

But still I am getting same error.

sub activate_storage {
my ($class, $storeid, $scfg, $cache) = @_;

#run_client_cmd($scfg, $storeid, "status");
# a 'status' client command is to expensive here
# TODO: use a dummy ping API call to ensure the PBS API daemon is available for real
my $server = $scfg->{server};
my $port = $scfg->{port} // 8007;
PVE::Network::tcp_ping($server, $port, 2);

return 1;
}


Error from syslog

Oct 16 21:08:44 proxmox1 vzdump[5847]: VM 111 qmp command failed - VM 111 qmp command 'backup' failed - Node 'drive-scsi0' is busy: block device is in use by block job: mirror
Oct 16 21:08:44 proxmox1 vzdump[5847]: ERROR: Backup of VM 111 failed - VM 111 qmp command 'backup' failed - Node 'drive-scsi0' is busy: block device is in use by block job: mirror
Oct 16 21:08:44 proxmox1 vzdump[5847]: INFO: Starting Backup of VM 112 (qemu)
Oct 16 21:08:48 proxmox1 vzdump[5847]: INFO: Finished Backup of VM 112 (00:00:04)
Oct 16 21:08:48 proxmox1 vzdump[5847]: INFO: Starting Backup of VM 113 (qemu)
Oct 16 21:09:15 proxmox1 vzdump[5847]: INFO: Finished Backup of VM 113 (00:00:27)
Oct 16 21:09:15 proxmox1 vzdump[5847]: INFO: Backup job finished with errors
Oct 16 21:09:15 proxmox1 vzdump[5847]: job errors
Oct 16 21:09:15 proxmox1 vzdump[5718]: <root@pam> end task UPID:proxmox1:000016D7:0B69B52D:5F89BC7A:vzdump::root@pam: job errors
 
Last edited:
hello,

I have modified the line as per patch and then restart all the service.

/usr/share/perl5/PVE/Storage/PBSPlugin.pm

But still I am getting same error.

sub activate_storage {
my ($class, $storeid, $scfg, $cache) = @_;

#run_client_cmd($scfg, $storeid, "status");
# a 'status' client command is to expensive here
# TODO: use a dummy ping API call to ensure the PBS API daemon is available for real
my $server = $scfg->{server};
my $port = $scfg->{port} // 8007;
PVE::Network::tcp_ping($server, $port, 2);

return 1;
}


Error from syslog

Oct 16 21:08:44 proxmox1 vzdump[5847]: VM 111 qmp command failed - VM 111 qmp command 'backup' failed - Node 'drive-scsi0' is busy: block device is in use by block job: mirror
Oct 16 21:08:44 proxmox1 vzdump[5847]: ERROR: Backup of VM 111 failed - VM 111 qmp command 'backup' failed - Node 'drive-scsi0' is busy: block device is in use by block job: mirror
Oct 16 21:08:44 proxmox1 vzdump[5847]: INFO: Starting Backup of VM 112 (qemu)
Oct 16 21:08:48 proxmox1 vzdump[5847]: INFO: Finished Backup of VM 112 (00:00:04)
Oct 16 21:08:48 proxmox1 vzdump[5847]: INFO: Starting Backup of VM 113 (qemu)
Oct 16 21:09:15 proxmox1 vzdump[5847]: INFO: Finished Backup of VM 113 (00:00:27)
Oct 16 21:09:15 proxmox1 vzdump[5847]: INFO: Backup job finished with errors
Oct 16 21:09:15 proxmox1 vzdump[5847]: job errors
Oct 16 21:09:15 proxmox1 vzdump[5718]: <root@pam> end task UPID:proxmox1:000016D7:0B69B52D:5F89BC7A:vzdump::root@pam: job errors
Hi @jegan

did this ever get resolved?

seeing the same error in our environment on 1 specific VM.

Bash:
ERROR: Backup of VM 114 failed - Node 'drive-virtio0' is busy: block device is in use by block job: mirror

""Cheers
G
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!