PVE hosts grayout if a SMB storage mount timeout

[DANILO MONTAGNA]

Hi,

We have a PVE cluster with 5 nodes, everything works great!!! the only problem is that when a SMB mount (PVE shared storage for VM backups) timeout ... the cluster stay all grayout... and we cant manage any VM on this cluster.. everything display as grayout and stays unavailable for view/editing until the SMB mount is done!! some times we need to reboot our SMB backup server for maintenance etc... and until this server comes alive again.. the PVE cluster stays all grayout for management...

any tips on how to resolve it?

 

[Moayad]

Hi,

there is any error in syslog/journalctl?

Please post output of pveversion -v and Status of services: pvedaemon,pveproxy,pvestatd

 

[DANILO MONTAGNA]

The only error/info in the syslog when it happens is below:

Nov 11 22:24:03 host03-pve kernel: [2418813.594238] CIFS VFS: \\192.168.64.37 has not responded in 180 seconds. Reconnecting...
Nov 11 22:25:04 host03-pve kernel: [2418875.031053] CIFS VFS: \\192.168.64.37 Send error in SessSetup = -11
Nov 11 22:32:58 host03-pve kernel: [2419349.457673] CIFS VFS: \\192.168.64.37 Cancelling wait for mid 14 cmd: 5
Nov 11 22:32:58 host03-pve kernel: [2419349.457749] CIFS VFS: \\192.168.64.37 Cancelling wait for mid 15 cmd: 16
Nov 11 22:35:58 host03-pve kernel: [2419529.383614] CIFS VFS: Close unmatched open
Nov 11 23:04:17 host03-pve kernel: [2421227.877423] CIFS VFS: \\192.168.64.37 has not responded in 180 seconds. Reconnecting...

service status is all running when this problem happens!!!

below is the package version..

root@host03-pve:~# pveversion -v
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.55-1-pve: 5.4.55-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.18-1-pve: 5.3.18-1
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.3.10-1-pve: 5.3.10-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.0-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-1
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

 

[Moayad]

Hi again,

Please try to upgrade your nodes to latest version and try again? - apt update && apt full-upgrade pve-manager version should be 6.2-15 (running version: 6.2-15/48bd51b6)

 

[DANILO MONTAGNA]

I can do that.. but this is happening for a while ... since version 6.2.1 when we installed this environment...

 

[Moayad]

is the SMB share for only backups? - please post output of /etc/pve/storage.cfg file

 

[DANILO MONTAGNA]

is the SMB share for only backups? - please post output of /etc/pve/storage.cfg file

yes.. only for backups..

root@host03-pve:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content iso,backup,vztmpl
maxfiles 1
shared 0

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

cifs: backup-vol01
path /mnt/pve/backup-vol01
server 192.168.64.37
share pve_backup
content images,backup
maxfiles 3
nodes host01-pve,host02-pve,host03-pve,host04-pve,host05-pve
username backup

lvm: stg-ssd-vol01
vgname vg_eql
content rootdir,images
nodes host01-pve,host02-pve,host03-pve,host04-pve,host05-pve
shared 1