Snapshot backup can cause KVM guest freeze on proxmox 3.2

[vasek125]

Hello. I am using proxmox 3.2. Currently I am using manual snapshot backups for my KVM guests. All my guests are KVM on pve LVM volume. I am using CIFS share mounted to /mnt/backup via fstab. Today my backup server was overloaded and there were some connection problems. I was expecting backup failure, but backup was still running (standing on some percentage) BUT one of my guest stopped working well. It was web server guest and some virtualhosts were not working. After some research I found some file system exceptions inside this guest (in dmesg). No issues with proxmox host in dmesg so probably not hardware issue. This problem was solved by manually stopping running snapshot backup. After that everything works well.

Is this issue related to "out of order" snapshot backup in Proxmox? Should I use NFS instead of CIFS?

 

[dietmar]

Should I use NFS instead of CIFS?

Please try an report back. What kind of CIFS server is that?

 

[vasek125]

Please try an report back. What kind of CIFS server is that?

CIFS server: Samba 3.6.22 on FreeBSD. I will try NFS.

 

[dietmar]

CIFS server: Samba 3.6.22 on FreeBSD.

Running inside a VM? If so, what kind of VM?

 

[vasek125]

Running inside a VM? If so, what kind of VM?

No. Backup server is physical server with CIFS share named "virt". This share is mounted on proxmox server to "/mnt/backup". There were some issues with backup server (overloaded, not related to proxmox). But this issue occured while running backup of "web" virtual machine so backup was freezed (not interrupted). This is OK, but running "web" virtual machine stopped working properly (file system exceptions, ...). After manual interruption of backup, "web" continued to work properly again. So problematic is proxmox snapshot backup in case of backup storage failure.


On proxmox machine was:
CIFS VFS: sends on sock ffff88081907a680 stuck for 15 seconds
CIFS VFS: Error -11 sending data on socket to server
CIFS VFS: sends on sock ffff88081907a680 stuck for 15 seconds
CIFS VFS: Error -11 sending data on socket to server

Backup share was accessible but probably very slow (maybe due to running another backups from another physical machines).

 

[vasek125]

I located the problem. It is kernel hang on cifs mount. I dont know why but even other apps (dd) hangs in case of cifs lost connection. But it is quite strange because I use "soft" as a mount option (which should be even default). It could be a good feature for snapshot system to be able to detect these situations.

 

[dietmar]

I located the problem. It is kernel hang on cifs mount. I dont know why but even other apps (dd) hangs in case of cifs lost connection. But it is quite strange because I use "soft" as a mount option (which should be even default). It could be a good feature for snapshot system to be able to detect these situations.

Please can you test with our new kernel fro pvetest repository:

# wget ftp://download.proxmox.com/debian/d...pve-kernel-2.6.32-28-pve_2.6.32-123_amd64.deb
# dpkg -i pve-kernel-2.6.32-28-pve_2.6.32-123_amd64.deb

There are some cifs related fixes.

 

[vasek125]

Please can you test with our new kernel fro pvetest repository:

# wget ftp://download.proxmox.com/debian/d...pve-kernel-2.6.32-28-pve_2.6.32-123_amd64.deb
# dpkg -i pve-kernel-2.6.32-28-pve_2.6.32-123_amd64.deb

There are some cifs related fixes.

Even with this kernel backup will stop (and all other transfers) when multiple transfers are running (e.g. manually dd if=/dev/pve/root ... ) together with proxmox backup. But it seems now it is killable - I can "kill -9" dd processes and proxmox backup will start/continue successfully.

Now I tested sshfs instead of cifs and it works perfectly all time. So I will probably use sshfs (with immutable bit set do directory dump before mounting remote directory with sshfs so backup will be never running "locally" without mounted fs) which is easier to configure than nfsv4.