KVM Snapshot backup failed ( stuck )

[EugenMayer]

Hello,

We have tried vzdump backups today using scheduled backups and the whole process stucked tonight. 2 of 2 proxmox nodes just did not finish their first backup (of the first VM to backup ) and stopped in between (rather stucked there, no progress )- those 2 VMs remained locked and not reachable until we killed the backup process ( task ) and unlocked the VMs using the cli.

- That is a status-screenshot of the running / stucked process:
- For the Local storage raw vm: http://screencast.com/t/Mo9PAnTiZtte
- For the LVM storage node: http://screencast.com/t/JY08gYiPOgfM
- It stucked for several hours (3-4 hours)

Any known issues here? I have seen some 1.x issues which should have been fixed a while ago in 2.x, but well, we are using:

Facts for both nodes:
- pve-manager/3.3-1/a06c9f73 (running kernel: 2.6.32-29-pve)
- active subscription
- one VM had a LVM storage
- the other a local raw storage
- both vms are kvm
- both proxmox storage are raid 1 hardware raids
- The backups are done on a samba share on all servers (1GBit internal connection to a netapp)
- backups on those to nodes started with 4 hours difference

What we tried before:
- Starting a backup manually worked without issues - also restoring on a different node

What we tried after:
- Samba share is mounted and writeable
- We unlocked both nodes and killed the tasks manually (using ps / kill )
- We also tried to run a new backup manually, which stopped on "ionice 7" and never even tried to create a dump. Looking like "waiting for ever"

 

[spirit]

Hi,

do you have tried with a nfs share on your netapp storage ?

 

[EugenMayer]

Hi,

do you have tried with a nfs share on your netapp storage ?

How is that a solution to the problem ? SMB is supposed to work and i cannot use NFS, since it is not available from my storage hoster.

 

[awesome]

How is that a solution to the problem ? SMB is supposed to work and i cannot use NFS, since it is not available from my storage hoster.

I had experienced this problem, when i used windows server 2012 as backup server with smb protocol.
When i switched to NFS service on my windows server, the problem with stucking goes away.

 

[EugenMayer]

I had experienced this problem, when i used windows server 2012 as backup server with smb protocol.
When i switched to NFS service on my windows server, the problem with stucking goes away.

Well thats a pitty, i cant change to NFS but i could try to backup localy - if the error does not occur, it must be SMB - maybe some mount options ( locking) are responsible for this?

Thanks for the reply

 

[awesome]

Well thats a pitty, i cant change to NFS but i could try to backup localy - if the error does not occur, it must be SMB - maybe some mount options ( locking) are responsible for this?

Thanks for the reply

I also tried local backup - all works fine. In conjunction with SMB backup process freezed at 94% and IO operation dropped to minimum with lzop process in the memory.
Try to google smb mount options and check them out.

 

[EugenMayer]

We solved the problem using mount-parameters (fstab)

,cache=none,forcedirectio,rsize=32768,wsize=32768,llock,rw,vers=3,prot=tcp,hard

Best