vzdump backup errors when source storage is SSD

[chunchunmaru]

Hello, I have a problem with backups on a couple of servers that have only SSD storage.
The backup storage is on a Synology NAS with 2 RAID1 WD reds, I tried both NFS and CIFS but the results ended up to be the same

I have tested this on multiple NAS's with the same disk configuration (and same disks type)

I have noticed an improvement when I force in /etc/vzdump.conf a consistent bandwidth limit reduction to 75000kbps, but now the backups are very very slow!

I am using LZO compression in snapshot mode, all the machines got the qemu-agent and the flag is enabled in Option from proxmox
All the VM's disk have iothread=1, discard and ssd=1 and are using the VirtIO SCSI Single controller, default chipset

What it looks like to me is that due to the SSD speed compared to the NAS proxmox thinks the storage is down or something similar, the error is "interrupted by signal"
The vzdump log in on the attachments

I only found a couple of useful logs in the dmesg

[17035.180050] CIFS VFS: Server 192.168.1.250 has not responded in 180 seconds. Reconnecting...
[146080.394873] CIFS VFS: Close unmatched open
[278619.536564] CIFS VFS: Server 192.168.1.250 has not responded in 180 seconds. Reconnecting...
[287550.356662] CIFS VFS: Server 192.168.1.250 has not responded in 180 seconds. Reconnecting...
[320173.205368] CIFS VFS: Close unmatched open
[320173.766890] CIFS VFS: cifs_invalidate_mapping: could not invalidate inode 00000000d8aeefd7

[16449.503242] FS-Cache: Loaded
[16449.509554] FS-Cache: Netfs 'cifs' registered for caching
[349031.069391] FS-Cache: Netfs 'nfs' registered for caching
[349049.522125] FS-Cache: Duplicate cookie detected
[349049.522224] FS-Cache: O-cookie c=00000000a25df93b [p=0000000039406dc3 fl=222 nc=0 na=1]
[349049.522340] FS-Cache: O-cookie d=0000000066061d72 n=000000009d0e9a0d
[349049.522446] FS-Cache: O-key=[10] '040002000801c0a801fa'
[349049.522568] FS-Cache: N-cookie c=00000000b453d807 [p=0000000039406dc3 fl=2 nc=0 na=1]
[349049.522690] FS-Cache: N-cookie d=0000000066061d72 n=000000001f248419
[349049.522815] FS-Cache: N-key=[10] '040002000801c0a801fa'

Can someone give me an hint here? Is there something wrong with my configuration?

Thanks in advance!

 

[wolfgang]

Hi,

this looks like the NAS is not able to write this data fast enough to the disks and then buffer run full.
Please check the logs of your NAS.

 

[chunchunmaru]

Hi,

this looks like the NAS is not able to write this data fast enough to the disks and then buffer run full.
Please check the logs of your NAS.

Hello, I checked the NAS logs but I couldn't see anything useful, instead I tried looking for the duplicate-cookie detected and found a bunch of bugzilla kernel reports

Also I tried changing the NAS filesystem from btrfs to ext4 I'll see how it goes I guess

 

[chunchunmaru]

Hello, I have noticed the latest proxmox update almost fixed the issue on all my setups
The changelog mentions "Improved iothread backup stability" or something similar

Thanks for the help anyway