uncompresed VZdump to SMB share fails

zemaphore

New Member
Nov 24, 2024
6
1
3
My vzdump task to SMB-NAS fails if uncompressed is selected. Switching to a compressed (zstd) dump works. Also an uncompressed vzdump to a NFS share on the same target machine works. Suspect it is something with the SMB share configuration - default done via the Datacenter Storage GUI. Also system log on source and target machine is "empty" in regards to the problem.

Uncompressed vzdump backup to SMB-share fails:

Bash:
INFO: starting new backup job: vzdump 300 --remove 0 --storage nas35-pve-backups --mode snapshot --notes-template '{{guestname}}' --notification-mode auto --node pve-1 --compress 0 
INFO: Starting Backup of VM 300 (qemu)
INFO: Backup started at 2024-11-26 11:16:44
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: vm300
INFO: exclude disk 'scsi0' 'local-lvm:vm-300-disk-0' (backup=no)
INFO: exclude disk 'scsi1' 'local-lvm:vm-300-disk-1' (backup=no)
INFO: exclude disk 'scsi2' 'local-lvm:vm-300-disk-2' (backup=no)
INFO: exclude disk 'scsi3' 'local-lvm:vm-300-disk-3' (backup=no)
INFO: exclude disk 'scsi4' 'local-lvm:vm-300-disk-4' (backup=no)
INFO: exclude disk 'scsi5' 'local-lvm:vm-300-disk-5' (backup=no)
INFO: include disk 'scsi6' 'local-lvm:vm-300-disk-6' 4G
INFO: creating vzdump archive '/mnt/pve/nas35-pve-backups/dump/vzdump-qemu-300-2024_11_26-11_16_44.vma'
INFO: starting kvm to execute backup task
INFO: started backup task '34a33bdc-14fc-49e8-82c2-65a983c510b9'
INFO:   0% (7.2 MiB of 4.0 GiB) in 1s, read: 7.2 MiB/s, write: 2.7 MiB/s
ERROR: vma_queue_write: write error - Interrupted system call
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 300 failed - vma_queue_write: write error - Interrupted system call
INFO: Failed at 2024-11-26 11:16:47
INFO: Backup job finished with errors
job errors

Compressed vzdump to same share works:

Bash:
INFO: starting new backup job: vzdump 300 --remove 0 --node pve-1 --compress zstd --mode snapshot --notes-template '{{guestname}}' --storage nas35-pve-backups --notification-mode auto
INFO: Starting Backup of VM 300 (qemu)
INFO: Backup started at 2024-11-26 11:18:12
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: vm300
INFO: exclude disk 'scsi0' 'local-lvm:vm-300-disk-0' (backup=no)
INFO: exclude disk 'scsi1' 'local-lvm:vm-300-disk-1' (backup=no)
INFO: exclude disk 'scsi2' 'local-lvm:vm-300-disk-2' (backup=no)
INFO: exclude disk 'scsi3' 'local-lvm:vm-300-disk-3' (backup=no)
INFO: exclude disk 'scsi4' 'local-lvm:vm-300-disk-4' (backup=no)
INFO: exclude disk 'scsi5' 'local-lvm:vm-300-disk-5' (backup=no)
INFO: include disk 'scsi6' 'local-lvm:vm-300-disk-6' 4G
INFO: creating vzdump archive '/mnt/pve/nas35-pve-backups/dump/vzdump-qemu-300-2024_11_26-11_18_12.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '1c6cd8f9-d708-4fa8-b171-f48497bbd950'
INFO:  25% (1.0 GiB of 4.0 GiB) in 3s, read: 344.4 MiB/s, write: 324.1 MiB/s
INFO:  33% (1.3 GiB of 4.0 GiB) in 6s, read: 114.8 MiB/s, write: 104.2 MiB/s
INFO:  40% (1.6 GiB of 4.0 GiB) in 9s, read: 97.2 MiB/s, write: 86.5 MiB/s
INFO:  47% (1.9 GiB of 4.0 GiB) in 12s, read: 87.7 MiB/s, write: 77.1 MiB/s
INFO:  53% (2.1 GiB of 4.0 GiB) in 15s, read: 85.8 MiB/s, write: 75.2 MiB/s
INFO:  58% (2.3 GiB of 4.0 GiB) in 18s, read: 71.3 MiB/s, write: 60.6 MiB/s
INFO:  64% (2.6 GiB of 4.0 GiB) in 21s, read: 77.3 MiB/s, write: 66.7 MiB/s
INFO:  70% (2.8 GiB of 4.0 GiB) in 24s, read: 81.0 MiB/s, write: 70.3 MiB/s
INFO:  76% (3.1 GiB of 4.0 GiB) in 27s, read: 84.6 MiB/s, write: 73.9 MiB/s
INFO:  82% (3.3 GiB of 4.0 GiB) in 30s, read: 82.2 MiB/s, write: 71.5 MiB/s
INFO:  87% (3.5 GiB of 4.0 GiB) in 33s, read: 71.3 MiB/s, write: 60.7 MiB/s
INFO:  93% (3.7 GiB of 4.0 GiB) in 36s, read: 78.5 MiB/s, write: 73.5 MiB/s
INFO: 100% (4.0 GiB of 4.0 GiB) in 37s, read: 268.0 MiB/s, write: 288.0 KiB/s
INFO: backup is sparse: 662.74 MiB (16%) total zero data
INFO: transferred 4.00 GiB in 37 seconds (110.7 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 3.29GB
INFO: adding notes to backup
INFO: Finished Backup of VM 300 (00:00:38)
INFO: Backup finished at 2024-11-26 11:18:50
INFO: Backup job finished successfully

Uncompressed vzdump to NFS share works:

Bash:
INFO: starting new backup job: vzdump 300 --mode snapshot --compress 0 --storage nas35-pve-nfs --notification-mode auto --remove 0 --node pve-1 --notes-template '{{guestname}}'
INFO: Starting Backup of VM 300 (qemu)
INFO: Backup started at 2024-11-26 11:19:23
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: vm300
INFO: exclude disk 'scsi0' 'local-lvm:vm-300-disk-0' (backup=no)
INFO: exclude disk 'scsi1' 'local-lvm:vm-300-disk-1' (backup=no)
INFO: exclude disk 'scsi2' 'local-lvm:vm-300-disk-2' (backup=no)
INFO: exclude disk 'scsi3' 'local-lvm:vm-300-disk-3' (backup=no)
INFO: exclude disk 'scsi4' 'local-lvm:vm-300-disk-4' (backup=no)
INFO: exclude disk 'scsi5' 'local-lvm:vm-300-disk-5' (backup=no)
INFO: include disk 'scsi6' 'local-lvm:vm-300-disk-6' 4G
INFO: creating vzdump archive '/mnt/pve/nas35-pve-nfs/dump/vzdump-qemu-300-2024_11_26-11_19_23.vma'
INFO: starting kvm to execute backup task
INFO: started backup task '35776cbe-87fa-4320-a2c9-30dbaaad9639'
INFO:  61% (2.5 GiB of 4.0 GiB) in 3s, read: 839.8 MiB/s, write: 766.3 MiB/s
INFO:  72% (2.9 GiB of 4.0 GiB) in 6s, read: 146.2 MiB/s, write: 124.9 MiB/s
INFO:  79% (3.2 GiB of 4.0 GiB) in 9s, read: 99.1 MiB/s, write: 88.4 MiB/s
INFO:  84% (3.4 GiB of 4.0 GiB) in 12s, read: 68.9 MiB/s, write: 58.2 MiB/s
INFO:  88% (3.6 GiB of 4.0 GiB) in 15s, read: 60.4 MiB/s, write: 49.8 MiB/s
INFO: 100% (4.0 GiB of 4.0 GiB) in 51s, read: 12.6 MiB/s, write: 4.7 MiB/s
INFO: backup is sparse: 662.74 MiB (16%) total zero data
INFO: transferred 4.00 GiB in 51 seconds (80.3 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 3.35GB
INFO: adding notes to backup
INFO: Finished Backup of VM 300 (00:01:00)
INFO: Backup finished at 2024-11-26 11:20:23
INFO: Backup job finished successfully

/etc/pve/storage.cfg

Code:
root@pve-1:/# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup


lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images


cifs: nas35-pve-backups
        path /mnt/pve/nas35-pve-backups
        server 192.168.20.35
        share pve-backups
        content backup
        prune-backups keep-all=1
        username pve-user


nfs: nas35-pve-nfs
        export /mnt/rust/pve/backup-nfs
        path /mnt/pve/nas35-pve-nfs
        server 192.168.20.35
        content backup
        prune-backups keep-all=1

Kernel
Code:
Linux pve-1 6.8.12-4-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-4 (2024-11-06T15:04Z) x86_64 GNU/Linux

pve
Code:
dpkg -l |grep pve-
ii  libpve-access-control                8.2.0                               all          Proxmox VE access control library
ii  libpve-apiclient-perl                3.3.2                               all          Proxmox VE API client library
ii  libpve-cluster-api-perl              8.0.10                              all          Proxmox Virtual Environment cluster Perl API modules.
ii  libpve-cluster-perl                  8.0.10                              all          Proxmox Virtual Environment cluster Perl modules.
ii  libpve-common-perl                   8.2.9                               all          Proxmox VE base library
ii  libpve-guest-common-perl             5.1.6                               all          Proxmox VE common guest-related modules
ii  libpve-http-server-perl              5.1.2                               all          Proxmox Asynchrounous HTTP Server Implementation
ii  libpve-network-api-perl              0.10.0                              all          API endpoints for Proxmox VE's SDN stack
ii  libpve-network-perl                  0.10.0                              all          Proxmox VE's SDN (Software Defined Network) stack
ii  libpve-notify-perl                   8.0.10                              all          Notify helper module.
ii  libpve-rs-perl                       0.9.0                               amd64        PVE parts which have been ported to Rust - Rust source code
ii  libpve-storage-perl                  8.2.9                               all          Proxmox VE storage management library
ii  libpve-u2f-server-perl               1.2.0                               amd64        Perl bindings for libu2f-server
ii  proxmox-kernel-6.8.12-4-pve-signed   6.8.12-4                            amd64        Proxmox Kernel Image (signed)
ii  proxmox-kernel-6.8.4-2-pve-signed    6.8.4-2                             amd64        Proxmox Kernel Image (signed)
ii  pve-cluster                          8.0.10                              amd64        "pmxcfs" distributed cluster filesystem for Proxmox Virtual Environment.
ii  pve-container                        5.2.2                               all          Proxmox VE Container management tool
ii  pve-docs                             8.3.1                               all          Proxmox VE Documentation
ii  pve-edk2-firmware                    4.2023.08-4                         all          edk2 based UEFI firmware modules for virtual machines
ii  pve-edk2-firmware-legacy             4.2023.08-4                         all          edk2 based legacy 2MB UEFI firmware modules for virtual machines
ii  pve-edk2-firmware-ovmf               4.2023.08-4                         all          edk2 based UEFI firmware modules for virtual machines
ii  pve-esxi-import-tools                0.7.2                               amd64        Tools to allow importing VMs from ESXi hosts
ii  pve-firewall                         5.1.0                               amd64        Proxmox VE Firewall
ii  pve-firmware                         3.14-1                              all          Binary firmware code for the pve-kernel
ii  pve-ha-manager                       4.0.6                               amd64        Proxmox VE HA Manager
ii  pve-i18n                             3.3.1                               all          Internationalization support for Proxmox VE
ii  pve-lxc-syscalld                     1.3.0                               amd64        PVE LXC syscall daemon
ii  pve-manager                          8.3.0                               all          Proxmox Virtual Environment Management Tools
ii  pve-qemu-kvm                         9.0.2-4                             amd64        Full virtualization on x86 hardware
ii  pve-xtermjs                          5.3.0-3                             all          HTML/TypeScript based fully-featured terminal for Proxmox projects

vzdump
Code:
dpkg -l |grep backup
ii  libproxmox-backup-qemu0              1.4.1                               amd64        Proxmox Backup Server client library for QEMU
ii  proxmox-backup-client                3.2.9-1                             amd64        Proxmox Backup Client tools
ii  proxmox-backup-file-restore          3.2.9-1                             amd64        Proxmox Backup single file restore tools for pxar and block device backups
ii  proxmox-backup-restore-image         0.6.1                               amd64        Kernel/initramfs images for Proxmox Backup single-file restore.
 
anything in the system logs on either the PVE or samba server side?
 
Nothing on the SMB srv side...

PVE side:
Bash:
Nov 26 11:49:30 pve-1 vzdump[1665796]: <root@pam> starting task UPID:pve-1:00196B05:00F6D955:6745A7BA:vzdump:300:root@pam:
Nov 26 11:49:31 pve-1 vzdump[1665797]: INFO: starting new backup job: vzdump 300 --notes-template '{{guestname}}' --mode snapshot --storage nas35-pve-backups --remove 0 --mailto node@pv>
Nov 26 11:49:31 pve-1 vzdump[1665797]: INFO: Starting Backup of VM 300 (qemu)
Nov 26 11:49:31 pve-1 systemd[1]: Started 300.scope.
Nov 26 11:49:32 pve-1 kernel: tap300i0: entered promiscuous mode
Nov 26 11:49:32 pve-1 kernel: vmbr0: port 10(fwpr300p0) entered blocking state
Nov 26 11:49:32 pve-1 kernel: vmbr0: port 10(fwpr300p0) entered disabled state
Nov 26 11:49:32 pve-1 kernel: fwpr300p0: entered allmulticast mode
Nov 26 11:49:32 pve-1 kernel: fwpr300p0: entered promiscuous mode
Nov 26 11:49:32 pve-1 kernel: vmbr0: port 10(fwpr300p0) entered blocking state
Nov 26 11:49:32 pve-1 kernel: vmbr0: port 10(fwpr300p0) entered forwarding state
Nov 26 11:49:32 pve-1 kernel: fwbr300i0: port 1(fwln300i0) entered blocking state
Nov 26 11:49:32 pve-1 kernel: fwbr300i0: port 1(fwln300i0) entered disabled state
Nov 26 11:49:32 pve-1 kernel: fwln300i0: entered allmulticast mode
Nov 26 11:49:32 pve-1 kernel: fwln300i0: entered promiscuous mode
Nov 26 11:49:32 pve-1 kernel: fwbr300i0: port 1(fwln300i0) entered blocking state
Nov 26 11:49:32 pve-1 kernel: fwbr300i0: port 1(fwln300i0) entered forwarding state
Nov 26 11:49:32 pve-1 kernel: fwbr300i0: port 2(tap300i0) entered blocking state
Nov 26 11:49:32 pve-1 kernel: fwbr300i0: port 2(tap300i0) entered disabled state
Nov 26 11:49:32 pve-1 kernel: tap300i0: entered allmulticast mode
Nov 26 11:49:32 pve-1 kernel: fwbr300i0: port 2(tap300i0) entered blocking state
Nov 26 11:49:32 pve-1 kernel: fwbr300i0: port 2(tap300i0) entered forwarding state
Nov 26 11:49:32 pve-1 vzdump[1665797]: VM 300 started with PID 1665819.
Nov 26 11:49:33 pve-1 kernel: tap300i0: left allmulticast mode
Nov 26 11:49:33 pve-1 kernel: fwbr300i0: port 2(tap300i0) entered disabled state
Nov 26 11:49:33 pve-1 kernel: fwbr300i0: port 1(fwln300i0) entered disabled state
Nov 26 11:49:33 pve-1 kernel: vmbr0: port 10(fwpr300p0) entered disabled state
Nov 26 11:49:33 pve-1 kernel: fwln300i0 (unregistering): left allmulticast mode
Nov 26 11:49:33 pve-1 kernel: fwln300i0 (unregistering): left promiscuous mode
Nov 26 11:49:33 pve-1 kernel: fwbr300i0: port 1(fwln300i0) entered disabled state
Nov 26 11:49:33 pve-1 kernel: fwpr300p0 (unregistering): left allmulticast mode
Nov 26 11:49:33 pve-1 kernel: fwpr300p0 (unregistering): left promiscuous mode
Nov 26 11:49:33 pve-1 kernel: vmbr0: port 10(fwpr300p0) entered disabled state
Nov 26 11:49:33 pve-1 qmeventd[25976]: read: Connection reset by peer
Nov 26 11:49:33 pve-1 vzdump[1665797]: ERROR: Backup of VM 300 failed - vma_queue_write: write error - Interrupted system call
Nov 26 11:49:33 pve-1 vzdump[1665797]: INFO: Backup job finished with errors
Nov 26 11:49:33 pve-1 systemd[1]: 300.scope: Deactivated successfully.
Nov 26 11:49:33 pve-1 systemd[1]: 300.scope: Consumed 1.132s CPU time.
Nov 26 11:49:33 pve-1 vzdump[1665797]: job errors
Nov 26 11:49:33 pve-1 vzdump[1665796]: <root@pam> end task UPID:pve-1:00196B05:00F6D955:6745A7BA:vzdump:300:root@pam: job errors
Nov 26 11:49:34 pve-1 qmeventd[1665935]: Starting cleanup for 300
Nov 26 11:49:34 pve-1 qmeventd[1665935]: Finished cleanup for 300
 
If you try copying/rsyncing a similar size file (4GB?) from the Node host to the SMB share, does this run normally?
 
Hey fabian,

I enabled CIFS debugging and did the failed run (uncompressed) and a successful (compressed) run. To me the output looks comparable.

dmesg with CIFS debugging enabled and failed run:

Code:
[164663.028930] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164663.036532] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164663.036927] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164663.039458] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164664.063190] tap300i0: entered promiscuous mode
[164664.106122] vmbr0: port 9(fwpr300p0) entered blocking state
[164664.106125] vmbr0: port 9(fwpr300p0) entered disabled state
[164664.106138] fwpr300p0: entered allmulticast mode
[164664.106174] fwpr300p0: entered promiscuous mode
[164664.106378] vmbr0: port 9(fwpr300p0) entered blocking state
[164664.106380] vmbr0: port 9(fwpr300p0) entered forwarding state
[164664.122719] fwbr300i0: port 1(fwln300i0) entered blocking state
[164664.122723] fwbr300i0: port 1(fwln300i0) entered disabled state
[164664.122735] fwln300i0: entered allmulticast mode
[164664.122766] fwln300i0: entered promiscuous mode
[164664.122817] fwbr300i0: port 1(fwln300i0) entered blocking state
[164664.122818] fwbr300i0: port 1(fwln300i0) entered forwarding state
[164664.130858] fwbr300i0: port 2(tap300i0) entered blocking state
[164664.130862] fwbr300i0: port 2(tap300i0) entered disabled state
[164664.130868] tap300i0: entered allmulticast mode
[164664.130910] fwbr300i0: port 2(tap300i0) entered blocking state
[164664.130911] fwbr300i0: port 2(tap300i0) entered forwarding state
[164664.241398] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164665.542204] tap300i0: left allmulticast mode
[164665.542222] fwbr300i0: port 2(tap300i0) entered disabled state
[164665.551880] fwbr300i0: port 1(fwln300i0) entered disabled state
[164665.551931] vmbr0: port 9(fwpr300p0) entered disabled state
[164665.552178] fwln300i0 (unregistering): left allmulticast mode
[164665.552180] fwln300i0 (unregistering): left promiscuous mode
[164665.552181] fwbr300i0: port 1(fwln300i0) entered disabled state
[164665.564891] fwpr300p0 (unregistering): left allmulticast mode
[164665.564894] fwpr300p0 (unregistering): left promiscuous mode
[164665.564895] vmbr0: port 9(fwpr300p0) entered disabled state
[164665.659020] CIFS: Status code returned 0x80000006 STATUS_NO_MORE_FILES
[164665.667531] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND

dmesg with CIFS debugging enabled and successful run (with compression):
Code:
[164779.840962] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164779.848178] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164779.848559] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164779.850386] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164780.895974] tap300i0: entered promiscuous mode
[164780.939360] vmbr0: port 9(fwpr300p0) entered blocking state
[164780.939364] vmbr0: port 9(fwpr300p0) entered disabled state
[164780.939377] fwpr300p0: entered allmulticast mode
[164780.939411] fwpr300p0: entered promiscuous mode
[164780.939616] vmbr0: port 9(fwpr300p0) entered blocking state
[164780.939617] vmbr0: port 9(fwpr300p0) entered forwarding state
[164780.955874] fwbr300i0: port 1(fwln300i0) entered blocking state
[164780.955879] fwbr300i0: port 1(fwln300i0) entered disabled state
[164780.955892] fwln300i0: entered allmulticast mode
[164780.955924] fwln300i0: entered promiscuous mode
[164780.955974] fwbr300i0: port 1(fwln300i0) entered blocking state
[164780.955975] fwbr300i0: port 1(fwln300i0) entered forwarding state
[164780.964833] fwbr300i0: port 2(tap300i0) entered blocking state
[164780.964837] fwbr300i0: port 2(tap300i0) entered disabled state
[164780.964843] tap300i0: entered allmulticast mode
[164780.964905] fwbr300i0: port 2(tap300i0) entered blocking state
[164780.964906] fwbr300i0: port 2(tap300i0) entered forwarding state
[164781.120854] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164815.498409] tap300i0: left allmulticast mode
[164815.498425] fwbr300i0: port 2(tap300i0) entered disabled state
[164815.516648] fwbr300i0: port 1(fwln300i0) entered disabled state
[164815.516834] vmbr0: port 9(fwpr300p0) entered disabled state
[164815.517122] fwln300i0 (unregistering): left allmulticast mode
[164815.517124] fwln300i0 (unregistering): left promiscuous mode
[164815.517126] fwbr300i0: port 1(fwln300i0) entered disabled state
[164815.532325] fwpr300p0 (unregistering): left allmulticast mode
[164815.532329] fwpr300p0 (unregistering): left promiscuous mode
[164815.532330] vmbr0: port 9(fwpr300p0) entered disabled state
[164815.623947] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164815.629695] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164815.635412] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
[164815.636570] CIFS: Status code returned 0x80000006 STATUS_NO_MORE_FILES
[164815.647384] CIFS: Status code returned 0xc0000034 STATUS_OBJECT_NAME_NOT_FOUND
 
If you try copying/rsyncing a similar size file (4GB?) from the Node host to the SMB share, does this run normally?
Yes ... I tried doing the uncompressed backup to a local directory and copied to the same SMB target. No issues
 
you could also try different kernel versions on the PVE side, maybe it is a regression there...
 
It works if I target the 2nd SMB server. So it must be something on the NAS side. Thank you anyways for the suggestions!
 
  • Like
Reactions: fabian