Backup to Samba Share freezes

[dva411]

I'm running 4 VMs on my Proxmox host. One of the VMs is Open Media Vault, with 2X 18tb USB drives passed through as scsi devices. The drives are shared back to the host and other VMs via CIFS.

I am able to backup 3 of my VMs to the samba shares. However, the largest VM (a 100 GiB Ubuntu 22.04 desktop instance running docker and 30+ containers) will not backup. It completes anywhere from 30%-90% and then freezes the socket connection. No error message is generated in the backup. It will hang indefinitely until I kill the process. When this happens, I can shut down all my VMs, but the host wont shut down gracefully, it hangs, and I have to kill the power. Rebooting is the only way I have been able to get access to the USB Samba Shares again. My other VMs are between 30-60% of the size, but they backup up successfully every time. One of the VMs is running frigate, and it is able to write 5X 1080 streams 24X7 to the samba shares without issue (yes, I've turned that off when running backups, but it hasn't helped). I can successfully restore one of my old backups of the Ubuntu VM from the samba shares without issue. However, I cant write a new backup. I used to have the drives connected to a raspberry pi and shared with native samba. I had no issue with my Proxmox VMs on my MS-01 backing up to the shares when they were connected to the pi. However throughput on the pi wasn't great, and I wanted to move the shares onto my MS-01 Proxmox host machine. I don't think its related, but the shares are encrypted with luks, and I have them network unlocking with clevis and mounting in Open Media Vault upon boot. Please Help, I have tried everything I can think of!

Journalctl

May 22 15:23:08 nuc kernel: CIFS: VFS: No writable handle in writepages rc=-9
May 22 15:23:11 nuc pvestatd[1623]: mkdir /mnt/MediaDriveBkupSmb: File exists at /usr/share/perl5/PVE/Storage/Plugin.pm line 1750

[  185.796713] CIFS: Attempting to mount //192.168.1.250/MediaDriveSmb
[  376.142909] CIFS: Attempting to mount //192.168.1.250/MediaDriveBkupSmb
[ 1288.001927] CIFS: VFS: \\192.168.1.250 Error -32 sending data on socket to server
root@nuc:/mnt/pve/MediaDriveBkupSmb# journalctl -f
May 22 18:07:37 nuc pvestatd[1616]: storage 'MediaDriveBkupSmb' is not online
May 22 18:07:47 nuc pvestatd[1616]: storage 'MediaDriveBkupSmb' is not online
May 22 18:07:49 nuc pveproxy[1651]: worker exit
May 22 18:07:49 nuc pveproxy[1650]: worker 1651 finished
May 22 18:07:49 nuc pveproxy[1650]: starting 1 worker(s)
May 22 18:07:49 nuc pveproxy[1650]: worker 9523 started
May 22 18:07:57 nuc pvestatd[1616]: storage 'MediaDriveBkupSmb' is not online
May 22 18:08:07 nuc pvestatd[1616]: storage 'MediaDriveBku!pSmb' is not online
May 22 18:08:17 nuc pvestatd[1616]: storage 'MediaDriveBkupSmb' is not online[/COD


PVEVERSION Proxmox host

root@nuc:~# pveversion --verbose
proxmox-ve: 8.2.0 (running kernel: 6.5.13-3-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.4-3
proxmox-kernel-6.8.4-3-pve-signed: 6.8.4-3
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
intel-microcode: 3.20240312.1
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.1
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.2-1
proxmox-backup-file-restore: 3.2.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.6
pve-container: 5.1.10
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.7
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2


And this is the VM that wont backup

root@nuc:/etc/pve/qemu-server# cat 101.conf
#scsi1%3A /dev/disk/by-uuid/9f1b8e21-c2b2-4610-a439-491e0fac6932,backup=0,size=17166334M
#scsi2%3A /dev/disk/by-uuid/073af071-c4a6-4e7d-a17a-903d998a971d,backup=0,size=17166334M
#scsi3%3A /dev/disk/by-uuid/f770f0e2-e9e1-4084-8543-38d831eb2cb2,backup=0,size=476939M
#scsi4%3A /dev/disk/by-uuid/fab9c4a9-40ba-419b-b338-94dfd060f030,backup=0,size=259961M
agent: 1,fstrim_cloned_disks=1
balloon: 0
bios: ovmf
boot: order=scsi0
cores: 6
cpu: x86-64-v2-AES
efidisk0: local-zfs:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:00:02.1,pcie=1,x-vga=1
machine: q35
memory: 20480
meta: creation-qemu=8.1.5,ctime=1710472890
name: AppServer
net0: virtio=BC:24:11:35:0D:CE,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-zfs:vm-101-disk-1,iothread=1,size=100G
scsihw: virtio-scsi-single
smbios1: uuid=96a279f9-bd02-4563-adcd-dacbbab31998
sockets: 1
vga: none
vmgenid: 4f27a9c9-1e0a-4ee2-8f89-575175853843

 

[Kingneutron]

> One of the VMs is Open Media Vault, with 2X 18tb USB drives passed through as scsi devices. The drives are shared back to the host and other VMs via CIFS

Sounds like a USB issue. Can you connect these drives as SATA?

It's really not recommended to try running a server off USB, it's horrible for reliability.

 

[dva411]

The server OS (Proxmox) and VMs are running on 2X 2TB internal NVME in zfs raid 1 configuration. The USB drives are media storage and backups. They stay connected and work great for media storage, streaming, constant NVR storage,etc. The only thing not working is backing up the one VM in proxmox. However, when the USB drives were connected to a raspberry pi and not my Proxmox host (MS-01), backups would work fine (the only difference is the samba shares/USB drives were on a different host machine). I suppose I can try passing them through as SATA.

 

[dva411]

I didn't try passing the disk through as SATA. I'm reluctant to make the switch as everything I have observed and read seems to indicate that scsi is faster and more stable. The drives are being written to over a samba connection 24X7 by my frigate VM. Additionally, yesterday, I backed up 5TB of data from one of the drives to the other using the borg plug-in in OMV. That wasn't over samba, but obviously levereged the SCSI interface on the OMV VM. The backup ran for more than 12 hours at high throughput, and completed successfully.

That seems to indicate the problem is not related to SCSI or USB. I guess I can try NFS to eliminate Samba as the root of the problem. Certainly samba by itself is not a problem because my frigate VM is connecting to the drives via Samba and is running fine. It seems it is samba on the Proxmox host or something specific to the Proxmox backup. Ideas?