USB drive disappears on VM backup and VM startup

[wulfwyn]

Hello all! Noob post...

I have a brand new Western Digital Elements 4TB USB drive (powered by USB cable), which bought to back up my proxmox VMs and CTs, and to use to backup my macbook via TimeMachine.

I formatted the drive via Disk Utility on my macbook. I created 2 partitions. A 1TB apfs partition for the macbooks' time machine. And a 3TB ext4 partition for the proxmox backups.

The drive works fine in the macbook, and it shows up under my node's disks on the proxmox gui. The partitions show as sda1 (efi), sda2 (apfs), sda3 (ext4), and these are also visible in the `/dev/` directory.

I followed this video guide in an attempt to create a directory on sda3 for my proxmox backups. In the end I have a storage item in the left sidebar which can hold all the content types i selected.

However... when I select a VM, click Backup in the VM's sidebar, and then the `Backup now` button... the backup begins but quickly fails. (note. no VMs or CTs running at the time)

And when I check the node Disks, the USB disk has vanished. Not wanting to unplug a USB drive, I find that the only way to bring the USB drive back to the node's disks is by restarting the node.

Similarly, when I start a VM, I noticed the USB disk vanishes from the node disks also.


I'm sure someone will tell me that initially formatting the disk on a mac is an issue. I wanted to format it on proxmox, but I don't know how to create partitions on proxmox (I would still need a APFS partition for my macbook's timemachine backups). Is it necessary / possible? Maybe this would be a better plan?

let me know if I need to share more information. I'm running Proxmox 7.3-3 and am feeling a little (very!) vulnerable without having my 3 VMs and fileserver CT backed up.


Ran the long smart test on the drive and it seems fine: SMART overall-health self-assessment test result: PASSED

Here's a screenshot of a log from when I was trying to backup a VM to sda1:


pveversions:

proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-8
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

 

[wulfwyn]

hmm... ok here's a thought.

i pass through my usb ports to the VMs. when i launch a VM, it unmounts the USB drive from the host, and then it gets mounted into the VM. indeed the USB drive appears in a Ubuntu VM. (when i power down the VM though, the drive does not re-appear on the host).

So maybe during the backup process, the USB is getting unmounted from the host for similar reasons (usb passthrough?)...

at least the USB drive disappearing from host when a vm is launched makes sense now... but the backup process causing it to disappear is still strange.

 

[wulfwyn]

OK... well... I am learning things.

I had my motherboard's USB controller passed through to my VMs as a PCI device.

As I theorised in my last comment, this passthrough indeed appears to play silly bugger with backups.

Removing the PCI passthrough for the USB controller allowed me to backup the Ubuntu VM.

I should mention, at least for the Ubuntu VM, I'm adding individual USB devices (keyboard, mouse etc) to passthrough. But NOT the USB hard drive. USB devices I've passed through still work when the VM starts up. So all good. And it backs up fine like this too.

I'm now backing up the fileserver container (this obviously hadn't anything passed through to it so this would have backed up fine anyway). Once this is done I will do the other VMs.

And just as I typed that, the backup failed, but for another reason. The directory I created on the USB partition is only 100gb, despite having 3tb available. So I'm going to try and solve that before getting back to backups. (I suspect I need to set the file system on the partition to match the partition size... we shall see)

 

[wulfwyn]

Finally figured resizing the directory.

So I don't know why proxmox defaults these directories to 100gb or whatever it choses. I think this only happens when creating a directory from the datacentre. I tried expanding the file system to fill the partition size (and changing the file system type to linux) using cfdisk but this didn't work.

1. Wipe the partition (sda3) in GUI
2. Create directory but not from the datacentre! Do it from node->disks->directory. The partition is now available (previously it wasn't as it had the exfat file system from when i formatted and partitioned the drive on the mac). Select Ext4 and name it.

This video was a MASSIVE help, especially at the end.
https://www.youtube.com/watch?v=eS4SNodCTcI

OK.. still need to see if I can backup everything... tbc.

 

[wulfwyn]

Not enjoying this enormously.

It seems impossible to eject the USB HDD safely. There's clearly no GUI for it. I've read about removing the directory from datacentre, and then unmounting via cli, even with the -R flag. But the disc is still spinning. i can hear it and feel it.

Not even shutting down proxmox will cause the disk to spin down.

This again... may be to do with having formatted the thing initially on a Mac. I think this because I tried ejecting the drive via the VMs. I couldn't eject it from Ubuntu but was able to eject it from a Sonoma VM, it span down.

 

[wulfwyn]

So in the end it all kind of works. I have my Proxmox set up with all the VMs and containers I would want, and all of them work great. And now I have an external USB drive for Proxmox backups of ISOs, VMs and CTs, which has a partition I can use to also backup my MacBook with TimeMachine.

I have an external USB hard drive with 2 partitions (3 with the EFI).

• Time Machine backups for my MacBook
• Proxmox backups with my VMs and CTs, as well as my ISOs

USB pass through issues affected backups. The drive would disappear from the Proxmox GUI whenever I tried to backup a VM which had the drive passed through. The cumbersome solution is to remove the passthrough before running the backup. Not ideal but hey. Until someone explains why this is necessary I'll just have to accept not knowing why.

Mounting USB drives in Linux is laborious. Hard to really accept.

Spinning down a USB drive so it can be safely ejected seems impossible in Proxmox. Why?!?! But at least I have a way around it by passing it through to a MacOS Sonoma VM (as a USB device only I think, it didn't work when I passed through the whole USB controller as a PCI device) and ejecting it there.

So, other things I'm more accepting of, some of the issues I have are mainly to do with running MacOS VMs. I have 2, each with a different version of MacOS. Both work really well. Pretty much perfectly. The issues I encountered are to do with USB devices. The easiest thing to do is to just pass the whole motherboard USB controller through as a PCI device. Running the VMs this way gives me access to all the USB devices. Ubuntu is more forgiving, allowing me to passthrough specific USB devices/ports instead of the whole controller as a PCI device. MacOS Sonoma also allows this (though audio/video devices may not work). Older versions of MacOS, like Mojave, seem less tolerant and require the whole controller to be passed through (I can otherwise use the mouse and keyboard at login, but as soon as the system is fully up the mouse and the keyboard die).