Mounted USB backup drive locking up host when full

[Louieo]

I have an external USB drive used for backing up all containers. The drive is mounted on the host under /mnt/Backups

• The host fs is on local
• all CTs are stored on local-lvm
• all other user data for fileshares etc is on zfspool

As the drive fills up, the over all disk usage reported on the host increases. When full the host essentially locks up, all CTs are inaccessible and I can only SSH into the host (albeit very slow). When I make some more space on the external Backups drive, everything frees back up and is accessible.

Why is the mounted backup drive affecting the host like this and what can I do?

Thanks in advance.

 

[Louieo]

Bump. Anyone?

 

[BobhWasatch]

Are you really, really, sure that the backup drive is mounted and it is mounted where you think it should be? Your symptoms sound a lot like the backup is being written to the rootfs rather than the external disk. That can happen if you have set up a mount point as directory storage but don't have the disk mounted.

 

[Louieo]

Are you really, really, sure that the backup drive is mounted and it is mounted where you think it should be? Your symptoms sound a lot like the backup is being written to the rootfs rather than the external disk. That can happen if you have set up a mount point as directory storage but don't have the disk mounted.

You are correct thanks! Upon rebooting the host, the USB drive is not auto mounted yet the mount directory and "Storage" item in pve manager remains. When backups resume, they are indeed writing to the host root drive. Is there a reliable solution to auto mounting USB drives yet? (apart from using Proxmox Backup Server - not always feasible in a home server environment)

 

[BobhWasatch]

Here is how I do it. In /etc/fstab:

LABEL=backup            /mnt/backup     ext4    noauto,x-systemd.automount      0 2

Then it will be auto-mounted as soon as PVE (or anything else) touches the mount point. Since PVE scans drives once per minute, it should get mounted within that time.

Use can use /dev/disk-by-id or a UUID in place of a volume label as I've done.

 

[Louieo]

Thanks I've implemented with UUID and works great! After a reboot and once the disk had automounted, I did also have to remove and re-add from Datacentre > Storage.

 

[VictorSTS]

There's also the property is_mountpoint [1] that you can set with pvesm set <STORAGE> --is_mountpoint yes, that once enabled will disable the storage in PVE if the path isn't mounted. The backup won't start if the storage is not available.

(apart from using Proxmox Backup Server - not always feasible in a home server environment)

Something that works for me is to have a VM with PBS and passthrough the USB drive (either by USB passthrough or full drive passthrough) to it. Backups are done using PBS and the datastore in the USB drive could be mounted to another PBS server if needed. Isn't superfast, but enough for ~3TB of source data.

[1] https://pve.proxmox.com/pve-docs/pvesm.1.html

 

[OberLeo]

Are you really, really, sure that the backup drive is mounted and it is mounted where you think it should be? Your symptoms sound a lot like the backup is being written to the rootfs rather than the external disk. That can happen if you have set up a mount point as directory storage but don't have the disk mounted.

Thanks a lot, this was indeed my problem for month (or even more than a year). I didn't understand, why I have seen Backups in proxmox GUI but my USB-Disk was empty. I also didn't understand why the proxmox GUI showed 100GB available space only while my USB-Disk is 1TB.

Maybe this will solve another problem also: My proxmox regularly (sometimes within days, sometimes within month) crashes because the main disk (root) is suddenly read only. Maybe it was full.