Running proxmox from a USB drive? - SOLVED

[SInisterPisces]

I would not rule out Proxmox as it does log quite a lot (and data for graphs) more than plain Debian (or other Linux). And maybe this has been reduced recently, but the pmxcfs also used to have a lot of write amplification.
But I agree that VMs with a lot of (synchronous) I/O, especially when the block sizes don't match, can easily become the major cause when the number of running VMs increases.

This is an interesting point that I need to do more research on. I'd like to set up a cluster, so I'd already planned to do the following to lessen boot disk writes:

1. Shared storage over NFS/ZFS over iSCSI (I'm still figuring out if I want to deal with ZFS over iSCSI … all the tutorial videos I watch are LVM-based Proxmox installs, so they all use NFS. I'm not sure whether ZFS over iSCSI is even considered a rock-solid feature or still more in-development/experimental like Ceph); and
2. Logging to an external syslog server.

I had not considered the need to handle logging specific to clusters, so I need to look into whether that's included in the general offload to a remote syslog server.

don't see that as such an obstacle. What I would do in your situation:

1. Use a small m.2 drive. 128gb?
2. From time to time (every kernel update etc.), make a dd zipped image of that boot drive to the storage area on the server.
3. Something goes wrong, write that dd image back to the boot drive (or a new one).
4. Make as little changes to the host OS as possible & document all those changes.
5. Document & backup all Proxmox host settings. (Search these forums on which files/folders to backup).
6. You could always re-install Proxmox & redo 4 & 5 above.

For all your VMs & LXCs you must anyway have full & restorable backups.

@gfngfn256 's advice on making backups is absolutely spot on. For something more automated, there's a PVE host backup script. I'm aware of it, but haven't deployed it, so I haven't read it to see exactly what it does, but it's recommended a lot:
https://community-scripts.github.io/ProxmoxVE/scripts?id=host-backup
That's also an interactive script; I'm not sure how to automate it (yet).

It's worth noting that implementing host (PVE node) backup is on the roadmap for Proxmox Backup Server, at https://pbs.proxmox.com/wiki/index.php/Roadmap . At the time I'm writing this, it's the last major server-side item on the roadmap, but we don't have any information on when it might land. So, hopefully, a standardized, easily automated official solution will be coming soon.

 

[hooman64]

I don't see that as such an obstacle. What I would do in your situation:

1. Use a small m.2 drive. 128gb?
2. From time to time (every kernel update etc.), make a dd zipped image of that boot drive to the storage area on the server.
3. Something goes wrong, write that dd image back to the boot drive (or a new one).
4. Make as little changes to the host OS as possible & document all those changes.
5. Document & backup all Proxmox host settings. (Search these forums on which files/folders to backup).
6. You could always re-install Proxmox & redo 4 & 5 above.

For all your VMs & LXCs you must anyway have full & restorable backups.

Thanks for your reply. To backup the VMc & LXCs, could I take the same approach and take snapshots? Or is it recommended to mirror the drive all together? Since my system doesn't support bifurcation and it only comes with one PCIe 4.0 interface, I most likely have to purchase an adapter with a switch to be able to run 2 NVMe drives off of that port. in that case, would it make more sense to put both the OS and the VMs on the same drive? Or running the OS from the USB port is still a better idea? and just mirror the VMs with that card?

 

[gfngfn256]

As you are probably aware a mirror of a drive is not a backup, data will just be copied between the two. Corrupted/deleted/erroneous data on one will be mirrored to the second one. It may be useful (& not always) if one of the drives just physically fails.

So you must anyway have a full backup system & storage location for all VM's & LXCs which should (at least) be on a separate drive. You have indicated that you will be using the 4 3.5" bays for data. I'm guessing you could somehow share that datastore as a backup location for these Proxmox backups? Whatever your strategy, you will have to make sure these backups will be accessible (somehow) even if the Proxmox node fails. You know the 3-2-1 strategy - so you'll also want another copy of those backups somewhere else.

I have no experience with a twin NVME (splitter) PCI card. But I can imagine there maybe a choking effect of the data transfer between the 2 drives. I don't think I'd go that route. In your position I'd probably go for one NVME with the VMs etc. & run the OS from an M.2 USB.

You will have to test the OS running on a USB connection, because on some hardware/software USB can be hit & miss.

If I really had my own way with your setup, I'd probably keep one of those 3.5" SATA ports as the OS drive & just leave the other 3 for a datastore.

In my home setup, I'm currently running the OS & 1 constantly used VM on one NVME (all the other VMs & LXCs are on a separate drive), for over 2 years & I've had no problems. I constantly have Proxmox backups for all my VMs & LXCs. I periodically dd image that NVME to an external location. I also periodically copy all backups to another location.

Good luck.

 

[hooman64]

As you are probably aware a mirror of a drive is not a backup, data will just be copied between the two. Corrupted/deleted/erroneous data on one will be mirrored to the second one. It may be useful (& not always) if one of the drives just physically fails.

So you must anyway have a full backup system & storage location for all VM's & LXCs which should (at least) be on a separate drive. You have indicated that you will be using the 4 3.5" bays for data. I'm guessing you could somehow share that datastore as a backup location for these Proxmox backups? Whatever your strategy, you will have to make sure these backups will be accessible (somehow) even if the Proxmox node fails. You know the 3-2-1 strategy - so you'll also want another copy of those backups somewhere else.

I have no experience with a twin NVME (splitter) PCI card. But I can imagine there maybe a choking effect of the data transfer between the 2 drives. I don't think I'd go that route. In your position I'd probably go for one NVME with the VMs etc. & run the OS from an M.2 USB.

You will have to test the OS running on a USB connection, because on some hardware/software USB can be hit & miss.

If I really had my own way with your setup, I'd probably keep one of those 3.5" SATA ports as the OS drive & just leave the other 3 for a datastore.

In my home setup, I'm currently running the OS & 1 constantly used VM on one NVME (all the other VMs & LXCs are on a separate drive), for over 2 years & I've had no problems. I constantly have Proxmox backups for all my VMs & LXCs. I periodically dd image that NVME to an external location. I also periodically copy all backups to another location.

Good luck.

Thanks these are some really good advise! Yes I think I can use the storage as a backup location and store one either on the cloud or somewhere else.

I'm new to this whole thing and am learning as I go. The reason I probably want to keep all 4 3.5"s for storage is that I probably will use either RAID10 or RaidZ1 and in either of those scenarios reducing my total number of drives means losing a lot of storage space potentially. So I probably have to stick with either running the OS off of a usb drive and VMs off of the NVMe or like you said having both the OS and the VMs on the same drive. I'm a bit paranoid and prefer not to put all of my eggs in one basket. I'm not sure how intensive the usage would be, the system is for hoem use, and the usage I had in mind is basically just media server, file storage and private cloud, some personal finance tools, pihole (I currently have that on a RPi 4), Home Assistant, some firewall systems (OPNSense/pfSense), and possibly some VMs to play around and test some other tools and not to worry about breaking things. Probably the list will grow as I learn more. I'm trying to somewhat start correctly so I don't have to keep modifying the system drastically and at the same time keep it simple.

 

[SInisterPisces]

Thanks these are some really good advise! Yes I think I can use the storage as a backup location and store one either on the cloud or somewhere else.

I'm new to this whole thing and am learning as I go. The reason I probably want to keep all 4 3.5"s for storage is that I probably will use either RAID10 or RaidZ1 and in either of those scenarios reducing my total number of drives means losing a lot of storage space potentially. So I probably have to stick with either running the OS off of a usb drive and VMs off of the NVMe or like you said having both the OS and the VMs on the same drive. I'm a bit paranoid and prefer not to put all of my eggs in one basket. I'm not sure how intensive the usage would be, the system is for hoem use, and the usage I had in mind is basically just media server, file storage and private cloud, some personal finance tools, pihole (I currently have that on a RPi 4), Home Assistant, some firewall systems (OPNSense/pfSense), and possibly some VMs to play around and test some other tools and not to worry about breaking things. Probably the list will grow as I learn more. I'm trying to somewhat start correctly so I don't have to keep modifying the system drastically and at the same time keep it simple.

How are you planning to use the 3.5" disks? TrueNAS VM? Direct storage on Proxmox itself (Proxmox as NAS)? My understanding from the thread is that you'll be setting them up as an array in Proxmox itself.

Just make sure you don't keep your VM and LXC containers' virtual boot disks on a RAIDZ1. RAIDZ1 will absolutely kill VM performance. There are a number of reasons for this, but the big thing is how much I/O overhead RAIDZ1's parity calculations create for a VM.

So, if you want redundant storage for VM and LXCs, you need a pool with at least one mirror VDEV for your VM boot disk store. (You can run VMs in a RAIDZ1, but you will have a bad time.) And ideally, your VMs should be running off solid state storage--either NVME or SATA SSDs are fine for home use. SATA SSDs are only "too slow" in enterprise/commercial applications, as I understand it. Consider: A SATA SSD mirror where the slowest SSD writes at 550 MB/s means you'll write at 550 MB/s at best and read at 1100 MB/s (or, ~10 Gbps, about 1 GB/s), at best. ZFS overhead for sync writes will slow this down to a degree, but it'll still be an order of magnitude faster than a Pi 4 on its best day.

If your 3.5" bays are for actual 3.5" mass storage drives, and you have no other plans for the PCIe slot, grab one of these. I haven't used it myself, but other people in your situation have used it to get dual NVME into their systems. It was actually recommended to me when I was looking at a similar PVE node setup. The one I'm linking includes coolers for the NVMEs. Theoretically unnecessary for PCIe 3.0x4 NVME disks, but still good to have--especially if you end up using PCIe 4.0x4 NVMEs or are using a case without great airflow.
https://www.amazon.com/gp/product/B0BCFZQZLR/ref=ox_sc_saved_title_6?smid=A36DOQ8QSJXCYP&psc=1

That's a PCIe 3.0x4 card with two NVME sockets and a PCIe splitter. So, at best, each NVME would get a 3.0x2 connection, which is two gigabytes/second per NVME. (Again, sync write overhead will make this less in practice.) At that point, your NICs are most likely to be your bottleneck, and even then, consider your workload: how likely are you to actually be pushing 10 Gbps (1 GB/s) of data read/writes around at any given time?

(Not trying to disparage you getting the setup you want, but don't fall into the trap I did of thinking purely about specs and not about what you actually need the system to do.)

Re: Backup
Do you have the budget for a small N100 (or similar) system to run Proxmox Backup Server on? A PBS server with a 2x SATA SSD mirror pool of appropriately sized SATA SSDs would backup everything but the Proxmox node itself. Something like a mini-ITX build with the