USB sticks vs SSD and DR

hacman

Renowned Member
Oct 11, 2013
86
8
73
Newcastle upon Tyne, UK
Hi all,

We're about to deploy a pair of Proxmox servers in an existing cluster, and have to decide how to store the OS. All 8 drive bays will be used for Ceph OSDs, so we were looking at either:

A) Install to internal USB stick. This stick will be a branded (SanDisk) model, so not cheap rubbish that is likely to die fast.
B) Install to a pair of cheap (30GB or so) SSDs that are mounted on a board in the PCI-E slot, which will be a software RAID.

Thus my questions are:

1) What has people's concern(s) been specifically with USB sticks in the past? Is it due to cheap sticks ceasing to work for no reason? Or more related to duty cycle/tolerance?
2) If we used either solution, and the node died, what sort of DR plans do people use in this case to essentially re-create the machine as an empty node in the cluster again, ready for the Ceph data to be re-populated.

I'd be quite interested to hear people's opinions on this.

Thanks,

Jon
 
For a time i was at the same questions. First i tryed an USBstick. But really... never, never, never. Single point of failure. The stick has no crashed, but i had a terrible feeling about. So we use on every server (when it make sense to install pve on extra part) on two good ssd's. Really here the same not the cheapest one. Samsung Pro is not needed, but use for example samsung evo.
 
Hi,

Sounds like we have similar reasoning then. That being said there should be minimal io to the drive looking at our current use so I'm torn.

The alternative is looking to be something like a pair of M-PCI-E modules on an adapter card, so I guess there is still a SPoF there too. For this we'd probably go cheap-er, something like the 30GB Kingston SSD Now devices.

I guess the question is how easy it is to recover from a failure without upsetting the PMVE cluster, as building clusters out such that disks and servers are essentially disposable is kind of what we aim for.

Jon
 
For recover we you another way. Boot with PXE and setup the base PVE system. Then whe writen an programm the does almost everything autmaticly. Not connect to cluster, but install packages, to configuration. We use puppet. But in the future PVE should automaticly setuped with foreman: https://theforeman.org We use it, for fullautomatics clientinstallation an configuration. So i think it can do it also for PVE. But... it is complex.
 
On my agenda is also to try booting via PXE and NFS. So you bootup your system completely from network/storage. (Besides direct iSCSI or FC-based SAN booting, but you're talking about Ceph, so you have no SAN).
 
PXE was an option we looked at, but as we're using Ceph to essentially aim for a no-SAN environment, it's a bit harder.

We did consider another option, which is USB sticks, but in ZFS RAID 1 (Proxmox now allows this at the install).

I've setup a test machine with this today, and it just worked. No messing around or anything.

It's not the fastest, but it's certainly more than sufficient. So if we used a USB 3.0 card to boost the speed a little, this may be a very good option.

Jon
 
I also considered ZFS RAID-1 on USB flash drives, but decided to use small SSDs for reliability. In case you run ceph mon and/or mds on the same host, I would also recommend SSDs.

Are there any reliable USB flash drives?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!