Proxmox over Debian installation
- Thread starter Reuven
- Start date
I don't get the discussion. PVE will write alot and sooner or later your SD card will fail, your server will crash or just don't boot anymore and you won't be able to use it until you buy a new disk, setup everything again from scratch, restore backups and so on so your server might be offline for hours or even days. If that is just a homeserver you don't rely on for work and you don't care about downtimes and additional hassle and work this might be fine. For everything else you should get a proper drive (or better even two for a mirrored setup) that fits PVEs minimum system requirements instead of screwing around with volatile ram disks or disabling PVE core components, which can cause additinal problems, to increase the life expencation of a SD card by some months. It doesn't really matter if that SD card will last for 3, 6 or 12 months. That all is not really suitable for a server and you could just get a 120GB SATA SSD for 20€ + a USB to SATA case/cable for 10€ that will be faster and last for many years.
Especially if you only got one SD card without any parity/redundancy. Other people here that are forced to use USB sticks atleast use 2 or 3 of them as a mirror so that the server will continue to operate if a stick fails. Then every several months a USB stick will die, they buy a new one and replace it without loosing data or any downtime.
And like fabian already said, that 30GB per day is with ZFS on an SSD. The main problem is not the amout of data that PVE writes, it is the number of small writes. Flash storages like SSDs, USB sticks and SD cards will degrade with every write and unlike HDDs they can't just write a 4KB file to a 4kb sector. NAND flash cells are grouped in big chunks and if you want to write a 4kb file it might need to erase and rewrite for example a big 1MB block. So if you write 100x 4kb it might actually write 100x 1MB to the storage (depending on the hardware that is used and how big this block size is). Because of this write amplification of flash based storages PVE writes terrible amounts of data even if the actual data PVE creates isn't that bad. And then USB sticks and SD cards don't got any cache for optimizing writes so its way more terrible compared to SSDs. And SSDs got garbage collection and wear leveling so that all NAND cells will degrade equally so it will last longer.
And with USB Sticks/SD cards you won't get SMART monitoring, so you never know how worn off that storage is so it might fail without any warning.
Especially if you only got one SD card without any parity/redundancy. Other people here that are forced to use USB sticks atleast use 2 or 3 of them as a mirror so that the server will continue to operate if a stick fails. Then every several months a USB stick will die, they buy a new one and replace it without loosing data or any downtime.
And like fabian already said, that 30GB per day is with ZFS on an SSD. The main problem is not the amout of data that PVE writes, it is the number of small writes. Flash storages like SSDs, USB sticks and SD cards will degrade with every write and unlike HDDs they can't just write a 4KB file to a 4kb sector. NAND flash cells are grouped in big chunks and if you want to write a 4kb file it might need to erase and rewrite for example a big 1MB block. So if you write 100x 4kb it might actually write 100x 1MB to the storage (depending on the hardware that is used and how big this block size is). Because of this write amplification of flash based storages PVE writes terrible amounts of data even if the actual data PVE creates isn't that bad. And then USB sticks and SD cards don't got any cache for optimizing writes so its way more terrible compared to SSDs. And SSDs got garbage collection and wear leveling so that all NAND cells will degrade equally so it will last longer.
And with USB Sticks/SD cards you won't get SMART monitoring, so you never know how worn off that storage is so it might fail without any warning.
Last edited:
Good point but in our case the hardware we are using is very low cost..almost disposable. and if the SDcards last a few years then we will replace them....that is why it is so important to know what is the real load on the SDcards and try to minimize it so as to extend the life of the these mini-servers...
the only realistic way to find out is to do a proof of concept with some real world load on the actual hardware you plan on using..
Its basically impossible to calculate/estimate how long your SD card will actually last because the SD card is a blackbox. Even if you use iostat to log how much PVE is writing to the card you will never know how big the cards internal write amplification is and no manufacturer will tell you that in the datasheets. Without SMART its even more a blackbox because you even cant read the wear for an estimation. And no consumer card will give you a TBW/DWDP so you also don't know what amout of data that card is atleast rated for. Maybe one SD card will last a year because it got a low internal write amplification. Some months later you buy the same model but the manufacturer changed the hardware and the card will die within 3 months.
The biggest problem here is the write amplification of the card itself. Maybe PVE is only writing 300MB of actual data per day. Then you got a card with a factor 100 write amplification and it will actually write 30 GB to the NAND (which you wont be able to see, you only see the 300MB because this is the last before the data enters the SD card). Another SD card might got a factor 200 write amplification and the same 300MB will cause 60 GB of writes to the NAND. Or you are lucky and the card only got a factor 10 write amplification and only 3GB are written.
So you basically can't rely on consumer SD cards. Professional enterprise/industrial SD cards might be better and they also give you a TBW sometimes, but I guess you wont pay 100+ € for a 16 or 32GB card.
The biggest problem here is the write amplification of the card itself. Maybe PVE is only writing 300MB of actual data per day. Then you got a card with a factor 100 write amplification and it will actually write 30 GB to the NAND (which you wont be able to see, you only see the 300MB because this is the last before the data enters the SD card). Another SD card might got a factor 200 write amplification and the same 300MB will cause 60 GB of writes to the NAND. Or you are lucky and the card only got a factor 10 write amplification and only 3GB are written.
So you basically can't rely on consumer SD cards. Professional enterprise/industrial SD cards might be better and they also give you a TBW sometimes, but I guess you wont pay 100+ € for a 16 or 32GB card.
Last edited:
