Best partitioning for my setup

[PB24]

Hello everybody,

I'm new to the forum, new to Proxmox and kind of new to linux.

I would like to convert an unused gaming setup into a small home server, I have run some test installation using pve in virtualbox to understand the basics but I'm still wondering on how to use physical drives.

I don't want to replace the hardware since it's pretty powerful and it is sitting there doing nothing (of course I'm going to remove the graphic card cause it'll be pointless) so please don't tell me "first of all trash everything and buy something used with ecc ram" - I know consumer grade stuff can fail but it's an home server to store some data, study and play with virtualization and different OS.

Coming to the storage I currently have 1x 500gb ssd nvm and 1x 4TB hdd.

I would like to add another 4TB internal disk (maybe an Ironwolf or WD red) and use these two storage to keep all my media (around 7TB) and to backup proxmox, VMs and containers - then once per couple weeks or month attach an external drive (8/10 TB) to perform a backup and store it in a safe place - the machine must be as silent as possible and not drain 8000000W

I'm down with the risk of losing data in between backups since I'm not going to load a huge amount daily

Is it possible to achieve something similar?
How should I partition the disks? I would like to have OS and VMs on the ssd and their backup (or snapshots) on the hdd
Which file system should I use? Does it make sense to use ZFS for a single drive pool?
Which file system for the external drive?
Is there an integrated function in proxmox to backup 2 drives into a single bigger drive?

Any idea, comment, suggestion is welcome and truly appreciated

THANK YOU

 

[Dunuin]

(of course I'm going to remove the graphic card cause it'll be pointless)

Its not useless. You can passthrough the GPU to a single VM, so the VM is capable of playing videos or do other hardware accelerated stuff like video encoding for plex/emby. Depending on your CPU your VMs might not be able to even play youtube videos if you don't passthrough a GPU.

so please don't tell me "first of all trash everything and buy something used with ecc ram" - I know consumer grade stuff can fail but it's an home server to store some data, study and play with virtualization and different OS.

Coming to the storage I currently have 1x 500gb ssd nvm and 1x 4TB hdd.

That ssd might be problematic. Keep in mind that consumer SSDs might die within months if confronted with virtualization and server workloads.
You really should monitor the SMART values of that drive and be ready to replace it sooner or later and do backups, especially if you don't want to use raid1.

I would like to add another 4TB internal disk (maybe an Ironwolf or WD red) and use these two storage to keep all my media (around 7TB) and to backup proxmox, VMs and containers - then once per couple weeks or month attach an external drive (8/10 TB) to perform a backup and store it in a safe place - the machine must be as silent as possible and not drain 8000000W

I'm down with the risk of losing data in between backups since I'm not going to load a huge amount daily

Is it possible to achieve something similar?

If you use ZFS as your filesystem (keep in mind that ZFS needs alot of RAM and ECC RAM is recommended) you could use replication to do a incremental sync from one disk to another on block level.
An alternative would be rsync or rdiff as a tool. These work on file level (and therefore should work with every filesystem) and can do a delta copy of your stuff. That way you don't need to copy the complete files again and again. It will only sync the parts of each file what changed.

How should I partition the disks? I would like to have OS and VMs on the ssd and their backup (or snapshots) on the hdd

You can't store snapshots on the HDD. If using copy-on-write filesystems snapshots will always be part of the drive you snapshot. So yes, you can backup your VMs from the SSD to the HDD but snapshots will stay on that SSD.

Which file system should I use? Does it make sense to use ZFS for a single drive pool?

Yes, if you want stuff like compression on blocklevel or want to use replication to backup stuff this is still a valid option. Just keep in mind that ZFS can't protect you against bit rot and can't self repair corrupted data if you are not using some kind of raid with parity.

Which file system for the external drive?

If you wan't to use ZFS with replication I would use ZFS for all drives. Internal and external ones.
If you don't need ZFS I would use LVM thin with ext4 or xfs for the internal drives and plain ext4 or xfs for the external drives.

Is there an integrated function in proxmox to backup 2 drives into a single bigger drive?

No.

 

[PB24]

Its not useless. You can passthrough the GPU to a single VM, so the VM is capable of playing videos or do other hardware accelerated stuff like video encoding for plex/emby. Depending on your CPU your VMs might not be able to even play youtube videos if you don't passthrough a GPU.

The integrated graphic of the cpu should be enough to run a GUI in case I want it - am I right? Intel 630

That ssd might be problematic. Keep in mind that consumer SSDs might die within months if confronted with virtualization and server workloads.
You really should monitor the SMART values of that drive and be ready to replace it sooner or later and do backups, especially if you don't want to use raid1.

That's why I want to backup the content elsewhere, I read some topics here and on the Internet on how to reduce file system workload on the ssd (for instance zfs lz4 compression and shorter recordsets)
I could plan to switch to a better SSD in next month, I see "prosumer" kingston for instance are quite cheap (around 100€) compared to consumer ones - can't afford 1k€ enterprise ssd for an home project

If you use ZFS as your filesystem (keep in mind that ZFS needs alot of RAM and ECC RAM is recommended) you could use replication to do a incremental sync from one disk to another on block level.
An alternative would be rsync or rdiff as a tool. These work on file level (and therefore should work with every filesystem) and can do a delta copy of your stuff. That way you don't need to copy the complete files again and again. It will only sync the parts of each file what changed.

Is the amount of ram proportional to the size of the pool or zfs is going to grab as much as he can? Right now there are 16gb on the system - I think that up to 4 or 6 should be dedicated to VMs, would about 10gb suffice?
rsync looks good also for my last question on backup two drives on a single one, am I right?

You can't store snapshots on the HDD. If using copy-on-write filesystems snapshots will always be part of the drive you snapshot. So yes, you can backup your VMs from the SSD to the HDD but snapshots will stay on that SSD.

Absolutely right, still making a bit of confusion - in few days everything will be clearer for me

Yes, if you want stuff like compression on blocklevel or want to use replication to backup stuff this is still a valid option. Just keep in mind that ZFS can't protect you against bit rot and can't self repair corrupted data if you are not using some kind of raid with parity.

In case of corruption I can grab my external drive and replace corrupted data, right?

If you wan't to use ZFS with replication I would use ZFS for all drives. Internal and external ones.
If you don't need ZFS I would use LVM thin with ext4 or xfs for the internal drives and plain ext4 or xfs for the external drives.

Let's go for ZFS so! Since I read a lot about it and see tons of people loving it and as many who lost sleep with it I'm very curious

No.

Rsync should work. I did try to syntesize my question and ended up being not clear. What I meant is that I use freefilesync for such task under Windows and I was looking for something similar which works in PVE with ZFS or any other supported file system.


Thank you for you help Dunuin

 

[Dunuin]

The integrated graphic of the cpu should be enough to run a GUI in case I want it - am I right? Intel 630

You need one GPU for the host itself so the host can boot and if you need to directly manage the host. For example if you need to change some BIOS options or Proxmox crashed and you can't use SSH or the WebGUI anymore so connecting a display and keyboard is your only option to access the host.
And NO VM can use your hosts GPU. Each VM needs its own dedicated GPU passed through to the VM or everything needs to be rendered by the CPU and that is super slow. For example...if I want to stream a youtube video inside a Win10 VM the decoding of that video stream will utilize 4 full CPU cores up to 100% and the video is still stuttering. Thats because no VM will be able to use your hosts GPUs hardware acceleration.

That's why I want to backup the content elsewhere, I read some topics here and on the Internet on how to reduce file system workload on the ssd (for instance zfs lz4 compression and shorter recordsets)
I could plan to switch to a better SSD in next month, I see "prosumer" kingston for instance are quite cheap (around 100€) compared to consumer ones - can't afford 1k€ enterprise ssd for an home project

Prousmer SSDs are as bad as consumer SSDs. You want a SSD with powerloss protection (prosumer SSDs don't offer that) and a high duarbility (a prosumer SSD might be 2 times more durable than a consumer SSD but a enterprise SSD might be 30 times more durable). If you take into account that a enterprise SSD will live 30 times longer, so you don't need to replace the SSD that often, buying an enterprise SSD might be the cheaper option. There are enterprise SSDs that aren't that expensive. For example you get a Intel S4610 960 GB enterprise SSD (with powerloss protection and a TBW of 5800 TB) for around 210€. A Samsung 860 Pro 1TB prosumer SSD (no powerloss protection and TBW is only 1200 TB) costs 192€. So you only pay 10% more but it will live almost 5 times longer.
If you really need NVMe there are also options.

Is the amount of ram proportional to the size of the pool or zfs is going to grab as much as he can? Right now there are 16gb on the system - I think that up to 4 or 6 should be dedicated to VMs, would about 10gb suffice?
rsync looks good also for my last question on backup two drives on a single one, am I right?

The more RAM you use for the ARC the better. Minimum should around 4GB. Rule of thumb is 4GB + 1GB RAM per 1TB of raw storage. So if you want to use 0,5+4+4+8 TB of raw storage the recommended ARC size would be something like 20GB of RAM. It will work with less RAM but the more you reduce the ARC size, the slower your drives will be. By default ZFS will use 50% of your hosts RAM for caching but you can change that.
You could start with 8GB for the ARC and look if that is fast enough for you. 16GB of RAM is really small for a hypervisor.

In case of corruption I can grab my external drive and replace corrupted data, right?

No, that is the point. Without ZFS, raid and ECC RAM you will never know if a file is corrupted or not. Your host will think everything is fine because it can't detect bit rot. If a file got corrupted 1 year ago but you never noticed it, you only backup a already corrupted file so you don't got any undamaged backups. If you don't keep your backups forever you will sooner or later overwrite your healthy files with corrupted ones.

ZFS can detect and report bit rot most of the time. But without ECC RAM ZFS can't detect all currupted data so you will still loose a small portion due to bit rot. And if you are not using raid1/raid5/raid6 ZFS can only detect but not fix the corrupted data.

 

[PB24]

You need one GPU for the host itself so the host can boot and if you need to directly manage the host. For example if you need to change some BIOS options or Proxmox crashed and you can't use SSH or the WebGUI anymore so connecting a display and keyboard is your only option to access the host.
And NO VM can use your hosts GPU. Each VM needs its own dedicated GPU passed through to the VM or everything needs to be rendered by the CPU and that is super slow. For example...if I want to stream a youtube video inside a Win10 VM the decoding of that video stream will utilize 4 full CPU cores up to 100% and the video is still stuttering. Thats because no VM will be able to use your hosts GPUs hardware acceleration.

This point was not clear to me - I use vmware daily for work on my company laptop due to policy restrictions for some software and never experienced any slow issue with videos so I was thinking proxmox would have no problem as well.

Prousmer SSDs are as bad as consumer SSDs. You want a SSD with powerloss protection (prosumer SSDs don't offer that) and a high duarbility (a prosumer SSD might be 2 times more durable than a consumer SSD but a enterprise SSD might be 30 times more durable). If you take into account that a enterprise SSD will live 30 times longer, so you don't need to replace the SSD that often, buying an enterprise SSD might be the cheaper option. There are enterprise SSDs that aren't that expensive. For example you get a Intel S4610 960 GB enterprise SSD (with powerloss protection and a TBW of 5800 TB) for around 210€. A Samsung 860 Pro 1TB prosumer SSD (no powerloss protection and TBW is only 1200 TB) costs 192€. So you only pay 10% more but it will live almost 5 times longer.
If you really need NVMe there are also options.

With prosumer I was pointing at something like kingston nvme dc1000b which has plp - it's around 110/120€ on the web.
I mixed price tags of 500gb with the 8tb enterprise class ssd - I was looking to not use hdd at all so I did a mess. Yeah this upgrade totally worth it

The more RAM you use for the ARC the better. Minimum should around 4GB. Rule of thumb is 4GB + 1GB RAM per 1TB of raw storage. So if you want to use 0,5+4+4+8 TB of raw storage the recommended ARC size would be something like 20GB of RAM. It will work with less RAM but the more you reduce the ARC size, the slower your drives will be. By default ZFS will use 50% of your hosts RAM for caching but you can change that.
You could start with 8GB for the ARC and look if that is fast enough for you. 16GB of RAM is really small for a hypervisor.

I should run test for this as soon as the system is up and running.

No, that is the point. Without ZFS, raid and ECC RAM you will never know if a file is corrupted or not. Your host will think everything is fine because it can't detect bit rot. If a file got corrupted 1 year ago but you never noticed it, you only backup a already corrupted file so you don't got any undamaged backups. If you don't keep your backups forever you will sooner or later overwrite your healthy files with corrupted ones
ZFS can detect and report bit rot most of the time. But without ECC RAM ZFS can't detect all currupted data so you will still loose a small portion due to bit rot. And if you are not using raid1/raid5/raid6 InZFS can only detect but not fix the corrupted data.

Setting up a raid1 with 2x8T without ecc would fix at least part of this problem, right?
I don't really want to replace all the hardware

 

[Dunuin]

With prosumer I was pointing at something like kingston nvme dc1000b which has plp - it's around 110/120€ on the web.
I mixed price tags of 500gb with the 8tb enterprise class ssd - I was looking to not use hdd at all so I did a mess. Yeah this upgrade totally worth it

The DC1000B is a enterprise grade SSD too. But I always hear people not to buy the DC1000B but the DC1000M instead. The DC1000M is the a little bit more durable and way faster U.2 version. You can connect a U.2 drive to your M.2 slot but you need a M.2 to U.2 converter and U.2 cable.

DC1000B 960GB: constant 4K reads/writes: 199.000/25.000 IOPS; 1095 TBW; 192€
DC1000M 960GB: constant 4K reads/writes: 400,000/125,000 IOPS; 1681 TBW; 226€ (+20€ for M.2 to U.2 converter +15€ U.2 MiniSAS cable)

Setting up a raid1 with 2x8T without ecc would fix at least part of this problem, right?
I don't really want to replace all the hardware

- ZFS without raid1 without ECC RAM = detection of most bit rot, but can't do anything against it (but a scrub task can tell you whats damaged so you can restore it from backups if that backup is older than your previous successfull scrub task. But you always need to run a scrub task before a backup to verify that you don't backup corrupted data.)
- ZFS with raid1 but without ECC RAM = can detect most of bit rot and repair it
- ZFS with raid1 and ECC RAM = can detect all bit rot and repair it

 

[PB24]

The DC1000B is a enterprise grade SSD too. But I always hear people not to buy the DC1000B but the DC1000M instead. The DC1000M is the a little bit more durable and way faster U.2 version. You can connect a U.2 drive to your M.2 slot but you need a M.2 to U.2 converter and U.2 cable.

DC1000B 960GB: constant 4K reads/writes: 199.000/25.000 IOPS; 1095 TBW; 192€
DC1000M 960GB: constant 4K reads/writes: 400,000/125,000 IOPS; 1681 TBW; 226€ (+20€ for M.2 to U.2 converter +15€ U.2 MiniSAS cable)

I ordered 480gb intel's one, best bang for the buck tho (110€). It's my home experiment so even if it's going to be a little bit slower it's not going to be a problem.

- ZFS without raid1 without ECC RAM = detection of most bit rot, but can't do anything against it (but a scrub task can tell you whats damaged so you can restore it from backups if that backup is older than your previous successfull scrub task. But you always need to run a scrub task before a backup to verify that you don't backup corrupted data.)
- ZFS with raid1 but without ECC RAM = can detect most of bit rot and repair it
- ZFS with raid1 and ECC RAM = can detect all bit rot and repair it

That's great!! My mind is completely clear now

Thank you so much Dunuin, I hope to learn enough from my experiment to give something back to the community a day.