Help for fist time set up Proxmox

[DK1992]

Hello All, Currently I’m using Dell Optiplex 7060 SFF for lab purpose, I’m concern about set up my storage for promox. My planning is running mirror ( 2 x 240 GB) zfs for promox OS, 1 ssd nvme 512 TB or 1 TB run lvm thin for VMs and lxc, and 1 lvm thin HDD 1 TB for iso image, log, camera , I’m not sure if it good?

So in summary :
2 x ssd 2.5 240 GB for OS ( ZFS - Mirror)
1 ssd Nmve (512 GB 1 TB) for VMs and lxc (lvm thin)
1 x HDD 3.5 (1 TB) for Camera, logs metric(prometheus, grafana, influxdb, zabbix, Centreon)(lvm thin)

Thank you.

 

[Dunuin]

I don't see the point mirroring the system disk when the VM/LXC storage disks and data disks are not mirrored.
I would just get two big enterprise grade SSDs for PVE system + VM/LXC storage + ISOs/templates and mirror them with ZFS. Then your system, as well as the VMs/LXCs, won't lose data and will continue running without downtime when any of the SSDs fails.
I guess you don't care that much about logs, metrics and CCTV, then a single HDD would be fine. But at least here a HDD won't work with zabbix, as Zabbix is writing too much to the MySQL DB that the HDD can't handle all the IOPS. Running heavy writing DBs on HDDs isn't great. So I personally probably just would get two 2TB enterprise grade mixed-worklaod SSDs for everything.
Something like a Samsung PM897 1.92TB, a Micron 5300 MAX 1.92TB or Micron 5400 MAX 1.92TB. Or if you think your not writing that much maybe something cheaper like a Samsung PM893 1.92TB, Solidigm S4510 1.92TB, Micron 5300 PRO 1.92TB, Micron 5400 PRO 1.92TB.

 

[DK1992]

I don't see the point mirroring the system disk when the VM/LXC storage disks and data disks are not mirrored.
I would just get two big enterprise grade SSDs for PVE system + VM/LXC storage + ISOs/templates and mirror them with ZFS. Then your system, as well as the VMs/LXCs, won't lose data and will continue running without downtime when any of the SSDs fails.
I guess you don't care that much about logs, metrics and CCTV, then a single HDD would be fine. But at least here a HDD won't work with zabbix, as Zabbix is writing too much to the MySQL DB that the HDD can't handle all the IOPS.

Actually I'm planning use CCTV on this but my budget is also low.

I guess you don't care that much about logs

Sorry but I as my using also for test (Centreon, Zabbix, Grafana, Prometheus, graylog ..) for monitoring i think this will write a lot and i have read many topic that the ZFS will burn the ssd that why i though may be use VMs and container on LVM Thin may be reduce my cost for SSD (Consumer SSD not enterprise SSD), and recording CCTV on HDD . I'm not sure that it good as i think. Thank you a lot for your help

 

[Dunuin]

Sorry but I as my using also for test (Centreon, Zabbix, Grafana, Prometheus, graylog ..) for monitoring i think this will write a lot and i have read many topic that the ZFS will burn the ssd that why i though may be use VMs and container on LVM Thin may be reduce my cost for SSD (Consumer SSD not enterprise SSD), and recording CCTV on HDD . I'm not sure that it good as i think. Thank you a lot for your help

Yes, ZFS wear the SSDs a lot. thats why you should get proper SSDs. I also started with consumer SSDs, replaced them with HDDs because of the wear (TBW would have been exceeded in below a year) but then everything was unusable slow because DBs and running multiple OSs need too IOPS performance the HDDs can't deliver. So I then replaced the HDDs with enterprise SSDs. Would be cheaper to directly get some proper SSDs instead of replacing the disks all the time because you realize that they can't handle the wear or are too slow.

For CCTV + ISOs + templates a single HDD would be fine. But I still would get some proper enterprise SSDs for system + VMs/LXCs + metrics/logs on a ZFS mirror.

And proper SSDs aren't that expensive anymore. Such a 1.92TB Intel S4510 or Samsung PM893 is only 160-175€. That's not much if you take into account that you need to replace them way less often, they are more predictable and you get magnitudes more of performance when using DBs doing sync writes.
A not well suited Samsung 870 EVO 2TB Consumer SSD would also cost 127€, so you just pay 25% more to get a proper enterprise SSD.

 

[DK1992]

Thank you, but i have a question about the mirror ZFS, ex: i use 2 disk have same size (240gb) and each disk have 600 TBW , so total 1200 TBW, if i using mirror mode it when it write it also write on 2 disk and if that it mean i aslo have only 600 for using ? Thank you

 

[Dunuin]

Thank you, but i have a question about the mirror ZFS, ex: i use 2 disk have same size (240gb) and each disk have 600 TBW , so total 1200 TBW, if i using mirror mode it when it write it also write on 2 disk and if that it mean i aslo have only 600 for using ? Thank you

Yes, everything needs to be written twice, one copy to each disk. When doing sync writes even 4 times. And then there is write amplification. I for example got an average write ampification of factor 20. So those 1200 TB TBW of the combined drives would be reached here after writing 60TB inside the VM, as those 60TB will be amplified to 1200TB of writes to the actual disks. All that overhead of sync writes, ZFS, nested filesystems, journaling, CoW, virtualization will not just add but multiply.

 

[DK1992]

T

Yes, everything needs to be written twice, one copy to each disk. When doing sync writes even 4 times. And then there is write amplification. I for example got an average write ampification of factor 20. So those 1200 TB TBW of the combined drives would be reached here after writing 60TB inside the VM, as those 60TB will be amplified to 1200TB of writes to the actual disks. All that overhead of sync writes, ZFS, nested filesystems, journaling, CoW, virtualization will not just add but multiply.

Thank you, can i ask my last question ? if i using HDD as stored log, does it also impact to write on root pve ?
ex:
My HDD write 1 TB data, on PVE (ZFS SSD) also write 1 TB ?
Thank you

 

[Dunuin]

if i using HDD as stored log, does it also impact to write on root pve ?
ex:
My HDD write 1 TB data, on PVE (ZFS SSD) also write 1 TB ?

I don't understand your question. Are you talking about your logs/metrics you want to write to your DBs (graphana and so on) or are you talking about a "log" in terms of ZFS (SLOG, so ZIL on SSD for your HDD).

 

[DK1992]

I don't understand your question. Are you talking about your logs/metrics you want to write to your DBs (graphana and so on) or are you talking about a "log" in terms of ZFS (SLOG, so ZIL on SSD for your HDD).

i mean the write amplification.
As the i use ZFS on PVE OS, i don't know if i create LVM Thin volume for my new HDD (storing log (for grafana, zabbix, prometheus ), Record Camera so does it also impact the write amplification on ZFS Volume.
EX: if my HDD need to write 1 TB data, it mean the promox ve (ZFS Volume) also write 1 TB ? it mean 1 TB write on SSD and 1 TB HDD?
Sorry english is not my native languague.
Thank you

 

[Dunuin]

When the HDD is using lvm-thin and you write data to it, it shouldn't effect the other SSDs.

 

[DK1992]

Thank you, i thought the PVE OS also affect on this. have a good day bro