Proxmox setup question

[SuperMiguel]

Good Evening i just installing proxmox, on 3 ssd in a raidz-1 configuration, i did it using the installer, i will using this raid of course for proxmox installation, and for installing vms

Now i have 4 HDD that i want to add to proxmox for data whats the best way to do this? Do i create a new pool and configure them to raidz-1

Also after this is set an done, are there any advantages to add another ssd for cache?

 

[SuperMiguel]

bump

 

[guletz]

Hi,

Do i create a new pool and configure them to raidz-1

Yes, it is possible. You must create this new pool from command line. Then you can use http proxmox interface to add this new pool. But remember that after this the both pools will use the same allocated memory(by default is 50% RAM). For raidz1, you can get the same IOPs as for a single HDD. If you need more IOPs, you can have something like RAID10, but you will loose the usable storage space.

Also after this is set an done, are there any advantages to add another ssd for cache?

This depends on your storage usage and the quantity RAM that you have. Adding zfs cache will eat more RAM, but it is possible to be usefull(especially if you use lxc containers with the same distribution).

 

[SuperMiguel]

I only have homeassistant and a samba containers installed so only two containers and they dont require much. Right now i only have 8gb, server supports up to 64gb i would be adding memory little bit at the time

 

[guletz]

I only have homeassistant and a samba containers installed so only two containers and they dont require much. Right now i only have 8gb, server supports up to 64gb i would be adding memory little bit at the time

In this case, it is not ok to add a zfs cache SSD. Instead add more RAM. Until then, try to limit the memory used by zfs at 2-3 GB.

 

[SuperMiguel]

In this case, it is not ok to add a zfs cache SSD. Instead add more RAM. Until then, try to limit the memory used by zfs at 2-3 GB.

Will do thanks one last question... For my main raidz-1 where i installed proxmox and all my vms.. is it better to leave it as raidz-1 or should i replace those 3 ssd with 2 bigger ones. Basically 2 240gb in raid 1 final storage 240gb 3 120gb in raidz-1 final storage 240gb

I have 3 120gb in raidz-1 but someone wondering if im better off returning those and get 2 240 and set them in raid1? I know standard raid 5 is way faster than raid 1, is it the same with zfs raidz-1 vs zfs raid 1

 

[guletz]

1. raidz1 is good for capacity, but IOPs is like for a single disk
2. mirror, is bad for capacity(you will loose 50% capacity), but the IOPs is like 2x single disk(but can be coverted if you need in the future in a let say raidz1, raidz2, or zfs raid10 if you have more SSD)
- for SSD, a bigger size is better for performance(as a general rule)
- in the near future, I can guess that it will be difficult to get anothers 120G SSD, but it will be possible to find 240G SSD

In the end, if you ask me, I will go to 2x 240 Gb SSD(zfs mirror)...

 

[SuperMiguel]

How is this possible? Raidz1 write in stripping with parity where raid 1 needs to write in mirror mode... shouldnt raidz1 have a higher read/write speed meaning higher iops?

 

[guletz]

shouldnt raidz1 have a higher read/write speed meaning higher iops?

higher read/write speed is-not-the-same-with higher iops.

Most of the time, any mirror is faster(in term of read/write speed AND iops) then raidz1/2/3. On any kind of raidzX, all the drives will need to wait untill the last data and parity blocks are written succesfully on the disks, then it could go to the next disk transaction. On a mirror, the same data can be written on the disks indepently. It is true, that SSD drives have higher IOPs, so the differences (mirror/raidzX) could not be so big. Also, a lots of others details/fact can have a influence/impact (like RAM, sequential operations, random operations, pool fragmentation, latency, and so on)

 

[SuperMiguel]

So what should one do with 3 drives then? Use 2 for raid-1 and other for cache? I dont care about size (gb) i only care about speed and protection in case one drive fails

Basically i only need about 60gb worth of containers so i could live with only 2 of the 120gb ssd just not sure if i can use the 3rd one for something usefull

 

[guletz]

For protection only, the best between mirror and raidz1 is for sure the mirror. Only for protection, you can have a 3 way mirror (3x ssd/disks) or more. In this last case, any read operations are faster (compared with 2 way mirror). And you can loose any 2 ssd/hdd from 3 and you still have your valuable data !

Another tip for 3 way mirror : from time to time, you can remove only one SSD, from the server, and do a fstrim, then put back in the server.


zfs cache for a exclusive SSD pool is not useful.

 

[SuperMiguel]

For protection only, the best between mirror and raidz1 is for sure the mirror. Only for protection, you can have a 3 way mirror (3x ssd/disks) or more. In this last case, any read operations are faster (compared with 2 way mirror). And you can loose any 2 ssd/hdd from 3 and you still have your valuable data !

Another tip for 3 way mirror : from time to time, you can remove only one SSD, from the server, and do a fstrim, then put back in the server.


zfs cache for a exclusive SSD pool is not useful.

Remember i have another hdd pool 4x 1tb drives

But i really like that 3 way mirror idea!!! i think thats the winner.. Thanks!!!!

 

[guletz]

Ok Miguel, then good luck !

... use a small amount of your SSD for pool (partitions ) and the rest for zfs cache for the hdd pool(5 Gb for example) so you can have 3 x 5 Gb for cache only