[SOLVED] Installation raidz-1

[Niko-9]

Hi all,
I am a windows administrator looking to switch to Proxmox, naturally I need help and forgive obvious questions.
I have installed it on a machine with 4 480GB ssd disks and during text installation I selected raidz-1 and left all default just deselect the USB disk, now I have it installed but I can not understand did it make Raid with zfs? All disks are selected and the system is on them also.

this is output from shell zpool status

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-KINGSTON_SA400S37480G_50026B728320DA6F-part3 ONLINE 0 0 0
ata-KINGSTON_SA400S37480G_50026B728320DA1F-part3 ONLINE 0 0 0
ata-KINGSTON_SA400S37480G_50026B728320DA5A-part3 ONLINE 0 0 0
ata-KINGSTON_SA400S37480G_50026B728320DA5D-part3 ONLINE 0 0 0

Also, when I go to server adn Disks And YFS I see this pool but siye is 1.9TB like this is stripe not raid 5?
What did I do wrong?

Thanks a lot
Niko

 

[KevinS]

Please post an output of

zpool list -v

1.9TB would be stripe and not RAIDZ1, but it looks like you setup a RAIDZ1 and not stripe

 

[leesteken]

It definitely looks like a RAIDz1, which is not exactly the same as (hardware) RAID5. It probably shows all space (including the parity) now but each file and virtual disk will take more space than you migh expect (due to padding and parity).
Please be aware that RAIDz1 is terrible for running VMs on because of the low IOPS, massive padding (when volblocksize is small) and huge write amplification (which will wear your consumer drives quickly without PLP) .

 

[gfngfn256]

Try the command zfs list which will show you the actual space (total raw capacity - all parity data).
AFAIK zpool status only shows total raw capacity.

 

[Niko-9]

It definitely looks like a RAIDz1, which is not exactly the same as (hardware) RAID5. It probably shows all space (including the parity) now but each file and virtual disk will take more space than you migh expect (due to padding and parity).
Please be aware that RAIDz1 is terrible for running VMs on because of the low IOPS, massive padding (when volblocksize is small) and huge write amplification (which will wear your consumer drives quickly without PLP) .

which zfs raid you would recommend for VM storage?

Thanks to all for reply!!!

All the best from Serbia

 

[Niko-9]

Please post an output of

zpool list -v

1.9TB would be stripe and not RAIDZ1, but it looks like you setup a RAIDZ1 and not stripe

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 1.73T 4.30G 1.73T - - 0% 0% 1.00x ONLINE -
raidz1-0 1.73T 4.30G 1.73T - - 0% 0.24% - ONLINE
ata-KINGSTON_SA400S37480G_50026B728320DA6F-part3 446G - - - - - - - ONLINE
ata-KINGSTON_SA400S37480G_50026B728320DA1F-part3 446G - - - - - - - ONLINE
ata-KINGSTON_SA400S37480G_50026B728320DA5A-part3 446G - - - - - - - ONLINE
ata-KINGSTON_SA400S37480G_50026B728320DA5D-part3 446G - - - - - - - ONLINE

 

[JensF]

My recommendation is: just don't use ZFS on consumer SSDs!
But there are tons of threads here about that...

 

[gfngfn256]

zpool list -v

As I said (above) try zfs list for actual capacity (-parity)

 

[Niko-9]

My recommendation is: just don't use ZFS on consumer SSDs!
But there are tons of threads here about that...

OK, then raid 10 on btrfs?

 

[JensF]

PVE on BTRFS is still "a technology preview". For home use I would say, just try it. And tell us how is it working.

 

[leesteken]

which zfs raid you would recommend for VM storage?

A stripe of mirrors (like RAID10), which combines redundancy and performance. Your consumer TLC drives will probably last for a few years if you don't run heavy IO.

EDIT: Btrfs is not much different. If you want the most performance run a LVM stripe but make regular backups (but you won't have checksums to detect silent corruption).

 

[Niko-9]

Ok my dear linux Admins, as per your kind advice I am reinstalling in the morning, ZFS raid 10 is the choice, and I will ducktape a “cold” spare ssd on the inside of old trusty Chieftec dragon tower. This is to be a host for a truenas VM which will use pass trough sata controller for storage with 3 WD red 8TB drives. The vm itself will be on ssd with the proxmox. The scenario is for the truenas to use rsync and be backup to my home QNAP and this server will be at wife’s company. I plan to have one more VM with windows server for administration of computers at her company and backup target(maybe), maybe one more win 10 workstation, and maybe Ubuntu file server along the way. Many maybe many possibilities ;-) I am also planing to fit a raspberry pi with a sata disk inside and install proxmox backup server on it so I can backup VMs.. CPU is AMD Ryzen 5 and I put 64GB of ram. I will advise of my progress thank you all for advice and all the best from hot Serbia!
Niko

 

[gfngfn256]

Looks like you are sorted. Maybe tag prefix the thread-title with [SOLVED], (upper right hand corner under title).

 

[leesteken]

I am also planing to fit a raspberry pi with a sata disk inside and install proxmox backup server on it so I can backup VMs.

Proxmox Backup Server (like Proxmox VE) does not support ARM: https://pbs.proxmox.com/docs/installation.html

This is to be a host for a truenas VM which will use pass trough sata controller for storage with 3 WD red 8TB drives. The vm itself will be on ssd with the proxmox.

Does TrueNAS use ZFS? Some WD Red drives are SMR and do not work well with ZFS. Also using ZFS on top of ZFS will be slow and wear out consumer SSDs really quickly.

 

[gfngfn256]

Proxmox Backup Server (like Proxmox VE) does not support ARM

Definitely true. But there are many (unsupported & unofficial) solutions out there. For instance here is a compiled arm64 version. Here is a PBS docker solution for RPI. The list goes on. Don't get me wrong - you've still got tinkering to do & updates are going to be hassle - but it is possible, all in the name of power-saving I guess, @5w.

Another possible solution for the OP, would be to just install a NAS on the RPI, & then just send the backups to that RPI.

 

[Niko-9]

Proxmox Backup Server (like Proxmox VE) does not support ARM: https://pbs.proxmox.com/docs/installation.html

Does TrueNAS use ZFS? Some WD Red drives are SMR and do not work well with ZFS. Also using ZFS on top of ZFS will be slow and wear out consumer SSDs really quickly.

True has does use zfs but the scenario is pretty common as I found numerous how to videos. The zfs for truenas storage is going to be on dedicated drives connected to separate sata controller which I plan to pass trough to the VM. The vm and its data will be stored on proxmox zfs ssd though, I do not know if setup will mitigate the wear problem, what do you think?
This proxmox server is a primary for backup of my home qnap via truenas. Since the cpu and memory allow it I will use it for maintenance and perhaps a file server for my wife’s company, also as a secondary server, not critical. I hope to learn to use proxmox in this setting so it can , maybe, replace hyper v and VMware in my organisation where I am head of IT. I am currently evaluating Rhel open shift and got intrigued by proxmox. After 30 years in windows with just some Linux machines I am not proficient on Linux as on windows and been using hyper v since it was introduced in 2008. It would also be a good solution for some side jobs where licensing is expensive.
Thanks for your thoughts!
All the best
Dragan

 

[Niko-9]

Definitely true. But there are many (unsupported & unofficial) solutions out there. For instance here is a compiled arm64 version. Here is a PBS docker solution for RPI. The list goes on. Don't get me wrong - you've still got tinkering to do & updates are going to be hassle - but it is possible, all in the name of power-saving I guess, @5w.

Another possible solution for the OP, would be to just install a NAS on the RPI, & then just send the backups to that RPI.

Thanks for the links and your ideas! I did not check if it’s available for ARM, that’s phase two ;-) I Liked the complete solution of proxmox, I will scrounge a mini pc which can fit in the tower or disembody an old laptop! I wish to familiarize with proxmox so I need a proxmox backup. All the best from Serbia!
Niko

 

[gfngfn256]

I will scrounge a mini pc which can fit in the tower or disembody an old laptop!

Go for it! That's the real way to master Proxmox!
Happy Proxmoxing!

 

[Niko-9]

Thank you all, I thought you would appreciate the build picture All the best from Serbia! Niko