Help needed > ZFS/storage setup with 3 NVMe and 1 SATA SSDs

[smellycat]

Hi everyone. I'm new with the Proxmox and I wanted to ask for advice with my storage setup.

I have an Intel Nuc and Internal SSD (Samsung_SSD_870_QVO) of 1 TB. I also have 3x1TB NVMe drives (Samsung nvme 990 pro) in NVME SSD Enclosure that support transmission rate on 10Gbps.

Connection to NUC will go over thunderbolt port for 2 drives and 1 drive will go over USB 3.2.

What would be the best option for ZFS creation? Will zfs raid 10 be a good solution?

 

[Dunuin]

I wouldn't use that 870 QVO with ZFS. Search the forum for all the complains about unusable performance of people who try to use ZFS with QLC SSDs, like your QVO.

 

[smellycat]

I also have inside nuc 1TB Samsung nvme 980 Pro but proxmox is installed on it. If I would use it then I would need to install all over again unless there are some better options?

 

[Dunuin]

If getting better suited hardware isn't an option I would create a raidz1 with those three 990 Pros, increase the blocksize of the ZFS storage to 16K before creating anything and use that QVO as a PBS datastore for your backups.

I also have inside nuc 1TB Samsung nvme 980 Pro but proxmox is installed on it. If I would use it then I would need to install all over again unless there are some better options?

In case you already installed PVE on it as a single disk ZFS raid0 you could convert it into a 4 disk raid10 without reinstallation.

 

[smellycat]

In case you already installed PVE on it as a single disk ZFS raid0 you could convert it into a 4 disk raid10 without reinstallation.

How can I do this?

 

[Dunuin]

You first clone the partition table from the existing to the new disks, then sync the bootloader using proxmox-boot-tool. Its basically whats written here in the paragraph "Changing a failed bootable device", just without the "zpool replace": https://pve.proxmox.com/wiki/ZFS_on_Linux#_zfs_administration

Instead, you then use the "zpool attach" command to add partition3 of one of the disks to the existing disk to turn that disk into in mirror (raid1).
And you can turn that mirror into a striped mirror (raid10) by using the "zpool add" command to add partition 3 of the remaining 2 disks as another mirror.

 

[smellycat]

@Dunuin Thanks for the info. I am afraid that I might mess up something by doing this. Is there somewhere step by step tutorial since I am a newby
This is the status when i run command proxmox-boot-tool status

root@pve01:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
E: /etc/kernel/proxmox-boot-uuids does not exist.

This is the current storage

 

[_gabriel]

your current pve is on ext4/lvm
so you can't convert it to zfs.
+ Config has 5 disks.
if you need ZFS and less than 1 day dataloss in case of crash, re install as mirror.
the internal + 1 external.
for the most important data.
second external nvme as single zfs
Same as Dunuin, keep QVO for proxmox backup server, as ext4 datastore.
the remaining drive can be single zfs if you need more storage or can be another backup datastore for proxmox backup server.

To me, for home lab, even pro usage, on not datacenter ssd drives, ext4/lvm + daily proxmox backup server is more reliable, wear level keep cool.

 

[smellycat]

if you need ZFS and less than 1 day dataloss in case of crash, re install as mirror.
the internal + 1 external.
for the most important data.

Hi @_gabriel, when you say
re install as mirror.
the internal + 1 external

Can you give me some info on how I can do this? Thanks

 

[_gabriel]

re install = start over, so need backup if you have already data on current installation.

 

[smellycat]

re install = start over, so need backup if you have already data on current installation.

My question is more about how to install it on internal + 1 external. How can I merge 2 drives during instalation

 

[_gabriel]

within installer,
zfs raid1 = mirror

 

[smellycat]

@_gabriel Will there be a problem if I use the external drive for proxmox installation? (internal + 1 external.)

 

[_gabriel]

external drives isn't recommended, neither for proxmox installation (the pve os) nor for the VM storage.
VM storage is more important than pve os itself when pve os has its own drive,
but in your case with 1TB disks , PVE OS + first VM storage will be on same disk.
zfs over external isn't recommended too, but you havn't choice if you want zfs mirror ...
in case of error / connection , zfs will fast do the correction automatically on the external mirror.

 

[smellycat]

I have one more option, to add additional nvme drive in INTEL NUC, but this drive would be KingSpec 1TB M.2 2242 NVME PCIE SSD. The speed for this drive is 900/1100. Will it make sense to take this and do a mirror with this one instead of the external drive, considering that the speed of this one is much lower than Samsung 980 Pro?

 

[_gabriel]

speed is not the problem, ZFS will wearout any consumer ssd, it require datacenter grade ssd with power protection.
it's not suited for home lab.

 

[smellycat]

@_gabriel Thank you. I finished the fresh installation and backup restore. I did the setup as you mention. Now I am wondering what would be the best option for the remaining 2 external drives since this is the only storage I have. I want to use this for media. Do you have a recommendation what is the best option to do here? Should I combine them as a RAID 0 or this is not smart because of data redundancy, or should I leave them separately?

 

[_gabriel]

it depends on your needs.
keep in mind backup.
the 1TB QVO can full backup the current 1TB zfs raid1 mirror pve + its VM storage.
I would use 1 remaining disk as Lvmthin for VM datastorage and the other for a second proxmox backup datastore as ext4 to able fully backup it.