New to Proxmox: Best practice for OS install

[Smvb64]

Hello all,

I have built a new server to replace my aging Raspberry Pi running OMV and would like to ask what would be the best practice for installing the OS and VMs

I currently have an SSD and an M2 drive. My original plan was to use the SSD to store the OS and VMs and the M2 to be a scratch drive.
I'm not sure if I should be using ZFS or LVM-thin for the SSD as I heard ZFS can wear them down. I would also like to set up the SSD to be in mirror mode.
Should buy another SSD or could I use an HDD for mirror mode for the SSD?

My main use for the server is for filesharing and dockers to use plex and other apps just for home use and to share with friends.

Thank you guys for your help.

Update, I'm thinking of doing this
LVM-thin for the SSD (boot and VM) in mirror mode

ZFS for the Harddrives for data and storage


Best,
Dan

 

[Dunuin]

I currently have an SSD and an M2 drive. My original plan was to use the SSD to store the OS and VMs and the M2 to be a scratch drive.
I'm not sure if I should be using ZFS or LVM-thin for the SSD as I heard ZFS can wear them down. I would also like to set up the SSD to be in mirror mode.

Out of the box PVE only supports ZFS as software raid and you are right, that ZFS can kill your SSDs, if you got no enterprise grade SSDs, in no time with specific workloads that for example do a lot of sync writes...like DBs. And ZFS on its own got a lot of overhead. And then there is virtualization and nested filesystem which add overhead too. And that overhead isn't adding up, it is multiplying, causing serious write amplification. There is still the option to use the raid of the mainboards disk controller or to install a Debian with mdadm software raid and convert that into a PVE later.

Should buy another SSD or could I use an HDD for mirror mode for the SSD?

That would be a terrible idea. Write performance of a mirror is only as fast as the slowest disk. So a HDD would slow everything down.

My main use for the server is for filesharing and dockers to use plex and other apps just for home use and to share with friends.

Thank you guys for your help.

Update, I'm thinking of doing this
LVM-thin for the SSD (boot and VM) in mirror mode

ZFS for the Harddrives for data and storage

I personally would get two SATA enterprise SSDs (second hand ones with good SMART values if you are on a tight budget), mirror them with ZFS, and use them for the system, as well as VM/LXC storage and as special devices to boost the performance of the HDDs.
Then a bunch of HDDs just as cold storage (for your videos that don't need great IOPS performance), maybe in a ZFS raidz1 (for 3-5 HDDs) or raidz2 (for 6-10 HDDs).
Another option would be to buy a HBA card, pass that through, using PCI passthrough, into a NAS VM (for example TrueNAS if you want ZFS or OMV if you want a simple NAS) and then create the HDD raid inside the VM and share it to the PVE host and other guests using SMB/NFS shares.

 

[Smvb64]

Out of the box PVE only supports ZFS as software raid and you are right, that ZFS can kill your SSDs, if you got no enterprise grade SSDs, in no time with specific workloads that for example do a lot of sync writes...like DBs. And ZFS on its own got a lot of overhead. And then there is virtualization and nested filesystem which add overhead too. And that overhead isn't adding up, it is multiplying, causing serious write maplification. There is still the option to use the raid of the mainboards disk controller or to install a Debian with mdadm software raid and convert that into a PVE later.

That would be a terrible idea. Write performance of a mirror is only as fast as the slowest disk. So a HDD would slow everything down.

I personally would get two SATA enterprise SSDs (second hand ones with good SMART values if you are on a tight budget), mirror them with ZFS, and use them for the system, as well as VM/LXC storage and as special devices to boost the performance of the HDDs.
Then a bunch of HDDs just as cold storage (for your videos that don't need great IOPS performance), maybe in a ZFS raidz1 (for 3-5 HDDs) or raidz2 (for 6-10 HDDs).
Another option would be to buy a HBA card, pass that through, using PCI passthrough, into a NAS VM (for example TrueNAS if you want ZFS or OMV if you want a simple NAS) and then create the HDD raid inside the VM and share it to the PVE host and other guests using SMB/NFS shares.

Thank you for your post. I am starting to understand Proxmox now and how it works.

Unfortunately, I do not have the budget for enterprise SDD. I currently have WD RED SSD.

For testing, I may keep the one SSD for now in LVM-thin just to play around with it and eventually move into the ZFS mirror setup as you mentioned.

Looking into SMB, I wanted to make an LXC container with Turnkey Linux Fileserver an LXC container for Plex, and finally one for docker.
Fileserver will just be primarily for my family, Only 4 users

The only question is will the SMB share work between containers?

Thank you for your help

Best,
Dan

 

[rason]

I would recommend using LVM-thin for your SSD setup. While ZFS can be a great option for data storage, it can be a bit too resource-intensive for use on an SSD, which can cause it to wear down faster. By using LVM-thin, you can create a logical volume on your SSD that you can use to store your OS and VMs. You can also set up the SSD in mirror mode, which will provide redundancy in case of failure.

As for your M2 drive, you can use it as a scratch drive for temporary data. Scratch drives are typically used for high-speed data access, so your M2 drive should be well-suited for this purpose.

If you want to set up mirror mode on your SSD, you can use another SSD or an HDD for the mirror. It's generally a good idea to use the same type of drive for the mirror as the primary drive, so if you're using an SSD for the primary, it's best to use another SSD for the mirror. This will ensure that the drives have similar performance characteristics and will help to prevent potential issues.

Overall, using LVM-thin on your SSD and setting it up in mirror mode should provide a good setup for your server. ZFS can be a great option for your hard drives, as it provides powerful data storage and management capabilities.

 

[Dunuin]

The only question is will the SMB share work between containers?

Depends:

VMs (most secure): can only use SMB/NFS shares
Privileged LXCs (worst security, so better not used for stuff that you port-forward): can use SMB/NFS shares + bind-mounts
Unprivileged LXCs (less secure): can only use bind-mounts (so you would need to indirectly access it by mounting the SMB/NFS share on the PVE host and then bind-mounting that mountpoint from the host into the unprivileged LXC)

 

[Smvb64]

I would recommend using LVM-thin for your SSD setup. While ZFS can be a great option for data storage, it can be a bit too resource-intensive for use on an SSD, which can cause it to wear down faster. By using LVM-thin, you can create a logical volume on your SSD that you can use to store your OS and VMs. You can also set up the SSD in mirror mode, which will provide redundancy in case of failure.

As for your M2 drive, you can use it as a scratch drive for temporary data. Scratch drives are typically used for high-speed data access, so your M2 drive should be well-suited for this purpose.

If you want to set up mirror mode on your SSD, you can use another SSD or an HDD for the mirror. It's generally a good idea to use the same type of drive for the mirror as the primary drive, so if you're using an SSD for the primary, it's best to use another SSD for the mirror. This will ensure that the drives have similar performance characteristics and will help to prevent potential issues.

Overall, using LVM-thin on your SSD and setting it up in mirror mode should provide a good setup for your server. ZFS can be a great option for your hard drives, as it provides powerful data storage and management capabilities.

Thank you for your reply! I am going to end up going this route. ZFS does sound great but for my home setup, just going to use LVM-thin with the SSD I currently own. I will use ZFS for the SMB shares on the HDD

By chance do you have any guide or link on how to make an LVM-thin mirror? I'm not too sure how I would set it up currently.

Thank you,
Dan

 

[Smvb64]

Depends:

VMs (most secure): can only use SMB/NFS shares
Privileged LXCs (worst security, so better not used for stuff that you port-forward): can use SMB/NFS shares + bind-mounts
Unprivileged LXCs (less secure): can only use bind-mounts (so you would need to indirectly access it by mounting the SMB/NFS share on the PVE host and then bind-mounting that mountpoint from the host into the unprivileged LXC)

I see, thanks for the heads up.
For Plex/torrents I will set up a VM since that will be shared and LXC for the docker that are just for the home

 

[rason]

Thank you for your reply! I am going to end up going this route. ZFS does sound great but for my home setup, just going to use LVM-thin with the SSD I currently own. I will use ZFS for the SMB shares on the HDD

By chance do you have any guide or link on how to make an LVM-thin mirror? I'm not too sure how I would set it up currently.

Thank you,
Dan

Great, I'm glad you've decided to go with LVM-thin for your OS and VMs. It's definitely a good choice for a home setup like yours.

Here are a few links to the Proxmox documentation that might be helpful:

• The Logical Volume Manager (LVM) page in the Proxmox documentation explains the basics of LVM and how to use it to manage your storage: https://pve.proxmox.com/wiki/Logical_Volume_Manager_(LVM)
• The Storage: LVM Thin page in the Proxmox documentation provides instructions on how to use LVM-thin to create thin-provisioned logical volumes: https://pve.proxmox.com/wiki/Storage:_LVM_Thin

I hope this helps, and good luck with your setup!

 

[Dunuin]

@rason: What LVM mirror are you talking about? LVM Thin ontop of HW/onboard raid or a real LVM mirror (so mirroring done by LVM in software)?
Because the PVE installer doesn't offer to use a LVM mirror and you can only use LVM-Thin with a single disk (where a "single disk" also could be a HW raid1 array presented to the installer as a singel disk). Not sure if it would be possible to mirror that single disk after an PE installation. Or if it is possible at all to use a LVM-Thin pool mirrored on two PVs.

 

[Smvb64]

@rason: What LVM mirror are you talking about? LVM Thin ontop of HW/onboard raid or a real LVM mirror (so mirroring done by LVM in software)?
Because the PVE installer doesn't offer to use a LVM mirror and you can only use LVM-Thin with a single disk (where a "single disk" also could be a HW raid1 array presented to the installer as a singel disk). Not sure if it would be possible to mirror that single disk after an PE installation. Or if it is possible at all to use a LVM-Thin pool mirrored on two PVs.

New to RAID but I see my mobo "MSI B560I" does have support for mirror mode in the BIOS level. I'm thinking of just going this route for now
and then have the ZFS raid for storage using proxmox