New Proxmox VE installation from scratch - Suggestions

[lucad]

Hello, I'm going to install a new fresh Proxmox server for my homelab and I'd like to ask you suggestions to configure it in a proper way (I've already a Proxmox server I'd like to reconfigure from scratch).

HW:​

• CPU Intel Core i5 4460 (4c/4t)
• RAM 16GB
• 8x SATA3 ports (2 on the mobo and 4 on a PCIe controller)
• 2x SATA2 ports
• 1x GbE
• ...
• 3x 1TB HDDs (Seagate SkyHawk)
• 1x 2TB HDD (Seagate Barracuda Compute)
• 1x 256GB SSD
• other two SATA2 disks (320 and 160GB)

I'd like to install the following SW:​

• openmediavault as NAS and Docker/Portainer (actually I've installed it in a VM and it does the job pretty well but I'd love to know if it is possible to install it on a LXC without issues and headaches)
• pihole (LXC)
• nginx as reverse proxy (LXC)
• shinobi-cctv (LXC or as docker container in omv)
• jellyfin (actually it is installed in a LXC)
• various ubuntu/debian instances to host webserver and applications (LXC or docker containers in omv)
• .. whatever

I was thinking to install proxmox on the SSD and then create a raid0 of 2x 1TB hdds that is snapshotted to the left 1TB hdd (using rsnapshot or RAID1, I accept tips).. finally use the 2TB hdd for data and shared folders (mounted in omv and shared to every other instance).
Do you think sould be better to use another strategy to achieve it?

I'd also understand which one to use for the disks and the ssd: LVM. ZFS or BTRFS, I'd like to be easy to manage and maybe creating/removing partitions having the possibility to use the snapshots feature of proxmox - actually my ssd is formatted as ext4 using lvm and I don't have this feature enabled.

I'm fine also to passthrough entire disks to omv because I'd like to manage shared folders from it.


Thank you!

 

[Dunuin]

• openmediavault as NAS and Docker/Portainer (actually I've installed it in a VM and it does the job pretty well but I'd love to know if it is possible to install it on a LXC without issues and headaches)

For me that worked fine with PVE6.4 but some people had problems running that as a LXC with PVE7.1.

I'd also understand which one to use for the disks and the ssd: LVM. ZFS or BTRFS, I'd like to be easy to manage and maybe creating/removing partitions having the possibility to use the snapshots feature of proxmox - actually my ssd is formatted as ext4 using lvm and I don't have this feature enabled.

According to the wiki LVM as a block storage doesn't support snapshots. But you could create a LV yourself, format it with for example ext4, mount it and add it as a "directory" storage. On a directory storage you can use qcow2 images that support snapshotting.
ZFS also would allow snapshots but ZFS needs alot of CPU power and RAM and you only got 16GB RAM which isn't that much if you for example need to give 4GB of RAM to ZFS.
Btrfs should also support snapshots but it was only build in some months ago, so I personally would avoid using that, until its better tested.

 

[lucad]

Thank you @Dunuin for your response.

I've made a simple scheme of how I'd like to use my HDDs for the Proxmox's storage. Doest it would work?
Any suggestion is appreciated.

 

[Dunuin]

HDDs make a terrible VM storage because they can't handle many IOPS and that is what you need when running multiple guests.
You possibly wouldn't think of installing a Windows to a HDD these days because it is so much faster with an SSD. Now think of running 10 OSs in parallel of the same HDD doing 10 times the read/write operations. So I would use that 250GB SSD as your VM/LXC storage.

And for HDD#3: You can't store snapshots on another drive. They are always part of the disk that you snapshot. And why do you want to manually mirror its data instead of using a proper software raid1 (for example using zfs or mdadm)?

 

[lucad]

So I would use that 250GB SSD as your VM/LXC storage.

Ok, so basically install proxmox on the SSD and create a partition to be used for the guests (VM & LXC) right?

And for HDD#3: You can't store snapshots on another drive. They are always part of the disk that you snapshot. And why do you want to manually mirror its data instead of using a proper software raid1 (for example using zfs or mdadm)?

Interesting.. can I add HDD#3 (and HDD#4 as RAID1) to the logical group where is installed proxmox and the guests to be able to take snapshots?

 

[Dunuin]

Ok, so basically install proxmox on the SSD and create a partition to be used for the guests (VM & LXC) right?

Jep, but thats already the default. But if you want you can edit it as described here in the "advanced LVM configuration options" paragraph. 16 to 32GB should be fine for your PVE, especially if you want to store ISOs on another disk.

Interesting.. can I add HDD#3 (and HDD#4 as RAID1) to the logical group where is installed proxmox and the guests to be able to take snapshots?

I don't get your question. You can create a software raid1 and then use it to store your ISOs, backups and so on there. Like I already said, if you create snapshots of VMs/LXCs and these are stored on the SSD, then your snapshots will consume the capacity of your SSD. But you can create full backups of your guests and store it on another drive. Or better you setup a Proxmox Backup Server (PBS) inside a LXC/VM or bare metal which will save alot of space because of its deduplication so full backups won't consume more space than snapshots.

 

[lucad]

I don't get your question.

Sorry, I'm attempting to learn things like these for the 1st time

You can create a software raid1 and then use it to store your ISOs, backups and so on there.

I don't understand what do you mean with "software RAID".. actually I've a RAID0 of old HDDs that is managed inside the VM of openmediavault and then shared as SMB/CIFS to proxmox. Probably it is not the best way to achieve this .. so do you mean using a software inside proxmox host that manage the raid of the disks right?

Like I already said, if you create snapshots of VMs/LXCs and these are stored on the SSD, then your snapshots will consume the capacity of your SSD. But you can create full backups of your guests and store it on another drive.

Actually I don't use snapshots (they are not enabled) but only backups. I'd like to have also this feature ready to be used.

Or better you setup a Proxmox Backup Server (PBS) inside a LXC/VM which will save alot of space because of its deduplication so full backups won't consume more space than snapshots.

This is a good point and I like it so probably i'll go in this direction. Do I need to mount the HDDS in proxmox as storages before using these in proxmox backup or I can handle this process directly in proxmox backup VM/LXC ?


Lets say I'd not use PBS, what is the best strategy to use the rest of HDDs ?

 

[Dunuin]

I don't understand what do you mean with "software RAID".. actually I've a RAID0 of old HDDs that is managed inside the VM of openmediavault and then shared as SMB/CIFS to proxmox. Probably it is not the best way to achieve this .. so do you mean using a software inside proxmox host that manage the raid of the disks right?

You can either use hardware raid where you got a dedicated raid controller PCIe card that is doing the raid. Or you use software raid where the raid is done in software by the OS without any additional hardware. And then there is the onboard raid of most mainboard chipsets which is a mix of both combining all the disadvantages.
So yes, raid is done in software. Using the WebUI only ZFS can be used as a software raid. But if you are not afraid of using the CLI and with some linux knowlage you can also manually setup a software raid using mdadm.

This is a good point and I like it so probably i'll go in this direction. Do I need to mount the HDDS in proxmox as storages before using these in proxmox backup or I can handle this process directly in proxmox backup VM/LXC ?

PBS just needs a simple folder to act as its datastore to store the backups there. So a passthroughed disk, bind-mount or NFS/SMB share would be fine. But a SMB/NFS share of your OMV VM wouldn't make sense because you wouldn't be able to access your backups if your OMV VM is down.
After your PBS is installed and running you add your PBS as a backup storage to your PVE. Would also make sense to create a vzdump backup of your PBS VM to one of your HDDs so you can restore the PBS VM from one of the disks so PBS is running again to be able to restore all the other guests from the PBS if your SSDs suddenly dies.

Lets say I'd not use PBS, what is the best strategy to use the rest of HDDs ?

I would add the second 2TB HDD to your OMV and either create a raid1 for redundancy or use it as two single disks so you got more cold storage that could be added to your other guests using NFS/SMB shares.

And I think mirroring HDD#3 and HDD#4 isn't that important. No problem to loose ISOs or LXC templates. Backups of Backups might be useful, but it would be more useful to store this on an external disk for an offsite backup. In case of a fire, theft, lightning strike, water damage and so on a mirror won't help you anything. Better to have a copy of your backups somewhere else.

 

[lucad]

Thank you @Dunuin !!

 

[Dunuin]

actually I've a RAID0 of old HDDs that is managed inside the VM of openmediavault and then shared as SMB/CIFS to proxmox

Keep in mind that you will loose all data if any of the disks dies. So if you for example got 2x 2TB in a raid0 it is more convinient to work with one big 4TB array and you get double the speed but you also double the chance to loose everything. I only would use raid0 if you got daily backups or the data stored on them is unimportant.

 

[lucad]

Keep in mind that you will loose all data if any of the disks dies. So if you for example got 2x 2TB in a raid0 it is more convinient to work with one big 4TB array and you get double the speed but you also double the chance to loose everything. I only would use raid0 if you got daily backups or the data stored on them is unimportant.

Ok, synthesizing:

• on the SSD I'll install proxmox on a smaller partition (16GB) and then install the guests on the bigger one
• 1 TB disk (hdd#3) for backups and snapshots (managed by a PBE LXC)
• 2x 2TB HDDS in raid0 for data, with a scheduled daily backup on a 1TB disk (#hdd4)
• 1 TB disk (hdd#5) mounted as pass-through to the Shinobi container to record videos

How do you suggest to format all these drives? I'd like to use some LVM feature like grouping (ZFS it is not an option because of 16GB of ram).

EDIT:
Because I've a VPS server with wireguard enabled could I install PBS on the vps? I can backup things on the vps drives or use some cloud storage, maybe multiple drives mounted with rclone
I don't like so much to have PBS on the same host of the PVE.

 

[Dunuin]

Ok, synthesizing:

• on the SSD I'll install proxmox on a smaller partition (16GB) and then install the guests on the bigger one

You don't partition it if you select LVM at the installation. It will create a bootloader and boot partition and a big partition for your LVM. Your LVM then got three LVs. One for the OS, one for swap and one for LVM-Thin for your VMs. Should be fine by default but you can lower the root (OS) LV to something like 16 to 32GB to have some more space for your guests.

• 2x 2TB HDDS in raid0 for data, with a scheduled daily backup on a 1TB disk (#hdd4)

Are you sure? Then you got 4.25TB of data that you need to backup on a 1TB disk. Also keep in mind that you might want to keep several backups of the same data.

Because I've a VPS server with wireguard enabled could I install PBS on the vps? I can backup things on the vps drives or use some cloud storage, maybe multiple drives mounted with rclone
I don't like so much to have PBS on the same host of the PVE.

PBS on another machine is always better. Just make sure you got enough storage (atleast 400GB I would say if you want to backup 200GB of guests). And your internet connection shouldn't be be too slow if you don't want to wait hours for an backup to restore. Even with a 100/20 Mbit connection you would need 4 1/2 hours to do a full restore and 22 hours for the initial backup.

 

[lucad]

is it possible to install PBS via apt on ubuntu server (I cannot use Debian right now in my vps) ?

 

[Dunuin]

As far as I know only Debian is supported and if that VPS is a LXC that might be a problem because PBS wants to install its custom kernel.

 

[lucad]

ok so I've tried multiple ways but using the vps to install PBS is not more an option :sadly:
can I run pbs on a mini pc with 2GB RAM and a 4c atom CPU ?

 

[Dunuin]

Yes but 2GB RAM is the absolute minimum.