Storage Layout Question for a beginner

[matthias-mw]

Hi together,

first of all I am quit new to proxmox and have some really low level questions. At the moment I am running a Linux server with different applications as homeserver, because I needed an upgrade anyway I decided to use proxmox and split things up a bit.

I would like to setup the following VMs for the beginning:
- ubuntu-server
- nextcould
- true-nas
- jellyfin mediaserver

Before i start, I need some best practice advice how to layout my storage.

Physically I have a 1TB M2.ssd and 2x 2TB HDDs. The 2HDDs are full with my data and were in a HW-Raid1 system In my old server.

1. Question: I guess there is no way to keep those 2 HDDs and use them directly in true-nas (with all the data)
2. Question: I installed proxmox to the ssd because it's a new disk. But the is now plenty of space left. Is it good to have the VMs and the backups on the same ssd as proxmox itself?
3. Question: Where is a good place to store backups?
4. Question: On my linux server I used lvm, so I could every now and than shift some free storage from e.G. nextcloud to the mediaserver. How does this works in proxmox with zfs?
5.Question: Is it possible or good practice that 2VMs use the same storage? Or is a net-share the preferred solution?

As you can see I am stuck at the very beginning. I have an additional SSD on hand and consider to by 2 new 4TB HDDs to not jeopardize my data...

Thanks for any advice how to layout the storage.

Matthias

 

[Dunuin]

1. Question: I guess there is no way to keep those 2 HDDs and use them directly in true-nas (with all the data)

No, TrueNAS only uses ZFS, disks need to be wiped and ZFS shouldn`t be used on top of a hardware raid and shouldn't be used with SMR HDD (so check that those are actually CMR).

2. Question: I installed proxmox to the ssd because it's a new disk. But the is now plenty of space left. Is it good to have the VMs and the backups on the same ssd as proxmox itself?

It's fine. But dedicated disk for PVE system would be preferable if money, slots and ports doesn't matter.

3. Question: Where is a good place to store backups?

See the "3-2-1 backup rule".
At least store them on a dedicated disk. And preferable a second copy offsite.
Setting up a PBS is way better than using VZDump for backups. Faster backups, incremental backups, encryption, integrity checks, ransomware protection, syncing between multiple PBS (useful for offsite backup!), waaaaay less space wasted.

4. Question: On my linux server I used lvm, so I could every now and than shift some free storage from e.G. nextcloud to the mediaserver. How does this works in proxmox with zfs?

First keep in mind that ZFS shouldn`t be used with consumer SSDs without a PLP. Especially not with QLC SSDs.
And with ZFS you can use thin provisioning. If you don't want to do the annoying resizing of partitions and filesystems you could just create very big virtual disks with more space than you probably need so you don't have to resize them in the first place. But a bit annoying with LXCs or when not using PBS, as this will then slow down backups.
Shinking of virtual disks is by the way not supported. You can only increase the size via CLI and webUI. And you then have to manually extend the LVM/partitions/filesystems/... inside the guestOS too.

5.Question: Is it possible or good practice that 2VMs use the same storage? Or is a net-share the preferred solution?

What storage? Mounting the same virtual disk in parallel with two VMs will corrupt the data on it. Usually you would work with NFS/SMB shares.

 

[EllyMae]

My first impression is that you don't have enough storage, except as a platform for learning.

Please let us know what processor you're using and how much memory.

1) I don't think you'll be able to migrate the HDDs to Proxmox unless you keep the hardware RAID setup. (Just curious: Are those 2 HDDs mirrors?) If your RAID is an HBA, you should be able to pass the HBA through Proxmox into a TrueNAS VM.

2) I like putting the Proxmox VE system (along with, perhaps, the Proxmox Backup Server system (not data)) on a separate device. But, if you must, there is an installation option to leave extra space on the installation target device. Then, you can put a partition in that area and use it as you please. Just think really hard about what that means in a disaster recovery situation.

3) In the trunk of your car? I'm only half joking. That's what I do. Best case, have a separate system with a backup server and plenty of storage (i.e., a physically separate pool). My compromise is to have Proxmox Backup Server installed as a VM and have physically separate devices for its ZFS storage pool. For offsite, I have separate devices. I define them as an additional PBS datastore and then do a PBS sync to copy the most recent backups into it. PBS is not hateful in terms of resource usage, except it is fairly I/O-heavy. It does a good job with deduplication. If you want duplication for redundancy, then you have to set up an additional datastore and sync them.

4) In Proxmox, both LVM and ZFS are available. You may have noticed that, by default, Proxmox installs in LVM. I have more experience with ZFS, so I may not give you the best answer to this. At the raw storage level, you can share your ZFS zpools any way you wish. But Proxmox will not let you share the same storage allocation (filesystem) directly across multiple VMs. At least, not in the GUI. That might be possible with containers, though; I personally don't have much need for them. To share filesystems, there are two ways that I know of. One is to set up a VM with an NFS server and install the NFS client to all the systems that want to use it. I use Open Media Vault because it greatly lessens the learning curve for setting up an NFS server. I don't use TrueNAS because AFAIK it only works with ZFS. The other way, I hear (but haven't tried), is to use ZFS's built-in NFS and/or CIFS (SMB) server. So far, I'm not convinced that that's the best way to go. It seems like trying to shoehorn in something that is not directly related.

5) In general, operating systems don't kindly share filesystems. If they had simultaneous access to the same filesystem, it would be destroyed quickly. Storage pools can be shared across VMs, however.

My opinion is that since you have an even number of like-sized devices, you could mirror them with either LVM or ZFS. If you want absolute maximum capacity (and, therefore, minimum fault tolerance), you could put everything in an LVM volume group and keep frequent backups. Or, you could use them as separate devices, each with its own filesystem. To use ZFS's RAID options, you'll need a minimum of three like-sized devices. Also, when choosing a filesystem, consider the advantages and disadvantages of checkpoints and on-the-fly error detection and recovery given your use cases.

These are my opinions based on my limited knowledge. On behalf of OP, I welcome additional input. Asking nicely.

 

[matthias-mw]

My first impression is that you don't have enough storage ...

After one day of playing around with proxmox I came to the same conclusion

Please let us know what processor you're using and how much memory.

As my old server has worked fine for 10Years (yes the hdds are 8years as well) but died now (possibly Motherboard) I decided for a upgrade to enhance performance. I choose AMD 4650G on a Asrock X570s board with on seagate firecuda 530 m2 with 1TB and 32GB ram. The other 2x 2TB HDDs and one old 256Gb SSD are from the old server, thats why I need to recover the data from that disks.

The new system should work for the next 10years, so I am willing to spend some more money if necessary and I will buy 2 more 4TB HDDs

First keep in mind that ZFS shouldn`t be used with consumer SSDs without a PLP.

Thank's didn't know that

In general, operating systems don't kindly share filesystems. If they had simultaneous access to the same filesystem, it would be destroyed quickly. Storage pools can be shared across VMs, however.

Understood, NFS is the way to go. What do you mean by "Storage pools can be shared" ....

To use ZFS's RAID options, you'll need a minimum of three like-sized devices.

I don't get that. I thought ZFS would be fine with 2x HDDs in mirror configuration?


With all that in mind I am thinking of the following Layout of Physical storage:
- 256GB SSD for Proxmox system, isos etc
- 1x 1TB SSD for the VMs and the filesystems
- 2x 4TB HDDs in mirrored config for all my standard data and standard backups
- 1x 2TB HDDs in a tray to backup important backups every 3month to be stored in an other location

Does this make sense?