Starting up my first server but I have a lot of questions

[Nicholas Phelps]

Sorry if this is not the correct place to post this. If it is not can you please point me in that direction?

Alright. I've been reading all day and I think I've just confused myself more. Maybe I can get some pointers in the right direction.

I know I want to use Proxmox so I can learn more about virtualization and everything I have read has pointed me in this direction. I want to run a plex server, octoprint for 3d printing, downloading software, and a NAS that I can access from outside my network. I also want to be able to add more storage drives to my pool as I go along. I think containers or Docker might be somewhere in there for some of those but I'm not sure.

I have at my disposal: 2 120GB SSD's, 3 4TB NAS HDDS, and 1 4TB HDD that currently has all my files and media on it.

At this moment my thoughts are that I want to install proxmox on one SSD then host my VMs on the second SSD because I can't run my VMs on the same drive as Proxmox. (Right?)

After that I need to set up my storage pool. I believe that is ZFS? Everyone recommends raid6, or in ZFS I think it's called raidz2, for redundancy which I like but I don't want to lose all of my data on my existing drive. Can I set up raid6/raidz2 with the 3 new 4 TB drives, migrate my data, then clear off the old drive and add it to the pool?

Is this even the best configuration for what I want? Expandable storage? Or should I use something else like Snapraid for that? I also know almost nothing about Snapraid other than what I have read today but I'm eager to learn.

Then just for simplicity I would love to just have that single array be my only "Hardrive" that all of my various VM's share with read and write capabilities. Is that possible?

 

[t.lamprecht]

Hi,

At this moment my thoughts are that I want to install proxmox on one SSD then host my VMs on the second SSD because I can't run my VMs on the same drive as Proxmox. (Right?)

You actually can run VMs on the disks you select in the installer. Some avoid this to not place the host system and the VM data on the same backing storage - if one is really corrupt it could make getting out of that situation slightly easier/faster (if you have backups ). But it can also be just fine to host them on the same storage, especially to try out and learn. Most of the services you named ain't probably highly critical, so if you really need to restore the octoprint or Plex from backup a few hours/days ago it is not that bad (as they normally do not host recent user data/documents).

After that I need to set up my storage pool. I believe that is ZFS? Everyone recommends raid6, or in ZFS I think it's called raidz2, for redundancy which I like but I don't want to lose all of my data on my existing drive. Can I set up raid6/raidz2 with the 3 new 4 TB drives, migrate my data, then clear off the old drive and add it to the pool?

For 3 drives you cannot use RAID-Z2, but RAID-Z1 - which allows only one, not two, disks to fail. the Z2 one would need at least 4 disks.

Is this even the best configuration for what I want? Expandable storage? Or should I use something else like Snapraid for that? I also know almost nothing about Snapraid other than what I have read today but I'm eager to learn.

So, you can add disks to ZFS, but you cannot remove them again. And adding is not as intuitive as one would think, i.e., ZFS does not rebalances the pools data on disk addition so that is evenly spread out.
You can workaround this in various ways, but the simplest is to just create a new completely new and separate ZFS pool with your new disks. You can move disks from VMs and CTs between those pools just fine.

Honestly, that and the fact that ZFS implementation in Linux cannot be integrated in the mainline kernel yet are the single drawbacks for it as a local storage system, it's active parity and redundancy checking, lot's of features, cheap snapshotting, .. are most unbeaten by other storage systems.

Then just for simplicity I would love to just have that single array be my only "Hardrive" that all of my various VM's share with read and write capabilities. Is that possible?

You mean a shared storage which the VM can use inside and see the other's data or a storage where all VM can be placed in an isolated way?

 

[Nicholas Phelps]

Thank you so much for all the answers!

You mean a shared storage which the VM can use inside and see the other's data or a storage where all VM can be placed in an isolated way?

What I want to do is just have one storage pool set up then basically have that accessible from any VM that wants it. I also want it to be a NAS so my other physical machines can access it. I think I am going to set up freeNAS and see if that does what I want it to.

 

[Nicholas Phelps]

So another question I have. Do I let FreeNAS set up my data pool or is that done by Proxmox. I am loading FreeNAS up right now and I can see it going two ways (Maybe, since I don't know what I'm doing). Either I make the pool with ZFS on Proxmox and give FreeNAS access to it or I has FreeNAS set up my storage pool then give proxmox/all my other VMs access to it. I don't know what is possible or what is best.

 

[budy]

It really depends on what you actually want. Personally, I'd always put the hypervisor first, which would be Proxmox in this case. The rest of the setup depends on the amount of storage vs. the performance you want to get out of your ZPOOL - and on a single server, I'd always choose ZFS (okay, been a ZFS junky for almost 10 years, so I am a bit biased). When running with spinning disks, I always try to getaway with mirrored stripes, which balance between redundancy and performance. However, you'd need a 4th 4TB drive for that. Then I'd install Proxmox on one of those SSDs and carve out two partitions for ZIL and L2ARC.

Then I'd install FreeNAS in a VM and feed the amount of storage you want.

 

[Nicholas Phelps]

It really depends on what you actually want. Personally, I'd always put the hypervisor first, which would be Proxmox in this case. The rest of the setup depends on the amount of storage vs. the performance you want to get out of your ZPOOL - and on a single server, I'd always choose ZFS (okay, been a ZFS junky for almost 10 years, so I am a bit biased). When running with spinning disks, I always try to getaway with mirrored stripes, which balance between redundancy and performance. However, you'd need a 4th 4TB drive for that. Then I'd install Proxmox on one of those SSDs and carve out two partitions for ZIL and L2ARC.

Then I'd install FreeNAS in a VM and feed the amount of storage you want.

This sounds like exactly what I want. I have my 3 4TB drives set up right now in raidz and SSDs are set up in raid1 and that's where proxmox and VMS are installed. Would like to have a 4th but my 4th has all my data on it that I don't want to lose if I don't have to. I'll have to do some research on ZIL and L2ARC to see what they are and what they do.

Should I be making positions on my OS drive before setting up virtual machines? Or does it automatically do it when I install them? On the harddrive menu screen when setting up the VM it lets you change the drive size. Is that creating a partition or just allocating the space?

 

[budy]

Afair, if you simply leave some space on the volume, you're installing proxmox to, you can also create the needed partitions for ZIL and L2ARC afterwards. You'd be using parted for that and note that the ZIL will only have to be a few GBs, depending on the size of your ZPOOL and the amount of sync I/O which your workload is actually producing, like DB writes.

You can setup your ZPOOL initially using two drives like this:

zpool create <zpool-name> /dev/<disk1> /dev/<disk2>

zpool attache <zpool-name> <disk1> /dev/<disk3>

That shoud create a ZPOO with the name zpool-name (change that to your liking), with two vdevs and then "promote" the first vdev to a mirror by attaching the 3rd disk. Then you could copy over all your data from your 4th drive and attach it to the 2nd vdev just as you did with the first. After the drive has reslivert, you will have a zpool which will look like this:

root@vm:~# zpool status
  pool: vmpool
 state: ONLINE
  scan: scrub repaired 0B in 0 days 01:55:20 with 0 errors on Sun Mar  8 02:19:21 2020
config:

    NAME        STATE     READ WRITE CKSUM
    vmpool      ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sda     ONLINE       0     0     0
        sdb     ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0

Then you can continue with addind a ZIL or L2ARC, if you want.