Newbie questions

[Blackmyre]

I want to use Proxmox to host a home file server and a few other things, and I'm new to Proxmox. One of my early thoughts was to simply create a directory on the host and for the [Linux] clients to mount it directly from there, either using SSHFS or installing Samba on the host. I got the impression from some forum posts that accessing things on the host directly is bad practice though. I'm not entirely sure I understand why that is (if anyone can point me at an explanation I'd appreciate it) but I'm happy to accept it so I've created a test Container with Turnkey File Server which looks like it should do the job nicely. Now I need to decide on the most appropriate Storage Type, and that's one of the key concepts I'm struggling to understand.

I only have a single data disk in the host (backups will be on a separate machine) and I gather that ZFS would be inappropriate on such a setup. I like the sound of thin provisioning so thought LVM-Thin might be the way to go, but I'm still not really clear about file vs block Storage Types, or why some types only support certain content types. I also think I saw a suggestion somewhere that Directory Storage might be "safer" in the event of some failure scenarios because recovery of data is much easier, not sure whether that's true or not.

So, I'm currently working with a Directory Storage on the host with a bind-mount to the container (and UID/GID tweaked +100,000 to avoid the nobody:nogroup issue). It's working but I wanted to check whether this is an appropriate approach before going too much further. Any thoughts?

 

[Dunuin]

I got the impression from some forum posts that accessing things on the host directly is bad practice though. I'm not entirely sure I understand why that is (if anyone can point me at an explanation I'd appreciate it) but I'm happy to accept it so I've created a test Container with Turnkey File Server which looks like it should do the job nicely.

You can't backup the PVE host without 3rd-party tools. So its way easier to backup and restore a LXC/VM than stuff on the host. And for the same reason you usually want the host as default as possible, so its easier to set it new PVE host up from scratch.
And when using multiple hosts in a cluster it's also not that easy to migrate VMs/LXCs between the nodes when the LXCs/VMs rely on software/folders/hardware on/of the host.

I only have a single data disk in the host (backups will be on a separate machine) and I gather that ZFS would be inappropriate on such a setup.

Depends...in case you got enough RAM (ECC prefered) and a enterprise SSD then ZFS still would be a good idea with a single disk if you care about your data. But with a single disk you only get bit rot detection, no bit rot protection. But I personally wouldn't use single disks. SSDs are so cheap these days (new proper enterprise SSDs with PLP starting at just 36€) and you will save so much work and trouble when mirrorings your disks. SSDs/HDDs are consumables failing sooner or later. With a single disk you lose your data (even with backups at least all the data since the last backup), you got probably many days of downtime where you can't access your services (and this can be terrible in case you virtualize your NAS or router so the whole house can'T access internet or inportant files for days...) and it can be a lot of work to set everything up again from scratch (especially when you care about performance or security, as you than have to edit endless of config files in CLI...). With a mirror a disk can fail and everything will continue working normally and you replace it in a few minutes once you got some free time.

I like the sound of thin provisioning so thought LVM-Thin might be the way to go, but I'm still not really clear about file vs block Storage Types, or why some types only support certain content types.

Block storage is block storage, so without a filesystem and therefore you can't store files/folders on it. But got the benefit that this missing filesystem also doesn't add additional overhead, so a better storage to store VMs virtual disks.

I also think I saw a suggestion somewhere that Directory Storage might be "safer" in the event of some failure scenarios because recovery of data is much easier, not sure whether that's true or not.

If you know what you are doing both are easy. Virtual disks as files are just a bit more noob-friendy, as you got files you could move around with some GUI like WinSCP. When working with ZFS or LVM there are no files, so you need to work with commands ZFS/LVM commands in CLI to do stuff with them.

 

[Blackmyre]

you usually want the host as default as possible, so its easier to set it new PVE host up from scratch.
And when using multiple hosts in a cluster it's also not that easy to migrate VMs/LXCs between the nodes when the LXCs/VMs rely on software/folders/hardware on/of the host.

I'm using a bind mount of a host directory directly onto a mountpoint in the container, so I guess that's a dependency. Is that a bad approach? I don't expect to be doing any clustering but I suppose it would be better to leave the door open.

in case you got enough RAM (ECC prefered) and a enterprise SSD then ZFS still would be a good idea

My current setup has 16GB RAM (not sure if it's ECC), a small SSD as system drive and a and a 3TB HDD for storage.

I personally wouldn't use single disks. SSDs are so cheap these days (new proper enterprise SSDs with PLP starting at just 36€)

Wow - what size are we talking about at that price? I just had a quick search and I was surprised at how much SSD prices seem to have fallen but it still wouldn't be especially cheap to get my current 3TB storage with mirrored SSDs (here in the UK abyway). It doesn't look like SSDs come in 3TB sizes and a pair of 4TB Crucial MX500 would be around £370 (about €430). I probably don't really need that much storage so I could get away with a pair of 2TB BX500 at about £150 (€175). That's still not trivial but I could consider it, though I don't know if you would consider either range (BX500 or MX500) to be enterprise class. Cheapest would be to bung another 3TB HDD in there, though ideally I was hoping to keep the power consumption down.

Block storage is block storage, so without a filesystem and therefore you can't store files/folders on it. But got the benefit that this missing filesystem also doesn't add additional overhead, so a better storage to store VMs virtual disks.

I really need to do some more background reading on block storage. I understand that HDDs are block storage devices at the lower level and that is normally accessed through a file system. I have a bit of trouble imagining how it would be used without some sort of file system. But I've just had a play and created an LVM-thin storage (which I believe is a block type). I then created a volume on that storage (if that's the correct term) in a container, and I was then able to happily create files on it via its mount point. I don't yet understand how it's different from file storage since I seem to be able to store files on both.

If you know what you are doing both are easy

Ah, but therein lies the problem for me

Thanks for the response.

 

[Dunuin]

That's still not trivial but I could consider it, though I don't know if you would consider either range (BX500 or MX500) to be enterprise clas

No. If it doesn't got power-loss protection and no 1+ DWPD thats not a enteprise disk. BX500 is terrible QLC consumer grade SSD and more usable MX500 is TLC consumer grade SSD. BX500 would be terrible with ZFS and MX500 are debatable. Some people use them and think they are fine, some complain about unusable performance.
Also keep in mind that you shouldn't fill a ZFS pool more than 80-90% and that you want some space for snapshotting. If you would want 3TB of usable storage you would need to buy a 4TB disk anyway.
And I woudn`t use HDDs as a VM/LXC storage. HDDs are fine as a cold storage where you store your movies or whatever. But the actual programs, OSs, DBs and so on of your guests should be on SSDs.

Wow - what size are we talking about at that price?

Samsung OEM Datacenter SSD PM883 240GB, SATA

I really need to do some more background reading on block storage. I understand that HDDs are block storage devices at the lower level and that is normally accessed through a file system. I have a bit of trouble imagining how it would be used without some sort of file system. But I've just had a play and created an LVM-thin storage (which I believe is a block type). I then created a volume on that storage (if that's the correct term) in a container, and I was then able to happily create files on it via its mount point. I don't yet understand how it's different from file storage since I seem to be able to store files on both.

Yes, when formating a block device with a filesystem you can also store files on it. But then it is a filesystem at the top level and not a block device. Virtualization and storage isn't easy. You get nested storages and so on.

Bad Example:
Physical disks (block devices) -> HW raid -> raid array (block device) -> ZFS -> Dataset (filesystem) -> qcow2 image file -> virtio SCSI -> virtual disk (block device) -> PV -> LVM -> VG -> LV (block device) -> ext4 (filesystem)

Good example:
Physical disk (block devices) -> ZFS -> Zvol (block device) -> virtio SCSI -> virtual disk (block device) -> ext4 (filesystem)

The more filesystems you are nesting and the more abstraction layers your storage-chain consists of, the faster your SSD will die and the worse will the performance be, the less space is actually usable and the more stuff that could go wrong.

 

[Blackmyre]

Samsung OEM Datacenter SSD PM883

A pair of those (actually PM893, which I gather is the replacement range) at ~2TB would set me back £265 (about €310). That's more than I really want to spend right now on a project whose main purpose is to replace my cheap & cheerful but ageing NAS (1x2TB HDD MyBookLive that hasn't put a foot wrong in 10 years).

Also, it's just occurred to me that I only have two SATA ports available. If I use a pair of SSDs then I guess I'd have to have the system volume and data volumes on the same physical device. I've always kept them separate where possible - is that advisable with Proxmox, or is it perhaps an old prejudice of mine that's no longer relevant?

Thanks for the details and the explanation of block devices & file systems.

 

[Dunuin]

If you primarily want some reliable bulk storage as a NAS I personally would buy at minimum:
- 2x small (lets say 480GB) SATA Enterprise SSD that are big enough to fit PVE + all the system disks of your VMs/LXCs and put them in a ZFS mirror (raid1). Would also be a good idea to use some of that space for "special device" partitions for the HDDs mirror.
- 2x SATA 3,5" HDDs (lets say 4TB and make sure it's CMR and not SMR!) and use them for your NAS storage as a ZFS mirror
- 2x USB HDDs (or even better two SATA HDDs in a caddy for one of those 5,25" hot swappable external drive bays) of a bit bigger size (lets say 6TB) for backups. One always inside your server and the other one stored offsite. You can then rotate the disks every week or month. Remember dthe 3-2-1 backup rule.

But yes, this would require at least 4x SATA ports, 2x 2,5"slots and 2x 3,5" slots or even better 5x SATA ports + 2x 2,5" slots + 2x 3,5" slots + 1x 5,25" slot.

I've always kept them separate where possible - is that advisable with Proxmox, or is it perhaps an old prejudice of mine that's no longer relevant?

Thats useful but not required. Main benefit would be, that you could replace your other disks without needing to reinstall PVE.

 

[Blackmyre]

Thanks, I'll give that some thought for a future build.

[Separate system and data volumes] useful but not required. Main benefit would be, that you could replace your other disks without needing to reinstall PVE.

I was also thinking from the other direction: that it allows a complete reinstall of the PVE without touching the data.

 

[Dunuin]

I was also thinking from the other direction: that it allows a complete reinstall of the PVE without touching the data.

Yes, there it helps too. But you should have proper backups of anything anyway. So only problem with data + PVE on the same disk would be the downtime while waiting to move your data from the backup storage back to the PVE node.

 

[Blackmyre]

True enough.