[TUTORIAL] If you are new to PVE, read this first; it might assist you with choices as you start your journey

[UdoB]

For me an "external" documentation site seems to be useful. The official https://pve.proxmox.com/pve-docs/pve-admin-guide.html is a little bit "dry" sometimes. That one is very good as a reference guide but it does fail to explain some basics in an understandable way for a beginner.

Nearly all of @Dunuin hints I need to agree to. Any recommendation should always bring a user to the save side: it is better to have enterprise grade hardware and redundancy. (But I would prefer ZFS also on a single disk - for its data integrity check and bitrot detection.)

The "safe side" includes hardware usually used in the Enterprise, not at home. Of course your https://portal.habitats.tech/Proxmox+VE+(PVE)/2.+PVE+7.x+-+Hardware does list some valid and interesting devices.

Just one detail popped up to me: as a recommendation you should at least mention ECC. In fact the full text search gives zero hits for this.

(FTR: my own homelab is only ~33% ECC - for the usual reasons.)


Basically I just wanted to give a little bit positive feedback

 

[habitats-tech]

I have actioned the changes as per discussions. Please take a look and let me know if you think they convey the message.

https://portal.habitats.tech/Proxmox+VE+(PVE)/1.+PVE+7.x+-+Introduction#Storage Considerations

 

[Dunuin]

The power-loss protection of an enterprise SSD is by the way not just for data integrity in case of a power failure. Without it, the SSD can't cache sync writes in DRAM cache, as DRAM is volatile and write-cached data would be lost. With a power-loss protection it can run for some seconds on the supercaps and switch to panic mode, quickly dumping all the data from volatile DRAM cache to non-volatile SLC cache before the power runs out.

Have a look at the Proxmox ZFS benchmark paper for 4K sync write performance and you will see how terrible sync write performance without power-loss protection is:

So IOPS performance dropping from 17485 down to 254 (both MLC SATA) just because Power-loss Protection (PLP) is missing, is a very big hit.

 

[habitats-tech]

Thank you to everyone for your feedback. More changes have been actioned (added ECC para).

https://portal.habitats.tech/Proxmox+VE+(PVE)/1.+PVE+7.x+-+Introduction

 

[Nabsta333]

You might want to update this post since you've taken down documentation for PVE 7.4 and generalized it to a PVE 7.x and PVE 8.X

PVE 7.X
PVE 8.X

 

[divStar]

The issue with newbies (like myself) in regards to ZFS (I am not a general Linux newbie, but I've used mdadm RAIDs for over a decade) is not knowing whether I'll need the extra features ZFS is providing. My old NAS survived over 10 years with no issues and I'd argue it was a consumer hardware build (except for WD Red 3TB HDDs).

I am now switching from some frankenstein build (that runs Ubuntu 22.04, non-ECC memory, some i5-3450 CPU etc.) to an enterprise-grade server in a tower (EPYC 8024P, 96GB of DDR5 ECC RDIMM RAM, 4*3.84TB Samsung PM983 U.2 SSDs, 2*480GB WD Ultrastar SA620 SATA SSDs). What I don't know, is, whether I need the ZFS features, because my old server survived without them just fine and still lives on.

Which is why even though I do know some stuff, I follow various tutorials and read various articles about the matter. And that's the point: in the end people have to learn to research enough to make an elaborate decision for themselves.

 

[Dunuin]

The issue with newbies (like myself) in regards to ZFS (I am not a general Linux newbie, but I've used mdadm RAIDs for over a decade) is not knowing whether I'll need the extra features ZFS is providing. My old NAS survived over 10 years with no issues and I'd argue it was a consumer hardware build (except for WD Red 3TB HDDs).

I am now switching from some frankenstein build (that runs Ubuntu 22.04, non-ECC memory, some i5-3450 CPU etc.) to an enterprise-grade server in a tower (EPYC 8024P, 96GB of DDR5 ECC RDIMM RAM, 4*3.84TB Samsung PM983 U.2 SSDs, 2*480GB WD Ultrastar SA620 SATA SSDs). What I don't know, is, whether I need the ZFS features, because my old server survived without them just fine and still lives on.

Which is why even though I do know some stuff, I follow various tutorials and read various articles about the matter. And that's the point: in the end people have to learn to research enough to make an elaborate decision for themselves.

It's not always about if your data is still there after 10 years. Another question is, if the data is still the same after all those years. Without ECC and some checksumming filesystem like ZFS you will never know how much of your data corrupted over all those years. Lets say you stored 100 videos 10 years ago. Those videos are still there and you can open and watch them. But maybe 1 of those 100 videos got corrupted and playback will stop in the middle and the second half of that video isn't watchable anymore. You probably don't watch all those videos regularily so you simply won't notice that data got corrupted. And when you notice it, that data might got corrupted some years ago and all those backups only contain that corrupted file as you usually don't have the money to store your backups for an eternity and you instead overwrite them from time to time.

 

[bossey1]

I think all contributors above have salient points. As a beginner (newcomer to proxmox and ZFS) myself, I very much appreciate tutorials that are tech-light and agree they are valuable as starters. Sometimes the complicated stuff can make you shy away. But I also have fallen into some of the traps highlighted by @Dunuin and @guletz. I like tutorials/how-tos that shield you from the expert in-depth stuff but they must also warn you strongly enough about your risks and where to seek further knowledge. Hopefully, people who are serious about what they eventually wish to achieve can go on to improve their knowledge and systems.

 

[guletz]

But maybe 1 of those 100 videos got corrupted and playback will stop in the middle and the second half of that video isn't watchable anymore. You probably don't watch all those videos regularily so you simply won't notice that data got corrupted.

Hi, and to give another point of view, similar of what @Dunuin tell, from my own experence:
- I make many photos in the past, many of them with many happy moments
- and I have store them on a non zfs storage during many years, and using different disks
- then I want see some of them

Surprise: some of them has multiple strange squares with unusual collors. So after that I decided to use my own mini server with zfs and proxmox.

Now I regret a lot, because I did not use zfs/Proxmox at home insted of a usual NAS.

So in the end is about how much value has your data for you, and also how much money you are willing to pay for it.


Good luck / Bafta !

 

[esi_y]

This is very strange feeling reading the OP, all the parts mentioned there already come with built-in VM solutions in regular (also Debian) repos. GUI comes with cockpit. ZFS comes with Ubuntu. Containers come with Incus (LXD). Clustering comes with pacemaker - and lots of users seem to struggle with this concept either running single node or badly trying to make 2/3-node setups work without adequate networking, etc.

Now I do not want to be a party pooper, but the mentioned selling points are absolutely NOT distinctive for PVE.

I suspect it's just the GUI, for most, after all.

 

[divStar]

This is very strange feeling reading the OP, all the parts mentioned there already come with built-in VM solutions in regular (also Debian) repos. GUI comes with cockpit. ZFS comes with Ubuntu. Containers come with Incus (LXD). Clustering comes with pacemaker - and lots of users seem to struggle with this concept either running single node or badly trying to make 2/3-node setups work without adequate networking, etc.

Now I do not want to be a party pooper, but the mentioned selling points are absolutely NOT distinctive for PVE.

I suspect it's just the GUI, for most, after all.

I'd also probably say, that some (like me) not only don't know what they'd need (pacemaker, Incus etc.), they don't know what the sensible default values are and sometimes they aren't provided. This means one would have to research even more upfront - perhaps to just find out, that it's overkill or is missing a feature.

This is why PVE is very nice. It's not just the UI, it's what I'd call "appliance", the combination of applications, with their respective default configurations. I am sure one could get similar things working with Debian or other distributions - in my case my old NAS is running on Ubuntu Server 22.04 and docker (with docker-compose), not even Kubernetes. And it's been running for over 10 years. It works, but now I would like to use Kubernetes and possibly have my data be preserved in a better way (tbh I wasn't aware of the bit-rot protection and even didn't really know it could happen and didn't think about it).

 

[esi_y]

I'd also probably say, that some (like me) not only don't know what they'd need (pacemaker, Incus etc.), they don't know what the sensible default values are and sometimes they aren't provided. This means one would have to research even more upfront - perhaps to just find out, that it's overkill or is missing a feature.

So you meant to say it is the first thing you heard about, it has a forum like this and it's easy to install without knowing much about it first, I suppose. That's all fair points, but the OP stated different reasons to somehow cherrypick PVE.

I also suspect for lots of users (that do not do the research) PVE actually is overkill.

This is why PVE is very nice. It's not just the UI, it's what I'd call "appliance", the combination of applications, with their respective default configurations. I am sure one could get similar things working with Debian or other distributions - in my case my old NAS is running on Ubuntu Server 22.04 and docker (with docker-compose), not even Kubernetes. And it's been running for over 10 years. It works, but now I would like to use Kubernetes and possibly have my data be preserved in a better way (tbh I wasn't aware of the bit-rot protection and even didn't really know it could happen and didn't think about it).

Do you know of any other options than ZFS for that as of today? One of the issues with appliances is that they kill .. curiosity.

 

[divStar]

Do you know of any other options than ZFS for that as of today?

I haven't checked (so this is not from experience), but I am confident I read about btrfs supporting some measures against bit rot if one uses it as the filesystem.

I also suspect for lots of users (that do not do the research) PVE actually is overkill.

Most definitely! To be honest, I am quite sure, that it's overkill for my use-case, too.
I only need a Kubernetes cluster (on one host, because I don't have nor currently intend to have a second server at home) for my services, perhaps a VM and some storage stuff. TrueNAS would fit this bill, Ubuntu Server would (with some customizations - I currently run this on my old NAS for already about a decade), NixOS with a proper set up would.

The key reason to dabbling into Proxmox is, that it's easy to install and there's really a ton of information about it everywhere on just about every topic and knowing at least some basics might benefit me at work (I am a software engineer, but not too much into Dev(Sec)Ops). I also love plugins like the proxmox-csi-plugin or the proxmox Terraform/OpenTofu provider for automations, because it seems natural to me, that I use a storage provider, that uses ZFS zvols or datasets (not sure which one it is), because I can then save snapshots in an easier manner.

But I am always kneen on learning more and I do consider myself a bloody beginner when it comes to setting up and maintaining a hypervisor or server.

 

[esi_y]

The key reason to dabbling into Proxmox is, that it's easy to install and there's really a ton of information about it everywhere

That often also means wrong information, though.

on just about every topic and knowing at least some basics might benefit me at work (I am a software engineer, but not too much into Dev(Sec)Ops). I also love plugins like the proxmox-csi-plugin or the proxmox Terraform/OpenTofu provider for automations, because it seems natural to me, that I use a storage provider, that uses ZFS zvols or datasets (not sure which one it is), because I can then save snapshots in an easier manner.

I get this, but I never e.g. understand how people come here asking about intricate e.g. PostgreSQL on Proxmox VE and HA, etc., same people who then run k8s instances as VMs, but do not e.g. go around architecting it with CloudNativePG.

But I am always kneen on learning more and I do consider myself a bloody beginner when it comes to setting up and maintaining a hypervisor or server.

I am not hiding it, I have lots of issues how Proxmox communicate, the ZFS (or rather, lack of support of other options), etc.

RAID: https://forum.proxmox.com/threads/proxmox-4-4-virtio_scsi-regression.31471/page-3#post-711018
Firewall: https://bugzilla.proxmox.com/show_bug.cgi?id=5759 (since you mentioned the "Sec")
Release notes quality: https://forum.proxmox.com/threads/o...h-known_hosts-bug-s.137809/page-4#post-713041
Approach to bugs (fixing the wrong thing): https://forum.proxmox.com/threads/etc-pve-pmxcfs-amplification-inefficiencies.154074/#post-701223
And when someone with poor ZFS experience shows up, no reply from staff at all, neither from the ZFS aficionados: https://forum.proxmox.com/threads/lost-all-data-on-zfs-raid10.154843/

They do not even use correct ZFS terms, after all: https://forum.proxmox.com/threads/raid0-1-10-do-not-exist-in-zfs.156143/

So yeah, sorry, for me the hype is not there. So the way this thread started was completely uninformed to me.

 

[esi_y]

And the most recent one, also no one, nothing:
https://forum.proxmox.com/threads/z...o-error-on-import-attempt.156075/#post-712516

 

[esi_y]

The Dunning–Kruger effect is also a problem.

This is completely hidden gem in this thread.