[TUTORIAL] If you are new to PVE, read this first; it might assist you with choices as you start your journey

[habitats-tech]

I have created this post with the hope it can assist newcomers, and not only, with decision making. This post uses acronyms and is a preamble to a full set of articles which are best viewed on my portal, as I provide full online glossary, links, graphics and easy to follow instructions. In any case, I hope it is useful to someone.

Full guide: https://portal.habitats.tech/Proxmox+VE+(PVE)/1.+PVE+7.4+-+Introduction

[[PVE]] is a type-1 [[hypervisor]], which provides infrastructure glue and a [[CLI]] & [[webUI]] to utilise all available Linux virtualisation technologies. It should be viewed as a productivity tool to manage Linux virtualisation, using a [[webUI]]. It is based on [[Debian]] and therefore GNU/Linux-standards-based with plenty of power at the frontend and the backend.

[[PVE]] offers two types of virtualisation:
- [[KVM]] / [[QEMU]]
- [[LXC]]

I highly recommend [[PVE]] over any other hypervisor for the following reasons:
- Based on open systems and technologies
- Utilises Linux standard technologies
- Based on [[Debian]] as the underlying OS and therefore it will run on any x86-64 hardware which supports virtualisation
- Open source
- Actively developed
- [[FOC]] for anyone who cannot afford a license, without software limits and restrictions
- Reasonable licensing cost for any size setup (good tiered system); [[IMHO]] virtually anyone in employment can afford it

Read more on my https://portal.habitats.tech, how to install, configure and use PVE as well as hardware choices for homelab and personal/commercial deployments. My guides are detailed, up to date, and you do not require a rocket science degree to follow.

 

[quanto11]

why is your tutorial better than the official documentation of Proxmox?

 

[Nuke Bloodaxe]

TBH, sometimes wading through the manual can be very intimidating for people. Having a quick guide, especially with "do this for this only thing you want to do" can be very helpful; not everyone wants to do more than the basics.

Ironically, I suspect that a reasonable amount of users would like a section on "how to make Proxmox look pretty"

 

[Dunuin]

Ironically, I suspect that a reasonable amount of users would like a section on "how to make Proxmox look pretty"

Yeah, dark mode was the most requested feature so far.

But I think simplifying stuff too much isn't really helping. Let's say you write a tutorial on how to setup a nextcloud on PVE. Then you skip the fail2ban, monitoring, public private key authentification, firewall, end-to-end encryption, VPN, reverse proxy, AMCE certs, 2FA, DMZ, intrusion detection, backups, security hardening, ... parts because that would go really into the details and you don't want to overwhelm the readers. Then they will only do the limited stuff you wrote, they will think everything is fine and secure, because they don't know it better. But they are actually running a very insecure webservice just waiting to be hacked. Best case they just lose all the data because of missing backups. Worst case the server gets hacked and they lose the data but some stranger also got access to all the files and maybe the police is knocking on their doors because their hacked servers were doing some crimes they are now responsible for, because they negligently ran a server without properly securing it.
If you go into the details and they don't understand 99% of what you are talking about, they at least get a feeling of how less they actually know about what they are planning to do and that they got a long road ahead and that it might need some weeks/months of learning to be able to do the job properly or even drop the whole idea because they are not interested in spending so much time on learning all that and just rent some properly secured cloud storage.

In my opinion, the web would be a way more secure place if everyone would follow this:

• Do I understand 100% of what's done there?
  • Yes: go on, you probably shouldn't have any problems following the tutorial as you already understand it
  • No: Do I want to spend a lot of time (might be months or years...) learning all that stuff to be able to do it properly and to keep everything updated, secure and reliable working over the years?
    • Yes: do all the tutorial steps with a test VM and read some books until you understand 100% of what's done there. Then do it again with your production system.
    • No: Do I have the budget to pay someone with proper knowledge to manage my service?
      • Yes: pay someone to set everything up for you and to keep in updated over the years on your own hardware. Or rent a hosted fully-managed solution somewhere.
      • No: drop the idea or at least do all offline or behind a VPN so you are not a threat to other people

The Dunning–Kruger effect is also a problem. Basically, you think you do something properly and securely, but that isn't the case, you can't know how bad you are doing things, because you are just missing the knowledge to undestand hard it would be to do it properly.
Simple tutorials not talking about the more complex topics would only validate their limited view, that stuff is easy and can be done properly without much effort.
I guess everyone experienced something like this. You learn a new programming language, write your first programs and years later you look back at your old code and you can only facepalm, as you only then realize how bad your code was.

 

[habitats-tech]

The motivation for this kind of guide it to get people up and running as fast and accurately as possible using visual aid. The official documentation is fantastic, but will not get you anywhere fast. Most people will either not read or will most likely not understand on first pass of the manual. To get people to use any product you need to get them up and running fast with no mistakes or ambiguities. Then they have all the time to dive deeper into to nitty gritty.

If you find something missing or inaccurate in any of my documentation I would be happy to correct.

 

[habitats-tech]

A follow up on my earlier post. To clarify, I am not trying to replace Proxmox documentation; I am trying to complement it and point to official resources people can explore on the spot while installing, something missing from reference/manual-style documentation because its intended purpose is, well, reference.

Additionally should we not try to democratise knowledge? Should we not try to expose more people to Proxmox and make it easier for them to get going? I understand the point, is my documentation correct. Well, I believe it is, however, if you think I am missing something or you found errors please point them to me and I will happily make the necessary corrections.

Please bear in mind knowledge acquisition is an additive process. My documentation is trying to build on this fact. I hope it helps new aspiring homelabers and anyone in a smaller setting requiring a solid foundation for their infrastructure.

 

[Dunuin]

But keeping things short without explanation can also be bad. Lets for example have a look at the partitioning part of your Installation guide. The only thing you write is "We will change the filesystem from ext4 (the default) to ZFS. If single disk is used: ZFS (raid0).".
I'm missing very much important information here. For example that by doing that a SSD will die multiple times faster because of the bad overhead of ZFS. And that you shouldn't use ZFS with consumer SSDs that are missing a power-loss protection, because of the low life expectation and often unusable write performance. That ZFS shouldn't be used with a raid card. and that ZFS shouldn't be used with cheap HDDs that make use of shingled magnetic recording. And that by using ZFS you will massively increase needed RAM which is especially important for old/cheap hosts, as ZFS will consume up to 50% of all of the host's RAM, but at least a couple of GBs. And that there is no bit rot protection with a ZFS raid0 as a raid0 got no parity data. There is a reason why ext4 was chosen as the default. ZFS is great, but only when using it with proper hardware (which most people, needing a very basic tutorial, don't got, as there small consumer boxes like a NUC, gaming PCs or old computers are mostly used).

So yes, it will work with a ZFS raid0 and in some situations this might also be a valid choice. But in most situations, outside of a server room, it is not.
If I would know all of this and got an thin-client with a consumer QLC SSD and 8GB of RAM, I would stick with ext4 and not use ZFS.
I would prefer to see an explanation why you have chosen a ZFS raid0 over an ext4 with an explanation of the benefits and downsides of both options, so people can choose the best option that fits their hardware and needs.

That's like writing a tutorial on how to use a gas station and telling everyone to fill diesel in the tank. Then they do that with their gasoline engines and it looks like it is working and after some months the engine will fail. Then they will be upset because their new car is broken and they don't know why. Next time they won't buy a car of that manufacturer again because it failed so fast but actually that all was user error caused by missing knowledge because of simplification. If you tell them to do something, they will just do what you say, without asking if that actually makes sense, as they can't know it better.

You at least need to educate the readers enough so they can make proper decisions on their own, even if that means it will get way more complex and time consuming.

But don't get me wrong. I don't want to badmouth your work. Great that people spend time writing tutorials.
I'm just not a big fan of oversimplified tutorials, as I daily read sentences here like "I don't know why it isn't working...I just did the same thing XYZ did in his tutorial video." and I then have to help them fix very obvious stuff that wouldn't happen when people actually would start reading the PVE/PBS/Debian/ZFS/LVM documentations.

 

[guletz]

HI,

Please bear in mind knowledge acquisition is an additive process.

This is a super super simplification! Human brain does not work like this!

I hope it helps new aspiring homelabers and anyone in a smaller setting requiring a solid foundation for their infrastructure

As a former teacher I can say that your Tutorials, will not help any new users who want to use Proxmox. As @Dunuin write, your tutorials, do not help such kind of persons, maybe on the short term, but sunner or later, they go into problems, because, your explenations or why they must do "that", is a no value for them.

But to be honest, your tutorials, has a value for a person who has medium knowladge about linux, networking, security, and want to have an ideea about Proxmox. For a such person, it is ok.

Somethig like, for example a average user of Vmware user(for many years) think that let try another platform...

Good luck / Bafta!

 

[habitats-tech]

Fair comment and no offence taken. ZFS on consumer SSDs should not be an issue as the disk will not be hammered as an enterprise one. I have done a lot of testing the last 3 years on the most basic NVMe SSDs, even on eMMC and ZFS has not failed once on any RAID-Z version (no ECC RAM was used and the SSDs were all basic NVMe or eMMC. I come from enterprise IT (compute, network, storage) background and I understand what you say, but reality is consumer SSDs and RAM are not as bad as people will have you believe, if sourced from known brands (e..g Crucial and Kingston).

The self healing properties of ZFS are hard to pass by and there are advantages running on ZFS especially if someone will add another disk for redundancy in the future; they will not have to rebuild their system. I was going to address the RAID-Z1 in a future document, which would link to the one you are referring to. Based on your feedback I think I will fast forward the RAID-Z1 document.

Any other comments you are very welcome to share.

 

[habitats-tech]

HI,


This is a super super simplification! Human brain does not work like this!



As a former teacher I can say that your Tutorials, will not help any new users who want to use Proxmox. As @Dunuin write, your tutorials, do not help such kind of persons, maybe on the short term, but sunner or later, they go into problems, because, your explenations or why they must do "that", is a no value for them.

But to be honest, your tutorials, has a value for a person who has medium knowladge about linux, networking, security, and want to have an ideea about Proxmox. For a such person, it is ok.

Somethig like, for example a average user of Vmware user(for many years) think that let try another platform...

Good luck / Bafta!

A person who understands their craft will not look at my tutorials, as they are not meant for that person. I have no intention replicating the fantastic Proxmox manual.

 

[Nuke Bloodaxe]

Building blocks: want to do X? You learn the base building blocks to reach X.
Your tutorial system can do this, and it would help address the "gotchas", as a user will cover it on the way to the destination.
Lots of things work well with a good foundation, which, in all honesty, is what a new user needs.

Also, if a user already knows one of the blocks, they can skip to the next one they don't know.

Note: I do work in education, but as IT, get to train a lot of people and children.

 

[Dunuin]

ZFS on consumer SSDs should not be an issue as the disk will not be hammered as an enterprise one.

I understand what you say, but reality is consumer SSDs and RAM are not as bad as people will have you believe, if sourced from known brands (e..g Crucial and Kingston).

At least here it is an issue. ZFS killed 4 consumer SSD in my homeservers the last year. And two of those are indeed Crucial TLC SSDs not a single year in use. And they all were just used as pure system/boot disks without any writes from guests. I don't really care about always replacing them, as they are cheap and I got everything mirrored, so there is no downtime or data loss. But I really wouldn't use them without redundancy. The Enterprise SSDs fail way less often. Or to quote the FAQ of the offical ZFS Benchmark Paper page 8:

Can I use consumer or pro-sumer SSDs, as these are much cheaper than enterprise-class SSDs?
No. Never. These SSDs wont provide the required performance, reliability or endurance. See the fio results from before and/or run your own fio tests.

If consumer SSDs can still be used with low-demanding workloads can be discussed. But you can't simplify that. Is it a QLC or TLC consumer SSD as QLC SSDs got a way worse write performance (even slower than a good HDD) and durability? Does it got a DRAM cache to be able to optimize async writes for less wear or not? How big is the spare area? Is it planned to run some services using databases doing a lot of sync writes (like a home assistant logging a lot of metrics to a DB) as sync write performance will be terrible without a power-loss protection and without that PLP sync writes will also cause massive additional wear, as without it sync writes can't be cached to optimize writes? Maybe the person just want to write a lot because it should be used to download TBs of "Linux ISOs" torrents per week?
A fact is that ZFS causes a way worse write amplification so it will write way more and therefore wearing the SSD way faster. If that additional wear is toleratable (maybe SSD failing in 5 instead of 15 years...or 1 instead of 3 years) really depends on the single case...
But people should still know about the problems so they can take their own decision.

Based on your feedback I think I will fast forward the RAID-Z1 document.

Please add at least a chapter about the relation between raidz1/raidz2/raidz3, number of disks, ashift and the volblocksize. Every week someone complains that a VM can't be created because the pool is running out of space, as no one reads the documentation telling that the volblocksize needs always (at least using modern 4K disks) to be increased when using any raidz1/2/3 if you don't want to waste a big portion of your space due to padding overhead. Every pool layout needs a different volblocksize, so you would need to create multiple big tables so they could pick a volblocksize from if you don't want to explain how to calculate the padding overhead.
See here for details.

And that you can't expand a raidz by adding single disks without destroying it.

And that a raidz of HDDs always got a terrible IOPS performance, no matter how many disks is consists of, as the IOPS performance won't scale with the number of disks (100 disk raidz as slow as a single disk raid0).

Most people don't get these 3 points.

 

[habitats-tech]

At least here it is an issue. ZFS killed 4 consumer SSD in my homeservers the last year. And two of those are indeed Crucial TLC SSDs not a single year in use. And they all were just used as system/boot disk without any writes from guests. The Enterprise SSDs fail way less often. Or to quote the FAQ of the offical ZFS Benchmark Paper page 8:


If consumer SSDs can still be used with low-demanding workloads can be discussed. But you can't simplify that. Is it a QLC or TLC consumer SSD as QLC SSDs got a way worse write performance and durability? Does it got a DRAM cache to be able to optimize async writes for less wear or not? How big is the spare area? Is it planned to run some services using databases doing a lot of sync writes as sync write performance will be terrible without a power-loss protection and without that sync writes will also cause massive additional wear, as without it sync writes can't be cached to optimize writes? A fact is that ZFS got a way worse write amplification so it will write way more and therefore wearing the SSD way faster. If that additional wear is toleratable (maybe SSD dfailing in 5 instead of 15 years...or 1 instead of 3 years) really depends on the single case...
But people should still know about the problems so they can do their own decision.

ANY disk can fail at any time, even enterprise grade. It could be your SSDs were living in a heat hostile environment. It could also be another 1000 reasons. My point is that ZFS is better than EXT4 even in RAID-Z0, for a variety of reasons. The fact that a single disk can fail is no reason to trash ZFS in favour of EXT4 in home or small business environments. CoW is hard to pass by, assuming thrashing and swapping are non issues. To explain that to my target audience it would be overkill as people start their journey. Just confuse them for no extra benefit. On systems with a single disk, if the disk fails, it is fatal regardless of EXT4 or ZFS. ZFS on light systems will not cause more wear and tear unless those systems run over the edge of their RAM resources. However ZFS will protect you from data corruption issues better than EXT4 can.

I am sorry I strongly believe ZFS is a better choice, even for home and small business environments, over anything else at the moment (especially EXT4), assuming RAM is not over-utilised. BTRFS might be a better choice but it still is in preview stage.

Even IT people cannot fully understand some of the storage choices you describe, let alone my target audience. Which is why I have a section over what hardware I suggest, although my list only covers what I have personal experience with. To explain storage is hard to people with no or limited tech experience. There are too many pieces in the puzzle, which if you try to explain people just switch off. I know because I train IT pros (most frequent joke is just buy enterprise grade - the end).

 

[habitats-tech]

At least here it is an issue. ZFS killed 4 consumer SSD in my homeservers the last year. And two of those are indeed Crucial TLC SSDs not a single year in use. And they all were just used as pure system/boot disks without any writes from guests. I don't really care about always replacing them, as they are cheap and I got everything mirrored, so there is no downtime or data loss. But I really wouldn't use them without redundancy. The Enterprise SSDs fail way less often. Or to quote the FAQ of the offical ZFS Benchmark Paper page 8:


If consumer SSDs can still be used with low-demanding workloads can be discussed. But you can't simplify that. Is it a QLC or TLC consumer SSD as QLC SSDs got a way worse write performance (even slower than a good HDD) and durability? Does it got a DRAM cache to be able to optimize async writes for less wear or not? How big is the spare area? Is it planned to run some services using databases doing a lot of sync writes (like a home assistant logging a lot of metrics to a DB) as sync write performance will be terrible without a power-loss protection and without that PLP sync writes will also cause massive additional wear, as without it sync writes can't be cached to optimize writes? Maybe the person just want to write a lot because it should be used to download TBs of "Linux ISOs" torrents per week?
A fact is that ZFS causes a way worse write amplification so it will write way more and therefore wearing the SSD way faster. If that additional wear is toleratable (maybe SSD failing in 5 instead of 15 years...or 1 instead of 3 years) really depends on the single case...
But people should still know about the problems so they can take their own decision.


Please add at least a chapter about the relation between raidz1/raidz2/raidz3, number of disks, ashift and the volblocksize. Every week someone complains that a VM can't be created because the pool is running out of space, as no one reads the documentation telling that the volblocksize needs always (at least using modern 4K disks) to be increased when using any raidz1/2/3 if you don't want to waste a big portion of your space due to padding overhead. Every pool layout needs a different volblocksize, so you would need to create multiple big tables so they could pick a volblocksize from if you don't want to explain how to calculate the padding overhead.
See here for details.

And that you can't expand a raidz by adding single disks without destroying it.

And that a raidz of HDDs always got a terrible IOPS performance, no matter how many disks is consists of, as the IOPS performance won't scale with the number of disks (100 disk raidz as slow as a single disk raid0).

Most people don't get these 3 points.

I will incorporate your suggestions onto the forthcoming subject of storage. I have another two docs in the works for networking and clustering. All targeting small scale setups.

 

[Dunuin]

My point is that ZFS is better than EXT4 even in RAID-Z0, for a variety of reasons. The fact that a single disk can fail is no reason to trash ZFS in favour of EXT4 in home or small business environments.

I don't say don't use ZFS in home environments. I just say get proper disks if you want to use it. See all the threads of people complaining about high IO delay and unusable write performance when using ZFS with QLC consumer SSDs. It always works like this:
"Beginner: PVE is too slow!"
"Me: Is the IO delay high?"
"Beginner: Yes."
"Me: What disks and storage you got?"
"Beginner: SSD and ZFS"
"Me: What SSD?"
"Beginner: SATA."
"Me: What Model?"
"Beginner: Crucial."
"Me: Thats the manufacturer, what Model?"
"Beginner: BX500."
"Me: Thats a QLC SSD with terrible performance. Get a proper Enterprise SSD."
"Beginner: Buy they cost too much. Can I make it fast by changing some setting?"
"Me: No, you can optimize some stuff to make it a bit faster but slow disk will always be a slow disk."
"Beginner: What should I buy?"
Me naming some proper enterprise SSD.
Beginner ordering and replacing SSD.
"Beginner: Thanks, no problems anymore."

You can search the forums yourself if you don't believe me. You rarely hear people complaing about enterprise SSDs but all the time when using cheap consumer SSDs...especially when using QLC SSDs.

On systems with a single disk, if the disk fails, it is fatal regardless of EXT4 or ZFS.

Yeah. I also wouldn't recommend to use single disks at home if you care about your data or you care about uptime like when using home automation or a virtualized router or a NAS storing important data. But a problem is that those people usually also don't do proper backups. Some may backup the guests, basically no one will use the proxmox-backup-client or tar.gz or whatever to backup the PVE config files. So its helpful when that disks isn't dying that fast. Maybe they then replace it because they need more space before it fails.

ZFS on light systems will not cause more wear and tear unless those systems run over the edge of their RAM resources. However ZFS will protect you from data corruption issues better than EXT4 can.

Did you actually measure that? I personally got a ZFS mirror, a ZFS raidz1 and a single LVM-Thin disk in my main homeserver. All using the same disk models monitored by zabbix logging the smart attributes. When moving a VM between the storages I see big differences in disk writes. On LVM-Thin that VM is only writing a fraction to the disk of what it is writing when using it on ZFS. It's indeed such a big difference, that I moved a few of my guests (that ones that are uncritical like Graylog collecting all logs and storing them in DB or monitoring writing a lot of metrics to DB) from ZFS to LVM-Thin and that basically halved the combines writes. And yes, I also prefer ZFS and use it nearly everywhere, despite the heavy wear.

I know because I train IT pros (most frequent joke is just buy enterprise grade - the end).

Yeah, but most of the time it works. Usually you get what you pay for....
Its hard to find a bad enterprise SSD. But its hard to find a good consumer SSD. They usually only range from horrible to usable while enterprise SSDs range from ok to great. So if you want to generalize, it would be better to recommend enterprise SSDs

 

[habitats-tech]

But keeping things short without explanation can also be bad. Lets for example have a look at the partitioning part of your Installation guide. The only thing you write is "We will change the filesystem from ext4 (the default) to ZFS. If single disk is used: ZFS (raid0).".
I'm missing very much important information here. For example that by doing that a SSD will die multiple times faster because of the bad overhead of ZFS. And that you shouldn't use ZFS with consumer SSDs that are missing a power-loss protection, because of the low life expectation and often unusable write performance. That ZFS shouldn't be used with a raid card. and that ZFS shouldn't be used with cheap HDDs that make use of shingled magnetic recording. And that by using ZFS you will massively increase needed RAM which is especially important for old/cheap hosts, as ZFS will consume up to 50% of all of the host's RAM, but at least a couple of GBs. And that there is no bit rot protection with a ZFS raid0 as a raid0 got no parity data. There is a reason why ext4 was chosen as the default. ZFS is great, but only when using it with proper hardware (which most people, needing a very basic tutorial, don't got, as there small consumer boxes like a NUC, gaming PCs or old computers are mostly used).

So yes, it will work with a ZFS raid0 and in some situations this might also be a valid choice. But in most situations, outside of a server room, it is not.
If I would know all of this and got an thin-client with a consumer QLC SSD and 8GB of RAM, I would stick with ext4 and not use ZFS.
I would prefer to see an explanation why you have chosen a ZFS raid0 over an ext4 with an explanation of the benefits and downsides of both options, so people can choose the best option that fits their hardware and needs.

That's like writing a tutorial on how to use a gas station and telling everyone to fill diesel in the tank. Then they do that with their gasoline engines and it looks like it is working and after some months the engine will fail. Then they will be upset because their new car is broken and they don't know why. Next time they won't buy a car of that manufacturer again because it failed so fast but actually that all was user error caused by missing knowledge because of simplification. If you tell them to do something, they will just do what you say, without asking if that actually makes sense, as they can't know it better.

You at least need to educate the readers enough so they can make proper decisions on their own, even if that means it will get way more complex and time consuming.

But don't get me wrong. I don't want to badmouth your work. Great that people spend time writing tutorials.
I'm just not a big fan of oversimplified tutorials, as I daily read sentences here like "I don't know why it isn't working...I just did the same thing XYZ did in his tutorial video." and I then have to help them fix very obvious stuff that wouldn't happen when people actually would start reading the PVE/PBS/Debian/ZFS/LVM documentations.

FYI: https://portal.habitats.tech/Proxmox+VE+(PVE)/1.+PVE+7.x+-+Introduction

• Use ZFS over any other file/storage system (e.g. EXT, RAID ) on PVEn with adequate CPU & RAM resources. ZFS can be used on featherweight PVE nodes, but the file system will be slower than if using EXT; e.g. on a PVEn with 4 cores, 8GB RAM, single drive eMMC storage avoid using ZFS as it will be slower than using EXT. Read SWAP on ZFS prior to making a decision. As SWAP is set on a per KVM / LXC you can disable SWAP or set it to low value. On a PVEn with fast cores, adequate RAM (e.g. upwards of 32G) and fast NVMe SSD, SWAP is rarely used in KVMs/LXCs, but even when used it presents no real performance issues, unless PVEn is memory starved. It is best practice if SWAP is always setup on an EXT volume or not used at all. If SWAP is not used, it requires PVEn to have enough RAM to serve all workloads without running into memory starvation mode)

 

[habitats-tech]

I don't say don't use ZFS in home environments. I just say get proper disks if you want to use it. See all the threads of people complaining about high IO delay and unusable write performance when using ZFS with QLC consumer SSDs. It always works like this:
"Beginner: PVE is too slow!"
"Me: Is the IO delay high?"
"Beginner: Yes."
"Me: What disks and storage you got?"
"Beginner: SSD and ZFS"
"Me: What SSD?"
"Beginner: SATA."
"Me: What Model?"
"Beginner: Crucial."
"Me: Thats the manufacturer, what Model?"
"Beginner: BX500."
"Me: Thats a QLC SSD with terrible performance. Get a proper Enterprise SSD."
"Beginner: Buy they cost too much. Can I make it fast by changing some setting?"
"Me: No, you can optimize some stuff to make it a bit faster but slow disk will always be a slow disk."
"Beginner: What should I buy?"
Me naming some proper enterprise SSD.
Beginner ordering and replacing SSD.
"Beginner: Thanks, no problems anymore."

You can search the forums yourself if you don't believe me. You rarely hear people complaing about enterprise SSDs but all the time when using cheap consumer SSDs...especially when using QLC SSDs.


Yeah. I also wouldn't recommend to use single disks at home if you care about your data or you care about uptime like when using home automation or a virtualized router or a NAS storing important data. But a problem is that those people usually also don't do proper backups. Some may backup the guests, basically no one will use the proxmox-backup-client or tar.gz or whatever to backup the PVE config files. So its helpful when that disks isn't dying that fast. Maybe they then replace it because they need more space before it fails.


Did you actually measure that? I personally got a ZFS mirror, a ZFS raidz1 and a single LVM-Thin disk in my main homeserver. All using the same disk models monitored by zabbix logging the smart attributes. When moving a VM between the storages I see big differences in disk writes. On LVM-Thin that VM is only writing a fraction to the disk of what it is writing when using it on ZFS. It's indeed such a big difference, that I moved a few of my guests (that ones that are uncritical like Graylog collecting all logs and storing them in DB or monitoring writing a lot of metrics to DB) from ZFS to LVM-Thin and that basically halved the combines writes. And yes, I also prefer ZFS and use it nearly everywhere, despite the heavy wear.


Yeah, but most of the time it works. Usually you get what you pay for....

I agree with all your points. I will add a bit more meat to the bone to clarify things without making it overly complex. That person was most likely running things on a potato, most likely CPU/RAM constrained as opposed to storage I/O issues, or running in IDE instead of AHCI. I have tested ZFS on celeron/pentium systems using eMMC storage only and was surprised how well it works.

 

[Dunuin]

Some while ago I did ~200 ZFS benchmarks on PVE. Here for example is one of the results. Total write amplification ("W.A. total" in the diagram) and total read amplification ("R.A. Total") measured from fixed amount of data written/read by fio inside a VM down to what SMART logged as "NAND_writes" or "host_reads" combined from all disks. Same VM, same disk models, same filesystem, same fio test. This one test was done writing/reading 4K random sync writes/reads by fio to various pools layouts using differnet volblocksizes:

The single disk LVM-Thin (light green in the diagram) for example only got a total write amplification factor of 8.41 while ZFS got a write amplification factor of 35.28 to 62.44. So with this specific workload the same amout of data would cause 4.2 to 7.43 times the SSD wear. Didn't tested a single ZFS disk (as I wouldn't use that without redundancy anyway) but even if you halve the write amplification of the 2 disk mirror (because ZFS only needs to store one copy of everything) what would be 3.37 times the SSD wear compared to LVM-Thin.
Factor 57 total write amplification of a 2 disk ZFS mirror by the way means that after writing 12.6 TBs of 4K sync writes inside a VM, two new "Samsung 870 QVO 1TB" would have exceeded their 360TB TBW rating and warranty would be lost.
This of cause is an extreme example. Using 4M sequential async writes got a way lower write amplification and it doesn't matter that much anymore:

Just wanted to show that it is as always...it depends...

 

[habitats-tech]

Some while ago I did ~200 ZFS benchmarks on PVE. Here for example is one of the results. All the same disks with total write amplification ("W.A. total" in the diagram) and total read amplification ("R.A. Total") measured from fixed amount of data written/read by fio inside a VM down to what SMART logged as "NAND_writes" or "host_reads" combined from all disks. This one test was done writing/reading 4K random sync writes/reads by fio to various pools layouts using differnet volblocksizes:
View attachment 49312
The single disk LVM-Thin for example only got a total write amplification factor of 8.41 while ZFS got a write amplification factor of 35.28 to 62.44. So with this specific workload the same amout of data would cause 4.2 to 7.43 times the SSD wear. Didn't tested a single ZFS disk (as I wouldn't use that without redundancy anyway) but even if you halve the write amplification of the 2 disk mirror (because ZFS only needs to store one copy of everything) what would be 3.37 times the SSD wear compared to LVM-Thin.
This of cause is an extreme example. Using 1M sequential async writes got a way lower write amplification and it doesn't matter that much anymore.
Just wanted to show that it is as always...it depends...

Agreed, however, IMHO real life is a bit different to lab tests. Very useful info though. I still believe the data corruption protection of ZFS and the fact you have a FS + LVM outweighs any "reasonable" wear reservations. For me, within reason, ease of use (e.g. how you expand and work with storage) outweighs any other reservation. From what I have seen people find it easier to work with ZFS storage rather than EXT4; less commands to learn, easier management, plus all other benefits of data storage reliability, especially if you add redundancy. Admin time is more expensive to replacing disks, but I understand your points, which will try to address in my docs.

 

[Dunuin]

Yep, ZFS is definitely worth the extra money for better disks (or crappy disks you need to replace more often). But I think most beginners here don't actually make use of all the features or know how to work with it. They just use it because they want raid and that is the only option in the webUI/installer, so they use it, without doing any research. I would bet most new ZFS users here don't even know what bit rot is, despite that ZFS can detect and correct it. Too bad that we can't create polls in this forum...that would be interesting. Those people also usually don't touch the CLI and you can't really do much with ZFS when limited to what the webUI is offering.
So when they have to choose between bit rot protection and paying less (often) money for SSDs, I would guess most of them would choose the alternative to ZFS that is cheaper.
You really need some ZFS knowledge first, before you can make use and appreciate its features.