I don't have any issues with ZFS before, except it eats huge amount of RAM. I don't have any comparision with LVM or any other system on Proxmox. But I am using ZFS for years on all proxmox servers, PBS too. Maybe it's because Proxmox installer can use RAID10 on ZFS only. There is no option to install on SW RAID 10 LVM (I know I can install Debian on LVM RAID10 and There install Proxmox)
Lost all data on ZFS RAID10
- Thread starter jhr
- Start date
I really meant the question on its face value - do you need any of the features ZFS provides?
I came to PVE rather late and I was surprised it's using ZFS this way, I have been using ZFS for long back on BSD, but always for storing dead data, not really primarily full of ZVOLs. When I first installed PVE, even I wanted to see how it performs with ZFS, I was completely surprised it also put the ZFS on root, it's not a filesystem I would choose for OS, let alone hypervisor.
PVE is generally strange (from where I came), it pushes ZFS, but then suggests not to really use replicas, but shared storage, it pushes ZFS, but PBS does not take advantage of the send/receive and keeps dirty blocks map of its own. And yes, it even calls striped mirror "RAID 10" which ZFS docs avoided for ages.
Yes and then be aware of: https://bugzilla.proxmox.com/show_bug.cgi?id=5235
Well, you puzzled me. I don't like have a data on non raid disks or partitions. If I have a HW raid, I am using it,if not I am using SW LVM RAID. The same scenario I choose when start using PVE. What better/other options do I have?
This will be a controversial piece of advice, but since you ALREADY got this experience with ZFS, in case you ACTUALLY need a copy-on-write filesystem, you might as well test BTRFS. Before someone links me some 10 years old post about upset RAID6 installs, if you are going to do mirror (stripes), I believe you are safe. There might be glitches with things like quotas, but I did not have it lose my data.
That said, I personally go for something like XFS on LVM and if I wanted RAID and a software one definitely mdadm. But this is just me because using LVM for e.g. mirroring feels so odd, i.e. not enough battle tested. Then again, if you have experience with it and it's working well for you, why not (LVM all the way).
As you discovered, you anyhow need backups (maybe more frequent) and you might as well get some normal performance on the HGSTs without over-the-top filesystem.
Funny bug with mdadm ... but I always have swap configured but disable usage ... because it doesn't make sense to me to overload a host until swapping as when I use a host/vm that way I still can use 5-10 years older hardware as well and don't need the actual hardware when down-performing is the way to go !
Yes, just it's good to keep it in the back of one's mind (since PVE does not bother to e.g. put in a simple check and recommend different chache).
SWAPs are generally funny, this reminds me - have you tried them on a ZVOL?
I'm coming out of the workstation and fileserver world, so all file storage is my world ...
Get a bunch of hardware which should be discommissed and we said stop for building a virtualization plattform for any testing and luckily we decided to use proxmox pve 1.5 years before. Then get in contact with zvols (block storage) ... but in my opinion I feel better with a filesystem to use for vm's/lxc's even if it's has overhead against block storage while is has filesystem (<-any of) features (readahead, zfs snapshot usage) and so even if using zfs I prefere a vm/lxc in a zfs dataset instead of in a zvol - but I think this is still my history owed. And there is told that zvols are double fast as a dataset what does it matter if I use xfs which is even double fast as zfs so I don't loose anythink
Get a bunch of hardware which should be discommissed and we said stop for building a virtualization plattform for any testing and luckily we decided to use proxmox pve 1.5 years before. Then get in contact with zvols (block storage) ... but in my opinion I feel better with a filesystem to use for vm's/lxc's even if it's has overhead against block storage while is has filesystem (<-any of) features (readahead, zfs snapshot usage) and so even if using zfs I prefere a vm/lxc in a zfs dataset instead of in a zvol - but I think this is still my history owed. And there is told that zvols are double fast as a dataset what does it matter if I use xfs which is even double fast as zfs so I don't loose anythink
To be fair, I used to run LXCs (not PVE) on ZFS datasets before, all was fine, but then stopped even that. ZVOLs are overrated and (actually) buggy, it got buried by now, but there was even a thread where someone's migration was failing all the time, then I suggested setting REFRESERV on EFI (!) disk and it was suddenly okay. To this day, no one analysed what made the difference. QCOW2 has nicer snapshotting for the purpose than any COW generic layer. Storing that on a ZFS dataset ... feels weird to me though.
And there is told that zvols are double fast as a dataset what does it matter if I use xfs which is even double fast as zfs so I don't loose anythink
It just feels somewhat strange about COW filesystem for anything that should perform. And in terms of checksum'ing, well, I will know when something is amiss if it's a mirror anyhow.
What if your house caught fire?
when defining your production environment, you really need to define what your disaster recovery criteria are. This is irrespective of what hardware or configuration you use. If you have a maximum downtime criteria, consider a replicated remote environment for proper BC (business continuity.) as others mentioned, this is above and beyond your backup strategy.
As for this particular issue. are you interested in troubleshooting it?
As storage we use 99% nfs to our 4 pve's, there doing reflink snapshots which then were copied to one of the pve's additional storage inside as fallback. That's quiet nice we are all really happy with. Easy, fast and working ... what do anybody need more ?!
As I wrote before, I can accept that PVE system hdd crashed, but I do not understand why it wiped all zvols on zpool. If there were some power outage and my server goes down, than maybe I can accept that ZFS could be broken. But that server is on UPS with other server, which are fine.
I am not saying these are factually incorrect, but with hindsight, coming to this forum, people take lots of myths for granted, e.g. ZFS being some sort of mystically dependable filesystem (i.e. must have been why even PVE took it for the popular option - in terms of support), when it is in fact not.
I find it bizzare that PVE install puts BTRFS on some "experimental" pedestal, but there's no warning next to ZFS. I have never even had old-fashioned ext4 fail in ways that ZFS can. When it does, ALL is gone. Maybe the official docs should go emhpasise that.
PS Sometimes people even ask what has better performance, i.e. ZFS or something else - where did they get such inference from (to even attempt to compare it) other than that even hypervisor supports it as if it was some performant choice.
I used to backup script on all PVEs (https://github.com/DerDanilo/proxmox-stuff/blob/master/prox_config_backup.sh), but when I started to use PBS I don't use that script anymore (edit: I lied my crontabs still have line with that skript and all my PROD PVE configurations are stored to NFS share). Could someone explain to me, where is the problem with configs on PBS ? I can restore all configs from all backups on PBS ?
Last edited:
Does PBS back up the hosts now? I apologise if I bring some more confusion, I have not looked for over 10 months now, but last time I checked it, it was backing up VMs alone.
I briefly checked the script you linked, it will make some backup, but what I do not like about these (in general) is that they e.g.:
Code:
tar --warning='no-file-ignored' -cvPf "$_filename9" --one-file-system /etc/pve/.
tar --warning='no-file-ignored' -cvPf "$_filename2" /var/lib/pve-cluster/.So you get some content of /etc/pve at that moment, but it cannot read it atomically, if there's some operation going on at the time, I wonder if you can end up with files that do not match (e.g. mid-migration).
The /var/lib/pve-cluster will backup config.db, but unfortunately while running, which SQLite does not really guarantee to be non-corrupt. You might catch it middle of performing WAL checkpoint. So that's why I put up the "tutorial" with the SQL dump at the least. But of course it's just a stub, would need timer/cron ...
This part I cannot answer, I do not want to criticise something that might have evolved in the meantime.
Yes there are many reasons to use ZFS, especially with proxmox, and yes ZFS was designed with Virtualisation, SSDs etc in mind.
I would higly recommend ZFS on a Single Node Proxmox Setup (in a cluster you might want to use Ceph).
As always, have backups. ProxmoxBackupServer is your friend
Irrelevant in this case, since the data loss is in the striped mirror (aka raid10)