This is not really helpful very likely, just an observation-comment.
My opinion on the topic of ZFS (or unsupported linux SW raid) and then vanilla Hardware Raid. Or no raid at all.
Raid is important for production generally.
data backups / VM Backups are really important
Part of raid being a good thing is that you now how to use it and how to recover from it.
this means either testing on devtest a full recovery cycle, or on prod when it is pre-prod and tests are ok/doable
proxmox-backup-server (or vanilla scheduled VM Dumps to NFS or other storage, etc, for old-style solution) are great ways to make sure you have your VM data backed up and easily accessible.
the fact that proxmox appliance installer supports a ZFS install config, is nice, but also means there is possible situation where you have setup something and don't really know how it works / or how to support it / or how to recover from a fail.
IMHO. If you are uncomfortable with ZFS then you should not use it. Personally I am not a big fan of ZFS so I don't use it. Just my own preference
I have some proxmox boxes deployed on "Unsupported LInux SW raid" because I am happy / comfortable supporting that / familiar with making it work
Similarly, I have some proxmox boxes deployed on "hardware raid" because generally that is fairly easy to support situations with a failed drive
End of the day, it is important that if you have big problems with proxmox server in production, you should always have the option of
Blow away the server
Clean install to your preferred config / good hardware / etc
Restore VMs from last recent good backup
Life goes on.
In your situation. This could mean possibly
-- attach external USB HDD with various TB of storage
-- manually backup your config files and then the raw VM data files. If you don't have good backups
-- make sure you are good. If you are not sure, use a new set of disks, in your server, and set old disks aside. that way you have a failback in case you screw up and destroy things by accident. ie, Disks are cheap in some situations compared to "oops I destroy my only copy of server VM ha ha oops". ie, disks do cost money, but destroying only copy of a VM by accident can cost more money quite possibly.
-- once you know for sure you are good, get good clean proxmox in place, on a config you are comfortable with and know how to support
-- bring your VMs back to life, ie, this might be "manually create one VM, do nothing with this place holder, now drop in your config and raw VM Disk files to suitable places, so that now your old VM exists and is bootable". once that is done for one VM successfully, rinse and repeat. for all your VM.
-- once all VM are alive and life is good, make sure to setup good backups (Proxmox backup server is recommended) with good suitable easy backups on desired schedule
-- then life moves forward smoothly, and there is less drama hopefully.
End of the day, hard drive failure is inevitable part of server deployment, so it is important that the platform is setup to anticipate this inevitable situation and move through it gracefully. Cars run out of gas, Tires wear out, hard drives fail. We generally avoid running out of gas on the highway, and we generally remove-and-replace bad/worn out care tires before they kill us / with no treads on them.
Sometimes we don't learn how important good reliable backups are until we learn the hard way. ie, lose something that is really important, or end up in an avoidable situation that could have been more more painless, if we had good backups that worked well.
hope this situation is one you can recover from without too much pain loss frustration
and that long term you end up in better place possibly
Tim
