PVE edge use case

[H25E]

First I will try to explain our use case:

- Remote industrial hardware
- Only two drive ports, 1x Sata + 1x M2
- HMI + DAQ functionality (Human Machine Interface and Data Acquisition)
- Deployed on hard to access areas, with no IT personal deployed in locally
- Uptime is important

Now I will try to explain why we are considering PVE for this use case:

- Solid and reliable (Debian based)
- Native ZFS (once of the very few distros that offer official ZFS support)
- Proxmox boot tool (ZFS would provide rootfs redundancy, and this would seal the deal providing ESP redundancy)

Why?

1. The fact that this HW is in remote hard-to-reach areas without having IT personal there makes it much easier to ship a new drive when one fails and have non-tech people physically change it. It's people that you would trust with their hands but never behind a keyboard and much less on a grub or dracut emergy shell. So a RAID is something that would give us a "progressive failure" scenario that would allow people on site give maintenance to the system without data loss and without uptime loss (planned maintenance)

2. Uptime is important. RAID increases uptime.

As HW raid is not an option (no PCIe expansion) and ZFS seems to be the only feasible software approach to an unattended software RAID (BTRFS refuses to boot from a degraded array and seems dangerous to run it with degraded flag perpetually), ZFS seems to be our only option. And as we don't really 100% trust community openZFS support for a production environment like this, we remain only with proxmox and NixOS as the only two distros with official ZFS support. And proxmox having proxmox-boot-tool for ESP redundancy, then proxmos feels like the ultimate software drive redundancy linux distribution.


The bad?

Since we don't need virtualization for anything in this application (or at least that's what we naively think) and sometimes it could be even counter productive like when using DAQ hardware with drivers that need direct kernel/HW access (it could work with hardware passthrough and things like that but it's not officially supported). Same for the graphical output with the HMI with iGPU passthrough... So our potential approach is to completely gut the PVE working phylosophy and install everything on host instance without using any of the virtualization features. This obviously sound as bad as it can, but what would be the real world consquences of that?

So the actual questions are:
1. What are the real consequencues of use PVE like that?
2. Can we uninstall all the virtualization related services and software?
3. Does any of this make any sense?

 

[_gabriel]

as we don't really 100% trust community openZFS support for a production environment like this, we remain only with proxmox and NixOS as the only two distros with official ZFS support.

There is only OpenZFS for Linux

Get datacenter flash disk to prevent replace them.

 

[H25E]

There is only OpenZFS for Linux

Get datacenter flash disk to prevent replace them.

1) I know. When I said community support, I mean distributions were no official support for openZFS exists, but there is a community project that keep an up to date openZFS repo compatible with the main repo, like opensuse or others.

2) I need to use temperature resistant drives because it's a fanless system. Still...

 

[Johannes S]

I might be wrong but doesn't Ubuntu support ZFS? If I remember correct that's the exact reason why Proxmox Server Solutions GmbH uses an Ubuntu kernel together with Debian bookworm as base for their products

 

[louie1961]

2. Can we uninstall all the virtualization related services and software?
3. Does any of this make any sense?

If you really want to remove all of the virtualization services and software, why not just use Debian with the Proxmox kernel? Seems safer and less "kludgy" to me.

Does it make sense? No, for a number of reasons. First, why start with Proxmox and basically remove 90% of the Proxmox functionality just to get ZFS. You will break something and spend a lot of time chasing your tail to solve it. Second, I wonder if you can really set up a ZFS mirror with an M.2 drive and a SATA drive. Third, your hardware doesn't sound like it is suited for the task at hand and you are trying to solve that issue with kludgy software fixes.

The real answer, if uptime is important, is to get the correct hardware for the job, and make this a situation where if the hardware fails, anyone would be able to unplug the failed device and plug in a replacement, You don't want to be changing drives in the field. Just replace the entire box.

 

[alexskysilk]

I wonder if you can really set up a ZFS mirror with an M.2 drive and a SATA drive.

You can. its not ideal but it would work.

The real answer, if uptime is important, is to get the correct hardware for the job,

THIS.

 

[H25E]

Thanks you all for your ansers! I will try to respond everybody:

I might be wrong but doesn't Ubuntu support ZFS? If I remember correct that's the exact reason why Proxmox Server Solutions GmbH uses an Ubuntu kernel together with Debian bookworm as base for their products

I thought that Ubuntu started to drop ZFS on 22.04, but seems like they only deprecated zsys and ZFS support was more or less maintained.

If you really want to remove all of the virtualization services and software, why not just use Debian with the Proxmox kernel? Seems safer and less "kludgy" to me.

Becuase I didn't think of this at all. Would be this an interesting approach? But then shouldn't I need to install ZFS mod and utils from proxmox repo so kernel and ZFS versions get sync? Also what about proxmox-boot-tool?

Second, I wonder if you can really set up a ZFS mirror with an M.2 drive and a SATA drive.

Won't make sense performance-wise obviously, but you would still get all the redundancy benefits, which is the most important part here.

The real answer, if uptime is important, is to get the correct hardware for the job

For sure. But sometimes you need to work with what you need to. Also, we are not looking for a bulletproof solution here, simply slightly improve reliability and uptime in the most consumable and failure prone component (drives).

make this a situation where if the hardware fails, anyone would be able to unplug the failed device and plug in a replacement

We have this also for critical projects. But still, this doesn't provide the same uptime than a RAID, which would allow to replace the box on non-operative hours in a planned mode. So both solutions are not exclusive nor equivalent.

You don't want to be changing drives in the field.

It won't be an issue in our case.

Thank you all for your feedback, much appreciated.

I could try Ubuntu then, seems like a more feasible option but I didn't know about. However, I'm still unsure what would be the actual drawbacks of using proxmox like that. It wouldn't even be necessary to uninstall all the proxmox software on top, ignoring it and close the firewall would be probably enough.