Are two PVE servers a bad idea?

[Ramalama]

I did mess up pve upgrade 7 to 8 last week (it was not dramatic, but took me a day to resolve).

I was able to get by without my VMs, but...

I can not stomach not having the internet alone with my VMs.

That's actually why I started this thread - to understand how to build a bulletproofed pve infrastructure.

TBH, bullet-proof pve infrastructure does't exist.
Everything can go wrong from time to time, especially on upgrading to a whole new release and im not talking about pve, im talking about a distro switch from bullseye to bookworm....
Which upgrades 800-1000 Packages...

The only thing that you can do to make it stable as possible is never touching or changing anything on the node itself, means no custom drivers, no custom packages on the pve host and so on.
Just everything in containers or VM's....

But anyway, this is the perfect example why we do run 2 nodes at least with opnsense/pfsense in HA...
If one pve node fails, you still have internet and so on.

 

[Dunuin]

I did mess up pve upgrade 7 to 8 last week (it was not dramatic, but took me a day to resolve).

I was able to get by without my VMs, but...

I can not stomach not having the internet alone with my VMs.

That's actually why I started this thread - to understand how to build a bulletproofed pve infrastructure.

Thats why you should run that OPNsense VM redundant with pfsync/CARP. If such a case happen and the PVE server with the primary OPNsense isn't working anymore, you still got perfectly fine working internet and routing by the secondary OPNsense on the second PVE node.

Emotional Damage xD

Everyone to his likings, but tbh, why not?
See the benefits, i have for example the Cpu Power of an Ryzen 5800x. Thats a lot better as some dedicated box with an Atom xD
Its more energy efficient either, since proxmox runs anyway 24/7.
You have less hardware to worry xD
You can do High Availability xD

I run it semi-virtual tho, means i passthrough always if i can an physical Nic Port to my Opnsense and do vlans there etc...
Gives me the benefit of activating some hardware acceleration either.
On my X550-T2, i use one 10GBe port for Proxmox and the other i passthrough to Opnsense completely, used earlier sr-iov for that, but i gone away from sr-iov and passthrough the whole physical port, which works a LOT better.
Have not to fiddle anymore with bridge fdb tables or any other issues i had with virtual functions.
And it allows me to use all hardware acceleration, which work btw flawless! (Just if you use surricata then you have to disable some hw-acc)

However, the benefits at least for me outperforms the downsides, if there are any at all.
Maybe the still broken ballooning driver in pf/opnsense is a downside, but i simply deactivate it anyway on freebsd....

Cheers

Yes, that you can't do consistent backups without shutting down the VM when using FreeBSD is really bad. Luckily I only use it for TrueNAS and OPNsense and those are appliances where you don't need a VM backup, as it can be redeployed easily as long as you got a recent backup of that single OPNsense/TrueNAS config file.
And OPNsense got a Nextcloud-Backup plugin, so I always got a daily backup of that config file on my Nextcloud VM and that is backed up to my two PBS VMs which are also redundant via sync jobs running one on each PVE node.

 

[Ramalama]

Yes, that you can't do consistent backups without shutting down the VM when using FreeBSD is really bad. Luckily I only use it for TrueNAS and OPNsense and those are appliances where you don't need a VM backup, as it can be redeployed easily as long as you got a recent backup of that single OPNsense/TrueNAS config file.
And OPNsense got a Nextcloud-Backup plugin, so I always got a daily backup of that config file on my Nextcloud VM and that is backed up to my two PBS VMs which are also redundant via sync jobs running one on each PVE node.

But Dunuin ...
I do consistent "live" Backups every weekend, i even restored once opnsense from a backup without issues.
I even did snapshots and rolled back without any issues, that way i tested crowdsec for example and rolled back a snapshot (all live) without any issues.

Did you had issues with consistent backups when you passthrough a nic? Or why you said that?

 

[Dunuin]

But Dunuin ...
I do consistent "live" Backups every weekend, i even restored once opnsense from a backup without issues.
I even did snapshots and rolled back without any issues, that way i tested crowdsec for example and rolled back a snapshot (all live) without any issues.

Did you had issues with consistent backups when you passthrough a nic? Or why you said that?

Snapshot-mode backups aren't consistent on FreeBSD because the write cache won't get flushed before doing the backup, so the recent async writes are lost. As the FreeBSD implementation of the QEMU guest agent doesn't support fsfreeze. So yes, it might work, if there wasn't important data in the cache that is then lost, but it's not guaranteed that a restore VM will be fine. A snapshot-mode backup on FreeBSD is better than nothing, but I wouldn`t rely on that. I still do a weekly stop-mode backup (not that bad thanks to high available OPNsense ) to have something to rely on. IThen I could restore that weekly stop-mode backup and import my daily config file from Nextcloud on it to get to a recent state.

Would you just unplug a USB-Disk or unmount it before doing that so you don't lose the data that is still in the write cache? Doing snapshot-mode backups with FreeBSD is just that...

 

[Ramalama]

Snapshot-mode backups aren't consistent on FreeBSD because the write cache won't get flushed before doing the backup, so the recent async writes are lost. As the FreeBSD implementation of the QEMU guest agent doesn't support fsfreeze. So yes, it might work, if there wasn't important data in the cache that is then lost, but it's not guaranteed that a restore VM will be fine. A snapshot-mode backup on FreeBSD is better than nothing, but I wouldn`t rely on that. I still do a weekly stop-mode backup (not that bad thanks to high available OPNsense ) to have something to rely on. IThen I could restore that weekly stop-mode backup and import my daily config file from Nextcloud on it to get to a recent state.

Thanks for the info, good to know!

However, i think that i didn't had any issues, because opnsense almost never writes anything, just if you change sth in the GUI.
Mostly everything else gets modified in memory, like states/connections and so on.

But i find it very weird the fsfreeze isn't supported, however ballooning doesn't work either. (Ballooning works half backed)

I need to test that, but as far i remember, at least some years ago, i worked in my old company a lot with pfsense, but that was on esxi tho...
Fsfreeze worked there :-(


However, thx for the info! Good to know!

 

[Ramalama]

Hey @Dunuin
https://github.com/opnsense/plugins/issues/3283#issuecomment-1637233731

If we have luck we probably get a full working qemu-guest-agent on Opnsense.

Hopefully they just don't ignore it xD