Manual install on Debian 12 (Proxmox 8.4.1) – network not persistent + cluster split (ZFS/RAID10)

[Kratus]

Hi guys,

I’m running a Proxmox cluster setup with two nodes (in different cities), but I had to install Proxmox manually on top of Debian 12 because the ISO wouldn’t load properly on my Dell R640 (Broadcom NICs). The installer freezes during driver loading (graphics/network), I tried iDRAC virtual media, physical USB, console boot options, safe mode, etc — nothing worked. So I went with Debian Netinstall and installed Proxmox manually afterwards.

System runs fine with ZFS on RAID 10, both for system and VMs. However, after rebooting, the network interfaces won’t come up automatically. Devices like ens1f0np0 and ens1f1np1 don’t show up or get IPs unless I manually run ip link set and ip addr add commands from a custom bash script at boot.

This node (Alpha) currently has no public IP — I used to assign one for internet access (for apt updates and package installs), but now it’s isolated and can’t reach the internet, which makes it difficult to install things like zfsutils, updates, etc.

Also, this Alpha node got disconnected from the original cluster. Now only the Bravo node appears active in the cluster. I need help deciding:

Should I recreate a new cluster from Alpha, which is stable now (even with manual network)?

Or should I try to rejoin Alpha into the old cluster, where Bravo is still running solo?

Both nodes run ZFS and RAID10, and my goal is to have replication and some level of HA between them.


Finally — how can I fix this persistent network issue where interfaces don’t auto-start after reboot? ifreload -a doesn’t help, and my bridges (vmbr0, vmbr1) and VLANs are properly configured.

Any suggestions or experience would be really appreciated.
Thanks

Proxmox version: pve-manager/8.4.1

 

[LnxBil]

This node (Alpha) currently has no public IP — I used to assign one for internet access (for apt updates and package installs), but now it’s isolated and can’t reach the internet, which makes it difficult to install things like zfsutils, updates, etc.

You also use a mirror, a webproxy or even a socks proxy over your connecting SSH connection.

However, after rebooting, the network interfaces won’t come up automatically. Devices like ens1f0np0 and ens1f1np1 don’t show up or get IPs unless I manually run ip link set and ip addr add commands from a custom bash script at boot.

How does your /etc/network/interfaces look like? Please post text in CODE tags.

Should I recreate a new cluster from Alpha, which is stable now (even with manual network)?

Or should I try to rejoin Alpha into the old cluster, where Bravo is still running solo?

Why should a single node be in a cluster? That does not make any sense. Besides, a two-node cluster is not supported in PVE, special versions work with a QDevice, but I don't see how that would work for your system without direct connection to each other.

 

[Kratus]

You also use a mirror, a webproxy or even a socks proxy over your connecting SSH connection.


How does your /etc/network/interfaces look like? Please post text in CODE tags.


Why should a single node be in a cluster? That does not make any sense. Besides, a two-node cluster is not supported in PVE, special versions work with a QDevice, but I don't see how that would work for your system without direct connection to each other.

Hi LnxBil,


Please look my interface configuration below. I added a pre-up command, but the interfaces still don't come up automatically on boot. When I try using ifup during boot, I get a message saying the interface is already in use — I believe this might be related to some backend service from Proxmox running during early startup.

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface enp216s0f0 inet manual

iface enp216s0f1 inet manual

iface enp216s0f2 inet manual

iface enp216s0f3 inet manual

auto ens1f0np0
iface ens1f0np0 inet manual
        mtu 9000
        pre-up ip link set ens1f0np0 up

auto ens1f1np1
iface ens1f1np1 inet manual
        mtu 9000
        pre-up ip link set ens1f1np1 up

auto vmbr0
iface vmbr0 inet static
        address 10.250.100.2/30
        gateway 10.250.100.1
        bridge-ports ens1f1np1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
# Internal Traffic // Fiber 1

auto vmbr1
iface vmbr1 inet manual
        bridge-ports ens1f0np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
# External Traffic // Fiber 2

Regarding the web proxy idea, I tried using Squid + NGINX but didn’t have much success. If you can suggest a better method, that would help a lot — I was thinking maybe a VM with internet access acting as a bridge/proxy for this node?


Currently, I'm using vmbr1 for internal traffic and to access the Proxmox GUI.


About the cluster part: I understand two nodes is not enough and doesn't guarantee quorum. I plan to deploy a QDevice soon to stabilize things until I migrate a third node (currently on VMware, but it will run Proxmox eventually).


I’m not trying to achieve full native HA, since I know that requires shared storage like Ceph — and with just two nodes it's a risky setup. I considered using Ceph, but with two nodes only, the chance of data loss or split-brain is too high.


If you have any suggestions or best practices for this kind of setup, I’d really appreciate it!


Thanks !

 

[LnxBil]

Regarding the web proxy idea, I tried using Squid + NGINX but didn’t have much success. If you can suggest a better method, that would help a lot — I was thinking maybe a VM with internet access acting as a bridge/proxy for this node?

Please look into socks proxy. You can piggy-back the connection to the outside world while logging in via SSH. You don't need any software besides SSH for this.

Please look my interface configuration below. I added a pre-up command, but the interfaces still don't come up automatically on boot.

I don't think the pre-up will do anything here. It's only run if you're "upping" the interface, and that is not happening, so the pre-commands will not work.

Can you look at the os logfiles in order to find out if there are problems prohibiting the interfaces from coming up? Maybe relevant entries in dmesg?