We bond all our Nics (Network interface Card as in eth0, eth1, ...) into a single Bond, then attach a bridge to it. To that bridge we add Ovs_InternalPorts.
We have different "sets" of Servers - depending on how old they are and what software they run - they have different connectivity. As an example:
- 1G onboard + quad 1G via pcie = 5G connectivity
- 1x dual 1G onboard + dual 10G via Pcie = 22G connectivity
- 1x dual10G onboard + dual 10G via Pcie = 40G connectivity
- 1x dual 10G onboard + 2x Dual 40G via Pcie = 100G connectivity
The reason beeing, that if we e.g. have dual 10G nics and use one dedicated Nic for Corosync/Proxmox, we'd be wasting tons of potential bandwith for e.g. ceph / NFS / VMs / whatever. It also makes it a bit more redundant for us, since the decision which nic/port is used by a VLan is up to openvswitch/physical switch and changes regularly
For our Backups. things start to get a bit crazy:
We have 3 Ceph Clusters (each with Erasure Coded Pools).
Ceph-CLuster 1 Is backing 24 HA-VM's (open media Vaults connected via NFS)
If our use-case requiers it, we have the Proxmox-Vm's setup to make snapshots every X-hours on the corresponding NFS-OpenMediaVault. up to 24x a day. We keep 2 copies. (so maximum of 48 hours of hourly backups)
- NFS-Servers are on 10.90.0.[1-24]
- NFS-Proxmox-Clients are on 10.90.[1-6].[1-10]
Ceph-Cluster 2 is backing 7 HA-VM's (open media Vaults connected via NFS)
This backup is done for every VM. Creates a Backup once per Day (staggered between 23:00-05:00). We keep again 2 copies for up to 14 days.
- NFS-Servers are on 10.91.0.[1-7]
- NFS-Proxmox-Clients are on 10.91.[1-6].[1-10]
Ceph-Cluster 3 is backing 4 HA-VM's (open media Vaults connected via NFS).
This type of Backup is only done for really important stuff, like e.g. our internal Wiki (where most of our configs and instructions are housed), ur Mail-Store and our document-management Software. Full backups are done once per week on sundays. we keep 18 copies thats basically 18 Month worth (only to safeguard against documents missing during the yearly revision of the company.
- NFS-Servers are on 10.92.0.[1-4]
- NFS-Proxmox-Clients are on 10.92.[1-6].[1-10]
For NTP this works similar
We have currently 7 NTP servers locally. if i'd had to guess somewhere around 2k-4k potential NTP-users. Even some Security-Cameras query NTP
- NTP-Servers are on 10.70.0.[1-7]
- NTP-Proxmox-Clients are on 10.70.[1-6].[1-10]
- NTP-VM-Clients are on 10.70.[150-200].x
- NTP-Office-Clients are on 10.70.[240-253].x
Perhaps it comes together the other way around
Example C1-ProxmoxNode-1 OvsIntPorts:
- Vlan1 Tag=101 IP=10.100.101.1
- Vlan20 Tag=20 Ip=10.20.101.1
- Vlan21 Tag=21 IP=10.21.101.1
- Vlan22 Tag=22 IP=10.22.101.1
- Vlan30 Tag=30 IP=10.30.101.1
- Vlan41 Tag=41 IP=10.41.101.1
- Vlan42 Tag=42 IP=10.42.101.1
- Vlan43 Tag=43 IP=10.43.101.1
- Vlan70 Tag=70 IP=10.70.101.1
- Vlan80 Tag=80 IP=10.80.101.1
- Vlan81 Tag=81 IP=10.81.101.1
- Vlan90 Tag=90 IP=10.90.101.1
- Vlan91 Tag=91 IP=10.91.101.1
- Vlan92 Tag=92 IP=10.92.101.1
- Vlan201 untagged IP=10.200.101.1 <-- thats where e.g. Admin Computer (e.g. 10.200.0.1) sit on.
If this were e.g. an Vm (lets say a Zimbra-MailServer) that had a to receive client traffic from e.g. an office computer then it would have an an Vlan like this:
Vlan300 untagged IP=192.168.130.1 <-- we have office-clients sitting on 192.168.0.0/16
I wanna point out that those IPs and Vlan Tags numbers are examples. we use different ones. Its completely reasonable i made typos while changing em.
Prioritising of our Vlans is done on the switches. e.g. FreeIPA > NTP > Proxmox > Ceph >NFS-Backups-C1 = NFS-backups-C2 = NFS-Backups-C3 ...
