[TUTORIAL] Сooking after installation

Pachakamak

New Member
Apr 9, 2021
1
0
1
31
Russia, Saint-Petersburg
Good time!
Previously, I worked with proxmox virtualization, but there was no need to assemble a cluster, and now it's time for clustering.
In particular, I was interested in a convenient and flexible network virtualization solution. The standard configuration by means of a Linux system seems to me cumbersome and not convenient for scaling.
The task was to be able to easily add an indefinite number of VLAN's and assign VLAN tags to virtual machines with the ability to change. The optimal solution, in my opinion, is to replace the standard Linux network configuration with an ovs switch.
Therefore, after studying a certain amount of information on the proxmox wiki and in the Internet, I came to the following network configuration solution:
auto lo iface lo inet loopback allow-vmbr0 ens17f0 auto ens17f0 iface ens17f0 inet manual mtu 9214 allow-vmbr0 ens17f1 auto ens17f1 iface ens17f1 inet manual mtu 9214 auto mgmt1 iface mgmt1 inet static address 192.168.10.205/20 gateway 192.168.10.1 ovs_type OVSIntPort ovs_bridge vmbr0 ovs_mtu 1500 ovs_options tag=1 ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif auto bond0 iface bond0 inet manual ovs_bonds ens17f0 ens17f1 ovs_type OVSBond ovs_bridge vmbr0 ovs_mtu 9214 ovs_options lacp=active trunks=1,10,50 vlan_mode=native-untagged bond_mode=balance-tcp tag=1 other_config:lacp-time=fast auto vmbr0 iface vmbr0 inet manual ovs_type OVSBridge ovs_ports bond0 mgmt1 ovs_mtu 9214

Also in this case, an aggregated channel is used via LACP.
For configuration trunk all you need to change "trunks=vlan_tags" to "trunks=all" or do not specify the trunk key at all.
As a result, we have only one network bridge to which all VM will be connected, and we can define the proxmox control interface to any convenient subnet.

The creation of a VM network interface will look like this:
1617959112318.png
To define the subnet of the VM, you just need to specify the tag. And to add a new subnet to the proxmox, specify the vlan tag in the network configuration.

I also add a sequence of actions to configure the system after installation.
Be sure to check the network configuration after reboot and before connecting the node to the cluster.

### this script is designed to work with the installation image proxmoxVE_6.3 on debian10 system ### remove enterprise repository rm /etc/apt/sources.list.d/pve-enterprise.list ### add installation repositoryes !!!! only for install PVE !!!! touch /etc/apt/sources.list.d/pve-install-repo.list echo "deb http://download.proxmox.com/debian/pve stretch pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list echo ' # Default debian repository deb http://deb.debian.org/debian/ stretch main deb-src http://deb.debian.org/debian/ stretch main deb http://deb.debian.org/debian/ stretch-updates main deb-src http://deb.debian.org/debian/ stretch-updates main deb http://security.debian.org/debian-security/ stretch/updates main deb-src http://security.debian.org/debian-security/ stretch/updates main # Security debian updates deb http://security.debian.org/debian-security buster/updates main contrib # PVE no subscription repository provided by proxmox.com deb http://download.proxmox.com/debian/pve buster pve-no-subscription' > /etc/apt/sources.list ### add additionaly repository for ceph !!!! only for install CEPH !!!! touch /etc/apt/sources.list.d/ceph.list echo "deb http://download.proxmox.com/debian/ceph-nautilus buster main" > /etc/apt/sources.list.d/ceph.list apt update apt dist-upgrade apt install net-tools apt install ethtool apt install openvswitch-switch apt install ifenslave-2.6 apt install ifupdown2 nano /etc/sysctl.conf ### next string should matter: net.ipv4.ip_forward = 1 ### check sysctl -p ### Include all nodes in cluster /etc/hosts ### Targets server addr hostname.domain hostname 127.0.0.1 localhost.localdomain localhost 192.168.10.201 hv01.your.company.domine hv01 ### Other nodes in cluster 192.168.10.202 hv02.your.company.domine hv02 192.168.10.203 hv03.your.company.domine hv03 192.168.10.204 hv04.your.company.domine hv04 192.168.10.205 hv05.your.company.domine hv05 192.168.10.206 hv06.your.company.domine hv06 192.168.10.207 hv07.your.company.domine hv07 192.168.10.208 hv08.your.company.domine hv08 ### Network configuration witch use OVS switch: SFP+ bound, lacp=active, trunk-native-untagged + used trunks ### change your network settings ############################################################################################################################## # network interface settings; autogenerated # Please do NOT modify this file directly, unless you know what # you're doing. # # If you want to manage parts of the network configuration manually, # please utilize the 'source' or 'source-directory' directives to do # so. # PVE will preserve these directives, but will NOT read its network # configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface ens11f0 inet manual iface ens11f1 inet manual allow-vmbr0 ens17f0 auto ens17f0 iface ens17f0 inet manual mtu 9214 allow-vmbr0 ens17f1 auto ens17f1 iface ens17f1 inet manual mtu 9214 auto mgmt1 iface mgmt1 inet static address 192.168.10.205/20 gateway 192.168.10.1 ovs_type OVSIntPort ovs_bridge vmbr0 ovs_mtu 1500 ovs_options tag=1 ovs_extra set interface ${IFACE} external-ids:iface-id=$(hostname -s)-${IFACE}-vif auto bond0 iface bond0 inet manual ovs_bonds ens17f0 ens17f1 ovs_type OVSBond ovs_bridge vmbr0 ovs_mtu 9214 ovs_options lacp=active trunks=1,10,50 vlan_mode=native-untagged bond_mode=balance-tcp tag=1 other_config:lacp-time=fast auto vmbr0 iface vmbr0 inet manual ovs_type OVSBridge ovs_ports bond0 mgmt1 ovs_mtu 9214 #################################################################################################################################### ### Cluster quorum tunes ### Witness Server utils apt install corosync-qdevice apt install corosync-qnetd ### Other nodes utils apt install corosync-qdevice ### VM agent for linux systems ############ apt install qemu-guest-agent
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!