PVE Supermicro SuperServer SYS-1029U-E1CR4T network configuration

[Borut]

Hi,
I just installed PVE 5.1-41 on a new server with 4 10GBLAN ports.
As a default, it was created vmbr0 Linux Bridge on port eno1. Why not vmbr1 on eno1?

Now I am trying to configure eno2. When I try to specify gateway I get:
Error Parameter verification failed. (400). gateway: Default gateway already exist on interface 'vmbr0'.
Is this a bug? All four ports should be configured independently of each other. What am I missing?

I created Linux Bridge vmbr2 on port eno2 and added network device in bridged mode on bridge vmbr2, model: Intel1000 to VM 102. As a result, I see a new network device as "Network Device (net0)" What does net0 stand for? Shouldn't be "Network Device (eno2)" better?

However, ssh connection to VM 102 (centos) doesn't work. I am able to login but very soon I get " packet_write_wait: Connection to 'VM 102' port 22: Broken pipe.

Connection to the node where is VM 102 is normal...

I need help. I would like to keep PVE!

Best regards,
Borut

 

[fabian]

Hi,
I just installed PVE 5.1-41 on a new server with 4 10GBLAN ports.
As a default, it was created vmbr0 Linux Bridge on port eno1. Why not vmbr1 on eno1?

because the default bridge is just called vmbr0, no matter how the underlying physical interface is called?

Now I am trying to configure eno2. When I try to specify gateway I get:
Error Parameter verification failed. (400). gateway: Default gateway already exist on interface 'vmbr0'.
Is this a bug? All four ports should be configured independently of each other. What am I missing?

there can only be one gateway across all interfaces.. everything else needs to be handled by routes.

I created Linux Bridge vmbr2 on port eno2 and added network device in bridged mode on bridge vmbr2, model: Intel1000 to VM 102. As a result, I see a new network device as "Network Device (net0)" What does net0 stand for? Shouldn't be "Network Device (eno2)" better?

netX are the config options in the guest config, they are not related to physical interfaces.

e.g., you can have net0, net2, net3, and net7 all on the same bridge (or on different bridges, PVE does not care ), if you want your guest to have 4 different virtual NICs. just like you can have multiple virtual disks on one or multiple storages. for containers, you can also specify the interface name inside the container (defaulting to eth0).

However, ssh connection to VM 102 (centos) doesn't work. I am able to login but very soon I get " packet_write_wait: Connection to 'VM 102' port 22: Broken pipe.

you'd need to provide more details in order to troubleshoot this issue.

 

[Borut]

Thank you for your answer Fabian.
I have no idea how to provide more details in order to troubleshoot ssh problem... To start with I will upload some PVE screenshots and ifconfig output from VM 102 (centos 7) to check network configuration:


After second try I was able to login for a short time:

[elara:~] borut% ssh -v stereo
OpenSSH_7.3p1, OpenSSL 1.0.2k 26 Jan 2017
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to stereo [134.76.239.192] port 22.
debug1: connect to address 134.76.239.192 port 22: Connection refused
ssh: connect to host stereo port 22: Connection refused

[elara:~] borut% ssh -v stereo
OpenSSH_7.3p1, OpenSSL 1.0.2k 26 Jan 2017
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to stereo [134.76.239.192] port 22.
debug1: connect to address 134.76.239.192 port 22: Connection refused
ssh: connect to host stereo port 22: Connection refused

[elara:~] borut% ssh -v stereo
OpenSSH_7.3p1, OpenSSL 1.0.2k 26 Jan 2017
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug1: Connecting to stereo [134.76.239.192] port 22.
debug1: Connection established.
debug1: identity file /Users/borut/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/borut/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/borut/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/borut/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/borut/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/borut/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/borut/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/borut/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to stereo:22 as 'borut'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:T/RE3KhQGt8/EX6OoT78xDBMYsjUflMUv8y8H+RsS60
debug1: Host 'stereo' is known and matches the ECDSA host key.
debug1: Found key in /Users/borut/.ssh/known_hosts:51
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/borut/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /Users/borut/.ssh/id_dsa
debug1: Trying private key: /Users/borut/.ssh/id_ecdsa
debug1: Trying private key: /Users/borut/.ssh/id_ed25519
debug1: Next authentication method: password
borut@stereo's password:
debug1: Authentication succeeded (password).
Authenticated to stereo ([134.76.239.192]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
Last login: Thu May 17 14:19:36 2018 from 134.76.239.196
[borut@stereo ~]$ cd Documents
[borut@stereo Documents]$ ls
admin_notes_stereo admin_notes_stereo_2
[borut@stereo Documents]$ lspacket_write_wait: Connection to 134.76.239.192 port 22: Broken pipe
[elara:~] borut%

 

[Borut]

[borut@stereo ~]$ ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 134.76.239.192 netmask 255.255.248.0 broadcast 134.76.239.255
inet6 fe80::1abc:48b7:60c1:6ff6 prefixlen 64 scopeid 0x20<link>
ether 00:03:ba:cd:a3:03 txqueuelen 1000 (Ethernet)
RX packets 121417 bytes 20440254 (19.4 MiB)
RX errors 0 dropped 5511 overruns 0 frame 0
TX packets 969 bytes 150045 (146.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 64 bytes 5600 (5.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 64 bytes 5600 (5.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 52:54:00:cf:f4:4a txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

virbr0-nic: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 52:54:00:cf:f4:4a txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[borut@stereo ~]$

 

[Borut]

I already increased ClientAliveInterval to 300 in /etc/ssh/sshd_config. This was not helpful. I think this problem is related to network configuration.

 

[dcsapak]

debug1: Connecting to stereo [134.76.239.192] port 22.
debug1: connect to address 134.76.239.192 port 22: Connection refused

you have the same ip in the vm as on the host, i am suprised this works at all

 

[Borut]

I am connecting from a remote machine with IP XXX.XXX.239.196 to the VM 134.76.239.192 which refused connection. This VM is on the node with IP address XXX.XXX.239.193. So, the IP's are not the same...

 

[dcsapak]

in your screenshot there is 134.76.239.192 on the vmbr2 which means the host has it

 

[Borut]

Which host? Looks like I don't understand what Linux Bridge is for. My node "starspot" has 4 10GB ports. vmbr2 is on port eon2 which is for VM with IP address 134.76.239.192. In production environment I want on port eon1 exclusive only node "starspot" - no VM's. So, what is the correct way to configure network for VM? For example I would like to have VM 102 configured on NIC eon2 with IP 134.76.239.192. What do you suggest?

 

[dcsapak]

a linux bridge is a virtual switch (the bridge ports are 'plugged in' to the bridge) where the vms plug into

so if you want to have the ip on the vm, you do not need to configure an ip on the bridge on the host, but on the network interface inside the vm
(it is totally valid to have a bridge without an ip)

but if you say you want to use the nics seperately, do you mean because they access different networks? or just for performance?
if for performance, maybe a bond would be more appropriate

e.g.

nic1/nic2 -> bond0 -> vmbr0 -> vmX/vmY/vmZ

 

[Borut]

I follow your suggestion and created a new Linux Bridge vmbr3 without an IP/subnet mask. The bridge is activated, network device on VM 102 (centos 7) modified to vmbr3 and VM 102 restarted. In this case, I don't get the network running (network settings were not changed are the same as when using vmbr2 with IP/subnet mask set). Going back to the bridge vmbr2 - the network is working...
Why is not working with vmbr3?

Best regards,
Borut

 

[Stoiko Ivanov]

could you please post the output of:

• ip link show (on the physical host and in the vm)
• ip address show (on the physical host and in the vm)
• ip route show (on the physical host and in the vm)
• brctl show (on the physical host)

 

[Borut]

root@starspot:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:0f:aa:d4 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:0f:aa:d5 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:0f:aa:d6 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:0f:aa:d7 brd ff:ff:ff:ff:ff:ff
12: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether d2:33:d2:4e:65:d5 brd ff:ff:ff:ff:ff:ff
13: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
link/ether a2:ca:b2:e5:85:67 brd ff:ff:ff:ff:ff:ff
15: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:0f:aa:d4 brd ff:ff:ff:ff:ff:ff
16: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ac:1f:6b:0f:aa:d5 brd ff:ff:ff:ff:ff:ff
17: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether c6:50:a6:e7:dd:21 brd ff:ff:ff:ff:ff:ff
22: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether c6:50:a6:e7:dd:21 brd ff:ff:ff:ff:ff:ff
root@starspot:~#

root@starspot:~# ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether ac:1f:6b:0f:aa:d4 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP group default qlen 1000
link/ether ac:1f:6b:0f:aa:d5 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether ac:1f:6b:0f:aa:d6 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether ac:1f:6b:0f:aa:d7 brd ff:ff:ff:ff:ff:ff
12: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
link/ether d2:33:d2:4e:65:d5 brd ff:ff:ff:ff:ff:ff
13: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
link/ether a2:ca:b2:e5:85:67 brd ff:ff:ff:ff:ff:ff
15: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ac:1f:6b:0f:aa:d4 brd ff:ff:ff:ff:ff:ff
inet 134.76.239.193/21 brd 134.76.239.255 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::ae1f:6bff:fe0f:aad4/64 scope link
valid_lft forever preferred_lft forever
16: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ac:1f:6b:0f:aa:d5 brd ff:ff:ff:ff:ff:ff
inet 134.76.239.192/21 brd 134.76.239.255 scope global vmbr2
valid_lft forever preferred_lft forever
inet6 fe80::ae1f:6bff:fe0f:aad5/64 scope link
valid_lft forever preferred_lft forever
17: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether c6:50:a6:e7:dd:21 brd ff:ff:ff:ff:ff:ff
inet6 fe80::88cb:d4ff:fe2c:9506/64 scope link
valid_lft forever preferred_lft forever
22: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN group default qlen 1000
link/ether c6:50:a6:e7:dd:21 brd ff:ff:ff:ff:ff:ff
root@starspot:~#


root@starspot:~# ip route show
default via 134.76.239.254 dev vmbr0 onlink
134.76.232.0/21 dev vmbr0 proto kernel scope link src 134.76.239.193
134.76.232.0/21 dev vmbr2 proto kernel scope link src 134.76.239.192
root@starspot:~#



root@starspot:~# brctl show
bridge name bridge id STP enabled interfaces
vmbr0 8000.ac1f6b0faad4 no eno1
vmbr2 8000.ac1f6b0faad5 no eno2
tap100i0
vmbr3 8000.c650a6e7dd21 no tap102i0
root@starspot:~#

I see something is broken, but I don't know how to fix it...

Best regards,
Borut

 

[Stoiko Ivanov]

• The bridge vmbr3 has only the virtual interface of the vm with id 102 (tap102i0) connected to it - and no physical nic (you could add eno3 to it, if it's plugged in and in the same vlan/network as the others).
• Using ips from the same network on 2 different physical interfaces on the same host (starspot) leads to problems IMO - While it may work with inbound TCP-connections in certain setups, you will have problems with connection-less protocols (UDP), going over the gateway. If you don't need the 134.76.239.192 on starspot - remove it from the network definition on starspot. If you need the IP on starspot - consider adding it as an alias to vmbr0.
• If you want to have 134.76.239.192 on the VM (cme) configure it there and not on starspot.
• The problems with the ssh-connection from within the same network breaking up from you previous posts (from 134.76.239.196 to 134.76.239.192) look to me like the ip 134.76.239.192 is used twice on the network - are you sure it isn't configured anywhere else?

 

[Borut]

The bridge vmbr3 is connected to eon2 port (NIC)...Is this not sufficient to work?


O.K. I removed IP/subnet mask from vmbr2... and still works. I will keep this setup as you suggested.
When do I set IP/Subnet mask when I am creating a Linux bridge?


I couldn't reproduce the problem with ssh-connection. I deleted VM and installed it again. I am still experimenting with PVE.

 

[Stoiko Ivanov]

You have a typo it's eno2 not eon2.

As for your when to set an ip on a bridge question: as @dcsapak already wrote: bridges are like a "switch" or rather a hub - every port you plug into the brige (physical like eno2 or virtual ones like the tap interfaces for the vms) see all packets on the bridge. - You can assign it an ip if you need to access the host via the physical nic that is in the bridge.

 

[Borut]

Thank you to find a typo... To avoid this, a pulldown menu with available/valid ports will be better.

By this mistake, I also learned that I can create one bridge for NIC. Is this correct?
Trying to create bridge vmbr3 failed with error: port 'eno2' is already used on interface 'vmbr2' (500)