PVE Supermicro SuperServer SYS-1029U-E1CR4T network configuration

Discussion in 'Proxmox VE: Networking and Firewall' started by Borut, May 17, 2018.

  1. Borut

    Borut New Member

    Joined:
    May 16, 2018
    Messages:
    16
    Likes Received:
    0
    Hi,
    I just installed PVE 5.1-41 on a new server with 4 10GBLAN ports.
    As a default, it was created vmbr0 Linux Bridge on port eno1. Why not vmbr1 on eno1?

    Now I am trying to configure eno2. When I try to specify gateway I get:
    Error Parameter verification failed. (400). gateway: Default gateway already exist on interface 'vmbr0'.
    Is this a bug? All four ports should be configured independently of each other. What am I missing?

    I created Linux Bridge vmbr2 on port eno2 and added network device in bridged mode on bridge vmbr2, model: Intel1000 to VM 102. As a result, I see a new network device as "Network Device (net0)" What does net0 stand for? Shouldn't be "Network Device (eno2)" better?

    However, ssh connection to VM 102 (centos) doesn't work. I am able to login but very soon I get " packet_write_wait: Connection to 'VM 102' port 22: Broken pipe.

    Connection to the node where is VM 102 is normal...

    I need help. I would like to keep PVE!

    Best regards,
    Borut
     
  2. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,035
    Likes Received:
    458
    because the default bridge is just called vmbr0, no matter how the underlying physical interface is called?

    there can only be one gateway across all interfaces.. everything else needs to be handled by routes.

    netX are the config options in the guest config, they are not related to physical interfaces.

    e.g., you can have net0, net2, net3, and net7 all on the same bridge (or on different bridges, PVE does not care ;)), if you want your guest to have 4 different virtual NICs. just like you can have multiple virtual disks on one or multiple storages. for containers, you can also specify the interface name inside the container (defaulting to eth0).

    you'd need to provide more details in order to troubleshoot this issue.
     
  3. Borut

    Borut New Member

    Joined:
    May 16, 2018
    Messages:
    16
    Likes Received:
    0
    Thank you for your answer Fabian.
    I have no idea how to provide more details in order to troubleshoot ssh problem... To start with I will upload some PVE screenshots and ifconfig output from VM 102 (centos 7) to check network configuration:
    Screen Shot 2018-05-17 at 13.57.19.png Screen Shot 2018-05-17 at 13.57.51.png Screen Shot 2018-05-17 at 13.58.14.png

    After second try I was able to login for a short time:

    [elara:~] borut% ssh -v stereo
    OpenSSH_7.3p1, OpenSSL 1.0.2k 26 Jan 2017
    debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
    debug1: Connecting to stereo [134.76.239.192] port 22.
    debug1: connect to address 134.76.239.192 port 22: Connection refused
    ssh: connect to host stereo port 22: Connection refused

    [elara:~] borut% ssh -v stereo
    OpenSSH_7.3p1, OpenSSL 1.0.2k 26 Jan 2017
    debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
    debug1: Connecting to stereo [134.76.239.192] port 22.
    debug1: connect to address 134.76.239.192 port 22: Connection refused
    ssh: connect to host stereo port 22: Connection refused

    [elara:~] borut% ssh -v stereo
    OpenSSH_7.3p1, OpenSSL 1.0.2k 26 Jan 2017
    debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
    debug1: Connecting to stereo [134.76.239.192] port 22.
    debug1: Connection established.
    debug1: identity file /Users/borut/.ssh/id_rsa type 1
    debug1: key_load_public: No such file or directory
    debug1: identity file /Users/borut/.ssh/id_rsa-cert type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /Users/borut/.ssh/id_dsa type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /Users/borut/.ssh/id_dsa-cert type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /Users/borut/.ssh/id_ecdsa type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /Users/borut/.ssh/id_ecdsa-cert type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /Users/borut/.ssh/id_ed25519 type -1
    debug1: key_load_public: No such file or directory
    debug1: identity file /Users/borut/.ssh/id_ed25519-cert type -1
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_7.3
    debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
    debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
    debug1: Authenticating to stereo:22 as 'borut'
    debug1: SSH2_MSG_KEXINIT sent
    debug1: SSH2_MSG_KEXINIT received
    debug1: kex: algorithm: curve25519-sha256@libssh.org
    debug1: kex: host key algorithm: ecdsa-sha2-nistp256
    debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
    debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
    debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
    debug1: Server host key: ecdsa-sha2-nistp256 SHA256:T/RE3KhQGt8/EX6OoT78xDBMYsjUflMUv8y8H+RsS60
    debug1: Host 'stereo' is known and matches the ECDSA host key.
    debug1: Found key in /Users/borut/.ssh/known_hosts:51
    debug1: rekey after 134217728 blocks
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: rekey after 134217728 blocks
    debug1: SSH2_MSG_NEWKEYS received
    debug1: SSH2_MSG_EXT_INFO received
    debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
    debug1: Next authentication method: publickey
    debug1: Offering RSA public key: /Users/borut/.ssh/id_rsa
    debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
    debug1: Trying private key: /Users/borut/.ssh/id_dsa
    debug1: Trying private key: /Users/borut/.ssh/id_ecdsa
    debug1: Trying private key: /Users/borut/.ssh/id_ed25519
    debug1: Next authentication method: password
    borut@stereo's password:
    debug1: Authentication succeeded (password).
    Authenticated to stereo ([134.76.239.192]:22).
    debug1: channel 0: new [client-session]
    debug1: Requesting no-more-sessions@openssh.com
    debug1: Entering interactive session.
    debug1: pledge: network
    debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
    Last login: Thu May 17 14:19:36 2018 from 134.76.239.196
    [borut@stereo ~]$ cd Documents
    [borut@stereo Documents]$ ls
    admin_notes_stereo admin_notes_stereo_2
    [borut@stereo Documents]$ lspacket_write_wait: Connection to 134.76.239.192 port 22: Broken pipe
    [elara:~] borut%
     
  4. Borut

    Borut New Member

    Joined:
    May 16, 2018
    Messages:
    16
    Likes Received:
    0
    [borut@stereo ~]$ ifconfig -a
    eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
    inet 134.76.239.192 netmask 255.255.248.0 broadcast 134.76.239.255
    inet6 fe80::1abc:48b7:60c1:6ff6 prefixlen 64 scopeid 0x20<link>
    ether 00:03:ba:cd:a3:03 txqueuelen 1000 (Ethernet)
    RX packets 121417 bytes 20440254 (19.4 MiB)
    RX errors 0 dropped 5511 overruns 0 frame 0
    TX packets 969 bytes 150045 (146.5 KiB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

    lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
    inet 127.0.0.1 netmask 255.0.0.0
    inet6 ::1 prefixlen 128 scopeid 0x10<host>
    loop txqueuelen 1 (Local Loopback)
    RX packets 64 bytes 5600 (5.4 KiB)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 64 bytes 5600 (5.4 KiB)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

    virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
    inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
    ether 52:54:00:cf:f4:4a txqueuelen 1000 (Ethernet)
    RX packets 0 bytes 0 (0.0 B)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 0 bytes 0 (0.0 B)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

    virbr0-nic: flags=4098<BROADCAST,MULTICAST> mtu 1500
    ether 52:54:00:cf:f4:4a txqueuelen 1000 (Ethernet)
    RX packets 0 bytes 0 (0.0 B)
    RX errors 0 dropped 0 overruns 0 frame 0
    TX packets 0 bytes 0 (0.0 B)
    TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

    [borut@stereo ~]$
     
  5. Borut

    Borut New Member

    Joined:
    May 16, 2018
    Messages:
    16
    Likes Received:
    0
    I already increased ClientAliveInterval to 300 in /etc/ssh/sshd_config. This was not helpful. I think this problem is related to network configuration.
     
  6. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    2,107
    Likes Received:
    183
    you have the same ip in the vm as on the host, i am suprised this works at all
     
  7. Borut

    Borut New Member

    Joined:
    May 16, 2018
    Messages:
    16
    Likes Received:
    0
    I am connecting from a remote machine with IP XXX.XXX.239.196 to the VM 134.76.239.192 which refused connection. This VM is on the node with IP address XXX.XXX.239.193. So, the IP's are not the same...
     
  8. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    2,107
    Likes Received:
    183
    in your screenshot there is 134.76.239.192 on the vmbr2 which means the host has it
     
  9. Borut

    Borut New Member

    Joined:
    May 16, 2018
    Messages:
    16
    Likes Received:
    0
    Which host? Looks like I don't understand what Linux Bridge is for. My node "starspot" has 4 10GB ports. vmbr2 is on port eon2 which is for VM with IP address 134.76.239.192. In production environment I want on port eon1 exclusive only node "starspot" - no VM's. So, what is the correct way to configure network for VM? For example I would like to have VM 102 configured on NIC eon2 with IP 134.76.239.192. What do you suggest?
     
  10. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    2,107
    Likes Received:
    183
    a linux bridge is a virtual switch (the bridge ports are 'plugged in' to the bridge) where the vms plug into

    so if you want to have the ip on the vm, you do not need to configure an ip on the bridge on the host, but on the network interface inside the vm
    (it is totally valid to have a bridge without an ip)

    but if you say you want to use the nics seperately, do you mean because they access different networks? or just for performance?
    if for performance, maybe a bond would be more appropriate

    e.g.

    nic1/nic2 -> bond0 -> vmbr0 -> vmX/vmY/vmZ
     
  11. Borut

    Borut New Member

    Joined:
    May 16, 2018
    Messages:
    16
    Likes Received:
    0
    I follow your suggestion and created a new Linux Bridge vmbr3 without an IP/subnet mask. The bridge is activated, network device on VM 102 (centos 7) modified to vmbr3 and VM 102 restarted. In this case, I don't get the network running (network settings were not changed are the same as when using vmbr2 with IP/subnet mask set). Going back to the bridge vmbr2 - the network is working...
    Why is not working with vmbr3?

    Best regards,
    Borut
     
  12. Stoiko Ivanov

    Stoiko Ivanov Proxmox Staff Member
    Staff Member

    Joined:
    May 2, 2018
    Messages:
    12
    Likes Received:
    0
    could you please post the output of:
    • ip link show (on the physical host and in the vm)
    • ip address show (on the physical host and in the vm)
    • ip route show (on the physical host and in the vm)
    • brctl show (on the physical host)
     
  13. Borut

    Borut New Member

    Joined:
    May 16, 2018
    Messages:
    16
    Likes Received:
    0
    root@starspot:~# ip link show
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d4 brd ff:ff:ff:ff:ff:ff
    3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP mode DEFAULT group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d5 brd ff:ff:ff:ff:ff:ff
    4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d6 brd ff:ff:ff:ff:ff:ff
    5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d7 brd ff:ff:ff:ff:ff:ff
    12: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether d2:33:d2:4e:65:d5 brd ff:ff:ff:ff:ff:ff
    13: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether a2:ca:b2:e5:85:67 brd ff:ff:ff:ff:ff:ff
    15: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d4 brd ff:ff:ff:ff:ff:ff
    16: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d5 brd ff:ff:ff:ff:ff:ff
    17: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether c6:50:a6:e7:dd:21 brd ff:ff:ff:ff:ff:ff
    22: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether c6:50:a6:e7:dd:21 brd ff:ff:ff:ff:ff:ff
    root@starspot:~#
    Screen Shot 2018-05-23 at 12.26.21.png
    root@starspot:~# ip address show
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
    valid_lft forever preferred_lft forever
    2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d4 brd ff:ff:ff:ff:ff:ff
    3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d5 brd ff:ff:ff:ff:ff:ff
    4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d6 brd ff:ff:ff:ff:ff:ff
    5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d7 brd ff:ff:ff:ff:ff:ff
    12: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr2 state UNKNOWN group default qlen 1000
    link/ether d2:33:d2:4e:65:d5 brd ff:ff:ff:ff:ff:ff
    13: tap101i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether a2:ca:b2:e5:85:67 brd ff:ff:ff:ff:ff:ff
    15: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d4 brd ff:ff:ff:ff:ff:ff
    inet 134.76.239.193/21 brd 134.76.239.255 scope global vmbr0
    valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:fe0f:aad4/64 scope link
    valid_lft forever preferred_lft forever
    16: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:0f:aa:d5 brd ff:ff:ff:ff:ff:ff
    inet 134.76.239.192/21 brd 134.76.239.255 scope global vmbr2
    valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:fe0f:aad5/64 scope link
    valid_lft forever preferred_lft forever
    17: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether c6:50:a6:e7:dd:21 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::88cb:d4ff:fe2c:9506/64 scope link
    valid_lft forever preferred_lft forever
    22: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr3 state UNKNOWN group default qlen 1000
    link/ether c6:50:a6:e7:dd:21 brd ff:ff:ff:ff:ff:ff
    root@starspot:~#
    Screen Shot 2018-05-23 at 12.29.32.png

    root@starspot:~# ip route show
    default via 134.76.239.254 dev vmbr0 onlink
    134.76.232.0/21 dev vmbr0 proto kernel scope link src 134.76.239.193
    134.76.232.0/21 dev vmbr2 proto kernel scope link src 134.76.239.192
    root@starspot:~#

    Screen Shot 2018-05-23 at 12.39.00.png

    root@starspot:~# brctl show
    bridge name bridge id STP enabled interfaces
    vmbr0 8000.ac1f6b0faad4 no eno1
    vmbr2 8000.ac1f6b0faad5 no eno2
    tap100i0
    vmbr3 8000.c650a6e7dd21 no tap102i0
    root@starspot:~#

    I see something is broken, but I don't know how to fix it...

    Best regards,
    Borut
     
  14. Stoiko Ivanov

    Stoiko Ivanov Proxmox Staff Member
    Staff Member

    Joined:
    May 2, 2018
    Messages:
    12
    Likes Received:
    0
    • The bridge vmbr3 has only the virtual interface of the vm with id 102 (tap102i0) connected to it - and no physical nic (you could add eno3 to it, if it's plugged in and in the same vlan/network as the others).
    • Using ips from the same network on 2 different physical interfaces on the same host (starspot) leads to problems IMO - While it may work with inbound TCP-connections in certain setups, you will have problems with connection-less protocols (UDP), going over the gateway. If you don't need the 134.76.239.192 on starspot - remove it from the network definition on starspot. If you need the IP on starspot - consider adding it as an alias to vmbr0.
    • If you want to have 134.76.239.192 on the VM (cme) configure it there and not on starspot.
    • The problems with the ssh-connection from within the same network breaking up from you previous posts (from 134.76.239.196 to 134.76.239.192) look to me like the ip 134.76.239.192 is used twice on the network - are you sure it isn't configured anywhere else?
     
  15. Borut

    Borut New Member

    Joined:
    May 16, 2018
    Messages:
    16
    Likes Received:
    0
    The bridge vmbr3 is connected to eon2 port (NIC)...Is this not sufficient to work?
    Screen Shot 2018-05-23 at 14.24.43.png

    O.K. I removed IP/subnet mask from vmbr2... and still works. I will keep this setup as you suggested.
    When do I set IP/Subnet mask when I am creating a Linux bridge?
    Screen Shot 2018-05-23 at 14.29.36.png

    I couldn't reproduce the problem with ssh-connection. I deleted VM and installed it again. I am still experimenting with PVE.
     

    Attached Files:

  16. Stoiko Ivanov

    Stoiko Ivanov Proxmox Staff Member
    Staff Member

    Joined:
    May 2, 2018
    Messages:
    12
    Likes Received:
    0
    You have a typo it's eno2 not eon2.

    As for your when to set an ip on a bridge question: as @dcsapak already wrote: bridges are like a "switch" or rather a hub - every port you plug into the brige (physical like eno2 or virtual ones like the tap interfaces for the vms) see all packets on the bridge. - You can assign it an ip if you need to access the host via the physical nic that is in the bridge.
     
  17. Borut

    Borut New Member

    Joined:
    May 16, 2018
    Messages:
    16
    Likes Received:
    0
    Thank you to find a typo... To avoid this, a pulldown menu with available/valid ports will be better.

    By this mistake, I also learned that I can create one bridge for NIC. Is this correct?
    Trying to create bridge vmbr3 failed with error: port 'eno2' is already used on interface 'vmbr2' (500)
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice