Issue with I-225v. Starting 2 VMs crashes web UI

[enderz]

I'm pretty new to Proxmox and a little experienced in Linux.
I am currently trying to use proxmox but when I spin up 2 or more VMs the NIC link seems to go down.

When I start the VMs without a network device attached, it works.

What I have tried:

- Changing cables
- PCIe Network Card
- Reinstalling Proxmox

Spoiler: result of ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp5s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether 50:eb:f6:29:29:5f brd ff:ff:ff:ff:ff:ff
3: wlo1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 44:e5:17:16:5d:55 brd ff:ff:ff:ff:ff:ff
altname wlp0s20f3
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 50:eb:f6:29:29:5f brd ff:ff:ff:ff:ff:ff
inet 10.247.160.69/22 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::52eb:f6ff:fe29:295f/64 scope link
valid_lft forever preferred_lft forever

Logs from proxmox node:

Jan 31 10:09:38 pve pvedaemon[1181]: <root@pam> end task UPID:pve:00000648:000013CF:679C9351:vncshell::root@pam: OK
Jan 31 10:09:48 pve pvedaemon[1675]: starting termproxy UPID:pve:0000068B:00001831:679C935C:vncshell::root@pam:
Jan 31 10:09:48 pve pvedaemon[1181]: <root@pam> starting task UPID:pve:0000068B:00001831:679C935C:vncshell::root@pam:
Jan 31 10:09:48 pve pvedaemon[1180]: <root@pam> successful auth for user 'root@pam'
Jan 31 10:09:48 pve login[1678]: pam_unix(login:session): session opened for user root(uid=0) by root(uid=0)
Jan 31 10:09:48 pve systemd-logind[824]: New session 4 of user root.
Jan 31 10:09:48 pve systemd[1]: Started session-4.scope - Session 4 of User root.
Jan 31 10:09:48 pve login[1683]: ROOT LOGIN  on '/dev/pts/0'
Jan 31 10:09:48 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Jan 31 10:09:48 pve kernel: vmbr0: port 1(enp5s0) entered disabled state
Jan 31 10:10:51 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 31 10:10:51 pve kernel: vmbr0: port 1(enp5s0) entered blocking state
Jan 31 10:10:51 pve kernel: vmbr0: port 1(enp5s0) entered forwarding state
Jan 31 10:10:52 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Jan 31 10:10:52 pve kernel: vmbr0: port 1(enp5s0) entered disabled state
Jan 31 10:11:55 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 31 10:11:55 pve kernel: vmbr0: port 1(enp5s0) entered blocking state
Jan 31 10:11:55 pve kernel: vmbr0: port 1(enp5s0) entered forwarding state
Jan 31 10:12:03 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Jan 31 10:12:03 pve kernel: vmbr0: port 1(enp5s0) entered disabled state
Jan 31 10:13:06 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Jan 31 10:13:06 pve kernel: vmbr0: port 1(enp5s0) entered blocking state
Jan 31 10:13:06 pve kernel: vmbr0: port 1(enp5s0) entered forwarding state

I can provide more information if needed.

 

[cheiss]

Hi,

could you please provide the output of pveversion -v and the configuration of both VMs, best using qm config <vmid>?

 

[enderz]

Hi,

could you please provide the output of pveversion -v and the configuration of both VMs, best using qm config <vmid>?

Hi Cheiss.

Here is the output of both pveversion -v and both vms.
Thanks.

Spoiler: PVE version

root@pve:~# pveversion
pve-manager/8.3.0/c1689ccb1065a83b (running kernel: 6.8.12-4-pve)
root@pve:~# pveversion -v
proxmox-ve: 8.3.0 (running kernel: 6.8.12-4-pve)
pve-manager: 8.3.0 (running version: 8.3.0/c1689ccb1065a83b)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-4
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.4
libpve-access-control: 8.2.0
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.10
libpve-cluster-perl: 8.0.10
libpve-common-perl: 8.2.9
libpve-guest-common-perl: 5.1.6
libpve-http-server-perl: 5.1.2
libpve-network-perl: 0.10.0
libpve-rs-perl: 0.9.0
libpve-storage-perl: 8.2.9
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.5.0-1
proxmox-backup-client: 3.2.9-1
proxmox-backup-file-restore: 3.2.9-1
proxmox-firewall: 0.6.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.1
pve-cluster: 8.0.10
pve-container: 5.2.2
pve-docs: 8.3.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.2
pve-firewall: 5.1.0
pve-firmware: 3.14-1
pve-ha-manager: 4.0.6
pve-i18n: 3.3.1
pve-qemu-kvm: 9.0.2-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.3.0
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.6-pve1

Config of VM 100:

root@pve:~# qm config 100

boot: order=scsi0;ide2;net0

cores: 3

cpu: x86-64-v2-AES

ide2: local:iso/ubuntu-24.04.1-desktop-amd64.iso,media=cdrom,size=6057964K

memory: 16000

meta: creation-qemu=9.0.2,ctime=1738312797

name: t

net0: virtio=BC:24:11:78:DD:41,bridge=vmbr0,firewall=1

numa: 0

ostype: l26

scsi0: local-lvm:vm-100-disk-0,iothread=1,size=32G

scsihw: virtio-scsi-single

smbios1: uuid=0b728376-a553-457f-a7c8-788e9dc9daad

sockets: 4

vmgenid: a7296c32-ef17-4828-b0e7-8d41dc180bdc

Config of VM 101:

root@pve:~# qm config 101

boot: order=scsi0;ide2;net0

cores: 3

cpu: x86-64-v2-AES

ide2: local:iso/ubuntu-24.04.1-desktop-amd64.iso,media=cdrom,size=6057964K

memory: 16000

meta: creation-qemu=9.0.2,ctime=1738312827

name: t2

net0: virtio=BC:24:11:CC:53:25,bridge=vmbr0,firewall=1

numa: 0

ostype: l26

scsi0: local-lvm:vm-101-disk-0,iothread=1,size=32G

scsihw: virtio-scsi-single

smbios1: uuid=7932a3ae-5e67-41c9-85cf-c6e2a1347baa

sockets: 4

vmgenid: 15340649-cd72-46e4-965e-6741f6ce4901

 

[cheiss]

pve-manager/8.3.0/c1689ccb1065a83b (running kernel: 6.8.12-4-pve)

I'd first try to install the opt-in 6.11 kernel, maybe there is some bug in the igc driver in combination with your card/firmware.

sockets: 4

Any specific reason for this setting? Normally this is not what you want.

memory: 16000

Also, your machine has 32+GB RAM? Just to be sure, as over-committing is something that should be done very consciously, if ever.

Can you further provide the network config? E.g. cat /etc/network/interfaces. Missed that above

 

[enderz]

I'd first try to install the opt-in 6.11 kernel, maybe there is some bug in the igc driver in combination with your card/firmware.


Any specific reason for this setting? Normally this is not what you want.


Also, your machine has 32+GB RAM? Just to be sure, as over-committing is something that should be done very consciously, if ever.

Can you further provide the network config? E.g. cat /etc/network/interfaces. Missed that above

root@pve:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface enp5s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.247.160.69/22
        gateway 10.247.160.1
        bridge-ports enp5s0
        bridge-stp off
        bridge-fd 0

iface wlo1 inet manual


source /etc/network/interfaces.d/*

I was testing around with the ram and found the 16gb to be the best but thinking back that may be overkill. Was using the ubuntu minimal as a baseline

I was also unsure what to set the sockets as, do you know what is generally recommended?

I will try the kernel update. I'll provide an update when that's done


Update:
NIC Link still went down.

Spoiler: Logs

Jan 31 10:55:44 pve pvedaemon[1328]: VM 101 started with PID 1341.
Jan 31 10:55:44 pve pvedaemon[1188]: <root@pam> end task UPIDve:00000530:00000BB5:679C9E1F:qmstart:101:root@pam: OK
Jan 31 10:55:45 pve pvedaemon[1437]: starting vnc proxy UPIDve:0000059D:00000C69:679C9E21:vncproxy:101:root@pam:
Jan 31 10:55:45 pve pvedaemon[1189]: <root@pam> starting task UPIDve:0000059D:00000C69:679C9E21:vncproxy:101:root@pam:
Jan 31 10:55:46 pve pvedaemon[1441]: start VM 100: UPIDve:000005A1:00000C90:679C9E22:qmstart:100:root@pam:
Jan 31 10:55:46 pve pvedaemon[1187]: <root@pam> starting task UPIDve:000005A1:00000C90:679C9E22:qmstart:100:root@pam:
Jan 31 10:55:46 pve systemd[1]: Started 100.scope.
Jan 31 10:55:46 pve kernel: tap100i0: entered promiscuous mode
Jan 31 10:55:46 pve kernel: vmbr0: port 3(fwpr100p0) entered blocking state
Jan 31 10:55:46 pve kernel: vmbr0: port 3(fwpr100p0) entered disabled state
Jan 31 10:55:46 pve kernel: fwpr100p0: entered allmulticast mode
Jan 31 10:55:46 pve kernel: fwpr100p0: entered promiscuous mode
Jan 31 10:55:46 pve kernel: vmbr0: port 3(fwpr100p0) entered blocking state
Jan 31 10:55:46 pve kernel: vmbr0: port 3(fwpr100p0) entered forwarding state
Jan 31 10:55:46 pve kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Jan 31 10:55:46 pve kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Jan 31 10:55:46 pve kernel: fwln100i0: entered allmulticast mode
Jan 31 10:55:46 pve kernel: fwln100i0: entered promiscuous mode
Jan 31 10:55:46 pve kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Jan 31 10:55:46 pve kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
Jan 31 10:55:46 pve kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Jan 31 10:55:46 pve kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Jan 31 10:55:46 pve kernel: tap100i0: entered allmulticast mode
Jan 31 10:55:46 pve kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Jan 31 10:55:46 pve kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
Jan 31 10:55:46 pve pvedaemon[1441]: VM 100 started with PID 1454.
Jan 31 10:55:46 pve pvedaemon[1187]: <root@pam> end task UPIDve:000005A1:00000C90:679C9E22:qmstart:100:root@pam: OK
Jan 31 10:55:46 pve systemd[1]: systemd-fsckd.service: Deactivated successfully.
Jan 31 10:55:47 pve pvedaemon[1546]: starting vnc proxy UPIDve:0000060A:00000D1A:679C9E23:vncproxy:100:root@pam:
Jan 31 10:55:47 pve pvedaemon[1188]: <root@pam> starting task UPIDve:0000060A:00000D1A:679C9E23:vncproxy:100:root@pam:
Jan 31 10:55:48 pve pvedaemon[1188]: <root@pam> end task UPIDve:0000060A:00000D1A:679C9E23:vncproxy:100:root@pam: OK
Jan 31 10:55:50 pve pvedaemon[1189]: <root@pam> end task UPIDve:0000059D:00000C69:679C9E21:vncproxy:101:root@pam: OK
Jan 31 10:55:54 pve pvedaemon[1188]: <root@pam> starting task UPIDve:00000626:00000FB9:679C9E2A:vncproxy:101:root@pam:
Jan 31 10:55:54 pve pvedaemon[1574]: starting vnc proxy UPIDve:00000626:00000FB9:679C9E2A:vncproxy:101:root@pam:
Jan 31 10:55:56 pve pvedaemon[1188]: <root@pam> end task UPIDve:00000626:00000FB9:679C9E2A:vncproxy:101:root@pam: OK
Jan 31 10:55:58 pve pvedaemon[1580]: starting vnc proxy UPIDve:0000062C:00001187:679C9E2E:vncproxy:100:root@pam:
Jan 31 10:55:58 pve pvedaemon[1188]: <root@pam> starting task UPIDve:0000062C:00001187:679C9E2E:vncproxy:100:root@pam:
Jan 31 10:56:02 pve pvedaemon[1188]: <root@pam> end task UPIDve:0000062C:00001187:679C9E2E:vncproxy:100:root@pam: OK
Jan 31 10:56:15 pve kernel: igc 0000:05:00.0 enp5s0: NIC Link is Down
Jan 31 10:56:15 pve kernel: vmbr0: port 1(enp5s0) entered disabled state
Jan 31 10:56:35 pve qm[1693]: <root@pam> starting task UPIDve:000006A0:00001FC2:679C9E53:qmstop:101:root@pam:
Jan 31 10:56:35 pve qm[1696]: stop VM 101: UPIDve:000006A0:00001FC2:679C9E53:qmstop:101:root@pam:
Jan 31 10:56:35 pve kernel: tap101i0: left allmulticast mode

 

[cheiss]

I was testing around with the ram and found the 16gb to be the best but thinking back that may be overkill.

Depends on the host, of course. Over-committing of RAM can be done of course, but should be done with care.

I was also unsure what to set the sockets as, do you know what is generally recommended?

Normally, 1. You don't specify anything about your system, but assuming it is a single-socket system, you really don't want to set anything else.
Especially with vCPUs set to 3.
This setting is only relevant on multi-socket systems, and then considerations about the topology w.r.t. NUMA must be made too.

See e.g. this thread, which goes into quite a bit more detail: https://forum.proxmox.com/threads/sockets-vs-cores-vs-vcpus.56339/

In any case, the timing when a VM is started and when the NIC goes down doesn't seem to correlate that much. Does this not happen in idle, i.e. if you just let the machine sit for a while? My hunch here would otherwise be a unstable connection physical connection, which the error message seems to most likely indicate. Maybe try some other cable, too.

Next, I'd verify the interface settings using ethtool enp5s0 and ip -s link show enp5s0, if anything unusual pops up and/or the latter command shows a high error rate.

 

[enderz]

Depends on the host, of course. Over-committing of RAM can be done of course, but should be done with care.


Normally, 1. You don't specify anything about your system, but assuming it is a single-socket system, you really don't want to set anything else.
Especially with vCPUs set to 3.
This setting is only relevant on multi-socket systems, and then considerations about the topology w.r.t. NUMA must be made too.

See e.g. this thread, which goes into quite a bit more detail: https://forum.proxmox.com/threads/sockets-vs-cores-vs-vcpus.56339/

In any case, the timing when a VM is started and when the NIC goes down doesn't seem to correlate that much. Does this not happen in idle, i.e. if you just let the machine sit for a while? My hunch here would otherwise be a unstable connection physical connection, which the error message seems to most likely indicate. Maybe try some other cable, too.

Next, I'd verify the interface settings using ethtool enp5s0 and ip -s link show enp5s0, if anything unusual pops up and/or the latter command shows a high error rate.

I'm seeing with ip-s link show enp5s0 that 47 packets are dropped.

Here is a picture of the commands. I started the vms and just let them sit.

Edit: WebUI seems to work now, just gonna sit on the System log screen and move around a bit for a few minutes see if that works.

Edit2: Just went down right after I wrote that.

 

[cheiss]

Yeah, the ethtool report is pretty conclusive, seems like a problem with the physical connection. There's no link.
I'd try swapping cables, some other switch/router to plug into, etc.
If you have some spare PCIe NIC laying around and means to install it into the server, maybe the (presumably) on-board NIC is faulty.

 

[enderz]

Yeah, the ethtool report is pretty conclusive, seems like a problem with the physical connection. There's no link.
I'd try swapping cables, some other switch/router to plug into, etc.
If you have some spare PCIe NIC laying around and means to install it into the server, maybe the (presumably) on-board NIC is faulty.

The cable is new, got it from IT yesterday. I'll try to talk to them to see if I can get a network with unrestricted access.

I tried a PCIe NIC but got the same result. Maybe because it was using R8169? Unsure. I'll get back to you about it.

 

[enderz]

I have connected my server to another router/network. Seems it was the router/switch being mad that it was receiving a lot of data from one point.

I can update if any other problem arises but seems it works now.
Thanks.