Debian 11 Cloud Image Networking broken

Vengance · Jul 1, 2022

Hi,

I' running some VMs based on the Debian KVM Cloud image.
However, the VM is stuck in the boot screen for at least a minute on startup due to network issues.

I did some digging in the log and apparently the interface changes names several times between ens18 <-> eth0, I suspect this is related to the problem.
I've attached are some screenshots.

Does anyone have any idea what this could be due to?
The image is based on the latest debian-11-generic-amd64.qcow2

Bildschirmfoto 2022-07-01 um 16.16.22.png

Bildschirmfoto 2022-07-01 um 16.15.38.png

Bildschirmfoto 2022-07-01 um 14.14.31.png

Moayad · Jul 4, 2022

Works on my test lab without any issue, you can set a rule in udev rules.

Can you please post the template config qm config <VMID> and the output of pveversion -v

for more information about network interface names, see below link

https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

Vengance · Jul 4, 2022

Maybe because I'm using DHCP/ radvd?
Just deployed another Debian VM, still the same issue.

Code:

proxmox-ve: 7.2-1 (running kernel: 5.15.35-3-pve)
pve-manager: 7.2-5 (running version: 7.2-5/12f1e639)
pve-kernel-5.15: 7.2-5
pve-kernel-helper: 7.2-5
pve-kernel-5.15.35-3-pve: 5.15.35-6
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-2
libpve-storage-perl: 7.2-5
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.12-1
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.3-1
proxmox-backup-file-restore: 2.2.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-10
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1

Code:

agent: 1
boot: c
bootdisk: scsi0
ciuser: root
cores: 1
cpu: host
ide2: local-zfs:vm-118-cloudinit,media=cdrom,size=4M
ipconfig0: ip6=auto
memory: 1024
meta: creation-qemu=6.2.0,ctime=1656709433
name: debian
net0: virtio=3A:AF:F0:7A:1B:72,bridge=vmbr0
numa: 0
scsi0: local-zfs:base-9000-disk-0/vm-118-disk-0,size=2G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=03cac89f-9bf8-4e03-9727-9c8579dc80a2
sockets: 1
sshkeys: ssh-rsa
vga: serial0
vmgenid: 8c018caf-b0ff-4ae9-96ba-1825312f9882

Vengance · Jul 5, 2022

Did some more testing and I can't get it to work, tried different cloud images but always the same.

bbgeek17 · Jul 5, 2022

Vengance said:
ipconfig0: ip6=auto

have you tried Ipv4 and/or static IP as a test?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Vengance · Jul 5, 2022

Setting a static IPv4 or IPv6 does not help, the network also fails during startup.

For testing purposes I just created a LXC container with the latest Debian 11 template and also set the network config to IPv6 SLAAC.

It worked like a charm

bbgeek17 · Jul 5, 2022

we use these images all the time without issues. Try to use en1000 hardware profile for NIC instead of virtio. You shouldnt have to, but something strange is going on with your setup.

You also said that the "image is based on" generic one, have you modified it in any way? Does the vanilla generic image work?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Vengance · Jul 5, 2022

bbgeek17 said:
we use these images all the time without issues. Try to use en1000 hardware profile for NIC instead of virtio. You shouldnt have to, but something strange is going on with your setup.

You also said that the "image is based on" generic one, have you modified it in any way? Does the vanilla generic image work?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Which image type are you using?
Generic, Generic Cloud?

bbgeek17 · Jul 5, 2022

Cloud, since we use cloud-init

Code:

+ qm create 100 --memory 512 --name vm100 --socket 1 --onboot no
+ qm importdisk 100 /mnt/pve/bbnas/template/iso/debian-11-genericcloud-amd64.qcow2 blockbridge --format raw
importing disk '/mnt/pve/bbnas/template/iso/debian-11-genericcloud-amd64.qcow2' to VM 100 ...
transferred 0.0 B of 2.0 GiB (0.00%)

skip

Code:

root@pve7demo1:~# qm start 100;qm terminal 100
generating cloud-init ISO
starting serial terminal on interface serial0 (press Ctrl+O to exit)
Booting from Hard Disk...
GRUB loading.
Welcome to GRUB
!
   Booting `Debian GNU/Linux' Booting `Debian GNU/Linux

Loading Linux 5.10.0-15-cloud-amd64 ...'

skip

Code:

[DEPEND] Dependency failed for ifup for ens18.
         Starting Raise network interfaces...
[  OK  ] Finished Raise network interfaces.
[  OK  ] Reached target Network.
         Starting Initial cloud-ini… (metadata service crawler)...
[   11.797921] cloud-init[480]: Cloud-init v. 20.4.1 running 'init' at Tue, 05 Jul 2022 17:32:39 +0000. Up 11.78 seconds.
[   11.807768] cloud-init[480]: ci-info: +++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
[   11.809606] cloud-init[480]: ci-info: +--------+------+------------------------------+---------------+--------+-------------------+
[   11.815728] cloud-init[480]: ci-info: | Device |  Up  |           Address            |      Mask     | Scope  |     Hw-Address    |
[   11.817645] cloud-init[480]: ci-info: +--------+------+------------------------------+---------------+--------+-------------------+
[   11.823742] cloud-init[480]: ci-info: |  eth0  | True |        172.16.101.78         | 255.255.255.0 | global | 2e:e5:b5:65:2b:d4 |
[   11.825620] cloud-init[480]: ci-info: |  eth0  | True |        172.16.101.79         | 255.255.255.0 | global | 2e:e5:b5:65:2b:d4 |
[   11.830427] cloud-init[480]: ci-info: |  eth0  | True | fe80::2ce5:b5ff:fe65:2bd4/64 |       .       |  link  | 2e:e5:b5:65:2b:d4 |
[   11.832428] cloud-init[480]: ci-info: |   lo   | True |          127.0.0.1           |   255.0.0.0   |  host  |         .         |
[   11.839731] cloud-init[480]: ci-info: |   lo   | True |           ::1/128            |       .       |  host  |         .         |
[   11.841634] cloud-init[480]: ci-info: +--------+------+------------------------------+---------------+--------+-------------------+
[   11.843509] cloud-init[480]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
[   11.847875] cloud-init[480]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
[   11.851752] cloud-init[480]: ci-info: | Route | Destination  |   Gateway    |    Genmask    | Interface | Flags |
[   11.855761] cloud-init[480]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
[   11.857340] cloud-init[480]: ci-info: |   0   |   0.0.0.0    | 172.16.101.1 |    0.0.0.0    |    eth0   |   UG  |
[   11.861416] cloud-init[480]: ci-info: |   1   | 172.16.101.0 |   0.0.0.0    | 255.255.255.0 |    eth0   |   U   |
[   11.863083] cloud-init[480]: ci-info: +-------+--------------+--------------+---------------+-----------+-------+
[   11.866645] cloud-init[480]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[   11.868576] cloud-init[480]: ci-info: +-------+-------------+---------+-----------+-------+
[   11.874954] cloud-init[480]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[   11.879904] cloud-init[480]: ci-info: +-------+-------------+---------+-----------+-------+
[   11.881391] cloud-init[480]: ci-info: |   1   |  fe80::/64  |    ::   |    eth0   |   U   |
[   11.883739] cloud-init[480]: ci-info: |   3   |    local    |    ::   |    eth0   |   U   |
[   11.887791] cloud-init[480]: ci-info: |   4   |  multicast  |    ::   |    eth0   |   U   |
[   11.889359] cloud-init[480]: ci-info: +-------+-------------+---------+-----------+-------+

skip

Code:

root@pve7demo1:~# qm config 100
agent: 1
boot: c
bootdisk: scsi0
cicustom: user=bbnas:snippets/centos.user
ide2: blockbridge:vm-100-cloudinit,media=cdrom
ipconfig0: ip=dhcp
memory: 512
meta: creation-qemu=6.2.0,ctime=1657042306
name: vm100
net0: virtio=2E:E5:B5:65:2B:D4,bridge=vmbr0,firewall=1
onboot: 0
scsi0: blockbridge:vm-100-disk-0,size=2G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=c1eca9c3-04fb-4c1c-8757-5a41b316fb9a
sockets: 1
sshkeys: ssh-rsa%
vga: serial0
vmgenid: 1cbde956-9fa7-4336-add9-109cb44aa182

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Vengance · Jul 5, 2022

Very strange, but I suspect my issue is related to the interface renaming, but why is this even happening?
Seems to work fine on other systems?

Bildschirmfoto 2022-07-05 um 19.39.16.png

bbgeek17 · Jul 5, 2022

as @Moayad pointed you earlier - "predictable network interfaces" .
The renames are normal and expected, on my VM:

Code:

Jul  5 17:45:51 vm2000 kernel: [    1.356570] virtio_net virtio3 ens18: renamed from eth0
Jul  5 17:45:51 vm2000 kernel: [    3.756320] virtio_net virtio3 eth0: renamed from ens18

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:63:e1:32:70:bd brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    altname ens18

If you are not using cloud-init, then you should probably not use Cloud image.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Vengance · Jul 5, 2022

I am using cloud-init.

Edit:
I mean my setup is quite similar to yours, the only thing I changed is using SLAAC for IPv6 and leaving IPv4 blank.
So I'm not sure what is going on here

bbgeek17 · Jul 5, 2022

Its not clear from your original post - does the VM come up in the end with fully functional network and your complaint is that its taking long? Or does the network not come up?
If its the latter - can you bring up the network manually with "ifup" or "network restart"? If you cant - then you need to troubleshoot that.

If you are just concerned about the time it takes to come up , have you already tried e1000? - may be your network device is not coming up quick enough.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Vengance · Jul 5, 2022

The network itself seems to work, the VM get's a IPv6 IP, there's just a 1minute delay when booting.
E1000 did not work in my testing, the VM was not able to get a IP.

virtio also seems to also have better performance overall

bbgeek17 · Jul 5, 2022

plug this into google "a job is running for raise network interfaces" and go over the suggestions to see if anything applies to you.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Vengance · Jul 5, 2022

https://askubuntu.com/a/1402305/1608729

Just found this by accident, this is exactly what I'm facing.
To my surprise there actually exists a file called
/run/network/interfaces.d/ens18

which configures the interface ens18 via dhcp

Code:

auto ens18
allow-hotplug ens18

iface ens18 inet dhcp

But why does this code exist in a cloud image which is supposed to configure the network via cloud-init?

bbgeek17 · Jul 5, 2022

Vengance said:
The image is based on the latest debian-11-generic-amd64.qcow2

bbgeek17 said:
You also said that the "image is based on" generic one, have you modified it in any way? Does the vanilla generic image work?

Have you modified your image file? Or do you mean "image" in proxmox sense?

/run is a temporary space for dynamically generated files.
https://wiki.debian.org/ReleaseGoals/RunDirectory
/run is a new cross-distribution location for the storage of transient state files—that is, files containing run-time information that may or may not need to be written early in the boot process and which does not require preserving across reboots.
/run is a tmpfs.

It shouldnt matter what the interface or network file is called, those are all aliases as you can see "ip a". The primary configuration file should be :
cat /etc/network/interfaces.d/50-cloud-init

good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Vengance · Jul 5, 2022

I did install qemu-guest-agent and resolvconf via virt-customize

But the issue is the same on the vanilla image.

Yes, but I guess the dhcp Definition itself is the problem, as there is no DHCP server running

erdnuesse · Oct 15, 2022

Hi, I know this thread is stale - but not too much so I hope I'm not resurrecting Zombies here.

Tldr; seems true, match the DHCP options for your network and the issue might go away.

I've had just the same problem with a debian 10 cloud image - I forgot 11, different story.
Thought, well, try 11 to iron out possible fixes, same issue. Found this thread as I was having them, and your comments helped me resolving the issue, so I might provide a solution, since there hasn't been one marked as such.

Tried e1000, no interfaces came up automatically despite guest agent already set up.

Reverted back to virtio, but changed ipv6 to SLAAC as mentioned right after, and the machine booted up just as fast as expected.

My homelab setup does not have a dhcpd for ipv6, so shame on me I guess for even trying that in the first place, but that's what it was for me. Maybe I just followed a checklist \ tutorial without thinking about my setup first, but just make sure that whatever DHCP options are present in your network are mirrored in your cloud-init IP config.
I'm also not planning on using ipv6 inside my network and probably never will, unless I absolutely have to.

Kindest regards and thanks for all your input - it definitely helped me and made a difference.

Vengance · Oct 15, 2022

Hi,

I eventually solved this issue by overwriting the included ens18 dhcp statement and setting the network config to manual.
It took me countless hours to find the root cause for this issue as this isn't documented anywhere.

The debian cloud team seems to assume there's always a DHCP server running, so the included this config by default.

Code:

virt-customize -a $IMAGE_NAME --install qemu-guest-agent --install resolvconf --update --run-command 'echo "auto ens18" >> /etc/network/interfaces.d/ens18' --run-command 'echo "iface ens18 inet manual" >> /etc/network/interfaces.d/ens18'

Debian 11 Cloud Image Networking broken

Renowned Member

Proxmox Staff Member

Renowned Member

Renowned Member

Distinguished Member

Renowned Member

Distinguished Member

Renowned Member

Distinguished Member

Renowned Member

Distinguished Member

Renowned Member

Distinguished Member

Renowned Member

Distinguished Member

Renowned Member

Distinguished Member

Renowned Member

New Member

Renowned Member