Proxmox 9 upgrade: Virt-customize no longer has internet access

[ruipin]

Hi,

I have upgraded my 2-node cluster to Proxmox 9 and everything went without a hitch. Thanks everyone for their efforts!

However, I have noticed one behaviour change: As part of my provisioning scripts, I use virt-customize to set up VM templates, running e.g.::

virt-customize --network --update --install qemu-guest-agent --hostname trixie-template --truncate /etc/machine-id -a /run/ansible-vmtemplate/debian-13-genericcloud-amd64.qcow2

Under Proxmox 9 however, I see the following errors:

...
W: Failed to fetch mirror+file:/etc/apt/mirrors/debian.list/dists/trixie/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch mirror+file:/etc/apt/mirrors/debian.list/dists/trixie-updates/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch mirror+file:/etc/apt/mirrors/debian.list/dists/trixie-backports/InRelease  Temporary failure resolving 'deb.debian.org'
W: Failed to fetch mirror+file:/etc/apt/mirrors/debian-security.list/dists/trixie-security/InRelease  Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package qemu-guest-agent
virt-customize: error:
      export DEBIAN_FRONTEND=noninteractive
      apt_opts='-q -y -o Dpkg::Options::=--force-confnew'
      apt-get $apt_opts update
      apt-get $apt_opts install 'qemu-guest-agent'
    : command exited with an error

Essentially, the virt-customize VM has no internet access, no matter what I do. I cannot seem to resolve domains or otherwise access the internet. Running "ip a" shows only the loopback interface and eth0, but the latter is DOWN. Bringing it up results in a link-local address only.

VMs in Proxmox work fine, as does creating a VM with the image and then booting it with Proxmox. It is just virt-customize that no longer works.

These commands have worked fine for years, but suddenly stopped working in Proxmox 9.
I noticed this after updating the first node, and tested it on my second node still on 8.4, where it worked correctly. I then upgraded that node, and it broke as well.

Does anyone have any idea what could be the problem? I've spent some time trying to debug it but have turned up no solution.

 

[bbgeek17]

Have you tried updating /etc/resolv.conf inside the image?

Since you had to install virt-customize separately (as it's not included in the default PVE installation), it’s worth noting that there’s no compatibility testing done on the Proxmox side. So you may be running into a package bug or mismatch. It's also possible that DHCP in your environment isn't behaving as expected.

Personally, when I need to make minor corrections to a VM image, I prefer using the nbd/chroot method. It’s far less intrusive and keeps with the "appliance" nature of PVE. If you prefer working with the virt-* tools, I’d recommend running them inside a dedicated VM for better isolation and reproducibility.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Zubin Singh Parihar]

Question for you about virt-customize and your Proxmox setup...

For Proxmox 8.4, if i run 'apt install guestfs-tools' it needs to remove the 'fuse' package. And if the 'fuse' package is uninstalled, then 'pve-cluster' and 'ceph-fuse' will have issues, because they both utilize 'fuse'.

Did you some how take care of this? Or are you not running 'pve-cluster' and 'ceph-fuse' at all?

 

[ruipin]

Thanks for the replies!

@Zubin Singh Parihar I didn't have that issue, although the package I installed was "libguestfs-tools" and I also installed it years ago pre-8.4 (carried over to 8.4 and 9 through upgrades). I just checked and my fuse package is still installed. For reference I don't use ceph but I do rely on pve-cluster and other fuse file systems and they are working fine.

@bbgeek17 I am aware there's no compatibility testing with virt-customize, I didn't mean to claim that this was something that was the Proxmox's team fault! I was just wondering if anyone had any idea how to resolve it as it used to work fine before.

I did try to patch resolv.conf but apparently that isn't really a thing within the libguestfs network stack - the file doesn't seem to exist (both in the working Proxmox 8 setup, and the broken Proxmox 9 setup).

I'll have to look into nbd/chroot, all guides back when I set up my VM templates were recommending virt-customize and it was very easy to use so I never really considered alternatives.

 

[proximoxi]

Edit:
This will cause different routing problems, so i will recommend to go with the nbd/chroot line or go on another machine with libguestfs.

Hi @ruipin

I got the same error since pve 9.

During debugging, I found the following entry:

Debug:

/init: line 140: dhcpcd: command not found

After installing the dhcpcd package on the Proxmox Host:

apt-get install -y dhcpcd

virt-customize working again.

Debug:

+ dhcpcd eth0
getpwnam: No such file or directory
dhcpcd-10.1.0 starting
[    1.056572] 8021q: 802.1Q VLAN Support v1.8
[    1.072184] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    1.073195] Loaded X.50...........
...................

root@pve:/var/isos# virt-customize -a templ-deb-controller.qcow2 --install htop
[   0.0] Examining the guest ...
[   8.8] Setting a random seed
virt-customize: warning: random seed could not be set for this type of
guest
[   8.8] Installing packages: htop
[  12.6] SELinux relabelling
[  13.0] Finishing off

hope it will help you.

 

[ruipin]

@proximoxi Thanks for the reply. You mention that this caused different routing issues for you - could you go into some detail what issues you saw?
I do plan to try nbd/chroot but it would be nice to have virt-customize as a backup solution regardless, especially since I already have my provisioning script able to use it.

 

[proximoxi]

I’m using LACP (bond) with different Linux bridges (VLANs).
Somehow, my Linux bridges started to obtain DHCP leases, and my Proxmox server suddenly became unreachable from some subnets.
After uninstalling dhcpcd, everything went back to normal.

What I did: I created a new VM and installed libguestfs on it.
I adapted my ansible scripts so that it performs the virt-customize tasks via this VM.

The VM is also running Debian 13 (and virt-customize works flawlessly), so in my opinion, the problem is not with Debian itself but rather somewhere in the Debian/Proxmox combination.

To be honest, I didn’t investigate the problem in detail. It seemed easier to quickly adjust the scripts than to spend more time debugging.

 

[ruipin]

I assume this is because installing dhcpc on the host enabled the dhcpc service, which automatically started requesting DHCP leases with the default configuration (i.e. on all interfaces).

It could suffice to disable this service, but I agree that is a bit brittle! Thanks for all the help