PVE 8 boot issue with Ubuntu Cloud Image due to broken symlink

F

foobar42

Member
Proxmox Subscriber
Dec 31, 2020
13
2
8
54
Hi,

after using qm importdisk 1013 noble-server-cloudimg-amd64.img local-zfs to import an Ubuntu Cloud Image, the symlink for disk-0 breaks for some reason:
Code: 
lrwxrwxrwx 1 root root 13 Apr 20 18:02 /dev/rpool/data/vm-1013-disk-0 -> ../../zd48p16
lrwxrwxrwx 1 root root 12 Apr 20 18:01 /dev/rpool/data/vm-1013-disk-0-part1 -> ../../zd48p1
lrwxrwxrwx 1 root root 13 Apr 20 18:01 /dev/rpool/data/vm-1013-disk-0-part14 -> ../../zd48p14
lrwxrwxrwx 1 root root 13 Apr 20 18:01 /dev/rpool/data/vm-1013-disk-0-part15 -> ../../zd48p15

When this happens, the VM doesn't boot anymore.

After applying a workaround like this:

Code: 
root@hyp-dus01-01:~# rm /dev/zvol/rpool/data/vm-1013-disk-0

root@hyp-dus01-01:~# ln -s /dev/zd48 /dev/zvol/rpool/data/vm-1013-disk-0

root@hyp-dus01-01:~# ls -lha /dev/zvol/rpool/data/vm-1013-di*
lrwxrwxrwx 1 root root  9 Apr 20 18:53 /dev/zvol/rpool/data/vm-1013-disk-0 -> /dev/zd48
lrwxrwxrwx 1 root root 15 Apr 20 18:45 /dev/zvol/rpool/data/vm-1013-disk-0-part1 -> ../../../zd48p1
lrwxrwxrwx 1 root root 16 Apr 20 18:45 /dev/zvol/rpool/data/vm-1013-disk-0-part14 -> ../../../zd48p14
lrwxrwxrwx 1 root root 16 Apr 20 18:45 /dev/zvol/rpool/data/vm-1013-disk-0-part15 -> ../../../zd48p15
lrwxrwxrwx 1 root root 14 Apr 20 18:45 /dev/zvol/rpool/data/vm-1013-disk-1 -> ../../../zd192

The VM boots again. This issue happens only with noble-server-cloudimg-amd64.img but not with ubuntu-22.04-server-cloudimg-amd64.img.

PVE is up2date:

Code: 
# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.13-1-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5: 6.5.13-1
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
proxmox-kernel-6.2.16-15-pve: 6.2.16-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 16.2.11+ds-2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.2
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.1
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.1.0
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
openvswitch-switch: 3.1.0-2+deb12u1
proxmox-backup-client: 3.1.4-1
proxmox-backup-file-restore: 3.1.4-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.5
proxmox-widget-toolkit: 4.1.4
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.4
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-3
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve2

This also happend when I convert the image to raw first. I'm not quite sure why the image messes up things on the hypervisor.
 
I had the same issue and was surprised that it worked when i imported the same image into a different vm.

I was able to reproduce the issue and it's not related to the vm but to the zd528 device. When i deleted both disks again and imported the image again in opposite order, the other vm had the issue and the vm that had the issue before worked.

The issue is that the disk-0 is linked to zd528p16 instead of zd528.

Here the logs of the first import:
Bash: 
root@pve1:~# qm clone 156 9012 --name 156-clone --full
create full clone of drive ide2 (local-zfs:vm-156-cloudinit)

root@pve1:~# ls -l /dev/zvol/rpool/data/vm-156* /dev/zvol/rpool/data/vm-9012*
lrwxrwxrwx 1 root root 14 May  8 15:50 /dev/zvol/rpool/data/vm-156-cloudinit -> ../../../zd976
lrwxrwxrwx 1 root root 14 May  8 16:11 /dev/zvol/rpool/data/vm-9012-cloudinit -> ../../../zd416

root@pve1:~# qm importdisk 156 noble-server-cloudimg-amd64.img local-zfs > /dev/null
root@pve1:~# qm importdisk 9012 noble-server-cloudimg-amd64.img local-zfs > /dev/null

root@pve1:~# ls -l /dev/zvol/rpool/data/vm-156* /dev/zvol/rpool/data/vm-9012*
lrwxrwxrwx 1 root root 14 May  8 15:50 /dev/zvol/rpool/data/vm-156-cloudinit -> ../../../zd976
lrwxrwxrwx 1 root root 17 May  8 16:11 /dev/zvol/rpool/data/vm-156-disk-0 -> ../../../zd528p16
lrwxrwxrwx 1 root root 16 May  8 16:11 /dev/zvol/rpool/data/vm-156-disk-0-part1 -> ../../../zd528p1
lrwxrwxrwx 1 root root 17 May  8 16:11 /dev/zvol/rpool/data/vm-156-disk-0-part14 -> ../../../zd528p14
lrwxrwxrwx 1 root root 17 May  8 16:11 /dev/zvol/rpool/data/vm-156-disk-0-part15 -> ../../../zd528p15
lrwxrwxrwx 1 root root 14 May  8 16:11 /dev/zvol/rpool/data/vm-9012-cloudinit -> ../../../zd416
lrwxrwxrwx 1 root root 14 May  8 16:12 /dev/zvol/rpool/data/vm-9012-disk-0 -> ../../../zd992
lrwxrwxrwx 1 root root 17 May  8 16:12 /dev/zvol/rpool/data/vm-9012-disk-0-part1 -> ../../../zd992p16
lrwxrwxrwx 1 root root 17 May  8 16:12 /dev/zvol/rpool/data/vm-9012-disk-0-part14 -> ../../../zd992p14
lrwxrwxrwx 1 root root 17 May  8 16:12 /dev/zvol/rpool/data/vm-9012-disk-0-part15 -> ../../../zd992p15

Here the logs of the import when i deleted both disks and imported into the other vm first:
Bash: 
root@pve1:~# ls -l /dev/zvol/rpool/data/vm-156* /dev/zvol/rpool/data/vm-9012*
lrwxrwxrwx 1 root root 14 May  8 15:50 /dev/zvol/rpool/data/vm-156-cloudinit -> ../../../zd976
lrwxrwxrwx 1 root root 14 May  8 16:11 /dev/zvol/rpool/data/vm-9012-cloudinit -> ../../../zd416

root@pve1:~# qm importdisk 9012 noble-server-cloudimg-amd64.img local-zfs > /dev/null
root@pve1:~# qm importdisk 156 noble-server-cloudimg-amd64.img local-zfs > /dev/null

root@pve1:~# ls -l /dev/zvol/rpool/data/vm-156* /dev/zvol/rpool/data/vm-9012*
lrwxrwxrwx 1 root root 14 May  8 15:50 /dev/zvol/rpool/data/vm-156-cloudinit -> ../../../zd976
lrwxrwxrwx 1 root root 14 May  8 16:16 /dev/zvol/rpool/data/vm-156-disk-0 -> ../../../zd992
lrwxrwxrwx 1 root root 17 May  8 16:16 /dev/zvol/rpool/data/vm-156-disk-0-part1 -> ../../../zd992p16
lrwxrwxrwx 1 root root 17 May  8 16:16 /dev/zvol/rpool/data/vm-156-disk-0-part14 -> ../../../zd992p14
lrwxrwxrwx 1 root root 17 May  8 16:16 /dev/zvol/rpool/data/vm-156-disk-0-part15 -> ../../../zd992p15
lrwxrwxrwx 1 root root 14 May  8 16:11 /dev/zvol/rpool/data/vm-9012-cloudinit -> ../../../zd416
lrwxrwxrwx 1 root root 17 May  8 16:16 /dev/zvol/rpool/data/vm-9012-disk-0 -> ../../../zd528p16
lrwxrwxrwx 1 root root 16 May  8 16:16 /dev/zvol/rpool/data/vm-9012-disk-0-part1 -> ../../../zd528p1
lrwxrwxrwx 1 root root 17 May  8 16:16 /dev/zvol/rpool/data/vm-9012-disk-0-part14 -> ../../../zd528p14
lrwxrwxrwx 1 root root 17 May  8 16:16 /dev/zvol/rpool/data/vm-9012-disk-0-part15 -> ../../../zd528p15

Not running the latest Proxmox version. 8.1 like OP.
Bash: 
root@pve1:~# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.11-7-pve)
pve-manager: 8.1.4 (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5: 6.5.11-7
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.2-1
proxmox-backup-file-restore: 3.1.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-3
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.2-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1
 
Last edited:
Just a quick update in case someone else encounters the same issue.

I updated two servers to PVE 8.2.2 and it seems like the issue is gone.

On one server i wasn't able to import the disk and get the same disk name (zd528) because i migrated all important vms to another node, but so far all disk imports were ok.

On the other server i imported the disk before the update and the issue was present (wrong partition links). After update and reboot the links of this vm disk were fixed.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back