Background
I'm creating a template for Ubuntu 22.04 and 24.04, which was working until a couple of days ago. After a Proxmox update (using the built in updater) it doesn't work anymore. Steps:
The Error
The last step is now failing (importing the disk):
Troubleshooting Output
The file exists, so does the CEPH RBD Pool and the config + keyring:
Also, the CEPH environment itself seems clean, I can also see list other objects:
Versions
Additional Context
The same issue also appears when cloning a template that existed for ages (full clone only, linked clone works and starts). Seems to be something deeper, although I have no idea what's going on.
I can work around it by creating an empty disk first and then overwrite it manually. then it does show the same behaviour as the existing template (full clones don't work)
I'm creating a template for Ubuntu 22.04 and 24.04, which was working until a couple of days ago. After a Proxmox update (using the built in updater) it doesn't work anymore. Steps:
Bash:
# For Ubuntu 22.04:
export OS_VERSION=jammy
export PM_TEMPLATE_NUMBER=9000
# Prepare image
wget https://cloud-images.ubuntu.com/${OS_VERSION}/current/${OS_VERSION}-server-cloudimg-amd64.img
virt-customize -a ${OS_VERSION}-server-cloudimg-amd64.img --install qemu-guest-agent --root-password password:mkspdefaultpass
qemu-img resize ${OS_VERSION}-server-cloudimg-amd64.img 16G
# Create VM and import
qm create ${PM_TEMPLATE_NUMBER} --name "Ubuntu-${OS_VERSION^}-Cloud-Init" --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
qm disk import ${PM_TEMPLATE_NUMBER} ${OS_VERSION}-server-cloudimg-amd64.img proxmox-vmsThe Error
The last step is now failing (importing the disk):
Code:
importing disk 'jammy-server-cloudimg-amd64.img' to VM 9001 ...
transferred 0.0 B of 16.0 GiB (0.00%)
qemu-img: Could not open 'zeroinit:rbd:proxmox-vms/vm-9001-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/proxmox-vms.keyring': Could not open 'rbd:proxmox-vms/vm-9001-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/proxmox-vms.keyring': No such file or directory
Removing image: 1% complete...
...
Removing image: 99% complete...
Removing image: 100% complete...done.
copy failed: command '/usr/bin/qemu-img convert -p -n -O raw jammy-server-cloudimg-amd64.img 'zeroinit:rbd:proxmox-vms/vm-9001-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/proxmox-vms.keyring'' failed: exit code 1Troubleshooting Output
The file exists, so does the CEPH RBD Pool and the config + keyring:
Code:
root@proxmox01:~# ls | grep jammy-server-cloudimg-amd64.img
jammy-server-cloudimg-amd64.img
root@proxmox01:~# ls /etc/pve/priv/ceph/proxmox-vms.keyring
/etc/pve/priv/ceph/proxmox-vms.keyring
root@proxmox01:~# ls /etc/pve/ceph.conf
/etc/pve/ceph.confAlso, the CEPH environment itself seems clean, I can also see list other objects:
Code:
root@proxmox01:~# rbd ls -l proxmox-vms
NAME SIZE PARENT FMT PROT LOCK
base-9000-disk-0 16 GiB 2
base-9000-disk-0@__base__ 16 GiB 2 yes
vm-100-cloudinit 4 MiB 2
vm-100-disk-0 16 GiB 2 excl
vm-100-disk-1 4 MiB 2 excl
vm-101-disk-0 128 GiB 2 excl
# ... More output omitted
vm-113-disk-0 2 GiB 2 excl
vm-9000-cloudinit 4 MiB 2
root@proxmox01:~# ceph status
cluster:
id: d88b5090-dde9-41c0-a3d4-e04a1e212112
health: HEALTH_OK
services:
mon: 3 daemons, quorum proxmox01,proxmox02,proxmox03 (age 13m)
mgr: proxmox01(active, since 72m), standbys: proxmox02, proxmox03
mds: 2/2 daemons up, 1 standby
osd: 12 osds: 12 up (since 66m), 12 in (since 6M)
data:
volumes: 2/2 healthy
pools: 7 pools, 289 pgs
objects: 241.41k objects, 907 GiB
usage: 2.6 TiB used, 7.8 TiB / 10 TiB avail
pgs: 289 active+clean
io:
client: 5.0 KiB/s rd, 1.6 MiB/s wr, 0 op/s rd, 176 op/s wrVersions
Code:
root@proxmox01:~# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-2-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.5.11-4-pve-signed: 6.5.11-4
ceph: 18.2.2-pve1
ceph-fuse: 18.2.2-pve1
corosync: 3.1.7-pve3
criu: 3.17.1-2
dnsmasq: 2.89-1
frr-pythontools: 8.5.2-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1Additional Context
The same issue also appears when cloning a template that existed for ages (full clone only, linked clone works and starts). Seems to be something deeper, although I have no idea what's going on.
Code:
create full clone of drive ide2 (proxmox-vms:vm-9000-cloudinit)
create full clone of drive scsi0 (proxmox-vms:base-9000-disk-0)
transferred 0.0 B of 16.0 GiB (0.00%)
qemu-img: Could not open 'zeroinit:rbd:proxmox-vms/vm-112-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/proxmox-vms.keyring': Could not open 'rbd:proxmox-vms/vm-112-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/proxmox-vms.keyring': No such file or directory
Removing image: 100% complete...done.
Removing image: 1% complete...
...
Removing image: 99% complete...
Removing image: 100% complete...done.
TASK ERROR: clone failed: copy failed: command '/usr/bin/qemu-img convert -p -n -f raw -O raw 'rbd:proxmox-vms/base-9000-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/proxmox-vms.keyring' 'zeroinit:rbd:proxmox-vms/vm-112-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/proxmox-vms.keyring'' failed: exit code 1I can work around it by creating an empty disk first and then overwrite it manually. then it does show the same behaviour as the existing template (full clones don't work)
Code:
qm set ${PM_TEMPLATE_NUMBER} --scsihw virtio-scsi-pci --scsi0 proxmox-vms:16
/usr/bin/qemu-img convert -p -n -O raw ${OS_VERSION}-server-cloudimg-amd64.img "rbd:proxmox-vms/vm-${PM_TEMPLATE_NUMBER}-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/proxmox-vms.keyring"
Last edited:
