Template on shared storage

[kverqus]

In my lab I'm running a three node cluster with a shared DAS storage solution. I have no issues manually installing virtual machines, but as soon as I create a template with the script below and clone a machine from the template it is unable to boot. When booting the template I end up at the "UEFI Interactive Shell v2.2" and I'm dropped into the shell. It would appear that the virtual machine fails to find its boot drive. However if I use the same script, but use local-lvm instead of shared_storage it works flawlessly to clone from the template. What could be the cause of this? My PVE environment is running on version 8.1.4.

#! /bin/bash
# https://github.com/UntouchedWagons/Ubuntu-CloudInit-Docs/tree/main

VMID=9000
STORAGE=shared_storage
USER=ansible

set -x
rm -f noble-server-cloudimg-amd64.img
wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
qemu-img resize noble-server-cloudimg-amd64.img 8G
qm destroy $VMID
qm create $VMID --name "ubuntu-noble-template" --ostype l26 \
    --memory 1024 --balloon 0 \
    --agent 1 \
    --bios ovmf --machine q35 --efidisk0 $STORAGE:0,pre-enrolled-keys=0 \
    --cpu host --cores 1 --numa 1 \
    --vga serial0 --serial0 socket  \
    --net0 virtio,bridge=vmbr0,mtu=1
qm importdisk $VMID noble-server-cloudimg-amd64.img $STORAGE
qm set $VMID --scsihw virtio-scsi-single --virtio0 $STORAGE:vm-$VMID-disk-1,iothread=1
qm set $VMID --boot order=virtio0
qm set $VMID --ide2 $STORAGE:cloudinit

cat << EOF | tee /var/lib/vz/snippets/ubuntu.yaml
#cloud-config
runcmd:
    - apt-get update
    - apt-get install -y qemu-guest-agent
    - systemctl enable ssh
    - reboot
# Taken from https://forum.proxmox.com/threads/combining-custom-cloud-init-with-auto-generated.59008/page-3#post-428772
EOF

qm set $VMID --cicustom "vendor=local:snippets/ubuntu.yaml"
qm set $VMID --tags ubuntu,noble,terraform
qm set $VMID --ciuser $USER
qm set $VMID --sshkeys ~/.ssh/ansible.pub
qm set $VMID --ipconfig0 ip=dhcp
qm template $VMID

 

[bbgeek17]

The script works fine for me. The template is created and when cloned boots without issues.
You can optimize it by commenting out image remove/download/resize on each script run.

May be your shared_storage (whatever it is) does not handle EFI disk correctly? As a test change the location of only EFI disk to local storage.
Are you saying when you use the script with shared_storage as target, and don't convert the VM to template as last step - it boots ok?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[kverqus]

The script works fine for me. The template is created and when cloned boots without issues.
You can optimize it by commenting out image remove/download/resize on each script run.

May be your shared_storage (whatever it is) does not handle EFI disk correctly? As a test change the location of only EFI disk to local storage.
Are you saying when you use the script with shared_storage as target, and don't convert the VM to template as last step - it boots ok?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

After writing my initial post I did do some tests of that nature. When virtio0 is moved from the shared storage back to the local storage (with the efi disk and cloudinit still on the shared storage) it boots just fine.

No, no matter what I've tried so far with keeping all of the files on shared_storage has resultet in the VM not booting. So far I've only managed to successfully boot an Ubuntu VM from the shared_storage when I either install it manually (from the regular Ubuntu iso) or when I've moved the virtio0 to the local storage.

 

[bbgeek17]

No, no matter what I've tried so far with keeping all of the files on shared_storage has resultet in the VM not booting. So far I've only managed to successfully boot an Ubuntu VM from the shared_storage when I either install it manually (from the regular Ubuntu iso) or when I've moved the virtio0 to the local storage.

Sounds like your shared storage may be lying about writes when a clone is done. What is this shared storage?
If it is file based - did you md5sum resulting files?

Again, I don't think anything is wrong with the script/procedure. It works fine with our storage, but it is certainly different than the one you have.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[kverqus]

Sounds like your shared storage may be lying about writes when a clone is done. What is this shared storage?
If it is file based - did you md5sum resulting files?

Again, I don't think anything is wrong with the script/procedure. It works fine with our storage, but it is certainly different than the one you have.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

The shared storage is a Dell PowerVault ME4012 configured as LVM, so it is block based.

I agree that something is up with the shared storage, perhaps not that it's lying but it's certainly doing something funny.