[SOLVED] Anyone else having issues shutting down Turnkey Linux appliances from PVE host?

[proxwolfe]

I noticed recently that a few of my VMs won't shutdown (the request times out) when I give the signal on the PVE host (or when this is triggered by, say, a backup run). I am using qemu-guest-agent in all my VMs.

Those VMs that don't shutdown seem to have in common that they are Turnkey Linux appliances. I have been using TKL appliances for a while but I don't remember ever seeing such a problem. However, recently, I have moved more VMs to TKL bases and I have upgraded them from Debian 10 to Debian 11 (or installed directly the newer version of the appliance).

There is this bug report according to which there was a problem in Debian 11 that caused the shutdown command not to work (if sent via qemu-guest-agent): #1001795 - sysvinit-core: Time parsing bug in shutdown - Debian Bug report logs which could fit my profile. But this bug seems to have been repaired long ago.

Is anyone else seeing this behavior or does anyone know the reason?

Thanks!

 

[Chris]

Hi,
do you see something in the journal of the guests? Is the qemu-guest-agent service up and running within the VM?

 

[proxwolfe]

Qemu-guest-agent is up and running - I can see the VM's IP address in the GUI.

Inside the VM, journalctl shows me: qemu-ga[xxx]: info: guest-shutdown called, mode: (null)

 

[Chris]

Inside the VM, journalctl show me: qemu-ga[xxx]: info: guest-shutdown called, mode: (null)

Nothing further? Can you still ping the guest agent after the guest-shutdown call? qm agent <vmid> ping

 

[proxwolfe]

No, nothing further.

When I ping the guest agent, I don't get any response (no time out or anything).

(I can ping the VM with a normal ping, though).

 

[Chris]

No, nothing further.

When I ping the guest agent, I don't get any response (no time out or anything).

(I can ping the VM with a normal ping, though).

If it just returns without an error, then the guest agent is still running... No issue there. Please share your pveversion -v and the Turnkey Linux Appliance Image you used.

 

[proxwolfe]

pveversion -v

proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-16 (running version: 7.4-16/0f39f621)
pve-kernel-5.15: 7.4-4
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.107-1-pve: 5.15.107-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.85-1-pve: 5.15.85-1
pve-kernel-5.15.83-1-pve: 5.15.83-1
pve-kernel-5.15.64-1-pve: 5.15.64-1
pve-kernel-5.15.53-1-pve: 5.15.53-1
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph: 17.2.6-pve1
ceph-fuse: 17.2.6-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-4
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

I first noticed it with TKL-Nextcloud 17.2 (fully updated to this day)

 

[proxwolfe]

So, have you been able to reproduce the issue?

 

[Chris]

Hi,
yes, sorry I did not find the time to look into this in more detail. It seems that the guest agent call to the VM gets passed correctly, but the execution of the shutdown within the VM fails.

You should see a

# qm agent <VMID> shutdown
{
   "error" : {
      "class" : "GenericError",
      "desc" : "child process has failed to shutdown"
   }
}

Unfortunately I did not find the time to look into this further, but this might be of interest to you https://www.mail-archive.com/pve-user@lists.proxmox.com/msg01640.html

Edit: Just tested, by installing systemd and dbus, shutdown via qemu guest agent works as expected on my test-installation, so you might give that a try.

 

[proxwolfe]

Happy to confirm that installing dbus on two of my TKL appliances did the trick (I did not also install systemd, that seems to be installed already).

The only other comment I have is that it did not work right after the installation of dbus, I needed to reboot the VM once.

Thanks for your help. Much appreciated!

 

[JeremyDavis]

Hi all,

I work with TurnKey and until I stumbled across this thread, we were unaware of this issue. So thanks for documenting it and it's cause.

I'll try to be a bit more vigilant checking these forums, but if you ever care to share this sort of stuff with us in future, please feel free to email support @ TurnKeyLinux.org and/or ping me here.

FWIW dbus is not a hard dependency of qemu-guest-agent[1] or systemd[2] (it is a "recommends" for systemd but to keep the ISOs as lean as possible, we do not install recommends by default). Hence why it was not pre-installed.

[1] https://packages.debian.org/bullseye/qemu-guest-agent
[2] https://packages.debian.org/bullseye/systemd

FWIW our next release (v18.0) will include dbus pre-installed by default. Despite it being a tool designed for use in desktops (hence the 'd' prefix), it seems that it has essentially become a hard requirement for all systems - even headless ones (despite it still not being a hard depends).

 

[proxwolfe]

Hi Jeremy,

Sorry, it would have made sense to report this to the TKL forums as well after resolving it.

I shall try and keep that in mind, should I ever have another TKL related issue.

Great to hear that dbus will be included in v18.

Cheers