Hi,
I have a PVE cluster and am having issues shutting down Centos servers.
If I install a fresh Centos 9 Stream image (CentOS-Stream-9-latest-x86_64-dvd1.iso - downloaded 25 August), without doing 'dnf update', I can shut the server down fine (either from within the guest or using the PVE GUI).
However, once I do 'dnf update' and reboot the server, I can no longer shut the server down. The console gets to the below, and I have to select 'Stop' in the GUI:
[ 42.361088] audit: type=1334 audit(1694234031.375:118): prog-id=19 op=UNLOAD
[ 42.361098] audit: type=1334 audit(1694234031.375:119): prog-id=18 op=UNLOAD
[ 42.406041] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 42.409091] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[ 42.412701] systemd-journald[565]: Received SIGTERM from PID 1 (systemd-shutdow).
[ 42.419066] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 42.421532] systemd-shutdown[1]: Unmounting file systems.
[ 42.422107] [4358]: Remounting '/' read-only with options 'seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota'.
[ 42.432987] systemd-shutdown[1]: All filesystems unmounted.
[ 42.432992] systemd-shutdown[1]: Deactivating swaps.
[ 42.433049] systemd-shutdown[1]: All swaps deactivated.
[ 42.433052] systemd-shutdown[1]: Detaching loop devices.
[ 42.433170] systemd-shutdown[1]: All loop devices detached.
[ 42.433173] systemd-shutdown[1]: Stopping MD devices.
[ 42.433230] systemd-shutdown[1]: All MD devices stopped.
[ 42.433233] systemd-shutdown[1]: Detaching DM devices.
[ 42.433830] systemd-shutdown[1]: Detaching DM /dev/dm-1 (253:1).
[ 42.434058] dm-1: detected capacity change from 1048576 to 0
[ 42.439425] systemd-shutdown[1]: Not all DM devices detached, 1 left.
[ 42.439509] systemd-shutdown[1]: Detaching DM devices.
[ 42.439910] systemd-shutdown[1]: Not all DM devices detached, 1 left.
[ 42.439914] systemd-shutdown[1]: Cannot finalize remaining DM devices, continuing.
[ 42.445432] systemd-shutdown[1]: Successfully changed into root pivot.
[ 42.445435] systemd-shutdown[1]: Returning to initrd...
dracut Warning: Killing all remaining processes
[ 42.486942] dracut Warning: Killing all remaining processes
[ 42.514147] XFS (dm-0): Unmounting Filesystem
dracut Warning: Unmounted /oldroot.
[ 42.521743] dracut Warning: Unmounted /oldroot.
[ 42.534802] dracut: Disassembling device-mapper devices
[ 42.539593] dm-0: detected capacity change from 30400512 to 0
Powering off.
[ 42.553601] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[ 42.554725] ACPI: PM: Preparing to enter system sleep state S5
[ 42.555506] reboot: Power down
[ 42.556475] dracut Warning: poweroff failed!
dracut Warning: poweroff failed!
I have looked at lots of other threads and have tried the following:
- check the qemu-guest-agent is running - OK
- looked at the journatctl logs - it all looks fine and gets to the journal shutting down
- tried ACPI installed and enabled/disabled
I have a snapshot of the guest before the update and whenever I revert to that it starts working again. Then I do 'dnf update' and it starts breaking. I've even tried completely from fresh again, and it does the same.
The two kernel versions are:
Works:
Linux localhost.localdomain 5.14.0-362.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 28 20:43:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Doesn't work:
Linux localhost.localdomain 5.14.0-354.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Aug 10 21:06:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Is there anywhere else I can look to narrow down what is happening.
Thanks in advance
I have a PVE cluster and am having issues shutting down Centos servers.
If I install a fresh Centos 9 Stream image (CentOS-Stream-9-latest-x86_64-dvd1.iso - downloaded 25 August), without doing 'dnf update', I can shut the server down fine (either from within the guest or using the PVE GUI).
However, once I do 'dnf update' and reboot the server, I can no longer shut the server down. The console gets to the below, and I have to select 'Stop' in the GUI:
[ 42.361088] audit: type=1334 audit(1694234031.375:118): prog-id=19 op=UNLOAD
[ 42.361098] audit: type=1334 audit(1694234031.375:119): prog-id=18 op=UNLOAD
[ 42.406041] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 42.409091] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[ 42.412701] systemd-journald[565]: Received SIGTERM from PID 1 (systemd-shutdow).
[ 42.419066] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 42.421532] systemd-shutdown[1]: Unmounting file systems.
[ 42.422107] [4358]: Remounting '/' read-only with options 'seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota'.
[ 42.432987] systemd-shutdown[1]: All filesystems unmounted.
[ 42.432992] systemd-shutdown[1]: Deactivating swaps.
[ 42.433049] systemd-shutdown[1]: All swaps deactivated.
[ 42.433052] systemd-shutdown[1]: Detaching loop devices.
[ 42.433170] systemd-shutdown[1]: All loop devices detached.
[ 42.433173] systemd-shutdown[1]: Stopping MD devices.
[ 42.433230] systemd-shutdown[1]: All MD devices stopped.
[ 42.433233] systemd-shutdown[1]: Detaching DM devices.
[ 42.433830] systemd-shutdown[1]: Detaching DM /dev/dm-1 (253:1).
[ 42.434058] dm-1: detected capacity change from 1048576 to 0
[ 42.439425] systemd-shutdown[1]: Not all DM devices detached, 1 left.
[ 42.439509] systemd-shutdown[1]: Detaching DM devices.
[ 42.439910] systemd-shutdown[1]: Not all DM devices detached, 1 left.
[ 42.439914] systemd-shutdown[1]: Cannot finalize remaining DM devices, continuing.
[ 42.445432] systemd-shutdown[1]: Successfully changed into root pivot.
[ 42.445435] systemd-shutdown[1]: Returning to initrd...
dracut Warning: Killing all remaining processes
[ 42.486942] dracut Warning: Killing all remaining processes
[ 42.514147] XFS (dm-0): Unmounting Filesystem
dracut Warning: Unmounted /oldroot.
[ 42.521743] dracut Warning: Unmounted /oldroot.
[ 42.534802] dracut: Disassembling device-mapper devices
[ 42.539593] dm-0: detected capacity change from 30400512 to 0
Powering off.
[ 42.553601] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[ 42.554725] ACPI: PM: Preparing to enter system sleep state S5
[ 42.555506] reboot: Power down
[ 42.556475] dracut Warning: poweroff failed!
dracut Warning: poweroff failed!
I have looked at lots of other threads and have tried the following:
- check the qemu-guest-agent is running - OK
- looked at the journatctl logs - it all looks fine and gets to the journal shutting down
- tried ACPI installed and enabled/disabled
I have a snapshot of the guest before the update and whenever I revert to that it starts working again. Then I do 'dnf update' and it starts breaking. I've even tried completely from fresh again, and it does the same.
The two kernel versions are:
Works:
Linux localhost.localdomain 5.14.0-362.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 28 20:43:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Doesn't work:
Linux localhost.localdomain 5.14.0-354.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Aug 10 21:06:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Is there anywhere else I can look to narrow down what is happening.
Thanks in advance
