PVE4 reboot hang at "reached target shutdown"

[ebiss]

pveversion  -v
proxmox-ve: 4.1-26 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-1 (running version: 4.1-1/2f9650d4)
pve-kernel-4.2.6-1-pve: 4.2.6-26
pve-kernel-4.2.2-1-pve: 4.2.2-16
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-29
qemu-server: 4.0-41
pve-firmware: 1.1-7
libpve-common-perl: 4.0-41
libpve-access-control: 4.0-10
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-17
pve-container: 1.0-32
pve-firewall: 2.0-14
pve-ha-manager: 1.0-14
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve1
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve6~jessie

The host is ProLiant DL360 G7 , have one vm (Windows 2008 R2) on it .

 

[dietmar]

longer than 3 minutes?

 

[ebiss]

Yes, seem longer than 3 minutes.

I try with https://pve.proxmox.com/wiki/Windows_2008_guest_best_practices#Power_settings

In order to make Windows 2008 R2 guest shutdown cleanly on the event of a shutdown (power button) on the Proxmox VE host you need make the following changes:
Control Panel -> System and Security -> Power Options -> Edit Plan Settings -> Turn off the display: Never

still hang there.

 

[dietmar]

Maybe you stopped pve-manager systemd unit manually before? As workaround, you should be able to open a guest console and stop the VM manually?

 

[ebiss]

Yes, first stop the VM manually, the host reboot is OK.

 

[marco44]

Hi,

I'm having this exact same problem. But I don't think I have any KVM VMs stuck while rebooting. I only have 3 linux KVM machines, which all shut down cleanly through the shutdown menu in the proxmox GUI. The rest is all LXC containers. The hang is not occurring all the time. But it is more than 3 minutes, that's for sure. I left the host stay in this state for 20 minutes last time.

 

[mar72kuss]

Hello, anyone has solved this problem? since a couple of weeks i'm performing the same issue. during shutdown process of pve (the physical server I mean) it hangs to " reached shutdown target" , thanks for the help

 

[silverado]

Same problem here: After a few days uptime I have shutdown all virtual machines first. Then I used the "Restart" Button in the Proxmox web GUI to reboot the host. The system hangs at "reached target shutdown". After approx. 10-15 minutes the reboot continues.

 

[fireon]

Sometimes we have the exact same problem on 4.1. Not on 4.2. Not yet.

 

[fireon]

Same problem still existing on actual PVE:

pveversion -v
proxmox-ve: 4.2-56 (running kernel: 4.4.13-1-pve)
pve-manager: 4.2-15 (running version: 4.2-15/6669ad2c)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.13-1-pve: 4.4.13-56
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-4.2.8-1-pve: 4.2.8-41
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.4.10-1-pve: 4.4.10-54
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-42
qemu-server: 4.0-83
pve-firmware: 1.1-8
libpve-common-perl: 4.0-70
libpve-access-control: 4.0-16
libpve-storage-perl: 4.0-55
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-19
pve-container: 1.0-70
pve-firewall: 2.0-29
pve-ha-manager: 1.0-32
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5.7-pve10~bpo80

On shutdown on reboot... but not always. Only sometimes. There is nothing what you can do, only wait... a long long time. On reboot/shutdown there are no VMs running.

Here some links:
http://superuser.com/questions/767414/why-does-systemd-hang-during-reboot
http://forums.debian.net/viewtopic.php?t=120835
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=788303
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1464917

I will not it clever...

 

[gosha]

The same problem too.
Even after manual shutdown all VM's and manual reboot nodes (for boot new kernel) - much more than three minutes waiting "reached target shutdown".

 

[silverado]

No progress here? I just tried it again, installed all available updates and... problem persists. After an uptime of 1 day I shut down all VMs first, then I hit the "Shutdown" button in the web GUI. The server hanged at "reached target shutdown". I waited 45 Minutes - nothing happened. Finally I had to power off the server manually.

I really can't understand how a modern operating system like Debian/Proxmox can fail in such a simple and basic task like shutting down.

 

[Boomstam]

I can confirm that this issue is not user specific. I have it too on the latest PVE 4.3 with just 1x VM Centos 7 that is already shutdown. When I hit Shutdown or /sbin/shutdown -h now via SSH, the server just hangs. Black screen on the physical monitor and totally unresponsive to SSH and keyboard input. I am really close to switch to Proxmox, but this sort of bugs keep me from coming over.

I am using 2x Intell 3510 SSD's on a intel onboard chip(ICH10R?) configured in ZFS raid-1.

--
I solved this issue for myself by updating the BIOS to the latest version. This problem was on a Supermicro machine. I didn't come across it with Dell or HP so far.

 

[pascal]

This just happened to me after updating from 4.3 to 4.4. This is on a Dell R710 using firmware (of october 2016 I think). I left it alone for 4h but it never reboots.

 

[joshin]

I've seen a similar I/O related hang & then reboot of the node when doing a disk migration to a pair of SSD drives in 'ZFS Raid1'. Shutdown works, but takes a long time.

I've freed up one server that has this issue, and will do some intensive testing on it over the weekend.

In addition, I have a possibly related issue with a 'can't start VM' error when the server thinks it's out of memory. A 64GB box should be able to run more than 3x 8GB virtual machines. Doing a force-clear of all Linux's caches fixes that.

I'm wondering if ZFS is too abusive on system memory by default, and when it clears its caches, takes long enough to cause the crash. A side effect of this memory use is the inability to start new VMs.

-J

 

[dnut]

I had similar issues using both Asus and supermicro boards (hadn't the chance to test other boards).
I noticed that proxmox hang during reboot/shutdown only when I configure access to nfs storage on proxmox.
I solved following the guidelines found here:
https://forum.proxmox.com/threads/nfs-share-umount-f-problem-device-is-busy.26777/#post-134490

Setting NFS "soft connection" options in each nfs storage in storage.cfg file, solved, more or less, this behaviour.

One of my NFS storage looks like this:

nfs: snapshot-daily
        path /mnt/pve/snapshot-daily
        server 192.168.1.25
        export /mnt/zvol/proxmox/snapshot-daily
        maxfiles 6
        content backup
        options vers=3,soft,timeo=1000,retrans=10,actimeo=1000,retry=10

This solved many issues I had with the reboot/shutdown process, but sometimes (randomly and rarely) never the less, proxmox hangs on reboot/shutdown at the stage "reached target shutdown".

Then sometimes, after 10-15 minutes it stops/reboot, sometimes it stays there until I manually reset the system.

 

[novascore.io]

+1 for mounted nfs share causing the issue, thanks dnut