[SOLVED] Virtual Machine reboot / shutdown / stop problems after upgrade from debian bullseye to bookworm.

A

AndrzejL

Member
Nov 8, 2020
40
12
13
I am running a proxmox 8.1.4

Code: 
root@andrzejl:~# apt search pve | grep installed
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
libproxmox-rs-perl/stable,now 0.3.3 amd64 [installed]
libpve-access-control/stable,now 8.0.7 all [installed]
libpve-apiclient-perl/stable,now 3.3.1 all [installed]
libpve-cluster-api-perl/stable,now 8.0.5 all [installed]
libpve-cluster-perl/stable,now 8.0.5 all [installed]
libpve-common-perl/stable,now 8.1.0 all [installed]
libpve-guest-common-perl/stable,now 5.0.6 all [installed]
libpve-http-server-perl/stable,now 5.0.5 all [installed]
libpve-network-perl/stable,now 0.9.5 all [installed,automatic]
libpve-notify-perl/stable,now 8.0.5 all [installed,automatic]
libpve-rs-perl/stable,now 0.8.8 amd64 [installed]
libpve-storage-perl/stable,now 8.0.5 all [installed]
libpve-u2f-server-perl/stable,now 1.2.0 amd64 [installed]
lxc-pve/stable,now 5.0.2-4 amd64 [installed]
novnc-pve/stable,now 1.4.0-3 all [installed]
proxmox-kernel-6.5.11-7-pve-signed/stable,now 6.5.11-7 amd64 [installed,automatic]
pve-cluster/stable,now 8.0.5 amd64 [installed]
pve-container/stable,now 5.0.8 all [installed]
pve-docs/stable,now 8.1.3 all [installed]
pve-edk2-firmware/stable,now 4.2023.08-3 all [installed]
pve-edk2-firmware-legacy/stable,now 4.2023.08-3 all [installed,automatic]
pve-edk2-firmware-ovmf/stable,now 4.2023.08-3 all [installed,automatic]
pve-firewall/stable,now 5.0.3 amd64 [installed]
pve-firmware/stable,now 3.9-1 all [installed]
pve-ha-manager/stable,now 4.0.3 amd64 [installed]
pve-i18n/stable,now 3.2.0 all [installed]
pve-kernel-5.15/now 7.4-9 all [installed,local]
pve-kernel-5.15.102-1-pve/now 5.15.102-1 amd64 [installed,local]
pve-kernel-5.15.131-2-pve/now 5.15.131-3 amd64 [installed,local]
pve-lxc-syscalld/stable,now 1.3.0 amd64 [installed]
pve-manager/stable,now 8.1.4 amd64 [installed]
pve-qemu-kvm/stable,now 8.1.2-6 amd64 [installed]
pve-xtermjs/stable,now 5.3.0-3 all [installed]
root@andrzejl:~#

It started as proxmox 7 installation but was upgraded to 8.

It is installed using the proxmox 7 iso not debian and built on top of that.

The version of debian it originally came with is bullseye but it was upgraded to bookworm.

Code: 
root@andrzejl:~# cat /etc/apt/sources.list
deb http://deb.debian.org/debian/ bookworm main contrib
deb http://security.debian.org/debian-security bookworm-security main contrib
deb http://deb.debian.org/debian/ bookworm-updates main contrib
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
root@andrzejl:~#

I do not have the paid subscription repo installed / active but the system is fully upgraded.

Code: 
apt -y update && apt -y upgrade && apt -y dist-upgrade && apt -y full-upgrade && apt -y --purge autoremove

It behaves very well as far as I can tell except for 1 virtual machine that gives me hassle.

To give you a bit of a background - I created a debian bullseye vm in the past, installed all my necessary basic apps and confed all the necessary configs and then this became my default debian VM. I do not install the system each time - I clone the default vm, assign new hostname and static IP etc and then all the necessary tools for this specific vm so lets say game server or some other tool. I have many of those cloned vms working with different purposes. At some stage all of them were upgraded from the debian bullseye to bookworm by changing the sources.list and running set of upgrade commands. ALL of my VMs behave well as far as I can tell except for 1 virtual machine that gives me hassle.

VM in question is ID 107 with IP address of 192.168.1.118

Its my nginx + ssl / php / mysqld machine.

It was created like every other machine on my server by cloning the default vm. It was upgraded the exactly same way as other vms... and yet its the only one that gives me issues.

It has qemu guest agent installed and active. And yet it throws a hissy fit every time I try to reboot / poweroff / stop the machine.
To stop it I need to remove the lock conf file... then I can stop it.

I cannot reboot the entire proxmox server because this particular vm will stop it until the vms lock conf is deleted and the machine is stopped. Only then the proxmox server lay down in peace. IF I run "poweroff -f" or "reboot -f" the vm will go down instantly.

I am requesting a help with trying to troubleshoot this particular machine.

2 things that may be of interest...

I have a repo added to sources.list:

Code: 
root@andrzejl:~# apt -y update && apt -y upgrade && apt -y dist-upgrade && apt -y full-upgrade && apt -y --purge autoremove

Code: 
W: https://packages.sury.org/php/dists/bookworm/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

Packages installed from the above repo are:

Code: 
libpcre3:
Installed: 2:8.45-1+0~20230620.10+debian12~1.gbp8792c4
Candidate: 2:8.45-1+0~20230620.10+debian12~1.gbp8792c4
Version table:
*** 2:8.45-1+0~20230620.10+debian12~1.gbp8792c4 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

2:8.39-15 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
php-common:
Installed: 2:94+0~20240121.50+debian12~1.gbpe06825
Candidate: 2:94+0~20240121.50+debian12~1.gbpe06825
Version table:
*** 2:94+0~20240121.50+debian12~1.gbpe06825 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

2:93 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
php8.2:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-apcu:
Installed: 5.1.23-1+0~20231125.39+debian12~1.gbp8bf49a
Candidate: 5.1.23-1+0~20231125.39+debian12~1.gbp8bf49a
Version table:
*** 5.1.23-1+0~20231125.39+debian12~1.gbp8bf49a 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

5.1.22+4.0.11-2 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
php8.2-bcmath:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-bz2:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-cli:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-common:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-curl:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-fpm:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-gd:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-gmp:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-igbinary:
Installed: 3.2.15-1+0~20231202.49+debian12~1.gbpe9ffe8
Candidate: 3.2.15-1+0~20231202.49+debian12~1.gbpe9ffe8
Version table:
*** 3.2.15-1+0~20231202.49+debian12~1.gbpe9ffe8 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

3.2.13-1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
php8.2-imagick:
Installed: 3.7.0-4+0~20231125.43+debian12~1.gbpbf7e27
Candidate: 3.7.0-4+0~20231125.43+debian12~1.gbpbf7e27
Version table:
*** 3.7.0-4+0~20231125.43+debian12~1.gbpbf7e27 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

3.7.0-4 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
php8.2-intl:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-ldap:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-mbstring:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-memcached:
Installed: 3.2.0++-1+0~20231206.58+debian12~1.gbpd57e5e
Candidate: 3.2.0++-1+0~20231206.58+debian12~1.gbpd57e5e
Version table:
*** 3.2.0++-1+0~20231206.58+debian12~1.gbpd57e5e 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

3.2.0+2.2.0-4 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
php8.2-msgpack:
Installed: 1:2.2.0-2+0~20231129.41+debian12~1.gbp8dd767
Candidate: 1:2.2.0-2+0~20231129.41+debian12~1.gbp8dd767
Version table:
*** 1:2.2.0-2+0~20231129.41+debian12~1.gbp8dd767 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

1:2.2.0~rc2-3 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
php8.2-mysql:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-opcache:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-readline:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-redis:
Installed: 6.0.2-1+0~20231128.56+debian12~1.gbp9f3071
Candidate: 6.0.2-1+0~20231128.56+debian12~1.gbp9f3071
Version table:
*** 6.0.2-1+0~20231128.56+debian12~1.gbp9f3071 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

5.3.7+4.3.0-3 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
php8.2-soap:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-sqlite3:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-xml:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages
php8.2-xmlrpc:
Installed: 3:1.0.0~rc3-6+0~20231125.13+debian12~1.gbp734d6c
Candidate: 3:1.0.0~rc3-6+0~20231125.13+debian12~1.gbp734d6c
Version table:
*** 3:1.0.0~rc3-6+0~20231125.13+debian12~1.gbp734d6c 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

3:1.0.0~rc3-6 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
php8.2-zip:
Installed: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Candidate: 8.2.15-1+0~20240120.39+debian12~1.gbp56f296
Version table:
*** 8.2.15-1+0~20240120.39+debian12~1.gbp56f296 500
500 https://packages.sury.org/php bookworm/main amd64 Packages

8.2.7-1~deb12u1 500
500 http://deb.debian.org/debian bookworm/main amd64 Packages
500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages

Everything else is from the default debian bookworm repo.

Not sure what else possibly I could add but I am guessing if you're interested in the subject and you know what you're doing - you're going to ask me the right questions to get this solved.

Thanks in advance for any replies / question and help provided.

Kindest regards.

AndrzejL
 

Attachments

  • 1706800713091.png
    1706800713091.png
    39.2 KB · Views: 3
Last edited:
AndrzejL said:
I do not have the paid subscription repo installed / active but the system is fully upgraded.
Click to expand...
So you didn't follow these instructions? https://pve.proxmox.com/wiki/Upgrade_from_7_to_8
AndrzejL said:
apt -y update && apt -y upgrade && apt -y dist-upgrade && apt -y full-upgrade && apt -y --purge autoremove
Click to expand...
Please check this: https://pve.proxmox.com/wiki/System_Software_Updates

AndrzejL said:
VM in question is ID 107 with IP address of 192.168.1.118
Click to expand...
Please post the config qm config 107.
 
sb-jw said:
So you didn't follow these instructions? https://pve.proxmox.com/wiki/Upgrade_from_7_to_8
Click to expand...

Partly, I did run the check script before upgrading but what I did was I replaced the bullseye in my sources.list with bookworm and run my upgrade routine listed above.

Code: 
root@andrzejl:~# cat /etc/apt/sources.list
deb http://deb.debian.org/debian/ bookworm main contrib
deb http://security.debian.org/debian-security bookworm-security main contrib
deb http://deb.debian.org/debian/ bookworm-updates main contrib
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription
root@andrzejl:~#

sb-jw said:
Please check this: https://pve.proxmox.com/wiki/System_Software_Updates
Click to expand...

Thank You for the link.

This is my upgrade routine so not very far away from what seems to be the right way.

Code: 
apt -y update && apt -y upgrade && apt -y dist-upgrade && apt -y full-upgrade && apt -y --purge autoremove

sb-jw said:
Please post the config qm config 107.
Click to expand...

Code: 
root@andrzejl:~# qm config 107
agent: 1
boot: order=sata0;ide2;net0
cores: 1
ide2: Storage_2TB_SSD_26032023:iso/gparted-live-1.5.0-1-amd64.iso,media=cdrom,size=488M
memory: 8192
meta: creation-qemu=7.2.0,ctime=1679794960
name: AndrzejLEUHTTPS1.118
net0: virtio=16:D4:90:9B:4E:0D,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
sata0: Storage_2TB_SSD_26032023:107/vm-107-disk-0.qcow2,size=500G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=44bcc70b-249a-408e-8ff5-02f4dbcddbbc
sockets: 2
startup: order=2
vmgenid: 3b4bceac-f915-4e5e-b25d-f65062dd6840
root@andrzejl:~#

Thank You for your reply. I appreciate You and Your time.

Please don't hesitate to scold me and ask for more details .

Kindest regards.

AndrzejL
 
AndrzejL said:
This is my upgrade routine so not very far away from what seems to be the right way.
Click to expand...
If you look at it in detail, you are already far from the recommended path. With an apt upgrade you can also completely kill the node.
AndrzejL said:
It has qemu guest agent installed and active. And yet it throws a hissy fit every time I try to reboot / poweroff / stop the machine.
To stop it I need to remove the lock conf file... then I can stop it.
Click to expand...
If you go to the VM in the Proxmox GUI and look at the overview, do you also see the IP address?

If you shut down the VM with poweroff, does it work quickly or does it also hang? If so, then it has nothing to do with Proxmox VE, then something is wrong in the VM.
 
sb-jw said:
If you look at it in detail, you are already far from the recommended path. With an apt upgrade you can also completely kill the node.
Click to expand...

I know... I like to live dangerously I am a badboy like that. But being serious - yes, I understand. Its not a production server and I am a risk taker. I learn by breaking and then trying to fix / understand my mess.

sb-jw said:
If you go to the VM in the Proxmox GUI and look at the overview, do you also see the IP address?

If you shut down the VM with poweroff, does it work quickly or does it also hang? If so, then it has nothing to do with Proxmox VE, then something is wrong in the VM.
Click to expand...

I was not sure which of the gui pages you meant as there is no overview tab. Did You meant the VM or the node?

1706807380749.png

1706807343659.png

When I run poweroff the process takes looong time and this is displayed in the journalctl of the node while the vm 107 is shutting down (and never finishes):

Code: 
Feb 01 17:10:01 andrzejl CRON[52457]: pam_unix(cron:session): session closed for user root
Feb 01 17:10:01 andrzejl CRON[52458]: pam_unix(cron:session): session closed for user root
Feb 01 17:10:15 andrzejl pvedaemon[1369]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
Feb 01 17:10:35 andrzejl pvedaemon[1370]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
Feb 01 17:10:55 andrzejl pvedaemon[1369]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
Feb 01 17:11:15 andrzejl pvedaemon[1368]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
Feb 01 17:11:35 andrzejl pvedaemon[1369]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
Feb 01 17:11:55 andrzejl pvedaemon[1368]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
Feb 01 17:12:15 andrzejl pvedaemon[1370]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
Feb 01 17:12:35 andrzejl pvedaemon[1368]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout
Feb 01 17:14:30 andrzejl pvedaemon[1369]: VM 107 qmp command failed - VM 107 qmp command 'guest-ping' failed - got timeout

Thanking You!

Kindest regards.

AndrzejL
 
You also have to install the agent in the VM, just activating it in Proxmox VE isn't enough, you're just saying that the host can reach the VM via agent and that's exactly what fails because you apparently haven't installed it.

The overview also says that it's not running, that's exactly where the IP address should be when it's running.

See: https://pve.proxmox.com/wiki/Qemu-guest-agent
 
sb-jw said:
You also have to install the agent in the VM, just activating it in Proxmox VE isn't enough, you're just saying that the host can reach the VM via agent and that's exactly what fails because you apparently haven't installed it.

The overview also says that it's not running, that's exactly where the IP address should be when it's running.

See: https://pve.proxmox.com/wiki/Qemu-guest-agent
Click to expand...

Qemu guest agent is installed and running in the guest kind Sir.

Code: 
root@andrzejl:~# apt search qemu-guest-agent
Sorting... Done
Full Text Search... Done
qemu-guest-agent/stable,now 1:7.2+dfsg-7+deb12u3 amd64 [installed]
  Guest-side qemu-system agent


root@andrzejl:~# systemctl status qemu-guest-agent
● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static)
     Active: active (running) since Thu 2024-02-01 17:20:32 GMT; 12min ago
   Main PID: 702 (qemu-ga)
      Tasks: 2 (limit: 9484)
     Memory: 1.5M
        CPU: 441ms
     CGroup: /system.slice/qemu-guest-agent.service
             └─702 /usr/sbin/qemu-ga


Feb 01 17:20:32 andrzejl.eu systemd[1]: Started qemu-guest-agent.service - QEMU Guest Agent.
root@andrzejl:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether a6:2c:20:2e:6c:e2 brd ff:ff:ff:ff:ff:ff permaddr 16:d4:90:9b:4e:0d
    altname enp0s18
    inet 192.168.1.118/24 brd 192.168.1.255 scope global ens18
       valid_lft forever preferred_lft forever
root@andrzejl:~#

Other machines that reboot / shutdown / stop without issues have the exactly same qemu-guest-agent package installed and running as it was configured on the original "default vm" before cloning. No machine shows the IP on the summary and options tab.

Kindest regards.

AndrzejL
 
sb-jw said:
But that is definitely not correct. Then there is something absolutely wrong.
Click to expand...

On the screenshot that You linked to here https://pve.proxmox.com/wiki/Qemu-guest-agent there is no IP either.

1706810339032.png

I've only seen IP on container's summary pages. Not once on the VM config or summary tabs.

Could You please take a screenshot on one of your machines to show me where am I missing it / suppose to look for it? Please and thank you!

Kindest regards.

AndrzejL
 
1706811598650.png

That's exactly where you've come to the right place and under "IPs" it says "Guest Agent not running" - that's exactly where the IP should be. And I'm thousand percent sure that these should be there.
 
sb-jw said:
That's exactly where you've come to the right place and under "IPs" it says "Guest Agent not running" - that's exactly where the IP should be. And I'm thousand percent sure that these should be there.
Click to expand...

My appologies. That's me being a dumb schmuck...

You asked if it took long to poweroff and I to show it I ran poweroff and left it in a state of "powered off but not" limbo.

1706812148051.png

1706812372371.png

I have since stopped and restarted the VM.

This is the correct screenshot.

Kindest regards.

AndrzejL
 
Last edited:

Any other ideas? I am out off. Its very annoying as every single other vm clone from the same base is behaving 100% correctly.

Cheers.

AndrzejL
 
AndrzejL said:
When I run poweroff the process takes looong time and this is displayed in the journalctl of the node while the vm 107 is shutting down (and never finishes):
Click to expand...
But I asked what it looks like when you do it in the VM, not on the node. The question is whether the VM has a problem.
 
sb-jw said:
But I asked what it looks like when you do it in the VM, not on the node. The question is whether the VM has a problem.
Click to expand...

Yes yes, I run poweroff inside the VM but I kept -f the journalctl on the node. When you run the poweroff on the vm it cuts you off so you cannot see the journalctl - I thought that the entries on the node itself might give a clue to what's wrong. So if I understand correctly - the VM is trying to power off and hangs, does not really shutdown / restart or stop. I believe that its the VM that has a problem. Every other VM on this node seems to be act exactly as expected.

Anyway - I took some nap and my brain started working again.

Code: 
journalctl -b -2 -l -x --no-pager

I ran the above command on the vm to show the past journalctl entries and I have some info. The lines from redis-server keep repeating themselves.

Code: 
Feb 01 17:06:36 andrzejl.eu systemd[1]: clamav-daemon.service: Consumed 22.938s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit clamav-daemon.service completed and consumed the indicated resources.
Feb 01 17:06:39 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:39.065 * 1 changes in 3600 seconds. Saving...
Feb 01 17:06:39 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:39.065 * Background saving started by pid 2582534
Feb 01 17:06:39 andrzejl.eu redis-server[2582534]: 2582534:C 01 Feb 2024 19:06:39.066 # Failed opening the temp RDB file temp-2582534.rdb (in server root dir /) for saving: Read-only file system
Feb 01 17:06:39 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:39.166 # Background saving error
Feb 01 17:06:45 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:45.083 * 1 changes in 3600 seconds. Saving...
Feb 01 17:06:45 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:45.084 * Background saving started by pid 2582535
Feb 01 17:06:45 andrzejl.eu redis-server[2582535]: 2582535:C 01 Feb 2024 19:06:45.084 # Failed opening the temp RDB file temp-2582535.rdb (in server root dir /) for saving: Read-only file system
Feb 01 17:06:45 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:45.184 # Background saving error
Feb 01 17:06:51 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:51.002 * 1 changes in 3600 seconds. Saving...
Feb 01 17:06:51 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:51.003 * Background saving started by pid 2582536
Feb 01 17:06:51 andrzejl.eu redis-server[2582536]: 2582536:C 01 Feb 2024 19:06:51.004 # Failed opening the temp RDB file temp-2582536.rdb (in server root dir /) for saving: Read-only file system
Feb 01 17:06:51 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:51.105 # Background saving error
Feb 01 17:06:57 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:57.022 * 1 changes in 3600 seconds. Saving...
Feb 01 17:06:57 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:57.023 * Background saving started by pid 2582537
Feb 01 17:06:57 andrzejl.eu redis-server[2582537]: 2582537:C 01 Feb 2024 19:06:57.024 # Failed opening the temp RDB file temp-2582537.rdb (in server root dir /) for saving: Read-only file system
Feb 01 17:06:57 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:06:57.124 # Background saving error
Feb 01 17:07:03 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:03.042 * 1 changes in 3600 seconds. Saving...
Feb 01 17:07:03 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:03.043 * Background saving started by pid 2582538
Feb 01 17:07:03 andrzejl.eu redis-server[2582538]: 2582538:C 01 Feb 2024 19:07:03.043 # Failed opening the temp RDB file temp-2582538.rdb (in server root dir /) for saving: Read-only file system
Feb 01 17:07:03 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:03.143 # Background saving error
Feb 01 17:07:09 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:09.061 * 1 changes in 3600 seconds. Saving...
Feb 01 17:07:09 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:09.062 * Background saving started by pid 2582539
Feb 01 17:07:09 andrzejl.eu redis-server[2582539]: 2582539:C 01 Feb 2024 19:07:09.062 # Failed opening the temp RDB file temp-2582539.rdb (in server root dir /) for saving: Read-only file system
Feb 01 17:07:09 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:09.162 # Background saving error
Feb 01 17:07:15 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:15.080 * 1 changes in 3600 seconds. Saving...
Feb 01 17:07:15 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:15.081 * Background saving started by pid 2582540
Feb 01 17:07:15 andrzejl.eu redis-server[2582540]: 2582540:C 01 Feb 2024 19:07:15.081 # Failed opening the temp RDB file temp-2582540.rdb (in server root dir /) for saving: Read-only file system
Feb 01 17:07:15 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:15.182 # Background saving error
Feb 01 17:07:21 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:21.000 * 1 changes in 3600 seconds. Saving...
Feb 01 17:07:21 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:21.001 * Background saving started by pid 2582541
Feb 01 17:07:21 andrzejl.eu redis-server[2582541]: 2582541:C 01 Feb 2024 19:07:21.001 # Failed opening the temp RDB file temp-2582541.rdb (in server root dir /) for saving: Read-only file system
Feb 01 17:07:21 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:21.101 # Background saving error
Feb 01 17:07:26 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:26.016 * 1 changes in 3600 seconds. Saving...
Feb 01 17:07:26 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:26.017 * Background saving started by pid 2582542
Feb 01 17:07:26 andrzejl.eu redis-server[2582542]: 2582542:C 01 Feb 2024 19:07:26.017 # Failed opening the temp RDB file temp-2582542.rdb (in server root dir /) for saving: Read-only file system
Feb 01 17:07:26 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:26.117 # Background saving error
Feb 01 17:07:32 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:32.035 * 1 changes in 3600 seconds. Saving...
Feb 01 17:07:32 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:32.036 * Background saving started by pid 2582543
Feb 01 17:07:32 andrzejl.eu redis-server[2582543]: 2582543:C 01 Feb 2024 19:07:32.037 # Failed opening the temp RDB file temp-2582543.rdb (in server root dir /) for saving: Read-only file system
Feb 01 17:07:32 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:32.137 # Background saving error
Feb 01 17:07:38 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:38.055 * 1 changes in 3600 seconds. Saving...
Feb 01 17:07:38 andrzejl.eu redis-server[737]: 737:M 01 Feb 2024 19:07:38.057 * Background saving started by pid 2582544
Feb 01 17:07:38 andrzejl.eu redis-server[2582544]: 2582544:C 01 Feb 2024 19:07:38.057 # Failed opening the temp RDB file temp-2582544.rdb (in server root dir /) for saving: Read-only file system.

Its redis server trying to save some data but failing. I have something to duckduckgo now...

Regards.

Andrzej
 
sb-jw said:
But I asked what it looks like when you do it in the VM, not on the node. The question is whether the VM has a problem.
Click to expand...

Thank You! Solved!

Yes - it was a problem with redis-server, it was unable to save to drive due to bad redis.conf file.

Regards.

AndrzejL
 
Last edited:
You must log in or register to reply here.
Top Bottom