8.2.4 VM reboot ohne ersichtlichen Grund

[dScha]

Hallo,

in einem Cluster mit 3 Knoten ( v 8.2.4 alle Updates eingespielt ) gibt es eine VM mit centos und einer Oracle Datenbank.
Sobald die Datenbank gestartet wird, friert die Maschine ein und wird nach einiger Zeit automatisch neu gestartet.
Weder in den logs der VM noch im syslog des Nodes gibt es Hinweise auf die Ursache bzw. den reboot. In der VM und der proxmox UI werden auch unterschiedliche Uptimes angezeigt.

Wo kann ich mehr Informationen zum Grund des reboots finden ?

Was schon probiert wurde: VM auf anderen Knoten schieben, vm auf einen anderen Storage verschieben. Vm löschen und aus einem Backup wieder herstellen.
Die gleiche VM auf einem andern Cluster (v 8.2.2 - keine updates eingespielt) aus dem Backup wiederhergestellt funktioniert ohne Probleme. Daraus kann man schließen dass die Ursache nicht an der VM liegt

Für Hinweise zur Fehlersuche wäre ich sehr dankbar
VG

 

[dScha]

die vm hat 4 virtuelle Kerne, Typ kvm64 und 8GB Ram
Der (aktuelle) Node hat 40 Kerne und 128GB Ram. Es laufen noch 2 weitere vm's mit jeweils 4cpu / 4gram
Es sind also ausreichend Resourcen auf dem Node frei (110GB RAM/ 28CPU)

 

[dScha]

Hallo,
nein ein guest-agent ist nicht installiert.
Danke für den Link zum remote syslog server. Aber kann ich die logs nicht auch lokal auf dem Node einsehen ?

Noch eine Korrektur des ursprünglichen Eintrags: die Nodes im Cluster haben 8.2.4 und der DR Server 8.2.2

Wir werden jetzt einen Standalone Server mit 8.2.2 aufsetzen und die VM dort wieder herstellen. Sollte sie funktionieren, führen wir das Upgrade auf 8.2.4 durch und schauen ob dies zum Fehler führt

 

[dScha]

auf einem frisch installierten 8.1.4 läuft die vm ohne crash

 

[fiona]

Hi,
könntest Du bitte die Konfiguration der VM qm config <ID> und Ausgabe von pveversion -v auf dem Knoten, wo die Probleme auftreten senden? Gibt es im System-Log/Journal irgendwelche Meldungen rund um die Zeit, wo das Problem auftritt? Gibt es irgendwas in den Logs innerhalb vom Gast?

 

[dScha]

Hi Fiona,
sehr gerne. Der frisch aufgesetzen Test-node wurde aktualisiert:

root@pvetst:~# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-3-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.8-3
proxmox-kernel-6.8.8-3-pve-signed: 6.8.8-3
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
ceph-fuse: 17.2.7-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.13-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.2
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1


root@pvetst:~# qm config 100
agent: 0
boot: order=scsi0
cores: 4
memory: 8192
name: vmqSrvIs
net0: virtio=00:0c:29:b2:7f:4f,bridge=vmbr0,firewall=1
scsi0: local-lvm:vm-100-disk-0,size=16G
scsi1: local-lvm:vm-100-disk-1,size=120G
scsi2: local-lvm:vm-100-disk-2,size=64G
scsi3: local-lvm:vm-100-disk-3,size=64G
scsi4: local-lvm:vm-100-disk-4,size=64G
smbios1: uuid=c29a94a0-d436-49dc-b051-57fe2d47c40b
tags: centos;db;mimir;oracle12
vmgenid: 0c73495e-3397-4031-a1d1-44a8fc3426bd

Gast vor dem starten der DB:

[root@vmqsrvifds12 ~]# tail -n 30 /var/log/messages
Jul 24 10:59:40 localhost kernel: Key type dns_resolver registered
Jul 24 10:59:40 localhost kernel: Slow work thread pool: Starting up
Jul 24 10:59:40 localhost kernel: Slow work thread pool: Ready
Jul 24 10:59:40 localhost acpid: starting up
Jul 24 10:59:40 localhost acpid: 1 rule loaded
Jul 24 10:59:40 localhost acpid: waiting for events: event logging is off
Jul 24 10:59:41 localhost acpid: client connected from 2078[68:68]
Jul 24 10:59:41 localhost acpid: 1 client rule loaded
Jul 24 10:59:41 localhost pcscd: pcscdaemon.c:506:main() pcsc-lite 1.5.2 daemon ready.
Jul 24 10:59:43 localhost automount[2123]: lookup_read_master: lookup(nisplus): couldn't locate nis+ table auto.master
Jul 24 10:59:43 localhost kdump: kexec: loaded kdump kernel
Jul 24 10:59:43 localhost kdump: started up
Jul 24 10:59:44 localhost ntpd[2270]: ntpd 4.2.6p5@1.2349-o Tue May 31 10:09:21 UTC 2016 (1)
Jul 24 10:59:44 localhost ntpd[2271]: proto: precision = 0.305 usec
Jul 24 10:59:44 localhost ntpd[2271]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Jul 24 10:59:44 localhost ntpd[2271]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Jul 24 10:59:44 localhost ntpd[2271]: Listen and drop on 1 v6wildcard :: UDP 123
Jul 24 10:59:44 localhost ntpd[2271]: Listen normally on 2 lo 127.0.0.1 UDP 123
Jul 24 10:59:44 localhost ntpd[2271]: Listen normally on 3 eth1 192.168.1.84 UDP 123
Jul 24 10:59:44 localhost ntpd[2271]: Listen normally on 4 lo ::1 UDP 123
Jul 24 10:59:44 localhost ntpd[2271]: Listen normally on 5 eth1 fe80::20c:29ff:feb2:7f4f UDP 123
Jul 24 10:59:44 localhost ntpd[2271]: Listening on routing socket on fd #22 for interface updates
Jul 24 10:59:44 localhost ntpd[2271]: 0.0.0.0 c016 06 restart
Jul 24 10:59:44 localhost ntpd[2271]: 0.0.0.0 c012 02 freq_set kernel 19.189 PPM
Jul 24 10:59:46 localhost ntpd[2271]: 0.0.0.0 c61c 0c clock_step +1.005025 s
Jul 24 10:59:46 localhost ntpd[2271]: 0.0.0.0 c614 04 freq_mode
Jul 24 10:59:46 localhost abrtd: Init complete, entering main loop
Jul 24 10:59:47 localhost ntpd[2271]: 0.0.0.0 c618 08 no_sys_peer
Jul 24 10:59:55 localhost polkitd[2655]: started daemon version 0.96 using authority implementation `local' version `0.96'
Jul 24 10:59:55 localhost rtkit-daemon[2666]: Sucessfully made thread 2664 of process 2664 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11.

Node vor dem starten

root@pvetst:~# journalctl
Jul 24 10:56:27 pvetst systemd[1]: Started 100.scope.
Jul 24 10:56:29 pvetst kernel: tap100i0: entered promiscuous mode
Jul 24 10:56:29 pvetst kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Jul 24 10:56:29 pvetst kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Jul 24 10:56:29 pvetst kernel: fwpr100p0: entered allmulticast mode
Jul 24 10:56:29 pvetst kernel: fwpr100p0: entered promiscuous mode
Jul 24 10:56:29 pvetst kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Jul 24 10:56:29 pvetst kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
Jul 24 10:56:29 pvetst kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Jul 24 10:56:29 pvetst kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Jul 24 10:56:29 pvetst kernel: fwln100i0: entered allmulticast mode
Jul 24 10:56:29 pvetst kernel: fwln100i0: entered promiscuous mode
Jul 24 10:56:29 pvetst kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Jul 24 10:56:29 pvetst kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
Jul 24 10:56:29 pvetst kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Jul 24 10:56:29 pvetst kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Jul 24 10:56:29 pvetst kernel: tap100i0: entered allmulticast mode
Jul 24 10:56:29 pvetst kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Jul 24 10:56:29 pvetst kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
Jul 24 10:56:30 pvetst pvedaemon[1288]: <root@pam> end task UPID:pvetst:00000974:00006428:66A0C1BB:qmstart:100:root@pam: OK
Jul 24 10:58:17 pvetst pvedaemon[1289]: <root@pam> starting task UPID:pvetst:00000C0A:00008F39:66A0C229:vncproxy:100:root@pam:
Jul 24 10:58:17 pvetst pvedaemon[3082]: starting vnc proxy UPID:pvetst:00000C0A:00008F39:66A0C229:vncproxy:100:root@pam:
Jul 24 10:58:29 pvetst pvedaemon[1289]: <root@pam> end task UPID:pvetst:00000C0A:00008F39:66A0C229:vncproxy:100:root@pam: OK
Jul 24 10:58:32 pvetst pvedaemon[3137]: starting vnc proxy UPID:pvetst:00000C41:000094D0:66A0C238:vncproxy:100:root@pam:
Jul 24 10:58:32 pvetst pvedaemon[1287]: <root@pam> starting task UPID:pvetst:00000C41:000094D0:66A0C238:vncproxy:100:root@pam:
Jul 24 10:58:44 pvetst pvedaemon[1287]: <root@pam> end task UPID:pvetst:00000C41:000094D0:66A0C238:vncproxy:100:root@pam: OK
Jul 24 10:58:58 pvetst pvedaemon[3286]: starting vnc proxy UPID:pvetst:00000CD6:00009F35:66A0C252:vncproxy:100:root@pam:
Jul 24 10:58:58 pvetst pvedaemon[1288]: <root@pam> starting task UPID:pvetst:00000CD6:00009F35:66A0C252:vncproxy:100:root@pam:
Jul 24 10:59:08 pvetst pvedaemon[1287]: <root@pam> starting task UPID:pvetst:00000D0A:0000A328:66A0C25C:qmreset:100:root@pam:
Jul 24 10:59:08 pvetst pvedaemon[1287]: <root@pam> end task UPID:pvetst:00000D0A:0000A328:66A0C25C:qmreset:100:root@pam: OK

Dann Start der - Gast lässt sich nicht mehr pingen, friert ein. Nach einigen Minuten der automatische reboot
Node nach reboot:

root@pvetst:~# journalctl -f
Jul 24 10:58:32 pvetst pvedaemon[3137]: starting vnc proxy UPID:pvetst:00000C41:000094D0:66A0C238:vncproxy:100:root@pam:
Jul 24 10:58:32 pvetst pvedaemon[1287]: <root@pam> starting task UPID:pvetst:00000C41:000094D0:66A0C238:vncproxy:100:root@pam:
Jul 24 10:58:44 pvetst pvedaemon[1287]: <root@pam> end task UPID:pvetst:00000C41:000094D0:66A0C238:vncproxy:100:root@pam: OK
Jul 24 10:58:58 pvetst pvedaemon[3286]: starting vnc proxy UPID:pvetst:00000CD6:00009F35:66A0C252:vncproxy:100:root@pam:
Jul 24 10:58:58 pvetst pvedaemon[1288]: <root@pam> starting task UPID:pvetst:00000CD6:00009F35:66A0C252:vncproxy:100:root@pam:
Jul 24 10:59:08 pvetst pvedaemon[1287]: <root@pam> starting task UPID:pvetst:00000D0A:0000A328:66A0C25C:qmreset:100:root@pam:
Jul 24 10:59:08 pvetst pvedaemon[1287]: <root@pam> end task UPID:pvetst:00000D0A:0000A328:66A0C25C:qmreset:100:root@pam: OK
Jul 24 11:04:35 pvetst pvedaemon[1288]: <root@pam> end task UPID:pvetst:00000CD6:00009F35:66A0C252:vncproxy:100:root@pam: OK
Jul 24 11:05:14 pvetst pvedaemon[5085]: starting vnc proxy UPID:pvetst:000013DD:000131FA:66A0C3CA:vncproxy:100:root@pam:
Jul 24 11:05:14 pvetst pvedaemon[1289]: <root@pam> starting task UPID:pvetst:000013DD:000131FA:66A0C3CA:vncproxy:100:root@pam:
Jul 24 11:07:51 pvetst systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Jul 24 11:07:51 pvetst systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Jul 24 11:07:51 pvetst systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Jul 24 11:07:51 pvetst systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Jul 24 11:08:25 pvetst pvedaemon[1288]: <root@pam> successful auth for user 'root@pam'

Gast nach dem reboot:

Jul 24 10:59:44 localhost ntpd[2271]: Listen normally on 4 lo ::1 UDP 123
Jul 24 10:59:44 localhost ntpd[2271]: Listen normally on 5 eth1 fe80::20c:29ff:feb2:7f4f UDP 123
Jul 24 10:59:44 localhost ntpd[2271]: Listening on routing socket on fd #22 for interface updates
Jul 24 10:59:44 localhost ntpd[2271]: 0.0.0.0 c016 06 restart
Jul 24 10:59:44 localhost ntpd[2271]: 0.0.0.0 c012 02 freq_set kernel 19.189 PPM
Jul 24 10:59:46 localhost ntpd[2271]: 0.0.0.0 c61c 0c clock_step +1.005025 s
Jul 24 10:59:46 localhost ntpd[2271]: 0.0.0.0 c614 04 freq_mode
Jul 24 10:59:46 localhost abrtd: Init complete, entering main loop
Jul 24 10:59:47 localhost ntpd[2271]: 0.0.0.0 c618 08 no_sys_peer
Jul 24 10:59:55 localhost polkitd[2655]: started daemon version 0.96 using authority implementation `local' version `0.96'
Jul 24 10:59:55 localhost rtkit-daemon[2666]: Sucessfully made thread 2664 of process 2664 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11.
Jul 24 11:06:43 localhost kernel: imklog 5.8.10, log source = /proc/kmsg started.
Jul 24 11:06:43 localhost rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1660" x-info="http://www.rsyslog.com"] start
Jul 24 11:06:43 localhost kernel: Initializing cgroup subsys cpuset
Jul 24 11:06:43 localhost kernel: Initializing cgroup subsys cpu
Jul 24 11:06:43 localhost kernel: Linux version 2.6.32-642.4.2.el6.x86_64 (mockbuild@worker1.bsys.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Tue Aug 23 19:58:13 UTC 2016
Jul 24 11:06:43 localhost kernel: Command line: ro root=/dev/mapper/vg_c65base-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD  KEYBOARDTYPE=pc KEYTABLE=de rd_LVM_LV=vg_c65base/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=128M rd_LVM_LV=vg_c65base/lv_root rd_NO_DM rhgb quiet
Jul 24 11:06:43 localhost kernel: KERNEL supported cpus:
Jul 24 11:06:43 localhost kernel:  Intel GenuineIntel
Jul 24 11:06:43 localhost kernel:  AMD AuthenticAMD
Jul 24 11:06:43 localhost kernel:  Centaur CentaurHauls
Jul 24 11:06:43 localhost kernel: BIOS-provided physical RAM map:
Jul 24 11:06:43 localhost kernel: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Jul 24 11:06:43 localhost kernel: BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Jul 24 11:06:43 localhost kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Jul 24 11:06:43 localhost kernel: BIOS-e820: 0000000000100000 - 00000000bffde000 (usable)
Jul 24 11:06:43 localhost kernel: BIOS-e820: 00000000bffde000 - 00000000c0000000 (reserved)
Jul 24 11:06:43 localhost kernel: BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved)
Jul 24 11:06:43 localhost kernel: BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved)
Jul 24 11:06:43 localhost kernel: BIOS-e820: 0000000100000000 - 0000000240000000 (usable)
Jul 24 11:06:43 localhost kernel: SMBIOS version 2.8 @ 0xF5270
Jul 24 11:06:43 localhost kernel: SMBIOS 2.8 present.
Jul 24 11:06:43 localhost kernel: Hypervisor detected: KVM
Jul 24 11:06:43 localhost kernel: No AGP bridge found
Jul 24 11:06:43 localhost kernel: last_pfn = 0x240000 max_arch_pfn = 0x400000000
...

Gibt es noch weitere logs die hilfreich sein könnten ?

 

[fiona]

Was ist die Ausgabe von qm status 100 --verbose nach dem Einfrieren?

Du könntest auch Debugger und Debug-Symbole installieren mit apt install pve-qemu-kvm-dbgsym gdb und nach dem Einfrieren einen Stack-Trace erhalten mit gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/100.pid). Wenn das Problem in QEMU selbst ist, enthält das mit etwas Glück einen Hinweis.

 

[fiona]

Einen Versuch wert wäre auch den CPU Typ der VM auf x86-64-v2-AES zu stellen, den SCSI Controller auf VirtIO SCSI single und auf den Disken IO Thread anzuschalten.

 

[dScha]

root@pvetst:~# qm status 100 --verbose
balloon: 8589934592
ballooninfo:
        actual: 8589934592
        free_mem: 5271588864
        last_update: 1721817968
        major_page_faults: 2314
        max_mem: 8589934592
        mem_swapped_in: 0
        mem_swapped_out: 0
        minor_page_faults: 2383689
        total_mem: 8255610880
blockstat:
        scsi0:
                account_failed: 1
                account_invalid: 1
                failed_flush_operations: 0
                failed_rd_operations: 0
                failed_unmap_operations: 0
                failed_wr_operations: 0
                failed_zone_append_operations: 0
                flush_operations: 1019
                flush_total_time_ns: 183398088
                idle_time_ns: 55164351465
                invalid_flush_operations: 0
                invalid_rd_operations: 0
                invalid_unmap_operations: 0
                invalid_wr_operations: 0
                invalid_zone_append_operations: 0
                rd_bytes: 587158016
                rd_merged: 0
                rd_operations: 18675
                rd_total_time_ns: 74542739772
                timed_stats:
                unmap_bytes: 0
                unmap_merged: 0
                unmap_operations: 0
                unmap_total_time_ns: 0
                wr_bytes: 17624064
                wr_highest_offset: 11264016384
                wr_merged: 0
                wr_operations: 2116
                wr_total_time_ns: 529196259
                zone_append_bytes: 0
                zone_append_merged: 0
                zone_append_operations: 0
                zone_append_total_time_ns: 0
        scsi1:
                account_failed: 1
                account_invalid: 1
                failed_flush_operations: 0
                failed_rd_operations: 0
                failed_unmap_operations: 0
                failed_wr_operations: 0
                failed_zone_append_operations: 0
                flush_operations: 772
                flush_total_time_ns: 117834559
                idle_time_ns: 55149919945
                invalid_flush_operations: 0
                invalid_rd_operations: 0
                invalid_unmap_operations: 0
                invalid_wr_operations: 0
                invalid_zone_append_operations: 0
                rd_bytes: 148509184
                rd_merged: 0
                rd_operations: 5664
                rd_total_time_ns: 47527586010
                timed_stats:
                unmap_bytes: 0
                unmap_merged: 0
                unmap_operations: 0
                unmap_total_time_ns: 0
                wr_bytes: 4599808
                wr_highest_offset: 119995932160
                wr_merged: 0
                wr_operations: 489
                wr_total_time_ns: 117321041
                zone_append_bytes: 0
                zone_append_merged: 0
                zone_append_operations: 0
                zone_append_total_time_ns: 0
        scsi2:
                account_failed: 1
                account_invalid: 1
                failed_flush_operations: 0
                failed_rd_operations: 0
                failed_unmap_operations: 0
                failed_wr_operations: 0
                failed_zone_append_operations: 0
                flush_operations: 772
                flush_total_time_ns: 14185575
                idle_time_ns: 55163801089
                invalid_flush_operations: 0
                invalid_rd_operations: 0
                invalid_unmap_operations: 0
                invalid_wr_operations: 0
                invalid_zone_append_operations: 0
                rd_bytes: 52889088
                rd_merged: 0
                rd_operations: 1031
                rd_total_time_ns: 8176330509
                timed_stats:
                unmap_bytes: 0
                unmap_merged: 0
                unmap_operations: 0
                unmap_total_time_ns: 0
                wr_bytes: 53248
                wr_highest_offset: 51246739456
                wr_merged: 0
                wr_operations: 7
                wr_total_time_ns: 1646183
                zone_append_bytes: 0
                zone_append_merged: 0
                zone_append_operations: 0
                zone_append_total_time_ns: 0
        scsi3:
                account_failed: 1
                account_invalid: 1
                failed_flush_operations: 0
                failed_rd_operations: 0
                failed_unmap_operations: 0
                failed_wr_operations: 0
                failed_zone_append_operations: 0
                flush_operations: 772
                flush_total_time_ns: 20699756
                idle_time_ns: 55163688443
                invalid_flush_operations: 0
                invalid_rd_operations: 0
                invalid_unmap_operations: 0
                invalid_wr_operations: 0
                invalid_zone_append_operations: 0
                rd_bytes: 199591424
                rd_merged: 0
                rd_operations: 3133
                rd_total_time_ns: 22815044578
                timed_stats:
                unmap_bytes: 0
                unmap_merged: 0
                unmap_operations: 0
                unmap_total_time_ns: 0
                wr_bytes: 491520
                wr_highest_offset: 13109374976
                wr_merged: 0
                wr_operations: 58
                wr_total_time_ns: 11855564
                zone_append_bytes: 0
                zone_append_merged: 0
                zone_append_operations: 0
                zone_append_total_time_ns: 0
        scsi4:
                account_failed: 1
                account_invalid: 1
                failed_flush_operations: 0
                failed_rd_operations: 0
                failed_unmap_operations: 0
                failed_wr_operations: 0
                failed_zone_append_operations: 0
                flush_operations: 772
                flush_total_time_ns: 26751569
                idle_time_ns: 55163112182
                invalid_flush_operations: 0
                invalid_rd_operations: 0
                invalid_unmap_operations: 0
                invalid_wr_operations: 0
                invalid_zone_append_operations: 0
                rd_bytes: 20621312
                rd_merged: 0
                rd_operations: 1210
                rd_total_time_ns: 3200747229
                timed_stats:
                unmap_bytes: 0
                unmap_merged: 0
                unmap_operations: 0
                unmap_total_time_ns: 0
                wr_bytes: 839680
                wr_highest_offset: 13928689152
                wr_merged: 0
                wr_operations: 97
                wr_total_time_ns: 28366388
                zone_append_bytes: 0
                zone_append_merged: 0
                zone_append_operations: 0
                zone_append_total_time_ns: 0
cpus: 4
disk: 0
diskread: 1008769024
diskwrite: 23608320
freemem: 5271588864
maxdisk: 17179869184
maxmem: 8589934592
mem: 2984022016
name: vmqSrvIFDS
netin: 12683342
netout: 1118413
nics:
        tap100i0:
                netin: 12683342
                netout: 1118413
pid: 2433
proxmox-support:
        backup-fleecing: 1
        backup-max-workers: 1
        pbs-dirty-bitmap: 1
        pbs-dirty-bitmap-migration: 1
        pbs-dirty-bitmap-savevm: 1
        pbs-library-version: 1.4.1 (UNKNOWN)
        pbs-masterkey: 1
        query-bitmap-info: 1
qmpstatus: running
running-machine: pc-i440fx-9.0+pve0
running-qemu: 9.0.0
status: running
tags: centos;db;mimir;oracle12
uptime: 6635
vmid: 100

root@pvetst:~# gdb --batch --ex 't a a bt' -p $(cat /var/run/qemu-server/100.pid)
[New LWP 2434]
[New LWP 2535]
[New LWP 2536]
[New LWP 2537]
[New LWP 2538]
[New LWP 2539]
[New LWP 2542]
[New LWP 2592]
[New LWP 2593]
[New LWP 5474]
[New LWP 6979]
[New LWP 34483]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x000074015df78256 in __ppoll (fds=0x5c1d0bc64e60, nfds=11, timeout=<optimized out>, timeout@entry=0x7fff84304fd0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42
42      ../sysdeps/unix/sysv/linux/ppoll.c: No such file or directory.

Thread 13 (Thread 0x7401598006c0 (LWP 34483) "iou-wrk-2537"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 12 (Thread 0x74015b51b480 (LWP 6979) "iou-wrk-2433"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 11 (Thread 0x74015a6006c0 (LWP 5474) "iou-wrk-2536"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 10 (Thread 0x740153e006c0 (LWP 2593) "iou-wrk-2539"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 9 (Thread 0x740158e006c0 (LWP 2592) "iou-wrk-2538"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 8 (Thread 0x73ff4b8006c0 (LWP 2542) "vnc_worker"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x5c1d0beb8e48) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (futex_word=futex_word@entry=0x5c1d0beb8e48, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0, cancel=cancel@entry=true) at ./nptl/futex-internal.c:87
#2  0x000074015df01efb in __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x5c1d0beb8e48, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x000074015df04558 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x5c1d0beb8e58, cond=0x5c1d0beb8e20) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=cond@entry=0x5c1d0beb8e20, mutex=mutex@entry=0x5c1d0beb8e58) at ./nptl/pthread_cond_wait.c:618
#5  0x00005c1d0a5bbbbb in qemu_cond_wait_impl (cond=0x5c1d0beb8e20, mutex=0x5c1d0beb8e58, file=0x5c1d0a6701d4 "../ui/vnc-jobs.c", line=248) at ../util/qemu-thread-posix.c:225
#6  0x00005c1d09fd616b in vnc_worker_thread_loop (queue=queue@entry=0x5c1d0beb8e20) at ../ui/vnc-jobs.c:248
#7  0x00005c1d09fd6e48 in vnc_worker_thread (arg=arg@entry=0x5c1d0beb8e20) at ../ui/vnc-jobs.c:362
#8  0x00005c1d0a5bafc8 in qemu_thread_start (args=0x5c1d0beb8eb0) at ../util/qemu-thread-posix.c:541
#9  0x000074015df05134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x000074015df857dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 7 (Thread 0x740153e006c0 (LWP 2539) "CPU 3/KVM"):
#0  __GI___ioctl (fd=32, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x00005c1d0a402a09 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5c1d0bc9c3e0, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3052
#2  0x00005c1d0a402f51 in kvm_cpu_exec (cpu=cpu@entry=0x5c1d0bc9c3e0) at ../accel/kvm/kvm-all.c:2869
#3  0x00005c1d0a404795 in kvm_vcpu_thread_fn (arg=arg@entry=0x5c1d0bc9c3e0) at ../accel/kvm/kvm-accel-ops.c:50
#4  0x00005c1d0a5bafc8 in qemu_thread_start (args=0x5c1d0bca5420) at ../util/qemu-thread-posix.c:541
#5  0x000074015df05134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x000074015df857dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x740158e006c0 (LWP 2538) "CPU 2/KVM"):
#0  __GI___ioctl (fd=30, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x00005c1d0a402a09 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5c1d0bc92b60, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3052
#2  0x00005c1d0a402f51 in kvm_cpu_exec (cpu=cpu@entry=0x5c1d0bc92b60) at ../accel/kvm/kvm-all.c:2869
#3  0x00005c1d0a404795 in kvm_vcpu_thread_fn (arg=arg@entry=0x5c1d0bc92b60) at ../accel/kvm/kvm-accel-ops.c:50
#4  0x00005c1d0a5bafc8 in qemu_thread_start (args=0x5c1d0bc9b9c0) at ../util/qemu-thread-posix.c:541
#5  0x000074015df05134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x000074015df857dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7401598006c0 (LWP 2537) "CPU 1/KVM"):
#0  __GI___ioctl (fd=28, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x00005c1d0a402a09 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5c1d0bc88260, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3052
#2  0x00005c1d0a402f51 in kvm_cpu_exec (cpu=cpu@entry=0x5c1d0bc88260) at ../accel/kvm/kvm-all.c:2869
#3  0x00005c1d0a404795 in kvm_vcpu_thread_fn (arg=arg@entry=0x5c1d0bc88260) at ../accel/kvm/kvm-accel-ops.c:50
#4  0x00005c1d0a5bafc8 in qemu_thread_start (args=0x5c1d0bc92140) at ../util/qemu-thread-posix.c:541
#5  0x000074015df05134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x000074015df857dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x74015a6006c0 (LWP 2536) "CPU 0/KVM"):
#0  __GI___ioctl (fd=26, request=request@entry=44672) at ../sysdeps/unix/sysv/linux/ioctl.c:36
#1  0x00005c1d0a402a09 in kvm_vcpu_ioctl (cpu=cpu@entry=0x5c1d0bc58530, type=type@entry=44672) at ../accel/kvm/kvm-all.c:3052
#2  0x00005c1d0a402f51 in kvm_cpu_exec (cpu=cpu@entry=0x5c1d0bc58530) at ../accel/kvm/kvm-all.c:2869
#3  0x00005c1d0a404795 in kvm_vcpu_thread_fn (arg=arg@entry=0x5c1d0bc58530) at ../accel/kvm/kvm-accel-ops.c:50
#4  0x00005c1d0a5bafc8 in qemu_thread_start (args=0x5c1d0b8b50b0) at ../util/qemu-thread-posix.c:541
#5  0x000074015df05134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x000074015df857dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x74015b51b480 (LWP 2535) "vhost-2433"):
#0  0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

Thread 2 (Thread 0x74015b0006c0 (LWP 2434) "call_rcu"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00005c1d0a5bc2ca in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at ./include/qemu/futex.h:29
#2  qemu_event_wait (ev=ev@entry=0x5c1d0b4a3548 <rcu_call_ready_event>) at ../util/qemu-thread-posix.c:464
#3  0x00005c1d0a5c7222 in call_rcu_thread (opaque=opaque@entry=0x0) at ../util/rcu.c:278
#4  0x00005c1d0a5bafc8 in qemu_thread_start (args=0x5c1d0b8b9790) at ../util/qemu-thread-posix.c:541
#5  0x000074015df05134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x000074015df857dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x74015b51b480 (LWP 2433) "kvm"):
#0  0x000074015df78256 in __ppoll (fds=0x5c1d0bc64e60, nfds=11, timeout=<optimized out>, timeout@entry=0x7fff84304fd0, sigmask=sigmask@entry=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:42
#1  0x00005c1d0a5d397e in ppoll (__ss=0x0, __timeout=0x7fff84304fd0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:64
#2  qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=timeout@entry=990608) at ../util/qemu-timer.c:351
#3  0x00005c1d0a5d0dae in os_host_main_loop_wait (timeout=990608) at ../util/main-loop.c:305
#4  main_loop_wait (nonblocking=nonblocking@entry=0) at ../util/main-loop.c:589
#5  0x00005c1d0a1eb1a9 in qemu_main_loop () at ../system/runstate.c:783
#6  0x00005c1d0a40e226 in qemu_default_main () at ../system/main.c:37
#7  0x000074015dea324a in __libc_start_call_main (main=main@entry=0x5c1d09f898e0 <main>, argc=argc@entry=74, argv=argv@entry=0x7fff843051e8) at ../sysdeps/nptl/libc_start_call_main.h:58
#8  0x000074015dea3305 in __libc_start_main_impl (main=0x5c1d09f898e0 <main>, argc=74, argv=0x7fff843051e8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff843051d8) at ../csu/libc-start.c:360
#9  0x00005c1d09f8b621 in _start ()
[Inferior 1 (process 2433) detached]
root@pvetst:~#

 

[dScha]

Einen Versuch wert wäre auch den CPU Typ der VM auf x86-64-v2-AES zu stellen, den SCSI Controller auf VirtIO SCSI single und auf den Disken IO Thread anzuschalten.

Umstellung der CPU - keine Änderung
Umstellung des SCSI-Controller aud VirtIO SCSI single und anschalten von IO Thread scheint das Problem zu beheben!

 

[dScha]

Hallo Fiona,
nochmal danke für den Hinweis, damit ist mein konkretest Problem gelöst.
Zum Umstand das eine funktionierende Konfiguration nach einem Update nicht mehr geht - soll ich einen Bug aufmachen, oder machst du das ?
Werden dazu noch weitere Informationen benötigt ?
VG

 

[fiona]

Es könnte auch einfach die folgende Limitierung sein: https://bugzilla.proxmox.com/show_bug.cgi?id=1453
Mit vielen Disken/viel IO sollte IO Thread benutzt werden, andernfalls können die virtuellen CPUs zu sehr blockiert werden. Warum es erst seit dem Update passiert ist nicht klar, es gab im Block-Layer in QEMU viele Änderungen zwischen QEMU 8.1 und 9.0, möglicherweise wird dadurch bei Dir diese Limitierung jetzt erst bemerkbar.