VM doesn't start Proxmox 6 - timeout waiting on systemd

[homozavrus]

Hi all,
i have a updated to 6.0 proxmox from 5.4 (without any issues). Community edition.
All containers and VM are on zfs pool (10 disks in zfs raid10).
I have tried to reset one stucked VM. I have stopped and when i try to start that,
i have strange error -

prx-06:~# qm start 6008
timeout waiting on systemd

Web-interface look strange too: all wm and pct are grey.
How can i debug that problem?
prx-06:~# uname -a
Linux hst-cl-prx-06 5.0.15-1-pve #1 SMP PVE 5.0.15-1 (Wed, 03 Jul 2019 10:51:57 +0200) x86_64 GNU/Linux
prx-06:~# pveversion
pve-manager/6.0-4/2a719255 (running kernel: 5.0.15-1-pve)
root@hst-cl-prx-06:~# zpool status
pool: rpool
state: ONLINE
scan: scrub repaired 0B in 0 days 00:05:54 with 0 errors on Sun Jul 14 06:29:56 2019
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdz3 ONLINE 0 0 0
sdy3 ONLINE 0 0 0

errors: No known data errors

pool: storagepool
state: ONLINE
scan: scrub repaired 0B in 0 days 05:05:06 with 0 errors on Sun Jul 14 11:29:09 2019
config:

NAME STATE READ WRITE CKSUM
storagepool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5000cca2326d67ac ONLINE 0 0 0
wwn-0x5000cca23271b308 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
wwn-0x5000cca23271ae98 ONLINE 0 0 0
wwn-0x5000cca2322788a0 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
wwn-0x5000cca23227ae20 ONLINE 0 0 0
wwn-0x5000cca23227b510 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
wwn-0x5000cca23227abf0 ONLINE 0 0 0
wwn-0x5000cca232276af0 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
wwn-0x5000cca23227b354 ONLINE 0 0 0
wwn-0x5000cca2327168a8 ONLINE 0 0 0

errors: No known data errors

 

[fabian]

please post "pveversion -v" and "systemctl status pvestatd" output...

 

[homozavrus]

I have rebooted the host.
i have no other options ((
All works for now. I think that issue coming again.

prx-06:~# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.15-1-pve)
pve-manager: 6.0-4 (running version: 6.0-4/2a719255)
pve-kernel-5.0: 6.0-5
pve-kernel-helper: 6.0-5
pve-kernel-4.15: 5.4-6
pve-kernel-5.0.15-1-pve: 5.0.15-1
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.18-17-pve: 4.15.18-43
ceph: 14.2.1-pve2
ceph-fuse: 14.2.1-pve2
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksmtuned: 4.20150325+b1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-2
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-5
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-4
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-5
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-3
pve-xtermjs: 3.13.2-1
pve-zsync: 2.0-1
qemu-server: 6.0-5
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1

prx-06:~# systemctl status pvestatd
● pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-07-22 19:34:53 +06; 25min ago
Process: 8683 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
Main PID: 8903 (pvestatd)
Tasks: 1 (limit: 4915)
Memory: 94.7M
CGroup: /system.slice/pvestatd.service
└─8903 pvestatd

Jul 22 19:35:33 hst-cl-prx-06 pvestatd[8903]: modified cpu set for lxc/609: 0
Jul 22 19:35:33 hst-cl-prx-06 pvestatd[8903]: modified cpu set for lxc/610: 1
Jul 22 19:35:33 hst-cl-prx-06 pvestatd[8903]: modified cpu set for lxc/611: 4
Jul 22 19:35:44 hst-cl-prx-06 pvestatd[8903]: modified cpu set for lxc/603: 5
Jul 22 19:35:44 hst-cl-prx-06 pvestatd[8903]: modified cpu set for lxc/607: 6
Jul 22 19:35:44 hst-cl-prx-06 pvestatd[8903]: modified cpu set for lxc/612: 0
Jul 22 19:35:44 hst-cl-prx-06 pvestatd[8903]: modified cpu set for lxc/613: 7
Jul 22 19:35:53 hst-cl-prx-06 pvestatd[8903]: modified cpu set for lxc/609: 8
Jul 22 19:46:53 hst-cl-prx-06 pvestatd[8903]: unable to get PID for CT 603 (not running?)
Jul 22 19:46:54 hst-cl-prx-06 pvestatd[8903]: unable to get PID for CT 603 (not running?)

 

[t.lamprecht]

All works for now. I think that issue coming again.

If it comes again see and post output of:

systemctl status qemu.slice

It lists all scopes, we now wait on a possible old $VMID.scope to be gone before we start the VM as we had problems with sopes still being around when we restarted a VM (e.g., for stop-mode backup).

If you still see the $vmid.scope around, you could try a manual:

systemctl stop VMID.scope

(note, just because above command returns it does not means that the VM scope is gone, re-check with the status command)

 

[Paddy972]

hello,
i have the same probleme, after upgrade to the 6th version of proxmox.
i have 2 VMs who freeze after 2 days activities .
with on the console systemd[1] : failed to start journalservice. line by line.

after a qmstop VMID
i have this
root@srvpve1:~# systemctl status qemu.slice
● qemu.slice
Loaded: loaded
Active: active since Wed 2019-07-24 09:35:36 AST; 4 days ago
Tasks: 13
Memory: 54.8G
CGroup: /qemu.slice
└─200.scope
└─4744 /usr/bin/kvm -id 200 -name srvalcasar -chardev socket,id=qmp,path=/var/run/qemu-server/200.qmp,server,nowait -mon chardev=qmp,mode=control -chard

Jul 24 09:35:47 srvpve1 ovs-vsctl[5223]: ovs|00002|db_ctl_base|ERR|no port named fwln208i0
Jul 24 09:35:47 srvpve1 ovs-vsctl[5237]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-port vmbr254 fwln208o0 -- set Interface fwln208o0 type=internal
Jul 26 09:59:49 srvpve1 ovs-vsctl[3036011]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap402i0
Jul 26 09:59:49 srvpve1 ovs-vsctl[3036011]: ovs|00002|db_ctl_base|ERR|no port named tap402i0
Jul 26 09:59:49 srvpve1 ovs-vsctl[3036013]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln402i0
Jul 26 09:59:49 srvpve1 ovs-vsctl[3036013]: ovs|00002|db_ctl_base|ERR|no port named fwln402i0
Jul 26 09:59:49 srvpve1 ovs-vsctl[3036014]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-port vmbr254 tap402i0
Jul 26 11:32:23 srvpve1 ovs-vsctl[3238665]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln402i0
Jul 26 11:32:23 srvpve1 ovs-vsctl[3238665]: ovs|00002|db_ctl_base|ERR|no port named fwln402i0
Jul 26 11:32:23 srvpve1 ovs-vsctl[3238666]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap402i0
root@srvpve1:~# qm start 208
timeout waiting on systemd
root@srvpve1:~#

CT is ok
VM on mageia 7 ok
VMs on ubuntu bionic freeze and con't restart with qm command

all restart if i reboot the server hardware

thanks for your help

 

[copcopcopcop]

Same problem here. Restarting qemu.slice and stopping 4000.scope did not help..

When I try to start the VM after it has crashed.

TASK ERROR: timeout waiting on systemd

Syslog...

Jul 30 13:26:08 pve pvedaemon[19630]: start VM 4000: UPID:pve:00004CAE:00842267:5D407DB0:qmstart:4000:root@pam:
Jul 30 13:26:08 pve pvedaemon[15793]: <root@pam> starting task UPID:pve:00004CAE:00842267:5D407DB0:qmstart:4000:root@pam:
Jul 30 13:26:13 pve pvedaemon[19630]: timeout waiting on systemd
Jul 30 13:26:13 pve pvedaemon[15793]: <root@pam> end task UPID:pve:00004CAE:00842267:5D407DB0:qmstart:4000:root@pam: timeout waiting on systemd

systemctl stop 4000.scope && systemctl status 4000.scope

root@pve:~# systemctl status 4000.scope
● 4000.scope
   Loaded: loaded (/run/systemd/transient/4000.scope; transient)
Transient: yes
   Active: inactive (dead) since Tue 2019-07-30 13:07:44 EDT; 26min ago
    Tasks: 2 (limit: 6143)
   Memory: 7.6G
   CGroup: /qemu.slice/4000.scope
           └─8589 /usr/bin/kvm -id 4000 -name rtps -chardev socket,id=qmp,path=/var/run/qemu-server/4000.qmp,server,nowait -mon chardev=qmp,mode=c

Jul 29 22:40:47 pve systemd[1]: Started 4000.scope.
Jul 29 22:40:48 pve ovs-vsctl[8599]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap4000i0
Jul 29 22:40:48 pve ovs-vsctl[8599]: ovs|00002|db_ctl_base|ERR|no port named tap4000i0
Jul 29 22:40:48 pve ovs-vsctl[8600]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln4000i0
Jul 29 22:40:48 pve ovs-vsctl[8600]: ovs|00002|db_ctl_base|ERR|no port named fwln4000i0
Jul 29 22:40:48 pve ovs-vsctl[8601]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl add-port vmbr10 tap4000i0 tag=20
Jul 30 13:07:44 pve systemd[1]: 4000.scope: Succeeded.
Jul 30 13:07:44 pve systemd[1]: Stopped 4000.scope.

systemctl status qemu.slice

root@pve:~# systemctl status qemu.slice
● qemu.slice
   Loaded: loaded
   Active: active since Tue 2019-07-30 13:25:44 EDT; 3min 52s ago
    Tasks: 53
   Memory: 27.4G
   CGroup: /qemu.slice
           ├─2000.scope
           │ └─39076 /usr/bin/kvm -id 2000 -name tt-rss -chardev socket,id=qmp,path=/var/run/qemu-server/2000.qmp,server,nowait -mon chardev=qmp,m
           ├─2100.scope
           │ └─8985 /usr/bin/kvm -id 2100 -name urlwatch -chardev socket,id=qmp,path=/var/run/qemu-server/2100.qmp,server,nowait -mon chardev=qmp,
           ├─2200.scope
           │ └─10296 /usr/bin/kvm -id 2200 -name docker -chardev socket,id=qmp,path=/var/run/qemu-server/2200.qmp,server,nowait -mon chardev=qmp,m
           ├─2300.scope
           │ └─13277 /usr/bin/kvm -id 2300 -name thelounge -chardev socket,id=qmp,path=/var/run/qemu-server/2300.qmp,server,nowait -mon chardev=qm
           ├─4000.scope
           │ └─8589 /usr/bin/kvm -id 4000 -name rtps -chardev socket,id=qmp,path=/var/run/qemu-server/4000.qmp,server,nowait -mon chardev=qmp,mode
           ├─4100.scope
           │ └─23483 /usr/bin/kvm -id 4100 -name sbxjb -chardev socket,id=qmp,path=/var/run/qemu-server/4100.qmp,server,nowait -mon chardev=qmp,mo
           ├─5000.scope
           │ └─32857 /usr/bin/kvm -id 5000 -name win10 -chardev socket,id=qmp,path=/var/run/qemu-server/5000.qmp,server,nowait -mon chardev=qmp,mo
           ├─5100.scope
           │ └─36667 /usr/bin/kvm -id 5100 -name kubuntu -chardev socket,id=qmp,path=/var/run/qemu-server/5100.qmp,server,nowait -mon chardev=qmp,
           └─9000.scope
             └─32122 /usr/bin/kvm -id 9000 -name ansible -chardev socket,id=qmp,path=/var/run/qemu-server/9000.qmp,server,nowait -mon chardev=qmp,

Jul 30 13:25:44 pve systemd[1]: Stopping qemu.slice.
Jul 30 13:25:44 pve systemd[1]: Created slice qemu.slice.

pveversion -v

root@pve:~# pveversion -v
proxmox-ve: 6.0-2 (running kernel: 5.0.15-1-pve)
pve-manager: 6.0-4 (running version: 6.0-4/2a719255)
pve-kernel-5.0: 6.0-5
pve-kernel-helper: 6.0-5
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1
corosync: 3.0.2-pve2
criu: 3.11-3
glusterfs-client: 5.5-3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.10-pve1
libpve-access-control: 6.0-2
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-2
libpve-guest-common-perl: 3.0-1
libpve-http-server-perl: 3.0-2
libpve-storage-perl: 6.0-5
libqb0: 1.0.5-1
lvm2: 2.03.02-pve3
lxc-pve: 3.1.0-61
lxcfs: 3.0.3-pve60
novnc-pve: 1.0.0-60
openvswitch-switch: 2.10.0+2018.08.28+git.8ca7c82b7d+ds1-12
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.0-5
pve-cluster: 6.0-4
pve-container: 3.0-4
pve-docs: 6.0-4
pve-edk2-firmware: 2.20190614-1
pve-firewall: 4.0-5
pve-firmware: 3.0-2
pve-ha-manager: 3.0-2
pve-i18n: 2.0-2
pve-qemu-kvm: 4.0.0-3
pve-xtermjs: 3.13.2-1
qemu-server: 6.0-5
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.1-pve1

 

[FcT]

Hello, same problem here (timeout waiting on systemd) on a fresh install of pve6 with zfs pools for the OS and VM storage. The VM freezes, the web console turns black and it cannot be restarted (gui or cli).
I had to electrically restart the server too, it won't gently stop or reboot (waiting endlessly for the zfs pool to stop/detach). It happened 2 times in four days on different VMs, in my case there is no backup tasks yet on the VMs running on that server.

 

[copcopcopcop]

Hello, same problem here (timeout waiting on systemd) on a fresh install of pve6 with zfs pools for the OS and VM storage. The VM freezes, the web console turns black and it cannot be restarted (gui or cli).
I had to electrically restart the server too, it won't gently stop or reboot (waiting endlessly for the zfs pool to stop/detach). It happened 2 times in four days on different VMs, in my case there is no backup tasks yet on the VMs running on that server.

Exactly the problem i’ve been having..

EDIT: 20 hours later, qm 4000 is still hung and won't start.

root@pve:~/rsync-logs# systemctl stop 4000.slice
root@pve:~/rsync-logs# systemctl status qemu.slice
● qemu.slice
   Loaded: loaded
   Active: active since Tue 2019-07-30 13:49:27 EDT; 20h ago
    Tasks: 44
   Memory: 14.6G
   CGroup: /qemu.slice
           ├─2000.scope
           │ └─7819 /usr/bin/kvm -id 2000 -name tt-rss -chardev socket,id=qmp,path=/var/run/qemu-server/2000.qmp,server,nowait -mon chardev=qmp,
           ├─2100.scope
           │ └─8123 /usr/bin/kvm -id 2100 -name urlwatch -chardev socket,id=qmp,path=/var/run/qemu-server/2100.qmp,server,nowait -mon chardev=qm
           ├─2200.scope
           │ └─8934 /usr/bin/kvm -id 2200 -name docker -chardev socket,id=qmp,path=/var/run/qemu-server/2200.qmp,server,nowait -mon chardev=qmp,
           ├─2300.scope
           │ └─9181 /usr/bin/kvm -id 2300 -name thelounge -chardev socket,id=qmp,path=/var/run/qemu-server/2300.qmp,server,nowait -mon chardev=q
           ├─4000.scope
           │ └─8589 /usr/bin/kvm -id 4000 -name rtps -chardev socket,id=qmp,path=/var/run/qemu-server/4000.qmp,server,nowait -mon chardev=qmp,mo
           ├─4100.scope
           │ └─19497 /usr/bin/kvm -id 4100 -name sbxjb -chardev socket,id=qmp,path=/var/run/qemu-server/4100.qmp,server,nowait -mon chardev=qmp,
           ├─4200.scope
           │ └─20356 /usr/bin/kvm -id 4200 -name nzbget -chardev socket,id=qmp,path=/var/run/qemu-server/4200.qmp,server,nowait -mon chardev=qmp
           └─9000.scope
             └─22216 /usr/bin/kvm -id 9000 -name ansible -chardev socket,id=qmp,path=/var/run/qemu-server/9000.qmp,server,nowait -mon chardev=qm

Jul 30 16:45:52 pve ovs-vsctl[18441]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln4200o0
Jul 30 16:45:52 pve ovs-vsctl[18451]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port fwln4200i0
Jul 30 16:45:52 pve ovs-vsctl[18451]: ovs|00002|db_ctl_base|ERR|no port named fwln4200i0
Jul 30 16:45:52 pve ovs-vsctl[18455]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap4200i0
Jul 30 16:45:52 pve ovs-vsctl[18455]: ovs|00002|db_ctl_base|ERR|no port named tap4200i0
Jul 30 16:46:25 pve ovs-vsctl[20490]: ovs|00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl del-port tap4200i0

 

[FcT]

Hi, two days without freezes, I keep fingers crossed...
As I don't have a clue if it could be hardware related, I am using a DELL R630 (latest firmware on every components) which boots PVE6 in UEFI mode. Do you guys have similar hardware ?
Greetings

 

[Paddy972]

Hi evrybody,
1 week without freeze, when i configure in hardware - hard disk with no cache in my VMs. i had write-back before.
Try and tell me if you have the same stability after

 

[homozavrus]

i've reinstalled proxmox.
For a week all goes without problem.
I use write-back for VM.

 

[scintilla13]

same problem for me, 6.0.5 fresh install.

 

[VTeronen]

Hello,
I have also encountered the problem on of my hosts with zfs and fresh 6-install.
Unfortunately I haven't been able to debug this too much since the host is currently running a vm that can't stay down too long.

 

[copcopcopcop]

Just had this happen again. Freshly installed VM running for about 2 days.

HDD was not set to write-back.

EDIT: The volume is still stuck and the VM won't start, but I can clone the VM and start it up again fine. pretty annoying. Also worth mentioning this causes shutdown to hang since the volume won't unmount.

 

[copcopcopcop]

still happening fairly frequently over here. it seems to happen most when trying to shut down a VM. the VM will hang while shutting down and has to force stopped. at that point, the virtual disk refuses to unmount until the host is shutdown which requires at least 5-minutes.

I may just revert back to PVE 5.

 

[Addy90]

I have the same problem. One VM keeps freezing the entire system. It seems I can log in via noVNC, but when I shutdown, it stops before killing the process. Stopping the process on Proxmox does not work either.
This is only on my home server: My CPU is an Intel i7-7700T on a GIGABYTE GA-Z270N-WIFI Mainboard with 16 GB DDR4-2400 RAM and Samsung Pro 256 GB SSD. Did not happen on PVE 5.4 before the upgrade. I already had no-cache active, not writeback. Did not help.

I could log the following from the PVE 6 host when trying to shut down the hung VM:
Aug 17 11:45:26 server kernel: INFO: task kvm:1732 blocked for more than 120 seconds.
Aug 17 11:45:26 server kernel: Tainted: P O 5.0.18-1-pve #1
Aug 17 11:45:26 server kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 17 11:45:26 server kernel: kvm D 0 1732 1 0x00000000
Aug 17 11:45:26 server kernel: Call Trace:
Aug 17 11:45:26 server kernel: __schedule+0x2d4/0x870
Aug 17 11:45:26 server kernel: ? wait_for_completion+0xc2/0x140
Aug 17 11:45:26 server kernel: ? wake_up_q+0x80/0x80
Aug 17 11:45:26 server kernel: schedule+0x2c/0x70
Aug 17 11:45:26 server kernel: vhost_net_ubuf_put_and_wait+0x60/0x90 [vhost_net]
Aug 17 11:45:26 server kernel: ? wait_woken+0x80/0x80
Aug 17 11:45:26 server kernel: vhost_net_ioctl+0x5fe/0xa50 [vhost_net]
Aug 17 11:45:26 server kernel: ? send_signal+0x3e/0x80
Aug 17 11:45:26 server kernel: do_vfs_ioctl+0xa9/0x640
Aug 17 11:45:26 server kernel: ksys_ioctl+0x67/0x90
Aug 17 11:45:26 server kernel: __x64_sys_ioctl+0x1a/0x20
Aug 17 11:45:26 server kernel: do_syscall_64+0x5a/0x110
Aug 17 11:45:26 server kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Aug 17 11:45:26 server kernel: RIP: 0033:0x7f03ce3bc427
Aug 17 11:45:26 server kernel: Code: Bad RIP value.
Aug 17 11:45:26 server kernel: RSP: 002b:00007ffd2eb8bd98 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Aug 17 11:45:26 server kernel: RAX: ffffffffffffffda RBX: 00007f03c16bb000 RCX: 00007f03ce3bc427
Aug 17 11:45:26 server kernel: RDX: 00007ffd2eb8bda0 RSI: 000000004008af30 RDI: 0000000000000017
Aug 17 11:45:26 server kernel: RBP: 00007ffd2eb8bda0 R08: 00007f03c1657760 R09: 00007f03c10e8420
Aug 17 11:45:26 server kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007f02bbf9a6b0
Aug 17 11:45:26 server kernel: R13: 0000000000000001 R14: 00007f02bbf9a638 R15: 00007f03c16880c0

 

[copcopcopcop]

These are the logs I was able to pull from one of the VM's on it's way to crashing. The first irregularity is the VM timing out and losing its connection to my NTP server. Soon after the syslog is flooded with VMMouse messages.

Aug 18 21:41:34 sbxjb systemd-timesyncd[928]: Timed out waiting for reply from 10.0.20.1:123 (10.0.20.1).
Aug 18 21:45:01 sbxjb CRON[3429]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 21:46:01 sbxjb CRON[3434]: (root) CMD (if test -x /usr/sbin/apticron; then /usr/sbin/apticron --cron; else true; fi)
Aug 18 21:55:01 sbxjb CRON[3455]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 22:05:01 sbxjb CRON[3478]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 22:15:01 sbxjb CRON[3500]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 22:15:53 sbxjb systemd-timesyncd[928]: Timed out waiting for reply from 10.0.20.1:123 (10.0.20.1).
Aug 18 22:16:03 sbxjb systemd-timesyncd[928]: Timed out waiting for reply from 10.0.20.1:123 (10.0.20.1).
Aug 18 22:17:01 sbxjb CRON[3507]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 18 22:23:19 sbxjb qemu-ga: info: guest-ping called
Aug 18 22:23:48 sbxjb kernel: [ 6744.917724] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:23:49 sbxjb kernel: [ 6745.265232] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:23:49 sbxjb kernel: [ 6746.018856] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:23:49 sbxjb kernel: [ 6746.019037] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:23:59 sbxjb kernel: [ 6755.761332] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:23:59 sbxjb kernel: [ 6756.072372] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:14 sbxjb systemd[1]: Created slice User Slice of user001.
Aug 18 22:24:14 sbxjb systemd[1]: Starting User Manager for UID 1000...
Aug 18 22:24:14 sbxjb systemd[1]: Started Session 16 of user user001.
Aug 18 22:24:14 sbxjb systemd[3541]: Reached target Timers.
Aug 18 22:24:14 sbxjb systemd[3541]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Aug 18 22:24:14 sbxjb systemd[3541]: Listening on GnuPG cryptographic agent and passphrase cache.
Aug 18 22:24:14 sbxjb systemd[3541]: Reached target Paths.
Aug 18 22:24:14 sbxjb systemd[3541]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Aug 18 22:24:14 sbxjb systemd[3541]: Starting D-Bus User Message Bus Socket.
Aug 18 22:24:14 sbxjb systemd[3541]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Aug 18 22:24:14 sbxjb systemd[3541]: Listening on GnuPG network certificate management daemon.
Aug 18 22:24:14 sbxjb systemd[3541]: Listening on D-Bus User Message Bus Socket.
Aug 18 22:24:14 sbxjb systemd[3541]: Reached target Sockets.
Aug 18 22:24:14 sbxjb systemd[3541]: Reached target Basic System.
Aug 18 22:24:14 sbxjb systemd[3541]: Reached target Default.
Aug 18 22:24:14 sbxjb systemd[3541]: Startup finished in 174ms.
Aug 18 22:24:14 sbxjb systemd[1]: Started User Manager for UID 1000.
Aug 18 22:24:15 sbxjb kernel: [ 6771.250767] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:15 sbxjb kernel: [ 6771.256900] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:15 sbxjb kernel: [ 6771.313194] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:15 sbxjb kernel: [ 6771.313948] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:16 sbxjb kernel: [ 6772.450324] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:16 sbxjb kernel: [ 6772.451229] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:33 sbxjb kernel: [ 6789.951615] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:33 sbxjb kernel: [ 6789.952803] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:35 sbxjb kernel: [ 6791.745145] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:35 sbxjb kernel: [ 6791.746403] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:36 sbxjb kernel: [ 6792.747651] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:40 sbxjb kernel: [ 6796.537925] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:40 sbxjb kernel: [ 6796.735395] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:40 sbxjb kernel: [ 6796.735492] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:41 sbxjb kernel: [ 6797.559736] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:41 sbxjb kernel: [ 6797.560343] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:42 sbxjb kernel: [ 6799.008623] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:42 sbxjb kernel: [ 6799.009098] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:52 sbxjb kernel: [ 6808.580412] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:52 sbxjb kernel: [ 6808.585290] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:53 sbxjb kernel: [ 6810.032023] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:53 sbxjb kernel: [ 6810.032757] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:24:54 sbxjb kernel: [ 6810.245250] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:24:54 sbxjb kernel: [ 6810.247907] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:25:01 sbxjb CRON[3745]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 18 22:25:32 sbxjb kernel: [ 6848.509425] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:25:32 sbxjb kernel: [ 6848.510078] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:25:32 sbxjb kernel: [ 6848.913519] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:25:32 sbxjb kernel: [ 6848.919190] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:25:32 sbxjb kernel: [ 6849.135081] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:25:32 sbxjb kernel: [ 6849.136687] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:25:54 sbxjb kernel: [ 6870.562283] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:25:54 sbxjb kernel: [ 6870.562853] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
Aug 18 22:25:55 sbxjb kernel: [ 6872.151393] psmouse serio1: VMMouse at isa0060/serio1/input0 lost sync at byte 1
Aug 18 22:25:57 sbxjb kernel: [ 6873.552463] psmouse serio1: VMMouse at isa0060/serio1/input0 - driver resynced.
[CODE]

 

[spirit]

Hi guys,

do you use openvswitch or linux-bridge ?

 

[FcT]

Hello, linux bridge in my case.
I've experienced crashes again (same thing as I exposed above), I'm sadly going to rollback to pve 5.4, I can't find out why the system is so unreliable for the moment and I need to put back the server in production.

 

[copcopcopcop]

Hi guys,

do you use openvswitch or linux-bridge ?

openvswitch