[SOLVED] Container kein Backup, kein shutdown , kein reboot

[volleasy]

Vor ein paar Wochen habe ich einen CT von einem 4.4 Cluster auf ein 5.2 cluster migriert.

Die Wiederherstellung habe ich aus einem Backup gemacht.

Jetzt habe ich festgestellt das kein Backup läuft.

TASK ERROR: CT is locked (snapshot-delete)

Ok snapshot gelöscht

rbd: sysfs write failed
TASK ERROR: can't unmap rbd volume vm-120-disk-0: rbd: sysfs write failed

Stopp
TASK ERROR: CT is locked (backup)

Ok unlocked

Dann Backup neu gestartet:

aber kein Erfolg

Dann umgestellt auf Backup Stopp

Danach war der Container nicht mehr erreichbar -

Jetzt komme ich zwar im Intranet noch mit putty drauf, aber in der shell kein reboot oder shutdown funzt.

lxc-stop --nokill funzt auch nicht.

status vom container

pve-container@120.service - PVE LXC Container: 120
Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
Active: active (running) since Sun 2018-11-04 20:21:12 EET; 1 months 8 days ago
Docs: man:lxc-start
man:lxc
manct
Process: 3518949 ExecStop=/usr/share/lxc/pve-container-stop-wrapper 120 (code=exited, status=0/SUCCESS)
Process: 3518957 ExecStart=/usr/bin/lxc-start -n 120 (code=exited, status=0/SUCCESS)
Main PID: 3518962 (lxc-start)
Tasks: 1 (limit: 25804)
CGroup: /system.slice/system-pve\x2dcontainer.slice/pve-container@120.service
└─3518962 [lxc monitor] /var/lib/lxc 120

Nov 04 20:21:04 tln-pve-srv03 systemd[1]: Starting PVE LXC Container: 120...
Nov 04 20:21:12 tln-pve-srv03 systemd[1]: Started PVE LXC Container: 120.

INFO: starting new backup job: vzdump 120 --node tln-pve-srv03 --compress lzo --storage OMV-Storage --remove 0 --mode snapshot
INFO: Starting Backup of VM 120 (lxc)
INFO: status = running
INFO: CT Name: owncl9
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
snapshot create failed: starting cleanup
no lock found trying to remove 'backup' lock
ERROR: Backup of VM 120 failed - rbd snapshot 'vm-120-disk-0' error: rbd: failed to create snapshot: (17) File exists

Ich suche dringend ein paar Tipps den Container sauber runterzufahren.

Ach ja, disk liegt in einem Ceph pool.



proxmox-ve: 5.2-2 (running kernel: 4.15.18-7-pve)
pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9)
pve-kernel-4.15: 5.2-10
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.10.17-2-pve: 4.10.17-20
ceph: 12.2.8-pve1
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-40
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-30
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-28
pve-docs: 5.2-8
pve-firewall: 3.0-14
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-36
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.11-pve1~bpo1

 

[volleasy]

Ok, habe jetzt den Service gestoppt.

status sagt
● pve-container@120.service - PVE LXC Container: 120
Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
Active: failed (Result: timeout) since Thu 2018-12-13 21:09:00 EET; 12s ago
Docs: man:lxc-start
man:lxc
manct
Process: 357628 ExecStop=/usr/share/lxc/pve-container-stop-wrapper 120 (code=killed, signal=TERM)
Process: 3518957 ExecStart=/usr/bin/lxc-start -n 120 (code=exited, status=0/SUCCESS)
Main PID: 3518962 (code=killed, signal=KILL)

Nov 04 20:21:04 tln-pve-srv03 systemd[1]: Starting PVE LXC Container: 120...
Nov 04 20:21:12 tln-pve-srv03 systemd[1]: Started PVE LXC Container: 120.
Dec 13 21:05:00 tln-pve-srv03 systemd[1]: Stopping PVE LXC Container: 120...
Dec 13 21:07:00 tln-pve-srv03 systemd[1]: pve-container@120.service: Stopping timed out. Terminating.
Dec 13 21:09:00 tln-pve-srv03 systemd[1]: pve-container@120.service: State 'stop-sigterm' timed out. Killing.
Dec 13 21:09:00 tln-pve-srv03 systemd[1]: pve-container@120.service: Killing process 3518962 (lxc-start) with signal SIGKILL.
Dec 13 21:09:00 tln-pve-srv03 systemd[1]: pve-container@120.service: Main process exited, code=killed, status=9/KILL
Dec 13 21:09:00 tln-pve-srv03 systemd[1]: Stopped PVE LXC Container: 120.
Dec 13 21:09:00 tln-pve-srv03 systemd[1]: pve-container@120.service: Unit entered failed state.
Dec 13 21:09:00 tln-pve-srv03 systemd[1]: pve-container@120.service: Failed with result 'timeout'.


systemctl start pve-container@120.service schlägt fehl

-- Support: https://www.debian.org/support
--
-- Unit pve-container@120.service has failed.
--
-- The result is failed.
Dec 13 21:10:47 tln-pve-srv03 systemd[1]: pve-container@120.service: Unit entered failed state.
Dec 13 21:10:47 tln-pve-srv03 systemd[1]: pve-container@120.service: Failed with result 'exit-code'.
Dec 13 21:10:47 tln-pve-srv03 pvedaemon[358376]: unable to get PID for CT 120 (not running?)
Dec 13 21:10:47 tln-pve-srv03 pvestatd[2484]: unable to get PID for CT 120 (not running?)
Dec 13 21:11:00 tln-pve-srv03 systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has begun starting up.
Dec 13 21:11:01 tln-pve-srv03 systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.


Seltsam ist das ich die IP des Containers noch pingen kann.

 

[volleasy]

Nach einem Neustart des Note, ist auch der CT wieder sauber gestartet und das Backup lief durch- insoweit gelöst. Offen bleibt warum es überhaupt passiert ist. Keine Zeit das noch mal zu reproduzieren.