Ceph RBD in readonly , need twice reboot for VM (QEMU/KVM) to be read and write

[atec666]

When i stop a VM with ceph (RBD) disk, we need to reboot twice the VM in order to have a file system with read/write.

● mariadb.service - MariaDB 10.3.23 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─limits.conf
   Active: failed (Result: exit-code) since Sat 2020-10-17 16:45:31 CEST; 1min 46s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 525 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
  Process: 548 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 552 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] 
  Process: 599 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=226/NAMESPACE)
 Main PID: 599 (code=exited, status=226/NAMESPACE)

Oct 17 16:45:30 web2 systemd[1]: Starting MariaDB 10.3.23 database server...
Oct 17 16:45:31 web2 sh[552]: mktemp: failed to create file via template ‘/tmp/wsrep_recovery.XXXXXX’: Read-only file system
Oct 17 16:45:31 web2 sh[552]: WSREP: mktemp failed
Oct 17 16:45:31 web2 systemd[599]: mariadb.service: Failed to set up mount namespacing: Read-only file system
Oct 17 16:45:31 web2 systemd[599]: mariadb.service: Failed at step NAMESPACE spawning /usr/sbin/mysqld: Read-only file system
Oct 17 16:45:31 web2 systemd[1]: mariadb.service: Main process exited, code=exited, status=226/NAMESPACE
Oct 17 16:45:31 web2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Oct 17 16:45:31 web2 systemd[1]: Failed to start MariaDB 10.3.23 database server.

 

[spirit]

This is really strange, I never see this.

it could be the exclusive lock not released on shutdown, but I never see this.
can you send your /etc/pve/storage.cfg and your vm config /etc/pve/nodes/<node>/qemu-server/<vmid>.conf ?

 

[Alwin]

When i stop a VM with ceph (RBD) disk, we need to reboot twice the VM in order to have a file system with read/write.

What exactly does stop mean, shutdown inside/outside the VM or really stop? And a second reboot then usually runs the fsck.

 

[atec666]

What exactly does stop mean, shutdown inside/outside the VM or really stop? And a second reboot then usually runs the fsck.

hello alwin,

this morning we have a power cut.

So need to power on our 3 nodes (supermicro, 3 x 4*4TB SAS, with hp NC523SFP 10Gbe for ceph on each node on a microtik CRS309 10GbE switch )
All VM and CT are on auto start.
we are using VLAN for promox management.
... So all VM reboot .
BUT when i do a systemctl on each ....
i can see error complain that RBD (ceph) is mounted in read-only.

 

[Alwin]

this morning we have a power cut.

So the VM/CT have been killed? Then corrupted filesystems inside the VM/CT are very likely and to be expected.

 

[atec666]

So the VM/CT have been killed? Then corrupted filesystems inside the VM/CT are very likely and to be expected.

yes. BUt why read-only ?

 

[Alwin]

yes. BUt why read-only ?

You mean, why are they mounted read-only when a corrupted fileystem is detected?

To not cause more harm and to allow easier intervention. It's the default in most OS when mounting filesystems (see fstab).

 

[atec666]

You mean, why are they mounted read-only when a corrupted fileystem is detected?

To not cause more harm and to allow easier intervention. It's the default in most OS when mounting filesystems (see fstab).

ok, thank you alwinn.
So normal OS behaviour

 

[atec666]

yes but with our last promox setup, with massively use of Ct (LXC) when we enconter a power cut : no need to reboot OS ?
CT reboot in read write , perhaps it's a behaviour of VM (QEMU+KVM) ?
or must i do something .... ?

 

[Alwin]

yes but with our last promox setup, with massively use of Ct (LXC) when we enconter a power cut : no need to reboot OS ?

Container are handled by the host, they use mapped (after boot) RBD images as local block devices. You can run a pct fsck <vmid> on a container to see if the FS has issues.