Ceph RBD in readonly , need twice reboot for VM (QEMU/KVM) to be read and write

A

atec666

Member
Mar 8, 2019
136
5
18
Issoire
When i stop a VM with ceph (RBD) disk, we need to reboot twice the VM in order to have a file system with read/write.

Code: 
● mariadb.service - MariaDB 10.3.23 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─limits.conf
   Active: failed (Result: exit-code) since Sat 2020-10-17 16:45:31 CEST; 1min 46s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 525 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
  Process: 548 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 552 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ] 
  Process: 599 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=226/NAMESPACE)
 Main PID: 599 (code=exited, status=226/NAMESPACE)

Oct 17 16:45:30 web2 systemd[1]: Starting MariaDB 10.3.23 database server...
Oct 17 16:45:31 web2 sh[552]: mktemp: failed to create file via template ‘/tmp/wsrep_recovery.XXXXXX’: Read-only file system
Oct 17 16:45:31 web2 sh[552]: WSREP: mktemp failed
Oct 17 16:45:31 web2 systemd[599]: mariadb.service: Failed to set up mount namespacing: Read-only file system
Oct 17 16:45:31 web2 systemd[599]: mariadb.service: Failed at step NAMESPACE spawning /usr/sbin/mysqld: Read-only file system
Oct 17 16:45:31 web2 systemd[1]: mariadb.service: Main process exited, code=exited, status=226/NAMESPACE
Oct 17 16:45:31 web2 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Oct 17 16:45:31 web2 systemd[1]: Failed to start MariaDB 10.3.23 database server.
 
atec666 said:
When i stop a VM with ceph (RBD) disk, we need to reboot twice the VM in order to have a file system with read/write.
Click to expand...
What exactly does stop mean, shutdown inside/outside the VM or really stop? And a second reboot then usually runs the fsck.
 
Alwin said:
What exactly does stop mean, shutdown inside/outside the VM or really stop? And a second reboot then usually runs the fsck.
Click to expand...
hello alwin,

this morning we have a power cut.

So need to power on our 3 nodes (supermicro, 3 x 4*4TB SAS, with hp NC523SFP 10Gbe for ceph on each node on a microtik CRS309 10GbE switch )
All VM and CT are on auto start.
we are using VLAN for promox management.
... So all VM reboot .
BUT when i do a systemctl on each ....
i can see error complain that RBD (ceph) is mounted in read-only.
 
atec666 said:
this morning we have a power cut.
Click to expand...
So the VM/CT have been killed? Then corrupted filesystems inside the VM/CT are very likely and to be expected.
 
atec666 said:
yes. BUt why read-only ?
Click to expand...
You mean, why are they mounted read-only when a corrupted fileystem is detected?

To not cause more harm and to allow easier intervention. It's the default in most OS when mounting filesystems (see fstab).
 
Alwin said:
You mean, why are they mounted read-only when a corrupted fileystem is detected?

To not cause more harm and to allow easier intervention. It's the default in most OS when mounting filesystems (see fstab).
Click to expand...
ok, thank you alwinn.
So normal OS behaviour
 
yes but with our last promox setup, with massively use of Ct (LXC) when we enconter a power cut : no need to reboot OS ?
CT reboot in read write , perhaps it's a behaviour of VM (QEMU+KVM) ?
or must i do something .... ?
 
atec666 said:
yes but with our last promox setup, with massively use of Ct (LXC) when we enconter a power cut : no need to reboot OS ?
Click to expand...
Container are handled by the host, they use mapped (after boot) RBD images as local block devices. You can run a pct fsck <vmid> on a container to see if the FS has issues.
 
You must log in or register to reply here.
Top Bottom