One Node going in read only mode

[sahostking]

Hi guys

We have many nodes but one new one with some VMs on are now having issues with read only file system.

It works fine for a day or two but when a new VPS is created it all of a sudden goes into read only mode until we reboot the server. Then all is well till the next VPS is created.

When we SSH in we see a line:

/dev/pts/10 : Read only File System

 

[acidrop]

Have you checked for any potential issues with the node's backing storage device(s), RAID or whatever else you are using?
Any clues in the system logs ?

 

[sahostking]

Syslog empty and it happened on new node aswell
Think its the new kernel as that is all we changed was updating the servers.

 

[sahostking]

root@vz-cpt-1:~# lxc-start -n 107 -F -l DEBUG -o /tmp/lxc-ID.log
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/freezer//lxc/107" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/freezer//lxc/107"
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/cpuset//lxc/107-1" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/cpuset//lxc/107-1"
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/freezer//lxc/107-2" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/freezer//lxc/107-2"
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/freezer//lxc/107-3" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/freezer//lxc/107-3"
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/pids//lxc/107-4" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/pids//lxc/107-4"
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/freezer//lxc/107-5" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/freezer//lxc/107-5"
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/freezer//lxc/107-6" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/freezer//lxc/107-6"
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/freezer//lxc/107-7" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/freezer//lxc/107-7"
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/cpuset//lxc/107-8" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/cpuset//lxc/107-8"
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/freezer//lxc/107-9" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/freezer//lxc/107-9"
lxc-start: 107: cgroups/cgfsng.c: create_path_for_hierarchy: 1752 Path "/sys/fs/cgroup/rdma//lxc/107-10" already existed.
lxc-start: 107: cgroups/cgfsng.c: cgfsng_create: 1862 Failed to create cgroup "/sys/fs/cgroup/rdma//lxc/107-10"
lxc-start: 107: conf.c: lxc_setup_ttydir_console: 1738 Read-only file system - Failed to set mode "0111" to "/dev/pts/10"
lxc-start: 107: conf.c: lxc_setup: 3415 Failed to setup console
lxc-start: 107: start.c: do_start: 1198 Failed to setup container "107"
lxc-start: 107: sync.c: __sync_wait: 57 An error occurred in another process (expected sequence number 5)
lxc-start: 107: start.c: __lxc_start: 1883 Failed to spawn container "107"
The container failed to start.

 

[sahostking]

Damn I'm silly
I decided to update another node to latest version and reboot. Then ran LXC backups and poof. Same thing occurred few minutes later.
Very weird.

Gonna move some VPS off to another while I try and fix this issue.

 

[sahostking]

Seems related to https://github.com/lxc/lxc/issues/1841

Any help? We use LVM thin with HW RAID 10

Seems like it starts after backups to our NFS server.