lxc and lvm thinissue

[sahostking]

We have an issue we cannot figure out. Currently the issue that when we run snapshots on lxc containers the backups are successful but then if someone stops any ct and tries to start it it will not start and gives a read only filesystem error.

We then reboot and all vps servers comr up again fine as the issue is then gone.

When someone however deletes a container, creates a new one or runs a snapshot again thr lvm partition again goes in readonly mode until we reboot once again.

Any ideas on what the issue could be?

 

[dietmar]

Maybe you run out of space (or metadata space) on lvm thin? Please check using 'lvs'.

 

[sahostking]

root@vz-cpt-1:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-aotz-- 5.30t 45.40 7.66
root pve -wi-ao---- 50.00g
swap pve -wi-ao---- 64.00g
vm-103-disk-1 pve Vwi-aotz-- 400.00g data 45.68
vm-104-disk-1 pve Vwi-aotz-- 200.00g data 70.47
vm-109-disk-1 pve Vwi-aotz-- 10.00g data 70.93
vm-112-disk-1 pve Vwi-aotz-- 10.00g data 90.20
vm-113-disk-1 pve Vwi-aotz-- 200.00g data 66.94
vm-115-disk-1 pve Vwi-aotz-- 200.00g data 3.20
vm-116-disk-1 pve Vwi-aotz-- 50.00g data 25.46
vm-117-disk-1 pve Vwi-aotz-- 100.00g data 48.78
vm-128-disk-1 pve Vwi-aotz-- 200.00g data 40.39
vm-129-disk-1 pve Vwi-aotz-- 25.00g data 92.18
vm-130-disk-1 pve Vwi-aotz-- 10.00g data 11.55
vm-131-disk-1 pve Vwi-aotz-- 100.00g data 54.29
vm-132-disk-1 pve Vwi-aotz-- 42.54g data 93.15
vm-136-disk-1 pve Vwi-a-tz-- 50.00g data 18.54
vm-138-disk-1 pve Vwi-aotz-- 100.00g data 57.41
vm-141-disk-1 pve Vwi-aotz-- 25.00g data 15.95
vm-146-disk-1 pve Vwi-a-tz-- 25.00g data 34.09
vm-147-disk-1 pve Vwi-aotz-- 200.00g data 92.79
vm-150-disk-1 pve Vwi-a-tz-- 400.00g data 32.97

 

[sahostking]

We always install using proxmox iso with lvm and ext4 and 50GB root and specify swap. The rest is all default.

Space is only half used.

 

[dietmar]

What version do you run exactly?

# pveversion -v

 

[sahostking]

root@vz-cpt-1:~# pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.17-3-pve)
pve-manager: 5.2-2 (running version: 5.2-2/b1d1c7f4)
pve-kernel-4.15: 5.2-3
pve-kernel-4.15.17-3-pve: 4.15.17-13
pve-kernel-4.15.17-2-pve: 4.15.17-10
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-4
libpve-common-perl: 5.0-33
libpve-guest-common-perl: 2.0-16
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-23
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-27
pve-container: 2.0-23
pve-docs: 5.2-4
pve-firewall: 3.0-12
pve-firmware: 2.0-4
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-5
qemu-server: 5.0-28
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9

 

[sahostking]

I am starting to think its overprovisioning. I moved some LARGE 800GB vps off that server to new one and it seems to be better.

Could I be right? Its never reached 90% of disksuage though always at 2.2T of 5.4T

 

[sahostking]

Seems its related to another post someone else posted earlier today.

I found the below:

---------------------------------------

I noticed this was mount read only on the problem node:

devpts on /dev/pts type devpts (ro,nosuid,noexec,relatime,mode=600,ptmxmode=000)

whereas this is rw on a node working fine:

devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

 

[sahostking]

Fixed it doing the following:

Getting closer.

I ran this :

mount -t devpts -o remount,gid=5,mode=620 devpts /dev/pts

and now it doesnt say read only but getting new error now which made me think some stale things still left behind.

So I ran
find /sys/fs/cgroup/*/lxc/113* -type d | tac | xargs rmdir

where 113 is the container ID and now I started it and all is good again in the world

Probably some bug somewhere.