[SOLVED] /var/lib/data not mounted

[BenDDD]

Good evening everyone,

The /var/lib/data partition is not mounted on one of the two nodes in my cluster. As a result, the / partition fills up to 100%:

df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                  2.9G     0  2.9G   0% /dev
tmpfs                 595M   61M  534M  11% /run
/dev/mapper/pve-root   15G   15G     0 100% /
tmpfs                 3.0G   57M  2.9G   2% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                 3.0G     0  3.0G   0% /sys/fs/cgroup
/dev/fuse              30M   28K   30M   1% /etc/pve
/dev/sdb1              97M  5.5M   92M   6% /var/lib/ceph/osd/ceph-1
tmpfs                 595M     0  595M   0% /run/user/1003

I compared the /etc/fstab file with the other node and it seems to be missing the mount:

cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

What do you recommend that I do to resolve this issue?

Thank you in advance for your help.

 

[BenDDD]

Hello,

Does anyone have any idea what I can do to resolve this issue?

Thank you in advance.

 

[fiona]

Hi,
please free up a little bit of space first. Do you have a storage that uses the mount point /var/lib/data? If so, disable the storage in PVE and move the data below /var/lib/data to somewhere else, e..g. rename the directory or move it to an external drive. Then mount the partition and create an entry in /etc/fstab to make sure it also mounts on subsequent boots. And now move the data back and re-enable the storage.

 

[BenDDD]

Hello @Fabian_E,

The 15 GB of the partition is taken by the image of the VM in /var/lib/data/images/.

Can I transfer the /var/lib/data/images/103/ folder via scp to another machine, disable the storage, rename the /var/lib/data directory, mount the partition, create an entry in /etc/fstab and transfer back the /var/lib/data/images/103/ folder via scp ?

 

[fiona]

You can do this, but it's not necessary to use scp. You can also do:

1. disable the storage in PVE
2. rename the /var/lib/data directory
3. mount the partition to /var/lib/data (you'll need create an empty directory at the mount point with mkdir /var/lib/data first) and create an entry in /etc/fstab
4. move the contents from the renamed directory to the newly mounted file system (to be on the safe side, you can also use copy and remove the renamed directory later after making sure everything worked)
5. re-enable the storage

 

[BenDDD]

I took your advice and get stuck when trying to mount the partition:

cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
/dev/pve/data /var/lib/data ext4 defaults 0 2

mount -a
mount: /var/lib/data: wrong fs type, bad option, bad superblock on /dev/mapper/pve-data, missing codepage or helper program, or other error.

 

[fiona]

Hi,
normally pve/data is an LVM thin pool, so it cannot be mounted as an ext4 file system. What is the ouput of the following:

lvs
pvesh get /storage/local-lvm

How/what did you mount on the other node? Did you format the logical volume with ext4 there?

 

[BenDDD]

Hello,

I am not the one who installed these nodes. The colleague has left and I must resume the rest.

Here is the / etc / fstab / file for the other node:

cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
/dev/pve/data /var/lib/data ext4 defaults 0 2

 

[fiona]

Yes, seems like it's not a standard setup on that node anymore. Could you do on both nodes:

cat /etc/pve/storage.cfg
lvs

 

[BenDDD]

Node 1 (OK) :

cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,backup,vztmpl

dir: data
        disable
        path /var/lib/data
        content rootdir,iso,images,vztmpl
        shared 0

rbd: osd.0
        content rootdir,images
        krbd 0
        pool galaxie-invites

rbd: osd.1
        content rootdir,images
        krbd 0
        pool galaxie-invites
        
lvs
  LV   VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data pve -wi-ao---- 118.07g
  root pve -wi-ao----  12.25g
  swap pve -wi-ao----   6.12g

Node 2 (KO) :

cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,backup,vztmpl

dir: data
        disable
        path /var/lib/data
        content rootdir,iso,images,vztmpl
        shared 0

rbd: osd.0
        content rootdir,images
        krbd 0
        pool galaxie-invites

rbd: osd.1
        content rootdir,images
        krbd 0
        pool galaxie-invites
        
lvs
  LV   VG  Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data pve twi-a-tz-- 24.37g             0.00   1.58
  root pve -wi-ao---- 15.25g
  swap pve -wi-a-----  1.00g

 

[fiona]

Note that the data LV on node 2 has different attributes than on node 1. On node 2 it's a thin pool, while on node 1 it's a normal LV. Since the thin pool is empty, you can re-create the data LV on node 2 as a normal LV (make sure you're on the correct node!):

lvremove pve/data
lvcreate pve -n data -l100%FREE

Afterwards it should show up as /dev/pve/data and you can format it with ext4 and mount it.

 

[BenDDD]

It worked. Thank you very much for your help.