[SOLVED] Container subvolume permissions changing on restart

Xyrgh

Member
Jun 15, 2020
8
1
8
Hello,

I appear to have an issue where permissions are changing on a reboot on a zfs volume. Volume is an SSD as zfs. All the container subvolumes below except 104 were created this morning the permissions somehow changed from 100000 to root and 700. After this, they will no longer start. These are all ubuntu 20.04 or 18.04 containers.

Code:
root@pve:/vm# ls -l
total 13
drwxr-----  2 root   root    2 Jul 10 14:13 subvol-100-disk-0
drwxr-----  2 root   root    2 Jul 10 11:14 subvol-101-disk-0
drwxr-----  2 root   root    2 Jul 10 11:14 subvol-102-disk-0
drwxr-----  2 root   root    2 Jul 10 11:15 subvol-103-disk-0
drwxr-xr-x 21 100000 100000 21 Jul 10 14:28 subvol-104-disk-0
drwxr-----  3 root   root    3 Jul 10 14:18 subvol-500-disk-0
drwxr-----  3 root   root    3 Jul 10 14:18 subvol-501-disk-0
drwxr-----  2 root   root    2 Jul  8 10:24 subvol-700-disk-0
drwxr-----  3 root   root    3 Jul 10 14:18 subvol-701-disk-0

I created 104 as a test container, works perfectly. As a test, I restarted the system and low and behold, that container also changed to root:root and 700 permissions:

Code:
root@pve:/vm# ls -l
total 5
drwxr----- 2 root root 2 Jul 10 14:13 subvol-100-disk-0
drwxr----- 3 root root 3 Jul 10 16:11 subvol-101-disk-0
drwxr----- 2 root root 2 Jul 10 11:14 subvol-102-disk-0
drwxr----- 2 root root 2 Jul 10 11:15 subvol-103-disk-0
drwxr----- 2 root root 2 Jul 10 14:28 subvol-104-disk-0
drwxr----- 3 root root 3 Jul 10 14:18 subvol-500-disk-0
drwxr----- 3 root root 3 Jul 10 14:18 subvol-501-disk-0
drwxr----- 2 root root 2 Jul  8 10:24 subvol-700-disk-0
drwxr----- 3 root root 3 Jul 10 14:18 subvol-701-disk-0

I originally only restarted because of some network issues, but this is going to be a major issue on something like a kernel upgrade.

Pveversion:


Code:
root@pve:/vm# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.44-2-pve)
pve-manager: 6.2-6 (running version: 6.2-6/ee1d7754)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.44-1-pve: 5.4.44-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve2
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-3
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-8
pve-cluster: 6.1-8
pve-container: 3.1-8
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-3
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

Can I please get some help?
 
Nothing which I can reproduce by the steps you indicated.

on a hunch - please post the output of:
Code:
zfs get all vm/subvol-104-disk-0
zpool status
zpool import
ls -la /vm/subvol-104-disk-0

(my guess is that the pool did not get imported - which is preventing the start of the container)
 
Just FYI, it;s all the containers except 104 that aren't working, so I've done a zfs get all for 100 and 104.


zfs get all 104:

Code:
NAME                  PROPERTY              VALUE                  SOURCE
vm/subvol-104-disk-0  type                  filesystem             -
vm/subvol-104-disk-0  creation              Fri Jul 10 14:28 2020  -
vm/subvol-104-disk-0  used                  363M                   -
vm/subvol-104-disk-0  available             661M                   -
vm/subvol-104-disk-0  referenced            363M                   -
vm/subvol-104-disk-0  compressratio         2.00x                  -
vm/subvol-104-disk-0  mounted               no                     -
vm/subvol-104-disk-0  quota                 none                   default
vm/subvol-104-disk-0  reservation           none                   default
vm/subvol-104-disk-0  recordsize            128K                   default
vm/subvol-104-disk-0  mountpoint            /vm/subvol-104-disk-0  default
vm/subvol-104-disk-0  sharenfs              off                    default
vm/subvol-104-disk-0  checksum              on                     default
vm/subvol-104-disk-0  compression           on                     inherited from vm
vm/subvol-104-disk-0  atime                 on                     default
vm/subvol-104-disk-0  devices               on                     default
vm/subvol-104-disk-0  exec                  on                     default
vm/subvol-104-disk-0  setuid                on                     default
vm/subvol-104-disk-0  readonly              off                    default
vm/subvol-104-disk-0  zoned                 off                    default
vm/subvol-104-disk-0  snapdir               hidden                 default
vm/subvol-104-disk-0  aclinherit            restricted             default
vm/subvol-104-disk-0  createtxg             164224                 -
vm/subvol-104-disk-0  canmount              on                     default
vm/subvol-104-disk-0  xattr                 sa                     local
vm/subvol-104-disk-0  copies                1                      default
vm/subvol-104-disk-0  version               5                      -
vm/subvol-104-disk-0  utf8only              off                    -
vm/subvol-104-disk-0  normalization         none                   -
vm/subvol-104-disk-0  casesensitivity       sensitive              -
vm/subvol-104-disk-0  vscan                 off                    default
vm/subvol-104-disk-0  nbmand                off                    default
vm/subvol-104-disk-0  sharesmb              off                    default
vm/subvol-104-disk-0  refquota              1G                     local
vm/subvol-104-disk-0  refreservation        none                   default
vm/subvol-104-disk-0  guid                  4585598861853904161    -
vm/subvol-104-disk-0  primarycache          all                    default
vm/subvol-104-disk-0  secondarycache        all                    default
vm/subvol-104-disk-0  usedbysnapshots       0B                     -
vm/subvol-104-disk-0  usedbydataset         363M                   -
vm/subvol-104-disk-0  usedbychildren        0B                     -
vm/subvol-104-disk-0  usedbyrefreservation  0B                     -
vm/subvol-104-disk-0  logbias               latency                default
vm/subvol-104-disk-0  objsetid              69                     -
vm/subvol-104-disk-0  dedup                 off                    default
vm/subvol-104-disk-0  mlslabel              none                   default
vm/subvol-104-disk-0  sync                  standard               default
vm/subvol-104-disk-0  dnodesize             legacy                 default
vm/subvol-104-disk-0  refcompressratio      2.00x                  -
vm/subvol-104-disk-0  written               363M                   -
vm/subvol-104-disk-0  logicalused           661M                   -
vm/subvol-104-disk-0  logicalreferenced     661M                   -
vm/subvol-104-disk-0  volmode               default                default
vm/subvol-104-disk-0  filesystem_limit      none                   default
vm/subvol-104-disk-0  snapshot_limit        none                   default
vm/subvol-104-disk-0  filesystem_count      none                   default
vm/subvol-104-disk-0  snapshot_count        none                   default
vm/subvol-104-disk-0  snapdev               hidden                 default
vm/subvol-104-disk-0  acltype               posixacl               local
vm/subvol-104-disk-0  context               none                   default
vm/subvol-104-disk-0  fscontext             none                   default
vm/subvol-104-disk-0  defcontext            none                   default
vm/subvol-104-disk-0  rootcontext           none                   default
vm/subvol-104-disk-0  relatime              off                    default
vm/subvol-104-disk-0  redundant_metadata    all                    default
vm/subvol-104-disk-0  overlay               off                    default
vm/subvol-104-disk-0  encryption            off                    default
vm/subvol-104-disk-0  keylocation           none                   default
vm/subvol-104-disk-0  keyformat             none                   default
vm/subvol-104-disk-0  pbkdf2iters           0                      default
vm/subvol-104-disk-0  special_small_blocks  0                      default
 
zfs get all 100:

Code:
NAME                  PROPERTY              VALUE                  SOURCE
vm/subvol-100-disk-0  type                  filesystem             -
vm/subvol-100-disk-0  creation              Fri Jul 10 14:13 2020  -
vm/subvol-100-disk-0  used                  363M                   -
vm/subvol-100-disk-0  available             661M                   -
vm/subvol-100-disk-0  referenced            363M                   -
vm/subvol-100-disk-0  compressratio         2.00x                  -
vm/subvol-100-disk-0  mounted               no                     -
vm/subvol-100-disk-0  quota                 none                   default
vm/subvol-100-disk-0  reservation           none                   default
vm/subvol-100-disk-0  recordsize            128K                   default
vm/subvol-100-disk-0  mountpoint            /vm/subvol-100-disk-0  default
vm/subvol-100-disk-0  sharenfs              off                    default
vm/subvol-100-disk-0  checksum              on                     default
vm/subvol-100-disk-0  compression           on                     inherited from vm
vm/subvol-100-disk-0  atime                 on                     default
vm/subvol-100-disk-0  devices               on                     default
vm/subvol-100-disk-0  exec                  on                     default
vm/subvol-100-disk-0  setuid                on                     default
vm/subvol-100-disk-0  readonly              off                    default
vm/subvol-100-disk-0  zoned                 off                    default
vm/subvol-100-disk-0  snapdir               hidden                 default
vm/subvol-100-disk-0  aclinherit            restricted             default
vm/subvol-100-disk-0  createtxg             164030                 -
vm/subvol-100-disk-0  canmount              on                     default
vm/subvol-100-disk-0  xattr                 sa                     local
vm/subvol-100-disk-0  copies                1                      default
vm/subvol-100-disk-0  version               5                      -
vm/subvol-100-disk-0  utf8only              off                    -
vm/subvol-100-disk-0  normalization         none                   -
vm/subvol-100-disk-0  casesensitivity       sensitive              -
vm/subvol-100-disk-0  vscan                 off                    default
vm/subvol-100-disk-0  nbmand                off                    default
vm/subvol-100-disk-0  sharesmb              off                    default
vm/subvol-100-disk-0  refquota              1G                     local
vm/subvol-100-disk-0  refreservation        none                   default
vm/subvol-100-disk-0  guid                  3273447436791900640    -
vm/subvol-100-disk-0  primarycache          all                    default
vm/subvol-100-disk-0  secondarycache        all                    default
vm/subvol-100-disk-0  usedbysnapshots       0B                     -
vm/subvol-100-disk-0  usedbydataset         363M                   -
vm/subvol-100-disk-0  usedbychildren        0B                     -
vm/subvol-100-disk-0  usedbyrefreservation  0B                     -
vm/subvol-100-disk-0  logbias               latency                default
vm/subvol-100-disk-0  objsetid              112                    -
vm/subvol-100-disk-0  dedup                 off                    default
vm/subvol-100-disk-0  mlslabel              none                   default
vm/subvol-100-disk-0  sync                  standard               default
vm/subvol-100-disk-0  dnodesize             legacy                 default
vm/subvol-100-disk-0  refcompressratio      2.00x                  -
vm/subvol-100-disk-0  written               363M                   -
vm/subvol-100-disk-0  logicalused           661M                   -
vm/subvol-100-disk-0  logicalreferenced     661M                   -
vm/subvol-100-disk-0  volmode               default                default
vm/subvol-100-disk-0  filesystem_limit      none                   default
vm/subvol-100-disk-0  snapshot_limit        none                   default
vm/subvol-100-disk-0  filesystem_count      none                   default
vm/subvol-100-disk-0  snapshot_count        none                   default
vm/subvol-100-disk-0  snapdev               hidden                 default
vm/subvol-100-disk-0  acltype               posixacl               local
vm/subvol-100-disk-0  context               none                   default
vm/subvol-100-disk-0  fscontext             none                   default
vm/subvol-100-disk-0  defcontext            none                   default
vm/subvol-100-disk-0  rootcontext           none                   default
vm/subvol-100-disk-0  relatime              off                    default
vm/subvol-100-disk-0  redundant_metadata    all                    default
vm/subvol-100-disk-0  overlay               off                    default
vm/subvol-100-disk-0  encryption            off                    default
vm/subvol-100-disk-0  keylocation           none                   default
vm/subvol-100-disk-0  keyformat             none                   default
vm/subvol-100-disk-0  pbkdf2iters           0                      default
vm/subvol-100-disk-0  special_small_blocks  0                      default

zfs status:

Code:
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME                                                  STATE     READ WRITE CKSUM
        rpool                                                 ONLINE       0     0     0
          mirror-0                                            ONLINE       0     0     0
            ata-INTEL_SSDSC2KW120H6_CVLT616504GC120GGN-part3  ONLINE       0     0     0
            ata-INTEL_SSDSC2KW120H6_CVLT62800490120GGN-part3  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb     ONLINE       0     0     0
            sdc     ONLINE       0     0     0

errors: No known data errors

  pool: vm
 state: ONLINE
  scan: none requested
config:

        NAME                      STATE     READ WRITE CKSUM
        vm                        ONLINE       0     0     0
          wwn-0x5002538840001372  ONLINE       0     0     0

zpool import:

Code:
no pools available to import

ls -la /vm/subvol-104-disk-0:

Code:
drwxr-----  3 root root  3 Jul 10 16:55 .
drwxr-xr-x 11 root root 11 Jul 10 14:28 ..
drwxr-xr-x  2 root root  2 Jul 10 16:55 dev

ls -la /vm/subvol-100-disk-0:

Code:
drwxr-----  2 root root  2 Jul 10 14:13 .
drwxr-xr-x 11 root root 11 Jul 10 14:28 ..
 
as guessed - the dataset is not mounted - this is a glitch - which others have seen and worked around - try the following:

https://forum.proxmox.com/threads/update-broke-lxc.59776/#post-277303

I hope this helps!

Thank you for the help. I followed the steps from your post there, but it didn't work, the zpools still won't mount.

I get the follwoing from journalctl -b

Code:
Jul 12 12:40:28 pve systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Jul 12 12:40:28 pve systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
Jul 12 12:40:28 pve systemd[1]: Failed to start Mount ZFS filesystems.
 
I ended up fixing this via another thread by turning zfs set overlay=on poolname on.
 
I ended up fixing this via another thread by turning zfs set overlay=on poolname on.
should also work - it might be a bit cleaner - to export the pool - and clean up the directories which were created in error beneath the pools mountpoint
 
Thanks. So originally, there were only two folders under those volumes when not mounted. Are they just 'rubbish folders', I just need to delete them and it should all be fine when I remove the overlay command?
 
as always - make sure you have a working backup!

Thanks. So originally, there were only two folders under those volumes when not mounted. Are they just 'rubbish folders', I just need to delete them and it should all be fine when I remove the overlay command?
yes - export the pool - check the contents of the mountpoint `ls -Ral /vm` - and if it's only the dev directories - delete them - then the overlay option should not be needed anymore.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!