[SOLVED] Unable to use lxc on ZFS rpool

[fireon]

Hello,

pve-manager/4.1-5/f910ef5c (running kernel: 4.2.6-1-pve)

We would like to use lxc for some applicationvirtualisation. We only have an rpool:

pool: rpool
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
     raidz1-0  ONLINE       0     0     0
       sda2    ONLINE       0     0     0
       sdb2    ONLINE       0     0     0
       sdc2    ONLINE       0     0     0
     raidz1-1  ONLINE       0     0     0
       sdd2    ONLINE       0     0     0
       sde2    ONLINE       0     0     0
       sdf2    ONLINE       0     0     0

errors: No known data errors

But i was able to create an lxc. No on pve-1 and not on an extra Dataset vor lxc. And yes also not local, but i know this is normal. Here are the error:

mount: special device /rpool/ROOT/pve-1/lxc/subvol-100-disk-1 does not exist
mounting container failed
TASK ERROR: command 'mount -o bind /rpool/ROOT/pve-1/lxc/subvol-100-disk-1 /var/lib/lxc/100/rootfs/' failed: exit code 32

So what i have to do?

Thanks a lot

 

[Nemesiz]

Proxmox had a bug with ZFS mountpoint. If your pool (let say name tank) mountpoint not starts with / (like /tank ) Proxmox stops with this error. The way to pass it create catalog path and set mount point to it.
example.
1. mkdir -p /rpool/ROOT/pve-1/lxc
2. zfs set mountpoint /rpool/ROOT/pve-1/lxc rpool/ROOT/pve-1/lxc

 

[fireon]

Nice Bug It is working fine! Thank you.

zfs set mountpoint=/rpool/ROOT/pve-1/lxc rpool/ROOT/pve-1/lxc

 

[fireon]

Wikientry is done: https://pve.proxmox.com/wiki/Linux_Container#LXC_on_ZFS_Rpool

 

[Nemesiz]

I recommend you to edit wiki because it does not cover the beginning of the problem . First of all you don't need to create ZFS file system if you are using one of existing file system ( or you wont get into this problem )

Before this problem

mount: special device /rpool/ROOT/pve-1/lxc/subvol-100-disk-1 does not exist
mounting container failed
TASK ERROR: command 'mount -o bind /rpool/ROOT/pve-1/lxc/subvol-100-disk-1 /var/lib/lxc/100/rootfs/' failed: exit code 32

you can catch another problem in creating LXC. To reproduce it do these steps.

1. Create new ZFS file system. ( tank/test_fs )
2. Add to PROXMOX storage. ( tank/test_fs . ID test )
3. Create LXC on test storage.
4. On creating time you can catch mount problem and container files will appear /var/lib/lxc/VM_ID/rootfs instead of tank/test_fs//subvol-VM_ID-disk-1 ( If I remember correct )
5. Trying to start LXC you will get into previously problem.

So the PROXMOX problem locates in the same place - the mount action script.

 

[wbumiller]

Could someone please describe the exactly how to trigger this bug? I've never encountered this, not with using the pool directly, not with using a subdirectory of the pool.
The pool's storage section (from /etc/pve/storage.cfg) as well as all the changes made to the pool and datasets via `zpool set` and `zfs set` would be interesting.

 

[Nemesiz]

To trigger this bug you need:

1. ZFS pool with another location of mountpoint. ( ZFS pool: zfs_mirror mountpoint: /media/zfs_mirror )
2. Create ZFS file system for example zfs_mirror/test ( mountpoint will be /media/zfs_mirror/test )
3. Add ZFS storage ( ID: test ZFS Pool : zfs_mirror/test )
4. Create LXC CT on test storage

Result:

mount: special device /zfs_mirror/test/subvol-106-disk-1 does not exist
mounting container failed - command 'mount -o bind /zfs_mirror/test/subvol-106-disk-1 /var/lib/lxc/106/rootfs/' failed: exit code 32
extracting archive '/var/lib/vz/template/cache/debian-8.0-standard_8.0-1_amd64.tar.gz'
Total bytes read: 533012480 (509MiB, 104MiB/s)
Detected container architecture: amd64
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
Creating SSH host key 'ssh_host_key' - this may take some time ...
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
Creating SSH host key 'ssh_host_dsa_key' - this may take some time ...
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
TASK OK

As you can see PROXMOX use different path to mount. All CT files are extracted to /var/lib/lxc/ID/rootfs

Starting CT manually

# lxc-start -n 106 -F
mount: special device /zfs_mirror/test/subvol-106-disk-1 does not exist
command 'mount -o bind /zfs_mirror/test/subvol-106-disk-1 /var/lib/lxc/106/rootfs/' failed: exit code 32
lxc-start: conf.c: run_buffer: 342 Script exited with status 32
lxc-start: start.c: lxc_init: 436 failed to run pre-start hooks for container '106'.
lxc-start: start.c: __lxc_start: 1170 failed to initialize the container
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.

 

[fireon]

Ok, nice. I've deleted my wikientry for the first time. It works fine, but machine is unable to boot. Had an grub-error to mount the lxc dataset. So what the real right procedure to use lxc on an zfs rpool?

 

[Nemesiz]

As mention before change ZFS storage mountpoint. Before added to storage or after its not matter. But before creating LXC CT