Hi! I am running an unprivileged LXC for Paperless-ngx, which uses ZFS storage on Proxmox (
So effectively I changed the LXC config from:
To:
Furthermore, today I moved Paperless LXC to another VLAN, and noticed that it cannot access the documents anymore. To my surprise there is only 50mb worth of files present in those datasets. I don't know whether the bind mount or VLAN changes had anything to do with this, but it worked just 2 day ago.
Detailed history (the only thing I did is loaded encryption key and mounted it after Proxmox VE reboot):
I now restored the config into previous
This is Paperless-ngx docker-compose file, which I haven't modified in the last months:
So in short, the only thing I touched on paperless side was the bind mount config. I also destroyed some temporary ZFS datasets day before this happened, but those have nothing to do with /
Any idea what could have happened, and if there is still any way to recover my files? It was 100% my fault for overlooking to snapshot and backup this dataset (I have a separate NAS to which I backup VMs/LXCs/ZFS snapshots), but I am just baffled how this could have happened. Has something like that happened before or is that certainly a user error?
Finally before I re-create this setup, is there anything I am doing wrong here (apart from lack of snapshots/backups)? I thought LXC with bind mounts to ZFS dataset would be a good approach.
Any advice much appreciated and happy to share any other logs.
tank/paperless dataset). Everything was working well for over a year, and yesterday I decided to change Bind mount config into lxc.mount.entry since that allows to make snapshots.So effectively I changed the LXC config from:
Bash:
mp0: VM-Drives:subvol-117-disk-1,mp=/data,backup=1,size=50G
mp1: /tank/paperless,mp=/mnt/paperless
mp2: /tank/paperless-db,mp=/mnt/paperless-db
Bash:
mp0: VM-Drives:subvol-117-disk-1,mp=/data,backup=1,size=50G
lxc.mount.entry: /tank/paperless mnt/paperless none bind,create=dir 0 0
lxc.mount.entry: /tank/paperless-db mnt/paperless-db none bind,create=dir 0 0Furthermore, today I moved Paperless LXC to another VLAN, and noticed that it cannot access the documents anymore. To my surprise there is only 50mb worth of files present in those datasets. I don't know whether the bind mount or VLAN changes had anything to do with this, but it worked just 2 day ago.
Bash:
[mylab] /tank/paperless/data# zfs get -o name,property,value encryption,encryptionroot,keystatus,mounted,mountpoint,used,referenced,available tank/paperless tank/paperless-db
NAME PROPERTY VALUE
tank/paperless encryption aes-256-gcm
tank/paperless encryptionroot tank/paperless
tank/paperless keystatus available
tank/paperless mounted yes
tank/paperless mountpoint /tank/paperless
tank/paperless used 54.4M
tank/paperless referenced 54.4M
tank/paperless available 541G
tank/paperless-db encryption aes-256-gcm
tank/paperless-db encryptionroot tank/paperless-db
tank/paperless-db keystatus available
tank/paperless-db mounted yes
tank/paperless-db mountpoint /tank/paperless-db
tank/paperless-db used 25.6M
tank/paperless-db referenced 25.6M
tank/paperless-db available 541GDetailed history (the only thing I did is loaded encryption key and mounted it after Proxmox VE reboot):
Bash:
[mylab] ~# zpool history tank -i -l | grep -i paperless
...
2026-03-21.06:32:13 zfs create -o encryption=aes-256-gcm -o keyformat=passphrase -o keylocation=prompt tank/paperless-db [user 0 (root) on mylab:linux]
2026-11-07.16:49:49 zfs load-key tank/paperless-db [user 0 (root) on mylab:linux]
2026-11-07.16:50:50 zfs load-key tank/paperless [user 0 (root) on mylab:linux]I now restored the config into previous
mp1/mp2 but nothing changed:
Bash:
[mylab] ~# pct config 117 | grep -E '^(mp|lxc.mount)'
mp0: VM-Drives:subvol-117-disk-1,mp=/data,backup=1,size=50G
mp1: /tank/paperless,mp=/mnt/paperless
mp2: /tank/paperless-db,mp=/mnt/paperless-db
[mylab] ~# pct exec 117 -- du -sh /mnt/paperless /mnt/paperless-db
55M/mnt/paperless
25M/mnt/paperless-dbThis is Paperless-ngx docker-compose file, which I haven't modified in the last months:
Bash:
paperless:
volumes:
- /mnt/paperless/data:/usr/src/paperless/data
- /mnt/paperless/media:/usr/src/paperless/media
- /mnt/paperless/export:/usr/src/paperless/export
- /mnt/paperless/consume:/usr/src/paperless/consume
paperless-postgres:
volumes:
- /var/lib/postgresql/data/var/lib/postgresql/data
[mylab] ~# pct exec 117 -- docker exec paperless sh -c 'du -sh /usr/src/paperless/media /usr/src/paperless/data; find /usr/src/paperless/media/documents -type f | head -20'
34K/usr/src/paperless/media
55M/usr/src/paperless/dataSo in short, the only thing I touched on paperless side was the bind mount config. I also destroyed some temporary ZFS datasets day before this happened, but those have nothing to do with /
tank/paperless{,-db}, which are used by Paperless-ngx:
Bash:
zpool history tank| grep -i destroy
...
2026-11-07.16:36:32 zfs destroy tank/permission-test/test2@migration
2026-11-07.16:37:37 zfs destroy -rv tank/permission-test
2026-11-07.16:38:06 zfs destroy -rv tank/test-incremental
2026-11-07.16:38:42 zfs destroy -rv tank/testAny idea what could have happened, and if there is still any way to recover my files? It was 100% my fault for overlooking to snapshot and backup this dataset (I have a separate NAS to which I backup VMs/LXCs/ZFS snapshots), but I am just baffled how this could have happened. Has something like that happened before or is that certainly a user error?
Finally before I re-create this setup, is there anything I am doing wrong here (apart from lack of snapshots/backups)? I thought LXC with bind mounts to ZFS dataset would be a good approach.
Any advice much appreciated and happy to share any other logs.
