Data on ZFS dataset mysteriously gone

W

westrangers

New Member
Jan 2, 2025
20
2
3
Hi! I am running an unprivileged LXC for Paperless-ngx, which uses ZFS storage on Proxmox (tank/paperless dataset). Everything was working well for over a year, and yesterday I decided to change Bind mount config into lxc.mount.entry since that allows to make snapshots.

So effectively I changed the LXC config from:
Bash: 
mp0: VM-Drives:subvol-117-disk-1,mp=/data,backup=1,size=50G
mp1: /tank/paperless,mp=/mnt/paperless
mp2: /tank/paperless-db,mp=/mnt/paperless-db
To:
Bash: 
mp0: VM-Drives:subvol-117-disk-1,mp=/data,backup=1,size=50G
lxc.mount.entry: /tank/paperless mnt/paperless none bind,create=dir 0 0
lxc.mount.entry: /tank/paperless-db mnt/paperless-db none bind,create=dir 0 0

Furthermore, today I moved Paperless LXC to another VLAN, and noticed that it cannot access the documents anymore. To my surprise there is only 50mb worth of files present in those datasets. I don't know whether the bind mount or VLAN changes had anything to do with this, but it worked just 2 day ago.
Bash: 
[mylab] /tank/paperless/data# zfs get -o name,property,value encryption,encryptionroot,keystatus,mounted,mountpoint,used,referenced,available tank/paperless tank/paperless-db
NAME               PROPERTY        VALUE
tank/paperless     encryption      aes-256-gcm
tank/paperless     encryptionroot  tank/paperless
tank/paperless     keystatus       available
tank/paperless     mounted         yes
tank/paperless     mountpoint      /tank/paperless
tank/paperless     used            54.4M
tank/paperless     referenced      54.4M
tank/paperless     available       541G
tank/paperless-db  encryption      aes-256-gcm
tank/paperless-db  encryptionroot  tank/paperless-db
tank/paperless-db  keystatus       available
tank/paperless-db  mounted         yes
tank/paperless-db  mountpoint      /tank/paperless-db
tank/paperless-db  used            25.6M
tank/paperless-db  referenced      25.6M
tank/paperless-db  available       541G

Detailed history (the only thing I did is loaded encryption key and mounted it after Proxmox VE reboot):
Bash: 
[mylab] ~# zpool history tank -i -l | grep -i paperless
...
2026-03-21.06:32:13 zfs create -o encryption=aes-256-gcm -o keyformat=passphrase -o keylocation=prompt tank/paperless-db [user 0 (root) on mylab:linux]
2026-11-07.16:49:49 zfs load-key tank/paperless-db [user 0 (root) on mylab:linux]
2026-11-07.16:50:50 zfs load-key tank/paperless [user 0 (root) on mylab:linux]

I now restored the config into previous mp1/mp2 but nothing changed:
Bash: 
[mylab] ~# pct config 117 | grep -E '^(mp|lxc.mount)'
mp0: VM-Drives:subvol-117-disk-1,mp=/data,backup=1,size=50G
mp1: /tank/paperless,mp=/mnt/paperless
mp2: /tank/paperless-db,mp=/mnt/paperless-db
[mylab] ~# pct exec 117 -- du -sh /mnt/paperless /mnt/paperless-db
55M/mnt/paperless
25M/mnt/paperless-db

This is Paperless-ngx docker-compose file, which I haven't modified in the last months:
Bash: 
paperless:
volumes:
- /mnt/paperless/data:/usr/src/paperless/data
- /mnt/paperless/media:/usr/src/paperless/media
- /mnt/paperless/export:/usr/src/paperless/export
- /mnt/paperless/consume:/usr/src/paperless/consume

paperless-postgres:
volumes:
- /var/lib/postgresql/data/var/lib/postgresql/data

[mylab] ~# pct exec 117 -- docker exec paperless sh -c 'du -sh /usr/src/paperless/media /usr/src/paperless/data; find /usr/src/paperless/media/documents -type f | head -20'
34K/usr/src/paperless/media
55M/usr/src/paperless/data

So in short, the only thing I touched on paperless side was the bind mount config. I also destroyed some temporary ZFS datasets day before this happened, but those have nothing to do with /tank/paperless{,-db}, which are used by Paperless-ngx:
Bash: 
zpool history tank| grep -i destroy
...
2026-11-07.16:36:32 zfs destroy tank/permission-test/test2@migration
2026-11-07.16:37:37 zfs destroy -rv tank/permission-test
2026-11-07.16:38:06 zfs destroy -rv tank/test-incremental
2026-11-07.16:38:42 zfs destroy -rv tank/test

Any idea what could have happened, and if there is still any way to recover my files? It was 100% my fault for overlooking to snapshot and backup this dataset (I have a separate NAS to which I backup VMs/LXCs/ZFS snapshots), but I am just baffled how this could have happened. Has something like that happened before or is that certainly a user error?

Finally before I re-create this setup, is there anything I am doing wrong here (apart from lack of snapshots/backups)? I thought LXC with bind mounts to ZFS dataset would be a good approach.

Any advice much appreciated and happy to share any other logs.
 
You have not shown us what actually exists in the datasets:

Code: 
find /tank/paperless-db /tank/paperless

There has to be something in there, because it uses storage according to your file
 
I quickly read the OP's post & will only comment the following;

westrangers said:
To my surprise there is only 50mb worth of files present in those datasets.
Click to expand...
According to your output, there is 54.4M in tank/paperless & 25.6M in tank/paperless-db of used data, so a total of 80M in "those datasets".
We can't tell what data loss has occurred, as OP provides no details of the actual loss. (I believe this is also LnxBill's point above). What size were these datasets before & what data did they contain that is now missing?

westrangers said:
Code: 
mp0: VM-Drives:subvol-117-disk-1,mp=/data,backup=1,size=50G
lxc.mount.entry: /tank/paperless mnt/paperless none bind,create=dir 0 0
lxc.mount.entry: /tank/paperless-db mnt/paperless-db none bind,create=dir 0 0
Click to expand...
This would not help to make snapshots, as you still have the first mp0.

westrangers said:
Code: 
paperless-postgres:
volumes: - /var/lib/postgresql/data/var/lib/postgresql/data
Click to expand...
colon missing?

Lastly, don't run docker in an LXC. Use a VM. (Search yourself on this topic).
 
  • Like
Reactions: Kingneutron
It seems like it's a freshly initialized version of paperless storage directory/db. There are no actual documents/PDFs in there, and whole space is used by 1 file (classification_model.pickle).
Bash: 
[mylab] ~# find /tank/paperless | head -20
/tank/paperless
/tank/paperless/media
/tank/paperless/media/documents
/tank/paperless/media/documents/thumbnails
/tank/paperless/media/documents/originals
/tank/paperless/export
/tank/paperless/consume
/tank/paperless/data
/tank/paperless/data/index
/tank/paperless/data/index/.index_settings.json
/tank/paperless/data/index/d878ffadb1304064b0f86bce0e273619.store
/tank/paperless/data/index/d878ffadb1304064b0f86bce0e273619.idx
/tank/paperless/data/index/d878ffadb1304064b0f86bce0e273619.fast
/tank/paperless/data/index/d878ffadb1304064b0f86bce0e273619.fieldnorm
/tank/paperless/data/index/2a57eadb9f074f06bd960e357f57441b.idx
/tank/paperless/data/index/.tantivy-writer.lock
/tank/paperless/data/index/meta.json
/tank/paperless/data/index/2a57eadb9f074f06bd960e357f57441b.fast
/tank/paperless/data/index/d878ffadb1304064b0f86bce0e273619.pos
/tank/paperless/data/index/2a57eadb9f074f06bd960e357f57441b.term
[mylab] ~# find /tank/paperless-db/ | head -20
/tank/paperless-db/
/tank/paperless-db/pg_twophase
/tank/paperless-db/pg_tblspc
/tank/paperless-db/pg_notify
/tank/paperless-db/pg_serial
/tank/paperless-db/pg_logical
/tank/paperless-db/pg_logical/replorigin_checkpoint
/tank/paperless-db/pg_logical/mappings
/tank/paperless-db/pg_logical/snapshots
/tank/paperless-db/postmaster.opts
/tank/paperless-db/pg_commit_ts
/tank/paperless-db/pg_wal
/tank/paperless-db/pg_wal/000000010000000000000004
/tank/paperless-db/pg_wal/000000010000000000000005
/tank/paperless-db/pg_wal/archive_status
/tank/paperless-db/pg_dynshmem
/tank/paperless-db/pg_multixact
/tank/paperless-db/pg_multixact/offsets
/tank/paperless-db/pg_multixact/offsets/0000
/tank/paperless-db/pg_multixact/members

[mylab] ~# ls /tank/paperless/media/documents
originals  thumbnails
[mylab] ~# ls /tank/paperless/media/documents/thumbnails/
[mylab] ~# ls /tank/paperless/media/documents/originals/

[mylab] /tank/paperless# du -sh data/*
4.5K    data/celerybeat-schedule.db
54M    data/classification_model.pickle
553K    data/index
83K    data/log
512    data/migration_lock
[mylab] ~

gfngfn256 said:
This would not help to make snapshots, as you still have the first mp0.


colon missing?

Lastly, don't run docker in an LXC. Use a VM. (Search yourself on this topic).
Click to expand...
Well after I changed 2 bind mounts to lxc.mount.entry I could immediately make a snapshot. Before that Proxmox GUI wouldn't allow me.

And copy paste error, colon is not missing in the original (- /mnt/paperless-db:/var/lib/postgresql/data).

Noted regarding Docker and LXC.. I did quite some research beforehand and people were using LXCs with docker without any issues. But now I will definitely reconsider.
 
  • Like
Reactions: Kingneutron
You must log in or register to reply here.
Top Bottom
Back