btrfs inside virtualdisk - errors

[techadmin-local]

Hello
We use btrfs in a couple of VMs in combination with samba to use the benefit of btrfs snapshots for file and directory history. We use it with Debian (UCS 5) and Ubuntu (Zentyal 7.0), but we have almost every week some errors. Sometimes we can fix it with btrfs scrub start -Bf /dev/... and sometimes we have to restore backups. We tried to find ways to handle this, but still no solution. We have good experience with btrfs, if we use it on hardware (no samba).
Do we have to configure any parameters or configs to run it smooth inside virtual disks?
Our disk setup:
scsi1, Default (no cache), IO threads enabled, Backup yes, Async IO Default (io_uring)

Thanks for any advice

 

[Maximiliano]

Hello,

To better understand the setup, could you please send us the contents of

- The VM config at /etc/pve/qemu-server/<VM_ID>.conf
- The storage config at /etc/pve/storage.cfg

When you say that you have "errors", what exactly do you mean?

 

[techadmin-local]

VM config:

#Restored on the 10. Sept. 2023
#restored from backup-date%3A 21%3A00h 09. Sept. 2023
agent: 1
boot: order=scsi0
cores: 5
ide2: none,media=cdrom
memory: 10240
meta: creation-qemu=6.1.1,ctime=1647613809
name: zentyal1
net0: virtio=DE....,bridge=vmbr2,firewall=1
numa: 0
onboot: 1
ostype: l26
parent: after_restore
protection: 1
scsi0: data1:vm-111-disk-0,size=30G
scsi1: data1:vm-111-disk-1,size=400G
scsihw: virtio-scsi-pci
smbios1: uuid=89296e91-61b0-4e55-b45a-a95facf78a66
sockets: 1
vmgenid: 5098974b-d1dc-4e36-a9a2-e743e4209e03

[after_restore]
agent: 1
boot: order=scsi0
cores: 5
ide2: none,media=cdrom
memory: 10240
meta: creation-qemu=6.1.1,ctime=1647613809
name: zentyal1
net0: virtio=DE:....,bridge=vmbr2,firewall=1
numa: 0
onboot: 1
ostype: l26
protection: 1
runningcpu: kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
runningmachine: pc-i440fx-9.0+pve0
scsi0: data1:vm-111-disk-0,size=30G
scsi1: data1:vm-111-disk-1,size=400G
scsihw: virtio-scsi-pci
smbios1: uuid=89296e91-61b0-4e55-b45a-a95facf78a66
snaptime: 1723398864
sockets: 1
vmgenid: 5098974b-d1dc-4e36-a9a2-e743e4209e03
vmstate: data1:vm-111-state-after_restore

storage config:

btrfs: local-btrfs
        path /var/lib/pve/local-btrfs
        content vztmpl,backup,rootdir,iso,images

zfspool: data1
        pool datapool/data1
        content images,rootdir
        mountpoint /datapool/data1
        sparse 0

Our monitoring is using the following command to check the health status of btrfs: btrfs check --force /dev/sd... - it shows up some errors
Unfortunately I don't have an error example right at the moment - I will submit it later, as soon I have one.

 

[Maximiliano]

I am a bit confused by your setup, the VMs are using the `data1` storage which is a zfs based storage as per the config. Do you use btrfs inside of the guests? Is this a san using brtfs and then you mount the storage somewhere and install zfs on top of that?

 

[techadmin-local]

We have a server with two disks, installed with a ZFS mirror setup, which is used as storage for all local VMs. One VM has a virtual disk, which is formatted as btrfs inside the VM.

 

[Kingneutron]

Yeah don't try combining 2x COW filesystems, you probably won't like the results.

btrfs code is shite and pretty much always has been. If you have ZFS on the "outside" just stick to that and format in-vm as ext4 or XFS.

If you really need snapshots in-vm, move the vdisk to non-COW storage (again, e.g. lvm-thin, ext4 / xfs) and use ZFS in-vm - it's much more reliable and tested over years than btrfs.

 

[techadmin-local]

Yeah, that's what we thought. So we already started to migrate the "btrfs" systems to the zfs version, which are perfect stable. We hoped to find a easier way to make it more stable.

 

[JensF]

Did you turn off COW in your VM? If not try the mount option nodatacow for your BTRFS mount point inside the VM.

 

[Maximiliano]

The problem with one COW filesystem on top of another COW filesystem is that you will have write amplification, in certain workloads the host could end up writing a dozen times the expected amount (c.f. [1]). This should not be an issue in theory, the guest will "just" see a extremely slow disk and the host will "just" use multiple times the expected storage for the VM image, but it is not a given that the guest will handle gracefully such slow IO.

One could disable COW on the guest, but at that point I would recommend to use another filesystem and leave the host filesystem do the data integrity checks.

[1] https://forum.proxmox.com/threads/understanding-and-minimizing-zfs-write-amplification.83096/

 

[techadmin-local]

Just to complete this reqeust, here is the error log after you run btrfs check --force /dev/sdb1

btrfs check --force /dev/sdb1
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
parent transid verify failed on 33390592 wanted 13846 found 14048
parent transid verify failed on 33390592 wanted 13846 found 14048
parent transid verify failed on 33390592 wanted 13846 found 14048
parent transid verify failed on 33390592 wanted 13846 found 14048
Ignoring transid failure
Checking filesystem on /dev/sdb1
UUID: 4d1db6f5-32aa-47e9-b786-1d3ecdd5591A
[1/7] checking root items
[2/7] checking extents
Chunk[256, 228, 13631488] stripe[1, 13631488] is not found in dev extent
Chunk[256, 228, 30408704] stripe[1, 38797312] is not found in dev extent
Chunk[256, 228, 30408704] stripe[1, 1112539136] is not found in dev extent
Chunk[256, 228, 1104150528] stripe[1, 2186280960] is not found in dev extent
Chunk[256, 228, 2177892352] stripe[1, 3260022784] is not found in dev extent
Chunk[256, 228, 3251634176] stripe[1, 4333764608] is not found in dev extent
Chunk[256, 228, 4325376000] stripe[1, 5407506432] is not found in dev extent
Chunk[256, 228, 5399117824] stripe[1, 6481248256] is not found in dev extent
Chunk[256, 228, 6472859648] stripe[1, 7554990080] is not found in dev extent
Chunk[256, 228, 7546601472] stripe[1, 8628731904] is not found in dev extent
Chunk[256, 228, 8620343296] stripe[1, 9702473728] is not found in dev extent
Chunk[256, 228, 9694085120] stripe[1, 10776215552] is not found in dev extent
Chunk[256, 228, 10767826944] stripe[1, 11849957376] is not found in dev extent
Chunk[256, 228, 11841568768] stripe[1, 12923699200] is not found in dev extent
Chunk[256, 228, 12915310592] stripe[1, 13997441024] is not found in dev extent
Chunk[256, 228, 13989052416] stripe[1, 15071182848] is not found in dev extent
Chunk[256, 228, 15062794240] stripe[1, 16144924672] is not found in dev extent
Chunk[256, 228, 16136536064] stripe[1, 17218666496] is not found in dev extent
Chunk[256, 228, 17210277888] stripe[1, 18292408320] is not found in dev extent
Chunk[256, 228, 18284019712] stripe[1, 19366150144] is not found in dev extent
Chunk[256, 228, 19357761536] stripe[1, 20439891968] is not found in dev extent
Chunk[256, 228, 20431503360] stripe[1, 21513633792] is not found in dev extent
Chunk[256, 228, 21505245184] stripe[1, 22587375616] is not found in dev extent
Chunk[256, 228, 22578987008] stripe[1, 23661117440] is not found in dev extent
Chunk[256, 228, 23652728832] stripe[1, 24734859264] is not found in dev extent
Chunk[256, 228, 24726470656] stripe[1, 25808601088] is not found in dev extent
Chunk[256, 228, 25800212480] stripe[1, 26882342912] is not found in dev extent
Chunk[256, 228, 26873954304] stripe[1, 27956084736] is not found in dev extent
Chunk[256, 228, 27947696128] stripe[1, 29029826560] is not found in dev extent
Chunk[256, 228, 29021437952] stripe[1, 30103568384] is not found in dev extent
Chunk[256, 228, 30095179776] stripe[1, 31177310208] is not found in dev extent
Chunk[256, 228, 31168921600] stripe[1, 32251052032] is not found in dev extent
Chunk[256, 228, 32242663424] stripe[1, 33324793856] is not found in dev extent
Chunk[256, 228, 33316405248] stripe[1, 34398535680] is not found in dev extent
Chunk[256, 228, 34390147072] stripe[1, 35472277504] is not found in dev extent
Chunk[256, 228, 35463888896] stripe[1, 36546019328] is not found in dev extent
Chunk[256, 228, 36537630720] stripe[1, 37619761152] is not found in dev extent
Chunk[256, 228, 37611372544] stripe[1, 38693502976] is not found in dev extent
Chunk[256, 228, 38685114368] stripe[1, 39767244800] is not found in dev extent
Chunk[256, 228, 39758856192] stripe[1, 40840986624] is not found in dev extent
Chunk[256, 228, 40832598016] stripe[1, 41914728448] is not found in dev extent
Chunk[256, 228, 41906339840] stripe[1, 42988470272] is not found in dev extent
Chunk[256, 228, 42980081664] stripe[1, 44062212096] is not found in dev extent
Chunk[256, 228, 44053823488] stripe[1, 45135953920] is not found in dev extent
Chunk[256, 228, 45127565312] stripe[1, 46209695744] is not found in dev extent
Chunk[256, 228, 46201307136] stripe[1, 47283437568] is not found in dev extent
Chunk[256, 228, 47275048960] stripe[1, 48357179392] is not found in dev extent
Chunk[256, 228, 48348790784] stripe[1, 49430921216] is not found in dev extent
Chunk[256, 228, 49422532608] stripe[1, 50504663040] is not found in dev extent
Chunk[256, 228, 50496274432] stripe[1, 51578404864] is not found in dev extent
Chunk[256, 228, 51570016256] stripe[1, 52652146688] is not found in dev extent
Chunk[256, 228, 52643758080] stripe[1, 53725888512] is not found in dev extent
Chunk[256, 228, 53717499904] stripe[1, 54799630336] is not found in dev extent
Chunk[256, 228, 54791241728] stripe[1, 55873372160] is not found in dev extent
Chunk[256, 228, 55864983552] stripe[1, 56947113984] is not found in dev extent
Chunk[256, 228, 56938725376] stripe[1, 58020855808] is not found in dev extent
Chunk[256, 228, 58012467200] stripe[1, 59094597632] is not found in dev extent
Chunk[256, 228, 59086209024] stripe[1, 60168339456] is not found in dev extent
Chunk[256, 228, 60159950848] stripe[1, 61242081280] is not found in dev extent
Chunk[256, 228, 61233692672] stripe[1, 62315823104] is not found in dev extent
Chunk[256, 228, 62307434496] stripe[1, 63389564928] is not found in dev extent
Chunk[256, 228, 63381176320] stripe[1, 64463306752] is not found in dev extent
Chunk[256, 228, 64454918144] stripe[1, 65537048576] is not found in dev extent
Chunk[256, 228, 65528659968] stripe[1, 66610790400] is not found in dev extent
Chunk[256, 228, 66602401792] stripe[1, 67684532224] is not found in dev extent
Chunk[256, 228, 67676143616] stripe[1, 68758274048] is not found in dev extent
Chunk[256, 228, 68749885440] stripe[1, 69832015872] is not found in dev extent
Chunk[256, 228, 69823627264] stripe[1, 70905757696] is not found in dev extent
Chunk[256, 228, 70897369088] stripe[1, 71979499520] is not found in dev extent
Chunk[256, 228, 71971110912] stripe[1, 73053241344] is not found in dev extent
Chunk[256, 228, 73044852736] stripe[1, 74126983168] is not found in dev extent
Chunk[256, 228, 74118594560] stripe[1, 75200724992] is not found in dev extent
Chunk[256, 228, 75192336384] stripe[1, 76274466816] is not found in dev extent
Chunk[256, 228, 76266078208] stripe[1, 77348208640] is not found in dev extent
Chunk[256, 228, 77339820032] stripe[1, 78421950464] is not found in dev extent
Chunk[256, 228, 78413561856] stripe[1, 79495692288] is not found in dev extent
Chunk[256, 228, 79487303680] stripe[1, 80569434112] is not found in dev extent
Chunk[256, 228, 80561045504] stripe[1, 81643175936] is not found in dev extent
Chunk[256, 228, 81634787328] stripe[1, 82716917760] is not found in dev extent
Chunk[256, 228, 82708529152] stripe[1, 83790659584] is not found in dev extent
Chunk[256, 228, 83782270976] stripe[1, 84864401408] is not found in dev extent
Chunk[256, 228, 84856012800] stripe[1, 85938143232] is not found in dev extent
Chunk[256, 228, 85929754624] stripe[1, 87011885056] is not found in dev extent
Chunk[256, 228, 87003496448] stripe[1, 88085626880] is not found in dev extent
Chunk[256, 228, 88077238272] stripe[1, 89159368704] is not found in dev extent
Chunk[256, 228, 89150980096] stripe[1, 90233110528] is not found in dev extent
Chunk[256, 228, 90224721920] stripe[1, 91306852352] is not found in dev extent
Chunk[256, 228, 91298463744] stripe[1, 92380594176] is not found in dev extent
Chunk[256, 228, 92372205568] stripe[1, 93454336000] is not found in dev extent
Chunk[256, 228, 93445947392] stripe[1, 94528077824] is not found in dev extent
Chunk[256, 228, 94519689216] stripe[1, 95601819648] is not found in dev extent
Chunk[256, 228, 95593431040] stripe[1, 96675561472] is not found in dev extent
Chunk[256, 228, 96667172864] stripe[1, 97749303296] is not found in dev extent
Chunk[256, 228, 97740914688] stripe[1, 98823045120] is not found in dev extent
Chunk[256, 228, 98814656512] stripe[1, 99896786944] is not found in dev extent
Chunk[256, 228, 99888398336] stripe[1, 100970528768] is not found in dev extent
Chunk[256, 228, 100962140160] stripe[1, 102044270592] is not found in dev extent
Chunk[256, 228, 102035881984] stripe[1, 103118012416] is not found in dev extent
Chunk[256, 228, 103109623808] stripe[1, 104191754240] is not found in dev extent
Chunk[256, 228, 104183365632] stripe[1, 105265496064] is not found in dev extent
Chunk[256, 228, 105257107456] stripe[1, 106339237888] is not found in dev extent
Chunk[256, 228, 106330849280] stripe[1, 107412979712] is not found in dev extent
Chunk[256, 228, 107404591104] stripe[1, 108486721536] is not found in dev extent
Chunk[256, 228, 108478332928] stripe[1, 109560463360] is not found in dev extent
Chunk[256, 228, 109552074752] stripe[1, 110634205184] is not found in dev extent
Chunk[256, 228, 110625816576] stripe[1, 111707947008] is not found in dev extent
Chunk[256, 228, 111699558400] stripe[1, 112781688832] is not found in dev extent
Chunk[256, 228, 112773300224] stripe[1, 113855430656] is not found in dev extent
Chunk[256, 228, 113847042048] stripe[1, 114929172480] is not found in dev extent
Chunk[256, 228, 114920783872] stripe[1, 116002914304] is not found in dev extent
Chunk[256, 228, 115994525696] stripe[1, 117076656128] is not found in dev extent
Chunk[256, 228, 117068267520] stripe[1, 118150397952] is not found in dev extent
Chunk[256, 228, 118142009344] stripe[1, 119224139776] is not found in dev extent
Chunk[256, 228, 119215751168] stripe[1, 120297881600] is not found in dev extent
Chunk[256, 228, 120289492992] stripe[1, 121371623424] is not found in dev extent
Chunk[256, 228, 121363234816] stripe[1, 122445365248] is not found in dev extent
Chunk[256, 228, 122436976640] stripe[1, 123519107072] is not found in dev extent
Chunk[256, 228, 123510718464] stripe[1, 124592848896] is not found in dev extent
Chunk[256, 228, 124584460288] stripe[1, 125666590720] is not found in dev extent
Chunk[256, 228, 125658202112] stripe[1, 126740332544] is not found in dev extent
Chunk[256, 228, 126731943936] stripe[1, 127814074368] is not found in dev extent
Chunk[256, 228, 127805685760] stripe[1, 128887816192] is not found in dev extent
Chunk[256, 228, 128879427584] stripe[1, 129961558016] is not found in dev extent
Chunk[256, 228, 129953169408] stripe[1, 131035299840] is not found in dev extent
Chunk[256, 228, 131026911232] stripe[1, 132109041664] is not found in dev extent
Chunk[256, 228, 132100653056] stripe[1, 133182783488] is not found in dev extent
Chunk[256, 228, 133174394880] stripe[1, 134256525312] is not found in dev extent
Chunk[256, 228, 134248136704] stripe[1, 135330267136] is not found in dev extent
Chunk[256, 228, 135321878528] stripe[1, 136404008960] is not found in dev extent
Chunk[256, 228, 136395620352] stripe[1, 137477750784] is not found in dev extent
Chunk[256, 228, 137469362176] stripe[1, 138551492608] is not found in dev extent
Chunk[256, 228, 138543104000] stripe[1, 139625234432] is not found in dev extent
Chunk[256, 228, 139616845824] stripe[1, 140698976256] is not found in dev extent
Chunk[256, 228, 140690587648] stripe[1, 141772718080] is not found in dev extent
Chunk[256, 228, 141764329472] stripe[1, 142846459904] is not found in dev extent
Chunk[256, 228, 142838071296] stripe[1, 143920201728] is not found in dev extent
Chunk[256, 228, 143911813120] stripe[1, 144993943552] is not found in dev extent
Chunk[256, 228, 144985554944] stripe[1, 146067685376] is not found in dev extent
Chunk[256, 228, 146059296768] stripe[1, 147141427200] is not found in dev extent
Chunk[256, 228, 147133038592] stripe[1, 148215169024] is not found in dev extent
Chunk[256, 228, 148206780416] stripe[1, 149288910848] is not found in dev extent
Chunk[256, 228, 149280522240] stripe[1, 150362652672] is not found in dev extent
Chunk[256, 228, 150354264064] stripe[1, 151436394496] is not found in dev extent
Chunk[256, 228, 151428005888] stripe[1, 152510136320] is not found in dev extent
Chunk[256, 228, 152501747712] stripe[1, 153583878144] is not found in dev extent
Chunk[256, 228, 153575489536] stripe[1, 154657619968] is not found in dev extent
Chunk[256, 228, 154649231360] stripe[1, 155731361792] is not found in dev extent
Chunk[256, 228, 155722973184] stripe[1, 156805103616] is not found in dev extent
Chunk[256, 228, 156796715008] stripe[1, 157878845440] is not found in dev extent
Chunk[256, 228, 157870456832] stripe[1, 158952587264] is not found in dev extent
Chunk[256, 228, 158944198656] stripe[1, 160026329088] is not found in dev extent
Chunk[256, 228, 160017940480] stripe[1, 161100070912] is not found in dev extent
Chunk[256, 228, 161091682304] stripe[1, 162173812736] is not found in dev extent
Chunk[256, 228, 162165424128] stripe[1, 163247554560] is not found in dev extent
Chunk[256, 228, 163239165952] stripe[1, 164321296384] is not found in dev extent
Chunk[256, 228, 164312907776] stripe[1, 165395038208] is not found in dev extent
Chunk[256, 228, 165386649600] stripe[1, 166468780032] is not found in dev extent
Chunk[256, 228, 166460391424] stripe[1, 167542521856] is not found in dev extent
Chunk[256, 228, 167534133248] stripe[1, 168616263680] is not found in dev extent
Chunk[256, 228, 168607875072] stripe[1, 169690005504] is not found in dev extent
Chunk[256, 228, 169681616896] stripe[1, 170763747328] is not found in dev extent
Chunk[256, 228, 170755358720] stripe[1, 171837489152] is not found in dev extent
Chunk[256, 228, 171829100544] stripe[1, 172911230976] is not found in dev extent
Chunk[256, 228, 172902842368] stripe[1, 173984972800] is not found in dev extent
Chunk[256, 228, 173976584192] stripe[1, 175058714624] is not found in dev extent
Chunk[256, 228, 175050326016] stripe[1, 176132456448] is not found in dev extent
Chunk[256, 228, 176124067840] stripe[1, 177206198272] is not found in dev extent
Chunk[256, 228, 177197809664] stripe[1, 178279940096] is not found in dev extent
Chunk[256, 228, 178271551488] stripe[1, 179353681920] is not found in dev extent
Chunk[256, 228, 179345293312] stripe[1, 180427423744] is not found in dev extent
Chunk[256, 228, 180419035136] stripe[1, 181501165568] is not found in dev extent
Chunk[256, 228, 181492776960] stripe[1, 182574907392] is not found in dev extent
Chunk[256, 228, 182566518784] stripe[1, 183648649216] is not found in dev extent
Chunk[256, 228, 183640260608] stripe[1, 184722391040] is not found in dev extent
Chunk[256, 228, 184714002432] stripe[1, 185796132864] is not found in dev extent
Chunk[256, 228, 185787744256] stripe[1, 186869874688] is not found in dev extent
owner ref check failed [33390592 16384]
ref mismatch on [33406976 16384] extent item 1, found 0
backref 33406976 root 4 not referenced back 0x55733cbe7890
incorrect global backref count on 33406976 found 1 wanted 0
backpointer mismatch on [33406976 16384]
owner ref check failed [33406976 16384]
ref mismatch on [334102528 16384] extent item 1, found 0
backref 334102528 root 4 not referenced back 0x55733cf82cc0
incorrect global backref count on 334102528 found 1 wanted 0
backpointer mismatch on [334102528 16384]
owner ref check failed [334102528 16384]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
ERROR: transid errors in file system
found 301326376960 bytes used, error(s) found
total csum bytes: 292324764
total tree bytes: 1906851840
total fs tree bytes: 1554890752
total extent tree bytes: 41074688
btree space waste bytes: 397896272
file data blocks allocated: 628467212288
 referenced 628466630656

However, we plan to rebuild those systems.