XFS filesystem errors after VM restore

[robhost]

Can you test with EL 8? Maybe kernel related or depending on the qemu-guest-agent version, as we're 100% on EL

Versions:

pve-manager/6.3-3/eee5f901 (running kernel: 5.4.78-2-pve)

proxmox-backup-server 1.0.8-1 running version: 1.0.8

And no, we have no Ceph, PVE oder PBS errors.
With vzdump it works, so I dont think PVE or Ceph in general ist the problem here.

 

[t.lamprecht]

Can you please post the full vm config (qm config VMID)?

Also, is the Ceph storage in Proxmox VE configured to always use KRBD or not?

With vzdump it works, so I dont think PVE or Ceph in general ist the problem here.

So a vzdump restore of the same VM to ceph always works?

What distro does the ceph cluster nodes run on and where do the packages come from?

 

[robhost]

So a vzdump restore of the same VM to ceph always works?

Thats right. The problem only happens with PBS.

Ceph runs on Ubuntu 18.04 LTS from the official Ceph repo .

 

[robhost]

Sorry, here comes the rest:

agent: 1
boot: order=scsi0
cores: 2
cpu: Haswell-noTSX-IBRS,flags=+pcid;+spec-ctrl;+ssbd
memory: 2048
migrate_speed: 512
name: vs5480
net0: virtio=52:54:6F:99:A5:90,bridge=vmbr504,firewall=1
onboot: 1
ostype: l26
protection: 1
scsi0: ceph-vs:vm-5480-disk-0,discard=on,iops_rd=500,iops_rd_max=20000,iops_wr=500,iops_wr_max=20000,size=100G
scsihw: virtio-scsi-pci
serial0: socket
smbios1: sku=dnM1NDgwLm15bWFuYWdlZC5ob3N0,product=Vk0=,base64=1,serial=NTQ4MA==,uuid=bf3a44e0-d5bc-4cf6-9a67-19cddd75dc7a
sockets: 1
vmgenid: a6a139c8-5427-496c-9b56-d76418f9a1d2

> Also, is the Ceph storage in Proxmox VE configured to always use KRBD or not?

No, we're not using KRBD.

 

[t.lamprecht]

Thats right. The problem only happens with PBS.

Just finished a few backup restore cycles with CentOS 8 on my ceph cluster, all went just fine.

Ceph runs on Ubuntu 18.04 LTS from the official Ceph repo .

Ceph's official Debian/Ubuntu packaging is quite lacking, to say the least, but let's just assume for now that they did not borked it that bad, as then there'd be more outcry on the ceph-user list, I guess.

Nonetheless, if you can it'd be interesting to test a (small) PVE ceph setup and compare results with that.

As currently those are the facts we know:
* Ceph backup/restore works in general (we and lots of users here heavily use it)
* CentOS with LVM+XFS backup restore works also in general, at least nobody in this thread could reproduce your symptoms in any way
* The "problematic" backup of yours works just fine on local storage
* The sole place where it fails is a non-PVE ceph setup, which we have no control (build & patch-wise) over

So currently I really suspect that the Ceph setup, meaning the cluster itself or its incorporation into PVE.

The latter could be actually interesting and cheap to check: what ceph/rbd client/library version are installed on the PVE side?

The pveversion -v command includes ceph and ceph-fuse, in addition to that the version of librbd1 would be interesting (e.g., apt show librbd1).

 

[t.lamprecht]

If ceph and librbd and such are from Debian native you could try to just add the Ceph Octopus repositories provided for PVE and run a normal

apt update
apt full-upgrade

package updae (no need for installing ceph server components itself) and retry the restore.

https://pve.proxmox.com/wiki/Package_Repositories#_ceph_octopus_test_repository

 

[robhost]

The packages on the Ceph nodes are the official ones from here: http://download.ceph.com/debian-octopus/dists/bionic/

Ceph on the PVE nodes comes from the Debian dist and is quite old: 12.2.11+dfsg1-2.1+b1

We'll try the upgrade to the Octopus packages on the PVE nodes as next step and report back.

Thanks so far for your help!

 

[robhost]

We updated the Ceph client packages on the PVE nodes to 15.2.8-pve2 and restore works now as expected. The rbd itself has the same features still, so this must somehow depend on librbd.

Thanks for that hint and your help, we will now do further tests with PBS

 

[t.lamprecht]

We updated the Ceph client packages on the PVE nodes to 15.2.8-pve2 and restore works now as expected. The rbd itself has the same features still, so this must somehow depend on librbd.

Thanks for that hint and your help, we will now do further tests with PBS

Great to hear! Thanks for the feedback.