Buffer I/O and EXT4-fs error/warning

[Iq6Nv]

Hey,

I don't know if this belongs more in Backup or Virtual Envoronment.


I have the problem that a VM returns the following errors during automatic backup:
~# dmesg -T | tail -n 10
[Wed Jan 19 21:52:44 2022] ? __sched_text_end+0x7/0x7
[Wed Jan 19 21:52:44 2022] default_idle+0x1c/0x140
[Wed Jan 19 21:52:44 2022] do_idle+0x1e3/0x270
[Wed Jan 19 21:52:44 2022] ? do_idle+0x18f/0x270
[Wed Jan 19 21:52:44 2022] cpu_startup_entry+0x6f/0x80
[Wed Jan 19 21:52:44 2022] start_secondary+0x1a4/0x200
[Wed Jan 19 21:52:44 2022] secondary_startup_64+0xa4/0xb0
[Wed Jan 19 21:52:44 2022] print_req_error: I/O error, dev vdf, sector 46786976
[Wed Jan 19 21:52:44 2022] EXT4-fs warning (device dm-0): ext4_end_bio:323: I/O error 10 writing to inode 394992 (offset 0 size 0 starting block 69419828)
[Wed Jan 19 21:52:44 2022] Buffer I/O error on device dm-0, logical block 69419828


Here the backup of the server seems to affect the VM somehow.

Is this a known error?

Setup:
Kernel: 4.19.0-18-amd64, Debian 10
Virtual Environment 6.3-3
SCSI Controller: VirtIO SCSI
Hard Disk: virtio0-5

 

[Iq6Nv]

No known errors or any tips on this?

 

[aaron]

The backup is going to a PBS? How fast is the network to the PBS and the PBS (disks) itself?
Are there other nodes that also run backups to the same PBS at the same time?

If a VM is backed up in snapshot mode, PVE is catching any write operation of the VM for parts of the disk which are not yet backed up. It then backs up those areas out of order and, once done, allows the write operation to continue. This way, a consistent backup of the VM disk from the time the backup is started can be guaranteed while the VM keeps running.

I can imagine that the VM reports those errors if it takes too long to back up the part of the disk where the VM wants to write to.

 

[Iq6Nv]

Yes, we have data centers with multiple VMs all going through the PBS. Ceph storages are mounted on the PBS. It is mainly limited by the HDDs. But the network in between is a 10G network.

Since there are several VMs and the affected one is the only one with this problem and at the same time not the biggest of the existing VMs, I don't assume that it is because of that.

But I will now look into it, that it does not come to several simultaneous backups and snapshots.
If it works after that or if there are problems again, I'll get back to you.

 

[linum]

I don't know if it's related. But that a look at Bug 1738839 - I/O error when virtio-blk disk is backed by a raw image on 4k disk. I had the described IO errors within a Debian based VM that uses ZFS as storage. Using the ZVOL outside the VM didn't trigger these IO errors. And it only occurs with ZVOL >= 8TB in my case. But, due to lack of time, I was never able to create a test case to report this problem/bug.

 

[Iq6Nv]

As I said, I'll get back to you.
I have analyzed our bandwidth usage and adjusted the backup time + buffer of each PBS accordingly. I also set all snapshots to times when the PBS does not make a backup.

Since 09.02. our monitoring has not even once reported an error. Therefore, I assume for now that the problem has been solved.

Thanks