dev loop0

[SamTzu]

Hi guys.

My big is itching
My big is a server with about 10 KVM clients that each host about 10 LXC containers (nested virtualization is the way to go these days.) Lately I have seen several "dev loop0" problems usually with I/O errors indicating a malfunctioning disk causing problems with the disk image causing the infernal "Structure needs cleaning" error when trying to rsync or backup. I have just stopped the client and run a fsck -v -y on the file and all is good again but it's happening with different disks... hence the itch. I'm thinking its a memory issue somewhere.

Some background on the host:
Uptime: 445 days
Proxmox 5.1-52
RAM: 128GB
Storage: Lots of different types of "storage" including NAS. Mostly SSD but some larger SATA thrown in.
At first I thought problem was just with SSD drives.

Some background on the KVM clients:
Proxmox 5.4-6 cluster
RAM: 16-32GB RAM each.
Storage: Share the same NAS drives + host drives.

Some background on the nested LXC clients:
Debian based containers, mostly v8-9. Some Ubuntu in the mix.
RAM:1M-8M each.
Storage: Share the same NAS drives + host drives.

Any thoughts are appreciated.

Sam

 

[Stefan_R]

I would take a close look in your log files, your case does sound like either a disk or a memory issue. Are you using storage redundancy (ZFS, RAID, etc...)?

If you are using ECC RAM you can also check your kernel log on the host, it might help you regarding memory errors.

 

[SamTzu]

ZFS/Ext4 disks on the Host (& KVM hosts) and LVM/Ext4 on the NAS.
I did find several error messages for dm-17 (one of the disks) that said "lost async page write."
Rest of the dmesg complains are about network interface detecting hardware unit hang (probably from the VPN server).

 

[Stefan_R]

It would help to know if your damaged disk images are located on the NAS or locally.

The errors you are seeing definitely indicates some kind of hardware fault though (see also https://serverfault.com/a/866150).

ZFS *should* prevent such errors from affecting anything, provided you are not just striping your disks. Does 'zpool status' report anything?

 

[SamTzu]

So far the few errors have been on both Host and NAS.
The ZFS on host reports no know data errors.

 

[Stefan_R]

In that case I assume it's either a hardware fault or an issue with your NAS.

I would suggest upgrading and rebooting your host system and running a memtest to check for RAM errors. It might also help running a 'smartctl' diagnostic on your disks.