[SOLVED] Read-Only-Filesystem on high I/O

[mbiberhofer]

Cheers,

since I started using proxmox, I am having serious issues in "high" (i.e. peak) I/O cases.

Setup description
Computiation power and memory is not an issue. The system is running on a HW-Managed RAID5 array, alltogether ~8TiB, about 20% used. About 10-15 VMs are running on it with different operating systems. pveversion reports "pve-manager/5.0-32/2560e073 (running kernel: 4.10.17-4-pve)"


Issue description
On Systems running on Linux with ext4, I regularly run into a quite troublesome issue: If Igenerate some I/O (i.e. some I/O delay happens), the root-file systems of the VMs are going into read-only error fallback. With "some I/O", I mean something like:

• A VM generates I/O by creating backups of application data
• A VM backup is generated by proxmox.

Sidenote: Every backup is timed so they do usually not run in parallel. On the Proxmox-Host, deadline is used as the I/O scheduler.


Question / Conclusion
Anyone here has an idea how to fix/avoid the issue? (Disabling backups is not an option)
This feels quite serious, and I'm running out of options.

Greetings,
Mario

P.S.: I just have the option to post to the german support forums. At least I didn't find any other option. Any clues how to reach the english support forums?

 

[dcsapak]

moved your thread

 

[czechsys]

Whats your HW config - disks, raid controller - cache, setup...etc. What is your backup app/vm size? What's your IO performance? I will say,it's will be slow - raid5 and 8TB = sata disks.

 

[mbiberhofer]

Can't access the specs right now; will deliver them later. I'll give you some pveperf-stats then, too .

Additional Information
For my part, I'll put my emphasis on a configuration issue (i.e. how proxmox is configured, and on that matter it has not been customized yet), due to the following reasons:

We've also got two proxmox-hosts running in a different cluster with Adaptec ASR-2405 RAID-Controllers, managing a 2x1.8TiB RAID1-Array (IIRC, it was "mostly autoconfigured"). We really have to generate *high* I/O on these for this effect to kick in, but it happened some times before.

This issue was also existent on hosts with ZFS-based VM storage. It was an experimental setup, therefore we had no ZIL-SSD installed there in the beginning - you could count on the fact that three of the ten test-VMs had remounted their root as R/O if we generated some I/O overnight - like a VM backup. Even after installing ZILs, this was still an issue. It happens quite frequently if you generated some "overall" I/O and then throw in some Peak I/O. With quite frequently, we're talking about "at least one VM has this issue once a day".

Also, I'd like to note that we migrated all of these hosts from Xen some time ago, where this issue was non-existent -- and we did not change the Hardware (neither the RAID, nor the general configuration) in any way. We also used ext4 as the default root file system for our VMs back then.

Cheers,
Mario

 

[mbiberhofer]

We changed the disk type to VirtIO for several VMs, now the problem is solved for these VMs.

Seems that not using VirtIO is not safe. Any other experiences on that?

 

[mbiberhofer]

Changing all Disks to using VirtIO, the problem disappeared. I'm gonna call it solved, though I guess there should be a warning about using any other block device virtualization driver.

 

[osoverflow]

I was testing this too, with two servers. One without SSD nor ZIL, and the other with ZIL into SSD. All the VMs with virtio-scsi. When I used a lot of IO on them the disks remount R/O.

I had writeback cache on the VMs.

Now I changed the cache to none, and ran the same task (uncompress a 17GB tar.gz full of files) and it is working nicely. Maybe something with the writeback cache?