VM with Ubuntu 18.04. Filesystem periodically enters in read-only mode

[antpervar]

Hi all,

We are experiencing a really annoying issue. From time to time and apparently without a reason (or we are not able to see it) all our Linux VM enter in read-only mode. The only way to get it work is restarting the VM with a LiveCD and executing a fsck over the filesystem. VM with Windows do work properly.

These are our specifications:


-> Proxmox : Proxmox 6.4.13
-> VM:
- OS: Ubuntu 18.04
- / mountpoint: is a ZFS Mirror disk. This partition (/) is around 25GB and when this issue happens it is 60% free. LVM activated.

Reviewing system logs, kernel messages, journalctl and our own monitoring systems, it has come to our attention that this issue is closely related with Proxmox Backup Server tasks. In our case, if a backup is frozen or if we stop the backup task manually, somehow, system gets corrupt and enters in read only mode. We are unsure if problem comes from a backup frozen (perhaps by a high I/O) or the cause is stopping manually the backup.

Backing up the system is a must for us, but at this moment we do not trust totally the backup mechanism as looks like it is affecting filesystem. But we are not sure about this diagnosis.

May you be so kind as to help us to find the cause of this behaviour?

 

[antpervar]

Hi all,

We are still analysing the incidence but unsuccessfully. May the issue come from LVM in Ubuntu instead of a high I/O when performing the backup?

Is ZFS safe enough to be used with Proxmox? All the VM we have experienced the issue with are mounted over ZFS.

Thanks!

 

[antpervar]

Hi all,

We would really appreciate any feedback from someone who has suffered the same issue. We are really lost and confused with this issue.

Thank you very much in advance!

 

[antpervar]

Hi all,

We have experienced the issue again. Executing a fsck fixed the problem but it is a random issue as it happens in different proxmox servers with different disks models where vmdisks are stored. We have experienced it in two different hosts, with Intel SSD disks in Mirror (ZFS) All the Linux VM have experienced this issue at this moment except the one without LVM implemented. Maybe this could be a clue to try to fix the issue. Disks are around 3 years old and smart values look fine.

Although this post is not answered, we will continue updating it, ust in case someone experience this issue. Hopefully we will be able to find the issue and could be helpful for someone.

Regards!

 

[antpervar]

Hi all,

Although we are not getting any answer (or properly speaking, we are answering ourselves ) we will continue updating the post for future new proxmox users just in case they experience the same issue.

Today we had a couple of systems in Read Only mode. Fortunately, we had configured our Zabbix to get the server's filesystem status and we had an alert when this condition happened. System's entered in Read Only mode while the backup was running (Local Backup) and failed due to lack of space in the destination disk. Once disk is full, backup failed and system entered in Read Only mode.

We will try to free up some space (if possible) or we will disable the backups, just in order to test if issues are related.

Regards!