VM-Disks are readonly / ZFS-Pool

[linedan]

Today we had / have the following Problem, I think that the server maybe had no power for a few minutes and shutdown unexpected.
The problem is now that all VM-Disks seem to be read-only,
We have two VMs on it, and one with Windows does not even find the C-Partiton anymore.
The Linux-Server could have been restored after we synced the backup of etc-dir back on the server.
But all partions, which are devided in single disks, are not mountable or only read-only.

The Setup is the following:

ZFS-Data-Pool:


I can not find any Log or Error which would show why or how it all went read-only.
The proxmox seems to be healthy.
the other thing about it is that i don't want to restore everything again if it happens again, so is there any solution or just a WHY this could have been happening?
Would to do if ZFS-VM-Disks get read-only? I did not find anything about it.

Thank you and if more information needed, just ask, i will provide it immediately.

 

[dcsapak]

did your pool run full? what is the output of

zfs list

 

[Dunuin]

zfs list

Or even better zfs list -o space -r rpool. That way we can also see if snapshots or missing discard is causing the pool to fill up, in case it's really read-only because it is full.

 

[linedan]

Thank you for the fast reply!
But i have to say that i do not think it was because the zfs-pool ran full.
But i also did not found any thing that shows if the server shutdown unexpected (maybe a tip how to find out? i already searched with grep in syslos)

The Output of "zfs list":





The Output of "zfs list -o space -r rpool":

 

[dcsapak]

you could post the syslog from the time the error occured (with some time before & after) and we can take a look if we see something

 

[Dunuin]

But i also did not found any thing that shows if the server shutdown unexpected (maybe a tip how to find out? i already searched with grep in syslos)

When reading the syslog instead of just grepping, you should easily be able to identify when the server was doing a graceful shutdown and when it was booting. When you just see the lines indicating a boot without the lines, before logging the shutdown of all the services, then the server probably crashed and you should have a look at the last lines before the boot happened.

 

[linedan]

you could post the syslog from the time the error occured (with some time before & after) and we can take a look if we see something

I can try, i already deleted all postfix messages in the syslog and there is one log from Saturday 9pm until sunday 9pm.
The proxmox server does a rsync backup everyday at 9pm, did it on saturday but no on sunday.
The other log is from friday 12pm until sunday 12am.
Maybe you see something i overlooked.
Thank you for the help!

 

[linedan]

The other question is, is there a possibility to make a zfs-vm-disk read-write again after such a crash?
How can I see if the vm-disk is read-only (except when the vm is not starting right)?
What would be the right way to fix this, or is it just a backup-restore?

 

[Dunuin]

Filesystems often enter a read-only state when something corrupted, for example by a hard reset. What filesystems are your guests using? For ext4 I would for example do a backup, then boot a Live Linux ISO and let fsck run over it to fix damaged parts.

 

[linedan]

Here is the last part of the syslog, after sunday 12am until monday 6pm.
Maybe there are also informationens which show the problem, i am also still searching.

 

[linedan]

Filesystems often enter a read-only state when something corrupted, for example by a hard reset. What filesystems are your guests using? For ext4 I would for example do a backup, then boot a Live Linux ISO and let fsck run over it to fix damaged parts.

I think the disks are all in raw format!

 

[Dunuin]

I mean the filesystem your virtual disks are formated with, not the filesystems of the OS. Or is the whole ZFS pool read only?
You can check the zvols with zfs get readonly.

 

[linedan]

I mean the filesystem your virtual disks are formated with, not the filesystems of the OS. Or is the whole ZFS pool read only?
You can check the zvols with zfs get readonly.

At the moment everything seems to be as it should:


I am not sure, my first thought was that the whole proxmox-filesystem is raed-only.
Because the two VMs startet but with errors and missing Configs / directories and partitions.
When we tried to bind in a partition with "mount -a" on the linux-server which is running as VM on this proxmox, it said something like "not possible, filesystem is readonly".

 

[linedan]

Any idea where this ready-only thing could have been come from?