VM-Disks are readonly / ZFS-Pool

linedan

Member
Feb 8, 2020
46
1
13
28
Today we had / have the following Problem, I think that the server maybe had no power for a few minutes and shutdown unexpected.
The problem is now that all VM-Disks seem to be read-only,
We have two VMs on it, and one with Windows does not even find the C-Partiton anymore.
The Linux-Server could have been restored after we synced the backup of etc-dir back on the server.
But all partions, which are devided in single disks, are not mountable or only read-only.

The Setup is the following:

ZFS-Data-Pool:
1666629821563.png

I can not find any Log or Error which would show why or how it all went read-only.
The proxmox seems to be healthy.
the other thing about it is that i don't want to restore everything again if it happens again, so is there any solution or just a WHY this could have been happening?
Would to do if ZFS-VM-Disks get read-only? I did not find anything about it.

Thank you and if more information needed, just ask, i will provide it immediately.
 
did your pool run full? what is the output of
Code:
zfs list
 
Thank you for the fast reply!
But i have to say that i do not think it was because the zfs-pool ran full.
But i also did not found any thing that shows if the server shutdown unexpected (maybe a tip how to find out? i already searched with grep in syslos)

The Output of "zfs list":

1666702542716.png



The Output of "zfs list -o space -r rpool":

1666702511361.png
 
you could post the syslog from the time the error occured (with some time before & after) and we can take a look if we see something
 
But i also did not found any thing that shows if the server shutdown unexpected (maybe a tip how to find out? i already searched with grep in syslos)
When reading the syslog instead of just grepping, you should easily be able to identify when the server was doing a graceful shutdown and when it was booting. When you just see the lines indicating a boot without the lines, before logging the shutdown of all the services, then the server probably crashed and you should have a look at the last lines before the boot happened.
 
you could post the syslog from the time the error occured (with some time before & after) and we can take a look if we see something
I can try, i already deleted all postfix messages in the syslog and there is one log from Saturday 9pm until sunday 9pm.
The proxmox server does a rsync backup everyday at 9pm, did it on saturday but no on sunday.
The other log is from friday 12pm until sunday 12am.
Maybe you see something i overlooked.
Thank you for the help!
 

Attachments

  • syslog.1_friday12am-sunday12am.txt
    26.5 KB · Views: 1
  • syslog.1_22.10.-9pm_23.10.-9pm.txt
    23.2 KB · Views: 1
The other question is, is there a possibility to make a zfs-vm-disk read-write again after such a crash?
How can I see if the vm-disk is read-only (except when the vm is not starting right)?
What would be the right way to fix this, or is it just a backup-restore?
 
Filesystems often enter a read-only state when something corrupted, for example by a hard reset. What filesystems are your guests using? For ext4 I would for example do a backup, then boot a Live Linux ISO and let fsck run over it to fix damaged parts.
 
Here is the last part of the syslog, after sunday 12am until monday 6pm.
Maybe there are also informationens which show the problem, i am also still searching.
 

Attachments

  • syslog.txt
    438.6 KB · Views: 0
Filesystems often enter a read-only state when something corrupted, for example by a hard reset. What filesystems are your guests using? For ext4 I would for example do a backup, then boot a Live Linux ISO and let fsck run over it to fix damaged parts.
1666709162788.png

1666709246090.png

I think the disks are all in raw format!
 
I mean the filesystem your virtual disks are formated with, not the filesystems of the OS. Or is the whole ZFS pool read only?
You can check the zvols with zfs get readonly.
 
I mean the filesystem your virtual disks are formated with, not the filesystems of the OS. Or is the whole ZFS pool read only?
You can check the zvols with zfs get readonly.
At the moment everything seems to be as it should:
1666875909183.png

I am not sure, my first thought was that the whole proxmox-filesystem is raed-only.
Because the two VMs startet but with errors and missing Configs / directories and partitions.
When we tried to bind in a partition with "mount -a" on the linux-server which is running as VM on this proxmox, it said something like "not possible, filesystem is readonly".
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!