root partition in ro but vms still running

[panucci_usr]

This morning one of our nodes went into ro mode. However, all the vms are still running. Trying to run a backup obviously fails with

unable to create output file '/var/log/pve/tasks/2/UPID:sxxx:0000A2D6:26430FCEE:66BB79A2:vzdump:103:root@pam:' - Read-only file system

Rebooting might or might not bring the system back. All vms are still running and responding to ping correctly.

Is there a way of backing up the vms externally without having to create an output file?

 

[gfngfn256]

Is there a way of backing up the vms externally without having to create an output file?

Manually copy all the VM's hard drives + each VM's .conf file to the external storage? I would probably shutdown those VMs before doing this.

If that external backup job/location is already setup within PVE - then possibly stopping rsyslog and journald, may help you to continue making a normal PVE backup.
If it is not already setup I doubt you'll manage to perform a standard PVE backup. But I'm only guessing.

Possibly if you could discover why your root FS has gone RO, you maybe able to fix it & be good to go. For instance maybe it is just filled up (e.g.. some mountpoint failed & something large got saved locally within the root).

Another thing I would run on the host pve-root is fsck to check for errors. Have you checked physical disk with smartctl?

Good luck.