Read-only file system problem when restarting CT

[gchad]

Hi guys!
All of a sudden, my containers have become unresponsive.

Running "pct list" tells me that all CTs are running fine.
But when I do "pct stop 101" I get:
unable to create output file '/var/log/pve/tasks/E/UPID:ns390657:000002FB:021B67F6:58D6C4CE:vzstop:101:root@pam:' - Read-only file system

Any idea of what is going on?

Thanks a bunch.

 

[dietmar]

Maybe your root file system is full? What is the output of 'df'?

 

[gchad]

Filesystem 1K-blocks Used Available Use% Mounted on

udev 10240 0 10240 0% /dev

tmpfs 26400356 82932 26317424 1% /run

/dev/sda2 20026236 1620360 17381928 9% /

tmpfs 66000888 34320 65966568 1% /dev/shm

tmpfs 5120 0 5120 0% /run/lock

tmpfs 66000888 0 66000888 0% /sys/fs/cgroup

/dev/mapper/pve-data 2858291064 197245364 2515846188 8% /var/lib/vz

cgmfs 100 0 100 0% /run/cgmanager/fs

/dev/fuse 30720 20 30700 1% /etc/pve

 

[dietmar]

Any hints in syslog?

 

[gchad]

So the syslog was not accessible.
nano /var/log/syslog was giving an error as well. (can't remember, but it was the same type of error with a permission denied).
So I asked for a manual restart of the machine. I am with OVH.
The machine has restarted on now everything is ok.

unfortunately syslog doesn't show anything passed yesterday... So I can't trouble shoot.

It is also worth noting that backups on some CTs have failed. One of them has gitlab installed on it. On the backup has failed 3 times already (not consecutively), causing the CT to be 'locked' and the overall system to crash. When I restarted it, I had to unlock the CT. I am on proxmox 4.3-10.
It is definitely not a space problem. There is plenty and daily backups or doing fine. It's just that sometime... one of them fails.... and blocks everything. Dunno in which respect it is correlated to my initial problem.

 

[sahostking]

How did you fix this?