cannot write von /etc/pve/* anymore

[Shagnar]

Hi,

today I found out that there is a problem on my Proxmox Host (no HA, 1 Host, 1x 120g SSD for host and VMs and some data disks, home network use, standing in my flat).

whenever i want to make a new VM i get the following error:

Formatting '/var/lib/vz/images/107/vm-107-disk-1.raw', fmt=raw size=21474836480 
TASK ERROR: create failed - unable to open file '/etc/pve/nodes/proxmox/qemu-server/107.conf.tmp.230485' - Input/output error

therefore i tried to copy the config file from 106 to create a new VM:

root@proxmox:/etc/pve/qemu-server# cp 106.conf 107.conf
cp: cannot create regular file `107.conf': Input/output error

debug output:

root@proxmox:/etc/pve/qemu-server# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  pve    1   3   0 wz--n- 111.29g 13.87g

root@proxmox:/etc/pve/qemu-server# lvs
  LV   VG   Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
  data pve  -wi-ao-- 83.42g
  root pve  -wi-ao-- 10.00g
  swap pve  -wi-ao--  4.00g

root@proxmox:/etc/pve/qemu-server# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/pve-root  9.9G  4.2G  5.2G  45% /
tmpfs                 3.9G     0  3.9G   0% /lib/init/rw
udev                  3.9G  236K  3.9G   1% /dev
tmpfs                 3.9G   19M  3.9G   1% /dev/shm
/dev/mapper/pve-data   83G   48G   36G  58% /var/lib/vz
/dev/sda1             495M   34M  436M   8% /boot
[...data disks...]
/dev/fuse              30M   16K   30M   1% /etc/pve

/var/log/messages and dmesg don't show any errors.
smartctl self test of hdd did not report any errors.

some more information:

top:

top - 00:00:37 up 63 days, 13:41,  2 users,  load average: 0.01, 0.02, 0.00
Tasks: 213 total,   1 running, 212 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.9%us,  2.9%sy,  0.0%ni, 92.1%id,  0.0%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   8056448k total,  7830228k used,   226220k free,    22164k buffers
Swap:  4194296k total,   334896k used,  3859400k free,  2621832k cached

The machine currently hosts 7 VMs, 5 of them are running. No Problems. Outside of /etc/pve (meaning outside of /dev/fuse) no problems seem to exist.

Can somebody help me? Despite of the fact, that it is not possible to create any new VMs, i am concerned about the stability of the system.

 

[dietmar]

Do you run a cluster? If so, please post the output of

# pvecm status

You probably lost quorum?

If you run a single node try:

# service pve-cluster restart

That re-mounts the file system at /etc/pve

 

[Shagnar]

Hi Dietmar,

I don't run a cluster, its just a single node.

I restarted pve-cluster service and it is working again. Thank you very much. Luckily, no VMs were shutdown (altough the file system their disk images are on got remountet?).

Is this a good time for running apt-get update / apt-get dist-upgrade?

 

[tom]

you should always run the latest stable version - so yes, update always.

 

[Shagnar]

Okay. Thank you very much for helping.