Transport endpoint is not connected

[ultrex]

Hello,

I want to create VM. I go to www interface and page ~90% load and crash system ?
I try restart pve-cluster but I get error:

root@i1:~# /etc/init.d/pve-cluster restart
Restarting pve cluster filesystem: pve-clusterstart-stop-daemon: warning: failed to kill 2605: No such process
[main] crit: unable to create lock '/var/lib/pve-cluster/.pmxcfs.lockfile': Read-only file system
[main] notice: exit proxmox configuration filesystem (-1)
(warning).
root@i1:~# ls -la /etc/pve/
ls: cannot access /etc/pve/: Transport endpoint is not connected
root@i1:~#

Can help ? This server is production server with active customers. I cannot reboot becouse boot time very long and customers virtual server timeout very bad


edit:
I cannot create file on /:
root@i1:~# touch /qe
touch: cannot touch `/qe': Read-only file system

 

[acidrop]

If you cannot create a file on / then you must have some problem with filesystem on that or it's backing storage(corruption?).Do you have free space on / ?
Also /etc/pve is read-only.Do you have cluster? What is the quorum status? (pvecm n , pvecm s)
Can you post some lines from dmesg and /var/log/messages?

 

[ultrex]

dmesg last lines:

EXT4-fs (sdb1): recovery complete
EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts:
EXT4-fs (sdd1): recovery complete
EXT4-fs (sdd1): mounted filesystem with ordered data mode. Opts:
EXT3-fs: barriers disabled
kjournald starting. Commit interval 5 seconds
EXT3-fs (sde1): using internal journal
EXT3-fs (sde1): recovery complete
EXT3-fs (sde1): mounted filesystem with ordered data mode
CT: 22194: stopped
CT: 22194: started
NOHZ: local_softirq_pending 100
..............
..............
..............
CPU1: Core temperature/speed normal
CPU5: Core temperature/speed normal
CPU0: Package temperature/speed normal
CPU6: Package temperature/speed normal
CPU7: Package temperature/speed normal
CPU4: Package temperature/speed normal
CPU3: Package temperature/speed normal
CPU2: Package temperature/speed normal
CPU5: Package temperature/speed normal
CPU1: Package temperature/speed normal
[Hardware Error]: Machine check events logged
TCP: Peer 80.217.137.59:55993/80 unexpectedly shrunk window 3538440733:3538457701 (repaired)
TCP: Peer 80.217.137.59:55993/80 unexpectedly shrunk window 3538454781:3538460621 (repaired)
TCP: Peer 80.217.137.59:55993/80 unexpectedly shrunk window 3538506433:3538507341 (repaired)
TCP: Peer 80.217.137.59:55993/80 unexpectedly shrunk window 3538506433:3538507341 (repaired)
TCP: Peer 80.217.137.59:55993/80 unexpectedly shrunk window 3538506433:3538507341 (repaired)
TCP: Peer 80.217.137.59:55993/80 unexpectedly shrunk window 3538520481:3538521941 (repaired)
TCP: Peer 80.217.137.59:55993/80 unexpectedly shrunk window 3538572133:3538574501 (repaired)
TCP: Peer 80.217.137.59:55993/80 unexpectedly shrunk window 3538572133:3538574501 (repaired)
TCP: Peer 80.217.137.59:55993/80 unexpectedly shrunk window 3538572133:3538574501 (repaired)
EXT3-fs error (device dm-0): ext3_lookup: deleted inode referenced: 5638753
Aborting journal on device dm-0.
EXT3-fs (dm-0): error: ext3_journal_start_sb: Detected aborted journal
EXT3-fs (dm-0): error: remounting filesystem read-only
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
CT: 22566: stopped
CT: 22585: stopped
CT: 22555: stopped
NOHZ: local_softirq_pending 100
NOHZ: local_softirq_pending 100
CT: 22584: stopped
__journal_remove_journal_head: freeing b_committed_data

System hdd is /dev/sda (/dev/sda1 495M 35M 435M 8% /boot). Non raid array

Free diskspace /dev/mapper/pve-data 318G 304G 14G 96% /var/lib/vz
and more hdd working (sdb, sdc...).

 

[udo]

...
EXT3-fs error (device dm-0): ext3_lookup: deleted inode referenced: 5638753
Aborting journal on device dm-0.
EXT3-fs (dm-0): error: ext3_journal_start_sb: Detected aborted journal
EXT3-fs (dm-0): error: remounting filesystem read-only
...

Hi,
your disk has an problem, try to fix with fsck (boot an live-cd like grml).

Perhaps you should change the disk?

Udo