/etc/pve/nodes/****/openvz - frozen / not responding

[Chris Rivera]

all nodes on the cluster are not able to be managed at this point. Seems to be some issue with the pve mount.

While i can access /etc/pve.... i have issues once getting to the nodes/****/openvz directory

if i run an ls on this directory for any of my nodes.... it hangs.


I cannot provision, delete, or modify any vms on the cluster now.... not sure what the issue is.


#############

Troubleshooting:

I went to cluster.conf and see node 11 is id 1 so i rebooted node 11 thinking that for some reason there was issues reading the pve mount from the master node... but after reboot the problem exists.

All nodes are online ( green ) but we cannot make any adjustments to the cluster


#############

output of cluster.conf

<?xml version="1.0"?>
<cluster config_version="49" name="FL-Cluster">
<cman keyfile="/var/lib/pve-cluster/corosync.authkey"/>
<clusternodes>




<clusternode name="proxmox4" votes="1" nodeid="4"/>
<clusternode name="poxmox5" votes="1" nodeid="5"/>
<clusternode name="proxmox6" votes="1" nodeid="6"/>
<clusternode name="proxmox7" votes="1" nodeid="7"/>
<clusternode name="proxmox9" votes="1" nodeid="9"/>
<clusternode name="Proxmox10" votes="1" nodeid="10"/>
<clusternode name="proxmox8" votes="1" nodeid="8"/>


<clusternode name="proxmox11" votes="1" nodeid="1"/>
<clusternode name="proxmox2" votes="1" nodeid="2"/>


<clusternode name="proxmox3a" votes="1" nodeid="3"/></clusternodes>
<rm/>
</cluster>



output of mount:

root@proxmox2:~# mount
/dev/mapper/pve-root on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
beancounter on /proc/vz/beancounter type cgroup (rw,name=beancounter)
container on /proc/vz/container type cgroup (rw,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,name=fairsched)
none on /sys/kernel/config type configfs (rw)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,default_permissions,allow_other)

#############

nothing in syslogs

#############

root@proxmox2:~# cat /var/log/syslog | tail
Feb 28 13:15:00 proxmox2 pmxcfs[525189]: [status] notice: received log
Feb 28 13:15:00 proxmox2 pmxcfs[525189]: [status] notice: received log
Feb 28 13:17:01 proxmox2 /USR/SBIN/CRON[788634]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 28 13:18:56 proxmox2 pvedaemon[525098]: <root@pam> successful auth for user 'root@pam'
Feb 28 13:27:12 proxmox2 pmxcfs[525189]: [status] notice: received log
Feb 28 13:28:24 proxmox2 rrdcached[1761]: flushing old values
Feb 28 13:28:24 proxmox2 rrdcached[1761]: rotating journals
Feb 28 13:28:24 proxmox2 rrdcached[1761]: started new journal /var/lib/rrdcached/journal//rrd.journal.1362076104.073732
Feb 28 13:28:24 proxmox2 rrdcached[1761]: removing old journal /var/lib/rrdcached/journal//rrd.journal.1362068904.074127
Feb 28 13:29:21 proxmox2 pmxcfs[525189]: [status] notice: received log

 

[dietmar]

What is the output of

# pvecm status

 

[Chris Rivera]

This issue has since cleared itself up.

I was able to navigate to the directories and run the command you provided

current status of the cloud its completely out of sync... All nodes show cman running but all non local nodes are offline... cannot stop cman.... service fails when stopping.

#######

root@proxmox2:~# pvecm status
Version: 6.2.0
Config Version: 49
Cluster Name: FL-Cluster
Cluster Id: 6836
Cluster Member: Yes
Cluster Generation: 1082608
Membership state: Cluster-Member
Nodes: 1
Expected votes: 10
Total votes: 1
Node votes: 1
Quorum: 6 Activity blocked
Active subsystems: 3
Flags:
Ports Bound: 0
Node name: proxmox2
Node ID: 2
Multicast addresses: 239.192.26.206
Node addresses: *.*.*.*