One node of a two node cluster node not start VM's

[stan]

Just got back from vacation and one node of a two machine cluster:
- displays status but no CPU usage, server load, memory usage or network traffic graphs.
-the VM's are all "blacked out", are not running and have no labels.

I have no idea where to look. Ideas please on where to look. I can log in to the machine by ssh just fine.


Thanks for the help!

Stan

 

[mir]

What proxmox version?

Normally the following should do it on the node with problems:
sudo service pvedaemon restart
sudo service pvestatd restart

 

[stan]

What proxmox version?

Normally the following should do it on the node with problems:
sudo service pvedaemon restart
sudo service pvestatd restart

My Proxmox version is: 3.1 and current.

The above commands executed with no squawks but there was no change to the above problem. :-(

I CAN right click on a blacked out VM with no lable, click start and it will start up but with no change to the datacenter/node display.

Stan

 

[mir]

My Proxmox version is: 3.1 and current.

The above commands executed with no squawks but there was no change to the above problem. :-(

I CAN right click on a blacked out VM with no lable, click start and it will start up but with no change to the datacenter/node display.

Stan

Then try this sequence:
sudo service pvedaemon restart
sudo service pvestatd restart
sudo service pveproxy restart

 

[stan]

Mir,

No change after executing the above sequence. I'm stumped.

Stan

 

[stan]

Further info.

If I access the www interface of the second of the two nodes, the display of the first node shows black mini screen indicators and no names after the VM number. The exact opposite happens if I access the www interface of the first node.

Still baffeled

Stan

 

[blackpaw]

Do the .conf files match on both nodes?

Sent from my MT27i using Tapatalk

 

[stan]

Do the .conf files match on both nodes?

Sent from my MT27i using Tapatalk

I'll check; exactly where are they, please?

Another data point is:

I access the www interface of node one. When I click on node two, it says "Unable to get IP from node `nodetwo` -node offline? (500)"

Keep in mind everything was running flawlessly for months prior to yesterday.

Stan

 

[mir]

What is the contents of /etc/hosts on both nodes?
What does ifconfig -a show on both nodes?

 

[m.ardito]

if nothing else works, you could obviously migrate via CLI all vm/ct to other nodes, and reboot that one.

Marco

 

[stan]

if nothing else works, you could obviously migrate via CLI all vm/ct to other nodes, and reboot that one.

Marco

Marco,

I've rebooted both nodes without migration. No change. All the VM's autostart and run but are not displayed properly when logged in to www on the other node.

Stan

 

[stan]

Here are my /etc/hosts and ifconfig -a output:


root@proxmox17:/etc# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.10.11.17 proxmox17.gaboon.com proxmox17 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

root@proxmox18:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.10.11.18 proxmox18.gaboon.com proxmox18 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

root@proxmox17:/etc# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:1c:c4:a5:6c:d9
inet6 addr: fe80::21c:c4ff:fea5:6cd9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:17469186 errors:0 dropped:0 overruns:0 frame:0
TX packets:13756282 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9566120795 (8.9 GiB) TX bytes:14979035055 (13.9 GiB)
Interrupt:16

eth1 Link encap:Ethernet HWaddr 00:1c:c4:a5:6c:da
inet6 addr: fe80::21c:c4ff:fea5:6cda/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4172040 errors:0 dropped:0 overruns:0 frame:0
TX packets:2472954 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:6045721913 (5.6 GiB) TX bytes:197794410 (188.6 MiB)
Interrupt:17

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:834252 errors:0 dropped:0 overruns:0 frame:0
TX packets:834252 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:117926816 (112.4 MiB) TX bytes:117926816 (112.4 MiB)

tap100i0 Link encap:Ethernet HWaddr 9a:79:d8:f4:06:a3
inet6 addr: fe80::9879:d8ff:fef4:6a3/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:5931 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:468 (468.0 B) TX bytes:613825 (599.4 KiB)

tap100i1 Link encap:Ethernet HWaddr be:6b:6e:9e:6f:16
inet6 addr: fe80::bc6b:6eff:fe9e:6f16/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:60384 errors:0 dropped:0 overruns:0 frame:0
TX packets:109963 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:5694906 (5.4 MiB) TX bytes:81599160 (77.8 MiB)

tap110i0 Link encap:Ethernet HWaddr fe:f8:d0:a3:89:3b
inet6 addr: fe80::fcf8:d0ff:fea3:893b/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:3326299 errors:0 dropped:0 overruns:0 frame:0
TX packets:11490112 errors:0 dropped:0 overruns:232 carrier:0
collisions:0 txqueuelen:500
RX bytes:9432287594 (8.7 GiB) TX bytes:6420120777 (5.9 GiB)

tap110i1 Link encap:Ethernet HWaddr 5a:79:69:41:90:01
inet6 addr: fe80::5879:69ff:fe41:9001/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:2410433 errors:0 dropped:0 overruns:0 frame:0
TX packets:4096590 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:182025926 (173.5 MiB) TX bytes:5949469880 (5.5 GiB)

venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet6 addr: fe80::1/128 Scope:Link
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:3 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vmbr0 Link encap:Ethernet HWaddr 00:1c:c4:a5:6c:d9
inet addr:10.10.11.17 Bcast:10.10.11.255 Mask:255.255.255.0
inet6 addr: fe80::21c:c4ff:fea5:6cd9/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5984840 errors:0 dropped:0 overruns:0 frame:0
TX packets:2382705 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2992886323 (2.7 GiB) TX bytes:4962655360 (4.6 GiB)

vmbr1 Link encap:Ethernet HWaddr 00:1c:c4:a5:6c:da
inet addr:192.xxx.xxx.17 Bcast:192.150.121.255 Mask:255.255.255.0
inet6 addr: fe80::21c:c4ff:fea5:6cda/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:36280 errors:0 dropped:0 overruns:0 frame:0
TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1682549 (1.6 MiB) TX bytes:1880 (1.8 KiB)

root@proxmox18:/etc# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:1c:c4:74:e3:b0
inet6 addr: fe80::21c:c4ff:fe74:e3b0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12609089 errors:0 dropped:0 overruns:0 frame:0
TX packets:8894444 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:10485024393 (9.7 GiB) TX bytes:5794151203 (5.3 GiB)
Interrupt:16

eth1 Link encap:Ethernet HWaddr 00:1c:c4:74:e3:b1
inet6 addr: fe80::21c:c4ff:fe74:e3b1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1240458 errors:0 dropped:0 overruns:0 frame:0
TX packets:1562375 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:302974371 (288.9 MiB) TX bytes:1752463089 (1.6 GiB)
Interrupt:17

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2731821 errors:0 dropped:0 overruns:0 frame:0
TX packets:2731821 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:418818393 (399.4 MiB) TX bytes:418818393 (399.4 MiB)

tap107i0 Link encap:Ethernet HWaddr 72:f4:82:05:06:c2
inet6 addr: fe80::70f4:82ff:fe05:6c2/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:1731 errors:0 dropped:0 overruns:0 frame:0
TX packets:14088 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:587349 (573.5 KiB) TX bytes:1487064 (1.4 MiB)

tap107i1 Link encap:Ethernet HWaddr ee:3b:dc:33:c1:8a
inet6 addr: fe80::ec3b:dcff:fe33:c18a/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:466309 errors:0 dropped:0 overruns:0 frame:0
TX packets:935383 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:1673099407 (1.5 GiB) TX bytes:269099294 (256.6 MiB)

tap113i0 Link encap:Ethernet HWaddr 4e:c4:de:9b:97:50
inet6 addr: fe80::4cc4:deff:fe9b:9750/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:130165 errors:0 dropped:0 overruns:0 frame:0
TX packets:142070 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:33159818 (31.6 MiB) TX bytes:14952723 (14.2 MiB)

tap113i1 Link encap:Ethernet HWaddr a2:a1:2a:1f:09:72
inet6 addr: fe80::a0a1:2aff:fe1f:972/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:297805 errors:0 dropped:0 overruns:0 frame:0
TX packets:408157 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:28530426 (27.2 MiB) TX bytes:35064573 (33.4 MiB)

venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet6 addr: fe80::1/128 Scope:Link
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:3 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vmbr0 Link encap:Ethernet HWaddr 00:1c:c4:74:e3:b0
inet addr:10.10.11.18 Bcast:10.10.11.255 Mask:255.255.255.0
inet6 addr: fe80::21c:c4ff:fe74:e3b0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12476322 errors:0 dropped:0 overruns:0 frame:0
TX packets:6640797 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:10245872198 (9.5 GiB) TX bytes:5593625036 (5.2 GiB)

vmbr1 Link encap:Ethernet HWaddr 00:1c:c4:74:e3:b1
inet addr:192.xxx.xxx.18 Bcast:192.150.121.255 Mask:255.255.255.0
inet6 addr: fe80::21c:c4ff:fe74:e3b1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:105639 errors:0 dropped:0 overruns:0 frame:0
TX packets:91 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:4885539 (4.6 MiB) TX bytes:4194 (4.0 KiB)

 

[m.ardito]

Marco,

I've rebooted both nodes without migration. No change. All the VM's autostart and run but are not displayed properly when logged in to www on the other node.

Stan

ok. as last resort, migrate vm to other node properly working and reinstall failing node from scratch

Marco

 

[stan]

SOLVED!

Cycle power on a Netgrear GS748T switch. All is back to normal. No fault of Proxmox at all! Whew!

Thanks for all the support!

Stan