perl error 500 read timeout

[conrad]

After upgrading our live servers to 1.4 we get the following perl error
after adding a 3rd node to the cluster.
The server is responding well and i can connect with ssh and web fast.

But when i want to add a vm on this 3rd node after selecting the 3rd node during creation of the vm this error occurs:

[5411]ERR: 24: Error in Perl code: 500 read timeout

nothing in syslog and other node responds within a fraction of a second.

any idea?

 

[rugby]

I got a similar error:

http://www.proxmox.com/forum/showthread.php?t=2646

 

[dietmar]

What is the output of

# pveca -l

(on master and failing node)

 

[conrad]

master:
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---DISK
1 : xx.xxx.xxx.220 M A 08:35 0.03 42% 6%
2 : xx.xxx.xxx.223 N A 18:42 0.02 8% 1%
3 : xx.xxx.xxx.224 N A 21:18 0.00 3% 1%

failing node 3 node:

1 : xx.xxx.xxx.220 M A 08:38 0.41 42% 6%
2 : xx.xxx.xxx.223 N A 18:45 0.00 8% 1%
3 : xx.xxx.xxx.224 N A 21:21 0.00 3% 1%

This is from the syslog of the failing node:
Nov 23 09:57:50 pve03 pvemirror[18773]: cluster syncronization finished (10.28 seconds (files 0.00, config 0.11))
Nov 23 09:58:39 pve03 pvemirror[18773]: starting cluster syncronization
Nov 23 09:58:39 pve03 pvemirror[18773]: syncing master configuration from 'xxx.xxx.xxx.220'
Nov 23 09:58:49 pve03 pvemirror[18773]: syncing vzlist from 'xxx.xxx.xxx.xxx' failed: 500 read timeout
Nov 23 09:58:49 pve03 pvemirror[18773]: syncing templates
Nov 23 09:58:49 pve03 pvemirror[18773]: cluster syncronization finished (10.28 seconds (files 0.00, config 0.11))
Nov 23 09:59:39 pve03 pvemirror[18773]: starting cluster syncronization
Nov 23 09:59:39 pve03 pvemirror[18773]: syncing master configuration from 'xxx.xxx.xxx.220'
Nov 23 09:59:39 pve03 pvemirror[18773]: syncing templates
Nov 23 09:59:39 pve03 pvemirror[18773]: cluster syncronization finished (0.29 seconds (files 0.00, config 0.11))

 

[dietmar]

Did you import any OpenVZ VMs from another system?

 

[dietmar]

Nov 23 09:59:39 pve03 pvemirror[18773]: starting cluster syncronization
Nov 23 09:59:39 pve03 pvemirror[18773]: syncing master configuration from 'xxx.xxx.xxx.220'
Nov 23 09:59:39 pve03 pvemirror[18773]: syncing templates
Nov 23 09:59:39 pve03 pvemirror[18773]: cluster syncronization finished (0.29 seconds (files 0.00, config 0.11))

Obviously the sync is succussful now? You still get the error?

 

[conrad]

no, node3 is completely empty (no vm's or containers)

 

[conrad]

yes, only when i want to create a vm or container on this node.

it looks like it cannot get the vzlist of this node or something.

 

[dietmar]

What is the output of (run on the failing node):

# vzlist -a;echo $?

 

[conrad]

pve03:/var/lib# vzlist -a;echo $?
CTID NPROC STATUS IP_ADDR HOSTNAME
0

 

[conrad]

Does it makes sense if i disconnect the node from de master and then
later add the node to the cluster again.

Its becoming a bit urgent now, what else can i do to find out whats wrong?

 

[dietmar]

Does it makes sense if i disconnect the node from de master and then
later add the node to the cluster again.

You can try that, yes.

 

[conrad]

bad luck:

[Mon Nov 23 13:58:31 2009] [error] [5411]ERR: 24: Error in Perl code: 500 read timeout
[Mon Nov 23 14:00:52 2009] [error] [5411]ERR: 24: Error in Perl code: 500 read timeout
[Mon Nov 23 14:03:03 2009] [error] [5411]ERR: 24: Error in Perl code: 500 read timeout

cluster sync is fine, template sync works well, just selecting node 3 during
creation of a container give this error.

must the the 0.conf exists in /etc/vz/conf on all three nodes?

 

[dietmar]

Does the following command works without errors on that node:

# pvesm list -a

 

[conrad]

Dietmar, you're the best.

after checking pvesm list -a i noticed that i forgot to add the 3rd node in the firewall of nfs (backup) server.
Apparently when choosing a destionation for creating a new container it times out because of the failure to connect ot the nfs server.

Running smooth now!

did i mention you're the best?

no? ok you're the best Dietmar, thanks.

 

[dietmar]

Running smooth now!

Great ;-)

 

[netbone]

I have the same error and found:

iscsiadm: No active sessions.
Logging in to [iface: default, target: storage.domain.com:disk1, portal: 194.0.yy.xx,3260]
iscsiadm: Could not login to [iface: default, target: storage.domain.com:disk1, portal: 194.0.xx.xx,3260]:
iscsiadm: initiator reported error (5 - encountered iSCSI login failure)
command '/usr/bin/iscsiadm --mode node --portal 194.0.xx.xx:3260 --targetname storage.domain.com:disk1 --login' failed with exit code 255


How to remove storage from cluster without webpanel?

 

[tom]

...

How to remove storage from cluster without webpanel?

remove it in /etc/pve/storage.cfg (master)

 

[netbone]

worked - thank you - perfect.