unable to start server: unable to create socket - PVE::APIDaemon: Address already in

C

charnov

Guest
One of my servers in the cluster is unreachable via web GUI but shows up in the cluster. Everything else is running great. It has been updated and rebooted.

On trying to restart pvedaemon, I get unable to start server: unable to create socket - PVE::APIDaemon: Address already in use
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

charnov said:
On trying to restart pvedaemon, I get unable to start server: unable to create socket - PVE::APIDaemon: Address already in use
Click to expand...

The daemon is already running? if so, try:

# /etc/init.d/pvedaemon stop
# /etc/init.d/pvedaemon start
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

On stop command I receive: Stopping PVE Daemon: pvedaemon failed!

On start command I receive: Starting PVE Daemon: pvedaemonunable to create socket - PVE::APIDaemon: Address already in use
(warning).
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

I tried re-installing pve-manager and that did not solve the issue. Here is the output for pveversion -v:

pve-manager: 2.1-13 (pve-manager/2.1/bdd3663d)
running kernel: 2.6.32-13-pve
proxmox-ve-2.6.32: 2.1-72
pve-kernel-2.6.32-13-pve: 2.6.32-72
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.92-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.8-1
pve-cluster: 1.0-27
qemu-server: 2.0-47
pve-firmware: 1.0-17
libpve-common-perl: 1.0-28
libpve-access-control: 1.0-24
libpve-storage-perl: 2.0-29
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.1-6
ksm-control-daemon: 1.1-1
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

pvedaemon[227615]: unable to start server: unable to create socket - PVE::APIDaemon: Address already in use
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

It's our big node so I cannot migrate everything off to reboot right now. When I can, I'll report back. I'm sure there is a perl process locking up the port somewhere where I haven't been able to find it.
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

What is the output of

# ps auxww|grep pvedaemon
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

root@virt3:~# ps auxww|grep pvedaemon
root 242472 0.0 0.1 180860 34944 ? Ss Aug07 0:00 pvedaemon worker
root 242474 0.0 0.1 191604 37932 ? S Aug07 0:00 pvedaemon
root 242475 0.0 0.1 191284 37652 ? S Aug07 0:00 pvedaemon worker
root 242476 0.0 0.1 191284 37652 ? S Aug07 0:00 pvedaemon worker
root 438516 0.0 0.0 7552 884 pts/0 S+ 00:09 0:00 grep pvedaemon
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

I'm having a similar issue currently with V1.9 where I can't login to web interface. After looking through logs and threads I've also tried to restart the pvedaemon and it won't stop:
Stopping PVE daemon: pvedaemon failed!

I've rebooted to no avail. All my KVM VM's are working fine but I do need to use the web interface.

hyper4:~# ps auxwww | grep pvedaemon
root 608394 0.0 0.0 95876 18320 ? D Jul27 0:11 pvedaemon worker
root 608481 0.0 0.0 89568 18180 ? D Jul27 0:17 pvedaemon worker
root 861228 0.0 0.0 3888 648 pts/3 S+ 21:35 0:00 grep pvedaemon

Not trying to hijack the thread... just adding my info in the hopes that it is similar and relevant and that if you find a fix for charnov that it will work for 1.9
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

What is the output of:

# pvesm status

Try to kill and restart the daemon:

# killall pvedaemon

or, if nothing helps

# killall -9 pvedaemon

The restart:

# /etc/init.d/pvedaemon start
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

Well... came in this morning and the server fixed itself. I am hoping it was a stalled or zombie process that finally died.
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

Spoke too soon. Now the other node is unreachable and is showing the same symptoms. I am seeing lots of pmxcfs[4177]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/virt3/Drive_Termsrv1: -1 errors in both nodes logs.
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

Hmmm.. on cluster FS restart:

pvedaemon[242475]: WARNING: ipcc_send_rec failed: Connection refused
Aug 8 09:41:10 virt3 pmxcfs[616234]: [status] notice: update cluster info (cluster name INDY-ARROW-CLS1, version = 4)
Aug 8 09:41:10 virt3 pmxcfs[616234]: [status] crit: cpg_initialize failed: 2
Aug 8 09:41:10 virt3 pmxcfs[616234]: [status] crit: can't initialize service
Aug 8 09:41:10 virt3 pmxcfs[616234]: [status] crit: cpg_send_message failed: 9
Aug 8 09:41:10 virt3 pmxcfs[616234]: [status] crit: cpg_send_message failed: 9
Aug 8 09:41:10 virt3 pmxcfs[616234]: [dcdb] notice: members: 1/616234, 2/3825
Aug 8 09:41:10 virt3 pmxcfs[616234]: [dcdb] notice: starting data syncronisation
Aug 8 09:41:10 virt3 pmxcfs[616234]: [dcdb] notice: received sync request (epoch 1/616234/00000001)
Aug 8 09:41:10 virt3 pvedaemon[242476]: <root@pam> end task UPID:virt3:0009671F:00DE5A7E:50226C74:srvrestart:pvecluster:root@pam: OK
Aug 8 09:41:10 virt3 pmxcfs[616234]: [status] crit: cpg_send_message failed: 9
Aug 8 09:41:10 virt3 pmxcfs[616234]: [status] crit: cpg_send_message failed: 9
Aug 8 09:41:10 virt3 pmxcfs[616234]: [dcdb] notice: received all states
Aug 8 09:41:10 virt3 pmxcfs[616234]: [dcdb] notice: leader is 1/616234
Aug 8 09:41:10 virt3 pmxcfs[616234]: [dcdb] notice: synced members: 1/616234, 2/3825
Aug 8 09:41:10 virt3 pmxcfs[616234]: [dcdb] notice: start sending inode updates
Aug 8 09:41:10 virt3 pmxcfs[616234]: [dcdb] notice: sent all (0) updates
Aug 8 09:41:10 virt3 pmxcfs[616234]: [dcdb] notice: all data is up to date
Aug 8 09:41:16 virt3 pmxcfs[616234]: [dcdb] notice: members: 1/616234, 2/3825
Aug 8 09:41:16 virt3 pmxcfs[616234]: [dcdb] notice: starting data syncronisation
Aug 8 09:41:16 virt3 pmxcfs[616234]: [dcdb] notice: received sync request (epoch 1/616234/00000001)
Aug 8 09:41:16 virt3 pmxcfs[616234]: [dcdb] notice: received all states
Aug 8 09:41:16 virt3 pmxcfs[616234]: [dcdb] notice: all data is up to date
Aug 8 09:41:17 virt3 pvestatd[204725]: WARNING: ipcc_send_rec failed: Transport endpoint is not connected
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

I'm seeing lots of this now:
Aug 8 10:05:37 virt2 pmxcfs[397356]: [status] crit: cpg_send_message failed: 9
Aug 8 10:05:38 virt2 corosync[398642]: [TOTEM ] Retransmit List: 6ba 6bb 6bc 6bd 6be 6bf 6c0 6c1 6c2 6c3
Aug 8 10:05:40 virt2 corosync[398642]: [TOTEM ] Retransmit List: 6c4 6c5 6c6 6c7 6c8 6c9 6ca 6cb 6cc 6cd
Aug 8 10:05:42 virt2 corosync[398642]: [TOTEM ] Retransmit List: 6ba 6bb 6bc 6bd 6be 6bf 6c0 6c1 6c2 6c3
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

charnov said:
Aug 8 10:05:38 virt2 corosync[398642]: [TOTEM ] Retransmit List: 6ba 6bb 6bc 6bd 6be 6bf 6c0 6c1 6c2 6c3
Aug 8 10:05:40 virt2 corosync[398642]: [TOTEM ] Retransmit List: 6c4 6c5 6c6 6c7 6c8 6c9 6ca 6cb 6cc 6cd
Aug 8 10:05:42 virt2 corosync[398642]: [TOTEM ] Retransmit List: 6ba 6bb 6bc 6bd 6be 6bf 6c0 6c1 6c2 6c3
Click to expand...

There is something wrong with cluster communication. Is there high load on the cluster network? Try

# /etc/init.d/pve-cluster stop
# /etc/init.d/pve-cluster start

Dos that change anything?
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

Do you use about the same hardware on all nodes? Or is there a slower node on the cluster?
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

dietmar said:
There is something wrong with cluster communication. Is there high load on the cluster network? Try

# /etc/init.d/pve-cluster stop
# /etc/init.d/pve-cluster start

Dos that change anything?
Click to expand...


Starting pve cluster filesystem : pve-cluster[main] notice: unable to aquire pmxcfs lock - trying again
[main] crit: unable to aquire pmxcfs lock: Resource temporarily unavailable
[main] notice: exit proxmox configuration filesystem (-1)
(warning).
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

Yes they are pretty close in specs. The last upgrade got a little messy on one node with it not booting with the right kernel. I fixed that and it WAS working for about 8 hours then ti started doing this. I can't get into either nodes web GUI, now. All the VMs are running great, though.
 
Re: unable to start server: unable to create socket - PVE::APIDaemon: Address already

charnov said:
Starting pve cluster filesystem : pve-cluster[main] notice: unable to aquire pmxcfs lock - trying again
Click to expand...

And the stop was successful?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back