No WEB gui, SSH but not pve command could be execurted

[afrugone]

Hi,
I've lost web gui, I've access to ssh, but any pve command I try to run never get an answer. All guest machines are working, but can not do anything else.
I'm working on pve version 4.1

Thanks for your help

 

[fireon]

Really no command? So when you say "qm list"?
Maybe an canceled updaten. Please say

apt install -f

Does this do something?

 

[afrugone]

Hi Thanks for your answer, I mean no proxmox command for example:

pveproxy --debug ---> wait forever no answer
pvecm status ---> wait forever no answer
pvecm nodes ---> wait forever no answer

apt install -f works

Now I run
pvedaemon stop
pvedaemon start

then:

/etc/init.d/pve-cluster start
[....] Starting pve-cluster (via systemctl): pve-cluster.serviceJob for pve-cluster.service failed. See 'systemctl status pve-cluster.service' and 'journalctl -xn' for details.

# pvecm status
ipcc_send_rec failed: Connection refused
ipcc_send_rec failed: Connection refused
ipcc_send_rec failed: Connection refused
pve configuration filesystem not mounted



Thanks and Regards
Alfredo

 

[afrugone]

In other machines of the cluster no command can be executed, I think is a problem with some nfs shares, how can I romove not used NFS shares?

 

[fireon]

When you can not deactivate the share, stop the nfs-kernelserver on your nfsharehost first. Then unmount it.

fusermount -uz /your/share

Don't use the TAB function, your terminal hangs by search over nfs.

 

[afrugone]

Can not stop NFS:
# service nfs-kernel-server stopReply
Failed to stop nfs-kernel-server.service: Unit nfs-kernel-server.service not loaded.

 

[fireon]

Can not stop NFS:
# service nfs-kernel-server stopReply
Failed to stop nfs-kernel-server.service: Unit nfs-kernel-server.service not loaded.

I don't know what system you have in use for your nfsserver.

 

[afrugone]

The NFS is not available, so I need to remove all references in Proxmox to this NFS server

 

[afrugone]

BIG PROBLEM - Now I lost connection to guest machines


# systemctl status pve-cluster.service
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled)
Active: failed (Result: signal) since Fri 2016-08-05 11:34:56 CLT; 5s ago
Process: 4702 ExecStart=/usr/bin/pmxcfs $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 4704 (code=killed, signal=KILL)

Aug 05 11:33:16 PVE007 pmxcfs[4704]: [status] notice: node has quorum
Aug 05 11:33:16 PVE007 pmxcfs[4704]: [dcdb] notice: members: 1/4704, 2/1391, 3/1390, 5/1419, 6/1383, 7/1348
Aug 05 11:33:16 PVE007 pmxcfs[4704]: [dcdb] notice: starting data syncronisation
Aug 05 11:33:16 PVE007 pmxcfs[4704]: [dcdb] notice: received sync request (epoch 1/4704/00000001)
Aug 05 11:33:16 PVE007 pmxcfs[4704]: [status] notice: members: 1/4704, 2/1391, 3/1390, 5/1419, 6/1383, 7/1348
Aug 05 11:33:16 PVE007 pmxcfs[4704]: [status] notice: starting data syncronisation
Aug 05 11:33:16 PVE007 pmxcfs[4704]: [status] notice: received sync request (epoch 1/4704/00000001)
Aug 05 11:33:16 PVE007 pvecm[4706]: ipcc_send_rec failed: Connection refused
Aug 05 11:33:16 PVE007 pvecm[4706]: ipcc_send_rec failed: Connection refused
Aug 05 11:33:16 PVE007 pvecm[4706]: ipcc_send_rec failed: Connection refused
Aug 05 11:34:46 PVE007 systemd[1]: pve-cluster.service start-post operation timed out. Stopping.
Aug 05 11:34:56 PVE007 systemd[1]: pve-cluster.service stop-sigterm timed out. Killing.
Aug 05 11:34:56 PVE007 systemd[1]: pve-cluster.service: main process exited, code=killed, status=9/KILL
Aug 05 11:34:56 PVE007 systemd[1]: Failed to start The Proxmox VE cluster filesystem.
Aug 05 11:34:56 PVE007 systemd[1]: Unit pve-cluster.service entered failed state.

 

[afrugone]

I try to modify the storage.cfg but I lost but is not possible.

# cd /etc/pve
-bash: cd: /etc/pve: Transport endpoint is not connected

 

[fireon]

This sounds not. Think there is really something broken. Can you boot with an livecd and check your hardware?

 

[afrugone]

The hardware is OK, all the problem starts when I try to connect to a new NFS storage that had problems, now that NFS storage is broken and I think that produce the problem with the Proxmox cluster.

 

[fireon]

The hardware is OK, all the problem starts when I try to connect to a new NFS storage that had problems, now that NFS storage is broken and I think that produce the problem with the Proxmox cluster.

This makes no sense.... hmm... boot with an livecd and remove the nfsstorage. Did this helping?

 

[afrugone]

The cluster is composed by 4 servers, I've to do that in all servers?

 

[fireon]

On all they are not running fine.

 

[afrugone]

Yes same problem with all servers. At least I could copy imagen files to other servers in a new cluster and have them running.

 

[afrugone]

Finally I've solved my problem, I can modify /etc/pve/storage.cfg file directly in local servers, I removed all references to difufunct NFSs and everything is working again, this modification must be done directly on server, can not be done using SSH.

Thanks for you help