Webinterface shows all systems in status unknown

[zahnfee]

Hi,
I'm a bit puzzled.
I have installed my system so that it is 99.9% complete.
The host boots from nvme1 and the vm/containers are on nvme2, which is mounted to /var

Then I decided to clone the system to identical hardware by

dd if=source-disk of=destination-disk

I edited the container config files to assign different IP-addresses and booted the system.
Initially there were complaints by the browser about identical certificate serial numbers, but that is solved.

I can see the containers, however their names are not shown as on the original system.

They are in status unknown in the web-interface. I can start them and ssh into them just fine
The cloned nameserver for example works just fine.

Is this a cosmetic issue and how can it be solved? I fail to see nay smoking guns in the logs...

kind regards

 

[bbgeek17]

The host boots from nvme1 and the vm/containers are on nvme2, which is mounted to /var

You overwritten a critical system directory /var with a mount to user-data disk?

There is a lot of information in /var/subfolders that is required by both the underlying Linux and PVE to function properly.
I'd recommend removing the custom mount. Examining the system properly for any breakage. And then point your container storage elsewhere, ie /mnt/containers

There is also a possibility that you have not relayed the situation clearly, or that I misunderstood.
You can provide command line outputs, encoded in CODE tags, for forum members to get a better visual of what your system looks like:

mount
df
cat /etc/fstab
journalctl -n 100
cat /etc/pve/storage.cfg
pvesm status
systemctl status pvestatd

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[zahnfee]

ok - what I omitted because i assumed it was clear is that I copied the entire /var to a separate disk, which also seems to be the default location for the container raw-disks (/var/lib/vz/images/101/vm-101-disk-0.raw ...). Don't worry, all files are there.
The handling of the container storage is still a mystery to me - none of my storage shows up anywhere in the user-interface, so I am forced to trick the system on the bare metal. I can live with it being at the current location.

 

[zahnfee]

This seems to be the culprit:

gatekeeper-1-(bookworm)-root# systemctl status pvestatd
× pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2024-02-27 16:08:38 UTC; 2h 54min ago
    Process: 1269 ExecStart=/usr/bin/pvestatd start (code=exited, status=111)
        CPU: 790ms

Feb 27 16:08:38 gatekeeper-1 pvestatd[1269]: ipcc_send_rec[1] failed: Connection refused
Feb 27 16:08:38 gatekeeper-1 pvestatd[1269]: ipcc_send_rec[2] failed: Connection refused
Feb 27 16:08:38 gatekeeper-1 pvestatd[1269]: ipcc_send_rec[3] failed: Connection refused
Feb 27 16:08:38 gatekeeper-1 pvestatd[1269]: ipcc_send_rec[1] failed: Connection refused
Feb 27 16:08:38 gatekeeper-1 pvestatd[1269]: ipcc_send_rec[2] failed: Connection refused
Feb 27 16:08:38 gatekeeper-1 pvestatd[1269]: Unable to load access control list: Connection refused
Feb 27 16:08:38 gatekeeper-1 pvestatd[1269]: ipcc_send_rec[3] failed: Connection refused
Feb 27 16:08:38 gatekeeper-1 systemd[1]: pvestatd.service: Control process exited, code=exited, status=111/n/a
Feb 27 16:08:38 gatekeeper-1 systemd[1]: pvestatd.service: Failed with result 'exit-code'.
Feb 27 16:08:38 gatekeeper-1 systemd[1]: Failed to start pvestatd.service - PVE Status Daemon.

After starting it the browser suddenly looks like the original system: with names and status

gatekeeper-1-(bookworm)-root# service pvestatd start
gatekeeper-1-(bookworm)-root# systemctl status pvestatd
● pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-02-27 19:04:16 UTC; 2s ago
    Process: 16553 ExecStart=/usr/bin/pvestatd start (code=exited, status=0/SUCCESS)
   Main PID: 16554 (pvestatd)
      Tasks: 1 (limit: 38292)
     Memory: 87.5M
        CPU: 694ms
     CGroup: /system.slice/pvestatd.service
             └─16554 pvestatd

Feb 27 19:04:16 gatekeeper-1 systemd[1]: Starting pvestatd.service - PVE Status Daemon...
Feb 27 19:04:16 gatekeeper-1 pvestatd[16554]: starting server
Feb 27 19:04:16 gatekeeper-1 systemd[1]: Started pvestatd.service - PVE Status Daemon.

Now the question is, why does it not start on the cloned system right away?
What refuses the connection?

 

[bbgeek17]

Omissions lead to assumptions, which lead to wrong advice or chasing a red herring.

Your system setup diverged from default installation sufficiently that it would require custom troubleshooting.

My suspicion is that you did not properly setup dependencies for your customizations, ie making sure that the mount of disk is a pre-requisite for practically all other services. Or I could be completely off base given lack of information.

Best of luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[zahnfee]

So the issue is decribed here: https://forum.proxmox.com/threads/unable-to-load-access-control-list-connection-refused.72245/
interestingly the /etc/hosts are identical (except of course for the IPs) on both systems.
/etc/hosts has an IP which is for a different network interface and network segment on both systems.

I had to manually correct the one on the clone machine.
Now on both systems hostname --ip-address presents the IP of the proxmox browser interface
Very confusing...

 

[zahnfee]

Thanks bbgeek17 for the pvestatd hint - I assume that this was the issue. I will shutdown for now and hope that the reboot tomorrow will show that all problems are gone.
Btw. the main system has the same setup: a mounted /var and has no issues.

 

[zahnfee]

Next morning, as expected: everything normal now