[SOLVED] 5 Brand new Proxmox installs that are all broken

O

ov3rwatch

New Member
Apr 7, 2024
13
2
3
Chicago
stingernetworks.us
So I have 5 Dell Poweredge R415s. I installed Proxmox VE 8.1-2 from the official download. it installs, I go to the IP addresses for each of these servers and there's nothing there, I can ping the devices so they're up but the pinwheel just spins then times out.

Connecting to any of the nodes via SSH, I can confirm Network is set up right but when checking the status of pveproxy I see that it is not working properly which is pretty impressive given they're fresh installs, normally i have to go and break something before they end up like this lol. I tried reinstalling with a fresh copy of the ISO on different drives thinking maybe i had a bad iso download but the nodes come up with the same issue. the result of systemctl status pveproxy shows the following, post install, and I think this is what's causing the issue but I'm honestly not sure the best way to resolve this error:

code_language.shell: 
pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service: enabled: preset: enabled)
Active: active (running)
Process: 2161 ExecStartPre=/usr/bin/pvecm updatecerts --client (code=exited status=111)
Process: 2164 ExecStart= /usr/bin/pveproxy start(code=exited status=0/SUCCESS)
MainPID: 2167 (pveproxy)

pvecm(2161): Unable to load access control list: CONNECTION REFUSED
pveproxy(2167): Starting server
pveproxy(2167): Starting 3 workers
pveproxy(2167): Worker 1268 Started.
pveproxy(2167): Worker 1269 Started.
pveproxy(2167): Worker 1270 Started.
systemd(1): Started pveproxy.service - PVE API Proxy Server
pveproxy(2168) /etc/pve/local/pve-ssl.key failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm at line 2009
pveproxy(2169) /etc/pve/local/pve-ssl.key failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm at line 2009
pveproxy(2170) /etc/pve/local/pve-ssl.key failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm at line 2009

I've done absolutely nothing to the systems. This is first boot straight off the installation media. Why is this happening and is this going to be a massive pain to fix it?

***Edited for clarity***
 
Last edited:
Pveproxy.service not being able to load pve-ssl.key usually means your PMXCFS isn't mounted, therefore /etc/pve is empty. Probably the pve-cluster.service failed. What is systemctl status pve-cluster.service reporting?
 
ov3rwatch said:
So I have 5 Dell Poweredge R415s. I installed Proxmox VE 8.1-2 from the official download. it installs, I go to the IP addresses for each of these servers and there's nothing there,
Click to expand...
Do you mean you've tried to access the GUI via https?
ov3rwatch said:
I can ping the devices so there up
Click to expand...
Can you list the IPs? Can you ssh into them? Can you provide "ip a" from each host? Use text with CODE tags.
ov3rwatch said:
ot sure the best way to resolve this error:
Click to expand...
If you have not created a cluster yet, then concentrate on single server. Provide the following information:
ip a
cat /etc/network/interfaces
hostname
cat /etc/hostname
cat /etc/hosts
curl -sk https://localhost:8006|grep -i title
systemctl status pveproxy
journalctl -n 100


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
Do you mean you've tried to access the GUI via https?
Click to expand...
Yes, HTTP doesn't work either on any of the nodes.

Can you list the IPs? Can you ssh into them? Can you provide "ip a" from each host? Use text with CODE tags.
Click to expand...
Yes as i said the IP configuration is correct. I can ping and ssh into them that is how I'm connected to the CLI. There is no cluster yet so like you said I am just going to focus on one unit since i can fix the rest after getting the first one sorted.

This node should be 172.31.87.13 which it is.

output of ip a:
Code: 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 08:9e:01:c7:99:1b brd ff:ff:ff:ff:ff:ff
    altname enp2s0f0
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 08:9e:01:c7:99:1c brd ff:ff:ff:ff:ff:ff
    altname enp2s0f1
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 08:9e:01:c7:99:1b brd ff:ff:ff:ff:ff:ff
    inet 172.31.87.13/23 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::a9e:1ff:fec7:991b/64 scope link
       valid_lft forever preferred_lft forever

/etc/network/interfaces
Code: 
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 172.31.87.13/23
        gateway 172.31.87.254
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

iface eno2 inet manual


source /etc/network/interfaces.d/*

Result of hostname and /etc/hostname:
Code: 
hostname
HYPERVISOR-03oakhills
cat /etc/hostname
HYPERVISOR-03@oakhills

curl -sk https://localhost:8006|grep -i title gives no output this just hangs forever until you cancel it.

service status of pveproxy:
Code: 
pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-04-11 12:16:46 CDT; 7h ago
    Process: 1261 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=111)
    Process: 1264 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 1267 (pveproxy)
      Tasks: 4 (limit: 38374)
     Memory: 147.3M
        CPU: 24min 34.403s
     CGroup: /system.slice/pveproxy.service
             ├─ 1267 pveproxy
             ├─19668 "pveproxy worker"
             ├─19669 "pveproxy worker"
             └─19670 "pveproxy worker"

Apr 11 19:46:35 HYPERVISOR-03oakhills pveproxy[19666]: worker exit
Apr 11 19:46:35 HYPERVISOR-03oakhills pveproxy[1267]: worker 19666 finished
Apr 11 19:46:35 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:46:35 HYPERVISOR-03oakhills pveproxy[1267]: worker 19669 started
Apr 11 19:46:35 HYPERVISOR-03oakhills pveproxy[19669]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Apr 11 19:46:35 HYPERVISOR-03oakhills pveproxy[19667]: worker exit
Apr 11 19:46:35 HYPERVISOR-03oakhills pveproxy[1267]: worker 19667 finished
Apr 11 19:46:35 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:46:35 HYPERVISOR-03oakhills pveproxy[1267]: worker 19670 started
Apr 11 19:46:35 HYPERVISOR-03oakhills pveproxy[19670]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.


JournalCTL output:
Code: 
root@HYPERVISOR-03oakhills:~# journalctl -n 100
Apr 11 19:48:06 HYPERVISOR-03oakhills pveproxy[19727]: worker exit
Apr 11 19:48:06 HYPERVISOR-03oakhills pveproxy[1267]: worker 19727 finished
Apr 11 19:48:06 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:48:06 HYPERVISOR-03oakhills pveproxy[1267]: worker 19730 started
Apr 11 19:48:06 HYPERVISOR-03oakhills pveproxy[19730]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Apr 11 19:48:06 HYPERVISOR-03oakhills pveproxy[19728]: worker exit
Apr 11 19:48:06 HYPERVISOR-03oakhills pveproxy[1267]: worker 19728 finished
Apr 11 19:48:06 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:48:06 HYPERVISOR-03oakhills pveproxy[1267]: worker 19731 started
Apr 11 19:48:07 HYPERVISOR-03oakhills pveproxy[19731]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Apr 11 19:48:11 HYPERVISOR-03oakhills pveproxy[19729]: worker exit
Apr 11 19:48:11 HYPERVISOR-03oakhills pveproxy[1267]: worker 19729 finished
Apr 11 19:48:11 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:48:11 HYPERVISOR-03oakhills pveproxy[1267]: worker 19732 started
Apr 11 19:48:11 HYPERVISOR-03oakhills pveproxy[19732]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Apr 11 19:48:11 HYPERVISOR-03oakhills pveproxy[19730]: worker exit
Apr 11 19:48:11 HYPERVISOR-03oakhills pveproxy[1267]: worker 19730 finished
Apr 11 19:48:11 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:48:11 HYPERVISOR-03oakhills pveproxy[1267]: worker 19733 started
Apr 11 19:48:12 HYPERVISOR-03oakhills pveproxy[19733]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Apr 11 19:48:12 HYPERVISOR-03oakhills pveproxy[19731]: worker exit
Apr 11 19:48:12 HYPERVISOR-03oakhills pveproxy[1267]: worker 19731 finished
Apr 11 19:48:12 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:48:12 HYPERVISOR-03oakhills pveproxy[1267]: worker 19734 started
Apr 11 19:48:12 HYPERVISOR-03oakhills pveproxy[19734]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Apr 11 19:48:16 HYPERVISOR-03oakhills pveproxy[19732]: worker exit
Apr 11 19:48:16 HYPERVISOR-03oakhills pveproxy[1267]: worker 19732 finished
Apr 11 19:48:16 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:48:16 HYPERVISOR-03oakhills pveproxy[1267]: worker 19735 started
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[19735]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[19733]: worker exit
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[1267]: worker 19733 finished
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[1267]: worker 19736 started
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[19736]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[19734]: worker exit
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[1267]: worker 19734 finished
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[1267]: worker 19737 started
Apr 11 19:48:17 HYPERVISOR-03oakhills pveproxy[19737]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Apr 11 19:48:22 HYPERVISOR-03oakhills pveproxy[19735]: worker exit
Apr 11 19:48:22 HYPERVISOR-03oakhills pveproxy[1267]: worker 19735 finished
Apr 11 19:48:22 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:48:22 HYPERVISOR-03oakhills pveproxy[1267]: worker 19743 started
Apr 11 19:48:22 HYPERVISOR-03oakhills pveproxy[19743]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Apr 11 19:48:22 HYPERVISOR-03oakhills pveproxy[19736]: worker exit
Apr 11 19:48:22 HYPERVISOR-03oakhills pveproxy[1267]: worker 19736 finished
Apr 11 19:48:22 HYPERVISOR-03oakhills pveproxy[1267]: starting 1 worker(s)
Apr 11 19:48:22 HYPERVISOR-03oakhills pveproxy[1267]: worker 19744 started
Apr 11 19:48:22 HYPERVISOR-03oakhills pveproxy[19744]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
 
Dunuin said:
Pveproxy.service not being able to load pve-ssl.key usually means your PMXCFS isn't mounted, therefore /etc/pve is empty. Probably the pve-cluster.service failed. What is systemctl status pve-cluster.service reporting?
Click to expand...

Code: 
# systemctl status pve-cluster.service
× pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-04-11 12:16:45 CDT; 7h ago
    Process: 1266 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
        CPU: 17ms

Apr 11 12:16:45 HYPERVISOR-03oakhills systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Apr 11 12:16:45 HYPERVISOR-03oakhills systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Apr 11 12:16:45 HYPERVISOR-03oakhills systemd[1]: pve-cluster.service: Start request repeated too quickly.
Apr 11 12:16:45 HYPERVISOR-03oakhills systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Apr 11 12:16:45 HYPERVISOR-03oakhills systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.

The output is basically the same on all the nodes. should I attempt to restart the service?
 
dietmar said:
Your hostname looks strange - A hostname must not contain an "@" character. Hostname is the part before the domain...
Click to expand...
Good catch! Also look at the inconsistency:
Code: 
hostname
HYPERVISOR-03oakhills
cat /etc/hostname
HYPERVISOR-03@oakhills
The user does not provide output from:
Code: 
cat /etc/hosts
as bbgeek17 requested, but my guess is it would also prove interesting.
 
  • Like
Reactions: ov3rwatch
I think you may have confused hostname syntax with email address syntax. So the correct hostname (FQDN) would be "HYPERVISOR-03.oakhills" instead of "HYPERVISOR-03@oakhills". Please try to reinstall.
 
  • Like
Reactions: ov3rwatch
The problem is definitely the hostname. Had a poster on r/proxmox that was unable to connect to the web UI and it came down to a hostname problem. As noted, no @ allowed in hostnames.
 
  • Like
Reactions: ov3rwatch
The value of sleep! Wake up and everything is solved :-) I agree with everything that's been posted here in last few hours - definitely a hostname issue.
@dietmar - perhaps it makes sense to add a sanity check into pveproxy startup code as well. Bad hostname is a common occurrence in the forum, it'd be useful to error out with a more detailed message.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: ov3rwatch
dietmar said:
I think you may have confused hostname syntax with email address syntax. So the correct hostname (FQDN) would be "HYPERVISOR-03.oakhills" instead of "HYPERVISOR-03@oakhills". Please try to reinstall.
Click to expand...
So I didn't create the hostname like this.

The installer configured the hostname like this on all hypervisors. It asked for a hostname; I entered HYPERVISOR-XX. It asked for a domain; I entered oakhills.local (matches my DC as I'll be syncing users with LDAP) and hit next. I've just done a fresh reinstall on this device, and it created the exact same hostname when the fields were filled out the same way. I can go through and do it again and take a picture of the install screen as configured if you'd like, but if the hostname is the issue here, then there is an installer bug because I did not put the @ symbol in anywhere.

gfngfn256 said:
The user does not provide output from:
Code: 
cat /etc/hosts
as bbgeek17 requested, but my guess is it would also prove interesting.
Click to expand...
yep i typo'd that good catch. here is that output on the freshly installed system as i can no longer access the original SSH shell sincei ve done a reinstall:
Code: 
127.0.0.1 localhost.localdomain localhost
172.31.87.13 HYPERVISOR-03@oakhills.local HYPERVISOR-03@oakhills

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

here is the output from another node that i didn't do a reinstall on: HYPERVISOR-04 was the hostname entered at install

Code: 
127.0.0.1 localhost.localdomain localhost
172.31.87.14 HYPERVISOR-04@oakhills.local HYPERVISOR-04@oakhills

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
 
Spot on. Changing the hostname to HV3 and restarting pveproxy, I now have a web GUI. I'll make this change on all the devices to confirm, but that looks to have solved it. To reiterate though, I did not put an @ into any hostnames at install time. I'm not sure where the illegal character came from.
 
ov3rwatch said:
The installer configured the hostname like this on all hypervisors. It asked for a hostname; I entered HYPERVISOR-XX. It asked for a domain; I entered oakhills.local (matches my DC as I'll be syncing users with LDAP) and hit next. I've just done a fresh reinstall on this device, and it created the exact same hostname when the fields were filled out the same way. I can go through and do it again and take a picture of the install screen as configured if you'd like, but if the hostname is the issue here, then there is an installer bug because I did not put the @ symbol in anywhere.
Click to expand...
I would actually like to see those pictures for my sanity. Nowhere in the GUI/TUI install I am being asked to put domain separately from hostname. I only have single FQDN field.
Is this in a hosted environment with custom installer by any chance?


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
I would actually like to see those pictures for my sanity. Nowhere in the GUI/TUI install I am being asked to put domain separately from hostname. I only have single FQDN field.
Is this in a hosted environment with custom installer by any chance?


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
No, local cluster. I'll go ahead and do a reinstall now. it actually looks like things still aren't working properly anyways inside the pve proxy interface the dialog boxes wont close and behind the dialog is an error that it cant resolve the hostname HV3 so it may still be a larger problem.
 
bbgeek17 said:
I would actually like to see those pictures for my sanity. Nowhere in the GUI/TUI install I am being asked to put domain separately from hostname. I only have single FQDN field.
Click to expand...
Well looks like I am just an idiot, as you're right. I distinctly remember this morning and during initial installs, there being 2 different fields one for hostname one for domain (though i wasn't really paying specific attention to it at the time) and testing the installer right now, I only see one field, FQDN... Evidently im the one losing my sanity. Taking over IT from a previous admin that let the hardware infrustructure litterally break down and fall apart, I've been burning the candle at both ends and I must just be losing it LOL.


bbgeek17 said:
The value of sleep! Wake up and everything is solved :) I agree with everything that's been posted here in last few hours - definitely a hostname issue.
@dietmar - perhaps it makes sense to add a sanity check into pveproxy startup code as well. Bad hostname is a common occurrence in the forum, it'd be useful to error out with a more detailed message.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
In any case definintly the bug report for allowing illegal characters in the fqdn feild in the installer is a good idea to stop sleep deprived admins from adding @s to their FQDNs :p
 
Last edited:
  • Like
Reactions: gfngfn256 and bbgeek17
Reminds me of the time I came to work and went into the lab to see that someone had stacked up about 6 adapters to connect a radio to a signal generator. After studying the setup for a few minutes, I removed all but two of the adapters and it worked fine with much less loss and strain on the connectors.

The previous guy had been working until 1am. I guess it all made sense at the time :).
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom