Continuing issues with pveproxy and pve-ssl.key

MikeC

Active Member
Jan 11, 2016
67
0
31
Bay Area, California
Hello, all.

I have two proxmox nodes (4.2-2 and 4.1-15) running on Debian 8 where access to the node's web site constantly stops working. Only after I restart pveproxy can I access the site again. Along with this behavior is constant errors in syslog:

Oct 19 13:15:13 proxmox5 pveproxy[18965]: worker exit
Oct 19 13:15:13 proxmox5 pveproxy[16128]: worker 18965 finished
Oct 19 13:15:13 proxmox5 pveproxy[16128]: starting 1 worker(s)
Oct 19 13:15:13 proxmox5 pveproxy[16128]: worker 18996 started
Oct 19 13:15:13 proxmox5 pveproxy[18996]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/HTTPServer.pm line 1639.
Oct 19 13:15:15 proxmox5 pveproxy[18993]: worker exit
Oct 19 13:15:15 proxmox5 pveproxy[16128]: worker 18993 finished
Oct 19 13:15:15 proxmox5 pveproxy[16128]: starting 1 worker(s)
Oct 19 13:15:15 proxmox5 pveproxy[16128]: worker 18997 started
Oct 19 13:15:15 proxmox5 pveproxy[18997]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/HTTPServer.pm line 1639.

over and over again.

The files exist...

-rw-r----- 1 root www-data 1679 Aug 31 12:23 pve-ssl.key
-rw-r----- 1 root www-data 1712 Aug 31 12:23 pve-ssl.pem

They're valid...

Issuer: CN=Proxmox Virtual Environment, OU=49c41bb2d7d4c96da43cf0ddf6e9b240, O=PVE Cluster Manager CA
Validity
Not Before: Aug 30 17:23:28 2016 GMT
Not After : Aug 28 17:23:28 2026 GMT

But the same errors keep coming up over and over and over again. My daily syslog file has 14,961 occurences so far. Does anyone have any idea why this continues to be a problem? Is there an open bug? Should I create one? None of my 3.x proxmox nodes have this problem. I've re-generated certificates a number of times on each machine, to no avail.

Thanks.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,357
1,657
174
are you sure the key is valid (as in, not corrupted somehow)? you only posted the output for the certificate. anyway, you can simply delete both files and generate new certificates with "pvecm updatecerts" (on the node in question)
 

MikeC

Active Member
Jan 11, 2016
67
0
31
Bay Area, California
Hi Fabian,
Thanks for the reply.
No, it's good. I've compared the modulus between the key and cert using openssl and they match.
I've also regenerated them at least four times so far with updatecerts.
It's something else.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,357
1,657
174
please post the output of "pveversion -v", and the content of "/etc/default/pveproxy"
 

MikeC

Active Member
Jan 11, 2016
67
0
31
Bay Area, California
Same continuing issue here, which I've seen now for almost a year:

Feb 16 18:14:49 util3 pveproxy[23421]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1618.

I have to stop pveproxy, delete the keys, re-issue them with pvecm, then start pveproxy before it'll work. The keys look fine using openssl to check their modulus:

Code:
root@util3:/etc/pve/local# openssl x509 -in pve-ssl.pem -noout | openssl md5
(stdin)= d41d8cd98f00b204e9800998ecf8427e
root@util3:/etc/pve/local# openssl rsa -in pve-ssl.key -noout | openssl md5
(stdin)= d41d8cd98f00b204e9800998ecf8427e

Here's my version info:

Code:
root@util3:/home/pilotmc# pveversion -v
proxmox-ve: 4.4-79 (running kernel: 4.4.35-2-pve)
pve-manager: 4.4-12 (running version: 4.4-12/e71b7a74)
pve-kernel-4.4.35-2-pve: 4.4.35-79
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-108
pve-firmware: 1.1-10
libpve-common-perl: 4.0-91
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-73
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.4-3
pve-qemu-kvm: 2.7.1-1
pve-container: 1.0-93
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-1
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80
 
Last edited:

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,357
1,657
174
Same continuing issue here, which I've seen now for almost a year:

Feb 16 18:14:49 util3 pveproxy[23421]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1618.

I have to stop pveproxy, delete the keys, re-issue them with pvecm, then start pveproxy before it'll work. The keys look fine using openssl to check their modulus:

Code:
root@util3:/etc/pve/local# openssl x509 -in pve-ssl.pem -noout | openssl md5
(stdin)= d41d8cd98f00b204e9800998ecf8427e
root@util3:/etc/pve/local# openssl rsa -in pve-ssl.key -noout | openssl md5
(stdin)= d41d8cd98f00b204e9800998ecf8427e

not sure what that is supposed to tell us, since you are just md5-ing an empty string:
Code:
echo -n "" | openssl md5
(stdin)= d41d8cd98f00b204e9800998ecf8427e

what you would need to do is check the following matches:
Code:
# openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -modulus
Modulus=A9D8EB8303AFD642C802F06D39BD7979422C6176107FD89F137F9F7B7FED5C32D88151810629D96FFC9896C9CF6B41FB244EBDD47FDA450930FA148EF427B5AD83CEEEC1664235C34FC1451CBEC94C9F7A48432E4EE20C82F4D721A65369219027500CBBCA7787914A1D2B920B3C210B49DC78995AA61602484171C8E180E3AFD1AD1A7F00483E1C2AC2D977978C4250F7632889BB2A0397B8D3ACC394C8D41CC1AC5AD8DA3AB4A4BC6866B17641AE621A5601364B84B96A869A1EA4E3EECD536F6665EA0CEF26F128DFBE81DB5D075D753E2F65143D4DEBEF794F65F2FBA7FE7802D4E365B056AE1056F9CCC1F5D3CA8F1BDC4BD394EC0CC649D2EC0424DF49
# openssl rsa -in /etc/pve/local/pve-ssl.key -noout -modulus
Modulus=A9D8EB8303AFD642C802F06D39BD7979422C6176107FD89F137F9F7B7FED5C32D88151810629D96FFC9896C9CF6B41FB244EBDD47FDA450930FA148EF427B5AD83CEEEC1664235C34FC1451CBEC94C9F7A48432E4EE20C82F4D721A65369219027500CBBCA7787914A1D2B920B3C210B49DC78995AA61602484171C8E180E3AFD1AD1A7F00483E1C2AC2D977978C4250F7632889BB2A0397B8D3ACC394C8D41CC1AC5AD8DA3AB4A4BC6866B17641AE621A5601364B84B96A869A1EA4E3EECD536F6665EA0CEF26F128DFBE81DB5D075D753E2F65143D4DEBEF794F65F2FBA7FE7802D4E365B056AE1056F9CCC1F5D3CA8F1BDC4BD394EC0CC649D2EC0424DF49

if the issue occurs again, please verify that the above gives identical output for both files. the complete log of pveproxy ("journalctl -u pveproxy") might also be helpful. I've only ever seen that error message when the file was actually not readable because the pmxcfs was not mounted - but in that case regenerating would also not work (unless the problem was temporary and resolved itself in the meantime).
 

MikeC

Active Member
Jan 11, 2016
67
0
31
Bay Area, California
THanks, Fabian. I see what I did. Okay, the modulus for both still match, when using the proper commands as you've shown.
This is what journalctl provides, which it appears also gets sent to syslog as that's how I found it:

Code:
Feb 15 19:47:23 util3 pveproxy[22681]: worker 18658 started
Feb 15 19:47:23 util3 pveproxy[18658]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) 
Feb 15 19:47:25 util3 pveproxy[18656]: worker exit
Feb 15 19:47:25 util3 pveproxy[18657]: worker exit
Feb 15 19:47:25 util3 pveproxy[22681]: worker 18656 finished
Feb 15 19:47:25 util3 pveproxy[22681]: starting 1 worker(s)
Feb 15 19:47:25 util3 pveproxy[22681]: worker 18659 started
Feb 15 19:47:25 util3 pveproxy[22681]: worker 18657 finished
Feb 15 19:47:25 util3 pveproxy[22681]: starting 1 worker(s)
Feb 15 19:47:25 util3 pveproxy[22681]: worker 18660 started
Feb 15 19:47:25 util3 pveproxy[18659]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) 
Feb 15 19:47:25 util3 pveproxy[18660]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) 
Feb 15 19:47:28 util3 pveproxy[18658]: worker exit
Feb 15 19:47:28 util3 pveproxy[22681]: worker 18658 finished
Feb 15 19:47:28 util3 pveproxy[22681]: starting 1 worker(s)
Feb 15 19:47:28 util3 pveproxy[22681]: worker 18662 started
Feb 15 19:47:28 util3 pveproxy[18662]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) 
Feb 15 19:47:30 util3 pveproxy[18659]: worker exit
Feb 15 19:47:30 util3 pveproxy[18660]: worker exit
Feb 15 19:47:30 util3 pveproxy[22681]: worker 18659 finished
Feb 15 19:47:30 util3 pveproxy[22681]: starting 1 worker(s)
Feb 15 19:47:30 util3 pveproxy[22681]: worker 18668 started
Feb 15 19:47:30 util3 pveproxy[22681]: worker 18660 finished
Feb 15 19:47:30 util3 pveproxy[22681]: starting 1 worker(s)
Feb 15 19:47:30 util3 pveproxy[22681]: worker 18669 started
Feb 15 19:47:30 util3 pveproxy[18668]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) 
Feb 15 19:47:30 util3 pveproxy[18669]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) 
Feb 15 19:47:33 util3 pveproxy[18662]: worker exit
Feb 15 19:47:33 util3 pveproxy[22681]: worker 18662 finished
Feb 15 19:47:33 util3 pveproxy[22681]: starting 1 worker(s)
Feb 15 19:47:33 util3 pveproxy[22681]: worker 18670 started

How can I check if the pmxcfs part is working properly?
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,357
1,657
174
if you can access /etc/pve, then the pmxcfs is at least readable (which is enough for pveproxy to start).

you could verify that the private key file only contains exactly one key, it should look like this:
Code:
-----BEGIN RSA PRIVATE KEY-----
RANDOM LOOKING STUFF
-----END RSA PRIVATE KEY-----

you wrote in your first post that you have regenerated the certificate "to no avail", but your first post also shows that your key and certificate have a timestamp from august.. this does not match ;) I suggest removing both pve-ssl.key and pve-ssl.pem from /etc/pve/local/ on the node which has this issue, and then running "pvecm updatecerts" again. both files should then be regenerated and have a current timestamp!
 

MikeC

Active Member
Jan 11, 2016
67
0
31
Bay Area, California
Thanks, Fabian.

Weird. I think that August file was from another server. I've since gone through and updated everything on my current problem server in the hopes that a newer release helps. Alas, it did not.

Code:
proxmox-ve: 4.4-79 (running kernel: 4.4.35-2-pve)
pve-manager: 4.4-12 (running version: 4.4-12/e71b7a74)
pve-kernel-4.4.35-2-pve: 4.4.35-79
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-48
qemu-server: 4.0-108
pve-firmware: 1.1-10
libpve-common-perl: 4.0-91
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-73
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-docs: 4.4-3
pve-qemu-kvm: 2.7.1-1
pve-container: 1.0-93
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-1
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
smartmontools: 6.5+svn4324-1~pve80

1) /etc/pve is accessible, so not pmxcfs
2) pve-ssl.key is a valid key according to openssl, and yes it contains the requisite open/close stanzas. So not format of key file.
3) My daily grind now is: service pveproxy stop; rm /etc/pve/local/pve-ssl.*; pvecm updatecerts; service pveproxy start.

I shouldn't have to do that, right? It works for a while, but several hours later it's already broken again with more of the same SSL errors.
Anything else you can think of? I'm seeing this on both my 4.2 and 4.4 instances. :/ The OS is Debian 8.7

I did see a different message today in syslog, but still same outcome: can't get the UI to load until I execute the actions in step 3 above.
Feb 20 11:39:16 util3 pveproxy[21731]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1618:

-> if ($self->{ssl}) {
$self->{tls_ctx} = AnyEvent::TLS->new(%{$self->{ssl}});
# TODO : openssl >= 1.0.2 supports SSL_CTX_set_ecdh_auto to select a curve depending on
# server and client availability from SSL_CTX_set1_curves.
# that way other curves like 25519 can be used.
# openssl 1.0.1 can only support 1 curve at a time.
 

MikeC

Active Member
Jan 11, 2016
67
0
31
Bay Area, California
Any other ideas out there? Several of my proxmox 4 installs are exhibiting this behavior, and now on some I can't get the UI to load at all, even going through the pveproxy stop/start, pvecm updatecerts, etc.
 

nexusguy59

Active Member
Jan 6, 2014
80
14
33
Ohio
www.rpihobby.us
I am seeing the issue as well, it doesn't make sense that only one node is afflicted. Can someone please give me some direction.

root@cfPVE2:~# /etc/pve/local# openssl x509 -in pve-ssl.pem -noout | openssl md5
-bash: /etc/pve/local#: No such file or directory
(stdin)= d41d8cd98f00b204e9800998ecf8427e
root@cfPVE2:~# /etc/pve/local# openssl rsa -in pve-ssl.key -noout | openssl md5
-bash: /etc/pve/local#: No such file or directory
(stdin)= d41d8cd98f00b204e9800998ecf8427e
root@cfPVE2:~#

Here is the current version

root@cfPVE2:~# pveversion -v
proxmox-ve: 5.1-26 (running kernel: 4.13.4-1-pve)
pve-manager: 5.1-36 (running version: 5.1-36/131401db)
pve-kernel-4.13.4-1-pve: 4.13.4-26
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-17
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9

The keys exist and match the md5
/etc/pve/nodes/cfPVE3/pve-ssl.key
/etc/pve/nodes/cfPVE4/pve-ssl.key
/etc/pve/nodes/cfpve2/pve-ssl.key
/etc/pve/nodes/cfPVE1/pve-ssl.key

So I am at a loss now, HELP
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,357
1,657
174
that is the md5sum of nothing (an empty string).. what does "ls -l /etc/pve" print?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!