Certificate Startup Problems

[Maximilian Sesterhenn]

Hello,

after upgrading to the latest version of Promox 4, i'm no longer able to start my VMs. I've tried to reinstall the complete node to eliminate all sources of errors but the output is still the same.

pveversion -V
proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
openvswitch-switch: 2.3.2-2

After starting a VM, the output shows this:

Running as unit 101.scope.
kvm: -vnc unix:/var/run/qemu-server/101.vnc,x509,password: Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer
TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 101 -p 'KillMode=none' -p 'CPUShares=50000' /usr/bin/kvm -id 101 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/101.vnc,x509,password -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=b2d31a83-8c47-41c5-8ad4-43f458244b9a' -name Backend-1 -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi -m 2048 -k de -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:3681fcbb6821' -drive 'file=/var/lib/vz/images/101/vm-101-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=200' -drive 'file=/var/lib/vz/template/iso/debian-8.3.0-amd64-DVD-1.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=02:00:00:C0:83:C0,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap101i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=1A:5E:AA:FF:03:57,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301'' failed: exit code 1

I've checked the cert, and it's definitely the correct one. My browser accepts the cert without any problems.

Thanks

Maximilian

 

[tom]

so you changed the default certificate?

 

[Maximilian Sesterhenn]

Yes, thats right.
It worked for years, the cert is valid. It works in my browser.
Atm it seems like one of the latest updates of ca-certificates for debian has removed the nessecary root-certificate (startssl). For now, i'm testing with LetsEncrypt. If that works, there is no direct relation to Proxmox.

I'm sorry for the confusion.

Best regards

Maximilian

 

[Maximilian Sesterhenn]

Ok, after testing with a cert from LetsEncrypt i can confirm that this is not a problem of proxmox. The cert was not valid because the server has got an new version of mozillas ca database which no longer contains the ca of StartSSL.

You can close this thread.

Thanks

 

[Jim O]

We had the same issue. Adding Startcom's intermediate certificate to the end of /etc/pve/pve-root-ca.pem solved the issue for us.

 

[sol]

Hello, I got same error. I have cluster with five nodes.

#pveversion --verbose
proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-22 (running version: 4.1-22/aca130cf)
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.2.2-1-pve: 4.2.2-16
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-32
qemu-server: 4.0-64
pve-firmware: 1.1-7
libpve-common-perl: 4.0-54
libpve-access-control: 4.0-13
libpve-storage-perl: 4.0-45
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-9
pve-container: 1.0-52
pve-firewall: 2.0-22
pve-ha-manager: 1.0-25
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
drbdmanage: 0.91-1

I generate LetsEncrypt cert for it with LetsEncrypt tool
#letsencrypt --agree-tos --renew-by-default --standalone --standalone-supported-challenges http-01 --http-01-port 9999 --server https://acme-v01.api.letsencrypt.org/directory certonly -d cluster1.domain.ltd -d n1.cluster1.domain.ltd -d n2.cluster1.domain.ltd -d n3.cluster1.domain.ltd -d n4.cluster1.domain.ltd -d n5.cluster1.domain.ltd --rsa-key-size 4096

I got positive result with it and four files:
lrwxrwxrwx 1 root root 43 Mar 30 23:21 cert.pem -> ../../archive/cluster1.domain.ltd/cert1.pem
lrwxrwxrwx 1 root root 44 Mar 30 23:21 chain.pem -> ../../archive/cluster1.domain.ltd/chain1.pem
lrwxrwxrwx 1 root root 48 Mar 30 23:21 fullchain.pem -> ../../archive/cluster1.domain.ltd/fullchain1.pem
lrwxrwxrwx 1 root root 46 Mar 30 23:21 privkey.pem -> ../../archive/cluster1.domain.ltd/privkey1.pem

Then i cp cert and keys according to wiki https://pve.proxmox.com/wiki/HTTPSCertificateConfiguration

cp fullchain.pem /etc/pve/nodes/<node>/pveproxy-ssl.pem
cp private-key.pem /etc/pve/nodes/<node>/pveproxy-ssl.key

#systemctl restart pveproxy - in all nodes

Then i go to browser https://<node>.cluster1.domain.ltd:8006 -cert OK

But when i try start vm i got error:
Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer

Then I
cp chain.pem to /etc/pve/pve-root-ca.pem
cp privkey.pem to /etc/pve/pve-www.key
and had
Error Permission denied - invalid csrf token (401)

# md5sum /etc/pve/pve-www.key - OK

Help me!

 

[fabian]

Please follow the instructions in the linked wiki article - including resetting the cluster CA and self signed certificate files as a first step!

 

[sol]

So in first step I must rm/mv

/etc/pve/pve-root-ca.pem
/etc/pve/priv/pve-root-ca.key
/etc/pve/nodes/<node>/pve-ssl.pem
/etc/pve/nodes/<node>/pve-ssl.key
​

then cp

fullchain.pem /etc/pve/nodes/<node>/pveproxy-ssl.pem
private-key.pem /etc/pve/nodes/<node>/pveproxy-ssl.key
​

at finish
systemctl restart pveproxy in all nodes

 

[fabian]

Like the howto states, you need to run "pvecm updatecerts -f" on every node to regenerate the cluster CA and self signed certificate files (after deleting/moving them). Then you can copy your LE files like described in the howto. I will not duplicate the whole howto here, just read it completely and then follow the steps..

 

[sol]

Thank you very much!!!

 

[tho1h]

We had the same issue. Adding Startcom's intermediate certificate to the end of /etc/pve/pve-root-ca.pem solved the issue for us.

Thanks! After adding GlobalSign intermediate certificate to pve-root-ca.pem I was able to start my VMs again. It solved my issue!