KVM failing to start after upgrading Proxmox 4.0 to 4.1

O

Ovidiu

Renowned Member
Proxmox Subscriber
Apr 27, 2014
326
12
83
I followed the wiki by turning all VMs of, then upgrading via apt-get update followed by apt-get dist-upgrade and restart.

LXC containers are up again but not my KVMs. Trying to start them I get this error, any hints?

Code: 
Running as unit 101.scope.
kvm: -vnc unix:/var/run/qemu-server/101.vnc,x509,password: Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer
TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 101 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 101 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/101.vnc,x509,password -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=dcc6a967-a39f-4d74-8dc8-45d0baf56ac2' -name thomas.ict-consult.co.za -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga std -no-hpet -cpu 'kvm64,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_relaxed,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce' -m 8196 -k de -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:3681fcbb6821' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100' -drive 'file=/var/lib/vz/images/101/vm-101-disk-1.qcow2,if=none,id=drive-virtio0,cache=unsafe,format=qcow2,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=200' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=02:00:00:90:96:87,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
 
Seems there is something wrong with the ssl certificate /etc/pve/pve-root-ca.pem.
Maybe you changed something manually?
 
Thanks for the replies but my intermediate certificate is already appended to the pve-root-ca.pem and it has been since I started using Proxmox with startssl's certificates.

So what exactly has the update changed that its no longer working?

###edit###
using a class 1 certificate for proxmox.mydomain.tld
 
Last edited:
I've tried to move the old certificates back as that woudl at least allow me to start my KVMs again, then did a
service pveproxy restart
service pvedaemon restart

Visited my proxmox.domain.tld in my browser and get a warning about a self signed certificate yet I cannot login as I always get a Login failed message. If I move my own certificates back, I can login just fine.

Anyone, help please?
 
Did you clear your browser cache after moving the self-signed certificates back in place (or try with a different, new browser/profile)? The certificates are not involved in the web GUI login..

If nothing else works, you should be able to return to the normal configuration (self-signed certificates for both cluster CA and individual nodes) by removing the cluster CA key and certificate:

Code: 
rm /etc/pve/pve-root-ca.pem /etc/pve/priv/pve-root-ca.key

followed by regenerating the node key and certificate and restarting pveproxy on each node (on the first node, this will generate a new cluster CA key and certificate first):

Code: 
pvecm updatecerts -f
Code: 
systemctl restart pveproxy

Hope this helps.
 
OK, thanks for the reply, so here is what I tried:

when I was replacing the original certificates with my own from starttls, I made a backup inside /etc/pve/local/old and the folder /etc/pve/local/new contains the new stuff.

So I tried copying everything from old back to /etc/pve/local restarted pveproxy pvedaemon and cannot log in.
Forgot to mention that I am using OTP too.

As soon as I copy all files from the new folder back to /etc/pve/local restart the two services I can log in again but hit the problem with not being able to restart the KVMs.
--------------------------------
@fabian I was just about to follow your advice to reset things when I realized you are talking about /etc/pve but everything I did was in /etc/pve/local according to this tutorial: http://blog.michaelboman.org/2012/04/getting-real-ssl-certificate-in-my.html
Would you mind having a look? Maybe you have a better tutorial or instructions to reset and then try again to import my StartSSL certificate?
 
"/etc/pve/local" contains files related to your local node, the cluster-wide CA files are in "/etc/pve/pve-root-ca.pem" (CA certificate) and "/etc/pve/priv/pve-root-ca.key" (CA private key). The web GUI only uses the certificate (or chain) and key from the local node, but KVM (VNC) and Spice use both the CA certificate and the certificate (chain) and key from the local node.
 
  • Like
Reactions: Ovidiu
Can the Proxmox team advise yet on what caused this issue?

We're seeing it happen for several people on here now, and something has clearly changed in Proxmox, as our existing certificates simply "stopped" working, when they'd been fine previously.

Also really looking forward to the improvements for SSL certificate management in coming versions!

Jon
 
  • Like
Reactions: Ovidiu
Qemu upstream changed their TLS validation checks with the 2.5 release. We will see whether we can implement a workaround to fix this.
 
  • Like
Reactions: Ovidiu
Thanks! my workaround was to copy my old certificates back restart pveproxy and pvedaemon then manually start the KVM, copy the new certs back, restart the 2 daemons again.
 
FWIW, I was able to get everything to work (though I have not tried adding a node) with a wildcard cert from RapidSSL chained certificate under GeoTrust:
  1. copy wildcard-ssl.key on top of /etc/pve/local/pve-ssl.key on each node
  2. copy wildcard-ssl.crt on top of /etc/pve/local/pve-ssl.pem on each node
  3. copy RapidSSL-CA-SHA256.crt to /etc/pve/pve-root-ca.pem (which is shared on all nodes)
Restart pvedaemon and pveproxy.

There is no need to install the GeoTrust root certificate as the validation is complete once a trusted cert is reached in the "root-ca" file.

I have not needed to include the RapidSSL intermediate cert into the pve-ssl.pem file. It seems Safari and Chrome already know it, or it is being presented by the pveproxy, I cannot tell which.

Up until this most recent change with the stricter kvm certificate checking, I had the RapidSSL CA intermediate cert appended to pve-ssl.pem and left the root-ca-pem as whatever was there on install of the cluster.
 
The setup you describe will break adding new nodes to the cluster and using the Spice console. We will probably have packages ready for testing next week that should make it easier to use setups like this, by providing alternate locations where the certificate chain and key for pveproxy (i.e., web interface and noVNC) can be provided, while leaving the self-signed certificates for the cluster, spice and old-school KVM VNC in place.
 
fabian said:
The setup you describe will break adding new nodes to the cluster and using the Spice console. We will probably have packages ready for testing next week that should make it easier to use setups like this, by providing alternate locations where the certificate chain and key for pveproxy (i.e., web interface and noVNC) can be provided, while leaving the self-signed certificates for the cluster, spice and old-school KVM VNC in place.
Click to expand...

Thanks for the info. Will there be instructions on how to "repair" my current setup after these packages come out?
 
vkhera said:
Thanks for the info. Will there be instructions on how to "repair" my current setup after these packages come out?
Click to expand...

yes, I will update the wiki article (and archive the current one) as soon as the packages are available for testing. it will boil down to resetting to the default self signed configuration and then configuring the new stuff ;)
 
  • Like
Reactions: vkhera
Yes, sorry! Updated my post.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back