Certificate Startup Problems

Discussion in 'Proxmox VE: Installation and configuration' started by Maximilian Sesterhenn, Jan 28, 2016.

  1. Maximilian Sesterhenn

    Joined:
    Jan 28, 2016
    Messages:
    3
    Likes Received:
    0
    Hello,

    after upgrading to the latest version of Promox 4, i'm no longer able to start my VMs. I've tried to reinstall the complete node to eliminate all sources of errors but the output is still the same.

    Code:
    pveversion -V
    proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
    pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
    pve-kernel-4.2.6-1-pve: 4.2.6-34
    lvm2: 2.02.116-pve2
    corosync-pve: 2.3.5-2
    libqb0: 0.17.2-1
    pve-cluster: 4.0-31
    qemu-server: 4.0-49
    pve-firmware: 1.1-7
    libpve-common-perl: 4.0-45
    libpve-access-control: 4.0-11
    libpve-storage-perl: 4.0-38
    pve-libspice-server1: 0.12.5-2
    vncterm: 1.2-1
    pve-qemu-kvm: 2.5-3
    pve-container: 1.0-39
    pve-firewall: 2.0-15
    pve-ha-manager: 1.0-19
    ksm-control-daemon: 1.2-1
    glusterfs-client: 3.5.2-2+deb8u1
    lxc-pve: 1.1.5-6
    lxcfs: 0.13-pve3
    cgmanager: 0.39-pve1
    criu: 1.6.0-1
    zfsutils: 0.6.5-pve7~jessie
    openvswitch-switch: 2.3.2-2
    
    After starting a VM, the output shows this:

    Code:
    Running as unit 101.scope.
    kvm: -vnc unix:/var/run/qemu-server/101.vnc,x509,password: Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer
    TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 101 -p 'KillMode=none' -p 'CPUShares=50000' /usr/bin/kvm -id 101 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/101.vnc,x509,password -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=b2d31a83-8c47-41c5-8ad4-43f458244b9a' -name Backend-1 -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi -m 2048 -k de -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:3681fcbb6821' -drive 'file=/var/lib/vz/images/101/vm-101-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=200' -drive 'file=/var/lib/vz/template/iso/debian-8.3.0-amd64-DVD-1.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=02:00:00:C0:83:C0,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap101i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=1A:5E:AA:FF:03:57,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301'' failed: exit code 1
    
    I've checked the cert, and it's definitely the correct one. My browser accepts the cert without any problems.

    Thanks

    Maximilian
     
  2. tom

    tom Proxmox Staff Member
    Staff Member

    Joined:
    Aug 29, 2006
    Messages:
    13,029
    Likes Received:
    333
    so you changed the default certificate?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Maximilian Sesterhenn

    Joined:
    Jan 28, 2016
    Messages:
    3
    Likes Received:
    0
    Yes, thats right.
    It worked for years, the cert is valid. It works in my browser.
    Atm it seems like one of the latest updates of ca-certificates for debian has removed the nessecary root-certificate (startssl). For now, i'm testing with LetsEncrypt. If that works, there is no direct relation to Proxmox.

    I'm sorry for the confusion.

    Best regards

    Maximilian
     
  4. Maximilian Sesterhenn

    Joined:
    Jan 28, 2016
    Messages:
    3
    Likes Received:
    0
    Ok, after testing with a cert from LetsEncrypt i can confirm that this is not a problem of proxmox. The cert was not valid because the server has got an new version of mozillas ca database which no longer contains the ca of StartSSL.

    You can close this thread.

    Thanks
     
  5. Jim O

    Jim O New Member

    Joined:
    Jan 2, 2016
    Messages:
    16
    Likes Received:
    5
    We had the same issue. Adding Startcom's intermediate certificate to the end of /etc/pve/pve-root-ca.pem solved the issue for us.
     
    tho1h and unze like this.
  6. sol

    sol New Member

    Joined:
    Apr 1, 2016
    Messages:
    3
    Likes Received:
    0
    Hello, I got same error. I have cluster with five nodes.

    #pveversion --verbose
    proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)
    pve-manager: 4.1-22 (running version: 4.1-22/aca130cf)
    pve-kernel-4.2.6-1-pve: 4.2.6-36
    pve-kernel-4.2.2-1-pve: 4.2.2-16
    lvm2: 2.02.116-pve2
    corosync-pve: 2.3.5-2
    libqb0: 1.0-1
    pve-cluster: 4.0-32
    qemu-server: 4.0-64
    pve-firmware: 1.1-7
    libpve-common-perl: 4.0-54
    libpve-access-control: 4.0-13
    libpve-storage-perl: 4.0-45
    pve-libspice-server1: 0.12.5-2
    vncterm: 1.2-1
    pve-qemu-kvm: 2.5-9
    pve-container: 1.0-52
    pve-firewall: 2.0-22
    pve-ha-manager: 1.0-25
    ksm-control-daemon: 1.2-1
    glusterfs-client: 3.5.2-2+deb8u1
    lxc-pve: 1.1.5-7
    lxcfs: 2.0.0-pve2
    cgmanager: 0.39-pve1
    criu: 1.6.0-1
    zfsutils: 0.6.5-pve7~jessie
    drbdmanage: 0.91-1

    I generate LetsEncrypt cert for it with LetsEncrypt tool
    #letsencrypt --agree-tos --renew-by-default --standalone --standalone-supported-challenges http-01 --http-01-port 9999 --server https://acme-v01.api.letsencrypt.org/directory certonly -d cluster1.domain.ltd -d n1.cluster1.domain.ltd -d n2.cluster1.domain.ltd -d n3.cluster1.domain.ltd -d n4.cluster1.domain.ltd -d n5.cluster1.domain.ltd --rsa-key-size 4096

    I got positive result with it and four files:
    lrwxrwxrwx 1 root root 43 Mar 30 23:21 cert.pem -> ../../archive/cluster1.domain.ltd/cert1.pem
    lrwxrwxrwx 1 root root 44 Mar 30 23:21 chain.pem -> ../../archive/cluster1.domain.ltd/chain1.pem
    lrwxrwxrwx 1 root root 48 Mar 30 23:21 fullchain.pem -> ../../archive/cluster1.domain.ltd/fullchain1.pem
    lrwxrwxrwx 1 root root 46 Mar 30 23:21 privkey.pem -> ../../archive/cluster1.domain.ltd/privkey1.pem

    Then i cp cert and keys according to wiki https://pve.proxmox.com/wiki/HTTPSCertificateConfiguration

    cp fullchain.pem /etc/pve/nodes/<node>/pveproxy-ssl.pem
    cp private-key.pem /etc/pve/nodes/<node>/pveproxy-ssl.key

    #systemctl restart pveproxy - in all nodes

    Then i go to browser https://<node>.cluster1.domain.ltd:8006 -cert OK

    But when i try start vm i got error:
    Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer

    Then I
    cp chain.pem to /etc/pve/pve-root-ca.pem
    cp privkey.pem to /etc/pve/pve-www.key
    and had
    Error Permission denied - invalid csrf token (401)

    # md5sum /etc/pve/pve-www.key - OK

    Help me!
     
  7. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,086
    Likes Received:
    470
    Please follow the instructions in the linked wiki article - including resetting the cluster CA and self signed certificate files as a first step!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. sol

    sol New Member

    Joined:
    Apr 1, 2016
    Messages:
    3
    Likes Received:
    0
    So in first step I must rm/mv

    /etc/pve/pve-root-ca.pem
    /etc/pve/priv/pve-root-ca.key
    /etc/pve/nodes/<node>/pve-ssl.pem
    /etc/pve/nodes/<node>/pve-ssl.key
    then cp

    fullchain.pem /etc/pve/nodes/<node>/pveproxy-ssl.pem
    private-key.pem /etc/pve/nodes/<node>/pveproxy-ssl.key
    at finish
    systemctl restart pveproxy in all nodes
     
  9. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,086
    Likes Received:
    470
    Like the howto states, you need to run "pvecm updatecerts -f" on every node to regenerate the cluster CA and self signed certificate files (after deleting/moving them). Then you can copy your LE files like described in the howto. I will not duplicate the whole howto here, just read it completely and then follow the steps..
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  10. sol

    sol New Member

    Joined:
    Apr 1, 2016
    Messages:
    3
    Likes Received:
    0
    Thank you very much!!!
     
  11. tho1h

    tho1h New Member

    Joined:
    Jun 6, 2018
    Messages:
    1
    Likes Received:
    0
    Thanks! After adding GlobalSign intermediate certificate to pve-root-ca.pem I was able to start my VMs again. It solved my issue!
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice