[SOLVED] Spice "Connecting to graphic server" indefinitely with ACME cert installed.

[Superfish1000]

I've been having this issue for over 6 months now but haven't put in the time to figure out what is causing it until now.

I configured a custom certificate for my primary server using the web GUI based ACME script. Ever since I did this spice has completely ceased to function from that node.

For my examples I have two nodes, [Scylla] and [Athena]. Scylla has a custom certificate to allow me to connect to it from the internet so that I don't get certificate warnings and I generated this certificate using the web GUI ACME menu.

Since I generated the certificate not one of my VM's has been viewable using Spice. Any attempt to view a VM from Scylla results in "Connecting to graphic server" indefinitely.
Connecting from Athena still works assuming the VM in question is running on Athena.

For the more visual people.


This happens regardless of if I connect to the server from the internet using the URL or if I connect locally using the servers local IP.

I've been trying to figure this out for a bit but I haven't had any luck. I am 100% sure this is certificate related because I had this issue before and removed the custom cert and it went away. I ended up putting the cert back however as at the time I needed external access more than I needed spice. This has gotten extremely annoying however and I'd like spice to be working again.

Version information:

proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-helper: 6.1-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-4-pve: 5.0.21-9
pve-kernel-5.0.21-3-pve: 5.0.21-7
pve-kernel-5.0.15-1-pve: 5.0.15-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-17
libpve-guest-common-perl: 3.0-5
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 4.0.1-pve1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-23
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-9
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-7
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Syslog:

[Scylla / FAILED]

Apr 05 00:19:11 scylla pvedaemon[16249]: <root@pam> end task UPID:scylla:000028B9:14954797:5E895C35:vncproxy:104:root@pam: OK
Apr 05 00:19:11 scylla pvedaemon[10643]: starting vnc proxy UPID:scylla:00002993:14954BC9:5E895C3F:vncproxy:104:root@pam:
Apr 05 00:19:11 scylla pvedaemon[16249]: <root@pam> starting task UPID:scylla:00002993:14954BC9:5E895C3F:vncproxy:104:root@pam:


[Athena / WORKING]

Apr 05 00:23:01 Athena pvedaemon[28460]: starting spiceterm UPID:Athena:00006F2C:08150A42:5E895D25:spiceproxy:116:root@pam: - CT 116
Apr 05 00:23:01 Athena pvedaemon[28460]: launch command: /usr/bin/spiceterm --port 61002 --addr localhost --timeout 40 --authpath /vms/116 --permissions VM.Console --keymap en-us -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole116 -r winch -z lxc-console -n 116
Apr 05 00:23:01 Athena pvedaemon[1024]: <root@pam> starting task UPID:Athena:00006F2C:08150A42:5E895D25:spiceproxy:116:root@pam:
Apr 05 00:23:01 Athena pvedaemon[1026]: <root@pam> end task UPID:Athena:00006E33:0814FFDE:5E895D0A:vncproxy:116:root@pam: OK
Apr 05 00:23:06 Athena pvedaemon[1026]: <root@pam> starting task UPID:Athena:00006F6A:08150C5B:5E895D2A:vncproxy:116:root@pam:
Apr 05 00:23:06 Athena pvedaemon[28522]: starting lxc termproxy UPID:Athena:00006F6A:08150C5B:5E895D2A:vncproxy:116:root@pam:
Apr 05 00:23:06 Athena pvedaemon[1024]: <root@pam> successful auth for user 'root@pam'

If anyone can point me in the right direction I'd apprecieate it. If at all possible I would like to keep the valid cert instead of wiping it just to make Spice work.

 

[Stoiko Ivanov]

just on a hunch - did you maybe replace the cluster-certificate (/etc/pve/nodes/nodename/pve-ssl.pem) instead of only the one for the web-connection (/etc/pve/nodes/nodename/pveproxy-ssl.pem)?

check the reference documentation:
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysadmin_certificate_management

I hope this helps!

 

[Superfish1000]

That was right on the money. I'm not sure why it was replaced though. Perhaps I did it by mistake when I was initially trying to get SSL up by manually configuring things the way the old guides said to. Everything seems to be functional now.


For anyone else who might run into this, here's what I did to fix it and put SSL back.
Note: I already configured ACME through the GUI before, so I was just doing it again.


First, I deleted and reset all the certificates on all my nodes as per the instructions here.
https://pve.proxmox.com/wiki/HTTPS_..._5.0_and_5.1)#Revert_to_default_configuration

Next, I reboot all the machines and reactivated ACME through the GUI on the node I am accessing from the internet.

Finally, I reboot this node to ensure it was using the new cert.

 

[Superfish1000]

So unfortunately I need to open this again as the issue came back up and I think this is a problem with Proxmox's ACME script. As I stated in my last post I configured the custom cert and everything worked just fine. A few days ago however Spice stopped working again.

Just as last time I went to check the

cluster-certificate (/etc/pve/nodes/nodename/pve-ssl.pem)

and it was changed on the 15th of July. I have not touched a single thing related to the cert or otherwise since my last post and it seems to haev changed itself.



My suspicion is that the ACME update script screwed it up when it renewed the cert. It has been 3 months since I did all of this so it seems almost certain that the ACME script is the culprit.

 

[Stoiko Ivanov]

and it was changed on the 15th of July. I have not touched a single thing related to the cert or otherwise since my last post and it seems to haev changed itself.

PVE's utilities for ACME processing only touch '/etc/pve/nodes/<nodename>/pveproxy-ssl.pem'.
The cluster certificate (which is issued from the cluster ca) has nothing to do with ACME and does get rotated automatically but the validity of these certificates is 2 years.

On a hunch - maybe you have some left-over ACME configuration, which changes pve-ssl.pem?

 

[Superfish1000]

How would I have changed this? I suppose it's possible but I would have had to manually edit the renewal script for the built in automated ACME GUI.
I didn't do anything (this time) that wasn't a GUI option. ACME didn't originally mess up the pve-ssl.pem file. It got messed up on automatic renewal.

EDIT:
I figured out what the issue is after some more digging. I think I followed the instructions here while trying to figure out how to set up HTTPS and didn't even realize it was still active.
https://pve.proxmox.com/wiki/HTTPS_...x,_5.0_and_5.1)#Let.27s_Encrypt_using_acme.sh

 

[Stoiko Ivanov]

Glad you found the issue - please mark the thread as 'SOLVED' - this helps others with a similar problem
Thanks!