serial console to VM hangs after extended time away and is unrevocerable except on reboot

D

dtreth

New Member
Apr 19, 2026
4
1
3
So I followed this guide: https://pve.proxmox.com/wiki/Serial_Terminal for setting up a terminal system on my VM because I like the xterm.js for the containers a lot, and I even installed the xterm tools to make it resize and whatnot. It works pretty well, except for needing to login separately,but I accept that as a limitation of being a VM rather than a container.
However, If I use the terminal for a while, then I leave and, say, get some food, most of the time when I come back the terminal is hung and unrecoverable. I have tried a LOT of tinkering and log chasing to figure it out but I have no idea why it just "dies". Rebooting the VM always brings it back. However, this is a Samba AD DC and rebooting constantly is really bad.

Anyone with some expertise on this? I would be extremely appreciative.
 
Welcome, @dtreth
I don't know why it hangs. I'm just thinking out loudly how to bypass the issue...
I would log in to the VM by other means (e.g., ssh) and inspect the situation when the console is hung.
Then maybe restart getty (or whatever is servicing the console).

I know that it isn't the substantial solution (if it works at all), but at least it would let avoid rebooting the Samba DC.
 
Last edited:
  • Like
Reactions: Kingneutron and dtreth
I can tell you that I used the noVNC to inspect and restart the getty. I tried to ask Claude (I know...) for some advice and using systemd overrides to tell the service to reset didn't help. It seems like the emulates serial device itself is what breaks, but I really can't tell. I can post any logs anyone who might have some expertise might need to figure it out.

It's a debian VM that was originally a Bullseye TurnkeyLinux Samba AD DC, but I ripped out the turnkey stuff and upgraded to Trixie and still have the same issues. I have the qemu system agent, fwiw, don't know if it does anything with this.
 
No I haven't set up backups and snapshots yet because I am playing around and learning. I was going to set up backups today actually, haha.

But I can try to reverse the steps from the tutorial. Basically the only difference between what you wrote up and what I have is the extra dydtemd getty invocation.

But before I do that, I will note some things from th eTask list in the Proxmox UI: there was this error:

Code: 
client connection: 127.0.0.1:59648
error reading from pty: Input/output error (os error 5)
TASK ERROR: command '/usr/bin/termproxy 5900 --path /vms/105 --perm VM.Console -- /usr/sbin/qm terminal 105 -escape 0' failed: exit code 1

And there were a couple open shells to VM 105 (that's the one) that had a "spinner" constantly. When I clicked on them to Stop they said (I cannot find the exact error right now) that the stop failed; received interrupt.

In the cluster log, however, these failures and lock ups aren't there.

After I killed the last of them I got the serial console back, so I think it's something on Proxmox's side holding onto things? I am not entirely sure.

Edit: I found the other error in another scratchpad:
Code: 
TASK ERROR: command '/usr/bin/termproxy 5900 --path /nodes/metal --perm Sys.Console --vncticket-endpoint -- /bin/login -f root' failed: received interrupt
BUT it actually DID seem to kill it, because like I said, it worksagain (for now, I am sure it will stick again when I take my next break)
 
Last edited:
I think I've seen this issue. It seems serial doesn't hang, but when client connection drops some process on the server side (kinda proxy or so) keeps serial open. When you try to connect it again, serial is still busy and unavailable. Besides rebooting, moving the affected vm to different host helps recover from this. After while it recovers by itself, but I didn't notice how long it takes
 
AH HA! I made it happen again! Left the console (closed the window but not the WebUI for the host) and came back today to a hung window and the Tasks still saying that the VM/CT 105 -- Console was still running. I clicked th elittle arrow, then the Stop button, go this error:

Code: 
client connection: 127.0.0.1:34330
TASK ERROR: command '/usr/bin/termproxy 5900 --path /vms/105 --perm VM.Console -- /usr/sbin/qm terminal 105 -escape 0' failed: received interrupt

Then opened the console and it worked! So it's definitely on the Proxmox side. I also remember that I went on to add the systemd directives from my first link after trying it without the directives--because of the comments that some VMs are already set up for it--and seeinng that it didn't work yet. I am pretty sure it's not an issue on the VM side.
On top of that, the tasks list itself didn't seem to update with my new logon and there was also a "shell" that seemed to be spinning. I hit F5 and it refreshed the tasks list, but I still had to kill the "shell" which I believe is the direct host terminal. I think Proxmox is in general not releasing the terminals correctly or something. BUT at least I found a kill button so I can get it back without having to revert to the painful noVNC.

I'm willing to provide any logs required to try to fix this issue (as it might ACTUALLY be a Proxmox bug that could be fixed).
 
  • Like
Reactions: Onslow
You must log in or register to reply here.
Top Bottom
Back