TIL: You can recover a lost PVE Node TTY Shell

Tmanok

Well-Known Member
Good evening everyone!

A few hours ago I was running some long running commands on one of my PVE hosts to install an LXC. However, as I had to close my laptop in a hurry, I was disconnected from my VPN and therefore lost my TTY session to the PVE node. As you'll probably all be aware, when you close a host / hypervisor / node NoVNC web-TTY, you generally do not simply re-open it later like an LXC or of course a virtual machine... Though, of course you can make a VM have selectable text by converting it to a TTY if you need say a lot of security to prevent the VPN from reaching the VM's network or don't want SSH... I digress.

So tonight, I had to reclaim this TTY shell and discovered REPTYR which will take over an entire terminal rather than bringing a command from one TTY to another or killing a process and restarting it elsewhere. So in this case:
Code:
root@pve5:~# ps aux | grep -e apt -e dpkg
root       45613  0.0  0.0   6332  1536 pts/0    S+   00:56   0:00 grep -e apt -e dpkg
root     3407589  0.0  0.0  10580  3456 pts/15   S+   Nov14   0:00 dpkg <redacted ;)>

My shell was under PID 3407589 so thanks to REPTYR all I needed to do was run: reptyr -T 3407589 and I regained my TTY completely! This isn't like when you latch back on to an LXC TTY either (or a VM with Serial TTY output) where you need to press ENTER/RETURN on your keyboard to bring anything up, it's exactly where I left it showing me a DPKG menu and everything). Maybe this is obvious already to many folks but somehow I've gone all these years without needing this (just my dumb luck today trying to mess with the hypervisor in a new lab environment, I'm preparing for a colocation facility move).
Thanks,


Tmanok
 
Maybe this is obvious already to many folks but somehow I've gone all these years without needing this
I only used it once or twice, because you really don't need it if you do everything in the beginning "correctly" - after stumbling over repytr of course. I had the same learning curse as you have, but in the end the lessons to take away was something else:

The underlying problem has been solved for decades: use a terminal multiplexer like tmux or screen and run whatever you want without any direct connection to your tty. This method does not care if you run it via physical tty, serial terminal, ssh, telnet, pct enter or X session.

You can configure autostart and auto attach to tmux session over whatever config you want so that everything already runs inside of tmux so you don't have to think about it. It's a great time saver.