Host won't boot with serial output any more?

[JonathanB19]

My PVE host machine is connected to a terminal server using the following grub.cfg:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,9600n8"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=9600 --unit=0 --word=8 --parity=no --stop=1"

Normally, when rebooting the host following a PVE kernel update, this works just fine and I can watch the machine go through it's POST routine, etc.

This time however, it seemed to get to the point of loading the OS and then hung with a flashing cursor. I tried power cycling but it still did the same thing.

When I connected a VGA screen to the box and booted it, the machine came up just fine. So I suspect a problem with the boot process when it's outputting to serial.

Does anyone know if anything in either Debian or PVE changed recently? Is the above grub.cfg no longer adequate or is else preventing the boot process?

Thanks for any clues...

proxmox-ve: 4.3-66 (running kernel: 4.4.19-1-pve)
pve-manager: 4.3-1 (running version: 4.3-1/e7cdc165)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.13-1-pve: 4.4.13-56
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-4.4.13-2-pve: 4.4.13-58
pve-kernel-4.2.8-1-pve: 4.2.8-41
pve-kernel-2.6.32-45-pve: 2.6.32-174
pve-kernel-4.4.19-1-pve: 4.4.19-66
pve-kernel-4.4.10-1-pve: 4.4.10-54
lvm2: 2.02.116-pve3
corosync-pve: 2.4.0-1
libqb0: 1.0-1
pve-cluster: 4.0-46
qemu-server: 4.0-88
pve-firmware: 1.1-9
libpve-common-perl: 4.0-73
libpve-access-control: 4.0-19
libpve-storage-perl: 4.0-61
pve-libspice-server1: 0.12.8-1
vncterm: 1.2-1
pve-qemu-kvm: 2.6.1-6
pve-container: 1.0-75
pve-firewall: 2.0-29
pve-ha-manager: 1.0-35
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 2.0.4-1
lxcfs: 2.0.3-pve1
criu: 1.6.0-1
novnc-pve: 0.5-8
​

Here's the apt history entry that contained all the updates I did before rebooting:

Start-Date: 2016-10-01 07:56:05
Commandline: apt upgrade
Install:
smartmontools:amd64 (6.3+svn4002-2+b2 automatic)
pve-kernel-4.4.19-1-pve:amd64 (4.4.19-66 automatic)
libcurl3:amd64 (7.38.0-4+deb8u4 automatic)
Upgrade: bind9-host:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
liblwres90:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
e2fslibs:amd64 (1.42.12-1.1 1.42.12-2)
lvm2:amd64 (2.02.116-pve2 2.02.116-pve3)
libpve-common-perl:amd64 (4.0-72 4.0-73)
libpython2.7-stdlib:amd64 (2.7.9-2 2.7.9-2+deb8u1)
multiarch-support:amd64 (2.19-18+deb8u4 2.19-18+deb8u6)
libdns100:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
libisc-export95:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
libisccfg90:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
libnet-ssleay-perl:amd64 (1.65-1+b1 1.65-1+deb8u1)
libssl1.0.0:amd64 (1.0.1t-1+deb8u2 1.0.1t-1+deb8u5)
libpve-storage-perl:amd64 (4.0-56 4.0-61)
libbind9-90:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
e2fsprogs:amd64 (1.42.12-1.1 1.42.12-2)
pve-manager:amd64 (4.2-17 4.3-1)
dmsetup:amd64 (1.02.93-pve2 1.02.93-pve3)
qemu-server:amd64 (4.0-85 4.0-88)
libarchive13:amd64 (3.1.2-11+deb8u2 3.1.2-11+deb8u3)
open-iscsi:amd64 (2.0.873+git0.3b4b4500-8+deb8u1 2.0.873+git0.3b4b4500-8+deb8u2)
iproute2:amd64 (4.4.0-1 4.4.0-2)
libcomerr2:amd64 (1.42.12-1.1 1.42.12-2)
gnupg-agent:amd64 (2.0.26-6 2.0.26-6+deb8u1)
liblvm2cmd2.02:amd64 (2.02.116-pve2 2.02.116-pve3)
file:amd64 (5.22+15-2+deb8u1 5.22+15-2+deb8u2)
libmagic1:amd64 (5.22+15-2+deb8u1 5.22+15-2+deb8u2)
libpam-systemd:amd64 (215-17+deb8u4 215-17+deb8u5)
libc-bin:amd64 (2.19-18+deb8u4 2.19-18+deb8u6)
libc6:amd64 (2.19-18+deb8u4 2.19-18+deb8u6)
python2.7:amd64 (2.7.9-2 2.7.9-2+deb8u1)
libss2:amd64 (1.42.12-1.1 1.42.12-2)
python-magic:amd64 (5.22+15-2+deb8u1 5.22+15-2+deb8u2)
dnsutils:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
pve-ha-manager:amd64 (1.0-32 1.0-35)
udev:amd64 (215-17+deb8u4 215-17+deb8u5)
base-files:amd64 (8+deb8u5 8+deb8u6)
gnupg:amd64 (1.4.18-7+deb8u2 1.4.18-7+deb8u3)
sqlite3:amd64 (3.8.7.1-1+deb8u1 3.8.7.1-1+deb8u2)
libsqlite3-0:amd64 (3.8.7.1-1+deb8u1 3.8.7.1-1+deb8u2)
lxcfs:amd64 (2.0.0-pve2 2.0.3-pve1)
pve-cluster:amd64 (4.0-43 4.0-46)
libxml2:amd64 (2.9.1+dfsg1-5+deb8u2 2.9.1+dfsg1-5+deb8u3)
libudev1:amd64 (215-17+deb8u4 215-17+deb8u5)
pve-qemu-kvm:amd64 (2.6-1 2.6.1-6)
libpve-access-control:amd64 (4.0-18 4.0-19)
pve-container:amd64 (1.0-71 1.0-75)
wget:amd64 (1.16-1 1.16-1+deb8u1)
libpython2.7:amd64 (2.7.9-2 2.7.9-2+deb8u1)
systemd-sysv:amd64 (215-17+deb8u4 215-17+deb8u5)
libcorosync4-pve:amd64 (2.3.5-2 2.4.0-1)
systemd:amd64 (215-17+deb8u4 215-17+deb8u5)
libdevmapper-event1.02.1:amd64 (1.02.93-pve2 1.02.93-pve3)
dmeventd:amd64 (1.02.93-pve2 1.02.93-pve3)
lxc-pve:amd64 (1.1.5-7 2.0.4-1)
libdevmapper1.02.1:amd64 (1.02.93-pve2 1.02.93-pve3)
gpgv:amd64 (1.4.18-7+deb8u2 1.4.18-7+deb8u3)
python2.7-minimal:amd64 (2.7.9-2 2.7.9-2+deb8u1)
proxmox-ve:amd64 (4.2-58 4.3-66)
gnupg2:amd64 (2.0.26-6 2.0.26-6+deb8u1)
pve-docs:amd64 (4.2-7 4.3-1)
tzdata:amd64 (2016d-0+deb8u1 2016f-0+deb8u1)
openssl:amd64 (1.0.1t-1+deb8u2 1.0.1t-1+deb8u5)
libsystemd0:amd64 (215-17+deb8u4 215-17+deb8u5)
liblvm2app2.2:amd64 (2.02.116-pve2 2.02.116-pve3)
novnc-pve:amd64 (0.5-6 0.5-8)
corosync-pve:amd64 (2.3.5-2 2.4.0-1)
libisccfg-export90:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
libdns-export100:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
libxml2-utils:amd64 (2.9.1+dfsg1-5+deb8u2 2.9.1+dfsg1-5+deb8u3)
pve-firmware:amd64 (1.1-8 1.1-9)
locales:amd64 (2.19-18+deb8u4 2.19-18+deb8u6)
libpython2.7-minimal:amd64 (2.7.9-2 2.7.9-2+deb8u1)
libirs-export91:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
libisccc90:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
libisc95:amd64 (9.9.5.dfsg-9+deb8u6 9.9.5.dfsg-9+deb8u7)
End-Date: 2016-10-01 07:57:43
​

 

[riptide_wave]

Have you confirmed that your grub config in /boot was updated? You can force this using the "update-grub2" command as root. I also personally remove "quiet" as this suppresses most of the kernel's boot information. If it helps, here is the config I use on my system:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0,115200n8 amd_iommu=on pcie_acs_override=downstream"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
# root FS on a local partition
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

 

[JonathanB19]

Thanks - interesting you're using console=ttyS0 on its own, while I've got both console=tty0 and ttyS0. You also have GRUB_TERMINAL=console while I have GRUB_TERMINAL=serial.

I think I need to read up more about how grub works with serial output as I suspect this might be the second time I've had problems with Debian booting using the configuration I have. Last time it was something to do with the bootlogger, which I think I had to disable (or hit CTRL+C in order for it to carry on booting).

BTW the grub config I'm using was suggested to me by my ISP (to whose terminal server the machine is connected) so it may not be right.

 

[riptide_wave]

Thanks - interesting you're using console=ttyS0 on its own, while I've got both console=tty0 and ttyS0. You also have GRUB_TERMINAL=console while I have GRUB_TERMINAL=serial.

I think I need to read up more about how grub works with serial output as I suspect this might be the second time I've had problems with Debian booting using the configuration I have. Last time it was something to do with the bootlogger, which I think I had to disable (or hit CTRL+C in order for it to carry on booting).

BTW the grub config I'm using was suggested to me by my ISP (to whose terminal server the machine is connected) so it may not be right.

Yeah, reason I only use console=ttyS0 is because my proxmox host has no GPU (embedded or external) so the only way in is via console, thus there are no ttyS "interfaces" available to me.

 

[JonathanB19]

Oh OK. I've got a GPU so I better not remove that bit!