"pct start" killed my whole Proxmox server (again)

[eth]

So I have been playing with my LXCs, turning them on and off (with "pxc start" and "pxc shutdown").
At one point "pxc start" started hanging for too long, so I stopped it with "ctrl-c".

At the same time I couldn't connect via SSH access to my system any more (old connection was fine, new connection fucked up). Got errors "server refused to allocate tty" and "stdin: is not a tty"


My proxmox web gui is fucked up:




This problem happened before and was solved by rebooting the whole proxmox server.
I am not happy that I will have to do this After all it is a live server.

 

[fabian]

Please update your installation to the current version. If you have other means of accessing the node (IPMI, serial console, ..) you can try to connect using that and retrieve log files. Otherwise it seems you will have to do a reboot.

If you have already enabled persistent logging, please provide the log output from the boot where the issue occured (i.e., "journalctl -b-1" for the previous boot). Otherwise, you will have to reboot, enable persistent logging ("mkdir /var/log/journal; systemctl restart systemd-journald.service") and wait for it to happen again in order to collect log output.

 

[eth]

Please update your installation to the current version. If you have other means of accessing the node (IPMI, serial console, ..) you can try to connect using that and retrieve log files. Otherwise it seems you will have to do a reboot.

If you have already enabled persistent logging, please provide the log output from the boot where the issue occured (i.e., "journalctl -b-1" for the previous boot). Otherwise, you will have to reboot, enable persistent logging ("mkdir /var/log/journal; systemctl restart systemd-journald.service") and wait for it to happen again in order to collect log output.

Thank you for the reply. I still have one single working SSH access left to the server + I have IPMI. Proxmox is relatively new (4.1 + did a package update lately). Nothing suspicious in "dmesg" or "/var/log/messages".
Should I still enable persistent logging? Does it show something else?

 

[fabian]

Yes, please save the output of "journalctl -b" (volatile log of the current boot) and then enable persistent logging afterwards. It seems like your node has run out of ptys, so check the logs for messages relating to pty and tty allocation, messages by the openssh server might also be relevant.

 

[eth]

You are correct, it did run out of ptys. How would I tune that?

 

[fabian]

Can you post the configurations of the containers you used for testing? Did you setup anything manually?

You can retrieve the configuration with "pct config <ID>".
Also, providing the full log outpu as text (attachment) instead of a part as screenshot might help in finding the root caues of this issue.

 

[eth]

Nothing out of the ordinary, regular CentOS machines:

The system and machines are still running normally, I am waiting for deep nighttime to do a proper reboot.

There is still some time for testing. Skype arkadiynight if you want to do this online.

 

[eth]

I googled and someone has already reported exactly the same bug:
https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1425477

 

[eth]

Here is what I tried to do (and what actually caused the problem). Looks like there is a bug with autodev and LXC:

cat > /usr/share/lxc/config/common.conf.d/02-openvpn-auto-tun.conf << EOL
lxc.hook.autodev = /usr/share/lxc/hooks/openvpn-auto-tun
EOL

cat > /usr/share/lxc/hooks/openvpn-auto-tun << EOL
#!/bin/bash
cd ${LXC_ROOTFS_MOUNT}/dev
mkdir net
mknod net/tun c 10 200
chmod 0666 net/tun
EOL

chmod 755 /usr/share/lxc/hooks/openvpn-auto-tun

I did this to setup a openvpn server inside the LXC and my system became irresponsive afterwards. Looks like autodev is in fact the problem. Rebooting and cleaning the autodev settings from config helped.

 

[is-max]

Hi

could you post your pveversion -v ?

Seems strange, all my vm use this script from my post, and anyway autodev is active by default on proxmox.
The unique difference between my setup and your, is that I use only debian or ubuntu vm, Im thinking it is something related with centos vm.

Regards

 

[eth]

Hi

could you post your pveversion -v ?

Seems strange, all my vm use this script from my post, and anyway autodev is active by default on proxmox.
The unique difference between my setup and your, is that I use only debian or ubuntu vm, Im thinking it is something related with centos vm.

Regards

root@pve:~# pveversion -v
proxmox-ve: 4.1-37 (running kernel: 4.2.8-1-pve)
pve-manager: 4.1-13 (running version: 4.1-13/cfb599fb)
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.2.8-1-pve: 4.2.8-37
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-32
qemu-server: 4.0-55
pve-firmware: 1.1-7
libpve-common-perl: 4.0-48
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-40
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-5
pve-container: 1.0-44
pve-firewall: 2.0-17
pve-ha-manager: 1.0-21
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-7
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie

I am using default proxmox 4.1 install from usb stick.

 

[is-max]

Hi

Are you using the enterprise repository?

Iv two server, one stock, and another one with nosubscription repository, looking at the version of your packages seems a middle-way from both stock and updated, did you tryed to do again: apt-get update && apt-get upgrade?

Later I'll post my both pveversion,now im from mobile.

Regards

Spoiler: ProxMox 4.1-39 (updated nosubscription repository)

proxmox-ve: 4.1-39 (running kernel: 4.2.8-1-pve)
pve-manager: 4.1-15 (running version: 4.1-15/8cd55b52)
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.2.8-1-pve: 4.2.8-39
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-33
qemu-server: 4.0-62
pve-firmware: 1.1-7
libpve-common-perl: 4.0-49
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-42
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-9
pve-container: 1.0-46
pve-firewall: 2.0-18
pve-ha-manager: 1.0-24
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve1
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie

Spoiler: ProxMox stock 4.1 ISO

proxmox-ve: 4.1-26 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-1 (running version: 4.1-1/2f9650d4)
pve-kernel-4.2.6-1-pve: 4.2.6-26
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-29
qemu-server: 4.0-41
pve-firmware: 1.1-7
libpve-common-perl: 4.0-41
libpve-access-control: 4.0-10
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-17
pve-container: 1.0-32
pve-firewall: 2.0-14
pve-ha-manager: 1.0-14
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve1
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve6~jessie

 

[eth]

I found a solution, how to start openvpn inside the lxc container without crashing proxmox: https://forum.proxmox.com/threads/tun-devices-in-ve-4-lxc.23473/#post-132999