apt-get upgrade broke my VE-3.2

[jkister]

running an older (but fully working) version of VE and ran 'apt-get update && apt-get upgrade'. it upgraded lots of packages.

apt-get update
apt-get upgrade

afterwards, the https gui started acting funky. timing out, not showing correct fields, just weird.

proxmox1> pveversion -v
proxmox-ve-2.6.32: 3.2-132 (running kernel: 2.6.32-31-pve)
pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)
pve-kernel-2.6.32-20-pve: 2.6.32-100
pve-kernel-2.6.32-12-pve: 2.6.32-68
pve-kernel-2.6.32-19-pve: 2.6.32-96
pve-kernel-2.6.32-28-pve: 2.6.32-124
pve-kernel-2.6.32-31-pve: 2.6.32-132
pve-kernel-2.6.32-26-pve: 2.6.32-114
pve-kernel-2.6.32-11-pve: 2.6.32-66
pve-kernel-2.6.32-23-pve: 2.6.32-109

i decided to log out of the gui, close the browser, and go back in. but now my https authentications all fail (ssh still works fine)

every few seconds in /var/log/syslog i see:
May 6 23:00:52 proxmox1 pmxcfs[541534]: [libqb] error: Could not accept client connection: Too many open files (24)

i've noticed:
proxmox1> pvecm nodes
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
cman_tool: Cannot open connection to cman, is it running ?
proxmox1> ^C
proxmox1> pvecm status
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
cman_tool: Cannot open connection to cman, is it running ?

cman is indeed not running. i have no /etc/cluster/cluster.conf as required by the /etc/init.d/cman

any clue how to fix all this? is cman the issue?

 

[spirit]

I think we already said everywhere in the forum, in the wiki and official documentation:

never use "apt-get upgrade" !

you need to use "apt-get dist-upgrade".

proxmox is a rolling released, and new packages can be installed when you upgrade.
if you upgrade instead dist-upgrade, it's only upgrade installed packages, but if new packages are needed, they are not installed.

 

[jkister]

ok, hindsight 20/20. is it safe to reboot now after dist-upgrade (after a regular upgrade) ? not sure what to do now.

after dist-upgrade:
proxmox1> qm suspend 100
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
unable to find configuration file for VM 100 - no such machine
proxmox1> df -h | grep 100
/var/lib/vz/private/100 120G 99G 22G 83% /var/lib/vz/root/100
tmpfs 820M 52K 820M 1% /var/lib/vz/root/100/run
tmpfs 5.0M 0 5.0M 0% /var/lib/vz/root/100/run/lock
tmpfs 1.8G 0 1.8G 0% /var/lib/vz/root/100/run/shm

it does seem the webgui is working again.

 

[jkister]

i havent rebooted this box yet -- hoping someone may be able to clue me in if a reboot is likely to succeed. the last thing i did was an apt-get dist-upgrade.

pvecm commands fail because of cman not running issue (see above)
webgui doesnt let me authenticate, when i try syslog shows: pveproxy[997690]: ipcc_send_rec failed: Resource temporarily unavailable"

but VMs still are working.

anyone have advice? can i slurp off the entire /var/lib/vz/ to another machine, reinstall proxmox and restore the /var/lib/vz somehow? and/or should i try to upgrade to the latest proxmox in this half-broken state ?

 

[jkister]

i'd be willing to pay a fee - maybe $50 in bitcoin/bch or paypal -- if a well known & capable member is willing to jump into this box via ssh to fix this up for me (and possibly upgrade to proxmox4 or 5 with the existing guest). i do not have access to the console because the console port is hung - and i cannt reboot the box because i dont know if it'll come back in its current state of br0ken.

 

[aderumier]

i'd be willing to pay a fee - maybe $50 in bitcoin/bch or paypal -- if a well known & capable member is willing to jump into this box via ssh to fix this up for me (and possibly upgrade to proxmox4 or 5 with the existing guest). i do not have access to the console because the console port is hung - and i cannt reboot the box because i dont know if it'll come back in its current state of br0ken.

if you still have access through ssh:

#apt-get install proxmox-ve

should fix it

 

[jkister]

#apt-get install proxmox-ve

proxmox1> apt-get install proxmox-ve
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package proxmox-ve is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
proxmox-ve-2.6.32

E: Package 'proxmox-ve' has no installation candidate
proxmox1> apt-get install proxmox-ve-2.6.32
Reading package lists... Done
Building dependency tree
Reading state information... Done
proxmox-ve-2.6.32 is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

do i want apt-get --reinstall ?

 

[udo]

Hi,
how looks the poutput of following commands?

dpkg -l | egrep 'pve|proxmox'
cat /proc/sys/fs/file-max
df -h

pve3 is a long time ago...

Udo

 

[jkister]

how looks the poutput of following commands?

proxmox1> dpkg -l | egrep 'pve|proxmox'
ii  clvm                             2.02.98-pve4                     amd64        Cluster LVM Daemon for lvm2
ii  corosync-pve                     1.4.7-1                          amd64        Standards-based cluster framework (daemon and modules)
ii  dmsetup                          2:1.02.77-pve4                   amd64        Linux Kernel Device Mapper userspace library
ii  fence-agents-pve                 4.0.10-3                         amd64        fence agents for redhat cluster suite
ii  libcorosync4-pve                 1.4.7-1                          amd64        Standards-based cluster framework (libraries)
ii  libdevmapper-event1.02.1:amd64   2:1.02.77-pve4                   amd64        Linux Kernel Device Mapper event support library
ii  libdevmapper1.02.1:amd64         2:1.02.77-pve4                   amd64        Linux Kernel Device Mapper userspace library
ii  liblvm2app2.2:amd64              2.02.98-pve4                     amd64        LVM2 application library
ii  libopenais3-pve                  1.1.4-3                          amd64        Standards-based cluster framework (libraries)
ii  libpve-access-control            3.0-16                           amd64        Proxmox VE access control library
ii  libpve-common-perl               3.0-27                           all          Proxmox VE base library
ii  libpve-storage-perl              3.0-35                           all          Proxmox VE storage management library
ii  lvm2                             2.02.98-pve4                     amd64        Linux Logical Volume Manager
ii  novnc-pve                        0.5-3                            amd64        HTML5 VNC client
ii  openais-pve                      1.1.4-3                          amd64        Standards-based cluster framework (daemon and modules)
ii  proxmox-ve-2.6.32                3.4-185                          all          The Proxmox Virtual Environment
ii  pve-cluster                      3.0-20                           amd64        Cluster Infrastructure for Proxmox Virtual Environment
ii  pve-firewall                     1.0-24                           amd64        Proxmox VE Firewall
ii  pve-firmware                     1.1-6                            all          Binary firmware code for the pve-kernel
ii  pve-kernel-2.6.32-11-pve         2.6.32-66                        amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-2.6.32-12-pve         2.6.32-68                        amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-2.6.32-19-pve         2.6.32-96                        amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-2.6.32-20-pve         2.6.32-100                       amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-2.6.32-23-pve         2.6.32-109                       amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-2.6.32-26-pve         2.6.32-114                       amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-2.6.32-28-pve         2.6.32-124                       amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-2.6.32-31-pve         2.6.32-132                       amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-2.6.32-48-pve         2.6.32-185                       amd64        The Proxmox PVE Kernel Image
ii  pve-libspice-server1             0.12.4-3                         amd64        SPICE remote display system server library
ii  pve-manager                      3.4-16                           amd64        The Proxmox Virtual Environment
ii  pve-qemu-kvm                     2.2-28                           amd64        Full virtualization on x86 hardware
ii  redhat-cluster-pve               3.2.0-2                          amd64        Red Hat cluster suite
ii  resource-agents-pve              3.9.2-4                          amd64        resource agents for redhat cluster suite
ii  tar                              1.27.1+pve.1                     amd64        GNU version of the tar archiving utility
ii  vzctl                            4.0-1pve6                        amd64        OpenVZ - server virtualization solution - control tools

proxmox1> cat /proc/sys/fs/file-max
1623878

proxmox1> df -h
Filesystem               Size  Used Avail Use% Mounted on
udev                      10M     0   10M   0% /dev
tmpfs                    1.6G  384K  1.6G   1% /run
/dev/mapper/pve-root      67G  3.1G   61G   5% /
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                    3.2G  3.1M  3.1G   1% /run/shm
/dev/mapper/pve-data     170G   99G   71G  59% /var/lib/vz
/dev/cciss/c0d0p1        495M  216M  254M  47% /boot
/var/lib/vz/private/100  120G   95G   26G  79% /var/lib/vz/root/100
tmpfs                    820M   68K  820M   1% /var/lib/vz/root/100/run
tmpfs                    5.0M     0  5.0M   0% /var/lib/vz/root/100/run/lock
tmpfs                    1.8G     0  1.8G   0% /var/lib/vz/root/100/run/shm
/dev/fuse                 30M   16K   30M   1% /etc/pve

i appreciate pve3 is old, trying my best to get the box fixed and upgraded. thanks for helping.

 

[udo]

Hi,
looks ok - all installed. Looks not for trouble with an reboot.

You don't wrote, if this is san single node or part of an cluster.


Udo

 

[jkister]

looks ok - all installed. Looks not for trouble with an reboot.

You don't wrote, if this is san single node or part of an cluster.

this is just a single server, no cluster. only 1 VM guest is installed, id 100.

do you think the only way to get commands working again is a reboot? like "pvecm status" shown at the top of this thread.

 

[udo]

this is just a single server, no cluster. only 1 VM guest is installed, id 100.

do you think the only way to get commands working again is a reboot? like "pvecm status" shown at the top of this thread.

Hi,
pvecm is for cluster (ClusterManager) - you don't need that on an single node.

Looks, that your CT (100) has used all open file discriptors - so you can't open new.

An reboot is not a bad idea, because you don't use the lates kernel/software... nevertheless it's not an good idea to run an pve3.x in this time (from security view).

And if you are now afraid to reboot - do you think it's better in some month? I can say no! If you do an reboot and all work after that, you can look forward, if you have an power outage or so - because you know your system is rebootable now.
And now you know what you have done - in some month you don't remind you right - don't make it's easier.

Of course you should allways have an valid backup of your CT.

Udo

 

[jkister]

Of course you should allways have an valid backup of your CT.

thank you for teaching me about pvecm. one less thing to worry about.

I would love to get a good backup. but I'm not sure how with the webgui broken.

i tried

proxmox1> vzdump 100 --mode suspend
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
can't use storage for backups - wrong content type

proxmox1> vzdump 100 --dumpdir /var/lib/vz/dump --mode suspend
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
ipcc_send_rec failed: Resource temporarily unavailable
INFO: starting new backup job: vzdump 100 --dumpdir /var/lib/vz/dump --mode suspend
ERROR: Backup of VM 100 failed - unable to find VM '100'
INFO: Backup job finished with errors
job errors
ipcc_send_rec failed: Resource temporarily unavailable

of note,

proxmox1> vzlist
      CTID      NPROC STATUS    IP_ADDR         HOSTNAME
       100        199 running   -               s1.kister.net

 

[udo]

Hmm,
perhaps first an reboot?

Do an backup of /etc/pve like "tar cvf /root/etc_pve.tar /etc/pve" before.

If you have realy trouble you can get your config files and the ct-files from the file-system


Udo