Server hacked and now Perl is broken... how can I recover?

[jh207]

This is fairly obscure server not part of any patch cycle, and I assume based on the symptoms Perl/Dashboard was exploited finally after years of being online. (Dashboard the only service that was open on it)

- After receiving alerts that some services went down, I was not able to access the dashboard due to a bad login.
- I accessed the server via console and reset the root password, but restarting the PVE services following this failed.
- In trying to get my VM back online, I noticed even `qm` was having the Perl issue.
- Attempts to reinstall/upgrade Perl seem to want to remove Proxmox which i'm afraid to commit to.

I have 2 applications on it that weren't critical or high profile enough to patch the hypervisor, but I do care about recovering their state. One is a container, the other a VM.

I would attach the container/vm in console to retrieve what I need and rebuild clean, but it seems even the CLI tools are broken in depending on Perl.

Can anyone advise on how I can access these systems from shell or bring services back up safely?




PVE services cite "Compilation failed at require" on these start failures.

 

[mira]

Please provide the output of pveversion -v, grep '' /etc/apt/sources.list and grep '' /etc/apt/sources.list.d/*

 

[jh207]

proxmox-ve: 6.4-1 (running kernel: 5.4.203-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.4-20
pve-kernel-helper: 6.4-20
pve-kernel-5.3: 6.1-6
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.4-8
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-1-pve: 5.3.18-1
pve-kernel-5.3.13-3-pve: 5.3.13-3
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-5.0.18-1-pve: 5.0.18-3
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 9.2-1
ifupdown: 0.8.36
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.6-2
lxcfs: 4.0.7-1
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-2
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.2-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve1

/etc/apt/sources.list:deb http://ftp.debian.org/debian bullseye main contrib
/etc/apt/sources.list:deb http://ftp.debian.org/debian bullseye-updates main contrib
/etc/apt/sources.list:
/etc/apt/sources.list:# security updates
/etc/apt/sources.list:deb http://security.debian.org/debian-security bullseye-security main contrib
grep: and: No such file or directory
grep: grep: No such file or directory
grep: : No such file or directory
/etc/apt/sources.list.d/corosync3.list:deb http://download.proxmox.com/debian/corosync-3/ stretch main
/etc/apt/sources.list.d/pve-enterprise.list:#deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise
/etc/apt/sources.list.d/pve-enterprise.list.dpkg-dist:deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise
/etc/apt/sources.list.d/pve-no-subscription.list:deb http://download.proxmox.com/debian buster pve-no-subscription
/etc/apt/sources.list.d/raid.list:# RAID Management Tools
/etc/apt/sources.list.d/raid.list:deb http://hwraid.le-vert.net/debian buster main

 

[jh207]

apt dist-upgrade warns that it'll remove pve which has been my hesitation to try that

 

[mira]

The issue is that you've configured conflicting repositories.
For Debian you have `bullseye` configured, while for PVE you have `buster`.

Were you planning on upgrading to PVE 7.0? If so, then please follow our upgrade guide: https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0

The repository configured in `raid.list` might conflict if you plan on upgrading to bullseye, unless it is available for bullseye as well.