[SOLVED] My install has destroyed Perl and prossibly Python

[PairedPrototype]

I checked on my server today and could not access the web GUI which sparked concern. I SSHed in and tried to run sudo pveproxy restart but was greeted with some error similar to this (this specific error was pulled from the syslog)

Spoiler

BEGIN failed--compilation aborted at /usr/bin/pveproxy line 11.
Compilation failed in require at /usr/bin/pveproxy line 11.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pveproxy.pm line 16.
Compilation failed in require at /usr/share/perl5/PVE/Service/pveproxy.pm line 16.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 14.
Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 14.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Cluster.pm line 28.
Compilation failed in require at /usr/share/perl5/PVE/API2/Cluster.pm line 28.
BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/ClusterConfig.pm line 13.
Compilation failed in require at /usr/share/perl5/PVE/API2/ClusterConfig.pm line 13.
Unrecognized character \x84; marked by <-- HERE after <-- HERE near column 1 at /usr/share/perl5/PVE/APIClient/LWP.pm line 1.

I checked out the file that looked to be the issue /usr/share/perl5/PVE/APIClient/LWP.pm and it turned out it was garbled as if it were corrupted, file /usr/share/perl5/PVE/APIClient/LWP.pm reported is as data rather than the expected Perl5 module source, ASCII text. All attempts to get apt/dpkg to reinstall it was failing. I ended up rsyncing the directory from a known good install of Proxmox 6 (I don't think the minor version match up exactly, but it was a desperate attempt to get it working). That seemed to fix errors I was seeing in apt at least.

But now I have this issue where apt was complaining about dpkg list files not ending with a newline, a bit of Googling suggested I deleted those files, instead, I just renamed them, just in case I need them. It turns the list file also looked corrupt, and 9 other list files were too, so I renamed them all. Which means many apt operations give this warning:

Spoiler

dpkg: warning: files list file for package 'python3-setuptools' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libcpg4:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'python3-zope.event' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'python3-pbr' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libquorum5:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'python3-zope.component' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libnl-route-3-200:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libpve-apiclient-perl' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libknet1:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libsndfile1:amd64' missing; assuming package has no files currently installed
dpkg: warning: files list file for package 'libvorbisenc2:amd64' missing; assuming package has no files currently installed

I've since been trying to reinstall these packages, but all attempts are failing. I've tried:

• sudo apt-get clean
• sudo apt reinstall python3-setuptools (pastebin)
• sudo dpkg-reconfigure python3-setuptools (pastebin)
• sudo dpkg --configure -a (pastebin)
• sudo apt --fix-broken install (patebin)
• (I downloaded the deb) sudo dpkg -i python3-setuptools_40.8.0-1_all.deb (pastebin)
• (I downloaded the deb) sudo dpkg -i python3_3.7.3-1_amd64.deb (pastebin)
• sudo dpkg --remove --force-remove-reinstreq python3-setuptools (pastebin)
• sudo apt-get --reinstall install -y -f python3-setuptools (pastebin)

Most things seem to point back to a broken python install but running a quick print line in Python2 and Python3 seems to work at least.
I'm really not how I can fix this. The sys logs have been plagued with errors relating to perl and python for a few days. My containers and VMs seem to be running fine at the moment which is why I didn't notice.

If I can avoid having to reinstall, I'd like to since I don't want to have to rsync 200GB+ of data over a 20/Mbps connection. I can have my containers and VMs offline if needed, I just don't want to have to move that much data if possible.

I'd greatly appreciate any help I can get to resolve this mess.
The worst thing is, I don't even know what caused this

TL;DR Somehow I think python and perl are screwed (corrupted?) and dpkg won't let me reinstall them.

EDIT:
I've managed to fix the dpkg list file issues by installing the same packages on a local good install of Proxmox 6 and rsyncing them to the broken remote install. However running a simple apt upgrade yeilds an error relating to the ifupdown2 package.

 

[t.lamprecht]

I've managed to fix the dpkg list file issues by installing the same packages on a local good install of Proxmox 6 and rsyncing them to the broken remote install. However running a simple apt upgrade yeilds an error relating to the ifupdown2 package.

Never do apt upgrade, always use apt full-upgrade

Traceback (most recent call last):
  File "/usr/bin/py3compile", line 34, in <module>
    from debpython.version import SUPPORTED, debsorted, vrepr, \
  File "/usr/share/python3/debpython/version.py", line 46, in <module>
    _config.read('/usr/share/python3/debian_defaults')
  File "/usr/lib/python3.7/configparser.py", line 696, in read
    self._read(fp, filename)
  File "/usr/lib/python3.7/configparser.py", line 1014, in _read
    for lineno, line in enumerate(fp, start=1):
  File "/usr/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x95 in position 1: invalid start byte

Seems like some files are still scrambled, not valid unicode could mean that there's garbage in that file.


So, files don't scramble themself, this could be data corruption by faulty disks or some power outage or what not.

Did you run a fsck and checked the disks (for example with smartctl)

 

[PairedPrototype]

Never do apt upgrade, always use apt full-upgrade

Noted. I'm still trying to remember to transition from apt-get lol.

Seems like some files are still scrambled, not valid unicode could mean that there's garbage in that file.

Yeah, I think so, I've been running a checksum on the files that I'm seeing in the apt logs and replacing the ones that don't match up with my local Proxmox install.

So, files don't scramble themself, this could be data corruption by faulty disks or some power outage or what not.

Did you run a fsck and checked the disks (for example with smartctl)

The server has an uptime of 16 days which is what I expected, so I don't think power outage was an issue.

I have already run smartctl on both NVMe drives (they're in soft raid 1 setup by OVH) and the SMART test reports a PASS for both drives.

I haven't run fsck yet though, as far as I know, that should only ever be run on unmounted filesystems, so I guess I'll have to reboot the server into rescue mode to do that, but I fear what the repercussions might be of rebooting it. All my containers and VMs are backed up on the OVH remote backup storage, so that should be fine but I'll have to clear some old backups on there first to make space and then I should be able to rsync that 200GB of data to it that I don't want to lose. So this might take a while, but I'll report back with my findings.

Thanks for your reply!

Edit:
I checked the raid array as well and all seems fine there, from what I can tell (pastebin).

 

[fabian]

you can use debsums to quickly check all files directly shipped by packages (note, generated files including some config files will not be checked as dpkg does not have checksums for those).

 

[PairedPrototype]

you can use debsums to quickly check all files directly shipped by packages (note, generated files including some config files will not be checked as dpkg does not have checksums for those).

I guess perl is still broken, debsums breaks

Spoiler

$ sudo debsums
Can't locate Dpkg/Conf.pm in @INC (you may need to install the Dpkg::Conf module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.28.1 /usr/local/share/perl/5.28.1 /usr/lib/x86_64-linux-gnu/perl5/5.28 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.28 /usr/share/perl/5.28 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/bin/debsums line 22.
BEGIN failed--compilation aborted at /usr/bin/debsums line 22.

 

[fabian]

that module is in libdpkg-perl

 

[PairedPrototype]

that module is in libdpkg-perl

dpkg seems to think that's already installed

$ sudo dpkg -l | grep libdpkg-perl
ii  libdpkg-perl                         1.19.7                          all          Dpkg perl modules

 

[fabian]

sounds like it is corrupt/missing? it's in /usr/share/perl5/Dpkg/Conf.pm

 

[PairedPrototype]

sounds like it is corrupt/missing? it's in /usr/share/perl5/Dpkg/Conf.pm

Looks like it was missing again, /usr/share/perl5/Dpkg.pm was also missing. I rsynced them from a known good machine and managed to run debsums and I saw a lot of fails in there when I ran it without the silent flag

Here's a random sample from the generated file $ sudo debsums -s 2> debsums.failures

debsums: invalid line (1) in md5sums for libproxmox-backup-qemu0: ia/focal/p/pi/pinta.desktop/3F732CD49EF8EB8E680C6A79A84B54C9/icons/128x128/pinta_pinta.png^@koÅ<82>u<90>^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@<80>^LJ<8d><97>U^@^@ <8c><89>G/^?^@^@>Pakiet q^A^@^@^@^@^@^@ð    J<8d><97>U^@^@ <8c><89>G/^?^@^@tegruje ±^@^@^@^@^@^@^@p^QJ<8d><97>U^@^@ <8c><89>G/^?^@^@adamiajÄ<91>^@^@^@^@^@^@^@Ð^OJ<8d><97>U^@^@ð¯*<8f><97>U^@^@Ä<85> okna wyskakujÄ<85>ce
debsums: changed file /usr/share/perl5/PVE/API2/AccessControl.pm (from libpve-access-control package)
debsums: changed file /usr/share/perl5/PVE/API2/User.pm (from libpve-access-control package)
debsums: changed file /usr/share/perl5/PVE/AccessControl.pm (from libpve-access-control package)
debsums: changed file /usr/share/perl5/PVE/Auth/AD.pm (from libpve-access-control package)
debsums: changed file /usr/share/perl5/PVE/Auth/LDAP.pm (from libpve-access-control package)
debsums: invalid line (1) in md5sums for libpve-apiclient-perl: J<8c>eG¹<9e>Þ÷;<89>¯·ø{}yMðà£<9c>r4ú8UÔÈÑ ^UÒWÛÜ45<97>ô^LöÎ<81>^[^Y{®^^^Cn<95>TH^@í*B   |^sd{¢ÃG^SÝ^U!èÑA34<8b>¤ã^C;ëÜè A8<83>¦=òqÀ'®t?ÛüÅ+ae2²x<90><98>|^PVr0^M^R<93>o²Æ^BÔ³ÑúnüCÌ<96>kÇÓ
debsums: invalid line (2) in md5sums for libpve-apiclient-perl: ¸^Â,<8d>+\·ÀÛ<9c>QßðvâL^G^G<93><90>2<9a><8d>(Â_^[¥ûo<9f>ýý÷ÞñóÍ<9f>^HT^]UÍI,<89>FÆ<9e>^L=ìÉfûÄÉ®^BÀ<84>ï`O<9e>^P^@Ôí#äúJ^?<8c>Jwz<83>Ê-G^Cz ^@¼Tô{Q^@À^YPeÇhà^L~¯å×éä?^Vï^T^D0á;Ø<93>'^D<90>^A@Áq^^^MéáÒ<8a>D,Þ^\ïà8^Eg %v%^S1<93>^Q^ÄZÞ¯^QhÂéòt<84><85><9d>ñ{^Vü^DìÉ^K~OÁ^EY^QÀ<86>ï`ÏÝ"ë° 8NÁ^Y0wl^S<9d>¬g^W<95>ïß<91>Çö^Sk<86>^C^@Î`ò^^5éø^Lò<84>^Uä3,È<8c>»À<95>k¹h±ê¼¼G<9c>»äiò
debsums: invalid line (3) in md5sums for libpve-apiclient-perl: <9f>¬^@^SÐTwYs<82>=JÿZÍ ö¨0·^Hm^EhL¡¼<87>%E<99>Û<89>}¦-^M7<93>+k^P<85>z<99>^OʶÎGe$å<9b>åXh='^Y<87>Xågï<8f>j    WÂ0<9c>s<9a>J<84><8a>S¯^\^^v^D<96>ÿ=Ij^Wú?©^ON(TêH¨^FMÏ[/^E<88>ùÿOýl ë?.q~¶Ag^[gWýÕ®5ª%Ç<80>~C^V^Gjy§e&ºB<90>wV)¤µófmò^X^M¤EdwÙúe<8b>ÇH»g^Yµ¥ÔM^]åüè®j.^YGk¤ót^H^Z^?éY+fWTóêGuKX#¨÷ôdIAw^A°òmÔm<94>ç^K/3<88>^Lì»UæÐ^@fn<88>®ð¤<82>Ä&^K¿Ëc^ST¥®øíiÊ°O<83>¦¨<91>~>uK¶º±û¹³êÊÒP´A«ÝYue^FmÕ^H¸±:^@-oî^@V|
debsums: changed file /usr/share/perl5/PVE/API2/ClusterConfig.pm (from libpve-cluster-api-perl package)
debsums: changed file /usr/share/perl5/PVE/CLI/pvecm.pm (from libpve-cluster-api-perl package)
debsums: changed file /usr/share/perl5/PVE/AbstractConfig.pm (from libpve-guest-common-perl package)

There are more lines of printed garbage like that. In total there are 1219 lines in that file, so I'm guessing that's potentially how many packages are corrupt?

 

[fabian]

that's looking like you should check your hardware (especially memory and anything disk related) and re-install afterwards, since you can't determine for all files whether they are correct or not (and some files might be rarely accessed).

e.g. this line alone:

debsums: invalid line (1) in md5sums for libproxmox-backup-qemu0: ia/focal/p/pi/pinta.desktop/3F732CD49EF8EB8E680C6A79A84B54C9/icons/128x128/pinta_pinta.png^@koÅ<82>u<90>^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@<80>^LJ<8d><97>U^@^@ <8c><89>G/^?^@^@>Pakiet q^A^@^@^@^@^@^@ð    J<8d><97>U^@^@ <8c><89>G/^?^@^@tegruje ±^@^@^@^@^@^@^@p^QJ<8d><97>U^@^@ <8c><89>G/^?^@^@adamiajÄ<91>^@^@^@^@^@^@^@Ð^OJ<8d><97>U^@^@ð¯*<8f><97>U^@^@Ä<85> okna wyskakujÄ<85>ce

looks like random content from some other file was written to the md5sums file for libproxmox-backup-qemu0, which is a very bad sign as that can't happen on just a simple 'botched upgrade'..

 

[PairedPrototype]

that's looking like you should check your hardware (especially memory and anything disk related) and re-install afterwards, since you can't determine for all files whether they are correct or not (and some files might be rarely accessed).

Okay, so a re-install is the best course of action at this point. I'll test the hardware as much as I can, but I think this might be something I'll have to delegate to OVH since it's a rented dedicated server from them.

Thanks so much for the help!