[SOLVED] Massive issues after cloned installation (ddrescue)

[keffo]

Hi all!

I've been trying to fix my installation the last few hours after "successfully" managed to clone my installation from an old disk to a new one. Good to know is that the device cloning from is a dying disk. All VM/CTS running just fine, but can't make backups or move storage due to I/O errors.

All good, clone done, system is booting up and disk looking good, but none of my PVE* services will start.

root@scumm-pve-01:~# systemctl --failed
  UNIT                 LOAD   ACTIVE SUB    DESCRIPTION
● pve-cluster.service  loaded failed failed The Proxmox VE cluster filesystem
● pve-firewall.service loaded failed failed Proxmox VE firewall
● pve-guests.service   loaded failed failed PVE guests
● pve-ha-crm.service   loaded failed failed PVE Cluster HA Resource Manager Daemon
● pve-ha-lrm.service   loaded failed failed PVE Local HA Resource Manager Daemon
● pvedaemon.service    loaded failed failed PVE API Daemon
● pveproxy.service     loaded failed failed PVE API Proxy Server
● pvescheduler.service loaded failed failed Proxmox VE scheduler
● pvestatd.service     loaded failed failed PVE Status Daemon
● rrdcached.service    loaded failed failed LSB: start or stop rrdcached

I managed to re-install my packages, since pve-manager failed to install correctly, for some reason.

proxmox-ve: 8.1.0 (running kernel: 6.5.13-3-pve)pve-manager: not correctly installed (running version: 8.1.10/4b06efb5db453f29)

At this stage I put my old disk back into my homeserver, booted it up, to see if everything was running fine. This to make sure nothing "happened" during the clone last night. As suspected, everything fired up as it should.

I'm running out of ideas honestly, and Google is of no use anymore.

When trying to start pvedaemon and looking into syslog this is what I get. I can see hints of perl issues.

2024-04-04T12:58:17.328791+02:00 scumm-pve-01 systemd[1]: Failed to start pvedaemon.service - PVE API Daemon.
2024-04-04T12:58:17.343317+02:00 scumm-pve-01 systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
2024-04-04T12:58:17.616285+02:00 scumm-pve-01 pvecm[4607]: ipcc_send_rec[1] failed: Connection refused
2024-04-04T12:58:17.616411+02:00 scumm-pve-01 pvecm[4607]: ipcc_send_rec[2] failed: Connection refused
2024-04-04T12:58:17.616472+02:00 scumm-pve-01 pvecm[4607]: ipcc_send_rec[3] failed: Connection refused
2024-04-04T12:58:17.616503+02:00 scumm-pve-01 pvecm[4607]: Unable to load access control list: Connection refused
2024-04-04T12:58:18.187792+02:00 scumm-pve-01 pveproxy[4608]: Can't load '/usr/lib/x86_64-linux-gnu/perl5/5.36/auto/RRDs/RRDs.so' for module RRDs: /lib/x86_64-linux-gnu/libpixman-1.so.0: invalid ELF header at /usr/lib/x86_64-linux-gnu/perl-base/DynaLoader.pm line 201.
2024-04-04T12:58:18.187883+02:00 scumm-pve-01 pveproxy[4608]:  at /usr/share/perl5/PVE/RRD.pm line 5.
2024-04-04T12:58:18.187929+02:00 scumm-pve-01 pveproxy[4608]: Compilation failed in require at /usr/share/perl5/PVE/RRD.pm line 5.
2024-04-04T12:58:18.187958+02:00 scumm-pve-01 pveproxy[4608]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/RRD.pm line 5.
2024-04-04T12:58:18.187973+02:00 scumm-pve-01 pveproxy[4608]: Compilation failed in require at /usr/share/perl5/PVE/API2/Qemu.pm line 18.
2024-04-04T12:58:18.187989+02:00 scumm-pve-01 pveproxy[4608]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Qemu.pm line 18.
2024-04-04T12:58:18.188018+02:00 scumm-pve-01 pveproxy[4608]: Compilation failed in require at /usr/share/perl5/PVE/HA/Resources/PVEVM.pm line 18.
2024-04-04T12:58:18.188032+02:00 scumm-pve-01 pveproxy[4608]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Resources/PVEVM.pm line 21.
2024-04-04T12:58:18.188047+02:00 scumm-pve-01 pveproxy[4608]: Compilation failed in require at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 23.
2024-04-04T12:58:18.188062+02:00 scumm-pve-01 pveproxy[4608]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/HA/Env/PVE2.pm line 23.
2024-04-04T12:58:18.188096+02:00 scumm-pve-01 pveproxy[4608]: Compilation failed in require at /usr/share/perl5/PVE/API2/Cluster.pm line 15.
2024-04-04T12:58:18.188126+02:00 scumm-pve-01 pveproxy[4608]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2/Cluster.pm line 15.
2024-04-04T12:58:18.188155+02:00 scumm-pve-01 pveproxy[4608]: Compilation failed in require at /usr/share/perl5/PVE/API2.pm line 15.
2024-04-04T12:58:18.188198+02:00 scumm-pve-01 pveproxy[4608]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/API2.pm line 15.
2024-04-04T12:58:18.188212+02:00 scumm-pve-01 pveproxy[4608]: Compilation failed in require at /usr/share/perl5/PVE/Service/pveproxy.pm line 13.
2024-04-04T12:58:18.188254+02:00 scumm-pve-01 pveproxy[4608]: BEGIN failed--compilation aborted at /usr/share/perl5/PVE/Service/pveproxy.pm line 13.
2024-04-04T12:58:18.188292+02:00 scumm-pve-01 pveproxy[4608]: Compilation failed in require at /usr/bin/pveproxy line 11.
2024-04-04T12:58:18.188322+02:00 scumm-pve-01 pveproxy[4608]: BEGIN failed--compilation aborted at /usr/bin/pveproxy line 11.
2024-04-04T12:58:18.197940+02:00 scumm-pve-01 systemd[1]: pveproxy.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
2024-04-04T12:58:18.198157+02:00 scumm-pve-01 systemd[1]: pveproxy.service: Failed with result 'exit-code'.

I did run debsums -s earlier and the only two missing / problematic files were associated to perl. But not the version we see here, but 5.32.
I removed the package, and reinstalled 5.36 - just to be sure.

As I said, I'm running out of ideas here guys. My goal was always to clone my installation to my new disk, and after that be able to backup my VM/CT's and the make a clean Proxmox install.

Thanks in advance!

 

[keffo]

Update and solved.

I re-ran ddrescue from a LIVE-USB instead from the host, cloned /dev/sda (old) to /dev/sdb. Took roughly 4.5 hours for 250gb-ish.

ddrescue -d -f -r3 /dev/sda /dev/sdb rescue.logfile

In retrospect, I realize my mistake in not using a LIVE CD from the start.