How to mount ZFS Install on Ubuntu Live to access Proxmox install files?

B

boethius

Well-Known Member
Feb 11, 2016
36
3
48
41
Hi,

I am troubleshooting a strange issue where I have lost root access to Proxmox. The WebUI would load, but my root login was rejected. SSH also doesn't work, an nmap scan now shows "TCPWrapped" as the service instead of SSH.

I also could not login with root at the console with a keyboard connected to the server. It would not allow for a password to be entered, simply saying root was an invalid username.

So I have booted a Ubuntu Live iso in an attempt to access the installation zpool, which is a mirror of 2x 14GB SataDOM.
I imported the zpool "rpool" which contains the directories ROOT and data. /rpool/ROOT/pve-1 is empty, as is /rpool/data.

ZFS status shows it is mounted, and shows it containing 12.9G of data, but nothing is visible within the directories.

What stupid error have I made, and how can I properly mount the pool to access the filesystem?
 
Now, the system will not boot into Ubuntu Live. I don't know what's going on with it, I haven't changed anything. It hangs indefinitely (ie. at least 6 hours) on the Ubuntu splash screen. I saw a very brief message (only captured as I was videotaping the bootup) "
Code: 
usbhid 3-11.4:1.2: couldn't find an input interrupt endpoint
" after selecting Try Ubuntu or Install at grub.
 
the symptoms kind of sound like your system was compromised and taken over by some kind of malware.. do you have physical access?
 
fabian said:
the symptoms kind of sound like your system was compromised and taken over by some kind of malware.. do you have physical access?
Click to expand...
I do have physical access.
Strange behavior I can't help but mention:
1. The monitor goes partially "technicolor" during boot (ie in BIOS). It is not a faulty cable, nor monitor. Once Ubuntu Live loaded, everything was normal, and it was during distinct phases of the bootup process where this visual corruption occurred.
2. The "patterning" of the "technicolor" was consistent for the ~3 reboots I did while Ubuntu Live would actually reach desktop. Now, the patterning of the technicolor output is different, much more subtle (horizontal striping vs an amorphous blob), and the OS never loads. The consistency of the visual artifacts and the pattern are hard to ignore and I'm not sure what to make of it.

It would seem to be an awfully advanced malware for it to reach the BIOS and be responsible for a complete lockout with me having physical access.
 
could also be your regular install being compromised, and your hardware requiring special tweaks to work with this ubuntu live version..
 
fabian said:
could also be your regular install being compromised, and your hardware requiring special tweaks to work with this ubuntu live version..
Click to expand...
Here's an update. I tried to boot with another USB drive in the same physical port of the machine.
The original USB of Ubuntu Live contained a persistence volume. The second USB drive was created with Rufus (as was the first) with the same Ubuntu ISO and settings as the other (GPT, UEFI), without a persistence partition.

This second USB drive booted differently. For instance, I was shown:
IMG_0410.jpg


I did not remove the media, instead I hit enter and it proceeded to boot showing the various ROMs being loaded such as my HBA, etc.
I landed at a working Ubuntu Live instance. I opened a terminal and did zpool list and saw no pools, as compared to earlier when all 3 pools were recognized and reported.

I then tried AGAIN with the initial USB drive and AGAIN the bootup has changed dramatically. The visual distortions are more extreme and there are not any distinguishable characters for the first minute of the bootup process, save for a brief flash of the Supermicro logo. Then, the keyboard I had plugged started to approximate the psychedelic affect of the visuals on the monitor and would not work to invoke any option (e.g. Press F11 for Boot Menu, F12 for PXE, DEL to enter setup, etc). I got another keyboard and plugged it in and it too would not work. The mobo's speaker beeped when it was connected and disconnected, and it briefly illuminated the LEDs for CAPS lock, NUM, SCROLL LOCK, but then nothing.

I am literally doing the same things and getting wildly different results. Is that the key to *insanity*?
 
well, it might be the point where it makes sense to get somebody else involved (potentially somebody that has experience trouble shooting complex issues like that). it might also just be your system having some sort of hardware meltdown ;)
 
fabian said:
well, it might be the point where it makes sense to get somebody else involved (potentially somebody that has experience trouble shooting complex issues like that). it might also just be your system having some sort of hardware meltdown ;)
Click to expand...
Please don't chalk this up as weird and unfixable! I know it is bizarre. But my hardware is not inherently janky!

The system is a Supermicro X10SRL-F w/ Xeon E5-2618L v3.

I had Proxmox running ZFS on root, 3 zpools:
rpool (ZFS mirror of 2x16GB SataDOM's)
ezstor (ZFS mirror of 8TB SATA)
tank01 (RaidZ1. 4x 4TB SATA connected via onboard SAS HBA.)

No disks had any signs of impending failure on SMART reports.

There is also a FusionIO 1.3TB PCIe which my VM's are loaded onto.
An Nvidia 1070 passed through to a Windows 10 guest.
Intel X520-SR2 NIC
And an Intel WiFi NIC 6250AGN passed through to a Ubuntu guest running Kismet.

If I had someone who was able to help, I wouldn't be presenting this quagmire to the forum. You know how they say that insanity is doing the same thing expecting different results? I am doing the same thing and getting wildly different results. Its weird, I know it sounds like I am doing something wrong but I am sure I at least know how to boot into Ubuntu Live, and don't have short term memory loss regarding any actions I've taken that would explain the machine seeming to be possessed.

BUT at least now (without any change on my part), the boot process is now normal, no technicolor. I am able to get to Ubuntu Live. Contrary to previous attempt (where I only had imported rpool, with all pools showing as healthy), now no pools are displayed when I run
Code: 
zpool list

GParted recognizes all the drives and displays them as having ZFS partitions.

Any ideas? I know this is not even strictly related to Proxmox at this point, but if you have a thought about next steps... I'd appreciate it a lot. :confused:
 
Last edited:
Nemesiz said:
If zpool list show empty did you tried zpool import ?
Click to expand...
Ok, I ran zpool import and then zfs status, which showed my 3 ZFS pools. Excellent.

I then run zpool import rpool and it is automounted at /rpool.

If I cd to /rpool, there is /rpool/PVE and /rpool/DATA, which are empty.

I was told I needed to mount the pool in a non-root directory, but when I issue umount /rpool it states target as busy.

Thank you for your advice I know I am just misunderstanding zfs concepts, I have tried to review them but something is not sticking as it relates to this process of trying to examine my PVE data from a LIVE USB of Ubuntu.
 
Mounting in Live CD you can use -R : zpool import pool -R /another/mount/point
So you could do what you need without mixing with Live CD / mountpoint

Or -N : Import the pool without mounting any file systems.

p.s. it is OK for /rpool/PVE and /rpool/DATA to be empty
 
Nemesiz said:
Mounting in Live CD you can use -R : zpool import pool -R /another/mount/point
So you could do what you need without mixing with Live CD / mountpoint

Or -N : Import the pool without mounting any file systems.

p.s. it is OK for /rpool/PVE and /rpool/DATA to be empty
Click to expand...

I don't know what you mean by the p.s.

I am trying to investigate why my Proxmox install was apparently compromised. I expect to see directories and files, not empty folders. How is it OK that they are inaccessible? At this point, I'm about to give up and just reinstall Proxmox and try to recover the other ZFS pools that have my actual data.
 
boethius said:
I don't know what you mean by the p.s.

I am trying to investigate why my Proxmox install was apparently compromised. I expect to see directories and files, not empty folders. How is it OK that they are inaccessible? At this point, I'm about to give up and just reinstall Proxmox and try to recover the other ZFS pools that have my actual data.
Click to expand...

  • rpool/data gets by default mounted at: /rpool/data and is the local-zfs storage for guest vDisks (ZVOLs for VMs and datasets for LXCs). You should see the vDisks with: zfs list, if you have stored any there at all.
  • rpool/ROOT/pve-1 gets by default mounted as the root: /
  • /etc/pve is not, what you might think it is: https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)

For completeness: rpool is mounted at: /rpool and rpool/ROOT at: /rpool/ROOT
 
I was able to mount rpool at /mnt/rpool and inspect the contents.

I found two scripts that I did not add, with modification date of around the time I lost access to the system. Logs were missing for after this date. The two scripts were https://github.com/iovisor/bcc/blob/master/tools/bpflist.py (without modification) and https://github.com/iovisor/bcc/blob/master/tools/tcptracer.py (with some customization).

Needless to say I am going to wipe and reinstall the latest version of Proxmox. However, if anyone has the expertise, I'd love to hear an explanation and/or further steps to investigate before I wipe these SataDOM's and reinstall. I'm worried there could be a persistence mechanism in the other zpools.
 
Last edited:
Nemesiz said:
It should make no problem to install but If you are struggling you can download directly - https://packages.debian.org/bookworm/all/debsums/download
Click to expand...
I downloaded from source and then installed 'build-essential' and the system crashed... I was on a console and it dropped me into the GUI after hanging. I think it installed successfully the second time running it. Can it be used to verify packages on another system, though? Remember, I'm in a Live distro of Ubuntu with the root directory of my Proxmox install. Can I restore root access to Proxmox from this configuration? Is that even desirable given the findings thus far (likely the existence of an advanced eBPF rootkit)?

Also, I was unable to compile debsums from source. I don't know if that is due to my massive gaps in linux knowledge, or something else...

I did
Code: 
wget http://deb.debian.org/debian/pool/main/d/debsums/debsums_2.2.3.tar.xz
apt update
apt install build-essential
---crash---
apt install build-essential #(no error)
tar -xf debsums_2.2.3
cd debsums_2.2.3
./configure
make
make install[/ICODE]
Which failed to install Debsums.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back