The TL;DR
I'd apperciate any comments on the below notes for capturing forensic data. I've highlighted a few specific questions, but any other notes, comments, corrections, or insight would be welcome.
Background
I work in the digital forensics field and, given Proxmox's recent jump in popularity, am attempting to find the easiest and most forensically sound method of obtaining the disk and memory for a system. In some ways this can be fairly straight forward and I have found various ways to get both. However, there can also be a lot of nuance to capture these correctly. For instance, in ESXi, you want to snapshot a system first to grab memory and -not- suspend it, as suspending the VM will break network connections that could be forensically relevant. You also need to grab both the VMEM file and the VMSN file from the same snapshot. So some methods work, but won't give you all the data. I don't know if that's the case here as I'm not nearly as framiliar with Proxmox and there's less of this kind of documentation. So I wanted to provide what I've found so far and see if the community add anything to it. Especially regarding something that might look correct, but could result in the loss of forensic data. I'd also be interested in any other alternatives, even if they seem silly or inconvenient. When an environment is completely broken by ransomware, sometimes it helps to know a really odd way to do something. I'm also particularly interested in anything from my notes/lab that doesn't match what would be in an enterprise production environment.
Memory
Disk Images
I'd apperciate any comments on the below notes for capturing forensic data. I've highlighted a few specific questions, but any other notes, comments, corrections, or insight would be welcome.
Background
I work in the digital forensics field and, given Proxmox's recent jump in popularity, am attempting to find the easiest and most forensically sound method of obtaining the disk and memory for a system. In some ways this can be fairly straight forward and I have found various ways to get both. However, there can also be a lot of nuance to capture these correctly. For instance, in ESXi, you want to snapshot a system first to grab memory and -not- suspend it, as suspending the VM will break network connections that could be forensically relevant. You also need to grab both the VMEM file and the VMSN file from the same snapshot. So some methods work, but won't give you all the data. I don't know if that's the case here as I'm not nearly as framiliar with Proxmox and there's less of this kind of documentation. So I wanted to provide what I've found so far and see if the community add anything to it. Especially regarding something that might look correct, but could result in the loss of forensic data. I'd also be interested in any other alternatives, even if they seem silly or inconvenient. When an environment is completely broken by ransomware, sometimes it helps to know a really odd way to do something. I'm also particularly interested in anything from my notes/lab that doesn't match what would be in an enterprise production environment.
Memory
- Memory can be extracted from both running and paused systems on the command line using the following steps:
- "qm monitor [ID]" e.g. qm monitor 101
- "dump-guest-memory -d /tmp/[dumpname].mem" e.g. dump-guest-memory -d /tmp/101-running.mem
- "help dump-guest-memory" to get options, "quit" to exit
- Both running and paused systems seem to provide the same physical size file output, both seem to contain network connections. I don't see a significant difference. Does anyone know if pausing helps in any way, such as to prvent memory smearing? Or if there's a downside such as some other artifact being lost?
- There is also the hibernante option. This results in a [VMID]-state-suspended-[DATE].raw file being written in /dev/pve/. The file is a QCOW suspended disk state file (QEVM header) that contains the volatile memory as well as other data, but cannot be processed by Volatility (not sure about other tools) as is.
- The memory can be extracted with vol3's LaywerWriter plugin.
- Get a list of layers with "vol -f vm-101-state-suspend-2024-04-08.raw layerwriter.LayerWriter --list"
- Extract the memory layer with "vol -f vm-101-state-suspend-2024-04-08.raw layerwriter.LayerWriter --layers memory_layer"
- This resulted in a "memory_layer.raw" file that was appx 2GB larger than the allocated memory (6 GB vs 4.2 GB with dump-guest-memory, host had 4GB assigned)
- The "memory_layer.raw" file was appx 2GB larger than the allocated memory (6 GB vs 4.2 GB with dump-guest-memory, host had 4GB assigned), but did seem to be parsable and have the same network connections.
- Given the larger file size and the need to extract the memory layer, I don't see an immediate benifit to using hibernate. Does anyone know of a reason why this might be prefered? Something additional in the memory layer? Something extra in the base_layer?
- The memory can be extracted with vol3's LaywerWriter plugin.
Disk Images
- There's no way to export these to your local system (e.g. right click -> download disk) that I've seen in the web interface, is this correct?
- Disks are probably (depending on setup and configuration) stored as block devices, not actual files on the file system (reference: https://forum.proxmox.com/threads/vm-disk-location-cant-find-the-vm-disk.101118/#post-436322).
- However, the disk can be written to a file. Find the primary disk name with "qm config [VMID]" e.g. qm config 101 | grep disk
- Disks should be in /dev/pve/, but the find command can be used to locate it once you know the name. Would this be the typical location in an enterprise environment?
- Copy the blockdevice to a physical file: dd if=/dev/pve/vm-101-disk-1 of=/tmp/vm-101-disk-1.raw
- This appears to work just fine while the VM is still running. Is there any downside? Would it be better to pause or hibernate the VM?
- There may be exceptions, such as if a NAS is used as storage, there is a .qcow2 (or file type of choice) that could be easily obtained.
- However, the disk can be written to a file. Find the primary disk name with "qm config [VMID]" e.g. qm config 101 | grep disk
- Alternativly, haven't tested this yet, but you should be able to go to Hardware -> Hard Disk -> Disk Actions -> Move Storage and put the disk somewhere that it is accessable and has a format that will result in a typical file (qcow2/vmdk on a NAS).
