MAKOP Ransomware attack on PVE 6.4.15 from within a VM guest because it had full access to my vdisks (qcow), allowed ransomware to encrypt my vdisks

[CloudArchive]

I am starting a thread to document my findings observations and experience of having my server wiped out by Makop Ransomware.

So see this as a heads up to check your own stuff.

I suspect, (can't prove) that it got into the 'share' where all my VM qcow files were because a guest was able to connect 'upstream' to the SMB share that was running on the same partition. It encrypted everything on my entire datastore (that was shared on SMB for a number of reasons) .
How did they gain access ? Apparantly RDP into the guest Windows 10 with weak password.

I did not know that the "VMBR3.190 VLAN" does not have to have an ip address in the setup of the PVE.
It is because it had an IP address in the same VLAN, that the Ransomware was able to detect the 'upstream' SMB share. (Can some-one comment on this ?)

I had 'elsewhere' backups in PBS on a different partition. Same machine and in a VM. (qcow on different drive though)
So, lots of data saved. No current guest VM (with regular backups) lost. All old VM lost.

Here's the kicker : There are several VM's still running and working fine despite their qcow files being encrypted.
On restart they will fail (and be gone forever) or on any attempt to do something low level like 'drive imaging' (Observed by other guest VM's)
In at least one case I was able to HOT plug a new qcow into a VM and inside the guest, 'robocopy' the important data files to it.
I'll still have to rebuild from scratch, once done, try attach the newest hotplugged qcow to get data files back in.
Any other backup solution that tries low level access instantly crashes it, because essentially it's drive is gone.

The lesson learnt here is : Guest VM may NEVER EVER EVER see into it's parent.
If guests have need for SMB common shared location, build that into a seperate guest.
Don't share it to them from your Proxmox (Debian) host.

ISOLATE ISOLATE ISOLATE
BACKUP BACKUP BACKUP

 

[ness1602]

Don't allow smb shares, atleast unauthenticated ones. First rule of anti-ransomware.
Second, do the backups and snapshots,

 

[_gabriel]

please rename your topic tilte, because of ransomware is within your Windows vm.
pve has nothing to do here.
and, of course, never open to public rdp or ssh, always use vpn or firewall rules.

edit: + by default VM use Lvmthin Block for virtual disks which are protected because can't be share over smb

 

[CloudArchive]

please rename your topic tilte, because of ransomware is within your Windows vm.
pve has nothing to do here.
and, of course, never open to public rdp or ssh, always use vpn or firewall rules.

edit: + by default VM use Lvmthin Block for virtual disks which are protected because can't be share over smb

I edited the heading and despite how it happened, the PVE and all it's files were wiped out .
Thus I believe it's relevant to other users not to make the mistakes I made.
As said above, I was using qcow. Noted however, that LVMTHIN virtual disks would have been immune.
Thank you.

 

[CloudArchive]

Don't allow smb shares, atleast unauthenticated ones. First rule of anti-ransomware.
Second, do the backups and snapshots,

Noted thank you, as I said I have now learnt, above.

 

[_gabriel]

sorry but the title promote a fail in PVE, but you will get same scenario with hyper v.
"crypt locked smb host share from Windows public rdp VM" should more revelant.
adapt wording because my bad english..

 

[leesteken]

Indeed, a title like "VM with full access to my virtual disks allowed ransomware to encrypt my virtual disks" would describe it and also function as a warning for others.

EDIT: And running an end-of-life/out-of-support version of Proxmox, that did not get any kernel updates from some time now, was not even the cause.

 

[CloudArchive]

sorry but the title promote a fail in PVE, but you will get same scenario with hyper v.
"crypt locked smb host share from Windows public rdp VM" should more revelant.
adapt wording because my bad english..

EDIT: And running an end-of-life/out-of-support version of Proxmox, that did not get any kernel updates from some time now, was not even the cause.

The PVE did fall ! The title does not state that it was PVE's fault, and I said that it was my bad pracice.

I never said it was PVE fault - it is a discussion about what happened. My PVE was wiped out was it not ?

 

[CloudArchive]

I regret posting here as I mistakenly thought it was a community forum where one could discuss what happened.

This is the most meaningful I learnt here :

by default VM use Lvmthin Block for virtual disks which are protected because can't be share over smb

 

[leesteken]

Did you try making a snapshot backup of the still running VMs to create a unencrypted backup of the whole VM?

 

[CloudArchive]

Did you try making a snapshot backup of the still running VMs to create a unencrypted backup of the whole VM?

Yes, tried that. Any attempt to do that instantly crashes the VM forever. Because the qcow file does not 'exist' anymore.

 

[leesteken]

Yes, tried that. Any attempt to do that instantly crashes the VM forever. Because the qcow file does not 'exist' anymore.

Yikes, I would have thought it might work because the a snapshot backup does not stop the VM. Good to know and sorry that you had to find out this way.

 

[CloudArchive]

Yikes, I would have thought it might work because the a snapshot backup does not stop the VM. Good to know and sorry that you had to find out this way.

My observation is any attempt to 'low level' access the drive, will make the system realise that it's not there any more.
Or anything that will trigger drive 'rescsan'

 

[pgro]

Question, does snapshots protecting you from Ransomwares? also for zvol and vdev. Also, is there any sophisticated Proxmox Ransomware attack that can gain and attach zfs and zfs snapshot system?

 

[LnxBil]

My observation is any attempt to 'low level' access the drive, will make the system realise that it's not there any more.
Or anything that will trigger drive 'rescsan'

A little bit background from the OS side:

No, the qcow2 file is opened on VM start and will not be closed. Therefore a deletion of the backend storage file will not influence the running VM until it is stopped.

Linux does not case about open files like windows does: a file name is just a name for the filesystem internals, e.g. inode and once opened, the application will read, slew and write to it despite the removal of the file on the filesystem. Windows will just block this. The name to the inode is still visible via the /proc filesystem and you can recover the deleted yet open file and it'll occupy space as long as it is still open. Therefore a (ransomware) attack on the file that encrypts it to a new file and delete the old one will NOT result in data loss unless the open file is closed. This concurs with your observation. So you should also be able to backup the running VM and back the actual data up and not the encrypted one.

Question, does snapshots protecting you from Ransomwares? also for zvol and vdev. Also, is there any sophisticated Proxmox Ransomware attack that can gain and attach zfs and zfs snapshot system?

I haven't seen ransomware do this, but of course it's possible. Just encrypt the dataset, delete the oldest snapshot, create a new one with the name and repeat unless everything is encrypted. You're only save if the ZFS is not on your local machine and you export it via iSCSI.

 

[_gabriel]

volumes blocks like zvol, lvmthin can't be shared over samba so it's not reproductible as same as the OP.
of course if the host is compromised, all is possible and never end.

 

[bbgeek17]

Also, is there any sophisticated Proxmox Ransomware attack

While none has been reported so far, based on recent PVE rise in popularity (due to VMWare debacle), its very likely that it will attract more attention from bad actors.

You're only save if the ZFS is not on your local machine and you export it via iSCSI.

I would advise against creating false sense of security here. If the attacker is in the network/PVE - they could encrypt raw device (iSCSI). Not to mention that there is passwordless authentication in place between PVE and Storage, which is a pre-requisite for setting up iSCSI/ZFS.

The goal should be full hypervisor/infrastructure isolation from the VMs.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

Question, does snapshots protecting you from Ransomwares? also for zvol and vdev. Also, is there any sophisticated Proxmox Ransomware attack that can gain and attach zfs and zfs snapshot system?

It can, so long as it was taken before the ransomware event. To my knowledge, it wouldn't be possible to alter an in-place snapshot without causing file system corruption. This is explicitly disallowed on a zpool with any fault tolerance.

 

[alyarb]

A public RDP host can serve over 40,000 password guesses per day.

Even with a complex login, you should not be hosting ANY administrative services directly on the internet without 2FA, an IP ACL, or a tool like RDPGuard or fail2ban, etc.

Your guest VMs should also never have direct access to their own underlying storage, what the hell...

 

[LnxBil]

I would advise against creating false sense of security here. If the attacker is in the network/PVE - they could encrypt raw device (iSCSI). Not to mention that there is passwordless authentication in place between PVE and Storage, which is a pre-requisite for setting up iSCSI/ZFS.

Yes sure, but the snapshots are not compromiseable over the network unless you have CLI access to the SAN. That was my point.