Better Protection against ransomware

[CCWTech]

Recently we had a client that had a machine on their network get ransomware. The virus encrypted every computer on the network, including the Proxmox server. They are in a Windows environment so I am not 100% sure how it spread to Proxmox, but it did. Including to the backups stored on a SYNOLOGY NAS.

Luckily they had online backups and we were able to get their data back but I am interested in how we can better protect our clients.

1 - How can we prevent it jumping from Windows to Proxmox
2 - How can we better isolate backups on a NAS to avoid being encrypted as well.

 

[bbgeek17]

1 - How can we prevent it jumping from Windows to Proxmox

Hi @CCWTech, sorry to hear about your client, but without you figuring out _how_ the jump occurred it would be impossible to give a definitive answer on how to prevent it.

Could they have had a keylogger and grabbed the password? Were there non-passphrase-protected SSH keys installed? Was the password stored in clear text in an excel spreadsheet called "All Company Passwords"?

Perhaps you can clarify what exactly "including Proxmox" means? Did it encrypt files on the root filesystem? Or were the qcows on SMB share encrypted?

The basic advice that could apply to any situation:
- don't use root password login daily, implement SSH keys for human logins
- create a read-only PVE user for monitoring
- limit access to proxmox console (ssh/ui/API)
- implement administrative vlan to isolate idrac, pve, etc administrative access
- avoid storing passwords in places that can be accessible in case of compromise
- place backups on isolated vlan, restrict access via IP to the backup store

In the end, there may be nothing you can do if the administrative account is compromised and attackers have enough time to spend and monitor the network.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[leesteken]

PBS has a section about ransomware: https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery

Proxmox Backup Server does not rewrite data for existing blocks. This means that a compromised Proxmox VE host or any other compromised system that uses the client to back up data cannot corrupt or modify existing backups in anyway.

Unless attackers get (root) access to the server running PBS, of course. Making the management network separate from the VM network also helps.

 

[CCWTech]

Well we *THINK* it was because their password for their Windows server and the password for the Proxmox server were the same. (Different usernames, but same actual password. We didn't have it stored anywhere locally.

 

[CCWTech]

PBS has a section about ransomware: https://pbs.proxmox.com/docs/storage.html#ransomware-protection-recovery

Unless attackers get (root) access to the server running PBS, of course. Making the management network separate from the VM network also helps.

I see that, in this case, it wasn't PBS, it was Proxmox that backs up directly to a NAS. But I understand some concepts could carry over.

 

[bbgeek17]

I see that, in this case, it wasn't PBS, it was Proxmox that backs up directly to a NAS.

Was it an NFS or CIFS share? If CIFS, then it was likely easily accessible from compromised Windows host with administrative account.
NFS would require a bit more intelligence from malware, i.e. unlikely directly accessible from Windows. But since they had access to PVE - the export was there for the taking.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[CCWTech]

NFS share for backups.

 

[dietmar]

In the end, there may be nothing you can do if the administrative account is compromised and attackers have enough time to spend and monitor the network.

3-2-1 backup rule: You should have three total copies, use two different media (disk and tape, usually), and keep one copy offsite (tape).
You will be always able to restore the offsite tape.

 

[bbgeek17]

3-2-1 backup rule: You should have three total copies, use two different media (disk and tape, usually), and keep one copy offsite (tape).
You will be always able to restore the offsite tape.

I should have been more clear - my comment was referring to the OP's first question on preventing access to PVE.

You are, of course, correct about the backup approach.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[CCWTech]

3-2-1 backup rule: You should have three total copies, use two different media (disk and tape, usually), and keep one copy offsite (tape).
You will be always able to restore the offsite tape.

Yes, we did, and that is what saved from data loss. I am more concerned about prevention in the first place as we are at about 140 tech hours right now into recovery.