Ransomware alert on Proxmox Backup Server 3.4.1

[duanvu]

We are currently using Proxmox Backup Server (PBS) 3.4.1 as our backup solution. The backups are performing as expected; however, after a few days, Microsoft Defender detected a potential threat and issued the following alert:

Alert Details:

• Process: proxmox-backup-proxy
• File Interaction: /mnt/datastore/backup2nas/.chunks/574c/574c8bde20722a5f76aff4c54042b849fc5e1fbb4fc875ce90eede004f4bd7cb
• Threat Detected: Ransom:HTML/Chicrypt.A
• Action Taken: Defender quarantined the file.

We need guidance on the following:

1. How can we prevent this issue?
2. What is the potential impact on our backups if a chunk is deleted or quarantined, as in this case?

Kind regards,
Duan.

 

[Johannes S]

How can we prevent this issue?

Uninstall Defender from the PBS or put an exception for the datastore path in it. If your corporate security compilance theater doesn't allow this for some reason you need to rethink your whole backup or security approach because a missing chunk renders backups non-restorable. My guess for the reason of this false positive is that you propably enabled encrypted backups (which are a good thing to have) and since antivirus snakeoil is really stupid it just saw "Oh it encrypted a file, surely this might be some ransomware (since ransomware encrypt files too, PANIC!")
BTW: How did you connect your NAS to the PBS? Network storage is not recommended due to their relative bad performance compared to local attached SSDs. It might be not only more performant but also more secure to isolate the PBS completely from the rest of your environment (by using enterprise SSDs directly attached to the PBS-Server and setting tight permissions on the PBS).
Sorry to be blunt: If you are not running a file server (for network share folders) or mail server there is never a good technical reason to install an antivirus software (even if it'S called enterprise endpoint security) on a Linux server. Except you need to check some boxes for a security compilance audit.

What is the potential impact on our backups if a chunk is deleted or quarantined, as in this case?

Any backup containing this chunk can not be restored it's as simple as this. So you will need to fix this issue. I'm not sure but If I recall correct you need to setup a verify task, uncheck the "skip verified" box and then run the verify job. Afterwards redo your backup tasks. They should detect that one chunk is missing and try to reupload it. But as said: Not sure about that, maybe somebody else can chime in on it. It might also be possible to move the chunk file and folder from your quarantine back to the datastore but since PBS relies on correct metadata of the chunks (stuff like access times) for the garbage collection this might not work out as intended.

 

[fabian]

it could also be that that chunk does contain that virus, because whatever you backed up interacted with it (e.g., an email server might very well have problematic bytes flowing through it that might end up in a backup). in any case, excempting the datastore path from your scans seems like the sensible way to go, like @Johannes S suggested

 

[Johannes S]

it could also be that that chunk does contain that virus, because whatever you backed up interacted with it (e.g., an email server might very well have problematic bytes flowing through it that might end up in a backup). in any case, excempting the datastore path from your scans seems like the sensible way to go, like @Johannes S suggested

Good point with the backup containing the virus. Can you confirm that my suggested procedure for ensuring a reupload of the chunk would work? Aka reverify everything and afterwards redoing all backup taks?

 

[fabian]

as soon as the last snapshot in a group is marked as failing verification, it is no longer used as base for an incremental backup, and all chunks will be uploaded for the next backup run in that group (the server will still deduplicate chunks which are not corrupt on the server side, of course). so any corrupt chunk where the original input data is still part of the backup stream should be fixed afterwards, yes.

 

[duanvu]

BTW: How did you connect your NAS to the PBS? Network storage is not recommended due to their relative bad performance compared to local attached SSDs. It might be not only more performant but also more secure to isolate the PBS completely from the rest of your environment (by using enterprise SSDs directly attached to the PBS-Server and setting tight permissions on the PBS).

The PBS is a VM on our PVE and I know that local disk should be prefered but NAS storage is more comfortable for us at the momment.

Sorry to be blunt: If you are not running a file server (for network share folders) or mail server there is never a good technical reason to install an antivirus software (even if it'S called enterprise endpoint security) on a Linux server. Except you need to check some boxes for a security compilance audit.

Yeah, Microsoft Defender for Server will be deploy on all our server (and Defender for Endpoint for all devices) for security concern and security compliance audit too.

I was excluding the ./.trunks/ from Defender scanning and wait for the result, I will update the result here.

 

[duanvu]

as soon as the last snapshot in a group is marked as failing verification, it is no longer used as base for an incremental backup, and all chunks will be uploaded for the next backup run in that group (the server will still deduplicate chunks which are not corrupt on the server side, of course). so any corrupt chunk where the original input data is still part of the backup stream should be fixed afterwards, yes.

Thank you for verifying this.

 

[fabian]

keep in mind that if the original data is no longer around at the source, the chunk can of course also not be recreated. so this fixing only applies to chunks which are still part of the data if a backup is taken *now*

 

[JensF]

I was excluding the ./.trunks/ from Defender scanning...

I hope you mean

./.chunks

and it was just a kind of autocorrect.

 

[Johannes S]

Yeah, Microsoft Defender for Server will be deploy on all our server (and Defender for Endpoint for all devices) for security concern and security compliance audit too.

And what would you do if the endpoint is not a server but an appilance like TrueNAS? I would highly suggest to think of PBS and PVE as appilances whose host system shouldn't be touched at all except for maintenance.