ZFS datasets zeroed

[voidic]

Hi,

I experienced data loss via zfs datasets being 0 wiped for no reason, im not entirely sure how this happened... there were potentially some bad actors probing the server at that time but i dont have reason to believe that the server was compromised in any way.

I use ZFS RAIDZ on SSD's, i logged onto the server to see one of my VM's was not responding, and there were FS errors on console so i rebooted to find there was no OS anymore.

None of the 30+ other VM's were affected in any way...

All 3 vm disks were zero wiped, they look something like this:

scsi2: /dev/nvme8n1,discard=on,size=488386584K,ssd=1
scsi1: local-zfs:vm-198-disk-2,backup=0,discard=on,size=4T,ssd=1
scsi0: local-zfs:vm-198-disk-1,discard=on,size=500G

High Disk IO / Unresponsive VM: ~14:37






The high disk read is unrelated and from a different VM.

If somebody could help me diagnose the issue i would greatly appreciate any help whatsoever.

Regards,
Kieran

 

[donhwyo]

Try zpool status
Then zpool import

 

[fabian]

so

- you had suspicious activity on that host
- the monitoring shows lots of disk writes
- the disk is zeroed afterwards

the logical conclusion would be somebody compromised your VM and deleted its disks?

 

[voidic]

so

- you had suspicious activity on that host
- the monitoring shows lots of disk writes
- the disk is zeroed afterwards

the logical conclusion would be somebody compromised your VM and deleted its disks?

Its a possibility but i can't imagine that being true, to have root access and zero wipe the disks when there are all kinds of API keys and secrets as well as bitcoin to steal...

 

[fabian]

well the Disk IO graph shows IO as seen by Qemu, so it must have been something inside the VM..

 

[fabian]

also just because the disks were zeroed doesn't mean that the sensitive things weren't first exfiltrated as well.. the zeroing might just have been a way of covering their tracks.. of course it could also have been an accident - but only you know who had access to that VM or what was running inside..

 

[voidic]

also just because the disks were zeroed doesn't mean that the sensitive things weren't first exfiltrated as well.. the zeroing might just have been a way of covering their tracks.. of course it could also have been an accident - but only you know who had access to that VM or what was running inside..

Thank you for your reply. Only I have access via ssh key.. this machine was running kubernetes so either the version i was running had an RCE or some credentials were stolen... thank you for your help