So I got home from work on friday and noticed that all my containers on a node had been shut down, with a lot of error messages saying my containers couldn't start.
I researched it for a bit, and after awhile understood that all my .raw files for the containers had been wiped. In fact, the entire storage had been wiped and all that was left was and empty folder hierarchy.
I checked the server logs and all I could see was that some of the containers running tasks had a lot of SIGKILL shutdowns due to out of memory exceptions. This was a bit expected as i was tuning the containers for optimal ram use the last few days. I wasn't too worried about it affecting the node, since they are in containers anyway right?
Well... Im not to familiar with container tech, but I couldnt see any reason why this would be able to wipe my storage completely?
I checked the nodes syslog and found something which *looks* to me like some kind of recovery process, but am not sure. It clear that there has been a restart. Anyway, after that block in the logs,I can see that it can't start the containers with the missing .raw file messages. So this happened right before the files disappeared.
So I am wondering what all this could because?
- Have I had some kind of security breach in on of my containers, which the intruder has somehow managed to wipe the storage drive?
This theory is only based on the fact that I was running a Cardano stake node which had internet access. The only containers had no access to internet. It doesnt make much sense though, as wiping the container like that would make the intruder lose his access to the container, and thus he wouldnt be able to do anything else. Also, why leave the folder structure?
- Is it possible that the multiple (once every 2 hours) out of memory exceptions from some of the containers, somehow kicked of a reset of the storage or what?
- Is it because of something else entirely?
What could in theory caused such an incident?
I can post the syslog here on what I believe is some form of recovery which happened before it couldn't start the containers, but if anyone has any idea where else to look for explanation please feel free to suggest anything! Because I am at a loss here!
/Thanks
I researched it for a bit, and after awhile understood that all my .raw files for the containers had been wiped. In fact, the entire storage had been wiped and all that was left was and empty folder hierarchy.
I checked the server logs and all I could see was that some of the containers running tasks had a lot of SIGKILL shutdowns due to out of memory exceptions. This was a bit expected as i was tuning the containers for optimal ram use the last few days. I wasn't too worried about it affecting the node, since they are in containers anyway right?
Well... Im not to familiar with container tech, but I couldnt see any reason why this would be able to wipe my storage completely?
I checked the nodes syslog and found something which *looks* to me like some kind of recovery process, but am not sure. It clear that there has been a restart. Anyway, after that block in the logs,I can see that it can't start the containers with the missing .raw file messages. So this happened right before the files disappeared.
So I am wondering what all this could because?
- Have I had some kind of security breach in on of my containers, which the intruder has somehow managed to wipe the storage drive?
This theory is only based on the fact that I was running a Cardano stake node which had internet access. The only containers had no access to internet. It doesnt make much sense though, as wiping the container like that would make the intruder lose his access to the container, and thus he wouldnt be able to do anything else. Also, why leave the folder structure?
- Is it possible that the multiple (once every 2 hours) out of memory exceptions from some of the containers, somehow kicked of a reset of the storage or what?
- Is it because of something else entirely?
What could in theory caused such an incident?
I can post the syslog here on what I believe is some form of recovery which happened before it couldn't start the containers, but if anyone has any idea where else to look for explanation please feel free to suggest anything! Because I am at a loss here!
/Thanks
