Weird issue: configuration done in a VM wasn't saved to disk

[trollmann]

Hello

I encountered a strange issue today. One of my VM servers had some additions/changes done to it about two weeks ago. Without reduntantly explaining too much, a site with a map was added to a service its running. This new site with the map had been running fine for two weeks, but today I restarted the service for regular maintenance and when the service resumed the new site had disappeared, just poof. It seems the changes that were done were never saved to the actual file system, but for some very strange reason just existed in RAM. I know this because when it happened, I restored a backup from 3 days ago and the site wasn't there either.

I have no idea if this is a Proxmox thing or not, but I had a session with the vendor and they couldn't fathom how it had happened. Is it at all possible that it has something to do with the virtualization?

I'm running regular ext4 with soft raid mirroring. The swap space has been at 100% for a good while too, if that could be relevant.

Any ideas? Thanks in advance.

 

[dcsapak]

if the application inside wrote to the (virtual) disk and there was no abrupt power outage, i fail to see how that could happen
even if there was some caching going on, the vm backup at least would have too read from the cache and would have included that

what exactly did you inside the guest? do you control how files get saved to the disk? is this a 3rd party application? can you reproduce it?

 

[trollmann]

First, thank you for the reply

This one has me stunned. All the configurations, files, everything that has been happening inside that application was suddenly reset 1.5 months back in time. The rest of the server is current. The only thing that in my mind can explain this, is if for some reason everything the app has done has resided exclusively in RAM, but even then that I cannot really fathom how it can happen. It writes to the file system the whole time and would error out if a write fails, meaning it must have gotten an "OK" for every write, without it actually being true. It could however explain why RAM usage seemed to just steadily climb over the course of weeks on that guest VM.

Caching is set to "writeback", which could lead to data loss, but there has been no power failures and that kind of data loss likely wouldn't look like this.

This is a third party application. It is a Windows 2019 guest, using qcow2, virtio for disk and network. I had ballooning enabled, but deactivated it now. I noticed an error in the config too, the OS was set to Win11, which is fixed now. Could ballooning in combination with it thinking its W11 cause something like this? I have no special settings done in Windows to control have data is saved.

It all happened when the service got restarted, which fits with the idea that it was all just residing in RAM. The application stopped, its RAM content got deleted and boom, back in time.

I do believe the application is Java based, which uses some virtualization too. Maybe that has had a part to play in this? I'll have to verify if its Java.

 

[dcsapak]

Could ballooning in combination with it thinking its W11 cause something like this?

no

since i don't know the app, i can only speculate, but is there maybe some kind of 'commit' function? such that you configure it, but have to actively write it back to disk/make it persistent?
i'm asking because i know this from managed switch firmware interfaces where you can save the configuration, but without 'applying' it, it's gone after a reboot/power cycle (bit me more than i care to admit )

 

[trollmann]

Yeah, wild guessing there. I've concluded the issue was not related to Proxmox. Most likely user error or a bug. Thank you though!