All recent files on a VM disappeared after reboot or VM restored some earlier state during reboot?

RodinM

Well-Known Member
Aug 1, 2011
83
0
71
Hello,
I faced a strange problem with one of my VMs on Proxmox VE.
This is an Ubuntu VM which resides on a ZFS thick storage.
On 31 October 2026 I reconfigured the VM, added CPUs and RAM but did not reboot the VM to apply changes because the VM was in use and these additional resources were not very critical.
On 6 November 2026 I added a disk space to the partition which is mounted as a large ext4 LVM volume on /var/www. Then I did standard live resize procedure (pvresize, lvextend -r -l +100%FREE ...), all completed successfully.
On 7 November 2026 I finally rebooted the VM and new resources in CPU adn RAM were applied.
On 9 November I got messages from the VM users that they do not see the files and Mysql DB changes which were done from approximately from 31 October and till 6-7 November.
I have a VM backup for 7 November which was finished before I rebooted it. After restoring it to another VM instance I saw that all the lost changes in files and DB were there, so the problem was partially solved.

But I still do not understand the source of the problem.
Here is what I found:
1) The VM had no snapshots. The last VM snapshot was done about half a year ago and deleted as unnecessary anymore.
2) The VM had cloud-init options applied, all of them very basic and done with Proxmox GUI. After reboot the cloud-init service was applied again because I did not disabled cloud init and previously unchecked the box "update packages".
So ssh host keys were changed after reboot but this was the only problem I noticed by myself.

For me it looks like rebooting restored the VM state to the date I changed the CPU and RAM settings and left them not applied but I do not see any " old state source" for this VM.

Now I am not sure if it is safe to reboot the VM once again. I know, I do have recent backups every day, I can do snapshots before reboot, that is ok. But the changes our users do to the VM and Mysql DB (this is a web portal) are not very easy to track and can be noticed long after reboot which makes snapshots not very useful.

Any help in understanding the cause of the problem would be very appreciated.

Code:
proxmox-ve: 8.4.0 (running kernel: 6.8.12-15-pve)
pve-manager: 8.4.14 (running version: 8.4.14/b502d23c55afcba1)
proxmox-kernel-helper: 8.1.4
proxmox-kernel-6.8: 6.8.12-15
proxmox-kernel-6.8.12-15-pve-signed: 6.8.12-15
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
ceph-fuse: 17.2.8-pve2
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u2
frr-pythontools: 10.2.3-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.2
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.2
libpve-cluster-perl: 8.1.2
libpve-common-perl: 8.3.4
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.7
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.6-1
proxmox-backup-file-restore: 3.4.6-1
proxmox-backup-restore-image: 0.7.0
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.4
proxmox-mail-forward: 0.3.3
proxmox-mini-journalreader: 1.5
proxmox-offline-mirror-helper: 0.6.8
proxmox-widget-toolkit: 4.3.13
pve-cluster: 8.1.2
pve-container: 5.3.3
pve-docs: 8.4.1
pve-edk2-firmware: 4.2025.02-4~bpo12+1
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.2
pve-firmware: 3.16-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.5
pve-qemu-kvm: 9.2.0-7
pve-xtermjs: 5.5.0-2
qemu-server: 8.4.3
smartmontools: 7.3-pve1
spiceterm: 3.3.1
swtpm: 0.8.0+pve1
vncterm: 1.8.1
zfsutils-linux: 2.2.8-pve1
 
What software are you using for the backup? There have been reports of systems being stuck with orphan QEMU layers due to failed backup. The new writes since then land in temporary storage and a cold start discards those.
Until this week we backed up our VMs using just built in backup by Proxmox VE to an NFS storage.
We really had sometimes failed backups (I would better say: failed backup tasks, not failed backups):
1) Two backup schedules sometimes overlapped which would cause the latter backup task to fail;
2) The NFS storage is a Synology NAS shared folder with WriteOnce enabled and backup retention policy sometimes caused the whole backup task to fail because the time of the oldest backup to remove was not old enough for Synology WriteOnce policy which allows the files to be deleted only if they are one month old;
So I am not sure if I can call this "failed backups".

If this is important: we had installed the veeam backup agent to the VM half a year ago, but did not have time to configure or use it till this week at all.

What can I do to be sure that all the new writes are really written where they belong in case of failed backup?
Is there a way to detect such problems with VMs writing to a temporary storage before they become a really big problem?
 
I would recommend going through this thread:

There was one or two others but I cant find them in my brief search.

Keep in mind this is just a guess on my part and the cause could be completely different. One way to find the RCA and prevention steps is to engage product support.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
I'd fix the two things that make your backup tasks fail first. Put the VMs into one backup job so runs can't overlap, and set the retention so PVE never tries to remove a backup that is younger than the one month WriteOnce period on the Synology. Before any planned reboot, take a manual backup of the VM and mark it protected, so normal retention can't remove it if a gap like this only shows up weeks later. That 7 November backup is what got the data back, so I'd also compare it with the live VM's files and database before you copy anything across, to know exactly which range went missing.

Disclosure: I work at CyberSnap.