All recent files on a VM disappeared after reboot or VM restored some earlier state during reboot?

RodinM

Well-Known Member
Aug 1, 2011
83
0
71
Hello,
I faced a strange problem with one of my VMs on Proxmox VE.
This is an Ubuntu VM which resides on a ZFS thick storage.
On 31 October 2026 I reconfigured the VM, added CPUs and RAM but did not reboot the VM to apply changes because the VM was in use and these additional resources were not very critical.
On 6 November 2026 I added a disk space to the partition which is mounted as a large ext4 LVM volume on /var/www. Then I did standard live resize procedure (pvresize, lvextend -r -l +100%FREE ...), all completed successfully.
On 7 November 2026 I finally rebooted the VM and new resources in CPU adn RAM were applied.
On 9 November I got messages from the VM users that they do not see the files and Mysql DB changes which were done from approximately from 31 October and till 6-7 November.
I have a VM backup for 7 November which was finished before I rebooted it. After restoring it to another VM instance I saw that all the lost changes in files and DB were there, so the problem was partially solved.

But I still do not understand the source of the problem.
Here is what I found:
1) The VM had no snapshots. The last VM snapshot was done about half a year ago and deleted as unnecessary anymore.
2) The VM had cloud-init options applied, all of them very basic and done with Proxmox GUI. After reboot the cloud-init service was applied again because I did not disabled cloud init and previously unchecked the box "update packages".
So ssh host keys were changed after reboot but this was the only problem I noticed by myself.

For me it looks like rebooting restored the VM state to the date I changed the CPU and RAM settings and left them not applied but I do not see any " old state source" for this VM.

Now I am not sure if it is safe to reboot the VM once again. I know, I do have recent backups every day, I can do snapshots before reboot, that is ok. But the changes our users do to the VM and Mysql DB (this is a web portal) are not very easy to track and can be noticed long after reboot which makes snapshots not very useful.

Any help in understanding the cause of the problem would be very appreciated.

Code:
proxmox-ve: 8.4.0 (running kernel: 6.8.12-15-pve)
pve-manager: 8.4.14 (running version: 8.4.14/b502d23c55afcba1)
proxmox-kernel-helper: 8.1.4
proxmox-kernel-6.8: 6.8.12-15
proxmox-kernel-6.8.12-15-pve-signed: 6.8.12-15
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
ceph-fuse: 17.2.8-pve2
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u2
frr-pythontools: 10.2.3-1+pve1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.2
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.2
libpve-cluster-perl: 8.1.2
libpve-common-perl: 8.3.4
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.7
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.6-1
proxmox-backup-file-restore: 3.4.6-1
proxmox-backup-restore-image: 0.7.0
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.4
proxmox-mail-forward: 0.3.3
proxmox-mini-journalreader: 1.5
proxmox-offline-mirror-helper: 0.6.8
proxmox-widget-toolkit: 4.3.13
pve-cluster: 8.1.2
pve-container: 5.3.3
pve-docs: 8.4.1
pve-edk2-firmware: 4.2025.02-4~bpo12+1
pve-esxi-import-tools: 0.7.4
pve-firewall: 5.1.2
pve-firmware: 3.16-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.5
pve-qemu-kvm: 9.2.0-7
pve-xtermjs: 5.5.0-2
qemu-server: 8.4.3
smartmontools: 7.3-pve1
spiceterm: 3.3.1
swtpm: 0.8.0+pve1
vncterm: 1.8.1
zfsutils-linux: 2.2.8-pve1
 
What software are you using for the backup? There have been reports of systems being stuck with orphan QEMU layers due to failed backup. The new writes since then land in temporary storage and a cold start discards those.
Until this week we backed up our VMs using just built in backup by Proxmox VE to an NFS storage.
We really had sometimes failed backups (I would better say: failed backup tasks, not failed backups):
1) Two backup schedules sometimes overlapped which would cause the latter backup task to fail;
2) The NFS storage is a Synology NAS shared folder with WriteOnce enabled and backup retention policy sometimes caused the whole backup task to fail because the time of the oldest backup to remove was not old enough for Synology WriteOnce policy which allows the files to be deleted only if they are one month old;
So I am not sure if I can call this "failed backups".

If this is important: we had installed the veeam backup agent to the VM half a year ago, but did not have time to configure or use it till this week at all.

What can I do to be sure that all the new writes are really written where they belong in case of failed backup?
Is there a way to detect such problems with VMs writing to a temporary storage before they become a really big problem?
 
I would recommend going through this thread:

There was one or two others but I cant find them in my brief search.

Keep in mind this is just a guess on my part and the cause could be completely different. One way to find the RCA and prevention steps is to engage product support.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
I'd fix the two things that make your backup tasks fail first. Put the VMs into one backup job so runs can't overlap, and set the retention so PVE never tries to remove a backup that is younger than the one month WriteOnce period on the Synology. Before any planned reboot, take a manual backup of the VM and mark it protected, so normal retention can't remove it if a gap like this only shows up weeks later. That 7 November backup is what got the data back, so I'd also compare it with the live VM's files and database before you copy anything across, to know exactly which range went missing.

Disclosure: I work at CyberSnap.
 
The failed task is the important part here. If a backup job crashes halfway through, the VM can end up writing to a temporary overlay instead of its actual disk. Everything may look perfectly normal, and the guest OS will keep running as usual. But once the VM is cold-booted, all those writes are gone. That's exactly why a reboot can expose the problem.

As for your two questions: the best way to verify that writes are going to the right place is to compare the VM's actual disk chain with what its configuration says it should be. An unexpected extra layer is usually what causes trouble when the VM reboots.

There's no reliable alert that will catch this, which is why I'd recommend doing an actual file restore at least once a month. It's a much better sanity check than relying on dashboards. If you can restore a file and its contents look right, that's a good indication that the disk chain is healthy.

Rebooting again should be fine, but I'd make sure you've successfully restored something from the backup first.
 
Can someone further clarify this "temporary" storage/overlay? Which filesystems and backup types does it happen under? With or without fleecing, or both?

I was under the impression that the VM writes during a backup go to the base virtual disk and the only thing that is supposed to be temporary is a copy of the old data that hasn't been sent to the backup server yet.

So a write comes in from the VM, a clone of that block is made to the temporary file or fleecing file, then the write is allowed to be made to the base virtual disk. When the backup is complete, the temporary or fleecing file is thrown out, but no data should be lost.
 
As a reminder, this was just a guess. There is no hard supporting evidence that we saw, ie logs. Only the result seems to match. The LLM summaries/suggestions are not a proof.
You have to go back in logs, hopefully you can go far enough, and trace the life of the VM.

The issue is generally that the "fleecing" layer is not removed due to some failure (busy signal, etc) if it was removed the writes would have been consolidated.
Note, not all shims are the same. There are different technics used by various backup vendors.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited: