ZFS Replication fails after creating a VM snapshot with RAM – no warning in the GUI

[martusi61]

Hello,

I'm running Proxmox VE 9.2.25 on a two-node cluster with ZFS storage and scheduled VM replication.

I encountered an issue with VM 101 (Home Assistant) after creating a snapshot using the default GUI settings, which include the VM's RAM state.

The procedure was straightforward:

1. Created a snapshot named Ok using the Proxmox GUI, leaving the default RAM inclusion option enabled.
2. Performed a VM update.
3. The next scheduled replication job failed with the following error:

101-0: start replication job
101-0: guest => VM 101, running => 1856
101-0: volumes => local-zfs:vm-101-disk-0,
local-zfs:vm-101-disk-1,local-zfs:vm-101-state-Ok
101-0: end replication job with error:
No common base snapshot on volume(s) local-zfs:vm-101-state-Ok

Please remove the problematic volume(s) from the replication target or delete and re-create the whole job 101-0

The two virtual disks were replicated normally in previous runs. The problem was caused by the additional VM state volume created by the snapshot.

I eventually resolved the issue by disabling the replication job, removing the snapshot (which I no longer needed), and re-enabling the replication job. Replication then completed successfully.

My questions and suggestions:

1. Is this behaviour expected when creating a snapshot with RAM enabled on a VM configured for ZFS replication?
2. Why doesn't Proxmox automatically handle or exclude the temporary VM state volume from replication?
3. Could the snapshot creation dialog detect an existing replication job and warn the administrator that including RAM may cause subsequent replication failures?
4. Could this limitation be documented more explicitly?

From an administrator's perspective, this is rather confusing: the snapshot operation completes successfully, there is no warning in the GUI, and the replication failure only becomes apparent during the next scheduled job.

I believe a preventive warning, or better integration between snapshot management and replication, would significantly improve the user experience and prevent unnecessary replication failures.

Thank you for your feedback.

 

[fabian]

please post in English (or German, in the German subsections ).

 

[martusi61]

please post in English (or German, in the German subsections ).

Sorry, I fixed it. The machine translation was giving me trouble.

 

[J-Rod]

I just tested this multiple times with my two Home Assistant VMs, ZFS replication and two hosts running 9.2.25. I went further by creating a snapshot on the first host, hot migrating the VM to the second, and then created a second snapshot, migrated it back, then deleted both snapshots. No errors and the logs show the replication picked up the snapshot disks each time and replicated them without incident.

Have you tested this again after re-creating the scheduled replication?

 

[martusi61]

I just tested this multiple times with my two Home Assistant VMs, ZFS replication and two hosts running 9.2.25. I went further by creating a snapshot on the first host, hot migrating the VM to the second, and then created a second snapshot, migrated it back, then deleted both snapshots. No errors and the logs show the replication picked up the snapshot disks each time and replicated them without incident.

I just tested this multiple times with my two Home Assistant VMs, ZFS replication and two hosts running 9.2.25. I went further by creating a snapshot on the first host, hot migrating the VM to the second, and then created a second snapshot, migrated it back, then deleted both snapshots. No errors and the logs show the replication picked up the snapshot disks each time and replicated them without incident.

Have you tested this again after re-creating the scheduled replication?

Have you tested this again after re-creating the scheduled replication?Snapshots with RAM checked? This is important because it seems like the memory tax is blocking replication.

 

[J-Rod]

What, you can't ask a new question before answering mine!

But of course this was all done with "Include RAM" checked.

 

[martusi61]

I'd never bothered with the RAM option in Snapshot before, but this is the second time it's happened to me, previously with an LXC Docker as well. Honestly, I don't know where to look.I always keep everything up to date and in fact my Cluster works flawlessly.Anyway, thanks to J-Rod for doing some testing.

 

[fabian]

if you manage to reproduce it again, please post the state of ZFS datasets and snapshots on both nodes. do you have HA enabled and were there any failover actions on that guest? or did you do a manual failover by moving the config file from one node to another?

 

[martusi61]

if you manage to reproduce it again, please post the state of ZFS datasets and snapshots on both nodes. do you have HA enabled and were there any failover actions on that guest? or did you do a manual failover by moving the config file from one node to another?

Hi Fabian,

Thank you for looking into this issue.

Here are some details about my setup:

- Proxmox VE 9.2.25.
- Two-node cluster: "pve1" and "pve2" with QDevice
- ZFS storage ("local-zfs") with scheduled replication between the two nodes.
- HA is enabled for VM 101 (Home Assistant).

Regarding migrations and failovers:

There were no migrations after creating snapshot "Ok", and no manual or automatic failovers occurred during the incident.

The last migration took place several days earlier, following a reboot of node "pve1" after a Proxmox update that required it.

Therefore, there was no migration or failover between the snapshot creation and the replication failure.

The issue appeared after creating snapshot "Ok" on VM 101, using the default GUI setting with "Include RAM" enabled, and subsequently updating the VM.

Replication had been working normally before this operation.

The replication job then failed with:

No common base snapshot on volume(s) local-zfs:vm-101-state-Ok

I resolved the problem by:

1. Disabling the replication job.
2. Removing snapshot "Ok", including its associated VM state volume.
3. Re-enabling the replication job.

The next replication completed successfully.

Unfortunately, I did not collect the ZFS dataset and snapshot information from both nodes before removing the snapshot, so I can no longer provide the exact ZFS state at the time of failure.

I also mentioned that I had encountered a similar replication issue previously with an LXC Docker container, although I cannot confirm whether it had the same underlying cause.

Since the issue is currently resolved, I would prefer not to deliberately reproduce the failure on my production cluster just to collect additional diagnostic information.

However, I would be happy to provide any additional information or run non-disruptive diagnostic commands that might help investigate this behaviour.

Thank you for your time and support.

Best regards