Are the backups "windows active directory aware" ?

[s_sasha]

Hi guys,

So the question is: are the incremental backups "windows active directory aware" - customer had asked us this question today because apparently there's an old issue with AD, that caused USN rollback errors when a virtual active directory server was restored from an image and we are clueless.

Thanks in advance,
Sasha

 

[t.lamprecht]

Hi,

The deduplication/incremental-send layer is one below QEMU backup itself and provided by PBS, so they're as aware as an old-style vzdump backup.

For guaranteeing a clean state of the guest during backup one would either use stop-mode, with its disadvantage of resulting in a downtime. Alternatively, to avoid downtime, one could set up the QEMU Guest Agent inside the VM and then set the respective "enable QGA" option for the VM in Proxmox VE, as then a filesystem freeze is done from inside the guest using the guest OS safe APIs (VSS in Windows) before snapshotting the disk state for the backup, which is enough for most software to have a clean state.

https://pve.proxmox.com/pve-docs/chapter-qm.html#qm_qemu_agent

Having separate AD specific backups could be still reasonable, also restore tests, which need to be done a bit more careful with things like AD, e.g., with network cut-off or isolated for the restore test VM to avoid causing possible havoc of multiple AD/DC instances in the production network.

 

[s_sasha]

Hi @t.lamprecht

That fully answers my question - thank you.

 

[larsen]

Be careful when there is more than one DC (which is recommended). You should never restore a DC in this case:
https://www.itprotoday.com/server-virtualization/never-snapshot-domain-controller-here-s-why

 

[s_sasha]

Be careful when there is more than one DC (which is recommended). You should never restore a DC in this case:
https://www.itprotoday.com/server-virtualization/never-snapshot-domain-controller-here-s-why

Hi ... although very useful, I don't think this relates to the topic, please corretct me if I'm wrong ?

 

[larsen]

You wrote "...caused USN rollback errors when a virtual active directory server was restored from an image...".
Exactly this might happen when you have more than one DC and restore a backup (even if that backup itself is consistent and would work if you only had one DC).

 

[VictorSTS]

Domain controller backup/restore is fully supported in AD for a while now. Using virtual machines as DCs are supported too. To avoid USN rollback issues with AD you need:

- At least Windows server 2012
- At least QEMU v5.0-36
- Consistent backups using QEMU Agent + check that VSS is working correctly when you take a backup, so AD flushes to disk any uncommitted changes.

QEMU 5.0-36 introduced support for VM generation ID, which is used by Windows >=2012 to resolve the conflicts that will arise when you restore a DC, as it's USN will be older than that of the other DCs. In theoretically, if you have just one DC you should never find a USN rollback issue. Also get AD backups in each DC using Windows native backup tools. If you just need to restore AD, Windows backup tools are USN rollback aware and will do their best to avoid it.

Please, do your best to understand what a USN rollback is because it can be hard to recover from it depending on how complex that AD is.

 

[larsen]

Thanks, wasn't aware of this.

I found this article explaining the technical stuff:
https://learn.microsoft.com/en-us/w...omain-services-ad-ds-virtualization-level-100

Interesting part:

When an administrator restores the virtual machine from a previous snapshot, the current value of the VM GenerationID from the virtual machine driver is compared against a value in the DIT.
If the two values are different, the invocationID is reset and the RID pool discarded thereby preventing USN re-use. If the values are the same, the transaction is committed as normal.

A more recent article about why restoring snapshots is still not recommended (at least in some cases):
https://www.semperis.com/blog/hyper...bstitute-for-proper-active-directory-backups/

 

[Darkk]

I am pretty much in the same boat. I have 4 DCs. Half are physical with FSMO role on one of them and other half as VMs. I am in the process of moving the VMs from vmware to ProxMox using the offline export OVF method. All DCs are running Server 2019 with 2008 R2 domain level. Can't raise it any higher right now due to on-prem Exchange 2010 which I am in the process of retiring.

From high availability perspective with ZFS replication (not using CEPH) between the 7 node cluster seems I am better off letting the VM DC fail and use the built-in backup tools in Windows 2019 for recovery. Best backups always been when the VM is off but not practical with DCs.

The smallest time for ZFS replication via snapshots is 5 minutes so may not be practical for DCs. Might be ok for failover between nodes but when it comes to DCs can't mess around with it. So I may end up using ZFS replication just to migrate the VM over in an off state and then bring it up on the new node.