Feature idea: saferemove mode to skip forced zero-write (blkdiscard --zeroout) on storage that guarantees zero-after-discard (e.g. NVMe DLFEAT)

M

mmercier

New Member
Proxmox Subscriber
May 16, 2025
26
2
3
Hello,

I'd like to gauge interest before filing this on Bugzilla - wanted to check the reasoning holds up and that I'm not missing an existing option.

Background: I run Proxmox VE on shared LVM backed by Dell PowerStore arrays over NVMe/TCP. saferemove is enabled on this storage to prevent a new VM from reading stale data left on physical extents freed by a deleted VM's disk (a real risk on thick LVM, since a freed extent goes straight back into the allocation pool with no zeroing).

Looking at LVMPlugin.pm, when saferemove is on and the LV's underlying device reports write_zeroes_max_bytes != 0, the code runs:

Code: 
blkdiscard $lvmpath -v --zeroout --step $stepsize

--zeroout maps to the BLKZEROOUT ioctl, which the kernel implements with BLKDEV_ZERO_NOUNMAP forced - i.e., it deliberately avoids relying on any "deallocate implies future reads return zero" guarantee the storage might offer, and instead performs a real, offloaded WRITE_ZEROES command that keeps the range provisioned. That's a deliberately conservative default and it's the right one when you don't know whether the storage backend actually guarantees zero-after-deallocate.

The issue: on storage that does provide that guarantee, this is much more expensive than it needs to be. I measured on our PowerStore-backed thick LVM (array in normal production use, not isolated/idle, so this reflects real-world contention rather than a best-case synthetic number):

OperationTime (100G LV)
Plain blkdiscard (no --zeroout)5.8s
blkdiscard --zeroout (current saferemove behavior)~101s

NVMe actually has a standard, machine-readable way to check this guarantee: the DLFEAT field in Identify Namespace data. Bits 2:0 = 001 means "reads of deallocated logical blocks return all-zeros," deterministically, per spec - not implementation-defined. I confirmed this is set (dlfeat: 9) on my PowerStore array. On a namespace with that guarantee, plain blkdiscard (UNMAP only) provides the exact same confidentiality property saferemove --zeroout is trying to guarantee, for a fraction of the cost.

I saw the recent work in this area (the Oct 2025 patch replacing the old cstream-based saferemove with blkdiscard --zeroout - thanks for that, it's already a big improvement over the old default!), so this feels like a natural next step rather than an unrelated ask.

Proposed idea: a new value/mode for the existing saferemove option (or a separate option, e.g. saferemove-mode = zero|discard) that, when set to discard, uses plain blkdiscard (no --zeroout) instead of forcing the write-zeroes path - intended for admins who have already verified their storage backend guarantees zero-after-deallocate (NVMe DLFEAT, or a documented SCSI/array-level equivalent) and are willing to opt in explicitly. This would keep the current safe-by-default behavior fully intact for everyone else, and wouldn't require Proxmox to auto-detect or trust arbitrary storage capability flags - the admin takes on that verification responsibility explicitly, similar to how saferemove_throughput/saferemove-stepsize already put tuning in the admin's hands.

Happy to open this on Bugzilla if there's interest — wanted to sanity-check the approach here first. Also curious if others running LVM-thick on NVMe-oF/NVMe-TCP-backed arrays have hit the same tradeoff.

Thanks,
Mike
 
Hi Mike,

It's a good idea, but it's worth noting that discard has a rough history. SATA TRIM often didn't guarantee zeros, a bunch of drives corrupted data on queued TRIM and are still on the kernel blacklist, and arrays have been all over the place with UNMAP. NVMe cleaned some of that up with the Deallocate Logical Block Features field (DLFEAT), but --zeroout is still the right default.

One thing to poke at: DLFEAT tells you what a read of a deallocated block returns. It doesn't tell you the deallocation actually happened. The spec says the controller "may" deallocate, and the Namespace Preferred Deallocate Granularity and Alignment fields (NPDG/NPDA) are only recommendations, so the command can come back fine without clearing everything.

If this gets added, the docs should advise that every PV in the VG must have the guarantee and recommend people read back a discarded LV and confirm it's zeros rather than relying on the flag.

Otherwise it's a sensible approach, and it puts the responsibility in the hands of an admin who can qualify their array. My only worry is the user - people love flipping switches without understanding them.

Happy to +1 it based on technical merit, though I can't say it affects us. Disk removal takes a few milliseconds in our system, and there's no such thing as an unsafe removal.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
but please note - we've had plenty of reports about storage appliances stating that they zero on unmap/discard, but empirical evidence shows they actually don't. or only do half of the time, if some block size and alignment assumptions are met. or if the phase of the moon and stars align. or ... ;)
 
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom
Back