Problem with ZFS RAIDZ and disk hotspot

[ccwb102]

I am currently virtualising my server landscape. As part of this activity I do quite a few migrations between different proxmox instances. So lot's of moving disk images around.

What I am observing is that one of the proxmox servers has an iowait issue when it is on the receiving end of a virtual disk. Target storage is zfs RAIDZ pool on magnetic 4 TB disks.

Looking at the relevant disks with iostat this is a typical picture:

It looks as if one of the disks (sdd) is a hotspot. Incidentally, this disk is a slightly different WD RED variant, actually newer than the other 3 disks.

Can someone point me at the possible cause of this asymmetry? Should I look more into the disk hardware or is it more likely a zfs hotspot issue?

Any help highly appreciated. Here goes some additionally info about the system:

Dell PowerEdge T30
3x WD40EFRX
1x WD40EFAX
(all on the internal SATA ports of the mainboard)
2x 1 TB NVME on PCIe controllers
2x 1 TB SATA SSD on 4 port SATA PCIe controller
1x 250 GB SATA SSD on the same 4 port controller
64 GB ECC RAM

PVE 8.1.4
Kernel 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1



Thanks,
Christoph

 

[Dunuin]

WD RED

Newer "WD Red" use terrible SMR and therefore shouldn't be used with ZFS. Only the "WD Red Pro" and "WD Red Plus" use CMR. Old "WD Red" could use SMR or CMR.

Your WD40EFAX uses SMR and therefore will horribly suck at writing once the cache is full. Similar to what you see with QLC SSDs.

 

[cwt]

Should be mentioned: every SMR disk performs crappy in RAID environments, not only with ZFS.

 

[ccwb102]

@cwt & @Dunuin: Thanks for the quick reply. And for stating the obvious . Will replace the disk ASAP.

 

[Dunuin]

Should be mentioned: every SMR disk performs crappy in RAID environments, not only with ZFS.

I find them even unusable and slowing down the whole system when using them as a single disk with NTFS/ext4 on top. Not even useful to store backups on it.

 

[cwt]

I find them even unusable and slowing down the whole system when using them as a single disk with NTFS/ext4 on top. Not even useful to store backups on it.

You can still use them as bricks for your house ;-)

 

[alexskysilk]

SMR drives get a bad rap; they are perfectly usable as long as expectations are set correctly to begin with.

- they are NOT FAST, especially for rapid writes, or once they've been written 100% to.
- they are TERRIBLE when resilvered. expect 10x resilver times so no raidz1- ever.
- may not be related to SMR but they tend to be cheaper and not last as long as CMR drives.

as long as you understand the implications, they're fine as warm/tier 4 storage- even in a zpool.