One zfs pool slows down all other zfs pools

[Jannoke]

It might be more like zfs question, but maybe someone has had experience.

So i have few zfs pools on one machine.
One of pools is running on simple consumer (dramless) nvme's (3 disk raidz1, one disk missing).. There is only one virtual machine on that specific pool . While filling disk on that pools guest machine using:

dd if=/dev/urandom of=filltest.bin bs=1M count=95000 status=progress

It starts at 220MB/s ..and in around 5 minutes and 20GB it has dropped to around 100-120MB/s. At the same time all other guest machines are starting to report high IO even thou they are on different pools. And verifiebly the io is very sluggish on all these virtual machines to the point some services start to report being down. If i cancel the DD, it will recover.

There is plenty of ram (512GB) and cpu (dual xeon 2699v4). Also machine is not that loaded either on cpu or disk side before activating dd.

So how can it be that one virtual machine with unrelated pool to other machines can take down entire server zfs storage? What am I missing here. Is it like trashing my ARC?

root@bla:~# pveversion
pve-manager/9.0.9/117b893e0e6a4fee (running kernel: 6.14.11-2-pve)
root@bla:/etc/modprobe.d# cat zfs.conf 
options zfs zfs_arc_min=10737418240
options zfs zfs_arc_max=64424509440

it happends with sync=disabled or sync=standard

 

[Kingneutron]

> One of pools is running on simple consumer (dramless) nvme's (3 disk raidz1, one disk missing)

Seriously, you're running a consumer-level 3-disk raidz DEGRADED with 1 disk MISSING, and posting about it here?? Fix your pool first.

If you want better speed, rebuild it as a mirror pool. With Enterprise-level SSD or at least high-TBW rated like Lexar NM790.

 

[Jannoke]

> One of pools is running on simple consumer (dramless) nvme's (3 disk raidz1, one disk missing)

Seriously, you're running a consumer-level 3-disk raidz DEGRADED with 1 disk MISSING, and posting about it here?? Fix your pool first.

If you want better speed, rebuild it as a mirror pool. With Enterprise-level SSD or at least high-TBW rated like Lexar NM790.

Hey chatgpt - It's always grumpy people like you who come to comment. Question is about what happens if situation like this arises. Throwing stuff like make it a mirror is useless - I know how to make pools and what different raid levels are. Also this is not hw purchase thread so you why are we recommending some cheapass lexars which are another variant of consumer drives. Why are you even talking about high-tbw drives if you don't even know what i have. High tbw does not mean anything on zfs, especially if you are not going to use it. These drives are currently 1600TBW drives @1TB size if it makes you feel better.

I don't want to fix problem blindly - i want to learn from it. This pool is not broken because my drive broke. I removed the drive. It's no problem for me to change these out to some enterprise Samsung drives like the other ones are in the pool.

But that's not the problem -

It was about what I saw happening to other pools on the same machine if one unrelated pool member loses drive, and is still being written on. Disk can be lost at any time , but question is why it bogs other unrelated pools.

Next time someone asks a question and you don't have an answer just move one and go comment to the mirror. If you don't know about this specific matter, stop forcing your "one-size-fits-all" solutions. You can be either problem solver or part changer and obviously you are latter.

 

[UdoB]

I don't want to fix problem blindly - i want to learn from it. This pool is not broken because my drive broke. I removed the drive.

That wasn't obvious from your first post. A lot of users come here with lowest end hardware and/or semi-damaged systems - wondering why they have problems.

but question is why it bogs other unrelated pools.

Good question!

I have seen similar behavior - a single high level task slowing down a whole system. And I have had systems with a classic system load of >700 and the foreground processes acted fine, being completely unimpressed.

Are the disks of the unrelated pools on the same controller? Maybe the controller is busy handling the missing drive again and again - and this slows down any additional command because the command queue is full. Sorry, no good answer from me...

 

[chrcoluk]

What is the outcome of 'zpool status' please, there is detail lacking here. Do all the pools have their own independent disks?