BTRFS awful disk performance

[daronhudson]

Hi, I'm fairly new to the forum, but have been using proxmox in some way for a few years now.
I recently acquired a large 1u server with an amd epyc 7571, 512gb of DDR4 ram, and 4x8tb intel gen 3 nvme drives. I have about 32 lxs/vms running in total.

I haven't really messed around with software raid much so I decided to go with something new using BTRFS for a raid0 array.

I'm not really doing anything that's super write intensive, but btrfs writes seem to be kinda slow and cause high io delay. I'm honestly not even sure where to start looking.
I gave iotop a quick look and it turned out to mostly be btrfs transactions doing most of it. Random writes seem to be around 80-90mb while sequential is about 1gb. I don't know if that's just a proxmox problem or not, but the Intel p4510's in my server should be able to do about 3gb according to the spec. Obviously it won't do that in all scenarios, but i would have assumed that sequential would be able to with just a large flat file.

Things seem to lock up when btrfs is causing high io delay at 20%+. I've looked around quite a bit and just can't find anything that leads to a solution. I can provide any logs or metrics needed if it'll help out, I just don't know what would be necessary at the moment.

Thanks for any help that can be provided. I'm really at a loss here.

 

[itNGO]

Use ZFS for VM Data.

 

[UdoB]

but btrfs writes seem to be kinda slow

This means nothing to an interested reader.

You need to qualify and quantify this in a repeatable and verifiable way. Look for "fio"-examples here in the forum; that's one of the usual benchmarking tools and its results are more useful than "seem to be slow".


Disclaimer: I have no comparable hardware and not much experience with btrfs, so I might not be able to help at all...

 

[bbrendon]

Hi, I'm fairly new to the forum, but have been using proxmox in some way for a few years now.
I recently acquired a large 1u server with an amd epyc 7571, 512gb of DDR4 ram, and 4x8tb intel gen 3 nvme drives. I have about 32 lxs/vms running in total.

I haven't really messed around with software raid much so I decided to go with something new using BTRFS for a raid0 array.

I'm not really doing anything that's super write intensive, but btrfs writes seem to be kinda slow and cause high io delay. I'm honestly not even sure where to start looking.
I gave iotop a quick look and it turned out to mostly be btrfs transactions doing most of it. Random writes seem to be around 80-90mb while sequential is about 1gb. I don't know if that's just a proxmox problem or not, but the Intel p4510's in my server should be able to do about 3gb according to the spec. Obviously it won't do that in all scenarios, but i would have assumed that sequential would be able to with just a large flat file.

Things seem to lock up when btrfs is causing high io delay at 20%+. I've looked around quite a bit and just can't find anything that leads to a solution. I can provide any logs or metrics needed if it'll help out, I just don't know what would be necessary at the moment.

Thanks for any help that can be provided. I'm really at a loss here.

It sounds like you have found some "features" of btrfs. If you're using snapshots on large & busy storage you'll have problems with dead locks.

I had similar issues ~ 3 years ago. I don't recall the details but it was related to dead locks and the generation metrics.

I have blacklisted btrfs from my life and went back to ZFS for basically everything linux running on physical hardware. Even small laptops. Life has been much better.

 

[Fantu]

That disk serie seems optimized for "read-intensive and/or mixed sequential workloads": https://www.solidigm.com/products/data-center/d7/p4510.html
random write seems low also checking this: https://versus.com/en/intel-dc-p4510-series-1tb

Vm storage on cow filesystem it seems to me that it does very little sequential and the writes are greater, on zfs when I tried it a few years ago seems even more write amplification than with btrfs but the greater use of cache can help to alleviate the performance problem (unless you have an excessive load of writes that the disks cannot support and create a bottleneck).

Write on the right section (https://forum.proxmox.com/forums/proxmox-ve-installation-and-configuration.16/) and give more details maybe you'll get more advice.