Server not able to sustain 1Gbps throughput

JurgenMK

Member
Mar 22, 2020
16
1
8
23
Hello,
My Proxmox server has a raid 5 array of 8 disks, so it should be able to sustain 1Gbps easily, however whenever I upload a big file to one of my containers, the first few seconds it goes at it's max speed, after which the IO delay skyrockets to sth like 20-30%, after which the transfer quits bc of timeout delay.
anyone know why it is happening/how it can be fixed?

Thanks in advance.
 
Could you describe your host a little bit more?
it's a Dell PowerEdge T610, with 2x Xeon X5650s, 92Gs of ram, a Perc 6/i raid controller, battery backed and 512G cache (I think, not sure what the capacity is), and is connected to my pc via a tp-link gigabit switch.
 
So, I think you went with LVM, right?

Can you test the disk speed locally by copying the file locally from one container to another?
 
So, I think you went with LVM, right?

Can you test the disk speed locally by copying the file locally from one container to another?
the link speed is around 250-280MB/s sustained internally, tested using a 27G datafile
 
the load is also massively increased, that's not good for a simple copy job.

Can you please go to the cli, install htop (apt-get install htop), and run it while copying the data? Please add a screenshot. I'd like to know what processes run in the background while the copying takes place.
 
the load is also massively increased, that's not good for a simple copy job.

Can you please go to the cli, install htop (apt-get install htop), and run it while copying the data? Please add a screenshot. I'd like to know what processes run in the background while the copying takes place.
as requested, top is host server, bottom left is source copy, bottom right is the recieving ct
1584888853009.pngas
 
My Proxmox server has a raid 5 array of 8 disks, so it should be able to sustain 1Gbps easily, however whenever I upload a big file to one of my containers, the first few seconds it goes at it's max speed, after which the IO delay skyrockets to sth like 20-30%, after which the transfer quits bc of timeout delay.

It's quite possible than you fill first the cache of your controller, then it need to flush data on disk.
As you use raid5, you have write speed of 1 disk.

BRW, perc6/I cache is around 256-512MB , not GB.
 
It's quite possible than you fill first the cache of your controller, then it need to flush data on disk.
As you use raid5, you have write speed of 1 disk.

BRW, perc6/I cache is around 256-512MB , not GB.
oh then it's 512MB cache, sorry,
but still, every drive in the raid array have more than 100MB/s write speed. so I don't think that is the problem, also on the host machine the upload works fine, it's just in a container that the issue arrises.
 
Maybe could you first do some disk wirte benchmark from host vs container (with fio for example).
(also, verify than your perc6/I has writeback enabled+battery ok)

also, about scp, I don't think you can do more than 1gbps (maybe 2gbps with high freq cpu), as you'll be cpu limited by 1core.
 
Maybe could you first do some disk wirte benchmark from host vs container (with fio for example).
(also, verify than your perc6/I has writeback enabled+battery ok)

also, about scp, I don't think you can do more than 1gbps (maybe 2gbps with high freq cpu), as you'll be cpu limited by 1core.
the fio result of the container:
1584903193897.png
the fio result of the host machine:
1584903213420.png
The perc 6/i controller is in write-back and is battery backed
 
Is this at the same time as you see the high io delay spike? There is nothing to see in htop
while copying i see the swp memory going to full temporarily, then I see a slight increase in ram, sustained throughput is now at 20-30MB/s
 
Wow, swap should never be used for such stuff. No wonder why the io delay spikes.
how do I configure the vm then, cus as far as I know I didn't change anything related to storage priority, just have the default linux lts cli installl on proxmox
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!