Server not able to sustain 1Gbps throughput

[JurgenMK]

Hello,
My Proxmox server has a raid 5 array of 8 disks, so it should be able to sustain 1Gbps easily, however whenever I upload a big file to one of my containers, the first few seconds it goes at it's max speed, after which the IO delay skyrockets to sth like 20-30%, after which the transfer quits bc of timeout delay.
anyone know why it is happening/how it can be fixed?

Thanks in advance.

 

[LnxBil]

Could you describe your host a little bit more?

 

[JurgenMK]

Could you describe your host a little bit more?

it's a Dell PowerEdge T610, with 2x Xeon X5650s, 92Gs of ram, a Perc 6/i raid controller, battery backed and 512G cache (I think, not sure what the capacity is), and is connected to my pc via a tp-link gigabit switch.

 

[LnxBil]

So, I think you went with LVM, right?

Can you test the disk speed locally by copying the file locally from one container to another?

 

[JurgenMK]

So, I think you went with LVM, right?

Can you test the disk speed locally by copying the file locally from one container to another?

the link speed is around 250-280MB/s sustained internally, tested using a 27G datafile

 

[LnxBil]

the link speed is around 250-280MB/s sustained internally, tested using a 27G datafile

Thats feasible.

How do you copy the file via the network to your container?

 

[JurgenMK]

Thats feasible.

How do you copy the file via the network to your container?

using filezilla scp, also while uploading the IO delay grew to 30%, if it is of any help, here is a screen of the host server:

 

[LnxBil]

the load is also massively increased, that's not good for a simple copy job.

Can you please go to the cli, install htop (apt-get install htop), and run it while copying the data? Please add a screenshot. I'd like to know what processes run in the background while the copying takes place.

 

[JurgenMK]

the load is also massively increased, that's not good for a simple copy job.

Can you please go to the cli, install htop (apt-get install htop), and run it while copying the data? Please add a screenshot. I'd like to know what processes run in the background while the copying takes place.

as requested, top is host server, bottom left is source copy, bottom right is the recieving ct
as

 

[spirit]

My Proxmox server has a raid 5 array of 8 disks, so it should be able to sustain 1Gbps easily, however whenever I upload a big file to one of my containers, the first few seconds it goes at it's max speed, after which the IO delay skyrockets to sth like 20-30%, after which the transfer quits bc of timeout delay.

It's quite possible than you fill first the cache of your controller, then it need to flush data on disk.
As you use raid5, you have write speed of 1 disk.

BRW, perc6/I cache is around 256-512MB , not GB.

 

[JurgenMK]

It's quite possible than you fill first the cache of your controller, then it need to flush data on disk.
As you use raid5, you have write speed of 1 disk.

BRW, perc6/I cache is around 256-512MB , not GB.

oh then it's 512MB cache, sorry,
but still, every drive in the raid array have more than 100MB/s write speed. so I don't think that is the problem, also on the host machine the upload works fine, it's just in a container that the issue arrises.

 

[spirit]

Maybe could you first do some disk wirte benchmark from host vs container (with fio for example).
(also, verify than your perc6/I has writeback enabled+battery ok)

also, about scp, I don't think you can do more than 1gbps (maybe 2gbps with high freq cpu), as you'll be cpu limited by 1core.

 

[JurgenMK]

Maybe could you first do some disk wirte benchmark from host vs container (with fio for example).
(also, verify than your perc6/I has writeback enabled+battery ok)

also, about scp, I don't think you can do more than 1gbps (maybe 2gbps with high freq cpu), as you'll be cpu limited by 1core.

the fio result of the container:

the fio result of the host machine:

The perc 6/i controller is in write-back and is battery backed

 

[LnxBil]

as requested, top is host server, bottom left is source copy, bottom right is the recieving ct
View attachment 15853as

Is this at the same time as you see the high io delay spike? There is nothing to see in htop

 

[JurgenMK]

Is this at the same time as you see the high io delay spike? There is nothing to see in htop

yes it is

 

[JurgenMK]

Is this at the same time as you see the high io delay spike? There is nothing to see in htop

while copying i see the swp memory going to full temporarily, then I see a slight increase in ram, sustained throughput is now at 20-30MB/s

 

[LnxBil]

while copying i see the swp memory going to full temporarily, then I see a slight increase in ram, sustained throughput is now at 20-30MB/s

Wow, swap should never be used for such stuff. No wonder why the io delay spikes.

 

[JurgenMK]

Wow, swap should never be used for such stuff. No wonder why the io delay spikes.

how do I configure the vm then, cus as far as I know I didn't change anything related to storage priority, just have the default linux lts cli installl on proxmox

 

[LnxBil]

Unfortunately, I do not know what causes this. I've never seen the swap used in a copy action.