Identify slow IO, even impacting ramdisk?

J

jw6677

Active Member
Oct 19, 2019
93
5
28
33
www.cayk.ca
Here's a google sheet full of my testing:
https://docs.google.com/spreadsheet...Od7tOf-cCyV24mY_4Uc3vOeVeY/edit#gid=194361085


When copying files to or from anywhere, I am limited to ~350MB/s transfer speed.

By anywhere, I mean:

To and From a Hardware Raid - 8 (cheap consumer) SSDs in Raid 0 | Each rated to 500/550MB/s R,W --> Total 4000/4400MB/s
To and From a Software Raid - 3 NVME SSDs Raid 0 with mdadm | Each rated to 3300/3000MB/s R,W --> Total 9900
even
To and From a tmpfs ramdisk, with two concurrent reads and writes to and from the same tmpfs (copying files to a new location)

Any combination always maxes at ~ 340MB file transfer speed with a single large (4GB) .tar.gz file, full of basically csv style data.

Interestingly, when copying tmpfs - tmpfs, top reports:
%Cpu(s): 4.2 us, 1.8 sy, 0.0 ni, 94.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

At "Idle":
%Cpu(s): 0.4 us, 0.1 sy, 0.0 ni, 99.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st


As well, Htop shows that a single core for each transfer is at 100% load, but I am under the impression that IO transfer is not CPU bound...?

Help!
 
Last edited:
  • Like
Reactions: merlin-vrn
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster

# uname -mrs
Linux 5.3.13-3-pve x86_64





lscpu

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 4
NUMA node(s): 4
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Model name: Intel(R) Xeon(R) CPU E5-4650 0 @ 2.70GHz
Stepping: 7
CPU MHz: 1672.593
CPU max MHz: 3300.0000
CPU min MHz: 1200.0000
BogoMIPS: 5399.65
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0,4,8,12,16,20,24,28,32,36,40,44,48,52,56,60
NUMA node1 CPU(s): 1,5,9,13,17,21,25,29,33,37,41,45,49,53,57,61
NUMA node2 CPU(s): 2,6,10,14,18,22,26,30,34,38,42,46,50,54,58,62
NUMA node3 CPU(s): 3,7,11,15,19,23,27,31,35,39,43,47,51,55,59,63
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm pti tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts




free

total used free shared buff/cache available
Mem: 726567720 5142736 720669224 80024 755760 718697784
Swap: 0 0 0



root@server:~# cat /proc/meminfo
MemTotal: 726567720 kB
MemFree: 720667304 kB
MemAvailable: 718695924 kB
Buffers: 38796 kB
Cached: 563516 kB
SwapCached: 0 kB
Active: 1050060 kB
Inactive: 274372 kB
Active(anon): 760112 kB
Inactive(anon): 58208 kB
Active(file): 289948 kB
Inactive(file): 216164 kB
Unevictable: 195092 kB
Mlocked: 195092 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 188 kB
Writeback: 0 kB
AnonPages: 917452 kB
Mapped: 119140 kB
Shmem: 80024 kB
KReclaimable: 153564 kB
Slab: 841472 kB
SReclaimable: 153564 kB
SUnreclaim: 687908 kB
KernelStack: 13728 kB
PageTables: 7048 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 363283860 kB
Committed_AS: 3921412 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 2682188 kB
VmallocChunk: 0 kB
Percpu: 241536 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 691008 kB
DirectMap2M: 13942784 kB
DirectMap1G: 725614592 kB






# lshw -short -C memory
H/W path Device Class Description
===========================================================
/0/0 memory 64KiB BIOS
/0/400/700 memory 256KiB L1 cache
/0/400/701 memory 2MiB L2 cache
/0/400/702 memory 20MiB L3 cache
/0/401/703 memory 256KiB L1 cache
/0/401/704 memory 2MiB L2 cache
/0/401/705 memory 20MiB L3 cache
/0/402/706 memory 256KiB L1 cache
/0/402/707 memory 2MiB L2 cache
/0/402/708 memory 20MiB L3 cache
/0/403/709 memory 256KiB L1 cache
/0/403/70a memory 2MiB L2 cache
/0/403/70b memory 20MiB L3 cache
/0/1000 memory 704GiB System Memory
/0/1000/0 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/1 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/2 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/3 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/4 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/5 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/6 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/7 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/8 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/9 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/a memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/b memory DIMM DDR3 Synchronous [empty]
/0/1000/c memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/d memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/e memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/f memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/10 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/11 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/12 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/13 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/14 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/15 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/16 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/17 memory DIMM DDR3 Synchronous [empty]
/0/1000/18 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/19 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/1a memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/1b memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/1c memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/1d memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/1e memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/1f memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/20 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/21 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/22 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/23 memory DIMM DDR3 Synchronous [empty]
/0/1000/24 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/25 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/26 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/27 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/28 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/29 memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/2a memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/2b memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/2c memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/2d memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/2e memory 16GiB DIMM DDR3 Synchronous Registered (Buffered) 1866 MHz (0.5 ns)
/0/1000/2f memory DIMM DDR3 Synchronous [empty]
 
To be clear, this is the host itself.

Is there some type of transparent compression or zero hole punching enabled with lvms or something?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom