File Copy - RAM Cache Inconsistent

[square_eyes]

I am copying 100GB files quite frequently - destination is a standard HDD. I have 192GB of RAM.

Sometimes the copy is reported to be completed in 5mins (presumably) due to RAM caching, with an average speed of 250MB/s. This is great.

However, quite often the RAM cache is not used and I average 20MB/s. It takes an hour or more.

How can I troubleshoot, or otherwise consistently make use of caching?

Other notes...

• There is a break after each file copy, so it's not hitting a bottleneck where a new copy is started before the cache is emptied from the previous copy.
• I have a couple of CTs on this system but nothing that is eating up the ample RAM I have.
• The copy is happening on the PVE Host, and the destination is a HDD on a SAS card + SAS Expander
• No ZFS

 

[_gabriel]

do you use zfs at destination ?

 

[Himcules]

So many details needed to even start...

192GB of RAM - Great!
A standard HDD - Just 1 HDD?
Is ZFS being used? ( say yes )
Do you have a ZIL in your pool? ( attach one! ) a - 10G NVMe or SSD will do! (even partitioned)

"consistently make use of caching?" - Keep the host online?

 

[RolandK]

what does '
time (cp /path/file /dest;sync)

vs
time (cp /path/file /dest

tell ?

 

[alexskysilk]

However, quite often the RAM cache is not used and I average 20MB/s. It takes an hour or more.

ram cache wouldnt come into play with a 100GB file. it would help with 100,000,000 1k files- and if you're doing that, cp is not an efficient method to copy.

Odds are your issues are described in your system log- problems with the source or destination storage.

 

[square_eyes]

do you use zfs at destination ?

No ZFS. Will update the original question.

 

[jlauro]

what does '
time (cp /path/file /dest;sync)

vs
time (cp /path/file /dest)

tell ?

or you could simply do:
time cp /path/file /dest ; time sync
and the first timing will be the copy and the second time will be the time for the buffer flush to commit to disk.