EXTREMELY SLOW WRITES - 2x Kingston SSD DC450R 960GB / ZFS / Raid

P

prjc

Member
May 31, 2021
11
0
6
41
Hello,
I am long-term fan of Proxmox ecosystem, a lot of work has beed done to bring cloud to masses. As I typically search a lot before posting to forums, I am pretty out of mind now. A problem:
1) Latest Proxmox 6.3 installation on 2x Kingston DC450R rpool.
2) Dell server with 192G RAM, 24x CPU etc... very fast in general
3) only a single VM wih 96G RAM (rest is left for ZFS so it can breathe) running, classic LAMP server on Centos 8, setted up all things like ulimit to raise file pointers limits.
- running only 20 low-traffic websites on WordPress. VM on the same rpool as system
4) tried multiple times to play with arc (returned to defaults then), with writeback cache,discard.
5) zpool trim rpool on daily basis on Proxmox root performed daily, drive full from 30%

Even though traffic is almost ZERO, a server is worth few thousands $ and Kingstons are not Intel DC for 2.000USD and are promoted to be more read-optimized, but I really expected that couple of SSD's from 2021 will easily beat 10 years old 5400RPM Seagates. The reality :
1) when running on config above, the WordPress administration take easily 10-20 seconds to load.
2) when I copy this website to a old-school server with 5400RPM Seagates, I load WP Admin area under 2s.

Did I really miss something when setting up the environment, or are SSD FOR 300$, called ,,DC" suitable just for Win10 and Youtube watching?
I would tip that trim would not come through Raid & ZFS to wipe the empty cells, since I think it used to be faster on the start. But I have not found how to check that.
Thank you, you probably save my day :)

1622452522780.png\

zpool iostat 30
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
rpool 244G 644G 57 136 739K 4.29M
seagate 195G 269G 2 1 600K 291K
---------- ----- ----- ----- ----- ----- -----
rpool 244G 644G 0 116 1.33K 3.44M
seagate 195G 269G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
rpool 244G 644G 0 85 409 2.02M
seagate 195G 269G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
rpool 244G 644G 0 114 409 2.87M
seagate 195G 269G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
rpool 244G 644G 0 104 2.27K 2.52M
seagate 195G 269G 0 0 0 0
---------- ----- ----- ----- ----- ----- -----
rpool 244G 644G 0 105 136 2.30M
seagate 195G 269G 0 0 0 0
 
It got no powerloss protection, bad durability (0,3 DWPD) and slow 4k write iops (17000-26000 IOPS). I don't really understand why it is marketed as a DC drive. On the paper it's just a prosumer drive...and not a fast one...but atleast they tell you that writes are bad.
 
Last edited:
I know, but even 17000 IOPS should perform better. I do not talk about highend database systems, but if SSD of this kind load 10x slower than old Seagates, something is wrong (probably with the market)
 
The missing powerloss protection could be a problem. Keep in mind that a drive can't cache sync writes if there is no powerloss protection. If you are running Wordpress you are running a MySQL db that is doing sync writes. So these db writes can't be done in parallel nor be optimized.
 
yy i know this, the usecase for them is fine. The websites have no problem to restore backup from X hours back elsewhere in case of problem. The problem is just the speed here.
 
Oh, i read it wrong. So you probably answered my question :( Thank you. So they are useless for that case..
 
But even with bad sync write performance it should be better than a 5400RPM HDD.

Did you flash the SSDs firmware to the lastest version?
Is your pool using the right ashift value?
Is your volblocksize optimized?
Did you optimize your php-fpm and mysql?
Is your VM using virtio SCSI?
Discard checkbox in VM settings is checked and you enabled discard in your guests fstab or are running fstrim -a on a daily/weekly basis inside your guest too?
 
Did you flash the SSDs firmware to the lastest version?
+++++ yes
Is your pool using the right ashift value?
++++ eeeh
Is your volblocksize optimized?
++++ eeeh
Did you optimize your php-fpm and mysql?
++++ yes, this is a level I understand pretty well
Is your VM using virtio SCSI?
++++ yes
Discard checkbox in VM settings is checked and you enabled discard in your guests fstab or are running fstrim -a on a daily/weekly basis inside your guest too?
++++ yes, discard setted up and fstrim -a in cron daily.

the rest it better like that:


root@dell:~# zpool get all | grep rpool
rpool size 888G -
rpool capacity 12% -
rpool altroot - default
rpool health ONLINE -
rpool guid 13854598641423436058 -
rpool version - default
rpool bootfs rpool/ROOT/pve-1 local
rpool delegation on default
rpool autoreplace off default
rpool cachefile - default
rpool failmode wait default
rpool listsnapshots off default
rpool autoexpand off default
rpool dedupditto 0 default
rpool dedupratio 1.00x -
rpool free 774G -
rpool allocated 114G -
rpool readonly off -
rpool ashift 12 local
rpool comment - default
rpool expandsize - -
rpool freeing 0 -
rpool fragmentation 25% -
rpool leaked 0 -
rpool multihost off default
rpool checkpoint - -
rpool load_guid 4949649479488749332 -
rpool autotrim off default
rpool feature@async_destroy enabled local
rpool feature@empty_bpobj active local
rpool feature@lz4_compress active local
rpool feature@multi_vdev_crash_dump enabled local
rpool feature@spacemap_histogram active local
rpool feature@enabled_txg active local
rpool feature@hole_birth active local
rpool feature@extensible_dataset active local
rpool feature@embedded_data active local
rpool feature@bookmarks enabled local
rpool feature@filesystem_limits enabled local
rpool feature@large_blocks enabled local
rpool feature@large_dnode enabled local
rpool feature@sha512 enabled local
rpool feature@skein enabled local
rpool feature@edonr enabled local
rpool feature@userobj_accounting active local
rpool feature@encryption enabled local
rpool feature@project_quota active local
rpool feature@device_removal enabled local
rpool feature@obsolete_counts enabled local
rpool feature@zpool_checkpoint enabled local
rpool feature@spacemap_v2 active local
rpool feature@allocation_classes enabled local
rpool feature@resilver_defer enabled local
rpool feature@bookmark_v2 enabled local
 

Attachments

  • zfs.dell.txt
    22.1 KB · Views: 2
Use Kingston DC500M (so not 450R, never "R" series, and 500 is better than 450 series) that is mixed use optimized and has powerloss protection. You as asking for "how to make 450R work better", but I think your starting point is hopeless in any case, and if you can effort "Dell server with 192G RAM, 24x CPU" you can spend some cash for 2 new SSD.
My pveperf on a single 1TB, zfs with dc500M is:
FSYNCS/SECOND: 9631.21
 
Guys, there must be a real issue somewhere. I know 450 is a piece of sh... now, but listen:
pveperf on (another boxes I have access to):

GIGABYTE GP-ASACNE2100TTTDR card : 30.000 fsyncs
Samsung SSD 860 - 20.000

My Kingstons - I am waiting almost 5 minutes and pveperf is unable to give me the results. It just hang after
CPU BOGOMIPS: 105780.00
REGEX/SECOND: 1353592
HD SIZE: 765.13 GB (rpool/ROOT/pve-1)
......
waiting..
Any clues? Thanks
 
Try to move online one disk from zfs to ext4 and relax :)
Zfs on linux can't be used in production.
 
guletz said:
Respond on my previous question and maybe we can find some better setup for your case
Click to expand...
If you mean if I do know the difference between ondemand/static FPM settings etc, apache fine-tuning parameters, this is already done - apache and PHP are both sleeping and have sufficient resources. The same configuration on a different server respond very quicky, with 5x more websites... Everything is sleeping on that machine including disk IO. But, the system is not able finish pveperf in 1 hour with just one very low-traffic VM running. So I assume apache+php on the upper SW level is not the problem.
 
hello,
dmesg or messages nothing special there

root@dell:~# fio --ioengine=libaio --filename=/dev/sdc --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio
fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=1692KiB/s][w=423 IOPS][eta 00m:00s]
fio: (groupid=0, jobs=1): err= 0: pid=11375: Tue Jun 1 14:57:35 2021
write: IOPS=433, BW=1733KiB/s (1775kB/s)(102MiB/60003msec); 0 zone resets
slat (usec): min=6, max=105, avg=15.49, stdev= 5.70
clat (usec): min=2009, max=14334, avg=2290.25, stdev=598.91
lat (usec): min=2021, max=14346, avg=2306.00, stdev=599.07
clat percentiles (usec):
| 1.00th=[ 2114], 5.00th=[ 2147], 10.00th=[ 2180], 20.00th=[ 2180],
| 30.00th=[ 2212], 40.00th=[ 2212], 50.00th=[ 2245], 60.00th=[ 2245],
| 70.00th=[ 2278], 80.00th=[ 2311], 90.00th=[ 2376], 95.00th=[ 2442],
| 99.00th=[ 2704], 99.50th=[ 3097], 99.90th=[13829], 99.95th=[13960],
| 99.99th=[13960]
bw ( KiB/s): min= 1624, max= 1816, per=100.00%, avg=1732.87, stdev=41.98, samples=120
iops : min= 406, max= 454, avg=433.22, stdev=10.50, samples=120
lat (msec) : 4=99.72%, 10=0.02%, 20=0.25%
cpu : usr=0.25%, sys=0.60%, ctx=51917, majf=7, minf=11
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,25995,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
WRITE: bw=1733KiB/s (1775kB/s), 1733KiB/s-1733KiB/s (1775kB/s-1775kB/s), io=102MiB (106MB), run=60003-60003msec

Disk stats (read/write):
sdc: ios=351/27745, merge=0/11, ticks=766/62390, in_queue=840, util=98.79%
 
Guys,
I completely wiped that ZFS installation, installed it again on EXT4 - one SSD for a system, second one for VM.. Pretty oldschool one should say.
The same VM with Centos 8, without ANY change inside the VM.

Result?
Load times from 20 seconds to 0.5 in admin area, bloody fast reading times
Probably ZFS is not for me, I understand its enterprise stuff, but setting it up is a real mess.
It's not about lowend SSD, even though they will not survive as long as better ones.
Bye Bye ZFS :(
 
prjc said:
Probably ZFS is not for me, I understand its enterprise stuff, but setting it up is a real mess.
Click to expand...

Hi,

No, is not any mess here, but you need to take your time and to understand how zfs is working. And I am agree that is not so easy as many new user think.

But if you want to invest time learning zfs, you will also learn many other things who can be useful in the future.

Is the same like linux/windows users. It is hard for a windows users to use linux, and could be a mess for many .... ;)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back