Hi,
Could anybody help me to pinpoint the source of my issue?
My issue is very bad disk speed for any drive.
My server configuration:
Dell R730XD E5-2680v4*2, 256GB RAM, PercH730 in HBA mode.
Disks:
Two SATA SSD, 256 GB each, in configuration ZFS RAID1 as OS drive.
6 SAS HDD,6TB each, in configuration ZFS RAIDz2 as “slow storage”
Two NVMe drives 1TB, I’ve installed 6 NVMe drives to make some tests.
Could anybody share experience on how to make proper configuration for ZFS and Proxmox on ZFS?
When I do fio tests on NVMe that configured as LVM, ZFS is not faster I just don’t want to mix issues, I have very bad results.
Here is how I did tests on my drives:
fio --bs=4k --ioengine=libaio --iodepth=512 --size=2g --group_reporting --name=nvme_4k_read --numjobs=16 --rw=randread --direct=0
I get these results:
Single NVME drive:
READ: bw=508MiB/s (532MB/s), 508MiB/s-508MiB/s (532MB/s-532MB/s), io=32.0GiB (34.4GB), run=64544-64544msec Disk stats (read/write): dm-0: ios=8384144/3, merge=0/0, ticks=732496/0, in_queue=732496, util=99.93%, aggrios=8388668/3, aggrmerge=0/0, aggrticks=759799/1, aggrin_queue=204, aggrutil=99.77% nvme0n1: ios=8388668/3, merge=0/0, ticks=759799/1, in_queue=204, util=99.77%
Stripe of two NVMe drives:
READ: bw=644MiB/s (675MB/s), 644MiB/s-644MiB/s (675MB/s-675MB/s), io=32.0GiB (34.4GB), run=50895-50895msec Disk stats (read/write): dm-2: ios=8388078/0, merge=0/0, ticks=588144/0, in_queue=588144, util=99.89%, aggrios=4194324/0, aggrmerge=30/0, aggrticks=311457/0, aggrin_queue=26, aggrutil=99.10% nvme1n1: ios=4194324/0, merge=30/0, ticks=326139/0, in_queue=0, util=99.10% nvme2n1: ios=4194324/0, merge=30/0, ticks=296775/0, in_queue=52, util=99.09%
and in any combination results are not better but if I use switch --direct=1 I get proper results like
READ: bw=2913MiB/s (3054MB/s) for a single drive
READ: bw=4629MiB/s (4854MB/s) for a stripe of two NVMe
I’ve changed only one thing in ZFS configuration that is:
/etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=25769803776
options zfs zfs_arc_min=8589934592
limit ARC to 24 GB since from ZFS I need snapshots and replication feature.
just an example of how bad performs my OS drive is that to run the test, it took 27m56.508s with laying out IO files and 1m18.233s without laying out IO files(second run)
Could anybody help me to pinpoint the source of my issue?
My issue is very bad disk speed for any drive.
My server configuration:
Dell R730XD E5-2680v4*2, 256GB RAM, PercH730 in HBA mode.
Disks:
Two SATA SSD, 256 GB each, in configuration ZFS RAID1 as OS drive.
6 SAS HDD,6TB each, in configuration ZFS RAIDz2 as “slow storage”
Two NVMe drives 1TB, I’ve installed 6 NVMe drives to make some tests.
Could anybody share experience on how to make proper configuration for ZFS and Proxmox on ZFS?
When I do fio tests on NVMe that configured as LVM, ZFS is not faster I just don’t want to mix issues, I have very bad results.
Here is how I did tests on my drives:
fio --bs=4k --ioengine=libaio --iodepth=512 --size=2g --group_reporting --name=nvme_4k_read --numjobs=16 --rw=randread --direct=0
I get these results:
Single NVME drive:
READ: bw=508MiB/s (532MB/s), 508MiB/s-508MiB/s (532MB/s-532MB/s), io=32.0GiB (34.4GB), run=64544-64544msec Disk stats (read/write): dm-0: ios=8384144/3, merge=0/0, ticks=732496/0, in_queue=732496, util=99.93%, aggrios=8388668/3, aggrmerge=0/0, aggrticks=759799/1, aggrin_queue=204, aggrutil=99.77% nvme0n1: ios=8388668/3, merge=0/0, ticks=759799/1, in_queue=204, util=99.77%
Stripe of two NVMe drives:
READ: bw=644MiB/s (675MB/s), 644MiB/s-644MiB/s (675MB/s-675MB/s), io=32.0GiB (34.4GB), run=50895-50895msec Disk stats (read/write): dm-2: ios=8388078/0, merge=0/0, ticks=588144/0, in_queue=588144, util=99.89%, aggrios=4194324/0, aggrmerge=30/0, aggrticks=311457/0, aggrin_queue=26, aggrutil=99.10% nvme1n1: ios=4194324/0, merge=30/0, ticks=326139/0, in_queue=0, util=99.10% nvme2n1: ios=4194324/0, merge=30/0, ticks=296775/0, in_queue=52, util=99.09%
and in any combination results are not better but if I use switch --direct=1 I get proper results like
READ: bw=2913MiB/s (3054MB/s) for a single drive
READ: bw=4629MiB/s (4854MB/s) for a stripe of two NVMe
I’ve changed only one thing in ZFS configuration that is:
/etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=25769803776
options zfs zfs_arc_min=8589934592
limit ARC to 24 GB since from ZFS I need snapshots and replication feature.
just an example of how bad performs my OS drive is that to run the test, it took 27m56.508s with laying out IO files and 1m18.233s without laying out IO files(second run)
