[SOLVED] Best Practice ZFS/SW Raid HDD

[cdn123]

Hello,
I‘ve reads lot in this forum but haven‘t find the best way for our setup.

Hardware:
AMD Ryzen™ 5 3600
64 GB DDR4 ECC
4 x 16 TB HDD 7200

We‘re running pve and pbs as VM with a sw raid-5. Additionally we have another VM with some NAS stuff on same node (not that much IO). We‘re backup about 35-40 VMs from 5 different nodes. Used backup space is about 6 TB in pbs but will grow in future.

The pbs is getting real slow in last weeks, so VMs crashed during backups. After reading a lot I changed max-workers to 1 und enable fleecing. The backups will need about 20h. That’s too long.

Do we have a better performance when we‘ll install pbs directly to metal? Should we change sw raid-5 to ZFS raidz1 for getting better performance?
What else can we do to use the pbs with reliable speed to finish in about 4-6 hours?

Alternative we can swap the hardware and add 2x1TB Nvme drives. Read about that special devices should bring a better speed but only in GC. There is no way to use only SSD for Backup (Server is in datacenter).

I‘m looking forward to your help. Thank you!

 

[ness1602]

With raid5 there isnt any performance gains,whatever you do. I have one PBS with it(since beta) and this is pretty much performance from it. If you need faster turnaround get ssds ,or atleast raid10 with special device.

 

[cdn123]

Would raid-5 with special device also bring a performance boost?
We need the disk space…

 

[ness1602]

Yeah it will bring a bit of performance, when vieweing bacukups or similar.

 

[waltar]

With which cmd you created your raid5 ? Did you use LVM also ? Which filesystem do you have on your raid5 (without lvm it's not ext4) ? Raid5 with special device is a xfs feature which isn't needed if you just have vm's/lxc's files as that are just a few files but useful if have 10th millions of files but that's uncommon on proxmox a server ...

 

[cdn123]

mdadm.No lvm. I‘m using ext3. But it‘s okay to recreate the pbs server if it helps due to VM crashes this is no state which we can use.
Yes but we are using the pbs which has millions of files in chunks. Right now the pbs is virtualized in pve as I wrote above.

 

[cdn123]

Yeah it will bring a bit of performance, when vieweing bacukups or similar.

What do you mean with viewing backups? It‘s a lot of work and will cost more money. So I should know if this will give us much better performance.

 

[waltar]

With which cmd you created your raid5 : mdadm -C ... ?
df -i ?

 

[cdn123]

With which cmd you created your raid5 : mdadm -C ... ?
df -i ?

Sorry I don‘t know. Used the hetzner installer image

 

[ness1602]

You won't get better write or access speed with special device. If it is hetzner, demolish the server and bring it up with Raid10 atleast.

 

[cdn123]

You won't get better write or access speed with special device. If it is hetzner, demolish the server and bring it up with Raid10 atleast.

With raid10 I‘ve too less space remaining…
Should I then switch to zfs and using special devices? I have 5 different datastores, can I use the special devices for all of them? Sorry but I cannot find any good tutorial oder documentation about the topic. That’s the point why I still using classic sw raid with ext3.

 

[waltar]

cat /proc/mdstat ?
df && df -i ?

 

[cdn123]

cat /proc/mdstat ?
df && df -i ?

PVE

Personalities : [raid1] [raid6] [raid5] [raid4] [raid0] [raid10]
md0 : active raid1 sdb1[2] sdc1[1] sda1[0] sdd1[3]
      33520640 blocks super 1.2 [4/4] [UUUU]
      
md1 : active raid1 sdb2[2] sda2[0] sdc2[1] sdd2[3]
      1046528 blocks super 1.2 [4/4] [UUUU]
      
md2 : active raid5 sdb3[2] sda3[0] sdc3[1] sdd3[4]
      46773425664 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 25/117 pages [100KB], 65536KB chunk

Filesystem       1K-blocks        Used   Available Use% Mounted on
udev              32841748           0    32841748   0% /dev
tmpfs              6575124        1048     6574076   1% /run
/dev/md2       46586233520 25585421444 18662124412  58% /
tmpfs             32875608       46800    32828808   1% /dev/shm
tmpfs                 5120           0        5120   0% /run/lock
/dev/md1           1012428      263804      696300  28% /boot
/dev/fuse           131072          20      131052   1% /etc/pve
tmpfs              6575120           0     6575120   0% /run/user/0

Filesystem        Inodes IUsed     IFree IUse% Mounted on
udev             8210437   592   8209845    1% /dev
tmpfs            8218902  1053   8217849    1% /run
/dev/md2       730834944 67540 730767404    1% /
tmpfs            8218902   106   8218796    1% /dev/shm
tmpfs            8218902    20   8218882    1% /run/lock
/dev/md1           65408   647     64761    1% /boot
/dev/fuse         262144    38    262106    1% /etc/pve
tmpfs            1643780    20   1643760    1% /run/user/0

PBS (VM)

Dateisystem      1K-Blöcke    Benutzt  Verfügbar Verw% Eingehängt auf
udev               7965924          0    7965924    0% /dev
tmpfs              1599040        724    1598316    1% /run
/dev/sdb1        101639152   10737676   85692288   12% /
tmpfs              7995184          0    7995184    0% /dev/shm
tmpfs                 5120          0       5120    0% /run/lock
/dev/sda       10401954720 6253503832 3624146504   64% /mnt/datastore
tmpfs              1599036          0    1599036    0% /run/user/0

Dateisystem       Inodes IBenutzt     IFrei IUse% Eingehängt auf
udev             1991481      462   1991019    1% /dev
tmpfs            1998796     1332   1997464    1% /run
/dev/sdb1        6496256    81535   6414721    2% /
tmpfs            1998796        1   1998795    1% /dev/shm
tmpfs            1998796        3   1998793    1% /run/lock
/dev/sda       327680000  2956972 324723028    1% /mnt/datastore
tmpfs             399759       14    399745    1% /run/user/0

 

[waltar]

Raid5 "md2" looks good with chunk 512k and internal bitmap on.
Your ~25TB used in md2 are just about small number of 67k files.
Nearly 3M moderate number of files in your pbs vm datastore mount - which is a virtual disk file out of your raid5 ?
You have 2 options without renew everythink - one or both:
a) tune virtual memory and scheduler on host and vm,
b) mv datastore to raid5, export to pbs vm and mount in vm, tune virtual memory, scheduler and nfs.
Further options are swich to xfs or zfs and as you have no additional disk slots special device is no option:
c) If a+b are not sufficient enough switch to xfs on raid5 with datastore onto and mount inside the vm,
d) Try raidz1 ... but as everybody knows single vdev is the slowest zfs ever ... that's nearly a theory.
So think a little about the way you feel to go and by the way install package "tuned" on host and pbs vm at first. Come back if you're done.

 

[cdn123]

Raid5 "md2" looks good with chunk 512k and internal bitmap on.
Your ~25TB used in md2 are just about small number of 67k files.
Nearly 3M moderate number of files in your pbs vm datastore mount - which is a virtual disk file out of your raid5 ?
You have 2 options without renew everythink - one or both:
a) tune virtual memory and scheduler on host and vm,
b) mv datastore to raid5, export to pbs vm and mount in vm, tune virtual memory, scheduler and nfs.
Further options are swich to xfs or zfs and as you have no additional disk slots special device is no option:
c) If a+b are not sufficient enough switch to xfs on raid5 with datastore onto and mount inside the vm,
d) Try raidz1 ... but as everybody knows single vdev is the slowest zfs ever ... that's nearly a theory.
So think a little about the way you feel to go and by the way install package "tuned" on host and pbs vm at first. Come back if you're done.

Thank you!
Just to be sure that I understand you right:
a) virtual memory: you mean the swap partition? My memory in use is about 20%. The rest is free.
b) should this bring a benefit when I „remove“ the layer of virtual disk? I could also directly switch to pve + pbs bare metal. Just using the vm because where I installed Idk that I can install in parallel.
d) yes I read about that raidz1 is slow like a single disk. So I am right that this is slower than a raid5? Right now I‘ve 5 Datastores (on same Raid5). When I create a vdev for each, should I have better performance? The space which is allocated to this vdev is thin provisioned? Sorry I‘m totally newbie in ZFS.
I have the option to switch the server by pay 30€/m more and have 2x1 TB NvmE extra for using as special drive.

 

[Dunuin]

There is no magic you can do to make it fast. PBS simply needs IOPS performance and HDDs can't deliver those. You either need as many small HDDs as possible striped in a raid10 or even better a few big SSDs. A traditional raid5 got single disk IOPS performance as well. So think of it like using a single 64TB HDD that can only handle ~100 IOPS and randomly accessing millions of ~2MB files...not great. A special device only moves metadata to the SSDs. All data is still stored on the slow HDDs.

 

[waltar]

@Dunuin: PBS is backing up files and for files you need streaming of them - first read and then write - which could indeed good done by hdd's. If you have these files in a virtual disks these are just blocks und you need random iops - but hey, that's exactly the designed I/O problem but nonetheless could be moved from block iops to file streaming. Normally the network is the problem, files could be easily read and further preread while even write faster as the network packages so raid5 hdd latency is not a problem.

 

[ness1602]

But this is not how PBS works, PBS doesnt use streaming, or anything inline, blocks are usually scattered on the whole raid, so you count random IOPS, not max.

 

[waltar]

I'm wondering of a user ever requests any random blocks to be restored instead of files ... When PBS don't backup files what is it's usecase ?
PS: I use /dev/urandom when needed random block data as it's easier to maintain as PBS.

 

[waltar]

Does anyone know the difference in linux standard filesystem and arc cache performance ? There's a factor of 4 between ...
Test yourself on 1 same server just with installed os data by eg.:
cd /usr ; tar cf /pool-name/dataset-name/test.tar bin etc include lib lib64 libexec local sbin share src /etc ; l -h /pool-name/dataset-name/test.tar ; dd if=/pool-name/dataset-name/test.tar of=/dev/null bs=32k
cd /usr ; tar cf /xfs-mount/test.tar bin config etc include lib lib64 libexec local sbin share src /etc ; l -h /xfs-mount/test.tar ; dd if=/xfs-mount/test.tar of=/dev/null bs=32k