[SOLVED] Shared cache and logs between two pools

WhiteTiger

Member
May 16, 2020
86
2
13
Italy
I have a zpool with three volumes in RAID-Z2, each with 7 disks.
I created the Zpool with volumes with the same number of disks.
This zpool is associated with a mirrored log and a cache, by NVMe.

I have two other unused HDDs and would like to create a new mirror pool.
Can I bind to this the same mirrored log and the cache used with the other zpool?
If not how can I use these two HDDs.
 
Last edited:
I have a zpool with three volumes in RAID-Z2, each with 7 disks.
I created the Zpool with volumes with the same number of disks.
This zpool is associated with a mirrored log and a cache, by NVMe.
So you basically got a single pool of 3x raidz2 of 7 disks each striped together?
If yes keep in mind to change the volblocksize from the default of 8K to somethign like 64K or above or you will waste half of the total capacity due to padding overhead.
Can I bind to this the same mirrored log and the cache used with the other zpool?
First you don't need to mirror a cache disk. Your L2ARC is only a read cache so if you loose it you wont loose any data. For the SLOG a mirror can be useful in the unlikely case that your SLOG drive is dying at the same time a power outage occures.

Second you should consider if using a L2ARC is useful in your case. The ARC in RAM is faster than a L2ARC on SSD and the more L2ARC you use the less ARC in RAM will be usable. So using a L2ARC you are basically sacrificing a bit of very fast read cache to get more but slower read cache. Its often said that L2ARC should only be used if you already maxed out your RAM, still need more read cache but can't add more because you ran out of RAM slots. So you should check if a L2ARC is actually helping you because it also can have a negative effect. If I remember right your L2ARC shouln't be higher than 10 times the size of your ARC.

And you don't need to add complete drives as SLOG or L2ARC. You can also use partitions. So you can for example partition both SSDs the same way using CLI with three partitions each. For example partition one and two using 10% of the capacity and partition three 80%. You can then use both first partitions as a mirrored SLOG for one pool, both second partitions as a mirrored SLOG for the second pool. The third partition of one of the SSDs (without mirroring) as a L2ARC for your first pool and the third partition of the other SSD as the L2ARC for your second pool.

But if you don't got alot of sync writes I guess using the SSDs as a special device would make more sense. But then a mirror of 3 SSDs would be best so that it fits the reliability of your raidz2 HDDs. Because if you loose the special devices all data on all HDDs is lost so a good redundancy is important.

If not how can I use these two HDDs.
You could use one of them as a hot spare for your pool and put the other one somewhere near so you already got one if you need to replace a disk. If one of your drives fails and there is a hot spare the pool will start the resilvering using that hot space disk as a temporary replacement for the failed disk. You then replace the failed disk with a new one (which you already have laying around), the hotspare will copy its data over to the new disk and will continue to work as a hot spare waiting for the next disk to die.
 
Last edited:
  • Like
Reactions: WhiteTiger
f yes keep in mind to change the volblocksize from the default of 8K to somethign like 64K or above or you will waste half of the total capacity due to padding overhead
What is the option in zpool to create blocksize = 64? and what value should ashift have?
With zpool create -f -o ashift=16 -o volblocksize=64K there is the error "property 'volblocksize' is not a valid pool property".
 
Last edited:
What is the option in zpool to create blocksize = 64? and what value should ashift have?
With zpool create -f -o ashift=16 -o volblocksize=64K there is the error "property 'volblocksize' is not a valid pool property".
Are you sure you want a ashift of 16? That means you want each of your HDDs to work with a 64K blocksize. And your volblocksize should atleast be 8 times the size of your disks blocksize, so the volblocksize would need to be atleast 512K if you don't want to waste too much capacity.

What ashift would make sense depends on the HDDs you are using. In general a ashift of 12 (resulting in HDDs operating with 4K blocksize) should be fine. The volblocksize you can't define for your pool. It needs to be defined for every single zvol at creation that you create. Proxmox will handle the vollbocksize for you but you need to tell it what volblocksize it should give all your newly created zvols. You can find that option under "Datacenter -> Storage -> YourZfsPool -> Edit -> Blocksize".
 
  • Like
Reactions: WhiteTiger
The volblocksize you can't define for your pool. It needs to be defined for every single zvol at creation that you create. Proxmox will handle the vollbocksize for you but you need to tell it what volblocksize it should give all your newly created zvols. You can find that option under "Datacenter -> Storage -> YourZfsPool -> Edit -> Blocksize".
OK, ashift = 12.

Zpool created from command line.
Then in Datacenter/Storage I added the pool created with "Thin Provisioning" option.
Finally, I went into edit and changed the blocksize 64k

Is there a command to view the pool configuration to see if everything is fine?
 
hello i have the same situation a raidz2 with 6 disk and a cache ssd disk:

Code:
~ zpool status
  pool: zfsfiles
 state: ONLINE
  scan: scrub repaired 0B in 02:16:37 with 0 errors on Sun Dec 12 02:40:38 2021
config:

    NAME                                         STATE     READ WRITE CKSUM
    zfsfiles                                     ONLINE       0     0     0
      raidz2-0                                   ONLINE       0     0     0
        ata-ST1000LM014-1EJ164-SSHD_W773JW09     ONLINE       0     0     0
        ata-ST1000LM014-1EJ164-SSHD_W773CFA9     ONLINE       0     0     0
        ata-HGST_HTS721010A9E630_JR10006P2H0X5F  ONLINE       0     0     0
        ata-ST1000LM049-2GH172_WGS70T3R          ONLINE       0     0     0
        ata-ST1000LM048-2E7172_WN93AZ5G          ONLINE       0     0     0
        ata-ST1000LM049-2GH172_WGS70BYW          ONLINE       0     0     0
    cache
      ata-SAMSUNG_SSD_830_Series_S0Z4NEAC827940  ONLINE       0     0     0

errors: No known data errors

i have 2 additional spare disks and i want to create a mirror, do i have to add also another cache disk or i can i use the one existing?
 
You can use a disk as a cache for multiple pools but only if you partition it. So you would need to remove the cache disk from your raidz2 pool, partition it (for example two partitions using 50% of the size each) and then you can add one partition as a cache to your raidz2 pool and the other one to your mirror pool.
 
You can use a disk as a cache for multiple pools but only if you partition it. So you would need to remove the cache disk from your raidz2 pool, partition it (for example two partitions using 50% of the size each) and then you can add one partition as a cache to your raidz2 pool and the other one to your mirror pool.
So I have to remove the cache disk from the pool and create 2 ext4 partition?
 
Yes. But you don't need to format the partitions. Just plain empty partitions. I guess you need to do that using the CLI.
 
Code:
~ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 223.6G  0 disk
├─sda1   8:1    0 215.6G  0 part /
├─sda2   8:2    0     1K  0 part
└─sda5   8:5    0     8G  0 part [SWAP]
sdb      8:16   0 238.5G  0 disk
├─sdb1   8:17   0   110G  0 part
└─sdb2   8:18   0   110G  0 part
sdc      8:32   0   1.8T  0 disk
├─sdc1   8:33   0   1.8T  0 part
└─sdc9   8:41   0     8M  0 part
sdd      8:48   0   1.8T  0 disk
├─sdd1   8:49   0   1.8T  0 part
└─sdd9   8:57   0     8M  0 part
sde      8:64   0 298.1G  0 disk
└─sde1   8:65   0 298.1G  0 part /srv/dev-disk-by-label-NoData
sdf      8:80   1 931.5G  0 disk
├─sdf1   8:81   1 931.5G  0 part
└─sdf9   8:89   1     8M  0 part
sdg      8:96   1 931.5G  0 disk
├─sdg1   8:97   1 931.5G  0 part
└─sdg9   8:105  1     8M  0 part
sdh      8:112  1 931.5G  0 disk
├─sdh1   8:113  1 931.5G  0 part
└─sdh9   8:121  1     8M  0 part
sdi      8:128  1 931.5G  0 disk
├─sdi1   8:129  1 931.5G  0 part
└─sdi9   8:137  1     8M  0 part
sdj      8:144  1 931.5G  0 disk
├─sdj1   8:145  1 931.5G  0 part
└─sdj9   8:153  1     8M  0 part
sdk      8:160  1 931.5G  0 disk
├─sdk1   8:161  1 931.5G  0 part
└─sdk9   8:169  1     8M  0 part



~ fdisk /dev/sdb -l
Disk /dev/sdb: 238.5 GiB, 256059432448 bytes, 500116079 sectors
Disk model: SAMSUNG SSD 830
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C294A829-A5FC-7E48-8460-BDE9B0DB06D3

Device         Start       End   Sectors  Size Type
/dev/sdb1       2048 230688767 230686720  110G Linux filesystem
/dev/sdb2  230688768 461375487 230686720  110G Linux filesystem
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!