[Suggestions] 8HDD's + 5NVME's (Z2 or Raid10 or something else)

[LnxBil]

Why don't you just have one pool with all the NVMe as special devices? Install on you harddisks and add the nvme as special devices afterwards. You then need to set special_small_blocks to 128K and send/receice your root pool to move it to the nvme.

 

[Ramalama]

Why don't you just have one pool with all the NVMe as special devices? Install on you harddisks and add the nvme as special devices afterwards. You then need to set special_small_blocks to 128K and send/receice your root pool to move it to the nvme.

Because its not managable here.

1. I need an mdadm raid10 (ext4) partition for docker as volume, since docker experience with zfs is horrible.
2. All VM's can run directly on the nvme storage, so im not wasting the "speedy storage"
Alone for the first point, i need to partition anyway.

I have now on my 4x2tb nvme drives:
2TB Zfs-Raid10 for Root+VM's
1,3TB mdadm-Raid10 (ext4 Directory) for Docker Containers
700GB ZFS-Raid10 as special vdev for the "HDD_Z2" 104TB HDD Pool.

special_small_blocks=64K & recordsize=256K for the "HDD_Z2" Pool.

Did an reinstall, configured everything again, now im done and happy xD

  pool: HDD_Z2
 state: ONLINE
config:

    NAME                                  STATE     READ WRITE CKSUM
    HDD_Z2                                ONLINE       0     0     0
      raidz2-0                            ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7M8RA  ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7ER9A  ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7KSTA  ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7XKNA  ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG64ZHE  ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7WHTE  ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7Y79A  ONLINE       0     0     0
        ata-WDC_WUH722020BLE6L4_8LG7VRXA  ONLINE       0     0     0
    special
      mirror-1                            ONLINE       0     0     0
        nvme0n1p5                         ONLINE       0     0     0
        nvme1n1p5                         ONLINE       0     0     0
      mirror-2                            ONLINE       0     0     0
        nvme3n1p5                         ONLINE       0     0     0
        nvme2n1p5                         ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
config:

    NAME                                 STATE     READ WRITE CKSUM
    rpool                                ONLINE       0     0     0
      mirror-0                           ONLINE       0     0     0
        nvme-eui.0025384431415a66-part3  ONLINE       0     0     0
        nvme-eui.0025384431414875-part3  ONLINE       0     0     0
      mirror-1                           ONLINE       0     0     0
        nvme-eui.00253844314008c6-part3  ONLINE       0     0     0
        nvme-eui.00253844314008e0-part3  ONLINE       0     0     0

errors: No known data errors

Filesystem        Size  Used Avail Use% Mounted on
udev               63G     0   63G   0% /dev
tmpfs              13G  1.4M   13G   1% /run
rpool/ROOT/pve-1  1.8T  1.4G  1.8T   1% /
tmpfs              63G   46M   63G   1% /dev/shm
tmpfs             5.0M     0  5.0M   0% /run/lock
/dev/md127        1.3T   48K  1.3T   1% /NVME_EXT4_R10
rpool             1.8T  128K  1.8T   1% /rpool
rpool/ROOT        1.8T  128K  1.8T   1% /rpool/ROOT
rpool/data        1.8T  128K  1.8T   1% /rpool/data
/dev/fuse         128M   16K  128M   1% /etc/pve
HDD_Z2            104T  256K  104T   1% /HDD_Z2
tmpfs              13G     0   13G   0% /run/user/0

Thanks everyone!
Now i have just to migrate my data back from the old drives xD

 

[LnxBil]

1. I need an mdadm raid10 (ext4) partition for docker as volume, since docker experience with zfs is horrible.

What? It's perfect and I would never use anything else but ZFS. Don't know where you get your false information.

2. All VM's can run directly on the nvme storage, so im not wasting the "speedy storage"

I don't get that. Isn't your NVMe your "speedy storage"?

 

[Ramalama]

What? It's perfect and I would never use anything else but ZFS. Don't know where you get your false information.

My Own findings, its works in general in an unpriviliged container with keyctl & nesting in zfs.
And it even uses overlay2 as driver since some latest Docker releases, but its slow.
Additionally you get some overlay2 errors in your logs.

However since some updates it works at all on zfs, in the earlier days, like 6 months ago, you had luck when you got it running. In general it didn't worked inside either an privileged or unprivileged container (don't remember exactly)

However, i always simply switched to an ext4 (lvm) partition, where docker runs inside an lxc container without any issues nearly 2times faster.
And im not changing that route, since i never had good experience with docker and zfs.

I don't get that. Isn't your NVMe your "speedy storage"?

It is, but if i add all 4 nvme drives completely as special devices to the big HDD pool, i loose the ability to manage myself what runs on the nvme's and what not.
At least managing gets harder.

 

[Dunuin]

It is, but if i add all 4 nvme drives completely as special devices to the big HDD pool, i loose the ability to manage myself what runs on the nvme's and what not.
At least managing gets harder.

Thats what the sepecial_small_blocks is for. Set specia_small_blocks=1M for a dataset or zvol and all data+metadata of that dataset/zvols will be stored on the NVMe SSDs only. Set special_small_blocks=0 for a dataset/zvol and all metadata will be stored on the NVMe SSDs and all data on the HDDs. Set special_small_blocks something in between and it depends on the volblocksize of the zvols and the size of the files of a dataset.
So you could use that for each single zvol/dataset to decide if it should only be stored on the HDDs, SSDs or a mix of them.

 

[LnxBil]

My Own findings, its works in general in an unpriviliged container with keyctl & nesting in zfs.
And it even uses overlay2 as driver since some latest Docker releases, but its slow.
Additionally you get some overlay2 errors in your logs.

However since some updates it works at all on zfs, in the earlier days, like 6 months ago, you had luck when you got it running. In general it didn't worked inside either an privileged or unprivileged container (don't remember exactly)

However, i always simply switched to an ext4 (lvm) partition, where docker runs inside an lxc container without any issues nearly 2times faster.

Oh okay ... container ... yeah .. then you don't have all the nice features you would get if you would use ZFS as your backing storage driver.

 

[Ramalama]

Oh okay ... container ... yeah .. then you don't have all the nice features you would get if you would use ZFS as your backing storage driver.

Running natively on the host, is sadly not a solution here, since im running 3 lxc docker containers, because of security reasons.
One of the is exposed public.
Tho it runs all through traefik, but anyway i secure myself as much i can

Cheers

 

[LnxBil]

Running natively on the host, is sadly not a solution here, since im running 3 lxc docker containers, because of security reasons.
One of the is exposed public.
Tho it runs all through traefik, but anyway i secure myself as much i can

LXC + Docker + Security? You can have many reasons to run Docker inside of an LX(C) container, but security it not one of them. Go with KVM, everything else is not secure at all.

 

[Ramalama]

LXC + Docker + Security? You can have many reasons to run Docker inside of an LX(C) container, but security it not one of them. Go with KVM, everything else is not secure at all.

There are ways to escape docker inside the container, even escaping an unprivileged container.
But you need first to hack your way through opnsense (port forward to traefik) only port 80&443 (http protocol).
I wouldn't say it's impossible, to gain somehow access, but at least it's very hard.

It's a lot easier to get docker/traefik to crash through the http protocol via ddos etc...
And for that im running it in a separate lxc container.

Even tho i have 128gb ram, it makes no sense for me to run docker inside an VM.

Cheers

 

[Dunuin]

But you need first to hack your way through opnsense (port forward to traefik) only port 80&443 (http protocol).

Unless you run a compromized container because you just deploy a container someone else created instead of creating your own containers or at least auditing all the code for backdoors or malware each time before deploying/updating them. Wouldn't be the first time a developers docker hub account gets hacked and the attacker is then uploading compromized containers there, thousands of people then are deploying.

 

[LnxBil]

Unless you run a compromized container because you just deploy a container someone else created instead of creating your own containers or at least auditing all the code for backdoors or malware each time before deploying/updating them. Wouldn't be the first time a developers docker hub account gets hacked and the attacker is then uploading compromized containers there, thousands of people then are deploying.

Exactly this. You beat me to the punch

 

[Ramalama]

Unless you run a compromized container because you just deploy a container someone else created instead of creating your own containers or at least auditing all the code for backdoors or malware each time before deploying/updating them. Wouldn't be the first time a developers docker hub account gets hacked and the attacker is then uploading compromized containers there, thousands of people then are deploying.

Well Dunuin, you thought 5 steps ahead xD
Sure if i get an compromized traefik container, im fu**ed.

But really, if you run your docker inside an VM, on most installations, the VM isnt inside an DMZ, and the Guy is inside your network anyway.

However, thats a risk i take, otherwise that discussion will never end.
If someone wants to be super secure, an closed down network without access to anything other then the FW for Port-Forwarding and Docker inside a VM, is the way to go.

Cheers

 

[LnxBil]

But really, if you run your docker inside an VM, on most installations, the VM isnt inside an DMZ, and the Guy is inside your network anyway.

Of course it can be in the DMZ. This depends on what you're doing inside of your PVE and how well you define your firewalling. Docker inside of a VM is per se much more secure than running inside of an LX(C) container, so it should be the way to go.

If someone wants to be super secure, an closed down network without access to anything other then the FW for Port-Forwarding and Docker inside a VM, is the way to go.

Of course it is and it should be the way to go. This is what we (@Dunuin and me) are talking about. You should also filter your ingress and egress.