Container creation takes ages

May 21, 2020
54
13
13
35
Hello everyone,

I started spinning up a turnkey nextclound instance on my HP Microserver Gen10+ with G5420 CPU and 16GB RAM.

It's already 8 min in and so far the allocated space from the container on my 3way ZFS mirror is

Code:
data/nextCloudApplicationContainer                                    382M   384G      200K  /data/nextCloudApplicationContainer
data/nextCloudApplicationContainer/subvol-101-disk-0   382M  99.6G      382M  /data/nextCloudApplicationContainer/subvol-101-disk-0

In only grows ~1-2MB/s
The image's .tar.gz is 461M

The ZIP and target volume for the container are on the same pool

Code:
    NAME                                            STATE     READ WRITE CKSUM
    data                                               ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        ata-WDC_WD4000FYYZ-03UL1B1_WD-WMC130D35YFK  ONLINE       0     0     0
        ata-ST4000NE001-2MA101_WJG27C1C                                ONLINE       0     0     0
        ata-WDC_WD40EZRZ-00GXCB0_WD-WCC7K1CZY7FP       ONLINE       0     0     0

How do I analyze what exactly is taking so long? Maybe it is allocating the 100GB which I gave the container? Or is it the deduplication? I also wonder whether the dedup table is too large for the RAM because the whole system only uses 5GB. But if the dedup table is too large, shouldn't at least a portion of it stay in RAM and almost all RAM be used up right now?



The ZFS properties for the source where the .tar.gz is placed are:

Code:
NAME                PROPERTY              VALUE                  SOURCE
data/docker-images  type                  filesystem             -
data/docker-images  creation              Sat Jan  9 21:03 2021  -
data/docker-images  used                  1.99G                  -
data/docker-images  available             382G                   -
data/docker-images  referenced            461M                   -
data/docker-images  compressratio         1.00x                  -
data/docker-images  mounted               yes                    -
data/docker-images  quota                 none                   default
data/docker-images  reservation           100G                   local
data/docker-images  recordsize            128K                   default
data/docker-images  mountpoint            /data/docker-images    default
data/docker-images  sharenfs              off                    default
data/docker-images  checksum              on                     local
data/docker-images  compression           off                    default
data/docker-images  atime                 off                    local
data/docker-images  devices               on                     default
data/docker-images  exec                  on                     default
data/docker-images  setuid                on                     default
data/docker-images  readonly              off                    default
data/docker-images  zoned                 off                    default
data/docker-images  snapdir               hidden                 default
data/docker-images  aclinherit            restricted             default
data/docker-images  createtxg             2542                   -
data/docker-images  canmount              on                     default
data/docker-images  xattr                 sa                     inherited from data
data/docker-images  copies                1                      default
data/docker-images  version               5                      -
data/docker-images  utf8only              off                    -
data/docker-images  normalization         none                   -
data/docker-images  casesensitivity       sensitive              -
data/docker-images  vscan                 off                    default
data/docker-images  nbmand                off                    default
data/docker-images  sharesmb              off                    default
data/docker-images  refquota              none                   default
data/docker-images  refreservation        none                   default
data/docker-images  guid                  6707254990687142115    -
data/docker-images  primarycache          all                    default
data/docker-images  secondarycache        all                    default
data/docker-images  usedbysnapshots       1.54G                  -
data/docker-images  usedbydataset         461M                   -
data/docker-images  usedbychildren        0B                     -
data/docker-images  usedbyrefreservation  0B                     -
data/docker-images  logbias               latency                default
data/docker-images  objsetid              419                    -
data/docker-images  dedup                 on                     local
data/docker-images  mlslabel              none                   default
data/docker-images  sync                  standard               default
data/docker-images  dnodesize             legacy                 default
data/docker-images  refcompressratio      1.00x                  -
data/docker-images  written               461M                   -
data/docker-images  logicalused           1.98G                  -
data/docker-images  logicalreferenced     461M                   -
data/docker-images  volmode               default                default
data/docker-images  filesystem_limit      none                   default
data/docker-images  snapshot_limit        none                   default
data/docker-images  filesystem_count      none                   default
data/docker-images  snapshot_count        none                   default
data/docker-images  snapdev               hidden                 default
data/docker-images  acltype               posixacl               inherited from data
data/docker-images  context               none                   default
data/docker-images  fscontext             none                   default
data/docker-images  defcontext            none                   default
data/docker-images  rootcontext           none                   default
data/docker-images  relatime              off                    default
data/docker-images  redundant_metadata    all                    default
data/docker-images  overlay               off                    default
data/docker-images  encryption            aes-256-gcm            -
data/docker-images  keylocation           none                   default
data/docker-images  keyformat             passphrase             -
data/docker-images  pbkdf2iters           342K                   -
data/docker-images  encryptionroot        data                   -
data/docker-images  keystatus             available              -
data/docker-images  special_small_blocks  0                      default

The ZFS properties for the destination are:

Code:
NAME                                PROPERTY              VALUE                                SOURCE
data/nextCloudApplicationContainer  type                  filesystem                           -
data/nextCloudApplicationContainer  creation              Sat Jan 23  9:43 2021                -
data/nextCloudApplicationContainer  used                  895M                                 -
data/nextCloudApplicationContainer  available             384G                                 -
data/nextCloudApplicationContainer  referenced            200K                                 -
data/nextCloudApplicationContainer  compressratio         1.45x                                -
data/nextCloudApplicationContainer  mounted               yes                                  -
data/nextCloudApplicationContainer  quota                 none                                 default
data/nextCloudApplicationContainer  reservation           100G                                 local
data/nextCloudApplicationContainer  recordsize            16K                                  local
data/nextCloudApplicationContainer  mountpoint            /data/nextCloudApplicationContainer  default
data/nextCloudApplicationContainer  sharenfs              off                                  default
data/nextCloudApplicationContainer  checksum              on                                   local
data/nextCloudApplicationContainer  compression           on                                   local
data/nextCloudApplicationContainer  atime                 on                                   default
data/nextCloudApplicationContainer  devices               on                                   default
data/nextCloudApplicationContainer  exec                  on                                   default
data/nextCloudApplicationContainer  setuid                on                                   default
data/nextCloudApplicationContainer  readonly              off                                  default
data/nextCloudApplicationContainer  zoned                 off                                  default
data/nextCloudApplicationContainer  snapdir               hidden                               default
data/nextCloudApplicationContainer  aclinherit            restricted                           default
data/nextCloudApplicationContainer  createtxg             61784                                -
data/nextCloudApplicationContainer  canmount              on                                   default
data/nextCloudApplicationContainer  xattr                 sa                                   inherited from data
data/nextCloudApplicationContainer  copies                1                                    default
data/nextCloudApplicationContainer  version               5                                    -
data/nextCloudApplicationContainer  utf8only              off                                  -
data/nextCloudApplicationContainer  normalization         none                                 -
data/nextCloudApplicationContainer  casesensitivity       sensitive                            -
data/nextCloudApplicationContainer  vscan                 off                                  default
data/nextCloudApplicationContainer  nbmand                off                                  default
data/nextCloudApplicationContainer  sharesmb              off                                  default
data/nextCloudApplicationContainer  refquota              none                                 default
data/nextCloudApplicationContainer  refreservation        none                                 default
data/nextCloudApplicationContainer  guid                  3846615381577726221                  -
data/nextCloudApplicationContainer  primarycache          all                                  default
data/nextCloudApplicationContainer  secondarycache        all                                  default
data/nextCloudApplicationContainer  usedbysnapshots       0B                                   -
data/nextCloudApplicationContainer  usedbydataset         200K                                 -
data/nextCloudApplicationContainer  usedbychildren        895M                                 -
data/nextCloudApplicationContainer  usedbyrefreservation  0B                                   -
data/nextCloudApplicationContainer  logbias               latency                              default
data/nextCloudApplicationContainer  objsetid              355                                  -
data/nextCloudApplicationContainer  dedup                 on                                   local
data/nextCloudApplicationContainer  mlslabel              none                                 default
data/nextCloudApplicationContainer  sync                  standard                             default
data/nextCloudApplicationContainer  dnodesize             legacy                               default
data/nextCloudApplicationContainer  refcompressratio      1.00x                                -
data/nextCloudApplicationContainer  written               200K                                 -
data/nextCloudApplicationContainer  logicalused           1.02G                                -
data/nextCloudApplicationContainer  logicalreferenced     69.5K                                -
data/nextCloudApplicationContainer  volmode               default                              default
data/nextCloudApplicationContainer  filesystem_limit      none                                 default
data/nextCloudApplicationContainer  snapshot_limit        none                                 default
data/nextCloudApplicationContainer  filesystem_count      none                                 default
data/nextCloudApplicationContainer  snapshot_count        none                                 default
data/nextCloudApplicationContainer  snapdev               hidden                               default
data/nextCloudApplicationContainer  acltype               posixacl                             inherited from data
data/nextCloudApplicationContainer  context               none                                 default
data/nextCloudApplicationContainer  fscontext             none                                 default
data/nextCloudApplicationContainer  defcontext            none                                 default
data/nextCloudApplicationContainer  rootcontext           none                                 default
data/nextCloudApplicationContainer  relatime              off                                  default
data/nextCloudApplicationContainer  redundant_metadata    all                                  default
data/nextCloudApplicationContainer  overlay               off                                  default
data/nextCloudApplicationContainer  encryption            aes-256-gcm                          -
data/nextCloudApplicationContainer  keylocation           none                                 default
data/nextCloudApplicationContainer  keyformat             passphrase                           -
data/nextCloudApplicationContainer  pbkdf2iters           342K                                 -
data/nextCloudApplicationContainer  encryptionroot        data                                 -
data/nextCloudApplicationContainer  keystatus             available                            -
data/nextCloudApplicationContainer  special_small_blocks  0                                    default

Also on configuration of turnkey linux wizard, during installation of the updates, the disks go crazy and it also takes very long :/
 
Last edited:
It's most likely due to deduplication.

But it could also be a bad disk in your 3 way mirror, given that they have to sync.

You only enabled dedup on the vdev so you can simply recreate them without dedup using send/receive and rename.


In general don't enable dedup, in most cases the overhead is much higher then the benefit.
 
But why does the dedup cause so much disk activity? I thought that's what the dedup table is for. I thought it would store hashes for all blocks so it doesn't have to read them from disk?
 
But why does the dedup cause so much disk activity? I thought that's what the dedup table is for. I thought it would store hashes for all blocks so it doesn't have to read them from disk?

The dedup table takes 20GB RAM for 1 TB pool size, this includes unused storage.

If the table does not fit into ram it's constantly read/written from disk.
 
Hmmm... why doesn't ZFS just store the hash of the most used blocks in the RAM if it has still RAM left (even if it's not enough for the whole dedup table)?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!