The guest operating system kills the performance of the disk subsystem.

[Shadow2091]

Good afternoon. I ask for help.
Installation:
Proxmox 6, Xeon 2665, 128Gb REG ECC RAM, ZFS, 16Gb ARC, 4 x 600Gb Enterprise 10kSAS HDD in RAIDZ-1.

root@virt:/mnt/zfs/RZHDD/vmstore# zpool status -v
  pool: RZHDD
 state: ONLINE
  scan: none requested
config:

        NAME                        STATE     READ WRITE CKSUM
        RZHDD                       ONLINE       0     0     0
          raidz1-0                  ONLINE       0     0     0
            scsi-350000393c8081118  ONLINE       0     0     0
            scsi-350000393c80812b0  ONLINE       0     0     0
            scsi-350000393c80830a4  ONLINE       0     0     0
            scsi-350000393c80999ec  ONLINE       0     0     0

errors: No known data errors

RZHDD                        type                  filesystem              -
RZHDD                        creation              Wed Sep 25 12:12 2019   -
RZHDD                        used                  50.2G                   -
RZHDD                        available             1.52T                   -
RZHDD                        referenced            34.4K                   -
RZHDD                        compressratio         1.00x                   -
RZHDD                        mounted               yes                     -
RZHDD                        quota                 none                    default
RZHDD                        reservation           none                    default
RZHDD                        recordsize            512K                    local
RZHDD                        mountpoint            /mnt/zfs/RZHDD          local
RZHDD                        sharenfs              off                     default
RZHDD                        checksum              on                      default
RZHDD                        compression           lz4                     local
RZHDD                        atime                 off                     local
RZHDD                        devices               on                      default
RZHDD                        exec                  on                      default
RZHDD                        setuid                on                      default
RZHDD                        readonly              off                     default
RZHDD                        zoned                 off                     default
RZHDD                        snapdir               hidden                  default
RZHDD                        aclinherit            restricted              default
RZHDD                        createtxg             1                       -
RZHDD                        canmount              on                      default
RZHDD                        xattr                 on                      default
RZHDD                        copies                1                       default
RZHDD                        version               5                       -
RZHDD                        utf8only              off                     -
RZHDD                        normalization         none                    -
RZHDD                        casesensitivity       sensitive               -
RZHDD                        vscan                 off                     default
RZHDD                        nbmand                off                     default
RZHDD                        sharesmb              off                     default
RZHDD                        refquota              none                    default
RZHDD                        refreservation        none                    default
RZHDD                        guid                  505271958176609401      -
RZHDD                        primarycache          all                     default
RZHDD                        secondarycache        all                     default
RZHDD                        usedbysnapshots       0B                      -
RZHDD                        usedbydataset         34.4K                   -
RZHDD                        usedbychildren        50.2G                   -
RZHDD                        usedbyrefreservation  0B                      -
RZHDD                        logbias               latency                 default
RZHDD                        objsetid              54                      -
RZHDD                        dedup                 off                     default
RZHDD                        mlslabel              none                    default
RZHDD                        sync                  standard                default
RZHDD                        dnodesize             legacy                  default
RZHDD                        refcompressratio      1.00x                   -
RZHDD                        written               34.4K                   -
RZHDD                        logicalused           50.0G                   -
RZHDD                        logicalreferenced     12.5K                   -
RZHDD                        volmode               default                 default
RZHDD                        filesystem_limit      none                    default
RZHDD                        snapshot_limit        none                    default
RZHDD                        filesystem_count      none                    default
RZHDD                        snapshot_count        none                    default
RZHDD                        snapdev               hidden                  default
RZHDD                        acltype               off                     default
RZHDD                        context               none                    default
RZHDD                        fscontext             none                    default
RZHDD                        defcontext            none                    default
RZHDD                        rootcontext           none                    default
RZHDD                        relatime              off                     default
RZHDD                        redundant_metadata    all                     default
RZHDD                        overlay               off                     default
RZHDD                        encryption            off                     default
RZHDD                        keylocation           none                    default
RZHDD                        keyformat             none                    default
RZHDD                        pbkdf2iters           0                       default
RZHDD                        special_small_blocks  0                       default

Testing streaming writing on a virtualization host gives excellent results:
mount -t tmpfs none /mnt/tmp/ -o size=64G dd if=/dev/urandom of=/mnt/tmp/random.dd bs=1M status=progress count=50000 dd if=/mnt/tmp/random.dd of=/mnt/zfs/RZHDD/vmstore/urandom.dd bs=1M status=progress oflag=direct [B]42722131968 bytes (43 GB, 40 GiB) copied, 99 s, 432 MB/s [/B]
Only one virtual machine is running. Centos7, kernel 5.2. VirtIO SCSI controller, iothread enabled, no cache, 24Gb guest RAM.
When I execute similar commands inside the guest OS, I get a speed of 20-40Mb/s. At this time, the speed of the disk system on the hypervisor drops to 10MB\s and remains so low for a while after the virtual machine finishes recording. If IOThread option is disabled in the VM settings, then during such testing the VM will stop serving I / O, and it will be impossible to turn off the hypervisor OS except by turning off the power.
System IOstat shows 100% disk utilization and recording at the level of several hundred kilobytes per second, while zpool iostat show:

                                                    capacity     operations     bandwidth
pool                                              alloc   free   read  write   read  write
------------------------------------------------  -----  -----  -----  -----  -----  -----
RZHDD                                             68.0G  2.11T      0  2.14K      0  34.4M
  raidz1                                          68.0G  2.11T      0  2.14K      0  34.4M
    scsi-350000393c8081118                            -      -      0    555      0  8.53M
    scsi-350000393c80812b0                            -      -      0    537      0  8.60M
    scsi-350000393c80830a4                            -      -      0    540      0  8.50M
    scsi-350000393c80999ec                            -      -      0    552      0  8.78M

But despite the low write speed, it is impossible to use drives at this time at all.
I'm new to both ZFS and Proxmox. What could be the cause of the problem?

 

[Klaus Steinberger]

Hi,

you mention 16 G ARC size, did you limit it in a conf file in /etc/modprobe.d (and rebuild the initram) so it get set again after reboot?

And for the disks:

you setup a 4 way mirror, was this intentional?

 

[Shadow2091]

you mention 16 G ARC size, did you limit it in a conf file in /etc/modprobe.d (and rebuild the initram) so it get set again after reboot?

Yep.

you setup a 4 way mirror, was this intentional?

As far as I understand, this is not a mirror, but an analog of RAID5 with parity.

 

[Shadow2091]

Well, changing the record block size to 256k in the Proxmox settings gave an increase in VM speed to 140-200Mb/s. The reason was this. But still, it is half the size of the drives on the host. I continue to dig.

 

[LnxBil]

Well, changing the record block size to 256k in the Proxmox settings gave an increase in VM speed to 140-200Mb/s. The reason was this. But still, it is half the size of the drives on the host. I continue to dig.

If you use a lot of snapshots, this is very bad concerning the copy-on-write behaviour and will slow down your system significantly. For every bit you change on disk, you have to read at most 256k, change a bit and write 256k. For random I/O, this is very, very bad. Compressibility and Read times for big files will increase however.