Slow disk freezing the system randomly

[paoloc68]

I have a running environment which I use to run also MacOs.
Sometimes the guest VM freezes and does not respond for some seconds, trying to see what happens I have found the following indication by looking in atop:

So it seems that one device of the pool gets busy, this is the pool:

root@proxmox:~# zpool status
  pool: fastzfs
 state: ONLINE
  scan: scrub repaired 0B in 00:22:49 with 0 errors on Sun Sep 11 00:46:50 2022
config:

    NAME                                  STATE     READ WRITE CKSUM
    fastzfs                               ONLINE       0     0     0
      raidz1-0                            ONLINE       0     0     0
        nvme-CT1000P1SSD8_1934E21B33B1    ONLINE       0     0     0
        ata-CT1000BX500SSD1_2150E5F04304  ONLINE       0     0     0
        ata-CT1000BX500SSD1_2150E5F06346  ONLINE       0     0     0

as you can see sdc is member of the pool, I know this is a spurious pool in which I mix an nvme disk with two sata:

root@proxmox:~# lsblk -o +FSTYPE
NAME                             MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT FSTYPE
sda                                8:0    0   1.8T  0 disk            LVM2_member
├─slow_storage-vm--1500--disk--0 253:2    0   500G  0 lvm
├─slow_storage-vm--1001--disk--0 253:3    0    20G  0 lvm             ext4
└─slow_storage-vm--1201--disk--0 253:4    0     4M  0 lvm
sdb                                8:16   0 931.5G  0 disk
├─sdb1                             8:17   0 931.5G  0 part            zfs_member
└─sdb9                             8:25   0     8M  0 part
sdc                                8:32   0 931.5G  0 disk
├─sdc1                             8:33   0 931.5G  0 part            zfs_member
└─sdc9                             8:41   0     8M  0 part
sdd                                8:48   1    15G  0 disk
└─sdd1                             8:49   1    15G  0 part            BitLocker
zd0                              230:0    0   500G  0 disk
├─zd0p1                          230:1    0   200M  0 part            vfat
└─zd0p2                          230:2    0 499.8G  0 part            apfs
zd16                             230:16   0     1M  0 disk
zd32                             230:32   0     1M  0 disk
zd48                             230:48   0    32G  0 disk
├─zd48p1                         230:49   0   512M  0 part            vfat
└─zd48p2                         230:50   0  31.5G  0 part            ext4
zd64                             230:64   0     1M  0 disk
zd80                             230:80   0   110G  0 disk
├─zd80p1                         230:81   0   100M  0 part            vfat
├─zd80p2                         230:82   0    16M  0 part
├─zd80p3                         230:83   0 109.4G  0 part            ntfs
└─zd80p4                         230:84   0   516M  0 part            ntfs
zd96                             230:96   0   250G  0 disk
├─zd96p1                         230:97   0   200M  0 part            vfat
└─zd96p2                         230:98   0 249.8G  0 part            apfs
nvme0n1                          259:0    0 931.5G  0 disk
├─nvme0n1p1                      259:1    0 931.5G  0 part            zfs_member
└─nvme0n1p9                      259:2    0     8M  0 part
nvme1n1                          259:3    0 465.8G  0 disk
├─nvme1n1p1                      259:4    0  1007K  0 part
├─nvme1n1p2                      259:5    0   512M  0 part /boot/efi  vfat
└─nvme1n1p3                      259:6    0 465.3G  0 part            LVM2_member
  ├─pve-swap                     253:0    0     8G  0 lvm  [SWAP]     swap
  └─pve-root                     253:1    0 457.3G  0 lvm  /          ext4

But is the "mix" the cause of the problem or it is else?
I am buying a third sata ssd so to rebalance that situation but I would like to know if that is indeed the cause of the problem

 

[Max Carrara]

I'm 99% sure that your issue stems from that one disk, as you correctly identified. Since your pool is in RAID-Z1, all data is spread across all three drives (obviously). Therefore, ZFS needs to read from and write to all three drives, and because that one drive is often busy, it will slow down your entire pool.

In other words, that one drive is almost certainly your bottleneck.

You could check the drive's SMART data using smartctl -a /dev/sdc to see if it's throwing any errors or if any attributes are suspicious, if you'd like. You could also perform a long SMART test using smartctl -t long /dev/sdc and check the drive's status again using smartctl -a /dev/sdc, as before.

Though, in my opinion, it's best if you just replaced that drive. I doubt the SMART data is going to tell you anything you don't already know (namely, that your drive is probably dying).

 

[Neobin]

As a sidenote:
The CT1000P1SSD8 as well as the CT1000BX500SSD1 are both hot garbage for things like ZFS.
Not only are those cheap consumer SSDs without PLP, but they also use QLC-nand.

Additionally: Also keep the cons of a raidZ in mind, like e.g. padding overhead and IOPS of only a single drive...

 

[paoloc68]

As a sidenote:
The CT1000P1SSD8 as well as the CT1000BX500SSD1 are both hot garbage for things like ZFS.
Not only are those cheap consumer SSDs without PLP, but they also use QLC-nand.

Additionally: Also keep the cons of a raidZ in mind, like e.g. padding overhead and IOPS of only a single drive...

Thanks for your suggestions, for the hardware I cannot invest a lot, I don't have any critical data and all my work gets anyway backed up. Following your valuable suggestion I configured a simple raid1 with the two NVME drives and any hiccups seem to be gone.
Now I would like to keep using the three SSD, what configuration would you recommend for obtaining a decent performance?

 

[Dunuin]

Decent performance and QLC NAND are two things that don't belong in one sentence.
QLC NAND SSDs are only good at reading, terrible at writing. So only thing they are usable for are workloads where you never try to write more than a few GBs at a time. For example as cold storage to store ISOs or something similar.

 

[Neobin]

Now I would like to keep using the three SSD, what configuration would you recommend for obtaining a decent performance?

What Dunuin said.
But to give you an answer: Since it is a single drive and depending on what you want to use it for, probably either LVM-thin or simply ext4, I guess.

Edit: Or do we talk about more than that single drive you got now free from the change to raid1?

 

[paoloc68]

What Dunuin said.
But to give you an answer: Since it is a single drive and depending on what you want to use it for, probably either LVM-thin or simply ext4, I guess.

Edit: Or do we talk about more than that single drive you got now free from the change to raid1?

I have three of them, I can move elsewhere what still remains and use them for a new pool, ideally a stripe mirror so I could increase the overall write speed. When I had chosen to make a raidz1-0 I was thinking that would have been working that way. Otherwise could be worth it to add one and get a RAID10?

 

[Dunuin]

I wouldn't use a copy-on-write filesystem like ZFS with those. Hardware raid or more simple software raids like mdraid would be better suited.
And yes, a raid10 would double the IOPS performance but you still got the problem that the write performance will drop to terrible levels as soon as the SLC and RAM caches get full.

 

[Neobin]

I assume, hardware raid is out of question here.

If you get a fourth disk, either a raid10 or two raid1.

Options for the raid(s) (repeating Dunuin):

• mdraid: Would probably be okay for those disks, but it is not (officially) supported by Proxmox and therefore you have to set it up and maintain completely on your own; even more, if they should become your boot-/system-disks, I guess.
• ZFS: CoW-filesystem. Officially supported. As already stated, really not good with those disks.
• BTRFS: CoW-filesystem. Officially supported; but currently in "technology preview" status. Before you go with this, you could also go directly with ZFS.

So yeah, choose the lesser evil, I would say.