[SOLVED] Moved Proxmox to HDD, VMs still on NVME but now got IO Delay

[xepa]

Hello,
I've been using Proxmox on a system with a 256GB NVME, I had everything there including some backups and the system worked perfectly for my needs.
I decided to also use an 500GB HDD that I had lying around, and installed on the following approach:

HDD
Proxmox OS
LVM for Backups

NVME
SWAP partition (8GB not in use atm)
LVM-Thin with VMs and LXCs data


After I did this, I noticed the whole system crashes if I copy large amounts of data between VMs or even on same VM, I also see huge IO Delay that I didn't saw before.
Any ideas on what is causing this?

Here's the lsblk:

NAME                                 MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                                    8:0    0 465.8G  0 disk
├─sda1                                 8:1    0  1007K  0 part
├─sda2                                 8:2    0     1G  0 part /boot/efi
└─sda3                                 8:3    0 464.8G  0 part
  ├─pve-root                         252:2    0    64G  0 lvm  /
  └─pve-local--hdd                   252:3    0 400.8G  0 lvm  /mnt/local-hdd
nvme0n1                              259:0    0 238.5G  0 disk
├─nvme0n1p1                          259:1    0 230.5G  0 part
│ ├─local--nvme-local--nvme_tmeta    252:0    0   2.3G  0 lvm 
│ │ └─local--nvme-local--nvme-tpool  252:4    0 225.7G  0 lvm 
│ │   ├─local--nvme-local--nvme      252:5    0 225.7G  1 lvm 
│ │   ├─local--nvme-vm--101--disk--0 252:6    0     2G  0 lvm 
│ │   ├─local--nvme-vm--102--disk--0 252:7    0     2G  0 lvm 
│ │   ├─local--nvme-vm--105--disk--0 252:8    0     5G  0 lvm 
│ │   ├─local--nvme-vm--103--disk--0 252:9    0    32G  0 lvm 
│ │   ├─local--nvme-vm--104--disk--2 252:10   0    64G  0 lvm 
│ │   ├─local--nvme-vm--104--disk--1 252:11   0     4M  0 lvm 
│ │   ├─local--nvme-vm--104--disk--3 252:12   0     4M  0 lvm 
│ │   ├─local--nvme-vm--100--disk--0 252:13   0     4M  0 lvm 
│ │   ├─local--nvme-vm--100--disk--1 252:14   0    32G  0 lvm 
│ │   └─local--nvme-vm--106--disk--0 252:15   0     5G  0 lvm 
│ └─local--nvme-local--nvme_tdata    252:1    0 225.7G  0 lvm 
│   └─local--nvme-local--nvme-tpool  252:4    0 225.7G  0 lvm 
│     ├─local--nvme-local--nvme      252:5    0 225.7G  1 lvm 
│     ├─local--nvme-vm--101--disk--0 252:6    0     2G  0 lvm 
│     ├─local--nvme-vm--102--disk--0 252:7    0     2G  0 lvm 
│     ├─local--nvme-vm--105--disk--0 252:8    0     5G  0 lvm 
│     ├─local--nvme-vm--103--disk--0 252:9    0    32G  0 lvm 
│     ├─local--nvme-vm--104--disk--2 252:10   0    64G  0 lvm 
│     ├─local--nvme-vm--104--disk--1 252:11   0     4M  0 lvm 
│     ├─local--nvme-vm--104--disk--3 252:12   0     4M  0 lvm 
│     ├─local--nvme-vm--100--disk--0 252:13   0     4M  0 lvm 
│     ├─local--nvme-vm--100--disk--1 252:14   0    32G  0 lvm 
│     └─local--nvme-vm--106--disk--0 252:15   0     5G  0 lvm 
└─nvme0n1p2                          259:2    0     8G  0 part

Thanks in advance

 

[_gabriel]

Any ideas on what is causing this?

it's not because PVE running on HDD, except if it has problems like bad sectors.
it's something else, like not exactly same conf as your previous.
Set VM Disk "Cache" to "Default (none) instead "Write cache" if used. This cause "double write cache" that slow SSD can't substain.
Remember regular consumer SSD can't substain large write, they have a write cache then slow dow.

 

[xepa]

Hi Gabriel,
I disabled cache, did some quick tests and seems to be solved. I will see during the next days if it happens again.
Thank you!

 

[xepa]

Hi,
Just to confirm, this was the issue, everything works great and even write speeds are better.
Guess that for some reason I haven't noticed this before moving Proxmox installation to HDD, but clearly that was not the issue, using cache on consumer grade SSD was.

Thanks once again Gabriel.