Proxmox IO slow - 17GB free

Thechaser · Saturday at 22:26

Hello all,

All my vm's are very slow, I point it down to the iops.
For some reason I see this, but I don't know what it means.
I have a NVME SSD of 2TB where all my hdd are on of the vm.
And a 480GB ssd for my proxmox install.

Can someone guide me how to read these numbers and what to do?

I use the latest proxmox version of this date.

Thank you

Impact · Saturday at 22:40

The picture looks normal. What is slow exactly? What is the model of the drive? What's the IOWait? How are the VMs configured? Also see here.

Thechaser · Saturday at 22:52

Hello thank you.
The IOWait is 0.00%

I looked at this picture:

This is an example of a vm:

but all the vm have the problem, even freshly installed ones. (linux and windows)

these are my disk:

The Lexar is my nvme where all the hdd are stored.
The 480 is the one where proxmox is installed by LVM

the Samsung one is only to make backups to.

the windows gives this in all vm's:

the problem even presist when only one vm is active.

Impact · Saturday at 23:00

I'd investigate further with the link. Also check the temperature of the drive. Can you share pvs and lvs too?

Thechaser · Saturday at 23:15

Ok, I'm following the link procedure right now

this is the temperature:

Code:

[/dev/nvme0n1]

Model Number:                       Lexar SSD NM620 2TB

Serial Number:                      NKM987R005782P110N

Temperature:                        39 Celsius

Warning  Comp. Temperature Time:    0

Critical Comp. Temperature Time:    0

Temperature Sensor 1:               39 Celsius

Temperature Sensor 2:               44 Celsius



[/dev/sda]

Device Model:     KINGSTON SA400S37480G

Serial Number:    50026B76868C955F

194 Temperature_Celsius     0x0022   032   047   000    Old_age   Always       -       32 (Min/Max 18/47)



[/dev/sdb]

Device Model:     INTEL SSDSC2KB038TZ

Serial Number:    PHYI2170004Q3P8EGN

190 Drive_Temperature       0x0022   064   063   000    Old_age   Always       -       36 (Min/Max 34/38)

194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       36



[/dev/sdc]

Model Number:                       Samsung SSD 990 EVO Plus 2TB

Serial Number:                      S7U7NU0YA96799N

Temperature:                        34 Celsius

Warning  Comp. Temperature Time:    0

Critical Comp. Temperature Time:    0

Temperature Sensor 1:               38 Celsius

Temperature Sensor 2:               34 Celsius

Impact · Saturday at 23:16

Please use code blocks. The temperature is in a good range so it shouldn't cause throttling.

Thechaser · Saturday at 23:28

My findings:

I see the red flashing often.

Code:

root@lenovo-one:~# fatrace -tf W

fatrace: Failed to add watch for /etc/pve: No such device

23:24:04.296712 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:06.296676 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:06.825229 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:06.825460 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:06.825597 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:06.825638 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:07.034305 systemd-journal(384): W   /var/log/journal/a062ea9990e9454dbdcc5ac30786e3e1/system.journal

23:24:07.300357 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:07.306256 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:08.296790 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:10.130929 pvestatd(1173): CW  /var/log/pve/tasks/.active.lock

23:24:10.312103 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:11.296820 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:11.303011 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:11.826938 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:11.827169 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:11.833333 pmxcfs(1005): W   /var/lib/pve-cluster/config.db

23:24:11.833402 pmxcfs(1005): W   /var/lib/pve-cluster/config.db

23:24:11.834286 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:11.834610 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:11.834638 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:12.296642 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:14.296803 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:15.297316 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:15.302498 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:16.297092 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:16.836112 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:16.836228 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:16.838573 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:16.838695 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:17.284408 systemd-journal(384): W   /var/log/journal/a062ea9990e9454dbdcc5ac30786e3e1/system.journal

23:24:18.300245 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:19.296863 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:19.302944 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:20.295181 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:20.339482 pvestatd(1173): CW  /var/log/pve/tasks/.active.lock

23:24:21.840052 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:21.840052 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:21.840157 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:21.840234 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:21.842690 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:21.843013 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:22.297175 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:23.297442 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:23.303576 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:23.309487 rrdcached(834): CW  /var/lib/rrdcached/db/pve-node-9.0/lenovo-one

23:24:23.312962 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:24.306772 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:26.294829 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:26.847344 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:26.847344 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:26.847431 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:26.847645 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:26.847849 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:27.296782 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:27.303801 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:27.534239 systemd-journal(384): W   /var/log/journal/a062ea9990e9454dbdcc5ac30786e3e1/system.journal

23:24:28.297809 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:30.296889 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:30.550860 pvestatd(1173): CW  /var/log/pve/tasks/.active.lock

23:24:31.301874 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:31.307942 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:31.848924 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:31.849139 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:31.851621 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:32.295413 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:34.297087 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:35.297392 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:35.302792 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:36.297081 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:36.852721 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:36.852851 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:36.852851 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:36.853058 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:36.853084 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:37.784432 systemd-journal(384): W   /var/log/journal/a062ea9990e9454dbdcc5ac30786e3e1/system.journal

23:24:38.303489 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:39.295026 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:39.302399 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:40.295522 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:40.755817 pvestatd(1173): CW  /var/log/pve/tasks/.active.lock

23:24:41.854471 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:41.854754 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:41.854754 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:41.854815 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:41.855064 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:42.297197 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:43.297763 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:43.304587 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:44.298418 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:46.295893 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:46.856348 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:46.856348 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:46.856348 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:46.856639 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:46.856639 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:46.856683 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:46.856982 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:46.857006 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:47.297458 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:47.303131 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:48.034411 systemd-journal(384): W   /var/log/journal/a062ea9990e9454dbdcc5ac30786e3e1/system.journal

23:24:48.296368 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:50.299578 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:50.961221 pvestatd(1173): CW  /var/log/pve/tasks/.active.lock

23:24:51.298589 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:51.304641 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:51.858260 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:51.858260 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:51.858382 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:51.858532 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:51.858532 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:51.858577 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:51.858764 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:51.858788 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:52.308921 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:54.298720 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:54.305594 rrdcached(834): CW  /var/lib/rrdcached/db/pve-node-9.0/lenovo-one

23:24:54.308937 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:55.321777 pveproxy worker(1216): W   /var/log/pveproxy/access.log

23:24:55.322498 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:56.299712 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:56.860194 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:56.860380 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:56.860380 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:56.860536 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:56.860639 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:56.860694 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:24:58.284428 systemd-journal(384): W   /var/log/journal/a062ea9990e9454dbdcc5ac30786e3e1/system.journal

23:24:58.298568 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:59.301321 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:24:59.302313 pveproxy worker(1216): W   /var/log/pveproxy/access.log

23:25:00.155326 pvestatd(1173): CW  /var/log/pve/tasks/.active.lock

23:25:00.297641 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:25:00.638651 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:25:00.642057 pvescheduler(8244): W   /var/lib/pve-manager/pve-replication-state.json.tmp.8244

23:25:00.642115 pvescheduler(8244): CW  /var/lib/pve-manager/pve-replication-state.json

23:25:00.642156 pvescheduler(8244): CW  /var/lib/pve-manager/pve-replication-state.lck

23:25:00.642509 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:25:00.642873 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:25:00.644163 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:25:00.644210 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:25:01.284438 systemd-journal(384): W   /var/log/journal/a062ea9990e9454dbdcc5ac30786e3e1/system.journal

23:25:01.784375 systemd-journal(384): W   /var/log/journal/a062ea9990e9454dbdcc5ac30786e3e1/system.journal

23:25:01.862137 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:25:01.862137 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:25:01.862209 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:25:01.862390 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:25:01.862620 pmxcfs(1005): W   /var/lib/pve-cluster/config.db-wal

23:25:02.297246 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:25:03.313094 pveproxy worker(1217): W   /var/log/pveproxy/access.log

23:25:03.313578 pveproxy worker(1216): W   /var/log/pveproxy/access.log

Tomorrow I will recieve a spare 2TB nvme disk. I will check if its a hardware error. and if needed I can reinstall the proxmox as well.

Impact · Saturday at 23:35

Yeah the lexar drive seems to be hurting pretty badly. I can't see a good reason why yet though.

Thechaser · Saturday at 23:41

Ok, I will get the spare drive tomorrow and will clone the drive and check if it presists. it has a 18% wear. and its 2years old.

_gabriel · 2026-03-29T10:27:41+0200

switch Lexar to backup then Samsung 990 as the Datastore

Thechaser · 2026-03-29T16:42:13+0200

In the end I deleted all the vm's (after backup) and datastore. plugged in the nvme and restored all the vm's.
Very fast now. It's active for a while now. I will monitor

Search

Search

Proxmox IO slow - 17GB free

Thechaser

Member

Impact

Distinguished Member

Thechaser

Member

Impact

Distinguished Member

Thechaser

Member

Impact

Distinguished Member

Thechaser

Member

Impact

Distinguished Member

Thechaser

Member

_gabriel

Famous Member

Thechaser

Member

We value your privacy