Proxmox: Influence of One VM on Others

[Nameless]

Good time of day!

There is a Proxmox setup with an NVMe disk without RAID, hosting a certain number of VMs. At one point, the disk hosting SQL databases in one of the VMs went offline. There wasn't an excessive load on the disk subsystem, the disk is fine, but a warning appears in the Windows event log: "A request to the device \Device\RaidPort0 was issued." Through trial and error, it was determined that this error can be reproduced in the Windows event log, for example, using Iometer, but it doesn't always result in the disk going offline. In Grafana, the read speed at this moment is 650 MB/s and the write speed is 260 MB/s. This might not be much for a 4-lane NVMe. The strangest thing is that when conducting the test on one VM, a similar issue arises in several other VMs, but not on all of them that are hosted on this disk. The file system is ZFS, and deduplication is turned off. Has anyone encountered something similar? Could VMs in Proxmox somehow affect each other?

 

[jsterr]

Havent seen this one but its always good to update the nvmes to its latest firmware. this can often be done via nvme-cli, but usually you need to contact your hardware-vendor. We had multiple issues with nvmes (u.2/u.3) that were fixed with fw updates.

Have you checked the nvme (smartctl) or health check with nvme-cli?

It would also be useful to post the vm config, to see what drivers and disk-types you use.

 

[Nameless]

=== START OF INFORMATION SECTION ===
Model Number: VO002000KWJSF
Serial Number: PHLF752100EY2P0HGN
Firmware Version: 4ICDHPK1
PCI Vendor ID: 0x8086
PCI Vendor Subsystem ID: 0x1590
IEEE OUI Identifier: 0x5cd2e4
Total NVM Capacity: 2,000,398,934,016 [2.00 TB]
Unallocated NVM Capacity: 0
Controller ID: 0
NVMe Version: 1.2
Number of Namespaces: 1
Namespace 1 Size/Capacity: 2,000,398,934,016 [2.00 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 5cd2e4 31460a0100
Local Time is: Tue Aug 22 17:13:00 2023 MSK
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0006): Format Frmw_DL
Optional NVM Commands (0x0006): Wr_Unc DS_Mngmt
Log Page Attributes (0x02): Cmd_Eff_Lg
Maximum Data Transfer Size: 32 Pages
Warning Comp. Temp. Threshold: 65 Celsius
Critical Comp. Temp. Threshold: 74 Celsius

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 25.00W - - 0 0 0 0 0 0

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2
1 - 4096 0 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 17 Celsius
Available Spare: 99%
Available Spare Threshold: 10%
Percentage Used: 2%
Data Units Read: 75,190,557 [38.4 TB]
Data Units Written: 94,774,843 [48.5 TB]
Host Read Commands: 1,763,348,129
Host Write Commands: 1,187,259,305
Controller Busy Time: 469
Power Cycles: 36,380
Power On Hours: 2,495
Unsafe Shutdowns: 36,373
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

=== START OF INFORMATION SECTION ===
Model Number: VO002000KWJSF
Serial Number: PHLF750501S42P0HGN
Firmware Version: 4ICDHPK1
PCI Vendor ID: 0x8086
PCI Vendor Subsystem ID: 0x1590
IEEE OUI Identifier: 0x5cd2e4
Total NVM Capacity: 2,000,398,934,016 [2.00 TB]
Unallocated NVM Capacity: 0
Controller ID: 0
NVMe Version: 1.2
Number of Namespaces: 1
Namespace 1 Size/Capacity: 2,000,398,934,016 [2.00 TB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 5cd2e4 a0b2060100
Local Time is: Tue Aug 22 17:14:00 2023 MSK
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0006): Format Frmw_DL
Optional NVM Commands (0x0006): Wr_Unc DS_Mngmt
Log Page Attributes (0x02): Cmd_Eff_Lg
Maximum Data Transfer Size: 32 Pages
Warning Comp. Temp. Threshold: 65 Celsius
Critical Comp. Temp. Threshold: 74 Celsius

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 25.00W - - 0 0 0 0 0 0

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2
1 - 4096 0 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 14 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 0%
Data Units Read: 50,621,057 [25.9 TB]
Data Units Written: 98,886,278 [50.6 TB]
Host Read Commands: 1,231,404,181
Host Write Commands: 1,122,689,044
Controller Busy Time: 321
Power Cycles: 29
Power On Hours: 1,947
Unsafe Shutdowns: 24
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged



agent: 1
boot: order=virtio0
cores: 10
hotplug: disk,network,usb,cpu
machine: pc-q35-7.2
memory: 32768
meta: creation-qemu=7.2.0,ctime=1686848947
name: Copy-of-VM-ZelSad
net0: virtio=,bridge=vmbr0,firewall=1,tag=464
numa: 1
onboot: 1
ostype: win10
scsihw: virtio-scsi-single
smbios1: uuid=
sockets: 1
tags:
unused0: local-zfs:vm-101-disk-2
virtio0: local-zfs:vm-101-disk-6,cache=writeback,format=raw,iothread=1,size=100G
virtio1: local-zfs:vm-101-disk-7,cache=writeback,format=raw,iothread=1,size=10G
virtio3: local-zfs:vm-101-disk-8,cache=writeback,format=raw,iothread=1,size=20G
virtio4: local-zfs:vm-101-disk-9,backup=0,cache=writeback,format=raw,iothread=1,size=300G
vmgenid:


The issue arose after migrating from one server to another. It's likely that the problem could be related to the virtual machine settings.