Hello everyone,
we are running a server under Proxmox 7.2 (7.2 since last night, 6.4 before, installed with 5.x some years ago). The CPU is 16 x Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz (1 Socket). The system is:
We use 2 6TB SATA hard disks connected to an Adaptec RAID Controller with BBU and WriteBack cache setting configured as RAID 1.
We have been observing poor performance for a long time when the Windows virtual machines install updates. The problem does not occur with other servers running Proxmox 7.2 (other harddisks). Installing a BBU for the raid controller and enabling the WriteBack cache did not help either. Disabling the linux swap didn't help (swapoff -a). We don't use LVM or ZFS. We suspect the hard disks as the cause (possibly https://forum.proxmox.com/threads/extremely-high-i-o-delay.102211/ or problems with 512e https://www.datacore.com/de/blog/ge...r-was-ist-512n-vs-512e-vs-4kn-vs-flash-pages/).
Does anybody have an idea what we can try to get this sorted out? Is it possible to change the hard drives one by one to another model and keep the data? Thank you very much for your help.
Best regards,
André
Btw. I have a valid subscription and added that in my profile but it is not showing.
we are running a server under Proxmox 7.2 (7.2 since last night, 6.4 before, installed with 5.x some years ago). The CPU is 16 x Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz (1 Socket). The system is:
Code:
proxmox-ve: 7.2-1 (running kernel: 5.15.60-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-helper: 7.2-12
pve-kernel-5.15: 7.2-11
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.60-1-pve: 5.15.60-1
pve-kernel-5.15.53-1-pve: 5.15.53-1
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: not correctly installed
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.6-1
proxmox-backup-file-restore: 2.2.6-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-3
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-3
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.5-pve1We use 2 6TB SATA hard disks connected to an Adaptec RAID Controller with BBU and WriteBack cache setting configured as RAID 1.
Code:
Controllers found: 1
----------------------------------------------------------------------
Controller information
----------------------------------------------------------------------
Controller Status : Optimal
Controller Mode : RAID (Expose RAW)
Controller Cache Preservation : Enabled
Channel description : SAS/SATA
Controller Model : Adaptec ASR8405
Controller Serial Number : 8A156311A8C
Controller World Wide Name : 50000D17045C7400
Controller Alarm : Disabled
Physical Slot : 20
Temperature : 73 C/ 163 F (Normal)
Installed memory : 1024 MB
Global task priority : High
Performance Mode : Default/Dynamic
Host bus type : PCIe
Host bus speed : 8000 MHz
Host bus link width : 8 bit(s)/link(s)
PCI Device ID : 653
Stayawake period : Disabled
Spinup limit internal drives : 0
Spinup limit external drives : 0
Defunct disk drive count : 0
Logical devices/Failed/Degraded : 1/0/0
NCQ status : Enabled
Statistics data collection mode : Disabled
Global Physical Device Write Cache Policy: Enabled
Monitor Log Severity Level : Informational
Global Max SAS Phy Link Rate : 12 Gbps
Verify Write Setting : Not Applicable
--------------------------------------------------------
RAID Properties
--------------------------------------------------------
Copyback : Disabled
Automatic Failover : Enabled
Background consistency check : Disabled
Background consistency check period : 0
Error Tunable Profile : Normal
--------------------------------------------------------
Controller BIOS Setting Information
--------------------------------------------------------
Runtime BIOS : Enabled
Array BBS Support : Disabled
Physical Drives Displayed during POST : Disabled
Backplane Mode : IBPI
BIOS Halt on Missing Drive Count : 255
--------------------------------------------------------
Controller Version Information
--------------------------------------------------------
BIOS : 7.18-0 (33556)
Firmware : 7.18-0 (33556)
Driver : 1.2-1 (50983)
Boot Flash : 7.18-0 (33556)
CPLD (Load version/ Flash version) : 10/ 11
SEEPROM (Load version/ Flash version) : 1/ 1
FCT Custom Init String Version : 0x0
--------------------------------------------------------
Controller Cache Backup Unit Information
--------------------------------------------------------
Overall Backup Unit Status : Ready
Backup Unit Type : AFM-700/700LP
Non-Volatile Storage Status : Ready
Supercap Status : Ready
-----------------------------------
Supercap Information
-----------------------------------
Current Temperature : 29 deg C
Threshold Temperature : 55 deg C
Life-time Temperature Recorded
(Min/Max) : 22 deg C/ 33 deg C
Voltage(Present/Max) : 4955 mV/ 5302 mV
Life-time Max Voltage Recorded : 5420 mV
Current Drawn(Present/Max) : -9 mA/ 560 mA
Health : 100 percent
Charge Level : 100 percent
Estimated Life-time : 5 Years, 0 Month
Serial Number : 0A28GD008D4
Learn Status : Idle
Next Scheduled Learn Cycle : 318 hours
Present Capacitance : 333 farads
----------------------------------------------------------------------
Logical device information
----------------------------------------------------------------------
Logical Device number 0
Logical Device name : Array 1
Block Size of member drives : 512 Bytes
RAID level : 1
Unique Identifier : BB581E73
Status of Logical Device : Optimal
Additional details : Quick initialized
Size : 5719030 MB
Parity space : 5719040 MB
Interface Type : Serial ATA
Device Type : HDD
Read-cache setting : Enabled
Read-cache status : On
Write-cache setting : On when protected by battery/ZMM
Write-cache status : On
Partitioned : Yes
Protected by Hot-Spare : No
Bootable : Yes
Failed stripes : No
Power settings : Disabled
--------------------------------------------------------
Logical Device segment information
--------------------------------------------------------
Segment 0 : Present (5723166MB, SATA, HDD, Connector:0, Device:0) Y7JGK04JFTTB
Segment 1 : Present (5723166MB, SATA, HDD, Connector:0, Device:1) Y7JGK04CFTTB
----------------------------------------------------------------------
Physical Device information
----------------------------------------------------------------------
Device #0
Device is a Hard drive
State : Online
Block Size : 512 Bytes
Supported : Yes
Programmed Max Speed : SATA 6.0 Gb/s
Transfer Speed : SATA 6.0 Gb/s
Reported Channel,Device(T:L) : 0,0(0:0)
Reported Location : Connector 0, Device 0
Vendor : ATA
Model : TOSHIBA MG04ACA6
Firmware : FT3B
Serial number : Y7JGK04JFTTB
World-wide name : 500003982BE81E9E
Reserved Size : 4225560 KB
Used Size : 5719040 MB
Unused Size : 64 KB
Total Size : 5723166 MB
Write Cache : Enabled (write-back)
FRU : None
S.M.A.R.T. : No
S.M.A.R.T. warnings : 0
Power State : Full rpm
Supported Power States : Full rpm,Powered off,Reduced rpm
SSD : No
Temperature : 36 C/ 96 F
NCQ status : Enabled
----------------------------------------------------------------
Device Phy Information
----------------------------------------------------------------
Phy #0
PHY Identifier : 0
SAS Address : 30000D17045C7403
Attached PHY Identifier : 3
Attached SAS Address : 50000D17045C7400
----------------------------------------------------------------
Runtime Error Counters
----------------------------------------------------------------
Hardware Error Count : 0
Medium Error Count : 0
Parity Error Count : 0
Link Failure Count : 0
Aborted Command Count : 0
SMART Warning Count : 0
Device #1
Device is a Hard drive
State : Online
Block Size : 512 Bytes
Supported : Yes
Programmed Max Speed : SATA 6.0 Gb/s
Transfer Speed : SATA 6.0 Gb/s
Reported Channel,Device(T:L) : 0,1(1:0)
Reported Location : Connector 0, Device 1
Vendor : ATA
Model : TOSHIBA MG04ACA6
Firmware : FT3B
Serial number : Y7JGK04CFTTB
World-wide name : 500003982BE81E97
Reserved Size : 4225560 KB
Used Size : 5719040 MB
Unused Size : 64 KB
Total Size : 5723166 MB
Write Cache : Enabled (write-back)
FRU : None
S.M.A.R.T. : No
S.M.A.R.T. warnings : 0
Power State : Full rpm
Supported Power States : Full rpm,Powered off,Reduced rpm
SSD : No
Temperature : 35 C/ 95 F
NCQ status : Enabled
----------------------------------------------------------------
Device Phy Information
----------------------------------------------------------------
Phy #0
PHY Identifier : 0
SAS Address : 30000D17045C7402
Attached PHY Identifier : 2
Attached SAS Address : 50000D17045C7400
----------------------------------------------------------------
Runtime Error Counters
----------------------------------------------------------------
Hardware Error Count : 0
Medium Error Count : 0
Parity Error Count : 0
Link Failure Count : 0
Aborted Command Count : 0
SMART Warning Count : 0
----------------------------------------------------------------------
Connector information
----------------------------------------------------------------------
Connector #0
Connector Name : CN0
-------------------------------------------------------------------
Lane Information
-------------------------------------------------------------------
Lane #0
Channel ID : 0
Device ID : 0
SAS Address : 50000D17045C7400
PHY Identifier : 3
----------------------------------------------------------------
Lane SAS Phy Information
----------------------------------------------------------------
SAS Address : 50000D17045C7400
Attached PHY Identifier : 0
Attached SAS Address : 30000D17045C7403
Negotiated Logical Link Rate : PHY enabled - 6 Gbps
Lane #1
Channel ID : 0
Device ID : 1
SAS Address : 50000D17045C7400
PHY Identifier : 2
----------------------------------------------------------------
Lane SAS Phy Information
----------------------------------------------------------------
SAS Address : 50000D17045C7400
Attached PHY Identifier : 0
Attached SAS Address : 30000D17045C7402
Negotiated Logical Link Rate : PHY enabled - 6 Gbps
Lane #2
Channel ID : 0
Device ID : 2
SAS Address : 50000D17045C7400
PHY Identifier : 0
----------------------------------------------------------------
Lane SAS Phy Information
----------------------------------------------------------------
SAS Address : 50000D17045C7400
Lane #3
Channel ID : 0
Device ID : 3
SAS Address : 50000D17045C7400
PHY Identifier : 1
----------------------------------------------------------------
Lane SAS Phy Information
----------------------------------------------------------------
SAS Address : 50000D17045C7400We have been observing poor performance for a long time when the Windows virtual machines install updates. The problem does not occur with other servers running Proxmox 7.2 (other harddisks). Installing a BBU for the raid controller and enabling the WriteBack cache did not help either. Disabling the linux swap didn't help (swapoff -a). We don't use LVM or ZFS. We suspect the hard disks as the cause (possibly https://forum.proxmox.com/threads/extremely-high-i-o-delay.102211/ or problems with 512e https://www.datacore.com/de/blog/ge...r-was-ist-512n-vs-512e-vs-4kn-vs-flash-pages/).
Does anybody have an idea what we can try to get this sorted out? Is it possible to change the hard drives one by one to another model and keep the data? Thank you very much for your help.
Best regards,
André
Btw. I have a valid subscription and added that in my profile but it is not showing.
Attachments
Last edited: