[SOLVED] Possible Bug in PVE 6.1: smartctl and physical sector size

[fx919]

Hi,

I just checked out Proxmox VE 6.1 and was trying to do some storage benchmarks.

When I want to figure out the hard disks physical sector size with smartctl, I get different values on every call:

smartctl -i /dev/sdc | grep -i Phy
Physical block size:  16384 bytes
root@pvetest:~/test/04.01# smartctl -i /dev/sdc | grep -i Phy
Physical block size:  4096 bytes
root@pvetest:~/test/04.01# smartctl -i /dev/sdc | grep -i Phy
Physical block size:  2097152 bytes
root@pvetest:~/test/04.01# smartctl -i /dev/sdc | grep -i Phy
Physical block size:  2048 bytes
root@pvetest:~/test/04.01# smartctl -i /dev/sdc | grep -i Phy
Physical block size:  16384 bytes
root@pvetest:~/test/04.01# smartctl -i /dev/sdc | grep -i Phy
Physical block size:  1048576 bytes

version information:

# dpkg -l | grep " pve-"

ii  pve-cluster                          6.1-2                       amd64        "pmxcfs" distributed cluster filesystem for Proxmox Virtual Environment.
ii  pve-container                        3.0-14                      all          Proxmox VE Container management tool
ii  pve-docs                             6.1-3                       all          Proxmox VE Documentation
ii  pve-edk2-firmware                    2.20191002-1                all          edk2 based firmware modules for virtual machines
ii  pve-firewall                         4.0-9                       amd64        Proxmox VE Firewall
ii  pve-firmware                         3.0-4                       all          Binary firmware code for the pve-kernel
ii  pve-ha-manager                       3.0-8                       amd64        Proxmox VE HA Manager
ii  pve-i18n                             2.0-3                       all          Internationalization support for Proxmox VE
ii  pve-kernel-5.3                       6.0-12                      all          Latest Proxmox VE Kernel Image
ii  pve-kernel-5.3.10-1-pve              5.3.10-1                    amd64        The Proxmox PVE Kernel Image
ii  pve-kernel-helper                    6.0-12                      all          Function for various kernel maintenance tasks.
ii  pve-manager                          6.1-3                       amd64        Proxmox Virtual Environment Management Tools
ii  pve-qemu-kvm                         4.1.1-2                     amd64        Full virtualization on x86 hardware
ii  pve-xtermjs                          3.13.2-1                    all          HTML/JS Shell client

Update

I just updated the system. The error had been already fixed.

Update - 2:

Issue still persists. Possibly it's connected to the old storage controller. The SSD connected to On-Board SATA-Controller reports the data correctly.

kern.log

Mar  6 12:57:02 pvetest kernel: [   15.645398] aacraid: Comm Interface enabled
Mar  6 12:57:02 pvetest kernel: [   15.948641] AAC0: kernel 5.2-0[18252] Nov 22 2010
Mar  6 12:57:02 pvetest kernel: [   15.948643] AAC0: monitor 5.2-0[18252]
Mar  6 12:57:02 pvetest kernel: [   15.948645] AAC0: bios 5.2-0[18252]
Mar  6 12:57:02 pvetest kernel: [   15.948647] AAC0: serial 0D221163C9E
Mar  6 12:57:02 pvetest kernel: [   15.948648] AAC0: Non-DASD support enabled.
Mar  6 12:57:02 pvetest kernel: [   15.948649] AAC0: 64bit support enabled.
Mar  6 12:57:02 pvetest kernel: [   15.948653] aacraid 0000:01:00.0: 64 Bit DAC enabled
Mar  6 12:57:02 pvetest kernel: [   28.315077] scsi host2: aacraid

lspci -v -s 01:00.0

01:00.0 RAID bus controller: Adaptec AAC-RAID (rev 09)
        Subsystem: Adaptec 5405
        Flags: bus master, fast devsel, latency 0, IRQ 16
        Memory at fb800000 (64-bit, non-prefetchable) [size=2M]
        Expansion ROM at fba00000 [disabled] [size=512K]
        Capabilities: [98] Power Management version 2
        Capabilities: [a0] MSI: Enable- Count=1/2 Maskable- 64bit+
        Capabilities: [d0] Express Endpoint, MSI 00
        Capabilities: [90] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: aacraid
        Kernel modules: aacraid

 

[t.lamprecht]

Hi

When I want to figure out the hard disks physical sector size with smartctl, I get different values on every call:

cannot reproduce, possibly drive related?

version information:

Please use pveversion -v to grab more relevant Proxmox VE packages, not fitting this search.

Issue still persists. Possibly it's connected to the old storage controller. The SSD connected to On-Board SATA-Controller reports the data correctly.

Oh, a HW RAID controller, seems suspicious.. What motherbaord do you use? And did you upgraded it (and thus hopefully also the RAID controller) to the latest available Firmware?

 

[fx919]

pveversion -v

proxmox-ve: 6.1-2 (running kernel: 5.3.10-1-pve)
pve-manager: 6.1-3 (running version: 6.1-3/37248ce6)
pve-kernel-5.3: 6.0-12
pve-kernel-helper: 6.0-12
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-2
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-14
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191002-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-2
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

The Controller is an external Controller Card(Adaptec ASR2405, reported as "Adaptec 5405"). The Mainboard is an "Intel DQ67SW". Both compoments are quite old as of now. I did not update the Firmware. I just reported the problem because it might point to a general bug.

 

[t.lamprecht]

I just reported the problem because it might point to a general bug.

Yes it might, but I honestly guess that's it's really not one in smartmontools but rather, as so often, one in the HW RAID controller firmware.
If you see this when attaching drives directly I'd be glad to hear from it.

 

[t.lamprecht]

smartmontools: 7.0-pve2

Are you sure you upgraded with setup repos, as there should be smartmontools: 7.1-pve2 available since quite a bit.

https://pve.proxmox.com/wiki/Package_Repositories

 

[fx919]

Ok. I'll try another Controller if possible. I can not use the onboard controller of the board, because the drives are SAS-Drives.

 

[t.lamprecht]

Thanks, I mean it's not impossible that this is an issue with smartmontools and certain controller/drive models, but it'd definitively surprise me more than the HW Controller being at fault. Plus, to be sure, i just checked over 15 drives from a total of four different servers, which all stayed stable - so it seems that there isn't something totally wrong in general.

 

[fx919]

repositories are pve default. (enterprise repo disabled) - because this is only a testing machine.

 

[t.lamprecht]

repositories are pve default. (enterprise repo disabled) - because this is only a testing machine.

Then I'd suggest to enable the no-subscription or pvetest one to get latest updates which got released after the 6.1 ISO release

 

[fx919]

Ok. Seems it is a typical PEBCAK problem:

I got some information from my smartctl command, but it's not the way smartctl is to be handled with that controller.

The correct command, I found out now, is:

# smartctl command for the first of the controllers SAS disks:

smartctl -i /dev/sg3 -d scsi -i

When I issue this command, the output is stable and the same for all 4 installed disks - even though the physical block size of 16384 is not quite in the range of the expected - maybe it's some setting of the raid controller.