Disk Serialnumber in PVE -> Host -> Disk

[FelixJ]

Hi,
i recently found out, that the value displayed in PVE-Manager under datacenter -> host -> Disks equals accordingly to the output of smarctcl to the "Logical unit id" rather then the "Serial number", which is confusing.
This happens on all Servers having a HP P420 HBA.
When using the command
smartcli -d cciss,X /dev/sdX --all
the value displayed in the PVE-Manager is associated to the "Logical unit id".
If a disk is connected via the internal SATA-Bus, the serial number in PVE-Manager is correctly (see /dev/sda) displayed.

Is there a setting which I don't know of, where I can tell PVE-Manager, that I use such a HBA to correct this?
Also, as you can see, the SMART Status cannot be displayed as well.

Regards,
Felix

 

[apoc]

SMART Status cannot be displayed as well.

Smart values of disks behind a raid controller always are a challenge, as the physical disk is masked an replaced by a virtual (RAID).
You are using smartcli on your commandlibe. Pve does grab the data with native tools and those don't look beyond the representation of your LUN

 

[FelixJ]

Smart values of disks behind a raid controller always are a challenge, as the physical disk is masked an replaced by a virtual (RAID).
You are using smartcli on your commandlibe. Pve does grab the data with native tools and those don't look beyond the representation of your LUN

That must be the explanation for that behavior. So hers' definitely room for improvement. I would love to see the correct serial numbers and the smart status via GUI!

 

[FelixJ]

Are there any plans to correct this, as the displayed information are clearly wrong...?

 

[aaron]

Are there any plans to correct this, as the displayed information are clearly wrong...?

I don't think so. Because the disks shown are as they are exposed to the system by the RAID controller. It is not necessarily a 1to1 relation, of the disk Proxmox VE (or any other OS) sees to the physical disks. So the one disk the OS sees, could actually be multiple disks used in some kind of RAID.

Additionally, different RAID controllers with different firmware can behave very differently. Older HP RAID controllers (and yours is one of those AFAICT) are rather bad at passing through single disks as raw as possible. Dell PERCs are usually quite a bit better if the single disk is configured as JBOD, here you should see the actual serial number because the controller doesn't mask it.

So in short, if you don't need HW RAID, try to avoid those cards or try to flash them with an IT-mode firmware where it will behave as a simple disk controller without masking anything.

 

[FelixJ]

I don't think so. Because the disks shown are as they are exposed to the system by the RAID controller. It is not necessarily a 1to1 relation, of the disk Proxmox VE (or any other OS) sees to the physical disks. So the one disk the OS sees, could actually be multiple disks used in some kind of RAID.

Additionally, different RAID controllers with different firmware can behave very differently. Older HP RAID controllers (and yours is one of those AFAICT) are rather bad at passing through single disks as raw as possible. Dell PERCs are usually quite a bit better if the single disk is configured as JBOD, here you should see the actual serial number because the controller doesn't mask it.

I really don't want to offend you, but using smartctl --d cciss,X /dev/sdX --all works fine.
So why re-invent the wheel in this case and not rely on known APIs and utilities (such as smartmontools)?

So in short, if you don't need HW RAID, try to avoid those cards or try to flash them with an IT-mode firmware where it will behave as a simple disk controller without masking anything.

The p420 is in HBA-Mode, so all disks are directly exposed as directly as one can think.

 

[aaron]

FYI: Proxmox VE gathers the disk infos from udev by calling

udevadm info -p /sys/block/<nvme0n1/sdX/...> --query all

This p420 is in a G8/G9 HP server right? It has been a while since I had to use them, but IIRC the RAID controller firmware, and especially the way it implemented HBA mode, was rather lackluster. One of my biggest gripes with it was, that you cannot boot from the disks once in HBA mode. Which is why a few years back, when I ran a G8, I bought a cheap HBA controller already flashed to IT mode. And on the G8 I also had to replace the cables to the disk backplane as the plugs to the onboard controller were 90° angled... After that things worked fine since the disks were actually presented directly as raw disks.

The p420 is in HBA-Mode, so all disks are directly exposed as directly as one can think.

Even in HBA mode, the disks are presented as cciss devices and not direct raw disks. Which is why you need to specify it in the smartctl command, as it otherwise would not be able to get the correct information.
See this thread in the HPE forums: https://community.hpe.com/t5/prolia...n-hba-mode/m-p/7138406/highlight/true#M174891


Newer HP servers / RAID controllers should be quite a bit better in that regard AFAIK, but I don't have first-hand experience, so take that with a grain of salt.

So while the situation is annoying, looking into udev would be what I would do if I would really want to get that working. Or go ahead and get some real HBA controller already flashed to IT mode and disable / replace the RAID controller. The latter is what I did some years ago. The H220 is one for example that should work fine in HP server and can be found on eBay for not too much money.

 

[FelixJ]

I tried the udevadm command, here's what I got:
udevadm info -p /sys/block/sdb --query all P: /devices/pci0000:00/0000:00:02.2/0000:02:00.0/host2/scsi_host/host2/port-2:2/end_device-2:2/target2:0:1/2:0:1:0/block/sdb N: sdb L: 0 S: disk/by-id/scsi-35000cca072116b7c S: disk/by-path/pci-0000:02:00.0-sas-0x5000cca072116b7d-lun-0 S: disk/by-id/lvm-pv-uuid-i163GU-0KRI-IHoR-kV2Z-BmQt-JM6a-DXgFJs S: disk/by-id/wwn-0x5000cca072116b7c E: DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:02:00.0/host2/scsi_host/host2/port-2:2/end_device-2:2/target2:0:1/2:0:1:0/block/sdb E: DEVNAME=/dev/sdb E: DEVTYPE=disk E: MAJOR=8 E: MINOR=16 E: SUBSYSTEM=block E: USEC_INITIALIZED=4209722 E: ID_SCSI=1 E: ID_VENDOR=HGST E: ID_VENDOR_ENC=HGST\x20\x20\x20\x20 E: ID_MODEL=HUC101212CSS600 E: ID_MODEL_ENC=HUC101212CSS600\x20 E: ID_REVISION=A469 E: ID_TYPE=disk E: [I][B]ID_SERIAL=35000cca072116b7c[/B][/I] E: ID_SERIAL_SHORT=5000cca072116b7c E: ID_WWN=0x5000cca072116b7c E: ID_WWN_WITH_EXTENSION=0x5000cca072116b7c E: [B][U]ID_SCSI_SERIAL=L0G9KZPG[/U][/B] E: ID_BUS=scsi E: ID_PATH=pci-0000:02:00.0-sas-0x5000cca072116b7d-lun-0 E: ID_PATH_TAG=pci-0000_02_00_0-sas-0x5000cca072116b7d-lun-0 E: ID_FS_UUID=i163GU-0KRI-IHoR-kV2Z-BmQt-JM6a-DXgFJs E: ID_FS_UUID_ENC=i163GU-0KRI-IHoR-kV2Z-BmQt-JM6a-DXgFJs E: ID_FS_VERSION=LVM2 001 E: ID_FS_TYPE=LVM2_member E: ID_FS_USAGE=raid E: SYSTEMD_READY=1 E: SYSTEMD_ALIAS=/dev/block/8:16 E: SYSTEMD_WANTS=lvm2-pvscan@8:16.service E: DEVLINKS=/dev/disk/by-id/scsi-35000cca072116b7c /dev/disk/by-path/pci-0000:02:00.0-sas-0x5000cca072116b7d-lun-0 /dev/disk/by-id/lvm-pv-uuid-i163GU-0KRI-IHoR-kV2Z-BmQt-JM6a-DXgFJs /dev/disk/by-id/wwn-0x5000cca072116b7c E: TAGS=:systemd:

As you can see, udevadm enumerates the disks correctly as sdX, how ever, the correct serialnumber "hides" behind the ID_SCSI_SERIAL - Parameter.
After all, I am not convinced, that this is an unresolveable issue to the gui.
Probably, a simple additional query, whether there is a ID_SCSI_SERIAL - Parameter along side to the ID_SERIAL might solve the problem?
If you can tell me, where I can try my luck in patching it myself, that would be great, then, at least I would have solved the problem to myself, if you do not see any further plan to mitigate that issue?
Of course, in the spirit of opensource I would be honored to share my code afterwards!
Thanks a lot.
Felix

 

[aaron]

Thanks for that info Could you please open a new enhancement request in our bugtracker with that info? Checking for the ID_SCSI_SERIAL field and using it if present really shouldn't be much of a hassle.

If you want to try it yourself, check the code I linked earlier, a bit further below, there is a line with a regex for the ID_SERIAL_SHORT. It probably should be enough to have a second regex right after it, overwriting the existing serial number if ID_SCSI_SERIAL exists.
If you haven't contributed any code yet, please check out the https://pve.proxmox.com/wiki/Developer_Documentation for the workflow and also the needed CLA (bottom of that page).

 

[FelixJ]

Hi Aron,
here's how far I came:
I even almost was able to get smart-data, how ever I don't know, how to handle my $data->{smartdparam} correctly to the get_smart_data function.
I altered the function call, but no luck:

my $smartdata = get_smart_data($devpath, !is_ssdlike($type),$data->{smartdparam});

sub get_smart_data {
    my ($disk, $healthonly, $smartdparam) = @_;

    assert_blockdev($disk);
    my $smartdata = {};
    my $type;

    if ($disk =~ m!^/dev/(nvme\d+n\d+)$!) {
        my $info = get_sysdir_info("/sys/block/$1");
        $disk = "/dev/".($info->{device}
            or die "failed to get nvme controller device for $disk\n");
    }

    my $cmd = [$SMARTCTL, '-H'];
    push @$cmd, '-A', '-f', 'brief' if !$healthonly;
    push @$cmd, $disk;
    #############
    #if (defined ($smartdparam})) {
    #       push @$cmd, $smartdparam;
    #}
    ##############

These are my changes to the get_udev_info function:

 ############
    #
      my $hbaline = "";
      $data->{hbapath} = $1 if $info =~ m/^E: DEVPATH=(\S+)$/m;
      $data->{hbaid} = ( split /\//, $data->{hbapath})[4];
      $data->{devport} = ( split /:/,( split /\//, $data->{hbapath})[8])[1]-2;
      $dev = "/sys/bus/pci/devices/".$data->{hbaid};
    eval {
        run_command(['udevadm', 'info', '-p', $dev, '--query', 'all'], outfunc => sub {
            my ($hbaline) = @_;
            $hbainfo .= "$hbaline\n";
        });
    };
    $data->{smartdriver}="none";
    if ($hbainfo =~ m/^E: ID_MODEL_FROM_DATABASE=(.*Smart.*)$/m) {
            $data->{smartdparam}="cciss,".$data->{devport};
            my $smartdparam = $data->{smartdparam};
    }

    $data->{serial} = 'unknown';
    if ($info =~ m/^E: ID_SCSI_SERIAL=(.+)$/m) {
        $data->{serial} = $1 if $info =~ m/^E: ID_SCSI_SERIAL=(\S+)$/m;
    } else {
        $data->{serial} = $1 if $info =~ m/^E: ID_SERIAL_SHORT=(\S+)$/m;
    }

So with a little help, so I can call the get_smart_data with my newly created SmartDriverParameter smartdparam - Variable, would be great!
Regards,
Felix

 

[aaron]

Hmm, if you want to show changes, it is easiest if you run git diff, optionally with the path to the file in question.

I don't really understand your approach, though. Might be a bit clearer with a git diff output

IIUC, the serial number is actually present in the output of udevadm, just under a different name?
Then the change should probably be only

diff --git a/PVE/Diskmanage.pm b/PVE/Diskmanage.pm
index a311ffd..1048732 100644
--- a/PVE/Diskmanage.pm
+++ b/PVE/Diskmanage.pm
@@ -325,6 +325,7 @@ sub get_udev_info {

     $data->{serial} = 'unknown';
     $data->{serial} = $1 if $info =~ m/^E: ID_SERIAL_SHORT=(\S+)$/m;
+    $data->{serial} = $1 if $info =~ m/^E: ID_SCSI_SERIAL==(\S+)$/m;

     $data->{gpt} = $info =~ m/^E: ID_PART_TABLE_TYPE=gpt$/m ? 1 : 0;

That should overwrite the serial number if the ID_SCSI_SERIAL line is present.

If you change that directly on your (test) system (/usr/share/perl5/PVE/..., don't forget to do a systemctl restart pvedaemon so that the changes will be active.

 

[FelixJ]

Hmm, if you want to show changes, it is easiest if you run git diff, optionally with the path to the file in question.

I don't really understand your approach, though. Might be a bit clearer with a git diff output

Sorry about that, I don't consider myself as a coder.... so I have no routine with git at all, but I'll figure it out.
Regarding the actual coding: I have totaly overseen, that there is already an if-statement so my if then else is not necessary.

How ever, my actual question is, how can I push the variable $data->{smartdparam}, which I create a little earlier, to the get_smart_data function?

IIUC, the serial number is actually present in the output of udevadm, just under a different name?
Then the change should probably be only

diff --git a/PVE/Diskmanage.pm b/PVE/Diskmanage.pm
index a311ffd..1048732 100644
--- a/PVE/Diskmanage.pm
+++ b/PVE/Diskmanage.pm
@@ -325,6 +325,7 @@ sub get_udev_info {

     $data->{serial} = 'unknown';
     $data->{serial} = $1 if $info =~ m/^E: ID_SERIAL_SHORT=(\S+)$/m;
+    $data->{serial} = $1 if $info =~ m/^E: ID_SCSI_SERIAL==(\S+)$/m;

     $data->{gpt} = $info =~ m/^E: ID_PART_TABLE_TYPE=gpt$/m ? 1 : 0;

That should overwrite the serial number if the ID_SCSI_SERIAL line is present.

If you change that directly on your (test) system (/usr/share/perl5/PVE/..., don't forget to do a systemctl restart pvedaemon so that the changes will be active.

Thanks for the example. I will give my best to learn how to properly create diffs I can send you!
F

 

[mr44er]

On my SAS2308 and SAS3008 both in IT-Mode I also get only the Logical unit id.
These can and will change, regardless of the OS...also happens on FreeBSD, so my workaround was just a manual sticker with the OSD-number on the server tray.

 

[FelixJ]

On my SAS2308 and SAS3008 both in IT-Mode I also get only the Logical unit id.
These can and will change, regardless of the OS...also happens on FreeBSD, so my workaround was just a manual sticker with the OSD-number on the server tray.

Sure, but as you can see above, with a little bit of udev debugging and adding a few code-lines, you might even resolve that without labeling stickers on your Drive-trays.
I encourage you, to run that udevadm info command from earlier and send me the results. I could implement that in my patch. And if you know how to handle the smartrctl command in regards of the -d option, I could implement that as well for you so that you can see the SMART-status in the GUI.
regards,F

 

[mr44er]

I encourage you, to run that udevadm info command from earlier and send me the results.

It is the same variable ID_SCSI_SERIAL here, so just the GUI and changing ID_SERIAL_SHORT to ID_SCSI_SERIAL should be good.
SMART works ootb, because these are SAS-disks and with these Avago/Broadcom-Controllers it's 'real' IT-Mode with raw passthrough.

 

[FelixJ]

It is the same variable ID_SCSI_SERIAL here, so just the GUI and changing ID_SERIAL_SHORT to ID_SCSI_SERIAL should be good.
SMART works ootb, because these are SAS-disks and with these Avago/Broadcom-Controllers it's 'real' IT-Mode with raw passthrough.

Very well. Then my patch will work fine for you.

 

[aaron]

How ever, my actual question is, how can I push the variable $data->{smartdparam}, which I create a little earlier, to the get_smart_data function?

If I understand you and the code you posted correctly, and I am not sure if I do , then you have $data->{smartdparam} as the 3rd param when you call get_smart_data. That looks all good so far. But it seems like you got a curly bracket where it doesn't belong:

    #if (defined ($smartdparam})) {
                              ^

A real diff would make it easier to check what has changed where in the file

On the other hand, can you check if the GUI shows the correct serial number with the changes in my previous answer?

 

[FelixJ]

Hi Aaron,
this time I managed to create a diff-output:
The curly bracket was indeed the problem why pvedaeom would not restart properly thanks for that.
How ever, how do I get the content from my $data->{smartdparam} to properly to the get_smart_data function?
In other words: why isn't it working? I'd assume, it'd work that way.
How can I debug the content of $data->{smartdparam} outside the get_udev_info function, where I can temporarily append it to the $data->{serial} to have it displayed..

Regards,
Felix

*** /usr/share/perl5/PVE/Diskmanage.pm  Sun Feb 19 20:57:12 2023
--- Diskmanage.pm       Thu Feb 16 12:55:34 2023
***************
*** 87,93 ****
  }

  sub get_smart_data {
!     my ($disk, $healthonly, $smartdparam) = @_;

      assert_blockdev($disk);
      my $smartdata = {};
--- 87,93 ----
  }

  sub get_smart_data {
!     my ($disk, $healthonly) = @_;

      assert_blockdev($disk);
      my $smartdata = {};
***************
*** 100,113 ****
      }

      my $cmd = [$SMARTCTL, '-H'];
      #############
!     if (defined ($smartdparam)) {
!       push @$cmd, '-A', $smartdparam ,'-f', 'brief' if !$healthonly;
!     } else {
!       push @$cmd, '-A', '-f', 'brief' if !$healthonly;
      }
      ##############
-    push @$cmd, $disk;
      my $returncode = eval {
        run_command($cmd, noerr => 1, outfunc => sub {
            my ($line) = @_;
--- 100,112 ----
      }

      my $cmd = [$SMARTCTL, '-H'];
+     push @$cmd, '-A', '-f', 'brief' if !$healthonly;
+     push @$cmd, $disk;
      #############
!     if (defined ($data->{smartdparam})) {
!           push @$cmd, $data->{smartdparam};
      }
      ##############
      my $returncode = eval {
        run_command($cmd, noerr => 1, outfunc => sub {
            my ($line) = @_;
***************
*** 348,355 ****
      };
      $data->{smartdriver}="none";
      if ($hbainfo =~ m/^E: ID_MODEL_FROM_DATABASE=(.*Smart.*)$/m) {
!           $data->{smartdparam}="-d cciss,".$data->{devport};
!           my $smartdparam = $data->{smartdparam};
      }


--- 347,353 ----
      };
      $data->{smartdriver}="none";
      if ($hbainfo =~ m/^E: ID_MODEL_FROM_DATABASE=(.*Smart.*)$/m) {
!           $data->{smartdparam}="cciss,".$data->{devport};
      }
      

***************
*** 361,367 ****
      } else {
          $data->{serial} = $1 if $info =~ m/^E: ID_SERIAL_SHORT=(\S+)$/m;
      }
!     # $data->{serial} = $data->{serial}." ".$data->{smartdparam};
      #
      ###########
      $data->{gpt} = $info =~ m/^E: ID_PART_TABLE_TYPE=gpt$/m ? 1 : 0;
--- 359,365 ----
      } else {
          $data->{serial} = $1 if $info =~ m/^E: ID_SERIAL_SHORT=(\S+)$/m;
      }
!
      #
      ###########
      $data->{gpt} = $info =~ m/^E: ID_PART_TABLE_TYPE=gpt$/m ? 1 : 0;
***************
*** 591,597 ****
        my ($health, $wearout) = ('UNKNOWN', 'N/A');
        if (!$nosmart) {
            eval {
!               my $smartdata = get_smart_data($devpath, !is_ssdlike($type),$data->{smartdparam});
                $health = $smartdata->{health} if $smartdata->{health};

                if (is_ssdlike($type)) { # if we have an ssd we try to get the wearout indicator
--- 589,595 ----
        my ($health, $wearout) = ('UNKNOWN', 'N/A');
        if (!$nosmart) {
            eval {
!               my $smartdata = get_smart_data($devpath, !is_ssdlike($type));
                $health = $smartdata->{health} if $smartdata->{health};

                if (is_ssdlike($type)) { # if we have an ssd we try to get the wearout indicator

 

[aaron]

hmm... Your diff has parts not marked as changed that I cannot find.

Can you please diff it against the git repo https://git.proxmox.com/?p=pve-storage.git;a=summary ?
You can clone it with git clone git://git.proxmox.com/git/pve-storage.git

Alternatively, you can reinstall the package to revert to file (maybe copy it away first): apt install --reinstall libpve-storage-perl
Then let me know which version it is, so I know what I should compare it against: dpkg -l|grep libpve-storage

 

[renovator0306]

Hmm, if you want to show changes, it is easiest if you run git diff, optionally with the path to the file in question.

I don't really understand your approach, though. Might be a bit clearer with a git diff output

IIUC, the serial number is actually present in the output of udevadm, just under a different name?
Then the change should probably be only

diff --git a/PVE/Diskmanage.pm b/PVE/Diskmanage.pm
index a311ffd..1048732 100644
--- a/PVE/Diskmanage.pm
+++ b/PVE/Diskmanage.pm
@@ -325,6 +325,7 @@ sub get_udev_info {

     $data->{serial} = 'unknown';
     $data->{serial} = $1 if $info =~ m/^E: ID_SERIAL_SHORT=(\S+)$/m;
+    $data->{serial} = $1 if $info =~ m/^E: ID_SCSI_SERIAL==(\S+)$/m;

     $data->{gpt} = $info =~ m/^E: ID_PART_TABLE_TYPE=gpt$/m ? 1 : 0;

That should overwrite the serial number if the ID_SCSI_SERIAL line is present.

If you change that directly on your (test) system (/usr/share/perl5/PVE/..., don't forget to do a systemctl restart pvedaemon so that the changes will be active.

I had the same issue, but in my case the Serial was showing as 0s. Your code has an error, you were using double equal signs after ID_SCSCI_SERIAL. I corrected and serials are showing correctly.

Before - notice the SPCC drives


After code change


Git diff

@@ -329,6 +329,7 @@ sub get_udev_info {

     $data->{serial} = 'unknown';
     $data->{serial} = $1 if $info =~ m/^E: ID_SERIAL=(\S+)$/m;
+    $data->{serial} = $1 if $info =~ m/^E: ID_SCSI_SERIAL=(\S+)$/m;

     $data->{gpt} = $info =~ m/^E: ID_PART_TABLE_TYPE=gpt$/m ? 1 : 0;