Proxmox smartmontools error

[Chris Lorenz]

In Proxmox 4.4 on a clean Debian 8.7 Installation we got following error every time, when we were writing a lot of small files to the hard drive (e.g when installing a package):
(part of the syslog)
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.868015] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.872013] sd 3:1:0:0: [sda]
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.876013] Sense Key : Illegal Request [current]
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.880012] sd 3:1:0:0: [sda]
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.888035] Add. Sense: No additional sense information
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.892860] sd 3:1:0:0: [sda] CDB:
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.897651] Write Same(10): 41 00 01 40 24 e8 00 00 08 00
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.902597] sda3: WRITE SAME failed. Manually zeroing.

After 2 days of research, I found out that this problem is related to smartmontools and is not only happening within proxmox, also with the stable debian jessie smartmontools package

Removing it from the systemd start process fixed the problem. It occurs with all type of disks together with LSI 1064e Raid Controllers with a Raid 1 on it. This is not a disk error it seems that smartmontools is interacting with the controller. It even lead to disk errors, where the OS Partition was not readable anymore. And no this is no hardware related disk error or controller error, as it is running stable without the smartmontools on 3 independat blade servers with 3 diffrent types of hard drive and SSDs
I think this is a bigger Problem, as all Systems based on an LSI 1064e controller have that problem. Tested with:
IBM HS22V, HS22, HS21
Is it possible to remove smartmontools completely from the system. The problem here is that the package is dependant to the proxmox-ve packages

Chris

 

[Rhinox]

smartmontools do not go well with lsi (or any) raid controllers, even with jbod. Somewhere on avago/lsi web I have seen recommendation not to use it. Even on smartmontools-web you can find a notice it works with *some* raid controllers, and does not work with others...

The other point is, there is not much sense in using hw raid controller (local) with PVE. But if you have to, at least re-flash it to IT-mode (if you have not done so already) so that presents drives to OS without any filtering...

 

[fabian]

In Proxmox 4.4 on a clean Debian 8.7 Installation we got following error every time, when we were writing a lot of small files to the hard drive (e.g when installing a package):
(part of the syslog)
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.868015] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.872013] sd 3:1:0:0: [sda]
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.876013] Sense Key : Illegal Request [current]
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.880012] sd 3:1:0:0: [sda]
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.888035] Add. Sense: No additional sense information
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.892860] sd 3:1:0:0: [sda] CDB:
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.897651] Write Same(10): 41 00 01 40 24 e8 00 00 08 00
Apr 11 04:08:34 v8d-16-177-14-1-alt kernel: [ 273.902597] sda3: WRITE SAME failed. Manually zeroing.

After 2 days of research, I found out that this problem is related to smartmontools and is not only happening within proxmox, also with the stable debian jessie smartmontools package

Removing it from the systemd start process fixed the problem. It occurs with all type of disks together with LSI 1064e Raid Controllers with a Raid 1 on it. This is not a disk error it seems that smartmontools is interacting with the controller. It even lead to disk errors, where the OS Partition was not readable anymore. And no this is no hardware related disk error or controller error, as it is running stable without the smartmontools on 3 independat blade servers with 3 diffrent types of hard drive and SSDs
I think this is a bigger Problem, as all Systems based on an LSI 1064e controller have that problem. Tested with:
IBM HS22V, HS22, HS21
Is it possible to remove smartmontools completely from the system. The problem here is that the package is dependant to the proxmox-ve packages

Chris

you can just tell smartd to not monitor the affected disks? see "man smartd.conf" (or disable smartd altogether if you don't want to use it).

 

[Alessandro 123]

How do you call smartctl ? Are you using "-d megaraid" as wrote in docs ?

 

[Chris Lorenz]

First of all Thanks for your replies

Yes I disabled it in smartd.conf, which had the same effect as disabling the service in systemd. The solution is basically ok, but the problem which I saw was already during the installation of proxmox, because the service was automatically started during the installation of all the packages. (I dont know if this also occures if you install proxmox from the prebuild DVD, because I have only tried to set it up on top of Debian 8 to have my own partition layout) Normally this doesn't lead to problems, but one time it destroyed my installation, as the partition was not readable anymore. But when the installation completes successfully (more or less as the errors occur during the installation, but with luck the partition doesn't get corrupted) you can disable the smartd daemon and everything is alright. The question is, is it possible to remove the dependency from the proxmox package list during install, because then I would be shure that the installation completes without any problems

@Rhinox: Yes I know that smartmontools are not working good with any hardware raid controllers, but for the operating system, I still want to use the hardware based raid systems. Simply making the controller only an HBA would be possible, but I do not like the idea of a software raid for the OS, swap etc. For the data part we are anyway not using the local disks installed in the blades, as they are way to small to be used for VMs

@Alessandro 123: I do not call smartctl, it is already the daemon, which is producing the SCSI Command errors installed together with the smartmontool package. https://forum.proxmox.com/members/rhinox.39782/
https://forum.proxmox.com/members/rhinox.39782/
https://forum.proxmox.com/members/alessandro-123.37750/

 

[Alessandro 123]

You have to configure the daemon to use the "-d megaraid" flag when trying to access devices under a megaraid controller

 

[mir]

@Alessandro 123: I do not call smartctl, it is already the daemon, which is producing the SCSI Command errors installed together with the smartmontool package.

In /etc/smartd.conf you can specify how smartd should call your disks.
An example from smartd.conf:
# Monitor 2 ATA disks connected to a 3ware 9000 controller which
# uses the 3w-9xxx driver (Linux, FreeBSD). Start long self-tests Tuesdays
# between 1-2 and 3-4 am.
#/dev/twa0 -d 3ware,0 -a -s L/../../2/01
#/dev/twa0 -d 3ware,1 -a -s L/../../2/03

 

[Chris Lorenz]

Thanks for your response mir, I´ll try it out after the easter holidays, when I have access to the systems again