Invisible SATA disk

[kana]

Hi all,
I've got a strange phenomenon where a SATA disk is connected to my mcomputer, but Proxmox is not able to see it, however, a Debian live installation disk can see it fine.
Here's what I tried in Proxmox:

• it's not listed with lsblk
• neither it is with fdisk -l
• ls /dev/sd* -> "ls: cannot access '/dev/sd*': No such file or directory"
• "smartctl --scan" results in only the NVME drive
• interestingly, there's a one hint that it can see something somehow (I've got only the one invisible SATA drive in the system):
  dmesg | grep -i sata
  ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
• to assure that the disk is not assigned to a VM, I checked them all manually and additionly I had script where this
  "qm config $vm_id | grep -E '(scsi|sata|ide|virtio|pci)'"
  loops through all VMs, disk is not assigned

When having booted the Debian live disk,

• lsblk listed the drive right away
• the partition manager would see it, there I could partition it etc.

The BIOS POST also shows it.

I think a HW issue can be excluded, what can be misconfigured in Proxmox that it cannot see the disk?
BTW, two USB sticks that I can see in live Debian (there I have /dev/sda, sdb and sdc) are invisible in Proxmox, too.

This is a mystery to me, has anbody got an idea?

 

[dcsapak]

HI,

can you post the output of 'dmesg' 'lsblk' and 'ls -l /dev' from the proxmox installation and from a working debian live disk?
it sounds like the sata controller/hba is either not recognized or the driver is broken (or something like that)

BTW, two USB sticks that I can see in live Debian (there I have /dev/sda, sdb and sdc) are invisible in Proxmox, too.

this also sounds very weird

 

[kana]

Hi,
yes of course!
I pasted lsblk here and attached the other outputs as files due to their size.

I just looked through the two dmesg logs, and there are the same messages concerning SATA and sda in both logs, except for one occurrence
[ 20.516242] sd 0:0:0:0: [sda] Synchronizing SCSI cache
which is unique to Proxmox.
Doesn't still make sense to me.

And I just noticed that nvme0n1 is not recognized either.

Proxmox:


root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
zd0 230:0 0 1M 0 disk
zd16 230:16 0 32G 0 disk
├─zd16p1 230:17 0 32M 0 part
├─zd16p2 230:18 0 24M 0 part
├─zd16p3 230:19 0 256M 0 part
├─zd16p4 230:20 0 24M 0 part
├─zd16p5 230:21 0 256M 0 part
├─zd16p6 230:22 0 8M 0 part
├─zd16p7 230:23 0 96M 0 part
└─zd16p8 230:24 0 31.3G 0 part
zd32 230:32 0 636M 0 disk
├─zd32p1 230:33 0 16M 0 part
├─zd32p2 230:34 0 619.7M 0 part
└─zd32p128 259:7 0 239K 0 part
zd48 230:48 0 64G 0 disk
└─zd48p1 230:49 0 64G 0 part
zd64 230:64 0 32G 0 disk
zd80 230:80 0 1M 0 disk
zd96 230:96 0 80G 0 disk
├─zd96p1 230:97 0 100M 0 part
├─zd96p2 230:98 0 16M 0 part
├─zd96p3 230:99 0 79.1G 0 part
└─zd96p4 230:100 0 775M 0 part
zd112 230:112 0 1M 0 disk
zd128 230:128 0 1M 0 disk
zd144 230:144 0 4M 0 disk
nvme1n1 259:0 0 953.9G 0 disk
├─nvme1n1p1 259:2 0 1007K 0 part
├─nvme1n1p2 259:3 0 1G 0 part
└─nvme1n1p3 259:4 0 952.9G 0 part
root@pve:~#

Debian Live:


user@debian:~$ lsblk

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0   2.8G  1 loop /usr/lib/live/mount/rootfs/filesystem.squashfs
                                      /run/live/rootfs/filesystem.squashfs
sda           8:0    0   7.3T  0 disk
└─sda1        8:1    0   7.3T  0 part
sdb           8:16   1  14.3G  0 disk
└─sdb1        8:17   1  14.3G  0 part
sdc           8:32   1  28.8G  0 disk
├─sdc1        8:33   1  28.8G  0 part
└─sdc2        8:34   1    32M  0 part
nvme0n1     259:0    0 447.1G  0 disk
├─nvme0n1p1 259:2    0 447.1G  0 part
└─nvme0n1p9 259:3    0     8M  0 part
nvme1n1     259:1    0 953.9G  0 disk
├─nvme1n1p1 259:4    0  1007K  0 part
├─nvme1n1p2 259:5    0     1G  0 part
└─nvme1n1p3 259:6 0 952.9G 0 part

 

[kana]

I made another experiment: I booted into the terminal from the Proxmox installation iso, and this one saw sda, sdb and sdc, also, fdisk -l listed both the missing 8TB disk as well as the nvme device.
lsblk seems not available for some reason in the installer.

So Proxmox has all the means it needs to see my disks, but I must have baadly misconfigured it...

 

[dcsapak]

Doesn't still make sense to me.

does not make much sense to me either... the only thing i could think of would be if your udev rules are somehow messed up, did you maybe change something there?
also could you maybe post a 'journalctl -b' from the proxmox installations? maybe i can see something there

 

[kana]

udev rules was the correct hint. I looked through the rules and found
/lib/udev/rules.d/60-block.rules
In there I saw this suspicious section and uncommented it:

ACTION!="remove", SUBSYSTEM=="block", \
 KERNEL=="loop*|mmcblk*[0-9]|msblk*[0-9]|mspblk*[0-9]|nvme*|sd*|vd*|xvd*|bcache*|cciss*|dasd*|ubd*|ubi*|scm>
  OPTIONS+="watch"

That has the bad effect, that I can access my server only via ssh but no more via web interface, so this cannot be the final solution, but Proxmox does now list the missing devices:

root@pve:~# lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0   7.3T  0 disk
└─sda1        8:1    0   7.3T  0 part
sdb           8:16   1  14.3G  0 disk
└─sdb1        8:17   1  14.3G  0 part
zd0         230:0    0     1M  0 disk
zd16        230:16   0    32G  0 disk
├─zd16p1    230:17   0    32M  0 part
├─zd16p2    230:18   0    24M  0 part
├─zd16p3    230:19   0   256M  0 part
├─zd16p4    230:20   0    24M  0 part
├─zd16p5    230:21   0   256M  0 part
├─zd16p6    230:22   0     8M  0 part
├─zd16p7    230:23   0    96M  0 part
└─zd16p8    230:24   0  31.3G  0 part
zd32        230:32   0   636M  0 disk
├─zd32p1    230:33   0    16M  0 part
├─zd32p2    230:34   0 619.7M  0 part
└─zd32p128  259:7    0   239K  0 part
zd48        230:48   0    64G  0 disk
└─zd48p1    230:49   0    64G  0 part
zd64        230:64   0    32G  0 disk
zd80        230:80   0     1M  0 disk
zd96        230:96   0    80G  0 disk
├─zd96p1    230:97   0   100M  0 part
├─zd96p2    230:98   0    16M  0 part
├─zd96p3    230:99   0  79.1G  0 part
└─zd96p4    230:100  0   775M  0 part
zd112       230:112  0     1M  0 disk
zd128       230:128  0     1M  0 disk
zd144       230:144  0     4M  0 disk
nvme0n1     259:0    0 953.9G  0 disk
├─nvme0n1p1 259:2    0  1007K  0 part
├─nvme0n1p2 259:3    0     1G  0 part
└─nvme0n1p3 259:4    0 952.9G  0 part
nvme1n1     259:1    0 447.1G  0 disk
├─nvme1n1p1 259:5    0 447.1G  0 part
└─nvme1n1p9 259:6    0     8M  0 part

Having no idea what this rule is about, what could I do next to get both the disks and the web interface?

 

[kana]

I did some research and now have a rough idea what udev rules are about. I also consulted an AI (which was helpful only to some extent).
What I did: I comment out only the "WATCH" part in that rule -> no effect
I also bootet into the initramfs shell and would see the sda, b, c devices, which is in line with the theory that they get removed somehow during the boot process.
I then captured the journal as you suggested, file is attached. sda is mentioned a lot in there, but no hint that it got removed at some point.