[SOLVED] Start job for removed drive and drive open failed

A

antipiot

Well-Known Member
Jan 26, 2019
67
5
48
37
Hello everybody!

i have 2 little issues when my PVE environnement is starting:

1st: theres a start job running for a drive that has been removed:
"a start job is running for /dev/disk/by-uuid/c3577............"
there's no entries for this in fstab nor in pve.

How can i remove this entry?

2nd: the host fail to open path where there's no drive: any chance to get rid of this?

1597823311789.png
lsblk shows:
1597823380303.png

Thanks for your work and help!
Long live Proxmox
 
Hi,
ad 1) was the device removed before the last reboot? What's the output of systemctl status /dev/disk/by-uuid/c3577...?
ad 2) what do pvs and vgs show?
 
Fabian_E said:
Hi,
ad 1) was the device removed before the last reboot? What's the output of systemctl status /dev/disk/by-uuid/c3577...?
ad 2) what do pvs and vgs show?
Click to expand...
Thanks for your help:.
The drive was removed many reboots ago.

here's the output for systemctl


Code: 
Aug 19 09:36:29 pve1 systemd[1]: dev-disk-by\x2duuid-c35eef55\x2dad70\x2d40e5\x2dab01\x2d0a2ab3560d92.device: Job dev-disk-by\x2duuid-c35eef55\x2dad70\x2d40e5\x2dab01\x2d0a2ab3560d92.device/start timed out.
Aug 19 09:36:29 pve1 systemd[1]: Timed out waiting for device /dev/disk/by-uuid/c35eef55-ad70-40e5-ab01-0a2ab3560d92.
Aug 19 09:36:29 pve1 systemd[1]: dev-disk-by\x2duuid-c35eef55\x2dad70\x2d40e5\x2dab01\x2d0a2ab3560d92.device: Job dev-disk-by\x2duuid-c35eef55\x2dad70\x2d40e5\x2dab01\x2d0a2ab3560d92.device/start failed with result 'timeout'.

PVS:
Code: 
root@pve1:~# pvs
  PV             VG  Fmt  Attr PSize    PFree
  /dev/nvme0n1p3 pve lvm2 a--  <953.37g <16.00g

VGS:
Code: 
  VG  #PV #LV #SN Attr   VSize    VFree
  pve   1  16   0 wz--n- <953.37g <16.00g
root@pve1:~#


Regards
 
Ad 1) Use ls -l /dev/disk/by-uuid/c35eef55-ad70-40e5-ab01-0a2ab3560d92 to see where it points to. Are you sure there is no entry in the fstab for this device? Is there no line like
Code: 
Device: /sys/devices/pci0000:00/0000:00:05.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda/sda2
in the output of the systemctl command? You could then use lspci to see where it is (supposedly) connected.

Ad 2) This kind of message might be coming from card readers. See this thread for the details. In short: Use dmesg | grep "sd[cdefhijk]" to see where these devices come from. If they come from things like card readers and you don't plan on using any of these for LVM, you can add it to the global_filter in /etc/lvm.conf, but it's most likely only a cosmetic issue.
 
Fabian_E said:
Ad 1) Use ls -l /dev/disk/by-uuid/c35eef55-ad70-40e5-ab01-0a2ab3560d92 to see where it points to. Are you sure there is no entry in the fstab for this device? Is there no line like
Code: 
Device: /sys/devices/pci0000:00/0000:00:05.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda/sda2
in the output of the systemctl command? You could then use lspci to see where it is (supposedly) connected.

Ad 2) This kind of message might be coming from card readers. See this thread for the details. In short: Use dmesg | grep "sd[cdefhijk]" to see where these devices come from. If they come from things like card readers and you don't plan on using any of these for LVM, you can add it to the global_filter in /etc/lvm.conf, but it's most likely only a cosmetic issue.
Click to expand...
The output shows this:
Code: 
root@pve1:~# ls -l /dev/disk/by-uuid/c35eef55-ad70-40e5-ab01-0a2ab3560d92
ls: cannot access '/dev/disk/by-uuid/c35eef55-ad70-40e5-ab01-0a2ab3560d92': No such file or directory

1:
Absolutly sure there's no entries in fstab:
Code: 
root@pve1:~# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

2:
This is the output: still no sure to understand where it comes from but yeah, no doubt it's a cosmetic issue :-)

Code: 
root@pve1:~# dmesg | grep "sd[cdefhijk]"
[    3.958472] sd 13:0:0:3: [sdf] Attached SCSI removable disk
[    3.958804] sd 13:0:0:2: [sde] Attached SCSI removable disk
[    3.959028] sd 13:0:0:1: [sdd] Attached SCSI removable disk
[    3.981857] sd 13:0:0:0: [sdc] Attached SCSI removable disk
[    4.214449] sd 14:0:0:1: [sdi] Attached SCSI removable disk
[    4.214718] sd 14:0:0:0: [sdh] Attached SCSI removable disk
[    4.214960] sd 14:0:0:3: [sdk] Attached SCSI removable disk
[    4.215192] sd 14:0:0:2: [sdj] Attached SCSI removable disk

Regards - JS
 
Ad 1) Could you check the /run/systemd/generator directory for the relevant .mount service? The SourcePath property (within the file) should indicate where it came from/why it was generated.

Ad 2) Maybe it's a controller/firmware bug mistakenly claiming that something is connected.
You can try using commands like
Code: 
find /sys/devices | grep 13:0:0:3$
in combination with lspci to see where the pseudo-device is supposedly connected.

For example what I get with a connected virtual disk (focus on the bold parts):
root@pbs ~ # dmesg| grep sda
[ 1.762438] sd 2:0:0:0: [sda] 67108864 512-byte logical blocks: (34.4 GB/32.0 GiB)
[ 1.762477] sd 2:0:0:0: [sda] Write Protect is off
[ 1.762479] sd 2:0:0:0: [sda] Mode Sense: 63 00 00 08
[ 1.762629] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 1.776263] sda: sda1 sda2 sda3
[ 1.777036] sd 2:0:0:0: [sda] Attached SCSI disk
root@pbs ~ # find /sys/devices | grep 2:0:0:0$
/sys/devices/pci0000:00/0000:00:05.0/virtio1/host2/target2:0:0/2:0:0:0
/sys/devices/pci0000:00/0000:00:05.0/virtio1/host2/target2:0:0/2:0:0:0/scsi_device/2:0:0:0
/sys/devices/pci0000:00/0000:00:05.0/virtio1/host2/target2:0:0/2:0:0:0/scsi_disk/2:0:0:0
/sys/devices/pci0000:00/0000:00:05.0/virtio1/host2/target2:0:0/2:0:0:0/bsg/2:0:0:0
root@pbs ~ # lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:03.0 Unclassified device [00ff]: Red Hat, Inc Virtio memory balloon
00:05.0 SCSI storage controller: Red Hat, Inc Virtio SCSI
00:12.0 Ethernet controller: Red Hat, Inc Virtio network device
00:13.0 Ethernet controller: Red Hat, Inc Virtio network device
00:14.0 Ethernet controller: Red Hat, Inc Virtio network device
00:1e.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
00:1f.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
Click to expand...
 
Fabian_E said:
Ad 1) Could you check the /run/systemd/generator directory for the relevant .mount service? The SourcePath property (within the file) should indicate where it came from/why it was generated.

Ad 2) Maybe it's a controller/firmware bug mistakenly claiming that something is connected.
Click to expand...


1:
This is the content of the .mount file (wich is named -.mount on my side :-/) is this related?!
EDIT: this looks to be correct indeed :-)
1598259252535.png
Code: 
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
Where=/
What=/dev/pve/root
Type=ext4
Options=errors=remount-ro

2:
Strange as i have nothing returned:

Code: 
root@pve1:~#  dmesg| grep sdf
[    3.965956] sd 13:0:0:3: [sdf] Attached SCSI removable disk


root@pve1:~# find /sys/devices | grep 13:0:0$
root@pve1:~#

root@pve1:~#  dmesg| grep sdc
[    3.966350] sd 13:0:0:0: [sdc] Attached SCSI removable disk
root@pve1:~# find /sys/devices | grep 13:0:0$
root@pve1:~#
 
Last edited:
antipiot said:
1:
This is the content of the .mount file (wich is named -.mount on my side :-/) is this related?!
EDIT: this looks to be correct indeed :)
View attachment 19349
Code: 
# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target

[Mount]
Where=/
What=/dev/pve/root
Type=ext4
Options=errors=remount-ro
Click to expand...

This is the service to mount the root partition, it should be there. What about the .swap service? Otherwise try grep -r c35eef55 /etc to see if there are any references in some configuration file.

2:
Strange as i have nothing returned:

Code: 
root@pve1:~#  dmesg| grep sdf
[    3.965956] sd 13:0:0:3: [sdf] Attached SCSI removable disk


root@pve1:~# find /sys/devices | grep 13:0:0$
root@pve1:~#

root@pve1:~#  dmesg| grep sdc
[    3.966350] sd 13:0:0:0: [sdc] Attached SCSI removable disk
root@pve1:~# find /sys/devices | grep 13:0:0$
root@pve1:~#
Click to expand...
Well since there is no device connected, it's not all that strange. I just thought it might be worth a shot. Could you post a larger portion of the dmesg output surrounding the Attached SCSI removable disk lines? There might be more information there.
 
Fabian_E said:
This is the service to mount the root partition, it should be there. What about the .swap service? Otherwise try grep -r c35eef55 /etc to see if there are any references in some configuration file.
Click to expand...

There it is!
Code: 
root@pve1:~# grep -r c35eef55 /etc
/etc/systemd/system/mnt-pve-BACKUPS.mount:What=/dev/disk/by-uuid/c35eef55-ad70-40e5-ab01-0a2ab3560d92

i guess i can remove this line (or at least comment it)?

Fabian_E said:
Well since there is no device connected, it's not all that strange. I just thought it might be worth a shot. Could you post a larger portion of the dmesg output surrounding the Attached SCSI removable disk lines? There might be more information there.
Click to expand...

Here you go!

Code: 
[    2.696820] sd 3:0:0:0: Attached scsi generic sg0 type 0
[    2.763231] sd 3:0:0:0: [sda] Attached SCSI disk
[    3.169415] sd 4:0:0:0: Attached scsi generic sg1 type 0
[    3.198658] sd 4:0:0:0: [sdb] Attached SCSI disk
[    3.436822] sr 12:0:0:0: Attached scsi CD-ROM sr0
[    3.436888] sr 12:0:0:0: Attached scsi generic sg2 type 5
[    3.464426] sr 12:0:0:1: Attached scsi CD-ROM sr1
[    3.464480] sr 12:0:0:1: Attached scsi generic sg3 type 5
[    3.832685] sr 12:0:0:2: Attached scsi CD-ROM sr2
[    3.832746] sr 12:0:0:2: Attached scsi generic sg4 type 5
[    3.872511] sr 12:0:0:3: Attached scsi CD-ROM sr3
[    3.872568] sr 12:0:0:3: Attached scsi generic sg5 type 5
[    3.937127] sd 13:0:0:0: Attached scsi generic sg6 type 0
[    3.937209] sd 13:0:0:1: Attached scsi generic sg7 type 0
[    3.937283] sd 13:0:0:2: Attached scsi generic sg8 type 0
[    3.937359] sd 13:0:0:3: Attached scsi generic sg9 type 0
[    3.961736] sd 6:0:0:0: Attached scsi generic sg10 type 0
[    3.965382] sd 13:0:0:1: [sdd] Attached SCSI removable disk
[    3.965712] sd 13:0:0:2: [sde] Attached SCSI removable disk
[    3.965956] sd 13:0:0:3: [sdf] Attached SCSI removable disk
[    3.966350] sd 13:0:0:0: [sdc] Attached SCSI removable disk
[    4.016876] sd 6:0:0:0: [sdg] Attached SCSI disk
[    4.192328] sd 14:0:0:0: Attached scsi generic sg11 type 0
[    4.192414] sd 14:0:0:1: Attached scsi generic sg12 type 0
[    4.192505] sd 14:0:0:2: Attached scsi generic sg13 type 0
[    4.192575] sd 14:0:0:3: Attached scsi generic sg14 type 0
[    4.213367] sd 14:0:0:0: [sdh] Attached SCSI removable disk
[    4.230281] sd 14:0:0:1: [sdi] Attached SCSI removable disk
[    4.230521] sd 14:0:0:3: [sdk] Attached SCSI removable disk
[    4.238266] sd 14:0:0:2: [sdj] Attached SCSI removable disk
[    4.438870] sd 7:0:0:0: Attached scsi generic sg15 type 0
[    4.439110] sd 8:0:0:0: Attached scsi generic sg16 type 0
[    4.478318] sd 7:0:0:0: [sdl] Attached SCSI disk
[    4.485145] sd 8:0:0:0: [sdm] Attached SCSI disk
[    4.919112] sd 9:0:0:0: Attached scsi generic sg17 type 0
[    4.966322] sd 9:0:0:0: [sdn] Attached SCSI disk

Thanks for your help
 
antipiot said:
There it is!
Code: 
root@pve1:~# grep -r c35eef55 /etc
/etc/systemd/system/mnt-pve-BACKUPS.mount:What=/dev/disk/by-uuid/c35eef55-ad70-40e5-ab01-0a2ab3560d92

i guess i can remove this line (or at least comment it)?
Click to expand...
Do you still have the pve/BACKUPS logical volume? I think you'd need to remove the whole file, otherwise you'll just have an invalid systemd mount unit.

Here you go!

Code: 
[    2.696820] sd 3:0:0:0: Attached scsi generic sg0 type 0
[    2.763231] sd 3:0:0:0: [sda] Attached SCSI disk
[    3.169415] sd 4:0:0:0: Attached scsi generic sg1 type 0
[    3.198658] sd 4:0:0:0: [sdb] Attached SCSI disk
[    3.436822] sr 12:0:0:0: Attached scsi CD-ROM sr0
[    3.436888] sr 12:0:0:0: Attached scsi generic sg2 type 5
[    3.464426] sr 12:0:0:1: Attached scsi CD-ROM sr1
[    3.464480] sr 12:0:0:1: Attached scsi generic sg3 type 5
[    3.832685] sr 12:0:0:2: Attached scsi CD-ROM sr2
[    3.832746] sr 12:0:0:2: Attached scsi generic sg4 type 5
[    3.872511] sr 12:0:0:3: Attached scsi CD-ROM sr3
[    3.872568] sr 12:0:0:3: Attached scsi generic sg5 type 5
[    3.937127] sd 13:0:0:0: Attached scsi generic sg6 type 0
[    3.937209] sd 13:0:0:1: Attached scsi generic sg7 type 0
[    3.937283] sd 13:0:0:2: Attached scsi generic sg8 type 0
[    3.937359] sd 13:0:0:3: Attached scsi generic sg9 type 0
[    3.961736] sd 6:0:0:0: Attached scsi generic sg10 type 0
[    3.965382] sd 13:0:0:1: [sdd] Attached SCSI removable disk
[    3.965712] sd 13:0:0:2: [sde] Attached SCSI removable disk
[    3.965956] sd 13:0:0:3: [sdf] Attached SCSI removable disk
[    3.966350] sd 13:0:0:0: [sdc] Attached SCSI removable disk
[    4.016876] sd 6:0:0:0: [sdg] Attached SCSI disk
[    4.192328] sd 14:0:0:0: Attached scsi generic sg11 type 0
[    4.192414] sd 14:0:0:1: Attached scsi generic sg12 type 0
[    4.192505] sd 14:0:0:2: Attached scsi generic sg13 type 0
[    4.192575] sd 14:0:0:3: Attached scsi generic sg14 type 0
[    4.213367] sd 14:0:0:0: [sdh] Attached SCSI removable disk
[    4.230281] sd 14:0:0:1: [sdi] Attached SCSI removable disk
[    4.230521] sd 14:0:0:3: [sdk] Attached SCSI removable disk
[    4.238266] sd 14:0:0:2: [sdj] Attached SCSI removable disk
[    4.438870] sd 7:0:0:0: Attached scsi generic sg15 type 0
[    4.439110] sd 8:0:0:0: Attached scsi generic sg16 type 0
[    4.478318] sd 7:0:0:0: [sdl] Attached SCSI disk
[    4.485145] sd 8:0:0:0: [sdm] Attached SCSI disk
[    4.919112] sd 9:0:0:0: Attached scsi generic sg17 type 0
[    4.966322] sd 9:0:0:0: [sdn] Attached SCSI disk

Thanks for your help
Click to expand...

Is there anything still earlier in the log that indicates what these are? For example, here it's some card reader connected via USB.
 
  • Like
Reactions: antipiot
Fabian_E said:
Do you still have the pve/BACKUPS logical volume? I think you'd need to remove the whole file, otherwise you'll just have an invalid systemd mount unit.



Is there anything still earlier in the log that indicates what these are? For example, here it's some card reader connected via USB.
Click to expand...

Many thanks to you: Both problems solved:

1:
Deleting the old "/etc/systemd/system/mnt-pve-BACKUPS.mount" file solved the start job error.

2: feeling dumb but:
Thoses fantomatic drives where coming from the ipmi of my Asrack motherboard:
Annotation 2020-08-25 160648.png

Annotation 2020-08-25 160703.png

Here there was 4 of each - setted to 0 - saved and rebooted proxmox host: all good.
Annotation 2020-08-25 160031.png
Many thanks for your help!
 
You must log in or register to reply here.
Top Bottom