3 node ceph cluster don't boot after some reboots

[cheissol]

Hi,
I configured a cluster with ceph to share disks with all 3 nodes. Everything worked fine but after some reboots in all nodes I get this error:

...
[TIME] Timed out waiting for device dev-pve-data.device
[DEPEND] Dependency failed for /var/lib/vz
[DEPEND] Dependency failed for Local File Systems
[DEPEND] Dependency failed for File System Check on /dev/pve/data
[DEPEND] Dependency failed for /var/lib/vz
[TIME] Timed out waiting for device dev-pve-swap.device
[DEPEND] Dependency failed for /dve/pve/swap
[DEPEND] Dependency failed for Swap
...

I've installed proxmox 4.1 in all nodes with subscription key so every paket is updated.

Where could be the error?

 

[wolfgang]

Hi,

can you be a bit more specific?
because this errors say you lvm can't mount.
What output you get with lsblk?

 

[cheissol]

This is node 3 (in every node there's the same problem):

#lsblk
NAME  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda  8:0  0 136,1G  0 disk
├─sda1  8:1  0  1007K  0 part
├─sda2  8:2  0  127M  0 part
└─sda3  8:3  0  136G  0 part
  ├─pve-root 252:0  0  34G  0 lvm  /
  ├─pve-swap 252:1  0  15G  0 lvm  [SWAP]
  └─pve-data 252:2  0  71G  0 lvm  /var/lib/vz
sdb  8:16  0 136,1G  0 disk
├─sdb1  8:17  0 131,1G  0 part
└─sdb2  8:18  0  5G  0 part
sdc  8:32  0 136,1G  0 disk
├─sdc1  8:33  0 131,1G  0 part
└─sdc2  8:34  0  5G  0 part
sdd  8:48  0 136,1G  0 disk
├─sdd1  8:49  0 131,1G  0 part
└─sdd2  8:50  0  5G  0 part
sde  8:64  0 558,4G  0 disk
├─sde1  8:65  0 553,4G  0 part
└─sde2  8:66  0  5G  0 part
sdf  8:80  0 558,4G  0 disk
├─sdf1  8:81  0 553,4G  0 part
└─sdf2  8:82  0  5G  0 part
sr0  11:0  1  1024M  0 rom 
sr1  11:1  1  1024M  0 rom

 

[wolfgang]

Your lvm looks ok.
Do you see more in the journal?

 

[cheissol]

# journalctl -b -p 3

mar 11 14:51:42 vs3 kernel: ERST: Failed to get Error Log Address Range.
mar 11 14:51:42 vs3 kernel: megasas: INIT adapter done
mar 11 14:51:42 vs3 systemd-sysv-generator[244]: Ignoring creation of an alias umountiscsi.service for itself
mar 11 14:51:43 vs3 kernel: Error: Driver 'pcspkr' is already registered, aborting...
mar 11 14:53:12 vs3 systemd[1]: Timed out waiting for device dev-pve-data.device.
mar 11 14:53:12 vs3 systemd[1]: Dependency failed for /var/lib/vz.
mar 11 14:53:12 vs3 systemd[1]: Dependency failed for Local File Systems.
mar 11 14:53:12 vs3 systemd[1]: Dependency failed for File System Check on /dev/pve/data.
mar 11 14:53:12 vs3 systemd[1]: Timed out waiting for device dev-pve-swap.device.
mar 11 14:53:12 vs3 systemd[1]: Dependency failed for /dev/pve/swap.
mar 11 14:53:12 vs3 systemd[1]: Dependency failed for Swap.
mar 11 14:53:13 vs3 systemd-udevd[309]: error changing net interface name eth6 to eth10: Success
mar 11 14:53:13 vs3 systemd-udevd[309]: timeout 'bridge-network-interface'
mar 11 14:53:13 vs3 systemd-udevd[309]: timeout 'net.agent'
mar 11 14:53:13 vs3 systemd-udevd[309]: timeout '/lib/systemd/systemd-sysctl --prefix=/proc/sys/net/ipv4/conf/eth10 --prefix=/proc/sys/net/ipv4/neigh/eth10 --prefix=/proc/sys/net/ipv6/conf/eth10 --prefix=/proc/sys/net/ipv6/neigh/eth10'
mar 11 14:53:17 vs3 systemd[883]: Failed at step EXEC spawning /bin/plymouth: No such file or directory
mar 11 14:53:17 vs3 pmxcfs[1374]: [quorum] crit: quorum_initialize failed: 2
mar 11 14:53:17 vs3 pmxcfs[1374]: [quorum] crit: can't initialize service
mar 11 14:53:17 vs3 pmxcfs[1374]: [confdb] crit: cmap_initialize failed: 2
mar 11 14:53:17 vs3 pmxcfs[1374]: [confdb] crit: can't initialize service
mar 11 14:53:17 vs3 pmxcfs[1374]: [dcdb] crit: cpg_initialize failed: 2
mar 11 14:53:17 vs3 pmxcfs[1374]: [dcdb] crit: can't initialize service
mar 11 14:53:17 vs3 pmxcfs[1374]: [status] crit: cpg_initialize failed: 2
mar 11 14:53:17 vs3 pmxcfs[1374]: [status] crit: can't initialize service
mar 11 14:53:18 vs3 iscsid[1378]: iSCSI daemon with pid=1379 started!
mar 11 14:53:23 vs3 pmxcfs[1374]: [quorum] crit: quorum_initialize failed: 2
mar 11 14:53:23 vs3 pmxcfs[1374]: [confdb] crit: cmap_initialize failed: 2
mar 11 14:53:23 vs3 pmxcfs[1374]: [dcdb] crit: cpg_initialize failed: 2
mar 11 14:53:23 vs3 pmxcfs[1374]: [status] crit: cpg_initialize failed: 2