SOLVED - 4.3 - /dev/pve/data disappears after adding to cluster

[term]

I built a new server running 4.3 and added it to a existing 4.3 cluster. I had 1 kvm on the machine that I knew would not have any conflicts with existing IDs. I also had a dump of this in case anything went wrong.
I used the --force option to force adding to the cluster, then rebooted. At that point, /dev/pve/data was gone from /etc/fstab. If I add it back in to /etc/fstab, the server boots into emergency mode. I can see the message: "a start job is running for dev-pve-data.device" on the console. I'm using the standard basic ext4 drive layout. When I don't attempt to mount the drive on boot, I can see the old data/partitions there. What do I need to do to get pve-data to mount on boot? Do I need to run fsck to make it happy?


lsblk:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 127M 0 part
└─sda3 8:3 0 930.9G 0 part
├─pve-root 251:0 0 96G 0 lvm /
├─pve-swap 251:1 0 8G 0 lvm [SWAP]
├─pve-data_tmeta 251:2 0 104M 0 lvm
│ └─pve-data-tpool 251:4 0 810.9G 0 lvm
│ ├─pve-data 251:5 0 810.9G 0 lvm
│ └─pve-vm--501--disk--1 251:6 0 65G 0 lvm
└─pve-data_tdata 251:3 0 810.9G 0 lvm
└─pve-data-tpool 251:4 0 810.9G 0 lvm
├─pve-data 251:5 0 810.9G 0 lvm
└─pve-vm--501--disk--1 251:6 0 65G 0 lvm
sdb 8:16 0 446.6G 0 disk
└─sdb1 8:17 0 416G 0 part /ssd
sdc 8:32 0 5.5T 0 disk
└─sdc1 8:33 0 5T 0 part /disk1
sdd 8:48 0 5.5T 0 disk
└─sdd1 8:49 0 5T 0 part /disk2
sde 8:64 0 7.3T 0 disk
└─sde1 8:65 0 6.7T 0 part /disk3

lvdisplay
--- Logical volume ---
LV Path /dev/pve/swap
LV Name swap
VG Name pve
LV UUID 1sk2i9-gInK-3HHR-gOa1-2GHS-r9ds-oSWSCs
LV Write Access read/write
LV Creation host, time proxmox, 2016-11-17 06:33:28 -0600
LV Status available
# open 2
LV Size 8.00 GiB
Current LE 2048
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 251:1

--- Logical volume ---
LV Path /dev/pve/root
LV Name root
VG Name pve
LV UUID 60AVG7-75l8-i25J-poFU-EHMR-fjcz-ACRKGP
LV Write Access read/write
LV Creation host, time proxmox, 2016-11-17 06:33:28 -0600
LV Status available
# open 1
LV Size 96.00 GiB
Current LE 24576
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 251:0

--- Logical volume ---
LV Name data
VG Name pve
LV UUID fTa18Z-FdP6-Ny06-VICk-lvc4-48Sa-aGmstZ
LV Write Access read/write
LV Creation host, time proxmox, 2016-11-17 06:33:30 -0600
LV Pool metadata data_tmeta
LV Pool data data_tdata
LV Status available
# open 2
LV Size 810.88 GiB
Allocated pool data 1.70%
Allocated metadata 1.16%
Current LE 207584
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 251:4

--- Logical volume ---
LV Path /dev/pve/vm-501-disk-1
LV Name vm-501-disk-1
VG Name pve
LV UUID eO4RRN-BHnP-zV8y-g5BI-vlzC-ga6E-udDlf7
LV Write Access read/write
LV Creation host, time proxmox5, 2016-11-29 15:41:55 -0600
LV Pool name data
LV Status available
# open 0
LV Size 65.00 GiB
Mapped size 21.17%
Current LE 16640
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 251:6

/etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext4 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0
/dev/sdb1 /ssd ext4 defaults,errors=remount-ro 0 1
/dev/sdc1 /disk1 ext4 defaults,errors=remount-ro 0 1
/dev/sdd1 /disk2 ext4 defaults,errors=remount-ro 0 1
/dev/sde1 /disk3 ext4 defaults,errors=remount-ro 0 1

 

[wolfgang]

Hi,

I don't know why your disk goes away but I know the cluster never touch the fstab.

And if /dev/pve/data goes away this sounds like a lvm or udev problem.
I would consult the log.

 

[fabian]

pve-data is a thin pool - you cannot mount it (this has nothing to do with joining a cluster or not!). you might not see it in PVE right now because the storage.cfg gets overwritten on the joining node (with the one from the cluster). in that case, you need to add the LVM-Thin storage again (limited to the node where it is actually available!)

 

[term]

Looks like lvm-thin was not added until recently based on the wiki entry. My other nodes are 3.x nodes that have been upgraded to 4.3, and I have confused myself by comparing their configuration files to the new one.

storage.cfg on the old nodes has the local storage set up like this:

dir: local
path /var/lib/vz
content images,iso,backup,rootdir,vztmpl
maxfiles 0

Sounds like the new one needs to be set up like this:

lvmthin: local-lvm
thinpool data
vgname pve
content rootdir,images

Is that correct?

 

[term]

Adding back the local storage worked. Thanks! I updated the wiki page on clusters with a warning for the next person that runs into this.