Basically what the title says. I have a few LXC containers that have filesystem mount points pointing to CephFS filesystems to sync containers across hosts. Sadly, they always fail to start on boot (I assume cause CephFS is just slower to start than they are, they have no problem being started manually after boot).
LXC Containers with CephFS Mountpoints Fail to Start at Boot
- Thread starter BlueMatt
- Start date
Try to give `startup delay` you can do that in the Web UI -> LXC -> Options -> Start/Shutdown order.
To investigate the issue further, could you please check the Syslog during the LXC startup? The Syslog from the PVE host should give us more information and help us understand the root cause.
It seems the mount was just too late.
The host came up at 00:34:34:
The container in question was the first thing it tried to start at 00:34:37:
But /mnt/pve mounts don't start getting processed until 00:34:45:
...and the mount we actually need isn't processed until 00:34:48:
The host came up at 00:34:34:
Nov 12 00:34:34 rackbeast corosync[7218]: [QUORUM] This node is within the primary component and will provide service.Nov 12 00:34:34 rackbeast corosync[7218]: [QUORUM] Members[4]: 1 2 3 4Nov 12 00:34:34 rackbeast corosync[7218]: [MAIN ] Completed service synchronization, ready to provide service.The container in question was the first thing it tried to start at 00:34:37:
Nov 12 00:34:37 rackbeast pve-guests[7642]: <root@pam> starting task UPID:rackbeast:00001DDB:00000E71:6732A29D:startall::root@pam:Nov 12 00:34:37 rackbeast pvesh[7642]: Starting CT 1013Nov 12 00:34:37 rackbeast pve-guests[7643]: <root@pam> starting task UPID:rackbeast:00001DDC:00000E73:6732A29D:vzstart:1013:root@pam:Nov 12 00:34:37 rackbeast pve-guests[7644]: starting CT 1013: UPID:rackbeast:00001DDC:00000E73:6732A29D:vzstart:1013:root@pam:Nov 12 00:34:41 rackbeast sh[7212]: Running command: /usr/sbin/ceph-volume lvm trigger 1-4cea8dc9-2d9a-43a2-91f8-c302faa33029Nov 12 00:34:41 rackbeast sh[7220]: Running command: /usr/sbin/ceph-volume lvm trigger 7-4a0453c7-2776-4c38-84ac-0ade91bc2658Nov 12 00:34:41 rackbeast sh[7206]: Running command: /usr/sbin/ceph-volume lvm trigger 0-2a9b8cf2-b802-4dda-b032-fe892191e517Nov 12 00:34:43 rackbeast ceph-mds[7182]: starting mds.rackbeast-2 atNov 12 00:34:43 rackbeast ceph-mds[7199]: starting mds.rackbeast atNov 12 00:34:43 rackbeast pmxcfs[6923]: [status] notice: received logNov 12 00:34:43 rackbeast kernel: netfs: FS-Cache loadedNov 12 00:34:43 rackbeast kernel: Key type cifs.spnego registeredNov 12 00:34:43 rackbeast kernel: Key type cifs.idmap registeredNov 12 00:34:43 rackbeast kernel: CIFS: Attempting to mount //69.59.18.197/mntNov 12 00:34:44 rackbeast kernel: Key type ceph registeredNov 12 00:34:44 rackbeast kernel: libceph: loaded (mon/osd proto 15/24)Nov 12 00:34:44 rackbeast kernel: rbd: loaded (major 251)Nov 12 00:34:44 rackbeast kernel: libceph: mon0 (1)69.59.18.247:6789 session establishedNov 12 00:34:44 rackbeast kernel: libceph: client58221316 fsid 0481f941-d8e3-4f47-8a8c-7c531ae0a614Nov 12 00:34:44 rackbeast kernel: rbd: rbd0: capacity 34359738368 features 0x3dNov 12 00:34:44 rackbeast systemd[1]: Created slice system-pve\x2dcontainer.slice - PVE LXC Container Slice.Nov 12 00:34:44 rackbeast systemd[1]: Started pve-container@1013.service - PVE LXC Container: 1013.Nov 12 00:34:44 rackbeast kernel: EXT4-fs (rbd0): mounted filesystem 2edfd69f-f74e-4a9d-bfdf-607d865a4259 r/w with ordered data mode. Quota mode: none.Nov 12 00:34:44 rackbeast pve-guests[7644]: startup for container '1013' failedNov 12 00:34:45 rackbeast kernel: EXT4-fs (rbd0): unmounting filesystem 2edfd69f-f74e-4a9d-bfdf-607d865a4259.Nov 12 00:34:45 rackbeast pvesh[7642]: Starting CT 1013 failed: startup for container '1013' failedBut /mnt/pve mounts don't start getting processed until 00:34:45:
Nov 12 00:34:45 rackbeast systemd[1]: Mounting mnt-pve-ceph_backups.mount - /mnt/pve/ceph_backups......and the mount we actually need isn't processed until 00:34:48:
Nov 12 00:34:48 rackbeast systemd[1]: Mounted mnt-pve-ceph_backups.mount - /mnt/pve/ceph_backups.Nov 12 00:34:48 rackbeast systemd[1]: Mounting mnt-pve-cephfs.mount - /mnt/pve/cephfs...Nov 12 00:34:49 rackbeast systemd[1]: Mounted mnt-pve-cephfs.mount - /mnt/pve/cephfs.