[SOLVED] Proxmox Ceph Cluster 4.4 to 5.1 upgrade - Problem with fstab

D

David Herselman

Renowned Member
Proxmox Subscriber
Jun 8, 2016
344
75
93
48
Johannesburg, South Africa
Firstly many thanks for the excellent work done by Proxmox, the Ceph upgrade to Luminous and subsequent upgrade to Proxmox 5.1 (Debian stretch) has run through perfectly.

The only nuance we have is that pve-cluster doesn't start when we define the following /etc/fstab entry:
Code: 
id=admin,conf=/etc/ceph/ceph.conf /var/lib/vz fuse.ceph defaults,_netdev,noauto,nonempty,x-systemd.requires=ceph.target,x-systemd.automount 0 0

We utilise CephFS to share '/var/lib/vz' between our cluster nodes, I wrote about how to set this up in the following post:
https://forum.proxmox.com/threads/proxmox-ve-5-0-beta2-released.34853/page-4#post-173582

If I take out 'x-systemd.automount' everything starts as expected, after which I can then issue a 'mount /var/lib/vz' and everything works perfectly...

What logging information could I provide to possibly investigate why Proxmox 5.1 has this issue?
 
Last edited:
you should probably write a full systemd mount unit for this, the fstab generator is not very smart regarding some defaults (e.g., your mount is probably added to remote-fs.target which comes earlier than ceph.target)

if you don't want to write your own unit, you can try checking the journal to see if an ordering cycle was broken, and check the generated mount unit ("systemctl show var-lib-vz.mount") for any obvious issues.
 
Thanks for the prompt reply Fabian, I've compared the output of that command on a Proxmox v4.4 and v5.1 system and it would appear systemd is not honoring the '_netdev' parameter.

'diff -uNr' from v5.1 to v4.4:
Code: 
-Requires=ceph.target -.mount system.slice
+Requires=-.mount
+Wants=network-online.target system.slice
 Conflicts=umount.target
-Before=umount.target local-fs.target
-After=ceph.target -.mount system.slice local-fs-pre.target
+Before=umount.target
+After=var-lib-vz.automount systemd-journald.socket remote-fs-pre.target network.target network-online.target system.slice -.mount

My understanding is that fstab is preferred over systemd mount units and I'm overwelmed by what systemd puts in the automatically generated file automatically.

For what it's worth, herewith the full auto generated mount units on:

v4.4:
Code: 
[root@kvm3 ~]# systemctl show var-lib-vz.mount
Where=/var/lib/vz
What=ceph-fuse
Options=rw,relatime,rw,user_id=0,group_id=0,default_permissions,allow_other
Type=fuse.ceph-fuse
TimeoutUSec=1min 30s
ControlPID=0
DirectoryMode=0755
SloppyOptions=no
Result=success
Slice=system.slice
ControlGroup=/system.slice/var-lib-vz.mount
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=(null)
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
UMask=0022
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=4096
LimitAS=18446744073709551615
LimitNPROC=193037
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=193037
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=yes
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=var-lib-vz.mount
Names=var-lib-vz.mount
Requires=-.mount
Wants=network-online.target system.slice
Conflicts=umount.target
Before=umount.target
After=var-lib-vz.automount systemd-journald.socket remote-fs-pre.target network.target network-online.target system.slice -.mount
TriggeredBy=var-lib-vz.automount
RequiresMountsFor=/var/lib
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Description=/var/lib/vz
LoadState=loaded
ActiveState=active
SubState=mounted
FragmentPath=/run/systemd/generator/var-lib-vz.mount
SourcePath=/etc/fstab
InactiveExitTimestamp=Sat 2017-07-01 10:34:35 SAST
InactiveExitTimestampMonotonic=646629848904
ActiveEnterTimestamp=Sat 2017-07-01 10:34:35 SAST
ActiveEnterTimestampMonotonic=646629848904
ActiveExitTimestamp=Sat 2017-07-01 10:29:22 SAST
ActiveExitTimestampMonotonic=646316716249
InactiveEnterTimestamp=Sat 2017-07-01 10:29:22 SAST
InactiveEnterTimestampMonotonic=646316737338
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=yes
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
ConditionResult=yes
ConditionTimestamp=Fri 2017-06-23 22:57:43 SAST
ConditionTimestampMonotonic=17928802
Transient=no

v5.1:
Code: 
[root@kvm2 ~]# systemctl show var-lib-vz.mount
Where=/var/lib/vz
What=ceph-fuse
Options=rw,relatime,user_id=0,group_id=0,allow_other
Type=fuse.ceph-fuse
TimeoutUSec=1min 30s
ControlPID=0
DirectoryMode=0755
SloppyOptions=no
LazyUnmount=no
ForceUnmount=no
Result=success
UID=4294967295
GID=4294967295
Slice=system.slice
ControlGroup=/system.slice/var-lib-vz.mount
MemoryCurrent=18446744073709551615
CPUUsageNSec=18446744073709551615
TasksCurrent=0
Delegate=no
CPUAccounting=no
CPUWeight=18446744073709551615
StartupCPUWeight=18446744073709551615
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
IOAccounting=no
IOWeight=18446744073709551615
StartupIOWeight=18446744073709551615
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLow=0
MemoryHigh=18446744073709551615
MemoryMax=18446744073709551615
MemorySwapMax=18446744073709551615
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=yes
TasksMax=4915
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
LimitFSIZE=18446744073709551615
LimitFSIZESoft=18446744073709551615
LimitDATA=18446744073709551615
LimitDATASoft=18446744073709551615
LimitSTACK=18446744073709551615
LimitSTACKSoft=8388608
LimitCORE=18446744073709551615
LimitCORESoft=0
LimitRSS=18446744073709551615
LimitRSSSoft=18446744073709551615
LimitNOFILE=4096
LimitNOFILESoft=1024
LimitAS=18446744073709551615
LimitASSoft=18446744073709551615
LimitNPROC=192990
LimitNPROCSoft=192990
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=18446744073709551615
LimitLOCKSSoft=18446744073709551615
LimitSIGPENDING=192990
LimitSIGPENDINGSoft=192990
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=18446744073709551615
LimitRTTIMESoft=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=inherit
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
DynamicUser=no
RemoveIPC=no
MountFlags=0
PrivateTmp=no
PrivateDevices=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=yes
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictNamespace=2114060288
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=var-lib-vz.mount
Names=var-lib-vz.mount
Requires=ceph.target -.mount system.slice
Conflicts=umount.target
Before=umount.target local-fs.target
After=ceph.target -.mount system.slice local-fs-pre.target
RequiresMountsFor=/var/lib
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Description=/var/lib/vz
LoadState=loaded
ActiveState=active
SubState=mounted
FragmentPath=/run/systemd/generator/var-lib-vz.mount
SourcePath=/etc/fstab
UnitFileState=generated
UnitFilePreset=enabled
StateChangeTimestamp=Wed 2017-11-15 15:34:38 SAST
StateChangeTimestampMonotonic=40594679
InactiveExitTimestamp=Wed 2017-11-15 15:34:38 SAST
InactiveExitTimestampMonotonic=40591079
ActiveEnterTimestamp=Wed 2017-11-15 15:34:38 SAST
ActiveEnterTimestampMonotonic=40591079
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=yes
NeedDaemonReload=yes
JobTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=no
AssertResult=no
ConditionTimestampMonotonic=0
AssertTimestampMonotonic=0
Transient=no
Perpetual=no
StartLimitIntervalSec=10000000
StartLimitBurst=5
StartLimitAction=none
InvocationID=6a0deaf6cfd549c8841b6aa1cb2e21ef
 
Just an update that adding 'x-systemd.after=network-online.target' unfortunately doesn't help.

Herewith the diff from v5.1 to v4.4:
Code: 
 Names=var-lib-vz.mount
-Requires=-.mount system.slice
-Wants=network-online.target
+Requires=-.mount
+Wants=network-online.target system.slice
 Conflicts=umount.target
 Before=umount.target
-After=var-lib-vz.automount -.mount system.slice remote-fs-pre.target network.target network-online.target
+After=var-lib-vz.automount systemd-journald.socket remote-fs-pre.target network.target network-online.target system.slice -.mount
 TriggeredBy=var-lib-vz.automount

Additional information (not very useful):
Code: 
[root@kvm2 ~]# systemctl status var-lib-vz.mount
● var-lib-vz.mount - /var/lib/vz
   Loaded: loaded (/etc/fstab; generated; vendor preset: enabled)
   Active: inactive (dead)
    Where: /var/lib/vz
     What: id=admin,conf=/etc/ceph/ceph.conf
     Docs: man:fstab(5)
           man:systemd-fstab-generator(8)
[root@kvm2 ~]# journalctl -u var-lib-vz.mount
-- No entries --

And again the full systems mount unit:
Code: 
[root@kvm2 ~]# systemctl show var-lib-vz.mount
Where=/var/lib/vz
What=id=admin,conf=/etc/ceph/ceph.conf
Options=defaults,_netdev,noauto,nonempty,x-systemd.after=network-online.target,x-systemd.requires=ceph.target,x-systemd.automount
Type=fuse.ceph
TimeoutUSec=1min 30s
ControlPID=0
DirectoryMode=0755
SloppyOptions=no
LazyUnmount=no
ForceUnmount=no
Result=success
UID=4294967295
GID=4294967295
Slice=system.slice
MemoryCurrent=18446744073709551615
CPUUsageNSec=18446744073709551615
TasksCurrent=18446744073709551615
Delegate=no
CPUAccounting=no
CPUWeight=18446744073709551615
StartupCPUWeight=18446744073709551615
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
IOAccounting=no
IOWeight=18446744073709551615
StartupIOWeight=18446744073709551615
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLow=0
MemoryHigh=18446744073709551615
MemoryMax=18446744073709551615
MemorySwapMax=18446744073709551615
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=yes
TasksMax=4915
UMask=0022
LimitCPU=18446744073709551615
LimitCPUSoft=18446744073709551615
LimitFSIZE=18446744073709551615
LimitFSIZESoft=18446744073709551615
LimitDATA=18446744073709551615
LimitDATASoft=18446744073709551615
LimitSTACK=18446744073709551615
LimitSTACKSoft=8388608
LimitCORE=18446744073709551615
LimitCORESoft=0
LimitRSS=18446744073709551615
LimitRSSSoft=18446744073709551615
LimitNOFILE=4096
LimitNOFILESoft=1024
LimitAS=18446744073709551615
LimitASSoft=18446744073709551615
LimitNPROC=192990
LimitNPROCSoft=192990
LimitMEMLOCK=65536
LimitMEMLOCKSoft=65536
LimitLOCKS=18446744073709551615
LimitLOCKSSoft=18446744073709551615
LimitSIGPENDING=192990
LimitSIGPENDINGSoft=192990
LimitMSGQUEUE=819200
LimitMSGQUEUESoft=819200
LimitNICE=0
LimitNICESoft=0
LimitRTPRIO=0
LimitRTPRIOSoft=0
LimitRTTIME=18446744073709551615
LimitRTTIMESoft=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=inherit
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SyslogLevel=6
SyslogFacility=3
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
DynamicUser=no
RemoveIPC=no
MountFlags=0
PrivateTmp=no
PrivateDevices=no
ProtectKernelTunables=no
ProtectKernelModules=no
ProtectControlGroups=no
PrivateNetwork=no
PrivateUsers=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=yes
UtmpMode=init
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
MemoryDenyWriteExecute=no
RestrictRealtime=no
RestrictNamespace=2114060288
KillMode=control-group
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=var-lib-vz.mount
Names=var-lib-vz.mount
Requires=-.mount system.slice
Wants=network-online.target
Conflicts=umount.target
Before=umount.target
After=var-lib-vz.automount -.mount system.slice remote-fs-pre.target network.target network-online.target
TriggeredBy=var-lib-vz.automount
RequiresMountsFor=/var/lib
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Description=/var/lib/vz
LoadState=loaded
ActiveState=inactive
SubState=dead
FragmentPath=/run/systemd/generator/var-lib-vz.mount
SourcePath=/etc/fstab
UnitFileState=generated
UnitFilePreset=enabled
StateChangeTimestampMonotonic=0
InactiveExitTimestampMonotonic=0
ActiveEnterTimestampMonotonic=0
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=yes
NeedDaemonReload=no
JobTimeoutUSec=infinity
JobTimeoutAction=none
ConditionResult=no
AssertResult=no
ConditionTimestampMonotonic=0
AssertTimestampMonotonic=0
Transient=no
Perpetual=no
StartLimitIntervalSec=10000000
StartLimitBurst=5
StartLimitAction=none
 
x-systemd.after is not supported in Debian Stretch's systemd (it's a very recent introduction to the fstab generator). I know that at least noauto is broken/ignored for CIFS. you can take the existing units as starting points to create your own - it's not that hard and the man pages (systemd.mount and systemd.unit) are pretty extensive.
 
Thanks Fabian, I added 'x-systemd.requires=network-online.target' which results in the following fstab entry:
Code: 
id=admin,conf=/etc/ceph/ceph.conf /var/lib/vz fuse.ceph defaults,_netdev,noauto,nonempty,x-systemd.requires=network-online.target,x-systemd.requires=ceph.target,x-systemd.automount 0 0

The resulting differences between v4.4 and v5.1 are now minimal (this is a diff from v5.1 to v4.4):
Code: 
 Names=var-lib-vz.mount
-Requires=system.slice -.mount
-Wants=network-online.target
+Requires=-.mount
+Wants=network-online.target system.slice
 Conflicts=umount.target
 Before=umount.target
-After=var-lib-vz.automount remote-fs-pre.target network-online.target network.target -.mount system.slice
+After=var-lib-vz.automount systemd-journald.socket remote-fs-pre.target network.target network-online.target system.slice -.mount
 TriggeredBy=var-lib-vz.automount

The mount point however doesn't automatically mount but it works perfectly when I run 'mount /var/lib/vz' manually.

Would anyone know if there is a x-systemd.requires target such as 'after everything else'? I presume the issue may be that Ceph reports itself as up but isn't able to serve CephFS until a couple of seconds afterwards...
 
but you don't need an automount unit, a regular mount should be okay? don't forget to set 'is_mountpoint' on the 'local' storage to prevent PVE from creating directories before it has been mounted.
 
Thanks for the tip Fabian, didn't know that option was available. Perhaps you could get it added to the documentation (https://pve.proxmox.com/wiki/Storage)?

If it helps others:
I puzzled around trying to solve this with a systemd mount unit but it would simply not automatically mount it. I also didn't like having a mount unit hidden away where it wouldn't easily be found by other staff or system administrators.

CephFS simply appears to require a little time after initializing, before it's ready to mount a volume...

My final 'hack':

/etc/fstab:
Code: 
id=admin,conf=/etc/ceph/ceph.conf /var/lib/vz fuse.ceph defaults,_netdev,noauto,nonempty,x-systemd
.requires=ceph.target 0 0

/etc/rc.local:
Code: 
sleep 60 && mount /var/lib/vz;
exit 0

/etc/pve/storage.cfg:
Code: 
dir: cephfs
        path /var/lib/vz
        maxfiles 0
        shared
        content backup,iso,vztmpl

NB: Ceph FUSE is slow, but absolutely perfect for sharing ISO images. Much better than landing up with different collections on your different servers...
 
You must log in or register to reply here.
Top Bottom