PBS cifs smb share not mounting on cold boot

[atkuzmanov]

Hi Folks,

Hope everyone is doing well!

Also thanks to anyone willing to help, I really appreciate it!

So, I have been stuck on this for a while and I am banging my head against the wall at this point...

I set up ProxMox Backup Server with a TrueNas SMB share following this tutorial:

https://www.youtube.com/watch?v=84QZc5cnKZc&ab_channel=Jim'sGarage

Everything worked ok, except for the auto mount of the smb share.

So, basically what I did was:
- Installed PBS (ProxMox Backup Server)
- Set up and SMB share on TrueNas Scale with credentials, so that it is accessible by PBS.
- Created a mount point on PBS and added the SMB share as a Datastore to PBS
- Added the PBS Datastore to the ProxMox Datacenter storage as ProxMox Backup Server

It mounts fine if I issue the mount command from the video:

https://github.com/JamesTurland/JimsGarage/tree/main/Proxmox-Backup-Server

mount -t cifs -o rw,vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34 //IP-OF-NAS/SHARE-NAME /mnt/truenas

I added the mount command in `/etc/fstab`:

//IP-OF-NAS/SHARE-NAME /mnt/test-pbs cifs vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34,defaults 0 0

But it never mounts on cold boot from fstab.

However it mounts fine if I restart the ProxMox Backup Server after it has already done a cold boot and failed once.

Therefore the fstab command must be correct?!

So, it looked to me like it could be a timing issue, where TrueNas has not started or become available yet, so I tried to add and order and startup delay of the VMs in ProxMox.
That did not help as well. It just looks like that the "Bulk start VMs and Containers" job is looping for longer before it finishes, but it looks like the actual VM is starting asap.

I also tried different network startup delay techniques below with no success so far.

---

Some screenshots showing current state and the issue:













This is how it mounts fine if I only reboot the PBS:

---

mount.cifs -V
mount.cifs version: 7.0

Errors:

journalctl -xe | grep cifs

Jun 17 10:29:49 nasko1 kernel: Key type cifs.spnego registered
Jun 17 10:29:49 nasko1 kernel: Key type cifs.idmap registered
Jun 17 10:29:56 nasko1 kernel: CIFS: VFS: cifs_mount failed w/return code = -113

journalctl -xe

Jun 17 10:32:51 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:5125>
Jun 17 10:32:51 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:5128>
Jun 17 10:32:52 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:5130>
Jun 17 10:32:54 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:5131>
Jun 17 10:32:57 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:3464>
Jun 17 10:32:58 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/namespace: 400 Bad Request: [client [::ffff:192.168.0.7]:49>
Jun 17 10:32:58 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status/?verbose=true: 400 Bad Request: [client [::ffff:192.>
Jun 17 10:32:58 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/rrd?timeframe=hour&cf=AVERAGE: 400 Bad Request: [client [::>
Jun 17 10:33:00 nasko1 proxmox-backup-proxy[586]: lookup_datastore failed - unable to open chunk store 'tnsproxbkps1' at "/mnt/tnsproxbkps1/.chunks" - No>
Jun 17 10:33:00 nasko1 proxmox-backup-proxy[586]: unable to start datastore prune job default-tnsproxbkps1-3eca9368-3d - unable to open chunk store 'tnsp>
Jun 17 10:33:01 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:3466>
Jun 17 10:33:01 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:3468>
Jun 17 10:33:04 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status/?verbose=true: 400 Bad Request: [client [::ffff:192.>
Jun 17 1

---

---

Things I have tried:

/etc/fstab


Here is what my fstab looks like:

/dev/pbs/root / ext4 errors=remount-ro 0 1
UUID=8327-698B /boot/efi vfat defaults 0 1
/dev/pbs/swap none swap sw 0 0
proc /proc proc defaults 0 0

# Mount cifs TrueNas Scale SMB share:

//192.168.0.43/tns-proxbkps-ds-1 /mnt/tnsproxbkps1 cifs vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34,dir_mode=0755,file_mode=0644 0 0

And here are the variations of the mount command I have tried, these commands and all sorts of combinations and permutations with different options for cifs:

//192.168.0.43/tns-proxbkps-ds-1 /mnt/tnsproxbkps1 cifs vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34,defaults 0 0

//192.168.0.43/tns-proxbkps-ds-1 /mnt/tnsproxbkps1 cifs vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34,dir_mode=0755,file_mode=0644 0 1

//192.168.0.43/tns-proxbkps-ds-1 /mnt/tnsproxbkps1 cifs rw,credentials=/etc/samba/.smbcreds,uid=34,gid=34,forceuid,forcegid,iocharset=utf8,dir_mode=0755,file_mode=0644 0 1


//192.168.0.43/tns-proxbkps-ds-1 /mnt/tnsproxbkps1 cifs _netdev,rw,credentials=/etc/samba/.smbcreds,uid=34,gid=34,noforceuid,noforcegid,iocharset=utf8,dir_mode=0777,file_mode=0777 0 1

I have even tried using _netdev, and it didn't work, based on this:
https://ubuntuforums.org/showthread.php?t=788146

---

---

I have tried the following based on this:
https://www.linuxquestions.org/questions/linux-software-2/cifs-not-mounting-on-boot-731689/

/etc/rc.local

chmod + x /etc/rc.local

#!/bin/bash

while ! ping -c 1 192.168.0.43 ;
do
    sleep 1 ;
done
sudo mount -t cifs -o rw,credentials=/etc/samba/.smbcreds,uid=34,noforceuid,gid=34,noforcegid,iocharset=utf8,dir_mode=0777,file_mode=0777 0 1

---

---

I have tried the following based on this:
https://www.linuxquestions.org/ques...-how-to-where-suggestions-are-welcome-757125/

/etc/network/if-up.d/mountcifs

chmod + x /etc/network/if-up.d/mountcifs

#!/bin/bash

SERVER="192.168.0.43"

until ping $SERVER -W 5 -c 1 -q; do continue; done

mount /home/bkpadmin/backups

exit 0

---

---

I have tried the following based on this:
https://forums.centos.org/viewtopic.php?t=72708


nano /etc/systemd/networkd.conf

[Network]
#SpeedMeter=no
#SpeedMeterIntervalSec=10sec
#ManageForeignRoutingPolicyRules=yes
#ManageForeignRoutes=yes
#RouteTable=

StartupDelaySec=180s

[DHCPv4]
#DUIDType=vendor
#DUIDRawData=

[DHCPv6]
#DUIDType=vendor
#DUIDRawData=


[Link]
LinkDelaySec=90s

---

---

I even tried the following from ChatGPT, but it completely bricked the server!
Luckily I had a snapshot after my initial instal and I was able to roll back.

ChatGPT:

"If the share mounts correctly with the right user when done manually but not automatically at system start, it could be due to timing issues or the order in which services start during the boot process. Here are steps to ensure that the CIFS mount is ready before the Proxmox Backup Server (PBS) tries to access it:

Delay PBS Startup:
Ensure that PBS starts after the CIFS mount is available. You can modify the PBS service file to depend on the CIFS mount.

Create a systemd mount unit for the CIFS share to ensure it is mounted properly before PBS starts.

Create the file `/etc/systemd/system/mnt-tnsproxbkps1.mount`:

[Unit]
Description=Mount TrueNAS datastore
After=network-online.target
Wants=network-online.target

[Mount]
What=//192.168.0.43/tns-proxbkps-ds-1
Where=/mnt/tnsproxbkps1
Type=cifs
Options=vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34,dir_mode=0755,file_mode=0644

[Install]
WantedBy=multi-user.target

Reload the systemd daemon and enable the mount unit:

systemctl daemon-reload
systemctl enable --now mnt-tnsproxbkps1.mount

Update PBS Service File:

Modify the Proxmox Backup Server service file to ensure it waits for the CIFS mount.

Create a drop-in directory for the PBS service:

mkdir -p /etc/systemd/system/proxmox-backup.service.d

Create a drop-in file `/etc/systemd/system/proxmox-backup.service.d/mounts.conf`:

[Unit]
After=mnt-tnsproxbkps1.mount
Requires=mnt-tnsproxbkps1.mount

Reload the systemd daemon:

systemctl daemon-reload

Verify and Test:
Reboot the system to ensure that the changes take effect and verify the order of the services.

reboot

After the system reboots, check the status of the mount and the PBS service:

systemctl status mnt-tnsproxbkps1.mount
systemctl status proxmox-backup

Ensure that the mount is active and owned by the correct user before the PBS service starts.

By ensuring the proper order of service starts and dependencies, you should be able to mount the CIFS share correctly before the PBS service tries to access it, maintaining the correct ownership and permissions."

---

---

I have exhausted all I can think of and I am bagging my head against a wall and any help is greatly valued and appreciated!

 

[atkuzmanov]

Anybody?

Okay, so far I have managed to use a workaround hack based on the comment of this guy Zorael here:

https://ubuntuforums.org/showthread.php?t=788146

Thanks Zorael!

Still open to advice and better/more elegant solutions. ( :

P.S.

These also look similar:

https://www.linuxquestions.org/ques...-how-to-where-suggestions-are-welcome-757125/

https://www.linuxquestions.org/questions/linux-software-2/cifs-not-mounting-on-boot-731689/

Hope this helps someone else as well.

 

[VictorSTS]

If your server breaks, what's your recovery plan? Hope you also have backups outside of that server.

Your backups depend on a PBS VM that depends on a TrueNAS VM, both hosted in the very same server. If you are lucky, if the server breaks you will still have PBS data in some disk(s) and maybe you can mount them somehow. If you are out of luck and disks break with the server (i.e. PSU failure, power spike, motherboard goes crazy) you will lose everything and your backups will be useless.

At the very least, sync PBS backups to a USB drive mapped datastore so you have some chance to recover.

 

[atkuzmanov]

If your server breaks, what's your recovery plan? Hope you also have backups outside of that server.

Your backups depend on a PBS VM that depends on a TrueNAS VM, both hosted in the very same server. If you are lucky, if the server breaks you will still have PBS data in some disk(s) and maybe you can mount them somehow. If you are out of luck and disks break with the server (i.e. PSU failure, power spike, motherboard goes crazy) you will lose everything and your backups will be useless.

At the very least, sync PBS backups to a USB drive mapped datastore so you have some chance to recover.

Thank you very much for your reply!
I am a newbie and what you are saying is a good point and makes sense!
I will see what I can think of and post back.
What would be a a good solution to this - host TrueNas and PBS on separate servers? - I currently have just the one server.