PBS cifs smb share not mounting on cold boot

[atkuzmanov]

Hi Folks,

Hope everyone is doing well!

Also thanks to anyone willing to help, I really appreciate it!

So, I have been stuck on this for a while and I am banging my head against the wall at this point...

I set up ProxMox Backup Server with a TrueNas SMB share following this tutorial:

https://www.youtube.com/watch?v=84QZc5cnKZc&ab_channel=Jim'sGarage

Everything worked ok, except for the auto mount of the smb share.

So, basically what I did was:
- Installed PBS (ProxMox Backup Server)
- Set up and SMB share on TrueNas Scale with credentials, so that it is accessible by PBS.
- Created a mount point on PBS and added the SMB share as a Datastore to PBS
- Added the PBS Datastore to the ProxMox Datacenter storage as ProxMox Backup Server

It mounts fine if I issue the mount command from the video:

https://github.com/JamesTurland/JimsGarage/tree/main/Proxmox-Backup-Server

mount -t cifs -o rw,vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34 //IP-OF-NAS/SHARE-NAME /mnt/truenas

I added the mount command in `/etc/fstab`:

//IP-OF-NAS/SHARE-NAME /mnt/test-pbs cifs vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34,defaults 0 0

But it never mounts on cold boot from fstab.

However it mounts fine if I restart the ProxMox Backup Server after it has already done a cold boot and failed once.

Therefore the fstab command must be correct?!

So, it looked to me like it could be a timing issue, where TrueNas has not started or become available yet, so I tried to add and order and startup delay of the VMs in ProxMox.
That did not help as well. It just looks like that the "Bulk start VMs and Containers" job is looping for longer before it finishes, but it looks like the actual VM is starting asap.

I also tried different network startup delay techniques below with no success so far.

---

Some screenshots showing current state and the issue:













This is how it mounts fine if I only reboot the PBS:

---

mount.cifs -V
mount.cifs version: 7.0

Errors:

journalctl -xe | grep cifs

Jun 17 10:29:49 nasko1 kernel: Key type cifs.spnego registered
Jun 17 10:29:49 nasko1 kernel: Key type cifs.idmap registered
Jun 17 10:29:56 nasko1 kernel: CIFS: VFS: cifs_mount failed w/return code = -113

journalctl -xe

Jun 17 10:32:51 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:5125>
Jun 17 10:32:51 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:5128>
Jun 17 10:32:52 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:5130>
Jun 17 10:32:54 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:5131>
Jun 17 10:32:57 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:3464>
Jun 17 10:32:58 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/namespace: 400 Bad Request: [client [::ffff:192.168.0.7]:49>
Jun 17 10:32:58 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status/?verbose=true: 400 Bad Request: [client [::ffff:192.>
Jun 17 10:32:58 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/rrd?timeframe=hour&cf=AVERAGE: 400 Bad Request: [client [::>
Jun 17 10:33:00 nasko1 proxmox-backup-proxy[586]: lookup_datastore failed - unable to open chunk store 'tnsproxbkps1' at "/mnt/tnsproxbkps1/.chunks" - No>
Jun 17 10:33:00 nasko1 proxmox-backup-proxy[586]: unable to start datastore prune job default-tnsproxbkps1-3eca9368-3d - unable to open chunk store 'tnsp>
Jun 17 10:33:01 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:3466>
Jun 17 10:33:01 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status: 400 Bad Request: [client [::ffff:192.168.0.42]:3468>
Jun 17 10:33:04 nasko1 proxmox-backup-proxy[586]: GET /api2/json/admin/datastore/tnsproxbkps1/status/?verbose=true: 400 Bad Request: [client [::ffff:192.>
Jun 17 1

---

---

Things I have tried:

/etc/fstab


Here is what my fstab looks like:

/dev/pbs/root / ext4 errors=remount-ro 0 1
UUID=8327-698B /boot/efi vfat defaults 0 1
/dev/pbs/swap none swap sw 0 0
proc /proc proc defaults 0 0

# Mount cifs TrueNas Scale SMB share:

//192.168.0.43/tns-proxbkps-ds-1 /mnt/tnsproxbkps1 cifs vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34,dir_mode=0755,file_mode=0644 0 0

And here are the variations of the mount command I have tried, these commands and all sorts of combinations and permutations with different options for cifs:

//192.168.0.43/tns-proxbkps-ds-1 /mnt/tnsproxbkps1 cifs vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34,defaults 0 0

//192.168.0.43/tns-proxbkps-ds-1 /mnt/tnsproxbkps1 cifs vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34,dir_mode=0755,file_mode=0644 0 1

//192.168.0.43/tns-proxbkps-ds-1 /mnt/tnsproxbkps1 cifs rw,credentials=/etc/samba/.smbcreds,uid=34,gid=34,forceuid,forcegid,iocharset=utf8,dir_mode=0755,file_mode=0644 0 1


//192.168.0.43/tns-proxbkps-ds-1 /mnt/tnsproxbkps1 cifs _netdev,rw,credentials=/etc/samba/.smbcreds,uid=34,gid=34,noforceuid,noforcegid,iocharset=utf8,dir_mode=0777,file_mode=0777 0 1

I have even tried using _netdev, and it didn't work, based on this:
https://ubuntuforums.org/showthread.php?t=788146

---

---

I have tried the following based on this:
https://www.linuxquestions.org/questions/linux-software-2/cifs-not-mounting-on-boot-731689/

/etc/rc.local

chmod + x /etc/rc.local

#!/bin/bash

while ! ping -c 1 192.168.0.43 ;
do
    sleep 1 ;
done
sudo mount -t cifs -o rw,credentials=/etc/samba/.smbcreds,uid=34,noforceuid,gid=34,noforcegid,iocharset=utf8,dir_mode=0777,file_mode=0777 0 1

---

---

I have tried the following based on this:
https://www.linuxquestions.org/ques...-how-to-where-suggestions-are-welcome-757125/

/etc/network/if-up.d/mountcifs

chmod + x /etc/network/if-up.d/mountcifs

#!/bin/bash

SERVER="192.168.0.43"

until ping $SERVER -W 5 -c 1 -q; do continue; done

mount /home/bkpadmin/backups

exit 0

---

---

I have tried the following based on this:
https://forums.centos.org/viewtopic.php?t=72708


nano /etc/systemd/networkd.conf

[Network]
#SpeedMeter=no
#SpeedMeterIntervalSec=10sec
#ManageForeignRoutingPolicyRules=yes
#ManageForeignRoutes=yes
#RouteTable=

StartupDelaySec=180s

[DHCPv4]
#DUIDType=vendor
#DUIDRawData=

[DHCPv6]
#DUIDType=vendor
#DUIDRawData=


[Link]
LinkDelaySec=90s

---

---

I even tried the following from ChatGPT, but it completely bricked the server!
Luckily I had a snapshot after my initial instal and I was able to roll back.

ChatGPT:

"If the share mounts correctly with the right user when done manually but not automatically at system start, it could be due to timing issues or the order in which services start during the boot process. Here are steps to ensure that the CIFS mount is ready before the Proxmox Backup Server (PBS) tries to access it:

Delay PBS Startup:
Ensure that PBS starts after the CIFS mount is available. You can modify the PBS service file to depend on the CIFS mount.

Create a systemd mount unit for the CIFS share to ensure it is mounted properly before PBS starts.

Create the file `/etc/systemd/system/mnt-tnsproxbkps1.mount`:

[Unit]
Description=Mount TrueNAS datastore
After=network-online.target
Wants=network-online.target

[Mount]
What=//192.168.0.43/tns-proxbkps-ds-1
Where=/mnt/tnsproxbkps1
Type=cifs
Options=vers=3.0,credentials=/etc/samba/.smbcreds,uid=34,gid=34,dir_mode=0755,file_mode=0644

[Install]
WantedBy=multi-user.target

Reload the systemd daemon and enable the mount unit:

systemctl daemon-reload
systemctl enable --now mnt-tnsproxbkps1.mount

Update PBS Service File:

Modify the Proxmox Backup Server service file to ensure it waits for the CIFS mount.

Create a drop-in directory for the PBS service:

mkdir -p /etc/systemd/system/proxmox-backup.service.d

Create a drop-in file `/etc/systemd/system/proxmox-backup.service.d/mounts.conf`:

[Unit]
After=mnt-tnsproxbkps1.mount
Requires=mnt-tnsproxbkps1.mount

Reload the systemd daemon:

systemctl daemon-reload

Verify and Test:
Reboot the system to ensure that the changes take effect and verify the order of the services.

reboot

After the system reboots, check the status of the mount and the PBS service:

systemctl status mnt-tnsproxbkps1.mount
systemctl status proxmox-backup

Ensure that the mount is active and owned by the correct user before the PBS service starts.

By ensuring the proper order of service starts and dependencies, you should be able to mount the CIFS share correctly before the PBS service tries to access it, maintaining the correct ownership and permissions."

---

---

I have exhausted all I can think of and I am bagging my head against a wall and any help is greatly valued and appreciated!