Hi,
Got two privileged Debian 11.6 standard LXCs where I installed the PBS packages with datastores on NFS shares.
They basically work but there are two problems:
1.) I boot PVE and PVE autostarts the TrueNAS VM first which autostarts the NFS server. 5 Minutes later PVE starts the PBS LXC (NFS is then up and running) and 10 Minutes after that PVE starts the backup jobs. Problem is that all PBS backups will fail with logs like these:
If I then open the TrueNAS webUI and stop+start the NFS service all backups are working. Not sure why getting shared locks isn't working on the first time when TrueNAS autostarts the NFS service. Some one got an idea? I would really like to get this fixed so I can automate my backups without trying some workarounds like using SMB shares or run a script that uses the TrueNAS API to stop+start the NFS service.
The LXC mounts the NFS share using fstab with these options:
2.) Each time I start that PBS LXC it will totally spam the hosts and LXCs syslog until my Graylog LXC can't handle it. Millions of logs in a few hours. Looks like there are several problems because of missing privileges:
zfs-zed:
I disabled that service as the LXC doesn't need to monitor my PVEs pools.
Logind:
Some other services are failing too:
Here is the LXCs config:
Got two privileged Debian 11.6 standard LXCs where I installed the PBS packages with datastores on NFS shares.
They basically work but there are two problems:
1.) I boot PVE and PVE autostarts the TrueNAS VM first which autostarts the NFS server. 5 Minutes later PVE starts the PBS LXC (NFS is then up and running) and 10 Minutes after that PVE starts the backup jobs. Problem is that all PBS backups will fail with logs like these:
Code:
INFO: Starting Backup of VM 111 (qemu)
INFO: Backup started at 2023-05-15 23:11:00
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: Minecraft
INFO: include disk 'scsi0' 'VMpool2_VLT_VM8K:vm-111-disk-0' 96G
INFO: include disk 'scsi1' 'VMpool2_VLT_VM8K:vm-111-disk-1' 2G
INFO: creating Proxmox Backup Server archive 'vm/111/2023-05-15T21:11:00Z'
INFO: starting kvm to execute backup task
ERROR: VM 111 qmp command 'backup' failed - backup register image failed: command error: unable to get shared lock - EIO: I/O error
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 111 failed - VM 111 qmp command 'backup' failed - backup register image failed: command error: unable to get shared lock - EIO: I/O error
INFO: Failed at 2023-05-15 23:11:15The LXC mounts the NFS share using fstab with these options:
defaults,nfsvers=32.) Each time I start that PBS LXC it will totally spam the hosts and LXCs syslog until my Graylog LXC can't handle it. Millions of logs in a few hours. Looks like there are several problems because of missing privileges:
zfs-zed:
Code:
Mar 19 22:39:19 PBS systemd[1]: Started ZFS Event Daemon (zed).
Mar 19 22:39:19 PBS zed[50954]: ZFS Event Daemon 2.1.9-pve1 (PID 50954)
Mar 19 22:39:29 PBS zed[50954]: Failed to initialize libzfs
Mar 19 22:39:29 PBS systemd[1]: zfs-zed.service: Main process exited, code=exited, status=1/FAILURE
Mar 19 22:39:29 PBS systemd[1]: zfs-zed.service: Failed with result 'exit-code'.
Mar 19 22:39:30 PBS systemd[1]: zfs-zed.service: Scheduled restart job, restart counter is at 6370.
Mar 19 22:39:30 PBS systemd[1]: Stopped ZFS Event Daemon (zed).Logind:
Code:
Mar 19 22:35:45 PBS systemd[1]: Condition check resulted in Load Kernel Module drm being skipped.
Mar 19 22:35:45 PBS kernel: [82607.249006] audit: type=1400 audit(1679261745.463:15098): apparmor="DENIED" operation="mount" info="failed perms check" error=-13 profile="lxc-145_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=2406085 comm="(d-logind)" srcname="/" flags="rw, rbind"
Mar 19 22:35:45 PBS systemd[1]: Starting User Login Management...
Mar 19 22:35:45 PBS systemd[50801]: systemd-logind.service: Failed to set up mount namespacing: Permission denied
Mar 19 22:35:45 PBS systemd[50801]: systemd-logind.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-logind: Permission denied
Mar 19 22:35:45 PBS systemd[1]: systemd-logind.service: Main process exited, code=exited, status=226/NAMESPACE
Mar 19 22:35:45 PBS systemd[1]: systemd-logind.service: Failed with result 'exit-code'.
Mar 19 22:35:45 PBS systemd[1]: Failed to start User Login Management.
Mar 19 22:35:45 PBS systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 2.
Mar 19 22:35:45 PBS systemd[1]: Stopped User Login Management.Some other services are failing too:
Code:
May 16 17:56:32 PBS systemd[51]: nftables.service: Failed at step NAMESPACE spawning /usr/sbin/nft: Permission denied
May 16 17:56:32 PBS mount[48]: mount: /run/rpc_pipefs: WARNING: source write-protected, mounted read-only.
May 16 17:56:32 PBS systemd[1]: Starting Flush Journal to Persistent Storage...
May 16 17:56:32 PBS systemd[1]: Finished Create Static Device Nodes in /dev.
May 16 17:56:32 PBS systemd[1]: Condition check resulted in Rule-based Manager for Device Events and Files being skipped.
May 16 17:56:32 PBS systemd[1]: Starting Network Service...
May 16 17:56:32 PBS systemd[65]: systemd-networkd.service: Failed to set up mount namespacing: Permission denied
May 16 17:56:32 PBS systemd[65]: systemd-networkd.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-networkd: Permission denied
May 16 17:56:32 PBS systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=226/NAMESPACE
May 16 17:56:32 PBS systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
May 16 17:56:32 PBS systemd[1]: Failed to start Network Service.
May 16 17:56:32 PBS systemd[1]: Dependency failed for Wait for Network to be Configured.
May 16 17:56:32 PBS systemd[1]: systemd-networkd-wait-online.service: Job systemd-networkd-wait-online.service/start failed with result 'dependency'.
May 16 17:56:32 PBS systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 1.
May 16 17:56:32 PBS systemd[1]: Stopped Network Service.
...
May 16 17:56:32 PBS systemd[1]: Starting Network Service...
May 16 17:56:32 PBS systemd[70]: systemd-networkd.service: Failed to set up mount namespacing: Permission denied
May 16 17:56:32 PBS systemd[70]: systemd-networkd.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-networkd: Permission denied
May 16 17:56:32 PBS systemd[1]: Finished Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
May 16 17:56:32 PBS systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=226/NAMESPACE
May 16 17:56:32 PBS systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
May 16 17:56:32 PBS systemd[1]: Failed to start Network Service.
May 16 17:56:32 PBS systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
May 16 17:56:32 PBS systemd[1]: Reached target Local File Systems (Pre).
May 16 17:56:32 PBS systemd[1]: Stopped Network Service.
May 16 17:56:32 PBS systemd[1]: systemd-networkd.service: Start request repeated too quickly.
May 16 17:56:32 PBS systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
May 16 17:56:32 PBS systemd[1]: Failed to start Network Service.
May 16 17:56:32 PBS systemd[1]: systemd-networkd.socket: Failed with result 'service-start-limit-hit'.
May 16 17:56:32 PBS systemd[1]: Finished Flush Journal to Persistent Storage.
May 16 17:56:32 PBS zfs[57]: /dev/zfs and /proc/self/mounts are required.
May 16 17:56:32 PBS zfs[57]: Try running 'udevadm trigger' and 'mount -t proc proc /proc' as root.
May 16 17:56:32 PBS systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
May 16 17:56:32 PBS systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
May 16 17:56:32 PBS systemd[1]: Failed to start Mount ZFS filesystems.
May 16 17:56:32 PBS systemd[1]: Reached target Local File Systems.
May 16 17:56:32 PBS systemd[1]: Starting Network initialization...
May 16 17:56:32 PBS systemd[1]: Starting Preprocess NFS configuration...
May 16 17:56:32 PBS zvol_wait[60]: /dev/zfs and /proc/self/mounts are required.
May 16 17:56:32 PBS zvol_wait[60]: Try running 'udevadm trigger' and 'mount -t proc proc /proc' as root.
May 16 17:56:32 PBS zvol_wait[58]: No zvols found, nothing to do.
May 16 17:56:32 PBS systemd[1]: Starting Proxmox Backup Server Login Banner...
May 16 17:56:32 PBS systemd[1]: Condition check resulted in Store a System Token in an EFI Variable being skipped.
May 16 17:56:32 PBS systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
May 16 17:56:32 PBS systemd[1]: Starting Create Volatile Files and Directories...
May 16 17:56:32 PBS systemd[1]: Finished Wait for ZFS Volume (zvol) links in /dev.
May 16 17:56:32 PBS systemd[1]: Reached target ZFS volumes are ready.
May 16 17:56:32 PBS systemd[1]: nfs-config.service: Succeeded.
May 16 17:56:32 PBS systemd[1]: Finished Preprocess NFS configuration.
May 16 17:56:32 PBS systemd[1]: Condition check resulted in RPC security service for NFS client and server being skipped.
May 16 17:56:32 PBS systemd[1]: Condition check resulted in RPC security service for NFS server being skipped.
May 16 17:56:32 PBS systemd[1]: Reached target NFS client services.
May 16 17:56:32 PBS systemd[1]: Finished Proxmox Backup Server Login Banner.
May 16 17:56:32 PBS networking[71]: networking: Configuring network interfaces
May 16 17:56:32 PBS systemd[1]: Finished Create Volatile Files and Directories.
May 16 17:56:32 PBS systemd[1]: Starting RPC bind portmap service...
May 16 17:56:32 PBS systemd[1]: Starting Network Name Resolution...
May 16 17:56:32 PBS systemd[1]: Condition check resulted in Network Time Synchronization being skipped.
May 16 17:56:32 PBS systemd[1]: Reached target System Time Set.
May 16 17:56:32 PBS systemd[1]: Reached target System Time Synchronized.
May 16 17:56:32 PBS systemd[1]: Starting Update UTMP about System Boot/Shutdown...
May 16 17:56:32 PBS systemd[84]: systemd-resolved.service: Failed to set up mount namespacing: Permission denied
May 16 17:56:32 PBS systemd[84]: systemd-resolved.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-resolved: Permission denied
May 16 17:56:32 PBS systemd[1]: Started RPC bind portmap service.
May 16 17:56:32 PBS systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=226/NAMESPACE
May 16 17:56:32 PBS systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
May 16 17:56:32 PBS systemd[1]: Failed to start Network Name Resolution.
May 16 17:56:32 PBS systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 1.
May 16 17:56:32 PBS systemd[1]: Reached target Remote File Systems (Pre).
May 16 17:56:32 PBS systemd[1]: Reached target Remote Encrypted Volumes.
May 16 17:56:32 PBS systemd[1]: Reached target RPC Port Mapper.
May 16 17:56:32 PBS systemd[1]: Stopped Network Name Resolution.
May 16 17:56:32 PBS systemd[1]: Starting Network Name Resolution...
May 16 17:56:32 PBS systemd[88]: systemd-resolved.service: Failed to set up mount namespacing: Permission denied
May 16 17:56:32 PBS systemd[88]: systemd-resolved.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-resolved: Permission denied
May 16 17:56:32 PBS systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=226/NAMESPACE
May 16 17:56:32 PBS systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
May 16 17:56:32 PBS systemd[1]: Failed to start Network Name Resolution.
May 16 17:56:32 PBS systemd[1]: Finished Update UTMP about System Boot/Shutdown.
May 16 17:56:32 PBS systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 2.
May 16 17:56:32 PBS systemd[1]: Reached target System Initialization.
May 16 17:56:32 PBS systemd[1]: Started Daily apt download activities.
May 16 17:56:32 PBS systemd[1]: Started Daily apt upgrade and clean activities.
May 16 17:56:32 PBS systemd[1]: Started Periodic ext4 Online Metadata Check for All Filesystems.
May 16 17:56:32 PBS systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped.
May 16 17:56:32 PBS systemd[1]: Started Daily rotation of log files.
May 16 17:56:32 PBS systemd[1]: Started Daily man-db regeneration.
May 16 17:56:32 PBS systemd[1]: Started Daily Proxmox Backup Server update and maintenance activities.
May 16 17:56:32 PBS systemd[1]: Started Daily Cleanup of Temporary Directories.
May 16 17:56:32 PBS systemd[1]: Reached target Timers.
May 16 17:56:32 PBS systemd[1]: Listening on D-Bus System Message Bus Socket.
May 16 17:56:32 PBS systemd[1]: Listening on OpenBSD Secure Shell server socket.
May 16 17:56:32 PBS systemd[1]: Reached target Sockets.
May 16 17:56:32 PBS systemd[1]: Reached target Basic System.Here is the LXCs config:
Code:
cat /etc/pve/lxc/145.conf
arch: amd64
cores: 4
features: mount=nfs;cifs
hostname: PBS
memory: 4096
nameserver: 192.168.43.1
net0: name=eth0,bridge=vmbr43,firewall=1,gw=192.168.43.1,hwaddr=EE:FF:60:55:67:CB,ip=192.168.43.63/24,mtu=1500,type=veth
net1: name=eth1,bridge=vmbr49,firewall=1,hwaddr=32:E2:30:F6:8D:48,ip=192.168.49.8/24,mtu=9000,type=veth
onboot: 1
ostype: debian
rootfs: VMpool2_VLT_LXC128K:subvol-145-disk-0,size=32G
swap: 1024
Last edited: