root disk filling up

what does 'df -h' 'mount' and 'proxmox-backup-manager datastore list' show?
 
Bash:
root@pbs-offsite-gcp:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           799M   81M  718M  11% /run
/dev/sda1       148G   78G   64G  55% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda15      124M  5.4M  119M   5% /boot/efi
backup-pbs-gcp   11T  8.7T  1.7T  84% /backup-pbs-gcp
tmpfs           799M     0  799M   0% /run/user/1000

Bash:
root@pbs-offsite-gcp:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=4075580k,nr_inodes=1018895,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=817204k,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,discard,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=44,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11018)
/dev/sda15 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
backup-pbs-gcp on /backup-pbs-gcp type zfs (rw,xattr,noacl)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=817200k,mode=700,uid=1000,gid=1001)

Bash:
root@pbs-offsite-gcp:~# proxmox-backup-manager datastore list
│ backup-office-offline │ /backup-pbs-gcp/backup-office-offline │         │
│ backup-pbs-gcp        │ /backup-pbs-gcp                       │         │
│ pd-shares-production  │ /backup-pbs-gcp/pd-shares-production  │         │
│ pd-shares-temp        │ /backup-pbs-gcp/pd-shares-temp        │         │
 
Last edited by a moderator:
Hi,

something which caught my eye but not directly related to your issue: You use "nested datastores", i.e., the path of backup-pbs-gcp contains the other three datastore backing paths, that is something which I'd try to avoid, and you may want to move the parent one (needs a bit of manual work).

Now regarding the root filesystem issue. The mounts and paths seem all good, can you check which folders contain a lot of data on the root partition:
Bash:
du -hxd3 /|sort -h
Just FYI: That command is directed to report human units (-h flag) stay on the root filesystem (-x flag), and while it checks all it only reports until a depth of 3 directory levels (-d3 flag) and then sort by size.
 
Hi

On my main PBS server all datastores have their own zfs filesystem. so you mean that datastore : "│ backup-pbs-gcp │ /backup-pbs-gcp " is a potential problem? - and if so how can I move the datastore?

*****
Offsite PBS server:
*****

root@pbs-offsite-gcp:~# ls -lah /backup-pbs-gcp/
total 8.2M
drwxr-xr-x 8 backup backup 10 Feb 15 03:03 .
drwxr-xr-x 19 root root 4.0K Jan 22 11:16 ..
drwxr-x--- 65538 backup backup 65K Oct 27 13:05 .chunks
-rw-r--r-- 1 backup backup 345 Feb 15 03:03 .gc-status
-rw-r--r-- 1 backup backup 0 Oct 27 13:05 .lock
drwxr-xr-x 4 backup backup 6 Dec 27 01:11 backup-office-offline
drwxr-xr-x 4 backup backup 4 Oct 30 20:20 ct
drwxr-xr-x 4 backup backup 6 Feb 15 00:32 pd-shares-production
drwxr-xr-x 4 backup backup 6 Feb 14 16:09 pd-shares-temp
drwxr-xr-x 11 backup backup 11 Oct 31 17:55 vm


*****
Code:
root@pbs-offsite-gcp:~# du -hxd3 /|sort -h
4.0K    /etc/apparmor.d/disable
4.0K    /etc/apparmor.d/force-complain
4.0K    /etc/bash_completion.d
4.0K    /etc/binfmt.d
4.0K    /etc/ca-certificates/update.d
4.0K    /etc/dbus-1/session.d
4.0K    /etc/dbus-1/system.d
4.0K    /etc/dhcp/dhclient-enter-hooks.d
4.0K    /etc/dpkg/dpkg.cfg.d
4.0K    /etc/gss/mech.d
4.0K    /etc/initramfs-tools/conf.d
4.0K    /etc/initramfs-tools/hooks
4.0K    /etc/kernel/install.d
4.0K    /etc/modules-load.d
4.0K    /etc/network/if-pre-up.d
4.0K    /etc/network/interfaces.d
4.0K    /etc/opt
4.0K    /etc/osconfig
4.0K    /etc/perl/CPAN
4.0K    /etc/postfix/dynamicmaps.cf.d
4.0K    /etc/postfix/postfix-files.d
4.0K    /etc/postfix/sasl
4.0K    /etc/profile.d
4.0K    /etc/rc0.d
4.0K    /etc/rc1.d
4.0K    /etc/rc2.d
4.0K    /etc/rc3.d
4.0K    /etc/rc4.d
4.0K    /etc/rc5.d
4.0K    /etc/rc6.d
4.0K    /etc/rcS.d
4.0K    /etc/security/limits.d
4.0K    /etc/security/namespace.d
4.0K    /etc/smartmontools/smartd_warning.d
4.0K    /etc/systemd/network
4.0K    /etc/systemd/user
4.0K    /etc/tmpfiles.d
4.0K    /etc/udev/hwdb.d
4.0K    /etc/xdg/systemd
4.0K    /etc/zfs/zpool.d
4.0K    /media
4.0K    /mnt
4.0K    /opt
4.0K    /srv
4.0K    /tmp/.ICE-unix
4.0K    /tmp/.Test-unix
4.0K    /tmp/.X11-unix
4.0K    /tmp/.XIM-unix
4.0K    /tmp/.font-unix
4.0K    /tmp/systemd-private-b3bb7668f39a44b882d621a39dc2d9d4-chrony.service-HZnbuu/tmp
4.0K    /tmp/systemd-private-b3bb7668f39a44b882d621a39dc2d9d4-haveged.service-jcYQNH/tmp
4.0K    /usr/games
4.0K    /usr/lib/bfd-plugins
4.0K    /usr/lib/binfmt.d
4.0K    /usr/lib/compat-ld
4.0K    /usr/lib/environment.d
4.0K    /usr/lib/gnupg2
4.0K    /usr/lib/gold-ld
4.0K    /usr/lib/sasl2
4.0K    /usr/lib32
4.0K    /usr/lib64
4.0K    /usr/libx32
4.0K    /usr/local/bin
4.0K    /usr/local/etc
4.0K    /usr/local/games
4.0K    /usr/local/include
4.0K    /usr/local/sbin
4.0K    /usr/local/src
4.0K    /usr/share/dict
4.0K    /usr/share/misc
4.0K    /usr/share/terminfo
4.0K    /var/cache/private
4.0K    /var/lib/dbus
4.0K    /var/lib/man-db
4.0K    /var/lib/misc
4.0K    /var/lib/os-prober
4.0K    /var/lib/private
4.0K    /var/lib/python
4.0K    /var/local
4.0K    /var/log/chrony
4.0K    /var/log/private
4.0K    /var/opt
4.0K    /var/spool/rsyslog
8.0K    /etc/NetworkManager/dispatcher.d
8.0K    /etc/X11/Xsession.d
8.0K    /etc/alternatives
8.0K    /etc/apparmor
8.0K    /etc/apparmor.d/local
8.0K    /etc/apt/auth.conf.d
8.0K    /etc/apt/preferences.d
8.0K    /etc/ca-certificates
8.0K    /etc/calendar
8.0K    /etc/cron.hourly
8.0K    /etc/cron.monthly
8.0K    /etc/dpkg/origins
8.0K    /etc/gss
8.0K    /etc/insserv.conf.d
8.0K    /etc/iproute2/rt_protos.d
8.0K    /etc/iproute2/rt_tables.d
8.0K    /etc/kernel/header_postinst.d
8.0K    /etc/kernel/prerm.d
8.0K    /etc/ldap
8.0K    /etc/network/if-down.d
8.0K    /etc/network/if-post-down.d
8.0K    /etc/perl/Net
8.0K    /etc/pm/sleep.d
8.0K    /etc/python3
8.0K    /etc/python3.7
8.0K    /etc/resolvconf/update-libc.d
8.0K    /etc/selinux
8.0K    /etc/smartmontools/run.d
8.0K    /etc/ssl/private
8.0K    /etc/terminfo
8.0K    /etc/udev/rules.d
8.0K    /etc/update-motd.d
8.0K    /etc/xdg
8.0K    /home/felixn/.gnupg
8.0K    /home/felixn/.ssh
8.0K    /root/.local/share
8.0K    /tmp/systemd-private-b3bb7668f39a44b882d621a39dc2d9d4-chrony.service-HZnbuu
8.0K    /tmp/systemd-private-b3bb7668f39a44b882d621a39dc2d9d4-haveged.service-jcYQNH
8.0K    /usr/include/iproute2
8.0K    /usr/include/netash
8.0K    /usr/include/netatalk
8.0K    /usr/include/neteconet
8.0K    /usr/include/netipx
8.0K    /usr/include/netiucv
8.0K    /usr/include/netpacket
8.0K    /usr/include/netrom
8.0K    /usr/include/netrose
8.0K    /usr/include/nfs
8.0K    /usr/lib/modules-load.d
8.0K    /usr/lib/python2.7
8.0K    /usr/lib/rsyslog
8.0K    /usr/share/adduser
8.0K    /usr/share/apparmor-features
8.0K    /usr/share/binfmts
8.0K    /usr/share/debianutils
8.0K    /usr/share/file
8.0K    /usr/share/glib-2.0
8.0K    /usr/share/google-guest-agent
8.0K    /usr/share/java
8.0K    /usr/share/libc-bin
8.0K    /usr/share/readline
8.0K    /usr/share/ssl-cert
8.0K    /var/lib/chrony
8.0K    /var/lib/google
8.0K    /var/lib/grub
8.0K    /var/lib/logrotate
8.0K    /var/lib/postfix
8.0K    /var/lib/sudo
8.0K    /var/lib/vim
8.0K    /var/spool/cron
8.0K    /var/tmp/systemd-private-b3bb7668f39a44b882d621a39dc2d9d4-chrony.service-DDKTdq
8.0K    /var/tmp/systemd-private-b3bb7668f39a44b882d621a39dc2d9d4-haveged.service-PPAuKy
12K    /etc/NetworkManager
12K    /etc/X11
12K    /etc/chrony
12K    /etc/cron.d
12K    /etc/cron.weekly
12K    /etc/dbus-1
12K    /etc/groff
12K    /etc/kernel/postrm.d
12K    /etc/network/if-up.d
12K    /etc/pm
12K    /etc/ppp/ip-down.d
12K    /etc/ppp/ip-up.d
12K    /etc/resolvconf
12K    /etc/rsyslog.d
12K    /etc/ufw/applications.d
12K    /etc/vim
12K    /root/.local
12K    /usr/include/misc
12K    /usr/include/netax25
12K    /usr/lib/chrony
12K    /usr/lib/networkd-dispatcher
12K    /usr/local/lib
12K    /usr/local/share
12K    /usr/share/applications
12K    /usr/share/base-passwd
12K    /usr/share/build-essential
12K    /usr/share/chrony
12K    /usr/share/distro-info
12K    /usr/share/iptables
12K    /usr/share/man-db
12K    /usr/share/openssh
12K    /usr/share/os-prober
12K    /usr/share/pkgconfig
12K    /usr/share/sensible-utils
16K    /etc/default/grub.d
16K    /etc/dhcp/dhclient-exit-hooks.d
16K    /etc/ld.so.conf.d
16K    /etc/logcheck/ignore.d.server
16K    /etc/modprobe.d
16K    /etc/perl
16K    /etc/skel
16K    /etc/smartmontools
16K    /etc/sudoers.d
16K    /etc/sysctl.d
16K    /etc/ufw
16K    /lost+found
16K    /usr/lib/ifupdown
16K    /usr/lib/init
16K    /usr/lib/modprobe.d
16K    /usr/lib/sysusers.d
16K    /usr/lib/valgrind
16K    /usr/share/debhelper
16K    /usr/share/menu
16K    /usr/share/pam-configs
16K    /usr/share/smartmontools
16K    /usr/share/systemd
16K    /var/lib/polkit-1
20K    /etc/apt/sources.list.d
20K    /etc/logcheck
20K    /etc/udev
20K    /etc/zfs/zed.d
20K    /usr/include/video
20K    /usr/lib/kernel
20K    /usr/lib/os-prober
20K    /usr/share/pyshared
20K    /usr/share/tabset
20K    /var/cache/ldconfig
20K    /var/lib/initramfs-tools
20K    /var/tmp
24K    /etc/kernel/postinst.d
24K    /usr/include/protocols
24K    /usr/lib/apparmor
24K    /usr/lib/dkms
24K    /usr/lib/ssl
24K    /var/lib/dhcp
28K    /etc/dpkg
28K    /etc/ppp
28K    /usr/include/scsi
28K    /usr/lib/linux-boot-probes
28K    /usr/lib/lsb
28K    /usr/share/debconf
28K    /var/lib/pam
32K    /etc/cron.daily
32K    /etc/dhcp
32K    /usr/include/xen
32K    /usr/share/doc-base
36K    /etc/dkms/template-dkms-mkbmdeb
36K    /etc/logrotate.d
36K    /etc/lvm/profile
36K    /home/felixn
36K    /usr/lib/dpkg
36K    /usr/share/apport
36K    /usr/share/base-files
40K    /etc/apt/apt.conf.d
40K    /home
40K    /tmp
40K    /usr/share/unattended-upgrades
44K    /etc/initramfs-tools/scripts
44K    /root
44K    /usr/lib/groff
44K    /usr/share/pam
48K    /etc/dkms/template-dkms-mkdeb
48K    /etc/proxmox-backup
48K    /etc/security
48K    /usr/include/mtd
48K    /usr/lib/mime
48K    /usr/share/pixmaps
52K    /etc/network
52K    /etc/zfs
52K    /usr/local
56K    /etc/iproute2
56K    /usr/include/arpa
56K    /usr/lib/dbus-1.0
60K    /etc/kernel
60K    /usr/include/net
60K    /usr/lib/tmpfiles.d
60K    /usr/share/dpkg
60K    /usr/share/gdb
64K    /etc/systemd/system
64K    /usr/share/polkit-1
64K    /usr/share/postfix
68K    /etc/initramfs-tools
72K    /etc/apt/trusted.gpg.d
72K    /var/lib/ucf
76K    /usr/share/icons
80K    /etc/apparmor.d/tunables
80K    /etc/grub.d
88K    /etc/default
88K    /etc/pam.d
88K    /usr/lib/os-probes
92K    /etc/dkms
92K    /var/log/apt
108K    /etc/systemd
112K    /etc/init.d
112K    /usr/include/rpc
112K    /var/log/unattended-upgrades
116K    /usr/include/netinet
120K    /usr/share/dbus-1
120K    /usr/share/gcc-8
124K    /etc/postfix
124K    /var/cache/apparmor
132K    /usr/lib/tc
136K    /usr/share/keyrings
140K    /etc/lvm
140K    /usr/lib/firmware
144K    /usr/share/python3
156K    /etc/apt
164K    /usr/share/zsh
184K    /usr/share/initramfs-tools
184K    /usr/share/nano
184K    /usr/share/python-apt
196K    /usr/include/asm-generic
208K    /usr/include/rdma
216K    /usr/lib/terminfo
216K    /var/lib/smartmontools
220K    /usr/include/sound
232K    /usr/share/pbs-i18n
260K    /usr/share/publicsuffix
264K    /usr/share/common-licenses
284K    /usr/include/rpcsvc
296K    /usr/share/gnupg
360K    /usr/lib/man-db
360K    /usr/lib/zfs-linux
364K    /usr/share/fonts-font-awesome
372K    /var/lib/systemd
440K    /usr/share/lintian
460K    /etc/apparmor.d/abstractions
460K    /usr/share/calendar
496K    /usr/share/bug
568K    /usr/lib/sudo
576K    /etc/apparmor.d
576K    /usr/share/bash-completion
588K    /etc/ssh
592K    /usr/share/ca-certificates
592K    /usr/share/google-cloud-sdk
608K    /var/lib/proxmox-backup
612K    /usr/lib/klibc
648K    /etc/ssl/certs
672K    /etc/ssl
692K    /usr/lib/gnupg
856K    /usr/lib/openssh
972K    /usr/lib/apt
1000K    /usr/share/info
1.1M    /usr/share/fonts
1.1M    /var/backups
1.2M    /usr/lib/linux-kbuild-4.19
1.4M    /usr/share/perl5
1.4M    /usr/share/pve-xtermjs
1.7M    /usr/lib/locale
1.7M    /usr/lib/python3
1.8M    /usr/share/groff
2.1M    /usr/include/x86_64-linux-gnu
2.3M    /boot/grub/fonts
2.4M    /boot/grub/i386-pc
2.4M    /usr/lib/postfix
2.4M    /var/cache/debconf
2.5M    /usr/share/grub
2.5M    /var/cache/man
3.2M    /boot/grub/x86_64-efi
3.5M    /usr/share/zoneinfo
4.2M    /boot/grub/locale
4.2M    /etc
5.2M    /usr/lib/file
5.4M    /usr/include/linux
6.3M    /usr/src/linux-headers-4.19.0-11-cloud-amd64
6.3M    /usr/src/linux-headers-4.19.0-12-cloud-amd64
6.3M    /usr/src/linux-headers-4.19.0-13-cloud-amd64
7.3M    /usr/lib/shim
7.3M    /var/spool
7.3M    /var/spool/postfix
9.9M    /usr/lib/systemd
11M    /usr/include/c++
11M    /usr/lib/grub
12M    /boot/grub
12M    /usr/src/linux-headers-4.19.0-13-amd64
12M    /usr/src/linux-headers-4.19.0-14-amd64
14M    /var/lib/dkms
16M    /usr/lib/udev
16M    /usr/src/zfs-0.8.6
21M    /usr/include
21M    /usr/share/perl
23M    /usr/lib/python3.7
23M    /usr/share/javascript
28M    /var/lib/dpkg
29M    /var/mail
32M    /usr/sbin
33M    /usr/share/vim
49M    /usr/share/doc
51M    /usr/share/man
63M    /usr/share/locale
67M    /usr/src/linux-headers-4.19.0-11-common
67M    /usr/src/linux-headers-4.19.0-12-common
67M    /usr/src/linux-headers-4.19.0-13-common
67M    /usr/src/linux-headers-4.19.0-14-common
94M    /usr/lib/gcc
95M    /var/cache/apt
100M    /var/cache
123M    /var/lib/apt
131M    /boot
144M    /usr/bin
166M    /var/lib
185M    /usr/lib/x86_64-linux-gnu
257M    /usr/share
324M    /usr/src
533M    /usr/lib/google-cloud-sdk
670M    /usr/lib/modules
1.6G    /usr/lib
2.3G    /usr
75G    /var
75G    /var/log
75G    /var/log/proxmox-backup
78G    /


*****
Main PBS server:
*****

root@pbs-office:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 48G 0 48G 0% /dev
tmpfs 9.5G 887M 8.6G 10% /run
rpool/ROOT/pbs-1 216G 2.2G 213G 2% /
tmpfs 48G 0 48G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 48G 0 48G 0% /sys/fs/cgroup
rpool 213G 128K 213G 1% /rpool
rpool/ROOT 213G 128K 213G 1% /rpool/ROOT
backup-office 26T 6.1T 20T 24% /backup-office
backup-office/offline 21T 234G 20T 2% /backup-office/offline
backup-office/shares-production 22T 2.0T 20T 10% /backup-office/shares-production
backup-office/shares-temp 22T 1.9T 20T 9% /backup-office/shares-temp
tmpfs 9.5G 0 9.5G 0% /run/user/0

------

root@pbs-office:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
backup-office 10.2T 19.9T 6.08T /backup-office
backup-office/offline 233G 19.9T 233G /backup-office/offline
backup-office/shares-production 2.00T 19.9T 2.00T /backup-office/shares-production
backup-office/shares-temp 1.88T 19.9T 1.88T /backup-office/shares-temp
rpool 2.20G 213G 96K /rpool
rpool/ROOT 2.19G 213G 96K /rpool/ROOT
rpool/ROOT/pbs-1 2.19G 213G 2.19G /

------

root@pbs-office:~# proxmox-backup-manager datastore list

| backup-office | /backup-office | |
| backup-office-offline | /backup-office/offline | |
| shares-production | /backup-office/shares-production | |
| shares-temp | /backup-office/shares-temp | |
 
Last edited by a moderator:
On my main PBS server all datastores have their own zfs filesystem. so you mean that datastore : "│ backup-pbs-gcp │ /backup-pbs-gcp " is a potential problem? - and if so how can I move the datastore?

If it causes real problematic problems: not 100% sure, I'd need to look closer into that. But what it definitively causes is that GC/Verify jobs for that datastores also check the other "sub" datatstore, and that results in a lot of extra work and possible also the big log messages (due to false-positives about missing chunks for the "sub" datastores).

Moving is not exatcly hard, but would need the backup daemons stopped, so that none access the datastores during it, which means no backup can be made or restored during that time (should be done in a few minutes at max though).

Bash:
# first stop the daemons, note that any running backup, GC, ... job may get aborted
systemctl stop proxmox-backup-proxy.service proxmox-backup.service
# now change directory to the separate mount
cd /backup-pbs-gcp
# create the new path so that we do not have multiple datastorews nested (you can give it any other unique folder name, I'd keep)
mkdir local
# ensure correct owner
chown backup:backup local
# move the data over, should be quick as it is a move on the same filesystem - it may complain about "No such file or directory" for some of those, that's OK
mv .chunks/ .gc-status .lock vm ct host local/
# now edit the config adapt the "path" property of the "backup-pbs-gcp" entry (just add local, or whatever you used to it)
nano /etc/proxmox-backup/datastore.cfg
# start the daemons again
systemctl start proxmox-backup-proxy.service proxmox-backup.service

If you're unsure I'd recommend adding a temporary datastore at some other path and play the path change through with that one (fyi: a new Datastore without backups may miss the vm/ct/host folders, that's fine)

75G /var/log/proxmox-backup
So it seems you get many log message and that fills up the root partition, let's check further for which log is causing this:
Bash:
du -hxd1 /var/log/proxmox-backup/
ls -lhSr /var/log/proxmox-backup/api
 
Last edited:
  • Like
Reactions: elBradford
So I have to move the datastore on my onsite PBS also? - meaning create another ZFS filesystem and move datastore to that one and make a folder on the offsite?

here is the offsite result:

root@pbs-offsite-gcp:~# du -hxd1 /var/log/proxmox-backup/
75G /var/log/proxmox-backup/tasks
63M /var/log/proxmox-backup/api
75G /var/log/proxmox-backup/

root@pbs-offsite-gcp:~# ls -lhSr /var/log/proxmox-backup/api
total 63M
-rw-r--r-- 1 backup backup 475K Feb 16 09:15 auth.log
-rw-r--r-- 1 backup backup 579K Dec 30 00:00 access.log.3.zst
-rw-r--r-- 1 backup backup 679K Nov 18 00:00 access.log.6.zst
-rw-r--r-- 1 backup backup 750K Nov 14 00:00 access.log.7.zst
-rw-r--r-- 1 backup backup 778K Jan 25 00:00 access.log.2.zst
-rw-r--r-- 1 backup backup 796K Dec 27 00:00 access.log.4.zst
-rw-r--r-- 1 backup backup 894K Dec 19 00:00 access.log.5.zst
-rw-r--r-- 1 backup backup 25M Feb 16 09:22 access.log
-rw-r--r-- 1 backup backup 33M Jan 24 11:22 access.log.1

root@pbs-offsite-gcp:~# du -hxd1 /var/log/proxmox-backup/tasks/
8.0K /var/log/proxmox-backup/tasks/FB
8.0K /var/log/proxmox-backup/tasks/50
8.0K /var/log/proxmox-backup/tasks/DC
8.0K /var/log/proxmox-backup/tasks/B5
8.0K /var/log/proxmox-backup/tasks/55
16K /var/log/proxmox-backup/tasks/35
8.0K /var/log/proxmox-backup/tasks/86
8.0K /var/log/proxmox-backup/tasks/DB
8.0K /var/log/proxmox-backup/tasks/A6
36G /var/log/proxmox-backup/tasks/03
31G /var/log/proxmox-backup/tasks/9C
8.0K /var/log/proxmox-backup/tasks/CE
12K /var/log/proxmox-backup/tasks/57
8.0K /var/log/proxmox-backup/tasks/3B
8.0K /var/log/proxmox-backup/tasks/29
8.0K /var/log/proxmox-backup/tasks/19
20K /var/log/proxmox-backup/tasks/BB
16K /var/log/proxmox-backup/tasks/4E
8.0K /var/log/proxmox-backup/tasks/70
8.0K /var/log/proxmox-backup/tasks/1E
8.0K /var/log/proxmox-backup/tasks/FC
8.0K /var/log/proxmox-backup/tasks/51
8.0K /var/log/proxmox-backup/tasks/45
12K /var/log/proxmox-backup/tasks/18
8.6G /var/log/proxmox-backup/tasks/A9
8.0K /var/log/proxmox-backup/tasks/16
32K /var/log/proxmox-backup/tasks/1C
12K /var/log/proxmox-backup/tasks/69
8.0K /var/log/proxmox-backup/tasks/EB
8.0K /var/log/proxmox-backup/tasks/32
8.0K /var/log/proxmox-backup/tasks/BC
8.0K /var/log/proxmox-backup/tasks/4B
8.0K /var/log/proxmox-backup/tasks/C7
8.0K /var/log/proxmox-backup/tasks/BD
8.0K /var/log/proxmox-backup/tasks/EE
8.0K /var/log/proxmox-backup/tasks/78
12K /var/log/proxmox-backup/tasks/CB
8.0K /var/log/proxmox-backup/tasks/8C
8.0K /var/log/proxmox-backup/tasks/6B
8.0K /var/log/proxmox-backup/tasks/43
8.0K /var/log/proxmox-backup/tasks/0B
8.0K /var/log/proxmox-backup/tasks/A7
8.0K /var/log/proxmox-backup/tasks/05
16K /var/log/proxmox-backup/tasks/D7
8.0K /var/log/proxmox-backup/tasks/67
8.0K /var/log/proxmox-backup/tasks/2B
8.0K /var/log/proxmox-backup/tasks/DD
8.0K /var/log/proxmox-backup/tasks/27
8.0K /var/log/proxmox-backup/tasks/75
8.0K /var/log/proxmox-backup/tasks/77
8.0K /var/log/proxmox-backup/tasks/65
8.0K /var/log/proxmox-backup/tasks/3A
75G /var/log/proxmox-backup/tasks/
 
So I have to move the datastore on my onsite PBS also? - meaning create another ZFS filesystem and move datastore to that one and make a folder on the offsite?

You have nested Datastores on both, so both would need to be moved.

Simply repeat the procedure for both, note that the datastore ID stays the same so, nothing else would need to be changed afterwards.

You do not necessarily need to create separate ZFS volumes, a separate directory can be enough, but if you want to have more clean separation you can also do that too.
 
alright I will move the data over to another folder
Just be sure to triple check commands so that nothing gets overwritten by mistake or so :)
can I cleanup the logs when datastore is moved?
Yes. Just clearing the inner files of, for example /var/log/proxmox-backup/tasks/03 (which alone golds 39G in logs o_O) should do it - you may not be able to view a few old task logs then, but those are probably not to interesting anyway if containing that much spam.

FYI: You could do that now already, but without moving them it will start to fill up again with the next GC/Verify/... job log.
 
Here is my changes, I did a "chown -R backup:backup xxxx" before moving the files, should I do anything else?

and thanks again

Felix

offsite:

Bash:
root@pbs-offsite-gcp:/backup-pbs-gcp# proxmox-backup-manager datastore list

│ backup-office-offline │ /backup-pbs-gcp/backup-office-offline │         │

│ backup-pbs-gcp        │ /backup-pbs-gcp/backup-office         │         │

│ pd-shares-production  │ /backup-pbs-gcp/pd-shares-production  │         │

│ pd-shares-temp        │ /backup-pbs-gcp/pd-shares-temp        │         │

root@pbs-offsite-gcp:/backup-pbs-gcp# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           799M   81M  718M  11% /run
/dev/sda1       148G   78G   64G  55% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda15      124M  5.4M  119M   5% /boot/efi
backup-pbs-gcp   11T  8.8T  1.6T  85% /backup-pbs-gcp
tmpfs           799M     0  799M   0% /run/user/1000

root@pbs-offsite-gcp:/backup-pbs-gcp#  ls -lah /backup-pbs-gcp/backup-office
total 8.1M
drwxr-xr-x     5 backup backup   7 Feb 16 09:50 .
drwxr-xr-x     6 backup backup   6 Feb 16 09:50 ..
drwxr-x--- 65538 backup backup 65K Oct 27 13:05 .chunks
-rw-r--r--     1 backup backup 345 Feb 15 03:03 .gc-status
-rw-r--r--     1 backup backup   0 Oct 27 13:05 .lock
drwxr-xr-x     4 backup backup   4 Oct 30 20:20 ct
drwxr-xr-x    11 backup backup  11 Oct 31 17:55 vm


onsite:

Bash:
root@pbs-office:~# proxmox-backup-manager datastore list

| backup-office         | /backup-office/office            |         |

| backup-office-offline | /backup-office/offline           |         |

| shares-production     | /backup-office/shares-production |         |

| shares-temp           | /backup-office/shares-temp       |         |


root@pbs-office:~# df -h
Filesystem                       Size  Used Avail Use% Mounted on
udev                              48G     0   48G   0% /dev
tmpfs                            9.5G  887M  8.6G  10% /run
rpool/ROOT/pbs-1                 216G  2.2G  213G   1% /
tmpfs                             48G     0   48G   0% /dev/shm
tmpfs                            5.0M     0  5.0M   0% /run/lock
tmpfs                             48G     0   48G   0% /sys/fs/cgroup
rpool                            213G  128K  213G   1% /rpool
rpool/ROOT                       213G  128K  213G   1% /rpool/ROOT
backup-office                     26T  6.1T   20T  24% /backup-office
backup-office/offline             21T  234G   20T   2% /backup-office/offline
backup-office/shares-production   22T  2.0T   20T  10% /backup-office/shares-production
backup-office/shares-temp         22T  1.9T   20T   9% /backup-office/shares-temp
tmpfs                            9.5G     0  9.5G   0% /run/user/0

root@pbs-office:~# ls -lah /backup-office/
total 62K
drwxr-xr-x  6 backup backup  6 Feb 16 10:35 .
drwxr-xr-x 19 root   root   25 Oct 26 00:17 ..
drwxr-xr-x  6 backup backup  8 Feb 16 10:35 office
drwxr-xr-x  4 backup backup  5 Oct 29 07:13 offline
drwxr-xr-x  4 backup backup  6 Dec 20 17:34 shares-production
drwxr-xr-x  4 backup backup  5 Nov  5 14:32 shares-temp

root@pbs-office:~# ls -lah /backup-office/office/
total 18M
drwxr-xr-x     6 backup backup   8 Feb 16 10:35 .
drwxr-xr-x     6 backup backup   6 Feb 16 10:35 ..
drwxr-x--- 65538 backup backup 65K Oct 26 07:57 .chunks
-rw-r--r--     1 backup backup 336 Dec 20 15:03 .gc-status
-rw-r--r--     1 backup backup   0 Oct 26 07:57 .lock
drwxr-xr-x     4 backup backup   4 Oct 27 02:53 ct
drwxr-xr-x     4 backup backup   4 Nov  3 08:36 host
drwxr-xr-x    15 backup backup  15 Oct 28 00:40 vm
 
Last edited by a moderator:
Here is my changes, I did a "chown -R backup:backup xxxx" before moving the files, should I do anything else?
good call with the chown, will update my post for others stumbling upon it.

The rest looks now OK.

I'd monitor the usage in the log directory a bit over the next days, just to ensure that this was all it took to fix that growth.

We'll look into either disallowing or improving setups with nested datastore directories in the future.
 
Great and thanks for the help, I will check the backup later - if it stil works :)

btw. Is there a plan for have file-restore from filebackups working from the GUI?

and the track-block-changes also working when machine has been rebooted?

and LOVE this product

Felix
 
btw. Is there a plan for have file-restore from filebackups working from the GUI?
For Container and file backups they already do in the PBS interface. Adding that to Proxmox VE and allowing it for VM (block level) backups is planned an in active work.

and the track-block-changes also working when machine has been rebooted?

The fast tracking (dirty-bitmap) only works after one backup run of a freshly booted VM. But, the block difference since the last backup on the server will always be checked, and only new blocks get uploaded, even if dirty-bitmap is not active yet.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!