not enough storage to apt update && upgrade

[MAX&GURGEN]

Hello, i have a proxmox ve in clusters, in the last our machine have 4 local storage.
MainStorage w 717Gb, created myself by using directory
backup storage w45Gb completely filled
local storage same as backup
and local-lvm storage empty.

Im guessing update && upgrade goes to local storage which is filled. is it possible to combine local and local-lvm storage or do somthing else?

 

[fabian]

please provide "pveversion -v", "/etc/pve/storage.cfg", "df -h", and "lvs". thanks!

 

[MAX&GURGEN]

please provide "pveversion -v", "/etc/pve/storage.cfg", "df -h", and "lvs". thanks!

root@pve02:~# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.13-3-pve)
pve-manager: 8.1.10 (running version: 8.1.10/4b06efb5db453f29)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5.13-3-pve-signed: 6.5.13-3
proxmox-kernel-6.5: 6.5.13-3
proxmox-kernel-6.5.13-1-pve-signed: 6.5.13-1
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
pve-kernel-6.2.16-5-pve: 6.2.16-6
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.3
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.5
libpve-cluster-perl: 8.0.5
libpve-common-perl: 8.1.1
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.6
libpve-network-perl: 0.9.6
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.1.4
libqb0: not correctly installed
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.5-1
proxmox-backup-file-restore: 3.1.5-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.5
pve-cluster: 8.0.5
pve-container: 5.0.9
pve-docs: 8.1.5
pve-edk2-firmware: 4.2023.08-4
pve-firewall: 5.0.3
pve-firmware: 3.9-2
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-4
pve-xtermjs: 5.3.0-3
qemu-server: 8.1.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve1

root@pve02:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-a-tz-- 65.22g 0.00 1.60
root pve -wi-ao---- 43.74g
swap pve -wi-ao---- 8.00g

root@pve02:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 63G 0 63G 0% /dev
tmpfs 13G 1.5M 13G 1% /run
/dev/mapper/pve-root 43G 42G 0 100% /
tmpfs 63G 63M 63G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
efivarfs 192K 44K 144K 24% /sys/firmware/efi/efivars
/dev/sdb1 669G 44G 592G 7% /mnt/pve/MainStorage
/dev/sda2 1022M 344K 1022M 1% /boot/efi
/dev/fuse 128M 32K 128M 1% /etc/pve
tmpfs 13G 0 13G 0% /run/user/0

root@pve02:/etc/pve# cat storage.cfg
dir: local
path /var/lib/vz
content vztmpl,iso,backup

lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir

dir: backup
path /mnt/backup
content backup,iso,vztmpl,images,rootdir,snippets
prune-backups keep-all=1
shared 0

dir: MainStorage
path /mnt/pve/MainStorage
content images,vztmpl,iso,snippets,rootdir,backup
is_mountpoint 1
nodes pve02

 

[fabian]

if you really don't need local-lvm, you can remove it (both from storage.cfg, and from the VG) and then expand the rootfs. your "backup" storage is sharing its space with the root file system (and the local storage), which is not a good idea (one backup that is too big can break your system!). I would advise rethinking this.

 

[MAX&GURGEN]

if you really don't need local-lvm, you can remove it (both from storage.cfg, and from the VG) and then expand the rootfs. your "backup" storage is sharing its space with the root file system (and the local storage), which is not a good idea (one backup that is too big can break your system!). I would advise rethinking this.

one moment, our backup storage isnt actually backup it is just a name (when we restored the system after a failure, they called the storage as backup). then i just need to clear space from local/backup sorage?

 

[fabian]

yes, that should also help.

 

[MAX&GURGEN]

maybe the last question. how can i break that sharing between backup and local storage?

 

[fabian]

what do you mean? right now both are just directories on the root file system. if you want to have them separate, you need to put them somewhere else (e.g., on their own partition/LV/filesystem).

 

[MAX&GURGEN]

ok, thanks. will teach more!

 

[Kingneutron]

/dev/mapper/pve-root 43G 42G 0 100% /

Your root filesystem is 100% full.

https://unix.stackexchange.com/ques...-up-the-rootfs-partition-or-increase-the-size


Start by deleting old log files ( /var/log is the usual candidate here if you're keeping logs for too long / not rotating them properly. The next usual suspect is likely /home ) Once you free up sufficient space, I recommend installing Midnight Commander to make future deleting and filesystem browsing easier (and safer!) at the commandline.

If you need temporary emergency space to do package updates, /var/cache/apt is where they get downloaded.
In a pinch, you could mount a USB3 disk partition over /var/cache/apt or possibly use a Samba mount if you have NAS with free space. But you still need enough root FS space for the installs to complete.

Since you're likely using an ext4 root FS, if you haven't done this yet this will give you some room to breathe:

' tune2fs -m1 /dev/mapper/pve-root ' # this lowers the reserved space from 5% to 1%, you should put it back higher after you're done troubleshooting to give yourself some leeway in the future.

 

[MAX&GURGEN]

Here are some commands I find that helped me a little to clean up rootfs.
## Clean apt cache
apt-get update
apt-get -f install
apt-get -y autoremove
apt-get clean
## Remove old kernels
dpkg --list | grep 'linux-image' | awk '{ print $2 }' | sort -V | sed -n '/'"$(uname -r | sed "s/\([0-9.-]*\)-\([^0-9]\+\)/\1/")"'/q;p' | xargs apt-get -y purge
dpkg --list | grep 'linux-headers' | awk '{ print $2 }' | sort -V | sed -n '/'"$(uname -r | sed "s/\([0-9.-]*\)-\([^0-9]\+\)/\1/")"'/q;p' | xargs apt-get -y purge