[SOLVED] pve-root directory runs out of space: Why and What should I do?

[jeremy_fritzen]

Hi!
I'm very new to Proxmox, since I just installed it for the first time few days ago.

Some context:
Before that, I was using ESXi. The ESXi OS was installed on a MicroSD Card (8 GB), in a HP Micro Server Gen8.
So, I thought I could just do the same for Proxmox and decided to install it an 8 GB MicroSD Card.

Problem:
Very quickly (maybe 1 or 2 days after the installation), /dev/mapper/pve-root got full.
Here is the result of df -hT:

Filesystem            Size  Used Avail Use% Mounted on
udev                  7.8G     0  7.8G   0% /dev
tmpfs                 1.6G  876K  1.6G   1% /run
/dev/mapper/pve-root  3.4G  3.3G     0 100% /
tmpfs                 7.8G   46M  7.8G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/fuse             128M   16K  128M   1% /etc/pve
/dev/sda1             2.7T   56K  2.6T   1% /mnt/pve/HDD1
SSD_01                225G  128K  225G   1% /SSD_01
tmpfs                 1.6G     0  1.6G   0% /run/user/0

So far, I don't have any VM or Container on it.

What is weird is that /var/log and /var/lib/pve-cluster don't seem to be huge:

root@pve:~# du -sh /var/log /var/lib/pve-cluster/
98M     /var/log
4.0M    /var/lib/pve-cluster/

So I'm wondering:
- What does get it so full so quickly? Can I optimize the configuration?
- Should I definitely move to a bigger MicroSD Card (and probably use other disk, such as an SSD, to store /var/log and /var/lib/pve-cluster?)?
- Should I use a USB Key instead of a MicroSD Card? I would prefer avoid using an SSD or HDD since I don't have a lot of room in my server (enough for 5 HDD and 1 SSD).


Thanks a lot for your help and take care!

 

[t.lamprecht]

So, I thought I could just do the same for Proxmox and decided to install it an 8 GB MicroSD Card.

Did you set the maxvz and swapsize value to zero in the installer? As else not all space is used for the root partition and 8 GiB is rather small for the root block device, so the resulting 3 GiB for the root partition is just not enough space for Proxmox VE.
Having swap or virtual guest data on the SD card wouldn't be that sensible anyway - we could try to make the installer select this a bit smarter in those cases in the future.

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html#advanced_lvm_options

Depending on how far you are with the setup and how comfortable you're with some basic Linux CLI commands it can be easier to re-run the installer, but it should be also possible to fix that up on the already installed system by removing the swap and data LVM logical volumes and then extend the root one to take up all the (new) space.

Already wrote above out when I saw your more specific questions, so a shortly answering them in a possible redundant way:

- What does get it so full so quickly?

Well, it was (auto-)configured that small, so getting full quickly was rather inevitable.

Can I optimize the configuration?

Yes, with some involvement, if you want to go down that route I can give you the rough steps required for that.

Should I definitely move to a bigger MicroSD Card

Well, a bit more space couldn't hurt, IMO 32 GiB+ are nice to have as lower limit to avoid messing much with the defaults and having to fine-tune to ensure updating and running a setup for some longer time is also pain free - a major upgrade to a next future major Proxmox VE version requires normally around 5 GiB of free space to ensure that it goes smoothly.

Should I use a USB Key instead of a MicroSD Card?

That won't make much difference. I mean, if you have a USB 3.x+ slot a fast USB pen drive could be an improvement but in general both are not ideal (due to reliability), but as you said, you have no other option, so it's what it is, and as space is the issue switching storage technology itself won't be enough.

 

[jeremy_fritzen]

Thank you very much for your help!

After all, I decided to reinstall Proxmox from scratch on a new 32GB USB device.
I thought Proxmox would fill the root partition but that's not what it did. I ended up with the following:

NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                  8:0    0   2.7T  0 disk
`-sda1               8:1    0   2.7T  0 part
sdb                  8:16   0   1.8T  0 disk
`-sdb1               8:17   0   1.8T  0 part
sdc                  8:32   1  28.7G  0 disk
|-sdc1               8:33   1  1007K  0 part
|-sdc2               8:34   1   512M  0 part
`-sdc3               8:35   1  28.2G  0 part
  |-pve-swap       253:0    0   3.5G  0 lvm  [SWAP]
  |-pve-root       253:1    0     7G  0 lvm  /
  |-pve-data_tmeta 253:2    0     1G  0 lvm 
  | `-pve-data     253:4    0  12.1G  0 lvm 
  `-pve-data_tdata 253:3    0  12.1G  0 lvm 
    `-pve-data     253:4    0  12.1G  0 lvm 
sdd                  8:48   0 232.9G  0 disk
|-sdd1               8:49   0 232.9G  0 part
`-sdd9               8:57   0     8M  0 part

As you see, sdc is the boot device for Proxmox.
Even if the device is 28GB, root partition is only 7G. I read in the Proxmox wiki that the maximum size of the root volume size is hdsize/4 (so 25% of the whole device size).
So, I end up with new questions :
- What is there this limit? keeping space for ISO/templates or for VM/container storage is good but most of the time, they will be stored on another device so I don't understand this limitation.
- Can I extend the root partition?
- I'm not very comfortable with LVM (though, I'm here also to learn ). Even if I could easily find a tuto to resize it, I don't want to break something on the Proxmox standard storage configuration.


Thanks again for your help, really appreciated

 

[t.lamprecht]

- What is there this limit? keeping space for ISO/templates or for VM/container storage is good but most of the time, they will be stored on another device so I don't understand this limitation.

This is an automatic setting you can override in the installer, as mentioned in my first post you'd set the maxvz and swapsize to 0 in the installer. To do that open the "Options" menu at the disk selection step.

- Can I extend the root partition?
- I'm not very comfortable with LVM (though, I'm here also to learn ). Even if I could easily find a tuto to resize it, I don't want to break something on the Proxmox standard storage configuration.

Yes, using the lvresize command, but you want to remove the data LVM-Thin pool first (as if you used maxvz = 0 in the installer). So, you'd do something like:

# remove the entry from PVE storage config first (this is not destructive)
pvesm remove local-lvm

# remove Thin LVM itself - this is destructive, but should be OK as long as you
# did not created any VM's on that storage
lvremove /dev/pve/data

# optional: remove swap, which may not be that good to have on a slow(er) USB drive
lvremove /dev/pve/data
# avoid mounting now nonexistent swap on boot
sed -i '/ swap /d' /etc/fstab

# resize root LV including the filesystem to the maximal free space available
lvresize -l +100%FREE --resizefs /dev/pve/root

 

[jeremy_fritzen]

Thank you so much! Very easy to follow and it seems to work like a charm

I have other questions related to storage on Proxmox.
But I close this topic since it's resolved and create a new one.

Thanks again!

 

[erikvanberkum]

Good post, here is my expanded version:

# lists information about all block devices
lsblk

# displays amount of space available on the file system
df -hT

# remove the entry from PVE storage config first (this is not destructive)
pvesm remove local-lvm

# remove Thin LVM itself - this is destructive, but should be OK as long as you  did not created any VM's on that storage
lvremove /dev/pve/data

# OPTIONAL: remove swap, which may not be that good to have on a slow(er) USB drive

# disable all swap partitions on the host
swapoff -a

# remove swap - this is destructive
lvremove /dev/pve/swap

# avoid mounting now nonexistent swap on boot
sed -i '/ swap /d' /etc/fstab

# RESIZE ROOT
# resize root LV including the filesystem to the maximal free space available or choose a percentage to your liking
lvresize -l +100%FREE --resizefs /dev/pve/root

 

[bladedude]

Be sure to check what is using up the space on pve-root via:
pvesm list local
cat /etc/pve/storage.cfg
df -hT
In my case, my 64GB pve-root was full of backups. I'd misconfigured the backups to keep all backups, and not rotate X number of backups.