Disk usage confusion - There are no machines or backups in use, but still PM reports disk full?

B

B0M0A0K

New Member
Jan 21, 2025
16
0
1
I'm looking for a little assistance with a scenario that has me completely stumped. VERY new to ProxMox and only a moderate user of the Linux system. Installed PM version 8.3.3 and started to try and create some storage. I had a couple of VMs installed and running, but kept finding that I had them located in the wrong place (I have two large external drives attached and want everything spilt between these so that the local drive partitions only get used for running PM). ProxMox has started to tell me that it can't do things because there is no space left. I was going to try and expand the partition but trying to use growpart or resize2fs tell me that the partition is already maxed out.

So I have no idea what is using the space on this partition /dev/sda3.
I have tried cleaning the cash, deleting temp files. Cleaning backup files and deleting unused VM disks. Still I am told that pve /dev/sda3 is 94% used.

Could someone please lend a hand and offer an explanation for what's going on here. I am temped to start from scratch but, if I don't know what I did wrong the first time round, then I am doomed to make the same mistake again.

Thanks in advance for any help.
FORUM-sda3.png
FORUM-Disks.png
 
Your main OS disk /dev/sda appears correctly partitioned & setup by your Proxmox installation, having an LVM (logical volume) occupying almost all of the sda3 partition. This LVM will be further divided into a local storage & LVM-Thin. You can click on these within the GUI (left-column) to see how much of these are used. Please note that local storage is a Directory type of storage & can contain any type, however LVM-Thin can only store Disk image & Container types. Please see here to get a better understanding of the different types of storages available in PVE.

It remains to be seen what & how your /dev/sdb & /dev/sdc have been setup within PVE. You are also showing a Terra storage that I cannot know what it is. (Maybe these 2 drives in some raid config?).

If you provide output of:
Code: 
cat /etc/pve/storage.cfg
maybe I could see what it is. Please copy the output & post it in the code-editor on the above formatting bar by pressing the "</>" button.
 
  • Like
Reactions: B0M0A0K
gfngfn256 said:
Your main OS disk /dev/sda appears correctly partitioned & setup by your Proxmox installation, having an LVM (logical volume) occupying almost all of the sda3 partition. This LVM will be further divided into a local storage & LVM-Thin. You can click on these within the GUI (left-column) to see how much of these are used. Please note that local storage is a Directory type of storage & can contain any type, however LVM-Thin can only store Disk image & Container types. Please see here to get a better understanding of the different types of storages available in PVE.

It remains to be seen what & how your /dev/sdb & /dev/sdc have been setup within PVE. You are also showing a Terra storage that I cannot know what it is. (Maybe these 2 drives in some raid config?).

If you provide output of:
Code: 
cat /etc/pve/storage.cfg
maybe I could see what it is. Please copy the output & post it in the code-editor on the above formatting bar by pressing the "</>" button.
Click to expand...
Hello and thanks for your reply @gfngfn256

I can confirm that "Terra" was my attempt to map one of my external storage devices in order to make it easier to access for storing VMs, etc. I also have a second external drive called "Transcend" which had not yet been mapped so is not appearing here. My intention is to have backups stored in a separate drive to the main VM repository.

Funnily enough, I was just reviewing storage.cfg when you message came in. I had removed the 'Terra' entry and rebooted the ProxMox server so I can confirm the change and try to start again.

FORUM-Storage-cfg.png

In investigating the 'local' and 'local-lvm' I see now that 'local-lvm' seems to be labelled 'data' and as I understand it this is like the 'working partition' - though I still don't really understand why it is displayed as 94% full, unless the system has pre-allocated all the space and that is what is being displayed, rather than what is actually being used.

I am trying to get the system configured so as the two external drives mentioned are visible under the 'prox' node and had assumed that I simply mounted these, each as a directory?
 
Based on the topic information and this discussion, I'll clarify the situation and help address your issues:
  1. LVM Storage Usage:
    According to your /etc/pve/storage.cfg, you have two storages configured:local, which is linked to the /var/lib/vz directory. This is used for storing files like ISO images, container templates, and backups. local-lvm, which is an LVM-Thin Pool used for storing virtual machine disks and containers.

    The LVM-Thin Pool (local-lvm) pre-allocates all available space in the pool, which is why it shows as "94% used." However, this doesn’t mean that all of the space is actually occupied—it indicates that the space is reserved and can be used for future virtual machines and containers.

    In my case:
    2025-01-27_19-49-27.png
  2. Adding External Drives:
    To connect your external drives (/dev/sdb and /dev/sdc) to Proxmox, follow these steps:
    1. Initialize the disks(if there’s no data to preserve on them):
      Example: for /dev/sdb
      Code: 
      wipefs -a /dev/sdb
parted /dev/sdb mklabel gpt
parted /dev/sdb mkpart primary ext4 0% 100%
mkfs.ext4 /dev/sdb1
    2. Create mount points:
      For example, create a directory /mnt/terra for one of the drives:
      Code: 
      /dev/sdb1 /mnt/terra

      For persistent mounting, add an entry to /etc/fstab:
      Code: 
      echo "/dev/sdb1 /mnt/terra ext4 defaults 0 2" >> /etc/fstab
    3. Add to Proxmox storage configuration:
      In the /etc/pve/storage.cfg file, add a new block:
      Code: 
      dir: terra
    path /mnt/terra
    content images,iso,backup
    nodes prox

      Restart Proxmox to apply the changes

I hope i able to help you.
 
  • Like
Reactions: B0M0A0K
GoXLd said:
Based on the topic information and this discussion, I'll clarify the situation and help address your issues:
  1. LVM Storage Usage:
    According to your /etc/pve/storage.cfg, you have two storages configured:local, which is linked to the /var/lib/vz directory. This is used for storing files like ISO images, container templates, and backups. local-lvm, which is an LVM-Thin Pool used for storing virtual machine disks and containers.

    The LVM-Thin Pool (local-lvm) pre-allocates all available space in the pool, which is why it shows as "94% used." However, this doesn’t mean that all of the space is actually occupied—it indicates that the space is reserved and can be used for future virtual machines and containers.

    In my case:
    View attachment 81463
  2. Adding External Drives:
    To connect your external drives (/dev/sdb and /dev/sdc) to Proxmox, follow these steps:
    1. Initialize the disks(if there’s no data to preserve on them):
      Example: for /dev/sdb
      Code: 
      wipefs -a /dev/sdb
parted /dev/sdb mklabel gpt
parted /dev/sdb mkpart primary ext4 0% 100%
mkfs.ext4 /dev/sdb1
    2. Create mount points:
      For example, create a directory /mnt/terra for one of the drives:
      Code: 
      /dev/sdb1 /mnt/terra

      For persistent mounting, add an entry to /etc/fstab:
      Code: 
      echo "/dev/sdb1 /mnt/terra ext4 defaults 0 2" >> /etc/fstab
    3. Add to Proxmox storage configuration:
      In the /etc/pve/storage.cfg file, add a new block:
      Code: 
      dir: terra
    path /mnt/terra
    content images,iso,backup
    nodes prox

      Restart Proxmox to apply the changes

I hope i able to help you.
Click to expand...
Outstanding information, thanks very much!
 
So I started running out of patience with this and called on a buddy with Linux admin history. He looked over the installation and even though he is not a Proxmox user, is of the opinion that something went amiss during the installation (or I have done something to stuff up the installation) . While 'local' is full, 'local-lvm' is not doing anything and remains empty rather than housing the LM's i have created. It's beginning to look like I may be better off to just do a clean reinstall if I can't find a way to recreate 'local-lvm' and initialize it so that it's seen as the main local storage area (rather than 'local').
 
B0M0A0K said:
While 'local' is full, 'local-lvm' is not doing anything and remains empty rather than housing the LM's i have created.
Click to expand...
When you create a VM (in the GUI wizard) you have to select that local-lvm as the Storage for that VM. I don't know what you have done till now.
 
Please show an image of the GUI selecting from the left-pane local & then local-lvm , from there you can check what is stored there by choosing the next column.
 
gfngfn256 said:
Please show an image of the GUI selecting from the left-pane local & then local-lvm , from there you can check what is stored there by choosing the next column.
Click to expand...
Here are the screenshots you were asking for. As you will see, local-lvm isn't getting used at all. With regards to your earlier reply, I have not been selecting 'local-lvm' as the install location because I want this all to go to my external drives - they have far more capacity than the local partitions. FORUM-Local.pngFORUM-local-lvm.png
 
Starting with the second image - you can see you have the full 166.61GB of the local-lvm available for VM Disks etc.

On the first image (local) - click on each item in the second column to see what is being used there - so start with Backups & then ISO Images & work your way down.

As I said above when you create a VM you choose the storage location where you want to store its storage.

I see you have 2 VMs (1101 & 1201) on your node. Click on the VM & then on Hardware to see where their disks are stored.

To check where all the space is being used on your local storage, you could try the following command:
Code: 
du -h -d1 -x /
 
Hi @gfngfn256
Running du -h -d1 -x / results in the following ;
FORUM-WhatsUsingWhat.png

When clicking through each item in the second column for 'local' there are no items visible in any of the sections. Nothing visible in Backups, ISO Images, CT Templates or Permissions. The Summary, however, shows the following;

FORUM-Local-Summary.png

As I say, no matter how hard I try, the system is storing data in pve rather than the external drives that I choose when creating VMs.
 
So you can see /mnt has 69GB of space used! That comes from a failed mountpoint/s you have setup!
You need to clear out that data to free up your root partition. If the root partition starts to fail on free space - your PVE instance will start to become unavailable. So delete that data now.

As far as I see it, what has/is probably happening, you are setting up external drives with mountpoints, but at some point they become inaccessible (USB flakiness & possible drive renumeration) , so data gets written to the local mountpoint. Another possibility is that you have completely incorrectly setup those external drives/mountpoints, & the data has always been written locally to /mnt.

Either way, you are going to have to learn how to correctly manage your PVE instance. Re-installing is not going to help - if you keep doing it wrong!

Proxmox has a learning-curve - but can also be easily mastered.

Happy Proxmoxing!
 
  • Like
Reactions: Johannes S and news
You must log in or register to reply here.
Top Bottom