[SOLVED] pve-root full

F

fostered

New Member
May 14, 2024
8
0
1
Hello everyone,

After several days of trying to configure my proxmox node (I had never worked with proxmox) I managed to get everything working correctly, but when I started transferring data from an external disk (via USB) to one of the virtual machines that will act as a NAS I received an error lack of space.

I have verified that the problem is in "/dev/mapper/pve-root", but I do not understand why it has been filled and especially how to delete the data, no matter how much I have searched it does not appear.

Code: 
root@PROXMOX:~# df -hT
Filesystem           Type      Size  Used Avail Use% Mounted on
udev                 devtmpfs   20G     0   20G   0% /dev
tmpfs                tmpfs     3.9G  960K  3.9G   1% /run
/dev/mapper/pve-root ext4       94G   83G  6.9G  93% /
tmpfs                tmpfs      20G   46M   20G   1% /dev/shm
tmpfs                tmpfs     5.0M     0  5.0M   0% /run/lock
efivarfs             efivarfs  176K   65K  106K  39% /sys/firmware/efi/efivars
/dev/sdd2            vfat     1022M   12M 1011M   2% /boot/efi
/dev/sdb             ext4      3.6T  5.2G  3.4T   1% /mnt/HD2
/dev/sdc             ext4      3.6T  4.1G  3.4T   1% /mnt/HD1
/dev/sda             ext4      3.6T  2.3G  3.4T   1% /mnt/HD3
/dev/fuse            fuse      128M   24K  128M   1% /etc/pve
tmpfs                tmpfs     3.9G     0  3.9G   0% /run/user/0

Code: 
root@PROXMOX:~# du -h -x -d1 /
2.3G    /usr
5.2M    /etc
4.0K    /opt
4.0K    /home
4.0K    /srv
4.0K    /media
16K     /lost+found
36K     /tmp
94M     /boot
4.0K    /mnt
60K     /root
7.7G    /var
10G     /

Code: 
root@PROXMOX:~# du -sh *
0       filename
0       wget-log
0       wget-log.1
0       wget-log.2

Code: 
root@PROXMOX:~# lvs
  LV            VG  Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve twi-aotz-- <319.61g             9.63   0.83                          
  root          pve -wi-ao----   96.00g                                                  
  swap          pve -wi-ao----    8.00g                                                  
  vm-100-disk-0 pve Vwi-aotz--    8.00g data        26.95                                
  vm-101-disk-0 pve Vwi-aotz--    4.00m data        0.00                                  
  vm-101-disk-1 pve Vwi-aotz--   32.00g data        17.55                                
  vm-102-disk-0 pve Vwi-a-tz--    1.00g data        56.76                                
  vm-103-disk-0 pve Vwi-a-tz--    4.00m data        14.06                                
  vm-103-disk-1 pve Vwi-a-tz--  600.00m data        99.70                                
  vm-103-disk-2 pve Vwi-a-tz--  120.00g data        18.21                                
  vm-103-disk-3 pve Vwi-a-tz--    4.00m data        1.56                                  
  vm-103-disk-4 pve Vwi-a-tz--   40.00g data        0.00

Code: 
root@PROXMOX:~# vgs
  VG  #PV #LV #SN Attr   VSize    VFree
  pve   1  12   0 wz--n- <446.13g 16.00g

Code: 
root@PROXMOX:~# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                            8:0    0   3.6T  0 disk /mnt/HD3
sdb                            8:16   0   3.6T  0 disk /mnt/HD2
sdc                            8:32   0   3.6T  0 disk /mnt/HD1
sdd                            8:48   0 447.1G  0 disk
├─sdd1                         8:49   0  1007K  0 part
├─sdd2                         8:50   0     1G  0 part /boot/efi
└─sdd3                         8:51   0 446.1G  0 part
  ├─pve-swap                 252:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                 252:1    0    96G  0 lvm  /
  ├─pve-data_tmeta           252:2    0   3.3G  0 lvm
  │ └─pve-data-tpool         252:4    0 319.6G  0 lvm
  │   ├─pve-data             252:5    0 319.6G  1 lvm
  │   ├─pve-vm--100--disk--0 252:6    0     8G  0 lvm
  │   ├─pve-vm--101--disk--0 252:7    0     4M  0 lvm
  │   ├─pve-vm--101--disk--1 252:8    0    32G  0 lvm
  │   ├─pve-vm--103--disk--0 252:9    0     4M  0 lvm
  │   ├─pve-vm--103--disk--1 252:10   0   600M  0 lvm
  │   ├─pve-vm--103--disk--2 252:11   0   120G  0 lvm
  │   ├─pve-vm--103--disk--3 252:12   0     4M  0 lvm
  │   ├─pve-vm--103--disk--4 252:13   0    40G  0 lvm
  │   └─pve-vm--102--disk--0 252:14   0     1G  0 lvm
  └─pve-data_tdata           252:3    0 319.6G  0 lvm
    └─pve-data-tpool         252:4    0 319.6G  0 lvm
      ├─pve-data             252:5    0 319.6G  1 lvm
      ├─pve-vm--100--disk--0 252:6    0     8G  0 lvm
      ├─pve-vm--101--disk--0 252:7    0     4M  0 lvm
      ├─pve-vm--101--disk--1 252:8    0    32G  0 lvm
      ├─pve-vm--103--disk--0 252:9    0     4M  0 lvm
      ├─pve-vm--103--disk--1 252:10   0   600M  0 lvm
      ├─pve-vm--103--disk--2 252:11   0   120G  0 lvm
      ├─pve-vm--103--disk--3 252:12   0     4M  0 lvm
      ├─pve-vm--103--disk--4 252:13   0    40G  0 lvm
      └─pve-vm--102--disk--0 252:14   0     1G  0 lvm
sde                            8:64   0   1.8T  0 disk
└─sde1                         8:65   0   1.8T  0 part
sr0                           11:0    1  1024M  0 rom
root@PROXMOX:~#

Surely it will be something simple to solve and very obvious, but I am not able to solve it even though I have reviewed all the threads that I have found with a similar issue.

Thank you very much in advance,
 
Last edited:
fostered said:
but when I started transferring data from an external disk (via USB) to one of the virtual machines
Click to expand...
Not sure what you mean by this.:

If you wanted to transfer "data" to the VM's virtual disk (or one of them) from the external USB disk. To do this you'd need to passthrough the USB port/external drive to the VM. If you did anything else, (& your lsblk output definitely doesn't look like you did this), its probably just going to take up place on your system/OS disk, without accomplishing much else. Remember to accomplish this, the VM's disk must have enough space to store that data.

Alternatively you could just attach (through passthrough) the external USB drive & use it as is in the VM.

However if what you want to do is attach another virtual disk to the VM from an image stored on the external USB, then depending on the image file, after mounting the USB drive to the host enter something like the following:

Code: 
qm importdisk {VMID} {path/to/mounted/externalUSB/image_name.img} {PVE_defined_Storage_name}
This will create an "unused" disk for that VM. You can later add it (GUI) to the VM to be used. You can also specify a format for the image to be used by adding --format <qcow2 | raw | vmdk> , as per the docs. For this you have to make sure you have enough space within the defined named storage for the virtual disk's file size.
 
  • Like
Reactions: fostered and leesteken
Thank you very much for your quick response,

The proxmox node runs on an ML30 Gen 9 server that has an SSD installed (where proxmox runs and saves VMs and backups) and three SATA disks of 4TB each that serve as storage for the VM with NAS functions. The three SATA disks are mounted and function correctly as independent volumes in the VM that acts as NAS.

Once I managed to install the basics (VMs, container, passthrough of the SATA disks...) I connected the USB disk to the server, added it to the VM and began to transfer information, thinking that I was transferring it to one of the SATA disks , but a few minutes later the problem of filling "/dev/mapper/pve-root" appeared.

For now I don't need to have the USB disk permanently connected, I just want to transfer the information I have to the indicated VM.

I take note and appreciate the instructions you give me to use the USB disk, but my main problem now is emptying "/dev/mapper/pve-root" and I can't find a way to do it.

Thank you and sorry for the bad explanations and my terrible English.
 
I'm guessing it started copying files (from your incorrect copy method) but somehow they are yet to be written/updated/cache etc..
Try rebooting the PVE node & I believe you'll get your space back.
 
  • Like
Reactions: fostered
Thanks for your help.

I have rebooting the node many times since the failure, but it is still in the same state.

I have tried all kinds of actions, but I can't empty the directory or see what fills it.
 
Last edited:
Interesting .....

What does this show:
Code: 
du -h -x -a -d1 /

What does this show:
Code: 
e2fsck -n /dev/mapper/pve-root
Its not perfect with the mounted fs - but maybe we'll get some idea
 
gfngfn256 said:
Interesting .....

What does this show:
Code: 
du -h -x -a -d1 /

What does this show:
Code: 
e2fsck -n /dev/mapper/pve-root
Its not perfect with the mounted fs - but maybe we'll get some idea
Click to expand...

Hi,

Code: 
root@PROXMOX:~# du -h -x -a -d1 /
2.3G    /usr
5.3M    /etc
4.0K    /opt
0       /lib64
0       /sbin
0       /lib
4.0K    /home
4.0K    /srv
4.0K    /media
16K     /lost+found
36K     /tmp
0       /bin
94M     /boot
4.0K    /mnt
64K     /root
7.7G    /var
10G     /

Code: 
root@PROXMOX:~# e2fsck -n /dev/mapper/pve-root
e2fsck 1.47.0 (5-Feb-2023)
Warning!  /dev/mapper/pve-root is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/mapper/pve-root: clean, 53741/6291456 files, 22121172/25165824 blocks

I don't know if it will help, but I have located the moment in which the failure occurred in the logs:

Code: 
May 10 11:00:48 PROXMOX kernel: usb 1-10: reset full-speed USB device number 3 using xhci_hcd
May 10 11:00:55 PROXMOX pveproxy[182714]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1901.
May 10 11:00:55 PROXMOX pveproxy[182714]: error writing access log

Thanks,
 
Last edited:
bbgeek17 said:
What happens when you execute:
umount /mnt/HD1
umount /mnt/HD2
umount /mnt/HD3
df -h -d1 /mnt

?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...

Hi,

Code: 
root@PROXMOX:~# umount /mnt/HD1
root@PROXMOX:~# umount /mnt/HD2
root@PROXMOX:~# umount /mnt/HD3
root@PROXMOX:~# df -h -d1 /mnt
df: invalid option -- 'd'
Try 'df --help' for more information.

Thanks!!
 
Looks like you found your lost occupied capacity.
As you can see your /mnt/HD2 alone is taking 69GB. This, of course, assuming you properly unmounted the external disk, as requested.

Essentially what happened is you wrote data to /mnt/HD2 when there was no disk mounted there. So you wrote data to your root filesystem.
Confirm that nothing is mounted "mount|grep HD", and if there is no output - go into those directories and delete the data you don't need.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: fostered
Well there you have it, if there are no drives now mounted in /mnt you've filled it with 73gb of local data.
First making sure nothing is mounted in /mnt, enter the following:
rm -r /mnt/*
 
  • Like
Reactions: fostered
I'm a little surprised that your umount commands above don't show the unmounting output.

After doing the above, you'll have to recreate the mountpoint dirs HD1, HD2 etc. & remount.
 
Last edited:
  • Like
Reactions: fostered
  • Like
Reactions: fostered
bbgeek17 said:
thats a bit drastic. If OP has anything else in there or a PVE cluster, the results can be disastrous.
I'd suggest more targeted clean up.
Click to expand...
I did tell him to check. But I agree with you fully.

bbgeek17 said:
a successful umount does not produce output by default
Click to expand...
Not in my experience: I always get back something like /mnt/HD1 is not mounted. True for Proxmox & all distros I've used.
 
  • Like
Reactions: fostered
SOLVED, thank you both very much!

There was nothing in /mnt. In fact I had backed up everything to start from scratch tonight if I couldn't fix it, so I could take my chances with the drastic solution.

I am clear that I still have a lot to learn. I get on with it.

Thank you very much and sorry for the inconvenience.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back