pve-root nearly full

ttist25 · Jan 16, 2013

Good morning,I'm having some difficulty with my ProxMox 2.1 installation. I seem to be running out of space on my /pve-root. I became aware of the issue because of failed backups. Here is the output of df -h:root@proxvs1:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 95G 85G 5.4G 95% /
tmpfs 83G 0 83G 0% /lib/init/rw
udev 83G 232K 83G 1% /dev
tmpfs 83G 19M 83G 1% /dev/shm
/dev/mapper/pve-data 1.5T 883G 595G 60% /var/lib/vz
/dev/sda1 495M 34M 436M 8% /boot
172.28.250.6:/storage
1006G 337G 619G 36% /mnt/pve/proxbackup
/dev/fuse 30M 16K 30M 1% /etc/pve

I'm not sure what has led to this situation but my first priority is freeing up space on /pve-root. Are there log files, tmp files, etc. which I can look for and safely delete to free up space? Recent changes I have made to this server are adding size: 6144 in vzdump.conf (to alleviate backups failing) and adding (physical) RAM to the server. Thanks in advance for any help you can provide.

udo · Jan 16, 2013

ttist25 said:
Good morning,I'm having some difficulty with my ProxMox 2.1 installation. I seem to be running out of space on my /pve-root. I became aware of the issue because of failed backups. Here is the output of df -h:root@proxvs1:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 95G 85G 5.4G 95% /
tmpfs 83G 0 83G 0% /lib/init/rw
udev 83G 232K 83G 1% /dev
tmpfs 83G 19M 83G 1% /dev/shm
/dev/mapper/pve-data 1.5T 883G 595G 60% /var/lib/vz
/dev/sda1 495M 34M 436M 8% /boot
172.28.250.6:/storage
1006G 337G 619G 36% /mnt/pve/proxbackup
/dev/fuse 30M 16K 30M 1% /etc/pve

I'm not sure what has led to this situation but my first priority is freeing up space on /pve-root. Are there log files, tmp files, etc. which I can look for and safely delete to free up space? Recent changes I have made to this server are adding size: 6144 in vzdump.conf (to alleviate backups failing) and adding (physical) RAM to the server. Thanks in advance for any help you can provide.

Hi,
to find where the "big" files are, is ncdu a nice tool. Use simply this

Code:

apt-get install ncdu
ncdu -x /

to see all stuff which are on the root-filesystem.

I assume you have your backups also on the root-filesystem?

Udo

ttist25 · Jan 16, 2013

udo said:
Hi,
to find where the "big" files are, is ncdu a nice tool. Use simply this

Code:

apt-get install ncdu ncdu -x /

to see all stuff which are on the root-filesystem.

I assume you have your backups also on the root-filesystem?

Udo

Hello Udo!

Thanks for your help. It is greatly appreciated. I installed ncdu.

Here is the output of ncdu -x /:
ncdu 1.6 ~ Use the arrow keys to navigate, press ? for help
--- / --------------------------------------------------------------------------
83.5GiB /mnt
382.2MiB /usr
186.4MiB /var
133.3MiB /lib
6.0MiB /etc
5.7MiB /bin
5.4MiB /sbin
2.4MiB /lib32
60.0kiB /root
28.0kiB /.gnupg
e 16.0kiB /lost+found
12.0kiB /run
12.0kiB /tmp
8.0kiB /home
e 4.0kiB /media
e 4.0kiB /opt
e 4.0kiB /selinux
e 4.0kiB /srv
> 0.0 B /boot
> 0.0 B /dev
> 0.0 B /proc
> 0.0 B /sys
Total disk usage: 84.2GiB Apparent size: 84.1GiB Items: 32522

If I enter the /mnt directory, there is 83.5GiB in /mnt/proxbackup-smb which is a mount on another physical server where I store vzdump backups.

Here is the output of cat /etc/fstab:
root@proxvs1:~# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext3 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext3 defaults 0 1
UUID=4507685b-19ab-4675-9c55-9a10bb90d508 /boot ext3 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

# Added for mount of .6/storage CIFS share
//172.28.250.6/storage /mnt/proxbackup-smb cifs username=root,password=mysupersecretpassword,domain=172.28.250.6 0 0

I'm stumped!

What else can I run for you?

Thanks again!

andydills · Jan 16, 2013

You sure that mount is mounted? Check df.

If it is indeed mounted, try unmounting it and then looking to see what's in the directory...pretty sure you'll find that at one point it was unmounted during a backup, and your backup was written to /, and the remounting of the remote filesystem is 'hiding' the files from you.

ttist25 · Jan 16, 2013

andydills said:
You sure that mount is mounted? Check df.

If it is indeed mounted, try unmounting it and then looking to see what's in the directory...pretty sure you'll find that at one point it was unmounted during a backup, and your backup was written to /, and the remounting of the remote filesystem is 'hiding' the files from you.

Thanks Andy -

Here is the output of df:

root@proxvs1:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 95G 85G 5.3G 95% /
tmpfs 83G 0 83G 0% /lib/init/rw
udev 83G 232K 83G 1% /dev
tmpfs 83G 16M 83G 1% /dev/shm
/dev/mapper/pve-data 1.5T 878G 600G 60% /var/lib/vz
/dev/sda1 495M 34M 436M 8% /boot
172.28.250.6:/storage
1006G 337G 619G 36% /mnt/pve/proxbackup
/dev/fuse 30M 16K 30M 1% /etc/pve

So it looks like it is mounted, correct (still finding my way with Linux so please forgive me)? I can browse the directory during an ssh session so I'm pretty certain it is. Given that, do you still suggest unmounting/remounting? I've got the mount defined in /etc/fstab. Can I use umount and mount to accomplish this without rebooting?

PS - not sure if this is important but, in the GUI under "Datacenter -> Storage" I have an NFS mount to the same location as my CIFS mount (172.28.250.6/storage). The Path/Target for the NFS mount is /mnt/pve/proxbackup. I used this NFS mount as my initial backup storage but added the CIFS share (to the same location) in an attempt to speed up my backups. The Path/Target for the CIFS mount is /mnt/proxbackup-smb.

Thanks again for any help.

andydills · Jan 16, 2013

Yes, you can umount any filesystem that isn't currently in use. If it won't let you, you can use lsof to determine what processes are still using it.

Try umount'ing the filesystem and then seeing what you see in the /mnt/pve/proxbackup directory.

udo · Jan 16, 2013

ttist25 said:
...

If I enter the /mnt directory, there is 83.5GiB in /mnt/proxbackup-smb which is a mount on another physical server where I store vzdump backups.

...

Hi,
due the "-x" are only space from this filesystem is showed - not other mount-points.
Andy is right, that can happens during lost of mount - but /mnt/proxbackup-smb != /mnt/pve/proxbackup !!!

Udo

ttist25 · Jan 16, 2013

andydills said:
Yes, you can umount any filesystem that isn't currently in use. If it won't let you, you can use lsof to determine what processes are still using it.

Try umount'ing the filesystem and then seeing what you see in the /mnt/pve/proxbackup directory.

You are definitely on to something Andy!

I haven't issued umount yet but from an SSH session in 172.28.250.2 (proxvs1) I get the following outputs:

root@proxvs1:/mnt/proxbackup-smb# ls
dump images

root@proxvs1:/mnt/pve/proxbackup# ls
data dump images test.txt

And from the file server where the mount should be pointing (172.28.250.6)

root@fileserver /storage# ls
data dump images test.txt

So - I'm thinking that /mnt/proxbackup-smb is not really mounted and that the files in it (including the log and tar.lzo files in /mnt/proxbackup-smb/dump) are the culprits taking up space in my pve-root.

Does that sound right?

udo · Jan 16, 2013

Hi,
look at your defined storage - and adjust from /mnt/proxbackup-smb to /mnt/pve/proxbackup.

Udo

ttist25 · Jan 16, 2013

udo said:
Hi,
look at your defined storage - and adjust from /mnt/proxbackup-smb to /mnt/pve/proxbackup.

Udo

Hi Udo,

Do you mean through the GUI or otherwise? I have clicked the item and "Edit" under "Storage" in the GUI but I'm not able to edit the path. Do you mean to do this in /etc/fstab?

udo · Jan 16, 2013

ttist25 said:
Hi Udo,

Do you mean through the GUI or otherwise? I have clicked the item and "Edit" under "Storage" in the GUI but I'm not able to edit the path. Do you mean to do this in /etc/fstab?

You can edit fstab and mount the nfs-server to the right position (first move the directoy and create an new one - otherwise you see only the space on the nfs-share and the big backup-files are hidden).

Other way is to edit your storage.cfg below /etc/pve - depends on your system (do you have an cluster and on the other nodes are the mountpoints right?)

Udo

ttist25 · Jan 17, 2013

udo said:
You can edit fstab and mount the nfs-server to the right position (first move the directoy and create an new one - otherwise you see only the space on the nfs-share and the big backup-files are hidden).

Other way is to edit your storage.cfg below /etc/pve - depends on your system (do you have an cluster and on the other nodes are the mountpoints right?)

Udo

Hi Guys,

Man - I've learned a lot today but there's still SO MUCH MORE!

I went in and deleted the .log and .lzo files from the /mnt/proxbackup-smb/dump folder and here's my new df output:
root@proxvs1:/# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 95G 928M 89G 2% /
tmpfs 83G 0 83G 0% /lib/init/rw
udev 83G 232K 83G 1% /dev
tmpfs 83G 19M 83G 1% /dev/shm
/dev/mapper/pve-data 1.5T 879G 599G 60% /var/lib/vz
/dev/sda1 495M 34M 436M 8% /boot
172.28.250.6:/storage
1006G 337G 619G 36% /mnt/pve/proxbackup
/dev/fuse 30M 16K 30M 1% /etc/pve

Yay! 2%!!!

So I'm trying to remove the Storage ID "proxbackup-smb" (which is listed as type "Directory" and Path/Target of "/mnt/proxbackup-smb") and I'm getting "delete storage failed: got lock request timeout (500)"

I commented out the entry in /etc/fstab. When I tried to modify stoarge.cfg I get an input/output error.

Any idea how I can remove that?

Thanks again.

udo · Jan 17, 2013

ttist25 said:
Hi Guys,

Man - I've learned a lot today but there's still SO MUCH MORE!

I went in and deleted the .log and .lzo files from the /mnt/proxbackup-smb/dump folder ...

Hi,
perhaps where a move to the nfs-share the safer way (because, now you lost your backups!)

So I'm trying to remove the Storage ID "proxbackup-smb" (which is listed as type "Directory" and Path/Target of "/mnt/proxbackup-smb") and I'm getting "delete storage failed: got lock request timeout (500)"

I commented out the entry in /etc/fstab. When I tried to modify stoarge.cfg I get an input/output error.

Any idea how I can remove that?

Thanks again.

sounds for me, that you don't have quorum in the cluster and /etc/pve is readonly?
Can you create an file on /etc/pve?

Code:

touch /etc/pve/xx
rm /etc/pve/xx

How is the quorum?

Code:

pvecm status

Udo

ttist25 · Jan 17, 2013

udo said:
Hi,
perhaps where a move to the nfs-share the safer way (because, now you lost your backups!)

sounds for me, that you don't have quorum in the cluster and /etc/pve is readonly?
Can you create an file on /etc/pve?

Code:

touch /etc/pve/xx rm /etc/pve/xx

How is the quorum?

Code:

pvecm status

Udo

Good morning Udo,

No worries on the backups. The files there were not completed successfully and I have successful VM backups on my backup server as well as data backups (belt and suspenders).

Here is the results of your queries:

root@proxvs1:~# touch /etc/pve/xx
touch: cannot touch `/etc/pve/xx': Input/output error
root@proxvs1:~# pvecm status
cman_tool: Cannot open connection to cman, is it running ?

Any help is greatly appreciated.

Thanks!

ttist25 · Jan 17, 2013

udo said:
Hi,
sounds for me, that you don't have quorum in the cluster and /etc/pve is readonly?
Udo

PS - I don't really understand "quorum" (I'm reading about it now) but, I only have a single node on this server if that helps.

Thanks again.

udo · Jan 17, 2013

ttist25 said:
PS - I don't really understand "quorum" (I'm reading about it now) but, I only have a single node on this server if that helps.

Thanks again.

Hi,
if I uderstand you right, that you have only one node and the file /etc/pve/cluster.conf don't excist?!

Simply update to the lates version (2.2) and your /etc/pve should be writable.

Udo

ttist25 · Jan 18, 2013

udo said:
Hi,
if I uderstand you right, that you have only one node and the file /etc/pve/cluster.conf don't excist?!

Simply update to the lates version (2.2) and your /etc/pve should be writable.

Udo

Hey guys. Thanks to your excellent knowledge and help, this is 100% resolved.

I ran /etc/init.d/pve-cluster restart and everything is working fine again.

Just to re-cap for anyone else that might have the same issue:

My pve-root was showing as 95% full.
I had mounted a CIFS share on a remote backup server using /etc/fstab. The local path for the mount was /mnt/proxbackup-smb.
I also had an NFS share on that same remote server which I mounted through the GUI. The local path was /mnt/pve/proxbackup.
My backups were set to backup to the CIFS instance of the share (at /mnt/proxbacup-smb)
The CIFS mount was lost (likely when I rebooted the servers for a RAM upgrade) but appeared to still be mounted when I ran DF (I think because of the NFS mount)
My backups were being placed in the local directory /mnt/proxbackup-smb and so my pve-root space was all being used by the backups.
After all of this I received Input/Output errors whenever I tried to edit anything in the /etc/pve directory and I received "...failed: got lock request timeout (500)" when I tried to modify items through the GUI
I ran /etc/init.d/pve-cluster restart and it fixed the lock issues.

Thanks again Andy and Udo!

ozgurerdogan · Oct 9, 2014

I tried every suggestion on forum about this issue. But just can not find what makes it full. Is not there an ieasy way? Or am I missing something like there are some hidden file filling up space?

ttist25 · Oct 9, 2014

Hey Ozguredogan,

I just happened to be researching another issue and came across my old post and saw you had posted today. What a coincidence!

Have you tried installing and running ncdu as Udo suggested early in this post? That was what pointed me towards the /mnt directory and to the final solution.

I would suggest you install ncdu "sudo apt-get install ncdu" and then run "ncdu -x /" to see where that leads you.

ozgurerdogan · Oct 9, 2014

I did that my friend. I also tried other file search options. Here is what I face;

Code:

root@vztl3:~# df -h
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/pve-root             95G   92G     0 100% /

Here is fstab:

Code:

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext3 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext3 defaults 0 1
UUID=751f483d-a7de-4586-a646-0bca76b94043 /boot ext3 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

And here is ncdu / - without -x;

Code:

ncdu 1.8 ~ Use the arrow keys to navigate, press ? for help
--- / ---------------------------------------------------------------------------------------------------------
    1.1TiB [##########] /storage-on-backup1
  613.3GiB [#####     ] /BACKUP1-ON-REMOTE-SERVER
  545.1GiB [####      ] /BACKUP2-ON-REMOTE-SERVER
  531.9GiB [####      ] /BACKUP3-ON-REMOTE-SERVER
  342.5GiB [##        ] /BACKUP4-ON-REMOTE-SERVER
  305.5GiB [##        ] /var
  568.2MiB [          ] /usr
  489.0MiB [          ] /lib
   92.1MiB [          ] /boot
   21.6MiB [          ] /run
    7.6MiB [          ] /sbin
    6.7MiB [          ] /bin
    4.3MiB [          ] /etc
  100.0KiB [          ] /root
   28.0KiB [          ] /.gnupg
e  16.0KiB [          ] /lost+found
    4.0KiB [          ] /lib64
e   4.0KiB [          ] /selinux
e   4.0KiB [          ] /opt
e   4.0KiB [          ] /mnt
e   4.0KiB [          ] /media
e   4.0KiB [          ] /home
    0.0  B [          ] /sys
    0.0  B [          ] /proc
    0.0  B [          ] /dev
    0.0  B [          ] /tmp

And Content of var have only kvm images no other big files.:

Code:

--- /var/lib/vz/images ----------------------------------------------------------------------------------------
                        /..
   99.3GiB [##########] /200
   82.3GiB [########  ] /104
   77.2GiB [#######   ] /107
   31.3GiB [###       ] /204
   15.2GiB [#         ] /302

I had same problem on a different server and I was able to find a big trash file with ls -a command and after deleting it it went fine. But on this server nothing is odd.

BUT I AM NEARLY SURE of that it is because of incomplete qm backups. But I can not find those backup files.

And also du -shx * :

Code:

root@vztl3:/# du -shx *
6.7M    bin
93M     boot
0       dev
4.4M    etc
4.0K    home
490M    lib
4.0K    lib64
16K     lost+found
4.0K    media
4.0K    mnt
4.0K    opt
du: cannot access `proc/24833/task/24833/fd/4': No such file or directory
du: cannot access `proc/24833/task/24833/fdinfo/4': No such file or directory
du: cannot access `proc/24833/fd/4': No such file or directory
du: cannot access `proc/24833/fdinfo/4': No such file or directory
0       proc
100K    root
356K    run
7.6M    sbin
4.0K    selinux
1.2T    storage-on-backup1
532G    BACKUP1-ON-REMOTE-SERVER
343G    BACKUP2-ON-REMOTE-SERVER
546G    BACKUP3-ON-REMOTE-SERVER
0       sys
0       tmp
569M    usr
149M    var
614G    BACKUP4-ON-REMOTE-SERVER

pve-root nearly full

Active Member

Distinguished Member

Active Member

Active Member

Active Member

Active Member

Distinguished Member

Active Member

Distinguished Member

Active Member

Distinguished Member

Active Member

Distinguished Member

Active Member

Active Member

Distinguished Member

Active Member

Renowned Member

Active Member

Renowned Member