1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

pve-root nearly full

Discussion in 'Proxmox VE: Installation and configuration' started by ttist25, Jan 16, 2013.

  1. ttist25

    ttist25 Member

    Joined:
    May 17, 2012
    Messages:
    32
    Likes Received:
    1
    Good morning,I'm having some difficulty with my ProxMox 2.1 installation. I seem to be running out of space on my /pve-root. I became aware of the issue because of failed backups. Here is the output of df -h:root@proxvs1:/# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/pve-root 95G 85G 5.4G 95% /
    tmpfs 83G 0 83G 0% /lib/init/rw
    udev 83G 232K 83G 1% /dev
    tmpfs 83G 19M 83G 1% /dev/shm
    /dev/mapper/pve-data 1.5T 883G 595G 60% /var/lib/vz
    /dev/sda1 495M 34M 436M 8% /boot
    172.28.250.6:/storage
    1006G 337G 619G 36% /mnt/pve/proxbackup
    /dev/fuse 30M 16K 30M 1% /etc/pve



    I'm not sure what has led to this situation but my first priority is freeing up space on /pve-root. Are there log files, tmp files, etc. which I can look for and safely delete to free up space? Recent changes I have made to this server are adding size: 6144 in vzdump.conf (to alleviate backups failing) and adding (physical) RAM to the server. Thanks in advance for any help you can provide.
     
  2. udo

    udo Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,366
    Likes Received:
    100
    Hi,
    to find where the "big" files are, is ncdu a nice tool. Use simply this
    Code:
    apt-get install ncdu
    ncdu -x /
    
    to see all stuff which are on the root-filesystem.

    I assume you have your backups also on the root-filesystem?

    Udo
     
  3. ttist25

    ttist25 Member

    Joined:
    May 17, 2012
    Messages:
    32
    Likes Received:
    1
    Hello Udo!

    Thanks for your help. It is greatly appreciated. I installed ncdu.

    Here is the output of ncdu -x /:
    ncdu 1.6 ~ Use the arrow keys to navigate, press ? for help
    --- / --------------------------------------------------------------------------
    83.5GiB /mnt
    382.2MiB /usr
    186.4MiB /var
    133.3MiB /lib
    6.0MiB /etc
    5.7MiB /bin
    5.4MiB /sbin
    2.4MiB /lib32
    60.0kiB /root
    28.0kiB /.gnupg
    e 16.0kiB /lost+found
    12.0kiB /run
    12.0kiB /tmp
    8.0kiB /home
    e 4.0kiB /media
    e 4.0kiB /opt
    e 4.0kiB /selinux
    e 4.0kiB /srv
    > 0.0 B /boot
    > 0.0 B /dev
    > 0.0 B /proc
    > 0.0 B /sys
    Total disk usage: 84.2GiB Apparent size: 84.1GiB Items: 32522



    If I enter the /mnt directory, there is 83.5GiB in /mnt/proxbackup-smb which is a mount on another physical server where I store vzdump backups.

    Here is the output of cat /etc/fstab:
    root@proxvs1:~# cat /etc/fstab
    # <file system> <mount point> <type> <options> <dump> <pass>
    /dev/pve/root / ext3 errors=remount-ro 0 1
    /dev/pve/data /var/lib/vz ext3 defaults 0 1
    UUID=4507685b-19ab-4675-9c55-9a10bb90d508 /boot ext3 defaults 0 1
    /dev/pve/swap none swap sw 0 0
    proc /proc proc defaults 0 0

    # Added for mount of .6/storage CIFS share
    //172.28.250.6/storage /mnt/proxbackup-smb cifs username=root,password=mysupersecretpassword,domain=172.28.250.6 0 0

    I'm stumped!

    What else can I run for you?

    Thanks again!
     
  4. andydills

    andydills Member

    Joined:
    Jan 5, 2013
    Messages:
    60
    Likes Received:
    0
    You sure that mount is mounted? Check df.

    If it is indeed mounted, try unmounting it and then looking to see what's in the directory...pretty sure you'll find that at one point it was unmounted during a backup, and your backup was written to /, and the remounting of the remote filesystem is 'hiding' the files from you.
     
  5. ttist25

    ttist25 Member

    Joined:
    May 17, 2012
    Messages:
    32
    Likes Received:
    1
    Thanks Andy -

    Here is the output of df:

    root@proxvs1:/# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/pve-root 95G 85G 5.3G 95% /
    tmpfs 83G 0 83G 0% /lib/init/rw
    udev 83G 232K 83G 1% /dev
    tmpfs 83G 16M 83G 1% /dev/shm
    /dev/mapper/pve-data 1.5T 878G 600G 60% /var/lib/vz
    /dev/sda1 495M 34M 436M 8% /boot
    172.28.250.6:/storage
    1006G 337G 619G 36% /mnt/pve/proxbackup
    /dev/fuse 30M 16K 30M 1% /etc/pve




    So it looks like it is mounted, correct (still finding my way with Linux so please forgive me)? I can browse the directory during an ssh session so I'm pretty certain it is. Given that, do you still suggest unmounting/remounting? I've got the mount defined in /etc/fstab. Can I use umount and mount to accomplish this without rebooting?

    PS - not sure if this is important but, in the GUI under "Datacenter -> Storage" I have an NFS mount to the same location as my CIFS mount (172.28.250.6/storage). The Path/Target for the NFS mount is /mnt/pve/proxbackup. I used this NFS mount as my initial backup storage but added the CIFS share (to the same location) in an attempt to speed up my backups. The Path/Target for the CIFS mount is /mnt/proxbackup-smb.

    Thanks again for any help.
     
  6. andydills

    andydills Member

    Joined:
    Jan 5, 2013
    Messages:
    60
    Likes Received:
    0
    Yes, you can umount any filesystem that isn't currently in use. If it won't let you, you can use lsof to determine what processes are still using it.

    Try umount'ing the filesystem and then seeing what you see in the /mnt/pve/proxbackup directory.
     
  7. udo

    udo Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,366
    Likes Received:
    100
    Hi,
    due the "-x" are only space from this filesystem is showed - not other mount-points.
    Andy is right, that can happens during lost of mount - but /mnt/proxbackup-smb != /mnt/pve/proxbackup !!!

    Udo
     
  8. ttist25

    ttist25 Member

    Joined:
    May 17, 2012
    Messages:
    32
    Likes Received:
    1

    You are definitely on to something Andy!

    I haven't issued umount yet but from an SSH session in 172.28.250.2 (proxvs1) I get the following outputs:

    root@proxvs1:/mnt/proxbackup-smb# ls
    dump images

    root@proxvs1:/mnt/pve/proxbackup# ls
    data dump images test.txt

    And from the file server where the mount should be pointing (172.28.250.6)

    root@fileserver /storage# ls
    data dump images test.txt

    So - I'm thinking that /mnt/proxbackup-smb is not really mounted and that the files in it (including the log and tar.lzo files in /mnt/proxbackup-smb/dump) are the culprits taking up space in my pve-root.


    Does that sound right?
     
  9. udo

    udo Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,366
    Likes Received:
    100
    Hi,
    look at your defined storage - and adjust from /mnt/proxbackup-smb to /mnt/pve/proxbackup.

    Udo
     
  10. ttist25

    ttist25 Member

    Joined:
    May 17, 2012
    Messages:
    32
    Likes Received:
    1
    Hi Udo,

    Do you mean through the GUI or otherwise? I have clicked the item and "Edit" under "Storage" in the GUI but I'm not able to edit the path. Do you mean to do this in /etc/fstab?
     
  11. udo

    udo Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,366
    Likes Received:
    100
    You can edit fstab and mount the nfs-server to the right position (first move the directoy and create an new one - otherwise you see only the space on the nfs-share and the big backup-files are hidden).

    Other way is to edit your storage.cfg below /etc/pve - depends on your system (do you have an cluster and on the other nodes are the mountpoints right?)

    Udo
     
  12. ttist25

    ttist25 Member

    Joined:
    May 17, 2012
    Messages:
    32
    Likes Received:
    1
    Hi Guys,


    Man - I've learned a lot today but there's still SO MUCH MORE! :)

    I went in and deleted the .log and .lzo files from the /mnt/proxbackup-smb/dump folder and here's my new df output:
    root@proxvs1:/# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/mapper/pve-root 95G 928M 89G 2% /
    tmpfs 83G 0 83G 0% /lib/init/rw
    udev 83G 232K 83G 1% /dev
    tmpfs 83G 19M 83G 1% /dev/shm
    /dev/mapper/pve-data 1.5T 879G 599G 60% /var/lib/vz
    /dev/sda1 495M 34M 436M 8% /boot
    172.28.250.6:/storage
    1006G 337G 619G 36% /mnt/pve/proxbackup
    /dev/fuse 30M 16K 30M 1% /etc/pve

    Yay! 2%!!!

    So I'm trying to remove the Storage ID "proxbackup-smb" (which is listed as type "Directory" and Path/Target of "/mnt/proxbackup-smb") and I'm getting "delete storage failed: got lock request timeout (500)"

    I commented out the entry in /etc/fstab. When I tried to modify stoarge.cfg I get an input/output error.

    Any idea how I can remove that?

    Thanks again.
     
  13. udo

    udo Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,366
    Likes Received:
    100
    Hi,
    perhaps where a move to the nfs-share the safer way (because, now you lost your backups!)
    sounds for me, that you don't have quorum in the cluster and /etc/pve is readonly?
    Can you create an file on /etc/pve?
    Code:
    touch /etc/pve/xx
    rm /etc/pve/xx
    
    How is the quorum?
    Code:
    pvecm status
    
    Udo
     
  14. ttist25

    ttist25 Member

    Joined:
    May 17, 2012
    Messages:
    32
    Likes Received:
    1

    Good morning Udo,

    No worries on the backups. The files there were not completed successfully and I have successful VM backups on my backup server as well as data backups (belt and suspenders).

    Here is the results of your queries:

    root@proxvs1:~# touch /etc/pve/xx
    touch: cannot touch `/etc/pve/xx': Input/output error
    root@proxvs1:~# pvecm status
    cman_tool: Cannot open connection to cman, is it running ?

    Any help is greatly appreciated.

    Thanks!
     
  15. ttist25

    ttist25 Member

    Joined:
    May 17, 2012
    Messages:
    32
    Likes Received:
    1
    PS - I don't really understand "quorum" (I'm reading about it now) but, I only have a single node on this server if that helps.

    Thanks again.
     
  16. udo

    udo Well-Known Member
    Proxmox VE Subscriber

    Joined:
    Apr 22, 2009
    Messages:
    5,366
    Likes Received:
    100
    Hi,
    if I uderstand you right, that you have only one node and the file /etc/pve/cluster.conf don't excist?!

    Simply update to the lates version (2.2) and your /etc/pve should be writable.

    Udo
     
  17. ttist25

    ttist25 Member

    Joined:
    May 17, 2012
    Messages:
    32
    Likes Received:
    1

    Hey guys. Thanks to your excellent knowledge and help, this is 100% resolved. :) I ran /etc/init.d/pve-cluster restart and everything is working fine again.

    Just to re-cap for anyone else that might have the same issue:

    • My pve-root was showing as 95% full.
    • I had mounted a CIFS share on a remote backup server using /etc/fstab. The local path for the mount was /mnt/proxbackup-smb.
    • I also had an NFS share on that same remote server which I mounted through the GUI. The local path was /mnt/pve/proxbackup.
    • My backups were set to backup to the CIFS instance of the share (at /mnt/proxbacup-smb)
    • The CIFS mount was lost (likely when I rebooted the servers for a RAM upgrade) but appeared to still be mounted when I ran DF (I think because of the NFS mount)
    • My backups were being placed in the local directory /mnt/proxbackup-smb and so my pve-root space was all being used by the backups.
    • After all of this I received Input/Output errors whenever I tried to edit anything in the /etc/pve directory and I received "...failed: got lock request timeout (500)" when I tried to modify items through the GUI
    • I ran /etc/init.d/pve-cluster restart and it fixed the lock issues.

    Thanks again Andy and Udo!
     
  18. ozgurerdogan

    ozgurerdogan Member

    Joined:
    May 2, 2010
    Messages:
    343
    Likes Received:
    0
    I tried every suggestion on forum about this issue. But just can not find what makes it full. Is not there an ieasy way? Or am I missing something like there are some hidden file filling up space?
     
  19. ttist25

    ttist25 Member

    Joined:
    May 17, 2012
    Messages:
    32
    Likes Received:
    1
    Hey Ozguredogan,

    I just happened to be researching another issue and came across my old post and saw you had posted today. What a coincidence!

    Have you tried installing and running ncdu as Udo suggested early in this post? That was what pointed me towards the /mnt directory and to the final solution.

    I would suggest you install ncdu "sudo apt-get install ncdu" and then run "ncdu -x /" to see where that leads you.
     
  20. ozgurerdogan

    ozgurerdogan Member

    Joined:
    May 2, 2010
    Messages:
    343
    Likes Received:
    0
    I did that my friend. I also tried other file search options. Here is what I face;

    Code:
    root@vztl3:~# df -h
    Filesystem                      Size  Used Avail Use% Mounted on
    /dev/mapper/pve-root             95G   92G     0 100% /
    
    Here is fstab:
    Code:
    # <file system> <mount point> <type> <options> <dump> <pass>
    /dev/pve/root / ext3 errors=remount-ro 0 1
    /dev/pve/data /var/lib/vz ext3 defaults 0 1
    UUID=751f483d-a7de-4586-a646-0bca76b94043 /boot ext3 defaults 0 1
    /dev/pve/swap none swap sw 0 0
    proc /proc proc defaults 0 0
    And here is ncdu / - without -x;
    Code:
    
    ncdu 1.8 ~ Use the arrow keys to navigate, press ? for help
    --- / ---------------------------------------------------------------------------------------------------------
        1.1TiB [##########] /storage-on-backup1
      613.3GiB [#####     ] /BACKUP1-ON-REMOTE-SERVER
      545.1GiB [####      ] /BACKUP2-ON-REMOTE-SERVER
      531.9GiB [####      ] /BACKUP3-ON-REMOTE-SERVER
      342.5GiB [##        ] /BACKUP4-ON-REMOTE-SERVER
      305.5GiB [##        ] /var
      568.2MiB [          ] /usr
      489.0MiB [          ] /lib
       92.1MiB [          ] /boot
       21.6MiB [          ] /run
        7.6MiB [          ] /sbin
        6.7MiB [          ] /bin
        4.3MiB [          ] /etc
      100.0KiB [          ] /root
       28.0KiB [          ] /.gnupg
    e  16.0KiB [          ] /lost+found
        4.0KiB [          ] /lib64
    e   4.0KiB [          ] /selinux
    e   4.0KiB [          ] /opt
    e   4.0KiB [          ] /mnt
    e   4.0KiB [          ] /media
    e   4.0KiB [          ] /home
        0.0  B [          ] /sys
        0.0  B [          ] /proc
        0.0  B [          ] /dev
        0.0  B [          ] /tmp
    

    And Content of var have only kvm images no other big files.:
    Code:
    --- /var/lib/vz/images ----------------------------------------------------------------------------------------
                            /..
       99.3GiB [##########] /200
       82.3GiB [########  ] /104
       77.2GiB [#######   ] /107
       31.3GiB [###       ] /204
       15.2GiB [#         ] /302
    
    
    I had same problem on a different server and I was able to find a big trash file with ls -a command and after deleting it it went fine. But on this server nothing is odd.

    BUT I AM NEARLY SURE of that it is because of incomplete qm backups. But I can not find those backup files.

    And also du -shx * :
    Code:
    root@vztl3:/# du -shx *
    6.7M    bin
    93M     boot
    0       dev
    4.4M    etc
    4.0K    home
    490M    lib
    4.0K    lib64
    16K     lost+found
    4.0K    media
    4.0K    mnt
    4.0K    opt
    du: cannot access `proc/24833/task/24833/fd/4': No such file or directory
    du: cannot access `proc/24833/task/24833/fdinfo/4': No such file or directory
    du: cannot access `proc/24833/fd/4': No such file or directory
    du: cannot access `proc/24833/fdinfo/4': No such file or directory
    0       proc
    100K    root
    356K    run
    7.6M    sbin
    4.0K    selinux
    1.2T    storage-on-backup1
    532G    BACKUP1-ON-REMOTE-SERVER
    343G    BACKUP2-ON-REMOTE-SERVER
    546G    BACKUP3-ON-REMOTE-SERVER
    0       sys
    0       tmp
    569M    usr
    149M    var
    614G    BACKUP4-ON-REMOTE-SERVER
    
    
     
    #20 ozgurerdogan, Oct 9, 2014
    Last edited: Oct 9, 2014

Share This Page