HDD space gets counted twice

Hook · Nov 21, 2012

Hey!

I got a problem with the current proxmox installation (problem also existed on 2.1 installations as far as i know).
On all my servers with proxmox installed (the default installation image from the website) the HDD space gets counted twice.

Here's the proof:

Code:

root@de-fra1-c10:/var/lib/vz# df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/pve-data   59G   58G  362M 100% /var/lib/vz

Code:

root@de-fra1-c10:/var/lib/vz# du -shx .
24G    .

Code:

root@de-fra1-c10:/var/lib/vz# du -sh 
4.0K    dump
4.0K    images
4.0K    lock
16K    lost+found
23G    private
23G    root
571M    template

Code:

root@de-fra1-c10:/var/lib/vz# du -shx 
4.0K    dump
4.0K    images
4.0K    lock
16K    lost+found
23G    private
128K    root
571M    template

I installed everything with default parameters (except of the IP's and the country). Also my data in the Containers doesn't exceed 23GB in total. However, on the host system, 58GB get counted.
I have a 120GB SSD installed, where 30 are for the root partition and 16 for swap (Which is way too much in my opinion, but it was a default installation, so this is okay)

Can anyone tell me why the space gets counted twice? Is there anything I could do to solve it? My HDD is always full because of this problem, although there should be enough space left as all this data doesn't even exist ...

Thanks in advance for your help!

tom · Nov 22, 2012

I don´t see it. please exlain you problem again.

btw, if you don´t like the swap or root size, you can set this values - see http://pve.proxmox.com/wiki/Quick_installation

Hook · Nov 22, 2012

Thank you for your response!

I think the problem is that everything gets symlinked in /var/lib/vz/root, the actual files are located in /var/lib/vz/private.
However, with df -h, also the symlinked files get counted, although they don't even exist. So the files get counted twice as you can see here:

Code:

root@de-fra1-c10:/var/lib/vz# du -sh
23G    private
23G    root

Now only the files on the filesystem:

Code:

root@de-fra1-c10:/var/lib/vz# du -shx  
23G    private 
128K    root

Also, as you can see here, there are 58GB used:

Code:

root@de-fra1-c10:/var/lib/vz# df -h . 
Filesystem            Size  Used Avail Use% Mounted on 
/dev/mapper/pve-data   59G   58G  362M 100% /var/lib/vz

The files which are actually there:

Code:

root@de-fra1-c10:/var/lib/vz# 
du -shx . 24G    .

There you can see that there should be more than enough free space, however it gets counted in a weird way, so it's impossible to use the whole space available.

udo · Nov 22, 2012

Hook said:
...

Code:

root@de-fra1-c10:/var/lib/vz# du -shx 23G private 128K root

Also, as you can see here, there are 58GB used:

Code:

root@de-fra1-c10:/var/lib/vz# df -h . Filesystem Size Used Avail Use% Mounted on /dev/mapper/pve-data 59G 58G 362M 100% /var/lib/vz

The files which are actually there:

Code:

root@de-fra1-c10:/var/lib/vz# du -shx . 24G .

There you can see that there should be more than enough free space, however it gets counted in a weird way, so it's impossible to use the whole space available.

Hi,
you don't give all infos - "du -shx" gives only the full amount of this directory, to get the subdirs is "du -shx *" the right but then you must have more directories (like images).

about the missed space - perhaps you have an running kvm-vm (as test), where you delete the disk-file? then the space is occoupied as long the kvm-precess is running but you don't see that file with "du". You can only find that file with lsof.

Udo

Hook · Nov 22, 2012

Same as before:

Code:

root@de-fra1-c10:/var/lib/vz# du -shx *
4.0K    dump
4.0K    images
4.0K    lock
16K    lost+found
23G    private
128K    root
571M    template

Code:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/pve-root   30G  855M   27G   4% /
tmpfs                 7.9G     0  7.9G   0% /lib/init/rw
udev                  7.9G  204K  7.9G   1% /dev
tmpfs                 7.9G   19M  7.9G   1% /dev/shm
/dev/mapper/pve-data   59G   43G   16G  74% /var/lib/vz
/dev/sda1             495M   35M  436M   8% /boot
/dev/fuse              30M   40K   30M   1% /etc/pve

As you can see, there's definitely something wrong there. I experienced this problem with all installations so far on different hardware configurations. I didn't notice this on HDD servers, but with a SSD this is definitely relevant.

I have the servers installed as cluster via the webinterface OR via this command:

Code:

pvectl create $slave_id_proxmox $template -cpus 2 -disk 20 -hostname $slave_id -ip_address $slave_ip -memory 4096 -nameserver 8.8.8.8 -onboot 1 -swap 0 -storage $storage

With both methods the used space is wrong. I have not even restarted the container, and no other settings have been changed as the ones you can find above.

joehopf · Nov 27, 2012

I can CONFIRM this problem!

I think I also know what is going on...

The problem does NOT occour on a Proxmox 2.0 installation. However on 2.1 each CT gests mounted separately.

The machines get mounted in to the scope of /var/lib/vz with the type simfs and are counted as used space, while they truly are only mounted.

Example of "mount":

Code:

/data/private/100403 on /var/lib/vz/root/100403 type simfs (rw,relatime)
proc on /var/lib/vz/root/100403/proc type proc (rw,relatime)
sysfs on /var/lib/vz/root/100403/sys type sysfs (rw,relatime)
none on /var/lib/vz/root/100403/proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
none on /var/lib/vz/root/100403/dev type devtmpfs (rw,relatime,size=2097152k,nr_inodes=524288)
none on /var/lib/vz/root/100403/dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
none on /var/lib/vz/root/100403/run type tmpfs (rw,nosuid,noexec,relatime,size=419432k,nr_inodes=524288,mode=755)
none on /var/lib/vz/root/100403/run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=524288)
none on /var/lib/vz/root/100403/run/shm type tmpfs (rw,relatime,size=2097152k,nr_inodes=524288)

I can also confirm the problem occurs with the default installation as well as 3 other isntallations that have been performed by independent 3rd party system engineers.

Also CTs that are on another, completely independent storage device, get mounted into the scope of /var/lib/vz and also do get counted to the used quota. So they effectively "steal" the free space of CTs on /var/lib/vz while in reality residing on a different disk!

An old installation of mine which is still Proxmox 2.0 does not have this problem. And the mount command also shows no separate mounts!
At this point I wondered how the contents of /var/lib/vz/private get accessible within /var/lib/vz./root then, so I checkt the inodes:

Code:

root@proxmox:/var/lib/vz# stat private/100 root/100
  File: `private/100'
  Size: 4096          Blocks: 8          IO Block: 4096   directory
Device: fd03h/64771d    Inode: 3448834     Links: 22
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-11-26 15:40:04.000000000 +0100
Modify: 2012-11-08 21:12:06.000000000 +0100
Change: 2012-11-08 21:12:06.000000000 +0100
  File: `root/100'
  Size: 4096          Blocks: 8          IO Block: 4096   directory
Device: 17h/23d    Inode: 3448834     Links: 22
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2012-11-26 15:40:04.000000000 +0100
Modify: 2012-11-08 21:12:06.000000000 +0100
Change: 2012-11-08 21:12:06.000000000 +0100

And surprise, surprise... the inode is the same.

It appears like Proxmox 2.0 solves the issue using hardlinks and Proxmox 2.1 moved to mounts, which produced this problem.

In my opinion this is really a critical blocking bug which already led to significant problems in my systems which have been running stable on old proxmox versions for years! Please do not ignore this.

Hook · Dec 8, 2012

It seems no one really cares about the problem, although there is definitely a problem with proxmox ...

dietmar · Dec 10, 2012

I simply cannot reproduce the problem. Use spaced is only counted one here.

Hook · Dec 11, 2012

So which further information do you need to solve the problem?
I have several servers, also with a different hardware, where the problem occures. They are all SSD servers if that's relevant.

udo · Dec 12, 2012

Hook said:
So which further information do you need to solve the problem?
I have several servers, also with a different hardware, where the problem occures. They are all SSD servers if that's relevant.

Hi,
just test on my Laptop - also an ssd-system. There is the space used only one time. Also on the other systems...

Do you see any very big files with "lsof | grep /var/lib/vz" (7. row for size)?

Udo

Hook · Dec 14, 2012

There's just too much output and all entries seem to be related to apache2 somehow ....
I have no idea what the output means, so I'll just post part of it here:

Code:

lsof: no pwd entry for UID 109
lsof: no pwd entry for UID 109
lsof: no pwd entry for UID 109
....
lsof: no pwd entry for UID 109
lsof: no pwd entry for UID 109
lsof: no pwd entry for UID 109
apache2      7045 www-data  cwd       DIR              0,167       4096      91955 /var/lib/vz/root/1156
apache2      7045 www-data  rtd       DIR              0,167       4096      91955 /var/lib/vz/root/1156
apache2      7045 www-data  txt       REG              0,167    1450994     803592 /var/lib/vz/root/1156/usr/lib/apache2/mpm-prefork/apache2
apache2      7045 www-data  mem       REG              253,0                804202 /var/lib/vz/root/1156/usr/lib/php5/20090626/pdo_mysql.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420263 /var/lib/vz/root/1156/lib/libgcc_s.so.1 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                804131 /var/lib/vz/root/1156/usr/lib/php5/20090626/pdo.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                804201 /var/lib/vz/root/1156/usr/lib/php5/20090626/mysqli.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                322053 /var/lib/vz/root/1156/usr/lib/libmysqlclient_r.so.16.0.0 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                804200 /var/lib/vz/root/1156/usr/lib/php5/20090626/mysql.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420214 /var/lib/vz/root/1156/lib/libgpg-error.so.0.4.0 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420316 /var/lib/vz/root/1156/lib/libgcrypt.so.11.5.2 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321938 /var/lib/vz/root/1156/usr/lib/libtasn1.so.3.1.7 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321952 /var/lib/vz/root/1156/usr/lib/libgnutls.so.26.14.12 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321981 /var/lib/vz/root/1156/usr/lib/libsasl2.so.2.0.23 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321979 /var/lib/vz/root/1156/usr/lib/libldap_r-2.4.so.2.5.4 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321953 /var/lib/vz/root/1156/usr/lib/liblber-2.4.so.2.5.4 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321923 /var/lib/vz/root/1156/usr/lib/libidn.so.11.5.44 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                322056 /var/lib/vz/root/1156/usr/lib/libcurl.so.4.1.1 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                804196 /var/lib/vz/root/1156/usr/lib/php5/20090626/curl.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803645 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_status.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803611 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_ssl.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803610 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_setenvif.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803628 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_rewrite.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803655 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_reqtimeout.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803651 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_proxy_connect.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803661 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_proxy.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420215 /var/lib/vz/root/1156/lib/libkeyutils-1.2.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321975 /var/lib/vz/root/1156/usr/lib/libkrb5support.so.0.1 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420412 /var/lib/vz/root/1156/lib/libcrypto.so.0.9.8 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                322007 /var/lib/vz/root/1156/usr/lib/libxml2.so.2.7.6 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420275 /var/lib/vz/root/1156/lib/libcom_err.so.2.1 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321974 /var/lib/vz/root/1156/usr/lib/libk5crypto.so.3.1 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321915 /var/lib/vz/root/1156/usr/lib/libkrb5.so.3.3 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321995 /var/lib/vz/root/1156/usr/lib/libgssapi_krb5.so.2.2 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420223 /var/lib/vz/root/1156/lib/libbz2.so.1.0.4 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                321988 /var/lib/vz/root/1156/usr/lib/libdb-4.8.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420413 /var/lib/vz/root/1156/lib/libssl.so.0.9.8 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                804173 /var/lib/vz/root/1156/usr/lib/apache2/modules/libphp5.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803638 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_negotiation.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803599 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_mime.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803643 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_headers.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803625 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_env.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420210 /var/lib/vz/root/1156/lib/libresolv-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420245 /var/lib/vz/root/1156/lib/libnss_dns-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420244 /var/lib/vz/root/1156/lib/libnss_files-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420301 /var/lib/vz/root/1156/lib/libnss_nis-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420262 /var/lib/vz/root/1156/lib/libnsl-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420330 /var/lib/vz/root/1156/lib/libnss_compat-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803612 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_dir.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803654 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_deflate.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803614 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_cgi.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803648 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_autoindex.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803606 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authz_user.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803604 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authz_host.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803601 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authz_groupfile.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803663 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authz_default.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803381 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authnz_external.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803650 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authn_file.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420311 /var/lib/vz/root/1156/lib/libz.so.1.2.3.3 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420319 /var/lib/vz/root/1156/lib/libm-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                322052 /var/lib/vz/root/1156/usr/lib/libmysqlclient.so.16.0.0 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                804118 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_auth_mysql.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803619 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_auth_basic.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803618 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_alias.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420408 /var/lib/vz/root/1156/lib/libexpat.so.1.5.2 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420218 /var/lib/vz/root/1156/lib/libdl-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420276 /var/lib/vz/root/1156/lib/libcrypt-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420256 /var/lib/vz/root/1156/lib/librt-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420323 /var/lib/vz/root/1156/lib/libuuid.so.1.3.0 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420287 /var/lib/vz/root/1156/lib/libc-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420261 /var/lib/vz/root/1156/lib/libpthread-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                322009 /var/lib/vz/root/1156/usr/lib/libapr-1.so.0.3.8 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                322011 /var/lib/vz/root/1156/usr/lib/libaprutil-1.so.0.3.9 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420238 /var/lib/vz/root/1156/lib/libpcre.so.3.12.1 (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                420304 /var/lib/vz/root/1156/lib/ld-2.11.1.so (path dev=0,167)
apache2      7045 www-data  mem       REG              253,0                803592 /var/lib/vz/root/1156/usr/lib/apache2/mpm-prefork/apache2 (path dev=0,167)
apache2      7045 www-data    0r      CHR                1,3        0t0     288401 /var/lib/vz/root/1156/dev/null
apache2      7045 www-data    1w      CHR                1,3        0t0     288401 /var/lib/vz/root/1156/dev/null
apache2      7045 www-data    2w      REG              0,167     114688     802864 /var/lib/vz/root/1156/var/log/apache2/error.log
apache2      7045 www-data    7w      REG              0,167      32768     802992 /var/lib/vz/root/1156/var/log/apache2/other_vhosts_access.log
apache2      7045 www-data    8w      REG              0,167          0     803014 /var/lib/vz/root/1156/var/log/apache2/dummy.log
apache2      7045 www-data    9w      REG              0,167    5125184     802859 /var/lib/vz/root/1156/var/log/apache2/access.log
apache2      7045 www-data   10w      REG              0,174          0  235577639  (deleted)/var/lib/vz/root/1156/var/run/apache2/ssl_mutex
apache2      8607 www-data  rtd       DIR              0,167       4096      91955 /var/lib/vz/root/1156
apache2      8607 www-data  txt       REG              0,167    1450994     803592 /var/lib/vz/root/1156/usr/lib/apache2/mpm-prefork/apache2
apache2      8607 www-data  mem       REG              253,0                804202 /var/lib/vz/root/1156/usr/lib/php5/20090626/pdo_mysql.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420263 /var/lib/vz/root/1156/lib/libgcc_s.so.1 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                804131 /var/lib/vz/root/1156/usr/lib/php5/20090626/pdo.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                804201 /var/lib/vz/root/1156/usr/lib/php5/20090626/mysqli.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                322053 /var/lib/vz/root/1156/usr/lib/libmysqlclient_r.so.16.0.0 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                804200 /var/lib/vz/root/1156/usr/lib/php5/20090626/mysql.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420214 /var/lib/vz/root/1156/lib/libgpg-error.so.0.4.0 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420316 /var/lib/vz/root/1156/lib/libgcrypt.so.11.5.2 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321938 /var/lib/vz/root/1156/usr/lib/libtasn1.so.3.1.7 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321952 /var/lib/vz/root/1156/usr/lib/libgnutls.so.26.14.12 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321981 /var/lib/vz/root/1156/usr/lib/libsasl2.so.2.0.23 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321979 /var/lib/vz/root/1156/usr/lib/libldap_r-2.4.so.2.5.4 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321953 /var/lib/vz/root/1156/usr/lib/liblber-2.4.so.2.5.4 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321923 /var/lib/vz/root/1156/usr/lib/libidn.so.11.5.44 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                322056 /var/lib/vz/root/1156/usr/lib/libcurl.so.4.1.1 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                804196 /var/lib/vz/root/1156/usr/lib/php5/20090626/curl.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803645 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_status.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803611 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_ssl.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803610 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_setenvif.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803628 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_rewrite.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803655 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_reqtimeout.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803651 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_proxy_connect.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803661 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_proxy.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420215 /var/lib/vz/root/1156/lib/libkeyutils-1.2.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321975 /var/lib/vz/root/1156/usr/lib/libkrb5support.so.0.1 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420412 /var/lib/vz/root/1156/lib/libcrypto.so.0.9.8 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                322007 /var/lib/vz/root/1156/usr/lib/libxml2.so.2.7.6 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420275 /var/lib/vz/root/1156/lib/libcom_err.so.2.1 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321974 /var/lib/vz/root/1156/usr/lib/libk5crypto.so.3.1 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321915 /var/lib/vz/root/1156/usr/lib/libkrb5.so.3.3 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321995 /var/lib/vz/root/1156/usr/lib/libgssapi_krb5.so.2.2 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420223 /var/lib/vz/root/1156/lib/libbz2.so.1.0.4 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                321988 /var/lib/vz/root/1156/usr/lib/libdb-4.8.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420413 /var/lib/vz/root/1156/lib/libssl.so.0.9.8 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                804173 /var/lib/vz/root/1156/usr/lib/apache2/modules/libphp5.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803638 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_negotiation.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803599 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_mime.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803643 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_headers.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803625 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_env.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420210 /var/lib/vz/root/1156/lib/libresolv-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420245 /var/lib/vz/root/1156/lib/libnss_dns-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420244 /var/lib/vz/root/1156/lib/libnss_files-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420301 /var/lib/vz/root/1156/lib/libnss_nis-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420262 /var/lib/vz/root/1156/lib/libnsl-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420330 /var/lib/vz/root/1156/lib/libnss_compat-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803612 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_dir.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803654 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_deflate.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803614 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_cgi.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803648 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_autoindex.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803606 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authz_user.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803604 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authz_host.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803601 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authz_groupfile.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803663 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authz_default.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803381 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authnz_external.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803650 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_authn_file.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420311 /var/lib/vz/root/1156/lib/libz.so.1.2.3.3 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420319 /var/lib/vz/root/1156/lib/libm-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                322052 /var/lib/vz/root/1156/usr/lib/libmysqlclient.so.16.0.0 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                804118 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_auth_mysql.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803619 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_auth_basic.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803618 /var/lib/vz/root/1156/usr/lib/apache2/modules/mod_alias.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420408 /var/lib/vz/root/1156/lib/libexpat.so.1.5.2 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420218 /var/lib/vz/root/1156/lib/libdl-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420276 /var/lib/vz/root/1156/lib/libcrypt-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420256 /var/lib/vz/root/1156/lib/librt-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420323 /var/lib/vz/root/1156/lib/libuuid.so.1.3.0 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420287 /var/lib/vz/root/1156/lib/libc-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420261 /var/lib/vz/root/1156/lib/libpthread-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                322009 /var/lib/vz/root/1156/usr/lib/libapr-1.so.0.3.8 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                322011 /var/lib/vz/root/1156/usr/lib/libaprutil-1.so.0.3.9 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420238 /var/lib/vz/root/1156/lib/libpcre.so.3.12.1 (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                420304 /var/lib/vz/root/1156/lib/ld-2.11.1.so (path dev=0,167)
apache2      8607 www-data  mem       REG              253,0                803592 /var/lib/vz/root/1156/usr/lib/apache2/mpm-prefork/apache2 (path dev=0,167)
apache2      8607 www-data    0r      CHR                1,3        0t0     288401 /var/lib/vz/root/1156/dev/null
apache2      8607 www-data    1w      CHR                1,3        0t0     288401 /var/lib/vz/root/1156/dev/null
apache2      8607 www-data    2w      REG              0,167    1309634     803146  (deleted)/var/lib/vz/root/1156/var/log/apache2/error.log.1
apache2      8607 www-data    7w      REG              0,167     577410     803191  (deleted)/var/lib/vz/root/1156/var/log/apache2/other_vhosts_access.log.1
apache2      8607 www-data    8w      REG              0,167   14994924     803088  (deleted)/var/lib/vz/root/1156/var/log/apache2/access.log.1
apache2      8607 www-data    9w      REG              0,167  135921683     803234  (deleted)/var/lib/vz/root/1156/var/log/apache2/rewrite.log.1
apache2      8607 www-data   10w      REG              0,174          0  235577639  (deleted)/var/lib/vz/root/1156/var/run/apache2/ssl_mutex
apache2      8882     root  cwd       DIR               0,68       4096    6268742 /var/lib/vz/root/1146
apache2      8882     root  rtd       DIR               0,68       4096    6268742 /var/lib/vz/root/1146
apache2      8882     root  txt       REG               0,68    1450994     131074 /var/lib/vz/root/1146/usr/lib/apache2/mpm-prefork/apache2
apache2      8882     root  mem       REG              253,0                131681 /var/lib/vz/root/1146/usr/lib/php5/20090626/pdo_mysql.so (path dev=0,68)
apache2      8882     root  mem       REG              253,0                131610 /var/lib/vz/root/1146/usr/lib/php5/20090626/pdo.so (path dev=0,68)
apache2      8882     root  mem       REG              253,0               6457504 /var/lib/vz/root/1146/lib/libgcc_s.so.1 (path dev=0,68)
apache2      8882     root  mem       REG              253,0                131680 /var/lib/vz/root/1146/usr/lib/php5/20090626/mysqli.so (path dev=0,68)
apache2      8882     root  mem       REG              253,0               6506935 /var/lib/vz/root/1146/usr/lib/libmysqlclient_r.so.16.0.0 (path dev=0,68)

I'm using the OpenVZ image ubuntu-12.04-x86_64.tar.gz which is avaliable here: http://wiki.openvz.org/Download/template/precreated

This is the output of "du -sh" and "df -h" on the virtual machine itself:

Code:

root@taurus:/# du -sh
du: cannot access `./proc/3635/task/3635/fd/4': No such file or directory
du: cannot access `./proc/3635/task/3635/fdinfo/4': No such file or directory
du: cannot access `./proc/3635/fd/4': No such file or directory
du: cannot access `./proc/3635/fdinfo/4': No such file or directory
2.4G    .
root@taurus:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/simfs       13G  6.7G  477M  94% /
none            5.0G  8.0K  5.0G   1% /dev
none            1.0G  1.1M 1023M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            5.0G     0  5.0G   0% /run/shm

Thanks in advance!

udo · Dec 14, 2012

Hook said:

There's just too much output and all entries seem to be related to apache2 somehow ....
I have no idea what the output means, so I'll just post part of it here:

Code:

....
apache2      8607 www-data    2w      REG              0,167    1309634     803146  (deleted)/var/lib/vz/root/1156/var/log/apache2/error.log.1
apache2      8607 www-data    7w      REG              0,167     577410     803191  (deleted)/var/lib/vz/root/1156/var/log/apache2/other_vhosts_access.log.1
apache2      8607 www-data    8w      REG              0,167   14994924     803088  (deleted)/var/lib/vz/root/1156/var/log/apache2/access.log.1
apache2      8607 www-data    9w      REG              0,167  135921683     803234  (deleted)/var/lib/vz/root/1156/var/log/apache2/rewrite.log.1
apache2      8607 www-data   10w      REG              0,174          0  235577639  (deleted)/var/lib/vz/root/1156/var/run/apache2/ssl_mutex

...

Hi,
this are the "interesting" files - they are delete (means "du" don't calc them) but still in use (means "df" calc them). If the process stops (apache2) the files are gone.
For this files it's only 150MB but perhaps there are more?
Try with

Code:

lsof | grep var/lib/vz | grep deleted

Udo

Hook · Dec 15, 2012

Thank you, Udo!
There are MANY files tagged with (deleted) ...
I'll write a script soon to parse the output, to see how much this really is ... But at least i know where all the disk space is gone to

Is there a way to get rid of these files without closing the apache2 process?

udo · Dec 15, 2012

Hook said:
...
Is there a way to get rid of these files without closing the apache2 process?

No,
except you convince the apache-process to use an new logfile (as example) and all subprocesses are use the new file too.

Perhaps you have only too much "old" subprocesses?

If you use logrototate or anything else, perhaps you should try an "kill -sighup" at this time?

Udo

Search

Search

HDD space gets counted twice

Hook

Renowned Member

tom

Proxmox Staff Member

Hook

Renowned Member

udo

Distinguished Member

Hook

Renowned Member

joehopf

Guest

Hook

Renowned Member

dietmar

Proxmox Staff Member

Hook

Renowned Member

udo

Distinguished Member

Hook

Renowned Member

udo

Distinguished Member

Hook

Renowned Member

udo

Distinguished Member