Disk full on / - move X to Y?

FuriousRage

Renowned Member
Oct 17, 2014
120
4
83
Hi, when i reinstalled my proxmox to get it working. I followed someones command that gave the disk around 10 GB only.. today when i got home, i got loads of errors due to that disk was full.
i Ran apt-get clean and i got around 120MB more free:d (still like 98% full)..


Code:
/dev/fuse              30M   16K   30M   1% /etc/pveroot@Caelus:/# du -ax --max-depth=3 / | sort -n | awk '{if($1 > 102400) print $1/1024 "MB" " " $2 }'
106.926MB /lib/modules/2.6.32-32-pve
106.93MB /lib/modules
125.445MB /usr/bin
132.203MB /var
165.723MB /lib
218.043MB /usr/lib
253.891MB /usr/share
664.152MB /usr
981.985MB /
What i was thinking.. to move /usr into one of the storage nodes that has terabytes of disk space..

How would i go about to do this the best and simplest way?


Additional information that might be usefull(?)
Code:
root@Caelus:/# df -hFilesystem            Size  Used Avail Use% Mounted on
udev                   10M     0   10M   0% /dev
tmpfs                 1.6G  552K  1.6G   1% /run
/dev/mapper/pve-root  9.9G  9.2G  192M  98% /
tmpfs                 5.0M  4.0K  5.0M   1% /run/lock
tmpfs                 3.2G   28M  3.1G   1% /run/shm
/dev/mapper/pve-data   67G  8.3G   59G  13% /var/lib/vz
/dev/sda2             494M   36M  433M   8% /boot
/dev/md0              7.3T  560G  6.7T   8% /storage
tmpfs                 7.2G     0  7.2G   0% /tmp
/dev/fuse              30M   16K   30M   1% /etc/pve
Code:
root@Caelus:/# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=2035237,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1630212k,mode=755)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro,barrier=1,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3260400k)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/mapper/pve-data on /var/lib/vz type ext4 (rw,relatime,barrier=1,data=ordered)
/dev/sda2 on /boot type ext4 (rw,relatime,barrier=1,data=ordered)
/dev/md0 on /storage type ext4 (rw,relatime,barrier=0,stripe=512,data=ordered)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,size=7454720k)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
beancounter on /proc/vz/beancounter type cgroup (rw,relatime,blkio,name=beancounter)
container on /proc/vz/container type cgroup (rw,relatime,freezer,devices,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,relatime,cpuacct,cpu,cpuset,name=fairsched)
 
Hi, when i reinstalled my proxmox to get it working. I followed someones command that gave the disk around 10 GB only.. today when i got home, i got loads of errors due to that disk was full.
i Ran apt-get clean and i got around 120MB more free:d (still like 98% full)..


Code:
/dev/fuse              30M   16K   30M   1% /etc/pveroot@Caelus:/# du -ax --max-depth=3 / | sort -n | awk '{if($1 > 102400) print $1/1024 "MB" " " $2 }'
106.926MB /lib/modules/2.6.32-32-pve
106.93MB /lib/modules
125.445MB /usr/bin
132.203MB /var
165.723MB /lib
218.043MB /usr/lib
253.891MB /usr/share
664.152MB /usr
981.985MB /
What i was thinking.. to move /usr into one of the storage nodes that has terabytes of disk space..

How would i go about to do this the best and simplest way?


Additional information that might be usefull(?)
Code:
root@Caelus:/# df -hFilesystem            Size  Used Avail Use% Mounted on
udev                   10M     0   10M   0% /dev
tmpfs                 1.6G  552K  1.6G   1% /run
/dev/mapper/pve-root  9.9G  9.2G  192M  98% /
tmpfs                 5.0M  4.0K  5.0M   1% /run/lock
tmpfs                 3.2G   28M  3.1G   1% /run/shm
/dev/mapper/pve-data   67G  8.3G   59G  13% /var/lib/vz
/dev/sda2             494M   36M  433M   8% /boot
/dev/md0              7.3T  560G  6.7T   8% /storage
tmpfs                 7.2G     0  7.2G   0% /tmp
/dev/fuse              30M   16K   30M   1% /etc/pve
Code:
root@Caelus:/# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=2035237,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1630212k,mode=755)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro,barrier=1,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3260400k)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/mapper/pve-data on /var/lib/vz type ext4 (rw,relatime,barrier=1,data=ordered)
/dev/sda2 on /boot type ext4 (rw,relatime,barrier=1,data=ordered)
/dev/md0 on /storage type ext4 (rw,relatime,barrier=0,stripe=512,data=ordered)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,size=7454720k)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
beancounter on /proc/vz/beancounter type cgroup (rw,relatime,blkio,name=beancounter)
container on /proc/vz/container type cgroup (rw,relatime,freezer,devices,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,relatime,cpuacct,cpu,cpuset,name=fairsched)
Hi,
for that you have lvm!
If you have enough free space in the volume group you can easy expand your pve-root and the filesystem.
Look with
Code:
pvs
vgs
lvs
But 10GB is not too small - look with ncdu where your big files are.
Code:
apt-get install ncdu
ncdu -x /
Udo
 
root@Caelus:/# pvs && vgs && lvs
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 111.29g 13.87g
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- 111.29g 13.87g
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
data pve -wi-ao--- 67.42g
root pve -wi-ao--- 10.00g
swap pve -wi-ao--- 20.00g
root@Caelus:/#

Hi,
for that you have lvm!
If you have enough free space in the volume group you can easy expand your pve-root and the filesystem.
Look with
Code:
pvs
vgs
lvs
But 10GB is not too small - look with ncdu where your big files are.
Code:
apt-get install ncdu
ncdu -x /
Udo

ncdu 1.8 ~ Use the arrow keys to navigate, press ? for help
--- / ----------------------------------------------------------------------------------------------------------------------------------------------
664.2MiB [##########] /usr
179.3MiB [## ] /var
165.7MiB [## ] /lib
8.7MiB [ ] /sbin
6.5MiB [ ] /bin
4.5MiB [ ] /etc
76.0KiB [ ] /root
28.0KiB [ ] /.gnupg
e 16.0KiB [ ] /lost+found
4.0KiB [ ] /lib64
e 4.0KiB [ ] /srv
e 4.0KiB [ ] /selinux
e 4.0KiB [ ] /opt
e 4.0KiB [ ] /mnt
e 4.0KiB [ ] /media
e 4.0KiB [ ] /home
4.0KiB [ ] apctest.output
4.0KiB [ ] .bash_history
> 0.0 B [ ] /tmp
> 0.0 B [ ] /sys
> 0.0 B [ ] /storage
> 0.0 B [ ] /run
> 0.0 B [ ] /proc
> 0.0 B [ ] /dev
> 0.0 B [ ] /boot
 
root@Caelus:/# pvs && vgs && lvs
PV VG Fmt Attr PSize PFree
/dev/sda3 pve lvm2 a-- 111.29g 13.87g
VG #PV #LV #SN Attr VSize VFree
pve 1 3 0 wz--n- 111.29g 13.87g
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
data pve -wi-ao--- 67.42g
root pve -wi-ao--- 10.00g
swap pve -wi-ao--- 20.00g
root@Caelus:/#



ncdu 1.8 ~ Use the arrow keys to navigate, press ? for help
--- / ----------------------------------------------------------------------------------------------------------------------------------------------
664.2MiB [##########] /usr
179.3MiB [## ] /var
165.7MiB [## ] /lib
8.7MiB [ ] /sbin
6.5MiB [ ] /bin
4.5MiB [ ] /etc
76.0KiB [ ] /root
28.0KiB [ ] /.gnupg
e 16.0KiB [ ] /lost+found
4.0KiB [ ] /lib64
e 4.0KiB [ ] /srv
e 4.0KiB [ ] /selinux
e 4.0KiB [ ] /opt
e 4.0KiB [ ] /mnt
e 4.0KiB [ ] /media
e 4.0KiB [ ] /home
4.0KiB [ ] apctest.output
4.0KiB [ ] .bash_history
> 0.0 B [ ] /tmp
> 0.0 B [ ] /sys
> 0.0 B [ ] /storage
> 0.0 B [ ] /run
> 0.0 B [ ] /proc
> 0.0 B [ ] /dev
> 0.0 B [ ] /boot
Hi,
this is round 1GB not 10GB!!
Perhaps there is an open but deleted big file, or you have an big directory, where you have mounted another share over - so you don't see the underlaying files.

For the first look with
Code:
lsof | grep deleted
another way is to use addition space (like 2GB) from VG pve (be sure to have an valid backup):
Code:
lvextend -L+2G /dev/pve/root
resize2fs /dev/mapper/pve-root
But leaf space in the VG free - if you want to take an snapshot.

IMHO swap is very big - I would shrink swap to 8 - 16GB.

Udo
 
I expanded the root drive with that command and now i atleast have 20% or so free. :D
thank you.

Hi,
this is round 1GB not 10GB!!
Perhaps there is an open but deleted big file, or you have an big directory, where you have mounted another share over - so you don't see the underlaying files.

For the first look with
Code:
lsof | grep deleted
another way is to use addition space (like 2GB) from VG pve (be sure to have an valid backup):
Code:
lvextend -L+2G /dev/pve/root
resize2fs /dev/mapper/pve-root
But leaf space in the VG free - if you want to take an snapshot.

IMHO swap is very big - I would shrink swap to 8 - 16GB.

Udo