URGENT MY server is down:::: Local disk is full

Ole

Member
May 17, 2013
41
0
6
Hi my local proxmox disk is full and I am trying to add size on it but I am not able to add the size....

I do not find my disk under sys folder but I find it under the mapper folder...

so when I try to use the command.. lvresize -L-10G /dev/mapper/vg0-root


but I am not able to increase the size..

what should I do to be able to re-size it..

(I know that I can re-size the size of the VM disk in the webadmin but that dosent help mutch if the local disk is smaller)

Thx

Ole
 
Hi my local proxmox disk is full and I am trying to add size on it but I am not able to add the size....

I do not find my disk under sys folder but I find it under the mapper folder...

so when I try to use the command.. lvresize -L-10G/dev/mapper/vg0-root


but I am not able to increase the size..

what should I do to be able to re-size it..

(I know that I can re-size the size of the VM disk in the webadmin but that dosent help mutch if the local disk is smaller)

Thx

Ole
Hi Ole,
you should find the reason, why your disk ist full.
Take a look in /var/log - perhaps there are a very huge logfile? (move to an other partition - restart syslog).
Or your Backup-files are on the root-volume??
If you have a little bit space (due removing logfiles), you can intall ncdu (apt-get install ncdu) which is very usefull to find the "wasted" space.
Code:
ncdu -x /
Udo
 
Hi that helped in some way I deleted som files

But now I get io error so server still don't start.

Thx.
 
Hi that helped in some way I deleted som files

But now I get io error so server still don't start.

Thx.

Hi,
your error message is very poor!

IO-Error? From which disks? Any hardwareproblem (check logfiles)?

IO-Error from a VM-disk (lvm)? What shows "dmsetup ls --tree" ?

Also after reboot?

Are there active snapshots? Look with "lvs".


Udo
 
Hi Udo,

Please check the screen attached. This is screen shot of Ole servers. We want to increase the space which is filed up now. We have 300 gb hdd but cant able to increase size of this 14 GB , Ican provide more screen shots if needed.

Please help :)Firefox20130708202519230.pngFirefox20130708202531718.pngFirefox20130708202130735.png
 
you have to resize the partition physically if the LVM physical disk is limited to 15GB first - a quick and dirty fix if the rest of the disk space is unpartitioned - create a new ext3/4 partition and mount it as something like /data then move/copy /var/lib/vz to /data/vz and then symlink /var/lib/vz to /data/vz

it does sound to me like one of your partitions created at setup never remounted properly after boot and proxmox has been writing to /var/lib/vz on the first root partition instead of the one it normally creates during install
 
Last edited:
Hi Udo,

Please check the screen attached. This is screen shot of Ole servers. We want to increase the space which is filed up now. We have 300 gb hdd but cant able to increase size of this 14 GB , Ican provide more screen shots if needed.

Please help :)
Hi,
please give output from the commandline (ssh to the host):
Code:
fdisk -l
pvs
vgs
lvs
mount
Udo
 
Please help

sorry, what is your relation with the original poster?
how about those i/o errors Ole cited before?

how can you keep asking for help by volunteers not replying with the right info they asked..?
helping on an unknown server is really difficult for anyone:
- if you need help, please provide the more details you can about your system,
- if you get errors from your system, please specify exactly where and when those error appeared,
- if someone (like udo above) asks you to report the output of some files or to do some simple test, please do what they ask, because it's needed to understand _your_ problems.

my 2c,
Marco
 
Hi Udo,

Here is output for the commands,hope this will help to track the issue


fdisk -l
root@Proxmox-VE ~ # fdisk -l

Disk /dev/sda: 750.2 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00004727

Device Boot Start End Blocks Id System
/dev/sda1 1 66 524288+ fd Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2 66 91202 732048123+ fd Linux raid autodetect

Disk /dev/sdb: 750.2 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000617e8

Device Boot Start End Blocks Id System
/dev/sdb1 1 66 524288+ fd Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sdb2 66 91202 732048123+ fd Linux raid autodetect

Disk /dev/md0: 536 MB, 536805376 bytes
2 heads, 4 sectors/track, 131056 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 749.6 GB, 749617152000 bytes
2 heads, 4 sectors/track, 183012000 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/dm-0: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-1 doesn't contain a valid partition table


pvs
PV VG Fmt Attr PSize PFree
/dev/md1 vg0 lvm2 a-- 698.13g 677.13g


vgs
root@Proxmox-VE ~ # vgs
VG #PV #LV #SN Attr VSize VFree
vg0 1 2 0 wz--n- 698.13g 677.13g


lvs
root@Proxmox-VE ~ # lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
root vg0 -wi-ao-- 15.00g
swap vg0 -wi-ao-- 6.00g

mount
root@Proxmox-VE ~ # mount
/dev/mapper/vg0-root on / type ext3 (rw)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/md0 on /boot type ext3 (rw)
overflow on /tmp type tmpfs (rw,size=1048576,mode=1777)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,default_permissions,allow_other)
beancounter on /proc/vz/beancounter type cgroup (rw,name=beancounter)
container on /proc/vz/container type cgroup (rw,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,name=fairsched)
 
Hi Udo,

Here is output for the commands,hope this will help to track the issue


...

pvs
PV VG Fmt Attr PSize PFree
/dev/md1 vg0 lvm2 a-- 698.13g 677.13g


vgs
root@Proxmox-VE ~ # vgs
VG #PV #LV #SN Attr VSize VFree
vg0 1 2 0 wz--n- 698.13g 677.13g


lvs
root@Proxmox-VE ~ # lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
root vg0 -wi-ao-- 15.00g
swap vg0 -wi-ao-- 6.00g

...
Hi,
you have plenty free space inside your VG.
Simply extend your logical volume:
Code:
lvextend -L+20G /dev/vg0/root
resize2fs /dev/vg0/root
After that, you should think about an exte LV for the data (not all on root).

Udo
 
Hi Udo

I receieved following msg


root@Proxmox-VE ~ # lvextend -L+20G /dev/vg0/root
/etc/lvm/archive: mkdir failed: No space left on device
 
Hi,
you have plenty free space inside your VG.
Simply extend your logical volume:
Code:
lvextend -L+20G /dev/vg0/root
resize2fs /dev/vg0/root
After that, you should think about an exte LV for the data (not all on root).

Udo

Hi Udo.

Thanks for all the help so faar!!

yes I think making a extended LV for data is very wise. Are there any docs on how to do this and how to move the vm to this disk?

thx

Ole
 
Hi Udo,

Thanks for ur help. Ur suggestion worked :) server is up again.

BTW , I see that our server space is filling up very fast , is there any way we can trace out reason behind this ?
 
Hi Udo.

Thanks for all the help so faar!!

yes I think making a extended LV for data is very wise. Are there any docs on how to do this and how to move the vm to this disk?

thx

Ole
Hi Ole,
there are plenty docs in the web about lvm and so on.
In short (see manpages to know what you are doing):
Code:
lvcreate -L 100G -n data /dev/vg0
mkfs.ext3 /dev/vg0/data
# shutdown all VMs and CTs
echo "/dev/vg0/data /var/lib/vz ext3 defaults 0 1" >> /etc/fstab
mv /var/lib/vz /var/lib/vz.org; mkdir /var/lib/vz; mount /var/lib/vz

cd /var/lib/vz.org
rsync -av . /var/lib/vz/

# control with 
df -h

# after that start all VMs - if all is running (also a reboot) you can remove /var/lib/vz.org
Udo
 
Hi Ole,
there are plenty docs in the web about lvm and so on.
In short (see manpages to know what you are doing):
Code:
lvcreate -L 100G -n data /dev/vg0
mkfs.ext3 /dev/vg0/data
# shutdown all VMs and CTs
echo "/dev/vg0/data /var/lib/vz ext3 defaults 0 1" >> /etc/fstab
mv /var/lib/vz /var/lib/vz.org; mkdir /var/lib/vz; mount /var/lib/vz

cd /var/lib/vz.org
rsync -av . /var/lib/vz/

# control with 
df -h

# after that start all VMs - if all is running (also a reboot) you can remove /var/lib/vz.org
Udo

Hi Udo

Thanks for the help.
I have tried ot study your example so I cna understand what happends :)
when I ran hte rsync for moving the vm it has taken some time..aroudn 15 min now and I am unsure what to do.

Can it be an issue with the rsync command? Any thought what to do to figure out this?

I have natually not been able to control things yet wiht df -h

Ole
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!