URGENT MY server is down:::: Local disk is full

[Ole]

Hi my local proxmox disk is full and I am trying to add size on it but I am not able to add the size....

I do not find my disk under sys folder but I find it under the mapper folder...

so when I try to use the command.. lvresize -L-10G /dev/mapper/vg0-root


but I am not able to increase the size..

what should I do to be able to re-size it..

(I know that I can re-size the size of the VM disk in the webadmin but that dosent help mutch if the local disk is smaller)

Thx

Ole

 

[udo]

Hi my local proxmox disk is full and I am trying to add size on it but I am not able to add the size....

I do not find my disk under sys folder but I find it under the mapper folder...

so when I try to use the command.. lvresize -L-10G/dev/mapper/vg0-root


but I am not able to increase the size..

what should I do to be able to re-size it..

(I know that I can re-size the size of the VM disk in the webadmin but that dosent help mutch if the local disk is smaller)

Thx

Ole

Hi Ole,
you should find the reason, why your disk ist full.
Take a look in /var/log - perhaps there are a very huge logfile? (move to an other partition - restart syslog).
Or your Backup-files are on the root-volume??
If you have a little bit space (due removing logfiles), you can intall ncdu (apt-get install ncdu) which is very usefull to find the "wasted" space.

ncdu -x /

Udo

 

[Ole]

Hi that helped in some way I deleted som files

But now I get io error so server still don't start.

Thx.

 

[udo]

Hi that helped in some way I deleted som files

But now I get io error so server still don't start.

Thx.

Hi,
your error message is very poor!

IO-Error? From which disks? Any hardwareproblem (check logfiles)?

IO-Error from a VM-disk (lvm)? What shows "dmsetup ls --tree" ?

Also after reboot?

Are there active snapshots? Look with "lvs".


Udo

 

[veenaay]

Hi Udo,

Please check the screen attached. This is screen shot of Ole servers. We want to increase the space which is filed up now. We have 300 gb hdd but cant able to increase size of this 14 GB , Ican provide more screen shots if needed.

Please help

 

[anthonysomerset]

you have to resize the partition physically if the LVM physical disk is limited to 15GB first - a quick and dirty fix if the rest of the disk space is unpartitioned - create a new ext3/4 partition and mount it as something like /data then move/copy /var/lib/vz to /data/vz and then symlink /var/lib/vz to /data/vz

it does sound to me like one of your partitions created at setup never remounted properly after boot and proxmox has been writing to /var/lib/vz on the first root partition instead of the one it normally creates during install

 

[udo]

Hi Udo,

Please check the screen attached. This is screen shot of Ole servers. We want to increase the space which is filed up now. We have 300 gb hdd but cant able to increase size of this 14 GB , Ican provide more screen shots if needed.

Please help

Hi,
please give output from the commandline (ssh to the host):

fdisk -l
pvs
vgs
lvs
mount

Udo

 

[m.ardito]

Please help

sorry, what is your relation with the original poster?
how about those i/o errors Ole cited before?

how can you keep asking for help by volunteers not replying with the right info they asked..?
helping on an unknown server is really difficult for anyone:
- if you need help, please provide the more details you can about your system,
- if you get errors from your system, please specify exactly where and when those error appeared,
- if someone (like udo above) asks you to report the output of some files or to do some simple test, please do what they ask, because it's needed to understand _your_ problems.

my 2c,
Marco

 

[veenaay]

Hi Udo,

Here is output for the commands,hope this will help to track the issue


fdisk -l
root@Proxmox-VE ~ # fdisk -l

Disk /dev/sda: 750.2 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00004727

Device Boot Start End Blocks Id System
/dev/sda1 1 66 524288+ fd Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2 66 91202 732048123+ fd Linux raid autodetect

Disk /dev/sdb: 750.2 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000617e8

Device Boot Start End Blocks Id System
/dev/sdb1 1 66 524288+ fd Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sdb2 66 91202 732048123+ fd Linux raid autodetect

Disk /dev/md0: 536 MB, 536805376 bytes
2 heads, 4 sectors/track, 131056 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

Disk /dev/md1: 749.6 GB, 749617152000 bytes
2 heads, 4 sectors/track, 183012000 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/dm-0: 16.1 GB, 16106127360 bytes
255 heads, 63 sectors/track, 1958 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 6442 MB, 6442450944 bytes
255 heads, 63 sectors/track, 783 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/dm-1 doesn't contain a valid partition table


pvs
PV VG Fmt Attr PSize PFree
/dev/md1 vg0 lvm2 a-- 698.13g 677.13g


vgs
root@Proxmox-VE ~ # vgs
VG #PV #LV #SN Attr VSize VFree
vg0 1 2 0 wz--n- 698.13g 677.13g


lvs
root@Proxmox-VE ~ # lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
root vg0 -wi-ao-- 15.00g
swap vg0 -wi-ao-- 6.00g

mount
root@Proxmox-VE ~ # mount
/dev/mapper/vg0-root on / type ext3 (rw)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/md0 on /boot type ext3 (rw)
overflow on /tmp type tmpfs (rw,size=1048576,mode=1777)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,default_permissions,allow_other)
beancounter on /proc/vz/beancounter type cgroup (rw,name=beancounter)
container on /proc/vz/container type cgroup (rw,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,name=fairsched)

 

[udo]

Hi Udo,

Here is output for the commands,hope this will help to track the issue


...

pvs
PV VG Fmt Attr PSize PFree
/dev/md1 vg0 lvm2 a-- 698.13g 677.13g


vgs
root@Proxmox-VE ~ # vgs
VG #PV #LV #SN Attr VSize VFree
vg0 1 2 0 wz--n- 698.13g 677.13g


lvs
root@Proxmox-VE ~ # lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
root vg0 -wi-ao-- 15.00g
swap vg0 -wi-ao-- 6.00g

...

Hi,
you have plenty free space inside your VG.
Simply extend your logical volume:

lvextend -L+20G /dev/vg0/root
resize2fs /dev/vg0/root

After that, you should think about an exte LV for the data (not all on root).

Udo

 

[veenaay]

Hi Udo

I receieved following msg


root@Proxmox-VE ~ # lvextend -L+20G /dev/vg0/root
/etc/lvm/archive: mkdir failed: No space left on device

 

[Ole]

Hi,
you have plenty free space inside your VG.
Simply extend your logical volume:

lvextend -L+20G /dev/vg0/root
resize2fs /dev/vg0/root

After that, you should think about an exte LV for the data (not all on root).

Udo

Hi Udo.

Thanks for all the help so faar!!

yes I think making a extended LV for data is very wise. Are there any docs on how to do this and how to move the vm to this disk?

thx

Ole

 

[veenaay]

Hi Udo,

Thanks for ur help. Ur suggestion worked server is up again.

BTW , I see that our server space is filling up very fast , is there any way we can trace out reason behind this ?

 

[udo]

Hi Udo,

Thanks for ur help. Ur suggestion worked server is up again.

fine, that all is running again.

BTW , I see that our server space is filling up very fast , is there any way we can trace out reason behind this ?

See my first post - ncdu is an nice tool for that. But 15GB for all (system, logs, VMs) is not too much.

Udo

 

[udo]

Hi Udo.

Thanks for all the help so faar!!

yes I think making a extended LV for data is very wise. Are there any docs on how to do this and how to move the vm to this disk?

thx

Ole

Hi Ole,
there are plenty docs in the web about lvm and so on.
In short (see manpages to know what you are doing):

lvcreate -L 100G -n data /dev/vg0
mkfs.ext3 /dev/vg0/data
# shutdown all VMs and CTs
echo "/dev/vg0/data /var/lib/vz ext3 defaults 0 1" >> /etc/fstab
mv /var/lib/vz /var/lib/vz.org; mkdir /var/lib/vz; mount /var/lib/vz

cd /var/lib/vz.org
rsync -av . /var/lib/vz/

# control with 
df -h

# after that start all VMs - if all is running (also a reboot) you can remove /var/lib/vz.org

Udo

 

[Ole]

Hi Ole,
there are plenty docs in the web about lvm and so on.
In short (see manpages to know what you are doing):

lvcreate -L 100G -n data /dev/vg0
mkfs.ext3 /dev/vg0/data
# shutdown all VMs and CTs
echo "/dev/vg0/data /var/lib/vz ext3 defaults 0 1" >> /etc/fstab
mv /var/lib/vz /var/lib/vz.org; mkdir /var/lib/vz; mount /var/lib/vz

cd /var/lib/vz.org
rsync -av . /var/lib/vz/

# control with 
df -h

# after that start all VMs - if all is running (also a reboot) you can remove /var/lib/vz.org

Udo

Hi Udo

Thanks for the help.
I have tried ot study your example so I cna understand what happends
when I ran hte rsync for moving the vm it has taken some time..aroudn 15 min now and I am unsure what to do.

Can it be an issue with the rsync command? Any thought what to do to figure out this?

I have natually not been able to control things yet wiht df -h

Ole