io-error

[eiger3970]

Hi, I have the io-error again.
How can I stop this. I think my HDD is full.
I'm not clear on which HDD to make space on?
local (proxmox) is full.
Also, how to make space on the HDD?

This is the server setup:

4 x 120 GB SSDs to setup RAID.
1 x 500 GB HDD for daily backups.

I set my local (proxmox) to backup automatically, but now a HDD is full?
Very confusing what the cause is and what HDD to fix.
So, I have local (proxmox) 54.75 GiB size and used. local-lvm (proxmox) has 143.70 GiB size and 5.99 GiB used.
I think I need to clear out the local (proxmox), but not sure how, without breaking whatever's on local (proxmox) by default?
I could SSH into the HDD and wipe it, but I think that is too much clearing of the HDD?

 

[LnxBil]

What kind of raid in on your SSDs?

Please post the output of df -h, lvs and vgs and pvs (inside a CODE tag)

 

[eiger3970]

root@proxmox:/var/lib/vz/images# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             10M     0   10M   0% /dev
tmpfs           3.2G  329M  2.9G  11% /run
/dev/dm-0        55G   55G     0 100% /
tmpfs           7.9G   37M  7.8G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs           100K     0  100K   0% /run/lxcfs/controllers
cgmfs           100K     0  100K   0% /run/cgmanager/fs
/dev/fuse        30M   16K   30M   1% /etc/pve
root@proxmox:/var/lib/vz/images# lvs
  LV            VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve  twi-aotz-- 143.70g             4.17   2.43                            
  root          pve  -wi-ao----  55.75g                                                    
  swap          pve  -wi-ao----   8.00g                                                    
  vm-143-disk-1 pve  Vwi-aotz--  30.00g data        19.96                                  
root@proxmox:/var/lib/vz/images# vgs
  VG   #PV #LV #SN Attr   VSize   VFree 
  pve    1   4   0 wz--n- 223.44g 15.85g
root@proxmox:/var/lib/vz/images# pvs
  PV         VG   Fmt  Attr PSize   PFree 
  /dev/sda3  pve  lvm2 a--  223.44g 15.85g

 

[manu]

so the problem is on the disk /dev/sda
( technically the logical volume where you have your root partition is full, which is inside a volume group called pve , inside the third partition of the disk /dev/sda )

you can install the ncdu package to know what is taking place in your root partition

ncdu /

 

[eiger3970]

Hmm, yum not intalled on Proxmox. I thought Proxmox was a Centos system. Might be Red Hat?

Ok, apparently Proxmox is based on Debian.
$ apt-get install ncdu
E: Unable to locate package ncdu

 

[e100]

Try running "apt-get update" first, then "apt-get install ncdu"

 

[eiger3970]

Try running "apt-get update" first, then "apt-get install ncdu"

Thank you. It looks like the original problem of no space is now effecting this install.

root@proxmox:/var/lib/vz/images# apt-get install ncdu
Reading package lists... Error!
E: Write error - write (28: No space left on device)
E: Can't mmap an empty file
E: Failed to truncate file - ftruncate (9: Bad file descriptor)
E: The package lists or status file could not be parsed or opened.

 

[eiger3970]

Well, not sure how you see the filled disk is /dev/sda third partition?

Anyways, maybe I can delete the tmpfs directory mounted on /run. This would clear 3.2G.
Then install ncdu, to find out why the disk is full, then clear out, then configure to avoid future issues like this.

 

[eiger3970]

How do I see what is on the /dev/dm-0 directory? I'm thinking of deleting this which seems to be taking up 55G.

Also, how to see what caused to overfilling of the disks. I'll check the logs, but need to configure Proxmox so Proxmox does not repeat this issue.

 

[eiger3970]

I deleted /mnt/backup/dump and now have on local (proxmox) Size 54.75 GiB, Used 33.59 GiB and Avail 18.36 GiB.
Servers now running.
I need to figure out what filled up the local (proxmox) disk. Proxmox does local backups initially. I made a script (or used the GUI) to transfer monthly to an external HDD. Seems something went awry.

 

[eiger3970]

Well, I notice I now have an error:

Partition 3 does not start on physical sector boundary.

 

[LnxBil]

Just do a

du -csm /var/lib/vz/*

this shows where the mail space is in your installation. You should monitor space usage in general to figure out if something fills up your disk. Also, put /var/lib/vz on a separate disk/volume such that if it is filled up, your system will continue to work properly.

 

[eiger3970]

Thanks. I'll ignore the error.

 

[newacecorp]

After the out-of-disk space issue is resolved (perhaps by deleting snapshots from available VMs) you can do a "qm resume <vmid>" to get the VM running again and flush whatever writes were pending to disk. The VM should resume without issue.