io-error

eiger3970

Active Member
Sep 9, 2012
273
3
38
Hi, I have the io-error again.
How can I stop this. I think my HDD is full.
I'm not clear on which HDD to make space on?
local (proxmox) is full.
Also, how to make space on the HDD?

This is the server setup:

4 x 120 GB SSDs to setup RAID.
1 x 500 GB HDD for daily backups.

I set my local (proxmox) to backup automatically, but now a HDD is full?
Very confusing what the cause is and what HDD to fix.
So, I have local (proxmox) 54.75 GiB size and used. local-lvm (proxmox) has 143.70 GiB size and 5.99 GiB used.
I think I need to clear out the local (proxmox), but not sure how, without breaking whatever's on local (proxmox) by default?
I could SSH into the HDD and wipe it, but I think that is too much clearing of the HDD?
 
Last edited:

LnxBil

Famous Member
Feb 21, 2015
5,482
607
133
Germany
What kind of raid in on your SSDs?

Please post the output of df -h, lvs and vgs and pvs (inside a CODE tag)
 

eiger3970

Active Member
Sep 9, 2012
273
3
38
Code:
root@proxmox:/var/lib/vz/images# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             10M     0   10M   0% /dev
tmpfs           3.2G  329M  2.9G  11% /run
/dev/dm-0        55G   55G     0 100% /
tmpfs           7.9G   37M  7.8G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs           100K     0  100K   0% /run/lxcfs/controllers
cgmfs           100K     0  100K   0% /run/cgmanager/fs
/dev/fuse        30M   16K   30M   1% /etc/pve
root@proxmox:/var/lib/vz/images# lvs
  LV            VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve  twi-aotz-- 143.70g             4.17   2.43                            
  root          pve  -wi-ao----  55.75g                                                    
  swap          pve  -wi-ao----   8.00g                                                    
  vm-143-disk-1 pve  Vwi-aotz--  30.00g data        19.96                                  
root@proxmox:/var/lib/vz/images# vgs
  VG   #PV #LV #SN Attr   VSize   VFree 
  pve    1   4   0 wz--n- 223.44g 15.85g
root@proxmox:/var/lib/vz/images# pvs
  PV         VG   Fmt  Attr PSize   PFree 
  /dev/sda3  pve  lvm2 a--  223.44g 15.85g
 

manu

Proxmox Staff Member
Mar 3, 2015
806
64
28
so the problem is on the disk /dev/sda
( technically the logical volume where you have your root partition is full, which is inside a volume group called pve , inside the third partition of the disk /dev/sda )

you can install the ncdu package to know what is taking place in your root partition

ncdu /
 

eiger3970

Active Member
Sep 9, 2012
273
3
38
Hmm, yum not intalled on Proxmox. I thought Proxmox was a Centos system. Might be Red Hat?

Ok, apparently Proxmox is based on Debian.
$ apt-get install ncdu
E: Unable to locate package ncdu
 

eiger3970

Active Member
Sep 9, 2012
273
3
38
Try running "apt-get update" first, then "apt-get install ncdu"
Thank you. It looks like the original problem of no space is now effecting this install.
Code:
root@proxmox:/var/lib/vz/images# apt-get install ncdu
Reading package lists... Error!
E: Write error - write (28: No space left on device)
E: Can't mmap an empty file
E: Failed to truncate file - ftruncate (9: Bad file descriptor)
E: The package lists or status file could not be parsed or opened.
 

eiger3970

Active Member
Sep 9, 2012
273
3
38
Well, not sure how you see the filled disk is /dev/sda third partition?

Anyways, maybe I can delete the tmpfs directory mounted on /run. This would clear 3.2G.
Then install ncdu, to find out why the disk is full, then clear out, then configure to avoid future issues like this.
 

eiger3970

Active Member
Sep 9, 2012
273
3
38
How do I see what is on the /dev/dm-0 directory? I'm thinking of deleting this which seems to be taking up 55G.

Also, how to see what caused to overfilling of the disks. I'll check the logs, but need to configure Proxmox so Proxmox does not repeat this issue.
 

eiger3970

Active Member
Sep 9, 2012
273
3
38
I deleted /mnt/backup/dump and now have on local (proxmox) Size 54.75 GiB, Used 33.59 GiB and Avail 18.36 GiB.
Servers now running.
I need to figure out what filled up the local (proxmox) disk. Proxmox does local backups initially. I made a script (or used the GUI) to transfer monthly to an external HDD. Seems something went awry.
 

eiger3970

Active Member
Sep 9, 2012
273
3
38
Well, I notice I now have an error:
Code:
Partition 3 does not start on physical sector boundary.
 

LnxBil

Famous Member
Feb 21, 2015
5,482
607
133
Germany
Just do a

Code:
du -csm /var/lib/vz/*

this shows where the mail space is in your installation. You should monitor space usage in general to figure out if something fills up your disk. Also, put /var/lib/vz on a separate disk/volume such that if it is filled up, your system will continue to work properly.
 
  • Like
Reactions: eiger3970

newacecorp

Active Member
Oct 14, 2012
38
0
26
After the out-of-disk space issue is resolved (perhaps by deleting snapshots from available VMs) you can do a "qm resume <vmid>" to get the VM running again and flush whatever writes were pending to disk. The VM should resume without issue.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!