error closing file '/var/log/pve/tasks/active.tmp.3156' failed - No space left on device (500)

[raj]

Hi team,

I am getting the following error and unsure why.

the error appears when I try to boot up the vms.

I checked the disk space and its full but unsure why.

here is a df -h


root@master:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 26G 2.5G 23G 10% /run
/dev/dm-0 33G 33G 0 100% /
tmpfs 63G 37M 63G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 63G 0 63G 0% /sys/fs/cgroup
/dev/mapper/pve-data 67G 52M 67G 1% /var/lib/vz
tmpfs 100K 0 100K 0% /run/lxcfs/controllers
cgmfs 100K 0 100K 0% /run/cgmanager/fs
/dev/fuse 30M 28K 30M 1% /etc/pve
192.168.0.63:/volume1/BackupNight 2.0T 1.6T 402G 80% /mnt/pve/NightlyBackup
192.168.0.62:/volume2/Datastore 3.6T 1.3T 2.3T 37% /mnt/pve/Datastore
192.168.0.62:/volume1/BackupDaily 1.8T 1.5T 394G 79% /mnt/pve/DailyBackup
192.168.0.63:/volume3/ISO 99G 70G 30G 71% /mnt/pve/ISO
/dev/sdb1 3.6T 865G 2.6T 25% /media/offsite



I am running 4.1

proxmox-ve: 4.1-26 (running kernel: 4.2.6-1-pve) pve-manager: 4.1-1 (running version: 4.1-1/2f9650d4) pve-kernel-4.2.6-1-pve: 4.2.6-26 lvm2: 2.02.116-pve2 corosync-pve: 2.3.5-2 libqb0: 0.17.2-1 pve-cluster: 4.0-29 qemu-server: 4.0-41 pve-firmware: 1.1-7 libpve-common-perl: 4.0-41 libpve-access-control: 4.0-10 libpve-storage-perl: 4.0-38 pve-libspice-server1: 0.12.5-2 vncterm: 1.2-1 pve-qemu-kvm: 2.4-17 pve-container: 1.0-32 pve-firewall: 2.0-14 pve-ha-manager: 1.0-14 ksm-control-daemon: 1.2-1 glusterfs-client: 3.5.2-2+deb8u1 lxc-pve: 1.1.5-5 lxcfs: 0.13-pve1 cgmanager: 0.39-pve1 criu: 1.6.0-1 zfsutils: 0.6.5-pve6~jessie

I have not rebooted yet but a bit fearful at this point.

Can anyone please assist.

Regards,

Raj

 

[raj]

anyone pls?

 

[windinternet]

Your root device /dev/dm-0 is full. Maybe your log files in /var/log have exploded or maybe there is something stored in /var/lib/vz/dump?

Look at the largest files with:

find / -xdev -size +100M

It being a LVM disk of course, if you have some extra medium you can attach you could also lvextend and resize2fs...

 

[raj]

hiya,

Thanks for the help.

Have tried to look for logs not but very much there no backup in the local disks as "/var/lib/vz/dump"

running
find / -xdev -size +100M

I cant find anything on the local disk:

root@master:~# find / -xdev -size +50M
/root/disk/MegaRAID_Storage_Manager-15.08.01-02.noarch.rpm
/root/MSM_linux_x64_installer-15.08.01-02.tar.gz
root@master:~# find / -xdev -size +1000M
root@master:~# find / -xdev -size +100M
root@master:~#


any other ideas pls

 

[raj]

ok i think found my problem,

I have a backup routine to backup to an external usb drive
with 3.4 i used to use usbmount but that does not work with 4.1 so i use udev from the link below
http://www.axllent.org/docs/view/auto-mounting-usb-storage/

Its mounted to /media/offsite

I believe the backup ran while there was no usb drive and it filled the / root

Now I could not find this as the usb drive was connected.

Read an old post from proxmox :
https://forum.proxmox.com/threads/proxmox-1-8-hd-space-root-nearly-100-full.6238/

When i removed the usb drive I was able to find the files on the root and delete them in the location of

/media/offsite/dump/

Now the only thing is why could I cant see that with the usb drive in if anyone can help on that pls?

Cheers,

Raj

 

[windinternet]

On your /var/lib/vz there would be enough room it is mounted on /dev/mapper/pve-data, but somehow your /dev/dm-0 block device is reading as absolutely full, 33G of 33G used. If it doesn't contain very large files, then it must be chock full of small files or the file system has become corrupted.

Can you show the output of

tune2fs -l /dev/dm-0

 

[windinternet]

Ah, yes if the USB drive is mounted, then its mount hides the underlying filesystem. Usually you mount a drive on an empty mount directory, but if it is filled with files, they will "just" disappear from view, until you umount the overmounted device.

Maybe set your backup to a subdirectory on the USB drive that is only present if the USB drive is actually mounted, then it won't fill your real root system.