[SOLVED] No space left on device

[manjotsc]

I am getting this error when access the shell and cannot start all vms,

"Connection failed (Error 500: unable to open file '/var/tmp/pve-reserved-ports.tmp.39578' - No space left on device)"

root@vms:~# df -kh
Filesystem                     Size  Used Avail Use% Mounted on
udev                           7.8G     0  7.8G   0% /dev
tmpfs                          1.6G   10M  1.6G   1% /run
/dev/mapper/pve-root            15G  3.2G   11G  24% /
tmpfs                          7.8G   63M  7.7G   1% /dev/shm
tmpfs                          5.0M     0  5.0M   0% /run/lock
tmpfs                          7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/sde1                      220G   37G  172G  18% /mnt/pve/VMDriveEXT
/dev/sda4                      161G   50G  103G  33% /mnt/VMDrive
/dev/sda2                      511M  312K  511M   1% /boot/efi
/dev/sdd1                      916G  823G   48G  95% /mnt/S1TB
/dev/sdb1                      916G   44G  827G   5% /mnt/pve/OneTB
/dev/sdc1                      3.6T  584G  2.9T  17% /mnt/4TB
/dev/fuse                       30M   32K   30M   1% /etc/pve
//repo.manjot.net/ProgramRepo  466G  396G   71G  85% /mnt/pve/ProgramRepo
//192.168.40.4/VMShare         223G  194G   30G  87% /mnt/pve/VMShare
tmpfs                          1.6G     0  1.6G   0% /run/user/0

starting worker failed: unable to create output file '/var/log/pve/tasks/5/UPID:vms:0000EC89:00016A2D:5F864DF5:vzstart:102:root@pam:' - No space left on device (500)

 

[Moayad]

Hi,

did you upgrade before this error comes? - if so, please reboot the node and try again

 

[manjotsc]

Hi,

did you upgrade before this error comes? - if so, please reboot the node and try again

No I haven't upgraded, I tried rebooting multiple times. But the issue remains, it started happening after I tried creating LXC Container, I have even tried removing that container. But It didn't fix.

 

[manjotsc]

Can somebody help, please

Thanks,

 

[manjotsc]

Solved have a look at https://www.reddit.com/r/Proxmox/comments/jc551l/no_space_left_on_device/

There has tons of files filled under /var/lib/samba/private/msg.sock/

Deleted them using for i in /var/lib/samba/private/msg.sock/*; do rm "$i"; done

 

[Moayad]

glad that you solved yourself!


Thanks for sharing your solution

 

[CvH]

That problem effects every proxmox installation that uses smb/samba at the proxmox host.
Open bugs for it
https://bugzilla.proxmox.com/show_bug.cgi?id=2333
https://bugzilla.samba.org/show_bug.cgi?id=12435

One time clean up is not a solution, you need to clean it frequently.
We had already the problem that it eat up all the free storage and that it eat up all free inodes. Be aware that find does not work if you never cleaned it manually before due the huge amount of files. It creates ~1files/2sec.

Our hacky solution is a cron that cleans up. Otherwise proxmox is not useable after some weeks/months of runtime.
/etc/cron.d/samba_clean

00 09 * * * root /usr/bin/find /run/samba/msg.lock -mtime +1 -type f -delete
05 09 * * * root /usr/bin/find /var/lib/samba/private/msg.sock -mtime +1 -type s -delete

 

[manjotsc]

@CvH
Thanks,

 

[RolandK]

news:
https://bugzilla.proxmox.com/show_bug.cgi?id=2333