Why is /dev full on my LXC container?

J

jimmyjoe

Active Member
Proxmox Subscriber
Jan 12, 2015
94
2
28
New to containers, so maybe overlooking something - why is /dev at 100% on my container? After a reboot it was fine.

5 days uptime:
$ df -mh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-vm--700--disk--1 50G 25G 22G 54% /
none 492K 492K 0 100% /dev
none 13G 136K 13G 1% /run
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 63G 0 63G 0% /run/shm
none 100M 0 100M 0% /run/user

After reboot:
$ df -mh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-vm--700--disk--1 50G 24G 23G 51% /
none 492K 8.0K 484K 2% /dev
none 13G 120K 13G 1% /run
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 63G 0 63G 0% /run/shm
none 100M 0 100M 0% /run/user
 
Looks like something using shared memory perhaps?

# find /dev/ -type f
/dev/shm/qb-pve2-event-1542-19201-26-data
/dev/shm/qb-pve2-event-1542-19201-26-header
/dev/shm/qb-pve2-response-1542-19201-26-data
/dev/shm/qb-pve2-response-1542-19201-26-header
/dev/shm/qb-pve2-request-1542-19201-26-data
/dev/shm/qb-pve2-request-1542-19201-26-header
/dev/shm/qb-pve2-event-1542-19203-25-data
...
 
The output in the OP shows a mounted /run/shm but no mounted /dev/shm. Usually one of them symlinks to the other (/run/shm to /dev/shm on all my systems actually), so you should check that. If - in your case - /dev/shm doesn't symlink to /run/shm then that would explain the issue as shared-memory-files are then created directly on the /dev mount point. The contents of that /dev/shm above are curious though, was that inside the container, are you running a nested pve inside?
 
Thanks @wbumiller ,
just getting back to this.. /dev/ full again today. All of the CLI pastes abover are from inside the container. I see those *pve2* files exist on the hypervisor (in /dev/shm) as well. Are they created inside the container also?

I should point out also that this container was created via rsync from a bare-metal host so I may have whacked something specific to Containerization when rsyncing into the rootfs.

# file /dev/shm
/dev/shm: symbolic link to `/run/shm'

# mount | grep shm
none on /run/shm type tmpfs (rw,nosuid,nodev)

# grep Mem /proc/meminfo
MemTotal: 65536000 kB
MemFree: 63711044 kB
MemAvailable: 63711044 kB

# find /dev/ -type f -ls
64 4 -rw-r----- 1 root adm 2374 Apr 4 11:08 /dev/tty10
26 4 -rw-r--r-- 1 root root 72 Apr 4 11:07 /dev/.udev/rules.d/root.rules
 
Decided to check my second "cloned" container on a different hypervisor. This one has a full /dev also.. hrm. This host was done from the rsync link above as well. Some CLI from the second container. 492K is not much but I can't tell where the space is being used.

# df -mh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg1.6-vm--5000--disk--1 79G 15G 61G 19% /
none 492K 492K 0 100% /dev
none 6.3G 112K 6.3G 1% /run
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 32G 0 32G 0% /run/shm
none 100M 0 100M 0% /run/user

# file /dev/shm
/dev/shm: symbolic link to `/run/shm'

# df -mh /run/
Filesystem Size Used Avail Use% Mounted on
none 6.3G 112K 6.3G 1% /run

# grep Mem /proc/meminfo
MemTotal: 65861076 kB
MemFree: 62594168 kB
MemAvailable: 62594168 kB


# find /dev/ -ls
2 0 drwxr-xr-x 8 root root 1160 Mar 31 03:00 /dev/
65 0 drwxr-xr-x 2 root root 60 Mar 31 03:00 /dev/mapper
66 0 crw------- 1 root root Mar 31 03:00 /dev/mapper/control
63 0 drwxr-xr-x 2 root root 60 Mar 30 15:19 /dev/net
64 0 crw-rw---- 1 root root Mar 30 15:19 /dev/net/tun
62 0 crw-rw---- 1 root dip Mar 30 15:19 /dev/ppp
17 0 lrwxrwxrwx 1 root root 4 Mar 30 15:19 /dev/stderr -> fd/2
16 0 lrwxrwxrwx 1 root root 4 Mar 30 15:19 /dev/stdout -> fd/1
15 0 lrwxrwxrwx 1 root root 4 Mar 30 15:19 /dev/stdin -> fd/0
14 0 lrwxrwxrwx 1 root root 13 Mar 30 15:19 /dev/fd -> /proc/self/fd
61 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/loop7
60 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/loop6
59 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/loop5
58 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/loop4
57 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/loop3
56 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/loop2
55 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/loop1
54 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/loop0
53 0 lrwxrwxrwx 1 root root 4 Mar 30 15:19 /dev/ram -> ram1
52 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram16
51 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram15
49 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram14
50 488 -rw-r----- 1 root adm 499712 Apr 4 11:42 /dev/tty10
48 0 srw-rw-rw- 1 root root 0 Mar 30 15:19 /dev/log
47 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram13
46 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram12
45 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram11
44 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram10
43 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram9
42 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram8
41 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram7
40 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram6
39 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram5
38 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram4
37 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram3
36 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram2
35 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram1
34 0 brw-rw---- 1 root disk Mar 30 15:19 /dev/ram0
10 0 prw-r----- 1 root adm 0 Apr 1 06:11 /dev/xconsole
8 0 crw-rw-rw- 1 root tty Apr 4 11:25 /dev/tty
9 0 crw-rw-rw- 1 root root Mar 30 15:19 /dev/urandom
7 0 crw-rw-rw- 1 root root Mar 30 15:19 /dev/random
28 0 crw-rw-rw- 1 root root Mar 30 15:19 /dev/full
6 0 lrwxrwxrwx 1 root root 11 Mar 30 15:19 /dev/core -> /proc/kcore
33 0 crw-rw-rw- 1 root root Mar 30 15:19 /dev/zero
32 0 crw-r----- 1 root kmem Mar 30 15:19 /dev/port
31 0 crw-rw-rw- 1 root root Mar 30 15:19 /dev/null
30 0 crw-r----- 1 root kmem Mar 30 15:19 /dev/kmem
29 0 crw-r----- 1 root kmem Mar 30 15:19 /dev/mem
27 0 lrwxrwxrwx 1 root root 8 Mar 30 15:19 /dev/shm -> /run/shm
24 0 drwxr-xr-x 3 root root 60 Mar 30 15:19 /dev/.udev
25 0 drwxr-xr-x 2 root root 60 Mar 30 15:19 /dev/.udev/rules.d
26 4 -rw-r--r-- 1 root root 72 Mar 30 15:19 /dev/.udev/rules.d/root.rules
23 0 crw------- 1 root root Mar 30 15:19 /dev/kmsg
22 0 lrwxrwxrwx 1 root root 8 Mar 30 15:19 /dev/tty2 -> lxc/tty2
20 0 lrwxrwxrwx 1 root root 8 Mar 30 15:19 /dev/tty1 -> lxc/tty1
18 0 lrwxrwxrwx 1 root root 13 Mar 30 15:19 /dev/ptmx -> /dev/pts/ptmx
13 0 lrwxrwxrwx 1 root root 11 Mar 30 15:19 /dev/console -> lxc/console
11 0 drwxr-xr-x 2 root root 100 Mar 30 15:19 /dev/lxc
4 0 crw-rw---- 1 root tty Mar 30 15:20 /dev/lxc/tty2
3 0 crw-rw---- 1 root tty Mar 30 15:20 /dev/lxc/tty1
4 0 crw-rw---- 1 root tty Mar 30 15:20 /dev/lxc/console
2695838 0 drwxrwxrwt 2 root root 40 Mar 30 15:19 /dev/mqueue
1 0 drwxr-xr-x 2 root root 0 Mar 30 15:19 /dev/pts
5 0 crw--w---- 1 user999 tty Apr 4 11:42 /dev/pts/2
4 0 crw-rw---- 1 root tty Mar 30 15:20 /dev/pts/1
3 0 crw-rw---- 1 root tty Mar 30 15:20 /dev/pts/0
2 0 crw-rw-rw- 1 root root Apr 4 11:42 /dev/pts/ptmx
 
Last edited:
I don't understand. -x instructs rsync not to cross filesystem boundaries so if you have separate partitions for usr, tmp, home and var this may not be useful.

Are you meaning to '--exclude=dev' in the rsync command?
 
jimmyjoe said:
I don't understand. -x instructs rsync not to cross filesystem boundaries so if you have separate partitions for usr, tmp, home and var this may not be useful.

Are you meaning to '--exclude=dev' in the rsync command?
Click to expand...

Then I'd use multiple rsync commands. You do not know what files exist that are accidentally named dev for e.g. for development.
 
Do you mean to say rsync will exclude a file such as dev.txt if --exclude=dev is used?
It seems to work differently than that. At least the way I've always used it (for excluding root directory entries).

# dev directory, and regular files containing 'dev'
localhost $ find src -ls
131086 4 drwxr-xr-x 3 skully sysadmin 4096 Apr 5 09:51 src
131087 4 drwxr-xr-x 2 skully sysadmin 4096 Apr 5 09:51 src/dev
131091 0 -rw-r--r-- 1 skully sysadmin 0 Apr 5 09:51 src/dev/dev-dir-dev-file.txt
131088 0 -rw-r--r-- 1 skully sysadmin 0 Apr 5 09:50 src/dev-file.txt
131089 0 -rw-r--r-- 1 skully sysadmin 0 Apr 5 09:50 src/my-dev-file.txt
131090 0 -rw-r--r-- 1 skully sysadmin 0 Apr 5 09:51 src/dev.txt

# sync it normally, everything goes..
localhost $ rsync -avn src/ dest/
sending incremental file list
./
dev-file.txt
dev.txt
my-dev-file.txt
dev/
dev/dev-dir-dev-file.txt

# exclude the dev directory
localhost $ rsync -avn src/ dest/ --exclude=dev
sending incremental file list
./
dev-file.txt
dev.txt
my-dev-file.txt
 
Only dev itself is excluded, but there are other directories named dev on my system:

Code: 
$ find / -name dev 2>/dev/null | wc -l
285

I'd loose a lot.
 
That would be disastrous. I'd like to get this thread back OT however..

Since rsync didn't touch /dev in the container, any thoughts why it's becoming full? On other systems (Virtuals) /dev is several gigabytes in size. Some up to 32G. On the proxmox containers it is 492K. That's quite small in comparison. I don't understand where a udev filesystem gets it's allocation. Maybe it is a configuration option I've overlooked?
 
jimmyjoe said:
Since rsync didn't touch /dev in the container, any thoughts why it's becoming full?
Click to expand...

You're writing on it, so there is something missing like shm mounts or device files like zero stdout ... you have to investigate each of them and look for errors in applications that would write to dev. What applications fail if dev is full?
 
Thanks for taking the time to look at the post. It seems to be OK for the past couple days now. I'll have to check it after the weekend and see if it's tanked again. `shm` is there and properly linked to /run/shm which is 32G in size (via df). I just don't see anything amiss yet.
 
Was 72% full again today. I think it's a configuration in syslog-ng which writes to the file /dev/tty10
removed the file and disabled the config. Must be leftover from an upgrade or something.

# pwd
/etc/syslog-ng

# grep tty10 *
grep: conf.d: Is a directory
grep: patterndb.d: Is a directory
syslog-ng.conf:mad:include "`scl-root`/system/tty10.conf"
syslog-ng.conf:destination d_console_all { file(`tty10`); };
 
I had the same thing after installing syslog-ng on my Gentoo container.
The default config writes to tty12 (so you can switch to that tty to look at the log). Removed the lines that did that:
Code: 
# By default messages are logged to tty12...
#destination console_all { file("/dev/tty12"); };
#log { source(src); destination(console_all); };

Hopefully, this will sort it out :)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back