No space left?

[nicedevil]

Hi I have a problem with one of my LXC containers. On starting it tells me that there is no space left on device but the pool is about 6 TB and the LXC is only using 5.1 TB
What are the right steps to debug this error?

root@node02:~# cat lxc-250.log
lxc-start 250 20230409125408.708 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 250 20230409125408.709 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "250", config section "lxc"
lxc-start 250 20230409125409.476 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 250 lxc pre-start produced output: closing file '/etc/systemd/network/eth0.network.tmp.3782' failed - No space left on device


lxc-start 250 20230409125409.486 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 250 lxc pre-start produced output: error in setup task PVE::LXC::Setup::pre_start_hook


lxc-start 250 20230409125409.556 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 1
lxc-start 250 20230409125409.556 ERROR    start - ../src/lxc/start.c:lxc_init:844 - Failed to run lxc.hook.pre-start for container "250"
lxc-start 250 20230409125409.557 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "250"
lxc-start 250 20230409125409.557 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "250", config section "lxc"
lxc-start 250 20230409125409.557 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "250", config section "lxc"
lxc-start 250 20230409125409.894 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 250 20230409125409.894 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

 

[floh8]

how about your root storage space?
type #df -h

 

[nicedevil]

how about your root storage space?
type #df -h

34 GB free from 40 GB

 

[floh8]

increase the size of the lxc hdd!

 

[nicedevil]

increase the size of the lxc hdd!

what the hell... ok my 10 gb on that drive were full

I already asked this question on discord channels... is there a monitoring solution for each root/boot drive of the LXCs?
the metric server (grafana/influx) only see the pools on the host itself.

 

[floh8]

you can use every monitoring solution and install the linux client inside a container or vm

 

[nicedevil]

you can use every monitoring solution and install the linux client inside a container or vm

Is there a recommended one that can be thrown to influxdbv2 ?

 

[floh8]

the most can be connected to influxdb if is that what u mean...i like zabbix