Can't remove Container

A

AlejandroRRG

New Member
Jun 28, 2024
3
0
1
Good afternoon Team, I have Proxmox cluster and I created a container on my node 1 and installed an application (UrBackup) on it, the problem is that after a few days the application stopped working, I restarted the container, and it never came back I was not able to turn it on, so I build everything from scratch on my node 2 and after a few days maybe a week the same thing happened I restarted the container to see if that solves the problem with my application since I thought that it was a problem with the application and I was not able to start the container. Today I created a new container, and I was installing Zabbix on it and I had to restart the container because an error with the zabbix installation and now it is happening the same thing with this new container once I restarted it never came back, so I had to stop it by using the Shell of the node.

I am trying to remove the container but is not letting me to do it.

See the attachments to see the errors.

Can some tell me why this is happening? why the container gets into that state.

I was able to stop the container by running these commands:
ps aux | grep CTID
kil -9

but now I am trying to delete the container and am getting the errors on the screenshots


Thank you in advance.
 

Attachments

  • proxmox error1.png
    proxmox error1.png
    10.3 KB · Views: 14
  • proxmox error2.png
    proxmox error2.png
    27.9 KB · Views: 19
Hi,
according to the error message there is still a filesystem mounted located on a partition of that logical volume. What is the output of lvs and mount? Also, please share your container configuration pct config VMID and try to startup the container with the debug flag, that might give some insights on why it is not starting pct start VMID --debug.
 
Hi Chris, thank you for your replaying.

Below is the output of the commands that you asked. I am going to add it as an attachment as well. Thank you in advance for your help

root@proxmox01:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-aotz-- 6.21t 22.94 2.89
root pve -wi-ao---- 96.00g
snap_vm-103-disk-0_AfterCopyingBKP pve Vri---tz-k 4.00m data vm-103-disk-0
snap_vm-103-disk-0_AfterRestoringtheDataBase pve Vri---tz-k 4.00m data vm-103-disk-0
snap_vm-103-disk-0_BeforeUpdateDBTo127v pve Vri---tz-k 4.00m data
snap_vm-103-disk-1_AfterCopyingBKP pve Vri---tz-k 900.00g data vm-103-disk-1
snap_vm-103-disk-1_AfterRestoringtheDataBase pve Vri---tz-k 900.00g data vm-103-disk-1
snap_vm-103-disk-1_BeforeUpdateDBTo127v pve Vri---tz-k 900.00g data
snap_vm-110-disk-0_BeforeUpdate pve Vri---tz-k 150.00g data vm-110-disk-0
snap_vm-111-disk-0_Fixed pve Vri---tz-k 50.00g data vm-111-disk-0
snap_vm-111-disk-0_broken pve Vri---tz-k 50.00g data
snap_vm-114-disk-0_clean pve Vri---tz-k 100.00g data
snap_vm-122-disk-0_BeforeHardening pve Vri---tz-k 128.00g data vm-122-disk-0
snap_vm-128-disk-0_ReadyInstallation pve Vri---tz-k 40.00g data
snap_vm-128-disk-0_WorkingPerfect pve Vri---tz-k 40.00g data vm-128-disk-0
snap_vm-130-disk-0_BeforeCleanup pve Vri---tz-k 4.00m data vm-130-disk-0
snap_vm-130-disk-0_BeforeDFSR pve Vri---tz-k 4.00m data vm-130-disk-0
snap_vm-130-disk-1_BeforeCleanup pve Vri---tz-k 250.00g data vm-130-disk-1
snap_vm-130-disk-1_BeforeDFSR pve Vri---tz-k 250.00g data vm-130-disk-1
snap_vm-131-disk-0_Clean pve Vri---tz-k 120.00g data
snap_vm-131-disk-0_Full_working pve Vri---tz-k 120.00g data
snap_vm-131-disk-0_Working pve Vri---tz-k 120.00g data
snap_vm-131-disk-0_fixed_working pve Vri---tz-k 120.00g data
swap pve -wi-ao---- 8.00g
vm-101-disk-0 pve Vwi-aotz-- 350.00g data 2.21
vm-103-disk-0 pve Vwi-aotz-- 4.00m data 14.06
vm-103-disk-1 pve Vwi-aotz-- 900.00g data 58.10
vm-103-state-AfterCopyingBKP pve Vwi-a-tz-- <64.49g data 47.54
vm-103-state-AfterRestoringtheDataBase pve Vwi-a-tz-- <64.49g data 48.14
vm-103-state-BeforeUpdateDBTo127v pve Vwi-a-tz-- <64.49g data 39.05
vm-107-disk-0 pve Vwi-aotz-- 80.00g data 6.14
vm-110-disk-0 pve Vwi-aotz-- 150.00g data 5.47
vm-111-disk-0 pve Vwi-aotz-- 50.00g data snap_vm-111-disk-0_broken 100.00
vm-111-state-Fixed pve Vwi-a-tz-- <20.49g data 29.97
vm-111-state-broken pve Vwi-a-tz-- <20.49g data 29.99
vm-112-disk-0 pve Vwi-a-tz-- 4.00m data 14.06
vm-112-disk-1 pve Vwi-a-tz-- 60.00g data 38.29
vm-114-disk-0 pve Vwi-aotz-- 100.00g data snap_vm-114-disk-0_clean 4.48
vm-115-disk-0 pve Vwi-a-tz-- 4.00m data 14.06
vm-115-disk-1 pve Vwi-a-tz-- 150.00g data 81.44
vm-116-disk-0 pve Vwi-aotz-- 50.00g data 11.09
vm-119-disk-0 pve Vwi-a-tz-- 100.00g data 6.05
vm-120-disk-0 pve Vwi-aotz-- 80.00g data 100.00
vm-122-disk-0 pve Vwi-a-tz-- 128.00g data 52.69
vm-123-disk-0 pve Vwi-aotz-- 250.00g data 3.47
vm-128-disk-0 pve Vwi-aotz-- 40.00g data snap_vm-128-disk-0_ReadyInstallation 11.96
vm-129-disk-0 pve Vwi-aotz-- 100.00g data 7.86
vm-130-disk-0 pve Vwi-aotz-- 4.00m data 14.06
vm-130-disk-1 pve Vwi-aotz-- 250.00g data 7.49
vm-130-state-BeforeDFSR pve Vwi-a-tz-- <16.49g data 39.90
vm-131-disk-0 pve Vwi-aotz-- 120.00g data snap_vm-131-disk-0_Full_working 3.25
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
root@proxmox01:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=98943604k,nr_inodes=24735901,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=19795488k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=41078)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
//192.168.168.2/backuppve on /mnt/pve/backuppvesmb type cifs (rw,relatime,vers=3.1.1,cache=strict,username=backupadmin,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.168.2,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=19795484k,nr_inodes=4948871,mode=700,inode64)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
root@proxmox01:~# pct config 101
arch: amd64
cores: 2
hostname: rrgzabbixprod
memory: 8192
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:2C:B7:5C,ip=dhcp,type=veth
ostype: centos
rootfs: local-lvm:vm-101-disk-0,size=350G
swap: 4096
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
root@proxmox01:~# pct start 101 --debug
run_buffer: 571 Script exited with status 16
lxc_init: 845 Failed to run lxc.hook.pre-start for container "101"
__lxc_start: 2034 Failed to initialize container "101"
rt-hook" for container "101", config section "lxc"
DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 101 lxc pre-start produced output: failed to remove directory '/sys/fs/cgroup/lxc/101/ns/system.slice/systemd-machine-id-commit.service': Device or resource busy

ERROR utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 16
ERROR start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "101"
ERROR start - ../src/lxc/start.c:__lxc_start:2034 - Failed to initialize container "101"
INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "101", config section "lxc"
startup for container '101' failed
 

Attachments

AlejandroRRG said:
Hi Chris, thank you for your replaying.

Below is the output of the commands that you asked. I am going to add it as an attachment as well. Thank you in advance for your help

root@proxmox01:~# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data pve twi-aotz-- 6.21t 22.94 2.89
root pve -wi-ao---- 96.00g
snap_vm-103-disk-0_AfterCopyingBKP pve Vri---tz-k 4.00m data vm-103-disk-0
snap_vm-103-disk-0_AfterRestoringtheDataBase pve Vri---tz-k 4.00m data vm-103-disk-0
snap_vm-103-disk-0_BeforeUpdateDBTo127v pve Vri---tz-k 4.00m data
snap_vm-103-disk-1_AfterCopyingBKP pve Vri---tz-k 900.00g data vm-103-disk-1
snap_vm-103-disk-1_AfterRestoringtheDataBase pve Vri---tz-k 900.00g data vm-103-disk-1
snap_vm-103-disk-1_BeforeUpdateDBTo127v pve Vri---tz-k 900.00g data
snap_vm-110-disk-0_BeforeUpdate pve Vri---tz-k 150.00g data vm-110-disk-0
snap_vm-111-disk-0_Fixed pve Vri---tz-k 50.00g data vm-111-disk-0
snap_vm-111-disk-0_broken pve Vri---tz-k 50.00g data
snap_vm-114-disk-0_clean pve Vri---tz-k 100.00g data
snap_vm-122-disk-0_BeforeHardening pve Vri---tz-k 128.00g data vm-122-disk-0
snap_vm-128-disk-0_ReadyInstallation pve Vri---tz-k 40.00g data
snap_vm-128-disk-0_WorkingPerfect pve Vri---tz-k 40.00g data vm-128-disk-0
snap_vm-130-disk-0_BeforeCleanup pve Vri---tz-k 4.00m data vm-130-disk-0
snap_vm-130-disk-0_BeforeDFSR pve Vri---tz-k 4.00m data vm-130-disk-0
snap_vm-130-disk-1_BeforeCleanup pve Vri---tz-k 250.00g data vm-130-disk-1
snap_vm-130-disk-1_BeforeDFSR pve Vri---tz-k 250.00g data vm-130-disk-1
snap_vm-131-disk-0_Clean pve Vri---tz-k 120.00g data
snap_vm-131-disk-0_Full_working pve Vri---tz-k 120.00g data
snap_vm-131-disk-0_Working pve Vri---tz-k 120.00g data
snap_vm-131-disk-0_fixed_working pve Vri---tz-k 120.00g data
swap pve -wi-ao---- 8.00g
vm-101-disk-0 pve Vwi-aotz-- 350.00g data 2.21
vm-103-disk-0 pve Vwi-aotz-- 4.00m data 14.06
vm-103-disk-1 pve Vwi-aotz-- 900.00g data 58.10
vm-103-state-AfterCopyingBKP pve Vwi-a-tz-- <64.49g data 47.54
vm-103-state-AfterRestoringtheDataBase pve Vwi-a-tz-- <64.49g data 48.14
vm-103-state-BeforeUpdateDBTo127v pve Vwi-a-tz-- <64.49g data 39.05
vm-107-disk-0 pve Vwi-aotz-- 80.00g data 6.14
vm-110-disk-0 pve Vwi-aotz-- 150.00g data 5.47
vm-111-disk-0 pve Vwi-aotz-- 50.00g data snap_vm-111-disk-0_broken 100.00
vm-111-state-Fixed pve Vwi-a-tz-- <20.49g data 29.97
vm-111-state-broken pve Vwi-a-tz-- <20.49g data 29.99
vm-112-disk-0 pve Vwi-a-tz-- 4.00m data 14.06
vm-112-disk-1 pve Vwi-a-tz-- 60.00g data 38.29
vm-114-disk-0 pve Vwi-aotz-- 100.00g data snap_vm-114-disk-0_clean 4.48
vm-115-disk-0 pve Vwi-a-tz-- 4.00m data 14.06
vm-115-disk-1 pve Vwi-a-tz-- 150.00g data 81.44
vm-116-disk-0 pve Vwi-aotz-- 50.00g data 11.09
vm-119-disk-0 pve Vwi-a-tz-- 100.00g data 6.05
vm-120-disk-0 pve Vwi-aotz-- 80.00g data 100.00
vm-122-disk-0 pve Vwi-a-tz-- 128.00g data 52.69
vm-123-disk-0 pve Vwi-aotz-- 250.00g data 3.47
vm-128-disk-0 pve Vwi-aotz-- 40.00g data snap_vm-128-disk-0_ReadyInstallation 11.96
vm-129-disk-0 pve Vwi-aotz-- 100.00g data 7.86
vm-130-disk-0 pve Vwi-aotz-- 4.00m data 14.06
vm-130-disk-1 pve Vwi-aotz-- 250.00g data 7.49
vm-130-state-BeforeDFSR pve Vwi-a-tz-- <16.49g data 39.90
vm-131-disk-0 pve Vwi-aotz-- 120.00g data snap_vm-131-disk-0_Full_working 3.25
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
root@proxmox01:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=98943604k,nr_inodes=24735901,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=19795488k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=41078)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
//192.168.168.2/backuppve on /mnt/pve/backuppvesmb type cifs (rw,relatime,vers=3.1.1,cache=strict,username=backupadmin,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.168.2,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=19795484k,nr_inodes=4948871,mode=700,inode64)
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
root@proxmox01:~# pct config 101
arch: amd64
cores: 2
hostname: rrgzabbixprod
memory: 8192
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:2C:B7:5C,ip=dhcp,type=veth
ostype: centos
rootfs: local-lvm:vm-101-disk-0,size=350G
swap: 4096
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
root@proxmox01:~# pct start 101 --debug
run_buffer: 571 Script exited with status 16
lxc_init: 845 Failed to run lxc.hook.pre-start for container "101"
__lxc_start: 2034 Failed to initialize container "101"
rt-hook" for container "101", config section "lxc"
DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 101 lxc pre-start produced output: failed to remove directory '/sys/fs/cgroup/lxc/101/ns/system.slice/systemd-machine-id-commit.service': Device or resource busy

ERROR utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 16
ERROR start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "101"
ERROR start - ../src/lxc/start.c:__lxc_start:2034 - Failed to initialize container "101"
INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "101", config section "lxc"
startup for container '101' failed
Click to expand...
Although you killed the parent process invoking the container, it seems you still might have some lingering processes holding open file handles to the containers rootfs disk and the systemd slice. Maybe you can find out which process this is by using fuser -v /sys/fs/cgroup/lxc/101/ns/system.slice/systemd-machine-id-commit.service and/or files contained within that directory. Also, check your systemd journal for errors and consider rebooting this node, as it seems in a rather inconsistent state.

Edit: Also check the output of systemctl status pve-container@101.service
 
Last edited:
Good morning Chris thank you very much for your help, yesterday I rebooted the node and after that I was able to delete the containers that were having issues for getting deleted.
The thing is that I had rebooted the node less than a month, so I am wondering what could be causing these problems? can it be a hardware issue? or maybe an update issue?

Can you please provide me more information in how diagnose this kind of issues.

Thank you very much for your time and help
 
AlejandroRRG said:
Good morning Chris thank you very much for your help, yesterday I rebooted the node and after that I was able to delete the containers that were having issues for getting deleted.
The thing is that I had rebooted the node less than a month, so I am wondering what could be causing these problems? can it be a hardware issue? or maybe an update issue?

Can you please provide me more information in how diagnose this kind of issues.

Thank you very much for your time and help
Click to expand...
Well, I would recommend to find the root cause of why the containers are becoming unresponsive and the reboot most likely fails because of the container being unresponsive to begin with. So you should check the logs inside the container for possible clues on why.

How does this manifest itself? Are you still able to connect to the container via a pct enter <VMID> when it is unresponsive. Any hints or errors in the systemd journal of the PVE host?
 
You must log in or register to reply here.
Top Bottom
Back