Problem with CT - dont start after shutdown

M

Maeg

New Member
May 30, 2019
11
0
1
36
I have PX 5.4.6 (kernel 4.15.18-15-pve) and 1 CT container with Red Hat 6.10 (krenel 4.15.18-15-pve).
If i try "pct shutdown @ID@" on host or "shutdown -r now" in CT already seems good, but after that i can't start my container without reboot Host-machine.
 

Attachments

Please post the output of 'pveversion -v' and try starting the container with
Code: 
lxc-start -n 300 -l DEBUG -F -o lxc-300.log
 
hi.

can you stop the container first, before running the debug command? it looks like your container is already running:

Code: 
lxc-start 300 20190531101118.552 DEBUG    commands - commands.c:lxc_cmd_get_state:630 - Container "300" is in "RUNNING" state
lxc-start 300 20190531101118.552 ERROR    lxc_start - tools/lxc_start.c:main:280 - Container is already running

* please stop your container (`pct stop CTID`) and then run the debug command again. post the output here in code tags (preferably not as a txt file)

* can you give us more info about the container:
** was it openvz before?
** did you have any problems with this container before? if yes, how long, and how did the problem first appear (maybe after an upgrade or migration or similar)
** you mentioned you couldn't start your CT without rebooting the host. what kind of error do you get?

EDIT:

also check this[0], because i saw this line in your log:
Code: 
lxc-start: 300: conf.c: lxc_setup_ttydir_console: 1857 Read-only file system - Failed to set mode "0110" to "/dev/pts/1"

[0]: https://pve.proxmox.com/wiki/Convert_OpenVZ_to_LXC#PTY_Allocation
 
* can you give us more info about the container:
** was it openvz before?
** did you have any problems with this container before? if yes, how long, and how did the problem first appear (maybe after an upgrade or migration or similar)
** you mentioned you couldn't start your CT without rebooting the host. what kind of error do you get?
Click to expand...

1. Yes, that was openVZ runnig by PX version 3.3
2. In PX 3.3 it works flawlessly, after i done backup it and convert to LXC in PX 5.4 i have that problem.
3. Yeah. If i shutdown CT i cant start in no way, unless reboot my HOST machine.
If i try do pct start 300 - here what i have:
Code: 
 pct start 300
Job for pve-container@300.service failed because the control process exited with error code.
See "systemctl status pve-container@300.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@300' failed: exit code 1

About information that you ask:
Code: 
root@px00:/etc/init.d# pct list
VMID       Status     Lock         Name
300        stopped                 nfs00

root@px00:/etc/init.d# lxc-start -n 300 -l DEBUG -F -o lxc-300.log
lxc-start: 300: cgroups/cgfsng.c: mkdir_eexist_on_last: 1301 File exists - Failed to create directory "/sys/fs/cgroup/systemd//lxc/300"
lxc-start: 300: cgroups/cgfsng.c: container_create_path_for_hierarchy: 1353 Failed to create cgroup "/sys/fs/cgroup/systemd//lxc/300"
lxc-start: 300: cgroups/cgfsng.c: cgfsng_payload_create: 1526 Failed to create cgroup "/sys/fs/cgroup/systemd//lxc/300"
lxc-start: 300: conf.c: lxc_setup_ttydir_console: 1857 Read-only file system - Failed to set mode "0110" to "/dev/pts/1"
lxc-start: 300: conf.c: lxc_setup: 3679 Failed to setup console
lxc-start: 300: start.c: do_start: 1279 Failed to setup container "300"
lxc-start: 300: sync.c: __sync_wait: 62 An error occurred in another process (expected sequence number 5)
lxc-start: 300: start.c: __lxc_start: 1989 Failed to spawn container "300"
lxc-start: 300: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: 300: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

So here is lxc-300.log:
Code: 
lxc-start 300 20190531111216.595 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 300 20190531111216.595 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start 300 20190531111216.595 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "300", config section "lxc"
lxc-start 300 20190531111217.580 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:714 - Using terminal "/dev/tty" as proxy
lxc-start 300 20190531111217.580 DEBUG    terminal - terminal.c:lxc_terminal_signal_init:192 - Created signal fd 9
lxc-start 300 20190531111217.580 DEBUG    terminal - terminal.c:lxc_terminal_winsz:90 - Set window size to 237 columns and 63 rows
lxc-start 300 20190531111217.702 INFO     start - start.c:lxc_init:906 - Container "300" is initialized
lxc-start 300 20190531111217.702 INFO     conf - conf.c:run_script:495 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "300", config section "net"
lxc-start 300 20190531111218.160 DEBUG    network - network.c:instantiate_veth:206 - Instantiated veth "veth300i0/vethEN7LEB", index is "26"
lxc-start 300 20190531111218.160 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1301 - File exists - Failed to create directory "/sys/fs/cgroup/systemd//lxc/300"
lxc-start 300 20190531111218.160 ERROR    cgfsng - cgroups/cgfsng.c:container_create_path_for_hierarchy:1353 - Failed to create cgroup "/sys/fs/cgroup/systemd//lxc/300"
lxc-start 300 20190531111218.160 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1526 - Failed to create cgroup "/sys/fs/cgroup/systemd//lxc/300"
lxc-start 300 20190531111218.161 DEBUG    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:619 - "cgroup.clone_children" was already set to "1"
lxc-start 300 20190531111218.161 INFO     cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1537 - The container uses "lxc/300-1" as cgroup
lxc-start 300 20190531111218.162 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWNS
lxc-start 300 20190531111218.162 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWPID
lxc-start 300 20190531111218.162 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWUTS
lxc-start 300 20190531111218.162 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWIPC
lxc-start 300 20190531111218.162 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWNET
lxc-start 300 20190531111218.162 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved mnt namespace via fd 15
lxc-start 300 20190531111218.162 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved pid namespace via fd 16
lxc-start 300 20190531111218.162 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved uts namespace via fd 17
lxc-start 300 20190531111218.162 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved ipc namespace via fd 18
lxc-start 300 20190531111218.162 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved net namespace via fd 19
lxc-start 300 20190531111218.162 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "memory.limit_in_bytes" set to "2147483648"
lxc-start 300 20190531111218.162 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "memory.memsw.limit_in_bytes" set to "3221225472"
lxc-start 300 20190531111218.162 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "cpu.cfs_period_us" set to "100000"
lxc-start 300 20190531111218.162 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "cpu.cfs_quota_us" set to "100000"
lxc-start 300 20190531111218.162 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "cpu.shares" set to "1024"
lxc-start 300 20190531111218.162 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "cpuset.cpus" set to "6,15"
lxc-start 300 20190531111218.162 INFO     cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2481 - Limits for the legacy cgroup hierarchies have been setup
lxc-start 300 20190531111218.162 DEBUG    start - start.c:lxc_spawn:1761 - Preserved net namespace via fd 10
lxc-start 300 20190531111218.216 DEBUG    network - network.c:lxc_network_move_created_netdev_priv:2505 - Moved network device "vethEN7LEB"/"eth0" to network namespace of 17723
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.deny" set to "a"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c *:* m"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "b *:* m"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 1:3 rwm"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 1:5 rwm"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 1:7 rwm"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 5:0 rwm"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 5:1 rwm"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 5:2 rwm"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 1:8 rwm"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 1:9 rwm"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 136:* rwm"
lxc-start 300 20190531111218.217 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 10:229 rwm"
lxc-start 300 20190531111218.217 INFO     cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2481 - Limits for the legacy cgroup hierarchies have been setup
lxc-start 300 20190531111218.217 INFO     start - start.c:do_start:1258 - Unshared CLONE_NEWCGROUP
lxc-start 300 20190531111218.217 DEBUG    storage - storage/storage.c:storage_query:253 - Detected rootfs type "dir"
lxc-start 300 20190531111218.217 DEBUG    conf - conf.c:lxc_mount_rootfs:1358 - Mounted rootfs "/var/lib/lxc/300/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
lxc-start 300 20190531111218.217 INFO     conf - conf.c:setup_utsname:817 - Set hostname to "nfs00"
lxc-start 300 20190531111218.240 DEBUG    network - network.c:setup_hw_addr:2772 - Mac address "4A:0F:39:15:3E:A0" on "eth0" has been setup
lxc-start 300 20190531111218.240 DEBUG    network - network.c:lxc_setup_netdev_in_child_namespaces:3037 - Network device "eth0" has been setup
lxc-start 300 20190531111218.240 INFO     network - network.c:lxc_setup_network_in_child_namespaces:3058 - network has been setup
lxc-start 300 20190531111218.240 INFO     conf - conf.c:mount_autodev:1144 - Preparing "/dev"
lxc-start 300 20190531111218.241 INFO     conf - conf.c:mount_autodev:1191 - Prepared "/dev"
lxc-start 300 20190531111218.241 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "300", config section "lxc"
lxc-start 300 20190531111218.255 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-autodev-hook" for container "300", config section "lxc"
lxc-start 300 20190531111218.343 INFO     conf - conf.c:lxc_fill_autodev:1235 - Populating "/dev"
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/full"
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/null"
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/random"
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/tty"
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/urandom"
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/zero"
lxc-start 300 20190531111218.343 INFO     conf - conf.c:lxc_fill_autodev:1312 - Populated "/dev"
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:mount_entry:2053 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:mount_entry:2074 - Flags for "/sys/fs/fuse/connections" were 4096, required extra flags are 0
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:mount_entry:2084 - Mountflags already were 4096, skipping remount
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:mount_entry:2128 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:mount_entry:2128 - Mounted "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/proc" with filesystem type "proc"
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:mount_entry:2128 - Mounted "sys" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/sys" with filesystem type "sysfs"
lxc-start 300 20190531111218.343 INFO     conf - conf.c:mount_file_entries:2359 - Finished setting up mounts
lxc-start 300 20190531111218.343 DEBUG    conf - conf.c:lxc_setup_ttydir_console:1822 - Created directory for console and tty devices at "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/lxc"
lxc-start 300 20190531111218.343 ERROR    conf - conf.c:lxc_setup_ttydir_console:1857 - Read-only file system - Failed to set mode "0110" to "/dev/pts/1"
lxc-start 300 20190531111218.343 ERROR    conf - conf.c:lxc_setup:3679 - Failed to setup console
lxc-start 300 20190531111218.343 ERROR    start - start.c:do_start:1279 - Failed to setup container "300"
lxc-start 300 20190531111218.343 ERROR    sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 5)
lxc-start 300 20190531111218.343 WARN     network - network.c:lxc_delete_network_priv:2594 - Operation not permitted - Failed to remove interface "eth0" with index 26
lxc-start 300 20190531111218.343 DEBUG    network - network.c:lxc_delete_network:3185 - Deleted network devices
lxc-start 300 20190531111218.343 ERROR    start - start.c:__lxc_start:1989 - Failed to spawn container "300"
lxc-start 300 20190531111218.425 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "300", config section "lxc"
lxc-start 300 20190531111219.546 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "300", config section "lxc"
lxc-start 300 20190531111219.704 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 300 20190531111219.704 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options


About PTY allocatio - thanks, cleaned that. (but thats dont help me with shutdown/start)
 
hmmm...

Code: 
lxc-start 300 20190531111218.160 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1301 - File exists - Failed to create directory "/sys/fs/cgroup/systemd//lxc/300"


run this command and try starting your container again:

Code: 
find /sys/fs/cgroup/*/lxc/CTID* -depth -type d -print -delete

if that doesn't work.. you could check your running processes for something using up a lot of cpu (there was a kworker bug at some point).

btw how did you convert your container? using the same article i've sent?
 
1. Yes, convertation was made by that article, only it was performed not on PX 4, but on PX 5.4 instead.
2. Here is command you write:
Code: 
 find /sys/fs/cgroup/*/lxc/300 -depth -type d -print -delete
/sys/fs/cgroup/blkio/lxc/300/ns
/sys/fs/cgroup/blkio/lxc/300
/sys/fs/cgroup/cpuacct/lxc/300/ns
/sys/fs/cgroup/cpuacct/lxc/300
find: ‘/sys/fs/cgroup/cpu,cpuacct/lxc/300’: No such file or directory
find: ‘/sys/fs/cgroup/cpu/lxc/300’: No such file or directory
/sys/fs/cgroup/cpuset/lxc/300/ns
/sys/fs/cgroup/cpuset/lxc/300
/sys/fs/cgroup/devices/lxc/300/ns
/sys/fs/cgroup/devices/lxc/300
/sys/fs/cgroup/freezer/lxc/300/ns
/sys/fs/cgroup/freezer/lxc/300
/sys/fs/cgroup/hugetlb/lxc/300/ns
/sys/fs/cgroup/hugetlb/lxc/300
/sys/fs/cgroup/memory/lxc/300/ns
/sys/fs/cgroup/memory/lxc/300
/sys/fs/cgroup/net_cls/lxc/300/ns
/sys/fs/cgroup/net_cls/lxc/300
find: ‘/sys/fs/cgroup/net_cls,net_prio/lxc/300’: No such file or directory
find: ‘/sys/fs/cgroup/net_prio/lxc/300’: No such file or directory
/sys/fs/cgroup/perf_event/lxc/300/ns
/sys/fs/cgroup/perf_event/lxc/300
/sys/fs/cgroup/pids/lxc/300/ns
/sys/fs/cgroup/pids/lxc/300
/sys/fs/cgroup/rdma/lxc/300/ns
/sys/fs/cgroup/rdma/lxc/300
/sys/fs/cgroup/systemd/lxc/300/ns
/sys/fs/cgroup/systemd/lxc/300

Then i try start my CT again:
Code: 
root@px00# pct list
VMID       Status     Lock         Name
300        stopped                 nfs00
root@px00:/sys/fs/cgroup# pct start 300
Job for pve-container@300.service failed because the control process exited with error code.
See "systemctl status pve-container@300.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@300' failed: exit code 1

Same result:(

About high CPU - no, host operating freely, just 2 Windows VM on it:
Code: 
top - 15:11:21 up 1 day,  1:14,  1 user,  load average: 0.54, 0.50, 0.42
Tasks: 334 total,   1 running, 189 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.2 us,  1.6 sy,  0.0 ni, 96.0 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem : 65933008 total, 51610968 free,  6903304 used,  7418736 buff/cache
KiB Swap:  3985404 total,  3985404 free,        0 used. 58357488 avail Mem
 
sorry for being unclear. can you try to start your ct with debug log again, after running the command i sent?
 
oguz said:
sorry for being unclear. can you try to start your ct with debug log again, after running the command i sent?
Click to expand...

Oh, ok
Code: 
l
lxc-start 300 20190531124322.364 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 300 20190531124322.364 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start 300 20190531124322.364 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "300", config section "lxc"
lxc-start 300 20190531124323.317 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:714 - Using terminal "/dev/tty" as proxy
lxc-start 300 20190531124323.317 DEBUG    terminal - terminal.c:lxc_terminal_signal_init:192 - Created signal fd 9
lxc-start 300 20190531124323.317 DEBUG    terminal - terminal.c:lxc_terminal_winsz:90 - Set window size to 105 columns and 32 rows
lxc-start 300 20190531124323.445 INFO     start - start.c:lxc_init:906 - Container "300" is initialized
lxc-start 300 20190531124323.445 DEBUG    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:619 - "cgroup.clone_children" was already set to "1"
lxc-start 300 20190531124323.446 INFO     cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1537 - The container uses "lxc/300" as cgroup
lxc-start 300 20190531124323.447 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWNS
lxc-start 300 20190531124323.447 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWPID
lxc-start 300 20190531124323.447 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWUTS
lxc-start 300 20190531124323.447 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWIPC
lxc-start 300 20190531124323.447 INFO     start - start.c:lxc_spawn:1707 - Cloned CLONE_NEWNET
lxc-start 300 20190531124323.447 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved mnt namespace via fd 15
lxc-start 300 20190531124323.447 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved pid namespace via fd 16
lxc-start 300 20190531124323.447 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved uts namespace via fd 17
lxc-start 300 20190531124323.447 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved ipc namespace via fd 18
lxc-start 300 20190531124323.447 DEBUG    start - start.c:lxc_try_preserve_namespaces:196 - Preserved net namespace via fd 19
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "memory.limit_in_bytes" set to "2147483648"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "memory.memsw.limit_in_bytes" set to "3221225472"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "cpu.cfs_period_us" set to "100000"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "cpu.cfs_quota_us" set to "100000"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "cpu.shares" set to "1024"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "cpuset.cpus" set to "0,3"
lxc-start 300 20190531124323.447 INFO     cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2481 - Limits for the legacy cgroup hierarchies have been setup
lxc-start 300 20190531124323.447 DEBUG    start - start.c:lxc_spawn:1761 - Preserved net namespace via fd 10
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.deny" set to "a"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c *:* m"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "b *:* m"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 1:3 rwm"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 1:5 rwm"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 1:7 rwm"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 5:0 rwm"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 5:1 rwm"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 5:2 rwm"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 1:8 rwm"
lxc-start 300 20190531124323.447 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 1:9 rwm"
lxc-start 300 20190531124323.448 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 136:* rwm"
lxc-start 300 20190531124323.448 DEBUG    cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2476 - Set controller "devices.allow" set to "c 10:229 rwm"
lxc-start 300 20190531124323.448 INFO     cgfsng - cgroups/cgfsng.c:__cg_legacy_setup_limits:2481 - Limits for the legacy cgroup hierarchies have been setup
lxc-start 300 20190531124323.449 INFO     start - start.c:do_start:1258 - Unshared CLONE_NEWCGROUP
lxc-start 300 20190531124323.449 DEBUG    storage - storage/storage.c:storage_query:253 - Detected rootfs type "dir"
lxc-start 300 20190531124323.449 DEBUG    conf - conf.c:lxc_mount_rootfs:1358 - Mounted rootfs "/var/lib/lxc/300/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
lxc-start 300 20190531124323.449 INFO     conf - conf.c:setup_utsname:817 - Set hostname to "nfs00"
lxc-start 300 20190531124323.449 INFO     conf - conf.c:mount_autodev:1144 - Preparing "/dev"
lxc-start 300 20190531124323.449 INFO     conf - conf.c:mount_autodev:1191 - Prepared "/dev"
lxc-start 300 20190531124323.449 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "300", config section "lxc"
lxc-start 300 20190531124323.463 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-autodev-hook" for container "300", config section "lxc"
lxc-start 300 20190531124323.551 INFO     conf - conf.c:lxc_fill_autodev:1235 - Populating "/dev"
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/full"
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/null"
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/random"
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/tty"
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/urandom"
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:lxc_fill_autodev:1250 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/zero"
lxc-start 300 20190531124323.551 INFO     conf - conf.c:lxc_fill_autodev:1312 - Populated "/dev"
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:mount_entry:2053 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:mount_entry:2074 - Flags for "/sys/fs/fuse/connections" were 4096, required extra flags are 0
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:mount_entry:2084 - Mountflags already were 4096, skipping remount
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:mount_entry:2128 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:mount_entry:2128 - Mounted "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/proc" with filesystem type "proc"
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:mount_entry:2128 - Mounted "sys" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/sys" with filesystem type "sysfs"
lxc-start 300 20190531124323.551 INFO     conf - conf.c:mount_file_entries:2359 - Finished setting up mounts
lxc-start 300 20190531124323.551 DEBUG    conf - conf.c:lxc_setup_ttydir_console:1822 - Created directory for console and tty devices at "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/lxc"
lxc-start 300 20190531124323.551 ERROR    conf - conf.c:lxc_setup_ttydir_console:1857 - Read-only file system - Failed to set mode "0110" to "/dev/pts/1"
lxc-start 300 20190531124323.551 ERROR    conf - conf.c:lxc_setup:3679 - Failed to setup console
lxc-start 300 20190531124323.551 ERROR    start - start.c:do_start:1279 - Failed to setup container "300"
lxc-start 300 20190531124323.552 ERROR    sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 5)
lxc-start 300 20190531124323.552 DEBUG    network - network.c:lxc_delete_network:3185 - Deleted network devices
lxc-start 300 20190531124323.552 ERROR    start - start.c:__lxc_start:1989 - Failed to spawn container "300"
lxc-start 300 20190531124323.633 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "300", config section "lxc"
lxc-start 300 20190531124324.123 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "300", config section "lxc"
lxc-start 300 20190531124324.824 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 300 20190531124324.824 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options
 
oguz said:
this[0] looks to be the same error you're getting. maybe the solution from there will work for you.

i can't think of much else.

[0]: https://forum.proxmox.com/threads/mesg-change-dev-pts-10-mode-failed-read-only-file-system.45080/
Click to expand...

Oh, try this solution.
What i have now.
Freshly rebboted node.
pct start 300 - all good.
pct shutdown 300 - all good.
pct start 300 - not working.

execute command from thread above
Code: 
mount -t devpts -o remount,gid=5,mode=620 devpts /dev/pts
root@px00:~# find /sys/fs/cgroup/*/lxc/300* -type d | tac | xargs rmdir
rmdir: failed to remove '/sys/fs/cgroup/net_cls,net_prio/lxc/300/ns': No such file or directory
rmdir: failed to remove '/sys/fs/cgroup/net_cls,net_prio/lxc/300': No such file or directory
rmdir: failed to remove '/sys/fs/cgroup/net_cls/lxc/300/ns': No such file or directory
rmdir: failed to remove '/sys/fs/cgroup/net_cls/lxc/300': No such file or directory
rmdir: failed to remove '/sys/fs/cgroup/cpu,cpuacct/lxc/300/ns': No such file or directory
rmdir: failed to remove '/sys/fs/cgroup/cpu,cpuacct/lxc/300': No such file or directory
rmdir: failed to remove '/sys/fs/cgroup/cpuacct/lxc/300/ns': No such file or directory
rmdir: failed to remove '/sys/fs/cgroup/cpuacct/lxc/300': No such file or directory
root@px00:~# pct start 300

So. Seems good. BUT. If i shutdown CT - i need to repeat @mount -t devpts -o remount,gid=5,mode=620 devpts /dev/pts@ again.
Can you help me how to emplement it to production without need to do @mount -t devpts -o remount,gid=5,mode=620 devpts /dev/pts@ each time after pct shutdown executes?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom