Greetings,
I'm new to PVE servers and proxmox and have been enjoying learning and building a homelab using Dell PowerEdge servers. I've had a pretty stable two container cluster for a while until recently when my web server no longer boots properly and I can no longer access the web pages it hosts...
I've restarted the environment and tried different settings for the container but the behavior is it "starts" and then seems to go into some kind of loop and never actually functions correctly.
I've also attempted to restore a backup I made when everything was in good order to attempt to get some functionality back, and despite completing ..I still am unable to access the console or the websites.
Please let me know what information I can provide to help with any suggestions as to how to restore this container. I'm kinda suprised that proxmox allowed it to get this bad, considering there is a working backup, but I'm very new to all of this.
Below is some log information provided by the proxmox server. Container 102 is the web server thats unable to fully online...
There is one error in the logs (at the end of the below code) that seems to be related..I'm not sure how to resolve this issue:
CT 102 - Move Volume Error: command 'mount -o ro /dev/loop0 /var/lib/lxc/102/.copy-volume-2//' failed: exit code 32
Thanks
I'm new to PVE servers and proxmox and have been enjoying learning and building a homelab using Dell PowerEdge servers. I've had a pretty stable two container cluster for a while until recently when my web server no longer boots properly and I can no longer access the web pages it hosts...
I've restarted the environment and tried different settings for the container but the behavior is it "starts" and then seems to go into some kind of loop and never actually functions correctly.
I've also attempted to restore a backup I made when everything was in good order to attempt to get some functionality back, and despite completing ..I still am unable to access the console or the websites.
Please let me know what information I can provide to help with any suggestions as to how to restore this container. I'm kinda suprised that proxmox allowed it to get this bad, considering there is a working backup, but I'm very new to all of this.
Below is some log information provided by the proxmox server. Container 102 is the web server thats unable to fully online...
There is one error in the logs (at the end of the below code) that seems to be related..I'm not sure how to resolve this issue:
CT 102 - Move Volume Error: command 'mount -o ro /dev/loop0 /var/lib/lxc/102/.copy-volume-2//' failed: exit code 32
Thanks
Code:
Update
Oct 24 16:10:31 pve kernel: device eno1 entered promiscuous mode
Oct 24 16:10:31 pve kernel: eth0: renamed from vethry1UvJ
Oct 24 16:10:31 pve pvedaemon[3117]: <root@pam> end task UPID:pve:0000118A:0000BAC4:6175D9D4:vzstart:100:root@pam: OK
Oct 24 16:10:31 pve pvestatd[3096]: modified cpu set for lxc/100: 0-5
Oct 24 16:10:34 pve kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 24 16:10:34 pve kernel: vmbr0: port 2(veth100i0) entered blocking state
Oct 24 16:10:34 pve kernel: vmbr0: port 2(veth100i0) entered forwarding state
Oct 24 16:10:37 pve kernel: overlayfs: unrecognized mount option "userxattr" or missing value
Oct 24 16:10:37 pve systemd[1]: Stopping User Manager for UID 0...
Oct 24 16:10:37 pve systemd[3297]: Stopped target Default.
Oct 24 16:10:37 pve systemd[3297]: Stopped target Basic System.
Oct 24 16:10:37 pve systemd[3297]: Stopped target Sockets.
Oct 24 16:10:37 pve systemd[3297]: gpg-agent-ssh.socket: Succeeded.
Oct 24 16:10:37 pve systemd[3297]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Oct 24 16:10:37 pve systemd[3297]: gpg-agent-browser.socket: Succeeded.
Oct 24 16:10:37 pve systemd[3297]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Oct 24 16:10:37 pve systemd[3297]: dirmngr.socket: Succeeded.
Oct 24 16:10:37 pve systemd[3297]: Closed GnuPG network certificate management daemon.
Oct 24 16:10:37 pve systemd[3297]: gpg-agent.socket: Succeeded.
Oct 24 16:10:37 pve systemd[3297]: Closed GnuPG cryptographic agent and passphrase cache.
Oct 24 16:10:37 pve systemd[3297]: gpg-agent-extra.socket: Succeeded.
Oct 24 16:10:37 pve systemd[3297]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Oct 24 16:10:37 pve systemd[3297]: Stopped target Timers.
Oct 24 16:10:37 pve systemd[3297]: Stopped target Paths.
Oct 24 16:10:37 pve systemd[3297]: Reached target Shutdown.
Oct 24 16:10:37 pve systemd[3297]: systemd-exit.service: Succeeded.
Oct 24 16:10:37 pve systemd[3297]: Started Exit the Session.
Oct 24 16:10:37 pve systemd[3297]: Reached target Exit the Session.
Oct 24 16:10:37 pve systemd[3298]: pam_unix(systemd-user:session): session closed for user root
Oct 24 16:10:37 pve systemd[1]: user@0.service: Succeeded.
Oct 24 16:10:37 pve systemd[1]: Stopped User Manager for UID 0.
Oct 24 16:10:37 pve systemd[1]: Stopping User Runtime Directory /run/user/0...
Oct 24 16:10:37 pve systemd[1]: run-user-0.mount: Succeeded.
Oct 24 16:10:37 pve systemd[1]: user-runtime-dir@0.service: Succeeded.
Oct 24 16:10:37 pve systemd[1]: Stopped User Runtime Directory /run/user/0.
Oct 24 16:10:37 pve systemd[1]: Removed slice User Slice of UID 0.
Oct 24 16:10:38 pve kernel: overlayfs: filesystem on '/var/lib/docker/check-overlayfs-support213399592/upper' not supported as upperdir
Oct 24 16:10:38 pve kernel: overlayfs: filesystem on '/var/lib/docker/check-overlayfs-support203064167/upper' not supported as upperdir
Oct 24 16:10:42 pve kernel: Initializing XFRM netlink socket
Oct 24 16:10:42 pve kernel: docker0: port 1(veth5518029) entered blocking state
Oct 24 16:10:42 pve kernel: docker0: port 1(veth5518029) entered disabled state
Oct 24 16:10:42 pve kernel: device veth5518029 entered promiscuous mode
Oct 24 16:10:42 pve kernel: docker0: port 1(veth5518029) entered blocking state
Oct 24 16:10:42 pve kernel: docker0: port 1(veth5518029) entered forwarding state
Oct 24 16:10:42 pve kernel: docker0: port 1(veth5518029) entered disabled state
Oct 24 16:10:43 pve kernel: docker0: port 2(vethbd7e55a) entered blocking state
Oct 24 16:10:43 pve kernel: docker0: port 2(vethbd7e55a) entered disabled state
Oct 24 16:10:43 pve kernel: device vethbd7e55a entered promiscuous mode
Oct 24 16:10:43 pve kernel: docker0: port 2(vethbd7e55a) entered blocking state
Oct 24 16:10:43 pve kernel: docker0: port 2(vethbd7e55a) entered forwarding state
Oct 24 16:10:43 pve kernel: docker0: port 2(vethbd7e55a) entered disabled state
Oct 24 16:10:45 pve kernel: eth0: renamed from vethb8b9ad1
Oct 24 16:10:45 pve kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth5518029: link becomes ready
Oct 24 16:10:45 pve kernel: docker0: port 1(veth5518029) entered blocking state
Oct 24 16:10:45 pve kernel: docker0: port 1(veth5518029) entered forwarding state
Oct 24 16:10:45 pve kernel: eth0: renamed from veth34939fd
Oct 24 16:10:45 pve kernel: IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
Oct 24 16:10:45 pve kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethbd7e55a: link becomes ready
Oct 24 16:10:45 pve kernel: docker0: port 2(vethbd7e55a) entered blocking state
Oct 24 16:10:45 pve kernel: docker0: port 2(vethbd7e55a) entered forwarding state
Oct 24 16:10:47 pve pvedaemon[3116]: <root@pam> starting task UPID:pve:00001EE5:0000C278:6175D9E7:vzstart:102:root@pam:
Oct 24 16:10:47 pve pvedaemon[7909]: starting CT 102: UPID:pve:00001EE5:0000C278:6175D9E7:vzstart:102:root@pam:
Oct 24 16:10:48 pve systemd[1]: Started PVE LXC Container: 102.
Oct 24 16:10:50 pve kernel: EXT4-fs warning (device loop0): ext4_multi_mount_protect:322: MMP interval 42 higher than expected, please wait.
Oct 24 16:11:00 pve systemd[1]: Starting Proxmox VE replication runner...
Oct 24 16:11:01 pve systemd[1]: pvesr.service: Succeeded.
Oct 24 16:11:01 pve systemd[1]: Started Proxmox VE replication runner.
Oct 24 16:11:34 pve kernel: EXT4-fs (loop0): recovery complete
Oct 24 16:11:34 pve kernel: EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Oct 24 16:11:35 pve kernel: EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null)
Oct 24 16:11:36 pve audit[10089]: AVC apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=10089 comm="apparmor_parser"
Oct 24 16:11:36 pve kernel: audit: type=1400 audit(1635113496.205:19): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=10089 comm="apparmor_parser"
Oct 24 16:11:36 pve systemd-udevd[10076]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 24 16:11:36 pve systemd-udevd[10076]: Using default interface naming scheme 'v240'.
Oct 24 16:11:37 pve kernel: vmbr0: port 3(veth102i0) entered blocking state
Oct 24 16:11:37 pve kernel: vmbr0: port 3(veth102i0) entered disabled state
Oct 24 16:11:37 pve kernel: device veth102i0 entered promiscuous mode
Oct 24 16:11:37 pve kernel: eth0: renamed from vethJmp376
Oct 24 16:11:37 pve systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 10096 (lxc-start)
Oct 24 16:11:37 pve systemd[1]: Mounting Arbitrary Executable File Formats File System...
Oct 24 16:11:37 pve systemd[1]: Mounted Arbitrary Executable File Formats File System.
Oct 24 16:11:37 pve pvedaemon[7909]: startup for container '102' failed
Oct 24 16:11:37 pve pvedaemon[3116]: <root@pam> end task UPID:pve:00001EE5:0000C278:6175D9E7:vzstart:102:root@pam: startup for container '102' failed
Oct 24 16:11:38 pve audit[10165]: AVC apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-102_</var/lib/lxc>" pid=10165 comm="apparmor_parser"
Logs
TasksCluster log
Start Time
End Time
Node
User name
Description
Status
Oct 24 20:58:34 Oct 24 20:58:41 pve root@pam VM/CT 102 - Console OK
Oct 24 19:54:56 Oct 24 19:55:01 pve root@pam VM/CT 102 - Console OK
Oct 24 18:30:59 Oct 24 18:31:03 pve root@pam VM/CT 102 - Console OK
Oct 24 18:07:06 Oct 24 18:07:12 pve root@pam VM/CT 102 - Console OK
Oct 24 18:05:16 Oct 24 18:05:19 pve root@pam CT 102 - Start OK
Oct 24 17:59:07 Oct 24 18:01:42 pve root@pam CT 102 - Restore OK
Oct 24 17:57:37 Oct 24 17:58:25 pve root@pam CT 102 - Start Error: startup for container '102' failed
Oct 24 16:12:00 Oct 24 17:25:01 pve root@pam CT 102 - Move Volume OK
Oct 24 16:10:47 Oct 24 16:11:37 pve root@pam CT 102 - Start Error: startup for container '102' failed
Oct 24 16:10:28 Oct 24 16:10:31 pve root@pam CT 100 - Start OK
Oct 24 16:10:18 Oct 24 16:10:27 pve root@pam Shell OK
Oct 24 16:10:11 Oct 24 16:10:13 pve root@pam Shell OK
Oct 24 16:03:44 Oct 24 16:10:05 pve root@pam Shell OK
Oct 24 16:03:07 Oct 24 16:10:26 pve root@pam Start all VMs and Containers OK
Oct 24 15:58:48 Oct 24 15:58:57 pve root@pam Stop all VMs and Containers OK
Oct 24 15:58:48 Oct 24 15:58:57 pve root@pam CT 100 - Shutdown OK
Oct 24 15:58:26 Oct 24 15:58:45 pve root@pam Shell OK
Oct 24 15:57:26 Oct 24 15:58:13 pve root@pam CT 102 - Start Error: startup for container '102' failed
Oct 24 15:56:47 Oct 24 15:56:50 pve root@pam CT 102 - Move Volume Error: command 'mount -o ro /dev/loop0 /var/lib/lxc/102/.copy-volume-2//' failed: exit code 32
Oct 24 15:56:23 Oct 24 15:56:23 pve root@pam VM/CT 102 - Console OK
Oct 24 15:55:41 Oct 24 15:55:41 pve root@pam VM/CT 102 - Console OK
Oct 24 15:54:34 Oct 24 15:55:21 pve root@pam CT 102 - Start Error: startup for container '102' failed
Oct 24 14:35:01 Oct 24 15:38:16 pve root@pam CT 102 - Restore OK
Oct 24 13:51:13 Oct 24 13:52:00 pve root@pam CT 102 - Start Error: startup for container '102' failed
Oct 24 13:49:52 Oct 24 13:50:40 pve root@pam CT 102 - Start Error: startup for container '102' failed
()