Hello everyone,
I just installed Proxmox the past week and I'm having some issues upgrading some packages, performance load on a single core, and systemd is unresponsive.
If I remember correctly, this started after performing
Basically, systemd could not upgrade successfully, and after rebooting the machine, ssh was taking so long to connect and journalctl showed the something like the following:
After waiting for 3 to 5 minutes to connect, I could issue some commands, but auto-complete [Tab] takes a long time too to show the result. Systemctl commands are also unresponsive (taking too long and does not execute system reboot, shutdown ... ). Here I tried to initiate the upgrade a second time, and I got the following:
And right now udev is stuck on this state (not fully installed or removed.), and getting invoked with every apt command.
I suspected a broken filesystem, but after looking on the fsck man page, it seems that the repair must be done while the partition is unmounted, and this is not possible for me since I'm connecting to the root partition remotely.
Here is the output of pveversion:
Please help me if possible.
I just installed Proxmox the past week and I'm having some issues upgrading some packages, performance load on a single core, and systemd is unresponsive.
If I remember correctly, this started after performing
apt update && apt upgrade
Code:
~# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
bind9-dnsutils bind9-host bind9-libs bsdextrautils chrony curl dirmngr distro-info-data eject gnupg gnupg-l10n gnupg-utils gpg gpg-agent
gpg-wks-client gpg-wks-server gpgconf gpgsm gpgv libarchive13 libc-bin libc-l10n libcryptsetup12 libcurl3-gnutls libcurl4 libexpat1 libflac8
libgmp10 liblzma5 libnss-systemd libnss3 libpam-systemd libsasl2-2 libsasl2-modules-db libseccomp2 libsmbclient libssl1.1 libsystemd0 libudev1
libwbclient0 libxml2 locales mount openssl samba-common samba-libs smbclient systemd systemd-sysv sysvinit-utils tasksel tasksel-data tzdata udev
vim-common vim-tiny wget xxd xz-utils zlib1g
60 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
Need to get 40.7 MB of archives.
After this operation, 37.9 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ftp.debian.org/debian bullseye/main amd64 libc-bin amd64 2.31-13+deb11u3 [821 kB]
Get:2 http://ftp.debian.org/debian bullseye/main amd64 sysvinit-utils amd64 2.96-7+deb11u1 [25.6 kB]
Get:3 http://ftp.debian.org/debian bullseye/main amd64 libnss-systemd amd64 247.3-7 [198 kB]
Get:4 http://ftp.debian.org/debian bullseye/main amd64 libsystemd0 amd64 247.3-7 [376 kB]
Get:5 http://ftp.debian.org/debian bullseye/main amd64 libpam-systemd amd64 247.3-7 [283 kB]
Get:6 http://ftp.debian.org/debian bullseye/main amd64 systemd amd64 247.3-7 [4,500 kB]
Get:7 http://security.debian.org bullseye-security/main amd64 liblzma5 amd64 5.2.5-2.1~deb11u1 [168 kB]
Get:8 http://security.debian.org bullseye-security/main amd64 zlib1g amd64 1:1.2.11.dfsg-2+deb11u1 [91.4 kB]
Get:9 http://security.debian.org bullseye-security/main amd64 xz-utils amd64 5.2.5-2.1~deb11u1 [220 kB]
Get:10 http://ftp.debian.org/debian bullseye/main amd64 udev amd64 247.3-7 [1,464 kB]
Get:11 http://ftp.debian.org/debian bullseye/main amd64 libudev1 amd64 247.3-7 [168 kB]
Get:12 http://ftp.debian.org/debian bullseye/main amd64 systemd-sysv amd64 247.3-7 [113 kB]
Get:13 http://ftp.debian.org/debian bullseye/main amd64 libssl1.1 amd64 1.1.1n-0+deb11u1 [1,557 kB]
Get:14 http://ftp.debian.org/debian bullseye/main amd64 libcryptsetup12 amd64 2:2.3.7-1+deb11u1 [248 kB]
Get:15 http://ftp.debian.org/debian bullseye/main amd64 libseccomp2 amd64 2.5.1-1+deb11u1 [48.5 kB]
Get:16 http://ftp.debian.org/debian bullseye/main amd64 mount amd64 2.36.1-8+deb11u1 [186 kB]
Get:17 http://ftp.debian.org/debian bullseye/main amd64 bsdextrautils amd64 2.36.1-8+deb11u1 [145 kB]
Get:18 http://ftp.debian.org/debian bullseye-updates/main amd64 tzdata all 2021a-1+deb11u3 [285 kB]
Get:19 http://ftp.debian.org/debian bullseye/main amd64 chrony amd64 4.0-8+deb11u2 [287 kB]
Get:20 http://ftp.debian.org/debian bullseye/main amd64 smbclient amd64 2:4.13.13+dfsg-1~deb11u3 [510 kB]
Get:21 http://ftp.debian.org/debian bullseye/main amd64 samba-common all 2:4.13.13+dfsg-1~deb11u3 [170 kB]
Get:22 http://ftp.debian.org/debian bullseye/main amd64 libxml2 amd64 2.9.10+dfsg-6.7+deb11u1 [693 kB]
Get:23 http://ftp.debian.org/debian bullseye/main amd64 libarchive13 amd64 3.4.3-2+deb11u1 [343 kB]
Get:24 http://ftp.debian.org/debian bullseye/main amd64 libsmbclient amd64 2:4.13.13+dfsg-1~deb11u3 [169 kB]
Get:25 http://ftp.debian.org/debian bullseye/main amd64 samba-libs amd64 2:4.13.13+dfsg-1~deb11u3 [5,771 kB]
Get:26 http://ftp.debian.org/debian bullseye/main amd64 libwbclient0 amd64 2:4.13.13+dfsg-1~deb11u3 [312 kB]
Get:27 http://ftp.debian.org/debian bullseye/main amd64 tasksel-data all 3.68+deb11u1 [18.0 kB]
Get:28 http://ftp.debian.org/debian bullseye/main amd64 tasksel all 3.68+deb11u1 [101 kB]
Get:29 http://ftp.debian.org/debian bullseye/main amd64 gpgv amd64 2.2.27-2+deb11u1 [626 kB]
Get:30 http://ftp.debian.org/debian bullseye/main amd64 libgmp10 amd64 2:6.2.1+dfsg-1+deb11u1 [563 kB]
Get:31 http://ftp.debian.org/debian bullseye/main amd64 vim-tiny amd64 2:8.2.2434-3+deb11u1 [744 kB]
Get:32 http://ftp.debian.org/debian bullseye/main amd64 xxd amd64 2:8.2.2434-3+deb11u1 [192 kB]
Get:33 http://ftp.debian.org/debian bullseye/main amd64 vim-common all 2:8.2.2434-3+deb11u1 [226 kB]
Get:34 http://ftp.debian.org/debian bullseye/main amd64 bind9-dnsutils amd64 1:9.16.27-1~deb11u1 [398 kB]
Get:35 http://ftp.debian.org/debian bullseye/main amd64 bind9-libs amd64 1:9.16.27-1~deb11u1 [1,413 kB]
Get:36 http://ftp.debian.org/debian bullseye/main amd64 bind9-host amd64 1:9.16.27-1~deb11u1 [302 kB]
Get:37 http://ftp.debian.org/debian bullseye/main amd64 libc-l10n all 2.31-13+deb11u3 [863 kB]
Get:38 http://ftp.debian.org/debian bullseye/main amd64 locales all 2.31-13+deb11u3 [4,084 kB]
Get:39 http://ftp.debian.org/debian bullseye/main amd64 wget amd64 1.21-1+deb11u1 [964 kB]
Get:40 http://ftp.debian.org/debian bullseye/main amd64 curl amd64 7.74.0-1.3+deb11u1 [267 kB]
Get:41 http://ftp.debian.org/debian bullseye/main amd64 libcurl4 amd64 7.74.0-1.3+deb11u1 [341 kB]
Get:42 http://ftp.debian.org/debian bullseye/main amd64 gpgsm amd64 2.2.27-2+deb11u1 [645 kB]
Get:43 http://ftp.debian.org/debian bullseye/main amd64 gpg-wks-client amd64 2.2.27-2+deb11u1 [524 kB]
Get:44 http://ftp.debian.org/debian bullseye/main amd64 gpg-wks-server amd64 2.2.27-2+deb11u1 [516 kB]
Get:45 http://ftp.debian.org/debian bullseye/main amd64 gpg amd64 2.2.27-2+deb11u1 [928 kB]
Get:46 http://ftp.debian.org/debian bullseye/main amd64 gnupg-utils amd64 2.2.27-2+deb11u1 [905 kB]
Get:47 http://ftp.debian.org/debian bullseye/main amd64 gnupg-l10n all 2.2.27-2+deb11u1 [1,085 kB]
Get:48 http://ftp.debian.org/debian bullseye/main amd64 dirmngr amd64 2.2.27-2+deb11u1 [763 kB]
Get:49 http://ftp.debian.org/debian bullseye/main amd64 gnupg all 2.2.27-2+deb11u1 [825 kB]
Get:50 http://ftp.debian.org/debian bullseye/main amd64 gpg-agent amd64 2.2.27-2+deb11u1 [669 kB]
Get:51 http://ftp.debian.org/debian bullseye/main amd64 gpgconf amd64 2.2.27-2+deb11u1 [548 kB]
Get:52 http://ftp.debian.org/debian bullseye/main amd64 distro-info-data all 0.51+deb11u1 [7,624 B]
Get:53 http://ftp.debian.org/debian bullseye/main amd64 eject amd64 2.36.1-8+deb11u1 [102 kB]
Get:54 http://ftp.debian.org/debian bullseye/main amd64 libcurl3-gnutls amd64 7.74.0-1.3+deb11u1 [338 kB]
Get:55 http://ftp.debian.org/debian bullseye/main amd64 libexpat1 amd64 2.2.10-2+deb11u3 [98.5 kB]
Get:56 http://ftp.debian.org/debian bullseye/main amd64 libflac8 amd64 1.3.3-2+deb11u1 [112 kB]
Get:57 http://ftp.debian.org/debian bullseye/main amd64 libnss3 amd64 2:3.61-1+deb11u2 [1,306 kB]
Get:58 http://ftp.debian.org/debian bullseye/main amd64 libsasl2-modules-db amd64 2.1.27+dfsg-2.1+deb11u1 [69.1 kB]
Get:59 http://ftp.debian.org/debian bullseye/main amd64 libsasl2-2 amd64 2.1.27+dfsg-2.1+deb11u1 [106 kB]
Get:60 http://ftp.debian.org/debian bullseye/main amd64 openssl amd64 1.1.1n-0+deb11u1 [853 kB]
Fetched 40.7 MB in 43s (939 kB/s)
Reading changelogs... Done
Extracting templates from packages: 100%
Preconfiguring packages ...
(Reading database ... 42884 files and directories currently installed.)
Preparing to unpack .../libc-bin_2.31-13+deb11u3_amd64.deb ...
Unpacking libc-bin (2.31-13+deb11u3) over (2.31-13+deb11u2) ...
Setting up libc-bin (2.31-13+deb11u3) ...
(Reading database ... 42884 files and directories currently installed.)
Preparing to unpack .../sysvinit-utils_2.96-7+deb11u1_amd64.deb ...
Unpacking sysvinit-utils (2.96-7+deb11u1) over (2.96-7) ...
Setting up sysvinit-utils (2.96-7+deb11u1) ...
(Reading database ... 42884 files and directories currently installed.)
Preparing to unpack .../libnss-systemd_247.3-7_amd64.deb ...
Unpacking libnss-systemd:amd64 (247.3-7) over (247.3-6) ...
Preparing to unpack .../libsystemd0_247.3-7_amd64.deb ...
Unpacking libsystemd0:amd64 (247.3-7) over (247.3-6) ...
Setting up libsystemd0:amd64 (247.3-7) ...
(Reading database ... 42884 files and directories currently installed.)
Preparing to unpack .../libpam-systemd_247.3-7_amd64.deb ...
Unpacking libpam-systemd:amd64 (247.3-7) over (247.3-6) ...
Preparing to unpack .../systemd_247.3-7_amd64.deb ...
Unpacking systemd (247.3-7) over (247.3-6) ...
Preparing to unpack .../udev_247.3-7_amd64.deb ...
Unpacking udev (247.3-7) over (247.3-6) ...
Preparing to unpack .../libudev1_247.3-7_amd64.deb ...
Unpacking libudev1:amd64 (247.3-7) over (247.3-6) ...
Setting up libudev1:amd64 (247.3-7) ...
Setting up systemd (247.3-7) ...
Failed to reload daemon: Transport endpoint is not connectedBasically, systemd could not upgrade successfully, and after rebooting the machine, ssh was taking so long to connect and journalctl showed the something like the following:
Code:
May 10 00:41:00 tower sshd[8798]: Accepted password for root from 192.168.1.2 port 55522 ssh2
May 10 00:41:00 tower sshd[8798]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
May 10 00:42:55 tower systemd-logind[792]: Failed to start user service 'user@0.service', ignoring: Connection timed out
May 10 00:43:20 tower systemd-logind[792]: Failed to start session scope session-2.scope: Connection timed out
May 10 00:43:20 tower sshd[8798]: pam_systemd(sshd:session): Failed to create session: Connection timed out
May 10 00:43:55 tower systemd-logind[792]: Failed to stop user service 'user@0.service', ignoring: Connection timed out
May 10 00:52:41 tower smartd[788]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
May 10 01:06:53 tower sshd[8798]: Received disconnect from 192.168.1.2 port 55522:11: disconnected by user
May 10 01:06:53 tower sshd[8798]: Disconnected from user root 192.168.1.2 port 55522After waiting for 3 to 5 minutes to connect, I could issue some commands, but auto-complete [Tab] takes a long time too to show the result. Systemctl commands are also unresponsive (taking too long and does not execute system reboot, shutdown ... ). Here I tried to initiate the upgrade a second time, and I got the following:
Code:
apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up udev (247.3-7) ...
Failed to reload daemon: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
invoke-rc.d: could not determine current runlevel
Failed to restart udev.service: Transport endpoint is not connected
See system logs and 'systemctl status udev.service' for details.
invoke-rc.d: initscript udev, action "restart" failed.
Failed to get properties: Transport endpoint is not connected
dpkg: error processing package udev (--configure):
installed udev package post-installation script subprocess returned error exit status 1
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.13.19-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
Errors were encountered while processing:
udev
E: Sub-process /usr/bin/dpkg returned an error code (1)And right now udev is stuck on this state (not fully installed or removed.), and getting invoked with every apt command.
I suspected a broken filesystem, but after looking on the fsck man page, it seems that the repair must be done while the partition is unmounted, and this is not possible for me since I'm connecting to the root partition remotely.
Here is the output of pveversion:
Code:
# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3Please help me if possible.
