broken system files after reboot

sudosu

New Member
May 9, 2022
8
0
1
Hello everyone,

I just installed Proxmox the past week and I'm having some issues upgrading some packages, performance load on a single core, and systemd is unresponsive.
If I remember correctly, this started after performing apt update && apt upgrade

Code:
~# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  bind9-dnsutils bind9-host bind9-libs bsdextrautils chrony curl dirmngr distro-info-data eject gnupg gnupg-l10n gnupg-utils gpg gpg-agent
  gpg-wks-client gpg-wks-server gpgconf gpgsm gpgv libarchive13 libc-bin libc-l10n libcryptsetup12 libcurl3-gnutls libcurl4 libexpat1 libflac8
  libgmp10 liblzma5 libnss-systemd libnss3 libpam-systemd libsasl2-2 libsasl2-modules-db libseccomp2 libsmbclient libssl1.1 libsystemd0 libudev1
  libwbclient0 libxml2 locales mount openssl samba-common samba-libs smbclient systemd systemd-sysv sysvinit-utils tasksel tasksel-data tzdata udev
  vim-common vim-tiny wget xxd xz-utils zlib1g
60 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
2 not fully installed or removed.
Need to get 40.7 MB of archives.
After this operation, 37.9 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ftp.debian.org/debian bullseye/main amd64 libc-bin amd64 2.31-13+deb11u3 [821 kB]
Get:2 http://ftp.debian.org/debian bullseye/main amd64 sysvinit-utils amd64 2.96-7+deb11u1 [25.6 kB]
Get:3 http://ftp.debian.org/debian bullseye/main amd64 libnss-systemd amd64 247.3-7 [198 kB]
Get:4 http://ftp.debian.org/debian bullseye/main amd64 libsystemd0 amd64 247.3-7 [376 kB]
Get:5 http://ftp.debian.org/debian bullseye/main amd64 libpam-systemd amd64 247.3-7 [283 kB]
Get:6 http://ftp.debian.org/debian bullseye/main amd64 systemd amd64 247.3-7 [4,500 kB]
Get:7 http://security.debian.org bullseye-security/main amd64 liblzma5 amd64 5.2.5-2.1~deb11u1 [168 kB]
Get:8 http://security.debian.org bullseye-security/main amd64 zlib1g amd64 1:1.2.11.dfsg-2+deb11u1 [91.4 kB]
Get:9 http://security.debian.org bullseye-security/main amd64 xz-utils amd64 5.2.5-2.1~deb11u1 [220 kB]
Get:10 http://ftp.debian.org/debian bullseye/main amd64 udev amd64 247.3-7 [1,464 kB]                                                               
Get:11 http://ftp.debian.org/debian bullseye/main amd64 libudev1 amd64 247.3-7 [168 kB]                                                             
Get:12 http://ftp.debian.org/debian bullseye/main amd64 systemd-sysv amd64 247.3-7 [113 kB]                                                         
Get:13 http://ftp.debian.org/debian bullseye/main amd64 libssl1.1 amd64 1.1.1n-0+deb11u1 [1,557 kB]                                                 
Get:14 http://ftp.debian.org/debian bullseye/main amd64 libcryptsetup12 amd64 2:2.3.7-1+deb11u1 [248 kB]                                             
Get:15 http://ftp.debian.org/debian bullseye/main amd64 libseccomp2 amd64 2.5.1-1+deb11u1 [48.5 kB]                                                 
Get:16 http://ftp.debian.org/debian bullseye/main amd64 mount amd64 2.36.1-8+deb11u1 [186 kB]                                                       
Get:17 http://ftp.debian.org/debian bullseye/main amd64 bsdextrautils amd64 2.36.1-8+deb11u1 [145 kB]                                               
Get:18 http://ftp.debian.org/debian bullseye-updates/main amd64 tzdata all 2021a-1+deb11u3 [285 kB]                                                 
Get:19 http://ftp.debian.org/debian bullseye/main amd64 chrony amd64 4.0-8+deb11u2 [287 kB]                                                         
Get:20 http://ftp.debian.org/debian bullseye/main amd64 smbclient amd64 2:4.13.13+dfsg-1~deb11u3 [510 kB]                                           
Get:21 http://ftp.debian.org/debian bullseye/main amd64 samba-common all 2:4.13.13+dfsg-1~deb11u3 [170 kB]                                           
Get:22 http://ftp.debian.org/debian bullseye/main amd64 libxml2 amd64 2.9.10+dfsg-6.7+deb11u1 [693 kB]                                               
Get:23 http://ftp.debian.org/debian bullseye/main amd64 libarchive13 amd64 3.4.3-2+deb11u1 [343 kB]                                                 
Get:24 http://ftp.debian.org/debian bullseye/main amd64 libsmbclient amd64 2:4.13.13+dfsg-1~deb11u3 [169 kB]                                         
Get:25 http://ftp.debian.org/debian bullseye/main amd64 samba-libs amd64 2:4.13.13+dfsg-1~deb11u3 [5,771 kB]                                         
Get:26 http://ftp.debian.org/debian bullseye/main amd64 libwbclient0 amd64 2:4.13.13+dfsg-1~deb11u3 [312 kB]                                         
Get:27 http://ftp.debian.org/debian bullseye/main amd64 tasksel-data all 3.68+deb11u1 [18.0 kB]                                                     
Get:28 http://ftp.debian.org/debian bullseye/main amd64 tasksel all 3.68+deb11u1 [101 kB]                                                           
Get:29 http://ftp.debian.org/debian bullseye/main amd64 gpgv amd64 2.2.27-2+deb11u1 [626 kB]                                                         
Get:30 http://ftp.debian.org/debian bullseye/main amd64 libgmp10 amd64 2:6.2.1+dfsg-1+deb11u1 [563 kB]                                               
Get:31 http://ftp.debian.org/debian bullseye/main amd64 vim-tiny amd64 2:8.2.2434-3+deb11u1 [744 kB]                                                 
Get:32 http://ftp.debian.org/debian bullseye/main amd64 xxd amd64 2:8.2.2434-3+deb11u1 [192 kB]                                                     
Get:33 http://ftp.debian.org/debian bullseye/main amd64 vim-common all 2:8.2.2434-3+deb11u1 [226 kB]                                                 
Get:34 http://ftp.debian.org/debian bullseye/main amd64 bind9-dnsutils amd64 1:9.16.27-1~deb11u1 [398 kB]                                           
Get:35 http://ftp.debian.org/debian bullseye/main amd64 bind9-libs amd64 1:9.16.27-1~deb11u1 [1,413 kB]                                             
Get:36 http://ftp.debian.org/debian bullseye/main amd64 bind9-host amd64 1:9.16.27-1~deb11u1 [302 kB]                                               
Get:37 http://ftp.debian.org/debian bullseye/main amd64 libc-l10n all 2.31-13+deb11u3 [863 kB]                                                       
Get:38 http://ftp.debian.org/debian bullseye/main amd64 locales all 2.31-13+deb11u3 [4,084 kB]                                                       
Get:39 http://ftp.debian.org/debian bullseye/main amd64 wget amd64 1.21-1+deb11u1 [964 kB]                                                           
Get:40 http://ftp.debian.org/debian bullseye/main amd64 curl amd64 7.74.0-1.3+deb11u1 [267 kB]                                                       
Get:41 http://ftp.debian.org/debian bullseye/main amd64 libcurl4 amd64 7.74.0-1.3+deb11u1 [341 kB]                                                   
Get:42 http://ftp.debian.org/debian bullseye/main amd64 gpgsm amd64 2.2.27-2+deb11u1 [645 kB]                                                       
Get:43 http://ftp.debian.org/debian bullseye/main amd64 gpg-wks-client amd64 2.2.27-2+deb11u1 [524 kB]                                               
Get:44 http://ftp.debian.org/debian bullseye/main amd64 gpg-wks-server amd64 2.2.27-2+deb11u1 [516 kB]                                               
Get:45 http://ftp.debian.org/debian bullseye/main amd64 gpg amd64 2.2.27-2+deb11u1 [928 kB]                                                         
Get:46 http://ftp.debian.org/debian bullseye/main amd64 gnupg-utils amd64 2.2.27-2+deb11u1 [905 kB]                                                 
Get:47 http://ftp.debian.org/debian bullseye/main amd64 gnupg-l10n all 2.2.27-2+deb11u1 [1,085 kB]                                                   
Get:48 http://ftp.debian.org/debian bullseye/main amd64 dirmngr amd64 2.2.27-2+deb11u1 [763 kB]                                                     
Get:49 http://ftp.debian.org/debian bullseye/main amd64 gnupg all 2.2.27-2+deb11u1 [825 kB]                                                         
Get:50 http://ftp.debian.org/debian bullseye/main amd64 gpg-agent amd64 2.2.27-2+deb11u1 [669 kB]                                                   
Get:51 http://ftp.debian.org/debian bullseye/main amd64 gpgconf amd64 2.2.27-2+deb11u1 [548 kB]                                                     
Get:52 http://ftp.debian.org/debian bullseye/main amd64 distro-info-data all 0.51+deb11u1 [7,624 B]                                                 
Get:53 http://ftp.debian.org/debian bullseye/main amd64 eject amd64 2.36.1-8+deb11u1 [102 kB]                                                       
Get:54 http://ftp.debian.org/debian bullseye/main amd64 libcurl3-gnutls amd64 7.74.0-1.3+deb11u1 [338 kB]                                           
Get:55 http://ftp.debian.org/debian bullseye/main amd64 libexpat1 amd64 2.2.10-2+deb11u3 [98.5 kB]                                                   
Get:56 http://ftp.debian.org/debian bullseye/main amd64 libflac8 amd64 1.3.3-2+deb11u1 [112 kB]                                                     
Get:57 http://ftp.debian.org/debian bullseye/main amd64 libnss3 amd64 2:3.61-1+deb11u2 [1,306 kB]                                                   
Get:58 http://ftp.debian.org/debian bullseye/main amd64 libsasl2-modules-db amd64 2.1.27+dfsg-2.1+deb11u1 [69.1 kB]                                 
Get:59 http://ftp.debian.org/debian bullseye/main amd64 libsasl2-2 amd64 2.1.27+dfsg-2.1+deb11u1 [106 kB]                                           
Get:60 http://ftp.debian.org/debian bullseye/main amd64 openssl amd64 1.1.1n-0+deb11u1 [853 kB]                                                     
Fetched 40.7 MB in 43s (939 kB/s)                                                                                                                   
Reading changelogs... Done
Extracting templates from packages: 100%
Preconfiguring packages ...
(Reading database ... 42884 files and directories currently installed.)
Preparing to unpack .../libc-bin_2.31-13+deb11u3_amd64.deb ...
Unpacking libc-bin (2.31-13+deb11u3) over (2.31-13+deb11u2) ...
Setting up libc-bin (2.31-13+deb11u3) ...
(Reading database ... 42884 files and directories currently installed.)
Preparing to unpack .../sysvinit-utils_2.96-7+deb11u1_amd64.deb ...
Unpacking sysvinit-utils (2.96-7+deb11u1) over (2.96-7) ...
Setting up sysvinit-utils (2.96-7+deb11u1) ...
(Reading database ... 42884 files and directories currently installed.)
Preparing to unpack .../libnss-systemd_247.3-7_amd64.deb ...
Unpacking libnss-systemd:amd64 (247.3-7) over (247.3-6) ...
Preparing to unpack .../libsystemd0_247.3-7_amd64.deb ...
Unpacking libsystemd0:amd64 (247.3-7) over (247.3-6) ...
Setting up libsystemd0:amd64 (247.3-7) ...
(Reading database ... 42884 files and directories currently installed.)
Preparing to unpack .../libpam-systemd_247.3-7_amd64.deb ...
Unpacking libpam-systemd:amd64 (247.3-7) over (247.3-6) ...
Preparing to unpack .../systemd_247.3-7_amd64.deb ...
Unpacking systemd (247.3-7) over (247.3-6) ...
Preparing to unpack .../udev_247.3-7_amd64.deb ...
Unpacking udev (247.3-7) over (247.3-6) ...
Preparing to unpack .../libudev1_247.3-7_amd64.deb ...
Unpacking libudev1:amd64 (247.3-7) over (247.3-6) ...
Setting up libudev1:amd64 (247.3-7) ...
Setting up systemd (247.3-7) ...
Failed to reload daemon: Transport endpoint is not connected

Basically, systemd could not upgrade successfully, and after rebooting the machine, ssh was taking so long to connect and journalctl showed the something like the following:
Code:
May 10 00:41:00 tower sshd[8798]: Accepted password for root from 192.168.1.2 port 55522 ssh2
May 10 00:41:00 tower sshd[8798]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
May 10 00:42:55 tower systemd-logind[792]: Failed to start user service 'user@0.service', ignoring: Connection timed out
May 10 00:43:20 tower systemd-logind[792]: Failed to start session scope session-2.scope: Connection timed out
May 10 00:43:20 tower sshd[8798]: pam_systemd(sshd:session): Failed to create session: Connection timed out
May 10 00:43:55 tower systemd-logind[792]: Failed to stop user service 'user@0.service', ignoring: Connection timed out
May 10 00:52:41 tower smartd[788]: Device: /dev/sdc [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 61
May 10 01:06:53 tower sshd[8798]: Received disconnect from 192.168.1.2 port 55522:11: disconnected by user
May 10 01:06:53 tower sshd[8798]: Disconnected from user root 192.168.1.2 port 55522

After waiting for 3 to 5 minutes to connect, I could issue some commands, but auto-complete [Tab] takes a long time too to show the result. Systemctl commands are also unresponsive (taking too long and does not execute system reboot, shutdown ... ). Here I tried to initiate the upgrade a second time, and I got the following:
Code:
apt-get upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up udev (247.3-7) ...
Failed to reload daemon: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
invoke-rc.d: could not determine current runlevel

Failed to restart udev.service: Transport endpoint is not connected
See system logs and 'systemctl status udev.service' for details.
invoke-rc.d: initscript udev, action "restart" failed.
Failed to get properties: Transport endpoint is not connected
dpkg: error processing package udev (--configure):
 installed udev package post-installation script subprocess returned error exit status 1
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.13.19-2-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
Errors were encountered while processing:
 udev
E: Sub-process /usr/bin/dpkg returned an error code (1)

And right now udev is stuck on this state (not fully installed or removed.), and getting invoked with every apt command.

I suspected a broken filesystem, but after looking on the fsck man page, it seems that the repair must be done while the partition is unmounted, and this is not possible for me since I'm connecting to the root partition remotely.

Here is the output of pveversion:
Code:
# pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-7 (running version: 7.1-7/df5740ad)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-2
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3

Please help me if possible.
 
  1. Did you setup a PVE repository? If you don't have a subscription, you want the no-subscription repo [1] and disable the enterprise one [2] (read the fine print there).
  2. Never use apt upgrade with Proxmox products; ever use apt full-upgrade (or apt dist-upgrade)! [3]
After configuring a valid PVE repo, you could try again with first apt update and then apt full-upgrade.
If there are errors, post the full output of both commands here.

But to be honest, I'm not sure if this will work/help in your current state. Maybe you want to wait for other answers.

[1] https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_no_subscription_repo
[2] https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_enterprise_repo
[3] https://forum.proxmox.com/threads/proxmox-ve-7-1-released.99847/page-8#post-463941
 
Hello Neobin,

I just performed the first step, and when updating and then upgrading I got the following errors:
Code:
# apt update
Hit:1 http://ftp.debian.org/debian bullseye InRelease
Hit:2 http://security.debian.org bullseye-security InRelease
Hit:3 http://download.proxmox.com/debian/pve bullseye InRelease
Hit:4 http://ftp.debian.org/debian bullseye-updates InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
45 packages can be upgraded. Run 'apt list --upgradable' to see them.
 
Code:
# apt full-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  libdrm-common libdrm2 libepoxy0 libgbm1 libproxmox-rs-perl libvirglrenderer1 libwayland-server0 proxmox-websocket-tunnel pve-kernel-5.13.19-6-pve
  pve-kernel-5.15 pve-kernel-5.15.35-1-pve
The following packages will be upgraded:
  btrfs-progs libnvpair3linux libproxmox-acme-perl libproxmox-acme-plugins libpve-access-control libpve-cluster-api-perl libpve-cluster-perl
  libpve-common-perl libpve-guest-common-perl libpve-http-server-perl libpve-rs-perl libpve-storage-perl libpve-u2f-server-perl libtpms0
  libuutil3linux libzfs4linux libzpool5linux lxc-pve lxcfs novnc-pve proxmox-backup-client proxmox-backup-file-restore proxmox-ve
  proxmox-widget-toolkit pve-cluster pve-container pve-docs pve-firmware pve-ha-manager pve-i18n pve-kernel-5.13 pve-kernel-helper pve-lxc-syscalld
  pve-manager pve-qemu-kvm pve-xtermjs qemu-server smartmontools spl swtpm swtpm-libs swtpm-tools zfs-initramfs zfs-zed zfsutils-linux
45 upgraded, 11 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 262 MB of archives.
After this operation, 756 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://download.proxmox.com/debian/pve bullseye/pve-no-subscription amd64 btrfs-progs amd64 5.16.2-1~bpo11+1 [725 kB]
Get:2 http://ftp.debian.org/debian bullseye/main amd64 libdrm-common all 2.4.104-1 [14.9 kB]
Get:3 http://ftp.debian.org/debian bullseye/main amd64 libdrm2 amd64 2.4.104-1 [41.5 kB]
...
Get:54 http://download.proxmox.com/debian/pve bullseye/pve-no-subscription amd64 zfs-initramfs all 2.1.4-pve1 [30.4 kB]                             
Get:55 http://download.proxmox.com/debian/pve bullseye/pve-no-subscription amd64 zfsutils-linux amd64 2.1.4-pve1 [492 kB]                           
Get:56 http://download.proxmox.com/debian/pve bullseye/pve-no-subscription amd64 zfs-zed amd64 2.1.4-pve1 [70.1 kB]                                 
Fetched 262 MB in 4min 17s (1,020 kB/s)                                                                                                             
Reading changelogs... Done
Extracting templates from packages: 100%
(Reading database ... 42896 files and directories currently installed.)
Preparing to unpack .../00-btrfs-progs_5.16.2-1~bpo11+1_amd64.deb ...
...
Preparing to unpack .../54-zfsutils-linux_2.1.4-pve1_amd64.deb ...
Unpacking zfsutils-linux (2.1.4-pve1) over (2.1.1-pve3) ...
Preparing to unpack .../55-zfs-zed_2.1.4-pve1_amd64.deb ...
Unpacking zfs-zed (2.1.4-pve1) over (2.1.1-pve3) ...
Setting up smartmontools (7.2-pve3) ...
Failed to reload daemon: Transport endpoint is not connected
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for smartmontools.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
smartmontools.service is a disabled or a static unit not running, not starting it.
Setting up proxmox-backup-file-restore (2.1.8-1) ...
Updating file-restore initramfs...
11316 blocks
Setting up libwayland-server0:amd64 (1.18.0-2~exp1.1) ...
Setting up libnvpair3linux (2.1.4-pve1) ...
Setting up proxmox-widget-toolkit (3.4-10) ...
Setting up pve-xtermjs (4.16.0-1) ...
Setting up pve-kernel-5.15.35-1-pve (5.15.35-2) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.15.35-1-pve /boot/vmlinuz-5.15.35-1-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.15.35-1-pve /boot/vmlinuz-5.15.35-1-pve
update-initramfs: Generating /boot/initrd.img-5.15.35-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 5.15.35-1-pve /boot/vmlinuz-5.15.35-1-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 5.15.35-1-pve /boot/vmlinuz-5.15.35-1-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 5.15.35-1-pve /boot/vmlinuz-5.15.35-1-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.35-1-pve
Found initrd image: /boot/initrd.img-5.15.35-1-pve
Found linux image: /boot/vmlinuz-5.13.19-6-pve
Found linux image: /boot/vmlinuz-5.13.19-2-pve
Found initrd image: /boot/initrd.img-5.13.19-2-pve
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Adding boot menu entry for EFI firmware configuration
done
Setting up proxmox-websocket-tunnel (0.1.0-1) ...
Setting up lxcfs (4.0.12-pve1) ...
Failed to get unit file state for lxcfs.service: Transport endpoint is not connected
Setting up btrfs-progs (5.16.2-1~bpo11+1) ...
Setting up pve-firmware (3.4-2) ...
Setting up libpve-u2f-server-perl (1.1-2) ...
Setting up pve-docs (7.2-2) ...
Setting up novnc-pve (1.3.0-3) ...
Setting up pve-kernel-5.13.19-6-pve (5.13.19-15) ...
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.13.19-6-pve /boot/vmlinuz-5.13.19-6-pve
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.13.19-6-pve /boot/vmlinuz-5.13.19-6-pve
update-initramfs: Generating /boot/initrd.img-5.13.19-6-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/proxmox-auto-removal 5.13.19-6-pve /boot/vmlinuz-5.13.19-6-pve
run-parts: executing /etc/kernel/postinst.d/zz-proxmox-boot 5.13.19-6-pve /boot/vmlinuz-5.13.19-6-pve
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postinst.d/zz-update-grub 5.13.19-6-pve /boot/vmlinuz-5.13.19-6-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.35-1-pve
Found initrd image: /boot/initrd.img-5.15.35-1-pve
Found linux image: /boot/vmlinuz-5.13.19-6-pve
Found initrd image: /boot/initrd.img-5.13.19-6-pve
Found linux image: /boot/vmlinuz-5.13.19-2-pve
Found initrd image: /boot/initrd.img-5.13.19-2-pve
Found memtest86+ image: /boot/memtest86+.bin
Found memtest86+ multiboot image: /boot/memtest86+_multiboot.bin
Adding boot menu entry for EFI firmware configuration
done
Setting up proxmox-backup-client (2.1.8-1) ...
Setting up libepoxy0:amd64 (1.5.5-1) ...
Setting up libpve-rs-perl (0.6.1) ...
Setting up libproxmox-acme-plugins (1.4.2) ...
Setting up pve-kernel-5.15 (7.2-3) ...
Setting up spl (2.1.4-pve1) ...
Setting up udev (247.3-7) ...
Failed to reload daemon: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
invoke-rc.d: could not determine current runlevel
Failed to restart udev.service: Transport endpoint is not connected
See system logs and 'systemctl status udev.service' for details.
invoke-rc.d: initscript udev, action "restart" failed.
Failed to get properties: Transport endpoint is not connected
dpkg: error processing package udev (--configure):
 installed udev package post-installation script subprocess returned error exit status 1
Setting up pve-i18n (2.7-1) ...
Setting up lxc-pve (4.0.12-1) ...
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for lxc-monitord.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
lxc-monitord.service is a disabled or a static unit, not starting it.
Failed to get unit file state for lxc-net.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
lxc-net.service is a disabled or a static unit, not starting it.
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for lxc.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
lxc.service is a disabled or a static unit, not starting it.
Failed to reload-or-try-restart lxc.service: Transport endpoint is not connected
See system logs and 'systemctl status lxc.service' for details.
dpkg: dependency problems prevent configuration of libpve-storage-perl:
 libpve-storage-perl depends on udev; however:
  Package udev is not configured yet.

dpkg: error processing package libpve-storage-perl (--configure):
 dependency problems - leaving unconfigured
Setting up libproxmox-rs-perl (0.1.1) ...
Setting up libtpms0:amd64 (0.9.2~bpo11+1) ...
Setting up libuutil3linux (2.1.4-pve1) ...
Setting up libdrm-common (2.4.104-1) ...
Setting up pve-lxc-syscalld (1.1.0-1) ...
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for pve-lxc-syscalld.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pve-lxc-syscalld.service is a disabled or a static unit not running, not starting it.
dpkg: dependency problems prevent configuration of pve-container:
 pve-container depends on libpve-storage-perl (>= 6.3-8); however:
  Package libpve-storage-perl is not configured yet.

dpkg: error processing package pve-container (--configure):
 dependency problems - leaving unconfigured
Setting up pve-kernel-5.13 (7.1-9) ...
Setting up swtpm-libs:amd64 (0.7.1~bpo11+1) ...
dpkg: dependency problems prevent configuration of pve-kernel-helper:
 pve-kernel-helper depends on udev; however:
  Package udev is not configured yet.

dpkg: error processing package pve-kernel-helper (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-manager:
 pve-manager depends on libpve-storage-perl (>= 7.1-2); however:
  Package libpve-storage-perl is not configured yet.
 pve-manager depends on pve-container (>= 4.0-9); however:
  Package pve-container is not configured yet.

dpkg: error processing package pve-manager (--configure):
 dependency problems - leaving unconfigured
Setting up libzfs4linux (2.1.4-pve1) ...
dpkg: dependency problems prevent configuration of qemu-server:
 qemu-server depends on libpve-storage-perl (>= 6.3-8); however:
  Package libpve-storage-perl is not configured yet.

dpkg: error processing package qemu-server (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-kernel-helper; however:
  Package pve-kernel-helper is not configured yet.
 proxmox-ve depends on pve-manager; however:
  Package pve-manager is not configured yet.
 proxmox-ve depends on qemu-server; however:
  Package qemu-server is not configured yet.

dpkg: error processing package proxmox-ve (--configure):
 dependency problems - leaving unconfigured
Setting up swtpm (0.7.1~bpo11+1) ...
dpkg: dependency problems prevent configuration of libpve-guest-common-perl:
 libpve-guest-common-perl depends on libpve-storage-perl (>= 7.0-14); however:
  Package libpve-storage-perl is not configured yet.

dpkg: error processing package libpve-guest-common-perl (--configure):
 dependency problems - leaving unconfigured
Setting up libdrm2:amd64 (2.4.104-1) ...
dpkg: dependency problems prevent configuration of pve-ha-manager:
 pve-ha-manager depends on pve-container; however:
  Package pve-container is not configured yet.
 pve-ha-manager depends on qemu-server (>= 6.0-15); however:
  Package qemu-server is not configured yet.

dpkg: error processing package pve-ha-manager (--configure):
 dependency problems - leaving unconfigured
Setting up libzpool5linux (2.1.4-pve1) ...
Setting up libgbm1:amd64 (20.3.5-1) ...
Setting up zfsutils-linux (2.1.4-pve1) ...
Installing new version of config file /etc/default/zfs ...
Installing new version of config file /etc/zfs/zfs-functions ...
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for zfs-import-cache.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-import-cache.service is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-import-scan.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-import-scan.service is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-import.target: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-import.target is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-mount.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-mount.service is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-share.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-share.service is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-volume-wait.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-volume-wait.service is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-volumes.target: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-volumes.target is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs.target: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs.target is a disabled or a static unit, not starting it.
Setting up swtpm-tools (0.7.1~bpo11+1) ...
Installing new version of config file /etc/swtpm_setup.conf ...
Setting up libvirglrenderer1:amd64 (0.8.2-5) ...
Setting up zfs-initramfs (2.1.4-pve1) ...
Setting up pve-qemu-kvm (6.2.0-5) ...
Setting up zfs-zed (2.1.4-pve1) ...
Installing new version of config file /etc/zfs/zed.d/zed-functions.sh ...
Installing new version of config file /etc/zfs/zed.d/zed.rc ...
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for zfs-zed.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-zed.service is a disabled or a static unit not running, not starting it.
Setting up libproxmox-acme-perl (1.4.2) ...
Setting up libpve-common-perl (7.1-6) ...
Setting up pve-cluster (7.2-1) ...
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for pve-cluster.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pve-cluster.service is a disabled or a static unit not running, not starting it.
Setting up libpve-cluster-perl (7.2-1) ...
Setting up libpve-http-server-perl (4.1-1) ...
Setting up libpve-access-control (7.1-8) ...
Setting up libpve-cluster-api-perl (7.2-1) ...
Processing triggers for initramfs-tools (0.140) ...
update-initramfs: Generating /boot/initrd.img-5.15.35-1-pve
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
Processing triggers for libc-bin (2.31-13+deb11u3) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for mailcap (3.69) ...
Errors were encountered while processing:
 udev
 libpve-storage-perl
 pve-container
 pve-kernel-helper
 pve-manager
 qemu-server
 proxmox-ve
 libpve-guest-common-perl
 pve-ha-manager
E: Sub-process /usr/bin/dpkg returned an error code (1)
 
Seems you missed posting some output. :)

Edit: I swear your last post was not visible at the time I wrote this post. o_O

Anyway: As I said above, I'm not sure if this will work/help in the current state and sadly it didn't.

Hopefully someone other has an idea.

But maybe it's easier and faster to reinstall PVE and restore from backups.
 
Last edited:
Seems you missed posting some output. :)

Edit: I swear your last post was not visible at the time I wrote this post. o_O

Anyway: As I said above, I'm not sure if this will work/help in the current state and sadly it didn't.

Hopefully someone other has an idea.

But maybe it's easier and faster to reinstall PVE and restore from backups.
Thank you for being so helpful anyway.

Yea, the post was waiting for mods' approval.

I wanted the possibility of reinstalling it as a last resort because this box is in a remote location. I'll wait to see if anyone can help out.

Best regards.
 
Hi all,

I've ended up here for same reason lol. I am having very similar problem with my Proxmox 7.3.3. I've noticed that one of my containers are down and can't be started. I had a snapshot of it so I restored it after which CT still would not start.
Then I just stopped all VMs and CTs and rebooted proxmox through GUI.

And... it never came back online after that. I have noticed that logging in through terminal takes so long, minutes. I did manage to get into faster through recovery mode and having shell `root@pve`

I was trying to figure out what is going on, then `cd ` into `/etc/pve/` and ls -l whick just shows empty folder. Ther's no configs, thre's nothing.

Then I took approach of updating and upgrading through non-enterprise repos. It's still upgrading and installig as I am typing this.
The log is quiet long and here is the latest snap of what I'm getting.



Bash:
Get:10 http://download.proFailed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for zfs-import-cache.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-import-cache.service is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-import-scan.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-import-scan.service is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-import.target: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-import.target is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-mount.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-mount.service is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-share.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-share.service is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-volume-wait.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-volume-wait.service is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs-volumes.target: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-volumes.target is a disabled or a static unit, not starting it.
Failed to get unit file state for zfs.target: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs.target is a disabled or a static unit, not starting it.
Setting up zfs-initramfs (2.1.9-pve1) ...
Setting up zfs-zed (2.1.9-pve1) ...
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for zfs-zed.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
zfs-zed.service is a disabled or a static unit not running, not starting it.
Setting up libproxmox-acme-perl (1.4.4) ...
Setting up libpve-common-perl (7.3-3) ...
Setting up pve-cluster (7.3-3) ...
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for pve-cluster.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pve-cluster.service is a disabled or a static unit not running, not starting it.
Setting up libpve-cluster-perl (7.3-3) ...
Setting up libpve-http-server-perl (4.2-1) ...
Setting up libpve-storage-perl (7.4-2) ...
Setting up libpve-access-control (7.4-2) ...
Setting up libpve-cluster-api-perl (7.3-3) ...
Setting up libpve-guest-common-perl (4.2-4) ...
Setting up pve-firewall (4.3-1) ...
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for pvefw-logger.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pvefw-logger.service is a disabled or a static unit not running, not starting it.
Failed to reload daemon: Transport endpoint is not connected
Failed to get unit file state for pve-firewall.service: Transport endpoint is not connected
Failed to retrieve unit state: Transport endpoint is not connected
pve-firewall.service is a disabled or a static unit, not starting it.
Failed to reload daemon: Transport endpoint is not connected


This is super slow, I am thinking could be NVMe that is failing (did pass a tests) or just something else on application level. Any suggestions?

Thanks!
 
I was trying to figure out what is going on, then `cd ` into `/etc/pve/` and ls -l whick just shows empty folder. Ther's no configs, thre's nothing.

Have a look here: [1] for the "why".

Then I took approach of updating and upgrading through non-enterprise repos.

I guess, updating in this state is/was not a good idea.

No clue, how to fix your current state; if it is possible at all, sorry.

[1] https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)
 
Hey there, this might be related to our issues in this thread. My /etc/pve is empty too but I think that's created on the services started.
 
Thanks for replies, @Neobin Last time I've logged in on web gui, I've noticed update attempt, not 100% sure but I think it was auto update. @chuffy yes that could be related, I am getting
Code:
Transport endpoint is not connected

What can be done in this situation? Maybe booting older IOS I've used for installation and "reinstalling" over if that is supported.
All my VMs and CTs are on same drive, I'll have to find a way to take them out.
 
Hi Markomo,

I not sure if my issue was also related to the timezone, but I can confirm that I made an 'apt upgrade' which made the distro go to an inconsistent state. I had to redo my setup, and until now it's still working great.
 
I have tried with updating and upgrading to latest version and kernel, it was super slow due CPU usage. Even the upgrade did not fix the issue. Then I applied timezone fix and problem went away.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!