[SOLVED] ZFS not enabled or running after update

[Curt Hall]

If try to create a zfs pool in the cli, i get this:

root@proxmox5:~# zpool create -f -o ashift=12 RAID10 /dev/sdb /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk
zpool: symbol lookup error: /lib/libzfs.so.2: undefined symbol: efi_rescan


If i click on the webGUI ZFS module, I get this:

It would seem the zfs service is offline or corrupt, I have tried a restart of the host and it makes no difference.

 

[LnxBil]

Looks like you have made a software update that was interrupted. The missing symbol indicates that the library or the binary has changed without its counterpart.

Please run

apt-get update
apt-get dist-upgrade

and report back

 

[Curt Hall]

When I run the update portion, I get this error:

Reading package lists... Error!
W: The repository 'https://enterprise.proxmox.com/debian/pve stretch Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://enterprise.proxmox.com/debian/pve/dists/stretch/pve-enterprise/binary-amd64/Packages 401 Unauthorized
E: Failed to fetch store:/var/lib/apt/lists/partial/ftp.debian.org_debian_dists_buster_contrib_binary-amd64_Packages.gz Hash Sum mismatch
Hashes of expected file:
- Filesize:230404 [weak]
- SHA256:4eab592c257683c0e7ceba69dcb497f218b942fa4f29eda108a5249b1b2ff531
- MD5Sum:27b2a3d7173452e43315ce76da6026c1 [weak]
Hashes of received file:
- SHA256:010ce393ee53e512c1320ae1e8842e2b0bf6d4a2400638114f5bc3525eae6d47
- MD5Sum:0faa2a70a58216c1ec7e2eae93242e57 [weak]
- Filesize:2789 [weak]
Release file created at: Sat, 09 May 2020 09:51:02 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.
E: Unable to parse package file /var/lib/apt/lists/ftp.debian.org_debian_dists_buster_main_binary-amd64_Packages (1)
E: Unable to parse package file /var/lib/apt/lists/ftp.debian.org_debian_dists_buster_contrib_binary-amd64_Packages (1)
W: You may want to run apt-get update to correct these problems
E: The package cache file is corrupted

 

[LnxBil]

Hmm... did you try to update your server manually? The Enterprise repository is stretch, but your other lists are buster. Please post the output of pveversion -v

 

[Curt Hall]

root@proxmox5:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libpve-access-control: 5.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.1.0-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 3.0-16
pve-firmware: 3.1-1
pve-ha-manager: 2.0-5
pve-i18n: 2.1-2
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 4.3.0-1
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

 

[LnxBil]

Yes, this looks like a uncomplete upgrade. There are version from 5.3 and 6.2 installed. What have you done to get this setup?

 

[Curt Hall]

That's the $64 question. I have 5 Proxmox boxes, all are or at least were 5.3
I have no clue how this one got half in and half out of an upgrade to 6. The good news is its the last one in the cluster and there are no VM's on it. So it can be wiped/changed/updated or whatever without production ramifications. I'm just not sure what best route would be to fix.

 

[Curt Hall]

Yes, this looks like a uncomplete upgrade. There are version from 5.3 and 6.2 installed. What have you done to get this setup?

Is there a way to piecemeal upgrade/update each service individually? Clearly doing it all at once with apt-get upgrade command is not going to work.

 

[H4R0]

You dont have a proxmox subscription do you ?

You have to remove the enterprise respository in that case and use no-subscription.

Only run the commands if you have backups, your system is already broken.
rm /etc/apt/sources.list.d/pve-enterprise.list
echo "deb http://download.proxmox.com/debian/pve buster pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list
mv /var/lib/apt/lists/ftp.debian.org_debian_dists_buster_main_binary-amd64_Packages /tmp
mv /var/lib/apt/lists/ftp.debian.org_debian_dists_buster_contrib_binary-amd64_Packages /tmp
apt update
apt full-upgrade

 

[Curt Hall]

You dont have a proxmox subscription do you ?

You have to remove the enterprise respository in that case and use no-subscription.

Only run the commands if you have backups, your system is already broken.
rm /etc/apt/sources.list.d/pve-enterprise.list
echo "deb http://download.proxmox.com/debian/pve buster pve-no-subscription" > /etc/apt/sources.list.d/pve-no-subscription.list
mv /var/lib/apt/lists/ftp.debian.org_debian_dists_buster_main_binary-amd64_Packages /tmp
mv /var/lib/apt/lists/ftp.debian.org_debian_dists_buster_contrib_binary-amd64_Packages /tmp
apt update
apt full-upgrade

Ran those and did a reboot, and this is what i get on the console screen

 

[H4R0]

Ran those and did a reboot, and this is what i get on the console screen

View attachment 17485

That usually only happens on hard shut down (power loss)

Or a damaged/broken hard disk... which would also make sense for the other problems.

Is your pve install on a raid ?


Run "fsck /dev/mapper/pve-root"

After that press ctrl+d and the machine should continue to boot as long as there are no other problems.


I kinda have the feeling a lot more is wrong with that machine, i would recommend to do a clean install and restore from backup.

Should be faster then to troubleshoot. Make sure your disks are good before tho.

 

[Curt Hall]

That usually only happens on hard shut down (power loss)

Or a damaged/broken hard disk... which would also make sense for the other problems.

Is your pve install on a raid ?


Run "fsck /dev/mapper/pve-root"

After that press ctrl+d and the machine should continue to boot as long as their are no other problems.


I kinda have the feeling a lot more is wrong with that machine, i would recommend to do a clean install and restore from backup.

Should be faster then to troubleshoot. Make sure your disks are good before tho.

The above steps does not work. I will have to do a clean re-install, sucks rejoining to cluster, and connecting to ISCSI targets, etc. but oh well, its junk.

 

[jim.bond.9862]

This sucks. I was kind of hoping you find a solution other than full reinstall.
I have a home server running 5.3

Would love to upgrade to 6. But it's the only machine I have. And I am still not sure how to system backup properly.

 

[LnxBil]

Would love to upgrade to 6. But it's the only machine I have. And I am still not sure how to system backup properly.

Unfortunately not ZFS-based? If so, just create a snapshot and use the rescue mode in new PVE install isos to get your system back up running.
I've updated quite a few PVE5 -> PVE6 and there was never a problem if you follow the upgrade guide.

 

[jim.bond.9862]

I am ,in fact, zfs based.
Just not sure how to do it safely.
Last update I tried, from 5.1 to 5.3 just hosed the system and I had to re do everything from scratch. At this moment I have only 3 VMs containers but doing the config again bites.

 

[H4R0]

I am ,in fact, zfs based.
Just not sure how to do it safely.
Last update I tried, from 5.1 to 5.3 just hosed the system and I had to re do everything from scratch. At this moment I have only 3 VMs containers but doing the config again bites.

Like @LnxBil said just create a snapshot before the upgrade, if something goes wrong you can simply boot recovery or live usb and rollback.

Also take a look at zfs-autosnapshot, i can rollback in 5 minute intervals with it. Doesnt add overhead.

zfs snapshot rpool@before_update

# from live cd:
zfs rollback rpool@before_update

 

[jim.bond.9862]

Thanks will try this.
Have to prepare livecd or USB
My server has no cd drive.

 

[LnxBil]

Last update I tried, from 5.1 to 5.3 just hosed the system and I had to re do everything from scratch. At this moment I have only 3 VMs containers but doing the config again bites.

Please try again and document every step. We can only help if we know what went wrong. Most common mistake is to use apt-get upgrade instead of apt-get dist-upgrade.

 

[Curt Hall]

Please try again and document every step. We can only help if we know what went wrong. Most common mistake is to use apt-get upgrade instead of apt-get dist-upgrade.

I tried that and it just stops at this point and seems to do nothing:

apt-listchanges: News
---------------------

ebtables (2.0.10.4+snapshot20181205-1) unstable; urgency=medium

All the ebtables binaries have been moved away from /sbin to
/usr/sbin. Some compatibility symlinks have been added for the
Buster release cycle, but please make sure your scripts aren't using
hardcoded binary paths.

The plan is to drop the symlinks in Bullseye, the release after
Buster.

-- Alberto Molina Coballes <alb.molina@gmail.com> Sun, 27 Dec 2018 21:29:00 +0100

fontconfig (2.12.3-0.2) unstable; urgency=medium

Starting with version 2.12, fontconfig is using "Slight" (hintslight) as
automatic hinting style. This might change the rendering of the fonts.

If you want the to restore the old hinting, run "dpkg-reconfigure
fontconfig-config" and select "Full" as hinting style.

-- Laurent Bigonville <bigon@debian.org> Tue, 04 Jul 2017 21:10:57 +0200

glibc (2.26-5) unstable; urgency=medium

Starting with version 2.26-1, the glibc requires a 3.2 or later Linux
kernel. If you use an older kernel, please upgrade it *before*
installing this glibc version. Failing to do so will end-up with the
following failure:

Preparing to unpack .../libc6_2.26-5_amd64.deb ...
ERROR: This version of the GNU libc requires kernel version
3.2 or later. Please upgrade your kernel before installing
glibc.

The decision to not support older kernels is a GNU libc upstream
decision.

Note: This obviously does not apply to non-Linux kernels.

-- Aurelien Jarno <aurel32@debian.org> Tue, 23 Jan 2018 22:03:12 +0100

gnupg2 (2.2.12-1+deb10u1) buster; urgency=medium

In this version we adopt GnuPG's upstream approach of making keyserver
access default to self-sigs-only. This defends against receiving

 

[Curt Hall]

I tried that and it just stops at this point and seems to do nothing:

apt-listchanges: News
---------------------

ebtables (2.0.10.4+snapshot20181205-1) unstable; urgency=medium

All the ebtables binaries have been moved away from /sbin to
/usr/sbin. Some compatibility symlinks have been added for the
Buster release cycle, but please make sure your scripts aren't using
hardcoded binary paths.

The plan is to drop the symlinks in Bullseye, the release after
Buster.

-- Alberto Molina Coballes <alb.molina@gmail.com> Sun, 27 Dec 2018 21:29:00 +0100

fontconfig (2.12.3-0.2) unstable; urgency=medium

Starting with version 2.12, fontconfig is using "Slight" (hintslight) as
automatic hinting style. This might change the rendering of the fonts.

If you want the to restore the old hinting, run "dpkg-reconfigure
fontconfig-config" and select "Full" as hinting style.

-- Laurent Bigonville <bigon@debian.org> Tue, 04 Jul 2017 21:10:57 +0200

glibc (2.26-5) unstable; urgency=medium

Starting with version 2.26-1, the glibc requires a 3.2 or later Linux
kernel. If you use an older kernel, please upgrade it *before*
installing this glibc version. Failing to do so will end-up with the
following failure:

Preparing to unpack .../libc6_2.26-5_amd64.deb ...
ERROR: This version of the GNU libc requires kernel version
3.2 or later. Please upgrade your kernel before installing
glibc.

The decision to not support older kernels is a GNU libc upstream
decision.

Note: This obviously does not apply to non-Linux kernels.

-- Aurelien Jarno <aurel32@debian.org> Tue, 23 Jan 2018 22:03:12 +0100

gnupg2 (2.2.12-1+deb10u1) buster; urgency=medium

In this version we adopt GnuPG's upstream approach of making keyserver
access default to self-sigs-only. This defends against receiving

oh, I see that was just the upgrade "news", I had to page down many times (not very intuitive) to get to the line that says "Yes" to continue.
Good grief Charlie Brown, can it be any more complicating?