[SOLVED] Problem: Upgrading CEPH Pacific to Quincy resulted in CEPH Storage pool to stop functioning.

S

ScoppyTheMehgnificent

New Member
Oct 9, 2023
4
1
1
I upgraded my CEPH cluster without properly following the mon upgrade so they were no longer on leveldb.

Proxmox and CEPH were updated to latest for current release.
https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy
  1. The upgrade to Quincy states a recommendation that Mons are using RocksDB.
  2. Leveldb support has been removed from quincy.

The monitors were still running as leveldb.
  1. Does this mean the mons cannot work at all since they are levelDB?

I upgraded all nodes to the quincy release 17.2.6 and restarted the mons.

At this point the cluster stopped responding.
`ceph` commands do not work since the service fails to start.

Are there steps for recovery?

1) Roll back to Pacific without being able to use CEPH commands (ceph orch upgrade start --ceph-version <version>).
2) Rebuild the monitors using data from the OSDs while maintaining Quincy release.
3) Is this actually the bug shared: https://tracker.ceph.com/issues/58156 about 17.2.6 (which is what Proxmox/CEPH upgrades to?


I ran the upgrade on another cluster prior to this without issue. The Mons were set with RocksDB and running on Quincy 17.2.6.


Code: 
 ceph-crash[53537]: WARNING:ceph-crash:post /var/lib/ceph/crash/2023-10-06T21:58:29.079369Z_b01e4225-9d3b-4fa4-9599-a4b6928f845c as client.crash.NODE failed: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
 
 
 -2> 2023-10-06T14:58:29.073-0700 7fa22e24ba00 -1 _open error initializing leveldb db back storage in /var/lib/ceph/mon/ceph-0/store.db
    -1> 2023-10-06T14:58:29.073-0700 7fa22e24ba00 -1 ./src/mon/MonitorDBStore.h: In function 'void MonitorDBStore::_open(const string&)' thread 7fa22e24ba00 time 2023-10-06T14:58:29.075051-0700
./src/mon/MonitorDBStore.h: 634: ceph_abort_msg("MonitorDBStore: error initializing keyvaluedb back storage")
 
Last edited:
journalctl -u ceph-mon@0.service

Code: 
systemd[1]: Started Ceph cluster monitor daemon.
ceph-mon[3181299]: 2023-10-06T14:5node.073-0700 7fa22e24ba00 -1 _open error initializing leveldb db back storage in /var/lib/ceph/mon/ceph-0/store.db
ceph-mon[3181299]: ./src/mon/MonitorDBStore.h: In function 'void MonitorDBStore::_open(const string&)' thread 7fa22e24ba00 time 2023-10-06T14:5node.075051-0700
ceph-mon[3181299]: ./src/mon/MonitorDBStore.h: 634: ceph_abort_msg("MonitorDBStore: error initializing keyvaluedb back storage")
ceph-mon[3181299]:  ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable)
ceph-mon[3181299]:  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd3) [0x7fa22f11a88f]
ceph-mon[3181299]:  2: (MonitorDBStore::_open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x74a) [0x56436b50100a]
ceph-mon[3181299]:  3: (MonitorDBStore::open(std::ostream&)+0x1a9) [0x56436b501589]
ceph-mon[3181299]:  4: main()
ceph-mon[3181299]:  5: __libc_start_main()
ceph-mon[3181299]:  6: _start()
ceph-mon[3181299]: 2023-10-06T14:5node.073-0700 7fa22e24ba00 -1 ./src/mon/MonitorDBStore.h: In function 'void MonitorDBStore::_open(const string&)' thread 7fa22e24ba00 time 2023-10-06T14:5node.075051-0700
ceph-mon[3181299]: ./src/mon/MonitorDBStore.h: 634: ceph_abort_msg("MonitorDBStore: error initializing keyvaluedb back storage")
ceph-mon[3181299]:  ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable)
ceph-mon[3181299]:  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd3) [0x7fa22f11a88f]
ceph-mon[3181299]:  2: (MonitorDBStore::_open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x74a) [0x56436b50100a]
ceph-mon[3181299]:  3: (MonitorDBStore::open(std::ostream&)+0x1a9) [0x56436b501589]
ceph-mon[3181299]:  4: main()
ceph-mon[3181299]:  5: __libc_start_main()
ceph-mon[3181299]:  6: _start()
ceph-mon[3181299]: *** Caught signal (Aborted) **
ceph-mon[3181299]:  in thread 7fa22e24ba00 thread_name:ceph-mon
ceph-mon[3181299]:  ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable)
ceph-mon[3181299]:  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7fa22ec16140]
ceph-mon[3181299]:  2: gsignal()
ceph-mon[3181299]:  3: abort()
ceph-mon[3181299]:  4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x18c) [0x7fa22f11a948]
ceph-mon[3181299]:  5: (MonitorDBStore::_open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x74a) [0x56436b50100a]
ceph-mon[3181299]:  6: (MonitorDBStore::open(std::ostream&)+0x1a9) [0x56436b501589]
ceph-mon[3181299]:  7: main()
ceph-mon[3181299]:  8: __libc_start_main()
ceph-mon[3181299]:  9: _start()
ceph-mon[3181299]: 2023-10-06T14:5node.077-0700 7fa22e24ba00 -1 *** Caught signal (Aborted) **
ceph-mon[3181299]:  in thread 7fa22e24ba00 thread_name:ceph-mon
ceph-mon[3181299]:  ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable)
ceph-mon[3181299]:  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7fa22ec16140]
ceph-mon[3181299]:  2: gsignal()
ceph-mon[3181299]:  3: abort()
ceph-mon[3181299]:  4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x18c) [0x7fa22f11a948]
ceph-mon[3181299]:  5: (MonitorDBStore::_open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x74a) [0x56436b50100a]
ceph-mon[3181299]:  6: (MonitorDBStore::open(std::ostream&)+0x1a9) [0x56436b501589]
ceph-mon[3181299]:  7: main()
ceph-mon[3181299]:  8: __libc_start_main()
ceph-mon[3181299]:  9: _start()
ceph-mon[3181299]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ceph-mon[3181299]:     -2> 2023-10-06T14:5node.073-0700 7fa22e24ba00 -1 _open error initializing leveldb db back storage in /var/lib/ceph/mon/ceph-0/store.db
ceph-mon[3181299]:     -1> 2023-10-06T14:5node.073-0700 7fa22e24ba00 -1 ./src/mon/MonitorDBStore.h: In function 'void MonitorDBStore::_open(const string&)' thread 7fa22e24ba00 time 2023-10-06T14:5node.075051-0700
ceph-mon[3181299]: ./src/mon/MonitorDBStore.h: 634: ceph_abort_msg("MonitorDBStore: error initializing keyvaluedb back storage")
ceph-mon[3181299]:  ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable)
ceph-mon[3181299]:  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd3) [0x7fa22f11a88f]
ceph-mon[3181299]:  2: (MonitorDBStore::_open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x74a) [0x56436b50100a]
ceph-mon[3181299]:  3: (MonitorDBStore::open(std::ostream&)+0x1a9) [0x56436b501589]
ceph-mon[3181299]:  4: main()
ceph-mon[3181299]:  5: __libc_start_main()
ceph-mon[3181299]:  6: _start()
ceph-mon[3181299]:      0> 2023-10-06T14:5node.077-0700 7fa22e24ba00 -1 *** Caught signal (Aborted) **
ceph-mon[3181299]:  in thread 7fa22e24ba00 thread_name:ceph-mon
ceph-mon[3181299]:  ceph version 17.2.6 (995dec2cdae920da21db2d455e55efbc339bde24) quincy (stable)
ceph-mon[3181299]:  1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140) [0x7fa22ec16140]
ceph-mon[3181299]:  2: gsignal()
ceph-mon[3181299]:  3: abort()
ceph-mon[3181299]:  4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x18c) [0x7fa22f11a948]
ceph-mon[3181299]:  5: (MonitorDBStore::_open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x74a) [0x56436b50100a]
ceph-mon[3181299]:  6: (MonitorDBStore::open(std::ostream&)+0x1a9) [0x56436b501589]
ceph-mon[3181299]:  7: main()
ceph-mon[3181299]:  8: __libc_start_main()
 
I fixed it by rolling back.

Code: 
# The following only need to be ran on the monitor nodes.
# A minimum of two nodes need to be running.

# Stop running CEPH services
systemctl stop ceph-mon.target
systemctl stop ceph-mgr.target
systemctl stop ceph.target

# Backup ceph configs.
rsync -uav /etc/ceph/ /etc/ceph-bak/
rsync -uav /var/lib/ceph/ /var/lib/ceph-bak/

# Based on: debian - Can I rollback an apt-get upgrade if something goes wrong? - Unix & Linux Stack Exchange
# Make a copy of the applicable log:
cp /var/log/dpkg.log ~
# Strip out the packages
awk '$3 ~ /upgrade/ {print $4"="$5}' ~/dpkg.log > ~/dpkg.log.clean
# Copy the log into a txt file (not necessary but I did it anyways)
cp dpkg.log.clean revert.txt

# Switch the repo back to pacific
sed -i 's/quincy/pacific/' /etc/apt/sources.list.d/ceph.list
apt update
# Run a test to see if the revert will hang up on anything.
apt-get -s install $(cat revert.txt | tr '\n' ' ')
# If everything goes as planned remove the -s and run.
apt-get install $(cat revert.txt | tr '\n' ' ')
# Pinning the apps might not be necessary since we will be rebuilding the mons to Rocksdb.  Then running the upgrade to Quincy again

# With the monitors running:
# 1) Delete one monitor
# 2) Add new monitor
# 3) Confirm new monitor is reflecting rocksdb
# 4) Repeat 1-3 on remaining mon.nodes
# 5) When all monitors are updated rerun the quincy upgrade portion of pve7to8 guide.
 
  • Like
Reactions: florian-n
Hi did this rollback worked for you ?
When i follow your list i get
root@compute01:~# apt-get -s install $(cat revert.txt | tr '\n' ' ') |grep ceph
E: Version '11.1+deb11u3' for 'base-files' was not found
E: Version '5.1-2+b3' for 'bash' was not found
E: Version '1.34+dfsg-1' for 'tar' was not found
E: Version '1.20.9' for 'dpkg' was not found
E: Version '3.6-1' for 'grep' was not found
E: Version '6.2+20201114-2' for 'ncurses-bin' was not found
E: Version '5.32.1-4+deb11u2' for 'libperl5.32' was not found
E: Version '5.32.1-4+deb11u2' for 'perl' was not found
E: Version '5.32.1-4+deb11u2' for 'perl-base' was not found
E: Version '5.32.1-4+deb11u2' for 'perl-modules-5.32' was not found
E: Version '2.31-13+deb11u3' for 'libc-devtools' was not found
E: Version '2.31-13+deb11u3' for 'libc6-dev' was not found
E: Version '2.31-13+deb11u3' for 'libc-dev-bin' was not found
E: Version '5.10.113-1' for 'linux-libc-dev' was not found
E: Version '2.31-13+deb11u3' for 'libc6' was not found
E: Version '1:1.2.11.dfsg-2+deb11u1' for 'zlib1g' was not found
E: Version '2.31-13+deb11u3' for 'libc-bin' was not found
E: Version '6.2+20201114-2' for 'ncurses-base' was not found
E: Version '3.118' for 'adduser' was not found
E: Version '247.3-7' for 'systemd-sysv' was not found
E: Version '247.3-7' for 'libsystemd0' was not found
E: Version '247.3-7' for 'systemd-timesyncd' was not found
E: Version '247.3-7' for 'libnss-systemd' was not found
E: Version '247.3-7' for 'libpam-systemd' was not found
E: Version '247.3-7' for 'systemd' was not found
E: Version '247.3-7' for 'udev' was not found
E: Version '247.3-7' for 'libudev1' was not found
E: Version '1.12.20-2' for 'dbus' was not found
E: Version '1.12.20-2' for 'libdbus-1-3' was not found
E: Version '2.2.10-2+deb11u3' for 'libexpat1' was not found
E: Version '3.7.1-5' for 'libgnutlsxx28' was not found
E: Version '3.7.1-5' for 'libgnutls-openssl27' was not found
E: Version '1.13.1-1' for 'libunbound8' was not found
E: Version '3.7.1-5' for 'libgnutls-dane0' was not found
E: Version '4.16.0-2' for 'libtasn1-6' was not found
E: Version '3.7.1-5' for 'libgnutls30' was not found
E: Version '8.2102.0-2' for 'rsyslog' was not found
E: Version '1.18.3-6+deb11u1' for 'libk5crypto3' was not found
E: Version '1.18.3-6+deb11u1' for 'libkrb5support0' was not found
E: Version '1.18.3-6+deb11u1' for 'libkrb5-3' was not found
E: Version '1.18.3-6+deb11u1' for 'libgssapi-krb5-2' was not found
E: Version '1.1.1n-0+deb11u1' for 'libssl1.1' was not found
E: Version '2:4.13.13+dfsg-1~deb11u3' for 'smbclient' was not found
E: Version '2:4.13.13+dfsg-1~deb11u3' for 'samba-common' was not found
E: Version '0.11.3-1' for 'libbsd0' was not found
E: Version '2:4.13.13+dfsg-1~deb11u3' for 'libsmbclient' was not found
E: Version '2:2.2.3-2~deb11u1' for 'python3-ldb' was not found
E: Version '2:2.2.3-2~deb11u1' for 'libldb2' was not found
E: Version '2:4.13.13+dfsg-1~deb11u3' for 'samba-libs' was not found
E: Version '2:4.13.13+dfsg-1~deb11u3' for 'libwbclient0' was not found
E: Version '2.4.57+dfsg-3' for 'libldap-2.4-2' was not found
E: Version '0.8-5' for 'libavahi-client3' was not found
E: Version '0.8-5' for 'libavahi-common3' was not found
E: Version '0.8-5' for 'libavahi-common-data' was not found
E: Version '2.3.3op2-3+deb11u1' for 'libcups2' was not found
E: Version '2.7.18-8' for 'libpython2.7' was not found
E: Version '2.7.18-8' for 'python2.7' was not found
E: Version '2.7.18-8' for 'libpython2.7-stdlib' was not found
E: Version '2.7.18-8' for 'python2.7-minimal' was not found
E: Version '2.7.18-8' for 'libpython2.7-minimal' was not found
E: Version '6.2+20201114-2' for 'libncurses6' was not found
E: Version '6.2+20201114-2' for 'libncursesw6' was not found
E: Version '6.2+20201114-2' for 'libtinfo6' was not found
E: Version '1.3.1-1' for 'libtirpc-common' was not found
E: Version '1.3.1-1' for 'libtirpc-dev' was not found
E: Version '1.3.1-1' for 'libtirpc3' was not found
E: Version '1:8.4p1-5' for 'openssh-sftp-server' was not found
E: Version '1:8.4p1-5' for 'openssh-server' was not found
E: Version '1:8.4p1-5' for 'openssh-client' was not found
E: Version '1:8.4p1-5' for 'ssh' was not found
E: Version '2021.1.1' for 'debian-archive-keyring' was not found
E: Version '1.5.0-3' for 'libksba8' was not found
E: Version '2.2.27-2+deb11u1' for 'gpg-wks-client' was not found
E: Version '2.2.27-2+deb11u1' for 'dirmngr' was not found
E: Version '2.2.27-2+deb11u1' for 'gnupg-utils' was not found
E: Version '2.2.27-2+deb11u1' for 'gpg-wks-server' was not found
E: Version '2.2.27-2+deb11u1' for 'gpg-agent' was not found
E: Version '2.2.27-2+deb11u1' for 'gpg' was not found
E: Version '2.2.27-2+deb11u1' for 'gpgconf' was not found
E: Version '2.2.27-2+deb11u1' for 'gnupg-l10n' was not found
E: Version '2.2.27-2+deb11u1' for 'gnupg' was not found
E: Version '2.2.27-2+deb11u1' for 'gpgsm' was not found
E: Version '2.2.27-2+deb11u1' for 'gpgv' was not found
E: Version '10.36-2' for 'libpcre2-8-0' was not found
E: Version '2021a-1+deb11u3' for 'tzdata' was not found
E: Version '2.13+dfsg-4' for 'cpio' was not found
E: Version '4.4.1-2.3' for 'isc-dhcp-client' was not found
E: Version '4.4.1-2.3' for 'isc-dhcp-common' was not found
E: Version '3.18.0-2' for 'logrotate' was not found
E: Version '5.4-2' for 'nano' was not found
E: Version '0.15-2' for 'libjson-c5' was not found
E: Version '2.9.10+dfsg-6.7+deb11u1' for 'libxml2' was not found
E: Version '1:9.16.27-1~deb11u1' for 'bind9-dnsutils' was not found
E: Version '1:9.16.27-1~deb11u1' for 'bind9-libs' was not found
E: Version '1:9.16.27-1~deb11u1' for 'bind9-host' was not found
E: Version '1:5.39-3' for 'file' was not found
E: Version '1:5.39-3' for 'libmagic1' was not found
E: Version '1:5.39-3' for 'libmagic-mgc' was not found
E: Version '1.18.3-6+deb11u1' for 'krb5-locales' was not found
E: Version '2.31-13+deb11u3' for 'libc-l10n' was not found
E: Version '2.31-13+deb11u3' for 'locales' was not found
E: Version '6.2+20201114-2' for 'ncurses-term' was not found
E: Version '1:2.1.0-2+b1' for 'traceroute' was not found
E: Version '7.74.0-1.3+deb11u1' for 'curl' was not found
E: Version '1.43.0-1' for 'libnghttp2-14' was not found
E: Version '7.74.0-1.3+deb11u1' for 'libcurl4' was not found
E: Version '5.3.3-1.1+b1' for 'liblua5.3-0' was not found
E: Version '16.2.7' for 'librgw2' was not found
E: Version '16.2.7' for 'libradosstriper1' was not found
E: Version '16.2.7' for 'ceph-mon' was not found
E: Version '16.2.7' for 'ceph-mgr-modules-core' was not found
E: Version '16.2.7' for 'ceph-mgr' was not found
E: Version '16.2.7' for 'ceph-mds' was not found
E: Version '16.2.7' for 'ceph' was not found
E: Version '16.2.7' for 'ceph-osd' was not found
E: Version '16.2.7' for 'ceph-base' was not found
E: Version '16.2.7' for 'ceph-common' was not found
E: Version '16.2.7' for 'librbd1' was not found
E: Version '16.2.7' for 'librados2' was not found
E: Version '16.2.7' for 'python3-cephfs' was not found
E: Version '16.2.7' for 'libcephfs2' was not found
E: Version '16.2.7' for 'python3-rgw' was not found
E: Version '16.2.7' for 'python3-rados' was not found
E: Version '16.2.7' for 'python3-ceph-argparse' was not found
E: Version '16.2.7' for 'python3-ceph-common' was not found
E: Version '16.2.7' for 'python3-rbd' was not found
E: Version '16.2.7' for 'libjaeger' was not found
E: Version '1.0.1+dfsg1-2' for 'python3-werkzeug' was not found
E: Version '1.9.5p2-3' for 'sudo' was not found
E: Version '16.2.7' for 'ceph-fuse' was not found
E: Version '2:6.11-3.1' for 'cifs-utils' was not found
E: Version '2:3.61-1+deb11u2' for 'libnss3' was not found
E: Version '1.1.34-4' for 'libxslt1.1' was not found
E: Version '1.1.34-4' for 'xsltproc' was not found
E: Version '1.0.123+deb11u1' for 'debootstrap' was not found
E: Version '0.51+deb11u1' for 'distro-info-data' was not found
E: Version '1:9.16.27-1~deb11u1' for 'dnsutils' was not found
E: Version '3.7.1-5' for 'gnutls-bin' was not found
E: Version '2.04-20' for 'grub2-common' was not found
E: Version '2.04-20' for 'grub-pc' was not found
E: Version '2.04-20' for 'grub-pc-bin' was not found
E: Version '2.04-20' for 'grub-efi-ia32-bin' was not found
E: Version '2.04-20' for 'grub-efi-amd64-bin' was not found
E: Version '2.10.4+dfsg-1' for 'libfreetype6' was not found
E: Version '2.04-20' for 'grub-common' was not found
E: Version '1.1.1n-0+deb11u1' for 'openssl' was not found
E: Version '2.66.8-1' for 'libglib2.0-0' was not found
E: Version '7.74.0-1.3+deb11u1' for 'libcurl3-gnutls' was not found
E: Version '0.40.0-1' for 'libpixman-1-0' was not found
E: Version '2:1.7.2-1' for 'libx11-data' was not found
E: Version '2:1.7.2-1' for 'libx11-6' was not found
E: Version '2:1.7.2-1' for 'libx11-xcb1' was not found
E: Version '0.8.2-5' for 'libvirglrenderer1' was not found
E: Version '3.5.6-1+b1' for 'postfix' was not found
E: Version '6.12-1' for 'libhttp-daemon-perl' was not found
E: Version '3.9.0-3' for 'libblas3' was not found
E: Version '1.3.3-2+deb11u1' for 'libflac8' was not found
E: Version '1.0.8-2' for 'libfribidi0' was not found
E: Version '1.18.4-2' for 'libgstreamer-plugins-base1.0-0' was not found
E: Version '2.4.57+dfsg-3' for 'libldap-common' was not found
E: Version '6.2+20201114-2+deb11u1' for 'libncurses5' was not found
E: Version '6.2+20201114-2+deb11u1' for 'libtinfo5' was not found
E: Version '3.12.4-1' for 'libprotobuf23' was not found
E: Version '0.6.1-2.1' for 'libwebp6' was not found
E: Version '4.2.0-1+deb11u1' for 'libtiff5' was not found
E: Version '1:3.5.12-1' for 'libxpm4' was not found
E: Version '3.22-26' for 'procmail' was not found
E: Version '3.12.4-1' for 'python3-protobuf' was not found
E: Version '1.4.4-1.1' for 'python3-waitress' was not found
E: Version '4.99.0-2' for 'tcpdump' was not found
E: Version '6.0-26' for 'unzip' was not found
E: Version '2022.02.15-0+deb11u1' for 'usb.ids' was not found
E: Version '16.2.14-pve1' for 'ceph-osd' was not found
E: Version '16.2.14-pve1' for 'ceph-mon' was not found
E: Version '16.2.14-pve1' for 'ceph-mds' was not found
E: Version '16.2.14-pve1' for 'ceph-base' was not found
E: Version '16.2.14-pve1' for 'ceph-common' was not found
E: Version '16.2.14-pve1' for 'ceph' was not found
E: Version '16.2.14-pve1' for 'ceph-mgr' was not found
E: Version '16.2.14-pve1' for 'ceph-mgr-modules-core' was not found
E: Version '16.2.14-pve1' for 'libsqlite3-mod-ceph' was not found
E: Version '16.2.14-pve1' for 'librgw2' was not found
E: Version '16.2.14-pve1' for 'python3-cephfs' was not found
E: Version '16.2.14-pve1' for 'libcephfs2' was not found
E: Version '16.2.14-pve1' for 'python3-rgw' was not found
E: Version '16.2.14-pve1' for 'python3-rados' was not found
E: Version '16.2.14-pve1' for 'python3-ceph-argparse' was not found
E: Version '16.2.14-pve1' for 'python3-ceph-common' was not found
E: Version '16.2.14-pve1' for 'python3-rbd' was not found
E: Version '16.2.14-pve1' for 'libradosstriper1' was not found
E: Version '16.2.14-pve1' for 'librbd1' was not found
E: Version '16.2.14-pve1' for 'librados2' was not found
E: Version '16.2.14-pve1' for 'ceph-fuse' was not found
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back