Hi Everyone
After upgrading ceph from 20.2.1 to 20.2.2 I'm getting the following error. Ceph 20.2.1 was working fine.
The error is preventing new drives from being recognised as available. Other than that, ceph appears to be working correctly.
Proxmox 9.2.4 with all updates.
Ceph was installed separate to proxmox. With cephadm and ceph dashboard
I've tried adding a lvm filter. global_filter = [ "r|/dev/zd.*|", "r|/dev/rbd.*|", "r|/dev/pve/.*|", "r|/dev/mapper/pve-.*|", "a|.*|" ]
Also tried creating a symbolic link. sudo ln -sf /dev/null /dev/pve/data
HEALTH_WARN: failed to probe daemons or devices
host host2 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/mon.host2/config
Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:55a5c2014b4db34589ad8886606409727f633319131bb663d37e0d489e350703 -e NODE_NAME=host2 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51:/var/run/ceph:z -v /var/log/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51:/var/log/ceph:z -v /var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9usyv3et:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:55a5c2014b4db34589ad8886606409727f633319131bb663d37e0d489e350703 inventory --format=json-pretty --with-lsm --list-all --filter-for-batch
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 33, in <module>
/usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')())
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 62, in __init__
/usr/bin/podman: stderr self.main(self.argv)
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in newfunc
/usr/bin/podman: stderr return f(*a, **kw)
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 174, in main
/usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args)
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch
/usr/bin/podman: stderr instance.main()
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/inventory/main.py", line 58, in main
/usr/bin/podman: stderr self.format_report(Devices(filter_for_batch=self.args.filter_for_batch,
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in __init__
/usr/bin/podman: stderr self._devices = [Device(k,
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in <listcomp>
/usr/bin/podman: stderr self._devices = [Device(k,
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 156, in __init__
/usr/bin/podman: stderr self.device_id = self._get_device_id()
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 322, in _get_device_id
/usr/bin/podman: stderr udev_data = disk.UdevData(self.path)
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/disk.py", line 1355, in __init__
/usr/bin/podman: stderr raise RuntimeError(f'{path} not found.')
/usr/bin/podman: stderr RuntimeError: /dev/pve/data_tmeta not found.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 5296, in <module>
main()
~~~~^^
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 5284, in main
r = ctx.func(ctx)
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 397, in _infer_config
return func(ctx)
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 312, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 425, in _infer_image
return func(ctx)
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 299, in _validate_fsid
return func(ctx)
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 3231, in command_ceph_volume
out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/cephadmlib/call_wrappers.py", line 310, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:55a5c2014b4db34589ad8886606409727f633319131bb663d37e0d489e350703 -e NODE_NAME=host2 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51:/var/run/ceph:z -v /var/log/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51:/var/log/ceph:z -v /var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9usyv3et:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:55a5c2014b4db34589ad8886606409727f633319131bb663d37e0d489e350703 inventory --format=json-pretty --with-lsm --list-all --filter-for-batch
After upgrading ceph from 20.2.1 to 20.2.2 I'm getting the following error. Ceph 20.2.1 was working fine.
The error is preventing new drives from being recognised as available. Other than that, ceph appears to be working correctly.
Proxmox 9.2.4 with all updates.
Ceph was installed separate to proxmox. With cephadm and ceph dashboard
I've tried adding a lvm filter. global_filter = [ "r|/dev/zd.*|", "r|/dev/rbd.*|", "r|/dev/pve/.*|", "r|/dev/mapper/pve-.*|", "a|.*|" ]
Also tried creating a symbolic link. sudo ln -sf /dev/null /dev/pve/data
HEALTH_WARN: failed to probe daemons or devices
host host2 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr: Inferring config /var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/mon.host2/config
Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:55a5c2014b4db34589ad8886606409727f633319131bb663d37e0d489e350703 -e NODE_NAME=host2 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51:/var/run/ceph:z -v /var/log/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51:/var/log/ceph:z -v /var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9usyv3et:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:55a5c2014b4db34589ad8886606409727f633319131bb663d37e0d489e350703 inventory --format=json-pretty --with-lsm --list-all --filter-for-batch
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr stderr: blkid: error: /dev/pve/data_tmeta: No such file or directory
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 33, in <module>
/usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')())
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 62, in __init__
/usr/bin/podman: stderr self.main(self.argv)
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in newfunc
/usr/bin/podman: stderr return f(*a, **kw)
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 174, in main
/usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args)
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in dispatch
/usr/bin/podman: stderr instance.main()
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/inventory/main.py", line 58, in main
/usr/bin/podman: stderr self.format_report(Devices(filter_for_batch=self.args.filter_for_batch,
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in __init__
/usr/bin/podman: stderr self._devices = [Device(k,
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 43, in <listcomp>
/usr/bin/podman: stderr self._devices = [Device(k,
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 156, in __init__
/usr/bin/podman: stderr self.device_id = self._get_device_id()
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/device.py", line 322, in _get_device_id
/usr/bin/podman: stderr udev_data = disk.UdevData(self.path)
/usr/bin/podman: stderr File "/usr/lib/python3.9/site-packages/ceph_volume/util/disk.py", line 1355, in __init__
/usr/bin/podman: stderr raise RuntimeError(f'{path} not found.')
/usr/bin/podman: stderr RuntimeError: /dev/pve/data_tmeta not found.
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 5296, in <module>
main()
~~~~^^
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 5284, in main
r = ctx.func(ctx)
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 397, in _infer_config
return func(ctx)
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 312, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 425, in _infer_image
return func(ctx)
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 299, in _validate_fsid
return func(ctx)
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/__main__.py", line 3231, in command_ceph_volume
out, err, code = call_throws(ctx, c.run_cmd(), verbosity=CallVerbosity.QUIET_UNLESS_ERROR)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/cephadm.42daa0d45411be4c8bb16fe92e265c59cc21fc86cd0040b96409c80ba0da884c/cephadmlib/call_wrappers.py", line 310, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:55a5c2014b4db34589ad8886606409727f633319131bb663d37e0d489e350703 -e NODE_NAME=host2 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51:/var/run/ceph:z -v /var/log/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51:/var/log/ceph:z -v /var/lib/ceph/bc732ddc-8001-11f0-8000-ac220b88ba51/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs:rslave -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmp9usyv3et:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:55a5c2014b4db34589ad8886606409727f633319131bb663d37e0d489e350703 inventory --format=json-pretty --with-lsm --list-all --filter-for-batch