Dear forum,
After ceph upgrade pacific to latest pacific 16.2.11 then to reef, rbd-mirror always crash and never create/continue image.
log :
After ceph upgrade pacific to latest pacific 16.2.11 then to reef, rbd-mirror always crash and never create/continue image.
log :
Code:
-86> 2024-01-19T00:20:44.333+0100 7f9a759546c0 4 set_mon_vals no callback set
-85> 2024-01-19T00:20:44.333+0100 7f9a791b2a80 5 monclient: authenticate success, global_id 46524
-84> 2024-01-19T00:20:44.333+0100 7f9a791b2a80 10 monclient: _renew_subs
-83> 2024-01-19T00:20:44.333+0100 7f9a791b2a80 10 monclient: _send_mon_message to mon.pve-145-239-10-207 at v2:[2001:41d0:203:14cf::]:3300/0
-82> 2024-01-19T00:20:44.333+0100 7f9a791b2a80 10 monclient: _renew_subs
-81> 2024-01-19T00:20:44.333+0100 7f9a791b2a80 10 monclient: _send_mon_message to mon.pve-145-239-10-207 at v2:[2001:41d0:203:14cf::]:3300/0
-80> 2024-01-19T00:20:44.333+0100 7f9a791b2a80 1 librados: init done
-79> 2024-01-19T00:20:44.333+0100 7f9a741516c0 4 mgrc handle_mgr_map Got map version 13
-78> 2024-01-19T00:20:44.333+0100 7f9a741516c0 4 mgrc handle_mgr_map Active mgr is now [v2:[2001:41d0:203:14cf::]:6834/21635,v1:[2001:41d0:203:14cf::]:6835/21635]
-77> 2024-01-19T00:20:44.333+0100 7f9a741516c0 4 mgrc reconnect Starting new session with [v2:[2001:41d0:203:14cf::]:6834/21635,v1:[2001:41d0:203:14cf::]:6835/21635]
-76> 2024-01-19T00:20:44.333+0100 7f9a791b2a80 1 mgrc service_daemon_register rbd-mirror.46524 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.1 (850293cdaae6621945e1191aa8c28ea2918269c3) reef (stable),ceph_version_short=18.2.1,cpu=Intel(R) Xeon(R) CPU D-1521 @ 2.40GHz,distro=debian,distro_description=Debian GNU/Linux 12 (bookworm),distro_version=12,hostname=pve-145-239-10-207,id=pve-145-239-10-207,instance_id=46524,kernel_description=#1 SMP PREEMPT_DYNAMIC PMX 6.5.11-7 (2023-12-05T09:44Z),kernel_version=6.5.11-7-pve,mem_swap_kb=0,mem_total_kb=32751332,os=Linux}
-75> 2024-01-19T00:20:44.333+0100 7f9a771576c0 10 monclient: get_auth_request con 0x55b608d86000 auth_method 0
-74> 2024-01-19T00:20:44.337+0100 7f9a791b2a80 10 monclient: _send_mon_message to mon.pve-145-239-10-207 at v2:[2001:41d0:203:14cf::]:3300/0
-73> 2024-01-19T00:20:44.337+0100 7f9a741516c0 10 monclient: handle_get_version_reply finishing 1 version 1405
-72> 2024-01-19T00:20:44.337+0100 7f9a741516c0 4 mgrc handle_mgr_configure stats_period=5
-71> 2024-01-19T00:20:44.337+0100 7f9a741516c0 4 mgrc handle_mgr_configure updated stats threshold: 5
-70> 2024-01-19T00:20:44.341+0100 7f9a781596c0 10 monclient: get_auth_request con 0x55b608d87800 auth_method 0
-69> 2024-01-19T00:20:44.353+0100 7f9a779586c0 10 monclient: get_auth_request con 0x55b608d87400 auth_method 0
-68> 2024-01-19T00:20:44.369+0100 7f9a771576c0 10 monclient: get_auth_request con 0x55b608dd2400 auth_method 0
-67> 2024-01-19T00:20:44.377+0100 7f9a781596c0 10 monclient: get_auth_request con 0x55b608dd2c00 auth_method 0
-66> 2024-01-19T00:20:44.381+0100 7f9a779586c0 10 monclient: get_auth_request con 0x55b608dd3000 auth_method 0
-65> 2024-01-19T00:20:44.393+0100 7f9a791b2a80 10 start_mon_command cmd=[{"prefix": "config-key get", "key": "rbd/mirror/peer/5/d12dea08-2218-4c70-9abc-f53c4a225c73"}]
-64> 2024-01-19T00:20:44.393+0100 7f9a791b2a80 10 monclient: _send_command 1 [{"prefix": "config-key get", "key": "rbd/mirror/peer/5/d12dea08-2218-4c70-9abc-f53c4a225c73"}]
-63> 2024-01-19T00:20:44.393+0100 7f9a791b2a80 10 monclient: _send_mon_message to mon.pve-145-239-10-207 at v2:[2001:41d0:203:14cf::]:3300/0
-62> 2024-01-19T00:20:44.393+0100 7f9a741516c0 10 monclient: handle_mon_command_ack 1 [{"prefix": "config-key get", "key": "rbd/mirror/peer/5/d12dea08-2218-4c70-9abc-f53c4a225c73"}]
-61> 2024-01-19T00:20:44.393+0100 7f9a741516c0 10 monclient: _finish_command 1 = mon:2
-60> 2024-01-19T00:20:44.393+0100 7f9a771576c0 10 monclient: get_auth_request con 0x55b608dd3c00 auth_method 0
-59> 2024-01-19T00:20:44.401+0100 7f9a781596c0 10 monclient: get_auth_request con 0x55b608df8000 auth_method 0
-58> 2024-01-19T00:20:44.409+0100 7f9a791b2a80 10 start_mon_command cmd=[{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]
-57> 2024-01-19T00:20:44.409+0100 7f9a791b2a80 10 monclient: _send_command 2 [{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]
-56> 2024-01-19T00:20:44.409+0100 7f9a791b2a80 10 monclient: _send_mon_message to mon.pve-145-239-10-207 at v2:[2001:41d0:203:14cf::]:3300/0
-55> 2024-01-19T00:20:44.409+0100 7f9a741516c0 10 monclient: handle_mon_command_ack 2 [{"prefix": "config-key get", "key": "rbd/mirror/site_name"}]
-54> 2024-01-19T00:20:44.409+0100 7f9a741516c0 10 monclient: _finish_command 2 = mon:2
-53> 2024-01-19T00:20:44.977+0100 7f9a681396c0 5 rbd::mirror::PoolReplayer: 0x55b608d50f00 handle_remote_pool_meta_updated: remote_pool_meta=mirror_uuid=5856167c-8896-4416-9ca1-6075a93de93b, mirror_peer_uuid=7b7c1a44-4368-48d8-a2f4-3a4932cb82df
-52> 2024-01-19T00:20:45.281+0100 7f9a791b2a80 5 prioritycache tune_memory target: 4294967296 mapped: 21438464 unmapped: 17850368 heap: 39288832 old mem: 2836086988 new mem: 2845369267
-51> 2024-01-19T00:20:45.281+0100 7f9a649326c0 5 asok(0x55b6081ea000) register_command rbd mirror status testpool1 ceph-master hook 0x55b608fac500
-50> 2024-01-19T00:20:45.281+0100 7f9a649326c0 5 asok(0x55b6081ea000) register_command rbd mirror start testpool1 ceph-master hook 0x55b608fac500
-49> 2024-01-19T00:20:45.281+0100 7f9a649326c0 5 asok(0x55b6081ea000) register_command rbd mirror stop testpool1 ceph-master hook 0x55b608fac500
-48> 2024-01-19T00:20:45.281+0100 7f9a649326c0 5 asok(0x55b6081ea000) register_command rbd mirror restart testpool1 ceph-master hook 0x55b608fac500
-47> 2024-01-19T00:20:45.281+0100 7f9a649326c0 5 asok(0x55b6081ea000) register_command rbd mirror flush testpool1 ceph-master hook 0x55b608fac500
-46> 2024-01-19T00:20:45.281+0100 7f9a649326c0 5 asok(0x55b6081ea000) register_command rbd mirror leader release testpool1 ceph-master hook 0x55b608fac500
-45> 2024-01-19T00:20:45.305+0100 7f9a6e9466c0 5 rbd::mirror::Instances: 0x55b608ebb0e0 handle_acked: instance_ids=[46527]
-44> 2024-01-19T00:20:45.309+0100 7f9a759546c0 5 rbd::mirror::PoolWatcher: 0x55b607f28960 init:
-43> 2024-01-19T00:20:45.309+0100 7f9a759546c0 5 rbd::mirror::PoolWatcher: 0x55b607f28960 register_watcher:
-42> 2024-01-19T00:20:45.333+0100 7f9a6f1476c0 5 rbd::mirror::PoolWatcher: 0x55b607f28960 handle_register_watcher: r=0
-41> 2024-01-19T00:20:45.333+0100 7f9a6f1476c0 5 rbd::mirror::PoolWatcher: 0x55b607f28960 refresh_images:
-40> 2024-01-19T00:20:45.337+0100 7f9a6e9466c0 5 rbd::mirror::PoolWatcher: 0x55b607f28960 handle_refresh_images: r=0
-39> 2024-01-19T00:20:45.337+0100 7f9a751536c0 5 rbd::mirror::ImageMap: 0x55b608ebb320 update_images: peer_uuid=, added_count=0, removed_count=0
-38> 2024-01-19T00:20:45.337+0100 7f9a751536c0 5 rbd::mirror::PoolWatcher: 0x55b607f29860 init:
-37> 2024-01-19T00:20:45.337+0100 7f9a751536c0 5 rbd::mirror::PoolWatcher: 0x55b607f29860 register_watcher:
-36> 2024-01-19T00:20:45.393+0100 7f9a681396c0 5 rbd::mirror::PoolWatcher: 0x55b607f29860 handle_register_watcher: r=0
-35> 2024-01-19T00:20:45.393+0100 7f9a681396c0 5 rbd::mirror::PoolWatcher: 0x55b607f29860 refresh_images:
-34> 2024-01-19T00:20:45.405+0100 7f9a6893a6c0 5 rbd::mirror::PoolWatcher: 0x55b607f29860 handle_refresh_images: r=0
-33> 2024-01-19T00:20:45.405+0100 7f9a759546c0 5 rbd::mirror::ImageMap: 0x55b608ebb320 update_images: peer_uuid=5856167c-8896-4416-9ca1-6075a93de93b, added_count=1, removed_count=0
-32> 2024-01-19T00:20:45.405+0100 7f9a759546c0 5 rbd::mirror::ImageMap: 0x55b608ebb320 update_images_added: peer_uuid=5856167c-8896-4416-9ca1-6075a93de93b, global_image_ids=[21b3a3b0-0ac8-417a-9443-a2ec22891c28]
-31> 2024-01-19T00:20:45.405+0100 7f9a759546c0 5 rbd::mirror::image_map::Policy: 0x55b607f4e540 add_image: global_image_id=21b3a3b0-0ac8-417a-9443-a2ec22891c28
-30> 2024-01-19T00:20:45.405+0100 7f9a759546c0 5 asok(0x55b6081ea000) register_command rbd mirror deletion status testpool1 hook 0x55b6090b1e00
-29> 2024-01-19T00:20:45.405+0100 7f9a759546c0 5 rbd::mirror::image_deleter::TrashWatcher: 0x55b6090a0900 init:
-28> 2024-01-19T00:20:45.437+0100 7f9a6f1476c0 5 rbd::mirror::image_deleter::TrashWatcher: 0x55b6090a0900 register_watcher:
-27> 2024-01-19T00:20:45.453+0100 7f9a6e9466c0 5 rbd::mirror::image_deleter::TrashWatcher: 0x55b6090a0900 handle_register_watcher: r=0
-26> 2024-01-19T00:20:45.453+0100 7f9a6e9466c0 5 rbd::mirror::image_deleter::TrashWatcher: 0x55b6090a0900 trash_list: last_image_id=
-25> 2024-01-19T00:20:45.457+0100 7f9a6f1476c0 5 rbd::mirror::image_deleter::TrashWatcher: 0x55b6090a0900 handle_trash_list: r=0
-24> 2024-01-19T00:20:45.461+0100 7f9a759546c0 5 rbd::mirror::Instances: 0x55b608ebb0e0 unblock_listener:
-23> 2024-01-19T00:20:45.461+0100 7f9a759546c0 5 rbd::mirror::Instances: 0x55b608ebb0e0 notify_instances_added: instance_ids=[46527]
-22> 2024-01-19T00:20:45.461+0100 7f9a759546c0 5 rbd::mirror::PoolReplayer: 0x55b608d50f00 handle_instances_added: instance_ids=[46527]
-21> 2024-01-19T00:20:45.461+0100 7f9a759546c0 5 rbd::mirror::image_map::Policy: 0x55b607f4e540 add_instances: instance_ids=[46527]
-20> 2024-01-19T00:20:45.461+0100 7f9a759546c0 5 rbd::mirror::image_map::Policy: 0x55b607f4e540 add_instances: initial instance update
-19> 2024-01-19T00:20:45.461+0100 7f9a759546c0 5 rbd::mirror::image_map::SimplePolicy: 0x55b607f4e540 do_shuffle_add_instances: images per instance=1
-18> 2024-01-19T00:20:45.461+0100 7f9a759546c0 5 rbd::mirror::image_map::Policy: 0x55b607f4e540 add_instances: shuffling global_image_ids=[]
-17> 2024-01-19T00:20:45.469+0100 7f9a751536c0 5 rbd::mirror::Instances: 0x55b608ebb0e0 handle_acked: instance_ids=[46527]
-16> 2024-01-19T00:20:46.405+0100 7f9a7214d6c0 5 rbd::mirror::image_map::Policy: 0x55b607f4e540 start_action: global_image_id=21b3a3b0-0ac8-417a-9443-a2ec22891c28, state=ASSOCIATING, action_type=MAP_UPDATE
-15> 2024-01-19T00:20:46.405+0100 7f9a7214d6c0 5 rbd::mirror::image_map::Policy: 0x55b607f4e540 execute_policy_action: global_image_id=21b3a3b0-0ac8-417a-9443-a2ec22891c28, policy_action=MAP
-14> 2024-01-19T00:20:46.405+0100 7f9a7214d6c0 5 rbd::mirror::image_map::Policy: 0x55b607f4e540 map: global_image_id=21b3a3b0-0ac8-417a-9443-a2ec22891c28, instance_id=46527
-13> 2024-01-19T00:20:46.405+0100 7f9a7214d6c0 5 rbd::mirror::ImageMap: 0x55b608ebb320 update_image_mapping: updates=[{global_image_id=21b3a3b0-0ac8-417a-9443-a2ec22891c28, instance_id=46527}], removes=[]
-12> 2024-01-19T00:20:46.433+0100 7f9a759546c0 5 rbd::mirror::image_map::Policy: 0x55b607f4e540 finish_action: global_image_id=21b3a3b0-0ac8-417a-9443-a2ec22891c28, state=ASSOCIATING, action_type=MAP_UPDATE, r=0
-11> 2024-01-19T00:20:47.337+0100 7f9a7314f6c0 10 monclient: tick
-10> 2024-01-19T00:20:47.337+0100 7f9a7314f6c0 10 monclient: _check_auth_tickets
-9> 2024-01-19T00:20:47.433+0100 7f9a7214d6c0 5 rbd::mirror::image_map::Policy: 0x55b607f4e540 start_action: global_image_id=21b3a3b0-0ac8-417a-9443-a2ec22891c28, state=ASSOCIATING, action_type=ACQUIRE
-8> 2024-01-19T00:20:47.433+0100 7f9a7214d6c0 5 rbd::mirror::ImageMap: 0x55b608ebb320 notify_listener_acquire_release_images: acquire=[{global_image_id=21b3a3b0-0ac8-417a-9443-a2ec22891c28, instance_id=46527}], release=[]
-7> 2024-01-19T00:20:47.433+0100 7f9a7214d6c0 5 rbd::mirror::NamespaceReplayer: 0x55b608e82d00 handle_acquire_image: global_image_id=21b3a3b0-0ac8-417a-9443-a2ec22891c28, instance_id=46527
-6> 2024-01-19T00:20:47.445+0100 7f9a751536c0 5 asok(0x55b6081ea000) register_command rbd mirror flush testpool1/21b3a3b0-0ac8-417a-9443-a2ec22891c28 hook 0x55b608fac900
-5> 2024-01-19T00:20:47.445+0100 7f9a751536c0 5 asok(0x55b6081ea000) register_command rbd mirror restart testpool1/21b3a3b0-0ac8-417a-9443-a2ec22891c28 hook 0x55b608fac900
-4> 2024-01-19T00:20:47.445+0100 7f9a751536c0 5 asok(0x55b6081ea000) register_command rbd mirror start testpool1/21b3a3b0-0ac8-417a-9443-a2ec22891c28 hook 0x55b608fac900
-3> 2024-01-19T00:20:47.445+0100 7f9a751536c0 5 asok(0x55b6081ea000) register_command rbd mirror status testpool1/21b3a3b0-0ac8-417a-9443-a2ec22891c28 hook 0x55b608fac900
-2> 2024-01-19T00:20:47.445+0100 7f9a751536c0 5 asok(0x55b6081ea000) register_command rbd mirror stop testpool1/21b3a3b0-0ac8-417a-9443-a2ec22891c28 hook 0x55b608fac900
-1> 2024-01-19T00:20:47.445+0100 7f9a759546c0 5 rbd::mirror::image_map::Policy: 0x55b607f4e540 finish_action: global_image_id=21b3a3b0-0ac8-417a-9443-a2ec22891c28, state=ASSOCIATING, action_type=ACQUIRE, r=0
0> 2024-01-19T00:20:47.617+0100 7f9a6893a6c0 -1 *** Caught signal (Aborted) **
in thread 7f9a6893a6c0 thread_name:io_context_pool