My next step was to re-create the monitors manually by disabling the service and removing /var/lib/ceph/mon/<hostname> directory.
Then ran 'pveceph mon create'. After awhile it timed-out. Running 'journalctl on the failed monitor service shows the following:
Jun 25 13:29:03 pve-test-7-to-8 systemd[1]: Started
ceph-mon@pve-test-7-to-8.service - Ceph cluster monitor daemon.
Jun 25 13:29:04 pve-test-7-to-8 ceph-mon[8161]: *** Caught signal (Illegal instruction) **
Jun 25 13:29:04 pve-test-7-to-8 ceph-mon[8161]: in thread 7fe8c0b1da00 thread_name:ceph-mon
Jun 25 13:29:04 pve-test-7-to-8 ceph-mon[8161]: ceph version 17.2.6 (810db68029296377607028a6c6da1ec06f5a2b27) quincy (stable)
Jun 25 13:29:04 pve-test-7-to-8 ceph-mon[8161]: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90) [0x7fe8c11bdf90]
...
Jun 25 13:29:55 pve-test-7-to-8 ceph-mon[9402]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jun 25 13:29:55 pve-test-7-to-8 systemd[1]:
ceph-mon@pve-test-7-to-8.service: Main process exited, code=killed, status=4/ILL
Jun 25 13:29:55 pve-test-7-to-8 systemd[1]:
ceph-mon@pve-test-7-to-8.service: Failed with result 'signal'.
Jun 25 13:30:05 pve-test-7-to-8 systemd[1]:
ceph-mon@pve-test-7-to-8.service: Scheduled restart job, restart counter is at 6.
Jun 25 13:30:05 pve-test-7-to-8 systemd[1]: Stopped
ceph-mon@pve-test-7-to-8.service - Ceph cluster monitor daemon.
Jun 25 13:30:05 pve-test-7-to-8 systemd[1]:
ceph-mon@pve-test-7-to-8.service: Start request repeated too quickly.
Jun 25 13:30:05 pve-test-7-to-8 systemd[1]:
ceph-mon@pve-test-7-to-8.service: Failed with result 'signal'.
Jun 25 13:30:05 pve-test-7-to-8 systemd[1]: Failed to start
ceph-mon@pve-test-7-to-8.service - Ceph cluster monitor daemon.
Seems to point to a corrupt binary, compile, or something else. No idea.
Going to do a clean install of Proxmox 8 and see if I get the same error when manually creating the monitors.
I'm testing the migration script on a test cluster which does have quorum and nodes can ping each other.
This is why you always test before pushing it to production.