We've stepped through the commands shown in the logs by hand, and managed to generate an strace of the ceph-osd command that fails (attached). This bit looks interesting:
...
madvise(0x55aaa41de000, 1990656, MADV_FREE) = 0
pread64(20, 0x55aaa41de000, 1048576, 3840699006976) = -1 EILSEQ...