you have a point, could be heat, what is the ambient temp. and the temp being reported by the hardware - check your ipmi
nvme drives are pretty reliable and fail rarely unlike earlier sata drives.
can you remove a few drives to test or is it running in production, can you put the vm's on the...