Yes. I think that is the cause.
I don't think it is stress as the server runs relatively chill (15% cpu, 0.4% io delay, nvme ssd)
However, I do suspect that the nic's drivers might be causing this issue. Running lspci gives me this result:
Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI...