Last thing I tried to find what's causing this issue was to try and monitor its temperature to figure out if the nvme ssd overheats for some reason, because when I lose access I get some EXT4 filesystem errors (see attached logs). I did some tests where I downloaded some huge (60GB) files through one container, noticed the temperatures increasing, but as soon as the file was downloaded they went back to normal.
I now have 15 days uptime and hadn't had the issue since and I'm thinking if the script I have set up to monitor the ssd temperatures is actually preventing the ssd from going into some kind of deep power saving state. My SSD is a Samsung 990 Pro SSD - 1TB and the command I have running every 5" in the background is nvme smart-log /dev/nvme0. Now I understand this might be a coincidence and the issue might return, but maybe if more people give it a go we have verify if it's the solution easier. Then we could find a proper way to prevent deep power-saving states (e.g.
GRUB_CMDLINE_LINUX_DEFAULT="quiet nvme_core.default_ps_max_latency_us=0" in /etc/default/grub). This will probably increase the idle power consumption a bit, but ensure stability in the system.