I'm running into similar things.
I have 3 nodes, (3x MS-01s, all running Proxmox. 2x 96GB ram, 1x 64GB, 1TB boot drive on each, 2x 2TB drives for ceph). Workload is 1VM for PBS, 1VM for ubuntu server for my talos config, and then 1 talos manager and 1 talos worker per node. For some reason...