I moved some VMs to another freshly setup host and could remove the USB network adapter.
With these actions, the memory usage dropped to around 50%, kernel logs during backup, see below.
But the VM connectivity issues are still there.
Mar 27 00:05:05 proxmox4 kernel: usb 2-7: reset SuperSpeed...