-What hardware do you use (cpu, nic, switch, storage)?
-What happens with CPU usage/IO delays when speed drops on the host?
-Did you install PVE from official ISO or on Debian?
-Can you test speed with Linux VM machine (ubuntu, debian) or with iperf directly from host to virtual machine and from...