I think you are in the same virtio IO wait/delay boat with us :(
https://forum.proxmox.com/threads/3000-msec-ping-and-packet-drops-with-virtio-under-load.36687/
Unfortunately there isn't any official Proxmox team response yet about this and the only temporary workaround found so far is to...