Hi,
It's strange indeed... I even found out that it's even not because of a OpenVZ VM itself. I found out that one of my VM's using OpenVZ has 4 IP-addresses attached, on 2 of those it seems the latency run up to 80-100ms, the others just seems to be normal between 15 and 20ms at the moment...