This is clearly a router/gateway issue, doing an ARP Cache broadcast towards the OVH Gateway, restores speed again for 7 minutes!
Do this on a VM and see for yourselves:
arping -c 1 -S VMIP GATEWAYIP
EDIT: arping included in iputils is not the same, i had to do apt-get install arping for it to...