You are not the only one. When more than 5 to 6 different users on different Workstations open their W8.1 or W2012 VM's in noVNC, this dramatically slows down also in actual firefox. I'm doing the same, restarting the pveproxy. This helps a little. The same thing happens when using spiceproxy...