Is it actually a problem? If you have a lot of disk I/O from the VMs, Linux will prefer to use the memory for file cache and buffers and push less often used memory to swap. Linux will not start pulling it from swap back to memory by itself unless there is actual demand for the memory stored in swap. Therefore, if you really have 8GB of memory that is allocated but not used, it will slowly move towards swap and stay there.
If you want Linux to read back all the swapped memory instead of using that memory file cache use the swapoff;swapon trick. Changing vm.swappiness in /etc/sysctl.conf can also be used to change this balance between more cache and less swap usage, but swap will still grow unless the swapped memory is requested again by applications.
