No, swap is used if Linux "feels like" the "unused" memory can be better used for file cache (which shows as unused because it can be make available instantly). Your system looks fine.
Only when Linux starts also reading from swap a lot, so called "thrashing" with high I/O delay, then there is a problem. The swap-line is unfortunately red because it uses the standard color scheme.