Follow up, after editing my dnsmasq server to increase the TTLs on the internal domains, proxmox is not honoring them. I still see a huge number of queries, so for now I'm going to set the metrics server to an IP address as a break fix.
Ahh doh. Good catch. Just looked around for that fix and found it:
I actually had to fix this on my OpenWRT dnsmaq config, by default it uses ttl=0 for local, just like Pihole does.