Hi everyone,
I ran into something today that cost me a couple of hours, and I thought it was worth sharing — partly because the failure is completely silent, and partly because I'm genuinely not sure whose job it is to fix it.
I had moved a node from my lab to a customer site and changed the DNS servers from the GUI (Datacenter → Node → DNS). Everything looked fine. Mail notifications, though, had quietly stopped arriving.
Here's what I eventually found:
```
# cat /etc/resolv.conf
search example.local
nameserver 192.168.5.4
nameserver 1.1.1.1
nameserver 8.8.8.8
# cat /var/spool/postfix/etc/resolv.conf
search example.local
nameserver 192.168.3.250 <-- the DNS of my lab, unreachable from here
```
Postfix runs chrooted, so it can't read `/etc/resolv.conf` and the Debian package keeps a copy inside the chroot. That copy is a snapshot taken when Postfix starts — and changing the DNS in the GUI doesn't touch it. From that moment Postfix kept resolving against a nameserver that doesn't exist on that network, so it couldn't look up any MX, and every notification just sat in the queue. No error anywhere. `mailq` was the only place that knew.
A couple of details I had to dig out, in case they save someone else the time:
- The sync is done by `/usr/lib/postfix/syncfiles.pl`, called from `configure-instance.sh` → `post-install`, and that runs on every instance start. It copies only if size or mtime differ, then matches the mtime — which is why on a healthy node the two files have the same mtime down to the nanosecond. If the mtimes differ, the copy is stale.
- `systemctl restart postfix` fixes it immediately.
- Running `/usr/lib/postfix/configure-instance.sh` by hand does not work: it expects the instance environment variables and exits with `E: This is internal script used by postfix`. I spent a while thinking I had re-synced when I hadn't — the error goes to stderr and is easy to miss.
To be fair, I don't think anybody is strictly doing anything wrong here.
PVE writes `/etc/resolv.conf`, which is its job. The Debian package syncs the chroot at startup, which is what it documents. Postfix itself says in its own documentation that the chroot needs copies of these files and that you have to keep them current. The gap is the interval between two Postfix starts, and nobody owns it.
What makes me think it's still worth raising here is that PVE is the only component that knows both things: that the DNS just changed, and that Postfix is present and is the transport its own notifications rely on. And the notification docs recommend sendmail over SMTP precisely because sendmail/Postfix gives you queuing and retry — so this hits the configuration the documentation points you to.
So, two questions rather than a demand:
1. Would it make sense for the DNS panel to refresh the Postfix chroot (or just warn) after writing `/etc/resolv.conf`? A `systemctl try-reload-or-restart postfix` would do it, though I appreciate that PVE doesn't want to own an MTA that's technically optional.
2. If that's out of scope — and it may well be — would a line in the notifications chapter be welcome? Something like "if you change the node's DNS, restart Postfix so the chroot copy is refreshed". I'd happily write it up if that's useful.
Either way, if you ever see notifications stop with no apparent cause, this is worth two seconds:
```
diff /etc/resolv.conf /var/spool/postfix/etc/resolv.conf && echo "in sync" mailq | tail -1
```
Thanks <3
I ran into something today that cost me a couple of hours, and I thought it was worth sharing — partly because the failure is completely silent, and partly because I'm genuinely not sure whose job it is to fix it.
I had moved a node from my lab to a customer site and changed the DNS servers from the GUI (Datacenter → Node → DNS). Everything looked fine. Mail notifications, though, had quietly stopped arriving.
Here's what I eventually found:
```
# cat /etc/resolv.conf
search example.local
nameserver 192.168.5.4
nameserver 1.1.1.1
nameserver 8.8.8.8
# cat /var/spool/postfix/etc/resolv.conf
search example.local
nameserver 192.168.3.250 <-- the DNS of my lab, unreachable from here
```
Postfix runs chrooted, so it can't read `/etc/resolv.conf` and the Debian package keeps a copy inside the chroot. That copy is a snapshot taken when Postfix starts — and changing the DNS in the GUI doesn't touch it. From that moment Postfix kept resolving against a nameserver that doesn't exist on that network, so it couldn't look up any MX, and every notification just sat in the queue. No error anywhere. `mailq` was the only place that knew.
A couple of details I had to dig out, in case they save someone else the time:
- The sync is done by `/usr/lib/postfix/syncfiles.pl`, called from `configure-instance.sh` → `post-install`, and that runs on every instance start. It copies only if size or mtime differ, then matches the mtime — which is why on a healthy node the two files have the same mtime down to the nanosecond. If the mtimes differ, the copy is stale.
- `systemctl restart postfix` fixes it immediately.
- Running `/usr/lib/postfix/configure-instance.sh` by hand does not work: it expects the instance environment variables and exits with `E: This is internal script used by postfix`. I spent a while thinking I had re-synced when I hadn't — the error goes to stderr and is easy to miss.
To be fair, I don't think anybody is strictly doing anything wrong here.
PVE writes `/etc/resolv.conf`, which is its job. The Debian package syncs the chroot at startup, which is what it documents. Postfix itself says in its own documentation that the chroot needs copies of these files and that you have to keep them current. The gap is the interval between two Postfix starts, and nobody owns it.
What makes me think it's still worth raising here is that PVE is the only component that knows both things: that the DNS just changed, and that Postfix is present and is the transport its own notifications rely on. And the notification docs recommend sendmail over SMTP precisely because sendmail/Postfix gives you queuing and retry — so this hits the configuration the documentation points you to.
So, two questions rather than a demand:
1. Would it make sense for the DNS panel to refresh the Postfix chroot (or just warn) after writing `/etc/resolv.conf`? A `systemctl try-reload-or-restart postfix` would do it, though I appreciate that PVE doesn't want to own an MTA that's technically optional.
2. If that's out of scope — and it may well be — would a line in the notifications chapter be welcome? Something like "if you change the node's DNS, restart Postfix so the chroot copy is refreshed". I'd happily write it up if that's useful.
Either way, if you ever see notifications stop with no apparent cause, this is worth two seconds:
```
diff /etc/resolv.conf /var/spool/postfix/etc/resolv.conf && echo "in sync" mailq | tail -1
```
Thanks <3
Last edited:
