Deferred Mail - Cannot Flush Queue

[vertigo2049]

proxmox-mailgateway: 8.0.1pmg-api: 8.0.11pmg-gui: 4.0.3proxmox-kernel-helper: 8.1.0pve-kernel-5.15: 7.4-9proxmox-kernel-6.5: 6.5.11-7proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7pve-kernel-5.4: 6.4-20proxmox-kernel-6.2.16-20-pve: 6.2.16-20proxmox-kernel-6.2: 6.2.16-20proxmox-kernel-6.2.16-19-pve: 6.2.16-19pve-kernel-5.15.131-2-pve: 5.15.131-3pve-kernel-5.15.126-1-pve: 5.15.126-1pve-kernel-5.15.107-2-pve: 5.15.107-2pve-kernel-5.15.85-1-pve: 5.15.85-1pve-kernel-5.15.83-1-pve: 5.15.83-1pve-kernel-5.15.74-1-pve: 5.15.74-1pve-kernel-5.4.203-1-pve: 5.4.203-1pve-kernel-5.4.30-1-pve: 5.4.30-1clamav-daemon: 1.0.3+dfsg-1~deb12u1ifupdown: 0.8.41libarchive-perl: 3.6.2libjs-extjs: 7.0.0-4libjs-framework7: 4.4.7-2libproxmox-acme-perl: 1.5.0libproxmox-acme-plugins: 1.5.0libpve-apiclient-perl: 3.3.0libpve-common-perl: 8.1.0libpve-http-server-perl: 5.0.5libxdgmime-perl: 1.1.0lvm2: 2.03.16-2pmg-docs: 8.0.1pmg-i18n: 3.1.5pmg-log-tracker: 2.4.1proxmox-mini-journalreader: 1.4.0proxmox-offline-mirror-helper: 0.6.3proxmox-spamassassin: 4.0.0-4proxmox-widget-toolkit: 4.1.3pve-firmware: 3.9-1pve-xtermjs: 5.3.0-3zfsutils-linux: 2.2.2-pve1

Suddenly mails are deferred and now i cannot get them re-delivered.

I have tried flushing from both the CLI and the GUI but they disappear then reappear.

The reason is status=deferred (delivery temporarily suspended: lost connection with 127.0.0.1[127.0.0.1] while sending end of data -- message may be sent more than once)

 

[vertigo2049]

Solved - ClamAV terrorizing the system.
Causes so many issues, ended up masking the services as we only use the spam filtering.

 

[gmpreussner]

I'm having the same problem as of this morning. Disabling and masking ClamAV with systemctl mask clamav-daemon.service clamav-freshclam.service did not work for me; rebooted PMG afterwards as well. My Queues are still filling up with undeliverable mail.

In the PMG syslog I'm seeing log entries of the form:

status=deferred (lost connection with mail.mycompany.com[x.y.z.w] while sending MAIL FROM)
status=deferred (lost connection with mail.mycompany.com[x.y.z.w] while sending end of data -- message may be sent more than once)
status=deferred (lost connection with someoneelse.com[h.i.j.k] while sending end of data -- message may be sent more than once)

It seems that connections to both destination mail servers for outbound email and relay transports for inbound emails are affected.

In the postfix logs of my incoming mail server that PMG is proxying to I'm seeing for every flush attempt from PMG:

lost connection after DATA (0 bytes) from pmg.mycompany.com[a.b.c.d]

I'm on PMG 7.3.11

 

[gmpreussner]

I upgraded to PMG 8.0, and the problem persists. It looks like the connections to all outgoing SMTP connections on port 25 to any destination server are prematurely terminated. This started happening today for the first time after the server has been running continuously for about six weeks.

I'm running PMG on Debian (11 initially, 12 as of today).

I made two interesting observations:
- I can send emails normally to destination SMTP servers via telnet from the command line
- Occasionally, PMG will successfully deliver an email on its own (approx. 1-2 in 100)

 

[gmpreussner]

So, after troubleshooting this for five hours, things suddenly started working again, and I have no idea why. I kept monitoring the postfix log and noticed that every few minutes PMG would attempt to deliver 1 email and succeed. I then flushed selected emails from the GUI, which also worked. Then I flushed the entire queue that has accumulated throughout the day, which also succeeded.

I was just about to take TCP packet captures on the PMG server, but for now everything appears to be working normally again :|

Edit: It might be worth pointing out that ClamAV has also updated itself and is now fully up to date. Perhaps that's why things started working again.