[SOLVED] PMG: No network after PVE upgrade from 6.4 to 7.0

[prahn]

Hi!

I just did the update from Proxmox 6.4 to 7.0. Everything worked fine, Proxmox 7.0 and all my VMs is running.
Except the LXC container of PMG 6.4: I do not have any network connection!

I have another LXC container (Dokuwiki) which is also running fine after the update, nor problems at all.
I thought about updating PMG to 7.0 as well, but since I don't havy any network connection here, I can not even start the update?!

Network config on PMG looks ok, as far as I can see:

root@pmg:~# ip a
2: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ca:78:42:e1:e0:9d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.2.5/24 brd 192.168.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::c878:42ff:fee1:e09d/64 scope link
       valid_lft forever preferred_lft forever

root@pmg:~# more /etc/network/interfaces
auto eth0
iface eth0 inet static
        address 192.168.2.5/24
        gateway 192.168.2.4

Anybody knows how to fix this?
Thank you.

Ingo

 

[Stoiko Ivanov]

* can you ping your gateway?
* else just on a hunch (although this usually does not make problems with the network working at all) - make sure nesting is enables (but only do so if it is an unprivileged container)

I hope this helps!

 

[prahn]

No, ping of the gateway does not work. :-(
I also tried to enable nesting and restarted the container but the network does not still not work...

Any other idea, what I can try?

I also wanted to install a fresh PMG 7.0 as a container for testing, but I can not find it in the repository. Is it not available as a template??

 

[Stoiko Ivanov]

No, ping of the gateway does not work. :-(
I also tried to enable nesting and restarted the container but the network does not still not work...

I'd check the journal inside the container for messages indicating the source of the error (boot it and then run `journalctl -b` inside the container)

I also wanted to install a fresh PMG 7.0 as a container for testing, but I can not find it in the repository. Is it not available as a template??

Currently it's not available - but we'll make it available soon. In the meantime you can start with a 6.4 PMG template and dist-upgrade to PMG 7.0 following the instruction in the wiki:
https://pmg.proxmox.com/wiki/index.php/Upgrade_from_6.x_to_7.0

 

[prahn]

I invstigated the journal, the only suspicious entry I found is on line 23? Is this correct?
Aug 16 08:15:29 pmg mv[63]: /bin/mv: cannot stat '/etc/network/interfaces.new': No such file or directory

Do you see any error here??

Thank you for help.

Ingo

Please find the journal here:

Aug 16 08:15:29 pmg systemd-journald[54]: Journal started
Aug 16 08:15:29 pmg systemd-journald[54]: Runtime journal (/run/log/journal/ce693d491df54d859b51fec362514136) is 8.0M, max 4.0G, 3.9G free.
Aug 16 08:15:29 pmg mount[57]: mount: /sys/kernel/config: permission denied.
Aug 16 08:15:29 pmg systemd[1]: Starting Flush Journal to Persistent Storage...
Aug 16 08:15:29 pmg systemd-journald[54]: Runtime journal (/run/log/journal/ce693d491df54d859b51fec362514136) is 8.0M, max 4.0G, 3.9G free.
Aug 16 08:15:29 pmg systemd[1]: Started Flush Journal to Persistent Storage.
Aug 16 08:15:29 pmg systemd-sysctl[58]: Couldn't write '1' to 'fs/protected_hardlinks', ignoring: Permission denied
Aug 16 08:15:29 pmg systemd-sysctl[58]: Couldn't write '1' to 'fs/protected_symlinks', ignoring: Permission denied
Aug 16 08:15:29 pmg systemd[1]: Started Create System Users.
Aug 16 08:15:29 pmg systemd[1]: Started Apply Kernel Variables.
Aug 16 08:15:29 pmg systemd[1]: Starting Create Static Device Nodes in /dev...
Aug 16 08:15:29 pmg systemd[1]: Started Helper to synchronize boot up for ifupdown.
Aug 16 08:15:29 pmg systemd[1]: Started Create Static Device Nodes in /dev.
Aug 16 08:15:29 pmg systemd[1]: Condition check resulted in udev Kernel Device Manager being skipped.
Aug 16 08:15:29 pmg systemd[1]: Reached target Local File Systems (Pre).
Aug 16 08:15:29 pmg systemd-journald[54]: Forwarding to syslog missed 1 messages.
Aug 16 08:15:29 pmg systemd[1]: Reached target Local File Systems.
Aug 16 08:15:29 pmg systemd[1]: Starting Commit Proxmox network changes...
Aug 16 08:15:29 pmg systemd[1]: Starting Raise network interfaces...
Aug 16 08:15:29 pmg systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
Aug 16 08:15:29 pmg systemd[1]: Starting Proxmox Mail Gateway Login Banner...
Aug 16 08:15:29 pmg systemd[1]: Starting Create Volatile Files and Directories...
Aug 16 08:15:29 pmg mv[63]: /bin/mv: cannot stat '/etc/network/interfaces.new': No such file or directory
Aug 16 08:15:29 pmg systemd[1]: Started Commit Proxmox network changes.
Aug 16 08:15:29 pmg systemd[1]: Started Create Volatile Files and Directories.
Aug 16 08:15:29 pmg systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Aug 16 08:15:29 pmg systemd[1]: Condition check resulted in Network Time Synchronization being skipped.
Aug 16 08:15:29 pmg systemd[1]: Reached target System Time Synchronized.
Aug 16 08:15:29 pmg systemd[1]: Started Update UTMP about System Boot/Shutdown.
Aug 16 08:15:29 pmg systemd[1]: Reached target System Initialization.
Aug 16 08:15:29 pmg systemd[1]: Started Daily Proxmox Mail Gateway activities.
Aug 16 08:15:29 pmg systemd[1]: Started Daily apt download activities.
Aug 16 08:15:29 pmg systemd[1]: Started Hourly Proxmox Mail Gateway activities.
Aug 16 08:15:29 pmg systemd[1]: Started Daily rotation of log files.
Aug 16 08:15:29 pmg systemd[1]: Started Send Daily Spam Report Mails.
Aug 16 08:15:29 pmg systemd[1]: Started Send Daily System Report Mail.
Aug 16 08:15:29 pmg systemd[1]: Started Daily apt upgrade and clean activities.
Aug 16 08:15:29 pmg systemd[1]: Reached target Basic System.
Aug 16 08:15:29 pmg systemd[1]: Started Regular background program processing daemon.
Aug 16 08:15:29 pmg systemd[1]: Condition check resulted in Login Service being skipped.
Aug 16 08:15:29 pmg systemd[1]: Condition check resulted in getty on tty2-tty6 if dbus and logind are not available being skipped.
Aug 16 08:15:29 pmg systemd[1]: Starting System Logging Service...
Aug 16 08:15:29 pmg systemd[1]: Started Daily Cleanup of Temporary Directories.
Aug 16 08:15:29 pmg systemd[1]: Reached target Timers.
Aug 16 08:15:29 pmg cron[87]: (CRON) INFO (pidfile fd = 3)
Aug 16 08:15:29 pmg systemd[1]: Starting Rotate log files...
Aug 16 08:15:29 pmg systemd[1]: Starting Daily Proxmox Mail Gateway activities...
Aug 16 08:15:30 pmg systemd[1]: Starting Send Daily System Report Mail...
Aug 16 08:15:30 pmg systemd[1]: Starting Send Daily Spam Report Mails...
Aug 16 08:15:30 pmg systemd[1]: Starting Hourly Proxmox Mail Gateway activities...
Aug 16 08:15:30 pmg cron[87]: (CRON) INFO (Running @reboot jobs)
Aug 16 08:15:30 pmg systemd[1]: Started Raise network interfaces.
Aug 16 08:15:30 pmg systemd[1]: Reached target Network.
Aug 16 08:15:30 pmg systemd[1]: Starting OpenBSD Secure Shell server...
Aug 16 08:15:30 pmg systemd[1]: Starting Permit User Sessions...
Aug 16 08:15:30 pmg systemd[1]: Reached target Network is Online.
Aug 16 08:15:30 pmg systemd[1]: Starting LSB: start or stop rrdcached...
Aug 16 08:15:30 pmg systemd[1]: Starting Daily apt download activities...
Aug 16 08:15:30 pmg systemd[1]: Starting PostgreSQL Cluster 11-main...
Aug 16 08:15:30 pmg systemd[1]: Condition check resulted in fast remote file copy program daemon being skipped.
Aug 16 08:15:30 pmg systemd[1]: Started Permit User Sessions.
Aug 16 08:15:30 pmg systemd[1]: Condition check resulted in Container Getty on /dev/tty0 being skipped.
Aug 16 08:15:30 pmg systemd[1]: Started Container Getty on /dev/tty1.
Aug 16 08:15:30 pmg systemd[1]: Started Container Getty on /dev/tty2.
Aug 16 08:15:30 pmg rsyslogd[89]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd.  [v8.1901.0]
Aug 16 08:15:30 pmg rsyslogd[89]: imklog: cannot open kernel log (/proc/kmsg): Permission denied.
Aug 16 08:15:30 pmg systemd[1]: Started System Logging Service.
Aug 16 08:15:30 pmg rsyslogd[89]: activation of module imklog failed [v8.1901.0 try https://www.rsyslog.com/e/2145 ]
Aug 16 08:15:30 pmg rsyslogd[89]:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="89" x-info="https://www.rsyslog.com"] start
Aug 16 08:15:31 pmg sshd[147]: Server listening on 0.0.0.0 port 22.
Aug 16 08:15:31 pmg sshd[147]: Server listening on :: port 22.
Aug 16 08:15:31 pmg systemd[1]: Started OpenBSD Secure Shell server.
Aug 16 08:15:32 pmg systemd[1]: Condition check resulted in Proxmox Mail Gateway Cluster Tunnel Daemon being skipped.
Aug 16 08:15:32 pmg rsyslogd[89]:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="89" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Aug 16 08:15:33 pmg rrdcached[118]: rrdcached started.
Aug 16 08:15:33 pmg systemd[1]: Started LSB: start or stop rrdcached.
Aug 16 08:15:36 pmg systemd[1]: apt-daily.service: Succeeded.
Aug 16 08:15:36 pmg systemd[1]: Started Daily apt download activities.
Aug 16 08:15:36 pmg systemd[1]: apt-daily.service: Consumed 271ms CPU time.
Aug 16 08:15:36 pmg systemd[1]: Starting Daily apt upgrade and clean activities...
Aug 16 08:15:36 pmg systemd[1]: apt-daily-upgrade.service: Succeeded.
Aug 16 08:15:36 pmg systemd[1]: Started Daily apt upgrade and clean activities.
Aug 16 08:15:36 pmg systemd[1]: apt-daily-upgrade.service: Consumed 255ms CPU time.
Aug 16 08:15:36 pmg rsyslogd[89]:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="89" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Aug 16 08:15:36 pmg systemd[1]: logrotate.service: Succeeded.
Aug 16 08:15:36 pmg systemd[1]: Started Rotate log files.
Aug 16 08:15:36 pmg systemd[1]: logrotate.service: Consumed 292ms CPU time.
Aug 16 08:15:37 pmg systemd[1]: Started Proxmox Mail Gateway Login Banner.
Aug 16 08:15:37 pmg systemd[1]: Started Console Getty.
Aug 16 08:15:37 pmg systemd[1]: Reached target Login Prompts.
Aug 16 08:15:37 pmg pmgqm[99]: purging database
Aug 16 08:15:38 pmg pmg-daily[96]: cleanup removed 65 entries from statistic database
Aug 16 08:15:38 pmg pmg-daily[96]: starting task UPID:pmg:00000121:01032E72:611A1EAA:aptupdate::root@pam:
Aug 16 08:15:38 pmg pmgqm[99]: removed 20 spam quarantine files
Aug 16 08:15:39 pmg pmgreport[98]: unable to connect to localhost at port 10025 at /usr/share/perl5/PMG/Utils.pm line 214.
Aug 16 08:15:39 pmg systemd[1]: pmgreport.service: Succeeded.
Aug 16 08:15:39 pmg systemd[1]: Started Send Daily System Report Mail.
Aug 16 08:15:39 pmg systemd[1]: pmgreport.service: Consumed 613ms CPU time.
Aug 16 08:15:39 pmg systemd[1]: Started PostgreSQL Cluster 11-main.
Aug 16 08:15:39 pmg systemd[1]: Starting PostgreSQL RDBMS...
Aug 16 08:15:39 pmg systemd[1]: Started PostgreSQL RDBMS.
Aug 16 08:15:39 pmg systemd[1]: Starting Sync Proxmox Configuration...
Aug 16 08:15:39 pmg systemd[1]: pmgspamreport.service: Succeeded.
Aug 16 08:15:39 pmg systemd[1]: Started Send Daily Spam Report Mails.
Aug 16 08:15:39 pmg systemd[1]: pmgspamreport.service: Consumed 956ms CPU time.
Aug 16 08:15:41 pmg pmgdb[305]: Analyzing/Upgrading existing Databases...done
Aug 16 08:15:42 pmg pmgconfig[324]: reloading pmg-smtp-filter: can't open '/run/pmg-smtp-filter.cfg' - No such file or directory
Aug 16 08:15:42 pmg systemd[1]: Started Sync Proxmox Configuration.
Aug 16 08:15:42 pmg systemd[1]: Starting Proxmox Mail Gateway Policy Daemon...
Aug 16 08:15:42 pmg systemd[1]: Starting Proxmox Mail Gateway API Daemon...
Aug 16 08:15:42 pmg systemd[1]: Started ClamAV virus database updater.
Aug 16 08:15:42 pmg systemd[1]: Condition check resulted in Proxmox Mail Gateway Database Mirror Daemon being skipped.
Aug 16 08:15:42 pmg systemd[1]: Starting Clam AntiVirus userspace daemon...
Aug 16 08:15:42 pmg systemd[1]: Starting Proxmox SMTP Filter Daemon...
Aug 16 08:15:42 pmg systemd[1]: Starting Postfix Mail Transport Agent (instance -)...
Aug 16 08:15:43 pmg systemd[1]: Started Clam AntiVirus userspace daemon.
Aug 16 08:15:43 pmg freshclam[439]: WARNING: Ignoring deprecated option SafeBrowsing at /etc/clamav/freshclam.conf:18
Aug 16 08:15:43 pmg clamd[447]: Received 0 file descriptor(s) from systemd.
Aug 16 08:15:43 pmg clamd[447]: clamd daemon 0.103.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Aug 16 08:15:43 pmg clamd[447]: Log file size limited to 4294967295 bytes.
Aug 16 08:15:43 pmg freshclam[439]: freshclam daemon 0.103.2 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
Aug 16 08:15:43 pmg freshclam[439]: ClamAV update process started at Mon Aug 16 08:15:43 2021
Aug 16 08:15:43 pmg freshclam[439]: ClamAV update process started at Mon Aug 16 08:15:43 2021
Aug 16 08:15:43 pmg clamd[447]: Reading databases from /var/lib/clamav/
Aug 16 08:15:43 pmg clamd[447]: Not loading PUA signatures.
Aug 16 08:15:43 pmg clamd[447]: Bytecode: Security mode set to "TrustSigned".
Aug 16 08:15:43 pmg pmgpolicy[494]: Process Backgrounded
Aug 16 08:15:43 pmg pmgpolicy[494]: 2021/08/16-08:15:43 main (type Net::Server::PreForkSimple) starting! pid(494)
Aug 16 08:15:43 pmg pmgpolicy[494]: Binding to TCP port 10022 on host 127.0.0.1 with IPv4
Aug 16 08:15:43 pmg pmgpolicy[494]: Group Not Defined.  Defaulting to EGID '0'
Aug 16 08:15:43 pmg pmgpolicy[494]: User Not Defined.  Defaulting to EUID '0'
Aug 16 08:15:43 pmg pmgpolicy[494]: Setting up serialization via flock
Aug 16 08:15:43 pmg pmgpolicy[494]: Policy daemon (re)started
Aug 16 08:15:43 pmg pmgpolicy[494]: Beginning prefork (5 processes)
Aug 16 08:15:43 pmg pmgpolicy[494]: Starting "5" children
Aug 16 08:15:43 pmg systemd[1]: Started Proxmox Mail Gateway Policy Daemon.
Aug 16 08:15:43 pmg pmg-hourly[100]: start syncing ldap profile 'zimbra' (192.168.2.2)
Aug 16 08:15:44 pmg systemd[1]: pmg-smtp-filter.service: Can't open PID file /run/pmg-smtp-filter.pid (yet?) after start: No such file or directory
Aug 16 08:15:44 pmg pmg-smtp-filter[521]: Process Backgrounded
Aug 16 08:15:44 pmg pmg-smtp-filter[521]: 2021/08/16-08:15:44 main (type Net::Server::PreFork) starting! pid(521)
Aug 16 08:15:44 pmg pmg-smtp-filter[521]: Binding to TCP port 10023 on host 127.0.0.1 with IPv4
Aug 16 08:15:44 pmg pmg-smtp-filter[521]: Binding to TCP port 10024 on host 127.0.0.1 with IPv4
Aug 16 08:15:44 pmg pmg-smtp-filter[521]: Group Not Defined.  Defaulting to EGID '0'
Aug 16 08:15:44 pmg pmg-smtp-filter[521]: User Not Defined.  Defaulting to EUID '0'
Aug 16 08:15:44 pmg pmg-smtp-filter[521]: Setting up serialization via flock
Aug 16 08:15:44 pmg pmg-smtp-filter[521]: Filter daemon (re)started (max. 34 processes)
Aug 16 08:15:44 pmg systemd[1]: Started Proxmox SMTP Filter Daemon.
Aug 16 08:15:44 pmg systemd[1]: pmgdaemon.service: Can't open PID file /run/pmgdaemon.pid (yet?) after start: No such file or directory
Aug 16 08:15:44 pmg pmgdaemon[549]: starting server
Aug 16 08:15:44 pmg pmgdaemon[549]: starting 3 worker(s)
Aug 16 08:15:44 pmg pmgdaemon[549]: worker 552 started
Aug 16 08:15:44 pmg pmgdaemon[549]: worker 553 started
Aug 16 08:15:44 pmg pmgdaemon[549]: worker 554 started
Aug 16 08:15:44 pmg systemd[1]: Started Proxmox Mail Gateway API Daemon.
Aug 16 08:15:44 pmg systemd[1]: Starting Proxmox Mail Gateway API...
Aug 16 08:15:44 pmg postfix/postfix-script[584]: warning: symlink leaves directory: /etc/postfix/./makedefs.out
Aug 16 08:15:45 pmg postfix/postfix-script[622]: starting the Postfix mail system
Aug 16 08:15:45 pmg postfix/master[624]: daemon started -- version 3.4.14, configuration /etc/postfix
Aug 16 08:15:45 pmg systemd[1]: Started Postfix Mail Transport Agent (instance -).
Aug 16 08:15:45 pmg systemd[1]: Starting Postfix Mail Transport Agent...
Aug 16 08:15:45 pmg systemd[1]: pmgproxy.service: Can't open PID file /run/pmgproxy/pmgproxy.pid (yet?) after start: No such file or directory

 

[Stoiko Ivanov]

invstigated the journal, the only suspicious entry I found is on line 23? Is this correct?
Aug 16 08:15:29 pmg mv[63]: /bin/mv: cannot stat '/etc/network/interfaces.new': No such file or directory

yes - log looks ok - an the warning message can be ignored (/etc/network/interfaces.new gets created if you change the network config on the GUI and then gets moved to /etc/network/interfaces upon reboot)

hmm - could you please post the complete container config (`pct config <VMID>` (replace <VMID> by the ID of the pmg container)

else describe your network setup? - where is 192.168.2.4 located - can you try to tcpdump (in the container, on the bridge on the PVE host, on the physical interface where the packets go out) and see where the packets get lost?

 

[prahn]

Here is the config of the container:

root@pve1:~# pct config 115
arch: amd64
cores: 2
features: nesting=1
hostname: pmg
memory: 4096
nameserver: 192.168.2.4
net0: name=eth0,bridge=vmbr1,gw=192.168.2.4,hwaddr=CA:78:42:E1:E0:9D,ip=192.168.2.5/24,type=veth
onboot: 0
ostype: debian
rootfs: pve1-storage:115/vm-115-disk-0.raw,size=40G
startup: order=2,up=60,down=60
swap: 4096
unprivileged: 1
root@pve1:~# pct config 115
arch: amd64
cores: 2
features: nesting=1
hostname: pmg
memory: 4096
nameserver: 192.168.2.4
net0: name=eth0,bridge=vmbr1,gw=192.168.2.4,hwaddr=CA:78:42:E1:E0:9D,ip=192.168.2.5/24,type=veth
onboot: 0
ostype: debian
rootfs: pve1-storage:115/vm-115-disk-0.raw,size=40G
startup: order=2,up=60,down=60
swap: 4096
unprivileged: 1

 

[prahn]

Ok, thank you for asking the right questions... error resolved!

This is the only VM on this host, which is connected to the DMZ interface (vmbr1). And it looks that with the update of the host from buster to bullseye the hardware ID of the network interface changed. (from ens3 to enp179s0)
After correcting the network config and rebooting the host, the container is back to work!

 

[Stoiko Ivanov]

After correcting the network config and rebooting the host, the container is back to work!

nice - glad you found and resolved the issue!

 

[prahn]

Thank you for your consistent help!