PVE 8.0, reloadnetworkall takes hours to complete

[islepnev]

Hello all,

Running SDN with vlans and evpn. Multiple clusters, PVE versions 7.4-15 and 8.0.3, upgraded with no-subscription repository.
After upgrading PVE from 7.4 to 8.0 I have noticed that SDN reload process slowed down by factor of ~100 and takes hours to complete.

Where should I look to diagnose the problem?

Thanks in advance!

 

[Philipp Hufnagl]

journalctl -b

 

[islepnev]

journalctl -b

Thanks.
After cleaning up, these lines look suspicious.
Interfaces are likely to be reloaded sequentially with regular time intervals: 20s, 40s, 60s.

Jul 12 15:48:50 c4n05 kernel: vmbr0: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:49:10 c4n05 kernel: vmbr1824: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:49:51 c4n05 kernel: vlan1805: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:50:51 c4n05 kernel: vlan1810: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:51:52 c4n05 kernel: vlan1811: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:52:53 c4n05 kernel: vlan1812: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:53:53 c4n05 kernel: vlan1813: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:54:54 c4n05 kernel: vlan1814: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:55:55 c4n05 kernel: vlan1815: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:56:56 c4n05 kernel: vlan1825: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:57:57 c4n05 kernel: vlan60: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:58:37 c4n05 kernel: vmbr0v1805: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:59:18 c4n05 kernel: vmbr0v1810: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:59:58 c4n05 kernel: vmbr0v1811: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:00:38 c4n05 kernel: vmbr0v1812: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:01:19 c4n05 kernel: vmbr0v1813: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:01:59 c4n05 kernel: vmbr0v1814: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:02:40 c4n05 kernel: vmbr0v1815: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:03:21 c4n05 kernel: vmbr0v1825: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:04:02 c4n05 kernel: vmbr0v60: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:04:42 c4n05 kernel: priv001: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:05:23 c4n05 kernel: priv002: the hash_elasticity option has been deprecated and is always 16

 

[Philipp Hufnagl]

Can I see more of the log? Having a 2nd eye on thinks often helps because its really easy to overlook something.

Also what does your cat /etc/network/interfaces say?

Also: Please use the code feature around the output when posting

 

[spirit]

can you try "ifreload -a -d" to reload with debug log ?

 

[islepnev]

can you try "ifreload -a -d" to reload with debug log ?

Thank you for hint,

... debug: bond0.1808: up : running script /etc/network/if-up.d/ntpsec-ntpdate info: executing /etc/network/if-up.d/ntpsec-ntpdate

stuck on this line. Checking processes:

root 2553911 0.0 0.0 2576 640 pts/0 S+ 12:21 0:00 /bin/sh /etc/network/if-up.d/ntpsec-ntpdate root 2553916 0.0 0.0 5460 1664 pts/0 S+ 12:21 0:00 \_ flock -n /run/lock/ntpsec-ntpdate /usr/sbin/ntpdate-debian -s root 2553917 0.0 0.0 2576 1536 pts/0 S+ 12:21 0:00 \_ /bin/sh /usr/sbin/ntpdate -s 0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org root 2553918 0.3 0.0 26280 18688 pts/0 S+ 12:21 0:00 \_ /usr/bin/python3 /bin/ntpdig -s -S -M 500 -t 1 0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp root 2553919 0.0 0.0 8924 2816 pts/0 S+ 12:21 0:00 \_ logger -t ntpdate

Isolated network, no internet, ntpdate has no luck contacting NTP servers and got stuck.

I definitely have no pool configured in /etc/ntp.conf:
root@pve:~# cat /etc/ntp.conf disable monitor statsdir /var/log/ntpstats restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery restrict 127.0.0.1 restrict -6 ::1 restrict 10.0.0.0 mask 255.255.0.0 nomodify notrap nopeer server 10.<hidden> iburst server 10.<hidden> iburst driftfile /var/lib/ntp/drift root@pve:~#

1) why /etc/ntp.conf is ignored and internet ntp servers are queried?
2) why network reload script tries to synchronise time at all?
I do not have /etc/network/if-up.d/ntpsec-ntpdate script in PVE 7 installations. That's why ifreload worked smoothly before upgrade.
From man page:
ntpdate-debian is identical to ntpdate(8) except that it uses the configuration in /etc/default/ntpsec-ntpdate by default.

Ok, here is problematic config file:

root@pve:~# cat /etc/default/ntpsec-ntpdate # The settings in this file are used by the program ntpdate-debian, but not # by the upstream program ntpdate. # Set to "yes" to take the server list from /etc/ntpsec/ntp.conf, so you only # have to keep it in one place. NTPDATE_USE_NTP_CONF=yes # List of NTP servers to use (Separate multiple servers with spaces.) # Not used if NTPDATE_USE_NTP_CONF is yes. NTPSERVERS="0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org" # Additional options to pass to ntpdate NTPOPTIONS="" # Set to "yes" to ignore DHCP servers returned by DHCP. IGNORE_DHCP=""

 

[aderumier]

Install chrony, this is the default ntp service installed with proxmox. (And purge old ntpsec ifup script )

 

[islepnev]

Install chrony, this is the default ntp service installed with proxmox. (And purge old ntpsec ifup script )

I had chrony installed and running before upgrading 7 to 8. No ntpd.
ntpsec-ntpdate package was automatically installed by dist-upgrade, that's nonsense.

apt purge ntpsec-ntpdate

 

[fabian]

could you post the entry in /var/log/apt/history.log corresponding to your 7 to 8 upgrade?

 

[islepnev]

could you post the entry in /var/log/apt/history.log corresponding to your 7 to 8 upgrade?

Start-Date: 2023-06-29 23:21:51
Commandline: apt dist-upgrade

Split log into install, upgrade and remove sections, one package per line.

root@pve8:~# grep ntp *.log install.log:ntpsec-ntpdate:amd64 (1.2.2+dfsg1-1, automatic), install.log:ntpsec-ntpdig:amd64 (1.2.2+dfsg1-1, automatic), install.log:python3-ntp:amd64 (1.2.2+dfsg1-1, automatic), upgrade.log:ntpdate:amd64 (1:4.2.8p15+dfsg-1, 1:4.2.8p15+dfsg-2~1.2.2+dfsg1-1),

ntpdate package dependencies have changed.

PVE7:
root@pve7:~# apt depends ntpdate ntpdate Depends: netbase Depends: libc6 (>= 2.17) Depends: libssl1.1 (>= 1.1.0)

PVE8:
root@pve8:~# apt depends ntpdate ntpdate Depends: ntpsec-ntpdate root@pve8:~# apt depends ntpsec-ntpdate ntpsec-ntpdate Depends: netbase Depends: ntpsec-ntpdig (= 1.2.2+dfsg1-1) Breaks: ntpdate (<< 1:4.2.8p15+dfsg-2~) Replaces: ntpdate (<< 1:4.2.8p15+dfsg-2~)

 

[fabian]

yes, because the ntpdate package is empty now, and just serves the purpose of upgrading to ntpdate-ntpsec. thanks!