PVE 8.0, reloadnetworkall takes hours to complete

I

islepnev

Active Member
May 2, 2013
6
0
41
Hello all,

Running SDN with vlans and evpn. Multiple clusters, PVE versions 7.4-15 and 8.0.3, upgraded with no-subscription repository.
After upgrading PVE from 7.4 to 8.0 I have noticed that SDN reload process slowed down by factor of ~100 and takes hours to complete.

Where should I look to diagnose the problem?

Thanks in advance!
 
Philipp Hufnagl said:
Code: 
journalctl -b
Click to expand...

Thanks.
After cleaning up, these lines look suspicious.
Interfaces are likely to be reloaded sequentially with regular time intervals: 20s, 40s, 60s.

Jul 12 15:48:50 c4n05 kernel: vmbr0: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:49:10 c4n05 kernel: vmbr1824: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:49:51 c4n05 kernel: vlan1805: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:50:51 c4n05 kernel: vlan1810: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:51:52 c4n05 kernel: vlan1811: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:52:53 c4n05 kernel: vlan1812: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:53:53 c4n05 kernel: vlan1813: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:54:54 c4n05 kernel: vlan1814: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:55:55 c4n05 kernel: vlan1815: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:56:56 c4n05 kernel: vlan1825: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:57:57 c4n05 kernel: vlan60: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:58:37 c4n05 kernel: vmbr0v1805: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:59:18 c4n05 kernel: vmbr0v1810: the hash_elasticity option has been deprecated and is always 16
Jul 12 15:59:58 c4n05 kernel: vmbr0v1811: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:00:38 c4n05 kernel: vmbr0v1812: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:01:19 c4n05 kernel: vmbr0v1813: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:01:59 c4n05 kernel: vmbr0v1814: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:02:40 c4n05 kernel: vmbr0v1815: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:03:21 c4n05 kernel: vmbr0v1825: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:04:02 c4n05 kernel: vmbr0v60: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:04:42 c4n05 kernel: priv001: the hash_elasticity option has been deprecated and is always 16
Jul 12 16:05:23 c4n05 kernel: priv002: the hash_elasticity option has been deprecated and is always 16
 
Can I see more of the log? Having a 2nd eye on thinks often helps because its really easy to overlook something.

Also what does your cat /etc/network/interfaces say?

Also: Please use the code feature around the output when posting
 
spirit said:
can you try "ifreload -a -d" to reload with debug log ?
Click to expand...

Thank you for hint,

... debug: bond0.1808: up : running script /etc/network/if-up.d/ntpsec-ntpdate info: executing /etc/network/if-up.d/ntpsec-ntpdate

stuck on this line. Checking processes:

root 2553911 0.0 0.0 2576 640 pts/0 S+ 12:21 0:00 /bin/sh /etc/network/if-up.d/ntpsec-ntpdate root 2553916 0.0 0.0 5460 1664 pts/0 S+ 12:21 0:00 \_ flock -n /run/lock/ntpsec-ntpdate /usr/sbin/ntpdate-debian -s root 2553917 0.0 0.0 2576 1536 pts/0 S+ 12:21 0:00 \_ /bin/sh /usr/sbin/ntpdate -s 0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org root 2553918 0.3 0.0 26280 18688 pts/0 S+ 12:21 0:00 \_ /usr/bin/python3 /bin/ntpdig -s -S -M 500 -t 1 0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp root 2553919 0.0 0.0 8924 2816 pts/0 S+ 12:21 0:00 \_ logger -t ntpdate

Isolated network, no internet, ntpdate has no luck contacting NTP servers and got stuck.

I definitely have no pool configured in /etc/ntp.conf:
root@pve:~# cat /etc/ntp.conf disable monitor statsdir /var/log/ntpstats restrict default kod nomodify notrap nopeer noquery restrict -6 default kod nomodify notrap nopeer noquery restrict 127.0.0.1 restrict -6 ::1 restrict 10.0.0.0 mask 255.255.0.0 nomodify notrap nopeer server 10.<hidden> iburst server 10.<hidden> iburst driftfile /var/lib/ntp/drift root@pve:~#

1) why /etc/ntp.conf is ignored and internet ntp servers are queried?
2) why network reload script tries to synchronise time at all?
I do not have /etc/network/if-up.d/ntpsec-ntpdate script in PVE 7 installations. That's why ifreload worked smoothly before upgrade.
From man page:
ntpdate-debian is identical to ntpdate(8) except that it uses the configuration in /etc/default/ntpsec-ntpdate by default.

Ok, here is problematic config file:

root@pve:~# cat /etc/default/ntpsec-ntpdate # The settings in this file are used by the program ntpdate-debian, but not # by the upstream program ntpdate. # Set to "yes" to take the server list from /etc/ntpsec/ntp.conf, so you only # have to keep it in one place. NTPDATE_USE_NTP_CONF=yes # List of NTP servers to use (Separate multiple servers with spaces.) # Not used if NTPDATE_USE_NTP_CONF is yes. NTPSERVERS="0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org 3.debian.pool.ntp.org" # Additional options to pass to ntpdate NTPOPTIONS="" # Set to "yes" to ignore DHCP servers returned by DHCP. IGNORE_DHCP=""
 
aderumier said:
Install chrony, this is the default ntp service installed with proxmox. (And purge old ntpsec ifup script )
Click to expand...
I had chrony installed and running before upgrading 7 to 8. No ntpd.
ntpsec-ntpdate package was automatically installed by dist-upgrade, that's nonsense.

apt purge ntpsec-ntpdate
 
could you post the entry in /var/log/apt/history.log corresponding to your 7 to 8 upgrade?
 
fabian said:
could you post the entry in /var/log/apt/history.log corresponding to your 7 to 8 upgrade?
Click to expand...
Start-Date: 2023-06-29 23:21:51
Commandline: apt dist-upgrade

Split log into install, upgrade and remove sections, one package per line.

root@pve8:~# grep ntp *.log install.log:ntpsec-ntpdate:amd64 (1.2.2+dfsg1-1, automatic), install.log:ntpsec-ntpdig:amd64 (1.2.2+dfsg1-1, automatic), install.log:python3-ntp:amd64 (1.2.2+dfsg1-1, automatic), upgrade.log:ntpdate:amd64 (1:4.2.8p15+dfsg-1, 1:4.2.8p15+dfsg-2~1.2.2+dfsg1-1),

ntpdate package dependencies have changed.

PVE7:
root@pve7:~# apt depends ntpdate ntpdate Depends: netbase Depends: libc6 (>= 2.17) Depends: libssl1.1 (>= 1.1.0)

PVE8:
root@pve8:~# apt depends ntpdate ntpdate Depends: ntpsec-ntpdate root@pve8:~# apt depends ntpsec-ntpdate ntpsec-ntpdate Depends: netbase Depends: ntpsec-ntpdig (= 1.2.2+dfsg1-1) Breaks: ntpdate (<< 1:4.2.8p15+dfsg-2~) Replaces: ntpdate (<< 1:4.2.8p15+dfsg-2~)
 

Attachments

yes, because the ntpdate package is empty now, and just serves the purpose of upgrading to ntpdate-ntpsec. thanks!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom