SRV networking - Reload: ifreload -a failed

J

jester

Renowned Member
Proxmox Subscriber
Aug 17, 2014
23
1
68
On two of our nodes we are getting the below error when trying to make simple network changes (like change a comment) and applying them.
A reboot does not solve the issue. The interfaces-file looks OK.

Anyone with an idea what's going on and how to solve this?

Code: 
bond0 : error: bond0: failed to set vid `{127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 

...

3813, 3814, 3815, 3816, 3817, 3818, 3819, 3820, 3821, 3822, 3823, 3824, 3825, 3826, 3827, 3828, 3829, 3830, 3831, 3832, 3833, 3834, 3835, 3836, 3837, 3838, 3839, 3840, 3841, 3842, 3843, 3844, 3845, 3846, 3847, 3848, 3849, 3850, 3851, 3852, 3853, 3854, 3855, 3856, 3857, 3858, 3859, 3860, 3861, 3862, 3863, 3864, 3865, 3866, 3867, 3868, 3869, 3870, 3871, 3872, 3873, 3874, 3875, 3876, 3877, 3878, 3879, 3880, 3881, 3882, 3883, 3884, 3885, 3886, 3887, 3888, 3889, 3890, 3891, 3892, 3893, 3894, 3895, 3896, 3897, 3898, 3899, 3900, 3901, 3902, 3903, 3904, 3905, 3906, 3907, 3908, 3909, 3910, 3911, 3912, 3913, 3914, 3915, 3916, 3917, 3918, 3919, 3920, 3921, 3922, 3923, 3924, 3925, 3926, 3927, 3928, 3929, 3930, 3931, 3932, 3933, 3934, 3935, 3936, 3937, 3938, 3939, 3940, 3941, 3942, 3943, 3944, 3945, 3946, 3947, 3948, 3949, 3950, 3951, 3952, 3953, 3954, 3955, 3956, 3957, 3958, 3959, 3960, 3961, 3962, 3963, 3964, 3965, 3966, 3967, 3968, 3969, 3970, 3971, 3972, 3973, 3974, 3975, 3976, 3977, 3978, 3979, 3980, 3981, 3982, 3983, 3984, 3985, 3986, 3987, 3988, 3989, 3990, 3991, 3992, 3993, 3994, 3995, 3996, 3997, 3998, 3999, 4000, 4001, 4002, 4003, 4004, 4005, 4006, 4007, 4008, 4009, 4010, 4011, 4012, 4013, 4014, 4015, 4016, 4017, 4018, 4019, 4020, 4021, 4022, 4023, 4024, 4025, 4026, 4027, 4028, 4029, 4030, 4031, 4032, 4033, 4034, 4035, 4036, 4037, 4038, 4039, 4040, 4041, 4042, 4043, 4044, 4045, 4046, 4047, 4048, 4049, 4050, 4051, 4052, 4053, 4054, 4055, 4056, 4057, 4058, 4059, 4060, 4061, 4062, 4063, 4064, 4065, 4066, 4067, 4068, 4069, 4070, 4071, 4072, 4073, 4074, 4075, 4076, 4077, 4078, 4079, 4080, 4081, 4082, 4083, 4084, 4085, 4086, 4087, 4088, 4089, 4090, 4091, 4092, 4093, 4094}` (cmd '/sbin/bridge -force -batch - [vlan add vid 127-4094 dev bond0 ]' failed: returned 1 (RTNETLINK answers: No space left on device
TASK ERROR: command 'ifreload -a' failed: exit code 1
 
jester said:
The interfaces-file looks OK.
Click to expand...
Please post your interfaces file, perhaps that will help us to understand.
I assume it has worked before... Were there any changes that could explain this behavior?
 
RTNETLINK answers: No space left on device
Click to expand...

Do you by chance have an old Mellanox NIC? Those can only do 512 VLANs and you get this message when the maximum amount of VLANs has been reached.
 
  • Like
Reactions: Stoiko Ivanov
shanreich said:
Do you by chance have an old Mellanox NIC? Those can only do 512 VLANs and you get this message when the maximum amount of VLANs has been reached.
Click to expand...
as similar issues exist for many vendors (Intel for certain, but I'd assume others as well) - I'd suggest checking the output of `journalctl -f` while running `ifreload -a` - if that does not provide the needed information - check `journalctl -b` - for the NIC initialization part.
 
Sorry for the late reply.

@fireon : our interfaces -file:
Code: 
auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto enp1s0
iface enp1s0 inet manual

auto enp1s0d1
iface enp1s0d1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1 enp1s0
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp1s0
#LAN Bond

auto bond1
iface bond1 inet manual
        bond-slaves eno2 enp1s0d1
        bond-miimon 100
        bond-mode active-backup
        bond-primary enp1s0d1
#SAN Bond

auto vmbr0
iface vmbr0 inet static
        address 192.9.200.21/24
        gateway 192.9.200.1
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
#LAN

auto vmbr1
iface vmbr1 inet static
        address 172.16.10.21/24
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#SAN

source /etc/network/interfaces.d/*

@shanreich & @Stoiko Ivanov : catching journalct -f did not render much useful i think:

Code: 
Nov 25 11:30:37 pmx01 systemd[1]: Reloading postfix@-.service - Postfix Mail Transport Agent (instance -)...
Nov 25 11:30:37 pmx01 postfix/postfix-script[2344591]: refreshing the Postfix mail system
Nov 25 11:30:37 pmx01 postfix/master[2004]: reload -- version 3.7.11, configuration /etc/postfix
Nov 25 11:30:37 pmx01 systemd[1]: Reloaded postfix@-.service - Postfix Mail Transport Agent (instance -).
Nov 25 11:30:37 pmx01 systemd[1]: Reloading postfix.service - Postfix Mail Transport Agent...
Nov 25 11:30:37 pmx01 systemd[1]: Reloaded postfix.service - Postfix Mail Transport Agent.
Nov 25 11:30:37 pmx01 systemd[1]: Reloading postfix@-.service - Postfix Mail Transport Agent (instance -)...
Nov 25 11:30:37 pmx01 postfix/postfix-script[2344622]: refreshing the Postfix mail system
Nov 25 11:30:37 pmx01 postfix/master[2004]: reload -- version 3.7.11, configuration /etc/postfix
Nov 25 11:30:37 pmx01 systemd[1]: Reloaded postfix@-.service - Postfix Mail Transport Agent (instance -).
Nov 25 11:30:37 pmx01 systemd[1]: Reloading postfix.service - Postfix Mail Transport Agent...
Nov 25 11:30:37 pmx01 systemd[1]: Reloaded postfix.service - Postfix Mail Transport Agent.
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix@-.service - Postfix Mail Transport Agent (instance -)...
Nov 25 11:30:38 pmx01 postfix/postfix-script[2344653]: refreshing the Postfix mail system
Nov 25 11:30:38 pmx01 postfix/master[2004]: reload -- version 3.7.11, configuration /etc/postfix
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix@-.service - Postfix Mail Transport Agent (instance -).
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix.service - Postfix Mail Transport Agent...
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix.service - Postfix Mail Transport Agent.
Nov 25 11:30:38 pmx01 kernel: vmbr0: the hash_elasticity option has been deprecated and is always 16
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix@-.service - Postfix Mail Transport Agent (instance -)...
Nov 25 11:30:38 pmx01 postfix/postfix-script[2344685]: refreshing the Postfix mail system
Nov 25 11:30:38 pmx01 postfix/master[2004]: reload -- version 3.7.11, configuration /etc/postfix
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix@-.service - Postfix Mail Transport Agent (instance -).
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix.service - Postfix Mail Transport Agent...
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix.service - Postfix Mail Transport Agent.
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix@-.service - Postfix Mail Transport Agent (instance -)...
Nov 25 11:30:38 pmx01 postfix/postfix-script[2344716]: refreshing the Postfix mail system
Nov 25 11:30:38 pmx01 postfix/master[2004]: reload -- version 3.7.11, configuration /etc/postfix
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix@-.service - Postfix Mail Transport Agent (instance -).
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix.service - Postfix Mail Transport Agent...
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix.service - Postfix Mail Transport Agent.
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix@-.service - Postfix Mail Transport Agent (instance -)...
Nov 25 11:30:38 pmx01 postfix/postfix-script[2344747]: refreshing the Postfix mail system
Nov 25 11:30:38 pmx01 postfix/master[2004]: reload -- version 3.7.11, configuration /etc/postfix
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix@-.service - Postfix Mail Transport Agent (instance -).
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix.service - Postfix Mail Transport Agent...
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix.service - Postfix Mail Transport Agent.
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix@-.service - Postfix Mail Transport Agent (instance -)...
Nov 25 11:30:38 pmx01 postfix/postfix-script[2344779]: refreshing the Postfix mail system
Nov 25 11:30:38 pmx01 postfix/master[2004]: reload -- version 3.7.11, configuration /etc/postfix
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix@-.service - Postfix Mail Transport Agent (instance -).
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix.service - Postfix Mail Transport Agent...
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix.service - Postfix Mail Transport Agent.
Nov 25 11:30:38 pmx01 kernel: vmbr1: the hash_elasticity option has been deprecated and is always 16
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix@-.service - Postfix Mail Transport Agent (instance -)...
Nov 25 11:30:38 pmx01 postfix/postfix-script[2344812]: refreshing the Postfix mail system
Nov 25 11:30:38 pmx01 postfix/master[2004]: reload -- version 3.7.11, configuration /etc/postfix
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix@-.service - Postfix Mail Transport Agent (instance -).
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix.service - Postfix Mail Transport Agent...
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix.service - Postfix Mail Transport Agent.
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix@-.service - Postfix Mail Transport Agent (instance -)...
Nov 25 11:30:38 pmx01 postfix/postfix-script[2344844]: refreshing the Postfix mail system
Nov 25 11:30:38 pmx01 postfix/master[2004]: reload -- version 3.7.11, configuration /etc/postfix
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix@-.service - Postfix Mail Transport Agent (instance -).
Nov 25 11:30:38 pmx01 systemd[1]: Reloading postfix.service - Postfix Mail Transport Agent...
Nov 25 11:30:38 pmx01 systemd[1]: Reloaded postfix.service - Postfix Mail Transport Agent.

The most relevant parts (i hope) from the journalctl -b output:
Code: 
# journalctl -b | grep kernel | grep -E 'igb|mlx4'

Nov 19 15:55:22 pmx01 kernel: mlx4_core: Mellanox ConnectX core driver v4.0-0
Nov 19 15:55:22 pmx01 kernel: mlx4_core: Initializing 0000:01:00.0
Nov 19 15:55:22 pmx01 kernel: igb: Intel(R) Gigabit Ethernet Network Driver
Nov 19 15:55:22 pmx01 kernel: igb: Copyright (c) 2007-2014 Intel Corporation.
Nov 19 15:55:22 pmx01 kernel: igb 0000:03:00.0: added PHC on eth0
Nov 19 15:55:22 pmx01 kernel: igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
Nov 19 15:55:22 pmx01 kernel: igb 0000:03:00.0: eth0: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:ec:b2:16
Nov 19 15:55:22 pmx01 kernel: igb 0000:03:00.0: eth0: PBA No: 010000-000
Nov 19 15:55:22 pmx01 kernel: igb 0000:03:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
Nov 19 15:55:22 pmx01 kernel: igb 0000:04:00.0: added PHC on eth1
Nov 19 15:55:22 pmx01 kernel: igb 0000:04:00.0: Intel(R) Gigabit Ethernet Network Connection
Nov 19 15:55:22 pmx01 kernel: igb 0000:04:00.0: eth1: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:ec:b2:17
Nov 19 15:55:22 pmx01 kernel: igb 0000:04:00.0: eth1: PBA No: 010000-000
Nov 19 15:55:22 pmx01 kernel: igb 0000:04:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
Nov 19 15:55:22 pmx01 kernel: igb 0000:04:00.0 eno2: renamed from eth1
Nov 19 15:55:22 pmx01 kernel: igb 0000:03:00.0 eno1: renamed from eth0
Nov 19 15:55:22 pmx01 kernel: mlx4_core 0000:01:00.0: DMFS high rate steer mode is: disabled performance optimized steering
Nov 19 15:55:22 pmx01 kernel: mlx4_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
Nov 19 15:55:22 pmx01 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
Nov 19 15:55:22 pmx01 kernel: mlx4_en 0000:01:00.0: Activating port:1
Nov 19 15:55:22 pmx01 kernel: mlx4_en: 0000:01:00.0: Port 1: Using 4 TX rings
Nov 19 15:55:22 pmx01 kernel: mlx4_en: 0000:01:00.0: Port 1: Using 4 RX rings
Nov 19 15:55:22 pmx01 kernel: mlx4_en: 0000:01:00.0: Port 1: Initializing port
Nov 19 15:55:22 pmx01 kernel: mlx4_en 0000:01:00.0: registered PHC clock
Nov 19 15:55:22 pmx01 kernel: mlx4_en 0000:01:00.0: Activating port:2
Nov 19 15:55:22 pmx01 kernel: mlx4_en: 0000:01:00.0: Port 2: Using 4 TX rings
Nov 19 15:55:22 pmx01 kernel: mlx4_en: 0000:01:00.0: Port 2: Using 4 RX rings
Nov 19 15:55:22 pmx01 kernel: mlx4_en: 0000:01:00.0: Port 2: Initializing port
Nov 19 15:55:22 pmx01 kernel: <mlx4_ib> mlx4_ib_probe: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
Nov 19 15:55:22 pmx01 kernel: <mlx4_ib> mlx4_ib_probe: counter index 2 for port 1 allocated 1
Nov 19 15:55:22 pmx01 kernel: <mlx4_ib> mlx4_ib_probe: counter index 3 for port 2 allocated 1
Nov 19 15:55:22 pmx01 kernel: mlx4_core 0000:01:00.0 enp1s0d1: renamed from eth1
Nov 19 15:55:22 pmx01 kernel: mlx4_core 0000:01:00.0 enp1s0: renamed from eth0
Nov 19 15:55:22 pmx01 kernel: mlx4_en: enp1s0d1: Link Up
Nov 19 15:55:22 pmx01 kernel: mlx4_en: enp1s0: Link Up
Nov 19 15:55:25 pmx01 kernel: mlx4_en: enp1s0: Steering Mode 1
Nov 19 15:55:25 pmx01 kernel: mlx4_en: enp1s0: Link Up
Nov 19 15:55:25 pmx01 kernel: mlx4_core 0000:01:00.0 enp1s0: entered allmulticast mode
Nov 19 15:55:25 pmx01 kernel: mlx4_core 0000:01:00.0 enp1s0: entered promiscuous mode
Nov 19 15:55:25 pmx01 kernel: mlx4_en: enp1s0d1: Steering Mode 1
Nov 19 15:55:25 pmx01 kernel: mlx4_en: enp1s0d1: Link Up
Nov 19 15:55:25 pmx01 kernel: mlx4_core 0000:01:00.0 enp1s0d1: entered allmulticast mode
 
Seems like you have a Mellanox ConnectX4 which, if my memory serves me correct, only supports 512 VLANs. You can customize the VLAN IDs available on a bridge with our newest release, 8.3, in the Network settings of the bridge when you check the advanced checkbox.
 
shanreich said:
Seems like you have a Mellanox ConnectX4 which, if my memory serves me correct, only supports 512 VLANs. You can customize the VLAN IDs available on a bridge with our newest release, 8.3, in the Network settings of the bridge when you check the advanced checkbox.
Click to expand...
You are right!
Just upgraded to 8.3 this morning, so I have the option set the VLAN IDs. I can set it to a range of 2-126, after that it starts erroring out. For the moment i'm just listing the used VLANs 'cause we're only using a few.

Thanks for the solution/workaround!
 
  • Like
Reactions: shanreich
You must log in or register to reply here.
Top Bottom