Nested interfaces.d/sdn directory in interfaces.d

[ns33]

I am unable to recreate this mishap, but I figured it's worth posting about in case anyone else has a "it worked a second ago"

I have cluster, with a handful of evpn zones, and an external bgp neighbor. Everything was great for months, until I made some changes to a few vnets, applied my changes and noticed all external routing stopped. (Routing within a zone was fine.) My external switch was still a bgp neighbor, but nothing could get in nor out. Tracing the route shows it would hit the gateway and just stop. From internal out, it would hit the vnet gateway and stop. Removing my vnet changes and re-applying made no difference. The SDN status page also showed everything was 'ok' and all zones were available.

It wasn't until I did an ifreload -a where I saw the error; warning: error processing file /etc/network/interfaces.d/interfaces.d ([Errno 21] Is a directory: '/etc/network/interfaces.d/interfaces.d') This warning only occurred on the node I have selected as an exit node in each zone.

The sdn file within the nested interfaces.d dir had an older version number and my previous configuration while the sdn file in the parent interfaces.d dir was what it should be running. The files also had different permissions:

interfaces.d/ (755)
.... sdn (644)
    interfaces.d/ (755)
    .... sdn (755)

Looks like the correct permissions for the sdn file is 755, based on my other nodes. So I copied the newer verison sdn file to the parent interfaces.d dir, changed the permissions and deleted the nested directory. After that, re-applied the sdn and just because, did an ifreload -a. I can now externally route again.

I'm ASSUMING, the nested directory is mimicking a "running config" at the root and "modified config" in the nested dir. And when the SDN configuration is applied, the modifid config is copied to the root level, deleting the nested dir. But somewhere along that save state it hung up, without an error....at least from what I can tell. This seems to be a one off case, but I have no idea why it occurred

 

[shanreich]

Do you have a task log from the task that regenerated the network configuration on that exit node and could post it here?

 

[ns33]

Do you have a task log from the task that regenerated the network configuration on that exit node and could post it here?

Maybe?

Is there a location where previous task logs are stored? The initial changed occurred about a day or so ago and the failed external routing I only noticed at the end of the workday yesterday. I'm the sole user of the cluster so I didn't notice the issue until I actually needed external systems

 

[shanreich]

You should be able to obtain them by clicking on the exit node in the left-hand resource tree, then go to task history and look for a task called 'SRV networking - Reload'. Double-click on it, then you should be able to view and download the task log. Make sure to check the output for information that you might not want to be publicly visible (e.g. public IPs).

 

[ns33]

This history in the docked window only goes back to this morning. I have since rebooted my whole cluster as that was my first...when in doubt fix.

On the exit node, I can cd to /var/log/pve/tasks which I'm guessing the storage place of these tasks. Just to confirm the correct file selection, I'm running find -name "*srvreload*" -type f -ls which is returning a list of those specific tasks from a handful of the numbered/lettered dirs. The earliest one is from Jul 29 16:03. Maybe that's when I made my first change. Regardless, if that's an okay way to obtain the task log, I can get that. It will just be a bit as I need to get approval to pull the file off the system