API for managing apt repos seems incoherent

[mikeely]

Suppose I want to enable/disable the "test" repository via the API, looking here this should be or in fact is possible:
https://pve.proxmox.com/pve-docs/api-viewer/#/nodes/{node}/apt/repositories
Except:
Required params are index, node, path. Let's look at the output from `get /nodes/$(hostname -s)/apt/repositories` for a moment:

{
  "files": [
    {
      "path": "/etc/apt/sources.list",
      "repositories": [
        {...},
        {
          "description": "This is the recommended repository for testing and non-production use. Its packages are not as heavily tested and validated as the production ready enterp
rise repository. You don't need a subscription key to access this repository.",
          "handle": "no-subscription",
          "name": "No-Subscription",
          "status": 1
        },
        {
          "description": "This repository contains the latest packages and is primarily used for test labs and by developers to test new features.",
          "handle": "test",
          "name": "Test",
          "status": 1
        },
        {...}
      ]
    },
    ...
  ],
  "infos": [
    {"index": 0, "path": "/etc/apt/sources.list"},
    {"index": 1, "path": "/etc/apt/sources.list"},
    ...
  ]
}

The problem here is that there is no direct mapping between index, which is required, and the actual repository that's to be enabled/disabled. The best I could figure is to count the number of entries under files/repositories and then use that to get the index, using files/path and index/path as a join.
Am I missing something here or is this a design bug?

 

[guruevi]

Why do you have all your repos in a single file? Typically your PVE repos would be individually listed in sources.list.d and only the Debian repos are in sources.list. Then you can delete/rename individual files to disable/enable repo.

 

[mikeely]

Why do you have all your repos in a single file? Typically your PVE repos would be individually listed in sources.list.d and only the Debian repos are in sources.list. Then you can delete/rename individual files to disable/enable repo.

That's how they came when the system was built. In any case the API still needs to surface some kind of mapping between index and repo.

 

[guruevi]

But repos are a child of the files object. So when you know the file you have to modify, you can just iterate over the object starting at 0 and you know the associated index when you find a matching property. The reason this isn't easy is because apt repo file structure was not built to be machine parseable.

Although I'm assuming you simply want to program your desired state (eg. the way Ansible does it), in that case, write exactly how you want your configuration to look like, clean up everything that deviates, then you always know eg. the enterprise repo is in /etc/apt/sources.list.d/pve-enterprise.repo index 0.

This will be simplified with this, which you could technically use today: https://repolib.readthedocs.io/en/latest/deb822-format.html - not sure how the API handles it though.

 

[mikeely]

Sure, managing apt repos with ansible is easy - the API is still constructed wrong given that a large sources.list file is a very common pattern so the infos array needs to be populated with something more useful than just path.
Taking a step back here, what is the overall guidance on when to use standard admin tools (ansible, vim, etc) and when it's better to use the API? I'm thinking in particular about networking/sdn although the question is general to the whole host. I could run the Proxmox hosts just like any other Debian server but would that cause Proxmox to get into an inconsistent state? It's somewhat puzzling that there are a ton of useful ansible modules in community.general that are about proxmox but not a single one of them is about managing the proxmox servers themselves.

 

[guruevi]

My point about the APT repos is they do not have anything unique to index by:

What would you make the "index" of this repo for example (this is a valid repo in my own cluster) if not the zero-based numerical one which indicates the order, because there is an exact copy of this object with types=deb-src and no, this isn't an official Debian or a Proxmox repo, but this is "valid" APT repository config.

{
                    "Components": [
                        "main"
                    ],
                    "Enabled": 0,
                    "FileType": "list",
                    "Options": [
                        {
                            "Key": "signed-by",
                            "Values": [
                                "/etc/apt/keyrings//somerepo.asc"
                            ]
                        }
                    ],
                    "Suites": [
                        "bullseye"
                    ],
                    "Types": [
                        "deb"
                    ],
                    "URIs": [
                        "http://somerepo"
                    ]
                }

In my case, the filename for this repo is known (so it is somerepo.list) and then index 0 is the deb and index 1 is the deb-src.

Managing apt repos with Ansible is not 'simple', what Ansible does is simply generates a repo with a unique filename after what the admin "names" the repo, but you can easily get duplicate repo configurations. If you're going to be using the API, my assumption is that you would query the current repos, do some kind of match for your desired configuration (eg. I will add the repo in a specific file at a specific index, or a regexp match) and then your desired config will be there.

My assumption is that for a fresh install, you already have everything in the 'correct' state. The recommendation is thus not to keep adding things to sources.list, you should really use individual repo files.

 

[aabraham]

That's how they came when the system was built. In any case the API still needs to surface some kind of mapping between index and repo.

That's why you should split your repositories across different files. I suspect You are seeing the same file for different indexes precisely because you did not separate your repositories.

 

[mikeely]

There are several ways to manage repos with ansible, the simplest arguably being with the copy module. Returning to the API, I still insist that it's going at things backwards. Look at the design goals of the apt section of the Proxmox API (at least, what I can make of the design goals based on capability):
1. Parse and return any valid sources.list or sources.d/*.list entry
2. Create new, valid apt sources.
3. Enable and disable existing sources.

Best practices or no, Debian does support having multiple repos appearing in a single file and the API does parse those correctly or else the web frontend wouldn't show them, and one can use the web frontend to enable/disable them. This is entirely consistent with the design goals.

To meet the design goal of allowing API calls to enable/disable while he indexing should be built off the _contents_ of the repo files and then use the path variable as confirmation. Suppose someone has been a sloppy sysadmin, or is taking over after a sloppy sysadmin, and encounters a repo configuration like so:

# /etc/apt/sources.list
...
deb http://foo.example main foo

# /etc/apt/sources.list.d/foo.list
Enabled: yes
Types: deb
URIs: http://foo.example
Suites: foo
Components: main

Awesome, now we have the identical foo repo defined in two places and want to disable one of them via the API. To do so we need the following params:
1. Index
2. Node
3. Path
So we can assume we already know which node we're looking at, so now all we need is index and path, right? Problem is, they're stored separately in the API and are in no way anchored to each other. Here's what gets returned when you look:

{
  "files": [
    {
      "path": "/etc/apt/sources.list",
      "repositories": [
        {...},
        {
          "description": "This is the foo repository for doing foo things.",
          "handle": "foo",
          "name": "Foo",
          "status": 1
        },
        {...},
          "description": "This is the foo repository for doing foo things.",
          "handle": "foo",
          "name": "Foo",
          "status": 1
      ]
    },
    ...
  ],
  "infos": [
    ...
    {"index": 3, "path": "/etc/apt/sources.list"},
    ...
    {"index": 7, "path": "/etc/apt/sources.list.d/foo.list"},
    ...
  ]
}

Obviously this is pretty easy from here but what really stands out to me is that you have the "status" field in one array but the "index" field is in the other array. Wouldn't it be much easier to have index and status in the same array, seeing as how the only thing we can really modify is status anyhow? Or better yet, merge the contents of the "infos" array into "files" since it's a distinction without meaning.

 

[guruevi]

When you talk arrays you inherently have the index of said array as you iterate over it. I’m not sure what exactly infos is doing, I’m assuming it is a shortcut to validate something, I haven’t dug into the source code to see where it is used.