POM - Copy only the updated files to air-gapped network

[rfrIII]

Is it possible to only copy only an incremental update of the repo to the air-gapped network or does the entire repo have to be moved every time? Once I copy the full repo to an air-gapped network I would only want to copy an incremental update, say once a week, to the air-gapped network. Is this possible, from reading the docs and the forum I wasn't able to determine if that was possible. I saw the proxmox-offline-mirror medium diff option, but the docs are not too clear on what that command does.

This is an issue for us because what goes on the air-gapped network doesn't leave the air-gapped network, an incremental update of the repo would save tons of time and money.


Thank you,

Rob

 

[LnxBil]

I saw the proxmox-offline-mirror medium diff option, but the docs are not too clear on what that command does.

I don't know either, yet debmirror is another choice (for everything except the enterprise repository) and it analyzes was is currently on the disk and just downloads the new files. Those downloaded files can easily be rsynced to another location, which also just copies new files.

This is an issue for us because what goes on the air-gapped network doesn't leave the air-gapped network, an incremental update of the repo would save tons of time and money.

I don't get this, please explain further.

As I understand it, you sync the repository to a host accessible to the internet and then what? Copy to usb? how does this work.

 

[rfrIII]

Once media is accessed by our air-gapped network it stays on that network or is destroyed. It doesn't matter if it is a usb stick, an SSD, or a DVD/BR it is gone.

What I want to know is if there is a way to identify only the updated files with POM and only copy those new files to some media. That media would then be used to copy only the new files to the air-gapped network.

 

[rfrIII]

I can write a script to do this, but I was hoping this was a feature of POM. The Debian mirror team suggested I just bite the bullet and copy the whole repo but that is not an option. We have this working with YUM repos and a couple others, it is just a feature of the provided tools, but this does not seem to be an apt/dpkg feature. I'll admit the Debian ecosystem is new to me though; likely I am missing something.

 

[LnxBil]

What I want to know is if there is a way to identify only the updated files with POM and only copy those new files to some media.

That would not always work: Imagine PVE adds a new feature and the pve packages have a new dependency, that is not new, but old and not present on the PVE nodes. If you would only copy new files, the old packages that have been present since the release of Debian, are not available.

The Debian mirror team suggested I just bite the bullet and copy the whole repo but that is not an option.

I also want to know what changed and I also did not get anything besides parsing the logfile until I used ZFS for the datastore and did a zfs diff. It'll output a list of changes files - for Debian the repository files Packages*, Release* and the actual deb files. This may be exactly what you need. You could snapshot the zfs dataset each time you create a "transfer medium" so that you'll always have the actual difference.

 

[rfrIII]

It would work because all the previously copied files (including and initial full repo copy) would still be there.

POM does not seem to support this, we have a license and I can't get POM to do this so I will roll my own solution.

 

[guruevi]

Have you looked at aptly? It can create static snapshot and then diff those into a new snapshot.

Although I believe the real issue is that Debian signs repo indexes, so you can’t just have a partial set of packages in a repo without breaking that signature. My solution would be rdiff-backup or syncthing or any system that produces a date based increment system in separate directories and use that more generally to move files in that kind of organization.

Actually: think about using git - make the entire file structure as a git repo, then every time you need to move, commit, push the changes to your DVD: https://git-scm.com/book/id/v2/Git-Tools-Bundling#_bundling that would even allow you to rollback repos and add or not add certain packages if you know they are “bad” and don’t want to see them in your other side (provided you never actually use them, you will get an error).