Proxmox Offline Mirror released!

[fabian]

sorry, kinda fell through the cracks.. I'll try to send the patch(es) this week!

 

[UweSauter]

Hi Fabian,

I think there hasn't been a release yet that include those patches, right? On my system, the currently installed proxmox-offline-mirror 0.6.5 was updated on 2024-02-22 according to my logs.
The symptoms that only parts of a repo gets mirrored still persist.

Regards,

Uwe

 

[fabian]

yes, we haven't cut a new release since.

 

[wond]

Hello team,

Quick question: I have POM running and I have done several snapshots for a specific mirror ex. debian_bullseye_main. So when I list the mirrors I get something like:

debian_bullseye_main (2 snapshots):
2024-04-03T08:XXXX
2024-04-02T09:XXXX

My question is: i would like to keep only one snapshot, the latest one. What happens if I remove the older one? Does the newer repository have any dependency on the older one which might cause it to break or each snapshot is completely independent?

Thank you!

 

[fabian]

the snapshots use hardlinks under the hood to only keep a single copy of each required file. removing older snapshots should not affect newer ones. you do need to call proxmox-offline-mirror mirror gc to run a garbage collection to actually remove files which are no longer referenced by any snapshot and free up the space.

but keep in mind - many repositories are append-only (exceptions like Debian testing or unstable excluded). so unless you also changed the filter settings, removing old snapshots usually doesn't free up much space

 

[wond]

the snapshots use hardlinks under the hood to only keep a single copy of each required file. removing older snapshots should not affect newer ones. you do need to call proxmox-offline-mirror mirror gc to run a garbage collection to actually remove files which are no longer referenced by any snapshot and free up the space.

but keep in mind - many repositories are append-only (exceptions like Debian testing or unstable excluded). so unless you also changed the filter settings, removing old snapshots usually doesn't free up much space

Ok Fabian, got it. Thanks a lot for the information.

Regards.

 

[wond]

Hello team, I am having an issue trying to sync the medium with mirrors.

The steps I did were:

1) Setup the mirrors
2) Create the snapshots
3) Syncing local mirrors to a medium (here I am having the error).

I used proxmox-offline-mirror-helper setup wizard to create the medium. I gave it a name of 100 and mount point to the location where I have all the local mirror files downloaded /mirror

When I try to sync it I get the following error:

proxmox-offline-mirror medium sync --config '/etc/proxmox-offline-mirror.cfg' '100'
will sync 0 subscription keys to medium
Syncing 8 mirrors ["debian_bullseye_main", "ceph_reef_bookworm", "debian_bookworm_security", "debian_bookworm_updates", "debian_bullseye_updates", "pve_bullseye_no_subscription", "pve_bookworm_no_subscription", "debian_bookworm_main"] to medium '100' ("/mirror")
Creating new statefile..
Sync timestamp: 2024-04-23T10:17:00Z
Previously synced mirrors: {}
Adding 8 new mirror(s) to target medium: {"pve_bullseye_no_subscription", "pve_bookworm_no_subscription", "ceph_reef_bookworm", "debian_bookworm_security", "debian_bullseye_main", "debian_bookworm_updates", "debian_bullseye_updates", "debian_bookworm_main"}

Starting sync now!

Syncing 'debian_bullseye_main' to "/mirror/debian_bullseye_main"..
Error: Pool dir "/mirror/.pool_cdf7f25e272cc8bddeffca7b6571f3b0c24a73d414fee59fd8167a3d35d1a5d1" doesn't exist.
This is the structure of /mirror directory
drwxr-xr-x 12 root root 4.0K Apr 23 09:40 .
drwxr-xr-x 19 root root 4.0K Apr 2 12:43 ..
drwxr-xr-x 3 root root 4.0K Apr 23 08:56 ceph_reef_bookworm
drwxr-xr-x 8 root root 4.0K Apr 23 08:57 debian_bookworm_main
drwxr-xr-x 3 root root 4.0K Apr 16 09:31 debian_bookworm_security
drwxr-xr-x 3 root root 4.0K Apr 16 09:31 debian_bookworm_updates
drwxr-xr-x 5 root root 4.0K Apr 23 08:57 debian_bullseye_main
drwxr-xr-x 5 root root 4.0K Apr 23 09:53 debian_bullseye_security
drwxr-xr-x 3 root root 4.0K Apr 23 08:58 debian_bullseye_updates
-rw-r--r-- 1 root root 0 Apr 23 09:40 .mirror-state.lock
drwxr-xr-x 4 root root 4.0K Apr 2 13:22 .pool
drwxr-xr-x 3 root root 4.0K Apr 16 09:31 pve_bookworm_no_subscription
drwxr-xr-x 3 root root 4.0K Apr 23 08:58 pve_bullseye_no_subscription


It`s looking for a file which actually doesnt exist in the .pool folder.

Next structure of .pool folder I have:

drwxr-xr-x 4 root root 4.0K Apr 2 13:22 .
drwxr-xr-x 12 root root 4.0K Apr 23 09:40 ..
-rwxr-xr-x 1 root root 0 Apr 2 13:22 .lock
drwxr-xr-x 2 root root 12M Apr 23 09:53 sha256
drwxr-xr-x 2 root root 704K Apr 23 09:44 sha512


Any ideas on how to fix the issue?

I appreciate for the support!
Thank you!!

 

[fabian]

please post the full config..

 

[wond]

please post the full config..

Hi Fabian, I managed to fix it. So I had to change the mountpoint from /mirror to /mirror/.pool/ in the /etc/proxmox-offline-mirror.cfg. Not sure why this was looking for wrong directory.

 

[wond]

Hi team, I have another question please:

I have a central POM server already configured with mirrors/medias and it`s configured with local HTTP server as medium. I am pointing the "client servers" via /etc/apt/sources.list and it`s working fine. The clients can access the mirrors and install packages.

Now the question comes when I try to use proxmox-offline-mirror-helper setup to deploy offline keys ( I have already deployed the subscription key to the POM server). On the client server, it`s asking me Path to medium mountpoint: what should I pass here? My POM server is running under https://pomserver.domain.example/. I have tried this address in different ways but it wont work saying that this mount point doesn't exist.

What am I doing wrong here? Is the offline activation key possible via HTTPS as mount point?

Thanks!

 

[fabian]

you need to provide the helper binary access to the mountpoint of the medium as a path at the moment (it can be a mountpoint of some kind though, like NFS). we do have plans to allow an API based access to the POM instance at some point, which would allow fetching the offline subscription data over the internal network as well, for systems that are no air-gapped completely.

 

[wond]

you need to provide the helper binary access to the mountpoint of the medium as a path at the moment (it can be a mountpoint of some kind though, like NFS). we do have plans to allow an API based access to the POM instance at some point, which would allow fetching the offline subscription data over the internal network as well, for systems that are no air-gapped completely.

thanks Fabian, understood. So in the end what I am trying to do is to have 3 clients servers to have their subscription activated from my central POM server. I bought 4 subscription licenses: 1 for the POM itself and 3 for the client nodes.

The one license for the POM I managed to activate easily because POM has access to the internet so I used proxmox-offline-mirror setup.

Now I need to get the activation on the 3 client servers. I will create a NFS mount point on the POM as your recommendation and use it as mount point from the client servers using proxmox-offline-mirror-helper.

My question is: how do I tell POM I have 3 more licenses for client servers? and how does POM "send" this subscriptions keys to the client servers? Is it via the mirror?

Thank you!

 

[liptech]

We are proud to announce the first release of our new Proxmox Offline Mirror tool.

With the Proxmox Offline Mirror tool, you can manage a local apt mirror for all package updates for Proxmox and Debian projects. From this local apt mirror you can create an external medium, for example a USB flash drive or a local network share, to update systems which cannot access the package repositories directly (or proxied) via the internet. Such systems might be restricted by policies to access the public internet or are completely air-gapped. Finally, you can also manage subscriptions for such restricted hosts.

Documentation
https://pom.proxmox.com

Community Forum
https://forum.proxmox.com

Source Code
https://git.proxmox.com

Bugtracker
https://bugzilla.proxmox.com

FAQ
Q: How can I install Proxmox Offline Mirror?
A: See the installation chapter in the reference documentation.

Q: Do I need a subscription to mirror APT repositories with Proxmox Offline Mirror?
A: No, you do not need a subscription for offline mirroring of APT repositories.

Q: Do I require a subscription to activate subscription keys offline with Proxmox Offline Mirror?
A: Yes, you require a special subscription to activate a subscription key of a Proxmox solution offline. Standard and Premium subscriptions include one for free.

Q: What types of repositories does Proxmox Offline Mirror support?
A: All APT-based repositories should work in theory, but we only test those of Proxmox and Debian projects.

Q: How is offline repository mirroring integrated with Proxmox VE, Proxmox Backup Server or Proxmox Mail Gateway?
A: Offline mirrors can be accessed like ordinary APT repositories. The proxmox-offline-mirror-helper utility facilitates the setup.

Q: What version of Proxmox VE, Proxmox Backup Server or Proxmox Mail Gateway supports offline subscription key activation?
A: See the docs for the minimum Package versions required for managing offline subscriptions.

Hello everybody

I'm trying Proxmox Offline Mirror and I have some doubts.

My main objective is to create a mirror in the company where I work, to speed up the updating of production and approval servers.

I managed to configure the mirrors and take their snapshots, what I still don't understand is how to publish, as it apparently doesn't generate a publication folder, much less a command to create it, it keeps the snapshot in a folder with the date it was taken. (2024-04-19T11:19:01Z) where they are maintained and updated with hardlinks.

This would give me the trouble of updating the repository folder settings manually or by script.

I would like to know if I am right in my analysis.

 

[fabian]

My question is: how do I tell POM I have 3 more licenses for client servers? and how does POM "send" this subscriptions keys to the client servers? Is it via the mirror?

see https://pom.proxmox.com/offline-keys.html#register-refresh-keys and https://pom.proxmox.com/offline-media.html#activating-an-subscription-key

basically, you give POM the server IDs and subscription keys for the offline systems. it will then validate them (since it has network access), and store the validated copy on a medium. if you then give the offline system access to the medium, the helper binary can find the pre-validated subscription info there and add it to the system.

 

[fabian]

Hello everybody

I'm trying Proxmox Offline Mirror and I have some doubts.

My main objective is to create a mirror in the company where I work, to speed up the updating of production and approval servers.

I managed to configure the mirrors and take their snapshots, what I still don't understand is how to publish, as it apparently doesn't generate a publication folder, much less a command to create it, it keeps the snapshot in a folder with the date it was taken. (2024-04-19T11:19:01Z) where they are maintained and updated with hardlinks.

This would give me the trouble of updating the repository folder settings manually or by script.

I would like to know if I am right in my analysis.

https://bugzilla.proxmox.com/show_bug.cgi?id=4379

you can just have your own symlink (outside of the POM managed directory for now) that you move whenever you make a new snapshot, until that gets implemented as a built-in feature.

note that the main use case for POM is not as an APT proxy (there are other solutions for that), but for updating air-gapped or heavily restricted installations (that usually don't get very frequent updates either).

 

[UweSauter]

note that the main use case for POM is not as an APT proxy (there are other solutions for that), but for updating air-gapped or heavily restricted installations (that usually don't get very frequent updates either).

But it works very well as an generic APT mirror solution, at least since update v0.6.6. And it is convenient to set up compared to other solutions.

I have a script that is executed as systemd timer which iterates over all configured mirrors and updates the "latest" symlink if creating the snapshot was successful.

One point I'd like to mention is that I still see a difference whether a system uses official Debian / Proxmox repositories in contrast to a mirror create using the exact same repository. Somehow metadata is lost on the way because security updates are not presented as such, e.g. when using aptitude. But that is a minor inconvenience.

 

[fabian]

One point I'd like to mention is that I still see a difference whether a system uses official Debian / Proxmox repositories in contrast to a mirror create using the exact same repository. Somehow metadata is lost on the way because security updates are not presented as such, e.g. when using aptitude. But that is a minor inconvenience.

the metadata that is mirrored, for example in the Release and Packages files, is unchanged (although some optional parts are not mirrored at all, like the pdiff files IIRC). I suspect aptitude might look at the URL of the repo, which is of course different when using a mirror..

the main difference between a proxy and a mirroring tool like POM is that the proxy only fetches and caches what is requested, and doesn't fetch pro-actively. of course you can use a full-fledged mirror as a stand-in for a proxy, if you update it often enough, but in many cases, it is overkill

 

[UweSauter]

of course you can use a full-fledged mirror as a stand-in for a proxy, if you update it often enough, but in many cases, it is overkill

You could also consider using an internal mirror as being a good netizen who saves bandwidth of the remote mirrors, especially if the number of servers that is used internally reaches a certain number

 

[fabian]

a proxy has the same effect though (as long as you configure its cache to have a reasonable size) but yes, like I said - both are valid solutions

 

[wond]

Hello everybody

I'm trying Proxmox Offline Mirror and I have some doubts.

My main objective is to create a mirror in the company where I work, to speed up the updating of production and approval servers.

I managed to configure the mirrors and take their snapshots, what I still don't understand is how to publish, as it apparently doesn't generate a publication folder, much less a command to create it, it keeps the snapshot in a folder with the date it was taken. (2024-04-19T11:19:01Z) where they are maintained and updated with hardlinks.

This would give me the trouble of updating the repository folder settings manually or by script.

I would like to know if I am right in my analysis.

Quick solution (maybe not the best).. create a soft link called latest which reads the latest snapshot file: used command is: ln -s ./`ls -rt | tail -n1` latest . You can create a bash routine/script which runs from time to time to validate if a new snapshot file appeared.