Proxmox Offline Mirror released!

[t.lamprecht]

We are proud to announce the first release of our new Proxmox Offline Mirror tool.

With the Proxmox Offline Mirror tool, you can manage a local apt mirror for all package updates for Proxmox and Debian projects. From this local apt mirror you can create an external medium, for example a USB flash drive or a local network share, to update systems which cannot access the package repositories directly (or proxied) via the internet. Such systems might be restricted by policies to access the public internet or are completely air-gapped. Finally, you can also manage subscriptions for such restricted hosts.

Documentation
https://pom.proxmox.com

Community Forum
https://forum.proxmox.com

Source Code
https://git.proxmox.com

Bugtracker
https://bugzilla.proxmox.com

FAQ
Q: How can I install Proxmox Offline Mirror?
A: See the installation chapter in the reference documentation.

Q: Do I need a subscription to mirror APT repositories with Proxmox Offline Mirror?
A: No, you do not need a subscription for offline mirroring of APT repositories.

Q: Do I require a subscription to activate subscription keys offline with Proxmox Offline Mirror?
A: Yes, you require a special subscription to activate a subscription key of a Proxmox solution offline. Standard and Premium subscriptions include one for free.

Q: What types of repositories does Proxmox Offline Mirror support?
A: All APT-based repositories should work in theory, but we only test those of Proxmox and Debian projects.

Q: How is offline repository mirroring integrated with Proxmox VE, Proxmox Backup Server or Proxmox Mail Gateway?
A: Offline mirrors can be accessed like ordinary APT repositories. The proxmox-offline-mirror-helper utility facilitates the setup.

Q: What version of Proxmox VE, Proxmox Backup Server or Proxmox Mail Gateway supports offline subscription key activation?
A: See the docs for the minimum Package versions required for managing offline subscriptions.

 

[Deepen Dhulla]

We would Checkout , Great to see Offline Updates for Customer who are looking for managing air-gap updates.

 

[mishki]

Great!

if i mount repo to windows machine i get error due to unsupported characters ":"

 

[t.lamprecht]

Is your local network share on windows? if not, what would you want to want to do with the mirror on Windows (out of interest)?

Also what filesystem are you using? FYI, the hard requirements from the docs list hardlinks as requirement for the deduplication pool layer, which excludes most Windows FS (so we did not bother to test with them at all) :
https://pom.proxmox.com/installation.html#system-requirements

I mean I knew that Windows is rather limited, but not supporting colons, or in fact any of <>:"/\|?*, in file names is really stretching it...

In general I'd not be opposed to add a opt-in mode that encodes the file names differently, but as said, hardlinks are still required, and if the systems not support colon in names overlap with their FS not being able to do hard links I see no point in doing so.
Until then I'd recommend using a Linux, BSD or another OS that can handle basic stuff like a colon in filenames and also hardlinks, as documented.

 

[mishki]

Is your local network share on windows? if not, what would you want to want to do with the mirror on Windows (out of interest)?

No, no. Windows Machine is like a link between me and a closed circuit with servers on PVE.
I downloaded Proxmox Offline Mirror to my nextcloud server (linux) and mounted it via webDAV to this Windows Machine.
After that, I copy them further via winscp to linux machines.

this is the most, and possibly the only way to transfer such a volume of data into a closed loop.

Is your local network share on windows? if not, what would you want to want to do with the mirror on Windows (out of interest)?

Also what filesystem are you using? FYI, the hard requirements from the docs list hardlinks as requirement for the deduplication pool layer, which excludes most Windows FS (so we did not bother to test with them at all) :
https://pom.proxmox.com/installation.html#system-requirements

Thanks for the info.
In general, my case is pretty rare and I should just use archiving instead of throwing files in folders as is.

 

[PanZvedavy]

Are there any plans to incorporate this feature into proxmox web interface? It would by a great addition to have something like linux WSUS for its hosts and virtuals. For standartisation and wider comunnity akceptance, this would need to have graphic administration.

 

[t.lamprecht]

Are there any plans to incorporate this feature into proxmox web interface?

The offline repo acts like any other apt repo, once it got set up you can already upgrade a host simply via the Proxmox VE, Proxmox Mail Gateway and Proxmox Backup Server's respective web interface.

It would by a great addition to have something like linux WSUS for its hosts and virtuals

POM got nothing specific to do with virtual guests, while you can also mirror for any VM or CT that gets updated via apt repos, we do not plan to integrate any such automated handling of something like that in the (foreseeable near) future, that would require to install further software in the guest, partially replacing, or at least puppeteering the package manager there, which is pretty finicky, especially on bigger updates and better done by the admin.

For standartisation and wider comunnity akceptance, this would need to have graphic administration.

What do you mean with standardization, as I do not see how some sort of additional GUI would help with that in my understanding of the word?
Running air gapped systems normally requires some advanced understanding of the system, to ensure its actually setup correctly in the limitations and protocols of an environment, and POM isn't that complex to set up, and especially once that initial setup is done the update work itself is something relatively simple but hard to improve with some GUI.

 

[PanZvedavy]

The offline repo acts like any other apt repo, once it got set up you can already upgrade a host simply via the Proxmox VE, Proxmox Mail Gateway and Proxmox Backup Server's respective web interface.

POM got nothing specific to do with virtual guests, while you can also mirror for any VM or CT that gets updated via apt repos, we do not plan to integrate any such automated handling of something like that in the (foreseeable near) future, that would require to install further software in the guest, partially replacing, or at least puppeteering the package manager there, which is pretty finicky, especially on bigger updates and better done by the admin.

What do you mean with standardization, as I do not see how some sort of additional GUI would help with that in my understanding of the word?
Running air gapped systems normally requires some advanced understanding of the system, to ensure its actually setup correctly in the limitations and protocols of an environment, and POM isn't that complex to set up, and especially once that initial setup is done the update work itself is something relatively simple but hard to improve with some GUI.

Well, I get you point with fidling with updates inside of vms. My primary idea was with web interface for POM having a simple one page, somewhere in administration section as submenu, with ability to manage repos..adding them, delete them, manage snapshots etc. Maybe after some time you will add support for rpm repos. I didnt ask for creating something like redhat satelite update system...just some hooks to qemu agent wich will communicate to proxmox host and offer new packages from POM(by simply calling apt update && apt upgrade inside vms) and let the OS inside vms do the rest. Its just idea about possibilities wich POM can bring.

 

[t.lamprecht]

This is rather out of scope of Proxmox Offline Mirror as it'd be completely independent of that, as such a solution wouldn't care if the updates came in over some local POM managed medium, or some other source; so this thread is the wrong one to discuss such a feature. That said, we do not plan to add some (general) guest update service in the foreseeable future.

As additional and last note for that off-topic thematic: For Debian and derivatives I'd rather recommend setting up some local nginx in the air gapped network partition/setup, configure that as repo inside guests and setup unattended/automated upgrades in said Debian based guests. That way, one would just need to periodically update the POM snapshot - which needs some manual action anyway (as otherwise the host isn't really air gapped, and doesn't need POM in the first place) - and be done, all else would happen automatically (less work and actually scalable if more than a handful of guests). See https://wiki.debian.org/UnattendedUpgrades

 

[alexskysilk]

Thanks Thomas! This is fantastic for normal size repos such as proxmox, but the full debian_bullseye mirror is over 500GB for just amd64; assuming this is for proxmox update consumption, about 90% of that mirror is of no use; do we have to mirror an entire dataset or is there/will there be something similar to debmirror?

 

[fabian]

Thanks Thomas! This is fantastic for normal size repos such as proxmox, but the full debian_bullseye mirror is over 500GB for just amd64; assuming this is for proxmox update consumption, about 90% of that mirror is of no use; do we have to mirror an entire dataset or is there/will there be something similar to debmirror?

a full mirror of bullseye (excluding backports, with a few snapshots of the main repository, -updates and -security each) is:

du -sh --apparent-size /mirror/debian/
105G    /mirror/debian/

did you maybe forget to select only architectures all/amd64? or used a different base directory for updates and security (these two share a lot of packages which are then not deduplicated!)?

that being said, a feature to exclude sections and packages from being mirrored is on the todo list, which should allow avoiding most of the unnecessary "big" things (game data, Debian kernel packages, ..).

 

[Machine Manager]

This is really good news! Thank you!

My only concern is how much disk space is going to be wasted with many packages that won't be used.
Is it possible to only download the specific packages that are going to be used by the machines? Maybe something similar to `apt-offline`?

 

[t.lamprecht]

My only concern is how much disk space is going to be wasted with many packages that won't be used.

It's a few tens of GB, not nothing but nowadays really not that much, I mean even SDcards come now with 1TB+
More serious, a 240 GB Crucial BX500 SSD is 25€ here and that includes a VAT of 20% (so ~20€ = ~ 20$ without VAT) - throw in an extra 15 bucks and you get the 500 GB model, which will be enough for mirroring three major Debian releases plus all the respective Proxmox repos at the same time..

IMO if anything is to worry its more time/traffic, but that is mostly "paid" only once on first sync, successive ones will only pull the normally quite small delta since the last sync.

Is it possible to only download the specific packages that are going to be used by the machines? Maybe something similar to `apt-offline`?

See Fabians previous reply:

that being said, a feature to exclude sections and packages from being mirrored is on the todo list, which should allow avoiding most of the unnecessary "big" things (game data, Debian kernel packages, ..).

With that the biggest chunk of the for most Proxmox VE hosts unused packages is filtered.

 

[DerDanilo]

@t.lamprecht @tom
May I suggest to add an automatic updated snapshot link that points to the latest snapshot? This way it's easier to use POM via network without the requirement to manually having to change either the nginx link/root or the apt source configuration for the clients using the mirror. This may also make updating media easier while being able to use a predefined ref to the latest snapshot.
Creating such a symlink might be possible if it was possible to define a custom snapshot name. I couldn't find information about this in the docs and cli.

I am also missing a command to clean up all but one snapshot (e.g. latest or specified). I just had to trigger a 'snapshot' multiple times since there were some issues while downloading packages from the upstream repo.

While testing POM I found also that it seems to cancel downloading further packages if there is any kind of interruption. As mentioned before one then has to trigger a new snapshot to have POM check all previously downloaded just to see POM fail again at some other point, having to create yet another snapshot. Maybe add an option to retry/update an existing snapshot?! This would also be a required feature to use POM as the proper apt-mirror replacement.

I know that this kind of was asked before but could POM be somewhat adjusted to be a capable replacement for 'apt-mirror' which isn't properly maintained anymore? Right now POM can kind of do that but one needs to write e.g. nginx/apache config to provide the proper package path that systems expect while not completely having to rewrite the sytax of the apt sources config. (yes, one could just configure the client apt sources but that kind of complicated things by a lot.
Hence it would be awesome if POM could provide all featues/settings to become an apt mirror without the need having to write a lot of other config files that might as well break during some updates.

As workarround I am thinking of creating symlinks in the webroot that mirror the layout of the default debian repos while pointing at the 'latest' snapshot of each repos mirror config. But this isn't possible without a reference to the 'latest' snapshot.

But maybe POM could alos provide a setting to expose the repos in a debian style default repo layout while keeping them in their base-dir.

Thanks in advance.

 

[fabian]

git already has an "ignore-errors" option that will continue if fetching a package file fails - there's a few more changes in the pipeline before the next release will be cut (among them, filtering by section/package name glob and proper deb-src support).

exposing a "latest" snapshot would be easy code-wise - although with the footgun that different mirrors would potentially point to totally different, possibly incompatible points in time with their "latest" snapshot.

with your example and exposing the whole thing via NGINX:
- PVE repo snapshot is created, "latest" updated
- client gets repo info
- debian bullseye repo snapshot is created, "latest" updated

could happen - and while it shouldn't cause any problems, it might be better to be safe than sorry and do the following (e.g. in a script/timer unit service):
- trigger snapshot creation
- wait for snapshots to be created
- list snapshots, update latest symlink

but I guess as long as the (lack of ) guarantees are documented automatically creating such a latest symlink automatically doesn't really hurt.

I agree that some sort of snapshot "pruning" similar to PBS would also be nice, but it's trivially do-able on your own for now (especially the "only keep last snapshot part). note that for most repositories, there's almost no practical difference between keeping only the last or keeping multiple snapshots except for more hardlinks, which barely cost any space

 

[jdancer]

Just created a Proxmox Offline Mirror instance.

I've noticed the setup wizard for creating a Ceph mirror does not include an option to mirror the Quincy release of Ceph.

How can I manually create it?

 

[t.lamprecht]

I've noticed the setup wizard for creating a Ceph mirror does not include an option to mirror the Quincy release of Ceph.

Thanks for the report, now fixed in git.

How can I manually create it?

See the docs. For Ceph Quincy that invocation would look something like:

proxmox-offline-mirror config mirror add \
 --id ceph-quincy \
 --architectures amd64 \
 --architectures all \
 --repository 'deb http://download.proxmox.com/debian/ceph-quincy bullseye main' \
 --key-path /etc/apt/trusted.gpg.d/proxmox-release-bullseye.gpg \
 --sync false \
 --verify true \
 --base-dir /path/to/mirror/base-dir

You basically only need to adapt the base-dir and maybe sync/verify behavior, if you want.

 

[t.lamprecht]

FYI, we just bumped a new version of the Proxmox Offline Mirror stack with quite a few new features and fixes:

  * fix #4259: mirror: add 'ignore-errors' option to make fetching errors from
    (technically) broken repositories non-fatal.

  * mirror: collect and summarize warnings to ensure that they ain't missed in
    the rather verbose output

  * pool: add a 'diff' command for snapshots and for mediums, to ease
    comparing the actual changes

  * mirror snapshot list: make the <id> parameter optional so that one can
    quickly list all snapshots from all mirrors.

  * mirror: add option to configure filtering one or more packages via GLOBs
    and also specific Debian packaging sections like 'games', 'kernel' or
    'debug' to reduce the pool size

  * mirror: implement source packages mirroring

  * fix #4264: only require either Release or InRelease as even though that
    InRelease is strictly speaking required, some repos (especially older
    ones) may not have it.

  * guided mirror setup: propose to filter out the 'games' and 'debug'
    sections by default, which, for example, reduces a mirror of PVE + Debian
    bullseye by about 27% (105GB->77GB).

  * docs: mention the new section and package filters

  * helper: add status command for showing information that is stored on the
    medium itself

  * guided setup: add Quincy as supported release for bullseye

At time of writing this, it is available on all test repos as version 0.5.0-1.

 

[DerDanilo]

git already has an "ignore-errors" option that will continue if fetching a package file fails - there's a few more changes in the pipeline before the next release will be cut (among them, filtering by section/package name glob and proper deb-src support).

exposing a "latest" snapshot would be easy code-wise - although with the footgun that different mirrors would potentially point to totally different, possibly incompatible points in time with their "latest" snapshot.

with your example and exposing the whole thing via NGINX:
- PVE repo snapshot is created, "latest" updated
- client gets repo info
- debian bullseye repo snapshot is created, "latest" updated

could happen - and while it shouldn't cause any problems, it might be better to be safe than sorry and do the following (e.g. in a script/timer unit service):
- trigger snapshot creation
- wait for snapshots to be created
- list snapshots, update latest symlink

but I guess as long as the (lack of ) guarantees are documented automatically creating such a latest symlink automatically doesn't really hurt.

I agree that some sort of snapshot "pruning" similar to PBS would also be nice, but it's trivially do-able on your own for now (especially the "only keep last snapshot part). note that for most repositories, there's almost no practical difference between keeping only the last or keeping multiple snapshots except for more hardlinks, which barely cost any space

Thanks for your reply. For now POM is not suitable to replace a classic APT mirror, which I now setup for a project. I hope it will be in future when it adds enough features to be an easy to configure netwerk mirror with build in web server for the files to download from. That should be fairly simple to implement once a "latest" snapshot is also build in. There could even be an option which each repo to enable that repo via netwerk etc.

There is much more potential for POM I think. But the Proxmox Team also needs to have the same idea of what POM can become, since you guys put most of the effort into developing it.

Thanks!

 

[t.lamprecht]

It's trivial to set up creating latest snapshot links and http export already if you want it though, so those two small things really should not make pom "unsuitable".

Setup nginx or caddy once, or even just run python3 -m http.server -d /path/to/pom/mirror/base 80 would be fine.
As it's just serving static files in either way, it can be done in maximal ~5 minutes, just change the web root it serves to the base-path of what you're using in POM.

Then, for your periodic script, you only need to cover link generation, which is rather trivial too, we can use the fact that ISO 8601 date/times are already ordered, so a simple ls -1 2* | tail -1 gives you the latest snapshot.

I actually wanted to see how long it takes me to do that and wrote the following small bash script you can use:

#!/bin/bash
set -e # exit on error

BASE_PATH="/mnt/offline-mirror" # <- adapt me

# only need to adapt below if not using the default /etc/proxmox-offline-mirror.cfg one
POM_CONFIG="/etc/proxmox-offline-mirror.cfg"

# trigger snapshot creation for all mirrors
proxmox-offline-mirror mirror snapshot create-all --config "$POM_CONFIG"

echo "mirroring done, creating link to most recent snapshot"
for f in "$BASE_PATH"/*; do
    [ -d "$f" ] || continue # loop over all directories to get all repos
    cd "$f"
    most_recent="$(find . -maxdepth 1 -name '2*' -printf '%f\n'  | sort | tail -1)"
    ln -Tsf "$most_recent" latest  # create link to latest snapshot
    echo "created 'latest' link for snapshot '$most_recent' of repo '$(basename "$f")'"
done

Took me about 10 minutes to write inclusive testing, with most part of that going into waiting until POM finished a snapshot round on my slow spinning-rust disk, and the dumb error of forgetting the -T flag on linkage.

There is much more potential for POM I think.

There's also much more potential in combining existing tools, one just needs to leverage them. The UNIX philosophy of "do one job but do it right" has its reasons and benefits, not everything needs to become a http exposing, email processing almost-mini-OS monster. I' mean sure, some things can be still nice for convenience, but a lot of them just can be easily workarounded with a simple script or the like.