Proxmox Offline Mirror released!

fabian said:
you can filter by architecture, "section" (part of the package metadata) and package name.
Click to expand...
There is no mention of a way to filter out the X11 related packages, there is no division along that line, as far as any documentation I have found is concerned.

fabian said:
well, all Proxmox software is Debian based. you might be able to get them to run on other distros as well, but it's not something that is officially supported.
Click to expand...
And Ubuntu is Debian based, it uses the same package manager (even if it is with different repo's, because Ubuntu has way longer support) so specifically with something that is build on that, I would expect it would work.
Besides, there is an official explanation from Proxmox, which does imply compatibility, but that is precisely the one that is incorrect and does not work.
 
emunt6 said:
Hi!

Is there any difference using POM instead of "apt-mirror" ?
Click to expand...
At least the repositories, and as far as I got, the location of the mirror is fixed with apt-mirror.

But you might be able to get the right repo's through apt-mirror.
I have not tried yet, but if I have more use for the POM, or specifically want more than just a Proxmox mirror (like for Ubuntu), I will test this and report the results.

Please let me know if you have tried before I have.
 
GHvM said:
There is no mention of a way to filter out the X11 related packages, there is no division along that line, as far as any documentation I have found is concerned.
Click to expand...

There's a x11 section, which e.g., the x11-apps package is listed as (check apt show x11-apps).
Or got a few examples for what packages you actually mean?

GHvM said:
because Ubuntu has way longer support
Click to expand...
FYI There's a Debian LTS project provides 5 years of support, and the Extended LTS project can extend it to up to 10 years, and that Debian provides seamless upgrades between major releases, making such LTS releases somewhat obsolete anyway.
GHvM said:
And Ubuntu is Debian based, it uses the same package manager (even if it is with different repo's, because Ubuntu has way longer support) so specifically with something that is build on that, I would expect it would work.
Click to expand...
The package manager has not really any say it what software runs and what not, it's just there for dependency resolution and metadata, it cannot magically create binary compatibility between different distros, or even different releases from the same distro.
Each distro release, be it from Debian or from Ubuntu is only guaranteed to be compatible with software that targets it, ideally build for exactly that release, with the libc version being the major factor most of the time.
I.e., you won't be able to use most .deb packages files from a modern Ubuntu 23.10 directly on a say, Ubuntu 22.04, even if they both use the same package manager.
GHvM said:
Besides, there is an official explanation from Proxmox, which does imply compatibility, but that is precisely the one that is incorrect and does not work.
Click to expand...
Why should a non-existing explanation or statement imply compatibility? By that logic POM would be also compatible with macOS and Windows DOS, we never state the contrary after all?

Our official builds target Debian, with the newest series targetting Debian Bookworm. If you got an Ubuntu that has the same libraries in compatible SO versions available it will run, just like it would on Fedora or ArchLinux in such a case if libraries are installed and compatible – sure the latter two might need manually extracting the .deb file as yes, their native package manager uses a different format, but that's about it.
 
t.lamprecht said:
There's a x11 section, which e.g., the x11-apps package is listed as (check apt show x11-apps).
Or got a few examples for what packages you actually mean?
Click to expand...
I have not found a distinct division like you say there is.
Unfortunately I can not check anything right now, but as far as the documentation that is provided it does not work like that.
t.lamprecht said:
FYI There's a Debian LTS project provides 5 years of support, and the Extended LTS project can extend it to up to 10 years, and that Debian provides seamless upgrades between major releases, making such LTS releases somewhat obsolete anyway.
Click to expand...
Debian may have the LTS version now, but it didn't when it mattered, and now the way things are setup by default is outdated.

As best practice, it is best to reevaluate a system's install (at least) every five years anyway, but the standard lifetime back then was not enough to be economically and maintenance-wise feasible.
The times have changed, and in all that time, I have seen Debian change the wrong way around and missing all the points that matter to me.
t.lamprecht said:
The package manager has not really any say it what software runs and what not, it's just there for dependency resolution and metadata, it cannot magically create binary compatibility between different distros, or even releases from the same distro.
Each distro release, be it from Debian or from Ubuntu is only guaranteed to be compatible with software that targets it, ideally build for exactly that release, with the libc version being the major factor most of the time.
I.e., you won't be able to use most .deb packages files from a modern Ubuntu 23.10 directly on a say, Ubuntu 22.04, even if they both use the same package manager.
Click to expand...
I am not talking about different versions of the same distro, I am talking about different flavours of the same version.
It might be that what's necessary for Proxmox has a different version id between Debian and Ubuntu, but that should not be a problem itself (if it is, someone is not doing the job they have taken on).
t.lamprecht said:
Why should a non-existing explanation or statement imply compatibility? By that POM would be also compatible with macOS and Windows DOS, we never state the contrary after all?
Click to expand...
The implication is not even subtle: it is stated literally in https://pom.proxmox.com/proxmox-offline-mirror.pdf that it should work in Ubuntu.
I see no mention of macOS or Windows/DOS in the same context.
t.lamprecht said:
Our official builds target Debian, with the newest series targetting Debian Bookworm. If you got an Ubuntu that has the same libraries in compatible SO versions available it will run, just like it would on Fedora or ArchLinux in such a case if libraries are installed and compatible – sure the latter two might need manually extracting the .deb file as yes, their native package manager uses a different format, but that's about it.
Click to expand...
As far as I have seen, Bookworm is not supported yet with POM, but it might have been changed in the last week or so.
I have not checked every single library, but as far as I could determine, it should have worked, but it didn't.

It seems you are missing most of my points about it, and interpret my objections exactly the wrong way.
For me this matter is closed, because there is no solution besides the Debian VM I have already setup and used.
 
GHvM said:
I have not found a distinct division like you say there is.
Click to expand...
I literally quoted an example and showed you how you can find out about which section a package is, which you can use for filtering, as Fabian mentioned.
GHvM said:
but as far as the documentation that is provided it does not work like that.
Click to expand...
Which documentation? What did you actually try? If you start posting specifics, people might be actually able to help you...
GHvM said:
Debian may have the LTS version now, but it didn't when it mattered, and now the way things are setup by default is outdated.
Click to expand...
Even the linked wiki page mentions that there was an LTS for Debian Squeeze, which was released in 2011, over a decade ago.
There's no Ubuntu release from back then that's still supported, so it definitively had one when you setup the last (still supported) Ubuntu release..
GHvM said:
I am talking about different flavours of the same version.
Click to expand...
What different flavors? Debian and Ubuntu are not just two different flavors, but rather two completely separate projects, that are having their own ABI versions and specialities. Compatibility for executables between Debian and Ubuntu might be only slightly better than between any random other Linux distro..
GHvM said:
It might be that what's necessary for Proxmox has a different version id between Debian and Ubuntu, but that should not be a problem itself (if it is, someone is not doing the job they have taken on).
Click to expand...
What? So please tell me how you magically keep up binary-ABI/soname compat between those two?

Some builds of POM indeed work on some Ubuntu versions, as long as the libraries of that Ubuntu release are neither to old nor to new, thus "should" work. But yeah, as there seems that some users do not know that Debian and Ubuntu are two different projects with their own decisions how to do things, impacting compat similar as between other, totally unrelated, distros, it may indeed be good to add some more context there.
GHvM said:
As far as I have seen, Bookworm is not supported yet with POM, but it might have been changed in the last week or so.
Click to expand...
There sure is, and that since over two months (i.e., before our final Bookworm based release even happened) as you can just check from the package changelog hosted on our repos:

Code: 
rust-proxmox-offline-mirror (0.6.1) bookworm; urgency=medium

  * add support for bookworm enterprise ceph repo

  * add bookworm to the list of releases for easier set up

 -- Proxmox Support Team <support@proxmox.com>  Fri, 16 Jun 2023 09:29:40 +0200
-- http://download.proxmox.com/debian/...-amd64/proxmox-offline-mirror_0.6.1.changelog
 
Hi,

thank you for your work on this project, looking forward to implementing it over the next weeks. I did notice two things in the docs that are a bit outdated, maybe you could update those:
Thank you!
 
Just tried to setup a test mirror with ondrej php ppa and got the following error:

Code: 
# curl -Lo /tmp/ppa-ondrej-php.asc "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x14aa40ec0831756756d7f66c4f4ea0aae5267a6c"
# proxmox-offline-mirror config mirror add --id ppa-ondrej-php --architectures amd64 --architectures all --architectures arm64 --repository "deb https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy main" --key-path /tmp/ppa-ondrej-php.asc --sync true --verify true --base-dir /tmp/ppa-ondrej-php --weak-crypto min-rsa-key-size=1024,allow-sha1=1
# proxmox-offline-mirror mirror snapshot create ppa-ondrej-php
Fetching Release/Release.gpg files
-> GET 'https://ppa.launchpadcontent.net/ondrej/php/ubuntu/dists/jammy/Release.gpg'..
-> GET 'https://ppa.launchpadcontent.net/ondrej/php/ubuntu/dists/jammy/Release'..
Verifying 'Release(.gpg)' signature using provided repository key..

Subkey of 14AA40EC0831756756D7F66C4F4EA0AAE5267A6C not bound: Policy rejected asymmetric algorithm
Caused by:
    0: RSA1024 is not considered secure since 2014-02-01T00:00:00Z

I've already added the weak-crypto param.
 
@morph027 there's a bug with the current package, could you also add "min-dsa-key-size=1024" as a workaround? the next version of POM should then work again without it.
 
  • Like
Reactions: morph027
POM 0.6.4 containing the fix is now available on the test repositories!
 
  • Like
Reactions: morph027
Do you mean for mirroring those images too (compared to asking if POM can run inside of them)?

As the former seems more likely, no currently POM cannot easily mirror the turnkey images too.
But it's actually an interesting request and I can see why it would be useful for air-gapped systems, can you please open an enhancement request for this over at https://bugzilla.proxmox.com/ so that we do not lose track of it? No promises though on how fast this gets added.

In the meantime you could download those files manually too from checking the locations in the aplinfo.dat appliance index, for TurnKey that would be https://releases.turnkeylinux.org/pve/aplinfo.dat

I quickly threw a small script together that query that and downloads all debian-12 based images, if not already present, and checks their sha256sum – disclaimer, it doesn't check the GPG signature of the index, but as that is served via TLS you should be fine as stop-gap as long as your DNS can be trusted.

Perl: 
#!/usr/bin/perl

use strict;
use warnings;

use LWP::UserAgent;

my $url = "https://releases.turnkeylinux.org/pve/aplinfo.dat";
my $filter_os = 'debian-12'; # only current Debian 12 Bookworm based templates

my $ua  = LWP::UserAgent->new(protocols_allowed => ['http', 'https'], timeout => 10);
$ua->agent('Simple Appliance Mirror 1.0');

my $apl_res = $ua->get($url);
die "could not retrieve aplinfo.dat index - " . $apl_res->status_line if !$apl_res->is_success;
my $aplinfo = $apl_res->decoded_content;

for my $entry (split(/\n\n/, $aplinfo)) {
    my ($os) = $entry =~ /^OS:\s*(.*)$/m;
    next if $os ne $filter_os;

    my ($location) = $entry =~ /^Location:\s*(.*)$/m;
    my ($sha512sum) = $entry =~ /^sha512sum:\s*(.*)$/m;
    next if !($location && $sha512sum);

    my ($filename) = $location =~ m{.*/(.*)$};
    next if !$filename;

    if (-e $filename) {
        print "$filename already exists.\n";
        next;
    }

    print "Downloading $filename...\n";
    my $download_res = $ua->get($location, ':content_file' => $filename);
    if (!$download_res->is_success) {
        warn "getstore of <$location> failed with ". $download_res->status_line ."\n";
        next;
    }

    my $actual_sha512sum = `sha512sum $filename |cut -d ' ' -f 1`;
    chomp $actual_sha512sum;

    my $match_str = $sha512sum eq $actual_sha512sum ? 'matches' : 'MISMATCH';
    print "SHA512 sum $match_str for $filename.\n";
}

Those archive files can then be copied to a template/cache (the default appliance directory) of a Proxmox VE storage.
 
Last edited:
Hi everyone, do the "clients" which will point to my offline repository server also needs subscription keys? or only the server which is running the proxmox-offline-mirror needs it?

Thanks
 
For repository mirroring and use thereof you do not need a subscription key at all.

The POM subscription is currently only required for offline subscription handling, i.e., if you want to set up longer-term subscriptions on the air-gapped PVE/PBS hosts.
 
t.lamprecht said:
For repository mirroring and use thereof you do not need a subscription key at all.

The POM subscription is currently only required for offline subscription handling, i.e., if you want to set up longer-term subscriptions on the air-gapped PVE/PBS hosts.
Click to expand...
Hi thanks for the reply. I have installed proxmox VÊ 8.1 from ISO and I am trying to install the proxmox-offiline- mirror package as indicated on the documentation (apt install proxmox-offline-mirror) and I am receiving an error saying that the package is not available so I am assuming apt is looking on the enterprise repo which I don’t have access as I don’t have a subscription isn’t it? I am using the very default installation of proxmox from ISO with original /etc/apt/souces.list
 
wond said:
I am receiving an error saying that the package is not available so I am assuming apt is looking on the enterprise repo which I don’t have access as I don’t have a subscription isn’t it? I am using the very default installation of proxmox from ISO with original /etc/apt/souces.list
Click to expand...
Yes, by default the enterprise repository is set up, and that one indeed needs a valid subscription for access.

But you can add the no-subscription repository on the web UI under Node -> Repositories and disable the enterprise one, after that refresh the repository state, either in the Updates section, or via executing apt update, after you should be able to install the proxmox-offiline- mirror package.

Our migration path for new package uploads is: internal -> pvetest repo -> pve-no-subscription repo -> pve-enterprise repo,
so the no-subscription holds a bit newer, and less tested packages compared to the enterprise repo, but otherwise it's feature complete.
 
Hi,

I'm trying to mirror the repo provided by Influx. Relevant section of the config file:

Code: 
mirror: influxdb
        architectures all
        architectures amd64
        base-dir /srv/www/influxdb/.mirror
        ignore-errors false
        key-path /srv/www/influxdb/influxdb.key
        repository deb https://repos.influxdata.com/debian stable main
        sync true
        verify true

But somehow not all of the repo is mirrored correctly:

Code: 
root@host:/srv/www/influxdb/.mirror/influxdb/2024-02-16T08:59:52Z/packages# ls -la
total 220898
drwxr-xr-x  2 root root       19 Feb 16 09:59 .
drwxr-xr-x  4 root root        4 Feb 16 09:59 ..
-rw-r--r-- 19 root root 44397274 Feb 12 22:37 chronograf_1.10.1_amd64.deb
-rw-r--r-- 19 root root 44505840 Feb 12 22:37 chronograf_1.10.2_amd64.deb
-rw-r--r-- 19 root root 33711326 Feb 12 22:37 chronograf_1.9.1_amd64.deb
-rw-r--r-- 19 root root 33718038 Feb 12 22:37 chronograf_1.9.4_amd64.deb
-rw-r--r-- 19 root root  4795820 Feb 12 22:37 influxctl_v2.0.0_amd64.deb
-rw-r--r-- 19 root root  4796402 Feb 12 22:37 influxctl_v2.0.1_amd64.deb
-rw-r--r-- 19 root root  4797184 Feb 12 22:37 influxctl_v2.0.2_amd64.deb
-rw-r--r-- 19 root root  4801728 Feb 12 22:37 influxctl_v2.0.3_amd64.deb
-rw-r--r-- 19 root root  4822656 Feb 12 22:37 influxctl_v2.0.4_amd64.deb
-rw-r--r-- 19 root root  4862386 Feb 12 22:37 influxctl_v2.1.0_amd64.deb
-rw-r--r-- 19 root root  4864304 Feb 12 22:37 influxctl_v2.2.0_amd64.deb
-rw-r--r-- 19 root root  4866594 Feb 12 22:37 influxctl_v2.3.0_amd64.deb
-rw-r--r-- 19 root root  4866364 Feb 12 22:37 influxctl_v2.3.1_amd64.deb
-rw-r--r-- 19 root root  6523578 Feb 12 22:37 influxctl_v2.4.0_amd64.deb
-rw-r--r-- 19 root root  6523632 Feb 12 22:37 influxctl_v2.4.1_amd64.deb
-rw-r--r-- 19 root root  6521926 Feb 12 22:37 influxctl_v2.4.2_amd64.deb
-rw-r--r-- 19 root root  6533662 Feb 12 22:37 influxctl_v2.4.3_amd64.deb

Relevant packages (influxdb2, influxdb2-client, telegraf) that are available at https://repos.influxdata.com/debian/packages/ are missing from the sync'd repo.

Output of sync:
Code: 
# proxmox-offline-mirror mirror snapshot create influxdb
Fetching Release/Release.gpg files
-> GET 'https://repos.influxdata.com/debian/dists/stable/Release.gpg'..
-> GET 'https://repos.influxdata.com/debian/dists/stable/Release'..
Verifying 'Release(.gpg)' signature using provided repository key..
Success
Parsing Release..
'Release' file has 16 referenced files..
Fetching InRelease file
-> GET 'https://repos.influxdata.com/debian/dists/stable/InRelease'..
Verifying 'InRelease' signature using provided repository key..
6019 bytes verified
Success
Parsing InRelease..
'InRelease' file has 16 referenced files..
Skipping main/binary-riscv64/Packages
Skipping main/binary-riscv64/Release
Skipping main/binary-arm64/Packages.gz
Skipping main/binary-armhf/Packages
Skipping main/binary-i386/Release
Skipping main/binary-armel/Release
Skipping main/binary-i386/Packages
Skipping main/binary-s390x/Packages
Skipping main/binary-armel/Packages.gz
Skipping main/binary-arm64/Release
Skipping main/binary-armhf/Release
Skipping main/binary-ppc64el/Release
Skipping main/binary-s390x/Release
Skipping main/binary-ppc64el/Packages

Component 'main'
        "main/binary-amd64/Packages": Packages("amd64", None)
        "main/binary-amd64/Packages.gz": Packages("amd64", Some(Gzip))
        "main/binary-amd64/Release": PseudoRelease(Some("amd64"))
Component references count: 2
Component indices size: 46259
Total indices count: 2
Total indices size: 46259
Skipped 14 references


Fetching indices for component 'main'
        Fetching 'main/binary-amd64/Packages'..
        17 packages totalling 225908714
Progress: 0 new files (0b), re-used 1 existing files (100.00% re-used)..
        Fetching 'main/binary-amd64/Release'..
Progress: 0 new files (0b), re-used 2 existing files (100.00% re-used)..
Total deb size for component: 225908714
Total dsc size for component: 0
Total deb size: 225908714

Fetching main packages..

main/binary-amd64/Packages - 17 total file(s)
        Progress: 0 new files (0b), re-used 1 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 2 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 3 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 4 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 5 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 6 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 7 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 8 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 9 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 10 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 11 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 12 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 13 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 14 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 15 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 16 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 17 existing files (100.00% re-used)..
        Progress: 0 new files (0b), re-used 17 existing files (100.00% re-used)..

Stats: 2 new files (13752b), re-used 19 existing files (90.48% re-used)..
Skipped downloading 0 packages totalling 0b

Rotating temp. snapshot in-place: "2024-02-16T10:45:34Z.tmp" -> "2024-02-16T10:45:34Z"

Is my configuration incorrect?

Regards,

Uwe
 
I can reproduce a similar issue - but for me it only downloads the "chronograf" packages..
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom