LXC template

[PaulVM]

Proxmox itself provides a variety of basic templates for the most common Linux distributions, Ok.
But I was wondering about the difference to use this templates Versus use an updated template.
Today I was planning to installa a new CentOS 8 container (I know CentOS 8 is declared dead).
I downloaded the template and noticied that it is an old initial 8.0:

centos-8-default_20191016_amd64.tar.xz

If I get:

http://uk.images.linuxcontainers.org/images/centos/8/amd64/default/20201210_07:08/rootfs.tar.xz

I have an updated 8.3

What are differencies, advantages/disadvantages to use the default templates instead of updated?
Are the default templates somehow optimized or have something different that have to be considered?

Thanks, P.

 

[t.lamprecht]

Hi,

One difference is that Debian, its derivative Ubuntu, and Arch Linux are built directly by us, not pulled from the images mirror you linked, so you won't get the same thing for those.
FYI, we use our Debian Appliance Builder and the Arch Linux Appliance Builder for that
https://git.proxmox.com/?p=dab.git;a=summary
https://git.proxmox.com/?p=aab.git;a=summary
(a bit dated docs are here: https://pve.proxmox.com/wiki/Debian_Appliance_Builder )

The next difference is that the one we distribute are always end to end verified, i.e., we do strong verification with the GPG keys of images we redistribute, and the tooling which provides integrated template download also checks the signatures - ensuring you get a clean, secure template.
If you manually download such images, especially over insecure channels like plain-text HTTP, I really advise to manually check the signatures with GPG, that should be also done when using secure channels, as a MITM is also possible there (a valid certificate does not necessarily mean that it was the correct certificate).

If those points are considered, then yes, you could use that CentOS rootfs image, without expecting issues.

We do not update those templates daily as often it's also fine to just use the older one and run an upgrade command after initial setup.

 

[PaulVM]

. . .

The next difference is that the one we distribute are always end to end verified, i.e., we do strong verification with the GPG keys of images we redistribute, and the tooling which provides integrated template download also checks the signatures - ensuring you get a clean, secure template.
If you manually download such images, especially over insecure channels like plain-text HTTP, I really advise to manually check the signatures with GPG, that should be also done when using secure channels, as a MITM is also possible there (a valid certificate does not necessarily mean that it was the correct certificate).

If those points are considered, then yes, you could use that CentOS rootfs image, without expecting issues.

That are some of the reasons I prefer to use the supplied templates instead of fetch something from external sources ...

We do not update those templates daily as often it's also fine to just use the older one and run an upgrade command after initial setup.

This is instead a reason sometime I search for supplementar sources ...
I agree that update the templates daily/weekly/.... is not very useful, but I suggest update them at least when there is a serious update (8.0 to 8.3 seems to me relevant).
It happen usually 1-2 times / year ;-)
And in my experience, a fresh .3 install is different (often really different) than a .0 install updated to .3 (I noticied many "interesting" differences for example in CentOS 7.x).
Thanks for clarifications and for your works ;-)

Thanks, P.

 

[moxfan]

And in my experience, a fresh .3 install is different (often really different) than a .0 install updated to .3 (I noticied many "interesting" differences for example in CentOS 7.x).

What type of differences?

 

[t.lamprecht]

It happen usually 1-2 times / year ;-)

That's the idea, and done for most - CentOS seems to have slipped and nobody cared to much or notified us to upload a newer template.

I did so now, if you run pveam update the template index will get updated (that happens normally automatically during the night) and it should be available for download.

 

[PaulVM]

What type of differences?

After a few years and many others things managed my memory can't be a perfect reference, but I assure you I lost many hours to understand/indagate differences.
A really evident difference was the /var/log/journal folder managment and various systemd settings.
Some packages have different settings
Some packages disapperas
file/folder permissions
....

Regards, P.

 

[PaulVM]

That's the idea, and done for most - CentOS seems to have slipped and nobody cared to much or notified us to upload a newer template.

When Proxmox used OpenVZ and RH kernel, it seems to me that CentOS/RH container were simpler to manage than in LXC (less strange issue/behauvoir).
So, often, I and many people I know, prefer to use KVM when there are no rilevant advantages in using CT.

I did so now, if you run pveam update the template index will get updated (that happens normally automatically during the night) and it should be available for download.

Thanks.
Curiosity. I noticied:

-rw-r--r-- 1 root root 106244064 Oct 16 2019 centos-8-default_20191016_amd64.tar.xz
-rw-r--r-- 1 root root 99098368 Dec 10 09:10 centos-8-default_20201210_amd64.tar.xz
-rw-r--r-- 1 root root 99098368 Dec 10 09:10 rootfs.tar.xz

The new C8 templates seems to be the same I have downloaded from linuxcontainers

# diff -s centos-8-default_20201210_amd64.tar.xz rootfs.tar.xz
Files centos-8-default_20201210_amd64.tar.xz and rootfs.tar.xz are identical

It is smaller than the old 8.0 and if you compare the contents it is evident that it miss some packages.
The old one has the same source or was generated from you or fetched from other source?

Thanks, P.

 

[t.lamprecht]

it seems to me that CentOS/RH container were simpler to manage than in LXC (less strange issue/behauvoir).

What strange behavior?

The old one has the same source or was generated from you or fetched from other source?

Both have the exact same source, quite possibly an effect of the changes happening in CentOS repos recently.
I mean, they differ only by 6.8 MiB in size, what packages are actually missing?

 

[PaulVM]

What strange behavior?

Time flows ... my memory is not very good at remember all the problems I went throught ... ;-)
The most remarkable was that sometime a CT became unresponsive and and you couldn't reboot it. You had to kill it and often reboot the server because a kworker* process saturated cpu and blocked some services.
With the firsts 5.x kernel a simple reboot of a CentOS 7 CT generated this issue.
With 6.x I havent' noticied this problem recently.
Many problems/bugs with console and other in CentOS 6/7 ,,,
Some also documented in Proxmox wiki.
I cant' remember or lists them all on the fly, but this lead me and other gui to use KVM instead of LXC CT when there is no great advantages in using CT (that for me are the mount points).

Both have the exact same source, quite possibly an effect of the changes happening in CentOS repos recently.
I mean, they differ only by 6.8 MiB in size, what packages are actually missing?

6.8 MB in .XZ can be many things ;-)
The answer is to install a couple of new CT, do a simple rpm -qa and compare the results.
Anyway, the default CT that I (and you) had fetched from linuxcontainers.org is a really really minimal install. That can be Ok, but only experienced sysadmin can transform it into a useful server installation.
For comparision, the Debian 10 templates (still very minimal as expected is 220 MB vs 100 MB of C8 (Ok, tgz Vs txz ...)).
At least it has sshd and postfix ...
I understand (think), the minimal filosophy of CT, but as I wrote above, at least some minimal tools probably have to be present for give the opportunity to use CTs to some sysadmin not able to build a system from scratch. Also the old net-tools and some standard packages can be sufficient. Or some docs that explain how to install a basic combination of packages.
CTs are one of the reasons I preferred PM instead of ESXi in the past years, but it seems that they are not really considered comparing to KVM. They are a really interesting option for someone that want something "lighter" than a full virtualized VM but more flexyble than a docker instance.
My personal thoughs. I am a quite experiences sysadmin, but not a "guru" ;-)

Thanks, P.

 

[t.lamprecht]

Some also documented in Proxmox wiki.

Care to post a link to that?
We had a bug with the console, but that one was completely orthogonal to the underlying implementation, would have happend too with openvz.

If you know of actual, reproducible issues, then please post them here (with full version info and steps to reproduce) or open a report over at https://bugzilla.proxmox.com/ - that can get them actually fixed.

6.8 MB in .XZ can be many things ;-)

Or just a bit of documentation..

The answer is to install a couple of new CT, do a simple rpm -qa and compare the results.

Why don't you just post it, you're the one implying stuff without sources, posting those normally that helps to back up such statements ;-)

At least it has sshd and postfix ...

You hopefully do not need to be an experienced sysadmin to be able to do: apt/dnf/yum/zypper/what-not install openssh-server, also you can always install your desired packages and create a Proxmox VE CT template where you can clone from.

I understand (think), the minimal filosophy of CT, but as I wrote above, at least some minimal tools probably have to be present for give the opportunity to use CTs to some sysadmin not able to build a system from scratch. Also the old net-tools and some standard packages can be sufficient.

That'd be best reported to CentOS upstream (is there still any?) or the lxc-template repository.

Honestly, personally I'd try to switch over to Debian, it's actually independent but still gets stuff done, has a very stable base and provides a clear upgrade path on major releases.

Or some docs that explain how to install a basic combination of packages.

That seems rather reasonable to me, could do that.

but it seems that they are not really considered comparing to KVM.

I mean, they are two completely different technologies, so comparing them will always lead to bigger differences.
That does not mean that LXC CT are no first level "citizen" in Proxmox VE.

 

[PaulVM]

Care to post a link to that?
We had a bug with the console, but that one was completely orthogonal to the underlying implementation, would have happend too with openvz.

The wiki:
https://pve.proxmox.com/wiki/Convert_OpenVZ_to_LXC#CentOS_6_OpenVZ_to_LXC_migration_issues
refer to:
https://forum.proxmox.com/threads/centos-6-openvz-to-lxc-migration-issues.35058/

And also some other sparse bit of info that I can't remember now where they are.
When I cross a problem I do what I can to solve it or to workaround it and then I forgot it because I am busy to new problems ;-)

If you know of actual, reproducible issues, then please post them here (with full version info and steps to reproduce) or open a report over at https://bugzilla.proxmox.com/ - that can get them actually fixed.

Ok. I consider this "offer" ;-)

Or just a bit of documentation..

Why don't you just post it, you're the one implying stuff without sources, posting those normally that helps to back up such statements ;-)

You are more lazy than me ;-)
Ok.
Done the CTs installation.
c80 ise the original Centos8,0
C83 is the latest CentOS 8.3.

@c80 /]# rpm -qa |wc -l
225

@c83 /]# rpm -qa |wc -l
193

I think it isn't useful to post the whole list of packages (if you want them simply ask ;-) ), so I report the differences only:

80-83 (present in 8.0, absent in 8.3):
centos-release
cpio
dracut
file
fipscheck
fipscheck-lib
freetype
gettext
gettext-libs
grub2-common
grub2-tools
grub2-tools-efi
grub2-tools-extra
grub2-tools-minimal
grubby
hardlink
kbd
kbd-legacy
kbd-misc
kmod
kpartx
libcroco
libgomp
libkcapi
libkcapi-hmaccalc
libmodulemd1
libpkgconf
libpng
libxkbcommon
os-prober
pigz
pkgconf
pkgconf-m4
pkgconf-pkg-config
polkit-libs
python3-iniparse
python3-six
systemd-udev
which
xkeyboard-config
xz

83-80 (present in 8.3, absent in 8.0):
centos-gpg-keys
centos-linux-release
centos-linux-repos
elfutils-debuginfod-client
libmodulemd
libssh-config
libzstd
python3-pip-wheel
python3-setuptools-wheel

You hopefully do not need to be an experienced sysadmin to be able to do: apt/dnf/yum/zypper/what-not install openssh-server, also you can always install your desired packages and create a Proxmox VE CT template where you can clone from.

Yes, but if you aren't an experienced sysadmin and you have network problems probably you don't know the ip syntax and the lack of the olds ifconfig, route, ... commands that everyone know can lead you to rid off the CT and install a VM (at laest some people told me this ;-) )

Honestly, personally I'd try to switch over to Debian, it's actually independent but still gets stuff done, has a very stable base and provides a clear upgrade path on major releases.

Debian 10 EOL is 2022
CentOS 8 EOL was 2029 now they say 31/12/2021, but I suppose there will be a RockyLinux or another project that can supply the continuity of the RH8 clone

That seems rather reasonable to me, could do that.

May I suggest to use a different style than the man-pages? If you want attract CTs users ... ;-)

I mean, they are two completely different technologies, so comparing them will always lead to bigger differences.
That does not mean that LXC CT are no first level "citizen" in Proxmox VE.

May be I am wrong, but as I told, seems to me that KVM was the main focus of Proxmox.
Since his replacement of OpenVZ, LXC give me this idea. May be because it wasn't solid and "good" like OpenVZ was in his era.

Thanks, P.

 

[Nuke Bloodaxe]

Apologies in advance if this seems a little off-topic, but I was wondering if an Oracle Linux CT might fit the bill here, they are still doing freely available builds of RHEL tracking the normal enterprise release schedule. This would get around the Centos Stream problem and the Centos EOL, while allowing a binary compatible orderly transition to a RHEL clone that already exists [like Centos did].

Note: Yes, I know it's Oracle, but in this case they do offer it freely; they only charge if you actually want support.

 

[PaulVM]

Apologies in advance if this seems a little off-topic, but I was wondering if an Oracle Linux CT might fit the bill here, they are still doing freely available builds of RHEL tracking the normal enterprise release schedule. This would get around the Centos Stream problem and the Centos EOL, while allowing a binary compatible orderly transition to a RHEL clone that already exists [like Centos did].

Note: Yes, I know it's Oracle, but in this case they do offer it freely; they only charge if you actually want support.

Oracle is Oracle ... ;-)
Like IBM is IBM ;-)
I don't think the problem is RH, CentOS, OL or may be RockyLinux or another RH clone distro.
My trust in Oracle is very limited, but I trust there will be a CentOS replacement if it really stops like they claimed.
There are so many interested ... (see https://top500.org/statistics/list/ Category OS )
And if we survived to the CentOS 5 --> 6 migration era ... ;-)

I like many aspects of Debian and also of other distros, but after the initial Slackware era, I choosed RH and derivates (so CentOS in last 13-14 years, after Trustix death ...), because it is the reference in business area (IMHO).
But this isn't a Proxmox "problem" ;-)
Proxmox had only to be "stimulated" to update the templates at least when a new dot release is published
Also the C7 templates is more than a year old.
May be that there is not so many Proxmox users insterested in the CentOS CTs
Or that they fetch otherwise the templates if the standard anren't updated ;-)