[TUTORIAL] HOWTO : Wrapper Script to Use Fedora CoreOS Ignition with Proxmox cloud-init system for Docker workloads

G

Geco-iT

Member
Mar 24, 2021
17
23
8
France
www.geco-it.fr
Hi,

we are a small compagny ( Geco-iT ) from France that strongly relies on Proxmox PVE every day and as we find proxmox more and more powerfull, we want to give back to the community by providing some of our tools for PVE.

We made a tool to use Fedora CoreOS as VM with proxmox cloud-init system !


Fedora CoreOS

For our Docker Swarm / Kubernetes container orchestration projects, we migrated from CoreOS distribution to Fedora CoreOS distribution.

Fedora CoreOS is an Operating system that is auto-updated, minimal and containers oriented. It is derived from Fedora Community distribution and especially designed for containerized environnement. The main goal is to support containerized workloads in a secure way.

The Fedora CoreOS distribution succeed to two projects, Fedora Atomic Host and CoreOS Container Linux. The idea is to merge, within a minimal operating system, OCI support and Atomic Host packaging with provisionning tools and automated roll-out of CoreOS.

Fedora CoreOS is the base of Red Hat Enterprise Linux CoreOS (RHCOS).

Fedora CoreOS features list contains :

* Automated update with stagged roll-out
* Base System : Last Fedora
* Last stable Linux Kernel
* Provisionning tools : Ignition
* OCI Support for Docker containers by Podman and Moby
* Hardenning by SeLinux
* Out of the box Cgroups support


Intégration of VM Fedora CoreOS within Proxmox VE


Proxmox VE use Cloud-Init as a provisionning tool for Virtual Machines (VM) Cloud-Init but Fedora CoreOS is only compatible with “Ignition”. So we made a « wrapper » that convert the Cloud-Init config of proxmox to an Ignition compatible config…

Our Cloud-Init ⇔ Ignition wrapper takes care of the following parameters :

* Username ; by default = admin
* Password
* DNS Domain
* DNS server(s)
* SSH Key(s)
* IP configuration(s) but only IPv4

Our tool will automatically :

1) Download the Fedora CoreOS Image
2) Create a virtual Machine
3) Import the Fedora CoreOS as a VM Disk
4) Add a cloud init config drive to that VM
5) Add the « hook-script » hook-fcos.sh at VM startup
6) Copy the Ignition template in a « Proxmox snippet » storage
7) Convert the VM to a Template that you can use after :)


Full explanation is here on our wiki : https://wiki.geco-it.net/public:pve_fcos
Source code can be seen here: https://git.geco-it.net/GECO-IT-PUBLIC/fedora-coreos-proxmox

Hope it will help someone !

Thanks to all the Proxmox Staff !

Have a nice day !
 
Last edited:
  • Like
Reactions: ZipTX, pille99, ophil and 7 others
Thank you. I added the Tutorial prefix to the thread so people who search for tutorials/howtos can filter it more easily. :)
 
  • Like
Reactions: guletz and Geco-iT
gecoit said:
Hi,

we are a small compagny ( Geco-iT ) from France that strongly relies on Proxmox PVE every day and as we find proxmox more and more powerfull, we want to give back to the community by providing some of our tools for PVE.

We made a tool to use Fedora CoreOS as VM with proxmox cloud-init system !


Fedora CoreOS

For our Docker Swarm / Kubernetes container orchestration projects, we migrated from CoreOS distribution to Fedora CoreOS distribution.

Fedora CoreOS is an Operating system that is auto-updated, minimal and containers oriented. It is derived from Fedora Community distribution and especially designed for containerized environnement. The main goal is to support containerized workloads in a secure way.

The Fedora CoreOS distribution succeed to two projects, Fedora Atomic Host and CoreOS Container Linux. The idea is to merge, within a minimal operating system, OCI support and Atomic Host packaging with provisionning tools and automated roll-out of CoreOS.

Fedora CoreOS is the base of Red Hat Enterprise Linux CoreOS (RHCOS).

Fedora CoreOS features list contains :

* Automated update with stagged roll-out
* Base System : Last Fedora
* Last stable Linux Kernel
* Provisionning tools : Ignition
* OCI Support for Docker containers by Podman and Moby
* Hardenning by SeLinux
* Out of the box Cgroups support


Intégration of VM Fedora CoreOS within Proxmox VE


Proxmox VE use Cloud-Init as a provisionning tool for Virtual Machines (VM) Cloud-Init but Fedora CoreOS is only compatible with “Ignition”. So we made a « wrapper » that convert the Cloud-Init config of proxmox to an Ignition compatible config…

Our Cloud-Init ⇔ Ignition wrapper takes care of the following parameters :

* Username ; by default = admin
* Password
* DNS Domain
* DNS server(s)
* SSH Key(s)
* IP configuration(s) but only IPv4

Our tool will automatically :

1) Download the Fedora CoreOS Image
2) Create a virtual Machine
3) Import the Fedora CoreOS as a VM Disk
4) Add a cloud init config drive to that VM
5) Add the « hook-script » hook-fcos.sh at VM startup
6) Copy the Ignition template in a « Proxmox snippet » storage
7) Convert the VM to a Template that you can use after :)


Full explanation is here on our wiki : https://wiki.geco-it.net/public:pve_fcos
Source code can be seen here: https://git.geco-it.net/GECO-IT-PUBLIC/fedora-coreos-proxmox

Hope it will help someone !

Thanks to all the Proxmox Staff !

Have a nice day !
Click to expand...

geco-it.net has been down for days - I would really like to see full explanation and source if possible.
 
Last edited:
gecoit said:
Please tell me what you have as ip address if you run dig wiki.geco-it.net or nslookup, or a ping ?
Click to expand...

dig wiki.geco-it.net

; <<>> DiG 9.10.6 <<>> wiki.geco-it.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3484
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;wiki.geco-it.net. IN A

;; ANSWER SECTION:
wiki.geco-it.net. 3556 IN CNAME buro.geco-it.net.
buro.geco-it.net. 16 IN CNAME burowan1.geco-it.net.
burowan1.geco-it.net. 3556 IN A 81.250.158.54

;; Query time: 1 msec
;; SERVER: 192.168.0.1#53(192.168.0.1)
;; WHEN: Thu May 06 07:05:38 AEST 2021
;; MSG SIZE rcvd: 125

----

nslookup wiki.geco-it.net
Server: 192.168.0.1
Address: 192.168.0.1#53

Non-authoritative answer:
wiki.geco-it.net canonical name = buro.geco-it.net.
buro.geco-it.net canonical name = burowan1.geco-it.net.
Name: burowan1.geco-it.net
Address: 81.250.158.54

---

ping wiki.geco-it.net
PING burowan1.geco-it.net (81.250.158.54): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5
Request timeout for icmp_seq 6
Request timeout for icmp_seq 7
Request timeout for icmp_seq 8
^C
--- burowan1.geco-it.net ping statistics ---
10 packets transmitted, 0 packets received, 100.0% packet loss
 
IP address is fine, without proper knowledge of your infrastructure i can't really help you, maybe firewall ?
I asked several people, and tried from different servers, it works everywere...
 
I have managed to access links by running VPN and pretending I am in France.
So somewhere along the route my IP or range is blocked.
 
This looks really great @gecoit

We use Butane to generate our ign configs. We have a set of generated ignition files (.ign's) already. Would it be possible to use your tool to deploy those to a fcos vm in proxmox?
 
Hi @ramblurr,

I think you can use your file, you have 3 options :

  • You want to tweak all VM ignition : modify the fcos-base-tmpl.yaml in the snipet proxmox storage to add your options
  • You want to tweak only one VM : modify the script with a IF statement, that check the vmid and cat your custom files in the snippet storage for that vmid before writing the result in /etc/pve/geco-pve/coreos/$vmid.ign
  • You want to use your ign files as they are : you want to modify the vmsetup script so that it does not cat cloud init and template but directly your files
 
I just set this up with the latest stable CoreOS, and had to do the following:

- The motd in CoreOS is now /run/motd.d, so had to modify the OUT variable in geco-motd accordingly
- qemu agent was failing to install with rpm-ostree because it was saying that a rpm-ostree transaction was already occurring. This was fixed by adding "Before=boot-complete.target" to the service definition.
 
Last edited:
MiLK_MaN said:
I just set this up with the latest stable CoreOS, and had to do the following:

- The motd in CoreOS is now /run/motd.d, so had to modify the OUT variable in geco-motd accordingly
- qemu agent was failing to install with rpm-ostree because it was saying that a rpm-ostree transaction was already occurring. This was fixed by adding "Before=boot-complete.target" to the service definition.
Click to expand...

I just began experimenting with this. My initial response in this posting was to ask for some updated source code, though I have since figured out the changes. These changes presume the latest version of Fedora CoreOS 37 is to be used.

These changes are made to the fcos-base-tmplt.yaml file:

8c8
< OUT=/etc/console-login-helper-messages/motd.d/22_geco.motd
---
> OUT=/etc/motd.d/22_geco.motd
264c264
< - path: /etc/console-login-helper-messages/issue.d/00_geco.issue
---
> - path: /etc/issue.d/00_geco.issue
339a340
> Before=boot-complete.target

I also wanted to point out that if you get an error from hook-fcos.sh that it had to restart the VM due to having to regenerate the ISO (cloud init) and it is really nothing to worry about. That said, placing an "set -v -x" at the beginning of the "pre-start" code and a "sev +v +x" at the end caused the entire output of the code section to become instantiated within the task log (thus making debugging much easier).

I hope this is of use to people in the future.

I will eventually setup a github repo with the totality of my changes and update this posting to make the link inclusive herein.


Stuart
 
Last edited:
Geco-iT said:
Hi,

we synced our repos to github, now everyone can contribute with his GH account and we will review pull requests !
You can find the Geco-iT org here : https://github.com/GECO-IT/

Have a nice day !
Click to expand...

Geco-IT:

Bonjour!

This is most excellent news! Indeed, I made some several updates to the code (the most prominent being the allowance for the latest Fedora CoreOS to work) as well as some others (one of which being the addition of a VLAN parameter).

I will fork the code and prepare some PRs later today. I made a few changes so I just need to break them out into separate PRs so you can choose each one individually without having to do an all or nothing choice on the code.

One small problem though... Follows hereupon the repo I was working on which was conspicuously NOT published on GitHub and the very one I have modified to function!

https://git.geco-it.net/GECO-IT-PUBLIC/fedora-coreos-proxmox


Stuart, N3GWG
 
Last edited:
jpbaril, et alia:

I first started using the Geco-IT version, I then found Jim Lee's version of the code and have been working off of his code. His code was based on an intermediate repository (by someone else) that had added flatcar support first, then Jim Lee added his changes to that code base. My version (I intend to publish my changes very soon, once I re-organize some 41 or so commits to my local repository and write some documentation) is based on Jim Lee's code version (that included prior flatcar support too).

Here is the link to Jim Lee's code:

https://github.com/jimlee2002/fedora-coreos-proxmox

Recently Geco-IT published to github, but in precedence to a few weeks ago that was not so. Thus, I searched GitHub and then thereafter they did publish there too.

I will post a message here once I have published the full repository of the changes I made. I am working on trying to support migration of a Fedora CoreOS VM (currently it reboots during the migration from node to node and I am not sure why yet). But once that change is made I will be ready to finish up documentation and publish the changes I made.


Stuart, N3GWG
 
Last edited:
  • Like
Reactions: ZipTX and jpbaril
Hi,
I could deploy fedora core OS template using geco-it scripts fine on 4 nodes cluster with shared storage (ceph for block devices and NFS for snippets) but i cannot live migrate VM due hookscript error during start of virtual machine on target node

Code: 
TASK ERROR: hookscript error for <VM_ID> on pre-start: command '/mnt/pve/NFS/snippets/hook-fcos.sh <VM_ID> pre-start' failed: exit code 1

Try to check hook-fcos.sh code to avoid error with no luck...
Also try to delete hookscript from vm config but get ign file error, check folder and ignition file was deleted after vm first restart as it were setup on script

Someone could deploy on cluster supporting vm live migration sucessfully?
PVE 7.3-3 cluster

UPDATE:
After more test I could do a live migration if a remove args from VM config file, but i need to remove this config on every clone from template
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom