[SOLVED] Debian 12 - Network / cloud-init / proxmox / netplan vs interfaces

A

andaga

Member
Proxmox Subscriber
May 24, 2022
133
17
23
Switzerland
Hello everyone!

I have a problem with Debian 12 which has been seriously bothering me for a few days and I really don't know how to solve it and above all I don't understand who is sending the bad information.

Well, let's start working on the latest build of ProxMox! So completely up to date!

My issue is the following:

I create a Debian 12 template VM with cloud-init! I fill in my fields in cloud-init and the network part!

My new VM is created magnificent everything is fine! EXCEPT the network which screws up every time!

By default Debian 12 uses netplan and no longer /etc/network/interfaces (which I liked this one)!

In the config of the 50-cloudinitt...... file of netplan we see that for the gateway part it uses the
gateway4: IP of the gateway

instead of using the

routes:
- on-link: true
to: default
via: STANDARD-GATEWAY OF ACTIVE INTERFACE

So I get errors all the time when I do "netplan apply" and especially if I put a /32 network mask I literally have no traffic at all because the information is missing:

- on-link: true

so that /32 for ipv4 and /128 for ipv6 works!

In summary, who sends the information in the vm? Is it ProxMox? Is it the qcow2 of the image I download? How do we fix this because I'm starting to go crazy! lol :)

Greetings
 
Last edited:
Since when does Debian 12 use netplan? I have a half dozen Debian installs and none of them use it. The desktop ones use network manager not netplan. Servers use ifupdown aka interfaces file. Ubuntu uses netplan but not Debian unless cloudinit is doing it different for some reason.

Maybe this is why you get errors.
 
  • Like
Reactions: ikarlo
andaga said:
I create a Debian 12 template VM with cloud-init!
Click to expand...
There are many places where this can go wrong. The issue can potentially be in your process and troubleshooting that is beyond the scope of this forum.
andaga said:
! I fill in my fields in cloud-init and the network part!
Click to expand...
Do you mean in PVE cloud-init wizard? Its possible that you will need to use custom cloudinit file.
andaga said:
In summary, who sends the information in the vm? Is it ProxMox? Is it the qcow2 of the image I download? How do we fix this because I'm starting to go crazy!
Click to expand...
I'd start with using a prebuild official cloud image, where chances are higher that its correct, and make sure you can get that working. Try both the built-in cloud-init wizard and via custom cloud-init files (user, network, vendor).

If everything works, then continue with your custom image and start with getting it working with custom cloud-init, using one of the guides on the net. When you have a working file you can compare it with what PVE creates and if there are radical differences.

Good luck

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
There are many places where this can go wrong. The issue can potentially be in your process and troubleshooting that is beyond the scope of this forum.

Do you mean in PVE cloud-init wizard? Its possible that you will need to use custom cloudinit file.

I'd start with using a prebuild official cloud image, where chances are higher that its correct, and make sure you can get that working. Try both the built-in cloud-init wizard and via custom cloud-init files (user, network, vendor).

If everything works, then continue with your custom image and start with getting it working with custom cloud-init, using one of the guides on the net. When you have a working file you can compare it with what PVE creates and if there are radical differences.

Good luck

Blockbridge: Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
Hello!
We have installed this qcow2 disk from this link:

https://cloud.debian.org/images/clo...ian-12-genericcloud-amd64-20231210-1591.qcow2

We have create the template and complete the field into the Cloud-init menu of the VM into ProxMox!

Everything is sent correctly during boot of the vm except the network side on the 50-cloudinit files into netplan folder!

It's using old value like gateway4 explained before!

But I don't understand what you mean about custom cloudinit files... You mean into the template? If yes! At which path??
Or at the ProxMox level?

Thank you for your time.
 
  • Like
Reactions: Der Harry
Hello! Finally I have found the issue after doing more hours of research!

In fact it was a cloud-init bug!

I explain:

Debian 12 Generic Cloud image we can dowload from Debian website come with default network the "netplan"
The latest version of cloud-init installed into Debian 12 over the normal repo are the Cloud-init 22.4.2

After some research into all the logs and changelog of cloud-init release we can see from the cloud-init 23.1.X version the problme about the netplan config with /32 subnet and the deprecated gateway4, the right on the directoy when you do netplan apply and they said the right on this folder is wrong... All of this thing is fixed on the new release.

For that I have downloaded the new release stable at this link:
https://packages.debian.org/sid/all/cloud-init/download

Installed into my debian 12 cloud image and now everything work.

I hope this thread can help other proxmox user to fix the problem.

Regards

Anthony
 
  • Like
Reactions: macboy80
This problem exists in Ubuntu below 22 and all Debian versions. It happens because Cloudinit uses gateway4 and gateway6 to apply static routes to the gateway but this is now depreciated. In Ubuntu 22 and netplan the routes command is used to apply the static routes to the gateway so you won't have the same issues. It's a real pain the in the fucking arse for people who rent or co-locate servers because it means you basically have to create a custom OS per server because the gateway to the outside world is usually different. I tend to create a systemd service that runs just after Cloudinit. The service runs a simple bash script that applies the static routes to the gateway. This way I just have to run something like virt-edit -a debian-12-generic-amd64-daily-20231218-1599.qcow2 /path/to/script to update the IP for the gateway and then package it as a template. You can save yourself 5 minutes of boot time by disabling the systemd-networkd-wait-online.service. Someone here might have a better way of doing this but I find running the script at boot is very reliable.
 
Hello!
I think the problem is deeper than this!

Because if Cloud-Init was updated in official rest to switch to 23.x there would no longer be this concern for Gateway4 etc ... but directly by the new road mode:

Having just updated manually by downloading the .deb package to update Cloud-Init at the OS level, I no longer have any worries! So no need to script or anything!

On the other hand, I would really like to understand why the update is not offered at the Debian 12 level to put the 23.x of Cloud-Init! It's just incredible!
 
When did release 23.1 of Cloud-Init come out? Debian 12 has been out since June and was frozen for many months before that. Debian don't update versions of packages between releases. They're like Red Hat Enterprise that way.

If you want a Linux that is up to the last five-minute-ago release you're on the wrong one. Or you should use testing or unstable builds.
 
Last edited:
andaga said:
Hello! Finally I have found the issue after doing more hours of research!

In fact it was a cloud-init bug!

I explain:

Debian 12 Generic Cloud image we can dowload from Debian website come with default network the "netplan"
The latest version of cloud-init installed into Debian 12 over the normal repo are the Cloud-init 22.4.2

After some research into all the logs and changelog of cloud-init release we can see from the cloud-init 23.1.X version the problme about the netplan config with /32 subnet and the deprecated gateway4, the right on the directoy when you do netplan apply and they said the right on this folder is wrong... All of this thing is fixed on the new release.

For that I have downloaded the new release stable at this link:
https://packages.debian.org/sid/all/cloud-init/download

Installed into my debian 12 cloud image and now everything work.

I hope this thread can help other proxmox user to fix the problem.

Regards

Anthony
Click to expand...
I would like to know how to update the cloud-init package in the default qcow2. can you point me in the right direction?
 
  • Like
Reactions: jt_telrite
jt_telrite said:
are you suggesting that I should be able to run 'apt upgrade" within the questfish env ?
Click to expand...
Yes. I solved it not by using questfish, but with virt-customize.

Modified the image with
virt-customize -a ~/debian-12-genericcloud-amd64-20240507-1740.qcow2 \
--run-command "printf 'Package: *\nPin: release n=trixie\nPin-Priority: 50' >> /etc/apt/preferences" \
--run-command "printf '\nTypes: deb deb-src\nURIs: mirror+file:///etc/apt/mirrors/debian.list\nSuites: trixie\nComponents: main' >> /etc/apt/sources.list.d/debian.sources" \
--run-command "apt-get update" \
--run-command "apt-get install cloud-init/testing -y"
 
  • Like
Reactions: justinclift and jt_telrite
Ahhh. I solved this in a different way.

For scripting we needed recently, I have it log into the new VM remotely over ssh after the first boot.

Because of the screwed up routing (ie broken default route), the VM can only be accessed by something else on the same local network as itself.

If you do have an existing server in the same local network as the VM, you can use the ProxyJump option in ssh to create connections to the vm.

Example ~/.ssh/config file:

Host new-vm
HostName 10.11.12.13
ProxyJump existing-server

Host existing-server
HostName 10.11.12.20
Click to expand...

With that in place, it's possible to ssh into the new vm to create a working netplan configuration (with the correct routing):

Bash: 
$ ssh new-vm "echo 'network: {config: disabled}' > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg"
$ ssh new-vm "cat <<EOF >/etc/netplan/50-cloud-init.yaml
network:
    version: 2
    ethernets:
        eth0:
            addresses:
            - ${VM_EXTERNAL_IP}/${VM_EXTERNAL_NETMASK}
            match:
                macaddress: ${VM_EXTERNAL_MAC}
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
                search:
                - ${DOMAIN}
            set-name: eth0
            routes:
                # IPv4 route
                - to: default
                  via: ${VM_EXTERNAL_GATEWAY}
EOF"
$ ssh new-vm "chmod 600 /etc/netplan/50-cloud-init.yaml"
$ ssh new-vm "netplan apply"

This approach has been working reliably in our dev environment for the past ~two weeks.

No flakiness whatsoever though, and I'm not really expecting any.
 
  • Like
Reactions: jt_telrite
justinclift said:
Ahhh. I solved this in a different way.

For scripting we needed recently, I have it log into the new VM remotely over ssh after the first boot.

Because of the screwed up routing (ie broken default route), the VM can only be accessed by something else on the same local network as itself.

If you do have an existing server in the same local network as the VM, you can use the ProxyJump option in ssh to create connections to the vm.

Example ~/.ssh/config file:



With that in place, it's possible to ssh into the new vm to create a working netplan configuration (with the correct routing):

Bash: 
$ ssh new-vm "echo 'network: {config: disabled}' > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg"
$ ssh new-vm "cat <<EOF >/etc/netplan/50-cloud-init.yaml
network:
    version: 2
    ethernets:
        eth0:
            addresses:
            - ${VM_EXTERNAL_IP}/${VM_EXTERNAL_NETMASK}
            match:
                macaddress: ${VM_EXTERNAL_MAC}
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
                search:
                - ${DOMAIN}
            set-name: eth0
            routes:
                # IPv4 route
                - to: default
                  via: ${VM_EXTERNAL_GATEWAY}
EOF"
$ ssh new-vm "chmod 600 /etc/netplan/50-cloud-init.yaml"
$ ssh new-vm "netplan apply"

This approach has been working reliably in our dev environment for the past ~two weeks.

No flakiness whatsoever though, and I'm not really expecting any.


I had Ansible fill in a template and push it to the vm

1 network:
2 version: 2
3 ethernets:
4 eth0:
5 ignore-carrier: true
6 addresses:
7 - {{ ip_address }}/{{ subnet_mask }}
8 match:
9 ⦙macaddress: {{ansible_facts['eth0']['macaddress']}}
10 nameservers:
11 ⦙ addresses:
12 ⦙ - 10.x.x.x
13 ⦙ - 10.x.x.x
14 ⦙ search:
15 ⦙ - example.com
16 routes:
17 ⦙ - to: default
18 ⦙ via: {{ gateway }}
19 optional: true
20 set-name: eth0
Click to expand...
 
  • Like
Reactions: justinclift
You gotta love Linux. I also have a different working combination of what's mentioned in this thread. Though I have been in IT for 25 years, I am just now learning Linux through my homelab. My goal with this was to preserve as much of the intent for ease of repetition with cloud-init and to make this easy enough for the novice. My procedure may very well be against best practices.

Assumption: You have a correctly configured, bootable Debian cloud-init template in PVE. Your only problem is the incomplete net plan.
Prerequisites: On the PVE Cloud-Init GUI - Static DNS, IP, Gateway are configured.
First: Boot the system until it is stable and at rest. The above prereqs should have allowed limited internet access. Verify this by checking that the apt update / upgrade occurred.

Now all you have to do is create the following script (once on your admin / ssh client machine.)
nano deb-cloud-init-net.sh
With contents:
Code: 
cd ~
wget http://ftp.us.debian.org/debian/pool/main/c/cloud-init/cloud-init_24.2-1_all.deb
sudo apt install ./cloud-init_24.2-1_all.deb
sudo cloud-init clean -l
rm cloud-init_24.2-1_all.deb
sudo reboot
Now you can run this on any future remote machine via ssh.
cat deb-cloud-init-net.sh | ssh username@xx.xx.xx.xx

After the reboot, cloud-init will rerun initial setup, but this time it will configure the net plan appropriately. You can verify all is well by running the following.
sudo nano /etc/netplan/50-cloud-init.yaml

The link to the new cloud-init version in my script comes from @andaga and special thanks to them.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back