[TUTORIAL] HOWTO: Scripts to make cloudbase work like cloudinit for your windows based instances
- Thread starter Geco-iT
- Start date
Hello everyone,
On my machines not working IPv4 and IPv6.
If I configure only IPv4 it works fine.
If I configure both(IPv4 and IPv6) cloud-init configures only IPv6.
Log:
Bash:2023-04-20 15:00:10.135 3620 DEBUG cloudbaseinit.utils.debiface [-] Found new interface: {'name': 'eth0', 'mac': None, 'address': '176.123.0.147', 'address6': None, 'netmask': '255.255.255.0', 'netmask6': None, 'broadcast': None, 'gateway': '176.123.0.1', 'gateway6': None, 'dnsnameservers': ['8.8.8.8']} _add_nic C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\debiface.py:104 2023-04-20 15:00:10.135 3620 DEBUG cloudbaseinit.utils.debiface [-] Found new interface: {'name': 'eth0', 'mac': None, 'address': '2001:678:6d4:4010::38', 'address6': None, 'netmask': '64', 'netmask6': None, 'broadcast': None, 'gateway': '2001:678:6d4:4010::1', 'gateway6': None, 'dnsnameservers': None} _add_nic C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\debiface.py:104 2023-04-20 15:00:10.151 3620 DEBUG cloudbaseinit.utils.classloader [-] Loading class 'cloudbaseinit.osutils.windows.WindowsUtils' load_class C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\utils\classloader.py:27 2023-04-20 15:00:10.354 3620 INFO cloudbaseinit.plugins.common.networkconfig [-] Configuring network adapter: Ethernet 2023-04-20 15:00:10.495 3620 DEBUG cloudbaseinit.osutils.windows [-] Removing existing IP address "fe80::39:63f9:9dd0:2f03%3" from adapter "Ethernet" _set_static_network_config C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:982 2023-04-20 15:00:10.510 3620 DEBUG cloudbaseinit.osutils.windows [-] Removing existing IP address "2001:678:6d4:4010::38" from adapter "Ethernet" _set_static_network_config C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:982 2023-04-20 15:00:10.557 3620 DEBUG cloudbaseinit.osutils.windows [-] Removing existing route "ff00::/8" from adapter "Ethernet" _set_static_network_config C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:991 2023-04-20 15:00:10.557 3620 DEBUG cloudbaseinit.osutils.windows [-] Removing existing route "::/0" from adapter "Ethernet" _set_static_network_config C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\lib\site-packages\cloudbaseinit\osutils\windows.py:991
This is due to cloudbase-init removing all assigned IPs from the NIC when assign a new one, which means that when assigning the IPv6 address in a dual stack scenario (comes after IPv4) the previously assigned IPv4 address gets wiped.
I'm looking for a way to fix this myself.
I tried geco-cloudbase-init and got it working to the point where the following works:
But lots of problems:
Update:
- IP-Adress manual or dhcp and dns settings working
- no errors on windows-ci clone and starting with cloudinit-drive (but it takes some time)
- unique mac-adresses (not sure if this comes from proxmox or from ci)
But lots of problems:
- If you use static for ipconfig, dns domain/dnsservers "use host settings" does not work (needs to be set manually, even though they match "host settings"
- I dont know why, but it does not create the user I put into the cloudinit drive
- It also does NOT set the password I set, even If I use a user, that was made before sysprepping
- two errors regarding logdirectory (see screenshot)
- I dont know how it was before, but the vm boots multiple times until its in the login screen
Update:
- I keep you updated, as I progress testing ci with windows
- this test was done using windows 10, I will try with windows server, as it might be different
- Tried the same with windows server 2022
- Username Password is not set, and I cant login, no matter what, seems like cloudinit changed the password to something I dont know (maybe some hardcoded stuff)
- Also getting this now: TASK ERROR: Can't use string ("#cloud-config hostname: test man"...) as a HASH ref while "strict refs" in use at /usr/share/perl5/PVE/QemuServer/Cloudinit.pm line 234. (this error occured first after restarting: pve-daemon)
Code:
2024-05-22 17:10:38.339 3844 DEBUG cloudbaseinit.plugins.common.setuserpassword [-] Generating a random user password _set_password C:\Program Files\Cloudbase Solutions\Cloudbase-Init\Python\Lib\site-packages\cloudbaseinit\plugins\common\setuserpassword.py:82
2024-05-22 17:10:38.354 3844 INFO cloudbaseinit.plugins.common.setuserpassword [-] Password succesfully updated for user Administrator
2024-05-22 17:10:38.354 3844 INFO cloudbaseinit.plugins.common.setuserpassword [-] Cannot set the password in the metadata as it is not supported by this service
Last edited:
@jsterr so... if you want to do something this weekend ;-)
I made a bug report beginning last year and @mira picked it up recently, a patch is now available,
I tried applying it to try tonight but am currently stuck doing so and out of time for this weekend mostly as my motorcycle calls for maintenance too...
I created two .patch files locally and tried to load them with patch --dry-run < cloudinit.patch but it cannot find the file to be patched,
even when I add the full path to them within the file.
The patch should fix most of the issues you mention, or explain them, you can start here:
https://bugzilla.proxmox.com/show_bug.cgi?id=4493#c5
With this patch we should be able to provision Windows VM's easily and handle the advanced configuration with other automation tools.
Edit:
Got the patch loaded as follows:
create two files with the patch contents and then patch the files
patch PVE/API2/Qemu.pm < qemu.patch
patch PVE/API2/Cloudinit.pm < cloudinit.patch
To undo the patch:
patch -R PVE/API2/Qemu.pm < qemu.patch
patch -R PVE/API2/Cloudinit.pm < cloudinit.patch
Now I have to install a Windows VM to test it
I made a bug report beginning last year and @mira picked it up recently, a patch is now available,
I tried applying it to try tonight but am currently stuck doing so and out of time for this weekend mostly as my motorcycle calls for maintenance too...
I created two .patch files locally and tried to load them with patch --dry-run < cloudinit.patch but it cannot find the file to be patched,
even when I add the full path to them within the file.
The patch should fix most of the issues you mention, or explain them, you can start here:
https://bugzilla.proxmox.com/show_bug.cgi?id=4493#c5
With this patch we should be able to provision Windows VM's easily and handle the advanced configuration with other automation tools.
Edit:
Got the patch loaded as follows:
create two files with the patch contents and then patch the files
patch PVE/API2/Qemu.pm < qemu.patch
patch PVE/API2/Cloudinit.pm < cloudinit.patch
To undo the patch:
patch -R PVE/API2/Qemu.pm < qemu.patch
patch -R PVE/API2/Cloudinit.pm < cloudinit.patch
Now I have to install a Windows VM to test it
Last edited:
No problem
Will do! Quick update in between.
Static network configuration works so far, not tested further yet.
Administrator and password setting works, but user is disabled so had to enable it, but I think that is a problem in my template.
Didn't get hostname setting working yet, it complains it is not found in meta_data.json, which is correct as it is defined in user_data.json currently.
We might need to further elaborate on (or refer) to a seperate wiki on how to create a windows cloudbase init image, it takes quite some tweaking.
The docs patch has all the options except for
In my tests the hostname was set reliably. This triggered a quick reboot on the first boot after installing cloudbase-init and adding a cloudinit disk. After the reboot all other settings (Admin user password, network, SSH keys) were set by cloudbase-init.
metadata_service: https://lists.proxmox.com/pipermail/pve-devel/2024-July/064558.htmlIn my tests the hostname was set reliably. This triggered a quick reboot on the first boot after installing cloudbase-init and adding a cloudinit disk. After the reboot all other settings (Admin user password, network, SSH keys) were set by cloudbase-init.
@mira @daviditty ,
Can anyone please provide the target files (after patch) for Proxmox 8.2.4, specifically Qemu.pm and Cloudinit.pm?
If not , Cloudinit.pm.patch and Qemu.pm.patch would also be appreciated.
Thank you for your help.
Can anyone please provide the target files (after patch) for Proxmox 8.2.4, specifically Qemu.pm and Cloudinit.pm?
If not , Cloudinit.pm.patch and Qemu.pm.patch would also be appreciated.
Thank you for your help.
@foura1512 see below, a patch file is just the patch contents within a file and applied with the command patch.
@mira not sure what I'm doing wrong with setting the hostname, it keeps complaining that it can't be found in the metadata_service file,
should I configure something special to make it look in the user_data.json for the hostname?
@mira not sure what I'm doing wrong with setting the hostname, it keeps complaining that it can't be found in the metadata_service file,
should I configure something special to make it look in the user_data.json for the hostname?
Perl:
--- /usr/share/perl5/PVE/QemuServer/Cloudinit.pm.origin
+++ /usr/share/perl5/PVE/QemuServer/Cloudinit.pm
@@ -8,6 +8,8 @@ use Digest::SHA;
use URI::Escape;
use MIME::Base64 qw(encode_base64);
use Storable qw(dclone);
+use JSON;
+use URI;
use PVE::Tools qw(run_command file_set_contents);
use PVE::Storage;
@@ -232,12 +234,23 @@ sub generate_configdrive2 {
my ($conf, $vmid, $drive, $volname, $storeid) = @_;
my ($user_data, $network_data, $meta_data, $vendor_data) = get_custom_cloudinit_files($conf);
- $user_data = cloudinit_userdata($conf, $vmid) if !defined($user_data);
- $network_data = configdrive2_network($conf) if !defined($network_data);
- $vendor_data = '' if !defined($vendor_data);
+ if (PVE::QemuServer::Helpers::windows_version($conf->{ostype})) {
+ $user_data = cloudinit_userdata($conf, $vmid) if !defined($user_data);
+ $network_data = cloudbase_network_eni($conf) if !defined($network_data);
+ $vendor_data = '' if !defined($vendor_data);
+
+ if (!defined($meta_data)) {
+ my $instance_id = cloudbase_gen_instance_id($user_data, $network_data);
+ $meta_data = cloudbase_configdrive2_metadata($instance_id, $conf);
+ }
+ } else {
+ $user_data = cloudinit_userdata($conf, $vmid) if !defined($user_data);
+ $network_data = configdrive2_network($conf) if !defined($network_data);
+ $vendor_data = '' if !defined($vendor_data);
- if (!defined($meta_data)) {
- $meta_data = configdrive2_gen_metadata($user_data, $network_data);
+ if (!defined($meta_data)) {
+ $meta_data = configdrive2_gen_metadata($user_data, $network_data);
+ }
}
# we always allocate a 4MiB disk for cloudinit and with the overhead of the ISO
@@ -254,6 +267,83 @@ sub generate_configdrive2 {
commit_cloudinit_disk($conf, $vmid, $drive, $volname, $storeid, $files, 'config-2');
}
+sub cloudbase_network_eni {
+ my ($conf) = @_;
+
+ my $content = "";
+
+ my ($searchdomains, $nameservers) = get_dns_conf($conf);
+ if ($nameservers && @$nameservers) {
+ $nameservers = join(' ', @$nameservers);
+ }
+
+ my @ifaces = grep { /^net(\d+)$/ } keys %$conf;
+ foreach my $iface (sort @ifaces) {
+ (my $id = $iface) =~ s/^net//;
+ next if !$conf->{"ipconfig$id"};
+ my $net = PVE::QemuServer::parse_ipconfig($conf->{"ipconfig$id"});
+ $id = "eth$id";
+
+ $content .="auto $id\n";
+ if ($net->{ip}) {
+ if ($net->{ip} eq 'dhcp') {
+ $content .= "iface $id inet dhcp\n";
+ } else {
+ my ($addr, $mask) = split_ip4($net->{ip});
+ $content .= "iface $id inet static\n";
+ $content .= " address $addr\n";
+ $content .= " netmask $mask\n";
+ $content .= " gateway $net->{gw}\n" if $net->{gw};
+ $content .= " dns-nameservers $nameservers\n" if $nameservers;
+ }
+ }
+ if ($net->{ip6}) {
+ if ($net->{ip6} =~ /^(auto|dhcp)$/) {
+ $content .= "iface $id inet6 $1\n";
+ } else {
+ my ($addr, $mask) = split('/', $net->{ip6});
+ $content .= "iface $id inet6 static\n";
+ $content .= " address $addr\n";
+ $content .= " netmask $mask\n";
+ $content .= " gateway $net->{gw6}\n" if $net->{gw6};
+ $content .= " dns-nameservers $nameservers\n" if $nameservers;
+ }
+ }
+ }
+
+ return $content;
+}
+
+sub cloudbase_configdrive2_metadata {
+ my ($uuid, $conf) = @_;
+ my $meta_data = {
+ uuid => $uuid,
+ 'network_config' => {
+ 'content_path' => '/content/0000',
+ },
+ };
+ $meta_data->{'admin_pass'} = $conf->{cipassword} if $conf->{cipassword};
+ if (defined(my $keys = $conf->{sshkeys})) {
+ $keys = URI::Escape::uri_unescape($keys);
+ $keys = [map { my $key = $_; chomp $key; $key } split(/\n/, $keys)];
+ $keys = [grep { /\S/ } @$keys];
+ my $i = 0;
+ foreach my $k (@$keys) {
+ $meta_data->{'public_keys'}->{"key-$i"} = $k;
+ $i++;
+ }
+ }
+ my $json = encode_json($meta_data);
+ return $json;
+}
+
+sub cloudbase_gen_instance_id {
+ my ($user, $network) = @_;
+
+ my $uuid_str = Digest::SHA::sha1_hex($user.$network);
+ return $uuid_str;
+}
+
sub generate_opennebula {
my ($conf, $vmid, $drive, $volname, $storeid) = @_;
Perl:
--- a/PVE/API2/Qemu.pm
+++ b/PVE/API2/Qemu.pm
@@ -1667,12 +1667,6 @@ my $update_vm_api = sub {
my $skip_cloud_init = extract_param($param, 'skip_cloud_init');
- if (defined(my $cipassword = $param->{cipassword})) {
- # Same logic as in cloud-init (but with the regex fixed...)
- $param->{cipassword} = PVE::Tools::encrypt_pw($cipassword)
- if $cipassword !~ /^\$(?:[156]|2[ay])(\$.+){2}/;
- }
-
my @paramarr = (); # used for log message
foreach my $key (sort keys %$param) {
my $value = $key eq 'cipassword' ? '<hidden>' : $param->{$key};
@@ -2022,6 +2016,13 @@ my $update_vm_api = sub {
my $machine_conf = PVE::QemuServer::Machine::parse_machine($param->{$opt});
PVE::QemuServer::Machine::assert_valid_machine_property($conf, $machine_conf);
$conf->{pending}->{$opt} = $param->{$opt};
+ } elsif ($opt eq 'cipassword') {
+ if (!PVE::QemuServer::Helpers::windows_version($conf->{ostype})) {
+ # Same logic as in cloud-init (but with the regex fixed...)
+ $param->{cipassword} = PVE::Tools::encrypt_pw($param->{cipassword})
+ if $param->{cipassword} !~ /^\$(?:[156]|2[ay])(\$.+){2}/;
+ }
+ $conf->{cipassword} = $param->{cipassword};
} else {
$conf->{pending}->{$opt} = $param->{$opt};Which Windows version did you try? And which cloudbase-init version? I'll try to reproduce it here.
With Windows Server 2022 and cloudbase-init 1.1.6dev20 and 1.1.7dev21 it worked out of the box.
I grabbed an install I already had and sysprepped it, mostly following the steps in the Geco-IT wiki and in the patch combined.
I used cloudbase-init 1.1.6dev20
And Windows 10 Pro 22H2
I used cloudbase-init 1.1.6dev20
And Windows 10 Pro 22H2
root@CPN:~# patch --force --forward --backup -p0 --directory / --input "/root/Cloudinit.pm.patch"
patching file /usr/share/perl5/PVE/QemuServer/Cloudinit.pm
Hunk #2 FAILED at 234.
1 out of 3 hunks FAILED -- saving rejects to file /usr/share/perl5/PVE/QemuServer/Cloudinit.pm.rej
root@CPN:~# cat /usr/share/perl5/PVE/QemuServer/Cloudinit.pm.rej
--- /usr/share/perl5/PVE/QemuServer/Cloudinit.pm.origin
+++ /usr/share/perl5/PVE/QemuServer/Cloudinit.pm
@@ -234,12 +236,23 @@ sub generate_configdrive2 {
my ($conf, $vmid, $drive, $volname, $storeid) = @_;
my ($user_data, $network_data, $meta_data, $vendor_data) = get_custom_cloudinit_files($conf);
- $user_data = cloudinit_userdata($conf, $vmid) if !defined($user_data);
- $network_data = configdrive2_network($conf) if !defined($network_data);
- $vendor_data = '' if !defined($vendor_data);
+ if (PVE::QemuServer::Helpers::windows_version($conf->{ostype})) {
+ $user_data = cloudinit_userdata($conf, $vmid) if !defined($user_data);
+ $network_data = cloudbase_network_eni($conf) if !defined($network_data);
+ $vendor_data = '' if !defined($vendor_data);
+
+ if (!defined($meta_data)) {
+ my $instance_id = cloudbase_gen_instance_id($user_data, $network_data);
+ $meta_data = cloudbase_configdrive2_metadata($instance_id, $conf);
+ }
+ } else {
+ $user_data = cloudinit_userdata($conf, $vmid) if !defined($user_data);
+ $network_data = configdrive2_network($conf) if !defined($network_data);
+ $vendor_data = '' if !defined($vendor_data);
- if (!defined($meta_data)) {
- $meta_data = configdrive2_gen_metadata($user_data, $network_data);
+ if (!defined($meta_data)) {
+ $meta_data = configdrive2_gen_metadata($user_data, $network_data);
+ }
}
# we always allocate a 4MiB disk for cloudinit and with the overhead of the ISO
@daviditty , Could you please provide the patched Cloudinit.pm and Qemu.pm that works with Prox 8.2.4 ?
Thanks.
Hi @foura1512,
Not sure, I applied it to PVE 8.2.2, as I run the enterprise repo in my homelab,
but I think the patch should work on no-subscription too, not sure what goes wrong though from the rejected output you mentioned.
That said, if we test it succesfully it gets merged upstream and no more manual patching is involved in the future, so it is important to test it,
if needed I can switch to no-subscription, but I'm leaving for holiday soon so will not be able to test further in the coming two weeks.
Not sure, I applied it to PVE 8.2.2, as I run the enterprise repo in my homelab,
but I think the patch should work on no-subscription too, not sure what goes wrong though from the rejected output you mentioned.
That said, if we test it succesfully it gets merged upstream and no more manual patching is involved in the future, so it is important to test it,
if needed I can switch to no-subscription, but I'm leaving for holiday soon so will not be able to test further in the coming two weeks.
I've tested no-sysprep and sysprep Win 10 with cloudbase-init. In both cases the hostname was set correctly (this required a reboot afterwards).
What did not work was renaming the Admin user with the sysprep image. It was disabled again, kept the name `Administrator` and the password was not set.
I'll look into this a bit more.
What did not work was renaming the Admin user with the sysprep image. It was disabled again, kept the name `Administrator` and the password was not set.
I'll look into this a bit more.
There's a v2 of the patches on the pve-devel list: https://lists.proxmox.com/pipermail/pve-devel/2024-July/064958.html
The qemu-server patch is unchanged, but the docs patch received a few additions including `sysprep`: https://lists.proxmox.com/pipermail/pve-devel/2024-July/064959.html
The qemu-server patch is unchanged, but the docs patch received a few additions including `sysprep`: https://lists.proxmox.com/pipermail/pve-devel/2024-July/064959.html
I too am getting issues applying these patches against PVE 8.2.4 from the no-subscription repository. The qemu API patch seems to think it's already been applied, and the cloud-init patch fails on the first hunk. Looking at the code from each of these files at the areas where the patch should be applied doesn't really give much in the way of an indication about why the patch would fail to apply. Any ideas here, @mira?
Looks like the v2 patch worked for me, the line numbers in the v1 patch for the API and Cloudinit endpoints appear slightly different.
In my testing, both the IP/DNS setting and the password application for the Administrator work well even without running one of the development versions of Cloudbase-init. My Windows Images use the latest release, 1.5, and set the password just fine.
When you say made available, do you mean they'll be pushed with a packaged update, rather than needing to patch?
Thanks for your work on this, @mira!
In my testing, both the IP/DNS setting and the password application for the Administrator work well even without running one of the development versions of Cloudbase-init. My Windows Images use the latest release, 1.5, and set the password just fine.
When you say made available, do you mean they'll be pushed with a packaged update, rather than needing to patch?
Thanks for your work on this, @mira!