Pve-kernel amd pstate not working (the _CPC object is not present)

[Grouack]

Hi everyone,

That some time I'm trying to change the scaling governor from performance to on demand. Every time I change it the server crash.
I have been searching some information about this and found that the amd pstate for zen2 and zen3 it's not manage correctly by the kernel.

by running dmesg | grep CPC I get the following result:

[    0.572740] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled

I have already check the bios and everything is activated. I have also tried to add amd_pstate.shared_mem=1 to grub but without result.

On this web site a patch is proposed to fix the amd pstate: https://bugzilla.kernel.org/show_bug.cgi?id=218171

So I tried to build a kernel using this repository https://github.com/proxmox/pve-kernel and adding the patch inside the folder patches/kernel. But I don't arrive to initialize submodules as they are not found.

That's my first time working and trying to build/patch a kernel, I'm a little bit lost and don't find a way to do it.
If someone have some advise or a tutorial to do it and be able to try this patch?

material and proxmox version:
amd thread ripper 3960x
proxmox 8.1.4
kernel 6.5.13-1-pve

 

[mikeone]

very interesting did you make any advancement on this ?

 

[Grouack]

Hi mikeone.
Thanks for the interest on the subject but I didn't really advance.
I have arrived to correct the problem by building a kernel with the patch.
With cpupower I arrive to see the driver set to amd-pstate ( if I remember correctly ) but it was not working properly. So I have let it.

I have just check with the new kernel provided by proxmox (6.8.4-3pve). It's look like the previous log is corrected because I get the following line in dmesg :

[    0.555910] amd_pstate: driver load is disabled, boot with specific mode to enable this

So I should retry to set the configuration in the bios and see if it's working now.

Until now I have used the schedutil policy that allow cores to go down at 2200MHz.
Some time I get the server which freeze, but I don't know if it come from this change, and I get this message in dmesg for several cores that I don't really understand:

[75723.686233] watchdog: BUG: soft lockup - CPU#8 stuck for 63315s! [pvestatd:1728]
[75723.686972] Modules linked in: cfg80211 veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter nf_tables bonding tls softdog nfnetlink_log sunrpc binfmt_misc nfnetlink ipmi_ssif intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd joydev input_leds hid_generic usbkbd usbmouse kvm snd_hda_codec_hdmi crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel snd_hda_intel sha256_ssse3 snd_intel_dspcfg usbhid sha1_ssse3 cdc_ether snd_intel_sdw_acpi hid usbnet aesni_intel mii snd_hda_codec crypto_simd snd_hda_core cryptd snd_hwdep snd_pcm ast snd_timer cp210x rapl wmi_bmof serio_raw pcspkr snd i2c_algo_bit usbserial ccp acpi_ipmi soundcore k10temp ipmi_si ipmi_devintf ipmi_msghandler mac_hid uas usb_storage zfs(PO) spl(O) vhost_net vhost vhost_iotlb tap vfio_pci vfio_pci_core irqbypass vfio_iommu_type1 vfio iommufd efi_pstore dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c xhci_pci xhci_pci_renesas nvme nvme_core ahci
[75723.687146]  i40e xhci_hcd psmouse igc crc32_pclmul libahci nvme_auth i2c_piix4 wmi
[75723.692841] CPU: 8 PID: 1728 Comm: pvestatd Tainted: P      D    O L     6.8.4-3-pve #1
[75723.693439] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./TRX40D8-2N2T, BIOS L1.19E 11/03/2021
[75723.694048] RIP: 0010:get_cpu_sleep_time_us.part.0+0x4c/0xb0
[75723.694665] Code: 89 c6 4d 85 ed 74 23 48 b8 cf f7 53 e3 a5 9b c4 20 48 f7 ee 48 89 f0 48 c1 f8 3f 48 c1 fa 07 48 29 c2 49 89 55 00 eb 02 f3 90 <8b> 53 70 f6 c2 01 75 f6 f6 03 04 74 05 45 84 e4 75 3a 49 8b 0e 8b
[75723.695935] RSP: 0018:ffffb72bc7897c80 EFLAGS: 00000202
[75723.696583] RAX: 0000070a631b0a61 RBX: ffffa0e13b425000 RCX: 0000000000000000
[75723.697242] RDX: 0000000001bcf7b3 RSI: 0000070a631b0a61 RDI: 0000000000000000
[75723.697901] RBP: ffffb72bc7897ca0 R08: 0000000000000000 R09: 0000000000000001
[75723.698562] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[75723.699226] R13: 0000000000000000 R14: ffffa0e13b4250c8 R15: 0000000000000004
[75723.699889] FS:  00007f1f6588f740(0000) GS:ffffa0e13b600000(0000) knlGS:0000000000000000
[75723.700562] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[75723.701235] CR2: 000056b235c7edb0 CR3: 00000001258fe000 CR4: 0000000000350ef0
[75723.701917] Call Trace:
[75723.702595]  <IRQ>
[75723.703270]  ? show_regs+0x6d/0x80
[75723.703950]  ? watchdog_timer_fn+0x206/0x290
[75723.704631]  ? __pfx_watchdog_timer_fn+0x10/0x10
[75723.705309]  ? __hrtimer_run_queues+0x108/0x280
[75723.705986]  ? srso_return_thunk+0x5/0x5f
[75723.706665]  ? hrtimer_interrupt+0xf6/0x250
[75723.707337]  ? __sysvec_apic_timer_interrupt+0x51/0x150
[75723.708011]  ? sysvec_apic_timer_interrupt+0x8d/0xd0
[75723.708686]  </IRQ>
[75723.709354]  <TASK>
[75723.710019]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
[75723.710695]  ? get_cpu_sleep_time_us.part.0+0x4c/0xb0
[75723.711345]  get_cpu_idle_time_us+0x4d/0x80
[75723.711971]  get_idle_time+0x35/0x50
[75723.712589]  uptime_proc_show+0x58/0x170
[75723.713199]  ? srso_return_thunk+0x5/0x5f
[75723.713806]  ? __kmalloc_node+0x41c/0x430
[75723.714416]  seq_read_iter+0x135/0x4a0
[75723.715021]  proc_reg_read_iter+0x32/0x90
[75723.715621]  vfs_read+0x258/0x390
[75723.716220]  ksys_read+0x73/0x100
[75723.716805]  __x64_sys_read+0x19/0x30
[75723.717377]  x64_sys_call+0x23f0/0x24b0
[75723.717938]  do_syscall_64+0x81/0x170
[75723.718486]  ? do_syscall_64+0x8d/0x170
[75723.719021]  ? exc_page_fault+0x94/0x1b0
[75723.719547]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[75723.720062] RIP: 0033:0x7f1f659c619d
[75723.720582] Code: 31 c0 e9 c6 fe ff ff 50 48 8d 3d 66 54 0a 00 e8 49 ff 01 00 66 0f 1f 84 00 00 00 00 00 80 3d 41 24 0e 00 00 74 17 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 5b c3 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec
[75723.721621] RSP: 002b:00007ffd63bc8cc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[75723.722149] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1f659c619d
[75723.722680] RDX: 0000000000002000 RSI: 000056b23a4c94e0 RDI: 0000000000000008
[75723.723191] RBP: 0000000000002000 R08: 0000000000000000 R09: 00007f1f65aa0cd0
[75723.723695] R10: 000056b23a4c94e0 R11: 0000000000000246 R12: 000056b23a4c94e0
[75723.724185] R13: 000056b2348dc2a0 R14: 0000000000000008 R15: 000056b23a4c6d90
[75723.724662]  </TASK>

edit:

I have try to set it up again it's look like now it's recognize but I'm still having trouble to make It work.
With CPPC enable and CPPC preferred core enable in bios

I get the following error in dmesg:

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.8.4-3-pve root=/dev/mapper/pve-root ro quiet iommu=pt initcall_blacklist=sysfb_init amd_pstate=active
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.4-3-pve root=/dev/mapper/pve-root ro quiet iommu=pt initcall_blacklist=sysfb_init amd_pstate=active
[    1.040586] amd_pstate: failed to register with return -19

also nominal and lowest frequency return 0:

root@eternium:~# cat  /sys/devices/system/cpu/cpu0/acpi_cppc/nominal_freq
0
root@eternium:~# cat  /sys/devices/system/cpu/cpu0/acpi_cppc/lowest_freq
0

cpupower frequency-info
analyzing CPU 0:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 0
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 2.20 GHz - 4.57 GHz
  available frequency steps:  3.80 GHz, 2.80 GHz, 2.20 GHz
  available cpufreq governors: conservative ondemand userspace powersave performance schedutil
  current policy: frequency should be within 2.20 GHz and 3.80 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency: 2.80 GHz (asserted by call to hardware)
  boost state support:
    Supported: yes
    Active: yes
    Boost States: 0
    Total States: 3
    Pstate-P0:  3800MHz
    Pstate-P1:  2800MHz
    Pstate-P2:  2200MHz

In this state, I see CPU going down to 500Mhz but I get poor performance for windows VM and moonlight and core seems to don't wake up properly when needed.

withCPPC preferred core disable in bios it's seems a little bite better but still poor performance for windows VM and moonlight

 

[Ramalama]

Im late to the party, but some things:

No Patch will work if:
If you enabled CPPC in Bios and get in dmesg:
amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled
And:
lscpu | grep -i cppc outputs nothing

1. You can disable x2apic (set xapic mode in bios), that will usually fix the dmesg error and you'll be able to load amd-pstate driver.
--> But its just that, youll be able to load it, but it wont work correctly, since your mainboard vendor fu...up acpi tables.

2. If lscpu outputs at least cppc, you are in halfway luck, then a patched kernel will work with amd-pstate as it should. (Proxmox kernel 6.8.4 wont)
Mainboards with broken acpi tables, asus/asrock/gigabyte (they don't even have a clue and no experts, what even acpi is), won't get working amd-pstate in any way. I think with kernel 6.9 those pstate patches will be available anyway. Or if Proxmox includes the pstate patches or ubuntu does...

There are ways to fix acpi tables and inject them, with a bootloader like Clover for example. It's not very hard.
Every hackintosh user with halfway good knowledge does that. So fixing _CPC with x2apic is not really hard.

3. if you get the error in dmesg, even with disabling x2apic (set to xapic) and you get no cppc in lscpu, you won't get amd-pstate working and there is no chance you ever will. Since the one or other is a requirement.
(Again there are ways to get the driver itself loaded, but its just stupid then)

PS:
4. forgot to mention:

wget http://ftp.debian.org/debian/pool/main/l/linux/libcpupower1_6.7.12-1~bpo12+1_amd64.deb
wget http://ftp.debian.org/debian/pool/main/l/linux/linux-cpupower_6.7.12-1~bpo12+1_amd64.deb
dpkg -i libcpupower1_6.7.12-1~bpo12+1_amd64.deb linux-cpupower_6.7.12-1~bpo12+1_amd64.deb

Update cpupower to a halfway actual version, because the debian 12 version is brutally outdated and doesn't show anything usefull for amd-pstate.

Cheers

 

[Grouack]

Hi Ramalama,
Never to late for find a solution

I have let the idea to do a patch since it was a little bit complicated even if I arrive to correct the _CPC object error.

Now with the kernel 6.8 this as been corrected. The error is explain in the linux patch that have been add to the kernel: https://bugzilla.kernel.org/show_bug.cgi?id=218171 (cf Gino Badouri comment)
When you read the exchange it was effectively a problem of reading the table.

But people arrive to make it work with thread reaper 3960x with passive, guided and active

The last try that I have done in dmesg no error for cppc in dmesg:

root@eternium:~# dmesg | grep pstate
[    0.554208] amd_pstate: driver load is disabled, boot with specific mode to enable this

lspcu | grep -i cppc output nothing

I have arrive to see the amd pstate loaded and used with cpupower but it was not working properly and I think is due to the nominal frequency

But what I don't get it is that the person who report the bug confirm after the patch that amd pstate work with threadripper 3960x and show the result.

when I try to print these informations:

echo "=============nominal freq==============================="
cat /sys/devices/system/cpu/cpu*/acpi_cppc/nominal_freq

echo "=============norminal perf==============================="
cat /sys/devices/system/cpu/cpu*/acpi_cppc/nominal_perf

echo "=============highest_perf perf==============================="
cat /sys/devices/system/cpu/cpu*/acpi_cppc/highest_perf

echo "=============lowest_nonlinear_perf perf==============================="
cat /sys/devices/system/cpu/cpu*/acpi_cppc/lowest_nonlinear_perf

echo "=============lowest perf==============================="
cat /sys/devices/system/cpu/cpu*/acpi_cppc/lowest_perf

I get the following result with nominal frequency at 0:

=============norminal freq===============================
0
=============norminal perf===============================
138
=============highest_perf perf===============================
231
=============lowest_nonlinear_perf perf===============================
81
=============lowest perf===============================
21

If we compare this result to my previous test (in the edit) we see that the lowest perf is set and before was at 0 (I don't what change this)

and when I try with the mode active I get th e following error in dmesg:

[    1.040586] amd_pstate: failed to register with return -19

I can try to look for x2apic option if it fix the return code.

 

[Grouack]

I have tried again this morning. disabling in the bios x2apic ( the majority of the other option are in auto juste enable CPPC)
installed newer cpu power and set pstate mode to active.
at boot dmesg return :

[    1.677664] amd_pstate: failed to register with return -19

cpupower return not using amd pstate and set the governor policy to performance to try:

  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 21
  CPUs which need to have their frequency coordinated by software: 21
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 2.20 GHz - 4.57 GHz
  available frequency steps:  3.80 GHz, 2.80 GHz, 2.20 GHz
  available cpufreq governors: conservative ondemand userspace powersave performance schedutil
  current policy: frequency should be within 2.20 GHz and 3.80 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency: 3.80 GHz (asserted by call to hardware)
  boost state support:
    Supported: yes
    Active: no
    Boost States: 0
    Total States: 3
    Pstate-P0:  3800MHz
    Pstate-P1:  2800MHz
    Pstate-P2:  2200MHz

cpu frequency go down to 500MHz that seems pretty nice as without amd_pstate in grub and schedutil cpu is always at 2200MHz
The problem is that now I get poor performance especially in a windows gaming machine (sunshine and moonlight).
FPS pass from 120 to 20. By looking at cpu frequency all core are at 500MHz only few of them jump directly to 3800MHz.

same result with ondemand policy.

So I don't really get it's look like it's half working but frequency are not set correctly according to the load. I'm thinking that it's due to nominal frequency reporting 0.

ps:
I have forgot that this week I get a new bios from asrock rack to correct and be able to enable resizable bar perhaps it have correct the lowest perf which was previously at 0

 

[Ramalama]

amd_pstate: failed to register with return -19

With amd_pstate=active ? But passive/guided (which is the same) works?
Thats a Completely new issue i never seen before.

This patch is not part of our PVE 6.8.4 Kernel:
https://bugzilla.kernel.org/show_bug.cgi?id=218171
I have 2 Systems, one Ryzen 5800x and 2x Genoa Servers with 9374f as CPU.
The Genoa Server was an absolute breeze, to enable amd-pstate-epp
The Ryzen 5800x was an absolute pain, took me a half day to find out the issue with x2apic, but amd-pstate-epp works now.

On the Ryzen System, i changed a TON in the bios settings, setted Custom TDP/PPT, df tables generation/c1 state tables generation, ton of things i already forgot, but i wanted to find out what the issue is.
Nothing made really a difference tbh. Just x2apic will break on that Asrock Mobo the _CPC tables. So i have to use or downgrade to xapic.

Then i searched around, and found that other people on asus and gigabyte have the exact same issues. Bootet my system with x2apic, dumped the acpi tables. Then booted again with xapic, and dumped.
And the tables are exactly the same, but there are functions inside, that refer to the mode, which seems to be broken. However almost all acpi tables are absolutely broken in the decompiled dump. And i don't think that it comes from decompiling.

However, back to your issue, i think at the moment you have to stick simply with good old acpi_cpufreq and try at another point again (with a newer kernel) the amd-pstate-epp driver.

https://github.com/Ramalama2/proxmox-edge-kernel
Watch my repo too, as soon as OpenZFS releases 2.2.5 (with 6.9 support), i will be able to compile the 6.9 kernel, and update my repo.

Cheers

 

[Ramalama]

About the new Bios:

Dont use Resizable Bar or Above 4G, with Proxmox.
No matter if you passthrough your GPU or use it for encoding, it will usually have the reset bug.
And you don't gain anything from on Proxmox anyway.

Cheers

 

[Grouack]

I will dig a little bit for pstate and share my result during the day with passive and guided.

For resizable bar, I wanted to give a try because I'm getting an intel arc 380 to transcode and apparently need it. I didn' hear about the reset bug but for now didn't get any trouble with an rtx3090 passthrough a windows 11 machine.

 

[Ramalama]

I will dig a little bit for pstate and share my result during the day with passive and guided.

For resizable bar, I wanted to give a try because I'm getting an intel arc 380 to transcode and apparently need it. I didn' hear about the reset bug but for now didn't get any trouble with an rtx3090 passthrough a windows 11 machine.

Im using an Arc380 (for transcoding) too, you don't need resizable bar, especially not for transcoding.

 

[Grouack]

Apparently the pve kernel 6.8.4 is based on ubuntu 24.04 node which is base on last linux kernel 6.8.
If this patch is not included how could I know in which kernel it have been added to give a new try when available?

For passive/guided mode:
- cpupower return usage of amd pstate but no frequency

analyzing CPU 38:
  driver: amd-pstate
  CPUs which run at the same hardware frequency: 38
  CPUs which need to have their frequency coordinated by software: 38
  maximum transition latency: 20.0 us
Not Available
  available cpufreq governors: conservative ondemand userspace powersave performance schedutil
  Unable to determine current policy
  current CPU frequency: Unable to call hardware
  current CPU frequency: 4.30 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: no
    AMD PSTATE Highest Performance: 166. Maximum Frequency: .
    AMD PSTATE Nominal Performance: 138. Nominal Frequency: .
    AMD PSTATE Lowest Non-linear Performance: 81. Lowest Non-linear Frequency: .
    AMD PSTATE Lowest Performance: 21. Lowest Frequency: .

- all cores are at a maximum speed (3800MHz)

thanks for the information about the intel arc 380.
I will keep an eyes on your repo too

 

[Ramalama]

I never tryed passive/guided tbh, so i don't know how the output should look like.

I can just say that with active (epp), i do have all frequencies, except the "current CPU frequency", i get the unable to call hardware too.
But for passive i have no clue.

Did you tryed to set the sheduler to ondemand?

 

[Grouack]

I have looked on other website and normally the frequency should be displayed.
On both I tried schedutil and ondemand no change at all.
Even with acpi_cpufreq and ondemand server crash only schedutil work.

Probably some tuning is needed in the bios but I'm really not expert on that point ^^

The acpi_cpufreq with schedutil it's always this and as you says I should perhaps wait an update of the kernel and give a try again.

I can also always ask asrockrack if they have a clue but I'm not quite sure is worth it. they have half abandon the motherboard that I have (trx40d8-2n2t)

 

[Ramalama]

I have looked on other website and normally the frequency should be displayed.
On both I tried schedutil and ondemand no change at all.
Even with acpi_cpufreq and ondemand server crash only schedutil work.

Probably some tuning is needed in the bios but I'm really not expert on that point ^^

The acpi_cpufreq with schedutil it's always this and as you says I should perhaps wait an update of the kernel and give a try again.

I can also always ask asrockrack if they have a clue but I'm not quite sure is worth it. they have half abandon the motherboard that I have (trx40d8-2n2t)

Forget asrock, i have the x570d4i-2t.
They have even less clue about the bios as you, believe me.

But at least they will give an Answer, im curious if they will just tell you "Reset your BIOS Settings to Optimized Defaults" or something different
But i never seen anyone, who have even broken acpi-cpufreq scaling driver.
Well it looks like your actual hope is to get amd-pstate working in 6.9.

But if you have time, you could at least play with bios settings, especially with x2apic/xapic, df states, c states, apbdis (should be 0), and anything else that is related to c states, like global c state support etc...
cppc and cppc preferred core, is up to you, if you want amd-pstate or not, but i don't think there is any harm to leave them both enabled.
preferred core support, will start working once the driver supports it, but atm moment the driver will ignore that, so in short, you can leave it enabled in bios, there shouldn't be any difference.
Precision boost overdrive should be enabled, eco mode should be disabled,
core-boost (i forgot the exact name), you should try disabled and enabled, i have it enabled, but maybe it breaks something, dunno...

Thats all i can help you with.
Disable autostart on all VM's, because you will reboot a shitton of times during bios changes... To see what effect they have,

On every reboot execute once:
dmesg --level err
dmesg --level warn
cpupower frequency-info

and check simply what changes there, maybe you get lucky.
Test the default "optimized" bios settings too, to be sure.
And just to be sure, we need with kernel 6.8 only "amd_pstate=active/passive/guided", no shared mem, no initcall acpi_cpufreq blocking, nothing.
add it to 2 places: /etc/kernel/cmdline & /etc/default/grub
update-initramfs -u -k all
update-grub2
proxmox-boot-tool refresh

I just do both, because on some systems here im myself confused, if they use grub or proxmox-boot, especially with uefi ones lol xD
So i do always both.
But im checking anyway afterwards always /proc/cmdline, if my changes were applied.

However, good luck!
Cheers

 

[Grouack]

Forget asrock, i have the x570d4i-2t.
They have even less clue about the bios as you, believe me.

But at least they will give an Answer, im curious if they will just tell you "Reset your BIOS Settings to Optimized Defaults" or something different
But i never seen anyone, who have even broken acpi-cpufreq scaling driver.
Well it looks like your actual hope is to get amd-pstate working in 6.9.

But if you have time, you could at least play with bios settings, especially with x2apic/xapic, df states, c states, apbdis (should be 0), and anything else that is related to c states, like global c state support etc...
cppc and cppc preferred core, is up to you, if you want amd-pstate or not, but i don't think there is any harm to leave them both enabled.
preferred core support, will start working once the driver supports it, but atm moment the driver will ignore that, so in short, you can leave it enabled in bios, there shouldn't be any difference.
Precision boost overdrive should be enabled, eco mode should be disabled,
core-boost (i forgot the exact name), you should try disabled and enabled, i have it enabled, but maybe it breaks something, dunno...

Thats all i can help you with.
Disable autostart on all VM's, because you will reboot a shitton of times during bios changes... To see what effect they have,

On every reboot execute once:
dmesg --level err
dmesg --level warn
cpupower frequency-info

and check simply what changes there, maybe you get lucky.
Test the default "optimized" bios settings too, to be sure.
And just to be sure, we need with kernel 6.8 only "amd_pstate=active/passive/guided", no shared mem, no initcall acpi_cpufreq blocking, nothing.
add it to 2 places: /etc/kernel/cmdline & /etc/default/grub
update-initramfs -u -k all
update-grub2
proxmox-boot-tool refresh

I just do both, because on some systems here im myself confused, if they use grub or proxmox-boot, especially with uefi ones lol xD
So i do always both.
But im chacking anyway afterwards always /proc/cmdline, if my changes were applied.

However, good luck!
Cheers

Thanks a lot for the time and your advice.
I will take look at all of this when I have a little bit of time and if I found a solution I will put it there