Problems with TwinCAT, Proxmox and isolated cores on Windows

[flrnwgnr]

Hello,

I am experiencing some weird problems with TwinCAT (TC) in a Windows-VM. Because these problems do not occur with a different hypervisor (vmware ESX or HyperV), I suspect that the problem is related to Proxmox, KVM and/or their implementation of nesting. But first things first:

I am currently in the process of installing the TwinCAT runtime in a Windows 10-VM. For those who are not familiar with TwinCAT: "The Windows Control and Automation Technology", TwinCAT, is the leading software for a soft-PLC that makes Windows real-time capable through a kernel extension. The most important requirement for 64-bit systems is access to the virtualization extension "VT-x".

There are then two modes for operating TwinCAT: "shared core" or "isolated core". With "shared core", TC shares a CPU core with Windows and both alternate in computing times.
In "isolated core"-mode, Windows is configured to use 1 core less than the CPU provides. This "isolated" core can then be used exclusively by TwinCAT (e.g. 4-core CPU: Windows uses 3 Cores and TwinCAT uses the fourth).
Also, this method is the only way to run TwinCAT in a 64-bit VM; "shared core" is only available on 32-bit VMs.


However, this is exactly where the problem lies:

In a 64-bit Windows VM with nesting enabled and operation mode "isolated core" (4 cores in Proxmox, 3 are for Windows and one for TwinCAT) the PLC can be started and runs without problems. The CPU load of all cores increases from 16% in idle mode to about 19% (visible in Proxmox).

But when I stop the PLC, the CPU usage jumps up by exactly 25% - the isolated core seems to be at 100% load. After that, TC cannot be started anymore and reports "startup of isolated core failed". The core hangs and remains in this state until the VM is rebooted.

However, this identical system configuration (64-Bit, Win10S, nesting enabled) works fine if another hypervisor, e.g. vmware ESX, HyperV or VirtualBox, is used - the PLC can be started and stopped multiple times.


Has anyone here had a similar problem? Is it possible that there is a bug in Proxmox or KVM, especially when a core is not used by Windows?

Thanks!

__

VM-Configuration:

(Most important is the "args"-line: TwinCAT doesn't want to start without it, even if the cpu-config has the same flags)

agent: 1
args: -cpu 'host,+pcid,+spec-ctrl,+ssbd'
boot: order=ide0;net0
cores: 3
cpu: host,flags=+pcid;+spec-ctrl;+ssbd
efidisk0: local-lvm:vm-100-disk-1,size=4M
ide0: local-lvm:vm-100-disk-0,size=32G
kvm: 1
machine: pc-i440fx-6.0
memory: 8192
name: Win10Eval-1
net0: virtio=0A:B1:A8:80:8F:ED,bridge=vmbr1,firewall=1
numa: 1
ostype: win10
parent: Ausgangsbedingungen
scsihw: virtio-scsi-pci
smbios1: uuid=c01feef2-02...
sockets: 1
vmgenid: 50f30ea3...

 

[0111]

Greetings,

Full disclosure I am not actually using Proxmox or know much about it. I had like issue on my hypervisor trying to run TwinCAT in Windows 10 Pro guest. I am running a host system on Debian 10 something. Hypervisor is kvm+qemu. I have a Intel core i5 k series so I can not do PCI pass-through. 32GB RAM DDR3. I am using virt-manager to handle management of xml files. I use spice server for my graphics and input and output control of guests. I have 15 guest installed on this hypervisor. A mixture of Windows 2000, Debian 10, Debian 11, Ubuntu something, Windows 10 Pro. Not all of them running at once. The boot disks are backed by two NVME PCI M.2 drives using LVM. Data storage is backed by 3 enterprise SATA drives forming a BTRFS raid 5 volume on host.

My issue was originally that when switching from config mode to run mode in Windows 10 Pro guest, TwinCAT would freeze and then the guest would reboot (crash). Never got it working until today. I found out that a Security system called Blue Iris would also crash on fresh install on Windows 10 Pro. There forum posts pointed me to passing the host a kernel option to turn off MSRS using for example "cat 'options kvm ignore_msrs=1' >> /etc/modeprobe.d/kvm.conf"

From my loose understanding of MSRS. The situation seems to be that Blue Iris and TwinCAT are asking for CPU specific registers even though they don't actually need them. By ignoring the requests the guest just continues to run and Blue Iris and TwinCAT carry on there marry way. Without this disabled the requests are answered and "maybe" QEMU injects the info into Windows 10 and Windows 10 freaks out that its CPU changed all of a sudden hence the crashing.

I don't know if this has actually solved the issue for me or how reliable it is. I just tested with a Hello World simple TwinCAT project and it did not crash. I was able to login and run the code. and start and stop execution and write variable and then logout.

Main.tc

binit : BOOL := 1;

IF binit THEN
binit := 0;
END_IF

That is all.

 

[flrnwgnr]

Hi, thanks for your reply!

Since you were able to start TwinCAT and your hypervisor is kvm+qemu (like proxmox), can you do a little test?

I just want to know if you are able to start TwinCAT "twice", because the second attempt always fails in my environment.

For example:

1. Start/Reboot the VM (TwinCAT-Icon should be blue/in config mode)
2. "Activate Configuration" (TwinCAT-Icon should become green/RUN-Mode)
3. Switch back to Config-Mode (Icon turns blue again)
4. "Activate Configuration" again (ideally the icon turns green again)

In my case the second attempt fails and I have to reboot the VM to set TwinCAT to RUN-Mode again. Is this also the case with your installation?

Thank you!

 

[0111]

Ok, can you confirm that ignore_msrs=1 had a positive impact?

And for the test:
1. I just rebooted the Windows VM and I reopen the project I am currently working on. Yes the the Icon is blue. Actually blue before I open Visual Studio 2017. (That I forgot to include in the last post. I have not had luck with Visual Studio 2019, install/ingratiation dose not complete for me)
2. I activate my configuration and yes the runtime goes into a green/run-mode state. I have the checkbox for boot configuration unchecked. So on reboot there was nothing running. In run mode there now is however note that I click the play button to actually start execution as its not auto run at boot.
3. I login to the PLC to start execution and then logout.
4. I select the blue icon for going back to config mode and it dose. My CPU utilization in TwinCat runtime goes form 1% to 0%
5. I select to activate the configuration and it dose go to green again.
6. I login again to start execution and logout when I am dose looking at variables.

So yes I can bouce back and forth no problem. But note I am not utilizing the boot configuration. I don't know if that matters. I am activly developing a application so I am bouncing back and forth all the time with no issues. Unless we count me using pointers and Interfaces that allocation write memory I didn't mean to and then yes the runtime crashed/halts and I have to go into config mode once I want to activate configuration again.

 

[flrnwgnr]

Ok, can you confirm that ignore_msrs=1 had a positive impact?

Not really - i can still start TwinCAT only once.. Are you using Windows 64-Bit or 32-Bit?

and it dose go to green again

But this is interesting! I tried it with your project but i still get an error after starting TwinCAT the second time. Can you post your VM-Config? Might be that proxmox and not KVM is the culprit..

 

[0111]

</os>
<features>
<acpi/>
<apic/>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="partial"/>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2"/>
<source file="/mnt/vm/machines/development_plc_twincat.qcow2"/>
<target dev="sda" bus="sata"/>
<address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pci-root"/>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
</controller>
<interface type="network">
<mac address="52:54:00:d7:49:ba"/>
<source network="default"/>
<model type="rtl8139"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
</interface>
<input type="tablet" bus="usb">
<address type="usb" bus="0" port="1"/>
</input>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<graphics type="spice" autoport="yes">
<listen type="address"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
</sound>
<video>
<model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
</video>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
</memballoon>
</devices>
</domain>

 

[aksu]

Hello,

I'm interested to know did you manage to get TwinCat3 working in the proxmox environment?

I'm trying to get my own virtualization environment to work, but when I try to start the runtime I get following error from the Twincat:
"Init4\RTime: Start Interrupt: Ticker started >> AdsError: 4116 (0x1014, RTIME: enter real-time mode fails!)"



Have you had same kind of issues ? Any ideas how to get rid of this?

 

[flrnwgnr]

Hello,

I'm interested to know did you manage to get TwinCat3 working in the proxmox environment?

I'm trying to get my own virtualization environment to work, but when I try to start the runtime I get following error from the Twincat:
"Init4\RTime: Start Interrupt: Ticker started >> AdsError: 4116 (0x1014, RTIME: enter real-time mode fails!)"

View attachment 35803

Have you had same kind of issues ? Any ideas how to get rid of this?

Try to use an isolated core within your TwinCAT-Project/VM.

 

[aksu]

I have tried this in two different Windows 10 distributions/installations, and now I managed to get the TC3 environment to work on one of these (after setting the isolated core for the TC3 project).

Runtime goes up and everything works nicely after booting the Windows BUT if I try to restart the runtime I get two same kind of errors:


But if I boot the machine, runtime starts and runs again nicely.

Any ideas how to get this work "as it should"?

 

[flrnwgnr]

I have tried this in two different Windows 10 distributions/installations, and now I managed to get the TC3 environment to work on one of these (after setting the isolated core for the TC3 project).

Runtime goes up and everything works nicely after booting the Windows BUT if I try to restart the runtime I get two same kind of errors:


But if I boot the machine, runtime starts and runs again nicely.

Any ideas how to get this work "as it should"?

This is the same problem i am experiencing and i haven't found a solution for it. It seems to work by using only KVM (without proxmox) so i assume the problem here is with proxmox..