Good improvement in linux Real Time.

2022-11-07T21:32:23.106000Z

Currently on Debian 11, but somehow i've managed to get it working with the steps you suggested.
On a i5-11th Gen. iEi DRPC industrial PC it's working for a few weeks without any problems.
The cores 2+3 are isolated from the rest and there is no conflict on the CODESYS runtime with the NodeJS/MongoDB server, also on heavy load.

This thread should be official included to the CODESYS FAQ, because of the rare infos about the Linux runtime.

Thanks to everyone here, especially @mondinmr

Good improvement in linux Real Time.

2022-10-25T08:59:48.588000Z

Here a i5 5th gen.
On same PC there are:

Runtime with ethercat master
Qt/C++ Settings manager, Gateway, logger and recorder/player to provide and store program recordings.
Qt/C++ HMI on LxQt Desktop.
X2GO remote desktop connected via VPN.
OpenVPN Server.
MySQL server for programs management and messages management.

As you can see, after about 2 hours max jitter is 45µs, but it remain many time under 10µs, there are spikes of 15-20µs in about 10-15 minutes and 30-45µs in about 30 minutes.

We have many machines working in this configuration out in last 3 years.

Good improvement in linux Real Time.

2022-10-20T09:21:00.759000Z

Also in Debian 10? In many machines using kernel 4.19.xRT Is working good. On medium load we have some spike after many hours but it's remaining <50-100us depending on machine.
Results obtained with LXQT, a heavy Qt/C++ application doing HMI and IoT works and a openVPN service running on other cores.

Good improvement in linux Real Time.

2022-10-14T07:09:14.375000Z

Hi again,

i did everything that mondinmr wrote in his tutorial above. If i'm starting the codesyscontrol and let my program run, there is a max. jitter with ~15us.
If i let the program run for lets say a few hours over night, there is a max. jitter with 300us. Is this because you cannot remove all IRQs from the two isolated cores and something is blocking the cycle?

I switched yesterday from Debian 10 to Debian 11 with RT patch (with all the tweaks from above) and except from that i think the IEC-Tasks and everything else works perfect.
Also the Intel I225-LM is working great with my TCP/IP Modbus controller with Debian 11 out of the box.

Greetings,
paho

Good improvement in linux Real Time.

2022-08-18T15:05:42.020000Z

Great, with that tutorial i've made it. Working like charm.

Another question, because there is a current project where the client asked me if we can put the CODESYS Runtime on a virtual system (because they've a big server farm with thousands of virtual systems) with soft real time (cycle times of 0,5-1ms)? Did anyone encountered something like this before?

Thanks a lot!

Good improvement in linux Real Time.

2022-08-16T07:16:22.497000Z

Hi @mondinmr,
thanks again for this nice summary about realtime optimizations!
And also thanks for the outlook regarding the Rasperry Pi 4.

Looking forward to hear from that.

Hint: If you post the Raspberry Pi results in CODESYS Talk, maybe use a different topic for RPI. Then it will be easier found.

Don't know if you found this already, but from CODESYS there is an FAQ, giving some optimization hints: https://faq.codesys.com/pages/viewpage.action?pageId=122748972

And soon, there will be a new chapter in the online help.

Keep up the great work!

Cheers,
Ingo

Good improvement in linux Real Time.

2022-08-12T07:58:37.007000Z

Hi! In latest versions of Codesys Control for linux everything is improved very well!

Only a few steps are required to obtain very low jitter on many CPUs.

0 - Use RT Preempt kernel (Debian has a precompiled RT kernel)

1 - DISABLE HYPERTHREADING FROM BIOS!!!

2 - In INTEL CPUS change in /etc/default/grub this line:

GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=2,3 processor.max_cstate=1 intel_idle.max_cstate=0 acpi_irq_nobalance noirqbalance quiet"

isolcpus detach cores 2 and 3 from Linux KERNEL.
High cores work better for me.
processor.max_cstate and intel_idle.max_cstate tune power management.
acpi_irq_nobalance noirqbalance avoid IRQs balancing. This has a big impact on jitter.

3 - Use folowing script to push all IRQs possible on low cores use mask 1 or 3 to use cores 0 and 1 and leave alone 2 and 3:

#!/bin/sh
if [ $# != 1 ]
then
    echo "Syntax: $0 cpu_hex_mask"
                echo "Mask is 1-F where 1 is CPU0"
                echo
    exit
fi
for i in /proc/irq/*; do
        if [ -d $i ]; then
                echo $1 > $i/smp_affinity || true
        else
                echo $1 > $i || true
        fi
done

4 - Modify /etc/initrd/codesyscontrol do_status:

    do_status
    if [ $? -eq 0 ]; then
        rm $PIDFILE
        echo "Error: Failed to start codesyscontrol"
        exit 1
    else
        PID=$(cat $PIDFILE)
        echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
        echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
        echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
        echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor
        taskset -p 4 $PID >>/root/codelog.txt
        echo 0 > /proc/sys/kernel/numa_balancing
        echo never > /sys/kernel/mm/transparent_hugepage/enabled
        echo 0 > /sys/kernel/mm/ksm/run
        echo "codesyscontrol started"
    fi

This move codesysbin to core 2, fix CPU frequency to maximum (This is very important for jitter) and make some irq tuning.

5 - In codesys go in TaskGroup and select CORE 3 for all tasks. Now you have GNU/Linux OS on cores 0 and 1, only CODESYS runtime on core 2 and only IEC tasks on core 3.

In 4.2 and 4.4.1 runtimes max jitter shoud be from 13µs to 40µs depending on luky of your hardware configuration.

We have many production machines out in different hw configurations (J1900, i5 5th gen, i5 6th gen) and jitter max never go over 40µs. In luky hardware remain under 13µs.

There many wonderful things about linux runtime:

1 - Is working in User Space, this make all stack exceptions bring debugger to yellow row indicating its.
In Windows RTE that is working in kernel space many time severe exceptions crash all device bringing Windows to reboot or to a blue screen of death. I such cases is very hard find error causes.
Kernel space is painful for safety of IoT.

2 - All INTEL and Realtek chips working on a PCIe line work in ethercat very well also in 250µs cyclic with standard linux kernel modules (drivers)!

3 - You can develop external components in C.

4 - GNU/Linux is hundreds time much safe and stable in production machines, you don't fall in painful Windows 10/11 update system that is became a nightmare in industrial automation!

As side note, we are also testing ARM 64 runtime in Raspberry 4, with a our distro built in yocto.
Raspberry 4 has ethernet connector on PCIe line in contrast to USB of old Raspberries until 3.
It's impressive! We tested an application for many hours, using a Ethercat SLAVE board SPI, a C++ application for slave, CODESYS as master on native ethernet connector connected master to slave on itself rpi and obtained a max jitter of 19µs in a 300µs cyclic.
In only master configuration I controlled a Lenze Servo Inverter down to 250µs cyclic.

The only difference in RPI 4 is kernel command line of step 2, all other steps remain unaltered.

Good improvement in linux Real Time.

2022-08-03T18:58:11.956000Z

Good to hear that, because i'm using the Linux SL on my i5 IPC too. Having Debian 4.19 with RT Preempt Kernel. The max. Jitter is 272us.
Are there additionally adjustments to do in the OS? My application is now fixed at CPU1 and i want to know if using the multicore brings really an improvement...

For my application, i'm not that unhappy with the jitter, but i'm using MODBUS TCP/IP bus couplers from WAGO and if i'm going to send data in short intervals (like every 50ms) then i can see some delays (not lost packets or erros) - i can see that on a LED that i'm going to switch on and of every 50ms. From time to time it seems that there is a delay of one 0/1 cycle, like a pause of 50ms.
The coupler is connected with a seperate ethernet interface and static configuration, point-to-point connection with a patchcable.

I think this can also be some network driver issue... maybe someone has a suggestion for my problem.

Thank you,
paho

Good improvement in linux Real Time.

2022-03-04T14:51:52.408000Z

Raspberry PI4, debian 64bit, Kernel RT stock from debian repo.
Runtime SL ARM64.
A little tuning in kernel parameters and CPU pinning.
Jitter <= 52µs with ethercat on native adapter.

Good improvement in linux Real Time.

2022-01-06T09:50:01.271000Z

thank a lot for your answer.
This is very helpful for me

Recent posts to Good improvement in linux Real Time.

Good improvement in linux Real Time.

Good improvement in linux Real Time.

Good improvement in linux Real Time.

Good improvement in linux Real Time.

Good improvement in linux Real Time.

Good improvement in linux Real Time.

Good improvement in linux Real Time.

Good improvement in linux Real Time.

Good improvement in linux Real Time.

Good improvement in linux Real Time.