Thanks a lot for your feedback. I will forward it to the team.

An additional information:
We are working on a students project to analyze the realtime performance on
Kubernetes. On the testsystem, we got a maximal Jitter below 10us. This is
achieved with several tweaks of the system, and by making heavy use of
multicore. Its very impressive what's possible with a current realtime
kernel!

Hehe, sounds great!

One of the "tricks" in our setup is currently, that we use "isolcpus", and
start the runtime without multicore support, bound to one of the isolated
cores.

So this core is free of any interrupts. Only some rare kernel locks are
producing some slight jitter.

Hi! In latest versions of Codesys Control for linux everything is improved very well!

Only a few steps are required to obtain very low jitter on many CPUs.

0 - Use RT Preempt kernel (Debian has a precompiled RT kernel)

1 - DISABLE HYPERTHREADING FROM UEFI Setup/BIOS!!!

2 - In INTEL CPUS change in /etc/default/grub this line:

GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=2,3 processor.max_cstate=1 intel_idle.max_cstate=0 acpi_irq_nobalance noirqbalance quiet"

isolcpus detach cores 2 and 3 from Linux KERNEL.
High cores work better for me.
processor.max_cstate and intel_idle.max_cstate tune power management.
acpi_irq_nobalance noirqbalance avoid IRQs balancing. This has a big impact on jitter.

3 - Use folowing script to push all IRQs possible on low cores use mask 1 or 3 to use cores 0 and 1 and leave alone 2 and 3:

#!/bin/sh
if [ $# != 1 ]
then
    echo "Syntax: $0 cpu_hex_mask"
                echo "Mask is 1-F where 1 is CPU0"
                echo
    exit
fi
for i in /proc/irq/*; do
        if [ -d $i ]; then
                echo $1 > $i/smp_affinity || true
        else
                echo $1 > $i || true
        fi
done

4 - Modify /etc/initrd/codesyscontrol do_status:

    do_status
    if [ $? -eq 0 ]; then
        rm $PIDFILE
        echo "Error: Failed to start codesyscontrol"
        exit 1
    else
        PID=$(cat $PIDFILE)
        echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
        echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
        echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
        echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor
        taskset -p 4 $PID >>/root/codelog.txt
        echo 0 > /proc/sys/kernel/numa_balancing
        echo never > /sys/kernel/mm/transparent_hugepage/enabled
        echo 0 > /sys/kernel/mm/ksm/run
        echo "codesyscontrol started"
    fi

This move codesysbin to core 2, fix CPU frequency to maximum (This is very important for jitter) and make some irq tuning.

5 - In codesys go in TaskGroup and select CORE 3 for all tasks. Now you have GNU/Linux OS on cores 0 and 1, only CODESYS runtime on core 2 and only IEC tasks on core 3.

6 - Tune /etc/CODESYSControl.cfg as follow:

[CmpSchedule]
SchedulerInterval=200 
ProcessorLoad.Enable=1
ProcessorLoad.Maximum=200
ProcessorLoad.Interval=200
DisableOmittedCycleWatchdog=1

In 4.2 and 4.4.1 runtimes max jitter shoud be from 13µs to 40µs depending on luky of your hardware configuration.

We have many production machines out in different hw configurations (J1900, i5 5th gen, i5 6th gen) and jitter max never go over 40µs. In luky hardware remain under 13µs.

There are many wonderful things about linux runtime:

1 - It's working in User Space, this make all stack exceptions bring debugger to yellow row indicating its.
In Windows RTE it's working in kernel space many time severe exceptions crash all device bringing Windows to reboot or to a blue screen of death or a black screen (I bring in all situations in some machine). In such cases it's very hard find error causes (I found only due a RPI4 in my bag connected in place of Windows controller and replaced back after solving problems).
Kernel space is painful for safety of IoT.

2 - All INTEL and Realtek chips working on a PCIe line work in ethercat very well also in 250µs cyclic with standard linux kernel modules (drivers)!

3 - You can develop external components in C.

4 - GNU/Linux is hundreds time much safe and stable in production machines, you don't fall in painful Windows 10/11 update system that is became a nightmare in industrial automation!

As side note, we are also testing ARM 64 runtime in Raspberry 4, with a our distro built in yocto.
Raspberry 4 has ethernet connector on PCIe line in contrast to USB of old Raspberries until 3.
It's impressive! We tested an application for many hours, using a Ethercat SLAVE board SPI, a C++ application for slave, CODESYS as master on native ethernet connector connected master to slave on itself rpi and obtained a max jitter of 19µs in a 300µs cyclic.
In only master configuration I controlled a Lenze Servo Inverter down to 250µs cyclic.

The only difference in RPI 4 is kernel command line of step 2, all other steps remain unaltered.

Also in Debian 10? In many machines using kernel 4.19.xRT Is working good. On medium load we have some spike after many hours but it's remaining <50-100µs depending on machine.
Results obtained with LXQT, a heavy Qt/C++ application doing HMI and IoT works, a mysql server and a openVPN service running on other cores.

In attached image you can see 65µs max after 2.5 hours.

It's a J1900 with medium load.

Here a i5 5th gen.
On same PC there are:

• Runtime with ethercat master
• Qt/C++ Settings manager, Gateway, logger and recorder/player to provide and store program recordings.
• Qt/C++ HMI on LxQt Desktop.
• X2GO remote desktop connected via VPN.
• OpenVPN Server.
• MySQL server for programs management and messages management.

As you can see, after about 2 hours max jitter is 45µs, but it remain many time under 10µs, there are spikes of 15-20µs in about 10-15 minutes and 30-45µs in about 30 minutes.

We have many machines working in this configuration out in last 3 years.