Welcome to our new forum
All users of the legacy CODESYS Forums, please create a new account at account.codesys.com. But make sure to use the same E-Mail address as in the old Forum. Then your posts will be matched. Close

Good improvement in linux Real Time.

mondinmr
2021-12-11
2022-01-06
  • mondinmr

    mondinmr - 2021-12-11

    We started using Linux SL since 3.5.12.
    On i5 6th gen., isolating 2 cores, using RT Preempt Kernel precompiled by DEBIAN, was very interesting see improvements update after update.

    3.5.12 Max Jitter in field application ~180Β΅s.
    3.5.14 Max Jitter in field application ~102Β΅s.
    4.2.0 Max Jitter in field application ~62Β΅s.

    In 4.2.0 is near RTE windows Jitter on same hardware, but with a big advantage on security side.
    Linux SL is running in user space and it's possible isolate it on a chroot jail!!!
    Windows RTE is in kernel ring and this is very dangerous on all sides in security meanings.
    Today continuos upgrading of Windows OS meke it a good Desktop OS, but create tons of problems on industrial field applications.

    This message is just to congratulate the team that is following the development of Codesys!

     
    πŸ‘
    2
    • Ingo

      Ingo - 2021-12-12

      Thanks a lot for your feedback. I will forward it to the team.

      An additional information:
      We are working on a students project to analyze the realtime performance on
      Kubernetes. On the testsystem, we got a maximal Jitter below 10us. This is
      achieved with several tweaks of the system, and by making heavy use of
      multicore. Its very impressive what's possible with a current realtime
      kernel!

       
  • mondinmr

    mondinmr - 2021-12-17

    Wow just tested on a J1900!!! 39Β΅s max jitter!!!
    We'll put in production next month.

     

    Last edit: mondinmr 2021-12-17
  • mondinmr

    mondinmr - 2021-12-17

    @Igno, you activated me! I played around with /etc/CODESYSControl.cfg, interrupts, kernel parameters etc ...
    22Β΅s on J1900 ...

    Not under 10Β΅s like your tests, but a good result for me.
    In a system running KDE as window manager!

     

    Last edit: mondinmr 2021-12-17
  • mondinmr

    mondinmr - 2021-12-17

    Another test!

    -Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
    -Debian/Linux Buster;
    -RT kernel;
    -Headless system;
    -KVM Hypervisor;
    -Hyperthreading disabled;
    -Core 2 and 3 detached from OS;
    -Core 0 and 1 used by OS and pinned to Windows10 VM;
    -Windows10 running with main GPU, USB and WIFI in passtrought;

    -Runtime Linux SL 4.0.2;
    -Codesyscontrol main process forced to core 2;
    -IEC tasks fixed and pinned to core 3;
    -Many tunings in kernel parameters;

    Max Jitter 18Β΅s after 20 minutes!!!

     
    • Ingo

      Ingo - 2021-12-17

      Hehe, sounds great!

      One of the "tricks" in our setup is currently, that we use "isolcpus", and
      start the runtime without multicore support, bound to one of the isolated
      cores.

      So this core is free of any interrupts. Only some rare kernel locks are
      producing some slight jitter.

       
  • mondinmr

    mondinmr - 2021-12-17

    Isolcpus is the basis. Then there are a couple of extra parameters on the kernel line.
    An additional script that shifyvinterrupts the kernel previously assigned is very useful in init.
    Then there would be the two interval parameters in codesyscontrol.cfg, but I haven't found any documentation and I still don't quite understand what they change. I know for a fact that they affect jitter a lot. And in last some tuning in /sys and /proc.

     
  • mondinmr

    mondinmr - 2021-12-20

    Intel(R) Core(TM) i5-6440EQ CPU @ 2.70GHz

    Same kernel tuning.
    10Β΅s!!!

    As in attached screenshot.

     
    πŸ‘
    2
  • mikuroung

    mikuroung - 2021-12-31

    what is the debian version.?

    I wonder what kind of tuning you did.

     
    • mondinmr

      mondinmr - 2022-01-03

      Buster. I'm still in Buster, Bullseye is too young.

      Here is my kernel line in i7 using KVM as hypervisor.

      GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=2,3 processor.max_cstate=1 intel_idle.max_cstate=0 acpi_irq_nobalance noirqbalance console=ttyS0 earlyprintk=ttyS0 quiet nofb loglevel=0 vfio-pci.ids=1002:6987,1002:aae0,8086:9dc8 nofb nomodese intel_iommu=on iommu=1 video=vesafb:off,efifb:off vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1"
      
      • In BIOS I disabled hyperthreading

      • isolcpus=2,3 turn off cores 2 and 3 from OS usage.
        I obtained better results on cores 2 and 3 leaving 0 and 1 to OS.

      • processor.max_cstate=1 intel_idle.max_cstate=0
        Force some power management in intel CPU (look frequency and disable power management in BIOS is not enough)

      • acpi_irq_nobalance noirqbalance
        Force IRQ balancing to off, avoiding kernel IRQ on cores 2 and 3

      • A script is useful to move preassigned IRQ out of cores 2 and 3.

      I found it in some forum.

       1
       2
       3
       4
       5
       6
       7
       8
       9
      10
      11
      12
      13
      14
      15
      #!/bin/sh
      if [ $# != 1 ]
      then
          echo "Syntax: $0 cpu_hex_mask"
                      echo "Mask is 1-F where 1 is CPU0"
                      echo
          exit
      fi
      for i in /proc/irq/*; do
              if [ -d $i ]; then
                      echo $1 > $i/smp_affinity || true
              else
                      echo $1 > $i || true
              fi
      done
      

      ForceIrqToCore.sh 1
      Force all movable IRQ to core 0.

      • I modified /etc/init.d/codesyscontrol start function
      start_runtime () {
          #exit script if package is not installed
          [ -x "$EXEC" ] || exit 5
      
          if [ ! -z "$DEBUGOUTPUT" ]; then
              ARGS="-d $ARGS"
              if [ -z "$DEBUGLOGFILE" ]; then
                  DEBUGLOGFILE=/tmp/codesyscontrol_debug.log
              fi
          else
              DEBUGLOGFILE=/dev/null
          fi
      
          mkdir -p $WORKDIR
          cd $WORKDIR && ( $DAEMON $DAEMON_ARGS $EXEC $CONFIGFILE $ARGS >$DEBUGLOGFILE 2>&1 & echo $! >$PIDFILE )
          sleep 1
          if [ ! -z $DAEMON ] && which pidof >/dev/null 2>&1; then
              # wait up to 10 seconds for process to become ready
              local TIMEOUT=10
              while ! pidof -s $EXEC >$PIDFILE 2>/dev/null; do
                  TIMEOUT=$(expr $TIMEOUT - 1)
                  if [ "$TIMEOUT" = "0" ]; then
                      break
                  fi
                  sleep 1
              done
          fi
      
          do_status
          if [ $? -eq 0 ]; then
              rm $PIDFILE
              echo "Error: Failed to start codesyscontrol"
              exit 1
          else
              PID=$(cat $PIDFILE)
              echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
              echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
              echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
              echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor
              taskset -p 4 $PID >>/root/codelog.txt
              renice -n -20 -p $PID >>/root/codelog.txt
              tuna --threads $PID --priority=RR:99
              echo 0 > /proc/sys/kernel/numa_balancing
              echo never > /sys/kernel/mm/transparent_hugepage/enabled
              echo 0 > /sys/kernel/mm/ksm/run
              echo "codesyscontrol started"
          fi
      }
      

      Forcing runtime to core 3 (Mask 4)
      I forced nice level and priority with tuna and renice.
      (apt-get install tuna)
      - I forced tasks in CODESYS in fixed pinning on core 2.

      • In last I made a tuning in /etc/CODESYSControl.cfg
      [CmpSchedule]
      SchedulerInterval=100 
      ProcessorLoad.Enable=1
      ProcessorLoad.Maximum=200
      ProcessorLoad.Interval=200
      DisableOmittedCycleWatchdog=1
      

      Lowering SchedulerInterval and ProcessorLoad.Interval until runtime take 70-80% of core 3 usage. (I don't know very well what they do respectively)

      I also removed irqbalance:
      apt-get purge irqbalance

       
      πŸ‘
      1

      Last edit: mondinmr 2022-01-07
      • Ingo

        Ingo - 2022-01-03

        Maybe I can explain the mysterium about the scheduler interval:
        It is an interval, in which the scheduler checks all IEC task watchdogs, as well as the processorload watchdog.

        The IEC task watchdog is what you define in the taskconfiguration in CODESYS. And when you increase this value in the. config file, your system will react later on a watchdog event.

        The interval of the processoload watchdog can't be lower than the scheduler interval (at least it would not make much sense), because it is the scheduler, which checks the processor load.

        What can you do with this intercal?
        You can reduce the CPU overhead, by increasing the interval. but it will not really have an effect on the maximum jitter.

         
  • MadsKaizer

    MadsKaizer - 2022-01-03

    Thank you for sharing these improvement speeds, its always nice to see some real numbers behind a patch note "Improved code execution" :)

     
  • mikuroung

    mikuroung - 2022-01-06

    thank a lot for your answer.
    This is very helpful for me

     

Log in to post a comment.