Codesys runtime break Python Scripts

Runtime 🇬🇧
arj3090
2019-09-06
2020-03-17

  • arj3090 - 2019-09-06

    I am trying to use Codesys on a BB-400 by Brainboxes and having trouble with Python services not running after installing the Codesys runtime. There are a series of python scripts that manage system services. One of which is shutting down the os when the UPS signals to.

    Looking at the log files and also running the scripts in ssh, I can see they run fine normally. As soon as I install the Codesys run-time, these python files no longer run or just hang. I tried using pdb to debug the python but it will also freeze up when I step through the code.

    I know this is related to the Codesys run time because I can remove the package and the Python scripts go back to working normal.

    Anyone have any hints of how I can troubleshoot this?

     

  • eschwellinger

    eschwellinger - 2019-09-07

    Hi,
    which CODESYS runtime and which RASBIAN (?) version is up and running on this system?

    BR
    Edwin

     

  • arj3090 - 2019-09-07

    I tried versions 3.5.14.30 and 3.5.15.0

    I will check the Raspbian version when I get to the office.

    The device is currently running :
    Linux 4.19.57-v7+

     

  • eschwellinger

    eschwellinger - 2019-09-07

    Hi,
    I have send out a release candidate of the 3.5.15.10 runtime,
    almost sure that this will solve it.
    Feedback well appreciated.

    BR
    Edwin

     

  • arj3090 - 2019-09-07

    I will test the new version right now and report back the results.

     

  • arj3090 - 2019-09-07

    So far that version seems to have fixed the issue. The python scripts are now running.

    The problem is now related to the BB-400 which I need to work with Brainboxes to resolve. I can call system shutdown on a UPS warning in the Python code and CodeSys will save persistent data value. But if I call shutdown on the UPS shutdown signal, Codesys does not save values. I am going to work on the Python and BB-400 side now.

    Thank you very much for the fast assistance and solution.

     

  • arj3090 - 2019-09-07

    I have found a way to make the BB-400 work well with Codesys, but it is kind of a hack because something doesn't work as expected.

    The BB-400 has 3 power states with 2 events that fire Python code.

    1) Running (normal power applied) - fires ups_power_change event
    2) UPS Alert (external power loss) - fires ups_power_change event
    3) UPS Shutdown - fires shutdown_signal event

    First Issue
    When the state goes to UPS Alert, the device also removes power from peripherals such as the Ethernet ports. I found this causes Codesys to stop the EtherCAT communcations in which it does not recover if returning to Running before going into UPS Shutdown.

    Second Issue
    Although the shutdown_signal event fires and the "shutdown now" is executed, Codesys still does not save persistent values. However if anything is done, such as shutdown, in the UPS Alert state, Codesys will save persistent values.

    My solution that resolves both issues is to stop Codesys when the BB-400 enter UPS Alert state and start Codesys when it goes into Running state.

    This is the modified Python code found in /usr/share/brainboxes/bb-core/bb_core_power.py

        def ups_power_change(self):
        logging.warning("bb_core_power : ups_power_change")
        isUPS = True if self.UPS_ALERT.value == 0 else False
        if isUPS:
            self.set_power_state(PowerStates.RUNNING)
        else:
            self.set_power_state(PowerStates.UPS_ALERT)
        self.power_changed_event(self.get_power_state())
        self.fire_event()
        # These must be done after firing the events because the events restart the peripherals
        if isUPS:
            # Try to start Codesys in case a UPS Alert stopped it
            subprocess.run("sudo bash /etc/init.d/codesyscontrol start", shell=True)
        else:
            # Stop the Codesys in order to force it to save persistent value
            # It is also necessary because Codesys does not recover from Ethernet
            # being powered off during UPS Alert
            subprocess.run("sudo bash /etc/init.d/codesyscontrol stop", shell=True)

    The proper solution is to figure why no values are saved during a shutdown in the shutdown_signal event. I know the event executes and the OS saves data because there is logging to a file that is updated in the event.

    Any possible ideas of why a shutdown will save data in a ups_power_change event, but not in the shutdown_signal event?

    Another thought I had was to use the GPIO signal in Codesys that comes from the Power Management Unit telling the OS to shutdown. Is there a way in Codesys to look at that signal and tell it to save its persistent data values?

     

  • eschwellinger

    eschwellinger - 2019-09-08

    Hi,
    this works if Linux does a gracefull shutdown, means the codesys runtime will be stopped/exit by the shutdown of the Linux System.

    what do this python exactly in this state.... shutdown the whole system? Then it must work.

    BR
    Edwin

     

  • arj3090 - 2019-09-08

    This is the code that executes when the UPS sends the signal to shutdown:

        def shutdown_signal(self):
        logging.warning("bb_core_power : entered shutdown_signal")
\#        subprocess.run('sudo killall codesyscontrol.bin', shell=True)
        subprocess.run("sudo bash /etc/init.d/codesyscontrol stop", shell=True)
        logging.warning("bb_core_power : after codesys kill")
        self.set_power_state(PowerStates.STOPPING)
        self.power_changed_event(self.get_power_state())
        self.fire_event()
        self.CMHalting.close()
        subprocess.run('sudo wall "Power failure, BB-400 will shut down"', shell=True, check=True)
        logging.warning("bb_core_power : About to  Shut Down")
        subprocess.run('sudo shutdown now', shell=True)
        logging.warning("bb_core_power : After Shut Down")

    You can see where I attempted to stop Codesys prior to the Shutdown Now. I definitely know the code is executing because I can check the log file and see in the log file. I also know the commands I am using to stop Codesys will save data because I tested them in the UPS Alert event.

    Considering the UPS Alert state powers down all peripherals, my next theory is that Codesys abruptly stops prior to the Shutdown_signal event, therefore not allowing a graceful shut down of Codesys. I am going to test this by listing the active processes to the log file so I can see if the Codesys process is still running when it reaches the Shutdown_Signal event.

    I will do some further testing and report back.

     

  • arj3090 - 2019-09-09

    I have confirmed the CodesysControl process abnormally stops prior to the Shutdown_Signal event, not allowing persistent data to be saved on the OS shutdown. To conclude this, I inserted a command into the Shutdown_Signal event to list all processes that contain "co". This was the result

    37 ? 00:00:00 kcompactd0
    69 ? 00:00:00 mmc_complete
    74 ? 00:00:00 ext4-rsv-conver
    75 ? 00:00:00 ipv6_addrconf

    So CodesysControl is no longer running when entering the Shutdown_Signal event. Is there a log or crash report that may give a clue as to why the Codesys process exits abruptly after the UPS goes into Alert state?

     

  • eschwellinger

    eschwellinger - 2019-09-09

    Hi,

    cat /tmp/codesyscontrol.log

    or even better:
    tail -f /tmp/codesyscontrol.log

    BR
    Edwin

     

  • arj3090 - 2019-09-09

    These are my latest results after further testing:

    In the UPS Alert event:

    Process List of co* : In UPS Alert Event
   37 ?        00:00:00 kcompactd0
   69 ?        00:00:00 mmc_complete
   73 ?        00:00:00 ext4-rsv-conver
   75 ?        00:00:00 ipv6_addrconf
  607 ?        00:00:01 codesyscontrol.
  714 ?        00:00:13 codesyscontrol.
 1080 ?        00:00:00 COEX-TX-Thread
 

    Codesys Log File
2019-09-09T08:48:43Z, 0x00000071, 1, 0, 0, **************************************************************
2019-09-09T08:48:58Z, 0x00000007, 1, 0, 6, Network interface: <ipaddress>192.168.63.1</ipaddress>, subnetmask <subnetmask>255.255.255.0</subnetmask>
2019-09-09T08:48:58Z, 0x00000018, 1, 0, 4, Network interface <interface>ether 5</interface> at router <instance>2</instance> registered
2019-09-09T08:48:58Z, 0x00000018, 1, 0, 1, Setting router <instance>2</instance> address to <address>(0001)</address>
2019-09-09T08:49:04Z, 0x00000007, 1, 0, 6, Network interface: <ipaddress>169.254.185.103</ipaddress>, subnetmask <subnetmask>255.255.0.0</subnetmask>
2019-09-09T08:49:04Z, 0x00000018, 1, 0, 4, Network interface <interface>ether 4</interface> at router <instance>3</instance> registered
2019-09-09T08:49:04Z, 0x00000018, 1, 0, 1, Setting router <instance>3</instance> address to <address>(0000:b967)</address>
2019-09-09T08:49:13Z, 0x00000111, 2, 0, 2, !!!! Warning: Duplicate IP detected. Skipping second one.
2019-09-09T08:49:46Z, 0x00000001, 1, 0, 0, runtime licensed

    In the Shutdown_Signal event:

    Process List of co* : In Shutdown_Event
   37 ?        00:00:00 kcompactd0
   69 ?        00:00:00 mmc_complete
   73 ?        00:00:00 ext4-rsv-conver
   75 ?        00:00:00 ipv6_addrconf

    Something has caused the Codesys process to close.

    Codesys Log File
2019-09-09T08:48:43Z, 0x00000071, 1, 0, 0, **************************************************************
2019-09-09T08:48:58Z, 0x00000007, 1, 0, 6, Network interface: <ipaddress>192.168.63.1</ipaddress>, subnetmask <subnetmask>255.255.255.0</subnetmask>
2019-09-09T08:48:58Z, 0x00000018, 1, 0, 4, Network interface <interface>ether 5</interface> at router <instance>2</instance> registered
2019-09-09T08:48:58Z, 0x00000018, 1, 0, 1, Setting router <instance>2</instance> address to <address>(0001)</address>
2019-09-09T08:49:04Z, 0x00000007, 1, 0, 6, Network interface: <ipaddress>169.254.185.103</ipaddress>, subnetmask <subnetmask>255.255.0.0</subnetmask>
2019-09-09T08:49:04Z, 0x00000018, 1, 0, 4, Network interface <interface>ether 4</interface> at router <instance>3</instance> registered
2019-09-09T08:49:04Z, 0x00000018, 1, 0, 1, Setting router <instance>3</instance> address to <address>(0000:b967)</address>
2019-09-09T08:49:13Z, 0x00000111, 2, 0, 2, !!!! Warning: Duplicate IP detected. Skipping second one.
2019-09-09T08:49:46Z, 0x00000001, 1, 0, 0, runtime licensed
2019-09-09T08:49:54Z, 0x00000018, 1, 0, 5, Network interface <interface>ether 5</interface> unregistered
2019-09-09T08:49:54Z, 0x00000018, 1, 0, 5, Network interface <interface>ether 4</interface> unregistered
2019-09-09T08:49:54Z, 0x000010f0, 1, 0, 1, <NetworkName>eth0</NetworkName><IPAddress>0.0.0.0</IPAddress><Subnetmask>0.0.0.0<Component>IoDrvEthernet</Component>

    The log file only reported "unregistered" of ethernet interfaces. I assume this is because the ethernet ports are powered down in UPS Alert state. Could this cause Codesys to close abruptly?

     

  • eschwellinger

    eschwellinger - 2019-09-10

    Hi,
    guess not.

    Could you please stop the plc , followed by a (as sudo):

    cd /var/opt/codesys
    /opt/codesys/bin/codesyscontrol.bin -d /etc/CODESYSControl.cfg

    Then you should a litte more information on this manual start of the runtime.

    Why does your UPS not a simple trigger a 'shutdown -h now' - then I think it will close/shutdown the runtime in correct way

    BR
    Edwin

     

  • arj3090 - 2019-09-10

    The BB-400 goes into alert mode, then powers down the peripherals, but keeps the CPU running. If power is restored in less than 60 seconds, it will power the peripherals back up. If no power longer than 60 seconds, it will perform the shutdown -h now

    Here is the last few lines from running the recommended test:

    SysTaskFrame[425]:  OPCUAServer(tid:1603,OSprio:0) running
SysTaskCreate[497]: create CommCycleHook(prio:223,interval:0us,stacksize:131072)
SysTaskFrame[425]:  CommCycleHook(tid:1604,OSprio:0) running
SysTaskCreate[497]: create X509AsyncTask(prio:255,interval:0us,stacksize:131072)
SysTaskFrame[425]:  X509AsyncTask(tid:1605,OSprio:0) running
SysSockSendTo[1268]: sendto(5,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(5,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(5,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(5,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(5,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(5,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(5,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(5,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(11,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(11,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(11,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(11,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(11,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(11,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(11,0x76a309f4,8,0x0,...):Invalid argument
SysSockSendTo[1268]: sendto(11,0x76a309f4,8,0x0,...):Invalid argument
ooops... this runtime was built for RASPBERRYPI (-22, 0x00000BB8, 0xFFFFFFFB)
     

  • eschwellinger

    eschwellinger - 2019-09-10

    Think if you let eth0 up and running it might work... give this a try if possible.
    Br Edwin

     

  • arj3090 - 2019-09-10

    Edwin Schwellinger hat geschrieben:
    Think if you let eth0 up and running it might work... give this a try if possible.

    Tech support at Brainboxes tells me it is not possible. They said it is a hardware design that shuts down peripheral power and cannot be controlled by the power management unit firmware. They would not share the schematics with me, so I am unable to verify this.

     

  • arj3090 - 2019-09-11

    Based on this:

    "ooops... this runtime was built for RASPBERRYPI (-22, 0x00000BB8, 0xFFFFFFFB)"

    Does Codesys use the Ethernet port in some way to determine if the device is a Raspberry Pi? If so, can that check be delayed for 2 minutes before closing out?

     

  • eschwellinger

    eschwellinger - 2019-09-11

    no, this is not possible!

     

  • arj3090 - 2019-09-11

    Do I have any options to keep Codesys from abruptly dumping out when the Ethernet port loses power?

     

  • eschwellinger

    eschwellinger - 2019-09-11

    Hi,
    no I see no way ... no chance to do this.
    BR
    Edwin

     

  • rckalex - 2020-03-17

    @eschwellinger

    Hi Edwin,

    I'm trying to replicate your solution by having the script Stop/Start the codesyscontrol with UPS state change but have been unsuccessful.

    Was your only change to add these lines to bb_core_power.py? Double quotes or single quotes around the start/stop command?

      if isUPS:
        subprocess.run("sudo bash /etc/init.d/codesyscontrol start", shell=True)
    else:
        subprocess.run("sudo bash /etc/init.d/codesyscontrol stop", shell=True)
     

Log in to post a comment.