Continuous Integration

CI / CD

CI / CD stands for continuous integration and continuous delivery. Generally spoken, those terms are describing a standard way to build, test and ship software automatically. A typical and famous tool to realize such an automation is Jenkins.

In the world of Open Source, Cloud and Github, another famous system developed, which has the very big advantage of a good integration with GitHub, as well as good integration of C, Go or Java compilers. So for a big range of projects, it is a natural fit.

CODESYS and Drone

Drone is an impressing project. It focuses mainly on providing a perfectly integrated CI / CD system for GitHub. But, as the engine is open source, you can also create your own installation of it.

Such an installation is now integrated into CODESYS Forge. It is used in the backend to build CODESYS projects, build CODESYS packages, run Unit Tests or deploy the packages to CODESYS Forge.

Pipelines and Staging

The main concept of most CI / CD systems is, to separate at least build, test and deploy processes into separate steps. But in practice you often need even more steps in your pipeline.

For example:

  • Build: Compile a CODESYS library
  • TestBuild: Create a Bootapplication of a testproject with the library
  • TestExec: Run the test application on a SoftPLC, like CODESYS Control for Linux SL
  • Deploy: Push the Library to an FTP server

This is an easy, but real world use case for a CI / CD system. And while it's easy to understand, it shows a few important concepts, every user of a such a system should be aware of:

  1. Every step depends on a successful previous step. So the pipeline stops when one of the steps fails.
  2. The different steps need different tools.
  3. Build and TestBuild need CODESYS
  4. TestExec needs CODESYS Control
  5. Deploy needs an FTP client

The first concept is known as pipelining, because everything is executed sequentially in a pipeline. The second step is known as staging, because every step is working on the same workspace, but with a different environment. This environment is seen as the stage on which the step of the pipeline takes place.

Build Artifacts

A build artifact is an output file, which was created during a build step. It can be an executable, a script, a library or even a complete package (nuget, npm, CODESYS Packages, ...). Many CI / CD systems (including Jenkins) define build artifacts to be passed between different steps of the pipeline.

Drone doesn't do that. It simply passes the whole workspace from stage to stage. With drone, build artifacts are collected in the last stage of a build pipeline. This stage is typically called, the deploy stage.

On CODESYS Forge, you just need to copy all files which you want to preserve into the subdirectory ".drone-artifacts" of your repository. Drone on CODESYS Forge will the serve those files under the URL "/download/repository-name". Note that those files will not be commited into your repository implicitely.

The name of the URL is choosen by purpose, as you may serve those files directly to your users. The access rights to the files are the same as the rights to the repository.

Download Links

If you now fear, that we repack your software and serve it to your users without your acknowledge, I can give you the all-clear.

We do build packages, but we don't link them anywhere. You can only find the links in the admin page of your projects.

Example Pipeline

By default, we use a generic drone configuration to build all repositories, which don't have one. This configuration is doing the following:

  • Compile Libraries
  • Extract Library Source Code
  • Create Packages
  • Build Packages

The built package is then served as a build artifact.

You can customize this automatism, by providing your own files for the different steps:

  • provide your own package.manifest, to have control over the content of your package
  • provide your own .drone.yml in the root of the repository

An example .drone.yml might look like this:

kind: pipeline
name: default
workspace:
  base: /working
steps:
  - name: compile
    image: codesys-ide:local
    commands:
      - codesys compile-library

All "codesys" commands are automatically collecting their output as artifacts. So when you call compile-library, you will get all created *.compiled-library files as build artifacts.

Adding own artifacts

If you want to save other files as build artifacts, you can simply add a copy command to your ".drone.yml".

kind: pipeline
name: default
workspace:
  base: /working
steps:
  - name: compile
    image: codesys-ide:local
    commands:
      - some command that creates some-file
      - cp some-file .drone-artifacts/

Scenarios

Drone is build and test system, which in its nature is highly flexible. You can automate many different tasks with it.

Such as:

  • build compiled libs
  • build test applications
  • build bootapplications
  • build packages
  • ...

All this can run automatically after every commit on CODESYS Forge. The only limit is actually the time it takes until a job is started, and the execution time of the job.

The free offer of the Drone CI/CD system limits the total execution time of a job to 5 minutes. This should be enough for all the above mentioned tasks, but it avoids that someone is ripping this service.

The time between the commit and the job being executed might vary, based on the load of the system. So there is no guarantee that the build jobs are running at a specific time.

Compile Libraries

kind: pipeline
name: default
workspace:
  base: /working/repo
steps:
  - name: compile
    image: codesys-ide:local
    commands:
      - codesys compile-library

In this example, we have a build pipeline with one step only. Every step defines its own stage, why I would recommend to keep the number of steps as small as possible.

To execute the command, we use a docker image called "codesys-ide:local". This image contains a full CODESYS installation, as well as some scripts to automate the most common CI/CD tasks. To execute those scripts, just run the wrapper "codesys" and pass the script name. In our case this was;

- codesys compile-library

This command searches for all files ending with ".library" and build a file called ".compiled-library". Those compiled libraries don't contain source code anymore, have no connection to SVN and are packed into project archives w/o warnings. So they are easier to handle for the users for everyday tasks.

Build Packages

- codesys build-package

This command searches for files called "package.manifest". The folder, containing this file is packed into a CODESYS package. Those packages can be easily installed by your users, and they can contain libraries, projects, device descriptions and also third party files.

Naturally this is a final step, and you should have done all your build actions before.

For example:

- codesys compile-library
- codesys build-package

Instead of:

- codesys build-package
- codesys compile-library

Generate Package Manifest

- codesys gen-package-manifest

If you don't have a package manifest, yet, you can let this script generate one for you. It will search for folders containing device descriptions, libraries or projects and create a package manifest, which installs those files.

If the repository is an SVN repository with the folders trunk, branches and tags, the scanning starts for all these folders separately.

Obviously it makes sense to call this action prior to "build-package".

For example:

- codesys compile-library
- codesys gen-package-manifest
- codesys build-package

Build Bootapplications

- codesys build-bootapp

This command searches for files ending with ".project" and tries to build a bootapplication out of them. The bootapplication contains of mainly two files, ending of ".app" and ".crc". The name of the files is equal to the name of the project file. This differs a bit from the default in CODESYS, as the bootapplications there are named after the application in the project.

Another deviation is, that we are limited to one application per project. We only build the active one if there are multiple applications in the project.

The reason is simply, that this behavior is much better for a generic build system, that knows nothing about the interna of the projects.

Export Library Documentation

- codesys export-libdoc

This command searches for all files ending with ".library" and tries to export the library documentation. To find out how to add documentation to your library, please checkout the CODESYS Online Help.


Related

Wiki: IndexMain