CI / CD stands for continuous integration and continuous delivery. Generally spoken, those terms are describing a standard way to build, test and ship software automatically. A typical and famous tool to realize such an automation is Jenkins.
In the world of Open Source, Cloud and Github, another famous system developed, which has the very big advantage of a good integration with GitHub, as well as good integration of C, Go or Java compilers. So for a big range of projects, it is a natural fit.
CODESYS and Drone
Drone is an impressing project. It focuses mainly on providing a perfectly integrated CI / CD system for GitHub. But, as the engine is open source, you can also create your own installation of it.
Such an installation is now integrated into CODESYS Forge. It is used in the backend to build CODESYS projects, build CODESYS packages, run Unit Tests or deploy the packages to CODESYS Forge.
Pipelines and Staging
The main concept of most CI / CD systems is, to separate at least build, test and deploy processes into separate steps. But in practice you often need even more steps in your pipeline.
Build: Compile a CODESYS library
TestBuild: Create a Bootapplication of a testproject with the library
TestExec: Run the test application on a SoftPLC, like CODESYS Control for Linux SL
Deploy: Push the Library to an FTP server
This is an easy, but real world use case for a CI / CD system. And while it's easy to understand, it shows a few important concepts, every user of a such a system should be aware of:
Every step depends on a successful previous step. So the pipeline stops when one of the steps fails.
The different steps need different tools.
Build and TestBuild need CODESYS
TestExec needs CODESYS Control
Deploy needs an FTP client
The first concept is known as pipelining, because everything is executed sequentially in a pipeline. The second step is known as staging, because every step is working on the same workspace, but with a different environment. This environment is seen as the stage on which the step of the pipeline takes place.
A build artifact is an output file, which was created during a build step. It can be an executable, a script, a library or even a complete package (nuget, npm, CODESYS Packages, ...). Many CI / CD systems (including Jenkins) define build artifacts to be passed between different steps of the pipeline.
Drone doesn't do that. It simply passes the whole workspace from stage to stage. With drone, build artifacts are collected in the last stage of a build pipeline. This stage is typically called, the deploy stage.
On CODESYS Forge, you just need to copy all files which you want to preserve into the subdirectory ".drone-artifacts" of your repository. Drone on CODESYS Forge will the serve those files under the URL "/download/<repository-name>".</repository-name>
The name of the URL is choosen by purpose, as you may serve those files directly to your users. The access rights to the files are the same as the rights to the repository.
If you now fear, that we repack your software and serve it to your users without your acknowledge, I can give you the all-clear.
We do build packages, but we don't link them anywhere. You can only find the links in the admin page of your projects.
By default, we use a generic drone configuration to build all repositories, which don't have one. This configuration is doing the following:
Extract Library Source Code
The built package is the served as a build artifact.
You can customuze this automatism, by providing your own files for the different steps:
provide your own package.manifest, to have control over the content of your package
provide your own .drone.yml in the root of the repository
Drone is build and test system, which in its nature is highly flexible. You can automate many different tasks with it.
build compiled libs
build test applications
All this can run automatically after every commit on CODESYS Forge. The only limit is actually the time it takes until a job is started, and the execution time of the job.
The free offer of the Drone CI/CD system limits the total execution time of a job to 5 minutes. This should be enough for all the above mentioned tasks, but it avoids that someone is ripping this service.
The time between the commit and the job being executed might vary, based on the load of the system. So there is no guarantee that the build jobs are running at a specific time.
In this example, we have a build pipeline with one step only. Every step defines its own stage, why I would recommend to keep the number of steps as small as possible.
To execute the command, we use a docker image called "codesys-ide:local". This image contains a full CODESYS installation, as well as some scripts to automate the most common CI/CD tasks. To execute those scripts, just run the wrapper "codesys" and pass the script name. In our case this was;
This command searches for all files ending with ".library" and build a file called ".compiled-library". Those compiled libraries don't contain source code anymore, have no connection to SVN and are packed into project archives w/o warnings. So they are easier to handle for the users for everyday tasks.
This command searches for files called "package.manifest". The folder, containing this file is packed into a CODESYS package. Those packages can be easily installed by your users, and they can contain libraries, projects, device descriptions and also third party files.
Naturally this is a final step, and you should have done all your build actions before.
Generate Package Manifest
If you don't have a package manifest, yet, you can let this script generate one for you. It will search for folders containing device descriptions, libraries or projects and create a package manifest, which installs those files.
If the repository is an SVN repository with the folders trunk, branches and tags, the scanning starts for all these folders separately.
Obviously it makes sense to call this action prior to "build-package".
This command searches for files ending with ".project" and tries to build a bootapplication out of them. The bootapplication contains of mainly two files, ending of ".app" and ".crc". The name of the files is equal to the name of the project file. This differs a bit from the default in CODESYS, as the bootapplications there are named after the application in the project.
Another deviation is, that we are limited to one application per project. We only build the active one if there are multiple applications in the project.
The reason is simply, that this behavior is much better for a generic build system, that knows nothing about the interna of the projects.