From a code change to a release candidate

A component can build successfully and still fail once it becomes part of a larger system. Its behaviour depends on the software around it, the hardware it runs on and the combinations of changes arriving from other teams.

The CI/CD environment supported embedded Linux software built with the Yocto Project, with components developed across hundreds of repositories. An integration repository brought those components together for build and testing.

The engineering work covered Jenkins build, test and delivery pipelines, their integration with the Eiffel event system, and the preparation of hardware resources for execution. These pipelines operated within an established integration process, where each candidate passed through progressively broader checks.

From a build to an integrated system

The first checks focused on the individual change: source verification, whether it could be built, and static analysis for code quality and security issues through SonarQube.

Passing those checks established an initial confidence level. The candidate could then progress to testing against a known software baseline, showing how the changed component behaved within an otherwise established version of the system.

Further stages expanded the scope to other devices and components. Software for one device could be tested as part of a cluster or a larger system, where its interactions with other components became relevant.

Progression depended on the checks required at each confidence level. Jenkins pipelines coordinated the build, test and delivery work, with results providing the basis for a candidate to move forward.

Testing combinations of candidate changes

Testing against a known baseline provided an initial point of comparison. A forthcoming release could also contain changes from several other repositories, introducing combinations that needed their own evaluation.

Candidates considered for release entered nightly test loops alongside other candidate packages. These runs examined how the proposed components worked together, extending the evidence beyond an individual change against a fixed baseline.

The software combination mattered when interpreting a result. A failure during broader integration needed to be investigated with the participating component versions and test conditions in view.

Preparing the hardware for each run

The test environment included hundreds of Linux devices, clusters and FPGA devices with different hardware versions and intended uses. Jenkins labels organised execution resources and matched pipelines with suitable equipment.

Each resource also needed a known starting state. After an execution, it left the available pool and returned only after restoration to a known baseline and successful completion of basic checks. Resources that failed validation remained unavailable.

This established a consistent preparation process before the next run. When a later test failed, the investigation could account for the fact that the equipment had been freshly reset and had passed its initial checks.

Hardware preparation was part of the execution process, alongside selecting the software and running the tests. It provided useful context for assessing the results that followed.

Keeping results tied to the software under test

Across multiple repositories, test stages and hardware environments, a failed pipeline result alone provided limited information.

Eiffel events linked artefacts with build and test activities, execution environments and reports. The pipeline implementation included message bus integration so that each run could be followed within the wider event history.

Those links helped establish which version was under test, what it was combined with, where it ran and which check failed. Logs and reports supplied the details needed to investigate the failure.

The resulting history followed a candidate through the integration process and retained the context behind its results. A candidate’s progress could be assessed alongside the checks it had completed, the software combinations involved and the environments in which it had been tested.