Pipeline build overview

This page documents specifics of different pipeline build scripts, as well as giving an overview of common Groovy code used from different pipeline scripts. Behavior shared across all pipeline builds is described in Jenkins scripts (releng Python module) and Jenkins configuration.

Matrix build launcher

matrix-launcher.groovy is a relatively simple pipeline script where the actual build is performed by a normal matrix build. The pipeline contains the following steps:

  1. In the initial node context, the bootstrap script calls loadMatrixConfigs() and passes the name of the matrix to use. This calls releng.prepare_multi_configuration_build(), and reads the build axis into a local variable.
  2. The bootstrap script calls doBuild() with the name of the matrix job to trigger. The pipeline triggers the matrix job, forwarding all relevant build parameters to it and adding the build configuration axis as an additional parameter.
  3. After the matrix build finishes, the pipeline calls releng.process_multi_configuration_build_results(), which uses the Jenkins REST API to get information about the results of the individual configurations. The information is used to construct a failure message to Gerrit, as well as checking that all configurations were actually built.
  4. The pipeline adds a link to the matrix build to the build summary page (while the build is running, the link can be found from the console log). The summary also includes the result of each configuration, and direct links to the build summary pages of each configuration. The build status of the matrix build is also propagated to the status of the pipeline job.
  5. As the last step in the build, the pipeline sets the URL to post back to Gerrit to point to the matrix build. This means that the presence of this launcher job is mostly invisible during normal usage. Only if the pipeline itself fails before reaching this step, you actually see a link to the launcher job in Gerrit.

Gromacs presubmit pipeline

For now, the gromacs-presubmit.groovy is essentially the same as the matrix launcher script, except that it supports multiple branches. This is done by receiving a prefix of the matrix jobs in doBuild(), and appending the name of the branch (deduced from the refspecs) in the pipeline.

The intention is for this pipeline to expand to cover also other presubmit verification, adding flexibility and reducing the need for separate builds for different purposes.

releng presubmit pipeline

releng-presubmit.groovy speficies steps to run to verify changes to releng repository in Gerrit.

As with gromacs-presubmit.groovy, the intention is for this to assume more responsibility for presubmit verification.

Clang static analysis

clang-analyzer.groovy is a simple pipeline that performs static analysis using Clang. The main reason for using a pipeline build instead of a freestyle job is to make it easy to dynamically decide the node where the analysis runs, depending on which version of the analyzer should be used. The sequence is:

  1. In the initial node context, the pipeline calls utils.read_build_script_config() to get the build options defined in the clang-analyzer.py build script in the source repo. This is stored in a local variable.
  2. The bootstrap script calls doBuild() without parameters. The pipeline allocates a node based on the build options, and runs the clang-analyzer.py build script there with releng.
  3. After the releng script finishes, the pipeline publishes a HTML report produced by the analyzer (if it exists), and scans for compiler warnings from the console log to show them on the build page.

Build & test release tarballs

release.groovy is a more complex pipeline that coordinates the building and testing of the tarballs for a release. The packaging of the tarballs is handled by two separate, non-pipeline Jenkins jobs, one for the source code and one for the regressiontests. The general sequence is:

  1. In addition to common preparation, the pipeline reads a set of configurations to test from release-matrix.txt in the source repo as with matrix builds. It also extracts version information from the source repository (using get-version-info.py build script), since the regressiontests repository does not contain this.
  2. The bootstrap script calls doBuild() with the names of the packaging jobs as parameters.
  3. The pipeline checks the latest successful builds in the packaging builds, and if these are not built from the correct commit, it triggers new builds for them. The regressiontests tarball is built first, and its MD5 sum is checked against the one specified in the source tarball. For a RELEASE build, a mismatch fails the build, otherwise it only produces a note in the console output. The packaging builds also compute MD5 sums for the tarballs, and these are accessible from Jenkins.
  4. After the tarballs are available, the pipeline runs each configuration from the test matrix in parallel, using run_build(), and the standard gromacs.py build script from the source tarball. A summary is posted to the build summary page (for each configuration, on which host it was built and whether it was successful, unstable, or failed), but compiler warnings etc. are currently only available from the console log (available for a single configuration with some browsing under “Pipeline Steps”).
  5. If all tests passed, the pipeline then does a final documentation build from the source tarball, which will produce the HTML pages for the documentation website. The generated pages are available from the Jenkins project page, as well as from a link on the build summary page. If the RELEASE build parameter is set, a tarball containing all the documentation is also archived as an artifact.

In addition to the refspecs to build, the pipeline uses two additional build parameters:

RELEASE
If set, the -dev suffix is stripped from all the tarballs, and from content within them. Note that currently, the -dev suffixes never appear in the generated website, irrespective of this
FORCE_REPACKAGING
If set, the tarballs are rebuilt, even if ones built from the correct refspecs and with the same value of RELEASE is available. This is useful if only releng or Jenkins configuration has changed in a way that influences the tarballs.

The pipeline and the level of testing is still a work-in-progress, but it already covers most of what the earlier builds did, and remaining content should not be too hard to add. Missing functionality is indicated with TODOs in the pipeline script or in the build scripts in the source repo.

On-demand launcher

ondemand.groovy handles builds that are triggered with a [JENKINS] comment from Gerrit. For many cases, the actual builds are done using separate, non-pipeline jobs triggered from the pipeline. The general sequence is:

  1. In the context of the initial checkout, the pipeline uses releng.get_actions_from_triggering_comment() to parse the comment from Gerrit, as well as the initial refspecs. This replaces utils.initBuildRevisions() from other pipelines, but returns exactly the same information to the pipeline script (in addition to the information specific to the on-demand build). This function will also read information from the gromacs repository, e.g., to fill out the matrix options into the returned data structure. It also posts cross-verify messages to Gerrit if needed.
  2. The bootstrap script calls doBuild() without parameters. The pipeline runs the requested builds in parallel, based on the data structure it got in step 1. All relevant build parameters are forwarded. Some actions are handled directly within the pipeline instead of triggering a separate build.
  3. After the builds finish, the pipeline adds links to the triggered builds to the build summary page (while the build is running, the link can be found from the console log). The pipeline then uses releng.do_ondemand_post_build() to construct the message to post back to Gerrit, as well as to perform other actions such as posting cross-verify messages. The combined build status of the builds is also propagated to the status of the pipeline job.

Common Groovy scripts

utils.groovy

TODO

matrixbuild.groovy

TODO

packaging.groovy

TODO