releng Python API

Build script definition

The build script is required to provide one function:

do_build(context)

Called to run the actual build. The context parameter is an instance of BuildContext, and can be used to access the build environment and to interact with Jenkins. The function can signal fatal build errors by raising BuildError directly; typically, this is done by methods in BuildContext if they fail to execute the requested commands.

When the function is called, the current working directory is set to the build directory (whether the build is in- or out-of-source).

The build script can also set a few global variables to influence the behavior of the build:

build_options

If this list value is set to a non-empty list, then these build options are used to initialize the build environment. Useful for non-matrix builds that want to, e.g., specify the compiler to use.

build_out_of_source

If this boolean value is set to True, the build will be executed out-of-source. By default, the build will be in-source.

extra_options

If this dictionary is set, it declares additional build options that the script understands. This can be used to declare options that only influence the build script; releng declares only options that affect the build environment or the build host assignment. Syntax is as follows:

extra_options = {
    'opt': Option.simple,
    'opt-bool': Option.bool,
    'opt-str': Option.string
}

The values of the build options can be read from context.opts in do_build(). See OptionTypes documentation for the available option types: in the build script, Option is bound to OptionTypes.

Technically, the value in the dictionary is a callable that gets called with the name of the option to create an internal handler class for processing the option.

extra_projects

If this list value is set to a non-empty list, then these repositories are also checked out before executing the build. releng and gromacs repositories are always checked out. Currently, only Project.REGRESSIONTESTS makes sense to specify here.

When the build script is loaded, various enums from the releng package are injected into the global scope to make them easy to access.

API for build scripts

The build script gets input and perfoms most tasks by using data and methods in a BuildContext instance:

class releng.context.BuildContext(factory, job_type, opts, extra_options)

Top-level interface for build scripts to the releng package.

job_type

JobType – Type/scope of the build job (e.g., per-patchset, nightly).

env

BuildEnvironment – Access to build environment details like paths to executables. Many of the environment properties, such as selecting the compiler, are handled by the build context transparently, without the build script needing to access this.

opts

BuildOptions – Access to all build options.

params

BuildParameters – Access to Jenkins build parameters (through environment variables).

workspace

Workspace – Access to the build workspace. Can be used to get paths to various parts in the workspace for changing directories and for producing log files.

build_target(target=None, parallel=True, keep_going=False, target_descr=None, failure_string=None, continue_on_failure=False)

Builds a given target.

run_cmake() must have been called to generate the build system.

Parameters:

• target (Optional[str]) – Name of the target to build. If None, the default (all) target is built.
• parallel (Optional[bool]) – Whether parallel building is supported.
• keep_going (Optional[bool]) – Whether to continue building after first error.
• target_descr (str or None) – If given, customizes the error message when the target fails to build. Should fit the initial part of the sentence “… failed to build”. Ignored if failure_string is specified.
• failure_string (str or None) – If given, this message is used as the failure message if the target fails to build.
• continue_on_failure (Optional[bool]) – If True and the target fails to build, the failure is only reported and self.failed is set to True.

Raises:

BuildError – If the target fails to build, and continue_on_failure is not specified.

chdir(path)

Changes the working directory for subsequent run_cmd() calls.

compute_md5(path)

Computes MD5 hash of a file.

Parameters:path (str) – Path to the file to compute the hash for. Returns:String with the computed hash in hexadecimal. Return type:str

failed

Whether the build has already failed.

This can be used in combination with build_target() argument continue_on_failure if the build script needs to test whether some previous target already has built. If the build script wants to stop the build in such a case, it can simply return; the build will always be marked failed if this property is True.

get_doc_cmake_options(doxygen_version, sphinx_version)

Returns non-GROMACS-specific CMake options to set for documentation builds.

make_archive(path, root_dir=None, use_git=False, prefix=None)

Creates a tar.gz archive.

Parameters:

• path (str) – Path to the archive to create without extension.
• root_dir (str) – Root directory from which the archive should be created.

mark_unstable(reason, details=None)

Marks the build unstable.

Parameters:

• reason (str) – Reason printed to the build console log for the failure.
• details (Optional[List[str]]) – Reason(s) reported back to Gerrit. If not provided, reason is used.

process_clang_analyzer_results()

Processes results from clang analyzer.

process_coverage_results(exclude=None)

Processes results from coverage runs.

Uses gcovr to process all coverage files found in the workspace (from running a build compiled with –coverage).

Parameters:exclude (List[str]) – Exclusions to pass to gcovr -e (regexs).

process_cppcheck_results(xml_pattern)

Processes results from cppcheck.

This method massages the XML output into a form the Jenkins CppCheck plugin can handle. It could also contain logic that would report additional information about the issues back to Jenkins.

Parameters:

• xml_pattern (str) – Pattern that matches all XML files produced by
• cppcheck. –

publish_logs(logs, category=None)

Copies provided log(s) to Jenkins.

This should be used for any files that are produced during the build and that need to be parsed by Jenkins (except for special cases where a separate method is provided, such as process_cppcheck_results()). This allows Jenkins configuration stay the same even if the files are relocated because of build system or repository reorganization. Alternatively, the build script can use workspace.get_path_for_logfile() to directly produce the log files into an invariant location.

Parameters:

• logs (List[str]) – Paths to files that need to be copied.
• category (Optional[str]) – Category for the log file. Log files in the same category are put into a common subdirectory (with the name of the category), allowing Jenkins to glob them for, e.g., parsing warnings.

read_cmake_variable_file(path)

Reads a file with CMake variable declarations (set commands).

Parameters:path (str) – Path to the file to read. Returns:variables found from the file, with their values. Return type:Dict

replace_in_file(path, pattern, repl)

Performs re.sub() on contents of a file.

Parameters:

• path (str) – Path to the file to process.
• pattern (str) – Pattern to replace.
• repl – See re.sub()

run_cmake(options)

Runs CMake with the provided options.

Options from the environment, such as for selecting compilers, are added automatically.

The working directory should be the build directory. Currently, does not support running CMake multiple times.

Parameters:options (Dict[str,str]) – Dictionary of macro definitions to pass to CMake using -D. Raises:BuildError – If CMake fails to configure the build system.

run_cmd(cmd, ignore_failure=False, use_return_code=False, use_output=False, failure_message=None, **kwargs)

Runs a command via subprocess.

This wraps subprocess.call() and check_call() with error-handling code and other generic handling such as ensuring proper output flushing and using bash as the shell on Unix.

Any arguments accepted by subprocess.call() or check_call() can also be passed, e.g. cwd or env to make such calls in stateless ways.

Parameters:

• cmd (str/list) – Command to execute (as for subprocess.call()).
• ignore_failure (Optional[bool]) – If True, failure to run the command is ignored.
• use_return_code (Optional[bool]) – If True, exit code from the command is returned. Otherwise, non-zero return code fails the build unless ignore_failure is set.
• use_output (Optional[bool]) – If True, the output from the command is returned. Mutually exclusive with use_return_code.
• failure_message (Optional[str]) – If set, provides a friendly message about what in the build fails if this command fails. This will be reported back to Gerrit.

Returns:

Command return code (if use_return_code=True).

Return type:

int

run_ctest(args, memcheck=False, failure_string=None)

Runs tests using CTest.

The build is marked unstable if any test fails.

Parameters:

• args (List[str]) – Additional arguments to pass to CTest.
• memcheck (Optional[bool]) – If true, run CTest with a memory checker.
• failure_string (Optional[str]) – If give, this message is used as the failure message reported to Gerrit if the tests fail.

set_version_info(version, regtest_md5sum)

Provides source version information from a build script.

This method supports the get-version-info.py build script and allows that script to pass out the version information without writing it into a file that would then be read back.

write_package_info(project, file_name, version)

Writes an information file for a tar package.

The file has a specific format that allows reading the information in a downstream build, and using the package instead of a git checkout.

Parameters:

• project (Project) – Project for which the package is done.
• file_name (str) – Name of the package file (with extension, without any path). Currently the file needs to be in the current working directory.
• version (str) – Version for the package.

write_property_file(path, values)

Writes a property file at given path.

Parameters:

• path (str) – Path to the file to write.
• values (Dict) – Dictionary of key/value pairs to write.

The build context contains attributes of the following classes to access additional information:

class releng.environment.BuildEnvironment(factory)

Provides access to the build environment.

Most details of the build environment are handled transparently based on the provided build options, and the build script does not need to consider this. For build scripts, the main interface this class provides is to find locations of some special executables (such as cppcheck) that may be needed for the build. Compiler selection is handled without special action from build scripts.

In rare cases, the build scripts may benefit from inspecting the attributes in this class to determine, e.g., the operating system running the build or the compiler being used.

system

System – Operating system of the build node.

compiler

Compiler or None – Selected compiler.

compiler_version

string – Version number for the selected compiler.

c_compiler

str or None – Name of the C compiler executable.

cxx_compiler

str or None – Name of the C++ compiler executable.

libcxx_version

str or None – Version of libc++ to use

doxygen_command

str – Name of the doxygen executable.

gcov_command

str – Name of the gcov executable.

cmake_command

str – Name of the CMake executable.

ctest_command

str – Name of the CTest executable.

cmake_version

str – Version of the CMake executable.

cmake_generator

str or None – CMake generator being used.

armhpc_version

str or None – The version of the ARM HPC toolchain.

armpl_dir

str or None – the ARM Perf Libraries directory

cuda_root

str or None – Root of the CUDA toolkit being used (for passing to CUDA_TOOLKIT_ROOT_DIR CMake option).

opencl_version

str or None – OpenCL API version expected to be detected.

cuda_host_compiler

str or None – Full path to the host compiler used with CUDA (for passing to CUDA_HOST_COMPILER CMake option).

amdappsdk_root

str or None – Root of the AMD SDK being used (for using as AMDAPPSDKROOT environment variable).

extra_cmake_options

Dict[str, str] – Additional options to pass to CMake.

append_path_env(path)

Appends a path to the executable search path (PATH).

get_cppcheck_command(version)

Returns path to the cppcheck executable of given version.

Parameters:version (str) – cppcheck version to use.

get_doxygen_command(version)

Returns path to the Doxygen executable of given version.

Parameters:version (str) – Doxygen version to use.

get_uncrustify_command()

Returns path to the uncrustify executable.

prepend_path_env(path)

Prepends a path to the executable search path (PATH).

set_env_var(variable, value)

Sets environment variable to be used for further commands.

All subsequent commands run with BuildContext.run_cmd() etc. will use the environment variable.

Parameters:

• variable (str) – Name of environment variable to set.
• value (str) – Value to set the variable to. As a convenience, if the value is None, nothing is done.

class releng.options.BuildOptions(handlers, opts)

Values for all build options.

This class provides read-only access to the values of all build options. A build option named mdrun-only is accessible as opts.mdrun_only and opts['mdrun-only'], whichever is more convenient. The keys for all build options are always available. If an option is not specified, the corresponding value is None.

For simple flag options that can only be set or unset, the stored value is True if the option is set.

For boolean options, the stored value is False or True. For example, no-mpi and mpi=no are both stored as opts.mpi == True.

For options like gcc-4.8, the value is stored as opts.gcc == '4.8'. Similarly, build-jobs=2 is stored as opts.build_jobs == '2'.

class releng.integration.BuildParameters(factory)

Access to build parameters.

class releng.workspace.Workspace(factory)

Provides access to set up, query, and act within the build workspace, particularly involving operations on the git repositories associated with the projects.

Methods are provided for accessing the build directory (whether in- or out-of-source), as well as the root directories of all checked-out projects. Also, methods to access a common log directory (for logs that need to be post-processed in Jenkins) are provided. Implements functionality for updating commits with new files.

root

str – Root directory of the workspace.

install_dir

str – Directory for test installation.

build_dir

Build directory for building gromacs.

Returns either the gromacs project directory or a separate build directory, depending on whether the build is in- or out-of-source.

clean_build_dir()

Ensures that the current build dir is in the initial state (empty).

get_log_dir(category=None)

Returns directory for log files.

The directory is created if necessary.

Parameters:category (Optional[str]) – Category for the log directory. Log files in the same category are put into a common subdirectory (with the name of the category), allowing Jenkins to glob them for, e.g., parsing warnings. Returns:Absolute path for the requested log directory. Return type:str

get_path_for_logfile(name, category=None)

Returns path for producing a log file in a common location.

Directories are created as necessary, but the caller is responsible of creating the actual file.

Parameters:

• name (str) – Name for the log file without directory components, but with extension.
• category (Optional[str]) – Category for the log file. Log files in the same category are put into a common subdirectory (with the name of the category), allowing Jenkins to glob them for, e.g., parsing warnings.

Returns:

Absolute path for the requested log.

Return type:

str

get_project_dir(project)

Returns project directory of a given project.

Parameters:project (Project) – Project whose directory should be returned. Returns:Absolute path to the project directory. Return type:str

upload_revision(project, file_glob=’*’)

Upload a new version of the patch that triggered this build, but only if files in the glob changed and it came from the specified project.

Parameters:

• project (Project) – Enum value to choose which project might be updated
• file_glob (str) – glob describing the files to add to the patch

API for Jenkins

The following functions from the releng package are intended to be called from scripts in Jenkins build configuration or from workflow scripts (see Jenkins configuration).

releng.run_build(build, job_type, opts, project=’gromacs’)

Main entry point for Jenkins builds.

Runs the build with the given build script and build parameters. Before calling this script, the job should have checked out the releng repository to a releng/ subdirectory of the workspace, and the repository that triggered the build, similarly in a subdirectory of the workspace.

See Jenkins scripts (releng Python module) for more details on the general build organization.

Parameters:

• build (str) – Build type identifying the build script to use. Names without directory separators are interpreted as gromacs/admin/builds/build.py, i.e., loaded from the main repository.
• job_type (JobType) – Type/scope of the job that can, e.g., influence the scope of testing. Not all build scripts use the value.
• opts (List[str]) – This is mainly intended for multi-configuration builds. Build scripts not intended for such builds may simply ignore most of the parameters that can be influenced by these options.

releng.read_build_script_config(script_name)

Reads build options specified in a build script.

Parameters:script_name (str) – Name of the build script (see run_build()).

releng.prepare_multi_configuration_build(configfile)

Main entry point for preparing matrix builds.

Reads a file with configurations to use (one configuration per line, with a list of space-separated build options on each line; comments starting with # and empty lines ignored).

Parameters:configfile (str) – File that contains the configurations to use. Names without directory separators are interpreted as gromacs/admin/builds/configfile.txt.

releng.get_actions_from_triggering_comment()

Processes Gerrit comment that triggered the build.

Parses the comment that triggered an on-demand build and returns a structure that tells the workflow build what it needs to do.

releng.do_ondemand_post_build(inputfile)

Does processing after on-demand builds have finished.

Reads a JSON file that provides information about the builds (and things forwarded from the output of get_actions_from_triggering_comment()), and returns a structure that specifies what to post back to Gerrit.

Can also perform other actions related to processing the build results, such as posting cross-verify messages.

Parameters:inputfile (str) – File to read the input from, relative to working dir.

releng.get_build_revisions()

Provides information about revisions used in the build.

Returns a structure that provides a list of projects and their revisions used in this build.

releng.read_source_version_info()

Reads version info from the source repository.

Returns a structure that provides version information from the source repository.