Yocto Project Test Environment Manual

Welcome to the Yocto Project Test Environment Manual! This manual is a work in progress. The manual contains information about the testing environment used by the Yocto Project to make sure each major and minor release works as planned. Other organizations can leverage off the process and testing environment used by the Yocto Project to create their own automated, production test environment.

Currently, the Yocto Project Test Environment Manual has no projected release date. This manual is a work-in-progress and is being initially loaded with information from the README files and notes from key engineers:

The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and procedures used by the Yocto Project to test released software across supported hardware in an automated and regular fashion. Basically, during the development of a Yocto Project release, the Autobuilder tests if things work. The Autobuilder builds all test targets and runs all the tests.

The Yocto Project uses the unpatched Buildbot Nine to drive its integration and testing. Buildbot Nine has a plug-in interface that the Yocto Project customizes using code from the yocto-autobuilder2 repository. The resulting customized UI plug-in allows you to visualize builds in a way suited to the project.

A "helper" layer provides configuration and job management through scripts found in the yocto-autobuilder-helper repository. The helper layer contains the bulk of the build configuration information and is release-specific, which makes it highly customizable on a per-project basis. The layer is CI system-agnostic and contains a number of helper scripts that can generate build configurations from simple JSON files.

The following figure shows the Yocto Project Autobuilder stack with a topology that includes a controller and a cluster of workers:

Two kinds of tests exist within the Yocto Project:

Beyond these types of testing, the Autobuilder tests different pieces by using the following types of tests:

Tests map into the codebase as follows:

This section provides example tests for each of the tests listed in the How Tests Map to Areas of Code" section.

The following is going to be the replacement content for the section on "Nightly Builds". Not sure what we are going to call these builds. We need a name to replace "Nightly Builds".

Here is the content from Richards email:

     In 1.6, we actually dropped the "nightly" bit pretty much everywhere.
     They are now named MACHINE or MACHINE-DISTRO, e.g. qemuarm or qemuarm-
     lsb (which tests poky-lsb with qemuarm). We now parallelise not just
     architecture but by machine so machine and real hardware are now
     separate. The flow is therefore to build the images+sdks, then test the
     images+sdks, trying to do as much as possible in parallel.

     We have two types of build trigger, "quick" and "full". quick runs all
     the things which commonly fail and one random oe-selftest. "full" runs
     all our targets, runs oe-selftest on all distros and includes ptest and
     build performance tests. Its slower but more complete and would be used
     for release builds.
            

Automatic testing is based on the workers executing builds using Buildbot Nine configured for specific build jobs triggered in an automatic and regular fashion. Worker Configuration and triggering is accomplished through the Yocto Project Autobuilder layer and a set of helper scripts.

The configuration and helper scripts have as little code and as few custom Buildbot extensions as possible. The configuration collects required input from the user to furnish the helper scripts with the input needed for workers to accomplish their builds. The input consists of minimal user-customizable parameters used to trigger the helper build scripts.

Each builder maps to a named configuration in the helper scripts. The configuration is created with the steps and properties required to invoke the helper scripts for a worker's builds.

Each worker has a custom scheduler created for it and contains parameters configured for the scheduler that can supply the custom versions of the required values for the helper script parameters.

Following is the code layout for the Autobuilder:

The goal is to keep custom code minimized throughout the Autobuilder. The few customizations implemented support the Yocto Project Autobuilder Helper Script workflows and help replicate the workflows established with the Yocto Autobuilder layer. In particular, the following files accomplish this customization:

Steps to deploy the Yocto Project Autobuilder assume each target system has a copy of Buildbot installed. Additionally, various pieces of functionality require that a copy of the Autobuilder Helper Scripts (i.e. yocto-autobuilder-helper) is available in the home directory at ~/yocto-autobuilder-helper of the user running Buildbot.

The following sections provide steps for Yocto Autobuilder deployment.

If you plan on using the Yocto Project Autobuilder to run headless sanity testing, you need to do the following:

The production Yocto Autobuilder uses a cluster of build workers. The cluster shares the same SSTATE_DIR and DL_DIR through an NFS4 mounted Network Attached Storage (NAS). The main nightly trigger pre-populates the DL_DIR, which allows the workers to not have to deal with a lot of downloading. In theory, you could also run your build workers with NO_NETWORK to enforce a single point for populating DL_DIR.

Running multiple build workers is fairly simple, but does require some setup:

Build History is used to track changes to packages and images. By default, the Autobuilder does not collect build history. The production Autobuilder does have this functionality enabled.

Setting up build history requires the following steps:

The helper scripts work in conjunction with the Yocto Project Autobuilder. These scripts do the actual build configuration and execution for tests on a per release basis.

You can use pre-commit-hook.sh to verify the JSON file before committing it. Create a symbolic link as follows:

     $ ln -s ../../scripts/pre-commit-hook.sh .git/hooks/pre-commit
            

Most users will have to customize the helper script repository to meet their needs. The repository is located at http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper. The scripts themselves should be more generically reusable. The config.json is less reusable as it represents the Yocto Project Autobuilder test matrix.

Two customization options are possible: 1) variable substitution, and 2) overlaying configuration files. The standard config.json minimally attempts to allow substitution of the paths. The helper script repository includes a local-example.json to show how you could override these from a separate configuration file. Pass the following into the environment of the autobuilder:

     ABHELPER_JSON="config.json local-example.json"
            

As another example, you could also pass the following into the environment:

     ABHELPER_JSON="config.json /some/location/local.json"