From 444666faa47186d30d222eb0ebe4a25f1649f6e5 Mon Sep 17 00:00:00 2001 From: Mark Morton Date: Tue, 16 Jun 2020 14:14:50 -0700 Subject: New source files and Makefile update for Test Manual (From yocto-docs rev: d7cff640569a5772f3c366b4136762628fca534d) Signed-off-by: Mark Morton Signed-off-by: Richard Purdie --- .../test-manual/test-manual-test-process.xml | 109 +++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 documentation/test-manual/test-manual-test-process.xml (limited to 'documentation/test-manual/test-manual-test-process.xml') diff --git a/documentation/test-manual/test-manual-test-process.xml b/documentation/test-manual/test-manual-test-process.xml new file mode 100644 index 0000000000..0b5036cd2c --- /dev/null +++ b/documentation/test-manual/test-manual-test-process.xml @@ -0,0 +1,109 @@ + %poky; ] > + + + +Project Testing and Release Process +
+ Day to Day Development + + This section details how the project tests changes, through automation on the + Autobuilder or with the assistance of QA teams, through to making releases. + + The project aims to test changes against our test matrix before those changes are + merged into the master branch. As such, changes are queued up in batches either in the + master-next branch in the main trees, or in user trees such as + ross/mut in poky-contrib (Ross Burton + helps review and test patches and this is his testing tree). + We have two broad categories of test builds, including "full" and "quick". On the + Autobuilder, these can be seen as "a-quick" and "a-full", simply for ease of sorting in + the UI. Use our Autobuilder console view to see where me manage most test-related items, + available at: https://autobuilder.yoctoproject.org/typhoon/#/console. + Builds are triggered manually when the test branches are ready. The builds are + monitored by the SWAT team. For additional information, see https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team. If + successful, the changes would usually be merged to the master + branch. If not successful, someone would respond to the changes on the mailing list + explaining that there was a failure in testing. The choice of quick or full would depend + on the type of changes and the speed with which the result was required. + The Autobuilder does build the master branch once daily for + several reasons, in particular, to ensure the current master branch + does build, but also to keep yocto-testresults (http://git.yoctoproject.org/cgit.cgi/yocto-testresults/), buildhistory + (http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/), + and our sstate up to date. On the weekend, there is a master-next build instead to + ensure the test results are updated for the less frequently run targets. + Performance builds (buildperf-* targets in the console) are triggered separately every + six hours and automatically push their results to the buildstats repository at: http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/. + The 'quick' targets have been selected to be the ones which catch the most failures or + give the most valuable data. We run 'fast' ptests in this case for example but not the + ones which take a long time. The quick target doesn't include *-lsb builds for all + architectures, some world builds and doesn't trigger performance tests or ltp testing. + The full build includes all these things and is slower but more comprehensive. +
+ +
+ Release Builds + + The project typically has two major releases a year with a six month cadence in April + and October. Between these there would be a number of milestone releases (usually four) + with the final one being stablization only along with point releases of our stable + branches. + The build and release process for these project releases is similar to that in Day to Day Development, in that the a-full target + of the Autobuilder is used but in addition the form is configured to generate and + publish artefacts and the milestone number, version, release candidate number and other + information is entered. The box to "generate an email to QA"is also checked. + When the build completes, an email is sent out using the send-qa-email script in the + yocto-autobuilder-helper repository to the list of people + configured for that release. Release builds are placed into a directory in https://autobuilder.yocto.io/pub/releases on the Autobuilder which + is included in the email. The process from here is more manual and control is + effectively passed to release engineering. The next steps include: + + QA teams respond to the email saying which tests they plan to run and when + the results will be available. + + + QA teams run their tests and share their results in the yocto- + testresults-contrib repository, along with a summary of their findings. + + + + Release engineering prepare the release as per their process. + + + Test results from the QA teams are included into the release in separate + directories and also uploaded to the yocto-testresults repository alongside + the other test results for the given revision. + + + The QA report in the final release is regenerated using resulttool to + include the new test results and the test summaries from the teams (as + headers to the generated report). + + + The release is checked against the release checklist and release readiness + criteria. + + + A final decision on whether to release is made by the YP TSC who have + final oversight on release readiness. + + +
+ + + + + + + + + + -- cgit v1.2.3-54-g00ecf