diff options
Diffstat (limited to 'documentation/test-manual/test-manual-test-process.xml')
| -rw-r--r-- | documentation/test-manual/test-manual-test-process.xml | 110 |
1 files changed, 0 insertions, 110 deletions
diff --git a/documentation/test-manual/test-manual-test-process.xml b/documentation/test-manual/test-manual-test-process.xml deleted file mode 100644 index 6e2157c621..0000000000 --- a/documentation/test-manual/test-manual-test-process.xml +++ /dev/null | |||
| @@ -1,110 +0,0 @@ | |||
| 1 | <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" | ||
| 2 | "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" | ||
| 3 | [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > | ||
| 4 | <!--SPDX-License-Identifier: CC-BY-2.0-UK--> | ||
| 5 | |||
| 6 | <chapter id='test-manual-test-process'> | ||
| 7 | |||
| 8 | <title>Project Testing and Release Process</title> | ||
| 9 | <section id='test-daily-devel'> | ||
| 10 | <title>Day to Day Development</title> | ||
| 11 | |||
| 12 | <para>This section details how the project tests changes, through automation on the | ||
| 13 | Autobuilder or with the assistance of QA teams, through to making releases.</para> | ||
| 14 | |||
| 15 | <para>The project aims to test changes against our test matrix before those changes are | ||
| 16 | merged into the master branch. As such, changes are queued up in batches either in the | ||
| 17 | <filename>master-next</filename> branch in the main trees, or in user trees such as | ||
| 18 | <filename>ross/mut</filename> in <filename>poky-contrib</filename> (Ross Burton | ||
| 19 | helps review and test patches and this is his testing tree).</para> | ||
| 20 | <para>We have two broad categories of test builds, including "full" and "quick". On the | ||
| 21 | Autobuilder, these can be seen as "a-quick" and "a-full", simply for ease of sorting in | ||
| 22 | the UI. Use our Autobuilder console view to see where me manage most test-related items, | ||
| 23 | available at: <link linkend="" | ||
| 24 | >https://autobuilder.yoctoproject.org/typhoon/#/console</link>.</para> | ||
| 25 | <para>Builds are triggered manually when the test branches are ready. The builds are | ||
| 26 | monitored by the SWAT team. For additional information, see <link linkend="" | ||
| 27 | >https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team</link>. If | ||
| 28 | successful, the changes would usually be merged to the <filename>master</filename> | ||
| 29 | branch. If not successful, someone would respond to the changes on the mailing list | ||
| 30 | explaining that there was a failure in testing. The choice of quick or full would depend | ||
| 31 | on the type of changes and the speed with which the result was required.</para> | ||
| 32 | <para>The Autobuilder does build the <filename>master</filename> branch once daily for | ||
| 33 | several reasons, in particular, to ensure the current <filename>master</filename> branch | ||
| 34 | does build, but also to keep <filename>yocto-testresults</filename> (<link linkend="" | ||
| 35 | >http://git.yoctoproject.org/cgit.cgi/yocto-testresults/</link>), buildhistory | ||
| 36 | (<link linkend="">http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/</link>), | ||
| 37 | and our sstate up to date. On the weekend, there is a master-next build instead to | ||
| 38 | ensure the test results are updated for the less frequently run targets.</para> | ||
| 39 | <para>Performance builds (buildperf-* targets in the console) are triggered separately every | ||
| 40 | six hours and automatically push their results to the buildstats repository at: <link | ||
| 41 | linkend="">http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/</link>. </para> | ||
| 42 | <para>The 'quick' targets have been selected to be the ones which catch the most failures or | ||
| 43 | give the most valuable data. We run 'fast' ptests in this case for example but not the | ||
| 44 | ones which take a long time. The quick target doesn't include *-lsb builds for all | ||
| 45 | architectures, some world builds and doesn't trigger performance tests or ltp testing. | ||
| 46 | The full build includes all these things and is slower but more comprehensive.</para> | ||
| 47 | </section> | ||
| 48 | |||
| 49 | <section id='test-yocto-project-autobuilder-overview'> | ||
| 50 | <title>Release Builds</title> | ||
| 51 | |||
| 52 | <para>The project typically has two major releases a year with a six month cadence in April | ||
| 53 | and October. Between these there would be a number of milestone releases (usually four) | ||
| 54 | with the final one being stablization only along with point releases of our stable | ||
| 55 | branches.</para> | ||
| 56 | <para>The build and release process for these project releases is similar to that in <link | ||
| 57 | linkend="test-daily-devel">Day to Day Development</link>, in that the a-full target | ||
| 58 | of the Autobuilder is used but in addition the form is configured to generate and | ||
| 59 | publish artefacts and the milestone number, version, release candidate number and other | ||
| 60 | information is entered. The box to "generate an email to QA"is also checked.</para> | ||
| 61 | <para>When the build completes, an email is sent out using the send-qa-email script in the | ||
| 62 | <filename>yocto-autobuilder-helper</filename> repository to the list of people | ||
| 63 | configured for that release. Release builds are placed into a directory in <link | ||
| 64 | linkend="">https://autobuilder.yocto.io/pub/releases</link> on the Autobuilder which | ||
| 65 | is included in the email. The process from here is more manual and control is | ||
| 66 | effectively passed to release engineering. The next steps include:<itemizedlist> | ||
| 67 | <listitem> | ||
| 68 | <para>QA teams respond to the email saying which tests they plan to run and when | ||
| 69 | the results will be available.</para> | ||
| 70 | </listitem> | ||
| 71 | <listitem> | ||
| 72 | <para>QA teams run their tests and share their results in the yocto- | ||
| 73 | testresults-contrib repository, along with a summary of their findings. | ||
| 74 | </para> | ||
| 75 | </listitem> | ||
| 76 | <listitem> | ||
| 77 | <para>Release engineering prepare the release as per their process. </para> | ||
| 78 | </listitem> | ||
| 79 | <listitem> | ||
| 80 | <para>Test results from the QA teams are included into the release in separate | ||
| 81 | directories and also uploaded to the yocto-testresults repository alongside | ||
| 82 | the other test results for the given revision.</para> | ||
| 83 | </listitem> | ||
| 84 | <listitem> | ||
| 85 | <para>The QA report in the final release is regenerated using resulttool to | ||
| 86 | include the new test results and the test summaries from the teams (as | ||
| 87 | headers to the generated report).</para> | ||
| 88 | </listitem> | ||
| 89 | <listitem> | ||
| 90 | <para>The release is checked against the release checklist and release readiness | ||
| 91 | criteria.</para> | ||
| 92 | </listitem> | ||
| 93 | <listitem> | ||
| 94 | <para>A final decision on whether to release is made by the YP TSC who have | ||
| 95 | final oversight on release readiness.</para> | ||
| 96 | </listitem> | ||
| 97 | </itemizedlist></para> | ||
| 98 | </section> | ||
| 99 | |||
| 100 | |||
| 101 | |||
| 102 | |||
| 103 | |||
| 104 | |||
| 105 | |||
| 106 | |||
| 107 | </chapter> | ||
| 108 | <!-- | ||
| 109 | vim: expandtab tw=80 ts=4 | ||
| 110 | --> | ||