diff options
Diffstat (limited to 'documentation/test-manual/test-process.rst')
-rw-r--r-- | documentation/test-manual/test-process.rst | 101 |
1 files changed, 101 insertions, 0 deletions
diff --git a/documentation/test-manual/test-process.rst b/documentation/test-manual/test-process.rst new file mode 100644 index 0000000000..8a5e29d922 --- /dev/null +++ b/documentation/test-manual/test-process.rst | |||
@@ -0,0 +1,101 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-SA-2.0-UK | ||
2 | |||
3 | *********************************** | ||
4 | Project Testing and Release Process | ||
5 | *********************************** | ||
6 | |||
7 | Day to Day Development | ||
8 | ====================== | ||
9 | |||
10 | This section details how the project tests changes, through automation | ||
11 | on the Autobuilder or with the assistance of QA teams, through to making | ||
12 | releases. | ||
13 | |||
14 | The project aims to test changes against our test matrix before those | ||
15 | changes are merged into the master branch. As such, changes are queued | ||
16 | up in batches either in the ``master-next`` branch in the main trees, or | ||
17 | in user trees such as ``ross/mut`` in ``poky-contrib`` (Ross Burton | ||
18 | helps review and test patches and this is his testing tree). | ||
19 | |||
20 | We have two broad categories of test builds, including "full" and | ||
21 | "quick". On the Autobuilder, these can be seen as "a-quick" and | ||
22 | "a-full", simply for ease of sorting in the UI. Use our Autobuilder | ||
23 | console view to see where me manage most test-related items, available | ||
24 | at: :yocto_ab:`/typhoon/#/console`. | ||
25 | |||
26 | Builds are triggered manually when the test branches are ready. The | ||
27 | builds are monitored by the SWAT team. For additional information, see | ||
28 | :yocto_wiki:`/Yocto_Build_Failure_Swat_Team`. | ||
29 | If successful, the changes would usually be merged to the ``master`` | ||
30 | branch. If not successful, someone would respond to the changes on the | ||
31 | mailing list explaining that there was a failure in testing. The choice | ||
32 | of quick or full would depend on the type of changes and the speed with | ||
33 | which the result was required. | ||
34 | |||
35 | The Autobuilder does build the ``master`` branch once daily for several | ||
36 | reasons, in particular, to ensure the current ``master`` branch does | ||
37 | build, but also to keep ``yocto-testresults`` | ||
38 | (:yocto_git:`/yocto-testresults/`), | ||
39 | buildhistory | ||
40 | (:yocto_git:`/poky-buildhistory/`), and | ||
41 | our sstate up to date. On the weekend, there is a master-next build | ||
42 | instead to ensure the test results are updated for the less frequently | ||
43 | run targets. | ||
44 | |||
45 | Performance builds (buildperf-\* targets in the console) are triggered | ||
46 | separately every six hours and automatically push their results to the | ||
47 | buildstats repository at: | ||
48 | :yocto_git:`/yocto-buildstats/`. | ||
49 | |||
50 | The 'quick' targets have been selected to be the ones which catch the | ||
51 | most failures or give the most valuable data. We run 'fast' ptests in | ||
52 | this case for example but not the ones which take a long time. The quick | ||
53 | target doesn't include \*-lsb builds for all architectures, some world | ||
54 | builds and doesn't trigger performance tests or ltp testing. The full | ||
55 | build includes all these things and is slower but more comprehensive. | ||
56 | |||
57 | Release Builds | ||
58 | ============== | ||
59 | |||
60 | The project typically has two major releases a year with a six month | ||
61 | cadence in April and October. Between these there would be a number of | ||
62 | milestone releases (usually four) with the final one being stablization | ||
63 | only along with point releases of our stable branches. | ||
64 | |||
65 | The build and release process for these project releases is similar to | ||
66 | that in `Day to Day Development <#test-daily-devel>`__, in that the | ||
67 | a-full target of the Autobuilder is used but in addition the form is | ||
68 | configured to generate and publish artefacts and the milestone number, | ||
69 | version, release candidate number and other information is entered. The | ||
70 | box to "generate an email to QA"is also checked. | ||
71 | |||
72 | When the build completes, an email is sent out using the send-qa-email | ||
73 | script in the ``yocto-autobuilder-helper`` repository to the list of | ||
74 | people configured for that release. Release builds are placed into a | ||
75 | directory in https://autobuilder.yocto.io/pub/releases on the | ||
76 | Autobuilder which is included in the email. The process from here is | ||
77 | more manual and control is effectively passed to release engineering. | ||
78 | The next steps include: | ||
79 | |||
80 | - QA teams respond to the email saying which tests they plan to run and | ||
81 | when the results will be available. | ||
82 | |||
83 | - QA teams run their tests and share their results in the yocto- | ||
84 | testresults-contrib repository, along with a summary of their | ||
85 | findings. | ||
86 | |||
87 | - Release engineering prepare the release as per their process. | ||
88 | |||
89 | - Test results from the QA teams are included into the release in | ||
90 | separate directories and also uploaded to the yocto-testresults | ||
91 | repository alongside the other test results for the given revision. | ||
92 | |||
93 | - The QA report in the final release is regenerated using resulttool to | ||
94 | include the new test results and the test summaries from the teams | ||
95 | (as headers to the generated report). | ||
96 | |||
97 | - The release is checked against the release checklist and release | ||
98 | readiness criteria. | ||
99 | |||
100 | - A final decision on whether to release is made by the YP TSC who have | ||
101 | final oversight on release readiness. | ||