diff options
Diffstat (limited to 'documentation/test-manual/test-manual-test-process.rst')
-rw-r--r-- | documentation/test-manual/test-manual-test-process.rst | 103 |
1 files changed, 0 insertions, 103 deletions
diff --git a/documentation/test-manual/test-manual-test-process.rst b/documentation/test-manual/test-manual-test-process.rst deleted file mode 100644 index 96e71bf314..0000000000 --- a/documentation/test-manual/test-manual-test-process.rst +++ /dev/null | |||
@@ -1,103 +0,0 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-2.0-UK | ||
2 | |||
3 | *********************************** | ||
4 | Project Testing and Release Process | ||
5 | *********************************** | ||
6 | |||
7 | .. _test-daily-devel: | ||
8 | |||
9 | Day to Day Development | ||
10 | ====================== | ||
11 | |||
12 | This section details how the project tests changes, through automation | ||
13 | on the Autobuilder or with the assistance of QA teams, through to making | ||
14 | releases. | ||
15 | |||
16 | The project aims to test changes against our test matrix before those | ||
17 | changes are merged into the master branch. As such, changes are queued | ||
18 | up in batches either in the ``master-next`` branch in the main trees, or | ||
19 | in user trees such as ``ross/mut`` in ``poky-contrib`` (Ross Burton | ||
20 | helps review and test patches and this is his testing tree). | ||
21 | |||
22 | We have two broad categories of test builds, including "full" and | ||
23 | "quick". On the Autobuilder, these can be seen as "a-quick" and | ||
24 | "a-full", simply for ease of sorting in the UI. Use our Autobuilder | ||
25 | console view to see where me manage most test-related items, available | ||
26 | at: :yocto_ab:`/typhoon/#/console`. | ||
27 | |||
28 | Builds are triggered manually when the test branches are ready. The | ||
29 | builds are monitored by the SWAT team. For additional information, see | ||
30 | :yocto_wiki:`/wiki/Yocto_Build_Failure_Swat_Team`. | ||
31 | If successful, the changes would usually be merged to the ``master`` | ||
32 | branch. If not successful, someone would respond to the changes on the | ||
33 | mailing list explaining that there was a failure in testing. The choice | ||
34 | of quick or full would depend on the type of changes and the speed with | ||
35 | which the result was required. | ||
36 | |||
37 | The Autobuilder does build the ``master`` branch once daily for several | ||
38 | reasons, in particular, to ensure the current ``master`` branch does | ||
39 | build, but also to keep ``yocto-testresults`` | ||
40 | (:yocto_git:`/cgit.cgi/yocto-testresults/`), | ||
41 | buildhistory | ||
42 | (:yocto_git:`/cgit.cgi/poky-buildhistory/`), and | ||
43 | our sstate up to date. On the weekend, there is a master-next build | ||
44 | instead to ensure the test results are updated for the less frequently | ||
45 | run targets. | ||
46 | |||
47 | Performance builds (buildperf-\* targets in the console) are triggered | ||
48 | separately every six hours and automatically push their results to the | ||
49 | buildstats repository at: | ||
50 | :yocto_git:`/cgit.cgi/yocto-buildstats/`. | ||
51 | |||
52 | The 'quick' targets have been selected to be the ones which catch the | ||
53 | most failures or give the most valuable data. We run 'fast' ptests in | ||
54 | this case for example but not the ones which take a long time. The quick | ||
55 | target doesn't include \*-lsb builds for all architectures, some world | ||
56 | builds and doesn't trigger performance tests or ltp testing. The full | ||
57 | build includes all these things and is slower but more comprehensive. | ||
58 | |||
59 | Release Builds | ||
60 | ============== | ||
61 | |||
62 | The project typically has two major releases a year with a six month | ||
63 | cadence in April and October. Between these there would be a number of | ||
64 | milestone releases (usually four) with the final one being stablization | ||
65 | only along with point releases of our stable branches. | ||
66 | |||
67 | The build and release process for these project releases is similar to | ||
68 | that in `Day to Day Development <#test-daily-devel>`__, in that the | ||
69 | a-full target of the Autobuilder is used but in addition the form is | ||
70 | configured to generate and publish artefacts and the milestone number, | ||
71 | version, release candidate number and other information is entered. The | ||
72 | box to "generate an email to QA"is also checked. | ||
73 | |||
74 | When the build completes, an email is sent out using the send-qa-email | ||
75 | script in the ``yocto-autobuilder-helper`` repository to the list of | ||
76 | people configured for that release. Release builds are placed into a | ||
77 | directory in https://autobuilder.yocto.io/pub/releases on the | ||
78 | Autobuilder which is included in the email. The process from here is | ||
79 | more manual and control is effectively passed to release engineering. | ||
80 | The next steps include: | ||
81 | |||
82 | - QA teams respond to the email saying which tests they plan to run and | ||
83 | when the results will be available. | ||
84 | |||
85 | - QA teams run their tests and share their results in the yocto- | ||
86 | testresults-contrib repository, along with a summary of their | ||
87 | findings. | ||
88 | |||
89 | - Release engineering prepare the release as per their process. | ||
90 | |||
91 | - Test results from the QA teams are included into the release in | ||
92 | separate directories and also uploaded to the yocto-testresults | ||
93 | repository alongside the other test results for the given revision. | ||
94 | |||
95 | - The QA report in the final release is regenerated using resulttool to | ||
96 | include the new test results and the test summaries from the teams | ||
97 | (as headers to the generated report). | ||
98 | |||
99 | - The release is checked against the release checklist and release | ||
100 | readiness criteria. | ||
101 | |||
102 | - A final decision on whether to release is made by the YP TSC who have | ||
103 | final oversight on release readiness. | ||