diff options
Diffstat (limited to 'documentation/test-manual/test-manual-test-process.rst')
-rw-r--r-- | documentation/test-manual/test-manual-test-process.rst | 103 |
1 files changed, 103 insertions, 0 deletions
diff --git a/documentation/test-manual/test-manual-test-process.rst b/documentation/test-manual/test-manual-test-process.rst new file mode 100644 index 0000000000..19c9b565de --- /dev/null +++ b/documentation/test-manual/test-manual-test-process.rst | |||
@@ -0,0 +1,103 @@ | |||
1 | *********************************** | ||
2 | Project Testing and Release Process | ||
3 | *********************************** | ||
4 | |||
5 | .. _test-daily-devel: | ||
6 | |||
7 | Day to Day Development | ||
8 | ====================== | ||
9 | |||
10 | This section details how the project tests changes, through automation | ||
11 | on the Autobuilder or with the assistance of QA teams, through to making | ||
12 | releases. | ||
13 | |||
14 | The project aims to test changes against our test matrix before those | ||
15 | changes are merged into the master branch. As such, changes are queued | ||
16 | up in batches either in the ``master-next`` branch in the main trees, or | ||
17 | in user trees such as ``ross/mut`` in ``poky-contrib`` (Ross Burton | ||
18 | helps review and test patches and this is his testing tree). | ||
19 | |||
20 | We have two broad categories of test builds, including "full" and | ||
21 | "quick". On the Autobuilder, these can be seen as "a-quick" and | ||
22 | "a-full", simply for ease of sorting in the UI. Use our Autobuilder | ||
23 | console view to see where me manage most test-related items, available | ||
24 | at: `https://autobuilder.yoctoproject.org/typhoon/#/console <#>`__. | ||
25 | |||
26 | Builds are triggered manually when the test branches are ready. The | ||
27 | builds are monitored by the SWAT team. For additional information, see | ||
28 | `https://wiki.yoctoproject.org/wiki/Yocto_Build_Failure_Swat_Team <#>`__. | ||
29 | If successful, the changes would usually be merged to the ``master`` | ||
30 | branch. If not successful, someone would respond to the changes on the | ||
31 | mailing list explaining that there was a failure in testing. The choice | ||
32 | of quick or full would depend on the type of changes and the speed with | ||
33 | which the result was required. | ||
34 | |||
35 | The Autobuilder does build the ``master`` branch once daily for several | ||
36 | reasons, in particular, to ensure the current ``master`` branch does | ||
37 | build, but also to keep ``yocto-testresults`` | ||
38 | (`http://git.yoctoproject.org/cgit.cgi/yocto-testresults/ <#>`__), | ||
39 | buildhistory | ||
40 | (`http://git.yoctoproject.org/cgit.cgi/poky-buildhistory/ <#>`__), and | ||
41 | our sstate up to date. On the weekend, there is a master-next build | ||
42 | instead to ensure the test results are updated for the less frequently | ||
43 | run targets. | ||
44 | |||
45 | Performance builds (buildperf-\* targets in the console) are triggered | ||
46 | separately every six hours and automatically push their results to the | ||
47 | buildstats repository at: | ||
48 | `http://git.yoctoproject.org/cgit.cgi/yocto-buildstats/ <#>`__. | ||
49 | |||
50 | The 'quick' targets have been selected to be the ones which catch the | ||
51 | most failures or give the most valuable data. We run 'fast' ptests in | ||
52 | this case for example but not the ones which take a long time. The quick | ||
53 | target doesn't include \*-lsb builds for all architectures, some world | ||
54 | builds and doesn't trigger performance tests or ltp testing. The full | ||
55 | build includes all these things and is slower but more comprehensive. | ||
56 | |||
57 | .. _test-yocto-project-autobuilder-overview: | ||
58 | |||
59 | Release Builds | ||
60 | ============== | ||
61 | |||
62 | The project typically has two major releases a year with a six month | ||
63 | cadence in April and October. Between these there would be a number of | ||
64 | milestone releases (usually four) with the final one being stablization | ||
65 | only along with point releases of our stable branches. | ||
66 | |||
67 | The build and release process for these project releases is similar to | ||
68 | that in `Day to Day Development <#test-daily-devel>`__, in that the | ||
69 | a-full target of the Autobuilder is used but in addition the form is | ||
70 | configured to generate and publish artefacts and the milestone number, | ||
71 | version, release candidate number and other information is entered. The | ||
72 | box to "generate an email to QA"is also checked. | ||
73 | |||
74 | When the build completes, an email is sent out using the send-qa-email | ||
75 | script in the ``yocto-autobuilder-helper`` repository to the list of | ||
76 | people configured for that release. Release builds are placed into a | ||
77 | directory in `https://autobuilder.yocto.io/pub/releases <#>`__ on the | ||
78 | Autobuilder which is included in the email. The process from here is | ||
79 | more manual and control is effectively passed to release engineering. | ||
80 | The next steps include: | ||
81 | |||
82 | - QA teams respond to the email saying which tests they plan to run and | ||
83 | when the results will be available. | ||
84 | |||
85 | - QA teams run their tests and share their results in the yocto- | ||
86 | testresults-contrib repository, along with a summary of their | ||
87 | findings. | ||
88 | |||
89 | - Release engineering prepare the release as per their process. | ||
90 | |||
91 | - Test results from the QA teams are included into the release in | ||
92 | separate directories and also uploaded to the yocto-testresults | ||
93 | repository alongside the other test results for the given revision. | ||
94 | |||
95 | - The QA report in the final release is regenerated using resulttool to | ||
96 | include the new test results and the test summaries from the teams | ||
97 | (as headers to the generated report). | ||
98 | |||
99 | - The release is checked against the release checklist and release | ||
100 | readiness criteria. | ||
101 | |||
102 | - A final decision on whether to release is made by the YP TSC who have | ||
103 | final oversight on release readiness. | ||