summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorScott Rifenbark <srifenbark@gmail.com>2017-03-30 07:51:19 -0700
committerRichard Purdie <richard.purdie@linuxfoundation.org>2017-04-12 19:35:00 +0100
commit43fefd121163e82575709fd75a318fe7f36c1e20 (patch)
tree844e0ba569f5f7b6159d030e6c2d2ee0cfce3e1d
parentc53ba427ef1d21cb5bd5412e9291e37d2e681a4c (diff)
downloadpoky-43fefd121163e82575709fd75a318fe7f36c1e20.tar.gz
ref-manual: Added section on test and QA infrastructure
Fixes [YOCTO #10887] There needed to be an overview of the test and QA infrastructure used by the YP development team for releases and that is visible to the developer. I added a new section for this information into the existing section on YP release processes. (From yocto-docs rev: 17b3d6898b37bdbfbd23a333a4d659a9746696d0) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-rw-r--r--documentation/ref-manual/ref-release-process.xml131
1 files changed, 131 insertions, 0 deletions
diff --git a/documentation/ref-manual/ref-release-process.xml b/documentation/ref-manual/ref-release-process.xml
index 1b337f8bd7..fe3ba0933e 100644
--- a/documentation/ref-manual/ref-release-process.xml
+++ b/documentation/ref-manual/ref-release-process.xml
@@ -117,6 +117,137 @@
117 </para> 117 </para>
118</section> 118</section>
119 119
120<section id='testing-and-quality-assurance'>
121 <title>Testing and Quality Assurance</title>
122
123 <para>
124 Part of the Yocto Project development and release process is quality
125 assurance through the execution of test strategies.
126 Test strategies provide the Yocto Project team a way to ensure a
127 release is validated.
128 Additionally, because the test strategies are visible to you as a
129 developer, you can validate your projects.
130 This section overviews the available test infrastructure used in the
131 Yocto Project.
132 For information on how to run available tests on your projects, see the
133 "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
134 section in the Yocto Project Development Manual.
135 </para>
136
137 <para>
138 The QA/testing infrastructure is woven into the project to the point
139 where core developers take some of it for granted.
140 The infrastructure consists of the following pieces:
141 <itemizedlist>
142 <listitem><para>
143 <filename>bitbake-selftest</filename>:
144 A standalone command that runs unit tests on key pieces of
145 BitBake and its fetchers.
146 </para></listitem>
147 <listitem><para>
148 <link linkend='ref-classes-sanity'><filename>sanity.bbclass</filename></link>:
149 This automatically included class checks the build environment
150 for missing tools (e.g. <filename>gcc</filename>) or common
151 misconfigurations such as
152 <link linkend='var-MACHINE'><filename>MACHINE</filename></link>
153 set incorrectly.
154 </para></listitem>
155 <listitem><para>
156 <link linkend='ref-classes-insane'><filename>insane.bbclass</filename></link>:
157 This class checks the generated output from builds for sanity.
158 For example, if building for an ARM target, did the build
159 produce ARM binaries.
160 If, for example, the build produced PPC binaries then there
161 is a problem.
162 </para></listitem>
163 <listitem><para>
164 <link linkend='ref-classes-testimage*'><filename>testimage.bbclass</filename></link>:
165 This class performs runtime testing of images after they are
166 built.
167 The tests are usually used with
168 <ulink url='&YOCTO_DOCS_DEV_URL;#dev-manual-qemu'>QEMU</ulink>
169 to boot the images and check the combined runtime result
170 boot operation and functions.
171 However, the test can also use the IP address of a machine to
172 test.
173 </para></listitem>
174 <listitem><para>
175 <ulink url='&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest'><filename>ptest</filename></ulink>:
176 Runs tests against packages produced during the build for a
177 given piece of software.
178 The test allows the packages to be be run within a target
179 image.
180 </para></listitem>
181 <listitem><para>
182 <filename>oe-selftest</filename>:
183 Tests combination BitBake invocations.
184 These tests operate outside the OpenEmbedded build system
185 itself.
186 The <filename>oe-selftest</filename> can run all tests by
187 default or can run selected tests or test suites.
188 <note>
189 Running <filename>oe-selftest</filename> requires
190 host packages beyond the "Essential" grouping.
191 See the
192 "<link linkend='required-packages-for-the-host-development-system'>Required Packages for the Host Development System</link>"
193 section for more information.
194 </note>
195 </para></listitem>
196 </itemizedlist>
197 </para>
198
199 <para>
200 Originally, much of this testing was done manually.
201 However, significant effort has been made to automate the tests so
202 that more people can use them and the Yocto Project development team
203 can run them faster and more efficiently.
204 </para>
205
206 <para>
207 The Yocto Project's main Autobuilder
208 (<filename>autobuilder.yoctoproject.org</filename>) publicly tests
209 each Yocto Project release's code in the OE-Core, Poky, and BitBake
210 repositories.
211 The testing occurs for both the current state of the
212 "master" branch and also for submitted patches.
213 Testing for submitted patches usually occurs in the
214 "ross/mut" branch in the <filename>poky-contrib</filename> repository
215 (i.e. the master-under-test branch) or in the "master-next" branch
216 in the <filename>poky</filename> repository.
217 <note>
218 You can find all these branches in the Yocto Project
219 <ulink url='&YOCTO_DOCS_DEV_URL;#source-repositories'>Source Repositories</ulink>.
220 </note>
221 Testing within these public branches ensures in a publicly visible way
222 that all of the main supposed architectures and recipes in OE-Core
223 successfully build and behave properly.
224 </para>
225
226 <para>
227 Various features such as <filename>multilib</filename>, sub
228 architectures (e.g. <filename>x32</filename>,
229 <filename>poky-tiny</filename>, <filename>musl</filename>,
230 <filename>no-x11</filename> and and so forth),
231 <filename>bitbake-selftest</filename>, and
232 <filename>oe-selftest</filename> are tested as part of
233 the QA process of a release.
234 Complete testing and validation for a release takes the Autobuilder
235 workers several hours.
236 <note>
237 The Autobuilder workers are non-homogeneous, which means regular
238 testing across a variety of Linux distributions occurs.
239 The Autobuilder is limited to only testing QEMU-based setups and
240 not real hardware.
241 </note>
242 </para>
243
244 <para>
245 Finally, in addition to the Autobuilder's tests, the Yocto Project
246 QA team also performs testing on a variety of platforms, which includes
247 actual hardware, to ensure expected results.
248 </para>
249</section>
250
120</chapter> 251</chapter>
121<!-- 252<!--
122vim: expandtab tw=80 ts=4 253vim: expandtab tw=80 ts=4