summaryrefslogtreecommitdiffstats
path: root/documentation/test-manual/test-manual-intro.xml
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/test-manual/test-manual-intro.xml')
-rw-r--r--documentation/test-manual/test-manual-intro.xml624
1 files changed, 0 insertions, 624 deletions
diff --git a/documentation/test-manual/test-manual-intro.xml b/documentation/test-manual/test-manual-intro.xml
deleted file mode 100644
index 0cdbee4d8e..0000000000
--- a/documentation/test-manual/test-manual-intro.xml
+++ /dev/null
@@ -1,624 +0,0 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4<!--SPDX-License-Identifier: CC-BY-2.0-UK-->
5
6<chapter id='test-manual-intro'>
7
8<title>The Yocto Project Test Environment Manual</title>
9 <section id='test-welcome'>
10 <title>Welcome</title>
11
12 <para> Welcome to the Yocto Project Test Environment Manual! This manual is a work in
13 progress. The manual contains information about the testing environment used by the
14 Yocto Project to make sure each major and minor release works as intended. All the
15 project's testing infrastructure and processes are publicly visible and available so
16 that the community can see what testing is being performed, how it's being done and the
17 current status of the tests and the project at any given time. It is intended that Other
18 organizations can leverage off the process and testing environment used by the Yocto
19 Project to create their own automated, production test environment, building upon the
20 foundations from the project core. </para>
21
22 <para> Currently, the Yocto Project Test Environment Manual has no projected release date.
23 This manual is a work-in-progress and is being initially loaded with information from
24 the <ulink url="">README</ulink> files and notes from key engineers: <itemizedlist>
25 <listitem>
26 <para>
27 <emphasis><filename>yocto-autobuilder2</filename>:</emphasis> This <ulink
28 url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md"
29 ><filename>README.md</filename></ulink> is the main README which
30 detials how to set up the Yocto Project Autobuilder. The
31 <filename>yocto-autobuilder2</filename> repository represents the Yocto
32 Project's console UI plugin to Buildbot and the configuration necessary to
33 configure Buildbot to perform the testing the project requires. </para>
34 </listitem>
35 <listitem>
36 <para>
37 <emphasis><filename>yocto-autobuilder-helper</filename>:</emphasis> This
38 <ulink
39 url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README"
40 ><filename>README</filename></ulink> and repository contains Yocto
41 Project Autobuilder Helper scripts and configuration. The
42 <filename>yocto-autobuilder-helper</filename> repository contains the
43 "glue" logic that defines which tests to run and how to run them. As a
44 result, it can be used by any Continuous Improvement (CI) system to run
45 builds, support getting the correct code revisions, configure builds and
46 layers, run builds, and collect results. The code is independent of any CI
47 system, which means the code can work Buildbot, Jenkins, or others. This
48 repository has a branch per release of the project defining the tests to run
49 on a per release basis.</para>
50 </listitem>
51 </itemizedlist>
52 </para>
53 </section>
54
55 <section id='test-yocto-project-autobuilder-overview'>
56 <title>Yocto Project Autobuilder Overview</title>
57
58 <para>The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and
59 procedures used by the Yocto Project to test released software across supported hardware
60 in an automated and regular fashion. Basically, during the development of a Yocto
61 Project release, the Autobuilder tests if things work. The Autobuilder builds all test
62 targets and runs all the tests. </para>
63
64 <para>The Yocto Project uses now uses standard upstream <ulink
65 url="https://docs.buildbot.net/0.9.15.post1/">Buildbot</ulink> (version 9) to drive
66 its integration and testing. Buildbot Nine has a plug-in interface that the Yocto
67 Project customizes using code from the <filename>yocto-autobuilder2</filename>
68 repository, adding its own console UI plugin. The resulting UI plug-in allows you to
69 visualize builds in a way suited to the project's needs.</para>
70
71 <para>A <filename>helper</filename> layer provides configuration and job management through
72 scripts found in the <filename>yocto-autobuilder-helper</filename> repository. The
73 <filename>helper</filename> layer contains the bulk of the build configuration
74 information and is release-specific, which makes it highly customizable on a per-project
75 basis. The layer is CI system-agnostic and contains a number of Helper scripts that can
76 generate build configurations from simple JSON files. <note>
77 <para>The project uses Buildbot for historical reasons but also because many of the
78 project developers have knowledge of python. It is possible to use the outer
79 layers from another Continuous Integration (CI) system such as <ulink
80 url="https://en.wikipedia.org/wiki/Jenkins_(software)">Jenkins</ulink>
81 instead of Buildbot. </para>
82 </note>
83 </para>
84
85 <para> The following figure shows the Yocto Project Autobuilder stack with a topology that
86 includes a controller and a cluster of workers: <imagedata
87 fileref="figures/ab-test-cluster.png" width="4.6in" depth="4.35in" align="center"
88 scalefit="1"/>
89 </para>
90 </section>
91
92 <section id='test-project-tests'>
93 <title>Yocto Project Tests - Types of Testing Overview</title>
94
95 <para>The Autobuilder tests different elements of the project by using thefollowing types of
96 tests: <itemizedlist>
97 <listitem>
98 <para>
99 <emphasis>Build Testing:</emphasis> Tests whether specific configurations
100 build by varying <ulink url="&YOCTO_DOCS_REF_URL;#var-MACHINE"
101 ><filename>MACHINE</filename></ulink>, <ulink
102 url="&YOCTO_DOCS_REF_URL;#var-DISTRO"
103 ><filename>DISTRO</filename></ulink>, other configuration options, and
104 the specific target images being built (or world). Used to trigger builds of
105 all the different test configurations on the Autobuilder. Builds usually
106 cover many different targets for different architectures, machines, and
107 distributions, as well as different configurations, such as different init
108 systems. The Autobuilder tests literally hundreds of configurations and
109 targets. <itemizedlist>
110 <listitem>
111 <para>
112 <emphasis>Sanity Checks During the Build Process:</emphasis>
113 Tests initiated through the <ulink
114 url="&YOCTO_DOCS_REF_URL;#ref-classes-insane"
115 ><filename>insane</filename></ulink> class. These checks
116 ensure the output of the builds are correct. For example, does
117 the ELF architecture in the generated binaries match the target
118 system? ARM binaries would not work in a MIPS system! </para>
119 </listitem>
120 </itemizedlist></para>
121 </listitem>
122 <listitem>
123 <para>
124 <emphasis>Build Performance Testing:</emphasis> Tests whether or not
125 commonly used steps during builds work efficiently and avoid regressions.
126 Tests to time commonly used usage scenarios are run through
127 <filename>oe-build-perf-test</filename>. These tests are run on isolated
128 machines so that the time measurements of the tests are accurate and no
129 other processes interfere with the timing results. The project currently
130 tests performance on two different distributions, Fedora and Ubuntu, to
131 ensure we have no single point of failure and can ensure the different
132 distros work effectively. </para>
133 </listitem>
134 <listitem>
135 <para>
136 <emphasis>eSDK Testing:</emphasis> Image tests initiated through the
137 following command:
138 <literallayout class="monospaced">
139 $ bitbake <replaceable>image</replaceable> -c testsdkext
140 </literallayout>
141 The tests utilize the <filename>testsdkext</filename> class and the
142 <filename>do_testsdkext</filename> task. </para>
143 </listitem>
144 <listitem>
145 <para>
146 <emphasis>Feature Testing:</emphasis> Various scenario-based tests are run
147 through the <ulink url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
148 >OpenEmbedded Self-Test</ulink> (oe-selftest). We test oe-selftest on
149 each of the main distrubutions we support. </para>
150 </listitem>
151 <listitem>
152 <para>
153 <emphasis>Image Testing:</emphasis> Image tests initiated through the
154 following command:
155 <literallayout class="monospaced">
156 $ bitbake <replaceable>image</replaceable> -c testimage
157 </literallayout>
158 The tests utilize the <ulink
159 url="&YOCTO_DOCS_REF_URL;#ref-classes-testimage*"
160 ><filename>testimage*</filename></ulink> classes and the <ulink
161 url="&YOCTO_DOCS_REF_URL;#ref-tasks-testimage"
162 ><filename>do_testimage</filename></ulink> task. </para>
163 </listitem>
164 <listitem>
165 <para>
166 <emphasis>Layer Testing:</emphasis> The Autobuilder has the possibility to
167 test whether specific layers work with the test of the system. The layers
168 tested may be selected by members of the project. Some key community layers
169 are also tested periodically.</para>
170 </listitem>
171 <listitem>
172 <para>
173 <emphasis>Package Testing:</emphasis> A Package Test (ptest) runs tests
174 against packages built by the OpenEmbedded build system on the target
175 machine. See the "<ulink
176 url="&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest">Testing Packages
177 With ptest</ulink>" section in the Yocto Project Development Tasks
178 Manual and the "<ulink url="&YOCTO_WIKI_URL;/wiki/Ptest">Ptest</ulink>" Wiki
179 page for more information on Ptest. </para>
180 </listitem>
181 <listitem>
182 <para>
183 <emphasis>SDK Testing:</emphasis> Image tests initiated through the
184 following command:
185 <literallayout class="monospaced">
186 $ bitbake <replaceable>image</replaceable> -c testsdk
187 </literallayout>
188 The tests utilize the <ulink url="&YOCTO_DOCS_REF_URL;#ref-classes-testsdk"
189 ><filename>testsdk</filename></ulink> class and the
190 <filename>do_testsdk</filename> task. </para>
191 </listitem>
192 <listitem>
193 <para>
194 <emphasis>Unit Testing:</emphasis> Unit tests on various components of the
195 system run through <filename>oe-selftest</filename> and <ulink
196 url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
197 ><filename>bitbake-selftest</filename></ulink>. </para>
198 </listitem>
199 <listitem>
200 <para>
201 <emphasis>Automatic Upgrade Helper:</emphasis> This target tests whether new
202 versions of software are available and whether we can automatically upgrade
203 to those new versions. If so, this target emails the maintainers with a
204 patch to let them know this is possible.</para>
205 </listitem>
206 </itemizedlist>
207 </para>
208 </section>
209
210 <section id='test-test-mapping'>
211 <title>How Tests Map to Areas of Code</title>
212
213 <para>
214 Tests map into the codebase as follows:
215 <itemizedlist>
216 <listitem><para>
217 <emphasis>bitbake-selftest</emphasis>: </para>
218 <para>These tests are self-contained and test BitBake as well as its APIs, which
219 include the fetchers. The tests are located in
220 <filename>bitbake/lib/*/tests</filename>. </para>
221 <para>From within the BitBake repository, run the following:
222 <literallayout class="monospaced">
223 $ bitbake-selftest
224 </literallayout>
225 </para>
226 <para>To skip tests that access the Internet, use the
227 <filename>BB_SKIP_NETTEST</filename> variable when running
228 "bitbake-selftest" as follows:
229 <literallayout class="monospaced">
230 $ BB_SKIP_NETTEST=yes bitbake-selftest
231 </literallayout></para>
232 <para>The default output is quiet and just prints a summary of what was run. To
233 see more information, there is a verbose
234 option:<literallayout class="monospaced">
235 $ bitbake-selftest -v
236 </literallayout></para>
237 <para>Use this option when you wish to skip tests that access the network, which
238 are mostly necessary to test the fetcher modules. To specify individual test
239 modules to run, append the test module name to the "bitbake-selftest"
240 command. For example, to specify the tests for the bb.data.module, run:
241 <literallayout class="monospaced">
242 $ bitbake-selftest bb.test.data.module
243 </literallayout>You
244 can also specify individual tests by defining the full name and module plus
245 the class path of the test, for example:
246 <literallayout class="monospaced">
247 $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
248 </literallayout></para>
249 <para>The tests are based on <ulink
250 url="https://docs.python.org/3/library/unittest.html">Python
251 unittest</ulink>. </para></listitem>
252 <listitem><para>
253 <emphasis>oe-selftest</emphasis>: <itemizedlist>
254 <listitem>
255 <para>These tests use OE to test the workflows, which include
256 testing specific features, behaviors of tasks, and API unit
257 tests. </para>
258 </listitem>
259 <listitem>
260 <para>The tests can take advantage of parallelism through the "-j"
261 option, which can specify a number of threads to spread the
262 tests across. Note that all tests from a given class of tests
263 will run in the same thread. To parallelize large numbers of
264 tests you can split the class into multiple units.</para>
265 </listitem>
266 <listitem>
267 <para>The tests are based on Python unittest. </para>
268 </listitem>
269 <listitem>
270 <para>The code for the tests resides in
271 <filename>meta/lib/oeqa/selftest/cases/</filename>. </para>
272 </listitem>
273 <listitem>
274 <para>To run all the tests, enter the following command:
275 <literallayout class="monospaced">
276 $ oe-selftest -a
277 </literallayout>
278 </para>
279 </listitem>
280 <listitem>
281 <para>To run a specific test, use the following command form where
282 <replaceable>testname</replaceable> is the name of the
283 specific test:
284 <literallayout class="monospaced">
285 $ oe-selftest -r <replaceable>testname</replaceable>
286 </literallayout>
287 For example, the following command would run the tinfoil getVar
288 API
289 test:<literallayout class="monospaced">
290 $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
291 </literallayout>It
292 is also possible to run a set of tests. For example the
293 following command will run all of the tinfoil
294 tests:<literallayout class="monospaced">
295 $ oe-selftest -r tinfoil
296 </literallayout></para>
297 </listitem>
298 </itemizedlist>
299 </para></listitem>
300 <listitem><para>
301 <emphasis>testimage:</emphasis>
302 <itemizedlist>
303 <listitem><para>
304 These tests build an image, boot it, and run tests
305 against the image's content.
306 </para></listitem>
307 <listitem><para> The code for these tests resides in <filename>meta/lib/oeqa/runtime/cases/</filename>. </para></listitem>
308 <listitem><para>
309 You need to set the
310 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_CLASSES'><filename>IMAGE_CLASSES</filename></ulink>
311 variable as follows:
312 <literallayout class='monospaced'>
313 IMAGE_CLASSES += "testimage"
314 </literallayout>
315 </para></listitem>
316 <listitem><para>
317 Run the tests using the following command form:
318 <literallayout class='monospaced'>
319 $ bitbake <replaceable>image</replaceable> -c testimage
320 </literallayout>
321 </para></listitem>
322 </itemizedlist>
323 </para></listitem>
324 <listitem><para>
325 <emphasis>testsdk:</emphasis>
326 <itemizedlist>
327 <listitem><para>These tests build an SDK, install it, and then run tests against that SDK. </para></listitem>
328 <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/sdk/cases/</filename>. </para></listitem>
329 <listitem><para>Run the test using the following command form:
330 <literallayout class="monospaced">
331 $ bitbake <replaceable>image</replaceable> -c testsdk
332 </literallayout>
333 </para></listitem>
334 </itemizedlist>
335 </para></listitem>
336 <listitem><para>
337 <emphasis>testsdk_ext:</emphasis>
338 <itemizedlist>
339 <listitem><para>These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. </para></listitem>
340 <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/esdk</filename>. </para></listitem>
341 <listitem><para>To run the tests, use the following command form:
342 <literallayout class="monospaced">
343 $ bitbake <replaceable>image</replaceable> -c testsdkext
344 </literallayout>
345 </para></listitem>
346 </itemizedlist>
347 </para></listitem>
348
349
350 <listitem><para>
351 <emphasis>oe-build-perf-test:</emphasis>
352 <itemizedlist>
353 <listitem><para>These tests run through commonly used usage scenarios and measure the performance times. </para></listitem>
354 <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/buildperf</filename>. </para></listitem>
355 <listitem><para>To run the tests, use the following command form:
356 <literallayout class="monospaced">
357 $ oe-build-perf-test <replaceable>options</replaceable>
358 </literallayout>The
359 command takes a number of options, such as where to place the
360 test results. The Autobuilder Helper Scripts include the
361 <filename>build-perf-test-wrapper</filename> script with
362 examples of how to use the oe-build-perf-test from the command
363 line.</para>
364 <para>Use the <filename>oe-git-archive</filename> command to store
365 test results into a Git repository. </para>
366 <para>Use the <filename>oe-build-perf-report</filename> command to
367 generate text reports and HTML reports with graphs of the
368 performance data. For examples, see <link linkend=""
369 >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html</link>
370 and <link linkend=""
371 >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt</link>.</para></listitem>
372 <listitem>
373 <para>The tests are contained in
374 <filename>lib/oeqa/buildperf/test_basic.py</filename>.</para>
375 </listitem>
376 </itemizedlist>
377 </para></listitem>
378
379
380
381
382 </itemizedlist>
383 </para>
384 </section>
385
386 <section id='test-examples'>
387 <title>Test Examples</title>
388
389 <para>This section provides example tests for each of the tests listed in the <link
390 linkend="test-test-mapping">How Tests Map to Areas of Code</link> section. </para>
391 <para>For oeqa tests, testcases for each area reside in the main test directory at
392 <filename>meta/lib/oeqa/selftest/cases</filename> directory.</para>
393 <para>For oe-selftest. bitbake testcases reside in the <filename>lib/bb/tests/</filename>
394 directory. </para>
395
396 <section id='bitbake-selftest-example'>
397 <title><filename>bitbake-selftest</filename></title>
398
399 <para>A simple test example from <filename>lib/bb/tests/data.py</filename> is:
400 <literallayout class="monospaced">
401 class DataExpansions(unittest.TestCase):
402 def setUp(self):
403 self.d = bb.data.init()
404 self.d["foo"] = "value_of_foo"
405 self.d["bar"] = "value_of_bar"
406 self.d["value_of_foo"] = "value_of_'value_of_foo'"
407
408 def test_one_var(self):
409 val = self.d.expand("${foo}")
410 self.assertEqual(str(val), "value_of_foo")
411 </literallayout>
412 </para>
413 <para>In this example, a <ulink url=""><filename>DataExpansions</filename></ulink> class
414 of tests is created, derived from standard python unittest. The class has a common
415 <filename>setUp</filename> function which is shared by all the tests in the
416 class. A simple test is then added to test that when a variable is expanded, the
417 correct value is found.</para>
418 <para>Bitbake selftests are straightforward python unittest. Refer to the Python
419 unittest documentation for additional information on writing these tests at: <link
420 linkend="">https://docs.python.org/3/library/unittest.html</link>.</para>
421 </section>
422
423 <section id='oe-selftest-example'>
424 <title><filename>oe-selftest</filename></title>
425
426 <para>These tests are more complex due to the setup required behind the scenes for full
427 builds. Rather than directly using Python's unittest, the code wraps most of the
428 standard objects. The tests can be simple, such as testing a command from within the
429 OE build environment using the following
430 example:<literallayout class="monospaced">
431 class BitbakeLayers(OESelftestTestCase):
432 def test_bitbakelayers_showcrossdepends(self):
433 result = runCmd('bitbake-layers show-cross-depends')
434 self.assertTrue('aspell' in result.output, msg = "No dependencies
435 were shown. bitbake-layers show-cross-depends output:
436 %s"% result.output)
437 </literallayout></para>
438 <para>This example, taken from
439 <filename>meta/lib/oeqa/selftest/cases/bblayers.py</filename>, creates a
440 testcase from the <ulink url=""><filename>OESelftestTestCase</filename></ulink>
441 class, derived from <filename>unittest.TestCase</filename>, which runs the
442 <filename>bitbake-layers</filename> command and checks the output to ensure it
443 contains something we know should be here.</para>
444 <para>The <filename>oeqa.utils.commands</filename> module contains Helpers which can
445 assist with common tasks, including:<itemizedlist>
446 <listitem>
447 <para><emphasis>Obtaining the value of a bitbake variable:</emphasis> Use
448 <filename>oeqa.utils.commands.get_bb_var()</filename> or use
449 <filename>oeqa.utils.commands.get_bb_vars()</filename> for more than
450 one variable</para>
451 </listitem>
452 <listitem>
453 <para><emphasis>Running a bitbake invocation for a build:</emphasis> Use
454 <filename>oeqa.utils.commands.bitbake()</filename></para>
455 </listitem>
456 <listitem>
457 <para><emphasis>Running a command:</emphasis> Use
458 <filename>oeqa.utils.commandsrunCmd()</filename></para>
459 </listitem>
460 </itemizedlist></para>
461 <para>There is also a <filename>oeqa.utils.commands.runqemu()</filename> function for
462 launching the <filename>runqemu</filename> command for testing things within a
463 running, virtualized image.</para>
464 <para>You can run these tests in parallel. Parallelism works per test class, so tests
465 within a given test class should always run in the same build, while tests in
466 different classes or modules may be split into different builds. There is no data
467 store available for these tests since the tests launch the
468 <filename>bitbake</filename> command and exist outside of its context. As a
469 result, common bitbake library functions (bb.*) are also unavailable.</para>
470 </section>
471
472 <section id='testimage-example'>
473 <title><filename>testimage</filename></title>
474
475 <para>These tests are run once an image is up and running, either on target hardware or
476 under QEMU. As a result, they are assumed to be running in a target image
477 environment, as opposed to a host build environment. A simple example from
478 <filename>meta/lib/oeqa/runtime/cases/python.py</filename> contains the
479 following:<literallayout class="monospaced">
480 class PythonTest(OERuntimeTestCase):
481 @OETestDepends(['ssh.SSHTest.test_ssh'])
482 @OEHasPackage(['python3-core'])
483 def test_python3(self):
484 cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb,
485 jbeyq', 'rot13'))\""
486 status, output = self.target.run(cmd)
487 msg = 'Exit status was not 0. Output: %s' % output
488 self.assertEqual(status, 0, msg=msg)
489 </literallayout></para>
490 <para>In this example, the <ulink url=""><filename>OERuntimeTestCase</filename></ulink>
491 class wraps <filename>unittest.TestCase</filename>. Within the test,
492 <filename>self.target</filename> represents the target system, where commands
493 can be run on it using the <filename>run()</filename> method. </para>
494 <para>To ensure certain test or package dependencies are met, you can use the
495 <filename>OETestDepends</filename> and <filename>OEHasPackage</filename>
496 decorators. For example, the test in this example would only make sense if
497 python3-core is installed in the image.</para>
498 </section>
499
500 <section id='testsdk_ext-example'>
501 <title><filename>testsdk_ext</filename></title>
502
503 <para>These tests are run against built extensible SDKs (eSDKs). The tests can assume
504 that the eSDK environment has already been setup. An example from
505 <filename>meta/lib/oeqa/sdk/cases/devtool.py</filename> contains the
506 following:<literallayout class="monospaced">
507 class DevtoolTest(OESDKExtTestCase):
508 @classmethod
509 def setUpClass(cls):
510 myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
511 cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
512 shutil.copytree(myapp_src, cls.myapp_dst)
513 subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
514 subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
515 subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
516
517 @classmethod
518 def tearDownClass(cls):
519 shutil.rmtree(cls.myapp_dst)
520 def _test_devtool_build(self, directory):
521 self._run('devtool add myapp %s' % directory)
522 try:
523 self._run('devtool build myapp')
524 finally:
525 self._run('devtool reset myapp')
526 def test_devtool_build_make(self):
527 self._test_devtool_build(self.myapp_dst)
528 </literallayout>In
529 this example, the <filename>devtool</filename> command is tested to see whether a
530 sample application can be built with the <filename>devtool build</filename> command
531 within the eSDK.</para>
532 </section>
533
534 <section id='testsdk-example'>
535 <title><filename>testsdk</filename></title>
536
537 <para>These tests are run against built SDKs. The tests can assume that an SDK has
538 already been extracted and its environment file has been sourced. A simple example
539 from <filename>meta/lib/oeqa/sdk/cases/python2.py</filename> contains the
540 following:<literallayout class="monospaced">
541 class Python3Test(OESDKTestCase):
542 def setUp(self):
543 if not (self.tc.hasHostPackage("nativesdk-python3-core") or
544 self.tc.hasHostPackage("python3-core-native")):
545 raise unittest.SkipTest("No python3 package in the SDK")
546
547 def test_python3(self):
548 cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
549 output = self._run(cmd)
550 self.assertEqual(output, "Hello, world\n")
551 </literallayout>In
552 this example, if nativesdk-python3-core has been installed into the SDK, the code
553 runs the python3 interpreter with a basic command to check it is working correctly.
554 The test would only run if python3 is installed in the SDK.</para>
555 </section>
556
557 <section id='oe-build-perf-test-example'>
558 <title><filename>oe-build-perf-test</filename></title>
559
560 <para>The performance tests usually measure how long operations take and the resource
561 utilisation as that happens. An example from
562 <filename>meta/lib/oeqa/buildperf/test_basic.py</filename> contains the
563 following:<literallayout class="monospaced">
564 class Test3(BuildPerfTestCase):
565
566 def test3(self):
567 """Bitbake parsing (bitbake -p)"""
568 # Drop all caches and parse
569 self.rm_cache()
570 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
571 self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
572 'bitbake -p (no caches)')
573 # Drop tmp/cache
574 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
575 self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
576 'bitbake -p (no tmp/cache)')
577 # Parse with fully cached data
578 self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
579 'bitbake -p (cached)')
580 </literallayout>This
581 example shows how three specific parsing timings are measured, with and without
582 various caches, to show how BitBake's parsing performance trends over time.</para>
583 </section>
584 </section>
585 <section id='test-writing-considerations'>
586 <title>Considerations When Writing Tests</title>
587 <para>When writing good tests, there are several things to keep in mind. Since things
588 running on the Autobuilder are accessed concurrently by multiple workers, consider the
589 following:</para>
590 <formalpara>
591 <title>Running "cleanall" is not permitted</title>
592 <para>This can delete files from DL_DIR which would potentially break other builds
593 running in parallel. If this is required, DL_DIR must be set to an isolated
594 directory.</para>
595 </formalpara>
596 <formalpara>
597 <title>Running "cleansstate" is not permitted</title>
598 <para>This can delete files from SSTATE_DIR which would potentially break other builds
599 running in parallel. If this is required, SSTATE_DIR must be set to an isolated
600 directory. Alternatively, you can use the "-f" option with the
601 <filename>bitbake</filename> command to "taint" tasks by changing the sstate
602 checksums to ensure sstate cache items will not be reused.</para>
603 </formalpara>
604 <formalpara>
605 <title>Tests should not change the metadata</title>
606 <para>This is particularly true for oe-selftests since these can run in parallel and
607 changing metadata leads to changing checksums, which confuses BitBake while running
608 in parallel. If this is necessary, copy layers to a temporary location and modify
609 them. Some tests need to change metadata, such as the devtool tests. To prevent the
610 metadate from changes, set up temporary copies of that data first.</para>
611 </formalpara>
612 </section>
613
614
615
616
617
618
619
620
621</chapter>
622<!--
623vim: expandtab tw=80 ts=4
624-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-14 00:36:30 +0000