summaryrefslogtreecommitdiffstats
path: root/documentation/test-manual/test-manual-intro.xml
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/test-manual/test-manual-intro.xml')
-rw-r--r--documentation/test-manual/test-manual-intro.xml634
1 files changed, 634 insertions, 0 deletions
diff --git a/documentation/test-manual/test-manual-intro.xml b/documentation/test-manual/test-manual-intro.xml
new file mode 100644
index 0000000000..5e9def894e
--- /dev/null
+++ b/documentation/test-manual/test-manual-intro.xml
@@ -0,0 +1,634 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='test-manual-intro'>
6
7<title>The Yocto Project Test Environment Manual</title>
8 <section id='test-welcome'>
9 <title>Welcome</title>
10
11 <para> Welcome to the Yocto Project Test Environment Manual! This manual is a work in
12 progress. The manual contains information about the testing environment used by the
13 Yocto Project to make sure each major and minor release works as intended. All the
14 project’s testing infrastructure and processes are publicly visible and available so
15 that the community can see what testing is being performed, how it’s being done and the
16 current status of the tests and the project at any given time. It is intended that Other
17 organizations can leverage off the process and testing environment used by the Yocto
18 Project to create their own automated, production test environment, building upon the
19 foundations from the project core. </para>
20
21 <para> Currently, the Yocto Project Test Environment Manual has no projected release date.
22 This manual is a work-in-progress and is being initially loaded with information from
23 the <ulink url="">README</ulink> files and notes from key engineers: <itemizedlist>
24 <listitem>
25 <para>
26 <emphasis><filename>yocto-autobuilder2</filename>:</emphasis> This <ulink
27 url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md"
28 ><filename>README.md</filename></ulink> is the main README which
29 detials how to set up the Yocto Project Autobuilder. The
30 <filename>yocto-autobuilder2</filename> repository represents the Yocto
31 Project's console UI plugin to Buildbot and the configuration necessary to
32 configure Buildbot to perform the testing the project requires. </para>
33 </listitem>
34 <listitem>
35 <para>
36 <emphasis><filename>yocto-autobuilder-helper</filename>:</emphasis> This
37 <ulink
38 url="http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README"
39 ><filename>README</filename></ulink> and repository contains Yocto
40 Project Autobuilder Helper scripts and configuration. The
41 <filename>yocto-autobuilder-helper</filename> repository contains the
42 "glue" logic that defines which tests to run and how to run them. As a
43 result, it can be used by any Continuous Improvement (CI) system to run
44 builds, support getting the correct code revisions, configure builds and
45 layers, run builds, and collect results. The code is independent of any CI
46 system, which means the code can work Buildbot, Jenkins, or others. This
47 repository has a branch per release of the project defining the tests to run
48 on a per release basis.</para>
49 </listitem>
50 </itemizedlist>
51 </para>
52 </section>
53
54 <section id='test-yocto-project-autobuilder-overview'>
55 <title>Yocto Project Autobuilder Overview</title>
56
57 <para>The Yocto Project Autobuilder collectively refers to the software, tools, scripts, and
58 procedures used by the Yocto Project to test released software across supported hardware
59 in an automated and regular fashion. Basically, during the development of a Yocto
60 Project release, the Autobuilder tests if things work. The Autobuilder builds all test
61 targets and runs all the tests. </para>
62
63 <para>The Yocto Project uses now uses standard upstream <ulink
64 url="https://docs.buildbot.net/0.9.15.post1/">Buildbot</ulink> (version 9) to drive
65 its integration and testing. Buildbot Nine has a plug-in interface that the Yocto
66 Project customizes using code from the <filename>yocto-autobuilder2</filename>
67 repository, adding its own console UI plugin. The resulting UI plug-in allows you to
68 visualize builds in a way suited to the project's needs.</para>
69
70 <para>A <filename>helper</filename> layer provides configuration and job management through
71 scripts found in the <filename>yocto-autobuilder-helper</filename> repository. The
72 <filename>helper</filename> layer contains the bulk of the build configuration
73 information and is release-specific, which makes it highly customizable on a per-project
74 basis. The layer is CI system-agnostic and contains a number of Helper scripts that can
75 generate build configurations from simple JSON files. <note>
76 <para>The project uses Buildbot for historical reasons but also because many of the
77 project developers have knowledge of python. It is possible to use the outer
78 layers from another Continuous Integration (CI) system such as <ulink
79 url="https://en.wikipedia.org/wiki/Jenkins_(software)">Jenkins</ulink>
80 instead of Buildbot. </para>
81 </note>
82 </para>
83
84 <para> The following figure shows the Yocto Project Autobuilder stack with a topology that
85 includes a controller and a cluster of workers: <imagedata
86 fileref="figures/ab-test-cluster.png" width="4.6in" depth="4.35in" align="center"
87 scalefit="1"/>
88 </para>
89 </section>
90
91 <section id='test-project-tests'>
92 <title>Yocto Project Tests - Types of Testing Overview</title>
93
94 <para>The Autobuilder tests different elements of the project by using thefollowing types of
95 tests: <itemizedlist>
96 <listitem>
97 <para>
98 <emphasis>Build Testing:</emphasis> Tests whether specific configurations
99 build by varying <ulink url="&YOCTO_DOCS_REF_URL;#var-MACHINE"
100 ><filename>MACHINE</filename></ulink>, <ulink
101 url="&YOCTO_DOCS_REF_URL;#var-DISTRO"
102 ><filename>DISTRO</filename></ulink>, other configuration options, and
103 the specific target images being built (or world). Used to trigger builds of
104 all the different test configurations on the Autobuilder. Builds usually
105 cover many different targets for different architectures, machines, and
106 distributions, as well as different configurations, such as different init
107 systems. The Autobuilder tests literally hundreds of configurations and
108 targets. <itemizedlist>
109 <listitem>
110 <para>
111 <emphasis>Sanity Checks During the Build Process:</emphasis>
112 Tests initiated through the <ulink
113 url="&YOCTO_DOCS_REF_URL;#ref-classes-insane"
114 ><filename>insane</filename></ulink> class. These checks
115 ensure the output of the builds are correct. For example, does
116 the ELF architecture in the generated binaries match the target
117 system? ARM binaries would not work in a MIPS system! </para>
118 </listitem>
119 </itemizedlist></para>
120 </listitem>
121 <listitem>
122 <para>
123 <emphasis>Build Performance Testing:</emphasis> Tests whether or not
124 commonly used steps during builds work efficiently and avoid regressions.
125 Tests to time commonly used usage scenarios are run through
126 <filename>oe-build-perf-test</filename>. These tests are run on isolated
127 machines so that the time measurements of the tests are accurate and no
128 other processes interfere with the timing results. The project currently
129 tests performance on two different distributions, Fedora and Ubuntu, to
130 ensure we have no single point of failure and can ensure the different
131 distros work effectively. </para>
132 </listitem>
133 <listitem>
134 <para>
135 <emphasis>eSDK Testing:</emphasis> Image tests initiated through the
136 following command:
137 <literallayout class="monospaced">
138 $ bitbake <replaceable>image</replaceable> -c testsdkext
139 </literallayout>
140 The tests utilize the <filename>testsdkext</filename> class and the
141 <filename>do_testsdkext</filename> task. </para>
142 </listitem>
143 <listitem>
144 <para>
145 <emphasis>Feature Testing:</emphasis> Various scenario-based tests are run
146 through the <ulink url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
147 >OpenEmbedded Self-Test</ulink> (oe-selftest). We test oe-selftest on
148 each of the main distrubutions we support. </para>
149 </listitem>
150 <listitem>
151 <para>
152 <emphasis>Image Testing:</emphasis> Image tests initiated through the
153 following command:
154 <literallayout class="monospaced">
155 $ bitbake <replaceable>image</replaceable> -c testimage
156 </literallayout>
157 The tests utilize the <ulink
158 url="&YOCTO_DOCS_REF_URL;#ref-classes-testimage*"
159 ><filename>testimage*</filename></ulink> classes and the <ulink
160 url="&YOCTO_DOCS_REF_URL;#ref-tasks-testimage"
161 ><filename>do_testimage</filename></ulink> task. </para>
162 </listitem>
163 <listitem>
164 <para>
165 <emphasis>Layer Testing:</emphasis> The Autobuilder has the possibility to
166 test whether specific layers work with the test of the system. The layers
167 tested may be selected by members of the project. Some key community layers
168 are also tested periodically.</para>
169 </listitem>
170 <listitem>
171 <para>
172 <emphasis>Package Testing:</emphasis> A Package Test (ptest) runs tests
173 against packages built by the OpenEmbedded build system on the target
174 machine. See the "<ulink
175 url="&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest">Testing Packages
176 With ptest</ulink>" section in the Yocto Project Development Tasks
177 Manual and the "<ulink url="&YOCTO_WIKI_URL;/wiki/Ptest">Ptest</ulink>" Wiki
178 page for more information on Ptest. </para>
179 </listitem>
180 <listitem>
181 <para>
182 <emphasis>SDK Testing:</emphasis> Image tests initiated through the
183 following command:
184 <literallayout class="monospaced">
185 $ bitbake <replaceable>image</replaceable> -c testsdk
186 </literallayout>
187 The tests utilize the <ulink url="&YOCTO_DOCS_REF_URL;#ref-classes-testsdk"
188 ><filename>testsdk</filename></ulink> class and the
189 <filename>do_testsdk</filename> task. </para>
190 </listitem>
191 <listitem>
192 <para>
193 <emphasis>Unit Testing:</emphasis> Unit tests on various components of the
194 system run through <filename>oe-selftest</filename> and <ulink
195 url="&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance"
196 ><filename>bitbake-selftest</filename></ulink>. </para>
197 </listitem>
198 <listitem>
199 <para>
200 <emphasis>Automatic Upgrade Helper:</emphasis> This target tests whether new
201 versions of software are available and whether we can automatically upgrade
202 to those new versions. If so, this target emails the maintainers with a
203 patch to let them know this is possible.</para>
204 </listitem>
205 </itemizedlist>
206 </para>
207 </section>
208
209 <section id='test-test-mapping'>
210 <title>How Tests Map to Areas of Code</title>
211
212 <para>
213 Tests map into the codebase as follows:
214 <itemizedlist>
215 <listitem><para>
216 <emphasis>bitbake-selftest</emphasis>: <itemizedlist>
217 <listitem>
218 <para>These tests are self-contained and test BitBake as well as its
219 APIs, which include the fetchers. The tests are located in
220 <filename>bitbake/lib/*/tests</filename>. </para>
221 </listitem>
222 <listitem>
223 <para>From within the BitBake repository, run the following:
224 <literallayout class="monospaced">
225 $ bitbake-selftest
226 </literallayout>
227 </para>
228 </listitem>
229 <listitem>
230 <para>To skip tests that access the Internet, use the
231 <filename>BB_SKIP_NETTEST</filename> variable when running
232 "bitbake-selftest" as follows:
233 <literallayout class="monospaced">
234 $ BB_SKIP_NETTEST=yes bitbake-selftest
235 </literallayout>The
236 default output is quiet and just prints a summary of what was
237 run. To see more information, there is a verbose
238 option:<literallayout>
239 $ bitbake-selftest -v
240 </literallayout></para>
241 <para>Use this option when you wish to skip tests that access the
242 network, which are mostly necessary to test the fetcher modules.
243 To specify individual test modules to run, append the test
244 module name to the "bitbake-selftest" command. For example, to
245 specify the tests for the bb.data.module, run:
246 <literallayout class="monospaced">
247 $ bitbake-selftest bb.test.data.module
248 </literallayout>You
249 can also specify individual tests by defining the full name and
250 module plus the class path of the test, for example:
251 <literallayout>
252 $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override
253 </literallayout></para>
254 </listitem>
255 <listitem>
256 <para>The tests are based on <ulink
257 url="https://docs.python.org/3/library/unittest.html">Python
258 unittest</ulink>. </para>
259 </listitem>
260 </itemizedlist>
261 </para></listitem>
262 <listitem><para>
263 <emphasis>oe-selftest</emphasis>: <itemizedlist>
264 <listitem>
265 <para>These tests use OE to test the workflows, which include
266 testing specific features, behaviors of tasks, and API unit
267 tests. </para>
268 </listitem>
269 <listitem>
270 <para>The tests can take advantage of parallelism through the "-j"
271 option, which can specify a number of threads to spread the
272 tests across. Note that all tests from a given class of tests
273 will run in the same thread. To parallelize large numbers of
274 tests you can split the class into multiple units.</para>
275 </listitem>
276 <listitem>
277 <para>The tests are based on Python unittest. </para>
278 </listitem>
279 <listitem>
280 <para>The code for the tests resides in
281 <filename>meta/lib/oeqa/selftest/cases/</filename>. </para>
282 </listitem>
283 <listitem>
284 <para>To run all the tests, enter the following command:
285 <literallayout class="monospaced">
286 $ oe-selftest -a
287 </literallayout>
288 </para>
289 </listitem>
290 <listitem>
291 <para>To run a specific test, use the following command form where
292 <replaceable>testname</replaceable> is the name of the
293 specific test:
294 <literallayout class="monospaced">
295 $ oe-selftest -r <replaceable>testname</replaceable>
296 </literallayout>
297 For example, the following command would run the tinfoil getVar
298 API
299 test:<literallayout>
300 $ oe-selftest -r tinfoil.TinfoilTests.test_getvar
301 </literallayout>It
302 is also possible to run a set of tests. For example the
303 following command will run all of the tinfoil
304 tests:<literallayout>
305 $ oe-selftest -r tinfoil
306 </literallayout></para>
307 </listitem>
308 </itemizedlist>
309 </para></listitem>
310 <listitem><para>
311 <emphasis>testimage:</emphasis>
312 <itemizedlist>
313 <listitem><para>
314 These tests build an image, boot it, and run tests
315 against the image's content.
316 </para></listitem>
317 <listitem><para> The code for these tests resides in <filename>meta/lib/oeqa/runtime/cases/</filename>. </para></listitem>
318 <listitem><para>
319 You need to set the
320 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_CLASSES'><filename>IMAGE_CLASSES</filename></ulink>
321 variable as follows:
322 <literallayout class='monospaced'>
323 IMAGE_CLASSES += "testimage"
324 </literallayout>
325 </para></listitem>
326 <listitem><para>
327 Run the tests using the following command form:
328 <literallayout class='monospaced'>
329 $ bitbake <replaceable>image</replaceable> -c testimage
330 </literallayout>
331 </para></listitem>
332 </itemizedlist>
333 </para></listitem>
334 <listitem><para>
335 <emphasis>testsdk:</emphasis>
336 <itemizedlist>
337 <listitem><para>These tests build an SDK, install it, and then run tests against that SDK. </para></listitem>
338 <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/sdk/cases/</filename>. </para></listitem>
339 <listitem><para>Run the test using the following command form:
340 <literallayout class="monospaced">
341 $ bitbake <replaceable>image</replaceable> -c testsdk
342 </literallayout>
343 </para></listitem>
344 </itemizedlist>
345 </para></listitem>
346 <listitem><para>
347 <emphasis>testsdk_ext:</emphasis>
348 <itemizedlist>
349 <listitem><para>These tests build an extended SDK (eSDK), install that eSDK, and run tests against the eSDK. </para></listitem>
350 <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/esdk</filename>. </para></listitem>
351 <listitem><para>To run the tests, use the following command form:
352 <literallayout class="monospaced">
353 $ bitbake <replaceable>image</replaceable> -c testsdkext
354 </literallayout>
355 </para></listitem>
356 </itemizedlist>
357 </para></listitem>
358
359
360 <listitem><para>
361 <emphasis>oe-build-perf-test:</emphasis>
362 <itemizedlist>
363 <listitem><para>These tests run through commonly used usage scenarios and measure the performance times. </para></listitem>
364 <listitem><para>The code for these tests resides in <filename>meta/lib/oeqa/buildperf</filename>. </para></listitem>
365 <listitem><para>To run the tests, use the following command form:
366 <literallayout class="monospaced">
367 $ oe-build-perf-test <replaceable>options</replaceable>
368 </literallayout>The
369 command takes a number of options, such as where to place the
370 test results. The Autobuilder Helper Scripts include the
371 <filename>build-perf-test-wrapper</filename> script with
372 examples of how to use the oe-build-perf-test from the command
373 line.</para>
374 <para>Use the <filename>oe-git-archive</filename> command to store
375 test results into a Git repository. </para>
376 <para>Use the <filename>oe-build-perf-report</filename> command to
377 generate text reports and HTML reports with graphs of the
378 performance data. For examples, see <link linkend=""
379 >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html</link>
380 and <link linkend=""
381 >http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt</link>.</para></listitem>
382 <listitem>
383 <para>The tests are contained in
384 <filename>lib/oeqa/buildperf/test_basic.py</filename>.</para>
385 </listitem>
386 </itemizedlist>
387 </para></listitem>
388
389
390
391
392 </itemizedlist>
393 </para>
394 </section>
395
396 <section id='test-examples'>
397 <title>Test Examples</title>
398
399 <para>This section provides example tests for each of the tests listed in the <link
400 linkend="test-test-mapping">How Tests Map to Areas of Code</link> section. </para>
401 <para>For oeqa tests, testcases for each area reside in the main test directory at
402 <filename>meta/lib/oeqa/selftest/cases</filename> directory.</para>
403 <para>For oe-selftest. bitbake testcases reside in the <filename>lib/bb/tests/</filename>
404 directory. </para>
405
406 <section id='bitbake-selftest-example'>
407 <title><filename>bitbake-selftest</filename></title>
408
409 <para>A simple test example from <filename>lib/bb/tests/data.py</filename> is:
410 <literallayout>
411 class DataExpansions(unittest.TestCase):
412 def setUp(self):
413 self.d = bb.data.init()
414 self.d["foo"] = "value_of_foo"
415 self.d["bar"] = "value_of_bar"
416 self.d["value_of_foo"] = "value_of_'value_of_foo'"
417
418 def test_one_var(self):
419 val = self.d.expand("${foo}")
420 self.assertEqual(str(val), "value_of_foo")
421 </literallayout>
422 </para>
423 <para>In this example, a <ulink url=""><filename>DataExpansions</filename></ulink> class
424 of tests is created, derived from standard python unittest. The class has a common
425 <filename>setUp</filename> function which is shared by all the tests in the
426 class. A simple test is then added to test that when a variable is expanded, the
427 correct value is found.</para>
428 <para>Bitbake selftests are straightforward python unittest. Refer to the Python
429 unittest documentation for additional information on writing these tests at: <link
430 linkend="">https://docs.python.org/3/library/unittest.html</link>.</para>
431 </section>
432
433 <section id='oe-selftest-example'>
434 <title><filename>oe-selftest</filename></title>
435
436 <para>These tests are more complex due to the setup required behind the scenes for full
437 builds. Rather than directly using Python's unittest, the code wraps most of the
438 standard objects. The tests can be simple, such as testing a command from within the
439 OE build environment using the following
440 example:<literallayout>
441 class BitbakeLayers(OESelftestTestCase):
442 def test_bitbakelayers_showcrossdepends(self):
443 result = runCmd('bitbake-layers show-cross-depends')
444 self.assertTrue('aspell' in result.output, msg = "No dependencies
445 were shown. bitbake-layers show-cross-depends output:
446 %s"% result.output)
447 </literallayout></para>
448 <para>This example, taken from
449 <filename>meta/lib/oeqa/selftest/cases/bblayers.py</filename>, creates a
450 testcase from the <ulink url=""><filename>OESelftestTestCase</filename></ulink>
451 class, derived from <filename>unittest.TestCase</filename>, which runs the
452 <filename>bitbake-layers</filename> command and checks the output to ensure it
453 contains something we know should be here.</para>
454 <para>The <filename>oeqa.utils.commands</filename> module contains Helpers which can
455 assist with common tasks, including:<itemizedlist>
456 <listitem>
457 <para><emphasis>Obtaining the value of a bitbake variable:</emphasis> Use
458 <filename>oeqa.utils.commands.get_bb_var()</filename> or use
459 <filename>oeqa.utils.commands.get_bb_vars()</filename> for more than
460 one variable</para>
461 </listitem>
462 <listitem>
463 <para><emphasis>Running a bitbake invocation for a build:</emphasis> Use
464 <filename>oeqa.utils.commands.bitbake()</filename></para>
465 </listitem>
466 <listitem>
467 <para><emphasis>Running a command:</emphasis> Use
468 <filename>oeqa.utils.commandsrunCmd()</filename></para>
469 </listitem>
470 </itemizedlist></para>
471 <para>There is also a <filename>oeqa.utils.commands.runqemu()</filename> function for
472 launching the <filename>runqemu</filename> command for testing things within a
473 running, virtualized image.</para>
474 <para>You can run these tests in parallel. Parallelism works per test class, so tests
475 within a given test class should always run in the same build, while tests in
476 different classes or modules may be split into different builds. There is no data
477 store available for these tests since the tests launch the
478 <filename>bitbake</filename> command and exist outside of its context. As a
479 result, common bitbake library functions (bb.*) are also unavailable.</para>
480 </section>
481
482 <section id='testimage-example'>
483 <title><filename>testimage</filename></title>
484
485 <para>These tests are run once an image is up and running, either on target hardware or
486 under QEMU. As a result, they are assumed to be running in a target image
487 environment, as opposed to a host build environment. A simple example from
488 <filename>meta/lib/oeqa/runtime/cases/python.py</filename> contains the
489 following:<literallayout>
490 class PythonTest(OERuntimeTestCase):
491 @OETestDepends(['ssh.SSHTest.test_ssh'])
492 @OEHasPackage(['python3-core'])
493 def test_python3(self):
494 cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb,
495 jbeyq', 'rot13'))\""
496 status, output = self.target.run(cmd)
497 msg = 'Exit status was not 0. Output: %s' % output
498 self.assertEqual(status, 0, msg=msg)
499 </literallayout></para>
500 <para>In this example, the <ulink url=""><filename>OERuntimeTestCase</filename></ulink>
501 class wraps <filename>unittest.TestCase</filename>. Within the test,
502 <filename>self.target</filename> represents the target system, where commands
503 can be run on it using the <filename>run()</filename> method. </para>
504 <para>To ensure certain test or package dependencies are met, you can use the
505 <filename>OETestDepends</filename> and <filename>OEHasPackage</filename>
506 decorators. For example, the test in this example would only make sense if
507 python3-core is installed in the image.</para>
508 </section>
509
510 <section id='testsdk_ext-example'>
511 <title><filename>testsdk_ext</filename></title>
512
513 <para>These tests are run against built extensible SDKs (eSDKs). The tests can assume
514 that the eSDK environment has already been setup. An example from
515 <filename>meta/lib/oeqa/sdk/cases/devtool.py</filename> contains the
516 following:<literallayout>
517 class DevtoolTest(OESDKExtTestCase):
518 @classmethod
519 def setUpClass(cls):
520 myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp")
521 cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp")
522 shutil.copytree(myapp_src, cls.myapp_dst)
523 subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst)
524 subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst)
525 subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst)
526
527 @classmethod
528 def tearDownClass(cls):
529 shutil.rmtree(cls.myapp_dst)
530 def _test_devtool_build(self, directory):
531 self._run('devtool add myapp %s' % directory)
532 try:
533 self._run('devtool build myapp')
534 finally:
535 self._run('devtool reset myapp')
536 def test_devtool_build_make(self):
537 self._test_devtool_build(self.myapp_dst)
538 </literallayout>In
539 this example, the <filename>devtool</filename> command is tested to see whether a
540 sample application can be built with the <filename>devtool build</filename> command
541 within the eSDK.</para>
542 </section>
543
544 <section id='testsdk-example'>
545 <title><filename>testsdk</filename></title>
546
547 <para>These tests are run against built SDKs. The tests can assume that an SDK has
548 already been extracted and its environment file has been sourced. A simple example
549 from <filename>meta/lib/oeqa/sdk/cases/python2.py</filename> contains the
550 following:<literallayout>
551 class Python3Test(OESDKTestCase):
552 def setUp(self):
553 if not (self.tc.hasHostPackage("nativesdk-python3-core") or
554 self.tc.hasHostPackage("python3-core-native")):
555 raise unittest.SkipTest("No python3 package in the SDK")
556
557 def test_python3(self):
558 cmd = "python3 -c \"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\""
559 output = self._run(cmd)
560 self.assertEqual(output, "Hello, world\n")
561 </literallayout>In
562 this example, if nativesdk-python3-core has been installed into the SDK, the code
563 runs the python3 interpreter with a basic command to check it is working correctly.
564 The test would only run if python3 is installed in the SDK.</para>
565 </section>
566
567 <section id='oe-build-perf-test-example'>
568 <title><filename>oe-build-perf-test</filename></title>
569
570 <para>The performance tests usually measure how long operations take and the resource
571 utilisation as that happens. An example from
572 <filename>meta/lib/oeqa/buildperf/test_basic.py</filename> contains the
573 following:<literallayout>
574 class Test3(BuildPerfTestCase):
575
576 def test3(self):
577 """Bitbake parsing (bitbake -p)"""
578 # Drop all caches and parse
579 self.rm_cache()
580 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
581 self.measure_cmd_resources(['bitbake', '-p'], 'parse_1',
582 'bitbake -p (no caches)')
583 # Drop tmp/cache
584 oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True)
585 self.measure_cmd_resources(['bitbake', '-p'], 'parse_2',
586 'bitbake -p (no tmp/cache)')
587 # Parse with fully cached data
588 self.measure_cmd_resources(['bitbake', '-p'], 'parse_3',
589 'bitbake -p (cached)')
590 </literallayout>This
591 example shows how three specific parsing timings are measured, with and without
592 various caches, to show how BitBake’s parsing performance trends over time.</para>
593 </section>
594 </section>
595 <section id='test-writing-considerations'>
596 <title>Considerations When Writing Tests</title>
597 <para>When writing good tests, there are several things to keep in mind. Since things
598 running on the Autobuilder are accessed concurrently by multiple workers, consider the
599 following:</para>
600 <formalpara>
601 <title>Running "cleanall" is not permitted</title>
602 <para>This can delete files from DL_DIR which would potentially break other builds
603 running in parallel. If this is required, DL_DIR must be set to an isolated
604 directory.</para>
605 </formalpara>
606 <formalpara>
607 <title>Running "cleansstate" is not permitted</title>
608 <para>This can delete files from SSTATE_DIR which would potentially break other builds
609 running in parallel. If this is required, SSTATE_DIR must be set to an isolated
610 directory. Alternatively, you can use the "-f" option with the
611 <filename>bitbake</filename> command to "taint" tasks by changing the sstate
612 checksums to ensure sstate cache items will not be reused.</para>
613 </formalpara>
614 <formalpara>
615 <title>Tests should not change the metadata</title>
616 <para>This is particularly true for oe-selftests since these can run in parallel and
617 changing metadata leads to changing checksums, which confuses BitBake while running
618 in parallel. If this is necessary, copy layers to a temporary location and modify
619 them. Some tests need to change metadata, such as the devtool tests. To prevent the
620 metadate from changes, set up temporary copies of that data first.</para>
621 </formalpara>
622 </section>
623
624
625
626
627
628
629
630
631</chapter>
632<!--
633vim: expandtab tw=80 ts=4
634-->
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-25 23:04:37 +0000