summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorScott Rifenbark <scott.m.rifenbark@intel.com>2013-09-23 09:19:53 -0700
committerRichard Purdie <richard.purdie@linuxfoundation.org>2013-10-01 22:52:51 +0100
commite44538f4e4d17c5fcdebc973e45e0b1b56e58803 (patch)
tree6fa92058f47c87a4f722ba981bcbf2f42f53e0f4
parent56c4de93b94c60a827d378488cbe13a0bb59c2e2 (diff)
downloadpoky-e44538f4e4d17c5fcdebc973e45e0b1b56e58803.tar.gz
ref-manual, dev-manual: Applied review comments for test section
The section and variables for performing automated runtime tests on images was reviewed by Paul Eggleton. The suggested changes were made. (From yocto-docs rev: c2f84ea3c162892e4da3df30fb833f88bab3d3cc) Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-rw-r--r--documentation/dev-manual/dev-manual-common-tasks.xml113
-rw-r--r--documentation/ref-manual/ref-classes.xml16
-rw-r--r--documentation/ref-manual/ref-variables.xml61
3 files changed, 109 insertions, 81 deletions
diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml
index 2dbc3d14b8..4d1b731fa3 100644
--- a/documentation/dev-manual/dev-manual-common-tasks.xml
+++ b/documentation/dev-manual/dev-manual-common-tasks.xml
@@ -4252,24 +4252,22 @@
4252 </section> 4252 </section>
4253 </section> 4253 </section>
4254 4254
4255 <section id="running-and-writing-tests-for-a-qemu-image"> 4255 <section id="performing-automated-runtime-testing">
4256 <title>Running and Writing Tests for a QEMU Image</title> 4256 <title>Performing Automated Runtime Testing</title>
4257 4257
4258 <para> 4258 <para>
4259 The OpenEmbedded build system makes available a series of automated 4259 The OpenEmbedded build system makes available a series of automated
4260 tests for QEMU images. 4260 tests for images.
4261 These tests are commands that run on the target system over 4261 <note>
4262 <filename>ssh</filename> and are written in Python, thus making 4262 The current release of Yocto Project supports these tests
4263 use of the <filename>unittest</filename> module. 4263 for QEMU images only.
4264 </para> 4264 </note>
4265 4265 These tests are written in Python making use of the
4266 <para> 4266 <filename>unittest</filename> module, and the majority of them
4267 run commands on the target system over
4268 <filename>ssh</filename>.
4267 This section describes how you set up the environment to use these 4269 This section describes how you set up the environment to use these
4268 tests, run available tests, and write and add your own tests. 4270 tests, run available tests, and write and add your own tests.
4269 You can the same information in a different form and with example
4270 log files of test runs on the
4271 <ulink url='https://wiki.yoctoproject.org/wiki/Image_tests'>Image Tests</ulink>
4272 Wiki page.
4273 </para> 4271 </para>
4274 4272
4275 <section id="qemu-image-enabling-tests"> 4273 <section id="qemu-image-enabling-tests">
@@ -4278,9 +4276,8 @@
4278 <para> 4276 <para>
4279 In order to run tests, you need to do the following: 4277 In order to run tests, you need to do the following:
4280 <itemizedlist> 4278 <itemizedlist>
4281 <listitem><para><emphasis>Ensure you run the test 4279 <listitem><para><emphasis>Set up to avoid interaction
4282 automatically with no interaction and under 4280 with <filename>sudo</filename> for networking:</emphasis>
4283 <filename>sudo</filename>:</emphasis>
4284 To accomplish this, you must do one of the 4281 To accomplish this, you must do one of the
4285 following: 4282 following:
4286 <itemizedlist> 4283 <itemizedlist>
@@ -4300,12 +4297,13 @@
4300 <listitem><para>Manually configure a tap interface 4297 <listitem><para>Manually configure a tap interface
4301 for your system.</para></listitem> 4298 for your system.</para></listitem>
4302 <listitem><para>Run as root the script in 4299 <listitem><para>Run as root the script in
4303 <filename>scripts/runqemu-gen-tapdev</filename>, 4300 <filename>scripts/runqemu-gen-tapdevs</filename>,
4304 which should generate a list of tap devices. 4301 which should generate a list of tap devices.
4305 List generation is usually done in environments 4302 This is the option typically chosen for
4306 similar to AutoBuilder.</para></listitem> 4303 Autobuilder-type environments.
4304 </para></listitem>
4307 </itemizedlist></para></listitem> 4305 </itemizedlist></para></listitem>
4308 <listitem><para><emphasis>Set up the 4306 <listitem><para><emphasis>Set the
4309 <filename>DISPLAY</filename> variable:</emphasis> 4307 <filename>DISPLAY</filename> variable:</emphasis>
4310 You need to set this variable so that you have an X 4308 You need to set this variable so that you have an X
4311 server available (e.g. start 4309 server available (e.g. start
@@ -4321,11 +4319,10 @@
4321 <filename>${DEPLOY_DIR}/rpm</filename> so it can run 4319 <filename>${DEPLOY_DIR}/rpm</filename> so it can run
4322 smart channel commands. That means your host's firewall 4320 smart channel commands. That means your host's firewall
4323 must accept incoming connections from 192.168.7.0/24, 4321 must accept incoming connections from 192.168.7.0/24,
4324 which is the default class used for 4322 which is the default IP range used for tap devices
4325 <filename>tap0</filename> devices by 4323 by <filename>runqemu</filename>.</para></listitem>
4326 <filename>runqemu</filename>.</para></listitem> 4324 <listitem><para><emphasis>Have your image built:</emphasis>
4327 <listitem><para><emphasis>Have your QEMU image built:</emphasis> 4325 The image on which you want to run tests needs to
4328 The QEMU image on which you want to run tests needs to
4329 be built. 4326 be built.
4330 For example, the following command would build the 4327 For example, the following command would build the
4331 <filename>core-image-sato</filename> image when 4328 <filename>core-image-sato</filename> image when
@@ -4333,18 +4330,34 @@
4333 <literallayout class='monospaced'> 4330 <literallayout class='monospaced'>
4334 bitbake core-image-sato 4331 bitbake core-image-sato
4335 </literallayout></para></listitem> 4332 </literallayout></para></listitem>
4336 <listitem><para><emphasis>Globally inherit 4333 <listitem><para><emphasis>Run the tests:</emphasis>
4337 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage'><filename>testimage.class</filename>:</ulink></emphasis> 4334 You can start the tests automatically or manually.
4338 Edit your <filename>local.conf</filename> file as 4335 To have the tests start automatically after the
4336 OpenEmbedded build system successfully creates a QEMU
4337 image, set the
4338 <ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_IMAGE'><filename>TEST_IMAGE</filename></ulink>
4339 variable to "1" in your
4340 <filename>local.conf</filename> file in the
4341 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
4342 To manually run the tests, globally inherit
4343 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage'><filename>testimage.class</filename>:</ulink>
4344 by editing your <filename>local.conf</filename> file as
4339 follows: 4345 follows:
4340 <literallayout class='monospaced'> 4346 <literallayout class='monospaced'>
4341 INHERIT += "testimage" 4347 INHERIT += "testimage"
4348 </literallayout>
4349 After editing your
4350 <filename>local.conf</filename> file, use BitBake to
4351 run the tests:
4352 <literallayout class='monospaced'>
4353 bitbake -c testimage &lt;qemu-image&gt;
4342 </literallayout></para></listitem> 4354 </literallayout></para></listitem>
4343 </itemizedlist> 4355 </itemizedlist>
4344 </para> 4356 </para>
4345 4357
4346 <note> 4358 <note>
4347 If you built your QEMU image using <filename>rm_work</filename>, 4359 Regardless of how you initiate the tests, if you built your
4360 image using <filename>rm_work</filename>,
4348 most of the tests will fail with errors because they rely on 4361 most of the tests will fail with errors because they rely on
4349 <filename>${WORKDIR}/installed_pkgs.txt</filename>. 4362 <filename>${WORKDIR}/installed_pkgs.txt</filename>.
4350 </note> 4363 </note>
@@ -4355,14 +4368,16 @@
4355 4368
4356 <para> 4369 <para>
4357 After setting up to run the tests, use BitBake to start 4370 After setting up to run the tests, use BitBake to start
4358 the standard suite of tests: 4371 the standard suite of tests manually or, if you are running
4372 them automatically, build your image:
4359 <literallayout class='monospaced'> 4373 <literallayout class='monospaced'>
4360 bitbake core-image-sato -c testimage 4374 bitbake core-image-sato -c testimage
4375 bitbake core-image-sato
4361 </literallayout> 4376 </literallayout>
4362 </para> 4377 </para>
4363 4378
4364 <para> 4379 <para>
4365 Once you start the tests, the following happens: 4380 Once the tests start, the following happens:
4366 <itemizedlist> 4381 <itemizedlist>
4367 <listitem><para>A copy of the root filesystem is written 4382 <listitem><para>A copy of the root filesystem is written
4368 to <filename>${WORKDIR}/testimage</filename>. 4383 to <filename>${WORKDIR}/testimage</filename>.
@@ -4400,10 +4415,10 @@
4400 All test files reside in 4415 All test files reside in
4401 <filename>meta/lib/oeqa/runtime</filename> in the 4416 <filename>meta/lib/oeqa/runtime</filename> in the
4402 <link linkend='source-directory'>Source Directory</link>. 4417 <link linkend='source-directory'>Source Directory</link>.
4403 Tests (also referred to as modules) have a one-to-one 4418 A test name (sometimes referred to as a module) maps to a
4404 relationship with the filenames in <filename>runtime</filename>. 4419 method name within a class within a module.
4405 Modules can have multiple classes and test methods and are 4420 Tests can be used within multiple classes and the tests
4406 usually grouped together by the area tested (e.g tests for 4421 are usually grouped together by the area tested (e.g tests for
4407 <filename>systemd</filename> reside in 4422 <filename>systemd</filename> reside in
4408 <filename>meta/lib/oeqa/runtime/systemd.py</filename>). 4423 <filename>meta/lib/oeqa/runtime/systemd.py</filename>).
4409 </para> 4424 </para>
@@ -4412,12 +4427,13 @@
4412 You can add tests to any layer provided you place them in the 4427 You can add tests to any layer provided you place them in the
4413 proper area and you extend 4428 proper area and you extend
4414 <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink> 4429 <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
4415 in the <filename>local.conf</filename> file. 4430 in the <filename>local.conf</filename> file as normal.
4416 Be sure that tests reside in 4431 Be sure that tests reside in
4417 <filename>&lt;layer&gt;/lib/oeqa/runtime</filename>. 4432 <filename>&lt;layer&gt;/lib/oeqa/runtime</filename>.
4418 <note> 4433 <note>
4419 Be sure that module names do not collide with module names 4434 Be sure that module names do not collide with module names
4420 provided by the Yocto Project. 4435 used in the default set of test modules in
4436 <filename>meta/lib/oeqa/runtime</filename>.
4421 </note> 4437 </note>
4422 </para> 4438 </para>
4423 4439
@@ -4458,12 +4474,12 @@
4458 <listitem><para>Run the default set of tests simply by 4474 <listitem><para>Run the default set of tests simply by
4459 running the following: 4475 running the following:
4460 <literallayout class='monospaced'> 4476 <literallayout class='monospaced'>
4461 bitbake &lt;qemu_image&gt; -c testimage 4477 bitbake &lt;image&gt; -c testimage
4462 </literallayout></para></listitem> 4478 </literallayout></para></listitem>
4463 <listitem><para>The default tests for the image are defined 4479 <listitem><para>The default tests for the image are defined
4464 as: 4480 as:
4465 <literallayout class='monospaced'> 4481 <literallayout class='monospaced'>
4466 DEFAULT_TEST_SUITES_pn-&lt;qemu_image&gt; = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg" 4482 DEFAULT_TEST_SUITES_pn-&lt;image&gt; = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg"
4467 </literallayout></para></listitem> 4483 </literallayout></para></listitem>
4468 <listitem><para>Add your own test to the list of the 4484 <listitem><para>Add your own test to the list of the
4469 by using the following: 4485 by using the following:
@@ -4487,15 +4503,16 @@
4487 <para> 4503 <para>
4488 As mentioned previously, all new test files need to be in the 4504 As mentioned previously, all new test files need to be in the
4489 proper place for the build system to find them. 4505 proper place for the build system to find them.
4490 New tests can go in the 4506 New tests for additional functionality outside of the core
4491 <link linkend='source-directory'>Source Directory</link> at 4507 should be added to the layer that adds the functionality, in
4492 <filename>meta/lib/oeqa/runtime</filename>. 4508 <filename>&lt;layer&gt;/lib/oeqa/runtime</filename> (as
4493 Alternatively, a layer can add its own tests in 4509 long as
4494 <filename>&lt;layer&gt;/lib/oeqa/runtime</filename> as long 4510 <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
4495 as you extend <filename>BBPATH</filename>. 4511 is extended in the layer's
4496 Just remember that filenames need to map directory to test 4512 <filename>layer.conf</filename> file as normal.
4497 (module) names and you not use module names that collide with 4513 Just remember that filenames need to map directly to test
4498 existing core tests. 4514 (module) names and that you do not use module names that
4515 collide with existing core tests.
4499 </para> 4516 </para>
4500 4517
4501 <para> 4518 <para>
diff --git a/documentation/ref-manual/ref-classes.xml b/documentation/ref-manual/ref-classes.xml
index 5870d0f44a..b1a8e64ce0 100644
--- a/documentation/ref-manual/ref-classes.xml
+++ b/documentation/ref-manual/ref-classes.xml
@@ -940,8 +940,12 @@
940 940
941 <para> 941 <para>
942 You can use this class to enable running a series of automated tests 942 You can use this class to enable running a series of automated tests
943 for QEMU images. 943 for images.
944 The class handles loading the tests and starting the QEMU image. 944 The class handles loading the tests and starting the image.
945 <note>
946 The current release of Yocto Project supports these tests
947 for QEMU images only.
948 </note>
945 </para> 949 </para>
946 950
947 <para> 951 <para>
@@ -950,15 +954,13 @@
950 The tests are commands that run on the target system over 954 The tests are commands that run on the target system over
951 <filename>ssh</filename>. 955 <filename>ssh</filename>.
952 they are written in Python and make use of the 956 they are written in Python and make use of the
953 <filename>unittest modulebuild</filename> software. 957 <filename>unittest</filename> module.
954 </para> 958 </para>
955 959
956 <para> 960 <para>
957 For information on how to enable, run, and create new tests, see the 961 For information on how to enable, run, and create new tests, see the
958 "<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>" 962 "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
959 section and the 963 section.
960 <ulink url='https://wiki.yoctoproject.org/wiki/Image_tests'>Image Tests</ulink>
961 Wiki page.
962 </para> 964 </para>
963</section> 965</section>
964 966
diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index 662a230650..d71378d271 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -5220,21 +5220,27 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
5220 <glossentry id='var-TEST_IMAGE'><glossterm>TEST_IMAGE</glossterm> 5220 <glossentry id='var-TEST_IMAGE'><glossterm>TEST_IMAGE</glossterm>
5221 <glossdef> 5221 <glossdef>
5222 <para> 5222 <para>
5223 Automatically runs the series of automated tests for QEMU 5223 Automatically runs the series of automated tests for
5224 images when a QEMU image is successfully built. 5224 images when an image is successfully built.
5225 These tests are commands that run on the target system over 5225 <note>
5226 <filename>ssh</filename> and are written in Python. 5226 The current release of Yocto Project supports these tests
5227 for QEMU images only.
5228 </note>
5229 These tests are written in Python making use of the
5230 <filename>unittest</filename> module, and the majority of
5231 them run commands on the target system over
5232 <filename>ssh</filename>.
5227 You can set this variable to "1" in your 5233 You can set this variable to "1" in your
5228 <filename>local.conf</filename> file in the 5234 <filename>local.conf</filename> file in the
5229 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink> 5235 <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
5230 to have the OpenEmbedded build system automatically run 5236 to have the OpenEmbedded build system automatically run
5231 these tests after a QEMU image successfully builds: 5237 these tests after an image successfully builds:
5232 <literallayout class='monospaced'> 5238 <literallayout class='monospaced'>
5233 TEST_IMAGE = "1" 5239 TEST_IMAGE = "1"
5234 </literallayout> 5240 </literallayout>
5235 For more information on enabling, running, and writing 5241 For more information on enabling, running, and writing
5236 these tests, see the 5242 these tests, see the
5237 "<ulink url='&YOCTO_DOCS_DEV_URL;#running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>" 5243 "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
5238 section in the Yocto Project Development Manual and the 5244 section in the Yocto Project Development Manual and the
5239 "<link linkend='ref-classes-testimage'><filename>testimage.bbclass</filename></link>" 5245 "<link linkend='ref-classes-testimage'><filename>testimage.bbclass</filename></link>"
5240 section. 5246 section.
@@ -5245,24 +5251,18 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
5245 <glossentry id='var-TEST_QEMUBOOT_TIMEOUT'><glossterm>TEST_QEMUBOOT_TIMEOUT</glossterm> 5251 <glossentry id='var-TEST_QEMUBOOT_TIMEOUT'><glossterm>TEST_QEMUBOOT_TIMEOUT</glossterm>
5246 <glossdef> 5252 <glossdef>
5247 <para> 5253 <para>
5248 The time in seconds allowed for a QEMU image to boot before 5254 The time in seconds allowed for an image to boot before
5249 tests begin to run against the image using the following 5255 automated runtime tests begin to run against an
5250 command: 5256 image.
5251 <literallayout class='monospaced'>
5252 bitbake &lt;qemu_image&gt; -c testimage
5253 </literallayout>
5254 </para>
5255
5256 <para>
5257 The default timeout period to allow the boot process to 5257 The default timeout period to allow the boot process to
5258 reach the login prompt is 500 seconds. 5258 reach the login prompt is 500 seconds.
5259 You can edit this default value in the 5259 You can specify a different value in the
5260 <filename>local.conf</filename> file. 5260 <filename>local.conf</filename> file.
5261 </para> 5261 </para>
5262 5262
5263 <para> 5263 <para>
5264 For more information on testing QEMU images, see the 5264 For more information on testing images, see the
5265 "<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>" 5265 "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
5266 section in the Yocto Project Development Manual. 5266 section in the Yocto Project Development Manual.
5267 </para> 5267 </para>
5268 </glossdef> 5268 </glossdef>
@@ -5272,15 +5272,16 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
5272 <glossdef> 5272 <glossdef>
5273 <para> 5273 <para>
5274 An ordered list of tests (modules) to run against 5274 An ordered list of tests (modules) to run against
5275 a QEMU image using the following command: 5275 an image when performing automated runtime testing.
5276 <literallayout class='monospaced'>
5277 bitbake &lt;qemu_image&gt; -c testimage
5278 </literallayout>
5279 </para> 5276 </para>
5280 5277
5281 <para> 5278 <para>
5282 The Yocto Project provides a core set of tests that can 5279 The OpenEmbedded build system provides a core set of tests
5283 be used against QEMU images. 5280 that can be used against images.
5281 <note>
5282 The current release of Yocto Project supports these
5283 tests for QEMU images only.
5284 </note>
5284 Tests include <filename>ping</filename>, 5285 Tests include <filename>ping</filename>,
5285 <filename>ssh</filename>, <filename>df</filename> among 5286 <filename>ssh</filename>, <filename>df</filename> among
5286 others. 5287 others.
@@ -5289,6 +5290,14 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
5289 <literallayout class='monospaced'> 5290 <literallayout class='monospaced'>
5290 TEST_SUITES_append = " mytest" 5291 TEST_SUITES_append = " mytest"
5291 </literallayout> 5292 </literallayout>
5293 Alternatively, you can provide the "auto" option to
5294 have all applicable tests run against the image.
5295 <literallayout class='monospaced'>
5296 TEST_SUITES_append = " auto"
5297 </literallayout>
5298 Using this option causes the build system to automatically
5299 run tests that are applicable to the image.
5300 Tests that are not applicable are skipped.
5292 </para> 5301 </para>
5293 5302
5294 <para> 5303 <para>
@@ -5307,8 +5316,8 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
5307 </para> 5316 </para>
5308 5317
5309 <para> 5318 <para>
5310 For more information on testing QEMU images, see the 5319 For more information on testing images, see the
5311 "<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>" 5320 "<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
5312 section in the Yocto Project Development Manual. 5321 section in the Yocto Project Development Manual.
5313 </para> 5322 </para>
5314 </glossdef> 5323 </glossdef>