From e44538f4e4d17c5fcdebc973e45e0b1b56e58803 Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Mon, 23 Sep 2013 09:19:53 -0700 Subject: ref-manual, dev-manual: Applied review comments for test section The section and variables for performing automated runtime tests on images was reviewed by Paul Eggleton. The suggested changes were made. (From yocto-docs rev: c2f84ea3c162892e4da3df30fb833f88bab3d3cc) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- .../dev-manual/dev-manual-common-tasks.xml | 113 ++++++++++++--------- documentation/ref-manual/ref-classes.xml | 16 +-- documentation/ref-manual/ref-variables.xml | 61 ++++++----- 3 files changed, 109 insertions(+), 81 deletions(-) (limited to 'documentation') diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml index 2dbc3d14b8..4d1b731fa3 100644 --- a/documentation/dev-manual/dev-manual-common-tasks.xml +++ b/documentation/dev-manual/dev-manual-common-tasks.xml @@ -4252,24 +4252,22 @@ -
- Running and Writing Tests for a QEMU Image +
+ Performing Automated Runtime Testing The OpenEmbedded build system makes available a series of automated - tests for QEMU images. - These tests are commands that run on the target system over - ssh and are written in Python, thus making - use of the unittest module. - - - + tests for images. + + The current release of Yocto Project supports these tests + for QEMU images only. + + These tests are written in Python making use of the + unittest module, and the majority of them + run commands on the target system over + ssh. This section describes how you set up the environment to use these tests, run available tests, and write and add your own tests. - You can the same information in a different form and with example - log files of test runs on the - Image Tests - Wiki page.
@@ -4278,9 +4276,8 @@ In order to run tests, you need to do the following: - Ensure you run the test - automatically with no interaction and under - sudo: + Set up to avoid interaction + with sudo for networking: To accomplish this, you must do one of the following: @@ -4300,12 +4297,13 @@ Manually configure a tap interface for your system. Run as root the script in - scripts/runqemu-gen-tapdev, + scripts/runqemu-gen-tapdevs, which should generate a list of tap devices. - List generation is usually done in environments - similar to AutoBuilder. + This is the option typically chosen for + Autobuilder-type environments. + - Set up the + Set the DISPLAY variable: You need to set this variable so that you have an X server available (e.g. start @@ -4321,11 +4319,10 @@ ${DEPLOY_DIR}/rpm so it can run smart channel commands. That means your host's firewall must accept incoming connections from 192.168.7.0/24, - which is the default class used for - tap0 devices by - runqemu. - Have your QEMU image built: - The QEMU image on which you want to run tests needs to + which is the default IP range used for tap devices + by runqemu. + Have your image built: + The image on which you want to run tests needs to be built. For example, the following command would build the core-image-sato image when @@ -4333,18 +4330,34 @@ bitbake core-image-sato - Globally inherit - testimage.class: - Edit your local.conf file as + Run the tests: + You can start the tests automatically or manually. + To have the tests start automatically after the + OpenEmbedded build system successfully creates a QEMU + image, set the + TEST_IMAGE + variable to "1" in your + local.conf file in the + Build Directory. + To manually run the tests, globally inherit + testimage.class: + by editing your local.conf file as follows: INHERIT += "testimage" + + After editing your + local.conf file, use BitBake to + run the tests: + + bitbake -c testimage <qemu-image> - If you built your QEMU image using rm_work, + Regardless of how you initiate the tests, if you built your + image using rm_work, most of the tests will fail with errors because they rely on ${WORKDIR}/installed_pkgs.txt. @@ -4355,14 +4368,16 @@ After setting up to run the tests, use BitBake to start - the standard suite of tests: + the standard suite of tests manually or, if you are running + them automatically, build your image: bitbake core-image-sato -c testimage + bitbake core-image-sato - Once you start the tests, the following happens: + Once the tests start, the following happens: A copy of the root filesystem is written to ${WORKDIR}/testimage. @@ -4400,10 +4415,10 @@ All test files reside in meta/lib/oeqa/runtime in the Source Directory. - Tests (also referred to as modules) have a one-to-one - relationship with the filenames in runtime. - Modules can have multiple classes and test methods and are - usually grouped together by the area tested (e.g tests for + A test name (sometimes referred to as a module) maps to a + method name within a class within a module. + Tests can be used within multiple classes and the tests + are usually grouped together by the area tested (e.g tests for systemd reside in meta/lib/oeqa/runtime/systemd.py). @@ -4412,12 +4427,13 @@ You can add tests to any layer provided you place them in the proper area and you extend BBPATH - in the local.conf file. + in the local.conf file as normal. Be sure that tests reside in <layer>/lib/oeqa/runtime. Be sure that module names do not collide with module names - provided by the Yocto Project. + used in the default set of test modules in + meta/lib/oeqa/runtime. @@ -4458,12 +4474,12 @@ Run the default set of tests simply by running the following: - bitbake <qemu_image> -c testimage + bitbake <image> -c testimage The default tests for the image are defined as: - DEFAULT_TEST_SUITES_pn-<qemu_image> = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg" + DEFAULT_TEST_SUITES_pn-<image> = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg" Add your own test to the list of the by using the following: @@ -4487,15 +4503,16 @@ As mentioned previously, all new test files need to be in the proper place for the build system to find them. - New tests can go in the - Source Directory at - meta/lib/oeqa/runtime. - Alternatively, a layer can add its own tests in - <layer>/lib/oeqa/runtime as long - as you extend BBPATH. - Just remember that filenames need to map directory to test - (module) names and you not use module names that collide with - existing core tests. + New tests for additional functionality outside of the core + should be added to the layer that adds the functionality, in + <layer>/lib/oeqa/runtime (as + long as + BBPATH + is extended in the layer's + layer.conf file as normal. + Just remember that filenames need to map directly to test + (module) names and that you do not use module names that + collide with existing core tests. diff --git a/documentation/ref-manual/ref-classes.xml b/documentation/ref-manual/ref-classes.xml index 5870d0f44a..b1a8e64ce0 100644 --- a/documentation/ref-manual/ref-classes.xml +++ b/documentation/ref-manual/ref-classes.xml @@ -940,8 +940,12 @@ You can use this class to enable running a series of automated tests - for QEMU images. - The class handles loading the tests and starting the QEMU image. + for images. + The class handles loading the tests and starting the image. + + The current release of Yocto Project supports these tests + for QEMU images only. + @@ -950,15 +954,13 @@ The tests are commands that run on the target system over ssh. they are written in Python and make use of the - unittest modulebuild software. + unittest module. For information on how to enable, run, and create new tests, see the - "Running and Writing Tests for a QEMU Image" - section and the - Image Tests - Wiki page. + "Performing Automated Runtime Testing" + section.
diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml index 662a230650..d71378d271 100644 --- a/documentation/ref-manual/ref-variables.xml +++ b/documentation/ref-manual/ref-variables.xml @@ -5220,21 +5220,27 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3" TEST_IMAGE - Automatically runs the series of automated tests for QEMU - images when a QEMU image is successfully built. - These tests are commands that run on the target system over - ssh and are written in Python. + Automatically runs the series of automated tests for + images when an image is successfully built. + + The current release of Yocto Project supports these tests + for QEMU images only. + + These tests are written in Python making use of the + unittest module, and the majority of + them run commands on the target system over + ssh. You can set this variable to "1" in your local.conf file in the Build Directory to have the OpenEmbedded build system automatically run - these tests after a QEMU image successfully builds: + these tests after an image successfully builds: TEST_IMAGE = "1" For more information on enabling, running, and writing these tests, see the - "Running and Writing Tests for a QEMU Image" + "Performing Automated Runtime Testing" section in the Yocto Project Development Manual and the "testimage.bbclass" section. @@ -5245,24 +5251,18 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3" TEST_QEMUBOOT_TIMEOUT - The time in seconds allowed for a QEMU image to boot before - tests begin to run against the image using the following - command: - - bitbake <qemu_image> -c testimage - - - - + The time in seconds allowed for an image to boot before + automated runtime tests begin to run against an + image. The default timeout period to allow the boot process to reach the login prompt is 500 seconds. - You can edit this default value in the + You can specify a different value in the local.conf file. - For more information on testing QEMU images, see the - "Running and Writing Tests for a QEMU Image" + For more information on testing images, see the + "Performing Automated Runtime Testing" section in the Yocto Project Development Manual. @@ -5272,15 +5272,16 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3" An ordered list of tests (modules) to run against - a QEMU image using the following command: - - bitbake <qemu_image> -c testimage - + an image when performing automated runtime testing. - The Yocto Project provides a core set of tests that can - be used against QEMU images. + The OpenEmbedded build system provides a core set of tests + that can be used against images. + + The current release of Yocto Project supports these + tests for QEMU images only. + Tests include ping, ssh, df among others. @@ -5289,6 +5290,14 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3" TEST_SUITES_append = " mytest" + Alternatively, you can provide the "auto" option to + have all applicable tests run against the image. + + TEST_SUITES_append = " auto" + + Using this option causes the build system to automatically + run tests that are applicable to the image. + Tests that are not applicable are skipped. @@ -5307,8 +5316,8 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3" - For more information on testing QEMU images, see the - "Running and Writing Tests for a QEMU Image" + For more information on testing images, see the + "Performing Automated Runtime Testing" section in the Yocto Project Development Manual. -- cgit v1.2.3-54-g00ecf