diff options
| -rw-r--r-- | documentation/dev-manual/packages.rst | 112 | ||||
| -rw-r--r-- | documentation/migration-guides/migration-1.6.rst | 2 | ||||
| -rw-r--r-- | documentation/ref-manual/classes.rst | 4 | ||||
| -rw-r--r-- | documentation/ref-manual/features.rst | 2 | ||||
| -rw-r--r-- | documentation/ref-manual/qa-checks.rst | 2 | ||||
| -rw-r--r-- | documentation/ref-manual/release-process.rst | 2 | ||||
| -rw-r--r-- | documentation/ref-manual/variables.rst | 2 | ||||
| -rw-r--r-- | documentation/test-manual/index.rst | 1 | ||||
| -rw-r--r-- | documentation/test-manual/intro.rst | 2 | ||||
| -rw-r--r-- | documentation/test-manual/ptest.rst | 114 |
10 files changed, 126 insertions, 117 deletions
diff --git a/documentation/dev-manual/packages.rst b/documentation/dev-manual/packages.rst index 33cf78747d..4ba2dcae3a 100644 --- a/documentation/dev-manual/packages.rst +++ b/documentation/dev-manual/packages.rst | |||
| @@ -16,7 +16,7 @@ This section describes a few tasks that involve packages: | |||
| 16 | - :ref:`dev-manual/packages:generating and using signed packages` | 16 | - :ref:`dev-manual/packages:generating and using signed packages` |
| 17 | 17 | ||
| 18 | - :ref:`Setting up and running package test | 18 | - :ref:`Setting up and running package test |
| 19 | (ptest) <dev-manual/packages:testing packages with ptest>` | 19 | (ptest) <test-manual/ptest:testing packages with ptest>` |
| 20 | 20 | ||
| 21 | - :ref:`dev-manual/packages:creating node package manager (npm) packages` | 21 | - :ref:`dev-manual/packages:creating node package manager (npm) packages` |
| 22 | 22 | ||
| @@ -882,114 +882,8 @@ related to signed package feeds are available: | |||
| 882 | Testing Packages With ptest | 882 | Testing Packages With ptest |
| 883 | =========================== | 883 | =========================== |
| 884 | 884 | ||
| 885 | A Package Test (ptest) runs tests against packages built by the | 885 | See the :ref:`test-manual/ptest:Testing Packages With ptest` section of the |
| 886 | OpenEmbedded build system on the target machine. A ptest contains at | 886 | Yocto Project Test Environment Manual. |
| 887 | least two items: the actual test, and a shell script (``run-ptest``) | ||
| 888 | that starts the test. The shell script that starts the test must not | ||
| 889 | contain the actual test --- the script only starts the test. On the other | ||
| 890 | hand, the test can be anything from a simple shell script that runs a | ||
| 891 | binary and checks the output to an elaborate system of test binaries and | ||
| 892 | data files. | ||
| 893 | |||
| 894 | The test generates output in the format used by Automake:: | ||
| 895 | |||
| 896 | result: testname | ||
| 897 | |||
| 898 | where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and | ||
| 899 | the testname can be any identifying string. | ||
| 900 | |||
| 901 | For a list of Yocto Project recipes that are already enabled with ptest, | ||
| 902 | see the :yocto_wiki:`Ptest </Ptest>` wiki page. | ||
| 903 | |||
| 904 | .. note:: | ||
| 905 | |||
| 906 | A recipe is "ptest-enabled" if it inherits the :ref:`ref-classes-ptest` | ||
| 907 | class. | ||
| 908 | |||
| 909 | Adding ptest to Your Build | ||
| 910 | -------------------------- | ||
| 911 | |||
| 912 | To add package testing to your build, add the :term:`DISTRO_FEATURES` and | ||
| 913 | :term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which | ||
| 914 | is found in the :term:`Build Directory`:: | ||
| 915 | |||
| 916 | DISTRO_FEATURES:append = " ptest" | ||
| 917 | EXTRA_IMAGE_FEATURES += "ptest-pkgs" | ||
| 918 | |||
| 919 | Once your build is complete, the ptest files are installed into the | ||
| 920 | ``/usr/lib/package/ptest`` directory within the image, where ``package`` | ||
| 921 | is the name of the package. | ||
| 922 | |||
| 923 | Running ptest | ||
| 924 | ------------- | ||
| 925 | |||
| 926 | The ``ptest-runner`` package installs a shell script that loops through | ||
| 927 | all installed ptest test suites and runs them in sequence. Consequently, | ||
| 928 | you might want to add this package to your image. | ||
| 929 | |||
| 930 | Getting Your Package Ready | ||
| 931 | -------------------------- | ||
| 932 | |||
| 933 | In order to enable a recipe to run installed ptests on target hardware, | ||
| 934 | you need to prepare the recipes that build the packages you want to | ||
| 935 | test. Here is what you have to do for each recipe: | ||
| 936 | |||
| 937 | - *Be sure the recipe inherits the* :ref:`ref-classes-ptest` *class:* | ||
| 938 | Include the following line in each recipe:: | ||
| 939 | |||
| 940 | inherit ptest | ||
| 941 | |||
| 942 | - *Create run-ptest:* This script starts your test. Locate the | ||
| 943 | script where you will refer to it using | ||
| 944 | :term:`SRC_URI`. Here is an | ||
| 945 | example that starts a test for ``dbus``:: | ||
| 946 | |||
| 947 | #!/bin/sh | ||
| 948 | cd test | ||
| 949 | make -k runtest-TESTS | ||
| 950 | |||
| 951 | - *Ensure dependencies are met:* If the test adds build or runtime | ||
| 952 | dependencies that normally do not exist for the package (such as | ||
| 953 | requiring "make" to run the test suite), use the | ||
| 954 | :term:`DEPENDS` and | ||
| 955 | :term:`RDEPENDS` variables in | ||
| 956 | your recipe in order for the package to meet the dependencies. Here | ||
| 957 | is an example where the package has a runtime dependency on "make":: | ||
| 958 | |||
| 959 | RDEPENDS:${PN}-ptest += "make" | ||
| 960 | |||
| 961 | - *Add a function to build the test suite:* Not many packages support | ||
| 962 | cross-compilation of their test suites. Consequently, you usually | ||
| 963 | need to add a cross-compilation function to the package. | ||
| 964 | |||
| 965 | Many packages based on Automake compile and run the test suite by | ||
| 966 | using a single command such as ``make check``. However, the host | ||
| 967 | ``make check`` builds and runs on the same computer, while | ||
| 968 | cross-compiling requires that the package is built on the host but | ||
| 969 | executed for the target architecture (though often, as in the case | ||
| 970 | for ptest, the execution occurs on the host). The built version of | ||
| 971 | Automake that ships with the Yocto Project includes a patch that | ||
| 972 | separates building and execution. Consequently, packages that use the | ||
| 973 | unaltered, patched version of ``make check`` automatically | ||
| 974 | cross-compiles. | ||
| 975 | |||
| 976 | Regardless, you still must add a ``do_compile_ptest`` function to | ||
| 977 | build the test suite. Add a function similar to the following to your | ||
| 978 | recipe:: | ||
| 979 | |||
| 980 | do_compile_ptest() { | ||
| 981 | oe_runmake buildtest-TESTS | ||
| 982 | } | ||
| 983 | |||
| 984 | - *Ensure special configurations are set:* If the package requires | ||
| 985 | special configurations prior to compiling the test code, you must | ||
| 986 | insert a ``do_configure_ptest`` function into the recipe. | ||
| 987 | |||
| 988 | - *Install the test suite:* The :ref:`ref-classes-ptest` class | ||
| 989 | automatically copies the file ``run-ptest`` to the target and then runs make | ||
| 990 | ``install-ptest`` to run the tests. If this is not enough, you need | ||
| 991 | to create a ``do_install_ptest`` function and make sure it gets | ||
| 992 | called after the "make install-ptest" completes. | ||
| 993 | 887 | ||
| 994 | Creating Node Package Manager (NPM) Packages | 888 | Creating Node Package Manager (NPM) Packages |
| 995 | ============================================ | 889 | ============================================ |
diff --git a/documentation/migration-guides/migration-1.6.rst b/documentation/migration-guides/migration-1.6.rst index 916169e836..b052a43a31 100644 --- a/documentation/migration-guides/migration-1.6.rst +++ b/documentation/migration-guides/migration-1.6.rst | |||
| @@ -221,7 +221,7 @@ Package Test (ptest) | |||
| 221 | 221 | ||
| 222 | Package Tests (ptest) are built but not installed by default. For | 222 | Package Tests (ptest) are built but not installed by default. For |
| 223 | information on using Package Tests, see the | 223 | information on using Package Tests, see the |
| 224 | ":ref:`dev-manual/packages:testing packages with ptest`" section in the | 224 | ":ref:`test-manual/ptest:testing packages with ptest`" section in the |
| 225 | Yocto Project Development Tasks Manual. See also the ":ref:`ref-classes-ptest`" | 225 | Yocto Project Development Tasks Manual. See also the ":ref:`ref-classes-ptest`" |
| 226 | section. | 226 | section. |
| 227 | 227 | ||
diff --git a/documentation/ref-manual/classes.rst b/documentation/ref-manual/classes.rst index 8c3cc0140f..f416a6d623 100644 --- a/documentation/ref-manual/classes.rst +++ b/documentation/ref-manual/classes.rst | |||
| @@ -2615,7 +2615,7 @@ runtime tests for recipes that build software that provides these tests. | |||
| 2615 | This class is intended to be inherited by individual recipes. However, | 2615 | This class is intended to be inherited by individual recipes. However, |
| 2616 | the class' functionality is largely disabled unless "ptest" appears in | 2616 | the class' functionality is largely disabled unless "ptest" appears in |
| 2617 | :term:`DISTRO_FEATURES`. See the | 2617 | :term:`DISTRO_FEATURES`. See the |
| 2618 | ":ref:`dev-manual/packages:testing packages with ptest`" | 2618 | ":ref:`test-manual/ptest:testing packages with ptest`" |
| 2619 | section in the Yocto Project Development Tasks Manual for more information | 2619 | section in the Yocto Project Development Tasks Manual for more information |
| 2620 | on ptest. | 2620 | on ptest. |
| 2621 | 2621 | ||
| @@ -2639,7 +2639,7 @@ Enables package tests (ptests) specifically for GNOME packages, which | |||
| 2639 | have tests intended to be executed with ``gnome-desktop-testing``. | 2639 | have tests intended to be executed with ``gnome-desktop-testing``. |
| 2640 | 2640 | ||
| 2641 | For information on setting up and running ptests, see the | 2641 | For information on setting up and running ptests, see the |
| 2642 | ":ref:`dev-manual/packages:testing packages with ptest`" | 2642 | ":ref:`test-manual/ptest:testing packages with ptest`" |
| 2643 | section in the Yocto Project Development Tasks Manual. | 2643 | section in the Yocto Project Development Tasks Manual. |
| 2644 | 2644 | ||
| 2645 | .. _ref-classes-python3-dir: | 2645 | .. _ref-classes-python3-dir: |
diff --git a/documentation/ref-manual/features.rst b/documentation/ref-manual/features.rst index b2ba731bb2..face6c1062 100644 --- a/documentation/ref-manual/features.rst +++ b/documentation/ref-manual/features.rst | |||
| @@ -207,7 +207,7 @@ metadata, as extra layers can define their own: | |||
| 207 | 207 | ||
| 208 | - *ptest:* Enables building the package tests where supported by | 208 | - *ptest:* Enables building the package tests where supported by |
| 209 | individual recipes. For more information on package tests, see the | 209 | individual recipes. For more information on package tests, see the |
| 210 | ":ref:`dev-manual/packages:testing packages with ptest`" section | 210 | ":ref:`test-manual/ptest:testing packages with ptest`" section |
| 211 | in the Yocto Project Development Tasks Manual. | 211 | in the Yocto Project Development Tasks Manual. |
| 212 | 212 | ||
| 213 | - *pulseaudio:* Include support for | 213 | - *pulseaudio:* Include support for |
diff --git a/documentation/ref-manual/qa-checks.rst b/documentation/ref-manual/qa-checks.rst index 1b83834df1..27d46de3fd 100644 --- a/documentation/ref-manual/qa-checks.rst +++ b/documentation/ref-manual/qa-checks.rst | |||
| @@ -791,7 +791,7 @@ Errors and Warnings | |||
| 791 | 791 | ||
| 792 | This check will detect if the source of the package contains some | 792 | This check will detect if the source of the package contains some |
| 793 | upstream-provided tests and, if so, that ptests are implemented for this | 793 | upstream-provided tests and, if so, that ptests are implemented for this |
| 794 | recipe. See the ":ref:`dev-manual/packages:testing packages with ptest`" | 794 | recipe. See the ":ref:`test-manual/ptest:testing packages with ptest`" |
| 795 | section in the Yocto Project Development Tasks Manual. See also the | 795 | section in the Yocto Project Development Tasks Manual. See also the |
| 796 | ":ref:`ref-classes-ptest`" section. | 796 | ":ref:`ref-classes-ptest`" section. |
| 797 | 797 | ||
diff --git a/documentation/ref-manual/release-process.rst b/documentation/ref-manual/release-process.rst index 691e7d3ed9..3383a4e1b0 100644 --- a/documentation/ref-manual/release-process.rst +++ b/documentation/ref-manual/release-process.rst | |||
| @@ -175,7 +175,7 @@ consists of the following pieces: | |||
| 175 | operation and functions. However, the test can also use the IP | 175 | operation and functions. However, the test can also use the IP |
| 176 | address of a machine to test. | 176 | address of a machine to test. |
| 177 | 177 | ||
| 178 | - :ref:`ptest <dev-manual/packages:testing packages with ptest>`: | 178 | - :ref:`ptest <test-manual/ptest:testing packages with ptest>`: |
| 179 | Runs tests against packages produced during the build for a given | 179 | Runs tests against packages produced during the build for a given |
| 180 | piece of software. The test allows the packages to be run within a | 180 | piece of software. The test allows the packages to be run within a |
| 181 | target image. | 181 | target image. |
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst index 869100ce34..200def084f 100644 --- a/documentation/ref-manual/variables.rst +++ b/documentation/ref-manual/variables.rst | |||
| @@ -6828,7 +6828,7 @@ system and gives an overview of their function and contents. | |||
| 6828 | 6828 | ||
| 6829 | :term:`PTEST_ENABLED` | 6829 | :term:`PTEST_ENABLED` |
| 6830 | Specifies whether or not :ref:`Package | 6830 | Specifies whether or not :ref:`Package |
| 6831 | Test <dev-manual/packages:testing packages with ptest>` (ptest) | 6831 | Test <test-manual/ptest:testing packages with ptest>` (ptest) |
| 6832 | functionality is enabled when building a recipe. You should not set | 6832 | functionality is enabled when building a recipe. You should not set |
| 6833 | this variable directly. Enabling and disabling building Package Tests | 6833 | this variable directly. Enabling and disabling building Package Tests |
| 6834 | at build time should be done by adding "ptest" to (or removing it | 6834 | at build time should be done by adding "ptest" to (or removing it |
diff --git a/documentation/test-manual/index.rst b/documentation/test-manual/index.rst index 86a2f436ea..ad71f37910 100644 --- a/documentation/test-manual/index.rst +++ b/documentation/test-manual/index.rst | |||
| @@ -12,6 +12,7 @@ Yocto Project Test Environment Manual | |||
| 12 | 12 | ||
| 13 | intro | 13 | intro |
| 14 | test-process | 14 | test-process |
| 15 | ptest | ||
| 15 | understand-autobuilder | 16 | understand-autobuilder |
| 16 | reproducible-builds | 17 | reproducible-builds |
| 17 | yocto-project-compatible | 18 | yocto-project-compatible |
diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst index c31fd11c7a..7bbd11295f 100644 --- a/documentation/test-manual/intro.rst +++ b/documentation/test-manual/intro.rst | |||
| @@ -140,7 +140,7 @@ the following types of tests: | |||
| 140 | - *Package Testing:* A Package Test (ptest) runs tests against packages | 140 | - *Package Testing:* A Package Test (ptest) runs tests against packages |
| 141 | built by the OpenEmbedded build system on the target machine. See the | 141 | built by the OpenEmbedded build system on the target machine. See the |
| 142 | :ref:`Testing Packages With | 142 | :ref:`Testing Packages With |
| 143 | ptest <dev-manual/packages:Testing Packages With ptest>` section | 143 | ptest <test-manual/ptest:Testing Packages With ptest>` section |
| 144 | in the Yocto Project Development Tasks Manual and the | 144 | in the Yocto Project Development Tasks Manual and the |
| 145 | ":yocto_wiki:`Ptest </Ptest>`" Wiki page for more | 145 | ":yocto_wiki:`Ptest </Ptest>`" Wiki page for more |
| 146 | information on Ptest. | 146 | information on Ptest. |
diff --git a/documentation/test-manual/ptest.rst b/documentation/test-manual/ptest.rst new file mode 100644 index 0000000000..dea1bad23b --- /dev/null +++ b/documentation/test-manual/ptest.rst | |||
| @@ -0,0 +1,114 @@ | |||
| 1 | .. SPDX-License-Identifier: CC-BY-SA-2.0-UK | ||
| 2 | |||
| 3 | *************************** | ||
| 4 | Testing Packages With ptest | ||
| 5 | *************************** | ||
| 6 | |||
| 7 | A Package Test (ptest) runs tests against packages built by the | ||
| 8 | OpenEmbedded build system on the target machine. A ptest contains at | ||
| 9 | least two items: the actual test, and a shell script (``run-ptest``) | ||
| 10 | that starts the test. The shell script that starts the test must not | ||
| 11 | contain the actual test --- the script only starts the test. On the other | ||
| 12 | hand, the test can be anything from a simple shell script that runs a | ||
| 13 | binary and checks the output to an elaborate system of test binaries and | ||
| 14 | data files. | ||
| 15 | |||
| 16 | The test generates output in the format used by Automake:: | ||
| 17 | |||
| 18 | result: testname | ||
| 19 | |||
| 20 | where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and | ||
| 21 | the testname can be any identifying string. | ||
| 22 | |||
| 23 | For a list of Yocto Project recipes that are already enabled with ptest, | ||
| 24 | see the :yocto_wiki:`Ptest </Ptest>` wiki page. | ||
| 25 | |||
| 26 | .. note:: | ||
| 27 | |||
| 28 | A recipe is "ptest-enabled" if it inherits the :ref:`ref-classes-ptest` | ||
| 29 | class. | ||
| 30 | |||
| 31 | Adding ptest to Your Build | ||
| 32 | ========================== | ||
| 33 | |||
| 34 | To add package testing to your build, add the :term:`DISTRO_FEATURES` and | ||
| 35 | :term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which | ||
| 36 | is found in the :term:`Build Directory`:: | ||
| 37 | |||
| 38 | DISTRO_FEATURES:append = " ptest" | ||
| 39 | EXTRA_IMAGE_FEATURES += "ptest-pkgs" | ||
| 40 | |||
| 41 | Once your build is complete, the ptest files are installed into the | ||
| 42 | ``/usr/lib/package/ptest`` directory within the image, where ``package`` | ||
| 43 | is the name of the package. | ||
| 44 | |||
| 45 | Running ptest | ||
| 46 | ============= | ||
| 47 | |||
| 48 | The ``ptest-runner`` package installs a shell script that loops through | ||
| 49 | all installed ptest test suites and runs them in sequence. Consequently, | ||
| 50 | you might want to add this package to your image. | ||
| 51 | |||
| 52 | Getting Your Package Ready | ||
| 53 | ========================== | ||
| 54 | |||
| 55 | In order to enable a recipe to run installed ptests on target hardware, | ||
| 56 | you need to prepare the recipes that build the packages you want to | ||
| 57 | test. Here is what you have to do for each recipe: | ||
| 58 | |||
| 59 | - *Be sure the recipe inherits the* :ref:`ref-classes-ptest` *class:* | ||
| 60 | Include the following line in each recipe:: | ||
| 61 | |||
| 62 | inherit ptest | ||
| 63 | |||
| 64 | - *Create run-ptest:* This script starts your test. Locate the | ||
| 65 | script where you will refer to it using | ||
| 66 | :term:`SRC_URI`. Here is an | ||
| 67 | example that starts a test for ``dbus``:: | ||
| 68 | |||
| 69 | #!/bin/sh | ||
| 70 | cd test | ||
| 71 | make -k runtest-TESTS | ||
| 72 | |||
| 73 | - *Ensure dependencies are met:* If the test adds build or runtime | ||
| 74 | dependencies that normally do not exist for the package (such as | ||
| 75 | requiring "make" to run the test suite), use the | ||
| 76 | :term:`DEPENDS` and | ||
| 77 | :term:`RDEPENDS` variables in | ||
| 78 | your recipe in order for the package to meet the dependencies. Here | ||
| 79 | is an example where the package has a runtime dependency on "make":: | ||
| 80 | |||
| 81 | RDEPENDS:${PN}-ptest += "make" | ||
| 82 | |||
| 83 | - *Add a function to build the test suite:* Not many packages support | ||
| 84 | cross-compilation of their test suites. Consequently, you usually | ||
| 85 | need to add a cross-compilation function to the package. | ||
| 86 | |||
| 87 | Many packages based on Automake compile and run the test suite by | ||
| 88 | using a single command such as ``make check``. However, the host | ||
| 89 | ``make check`` builds and runs on the same computer, while | ||
| 90 | cross-compiling requires that the package is built on the host but | ||
| 91 | executed for the target architecture (though often, as in the case | ||
| 92 | for ptest, the execution occurs on the host). The built version of | ||
| 93 | Automake that ships with the Yocto Project includes a patch that | ||
| 94 | separates building and execution. Consequently, packages that use the | ||
| 95 | unaltered, patched version of ``make check`` automatically | ||
| 96 | cross-compiles. | ||
| 97 | |||
| 98 | Regardless, you still must add a ``do_compile_ptest`` function to | ||
| 99 | build the test suite. Add a function similar to the following to your | ||
| 100 | recipe:: | ||
| 101 | |||
| 102 | do_compile_ptest() { | ||
| 103 | oe_runmake buildtest-TESTS | ||
| 104 | } | ||
| 105 | |||
| 106 | - *Ensure special configurations are set:* If the package requires | ||
| 107 | special configurations prior to compiling the test code, you must | ||
| 108 | insert a ``do_configure_ptest`` function into the recipe. | ||
| 109 | |||
| 110 | - *Install the test suite:* The :ref:`ref-classes-ptest` class | ||
| 111 | automatically copies the file ``run-ptest`` to the target and then runs make | ||
| 112 | ``install-ptest`` to run the tests. If this is not enough, you need | ||
| 113 | to create a ``do_install_ptest`` function and make sure it gets | ||
| 114 | called after the "make install-ptest" completes. | ||
