diff options
author | Mark Morton <Mark.Morton@windriver.com> | 2020-09-14 16:11:31 -0700 |
---|---|---|
committer | Richard Purdie <richard.purdie@linuxfoundation.org> | 2020-09-17 10:09:35 +0100 |
commit | 74f212d2e1db87ccfecaf9385ab280123d5fb168 (patch) | |
tree | 79e85fac733b78eb11cb19ca9753ed44dbf8358e /documentation/test-manual/test-manual-intro.rst | |
parent | fc876832cb57a1a3e61150778a9cfd7acffbc2b6 (diff) | |
download | poky-74f212d2e1db87ccfecaf9385ab280123d5fb168.tar.gz |
sphinx: test-manual code block, link, and format update
(From yocto-docs rev: 928c212ec4ad6e09524fdf8147aa7daf244677b0)
Signed-off-by: Mark Morton <mark.morton@windriver.com>
Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/test-manual/test-manual-intro.rst')
-rw-r--r-- | documentation/test-manual/test-manual-intro.rst | 287 |
1 files changed, 173 insertions, 114 deletions
diff --git a/documentation/test-manual/test-manual-intro.rst b/documentation/test-manual/test-manual-intro.rst index 2f11e2beb3..b0bc24c9b0 100644 --- a/documentation/test-manual/test-manual-intro.rst +++ b/documentation/test-manual/test-manual-intro.rst | |||
@@ -23,26 +23,25 @@ project core. | |||
23 | 23 | ||
24 | Currently, the Yocto Project Test Environment Manual has no projected | 24 | Currently, the Yocto Project Test Environment Manual has no projected |
25 | release date. This manual is a work-in-progress and is being initially | 25 | release date. This manual is a work-in-progress and is being initially |
26 | loaded with information from the `README <>`__ files and notes from key | 26 | loaded with information from the README files and notes from key |
27 | engineers: | 27 | engineers: |
28 | 28 | ||
29 | - *``yocto-autobuilder2``:* This | 29 | - *yocto-autobuilder2:* This |
30 | ```README.md`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder2/tree/README.md | 30 | :yocto_git:`README.md </cgit.cgi/yocto-autobuilder2/tree/README.md>` |
31 | is the main README which detials how to set up the Yocto Project | 31 | is the main README which detials how to set up the Yocto Project |
32 | Autobuilder. The ``yocto-autobuilder2`` repository represents the | 32 | Autobuilder. The ``yocto-autobuilder2`` repository represents the |
33 | Yocto Project's console UI plugin to Buildbot and the configuration | 33 | Yocto Project's console UI plugin to Buildbot and the configuration |
34 | necessary to configure Buildbot to perform the testing the project | 34 | necessary to configure Buildbot to perform the testing the project |
35 | requires. | 35 | requires. |
36 | 36 | ||
37 | - *``yocto-autobuilder-helper``:* This | 37 | - *yocto-autobuilder-helper:* This :yocto_git:`README </cgit.cgi/yocto-autobuilder-helper/tree/README/>` |
38 | ```README`http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder-helper/tree/README | ||
39 | and repository contains Yocto Project Autobuilder Helper scripts and | 38 | and repository contains Yocto Project Autobuilder Helper scripts and |
40 | configuration. The ``yocto-autobuilder-helper`` repository contains | 39 | configuration. The ``yocto-autobuilder-helper`` repository contains |
41 | the "glue" logic that defines which tests to run and how to run them. | 40 | the "glue" logic that defines which tests to run and how to run them. |
42 | As a result, it can be used by any Continuous Improvement (CI) system | 41 | As a result, it can be used by any Continuous Improvement (CI) system |
43 | to run builds, support getting the correct code revisions, configure | 42 | to run builds, support getting the correct code revisions, configure |
44 | builds and layers, run builds, and collect results. The code is | 43 | builds and layers, run builds, and collect results. The code is |
45 | independent of any CI system, which means the code can work Buildbot, | 44 | independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__, |
46 | Jenkins, or others. This repository has a branch per release of the | 45 | Jenkins, or others. This repository has a branch per release of the |
47 | project defining the tests to run on a per release basis. | 46 | project defining the tests to run on a per release basis. |
48 | 47 | ||
@@ -124,21 +123,22 @@ thefollowing types of tests: | |||
124 | ensure we have no single point of failure and can ensure the | 123 | ensure we have no single point of failure and can ensure the |
125 | different distros work effectively. | 124 | different distros work effectively. |
126 | 125 | ||
127 | - *eSDK Testing:* Image tests initiated through the following command: | 126 | - *eSDK Testing:* Image tests initiated through the following command:: |
128 | $ bitbake image -c testsdkext The tests utilize the ``testsdkext`` | 127 | |
129 | class and the ``do_testsdkext`` task. | 128 | $ bitbake image -c testsdkext |
129 | |||
130 | The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task. | ||
130 | 131 | ||
131 | - *Feature Testing:* Various scenario-based tests are run through the | 132 | - *Feature Testing:* Various scenario-based tests are run through the |
132 | `OpenEmbedded | 133 | :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions |
133 | Self-Test <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__ | ||
134 | (oe-selftest). We test oe-selftest on each of the main distrubutions | ||
135 | we support. | 134 | we support. |
136 | 135 | ||
137 | - *Image Testing:* Image tests initiated through the following command: | 136 | - *Image Testing:* Image tests initiated through the following command:: |
138 | $ bitbake image -c testimage The tests utilize the | 137 | |
139 | :ref:`testimage* <ref-classes-testimage*>` | 138 | $ bitbake image -c testimage |
140 | classes and the | 139 | |
141 | :ref:`ref-tasks-testimage` task. | 140 | The tests utilize the :ref:`testimage* <ref-classes-testimage*>` |
141 | classes and the :ref:`ref-tasks-testimage` task. | ||
142 | 142 | ||
143 | - *Layer Testing:* The Autobuilder has the possibility to test whether | 143 | - *Layer Testing:* The Autobuilder has the possibility to test whether |
144 | specific layers work with the test of the system. The layers tested | 144 | specific layers work with the test of the system. The layers tested |
@@ -147,20 +147,22 @@ thefollowing types of tests: | |||
147 | 147 | ||
148 | - *Package Testing:* A Package Test (ptest) runs tests against packages | 148 | - *Package Testing:* A Package Test (ptest) runs tests against packages |
149 | built by the OpenEmbedded build system on the target machine. See the | 149 | built by the OpenEmbedded build system on the target machine. See the |
150 | "`Testing Packages With | 150 | :ref:`Testing Packages With |
151 | ptest <&YOCTO_DOCS_DEV_URL;#testing-packages-with-ptest>`__" section | 151 | ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section |
152 | in the Yocto Project Development Tasks Manual and the | 152 | in the Yocto Project Development Tasks Manual and the |
153 | ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more | 153 | ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more |
154 | information on Ptest. | 154 | information on Ptest. |
155 | 155 | ||
156 | - *SDK Testing:* Image tests initiated through the following command: $ | 156 | - *SDK Testing:* Image tests initiated through the following command:: |
157 | bitbake image -c testsdk The tests utilize the | 157 | |
158 | :ref:`testsdk <ref-classes-testsdk>` class and | 158 | $ bitbake image -c testsdk |
159 | |||
160 | The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and | ||
159 | the ``do_testsdk`` task. | 161 | the ``do_testsdk`` task. |
160 | 162 | ||
161 | - *Unit Testing:* Unit tests on various components of the system run | 163 | - *Unit Testing:* Unit tests on various components of the system run |
162 | through ``oe-selftest`` and | 164 | through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and |
163 | ```bitbake-selftest`` <&YOCTO_DOCS_REF_URL;#testing-and-quality-assurance>`__. | 165 | :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`. |
164 | 166 | ||
165 | - *Automatic Upgrade Helper:* This target tests whether new versions of | 167 | - *Automatic Upgrade Helper:* This target tests whether new versions of |
166 | software are available and whether we can automatically upgrade to | 168 | software are available and whether we can automatically upgrade to |
@@ -174,36 +176,43 @@ How Tests Map to Areas of Code | |||
174 | 176 | ||
175 | Tests map into the codebase as follows: | 177 | Tests map into the codebase as follows: |
176 | 178 | ||
177 | - *bitbake-selftest*: | 179 | - *bitbake-selftest:* |
178 | 180 | ||
179 | These tests are self-contained and test BitBake as well as its APIs, | 181 | These tests are self-contained and test BitBake as well as its APIs, |
180 | which include the fetchers. The tests are located in | 182 | which include the fetchers. The tests are located in |
181 | ``bitbake/lib/*/tests``. | 183 | ``bitbake/lib/*/tests``. |
182 | 184 | ||
183 | From within the BitBake repository, run the following: $ | 185 | From within the BitBake repository, run the following:: |
184 | bitbake-selftest | 186 | |
187 | $ bitbake-selftest | ||
185 | 188 | ||
186 | To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` | 189 | To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` |
187 | variable when running "bitbake-selftest" as follows: $ | 190 | variable when running "bitbake-selftest" as follows:: |
188 | BB_SKIP_NETTEST=yes bitbake-selftest | 191 | |
192 | $ BB_SKIP_NETTEST=yes bitbake-selftest | ||
189 | 193 | ||
190 | The default output is quiet and just prints a summary of what was | 194 | The default output is quiet and just prints a summary of what was |
191 | run. To see more information, there is a verbose option:$ | 195 | run. To see more information, there is a verbose option:: |
192 | bitbake-selftest -v | 196 | |
197 | $ bitbake-selftest -v | ||
193 | 198 | ||
194 | Use this option when you wish to skip tests that access the network, | 199 | Use this option when you wish to skip tests that access the network, |
195 | which are mostly necessary to test the fetcher modules. To specify | 200 | which are mostly necessary to test the fetcher modules. To specify |
196 | individual test modules to run, append the test module name to the | 201 | individual test modules to run, append the test module name to the |
197 | "bitbake-selftest" command. For example, to specify the tests for the | 202 | "bitbake-selftest" command. For example, to specify the tests for the |
198 | bb.data.module, run: $ bitbake-selftest bb.test.data.moduleYou can | 203 | bb.data.module, run:: |
199 | also specify individual tests by defining the full name and module | 204 | |
200 | plus the class path of the test, for example: $ bitbake-selftest | 205 | $ bitbake-selftest bb.test.data.module |
201 | bb.tests.data.TestOverrides.test_one_override | 206 | |
207 | You can also specify individual tests by defining the full name and module | ||
208 | plus the class path of the test, for example:: | ||
209 | |||
210 | $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override | ||
202 | 211 | ||
203 | The tests are based on `Python | 212 | The tests are based on `Python |
204 | unittest <https://docs.python.org/3/library/unittest.html>`__. | 213 | unittest <https://docs.python.org/3/library/unittest.html>`__. |
205 | 214 | ||
206 | - *oe-selftest*: | 215 | - *oe-selftest:* |
207 | 216 | ||
208 | - These tests use OE to test the workflows, which include testing | 217 | - These tests use OE to test the workflows, which include testing |
209 | specific features, behaviors of tasks, and API unit tests. | 218 | specific features, behaviors of tasks, and API unit tests. |
@@ -219,31 +228,40 @@ Tests map into the codebase as follows: | |||
219 | - The code for the tests resides in | 228 | - The code for the tests resides in |
220 | ``meta/lib/oeqa/selftest/cases/``. | 229 | ``meta/lib/oeqa/selftest/cases/``. |
221 | 230 | ||
222 | - To run all the tests, enter the following command: $ oe-selftest | 231 | - To run all the tests, enter the following command:: |
223 | -a | 232 | |
233 | $ oe-selftest -a | ||
224 | 234 | ||
225 | - To run a specific test, use the following command form where | 235 | - To run a specific test, use the following command form where |
226 | testname is the name of the specific test: $ oe-selftest -r | 236 | testname is the name of the specific test:: |
227 | testname For example, the following command would run the tinfoil | 237 | |
228 | getVar API test:$ oe-selftest -r | 238 | $ oe-selftest -r <testname> |
229 | tinfoil.TinfoilTests.test_getvarIt is also possible to run a set | 239 | |
240 | For example, the following command would run the tinfoil | ||
241 | getVar API test:: | ||
242 | |||
243 | $ oe-selftest -r tinfoil.TinfoilTests.test_getvar | ||
244 | |||
245 | It is also possible to run a set | ||
230 | of tests. For example the following command will run all of the | 246 | of tests. For example the following command will run all of the |
231 | tinfoil tests:$ oe-selftest -r tinfoil | 247 | tinfoil tests:: |
248 | |||
249 | $ oe-selftest -r tinfoil | ||
232 | 250 | ||
233 | - *testimage:* | 251 | - *testimage:* |
234 | 252 | ||
235 | - These tests build an image, boot it, and run tests against the | 253 | - These tests build an image, boot it, and run tests against the |
236 | image's content. | 254 | image's content. |
237 | 255 | ||
238 | - The code for these tests resides in | 256 | - The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``. |
239 | ``meta/lib/oeqa/runtime/cases/``. | 257 | |
258 | - You need to set the :term:`IMAGE_CLASSES` variable as follows:: | ||
240 | 259 | ||
241 | - You need to set the | 260 | IMAGE_CLASSES += "testimage" |
242 | :term:`IMAGE_CLASSES` | ||
243 | variable as follows: IMAGE_CLASSES += "testimage" | ||
244 | 261 | ||
245 | - Run the tests using the following command form: $ bitbake image -c | 262 | - Run the tests using the following command form:: |
246 | testimage | 263 | |
264 | $ bitbake image -c testimage | ||
247 | 265 | ||
248 | - *testsdk:* | 266 | - *testsdk:* |
249 | 267 | ||
@@ -252,8 +270,9 @@ Tests map into the codebase as follows: | |||
252 | 270 | ||
253 | - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``. | 271 | - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``. |
254 | 272 | ||
255 | - Run the test using the following command form: $ bitbake image -c | 273 | - Run the test using the following command form:: |
256 | testsdk | 274 | |
275 | $ bitbake image -c testsdk | ||
257 | 276 | ||
258 | - *testsdk_ext:* | 277 | - *testsdk_ext:* |
259 | 278 | ||
@@ -262,8 +281,9 @@ Tests map into the codebase as follows: | |||
262 | 281 | ||
263 | - The code for these tests resides in ``meta/lib/oeqa/esdk``. | 282 | - The code for these tests resides in ``meta/lib/oeqa/esdk``. |
264 | 283 | ||
265 | - To run the tests, use the following command form: $ bitbake image | 284 | - To run the tests, use the following command form:: |
266 | -c testsdkext | 285 | |
286 | $ bitbake image -c testsdkext | ||
267 | 287 | ||
268 | - *oe-build-perf-test:* | 288 | - *oe-build-perf-test:* |
269 | 289 | ||
@@ -272,8 +292,11 @@ Tests map into the codebase as follows: | |||
272 | 292 | ||
273 | - The code for these tests resides in ``meta/lib/oeqa/buildperf``. | 293 | - The code for these tests resides in ``meta/lib/oeqa/buildperf``. |
274 | 294 | ||
275 | - To run the tests, use the following command form: $ | 295 | - To run the tests, use the following command form:: |
276 | oe-build-perf-test optionsThe command takes a number of options, | 296 | |
297 | $ oe-build-perf-test <options> | ||
298 | |||
299 | The command takes a number of options, | ||
277 | such as where to place the test results. The Autobuilder Helper | 300 | such as where to place the test results. The Autobuilder Helper |
278 | Scripts include the ``build-perf-test-wrapper`` script with | 301 | Scripts include the ``build-perf-test-wrapper`` script with |
279 | examples of how to use the oe-build-perf-test from the command | 302 | examples of how to use the oe-build-perf-test from the command |
@@ -285,9 +308,9 @@ Tests map into the codebase as follows: | |||
285 | Use the ``oe-build-perf-report`` command to generate text reports | 308 | Use the ``oe-build-perf-report`` command to generate text reports |
286 | and HTML reports with graphs of the performance data. For | 309 | and HTML reports with graphs of the performance data. For |
287 | examples, see | 310 | examples, see |
288 | `http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html <#>`__ | 311 | :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html` |
289 | and | 312 | and |
290 | `http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt <#>`__. | 313 | :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`. |
291 | 314 | ||
292 | - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. | 315 | - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. |
293 | 316 | ||
@@ -295,7 +318,7 @@ Test Examples | |||
295 | ============= | 318 | ============= |
296 | 319 | ||
297 | This section provides example tests for each of the tests listed in the | 320 | This section provides example tests for each of the tests listed in the |
298 | `How Tests Map to Areas of Code <#test-test-mapping>`__ section. | 321 | :ref:`test-manual/test-manual-intro:How Tests Map to Areas of Code` section. |
299 | 322 | ||
300 | For oeqa tests, testcases for each area reside in the main test | 323 | For oeqa tests, testcases for each area reside in the main test |
301 | directory at ``meta/lib/oeqa/selftest/cases`` directory. | 324 | directory at ``meta/lib/oeqa/selftest/cases`` directory. |
@@ -308,14 +331,20 @@ directory. | |||
308 | ``bitbake-selftest`` | 331 | ``bitbake-selftest`` |
309 | -------------------- | 332 | -------------------- |
310 | 333 | ||
311 | A simple test example from ``lib/bb/tests/data.py`` is: class | 334 | A simple test example from ``lib/bb/tests/data.py`` is:: |
312 | DataExpansions(unittest.TestCase): def setUp(self): self.d = | 335 | |
313 | bb.data.init() self.d["foo"] = "value_of_foo" self.d["bar"] = | 336 | class DataExpansions(unittest.TestCase): |
314 | "value_of_bar" self.d["value_of_foo"] = "value_of_'value_of_foo'" def | 337 | def setUp(self): |
315 | test_one_var(self): val = self.d.expand("${foo}") | 338 | self.d = bb.data.init() |
316 | self.assertEqual(str(val), "value_of_foo") | 339 | self.d["foo"] = "value_of_foo" |
340 | self.d["bar"] = "value_of_bar" | ||
341 | self.d["value_of_foo"] = "value_of_'value_of_foo'" | ||
317 | 342 | ||
318 | In this example, a ```DataExpansions`` <>`__ class of tests is created, | 343 | def test_one_var(self): |
344 | val = self.d.expand("${foo}") | ||
345 | self.assertEqual(str(val), "value_of_foo") | ||
346 | |||
347 | In this example, a ``DataExpansions`` class of tests is created, | ||
319 | derived from standard python unittest. The class has a common ``setUp`` | 348 | derived from standard python unittest. The class has a common ``setUp`` |
320 | function which is shared by all the tests in the class. A simple test is | 349 | function which is shared by all the tests in the class. A simple test is |
321 | then added to test that when a variable is expanded, the correct value | 350 | then added to test that when a variable is expanded, the correct value |
@@ -323,7 +352,7 @@ is found. | |||
323 | 352 | ||
324 | Bitbake selftests are straightforward python unittest. Refer to the | 353 | Bitbake selftests are straightforward python unittest. Refer to the |
325 | Python unittest documentation for additional information on writing | 354 | Python unittest documentation for additional information on writing |
326 | these tests at: `https://docs.python.org/3/library/unittest.html <#>`__. | 355 | these tests at: https://docs.python.org/3/library/unittest.html. |
327 | 356 | ||
328 | .. _oe-selftest-example: | 357 | .. _oe-selftest-example: |
329 | 358 | ||
@@ -334,14 +363,15 @@ These tests are more complex due to the setup required behind the scenes | |||
334 | for full builds. Rather than directly using Python's unittest, the code | 363 | for full builds. Rather than directly using Python's unittest, the code |
335 | wraps most of the standard objects. The tests can be simple, such as | 364 | wraps most of the standard objects. The tests can be simple, such as |
336 | testing a command from within the OE build environment using the | 365 | testing a command from within the OE build environment using the |
337 | following example:class BitbakeLayers(OESelftestTestCase): def | 366 | following example:: |
338 | test_bitbakelayers_showcrossdepends(self): result = | 367 | |
339 | runCmd('bitbake-layers show-cross-depends') self.assertTrue('aspell' in | 368 | class BitbakeLayers(OESelftestTestCase): |
340 | result.output, msg = "No dependencies were shown. bitbake-layers | 369 | def test_bitbakelayers_showcrossdepends(self): |
341 | show-cross-depends output: %s"% result.output) | 370 | result = runCmd('bitbake-layers show-cross-depends') |
371 | self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output) | ||
342 | 372 | ||
343 | This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``, | 373 | This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``, |
344 | creates a testcase from the ```OESelftestTestCase`` <>`__ class, derived | 374 | creates a testcase from the ``OESelftestTestCase`` class, derived |
345 | from ``unittest.TestCase``, which runs the ``bitbake-layers`` command | 375 | from ``unittest.TestCase``, which runs the ``bitbake-layers`` command |
346 | and checks the output to ensure it contains something we know should be | 376 | and checks the output to ensure it contains something we know should be |
347 | here. | 377 | here. |
@@ -367,7 +397,7 @@ so tests within a given test class should always run in the same build, | |||
367 | while tests in different classes or modules may be split into different | 397 | while tests in different classes or modules may be split into different |
368 | builds. There is no data store available for these tests since the tests | 398 | builds. There is no data store available for these tests since the tests |
369 | launch the ``bitbake`` command and exist outside of its context. As a | 399 | launch the ``bitbake`` command and exist outside of its context. As a |
370 | result, common bitbake library functions (bb.*) are also unavailable. | 400 | result, common bitbake library functions (bb.\*) are also unavailable. |
371 | 401 | ||
372 | .. _testimage-example: | 402 | .. _testimage-example: |
373 | 403 | ||
@@ -378,14 +408,18 @@ These tests are run once an image is up and running, either on target | |||
378 | hardware or under QEMU. As a result, they are assumed to be running in a | 408 | hardware or under QEMU. As a result, they are assumed to be running in a |
379 | target image environment, as opposed to a host build environment. A | 409 | target image environment, as opposed to a host build environment. A |
380 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains | 410 | simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains |
381 | the following:class PythonTest(OERuntimeTestCase): | 411 | the following:: |
382 | @OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['python3-core']) | 412 | |
383 | def test_python3(self): cmd = "python3 -c \\"import codecs; | 413 | class PythonTest(OERuntimeTestCase): |
384 | print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" status, output = | 414 | @OETestDepends(['ssh.SSHTest.test_ssh']) |
385 | self.target.run(cmd) msg = 'Exit status was not 0. Output: %s' % output | 415 | @OEHasPackage(['python3-core']) |
386 | self.assertEqual(status, 0, msg=msg) | 416 | def test_python3(self): |
387 | 417 | cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" | |
388 | In this example, the ```OERuntimeTestCase`` <>`__ class wraps | 418 | status, output = self.target.run(cmd) |
419 | msg = 'Exit status was not 0. Output: %s' % output | ||
420 | self.assertEqual(status, 0, msg=msg) | ||
421 | |||
422 | In this example, the ``OERuntimeTestCase`` class wraps | ||
389 | ``unittest.TestCase``. Within the test, ``self.target`` represents the | 423 | ``unittest.TestCase``. Within the test, ``self.target`` represents the |
390 | target system, where commands can be run on it using the ``run()`` | 424 | target system, where commands can be run on it using the ``run()`` |
391 | method. | 425 | method. |
@@ -402,19 +436,30 @@ the image. | |||
402 | 436 | ||
403 | These tests are run against built extensible SDKs (eSDKs). The tests can | 437 | These tests are run against built extensible SDKs (eSDKs). The tests can |
404 | assume that the eSDK environment has already been setup. An example from | 438 | assume that the eSDK environment has already been setup. An example from |
405 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:class | 439 | ``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: |
406 | DevtoolTest(OESDKExtTestCase): @classmethod def setUpClass(cls): | 440 | |
407 | myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") cls.myapp_dst = | 441 | class DevtoolTest(OESDKExtTestCase): |
408 | os.path.join(cls.tc.sdk_dir, "myapp") shutil.copytree(myapp_src, | 442 | @classmethod def setUpClass(cls): |
409 | cls.myapp_dst) subprocess.check_output(['git', 'init', '.'], | 443 | myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") |
410 | cwd=cls.myapp_dst) subprocess.check_output(['git', 'add', '.'], | 444 | cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") |
411 | cwd=cls.myapp_dst) subprocess.check_output(['git', 'commit', '-m', | 445 | shutil.copytree(myapp_src, cls.myapp_dst) |
412 | "'test commit'"], cwd=cls.myapp_dst) @classmethod def | 446 | subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) |
413 | tearDownClass(cls): shutil.rmtree(cls.myapp_dst) def | 447 | subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) |
414 | \_test_devtool_build(self, directory): self._run('devtool add myapp %s' | 448 | subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) |
415 | % directory) try: self._run('devtool build myapp') finally: | 449 | |
416 | self._run('devtool reset myapp') def test_devtool_build_make(self): | 450 | @classmethod |
417 | self._test_devtool_build(self.myapp_dst)In this example, the ``devtool`` | 451 | def tearDownClass(cls): |
452 | shutil.rmtree(cls.myapp_dst) | ||
453 | def _test_devtool_build(self, directory): | ||
454 | self._run('devtool add myapp %s' % directory) | ||
455 | try: | ||
456 | self._run('devtool build myapp') | ||
457 | finally: | ||
458 | self._run('devtool reset myapp') | ||
459 | def test_devtool_build_make(self): | ||
460 | self._test_devtool_build(self.myapp_dst) | ||
461 | |||
462 | In this example, the ``devtool`` | ||
418 | command is tested to see whether a sample application can be built with | 463 | command is tested to see whether a sample application can be built with |
419 | the ``devtool build`` command within the eSDK. | 464 | the ``devtool build`` command within the eSDK. |
420 | 465 | ||
@@ -426,14 +471,20 @@ the ``devtool build`` command within the eSDK. | |||
426 | These tests are run against built SDKs. The tests can assume that an SDK | 471 | These tests are run against built SDKs. The tests can assume that an SDK |
427 | has already been extracted and its environment file has been sourced. A | 472 | has already been extracted and its environment file has been sourced. A |
428 | simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the | 473 | simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the |
429 | following:class Python3Test(OESDKTestCase): def setUp(self): if not | 474 | following:: |
430 | (self.tc.hasHostPackage("nativesdk-python3-core") or | 475 | |
431 | self.tc.hasHostPackage("python3-core-native")): raise | 476 | class Python3Test(OESDKTestCase): |
432 | unittest.SkipTest("No python3 package in the SDK") def | 477 | def setUp(self): |
433 | test_python3(self): cmd = "python3 -c \\"import codecs; | 478 | if not (self.tc.hasHostPackage("nativesdk-python3-core") or |
434 | print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" output = self._run(cmd) | 479 | self.tc.hasHostPackage("python3-core-native")): |
435 | self.assertEqual(output, "Hello, world\n")In this example, if | 480 | raise unittest.SkipTest("No python3 package in the SDK") |
436 | nativesdk-python3-core has been installed into the SDK, the code runs | 481 | |
482 | def test_python3(self): | ||
483 | cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" | ||
484 | output = self._run(cmd) | ||
485 | self.assertEqual(output, "Hello, world\n") | ||
486 | |||
487 | In this example, if nativesdk-python3-core has been installed into the SDK, the code runs | ||
437 | the python3 interpreter with a basic command to check it is working | 488 | the python3 interpreter with a basic command to check it is working |
438 | correctly. The test would only run if python3 is installed in the SDK. | 489 | correctly. The test would only run if python3 is installed in the SDK. |
439 | 490 | ||
@@ -444,17 +495,25 @@ correctly. The test would only run if python3 is installed in the SDK. | |||
444 | 495 | ||
445 | The performance tests usually measure how long operations take and the | 496 | The performance tests usually measure how long operations take and the |
446 | resource utilisation as that happens. An example from | 497 | resource utilisation as that happens. An example from |
447 | ``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:class | 498 | ``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:: |
448 | Test3(BuildPerfTestCase): def test3(self): """Bitbake parsing (bitbake | 499 | |
449 | -p)""" # Drop all caches and parse self.rm_cache() | 500 | class Test3(BuildPerfTestCase): |
450 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) | 501 | def test3(self): |
451 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', 'bitbake -p (no | 502 | """Bitbake parsing (bitbake -p)""" |
452 | caches)') # Drop tmp/cache | 503 | # Drop all caches and parse |
453 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) | 504 | self.rm_cache() |
454 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', 'bitbake -p (no | 505 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) |
455 | tmp/cache)') # Parse with fully cached data | 506 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', |
456 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', 'bitbake -p | 507 | 'bitbake -p (no caches)') |
457 | (cached)')This example shows how three specific parsing timings are | 508 | # Drop tmp/cache |
509 | oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) | ||
510 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', | ||
511 | 'bitbake -p (no tmp/cache)') | ||
512 | # Parse with fully cached data | ||
513 | self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', | ||
514 | 'bitbake -p (cached)') | ||
515 | |||
516 | This example shows how three specific parsing timings are | ||
458 | measured, with and without various caches, to show how BitBake’s parsing | 517 | measured, with and without various caches, to show how BitBake’s parsing |
459 | performance trends over time. | 518 | performance trends over time. |
460 | 519 | ||