diff options
Diffstat (limited to 'documentation/dev-manual/runtime-testing.rst')
-rw-r--r-- | documentation/dev-manual/runtime-testing.rst | 599 |
1 files changed, 599 insertions, 0 deletions
diff --git a/documentation/dev-manual/runtime-testing.rst b/documentation/dev-manual/runtime-testing.rst new file mode 100644 index 0000000000..77e1e6545f --- /dev/null +++ b/documentation/dev-manual/runtime-testing.rst | |||
@@ -0,0 +1,599 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-SA-2.0-UK | ||
2 | |||
3 | Performing Automated Runtime Testing | ||
4 | ************************************ | ||
5 | |||
6 | The OpenEmbedded build system makes available a series of automated | ||
7 | tests for images to verify runtime functionality. You can run these | ||
8 | tests on either QEMU or actual target hardware. Tests are written in | ||
9 | Python making use of the ``unittest`` module, and the majority of them | ||
10 | run commands on the target system over SSH. This section describes how | ||
11 | you set up the environment to use these tests, run available tests, and | ||
12 | write and add your own tests. | ||
13 | |||
14 | For information on the test and QA infrastructure available within the | ||
15 | Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`" | ||
16 | section in the Yocto Project Reference Manual. | ||
17 | |||
18 | Enabling Tests | ||
19 | ============== | ||
20 | |||
21 | Depending on whether you are planning to run tests using QEMU or on the | ||
22 | hardware, you have to take different steps to enable the tests. See the | ||
23 | following subsections for information on how to enable both types of | ||
24 | tests. | ||
25 | |||
26 | Enabling Runtime Tests on QEMU | ||
27 | ------------------------------ | ||
28 | |||
29 | In order to run tests, you need to do the following: | ||
30 | |||
31 | - *Set up to avoid interaction with sudo for networking:* To | ||
32 | accomplish this, you must do one of the following: | ||
33 | |||
34 | - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all | ||
35 | commands or just for ``runqemu-ifup``. You must provide the full | ||
36 | path as that can change if you are using multiple clones of the | ||
37 | source repository. | ||
38 | |||
39 | .. note:: | ||
40 | |||
41 | On some distributions, you also need to comment out "Defaults | ||
42 | requiretty" in ``/etc/sudoers``. | ||
43 | |||
44 | - Manually configure a tap interface for your system. | ||
45 | |||
46 | - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which | ||
47 | should generate a list of tap devices. This is the option | ||
48 | typically chosen for Autobuilder-type environments. | ||
49 | |||
50 | .. note:: | ||
51 | |||
52 | - Be sure to use an absolute path when calling this script | ||
53 | with sudo. | ||
54 | |||
55 | - The package recipe ``qemu-helper-native`` is required to run | ||
56 | this script. Build the package using the following command:: | ||
57 | |||
58 | $ bitbake qemu-helper-native | ||
59 | |||
60 | - *Set the DISPLAY variable:* You need to set this variable so that | ||
61 | you have an X server available (e.g. start ``vncserver`` for a | ||
62 | headless machine). | ||
63 | |||
64 | - *Be sure your host's firewall accepts incoming connections from | ||
65 | 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an | ||
66 | HTTP server on a random high number port, which is used to serve | ||
67 | files to the target. The DNF module serves | ||
68 | ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands. | ||
69 | That means your host's firewall must accept incoming connections from | ||
70 | 192.168.7.0/24, which is the default IP range used for tap devices by | ||
71 | ``runqemu``. | ||
72 | |||
73 | - *Be sure your host has the correct packages installed:* Depending | ||
74 | your host's distribution, you need to have the following packages | ||
75 | installed: | ||
76 | |||
77 | - Ubuntu and Debian: ``sysstat`` and ``iproute2`` | ||
78 | |||
79 | - openSUSE: ``sysstat`` and ``iproute2`` | ||
80 | |||
81 | - Fedora: ``sysstat`` and ``iproute`` | ||
82 | |||
83 | - CentOS: ``sysstat`` and ``iproute`` | ||
84 | |||
85 | Once you start running the tests, the following happens: | ||
86 | |||
87 | 1. A copy of the root filesystem is written to ``${WORKDIR}/testimage``. | ||
88 | |||
89 | 2. The image is booted under QEMU using the standard ``runqemu`` script. | ||
90 | |||
91 | 3. A default timeout of 500 seconds occurs to allow for the boot process | ||
92 | to reach the login prompt. You can change the timeout period by | ||
93 | setting | ||
94 | :term:`TEST_QEMUBOOT_TIMEOUT` | ||
95 | in the ``local.conf`` file. | ||
96 | |||
97 | 4. Once the boot process is reached and the login prompt appears, the | ||
98 | tests run. The full boot log is written to | ||
99 | ``${WORKDIR}/testimage/qemu_boot_log``. | ||
100 | |||
101 | 5. Each test module loads in the order found in :term:`TEST_SUITES`. You can | ||
102 | find the full output of the commands run over SSH in | ||
103 | ``${WORKDIR}/testimgage/ssh_target_log``. | ||
104 | |||
105 | 6. If no failures occur, the task running the tests ends successfully. | ||
106 | You can find the output from the ``unittest`` in the task log at | ||
107 | ``${WORKDIR}/temp/log.do_testimage``. | ||
108 | |||
109 | Enabling Runtime Tests on Hardware | ||
110 | ---------------------------------- | ||
111 | |||
112 | The OpenEmbedded build system can run tests on real hardware, and for | ||
113 | certain devices it can also deploy the image to be tested onto the | ||
114 | device beforehand. | ||
115 | |||
116 | For automated deployment, a "controller image" is installed onto the | ||
117 | hardware once as part of setup. Then, each time tests are to be run, the | ||
118 | following occurs: | ||
119 | |||
120 | 1. The controller image is booted into and used to write the image to be | ||
121 | tested to a second partition. | ||
122 | |||
123 | 2. The device is then rebooted using an external script that you need to | ||
124 | provide. | ||
125 | |||
126 | 3. The device boots into the image to be tested. | ||
127 | |||
128 | When running tests (independent of whether the image has been deployed | ||
129 | automatically or not), the device is expected to be connected to a | ||
130 | network on a pre-determined IP address. You can either use static IP | ||
131 | addresses written into the image, or set the image to use DHCP and have | ||
132 | your DHCP server on the test network assign a known IP address based on | ||
133 | the MAC address of the device. | ||
134 | |||
135 | In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an | ||
136 | appropriate value. For QEMU, you do not have to change anything, the | ||
137 | default value is "qemu". For running tests on hardware, the following | ||
138 | options are available: | ||
139 | |||
140 | - *"simpleremote":* Choose "simpleremote" if you are going to run tests | ||
141 | on a target system that is already running the image to be tested and | ||
142 | is available on the network. You can use "simpleremote" in | ||
143 | conjunction with either real hardware or an image running within a | ||
144 | separately started QEMU or any other virtual machine manager. | ||
145 | |||
146 | - *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is | ||
147 | an EFI-based machine with ``systemd-boot`` as bootloader and | ||
148 | ``core-image-testmaster`` (or something similar) is installed. Also, | ||
149 | your hardware under test must be in a DHCP-enabled network that gives | ||
150 | it the same IP address for each reboot. | ||
151 | |||
152 | If you choose "SystemdbootTarget", there are additional requirements | ||
153 | and considerations. See the | ||
154 | ":ref:`dev-manual/runtime-testing:selecting systemdboottarget`" section, which | ||
155 | follows, for more information. | ||
156 | |||
157 | - *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying | ||
158 | images and running tests on the BeagleBone "Black" or original | ||
159 | "White" hardware. For information on how to use these tests, see the | ||
160 | comments at the top of the BeagleBoneTarget | ||
161 | ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file. | ||
162 | |||
163 | - *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying | ||
164 | images and running tests on the Ubiquiti Networks EdgeRouter Lite. | ||
165 | For information on how to use these tests, see the comments at the | ||
166 | top of the EdgeRouterTarget | ||
167 | ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file. | ||
168 | |||
169 | - *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running | ||
170 | tests on any generic PC that boots using GRUB. For information on how | ||
171 | to use these tests, see the comments at the top of the GrubTarget | ||
172 | ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file. | ||
173 | |||
174 | - *"your-target":* Create your own custom target if you want to run | ||
175 | tests when you are deploying images and running tests on a custom | ||
176 | machine within your BSP layer. To do this, you need to add a Python | ||
177 | unit that defines the target class under ``lib/oeqa/controllers/`` | ||
178 | within your layer. You must also provide an empty ``__init__.py``. | ||
179 | For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``. | ||
180 | |||
181 | Selecting SystemdbootTarget | ||
182 | --------------------------- | ||
183 | |||
184 | If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do | ||
185 | not need any information in this section. You can skip down to the | ||
186 | ":ref:`dev-manual/runtime-testing:running tests`" section. | ||
187 | |||
188 | If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to | ||
189 | perform a one-time setup of your controller image by doing the following: | ||
190 | |||
191 | 1. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows:: | ||
192 | |||
193 | EFI_PROVIDER = "systemd-boot" | ||
194 | |||
195 | 2. *Build the controller image:* Build the ``core-image-testmaster`` image. | ||
196 | The ``core-image-testmaster`` recipe is provided as an example for a | ||
197 | "controller" image and you can customize the image recipe as you would | ||
198 | any other recipe. | ||
199 | |||
200 | Here are the image recipe requirements: | ||
201 | |||
202 | - Inherits ``core-image`` so that kernel modules are installed. | ||
203 | |||
204 | - Installs normal linux utilities not BusyBox ones (e.g. ``bash``, | ||
205 | ``coreutils``, ``tar``, ``gzip``, and ``kmod``). | ||
206 | |||
207 | - Uses a custom :term:`Initramfs` image with a custom | ||
208 | installer. A normal image that you can install usually creates a | ||
209 | single root filesystem partition. This image uses another installer that | ||
210 | creates a specific partition layout. Not all Board Support | ||
211 | Packages (BSPs) can use an installer. For such cases, you need to | ||
212 | manually create the following partition layout on the target: | ||
213 | |||
214 | - First partition mounted under ``/boot``, labeled "boot". | ||
215 | |||
216 | - The main root filesystem partition where this image gets installed, | ||
217 | which is mounted under ``/``. | ||
218 | |||
219 | - Another partition labeled "testrootfs" where test images get | ||
220 | deployed. | ||
221 | |||
222 | 3. *Install image:* Install the image that you just built on the target | ||
223 | system. | ||
224 | |||
225 | The final thing you need to do when setting :term:`TEST_TARGET` to | ||
226 | "SystemdbootTarget" is to set up the test image: | ||
227 | |||
228 | 1. *Set up your local.conf file:* Make sure you have the following | ||
229 | statements in your ``local.conf`` file:: | ||
230 | |||
231 | IMAGE_FSTYPES += "tar.gz" | ||
232 | INHERIT += "testimage" | ||
233 | TEST_TARGET = "SystemdbootTarget" | ||
234 | TEST_TARGET_IP = "192.168.2.3" | ||
235 | |||
236 | 2. *Build your test image:* Use BitBake to build the image:: | ||
237 | |||
238 | $ bitbake core-image-sato | ||
239 | |||
240 | Power Control | ||
241 | ------------- | ||
242 | |||
243 | For most hardware targets other than "simpleremote", you can control | ||
244 | power: | ||
245 | |||
246 | - You can use :term:`TEST_POWERCONTROL_CMD` together with | ||
247 | :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host | ||
248 | and does power cycling. The test code passes one argument to that | ||
249 | command: off, on or cycle (off then on). Here is an example that | ||
250 | could appear in your ``local.conf`` file:: | ||
251 | |||
252 | TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1" | ||
253 | |||
254 | In this example, the expect | ||
255 | script does the following: | ||
256 | |||
257 | .. code-block:: shell | ||
258 | |||
259 | ssh test@10.11.12.1 "pyctl nuc1 arg" | ||
260 | |||
261 | It then runs a Python script that controls power for a label called | ||
262 | ``nuc1``. | ||
263 | |||
264 | .. note:: | ||
265 | |||
266 | You need to customize :term:`TEST_POWERCONTROL_CMD` and | ||
267 | :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement | ||
268 | is that it accepts "on", "off", and "cycle" as the last argument. | ||
269 | |||
270 | - When no command is defined, it connects to the device over SSH and | ||
271 | uses the classic reboot command to reboot the device. Classic reboot | ||
272 | is fine as long as the machine actually reboots (i.e. the SSH test | ||
273 | has not failed). It is useful for scenarios where you have a simple | ||
274 | setup, typically with a single board, and where some manual | ||
275 | interaction is okay from time to time. | ||
276 | |||
277 | If you have no hardware to automatically perform power control but still | ||
278 | wish to experiment with automated hardware testing, you can use the | ||
279 | ``dialog-power-control`` script that shows a dialog prompting you to perform | ||
280 | the required power action. This script requires either KDialog or Zenity | ||
281 | to be installed. To use this script, set the | ||
282 | :term:`TEST_POWERCONTROL_CMD` | ||
283 | variable as follows:: | ||
284 | |||
285 | TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control" | ||
286 | |||
287 | Serial Console Connection | ||
288 | ------------------------- | ||
289 | |||
290 | For test target classes requiring a serial console to interact with the | ||
291 | bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget), | ||
292 | you need to specify a command to use to connect to the serial console of | ||
293 | the target machine by using the | ||
294 | :term:`TEST_SERIALCONTROL_CMD` | ||
295 | variable and optionally the | ||
296 | :term:`TEST_SERIALCONTROL_EXTRA_ARGS` | ||
297 | variable. | ||
298 | |||
299 | These cases could be a serial terminal program if the machine is | ||
300 | connected to a local serial port, or a ``telnet`` or ``ssh`` command | ||
301 | connecting to a remote console server. Regardless of the case, the | ||
302 | command simply needs to connect to the serial console and forward that | ||
303 | connection to standard input and output as any normal terminal program | ||
304 | does. For example, to use the picocom terminal program on serial device | ||
305 | ``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows:: | ||
306 | |||
307 | TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200" | ||
308 | |||
309 | For local | ||
310 | devices where the serial port device disappears when the device reboots, | ||
311 | an additional "serdevtry" wrapper script is provided. To use this | ||
312 | wrapper, simply prefix the terminal command with | ||
313 | ``${COREBASE}/scripts/contrib/serdevtry``:: | ||
314 | |||
315 | TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0" | ||
316 | |||
317 | Running Tests | ||
318 | ============= | ||
319 | |||
320 | You can start the tests automatically or manually: | ||
321 | |||
322 | - *Automatically running tests:* To run the tests automatically after the | ||
323 | OpenEmbedded build system successfully creates an image, first set the | ||
324 | :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the | ||
325 | :term:`Build Directory`:: | ||
326 | |||
327 | TESTIMAGE_AUTO = "1" | ||
328 | |||
329 | Next, build your image. If the image successfully builds, the | ||
330 | tests run:: | ||
331 | |||
332 | bitbake core-image-sato | ||
333 | |||
334 | - *Manually running tests:* To manually run the tests, first globally | ||
335 | inherit the :ref:`testimage <ref-classes-testimage>` class | ||
336 | by editing your ``local.conf`` file:: | ||
337 | |||
338 | INHERIT += "testimage" | ||
339 | |||
340 | Next, use BitBake to run the tests:: | ||
341 | |||
342 | bitbake -c testimage image | ||
343 | |||
344 | All test files reside in ``meta/lib/oeqa/runtime`` in the | ||
345 | :term:`Source Directory`. A test name maps | ||
346 | directly to a Python module. Each test module may contain a number of | ||
347 | individual tests. Tests are usually grouped together by the area tested | ||
348 | (e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``). | ||
349 | |||
350 | You can add tests to any layer provided you place them in the proper | ||
351 | area and you extend :term:`BBPATH` in | ||
352 | the ``local.conf`` file as normal. Be sure that tests reside in | ||
353 | ``layer/lib/oeqa/runtime``. | ||
354 | |||
355 | .. note:: | ||
356 | |||
357 | Be sure that module names do not collide with module names used in | ||
358 | the default set of test modules in ``meta/lib/oeqa/runtime``. | ||
359 | |||
360 | You can change the set of tests run by appending or overriding | ||
361 | :term:`TEST_SUITES` variable in | ||
362 | ``local.conf``. Each name in :term:`TEST_SUITES` represents a required test | ||
363 | for the image. Test modules named within :term:`TEST_SUITES` cannot be | ||
364 | skipped even if a test is not suitable for an image (e.g. running the | ||
365 | RPM tests on an image without ``rpm``). Appending "auto" to | ||
366 | :term:`TEST_SUITES` causes the build system to try to run all tests that are | ||
367 | suitable for the image (i.e. each test module may elect to skip itself). | ||
368 | |||
369 | The order you list tests in :term:`TEST_SUITES` is important and influences | ||
370 | test dependencies. Consequently, tests that depend on other tests should | ||
371 | be added after the test on which they depend. For example, since the | ||
372 | ``ssh`` test depends on the ``ping`` test, "ssh" needs to come after | ||
373 | "ping" in the list. The test class provides no re-ordering or dependency | ||
374 | handling. | ||
375 | |||
376 | .. note:: | ||
377 | |||
378 | Each module can have multiple classes with multiple test methods. | ||
379 | And, Python ``unittest`` rules apply. | ||
380 | |||
381 | Here are some things to keep in mind when running tests: | ||
382 | |||
383 | - The default tests for the image are defined as:: | ||
384 | |||
385 | DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg" | ||
386 | |||
387 | - Add your own test to the list of the by using the following:: | ||
388 | |||
389 | TEST_SUITES:append = " mytest" | ||
390 | |||
391 | - Run a specific list of tests as follows:: | ||
392 | |||
393 | TEST_SUITES = "test1 test2 test3" | ||
394 | |||
395 | Remember, order is important. Be sure to place a test that is | ||
396 | dependent on another test later in the order. | ||
397 | |||
398 | Exporting Tests | ||
399 | =============== | ||
400 | |||
401 | You can export tests so that they can run independently of the build | ||
402 | system. Exporting tests is required if you want to be able to hand the | ||
403 | test execution off to a scheduler. You can only export tests that are | ||
404 | defined in :term:`TEST_SUITES`. | ||
405 | |||
406 | If your image is already built, make sure the following are set in your | ||
407 | ``local.conf`` file:: | ||
408 | |||
409 | INHERIT += "testexport" | ||
410 | TEST_TARGET_IP = "IP-address-for-the-test-target" | ||
411 | TEST_SERVER_IP = "IP-address-for-the-test-server" | ||
412 | |||
413 | You can then export the tests with the | ||
414 | following BitBake command form:: | ||
415 | |||
416 | $ bitbake image -c testexport | ||
417 | |||
418 | Exporting the tests places them in the :term:`Build Directory` in | ||
419 | ``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR` | ||
420 | variable. | ||
421 | |||
422 | You can now run the tests outside of the build environment:: | ||
423 | |||
424 | $ cd tmp/testexport/image | ||
425 | $ ./runexported.py testdata.json | ||
426 | |||
427 | Here is a complete example that shows IP addresses and uses the | ||
428 | ``core-image-sato`` image:: | ||
429 | |||
430 | INHERIT += "testexport" | ||
431 | TEST_TARGET_IP = "192.168.7.2" | ||
432 | TEST_SERVER_IP = "192.168.7.1" | ||
433 | |||
434 | Use BitBake to export the tests:: | ||
435 | |||
436 | $ bitbake core-image-sato -c testexport | ||
437 | |||
438 | Run the tests outside of | ||
439 | the build environment using the following:: | ||
440 | |||
441 | $ cd tmp/testexport/core-image-sato | ||
442 | $ ./runexported.py testdata.json | ||
443 | |||
444 | Writing New Tests | ||
445 | ================= | ||
446 | |||
447 | As mentioned previously, all new test files need to be in the proper | ||
448 | place for the build system to find them. New tests for additional | ||
449 | functionality outside of the core should be added to the layer that adds | ||
450 | the functionality, in ``layer/lib/oeqa/runtime`` (as long as | ||
451 | :term:`BBPATH` is extended in the | ||
452 | layer's ``layer.conf`` file as normal). Just remember the following: | ||
453 | |||
454 | - Filenames need to map directly to test (module) names. | ||
455 | |||
456 | - Do not use module names that collide with existing core tests. | ||
457 | |||
458 | - Minimally, an empty ``__init__.py`` file must be present in the runtime | ||
459 | directory. | ||
460 | |||
461 | To create a new test, start by copying an existing module (e.g. | ||
462 | ``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use | ||
463 | code from ``meta/lib/oeqa/utils``, which are helper classes. | ||
464 | |||
465 | .. note:: | ||
466 | |||
467 | Structure shell commands such that you rely on them and they return a | ||
468 | single code for success. Be aware that sometimes you will need to | ||
469 | parse the output. See the ``df.py`` and ``date.py`` modules for examples. | ||
470 | |||
471 | You will notice that all test classes inherit ``oeRuntimeTest``, which | ||
472 | is found in ``meta/lib/oetest.py``. This base class offers some helper | ||
473 | attributes, which are described in the following sections: | ||
474 | |||
475 | Class Methods | ||
476 | ------------- | ||
477 | |||
478 | Class methods are as follows: | ||
479 | |||
480 | - *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed | ||
481 | package list of the image, which is based on the manifest file that | ||
482 | is generated during the :ref:`ref-tasks-rootfs` task. | ||
483 | |||
484 | - *hasFeature(feature):* Returns "True" if the feature is in | ||
485 | :term:`IMAGE_FEATURES` or | ||
486 | :term:`DISTRO_FEATURES`. | ||
487 | |||
488 | Class Attributes | ||
489 | ---------------- | ||
490 | |||
491 | Class attributes are as follows: | ||
492 | |||
493 | - *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image. | ||
494 | Otherwise, ``pscmd`` equals "ps" (busybox). | ||
495 | |||
496 | - *tc:* The called test context, which gives access to the | ||
497 | following attributes: | ||
498 | |||
499 | - *d:* The BitBake datastore, which allows you to use stuff such | ||
500 | as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``. | ||
501 | |||
502 | - *testslist and testsrequired:* Used internally. The tests | ||
503 | do not need these. | ||
504 | |||
505 | - *filesdir:* The absolute path to | ||
506 | ``meta/lib/oeqa/runtime/files``, which contains helper files for | ||
507 | tests meant for copying on the target such as small files written | ||
508 | in C for compilation. | ||
509 | |||
510 | - *target:* The target controller object used to deploy and | ||
511 | start an image on a particular target (e.g. Qemu, SimpleRemote, | ||
512 | and SystemdbootTarget). Tests usually use the following: | ||
513 | |||
514 | - *ip:* The target's IP address. | ||
515 | |||
516 | - *server_ip:* The host's IP address, which is usually used | ||
517 | by the DNF test suite. | ||
518 | |||
519 | - *run(cmd, timeout=None):* The single, most used method. | ||
520 | This command is a wrapper for: ``ssh root@host "cmd"``. The | ||
521 | command returns a tuple: (status, output), which are what their | ||
522 | names imply - the return code of "cmd" and whatever output it | ||
523 | produces. The optional timeout argument represents the number | ||
524 | of seconds the test should wait for "cmd" to return. If the | ||
525 | argument is "None", the test uses the default instance's | ||
526 | timeout period, which is 300 seconds. If the argument is "0", | ||
527 | the test runs until the command returns. | ||
528 | |||
529 | - *copy_to(localpath, remotepath):* | ||
530 | ``scp localpath root@ip:remotepath``. | ||
531 | |||
532 | - *copy_from(remotepath, localpath):* | ||
533 | ``scp root@host:remotepath localpath``. | ||
534 | |||
535 | Instance Attributes | ||
536 | ------------------- | ||
537 | |||
538 | There is a single instance attribute, which is ``target``. The ``target`` | ||
539 | instance attribute is identical to the class attribute of the same name, | ||
540 | which is described in the previous section. This attribute exists as | ||
541 | both an instance and class attribute so tests can use | ||
542 | ``self.target.run(cmd)`` in instance methods instead of | ||
543 | ``oeRuntimeTest.tc.target.run(cmd)``. | ||
544 | |||
545 | Installing Packages in the DUT Without the Package Manager | ||
546 | ========================================================== | ||
547 | |||
548 | When a test requires a package built by BitBake, it is possible to | ||
549 | install that package. Installing the package does not require a package | ||
550 | manager be installed in the device under test (DUT). It does, however, | ||
551 | require an SSH connection and the target must be using the | ||
552 | ``sshcontrol`` class. | ||
553 | |||
554 | .. note:: | ||
555 | |||
556 | This method uses ``scp`` to copy files from the host to the target, which | ||
557 | causes permissions and special attributes to be lost. | ||
558 | |||
559 | A JSON file is used to define the packages needed by a test. This file | ||
560 | must be in the same path as the file used to define the tests. | ||
561 | Furthermore, the filename must map directly to the test module name with | ||
562 | a ``.json`` extension. | ||
563 | |||
564 | The JSON file must include an object with the test name as keys of an | ||
565 | object or an array. This object (or array of objects) uses the following | ||
566 | data: | ||
567 | |||
568 | - "pkg" --- a mandatory string that is the name of the package to be | ||
569 | installed. | ||
570 | |||
571 | - "rm" --- an optional boolean, which defaults to "false", that specifies | ||
572 | to remove the package after the test. | ||
573 | |||
574 | - "extract" --- an optional boolean, which defaults to "false", that | ||
575 | specifies if the package must be extracted from the package format. | ||
576 | When set to "true", the package is not automatically installed into | ||
577 | the DUT. | ||
578 | |||
579 | Following is an example JSON file that handles test "foo" installing | ||
580 | package "bar" and test "foobar" installing packages "foo" and "bar". | ||
581 | Once the test is complete, the packages are removed from the DUT. | ||
582 | :: | ||
583 | |||
584 | { | ||
585 | "foo": { | ||
586 | "pkg": "bar" | ||
587 | }, | ||
588 | "foobar": [ | ||
589 | { | ||
590 | "pkg": "foo", | ||
591 | "rm": true | ||
592 | }, | ||
593 | { | ||
594 | "pkg": "bar", | ||
595 | "rm": true | ||
596 | } | ||
597 | ] | ||
598 | } | ||
599 | |||