summaryrefslogtreecommitdiffstats
path: root/documentation/dev-manual/common-tasks.rst
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/dev-manual/common-tasks.rst')
-rw-r--r--documentation/dev-manual/common-tasks.rst11587
1 files changed, 0 insertions, 11587 deletions
diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
deleted file mode 100644
index 65db4aed33..0000000000
--- a/documentation/dev-manual/common-tasks.rst
+++ /dev/null
@@ -1,11587 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
21":ref:`overview-manual/yp-intro:the yocto project layer model`"
22section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
27It is very easy to create your own layers to use with the OpenEmbedded
28build system. The Yocto Project ships with tools that speed up creating
29layers. This section describes the steps you perform by hand to create
30layers so that you can better understand them. For information about the
31layer-creation tools, see the
32":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
33section in the Yocto Project Board Support Package (BSP) Developer's
34Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
35section further down in this manual.
36
37Follow these general steps to create your layer without using tools:
38
391. *Check Existing Layers:* Before creating a new layer, you should be
40 sure someone has not already created a layer containing the Metadata
41 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
42 for a list of layers from the OpenEmbedded community that can be used in
43 the Yocto Project. You could find a layer that is identical or close
44 to what you need.
45
462. *Create a Directory:* Create the directory for your layer. When you
47 create the layer, be sure to create the directory in an area not
48 associated with the Yocto Project :term:`Source Directory`
49 (e.g. the cloned ``poky`` repository).
50
51 While not strictly required, prepend the name of the directory with
52 the string "meta-". For example:
53 ::
54
55 meta-mylayer
56 meta-GUI_xyz
57 meta-mymachine
58
59 With rare exceptions, a layer's name follows this form:
60 ::
61
62 meta-root_name
63
64 Following this layer naming convention can save
65 you trouble later when tools, components, or variables "assume" your
66 layer name begins with "meta-". A notable example is in configuration
67 files as shown in the following step where layer names without the
68 "meta-" string are appended to several variables used in the
69 configuration.
70
713. *Create a Layer Configuration File:* Inside your new layer folder,
72 you need to create a ``conf/layer.conf`` file. It is easiest to take
73 an existing layer configuration file and copy that to your layer's
74 ``conf`` directory and then modify the file as needed.
75
76 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
77 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
78 demonstrates the required syntax. For your layer, you need to replace
79 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
80 for a layer named "meta-machinexyz"):
81 ::
82
83 # We have a conf and classes directory, add to BBPATH
84 BBPATH .= ":${LAYERDIR}"
85
86 # We have recipes-* directories, add to BBFILES
87 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
88 ${LAYERDIR}/recipes-*/*/*.bbappend"
89
90 BBFILE_COLLECTIONS += "yoctobsp"
91 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
92 BBFILE_PRIORITY_yoctobsp = "5"
93 LAYERVERSION_yoctobsp = "4"
94 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
95
96 Following is an explanation of the layer configuration file:
97
98 - :term:`BBPATH`: Adds the layer's
99 root directory to BitBake's search path. Through the use of the
100 ``BBPATH`` variable, BitBake locates class files (``.bbclass``),
101 configuration files, and files that are included with ``include``
102 and ``require`` statements. For these cases, BitBake uses the
103 first file that matches the name found in ``BBPATH``. This is
104 similar to the way the ``PATH`` variable is used for binaries. It
105 is recommended, therefore, that you use unique class and
106 configuration filenames in your custom layer.
107
108 - :term:`BBFILES`: Defines the
109 location for all recipes in the layer.
110
111 - :term:`BBFILE_COLLECTIONS`:
112 Establishes the current layer through a unique identifier that is
113 used throughout the OpenEmbedded build system to refer to the
114 layer. In this example, the identifier "yoctobsp" is the
115 representation for the container layer named "meta-yocto-bsp".
116
117 - :term:`BBFILE_PATTERN`:
118 Expands immediately during parsing to provide the directory of the
119 layer.
120
121 - :term:`BBFILE_PRIORITY`:
122 Establishes a priority to use for recipes in the layer when the
123 OpenEmbedded build finds recipes of the same name in different
124 layers.
125
126 - :term:`LAYERVERSION`:
127 Establishes a version number for the layer. You can use this
128 version number to specify this exact version of the layer as a
129 dependency when using the
130 :term:`LAYERDEPENDS`
131 variable.
132
133 - :term:`LAYERDEPENDS`:
134 Lists all layers on which this layer depends (if any).
135
136 - :term:`LAYERSERIES_COMPAT`:
137 Lists the :yocto_wiki:`Yocto Project </Releases>`
138 releases for which the current version is compatible. This
139 variable is a good way to indicate if your particular layer is
140 current.
141
1424. *Add Content:* Depending on the type of layer, add the content. If
143 the layer adds support for a machine, add the machine configuration
144 in a ``conf/machine/`` file within the layer. If the layer adds
145 distro policy, add the distro configuration in a ``conf/distro/``
146 file within the layer. If the layer introduces new recipes, put the
147 recipes you need in ``recipes-*`` subdirectories within the layer.
148
149 .. note::
150
151 For an explanation of layer hierarchy that is compliant with the
152 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
153 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
154
1555. *Optionally Test for Compatibility:* If you want permission to use
156 the Yocto Project Compatibility logo with your layer or application
157 that uses your layer, perform the steps to apply for compatibility.
158 See the "`Making Sure Your Layer is Compatible With Yocto
159 Project <#making-sure-your-layer-is-compatible-with-yocto-project>`__"
160 section for more information.
161
162Following Best Practices When Creating Layers
163---------------------------------------------
164
165To create layers that are easier to maintain and that will not impact
166builds for other machines, you should consider the information in the
167following list:
168
169- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
170 Configuration:* In other words, do not copy an entire recipe into
171 your layer and then modify it. Rather, use an append file
172 (``.bbappend``) to override only those parts of the original recipe
173 you need to modify.
174
175- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
176 for each recipe that uses an include file. Or, if you are introducing
177 a new recipe that requires the included file, use the path relative
178 to the original layer directory to refer to the file. For example,
179 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
180 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
181 the include file, it could indicate a deficiency in the include file
182 in the layer to which it originally belongs. If this is the case, you
183 should try to address that deficiency instead of overlaying the
184 include file. For example, you could address this by getting the
185 maintainer of the include file to add a variable or variables to make
186 it easy to override the parts needing to be overridden.
187
188- *Structure Your Layers:* Proper use of overrides within append files
189 and placement of machine-specific files within your layer can ensure
190 that a build is not using the wrong Metadata and negatively impacting
191 a build for a different machine. Following are some examples:
192
193 - *Modify Variables to Support a Different Machine:* Suppose you
194 have a layer named ``meta-one`` that adds support for building
195 machine "one". To do so, you use an append file named
196 ``base-files.bbappend`` and create a dependency on "foo" by
197 altering the :term:`DEPENDS`
198 variable:
199 ::
200
201 DEPENDS = "foo"
202
203 The dependency is created during any
204 build that includes the layer ``meta-one``. However, you might not
205 want this dependency for all machines. For example, suppose you
206 are building for machine "two" but your ``bblayers.conf`` file has
207 the ``meta-one`` layer included. During the build, the
208 ``base-files`` for machine "two" will also have the dependency on
209 ``foo``.
210
211 To make sure your changes apply only when building machine "one",
212 use a machine override with the ``DEPENDS`` statement:
213 ::
214
215 DEPENDS_one = "foo"
216
217 You should follow the same strategy when using ``_append``
218 and ``_prepend`` operations:
219 ::
220
221 DEPENDS_append_one = " foo"
222 DEPENDS_prepend_one = "foo "
223
224 As an actual example, here's a
225 snippet from the generic kernel include file ``linux-yocto.inc``,
226 wherein the kernel compile and link options are adjusted in the
227 case of a subset of the supported architectures:
228 ::
229
230 DEPENDS_append_aarch64 = " libgcc"
231 KERNEL_CC_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
232 KERNEL_LD_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
233
234 DEPENDS_append_nios2 = " libgcc"
235 KERNEL_CC_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
236 KERNEL_LD_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
237
238 DEPENDS_append_arc = " libgcc"
239 KERNEL_CC_append_arc = " ${TOOLCHAIN_OPTIONS}"
240 KERNEL_LD_append_arc = " ${TOOLCHAIN_OPTIONS}"
241
242 KERNEL_FEATURES_append_qemuall=" features/debug/printk.scc"
243
244 .. note::
245
246 Avoiding "+=" and "=+" and using machine-specific ``_append``
247 and ``_prepend`` operations is recommended as well.
248
249 - *Place Machine-Specific Files in Machine-Specific Locations:* When
250 you have a base recipe, such as ``base-files.bb``, that contains a
251 :term:`SRC_URI` statement to a
252 file, you can use an append file to cause the build to use your
253 own version of the file. For example, an append file in your layer
254 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
255 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows:
256 ::
257
258 FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}:"
259
260 The build for machine "one" will pick up your machine-specific file as
261 long as you have the file in
262 ``meta-one/recipes-core/base-files/base-files/``. However, if you
263 are building for a different machine and the ``bblayers.conf``
264 file includes the ``meta-one`` layer and the location of your
265 machine-specific file is the first location where that file is
266 found according to ``FILESPATH``, builds for all machines will
267 also use that machine-specific file.
268
269 You can make sure that a machine-specific file is used for a
270 particular machine by putting the file in a subdirectory specific
271 to the machine. For example, rather than placing the file in
272 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
273 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
274 Not only does this make sure the file is used only when building
275 for machine "one", but the build process locates the file more
276 quickly.
277
278 In summary, you need to place all files referenced from
279 ``SRC_URI`` in a machine-specific subdirectory within the layer in
280 order to restrict those files to machine-specific builds.
281
282- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
283 permission to use the Yocto Project Compatibility logo with your
284 layer or application that uses your layer, perform the steps to apply
285 for compatibility. See the "`Making Sure Your Layer is Compatible
286 With Yocto
287 Project <#making-sure-your-layer-is-compatible-with-yocto-project>`__"
288 section for more information.
289
290- *Follow the Layer Naming Convention:* Store custom layers in a Git
291 repository that use the ``meta-layer_name`` format.
292
293- *Group Your Layers Locally:* Clone your repository alongside other
294 cloned ``meta`` directories from the :term:`Source Directory`.
295
296Making Sure Your Layer is Compatible With Yocto Project
297-------------------------------------------------------
298
299When you create a layer used with the Yocto Project, it is advantageous
300to make sure that the layer interacts well with existing Yocto Project
301layers (i.e. the layer is compatible with the Yocto Project). Ensuring
302compatibility makes the layer easy to be consumed by others in the Yocto
303Project community and could allow you permission to use the Yocto
304Project Compatible Logo.
305
306.. note::
307
308 Only Yocto Project member organizations are permitted to use the
309 Yocto Project Compatible Logo. The logo is not available for general
310 use. For information on how to become a Yocto Project member
311 organization, see the :yocto_home:`Yocto Project Website <>`.
312
313The Yocto Project Compatibility Program consists of a layer application
314process that requests permission to use the Yocto Project Compatibility
315Logo for your layer and application. The process consists of two parts:
316
3171. Successfully passing a script (``yocto-check-layer``) that when run
318 against your layer, tests it against constraints based on experiences
319 of how layers have worked in the real world and where pitfalls have
320 been found. Getting a "PASS" result from the script is required for
321 successful compatibility registration.
322
3232. Completion of an application acceptance form, which you can find at
324 :yocto_home:`/webform/yocto-project-compatible-registration`.
325
326To be granted permission to use the logo, you need to satisfy the
327following:
328
329- Be able to check the box indicating that you got a "PASS" when
330 running the script against your layer.
331
332- Answer "Yes" to the questions on the form or have an acceptable
333 explanation for any questions answered "No".
334
335- Be a Yocto Project Member Organization.
336
337The remainder of this section presents information on the registration
338form and on the ``yocto-check-layer`` script.
339
340Yocto Project Compatible Program Application
341~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
342
343Use the form to apply for your layer's approval. Upon successful
344application, you can use the Yocto Project Compatibility Logo with your
345layer and the application that uses your layer.
346
347To access the form, use this link:
348:yocto_home:`/webform/yocto-project-compatible-registration`.
349Follow the instructions on the form to complete your application.
350
351The application consists of the following sections:
352
353- *Contact Information:* Provide your contact information as the fields
354 require. Along with your information, provide the released versions
355 of the Yocto Project for which your layer is compatible.
356
357- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
358 items in the checklist. Space exists at the bottom of the form for
359 any explanations for items for which you answered "No".
360
361- *Recommendations:* Provide answers for the questions regarding Linux
362 kernel use and build success.
363
364``yocto-check-layer`` Script
365~~~~~~~~~~~~~~~~~~~~~~~~~~~~
366
367The ``yocto-check-layer`` script provides you a way to assess how
368compatible your layer is with the Yocto Project. You should run this
369script prior to using the form to apply for compatibility as described
370in the previous section. You need to achieve a "PASS" result in order to
371have your application form successfully processed.
372
373The script divides tests into three areas: COMMON, BSP, and DISTRO. For
374example, given a distribution layer (DISTRO), the layer must pass both
375the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
376layer, the layer must pass the COMMON and BSP set of tests.
377
378To execute the script, enter the following commands from your build
379directory:
380::
381
382 $ source oe-init-build-env
383 $ yocto-check-layer your_layer_directory
384
385Be sure to provide the actual directory for your
386layer as part of the command.
387
388Entering the command causes the script to determine the type of layer
389and then to execute a set of specific tests against the layer. The
390following list overviews the test:
391
392- ``common.test_readme``: Tests if a ``README`` file exists in the
393 layer and the file is not empty.
394
395- ``common.test_parse``: Tests to make sure that BitBake can parse the
396 files without error (i.e. ``bitbake -p``).
397
398- ``common.test_show_environment``: Tests that the global or per-recipe
399 environment is in order without errors (i.e. ``bitbake -e``).
400
401- ``common.test_world``: Verifies that ``bitbake world`` works.
402
403- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
404 layers do not come with recipes that change signatures.
405
406- ``common.test_layerseries_compat``: Verifies layer compatibility is
407 set properly.
408
409- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
410 configurations.
411
412- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
413 set the machine when the layer is added.
414
415- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
416 regardless of which machine is selected.
417
418- ``bsp.test_machine_signatures``: Verifies that building for a
419 particular machine affects only the signature of tasks specific to
420 that machine.
421
422- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
423 distro configurations.
424
425- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
426 does not set the distribution when the layer is added.
427
428Enabling Your Layer
429-------------------
430
431Before the OpenEmbedded build system can use your new layer, you need to
432enable it. To enable your layer, simply add your layer's path to the
433``BBLAYERS`` variable in your ``conf/bblayers.conf`` file, which is
434found in the :term:`Build Directory`.
435The following example shows how to enable a layer named
436``meta-mylayer``:
437::
438
439 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
440 # changes incompatibly
441 POKY_BBLAYERS_CONF_VERSION = "2"
442 BBPATH = "${TOPDIR}"
443 BBFILES ?= ""
444 BBLAYERS ?= " \
445 /home/user/poky/meta \
446 /home/user/poky/meta-poky \
447 /home/user/poky/meta-yocto-bsp \
448 /home/user/poky/meta-mylayer \
449 "
450
451BitBake parses each ``conf/layer.conf`` file from the top down as
452specified in the ``BBLAYERS`` variable within the ``conf/bblayers.conf``
453file. During the processing of each ``conf/layer.conf`` file, BitBake
454adds the recipes, classes and configurations contained within the
455particular layer to the source directory.
456
457Using .bbappend Files in Your Layer
458-----------------------------------
459
460A recipe that appends Metadata to another recipe is called a BitBake
461append file. A BitBake append file uses the ``.bbappend`` file type
462suffix, while the corresponding recipe to which Metadata is being
463appended uses the ``.bb`` file type suffix.
464
465You can use a ``.bbappend`` file in your layer to make additions or
466changes to the content of another layer's recipe without having to copy
467the other layer's recipe into your layer. Your ``.bbappend`` file
468resides in your layer, while the main ``.bb`` recipe file to which you
469are appending Metadata resides in a different layer.
470
471Being able to append information to an existing recipe not only avoids
472duplication, but also automatically applies recipe changes from a
473different layer into your layer. If you were copying recipes, you would
474have to manually merge changes as they occur.
475
476When you create an append file, you must use the same root name as the
477corresponding recipe file. For example, the append file
478``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
479means the original recipe and append file names are version
480number-specific. If the corresponding recipe is renamed to update to a
481newer version, you must also rename and possibly update the
482corresponding ``.bbappend`` as well. During the build process, BitBake
483displays an error on starting if it detects a ``.bbappend`` file that
484does not have a corresponding recipe with a matching name. See the
485:term:`BB_DANGLINGAPPENDS_WARNONLY`
486variable for information on how to handle this error.
487
488As an example, consider the main formfactor recipe and a corresponding
489formfactor append file both from the :term:`Source Directory`.
490Here is the main
491formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
492the "meta" layer at ``meta/recipes-bsp/formfactor``:
493::
494
495 SUMMARY = "Device formfactor information"
496 DESCRIPTION = "A formfactor configuration file provides information about the \
497 target hardware for which the image is being built and information that the \
498 build system cannot obtain from other sources such as the kernel."
499 SECTION = "base"
500 LICENSE = "MIT"
501 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
502 PR = "r45"
503
504 SRC_URI = "file://config file://machconfig"
505 S = "${WORKDIR}"
506
507 PACKAGE_ARCH = "${MACHINE_ARCH}"
508 INHIBIT_DEFAULT_DEPS = "1"
509
510 do_install() {
511 # Install file only if it has contents
512 install -d ${D}${sysconfdir}/formfactor/
513 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
514 if [ -s "${S}/machconfig" ]; then
515 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
516 fi
517 }
518
519In the main recipe, note the :term:`SRC_URI`
520variable, which tells the OpenEmbedded build system where to find files
521during the build.
522
523Following is the append file, which is named ``formfactor_0.0.bbappend``
524and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
525file is in the layer at ``recipes-bsp/formfactor``:
526::
527
528 FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
529
530By default, the build system uses the
531:term:`FILESPATH` variable to
532locate files. This append file extends the locations by setting the
533:term:`FILESEXTRAPATHS`
534variable. Setting this variable in the ``.bbappend`` file is the most
535reliable and recommended method for adding directories to the search
536path used by the build system to find files.
537
538The statement in this example extends the directories to include
539``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
540which resolves to a directory named ``formfactor`` in the same directory
541in which the append file resides (i.e.
542``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
543have the supporting directory structure set up that will contain any
544files or patches you will be including from the layer.
545
546Using the immediate expansion assignment operator ``:=`` is important
547because of the reference to ``THISDIR``. The trailing colon character is
548important as it ensures that items in the list remain colon-separated.
549
550.. note::
551
552 BitBake automatically defines the ``THISDIR`` variable. You should
553 never set this variable yourself. Using "_prepend" as part of the
554 ``FILESEXTRAPATHS`` ensures your path will be searched prior to other
555 paths in the final list.
556
557 Also, not all append files add extra files. Many append files simply
558 exist to add build options (e.g. ``systemd``). For these cases, your
559 append file would not even use the ``FILESEXTRAPATHS`` statement.
560
561Prioritizing Your Layer
562-----------------------
563
564Each layer is assigned a priority value. Priority values control which
565layer takes precedence if there are recipe files with the same name in
566multiple layers. For these cases, the recipe file from the layer with a
567higher priority number takes precedence. Priority values also affect the
568order in which multiple ``.bbappend`` files for the same recipe are
569applied. You can either specify the priority manually, or allow the
570build system to calculate it based on the layer's dependencies.
571
572To specify the layer's priority manually, use the
573:term:`BBFILE_PRIORITY`
574variable and append the layer's root name:
575::
576
577 BBFILE_PRIORITY_mylayer = "1"
578
579.. note::
580
581 It is possible for a recipe with a lower version number
582 :term:`PV` in a layer that has a higher
583 priority to take precedence.
584
585 Also, the layer priority does not currently affect the precedence
586 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
587 might address this.
588
589Managing Layers
590---------------
591
592You can use the BitBake layer management tool ``bitbake-layers`` to
593provide a view into the structure of recipes across a multi-layer
594project. Being able to generate output that reports on configured layers
595with their paths and priorities and on ``.bbappend`` files and their
596applicable recipes can help to reveal potential problems.
597
598For help on the BitBake layer management tool, use the following
599command:
600::
601
602 $ bitbake-layers --help
603 NOTE: Starting bitbake server...
604 usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
605
606 BitBake layers utility
607
608 optional arguments:
609 -d, --debug Enable debug output
610 -q, --quiet Print only errors
611 -F, --force Force add without recipe parse verification
612 --color COLOR Colorize output (where COLOR is auto, always, never)
613 -h, --help show this help message and exit
614
615 subcommands:
616 <subcommand>
617 layerindex-fetch Fetches a layer from a layer index along with its
618 dependent layers, and adds them to conf/bblayers.conf.
619 layerindex-show-depends
620 Find layer dependencies from layer index.
621 add-layer Add one or more layers to bblayers.conf.
622 remove-layer Remove one or more layers from bblayers.conf.
623 flatten flatten layer configuration into a separate output
624 directory.
625 show-layers show current configured layers.
626 show-overlayed list overlayed recipes (where the same recipe exists
627 in another layer)
628 show-recipes list available recipes, showing the layer they are
629 provided by
630 show-appends list bbappend files and recipe files they apply to
631 show-cross-depends Show dependencies between recipes that cross layer
632 boundaries.
633 create-layer Create a basic layer
634
635 Use bitbake-layers <subcommand> --help to get help on a specific command
636
637The following list describes the available commands:
638
639- ``help:`` Displays general help or help on a specified command.
640
641- ``show-layers:`` Shows the current configured layers.
642
643- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
644 when a recipe with the same name exists in another layer that has a
645 higher layer priority.
646
647- ``show-recipes:`` Lists available recipes and the layers that
648 provide them.
649
650- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
651 which they apply.
652
653- ``show-cross-depends:`` Lists dependency relationships between
654 recipes that cross layer boundaries.
655
656- ``add-layer:`` Adds a layer to ``bblayers.conf``.
657
658- ``remove-layer:`` Removes a layer from ``bblayers.conf``
659
660- ``flatten:`` Flattens the layer configuration into a separate
661 output directory. Flattening your layer configuration builds a
662 "flattened" directory that contains the contents of all layers, with
663 any overlayed recipes removed and any ``.bbappend`` files appended to
664 the corresponding recipes. You might have to perform some manual
665 cleanup of the flattened layer as follows:
666
667 - Non-recipe files (such as patches) are overwritten. The flatten
668 command shows a warning for these files.
669
670 - Anything beyond the normal layer setup has been added to the
671 ``layer.conf`` file. Only the lowest priority layer's
672 ``layer.conf`` is used.
673
674 - Overridden and appended items from ``.bbappend`` files need to be
675 cleaned up. The contents of each ``.bbappend`` end up in the
676 flattened recipe. However, if there are appended or changed
677 variable values, you need to tidy these up yourself. Consider the
678 following example. Here, the ``bitbake-layers`` command adds the
679 line ``#### bbappended ...`` so that you know where the following
680 lines originate:
681 ::
682
683 ...
684 DESCRIPTION = "A useful utility"
685 ...
686 EXTRA_OECONF = "--enable-something"
687 ...
688
689 #### bbappended from meta-anotherlayer ####
690
691 DESCRIPTION = "Customized utility"
692 EXTRA_OECONF += "--enable-somethingelse"
693
694
695 Ideally, you would tidy up these utilities as follows:
696 ::
697
698 ...
699 DESCRIPTION = "Customized utility"
700 ...
701 EXTRA_OECONF = "--enable-something --enable-somethingelse"
702 ...
703
704- ``layerindex-fetch``: Fetches a layer from a layer index, along
705 with its dependent layers, and adds the layers to the
706 ``conf/bblayers.conf`` file.
707
708- ``layerindex-show-depends``: Finds layer dependencies from the
709 layer index.
710
711- ``create-layer``: Creates a basic layer.
712
713Creating a General Layer Using the ``bitbake-layers`` Script
714------------------------------------------------------------
715
716The ``bitbake-layers`` script with the ``create-layer`` subcommand
717simplifies creating a new general layer.
718
719.. note::
720
721 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
722 section in the Yocto
723 Project Board Specific (BSP) Developer's Guide.
724
725 - In order to use a layer with the OpenEmbedded build system, you
726 need to add the layer to your ``bblayers.conf`` configuration
727 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
728 section for more information.
729
730The default mode of the script's operation with this subcommand is to
731create a layer with the following:
732
733- A layer priority of 6.
734
735- A ``conf`` subdirectory that contains a ``layer.conf`` file.
736
737- A ``recipes-example`` subdirectory that contains a further
738 subdirectory named ``example``, which contains an ``example.bb``
739 recipe file.
740
741- A ``COPYING.MIT``, which is the license statement for the layer. The
742 script assumes you want to use the MIT license, which is typical for
743 most layers, for the contents of the layer itself.
744
745- A ``README`` file, which is a file describing the contents of your
746 new layer.
747
748In its simplest form, you can use the following command form to create a
749layer. The command creates a layer whose name corresponds to
750"your_layer_name" in the current directory:
751::
752
753 $ bitbake-layers create-layer your_layer_name
754
755As an example, the following command creates a layer named ``meta-scottrif``
756in your home directory:
757::
758
759 $ cd /usr/home
760 $ bitbake-layers create-layer meta-scottrif
761 NOTE: Starting bitbake server...
762 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
763
764If you want to set the priority of the layer to other than the default
765value of "6", you can either use the ``--priority`` option or you
766can edit the
767:term:`BBFILE_PRIORITY` value
768in the ``conf/layer.conf`` after the script creates it. Furthermore, if
769you want to give the example recipe file some name other than the
770default, you can use the ``--example-recipe-name`` option.
771
772The easiest way to see how the ``bitbake-layers create-layer`` command
773works is to experiment with the script. You can also read the usage
774information by entering the following:
775::
776
777 $ bitbake-layers create-layer --help
778 NOTE: Starting bitbake server...
779 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
780 [--example-recipe-name EXAMPLERECIPE]
781 layerdir
782
783 Create a basic layer
784
785 positional arguments:
786 layerdir Layer directory to create
787
788 optional arguments:
789 -h, --help show this help message and exit
790 --priority PRIORITY, -p PRIORITY
791 Layer directory to create
792 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
793 Filename of the example recipe
794
795Adding a Layer Using the ``bitbake-layers`` Script
796--------------------------------------------------
797
798Once you create your general layer, you must add it to your
799``bblayers.conf`` file. Adding the layer to this configuration file
800makes the OpenEmbedded build system aware of your layer so that it can
801search it for metadata.
802
803Add your layer by using the ``bitbake-layers add-layer`` command:
804::
805
806 $ bitbake-layers add-layer your_layer_name
807
808Here is an example that adds a
809layer named ``meta-scottrif`` to the configuration file. Following the
810command that adds the layer is another ``bitbake-layers`` command that
811shows the layers that are in your ``bblayers.conf`` file:
812::
813
814 $ bitbake-layers add-layer meta-scottrif
815 NOTE: Starting bitbake server...
816 Parsing recipes: 100% |##########################################################| Time: 0:00:49
817 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
818 $ bitbake-layers show-layers
819 NOTE: Starting bitbake server...
820 layer path priority
821 ==========================================================================
822 meta /home/scottrif/poky/meta 5
823 meta-poky /home/scottrif/poky/meta-poky 5
824 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
825 workspace /home/scottrif/poky/build/workspace 99
826 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
827
828
829Adding the layer to this file
830enables the build system to locate the layer during the build.
831
832.. note::
833
834 During a build, the OpenEmbedded build system looks in the layers
835 from the top of the list down to the bottom in that order.
836
837Customizing Images
838==================
839
840You can customize images to satisfy particular requirements. This
841section describes several methods and provides guidelines for each.
842
843Customizing Images Using ``local.conf``
844---------------------------------------
845
846Probably the easiest way to customize an image is to add a package by
847way of the ``local.conf`` configuration file. Because it is limited to
848local use, this method generally only allows you to add packages and is
849not as flexible as creating your own customized image. When you add
850packages using local variables this way, you need to realize that these
851variable changes are in effect for every build and consequently affect
852all images, which might not be what you require.
853
854To add a package to your image using the local configuration file, use
855the ``IMAGE_INSTALL`` variable with the ``_append`` operator:
856::
857
858 IMAGE_INSTALL_append = " strace"
859
860Use of the syntax is important -
861specifically, the space between the quote and the package name, which is
862``strace`` in this example. This space is required since the ``_append``
863operator does not add the space.
864
865Furthermore, you must use ``_append`` instead of the ``+=`` operator if
866you want to avoid ordering issues. The reason for this is because doing
867so unconditionally appends to the variable and avoids ordering problems
868due to the variable being set in image recipes and ``.bbclass`` files
869with operators like ``?=``. Using ``_append`` ensures the operation
870takes effect.
871
872As shown in its simplest use, ``IMAGE_INSTALL_append`` affects all
873images. It is possible to extend the syntax so that the variable applies
874to a specific image only. Here is an example:
875::
876
877 IMAGE_INSTALL_append_pn-core-image-minimal = " strace"
878
879This example adds ``strace`` to the ``core-image-minimal`` image only.
880
881You can add packages using a similar approach through the
882``CORE_IMAGE_EXTRA_INSTALL`` variable. If you use this variable, only
883``core-image-*`` images are affected.
884
885Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
886-------------------------------------------------------------------------------
887
888Another method for customizing your image is to enable or disable
889high-level image features by using the
890:term:`IMAGE_FEATURES` and
891:term:`EXTRA_IMAGE_FEATURES`
892variables. Although the functions for both variables are nearly
893equivalent, best practices dictate using ``IMAGE_FEATURES`` from within
894a recipe and using ``EXTRA_IMAGE_FEATURES`` from within your
895``local.conf`` file, which is found in the
896:term:`Build Directory`.
897
898To understand how these features work, the best reference is
899``meta/classes/core-image.bbclass``. This class lists out the available
900``IMAGE_FEATURES`` of which most map to package groups while some, such
901as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
902configuration settings.
903
904In summary, the file looks at the contents of the ``IMAGE_FEATURES``
905variable and then maps or configures the feature accordingly. Based on
906this information, the build system automatically adds the appropriate
907packages or configurations to the
908:term:`IMAGE_INSTALL` variable.
909Effectively, you are enabling extra features by extending the class or
910creating a custom class for use with specialized image ``.bb`` files.
911
912Use the ``EXTRA_IMAGE_FEATURES`` variable from within your local
913configuration file. Using a separate area from which to enable features
914with this variable helps you avoid overwriting the features in the image
915recipe that are enabled with ``IMAGE_FEATURES``. The value of
916``EXTRA_IMAGE_FEATURES`` is added to ``IMAGE_FEATURES`` within
917``meta/conf/bitbake.conf``.
918
919To illustrate how you can use these variables to modify your image,
920consider an example that selects the SSH server. The Yocto Project ships
921with two SSH servers you can use with your images: Dropbear and OpenSSH.
922Dropbear is a minimal SSH server appropriate for resource-constrained
923environments, while OpenSSH is a well-known standard SSH server
924implementation. By default, the ``core-image-sato`` image is configured
925to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
926images both include OpenSSH. The ``core-image-minimal`` image does not
927contain an SSH server.
928
929You can customize your image and change these defaults. Edit the
930``IMAGE_FEATURES`` variable in your recipe or use the
931``EXTRA_IMAGE_FEATURES`` in your ``local.conf`` file so that it
932configures the image you are working with to include
933``ssh-server-dropbear`` or ``ssh-server-openssh``.
934
935.. note::
936
937 See the ":ref:`ref-manual/features:image features`" section in the Yocto
938 Project Reference Manual for a complete list of image features that ship
939 with the Yocto Project.
940
941Customizing Images Using Custom .bb Files
942-----------------------------------------
943
944You can also customize an image by creating a custom recipe that defines
945additional software as part of the image. The following example shows
946the form for the two lines you need:
947::
948
949 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
950 inherit core-image
951
952Defining the software using a custom recipe gives you total control over
953the contents of the image. It is important to use the correct names of
954packages in the ``IMAGE_INSTALL`` variable. You must use the
955OpenEmbedded notation and not the Debian notation for the names (e.g.
956``glibc-dev`` instead of ``libc6-dev``).
957
958The other method for creating a custom image is to base it on an
959existing image. For example, if you want to create an image based on
960``core-image-sato`` but add the additional package ``strace`` to the
961image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
962``.bb`` and add the following line to the end of the copy:
963::
964
965 IMAGE_INSTALL += "strace"
966
967Customizing Images Using Custom Package Groups
968----------------------------------------------
969
970For complex custom images, the best approach for customizing an image is
971to create a custom package group recipe that is used to build the image
972or images. A good example of a package group recipe is
973``meta/recipes-core/packagegroups/packagegroup-base.bb``.
974
975If you examine that recipe, you see that the ``PACKAGES`` variable lists
976the package group packages to produce. The ``inherit packagegroup``
977statement sets appropriate default values and automatically adds
978``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
979package specified in the ``PACKAGES`` statement.
980
981.. note::
982
983 The ``inherit packagegroup`` line should be located near the top of the
984 recipe, certainly before the ``PACKAGES`` statement.
985
986For each package you specify in ``PACKAGES``, you can use ``RDEPENDS``
987and ``RRECOMMENDS`` entries to provide a list of packages the parent
988task package should contain. You can see examples of these further down
989in the ``packagegroup-base.bb`` recipe.
990
991Here is a short, fabricated example showing the same basic pieces for a
992hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
993the variable ``PN`` is the standard way to abbreviate the reference to
994the full packagegroup name ``packagegroup-custom``:
995::
996
997 DESCRIPTION = "My Custom Package Groups"
998
999 inherit packagegroup
1000
1001 PACKAGES = "\
1002 ${PN}-apps \
1003 ${PN}-tools \
1004 "
1005
1006 RDEPENDS_${PN}-apps = "\
1007 dropbear \
1008 portmap \
1009 psplash"
1010
1011 RDEPENDS_${PN}-tools = "\
1012 oprofile \
1013 oprofileui-server \
1014 lttng-tools"
1015
1016 RRECOMMENDS_${PN}-tools = "\
1017 kernel-module-oprofile"
1018
1019In the previous example, two package group packages are created with
1020their dependencies and their recommended package dependencies listed:
1021``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1022build an image using these package group packages, you need to add
1023``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
1024``IMAGE_INSTALL``. For other forms of image dependencies see the other
1025areas of this section.
1026
1027Customizing an Image Hostname
1028-----------------------------
1029
1030By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1031is the same as the machine name. For example, if
1032:term:`MACHINE` equals "qemux86", the
1033configured hostname written to ``/etc/hostname`` is "qemux86".
1034
1035You can customize this name by altering the value of the "hostname"
1036variable in the ``base-files`` recipe using either an append file or a
1037configuration file. Use the following in an append file:
1038::
1039
1040 hostname = "myhostname"
1041
1042Use the following in a configuration file:
1043::
1044
1045 hostname_pn-base-files = "myhostname"
1046
1047Changing the default value of the variable "hostname" can be useful in
1048certain situations. For example, suppose you need to do extensive
1049testing on an image and you would like to easily identify the image
1050under test from existing images with typical default hostnames. In this
1051situation, you could change the default hostname to "testme", which
1052results in all the images using the name "testme". Once testing is
1053complete and you do not need to rebuild the image for test any longer,
1054you can easily reset the default hostname.
1055
1056Another point of interest is that if you unset the variable, the image
1057will have no default hostname in the filesystem. Here is an example that
1058unsets the variable in a configuration file:
1059::
1060
1061 hostname_pn-base-files = ""
1062
1063Having no default hostname in the filesystem is suitable for
1064environments that use dynamic hostnames such as virtual machines.
1065
1066Writing a New Recipe
1067====================
1068
1069Recipes (``.bb`` files) are fundamental components in the Yocto Project
1070environment. Each software component built by the OpenEmbedded build
1071system requires a recipe to define the component. This section describes
1072how to create, write, and test a new recipe.
1073
1074.. note::
1075
1076 For information on variables that are useful for recipes and for
1077 information about recipe naming issues, see the
1078 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
1079 Reference Manual.
1080
1081Overview
1082--------
1083
1084The following figure shows the basic process for creating a new recipe.
1085The remainder of the section provides details for the steps.
1086
1087.. image:: figures/recipe-workflow.png
1088 :align: center
1089
1090Locate or Automatically Create a Base Recipe
1091--------------------------------------------
1092
1093You can always write a recipe from scratch. However, three choices exist
1094that can help you quickly get a start on a new recipe:
1095
1096- ``devtool add``: A command that assists in creating a recipe and an
1097 environment conducive to development.
1098
1099- ``recipetool create``: A command provided by the Yocto Project that
1100 automates creation of a base recipe based on the source files.
1101
1102- *Existing Recipes:* Location and modification of an existing recipe
1103 that is similar in function to the recipe you need.
1104
1105.. note::
1106
1107 For information on recipe syntax, see the
1108 ":ref:`dev-manual/common-tasks:recipe syntax`" section.
1109
1110Creating the Base Recipe Using ``devtool add``
1111~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1112
1113The ``devtool add`` command uses the same logic for auto-creating the
1114recipe as ``recipetool create``, which is listed below. Additionally,
1115however, ``devtool add`` sets up an environment that makes it easy for
1116you to patch the source and to make changes to the recipe as is often
1117necessary when adding a recipe to build a new piece of software to be
1118included in a build.
1119
1120You can find a complete description of the ``devtool add`` command in
1121the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
1122in the Yocto Project Application Development and the Extensible Software
1123Development Kit (eSDK) manual.
1124
1125Creating the Base Recipe Using ``recipetool create``
1126~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1127
1128``recipetool create`` automates creation of a base recipe given a set of
1129source code files. As long as you can extract or point to the source
1130files, the tool will construct a recipe and automatically configure all
1131pre-build information into the recipe. For example, suppose you have an
1132application that builds using Autotools. Creating the base recipe using
1133``recipetool`` results in a recipe that has the pre-build dependencies,
1134license requirements, and checksums configured.
1135
1136To run the tool, you just need to be in your
1137:term:`Build Directory` and have sourced the
1138build environment setup script (i.e.
1139:ref:`structure-core-script`).
1140To get help on the tool, use the following command:
1141::
1142
1143 $ recipetool -h
1144 NOTE: Starting bitbake server...
1145 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1146
1147 OpenEmbedded recipe tool
1148
1149 options:
1150 -d, --debug Enable debug output
1151 -q, --quiet Print only errors
1152 --color COLOR Colorize output (where COLOR is auto, always, never)
1153 -h, --help show this help message and exit
1154
1155 subcommands:
1156 create Create a new recipe
1157 newappend Create a bbappend for the specified target in the specified
1158 layer
1159 setvar Set a variable within a recipe
1160 appendfile Create/update a bbappend to replace a target file
1161 appendsrcfiles Create/update a bbappend to add or replace source files
1162 appendsrcfile Create/update a bbappend to add or replace a source file
1163 Use recipetool <subcommand> --help to get help on a specific command
1164
1165Running ``recipetool create -o OUTFILE`` creates the base recipe and
1166locates it properly in the layer that contains your source files.
1167Following are some syntax examples:
1168
1169 - Use this syntax to generate a recipe based on source. Once generated,
1170 the recipe resides in the existing source code layer:
1171 ::
1172
1173 recipetool create -o OUTFILE source
1174
1175 - Use this syntax to generate a recipe using code that
1176 you extract from source. The extracted code is placed in its own layer
1177 defined by ``EXTERNALSRC``.
1178 ::
1179
1180 recipetool create -o OUTFILE -x EXTERNALSRC source
1181
1182 - Use this syntax to generate a recipe based on source. The options
1183 direct ``recipetool`` to generate debugging information. Once generated,
1184 the recipe resides in the existing source code layer:
1185 ::
1186
1187 recipetool create -d -o OUTFILE source
1188
1189Locating and Using a Similar Recipe
1190~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1191
1192Before writing a recipe from scratch, it is often useful to discover
1193whether someone else has already written one that meets (or comes close
1194to meeting) your needs. The Yocto Project and OpenEmbedded communities
1195maintain many recipes that might be candidates for what you are doing.
1196You can find a good central index of these recipes in the
1197:oe_layerindex:`OpenEmbedded Layer Index <>`.
1198
1199Working from an existing recipe or a skeleton recipe is the best way to
1200get started. Here are some points on both methods:
1201
1202- *Locate and modify a recipe that is close to what you want to do:*
1203 This method works when you are familiar with the current recipe
1204 space. The method does not work so well for those new to the Yocto
1205 Project or writing recipes.
1206
1207 Some risks associated with this method are using a recipe that has
1208 areas totally unrelated to what you are trying to accomplish with
1209 your recipe, not recognizing areas of the recipe that you might have
1210 to add from scratch, and so forth. All these risks stem from
1211 unfamiliarity with the existing recipe space.
1212
1213- *Use and modify the following skeleton recipe:* If for some reason
1214 you do not want to use ``recipetool`` and you cannot find an existing
1215 recipe that is close to meeting your needs, you can use the following
1216 structure to provide the fundamental areas of a new recipe.
1217 ::
1218
1219 DESCRIPTION = ""
1220 HOMEPAGE = ""
1221 LICENSE = ""
1222 SECTION = ""
1223 DEPENDS = ""
1224 LIC_FILES_CHKSUM = ""
1225
1226 SRC_URI = ""
1227
1228Storing and Naming the Recipe
1229-----------------------------
1230
1231Once you have your base recipe, you should put it in your own layer and
1232name it appropriately. Locating it correctly ensures that the
1233OpenEmbedded build system can find it when you use BitBake to process
1234the recipe.
1235
1236- *Storing Your Recipe:* The OpenEmbedded build system locates your
1237 recipe through the layer's ``conf/layer.conf`` file and the
1238 :term:`BBFILES` variable. This
1239 variable sets up a path from which the build system can locate
1240 recipes. Here is the typical use:
1241 ::
1242
1243 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1244 ${LAYERDIR}/recipes-*/*/*.bbappend"
1245
1246 Consequently, you need to be sure you locate your new recipe inside
1247 your layer such that it can be found.
1248
1249 You can find more information on how layers are structured in the
1250 "`Understanding and Creating
1251 Layers <#understanding-and-creating-layers>`__" section.
1252
1253- *Naming Your Recipe:* When you name your recipe, you need to follow
1254 this naming convention:
1255 ::
1256
1257 basename_version.bb
1258
1259 Use lower-cased characters and do not include the reserved suffixes
1260 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1261 them as part of your recipe name unless the string applies). Here are some
1262 examples:
1263
1264 .. code-block:: none
1265
1266 cups_1.7.0.bb
1267 gawk_4.0.2.bb
1268 irssi_0.8.16-rc1.bb
1269
1270Running a Build on the Recipe
1271-----------------------------
1272
1273Creating a new recipe is usually an iterative process that requires
1274using BitBake to process the recipe multiple times in order to
1275progressively discover and add information to the recipe file.
1276
1277Assuming you have sourced the build environment setup script (i.e.
1278:ref:`structure-core-script`) and you are in
1279the :term:`Build Directory`, use
1280BitBake to process your recipe. All you need to provide is the
1281``basename`` of the recipe as described in the previous section:
1282::
1283
1284 $ bitbake basename
1285
1286During the build, the OpenEmbedded build system creates a temporary work
1287directory for each recipe
1288(``${``\ :term:`WORKDIR`\ ``}``)
1289where it keeps extracted source files, log files, intermediate
1290compilation and packaging files, and so forth.
1291
1292The path to the per-recipe temporary work directory depends on the
1293context in which it is being built. The quickest way to find this path
1294is to have BitBake return it by running the following:
1295::
1296
1297 $ bitbake -e basename | grep ^WORKDIR=
1298
1299As an example, assume a Source Directory
1300top-level folder named ``poky``, a default Build Directory at
1301``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1302Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1303case, the work directory the build system uses to build the package
1304would be as follows:
1305::
1306
1307 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1308
1309Inside this directory you can find sub-directories such as ``image``,
1310``packages-split``, and ``temp``. After the build, you can examine these
1311to determine how well the build went.
1312
1313.. note::
1314
1315 You can find log files for each task in the recipe's ``temp``
1316 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1317 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1318 ``log.do_fetch``, and ``log.do_compile``).
1319
1320You can find more information about the build process in
1321":doc:`/overview-manual/development-environment`"
1322chapter of the Yocto Project Overview and Concepts Manual.
1323
1324Fetching Code
1325-------------
1326
1327The first thing your recipe must do is specify how to fetch the source
1328files. Fetching is controlled mainly through the
1329:term:`SRC_URI` variable. Your recipe
1330must have a ``SRC_URI`` variable that points to where the source is
1331located. For a graphical representation of source locations, see the
1332":ref:`overview-manual/concepts:sources`" section in
1333the Yocto Project Overview and Concepts Manual.
1334
1335The :ref:`ref-tasks-fetch` task uses
1336the prefix of each entry in the ``SRC_URI`` variable value to determine
1337which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
1338source files. It is the ``SRC_URI`` variable that triggers the fetcher.
1339The :ref:`ref-tasks-patch` task uses
1340the variable after source is fetched to apply patches. The OpenEmbedded
1341build system uses
1342:term:`FILESOVERRIDES` for
1343scanning directory locations for local files in ``SRC_URI``.
1344
1345The ``SRC_URI`` variable in your recipe must define each unique location
1346for your source files. It is good practice to not hard-code version
1347numbers in a URL used in ``SRC_URI``. Rather than hard-code these
1348values, use ``${``\ :term:`PV`\ ``}``,
1349which causes the fetch process to use the version specified in the
1350recipe filename. Specifying the version in this manner means that
1351upgrading the recipe to a future version is as simple as renaming the
1352recipe to match the new version.
1353
1354Here is a simple example from the
1355``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1356comes from a single tarball. Notice the use of the
1357:term:`PV` variable:
1358::
1359
1360 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
1361
1362Files mentioned in ``SRC_URI`` whose names end in a typical archive
1363extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1364forth), are automatically extracted during the
1365:ref:`ref-tasks-unpack` task. For
1366another example that specifies these types of files, see the
1367"`Autotooled Package <#new-recipe-autotooled-package>`__" section.
1368
1369Another way of specifying source is from an SCM. For Git repositories,
1370you must specify :term:`SRCREV` and
1371you should specify :term:`PV` to include
1372the revision with :term:`SRCPV`. Here
1373is an example from the recipe
1374``meta/recipes-kernel/blktrace/blktrace_git.bb``:
1375::
1376
1377 SRCREV = "d6918c8832793b4205ed3bfede78c2f915c23385"
1378
1379 PR = "r6"
1380 PV = "1.0.5+git${SRCPV}"
1381
1382 SRC_URI = "git://git.kernel.dk/blktrace.git \
1383 file://ldflags.patch"
1384
1385If your ``SRC_URI`` statement includes URLs pointing to individual files
1386fetched from a remote server other than a version control system,
1387BitBake attempts to verify the files against checksums defined in your
1388recipe to ensure they have not been tampered with or otherwise modified
1389since the recipe was written. Two checksums are used:
1390``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1391
1392If your ``SRC_URI`` variable points to more than a single URL (excluding
1393SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1394each URL. For these cases, you provide a name for each URL as part of
1395the ``SRC_URI`` and then reference that name in the subsequent checksum
1396statements. Here is an example combining lines from the files
1397``git.inc`` and ``git_2.24.1.bb``:
1398::
1399
1400 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1401 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1402
1403 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1404 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1405 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1406 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1407
1408Proper values for ``md5`` and ``sha256`` checksums might be available
1409with other signatures on the download page for the upstream source (e.g.
1410``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1411OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1412you should verify all the signatures you find by hand.
1413
1414If no ``SRC_URI`` checksums are specified when you attempt to build the
1415recipe, or you provide an incorrect checksum, the build will produce an
1416error for each missing or incorrect checksum. As part of the error
1417message, the build system provides the checksum string corresponding to
1418the fetched file. Once you have the correct checksums, you can copy and
1419paste them into your recipe and then run the build again to continue.
1420
1421.. note::
1422
1423 As mentioned, if the upstream source provides signatures for
1424 verifying the downloaded source code, you should verify those
1425 manually before setting the checksum values in the recipe and
1426 continuing with the build.
1427
1428This final example is a bit more complicated and is from the
1429``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
1430example's ``SRC_URI`` statement identifies multiple files as the source
1431files for the recipe: a tarball, a patch file, a desktop file, and an
1432icon.
1433::
1434
1435 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1436 file://xwc.patch \
1437 file://rxvt.desktop \
1438 file://rxvt.png"
1439
1440When you specify local files using the ``file://`` URI protocol, the
1441build system fetches files from the local machine. The path is relative
1442to the :term:`FILESPATH` variable
1443and searches specific directories in a certain order:
1444``${``\ :term:`BP`\ ``}``,
1445``${``\ :term:`BPN`\ ``}``, and
1446``files``. The directories are assumed to be subdirectories of the
1447directory in which the recipe or append file resides. For another
1448example that specifies these types of files, see the "`Single .c File
1449Package (Hello
1450World!) <#new-recipe-single-c-file-package-hello-world>`__" section.
1451
1452The previous example also specifies a patch file. Patch files are files
1453whose names usually end in ``.patch`` or ``.diff`` but can end with
1454compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1455The build system automatically applies patches as described in the
1456"`Patching Code <#new-recipe-patching-code>`__" section.
1457
1458Unpacking Code
1459--------------
1460
1461During the build, the
1462:ref:`ref-tasks-unpack` task unpacks
1463the source with ``${``\ :term:`S`\ ``}``
1464pointing to where it is unpacked.
1465
1466If you are fetching your source files from an upstream source archived
1467tarball and the tarball's internal structure matches the common
1468convention of a top-level subdirectory named
1469``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
1470then you do not need to set ``S``. However, if ``SRC_URI`` specifies to
1471fetch source from an archive that does not use this convention, or from
1472an SCM like Git or Subversion, your recipe needs to define ``S``.
1473
1474If processing your recipe using BitBake successfully unpacks the source
1475files, you need to be sure that the directory pointed to by ``${S}``
1476matches the structure of the source.
1477
1478Patching Code
1479-------------
1480
1481Sometimes it is necessary to patch code after it has been fetched. Any
1482files mentioned in ``SRC_URI`` whose names end in ``.patch`` or
1483``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1484treated as patches. The
1485:ref:`ref-tasks-patch` task
1486automatically applies these patches.
1487
1488The build system should be able to apply patches with the "-p1" option
1489(i.e. one directory level in the path will be stripped off). If your
1490patch needs to have more directory levels stripped off, specify the
1491number of levels using the "striplevel" option in the ``SRC_URI`` entry
1492for the patch. Alternatively, if your patch needs to be applied in a
1493specific subdirectory that is not specified in the patch file, use the
1494"patchdir" option in the entry.
1495
1496As with all local files referenced in
1497:term:`SRC_URI` using ``file://``,
1498you should place patch files in a directory next to the recipe either
1499named the same as the base name of the recipe
1500(:term:`BP` and
1501:term:`BPN`) or "files".
1502
1503Licensing
1504---------
1505
1506Your recipe needs to have both the
1507:term:`LICENSE` and
1508:term:`LIC_FILES_CHKSUM`
1509variables:
1510
1511- ``LICENSE``: This variable specifies the license for the software.
1512 If you do not know the license under which the software you are
1513 building is distributed, you should go to the source code and look
1514 for that information. Typical files containing this information
1515 include ``COPYING``, ``LICENSE``, and ``README`` files. You could
1516 also find the information near the top of a source file. For example,
1517 given a piece of software licensed under the GNU General Public
1518 License version 2, you would set ``LICENSE`` as follows:
1519 ::
1520
1521 LICENSE = "GPLv2"
1522
1523 The licenses you specify within ``LICENSE`` can have any name as long
1524 as you do not use spaces, since spaces are used as separators between
1525 license names. For standard licenses, use the names of the files in
1526 ``meta/files/common-licenses/`` or the ``SPDXLICENSEMAP`` flag names
1527 defined in ``meta/conf/licenses.conf``.
1528
1529- ``LIC_FILES_CHKSUM``: The OpenEmbedded build system uses this
1530 variable to make sure the license text has not changed. If it has,
1531 the build produces an error and it affords you the chance to figure
1532 it out and correct the problem.
1533
1534 You need to specify all applicable licensing files for the software.
1535 At the end of the configuration step, the build process will compare
1536 the checksums of the files to be sure the text has not changed. Any
1537 differences result in an error with the message containing the
1538 current checksum. For more explanation and examples of how to set the
1539 ``LIC_FILES_CHKSUM`` variable, see the
1540 ":ref:`dev-manual/common-tasks:tracking license changes`" section.
1541
1542 To determine the correct checksum string, you can list the
1543 appropriate files in the ``LIC_FILES_CHKSUM`` variable with incorrect
1544 md5 strings, attempt to build the software, and then note the
1545 resulting error messages that will report the correct md5 strings.
1546 See the "`Fetching Code <#new-recipe-fetching-code>`__" section for
1547 additional information.
1548
1549 Here is an example that assumes the software has a ``COPYING`` file:
1550 ::
1551
1552 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1553
1554 When you try to build the
1555 software, the build system will produce an error and give you the
1556 correct string that you can substitute into the recipe file for a
1557 subsequent build.
1558
1559Dependencies
1560------------
1561
1562Most software packages have a short list of other packages that they
1563require, which are called dependencies. These dependencies fall into two
1564main categories: build-time dependencies, which are required when the
1565software is built; and runtime dependencies, which are required to be
1566installed on the target in order for the software to run.
1567
1568Within a recipe, you specify build-time dependencies using the
1569:term:`DEPENDS` variable. Although
1570nuances exist, items specified in ``DEPENDS`` should be names of other
1571recipes. It is important that you specify all build-time dependencies
1572explicitly.
1573
1574Another consideration is that configure scripts might automatically
1575check for optional dependencies and enable corresponding functionality
1576if those dependencies are found. If you wish to make a recipe that is
1577more generally useful (e.g. publish the recipe in a layer for others to
1578use), instead of hard-disabling the functionality, you can use the
1579:term:`PACKAGECONFIG` variable to allow functionality and the
1580corresponding dependencies to be enabled and disabled easily by other
1581users of the recipe.
1582
1583Similar to build-time dependencies, you specify runtime dependencies
1584through a variable -
1585:term:`RDEPENDS`, which is
1586package-specific. All variables that are package-specific need to have
1587the name of the package added to the end as an override. Since the main
1588package for a recipe has the same name as the recipe, and the recipe's
1589name can be found through the
1590``${``\ :term:`PN`\ ``}`` variable, then
1591you specify the dependencies for the main package by setting
1592``RDEPENDS_${PN}``. If the package were named ``${PN}-tools``, then you
1593would set ``RDEPENDS_${PN}-tools``, and so forth.
1594
1595Some runtime dependencies will be set automatically at packaging time.
1596These dependencies include any shared library dependencies (i.e. if a
1597package "example" contains "libexample" and another package "mypackage"
1598contains a binary that links to "libexample" then the OpenEmbedded build
1599system will automatically add a runtime dependency to "mypackage" on
1600"example"). See the
1601":ref:`overview-manual/concepts:automatically added runtime dependencies`"
1602section in the Yocto Project Overview and Concepts Manual for further
1603details.
1604
1605Configuring the Recipe
1606----------------------
1607
1608Most software provides some means of setting build-time configuration
1609options before compilation. Typically, setting these options is
1610accomplished by running a configure script with options, or by modifying
1611a build configuration file.
1612
1613.. note::
1614
1615 As of Yocto Project Release 1.7, some of the core recipes that
1616 package binary configuration scripts now disable the scripts due to
1617 the scripts previously requiring error-prone path substitution. The
1618 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1619 robust. You can find a list of the ``*-config`` scripts that are disabled
1620 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1621 in the Yocto Project Reference Manual.
1622
1623A major part of build-time configuration is about checking for
1624build-time dependencies and possibly enabling optional functionality as
1625a result. You need to specify any build-time dependencies for the
1626software you are building in your recipe's
1627:term:`DEPENDS` value, in terms of
1628other recipes that satisfy those dependencies. You can often find
1629build-time or runtime dependencies described in the software's
1630documentation.
1631
1632The following list provides configuration items of note based on how
1633your software is built:
1634
1635- *Autotools:* If your source files have a ``configure.ac`` file, then
1636 your software is built using Autotools. If this is the case, you just
1637 need to worry about modifying the configuration.
1638
1639 When using Autotools, your recipe needs to inherit the
1640 :ref:`autotools <ref-classes-autotools>` class
1641 and your recipe does not have to contain a
1642 :ref:`ref-tasks-configure` task.
1643 However, you might still want to make some adjustments. For example,
1644 you can set
1645 :term:`EXTRA_OECONF` or
1646 :term:`PACKAGECONFIG_CONFARGS`
1647 to pass any needed configure options that are specific to the recipe.
1648
1649- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1650 your software is built using CMake. If this is the case, you just
1651 need to worry about modifying the configuration.
1652
1653 When you use CMake, your recipe needs to inherit the
1654 :ref:`cmake <ref-classes-cmake>` class and your
1655 recipe does not have to contain a
1656 :ref:`ref-tasks-configure` task.
1657 You can make some adjustments by setting
1658 :term:`EXTRA_OECMAKE` to
1659 pass any needed configure options that are specific to the recipe.
1660
1661 .. note::
1662
1663 If you need to install one or more custom CMake toolchain files
1664 that are supplied by the application you are building, install the
1665 files to ``${D}${datadir}/cmake/Modules`` during ``do_install``.
1666
1667- *Other:* If your source files do not have a ``configure.ac`` or
1668 ``CMakeLists.txt`` file, then your software is built using some
1669 method other than Autotools or CMake. If this is the case, you
1670 normally need to provide a
1671 :ref:`ref-tasks-configure` task
1672 in your recipe unless, of course, there is nothing to configure.
1673
1674 Even if your software is not being built by Autotools or CMake, you
1675 still might not need to deal with any configuration issues. You need
1676 to determine if configuration is even a required step. You might need
1677 to modify a Makefile or some configuration file used for the build to
1678 specify necessary build options. Or, perhaps you might need to run a
1679 provided, custom configure script with the appropriate options.
1680
1681 For the case involving a custom configure script, you would run
1682 ``./configure --help`` and look for the options you need to set.
1683
1684Once configuration succeeds, it is always good practice to look at the
1685``log.do_configure`` file to ensure that the appropriate options have
1686been enabled and no additional build-time dependencies need to be added
1687to ``DEPENDS``. For example, if the configure script reports that it
1688found something not mentioned in ``DEPENDS``, or that it did not find
1689something that it needed for some desired optional functionality, then
1690you would need to add those to ``DEPENDS``. Looking at the log might
1691also reveal items being checked for, enabled, or both that you do not
1692want, or items not being found that are in ``DEPENDS``, in which case
1693you would need to look at passing extra options to the configure script
1694as needed. For reference information on configure options specific to
1695the software you are building, you can consult the output of the
1696``./configure --help`` command within ``${S}`` or consult the software's
1697upstream documentation.
1698
1699Using Headers to Interface with Devices
1700---------------------------------------
1701
1702If your recipe builds an application that needs to communicate with some
1703device or needs an API into a custom kernel, you will need to provide
1704appropriate header files. Under no circumstances should you ever modify
1705the existing
1706``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1707These headers are used to build ``libc`` and must not be compromised
1708with custom or machine-specific header information. If you customize
1709``libc`` through modified headers all other applications that use
1710``libc`` thus become affected.
1711
1712.. note::
1713
1714 Never copy and customize the ``libc`` header file (i.e.
1715 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
1716
1717The correct way to interface to a device or custom kernel is to use a
1718separate package that provides the additional headers for the driver or
1719other unique interfaces. When doing so, your application also becomes
1720responsible for creating a dependency on that specific provider.
1721
1722Consider the following:
1723
1724- Never modify ``linux-libc-headers.inc``. Consider that file to be
1725 part of the ``libc`` system, and not something you use to access the
1726 kernel directly. You should access ``libc`` through specific ``libc``
1727 calls.
1728
1729- Applications that must talk directly to devices should either provide
1730 necessary headers themselves, or establish a dependency on a special
1731 headers package that is specific to that driver.
1732
1733For example, suppose you want to modify an existing header that adds I/O
1734control or network support. If the modifications are used by a small
1735number programs, providing a unique version of a header is easy and has
1736little impact. When doing so, bear in mind the guidelines in the
1737previous list.
1738
1739.. note::
1740
1741 If for some reason your changes need to modify the behavior of the ``libc``,
1742 and subsequently all other applications on the system, use a ``.bbappend``
1743 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1744 make the changes machine specific.
1745
1746Consider a case where your kernel is older and you need an older
1747``libc`` ABI. The headers installed by your recipe should still be a
1748standard mainline kernel, not your own custom one.
1749
1750When you use custom kernel headers you need to get them from
1751:term:`STAGING_KERNEL_DIR`,
1752which is the directory with kernel headers that are required to build
1753out-of-tree modules. Your recipe will also need the following:
1754::
1755
1756 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1757
1758Compilation
1759-----------
1760
1761During a build, the ``do_compile`` task happens after source is fetched,
1762unpacked, and configured. If the recipe passes through ``do_compile``
1763successfully, nothing needs to be done.
1764
1765However, if the compile step fails, you need to diagnose the failure.
1766Here are some common issues that cause failures.
1767
1768.. note::
1769
1770 For cases where improper paths are detected for configuration files
1771 or for when libraries/headers cannot be found, be sure you are using
1772 the more robust ``pkg-config``. See the note in section
1773 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information.
1774
1775- *Parallel build failures:* These failures manifest themselves as
1776 intermittent errors, or errors reporting that a file or directory
1777 that should be created by some other part of the build process could
1778 not be found. This type of failure can occur even if, upon
1779 inspection, the file or directory does exist after the build has
1780 failed, because that part of the build process happened in the wrong
1781 order.
1782
1783 To fix the problem, you need to either satisfy the missing dependency
1784 in the Makefile or whatever script produced the Makefile, or (as a
1785 workaround) set :term:`PARALLEL_MAKE` to an empty string:
1786 ::
1787
1788 PARALLEL_MAKE = ""
1789
1790 For information on parallel Makefile issues, see the "`Debugging
1791 Parallel Make Races <#debugging-parallel-make-races>`__" section.
1792
1793- *Improper host path usage:* This failure applies to recipes building
1794 for the target or ``nativesdk`` only. The failure occurs when the
1795 compilation process uses improper headers, libraries, or other files
1796 from the host system when cross-compiling for the target.
1797
1798 To fix the problem, examine the ``log.do_compile`` file to identify
1799 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1800 so forth) and then either add configure options, apply a patch, or do
1801 both.
1802
1803- *Failure to find required libraries/headers:* If a build-time
1804 dependency is missing because it has not been declared in
1805 :term:`DEPENDS`, or because the
1806 dependency exists but the path used by the build process to find the
1807 file is incorrect and the configure step did not detect it, the
1808 compilation process could fail. For either of these failures, the
1809 compilation process notes that files could not be found. In these
1810 cases, you need to go back and add additional options to the
1811 configure script as well as possibly add additional build-time
1812 dependencies to ``DEPENDS``.
1813
1814 Occasionally, it is necessary to apply a patch to the source to
1815 ensure the correct paths are used. If you need to specify paths to
1816 find files staged into the sysroot from other recipes, use the
1817 variables that the OpenEmbedded build system provides (e.g.
1818 ``STAGING_BINDIR``, ``STAGING_INCDIR``, ``STAGING_DATADIR``, and so
1819 forth).
1820
1821Installing
1822----------
1823
1824During ``do_install``, the task copies the built files along with their
1825hierarchy to locations that would mirror their locations on the target
1826device. The installation process copies files from the
1827``${``\ :term:`S`\ ``}``,
1828``${``\ :term:`B`\ ``}``, and
1829``${``\ :term:`WORKDIR`\ ``}``
1830directories to the ``${``\ :term:`D`\ ``}``
1831directory to create the structure as it should appear on the target
1832system.
1833
1834How your software is built affects what you must do to be sure your
1835software is installed correctly. The following list describes what you
1836must do for installation depending on the type of build system used by
1837the software being built:
1838
1839- *Autotools and CMake:* If the software your recipe is building uses
1840 Autotools or CMake, the OpenEmbedded build system understands how to
1841 install the software. Consequently, you do not have to have a
1842 ``do_install`` task as part of your recipe. You just need to make
1843 sure the install portion of the build completes with no issues.
1844 However, if you wish to install additional files not already being
1845 installed by ``make install``, you should do this using a
1846 ``do_install_append`` function using the install command as described
1847 in the "Manual" bulleted item later in this list.
1848
1849- *Other (using* ``make install``\ *)*: You need to define a ``do_install``
1850 function in your recipe. The function should call
1851 ``oe_runmake install`` and will likely need to pass in the
1852 destination directory as well. How you pass that path is dependent on
1853 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1854 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1855
1856 For an example recipe using ``make install``, see the
1857 "`Makefile-Based Package <#new-recipe-makefile-based-package>`__"
1858 section.
1859
1860- *Manual:* You need to define a ``do_install`` function in your
1861 recipe. The function must first use ``install -d`` to create the
1862 directories under
1863 ``${``\ :term:`D`\ ``}``. Once the
1864 directories exist, your function can use ``install`` to manually
1865 install the built software into the directories.
1866
1867 You can find more information on ``install`` at
1868 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
1869
1870For the scenarios that do not use Autotools or CMake, you need to track
1871the installation and diagnose and fix any issues until everything
1872installs correctly. You need to look in the default location of
1873``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1874installed correctly.
1875
1876.. note::
1877
1878 - During the installation process, you might need to modify some of
1879 the installed files to suit the target layout. For example, you
1880 might need to replace hard-coded paths in an initscript with
1881 values of variables provided by the build system, such as
1882 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1883 modifications during ``do_install``, be sure to modify the
1884 destination file after copying rather than before copying.
1885 Modifying after copying ensures that the build system can
1886 re-execute ``do_install`` if needed.
1887
1888 - ``oe_runmake install``, which can be run directly or can be run
1889 indirectly by the
1890 :ref:`autotools <ref-classes-autotools>` and
1891 :ref:`cmake <ref-classes-cmake>` classes,
1892 runs ``make install`` in parallel. Sometimes, a Makefile can have
1893 missing dependencies between targets that can result in race
1894 conditions. If you experience intermittent failures during
1895 ``do_install``, you might be able to work around them by disabling
1896 parallel Makefile installs by adding the following to the recipe:
1897 ::
1898
1899 PARALLEL_MAKEINST = ""
1900
1901 See :term:`PARALLEL_MAKEINST` for additional information.
1902
1903 - If you need to install one or more custom CMake toolchain files
1904 that are supplied by the application you are building, install the
1905 files to ``${D}${datadir}/cmake/Modules`` during
1906 :ref:`ref-tasks-install`.
1907
1908Enabling System Services
1909------------------------
1910
1911If you want to install a service, which is a process that usually starts
1912on boot and runs in the background, then you must include some
1913additional definitions in your recipe.
1914
1915If you are adding services and the service initialization script or the
1916service file itself is not installed, you must provide for that
1917installation in your recipe using a ``do_install_append`` function. If
1918your recipe already has a ``do_install`` function, update the function
1919near its end rather than adding an additional ``do_install_append``
1920function.
1921
1922When you create the installation for your services, you need to
1923accomplish what is normally done by ``make install``. In other words,
1924make sure your installation arranges the output similar to how it is
1925arranged on the target system.
1926
1927The OpenEmbedded build system provides support for starting services two
1928different ways:
1929
1930- *SysVinit:* SysVinit is a system and service manager that manages the
1931 init system used to control the very basic functions of your system.
1932 The init program is the first program started by the Linux kernel
1933 when the system boots. Init then controls the startup, running and
1934 shutdown of all other programs.
1935
1936 To enable a service using SysVinit, your recipe needs to inherit the
1937 :ref:`update-rc.d <ref-classes-update-rc.d>`
1938 class. The class helps facilitate safely installing the package on
1939 the target.
1940
1941 You will need to set the
1942 :term:`INITSCRIPT_PACKAGES`,
1943 :term:`INITSCRIPT_NAME`,
1944 and
1945 :term:`INITSCRIPT_PARAMS`
1946 variables within your recipe.
1947
1948- *systemd:* System Management Daemon (systemd) was designed to replace
1949 SysVinit and to provide enhanced management of services. For more
1950 information on systemd, see the systemd homepage at
1951 https://freedesktop.org/wiki/Software/systemd/.
1952
1953 To enable a service using systemd, your recipe needs to inherit the
1954 :ref:`systemd <ref-classes-systemd>` class. See
1955 the ``systemd.bbclass`` file located in your :term:`Source Directory`
1956 section for
1957 more information.
1958
1959Packaging
1960---------
1961
1962Successful packaging is a combination of automated processes performed
1963by the OpenEmbedded build system and some specific steps you need to
1964take. The following list describes the process:
1965
1966- *Splitting Files*: The ``do_package`` task splits the files produced
1967 by the recipe into logical components. Even software that produces a
1968 single binary might still have debug symbols, documentation, and
1969 other logical components that should be split out. The ``do_package``
1970 task ensures that files are split up and packaged correctly.
1971
1972- *Running QA Checks*: The
1973 :ref:`insane <ref-classes-insane>` class adds a
1974 step to the package generation process so that output quality
1975 assurance checks are generated by the OpenEmbedded build system. This
1976 step performs a range of checks to be sure the build's output is free
1977 of common problems that show up during runtime. For information on
1978 these checks, see the
1979 :ref:`insane <ref-classes-insane>` class and
1980 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
1981 chapter in the Yocto Project Reference Manual.
1982
1983- *Hand-Checking Your Packages*: After you build your software, you
1984 need to be sure your packages are correct. Examine the
1985 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
1986 directory and make sure files are where you expect them to be. If you
1987 discover problems, you can set
1988 :term:`PACKAGES`,
1989 :term:`FILES`,
1990 ``do_install(_append)``, and so forth as needed.
1991
1992- *Splitting an Application into Multiple Packages*: If you need to
1993 split an application into several packages, see the "`Splitting an
1994 Application into Multiple
1995 Packages <#splitting-an-application-into-multiple-packages>`__"
1996 section for an example.
1997
1998- *Installing a Post-Installation Script*: For an example showing how
1999 to install a post-installation script, see the "`Post-Installation
2000 Scripts <#new-recipe-post-installation-scripts>`__" section.
2001
2002- *Marking Package Architecture*: Depending on what your recipe is
2003 building and how it is configured, it might be important to mark the
2004 packages produced as being specific to a particular machine, or to
2005 mark them as not being specific to a particular machine or
2006 architecture at all.
2007
2008 By default, packages apply to any machine with the same architecture
2009 as the target machine. When a recipe produces packages that are
2010 machine-specific (e.g. the
2011 :term:`MACHINE` value is passed
2012 into the configure script or a patch is applied only for a particular
2013 machine), you should mark them as such by adding the following to the
2014 recipe:
2015 ::
2016
2017 PACKAGE_ARCH = "${MACHINE_ARCH}"
2018
2019 On the other hand, if the recipe produces packages that do not
2020 contain anything specific to the target machine or architecture at
2021 all (e.g. recipes that simply package script files or configuration
2022 files), you should use the
2023 :ref:`allarch <ref-classes-allarch>` class to
2024 do this for you by adding this to your recipe:
2025 ::
2026
2027 inherit allarch
2028
2029 Ensuring that the package architecture is correct is not critical
2030 while you are doing the first few builds of your recipe. However, it
2031 is important in order to ensure that your recipe rebuilds (or does
2032 not rebuild) appropriately in response to changes in configuration,
2033 and to ensure that you get the appropriate packages installed on the
2034 target machine, particularly if you run separate builds for more than
2035 one target machine.
2036
2037Sharing Files Between Recipes
2038-----------------------------
2039
2040Recipes often need to use files provided by other recipes on the build
2041host. For example, an application linking to a common library needs
2042access to the library itself and its associated headers. The way this
2043access is accomplished is by populating a sysroot with files. Each
2044recipe has two sysroots in its work directory, one for target files
2045(``recipe-sysroot``) and one for files that are native to the build host
2046(``recipe-sysroot-native``).
2047
2048.. note::
2049
2050 You could find the term "staging" used within the Yocto project
2051 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
2052 variable).
2053
2054Recipes should never populate the sysroot directly (i.e. write files
2055into sysroot). Instead, files should be installed into standard
2056locations during the
2057:ref:`ref-tasks-install` task within
2058the ``${``\ :term:`D`\ ``}`` directory. The
2059reason for this limitation is that almost all files that populate the
2060sysroot are cataloged in manifests in order to ensure the files can be
2061removed later when a recipe is either modified or removed. Thus, the
2062sysroot is able to remain free from stale files.
2063
2064A subset of the files installed by the
2065:ref:`ref-tasks-install` task are
2066used by the
2067:ref:`ref-tasks-populate_sysroot`
2068task as defined by the the
2069:term:`SYSROOT_DIRS` variable to
2070automatically populate the sysroot. It is possible to modify the list of
2071directories that populate the sysroot. The following example shows how
2072you could add the ``/opt`` directory to the list of directories within a
2073recipe:
2074::
2075
2076 SYSROOT_DIRS += "/opt"
2077
2078.. note::
2079
2080 The `/sysroot-only` is to be used by recipes that generate artifacts
2081 that are not included in the target filesystem, allowing them to share
2082 these artifacts without needing to use the ``DEPLOY_DIR``.
2083
2084For a more complete description of the
2085:ref:`ref-tasks-populate_sysroot`
2086task and its associated functions, see the
2087:ref:`staging <ref-classes-staging>` class.
2088
2089Using Virtual Providers
2090-----------------------
2091
2092Prior to a build, if you know that several different recipes provide the
2093same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2094as a placeholder for the actual provider. The actual provider is
2095determined at build-time.
2096
2097A common scenario where a virtual provider is used would be for the
2098kernel recipe. Suppose you have three kernel recipes whose
2099:term:`PN` values map to ``kernel-big``,
2100``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2101in some way uses a :term:`PROVIDES`
2102statement that essentially identifies itself as being able to provide
2103``virtual/kernel``. Here is one way through the
2104:ref:`kernel <ref-classes-kernel>` class:
2105::
2106
2107 PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }"
2108
2109Any recipe that inherits the ``kernel`` class is
2110going to utilize a ``PROVIDES`` statement that identifies that recipe as
2111being able to provide the ``virtual/kernel`` item.
2112
2113Now comes the time to actually build an image and you need a kernel
2114recipe, but which one? You can configure your build to call out the
2115kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
2116an example, consider the :yocto_git:`x86-base.inc
2117</poky/tree/meta/conf/machine/include/x86-base.inc>` include file, which is a
2118machine (i.e. :term:`MACHINE`) configuration file. This include file is the
2119reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
2120relevant lines from the include file:
2121::
2122
2123 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2124 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2125
2126When you use a virtual provider, you do not have to "hard code" a recipe
2127name as a build dependency. You can use the
2128:term:`DEPENDS` variable to state the
2129build is dependent on ``virtual/kernel`` for example:
2130::
2131
2132 DEPENDS = "virtual/kernel"
2133
2134During the build, the OpenEmbedded build system picks
2135the correct recipe needed for the ``virtual/kernel`` dependency based on
2136the ``PREFERRED_PROVIDER`` variable. If you want to use the small kernel
2137mentioned at the beginning of this section, configure your build as
2138follows:
2139::
2140
2141 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
2142
2143.. note::
2144
2145 Any recipe that ``PROVIDES`` a ``virtual/*`` item that is ultimately not
2146 selected through ``PREFERRED_PROVIDER`` does not get built. Preventing these
2147 recipes from building is usually the desired behavior since this mechanism's
2148 purpose is to select between mutually exclusive alternative providers.
2149
2150The following lists specific examples of virtual providers:
2151
2152- ``virtual/kernel``: Provides the name of the kernel recipe to use
2153 when building a kernel image.
2154
2155- ``virtual/bootloader``: Provides the name of the bootloader to use
2156 when building an image.
2157
2158- ``virtual/libgbm``: Provides ``gbm.pc``.
2159
2160- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2161
2162- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2163
2164- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2165
2166- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2167
2168.. note::
2169
2170 Virtual providers only apply to build time dependencies specified with
2171 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2172 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2173
2174Properly Versioning Pre-Release Recipes
2175---------------------------------------
2176
2177Sometimes the name of a recipe can lead to versioning problems when the
2178recipe is upgraded to a final release. For example, consider the
2179``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
2180the "`Storing and Naming the
2181Recipe <#new-recipe-storing-and-naming-the-recipe>`__" section. This
2182recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
2183released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2184change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2185build system and package managers, so the resulting packages will not
2186correctly trigger an upgrade.
2187
2188In order to ensure the versions compare properly, the recommended
2189convention is to set :term:`PV` within the
2190recipe to "previous_version+current_version". You can use an additional
2191variable so that you can use the current version elsewhere. Here is an
2192example:
2193::
2194
2195 REALPV = "0.8.16-rc1"
2196 PV = "0.8.15+${REALPV}"
2197
2198Post-Installation Scripts
2199-------------------------
2200
2201Post-installation scripts run immediately after installing a package on
2202the target or during image creation when a package is included in an
2203image. To add a post-installation script to a package, add a
2204``pkg_postinst_``\ `PACKAGENAME`\ ``()`` function to the recipe file
2205(``.bb``) and replace `PACKAGENAME` with the name of the package you want
2206to attach to the ``postinst`` script. To apply the post-installation
2207script to the main package for the recipe, which is usually what is
2208required, specify
2209``${``\ :term:`PN`\ ``}`` in place of
2210PACKAGENAME.
2211
2212A post-installation function has the following structure:
2213::
2214
2215 pkg_postinst_PACKAGENAME() {
2216 # Commands to carry out
2217 }
2218
2219The script defined in the post-installation function is called when the
2220root filesystem is created. If the script succeeds, the package is
2221marked as installed.
2222
2223.. note::
2224
2225 Any RPM post-installation script that runs on the target should
2226 return a 0 exit code. RPM does not allow non-zero exit codes for
2227 these scripts, and the RPM package manager will cause the package to
2228 fail installation on the target.
2229
2230Sometimes it is necessary for the execution of a post-installation
2231script to be delayed until the first boot. For example, the script might
2232need to be executed on the device itself. To delay script execution
2233until boot time, you must explicitly mark post installs to defer to the
2234target. You can use ``pkg_postinst_ontarget()`` or call
2235``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2236failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2237error during the
2238:ref:`ref-tasks-rootfs` task.
2239
2240If you have recipes that use ``pkg_postinst`` function and they require
2241the use of non-standard native tools that have dependencies during
2242rootfs construction, you need to use the
2243:term:`PACKAGE_WRITE_DEPS`
2244variable in your recipe to list these tools. If you do not use this
2245variable, the tools might be missing and execution of the
2246post-installation script is deferred until first boot. Deferring the
2247script to first boot is undesirable and for read-only rootfs impossible.
2248
2249.. note::
2250
2251 Equivalent support for pre-install, pre-uninstall, and post-uninstall
2252 scripts exist by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
2253 respectively. These scrips work in exactly the same way as does
2254 ``pkg_postinst`` with the exception that they run at different times. Also,
2255 because of when they run, they are not applicable to being run at image
2256 creation time like ``pkg_postinst``.
2257
2258Testing
2259-------
2260
2261The final step for completing your recipe is to be sure that the
2262software you built runs correctly. To accomplish runtime testing, add
2263the build's output packages to your image and test them on the target.
2264
2265For information on how to customize your image by adding specific
2266packages, see the "`Customizing
2267Images <#usingpoky-extend-customimage>`__" section.
2268
2269Examples
2270--------
2271
2272To help summarize how to write a recipe, this section provides some
2273examples given various scenarios:
2274
2275- Recipes that use local files
2276
2277- Using an Autotooled package
2278
2279- Using a Makefile-based package
2280
2281- Splitting an application into multiple packages
2282
2283- Adding binaries to an image
2284
2285Single .c File Package (Hello World!)
2286~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2287
2288Building an application from a single file that is stored locally (e.g.
2289under ``files``) requires a recipe that has the file listed in the
2290``SRC_URI`` variable. Additionally, you need to manually write the
2291``do_compile`` and ``do_install`` tasks. The ``S`` variable defines the
2292directory containing the source code, which is set to
2293:term:`WORKDIR` in this case - the
2294directory BitBake uses for the build.
2295::
2296
2297 SUMMARY = "Simple helloworld application"
2298 SECTION = "examples"
2299 LICENSE = "MIT"
2300 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2301
2302 SRC_URI = "file://helloworld.c"
2303
2304 S = "${WORKDIR}"
2305
2306 do_compile() {
2307 ${CC} ${LDFLAGS} helloworld.c -o helloworld
2308 }
2309
2310 do_install() {
2311 install -d ${D}${bindir}
2312 install -m 0755 helloworld ${D}${bindir}
2313 }
2314
2315By default, the ``helloworld``, ``helloworld-dbg``, and
2316``helloworld-dev`` packages are built. For information on how to
2317customize the packaging process, see the "`Splitting an Application into
2318Multiple Packages <#splitting-an-application-into-multiple-packages>`__"
2319section.
2320
2321Autotooled Package
2322~~~~~~~~~~~~~~~~~~
2323
2324Applications that use Autotools such as ``autoconf`` and ``automake``
2325require a recipe that has a source archive listed in ``SRC_URI`` and
2326also inherit the
2327:ref:`autotools <ref-classes-autotools>` class,
2328which contains the definitions of all the steps needed to build an
2329Autotool-based application. The result of the build is automatically
2330packaged. And, if the application uses NLS for localization, packages
2331with local information are generated (one package per language).
2332Following is one example: (``hello_2.3.bb``)
2333::
2334
2335 SUMMARY = "GNU Helloworld application"
2336 SECTION = "examples"
2337 LICENSE = "GPLv2+"
2338 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2339
2340 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2341
2342 inherit autotools gettext
2343
2344The variable ``LIC_FILES_CHKSUM`` is used to track source license
2345changes as described in the
2346":ref:`dev-manual/common-tasks:tracking license changes`" section in
2347the Yocto Project Overview and Concepts Manual. You can quickly create
2348Autotool-based recipes in a manner similar to the previous example.
2349
2350Makefile-Based Package
2351~~~~~~~~~~~~~~~~~~~~~~
2352
2353Applications that use GNU ``make`` also require a recipe that has the
2354source archive listed in ``SRC_URI``. You do not need to add a
2355``do_compile`` step since by default BitBake starts the ``make`` command
2356to compile the application. If you need additional ``make`` options, you
2357should store them in the
2358:term:`EXTRA_OEMAKE` or
2359:term:`PACKAGECONFIG_CONFARGS`
2360variables. BitBake passes these options into the GNU ``make``
2361invocation. Note that a ``do_install`` task is still required.
2362Otherwise, BitBake runs an empty ``do_install`` task by default.
2363
2364Some applications might require extra parameters to be passed to the
2365compiler. For example, the application might need an additional header
2366path. You can accomplish this by adding to the ``CFLAGS`` variable. The
2367following example shows this:
2368::
2369
2370 CFLAGS_prepend = "-I ${S}/include "
2371
2372In the following example, ``mtd-utils`` is a makefile-based package:
2373::
2374
2375 SUMMARY = "Tools for managing memory technology devices"
2376 SECTION = "base"
2377 DEPENDS = "zlib lzo e2fsprogs util-linux"
2378 HOMEPAGE = "http://www.linux-mtd.infradead.org/"
2379 LICENSE = "GPLv2+"
2380 LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3 \
2381 file://include/common.h;beginline=1;endline=17;md5=ba05b07912a44ea2bf81ce409380049c"
2382
2383 # Use the latest version at 26 Oct, 2013
2384 SRCREV = "9f107132a6a073cce37434ca9cda6917dd8d866b"
2385 SRC_URI = "git://git.infradead.org/mtd-utils.git \
2386 file://add-exclusion-to-mkfs-jffs2-git-2.patch \
2387 "
2388
2389 PV = "1.5.1+git${SRCPV}"
2390
2391 S = "${WORKDIR}/git"
2392
2393 EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'"
2394
2395 do_install () {
2396 oe_runmake install DESTDIR=${D} SBINDIR=${sbindir} MANDIR=${mandir} INCLUDEDIR=${includedir}
2397 }
2398
2399 PACKAGES =+ "mtd-utils-jffs2 mtd-utils-ubifs mtd-utils-misc"
2400
2401 FILES_mtd-utils-jffs2 = "${sbindir}/mkfs.jffs2 ${sbindir}/jffs2dump ${sbindir}/jffs2reader ${sbindir}/sumtool"
2402 FILES_mtd-utils-ubifs = "${sbindir}/mkfs.ubifs ${sbindir}/ubi*"
2403 FILES_mtd-utils-misc = "${sbindir}/nftl* ${sbindir}/ftl* ${sbindir}/rfd* ${sbindir}/doc* ${sbindir}/serve_image ${sbindir}/recv_image"
2404
2405 PARALLEL_MAKE = ""
2406
2407 BBCLASSEXTEND = "native"
2408
2409Splitting an Application into Multiple Packages
2410~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2411
2412You can use the variables ``PACKAGES`` and ``FILES`` to split an
2413application into multiple packages.
2414
2415Following is an example that uses the ``libxpm`` recipe. By default,
2416this recipe generates a single package that contains the library along
2417with a few binaries. You can modify the recipe to split the binaries
2418into separate packages:
2419::
2420
2421 require xorg-lib-common.inc
2422
2423 SUMMARY = "Xpm: X Pixmap extension library"
2424 LICENSE = "BSD"
2425 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2426 DEPENDS += "libxext libsm libxt"
2427 PE = "1"
2428
2429 XORG_PN = "libXpm"
2430
2431 PACKAGES =+ "sxpm cxpm"
2432 FILES_cxpm = "${bindir}/cxpm"
2433 FILES_sxpm = "${bindir}/sxpm"
2434
2435In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2436binaries in separate packages. Since ``bindir`` would be packaged into
2437the main ``PN`` package by default, we prepend the ``PACKAGES`` variable
2438so additional package names are added to the start of list. This results
2439in the extra ``FILES_*`` variables then containing information that
2440define which files and directories go into which packages. Files
2441included by earlier packages are skipped by latter packages. Thus, the
2442main ``PN`` package does not include the above listed files.
2443
2444Packaging Externally Produced Binaries
2445~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2446
2447Sometimes, you need to add pre-compiled binaries to an image. For
2448example, suppose that binaries for proprietary code exist, which are
2449created by a particular division of a company. Your part of the company
2450needs to use those binaries as part of an image that you are building
2451using the OpenEmbedded build system. Since you only have the binaries
2452and not the source code, you cannot use a typical recipe that expects to
2453fetch the source specified in
2454:term:`SRC_URI` and then compile it.
2455
2456One method is to package the binaries and then install them as part of
2457the image. Generally, it is not a good idea to package binaries since,
2458among other things, it can hinder the ability to reproduce builds and
2459could lead to compatibility problems with ABI in the future. However,
2460sometimes you have no choice.
2461
2462The easiest solution is to create a recipe that uses the
2463:ref:`bin_package <ref-classes-bin-package>` class
2464and to be sure that you are using default locations for build artifacts.
2465In most cases, the ``bin_package`` class handles "skipping" the
2466configure and compile steps as well as sets things up to grab packages
2467from the appropriate area. In particular, this class sets ``noexec`` on
2468both the :ref:`ref-tasks-configure`
2469and :ref:`ref-tasks-compile` tasks,
2470sets ``FILES_${PN}`` to "/" so that it picks up all files, and sets up a
2471:ref:`ref-tasks-install` task, which
2472effectively copies all files from ``${S}`` to ``${D}``. The
2473``bin_package`` class works well when the files extracted into ``${S}``
2474are already laid out in the way they should be laid out on the target.
2475For more information on these variables, see the
2476:term:`FILES`,
2477:term:`PN`,
2478:term:`S`, and
2479:term:`D` variables in the Yocto Project
2480Reference Manual's variable glossary.
2481
2482.. note::
2483
2484 - Using :term:`DEPENDS` is a good
2485 idea even for components distributed in binary form, and is often
2486 necessary for shared libraries. For a shared library, listing the
2487 library dependencies in ``DEPENDS`` makes sure that the libraries
2488 are available in the staging sysroot when other recipes link
2489 against the library, which might be necessary for successful
2490 linking.
2491
2492 - Using ``DEPENDS`` also allows runtime dependencies between
2493 packages to be added automatically. See the
2494 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
2495 section in the Yocto Project Overview and Concepts Manual for more
2496 information.
2497
2498If you cannot use the ``bin_package`` class, you need to be sure you are
2499doing the following:
2500
2501- Create a recipe where the
2502 :ref:`ref-tasks-configure` and
2503 :ref:`ref-tasks-compile` tasks do
2504 nothing: It is usually sufficient to just not define these tasks in
2505 the recipe, because the default implementations do nothing unless a
2506 Makefile is found in
2507 ``${``\ :term:`S`\ ``}``.
2508
2509 If ``${S}`` might contain a Makefile, or if you inherit some class
2510 that replaces ``do_configure`` and ``do_compile`` with custom
2511 versions, then you can use the
2512 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2513 flag to turn the tasks into no-ops, as follows:
2514 ::
2515
2516 do_configure[noexec] = "1"
2517 do_compile[noexec] = "1"
2518
2519 Unlike
2520 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2521 using the flag preserves the dependency chain from the
2522 :ref:`ref-tasks-fetch`,
2523 :ref:`ref-tasks-unpack`, and
2524 :ref:`ref-tasks-patch` tasks to the
2525 :ref:`ref-tasks-install` task.
2526
2527- Make sure your ``do_install`` task installs the binaries
2528 appropriately.
2529
2530- Ensure that you set up :term:`FILES`
2531 (usually
2532 ``FILES_${``\ :term:`PN`\ ``}``) to
2533 point to the files you have installed, which of course depends on
2534 where you have installed them and whether those files are in
2535 different locations than the defaults.
2536
2537.. note::
2538
2539 If image prelinking is enabled (e.g. "image-prelink" is in :term:`USER_CLASSES`
2540 which it is by default), prelink will change the binaries in the generated images
2541 and this often catches people out. Remove that class to ensure binaries are
2542 preserved exactly if that is necessary.
2543
2544Following Recipe Style Guidelines
2545---------------------------------
2546
2547When writing recipes, it is good to conform to existing style
2548guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
2549provides rough guidelines for preferred recipe style.
2550
2551It is common for existing recipes to deviate a bit from this style.
2552However, aiming for at least a consistent style is a good idea. Some
2553practices, such as omitting spaces around ``=`` operators in assignments
2554or ordering recipe components in an erratic way, are widely seen as poor
2555style.
2556
2557Recipe Syntax
2558-------------
2559
2560Understanding recipe file syntax is important for writing recipes. The
2561following list overviews the basic items that make up a BitBake recipe
2562file. For more complete BitBake syntax descriptions, see the
2563":doc:`bitbake-user-manual/bitbake-user-manual-metadata`"
2564chapter of the BitBake User Manual.
2565
2566- *Variable Assignments and Manipulations:* Variable assignments allow
2567 a value to be assigned to a variable. The assignment can be static
2568 text or might include the contents of other variables. In addition to
2569 the assignment, appending and prepending operations are also
2570 supported.
2571
2572 The following example shows some of the ways you can use variables in
2573 recipes:
2574 ::
2575
2576 S = "${WORKDIR}/postfix-${PV}"
2577 CFLAGS += "-DNO_ASM"
2578 SRC_URI_append = " file://fixup.patch"
2579
2580- *Functions:* Functions provide a series of actions to be performed.
2581 You usually use functions to override the default implementation of a
2582 task function or to complement a default function (i.e. append or
2583 prepend to an existing function). Standard functions use ``sh`` shell
2584 syntax, although access to OpenEmbedded variables and internal
2585 methods are also available.
2586
2587 The following is an example function from the ``sed`` recipe:
2588 ::
2589
2590 do_install () {
2591 autotools_do_install
2592 install -d ${D}${base_bindir}
2593 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2594 rmdir ${D}${bindir}/
2595 }
2596
2597 It is
2598 also possible to implement new functions that are called between
2599 existing tasks as long as the new functions are not replacing or
2600 complementing the default functions. You can implement functions in
2601 Python instead of shell. Both of these options are not seen in the
2602 majority of recipes.
2603
2604- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2605 to include common functions (``inherit``), load parts of a recipe
2606 from other files (``include`` and ``require``) and export variables
2607 to the environment (``export``).
2608
2609 The following example shows the use of some of these keywords:
2610 ::
2611
2612 export POSTCONF = "${STAGING_BINDIR}/postconf"
2613 inherit autoconf
2614 require otherfile.inc
2615
2616- *Comments (#):* Any lines that begin with the hash character (``#``)
2617 are treated as comment lines and are ignored:
2618 ::
2619
2620 # This is a comment
2621
2622This next list summarizes the most important and most commonly used
2623parts of the recipe syntax. For more information on these parts of the
2624syntax, you can reference the
2625:doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata` chapter
2626in the BitBake User Manual.
2627
2628- *Line Continuation (\\):* Use the backward slash (``\``) character to
2629 split a statement over multiple lines. Place the slash character at
2630 the end of the line that is to be continued on the next line:
2631 ::
2632
2633 VAR = "A really long \
2634 line"
2635
2636 .. note::
2637
2638 You cannot have any characters including spaces or tabs after the
2639 slash character.
2640
2641- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
2642 access the contents of a variable:
2643 ::
2644
2645 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2646
2647 .. note::
2648
2649 It is important to understand that the value of a variable
2650 expressed in this form does not get substituted automatically. The
2651 expansion of these expressions happens on-demand later (e.g.
2652 usually when a function that makes reference to the variable
2653 executes). This behavior ensures that the values are most
2654 appropriate for the context in which they are finally used. On the
2655 rare occasion that you do need the variable expression to be
2656 expanded immediately, you can use the
2657 :=
2658 operator instead of
2659 =
2660 when you make the assignment, but this is not generally needed.
2661
2662- *Quote All Assignments ("value"):* Use double quotes around values in
2663 all variable assignments (e.g. ``"value"``). Following is an example:
2664 ::
2665
2666 VAR1 = "${OTHERVAR}"
2667 VAR2 = "The version is ${PV}"
2668
2669- *Conditional Assignment (?=):* Conditional assignment is used to
2670 assign a value to a variable, but only when the variable is currently
2671 unset. Use the question mark followed by the equal sign (``?=``) to
2672 make a "soft" assignment used for conditional assignment. Typically,
2673 "soft" assignments are used in the ``local.conf`` file for variables
2674 that are allowed to come through from the external environment.
2675
2676 Here is an example where ``VAR1`` is set to "New value" if it is
2677 currently empty. However, if ``VAR1`` has already been set, it
2678 remains unchanged:
2679 ::
2680
2681 VAR1 ?= "New value"
2682
2683 In this next example, ``VAR1`` is left with the value "Original value":
2684 ::
2685
2686 VAR1 = "Original value"
2687 VAR1 ?= "New value"
2688
2689- *Appending (+=):* Use the plus character followed by the equals sign
2690 (``+=``) to append values to existing variables.
2691
2692 .. note::
2693
2694 This operator adds a space between the existing content of the
2695 variable and the new content.
2696
2697 Here is an example:
2698 ::
2699
2700 SRC_URI += "file://fix-makefile.patch"
2701
2702- *Prepending (=+):* Use the equals sign followed by the plus character
2703 (``=+``) to prepend values to existing variables.
2704
2705 .. note::
2706
2707 This operator adds a space between the new content and the
2708 existing content of the variable.
2709
2710 Here is an example:
2711 ::
2712
2713 VAR =+ "Starts"
2714
2715- *Appending (_append):* Use the ``_append`` operator to append values
2716 to existing variables. This operator does not add any additional
2717 space. Also, the operator is applied after all the ``+=``, and ``=+``
2718 operators have been applied and after all ``=`` assignments have
2719 occurred.
2720
2721 The following example shows the space being explicitly added to the
2722 start to ensure the appended value is not merged with the existing
2723 value:
2724 ::
2725
2726 SRC_URI_append = " file://fix-makefile.patch"
2727
2728 You can also use
2729 the ``_append`` operator with overrides, which results in the actions
2730 only being performed for the specified target or machine:
2731 ::
2732
2733 SRC_URI_append_sh4 = " file://fix-makefile.patch"
2734
2735- *Prepending (_prepend):* Use the ``_prepend`` operator to prepend
2736 values to existing variables. This operator does not add any
2737 additional space. Also, the operator is applied after all the ``+=``,
2738 and ``=+`` operators have been applied and after all ``=``
2739 assignments have occurred.
2740
2741 The following example shows the space being explicitly added to the
2742 end to ensure the prepended value is not merged with the existing
2743 value:
2744 ::
2745
2746 CFLAGS_prepend = "-I${S}/myincludes "
2747
2748 You can also use the
2749 ``_prepend`` operator with overrides, which results in the actions
2750 only being performed for the specified target or machine:
2751 ::
2752
2753 CFLAGS_prepend_sh4 = "-I${S}/myincludes "
2754
2755- *Overrides:* You can use overrides to set a value conditionally,
2756 typically based on how the recipe is being built. For example, to set
2757 the :term:`KBRANCH` variable's
2758 value to "standard/base" for any target
2759 :term:`MACHINE`, except for
2760 qemuarm where it should be set to "standard/arm-versatile-926ejs",
2761 you would do the following:
2762 ::
2763
2764 KBRANCH = "standard/base"
2765 KBRANCH_qemuarm = "standard/arm-versatile-926ejs"
2766
2767 Overrides are also used to separate
2768 alternate values of a variable in other situations. For example, when
2769 setting variables such as
2770 :term:`FILES` and
2771 :term:`RDEPENDS` that are
2772 specific to individual packages produced by a recipe, you should
2773 always use an override that specifies the name of the package.
2774
2775- *Indentation:* Use spaces for indentation rather than tabs. For
2776 shell functions, both currently work. However, it is a policy
2777 decision of the Yocto Project to use tabs in shell functions. Realize
2778 that some layers have a policy to use spaces for all indentation.
2779
2780- *Using Python for Complex Operations:* For more advanced processing,
2781 it is possible to use Python code during variable assignments (e.g.
2782 search and replacement on a variable).
2783
2784 You indicate Python code using the ``${@python_code}`` syntax for the
2785 variable assignment:
2786 ::
2787
2788 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2789
2790- *Shell Function Syntax:* Write shell functions as if you were writing
2791 a shell script when you describe a list of actions to take. You
2792 should ensure that your script works with a generic ``sh`` and that
2793 it does not require any ``bash`` or other shell-specific
2794 functionality. The same considerations apply to various system
2795 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2796 might wish to use. If in doubt, you should check with multiple
2797 implementations - including those from BusyBox.
2798
2799Adding a New Machine
2800====================
2801
2802Adding a new machine to the Yocto Project is a straightforward process.
2803This section describes how to add machines that are similar to those
2804that the Yocto Project already supports.
2805
2806.. note::
2807
2808 Although well within the capabilities of the Yocto Project, adding a
2809 totally new architecture might require changes to ``gcc``/``glibc``
2810 and to the site information, which is beyond the scope of this
2811 manual.
2812
2813For a complete example that shows how to add a new machine, see the
2814":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2815section in the Yocto Project Board Support Package (BSP) Developer's
2816Guide.
2817
2818Adding the Machine Configuration File
2819-------------------------------------
2820
2821To add a new machine, you need to add a new machine configuration file
2822to the layer's ``conf/machine`` directory. This configuration file
2823provides details about the device you are adding.
2824
2825The OpenEmbedded build system uses the root name of the machine
2826configuration file to reference the new machine. For example, given a
2827machine configuration file named ``crownbay.conf``, the build system
2828recognizes the machine as "crownbay".
2829
2830The most important variables you must set in your machine configuration
2831file or include from a lower-level configuration file are as follows:
2832
2833- ``TARGET_ARCH`` (e.g. "arm")
2834
2835- ``PREFERRED_PROVIDER_virtual/kernel``
2836
2837- ``MACHINE_FEATURES`` (e.g. "apm screen wifi")
2838
2839You might also need these variables:
2840
2841- ``SERIAL_CONSOLES`` (e.g. "115200;ttyS0 115200;ttyS1")
2842
2843- ``KERNEL_IMAGETYPE`` (e.g. "zImage")
2844
2845- ``IMAGE_FSTYPES`` (e.g. "tar.gz jffs2")
2846
2847You can find full details on these variables in the reference section.
2848You can leverage existing machine ``.conf`` files from
2849``meta-yocto-bsp/conf/machine/``.
2850
2851Adding a Kernel for the Machine
2852-------------------------------
2853
2854The OpenEmbedded build system needs to be able to build a kernel for the
2855machine. You need to either create a new kernel recipe for this machine,
2856or extend an existing kernel recipe. You can find several kernel recipe
2857examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2858you can use as references.
2859
2860If you are creating a new kernel recipe, normal recipe-writing rules
2861apply for setting up a ``SRC_URI``. Thus, you need to specify any
2862necessary patches and set ``S`` to point at the source code. You need to
2863create a ``do_configure`` task that configures the unpacked kernel with
2864a ``defconfig`` file. You can do this by using a ``make defconfig``
2865command or, more commonly, by copying in a suitable ``defconfig`` file
2866and then running ``make oldconfig``. By making use of ``inherit kernel``
2867and potentially some of the ``linux-*.inc`` files, most other
2868functionality is centralized and the defaults of the class normally work
2869well.
2870
2871If you are extending an existing kernel recipe, it is usually a matter
2872of adding a suitable ``defconfig`` file. The file needs to be added into
2873a location similar to ``defconfig`` files used for other machines in a
2874given kernel recipe. A possible way to do this is by listing the file in
2875the ``SRC_URI`` and adding the machine to the expression in
2876``COMPATIBLE_MACHINE``:
2877::
2878
2879 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2880
2881For more information on ``defconfig`` files, see the
2882":ref:`kernel-dev/common:changing the configuration`"
2883section in the Yocto Project Linux Kernel Development Manual.
2884
2885Adding a Formfactor Configuration File
2886--------------------------------------
2887
2888A formfactor configuration file provides information about the target
2889hardware for which the image is being built and information that the
2890build system cannot obtain from other sources such as the kernel. Some
2891examples of information contained in a formfactor configuration file
2892include framebuffer orientation, whether or not the system has a
2893keyboard, the positioning of the keyboard in relation to the screen, and
2894the screen resolution.
2895
2896The build system uses reasonable defaults in most cases. However, if
2897customization is necessary, you need to create a ``machconfig`` file in
2898the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2899contains directories for specific machines such as ``qemuarm`` and
2900``qemux86``. For information about the settings available and the
2901defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2902found in the same area.
2903
2904Following is an example for "qemuarm" machine:
2905::
2906
2907 HAVE_TOUCHSCREEN=1
2908 HAVE_KEYBOARD=1
2909 DISPLAY_CAN_ROTATE=0
2910 DISPLAY_ORIENTATION=0
2911 #DISPLAY_WIDTH_PIXELS=640
2912 #DISPLAY_HEIGHT_PIXELS=480
2913 #DISPLAY_BPP=16
2914 DISPLAY_DPI=150
2915 DISPLAY_SUBPIXEL_ORDER=vrgb
2916
2917Upgrading Recipes
2918=================
2919
2920Over time, upstream developers publish new versions for software built
2921by layer recipes. It is recommended to keep recipes up-to-date with
2922upstream version releases.
2923
2924While several methods exist that allow you upgrade a recipe, you might
2925consider checking on the upgrade status of a recipe first. You can do so
2926using the ``devtool check-upgrade-status`` command. See the
2927":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2928section in the Yocto Project Reference Manual for more information.
2929
2930The remainder of this section describes three ways you can upgrade a
2931recipe. You can use the Automated Upgrade Helper (AUH) to set up
2932automatic version upgrades. Alternatively, you can use
2933``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
2934you can manually upgrade a recipe by editing the recipe itself.
2935
2936Using the Auto Upgrade Helper (AUH)
2937-----------------------------------
2938
2939The AUH utility works in conjunction with the OpenEmbedded build system
2940in order to automatically generate upgrades for recipes based on new
2941versions being published upstream. Use AUH when you want to create a
2942service that performs the upgrades automatically and optionally sends
2943you an email with the results.
2944
2945AUH allows you to update several recipes with a single use. You can also
2946optionally perform build and integration tests using images with the
2947results saved to your hard drive and emails of results optionally sent
2948to recipe maintainers. Finally, AUH creates Git commits with appropriate
2949commit messages in the layer's tree for the changes made to recipes.
2950
2951.. note::
2952
2953 Conditions do exist when you should not use AUH to upgrade recipes
2954 and you should instead use either ``devtool upgrade`` or upgrade your
2955 recipes manually:
2956
2957 - When AUH cannot complete the upgrade sequence. This situation
2958 usually results because custom patches carried by the recipe
2959 cannot be automatically rebased to the new version. In this case,
2960 ``devtool upgrade`` allows you to manually resolve conflicts.
2961
2962 - When for any reason you want fuller control over the upgrade
2963 process. For example, when you want special arrangements for
2964 testing.
2965
2966The following steps describe how to set up the AUH utility:
2967
29681. *Be Sure the Development Host is Set Up:* You need to be sure that
2969 your development host is set up to use the Yocto Project. For
2970 information on how to set up your host, see the
2971 ":ref:`dev-manual/start:Preparing the Build Host`" section.
2972
29732. *Make Sure Git is Configured:* The AUH utility requires Git to be
2974 configured because AUH uses Git to save upgrades. Thus, you must have
2975 Git user and email configured. The following command shows your
2976 configurations:
2977 ::
2978
2979 $ git config --list
2980
2981 If you do not have the user and
2982 email configured, you can use the following commands to do so:
2983 ::
2984
2985 $ git config --global user.name some_name
2986 $ git config --global user.email username@domain.com
2987
29883. *Clone the AUH Repository:* To use AUH, you must clone the repository
2989 onto your development host. The following command uses Git to create
2990 a local copy of the repository on your system:
2991 ::
2992
2993 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
2994 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
2995 remote: Compressing objects: 100% (300/300), done.
2996 remote: Total 768 (delta 499), reused 703 (delta 434)
2997 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
2998 Resolving deltas: 100% (499/499), done.
2999 Checking connectivity... done.
3000
3001 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
3002 :term:`Poky` repositories.
3003
30044. *Create a Dedicated Build Directory:* Run the
3005 :ref:`structure-core-script`
3006 script to create a fresh build directory that you use exclusively for
3007 running the AUH utility:
3008 ::
3009
3010 $ cd ~/poky
3011 $ source oe-init-build-env your_AUH_build_directory
3012
3013 Re-using an existing build directory and its configurations is not
3014 recommended as existing settings could cause AUH to fail or behave
3015 undesirably.
3016
30175. *Make Configurations in Your Local Configuration File:* Several
3018 settings need to exist in the ``local.conf`` file in the build
3019 directory you just created for AUH. Make these following
3020 configurations:
3021
3022 - If you want to enable :ref:`Build
3023 History <dev-manual/common-tasks:maintaining build output quality>`,
3024 which is optional, you need the following lines in the
3025 ``conf/local.conf`` file:
3026 ::
3027
3028 INHERIT =+ "buildhistory"
3029 BUILDHISTORY_COMMIT = "1"
3030
3031 With this configuration and a successful
3032 upgrade, a build history "diff" file appears in the
3033 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3034 your build directory.
3035
3036 - If you want to enable testing through the
3037 :ref:`testimage <ref-classes-testimage*>`
3038 class, which is optional, you need to have the following set in
3039 your ``conf/local.conf`` file:
3040 ::
3041
3042 INHERIT += "testimage"
3043
3044 .. note::
3045
3046 If your distro does not enable by default ptest, which Poky
3047 does, you need the following in your ``local.conf`` file:
3048 ::
3049
3050 DISTRO_FEATURES_append = " ptest"
3051
3052
30536. *Optionally Start a vncserver:* If you are running in a server
3054 without an X11 session, you need to start a vncserver:
3055 ::
3056
3057 $ vncserver :1
3058 $ export DISPLAY=:1
3059
30607. *Create and Edit an AUH Configuration File:* You need to have the
3061 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
3062 build directory. You can find a sample configuration file in the
3063 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
3064
3065 Read through the sample file and make configurations as needed. For
3066 example, if you enabled build history in your ``local.conf`` as
3067 described earlier, you must enable it in ``upgrade-helper.conf``.
3068
3069 Also, if you are using the default ``maintainers.inc`` file supplied
3070 with Poky and located in ``meta-yocto`` and you do not set a
3071 "maintainers_whitelist" or "global_maintainer_override" in the
3072 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3073 the AUH command-line, the utility automatically sends out emails to
3074 all the default maintainers. Please avoid this.
3075
3076This next set of examples describes how to use the AUH:
3077
3078- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
3079 following form:
3080 ::
3081
3082 $ upgrade-helper.py recipe_name
3083
3084 For example, this command upgrades the ``xmodmap`` recipe:
3085 ::
3086
3087 $ upgrade-helper.py xmodmap
3088
3089- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
3090 specific recipe to a particular version, use the following form:
3091 ::
3092
3093 $ upgrade-helper.py recipe_name -t version
3094
3095 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3:
3096 ::
3097
3098 $ upgrade-helper.py xmodmap -t 1.2.3
3099
3100- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3101 Notifications:* To upgrade all recipes to their most recent versions
3102 and suppress the email notifications, use the following command:
3103 ::
3104
3105 $ upgrade-helper.py all
3106
3107- *Upgrading all Recipes to the Latest Versions and Send Email
3108 Notifications:* To upgrade all recipes to their most recent versions
3109 and send email messages to maintainers for each attempted recipe as
3110 well as a status email, use the following command:
3111 ::
3112
3113 $ upgrade-helper.py -e all
3114
3115Once you have run the AUH utility, you can find the results in the AUH
3116build directory:
3117::
3118
3119 ${BUILDDIR}/upgrade-helper/timestamp
3120
3121The AUH utility
3122also creates recipe update commits from successful upgrade attempts in
3123the layer tree.
3124
3125You can easily set up to run the AUH utility on a regular basis by using
3126a cron job. See the
3127:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
3128file distributed with the utility for an example.
3129
3130Using ``devtool upgrade``
3131-------------------------
3132
3133As mentioned earlier, an alternative method for upgrading recipes to
3134newer versions is to use
3135:doc:`devtool upgrade </ref-manual/devtool-reference>`.
3136You can read about ``devtool upgrade`` in general in the
3137":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
3138section in the Yocto Project Application Development and the Extensible
3139Software Development Kit (eSDK) Manual.
3140
3141To see all the command-line options available with ``devtool upgrade``,
3142use the following help command:
3143::
3144
3145 $ devtool upgrade -h
3146
3147If you want to find out what version a recipe is currently at upstream
3148without any attempt to upgrade your local version of the recipe, you can
3149use the following command:
3150::
3151
3152 $ devtool latest-version recipe_name
3153
3154As mentioned in the previous section describing AUH, ``devtool upgrade``
3155works in a less-automated manner than AUH. Specifically,
3156``devtool upgrade`` only works on a single recipe that you name on the
3157command line, cannot perform build and integration testing using images,
3158and does not automatically generate commits for changes in the source
3159tree. Despite all these "limitations", ``devtool upgrade`` updates the
3160recipe file to the new upstream version and attempts to rebase custom
3161patches contained by the recipe as needed.
3162
3163.. note::
3164
3165 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3166 of a "wrapper" application for ``devtool upgrade``.
3167
3168A typical scenario involves having used Git to clone an upstream
3169repository that you use during build operations. Because you have built the
3170recipe in the past, the layer is likely added to your
3171configuration already. If for some reason, the layer is not added, you
3172could add it easily using the
3173":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3174script. For example, suppose you use the ``nano.bb`` recipe from the
3175``meta-oe`` layer in the ``meta-openembedded`` repository. For this
3176example, assume that the layer has been cloned into following area:
3177::
3178
3179 /home/scottrif/meta-openembedded
3180
3181The following command from your
3182:term:`Build Directory` adds the layer to
3183your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``):
3184::
3185
3186 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3187 NOTE: Starting bitbake server...
3188 Parsing recipes: 100% |##########################################| Time: 0:00:55
3189 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3190 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3191 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3192 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3193 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3194
3195For this example, assume that the ``nano.bb`` recipe that
3196is upstream has a 2.9.3 version number. However, the version in the
3197local repository is 2.7.4. The following command from your build
3198directory automatically upgrades the recipe for you:
3199
3200.. note::
3201
3202 Using the ``-V`` option is not necessary. Omitting the version number causes
3203 ``devtool upgrade`` to upgrade the recipe to the most recent version.
3204
3205::
3206
3207 $ devtool upgrade nano -V 2.9.3
3208 NOTE: Starting bitbake server...
3209 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3210 Parsing recipes: 100% |##########################################| Time: 0:00:46
3211 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3212 NOTE: Extracting current version source...
3213 NOTE: Resolving any missing task queue dependencies
3214 .
3215 .
3216 .
3217 NOTE: Executing SetScene Tasks
3218 NOTE: Executing RunQueue Tasks
3219 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3220 Adding changed files: 100% |#####################################| Time: 0:00:00
3221 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3222 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3223
3224Continuing with this example, you can use ``devtool build`` to build the
3225newly upgraded recipe:
3226::
3227
3228 $ devtool build nano
3229 NOTE: Starting bitbake server...
3230 Loading cache: 100% |################################################################################################| Time: 0:00:01
3231 Loaded 2040 entries from dependency cache.
3232 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3233 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3234 NOTE: Resolving any missing task queue dependencies
3235 .
3236 .
3237 .
3238 NOTE: Executing SetScene Tasks
3239 NOTE: Executing RunQueue Tasks
3240 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3241 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3242
3243Within the ``devtool upgrade`` workflow, opportunity
3244exists to deploy and test your rebuilt software. For this example,
3245however, running ``devtool finish`` cleans up the workspace once the
3246source in your workspace is clean. This usually means using Git to stage
3247and submit commits for the changes generated by the upgrade process.
3248
3249Once the tree is clean, you can clean things up in this example with the
3250following command from the ``${BUILDDIR}/workspace/sources/nano``
3251directory:
3252::
3253
3254 $ devtool finish nano meta-oe
3255 NOTE: Starting bitbake server...
3256 Loading cache: 100% |################################################################################################| Time: 0:00:00
3257 Loaded 2040 entries from dependency cache.
3258 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3259 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3260 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3261 NOTE: Updating recipe nano_2.9.3.bb
3262 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3263 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3264 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3265
3266
3267Using the ``devtool finish`` command cleans up the workspace and creates a patch
3268file based on your commits. The tool puts all patch files back into the
3269source directory in a sub-directory named ``nano`` in this case.
3270
3271Manually Upgrading a Recipe
3272---------------------------
3273
3274If for some reason you choose not to upgrade recipes using
3275:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or
3276by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``,
3277you can manually edit the recipe files to upgrade the versions.
3278
3279.. note::
3280
3281 Manually updating multiple recipes scales poorly and involves many
3282 steps. The recommendation to upgrade recipe versions is through AUH
3283 or ``devtool upgrade``, both of which automate some steps and provide
3284 guidance for others needed for the manual process.
3285
3286To manually upgrade recipe versions, follow these general steps:
3287
32881. *Change the Version:* Rename the recipe such that the version (i.e.
3289 the :term:`PV` part of the recipe name)
3290 changes appropriately. If the version is not part of the recipe name,
3291 change the value as it is set for ``PV`` within the recipe itself.
3292
32932. *Update* ``SRCREV`` *if Needed*: If the source code your recipe builds
3294 is fetched from Git or some other version control system, update
3295 :term:`SRCREV` to point to the
3296 commit hash that matches the new version.
3297
32983. *Build the Software:* Try to build the recipe using BitBake. Typical
3299 build failures include the following:
3300
3301 - License statements were updated for the new version. For this
3302 case, you need to review any changes to the license and update the
3303 values of :term:`LICENSE` and
3304 :term:`LIC_FILES_CHKSUM`
3305 as needed.
3306
3307 .. note::
3308
3309 License changes are often inconsequential. For example, the
3310 license text's copyright year might have changed.
3311
3312 - Custom patches carried by the older version of the recipe might
3313 fail to apply to the new version. For these cases, you need to
3314 review the failures. Patches might not be necessary for the new
3315 version of the software if the upgraded version has fixed those
3316 issues. If a patch is necessary and failing, you need to rebase it
3317 into the new version.
3318
33194. *Optionally Attempt to Build for Several Architectures:* Once you
3320 successfully build the new software for a given architecture, you
3321 could test the build for other architectures by changing the
3322 :term:`MACHINE` variable and
3323 rebuilding the software. This optional step is especially important
3324 if the recipe is to be released publicly.
3325
33265. *Check the Upstream Change Log or Release Notes:* Checking both these
3327 reveals if new features exist that could break
3328 backwards-compatibility. If so, you need to take steps to mitigate or
3329 eliminate that situation.
3330
33316. *Optionally Create a Bootable Image and Test:* If you want, you can
3332 test the new software by booting it onto actual hardware.
3333
33347. *Create a Commit with the Change in the Layer Repository:* After all
3335 builds work and any testing is successful, you can create commits for
3336 any changes in the layer holding your upgraded recipe.
3337
3338Finding Temporary Source Code
3339=============================
3340
3341You might find it helpful during development to modify the temporary
3342source code used by recipes to build packages. For example, suppose you
3343are developing a patch and you need to experiment a bit to figure out
3344your solution. After you have initially built the package, you can
3345iteratively tweak the source code, which is located in the
3346:term:`Build Directory`, and then you can
3347force a re-compile and quickly test your altered code. Once you settle
3348on a solution, you can then preserve your changes in the form of
3349patches.
3350
3351During a build, the unpacked temporary source code used by recipes to
3352build packages is available in the Build Directory as defined by the
3353:term:`S` variable. Below is the default
3354value for the ``S`` variable as defined in the
3355``meta/conf/bitbake.conf`` configuration file in the
3356:term:`Source Directory`:
3357::
3358
3359 S = "${WORKDIR}/${BP}"
3360
3361You should be aware that many recipes override the
3362``S`` variable. For example, recipes that fetch their source from Git
3363usually set ``S`` to ``${WORKDIR}/git``.
3364
3365.. note::
3366
3367 The :term:`BP` represents the base recipe name, which consists of the name
3368 and version:
3369 ::
3370
3371 BP = "${BPN}-${PV}"
3372
3373
3374The path to the work directory for the recipe
3375(:term:`WORKDIR`) is defined as
3376follows:
3377::
3378
3379 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3380
3381The actual directory depends on several things:
3382
3383- :term:`TMPDIR`: The top-level build
3384 output directory.
3385
3386- :term:`MULTIMACH_TARGET_SYS`:
3387 The target system identifier.
3388
3389- :term:`PN`: The recipe name.
3390
3391- :term:`EXTENDPE`: The epoch - (if
3392 :term:`PE` is not specified, which is
3393 usually the case for most recipes, then ``EXTENDPE`` is blank).
3394
3395- :term:`PV`: The recipe version.
3396
3397- :term:`PR`: The recipe revision.
3398
3399As an example, assume a Source Directory top-level folder named
3400``poky``, a default Build Directory at ``poky/build``, and a
3401``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3402recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
3403build system uses to build the package would be as follows:
3404::
3405
3406 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3407
3408Using Quilt in Your Workflow
3409============================
3410
3411`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
3412that allows you to capture source code changes without having a clean
3413source tree. This section outlines the typical workflow you can use to
3414modify source code, test changes, and then preserve the changes in the
3415form of a patch all using Quilt.
3416
3417.. note::
3418
3419 With regard to preserving changes to source files, if you clean a
3420 recipe or have ``rm_work`` enabled, the
3421 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
3422 as described in the Yocto Project Application Development and the
3423 Extensible Software Development Kit (eSDK) manual is a safer
3424 development flow than the flow that uses Quilt.
3425
3426Follow these general steps:
3427
34281. *Find the Source Code:* Temporary source code used by the
3429 OpenEmbedded build system is kept in the
3430 :term:`Build Directory`. See the
3431 "`Finding Temporary Source
3432 Code <#finding-the-temporary-source-code>`__" section to learn how to
3433 locate the directory that has the temporary source code for a
3434 particular package.
3435
34362. *Change Your Working Directory:* You need to be in the directory that
3437 has the temporary source code. That directory is defined by the
3438 :term:`S` variable.
3439
34403. *Create a New Patch:* Before modifying source code, you need to
3441 create a new patch. To create a new patch file, use ``quilt new`` as
3442 below:
3443 ::
3444
3445 $ quilt new my_changes.patch
3446
34474. *Notify Quilt and Add Files:* After creating the patch, you need to
3448 notify Quilt about the files you plan to edit. You notify Quilt by
3449 adding the files to the patch you just created:
3450 ::
3451
3452 $ quilt add file1.c file2.c file3.c
3453
34545. *Edit the Files:* Make your changes in the source code to the files
3455 you added to the patch.
3456
34576. *Test Your Changes:* Once you have modified the source code, the
3458 easiest way to test your changes is by calling the ``do_compile``
3459 task as shown in the following example:
3460 ::
3461
3462 $ bitbake -c compile -f package
3463
3464 The ``-f`` or ``--force`` option forces the specified task to
3465 execute. If you find problems with your code, you can just keep
3466 editing and re-testing iteratively until things work as expected.
3467
3468 .. note::
3469
3470 All the modifications you make to the temporary source code disappear
3471 once you run the ``do_clean`` or ``do_cleanall`` tasks using BitBake
3472 (i.e. ``bitbake -c clean package`` and ``bitbake -c cleanall package``).
3473 Modifications will also disappear if you use the ``rm_work`` feature as
3474 described in the
3475 ":ref:`dev-manual/common-tasks:conserving disk space during builds`"
3476 section.
3477
34787. *Generate the Patch:* Once your changes work as expected, you need to
3479 use Quilt to generate the final patch that contains all your
3480 modifications.
3481 ::
3482
3483 $ quilt refresh
3484
3485 At this point, the
3486 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3487 ``file2.c``, and ``file3.c`` files.
3488
3489 You can find the resulting patch file in the ``patches/``
3490 subdirectory of the source (``S``) directory.
3491
34928. *Copy the Patch File:* For simplicity, copy the patch file into a
3493 directory named ``files``, which you can create in the same directory
3494 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3495 file. Placing the patch here guarantees that the OpenEmbedded build
3496 system will find the patch. Next, add the patch into the ``SRC_URI``
3497 of the recipe. Here is an example:
3498 ::
3499
3500 SRC_URI += "file://my_changes.patch"
3501
3502Using a Development Shell
3503=========================
3504
3505When debugging certain commands or even when just editing packages,
3506``devshell`` can be a useful tool. When you invoke ``devshell``, all
3507tasks up to and including
3508:ref:`ref-tasks-patch` are run for the
3509specified target. Then, a new terminal is opened and you are placed in
3510``${``\ :term:`S`\ ``}``, the source
3511directory. In the new terminal, all the OpenEmbedded build-related
3512environment variables are still defined so you can use commands such as
3513``configure`` and ``make``. The commands execute just as if the
3514OpenEmbedded build system were executing them. Consequently, working
3515this way can be helpful when debugging a build or preparing software to
3516be used with the OpenEmbedded build system.
3517
3518Following is an example that uses ``devshell`` on a target named
3519``matchbox-desktop``:
3520::
3521
3522 $ bitbake matchbox-desktop -c devshell
3523
3524This command spawns a terminal with a shell prompt within the
3525OpenEmbedded build environment. The
3526:term:`OE_TERMINAL` variable
3527controls what type of shell is opened.
3528
3529For spawned terminals, the following occurs:
3530
3531- The ``PATH`` variable includes the cross-toolchain.
3532
3533- The ``pkgconfig`` variables find the correct ``.pc`` files.
3534
3535- The ``configure`` command finds the Yocto Project site files as well
3536 as any other necessary files.
3537
3538Within this environment, you can run configure or compile commands as if
3539they were being run by the OpenEmbedded build system itself. As noted
3540earlier, the working directory also automatically changes to the Source
3541Directory (:term:`S`).
3542
3543To manually run a specific task using ``devshell``, run the
3544corresponding ``run.*`` script in the
3545``${``\ :term:`WORKDIR`\ ``}/temp``
3546directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
3547not exist, which would be the case if the task was skipped by way of the
3548sstate cache, you can create the task by first running it outside of the
3549``devshell``:
3550::
3551
3552 $ bitbake -c task
3553
3554.. note::
3555
3556 - Execution of a task's ``run.*`` script and BitBake's execution of
3557 a task are identical. In other words, running the script re-runs
3558 the task just as it would be run using the ``bitbake -c`` command.
3559
3560 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3561 symbolic link (symlink) to the most recent version of that file.
3562
3563Remember, that the ``devshell`` is a mechanism that allows you to get
3564into the BitBake task execution environment. And as such, all commands
3565must be called just as BitBake would call them. That means you need to
3566provide the appropriate options for cross-compilation and so forth as
3567applicable.
3568
3569When you are finished using ``devshell``, exit the shell or close the
3570terminal window.
3571
3572.. note::
3573
3574 - It is worth remembering that when using ``devshell`` you need to
3575 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3576 instead of just using ``gcc``. The same applies to other
3577 applications such as ``binutils``, ``libtool`` and so forth.
3578 BitBake sets up environment variables such as ``CC`` to assist
3579 applications, such as ``make`` to find the correct tools.
3580
3581 - It is also worth noting that ``devshell`` still works over X11
3582 forwarding and similar situations.
3583
3584Using a Development Python Shell
3585================================
3586
3587Similar to working within a development shell as described in the
3588previous section, you can also spawn and work within an interactive
3589Python development shell. When debugging certain commands or even when
3590just editing packages, ``devpyshell`` can be a useful tool. When you
3591invoke ``devpyshell``, all tasks up to and including
3592:ref:`ref-tasks-patch` are run for the
3593specified target. Then a new terminal is opened. Additionally, key
3594Python objects and code are available in the same way they are to
3595BitBake tasks, in particular, the data store 'd'. So, commands such as
3596the following are useful when exploring the data store and running
3597functions:
3598::
3599
3600 pydevshell> d.getVar("STAGING_DIR")
3601 '/media/build1/poky/build/tmp/sysroots'
3602 pydevshell> d.getVar("STAGING_DIR")
3603 '${TMPDIR}/sysroots'
3604 pydevshell> d.setVar("FOO", "bar")
3605 pydevshell> d.getVar("FOO")
3606 'bar'
3607 pydevshell> d.delVar("FOO")
3608 pydevshell> d.getVar("FOO")
3609 pydevshell> bb.build.exec_func("do_unpack", d)
3610 pydevshell>
3611
3612The commands execute just as if the OpenEmbedded build
3613system were executing them. Consequently, working this way can be
3614helpful when debugging a build or preparing software to be used with the
3615OpenEmbedded build system.
3616
3617Following is an example that uses ``devpyshell`` on a target named
3618``matchbox-desktop``:
3619::
3620
3621 $ bitbake matchbox-desktop -c devpyshell
3622
3623This command spawns a terminal and places you in an interactive Python
3624interpreter within the OpenEmbedded build environment. The
3625:term:`OE_TERMINAL` variable
3626controls what type of shell is opened.
3627
3628When you are finished using ``devpyshell``, you can exit the shell
3629either by using Ctrl+d or closing the terminal window.
3630
3631Building
3632========
3633
3634This section describes various build procedures. For example, the steps
3635needed for a simple build, a target that uses multiple configurations,
3636building an image for more than one machine, and so forth.
3637
3638Building a Simple Image
3639-----------------------
3640
3641In the development environment, you need to build an image whenever you
3642change hardware support, add or change system libraries, or add or
3643change services that have dependencies. Several methods exist that allow
3644you to build an image within the Yocto Project. This section presents
3645the basic steps you need to build a simple image using BitBake from a
3646build host running Linux.
3647
3648.. note::
3649
3650 - For information on how to build an image using
3651 :term:`Toaster`, see the
3652 :doc:`/toaster-manual/index`.
3653
3654 - For information on how to use ``devtool`` to build images, see the
3655 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
3656 section in the Yocto Project Application Development and the
3657 Extensible Software Development Kit (eSDK) manual.
3658
3659 - For a quick example on how to build an image using the
3660 OpenEmbedded build system, see the
3661 :doc:`/brief-yoctoprojectqs/index` document.
3662
3663The build process creates an entire Linux distribution from source and
3664places it in your :term:`Build Directory` under
3665``tmp/deploy/images``. For detailed information on the build process
3666using BitBake, see the ":ref:`overview-manual/concepts:images`" section in the
3667Yocto Project Overview and Concepts Manual.
3668
3669The following figure and list overviews the build process:
3670
3671.. image:: figures/bitbake-build-flow.png
3672 :align: center
3673
36741. *Set up Your Host Development System to Support Development Using the
3675 Yocto Project*: See the ":doc:`start`" section for options on how to get a
3676 build host ready to use the Yocto Project.
3677
36782. *Initialize the Build Environment:* Initialize the build environment
3679 by sourcing the build environment script (i.e.
3680 :ref:`structure-core-script`):
3681 ::
3682
3683 $ source oe-init-build-env [build_dir]
3684
3685 When you use the initialization script, the OpenEmbedded build system
3686 uses ``build`` as the default :term:`Build Directory` in your current work
3687 directory. You can use a `build_dir` argument with the script to
3688 specify a different build directory.
3689
3690 .. note::
3691
3692 A common practice is to use a different Build Directory for
3693 different targets. For example, ``~/build/x86`` for a ``qemux86``
3694 target, and ``~/build/arm`` for a ``qemuarm`` target.
3695
36963. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
3697 ``conf/local.conf`` configuration file, which is found in the Build
3698 Directory, is set up how you want it. This file defines many aspects
3699 of the build environment including the target machine architecture
3700 through the ``MACHINE`` variable, the packaging format used during
3701 the build
3702 (:term:`PACKAGE_CLASSES`),
3703 and a centralized tarball download directory through the
3704 :term:`DL_DIR` variable.
3705
37064. *Build the Image:* Build the image using the ``bitbake`` command:
3707 ::
3708
3709 $ bitbake target
3710
3711 .. note::
3712
3713 For information on BitBake, see the :doc:`bitbake:index`.
3714
3715 The target is the name of the recipe you want to build. Common
3716 targets are the images in ``meta/recipes-core/images``,
3717 ``meta/recipes-sato/images``, and so forth all found in the
3718 :term:`Source Directory`. Or, the target
3719 can be the name of a recipe for a specific piece of software such as
3720 BusyBox. For more details about the images the OpenEmbedded build
3721 system supports, see the
3722 ":ref:`ref-manual/images:Images`" chapter in the Yocto
3723 Project Reference Manual.
3724
3725 As an example, the following command builds the
3726 ``core-image-minimal`` image:
3727 ::
3728
3729 $ bitbake core-image-minimal
3730
3731 Once an
3732 image has been built, it often needs to be installed. The images and
3733 kernels built by the OpenEmbedded build system are placed in the
3734 Build Directory in ``tmp/deploy/images``. For information on how to
3735 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
3736 :doc:`/sdk-manual/index` manual. For
3737 information about how to install these images, see the documentation
3738 for your particular board or machine.
3739
3740Building Images for Multiple Targets Using Multiple Configurations
3741------------------------------------------------------------------
3742
3743You can use a single ``bitbake`` command to build multiple images or
3744packages for different targets where each image or package requires a
3745different configuration (multiple configuration builds). The builds, in
3746this scenario, are sometimes referred to as "multiconfigs", and this
3747section uses that term throughout.
3748
3749This section describes how to set up for multiple configuration builds
3750and how to account for cross-build dependencies between the
3751multiconfigs.
3752
3753Setting Up and Running a Multiple Configuration Build
3754~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3755
3756To accomplish a multiple configuration build, you must define each
3757target's configuration separately using a parallel configuration file in
3758the :term:`Build Directory`, and you
3759must follow a required file hierarchy. Additionally, you must enable the
3760multiple configuration builds in your ``local.conf`` file.
3761
3762Follow these steps to set up and execute multiple configuration builds:
3763
3764- *Create Separate Configuration Files*: You need to create a single
3765 configuration file for each build target (each multiconfig).
3766 Minimally, each configuration file must define the machine and the
3767 temporary directory BitBake uses for the build. Suggested practice
3768 dictates that you do not overlap the temporary directories used
3769 during the builds. However, it is possible that you can share the
3770 temporary directory
3771 (:term:`TMPDIR`). For example,
3772 consider a scenario with two different multiconfigs for the same
3773 :term:`MACHINE`: "qemux86" built
3774 for two distributions such as "poky" and "poky-lsb". In this case,
3775 you might want to use the same ``TMPDIR``.
3776
3777 Here is an example showing the minimal statements needed in a
3778 configuration file for a "qemux86" target whose temporary build
3779 directory is ``tmpmultix86``:
3780 ::
3781
3782 MACHINE = "qemux86"
3783 TMPDIR = "${TOPDIR}/tmpmultix86"
3784
3785 The location for these multiconfig configuration files is specific.
3786 They must reside in the current build directory in a sub-directory of
3787 ``conf`` named ``multiconfig``. Following is an example that defines
3788 two configuration files for the "x86" and "arm" multiconfigs:
3789
3790 .. image:: figures/multiconfig_files.png
3791 :align: center
3792
3793 The reason for this required file hierarchy is because the ``BBPATH``
3794 variable is not constructed until the layers are parsed.
3795 Consequently, using the configuration file as a pre-configuration
3796 file is not possible unless it is located in the current working
3797 directory.
3798
3799- *Add the BitBake Multi-configuration Variable to the Local
3800 Configuration File*: Use the
3801 :term:`BBMULTICONFIG`
3802 variable in your ``conf/local.conf`` configuration file to specify
3803 each multiconfig. Continuing with the example from the previous
3804 figure, the ``BBMULTICONFIG`` variable needs to enable two
3805 multiconfigs: "x86" and "arm" by specifying each configuration file:
3806 ::
3807
3808 BBMULTICONFIG = "x86 arm"
3809
3810 .. note::
3811
3812 A "default" configuration already exists by definition. This
3813 configuration is named: "" (i.e. empty string) and is defined by
3814 the variables coming from your ``local.conf``
3815 file. Consequently, the previous example actually adds two
3816 additional configurations to your build: "arm" and "x86" along
3817 with "".
3818
3819- *Launch BitBake*: Use the following BitBake command form to launch
3820 the multiple configuration build:
3821 ::
3822
3823 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3824
3825 For the example in this section, the following command applies:
3826 ::
3827
3828 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3829
3830 The previous BitBake command builds a ``core-image-minimal`` image
3831 that is configured through the ``x86.conf`` configuration file, a
3832 ``core-image-sato`` image that is configured through the ``arm.conf``
3833 configuration file and a ``core-image-base`` that is configured
3834 through your ``local.conf`` configuration file.
3835
3836.. note::
3837
3838 Support for multiple configuration builds in the Yocto Project &DISTRO;
3839 (&DISTRO_NAME;) Release does not include Shared State (sstate)
3840 optimizations. Consequently, if a build uses the same object twice
3841 in, for example, two different ``TMPDIR``
3842 directories, the build either loads from an existing sstate cache for
3843 that build at the start or builds the object fresh.
3844
3845Enabling Multiple Configuration Build Dependencies
3846~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3847
3848Sometimes dependencies can exist between targets (multiconfigs) in a
3849multiple configuration build. For example, suppose that in order to
3850build a ``core-image-sato`` image for an "x86" multiconfig, the root
3851filesystem of an "arm" multiconfig must exist. This dependency is
3852essentially that the
3853:ref:`ref-tasks-image` task in the
3854``core-image-sato`` recipe depends on the completion of the
3855:ref:`ref-tasks-rootfs` task of the
3856``core-image-minimal`` recipe.
3857
3858To enable dependencies in a multiple configuration build, you must
3859declare the dependencies in the recipe using the following statement
3860form:
3861::
3862
3863 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3864
3865To better show how to use this statement, consider the example scenario
3866from the first paragraph of this section. The following statement needs
3867to be added to the recipe that builds the ``core-image-sato`` image:
3868::
3869
3870 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3871
3872In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
3873task on which the ``do_image`` task in the recipe depends is the
3874``do_rootfs`` task from the ``core-image-minimal`` recipe associated
3875with the "arm" multiconfig.
3876
3877Once you set up this dependency, you can build the "x86" multiconfig
3878using a BitBake command as follows:
3879::
3880
3881 $ bitbake mc:x86:core-image-sato
3882
3883This command executes all the tasks needed to create the
3884``core-image-sato`` image for the "x86" multiconfig. Because of the
3885dependency, BitBake also executes through the ``do_rootfs`` task for the
3886"arm" multiconfig build.
3887
3888Having a recipe depend on the root filesystem of another build might not
3889seem that useful. Consider this change to the statement in the
3890``core-image-sato`` recipe:
3891::
3892
3893 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3894
3895In this case, BitBake must
3896create the ``core-image-minimal`` image for the "arm" build since the
3897"x86" build depends on it.
3898
3899Because "x86" and "arm" are enabled for multiple configuration builds
3900and have separate configuration files, BitBake places the artifacts for
3901each build in the respective temporary build directories (i.e.
3902:term:`TMPDIR`).
3903
3904Building an Initial RAM Filesystem (initramfs) Image
3905----------------------------------------------------
3906
3907An initial RAM filesystem (initramfs) image provides a temporary root
3908filesystem used for early system initialization (e.g. loading of modules
3909needed to locate and mount the "real" root filesystem).
3910
3911.. note::
3912
3913 The initramfs image is the successor of initial RAM disk (initrd). It
3914 is a "copy in and out" (cpio) archive of the initial filesystem that
3915 gets loaded into memory during the Linux startup process. Because
3916 Linux uses the contents of the archive during initialization, the
3917 initramfs image needs to contain all of the device drivers and tools
3918 needed to mount the final root filesystem.
3919
3920Follow these steps to create an initramfs image:
3921
39221. *Create the initramfs Image Recipe:* You can reference the
3923 ``core-image-minimal-initramfs.bb`` recipe found in the
3924 ``meta/recipes-core`` directory of the :term:`Source Directory`
3925 as an example
3926 from which to work.
3927
39282. *Decide if You Need to Bundle the initramfs Image Into the Kernel
3929 Image:* If you want the initramfs image that is built to be bundled
3930 in with the kernel image, set the
3931 :term:`INITRAMFS_IMAGE_BUNDLE`
3932 variable to "1" in your ``local.conf`` configuration file and set the
3933 :term:`INITRAMFS_IMAGE`
3934 variable in the recipe that builds the kernel image.
3935
3936 .. note::
3937
3938 It is recommended that you do bundle the initramfs image with the
3939 kernel image to avoid circular dependencies between the kernel
3940 recipe and the initramfs recipe should the initramfs image include
3941 kernel modules.
3942
3943 Setting the ``INITRAMFS_IMAGE_BUNDLE`` flag causes the initramfs
3944 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
3945 initramfs image is then passed to the kernel's ``Makefile`` using the
3946 :term:`CONFIG_INITRAMFS_SOURCE`
3947 variable, allowing the initramfs image to be built into the kernel
3948 normally.
3949
3950 .. note::
3951
3952 If you choose to not bundle the initramfs image with the kernel
3953 image, you are essentially using an
3954 `Initial RAM Disk (initrd) <https://en.wikipedia.org/wiki/Initrd>`__.
3955 Creating an initrd is handled primarily through the :term:`INITRD_IMAGE`,
3956 ``INITRD_LIVE``, and ``INITRD_IMAGE_LIVE`` variables. For more
3957 information, see the :ref:`ref-classes-image-live` file.
3958
39593. *Optionally Add Items to the initramfs Image Through the initramfs
3960 Image Recipe:* If you add items to the initramfs image by way of its
3961 recipe, you should use
3962 :term:`PACKAGE_INSTALL`
3963 rather than
3964 :term:`IMAGE_INSTALL`.
3965 ``PACKAGE_INSTALL`` gives more direct control of what is added to the
3966 image as compared to the defaults you might not necessarily want that
3967 are set by the :ref:`image <ref-classes-image>`
3968 or :ref:`core-image <ref-classes-core-image>`
3969 classes.
3970
39714. *Build the Kernel Image and the initramfs Image:* Build your kernel
3972 image using BitBake. Because the initramfs image recipe is a
3973 dependency of the kernel image, the initramfs image is built as well
3974 and bundled with the kernel image if you used the
3975 :term:`INITRAMFS_IMAGE_BUNDLE`
3976 variable described earlier.
3977
3978Building a Tiny System
3979----------------------
3980
3981Very small distributions have some significant advantages such as
3982requiring less on-die or in-package memory (cheaper), better performance
3983through efficient cache usage, lower power requirements due to less
3984memory, faster boot times, and reduced development overhead. Some
3985real-world examples where a very small distribution gives you distinct
3986advantages are digital cameras, medical devices, and small headless
3987systems.
3988
3989This section presents information that shows you how you can trim your
3990distribution to even smaller sizes than the ``poky-tiny`` distribution,
3991which is around 5 Mbytes, that can be built out-of-the-box using the
3992Yocto Project.
3993
3994Tiny System Overview
3995~~~~~~~~~~~~~~~~~~~~
3996
3997The following list presents the overall steps you need to consider and
3998perform to create distributions with smaller root filesystems, achieve
3999faster boot times, maintain your critical functionality, and avoid
4000initial RAM disks:
4001
4002- `Determine your goals and guiding
4003 principles. <#goals-and-guiding-principles>`__
4004
4005- `Understand what contributes to your image
4006 size. <#understand-what-gives-your-image-size>`__
4007
4008- `Reduce the size of the root
4009 filesystem. <#trim-the-root-filesystem>`__
4010
4011- `Reduce the size of the kernel. <#trim-the-kernel>`__
4012
4013- `Eliminate packaging
4014 requirements. <#remove-package-management-requirements>`__
4015
4016- `Look for other ways to minimize
4017 size. <#look-for-other-ways-to-minimize-size>`__
4018
4019- `Iterate on the process. <#iterate-on-the-process>`__
4020
4021Goals and Guiding Principles
4022~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4023
4024Before you can reach your destination, you need to know where you are
4025going. Here is an example list that you can use as a guide when creating
4026very small distributions:
4027
4028- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
4029 less and a root filesystem that is 3 Mbytes or less).
4030
4031- Find the areas that are currently taking 90% of the space and
4032 concentrate on reducing those areas.
4033
4034- Do not create any difficult "hacks" to achieve your goals.
4035
4036- Leverage the device-specific options.
4037
4038- Work in a separate layer so that you keep changes isolated. For
4039 information on how to create layers, see the "`Understanding and
4040 Creating Layers <#understanding-and-creating-layers>`__" section.
4041
4042Understand What Contributes to Your Image Size
4043~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4044
4045It is easiest to have something to start with when creating your own
4046distribution. You can use the Yocto Project out-of-the-box to create the
4047``poky-tiny`` distribution. Ultimately, you will want to make changes in
4048your own distribution that are likely modeled after ``poky-tiny``.
4049
4050.. note::
4051
4052 To use ``poky-tiny`` in your build, set the ``DISTRO`` variable in your
4053 ``local.conf`` file to "poky-tiny" as described in the
4054 ":ref:`dev-manual/common-tasks:creating your own distribution`"
4055 section.
4056
4057Understanding some memory concepts will help you reduce the system size.
4058Memory consists of static, dynamic, and temporary memory. Static memory
4059is the TEXT (code), DATA (initialized data in the code), and BSS
4060(uninitialized data) sections. Dynamic memory represents memory that is
4061allocated at runtime: stacks, hash tables, and so forth. Temporary
4062memory is recovered after the boot process. This memory consists of
4063memory used for decompressing the kernel and for the ``__init__``
4064functions.
4065
4066To help you see where you currently are with kernel and root filesystem
4067sizes, you can use two tools found in the :term:`Source Directory`
4068in the
4069``scripts/tiny/`` directory:
4070
4071- ``ksize.py``: Reports component sizes for the kernel build objects.
4072
4073- ``dirsize.py``: Reports component sizes for the root filesystem.
4074
4075This next tool and command help you organize configuration fragments and
4076view file dependencies in a human-readable form:
4077
4078- ``merge_config.sh``: Helps you manage configuration files and
4079 fragments within the kernel. With this tool, you can merge individual
4080 configuration fragments together. The tool allows you to make
4081 overrides and warns you of any missing configuration options. The
4082 tool is ideal for allowing you to iterate on configurations, create
4083 minimal configurations, and create configuration files for different
4084 machines without having to duplicate your process.
4085
4086 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4087 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4088 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4089 directory.
4090
4091 For more information on configuration fragments, see the
4092 ":ref:`kernel-dev/common:creating configuration fragments`"
4093 section in the Yocto Project Linux Kernel Development Manual.
4094
4095- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4096 with these options brings up a Dependency Explorer from which you can
4097 view file dependencies. Understanding these dependencies allows you
4098 to make informed decisions when cutting out various pieces of the
4099 kernel and root filesystem.
4100
4101Trim the Root Filesystem
4102~~~~~~~~~~~~~~~~~~~~~~~~
4103
4104The root filesystem is made up of packages for booting, libraries, and
4105applications. To change things, you can configure how the packaging
4106happens, which changes the way you build them. You can also modify the
4107filesystem itself or select a different filesystem.
4108
4109First, find out what is hogging your root filesystem by running the
4110``dirsize.py`` script from your root directory:
4111::
4112
4113 $ cd root-directory-of-image
4114 $ dirsize.py 100000 > dirsize-100k.log
4115 $ cat dirsize-100k.log
4116
4117You can apply a filter to the script to ignore files
4118under a certain size. The previous example filters out any files below
4119100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4120will be smaller by a relatively constant factor in a compressed root
4121filesystem. When you examine your log file, you can focus on areas of
4122the root filesystem that take up large amounts of memory.
4123
4124You need to be sure that what you eliminate does not cripple the
4125functionality you need. One way to see how packages relate to each other
4126is by using the Dependency Explorer UI with the BitBake command:
4127::
4128
4129 $ cd image-directory
4130 $ bitbake -u taskexp -g image
4131
4132Use the interface to
4133select potential packages you wish to eliminate and see their dependency
4134relationships.
4135
4136When deciding how to reduce the size, get rid of packages that result in
4137minimal impact on the feature set. For example, you might not need a VGA
4138display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4139instead of ``udev``.
4140
4141Use your ``local.conf`` file to make changes. For example, to eliminate
4142``udev`` and ``glib``, set the following in the local configuration
4143file:
4144::
4145
4146 VIRTUAL-RUNTIME_dev_manager = ""
4147
4148Finally, you should consider exactly the type of root filesystem you
4149need to meet your needs while also reducing its size. For example,
4150consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4151``initramfs`` using ``initramfs``. Be aware that ``ext3`` requires a 1
4152Mbyte journal. If you are okay with running read-only, you do not need
4153this journal.
4154
4155.. note::
4156
4157 After each round of elimination, you need to rebuild your system and
4158 then use the tools to see the effects of your reductions.
4159
4160Trim the Kernel
4161~~~~~~~~~~~~~~~
4162
4163The kernel is built by including policies for hardware-independent
4164aspects. What subsystems do you enable? For what architecture are you
4165building? Which drivers do you build by default?
4166
4167.. note::
4168
4169 You can modify the kernel source if you want to help with boot time.
4170
4171Run the ``ksize.py`` script from the top-level Linux build directory to
4172get an idea of what is making up the kernel:
4173::
4174
4175 $ cd top-level-linux-build-directory
4176 $ ksize.py > ksize.log
4177 $ cat ksize.log
4178
4179When you examine the log, you will see how much space is taken up with
4180the built-in ``.o`` files for drivers, networking, core kernel files,
4181filesystem, sound, and so forth. The sizes reported by the tool are
4182uncompressed, and thus will be smaller by a relatively constant factor
4183in a compressed kernel image. Look to reduce the areas that are large
4184and taking up around the "90% rule."
4185
4186To examine, or drill down, into any particular area, use the ``-d``
4187option with the script:
4188::
4189
4190 $ ksize.py -d > ksize.log
4191
4192Using this option
4193breaks out the individual file information for each area of the kernel
4194(e.g. drivers, networking, and so forth).
4195
4196Use your log file to see what you can eliminate from the kernel based on
4197features you can let go. For example, if you are not going to need
4198sound, you do not need any drivers that support sound.
4199
4200After figuring out what to eliminate, you need to reconfigure the kernel
4201to reflect those changes during the next build. You could run
4202``menuconfig`` and make all your changes at once. However, that makes it
4203difficult to see the effects of your individual eliminations and also
4204makes it difficult to replicate the changes for perhaps another target
4205device. A better method is to start with no configurations using
4206``allnoconfig``, create configuration fragments for individual changes,
4207and then manage the fragments into a single configuration file using
4208``merge_config.sh``. The tool makes it easy for you to iterate using the
4209configuration change and build cycle.
4210
4211Each time you make configuration changes, you need to rebuild the kernel
4212and check to see what impact your changes had on the overall size.
4213
4214Remove Package Management Requirements
4215~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4216
4217Packaging requirements add size to the image. One way to reduce the size
4218of the image is to remove all the packaging requirements from the image.
4219This reduction includes both removing the package manager and its unique
4220dependencies as well as removing the package management data itself.
4221
4222To eliminate all the packaging requirements for an image, be sure that
4223"package-management" is not part of your
4224:term:`IMAGE_FEATURES`
4225statement for the image. When you remove this feature, you are removing
4226the package manager as well as its dependencies from the root
4227filesystem.
4228
4229Look for Other Ways to Minimize Size
4230~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4231
4232Depending on your particular circumstances, other areas that you can
4233trim likely exist. The key to finding these areas is through tools and
4234methods described here combined with experimentation and iteration. Here
4235are a couple of areas to experiment with:
4236
4237- ``glibc``: In general, follow this process:
4238
4239 1. Remove ``glibc`` features from
4240 :term:`DISTRO_FEATURES`
4241 that you think you do not need.
4242
4243 2. Build your distribution.
4244
4245 3. If the build fails due to missing symbols in a package, determine
4246 if you can reconfigure the package to not need those features. For
4247 example, change the configuration to not support wide character
4248 support as is done for ``ncurses``. Or, if support for those
4249 characters is needed, determine what ``glibc`` features provide
4250 the support and restore the configuration.
4251
4252 4. Rebuild and repeat the process.
4253
4254- ``busybox``: For BusyBox, use a process similar as described for
4255 ``glibc``. A difference is you will need to boot the resulting system
4256 to see if you are able to do everything you expect from the running
4257 system. You need to be sure to integrate configuration fragments into
4258 Busybox because BusyBox handles its own core features and then allows
4259 you to add configuration fragments on top.
4260
4261Iterate on the Process
4262~~~~~~~~~~~~~~~~~~~~~~
4263
4264If you have not reached your goals on system size, you need to iterate
4265on the process. The process is the same. Use the tools and see just what
4266is taking up 90% of the root filesystem and the kernel. Decide what you
4267can eliminate without limiting your device beyond what you need.
4268
4269Depending on your system, a good place to look might be Busybox, which
4270provides a stripped down version of Unix tools in a single, executable
4271file. You might be able to drop virtual terminal services or perhaps
4272ipv6.
4273
4274Building Images for More than One Machine
4275-----------------------------------------
4276
4277A common scenario developers face is creating images for several
4278different machines that use the same software environment. In this
4279situation, it is tempting to set the tunings and optimization flags for
4280each build specifically for the targeted hardware (i.e. "maxing out" the
4281tunings). Doing so can considerably add to build times and package feed
4282maintenance collectively for the machines. For example, selecting tunes
4283that are extremely specific to a CPU core used in a system might enable
4284some micro optimizations in GCC for that particular system but would
4285otherwise not gain you much of a performance difference across the other
4286systems as compared to using a more general tuning across all the builds
4287(e.g. setting :term:`DEFAULTTUNE`
4288specifically for each machine's build). Rather than "max out" each
4289build's tunings, you can take steps that cause the OpenEmbedded build
4290system to reuse software across the various machines where it makes
4291sense.
4292
4293If build speed and package feed maintenance are considerations, you
4294should consider the points in this section that can help you optimize
4295your tunings to best consider build times and package feed maintenance.
4296
4297- *Share the Build Directory:* If at all possible, share the
4298 :term:`TMPDIR` across builds. The
4299 Yocto Project supports switching between different
4300 :term:`MACHINE` values in the same
4301 ``TMPDIR``. This practice is well supported and regularly used by
4302 developers when building for multiple machines. When you use the same
4303 ``TMPDIR`` for multiple machine builds, the OpenEmbedded build system
4304 can reuse the existing native and often cross-recipes for multiple
4305 machines. Thus, build time decreases.
4306
4307 .. note::
4308
4309 If :term:`DISTRO` settings change or fundamental configuration settings
4310 such as the filesystem layout, you need to work with a clean ``TMPDIR``.
4311 Sharing ``TMPDIR`` under these circumstances might work but since it is
4312 not guaranteed, you should use a clean ``TMPDIR``.
4313
4314- *Enable the Appropriate Package Architecture:* By default, the
4315 OpenEmbedded build system enables three levels of package
4316 architectures: "all", "tune" or "package", and "machine". Any given
4317 recipe usually selects one of these package architectures (types) for
4318 its output. Depending for what a given recipe creates packages,
4319 making sure you enable the appropriate package architecture can
4320 directly impact the build time.
4321
4322 A recipe that just generates scripts can enable "all" architecture
4323 because there are no binaries to build. To specifically enable "all"
4324 architecture, be sure your recipe inherits the
4325 :ref:`allarch <ref-classes-allarch>` class.
4326 This class is useful for "all" architectures because it configures
4327 many variables so packages can be used across multiple architectures.
4328
4329 If your recipe needs to generate packages that are machine-specific
4330 or when one of the build or runtime dependencies is already
4331 machine-architecture dependent, which makes your recipe also
4332 machine-architecture dependent, make sure your recipe enables the
4333 "machine" package architecture through the
4334 :term:`MACHINE_ARCH`
4335 variable:
4336 ::
4337
4338 PACKAGE_ARCH = "${MACHINE_ARCH}"
4339
4340 When you do not
4341 specifically enable a package architecture through the
4342 :term:`PACKAGE_ARCH`, The
4343 OpenEmbedded build system defaults to the
4344 :term:`TUNE_PKGARCH` setting:
4345 ::
4346
4347 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4348
4349- *Choose a Generic Tuning File if Possible:* Some tunes are more
4350 generic and can run on multiple targets (e.g. an ``armv5`` set of
4351 packages could run on ``armv6`` and ``armv7`` processors in most
4352 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4353 higher processors. You should realize, however, that advances on
4354 newer processor versions would not be used.
4355
4356 If you select the same tune for several different machines, the
4357 OpenEmbedded build system reuses software previously built, thus
4358 speeding up the overall build time. Realize that even though a new
4359 sysroot for each machine is generated, the software is not recompiled
4360 and only one package feed exists.
4361
4362- *Manage Granular Level Packaging:* Sometimes cases exist where
4363 injecting another level of package architecture beyond the three
4364 higher levels noted earlier can be useful. For example, consider how
4365 NXP (formerly Freescale) allows for the easy reuse of binary packages
4366 in their layer
4367 :yocto_git:`meta-freescale </meta-freescale/>`.
4368 In this example, the
4369 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
4370 class shares GPU packages for i.MX53 boards because all boards share
4371 the AMD GPU. The i.MX6-based boards can do the same because all
4372 boards share the Vivante GPU. This class inspects the BitBake
4373 datastore to identify if the package provides or depends on one of
4374 the sub-architecture values. If so, the class sets the
4375 :term:`PACKAGE_ARCH` value
4376 based on the ``MACHINE_SUBARCH`` value. If the package does not
4377 provide or depend on one of the sub-architecture values but it
4378 matches a value in the machine-specific filter, it sets
4379 :term:`MACHINE_ARCH`. This
4380 behavior reduces the number of packages built and saves build time by
4381 reusing binaries.
4382
4383- *Use Tools to Debug Issues:* Sometimes you can run into situations
4384 where software is being rebuilt when you think it should not be. For
4385 example, the OpenEmbedded build system might not be using shared
4386 state between machines when you think it should be. These types of
4387 situations are usually due to references to machine-specific
4388 variables such as :term:`MACHINE`,
4389 :term:`SERIAL_CONSOLES`,
4390 :term:`XSERVER`,
4391 :term:`MACHINE_FEATURES`,
4392 and so forth in code that is supposed to only be tune-specific or
4393 when the recipe depends
4394 (:term:`DEPENDS`,
4395 :term:`RDEPENDS`,
4396 :term:`RRECOMMENDS`,
4397 :term:`RSUGGESTS`, and so forth)
4398 on some other recipe that already has
4399 :term:`PACKAGE_ARCH` defined
4400 as "${MACHINE_ARCH}".
4401
4402 .. note::
4403
4404 Patches to fix any issues identified are most welcome as these
4405 issues occasionally do occur.
4406
4407 For such cases, you can use some tools to help you sort out the
4408 situation:
4409
4410 - ``state-diff-machines.sh``*:* You can find this tool in the
4411 ``scripts`` directory of the Source Repositories. See the comments
4412 in the script for information on how to use the tool.
4413
4414 - *BitBake's "-S printdiff" Option:* Using this option causes
4415 BitBake to try to establish the closest signature match it can
4416 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4417 over the matches to determine the stamps and delta where these two
4418 stamp trees diverge.
4419
4420Building Software from an External Source
4421-----------------------------------------
4422
4423By default, the OpenEmbedded build system uses the
4424:term:`Build Directory` when building source
4425code. The build process involves fetching the source files, unpacking
4426them, and then patching them if necessary before the build takes place.
4427
4428Situations exist where you might want to build software from source
4429files that are external to and thus outside of the OpenEmbedded build
4430system. For example, suppose you have a project that includes a new BSP
4431with a heavily customized kernel. And, you want to minimize exposing the
4432build system to the development team so that they can focus on their
4433project and maintain everyone's workflow as much as possible. In this
4434case, you want a kernel source directory on the development machine
4435where the development occurs. You want the recipe's
4436:term:`SRC_URI` variable to point to
4437the external directory and use it as is, not copy it.
4438
4439To build from software that comes from an external source, all you need
4440to do is inherit the
4441:ref:`externalsrc <ref-classes-externalsrc>` class
4442and then set the
4443:term:`EXTERNALSRC` variable to
4444point to your external source code. Here are the statements to put in
4445your ``local.conf`` file:
4446::
4447
4448 INHERIT += "externalsrc"
4449 EXTERNALSRC_pn-myrecipe = "path-to-your-source-tree"
4450
4451This next example shows how to accomplish the same thing by setting
4452``EXTERNALSRC`` in the recipe itself or in the recipe's append file:
4453::
4454
4455 EXTERNALSRC = "path"
4456 EXTERNALSRC_BUILD = "path"
4457
4458.. note::
4459
4460 In order for these settings to take effect, you must globally or
4461 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
4462 class.
4463
4464By default, ``externalsrc.bbclass`` builds the source code in a
4465directory separate from the external source directory as specified by
4466:term:`EXTERNALSRC`. If you need
4467to have the source built in the same directory in which it resides, or
4468some other nominated directory, you can set
4469:term:`EXTERNALSRC_BUILD`
4470to point to that directory:
4471::
4472
4473 EXTERNALSRC_BUILD_pn-myrecipe = "path-to-your-source-tree"
4474
4475Replicating a Build Offline
4476---------------------------
4477
4478It can be useful to take a "snapshot" of upstream sources used in a
4479build and then use that "snapshot" later to replicate the build offline.
4480To do so, you need to first prepare and populate your downloads
4481directory your "snapshot" of files. Once your downloads directory is
4482ready, you can use it at any time and from any machine to replicate your
4483build.
4484
4485Follow these steps to populate your Downloads directory:
4486
44871. *Create a Clean Downloads Directory:* Start with an empty downloads
4488 directory (:term:`DL_DIR`). You
4489 start with an empty downloads directory by either removing the files
4490 in the existing directory or by setting ``DL_DIR`` to point to either
4491 an empty location or one that does not yet exist.
4492
44932. *Generate Tarballs of the Source Git Repositories:* Edit your
4494 ``local.conf`` configuration file as follows:
4495 ::
4496
4497 DL_DIR = "/home/your-download-dir/"
4498 BB_GENERATE_MIRROR_TARBALLS = "1"
4499
4500 During
4501 the fetch process in the next step, BitBake gathers the source files
4502 and creates tarballs in the directory pointed to by ``DL_DIR``. See
4503 the
4504 :term:`BB_GENERATE_MIRROR_TARBALLS`
4505 variable for more information.
4506
45073. *Populate Your Downloads Directory Without Building:* Use BitBake to
4508 fetch your sources but inhibit the build:
4509 ::
4510
4511 $ bitbake target --runonly=fetch
4512
4513 The downloads directory (i.e. ``${DL_DIR}``) now has
4514 a "snapshot" of the source files in the form of tarballs, which can
4515 be used for the build.
4516
45174. *Optionally Remove Any Git or other SCM Subdirectories From the
4518 Downloads Directory:* If you want, you can clean up your downloads
4519 directory by removing any Git or other Source Control Management
4520 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4521 already contain these subdirectories.
4522
4523Once your downloads directory has everything it needs regarding source
4524files, you can create your "own-mirror" and build your target.
4525Understand that you can use the files to build the target offline from
4526any machine and at any time.
4527
4528Follow these steps to build your target using the files in the downloads
4529directory:
4530
45311. *Using Local Files Only:* Inside your ``local.conf`` file, add the
4532 :term:`SOURCE_MIRROR_URL`
4533 variable, inherit the
4534 :ref:`own-mirrors <ref-classes-own-mirrors>`
4535 class, and use the
4536 :term:`bitbake:BB_NO_NETWORK`
4537 variable to your ``local.conf``.
4538 ::
4539
4540 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4541 INHERIT += "own-mirrors"
4542 BB_NO_NETWORK = "1"
4543
4544 The ``SOURCE_MIRROR_URL`` and ``own-mirror``
4545 class set up the system to use the downloads directory as your "own
4546 mirror". Using the ``BB_NO_NETWORK`` variable makes sure that
4547 BitBake's fetching process in step 3 stays local, which means files
4548 from your "own-mirror" are used.
4549
45502. *Start With a Clean Build:* You can start with a clean build by
4551 removing the
4552 ``${``\ :term:`TMPDIR`\ ``}``
4553 directory or using a new :term:`Build Directory`.
4554
45553. *Build Your Target:* Use BitBake to build your target:
4556 ::
4557
4558 $ bitbake target
4559
4560 The build completes using the known local "snapshot" of source
4561 files from your mirror. The resulting tarballs for your "snapshot" of
4562 source files are in the downloads directory.
4563
4564 .. note::
4565
4566 The offline build does not work if recipes attempt to find the
4567 latest version of software by setting
4568 :term:`SRCREV` to
4569 ``${``\ :term:`AUTOREV`\ ``}``:
4570 ::
4571
4572 SRCREV = "${AUTOREV}"
4573
4574 When a recipe sets ``SRCREV`` to
4575 ``${AUTOREV}``, the build system accesses the network in an
4576 attempt to determine the latest version of software from the SCM.
4577 Typically, recipes that use ``AUTOREV`` are custom or modified
4578 recipes. Recipes that reside in public repositories usually do not
4579 use ``AUTOREV``.
4580
4581 If you do have recipes that use ``AUTOREV``, you can take steps to
4582 still use the recipes in an offline build. Do the following:
4583
4584 1. Use a configuration generated by enabling `build
4585 history <#maintaining-build-output-quality>`__.
4586
4587 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
4588 stored ``SRCREV`` values from the build's history. For more
4589 information on collecting these values, see the "`Build History
4590 Package Information <#build-history-package-information>`__"
4591 section.
4592
4593 3. Once you have the correct source revisions, you can modify
4594 those recipes to to set ``SRCREV`` to specific versions of the
4595 software.
4596
4597Speeding Up a Build
4598===================
4599
4600Build time can be an issue. By default, the build system uses simple
4601controls to try and maximize build efficiency. In general, the default
4602settings for all the following variables result in the most efficient
4603build times when dealing with single socket systems (i.e. a single CPU).
4604If you have multiple CPUs, you might try increasing the default values
4605to gain more speed. See the descriptions in the glossary for each
4606variable for more information:
4607
4608- :term:`BB_NUMBER_THREADS`:
4609 The maximum number of threads BitBake simultaneously executes.
4610
4611- :term:`bitbake:BB_NUMBER_PARSE_THREADS`:
4612 The number of threads BitBake uses during parsing.
4613
4614- :term:`PARALLEL_MAKE`: Extra
4615 options passed to the ``make`` command during the
4616 :ref:`ref-tasks-compile` task in
4617 order to specify parallel compilation on the local build host.
4618
4619- :term:`PARALLEL_MAKEINST`:
4620 Extra options passed to the ``make`` command during the
4621 :ref:`ref-tasks-install` task in
4622 order to specify parallel installation on the local build host.
4623
4624As mentioned, these variables all scale to the number of processor cores
4625available on the build system. For single socket systems, this
4626auto-scaling ensures that the build system fundamentally takes advantage
4627of potential parallel operations during the build based on the build
4628machine's capabilities.
4629
4630Following are additional factors that can affect build speed:
4631
4632- File system type: The file system type that the build is being
4633 performed on can also influence performance. Using ``ext4`` is
4634 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4635 improved features such as extents.
4636
4637- Disabling the updating of access time using ``noatime``: The
4638 ``noatime`` mount option prevents the build system from updating file
4639 and directory access times.
4640
4641- Setting a longer commit: Using the "commit=" mount option increases
4642 the interval in seconds between disk cache writes. Changing this
4643 interval from the five second default to something longer increases
4644 the risk of data loss but decreases the need to write to the disk,
4645 thus increasing the build performance.
4646
4647- Choosing the packaging backend: Of the available packaging backends,
4648 IPK is the fastest. Additionally, selecting a singular packaging
4649 backend also helps.
4650
4651- Using ``tmpfs`` for :term:`TMPDIR`
4652 as a temporary file system: While this can help speed up the build,
4653 the benefits are limited due to the compiler using ``-pipe``. The
4654 build system goes to some lengths to avoid ``sync()`` calls into the
4655 file system on the principle that if there was a significant failure,
4656 the :term:`Build Directory`
4657 contents could easily be rebuilt.
4658
4659- Inheriting the
4660 :ref:`rm_work <ref-classes-rm-work>` class:
4661 Inheriting this class has shown to speed up builds due to
4662 significantly lower amounts of data stored in the data cache as well
4663 as on disk. Inheriting this class also makes cleanup of
4664 :term:`TMPDIR` faster, at the
4665 expense of being easily able to dive into the source code. File
4666 system maintainers have recommended that the fastest way to clean up
4667 large numbers of files is to reformat partitions rather than delete
4668 files due to the linear nature of partitions. This, of course,
4669 assumes you structure the disk partitions and file systems in a way
4670 that this is practical.
4671
4672Aside from the previous list, you should keep some trade offs in mind
4673that can help you speed up the build:
4674
4675- Remove items from
4676 :term:`DISTRO_FEATURES`
4677 that you might not need.
4678
4679- Exclude debug symbols and other debug information: If you do not need
4680 these symbols and other debug information, disabling the ``*-dbg``
4681 package generation can speed up the build. You can disable this
4682 generation by setting the
4683 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4684 variable to "1".
4685
4686- Disable static library generation for recipes derived from
4687 ``autoconf`` or ``libtool``: Following is an example showing how to
4688 disable static libraries and still provide an override to handle
4689 exceptions:
4690 ::
4691
4692 STATICLIBCONF = "--disable-static"
4693 STATICLIBCONF_sqlite3-native = ""
4694 EXTRA_OECONF += "${STATICLIBCONF}"
4695
4696 .. note::
4697
4698 - Some recipes need static libraries in order to work correctly
4699 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4700 as in the previous example, account for these kinds of
4701 exceptions.
4702
4703 - Some packages have packaging code that assumes the presence of
4704 the static libraries. If so, you might need to exclude them as
4705 well.
4706
4707Working With Libraries
4708======================
4709
4710Libraries are an integral part of your system. This section describes
4711some common practices you might find helpful when working with libraries
4712to build your system:
4713
4714- `How to include static library
4715 files <#including-static-library-files>`__
4716
4717- `How to use the Multilib feature to combine multiple versions of
4718 library files into a single
4719 image <#combining-multiple-versions-library-files-into-one-image>`__
4720
4721- `How to install multiple versions of the same library in parallel on
4722 the same
4723 system <#installing-multiple-versions-of-the-same-library>`__
4724
4725Including Static Library Files
4726------------------------------
4727
4728If you are building a library and the library offers static linking, you
4729can control which static library files (``*.a`` files) get included in
4730the built library.
4731
4732The :term:`PACKAGES` and
4733:term:`FILES_* <FILES>` variables in the
4734``meta/conf/bitbake.conf`` configuration file define how files installed
4735by the ``do_install`` task are packaged. By default, the ``PACKAGES``
4736variable includes ``${PN}-staticdev``, which represents all static
4737library files.
4738
4739.. note::
4740
4741 Some previously released versions of the Yocto Project defined the
4742 static library files through ``${PN}-dev``.
4743
4744Following is part of the BitBake configuration file, where you can see
4745how the static library files are defined:
4746::
4747
4748 PACKAGE_BEFORE_PN ?= ""
4749 PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
4750 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4751 FILES = ""
4752
4753 FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
4754 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4755 ${base_bindir}/* ${base_sbindir}/* \
4756 ${base_libdir}/*${SOLIBS} \
4757 ${base_prefix}/lib/udev/rules.d ${prefix}/lib/udev/rules.d \
4758 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4759 ${datadir}/pixmaps ${datadir}/applications \
4760 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4761 ${libdir}/bonobo/servers"
4762
4763 FILES_${PN}-bin = "${bindir}/* ${sbindir}/*"
4764
4765 FILES_${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
4766 ${datadir}/gnome/help"
4767 SECTION_${PN}-doc = "doc"
4768
4769 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
4770 FILES_${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
4771 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4772 ${datadir}/aclocal ${base_libdir}/*.o \
4773 ${libdir}/${BPN}/*.la ${base_libdir}/*.la"
4774 SECTION_${PN}-dev = "devel"
4775 ALLOW_EMPTY_${PN}-dev = "1"
4776 RDEPENDS_${PN}-dev = "${PN} (= ${EXTENDPKGV})"
4777
4778 FILES_${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4779 SECTION_${PN}-staticdev = "devel"
4780 RDEPENDS_${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
4781
4782Combining Multiple Versions of Library Files into One Image
4783-----------------------------------------------------------
4784
4785The build system offers the ability to build libraries with different
4786target optimizations or architecture formats and combine these together
4787into one system image. You can link different binaries in the image
4788against the different libraries as needed for specific use cases. This
4789feature is called "Multilib".
4790
4791An example would be where you have most of a system compiled in 32-bit
4792mode using 32-bit libraries, but you have something large, like a
4793database engine, that needs to be a 64-bit application and uses 64-bit
4794libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4795libraries.
4796
4797While the Multilib feature is most commonly used for 32 and 64-bit
4798differences, the approach the build system uses facilitates different
4799target optimizations. You could compile some binaries to use one set of
4800libraries and other binaries to use a different set of libraries. The
4801libraries could differ in architecture, compiler options, or other
4802optimizations.
4803
4804Several examples exist in the ``meta-skeleton`` layer found in the
4805:term:`Source Directory`:
4806
4807- ``conf/multilib-example.conf`` configuration file
4808
4809- ``conf/multilib-example2.conf`` configuration file
4810
4811- ``recipes-multilib/images/core-image-multilib-example.bb`` recipe
4812
4813Preparing to Use Multilib
4814~~~~~~~~~~~~~~~~~~~~~~~~~
4815
4816User-specific requirements drive the Multilib feature. Consequently,
4817there is no one "out-of-the-box" configuration that likely exists to
4818meet your needs.
4819
4820In order to enable Multilib, you first need to ensure your recipe is
4821extended to support multiple libraries. Many standard recipes are
4822already extended and support multiple libraries. You can check in the
4823``meta/conf/multilib.conf`` configuration file in the
4824:term:`Source Directory` to see how this is
4825done using the
4826:term:`BBCLASSEXTEND` variable.
4827Eventually, all recipes will be covered and this list will not be
4828needed.
4829
4830For the most part, the Multilib class extension works automatically to
4831extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
4832``MLPREFIX`` is the particular multilib (e.g. "lib32-" or "lib64-").
4833Standard variables such as
4834:term:`DEPENDS`,
4835:term:`RDEPENDS`,
4836:term:`RPROVIDES`,
4837:term:`RRECOMMENDS`,
4838:term:`PACKAGES`, and
4839:term:`PACKAGES_DYNAMIC` are
4840automatically extended by the system. If you are extending any manual
4841code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
4842those names are extended correctly. This automatic extension code
4843resides in ``multilib.bbclass``.
4844
4845Using Multilib
4846~~~~~~~~~~~~~~
4847
4848After you have set up the recipes, you need to define the actual
4849combination of multiple libraries you want to build. You accomplish this
4850through your ``local.conf`` configuration file in the
4851:term:`Build Directory`. An example
4852configuration would be as follows:
4853::
4854
4855 MACHINE = "qemux86-64"
4856 require conf/multilib.conf
4857 MULTILIBS = "multilib:lib32"
4858 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
4859 IMAGE_INSTALL_append = "lib32-glib-2.0"
4860
4861This example enables an additional library named
4862``lib32`` alongside the normal target packages. When combining these
4863"lib32" alternatives, the example uses "x86" for tuning. For information
4864on this particular tuning, see
4865``meta/conf/machine/include/ia32/arch-ia32.inc``.
4866
4867The example then includes ``lib32-glib-2.0`` in all the images, which
4868illustrates one method of including a multiple library dependency. You
4869can use a normal image build to include this dependency, for example:
4870::
4871
4872 $ bitbake core-image-sato
4873
4874You can also build Multilib packages
4875specifically with a command like this:
4876::
4877
4878 $ bitbake lib32-glib-2.0
4879
4880Additional Implementation Details
4881~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4882
4883Generic implementation details as well as details that are specific to
4884package management systems exist. Following are implementation details
4885that exist regardless of the package management system:
4886
4887- The typical convention used for the class extension code as used by
4888 Multilib assumes that all package names specified in
4889 :term:`PACKAGES` that contain
4890 ``${PN}`` have ``${PN}`` at the start of the name. When that
4891 convention is not followed and ``${PN}`` appears at the middle or the
4892 end of a name, problems occur.
4893
4894- The :term:`TARGET_VENDOR`
4895 value under Multilib will be extended to "-vendormlmultilib" (e.g.
4896 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
4897 slightly unwieldy contraction is that any "-" characters in the
4898 vendor string presently break Autoconf's ``config.sub``, and other
4899 separators are problematic for different reasons.
4900
4901For the RPM Package Management System, the following implementation
4902details exist:
4903
4904- A unique architecture is defined for the Multilib packages, along
4905 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
4906 :term:`Build Directory`. For
4907 example, consider ``lib32`` in a ``qemux86-64`` image. The possible
4908 architectures in the system are "all", "qemux86_64",
4909 "lib32_qemux86_64", and "lib32_x86".
4910
4911- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
4912 packaging. The naming for a normal RPM package and a Multilib RPM
4913 package in a ``qemux86-64`` system resolves to something similar to
4914 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
4915 respectively.
4916
4917- When installing a Multilib image, the RPM backend first installs the
4918 base image and then installs the Multilib libraries.
4919
4920- The build system relies on RPM to resolve the identical files in the
4921 two (or more) Multilib packages.
4922
4923For the IPK Package Management System, the following implementation
4924details exist:
4925
4926- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
4927 packaging. The naming for a normal RPM package and a Multilib IPK
4928 package in a ``qemux86-64`` system resolves to something like
4929 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw_x86.ipk``,
4930 respectively.
4931
4932- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
4933 packages with and without the Multilib feature can exist in the same
4934 folder due to the ``${PN}`` differences.
4935
4936- IPK defines a sanity check for Multilib installation using certain
4937 rules for file comparison, overridden, etc.
4938
4939Installing Multiple Versions of the Same Library
4940------------------------------------------------
4941
4942Situations can exist where you need to install and use multiple versions
4943of the same library on the same system at the same time. These
4944situations almost always exist when a library API changes and you have
4945multiple pieces of software that depend on the separate versions of the
4946library. To accommodate these situations, you can install multiple
4947versions of the same library in parallel on the same system.
4948
4949The process is straightforward as long as the libraries use proper
4950versioning. With properly versioned libraries, all you need to do to
4951individually specify the libraries is create separate, appropriately
4952named recipes where the :term:`PN` part of
4953the name includes a portion that differentiates each library version
4954(e.g. the major part of the version number). Thus, instead of having a
4955single recipe that loads one version of a library (e.g. ``clutter``),
4956you provide multiple recipes that result in different versions of the
4957libraries you want. As an example, the following two recipes would allow
4958the two separate versions of the ``clutter`` library to co-exist on the
4959same system:
4960
4961.. code-block:: none
4962
4963 clutter-1.6_1.6.20.bb
4964 clutter-1.8_1.8.4.bb
4965
4966Additionally, if
4967you have other recipes that depend on a given library, you need to use
4968the :term:`DEPENDS` variable to
4969create the dependency. Continuing with the same example, if you want to
4970have a recipe depend on the 1.8 version of the ``clutter`` library, use
4971the following in your recipe:
4972::
4973
4974 DEPENDS = "clutter-1.8"
4975
4976Using x32 psABI
4977===============
4978
4979x32 processor-specific Application Binary Interface (`x32
4980psABI <https://software.intel.com/en-us/node/628948>`__) is a native
498132-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
4982ABI defines the calling conventions between functions in a processing
4983environment. The interface determines what registers are used and what
4984the sizes are for various C data types.
4985
4986Some processing environments prefer using 32-bit applications even when
4987running on Intel 64-bit platforms. Consider the i386 psABI, which is a
4988very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
4989provide efficient use and access of the Intel 64-bit processor
4990resources, leaving the system underutilized. Now consider the x86_64
4991psABI. This ABI is newer and uses 64-bits for data sizes and program
4992pointers. The extra bits increase the footprint size of the programs,
4993libraries, and also increases the memory and file system size
4994requirements. Executing under the x32 psABI enables user programs to
4995utilize CPU and system resources more efficiently while keeping the
4996memory footprint of the applications low. Extra bits are used for
4997registers but not for addressing mechanisms.
4998
4999The Yocto Project supports the final specifications of x32 psABI as
5000follows:
5001
5002- You can create packages and images in x32 psABI format on x86_64
5003 architecture targets.
5004
5005- You can successfully build recipes with the x32 toolchain.
5006
5007- You can create and boot ``core-image-minimal`` and
5008 ``core-image-sato`` images.
5009
5010- RPM Package Manager (RPM) support exists for x32 binaries.
5011
5012- Support for large images exists.
5013
5014To use the x32 psABI, you need to edit your ``conf/local.conf``
5015configuration file as follows:
5016::
5017
5018 MACHINE = "qemux86-64"
5019 DEFAULTTUNE = "x86-64-x32"
5020 baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE') \
5021 or 'INVALID')) or 'lib'}"
5022
5023Once you have set
5024up your configuration file, use BitBake to build an image that supports
5025the x32 psABI. Here is an example:
5026::
5027
5028 $ bitbake core-image-sato
5029
5030Enabling GObject Introspection Support
5031======================================
5032
5033`GObject
5034introspection <https://wiki.gnome.org/Projects/GObjectIntrospection>`__
5035is the standard mechanism for accessing GObject-based software from
5036runtime environments. GObject is a feature of the GLib library that
5037provides an object framework for the GNOME desktop and related software.
5038GObject Introspection adds information to GObject that allows objects
5039created within it to be represented across different programming
5040languages. If you want to construct GStreamer pipelines using Python, or
5041control UPnP infrastructure using Javascript and GUPnP, GObject
5042introspection is the only way to do it.
5043
5044This section describes the Yocto Project support for generating and
5045packaging GObject introspection data. GObject introspection data is a
5046description of the API provided by libraries built on top of GLib
5047framework, and, in particular, that framework's GObject mechanism.
5048GObject Introspection Repository (GIR) files go to ``-dev`` packages,
5049``typelib`` files go to main packages as they are packaged together with
5050libraries that are introspected.
5051
5052The data is generated when building such a library, by linking the
5053library with a small executable binary that asks the library to describe
5054itself, and then executing the binary and processing its output.
5055
5056Generating this data in a cross-compilation environment is difficult
5057because the library is produced for the target architecture, but its
5058code needs to be executed on the build host. This problem is solved with
5059the OpenEmbedded build system by running the code through QEMU, which
5060allows precisely that. Unfortunately, QEMU does not always work
5061perfectly as mentioned in the "`Known Issues <#known-issues>`__"
5062section.
5063
5064Enabling the Generation of Introspection Data
5065---------------------------------------------
5066
5067Enabling the generation of introspection data (GIR files) in your
5068library package involves the following:
5069
50701. Inherit the
5071 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
5072 class.
5073
50742. Make sure introspection is not disabled anywhere in the recipe or
5075 from anything the recipe includes. Also, make sure that
5076 "gobject-introspection-data" is not in
5077 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5078 and that "qemu-usermode" is not in
5079 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5080 If either of these conditions exist, nothing will happen.
5081
50823. Try to build the recipe. If you encounter build errors that look like
5083 something is unable to find ``.so`` libraries, check where these
5084 libraries are located in the source tree and add the following to the
5085 recipe:
5086 ::
5087
5088 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5089
5090 .. note::
5091
5092 See recipes in the ``oe-core`` repository that use that
5093 ``GIR_EXTRA_LIBS_PATH`` variable as an example.
5094
50954. Look for any other errors, which probably mean that introspection
5096 support in a package is not entirely standard, and thus breaks down
5097 in a cross-compilation environment. For such cases, custom-made fixes
5098 are needed. A good place to ask and receive help in these cases is
5099 the :ref:`Yocto Project mailing
5100 lists <resources-mailinglist>`.
5101
5102.. note::
5103
5104 Using a library that no longer builds against the latest Yocto
5105 Project release and prints introspection related errors is a good
5106 candidate for the previous procedure.
5107
5108Disabling the Generation of Introspection Data
5109----------------------------------------------
5110
5111You might find that you do not want to generate introspection data. Or,
5112perhaps QEMU does not work on your build host and target architecture
5113combination. If so, you can use either of the following methods to
5114disable GIR file generations:
5115
5116- Add the following to your distro configuration:
5117 ::
5118
5119 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5120
5121 Adding this statement disables generating introspection data using
5122 QEMU but will still enable building introspection tools and libraries
5123 (i.e. building them does not require the use of QEMU).
5124
5125- Add the following to your machine configuration:
5126 ::
5127
5128 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5129
5130 Adding this statement disables the use of QEMU when building packages for your
5131 machine. Currently, this feature is used only by introspection
5132 recipes and has the same effect as the previously described option.
5133
5134 .. note::
5135
5136 Future releases of the Yocto Project might have other features
5137 affected by this option.
5138
5139If you disable introspection data, you can still obtain it through other
5140means such as copying the data from a suitable sysroot, or by generating
5141it on the target hardware. The OpenEmbedded build system does not
5142currently provide specific support for these techniques.
5143
5144Testing that Introspection Works in an Image
5145--------------------------------------------
5146
5147Use the following procedure to test if generating introspection data is
5148working in an image:
5149
51501. Make sure that "gobject-introspection-data" is not in
5151 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5152 and that "qemu-usermode" is not in
5153 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5154
51552. Build ``core-image-sato``.
5156
51573. Launch a Terminal and then start Python in the terminal.
5158
51594. Enter the following in the terminal:
5160 ::
5161
5162 >>> from gi.repository import GLib
5163 >>> GLib.get_host_name()
5164
51655. For something a little more advanced, enter the following see:
5166 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
5167
5168Known Issues
5169------------
5170
5171The following know issues exist for GObject Introspection Support:
5172
5173- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5174 introspection data on that architecture.
5175
5176- x32 is not supported by QEMU. Consequently, introspection data is
5177 disabled.
5178
5179- musl causes transient GLib binaries to crash on assertion failures.
5180 Consequently, generating introspection data is disabled.
5181
5182- Because QEMU is not able to run the binaries correctly, introspection
5183 is disabled for some specific packages under specific architectures
5184 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5185
5186- QEMU usermode might not work properly when running 64-bit binaries
5187 under 32-bit host machines. In particular, "qemumips64" is known to
5188 not work under i686.
5189
5190Optionally Using an External Toolchain
5191======================================
5192
5193You might want to use an external toolchain as part of your development.
5194If this is the case, the fundamental steps you need to accomplish are as
5195follows:
5196
5197- Understand where the installed toolchain resides. For cases where you
5198 need to build the external toolchain, you would need to take separate
5199 steps to build and install the toolchain.
5200
5201- Make sure you add the layer that contains the toolchain to your
5202 ``bblayers.conf`` file through the
5203 :term:`BBLAYERS` variable.
5204
5205- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5206 to the location in which you installed the toolchain.
5207
5208A good example of an external toolchain used with the Yocto Project is
5209Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5210to use that particular layer in the ``README`` file at
5211https://github.com/MentorEmbedded/meta-sourcery/. You can find
5212further information by reading about the
5213:term:`TCMODE` variable in the Yocto
5214Project Reference Manual's variable glossary.
5215
5216Creating Partitioned Images Using Wic
5217=====================================
5218
5219Creating an image for a particular hardware target using the
5220OpenEmbedded build system does not necessarily mean you can boot that
5221image as is on your device. Physical devices accept and boot images in
5222various ways depending on the specifics of the device. Usually,
5223information about the hardware can tell you what image format the device
5224requires. Should your device require multiple partitions on an SD card,
5225flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5226create the properly partitioned image.
5227
5228The ``wic`` command generates partitioned images from existing
5229OpenEmbedded build artifacts. Image generation is driven by partitioning
5230commands contained in an Openembedded kickstart file (``.wks``)
5231specified either directly on the command line or as one of a selection
5232of canned kickstart files as shown with the ``wic list images`` command
5233in the "`Using an Existing Kickstart
5234File <#using-a-provided-kickstart-file>`__" section. When you apply the
5235command to a given set of build artifacts, the result is an image or set
5236of images that can be directly written onto media and used on a
5237particular system.
5238
5239.. note::
5240
5241 For a kickstart file reference, see the
5242 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
5243 Chapter in the Yocto Project Reference Manual.
5244
5245The ``wic`` command and the infrastructure it is based on is by
5246definition incomplete. The purpose of the command is to allow the
5247generation of customized images, and as such, was designed to be
5248completely extensible through a plugin interface. See the "`Using the
5249Wic PlugIn Interface <#wic-using-the-wic-plugin-interface>`__" section
5250for information on these plugins.
5251
5252This section provides some background information on Wic, describes what
5253you need to have in place to run the tool, provides instruction on how
5254to use the Wic utility, provides information on using the Wic plugins
5255interface, and provides several examples that show how to use Wic.
5256
5257Background
5258----------
5259
5260This section provides some background on the Wic utility. While none of
5261this information is required to use Wic, you might find it interesting.
5262
5263- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5264 "oe" diphthong in "oeic" was promoted to the letter "w", because
5265 "oeic" is both difficult to remember and to pronounce.
5266
5267- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5268 The Wic implementation has been heavily modified to make direct use
5269 of OpenEmbedded build artifacts instead of package installation and
5270 configuration, which are already incorporated within the OpenEmbedded
5271 artifacts.
5272
5273- Wic is a completely independent standalone utility that initially
5274 provides easier-to-use and more flexible replacements for an existing
5275 functionality in OE-Core's
5276 :ref:`image-live <ref-classes-image-live>`
5277 class. The difference between Wic and those examples is that with Wic
5278 the functionality of those scripts is implemented by a
5279 general-purpose partitioning language, which is based on Redhat
5280 kickstart syntax.
5281
5282Requirements
5283------------
5284
5285In order to use the Wic utility with the OpenEmbedded Build system, your
5286system needs to meet the following requirements:
5287
5288- The Linux distribution on your development host must support the
5289 Yocto Project. See the ":ref:`detailed-supported-distros`"
5290 section in the Yocto Project Reference Manual for the list of
5291 distributions that support the Yocto Project.
5292
5293- The standard system utilities, such as ``cp``, must be installed on
5294 your development host system.
5295
5296- You must have sourced the build environment setup script (i.e.
5297 :ref:`structure-core-script`) found in the
5298 :term:`Build Directory`.
5299
5300- You need to have the build artifacts already available, which
5301 typically means that you must have already created an image using the
5302 Openembedded build system (e.g. ``core-image-minimal``). While it
5303 might seem redundant to generate an image in order to create an image
5304 using Wic, the current version of Wic requires the artifacts in the
5305 form generated by the OpenEmbedded build system.
5306
5307- You must build several native tools, which are built to run on the
5308 build system:
5309 ::
5310
5311 $ bitbake parted-native dosfstools-native mtools-native
5312
5313- Include "wic" as part of the
5314 :term:`IMAGE_FSTYPES`
5315 variable.
5316
5317- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
5318 as part of the :term:`WKS_FILE` variable
5319
5320Getting Help
5321------------
5322
5323You can get general help for the ``wic`` command by entering the ``wic``
5324command by itself or by entering the command with a help argument as
5325follows:
5326::
5327
5328 $ wic -h
5329 $ wic --help
5330 $ wic help
5331
5332Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5333``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
5334commands except "help" by using the following form:
5335::
5336
5337 $ wic help command
5338
5339For example, the following command returns help for the ``write``
5340command:
5341::
5342
5343 $ wic help write
5344
5345Wic supports help for three topics: ``overview``, ``plugins``, and
5346``kickstart``. You can get help for any topic using the following form:
5347::
5348
5349 $ wic help topic
5350
5351For example, the following returns overview help for Wic:
5352::
5353
5354 $ wic help overview
5355
5356One additional level of help exists for Wic. You can get help on
5357individual images through the ``list`` command. You can use the ``list``
5358command to return the available Wic images as follows:
5359::
5360
5361 $ wic list images
5362 genericx86 Create an EFI disk image for genericx86*
5363 beaglebone-yocto Create SD card image for Beaglebone
5364 edgerouter Create SD card image for Edgerouter
5365 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5366 directdisk-gpt Create a 'pcbios' direct disk image
5367 mkefidisk Create an EFI disk image
5368 directdisk Create a 'pcbios' direct disk image
5369 systemd-bootdisk Create an EFI disk image with systemd-boot
5370 mkhybridiso Create a hybrid ISO image
5371 sdimage-bootpart Create SD card image with a boot partition
5372 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5373 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5374
5375Once you know the list of available
5376Wic images, you can use ``help`` with the command to get help on a
5377particular image. For example, the following command returns help on the
5378"beaglebone-yocto" image:
5379::
5380
5381 $ wic list beaglebone-yocto help
5382
5383 Creates a partitioned SD card image for Beaglebone.
5384 Boot files are located in the first vfat partition.
5385
5386Operational Modes
5387-----------------
5388
5389You can use Wic in two different modes, depending on how much control
5390you need for specifying the Openembedded build artifacts that are used
5391for creating the image: Raw and Cooked:
5392
5393- *Raw Mode:* You explicitly specify build artifacts through Wic
5394 command-line arguments.
5395
5396- *Cooked Mode:* The current
5397 :term:`MACHINE` setting and image
5398 name are used to automatically locate and provide the build
5399 artifacts. You just supply a kickstart file and the name of the image
5400 from which to use artifacts.
5401
5402Regardless of the mode you use, you need to have the build artifacts
5403ready and available.
5404
5405Raw Mode
5406~~~~~~~~
5407
5408Running Wic in raw mode allows you to specify all the partitions through
5409the ``wic`` command line. The primary use for raw mode is if you have
5410built your kernel outside of the Yocto Project
5411:term:`Build Directory`. In other words, you
5412can point to arbitrary kernel, root filesystem locations, and so forth.
5413Contrast this behavior with cooked mode where Wic looks in the Build
5414Directory (e.g. ``tmp/deploy/images/``\ machine).
5415
5416The general form of the ``wic`` command in raw mode is:
5417::
5418
5419 $ wic create wks_file options ...
5420
5421 Where:
5422
5423 wks_file:
5424 An OpenEmbedded kickstart file. You can provide
5425 your own custom file or use a file from a set of
5426 existing files as described by further options.
5427
5428 optional arguments:
5429 -h, --help show this help message and exit
5430 -o OUTDIR, --outdir OUTDIR
5431 name of directory to create image in
5432 -e IMAGE_NAME, --image-name IMAGE_NAME
5433 name of the image to use the artifacts from e.g. core-
5434 image-sato
5435 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5436 path to the /rootfs dir to use as the .wks rootfs
5437 source
5438 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5439 path to the dir containing the boot artifacts (e.g.
5440 /EFI or /syslinux dirs) to use as the .wks bootimg
5441 source
5442 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5443 path to the dir containing the kernel to use in the
5444 .wks bootimg
5445 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5446 path to the native sysroot containing the tools to use
5447 to build the image
5448 -s, --skip-build-check
5449 skip the build check
5450 -f, --build-rootfs build rootfs
5451 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5452 compress image with specified compressor
5453 -m, --bmap generate .bmap
5454 --no-fstab-update Do not change fstab file.
5455 -v VARS_DIR, --vars VARS_DIR
5456 directory with <image>.env files that store bitbake
5457 variables
5458 -D, --debug output debug information
5459
5460.. note::
5461
5462 You do not need root privileges to run Wic. In fact, you should not
5463 run as root when using the utility.
5464
5465Cooked Mode
5466~~~~~~~~~~~
5467
5468Running Wic in cooked mode leverages off artifacts in the Build
5469Directory. In other words, you do not have to specify kernel or root
5470filesystem locations as part of the command. All you need to provide is
5471a kickstart file and the name of the image from which to use artifacts
5472by using the "-e" option. Wic looks in the Build Directory (e.g.
5473``tmp/deploy/images/``\ machine) for artifacts.
5474
5475The general form of the ``wic`` command using Cooked Mode is as follows:
5476::
5477
5478 $ wic create wks_file -e IMAGE_NAME
5479
5480 Where:
5481
5482 wks_file:
5483 An OpenEmbedded kickstart file. You can provide
5484 your own custom file or use a file from a set of
5485 existing files provided with the Yocto Project
5486 release.
5487
5488 required argument:
5489 -e IMAGE_NAME, --image-name IMAGE_NAME
5490 name of the image to use the artifacts from e.g. core-
5491 image-sato
5492
5493Using an Existing Kickstart File
5494--------------------------------
5495
5496If you do not want to create your own kickstart file, you can use an
5497existing file provided by the Wic installation. As shipped, kickstart
5498files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
5499following two locations:
5500::
5501
5502 poky/meta-yocto-bsp/wic
5503 poky/scripts/lib/wic/canned-wks
5504
5505Use the following command to list the available kickstart files:
5506::
5507
5508 $ wic list images
5509 genericx86 Create an EFI disk image for genericx86*
5510 beaglebone-yocto Create SD card image for Beaglebone
5511 edgerouter Create SD card image for Edgerouter
5512 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5513 directdisk-gpt Create a 'pcbios' direct disk image
5514 mkefidisk Create an EFI disk image
5515 directdisk Create a 'pcbios' direct disk image
5516 systemd-bootdisk Create an EFI disk image with systemd-boot
5517 mkhybridiso Create a hybrid ISO image
5518 sdimage-bootpart Create SD card image with a boot partition
5519 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5520 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5521
5522When you use an existing file, you
5523do not have to use the ``.wks`` extension. Here is an example in Raw
5524Mode that uses the ``directdisk`` file:
5525::
5526
5527 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5528 -k kernel_dir -n native_sysroot
5529
5530Here are the actual partition language commands used in the
5531``genericx86.wks`` file to generate an image:
5532::
5533
5534 # short-description: Create an EFI disk image for genericx86*
5535 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5536 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5537 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5538 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5539
5540 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5541
5542Using the Wic Plugin Interface
5543------------------------------
5544
5545You can extend and specialize Wic functionality by using Wic plugins.
5546This section explains the Wic plugin interface.
5547
5548.. note::
5549
5550 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5551 are beyond the scope of this section.
5552
5553Source plugins provide a mechanism to customize partition content during
5554the Wic image generation process. You can use source plugins to map
5555values that you specify using ``--source`` commands in kickstart files
5556(i.e. ``*.wks``) to a plugin implementation used to populate a given
5557partition.
5558
5559.. note::
5560
5561 If you use plugins that have build-time dependencies (e.g. native
5562 tools, bootloaders, and so forth) when building a Wic image, you need
5563 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
5564 variable.
5565
5566Source plugins are subclasses defined in plugin files. As shipped, the
5567Yocto Project provides several plugin files. You can see the source
5568plugin files that ship with the Yocto Project
5569:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
5570Each of these plugin files contains source plugins that are designed to
5571populate a specific Wic image partition.
5572
5573Source plugins are subclasses of the ``SourcePlugin`` class, which is
5574defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5575the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5576file is a subclass of the ``SourcePlugin`` class, which is found in the
5577``pluginbase.py`` file.
5578
5579You can also implement source plugins in a layer outside of the Source
5580Repositories (external layer). To do so, be sure that your plugin files
5581are located in a directory whose path is
5582``scripts/lib/wic/plugins/source/`` within your external layer. When the
5583plugin files are located there, the source plugins they contain are made
5584available to Wic.
5585
5586When the Wic implementation needs to invoke a partition-specific
5587implementation, it looks for the plugin with the same name as the
5588``--source`` parameter used in the kickstart file given to that
5589partition. For example, if the partition is set up using the following
5590command in a kickstart file:
5591::
5592
5593 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5594
5595The methods defined as class
5596members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5597``bootimg-pcbios.py`` plugin file are used.
5598
5599To be more concrete, here is the corresponding plugin definition from
5600the ``bootimg-pcbios.py`` file for the previous command along with an
5601example method called by the Wic implementation when it needs to prepare
5602a partition using an implementation-specific function:
5603::
5604
5605 .
5606 .
5607 .
5608 class BootimgPcbiosPlugin(SourcePlugin):
5609 """
5610 Create MBR boot partition and install syslinux on it.
5611 """
5612
5613 name = 'bootimg-pcbios'
5614 .
5615 .
5616 .
5617 @classmethod
5618 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5619 oe_builddir, bootimg_dir, kernel_dir,
5620 rootfs_dir, native_sysroot):
5621 """
5622 Called to do the actual content population for a partition i.e. it
5623 'prepares' the partition to be incorporated into the image.
5624 In this case, prepare content for legacy bios boot partition.
5625 """
5626 .
5627 .
5628 .
5629
5630If a
5631subclass (plugin) itself does not implement a particular function, Wic
5632locates and uses the default version in the superclass. It is for this
5633reason that all source plugins are derived from the ``SourcePlugin``
5634class.
5635
5636The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5637a set of methods that source plugins can implement or override. Any
5638plugins (subclass of ``SourcePlugin``) that do not implement a
5639particular method inherit the implementation of the method from the
5640``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5641class in the ``pluginbase.py`` file for details:
5642
5643The following list describes the methods implemented in the
5644``SourcePlugin`` class:
5645
5646- ``do_prepare_partition()``: Called to populate a partition with
5647 actual content. In other words, the method prepares the final
5648 partition image that is incorporated into the disk image.
5649
5650- ``do_configure_partition()``: Called before
5651 ``do_prepare_partition()`` to create custom configuration files for a
5652 partition (e.g. syslinux or grub configuration files).
5653
5654- ``do_install_disk()``: Called after all partitions have been
5655 prepared and assembled into a disk image. This method provides a hook
5656 to allow finalization of a disk image (e.g. writing an MBR).
5657
5658- ``do_stage_partition()``: Special content-staging hook called
5659 before ``do_prepare_partition()``. This method is normally empty.
5660
5661 Typically, a partition just uses the passed-in parameters (e.g. the
5662 unmodified value of ``bootimg_dir``). However, in some cases, things
5663 might need to be more tailored. As an example, certain files might
5664 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5665 allows those files to be staged in a customized fashion.
5666
5667 .. note::
5668
5669 ``get_bitbake_var()`` allows you to access non-standard variables that
5670 you might want to use for this behavior.
5671
5672You can extend the source plugin mechanism. To add more hooks, create
5673more source plugin methods within ``SourcePlugin`` and the corresponding
5674derived subclasses. The code that calls the plugin methods uses the
5675``plugin.get_source_plugin_methods()`` function to find the method or
5676methods needed by the call. Retrieval of those methods is accomplished
5677by filling up a dict with keys that contain the method names of
5678interest. On success, these will be filled in with the actual methods.
5679See the Wic implementation for examples and details.
5680
5681Wic Examples
5682------------
5683
5684This section provides several examples that show how to use the Wic
5685utility. All the examples assume the list of requirements in the
5686"`Requirements <#wic-requirements>`__" section have been met. The
5687examples assume the previously generated image is
5688``core-image-minimal``.
5689
5690Generate an Image using an Existing Kickstart File
5691~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5692
5693This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
5694file:
5695::
5696
5697 $ wic create mkefidisk -e core-image-minimal
5698 INFO: Building wic-tools...
5699 .
5700 .
5701 .
5702 INFO: The new image(s) can be found here:
5703 ./mkefidisk-201804191017-sda.direct
5704
5705 The following build artifacts were used to create the image(s):
5706 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5707 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5708 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5709 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5710
5711 INFO: The image(s) were created using OE kickstart file:
5712 /home/stephano/build/master/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
5713
5714The previous example shows the easiest way to create an image by running
5715in cooked mode and supplying a kickstart file and the "-e" option to
5716point to the existing build artifacts. Your ``local.conf`` file needs to
5717have the :term:`MACHINE` variable set
5718to the machine you are using, which is "qemux86" in this example.
5719
5720Once the image builds, the output provides image location, artifact use,
5721and kickstart file information.
5722
5723.. note::
5724
5725 You should always verify the details provided in the output to make
5726 sure that the image was indeed created exactly as expected.
5727
5728Continuing with the example, you can now write the image from the Build
5729Directory onto a USB stick, or whatever media for which you built your
5730image, and boot from the media. You can write the image by using
5731``bmaptool`` or ``dd``:
5732::
5733
5734 $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
5735
5736or ::
5737
5738 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5739
5740.. note::
5741
5742 For more information on how to use the ``bmaptool``
5743 to flash a device with an image, see the
5744 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``"
5745 section.
5746
5747Using a Modified Kickstart File
5748~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5749
5750Because partitioned image creation is driven by the kickstart file, it
5751is easy to affect image creation by changing the parameters in the file.
5752This next example demonstrates that through modification of the
5753``directdisk-gpt`` kickstart file.
5754
5755As mentioned earlier, you can use the command ``wic list images`` to
5756show the list of existing kickstart files. The directory in which the
5757``directdisk-gpt.wks`` file resides is
5758``scripts/lib/image/canned-wks/``, which is located in the
5759:term:`Source Directory` (e.g. ``poky``).
5760Because available files reside in this directory, you can create and add
5761your own custom files to the directory. Subsequent use of the
5762``wic list images`` command would then include your kickstart files.
5763
5764In this example, the existing ``directdisk-gpt`` file already does most
5765of what is needed. However, for the hardware in this example, the image
5766will need to boot from ``sdb`` instead of ``sda``, which is what the
5767``directdisk-gpt`` kickstart file uses.
5768
5769The example begins by making a copy of the ``directdisk-gpt.wks`` file
5770in the ``scripts/lib/image/canned-wks`` directory and then by changing
5771the lines that specify the target disk from which to boot.
5772::
5773
5774 $ cp /home/stephano/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5775 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5776
5777Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5778changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5779example changes the following two lines and leaves the remaining lines
5780untouched:
5781::
5782
5783 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5784 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5785
5786Once the lines are changed, the
5787example generates the ``directdisksdb-gpt`` image. The command points
5788the process at the ``core-image-minimal`` artifacts for the Next Unit of
5789Computing (nuc) :term:`MACHINE` the
5790``local.conf``.
5791::
5792
5793 $ wic create directdisksdb-gpt -e core-image-minimal
5794 INFO: Building wic-tools...
5795 .
5796 .
5797 .
5798 Initialising tasks: 100% |#######################################| Time: 0:00:01
5799 NOTE: Executing SetScene Tasks
5800 NOTE: Executing RunQueue Tasks
5801 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5802 INFO: Creating image(s)...
5803
5804 INFO: The new image(s) can be found here:
5805 ./directdisksdb-gpt-201710090938-sdb.direct
5806
5807 The following build artifacts were used to create the image(s):
5808 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5809 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5810 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5811 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5812
5813 INFO: The image(s) were created using OE kickstart file:
5814 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5815
5816Continuing with the example, you can now directly ``dd`` the image to a
5817USB stick, or whatever media for which you built your image, and boot
5818the resulting media:
5819::
5820
5821 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5822 140966+0 records in
5823 140966+0 records out
5824 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5825 $ sudo eject /dev/sdb
5826
5827Using a Modified Kickstart File and Running in Raw Mode
5828~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5829
5830This next example manually specifies each build artifact (runs in Raw
5831Mode) and uses a modified kickstart file. The example also uses the
5832``-o`` option to cause Wic to create the output somewhere other than the
5833default output directory, which is the current directory:
5834::
5835
5836 $ wic create /home/stephano/my_yocto/test.wks -o /home/stephano/testwic \
5837 --rootfs-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5838 --bootimg-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5839 --kernel-dir /home/stephano/build/master/build/tmp/deploy/images/qemux86 \
5840 --native-sysroot /home/stephano/build/master/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
5841
5842 INFO: Creating image(s)...
5843
5844 INFO: The new image(s) can be found here:
5845 /home/stephano/testwic/test-201710091445-sdb.direct
5846
5847 The following build artifacts were used to create the image(s):
5848 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5849 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5850 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5851 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5852
5853 INFO: The image(s) were created using OE kickstart file:
5854 /home/stephano/my_yocto/test.wks
5855
5856For this example,
5857:term:`MACHINE` did not have to be
5858specified in the ``local.conf`` file since the artifact is manually
5859specified.
5860
5861Using Wic to Manipulate an Image
5862~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5863
5864Wic image manipulation allows you to shorten turnaround time during
5865image development. For example, you can use Wic to delete the kernel
5866partition of a Wic image and then insert a newly built kernel. This
5867saves you time from having to rebuild the entire image each time you
5868modify the kernel.
5869
5870.. note::
5871
5872 In order to use Wic to manipulate a Wic image as in this example,
5873 your development machine must have the ``mtools`` package installed.
5874
5875The following example examines the contents of the Wic image, deletes
5876the existing kernel, and then inserts a new kernel:
5877
58781. *List the Partitions:* Use the ``wic ls`` command to list all the
5879 partitions in the Wic image:
5880 ::
5881
5882 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
5883 Num Start End Size Fstype
5884 1 1048576 25041919 23993344 fat16
5885 2 25165824 72157183 46991360 ext4
5886
5887 The previous output shows two partitions in the
5888 ``core-image-minimal-qemux86.wic`` image.
5889
58902. *Examine a Particular Partition:* Use the ``wic ls`` command again
5891 but in a different form to examine a particular partition.
5892
5893 .. note::
5894
5895 You can get command usage on any Wic command using the following
5896 form:
5897 ::
5898
5899 $ wic help command
5900
5901
5902 For example, the following command shows you the various ways to
5903 use the
5904 wic ls
5905 command:
5906 ::
5907
5908 $ wic help ls
5909
5910
5911 The following command shows what is in Partition one:
5912 ::
5913
5914 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
5915 Volume in drive : is boot
5916 Volume Serial Number is E894-1809
5917 Directory for ::/
5918
5919 libcom32 c32 186500 2017-10-09 16:06
5920 libutil c32 24148 2017-10-09 16:06
5921 syslinux cfg 220 2017-10-09 16:06
5922 vesamenu c32 27104 2017-10-09 16:06
5923 vmlinuz 6904608 2017-10-09 16:06
5924 5 files 7 142 580 bytes
5925 16 582 656 bytes free
5926
5927 The previous output shows five files, with the
5928 ``vmlinuz`` being the kernel.
5929
5930 .. note::
5931
5932 If you see the following error, you need to update or create a
5933 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
5934 in the file. Then, run the Wic command again:
5935 ::
5936
5937 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
5938 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
5939 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
5940
5941
59423. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
5943 ``vmlinuz`` file (kernel):
5944 ::
5945
5946 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
5947
59484. *Add In the New Kernel:* Use the ``wic cp`` command to add the
5949 updated kernel to the Wic image. Depending on how you built your
5950 kernel, it could be in different places. If you used ``devtool`` and
5951 an SDK to build your kernel, it resides in the ``tmp/work`` directory
5952 of the extensible SDK. If you used ``make`` to build the kernel, the
5953 kernel will be in the ``workspace/sources`` area.
5954
5955 The following example assumes ``devtool`` was used to build the
5956 kernel:
5957 ::
5958
5959 cp ~/poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
5960 ~/poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
5961
5962 Once the new kernel is added back into the image, you can use the
5963 ``dd`` command or :ref:`bmaptool
5964 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>`
5965 to flash your wic image onto an SD card or USB stick and test your
5966 target.
5967
5968 .. note::
5969
5970 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
5971
5972Flashing Images Using ``bmaptool``
5973==================================
5974
5975A fast and easy way to flash an image to a bootable device is to use
5976Bmaptool, which is integrated into the OpenEmbedded build system.
5977Bmaptool is a generic tool that creates a file's block map (bmap) and
5978then uses that map to copy the file. As compared to traditional tools
5979such as dd or cp, Bmaptool can copy (or flash) large files like raw
5980system image files much faster.
5981
5982.. note::
5983
5984 - If you are using Ubuntu or Debian distributions, you can install
5985 the ``bmap-tools`` package using the following command and then
5986 use the tool without specifying ``PATH`` even from the root
5987 account:
5988 ::
5989
5990 $ sudo apt-get install bmap-tools
5991
5992 - If you are unable to install the ``bmap-tools`` package, you will
5993 need to build Bmaptool before using it. Use the following command:
5994 ::
5995
5996 $ bitbake bmap-tools-native
5997
5998Following, is an example that shows how to flash a Wic image. Realize
5999that while this example uses a Wic image, you can use Bmaptool to flash
6000any type of image. Use these steps to flash an image using Bmaptool:
6001
60021. *Update your local.conf File:* You need to have the following set
6003 in your ``local.conf`` file before building your image:
6004 ::
6005
6006 IMAGE_FSTYPES += "wic wic.bmap"
6007
60082. *Get Your Image:* Either have your image ready (pre-built with the
6009 :term:`IMAGE_FSTYPES`
6010 setting previously mentioned) or take the step to build the image:
6011 ::
6012
6013 $ bitbake image
6014
60153. *Flash the Device:* Flash the device with the image by using Bmaptool
6016 depending on your particular setup. The following commands assume the
6017 image resides in the Build Directory's ``deploy/images/`` area:
6018
6019 - If you have write access to the media, use this command form:
6020 ::
6021
6022 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6023
6024 - If you do not have write access to the media, set your permissions
6025 first and then use the same command form:
6026 ::
6027
6028 $ sudo chmod 666 /dev/sdX
6029 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6030
6031For help on the ``bmaptool`` command, use the following command:
6032::
6033
6034 $ bmaptool --help
6035
6036Making Images More Secure
6037=========================
6038
6039Security is of increasing concern for embedded devices. Consider the
6040issues and problems discussed in just this sampling of work found across
6041the Internet:
6042
6043- *"*\ `Security Risks of Embedded
6044 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
6045 by Bruce Schneier
6046
6047- *"*\ `Internet Census
6048 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
6049 Botnet
6050
6051- *"*\ `Security Issues for Embedded
6052 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
6053 by Jake Edge
6054
6055When securing your image is of concern, there are steps, tools, and
6056variables that you can consider to help you reach the security goals you
6057need for your particular device. Not all situations are identical when
6058it comes to making an image secure. Consequently, this section provides
6059some guidance and suggestions for consideration when you want to make
6060your image more secure.
6061
6062.. note::
6063
6064 Because the security requirements and risks are different for every
6065 type of device, this section cannot provide a complete reference on
6066 securing your custom OS. It is strongly recommended that you also
6067 consult other sources of information on embedded Linux system
6068 hardening and on security.
6069
6070General Considerations
6071----------------------
6072
6073General considerations exist that help you create more secure images.
6074You should consider the following suggestions to help make your device
6075more secure:
6076
6077- Scan additional code you are adding to the system (e.g. application
6078 code) by using static analysis tools. Look for buffer overflows and
6079 other potential security problems.
6080
6081- Pay particular attention to the security for any web-based
6082 administration interface.
6083
6084 Web interfaces typically need to perform administrative functions and
6085 tend to need to run with elevated privileges. Thus, the consequences
6086 resulting from the interface's security becoming compromised can be
6087 serious. Look for common web vulnerabilities such as
6088 cross-site-scripting (XSS), unvalidated inputs, and so forth.
6089
6090 As with system passwords, the default credentials for accessing a
6091 web-based interface should not be the same across all devices. This
6092 is particularly true if the interface is enabled by default as it can
6093 be assumed that many end-users will not change the credentials.
6094
6095- Ensure you can update the software on the device to mitigate
6096 vulnerabilities discovered in the future. This consideration
6097 especially applies when your device is network-enabled.
6098
6099- Ensure you remove or disable debugging functionality before producing
6100 the final image. For information on how to do this, see the
6101 "`Considerations Specific to the OpenEmbedded Build
6102 System <#considerations-specific-to-the-openembedded-build-system>`__"
6103 section.
6104
6105- Ensure you have no network services listening that are not needed.
6106
6107- Remove any software from the image that is not needed.
6108
6109- Enable hardware support for secure boot functionality when your
6110 device supports this functionality.
6111
6112Security Flags
6113--------------
6114
6115The Yocto Project has security flags that you can enable that help make
6116your build output more secure. The security flags are in the
6117``meta/conf/distro/include/security_flags.inc`` file in your
6118:term:`Source Directory` (e.g. ``poky``).
6119
6120.. note::
6121
6122 Depending on the recipe, certain security flags are enabled and
6123 disabled by default.
6124
6125Use the following line in your ``local.conf`` file or in your custom
6126distribution configuration file to enable the security compiler and
6127linker flags for your build:
6128::
6129
6130 require conf/distro/include/security_flags.inc
6131
6132Considerations Specific to the OpenEmbedded Build System
6133--------------------------------------------------------
6134
6135You can take some steps that are specific to the OpenEmbedded build
6136system to make your images more secure:
6137
6138- Ensure "debug-tweaks" is not one of your selected
6139 :term:`IMAGE_FEATURES`.
6140 When creating a new project, the default is to provide you with an
6141 initial ``local.conf`` file that enables this feature using the
6142 :term:`EXTRA_IMAGE_FEATURES`
6143 variable with the line:
6144 ::
6145
6146 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6147
6148 To disable that feature, simply comment out that line in your
6149 ``local.conf`` file, or make sure ``IMAGE_FEATURES`` does not contain
6150 "debug-tweaks" before producing your final image. Among other things,
6151 leaving this in place sets the root password as blank, which makes
6152 logging in for debugging or inspection easy during development but
6153 also means anyone can easily log in during production.
6154
6155- It is possible to set a root password for the image and also to set
6156 passwords for any extra users you might add (e.g. administrative or
6157 service type users). When you set up passwords for multiple images or
6158 users, you should not duplicate passwords.
6159
6160 To set up passwords, use the
6161 :ref:`extrausers <ref-classes-extrausers>`
6162 class, which is the preferred method. For an example on how to set up
6163 both root and user passwords, see the
6164 ":ref:`extrausers.bbclass <ref-classes-extrausers>`"
6165 section.
6166
6167 .. note::
6168
6169 When adding extra user accounts or setting a root password, be
6170 cautious about setting the same password on every device. If you
6171 do this, and the password you have set is exposed, then every
6172 device is now potentially compromised. If you need this access but
6173 want to ensure security, consider setting a different, random
6174 password for each device. Typically, you do this as a separate
6175 step after you deploy the image onto the device.
6176
6177- Consider enabling a Mandatory Access Control (MAC) framework such as
6178 SMACK or SELinux and tuning it appropriately for your device's usage.
6179 You can find more information in the
6180 :yocto_git:`meta-selinux </meta-selinux/>` layer.
6181
6182Tools for Hardening Your Image
6183------------------------------
6184
6185The Yocto Project provides tools for making your image more secure. You
6186can find these tools in the ``meta-security`` layer of the
6187:yocto_git:`Yocto Project Source Repositories <>`.
6188
6189Creating Your Own Distribution
6190==============================
6191
6192When you build an image using the Yocto Project and do not alter any
6193distribution :term:`Metadata`, you are
6194creating a Poky distribution. If you wish to gain more control over
6195package alternative selections, compile-time options, and other
6196low-level configurations, you can create your own distribution.
6197
6198To create your own distribution, the basic steps consist of creating
6199your own distribution layer, creating your own distribution
6200configuration file, and then adding any needed code and Metadata to the
6201layer. The following steps provide some more detail:
6202
6203- *Create a layer for your new distro:* Create your distribution layer
6204 so that you can keep your Metadata and code for the distribution
6205 separate. It is strongly recommended that you create and use your own
6206 layer for configuration and code. Using your own layer as compared to
6207 just placing configurations in a ``local.conf`` configuration file
6208 makes it easier to reproduce the same build configuration when using
6209 multiple build machines. See the
6210 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
6211 section for information on how to quickly set up a layer.
6212
6213- *Create the distribution configuration file:* The distribution
6214 configuration file needs to be created in the ``conf/distro``
6215 directory of your layer. You need to name it using your distribution
6216 name (e.g. ``mydistro.conf``).
6217
6218 .. note::
6219
6220 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6221 name of your distribution.
6222
6223 You can split out parts of your configuration file into include files
6224 and then "require" them from within your distribution configuration
6225 file. Be sure to place the include files in the
6226 ``conf/distro/include`` directory of your layer. A common example
6227 usage of include files would be to separate out the selection of
6228 desired version and revisions for individual recipes.
6229
6230 Your configuration file needs to set the following required
6231 variables:
6232
6233 - :term:`DISTRO_NAME`
6234
6235 - :term:`DISTRO_VERSION`
6236
6237 These following variables are optional and you typically set them
6238 from the distribution configuration file:
6239
6240 - :term:`DISTRO_FEATURES`
6241
6242 - :term:`DISTRO_EXTRA_RDEPENDS`
6243
6244 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6245
6246 - :term:`TCLIBC`
6247
6248 .. tip::
6249
6250 If you want to base your distribution configuration file on the
6251 very basic configuration from OE-Core, you can use
6252 ``conf/distro/defaultsetup.conf`` as a reference and just include
6253 variables that differ as compared to ``defaultsetup.conf``.
6254 Alternatively, you can create a distribution configuration file
6255 from scratch using the ``defaultsetup.conf`` file or configuration files
6256 from other distributions such as Poky or Angstrom as references.
6257
6258- *Provide miscellaneous variables:* Be sure to define any other
6259 variables for which you want to create a default or enforce as part
6260 of the distribution configuration. You can include nearly any
6261 variable from the ``local.conf`` file. The variables you use are not
6262 limited to the list in the previous bulleted item.
6263
6264- *Point to Your distribution configuration file:* In your
6265 ``local.conf`` file in the :term:`Build Directory`,
6266 set your
6267 :term:`DISTRO` variable to point to
6268 your distribution's configuration file. For example, if your
6269 distribution's configuration file is named ``mydistro.conf``, then
6270 you point to it as follows:
6271 ::
6272
6273 DISTRO = "mydistro"
6274
6275- *Add more to the layer if necessary:* Use your layer to hold other
6276 information needed for the distribution:
6277
6278 - Add recipes for installing distro-specific configuration files
6279 that are not already installed by another recipe. If you have
6280 distro-specific configuration files that are included by an
6281 existing recipe, you should add an append file (``.bbappend``) for
6282 those. For general information and recommendations on how to add
6283 recipes to your layer, see the "`Creating Your Own
6284 Layer <#creating-your-own-layer>`__" and "`Following Best
6285 Practices When Creating
6286 Layers <#best-practices-to-follow-when-creating-layers>`__"
6287 sections.
6288
6289 - Add any image recipes that are specific to your distribution.
6290
6291 - Add a ``psplash`` append file for a branded splash screen. For
6292 information on append files, see the "`Using .bbappend Files in
6293 Your Layer <#using-bbappend-files>`__" section.
6294
6295 - Add any other append files to make custom changes that are
6296 specific to individual recipes.
6297
6298Creating a Custom Template Configuration Directory
6299==================================================
6300
6301If you are producing your own customized version of the build system for
6302use by other users, you might want to customize the message shown by the
6303setup script or you might want to change the template configuration
6304files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
6305new build directory.
6306
6307The OpenEmbedded build system uses the environment variable
6308``TEMPLATECONF`` to locate the directory from which it gathers
6309configuration information that ultimately ends up in the
6310:term:`Build Directory` ``conf`` directory.
6311By default, ``TEMPLATECONF`` is set as follows in the ``poky``
6312repository:
6313::
6314
6315 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
6316
6317This is the
6318directory used by the build system to find templates from which to build
6319some key configuration files. If you look at this directory, you will
6320see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6321``conf-notes.txt`` files. The build system uses these files to form the
6322respective ``bblayers.conf`` file, ``local.conf`` file, and display the
6323list of BitBake targets when running the setup script.
6324
6325To override these default configuration files with configurations you
6326want used within every new Build Directory, simply set the
6327``TEMPLATECONF`` variable to your directory. The ``TEMPLATECONF``
6328variable is set in the ``.templateconf`` file, which is in the top-level
6329:term:`Source Directory` folder
6330(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
6331directory.
6332
6333Best practices dictate that you should keep your template configuration
6334directory in your custom distribution layer. For example, suppose you
6335have a layer named ``meta-mylayer`` located in your home directory and
6336you want your template configuration directory named ``myconf``.
6337Changing the ``.templateconf`` as follows causes the OpenEmbedded build
6338system to look in your directory and base its configuration files on the
6339``*.sample`` configuration files it finds. The final configuration files
6340(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
6341your Build Directory, but they are based on your ``*.sample`` files.
6342::
6343
6344 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
6345
6346Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
6347also resides in the default ``meta-poky/conf`` directory. The script
6348that sets up the build environment (i.e.
6349:ref:`structure-core-script`) uses this file to
6350display BitBake targets as part of the script output. Customizing this
6351``conf-notes.txt`` file is a good way to make sure your list of custom
6352targets appears as part of the script's output.
6353
6354Here is the default list of targets displayed as a result of running
6355either of the setup scripts:
6356::
6357
6358 You can now run 'bitbake <target>'
6359
6360 Common targets are:
6361 core-image-minimal
6362 core-image-sato
6363 meta-toolchain
6364 meta-ide-support
6365
6366Changing the listed common targets is as easy as editing your version of
6367``conf-notes.txt`` in your custom template configuration directory and
6368making sure you have ``TEMPLATECONF`` set to your directory.
6369
6370Conserving Disk Space During Builds
6371===================================
6372
6373To help conserve disk space during builds, you can add the following
6374statement to your project's ``local.conf`` configuration file found in
6375the :term:`Build Directory`:
6376::
6377
6378 INHERIT += "rm_work"
6379
6380Adding this statement deletes the work directory used for
6381building a recipe once the recipe is built. For more information on
6382"rm_work", see the
6383:ref:`rm_work <ref-classes-rm-work>` class in the
6384Yocto Project Reference Manual.
6385
6386Working with Packages
6387=====================
6388
6389This section describes a few tasks that involve packages:
6390
6391- `Excluding packages from an
6392 image <#excluding-packages-from-an-image>`__
6393
6394- `Incrementing a binary package
6395 version <#incrementing-a-binary-package-version>`__
6396
6397- `Handling optional module
6398 packaging <#handling-optional-module-packaging>`__
6399
6400- `Using runtime package
6401 management <#using-runtime-package-management>`__
6402
6403- `Generating and using signed
6404 packages <#generating-and-using-signed-packages>`__
6405
6406- `Setting up and running package test
6407 (ptest) <#testing-packages-with-ptest>`__
6408
6409- `Creating node package manager (NPM)
6410 packages <#creating-node-package-manager-npm-packages>`__
6411
6412- `Adding custom metadata to
6413 packages <#adding-custom-metadata-to-packages>`__
6414
6415Excluding Packages from an Image
6416--------------------------------
6417
6418You might find it necessary to prevent specific packages from being
6419installed into an image. If so, you can use several variables to direct
6420the build system to essentially ignore installing recommended packages
6421or to not install a package at all.
6422
6423The following list introduces variables you can use to prevent packages
6424from being installed into your image. Each of these variables only works
6425with IPK and RPM package types. Support for Debian packages does not
6426exist. Also, you can use these variables from your ``local.conf`` file
6427or attach them to a specific image recipe by using a recipe name
6428override. For more detail on the variables, see the descriptions in the
6429Yocto Project Reference Manual's glossary chapter.
6430
6431- :term:`BAD_RECOMMENDATIONS`:
6432 Use this variable to specify "recommended-only" packages that you do
6433 not want installed.
6434
6435- :term:`NO_RECOMMENDATIONS`:
6436 Use this variable to prevent all "recommended-only" packages from
6437 being installed.
6438
6439- :term:`PACKAGE_EXCLUDE`:
6440 Use this variable to prevent specific packages from being installed
6441 regardless of whether they are "recommended-only" or not. You need to
6442 realize that the build process could fail with an error when you
6443 prevent the installation of a package whose presence is required by
6444 an installed package.
6445
6446Incrementing a Package Version
6447------------------------------
6448
6449This section provides some background on how binary package versioning
6450is accomplished and presents some of the services, variables, and
6451terminology involved.
6452
6453In order to understand binary package versioning, you need to consider
6454the following:
6455
6456- Binary Package: The binary package that is eventually built and
6457 installed into an image.
6458
6459- Binary Package Version: The binary package version is composed of two
6460 components - a version and a revision.
6461
6462 .. note::
6463
6464 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
6465 but this discussion for the most part ignores ``PE``.
6466
6467 The version and revision are taken from the
6468 :term:`PV` and
6469 :term:`PR` variables, respectively.
6470
6471- ``PV``: The recipe version. ``PV`` represents the version of the
6472 software being packaged. Do not confuse ``PV`` with the binary
6473 package version.
6474
6475- ``PR``: The recipe revision.
6476
6477- :term:`SRCPV`: The OpenEmbedded
6478 build system uses this string to help define the value of ``PV`` when
6479 the source code revision needs to be included in it.
6480
6481- :yocto_wiki:`PR Service </PR_Service>`: A
6482 network-based service that helps automate keeping package feeds
6483 compatible with existing package manager applications such as RPM,
6484 APT, and OPKG.
6485
6486Whenever the binary package content changes, the binary package version
6487must change. Changing the binary package version is accomplished by
6488changing or "bumping" the ``PR`` and/or ``PV`` values. Increasing these
6489values occurs one of two ways:
6490
6491- Automatically using a Package Revision Service (PR Service).
6492
6493- Manually incrementing the ``PR`` and/or ``PV`` variables.
6494
6495Given a primary challenge of any build system and its users is how to
6496maintain a package feed that is compatible with existing package manager
6497applications such as RPM, APT, and OPKG, using an automated system is
6498much preferred over a manual system. In either system, the main
6499requirement is that binary package version numbering increases in a
6500linear fashion and that a number of version components exist that
6501support that linear progression. For information on how to ensure
6502package revisioning remains linear, see the "`Automatically Incrementing
6503a Binary Package Revision
6504Number <#automatically-incrementing-a-binary-package-revision-number>`__"
6505section.
6506
6507The following three sections provide related information on the PR
6508Service, the manual method for "bumping" ``PR`` and/or ``PV``, and on
6509how to ensure binary package revisioning remains linear.
6510
6511Working With a PR Service
6512~~~~~~~~~~~~~~~~~~~~~~~~~
6513
6514As mentioned, attempting to maintain revision numbers in the
6515:term:`Metadata` is error prone, inaccurate,
6516and causes problems for people submitting recipes. Conversely, the PR
6517Service automatically generates increasing numbers, particularly the
6518revision field, which removes the human element.
6519
6520.. note::
6521
6522 For additional information on using a PR Service, you can see the
6523 :yocto_wiki:`PR Service </PR_Service>` wiki page.
6524
6525The Yocto Project uses variables in order of decreasing priority to
6526facilitate revision numbering (i.e.
6527:term:`PE`,
6528:term:`PV`, and
6529:term:`PR` for epoch, version, and
6530revision, respectively). The values are highly dependent on the policies
6531and procedures of a given distribution and package feed.
6532
6533Because the OpenEmbedded build system uses
6534":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
6535unique to a given build, the build system knows when to rebuild
6536packages. All the inputs into a given task are represented by a
6537signature, which can trigger a rebuild when different. Thus, the build
6538system itself does not rely on the ``PR``, ``PV``, and ``PE`` numbers to
6539trigger a rebuild. The signatures, however, can be used to generate
6540these values.
6541
6542The PR Service works with both ``OEBasic`` and ``OEBasicHash``
6543generators. The value of ``PR`` bumps when the checksum changes and the
6544different generator mechanisms change signatures under different
6545circumstances.
6546
6547As implemented, the build system includes values from the PR Service
6548into the ``PR`` field as an addition using the form "``.x``" so ``r0``
6549becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
6550``PR`` values to be used for whatever reasons, which include manual
6551``PR`` bumps, should it be necessary.
6552
6553By default, the PR Service is not enabled or running. Thus, the packages
6554generated are just "self consistent". The build system adds and removes
6555packages and there are no guarantees about upgrade paths but images will
6556be consistent and correct with the latest changes.
6557
6558The simplest form for a PR Service is for it to exist for a single host
6559development system that builds the package feed (building system). For
6560this scenario, you can enable a local PR Service by setting
6561:term:`PRSERV_HOST` in your
6562``local.conf`` file in the :term:`Build Directory`:
6563::
6564
6565 PRSERV_HOST = "localhost:0"
6566
6567Once the service is started, packages will automatically
6568get increasing ``PR`` values and BitBake takes care of starting and
6569stopping the server.
6570
6571If you have a more complex setup where multiple host development systems
6572work against a common, shared package feed, you have a single PR Service
6573running and it is connected to each building system. For this scenario,
6574you need to start the PR Service using the ``bitbake-prserv`` command:
6575::
6576
6577 bitbake-prserv --host ip --port port --start
6578
6579In addition to
6580hand-starting the service, you need to update the ``local.conf`` file of
6581each building system as described earlier so each system points to the
6582server and port.
6583
6584It is also recommended you use build history, which adds some sanity
6585checks to binary package versions, in conjunction with the server that
6586is running the PR Service. To enable build history, add the following to
6587each building system's ``local.conf`` file:
6588::
6589
6590 # It is recommended to activate "buildhistory" for testing the PR service
6591 INHERIT += "buildhistory"
6592 BUILDHISTORY_COMMIT = "1"
6593
6594For information on build
6595history, see the "`Maintaining Build Output
6596Quality <#maintaining-build-output-quality>`__" section.
6597
6598.. note::
6599
6600 The OpenEmbedded build system does not maintain ``PR`` information as
6601 part of the shared state (sstate) packages. If you maintain an sstate
6602 feed, its expected that either all your building systems that
6603 contribute to the sstate feed use a shared PR Service, or you do not
6604 run a PR Service on any of your building systems. Having some systems
6605 use a PR Service while others do not leads to obvious problems.
6606
6607 For more information on shared state, see the
6608 ":ref:`overview-manual/concepts:shared state cache`"
6609 section in the Yocto Project Overview and Concepts Manual.
6610
6611Manually Bumping PR
6612~~~~~~~~~~~~~~~~~~~
6613
6614The alternative to setting up a PR Service is to manually "bump" the
6615:term:`PR` variable.
6616
6617If a committed change results in changing the package output, then the
6618value of the PR variable needs to be increased (or "bumped") as part of
6619that commit. For new recipes you should add the ``PR`` variable and set
6620its initial value equal to "r0", which is the default. Even though the
6621default value is "r0", the practice of adding it to a new recipe makes
6622it harder to forget to bump the variable when you make changes to the
6623recipe in future.
6624
6625If you are sharing a common ``.inc`` file with multiple recipes, you can
6626also use the ``INC_PR`` variable to ensure that the recipes sharing the
6627``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
6628``.inc`` file must set ``INC_PR`` (initially to "r0"), and all recipes
6629referring to it should set ``PR`` to "${INC_PR}.0" initially,
6630incrementing the last number when the recipe is changed. If the ``.inc``
6631file is changed then its ``INC_PR`` should be incremented.
6632
6633When upgrading the version of a binary package, assuming the ``PV``
6634changes, the ``PR`` variable should be reset to "r0" (or "${INC_PR}.0"
6635if you are using ``INC_PR``).
6636
6637Usually, version increases occur only to binary packages. However, if
6638for some reason ``PV`` changes but does not increase, you can increase
6639the ``PE`` variable (Package Epoch). The ``PE`` variable defaults to
6640"0".
6641
6642Binary package version numbering strives to follow the `Debian Version
6643Field Policy
6644Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
6645These guidelines define how versions are compared and what "increasing"
6646a version means.
6647
6648Automatically Incrementing a Package Version Number
6649~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6650
6651When fetching a repository, BitBake uses the
6652:term:`SRCREV` variable to determine
6653the specific source code revision from which to build. You set the
6654``SRCREV`` variable to
6655:term:`AUTOREV` to cause the
6656OpenEmbedded build system to automatically use the latest revision of
6657the software:
6658::
6659
6660 SRCREV = "${AUTOREV}"
6661
6662Furthermore, you need to reference ``SRCPV`` in ``PV`` in order to
6663automatically update the version whenever the revision of the source
6664code changes. Here is an example:
6665::
6666
6667 PV = "1.0+git${SRCPV}"
6668
6669The OpenEmbedded build system substitutes ``SRCPV`` with the following:
6670
6671.. code-block:: none
6672
6673 AUTOINC+source_code_revision
6674
6675The build system replaces the ``AUTOINC``
6676with a number. The number used depends on the state of the PR Service:
6677
6678- If PR Service is enabled, the build system increments the number,
6679 which is similar to the behavior of
6680 :term:`PR`. This behavior results in
6681 linearly increasing package versions, which is desirable. Here is an
6682 example:
6683
6684 .. code-block:: none
6685
6686 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6687 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6688
6689- If PR Service is not enabled, the build system replaces the
6690 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6691 changing the package version since the source revision is included.
6692 However, package versions are not increased linearly. Here is an
6693 example:
6694
6695 .. code-block:: none
6696
6697 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6698 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6699
6700In summary, the OpenEmbedded build system does not track the history of
6701binary package versions for this purpose. ``AUTOINC``, in this case, is
6702comparable to ``PR``. If PR server is not enabled, ``AUTOINC`` in the
6703package version is simply replaced by "0". If PR server is enabled, the
6704build system keeps track of the package versions and bumps the number
6705when the package revision changes.
6706
6707Handling Optional Module Packaging
6708----------------------------------
6709
6710Many pieces of software split functionality into optional modules (or
6711plugins) and the plugins that are built might depend on configuration
6712options. To avoid having to duplicate the logic that determines what
6713modules are available in your recipe or to avoid having to package each
6714module by hand, the OpenEmbedded build system provides functionality to
6715handle module packaging dynamically.
6716
6717To handle optional module packaging, you need to do two things:
6718
6719- Ensure the module packaging is actually done.
6720
6721- Ensure that any dependencies on optional modules from other recipes
6722 are satisfied by your recipe.
6723
6724Making Sure the Packaging is Done
6725~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6726
6727To ensure the module packaging actually gets done, you use the
6728``do_split_packages`` function within the ``populate_packages`` Python
6729function in your recipe. The ``do_split_packages`` function searches for
6730a pattern of files or directories under a specified path and creates a
6731package for each one it finds by appending to the
6732:term:`PACKAGES` variable and
6733setting the appropriate values for ``FILES_packagename``,
6734``RDEPENDS_packagename``, ``DESCRIPTION_packagename``, and so forth.
6735Here is an example from the ``lighttpd`` recipe:
6736::
6737
6738 python populate_packages_prepend () {
6739 lighttpd_libdir = d.expand('${libdir}')
6740 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6741 'lighttpd-module-%s', 'Lighttpd module for %s',
6742 extra_depends='')
6743 }
6744
6745The previous example specifies a number of things in the call to
6746``do_split_packages``.
6747
6748- A directory within the files installed by your recipe through
6749 ``do_install`` in which to search.
6750
6751- A regular expression used to match module files in that directory. In
6752 the example, note the parentheses () that mark the part of the
6753 expression from which the module name should be derived.
6754
6755- A pattern to use for the package names.
6756
6757- A description for each package.
6758
6759- An empty string for ``extra_depends``, which disables the default
6760 dependency on the main ``lighttpd`` package. Thus, if a file in
6761 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6762 ``lighttpd-module-alias`` is created for it and the
6763 :term:`DESCRIPTION` is set to
6764 "Lighttpd module for alias".
6765
6766Often, packaging modules is as simple as the previous example. However,
6767more advanced options exist that you can use within
6768``do_split_packages`` to modify its behavior. And, if you need to, you
6769can add more logic by specifying a hook function that is called for each
6770package. It is also perfectly acceptable to call ``do_split_packages``
6771multiple times if you have more than one set of modules to package.
6772
6773For more examples that show how to use ``do_split_packages``, see the
6774``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
6775directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
6776also find examples in ``meta/classes/kernel.bbclass``.
6777
6778Following is a reference that shows ``do_split_packages`` mandatory and
6779optional arguments:
6780::
6781
6782 Mandatory arguments
6783
6784 root
6785 The path in which to search
6786 file_regex
6787 Regular expression to match searched files.
6788 Use parentheses () to mark the part of this
6789 expression that should be used to derive the
6790 module name (to be substituted where %s is
6791 used in other function arguments as noted below)
6792 output_pattern
6793 Pattern to use for the package names. Must
6794 include %s.
6795 description
6796 Description to set for each package. Must
6797 include %s.
6798
6799 Optional arguments
6800
6801 postinst
6802 Postinstall script to use for all packages
6803 (as a string)
6804 recursive
6805 True to perform a recursive search - default
6806 False
6807 hook
6808 A hook function to be called for every match.
6809 The function will be called with the following
6810 arguments (in the order listed):
6811
6812 f
6813 Full path to the file/directory match
6814 pkg
6815 The package name
6816 file_regex
6817 As above
6818 output_pattern
6819 As above
6820 modulename
6821 The module name derived using file_regex
6822 extra_depends
6823 Extra runtime dependencies (RDEPENDS) to be
6824 set for all packages. The default value of None
6825 causes a dependency on the main package
6826 (${PN}) - if you do not want this, pass empty
6827 string '' for this parameter.
6828 aux_files_pattern
6829 Extra item(s) to be added to FILES for each
6830 package. Can be a single string item or a list
6831 of strings for multiple items. Must include %s.
6832 postrm
6833 postrm script to use for all packages (as a
6834 string)
6835 allow_dirs
6836 True to allow directories to be matched -
6837 default False
6838 prepend
6839 If True, prepend created packages to PACKAGES
6840 instead of the default False which appends them
6841 match_path
6842 match file_regex on the whole relative path to
6843 the root rather than just the file name
6844 aux_files_pattern_verbatim
6845 Extra item(s) to be added to FILES for each
6846 package, using the actual derived module name
6847 rather than converting it to something legal
6848 for a package name. Can be a single string item
6849 or a list of strings for multiple items. Must
6850 include %s.
6851 allow_links
6852 True to allow symlinks to be matched - default
6853 False
6854 summary
6855 Summary to set for each package. Must include %s;
6856 defaults to description if not set.
6857
6858
6859
6860Satisfying Dependencies
6861~~~~~~~~~~~~~~~~~~~~~~~
6862
6863The second part for handling optional module packaging is to ensure that
6864any dependencies on optional modules from other recipes are satisfied by
6865your recipe. You can be sure these dependencies are satisfied by using
6866the :term:`PACKAGES_DYNAMIC`
6867variable. Here is an example that continues with the ``lighttpd`` recipe
6868shown earlier:
6869::
6870
6871 PACKAGES_DYNAMIC = "lighttpd-module-.*"
6872
6873The name
6874specified in the regular expression can of course be anything. In this
6875example, it is ``lighttpd-module-`` and is specified as the prefix to
6876ensure that any :term:`RDEPENDS` and
6877:term:`RRECOMMENDS` on a package
6878name starting with the prefix are satisfied during build time. If you
6879are using ``do_split_packages`` as described in the previous section,
6880the value you put in ``PACKAGES_DYNAMIC`` should correspond to the name
6881pattern specified in the call to ``do_split_packages``.
6882
6883Using Runtime Package Management
6884--------------------------------
6885
6886During a build, BitBake always transforms a recipe into one or more
6887packages. For example, BitBake takes the ``bash`` recipe and produces a
6888number of packages (e.g. ``bash``, ``bash-bashbug``,
6889``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
6890``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
6891generated packages are included in an image.
6892
6893In several situations, you might need to update, add, remove, or query
6894the packages on a target device at runtime (i.e. without having to
6895generate a new image). Examples of such situations include:
6896
6897- You want to provide in-the-field updates to deployed devices (e.g.
6898 security updates).
6899
6900- You want to have a fast turn-around development cycle for one or more
6901 applications that run on your device.
6902
6903- You want to temporarily install the "debug" packages of various
6904 applications on your device so that debugging can be greatly improved
6905 by allowing access to symbols and source debugging.
6906
6907- You want to deploy a more minimal package selection of your device
6908 but allow in-the-field updates to add a larger selection for
6909 customization.
6910
6911In all these situations, you have something similar to a more
6912traditional Linux distribution in that in-field devices are able to
6913receive pre-compiled packages from a server for installation or update.
6914Being able to install these packages on a running, in-field device is
6915what is termed "runtime package management".
6916
6917In order to use runtime package management, you need a host or server
6918machine that serves up the pre-compiled packages plus the required
6919metadata. You also need package manipulation tools on the target. The
6920build machine is a likely candidate to act as the server. However, that
6921machine does not necessarily have to be the package server. The build
6922machine could push its artifacts to another machine that acts as the
6923server (e.g. Internet-facing). In fact, doing so is advantageous for a
6924production environment as getting the packages away from the development
6925system's build directory prevents accidental overwrites.
6926
6927A simple build that targets just one device produces more than one
6928package database. In other words, the packages produced by a build are
6929separated out into a couple of different package groupings based on
6930criteria such as the target's CPU architecture, the target board, or the
6931C library used on the target. For example, a build targeting the
6932``qemux86`` device produces the following three package databases:
6933``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
6934device to be aware of all the packages that were available to it, you
6935would need to point it to each of these databases individually. In a
6936similar way, a traditional Linux distribution usually is configured to
6937be aware of a number of software repositories from which it retrieves
6938packages.
6939
6940Using runtime package management is completely optional and not required
6941for a successful build or deployment in any way. But if you want to make
6942use of runtime package management, you need to do a couple things above
6943and beyond the basics. The remainder of this section describes what you
6944need to do.
6945
6946Build Considerations
6947~~~~~~~~~~~~~~~~~~~~
6948
6949This section describes build considerations of which you need to be
6950aware in order to provide support for runtime package management.
6951
6952When BitBake generates packages, it needs to know what format or formats
6953to use. In your configuration, you use the
6954:term:`PACKAGE_CLASSES`
6955variable to specify the format:
6956
69571. Open the ``local.conf`` file inside your
6958 :term:`Build Directory` (e.g.
6959 ``~/poky/build/conf/local.conf``).
6960
69612. Select the desired package format as follows:
6962 ::
6963
6964 PACKAGE_CLASSES ?= "package_packageformat"
6965
6966 where packageformat can be "ipk", "rpm",
6967 "deb", or "tar" which are the supported package formats.
6968
6969 .. note::
6970
6971 Because the Yocto Project supports four different package formats,
6972 you can set the variable with more than one argument. However, the
6973 OpenEmbedded build system only uses the first argument when
6974 creating an image or Software Development Kit (SDK).
6975
6976If you would like your image to start off with a basic package database
6977containing the packages in your current build as well as to have the
6978relevant tools available on the target for runtime package management,
6979you can include "package-management" in the
6980:term:`IMAGE_FEATURES`
6981variable. Including "package-management" in this configuration variable
6982ensures that when the image is assembled for your target, the image
6983includes the currently-known package databases as well as the
6984target-specific tools required for runtime package management to be
6985performed on the target. However, this is not strictly necessary. You
6986could start your image off without any databases but only include the
6987required on-target package tool(s). As an example, you could include
6988"opkg" in your
6989:term:`IMAGE_INSTALL` variable
6990if you are using the IPK package format. You can then initialize your
6991target's package database(s) later once your image is up and running.
6992
6993Whenever you perform any sort of build step that can potentially
6994generate a package or modify existing package, it is always a good idea
6995to re-generate the package index after the build by using the following
6996command:
6997::
6998
6999 $ bitbake package-index
7000
7001It might be tempting to build the
7002package and the package index at the same time with a command such as
7003the following:
7004::
7005
7006 $ bitbake some-package package-index
7007
7008Do not do this as
7009BitBake does not schedule the package index for after the completion of
7010the package you are building. Consequently, you cannot be sure of the
7011package index including information for the package you just built.
7012Thus, be sure to run the package update step separately after building
7013any packages.
7014
7015You can use the
7016:term:`PACKAGE_FEED_ARCHS`,
7017:term:`PACKAGE_FEED_BASE_PATHS`,
7018and
7019:term:`PACKAGE_FEED_URIS`
7020variables to pre-configure target images to use a package feed. If you
7021do not define these variables, then manual steps as described in the
7022subsequent sections are necessary to configure the target. You should
7023set these variables before building the image in order to produce a
7024correctly configured image.
7025
7026When your build is complete, your packages reside in the
7027``${TMPDIR}/deploy/packageformat`` directory. For example, if
7028``${``\ :term:`TMPDIR`\ ``}`` is
7029``tmp`` and your selected package type is RPM, then your RPM packages
7030are available in ``tmp/deploy/rpm``.
7031
7032Host or Server Machine Setup
7033~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7034
7035Although other protocols are possible, a server using HTTP typically
7036serves packages. If you want to use HTTP, then set up and configure a
7037web server such as Apache 2, lighttpd, or Python web server on the
7038machine serving the packages.
7039
7040To keep things simple, this section describes how to set up a
7041Python web server to share package feeds from the developer's
7042machine. Although this server might not be the best for a production
7043environment, the setup is simple and straight forward. Should you want
7044to use a different server more suited for production (e.g. Apache 2,
7045Lighttpd, or Nginx), take the appropriate steps to do so.
7046
7047From within the build directory where you have built an image based on
7048your packaging choice (i.e. the
7049:term:`PACKAGE_CLASSES`
7050setting), simply start the server. The following example assumes a build
7051directory of ``~/poky/build/tmp/deploy/rpm`` and a ``PACKAGE_CLASSES``
7052setting of "package_rpm":
7053::
7054
7055 $ cd ~/poky/build/tmp/deploy/rpm
7056 $ python3 -m http.server
7057
7058Target Setup
7059~~~~~~~~~~~~
7060
7061Setting up the target differs depending on the package management
7062system. This section provides information for RPM, IPK, and DEB.
7063
7064Using RPM
7065^^^^^^^^^
7066
7067The `Dandified Packaging
7068Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs
7069runtime package management of RPM packages. In order to use DNF for
7070runtime package management, you must perform an initial setup on the
7071target machine for cases where the ``PACKAGE_FEED_*`` variables were not
7072set as part of the image that is running on the target. This means if
7073you built your image and did not not use these variables as part of the
7074build and your image is now running on the target, you need to perform
7075the steps in this section if you want to use runtime package management.
7076
7077.. note::
7078
7079 For information on the ``PACKAGE_FEED_*`` variables, see
7080 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
7081 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
7082 glossary.
7083
7084On the target, you must inform DNF that package databases are available.
7085You do this by creating a file named
7086``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
7087
7088As an example, assume the target is able to use the following package
7089databases: ``all``, ``i586``, and ``qemux86`` from a server named
7090``my.server``. The specifics for setting up the web server are up to
7091you. The critical requirement is that the URIs in the target repository
7092configuration point to the correct remote location for the feeds.
7093
7094.. note::
7095
7096 For development purposes, you can point the web server to the build
7097 system's ``deploy`` directory. However, for production use, it is better to
7098 copy the package directories to a location outside of the build area and use
7099 that location. Doing so avoids situations where the build system
7100 overwrites or changes the ``deploy`` directory.
7101
7102When telling DNF where to look for the package databases, you must
7103declare individual locations per architecture or a single location used
7104for all architectures. You cannot do both:
7105
7106- *Create an Explicit List of Architectures:* Define individual base
7107 URLs to identify where each package database is located:
7108
7109 .. code-block:: none
7110
7111 [oe-packages]
7112 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
7113
7114 This example
7115 informs DNF about individual package databases for all three
7116 architectures.
7117
7118- *Create a Single (Full) Package Index:* Define a single base URL that
7119 identifies where a full package database is located:
7120 ::
7121
7122 [oe-packages]
7123 baseurl=http://my.server/rpm
7124
7125 This example informs DNF about a single
7126 package database that contains all the package index information for
7127 all supported architectures.
7128
7129Once you have informed DNF where to find the package databases, you need
7130to fetch them:
7131
7132.. code-block:: none
7133
7134 # dnf makecache
7135
7136DNF is now able to find, install, and
7137upgrade packages from the specified repository or repositories.
7138
7139.. note::
7140
7141 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
7142 additional information.
7143
7144Using IPK
7145^^^^^^^^^
7146
7147The ``opkg`` application performs runtime package management of IPK
7148packages. You must perform an initial setup for ``opkg`` on the target
7149machine if the
7150:term:`PACKAGE_FEED_ARCHS`,
7151:term:`PACKAGE_FEED_BASE_PATHS`,
7152and
7153:term:`PACKAGE_FEED_URIS`
7154variables have not been set or the target image was built before the
7155variables were set.
7156
7157The ``opkg`` application uses configuration files to find available
7158package databases. Thus, you need to create a configuration file inside
7159the ``/etc/opkg/`` direction, which informs ``opkg`` of any repository
7160you want to use.
7161
7162As an example, suppose you are serving packages from a ``ipk/``
7163directory containing the ``i586``, ``all``, and ``qemux86`` databases
7164through an HTTP server named ``my.server``. On the target, create a
7165configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7166directory containing the following:
7167
7168.. code-block:: none
7169
7170 src/gz all http://my.server/ipk/all
7171 src/gz i586 http://my.server/ipk/i586
7172 src/gz qemux86 http://my.server/ipk/qemux86
7173
7174Next, instruct ``opkg`` to fetch the
7175repository information:
7176
7177.. code-block:: none
7178
7179 # opkg update
7180
7181The ``opkg`` application is now able to find, install, and upgrade packages
7182from the specified repository.
7183
7184Using DEB
7185^^^^^^^^^
7186
7187The ``apt`` application performs runtime package management of DEB
7188packages. This application uses a source list file to find available
7189package databases. You must perform an initial setup for ``apt`` on the
7190target machine if the
7191:term:`PACKAGE_FEED_ARCHS`,
7192:term:`PACKAGE_FEED_BASE_PATHS`,
7193and
7194:term:`PACKAGE_FEED_URIS`
7195variables have not been set or the target image was built before the
7196variables were set.
7197
7198To inform ``apt`` of the repository you want to use, you might create a
7199list file (e.g. ``my_repo.list``) inside the
7200``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7201serving packages from a ``deb/`` directory containing the ``i586``,
7202``all``, and ``qemux86`` databases through an HTTP server named
7203``my.server``. The list file should contain:
7204
7205.. code-block:: none
7206
7207 deb http://my.server/deb/all ./
7208 deb http://my.server/deb/i586 ./
7209 deb http://my.server/deb/qemux86 ./
7210
7211Next, instruct the ``apt`` application
7212to fetch the repository information:
7213
7214.. code-block:: none
7215
7216 # apt-get update
7217
7218After this step,
7219``apt`` is able to find, install, and upgrade packages from the
7220specified repository.
7221
7222Generating and Using Signed Packages
7223------------------------------------
7224
7225In order to add security to RPM packages used during a build, you can
7226take steps to securely sign them. Once a signature is verified, the
7227OpenEmbedded build system can use the package in the build. If security
7228fails for a signed package, the build system aborts the build.
7229
7230This section describes how to sign RPM packages during a build and how
7231to use signed package feeds (repositories) when doing a build.
7232
7233Signing RPM Packages
7234~~~~~~~~~~~~~~~~~~~~
7235
7236To enable signing RPM packages, you must set up the following
7237configurations in either your ``local.config`` or ``distro.config``
7238file:
7239::
7240
7241 # Inherit sign_rpm.bbclass to enable signing functionality
7242 INHERIT += " sign_rpm"
7243 # Define the GPG key that will be used for signing.
7244 RPM_GPG_NAME = "key_name"
7245 # Provide passphrase for the key
7246 RPM_GPG_PASSPHRASE = "passphrase"
7247
7248.. note::
7249
7250 Be sure to supply appropriate values for both `key_name` and
7251 `passphrase`.
7252
7253Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
7254the previous example, two optional variables related to signing exist:
7255
7256- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7257 when the package is signed.
7258
7259- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7260 package is signed.
7261
7262Processing Package Feeds
7263~~~~~~~~~~~~~~~~~~~~~~~~
7264
7265In addition to being able to sign RPM packages, you can also enable
7266signed package feeds for IPK and RPM packages.
7267
7268The steps you need to take to enable signed package feed use are similar
7269to the steps used to sign RPM packages. You must define the following in
7270your ``local.config`` or ``distro.config`` file:
7271::
7272
7273 INHERIT += "sign_package_feed"
7274 PACKAGE_FEED_GPG_NAME = "key_name"
7275 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7276
7277For signed package feeds, the passphrase must exist in a separate file,
7278which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7279variable. Regarding security, keeping a plain text passphrase out of the
7280configuration is more secure.
7281
7282Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7283``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
7284related to signed package feeds exist:
7285
7286- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7287 when the package is signed.
7288
7289- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7290 package is signed.
7291
7292- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7293 signature. This variable applies only to RPM and IPK package feeds.
7294 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7295 "ASC", which is the default and specifies ascii armored, and "BIN",
7296 which specifies binary.
7297
7298Testing Packages With ptest
7299---------------------------
7300
7301A Package Test (ptest) runs tests against packages built by the
7302OpenEmbedded build system on the target machine. A ptest contains at
7303least two items: the actual test, and a shell script (``run-ptest``)
7304that starts the test. The shell script that starts the test must not
7305contain the actual test - the script only starts the test. On the other
7306hand, the test can be anything from a simple shell script that runs a
7307binary and checks the output to an elaborate system of test binaries and
7308data files.
7309
7310The test generates output in the format used by Automake:
7311::
7312
7313 result: testname
7314
7315where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7316the testname can be any identifying string.
7317
7318For a list of Yocto Project recipes that are already enabled with ptest,
7319see the :yocto_wiki:`Ptest </Ptest>` wiki page.
7320
7321.. note::
7322
7323 A recipe is "ptest-enabled" if it inherits the
7324 :ref:`ptest <ref-classes-ptest>` class.
7325
7326Adding ptest to Your Build
7327~~~~~~~~~~~~~~~~~~~~~~~~~~
7328
7329To add package testing to your build, add the
7330:term:`DISTRO_FEATURES` and
7331:term:`EXTRA_IMAGE_FEATURES`
7332variables to your ``local.conf`` file, which is found in the
7333:term:`Build Directory`:
7334::
7335
7336 DISTRO_FEATURES_append = " ptest"
7337 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7338
7339Once your build is complete, the ptest files are installed into the
7340``/usr/lib/package/ptest`` directory within the image, where ``package``
7341is the name of the package.
7342
7343Running ptest
7344~~~~~~~~~~~~~
7345
7346The ``ptest-runner`` package installs a shell script that loops through
7347all installed ptest test suites and runs them in sequence. Consequently,
7348you might want to add this package to your image.
7349
7350Getting Your Package Ready
7351~~~~~~~~~~~~~~~~~~~~~~~~~~
7352
7353In order to enable a recipe to run installed ptests on target hardware,
7354you need to prepare the recipes that build the packages you want to
7355test. Here is what you have to do for each recipe:
7356
7357- *Be sure the recipe inherits
7358 the* :ref:`ptest <ref-classes-ptest>` *class:*
7359 Include the following line in each recipe:
7360 ::
7361
7362 inherit ptest
7363
7364- *Create run-ptest:* This script starts your test. Locate the
7365 script where you will refer to it using
7366 :term:`SRC_URI`. Here is an
7367 example that starts a test for ``dbus``:
7368 ::
7369
7370 #!/bin/sh
7371 cd test
7372 make -k runtest-TESTS
7373
7374- *Ensure dependencies are met:* If the test adds build or runtime
7375 dependencies that normally do not exist for the package (such as
7376 requiring "make" to run the test suite), use the
7377 :term:`DEPENDS` and
7378 :term:`RDEPENDS` variables in
7379 your recipe in order for the package to meet the dependencies. Here
7380 is an example where the package has a runtime dependency on "make":
7381 ::
7382
7383 RDEPENDS_${PN}-ptest += "make"
7384
7385- *Add a function to build the test suite:* Not many packages support
7386 cross-compilation of their test suites. Consequently, you usually
7387 need to add a cross-compilation function to the package.
7388
7389 Many packages based on Automake compile and run the test suite by
7390 using a single command such as ``make check``. However, the host
7391 ``make check`` builds and runs on the same computer, while
7392 cross-compiling requires that the package is built on the host but
7393 executed for the target architecture (though often, as in the case
7394 for ptest, the execution occurs on the host). The built version of
7395 Automake that ships with the Yocto Project includes a patch that
7396 separates building and execution. Consequently, packages that use the
7397 unaltered, patched version of ``make check`` automatically
7398 cross-compiles.
7399
7400 Regardless, you still must add a ``do_compile_ptest`` function to
7401 build the test suite. Add a function similar to the following to your
7402 recipe:
7403 ::
7404
7405 do_compile_ptest() {
7406 oe_runmake buildtest-TESTS
7407 }
7408
7409- *Ensure special configurations are set:* If the package requires
7410 special configurations prior to compiling the test code, you must
7411 insert a ``do_configure_ptest`` function into the recipe.
7412
7413- *Install the test suite:* The ``ptest`` class automatically copies
7414 the file ``run-ptest`` to the target and then runs make
7415 ``install-ptest`` to run the tests. If this is not enough, you need
7416 to create a ``do_install_ptest`` function and make sure it gets
7417 called after the "make install-ptest" completes.
7418
7419Creating Node Package Manager (NPM) Packages
7420--------------------------------------------
7421
7422`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package
7423manager for the JavaScript programming language. The Yocto Project
7424supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
7425use this fetcher in combination with
7426:doc:`devtool </ref-manual/devtool-reference>` to create
7427recipes that produce NPM packages.
7428
7429Two workflows exist that allow you to create NPM packages using
7430``devtool``: the NPM registry modules method and the NPM project code
7431method.
7432
7433.. note::
7434
7435 While it is possible to create NPM recipes manually, using
7436 ``devtool`` is far simpler.
7437
7438Additionally, some requirements and caveats exist.
7439
7440Requirements and Caveats
7441~~~~~~~~~~~~~~~~~~~~~~~~
7442
7443You need to be aware of the following before using ``devtool`` to create
7444NPM packages:
7445
7446- Of the two methods that you can use ``devtool`` to create NPM
7447 packages, the registry approach is slightly simpler. However, you
7448 might consider the project approach because you do not have to
7449 publish your module in the NPM registry
7450 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which
7451 is NPM's public registry.
7452
7453- Be familiar with
7454 :doc:`devtool </ref-manual/devtool-reference>`.
7455
7456- The NPM host tools need the native ``nodejs-npm`` package, which is
7457 part of the OpenEmbedded environment. You need to get the package by
7458 cloning the https://github.com/openembedded/meta-openembedded
7459 repository out of GitHub. Be sure to add the path to your local copy
7460 to your ``bblayers.conf`` file.
7461
7462- ``devtool`` cannot detect native libraries in module dependencies.
7463 Consequently, you must manually add packages to your recipe.
7464
7465- While deploying NPM packages, ``devtool`` cannot determine which
7466 dependent packages are missing on the target (e.g. the node runtime
7467 ``nodejs``). Consequently, you need to find out what files are
7468 missing and be sure they are on the target.
7469
7470- Although you might not need NPM to run your node package, it is
7471 useful to have NPM on your target. The NPM package name is
7472 ``nodejs-npm``.
7473
7474Using the Registry Modules Method
7475~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7476
7477This section presents an example that uses the ``cute-files`` module,
7478which is a file browser web application.
7479
7480.. note::
7481
7482 You must know the ``cute-files`` module version.
7483
7484The first thing you need to do is use ``devtool`` and the NPM fetcher to
7485create the recipe:
7486::
7487
7488 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7489
7490The
7491``devtool add`` command runs ``recipetool create`` and uses the same
7492fetch URI to download each dependency and capture license details where
7493possible. The result is a generated recipe.
7494
7495The recipe file is fairly simple and contains every license that
7496``recipetool`` finds and includes the licenses in the recipe's
7497:term:`LIC_FILES_CHKSUM`
7498variables. You need to examine the variables and look for those with
7499"unknown" in the :term:`LICENSE`
7500field. You need to track down the license information for "unknown"
7501modules and manually add the information to the recipe.
7502
7503``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7504files capture the version of all dependent modules. Many packages do not
7505provide shrinkwrap files. ``recipetool`` create a shrinkwrap file as it
7506runs.
7507
7508.. note::
7509
7510 A package is created for each sub-module. This policy is the only
7511 practical way to have the licenses for all of the dependencies
7512 represented in the license manifest of the image.
7513
7514The ``devtool edit-recipe`` command lets you take a look at the recipe:
7515::
7516
7517 $ devtool edit-recipe cute-files
7518 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7519 LICENSE = "MIT & ISC & Unknown"
7520 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
7521 file://node_modules/toidentifier/LICENSE;md5=1a261071a044d02eb6f2bb47f51a3502 \
7522 file://node_modules/debug/LICENSE;md5=ddd815a475e7338b0be7a14d8ee35a99 \
7523 ...
7524 SRC_URI = " \
7525 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7526 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7527 "
7528 S = "${WORKDIR}/npm"
7529 inherit npm LICENSE_${PN} = "MIT"
7530 LICENSE_${PN}-accepts = "MIT"
7531 LICENSE_${PN}-array-flatten = "MIT"
7532 ...
7533 LICENSE_${PN}-vary = "MIT"
7534
7535Three key points exist in the previous example:
7536
7537- :term:`SRC_URI` uses the NPM
7538 scheme so that the NPM fetcher is used.
7539
7540- ``recipetool`` collects all the license information. If a
7541 sub-module's license is unavailable, the sub-module's name appears in
7542 the comments.
7543
7544- The ``inherit npm`` statement causes the
7545 :ref:`npm <ref-classes-npm>` class to package
7546 up all the modules.
7547
7548You can run the following command to build the ``cute-files`` package:
7549::
7550
7551 $ devtool build cute-files
7552
7553Remember that ``nodejs`` must be installed on
7554the target before your package.
7555
7556Assuming 192.168.7.2 for the target's IP address, use the following
7557command to deploy your package:
7558::
7559
7560 $ devtool deploy-target -s cute-files root@192.168.7.2
7561
7562Once the package is installed on the target, you can
7563test the application:
7564
7565.. note::
7566
7567 Because of a known issue, you cannot simply run ``cute-files`` as you would
7568 if you had run ``npm install``.
7569
7570::
7571
7572 $ cd /usr/lib/node_modules/cute-files
7573 $ node cute-files.js
7574
7575On a browser,
7576go to ``http://192.168.7.2:3000`` and you see the following:
7577
7578.. image:: figures/cute-files-npm-example.png
7579 :align: center
7580
7581You can find the recipe in ``workspace/recipes/cute-files``. You can use
7582the recipe in any layer you choose.
7583
7584Using the NPM Projects Code Method
7585~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7586
7587Although it is useful to package modules already in the NPM registry,
7588adding ``node.js`` projects under development is a more common developer
7589use case.
7590
7591This section covers the NPM projects code method, which is very similar
7592to the "registry" approach described in the previous section. In the NPM
7593projects method, you provide ``devtool`` with an URL that points to the
7594source files.
7595
7596Replicating the same example, (i.e. ``cute-files``) use the following
7597command:
7598::
7599
7600 $ devtool add https://github.com/martinaglv/cute-files.git
7601
7602The
7603recipe this command generates is very similar to the recipe created in
7604the previous section. However, the ``SRC_URI`` looks like the following:
7605::
7606
7607 SRC_URI = " \
7608 git://github.com/martinaglv/cute-files.git;protocol=https \
7609 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7610 "
7611
7612In this example,
7613the main module is taken from the Git repository and dependencies are
7614taken from the NPM registry. Other than those differences, the recipe is
7615basically the same between the two methods. You can build and deploy the
7616package exactly as described in the previous section that uses the
7617registry modules method.
7618
7619Adding custom metadata to packages
7620----------------------------------
7621
7622The variable
7623:term:`PACKAGE_ADD_METADATA`
7624can be used to add additional metadata to packages. This is reflected in
7625the package control/spec file. To take the ipk format for example, the
7626CONTROL file stored inside would contain the additional metadata as
7627additional lines.
7628
7629The variable can be used in multiple ways, including using suffixes to
7630set it for a specific package type and/or package. Note that the order
7631of precedence is the same as this list:
7632
7633- ``PACKAGE_ADD_METADATA_<PKGTYPE>_<PN>``
7634
7635- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7636
7637- ``PACKAGE_ADD_METADATA_<PN>``
7638
7639- ``PACKAGE_ADD_METADATA``
7640
7641`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
7642package type:
7643
7644- IPK for .ipk packages
7645
7646- DEB for .deb packages
7647
7648- RPM for .rpm packages
7649
7650`<PN>` is a parameter and expected to be a package name.
7651
7652The variable can contain multiple [one-line] metadata fields separated
7653by the literal sequence '\\n'. The separator can be redefined using the
7654variable flag ``separator``.
7655
7656The following is an example that adds two custom fields for ipk
7657packages:
7658::
7659
7660 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
7661
7662Efficiently Fetching Source Files During a Build
7663================================================
7664
7665The OpenEmbedded build system works with source files located through
7666the :term:`SRC_URI` variable. When
7667you build something using BitBake, a big part of the operation is
7668locating and downloading all the source tarballs. For images,
7669downloading all the source for various packages can take a significant
7670amount of time.
7671
7672This section shows you how you can use mirrors to speed up fetching
7673source files and how you can pre-fetch files all of which leads to more
7674efficient use of resources and time.
7675
7676Setting up Effective Mirrors
7677----------------------------
7678
7679A good deal that goes into a Yocto Project build is simply downloading
7680all of the source tarballs. Maybe you have been working with another
7681build system (OpenEmbedded or Angstrom) for which you have built up a
7682sizable directory of source tarballs. Or, perhaps someone else has such
7683a directory for which you have read access. If so, you can save time by
7684adding statements to your configuration file so that the build process
7685checks local directories first for existing tarballs before checking the
7686Internet.
7687
7688Here is an efficient way to set it up in your ``local.conf`` file:
7689::
7690
7691 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7692 INHERIT += "own-mirrors"
7693 BB_GENERATE_MIRROR_TARBALLS = "1"
7694 # BB_NO_NETWORK = "1"
7695
7696In the previous example, the
7697:term:`BB_GENERATE_MIRROR_TARBALLS`
7698variable causes the OpenEmbedded build system to generate tarballs of
7699the Git repositories and store them in the
7700:term:`DL_DIR` directory. Due to
7701performance reasons, generating and storing these tarballs is not the
7702build system's default behavior.
7703
7704You can also use the
7705:term:`PREMIRRORS` variable. For
7706an example, see the variable's glossary entry in the Yocto Project
7707Reference Manual.
7708
7709Getting Source Files and Suppressing the Build
7710----------------------------------------------
7711
7712Another technique you can use to ready yourself for a successive string
7713of build operations, is to pre-fetch all the source files without
7714actually starting a build. This technique lets you work through any
7715download issues and ultimately gathers all the source files into your
7716download directory :ref:`structure-build-downloads`,
7717which is located with :term:`DL_DIR`.
7718
7719Use the following BitBake command form to fetch all the necessary
7720sources without starting the build:
7721::
7722
7723 $ bitbake target --runall=fetch
7724
7725This
7726variation of the BitBake command guarantees that you have all the
7727sources for that BitBake target should you disconnect from the Internet
7728and want to do the build later offline.
7729
7730Selecting an Initialization Manager
7731===================================
7732
7733By default, the Yocto Project uses SysVinit as the initialization
7734manager. However, support also exists for systemd, which is a full
7735replacement for init with parallel starting of services, reduced shell
7736overhead and other features that are used by many distributions.
7737
7738Within the system, SysVinit treats system components as services. These
7739services are maintained as shell scripts stored in the ``/etc/init.d/``
7740directory. Services organize into different run levels. This
7741organization is maintained by putting links to the services in the
7742``/etc/rcN.d/`` directories, where `N/` is one of the following options:
7743"S", "0", "1", "2", "3", "4", "5", or "6".
7744
7745.. note::
7746
7747 Each runlevel has a dependency on the previous runlevel. This
7748 dependency allows the services to work properly.
7749
7750In comparison, systemd treats components as units. Using units is a
7751broader concept as compared to using a service. A unit includes several
7752different types of entities. Service is one of the types of entities.
7753The runlevel concept in SysVinit corresponds to the concept of a target
7754in systemd, where target is also a type of supported unit.
7755
7756In a SysVinit-based system, services load sequentially (i.e. one by one)
7757during init and parallelization is not supported. With systemd, services
7758start in parallel. Needless to say, the method can have an impact on
7759system startup performance.
7760
7761If you want to use SysVinit, you do not have to do anything. But, if you
7762want to use systemd, you must take some steps as described in the
7763following sections.
7764
7765Using systemd Exclusively
7766-------------------------
7767
7768Set these variables in your distribution configuration file as follows:
7769::
7770
7771 DISTRO_FEATURES_append = " systemd"
7772 VIRTUAL-RUNTIME_init_manager = "systemd"
7773
7774You can also prevent the SysVinit distribution feature from
7775being automatically enabled as follows:
7776::
7777
7778 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7779
7780Doing so removes any
7781redundant SysVinit scripts.
7782
7783To remove initscripts from your image altogether, set this variable
7784also:
7785::
7786
7787 VIRTUAL-RUNTIME_initscripts = ""
7788
7789For information on the backfill variable, see
7790:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7791
7792Using systemd for the Main Image and Using SysVinit for the Rescue Image
7793------------------------------------------------------------------------
7794
7795Set these variables in your distribution configuration file as follows:
7796::
7797
7798 DISTRO_FEATURES_append = " systemd"
7799 VIRTUAL-RUNTIME_init_manager = "systemd"
7800
7801Doing so causes your main image to use the
7802``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7803image cannot use this package group. However, it can install SysVinit
7804and the appropriate packages will have support for both systemd and
7805SysVinit.
7806
7807Selecting a Device Manager
7808==========================
7809
7810The Yocto Project provides multiple ways to manage the device manager
7811(``/dev``):
7812
7813- Persistent and Pre-Populated\ ``/dev``: For this case, the ``/dev``
7814 directory is persistent and the required device nodes are created
7815 during the build.
7816
7817- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7818 directory is provided by the kernel as an in-memory file system and
7819 is automatically populated by the kernel at runtime. Additional
7820 configuration of device nodes is done in user space by a device
7821 manager like ``udev`` or ``busybox-mdev``.
7822
7823Using Persistent and Pre-Populated\ ``/dev``
7824--------------------------------------------
7825
7826To use the static method for device population, you need to set the
7827:term:`USE_DEVFS` variable to "0"
7828as follows:
7829::
7830
7831 USE_DEVFS = "0"
7832
7833The content of the resulting ``/dev`` directory is defined in a Device
7834Table file. The
7835:term:`IMAGE_DEVICE_TABLES`
7836variable defines the Device Table to use and should be set in the
7837machine or distro configuration file. Alternatively, you can set this
7838variable in your ``local.conf`` configuration file.
7839
7840If you do not define the ``IMAGE_DEVICE_TABLES`` variable, the default
7841``device_table-minimal.txt`` is used:
7842::
7843
7844 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7845
7846The population is handled by the ``makedevs`` utility during image
7847creation:
7848
7849Using ``devtmpfs`` and a Device Manager
7850---------------------------------------
7851
7852To use the dynamic method for device population, you need to use (or be
7853sure to set) the :term:`USE_DEVFS`
7854variable to "1", which is the default:
7855::
7856
7857 USE_DEVFS = "1"
7858
7859With this
7860setting, the resulting ``/dev`` directory is populated by the kernel
7861using ``devtmpfs``. Make sure the corresponding kernel configuration
7862variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
7863kernel.
7864
7865All devices created by ``devtmpfs`` will be owned by ``root`` and have
7866permissions ``0600``.
7867
7868To have more control over the device nodes, you can use a device manager
7869like ``udev`` or ``busybox-mdev``. You choose the device manager by
7870defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
7871distro configuration file. Alternatively, you can set this variable in
7872your ``local.conf`` configuration file:
7873::
7874
7875 VIRTUAL-RUNTIME_dev_manager = "udev"
7876
7877 # Some alternative values
7878 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
7879 # VIRTUAL-RUNTIME_dev_manager = "systemd"
7880
7881Using an External SCM
7882=====================
7883
7884If you're working on a recipe that pulls from an external Source Code
7885Manager (SCM), it is possible to have the OpenEmbedded build system
7886notice new recipe changes added to the SCM and then build the resulting
7887packages that depend on the new recipes by using the latest versions.
7888This only works for SCMs from which it is possible to get a sensible
7889revision number for changes. Currently, you can do this with Apache
7890Subversion (SVN), Git, and Bazaar (BZR) repositories.
7891
7892To enable this behavior, the :term:`PV` of
7893the recipe needs to reference
7894:term:`SRCPV`. Here is an example:
7895::
7896
7897 PV = "1.2.3+git${SRCPV}"
7898
7899Then, you can add the following to your
7900``local.conf``:
7901::
7902
7903 SRCREV_pn-PN = "${AUTOREV}"
7904
7905:term:`PN` is the name of the recipe for
7906which you want to enable automatic source revision updating.
7907
7908If you do not want to update your local configuration file, you can add
7909the following directly to the recipe to finish enabling the feature:
7910::
7911
7912 SRCREV = "${AUTOREV}"
7913
7914The Yocto Project provides a distribution named ``poky-bleeding``, whose
7915configuration file contains the line:
7916::
7917
7918 require conf/distro/include/poky-floating-revisions.inc
7919
7920This line pulls in the
7921listed include file that contains numerous lines of exactly that form:
7922::
7923
7924 #SRCREV_pn-opkg-native ?= "${AUTOREV}"
7925 #SRCREV_pn-opkg-sdk ?= "${AUTOREV}"
7926 #SRCREV_pn-opkg ?= "${AUTOREV}"
7927 #SRCREV_pn-opkg-utils-native ?= "${AUTOREV}"
7928 #SRCREV_pn-opkg-utils ?= "${AUTOREV}"
7929 SRCREV_pn-gconf-dbus ?= "${AUTOREV}"
7930 SRCREV_pn-matchbox-common ?= "${AUTOREV}"
7931 SRCREV_pn-matchbox-config-gtk ?= "${AUTOREV}"
7932 SRCREV_pn-matchbox-desktop ?= "${AUTOREV}"
7933 SRCREV_pn-matchbox-keyboard ?= "${AUTOREV}"
7934 SRCREV_pn-matchbox-panel-2 ?= "${AUTOREV}"
7935 SRCREV_pn-matchbox-themes-extra ?= "${AUTOREV}"
7936 SRCREV_pn-matchbox-terminal ?= "${AUTOREV}"
7937 SRCREV_pn-matchbox-wm ?= "${AUTOREV}"
7938 SRCREV_pn-settings-daemon ?= "${AUTOREV}"
7939 SRCREV_pn-screenshot ?= "${AUTOREV}"
7940 . . .
7941
7942These lines allow you to
7943experiment with building a distribution that tracks the latest
7944development source for numerous packages.
7945
7946.. note::
7947
7948 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
7949 this in mind if you use it.
7950
7951Creating a Read-Only Root Filesystem
7952====================================
7953
7954Suppose, for security reasons, you need to disable your target device's
7955root filesystem's write permissions (i.e. you need a read-only root
7956filesystem). Or, perhaps you are running the device's operating system
7957from a read-only storage device. For either case, you can customize your
7958image for that behavior.
7959
7960.. note::
7961
7962 Supporting a read-only root filesystem requires that the system and
7963 applications do not try to write to the root filesystem. You must
7964 configure all parts of the target system to write elsewhere, or to
7965 gracefully fail in the event of attempting to write to the root
7966 filesystem.
7967
7968Creating the Root Filesystem
7969----------------------------
7970
7971To create the read-only root filesystem, simply add the
7972"read-only-rootfs" feature to your image, normally in one of two ways.
7973The first way is to add the "read-only-rootfs" image feature in the
7974image's recipe file via the ``IMAGE_FEATURES`` variable:
7975::
7976
7977 IMAGE_FEATURES += "read-only-rootfs"
7978
7979As an alternative, you can add the same feature
7980from within your build directory's ``local.conf`` file with the
7981associated ``EXTRA_IMAGE_FEATURES`` variable, as in:
7982::
7983
7984 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
7985
7986For more information on how to use these variables, see the
7987":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
7988section. For information on the variables, see
7989:term:`IMAGE_FEATURES` and
7990:term:`EXTRA_IMAGE_FEATURES`.
7991
7992Post-Installation Scripts and Read-Only Root Filesystem
7993-------------------------------------------------------
7994
7995It is very important that you make sure all post-Installation
7996(``pkg_postinst``) scripts for packages that are installed into the
7997image can be run at the time when the root filesystem is created during
7998the build on the host system. These scripts cannot attempt to run during
7999first-boot on the target device. With the "read-only-rootfs" feature
8000enabled, the build system checks during root filesystem creation to make
8001sure all post-installation scripts succeed. If any of these scripts
8002still need to be run after the root filesystem is created, the build
8003immediately fails. These build-time checks ensure that the build fails
8004rather than the target device fails later during its initial boot
8005operation.
8006
8007Most of the common post-installation scripts generated by the build
8008system for the out-of-the-box Yocto Project are engineered so that they
8009can run during root filesystem creation (e.g. post-installation scripts
8010for caching fonts). However, if you create and add custom scripts, you
8011need to be sure they can be run during this file system creation.
8012
8013Here are some common problems that prevent post-installation scripts
8014from running during root filesystem creation:
8015
8016- *Not using $D in front of absolute paths:* The build system defines
8017 ``$``\ :term:`D` when the root
8018 filesystem is created. Furthermore, ``$D`` is blank when the script
8019 is run on the target device. This implies two purposes for ``$D``:
8020 ensuring paths are valid in both the host and target environments,
8021 and checking to determine which environment is being used as a method
8022 for taking appropriate actions.
8023
8024- *Attempting to run processes that are specific to or dependent on the
8025 target architecture:* You can work around these attempts by using
8026 native tools, which run on the host system, to accomplish the same
8027 tasks, or by alternatively running the processes under QEMU, which
8028 has the ``qemu_run_binary`` function. For more information, see the
8029 :ref:`qemu <ref-classes-qemu>` class.
8030
8031Areas With Write Access
8032-----------------------
8033
8034With the "read-only-rootfs" feature enabled, any attempt by the target
8035to write to the root filesystem at runtime fails. Consequently, you must
8036make sure that you configure processes and applications that attempt
8037these types of writes do so to directories with write access (e.g.
8038``/tmp`` or ``/var/run``).
8039
8040Maintaining Build Output Quality
8041================================
8042
8043Many factors can influence the quality of a build. For example, if you
8044upgrade a recipe to use a new version of an upstream software package or
8045you experiment with some new configuration options, subtle changes can
8046occur that you might not detect until later. Consider the case where
8047your recipe is using a newer version of an upstream package. In this
8048case, a new version of a piece of software might introduce an optional
8049dependency on another library, which is auto-detected. If that library
8050has already been built when the software is building, the software will
8051link to the built library and that library will be pulled into your
8052image along with the new software even if you did not want the library.
8053
8054The :ref:`buildhistory <ref-classes-buildhistory>`
8055class exists to help you maintain the quality of your build output. You
8056can use the class to highlight unexpected and possibly unwanted changes
8057in the build output. When you enable build history, it records
8058information about the contents of each package and image and then
8059commits that information to a local Git repository where you can examine
8060the information.
8061
8062The remainder of this section describes the following:
8063
8064- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>`
8065
8066- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>`
8067
8068- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>`
8069
8070- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>`
8071
8072Enabling and Disabling Build History
8073------------------------------------
8074
8075Build history is disabled by default. To enable it, add the following
8076``INHERIT`` statement and set the
8077:term:`BUILDHISTORY_COMMIT`
8078variable to "1" at the end of your ``conf/local.conf`` file found in the
8079:term:`Build Directory`:
8080::
8081
8082 INHERIT += "buildhistory"
8083 BUILDHISTORY_COMMIT = "1"
8084
8085Enabling build history as
8086previously described causes the OpenEmbedded build system to collect
8087build output information and commit it as a single commit to a local
8088:ref:`overview-manual/development-environment:git` repository.
8089
8090.. note::
8091
8092 Enabling build history increases your build times slightly,
8093 particularly for images, and increases the amount of disk space used
8094 during the build.
8095
8096You can disable build history by removing the previous statements from
8097your ``conf/local.conf`` file.
8098
8099Understanding What the Build History Contains
8100---------------------------------------------
8101
8102Build history information is kept in
8103``${``\ :term:`TOPDIR`\ ``}/buildhistory``
8104in the Build Directory as defined by the
8105:term:`BUILDHISTORY_DIR`
8106variable. The following is an example abbreviated listing:
8107
8108.. image:: figures/buildhistory.png
8109 :align: center
8110
8111At the top level, a ``metadata-revs`` file exists that lists the
8112revisions of the repositories for the enabled layers when the build was
8113produced. The rest of the data splits into separate ``packages``,
8114``images`` and ``sdk`` directories, the contents of which are described
8115as follows.
8116
8117Build History Package Information
8118~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8119
8120The history for each package contains a text file that has name-value
8121pairs with information about the package. For example,
8122``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
8123contains the following:
8124
8125.. code-block:: none
8126
8127 PV = 1.22.1
8128 PR = r32
8129 RPROVIDES =
8130 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
8131 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
8132 PKGSIZE = 540168
8133 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
8134 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
8135 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
8136 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
8137 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
8138 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
8139 /etc/busybox.links.nosuid /etc/busybox.links.suid
8140
8141Most of these
8142name-value pairs correspond to variables used to produce the package.
8143The exceptions are ``FILELIST``, which is the actual list of files in
8144the package, and ``PKGSIZE``, which is the total size of files in the
8145package in bytes.
8146
8147A file also exists that corresponds to the recipe from which the package
8148came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
8149
8150.. code-block:: none
8151
8152 PV = 1.22.1
8153 PR = r32
8154 DEPENDS = initscripts kern-tools-native update-rc.d-native \
8155 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
8156 virtual/libc virtual/update-alternatives
8157 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
8158 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
8159 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
8160
8161Finally, for those recipes fetched from a version control system (e.g.,
8162Git), a file exists that lists source revisions that are specified in
8163the recipe and lists the actual revisions used during the build. Listed
8164and actual revisions might differ when
8165:term:`SRCREV` is set to
8166${:term:`AUTOREV`}. Here is an
8167example assuming
8168``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``):
8169::
8170
8171 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8172 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8173 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
8174 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
8175
8176You can use the
8177``buildhistory-collect-srcrevs`` command with the ``-a`` option to
8178collect the stored ``SRCREV`` values from build history and report them
8179in a format suitable for use in global configuration (e.g.,
8180``local.conf`` or a distro include file) to override floating
8181``AUTOREV`` values to a fixed set of revisions. Here is some example
8182output from this command:
8183::
8184
8185 $ buildhistory-collect-srcrevs -a
8186 # i586-poky-linux
8187 SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
8188 SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
8189 SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
8190 SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
8191 # x86_64-linux
8192 SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
8193 SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
8194 SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
8195 SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
8196 SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
8197 SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
8198 SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
8199 SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
8200 SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
8201 # qemux86-poky-linux
8202 SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8203 SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
8204 # all-poky-linux
8205 SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
8206
8207.. note::
8208
8209 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
8210
8211 - By default, only values where the ``SRCREV`` was not hardcoded
8212 (usually when ``AUTOREV`` is used) are reported. Use the ``-a``
8213 option to see all ``SRCREV`` values.
8214
8215 - The output statements might not have any effect if overrides are
8216 applied elsewhere in the build system configuration. Use the
8217 ``-f`` option to add the ``forcevariable`` override to each output
8218 line if you need to work around this restriction.
8219
8220 - The script does apply special handling when building for multiple
8221 machines. However, the script does place a comment before each set
8222 of values that specifies which triplet to which they belong as
8223 previously shown (e.g., ``i586-poky-linux``).
8224
8225Build History Image Information
8226~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8227
8228The files produced for each image are as follows:
8229
8230- ``image-files:`` A directory containing selected files from the root
8231 filesystem. The files are defined by
8232 :term:`BUILDHISTORY_IMAGE_FILES`.
8233
8234- ``build-id.txt:`` Human-readable information about the build
8235 configuration and metadata source revisions. This file contains the
8236 full build header as printed by BitBake.
8237
8238- ``*.dot:`` Dependency graphs for the image that are compatible with
8239 ``graphviz``.
8240
8241- ``files-in-image.txt:`` A list of files in the image with
8242 permissions, owner, group, size, and symlink information.
8243
8244- ``image-info.txt:`` A text file containing name-value pairs with
8245 information about the image. See the following listing example for
8246 more information.
8247
8248- ``installed-package-names.txt:`` A list of installed packages by name
8249 only.
8250
8251- ``installed-package-sizes.txt:`` A list of installed packages ordered
8252 by size.
8253
8254- ``installed-packages.txt:`` A list of installed packages with full
8255 package filenames.
8256
8257.. note::
8258
8259 Installed package information is able to be gathered and produced
8260 even if package management is disabled for the final image.
8261
8262Here is an example of ``image-info.txt``:
8263
8264.. code-block:: none
8265
8266 DISTRO = poky
8267 DISTRO_VERSION = 1.7
8268 USER_CLASSES = buildstats image-mklibs image-prelink
8269 IMAGE_CLASSES = image_types
8270 IMAGE_FEATURES = debug-tweaks
8271 IMAGE_LINGUAS =
8272 IMAGE_INSTALL = packagegroup-core-boot run-postinsts
8273 BAD_RECOMMENDATIONS =
8274 NO_RECOMMENDATIONS =
8275 PACKAGE_EXCLUDE =
8276 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
8277 write_image_manifest ; buildhistory_list_installed_image ; \
8278 buildhistory_get_image_installed ; ssh_allow_empty_password; \
8279 postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
8280 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8281 IMAGESIZE = 6900
8282
8283Other than ``IMAGESIZE``,
8284which is the total size of the files in the image in Kbytes, the
8285name-value pairs are variables that may have influenced the content of
8286the image. This information is often useful when you are trying to
8287determine why a change in the package or file listings has occurred.
8288
8289Using Build History to Gather Image Information Only
8290~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8291
8292As you can see, build history produces image information, including
8293dependency graphs, so you can see why something was pulled into the
8294image. If you are just interested in this information and not interested
8295in collecting specific package or SDK information, you can enable
8296writing only image information without any history by adding the
8297following to your ``conf/local.conf`` file found in the
8298:term:`Build Directory`:
8299::
8300
8301 INHERIT += "buildhistory"
8302 BUILDHISTORY_COMMIT = "0"
8303 BUILDHISTORY_FEATURES = "image"
8304
8305Here, you set the
8306:term:`BUILDHISTORY_FEATURES`
8307variable to use the image feature only.
8308
8309Build History SDK Information
8310~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8311
8312Build history collects similar information on the contents of SDKs (e.g.
8313``bitbake -c populate_sdk imagename``) as compared to information it
8314collects for images. Furthermore, this information differs depending on
8315whether an extensible or standard SDK is being produced.
8316
8317The following list shows the files produced for SDKs:
8318
8319- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8320 owner, group, size, and symlink information. This list includes both
8321 the host and target parts of the SDK.
8322
8323- ``sdk-info.txt:`` A text file containing name-value pairs with
8324 information about the SDK. See the following listing example for more
8325 information.
8326
8327- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8328 with information about task group sizes (e.g. ``do_populate_sysroot``
8329 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8330 only when an extensible SDK is created.
8331
8332- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8333 with information for the shared-state packages and sizes in the SDK.
8334 The ``sstate-package-sizes.txt`` file exists only when an extensible
8335 SDK is created.
8336
8337- ``sdk-files:`` A folder that contains copies of the files mentioned
8338 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8339 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8340 specific to the extensible SDK although you can set it differently if
8341 you would like to pull in specific files from the standard SDK.
8342
8343 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8344 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8345 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8346 copied into the ``sdk-files`` directory.
8347
8348- The following information appears under each of the ``host`` and
8349 ``target`` directories for the portions of the SDK that run on the
8350 host and on the target, respectively:
8351
8352 .. note::
8353
8354 The following files for the most part are empty when producing an
8355 extensible SDK because this type of SDK is not constructed from
8356 packages as is the standard SDK.
8357
8358 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8359 with ``graphviz``.
8360
8361 - ``installed-package-names.txt:`` A list of installed packages by
8362 name only.
8363
8364 - ``installed-package-sizes.txt:`` A list of installed packages
8365 ordered by size.
8366
8367 - ``installed-packages.txt:`` A list of installed packages with full
8368 package filenames.
8369
8370Here is an example of ``sdk-info.txt``:
8371
8372.. code-block:: none
8373
8374 DISTRO = poky
8375 DISTRO_VERSION = 1.3+snapshot-20130327
8376 SDK_NAME = poky-glibc-i686-arm
8377 SDK_VERSION = 1.3+snapshot
8378 SDKMACHINE =
8379 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8380 BAD_RECOMMENDATIONS =
8381 SDKSIZE = 352712
8382
8383Other than ``SDKSIZE``, which is
8384the total size of the files in the SDK in Kbytes, the name-value pairs
8385are variables that might have influenced the content of the SDK. This
8386information is often useful when you are trying to determine why a
8387change in the package or file listings has occurred.
8388
8389Examining Build History Information
8390~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8391
8392You can examine build history output from the command line or from a web
8393interface.
8394
8395To see any changes that have occurred (assuming you have
8396:term:`BUILDHISTORY_COMMIT` = "1"),
8397you can simply use any Git command that allows you to view the history
8398of a repository. Here is one method:
8399::
8400
8401 $ git log -p
8402
8403You need to realize,
8404however, that this method does show changes that are not significant
8405(e.g. a package's size changing by a few bytes).
8406
8407A command-line tool called ``buildhistory-diff`` does exist, though,
8408that queries the Git repository and prints just the differences that
8409might be significant in human-readable form. Here is an example:
8410::
8411
8412 $ ~/poky/poky/scripts/buildhistory-diff . HEAD^
8413 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8414 /etc/anotherpkg.conf was added
8415 /sbin/anotherpkg was added
8416 * (installed-package-names.txt):
8417 * anotherpkg was added
8418 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8419 anotherpkg was added
8420 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8421 * PR changed from "r0" to "r1"
8422 * PV changed from "0.1.10" to "0.1.12"
8423 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8424 * PR changed from "r0" to "r1"
8425 * PV changed from "0.1.10" to "0.1.12"
8426
8427.. note::
8428
8429 The ``buildhistory-diff`` tool requires the ``GitPython``
8430 package. Be sure to install it using Pip3 as follows:
8431 ::
8432
8433 $ pip3 install GitPython --user
8434
8435
8436 Alternatively, you can install ``python3-git`` using the appropriate
8437 distribution package manager (e.g. ``apt-get``, ``dnf``, or ``zipper``).
8438
8439To see changes to the build history using a web interface, follow the
8440instruction in the ``README`` file
8441:yocto_git:`here </buildhistory-web/>`.
8442
8443Here is a sample screenshot of the interface:
8444
8445.. image:: figures/buildhistory-web.png
8446 :align: center
8447
8448Performing Automated Runtime Testing
8449====================================
8450
8451The OpenEmbedded build system makes available a series of automated
8452tests for images to verify runtime functionality. You can run these
8453tests on either QEMU or actual target hardware. Tests are written in
8454Python making use of the ``unittest`` module, and the majority of them
8455run commands on the target system over SSH. This section describes how
8456you set up the environment to use these tests, run available tests, and
8457write and add your own tests.
8458
8459For information on the test and QA infrastructure available within the
8460Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
8461section in the Yocto Project Reference Manual.
8462
8463Enabling Tests
8464--------------
8465
8466Depending on whether you are planning to run tests using QEMU or on the
8467hardware, you have to take different steps to enable the tests. See the
8468following subsections for information on how to enable both types of
8469tests.
8470
8471Enabling Runtime Tests on QEMU
8472~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8473
8474In order to run tests, you need to do the following:
8475
8476- *Set up to avoid interaction with sudo for networking:* To
8477 accomplish this, you must do one of the following:
8478
8479 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8480 commands or just for ``runqemu-ifup``. You must provide the full
8481 path as that can change if you are using multiple clones of the
8482 source repository.
8483
8484 .. note::
8485
8486 On some distributions, you also need to comment out "Defaults
8487 requiretty" in ``/etc/sudoers``.
8488
8489 - Manually configure a tap interface for your system.
8490
8491 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8492 should generate a list of tap devices. This is the option
8493 typically chosen for Autobuilder-type environments.
8494
8495 .. note::
8496
8497 - Be sure to use an absolute path when calling this script
8498 with sudo.
8499
8500 - The package recipe ``qemu-helper-native`` is required to run
8501 this script. Build the package using the following command:
8502 ::
8503
8504 $ bitbake qemu-helper-native
8505
8506- *Set the DISPLAY variable:* You need to set this variable so that
8507 you have an X server available (e.g. start ``vncserver`` for a
8508 headless machine).
8509
8510- *Be sure your host's firewall accepts incoming connections from
8511 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8512 HTTP server on a random high number port, which is used to serve
8513 files to the target. The DNF module serves
8514 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8515 That means your host's firewall must accept incoming connections from
8516 192.168.7.0/24, which is the default IP range used for tap devices by
8517 ``runqemu``.
8518
8519- *Be sure your host has the correct packages installed:* Depending
8520 your host's distribution, you need to have the following packages
8521 installed:
8522
8523 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8524
8525 - OpenSUSE: ``sysstat`` and ``iproute2``
8526
8527 - Fedora: ``sysstat`` and ``iproute``
8528
8529 - CentOS: ``sysstat`` and ``iproute``
8530
8531Once you start running the tests, the following happens:
8532
85331. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8534
85352. The image is booted under QEMU using the standard ``runqemu`` script.
8536
85373. A default timeout of 500 seconds occurs to allow for the boot process
8538 to reach the login prompt. You can change the timeout period by
8539 setting
8540 :term:`TEST_QEMUBOOT_TIMEOUT`
8541 in the ``local.conf`` file.
8542
85434. Once the boot process is reached and the login prompt appears, the
8544 tests run. The full boot log is written to
8545 ``${WORKDIR}/testimage/qemu_boot_log``.
8546
85475. Each test module loads in the order found in ``TEST_SUITES``. You can
8548 find the full output of the commands run over SSH in
8549 ``${WORKDIR}/testimgage/ssh_target_log``.
8550
85516. If no failures occur, the task running the tests ends successfully.
8552 You can find the output from the ``unittest`` in the task log at
8553 ``${WORKDIR}/temp/log.do_testimage``.
8554
8555Enabling Runtime Tests on Hardware
8556~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8557
8558The OpenEmbedded build system can run tests on real hardware, and for
8559certain devices it can also deploy the image to be tested onto the
8560device beforehand.
8561
8562For automated deployment, a "master image" is installed onto the
8563hardware once as part of setup. Then, each time tests are to be run, the
8564following occurs:
8565
85661. The master image is booted into and used to write the image to be
8567 tested to a second partition.
8568
85692. The device is then rebooted using an external script that you need to
8570 provide.
8571
85723. The device boots into the image to be tested.
8573
8574When running tests (independent of whether the image has been deployed
8575automatically or not), the device is expected to be connected to a
8576network on a pre-determined IP address. You can either use static IP
8577addresses written into the image, or set the image to use DHCP and have
8578your DHCP server on the test network assign a known IP address based on
8579the MAC address of the device.
8580
8581In order to run tests on hardware, you need to set ``TEST_TARGET`` to an
8582appropriate value. For QEMU, you do not have to change anything, the
8583default value is "qemu". For running tests on hardware, the following
8584options exist:
8585
8586- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8587 on a target system that is already running the image to be tested and
8588 is available on the network. You can use "simpleremote" in
8589 conjunction with either real hardware or an image running within a
8590 separately started QEMU or any other virtual machine manager.
8591
8592- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8593 an EFI-based machine with ``systemd-boot`` as bootloader and
8594 ``core-image-testmaster`` (or something similar) is installed. Also,
8595 your hardware under test must be in a DHCP-enabled network that gives
8596 it the same IP address for each reboot.
8597
8598 If you choose "SystemdbootTarget", there are additional requirements
8599 and considerations. See the "`Selecting
8600 SystemdbootTarget <#selecting-systemdboottarget>`__" section, which
8601 follows, for more information.
8602
8603- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8604 images and running tests on the BeagleBone "Black" or original
8605 "White" hardware. For information on how to use these tests, see the
8606 comments at the top of the BeagleBoneTarget
8607 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8608
8609- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
8610 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8611 For information on how to use these tests, see the comments at the
8612 top of the EdgeRouterTarget
8613 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8614
8615- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
8616 tests on any generic PC that boots using GRUB. For information on how
8617 to use these tests, see the comments at the top of the GrubTarget
8618 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8619
8620- *"your-target":* Create your own custom target if you want to run
8621 tests when you are deploying images and running tests on a custom
8622 machine within your BSP layer. To do this, you need to add a Python
8623 unit that defines the target class under ``lib/oeqa/controllers/``
8624 within your layer. You must also provide an empty ``__init__.py``.
8625 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8626
8627Selecting SystemdbootTarget
8628~~~~~~~~~~~~~~~~~~~~~~~~~~~
8629
8630If you did not set ``TEST_TARGET`` to "SystemdbootTarget", then you do
8631not need any information in this section. You can skip down to the
8632"`Running Tests <#qemu-image-running-tests>`__" section.
8633
8634If you did set ``TEST_TARGET`` to "SystemdbootTarget", you also need to
8635perform a one-time setup of your master image by doing the following:
8636
86371. *Set EFI_PROVIDER:* Be sure that ``EFI_PROVIDER`` is as follows:
8638 ::
8639
8640 EFI_PROVIDER = "systemd-boot"
8641
86422. *Build the master image:* Build the ``core-image-testmaster`` image.
8643 The ``core-image-testmaster`` recipe is provided as an example for a
8644 "master" image and you can customize the image recipe as you would
8645 any other recipe.
8646
8647 Here are the image recipe requirements:
8648
8649 - Inherits ``core-image`` so that kernel modules are installed.
8650
8651 - Installs normal linux utilities not busybox ones (e.g. ``bash``,
8652 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8653
8654 - Uses a custom Initial RAM Disk (initramfs) image with a custom
8655 installer. A normal image that you can install usually creates a
8656 single rootfs partition. This image uses another installer that
8657 creates a specific partition layout. Not all Board Support
8658 Packages (BSPs) can use an installer. For such cases, you need to
8659 manually create the following partition layout on the target:
8660
8661 - First partition mounted under ``/boot``, labeled "boot".
8662
8663 - The main rootfs partition where this image gets installed,
8664 which is mounted under ``/``.
8665
8666 - Another partition labeled "testrootfs" where test images get
8667 deployed.
8668
86693. *Install image:* Install the image that you just built on the target
8670 system.
8671
8672The final thing you need to do when setting ``TEST_TARGET`` to
8673"SystemdbootTarget" is to set up the test image:
8674
86751. *Set up your local.conf file:* Make sure you have the following
8676 statements in your ``local.conf`` file:
8677 ::
8678
8679 IMAGE_FSTYPES += "tar.gz"
8680 INHERIT += "testimage"
8681 TEST_TARGET = "SystemdbootTarget"
8682 TEST_TARGET_IP = "192.168.2.3"
8683
86842. *Build your test image:* Use BitBake to build the image:
8685 ::
8686
8687 $ bitbake core-image-sato
8688
8689Power Control
8690~~~~~~~~~~~~~
8691
8692For most hardware targets other than "simpleremote", you can control
8693power:
8694
8695- You can use ``TEST_POWERCONTROL_CMD`` together with
8696 ``TEST_POWERCONTROL_EXTRA_ARGS`` as a command that runs on the host
8697 and does power cycling. The test code passes one argument to that
8698 command: off, on or cycle (off then on). Here is an example that
8699 could appear in your ``local.conf`` file:
8700 ::
8701
8702 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8703
8704 In this example, the expect
8705 script does the following:
8706
8707 .. code-block:: shell
8708
8709 ssh test@10.11.12.1 "pyctl nuc1 arg"
8710
8711 It then runs a Python script that controls power for a label called
8712 ``nuc1``.
8713
8714 .. note::
8715
8716 You need to customize ``TEST_POWERCONTROL_CMD`` and
8717 ``TEST_POWERCONTROL_EXTRA_ARGS`` for your own setup. The one requirement
8718 is that it accepts "on", "off", and "cycle" as the last argument.
8719
8720- When no command is defined, it connects to the device over SSH and
8721 uses the classic reboot command to reboot the device. Classic reboot
8722 is fine as long as the machine actually reboots (i.e. the SSH test
8723 has not failed). It is useful for scenarios where you have a simple
8724 setup, typically with a single board, and where some manual
8725 interaction is okay from time to time.
8726
8727If you have no hardware to automatically perform power control but still
8728wish to experiment with automated hardware testing, you can use the
8729``dialog-power-control`` script that shows a dialog prompting you to perform
8730the required power action. This script requires either KDialog or Zenity
8731to be installed. To use this script, set the
8732:term:`TEST_POWERCONTROL_CMD`
8733variable as follows:
8734::
8735
8736 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8737
8738Serial Console Connection
8739~~~~~~~~~~~~~~~~~~~~~~~~~
8740
8741For test target classes requiring a serial console to interact with the
8742bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8743you need to specify a command to use to connect to the serial console of
8744the target machine by using the
8745:term:`TEST_SERIALCONTROL_CMD`
8746variable and optionally the
8747:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8748variable.
8749
8750These cases could be a serial terminal program if the machine is
8751connected to a local serial port, or a ``telnet`` or ``ssh`` command
8752connecting to a remote console server. Regardless of the case, the
8753command simply needs to connect to the serial console and forward that
8754connection to standard input and output as any normal terminal program
8755does. For example, to use the picocom terminal program on serial device
8756``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows:
8757::
8758
8759 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8760
8761For local
8762devices where the serial port device disappears when the device reboots,
8763an additional "serdevtry" wrapper script is provided. To use this
8764wrapper, simply prefix the terminal command with
8765``${COREBASE}/scripts/contrib/serdevtry``:
8766::
8767
8768 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8769
8770Running Tests
8771-------------
8772
8773You can start the tests automatically or manually:
8774
8775- *Automatically running tests:* To run the tests automatically after
8776 the OpenEmbedded build system successfully creates an image, first
8777 set the
8778 :term:`TESTIMAGE_AUTO`
8779 variable to "1" in your ``local.conf`` file in the
8780 :term:`Build Directory`:
8781 ::
8782
8783 TESTIMAGE_AUTO = "1"
8784
8785 Next, build your image. If the image successfully builds, the
8786 tests run:
8787 ::
8788
8789 bitbake core-image-sato
8790
8791- *Manually running tests:* To manually run the tests, first globally
8792 inherit the
8793 :ref:`testimage <ref-classes-testimage*>` class
8794 by editing your ``local.conf`` file:
8795 ::
8796
8797 INHERIT += "testimage"
8798
8799 Next, use BitBake to run the tests:
8800 ::
8801
8802 bitbake -c testimage image
8803
8804All test files reside in ``meta/lib/oeqa/runtime`` in the
8805:term:`Source Directory`. A test name maps
8806directly to a Python module. Each test module may contain a number of
8807individual tests. Tests are usually grouped together by the area tested
8808(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
8809
8810You can add tests to any layer provided you place them in the proper
8811area and you extend :term:`BBPATH` in
8812the ``local.conf`` file as normal. Be sure that tests reside in
8813``layer/lib/oeqa/runtime``.
8814
8815.. note::
8816
8817 Be sure that module names do not collide with module names used in
8818 the default set of test modules in ``meta/lib/oeqa/runtime``.
8819
8820You can change the set of tests run by appending or overriding
8821:term:`TEST_SUITES` variable in
8822``local.conf``. Each name in ``TEST_SUITES`` represents a required test
8823for the image. Test modules named within ``TEST_SUITES`` cannot be
8824skipped even if a test is not suitable for an image (e.g. running the
8825RPM tests on an image without ``rpm``). Appending "auto" to
8826``TEST_SUITES`` causes the build system to try to run all tests that are
8827suitable for the image (i.e. each test module may elect to skip itself).
8828
8829The order you list tests in ``TEST_SUITES`` is important and influences
8830test dependencies. Consequently, tests that depend on other tests should
8831be added after the test on which they depend. For example, since the
8832``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8833"ping" in the list. The test class provides no re-ordering or dependency
8834handling.
8835
8836.. note::
8837
8838 Each module can have multiple classes with multiple test methods.
8839 And, Python ``unittest`` rules apply.
8840
8841Here are some things to keep in mind when running tests:
8842
8843- The default tests for the image are defined as:
8844 ::
8845
8846 DEFAULT_TEST_SUITES_pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
8847
8848- Add your own test to the list of the by using the following:
8849 ::
8850
8851 TEST_SUITES_append = " mytest"
8852
8853- Run a specific list of tests as follows:
8854 ::
8855
8856 TEST_SUITES = "test1 test2 test3"
8857
8858 Remember, order is important. Be sure to place a test that is
8859 dependent on another test later in the order.
8860
8861Exporting Tests
8862---------------
8863
8864You can export tests so that they can run independently of the build
8865system. Exporting tests is required if you want to be able to hand the
8866test execution off to a scheduler. You can only export tests that are
8867defined in :term:`TEST_SUITES`.
8868
8869If your image is already built, make sure the following are set in your
8870``local.conf`` file:
8871::
8872
8873 INHERIT += "testexport"
8874 TEST_TARGET_IP = "IP-address-for-the-test-target"
8875 TEST_SERVER_IP = "IP-address-for-the-test-server"
8876
8877You can then export the tests with the
8878following BitBake command form:
8879::
8880
8881 $ bitbake image -c testexport
8882
8883Exporting the tests places them in the
8884:term:`Build Directory` in
8885``tmp/testexport/``\ image, which is controlled by the
8886``TEST_EXPORT_DIR`` variable.
8887
8888You can now run the tests outside of the build environment:
8889::
8890
8891 $ cd tmp/testexport/image
8892 $ ./runexported.py testdata.json
8893
8894Here is a complete example that shows IP addresses and uses the
8895``core-image-sato`` image:
8896::
8897
8898 INHERIT += "testexport"
8899 TEST_TARGET_IP = "192.168.7.2"
8900 TEST_SERVER_IP = "192.168.7.1"
8901
8902Use BitBake to export the tests:
8903::
8904
8905 $ bitbake core-image-sato -c testexport
8906
8907Run the tests outside of
8908the build environment using the following:
8909::
8910
8911 $ cd tmp/testexport/core-image-sato
8912 $ ./runexported.py testdata.json
8913
8914Writing New Tests
8915-----------------
8916
8917As mentioned previously, all new test files need to be in the proper
8918place for the build system to find them. New tests for additional
8919functionality outside of the core should be added to the layer that adds
8920the functionality, in ``layer/lib/oeqa/runtime`` (as long as
8921:term:`BBPATH` is extended in the
8922layer's ``layer.conf`` file as normal). Just remember the following:
8923
8924- Filenames need to map directly to test (module) names.
8925
8926- Do not use module names that collide with existing core tests.
8927
8928- Minimally, an empty ``__init__.py`` file must exist in the runtime
8929 directory.
8930
8931To create a new test, start by copying an existing module (e.g.
8932``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
8933code from ``meta/lib/oeqa/utils``, which are helper classes.
8934
8935.. note::
8936
8937 Structure shell commands such that you rely on them and they return a
8938 single code for success. Be aware that sometimes you will need to
8939 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
8940
8941You will notice that all test classes inherit ``oeRuntimeTest``, which
8942is found in ``meta/lib/oetest.py``. This base class offers some helper
8943attributes, which are described in the following sections:
8944
8945Class Methods
8946~~~~~~~~~~~~~
8947
8948Class methods are as follows:
8949
8950- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
8951 package list of the image, which is based on the manifest file that
8952 is generated during the ``do_rootfs`` task.
8953
8954- *hasFeature(feature):* Returns "True" if the feature is in
8955 :term:`IMAGE_FEATURES` or
8956 :term:`DISTRO_FEATURES`.
8957
8958Class Attributes
8959~~~~~~~~~~~~~~~~
8960
8961Class attributes are as follows:
8962
8963- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
8964 Otherwise, ``pscmd`` equals "ps" (busybox).
8965
8966- *tc:* The called test context, which gives access to the
8967 following attributes:
8968
8969 - *d:* The BitBake datastore, which allows you to use stuff such
8970 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
8971
8972 - *testslist and testsrequired:* Used internally. The tests
8973 do not need these.
8974
8975 - *filesdir:* The absolute path to
8976 ``meta/lib/oeqa/runtime/files``, which contains helper files for
8977 tests meant for copying on the target such as small files written
8978 in C for compilation.
8979
8980 - *target:* The target controller object used to deploy and
8981 start an image on a particular target (e.g. Qemu, SimpleRemote,
8982 and SystemdbootTarget). Tests usually use the following:
8983
8984 - *ip:* The target's IP address.
8985
8986 - *server_ip:* The host's IP address, which is usually used
8987 by the DNF test suite.
8988
8989 - *run(cmd, timeout=None):* The single, most used method.
8990 This command is a wrapper for: ``ssh root@host "cmd"``. The
8991 command returns a tuple: (status, output), which are what their
8992 names imply - the return code of "cmd" and whatever output it
8993 produces. The optional timeout argument represents the number
8994 of seconds the test should wait for "cmd" to return. If the
8995 argument is "None", the test uses the default instance's
8996 timeout period, which is 300 seconds. If the argument is "0",
8997 the test runs until the command returns.
8998
8999 - *copy_to(localpath, remotepath):*
9000 ``scp localpath root@ip:remotepath``.
9001
9002 - *copy_from(remotepath, localpath):*
9003 ``scp root@host:remotepath localpath``.
9004
9005Instance Attributes
9006~~~~~~~~~~~~~~~~~~~
9007
9008A single instance attribute exists, which is ``target``. The ``target``
9009instance attribute is identical to the class attribute of the same name,
9010which is described in the previous section. This attribute exists as
9011both an instance and class attribute so tests can use
9012``self.target.run(cmd)`` in instance methods instead of
9013``oeRuntimeTest.tc.target.run(cmd)``.
9014
9015Installing Packages in the DUT Without the Package Manager
9016----------------------------------------------------------
9017
9018When a test requires a package built by BitBake, it is possible to
9019install that package. Installing the package does not require a package
9020manager be installed in the device under test (DUT). It does, however,
9021require an SSH connection and the target must be using the
9022``sshcontrol`` class.
9023
9024.. note::
9025
9026 This method uses ``scp`` to copy files from the host to the target, which
9027 causes permissions and special attributes to be lost.
9028
9029A JSON file is used to define the packages needed by a test. This file
9030must be in the same path as the file used to define the tests.
9031Furthermore, the filename must map directly to the test module name with
9032a ``.json`` extension.
9033
9034The JSON file must include an object with the test name as keys of an
9035object or an array. This object (or array of objects) uses the following
9036data:
9037
9038- "pkg" - A mandatory string that is the name of the package to be
9039 installed.
9040
9041- "rm" - An optional boolean, which defaults to "false", that specifies
9042 to remove the package after the test.
9043
9044- "extract" - An optional boolean, which defaults to "false", that
9045 specifies if the package must be extracted from the package format.
9046 When set to "true", the package is not automatically installed into
9047 the DUT.
9048
9049Following is an example JSON file that handles test "foo" installing
9050package "bar" and test "foobar" installing packages "foo" and "bar".
9051Once the test is complete, the packages are removed from the DUT.
9052::
9053
9054 {
9055 "foo": {
9056 "pkg": "bar"
9057 },
9058 "foobar": [
9059 {
9060 "pkg": "foo",
9061 "rm": true
9062 },
9063 {
9064 "pkg": "bar",
9065 "rm": true
9066 }
9067 ]
9068 }
9069
9070Debugging Tools and Techniques
9071==============================
9072
9073The exact method for debugging build failures depends on the nature of
9074the problem and on the system's area from which the bug originates.
9075Standard debugging practices such as comparison against the last known
9076working version with examination of the changes and the re-application
9077of steps to identify the one causing the problem are valid for the Yocto
9078Project just as they are for any other system. Even though it is
9079impossible to detail every possible potential failure, this section
9080provides some general tips to aid in debugging given a variety of
9081situations.
9082
9083.. note::
9084
9085 A useful feature for debugging is the error reporting tool.
9086 Configuring the Yocto Project to use this tool causes the
9087 OpenEmbedded build system to produce error reporting commands as part
9088 of the console output. You can enter the commands after the build
9089 completes to log error information into a common database, that can
9090 help you figure out what might be going wrong. For information on how
9091 to enable and use this feature, see the
9092 ":ref:`dev-manual/common-tasks:using the error reporting tool`"
9093 section.
9094
9095The following list shows the debugging topics in the remainder of this
9096section:
9097
9098- "`Viewing Logs from Failed
9099 Tasks <#dev-debugging-viewing-logs-from-failed-tasks>`__" describes
9100 how to find and view logs from tasks that failed during the build
9101 process.
9102
9103- "`Viewing Variable
9104 Values <#dev-debugging-viewing-variable-values>`__" describes how to
9105 use the BitBake ``-e`` option to examine variable values after a
9106 recipe has been parsed.
9107
9108- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
9109 describes how to use the ``oe-pkgdata-util`` utility to query
9110 :term:`PKGDATA_DIR` and
9111 display package-related information for built packages.
9112
9113- "`Viewing Dependencies Between Recipes and
9114 Tasks <#dev-viewing-dependencies-between-recipes-and-tasks>`__"
9115 describes how to use the BitBake ``-g`` option to display recipe
9116 dependency information used during the build.
9117
9118- "`Viewing Task Variable
9119 Dependencies <#dev-viewing-task-variable-dependencies>`__" describes
9120 how to use the ``bitbake-dumpsig`` command in conjunction with key
9121 subdirectories in the
9122 :term:`Build Directory` to determine
9123 variable dependencies.
9124
9125- "`Running Specific Tasks <#dev-debugging-taskrunning>`__" describes
9126 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
9127 to run specific tasks in the build chain. It can be useful to run
9128 tasks "out-of-order" when trying isolate build issues.
9129
9130- "`General BitBake Problems <#dev-debugging-bitbake>`__" describes how
9131 to use BitBake's ``-D`` debug output option to reveal more about what
9132 BitBake is doing during the build.
9133
9134- "`Building with No Dependencies <#dev-debugging-buildfile>`__"
9135 describes how to use the BitBake ``-b`` option to build a recipe
9136 while ignoring dependencies.
9137
9138- "`Recipe Logging Mechanisms <#recipe-logging-mechanisms>`__"
9139 describes how to use the many recipe logging functions to produce
9140 debugging output and report errors and warnings.
9141
9142- "`Debugging Parallel Make Races <#debugging-parallel-make-races>`__"
9143 describes how to debug situations where the build consists of several
9144 parts that are run simultaneously and when the output or result of
9145 one part is not ready for use with a different part of the build that
9146 depends on that output.
9147
9148- "`Debugging With the GNU Project Debugger (GDB)
9149 Remotely <#platdev-gdb-remotedebug>`__" describes how to use GDB to
9150 allow you to examine running programs, which can help you fix
9151 problems.
9152
9153- "`Debugging with the GNU Project Debugger (GDB) on the
9154 Target <#debugging-with-the-gnu-project-debugger-gdb-on-the-target>`__"
9155 describes how to use GDB directly on target hardware for debugging.
9156
9157- "`Other Debugging Tips <#dev-other-debugging-others>`__" describes
9158 miscellaneous debugging tips that can be useful.
9159
9160Viewing Logs from Failed Tasks
9161------------------------------
9162
9163You can find the log for a task in the file
9164``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
9165For example, the log for the
9166:ref:`ref-tasks-compile` task of the
9167QEMU minimal image for the x86 machine (``qemux86``) might be in
9168``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
9169To see the commands :term:`BitBake` ran
9170to generate a log, look at the corresponding ``run.do_``\ `taskname` file
9171in the same directory.
9172
9173``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
9174links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
9175``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
9176when it ran. The symlinks always point to the files corresponding to the
9177most recent run.
9178
9179Viewing Variable Values
9180-----------------------
9181
9182Sometimes you need to know the value of a variable as a result of
9183BitBake's parsing step. This could be because some unexpected behavior
9184occurred in your project. Perhaps an attempt to :ref:`modify a variable
9185<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
9186variables>` did not work out as expected.
9187
9188BitBake's ``-e`` option is used to display variable values after
9189parsing. The following command displays the variable values after the
9190configuration files (i.e. ``local.conf``, ``bblayers.conf``,
9191``bitbake.conf`` and so forth) have been parsed:
9192::
9193
9194 $ bitbake -e
9195
9196The following command displays variable values after a specific recipe has
9197been parsed. The variables include those from the configuration as well:
9198::
9199
9200 $ bitbake -e recipename
9201
9202.. note::
9203
9204 Each recipe has its own private set of variables (datastore).
9205 Internally, after parsing the configuration, a copy of the resulting
9206 datastore is made prior to parsing each recipe. This copying implies
9207 that variables set in one recipe will not be visible to other
9208 recipes.
9209
9210 Likewise, each task within a recipe gets a private datastore based on
9211 the recipe datastore, which means that variables set within one task
9212 will not be visible to other tasks.
9213
9214In the output of ``bitbake -e``, each variable is preceded by a
9215description of how the variable got its value, including temporary
9216values that were later overridden. This description also includes
9217variable flags (varflags) set on the variable. The output can be very
9218helpful during debugging.
9219
9220Variables that are exported to the environment are preceded by
9221``export`` in the output of ``bitbake -e``. See the following example:
9222::
9223
9224 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9225
9226In addition to variable values, the output of the ``bitbake -e`` and
9227``bitbake -e`` recipe commands includes the following information:
9228
9229- The output starts with a tree listing all configuration files and
9230 classes included globally, recursively listing the files they include
9231 or inherit in turn. Much of the behavior of the OpenEmbedded build
9232 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
9233 implemented in the
9234 :ref:`base <ref-classes-base>` class and the
9235 classes it inherits, rather than being built into BitBake itself.
9236
9237- After the variable values, all functions appear in the output. For
9238 shell functions, variables referenced within the function body are
9239 expanded. If a function has been modified using overrides or using
9240 override-style operators like ``_append`` and ``_prepend``, then the
9241 final assembled function body appears in the output.
9242
9243Viewing Package Information with ``oe-pkgdata-util``
9244----------------------------------------------------
9245
9246You can use the ``oe-pkgdata-util`` command-line utility to query
9247:term:`PKGDATA_DIR` and display
9248various package-related information. When you use the utility, you must
9249use it to view information on packages that have already been built.
9250
9251Following are a few of the available ``oe-pkgdata-util`` subcommands.
9252
9253.. note::
9254
9255 You can use the standard \* and ? globbing wildcards as part of
9256 package names and paths.
9257
9258- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9259 that have been built, optionally limiting the match to packages that
9260 match pattern.
9261
9262- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9263 files and directories contained in the given packages.
9264
9265 .. note::
9266
9267 A different way to view the contents of a package is to look at
9268 the
9269 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9270 directory of the recipe that generates the package. This directory
9271 is created by the
9272 :ref:`ref-tasks-package` task
9273 and has one subdirectory for each package the recipe generates,
9274 which contains the files stored in that package.
9275
9276 If you want to inspect the ``${WORKDIR}/packages-split``
9277 directory, make sure that
9278 :ref:`rm_work <ref-classes-rm-work>` is not
9279 enabled when you build the recipe.
9280
9281- ``oe-pkgdata-util find-path path ...``: Lists the names of
9282 the packages that contain the given paths. For example, the following
9283 tells us that ``/usr/share/man/man1/make.1`` is contained in the
9284 ``make-doc`` package:
9285 ::
9286
9287 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
9288 make-doc: /usr/share/man/man1/make.1
9289
9290- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9291 of the recipes that produce the given packages.
9292
9293For more information on the ``oe-pkgdata-util`` command, use the help
9294facility:
9295::
9296
9297 $ oe-pkgdata-util --help
9298 $ oe-pkgdata-util subcommand --help
9299
9300Viewing Dependencies Between Recipes and Tasks
9301----------------------------------------------
9302
9303Sometimes it can be hard to see why BitBake wants to build other recipes
9304before the one you have specified. Dependency information can help you
9305understand why a recipe is built.
9306
9307To generate dependency information for a recipe, run the following
9308command:
9309::
9310
9311 $ bitbake -g recipename
9312
9313This command writes the following files in the current directory:
9314
9315- ``pn-buildlist``: A list of recipes/targets involved in building
9316 `recipename`. "Involved" here means that at least one task from the
9317 recipe needs to run when building `recipename` from scratch. Targets
9318 that are in
9319 :term:`ASSUME_PROVIDED`
9320 are not listed.
9321
9322- ``task-depends.dot``: A graph showing dependencies between tasks.
9323
9324The graphs are in
9325`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__
9326format and can be converted to images (e.g. using the ``dot`` tool from
9327`Graphviz <https://www.graphviz.org/>`__).
9328
9329.. note::
9330
9331 - DOT files use a plain text format. The graphs generated using the
9332 ``bitbake -g`` command are often so large as to be difficult to
9333 read without special pruning (e.g. with Bitbake's ``-I`` option)
9334 and processing. Despite the form and size of the graphs, the
9335 corresponding ``.dot`` files can still be possible to read and
9336 provide useful information.
9337
9338 As an example, the ``task-depends.dot`` file contains lines such
9339 as the following:
9340 ::
9341
9342 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9343
9344 The above example line reveals that the
9345 :ref:`ref-tasks-configure`
9346 task in ``libxslt`` depends on the
9347 :ref:`ref-tasks-populate_sysroot`
9348 task in ``libxml2``, which is a normal
9349 :term:`DEPENDS` dependency
9350 between the two recipes.
9351
9352 - For an example of how ``.dot`` files can be processed, see the
9353 ``scripts/contrib/graph-tool`` Python script, which finds and
9354 displays paths between graph nodes.
9355
9356You can use a different method to view dependency information by using
9357the following command:
9358::
9359
9360 $ bitbake -g -u taskexp recipename
9361
9362This command
9363displays a GUI window from which you can view build-time and runtime
9364dependencies for the recipes involved in building recipename.
9365
9366Viewing Task Variable Dependencies
9367----------------------------------
9368
9369As mentioned in the
9370":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9371User Manual, BitBake tries to automatically determine what variables a
9372task depends on so that it can rerun the task if any values of the
9373variables change. This determination is usually reliable. However, if
9374you do things like construct variable names at runtime, then you might
9375have to manually declare dependencies on those variables using
9376``vardeps`` as described in the
9377":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9378User Manual.
9379
9380If you are unsure whether a variable dependency is being picked up
9381automatically for a given task, you can list the variable dependencies
9382BitBake has determined by doing the following:
9383
93841. Build the recipe containing the task:
9385::
9386
9387 $ bitbake recipename
9388
93892. Inside the :term:`STAMPS_DIR`
9390 directory, find the signature data (``sigdata``) file that
9391 corresponds to the task. The ``sigdata`` files contain a pickled
9392 Python database of all the metadata that went into creating the input
9393 checksum for the task. As an example, for the
9394 :ref:`ref-tasks-fetch` task of the
9395 ``db`` recipe, the ``sigdata`` file might be found in the following
9396 location:
9397 ::
9398
9399 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9400
9401 For tasks that are accelerated through the shared state
9402 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
9403 additional ``siginfo`` file is written into
9404 :term:`SSTATE_DIR` along with
9405 the cached task output. The ``siginfo`` files contain exactly the
9406 same information as ``sigdata`` files.
9407
94083. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
9409 is an example:
9410 ::
9411
9412 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9413
9414 In the output of the above command, you will find a line like the
9415 following, which lists all the (inferred) variable dependencies for
9416 the task. This list also includes indirect dependencies from
9417 variables depending on other variables, recursively.
9418 ::
9419
9420 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9421
9422 .. note::
9423
9424 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9425 These functions in turn depend on the variables they reference.
9426
9427 The output of ``bitbake-dumpsig`` also includes the value each
9428 variable had, a list of dependencies for each variable, and
9429 :term:`bitbake:BB_HASHBASE_WHITELIST`
9430 information.
9431
9432There is also a ``bitbake-diffsigs`` command for comparing two
9433``siginfo`` or ``sigdata`` files. This command can be helpful when
9434trying to figure out what changed between two versions of a task. If you
9435call ``bitbake-diffsigs`` with just one file, the command behaves like
9436``bitbake-dumpsig``.
9437
9438You can also use BitBake to dump out the signature construction
9439information without executing tasks by using either of the following
9440BitBake command-line options:
9441::
9442
9443 ‐‐dump-signatures=SIGNATURE_HANDLER
9444 -S SIGNATURE_HANDLER
9445
9446
9447.. note::
9448
9449 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9450 dump only the signature or compare the dumped signature with the cached one,
9451 respectively.
9452
9453Using BitBake with either of these options causes BitBake to dump out
9454``sigdata`` files in the ``stamps`` directory for every task it would
9455have executed instead of building the specified target package.
9456
9457Viewing Metadata Used to Create the Input Signature of a Shared State Task
9458--------------------------------------------------------------------------
9459
9460Seeing what metadata went into creating the input signature of a shared
9461state (sstate) task can be a useful debugging aid. This information is
9462available in signature information (``siginfo``) files in
9463:term:`SSTATE_DIR`. For
9464information on how to view and interpret information in ``siginfo``
9465files, see the "`Viewing Task Variable
9466Dependencies <#dev-viewing-task-variable-dependencies>`__" section.
9467
9468For conceptual information on shared state, see the
9469":ref:`overview-manual/concepts:shared state`"
9470section in the Yocto Project Overview and Concepts Manual.
9471
9472Invalidating Shared State to Force a Task to Run
9473------------------------------------------------
9474
9475The OpenEmbedded build system uses
9476:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
9477:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
9478rebuilding tasks. Collectively, this scheme is known as "shared state
9479code".
9480
9481As with all schemes, this one has some drawbacks. It is possible that
9482you could make implicit changes to your code that the checksum
9483calculations do not take into account. These implicit changes affect a
9484task's output but do not trigger the shared state code into rebuilding a
9485recipe. Consider an example during which a tool changes its output.
9486Assume that the output of ``rpmdeps`` changes. The result of the change
9487should be that all the ``package`` and ``package_write_rpm`` shared
9488state cache items become invalid. However, because the change to the
9489output is external to the code and therefore implicit, the associated
9490shared state cache items do not become invalidated. In this case, the
9491build process uses the cached items rather than running the task again.
9492Obviously, these types of implicit changes can cause problems.
9493
9494To avoid these problems during the build, you need to understand the
9495effects of any changes you make. Realize that changes you make directly
9496to a function are automatically factored into the checksum calculation.
9497Thus, these explicit changes invalidate the associated area of shared
9498state cache. However, you need to be aware of any implicit changes that
9499are not obvious changes to the code and could affect the output of a
9500given task.
9501
9502When you identify an implicit change, you can easily take steps to
9503invalidate the cache and force the tasks to run. The steps you can take
9504are as simple as changing a function's comments in the source code. For
9505example, to invalidate package shared state files, change the comment
9506statements of
9507:ref:`ref-tasks-package` or the
9508comments of one of the functions it calls. Even though the change is
9509purely cosmetic, it causes the checksum to be recalculated and forces
9510the build system to run the task again.
9511
9512.. note::
9513
9514 For an example of a commit that makes a cosmetic change to invalidate
9515 shared state, see this
9516 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
9517
9518Running Specific Tasks
9519----------------------
9520
9521Any given recipe consists of a set of tasks. The standard BitBake
9522behavior in most cases is: ``do_fetch``, ``do_unpack``, ``do_patch``,
9523``do_configure``, ``do_compile``, ``do_install``, ``do_package``,
9524``do_package_write_*``, and ``do_build``. The default task is
9525``do_build`` and any tasks on which it depends build first. Some tasks,
9526such as ``do_devshell``, are not part of the default build chain. If you
9527wish to run a task that is not part of the default build chain, you can
9528use the ``-c`` option in BitBake. Here is an example:
9529::
9530
9531 $ bitbake matchbox-desktop -c devshell
9532
9533The ``-c`` option respects task dependencies, which means that all other
9534tasks (including tasks from other recipes) that the specified task
9535depends on will be run before the task. Even when you manually specify a
9536task to run with ``-c``, BitBake will only run the task if it considers
9537it "out of date". See the
9538":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
9539section in the Yocto Project Overview and Concepts Manual for how
9540BitBake determines whether a task is "out of date".
9541
9542If you want to force an up-to-date task to be rerun (e.g. because you
9543made manual modifications to the recipe's
9544:term:`WORKDIR` that you want to try
9545out), then you can use the ``-f`` option.
9546
9547.. note::
9548
9549 The reason ``-f`` is never required when running the
9550 :ref:`ref-tasks-devshell` task is because the
9551 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
9552 variable flag is already set for the task.
9553
9554The following example shows one way you can use the ``-f`` option:
9555::
9556
9557 $ bitbake matchbox-desktop
9558 .
9559 .
9560 make some changes to the source code in the work directory
9561 .
9562 .
9563 $ bitbake matchbox-desktop -c compile -f
9564 $ bitbake matchbox-desktop
9565
9566This sequence first builds and then recompiles ``matchbox-desktop``. The
9567last command reruns all tasks (basically the packaging tasks) after the
9568compile. BitBake recognizes that the ``do_compile`` task was rerun and
9569therefore understands that the other tasks also need to be run again.
9570
9571Another, shorter way to rerun a task and all
9572:ref:`ref-manual/tasks:normal recipe build tasks`
9573that depend on it is to use the ``-C`` option.
9574
9575.. note::
9576
9577 This option is upper-cased and is separate from the ``-c``
9578 option, which is lower-cased.
9579
9580Using this option invalidates the given task and then runs the
9581:ref:`ref-tasks-build` task, which is
9582the default task if no task is given, and the tasks on which it depends.
9583You could replace the final two commands in the previous example with
9584the following single command:
9585::
9586
9587 $ bitbake matchbox-desktop -C compile
9588
9589Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9590the input checksum of the specified task. This tainting indirectly
9591causes the task and its dependent tasks to be rerun through the normal
9592task dependency mechanisms.
9593
9594.. note::
9595
9596 BitBake explicitly keeps track of which tasks have been tainted in
9597 this fashion, and will print warnings such as the following for
9598 builds involving such tasks:
9599
9600 .. code-block:: none
9601
9602 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9603
9604
9605 The purpose of the warning is to let you know that the work directory
9606 and build output might not be in the clean state they would be in for
9607 a "normal" build, depending on what actions you took. To get rid of
9608 such warnings, you can remove the work directory and rebuild the
9609 recipe, as follows:
9610 ::
9611
9612 $ bitbake matchbox-desktop -c clean
9613 $ bitbake matchbox-desktop
9614
9615
9616You can view a list of tasks in a given package by running the
9617``do_listtasks`` task as follows:
9618::
9619
9620 $ bitbake matchbox-desktop -c listtasks
9621
9622The results appear as output to the console and are also in
9623the file ``${WORKDIR}/temp/log.do_listtasks``.
9624
9625General BitBake Problems
9626------------------------
9627
9628You can see debug output from BitBake by using the ``-D`` option. The
9629debug output gives more information about what BitBake is doing and the
9630reason behind it. Each ``-D`` option you use increases the logging
9631level. The most common usage is ``-DDD``.
9632
9633The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
9634chose a certain version of a package or why BitBake picked a certain
9635provider. This command could also help you in a situation where you
9636think BitBake did something unexpected.
9637
9638Building with No Dependencies
9639-----------------------------
9640
9641To build a specific recipe (``.bb`` file), you can use the following
9642command form:
9643::
9644
9645 $ bitbake -b somepath/somerecipe.bb
9646
9647This command form does
9648not check for dependencies. Consequently, you should use it only when
9649you know existing dependencies have been met.
9650
9651.. note::
9652
9653 You can also specify fragments of the filename. In this case, BitBake
9654 checks for a unique match.
9655
9656Recipe Logging Mechanisms
9657-------------------------
9658
9659The Yocto Project provides several logging functions for producing
9660debugging output and reporting errors and warnings. For Python
9661functions, the following logging functions exist. All of these functions
9662log to ``${T}/log.do_``\ `task`, and can also log to standard output
9663(stdout) with the right settings:
9664
9665- ``bb.plain(msg)``: Writes msg as is to the log while also
9666 logging to stdout.
9667
9668- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9669 stdout if BitBake is called with "-v".
9670
9671- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9672 log. Also logs to stdout if the log level is greater than or equal to
9673 level. See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option
9674 in the BitBake User Manual for more information.
9675
9676- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9677 logging to stdout.
9678
9679- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9680 logging to standard out (stdout).
9681
9682 .. note::
9683
9684 Calling this function does not cause the task to fail.
9685
9686- ``bb.fatal(``\ msg\ ``)``: This logging function is similar to
9687 ``bb.error(``\ msg\ ``)`` but also causes the calling task to fail.
9688
9689 .. note::
9690
9691 ``bb.fatal()`` raises an exception, which means you do not need to put a
9692 "return" statement after the function.
9693
9694The same logging functions are also available in shell functions, under
9695the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9696and ``bbfatal``. The
9697:ref:`logging <ref-classes-logging>` class
9698implements these functions. See that class in the ``meta/classes``
9699folder of the :term:`Source Directory` for information.
9700
9701Logging With Python
9702~~~~~~~~~~~~~~~~~~~
9703
9704When creating recipes using Python and inserting code that handles build
9705logs, keep in mind the goal is to have informative logs while keeping
9706the console as "silent" as possible. Also, if you want status messages
9707in the log, use the "debug" loglevel.
9708
9709Following is an example written in Python. The code handles logging for
9710a function that determines the number of tasks needed to be run. See the
9711":ref:`ref-tasks-listtasks`"
9712section for additional information:
9713::
9714
9715 python do_listtasks() {
9716 bb.debug(2, "Starting to figure out the task list")
9717 if noteworthy_condition:
9718 bb.note("There are 47 tasks to run")
9719 bb.debug(2, "Got to point xyz")
9720 if warning_trigger:
9721 bb.warn("Detected warning_trigger, this might be a problem later.")
9722 if recoverable_error:
9723 bb.error("Hit recoverable_error, you really need to fix this!")
9724 if fatal_error:
9725 bb.fatal("fatal_error detected, unable to print the task list")
9726 bb.plain("The tasks present are abc")
9727 bb.debug(2, "Finished figuring out the tasklist")
9728 }
9729
9730Logging With Bash
9731~~~~~~~~~~~~~~~~~
9732
9733When creating recipes using Bash and inserting code that handles build
9734logs, you have the same goals - informative with minimal console output.
9735The syntax you use for recipes written in Bash is similar to that of
9736recipes written in Python described in the previous section.
9737
9738Following is an example written in Bash. The code logs the progress of
9739the ``do_my_function`` function.
9740::
9741
9742 do_my_function() {
9743 bbdebug 2 "Running do_my_function"
9744 if [ exceptional_condition ]; then
9745 bbnote "Hit exceptional_condition"
9746 fi
9747 bbdebug 2 "Got to point xyz"
9748 if [ warning_trigger ]; then
9749 bbwarn "Detected warning_trigger, this might cause a problem later."
9750 fi
9751 if [ recoverable_error ]; then
9752 bberror "Hit recoverable_error, correcting"
9753 fi
9754 if [ fatal_error ]; then
9755 bbfatal "fatal_error detected"
9756 fi
9757 bbdebug 2 "Completed do_my_function"
9758 }
9759
9760
9761Debugging Parallel Make Races
9762-----------------------------
9763
9764A parallel ``make`` race occurs when the build consists of several parts
9765that are run simultaneously and a situation occurs when the output or
9766result of one part is not ready for use with a different part of the
9767build that depends on that output. Parallel make races are annoying and
9768can sometimes be difficult to reproduce and fix. However, some simple
9769tips and tricks exist that can help you debug and fix them. This section
9770presents a real-world example of an error encountered on the Yocto
9771Project autobuilder and the process used to fix it.
9772
9773.. note::
9774
9775 If you cannot properly fix a ``make`` race condition, you can work around it
9776 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
9777 variables.
9778
9779The Failure
9780~~~~~~~~~~~
9781
9782For this example, assume that you are building an image that depends on
9783the "neard" package. And, during the build, BitBake runs into problems
9784and creates the following output.
9785
9786.. note::
9787
9788 This example log file has longer lines artificially broken to make
9789 the listing easier to read.
9790
9791If you examine the output or the log file, you see the failure during
9792``make``:
9793
9794.. code-block:: none
9795
9796 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9797 | DEBUG: Executing shell function do_compile
9798 | NOTE: make -j 16
9799 | make --no-print-directory all-am
9800 | /bin/mkdir -p include/near
9801 | /bin/mkdir -p include/near
9802 | /bin/mkdir -p include/near
9803 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9804 0.14-r0/neard-0.14/include/types.h include/near/types.h
9805 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9806 0.14-r0/neard-0.14/include/log.h include/near/log.h
9807 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9808 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9809 | /bin/mkdir -p include/near
9810 | /bin/mkdir -p include/near
9811 | /bin/mkdir -p include/near
9812 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9813 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
9814 | /bin/mkdir -p include/near
9815 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9816 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
9817 | /bin/mkdir -p include/near
9818 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9819 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
9820 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9821 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
9822 | /bin/mkdir -p include/near
9823 | /bin/mkdir -p include/near
9824 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9825 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
9826 | /bin/mkdir -p include/near
9827 | /bin/mkdir -p include/near
9828 | /bin/mkdir -p include/near
9829 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9830 0.14-r0/neard-0.14/include/device.h include/near/device.h
9831 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9832 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
9833 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9834 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
9835 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9836 0.14-r0/neard-0.14/include/version.h include/near/version.h
9837 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9838 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
9839 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
9840 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/
9841 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
9842 yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
9843 -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
9844 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/
9845 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/yocto-slave/
9846 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
9847 yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
9848 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
9849 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
9850 -o tools/snep-send.o tools/snep-send.c
9851 | In file included from tools/snep-send.c:16:0:
9852 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9853 | #include <near/dbus.h>
9854 | ^
9855 | compilation terminated.
9856 | make[1]: *** [tools/snep-send.o] Error 1
9857 | make[1]: *** Waiting for unfinished jobs....
9858 | make: *** [all] Error 2
9859 | ERROR: oe_runmake failed
9860
9861Reproducing the Error
9862~~~~~~~~~~~~~~~~~~~~~
9863
9864Because race conditions are intermittent, they do not manifest
9865themselves every time you do the build. In fact, most times the build
9866will complete without problems even though the potential race condition
9867exists. Thus, once the error surfaces, you need a way to reproduce it.
9868
9869In this example, compiling the "neard" package is causing the problem.
9870So the first thing to do is build "neard" locally. Before you start the
9871build, set the
9872:term:`PARALLEL_MAKE` variable
9873in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
9874high value for ``PARALLEL_MAKE`` increases the chances of the race
9875condition showing up:
9876::
9877
9878 $ bitbake neard
9879
9880Once the local build for "neard" completes, start a ``devshell`` build:
9881::
9882
9883 $ bitbake neard -c devshell
9884
9885For information on how to use a
9886``devshell``, see the "`Using a Development
9887Shell <#platdev-appdev-devshell>`__" section.
9888
9889In the ``devshell``, do the following:
9890::
9891
9892 $ make clean
9893 $ make tools/snep-send.o
9894
9895The ``devshell`` commands cause the failure to clearly
9896be visible. In this case, a missing dependency exists for the "neard"
9897Makefile target. Here is some abbreviated, sample output with the
9898missing dependency clearly visible at the end:
9899::
9900
9901 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
9902 .
9903 .
9904 .
9905 tools/snep-send.c
9906 In file included from tools/snep-send.c:16:0:
9907 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9908 #include <near/dbus.h>
9909 ^
9910 compilation terminated.
9911 make: *** [tools/snep-send.o] Error 1
9912 $
9913
9914
9915Creating a Patch for the Fix
9916~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9917
9918Because there is a missing dependency for the Makefile target, you need
9919to patch the ``Makefile.am`` file, which is generated from
9920``Makefile.in``. You can use Quilt to create the patch:
9921::
9922
9923 $ quilt new parallelmake.patch
9924 Patch patches/parallelmake.patch is now on top
9925 $ quilt add Makefile.am
9926 File Makefile.am added to patch patches/parallelmake.patch
9927
9928For more information on using Quilt, see the
9929"`Using Quilt in Your Workflow <#using-a-quilt-workflow>`__" section.
9930
9931At this point you need to make the edits to ``Makefile.am`` to add the
9932missing dependency. For our example, you have to add the following line
9933to the file:
9934::
9935
9936 tools/snep-send.$(OBJEXT): include/near/dbus.h
9937
9938Once you have edited the file, use the ``refresh`` command to create the
9939patch:
9940::
9941
9942 $ quilt refresh
9943 Refreshed patch patches/parallelmake.patch
9944
9945Once
9946the patch file exists, you need to add it back to the originating recipe
9947folder. Here is an example assuming a top-level
9948:term:`Source Directory` named ``poky``:
9949::
9950
9951 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
9952
9953The final thing you need to do to implement the fix in the build is to
9954update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
9955:term:`SRC_URI` statement includes
9956the patch file. The recipe file is in the folder above the patch. Here
9957is what the edited ``SRC_URI`` statement would look like:
9958::
9959
9960 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
9961 file://neard.in \
9962 file://neard.service.in \
9963 file://parallelmake.patch \
9964 "
9965
9966With the patch complete and moved to the correct folder and the
9967``SRC_URI`` statement updated, you can exit the ``devshell``:
9968::
9969
9970 $ exit
9971
9972Testing the Build
9973~~~~~~~~~~~~~~~~~
9974
9975With everything in place, you can get back to trying the build again
9976locally:
9977::
9978
9979 $ bitbake neard
9980
9981This build should succeed.
9982
9983Now you can open up a ``devshell`` again and repeat the clean and make
9984operations as follows:
9985::
9986
9987 $ bitbake neard -c devshell
9988 $ make clean
9989 $ make tools/snep-send.o
9990
9991The build should work without issue.
9992
9993As with all solved problems, if they originated upstream, you need to
9994submit the fix for the recipe in OE-Core and upstream so that the
9995problem is taken care of at its source. See the "`Submitting a Change to
9996the Yocto Project <#how-to-submit-a-change>`__" section for more
9997information.
9998
9999Debugging With the GNU Project Debugger (GDB) Remotely
10000------------------------------------------------------
10001
10002GDB allows you to examine running programs, which in turn helps you to
10003understand and fix problems. It also allows you to perform post-mortem
10004style analysis of program crashes. GDB is available as a package within
10005the Yocto Project and is installed in SDK images by default. See the
10006":ref:`ref-manual/images:Images`" chapter in the Yocto
10007Project Reference Manual for a description of these images. You can find
10008information on GDB at https://sourceware.org/gdb/.
10009
10010.. note::
10011
10012 For best results, install debug (``-dbg``) packages for the applications you
10013 are going to debug. Doing so makes extra debug symbols available that give
10014 you more meaningful output.
10015
10016Sometimes, due to memory or disk space constraints, it is not possible
10017to use GDB directly on the remote target to debug applications. These
10018constraints arise because GDB needs to load the debugging information
10019and the binaries of the process being debugged. Additionally, GDB needs
10020to perform many computations to locate information such as function
10021names, variable names and values, stack traces and so forth - even
10022before starting the debugging process. These extra computations place
10023more load on the target system and can alter the characteristics of the
10024program being debugged.
10025
10026To help get past the previously mentioned constraints, you can use
10027gdbserver, which runs on the remote target and does not load any
10028debugging information from the debugged process. Instead, a GDB instance
10029processes the debugging information that is run on a remote computer -
10030the host GDB. The host GDB then sends control commands to gdbserver to
10031make it stop or start the debugged program, as well as read or write
10032memory regions of that debugged program. All the debugging information
10033loaded and processed as well as all the heavy debugging is done by the
10034host GDB. Offloading these processes gives the gdbserver running on the
10035target a chance to remain small and fast.
10036
10037Because the host GDB is responsible for loading the debugging
10038information and for doing the necessary processing to make actual
10039debugging happen, you have to make sure the host can access the
10040unstripped binaries complete with their debugging information and also
10041be sure the target is compiled with no optimizations. The host GDB must
10042also have local access to all the libraries used by the debugged
10043program. Because gdbserver does not need any local debugging
10044information, the binaries on the remote target can remain stripped.
10045However, the binaries must also be compiled without optimization so they
10046match the host's binaries.
10047
10048To remain consistent with GDB documentation and terminology, the binary
10049being debugged on the remote target machine is referred to as the
10050"inferior" binary. For documentation on GDB see the `GDB
10051site <https://sourceware.org/gdb/documentation/>`__.
10052
10053The following steps show you how to debug using the GNU project
10054debugger.
10055
100561. *Configure your build system to construct the companion debug
10057 filesystem:*
10058
10059 In your ``local.conf`` file, set the following:
10060 ::
10061
10062 IMAGE_GEN_DEBUGFS = "1"
10063 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
10064
10065 These options cause the
10066 OpenEmbedded build system to generate a special companion filesystem
10067 fragment, which contains the matching source and debug symbols to
10068 your deployable filesystem. The build system does this by looking at
10069 what is in the deployed filesystem, and pulling the corresponding
10070 ``-dbg`` packages.
10071
10072 The companion debug filesystem is not a complete filesystem, but only
10073 contains the debug fragments. This filesystem must be combined with
10074 the full filesystem for debugging. Subsequent steps in this procedure
10075 show how to combine the partial filesystem with the full filesystem.
10076
100772. *Configure the system to include gdbserver in the target filesystem:*
10078
10079 Make the following addition in either your ``local.conf`` file or in
10080 an image recipe:
10081 ::
10082
10083 IMAGE_INSTALL_append = " gdbserver"
10084
10085 The change makes
10086 sure the ``gdbserver`` package is included.
10087
100883. *Build the environment:*
10089
10090 Use the following command to construct the image and the companion
10091 Debug Filesystem:
10092 ::
10093
10094 $ bitbake image
10095
10096 Build the cross GDB component and
10097 make it available for debugging. Build the SDK that matches the
10098 image. Building the SDK is best for a production build that can be
10099 used later for debugging, especially during long term maintenance:
10100 ::
10101
10102 $ bitbake -c populate_sdk image
10103
10104 Alternatively, you can build the minimal toolchain components that
10105 match the target. Doing so creates a smaller than typical SDK and
10106 only contains a minimal set of components with which to build simple
10107 test applications, as well as run the debugger:
10108 ::
10109
10110 $ bitbake meta-toolchain
10111
10112 A final method is to build Gdb itself within the build system:
10113 ::
10114
10115 $ bitbake gdb-cross-<architecture>
10116
10117 Doing so produces a temporary copy of
10118 ``cross-gdb`` you can use for debugging during development. While
10119 this is the quickest approach, the two previous methods in this step
10120 are better when considering long-term maintenance strategies.
10121
10122 .. note::
10123
10124 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
10125 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
10126 actual name you want to use.
10127
101284. *Set up the* ``debugfs``\ *:*
10129
10130 Run the following commands to set up the ``debugfs``:
10131 ::
10132
10133 $ mkdir debugfs
10134 $ cd debugfs
10135 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
10136 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2
10137
101385. *Set up GDB:*
10139
10140 Install the SDK (if you built one) and then source the correct
10141 environment file. Sourcing the environment file puts the SDK in your
10142 ``PATH`` environment variable.
10143
10144 If you are using the build system, Gdb is located in
10145 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
10146
101476. *Boot the target:*
10148
10149 For information on how to run QEMU, see the `QEMU
10150 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
10151
10152 .. note::
10153
10154 Be sure to verify that your host can access the target via TCP.
10155
101567. *Debug a program:*
10157
10158 Debugging a program involves running gdbserver on the target and then
10159 running Gdb on the host. The example in this step debugs ``gzip``:
10160
10161 .. code-block:: shell
10162
10163 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
10164
10165 For
10166 additional gdbserver options, see the `GDB Server
10167 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
10168
10169 After running gdbserver on the target, you need to run Gdb on the
10170 host and configure it and connect to the target. Use these commands:
10171 ::
10172
10173 $ cd directory-holding-the-debugfs-directory
10174 $ arch-gdb
10175 (gdb) set sysroot debugfs
10176 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
10177 (gdb) target remote IP-of-target:1234
10178
10179 At this
10180 point, everything should automatically load (i.e. matching binaries,
10181 symbols and headers).
10182
10183 .. note::
10184
10185 The Gdb ``set`` commands in the previous example can be placed into the
10186 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
10187 commands are in that file.
10188
101898. *Deploying without a full image rebuild:*
10190
10191 In many cases, during development you want a quick method to deploy a
10192 new binary to the target and debug it, without waiting for a full
10193 image build.
10194
10195 One approach to solving this situation is to just build the component
10196 you want to debug. Once you have built the component, copy the
10197 executable directly to both the target and the host ``debugfs``.
10198
10199 If the binary is processed through the debug splitting in
10200 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
10201 contents and corresponding ``/usr/src/debug`` files) from the work
10202 directory. Here is an example:
10203 ::
10204
10205 $ bitbake bash
10206 $ bitbake -c devshell bash
10207 $ cd ..
10208 $ scp packages-split/bash/bin/bash target:/bin/bash
10209 $ cp -a packages-split/bash-dbg/\* path/debugfs
10210
10211Debugging with the GNU Project Debugger (GDB) on the Target
10212-----------------------------------------------------------
10213
10214The previous section addressed using GDB remotely for debugging
10215purposes, which is the most usual case due to the inherent hardware
10216limitations on many embedded devices. However, debugging in the target
10217hardware itself is also possible with more powerful devices. This
10218section describes what you need to do in order to support using GDB to
10219debug on the target hardware.
10220
10221To support this kind of debugging, you need do the following:
10222
10223- Ensure that GDB is on the target. You can do this by adding "gdb" to
10224 :term:`IMAGE_INSTALL`:
10225 ::
10226
10227 IMAGE_INSTALL_append = " gdb"
10228
10229 Alternatively, you can add "tools-debug" to :term:`IMAGE_FEATURES`:
10230 ::
10231
10232 IMAGE_FEATURES_append = " tools-debug"
10233
10234- Ensure that debug symbols are present. You can make sure these
10235 symbols are present by installing ``-dbg``:
10236 ::
10237
10238 IMAGE_INSTALL_append = "packagename-dbg"
10239
10240 Alternatively, you can do the following to include
10241 all the debug symbols:
10242 ::
10243
10244 IMAGE_FEATURES_append = " dbg-pkgs"
10245
10246.. note::
10247
10248 To improve the debug information accuracy, you can reduce the level
10249 of optimization used by the compiler. For example, when adding the
10250 following line to your ``local.conf`` file, you will reduce optimization
10251 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
10252 of "-O -fno-omit-frame-pointer":
10253 ::
10254
10255 DEBUG_BUILD = "1"
10256
10257 Consider that this will reduce the application's performance and is
10258 recommended only for debugging purposes.
10259
10260Other Debugging Tips
10261--------------------
10262
10263Here are some other tips that you might find useful:
10264
10265- When adding new packages, it is worth watching for undesirable items
10266 making their way into compiler command lines. For example, you do not
10267 want references to local system files like ``/usr/lib/`` or
10268 ``/usr/include/``.
10269
10270- If you want to remove the ``psplash`` boot splashscreen, add
10271 ``psplash=false`` to the kernel command line. Doing so prevents
10272 ``psplash`` from loading and thus allows you to see the console. It
10273 is also possible to switch out of the splashscreen by switching the
10274 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10275
10276- Removing :term:`TMPDIR` (usually
10277 ``tmp/``, within the
10278 :term:`Build Directory`) can often fix
10279 temporary build issues. Removing ``TMPDIR`` is usually a relatively
10280 cheap operation, because task output will be cached in
10281 :term:`SSTATE_DIR` (usually
10282 ``sstate-cache/``, which is also in the Build Directory).
10283
10284 .. note::
10285
10286 Removing ``TMPDIR`` might be a workaround rather than a fix.
10287 Consequently, trying to determine the underlying cause of an issue before
10288 removing the directory is a good idea.
10289
10290- Understanding how a feature is used in practice within existing
10291 recipes can be very helpful. It is recommended that you configure
10292 some method that allows you to quickly search through files.
10293
10294 Using GNU Grep, you can use the following shell function to
10295 recursively search through common recipe-related files, skipping
10296 binary files, ``.git`` directories, and the Build Directory (assuming
10297 its name starts with "build"):
10298 ::
10299
10300 g() {
10301 grep -Ir \
10302 --exclude-dir=.git \
10303 --exclude-dir='build*' \
10304 --include='*.bb*' \
10305 --include='*.inc*' \
10306 --include='*.conf*' \
10307 --include='*.py*' \
10308 "$@"
10309 }
10310
10311 Following are some usage examples:
10312 ::
10313
10314 $ g FOO # Search recursively for "FOO"
10315 $ g -i foo # Search recursively for "foo", ignoring case
10316 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10317
10318 If figuring
10319 out how some feature works requires a lot of searching, it might
10320 indicate that the documentation should be extended or improved. In
10321 such cases, consider filing a documentation bug using the Yocto
10322 Project implementation of
10323 :yocto_bugs:`Bugzilla <>`. For information on
10324 how to submit a bug against the Yocto Project, see the Yocto Project
10325 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
10326 and the "`Submitting a Defect Against the Yocto
10327 Project <#submitting-a-defect-against-the-yocto-project>`__" section.
10328
10329 .. note::
10330
10331 The manuals might not be the right place to document variables
10332 that are purely internal and have a limited scope (e.g. internal
10333 variables used to implement a single ``.bbclass`` file).
10334
10335Making Changes to the Yocto Project
10336===================================
10337
10338Because the Yocto Project is an open-source, community-based project,
10339you can effect changes to the project. This section presents procedures
10340that show you how to submit a defect against the project and how to
10341submit a change.
10342
10343Submitting a Defect Against the Yocto Project
10344---------------------------------------------
10345
10346Use the Yocto Project implementation of
10347`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
10348against the Yocto Project. For additional information on this
10349implementation of Bugzilla see the ":ref:`Yocto Project
10350Bugzilla <resources-bugtracker>`" section in the
10351Yocto Project Reference Manual. For more detail on any of the following
10352steps, see the Yocto Project
10353:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`.
10354
10355Use the following general steps to submit a bug:
10356
103571. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10358
103592. Click "File a Bug" to enter a new bug.
10360
103613. Choose the appropriate "Classification", "Product", and "Component"
10362 for which the bug was found. Bugs for the Yocto Project fall into
10363 one of several classifications, which in turn break down into
10364 several products and components. For example, for a bug against the
10365 ``meta-intel`` layer, you would choose "Build System, Metadata &
10366 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10367
103684. Choose the "Version" of the Yocto Project for which you found the
10369 bug (e.g. &DISTRO;).
10370
103715. Determine and select the "Severity" of the bug. The severity
10372 indicates how the bug impacted your work.
10373
103746. Choose the "Hardware" that the bug impacts.
10375
103767. Choose the "Architecture" that the bug impacts.
10377
103788. Choose a "Documentation change" item for the bug. Fixing a bug might
10379 or might not affect the Yocto Project documentation. If you are
10380 unsure of the impact to the documentation, select "Don't Know".
10381
103829. Provide a brief "Summary" of the bug. Try to limit your summary to
10383 just a line or two and be sure to capture the essence of the bug.
10384
1038510. Provide a detailed "Description" of the bug. You should provide as
10386 much detail as you can about the context, behavior, output, and so
10387 forth that surrounds the bug. You can even attach supporting files
10388 for output from logs by using the "Add an attachment" button.
10389
1039011. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10391 is assigned to the bug and the defect is logged in the bug tracking
10392 system.
10393
10394Once you file a bug, the bug is processed by the Yocto Project Bug
10395Triage Team and further details concerning the bug are assigned (e.g.
10396priority and owner). You are the "Submitter" of the bug and any further
10397categorization, progress, or comments on the bug result in Bugzilla
10398sending you an automated email concerning the particular change or
10399progress to the bug.
10400
10401Submitting a Change to the Yocto Project
10402----------------------------------------
10403
10404Contributions to the Yocto Project and OpenEmbedded are very welcome.
10405Because the system is extremely configurable and flexible, we recognize
10406that developers will want to extend, configure or optimize it for their
10407specific uses.
10408
10409The Yocto Project uses a mailing list and a patch-based workflow that is
10410similar to the Linux kernel but contains important differences. In
10411general, a mailing list exists through which you can submit patches. You
10412should send patches to the appropriate mailing list so that they can be
10413reviewed and merged by the appropriate maintainer. The specific mailing
10414list you need to use depends on the location of the code you are
10415changing. Each component (e.g. layer) should have a ``README`` file that
10416indicates where to send the changes and which process to follow.
10417
10418You can send the patch to the mailing list using whichever approach you
10419feel comfortable with to generate the patch. Once sent, the patch is
10420usually reviewed by the community at large. If somebody has concerns
10421with the patch, they will usually voice their concern over the mailing
10422list. If a patch does not receive any negative reviews, the maintainer
10423of the affected layer typically takes the patch, tests it, and then
10424based on successful testing, merges the patch.
10425
10426The "poky" repository, which is the Yocto Project's reference build
10427environment, is a hybrid repository that contains several individual
10428pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10429the combo-layer tool. The upstream location used for submitting changes
10430varies by component:
10431
10432- *Core Metadata:* Send your patch to the
10433 :oe_lists:`openembedded-core </g/openembedded-core>`
10434 mailing list. For example, a change to anything under the ``meta`` or
10435 ``scripts`` directories should be sent to this mailing list.
10436
10437- *BitBake:* For changes to BitBake (i.e. anything under the
10438 ``bitbake`` directory), send your patch to the
10439 :oe_lists:`bitbake-devel </g/bitbake-devel>`
10440 mailing list.
10441
10442- *"meta-\*" trees:* These trees contain Metadata. Use the
10443 :yocto_lists:`poky </g/poky>` mailing list.
10444
10445- *Documentation*: For changes to the Yocto Project documentation, use the
10446 :yocto_lists:`docs </g/docs>` mailing list.
10447
10448For changes to other layers hosted in the Yocto Project source
10449repositories (i.e. ``yoctoproject.org``) and tools use the
10450:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
10451
10452.. note::
10453
10454 Sometimes a layer's documentation specifies to use a particular
10455 mailing list. If so, use that list.
10456
10457For additional recipes that do not fit into the core Metadata, you
10458should determine which layer the recipe should go into and submit the
10459change in the manner recommended by the documentation (e.g. the
10460``README`` file) supplied with the layer. If in doubt, please ask on the
10461Yocto general mailing list or on the openembedded-devel mailing list.
10462
10463You can also push a change upstream and request a maintainer to pull the
10464change into the component's upstream repository. You do this by pushing
10465to a contribution repository that is upstream. See the
10466":ref:`overview-manual/development-environment:git workflows and the yocto project`"
10467section in the Yocto Project Overview and Concepts Manual for additional
10468concepts on working in the Yocto Project development environment.
10469
10470Maintainers commonly use ``-next`` branches to test submissions prior to
10471merging patches. Thus, you can get an idea of the status of a patch based on
10472whether the patch has been merged into one of these branches. The commonly
10473used testing branches for OpenEmbedded-Core are as follows:
10474
10475- *openembedded-core "master-next" branch:* This branch is part of the
10476 :oe_git:`openembedded-core </openembedded-core/>` repository and contains
10477 proposed changes to the core metadata.
10478
10479- *poky "master-next" branch:* This branch is part of the
10480 :yocto_git:`poky </poky/>` repository and combines proposed
10481 changes to bitbake, the core metadata and the poky distro.
10482
10483Similarly, stable branches maintained by the project may have corresponding
10484``-next`` branches which collect proposed changes. For example,
10485``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next``
10486branches in both the "openembdedded-core" and "poky" repositories.
10487
10488Other layers may have similar testing branches but there is no formal
10489requirement or standard for these so please check the documentation for the
10490layers you are contributing to.
10491
10492The following sections provide procedures for submitting a change.
10493
10494Preparing Changes for Submission
10495~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10496
104971. *Make Your Changes Locally:* Make your changes in your local Git
10498 repository. You should make small, controlled, isolated changes.
10499 Keeping changes small and isolated aids review, makes
10500 merging/rebasing easier and keeps the change history clean should
10501 anyone need to refer to it in future.
10502
105032. *Stage Your Changes:* Stage your changes by using the ``git add``
10504 command on each file you changed.
10505
105063. *Commit Your Changes:* Commit the change by using the ``git commit``
10507 command. Make sure your commit information follows standards by
10508 following these accepted conventions:
10509
10510 - Be sure to include a "Signed-off-by:" line in the same style as
10511 required by the Linux kernel. Adding this line signifies that you,
10512 the submitter, have agreed to the Developer's Certificate of
10513 Origin 1.1 as follows:
10514
10515 .. code-block:: none
10516
10517 Developer's Certificate of Origin 1.1
10518
10519 By making a contribution to this project, I certify that:
10520
10521 (a) The contribution was created in whole or in part by me and I
10522 have the right to submit it under the open source license
10523 indicated in the file; or
10524
10525 (b) The contribution is based upon previous work that, to the best
10526 of my knowledge, is covered under an appropriate open source
10527 license and I have the right under that license to submit that
10528 work with modifications, whether created in whole or in part
10529 by me, under the same open source license (unless I am
10530 permitted to submit under a different license), as indicated
10531 in the file; or
10532
10533 (c) The contribution was provided directly to me by some other
10534 person who certified (a), (b) or (c) and I have not modified
10535 it.
10536
10537 (d) I understand and agree that this project and the contribution
10538 are public and that a record of the contribution (including all
10539 personal information I submit with it, including my sign-off) is
10540 maintained indefinitely and may be redistributed consistent with
10541 this project or the open source license(s) involved.
10542
10543 - Provide a single-line summary of the change and, if more
10544 explanation is needed, provide more detail in the body of the
10545 commit. This summary is typically viewable in the "shortlist" of
10546 changes. Thus, providing something short and descriptive that
10547 gives the reader a summary of the change is useful when viewing a
10548 list of many commits. You should prefix this short description
10549 with the recipe name (if changing a recipe), or else with the
10550 short form path to the file being changed.
10551
10552 - For the body of the commit message, provide detailed information
10553 that describes what you changed, why you made the change, and the
10554 approach you used. It might also be helpful if you mention how you
10555 tested the change. Provide as much detail as you can in the body
10556 of the commit message.
10557
10558 .. note::
10559
10560 You do not need to provide a more detailed explanation of a
10561 change if the change is minor to the point of the single line
10562 summary providing all the information.
10563
10564 - If the change addresses a specific bug or issue that is associated
10565 with a bug-tracking ID, include a reference to that ID in your
10566 detailed description. For example, the Yocto Project uses a
10567 specific convention for bug references - any commit that addresses
10568 a specific bug should use the following form for the detailed
10569 description. Be sure to use the actual bug-tracking ID from
10570 Bugzilla for bug-id:
10571 ::
10572
10573 Fixes [YOCTO #bug-id]
10574
10575 detailed description of change
10576
10577Using Email to Submit a Patch
10578~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10579
10580Depending on the components changed, you need to submit the email to a
10581specific mailing list. For some guidance on which mailing list to use,
10582see the `list <#figuring-out-the-mailing-list-to-use>`__ at the
10583beginning of this section. For a description of all the available
10584mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
10585Yocto Project Reference Manual.
10586
10587Here is the general procedure on how to submit a patch through email
10588without using the scripts once the steps in
10589:ref:`dev-manual/common-tasks:preparing changes for submission` have been followed:
10590
105911. *Format the Commit:* Format the commit into an email message. To
10592 format commits, use the ``git format-patch`` command. When you
10593 provide the command, you must include a revision list or a number of
10594 patches as part of the command. For example, either of these two
10595 commands takes your most recent single commit and formats it as an
10596 email message in the current directory:
10597 ::
10598
10599 $ git format-patch -1
10600
10601 or ::
10602
10603 $ git format-patch HEAD~
10604
10605 After the command is run, the current directory contains a numbered
10606 ``.patch`` file for the commit.
10607
10608 If you provide several commits as part of the command, the
10609 ``git format-patch`` command produces a series of numbered files in
10610 the current directory – one for each commit. If you have more than
10611 one patch, you should also use the ``--cover`` option with the
10612 command, which generates a cover letter as the first "patch" in the
10613 series. You can then edit the cover letter to provide a description
10614 for the series of patches. For information on the
10615 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
10616 using the ``man git-format-patch`` command.
10617
10618 .. note::
10619
10620 If you are or will be a frequent contributor to the Yocto Project
10621 or to OpenEmbedded, you might consider requesting a contrib area
10622 and the necessary associated rights.
10623
106242. *Send the patches via email:* Send the patches to the recipients and
10625 relevant mailing lists by using the ``git send-email`` command.
10626
10627 .. note::
10628
10629 In order to use ``git send-email``, you must have the proper Git packages
10630 installed on your host.
10631 For Ubuntu, Debian, and Fedora the package is ``git-email``.
10632
10633 The ``git send-email`` command sends email by using a local or remote
10634 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
10635 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
10636 file. If you are submitting patches through email only, it is very
10637 important that you submit them without any whitespace or HTML
10638 formatting that either you or your mailer introduces. The maintainer
10639 that receives your patches needs to be able to save and apply them
10640 directly from your emails. A good way to verify that what you are
10641 sending will be applicable by the maintainer is to do a dry run and
10642 send them to yourself and then save and apply them as the maintainer
10643 would.
10644
10645 The ``git send-email`` command is the preferred method for sending
10646 your patches using email since there is no risk of compromising
10647 whitespace in the body of the message, which can occur when you use
10648 your own mail client. The command also has several options that let
10649 you specify recipients and perform further editing of the email
10650 message. For information on how to use the ``git send-email``
10651 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
10652 ``man git-send-email`` command.
10653
10654The Yocto Project uses a `Patchwork instance <https://patchwork.openembedded.org/>`__
10655to track the status of patches submitted to the various mailing lists and to
10656support automated patch testing. Each submitted patch is checked for common
10657mistakes and deviations from the expected patch format and submitters are
10658notified by patchtest if such mistakes are found. This process helps to
10659reduce the burden of patch review on maintainers.
10660
10661.. note::
10662
10663 This system is imperfect and changes can sometimes get lost in the flow.
10664 Asking about the status of a patch or change is reasonable if the change
10665 has been idle for a while with no feedback.
10666
10667Using Scripts to Push a Change Upstream and Request a Pull
10668~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10669
10670For larger patch series it is preferable to send a pull request which not
10671only includes the patch but also a pointer to a branch that can be pulled
10672from. This involves making a local branch for your changes, pushing this
10673branch to an accessible repository and then using the ``create-pull-request``
10674and ``send-pull-request`` scripts from openembedded-core to create and send a
10675patch series with a link to the branch for review.
10676
10677Follow this procedure to push a change to an upstream "contrib" Git
10678repository once the steps in :ref:`dev-manual/common-tasks:preparing changes for submission` have
10679been followed:
10680
10681.. note::
10682
10683 You can find general Git information on how to push a change upstream
10684 in the
10685 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
10686
106871. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
10688 permissions to push to an upstream contrib repository, push the
10689 change to that repository:
10690 ::
10691
10692 $ git push upstream_remote_repo local_branch_name
10693
10694 For example, suppose you have permissions to push
10695 into the upstream ``meta-intel-contrib`` repository and you are
10696 working in a local branch named `your_name`\ ``/README``. The following
10697 command pushes your local commits to the ``meta-intel-contrib``
10698 upstream repository and puts the commit in a branch named
10699 `your_name`\ ``/README``:
10700 ::
10701
10702 $ git push meta-intel-contrib your_name/README
10703
107042. *Determine Who to Notify:* Determine the maintainer or the mailing
10705 list that you need to notify for the change.
10706
10707 Before submitting any change, you need to be sure who the maintainer
10708 is or what mailing list that you need to notify. Use either these
10709 methods to find out:
10710
10711 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10712 located in the :term:`Source Directory` at
10713 ``meta/conf/distro/include``, to see who is responsible for code.
10714
10715 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can
10716 enter the following command to bring up a short list of all
10717 commits against a specific file:
10718 ::
10719
10720 git shortlog -- filename
10721
10722 Just provide the name of the file for which you are interested. The
10723 information returned is not ordered by history but does include a
10724 list of everyone who has committed grouped by name. From the list,
10725 you can see who is responsible for the bulk of the changes against
10726 the file.
10727
10728 - *Examine the List of Mailing Lists:* For a list of the Yocto
10729 Project and related mailing lists, see the ":ref:`Mailing
10730 lists <resources-mailinglist>`" section in
10731 the Yocto Project Reference Manual.
10732
107333. *Make a Pull Request:* Notify the maintainer or the mailing list that
10734 you have pushed a change by making a pull request.
10735
10736 The Yocto Project provides two scripts that conveniently let you
10737 generate and send pull requests to the Yocto Project. These scripts
10738 are ``create-pull-request`` and ``send-pull-request``. You can find
10739 these scripts in the ``scripts`` directory within the
10740 :term:`Source Directory` (e.g.
10741 ``~/poky/scripts``).
10742
10743 Using these scripts correctly formats the requests without
10744 introducing any whitespace or HTML formatting. The maintainer that
10745 receives your patches either directly or through the mailing list
10746 needs to be able to save and apply them directly from your emails.
10747 Using these scripts is the preferred method for sending patches.
10748
10749 First, create the pull request. For example, the following command
10750 runs the script, specifies the upstream repository in the contrib
10751 directory into which you pushed the change, and provides a subject
10752 line in the created patch files:
10753 ::
10754
10755 $ ~/poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
10756
10757 Running this script forms ``*.patch`` files in a folder named
10758 ``pull-``\ `PID` in the current directory. One of the patch files is a
10759 cover letter.
10760
10761 Before running the ``send-pull-request`` script, you must edit the
10762 cover letter patch to insert information about your change. After
10763 editing the cover letter, send the pull request. For example, the
10764 following command runs the script and specifies the patch directory
10765 and email address. In this example, the email address is a mailing
10766 list:
10767 ::
10768
10769 $ ~/poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@yoctoproject.org
10770
10771 You need to follow the prompts as the script is interactive.
10772
10773 .. note::
10774
10775 For help on using these scripts, simply provide the ``-h``
10776 argument as follows:
10777 ::
10778
10779 $ poky/scripts/create-pull-request -h
10780 $ poky/scripts/send-pull-request -h
10781
10782Responding to Patch Review
10783~~~~~~~~~~~~~~~~~~~~~~~~~~
10784
10785You may get feedback on your submitted patches from other community members
10786or from the automated patchtest service. If issues are identified in your
10787patch then it is usually necessary to address these before the patch will be
10788accepted into the project. In this case you should amend the patch according
10789to the feedback and submit an updated version to the relevant mailing list,
10790copying in the reviewers who provided feedback to the previous version of the
10791patch.
10792
10793The patch should be amended using ``git commit --amend`` or perhaps ``git
10794rebase`` for more expert git users. You should also modify the ``[PATCH]``
10795tag in the email subject line when sending the revised patch to mark the new
10796iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be
10797done by passing the ``-v`` argument to ``git format-patch`` with a version
10798number.
10799
10800Lastly please ensure that you also test your revised changes. In particular
10801please don't just edit the patch file written out by ``git format-patch`` and
10802resend it.
10803
10804Submitting Changes to Stable Release Branches
10805~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10806
10807The process for proposing changes to a Yocto Project stable branch differs
10808from the steps described above. Changes to a stable branch must address
10809identified bugs or CVEs and should be made carefully in order to avoid the
10810risk of introducing new bugs or breaking backwards compatibility. Typically
10811bug fixes must already be accepted into the master branch before they can be
10812backported to a stable branch unless the bug in question does not affect the
10813master branch or the fix on the master branch is unsuitable for backporting.
10814
10815The list of stable branches along with the status and maintainer for each
10816branch can be obtained from the
10817:yocto_wiki:`Releases wiki page </Releases>`.
10818
10819.. note::
10820
10821 Changes will not typically be accepted for branches which are marked as
10822 End-Of-Life (EOL).
10823
10824With this in mind, the steps to submit a change for a stable branch are as
10825follows:
10826
108271. *Identify the bug or CVE to be fixed:* This information should be
10828 collected so that it can be included in your submission.
10829
108302. *Check if the fix is already present in the master branch:* This will
10831 result in the most straightforward path into the stable branch for the
10832 fix.
10833
10834 a. *If the fix is present in the master branch - Submit a backport request
10835 by email:* You should send an email to the relevant stable branch
10836 maintainer and the mailing list with details of the bug or CVE to be
10837 fixed, the commit hash on the master branch that fixes the issue and
10838 the stable branches which you would like this fix to be backported to.
10839
10840 b. *If the fix is not present in the master branch - Submit the fix to the
10841 master branch first:* This will ensure that the fix passes through the
10842 project's usual patch review and test processes before being accepted.
10843 It will also ensure that bugs are not left unresolved in the master
10844 branch itself. Once the fix is accepted in the master branch a backport
10845 request can be submitted as above.
10846
10847 c. *If the fix is unsuitable for the master branch - Submit a patch
10848 directly for the stable branch:* This method should be considered as a
10849 last resort. It is typically necessary when the master branch is using
10850 a newer version of the software which includes an upstream fix for the
10851 issue or when the issue has been fixed on the master branch in a way
10852 that introduces backwards incompatible changes. In this case follow the
10853 steps in :ref:`dev-manual/common-tasks:preparing changes for submission` and
10854 :ref:`dev-manual/common-tasks:using email to submit a patch` but modify the subject header of your patch
10855 email to include the name of the stable branch which you are
10856 targetting. This can be done using the ``--subject-prefix`` argument to
10857 ``git format-patch``, for example to submit a patch to the dunfell
10858 branch use
10859 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``.
10860
10861Working With Licenses
10862=====================
10863
10864As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
10865section in the Yocto Project Overview and Concepts Manual, open source
10866projects are open to the public and they consequently have different
10867licensing structures in place. This section describes the mechanism by
10868which the :term:`OpenEmbedded Build System`
10869tracks changes to
10870licensing text and covers how to maintain open source license compliance
10871during your project's lifecycle. The section also describes how to
10872enable commercially licensed recipes, which by default are disabled.
10873
10874Tracking License Changes
10875------------------------
10876
10877The license of an upstream project might change in the future. In order
10878to prevent these changes going unnoticed, the
10879:term:`LIC_FILES_CHKSUM`
10880variable tracks changes to the license text. The checksums are validated
10881at the end of the configure step, and if the checksums do not match, the
10882build will fail.
10883
10884Specifying the ``LIC_FILES_CHKSUM`` Variable
10885~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10886
10887The ``LIC_FILES_CHKSUM`` variable contains checksums of the license text
10888in the source code for the recipe. Following is an example of how to
10889specify ``LIC_FILES_CHKSUM``:
10890::
10891
10892 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
10893 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
10894 file://licfile2.txt;endline=50;md5=zzzz \
10895 ..."
10896
10897.. note::
10898
10899 - When using "beginline" and "endline", realize that line numbering
10900 begins with one and not zero. Also, the included lines are
10901 inclusive (i.e. lines five through and including 29 in the
10902 previous example for ``licfile1.txt``).
10903
10904 - When a license check fails, the selected license text is included
10905 as part of the QA message. Using this output, you can determine
10906 the exact start and finish for the needed license text.
10907
10908The build system uses the :term:`S`
10909variable as the default directory when searching files listed in
10910``LIC_FILES_CHKSUM``. The previous example employs the default
10911directory.
10912
10913Consider this next example:
10914::
10915
10916 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
10917 md5=bb14ed3c4cda583abc85401304b5cd4e"
10918 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
10919
10920The first line locates a file in ``${S}/src/ls.c`` and isolates lines
10921five through 16 as license text. The second line refers to a file in
10922:term:`WORKDIR`.
10923
10924Note that ``LIC_FILES_CHKSUM`` variable is mandatory for all recipes,
10925unless the ``LICENSE`` variable is set to "CLOSED".
10926
10927Explanation of Syntax
10928~~~~~~~~~~~~~~~~~~~~~
10929
10930As mentioned in the previous section, the ``LIC_FILES_CHKSUM`` variable
10931lists all the important files that contain the license text for the
10932source code. It is possible to specify a checksum for an entire file, or
10933a specific section of a file (specified by beginning and ending line
10934numbers with the "beginline" and "endline" parameters, respectively).
10935The latter is useful for source files with a license notice header,
10936README documents, and so forth. If you do not use the "beginline"
10937parameter, then it is assumed that the text begins on the first line of
10938the file. Similarly, if you do not use the "endline" parameter, it is
10939assumed that the license text ends with the last line of the file.
10940
10941The "md5" parameter stores the md5 checksum of the license text. If the
10942license text changes in any way as compared to this parameter then a
10943mismatch occurs. This mismatch triggers a build failure and notifies the
10944developer. Notification allows the developer to review and address the
10945license text changes. Also note that if a mismatch occurs during the
10946build, the correct md5 checksum is placed in the build log and can be
10947easily copied to the recipe.
10948
10949There is no limit to how many files you can specify using the
10950``LIC_FILES_CHKSUM`` variable. Generally, however, every project
10951requires a few specifications for license tracking. Many projects have a
10952"COPYING" file that stores the license information for all the source
10953code files. This practice allows you to just track the "COPYING" file as
10954long as it is kept up to date.
10955
10956.. note::
10957
10958 - If you specify an empty or invalid "md5" parameter,
10959 :term:`BitBake` returns an md5
10960 mis-match error and displays the correct "md5" parameter value
10961 during the build. The correct parameter is also captured in the
10962 build log.
10963
10964 - If the whole file contains only license text, you do not need to
10965 use the "beginline" and "endline" parameters.
10966
10967Enabling Commercially Licensed Recipes
10968--------------------------------------
10969
10970By default, the OpenEmbedded build system disables components that have
10971commercial or other special licensing requirements. Such requirements
10972are defined on a recipe-by-recipe basis through the
10973:term:`LICENSE_FLAGS` variable
10974definition in the affected recipe. For instance, the
10975``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
10976contains the following statement:
10977::
10978
10979 LICENSE_FLAGS = "commercial"
10980
10981Here is a
10982slightly more complicated example that contains both an explicit recipe
10983name and version (after variable expansion):
10984::
10985
10986 LICENSE_FLAGS = "license_${PN}_${PV}"
10987
10988In order for a component restricted by a
10989``LICENSE_FLAGS`` definition to be enabled and included in an image, it
10990needs to have a matching entry in the global
10991:term:`LICENSE_FLAGS_WHITELIST`
10992variable, which is a variable typically defined in your ``local.conf``
10993file. For example, to enable the
10994``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
10995could add either the string "commercial_gst-plugins-ugly" or the more
10996general string "commercial" to ``LICENSE_FLAGS_WHITELIST``. See the
10997"`License Flag Matching <#license-flag-matching>`__" section for a full
10998explanation of how ``LICENSE_FLAGS`` matching works. Here is the
10999example:
11000::
11001
11002 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
11003
11004Likewise, to additionally enable the package built from the recipe
11005containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
11006the actual recipe name was ``emgd_1.10.bb``, the following string would
11007enable that package as well as the original ``gst-plugins-ugly``
11008package:
11009::
11010
11011 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
11012
11013As a convenience, you do not need to specify the
11014complete license string in the whitelist for every package. You can use
11015an abbreviated form, which consists of just the first portion or
11016portions of the license string before the initial underscore character
11017or characters. A partial string will match any license that contains the
11018given string as the first portion of its license. For example, the
11019following whitelist string will also match both of the packages
11020previously mentioned as well as any other packages that have licenses
11021starting with "commercial" or "license".
11022::
11023
11024 LICENSE_FLAGS_WHITELIST = "commercial license"
11025
11026License Flag Matching
11027~~~~~~~~~~~~~~~~~~~~~
11028
11029License flag matching allows you to control what recipes the
11030OpenEmbedded build system includes in the build. Fundamentally, the
11031build system attempts to match ``LICENSE_FLAGS`` strings found in
11032recipes against ``LICENSE_FLAGS_WHITELIST`` strings found in the
11033whitelist. A match causes the build system to include a recipe in the
11034build, while failure to find a match causes the build system to exclude
11035a recipe.
11036
11037In general, license flag matching is simple. However, understanding some
11038concepts will help you correctly and effectively use matching.
11039
11040Before a flag defined by a particular recipe is tested against the
11041contents of the whitelist, the expanded string ``_${PN}`` is appended to
11042the flag. This expansion makes each ``LICENSE_FLAGS`` value
11043recipe-specific. After expansion, the string is then matched against the
11044whitelist. Thus, specifying ``LICENSE_FLAGS = "commercial"`` in recipe
11045"foo", for example, results in the string ``"commercial_foo"``. And, to
11046create a match, that string must appear in the whitelist.
11047
11048Judicious use of the ``LICENSE_FLAGS`` strings and the contents of the
11049``LICENSE_FLAGS_WHITELIST`` variable allows you a lot of flexibility for
11050including or excluding recipes based on licensing. For example, you can
11051broaden the matching capabilities by using license flags string subsets
11052in the whitelist.
11053
11054.. note::
11055
11056 When using a string subset, be sure to use the part of the expanded
11057 string that precedes the appended underscore character (e.g.
11058 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
11059
11060For example, simply specifying the string "commercial" in the whitelist
11061matches any expanded ``LICENSE_FLAGS`` definition that starts with the
11062string "commercial" such as "commercial_foo" and "commercial_bar", which
11063are the strings the build system automatically generates for
11064hypothetical recipes named "foo" and "bar" assuming those recipes simply
11065specify the following:
11066::
11067
11068 LICENSE_FLAGS = "commercial"
11069
11070Thus, you can choose
11071to exhaustively enumerate each license flag in the whitelist and allow
11072only specific recipes into the image, or you can use a string subset
11073that causes a broader range of matches to allow a range of recipes into
11074the image.
11075
11076This scheme works even if the ``LICENSE_FLAGS`` string already has
11077``_${PN}`` appended. For example, the build system turns the license
11078flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
11079both the general "commercial" and the specific "commercial_1.2_foo"
11080strings found in the whitelist, as expected.
11081
11082Here are some other scenarios:
11083
11084- You can specify a versioned string in the recipe such as
11085 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
11086 string to "commercial_foo_1.2_foo". Combine this license flag with a
11087 whitelist that has the string "commercial" and you match the flag
11088 along with any other flag that starts with the string "commercial".
11089
11090- Under the same circumstances, you can use "commercial_foo" in the
11091 whitelist and the build system not only matches "commercial_foo_1.2"
11092 but also matches any license flag with the string "commercial_foo",
11093 regardless of the version.
11094
11095- You can be very specific and use both the package and version parts
11096 in the whitelist (e.g. "commercial_foo_1.2") to specifically match a
11097 versioned recipe.
11098
11099Other Variables Related to Commercial Licenses
11100~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11101
11102Other helpful variables related to commercial license handling exist and
11103are defined in the
11104``poky/meta/conf/distro/include/default-distrovars.inc`` file:
11105::
11106
11107 COMMERCIAL_AUDIO_PLUGINS ?= ""
11108 COMMERCIAL_VIDEO_PLUGINS ?= ""
11109
11110If you
11111want to enable these components, you can do so by making sure you have
11112statements similar to the following in your ``local.conf`` configuration
11113file:
11114::
11115
11116 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
11117 gst-plugins-ugly-mpegaudioparse"
11118 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
11119 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
11120 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
11121
11122
11123Of course, you could also create a matching whitelist for those
11124components using the more general "commercial" in the whitelist, but
11125that would also enable all the other packages with ``LICENSE_FLAGS``
11126containing "commercial", which you may or may not want:
11127::
11128
11129 LICENSE_FLAGS_WHITELIST = "commercial"
11130
11131Specifying audio and video plugins as part of the
11132``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
11133(along with the enabling ``LICENSE_FLAGS_WHITELIST``) includes the
11134plugins or components into built images, thus adding support for media
11135formats or components.
11136
11137Maintaining Open Source License Compliance During Your Product's Lifecycle
11138--------------------------------------------------------------------------
11139
11140One of the concerns for a development organization using open source
11141software is how to maintain compliance with various open source
11142licensing during the lifecycle of the product. While this section does
11143not provide legal advice or comprehensively cover all scenarios, it does
11144present methods that you can use to assist you in meeting the compliance
11145requirements during a software release.
11146
11147With hundreds of different open source licenses that the Yocto Project
11148tracks, it is difficult to know the requirements of each and every
11149license. However, the requirements of the major FLOSS licenses can begin
11150to be covered by assuming that three main areas of concern exist:
11151
11152- Source code must be provided.
11153
11154- License text for the software must be provided.
11155
11156- Compilation scripts and modifications to the source code must be
11157 provided.
11158
11159- spdx files can be provided.
11160
11161There are other requirements beyond the scope of these three and the
11162methods described in this section (e.g. the mechanism through which
11163source code is distributed).
11164
11165As different organizations have different methods of complying with open
11166source licensing, this section is not meant to imply that there is only
11167one single way to meet your compliance obligations, but rather to
11168describe one method of achieving compliance. The remainder of this
11169section describes methods supported to meet the previously mentioned
11170three requirements. Once you take steps to meet these requirements, and
11171prior to releasing images, sources, and the build system, you should
11172audit all artifacts to ensure completeness.
11173
11174.. note::
11175
11176 The Yocto Project generates a license manifest during image creation
11177 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
11178 to assist with any audits.
11179
11180Providing the Source Code
11181~~~~~~~~~~~~~~~~~~~~~~~~~
11182
11183Compliance activities should begin before you generate the final image.
11184The first thing you should look at is the requirement that tops the list
11185for most compliance groups - providing the source. The Yocto Project has
11186a few ways of meeting this requirement.
11187
11188One of the easiest ways to meet this requirement is to provide the
11189entire :term:`DL_DIR` used by the
11190build. This method, however, has a few issues. The most obvious is the
11191size of the directory since it includes all sources used in the build
11192and not just the source used in the released image. It will include
11193toolchain source, and other artifacts, which you would not generally
11194release. However, the more serious issue for most companies is
11195accidental release of proprietary software. The Yocto Project provides
11196an :ref:`archiver <ref-classes-archiver>` class to
11197help avoid some of these concerns.
11198
11199Before you employ ``DL_DIR`` or the ``archiver`` class, you need to
11200decide how you choose to provide source. The source ``archiver`` class
11201can generate tarballs and SRPMs and can create them with various levels
11202of compliance in mind.
11203
11204One way of doing this (but certainly not the only way) is to release
11205just the source as a tarball. You can do this by adding the following to
11206the ``local.conf`` file found in the
11207:term:`Build Directory`:
11208::
11209
11210 INHERIT += "archiver"
11211 ARCHIVER_MODE[src] = "original"
11212
11213During the creation of your
11214image, the source from all recipes that deploy packages to the image is
11215placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
11216:term:`LICENSE` for each recipe.
11217Releasing the entire directory enables you to comply with requirements
11218concerning providing the unmodified source. It is important to note that
11219the size of the directory can get large.
11220
11221A way to help mitigate the size issue is to only release tarballs for
11222licenses that require the release of source. Let us assume you are only
11223concerned with GPL code as identified by running the following script:
11224
11225.. code-block:: shell
11226
11227 # Script to archive a subset of packages matching specific license(s)
11228 # Source and license files are copied into sub folders of package folder
11229 # Must be run from build folder
11230 #!/bin/bash
11231 src_release_dir="source-release"
11232 mkdir -p $src_release_dir
11233 for a in tmp/deploy/sources/*; do
11234 for d in $a/*; do
11235 # Get package name from path
11236 p=`basename $d`
11237 p=${p%-*}
11238 p=${p%-*}
11239 # Only archive GPL packages (update *GPL* regex for your license check)
11240 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
11241 if [ $numfiles -gt 1 ]; then
11242 echo Archiving $p
11243 mkdir -p $src_release_dir/$p/source
11244 cp $d/* $src_release_dir/$p/source 2> /dev/null
11245 mkdir -p $src_release_dir/$p/license
11246 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
11247 fi
11248 done
11249 done
11250
11251At this point, you
11252could create a tarball from the ``gpl_source_release`` directory and
11253provide that to the end user. This method would be a step toward
11254achieving compliance with section 3a of GPLv2 and with section 6 of
11255GPLv3.
11256
11257Providing License Text
11258~~~~~~~~~~~~~~~~~~~~~~
11259
11260One requirement that is often overlooked is inclusion of license text.
11261This requirement also needs to be dealt with prior to generating the
11262final image. Some licenses require the license text to accompany the
11263binary. You can achieve this by adding the following to your
11264``local.conf`` file:
11265::
11266
11267 COPY_LIC_MANIFEST = "1"
11268 COPY_LIC_DIRS = "1"
11269 LICENSE_CREATE_PACKAGE = "1"
11270
11271Adding these statements to the
11272configuration file ensures that the licenses collected during package
11273generation are included on your image.
11274
11275.. note::
11276
11277 Setting all three variables to "1" results in the image having two
11278 copies of the same license file. One copy resides in
11279 ``/usr/share/common-licenses`` and the other resides in
11280 ``/usr/share/license``.
11281
11282 The reason for this behavior is because
11283 :term:`COPY_LIC_DIRS` and
11284 :term:`COPY_LIC_MANIFEST`
11285 add a copy of the license when the image is built but do not offer a
11286 path for adding licenses for newly installed packages to an image.
11287 :term:`LICENSE_CREATE_PACKAGE`
11288 adds a separate package and an upgrade path for adding licenses to an
11289 image.
11290
11291As the source ``archiver`` class has already archived the original
11292unmodified source that contains the license files, you would have
11293already met the requirements for inclusion of the license information
11294with source as defined by the GPL and other open source licenses.
11295
11296Providing Compilation Scripts and Source Code Modifications
11297~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11298
11299At this point, we have addressed all we need to prior to generating the
11300image. The next two requirements are addressed during the final
11301packaging of the release.
11302
11303By releasing the version of the OpenEmbedded build system and the layers
11304used during the build, you will be providing both compilation scripts
11305and the source code modifications in one step.
11306
11307If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
11308and a distro layer, and those
11309those layers are used to patch, compile, package, or modify (in any way)
11310any open source software included in your released images, you might be
11311required to release those layers under section 3 of GPLv2 or section 1
11312of GPLv3. One way of doing that is with a clean checkout of the version
11313of the Yocto Project and layers used during your build. Here is an
11314example:
11315
11316.. code-block:: shell
11317
11318 # We built using the dunfell branch of the poky repo
11319 $ git clone -b dunfell git://git.yoctoproject.org/poky
11320 $ cd poky
11321 # We built using the release_branch for our layers
11322 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11323 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11324 # clean up the .git repos
11325 $ find . -name ".git" -type d -exec rm -rf {} \;
11326
11327One
11328thing a development organization might want to consider for end-user
11329convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to
11330ensure that when the end user utilizes the released build system to
11331build an image, the development organization's layers are included in
11332the ``bblayers.conf`` file automatically:
11333::
11334
11335 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11336 # changes incompatibly
11337 POKY_BBLAYERS_CONF_VERSION = "2"
11338
11339 BBPATH = "${TOPDIR}"
11340 BBFILES ?= ""
11341
11342 BBLAYERS ?= " \
11343 ##OEROOT##/meta \
11344 ##OEROOT##/meta-poky \
11345 ##OEROOT##/meta-yocto-bsp \
11346 ##OEROOT##/meta-mylayer \
11347 "
11348
11349Creating and
11350providing an archive of the :term:`Metadata`
11351layers (recipes, configuration files, and so forth) enables you to meet
11352your requirements to include the scripts to control compilation as well
11353as any modifications to the original source.
11354
11355Providing spdx files
11356~~~~~~~~~~~~~~~~~~~~~~~~~
11357
11358The spdx module has been integrated to a layer named meta-spdxscanner.
11359meta-spdxscanner provides several kinds of scanner. If you want to enable
11360this function, you have to follow the following steps:
11361
113621. Add meta-spdxscanner layer into ``bblayers.conf``.
11363
113642. Refer to the README in meta-spdxscanner to setup the environment (e.g,
11365 setup a fossology server) needed for the scanner.
11366
113673. Meta-spdxscanner provides several methods within the bbclass to create spdx files.
11368 Please choose one that you want to use and enable the spdx task. You have to
11369 add some config options in ``local.conf`` file in your :term:`Build
11370 Directory`. The following is an example showing how to generate spdx files
11371 during bitbake using the fossology-python.bbclass::
11372
11373 # Select fossology-python.bbclass.
11374 INHERIT += "fossology-python"
11375 # For fossology-python.bbclass, TOKEN is necessary, so, after setup a
11376 # Fossology server, you have to create a token.
11377 TOKEN = "eyJ0eXAiO..."
11378 # The fossology server is necessary for fossology-python.bbclass.
11379 FOSSOLOGY_SERVER = "http://xx.xx.xx.xx:8081/repo"
11380 # If you want to upload the source code to a special folder:
11381 FOLDER_NAME = "xxxx" //Optional
11382 # If you don't want to put spdx files in tmp/deploy/spdx, you can enable:
11383 SPDX_DEPLOY_DIR = "${DEPLOY_DIR}" //Optional
11384
11385For more usage information refer to :yocto_git:`the meta-spdxscanner repository
11386</meta-spdxscanner/>`.
11387
11388
11389Copying Licenses that Do Not Exist
11390----------------------------------
11391
11392Some packages, such as the linux-firmware package, have many licenses
11393that are not in any way common. You can avoid adding a lot of these
11394types of common license files, which are only applicable to a specific
11395package, by using the
11396:term:`NO_GENERIC_LICENSE`
11397variable. Using this variable also avoids QA errors when you use a
11398non-common, non-CLOSED license in a recipe.
11399
11400The following is an example that uses the ``LICENSE.Abilis.txt`` file as
11401the license from the fetched source:
11402::
11403
11404 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11405
11406Using the Error Reporting Tool
11407==============================
11408
11409The error reporting tool allows you to submit errors encountered during
11410builds to a central database. Outside of the build environment, you can
11411use a web interface to browse errors, view statistics, and query for
11412errors. The tool works using a client-server system where the client
11413portion is integrated with the installed Yocto Project
11414:term:`Source Directory` (e.g. ``poky``).
11415The server receives the information collected and saves it in a
11416database.
11417
11418A live instance of the error reporting server exists at
11419https://errors.yoctoproject.org. This server exists so that when
11420you want to get help with build failures, you can submit all of the
11421information on the failure easily and then point to the URL in your bug
11422report or send an email to the mailing list.
11423
11424.. note::
11425
11426 If you send error reports to this server, the reports become publicly
11427 visible.
11428
11429Enabling and Using the Tool
11430---------------------------
11431
11432By default, the error reporting tool is disabled. You can enable it by
11433inheriting the
11434:ref:`report-error <ref-classes-report-error>`
11435class by adding the following statement to the end of your
11436``local.conf`` file in your
11437:term:`Build Directory`.
11438::
11439
11440 INHERIT += "report-error"
11441
11442By default, the error reporting feature stores information in
11443``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11444However, you can specify a directory to use by adding the following to
11445your ``local.conf`` file:
11446::
11447
11448 ERR_REPORT_DIR = "path"
11449
11450Enabling error
11451reporting causes the build process to collect the errors and store them
11452in a file as previously described. When the build system encounters an
11453error, it includes a command as part of the console output. You can run
11454the command to send the error file to the server. For example, the
11455following command sends the errors to an upstream server:
11456::
11457
11458 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11459
11460In the previous example, the errors are sent to a public database
11461available at https://errors.yoctoproject.org, which is used by the
11462entire community. If you specify a particular server, you can send the
11463errors to a different database. Use the following command for more
11464information on available options:
11465::
11466
11467 $ send-error-report --help
11468
11469When sending the error file, you are prompted to review the data being
11470sent as well as to provide a name and optional email address. Once you
11471satisfy these prompts, the command returns a link from the server that
11472corresponds to your entry in the database. For example, here is a
11473typical link: https://errors.yoctoproject.org/Errors/Details/9522/
11474
11475Following the link takes you to a web interface where you can browse,
11476query the errors, and view statistics.
11477
11478Disabling the Tool
11479------------------
11480
11481To disable the error reporting feature, simply remove or comment out the
11482following statement from the end of your ``local.conf`` file in your
11483:term:`Build Directory`.
11484::
11485
11486 INHERIT += "report-error"
11487
11488Setting Up Your Own Error Reporting Server
11489------------------------------------------
11490
11491If you want to set up your own error reporting server, you can obtain
11492the code from the Git repository at :yocto_git:`/error-report-web/`.
11493Instructions on how to set it up are in the README document.
11494
11495Using Wayland and Weston
11496========================
11497
11498`Wayland <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__
11499is a computer display server protocol that provides a method for
11500compositing window managers to communicate directly with applications
11501and video hardware and expects them to communicate with input hardware
11502using other libraries. Using Wayland with supporting targets can result
11503in better control over graphics frame rendering than an application
11504might otherwise achieve.
11505
11506The Yocto Project provides the Wayland protocol libraries and the
11507reference
11508`Weston <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__
11509compositor as part of its release. You can find the integrated packages
11510in the ``meta`` layer of the :term:`Source Directory`.
11511Specifically, you
11512can find the recipes that build both Wayland and Weston at
11513``meta/recipes-graphics/wayland``.
11514
11515You can build both the Wayland and Weston packages for use only with
11516targets that accept the `Mesa 3D and Direct Rendering
11517Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__,
11518which is also known as Mesa DRI. This implies that you cannot build and
11519use the packages if your target uses, for example, the Intel Embedded
11520Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11521
11522.. note::
11523
11524 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11525 emulated QEMU hardware. However, this version of Weston will run
11526 under X emulation without issues.
11527
11528This section describes what you need to do to implement Wayland and use
11529the Weston compositor when building an image for a supporting target.
11530
11531Enabling Wayland in an Image
11532----------------------------
11533
11534To enable Wayland, you need to enable it to be built and enable it to be
11535included (installed) in the image.
11536
11537Building Wayland
11538~~~~~~~~~~~~~~~~
11539
11540To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11541Wayland with Kernel Mode Setting
11542(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11543support, include the "wayland" flag in the
11544:term:`DISTRO_FEATURES`
11545statement in your ``local.conf`` file:
11546::
11547
11548 DISTRO_FEATURES_append = " wayland"
11549
11550.. note::
11551
11552 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11553 support
11554
11555Installing Wayland and Weston
11556~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11557
11558To install the Wayland feature into an image, you must include the
11559following
11560:term:`CORE_IMAGE_EXTRA_INSTALL`
11561statement in your ``local.conf`` file:
11562::
11563
11564 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11565
11566Running Weston
11567--------------
11568
11569To run Weston inside X11, enabling it as described earlier and building
11570a Sato image is sufficient. If you are running your image under Sato, a
11571Weston Launcher appears in the "Utility" category.
11572
11573Alternatively, you can run Weston through the command-line interpretor
11574(CLI), which is better suited for development work. To run Weston under
11575the CLI, you need to do the following after your image is built:
11576
115771. Run these commands to export ``XDG_RUNTIME_DIR``:
11578 ::
11579
11580 mkdir -p /tmp/$USER-weston
11581 chmod 0700 /tmp/$USER-weston
11582 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11583
115842. Launch Weston in the shell:
11585 ::
11586
11587 weston