summaryrefslogtreecommitdiffstats
path: root/documentation/dev-manual/common-tasks.rst
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/dev-manual/common-tasks.rst')
-rw-r--r--documentation/dev-manual/common-tasks.rst11861
1 files changed, 0 insertions, 11861 deletions
diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
deleted file mode 100644
index 3ba64e1477..0000000000
--- a/documentation/dev-manual/common-tasks.rst
+++ /dev/null
@@ -1,11861 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
21":ref:`overview-manual/yp-intro:the yocto project layer model`"
22section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
27.. note::
28
29 It is very easy to create your own layers to use with the OpenEmbedded
30 build system, as the Yocto Project ships with tools that speed up creating
31 layers. This section describes the steps you perform by hand to create
32 layers so that you can better understand them. For information about the
33 layer-creation tools, see the
34 ":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
35 section in the Yocto Project Board Support Package (BSP) Developer's
36 Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
37 section further down in this manual.
38
39Follow these general steps to create your layer without using tools:
40
411. *Check Existing Layers:* Before creating a new layer, you should be
42 sure someone has not already created a layer containing the Metadata
43 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
44 for a list of layers from the OpenEmbedded community that can be used in
45 the Yocto Project. You could find a layer that is identical or close
46 to what you need.
47
482. *Create a Directory:* Create the directory for your layer. When you
49 create the layer, be sure to create the directory in an area not
50 associated with the Yocto Project :term:`Source Directory`
51 (e.g. the cloned ``poky`` repository).
52
53 While not strictly required, prepend the name of the directory with
54 the string "meta-". For example::
55
56 meta-mylayer
57 meta-GUI_xyz
58 meta-mymachine
59
60 With rare exceptions, a layer's name follows this form::
61
62 meta-root_name
63
64 Following this layer naming convention can save
65 you trouble later when tools, components, or variables "assume" your
66 layer name begins with "meta-". A notable example is in configuration
67 files as shown in the following step where layer names without the
68 "meta-" string are appended to several variables used in the
69 configuration.
70
713. *Create a Layer Configuration File:* Inside your new layer folder,
72 you need to create a ``conf/layer.conf`` file. It is easiest to take
73 an existing layer configuration file and copy that to your layer's
74 ``conf`` directory and then modify the file as needed.
75
76 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
77 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
78 demonstrates the required syntax. For your layer, you need to replace
79 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
80 for a layer named "meta-machinexyz")::
81
82 # We have a conf and classes directory, add to BBPATH
83 BBPATH .= ":${LAYERDIR}"
84
85 # We have recipes-* directories, add to BBFILES
86 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
87 ${LAYERDIR}/recipes-*/*/*.bbappend"
88
89 BBFILE_COLLECTIONS += "yoctobsp"
90 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
91 BBFILE_PRIORITY_yoctobsp = "5"
92 LAYERVERSION_yoctobsp = "4"
93 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
94
95 Following is an explanation of the layer configuration file:
96
97 - :term:`BBPATH`: Adds the layer's
98 root directory to BitBake's search path. Through the use of the
99 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
100 configuration files, and files that are included with ``include``
101 and ``require`` statements. For these cases, BitBake uses the
102 first file that matches the name found in :term:`BBPATH`. This is
103 similar to the way the ``PATH`` variable is used for binaries. It
104 is recommended, therefore, that you use unique class and
105 configuration filenames in your custom layer.
106
107 - :term:`BBFILES`: Defines the
108 location for all recipes in the layer.
109
110 - :term:`BBFILE_COLLECTIONS`:
111 Establishes the current layer through a unique identifier that is
112 used throughout the OpenEmbedded build system to refer to the
113 layer. In this example, the identifier "yoctobsp" is the
114 representation for the container layer named "meta-yocto-bsp".
115
116 - :term:`BBFILE_PATTERN`:
117 Expands immediately during parsing to provide the directory of the
118 layer.
119
120 - :term:`BBFILE_PRIORITY`:
121 Establishes a priority to use for recipes in the layer when the
122 OpenEmbedded build finds recipes of the same name in different
123 layers.
124
125 - :term:`LAYERVERSION`:
126 Establishes a version number for the layer. You can use this
127 version number to specify this exact version of the layer as a
128 dependency when using the
129 :term:`LAYERDEPENDS`
130 variable.
131
132 - :term:`LAYERDEPENDS`:
133 Lists all layers on which this layer depends (if any).
134
135 - :term:`LAYERSERIES_COMPAT`:
136 Lists the :yocto_wiki:`Yocto Project </Releases>`
137 releases for which the current version is compatible. This
138 variable is a good way to indicate if your particular layer is
139 current.
140
1414. *Add Content:* Depending on the type of layer, add the content. If
142 the layer adds support for a machine, add the machine configuration
143 in a ``conf/machine/`` file within the layer. If the layer adds
144 distro policy, add the distro configuration in a ``conf/distro/``
145 file within the layer. If the layer introduces new recipes, put the
146 recipes you need in ``recipes-*`` subdirectories within the layer.
147
148 .. note::
149
150 For an explanation of layer hierarchy that is compliant with the
151 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
152 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
153
1545. *Optionally Test for Compatibility:* If you want permission to use
155 the Yocto Project Compatibility logo with your layer or application
156 that uses your layer, perform the steps to apply for compatibility.
157 See the
158 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
159 section for more information.
160
161Following Best Practices When Creating Layers
162---------------------------------------------
163
164To create layers that are easier to maintain and that will not impact
165builds for other machines, you should consider the information in the
166following list:
167
168- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
169 Configuration:* In other words, do not copy an entire recipe into
170 your layer and then modify it. Rather, use an append file
171 (``.bbappend``) to override only those parts of the original recipe
172 you need to modify.
173
174- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
175 for each recipe that uses an include file. Or, if you are introducing
176 a new recipe that requires the included file, use the path relative
177 to the original layer directory to refer to the file. For example,
178 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
179 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
180 the include file, it could indicate a deficiency in the include file
181 in the layer to which it originally belongs. If this is the case, you
182 should try to address that deficiency instead of overlaying the
183 include file. For example, you could address this by getting the
184 maintainer of the include file to add a variable or variables to make
185 it easy to override the parts needing to be overridden.
186
187- *Structure Your Layers:* Proper use of overrides within append files
188 and placement of machine-specific files within your layer can ensure
189 that a build is not using the wrong Metadata and negatively impacting
190 a build for a different machine. Following are some examples:
191
192 - *Modify Variables to Support a Different Machine:* Suppose you
193 have a layer named ``meta-one`` that adds support for building
194 machine "one". To do so, you use an append file named
195 ``base-files.bbappend`` and create a dependency on "foo" by
196 altering the :term:`DEPENDS`
197 variable::
198
199 DEPENDS = "foo"
200
201 The dependency is created during any
202 build that includes the layer ``meta-one``. However, you might not
203 want this dependency for all machines. For example, suppose you
204 are building for machine "two" but your ``bblayers.conf`` file has
205 the ``meta-one`` layer included. During the build, the
206 ``base-files`` for machine "two" will also have the dependency on
207 ``foo``.
208
209 To make sure your changes apply only when building machine "one",
210 use a machine override with the :term:`DEPENDS` statement::
211
212 DEPENDS:one = "foo"
213
214 You should follow the same strategy when using ``:append``
215 and ``:prepend`` operations::
216
217 DEPENDS:append:one = " foo"
218 DEPENDS:prepend:one = "foo "
219
220 As an actual example, here's a
221 snippet from the generic kernel include file ``linux-yocto.inc``,
222 wherein the kernel compile and link options are adjusted in the
223 case of a subset of the supported architectures::
224
225 DEPENDS:append:aarch64 = " libgcc"
226 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
227 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
228
229 DEPENDS:append:nios2 = " libgcc"
230 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
231 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
232
233 DEPENDS:append:arc = " libgcc"
234 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
235 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
236
237 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
238
239 - *Place Machine-Specific Files in Machine-Specific Locations:* When
240 you have a base recipe, such as ``base-files.bb``, that contains a
241 :term:`SRC_URI` statement to a
242 file, you can use an append file to cause the build to use your
243 own version of the file. For example, an append file in your layer
244 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
245 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
246
247 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
248
249 The build for machine "one" will pick up your machine-specific file as
250 long as you have the file in
251 ``meta-one/recipes-core/base-files/base-files/``. However, if you
252 are building for a different machine and the ``bblayers.conf``
253 file includes the ``meta-one`` layer and the location of your
254 machine-specific file is the first location where that file is
255 found according to :term:`FILESPATH`, builds for all machines will
256 also use that machine-specific file.
257
258 You can make sure that a machine-specific file is used for a
259 particular machine by putting the file in a subdirectory specific
260 to the machine. For example, rather than placing the file in
261 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
262 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
263 Not only does this make sure the file is used only when building
264 for machine "one", but the build process locates the file more
265 quickly.
266
267 In summary, you need to place all files referenced from
268 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
269 order to restrict those files to machine-specific builds.
270
271- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
272 permission to use the Yocto Project Compatibility logo with your
273 layer or application that uses your layer, perform the steps to apply
274 for compatibility. See the
275 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
276 section for more information.
277
278- *Follow the Layer Naming Convention:* Store custom layers in a Git
279 repository that use the ``meta-layer_name`` format.
280
281- *Group Your Layers Locally:* Clone your repository alongside other
282 cloned ``meta`` directories from the :term:`Source Directory`.
283
284Making Sure Your Layer is Compatible With Yocto Project
285-------------------------------------------------------
286
287When you create a layer used with the Yocto Project, it is advantageous
288to make sure that the layer interacts well with existing Yocto Project
289layers (i.e. the layer is compatible with the Yocto Project). Ensuring
290compatibility makes the layer easy to be consumed by others in the Yocto
291Project community and could allow you permission to use the Yocto
292Project Compatible Logo.
293
294.. note::
295
296 Only Yocto Project member organizations are permitted to use the
297 Yocto Project Compatible Logo. The logo is not available for general
298 use. For information on how to become a Yocto Project member
299 organization, see the :yocto_home:`Yocto Project Website <>`.
300
301The Yocto Project Compatibility Program consists of a layer application
302process that requests permission to use the Yocto Project Compatibility
303Logo for your layer and application. The process consists of two parts:
304
3051. Successfully passing a script (``yocto-check-layer``) that when run
306 against your layer, tests it against constraints based on experiences
307 of how layers have worked in the real world and where pitfalls have
308 been found. Getting a "PASS" result from the script is required for
309 successful compatibility registration.
310
3112. Completion of an application acceptance form, which you can find at
312 :yocto_home:`/webform/yocto-project-compatible-registration`.
313
314To be granted permission to use the logo, you need to satisfy the
315following:
316
317- Be able to check the box indicating that you got a "PASS" when
318 running the script against your layer.
319
320- Answer "Yes" to the questions on the form or have an acceptable
321 explanation for any questions answered "No".
322
323- Be a Yocto Project Member Organization.
324
325The remainder of this section presents information on the registration
326form and on the ``yocto-check-layer`` script.
327
328Yocto Project Compatible Program Application
329~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
330
331Use the form to apply for your layer's approval. Upon successful
332application, you can use the Yocto Project Compatibility Logo with your
333layer and the application that uses your layer.
334
335To access the form, use this link:
336:yocto_home:`/webform/yocto-project-compatible-registration`.
337Follow the instructions on the form to complete your application.
338
339The application consists of the following sections:
340
341- *Contact Information:* Provide your contact information as the fields
342 require. Along with your information, provide the released versions
343 of the Yocto Project for which your layer is compatible.
344
345- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
346 items in the checklist. There is space at the bottom of the form for
347 any explanations for items for which you answered "No".
348
349- *Recommendations:* Provide answers for the questions regarding Linux
350 kernel use and build success.
351
352``yocto-check-layer`` Script
353~~~~~~~~~~~~~~~~~~~~~~~~~~~~
354
355The ``yocto-check-layer`` script provides you a way to assess how
356compatible your layer is with the Yocto Project. You should run this
357script prior to using the form to apply for compatibility as described
358in the previous section. You need to achieve a "PASS" result in order to
359have your application form successfully processed.
360
361The script divides tests into three areas: COMMON, BSP, and DISTRO. For
362example, given a distribution layer (DISTRO), the layer must pass both
363the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
364layer, the layer must pass the COMMON and BSP set of tests.
365
366To execute the script, enter the following commands from your build
367directory::
368
369 $ source oe-init-build-env
370 $ yocto-check-layer your_layer_directory
371
372Be sure to provide the actual directory for your
373layer as part of the command.
374
375Entering the command causes the script to determine the type of layer
376and then to execute a set of specific tests against the layer. The
377following list overviews the test:
378
379- ``common.test_readme``: Tests if a ``README`` file exists in the
380 layer and the file is not empty.
381
382- ``common.test_parse``: Tests to make sure that BitBake can parse the
383 files without error (i.e. ``bitbake -p``).
384
385- ``common.test_show_environment``: Tests that the global or per-recipe
386 environment is in order without errors (i.e. ``bitbake -e``).
387
388- ``common.test_world``: Verifies that ``bitbake world`` works.
389
390- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
391 layers do not come with recipes that change signatures.
392
393- ``common.test_layerseries_compat``: Verifies layer compatibility is
394 set properly.
395
396- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
397 configurations.
398
399- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
400 set the machine when the layer is added.
401
402- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
403 regardless of which machine is selected.
404
405- ``bsp.test_machine_signatures``: Verifies that building for a
406 particular machine affects only the signature of tasks specific to
407 that machine.
408
409- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
410 distro configurations.
411
412- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
413 does not set the distribution when the layer is added.
414
415Enabling Your Layer
416-------------------
417
418Before the OpenEmbedded build system can use your new layer, you need to
419enable it. To enable your layer, simply add your layer's path to the
420:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
421found in the :term:`Build Directory`. The following example shows how to
422enable your new ``meta-mylayer`` layer (note how your new layer exists
423outside of the official ``poky`` repository which you would have checked
424out earlier)::
425
426 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
427 # changes incompatibly
428 POKY_BBLAYERS_CONF_VERSION = "2"
429 BBPATH = "${TOPDIR}"
430 BBFILES ?= ""
431 BBLAYERS ?= " \
432 /home/user/poky/meta \
433 /home/user/poky/meta-poky \
434 /home/user/poky/meta-yocto-bsp \
435 /home/user/mystuff/meta-mylayer \
436 "
437
438BitBake parses each ``conf/layer.conf`` file from the top down as
439specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
440file. During the processing of each ``conf/layer.conf`` file, BitBake
441adds the recipes, classes and configurations contained within the
442particular layer to the source directory.
443
444Appending Other Layers Metadata With Your Layer
445-----------------------------------------------
446
447A recipe that appends Metadata to another recipe is called a BitBake
448append file. A BitBake append file uses the ``.bbappend`` file type
449suffix, while the corresponding recipe to which Metadata is being
450appended uses the ``.bb`` file type suffix.
451
452You can use a ``.bbappend`` file in your layer to make additions or
453changes to the content of another layer's recipe without having to copy
454the other layer's recipe into your layer. Your ``.bbappend`` file
455resides in your layer, while the main ``.bb`` recipe file to which you
456are appending Metadata resides in a different layer.
457
458Being able to append information to an existing recipe not only avoids
459duplication, but also automatically applies recipe changes from a
460different layer into your layer. If you were copying recipes, you would
461have to manually merge changes as they occur.
462
463When you create an append file, you must use the same root name as the
464corresponding recipe file. For example, the append file
465``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
466means the original recipe and append filenames are version
467number-specific. If the corresponding recipe is renamed to update to a
468newer version, you must also rename and possibly update the
469corresponding ``.bbappend`` as well. During the build process, BitBake
470displays an error on starting if it detects a ``.bbappend`` file that
471does not have a corresponding recipe with a matching name. See the
472:term:`BB_DANGLINGAPPENDS_WARNONLY`
473variable for information on how to handle this error.
474
475Overlaying a File Using Your Layer
476~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
477
478As an example, consider the main formfactor recipe and a corresponding
479formfactor append file both from the :term:`Source Directory`.
480Here is the main
481formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
482the "meta" layer at ``meta/recipes-bsp/formfactor``::
483
484 SUMMARY = "Device formfactor information"
485 DESCRIPTION = "A formfactor configuration file provides information about the \
486 target hardware for which the image is being built and information that the \
487 build system cannot obtain from other sources such as the kernel."
488 SECTION = "base"
489 LICENSE = "MIT"
490 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
491 PR = "r45"
492
493 SRC_URI = "file://config file://machconfig"
494 S = "${WORKDIR}"
495
496 PACKAGE_ARCH = "${MACHINE_ARCH}"
497 INHIBIT_DEFAULT_DEPS = "1"
498
499 do_install() {
500 # Install file only if it has contents
501 install -d ${D}${sysconfdir}/formfactor/
502 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
503 if [ -s "${S}/machconfig" ]; then
504 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
505 fi
506 }
507
508In the main recipe, note the :term:`SRC_URI`
509variable, which tells the OpenEmbedded build system where to find files
510during the build.
511
512Following is the append file, which is named ``formfactor_0.0.bbappend``
513and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
514file is in the layer at ``recipes-bsp/formfactor``::
515
516 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
517
518By default, the build system uses the
519:term:`FILESPATH` variable to
520locate files. This append file extends the locations by setting the
521:term:`FILESEXTRAPATHS`
522variable. Setting this variable in the ``.bbappend`` file is the most
523reliable and recommended method for adding directories to the search
524path used by the build system to find files.
525
526The statement in this example extends the directories to include
527``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
528which resolves to a directory named ``formfactor`` in the same directory
529in which the append file resides (i.e.
530``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
531have the supporting directory structure set up that will contain any
532files or patches you will be including from the layer.
533
534Using the immediate expansion assignment operator ``:=`` is important
535because of the reference to :term:`THISDIR`. The trailing colon character is
536important as it ensures that items in the list remain colon-separated.
537
538.. note::
539
540 BitBake automatically defines the :term:`THISDIR` variable. You should
541 never set this variable yourself. Using ":prepend" as part of the
542 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
543 paths in the final list.
544
545 Also, not all append files add extra files. Many append files simply
546 allow to add build options (e.g. ``systemd``). For these cases, your
547 append file would not even use the :term:`FILESEXTRAPATHS` statement.
548
549The end result of this ``.bbappend`` file is that on a Raspberry Pi, where
550``rpi`` will exist in the list of :term:`OVERRIDES`, the file
551``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be
552used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in
553:ref:`ref-tasks-install` will return true, and the file will be installed.
554
555Installing Additional Files Using Your Layer
556~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
557
558As another example, consider the main ``xserver-xf86-config`` recipe and a
559corresponding ``xserver-xf86-config`` append file both from the :term:`Source
560Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named
561``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at
562``meta/recipes-graphics/xorg-xserver``::
563
564 SUMMARY = "X.Org X server configuration file"
565 HOMEPAGE = "http://www.x.org"
566 SECTION = "x11/base"
567 LICENSE = "MIT"
568 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
569 PR = "r33"
570
571 SRC_URI = "file://xorg.conf"
572
573 S = "${WORKDIR}"
574
575 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf"
576
577 PACKAGE_ARCH = "${MACHINE_ARCH}"
578 ALLOW_EMPTY:${PN} = "1"
579
580 do_install () {
581 if test -s ${WORKDIR}/xorg.conf; then
582 install -d ${D}/${sysconfdir}/X11
583 install -m 0644 ${WORKDIR}/xorg.conf ${D}/${sysconfdir}/X11/
584 fi
585 }
586
587Following is the append file, which is named ``xserver-xf86-config_%.bbappend``
588and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
589file is in the layer at ``recipes-graphics/xorg-xserver``::
590
591 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
592
593 SRC_URI:append:rpi = " \
594 file://xorg.conf.d/98-pitft.conf \
595 file://xorg.conf.d/99-calibration.conf \
596 "
597 do_install:append:rpi () {
598 PITFT="${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}"
599 if [ "${PITFT}" = "1" ]; then
600 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/
601 install -m 0644 ${WORKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
602 install -m 0644 ${WORKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
603 fi
604 }
605
606 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*"
607
608Building off of the previous example, we once again are setting the
609:term:`FILESEXTRAPATHS` variable. In this case we are also using
610:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in
611the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a
612check for an additional :term:`MACHINE_FEATURES` that if set will cause these
613additional files to be installed. These additional files are listed in
614:term:`FILES` so that they will be packaged.
615
616Prioritizing Your Layer
617-----------------------
618
619Each layer is assigned a priority value. Priority values control which
620layer takes precedence if there are recipe files with the same name in
621multiple layers. For these cases, the recipe file from the layer with a
622higher priority number takes precedence. Priority values also affect the
623order in which multiple ``.bbappend`` files for the same recipe are
624applied. You can either specify the priority manually, or allow the
625build system to calculate it based on the layer's dependencies.
626
627To specify the layer's priority manually, use the
628:term:`BBFILE_PRIORITY`
629variable and append the layer's root name::
630
631 BBFILE_PRIORITY_mylayer = "1"
632
633.. note::
634
635 It is possible for a recipe with a lower version number
636 :term:`PV` in a layer that has a higher
637 priority to take precedence.
638
639 Also, the layer priority does not currently affect the precedence
640 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
641 might address this.
642
643Managing Layers
644---------------
645
646You can use the BitBake layer management tool ``bitbake-layers`` to
647provide a view into the structure of recipes across a multi-layer
648project. Being able to generate output that reports on configured layers
649with their paths and priorities and on ``.bbappend`` files and their
650applicable recipes can help to reveal potential problems.
651
652For help on the BitBake layer management tool, use the following
653command::
654
655 $ bitbake-layers --help
656
657The following list describes the available commands:
658
659- ``help:`` Displays general help or help on a specified command.
660
661- ``show-layers:`` Shows the current configured layers.
662
663- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
664 when a recipe with the same name exists in another layer that has a
665 higher layer priority.
666
667- ``show-recipes:`` Lists available recipes and the layers that
668 provide them.
669
670- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
671 which they apply.
672
673- ``show-cross-depends:`` Lists dependency relationships between
674 recipes that cross layer boundaries.
675
676- ``add-layer:`` Adds a layer to ``bblayers.conf``.
677
678- ``remove-layer:`` Removes a layer from ``bblayers.conf``
679
680- ``flatten:`` Flattens the layer configuration into a separate
681 output directory. Flattening your layer configuration builds a
682 "flattened" directory that contains the contents of all layers, with
683 any overlayed recipes removed and any ``.bbappend`` files appended to
684 the corresponding recipes. You might have to perform some manual
685 cleanup of the flattened layer as follows:
686
687 - Non-recipe files (such as patches) are overwritten. The flatten
688 command shows a warning for these files.
689
690 - Anything beyond the normal layer setup has been added to the
691 ``layer.conf`` file. Only the lowest priority layer's
692 ``layer.conf`` is used.
693
694 - Overridden and appended items from ``.bbappend`` files need to be
695 cleaned up. The contents of each ``.bbappend`` end up in the
696 flattened recipe. However, if there are appended or changed
697 variable values, you need to tidy these up yourself. Consider the
698 following example. Here, the ``bitbake-layers`` command adds the
699 line ``#### bbappended ...`` so that you know where the following
700 lines originate::
701
702 ...
703 DESCRIPTION = "A useful utility"
704 ...
705 EXTRA_OECONF = "--enable-something"
706 ...
707
708 #### bbappended from meta-anotherlayer ####
709
710 DESCRIPTION = "Customized utility"
711 EXTRA_OECONF += "--enable-somethingelse"
712
713
714 Ideally, you would tidy up these utilities as follows::
715
716 ...
717 DESCRIPTION = "Customized utility"
718 ...
719 EXTRA_OECONF = "--enable-something --enable-somethingelse"
720 ...
721
722- ``layerindex-fetch``: Fetches a layer from a layer index, along
723 with its dependent layers, and adds the layers to the
724 ``conf/bblayers.conf`` file.
725
726- ``layerindex-show-depends``: Finds layer dependencies from the
727 layer index.
728
729- ``save-build-conf``: Saves the currently active build configuration
730 (``conf/local.conf``, ``conf/bblayers.conf``) as a template into a layer.
731 This template can later be used for setting up builds via :term:``TEMPLATECONF``.
732 For information about saving and using configuration templates, see
733 ":ref:`dev-manual/common-tasks:creating a custom template configuration directory`".
734
735- ``create-layer``: Creates a basic layer.
736
737- ``create-layers-setup``: Writes out a configuration file and/or a script that
738 can replicate the directory structure and revisions of the layers in a current build.
739 For more information, see ":ref:`dev-manual/common-tasks:saving and restoring the layers setup`".
740
741Creating a General Layer Using the ``bitbake-layers`` Script
742------------------------------------------------------------
743
744The ``bitbake-layers`` script with the ``create-layer`` subcommand
745simplifies creating a new general layer.
746
747.. note::
748
749 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
750 section in the Yocto
751 Project Board Specific (BSP) Developer's Guide.
752
753 - In order to use a layer with the OpenEmbedded build system, you
754 need to add the layer to your ``bblayers.conf`` configuration
755 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
756 section for more information.
757
758The default mode of the script's operation with this subcommand is to
759create a layer with the following:
760
761- A layer priority of 6.
762
763- A ``conf`` subdirectory that contains a ``layer.conf`` file.
764
765- A ``recipes-example`` subdirectory that contains a further
766 subdirectory named ``example``, which contains an ``example.bb``
767 recipe file.
768
769- A ``COPYING.MIT``, which is the license statement for the layer. The
770 script assumes you want to use the MIT license, which is typical for
771 most layers, for the contents of the layer itself.
772
773- A ``README`` file, which is a file describing the contents of your
774 new layer.
775
776In its simplest form, you can use the following command form to create a
777layer. The command creates a layer whose name corresponds to
778"your_layer_name" in the current directory::
779
780 $ bitbake-layers create-layer your_layer_name
781
782As an example, the following command creates a layer named ``meta-scottrif``
783in your home directory::
784
785 $ cd /usr/home
786 $ bitbake-layers create-layer meta-scottrif
787 NOTE: Starting bitbake server...
788 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
789
790If you want to set the priority of the layer to other than the default
791value of "6", you can either use the ``--priority`` option or you
792can edit the
793:term:`BBFILE_PRIORITY` value
794in the ``conf/layer.conf`` after the script creates it. Furthermore, if
795you want to give the example recipe file some name other than the
796default, you can use the ``--example-recipe-name`` option.
797
798The easiest way to see how the ``bitbake-layers create-layer`` command
799works is to experiment with the script. You can also read the usage
800information by entering the following::
801
802 $ bitbake-layers create-layer --help
803 NOTE: Starting bitbake server...
804 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
805 [--example-recipe-name EXAMPLERECIPE]
806 layerdir
807
808 Create a basic layer
809
810 positional arguments:
811 layerdir Layer directory to create
812
813 optional arguments:
814 -h, --help show this help message and exit
815 --priority PRIORITY, -p PRIORITY
816 Layer directory to create
817 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
818 Filename of the example recipe
819
820Adding a Layer Using the ``bitbake-layers`` Script
821--------------------------------------------------
822
823Once you create your general layer, you must add it to your
824``bblayers.conf`` file. Adding the layer to this configuration file
825makes the OpenEmbedded build system aware of your layer so that it can
826search it for metadata.
827
828Add your layer by using the ``bitbake-layers add-layer`` command::
829
830 $ bitbake-layers add-layer your_layer_name
831
832Here is an example that adds a
833layer named ``meta-scottrif`` to the configuration file. Following the
834command that adds the layer is another ``bitbake-layers`` command that
835shows the layers that are in your ``bblayers.conf`` file::
836
837 $ bitbake-layers add-layer meta-scottrif
838 NOTE: Starting bitbake server...
839 Parsing recipes: 100% |##########################################################| Time: 0:00:49
840 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
841 $ bitbake-layers show-layers
842 NOTE: Starting bitbake server...
843 layer path priority
844 ==========================================================================
845 meta /home/scottrif/poky/meta 5
846 meta-poky /home/scottrif/poky/meta-poky 5
847 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
848 workspace /home/scottrif/poky/build/workspace 99
849 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
850
851
852Adding the layer to this file
853enables the build system to locate the layer during the build.
854
855.. note::
856
857 During a build, the OpenEmbedded build system looks in the layers
858 from the top of the list down to the bottom in that order.
859
860Saving and restoring the layers setup
861-------------------------------------
862
863Once you have a working build with the correct set of layers, it is beneficial
864to capture the layer setup --- what they are, which repositories they come from
865and which SCM revisions they're at --- into a configuration file, so that this
866setup can be easily replicated later, perhaps on a different machine. Here's
867how to do this::
868
869 $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/
870 NOTE: Starting bitbake server...
871 NOTE: Created /srv/work/alex/meta-alex/setup-layers.json
872 NOTE: Created /srv/work/alex/meta-alex/setup-layers
873
874The tool needs a single argument which tells where to place the output, consisting
875of a json formatted layer configuration, and a ``setup-layers`` script that can use that configuration
876to restore the layers in a different location, or on a different host machine. The argument
877can point to a custom layer (which is then deemed a "bootstrap" layer that needs to be
878checked out first), or into a completely independent location.
879
880The replication of the layers is performed by running the ``setup-layers`` script provided
881above:
882
8831. Clone the bootstrap layer or some other repository to obtain
884 the json config and the setup script that can use it.
885
8862. Run the script directly with no options::
887
888 alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers
889 Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override.
890
891 Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master
892 Running 'git init -q /srv/work/alex/my-build/meta-intel'
893 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel
894 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel
895 Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel
896
897 Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers
898 Running 'git init -q /srv/work/alex/my-build/poky'
899 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky
900 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky
901 Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky
902 Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky
903 Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky
904
905.. note::
906 This will work to update an existing checkout as well.
907
908.. note::
909 The script is self-sufficient and requires only python3
910 and git on the build machine.
911
912.. note::
913 Both the ``create-layers-setup`` and the ``setup-layers`` provided several additional options
914 that customize their behavior - you are welcome to study them via ``--help`` command line parameter.
915
916Customizing Images
917==================
918
919You can customize images to satisfy particular requirements. This
920section describes several methods and provides guidelines for each.
921
922Customizing Images Using ``local.conf``
923---------------------------------------
924
925Probably the easiest way to customize an image is to add a package by
926way of the ``local.conf`` configuration file. Because it is limited to
927local use, this method generally only allows you to add packages and is
928not as flexible as creating your own customized image. When you add
929packages using local variables this way, you need to realize that these
930variable changes are in effect for every build and consequently affect
931all images, which might not be what you require.
932
933To add a package to your image using the local configuration file, use
934the :term:`IMAGE_INSTALL` variable with the ``:append`` operator::
935
936 IMAGE_INSTALL:append = " strace"
937
938Use of the syntax is important; specifically, the leading space
939after the opening quote and before the package name, which is
940``strace`` in this example. This space is required since the ``:append``
941operator does not add the space.
942
943Furthermore, you must use ``:append`` instead of the ``+=`` operator if
944you want to avoid ordering issues. The reason for this is because doing
945so unconditionally appends to the variable and avoids ordering problems
946due to the variable being set in image recipes and ``.bbclass`` files
947with operators like ``?=``. Using ``:append`` ensures the operation
948takes effect.
949
950As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all
951images. It is possible to extend the syntax so that the variable applies
952to a specific image only. Here is an example::
953
954 IMAGE_INSTALL:append:pn-core-image-minimal = " strace"
955
956This example adds ``strace`` to the ``core-image-minimal`` image only.
957
958You can add packages using a similar approach through the
959:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
960``core-image-*`` images are affected.
961
962Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
963-------------------------------------------------------------------------------
964
965Another method for customizing your image is to enable or disable
966high-level image features by using the
967:term:`IMAGE_FEATURES` and
968:term:`EXTRA_IMAGE_FEATURES`
969variables. Although the functions for both variables are nearly
970equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
971a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
972``local.conf`` file, which is found in the :term:`Build Directory`.
973
974To understand how these features work, the best reference is
975:ref:`meta/classes-recipe/image.bbclass <ref-classes-image>`.
976This class lists out the available
977:term:`IMAGE_FEATURES` of which most map to package groups while some, such
978as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
979configuration settings.
980
981In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
982variable and then maps or configures the feature accordingly. Based on
983this information, the build system automatically adds the appropriate
984packages or configurations to the
985:term:`IMAGE_INSTALL` variable.
986Effectively, you are enabling extra features by extending the class or
987creating a custom class for use with specialized image ``.bb`` files.
988
989Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
990configuration file. Using a separate area from which to enable features
991with this variable helps you avoid overwriting the features in the image
992recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
993:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
994``meta/conf/bitbake.conf``.
995
996To illustrate how you can use these variables to modify your image,
997consider an example that selects the SSH server. The Yocto Project ships
998with two SSH servers you can use with your images: Dropbear and OpenSSH.
999Dropbear is a minimal SSH server appropriate for resource-constrained
1000environments, while OpenSSH is a well-known standard SSH server
1001implementation. By default, the ``core-image-sato`` image is configured
1002to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
1003images both include OpenSSH. The ``core-image-minimal`` image does not
1004contain an SSH server.
1005
1006You can customize your image and change these defaults. Edit the
1007:term:`IMAGE_FEATURES` variable in your recipe or use the
1008:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
1009configures the image you are working with to include
1010``ssh-server-dropbear`` or ``ssh-server-openssh``.
1011
1012.. note::
1013
1014 See the ":ref:`ref-manual/features:image features`" section in the Yocto
1015 Project Reference Manual for a complete list of image features that ship
1016 with the Yocto Project.
1017
1018Customizing Images Using Custom .bb Files
1019-----------------------------------------
1020
1021You can also customize an image by creating a custom recipe that defines
1022additional software as part of the image. The following example shows
1023the form for the two lines you need::
1024
1025 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
1026 inherit core-image
1027
1028Defining the software using a custom recipe gives you total control over
1029the contents of the image. It is important to use the correct names of
1030packages in the :term:`IMAGE_INSTALL` variable. You must use the
1031OpenEmbedded notation and not the Debian notation for the names (e.g.
1032``glibc-dev`` instead of ``libc6-dev``).
1033
1034The other method for creating a custom image is to base it on an
1035existing image. For example, if you want to create an image based on
1036``core-image-sato`` but add the additional package ``strace`` to the
1037image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
1038``.bb`` and add the following line to the end of the copy::
1039
1040 IMAGE_INSTALL += "strace"
1041
1042Customizing Images Using Custom Package Groups
1043----------------------------------------------
1044
1045For complex custom images, the best approach for customizing an image is
1046to create a custom package group recipe that is used to build the image
1047or images. A good example of a package group recipe is
1048``meta/recipes-core/packagegroups/packagegroup-base.bb``.
1049
1050If you examine that recipe, you see that the :term:`PACKAGES` variable lists
1051the package group packages to produce. The ``inherit packagegroup``
1052statement sets appropriate default values and automatically adds
1053``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
1054package specified in the :term:`PACKAGES` statement.
1055
1056.. note::
1057
1058 The ``inherit packagegroup`` line should be located near the top of the
1059 recipe, certainly before the :term:`PACKAGES` statement.
1060
1061For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
1062and :term:`RRECOMMENDS` entries to provide a list of packages the parent
1063task package should contain. You can see examples of these further down
1064in the ``packagegroup-base.bb`` recipe.
1065
1066Here is a short, fabricated example showing the same basic pieces for a
1067hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
1068the variable :term:`PN` is the standard way to abbreviate the reference to
1069the full packagegroup name ``packagegroup-custom``::
1070
1071 DESCRIPTION = "My Custom Package Groups"
1072
1073 inherit packagegroup
1074
1075 PACKAGES = "\
1076 ${PN}-apps \
1077 ${PN}-tools \
1078 "
1079
1080 RDEPENDS:${PN}-apps = "\
1081 dropbear \
1082 portmap \
1083 psplash"
1084
1085 RDEPENDS:${PN}-tools = "\
1086 oprofile \
1087 oprofileui-server \
1088 lttng-tools"
1089
1090 RRECOMMENDS:${PN}-tools = "\
1091 kernel-module-oprofile"
1092
1093In the previous example, two package group packages are created with
1094their dependencies and their recommended package dependencies listed:
1095``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1096build an image using these package group packages, you need to add
1097``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
1098:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
1099areas of this section.
1100
1101Customizing an Image Hostname
1102-----------------------------
1103
1104By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1105is the same as the machine name. For example, if
1106:term:`MACHINE` equals "qemux86", the
1107configured hostname written to ``/etc/hostname`` is "qemux86".
1108
1109You can customize this name by altering the value of the "hostname"
1110variable in the ``base-files`` recipe using either an append file or a
1111configuration file. Use the following in an append file::
1112
1113 hostname = "myhostname"
1114
1115Use the following in a configuration file::
1116
1117 hostname:pn-base-files = "myhostname"
1118
1119Changing the default value of the variable "hostname" can be useful in
1120certain situations. For example, suppose you need to do extensive
1121testing on an image and you would like to easily identify the image
1122under test from existing images with typical default hostnames. In this
1123situation, you could change the default hostname to "testme", which
1124results in all the images using the name "testme". Once testing is
1125complete and you do not need to rebuild the image for test any longer,
1126you can easily reset the default hostname.
1127
1128Another point of interest is that if you unset the variable, the image
1129will have no default hostname in the filesystem. Here is an example that
1130unsets the variable in a configuration file::
1131
1132 hostname:pn-base-files = ""
1133
1134Having no default hostname in the filesystem is suitable for
1135environments that use dynamic hostnames such as virtual machines.
1136
1137Writing a New Recipe
1138====================
1139
1140Recipes (``.bb`` files) are fundamental components in the Yocto Project
1141environment. Each software component built by the OpenEmbedded build
1142system requires a recipe to define the component. This section describes
1143how to create, write, and test a new recipe.
1144
1145.. note::
1146
1147 For information on variables that are useful for recipes and for
1148 information about recipe naming issues, see the
1149 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
1150 Reference Manual.
1151
1152Overview
1153--------
1154
1155The following figure shows the basic process for creating a new recipe.
1156The remainder of the section provides details for the steps.
1157
1158.. image:: figures/recipe-workflow.png
1159 :align: center
1160 :width: 50%
1161
1162Locate or Automatically Create a Base Recipe
1163--------------------------------------------
1164
1165You can always write a recipe from scratch. However, there are three choices
1166that can help you quickly get started with a new recipe:
1167
1168- ``devtool add``: A command that assists in creating a recipe and an
1169 environment conducive to development.
1170
1171- ``recipetool create``: A command provided by the Yocto Project that
1172 automates creation of a base recipe based on the source files.
1173
1174- *Existing Recipes:* Location and modification of an existing recipe
1175 that is similar in function to the recipe you need.
1176
1177.. note::
1178
1179 For information on recipe syntax, see the
1180 ":ref:`dev-manual/common-tasks:recipe syntax`" section.
1181
1182Creating the Base Recipe Using ``devtool add``
1183~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1184
1185The ``devtool add`` command uses the same logic for auto-creating the
1186recipe as ``recipetool create``, which is listed below. Additionally,
1187however, ``devtool add`` sets up an environment that makes it easy for
1188you to patch the source and to make changes to the recipe as is often
1189necessary when adding a recipe to build a new piece of software to be
1190included in a build.
1191
1192You can find a complete description of the ``devtool add`` command in
1193the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
1194in the Yocto Project Application Development and the Extensible Software
1195Development Kit (eSDK) manual.
1196
1197Creating the Base Recipe Using ``recipetool create``
1198~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1199
1200``recipetool create`` automates creation of a base recipe given a set of
1201source code files. As long as you can extract or point to the source
1202files, the tool will construct a recipe and automatically configure all
1203pre-build information into the recipe. For example, suppose you have an
1204application that builds using Autotools. Creating the base recipe using
1205``recipetool`` results in a recipe that has the pre-build dependencies,
1206license requirements, and checksums configured.
1207
1208To run the tool, you just need to be in your :term:`Build Directory` and
1209have sourced the build environment setup script (i.e.
1210:ref:`structure-core-script`). To get help on the tool, use the following
1211command::
1212
1213 $ recipetool -h
1214 NOTE: Starting bitbake server...
1215 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1216
1217 OpenEmbedded recipe tool
1218
1219 options:
1220 -d, --debug Enable debug output
1221 -q, --quiet Print only errors
1222 --color COLOR Colorize output (where COLOR is auto, always, never)
1223 -h, --help show this help message and exit
1224
1225 subcommands:
1226 create Create a new recipe
1227 newappend Create a bbappend for the specified target in the specified
1228 layer
1229 setvar Set a variable within a recipe
1230 appendfile Create/update a bbappend to replace a target file
1231 appendsrcfiles Create/update a bbappend to add or replace source files
1232 appendsrcfile Create/update a bbappend to add or replace a source file
1233 Use recipetool <subcommand> --help to get help on a specific command
1234
1235Running ``recipetool create -o OUTFILE`` creates the base recipe and
1236locates it properly in the layer that contains your source files.
1237Following are some syntax examples:
1238
1239 - Use this syntax to generate a recipe based on source. Once generated,
1240 the recipe resides in the existing source code layer::
1241
1242 recipetool create -o OUTFILE source
1243
1244 - Use this syntax to generate a recipe using code that
1245 you extract from source. The extracted code is placed in its own layer
1246 defined by :term:`EXTERNALSRC`.
1247 ::
1248
1249 recipetool create -o OUTFILE -x EXTERNALSRC source
1250
1251 - Use this syntax to generate a recipe based on source. The options
1252 direct ``recipetool`` to generate debugging information. Once generated,
1253 the recipe resides in the existing source code layer::
1254
1255 recipetool create -d -o OUTFILE source
1256
1257Locating and Using a Similar Recipe
1258~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1259
1260Before writing a recipe from scratch, it is often useful to discover
1261whether someone else has already written one that meets (or comes close
1262to meeting) your needs. The Yocto Project and OpenEmbedded communities
1263maintain many recipes that might be candidates for what you are doing.
1264You can find a good central index of these recipes in the
1265:oe_layerindex:`OpenEmbedded Layer Index <>`.
1266
1267Working from an existing recipe or a skeleton recipe is the best way to
1268get started. Here are some points on both methods:
1269
1270- *Locate and modify a recipe that is close to what you want to do:*
1271 This method works when you are familiar with the current recipe
1272 space. The method does not work so well for those new to the Yocto
1273 Project or writing recipes.
1274
1275 Some risks associated with this method are using a recipe that has
1276 areas totally unrelated to what you are trying to accomplish with
1277 your recipe, not recognizing areas of the recipe that you might have
1278 to add from scratch, and so forth. All these risks stem from
1279 unfamiliarity with the existing recipe space.
1280
1281- *Use and modify the following skeleton recipe:* If for some reason
1282 you do not want to use ``recipetool`` and you cannot find an existing
1283 recipe that is close to meeting your needs, you can use the following
1284 structure to provide the fundamental areas of a new recipe.
1285 ::
1286
1287 DESCRIPTION = ""
1288 HOMEPAGE = ""
1289 LICENSE = ""
1290 SECTION = ""
1291 DEPENDS = ""
1292 LIC_FILES_CHKSUM = ""
1293
1294 SRC_URI = ""
1295
1296Storing and Naming the Recipe
1297-----------------------------
1298
1299Once you have your base recipe, you should put it in your own layer and
1300name it appropriately. Locating it correctly ensures that the
1301OpenEmbedded build system can find it when you use BitBake to process
1302the recipe.
1303
1304- *Storing Your Recipe:* The OpenEmbedded build system locates your
1305 recipe through the layer's ``conf/layer.conf`` file and the
1306 :term:`BBFILES` variable. This
1307 variable sets up a path from which the build system can locate
1308 recipes. Here is the typical use::
1309
1310 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1311 ${LAYERDIR}/recipes-*/*/*.bbappend"
1312
1313 Consequently, you need to be sure you locate your new recipe inside
1314 your layer such that it can be found.
1315
1316 You can find more information on how layers are structured in the
1317 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
1318
1319- *Naming Your Recipe:* When you name your recipe, you need to follow
1320 this naming convention::
1321
1322 basename_version.bb
1323
1324 Use lower-cased characters and do not include the reserved suffixes
1325 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1326 them as part of your recipe name unless the string applies). Here are some
1327 examples:
1328
1329 .. code-block:: none
1330
1331 cups_1.7.0.bb
1332 gawk_4.0.2.bb
1333 irssi_0.8.16-rc1.bb
1334
1335Running a Build on the Recipe
1336-----------------------------
1337
1338Creating a new recipe is usually an iterative process that requires
1339using BitBake to process the recipe multiple times in order to
1340progressively discover and add information to the recipe file.
1341
1342Assuming you have sourced the build environment setup script (i.e.
1343:ref:`structure-core-script`) and you are in the :term:`Build Directory`, use
1344BitBake to process your recipe. All you need to provide is the
1345``basename`` of the recipe as described in the previous section::
1346
1347 $ bitbake basename
1348
1349During the build, the OpenEmbedded build system creates a temporary work
1350directory for each recipe
1351(``${``\ :term:`WORKDIR`\ ``}``)
1352where it keeps extracted source files, log files, intermediate
1353compilation and packaging files, and so forth.
1354
1355The path to the per-recipe temporary work directory depends on the
1356context in which it is being built. The quickest way to find this path
1357is to have BitBake return it by running the following::
1358
1359 $ bitbake -e basename | grep ^WORKDIR=
1360
1361As an example, assume a Source Directory
1362top-level folder named ``poky``, a default :term:`Build Directory` at
1363``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1364Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1365case, the work directory the build system uses to build the package
1366would be as follows::
1367
1368 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1369
1370Inside this directory you can find sub-directories such as ``image``,
1371``packages-split``, and ``temp``. After the build, you can examine these
1372to determine how well the build went.
1373
1374.. note::
1375
1376 You can find log files for each task in the recipe's ``temp``
1377 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1378 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1379 ``log.do_fetch``, and ``log.do_compile``).
1380
1381You can find more information about the build process in
1382":doc:`/overview-manual/development-environment`"
1383chapter of the Yocto Project Overview and Concepts Manual.
1384
1385Fetching Code
1386-------------
1387
1388The first thing your recipe must do is specify how to fetch the source
1389files. Fetching is controlled mainly through the
1390:term:`SRC_URI` variable. Your recipe
1391must have a :term:`SRC_URI` variable that points to where the source is
1392located. For a graphical representation of source locations, see the
1393":ref:`overview-manual/concepts:sources`" section in
1394the Yocto Project Overview and Concepts Manual.
1395
1396The :ref:`ref-tasks-fetch` task uses
1397the prefix of each entry in the :term:`SRC_URI` variable value to determine
1398which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
1399source files. It is the :term:`SRC_URI` variable that triggers the fetcher.
1400The :ref:`ref-tasks-patch` task uses
1401the variable after source is fetched to apply patches. The OpenEmbedded
1402build system uses
1403:term:`FILESOVERRIDES` for
1404scanning directory locations for local files in :term:`SRC_URI`.
1405
1406The :term:`SRC_URI` variable in your recipe must define each unique location
1407for your source files. It is good practice to not hard-code version
1408numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
1409values, use ``${``\ :term:`PV`\ ``}``,
1410which causes the fetch process to use the version specified in the
1411recipe filename. Specifying the version in this manner means that
1412upgrading the recipe to a future version is as simple as renaming the
1413recipe to match the new version.
1414
1415Here is a simple example from the
1416``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1417comes from a single tarball. Notice the use of the
1418:term:`PV` variable::
1419
1420 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
1421
1422Files mentioned in :term:`SRC_URI` whose names end in a typical archive
1423extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1424forth), are automatically extracted during the
1425:ref:`ref-tasks-unpack` task. For
1426another example that specifies these types of files, see the
1427":ref:`dev-manual/common-tasks:autotooled package`" section.
1428
1429Another way of specifying source is from an SCM. For Git repositories,
1430you must specify :term:`SRCREV` and you should specify :term:`PV` to include
1431the revision with :term:`SRCPV`. Here is an example from the recipe
1432``meta/recipes-core/musl/gcompat_git.bb``::
1433
1434 SRC_URI = "git://git.adelielinux.org/adelie/gcompat.git;protocol=https;branch=current"
1435
1436 PV = "1.0.0+1.1+git${SRCPV}"
1437 SRCREV = "af5a49e489fdc04b9cf02547650d7aeaccd43793"
1438
1439If your :term:`SRC_URI` statement includes URLs pointing to individual files
1440fetched from a remote server other than a version control system,
1441BitBake attempts to verify the files against checksums defined in your
1442recipe to ensure they have not been tampered with or otherwise modified
1443since the recipe was written. Two checksums are used:
1444``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1445
1446If your :term:`SRC_URI` variable points to more than a single URL (excluding
1447SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1448each URL. For these cases, you provide a name for each URL as part of
1449the :term:`SRC_URI` and then reference that name in the subsequent checksum
1450statements. Here is an example combining lines from the files
1451``git.inc`` and ``git_2.24.1.bb``::
1452
1453 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1454 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1455
1456 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1457 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1458 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1459 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1460
1461Proper values for ``md5`` and ``sha256`` checksums might be available
1462with other signatures on the download page for the upstream source (e.g.
1463``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1464OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1465you should verify all the signatures you find by hand.
1466
1467If no :term:`SRC_URI` checksums are specified when you attempt to build the
1468recipe, or you provide an incorrect checksum, the build will produce an
1469error for each missing or incorrect checksum. As part of the error
1470message, the build system provides the checksum string corresponding to
1471the fetched file. Once you have the correct checksums, you can copy and
1472paste them into your recipe and then run the build again to continue.
1473
1474.. note::
1475
1476 As mentioned, if the upstream source provides signatures for
1477 verifying the downloaded source code, you should verify those
1478 manually before setting the checksum values in the recipe and
1479 continuing with the build.
1480
1481This final example is a bit more complicated and is from the
1482``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
1483example's :term:`SRC_URI` statement identifies multiple files as the source
1484files for the recipe: a tarball, a patch file, a desktop file, and an
1485icon.
1486::
1487
1488 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1489 file://xwc.patch \
1490 file://rxvt.desktop \
1491 file://rxvt.png"
1492
1493When you specify local files using the ``file://`` URI protocol, the
1494build system fetches files from the local machine. The path is relative
1495to the :term:`FILESPATH` variable
1496and searches specific directories in a certain order:
1497``${``\ :term:`BP`\ ``}``,
1498``${``\ :term:`BPN`\ ``}``, and
1499``files``. The directories are assumed to be subdirectories of the
1500directory in which the recipe or append file resides. For another
1501example that specifies these types of files, see the
1502":ref:`dev-manual/common-tasks:single .c file package (hello world!)`" section.
1503
1504The previous example also specifies a patch file. Patch files are files
1505whose names usually end in ``.patch`` or ``.diff`` but can end with
1506compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1507The build system automatically applies patches as described in the
1508":ref:`dev-manual/common-tasks:patching code`" section.
1509
1510Fetching Code Through Firewalls
1511~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1512
1513Some users are behind firewalls and need to fetch code through a proxy.
1514See the ":doc:`/ref-manual/faq`" chapter for advice.
1515
1516Limiting the Number of Parallel Connections
1517~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1518
1519Some users are behind firewalls or use servers where the number of parallel
1520connections is limited. In such cases, you can limit the number of fetch
1521tasks being run in parallel by adding the following to your ``local.conf``
1522file::
1523
1524 do_fetch[number_threads] = "4"
1525
1526Unpacking Code
1527--------------
1528
1529During the build, the
1530:ref:`ref-tasks-unpack` task unpacks
1531the source with ``${``\ :term:`S`\ ``}``
1532pointing to where it is unpacked.
1533
1534If you are fetching your source files from an upstream source archived
1535tarball and the tarball's internal structure matches the common
1536convention of a top-level subdirectory named
1537``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
1538then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
1539fetch source from an archive that does not use this convention, or from
1540an SCM like Git or Subversion, your recipe needs to define :term:`S`.
1541
1542If processing your recipe using BitBake successfully unpacks the source
1543files, you need to be sure that the directory pointed to by ``${S}``
1544matches the structure of the source.
1545
1546Patching Code
1547-------------
1548
1549Sometimes it is necessary to patch code after it has been fetched. Any
1550files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
1551``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1552treated as patches. The
1553:ref:`ref-tasks-patch` task
1554automatically applies these patches.
1555
1556The build system should be able to apply patches with the "-p1" option
1557(i.e. one directory level in the path will be stripped off). If your
1558patch needs to have more directory levels stripped off, specify the
1559number of levels using the "striplevel" option in the :term:`SRC_URI` entry
1560for the patch. Alternatively, if your patch needs to be applied in a
1561specific subdirectory that is not specified in the patch file, use the
1562"patchdir" option in the entry.
1563
1564As with all local files referenced in
1565:term:`SRC_URI` using ``file://``,
1566you should place patch files in a directory next to the recipe either
1567named the same as the base name of the recipe
1568(:term:`BP` and
1569:term:`BPN`) or "files".
1570
1571Licensing
1572---------
1573
1574Your recipe needs to have both the
1575:term:`LICENSE` and
1576:term:`LIC_FILES_CHKSUM`
1577variables:
1578
1579- :term:`LICENSE`: This variable specifies the license for the software.
1580 If you do not know the license under which the software you are
1581 building is distributed, you should go to the source code and look
1582 for that information. Typical files containing this information
1583 include ``COPYING``, :term:`LICENSE`, and ``README`` files. You could
1584 also find the information near the top of a source file. For example,
1585 given a piece of software licensed under the GNU General Public
1586 License version 2, you would set :term:`LICENSE` as follows::
1587
1588 LICENSE = "GPL-2.0-only"
1589
1590 The licenses you specify within :term:`LICENSE` can have any name as long
1591 as you do not use spaces, since spaces are used as separators between
1592 license names. For standard licenses, use the names of the files in
1593 ``meta/files/common-licenses/`` or the :term:`SPDXLICENSEMAP` flag names
1594 defined in ``meta/conf/licenses.conf``.
1595
1596- :term:`LIC_FILES_CHKSUM`: The OpenEmbedded build system uses this
1597 variable to make sure the license text has not changed. If it has,
1598 the build produces an error and it affords you the chance to figure
1599 it out and correct the problem.
1600
1601 You need to specify all applicable licensing files for the software.
1602 At the end of the configuration step, the build process will compare
1603 the checksums of the files to be sure the text has not changed. Any
1604 differences result in an error with the message containing the
1605 current checksum. For more explanation and examples of how to set the
1606 :term:`LIC_FILES_CHKSUM` variable, see the
1607 ":ref:`dev-manual/common-tasks:tracking license changes`" section.
1608
1609 To determine the correct checksum string, you can list the
1610 appropriate files in the :term:`LIC_FILES_CHKSUM` variable with incorrect
1611 md5 strings, attempt to build the software, and then note the
1612 resulting error messages that will report the correct md5 strings.
1613 See the ":ref:`dev-manual/common-tasks:fetching code`" section for
1614 additional information.
1615
1616 Here is an example that assumes the software has a ``COPYING`` file::
1617
1618 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1619
1620 When you try to build the
1621 software, the build system will produce an error and give you the
1622 correct string that you can substitute into the recipe file for a
1623 subsequent build.
1624
1625Dependencies
1626------------
1627
1628Most software packages have a short list of other packages that they
1629require, which are called dependencies. These dependencies fall into two
1630main categories: build-time dependencies, which are required when the
1631software is built; and runtime dependencies, which are required to be
1632installed on the target in order for the software to run.
1633
1634Within a recipe, you specify build-time dependencies using the
1635:term:`DEPENDS` variable. Although there are nuances,
1636items specified in :term:`DEPENDS` should be names of other
1637recipes. It is important that you specify all build-time dependencies
1638explicitly.
1639
1640Another consideration is that configure scripts might automatically
1641check for optional dependencies and enable corresponding functionality
1642if those dependencies are found. If you wish to make a recipe that is
1643more generally useful (e.g. publish the recipe in a layer for others to
1644use), instead of hard-disabling the functionality, you can use the
1645:term:`PACKAGECONFIG` variable to allow functionality and the
1646corresponding dependencies to be enabled and disabled easily by other
1647users of the recipe.
1648
1649Similar to build-time dependencies, you specify runtime dependencies
1650through a variable -
1651:term:`RDEPENDS`, which is
1652package-specific. All variables that are package-specific need to have
1653the name of the package added to the end as an override. Since the main
1654package for a recipe has the same name as the recipe, and the recipe's
1655name can be found through the
1656``${``\ :term:`PN`\ ``}`` variable, then
1657you specify the dependencies for the main package by setting
1658``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you
1659would set ``RDEPENDS:${PN}-tools``, and so forth.
1660
1661Some runtime dependencies will be set automatically at packaging time.
1662These dependencies include any shared library dependencies (i.e. if a
1663package "example" contains "libexample" and another package "mypackage"
1664contains a binary that links to "libexample" then the OpenEmbedded build
1665system will automatically add a runtime dependency to "mypackage" on
1666"example"). See the
1667":ref:`overview-manual/concepts:automatically added runtime dependencies`"
1668section in the Yocto Project Overview and Concepts Manual for further
1669details.
1670
1671Configuring the Recipe
1672----------------------
1673
1674Most software provides some means of setting build-time configuration
1675options before compilation. Typically, setting these options is
1676accomplished by running a configure script with options, or by modifying
1677a build configuration file.
1678
1679.. note::
1680
1681 As of Yocto Project Release 1.7, some of the core recipes that
1682 package binary configuration scripts now disable the scripts due to
1683 the scripts previously requiring error-prone path substitution. The
1684 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1685 robust. You can find a list of the ``*-config`` scripts that are disabled
1686 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1687 in the Yocto Project Reference Manual.
1688
1689A major part of build-time configuration is about checking for
1690build-time dependencies and possibly enabling optional functionality as
1691a result. You need to specify any build-time dependencies for the
1692software you are building in your recipe's
1693:term:`DEPENDS` value, in terms of
1694other recipes that satisfy those dependencies. You can often find
1695build-time or runtime dependencies described in the software's
1696documentation.
1697
1698The following list provides configuration items of note based on how
1699your software is built:
1700
1701- *Autotools:* If your source files have a ``configure.ac`` file, then
1702 your software is built using Autotools. If this is the case, you just
1703 need to modify the configuration.
1704
1705 When using Autotools, your recipe needs to inherit the
1706 :ref:`autotools <ref-classes-autotools>` class and it does not have to
1707 contain a :ref:`ref-tasks-configure` task. However, you might still want to
1708 make some adjustments. For example, you can set :term:`EXTRA_OECONF` or
1709 :term:`PACKAGECONFIG_CONFARGS` to pass any needed configure options that
1710 are specific to the recipe.
1711
1712- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1713 your software is built using CMake. If this is the case, you just
1714 need to modify the configuration.
1715
1716 When you use CMake, your recipe needs to inherit the
1717 :ref:`cmake <ref-classes-cmake>` class and it does not have to contain a
1718 :ref:`ref-tasks-configure` task. You can make some adjustments by setting
1719 :term:`EXTRA_OECMAKE` to pass any needed configure options that are
1720 specific to the recipe.
1721
1722 .. note::
1723
1724 If you need to install one or more custom CMake toolchain files
1725 that are supplied by the application you are building, install the
1726 files to ``${D}${datadir}/cmake/Modules`` during :ref:`ref-tasks-install`.
1727
1728- *Other:* If your source files do not have a ``configure.ac`` or
1729 ``CMakeLists.txt`` file, then your software is built using some
1730 method other than Autotools or CMake. If this is the case, you
1731 normally need to provide a
1732 :ref:`ref-tasks-configure` task
1733 in your recipe unless, of course, there is nothing to configure.
1734
1735 Even if your software is not being built by Autotools or CMake, you
1736 still might not need to deal with any configuration issues. You need
1737 to determine if configuration is even a required step. You might need
1738 to modify a Makefile or some configuration file used for the build to
1739 specify necessary build options. Or, perhaps you might need to run a
1740 provided, custom configure script with the appropriate options.
1741
1742 For the case involving a custom configure script, you would run
1743 ``./configure --help`` and look for the options you need to set.
1744
1745Once configuration succeeds, it is always good practice to look at the
1746``log.do_configure`` file to ensure that the appropriate options have
1747been enabled and no additional build-time dependencies need to be added
1748to :term:`DEPENDS`. For example, if the configure script reports that it
1749found something not mentioned in :term:`DEPENDS`, or that it did not find
1750something that it needed for some desired optional functionality, then
1751you would need to add those to :term:`DEPENDS`. Looking at the log might
1752also reveal items being checked for, enabled, or both that you do not
1753want, or items not being found that are in :term:`DEPENDS`, in which case
1754you would need to look at passing extra options to the configure script
1755as needed. For reference information on configure options specific to
1756the software you are building, you can consult the output of the
1757``./configure --help`` command within ``${S}`` or consult the software's
1758upstream documentation.
1759
1760Using Headers to Interface with Devices
1761---------------------------------------
1762
1763If your recipe builds an application that needs to communicate with some
1764device or needs an API into a custom kernel, you will need to provide
1765appropriate header files. Under no circumstances should you ever modify
1766the existing
1767``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1768These headers are used to build ``libc`` and must not be compromised
1769with custom or machine-specific header information. If you customize
1770``libc`` through modified headers all other applications that use
1771``libc`` thus become affected.
1772
1773.. note::
1774
1775 Never copy and customize the ``libc`` header file (i.e.
1776 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
1777
1778The correct way to interface to a device or custom kernel is to use a
1779separate package that provides the additional headers for the driver or
1780other unique interfaces. When doing so, your application also becomes
1781responsible for creating a dependency on that specific provider.
1782
1783Consider the following:
1784
1785- Never modify ``linux-libc-headers.inc``. Consider that file to be
1786 part of the ``libc`` system, and not something you use to access the
1787 kernel directly. You should access ``libc`` through specific ``libc``
1788 calls.
1789
1790- Applications that must talk directly to devices should either provide
1791 necessary headers themselves, or establish a dependency on a special
1792 headers package that is specific to that driver.
1793
1794For example, suppose you want to modify an existing header that adds I/O
1795control or network support. If the modifications are used by a small
1796number programs, providing a unique version of a header is easy and has
1797little impact. When doing so, bear in mind the guidelines in the
1798previous list.
1799
1800.. note::
1801
1802 If for some reason your changes need to modify the behavior of the ``libc``,
1803 and subsequently all other applications on the system, use a ``.bbappend``
1804 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1805 make the changes machine specific.
1806
1807Consider a case where your kernel is older and you need an older
1808``libc`` ABI. The headers installed by your recipe should still be a
1809standard mainline kernel, not your own custom one.
1810
1811When you use custom kernel headers you need to get them from
1812:term:`STAGING_KERNEL_DIR`,
1813which is the directory with kernel headers that are required to build
1814out-of-tree modules. Your recipe will also need the following::
1815
1816 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1817
1818Compilation
1819-----------
1820
1821During a build, the :ref:`ref-tasks-compile` task happens after source is fetched,
1822unpacked, and configured. If the recipe passes through :ref:`ref-tasks-compile`
1823successfully, nothing needs to be done.
1824
1825However, if the compile step fails, you need to diagnose the failure.
1826Here are some common issues that cause failures.
1827
1828.. note::
1829
1830 For cases where improper paths are detected for configuration files
1831 or for when libraries/headers cannot be found, be sure you are using
1832 the more robust ``pkg-config``. See the note in section
1833 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information.
1834
1835- *Parallel build failures:* These failures manifest themselves as
1836 intermittent errors, or errors reporting that a file or directory
1837 that should be created by some other part of the build process could
1838 not be found. This type of failure can occur even if, upon
1839 inspection, the file or directory does exist after the build has
1840 failed, because that part of the build process happened in the wrong
1841 order.
1842
1843 To fix the problem, you need to either satisfy the missing dependency
1844 in the Makefile or whatever script produced the Makefile, or (as a
1845 workaround) set :term:`PARALLEL_MAKE` to an empty string::
1846
1847 PARALLEL_MAKE = ""
1848
1849 For information on parallel Makefile issues, see the
1850 ":ref:`dev-manual/common-tasks:debugging parallel make races`" section.
1851
1852- *Improper host path usage:* This failure applies to recipes building
1853 for the target or ":ref:`nativesdk <ref-classes-nativesdk>`" only. The
1854 failure occurs when the compilation process uses improper headers,
1855 libraries, or other files from the host system when cross-compiling for
1856 the target.
1857
1858 To fix the problem, examine the ``log.do_compile`` file to identify
1859 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1860 so forth) and then either add configure options, apply a patch, or do
1861 both.
1862
1863- *Failure to find required libraries/headers:* If a build-time
1864 dependency is missing because it has not been declared in
1865 :term:`DEPENDS`, or because the
1866 dependency exists but the path used by the build process to find the
1867 file is incorrect and the configure step did not detect it, the
1868 compilation process could fail. For either of these failures, the
1869 compilation process notes that files could not be found. In these
1870 cases, you need to go back and add additional options to the
1871 configure script as well as possibly add additional build-time
1872 dependencies to :term:`DEPENDS`.
1873
1874 Occasionally, it is necessary to apply a patch to the source to
1875 ensure the correct paths are used. If you need to specify paths to
1876 find files staged into the sysroot from other recipes, use the
1877 variables that the OpenEmbedded build system provides (e.g.
1878 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
1879 forth).
1880
1881Installing
1882----------
1883
1884During :ref:`ref-tasks-install`, the task copies the built files along with their
1885hierarchy to locations that would mirror their locations on the target
1886device. The installation process copies files from the
1887``${``\ :term:`S`\ ``}``,
1888``${``\ :term:`B`\ ``}``, and
1889``${``\ :term:`WORKDIR`\ ``}``
1890directories to the ``${``\ :term:`D`\ ``}``
1891directory to create the structure as it should appear on the target
1892system.
1893
1894How your software is built affects what you must do to be sure your
1895software is installed correctly. The following list describes what you
1896must do for installation depending on the type of build system used by
1897the software being built:
1898
1899- *Autotools and CMake:* If the software your recipe is building uses
1900 Autotools or CMake, the OpenEmbedded build system understands how to
1901 install the software. Consequently, you do not have to have a
1902 :ref:`ref-tasks-install` task as part of your recipe. You just need to make
1903 sure the install portion of the build completes with no issues.
1904 However, if you wish to install additional files not already being
1905 installed by ``make install``, you should do this using a
1906 ``do_install:append`` function using the install command as described
1907 in the "Manual" bulleted item later in this list.
1908
1909- *Other (using* ``make install``\ *)*: You need to define a :ref:`ref-tasks-install`
1910 function in your recipe. The function should call
1911 ``oe_runmake install`` and will likely need to pass in the
1912 destination directory as well. How you pass that path is dependent on
1913 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1914 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1915
1916 For an example recipe using ``make install``, see the
1917 ":ref:`dev-manual/common-tasks:makefile-based package`" section.
1918
1919- *Manual:* You need to define a :ref:`ref-tasks-install` function in your
1920 recipe. The function must first use ``install -d`` to create the
1921 directories under
1922 ``${``\ :term:`D`\ ``}``. Once the
1923 directories exist, your function can use ``install`` to manually
1924 install the built software into the directories.
1925
1926 You can find more information on ``install`` at
1927 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
1928
1929For the scenarios that do not use Autotools or CMake, you need to track
1930the installation and diagnose and fix any issues until everything
1931installs correctly. You need to look in the default location of
1932``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1933installed correctly.
1934
1935.. note::
1936
1937 - During the installation process, you might need to modify some of
1938 the installed files to suit the target layout. For example, you
1939 might need to replace hard-coded paths in an initscript with
1940 values of variables provided by the build system, such as
1941 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1942 modifications during :ref:`ref-tasks-install`, be sure to modify the
1943 destination file after copying rather than before copying.
1944 Modifying after copying ensures that the build system can
1945 re-execute :ref:`ref-tasks-install` if needed.
1946
1947 - ``oe_runmake install``, which can be run directly or can be run
1948 indirectly by the
1949 :ref:`autotools <ref-classes-autotools>` and
1950 :ref:`cmake <ref-classes-cmake>` classes,
1951 runs ``make install`` in parallel. Sometimes, a Makefile can have
1952 missing dependencies between targets that can result in race
1953 conditions. If you experience intermittent failures during
1954 :ref:`ref-tasks-install`, you might be able to work around them by disabling
1955 parallel Makefile installs by adding the following to the recipe::
1956
1957 PARALLEL_MAKEINST = ""
1958
1959 See :term:`PARALLEL_MAKEINST` for additional information.
1960
1961 - If you need to install one or more custom CMake toolchain files
1962 that are supplied by the application you are building, install the
1963 files to ``${D}${datadir}/cmake/Modules`` during
1964 :ref:`ref-tasks-install`.
1965
1966Enabling System Services
1967------------------------
1968
1969If you want to install a service, which is a process that usually starts
1970on boot and runs in the background, then you must include some
1971additional definitions in your recipe.
1972
1973If you are adding services and the service initialization script or the
1974service file itself is not installed, you must provide for that
1975installation in your recipe using a ``do_install:append`` function. If
1976your recipe already has a :ref:`ref-tasks-install` function, update the function
1977near its end rather than adding an additional ``do_install:append``
1978function.
1979
1980When you create the installation for your services, you need to
1981accomplish what is normally done by ``make install``. In other words,
1982make sure your installation arranges the output similar to how it is
1983arranged on the target system.
1984
1985The OpenEmbedded build system provides support for starting services two
1986different ways:
1987
1988- *SysVinit:* SysVinit is a system and service manager that manages the
1989 init system used to control the very basic functions of your system.
1990 The init program is the first program started by the Linux kernel
1991 when the system boots. Init then controls the startup, running and
1992 shutdown of all other programs.
1993
1994 To enable a service using SysVinit, your recipe needs to inherit the
1995 :ref:`update-rc.d <ref-classes-update-rc.d>` class. The class helps
1996 facilitate safely installing the package on the target.
1997
1998 You will need to set the
1999 :term:`INITSCRIPT_PACKAGES`,
2000 :term:`INITSCRIPT_NAME`,
2001 and
2002 :term:`INITSCRIPT_PARAMS`
2003 variables within your recipe.
2004
2005- *systemd:* System Management Daemon (systemd) was designed to replace
2006 SysVinit and to provide enhanced management of services. For more
2007 information on systemd, see the systemd homepage at
2008 https://freedesktop.org/wiki/Software/systemd/.
2009
2010 To enable a service using systemd, your recipe needs to inherit the
2011 :ref:`systemd <ref-classes-systemd>` class. See the ``systemd.bbclass`` file
2012 located in your :term:`Source Directory` section for more information.
2013
2014Packaging
2015---------
2016
2017Successful packaging is a combination of automated processes performed
2018by the OpenEmbedded build system and some specific steps you need to
2019take. The following list describes the process:
2020
2021- *Splitting Files*: The :ref:`ref-tasks-package` task splits the files produced
2022 by the recipe into logical components. Even software that produces a
2023 single binary might still have debug symbols, documentation, and
2024 other logical components that should be split out. The :ref:`ref-tasks-package`
2025 task ensures that files are split up and packaged correctly.
2026
2027- *Running QA Checks*: The
2028 :ref:`insane <ref-classes-insane>` class adds a
2029 step to the package generation process so that output quality
2030 assurance checks are generated by the OpenEmbedded build system. This
2031 step performs a range of checks to be sure the build's output is free
2032 of common problems that show up during runtime. For information on
2033 these checks, see the
2034 :ref:`insane <ref-classes-insane>` class and
2035 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
2036 chapter in the Yocto Project Reference Manual.
2037
2038- *Hand-Checking Your Packages*: After you build your software, you
2039 need to be sure your packages are correct. Examine the
2040 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
2041 directory and make sure files are where you expect them to be. If you
2042 discover problems, you can set
2043 :term:`PACKAGES`,
2044 :term:`FILES`,
2045 ``do_install(:append)``, and so forth as needed.
2046
2047- *Splitting an Application into Multiple Packages*: If you need to
2048 split an application into several packages, see the
2049 ":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
2050 section for an example.
2051
2052- *Installing a Post-Installation Script*: For an example showing how
2053 to install a post-installation script, see the
2054 ":ref:`dev-manual/common-tasks:post-installation scripts`" section.
2055
2056- *Marking Package Architecture*: Depending on what your recipe is
2057 building and how it is configured, it might be important to mark the
2058 packages produced as being specific to a particular machine, or to
2059 mark them as not being specific to a particular machine or
2060 architecture at all.
2061
2062 By default, packages apply to any machine with the same architecture
2063 as the target machine. When a recipe produces packages that are
2064 machine-specific (e.g. the
2065 :term:`MACHINE` value is passed
2066 into the configure script or a patch is applied only for a particular
2067 machine), you should mark them as such by adding the following to the
2068 recipe::
2069
2070 PACKAGE_ARCH = "${MACHINE_ARCH}"
2071
2072 On the other hand, if the recipe produces packages that do not
2073 contain anything specific to the target machine or architecture at
2074 all (e.g. recipes that simply package script files or configuration
2075 files), you should use the
2076 :ref:`allarch <ref-classes-allarch>` class to
2077 do this for you by adding this to your recipe::
2078
2079 inherit allarch
2080
2081 Ensuring that the package architecture is correct is not critical
2082 while you are doing the first few builds of your recipe. However, it
2083 is important in order to ensure that your recipe rebuilds (or does
2084 not rebuild) appropriately in response to changes in configuration,
2085 and to ensure that you get the appropriate packages installed on the
2086 target machine, particularly if you run separate builds for more than
2087 one target machine.
2088
2089Sharing Files Between Recipes
2090-----------------------------
2091
2092Recipes often need to use files provided by other recipes on the build
2093host. For example, an application linking to a common library needs
2094access to the library itself and its associated headers. The way this
2095access is accomplished is by populating a sysroot with files. Each
2096recipe has two sysroots in its work directory, one for target files
2097(``recipe-sysroot``) and one for files that are native to the build host
2098(``recipe-sysroot-native``).
2099
2100.. note::
2101
2102 You could find the term "staging" used within the Yocto project
2103 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
2104 variable).
2105
2106Recipes should never populate the sysroot directly (i.e. write files
2107into sysroot). Instead, files should be installed into standard
2108locations during the
2109:ref:`ref-tasks-install` task within
2110the ``${``\ :term:`D`\ ``}`` directory. The
2111reason for this limitation is that almost all files that populate the
2112sysroot are cataloged in manifests in order to ensure the files can be
2113removed later when a recipe is either modified or removed. Thus, the
2114sysroot is able to remain free from stale files.
2115
2116A subset of the files installed by the :ref:`ref-tasks-install` task are
2117used by the :ref:`ref-tasks-populate_sysroot` task as defined by the
2118:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
2119is possible to modify the list of directories that populate the sysroot.
2120The following example shows how you could add the ``/opt`` directory to
2121the list of directories within a recipe::
2122
2123 SYSROOT_DIRS += "/opt"
2124
2125.. note::
2126
2127 The `/sysroot-only` is to be used by recipes that generate artifacts
2128 that are not included in the target filesystem, allowing them to share
2129 these artifacts without needing to use the :term:`DEPLOY_DIR`.
2130
2131For a more complete description of the :ref:`ref-tasks-populate_sysroot`
2132task and its associated functions, see the
2133:ref:`staging <ref-classes-staging>` class.
2134
2135Using Virtual Providers
2136-----------------------
2137
2138Prior to a build, if you know that several different recipes provide the
2139same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2140as a placeholder for the actual provider. The actual provider is
2141determined at build-time.
2142
2143A common scenario where a virtual provider is used would be for the
2144kernel recipe. Suppose you have three kernel recipes whose
2145:term:`PN` values map to ``kernel-big``,
2146``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2147in some way uses a :term:`PROVIDES`
2148statement that essentially identifies itself as being able to provide
2149``virtual/kernel``. Here is one way through the
2150:ref:`kernel <ref-classes-kernel>` class::
2151
2152 PROVIDES += "virtual/kernel"
2153
2154Any recipe that inherits the :ref:`kernel <ref-classes-kernel>` class is
2155going to utilize a :term:`PROVIDES` statement that identifies that recipe as
2156being able to provide the ``virtual/kernel`` item.
2157
2158Now comes the time to actually build an image and you need a kernel
2159recipe, but which one? You can configure your build to call out the
2160kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
2161an example, consider the :yocto_git:`x86-base.inc
2162</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a
2163machine (i.e. :term:`MACHINE`) configuration file. This include file is the
2164reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
2165relevant lines from the include file::
2166
2167 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2168 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2169
2170When you use a virtual provider, you do not have to "hard code" a recipe
2171name as a build dependency. You can use the
2172:term:`DEPENDS` variable to state the
2173build is dependent on ``virtual/kernel`` for example::
2174
2175 DEPENDS = "virtual/kernel"
2176
2177During the build, the OpenEmbedded build system picks
2178the correct recipe needed for the ``virtual/kernel`` dependency based on
2179the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
2180mentioned at the beginning of this section, configure your build as
2181follows::
2182
2183 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
2184
2185.. note::
2186
2187 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
2188 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
2189 recipes from building is usually the desired behavior since this mechanism's
2190 purpose is to select between mutually exclusive alternative providers.
2191
2192The following lists specific examples of virtual providers:
2193
2194- ``virtual/kernel``: Provides the name of the kernel recipe to use
2195 when building a kernel image.
2196
2197- ``virtual/bootloader``: Provides the name of the bootloader to use
2198 when building an image.
2199
2200- ``virtual/libgbm``: Provides ``gbm.pc``.
2201
2202- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2203
2204- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2205
2206- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2207
2208- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2209
2210.. note::
2211
2212 Virtual providers only apply to build time dependencies specified with
2213 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2214 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2215
2216Properly Versioning Pre-Release Recipes
2217---------------------------------------
2218
2219Sometimes the name of a recipe can lead to versioning problems when the
2220recipe is upgraded to a final release. For example, consider the
2221``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
2222the ":ref:`dev-manual/common-tasks:storing and naming the recipe`" section.
2223This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
2224released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2225change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2226build system and package managers, so the resulting packages will not
2227correctly trigger an upgrade.
2228
2229In order to ensure the versions compare properly, the recommended
2230convention is to set :term:`PV` within the
2231recipe to "previous_version+current_version". You can use an additional
2232variable so that you can use the current version elsewhere. Here is an
2233example::
2234
2235 REALPV = "0.8.16-rc1"
2236 PV = "0.8.15+${REALPV}"
2237
2238Post-Installation Scripts
2239-------------------------
2240
2241Post-installation scripts run immediately after installing a package on
2242the target or during image creation when a package is included in an
2243image. To add a post-installation script to a package, add a
2244``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file
2245(``.bb``) and replace `PACKAGENAME` with the name of the package you want
2246to attach to the ``postinst`` script. To apply the post-installation
2247script to the main package for the recipe, which is usually what is
2248required, specify
2249``${``\ :term:`PN`\ ``}`` in place of
2250PACKAGENAME.
2251
2252A post-installation function has the following structure::
2253
2254 pkg_postinst:PACKAGENAME() {
2255 # Commands to carry out
2256 }
2257
2258The script defined in the post-installation function is called when the
2259root filesystem is created. If the script succeeds, the package is
2260marked as installed.
2261
2262.. note::
2263
2264 Any RPM post-installation script that runs on the target should
2265 return a 0 exit code. RPM does not allow non-zero exit codes for
2266 these scripts, and the RPM package manager will cause the package to
2267 fail installation on the target.
2268
2269Sometimes it is necessary for the execution of a post-installation
2270script to be delayed until the first boot. For example, the script might
2271need to be executed on the device itself. To delay script execution
2272until boot time, you must explicitly mark post installs to defer to the
2273target. You can use ``pkg_postinst_ontarget()`` or call
2274``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2275failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2276error during the
2277:ref:`ref-tasks-rootfs` task.
2278
2279If you have recipes that use ``pkg_postinst`` function and they require
2280the use of non-standard native tools that have dependencies during
2281root filesystem construction, you need to use the
2282:term:`PACKAGE_WRITE_DEPS`
2283variable in your recipe to list these tools. If you do not use this
2284variable, the tools might be missing and execution of the
2285post-installation script is deferred until first boot. Deferring the
2286script to the first boot is undesirable and impossible for read-only
2287root filesystems.
2288
2289.. note::
2290
2291 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
2292 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
2293 respectively. These scrips work in exactly the same way as does
2294 ``pkg_postinst`` with the exception that they run at different times. Also,
2295 because of when they run, they are not applicable to being run at image
2296 creation time like ``pkg_postinst``.
2297
2298Testing
2299-------
2300
2301The final step for completing your recipe is to be sure that the
2302software you built runs correctly. To accomplish runtime testing, add
2303the build's output packages to your image and test them on the target.
2304
2305For information on how to customize your image by adding specific
2306packages, see ":ref:`dev-manual/common-tasks:customizing images`" section.
2307
2308Examples
2309--------
2310
2311To help summarize how to write a recipe, this section provides some
2312examples given various scenarios:
2313
2314- Recipes that use local files
2315
2316- Using an Autotooled package
2317
2318- Using a Makefile-based package
2319
2320- Splitting an application into multiple packages
2321
2322- Adding binaries to an image
2323
2324Single .c File Package (Hello World!)
2325~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2326
2327Building an application from a single file that is stored locally (e.g.
2328under ``files``) requires a recipe that has the file listed in the
2329:term:`SRC_URI` variable. Additionally, you need to manually write the
2330:ref:`ref-tasks-compile` and :ref:`ref-tasks-install` tasks. The :term:`S` variable defines the
2331directory containing the source code, which is set to
2332:term:`WORKDIR` in this case --- the
2333directory BitBake uses for the build.
2334::
2335
2336 SUMMARY = "Simple helloworld application"
2337 SECTION = "examples"
2338 LICENSE = "MIT"
2339 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2340
2341 SRC_URI = "file://helloworld.c"
2342
2343 S = "${WORKDIR}"
2344
2345 do_compile() {
2346 ${CC} ${LDFLAGS} helloworld.c -o helloworld
2347 }
2348
2349 do_install() {
2350 install -d ${D}${bindir}
2351 install -m 0755 helloworld ${D}${bindir}
2352 }
2353
2354By default, the ``helloworld``, ``helloworld-dbg``, and
2355``helloworld-dev`` packages are built. For information on how to
2356customize the packaging process, see the
2357":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
2358section.
2359
2360Autotooled Package
2361~~~~~~~~~~~~~~~~~~
2362
2363Applications that use Autotools such as ``autoconf`` and ``automake``
2364require a recipe that has a source archive listed in :term:`SRC_URI` and
2365also inherit the :ref:`autotools <ref-classes-autotools>` class,
2366which contains the definitions of all the steps needed to build an
2367Autotool-based application. The result of the build is automatically
2368packaged. And, if the application uses NLS for localization, packages
2369with local information are generated (one package per language).
2370Following is one example: (``hello_2.3.bb``)
2371::
2372
2373 SUMMARY = "GNU Helloworld application"
2374 SECTION = "examples"
2375 LICENSE = "GPL-2.0-or-later"
2376 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2377
2378 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2379
2380 inherit autotools gettext
2381
2382The variable :term:`LIC_FILES_CHKSUM` is used to track source license
2383changes as described in the
2384":ref:`dev-manual/common-tasks:tracking license changes`" section in
2385the Yocto Project Overview and Concepts Manual. You can quickly create
2386Autotool-based recipes in a manner similar to the previous example.
2387
2388Makefile-Based Package
2389~~~~~~~~~~~~~~~~~~~~~~
2390
2391Applications that use GNU ``make`` also require a recipe that has the
2392source archive listed in :term:`SRC_URI`. You do not need to add a
2393:ref:`ref-tasks-compile` step since by default BitBake starts the ``make`` command
2394to compile the application. If you need additional ``make`` options, you
2395should store them in the
2396:term:`EXTRA_OEMAKE` or
2397:term:`PACKAGECONFIG_CONFARGS`
2398variables. BitBake passes these options into the GNU ``make``
2399invocation. Note that a :ref:`ref-tasks-install` task is still required.
2400Otherwise, BitBake runs an empty :ref:`ref-tasks-install` task by default.
2401
2402Some applications might require extra parameters to be passed to the
2403compiler. For example, the application might need an additional header
2404path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
2405following example shows this::
2406
2407 CFLAGS:prepend = "-I ${S}/include "
2408
2409In the following example, ``lz4`` is a makefile-based package::
2410
2411 SUMMARY = "Extremely Fast Compression algorithm"
2412 DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems."
2413 HOMEPAGE = "https://github.com/lz4/lz4"
2414
2415 LICENSE = "BSD-2-Clause | GPL-2.0-only"
2416 LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc \
2417 file://programs/COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
2418 file://LICENSE;md5=d57c0d21cb917fb4e0af2454aa48b956 \
2419 "
2420
2421 PE = "1"
2422
2423 SRCREV = "d44371841a2f1728a3f36839fd4b7e872d0927d3"
2424
2425 SRC_URI = "git://github.com/lz4/lz4.git;branch=release;protocol=https \
2426 file://CVE-2021-3520.patch \
2427 "
2428 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>.*)"
2429
2430 S = "${WORKDIR}/git"
2431
2432 # Fixed in r118, which is larger than the current version.
2433 CVE_CHECK_IGNORE += "CVE-2014-4715"
2434
2435 EXTRA_OEMAKE = "PREFIX=${prefix} CC='${CC}' CFLAGS='${CFLAGS}' DESTDIR=${D} LIBDIR=${libdir} INCLUDEDIR=${includedir} BUILD_STATIC=no"
2436
2437 do_install() {
2438 oe_runmake install
2439 }
2440
2441 BBCLASSEXTEND = "native nativesdk"
2442
2443Splitting an Application into Multiple Packages
2444~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2445
2446You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
2447application into multiple packages.
2448
2449Following is an example that uses the ``libxpm`` recipe. By default,
2450this recipe generates a single package that contains the library along
2451with a few binaries. You can modify the recipe to split the binaries
2452into separate packages::
2453
2454 require xorg-lib-common.inc
2455
2456 SUMMARY = "Xpm: X Pixmap extension library"
2457 LICENSE = "MIT"
2458 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2459 DEPENDS += "libxext libsm libxt"
2460 PE = "1"
2461
2462 XORG_PN = "libXpm"
2463
2464 PACKAGES =+ "sxpm cxpm"
2465 FILES:cxpm = "${bindir}/cxpm"
2466 FILES:sxpm = "${bindir}/sxpm"
2467
2468In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2469binaries in separate packages. Since ``bindir`` would be packaged into
2470the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
2471so additional package names are added to the start of list. This results
2472in the extra ``FILES:*`` variables then containing information that
2473define which files and directories go into which packages. Files
2474included by earlier packages are skipped by latter packages. Thus, the
2475main :term:`PN` package does not include the above listed files.
2476
2477Packaging Externally Produced Binaries
2478~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2479
2480Sometimes, you need to add pre-compiled binaries to an image. For
2481example, suppose that there are binaries for proprietary code,
2482created by a particular division of a company. Your part of the company
2483needs to use those binaries as part of an image that you are building
2484using the OpenEmbedded build system. Since you only have the binaries
2485and not the source code, you cannot use a typical recipe that expects to
2486fetch the source specified in
2487:term:`SRC_URI` and then compile it.
2488
2489One method is to package the binaries and then install them as part of
2490the image. Generally, it is not a good idea to package binaries since,
2491among other things, it can hinder the ability to reproduce builds and
2492could lead to compatibility problems with ABI in the future. However,
2493sometimes you have no choice.
2494
2495The easiest solution is to create a recipe that uses the
2496:ref:`bin_package <ref-classes-bin-package>` class
2497and to be sure that you are using default locations for build artifacts.
2498In most cases, the :ref:`bin_package <ref-classes-bin-package>` class handles "skipping" the
2499configure and compile steps as well as sets things up to grab packages
2500from the appropriate area. In particular, this class sets ``noexec`` on
2501both the :ref:`ref-tasks-configure`
2502and :ref:`ref-tasks-compile` tasks,
2503sets ``FILES:${PN}`` to "/" so that it picks up all files, and sets up a
2504:ref:`ref-tasks-install` task, which
2505effectively copies all files from ``${S}`` to ``${D}``. The
2506:ref:`bin_package <ref-classes-bin-package>` class works well when the files extracted into ``${S}``
2507are already laid out in the way they should be laid out on the target.
2508For more information on these variables, see the
2509:term:`FILES`,
2510:term:`PN`,
2511:term:`S`, and
2512:term:`D` variables in the Yocto Project
2513Reference Manual's variable glossary.
2514
2515.. note::
2516
2517 - Using :term:`DEPENDS` is a good
2518 idea even for components distributed in binary form, and is often
2519 necessary for shared libraries. For a shared library, listing the
2520 library dependencies in :term:`DEPENDS` makes sure that the libraries
2521 are available in the staging sysroot when other recipes link
2522 against the library, which might be necessary for successful
2523 linking.
2524
2525 - Using :term:`DEPENDS` also allows runtime dependencies between
2526 packages to be added automatically. See the
2527 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
2528 section in the Yocto Project Overview and Concepts Manual for more
2529 information.
2530
2531If you cannot use the :ref:`bin_package <ref-classes-bin-package>` class, you need to be sure you are
2532doing the following:
2533
2534- Create a recipe where the
2535 :ref:`ref-tasks-configure` and
2536 :ref:`ref-tasks-compile` tasks do
2537 nothing: It is usually sufficient to just not define these tasks in
2538 the recipe, because the default implementations do nothing unless a
2539 Makefile is found in
2540 ``${``\ :term:`S`\ ``}``.
2541
2542 If ``${S}`` might contain a Makefile, or if you inherit some class
2543 that replaces :ref:`ref-tasks-configure` and :ref:`ref-tasks-compile` with custom
2544 versions, then you can use the
2545 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2546 flag to turn the tasks into no-ops, as follows::
2547
2548 do_configure[noexec] = "1"
2549 do_compile[noexec] = "1"
2550
2551 Unlike
2552 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2553 using the flag preserves the dependency chain from the
2554 :ref:`ref-tasks-fetch`,
2555 :ref:`ref-tasks-unpack`, and
2556 :ref:`ref-tasks-patch` tasks to the
2557 :ref:`ref-tasks-install` task.
2558
2559- Make sure your :ref:`ref-tasks-install` task installs the binaries
2560 appropriately.
2561
2562- Ensure that you set up :term:`FILES`
2563 (usually
2564 ``FILES:${``\ :term:`PN`\ ``}``) to
2565 point to the files you have installed, which of course depends on
2566 where you have installed them and whether those files are in
2567 different locations than the defaults.
2568
2569Following Recipe Style Guidelines
2570---------------------------------
2571
2572When writing recipes, it is good to conform to existing style
2573guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
2574provides rough guidelines for preferred recipe style.
2575
2576It is common for existing recipes to deviate a bit from this style.
2577However, aiming for at least a consistent style is a good idea. Some
2578practices, such as omitting spaces around ``=`` operators in assignments
2579or ordering recipe components in an erratic way, are widely seen as poor
2580style.
2581
2582Recipe Syntax
2583-------------
2584
2585Understanding recipe file syntax is important for writing recipes. The
2586following list overviews the basic items that make up a BitBake recipe
2587file. For more complete BitBake syntax descriptions, see the
2588":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`"
2589chapter of the BitBake User Manual.
2590
2591- *Variable Assignments and Manipulations:* Variable assignments allow
2592 a value to be assigned to a variable. The assignment can be static
2593 text or might include the contents of other variables. In addition to
2594 the assignment, appending and prepending operations are also
2595 supported.
2596
2597 The following example shows some of the ways you can use variables in
2598 recipes::
2599
2600 S = "${WORKDIR}/postfix-${PV}"
2601 CFLAGS += "-DNO_ASM"
2602 CFLAGS:append = " --enable-important-feature"
2603
2604- *Functions:* Functions provide a series of actions to be performed.
2605 You usually use functions to override the default implementation of a
2606 task function or to complement a default function (i.e. append or
2607 prepend to an existing function). Standard functions use ``sh`` shell
2608 syntax, although access to OpenEmbedded variables and internal
2609 methods are also available.
2610
2611 Here is an example function from the ``sed`` recipe::
2612
2613 do_install () {
2614 autotools_do_install
2615 install -d ${D}${base_bindir}
2616 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2617 rmdir ${D}${bindir}/
2618 }
2619
2620 It is
2621 also possible to implement new functions that are called between
2622 existing tasks as long as the new functions are not replacing or
2623 complementing the default functions. You can implement functions in
2624 Python instead of shell. Both of these options are not seen in the
2625 majority of recipes.
2626
2627- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2628 to include common functions (``inherit``), load parts of a recipe
2629 from other files (``include`` and ``require``) and export variables
2630 to the environment (``export``).
2631
2632 The following example shows the use of some of these keywords::
2633
2634 export POSTCONF = "${STAGING_BINDIR}/postconf"
2635 inherit autoconf
2636 require otherfile.inc
2637
2638- *Comments (#):* Any lines that begin with the hash character (``#``)
2639 are treated as comment lines and are ignored::
2640
2641 # This is a comment
2642
2643This next list summarizes the most important and most commonly used
2644parts of the recipe syntax. For more information on these parts of the
2645syntax, you can reference the
2646":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
2647in the BitBake User Manual.
2648
2649- *Line Continuation (\\):* Use the backward slash (``\``) character to
2650 split a statement over multiple lines. Place the slash character at
2651 the end of the line that is to be continued on the next line::
2652
2653 VAR = "A really long \
2654 line"
2655
2656 .. note::
2657
2658 You cannot have any characters including spaces or tabs after the
2659 slash character.
2660
2661- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
2662 access the contents of a variable::
2663
2664 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2665
2666 .. note::
2667
2668 It is important to understand that the value of a variable
2669 expressed in this form does not get substituted automatically. The
2670 expansion of these expressions happens on-demand later (e.g.
2671 usually when a function that makes reference to the variable
2672 executes). This behavior ensures that the values are most
2673 appropriate for the context in which they are finally used. On the
2674 rare occasion that you do need the variable expression to be
2675 expanded immediately, you can use the
2676 :=
2677 operator instead of
2678 =
2679 when you make the assignment, but this is not generally needed.
2680
2681- *Quote All Assignments ("value"):* Use double quotes around values in
2682 all variable assignments (e.g. ``"value"``). Following is an example::
2683
2684 VAR1 = "${OTHERVAR}"
2685 VAR2 = "The version is ${PV}"
2686
2687- *Conditional Assignment (?=):* Conditional assignment is used to
2688 assign a value to a variable, but only when the variable is currently
2689 unset. Use the question mark followed by the equal sign (``?=``) to
2690 make a "soft" assignment used for conditional assignment. Typically,
2691 "soft" assignments are used in the ``local.conf`` file for variables
2692 that are allowed to come through from the external environment.
2693
2694 Here is an example where ``VAR1`` is set to "New value" if it is
2695 currently empty. However, if ``VAR1`` has already been set, it
2696 remains unchanged::
2697
2698 VAR1 ?= "New value"
2699
2700 In this next example, ``VAR1`` is left with the value "Original value"::
2701
2702 VAR1 = "Original value"
2703 VAR1 ?= "New value"
2704
2705- *Appending (+=):* Use the plus character followed by the equals sign
2706 (``+=``) to append values to existing variables.
2707
2708 .. note::
2709
2710 This operator adds a space between the existing content of the
2711 variable and the new content.
2712
2713 Here is an example::
2714
2715 SRC_URI += "file://fix-makefile.patch"
2716
2717- *Prepending (=+):* Use the equals sign followed by the plus character
2718 (``=+``) to prepend values to existing variables.
2719
2720 .. note::
2721
2722 This operator adds a space between the new content and the
2723 existing content of the variable.
2724
2725 Here is an example::
2726
2727 VAR =+ "Starts"
2728
2729- *Appending (:append):* Use the ``:append`` operator to append values
2730 to existing variables. This operator does not add any additional
2731 space. Also, the operator is applied after all the ``+=``, and ``=+``
2732 operators have been applied and after all ``=`` assignments have
2733 occurred. This means that if ``:append`` is used in a recipe, it can
2734 only be overridden by another layer using the special ``:remove``
2735 operator, which in turn will prevent further layers from adding it back.
2736
2737 The following example shows the space being explicitly added to the
2738 start to ensure the appended value is not merged with the existing
2739 value::
2740
2741 CFLAGS:append = " --enable-important-feature"
2742
2743 You can also use
2744 the ``:append`` operator with overrides, which results in the actions
2745 only being performed for the specified target or machine::
2746
2747 CFLAGS:append:sh4 = " --enable-important-sh4-specific-feature"
2748
2749- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend
2750 values to existing variables. This operator does not add any
2751 additional space. Also, the operator is applied after all the ``+=``,
2752 and ``=+`` operators have been applied and after all ``=``
2753 assignments have occurred.
2754
2755 The following example shows the space being explicitly added to the
2756 end to ensure the prepended value is not merged with the existing
2757 value::
2758
2759 CFLAGS:prepend = "-I${S}/myincludes "
2760
2761 You can also use the
2762 ``:prepend`` operator with overrides, which results in the actions
2763 only being performed for the specified target or machine::
2764
2765 CFLAGS:prepend:sh4 = "-I${S}/myincludes "
2766
2767- *Overrides:* You can use overrides to set a value conditionally,
2768 typically based on how the recipe is being built. For example, to set
2769 the :term:`KBRANCH` variable's
2770 value to "standard/base" for any target
2771 :term:`MACHINE`, except for
2772 qemuarm where it should be set to "standard/arm-versatile-926ejs",
2773 you would do the following::
2774
2775 KBRANCH = "standard/base"
2776 KBRANCH:qemuarm = "standard/arm-versatile-926ejs"
2777
2778 Overrides are also used to separate
2779 alternate values of a variable in other situations. For example, when
2780 setting variables such as
2781 :term:`FILES` and
2782 :term:`RDEPENDS` that are
2783 specific to individual packages produced by a recipe, you should
2784 always use an override that specifies the name of the package.
2785
2786- *Indentation:* Use spaces for indentation rather than tabs. For
2787 shell functions, both currently work. However, it is a policy
2788 decision of the Yocto Project to use tabs in shell functions. Realize
2789 that some layers have a policy to use spaces for all indentation.
2790
2791- *Using Python for Complex Operations:* For more advanced processing,
2792 it is possible to use Python code during variable assignments (e.g.
2793 search and replacement on a variable).
2794
2795 You indicate Python code using the ``${@python_code}`` syntax for the
2796 variable assignment::
2797
2798 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2799
2800- *Shell Function Syntax:* Write shell functions as if you were writing
2801 a shell script when you describe a list of actions to take. You
2802 should ensure that your script works with a generic ``sh`` and that
2803 it does not require any ``bash`` or other shell-specific
2804 functionality. The same considerations apply to various system
2805 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2806 might wish to use. If in doubt, you should check with multiple
2807 implementations --- including those from BusyBox.
2808
2809Adding a New Machine
2810====================
2811
2812Adding a new machine to the Yocto Project is a straightforward process.
2813This section describes how to add machines that are similar to those
2814that the Yocto Project already supports.
2815
2816.. note::
2817
2818 Although well within the capabilities of the Yocto Project, adding a
2819 totally new architecture might require changes to ``gcc``/``glibc``
2820 and to the site information, which is beyond the scope of this
2821 manual.
2822
2823For a complete example that shows how to add a new machine, see the
2824":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2825section in the Yocto Project Board Support Package (BSP) Developer's
2826Guide.
2827
2828Adding the Machine Configuration File
2829-------------------------------------
2830
2831To add a new machine, you need to add a new machine configuration file
2832to the layer's ``conf/machine`` directory. This configuration file
2833provides details about the device you are adding.
2834
2835The OpenEmbedded build system uses the root name of the machine
2836configuration file to reference the new machine. For example, given a
2837machine configuration file named ``crownbay.conf``, the build system
2838recognizes the machine as "crownbay".
2839
2840The most important variables you must set in your machine configuration
2841file or include from a lower-level configuration file are as follows:
2842
2843- :term:`TARGET_ARCH` (e.g. "arm")
2844
2845- ``PREFERRED_PROVIDER_virtual/kernel``
2846
2847- :term:`MACHINE_FEATURES` (e.g. "apm screen wifi")
2848
2849You might also need these variables:
2850
2851- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
2852
2853- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
2854
2855- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
2856
2857You can find full details on these variables in the reference section.
2858You can leverage existing machine ``.conf`` files from
2859``meta-yocto-bsp/conf/machine/``.
2860
2861Adding a Kernel for the Machine
2862-------------------------------
2863
2864The OpenEmbedded build system needs to be able to build a kernel for the
2865machine. You need to either create a new kernel recipe for this machine,
2866or extend an existing kernel recipe. You can find several kernel recipe
2867examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2868you can use as references.
2869
2870If you are creating a new kernel recipe, normal recipe-writing rules
2871apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
2872necessary patches and set :term:`S` to point at the source code. You need to
2873create a :ref:`ref-tasks-configure` task that configures the unpacked kernel with
2874a ``defconfig`` file. You can do this by using a ``make defconfig``
2875command or, more commonly, by copying in a suitable ``defconfig`` file
2876and then running ``make oldconfig``. By making use of ``inherit kernel``
2877and potentially some of the ``linux-*.inc`` files, most other
2878functionality is centralized and the defaults of the class normally work
2879well.
2880
2881If you are extending an existing kernel recipe, it is usually a matter
2882of adding a suitable ``defconfig`` file. The file needs to be added into
2883a location similar to ``defconfig`` files used for other machines in a
2884given kernel recipe. A possible way to do this is by listing the file in
2885the :term:`SRC_URI` and adding the machine to the expression in
2886:term:`COMPATIBLE_MACHINE`::
2887
2888 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2889
2890For more information on ``defconfig`` files, see the
2891":ref:`kernel-dev/common:changing the configuration`"
2892section in the Yocto Project Linux Kernel Development Manual.
2893
2894Adding a Formfactor Configuration File
2895--------------------------------------
2896
2897A formfactor configuration file provides information about the target
2898hardware for which the image is being built and information that the
2899build system cannot obtain from other sources such as the kernel. Some
2900examples of information contained in a formfactor configuration file
2901include framebuffer orientation, whether or not the system has a
2902keyboard, the positioning of the keyboard in relation to the screen, and
2903the screen resolution.
2904
2905The build system uses reasonable defaults in most cases. However, if
2906customization is necessary, you need to create a ``machconfig`` file in
2907the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2908contains directories for specific machines such as ``qemuarm`` and
2909``qemux86``. For information about the settings available and the
2910defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2911found in the same area.
2912
2913Following is an example for "qemuarm" machine::
2914
2915 HAVE_TOUCHSCREEN=1
2916 HAVE_KEYBOARD=1
2917 DISPLAY_CAN_ROTATE=0
2918 DISPLAY_ORIENTATION=0
2919 #DISPLAY_WIDTH_PIXELS=640
2920 #DISPLAY_HEIGHT_PIXELS=480
2921 #DISPLAY_BPP=16
2922 DISPLAY_DPI=150
2923 DISPLAY_SUBPIXEL_ORDER=vrgb
2924
2925Upgrading Recipes
2926=================
2927
2928Over time, upstream developers publish new versions for software built
2929by layer recipes. It is recommended to keep recipes up-to-date with
2930upstream version releases.
2931
2932While there are several methods to upgrade a recipe, you might
2933consider checking on the upgrade status of a recipe first. You can do so
2934using the ``devtool check-upgrade-status`` command. See the
2935":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2936section in the Yocto Project Reference Manual for more information.
2937
2938The remainder of this section describes three ways you can upgrade a
2939recipe. You can use the Automated Upgrade Helper (AUH) to set up
2940automatic version upgrades. Alternatively, you can use
2941``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
2942you can manually upgrade a recipe by editing the recipe itself.
2943
2944Using the Auto Upgrade Helper (AUH)
2945-----------------------------------
2946
2947The AUH utility works in conjunction with the OpenEmbedded build system
2948in order to automatically generate upgrades for recipes based on new
2949versions being published upstream. Use AUH when you want to create a
2950service that performs the upgrades automatically and optionally sends
2951you an email with the results.
2952
2953AUH allows you to update several recipes with a single use. You can also
2954optionally perform build and integration tests using images with the
2955results saved to your hard drive and emails of results optionally sent
2956to recipe maintainers. Finally, AUH creates Git commits with appropriate
2957commit messages in the layer's tree for the changes made to recipes.
2958
2959.. note::
2960
2961 In some conditions, you should not use AUH to upgrade recipes
2962 and should instead use either ``devtool upgrade`` or upgrade your
2963 recipes manually:
2964
2965 - When AUH cannot complete the upgrade sequence. This situation
2966 usually results because custom patches carried by the recipe
2967 cannot be automatically rebased to the new version. In this case,
2968 ``devtool upgrade`` allows you to manually resolve conflicts.
2969
2970 - When for any reason you want fuller control over the upgrade
2971 process. For example, when you want special arrangements for
2972 testing.
2973
2974The following steps describe how to set up the AUH utility:
2975
29761. *Be Sure the Development Host is Set Up:* You need to be sure that
2977 your development host is set up to use the Yocto Project. For
2978 information on how to set up your host, see the
2979 ":ref:`dev-manual/start:Preparing the Build Host`" section.
2980
29812. *Make Sure Git is Configured:* The AUH utility requires Git to be
2982 configured because AUH uses Git to save upgrades. Thus, you must have
2983 Git user and email configured. The following command shows your
2984 configurations::
2985
2986 $ git config --list
2987
2988 If you do not have the user and
2989 email configured, you can use the following commands to do so::
2990
2991 $ git config --global user.name some_name
2992 $ git config --global user.email username@domain.com
2993
29943. *Clone the AUH Repository:* To use AUH, you must clone the repository
2995 onto your development host. The following command uses Git to create
2996 a local copy of the repository on your system::
2997
2998 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
2999 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
3000 remote: Compressing objects: 100% (300/300), done.
3001 remote: Total 768 (delta 499), reused 703 (delta 434)
3002 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
3003 Resolving deltas: 100% (499/499), done.
3004 Checking connectivity... done.
3005
3006 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
3007 :term:`Poky` repositories.
3008
30094. *Create a Dedicated Build Directory:* Run the :ref:`structure-core-script`
3010 script to create a fresh :term:`Build Directory` that you use exclusively
3011 for running the AUH utility::
3012
3013 $ cd poky
3014 $ source oe-init-build-env your_AUH_build_directory
3015
3016 Re-using an existing :term:`Build Directory` and its configurations is not
3017 recommended as existing settings could cause AUH to fail or behave
3018 undesirably.
3019
30205. *Make Configurations in Your Local Configuration File:* Several
3021 settings are needed in the ``local.conf`` file in the build
3022 directory you just created for AUH. Make these following
3023 configurations:
3024
3025 - If you want to enable :ref:`Build
3026 History <dev-manual/common-tasks:maintaining build output quality>`,
3027 which is optional, you need the following lines in the
3028 ``conf/local.conf`` file::
3029
3030 INHERIT =+ "buildhistory"
3031 BUILDHISTORY_COMMIT = "1"
3032
3033 With this configuration and a successful
3034 upgrade, a build history "diff" file appears in the
3035 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3036 your :term:`Build Directory`.
3037
3038 - If you want to enable testing through the
3039 :ref:`testimage <ref-classes-testimage>`
3040 class, which is optional, you need to have the following set in
3041 your ``conf/local.conf`` file::
3042
3043 INHERIT += "testimage"
3044
3045 .. note::
3046
3047 If your distro does not enable by default ptest, which Poky
3048 does, you need the following in your ``local.conf`` file::
3049
3050 DISTRO_FEATURES:append = " ptest"
3051
3052
30536. *Optionally Start a vncserver:* If you are running in a server
3054 without an X11 session, you need to start a vncserver::
3055
3056 $ vncserver :1
3057 $ export DISPLAY=:1
3058
30597. *Create and Edit an AUH Configuration File:* You need to have the
3060 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
3061 :term:`Build Directory`. You can find a sample configuration file in the
3062 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
3063
3064 Read through the sample file and make configurations as needed. For
3065 example, if you enabled build history in your ``local.conf`` as
3066 described earlier, you must enable it in ``upgrade-helper.conf``.
3067
3068 Also, if you are using the default ``maintainers.inc`` file supplied
3069 with Poky and located in ``meta-yocto`` and you do not set a
3070 "maintainers_whitelist" or "global_maintainer_override" in the
3071 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3072 the AUH command-line, the utility automatically sends out emails to
3073 all the default maintainers. Please avoid this.
3074
3075This next set of examples describes how to use the AUH:
3076
3077- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
3078 following form::
3079
3080 $ upgrade-helper.py recipe_name
3081
3082 For example, this command upgrades the ``xmodmap`` recipe::
3083
3084 $ upgrade-helper.py xmodmap
3085
3086- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
3087 specific recipe to a particular version, use the following form::
3088
3089 $ upgrade-helper.py recipe_name -t version
3090
3091 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
3092
3093 $ upgrade-helper.py xmodmap -t 1.2.3
3094
3095- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3096 Notifications:* To upgrade all recipes to their most recent versions
3097 and suppress the email notifications, use the following command::
3098
3099 $ upgrade-helper.py all
3100
3101- *Upgrading all Recipes to the Latest Versions and Send Email
3102 Notifications:* To upgrade all recipes to their most recent versions
3103 and send email messages to maintainers for each attempted recipe as
3104 well as a status email, use the following command::
3105
3106 $ upgrade-helper.py -e all
3107
3108Once you have run the AUH utility, you can find the results in the AUH
3109:term:`Build Directory`::
3110
3111 ${BUILDDIR}/upgrade-helper/timestamp
3112
3113The AUH utility
3114also creates recipe update commits from successful upgrade attempts in
3115the layer tree.
3116
3117You can easily set up to run the AUH utility on a regular basis by using
3118a cron job. See the
3119:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
3120file distributed with the utility for an example.
3121
3122Using ``devtool upgrade``
3123-------------------------
3124
3125As mentioned earlier, an alternative method for upgrading recipes to
3126newer versions is to use
3127:doc:`devtool upgrade </ref-manual/devtool-reference>`.
3128You can read about ``devtool upgrade`` in general in the
3129":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
3130section in the Yocto Project Application Development and the Extensible
3131Software Development Kit (eSDK) Manual.
3132
3133To see all the command-line options available with ``devtool upgrade``,
3134use the following help command::
3135
3136 $ devtool upgrade -h
3137
3138If you want to find out what version a recipe is currently at upstream
3139without any attempt to upgrade your local version of the recipe, you can
3140use the following command::
3141
3142 $ devtool latest-version recipe_name
3143
3144As mentioned in the previous section describing AUH, ``devtool upgrade``
3145works in a less-automated manner than AUH. Specifically,
3146``devtool upgrade`` only works on a single recipe that you name on the
3147command line, cannot perform build and integration testing using images,
3148and does not automatically generate commits for changes in the source
3149tree. Despite all these "limitations", ``devtool upgrade`` updates the
3150recipe file to the new upstream version and attempts to rebase custom
3151patches contained by the recipe as needed.
3152
3153.. note::
3154
3155 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3156 of a "wrapper" application for ``devtool upgrade``.
3157
3158A typical scenario involves having used Git to clone an upstream
3159repository that you use during build operations. Because you have built the
3160recipe in the past, the layer is likely added to your
3161configuration already. If for some reason, the layer is not added, you
3162could add it easily using the
3163":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3164script. For example, suppose you use the ``nano.bb`` recipe from the
3165``meta-oe`` layer in the ``meta-openembedded`` repository. For this
3166example, assume that the layer has been cloned into following area::
3167
3168 /home/scottrif/meta-openembedded
3169
3170The following command from your :term:`Build Directory` adds the layer to
3171your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
3172
3173 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3174 NOTE: Starting bitbake server...
3175 Parsing recipes: 100% |##########################################| Time: 0:00:55
3176 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3177 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3178 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3179 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3180 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3181
3182For this example, assume that the ``nano.bb`` recipe that
3183is upstream has a 2.9.3 version number. However, the version in the
3184local repository is 2.7.4. The following command from your build
3185directory automatically upgrades the recipe for you:
3186
3187.. note::
3188
3189 Using the ``-V`` option is not necessary. Omitting the version number causes
3190 ``devtool upgrade`` to upgrade the recipe to the most recent version.
3191
3192::
3193
3194 $ devtool upgrade nano -V 2.9.3
3195 NOTE: Starting bitbake server...
3196 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3197 Parsing recipes: 100% |##########################################| Time: 0:00:46
3198 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3199 NOTE: Extracting current version source...
3200 NOTE: Resolving any missing task queue dependencies
3201 .
3202 .
3203 .
3204 NOTE: Executing SetScene Tasks
3205 NOTE: Executing RunQueue Tasks
3206 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3207 Adding changed files: 100% |#####################################| Time: 0:00:00
3208 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3209 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3210
3211Continuing with this example, you can use ``devtool build`` to build the
3212newly upgraded recipe::
3213
3214 $ devtool build nano
3215 NOTE: Starting bitbake server...
3216 Loading cache: 100% |################################################################################################| Time: 0:00:01
3217 Loaded 2040 entries from dependency cache.
3218 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3219 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3220 NOTE: Resolving any missing task queue dependencies
3221 .
3222 .
3223 .
3224 NOTE: Executing SetScene Tasks
3225 NOTE: Executing RunQueue Tasks
3226 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3227 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3228
3229Within the ``devtool upgrade`` workflow, you can
3230deploy and test your rebuilt software. For this example,
3231however, running ``devtool finish`` cleans up the workspace once the
3232source in your workspace is clean. This usually means using Git to stage
3233and submit commits for the changes generated by the upgrade process.
3234
3235Once the tree is clean, you can clean things up in this example with the
3236following command from the ``${BUILDDIR}/workspace/sources/nano``
3237directory::
3238
3239 $ devtool finish nano meta-oe
3240 NOTE: Starting bitbake server...
3241 Loading cache: 100% |################################################################################################| Time: 0:00:00
3242 Loaded 2040 entries from dependency cache.
3243 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3244 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3245 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3246 NOTE: Updating recipe nano_2.9.3.bb
3247 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3248 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3249 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3250
3251
3252Using the ``devtool finish`` command cleans up the workspace and creates a patch
3253file based on your commits. The tool puts all patch files back into the
3254source directory in a sub-directory named ``nano`` in this case.
3255
3256Manually Upgrading a Recipe
3257---------------------------
3258
3259If for some reason you choose not to upgrade recipes using
3260:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or
3261by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``,
3262you can manually edit the recipe files to upgrade the versions.
3263
3264.. note::
3265
3266 Manually updating multiple recipes scales poorly and involves many
3267 steps. The recommendation to upgrade recipe versions is through AUH
3268 or ``devtool upgrade``, both of which automate some steps and provide
3269 guidance for others needed for the manual process.
3270
3271To manually upgrade recipe versions, follow these general steps:
3272
32731. *Change the Version:* Rename the recipe such that the version (i.e.
3274 the :term:`PV` part of the recipe name)
3275 changes appropriately. If the version is not part of the recipe name,
3276 change the value as it is set for :term:`PV` within the recipe itself.
3277
32782. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
3279 is fetched from Git or some other version control system, update
3280 :term:`SRCREV` to point to the
3281 commit hash that matches the new version.
3282
32833. *Build the Software:* Try to build the recipe using BitBake. Typical
3284 build failures include the following:
3285
3286 - License statements were updated for the new version. For this
3287 case, you need to review any changes to the license and update the
3288 values of :term:`LICENSE` and
3289 :term:`LIC_FILES_CHKSUM`
3290 as needed.
3291
3292 .. note::
3293
3294 License changes are often inconsequential. For example, the
3295 license text's copyright year might have changed.
3296
3297 - Custom patches carried by the older version of the recipe might
3298 fail to apply to the new version. For these cases, you need to
3299 review the failures. Patches might not be necessary for the new
3300 version of the software if the upgraded version has fixed those
3301 issues. If a patch is necessary and failing, you need to rebase it
3302 into the new version.
3303
33044. *Optionally Attempt to Build for Several Architectures:* Once you
3305 successfully build the new software for a given architecture, you
3306 could test the build for other architectures by changing the
3307 :term:`MACHINE` variable and
3308 rebuilding the software. This optional step is especially important
3309 if the recipe is to be released publicly.
3310
33115. *Check the Upstream Change Log or Release Notes:* Checking both these
3312 reveals if there are new features that could break
3313 backwards-compatibility. If so, you need to take steps to mitigate or
3314 eliminate that situation.
3315
33166. *Optionally Create a Bootable Image and Test:* If you want, you can
3317 test the new software by booting it onto actual hardware.
3318
33197. *Create a Commit with the Change in the Layer Repository:* After all
3320 builds work and any testing is successful, you can create commits for
3321 any changes in the layer holding your upgraded recipe.
3322
3323Finding Temporary Source Code
3324=============================
3325
3326You might find it helpful during development to modify the temporary
3327source code used by recipes to build packages. For example, suppose you
3328are developing a patch and you need to experiment a bit to figure out
3329your solution. After you have initially built the package, you can
3330iteratively tweak the source code, which is located in the
3331:term:`Build Directory`, and then you can force a re-compile and quickly
3332test your altered code. Once you settle on a solution, you can then preserve
3333your changes in the form of patches.
3334
3335During a build, the unpacked temporary source code used by recipes to
3336build packages is available in the :term:`Build Directory` as defined by the
3337:term:`S` variable. Below is the default value for the :term:`S` variable as
3338defined in the ``meta/conf/bitbake.conf`` configuration file in the
3339:term:`Source Directory`::
3340
3341 S = "${WORKDIR}/${BP}"
3342
3343You should be aware that many recipes override the
3344:term:`S` variable. For example, recipes that fetch their source from Git
3345usually set :term:`S` to ``${WORKDIR}/git``.
3346
3347.. note::
3348
3349 The :term:`BP` represents the base recipe name, which consists of the name
3350 and version::
3351
3352 BP = "${BPN}-${PV}"
3353
3354
3355The path to the work directory for the recipe
3356(:term:`WORKDIR`) is defined as
3357follows::
3358
3359 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3360
3361The actual directory depends on several things:
3362
3363- :term:`TMPDIR`: The top-level build
3364 output directory.
3365
3366- :term:`MULTIMACH_TARGET_SYS`:
3367 The target system identifier.
3368
3369- :term:`PN`: The recipe name.
3370
3371- :term:`EXTENDPE`: The epoch --- if
3372 :term:`PE` is not specified, which is
3373 usually the case for most recipes, then :term:`EXTENDPE` is blank.
3374
3375- :term:`PV`: The recipe version.
3376
3377- :term:`PR`: The recipe revision.
3378
3379As an example, assume a Source Directory top-level folder named
3380``poky``, a default :term:`Build Directory` at ``poky/build``, and a
3381``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3382recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
3383build system uses to build the package would be as follows::
3384
3385 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3386
3387Using Quilt in Your Workflow
3388============================
3389
3390`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
3391that allows you to capture source code changes without having a clean
3392source tree. This section outlines the typical workflow you can use to
3393modify source code, test changes, and then preserve the changes in the
3394form of a patch all using Quilt.
3395
3396.. note::
3397
3398 With regard to preserving changes to source files, if you clean a
3399 recipe or have :ref:`rm_work <ref-classes-rm-work>` enabled, the
3400 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
3401 as described in the Yocto Project Application Development and the
3402 Extensible Software Development Kit (eSDK) manual is a safer
3403 development flow than the flow that uses Quilt.
3404
3405Follow these general steps:
3406
34071. *Find the Source Code:* Temporary source code used by the
3408 OpenEmbedded build system is kept in the :term:`Build Directory`. See the
3409 ":ref:`dev-manual/common-tasks:finding temporary source code`" section to
3410 learn how to locate the directory that has the temporary source code for a
3411 particular package.
3412
34132. *Change Your Working Directory:* You need to be in the directory that
3414 has the temporary source code. That directory is defined by the
3415 :term:`S` variable.
3416
34173. *Create a New Patch:* Before modifying source code, you need to
3418 create a new patch. To create a new patch file, use ``quilt new`` as
3419 below::
3420
3421 $ quilt new my_changes.patch
3422
34234. *Notify Quilt and Add Files:* After creating the patch, you need to
3424 notify Quilt about the files you plan to edit. You notify Quilt by
3425 adding the files to the patch you just created::
3426
3427 $ quilt add file1.c file2.c file3.c
3428
34295. *Edit the Files:* Make your changes in the source code to the files
3430 you added to the patch.
3431
34326. *Test Your Changes:* Once you have modified the source code, the
3433 easiest way to test your changes is by calling the :ref:`ref-tasks-compile`
3434 task as shown in the following example::
3435
3436 $ bitbake -c compile -f package
3437
3438 The ``-f`` or ``--force`` option forces the specified task to
3439 execute. If you find problems with your code, you can just keep
3440 editing and re-testing iteratively until things work as expected.
3441
3442 .. note::
3443
3444 All the modifications you make to the temporary source code disappear
3445 once you run the :ref:`ref-tasks-clean` or :ref:`ref-tasks-cleanall`
3446 tasks using BitBake (i.e. ``bitbake -c clean package`` and
3447 ``bitbake -c cleanall package``). Modifications will also disappear if
3448 you use the :ref:`rm_work <ref-classes-rm-work>` feature as described in
3449 the ":ref:`dev-manual/common-tasks:conserving disk space during builds`"
3450 section.
3451
34527. *Generate the Patch:* Once your changes work as expected, you need to
3453 use Quilt to generate the final patch that contains all your
3454 modifications.
3455 ::
3456
3457 $ quilt refresh
3458
3459 At this point, the
3460 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3461 ``file2.c``, and ``file3.c`` files.
3462
3463 You can find the resulting patch file in the ``patches/``
3464 subdirectory of the source (:term:`S`) directory.
3465
34668. *Copy the Patch File:* For simplicity, copy the patch file into a
3467 directory named ``files``, which you can create in the same directory
3468 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3469 file. Placing the patch here guarantees that the OpenEmbedded build
3470 system will find the patch. Next, add the patch into the :term:`SRC_URI`
3471 of the recipe. Here is an example::
3472
3473 SRC_URI += "file://my_changes.patch"
3474
3475Using a Development Shell
3476=========================
3477
3478When debugging certain commands or even when just editing packages,
3479``devshell`` can be a useful tool. When you invoke ``devshell``, all
3480tasks up to and including
3481:ref:`ref-tasks-patch` are run for the
3482specified target. Then, a new terminal is opened and you are placed in
3483``${``\ :term:`S`\ ``}``, the source
3484directory. In the new terminal, all the OpenEmbedded build-related
3485environment variables are still defined so you can use commands such as
3486``configure`` and ``make``. The commands execute just as if the
3487OpenEmbedded build system were executing them. Consequently, working
3488this way can be helpful when debugging a build or preparing software to
3489be used with the OpenEmbedded build system.
3490
3491Following is an example that uses ``devshell`` on a target named
3492``matchbox-desktop``::
3493
3494 $ bitbake matchbox-desktop -c devshell
3495
3496This command spawns a terminal with a shell prompt within the
3497OpenEmbedded build environment. The
3498:term:`OE_TERMINAL` variable
3499controls what type of shell is opened.
3500
3501For spawned terminals, the following occurs:
3502
3503- The ``PATH`` variable includes the cross-toolchain.
3504
3505- The ``pkgconfig`` variables find the correct ``.pc`` files.
3506
3507- The ``configure`` command finds the Yocto Project site files as well
3508 as any other necessary files.
3509
3510Within this environment, you can run configure or compile commands as if
3511they were being run by the OpenEmbedded build system itself. As noted
3512earlier, the working directory also automatically changes to the Source
3513Directory (:term:`S`).
3514
3515To manually run a specific task using ``devshell``, run the
3516corresponding ``run.*`` script in the
3517``${``\ :term:`WORKDIR`\ ``}/temp``
3518directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
3519not exist, which would be the case if the task was skipped by way of the
3520sstate cache, you can create the task by first running it outside of the
3521``devshell``::
3522
3523 $ bitbake -c task
3524
3525.. note::
3526
3527 - Execution of a task's ``run.*`` script and BitBake's execution of
3528 a task are identical. In other words, running the script re-runs
3529 the task just as it would be run using the ``bitbake -c`` command.
3530
3531 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3532 symbolic link (symlink) to the most recent version of that file.
3533
3534Remember, that the ``devshell`` is a mechanism that allows you to get
3535into the BitBake task execution environment. And as such, all commands
3536must be called just as BitBake would call them. That means you need to
3537provide the appropriate options for cross-compilation and so forth as
3538applicable.
3539
3540When you are finished using ``devshell``, exit the shell or close the
3541terminal window.
3542
3543.. note::
3544
3545 - It is worth remembering that when using ``devshell`` you need to
3546 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3547 instead of just using ``gcc``. The same applies to other
3548 applications such as ``binutils``, ``libtool`` and so forth.
3549 BitBake sets up environment variables such as :term:`CC` to assist
3550 applications, such as ``make`` to find the correct tools.
3551
3552 - It is also worth noting that ``devshell`` still works over X11
3553 forwarding and similar situations.
3554
3555Using a Python Development Shell
3556================================
3557
3558Similar to working within a development shell as described in the
3559previous section, you can also spawn and work within an interactive
3560Python development shell. When debugging certain commands or even when
3561just editing packages, ``pydevshell`` can be a useful tool. When you
3562invoke the ``pydevshell`` task, all tasks up to and including
3563:ref:`ref-tasks-patch` are run for the
3564specified target. Then a new terminal is opened. Additionally, key
3565Python objects and code are available in the same way they are to
3566BitBake tasks, in particular, the data store 'd'. So, commands such as
3567the following are useful when exploring the data store and running
3568functions::
3569
3570 pydevshell> d.getVar("STAGING_DIR")
3571 '/media/build1/poky/build/tmp/sysroots'
3572 pydevshell> d.getVar("STAGING_DIR", False)
3573 '${TMPDIR}/sysroots'
3574 pydevshell> d.setVar("FOO", "bar")
3575 pydevshell> d.getVar("FOO")
3576 'bar'
3577 pydevshell> d.delVar("FOO")
3578 pydevshell> d.getVar("FOO")
3579 pydevshell> bb.build.exec_func("do_unpack", d)
3580 pydevshell>
3581
3582See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:functions you can call from within python`"
3583section in the BitBake User Manual for details about available functions.
3584
3585The commands execute just as if the OpenEmbedded build
3586system were executing them. Consequently, working this way can be
3587helpful when debugging a build or preparing software to be used with the
3588OpenEmbedded build system.
3589
3590Following is an example that uses ``pydevshell`` on a target named
3591``matchbox-desktop``::
3592
3593 $ bitbake matchbox-desktop -c pydevshell
3594
3595This command spawns a terminal and places you in an interactive Python
3596interpreter within the OpenEmbedded build environment. The
3597:term:`OE_TERMINAL` variable
3598controls what type of shell is opened.
3599
3600When you are finished using ``pydevshell``, you can exit the shell
3601either by using Ctrl+d or closing the terminal window.
3602
3603Building
3604========
3605
3606This section describes various build procedures, such as the steps
3607needed for a simple build, building a target for multiple configurations,
3608generating an image for more than one machine, and so forth.
3609
3610Building a Simple Image
3611-----------------------
3612
3613In the development environment, you need to build an image whenever you
3614change hardware support, add or change system libraries, or add or
3615change services that have dependencies. There are several methods that allow
3616you to build an image within the Yocto Project. This section presents
3617the basic steps you need to build a simple image using BitBake from a
3618build host running Linux.
3619
3620.. note::
3621
3622 - For information on how to build an image using
3623 :term:`Toaster`, see the
3624 :doc:`/toaster-manual/index`.
3625
3626 - For information on how to use ``devtool`` to build images, see the
3627 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
3628 section in the Yocto Project Application Development and the
3629 Extensible Software Development Kit (eSDK) manual.
3630
3631 - For a quick example on how to build an image using the
3632 OpenEmbedded build system, see the
3633 :doc:`/brief-yoctoprojectqs/index` document.
3634
3635The build process creates an entire Linux distribution from source and
3636places it in your :term:`Build Directory` under ``tmp/deploy/images``. For
3637detailed information on the build process using BitBake, see the
3638":ref:`overview-manual/concepts:images`" section in the Yocto Project Overview
3639and Concepts Manual.
3640
3641The following figure and list overviews the build process:
3642
3643.. image:: figures/bitbake-build-flow.png
3644 :width: 100%
3645
36461. *Set up Your Host Development System to Support Development Using the
3647 Yocto Project*: See the ":doc:`start`" section for options on how to get a
3648 build host ready to use the Yocto Project.
3649
36502. *Initialize the Build Environment:* Initialize the build environment
3651 by sourcing the build environment script (i.e.
3652 :ref:`structure-core-script`)::
3653
3654 $ source oe-init-build-env [build_dir]
3655
3656 When you use the initialization script, the OpenEmbedded build system
3657 uses ``build`` as the default :term:`Build Directory` in your current work
3658 directory. You can use a `build_dir` argument with the script to
3659 specify a different :term:`Build Directory`.
3660
3661 .. note::
3662
3663 A common practice is to use a different :term:`Build Directory` for
3664 different targets; for example, ``~/build/x86`` for a ``qemux86``
3665 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
3666 event, it's typically cleaner to locate the :term:`Build Directory`
3667 somewhere outside of your source directory.
3668
36693. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
3670 ``conf/local.conf`` configuration file, which is found in the
3671 :term:`Build Directory`, is set up how you want it. This file defines many
3672 aspects of the build environment including the target machine architecture
3673 through the :term:`MACHINE` variable, the packaging format used during
3674 the build (:term:`PACKAGE_CLASSES`), and a centralized tarball download
3675 directory through the :term:`DL_DIR` variable.
3676
36774. *Build the Image:* Build the image using the ``bitbake`` command::
3678
3679 $ bitbake target
3680
3681 .. note::
3682
3683 For information on BitBake, see the :doc:`bitbake:index`.
3684
3685 The target is the name of the recipe you want to build. Common
3686 targets are the images in ``meta/recipes-core/images``,
3687 ``meta/recipes-sato/images``, and so forth all found in the
3688 :term:`Source Directory`. Alternatively, the target
3689 can be the name of a recipe for a specific piece of software such as
3690 BusyBox. For more details about the images the OpenEmbedded build
3691 system supports, see the
3692 ":ref:`ref-manual/images:Images`" chapter in the Yocto
3693 Project Reference Manual.
3694
3695 As an example, the following command builds the
3696 ``core-image-minimal`` image::
3697
3698 $ bitbake core-image-minimal
3699
3700 Once an
3701 image has been built, it often needs to be installed. The images and
3702 kernels built by the OpenEmbedded build system are placed in the
3703 :term:`Build Directory` in ``tmp/deploy/images``. For information on how to
3704 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
3705 :doc:`/sdk-manual/index` manual. For
3706 information about how to install these images, see the documentation
3707 for your particular board or machine.
3708
3709Building Images for Multiple Targets Using Multiple Configurations
3710------------------------------------------------------------------
3711
3712You can use a single ``bitbake`` command to build multiple images or
3713packages for different targets where each image or package requires a
3714different configuration (multiple configuration builds). The builds, in
3715this scenario, are sometimes referred to as "multiconfigs", and this
3716section uses that term throughout.
3717
3718This section describes how to set up for multiple configuration builds
3719and how to account for cross-build dependencies between the
3720multiconfigs.
3721
3722Setting Up and Running a Multiple Configuration Build
3723~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3724
3725To accomplish a multiple configuration build, you must define each
3726target's configuration separately using a parallel configuration file in
3727the :term:`Build Directory` or configuration directory within a layer, and you
3728must follow a required file hierarchy. Additionally, you must enable the
3729multiple configuration builds in your ``local.conf`` file.
3730
3731Follow these steps to set up and execute multiple configuration builds:
3732
3733- *Create Separate Configuration Files*: You need to create a single
3734 configuration file for each build target (each multiconfig).
3735 The configuration definitions are implementation dependent but often
3736 each configuration file will define the machine and the
3737 temporary directory BitBake uses for the build. Whether the same
3738 temporary directory (:term:`TMPDIR`) can be shared will depend on what is
3739 similar and what is different between the configurations. Multiple MACHINE
3740 targets can share the same (:term:`TMPDIR`) as long as the rest of the
3741 configuration is the same, multiple DISTRO settings would need separate
3742 (:term:`TMPDIR`) directories.
3743
3744 For example, consider a scenario with two different multiconfigs for the same
3745 :term:`MACHINE`: "qemux86" built
3746 for two distributions such as "poky" and "poky-lsb". In this case,
3747 you would need to use the different :term:`TMPDIR`.
3748
3749 Here is an example showing the minimal statements needed in a
3750 configuration file for a "qemux86" target whose temporary build
3751 directory is ``tmpmultix86``::
3752
3753 MACHINE = "qemux86"
3754 TMPDIR = "${TOPDIR}/tmpmultix86"
3755
3756 The location for these multiconfig configuration files is specific.
3757 They must reside in the current :term:`Build Directory` in a sub-directory of
3758 ``conf`` named ``multiconfig`` or within a layer's ``conf`` directory
3759 under a directory named ``multiconfig``. Following is an example that defines
3760 two configuration files for the "x86" and "arm" multiconfigs:
3761
3762 .. image:: figures/multiconfig_files.png
3763 :align: center
3764 :width: 50%
3765
3766 The usual :term:`BBPATH` search path is used to locate multiconfig files in
3767 a similar way to other conf files.
3768
3769- *Add the BitBake Multi-configuration Variable to the Local
3770 Configuration File*: Use the
3771 :term:`BBMULTICONFIG`
3772 variable in your ``conf/local.conf`` configuration file to specify
3773 each multiconfig. Continuing with the example from the previous
3774 figure, the :term:`BBMULTICONFIG` variable needs to enable two
3775 multiconfigs: "x86" and "arm" by specifying each configuration file::
3776
3777 BBMULTICONFIG = "x86 arm"
3778
3779 .. note::
3780
3781 A "default" configuration already exists by definition. This
3782 configuration is named: "" (i.e. empty string) and is defined by
3783 the variables coming from your ``local.conf``
3784 file. Consequently, the previous example actually adds two
3785 additional configurations to your build: "arm" and "x86" along
3786 with "".
3787
3788- *Launch BitBake*: Use the following BitBake command form to launch
3789 the multiple configuration build::
3790
3791 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3792
3793 For the example in this section, the following command applies::
3794
3795 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3796
3797 The previous BitBake command builds a ``core-image-minimal`` image
3798 that is configured through the ``x86.conf`` configuration file, a
3799 ``core-image-sato`` image that is configured through the ``arm.conf``
3800 configuration file and a ``core-image-base`` that is configured
3801 through your ``local.conf`` configuration file.
3802
3803.. note::
3804
3805 Support for multiple configuration builds in the Yocto Project &DISTRO;
3806 (&DISTRO_NAME;) Release does not include Shared State (sstate)
3807 optimizations. Consequently, if a build uses the same object twice
3808 in, for example, two different :term:`TMPDIR`
3809 directories, the build either loads from an existing sstate cache for
3810 that build at the start or builds the object fresh.
3811
3812Enabling Multiple Configuration Build Dependencies
3813~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3814
3815Sometimes dependencies can exist between targets (multiconfigs) in a
3816multiple configuration build. For example, suppose that in order to
3817build a ``core-image-sato`` image for an "x86" multiconfig, the root
3818filesystem of an "arm" multiconfig must exist. This dependency is
3819essentially that the
3820:ref:`ref-tasks-image` task in the
3821``core-image-sato`` recipe depends on the completion of the
3822:ref:`ref-tasks-rootfs` task of the
3823``core-image-minimal`` recipe.
3824
3825To enable dependencies in a multiple configuration build, you must
3826declare the dependencies in the recipe using the following statement
3827form::
3828
3829 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3830
3831To better show how to use this statement, consider the example scenario
3832from the first paragraph of this section. The following statement needs
3833to be added to the recipe that builds the ``core-image-sato`` image::
3834
3835 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3836
3837In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
3838task on which the :ref:`ref-tasks-image` task in the recipe depends is the
3839:ref:`ref-tasks-rootfs` task from the ``core-image-minimal`` recipe associated
3840with the "arm" multiconfig.
3841
3842Once you set up this dependency, you can build the "x86" multiconfig
3843using a BitBake command as follows::
3844
3845 $ bitbake mc:x86:core-image-sato
3846
3847This command executes all the tasks needed to create the
3848``core-image-sato`` image for the "x86" multiconfig. Because of the
3849dependency, BitBake also executes through the :ref:`ref-tasks-rootfs` task for the
3850"arm" multiconfig build.
3851
3852Having a recipe depend on the root filesystem of another build might not
3853seem that useful. Consider this change to the statement in the
3854``core-image-sato`` recipe::
3855
3856 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3857
3858In this case, BitBake must
3859create the ``core-image-minimal`` image for the "arm" build since the
3860"x86" build depends on it.
3861
3862Because "x86" and "arm" are enabled for multiple configuration builds
3863and have separate configuration files, BitBake places the artifacts for
3864each build in the respective temporary build directories (i.e.
3865:term:`TMPDIR`).
3866
3867Building an Initial RAM Filesystem (Initramfs) Image
3868----------------------------------------------------
3869
3870An initial RAM filesystem (:term:`Initramfs`) image provides a temporary root
3871filesystem used for early system initialization, typically providing tools and
3872loading modules needed to locate and mount the final root filesystem.
3873
3874Follow these steps to create an :term:`Initramfs` image:
3875
38761. *Create the Initramfs Image Recipe:* You can reference the
3877 ``core-image-minimal-initramfs.bb`` recipe found in the
3878 ``meta/recipes-core`` directory of the :term:`Source Directory`
3879 as an example from which to work.
3880
38812. *Decide if You Need to Bundle the Initramfs Image Into the Kernel
3882 Image:* If you want the :term:`Initramfs` image that is built to be bundled
3883 in with the kernel image, set the :term:`INITRAMFS_IMAGE_BUNDLE`
3884 variable to ``"1"`` in your ``local.conf`` configuration file and set the
3885 :term:`INITRAMFS_IMAGE` variable in the recipe that builds the kernel image.
3886
3887 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the :term:`Initramfs`
3888 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
3889 :term:`Initramfs` image is then passed to the kernel's ``Makefile`` using the
3890 :term:`CONFIG_INITRAMFS_SOURCE` variable, allowing the :term:`Initramfs`
3891 image to be built into the kernel normally.
3892
38933. *Optionally Add Items to the Initramfs Image Through the Initramfs
3894 Image Recipe:* If you add items to the :term:`Initramfs` image by way of its
3895 recipe, you should use :term:`PACKAGE_INSTALL` rather than
3896 :term:`IMAGE_INSTALL`. :term:`PACKAGE_INSTALL` gives more direct control of
3897 what is added to the image as compared to the defaults you might not
3898 necessarily want that are set by the :ref:`image <ref-classes-image>`
3899 or :ref:`core-image <ref-classes-core-image>` classes.
3900
39014. *Build the Kernel Image and the Initramfs Image:* Build your kernel
3902 image using BitBake. Because the :term:`Initramfs` image recipe is a
3903 dependency of the kernel image, the :term:`Initramfs` image is built as well
3904 and bundled with the kernel image if you used the
3905 :term:`INITRAMFS_IMAGE_BUNDLE` variable described earlier.
3906
3907Bundling an Initramfs Image From a Separate Multiconfig
3908~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3909
3910There may be a case where we want to build an :term:`Initramfs` image which does not
3911inherit the same distro policy as our main image, for example, we may want
3912our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our :term:`Initramfs`
3913image to keep a smaller footprint. However, by performing the steps mentioned
3914above the :term:`Initramfs` image will inherit ``TCLIBC="glibc"`` without allowing us
3915to override it.
3916
3917To achieve this, you need to perform some additional steps:
3918
39191. *Create a multiconfig for your Initramfs image:* You can perform the steps
3920 on ":ref:`dev-manual/common-tasks:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
3921 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
3922 contains the variables::
3923
3924 TMPDIR="${TOPDIR}/tmp-initramfscfg"
3925 TCLIBC="musl"
3926
39272. *Set additional Initramfs variables on your main configuration:*
3928 Additionally, on your main configuration (``local.conf``) you need to set the
3929 variables::
3930
3931 INITRAMFS_MULTICONFIG = "initramfscfg"
3932 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
3933
3934 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
3935 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
3936 to be built coming from the ``initramfscfg`` multiconfig, and to let the
3937 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
3938
3939 Building a system with such configuration will build the kernel using the
3940 main configuration but the :ref:`ref-tasks-bundle_initramfs` task will grab the
3941 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
3942 instead, resulting in a musl based :term:`Initramfs` image bundled in the kernel
3943 but a glibc based main image.
3944
3945 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
3946 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
3947
3948
3949Building a Tiny System
3950----------------------
3951
3952Very small distributions have some significant advantages such as
3953requiring less on-die or in-package memory (cheaper), better performance
3954through efficient cache usage, lower power requirements due to less
3955memory, faster boot times, and reduced development overhead. Some
3956real-world examples where a very small distribution gives you distinct
3957advantages are digital cameras, medical devices, and small headless
3958systems.
3959
3960This section presents information that shows you how you can trim your
3961distribution to even smaller sizes than the ``poky-tiny`` distribution,
3962which is around 5 Mbytes, that can be built out-of-the-box using the
3963Yocto Project.
3964
3965Tiny System Overview
3966~~~~~~~~~~~~~~~~~~~~
3967
3968The following list presents the overall steps you need to consider and
3969perform to create distributions with smaller root filesystems, achieve
3970faster boot times, maintain your critical functionality, and avoid
3971initial RAM disks:
3972
3973- :ref:`Determine your goals and guiding principles
3974 <dev-manual/common-tasks:goals and guiding principles>`
3975
3976- :ref:`dev-manual/common-tasks:understand what contributes to your image size`
3977
3978- :ref:`Reduce the size of the root filesystem
3979 <dev-manual/common-tasks:trim the root filesystem>`
3980
3981- :ref:`Reduce the size of the kernel <dev-manual/common-tasks:trim the kernel>`
3982
3983- :ref:`dev-manual/common-tasks:remove package management requirements`
3984
3985- :ref:`dev-manual/common-tasks:look for other ways to minimize size`
3986
3987- :ref:`dev-manual/common-tasks:iterate on the process`
3988
3989Goals and Guiding Principles
3990~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3991
3992Before you can reach your destination, you need to know where you are
3993going. Here is an example list that you can use as a guide when creating
3994very small distributions:
3995
3996- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
3997 less and a root filesystem that is 3 Mbytes or less).
3998
3999- Find the areas that are currently taking 90% of the space and
4000 concentrate on reducing those areas.
4001
4002- Do not create any difficult "hacks" to achieve your goals.
4003
4004- Leverage the device-specific options.
4005
4006- Work in a separate layer so that you keep changes isolated. For
4007 information on how to create layers, see the
4008 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
4009
4010Understand What Contributes to Your Image Size
4011~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4012
4013It is easiest to have something to start with when creating your own
4014distribution. You can use the Yocto Project out-of-the-box to create the
4015``poky-tiny`` distribution. Ultimately, you will want to make changes in
4016your own distribution that are likely modeled after ``poky-tiny``.
4017
4018.. note::
4019
4020 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
4021 ``local.conf`` file to "poky-tiny" as described in the
4022 ":ref:`dev-manual/common-tasks:creating your own distribution`"
4023 section.
4024
4025Understanding some memory concepts will help you reduce the system size.
4026Memory consists of static, dynamic, and temporary memory. Static memory
4027is the TEXT (code), DATA (initialized data in the code), and BSS
4028(uninitialized data) sections. Dynamic memory represents memory that is
4029allocated at runtime: stacks, hash tables, and so forth. Temporary
4030memory is recovered after the boot process. This memory consists of
4031memory used for decompressing the kernel and for the ``__init__``
4032functions.
4033
4034To help you see where you currently are with kernel and root filesystem
4035sizes, you can use two tools found in the :term:`Source Directory`
4036in the
4037``scripts/tiny/`` directory:
4038
4039- ``ksize.py``: Reports component sizes for the kernel build objects.
4040
4041- ``dirsize.py``: Reports component sizes for the root filesystem.
4042
4043This next tool and command help you organize configuration fragments and
4044view file dependencies in a human-readable form:
4045
4046- ``merge_config.sh``: Helps you manage configuration files and
4047 fragments within the kernel. With this tool, you can merge individual
4048 configuration fragments together. The tool allows you to make
4049 overrides and warns you of any missing configuration options. The
4050 tool is ideal for allowing you to iterate on configurations, create
4051 minimal configurations, and create configuration files for different
4052 machines without having to duplicate your process.
4053
4054 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4055 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4056 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4057 directory.
4058
4059 For more information on configuration fragments, see the
4060 ":ref:`kernel-dev/common:creating configuration fragments`"
4061 section in the Yocto Project Linux Kernel Development Manual.
4062
4063- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4064 with these options brings up a Dependency Explorer from which you can
4065 view file dependencies. Understanding these dependencies allows you
4066 to make informed decisions when cutting out various pieces of the
4067 kernel and root filesystem.
4068
4069Trim the Root Filesystem
4070~~~~~~~~~~~~~~~~~~~~~~~~
4071
4072The root filesystem is made up of packages for booting, libraries, and
4073applications. To change things, you can configure how the packaging
4074happens, which changes the way you build them. You can also modify the
4075filesystem itself or select a different filesystem.
4076
4077First, find out what is hogging your root filesystem by running the
4078``dirsize.py`` script from your root directory::
4079
4080 $ cd root-directory-of-image
4081 $ dirsize.py 100000 > dirsize-100k.log
4082 $ cat dirsize-100k.log
4083
4084You can apply a filter to the script to ignore files
4085under a certain size. The previous example filters out any files below
4086100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4087will be smaller by a relatively constant factor in a compressed root
4088filesystem. When you examine your log file, you can focus on areas of
4089the root filesystem that take up large amounts of memory.
4090
4091You need to be sure that what you eliminate does not cripple the
4092functionality you need. One way to see how packages relate to each other
4093is by using the Dependency Explorer UI with the BitBake command::
4094
4095 $ cd image-directory
4096 $ bitbake -u taskexp -g image
4097
4098Use the interface to
4099select potential packages you wish to eliminate and see their dependency
4100relationships.
4101
4102When deciding how to reduce the size, get rid of packages that result in
4103minimal impact on the feature set. For example, you might not need a VGA
4104display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4105instead of ``udev``.
4106
4107Use your ``local.conf`` file to make changes. For example, to eliminate
4108``udev`` and ``glib``, set the following in the local configuration
4109file::
4110
4111 VIRTUAL-RUNTIME_dev_manager = ""
4112
4113Finally, you should consider exactly the type of root filesystem you
4114need to meet your needs while also reducing its size. For example,
4115consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4116:term:`Initramfs` using ``initramfs``. Be aware that ``ext3`` requires a 1
4117Mbyte journal. If you are okay with running read-only, you do not need
4118this journal.
4119
4120.. note::
4121
4122 After each round of elimination, you need to rebuild your system and
4123 then use the tools to see the effects of your reductions.
4124
4125Trim the Kernel
4126~~~~~~~~~~~~~~~
4127
4128The kernel is built by including policies for hardware-independent
4129aspects. What subsystems do you enable? For what architecture are you
4130building? Which drivers do you build by default?
4131
4132.. note::
4133
4134 You can modify the kernel source if you want to help with boot time.
4135
4136Run the ``ksize.py`` script from the top-level Linux build directory to
4137get an idea of what is making up the kernel::
4138
4139 $ cd top-level-linux-build-directory
4140 $ ksize.py > ksize.log
4141 $ cat ksize.log
4142
4143When you examine the log, you will see how much space is taken up with
4144the built-in ``.o`` files for drivers, networking, core kernel files,
4145filesystem, sound, and so forth. The sizes reported by the tool are
4146uncompressed, and thus will be smaller by a relatively constant factor
4147in a compressed kernel image. Look to reduce the areas that are large
4148and taking up around the "90% rule."
4149
4150To examine, or drill down, into any particular area, use the ``-d``
4151option with the script::
4152
4153 $ ksize.py -d > ksize.log
4154
4155Using this option
4156breaks out the individual file information for each area of the kernel
4157(e.g. drivers, networking, and so forth).
4158
4159Use your log file to see what you can eliminate from the kernel based on
4160features you can let go. For example, if you are not going to need
4161sound, you do not need any drivers that support sound.
4162
4163After figuring out what to eliminate, you need to reconfigure the kernel
4164to reflect those changes during the next build. You could run
4165``menuconfig`` and make all your changes at once. However, that makes it
4166difficult to see the effects of your individual eliminations and also
4167makes it difficult to replicate the changes for perhaps another target
4168device. A better method is to start with no configurations using
4169``allnoconfig``, create configuration fragments for individual changes,
4170and then manage the fragments into a single configuration file using
4171``merge_config.sh``. The tool makes it easy for you to iterate using the
4172configuration change and build cycle.
4173
4174Each time you make configuration changes, you need to rebuild the kernel
4175and check to see what impact your changes had on the overall size.
4176
4177Remove Package Management Requirements
4178~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4179
4180Packaging requirements add size to the image. One way to reduce the size
4181of the image is to remove all the packaging requirements from the image.
4182This reduction includes both removing the package manager and its unique
4183dependencies as well as removing the package management data itself.
4184
4185To eliminate all the packaging requirements for an image, be sure that
4186"package-management" is not part of your
4187:term:`IMAGE_FEATURES`
4188statement for the image. When you remove this feature, you are removing
4189the package manager as well as its dependencies from the root
4190filesystem.
4191
4192Look for Other Ways to Minimize Size
4193~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4194
4195Depending on your particular circumstances, other areas that you can
4196trim likely exist. The key to finding these areas is through tools and
4197methods described here combined with experimentation and iteration. Here
4198are a couple of areas to experiment with:
4199
4200- ``glibc``: In general, follow this process:
4201
4202 1. Remove ``glibc`` features from
4203 :term:`DISTRO_FEATURES`
4204 that you think you do not need.
4205
4206 2. Build your distribution.
4207
4208 3. If the build fails due to missing symbols in a package, determine
4209 if you can reconfigure the package to not need those features. For
4210 example, change the configuration to not support wide character
4211 support as is done for ``ncurses``. Or, if support for those
4212 characters is needed, determine what ``glibc`` features provide
4213 the support and restore the configuration.
4214
4215 4. Rebuild and repeat the process.
4216
4217- ``busybox``: For BusyBox, use a process similar as described for
4218 ``glibc``. A difference is you will need to boot the resulting system
4219 to see if you are able to do everything you expect from the running
4220 system. You need to be sure to integrate configuration fragments into
4221 Busybox because BusyBox handles its own core features and then allows
4222 you to add configuration fragments on top.
4223
4224Iterate on the Process
4225~~~~~~~~~~~~~~~~~~~~~~
4226
4227If you have not reached your goals on system size, you need to iterate
4228on the process. The process is the same. Use the tools and see just what
4229is taking up 90% of the root filesystem and the kernel. Decide what you
4230can eliminate without limiting your device beyond what you need.
4231
4232Depending on your system, a good place to look might be Busybox, which
4233provides a stripped down version of Unix tools in a single, executable
4234file. You might be able to drop virtual terminal services or perhaps
4235ipv6.
4236
4237Building Images for More than One Machine
4238-----------------------------------------
4239
4240A common scenario developers face is creating images for several
4241different machines that use the same software environment. In this
4242situation, it is tempting to set the tunings and optimization flags for
4243each build specifically for the targeted hardware (i.e. "maxing out" the
4244tunings). Doing so can considerably add to build times and package feed
4245maintenance collectively for the machines. For example, selecting tunes
4246that are extremely specific to a CPU core used in a system might enable
4247some micro optimizations in GCC for that particular system but would
4248otherwise not gain you much of a performance difference across the other
4249systems as compared to using a more general tuning across all the builds
4250(e.g. setting :term:`DEFAULTTUNE`
4251specifically for each machine's build). Rather than "max out" each
4252build's tunings, you can take steps that cause the OpenEmbedded build
4253system to reuse software across the various machines where it makes
4254sense.
4255
4256If build speed and package feed maintenance are considerations, you
4257should consider the points in this section that can help you optimize
4258your tunings to best consider build times and package feed maintenance.
4259
4260- *Share the :term:`Build Directory`:* If at all possible, share the
4261 :term:`TMPDIR` across builds. The Yocto Project supports switching between
4262 different :term:`MACHINE` values in the same :term:`TMPDIR`. This practice
4263 is well supported and regularly used by developers when building for
4264 multiple machines. When you use the same :term:`TMPDIR` for multiple
4265 machine builds, the OpenEmbedded build system can reuse the existing native
4266 and often cross-recipes for multiple machines. Thus, build time decreases.
4267
4268 .. note::
4269
4270 If :term:`DISTRO` settings change or fundamental configuration settings
4271 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
4272 Sharing :term:`TMPDIR` under these circumstances might work but since it is
4273 not guaranteed, you should use a clean :term:`TMPDIR`.
4274
4275- *Enable the Appropriate Package Architecture:* By default, the
4276 OpenEmbedded build system enables three levels of package
4277 architectures: "all", "tune" or "package", and "machine". Any given
4278 recipe usually selects one of these package architectures (types) for
4279 its output. Depending for what a given recipe creates packages,
4280 making sure you enable the appropriate package architecture can
4281 directly impact the build time.
4282
4283 A recipe that just generates scripts can enable "all" architecture
4284 because there are no binaries to build. To specifically enable "all"
4285 architecture, be sure your recipe inherits the
4286 :ref:`allarch <ref-classes-allarch>` class.
4287 This class is useful for "all" architectures because it configures
4288 many variables so packages can be used across multiple architectures.
4289
4290 If your recipe needs to generate packages that are machine-specific
4291 or when one of the build or runtime dependencies is already
4292 machine-architecture dependent, which makes your recipe also
4293 machine-architecture dependent, make sure your recipe enables the
4294 "machine" package architecture through the
4295 :term:`MACHINE_ARCH`
4296 variable::
4297
4298 PACKAGE_ARCH = "${MACHINE_ARCH}"
4299
4300 When you do not
4301 specifically enable a package architecture through the
4302 :term:`PACKAGE_ARCH`, The
4303 OpenEmbedded build system defaults to the
4304 :term:`TUNE_PKGARCH` setting::
4305
4306 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4307
4308- *Choose a Generic Tuning File if Possible:* Some tunes are more
4309 generic and can run on multiple targets (e.g. an ``armv5`` set of
4310 packages could run on ``armv6`` and ``armv7`` processors in most
4311 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4312 higher processors. You should realize, however, that advances on
4313 newer processor versions would not be used.
4314
4315 If you select the same tune for several different machines, the
4316 OpenEmbedded build system reuses software previously built, thus
4317 speeding up the overall build time. Realize that even though a new
4318 sysroot for each machine is generated, the software is not recompiled
4319 and only one package feed exists.
4320
4321- *Manage Granular Level Packaging:* Sometimes there are cases where
4322 injecting another level of package architecture beyond the three
4323 higher levels noted earlier can be useful. For example, consider how
4324 NXP (formerly Freescale) allows for the easy reuse of binary packages
4325 in their layer
4326 :yocto_git:`meta-freescale </meta-freescale/>`.
4327 In this example, the
4328 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
4329 class shares GPU packages for i.MX53 boards because all boards share
4330 the AMD GPU. The i.MX6-based boards can do the same because all
4331 boards share the Vivante GPU. This class inspects the BitBake
4332 datastore to identify if the package provides or depends on one of
4333 the sub-architecture values. If so, the class sets the
4334 :term:`PACKAGE_ARCH` value
4335 based on the ``MACHINE_SUBARCH`` value. If the package does not
4336 provide or depend on one of the sub-architecture values but it
4337 matches a value in the machine-specific filter, it sets
4338 :term:`MACHINE_ARCH`. This
4339 behavior reduces the number of packages built and saves build time by
4340 reusing binaries.
4341
4342- *Use Tools to Debug Issues:* Sometimes you can run into situations
4343 where software is being rebuilt when you think it should not be. For
4344 example, the OpenEmbedded build system might not be using shared
4345 state between machines when you think it should be. These types of
4346 situations are usually due to references to machine-specific
4347 variables such as :term:`MACHINE`,
4348 :term:`SERIAL_CONSOLES`,
4349 :term:`XSERVER`,
4350 :term:`MACHINE_FEATURES`,
4351 and so forth in code that is supposed to only be tune-specific or
4352 when the recipe depends
4353 (:term:`DEPENDS`,
4354 :term:`RDEPENDS`,
4355 :term:`RRECOMMENDS`,
4356 :term:`RSUGGESTS`, and so forth)
4357 on some other recipe that already has
4358 :term:`PACKAGE_ARCH` defined
4359 as "${MACHINE_ARCH}".
4360
4361 .. note::
4362
4363 Patches to fix any issues identified are most welcome as these
4364 issues occasionally do occur.
4365
4366 For such cases, you can use some tools to help you sort out the
4367 situation:
4368
4369 - ``state-diff-machines.sh``*:* You can find this tool in the
4370 ``scripts`` directory of the Source Repositories. See the comments
4371 in the script for information on how to use the tool.
4372
4373 - *BitBake's "-S printdiff" Option:* Using this option causes
4374 BitBake to try to establish the closest signature match it can
4375 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4376 over the matches to determine the stamps and delta where these two
4377 stamp trees diverge.
4378
4379Building Software from an External Source
4380-----------------------------------------
4381
4382By default, the OpenEmbedded build system uses the :term:`Build Directory`
4383when building source code. The build process involves fetching the source
4384files, unpacking them, and then patching them if necessary before the build
4385takes place.
4386
4387There are situations where you might want to build software from source
4388files that are external to and thus outside of the OpenEmbedded build
4389system. For example, suppose you have a project that includes a new BSP
4390with a heavily customized kernel. And, you want to minimize exposing the
4391build system to the development team so that they can focus on their
4392project and maintain everyone's workflow as much as possible. In this
4393case, you want a kernel source directory on the development machine
4394where the development occurs. You want the recipe's
4395:term:`SRC_URI` variable to point to
4396the external directory and use it as is, not copy it.
4397
4398To build from software that comes from an external source, all you need to do
4399is inherit the :ref:`externalsrc <ref-classes-externalsrc>` class and then set
4400the :term:`EXTERNALSRC` variable to point to your external source code. Here
4401are the statements to put in your ``local.conf`` file::
4402
4403 INHERIT += "externalsrc"
4404 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
4405
4406This next example shows how to accomplish the same thing by setting
4407:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
4408
4409 EXTERNALSRC = "path"
4410 EXTERNALSRC_BUILD = "path"
4411
4412.. note::
4413
4414 In order for these settings to take effect, you must globally or
4415 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
4416 class.
4417
4418By default, :ref:`ref-classes-externalsrc` builds the source code in a
4419directory separate from the external source directory as specified by
4420:term:`EXTERNALSRC`. If you need
4421to have the source built in the same directory in which it resides, or
4422some other nominated directory, you can set
4423:term:`EXTERNALSRC_BUILD`
4424to point to that directory::
4425
4426 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
4427
4428Replicating a Build Offline
4429---------------------------
4430
4431It can be useful to take a "snapshot" of upstream sources used in a
4432build and then use that "snapshot" later to replicate the build offline.
4433To do so, you need to first prepare and populate your downloads
4434directory your "snapshot" of files. Once your downloads directory is
4435ready, you can use it at any time and from any machine to replicate your
4436build.
4437
4438Follow these steps to populate your Downloads directory:
4439
44401. *Create a Clean Downloads Directory:* Start with an empty downloads
4441 directory (:term:`DL_DIR`). You
4442 start with an empty downloads directory by either removing the files
4443 in the existing directory or by setting :term:`DL_DIR` to point to either
4444 an empty location or one that does not yet exist.
4445
44462. *Generate Tarballs of the Source Git Repositories:* Edit your
4447 ``local.conf`` configuration file as follows::
4448
4449 DL_DIR = "/home/your-download-dir/"
4450 BB_GENERATE_MIRROR_TARBALLS = "1"
4451
4452 During
4453 the fetch process in the next step, BitBake gathers the source files
4454 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
4455 the
4456 :term:`BB_GENERATE_MIRROR_TARBALLS`
4457 variable for more information.
4458
44593. *Populate Your Downloads Directory Without Building:* Use BitBake to
4460 fetch your sources but inhibit the build::
4461
4462 $ bitbake target --runonly=fetch
4463
4464 The downloads directory (i.e. ``${DL_DIR}``) now has
4465 a "snapshot" of the source files in the form of tarballs, which can
4466 be used for the build.
4467
44684. *Optionally Remove Any Git or other SCM Subdirectories From the
4469 Downloads Directory:* If you want, you can clean up your downloads
4470 directory by removing any Git or other Source Control Management
4471 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4472 already contain these subdirectories.
4473
4474Once your downloads directory has everything it needs regarding source
4475files, you can create your "own-mirror" and build your target.
4476Understand that you can use the files to build the target offline from
4477any machine and at any time.
4478
4479Follow these steps to build your target using the files in the downloads
4480directory:
4481
44821. *Using Local Files Only:* Inside your ``local.conf`` file, add the
4483 :term:`SOURCE_MIRROR_URL` variable, inherit the
4484 :ref:`own-mirrors <ref-classes-own-mirrors>` class, and use the
4485 :term:`BB_NO_NETWORK` variable to your ``local.conf``::
4486
4487 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4488 INHERIT += "own-mirrors"
4489 BB_NO_NETWORK = "1"
4490
4491 The :term:`SOURCE_MIRROR_URL` and :ref:`own-mirrors <ref-classes-own-mirrors>`
4492 class set up the system to use the downloads directory as your "own
4493 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
4494 BitBake's fetching process in step 3 stays local, which means files
4495 from your "own-mirror" are used.
4496
44972. *Start With a Clean Build:* You can start with a clean build by
4498 removing the ``${``\ :term:`TMPDIR`\ ``}`` directory or using a new
4499 :term:`Build Directory`.
4500
45013. *Build Your Target:* Use BitBake to build your target::
4502
4503 $ bitbake target
4504
4505 The build completes using the known local "snapshot" of source
4506 files from your mirror. The resulting tarballs for your "snapshot" of
4507 source files are in the downloads directory.
4508
4509 .. note::
4510
4511 The offline build does not work if recipes attempt to find the
4512 latest version of software by setting
4513 :term:`SRCREV` to
4514 ``${``\ :term:`AUTOREV`\ ``}``::
4515
4516 SRCREV = "${AUTOREV}"
4517
4518 When a recipe sets :term:`SRCREV` to
4519 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
4520 attempt to determine the latest version of software from the SCM.
4521 Typically, recipes that use :term:`AUTOREV` are custom or modified
4522 recipes. Recipes that reside in public repositories usually do not
4523 use :term:`AUTOREV`.
4524
4525 If you do have recipes that use :term:`AUTOREV`, you can take steps to
4526 still use the recipes in an offline build. Do the following:
4527
4528 1. Use a configuration generated by enabling :ref:`build
4529 history <dev-manual/common-tasks:maintaining build output quality>`.
4530
4531 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
4532 stored :term:`SRCREV` values from the build's history. For more
4533 information on collecting these values, see the
4534 ":ref:`dev-manual/common-tasks:build history package information`"
4535 section.
4536
4537 3. Once you have the correct source revisions, you can modify
4538 those recipes to set :term:`SRCREV` to specific versions of the
4539 software.
4540
4541Speeding Up a Build
4542===================
4543
4544Build time can be an issue. By default, the build system uses simple
4545controls to try and maximize build efficiency. In general, the default
4546settings for all the following variables result in the most efficient
4547build times when dealing with single socket systems (i.e. a single CPU).
4548If you have multiple CPUs, you might try increasing the default values
4549to gain more speed. See the descriptions in the glossary for each
4550variable for more information:
4551
4552- :term:`BB_NUMBER_THREADS`:
4553 The maximum number of threads BitBake simultaneously executes.
4554
4555- :term:`BB_NUMBER_PARSE_THREADS`:
4556 The number of threads BitBake uses during parsing.
4557
4558- :term:`PARALLEL_MAKE`: Extra
4559 options passed to the ``make`` command during the
4560 :ref:`ref-tasks-compile` task in
4561 order to specify parallel compilation on the local build host.
4562
4563- :term:`PARALLEL_MAKEINST`:
4564 Extra options passed to the ``make`` command during the
4565 :ref:`ref-tasks-install` task in
4566 order to specify parallel installation on the local build host.
4567
4568As mentioned, these variables all scale to the number of processor cores
4569available on the build system. For single socket systems, this
4570auto-scaling ensures that the build system fundamentally takes advantage
4571of potential parallel operations during the build based on the build
4572machine's capabilities.
4573
4574Following are additional factors that can affect build speed:
4575
4576- File system type: The file system type that the build is being
4577 performed on can also influence performance. Using ``ext4`` is
4578 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4579 improved features such as extents.
4580
4581- Disabling the updating of access time using ``noatime``: The
4582 ``noatime`` mount option prevents the build system from updating file
4583 and directory access times.
4584
4585- Setting a longer commit: Using the "commit=" mount option increases
4586 the interval in seconds between disk cache writes. Changing this
4587 interval from the five second default to something longer increases
4588 the risk of data loss but decreases the need to write to the disk,
4589 thus increasing the build performance.
4590
4591- Choosing the packaging backend: Of the available packaging backends,
4592 IPK is the fastest. Additionally, selecting a singular packaging
4593 backend also helps.
4594
4595- Using ``tmpfs`` for :term:`TMPDIR`
4596 as a temporary file system: While this can help speed up the build,
4597 the benefits are limited due to the compiler using ``-pipe``. The
4598 build system goes to some lengths to avoid ``sync()`` calls into the
4599 file system on the principle that if there was a significant failure,
4600 the :term:`Build Directory` contents could easily be rebuilt.
4601
4602- Inheriting the
4603 :ref:`rm_work <ref-classes-rm-work>` class:
4604 Inheriting this class has shown to speed up builds due to
4605 significantly lower amounts of data stored in the data cache as well
4606 as on disk. Inheriting this class also makes cleanup of
4607 :term:`TMPDIR` faster, at the
4608 expense of being easily able to dive into the source code. File
4609 system maintainers have recommended that the fastest way to clean up
4610 large numbers of files is to reformat partitions rather than delete
4611 files due to the linear nature of partitions. This, of course,
4612 assumes you structure the disk partitions and file systems in a way
4613 that this is practical.
4614
4615Aside from the previous list, you should keep some trade offs in mind
4616that can help you speed up the build:
4617
4618- Remove items from
4619 :term:`DISTRO_FEATURES`
4620 that you might not need.
4621
4622- Exclude debug symbols and other debug information: If you do not need
4623 these symbols and other debug information, disabling the ``*-dbg``
4624 package generation can speed up the build. You can disable this
4625 generation by setting the
4626 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4627 variable to "1".
4628
4629- Disable static library generation for recipes derived from
4630 ``autoconf`` or ``libtool``: Following is an example showing how to
4631 disable static libraries and still provide an override to handle
4632 exceptions::
4633
4634 STATICLIBCONF = "--disable-static"
4635 STATICLIBCONF:sqlite3-native = ""
4636 EXTRA_OECONF += "${STATICLIBCONF}"
4637
4638 .. note::
4639
4640 - Some recipes need static libraries in order to work correctly
4641 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4642 as in the previous example, account for these kinds of
4643 exceptions.
4644
4645 - Some packages have packaging code that assumes the presence of
4646 the static libraries. If so, you might need to exclude them as
4647 well.
4648
4649Working With Libraries
4650======================
4651
4652Libraries are an integral part of your system. This section describes
4653some common practices you might find helpful when working with libraries
4654to build your system:
4655
4656- :ref:`How to include static library files
4657 <dev-manual/common-tasks:including static library files>`
4658
4659- :ref:`How to use the Multilib feature to combine multiple versions of
4660 library files into a single image
4661 <dev-manual/common-tasks:combining multiple versions of library files into one image>`
4662
4663- :ref:`How to install multiple versions of the same library in parallel on
4664 the same system
4665 <dev-manual/common-tasks:installing multiple versions of the same library>`
4666
4667Including Static Library Files
4668------------------------------
4669
4670If you are building a library and the library offers static linking, you
4671can control which static library files (``*.a`` files) get included in
4672the built library.
4673
4674The :term:`PACKAGES` and
4675:term:`FILES:* <FILES>` variables in the
4676``meta/conf/bitbake.conf`` configuration file define how files installed
4677by the :ref:`ref-tasks-install` task are packaged. By default, the :term:`PACKAGES`
4678variable includes ``${PN}-staticdev``, which represents all static
4679library files.
4680
4681.. note::
4682
4683 Some previously released versions of the Yocto Project defined the
4684 static library files through ``${PN}-dev``.
4685
4686Following is part of the BitBake configuration file, where you can see
4687how the static library files are defined::
4688
4689 PACKAGE_BEFORE_PN ?= ""
4690 PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
4691 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4692 FILES = ""
4693
4694 FILES:${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
4695 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4696 ${base_bindir}/* ${base_sbindir}/* \
4697 ${base_libdir}/*${SOLIBS} \
4698 ${base_prefix}/lib/udev ${prefix}/lib/udev \
4699 ${base_libdir}/udev ${libdir}/udev \
4700 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4701 ${datadir}/pixmaps ${datadir}/applications \
4702 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4703 ${libdir}/bonobo/servers"
4704
4705 FILES:${PN}-bin = "${bindir}/* ${sbindir}/*"
4706
4707 FILES:${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
4708 ${datadir}/gnome/help"
4709 SECTION:${PN}-doc = "doc"
4710
4711 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
4712 FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
4713 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4714 ${datadir}/aclocal ${base_libdir}/*.o \
4715 ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
4716 ${libdir}/cmake ${datadir}/cmake"
4717 SECTION:${PN}-dev = "devel"
4718 ALLOW_EMPTY:${PN}-dev = "1"
4719 RDEPENDS:${PN}-dev = "${PN} (= ${EXTENDPKGV})"
4720
4721 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4722 SECTION:${PN}-staticdev = "devel"
4723 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
4724
4725Combining Multiple Versions of Library Files into One Image
4726-----------------------------------------------------------
4727
4728The build system offers the ability to build libraries with different
4729target optimizations or architecture formats and combine these together
4730into one system image. You can link different binaries in the image
4731against the different libraries as needed for specific use cases. This
4732feature is called "Multilib".
4733
4734An example would be where you have most of a system compiled in 32-bit
4735mode using 32-bit libraries, but you have something large, like a
4736database engine, that needs to be a 64-bit application and uses 64-bit
4737libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4738libraries.
4739
4740While the Multilib feature is most commonly used for 32 and 64-bit
4741differences, the approach the build system uses facilitates different
4742target optimizations. You could compile some binaries to use one set of
4743libraries and other binaries to use a different set of libraries. The
4744libraries could differ in architecture, compiler options, or other
4745optimizations.
4746
4747There are several examples in the ``meta-skeleton`` layer found in the
4748:term:`Source Directory`:
4749
4750- :oe_git:`conf/multilib-example.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example.conf>`
4751 configuration file.
4752
4753- :oe_git:`conf/multilib-example2.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example2.conf>`
4754 configuration file.
4755
4756- :oe_git:`recipes-multilib/images/core-image-multilib-example.bb </openembedded-core/tree/meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb>`
4757 recipe
4758
4759Preparing to Use Multilib
4760~~~~~~~~~~~~~~~~~~~~~~~~~
4761
4762User-specific requirements drive the Multilib feature. Consequently,
4763there is no one "out-of-the-box" configuration that would
4764meet your needs.
4765
4766In order to enable Multilib, you first need to ensure your recipe is
4767extended to support multiple libraries. Many standard recipes are
4768already extended and support multiple libraries. You can check in the
4769``meta/conf/multilib.conf`` configuration file in the
4770:term:`Source Directory` to see how this is
4771done using the
4772:term:`BBCLASSEXTEND` variable.
4773Eventually, all recipes will be covered and this list will not be
4774needed.
4775
4776For the most part, the :ref:`Multilib <ref-classes-multilib*>`
4777class extension works automatically to
4778extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
4779:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
4780Standard variables such as
4781:term:`DEPENDS`,
4782:term:`RDEPENDS`,
4783:term:`RPROVIDES`,
4784:term:`RRECOMMENDS`,
4785:term:`PACKAGES`, and
4786:term:`PACKAGES_DYNAMIC` are
4787automatically extended by the system. If you are extending any manual
4788code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
4789those names are extended correctly.
4790
4791Using Multilib
4792~~~~~~~~~~~~~~
4793
4794After you have set up the recipes, you need to define the actual
4795combination of multiple libraries you want to build. You accomplish this
4796through your ``local.conf`` configuration file in the
4797:term:`Build Directory`. An example configuration would be as follows::
4798
4799 MACHINE = "qemux86-64"
4800 require conf/multilib.conf
4801 MULTILIBS = "multilib:lib32"
4802 DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
4803 IMAGE_INSTALL:append = " lib32-glib-2.0"
4804
4805This example enables an additional library named
4806``lib32`` alongside the normal target packages. When combining these
4807"lib32" alternatives, the example uses "x86" for tuning. For information
4808on this particular tuning, see
4809``meta/conf/machine/include/ia32/arch-ia32.inc``.
4810
4811The example then includes ``lib32-glib-2.0`` in all the images, which
4812illustrates one method of including a multiple library dependency. You
4813can use a normal image build to include this dependency, for example::
4814
4815 $ bitbake core-image-sato
4816
4817You can also build Multilib packages
4818specifically with a command like this::
4819
4820 $ bitbake lib32-glib-2.0
4821
4822Additional Implementation Details
4823~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4824
4825There are generic implementation details as well as details that are specific to
4826package management systems. Following are implementation details
4827that exist regardless of the package management system:
4828
4829- The typical convention used for the class extension code as used by
4830 Multilib assumes that all package names specified in
4831 :term:`PACKAGES` that contain
4832 ``${PN}`` have ``${PN}`` at the start of the name. When that
4833 convention is not followed and ``${PN}`` appears at the middle or the
4834 end of a name, problems occur.
4835
4836- The :term:`TARGET_VENDOR`
4837 value under Multilib will be extended to "-vendormlmultilib" (e.g.
4838 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
4839 slightly unwieldy contraction is that any "-" characters in the
4840 vendor string presently break Autoconf's ``config.sub``, and other
4841 separators are problematic for different reasons.
4842
4843Here are the implementation details for the RPM Package Management System:
4844
4845- A unique architecture is defined for the Multilib packages, along
4846 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
4847 :term:`Build Directory`. For example, consider ``lib32`` in a
4848 ``qemux86-64`` image. The possible architectures in the system are "all",
4849 "qemux86_64", "lib32:qemux86_64", and "lib32:x86".
4850
4851- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
4852 packaging. The naming for a normal RPM package and a Multilib RPM
4853 package in a ``qemux86-64`` system resolves to something similar to
4854 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
4855 respectively.
4856
4857- When installing a Multilib image, the RPM backend first installs the
4858 base image and then installs the Multilib libraries.
4859
4860- The build system relies on RPM to resolve the identical files in the
4861 two (or more) Multilib packages.
4862
4863Here are the implementation details for the IPK Package Management System:
4864
4865- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
4866 packaging. The naming for a normal RPM package and a Multilib IPK
4867 package in a ``qemux86-64`` system resolves to something like
4868 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``,
4869 respectively.
4870
4871- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
4872 packages with and without the Multilib feature can exist in the same
4873 folder due to the ``${PN}`` differences.
4874
4875- IPK defines a sanity check for Multilib installation using certain
4876 rules for file comparison, overridden, etc.
4877
4878Installing Multiple Versions of the Same Library
4879------------------------------------------------
4880
4881There are be situations where you need to install and use multiple versions
4882of the same library on the same system at the same time. This
4883almost always happens when a library API changes and you have
4884multiple pieces of software that depend on the separate versions of the
4885library. To accommodate these situations, you can install multiple
4886versions of the same library in parallel on the same system.
4887
4888The process is straightforward as long as the libraries use proper
4889versioning. With properly versioned libraries, all you need to do to
4890individually specify the libraries is create separate, appropriately
4891named recipes where the :term:`PN` part of
4892the name includes a portion that differentiates each library version
4893(e.g. the major part of the version number). Thus, instead of having a
4894single recipe that loads one version of a library (e.g. ``clutter``),
4895you provide multiple recipes that result in different versions of the
4896libraries you want. As an example, the following two recipes would allow
4897the two separate versions of the ``clutter`` library to co-exist on the
4898same system:
4899
4900.. code-block:: none
4901
4902 clutter-1.6_1.6.20.bb
4903 clutter-1.8_1.8.4.bb
4904
4905Additionally, if
4906you have other recipes that depend on a given library, you need to use
4907the :term:`DEPENDS` variable to
4908create the dependency. Continuing with the same example, if you want to
4909have a recipe depend on the 1.8 version of the ``clutter`` library, use
4910the following in your recipe::
4911
4912 DEPENDS = "clutter-1.8"
4913
4914Working with Pre-Built Libraries
4915================================
4916
4917Introduction
4918-------------
4919
4920Some library vendors do not release source code for their software but do
4921release pre-built binaries. When shared libraries are built, they should
4922be versioned (see `this article
4923<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__
4924for some background), but sometimes this is not done.
4925
4926To summarize, a versioned library must meet two conditions:
4927
4928#. The filename must have the version appended, for example: ``libfoo.so.1.2.3``.
4929#. The library must have the ELF tag ``SONAME`` set to the major version
4930 of the library, for example: ``libfoo.so.1``. You can check this by
4931 running ``readelf -d filename | grep SONAME``.
4932
4933This section shows how to deal with both versioned and unversioned
4934pre-built libraries.
4935
4936Versioned Libraries
4937-------------------
4938
4939In this example we work with pre-built libraries for the FT4222H USB I/O chip.
4940Libraries are built for several target architecture variants and packaged in
4941an archive as follows::
4942
4943 ├── build-arm-hisiv300
4944 │   └── libft4222.so.1.4.4.44
4945 ├── build-arm-v5-sf
4946 │   └── libft4222.so.1.4.4.44
4947 ├── build-arm-v6-hf
4948 │   └── libft4222.so.1.4.4.44
4949 ├── build-arm-v7-hf
4950 │   └── libft4222.so.1.4.4.44
4951 ├── build-arm-v8
4952 │   └── libft4222.so.1.4.4.44
4953 ├── build-i386
4954 │   └── libft4222.so.1.4.4.44
4955 ├── build-i486
4956 │   └── libft4222.so.1.4.4.44
4957 ├── build-mips-eglibc-hf
4958 │   └── libft4222.so.1.4.4.44
4959 ├── build-pentium
4960 │   └── libft4222.so.1.4.4.44
4961 ├── build-x86_64
4962 │   └── libft4222.so.1.4.4.44
4963 ├── examples
4964 │   ├── get-version.c
4965 │   ├── i2cm.c
4966 │   ├── spim.c
4967 │   └── spis.c
4968 ├── ftd2xx.h
4969 ├── install4222.sh
4970 ├── libft4222.h
4971 ├── ReadMe.txt
4972 └── WinTypes.h
4973
4974To write a recipe to use such a library in your system:
4975
4976- The vendor will probably have a proprietary licence, so set
4977 :term:`LICENSE_FLAGS` in your recipe.
4978- The vendor provides a tarball containing libraries so set :term:`SRC_URI`
4979 appropriately.
4980- Set :term:`COMPATIBLE_HOST` so that the recipe cannot be used with an
4981 unsupported architecture. In the following example, we only support the 32
4982 and 64 bit variants of the ``x86`` architecture.
4983- As the vendor provides versioned libraries, we can use ``oe_soinstall``
4984 from :ref:`ref-classes-utils` to install the shared library and create
4985 symbolic links. If the vendor does not do this, we need to follow the
4986 non-versioned library guidelines in the next section.
4987- As the vendor likely used :term:`LDFLAGS` different from those in your Yocto
4988 Project build, disable the corresponding checks by adding ``ldflags``
4989 to :term:`INSANE_SKIP`.
4990- The vendor will typically ship release builds without debugging symbols.
4991 Avoid errors by preventing the packaging task from stripping out the symbols
4992 and adding them to a separate debug package. This is done by setting the
4993 ``INHIBIT_`` flags shown below.
4994
4995The complete recipe would look like this::
4996
4997 SUMMARY = "FTDI FT4222H Library"
4998 SECTION = "libs"
4999 LICENSE_FLAGS = "ftdi"
5000 LICENSE = "CLOSED"
5001
5002 COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
5003
5004 # Sources available in a .tgz file in .zip archive
5005 # at https://ftdichip.com/wp-content/uploads/2021/01/libft4222-linux-1.4.4.44.zip
5006 # Found on https://ftdichip.com/software-examples/ft4222h-software-examples/
5007 # Since dealing with this particular type of archive is out of topic here,
5008 # we use a local link.
5009 SRC_URI = "file://libft4222-linux-${PV}.tgz"
5010
5011 S = "${WORKDIR}"
5012
5013 ARCH_DIR:x86-64 = "build-x86_64"
5014 ARCH_DIR:i586 = "build-i386"
5015 ARCH_DIR:i686 = "build-i386"
5016
5017 INSANE_SKIP:${PN} = "ldflags"
5018 INHIBIT_PACKAGE_STRIP = "1"
5019 INHIBIT_SYSROOT_STRIP = "1"
5020 INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
5021
5022 do_install () {
5023 install -m 0755 -d ${D}${libdir}
5024 oe_soinstall ${S}/${ARCH_DIR}/libft4222.so.${PV} ${D}${libdir}
5025 install -d ${D}${includedir}
5026 install -m 0755 ${S}/*.h ${D}${includedir}
5027 }
5028
5029If the precompiled binaries are not statically linked and have dependencies on
5030other libraries, then by adding those libraries to :term:`DEPENDS`, the linking
5031can be examined and the appropriate :term:`RDEPENDS` automatically added.
5032
5033Non-Versioned Libraries
5034-----------------------
5035
5036Some Background
5037~~~~~~~~~~~~~~~
5038
5039Libraries in Linux systems are generally versioned so that it is possible
5040to have multiple versions of the same library installed, which eases upgrades
5041and support for older software. For example, suppose that in a versioned
5042library, an actual library is called ``libfoo.so.1.2``, a symbolic link named
5043``libfoo.so.1`` points to ``libfoo.so.1.2``, and a symbolic link named
5044``libfoo.so`` points to ``libfoo.so.1.2``. Given these conditions, when you
5045link a binary against a library, you typically provide the unversioned file
5046name (i.e. ``-lfoo`` to the linker). However, the linker follows the symbolic
5047link and actually links against the versioned filename. The unversioned symbolic
5048link is only used at development time. Consequently, the library is packaged
5049along with the headers in the development package ``${PN}-dev`` along with the
5050actual library and versioned symbolic links in ``${PN}``. Because versioned
5051libraries are far more common than unversioned libraries, the default packaging
5052rules assume versioned libraries.
5053
5054Yocto Library Packaging Overview
5055~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5056
5057It follows that packaging an unversioned library requires a bit of work in the
5058recipe. By default, ``libfoo.so`` gets packaged into ``${PN}-dev``, which
5059triggers a QA warning that a non-symlink library is in a ``-dev`` package,
5060and binaries in the same recipe link to the library in ``${PN}-dev``,
5061which triggers more QA warnings. To solve this problem, you need to package the
5062unversioned library into ``${PN}`` where it belongs. The following are the abridged
5063default :term:`FILES` variables in ``bitbake.conf``::
5064
5065 SOLIBS = ".so.*"
5066 SOLIBSDEV = ".so"
5067 FILES_${PN} = "... ${libdir}/lib*${SOLIBS} ..."
5068 FILES_SOLIBSDEV ?= "... ${libdir}/lib*${SOLIBSDEV} ..."
5069 FILES_${PN}-dev = "... ${FILES_SOLIBSDEV} ..."
5070
5071:term:`SOLIBS` defines a pattern that matches real shared object libraries.
5072:term:`SOLIBSDEV` matches the development form (unversioned symlink). These two
5073variables are then used in ``FILES:${PN}`` and ``FILES:${PN}-dev``, which puts
5074the real libraries into ``${PN}`` and the unversioned symbolic link into ``${PN}-dev``.
5075To package unversioned libraries, you need to modify the variables in the recipe
5076as follows::
5077
5078 SOLIBS = ".so"
5079 FILES_SOLIBSDEV = ""
5080
5081The modifications cause the ``.so`` file to be the real library
5082and unset :term:`FILES_SOLIBSDEV` so that no libraries get packaged into
5083``${PN}-dev``. The changes are required because unless :term:`PACKAGES` is changed,
5084``${PN}-dev`` collects files before `${PN}`. ``${PN}-dev`` must not collect any of
5085the files you want in ``${PN}``.
5086
5087Finally, loadable modules, essentially unversioned libraries that are linked
5088at runtime using ``dlopen()`` instead of at build time, should generally be
5089installed in a private directory. However, if they are installed in ``${libdir}``,
5090then the modules can be treated as unversioned libraries.
5091
5092Example
5093~~~~~~~
5094
5095The example below installs an unversioned x86-64 pre-built library named
5096``libfoo.so``. The :term:`COMPATIBLE_HOST` variable limits recipes to the
5097x86-64 architecture while the :term:`INSANE_SKIP`, :term:`INHIBIT_PACKAGE_STRIP`
5098and :term:`INHIBIT_SYSROOT_STRIP` variables are all set as in the above
5099versioned library example. The "magic" is setting the :term:`SOLIBS` and
5100:term:`FILES_SOLIBSDEV` variables as explained above::
5101
5102 SUMMARY = "libfoo sample recipe"
5103 SECTION = "libs"
5104 LICENSE = "CLOSED"
5105
5106 SRC_URI = "file://libfoo.so"
5107
5108 COMPATIBLE_HOST = "x86_64.*-linux"
5109
5110 INSANE_SKIP:${PN} = "ldflags"
5111 INHIBIT_PACKAGE_STRIP = "1"
5112 INHIBIT_SYSROOT_STRIP = "1"
5113 SOLIBS = ".so"
5114 FILES_SOLIBSDEV = ""
5115
5116 do_install () {
5117 install -d ${D}${libdir}
5118 install -m 0755 ${WORKDIR}/libfoo.so ${D}${libdir}
5119 }
5120
5121Using x32 psABI
5122===============
5123
5124x32 processor-specific Application Binary Interface (`x32
5125psABI <https://software.intel.com/en-us/node/628948>`__) is a native
512632-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
5127ABI defines the calling conventions between functions in a processing
5128environment. The interface determines what registers are used and what
5129the sizes are for various C data types.
5130
5131Some processing environments prefer using 32-bit applications even when
5132running on Intel 64-bit platforms. Consider the i386 psABI, which is a
5133very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
5134provide efficient use and access of the Intel 64-bit processor
5135resources, leaving the system underutilized. Now consider the x86_64
5136psABI. This ABI is newer and uses 64-bits for data sizes and program
5137pointers. The extra bits increase the footprint size of the programs,
5138libraries, and also increases the memory and file system size
5139requirements. Executing under the x32 psABI enables user programs to
5140utilize CPU and system resources more efficiently while keeping the
5141memory footprint of the applications low. Extra bits are used for
5142registers but not for addressing mechanisms.
5143
5144The Yocto Project supports the final specifications of x32 psABI as
5145follows:
5146
5147- You can create packages and images in x32 psABI format on x86_64
5148 architecture targets.
5149
5150- You can successfully build recipes with the x32 toolchain.
5151
5152- You can create and boot ``core-image-minimal`` and
5153 ``core-image-sato`` images.
5154
5155- There is RPM Package Manager (RPM) support for x32 binaries.
5156
5157- There is support for large images.
5158
5159To use the x32 psABI, you need to edit your ``conf/local.conf``
5160configuration file as follows::
5161
5162 MACHINE = "qemux86-64"
5163 DEFAULTTUNE = "x86-64-x32"
5164 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \
5165 or 'INVALID')) or 'lib'}"
5166
5167Once you have set
5168up your configuration file, use BitBake to build an image that supports
5169the x32 psABI. Here is an example::
5170
5171 $ bitbake core-image-sato
5172
5173Enabling GObject Introspection Support
5174======================================
5175
5176`GObject introspection <https://gi.readthedocs.io/en/latest/>`__
5177is the standard mechanism for accessing GObject-based software from
5178runtime environments. GObject is a feature of the GLib library that
5179provides an object framework for the GNOME desktop and related software.
5180GObject Introspection adds information to GObject that allows objects
5181created within it to be represented across different programming
5182languages. If you want to construct GStreamer pipelines using Python, or
5183control UPnP infrastructure using Javascript and GUPnP, GObject
5184introspection is the only way to do it.
5185
5186This section describes the Yocto Project support for generating and
5187packaging GObject introspection data. GObject introspection data is a
5188description of the API provided by libraries built on top of the GLib
5189framework, and, in particular, that framework's GObject mechanism.
5190GObject Introspection Repository (GIR) files go to ``-dev`` packages,
5191``typelib`` files go to main packages as they are packaged together with
5192libraries that are introspected.
5193
5194The data is generated when building such a library, by linking the
5195library with a small executable binary that asks the library to describe
5196itself, and then executing the binary and processing its output.
5197
5198Generating this data in a cross-compilation environment is difficult
5199because the library is produced for the target architecture, but its
5200code needs to be executed on the build host. This problem is solved with
5201the OpenEmbedded build system by running the code through QEMU, which
5202allows precisely that. Unfortunately, QEMU does not always work
5203perfectly as mentioned in the ":ref:`dev-manual/common-tasks:known issues`"
5204section.
5205
5206Enabling the Generation of Introspection Data
5207---------------------------------------------
5208
5209Enabling the generation of introspection data (GIR files) in your
5210library package involves the following:
5211
52121. Inherit the
5213 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
5214 class.
5215
52162. Make sure introspection is not disabled anywhere in the recipe or
5217 from anything the recipe includes. Also, make sure that
5218 "gobject-introspection-data" is not in
5219 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5220 and that "qemu-usermode" is not in
5221 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5222 In either of these conditions, nothing will happen.
5223
52243. Try to build the recipe. If you encounter build errors that look like
5225 something is unable to find ``.so`` libraries, check where these
5226 libraries are located in the source tree and add the following to the
5227 recipe::
5228
5229 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5230
5231 .. note::
5232
5233 See recipes in the ``oe-core`` repository that use that
5234 :term:`GIR_EXTRA_LIBS_PATH` variable as an example.
5235
52364. Look for any other errors, which probably mean that introspection
5237 support in a package is not entirely standard, and thus breaks down
5238 in a cross-compilation environment. For such cases, custom-made fixes
5239 are needed. A good place to ask and receive help in these cases is
5240 the :ref:`Yocto Project mailing
5241 lists <resources-mailinglist>`.
5242
5243.. note::
5244
5245 Using a library that no longer builds against the latest Yocto
5246 Project release and prints introspection related errors is a good
5247 candidate for the previous procedure.
5248
5249Disabling the Generation of Introspection Data
5250----------------------------------------------
5251
5252You might find that you do not want to generate introspection data. Or,
5253perhaps QEMU does not work on your build host and target architecture
5254combination. If so, you can use either of the following methods to
5255disable GIR file generations:
5256
5257- Add the following to your distro configuration::
5258
5259 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5260
5261 Adding this statement disables generating introspection data using
5262 QEMU but will still enable building introspection tools and libraries
5263 (i.e. building them does not require the use of QEMU).
5264
5265- Add the following to your machine configuration::
5266
5267 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5268
5269 Adding this statement disables the use of QEMU when building packages for your
5270 machine. Currently, this feature is used only by introspection
5271 recipes and has the same effect as the previously described option.
5272
5273 .. note::
5274
5275 Future releases of the Yocto Project might have other features
5276 affected by this option.
5277
5278If you disable introspection data, you can still obtain it through other
5279means such as copying the data from a suitable sysroot, or by generating
5280it on the target hardware. The OpenEmbedded build system does not
5281currently provide specific support for these techniques.
5282
5283Testing that Introspection Works in an Image
5284--------------------------------------------
5285
5286Use the following procedure to test if generating introspection data is
5287working in an image:
5288
52891. Make sure that "gobject-introspection-data" is not in
5290 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5291 and that "qemu-usermode" is not in
5292 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5293
52942. Build ``core-image-sato``.
5295
52963. Launch a Terminal and then start Python in the terminal.
5297
52984. Enter the following in the terminal::
5299
5300 >>> from gi.repository import GLib
5301 >>> GLib.get_host_name()
5302
53035. For something a little more advanced, enter the following see:
5304 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
5305
5306Known Issues
5307------------
5308
5309Here are know issues in GObject Introspection Support:
5310
5311- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5312 introspection data on that architecture.
5313
5314- x32 is not supported by QEMU. Consequently, introspection data is
5315 disabled.
5316
5317- musl causes transient GLib binaries to crash on assertion failures.
5318 Consequently, generating introspection data is disabled.
5319
5320- Because QEMU is not able to run the binaries correctly, introspection
5321 is disabled for some specific packages under specific architectures
5322 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5323
5324- QEMU usermode might not work properly when running 64-bit binaries
5325 under 32-bit host machines. In particular, "qemumips64" is known to
5326 not work under i686.
5327
5328Optionally Using an External Toolchain
5329======================================
5330
5331You might want to use an external toolchain as part of your development.
5332If this is the case, the fundamental steps you need to accomplish are as
5333follows:
5334
5335- Understand where the installed toolchain resides. For cases where you
5336 need to build the external toolchain, you would need to take separate
5337 steps to build and install the toolchain.
5338
5339- Make sure you add the layer that contains the toolchain to your
5340 ``bblayers.conf`` file through the
5341 :term:`BBLAYERS` variable.
5342
5343- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5344 to the location in which you installed the toolchain.
5345
5346A good example of an external toolchain used with the Yocto Project is
5347Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5348to use that particular layer in the ``README`` file at
5349https://github.com/MentorEmbedded/meta-sourcery/. You can find
5350further information by reading about the
5351:term:`TCMODE` variable in the Yocto
5352Project Reference Manual's variable glossary.
5353
5354Creating Partitioned Images Using Wic
5355=====================================
5356
5357Creating an image for a particular hardware target using the
5358OpenEmbedded build system does not necessarily mean you can boot that
5359image as is on your device. Physical devices accept and boot images in
5360various ways depending on the specifics of the device. Usually,
5361information about the hardware can tell you what image format the device
5362requires. Should your device require multiple partitions on an SD card,
5363flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5364create the properly partitioned image.
5365
5366The ``wic`` command generates partitioned images from existing
5367OpenEmbedded build artifacts. Image generation is driven by partitioning
5368commands contained in an OpenEmbedded kickstart file (``.wks``)
5369specified either directly on the command line or as one of a selection
5370of canned kickstart files as shown with the ``wic list images`` command
5371in the
5372":ref:`dev-manual/common-tasks:generate an image using an existing kickstart file`"
5373section. When you apply the command to a given set of build artifacts, the
5374result is an image or set of images that can be directly written onto media and
5375used on a particular system.
5376
5377.. note::
5378
5379 For a kickstart file reference, see the
5380 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
5381 Chapter in the Yocto Project Reference Manual.
5382
5383The ``wic`` command and the infrastructure it is based on is by
5384definition incomplete. The purpose of the command is to allow the
5385generation of customized images, and as such, was designed to be
5386completely extensible through a plugin interface. See the
5387":ref:`dev-manual/common-tasks:using the wic plugin interface`" section
5388for information on these plugins.
5389
5390This section provides some background information on Wic, describes what
5391you need to have in place to run the tool, provides instruction on how
5392to use the Wic utility, provides information on using the Wic plugins
5393interface, and provides several examples that show how to use Wic.
5394
5395Background
5396----------
5397
5398This section provides some background on the Wic utility. While none of
5399this information is required to use Wic, you might find it interesting.
5400
5401- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5402 "oe" diphthong in "oeic" was promoted to the letter "w", because
5403 "oeic" is both difficult to remember and to pronounce.
5404
5405- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5406 The Wic implementation has been heavily modified to make direct use
5407 of OpenEmbedded build artifacts instead of package installation and
5408 configuration, which are already incorporated within the OpenEmbedded
5409 artifacts.
5410
5411- Wic is a completely independent standalone utility that initially
5412 provides easier-to-use and more flexible replacements for an existing
5413 functionality in OE-Core's
5414 :ref:`image-live <ref-classes-image-live>`
5415 class. The difference between Wic and those examples is that with Wic
5416 the functionality of those scripts is implemented by a
5417 general-purpose partitioning language, which is based on Redhat
5418 kickstart syntax.
5419
5420Requirements
5421------------
5422
5423In order to use the Wic utility with the OpenEmbedded Build system, your
5424system needs to meet the following requirements:
5425
5426- The Linux distribution on your development host must support the
5427 Yocto Project. See the ":ref:`detailed-supported-distros`"
5428 section in the Yocto Project Reference Manual for the list of
5429 distributions that support the Yocto Project.
5430
5431- The standard system utilities, such as ``cp``, must be installed on
5432 your development host system.
5433
5434- You must have sourced the build environment setup script (i.e.
5435 :ref:`structure-core-script`) found in the :term:`Build Directory`.
5436
5437- You need to have the build artifacts already available, which
5438 typically means that you must have already created an image using the
5439 OpenEmbedded build system (e.g. ``core-image-minimal``). While it
5440 might seem redundant to generate an image in order to create an image
5441 using Wic, the current version of Wic requires the artifacts in the
5442 form generated by the OpenEmbedded build system.
5443
5444- You must build several native tools, which are built to run on the
5445 build system::
5446
5447 $ bitbake parted-native dosfstools-native mtools-native
5448
5449- Include "wic" as part of the
5450 :term:`IMAGE_FSTYPES`
5451 variable.
5452
5453- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
5454 as part of the :term:`WKS_FILE` variable. If multiple candidate files can
5455 be provided by different layers, specify all the possible names through the
5456 :term:`WKS_FILES` variable instead.
5457
5458Getting Help
5459------------
5460
5461You can get general help for the ``wic`` command by entering the ``wic``
5462command by itself or by entering the command with a help argument as
5463follows::
5464
5465 $ wic -h
5466 $ wic --help
5467 $ wic help
5468
5469Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5470``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
5471commands except "help" by using the following form::
5472
5473 $ wic help command
5474
5475For example, the following command returns help for the ``write``
5476command::
5477
5478 $ wic help write
5479
5480Wic supports help for three topics: ``overview``, ``plugins``, and
5481``kickstart``. You can get help for any topic using the following form::
5482
5483 $ wic help topic
5484
5485For example, the following returns overview help for Wic::
5486
5487 $ wic help overview
5488
5489There is one additional level of help for Wic. You can get help on
5490individual images through the ``list`` command. You can use the ``list``
5491command to return the available Wic images as follows::
5492
5493 $ wic list images
5494 genericx86 Create an EFI disk image for genericx86*
5495 edgerouter Create SD card image for Edgerouter
5496 beaglebone-yocto Create SD card image for Beaglebone
5497 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5498 systemd-bootdisk Create an EFI disk image with systemd-boot
5499 mkhybridiso Create a hybrid ISO image
5500 mkefidisk Create an EFI disk image
5501 sdimage-bootpart Create SD card image with a boot partition
5502 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5503 directdisk Create a 'pcbios' direct disk image
5504 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5505 qemuriscv Create qcow2 image for RISC-V QEMU machines
5506 directdisk-gpt Create a 'pcbios' direct disk image
5507 efi-bootdisk
5508
5509Once you know the list of available
5510Wic images, you can use ``help`` with the command to get help on a
5511particular image. For example, the following command returns help on the
5512"beaglebone-yocto" image::
5513
5514 $ wic list beaglebone-yocto help
5515
5516 Creates a partitioned SD card image for Beaglebone.
5517 Boot files are located in the first vfat partition.
5518
5519Operational Modes
5520-----------------
5521
5522You can use Wic in two different modes, depending on how much control
5523you need for specifying the OpenEmbedded build artifacts that are used
5524for creating the image: Raw and Cooked:
5525
5526- *Raw Mode:* You explicitly specify build artifacts through Wic
5527 command-line arguments.
5528
5529- *Cooked Mode:* The current
5530 :term:`MACHINE` setting and image
5531 name are used to automatically locate and provide the build
5532 artifacts. You just supply a kickstart file and the name of the image
5533 from which to use artifacts.
5534
5535Regardless of the mode you use, you need to have the build artifacts
5536ready and available.
5537
5538Raw Mode
5539~~~~~~~~
5540
5541Running Wic in raw mode allows you to specify all the partitions through
5542the ``wic`` command line. The primary use for raw mode is if you have
5543built your kernel outside of the Yocto Project :term:`Build Directory`.
5544In other words, you can point to arbitrary kernel, root filesystem locations,
5545and so forth. Contrast this behavior with cooked mode where Wic looks in the
5546:term:`Build Directory` (e.g. ``tmp/deploy/images/``\ machine).
5547
5548The general form of the ``wic`` command in raw mode is::
5549
5550 $ wic create wks_file options ...
5551
5552 Where:
5553
5554 wks_file:
5555 An OpenEmbedded kickstart file. You can provide
5556 your own custom file or use a file from a set of
5557 existing files as described by further options.
5558
5559 optional arguments:
5560 -h, --help show this help message and exit
5561 -o OUTDIR, --outdir OUTDIR
5562 name of directory to create image in
5563 -e IMAGE_NAME, --image-name IMAGE_NAME
5564 name of the image to use the artifacts from e.g. core-
5565 image-sato
5566 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5567 path to the /rootfs dir to use as the .wks rootfs
5568 source
5569 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5570 path to the dir containing the boot artifacts (e.g.
5571 /EFI or /syslinux dirs) to use as the .wks bootimg
5572 source
5573 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5574 path to the dir containing the kernel to use in the
5575 .wks bootimg
5576 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5577 path to the native sysroot containing the tools to use
5578 to build the image
5579 -s, --skip-build-check
5580 skip the build check
5581 -f, --build-rootfs build rootfs
5582 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5583 compress image with specified compressor
5584 -m, --bmap generate .bmap
5585 --no-fstab-update Do not change fstab file.
5586 -v VARS_DIR, --vars VARS_DIR
5587 directory with <image>.env files that store bitbake
5588 variables
5589 -D, --debug output debug information
5590
5591.. note::
5592
5593 You do not need root privileges to run Wic. In fact, you should not
5594 run as root when using the utility.
5595
5596Cooked Mode
5597~~~~~~~~~~~
5598
5599Running Wic in cooked mode leverages off artifacts in the
5600:term:`Build Directory`. In other words, you do not have to specify kernel or
5601root filesystem locations as part of the command. All you need to provide is
5602a kickstart file and the name of the image from which to use artifacts
5603by using the "-e" option. Wic looks in the :term:`Build Directory` (e.g.
5604``tmp/deploy/images/``\ machine) for artifacts.
5605
5606The general form of the ``wic`` command using Cooked Mode is as follows::
5607
5608 $ wic create wks_file -e IMAGE_NAME
5609
5610 Where:
5611
5612 wks_file:
5613 An OpenEmbedded kickstart file. You can provide
5614 your own custom file or use a file from a set of
5615 existing files provided with the Yocto Project
5616 release.
5617
5618 required argument:
5619 -e IMAGE_NAME, --image-name IMAGE_NAME
5620 name of the image to use the artifacts from e.g. core-
5621 image-sato
5622
5623Using an Existing Kickstart File
5624--------------------------------
5625
5626If you do not want to create your own kickstart file, you can use an
5627existing file provided by the Wic installation. As shipped, kickstart
5628files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
5629following two locations::
5630
5631 poky/meta-yocto-bsp/wic
5632 poky/scripts/lib/wic/canned-wks
5633
5634Use the following command to list the available kickstart files::
5635
5636 $ wic list images
5637 genericx86 Create an EFI disk image for genericx86*
5638 beaglebone-yocto Create SD card image for Beaglebone
5639 edgerouter Create SD card image for Edgerouter
5640 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
5641 directdisk-gpt Create a 'pcbios' direct disk image
5642 mkefidisk Create an EFI disk image
5643 directdisk Create a 'pcbios' direct disk image
5644 systemd-bootdisk Create an EFI disk image with systemd-boot
5645 mkhybridiso Create a hybrid ISO image
5646 sdimage-bootpart Create SD card image with a boot partition
5647 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5648 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5649
5650When you use an existing file, you
5651do not have to use the ``.wks`` extension. Here is an example in Raw
5652Mode that uses the ``directdisk`` file::
5653
5654 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5655 -k kernel_dir -n native_sysroot
5656
5657Here are the actual partition language commands used in the
5658``genericx86.wks`` file to generate an image::
5659
5660 # short-description: Create an EFI disk image for genericx86*
5661 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5662 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5663 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5664 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5665
5666 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5667
5668Using the Wic Plugin Interface
5669------------------------------
5670
5671You can extend and specialize Wic functionality by using Wic plugins.
5672This section explains the Wic plugin interface.
5673
5674.. note::
5675
5676 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5677 are beyond the scope of this section.
5678
5679Source plugins provide a mechanism to customize partition content during
5680the Wic image generation process. You can use source plugins to map
5681values that you specify using ``--source`` commands in kickstart files
5682(i.e. ``*.wks``) to a plugin implementation used to populate a given
5683partition.
5684
5685.. note::
5686
5687 If you use plugins that have build-time dependencies (e.g. native
5688 tools, bootloaders, and so forth) when building a Wic image, you need
5689 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
5690 variable.
5691
5692Source plugins are subclasses defined in plugin files. As shipped, the
5693Yocto Project provides several plugin files. You can see the source
5694plugin files that ship with the Yocto Project
5695:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
5696Each of these plugin files contains source plugins that are designed to
5697populate a specific Wic image partition.
5698
5699Source plugins are subclasses of the ``SourcePlugin`` class, which is
5700defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5701the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5702file is a subclass of the ``SourcePlugin`` class, which is found in the
5703``pluginbase.py`` file.
5704
5705You can also implement source plugins in a layer outside of the Source
5706Repositories (external layer). To do so, be sure that your plugin files
5707are located in a directory whose path is
5708``scripts/lib/wic/plugins/source/`` within your external layer. When the
5709plugin files are located there, the source plugins they contain are made
5710available to Wic.
5711
5712When the Wic implementation needs to invoke a partition-specific
5713implementation, it looks for the plugin with the same name as the
5714``--source`` parameter used in the kickstart file given to that
5715partition. For example, if the partition is set up using the following
5716command in a kickstart file::
5717
5718 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5719
5720The methods defined as class
5721members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5722``bootimg-pcbios.py`` plugin file are used.
5723
5724To be more concrete, here is the corresponding plugin definition from
5725the ``bootimg-pcbios.py`` file for the previous command along with an
5726example method called by the Wic implementation when it needs to prepare
5727a partition using an implementation-specific function::
5728
5729 .
5730 .
5731 .
5732 class BootimgPcbiosPlugin(SourcePlugin):
5733 """
5734 Create MBR boot partition and install syslinux on it.
5735 """
5736
5737 name = 'bootimg-pcbios'
5738 .
5739 .
5740 .
5741 @classmethod
5742 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5743 oe_builddir, bootimg_dir, kernel_dir,
5744 rootfs_dir, native_sysroot):
5745 """
5746 Called to do the actual content population for a partition i.e. it
5747 'prepares' the partition to be incorporated into the image.
5748 In this case, prepare content for legacy bios boot partition.
5749 """
5750 .
5751 .
5752 .
5753
5754If a
5755subclass (plugin) itself does not implement a particular function, Wic
5756locates and uses the default version in the superclass. It is for this
5757reason that all source plugins are derived from the ``SourcePlugin``
5758class.
5759
5760The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5761a set of methods that source plugins can implement or override. Any
5762plugins (subclass of ``SourcePlugin``) that do not implement a
5763particular method inherit the implementation of the method from the
5764``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5765class in the ``pluginbase.py`` file for details:
5766
5767The following list describes the methods implemented in the
5768``SourcePlugin`` class:
5769
5770- ``do_prepare_partition()``: Called to populate a partition with
5771 actual content. In other words, the method prepares the final
5772 partition image that is incorporated into the disk image.
5773
5774- ``do_configure_partition()``: Called before
5775 ``do_prepare_partition()`` to create custom configuration files for a
5776 partition (e.g. syslinux or grub configuration files).
5777
5778- ``do_install_disk()``: Called after all partitions have been
5779 prepared and assembled into a disk image. This method provides a hook
5780 to allow finalization of a disk image (e.g. writing an MBR).
5781
5782- ``do_stage_partition()``: Special content-staging hook called
5783 before ``do_prepare_partition()``. This method is normally empty.
5784
5785 Typically, a partition just uses the passed-in parameters (e.g. the
5786 unmodified value of ``bootimg_dir``). However, in some cases, things
5787 might need to be more tailored. As an example, certain files might
5788 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5789 allows those files to be staged in a customized fashion.
5790
5791 .. note::
5792
5793 ``get_bitbake_var()`` allows you to access non-standard variables that
5794 you might want to use for this behavior.
5795
5796You can extend the source plugin mechanism. To add more hooks, create
5797more source plugin methods within ``SourcePlugin`` and the corresponding
5798derived subclasses. The code that calls the plugin methods uses the
5799``plugin.get_source_plugin_methods()`` function to find the method or
5800methods needed by the call. Retrieval of those methods is accomplished
5801by filling up a dict with keys that contain the method names of
5802interest. On success, these will be filled in with the actual methods.
5803See the Wic implementation for examples and details.
5804
5805Wic Examples
5806------------
5807
5808This section provides several examples that show how to use the Wic
5809utility. All the examples assume the list of requirements in the
5810":ref:`dev-manual/common-tasks:requirements`" section have been met. The
5811examples assume the previously generated image is
5812``core-image-minimal``.
5813
5814Generate an Image using an Existing Kickstart File
5815~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5816
5817This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
5818file::
5819
5820 $ wic create mkefidisk -e core-image-minimal
5821 INFO: Building wic-tools...
5822 .
5823 .
5824 .
5825 INFO: The new image(s) can be found here:
5826 ./mkefidisk-201804191017-sda.direct
5827
5828 The following build artifacts were used to create the image(s):
5829 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5830 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5831 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5832 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5833
5834 INFO: The image(s) were created using OE kickstart file:
5835 /home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
5836
5837The previous example shows the easiest way to create an image by running
5838in cooked mode and supplying a kickstart file and the "-e" option to
5839point to the existing build artifacts. Your ``local.conf`` file needs to
5840have the :term:`MACHINE` variable set
5841to the machine you are using, which is "qemux86" in this example.
5842
5843Once the image builds, the output provides image location, artifact use,
5844and kickstart file information.
5845
5846.. note::
5847
5848 You should always verify the details provided in the output to make
5849 sure that the image was indeed created exactly as expected.
5850
5851Continuing with the example, you can now write the image from the
5852:term:`Build Directory` onto a USB stick, or whatever media for which you
5853built your image, and boot from the media. You can write the image by using
5854``bmaptool`` or ``dd``::
5855
5856 $ oe-run-native bmap-tools-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
5857
5858or ::
5859
5860 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5861
5862.. note::
5863
5864 For more information on how to use the ``bmaptool``
5865 to flash a device with an image, see the
5866 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``"
5867 section.
5868
5869Using a Modified Kickstart File
5870~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5871
5872Because partitioned image creation is driven by the kickstart file, it
5873is easy to affect image creation by changing the parameters in the file.
5874This next example demonstrates that through modification of the
5875``directdisk-gpt`` kickstart file.
5876
5877As mentioned earlier, you can use the command ``wic list images`` to
5878show the list of existing kickstart files. The directory in which the
5879``directdisk-gpt.wks`` file resides is
5880``scripts/lib/image/canned-wks/``, which is located in the
5881:term:`Source Directory` (e.g. ``poky``).
5882Because available files reside in this directory, you can create and add
5883your own custom files to the directory. Subsequent use of the
5884``wic list images`` command would then include your kickstart files.
5885
5886In this example, the existing ``directdisk-gpt`` file already does most
5887of what is needed. However, for the hardware in this example, the image
5888will need to boot from ``sdb`` instead of ``sda``, which is what the
5889``directdisk-gpt`` kickstart file uses.
5890
5891The example begins by making a copy of the ``directdisk-gpt.wks`` file
5892in the ``scripts/lib/image/canned-wks`` directory and then by changing
5893the lines that specify the target disk from which to boot.
5894::
5895
5896 $ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5897 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5898
5899Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5900changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5901example changes the following two lines and leaves the remaining lines
5902untouched::
5903
5904 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5905 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5906
5907Once the lines are changed, the
5908example generates the ``directdisksdb-gpt`` image. The command points
5909the process at the ``core-image-minimal`` artifacts for the Next Unit of
5910Computing (nuc) :term:`MACHINE` the
5911``local.conf``.
5912::
5913
5914 $ wic create directdisksdb-gpt -e core-image-minimal
5915 INFO: Building wic-tools...
5916 .
5917 .
5918 .
5919 Initialising tasks: 100% |#######################################| Time: 0:00:01
5920 NOTE: Executing SetScene Tasks
5921 NOTE: Executing RunQueue Tasks
5922 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5923 INFO: Creating image(s)...
5924
5925 INFO: The new image(s) can be found here:
5926 ./directdisksdb-gpt-201710090938-sdb.direct
5927
5928 The following build artifacts were used to create the image(s):
5929 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5930 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5931 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5932 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5933
5934 INFO: The image(s) were created using OE kickstart file:
5935 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5936
5937Continuing with the example, you can now directly ``dd`` the image to a
5938USB stick, or whatever media for which you built your image, and boot
5939the resulting media::
5940
5941 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5942 140966+0 records in
5943 140966+0 records out
5944 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5945 $ sudo eject /dev/sdb
5946
5947Using a Modified Kickstart File and Running in Raw Mode
5948~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5949
5950This next example manually specifies each build artifact (runs in Raw
5951Mode) and uses a modified kickstart file. The example also uses the
5952``-o`` option to cause Wic to create the output somewhere other than the
5953default output directory, which is the current directory::
5954
5955 $ wic create test.wks -o /home/stephano/testwic \
5956 --rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5957 --bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5958 --kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \
5959 --native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
5960
5961 INFO: Creating image(s)...
5962
5963 INFO: The new image(s) can be found here:
5964 /home/stephano/testwic/test-201710091445-sdb.direct
5965
5966 The following build artifacts were used to create the image(s):
5967 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5968 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5969 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5970 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5971
5972 INFO: The image(s) were created using OE kickstart file:
5973 test.wks
5974
5975For this example,
5976:term:`MACHINE` did not have to be
5977specified in the ``local.conf`` file since the artifact is manually
5978specified.
5979
5980Using Wic to Manipulate an Image
5981~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5982
5983Wic image manipulation allows you to shorten turnaround time during
5984image development. For example, you can use Wic to delete the kernel
5985partition of a Wic image and then insert a newly built kernel. This
5986saves you time from having to rebuild the entire image each time you
5987modify the kernel.
5988
5989.. note::
5990
5991 In order to use Wic to manipulate a Wic image as in this example,
5992 your development machine must have the ``mtools`` package installed.
5993
5994The following example examines the contents of the Wic image, deletes
5995the existing kernel, and then inserts a new kernel:
5996
59971. *List the Partitions:* Use the ``wic ls`` command to list all the
5998 partitions in the Wic image::
5999
6000 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
6001 Num Start End Size Fstype
6002 1 1048576 25041919 23993344 fat16
6003 2 25165824 72157183 46991360 ext4
6004
6005 The previous output shows two partitions in the
6006 ``core-image-minimal-qemux86.wic`` image.
6007
60082. *Examine a Particular Partition:* Use the ``wic ls`` command again
6009 but in a different form to examine a particular partition.
6010
6011 .. note::
6012
6013 You can get command usage on any Wic command using the following
6014 form::
6015
6016 $ wic help command
6017
6018
6019 For example, the following command shows you the various ways to
6020 use the
6021 wic ls
6022 command::
6023
6024 $ wic help ls
6025
6026
6027 The following command shows what is in partition one::
6028
6029 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
6030 Volume in drive : is boot
6031 Volume Serial Number is E894-1809
6032 Directory for ::/
6033
6034 libcom32 c32 186500 2017-10-09 16:06
6035 libutil c32 24148 2017-10-09 16:06
6036 syslinux cfg 220 2017-10-09 16:06
6037 vesamenu c32 27104 2017-10-09 16:06
6038 vmlinuz 6904608 2017-10-09 16:06
6039 5 files 7 142 580 bytes
6040 16 582 656 bytes free
6041
6042 The previous output shows five files, with the
6043 ``vmlinuz`` being the kernel.
6044
6045 .. note::
6046
6047 If you see the following error, you need to update or create a
6048 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
6049 in the file. Then, run the Wic command again::
6050
6051 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
6052 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
6053 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
6054
6055
60563. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
6057 ``vmlinuz`` file (kernel)::
6058
6059 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6060
60614. *Add In the New Kernel:* Use the ``wic cp`` command to add the
6062 updated kernel to the Wic image. Depending on how you built your
6063 kernel, it could be in different places. If you used ``devtool`` and
6064 an SDK to build your kernel, it resides in the ``tmp/work`` directory
6065 of the extensible SDK. If you used ``make`` to build the kernel, the
6066 kernel will be in the ``workspace/sources`` area.
6067
6068 The following example assumes ``devtool`` was used to build the
6069 kernel::
6070
6071 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
6072 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6073
6074 Once the new kernel is added back into the image, you can use the
6075 ``dd`` command or :ref:`bmaptool
6076 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>`
6077 to flash your wic image onto an SD card or USB stick and test your
6078 target.
6079
6080 .. note::
6081
6082 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
6083
6084Flashing Images Using ``bmaptool``
6085==================================
6086
6087A fast and easy way to flash an image to a bootable device is to use
6088Bmaptool, which is integrated into the OpenEmbedded build system.
6089Bmaptool is a generic tool that creates a file's block map (bmap) and
6090then uses that map to copy the file. As compared to traditional tools
6091such as dd or cp, Bmaptool can copy (or flash) large files like raw
6092system image files much faster.
6093
6094.. note::
6095
6096 - If you are using Ubuntu or Debian distributions, you can install
6097 the ``bmap-tools`` package using the following command and then
6098 use the tool without specifying ``PATH`` even from the root
6099 account::
6100
6101 $ sudo apt install bmap-tools
6102
6103 - If you are unable to install the ``bmap-tools`` package, you will
6104 need to build Bmaptool before using it. Use the following command::
6105
6106 $ bitbake bmap-tools-native
6107
6108Following, is an example that shows how to flash a Wic image. Realize
6109that while this example uses a Wic image, you can use Bmaptool to flash
6110any type of image. Use these steps to flash an image using Bmaptool:
6111
61121. *Update your local.conf File:* You need to have the following set
6113 in your ``local.conf`` file before building your image::
6114
6115 IMAGE_FSTYPES += "wic wic.bmap"
6116
61172. *Get Your Image:* Either have your image ready (pre-built with the
6118 :term:`IMAGE_FSTYPES`
6119 setting previously mentioned) or take the step to build the image::
6120
6121 $ bitbake image
6122
61233. *Flash the Device:* Flash the device with the image by using Bmaptool
6124 depending on your particular setup. The following commands assume the
6125 image resides in the :term:`Build Directory`'s ``deploy/images/`` area:
6126
6127 - If you have write access to the media, use this command form::
6128
6129 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6130
6131 - If you do not have write access to the media, set your permissions
6132 first and then use the same command form::
6133
6134 $ sudo chmod 666 /dev/sdX
6135 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6136
6137For help on the ``bmaptool`` command, use the following command::
6138
6139 $ bmaptool --help
6140
6141Making Images More Secure
6142=========================
6143
6144Security is of increasing concern for embedded devices. Consider the
6145issues and problems discussed in just this sampling of work found across
6146the Internet:
6147
6148- *"*\ `Security Risks of Embedded
6149 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
6150 by Bruce Schneier
6151
6152- *"*\ `Internet Census
6153 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
6154 Botnet
6155
6156- *"*\ `Security Issues for Embedded
6157 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
6158 by Jake Edge
6159
6160When securing your image is of concern, there are steps, tools, and
6161variables that you can consider to help you reach the security goals you
6162need for your particular device. Not all situations are identical when
6163it comes to making an image secure. Consequently, this section provides
6164some guidance and suggestions for consideration when you want to make
6165your image more secure.
6166
6167.. note::
6168
6169 Because the security requirements and risks are different for every
6170 type of device, this section cannot provide a complete reference on
6171 securing your custom OS. It is strongly recommended that you also
6172 consult other sources of information on embedded Linux system
6173 hardening and on security.
6174
6175General Considerations
6176----------------------
6177
6178There are general considerations that help you create more secure images.
6179You should consider the following suggestions to make your device
6180more secure:
6181
6182- Scan additional code you are adding to the system (e.g. application
6183 code) by using static analysis tools. Look for buffer overflows and
6184 other potential security problems.
6185
6186- Pay particular attention to the security for any web-based
6187 administration interface.
6188
6189 Web interfaces typically need to perform administrative functions and
6190 tend to need to run with elevated privileges. Thus, the consequences
6191 resulting from the interface's security becoming compromised can be
6192 serious. Look for common web vulnerabilities such as
6193 cross-site-scripting (XSS), unvalidated inputs, and so forth.
6194
6195 As with system passwords, the default credentials for accessing a
6196 web-based interface should not be the same across all devices. This
6197 is particularly true if the interface is enabled by default as it can
6198 be assumed that many end-users will not change the credentials.
6199
6200- Ensure you can update the software on the device to mitigate
6201 vulnerabilities discovered in the future. This consideration
6202 especially applies when your device is network-enabled.
6203
6204- Regularly scan and apply fixes for CVE security issues affecting
6205 all software components in the product, see ":ref:`dev-manual/common-tasks:checking for vulnerabilities`".
6206
6207- Regularly update your version of Poky and OE-Core from their upstream
6208 developers, e.g. to apply updates and security fixes from stable
6209 and LTS branches.
6210
6211- Ensure you remove or disable debugging functionality before producing
6212 the final image. For information on how to do this, see the
6213 ":ref:`dev-manual/common-tasks:considerations specific to the openembedded build system`"
6214 section.
6215
6216- Ensure you have no network services listening that are not needed.
6217
6218- Remove any software from the image that is not needed.
6219
6220- Enable hardware support for secure boot functionality when your
6221 device supports this functionality.
6222
6223Security Flags
6224--------------
6225
6226The Yocto Project has security flags that you can enable that help make
6227your build output more secure. The security flags are in the
6228``meta/conf/distro/include/security_flags.inc`` file in your
6229:term:`Source Directory` (e.g. ``poky``).
6230
6231.. note::
6232
6233 Depending on the recipe, certain security flags are enabled and
6234 disabled by default.
6235
6236Use the following line in your ``local.conf`` file or in your custom
6237distribution configuration file to enable the security compiler and
6238linker flags for your build::
6239
6240 require conf/distro/include/security_flags.inc
6241
6242Considerations Specific to the OpenEmbedded Build System
6243--------------------------------------------------------
6244
6245You can take some steps that are specific to the OpenEmbedded build
6246system to make your images more secure:
6247
6248- Ensure "debug-tweaks" is not one of your selected
6249 :term:`IMAGE_FEATURES`.
6250 When creating a new project, the default is to provide you with an
6251 initial ``local.conf`` file that enables this feature using the
6252 :term:`EXTRA_IMAGE_FEATURES`
6253 variable with the line::
6254
6255 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6256
6257 To disable that feature, simply comment out that line in your
6258 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
6259 "debug-tweaks" before producing your final image. Among other things,
6260 leaving this in place sets the root password as blank, which makes
6261 logging in for debugging or inspection easy during development but
6262 also means anyone can easily log in during production.
6263
6264- It is possible to set a root password for the image and also to set
6265 passwords for any extra users you might add (e.g. administrative or
6266 service type users). When you set up passwords for multiple images or
6267 users, you should not duplicate passwords.
6268
6269 To set up passwords, use the
6270 :ref:`extrausers <ref-classes-extrausers>`
6271 class, which is the preferred method. For an example on how to set up
6272 both root and user passwords, see the
6273 ":ref:`ref-classes-extrausers`" section.
6274
6275 .. note::
6276
6277 When adding extra user accounts or setting a root password, be
6278 cautious about setting the same password on every device. If you
6279 do this, and the password you have set is exposed, then every
6280 device is now potentially compromised. If you need this access but
6281 want to ensure security, consider setting a different, random
6282 password for each device. Typically, you do this as a separate
6283 step after you deploy the image onto the device.
6284
6285- Consider enabling a Mandatory Access Control (MAC) framework such as
6286 SMACK or SELinux and tuning it appropriately for your device's usage.
6287 You can find more information in the
6288 :yocto_git:`meta-selinux </meta-selinux/>` layer.
6289
6290Tools for Hardening Your Image
6291------------------------------
6292
6293The Yocto Project provides tools for making your image more secure. You
6294can find these tools in the ``meta-security`` layer of the
6295:yocto_git:`Yocto Project Source Repositories <>`.
6296
6297Creating Your Own Distribution
6298==============================
6299
6300When you build an image using the Yocto Project and do not alter any
6301distribution :term:`Metadata`, you are
6302creating a Poky distribution. If you wish to gain more control over
6303package alternative selections, compile-time options, and other
6304low-level configurations, you can create your own distribution.
6305
6306To create your own distribution, the basic steps consist of creating
6307your own distribution layer, creating your own distribution
6308configuration file, and then adding any needed code and Metadata to the
6309layer. The following steps provide some more detail:
6310
6311- *Create a layer for your new distro:* Create your distribution layer
6312 so that you can keep your Metadata and code for the distribution
6313 separate. It is strongly recommended that you create and use your own
6314 layer for configuration and code. Using your own layer as compared to
6315 just placing configurations in a ``local.conf`` configuration file
6316 makes it easier to reproduce the same build configuration when using
6317 multiple build machines. See the
6318 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
6319 section for information on how to quickly set up a layer.
6320
6321- *Create the distribution configuration file:* The distribution
6322 configuration file needs to be created in the ``conf/distro``
6323 directory of your layer. You need to name it using your distribution
6324 name (e.g. ``mydistro.conf``).
6325
6326 .. note::
6327
6328 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6329 name of your distribution.
6330
6331 You can split out parts of your configuration file into include files
6332 and then "require" them from within your distribution configuration
6333 file. Be sure to place the include files in the
6334 ``conf/distro/include`` directory of your layer. A common example
6335 usage of include files would be to separate out the selection of
6336 desired version and revisions for individual recipes.
6337
6338 Your configuration file needs to set the following required
6339 variables:
6340
6341 - :term:`DISTRO_NAME`
6342
6343 - :term:`DISTRO_VERSION`
6344
6345 These following variables are optional and you typically set them
6346 from the distribution configuration file:
6347
6348 - :term:`DISTRO_FEATURES`
6349
6350 - :term:`DISTRO_EXTRA_RDEPENDS`
6351
6352 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6353
6354 - :term:`TCLIBC`
6355
6356 .. tip::
6357
6358 If you want to base your distribution configuration file on the
6359 very basic configuration from OE-Core, you can use
6360 ``conf/distro/defaultsetup.conf`` as a reference and just include
6361 variables that differ as compared to ``defaultsetup.conf``.
6362 Alternatively, you can create a distribution configuration file
6363 from scratch using the ``defaultsetup.conf`` file or configuration files
6364 from another distribution such as Poky as a reference.
6365
6366- *Provide miscellaneous variables:* Be sure to define any other
6367 variables for which you want to create a default or enforce as part
6368 of the distribution configuration. You can include nearly any
6369 variable from the ``local.conf`` file. The variables you use are not
6370 limited to the list in the previous bulleted item.
6371
6372- *Point to Your distribution configuration file:* In your ``local.conf``
6373 file in the :term:`Build Directory`, set your :term:`DISTRO` variable to
6374 point to your distribution's configuration file. For example, if your
6375 distribution's configuration file is named ``mydistro.conf``, then
6376 you point to it as follows::
6377
6378 DISTRO = "mydistro"
6379
6380- *Add more to the layer if necessary:* Use your layer to hold other
6381 information needed for the distribution:
6382
6383 - Add recipes for installing distro-specific configuration files
6384 that are not already installed by another recipe. If you have
6385 distro-specific configuration files that are included by an
6386 existing recipe, you should add an append file (``.bbappend``) for
6387 those. For general information and recommendations on how to add
6388 recipes to your layer, see the
6389 ":ref:`dev-manual/common-tasks:creating your own layer`" and
6390 ":ref:`dev-manual/common-tasks:following best practices when creating layers`"
6391 sections.
6392
6393 - Add any image recipes that are specific to your distribution.
6394
6395 - Add a ``psplash`` append file for a branded splash screen. For
6396 information on append files, see the
6397 ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`"
6398 section.
6399
6400 - Add any other append files to make custom changes that are
6401 specific to individual recipes.
6402
6403Creating a Custom Template Configuration Directory
6404==================================================
6405
6406If you are producing your own customized version of the build system for
6407use by other users, you might want to provide a custom build configuration
6408that includes all the necessary settings and layers (i.e. ``local.conf`` and
6409``bblayers.conf`` that are created in a new :term:`Build Directory`) and a custom
6410message that is shown when setting up the build. This can be done by
6411creating one or more template configuration directories in your
6412custom distribution layer.
6413
6414This can be done by using ``bitbake-layers save-build-conf``::
6415
6416 $ bitbake-layers save-build-conf ../../meta-alex/ test-1
6417 NOTE: Starting bitbake server...
6418 NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
6419 Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
6420 You can try out the configuration with
6421 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
6422
6423The above command takes the config files from the currently active :term:`Build Directory` under ``conf``,
6424replaces site-specific paths in ``bblayers.conf`` with ``##OECORE##``-relative paths, and copies
6425the config files into a specified layer under a specified template name.
6426
6427To use those saved templates as a starting point for a build, users should point
6428to one of them with :term:`TEMPLATECONF` environment variable::
6429
6430 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
6431
6432The OpenEmbedded build system uses the environment variable
6433:term:`TEMPLATECONF` to locate the directory from which it gathers
6434configuration information that ultimately ends up in the
6435:term:`Build Directory` ``conf`` directory.
6436
6437If :term:`TEMPLATECONF` is not set, the default value is obtained
6438from ``.templateconf`` file that is read from the same directory as
6439``oe-init-build-env`` script. For the Poky reference distribution this
6440would be::
6441
6442 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf/templates/default}
6443
6444If you look at a configuration template directory, you will
6445see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6446``conf-notes.txt`` files. The build system uses these files to form the
6447respective ``bblayers.conf`` file, ``local.conf`` file, and show
6448users a note about the build they're setting up
6449when running the ``oe-init-build-env`` setup script. These can be
6450edited further if needed to improve or change the build configurations
6451available to the users.
6452
6453Conserving Disk Space
6454=====================
6455
6456Conserving Disk Space During Builds
6457-----------------------------------
6458
6459To help conserve disk space during builds, you can add the following
6460statement to your project's ``local.conf`` configuration file found in
6461the :term:`Build Directory`::
6462
6463 INHERIT += "rm_work"
6464
6465Adding this statement deletes the work directory used for
6466building a recipe once the recipe is built. For more information on
6467"rm_work", see the
6468:ref:`rm_work <ref-classes-rm-work>` class in the
6469Yocto Project Reference Manual.
6470
6471Purging Duplicate Shared State Cache Files
6472-------------------------------------------
6473
6474After multiple build iterations, the Shared State (sstate) cache can contain
6475duplicate cache files for a given package, while only the most recent one
6476is likely to be reusable. The following command purges all but the
6477newest sstate cache file for each package::
6478
6479 sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
6480
6481This command will ask you to confirm the deletions it identifies.
6482
6483.. note::
6484
6485 The duplicated sstate cache files of one package must have the same
6486 architecture, which means that sstate cache files with multiple
6487 architectures are not considered as duplicate.
6488
6489Run ``sstate-cache-management.sh`` for more details about this script.
6490
6491Working with Packages
6492=====================
6493
6494This section describes a few tasks that involve packages:
6495
6496- :ref:`dev-manual/common-tasks:excluding packages from an image`
6497
6498- :ref:`dev-manual/common-tasks:incrementing a package version`
6499
6500- :ref:`dev-manual/common-tasks:handling optional module packaging`
6501
6502- :ref:`dev-manual/common-tasks:using runtime package management`
6503
6504- :ref:`dev-manual/common-tasks:generating and using signed packages`
6505
6506- :ref:`Setting up and running package test
6507 (ptest) <dev-manual/common-tasks:testing packages with ptest>`
6508
6509- :ref:`dev-manual/common-tasks:creating node package manager (npm) packages`
6510
6511- :ref:`dev-manual/common-tasks:adding custom metadata to packages`
6512
6513Excluding Packages from an Image
6514--------------------------------
6515
6516You might find it necessary to prevent specific packages from being
6517installed into an image. If so, you can use several variables to direct
6518the build system to essentially ignore installing recommended packages
6519or to not install a package at all.
6520
6521The following list introduces variables you can use to prevent packages
6522from being installed into your image. Each of these variables only works
6523with IPK and RPM package types, not for Debian packages.
6524Also, you can use these variables from your ``local.conf`` file
6525or attach them to a specific image recipe by using a recipe name
6526override. For more detail on the variables, see the descriptions in the
6527Yocto Project Reference Manual's glossary chapter.
6528
6529- :term:`BAD_RECOMMENDATIONS`:
6530 Use this variable to specify "recommended-only" packages that you do
6531 not want installed.
6532
6533- :term:`NO_RECOMMENDATIONS`:
6534 Use this variable to prevent all "recommended-only" packages from
6535 being installed.
6536
6537- :term:`PACKAGE_EXCLUDE`:
6538 Use this variable to prevent specific packages from being installed
6539 regardless of whether they are "recommended-only" or not. You need to
6540 realize that the build process could fail with an error when you
6541 prevent the installation of a package whose presence is required by
6542 an installed package.
6543
6544Incrementing a Package Version
6545------------------------------
6546
6547This section provides some background on how binary package versioning
6548is accomplished and presents some of the services, variables, and
6549terminology involved.
6550
6551In order to understand binary package versioning, you need to consider
6552the following:
6553
6554- Binary Package: The binary package that is eventually built and
6555 installed into an image.
6556
6557- Binary Package Version: The binary package version is composed of two
6558 components --- a version and a revision.
6559
6560 .. note::
6561
6562 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
6563 but this discussion for the most part ignores :term:`PE`.
6564
6565 The version and revision are taken from the
6566 :term:`PV` and
6567 :term:`PR` variables, respectively.
6568
6569- :term:`PV`: The recipe version. :term:`PV` represents the version of the
6570 software being packaged. Do not confuse :term:`PV` with the binary
6571 package version.
6572
6573- :term:`PR`: The recipe revision.
6574
6575- :term:`SRCPV`: The OpenEmbedded
6576 build system uses this string to help define the value of :term:`PV` when
6577 the source code revision needs to be included in it.
6578
6579- :yocto_wiki:`PR Service </PR_Service>`: A
6580 network-based service that helps automate keeping package feeds
6581 compatible with existing package manager applications such as RPM,
6582 APT, and OPKG.
6583
6584Whenever the binary package content changes, the binary package version
6585must change. Changing the binary package version is accomplished by
6586changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
6587values occurs one of two ways:
6588
6589- Automatically using a Package Revision Service (PR Service).
6590
6591- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
6592
6593Given a primary challenge of any build system and its users is how to
6594maintain a package feed that is compatible with existing package manager
6595applications such as RPM, APT, and OPKG, using an automated system is
6596much preferred over a manual system. In either system, the main
6597requirement is that binary package version numbering increases in a
6598linear fashion and that there is a number of version components that
6599support that linear progression. For information on how to ensure
6600package revisioning remains linear, see the
6601":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
6602section.
6603
6604The following three sections provide related information on the PR
6605Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
6606how to ensure binary package revisioning remains linear.
6607
6608Working With a PR Service
6609~~~~~~~~~~~~~~~~~~~~~~~~~
6610
6611As mentioned, attempting to maintain revision numbers in the
6612:term:`Metadata` is error prone, inaccurate,
6613and causes problems for people submitting recipes. Conversely, the PR
6614Service automatically generates increasing numbers, particularly the
6615revision field, which removes the human element.
6616
6617.. note::
6618
6619 For additional information on using a PR Service, you can see the
6620 :yocto_wiki:`PR Service </PR_Service>` wiki page.
6621
6622The Yocto Project uses variables in order of decreasing priority to
6623facilitate revision numbering (i.e.
6624:term:`PE`,
6625:term:`PV`, and
6626:term:`PR` for epoch, version, and
6627revision, respectively). The values are highly dependent on the policies
6628and procedures of a given distribution and package feed.
6629
6630Because the OpenEmbedded build system uses
6631":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
6632unique to a given build, the build system knows when to rebuild
6633packages. All the inputs into a given task are represented by a
6634signature, which can trigger a rebuild when different. Thus, the build
6635system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
6636trigger a rebuild. The signatures, however, can be used to generate
6637these values.
6638
6639The PR Service works with both ``OEBasic`` and ``OEBasicHash``
6640generators. The value of :term:`PR` bumps when the checksum changes and the
6641different generator mechanisms change signatures under different
6642circumstances.
6643
6644As implemented, the build system includes values from the PR Service
6645into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
6646becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
6647:term:`PR` values to be used for whatever reasons, which include manual
6648:term:`PR` bumps, should it be necessary.
6649
6650By default, the PR Service is not enabled or running. Thus, the packages
6651generated are just "self consistent". The build system adds and removes
6652packages and there are no guarantees about upgrade paths but images will
6653be consistent and correct with the latest changes.
6654
6655The simplest form for a PR Service is for a single host development system
6656that builds the package feed (building system). For this scenario, you can
6657enable a local PR Service by setting :term:`PRSERV_HOST` in your
6658``local.conf`` file in the :term:`Build Directory`::
6659
6660 PRSERV_HOST = "localhost:0"
6661
6662Once the service is started, packages will automatically
6663get increasing :term:`PR` values and BitBake takes care of starting and
6664stopping the server.
6665
6666If you have a more complex setup where multiple host development systems
6667work against a common, shared package feed, you have a single PR Service
6668running and it is connected to each building system. For this scenario,
6669you need to start the PR Service using the ``bitbake-prserv`` command::
6670
6671 bitbake-prserv --host ip --port port --start
6672
6673In addition to
6674hand-starting the service, you need to update the ``local.conf`` file of
6675each building system as described earlier so each system points to the
6676server and port.
6677
6678It is also recommended you use build history, which adds some sanity
6679checks to binary package versions, in conjunction with the server that
6680is running the PR Service. To enable build history, add the following to
6681each building system's ``local.conf`` file::
6682
6683 # It is recommended to activate "buildhistory" for testing the PR service
6684 INHERIT += "buildhistory"
6685 BUILDHISTORY_COMMIT = "1"
6686
6687For information on build
6688history, see the
6689":ref:`dev-manual/common-tasks:maintaining build output quality`" section.
6690
6691.. note::
6692
6693 The OpenEmbedded build system does not maintain :term:`PR` information as
6694 part of the shared state (sstate) packages. If you maintain an sstate
6695 feed, it's expected that either all your building systems that
6696 contribute to the sstate feed use a shared PR Service, or you do not
6697 run a PR Service on any of your building systems. Having some systems
6698 use a PR Service while others do not leads to obvious problems.
6699
6700 For more information on shared state, see the
6701 ":ref:`overview-manual/concepts:shared state cache`"
6702 section in the Yocto Project Overview and Concepts Manual.
6703
6704Manually Bumping PR
6705~~~~~~~~~~~~~~~~~~~
6706
6707The alternative to setting up a PR Service is to manually "bump" the
6708:term:`PR` variable.
6709
6710If a committed change results in changing the package output, then the
6711value of the PR variable needs to be increased (or "bumped") as part of
6712that commit. For new recipes you should add the :term:`PR` variable and set
6713its initial value equal to "r0", which is the default. Even though the
6714default value is "r0", the practice of adding it to a new recipe makes
6715it harder to forget to bump the variable when you make changes to the
6716recipe in future.
6717
6718If you are sharing a common ``.inc`` file with multiple recipes, you can
6719also use the :term:`INC_PR` variable to ensure that the recipes sharing the
6720``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
6721``.inc`` file must set :term:`INC_PR` (initially to "r0"), and all recipes
6722referring to it should set :term:`PR` to "${INC_PR}.0" initially,
6723incrementing the last number when the recipe is changed. If the ``.inc``
6724file is changed then its :term:`INC_PR` should be incremented.
6725
6726When upgrading the version of a binary package, assuming the :term:`PV`
6727changes, the :term:`PR` variable should be reset to "r0" (or "${INC_PR}.0"
6728if you are using :term:`INC_PR`).
6729
6730Usually, version increases occur only to binary packages. However, if
6731for some reason :term:`PV` changes but does not increase, you can increase
6732the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
6733"0".
6734
6735Binary package version numbering strives to follow the `Debian Version
6736Field Policy
6737Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
6738These guidelines define how versions are compared and what "increasing"
6739a version means.
6740
6741Automatically Incrementing a Package Version Number
6742~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6743
6744When fetching a repository, BitBake uses the
6745:term:`SRCREV` variable to determine
6746the specific source code revision from which to build. You set the
6747:term:`SRCREV` variable to
6748:term:`AUTOREV` to cause the
6749OpenEmbedded build system to automatically use the latest revision of
6750the software::
6751
6752 SRCREV = "${AUTOREV}"
6753
6754Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
6755automatically update the version whenever the revision of the source
6756code changes. Here is an example::
6757
6758 PV = "1.0+git${SRCPV}"
6759
6760The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
6761
6762.. code-block:: none
6763
6764 AUTOINC+source_code_revision
6765
6766The build system replaces the ``AUTOINC``
6767with a number. The number used depends on the state of the PR Service:
6768
6769- If PR Service is enabled, the build system increments the number,
6770 which is similar to the behavior of
6771 :term:`PR`. This behavior results in
6772 linearly increasing package versions, which is desirable. Here is an
6773 example:
6774
6775 .. code-block:: none
6776
6777 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6778 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6779
6780- If PR Service is not enabled, the build system replaces the
6781 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6782 changing the package version since the source revision is included.
6783 However, package versions are not increased linearly. Here is an
6784 example:
6785
6786 .. code-block:: none
6787
6788 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6789 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6790
6791In summary, the OpenEmbedded build system does not track the history of
6792binary package versions for this purpose. ``AUTOINC``, in this case, is
6793comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
6794package version is simply replaced by "0". If PR server is enabled, the
6795build system keeps track of the package versions and bumps the number
6796when the package revision changes.
6797
6798Handling Optional Module Packaging
6799----------------------------------
6800
6801Many pieces of software split functionality into optional modules (or
6802plugins) and the plugins that are built might depend on configuration
6803options. To avoid having to duplicate the logic that determines what
6804modules are available in your recipe or to avoid having to package each
6805module by hand, the OpenEmbedded build system provides functionality to
6806handle module packaging dynamically.
6807
6808To handle optional module packaging, you need to do two things:
6809
6810- Ensure the module packaging is actually done.
6811
6812- Ensure that any dependencies on optional modules from other recipes
6813 are satisfied by your recipe.
6814
6815Making Sure the Packaging is Done
6816~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6817
6818To ensure the module packaging actually gets done, you use the
6819``do_split_packages`` function within the ``populate_packages`` Python
6820function in your recipe. The ``do_split_packages`` function searches for
6821a pattern of files or directories under a specified path and creates a
6822package for each one it finds by appending to the
6823:term:`PACKAGES` variable and
6824setting the appropriate values for ``FILES:packagename``,
6825``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
6826Here is an example from the ``lighttpd`` recipe::
6827
6828 python populate_packages:prepend () {
6829 lighttpd_libdir = d.expand('${libdir}')
6830 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6831 'lighttpd-module-%s', 'Lighttpd module for %s',
6832 extra_depends='')
6833 }
6834
6835The previous example specifies a number of things in the call to
6836``do_split_packages``.
6837
6838- A directory within the files installed by your recipe through
6839 :ref:`ref-tasks-install` in which to search.
6840
6841- A regular expression used to match module files in that directory. In
6842 the example, note the parentheses () that mark the part of the
6843 expression from which the module name should be derived.
6844
6845- A pattern to use for the package names.
6846
6847- A description for each package.
6848
6849- An empty string for ``extra_depends``, which disables the default
6850 dependency on the main ``lighttpd`` package. Thus, if a file in
6851 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6852 ``lighttpd-module-alias`` is created for it and the
6853 :term:`DESCRIPTION` is set to
6854 "Lighttpd module for alias".
6855
6856Often, packaging modules is as simple as the previous example. However,
6857there are more advanced options that you can use within
6858``do_split_packages`` to modify its behavior. And, if you need to, you
6859can add more logic by specifying a hook function that is called for each
6860package. It is also perfectly acceptable to call ``do_split_packages``
6861multiple times if you have more than one set of modules to package.
6862
6863For more examples that show how to use ``do_split_packages``, see the
6864``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
6865directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
6866also find examples in ``meta/classes-recipe/kernel.bbclass``.
6867
6868Following is a reference that shows ``do_split_packages`` mandatory and
6869optional arguments::
6870
6871 Mandatory arguments
6872
6873 root
6874 The path in which to search
6875 file_regex
6876 Regular expression to match searched files.
6877 Use parentheses () to mark the part of this
6878 expression that should be used to derive the
6879 module name (to be substituted where %s is
6880 used in other function arguments as noted below)
6881 output_pattern
6882 Pattern to use for the package names. Must
6883 include %s.
6884 description
6885 Description to set for each package. Must
6886 include %s.
6887
6888 Optional arguments
6889
6890 postinst
6891 Postinstall script to use for all packages
6892 (as a string)
6893 recursive
6894 True to perform a recursive search --- default
6895 False
6896 hook
6897 A hook function to be called for every match.
6898 The function will be called with the following
6899 arguments (in the order listed):
6900
6901 f
6902 Full path to the file/directory match
6903 pkg
6904 The package name
6905 file_regex
6906 As above
6907 output_pattern
6908 As above
6909 modulename
6910 The module name derived using file_regex
6911 extra_depends
6912 Extra runtime dependencies (RDEPENDS) to be
6913 set for all packages. The default value of None
6914 causes a dependency on the main package
6915 (${PN}) --- if you do not want this, pass empty
6916 string '' for this parameter.
6917 aux_files_pattern
6918 Extra item(s) to be added to FILES for each
6919 package. Can be a single string item or a list
6920 of strings for multiple items. Must include %s.
6921 postrm
6922 postrm script to use for all packages (as a
6923 string)
6924 allow_dirs
6925 True to allow directories to be matched -
6926 default False
6927 prepend
6928 If True, prepend created packages to PACKAGES
6929 instead of the default False which appends them
6930 match_path
6931 match file_regex on the whole relative path to
6932 the root rather than just the filename
6933 aux_files_pattern_verbatim
6934 Extra item(s) to be added to FILES for each
6935 package, using the actual derived module name
6936 rather than converting it to something legal
6937 for a package name. Can be a single string item
6938 or a list of strings for multiple items. Must
6939 include %s.
6940 allow_links
6941 True to allow symlinks to be matched --- default
6942 False
6943 summary
6944 Summary to set for each package. Must include %s;
6945 defaults to description if not set.
6946
6947
6948
6949Satisfying Dependencies
6950~~~~~~~~~~~~~~~~~~~~~~~
6951
6952The second part for handling optional module packaging is to ensure that
6953any dependencies on optional modules from other recipes are satisfied by
6954your recipe. You can be sure these dependencies are satisfied by using
6955the :term:`PACKAGES_DYNAMIC`
6956variable. Here is an example that continues with the ``lighttpd`` recipe
6957shown earlier::
6958
6959 PACKAGES_DYNAMIC = "lighttpd-module-.*"
6960
6961The name
6962specified in the regular expression can of course be anything. In this
6963example, it is ``lighttpd-module-`` and is specified as the prefix to
6964ensure that any :term:`RDEPENDS` and
6965:term:`RRECOMMENDS` on a package
6966name starting with the prefix are satisfied during build time. If you
6967are using ``do_split_packages`` as described in the previous section,
6968the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
6969pattern specified in the call to ``do_split_packages``.
6970
6971Using Runtime Package Management
6972--------------------------------
6973
6974During a build, BitBake always transforms a recipe into one or more
6975packages. For example, BitBake takes the ``bash`` recipe and produces a
6976number of packages (e.g. ``bash``, ``bash-bashbug``,
6977``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
6978``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
6979generated packages are included in an image.
6980
6981In several situations, you might need to update, add, remove, or query
6982the packages on a target device at runtime (i.e. without having to
6983generate a new image). Examples of such situations include:
6984
6985- You want to provide in-the-field updates to deployed devices (e.g.
6986 security updates).
6987
6988- You want to have a fast turn-around development cycle for one or more
6989 applications that run on your device.
6990
6991- You want to temporarily install the "debug" packages of various
6992 applications on your device so that debugging can be greatly improved
6993 by allowing access to symbols and source debugging.
6994
6995- You want to deploy a more minimal package selection of your device
6996 but allow in-the-field updates to add a larger selection for
6997 customization.
6998
6999In all these situations, you have something similar to a more
7000traditional Linux distribution in that in-field devices are able to
7001receive pre-compiled packages from a server for installation or update.
7002Being able to install these packages on a running, in-field device is
7003what is termed "runtime package management".
7004
7005In order to use runtime package management, you need a host or server
7006machine that serves up the pre-compiled packages plus the required
7007metadata. You also need package manipulation tools on the target. The
7008build machine is a likely candidate to act as the server. However, that
7009machine does not necessarily have to be the package server. The build
7010machine could push its artifacts to another machine that acts as the
7011server (e.g. Internet-facing). In fact, doing so is advantageous for a
7012production environment as getting the packages away from the development
7013system's :term:`Build Directory` prevents accidental overwrites.
7014
7015A simple build that targets just one device produces more than one
7016package database. In other words, the packages produced by a build are
7017separated out into a couple of different package groupings based on
7018criteria such as the target's CPU architecture, the target board, or the
7019C library used on the target. For example, a build targeting the
7020``qemux86`` device produces the following three package databases:
7021``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
7022device to be aware of all the packages that were available to it, you
7023would need to point it to each of these databases individually. In a
7024similar way, a traditional Linux distribution usually is configured to
7025be aware of a number of software repositories from which it retrieves
7026packages.
7027
7028Using runtime package management is completely optional and not required
7029for a successful build or deployment in any way. But if you want to make
7030use of runtime package management, you need to do a couple things above
7031and beyond the basics. The remainder of this section describes what you
7032need to do.
7033
7034Build Considerations
7035~~~~~~~~~~~~~~~~~~~~
7036
7037This section describes build considerations of which you need to be
7038aware in order to provide support for runtime package management.
7039
7040When BitBake generates packages, it needs to know what format or formats
7041to use. In your configuration, you use the
7042:term:`PACKAGE_CLASSES`
7043variable to specify the format:
7044
70451. Open the ``local.conf`` file inside your :term:`Build Directory` (e.g.
7046 ``poky/build/conf/local.conf``).
7047
70482. Select the desired package format as follows::
7049
7050 PACKAGE_CLASSES ?= "package_packageformat"
7051
7052 where packageformat can be "ipk", "rpm",
7053 "deb", or "tar" which are the supported package formats.
7054
7055 .. note::
7056
7057 Because the Yocto Project supports four different package formats,
7058 you can set the variable with more than one argument. However, the
7059 OpenEmbedded build system only uses the first argument when
7060 creating an image or Software Development Kit (SDK).
7061
7062If you would like your image to start off with a basic package database
7063containing the packages in your current build as well as to have the
7064relevant tools available on the target for runtime package management,
7065you can include "package-management" in the
7066:term:`IMAGE_FEATURES`
7067variable. Including "package-management" in this configuration variable
7068ensures that when the image is assembled for your target, the image
7069includes the currently-known package databases as well as the
7070target-specific tools required for runtime package management to be
7071performed on the target. However, this is not strictly necessary. You
7072could start your image off without any databases but only include the
7073required on-target package tool(s). As an example, you could include
7074"opkg" in your
7075:term:`IMAGE_INSTALL` variable
7076if you are using the IPK package format. You can then initialize your
7077target's package database(s) later once your image is up and running.
7078
7079Whenever you perform any sort of build step that can potentially
7080generate a package or modify existing package, it is always a good idea
7081to re-generate the package index after the build by using the following
7082command::
7083
7084 $ bitbake package-index
7085
7086It might be tempting to build the
7087package and the package index at the same time with a command such as
7088the following::
7089
7090 $ bitbake some-package package-index
7091
7092Do not do this as
7093BitBake does not schedule the package index for after the completion of
7094the package you are building. Consequently, you cannot be sure of the
7095package index including information for the package you just built.
7096Thus, be sure to run the package update step separately after building
7097any packages.
7098
7099You can use the
7100:term:`PACKAGE_FEED_ARCHS`,
7101:term:`PACKAGE_FEED_BASE_PATHS`,
7102and
7103:term:`PACKAGE_FEED_URIS`
7104variables to pre-configure target images to use a package feed. If you
7105do not define these variables, then manual steps as described in the
7106subsequent sections are necessary to configure the target. You should
7107set these variables before building the image in order to produce a
7108correctly configured image.
7109
7110When your build is complete, your packages reside in the
7111``${TMPDIR}/deploy/packageformat`` directory. For example, if
7112``${``\ :term:`TMPDIR`\ ``}`` is
7113``tmp`` and your selected package type is RPM, then your RPM packages
7114are available in ``tmp/deploy/rpm``.
7115
7116Host or Server Machine Setup
7117~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7118
7119Although other protocols are possible, a server using HTTP typically
7120serves packages. If you want to use HTTP, then set up and configure a
7121web server such as Apache 2, lighttpd, or Python web server on the
7122machine serving the packages.
7123
7124To keep things simple, this section describes how to set up a
7125Python web server to share package feeds from the developer's
7126machine. Although this server might not be the best for a production
7127environment, the setup is simple and straight forward. Should you want
7128to use a different server more suited for production (e.g. Apache 2,
7129Lighttpd, or Nginx), take the appropriate steps to do so.
7130
7131From within the :term:`Build Directory` where you have built an image based on
7132your packaging choice (i.e. the :term:`PACKAGE_CLASSES` setting), simply start
7133the server. The following example assumes a :term:`Build Directory` of ``poky/build``
7134and a :term:`PACKAGE_CLASSES` setting of
7135":ref:`package_rpm <ref-classes-package_rpm>`"::
7136
7137 $ cd poky/build/tmp/deploy/rpm
7138 $ python3 -m http.server
7139
7140Target Setup
7141~~~~~~~~~~~~
7142
7143Setting up the target differs depending on the package management
7144system. This section provides information for RPM, IPK, and DEB.
7145
7146Using RPM
7147^^^^^^^^^
7148
7149The :wikipedia:`Dandified Packaging <DNF_(software)>` (DNF) performs
7150runtime package management of RPM packages. In order to use DNF for
7151runtime package management, you must perform an initial setup on the
7152target machine for cases where the ``PACKAGE_FEED_*`` variables were not
7153set as part of the image that is running on the target. This means if
7154you built your image and did not use these variables as part of the
7155build and your image is now running on the target, you need to perform
7156the steps in this section if you want to use runtime package management.
7157
7158.. note::
7159
7160 For information on the ``PACKAGE_FEED_*`` variables, see
7161 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
7162 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
7163 glossary.
7164
7165On the target, you must inform DNF that package databases are available.
7166You do this by creating a file named
7167``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
7168
7169As an example, assume the target is able to use the following package
7170databases: ``all``, ``i586``, and ``qemux86`` from a server named
7171``my.server``. The specifics for setting up the web server are up to
7172you. The critical requirement is that the URIs in the target repository
7173configuration point to the correct remote location for the feeds.
7174
7175.. note::
7176
7177 For development purposes, you can point the web server to the build
7178 system's ``deploy`` directory. However, for production use, it is better to
7179 copy the package directories to a location outside of the build area and use
7180 that location. Doing so avoids situations where the build system
7181 overwrites or changes the ``deploy`` directory.
7182
7183When telling DNF where to look for the package databases, you must
7184declare individual locations per architecture or a single location used
7185for all architectures. You cannot do both:
7186
7187- *Create an Explicit List of Architectures:* Define individual base
7188 URLs to identify where each package database is located:
7189
7190 .. code-block:: none
7191
7192 [oe-packages]
7193 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
7194
7195 This example
7196 informs DNF about individual package databases for all three
7197 architectures.
7198
7199- *Create a Single (Full) Package Index:* Define a single base URL that
7200 identifies where a full package database is located::
7201
7202 [oe-packages]
7203 baseurl=http://my.server/rpm
7204
7205 This example informs DNF about a single
7206 package database that contains all the package index information for
7207 all supported architectures.
7208
7209Once you have informed DNF where to find the package databases, you need
7210to fetch them:
7211
7212.. code-block:: none
7213
7214 # dnf makecache
7215
7216DNF is now able to find, install, and
7217upgrade packages from the specified repository or repositories.
7218
7219.. note::
7220
7221 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
7222 additional information.
7223
7224Using IPK
7225^^^^^^^^^
7226
7227The ``opkg`` application performs runtime package management of IPK
7228packages. You must perform an initial setup for ``opkg`` on the target
7229machine if the
7230:term:`PACKAGE_FEED_ARCHS`,
7231:term:`PACKAGE_FEED_BASE_PATHS`,
7232and
7233:term:`PACKAGE_FEED_URIS`
7234variables have not been set or the target image was built before the
7235variables were set.
7236
7237The ``opkg`` application uses configuration files to find available
7238package databases. Thus, you need to create a configuration file inside
7239the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
7240you want to use.
7241
7242As an example, suppose you are serving packages from a ``ipk/``
7243directory containing the ``i586``, ``all``, and ``qemux86`` databases
7244through an HTTP server named ``my.server``. On the target, create a
7245configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7246directory containing the following:
7247
7248.. code-block:: none
7249
7250 src/gz all http://my.server/ipk/all
7251 src/gz i586 http://my.server/ipk/i586
7252 src/gz qemux86 http://my.server/ipk/qemux86
7253
7254Next, instruct ``opkg`` to fetch the
7255repository information:
7256
7257.. code-block:: none
7258
7259 # opkg update
7260
7261The ``opkg`` application is now able to find, install, and upgrade packages
7262from the specified repository.
7263
7264Using DEB
7265^^^^^^^^^
7266
7267The ``apt`` application performs runtime package management of DEB
7268packages. This application uses a source list file to find available
7269package databases. You must perform an initial setup for ``apt`` on the
7270target machine if the
7271:term:`PACKAGE_FEED_ARCHS`,
7272:term:`PACKAGE_FEED_BASE_PATHS`,
7273and
7274:term:`PACKAGE_FEED_URIS`
7275variables have not been set or the target image was built before the
7276variables were set.
7277
7278To inform ``apt`` of the repository you want to use, you might create a
7279list file (e.g. ``my_repo.list``) inside the
7280``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7281serving packages from a ``deb/`` directory containing the ``i586``,
7282``all``, and ``qemux86`` databases through an HTTP server named
7283``my.server``. The list file should contain:
7284
7285.. code-block:: none
7286
7287 deb http://my.server/deb/all ./
7288 deb http://my.server/deb/i586 ./
7289 deb http://my.server/deb/qemux86 ./
7290
7291Next, instruct the ``apt`` application
7292to fetch the repository information:
7293
7294.. code-block:: none
7295
7296 $ sudo apt update
7297
7298After this step,
7299``apt`` is able to find, install, and upgrade packages from the
7300specified repository.
7301
7302Generating and Using Signed Packages
7303------------------------------------
7304
7305In order to add security to RPM packages used during a build, you can
7306take steps to securely sign them. Once a signature is verified, the
7307OpenEmbedded build system can use the package in the build. If security
7308fails for a signed package, the build system stops the build.
7309
7310This section describes how to sign RPM packages during a build and how
7311to use signed package feeds (repositories) when doing a build.
7312
7313Signing RPM Packages
7314~~~~~~~~~~~~~~~~~~~~
7315
7316To enable signing RPM packages, you must set up the following
7317configurations in either your ``local.config`` or ``distro.config``
7318file::
7319
7320 # Inherit sign_rpm.bbclass to enable signing functionality
7321 INHERIT += " sign_rpm"
7322 # Define the GPG key that will be used for signing.
7323 RPM_GPG_NAME = "key_name"
7324 # Provide passphrase for the key
7325 RPM_GPG_PASSPHRASE = "passphrase"
7326
7327.. note::
7328
7329 Be sure to supply appropriate values for both `key_name` and
7330 `passphrase`.
7331
7332Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
7333the previous example, two optional variables related to signing are available:
7334
7335- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7336 when the package is signed.
7337
7338- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7339 package is signed.
7340
7341Processing Package Feeds
7342~~~~~~~~~~~~~~~~~~~~~~~~
7343
7344In addition to being able to sign RPM packages, you can also enable
7345signed package feeds for IPK and RPM packages.
7346
7347The steps you need to take to enable signed package feed use are similar
7348to the steps used to sign RPM packages. You must define the following in
7349your ``local.config`` or ``distro.config`` file::
7350
7351 INHERIT += "sign_package_feed"
7352 PACKAGE_FEED_GPG_NAME = "key_name"
7353 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7354
7355For signed package feeds, the passphrase must be specified in a separate file,
7356which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7357variable. Regarding security, keeping a plain text passphrase out of the
7358configuration is more secure.
7359
7360Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7361``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
7362related to signed package feeds are available:
7363
7364- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7365 when the package is signed.
7366
7367- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7368 package is signed.
7369
7370- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7371 signature. This variable applies only to RPM and IPK package feeds.
7372 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7373 "ASC", which is the default and specifies ascii armored, and "BIN",
7374 which specifies binary.
7375
7376Testing Packages With ptest
7377---------------------------
7378
7379A Package Test (ptest) runs tests against packages built by the
7380OpenEmbedded build system on the target machine. A ptest contains at
7381least two items: the actual test, and a shell script (``run-ptest``)
7382that starts the test. The shell script that starts the test must not
7383contain the actual test --- the script only starts the test. On the other
7384hand, the test can be anything from a simple shell script that runs a
7385binary and checks the output to an elaborate system of test binaries and
7386data files.
7387
7388The test generates output in the format used by Automake::
7389
7390 result: testname
7391
7392where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7393the testname can be any identifying string.
7394
7395For a list of Yocto Project recipes that are already enabled with ptest,
7396see the :yocto_wiki:`Ptest </Ptest>` wiki page.
7397
7398.. note::
7399
7400 A recipe is "ptest-enabled" if it inherits the
7401 :ref:`ptest <ref-classes-ptest>` class.
7402
7403Adding ptest to Your Build
7404~~~~~~~~~~~~~~~~~~~~~~~~~~
7405
7406To add package testing to your build, add the :term:`DISTRO_FEATURES` and
7407:term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which
7408is found in the :term:`Build Directory`::
7409
7410 DISTRO_FEATURES:append = " ptest"
7411 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7412
7413Once your build is complete, the ptest files are installed into the
7414``/usr/lib/package/ptest`` directory within the image, where ``package``
7415is the name of the package.
7416
7417Running ptest
7418~~~~~~~~~~~~~
7419
7420The ``ptest-runner`` package installs a shell script that loops through
7421all installed ptest test suites and runs them in sequence. Consequently,
7422you might want to add this package to your image.
7423
7424Getting Your Package Ready
7425~~~~~~~~~~~~~~~~~~~~~~~~~~
7426
7427In order to enable a recipe to run installed ptests on target hardware,
7428you need to prepare the recipes that build the packages you want to
7429test. Here is what you have to do for each recipe:
7430
7431- *Be sure the recipe inherits the* :ref:`ptest <ref-classes-ptest>` *class:*
7432 Include the following line in each recipe::
7433
7434 inherit ptest
7435
7436- *Create run-ptest:* This script starts your test. Locate the
7437 script where you will refer to it using
7438 :term:`SRC_URI`. Here is an
7439 example that starts a test for ``dbus``::
7440
7441 #!/bin/sh
7442 cd test
7443 make -k runtest-TESTS
7444
7445- *Ensure dependencies are met:* If the test adds build or runtime
7446 dependencies that normally do not exist for the package (such as
7447 requiring "make" to run the test suite), use the
7448 :term:`DEPENDS` and
7449 :term:`RDEPENDS` variables in
7450 your recipe in order for the package to meet the dependencies. Here
7451 is an example where the package has a runtime dependency on "make"::
7452
7453 RDEPENDS:${PN}-ptest += "make"
7454
7455- *Add a function to build the test suite:* Not many packages support
7456 cross-compilation of their test suites. Consequently, you usually
7457 need to add a cross-compilation function to the package.
7458
7459 Many packages based on Automake compile and run the test suite by
7460 using a single command such as ``make check``. However, the host
7461 ``make check`` builds and runs on the same computer, while
7462 cross-compiling requires that the package is built on the host but
7463 executed for the target architecture (though often, as in the case
7464 for ptest, the execution occurs on the host). The built version of
7465 Automake that ships with the Yocto Project includes a patch that
7466 separates building and execution. Consequently, packages that use the
7467 unaltered, patched version of ``make check`` automatically
7468 cross-compiles.
7469
7470 Regardless, you still must add a ``do_compile_ptest`` function to
7471 build the test suite. Add a function similar to the following to your
7472 recipe::
7473
7474 do_compile_ptest() {
7475 oe_runmake buildtest-TESTS
7476 }
7477
7478- *Ensure special configurations are set:* If the package requires
7479 special configurations prior to compiling the test code, you must
7480 insert a ``do_configure_ptest`` function into the recipe.
7481
7482- *Install the test suite:* The :ref:`ptest <ref-classes-ptest>` class
7483 automatically copies the file ``run-ptest`` to the target and then runs make
7484 ``install-ptest`` to run the tests. If this is not enough, you need
7485 to create a ``do_install_ptest`` function and make sure it gets
7486 called after the "make install-ptest" completes.
7487
7488Creating Node Package Manager (NPM) Packages
7489--------------------------------------------
7490
7491:wikipedia:`NPM <Npm_(software)>` is a package
7492manager for the JavaScript programming language. The Yocto Project
7493supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
7494use this fetcher in combination with
7495:doc:`devtool </ref-manual/devtool-reference>` to create
7496recipes that produce NPM packages.
7497
7498There are two workflows that allow you to create NPM packages using
7499``devtool``: the NPM registry modules method and the NPM project code
7500method.
7501
7502.. note::
7503
7504 While it is possible to create NPM recipes manually, using
7505 ``devtool`` is far simpler.
7506
7507Additionally, some requirements and caveats exist.
7508
7509Requirements and Caveats
7510~~~~~~~~~~~~~~~~~~~~~~~~
7511
7512You need to be aware of the following before using ``devtool`` to create
7513NPM packages:
7514
7515- Of the two methods that you can use ``devtool`` to create NPM
7516 packages, the registry approach is slightly simpler. However, you
7517 might consider the project approach because you do not have to
7518 publish your module in the `NPM registry <https://docs.npmjs.com/misc/registry>`__,
7519 which is NPM's public registry.
7520
7521- Be familiar with
7522 :doc:`devtool </ref-manual/devtool-reference>`.
7523
7524- The NPM host tools need the native ``nodejs-npm`` package, which is
7525 part of the OpenEmbedded environment. You need to get the package by
7526 cloning the :oe_git:`meta-openembedded </meta-openembedded>`
7527 repository. Be sure to add the path to your local copy
7528 to your ``bblayers.conf`` file.
7529
7530- ``devtool`` cannot detect native libraries in module dependencies.
7531 Consequently, you must manually add packages to your recipe.
7532
7533- While deploying NPM packages, ``devtool`` cannot determine which
7534 dependent packages are missing on the target (e.g. the node runtime
7535 ``nodejs``). Consequently, you need to find out what files are
7536 missing and be sure they are on the target.
7537
7538- Although you might not need NPM to run your node package, it is
7539 useful to have NPM on your target. The NPM package name is
7540 ``nodejs-npm``.
7541
7542Using the Registry Modules Method
7543~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7544
7545This section presents an example that uses the ``cute-files`` module,
7546which is a file browser web application.
7547
7548.. note::
7549
7550 You must know the ``cute-files`` module version.
7551
7552The first thing you need to do is use ``devtool`` and the NPM fetcher to
7553create the recipe::
7554
7555 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7556
7557The
7558``devtool add`` command runs ``recipetool create`` and uses the same
7559fetch URI to download each dependency and capture license details where
7560possible. The result is a generated recipe.
7561
7562After running for quite a long time, in particular building the
7563``nodejs-native`` package, the command should end as follows::
7564
7565 INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional
7566
7567The recipe file is fairly simple and contains every license that
7568``recipetool`` finds and includes the licenses in the recipe's
7569:term:`LIC_FILES_CHKSUM`
7570variables. You need to examine the variables and look for those with
7571"unknown" in the :term:`LICENSE`
7572field. You need to track down the license information for "unknown"
7573modules and manually add the information to the recipe.
7574
7575``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7576files capture the version of all dependent modules. Many packages do not
7577provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it
7578runs.
7579
7580.. note::
7581
7582 A package is created for each sub-module. This policy is the only
7583 practical way to have the licenses for all of the dependencies
7584 represented in the license manifest of the image.
7585
7586The ``devtool edit-recipe`` command lets you take a look at the recipe::
7587
7588 $ devtool edit-recipe cute-files
7589 # Recipe created by recipetool
7590 # This is the basis of a recipe and may need further editing in order to be fully functional.
7591 # (Feel free to remove these comments when editing.)
7592
7593 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7594 # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
7595 # your responsibility to verify that the values are complete and correct.
7596 #
7597 # NOTE: multiple licenses have been detected; they have been separated with &
7598 # in the LICENSE value for now since it is a reasonable assumption that all
7599 # of the licenses apply. If instead there is a choice between the multiple
7600 # licenses then you should change the value to separate the licenses with |
7601 # instead of &. If there is any doubt, check the accompanying documentation
7602 # to determine which situation is applicable.
7603
7604 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7605 LICENSE = "BSD-3-Clause & ISC & MIT"
7606 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
7607 file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \
7608 file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \
7609 ...
7610 file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a"
7611
7612 SRC_URI = " \
7613 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7614 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7615 "
7616
7617 S = "${WORKDIR}/npm"
7618
7619 inherit npm
7620
7621 LICENSE:${PN} = "MIT"
7622 LICENSE:${PN}-accepts = "MIT"
7623 LICENSE:${PN}-array-flatten = "MIT"
7624 ...
7625 LICENSE:${PN}-vary = "MIT"
7626
7627Here are three key points in the previous example:
7628
7629- :term:`SRC_URI` uses the NPM
7630 scheme so that the NPM fetcher is used.
7631
7632- ``recipetool`` collects all the license information. If a
7633 sub-module's license is unavailable, the sub-module's name appears in
7634 the comments.
7635
7636- The ``inherit npm`` statement causes the
7637 :ref:`npm <ref-classes-npm>` class to package
7638 up all the modules.
7639
7640You can run the following command to build the ``cute-files`` package::
7641
7642 $ devtool build cute-files
7643
7644Remember that ``nodejs`` must be installed on
7645the target before your package.
7646
7647Assuming 192.168.7.2 for the target's IP address, use the following
7648command to deploy your package::
7649
7650 $ devtool deploy-target -s cute-files root@192.168.7.2
7651
7652Once the package is installed on the target, you can
7653test the application to show the contents of any directory::
7654
7655 $ cd /usr/lib/node_modules/cute-files
7656 $ cute-files
7657
7658On a browser,
7659go to ``http://192.168.7.2:3000`` and you see the following:
7660
7661.. image:: figures/cute-files-npm-example.png
7662 :width: 100%
7663
7664You can find the recipe in ``workspace/recipes/cute-files``. You can use
7665the recipe in any layer you choose.
7666
7667Using the NPM Projects Code Method
7668~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7669
7670Although it is useful to package modules already in the NPM registry,
7671adding ``node.js`` projects under development is a more common developer
7672use case.
7673
7674This section covers the NPM projects code method, which is very similar
7675to the "registry" approach described in the previous section. In the NPM
7676projects method, you provide ``devtool`` with an URL that points to the
7677source files.
7678
7679Replicating the same example, (i.e. ``cute-files``) use the following
7680command::
7681
7682 $ devtool add https://github.com/martinaglv/cute-files.git
7683
7684The recipe this command generates is very similar to the recipe created in
7685the previous section. However, the :term:`SRC_URI` looks like the following::
7686
7687 SRC_URI = " \
7688 git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \
7689 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7690 "
7691
7692In this example,
7693the main module is taken from the Git repository and dependencies are
7694taken from the NPM registry. Other than those differences, the recipe is
7695basically the same between the two methods. You can build and deploy the
7696package exactly as described in the previous section that uses the
7697registry modules method.
7698
7699Adding custom metadata to packages
7700----------------------------------
7701
7702The variable
7703:term:`PACKAGE_ADD_METADATA`
7704can be used to add additional metadata to packages. This is reflected in
7705the package control/spec file. To take the ipk format for example, the
7706CONTROL file stored inside would contain the additional metadata as
7707additional lines.
7708
7709The variable can be used in multiple ways, including using suffixes to
7710set it for a specific package type and/or package. Note that the order
7711of precedence is the same as this list:
7712
7713- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
7714
7715- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7716
7717- ``PACKAGE_ADD_METADATA:<PN>``
7718
7719- :term:`PACKAGE_ADD_METADATA`
7720
7721`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
7722package type:
7723
7724- IPK for .ipk packages
7725
7726- DEB for .deb packages
7727
7728- RPM for .rpm packages
7729
7730`<PN>` is a parameter and expected to be a package name.
7731
7732The variable can contain multiple [one-line] metadata fields separated
7733by the literal sequence '\\n'. The separator can be redefined using the
7734variable flag ``separator``.
7735
7736Here is an example that adds two custom fields for ipk
7737packages::
7738
7739 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
7740
7741Efficiently Fetching Source Files During a Build
7742================================================
7743
7744The OpenEmbedded build system works with source files located through
7745the :term:`SRC_URI` variable. When
7746you build something using BitBake, a big part of the operation is
7747locating and downloading all the source tarballs. For images,
7748downloading all the source for various packages can take a significant
7749amount of time.
7750
7751This section shows you how you can use mirrors to speed up fetching
7752source files and how you can pre-fetch files all of which leads to more
7753efficient use of resources and time.
7754
7755Setting up Effective Mirrors
7756----------------------------
7757
7758A good deal that goes into a Yocto Project build is simply downloading
7759all of the source tarballs. Maybe you have been working with another
7760build system for which you have built up a
7761sizable directory of source tarballs. Or, perhaps someone else has such
7762a directory for which you have read access. If so, you can save time by
7763adding statements to your configuration file so that the build process
7764checks local directories first for existing tarballs before checking the
7765Internet.
7766
7767Here is an efficient way to set it up in your ``local.conf`` file::
7768
7769 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7770 INHERIT += "own-mirrors"
7771 BB_GENERATE_MIRROR_TARBALLS = "1"
7772 # BB_NO_NETWORK = "1"
7773
7774In the previous example, the
7775:term:`BB_GENERATE_MIRROR_TARBALLS`
7776variable causes the OpenEmbedded build system to generate tarballs of
7777the Git repositories and store them in the
7778:term:`DL_DIR` directory. Due to
7779performance reasons, generating and storing these tarballs is not the
7780build system's default behavior.
7781
7782You can also use the
7783:term:`PREMIRRORS` variable. For
7784an example, see the variable's glossary entry in the Yocto Project
7785Reference Manual.
7786
7787Getting Source Files and Suppressing the Build
7788----------------------------------------------
7789
7790Another technique you can use to ready yourself for a successive string
7791of build operations, is to pre-fetch all the source files without
7792actually starting a build. This technique lets you work through any
7793download issues and ultimately gathers all the source files into your
7794download directory :ref:`structure-build-downloads`,
7795which is located with :term:`DL_DIR`.
7796
7797Use the following BitBake command form to fetch all the necessary
7798sources without starting the build::
7799
7800 $ bitbake target --runall=fetch
7801
7802This
7803variation of the BitBake command guarantees that you have all the
7804sources for that BitBake target should you disconnect from the Internet
7805and want to do the build later offline.
7806
7807Selecting an Initialization Manager
7808===================================
7809
7810By default, the Yocto Project uses SysVinit as the initialization
7811manager. However, there is also support for systemd, which is a full
7812replacement for init with parallel starting of services, reduced shell
7813overhead and other features that are used by many distributions.
7814
7815Within the system, SysVinit treats system components as services. These
7816services are maintained as shell scripts stored in the ``/etc/init.d/``
7817directory. Services organize into different run levels. This
7818organization is maintained by putting links to the services in the
7819``/etc/rcN.d/`` directories, where `N/` is one of the following options:
7820"S", "0", "1", "2", "3", "4", "5", or "6".
7821
7822.. note::
7823
7824 Each runlevel has a dependency on the previous runlevel. This
7825 dependency allows the services to work properly.
7826
7827In comparison, systemd treats components as units. Using units is a
7828broader concept as compared to using a service. A unit includes several
7829different types of entities. Service is one of the types of entities.
7830The runlevel concept in SysVinit corresponds to the concept of a target
7831in systemd, where target is also a type of supported unit.
7832
7833In a SysVinit-based system, services load sequentially (i.e. one by one)
7834during init and parallelization is not supported. With systemd, services
7835start in parallel. Needless to say, the method can have an impact on
7836system startup performance.
7837
7838If you want to use SysVinit, you do not have to do anything. But, if you
7839want to use systemd, you must take some steps as described in the
7840following sections.
7841
7842Using systemd Exclusively
7843-------------------------
7844
7845Set these variables in your distribution configuration file as follows::
7846
7847 DISTRO_FEATURES:append = " systemd"
7848 VIRTUAL-RUNTIME_init_manager = "systemd"
7849
7850You can also prevent the SysVinit distribution feature from
7851being automatically enabled as follows::
7852
7853 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7854
7855Doing so removes any
7856redundant SysVinit scripts.
7857
7858To remove initscripts from your image altogether, set this variable
7859also::
7860
7861 VIRTUAL-RUNTIME_initscripts = ""
7862
7863For information on the backfill variable, see
7864:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7865
7866Using systemd for the Main Image and Using SysVinit for the Rescue Image
7867------------------------------------------------------------------------
7868
7869Set these variables in your distribution configuration file as follows::
7870
7871 DISTRO_FEATURES:append = " systemd"
7872 VIRTUAL-RUNTIME_init_manager = "systemd"
7873
7874Doing so causes your main image to use the
7875``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7876image cannot use this package group. However, it can install SysVinit
7877and the appropriate packages will have support for both systemd and
7878SysVinit.
7879
7880Using systemd-journald without a traditional syslog daemon
7881----------------------------------------------------------
7882
7883Counter-intuitively, ``systemd-journald`` is not a syslog runtime or provider,
7884and the proper way to use systemd-journald as your sole logging mechanism is to
7885effectively disable syslog entirely by setting these variables in your distribution
7886configuration file::
7887
7888 VIRTUAL-RUNTIME_syslog = ""
7889 VIRTUAL-RUNTIME_base-utils-syslog = ""
7890
7891Doing so will prevent ``rsyslog`` / ``busybox-syslog`` from being pulled in by
7892default, leaving only ``journald``.
7893
7894Selecting a Device Manager
7895==========================
7896
7897The Yocto Project provides multiple ways to manage the device manager
7898(``/dev``):
7899
7900- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev``
7901 directory is persistent and the required device nodes are created
7902 during the build.
7903
7904- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7905 directory is provided by the kernel as an in-memory file system and
7906 is automatically populated by the kernel at runtime. Additional
7907 configuration of device nodes is done in user space by a device
7908 manager like ``udev`` or ``busybox-mdev``.
7909
7910Using Persistent and Pre-Populated ``/dev``
7911--------------------------------------------
7912
7913To use the static method for device population, you need to set the
7914:term:`USE_DEVFS` variable to "0"
7915as follows::
7916
7917 USE_DEVFS = "0"
7918
7919The content of the resulting ``/dev`` directory is defined in a Device
7920Table file. The
7921:term:`IMAGE_DEVICE_TABLES`
7922variable defines the Device Table to use and should be set in the
7923machine or distro configuration file. Alternatively, you can set this
7924variable in your ``local.conf`` configuration file.
7925
7926If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
7927``device_table-minimal.txt`` is used::
7928
7929 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7930
7931The population is handled by the ``makedevs`` utility during image
7932creation:
7933
7934Using ``devtmpfs`` and a Device Manager
7935---------------------------------------
7936
7937To use the dynamic method for device population, you need to use (or be
7938sure to set) the :term:`USE_DEVFS`
7939variable to "1", which is the default::
7940
7941 USE_DEVFS = "1"
7942
7943With this
7944setting, the resulting ``/dev`` directory is populated by the kernel
7945using ``devtmpfs``. Make sure the corresponding kernel configuration
7946variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
7947kernel.
7948
7949All devices created by ``devtmpfs`` will be owned by ``root`` and have
7950permissions ``0600``.
7951
7952To have more control over the device nodes, you can use a device manager
7953like ``udev`` or ``busybox-mdev``. You choose the device manager by
7954defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
7955distro configuration file. Alternatively, you can set this variable in
7956your ``local.conf`` configuration file::
7957
7958 VIRTUAL-RUNTIME_dev_manager = "udev"
7959
7960 # Some alternative values
7961 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
7962 # VIRTUAL-RUNTIME_dev_manager = "systemd"
7963
7964Using an External SCM
7965=====================
7966
7967If you're working on a recipe that pulls from an external Source Code
7968Manager (SCM), it is possible to have the OpenEmbedded build system
7969notice new recipe changes added to the SCM and then build the resulting
7970packages that depend on the new recipes by using the latest versions.
7971This only works for SCMs from which it is possible to get a sensible
7972revision number for changes. Currently, you can do this with Apache
7973Subversion (SVN), Git, and Bazaar (BZR) repositories.
7974
7975To enable this behavior, the :term:`PV` of
7976the recipe needs to reference
7977:term:`SRCPV`. Here is an example::
7978
7979 PV = "1.2.3+git${SRCPV}"
7980
7981Then, you can add the following to your
7982``local.conf``::
7983
7984 SRCREV:pn-PN = "${AUTOREV}"
7985
7986:term:`PN` is the name of the recipe for
7987which you want to enable automatic source revision updating.
7988
7989If you do not want to update your local configuration file, you can add
7990the following directly to the recipe to finish enabling the feature::
7991
7992 SRCREV = "${AUTOREV}"
7993
7994The Yocto Project provides a distribution named ``poky-bleeding``, whose
7995configuration file contains the line::
7996
7997 require conf/distro/include/poky-floating-revisions.inc
7998
7999This line pulls in the
8000listed include file that contains numerous lines of exactly that form::
8001
8002 #SRCREV:pn-opkg-native ?= "${AUTOREV}"
8003 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}"
8004 #SRCREV:pn-opkg ?= "${AUTOREV}"
8005 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}"
8006 #SRCREV:pn-opkg-utils ?= "${AUTOREV}"
8007 SRCREV:pn-gconf-dbus ?= "${AUTOREV}"
8008 SRCREV:pn-matchbox-common ?= "${AUTOREV}"
8009 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}"
8010 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}"
8011 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}"
8012 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}"
8013 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}"
8014 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}"
8015 SRCREV:pn-matchbox-wm ?= "${AUTOREV}"
8016 SRCREV:pn-settings-daemon ?= "${AUTOREV}"
8017 SRCREV:pn-screenshot ?= "${AUTOREV}"
8018 . . .
8019
8020These lines allow you to
8021experiment with building a distribution that tracks the latest
8022development source for numerous packages.
8023
8024.. note::
8025
8026 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
8027 this in mind if you use it.
8028
8029Creating a Read-Only Root Filesystem
8030====================================
8031
8032Suppose, for security reasons, you need to disable your target device's
8033root filesystem's write permissions (i.e. you need a read-only root
8034filesystem). Or, perhaps you are running the device's operating system
8035from a read-only storage device. For either case, you can customize your
8036image for that behavior.
8037
8038.. note::
8039
8040 Supporting a read-only root filesystem requires that the system and
8041 applications do not try to write to the root filesystem. You must
8042 configure all parts of the target system to write elsewhere, or to
8043 gracefully fail in the event of attempting to write to the root
8044 filesystem.
8045
8046Creating the Root Filesystem
8047----------------------------
8048
8049To create the read-only root filesystem, simply add the
8050"read-only-rootfs" feature to your image, normally in one of two ways.
8051The first way is to add the "read-only-rootfs" image feature in the
8052image's recipe file via the :term:`IMAGE_FEATURES` variable::
8053
8054 IMAGE_FEATURES += "read-only-rootfs"
8055
8056As an alternative, you can add the same feature
8057from within your :term:`Build Directory`'s ``local.conf`` file with the
8058associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
8059
8060 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
8061
8062For more information on how to use these variables, see the
8063":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
8064section. For information on the variables, see
8065:term:`IMAGE_FEATURES` and
8066:term:`EXTRA_IMAGE_FEATURES`.
8067
8068Post-Installation Scripts and Read-Only Root Filesystem
8069-------------------------------------------------------
8070
8071It is very important that you make sure all post-Installation
8072(``pkg_postinst``) scripts for packages that are installed into the
8073image can be run at the time when the root filesystem is created during
8074the build on the host system. These scripts cannot attempt to run during
8075the first boot on the target device. With the "read-only-rootfs" feature
8076enabled, the build system makes sure that all post-installation scripts
8077succeed at file system creation time. If any of these scripts
8078still need to be run after the root filesystem is created, the build
8079immediately fails. These build-time checks ensure that the build fails
8080rather than the target device fails later during its initial boot
8081operation.
8082
8083Most of the common post-installation scripts generated by the build
8084system for the out-of-the-box Yocto Project are engineered so that they
8085can run during root filesystem creation (e.g. post-installation scripts
8086for caching fonts). However, if you create and add custom scripts, you
8087need to be sure they can be run during this file system creation.
8088
8089Here are some common problems that prevent post-installation scripts
8090from running during root filesystem creation:
8091
8092- *Not using $D in front of absolute paths:* The build system defines
8093 ``$``\ :term:`D` when the root
8094 filesystem is created. Furthermore, ``$D`` is blank when the script
8095 is run on the target device. This implies two purposes for ``$D``:
8096 ensuring paths are valid in both the host and target environments,
8097 and checking to determine which environment is being used as a method
8098 for taking appropriate actions.
8099
8100- *Attempting to run processes that are specific to or dependent on the
8101 target architecture:* You can work around these attempts by using
8102 native tools, which run on the host system, to accomplish the same
8103 tasks, or by alternatively running the processes under QEMU, which
8104 has the ``qemu_run_binary`` function. For more information, see the
8105 :ref:`qemu <ref-classes-qemu>` class.
8106
8107Areas With Write Access
8108-----------------------
8109
8110With the "read-only-rootfs" feature enabled, any attempt by the target
8111to write to the root filesystem at runtime fails. Consequently, you must
8112make sure that you configure processes and applications that attempt
8113these types of writes do so to directories with write access (e.g.
8114``/tmp`` or ``/var/run``).
8115
8116Maintaining Build Output Quality
8117================================
8118
8119Many factors can influence the quality of a build. For example, if you
8120upgrade a recipe to use a new version of an upstream software package or
8121you experiment with some new configuration options, subtle changes can
8122occur that you might not detect until later. Consider the case where
8123your recipe is using a newer version of an upstream package. In this
8124case, a new version of a piece of software might introduce an optional
8125dependency on another library, which is auto-detected. If that library
8126has already been built when the software is building, the software will
8127link to the built library and that library will be pulled into your
8128image along with the new software even if you did not want the library.
8129
8130The :ref:`buildhistory <ref-classes-buildhistory>`
8131class helps you maintain the quality of your build output. You
8132can use the class to highlight unexpected and possibly unwanted changes
8133in the build output. When you enable build history, it records
8134information about the contents of each package and image and then
8135commits that information to a local Git repository where you can examine
8136the information.
8137
8138The remainder of this section describes the following:
8139
8140- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>`
8141
8142- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>`
8143
8144- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>`
8145
8146- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>`
8147
8148Enabling and Disabling Build History
8149------------------------------------
8150
8151Build history is disabled by default. To enable it, add the following
8152:term:`INHERIT` statement and set the :term:`BUILDHISTORY_COMMIT` variable to
8153"1" at the end of your ``conf/local.conf`` file found in the
8154:term:`Build Directory`::
8155
8156 INHERIT += "buildhistory"
8157 BUILDHISTORY_COMMIT = "1"
8158
8159Enabling build history as
8160previously described causes the OpenEmbedded build system to collect
8161build output information and commit it as a single commit to a local
8162:ref:`overview-manual/development-environment:git` repository.
8163
8164.. note::
8165
8166 Enabling build history increases your build times slightly,
8167 particularly for images, and increases the amount of disk space used
8168 during the build.
8169
8170You can disable build history by removing the previous statements from
8171your ``conf/local.conf`` file.
8172
8173Understanding What the Build History Contains
8174---------------------------------------------
8175
8176Build history information is kept in ``${``\ :term:`TOPDIR`\ ``}/buildhistory``
8177in the :term:`Build Directory` as defined by the :term:`BUILDHISTORY_DIR`
8178variable. Here is an example abbreviated listing:
8179
8180.. image:: figures/buildhistory.png
8181 :align: center
8182 :width: 50%
8183
8184At the top level, there is a ``metadata-revs`` file that lists the
8185revisions of the repositories for the enabled layers when the build was
8186produced. The rest of the data splits into separate ``packages``,
8187``images`` and ``sdk`` directories, the contents of which are described
8188as follows.
8189
8190Build History Package Information
8191~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8192
8193The history for each package contains a text file that has name-value
8194pairs with information about the package. For example,
8195``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
8196contains the following:
8197
8198.. code-block:: none
8199
8200 PV = 1.22.1
8201 PR = r32
8202 RPROVIDES =
8203 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
8204 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
8205 PKGSIZE = 540168
8206 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
8207 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
8208 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
8209 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
8210 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
8211 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
8212 /etc/busybox.links.nosuid /etc/busybox.links.suid
8213
8214Most of these
8215name-value pairs correspond to variables used to produce the package.
8216The exceptions are ``FILELIST``, which is the actual list of files in
8217the package, and ``PKGSIZE``, which is the total size of files in the
8218package in bytes.
8219
8220There is also a file that corresponds to the recipe from which the package
8221came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
8222
8223.. code-block:: none
8224
8225 PV = 1.22.1
8226 PR = r32
8227 DEPENDS = initscripts kern-tools-native update-rc.d-native \
8228 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
8229 virtual/libc virtual/update-alternatives
8230 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
8231 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
8232 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
8233
8234Finally, for those recipes fetched from a version control system (e.g.,
8235Git), there is a file that lists source revisions that are specified in
8236the recipe and the actual revisions used during the build. Listed
8237and actual revisions might differ when
8238:term:`SRCREV` is set to
8239${:term:`AUTOREV`}. Here is an
8240example assuming
8241``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
8242
8243 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8244 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8245 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
8246 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
8247
8248You can use the
8249``buildhistory-collect-srcrevs`` command with the ``-a`` option to
8250collect the stored :term:`SRCREV` values from build history and report them
8251in a format suitable for use in global configuration (e.g.,
8252``local.conf`` or a distro include file) to override floating
8253:term:`AUTOREV` values to a fixed set of revisions. Here is some example
8254output from this command::
8255
8256 $ buildhistory-collect-srcrevs -a
8257 # all-poky-linux
8258 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8"
8259 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac"
8260 # core2-64-poky-linux
8261 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66"
8262 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8"
8263 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0"
8264 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0"
8265 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1"
8266 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c"
8267 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7"
8268 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee"
8269 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1"
8270 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100"
8271 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3"
8272 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf"
8273 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced"
8274 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8"
8275 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f"
8276 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa"
8277 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70"
8278 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
8279 # qemux86_64-poky-linux
8280 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c"
8281 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0"
8282 # x86_64-linux
8283 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66"
8284 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c"
8285 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87"
8286 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57"
8287 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb"
8288 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b"
8289 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
8290
8291.. note::
8292
8293 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
8294
8295 - By default, only values where the :term:`SRCREV` was not hardcoded
8296 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
8297 option to see all :term:`SRCREV` values.
8298
8299 - The output statements might not have any effect if overrides are
8300 applied elsewhere in the build system configuration. Use the
8301 ``-f`` option to add the ``forcevariable`` override to each output
8302 line if you need to work around this restriction.
8303
8304 - The script does apply special handling when building for multiple
8305 machines. However, the script does place a comment before each set
8306 of values that specifies which triplet to which they belong as
8307 previously shown (e.g., ``i586-poky-linux``).
8308
8309Build History Image Information
8310~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8311
8312The files produced for each image are as follows:
8313
8314- ``image-files:`` A directory containing selected files from the root
8315 filesystem. The files are defined by
8316 :term:`BUILDHISTORY_IMAGE_FILES`.
8317
8318- ``build-id.txt:`` Human-readable information about the build
8319 configuration and metadata source revisions. This file contains the
8320 full build header as printed by BitBake.
8321
8322- ``*.dot:`` Dependency graphs for the image that are compatible with
8323 ``graphviz``.
8324
8325- ``files-in-image.txt:`` A list of files in the image with
8326 permissions, owner, group, size, and symlink information.
8327
8328- ``image-info.txt:`` A text file containing name-value pairs with
8329 information about the image. See the following listing example for
8330 more information.
8331
8332- ``installed-package-names.txt:`` A list of installed packages by name
8333 only.
8334
8335- ``installed-package-sizes.txt:`` A list of installed packages ordered
8336 by size.
8337
8338- ``installed-packages.txt:`` A list of installed packages with full
8339 package filenames.
8340
8341.. note::
8342
8343 Installed package information is able to be gathered and produced
8344 even if package management is disabled for the final image.
8345
8346Here is an example of ``image-info.txt``:
8347
8348.. code-block:: none
8349
8350 DISTRO = poky
8351 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be
8352 USER_CLASSES = buildstats
8353 IMAGE_CLASSES = qemuboot qemuboot license_image
8354 IMAGE_FEATURES = debug-tweaks
8355 IMAGE_LINGUAS =
8356 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp
8357 BAD_RECOMMENDATIONS =
8358 NO_RECOMMENDATIONS =
8359 PACKAGE_EXCLUDE =
8360 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible;
8361 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8362 IMAGESIZE = 9265
8363
8364Other than ``IMAGESIZE``,
8365which is the total size of the files in the image in Kbytes, the
8366name-value pairs are variables that may have influenced the content of
8367the image. This information is often useful when you are trying to
8368determine why a change in the package or file listings has occurred.
8369
8370Using Build History to Gather Image Information Only
8371~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8372
8373As you can see, build history produces image information, including
8374dependency graphs, so you can see why something was pulled into the
8375image. If you are just interested in this information and not interested
8376in collecting specific package or SDK information, you can enable
8377writing only image information without any history by adding the
8378following to your ``conf/local.conf`` file found in the
8379:term:`Build Directory`::
8380
8381 INHERIT += "buildhistory"
8382 BUILDHISTORY_COMMIT = "0"
8383 BUILDHISTORY_FEATURES = "image"
8384
8385Here, you set the
8386:term:`BUILDHISTORY_FEATURES`
8387variable to use the image feature only.
8388
8389Build History SDK Information
8390~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8391
8392Build history collects similar information on the contents of SDKs (e.g.
8393``bitbake -c populate_sdk imagename``) as compared to information it
8394collects for images. Furthermore, this information differs depending on
8395whether an extensible or standard SDK is being produced.
8396
8397The following list shows the files produced for SDKs:
8398
8399- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8400 owner, group, size, and symlink information. This list includes both
8401 the host and target parts of the SDK.
8402
8403- ``sdk-info.txt:`` A text file containing name-value pairs with
8404 information about the SDK. See the following listing example for more
8405 information.
8406
8407- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8408 with information about task group sizes (e.g. :ref:`ref-tasks-populate_sysroot`
8409 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8410 only when an extensible SDK is created.
8411
8412- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8413 with information for the shared-state packages and sizes in the SDK.
8414 The ``sstate-package-sizes.txt`` file exists only when an extensible
8415 SDK is created.
8416
8417- ``sdk-files:`` A folder that contains copies of the files mentioned
8418 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8419 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8420 specific to the extensible SDK although you can set it differently if
8421 you would like to pull in specific files from the standard SDK.
8422
8423 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8424 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8425 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8426 copied into the ``sdk-files`` directory.
8427
8428- The following information appears under each of the ``host`` and
8429 ``target`` directories for the portions of the SDK that run on the
8430 host and on the target, respectively:
8431
8432 .. note::
8433
8434 The following files for the most part are empty when producing an
8435 extensible SDK because this type of SDK is not constructed from
8436 packages as is the standard SDK.
8437
8438 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8439 with ``graphviz``.
8440
8441 - ``installed-package-names.txt:`` A list of installed packages by
8442 name only.
8443
8444 - ``installed-package-sizes.txt:`` A list of installed packages
8445 ordered by size.
8446
8447 - ``installed-packages.txt:`` A list of installed packages with full
8448 package filenames.
8449
8450Here is an example of ``sdk-info.txt``:
8451
8452.. code-block:: none
8453
8454 DISTRO = poky
8455 DISTRO_VERSION = 1.3+snapshot-20130327
8456 SDK_NAME = poky-glibc-i686-arm
8457 SDK_VERSION = 1.3+snapshot
8458 SDKMACHINE =
8459 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8460 BAD_RECOMMENDATIONS =
8461 SDKSIZE = 352712
8462
8463Other than ``SDKSIZE``, which is
8464the total size of the files in the SDK in Kbytes, the name-value pairs
8465are variables that might have influenced the content of the SDK. This
8466information is often useful when you are trying to determine why a
8467change in the package or file listings has occurred.
8468
8469Examining Build History Information
8470~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8471
8472You can examine build history output from the command line or from a web
8473interface.
8474
8475To see any changes that have occurred (assuming you have
8476:term:`BUILDHISTORY_COMMIT` = "1"),
8477you can simply use any Git command that allows you to view the history
8478of a repository. Here is one method::
8479
8480 $ git log -p
8481
8482You need to realize,
8483however, that this method does show changes that are not significant
8484(e.g. a package's size changing by a few bytes).
8485
8486There is a command-line tool called ``buildhistory-diff``, though,
8487that queries the Git repository and prints just the differences that
8488might be significant in human-readable form. Here is an example::
8489
8490 $ poky/poky/scripts/buildhistory-diff . HEAD^
8491 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8492 /etc/anotherpkg.conf was added
8493 /sbin/anotherpkg was added
8494 * (installed-package-names.txt):
8495 * anotherpkg was added
8496 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8497 anotherpkg was added
8498 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8499 * PR changed from "r0" to "r1"
8500 * PV changed from "0.1.10" to "0.1.12"
8501 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8502 * PR changed from "r0" to "r1"
8503 * PV changed from "0.1.10" to "0.1.12"
8504
8505.. note::
8506
8507 The ``buildhistory-diff`` tool requires the ``GitPython``
8508 package. Be sure to install it using Pip3 as follows::
8509
8510 $ pip3 install GitPython --user
8511
8512
8513 Alternatively, you can install ``python3-git`` using the appropriate
8514 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``).
8515
8516To see changes to the build history using a web interface, follow the
8517instruction in the ``README`` file
8518:yocto_git:`here </buildhistory-web/>`.
8519
8520Here is a sample screenshot of the interface:
8521
8522.. image:: figures/buildhistory-web.png
8523 :width: 100%
8524
8525Performing Automated Runtime Testing
8526====================================
8527
8528The OpenEmbedded build system makes available a series of automated
8529tests for images to verify runtime functionality. You can run these
8530tests on either QEMU or actual target hardware. Tests are written in
8531Python making use of the ``unittest`` module, and the majority of them
8532run commands on the target system over SSH. This section describes how
8533you set up the environment to use these tests, run available tests, and
8534write and add your own tests.
8535
8536For information on the test and QA infrastructure available within the
8537Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
8538section in the Yocto Project Reference Manual.
8539
8540Enabling Tests
8541--------------
8542
8543Depending on whether you are planning to run tests using QEMU or on the
8544hardware, you have to take different steps to enable the tests. See the
8545following subsections for information on how to enable both types of
8546tests.
8547
8548Enabling Runtime Tests on QEMU
8549~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8550
8551In order to run tests, you need to do the following:
8552
8553- *Set up to avoid interaction with sudo for networking:* To
8554 accomplish this, you must do one of the following:
8555
8556 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8557 commands or just for ``runqemu-ifup``. You must provide the full
8558 path as that can change if you are using multiple clones of the
8559 source repository.
8560
8561 .. note::
8562
8563 On some distributions, you also need to comment out "Defaults
8564 requiretty" in ``/etc/sudoers``.
8565
8566 - Manually configure a tap interface for your system.
8567
8568 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8569 should generate a list of tap devices. This is the option
8570 typically chosen for Autobuilder-type environments.
8571
8572 .. note::
8573
8574 - Be sure to use an absolute path when calling this script
8575 with sudo.
8576
8577 - The package recipe ``qemu-helper-native`` is required to run
8578 this script. Build the package using the following command::
8579
8580 $ bitbake qemu-helper-native
8581
8582- *Set the DISPLAY variable:* You need to set this variable so that
8583 you have an X server available (e.g. start ``vncserver`` for a
8584 headless machine).
8585
8586- *Be sure your host's firewall accepts incoming connections from
8587 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8588 HTTP server on a random high number port, which is used to serve
8589 files to the target. The DNF module serves
8590 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8591 That means your host's firewall must accept incoming connections from
8592 192.168.7.0/24, which is the default IP range used for tap devices by
8593 ``runqemu``.
8594
8595- *Be sure your host has the correct packages installed:* Depending
8596 your host's distribution, you need to have the following packages
8597 installed:
8598
8599 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8600
8601 - openSUSE: ``sysstat`` and ``iproute2``
8602
8603 - Fedora: ``sysstat`` and ``iproute``
8604
8605 - CentOS: ``sysstat`` and ``iproute``
8606
8607Once you start running the tests, the following happens:
8608
86091. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8610
86112. The image is booted under QEMU using the standard ``runqemu`` script.
8612
86133. A default timeout of 500 seconds occurs to allow for the boot process
8614 to reach the login prompt. You can change the timeout period by
8615 setting
8616 :term:`TEST_QEMUBOOT_TIMEOUT`
8617 in the ``local.conf`` file.
8618
86194. Once the boot process is reached and the login prompt appears, the
8620 tests run. The full boot log is written to
8621 ``${WORKDIR}/testimage/qemu_boot_log``.
8622
86235. Each test module loads in the order found in :term:`TEST_SUITES`. You can
8624 find the full output of the commands run over SSH in
8625 ``${WORKDIR}/testimgage/ssh_target_log``.
8626
86276. If no failures occur, the task running the tests ends successfully.
8628 You can find the output from the ``unittest`` in the task log at
8629 ``${WORKDIR}/temp/log.do_testimage``.
8630
8631Enabling Runtime Tests on Hardware
8632~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8633
8634The OpenEmbedded build system can run tests on real hardware, and for
8635certain devices it can also deploy the image to be tested onto the
8636device beforehand.
8637
8638For automated deployment, a "controller image" is installed onto the
8639hardware once as part of setup. Then, each time tests are to be run, the
8640following occurs:
8641
86421. The controller image is booted into and used to write the image to be
8643 tested to a second partition.
8644
86452. The device is then rebooted using an external script that you need to
8646 provide.
8647
86483. The device boots into the image to be tested.
8649
8650When running tests (independent of whether the image has been deployed
8651automatically or not), the device is expected to be connected to a
8652network on a pre-determined IP address. You can either use static IP
8653addresses written into the image, or set the image to use DHCP and have
8654your DHCP server on the test network assign a known IP address based on
8655the MAC address of the device.
8656
8657In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
8658appropriate value. For QEMU, you do not have to change anything, the
8659default value is "qemu". For running tests on hardware, the following
8660options are available:
8661
8662- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8663 on a target system that is already running the image to be tested and
8664 is available on the network. You can use "simpleremote" in
8665 conjunction with either real hardware or an image running within a
8666 separately started QEMU or any other virtual machine manager.
8667
8668- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8669 an EFI-based machine with ``systemd-boot`` as bootloader and
8670 ``core-image-testmaster`` (or something similar) is installed. Also,
8671 your hardware under test must be in a DHCP-enabled network that gives
8672 it the same IP address for each reboot.
8673
8674 If you choose "SystemdbootTarget", there are additional requirements
8675 and considerations. See the
8676 ":ref:`dev-manual/common-tasks:selecting systemdboottarget`" section, which
8677 follows, for more information.
8678
8679- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8680 images and running tests on the BeagleBone "Black" or original
8681 "White" hardware. For information on how to use these tests, see the
8682 comments at the top of the BeagleBoneTarget
8683 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8684
8685- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
8686 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8687 For information on how to use these tests, see the comments at the
8688 top of the EdgeRouterTarget
8689 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8690
8691- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
8692 tests on any generic PC that boots using GRUB. For information on how
8693 to use these tests, see the comments at the top of the GrubTarget
8694 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8695
8696- *"your-target":* Create your own custom target if you want to run
8697 tests when you are deploying images and running tests on a custom
8698 machine within your BSP layer. To do this, you need to add a Python
8699 unit that defines the target class under ``lib/oeqa/controllers/``
8700 within your layer. You must also provide an empty ``__init__.py``.
8701 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8702
8703Selecting SystemdbootTarget
8704~~~~~~~~~~~~~~~~~~~~~~~~~~~
8705
8706If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
8707not need any information in this section. You can skip down to the
8708":ref:`dev-manual/common-tasks:running tests`" section.
8709
8710If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
8711perform a one-time setup of your controller image by doing the following:
8712
87131. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
8714
8715 EFI_PROVIDER = "systemd-boot"
8716
87172. *Build the controller image:* Build the ``core-image-testmaster`` image.
8718 The ``core-image-testmaster`` recipe is provided as an example for a
8719 "controller" image and you can customize the image recipe as you would
8720 any other recipe.
8721
8722 Here are the image recipe requirements:
8723
8724 - Inherits ``core-image`` so that kernel modules are installed.
8725
8726 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
8727 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8728
8729 - Uses a custom :term:`Initramfs` image with a custom
8730 installer. A normal image that you can install usually creates a
8731 single root filesystem partition. This image uses another installer that
8732 creates a specific partition layout. Not all Board Support
8733 Packages (BSPs) can use an installer. For such cases, you need to
8734 manually create the following partition layout on the target:
8735
8736 - First partition mounted under ``/boot``, labeled "boot".
8737
8738 - The main root filesystem partition where this image gets installed,
8739 which is mounted under ``/``.
8740
8741 - Another partition labeled "testrootfs" where test images get
8742 deployed.
8743
87443. *Install image:* Install the image that you just built on the target
8745 system.
8746
8747The final thing you need to do when setting :term:`TEST_TARGET` to
8748"SystemdbootTarget" is to set up the test image:
8749
87501. *Set up your local.conf file:* Make sure you have the following
8751 statements in your ``local.conf`` file::
8752
8753 IMAGE_FSTYPES += "tar.gz"
8754 INHERIT += "testimage"
8755 TEST_TARGET = "SystemdbootTarget"
8756 TEST_TARGET_IP = "192.168.2.3"
8757
87582. *Build your test image:* Use BitBake to build the image::
8759
8760 $ bitbake core-image-sato
8761
8762Power Control
8763~~~~~~~~~~~~~
8764
8765For most hardware targets other than "simpleremote", you can control
8766power:
8767
8768- You can use :term:`TEST_POWERCONTROL_CMD` together with
8769 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
8770 and does power cycling. The test code passes one argument to that
8771 command: off, on or cycle (off then on). Here is an example that
8772 could appear in your ``local.conf`` file::
8773
8774 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8775
8776 In this example, the expect
8777 script does the following:
8778
8779 .. code-block:: shell
8780
8781 ssh test@10.11.12.1 "pyctl nuc1 arg"
8782
8783 It then runs a Python script that controls power for a label called
8784 ``nuc1``.
8785
8786 .. note::
8787
8788 You need to customize :term:`TEST_POWERCONTROL_CMD` and
8789 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
8790 is that it accepts "on", "off", and "cycle" as the last argument.
8791
8792- When no command is defined, it connects to the device over SSH and
8793 uses the classic reboot command to reboot the device. Classic reboot
8794 is fine as long as the machine actually reboots (i.e. the SSH test
8795 has not failed). It is useful for scenarios where you have a simple
8796 setup, typically with a single board, and where some manual
8797 interaction is okay from time to time.
8798
8799If you have no hardware to automatically perform power control but still
8800wish to experiment with automated hardware testing, you can use the
8801``dialog-power-control`` script that shows a dialog prompting you to perform
8802the required power action. This script requires either KDialog or Zenity
8803to be installed. To use this script, set the
8804:term:`TEST_POWERCONTROL_CMD`
8805variable as follows::
8806
8807 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8808
8809Serial Console Connection
8810~~~~~~~~~~~~~~~~~~~~~~~~~
8811
8812For test target classes requiring a serial console to interact with the
8813bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8814you need to specify a command to use to connect to the serial console of
8815the target machine by using the
8816:term:`TEST_SERIALCONTROL_CMD`
8817variable and optionally the
8818:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8819variable.
8820
8821These cases could be a serial terminal program if the machine is
8822connected to a local serial port, or a ``telnet`` or ``ssh`` command
8823connecting to a remote console server. Regardless of the case, the
8824command simply needs to connect to the serial console and forward that
8825connection to standard input and output as any normal terminal program
8826does. For example, to use the picocom terminal program on serial device
8827``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
8828
8829 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8830
8831For local
8832devices where the serial port device disappears when the device reboots,
8833an additional "serdevtry" wrapper script is provided. To use this
8834wrapper, simply prefix the terminal command with
8835``${COREBASE}/scripts/contrib/serdevtry``::
8836
8837 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8838
8839Running Tests
8840-------------
8841
8842You can start the tests automatically or manually:
8843
8844- *Automatically running tests:* To run the tests automatically after the
8845 OpenEmbedded build system successfully creates an image, first set the
8846 :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the
8847 :term:`Build Directory`::
8848
8849 TESTIMAGE_AUTO = "1"
8850
8851 Next, build your image. If the image successfully builds, the
8852 tests run::
8853
8854 bitbake core-image-sato
8855
8856- *Manually running tests:* To manually run the tests, first globally
8857 inherit the :ref:`testimage <ref-classes-testimage>` class
8858 by editing your ``local.conf`` file::
8859
8860 INHERIT += "testimage"
8861
8862 Next, use BitBake to run the tests::
8863
8864 bitbake -c testimage image
8865
8866All test files reside in ``meta/lib/oeqa/runtime`` in the
8867:term:`Source Directory`. A test name maps
8868directly to a Python module. Each test module may contain a number of
8869individual tests. Tests are usually grouped together by the area tested
8870(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
8871
8872You can add tests to any layer provided you place them in the proper
8873area and you extend :term:`BBPATH` in
8874the ``local.conf`` file as normal. Be sure that tests reside in
8875``layer/lib/oeqa/runtime``.
8876
8877.. note::
8878
8879 Be sure that module names do not collide with module names used in
8880 the default set of test modules in ``meta/lib/oeqa/runtime``.
8881
8882You can change the set of tests run by appending or overriding
8883:term:`TEST_SUITES` variable in
8884``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
8885for the image. Test modules named within :term:`TEST_SUITES` cannot be
8886skipped even if a test is not suitable for an image (e.g. running the
8887RPM tests on an image without ``rpm``). Appending "auto" to
8888:term:`TEST_SUITES` causes the build system to try to run all tests that are
8889suitable for the image (i.e. each test module may elect to skip itself).
8890
8891The order you list tests in :term:`TEST_SUITES` is important and influences
8892test dependencies. Consequently, tests that depend on other tests should
8893be added after the test on which they depend. For example, since the
8894``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8895"ping" in the list. The test class provides no re-ordering or dependency
8896handling.
8897
8898.. note::
8899
8900 Each module can have multiple classes with multiple test methods.
8901 And, Python ``unittest`` rules apply.
8902
8903Here are some things to keep in mind when running tests:
8904
8905- The default tests for the image are defined as::
8906
8907 DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
8908
8909- Add your own test to the list of the by using the following::
8910
8911 TEST_SUITES:append = " mytest"
8912
8913- Run a specific list of tests as follows::
8914
8915 TEST_SUITES = "test1 test2 test3"
8916
8917 Remember, order is important. Be sure to place a test that is
8918 dependent on another test later in the order.
8919
8920Exporting Tests
8921---------------
8922
8923You can export tests so that they can run independently of the build
8924system. Exporting tests is required if you want to be able to hand the
8925test execution off to a scheduler. You can only export tests that are
8926defined in :term:`TEST_SUITES`.
8927
8928If your image is already built, make sure the following are set in your
8929``local.conf`` file::
8930
8931 INHERIT += "testexport"
8932 TEST_TARGET_IP = "IP-address-for-the-test-target"
8933 TEST_SERVER_IP = "IP-address-for-the-test-server"
8934
8935You can then export the tests with the
8936following BitBake command form::
8937
8938 $ bitbake image -c testexport
8939
8940Exporting the tests places them in the :term:`Build Directory` in
8941``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
8942variable.
8943
8944You can now run the tests outside of the build environment::
8945
8946 $ cd tmp/testexport/image
8947 $ ./runexported.py testdata.json
8948
8949Here is a complete example that shows IP addresses and uses the
8950``core-image-sato`` image::
8951
8952 INHERIT += "testexport"
8953 TEST_TARGET_IP = "192.168.7.2"
8954 TEST_SERVER_IP = "192.168.7.1"
8955
8956Use BitBake to export the tests::
8957
8958 $ bitbake core-image-sato -c testexport
8959
8960Run the tests outside of
8961the build environment using the following::
8962
8963 $ cd tmp/testexport/core-image-sato
8964 $ ./runexported.py testdata.json
8965
8966Writing New Tests
8967-----------------
8968
8969As mentioned previously, all new test files need to be in the proper
8970place for the build system to find them. New tests for additional
8971functionality outside of the core should be added to the layer that adds
8972the functionality, in ``layer/lib/oeqa/runtime`` (as long as
8973:term:`BBPATH` is extended in the
8974layer's ``layer.conf`` file as normal). Just remember the following:
8975
8976- Filenames need to map directly to test (module) names.
8977
8978- Do not use module names that collide with existing core tests.
8979
8980- Minimally, an empty ``__init__.py`` file must be present in the runtime
8981 directory.
8982
8983To create a new test, start by copying an existing module (e.g.
8984``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
8985code from ``meta/lib/oeqa/utils``, which are helper classes.
8986
8987.. note::
8988
8989 Structure shell commands such that you rely on them and they return a
8990 single code for success. Be aware that sometimes you will need to
8991 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
8992
8993You will notice that all test classes inherit ``oeRuntimeTest``, which
8994is found in ``meta/lib/oetest.py``. This base class offers some helper
8995attributes, which are described in the following sections:
8996
8997Class Methods
8998~~~~~~~~~~~~~
8999
9000Class methods are as follows:
9001
9002- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
9003 package list of the image, which is based on the manifest file that
9004 is generated during the :ref:`ref-tasks-rootfs` task.
9005
9006- *hasFeature(feature):* Returns "True" if the feature is in
9007 :term:`IMAGE_FEATURES` or
9008 :term:`DISTRO_FEATURES`.
9009
9010Class Attributes
9011~~~~~~~~~~~~~~~~
9012
9013Class attributes are as follows:
9014
9015- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
9016 Otherwise, ``pscmd`` equals "ps" (busybox).
9017
9018- *tc:* The called test context, which gives access to the
9019 following attributes:
9020
9021 - *d:* The BitBake datastore, which allows you to use stuff such
9022 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
9023
9024 - *testslist and testsrequired:* Used internally. The tests
9025 do not need these.
9026
9027 - *filesdir:* The absolute path to
9028 ``meta/lib/oeqa/runtime/files``, which contains helper files for
9029 tests meant for copying on the target such as small files written
9030 in C for compilation.
9031
9032 - *target:* The target controller object used to deploy and
9033 start an image on a particular target (e.g. Qemu, SimpleRemote,
9034 and SystemdbootTarget). Tests usually use the following:
9035
9036 - *ip:* The target's IP address.
9037
9038 - *server_ip:* The host's IP address, which is usually used
9039 by the DNF test suite.
9040
9041 - *run(cmd, timeout=None):* The single, most used method.
9042 This command is a wrapper for: ``ssh root@host "cmd"``. The
9043 command returns a tuple: (status, output), which are what their
9044 names imply - the return code of "cmd" and whatever output it
9045 produces. The optional timeout argument represents the number
9046 of seconds the test should wait for "cmd" to return. If the
9047 argument is "None", the test uses the default instance's
9048 timeout period, which is 300 seconds. If the argument is "0",
9049 the test runs until the command returns.
9050
9051 - *copy_to(localpath, remotepath):*
9052 ``scp localpath root@ip:remotepath``.
9053
9054 - *copy_from(remotepath, localpath):*
9055 ``scp root@host:remotepath localpath``.
9056
9057Instance Attributes
9058~~~~~~~~~~~~~~~~~~~
9059
9060There is a single instance attribute, which is ``target``. The ``target``
9061instance attribute is identical to the class attribute of the same name,
9062which is described in the previous section. This attribute exists as
9063both an instance and class attribute so tests can use
9064``self.target.run(cmd)`` in instance methods instead of
9065``oeRuntimeTest.tc.target.run(cmd)``.
9066
9067Installing Packages in the DUT Without the Package Manager
9068----------------------------------------------------------
9069
9070When a test requires a package built by BitBake, it is possible to
9071install that package. Installing the package does not require a package
9072manager be installed in the device under test (DUT). It does, however,
9073require an SSH connection and the target must be using the
9074``sshcontrol`` class.
9075
9076.. note::
9077
9078 This method uses ``scp`` to copy files from the host to the target, which
9079 causes permissions and special attributes to be lost.
9080
9081A JSON file is used to define the packages needed by a test. This file
9082must be in the same path as the file used to define the tests.
9083Furthermore, the filename must map directly to the test module name with
9084a ``.json`` extension.
9085
9086The JSON file must include an object with the test name as keys of an
9087object or an array. This object (or array of objects) uses the following
9088data:
9089
9090- "pkg" --- a mandatory string that is the name of the package to be
9091 installed.
9092
9093- "rm" --- an optional boolean, which defaults to "false", that specifies
9094 to remove the package after the test.
9095
9096- "extract" --- an optional boolean, which defaults to "false", that
9097 specifies if the package must be extracted from the package format.
9098 When set to "true", the package is not automatically installed into
9099 the DUT.
9100
9101Following is an example JSON file that handles test "foo" installing
9102package "bar" and test "foobar" installing packages "foo" and "bar".
9103Once the test is complete, the packages are removed from the DUT.
9104::
9105
9106 {
9107 "foo": {
9108 "pkg": "bar"
9109 },
9110 "foobar": [
9111 {
9112 "pkg": "foo",
9113 "rm": true
9114 },
9115 {
9116 "pkg": "bar",
9117 "rm": true
9118 }
9119 ]
9120 }
9121
9122Debugging Tools and Techniques
9123==============================
9124
9125The exact method for debugging build failures depends on the nature of
9126the problem and on the system's area from which the bug originates.
9127Standard debugging practices such as comparison against the last known
9128working version with examination of the changes and the re-application
9129of steps to identify the one causing the problem are valid for the Yocto
9130Project just as they are for any other system. Even though it is
9131impossible to detail every possible potential failure, this section
9132provides some general tips to aid in debugging given a variety of
9133situations.
9134
9135.. note::
9136
9137 A useful feature for debugging is the error reporting tool.
9138 Configuring the Yocto Project to use this tool causes the
9139 OpenEmbedded build system to produce error reporting commands as part
9140 of the console output. You can enter the commands after the build
9141 completes to log error information into a common database, that can
9142 help you figure out what might be going wrong. For information on how
9143 to enable and use this feature, see the
9144 ":ref:`dev-manual/common-tasks:using the error reporting tool`"
9145 section.
9146
9147The following list shows the debugging topics in the remainder of this
9148section:
9149
9150- ":ref:`dev-manual/common-tasks:viewing logs from failed tasks`" describes
9151 how to find and view logs from tasks that failed during the build
9152 process.
9153
9154- ":ref:`dev-manual/common-tasks:viewing variable values`" describes how to
9155 use the BitBake ``-e`` option to examine variable values after a
9156 recipe has been parsed.
9157
9158- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
9159 describes how to use the ``oe-pkgdata-util`` utility to query
9160 :term:`PKGDATA_DIR` and
9161 display package-related information for built packages.
9162
9163- ":ref:`dev-manual/common-tasks:viewing dependencies between recipes and tasks`"
9164 describes how to use the BitBake ``-g`` option to display recipe
9165 dependency information used during the build.
9166
9167- ":ref:`dev-manual/common-tasks:viewing task variable dependencies`" describes
9168 how to use the ``bitbake-dumpsig`` command in conjunction with key
9169 subdirectories in the :term:`Build Directory` to determine variable
9170 dependencies.
9171
9172- ":ref:`dev-manual/common-tasks:running specific tasks`" describes
9173 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
9174 to run specific tasks in the build chain. It can be useful to run
9175 tasks "out-of-order" when trying isolate build issues.
9176
9177- ":ref:`dev-manual/common-tasks:general BitBake problems`" describes how
9178 to use BitBake's ``-D`` debug output option to reveal more about what
9179 BitBake is doing during the build.
9180
9181- ":ref:`dev-manual/common-tasks:building with no dependencies`"
9182 describes how to use the BitBake ``-b`` option to build a recipe
9183 while ignoring dependencies.
9184
9185- ":ref:`dev-manual/common-tasks:recipe logging mechanisms`"
9186 describes how to use the many recipe logging functions to produce
9187 debugging output and report errors and warnings.
9188
9189- ":ref:`dev-manual/common-tasks:debugging parallel make races`"
9190 describes how to debug situations where the build consists of several
9191 parts that are run simultaneously and when the output or result of
9192 one part is not ready for use with a different part of the build that
9193 depends on that output.
9194
9195- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) remotely`"
9196 describes how to use GDB to allow you to examine running programs, which can
9197 help you fix problems.
9198
9199- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) on the target`"
9200 describes how to use GDB directly on target hardware for debugging.
9201
9202- ":ref:`dev-manual/common-tasks:other debugging tips`" describes
9203 miscellaneous debugging tips that can be useful.
9204
9205Viewing Logs from Failed Tasks
9206------------------------------
9207
9208You can find the log for a task in the file
9209``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
9210For example, the log for the
9211:ref:`ref-tasks-compile` task of the
9212QEMU minimal image for the x86 machine (``qemux86``) might be in
9213``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
9214To see the commands :term:`BitBake` ran
9215to generate a log, look at the corresponding ``run.do_``\ `taskname` file
9216in the same directory.
9217
9218``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
9219links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
9220``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
9221when it ran. The symlinks always point to the files corresponding to the
9222most recent run.
9223
9224Viewing Variable Values
9225-----------------------
9226
9227Sometimes you need to know the value of a variable as a result of
9228BitBake's parsing step. This could be because some unexpected behavior
9229occurred in your project. Perhaps an attempt to :ref:`modify a variable
9230<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
9231variables>` did not work out as expected.
9232
9233BitBake's ``-e`` option is used to display variable values after
9234parsing. The following command displays the variable values after the
9235configuration files (i.e. ``local.conf``, ``bblayers.conf``,
9236``bitbake.conf`` and so forth) have been parsed::
9237
9238 $ bitbake -e
9239
9240The following command displays variable values after a specific recipe has
9241been parsed. The variables include those from the configuration as well::
9242
9243 $ bitbake -e recipename
9244
9245.. note::
9246
9247 Each recipe has its own private set of variables (datastore).
9248 Internally, after parsing the configuration, a copy of the resulting
9249 datastore is made prior to parsing each recipe. This copying implies
9250 that variables set in one recipe will not be visible to other
9251 recipes.
9252
9253 Likewise, each task within a recipe gets a private datastore based on
9254 the recipe datastore, which means that variables set within one task
9255 will not be visible to other tasks.
9256
9257In the output of ``bitbake -e``, each variable is preceded by a
9258description of how the variable got its value, including temporary
9259values that were later overridden. This description also includes
9260variable flags (varflags) set on the variable. The output can be very
9261helpful during debugging.
9262
9263Variables that are exported to the environment are preceded by
9264``export`` in the output of ``bitbake -e``. See the following example::
9265
9266 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9267
9268In addition to variable values, the output of the ``bitbake -e`` and
9269``bitbake -e`` recipe commands includes the following information:
9270
9271- The output starts with a tree listing all configuration files and
9272 classes included globally, recursively listing the files they include
9273 or inherit in turn. Much of the behavior of the OpenEmbedded build
9274 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
9275 implemented in the :ref:`base <ref-classes-base>` class and the
9276 classes it inherits, rather than being built into BitBake itself.
9277
9278- After the variable values, all functions appear in the output. For
9279 shell functions, variables referenced within the function body are
9280 expanded. If a function has been modified using overrides or using
9281 override-style operators like ``:append`` and ``:prepend``, then the
9282 final assembled function body appears in the output.
9283
9284Viewing Package Information with ``oe-pkgdata-util``
9285----------------------------------------------------
9286
9287You can use the ``oe-pkgdata-util`` command-line utility to query
9288:term:`PKGDATA_DIR` and display
9289various package-related information. When you use the utility, you must
9290use it to view information on packages that have already been built.
9291
9292Following are a few of the available ``oe-pkgdata-util`` subcommands.
9293
9294.. note::
9295
9296 You can use the standard \* and ? globbing wildcards as part of
9297 package names and paths.
9298
9299- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9300 that have been built, optionally limiting the match to packages that
9301 match pattern.
9302
9303- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9304 files and directories contained in the given packages.
9305
9306 .. note::
9307
9308 A different way to view the contents of a package is to look at
9309 the
9310 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9311 directory of the recipe that generates the package. This directory
9312 is created by the
9313 :ref:`ref-tasks-package` task
9314 and has one subdirectory for each package the recipe generates,
9315 which contains the files stored in that package.
9316
9317 If you want to inspect the ``${WORKDIR}/packages-split``
9318 directory, make sure that
9319 :ref:`rm_work <ref-classes-rm-work>` is not
9320 enabled when you build the recipe.
9321
9322- ``oe-pkgdata-util find-path path ...``: Lists the names of
9323 the packages that contain the given paths. For example, the following
9324 tells us that ``/usr/share/man/man1/make.1`` is contained in the
9325 ``make-doc`` package::
9326
9327 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
9328 make-doc: /usr/share/man/man1/make.1
9329
9330- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9331 of the recipes that produce the given packages.
9332
9333For more information on the ``oe-pkgdata-util`` command, use the help
9334facility::
9335
9336 $ oe-pkgdata-util --help
9337 $ oe-pkgdata-util subcommand --help
9338
9339Viewing Dependencies Between Recipes and Tasks
9340----------------------------------------------
9341
9342Sometimes it can be hard to see why BitBake wants to build other recipes
9343before the one you have specified. Dependency information can help you
9344understand why a recipe is built.
9345
9346To generate dependency information for a recipe, run the following
9347command::
9348
9349 $ bitbake -g recipename
9350
9351This command writes the following files in the current directory:
9352
9353- ``pn-buildlist``: A list of recipes/targets involved in building
9354 `recipename`. "Involved" here means that at least one task from the
9355 recipe needs to run when building `recipename` from scratch. Targets
9356 that are in
9357 :term:`ASSUME_PROVIDED`
9358 are not listed.
9359
9360- ``task-depends.dot``: A graph showing dependencies between tasks.
9361
9362The graphs are in :wikipedia:`DOT <DOT_%28graph_description_language%29>`
9363format and can be converted to images (e.g. using the ``dot`` tool from
9364`Graphviz <https://www.graphviz.org/>`__).
9365
9366.. note::
9367
9368 - DOT files use a plain text format. The graphs generated using the
9369 ``bitbake -g`` command are often so large as to be difficult to
9370 read without special pruning (e.g. with BitBake's ``-I`` option)
9371 and processing. Despite the form and size of the graphs, the
9372 corresponding ``.dot`` files can still be possible to read and
9373 provide useful information.
9374
9375 As an example, the ``task-depends.dot`` file contains lines such
9376 as the following::
9377
9378 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9379
9380 The above example line reveals that the
9381 :ref:`ref-tasks-configure`
9382 task in ``libxslt`` depends on the
9383 :ref:`ref-tasks-populate_sysroot`
9384 task in ``libxml2``, which is a normal
9385 :term:`DEPENDS` dependency
9386 between the two recipes.
9387
9388 - For an example of how ``.dot`` files can be processed, see the
9389 ``scripts/contrib/graph-tool`` Python script, which finds and
9390 displays paths between graph nodes.
9391
9392You can use a different method to view dependency information by using
9393the following command::
9394
9395 $ bitbake -g -u taskexp recipename
9396
9397This command
9398displays a GUI window from which you can view build-time and runtime
9399dependencies for the recipes involved in building recipename.
9400
9401Viewing Task Variable Dependencies
9402----------------------------------
9403
9404As mentioned in the
9405":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9406User Manual, BitBake tries to automatically determine what variables a
9407task depends on so that it can rerun the task if any values of the
9408variables change. This determination is usually reliable. However, if
9409you do things like construct variable names at runtime, then you might
9410have to manually declare dependencies on those variables using
9411``vardeps`` as described in the
9412":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9413User Manual.
9414
9415If you are unsure whether a variable dependency is being picked up
9416automatically for a given task, you can list the variable dependencies
9417BitBake has determined by doing the following:
9418
94191. Build the recipe containing the task::
9420
9421 $ bitbake recipename
9422
94232. Inside the :term:`STAMPS_DIR`
9424 directory, find the signature data (``sigdata``) file that
9425 corresponds to the task. The ``sigdata`` files contain a pickled
9426 Python database of all the metadata that went into creating the input
9427 checksum for the task. As an example, for the
9428 :ref:`ref-tasks-fetch` task of the
9429 ``db`` recipe, the ``sigdata`` file might be found in the following
9430 location::
9431
9432 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9433
9434 For tasks that are accelerated through the shared state
9435 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
9436 additional ``siginfo`` file is written into
9437 :term:`SSTATE_DIR` along with
9438 the cached task output. The ``siginfo`` files contain exactly the
9439 same information as ``sigdata`` files.
9440
94413. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
9442 is an example::
9443
9444 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9445
9446 In the output of the above command, you will find a line like the
9447 following, which lists all the (inferred) variable dependencies for
9448 the task. This list also includes indirect dependencies from
9449 variables depending on other variables, recursively.
9450 ::
9451
9452 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9453
9454 .. note::
9455
9456 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9457 These functions in turn depend on the variables they reference.
9458
9459 The output of ``bitbake-dumpsig`` also includes the value each
9460 variable had, a list of dependencies for each variable, and
9461 :term:`BB_BASEHASH_IGNORE_VARS`
9462 information.
9463
9464There is also a ``bitbake-diffsigs`` command for comparing two
9465``siginfo`` or ``sigdata`` files. This command can be helpful when
9466trying to figure out what changed between two versions of a task. If you
9467call ``bitbake-diffsigs`` with just one file, the command behaves like
9468``bitbake-dumpsig``.
9469
9470You can also use BitBake to dump out the signature construction
9471information without executing tasks by using either of the following
9472BitBake command-line options::
9473
9474 ‐‐dump-signatures=SIGNATURE_HANDLER
9475 -S SIGNATURE_HANDLER
9476
9477
9478.. note::
9479
9480 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9481 dump only the signature or compare the dumped signature with the cached one,
9482 respectively.
9483
9484Using BitBake with either of these options causes BitBake to dump out
9485``sigdata`` files in the ``stamps`` directory for every task it would
9486have executed instead of building the specified target package.
9487
9488Viewing Metadata Used to Create the Input Signature of a Shared State Task
9489--------------------------------------------------------------------------
9490
9491Seeing what metadata went into creating the input signature of a shared
9492state (sstate) task can be a useful debugging aid. This information is
9493available in signature information (``siginfo``) files in
9494:term:`SSTATE_DIR`. For
9495information on how to view and interpret information in ``siginfo``
9496files, see the
9497":ref:`dev-manual/common-tasks:viewing task variable dependencies`" section.
9498
9499For conceptual information on shared state, see the
9500":ref:`overview-manual/concepts:shared state`"
9501section in the Yocto Project Overview and Concepts Manual.
9502
9503Invalidating Shared State to Force a Task to Run
9504------------------------------------------------
9505
9506The OpenEmbedded build system uses
9507:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
9508:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
9509rebuilding tasks. Collectively, this scheme is known as "shared state
9510code".
9511
9512As with all schemes, this one has some drawbacks. It is possible that
9513you could make implicit changes to your code that the checksum
9514calculations do not take into account. These implicit changes affect a
9515task's output but do not trigger the shared state code into rebuilding a
9516recipe. Consider an example during which a tool changes its output.
9517Assume that the output of ``rpmdeps`` changes. The result of the change
9518should be that all the ``package`` and ``package_write_rpm`` shared
9519state cache items become invalid. However, because the change to the
9520output is external to the code and therefore implicit, the associated
9521shared state cache items do not become invalidated. In this case, the
9522build process uses the cached items rather than running the task again.
9523Obviously, these types of implicit changes can cause problems.
9524
9525To avoid these problems during the build, you need to understand the
9526effects of any changes you make. Realize that changes you make directly
9527to a function are automatically factored into the checksum calculation.
9528Thus, these explicit changes invalidate the associated area of shared
9529state cache. However, you need to be aware of any implicit changes that
9530are not obvious changes to the code and could affect the output of a
9531given task.
9532
9533When you identify an implicit change, you can easily take steps to
9534invalidate the cache and force the tasks to run. The steps you can take
9535are as simple as changing a function's comments in the source code. For
9536example, to invalidate package shared state files, change the comment
9537statements of
9538:ref:`ref-tasks-package` or the
9539comments of one of the functions it calls. Even though the change is
9540purely cosmetic, it causes the checksum to be recalculated and forces
9541the build system to run the task again.
9542
9543.. note::
9544
9545 For an example of a commit that makes a cosmetic change to invalidate
9546 shared state, see this
9547 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
9548
9549Running Specific Tasks
9550----------------------
9551
9552Any given recipe consists of a set of tasks. The standard BitBake
9553behavior in most cases is: :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`,
9554:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, :ref:`ref-tasks-install`, :ref:`ref-tasks-package`,
9555:ref:`do_package_write_* <ref-tasks-package_write_deb>`, and :ref:`ref-tasks-build`. The default task is
9556:ref:`ref-tasks-build` and any tasks on which it depends build first. Some tasks,
9557such as :ref:`ref-tasks-devshell`, are not part of the default build chain. If you
9558wish to run a task that is not part of the default build chain, you can
9559use the ``-c`` option in BitBake. Here is an example::
9560
9561 $ bitbake matchbox-desktop -c devshell
9562
9563The ``-c`` option respects task dependencies, which means that all other
9564tasks (including tasks from other recipes) that the specified task
9565depends on will be run before the task. Even when you manually specify a
9566task to run with ``-c``, BitBake will only run the task if it considers
9567it "out of date". See the
9568":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
9569section in the Yocto Project Overview and Concepts Manual for how
9570BitBake determines whether a task is "out of date".
9571
9572If you want to force an up-to-date task to be rerun (e.g. because you
9573made manual modifications to the recipe's
9574:term:`WORKDIR` that you want to try
9575out), then you can use the ``-f`` option.
9576
9577.. note::
9578
9579 The reason ``-f`` is never required when running the
9580 :ref:`ref-tasks-devshell` task is because the
9581 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
9582 variable flag is already set for the task.
9583
9584The following example shows one way you can use the ``-f`` option::
9585
9586 $ bitbake matchbox-desktop
9587 .
9588 .
9589 make some changes to the source code in the work directory
9590 .
9591 .
9592 $ bitbake matchbox-desktop -c compile -f
9593 $ bitbake matchbox-desktop
9594
9595This sequence first builds and then recompiles ``matchbox-desktop``. The
9596last command reruns all tasks (basically the packaging tasks) after the
9597compile. BitBake recognizes that the :ref:`ref-tasks-compile` task was rerun and
9598therefore understands that the other tasks also need to be run again.
9599
9600Another, shorter way to rerun a task and all
9601:ref:`ref-manual/tasks:normal recipe build tasks`
9602that depend on it is to use the ``-C`` option.
9603
9604.. note::
9605
9606 This option is upper-cased and is separate from the ``-c``
9607 option, which is lower-cased.
9608
9609Using this option invalidates the given task and then runs the
9610:ref:`ref-tasks-build` task, which is
9611the default task if no task is given, and the tasks on which it depends.
9612You could replace the final two commands in the previous example with
9613the following single command::
9614
9615 $ bitbake matchbox-desktop -C compile
9616
9617Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9618the input checksum of the specified task. This tainting indirectly
9619causes the task and its dependent tasks to be rerun through the normal
9620task dependency mechanisms.
9621
9622.. note::
9623
9624 BitBake explicitly keeps track of which tasks have been tainted in
9625 this fashion, and will print warnings such as the following for
9626 builds involving such tasks:
9627
9628 .. code-block:: none
9629
9630 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9631
9632
9633 The purpose of the warning is to let you know that the work directory
9634 and build output might not be in the clean state they would be in for
9635 a "normal" build, depending on what actions you took. To get rid of
9636 such warnings, you can remove the work directory and rebuild the
9637 recipe, as follows::
9638
9639 $ bitbake matchbox-desktop -c clean
9640 $ bitbake matchbox-desktop
9641
9642
9643You can view a list of tasks in a given package by running the
9644:ref:`ref-tasks-listtasks` task as follows::
9645
9646 $ bitbake matchbox-desktop -c listtasks
9647
9648The results appear as output to the console and are also in
9649the file ``${WORKDIR}/temp/log.do_listtasks``.
9650
9651General BitBake Problems
9652------------------------
9653
9654You can see debug output from BitBake by using the ``-D`` option. The
9655debug output gives more information about what BitBake is doing and the
9656reason behind it. Each ``-D`` option you use increases the logging
9657level. The most common usage is ``-DDD``.
9658
9659The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
9660chose a certain version of a package or why BitBake picked a certain
9661provider. This command could also help you in a situation where you
9662think BitBake did something unexpected.
9663
9664Building with No Dependencies
9665-----------------------------
9666
9667To build a specific recipe (``.bb`` file), you can use the following
9668command form::
9669
9670 $ bitbake -b somepath/somerecipe.bb
9671
9672This command form does
9673not check for dependencies. Consequently, you should use it only when
9674you know existing dependencies have been met.
9675
9676.. note::
9677
9678 You can also specify fragments of the filename. In this case, BitBake
9679 checks for a unique match.
9680
9681Recipe Logging Mechanisms
9682-------------------------
9683
9684The Yocto Project provides several logging functions for producing
9685debugging output and reporting errors and warnings. For Python
9686functions, the following logging functions are available. All of these functions
9687log to ``${T}/log.do_``\ `task`, and can also log to standard output
9688(stdout) with the right settings:
9689
9690- ``bb.plain(msg)``: Writes msg as is to the log while also
9691 logging to stdout.
9692
9693- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9694 stdout if BitBake is called with "-v".
9695
9696- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9697 log. Also logs to stdout if the log level is greater than or equal to
9698 level. See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`" option
9699 in the BitBake User Manual for more information.
9700
9701- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9702 logging to stdout.
9703
9704- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9705 logging to standard out (stdout).
9706
9707 .. note::
9708
9709 Calling this function does not cause the task to fail.
9710
9711- ``bb.fatal(msg)``: This logging function is similar to
9712 ``bb.error(msg)`` but also causes the calling task to fail.
9713
9714 .. note::
9715
9716 ``bb.fatal()`` raises an exception, which means you do not need to put a
9717 "return" statement after the function.
9718
9719The same logging functions are also available in shell functions, under
9720the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9721and ``bbfatal``. The
9722:ref:`logging <ref-classes-logging>` class
9723implements these functions. See that class in the ``meta/classes``
9724folder of the :term:`Source Directory` for information.
9725
9726Logging With Python
9727~~~~~~~~~~~~~~~~~~~
9728
9729When creating recipes using Python and inserting code that handles build
9730logs, keep in mind the goal is to have informative logs while keeping
9731the console as "silent" as possible. Also, if you want status messages
9732in the log, use the "debug" loglevel.
9733
9734Following is an example written in Python. The code handles logging for
9735a function that determines the number of tasks needed to be run. See the
9736":ref:`ref-tasks-listtasks`"
9737section for additional information::
9738
9739 python do_listtasks() {
9740 bb.debug(2, "Starting to figure out the task list")
9741 if noteworthy_condition:
9742 bb.note("There are 47 tasks to run")
9743 bb.debug(2, "Got to point xyz")
9744 if warning_trigger:
9745 bb.warn("Detected warning_trigger, this might be a problem later.")
9746 if recoverable_error:
9747 bb.error("Hit recoverable_error, you really need to fix this!")
9748 if fatal_error:
9749 bb.fatal("fatal_error detected, unable to print the task list")
9750 bb.plain("The tasks present are abc")
9751 bb.debug(2, "Finished figuring out the tasklist")
9752 }
9753
9754Logging With Bash
9755~~~~~~~~~~~~~~~~~
9756
9757When creating recipes using Bash and inserting code that handles build
9758logs, you have the same goals --- informative with minimal console output.
9759The syntax you use for recipes written in Bash is similar to that of
9760recipes written in Python described in the previous section.
9761
9762Following is an example written in Bash. The code logs the progress of
9763the ``do_my_function`` function.
9764::
9765
9766 do_my_function() {
9767 bbdebug 2 "Running do_my_function"
9768 if [ exceptional_condition ]; then
9769 bbnote "Hit exceptional_condition"
9770 fi
9771 bbdebug 2 "Got to point xyz"
9772 if [ warning_trigger ]; then
9773 bbwarn "Detected warning_trigger, this might cause a problem later."
9774 fi
9775 if [ recoverable_error ]; then
9776 bberror "Hit recoverable_error, correcting"
9777 fi
9778 if [ fatal_error ]; then
9779 bbfatal "fatal_error detected"
9780 fi
9781 bbdebug 2 "Completed do_my_function"
9782 }
9783
9784
9785Debugging Parallel Make Races
9786-----------------------------
9787
9788A parallel ``make`` race occurs when the build consists of several parts
9789that are run simultaneously and a situation occurs when the output or
9790result of one part is not ready for use with a different part of the
9791build that depends on that output. Parallel make races are annoying and
9792can sometimes be difficult to reproduce and fix. However, there are some simple
9793tips and tricks that can help you debug and fix them. This section
9794presents a real-world example of an error encountered on the Yocto
9795Project autobuilder and the process used to fix it.
9796
9797.. note::
9798
9799 If you cannot properly fix a ``make`` race condition, you can work around it
9800 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
9801 variables.
9802
9803The Failure
9804~~~~~~~~~~~
9805
9806For this example, assume that you are building an image that depends on
9807the "neard" package. And, during the build, BitBake runs into problems
9808and creates the following output.
9809
9810.. note::
9811
9812 This example log file has longer lines artificially broken to make
9813 the listing easier to read.
9814
9815If you examine the output or the log file, you see the failure during
9816``make``:
9817
9818.. code-block:: none
9819
9820 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9821 | DEBUG: Executing shell function do_compile
9822 | NOTE: make -j 16
9823 | make --no-print-directory all-am
9824 | /bin/mkdir -p include/near
9825 | /bin/mkdir -p include/near
9826 | /bin/mkdir -p include/near
9827 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9828 0.14-r0/neard-0.14/include/types.h include/near/types.h
9829 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9830 0.14-r0/neard-0.14/include/log.h include/near/log.h
9831 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9832 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9833 | /bin/mkdir -p include/near
9834 | /bin/mkdir -p include/near
9835 | /bin/mkdir -p include/near
9836 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9837 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
9838 | /bin/mkdir -p include/near
9839 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9840 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
9841 | /bin/mkdir -p include/near
9842 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9843 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
9844 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9845 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
9846 | /bin/mkdir -p include/near
9847 | /bin/mkdir -p include/near
9848 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9849 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
9850 | /bin/mkdir -p include/near
9851 | /bin/mkdir -p include/near
9852 | /bin/mkdir -p include/near
9853 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9854 0.14-r0/neard-0.14/include/device.h include/near/device.h
9855 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9856 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
9857 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9858 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
9859 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9860 0.14-r0/neard-0.14/include/version.h include/near/version.h
9861 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9862 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
9863 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
9864 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/
9865 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
9866 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
9867 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
9868 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/
9869 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/
9870 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
9871 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
9872 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
9873 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
9874 -o tools/snep-send.o tools/snep-send.c
9875 | In file included from tools/snep-send.c:16:0:
9876 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9877 | #include <near/dbus.h>
9878 | ^
9879 | compilation terminated.
9880 | make[1]: *** [tools/snep-send.o] Error 1
9881 | make[1]: *** Waiting for unfinished jobs....
9882 | make: *** [all] Error 2
9883 | ERROR: oe_runmake failed
9884
9885Reproducing the Error
9886~~~~~~~~~~~~~~~~~~~~~
9887
9888Because race conditions are intermittent, they do not manifest
9889themselves every time you do the build. In fact, most times the build
9890will complete without problems even though the potential race condition
9891exists. Thus, once the error surfaces, you need a way to reproduce it.
9892
9893In this example, compiling the "neard" package is causing the problem.
9894So the first thing to do is build "neard" locally. Before you start the
9895build, set the
9896:term:`PARALLEL_MAKE` variable
9897in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
9898high value for :term:`PARALLEL_MAKE` increases the chances of the race
9899condition showing up::
9900
9901 $ bitbake neard
9902
9903Once the local build for "neard" completes, start a ``devshell`` build::
9904
9905 $ bitbake neard -c devshell
9906
9907For information on how to use a ``devshell``, see the
9908":ref:`dev-manual/common-tasks:using a development shell`" section.
9909
9910In the ``devshell``, do the following::
9911
9912 $ make clean
9913 $ make tools/snep-send.o
9914
9915The ``devshell`` commands cause the failure to clearly
9916be visible. In this case, there is a missing dependency for the ``neard``
9917Makefile target. Here is some abbreviated, sample output with the
9918missing dependency clearly visible at the end::
9919
9920 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
9921 .
9922 .
9923 .
9924 tools/snep-send.c
9925 In file included from tools/snep-send.c:16:0:
9926 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9927 #include <near/dbus.h>
9928 ^
9929 compilation terminated.
9930 make: *** [tools/snep-send.o] Error 1
9931 $
9932
9933
9934Creating a Patch for the Fix
9935~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9936
9937Because there is a missing dependency for the Makefile target, you need
9938to patch the ``Makefile.am`` file, which is generated from
9939``Makefile.in``. You can use Quilt to create the patch::
9940
9941 $ quilt new parallelmake.patch
9942 Patch patches/parallelmake.patch is now on top
9943 $ quilt add Makefile.am
9944 File Makefile.am added to patch patches/parallelmake.patch
9945
9946For more information on using Quilt, see the
9947":ref:`dev-manual/common-tasks:using quilt in your workflow`" section.
9948
9949At this point you need to make the edits to ``Makefile.am`` to add the
9950missing dependency. For our example, you have to add the following line
9951to the file::
9952
9953 tools/snep-send.$(OBJEXT): include/near/dbus.h
9954
9955Once you have edited the file, use the ``refresh`` command to create the
9956patch::
9957
9958 $ quilt refresh
9959 Refreshed patch patches/parallelmake.patch
9960
9961Once the patch file is created, you need to add it back to the originating
9962recipe folder. Here is an example assuming a top-level
9963:term:`Source Directory` named ``poky``::
9964
9965 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
9966
9967The final thing you need to do to implement the fix in the build is to
9968update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
9969:term:`SRC_URI` statement includes
9970the patch file. The recipe file is in the folder above the patch. Here
9971is what the edited :term:`SRC_URI` statement would look like::
9972
9973 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
9974 file://neard.in \
9975 file://neard.service.in \
9976 file://parallelmake.patch \
9977 "
9978
9979With the patch complete and moved to the correct folder and the
9980:term:`SRC_URI` statement updated, you can exit the ``devshell``::
9981
9982 $ exit
9983
9984Testing the Build
9985~~~~~~~~~~~~~~~~~
9986
9987With everything in place, you can get back to trying the build again
9988locally::
9989
9990 $ bitbake neard
9991
9992This build should succeed.
9993
9994Now you can open up a ``devshell`` again and repeat the clean and make
9995operations as follows::
9996
9997 $ bitbake neard -c devshell
9998 $ make clean
9999 $ make tools/snep-send.o
10000
10001The build should work without issue.
10002
10003As with all solved problems, if they originated upstream, you need to
10004submit the fix for the recipe in OE-Core and upstream so that the
10005problem is taken care of at its source. See the
10006":ref:`dev-manual/common-tasks:submitting a change to the yocto project`"
10007section for more information.
10008
10009Debugging With the GNU Project Debugger (GDB) Remotely
10010------------------------------------------------------
10011
10012GDB allows you to examine running programs, which in turn helps you to
10013understand and fix problems. It also allows you to perform post-mortem
10014style analysis of program crashes. GDB is available as a package within
10015the Yocto Project and is installed in SDK images by default. See the
10016":ref:`ref-manual/images:Images`" chapter in the Yocto
10017Project Reference Manual for a description of these images. You can find
10018information on GDB at https://sourceware.org/gdb/.
10019
10020.. note::
10021
10022 For best results, install debug (``-dbg``) packages for the applications you
10023 are going to debug. Doing so makes extra debug symbols available that give
10024 you more meaningful output.
10025
10026Sometimes, due to memory or disk space constraints, it is not possible
10027to use GDB directly on the remote target to debug applications. These
10028constraints arise because GDB needs to load the debugging information
10029and the binaries of the process being debugged. Additionally, GDB needs
10030to perform many computations to locate information such as function
10031names, variable names and values, stack traces and so forth --- even
10032before starting the debugging process. These extra computations place
10033more load on the target system and can alter the characteristics of the
10034program being debugged.
10035
10036To help get past the previously mentioned constraints, there are two
10037methods you can use: running a debuginfod server and using gdbserver.
10038
10039Using the debuginfod server method
10040~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10041
10042``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
10043Running a ``debuginfod`` server makes debug symbols readily available,
10044which means you don't need to download debugging information
10045and the binaries of the process being debugged. You can just fetch
10046debug symbols from the server.
10047
10048To run a ``debuginfod`` server, you need to do the following:
10049
10050- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
10051 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
10052 If not, set in your distro config file or in ``local.conf``::
10053
10054 DISTRO_FEATURES:append = " debuginfod"
10055
10056 This distro feature enables the server and client library in ``elfutils``,
10057 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
10058
10059- Run the following commands to launch the ``debuginfod`` server on the host::
10060
10061 $ oe-debuginfod
10062
10063- To use ``debuginfod`` on the target, you need to know the ip:port where
10064 ``debuginfod`` is listening on the host (port defaults to 8002), and export
10065 that into the shell environment, for example in ``qemu``::
10066
10067 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
10068
10069- Then debug info fetching should simply work when running the target ``gdb``,
10070 ``readelf`` or ``objdump``, for example::
10071
10072 root@qemux86-64:~# gdb /bin/cat
10073 ...
10074 Reading symbols from /bin/cat...
10075 Downloading separate debug info for /bin/cat...
10076 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
10077
10078- It's also possible to use ``debuginfod-find`` to just query the server::
10079
10080 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
10081 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
10082
10083
10084Using the gdbserver method
10085~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10086
10087gdbserver, which runs on the remote target and does not load any
10088debugging information from the debugged process. Instead, a GDB instance
10089processes the debugging information that is run on a remote computer -
10090the host GDB. The host GDB then sends control commands to gdbserver to
10091make it stop or start the debugged program, as well as read or write
10092memory regions of that debugged program. All the debugging information
10093loaded and processed as well as all the heavy debugging is done by the
10094host GDB. Offloading these processes gives the gdbserver running on the
10095target a chance to remain small and fast.
10096
10097Because the host GDB is responsible for loading the debugging
10098information and for doing the necessary processing to make actual
10099debugging happen, you have to make sure the host can access the
10100unstripped binaries complete with their debugging information and also
10101be sure the target is compiled with no optimizations. The host GDB must
10102also have local access to all the libraries used by the debugged
10103program. Because gdbserver does not need any local debugging
10104information, the binaries on the remote target can remain stripped.
10105However, the binaries must also be compiled without optimization so they
10106match the host's binaries.
10107
10108To remain consistent with GDB documentation and terminology, the binary
10109being debugged on the remote target machine is referred to as the
10110"inferior" binary. For documentation on GDB see the `GDB
10111site <https://sourceware.org/gdb/documentation/>`__.
10112
10113The following steps show you how to debug using the GNU project
10114debugger.
10115
101161. *Configure your build system to construct the companion debug
10117 filesystem:*
10118
10119 In your ``local.conf`` file, set the following::
10120
10121 IMAGE_GEN_DEBUGFS = "1"
10122 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
10123
10124 These options cause the
10125 OpenEmbedded build system to generate a special companion filesystem
10126 fragment, which contains the matching source and debug symbols to
10127 your deployable filesystem. The build system does this by looking at
10128 what is in the deployed filesystem, and pulling the corresponding
10129 ``-dbg`` packages.
10130
10131 The companion debug filesystem is not a complete filesystem, but only
10132 contains the debug fragments. This filesystem must be combined with
10133 the full filesystem for debugging. Subsequent steps in this procedure
10134 show how to combine the partial filesystem with the full filesystem.
10135
101362. *Configure the system to include gdbserver in the target filesystem:*
10137
10138 Make the following addition in your ``local.conf`` file::
10139
10140 EXTRA_IMAGE_FEATURES:append = " tools-debug"
10141
10142 The change makes
10143 sure the ``gdbserver`` package is included.
10144
101453. *Build the environment:*
10146
10147 Use the following command to construct the image and the companion
10148 Debug Filesystem::
10149
10150 $ bitbake image
10151
10152 Build the cross GDB component and
10153 make it available for debugging. Build the SDK that matches the
10154 image. Building the SDK is best for a production build that can be
10155 used later for debugging, especially during long term maintenance::
10156
10157 $ bitbake -c populate_sdk image
10158
10159 Alternatively, you can build the minimal toolchain components that
10160 match the target. Doing so creates a smaller than typical SDK and
10161 only contains a minimal set of components with which to build simple
10162 test applications, as well as run the debugger::
10163
10164 $ bitbake meta-toolchain
10165
10166 A final method is to build Gdb itself within the build system::
10167
10168 $ bitbake gdb-cross-<architecture>
10169
10170 Doing so produces a temporary copy of
10171 ``cross-gdb`` you can use for debugging during development. While
10172 this is the quickest approach, the two previous methods in this step
10173 are better when considering long-term maintenance strategies.
10174
10175 .. note::
10176
10177 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
10178 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
10179 actual name you want to use.
10180
101814. *Set up the* ``debugfs``\ *:*
10182
10183 Run the following commands to set up the ``debugfs``::
10184
10185 $ mkdir debugfs
10186 $ cd debugfs
10187 $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2
10188 $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2
10189
101905. *Set up GDB:*
10191
10192 Install the SDK (if you built one) and then source the correct
10193 environment file. Sourcing the environment file puts the SDK in your
10194 ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger.
10195
10196 If you are using the build system, Gdb is located in
10197 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
10198
101996. *Boot the target:*
10200
10201 For information on how to run QEMU, see the `QEMU
10202 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
10203
10204 .. note::
10205
10206 Be sure to verify that your host can access the target via TCP.
10207
102087. *Debug a program:*
10209
10210 Debugging a program involves running gdbserver on the target and then
10211 running Gdb on the host. The example in this step debugs ``gzip``:
10212
10213 .. code-block:: shell
10214
10215 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
10216
10217 For
10218 additional gdbserver options, see the `GDB Server
10219 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
10220
10221 After running gdbserver on the target, you need to run Gdb on the
10222 host and configure it and connect to the target. Use these commands::
10223
10224 $ cd directory-holding-the-debugfs-directory
10225 $ arch-gdb
10226 (gdb) set sysroot debugfs
10227 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
10228 (gdb) target remote IP-of-target:1234
10229
10230 At this
10231 point, everything should automatically load (i.e. matching binaries,
10232 symbols and headers).
10233
10234 .. note::
10235
10236 The Gdb ``set`` commands in the previous example can be placed into the
10237 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
10238 commands are in that file.
10239
102408. *Deploying without a full image rebuild:*
10241
10242 In many cases, during development you want a quick method to deploy a
10243 new binary to the target and debug it, without waiting for a full
10244 image build.
10245
10246 One approach to solving this situation is to just build the component
10247 you want to debug. Once you have built the component, copy the
10248 executable directly to both the target and the host ``debugfs``.
10249
10250 If the binary is processed through the debug splitting in
10251 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
10252 contents and corresponding ``/usr/src/debug`` files) from the work
10253 directory. Here is an example::
10254
10255 $ bitbake bash
10256 $ bitbake -c devshell bash
10257 $ cd ..
10258 $ scp packages-split/bash/bin/bash target:/bin/bash
10259 $ cp -a packages-split/bash-dbg/\* path/debugfs
10260
10261Debugging with the GNU Project Debugger (GDB) on the Target
10262-----------------------------------------------------------
10263
10264The previous section addressed using GDB remotely for debugging
10265purposes, which is the most usual case due to the inherent hardware
10266limitations on many embedded devices. However, debugging in the target
10267hardware itself is also possible with more powerful devices. This
10268section describes what you need to do in order to support using GDB to
10269debug on the target hardware.
10270
10271To support this kind of debugging, you need do the following:
10272
10273- Ensure that GDB is on the target. You can do this by making
10274 the following addition to your ``local.conf`` file::
10275
10276 EXTRA_IMAGE_FEATURES:append = " tools-debug"
10277
10278- Ensure that debug symbols are present. You can do so by adding the
10279 corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`::
10280
10281 IMAGE_INSTALL:append = " packagename-dbg"
10282
10283 Alternatively, you can add the following to ``local.conf`` to include
10284 all the debug symbols::
10285
10286 EXTRA_IMAGE_FEATURES:append = " dbg-pkgs"
10287
10288.. note::
10289
10290 To improve the debug information accuracy, you can reduce the level
10291 of optimization used by the compiler. For example, when adding the
10292 following line to your ``local.conf`` file, you will reduce optimization
10293 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
10294 of "-O -fno-omit-frame-pointer"::
10295
10296 DEBUG_BUILD = "1"
10297
10298 Consider that this will reduce the application's performance and is
10299 recommended only for debugging purposes.
10300
10301Other Debugging Tips
10302--------------------
10303
10304Here are some other tips that you might find useful:
10305
10306- When adding new packages, it is worth watching for undesirable items
10307 making their way into compiler command lines. For example, you do not
10308 want references to local system files like ``/usr/lib/`` or
10309 ``/usr/include/``.
10310
10311- If you want to remove the ``psplash`` boot splashscreen, add
10312 ``psplash=false`` to the kernel command line. Doing so prevents
10313 ``psplash`` from loading and thus allows you to see the console. It
10314 is also possible to switch out of the splashscreen by switching the
10315 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10316
10317- Removing :term:`TMPDIR` (usually ``tmp/``, within the
10318 :term:`Build Directory`) can often fix temporary build issues. Removing
10319 :term:`TMPDIR` is usually a relatively cheap operation, because task output
10320 will be cached in :term:`SSTATE_DIR` (usually ``sstate-cache/``, which is
10321 also in the :term:`Build Directory`).
10322
10323 .. note::
10324
10325 Removing :term:`TMPDIR` might be a workaround rather than a fix.
10326 Consequently, trying to determine the underlying cause of an issue before
10327 removing the directory is a good idea.
10328
10329- Understanding how a feature is used in practice within existing
10330 recipes can be very helpful. It is recommended that you configure
10331 some method that allows you to quickly search through files.
10332
10333 Using GNU Grep, you can use the following shell function to
10334 recursively search through common recipe-related files, skipping
10335 binary files, ``.git`` directories, and the :term:`Build Directory`
10336 (assuming its name starts with "build")::
10337
10338 g() {
10339 grep -Ir \
10340 --exclude-dir=.git \
10341 --exclude-dir='build*' \
10342 --include='*.bb*' \
10343 --include='*.inc*' \
10344 --include='*.conf*' \
10345 --include='*.py*' \
10346 "$@"
10347 }
10348
10349 Following are some usage examples::
10350
10351 $ g FOO # Search recursively for "FOO"
10352 $ g -i foo # Search recursively for "foo", ignoring case
10353 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10354
10355 If figuring
10356 out how some feature works requires a lot of searching, it might
10357 indicate that the documentation should be extended or improved. In
10358 such cases, consider filing a documentation bug using the Yocto
10359 Project implementation of
10360 :yocto_bugs:`Bugzilla <>`. For information on
10361 how to submit a bug against the Yocto Project, see the Yocto Project
10362 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
10363 and the
10364 ":ref:`dev-manual/common-tasks:submitting a defect against the yocto project`"
10365 section.
10366
10367 .. note::
10368
10369 The manuals might not be the right place to document variables
10370 that are purely internal and have a limited scope (e.g. internal
10371 variables used to implement a single ``.bbclass`` file).
10372
10373Making Changes to the Yocto Project
10374===================================
10375
10376Because the Yocto Project is an open-source, community-based project,
10377you can effect changes to the project. This section presents procedures
10378that show you how to submit a defect against the project and how to
10379submit a change.
10380
10381Submitting a Defect Against the Yocto Project
10382---------------------------------------------
10383
10384Use the Yocto Project implementation of
10385`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
10386against the Yocto Project. For additional information on this
10387implementation of Bugzilla see the ":ref:`Yocto Project
10388Bugzilla <resources-bugtracker>`" section in the
10389Yocto Project Reference Manual. For more detail on any of the following
10390steps, see the Yocto Project
10391:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`.
10392
10393Use the following general steps to submit a bug:
10394
103951. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10396
103972. Click "File a Bug" to enter a new bug.
10398
103993. Choose the appropriate "Classification", "Product", and "Component"
10400 for which the bug was found. Bugs for the Yocto Project fall into
10401 one of several classifications, which in turn break down into
10402 several products and components. For example, for a bug against the
10403 ``meta-intel`` layer, you would choose "Build System, Metadata &
10404 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10405
104064. Choose the "Version" of the Yocto Project for which you found the
10407 bug (e.g. &DISTRO;).
10408
104095. Determine and select the "Severity" of the bug. The severity
10410 indicates how the bug impacted your work.
10411
104126. Choose the "Hardware" that the bug impacts.
10413
104147. Choose the "Architecture" that the bug impacts.
10415
104168. Choose a "Documentation change" item for the bug. Fixing a bug might
10417 or might not affect the Yocto Project documentation. If you are
10418 unsure of the impact to the documentation, select "Don't Know".
10419
104209. Provide a brief "Summary" of the bug. Try to limit your summary to
10421 just a line or two and be sure to capture the essence of the bug.
10422
1042310. Provide a detailed "Description" of the bug. You should provide as
10424 much detail as you can about the context, behavior, output, and so
10425 forth that surrounds the bug. You can even attach supporting files
10426 for output from logs by using the "Add an attachment" button.
10427
1042811. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10429 is assigned to the bug and the defect is logged in the bug tracking
10430 system.
10431
10432Once you file a bug, the bug is processed by the Yocto Project Bug
10433Triage Team and further details concerning the bug are assigned (e.g.
10434priority and owner). You are the "Submitter" of the bug and any further
10435categorization, progress, or comments on the bug result in Bugzilla
10436sending you an automated email concerning the particular change or
10437progress to the bug.
10438
10439Submitting a Change to the Yocto Project
10440----------------------------------------
10441
10442Contributions to the Yocto Project and OpenEmbedded are very welcome.
10443Because the system is extremely configurable and flexible, we recognize
10444that developers will want to extend, configure or optimize it for their
10445specific uses.
10446
10447The Yocto Project uses a mailing list and a patch-based workflow that is
10448similar to the Linux kernel but contains important differences. In
10449general, there is a mailing list through which you can submit patches. You
10450should send patches to the appropriate mailing list so that they can be
10451reviewed and merged by the appropriate maintainer. The specific mailing
10452list you need to use depends on the location of the code you are
10453changing. Each component (e.g. layer) should have a ``README`` file that
10454indicates where to send the changes and which process to follow.
10455
10456You can send the patch to the mailing list using whichever approach you
10457feel comfortable with to generate the patch. Once sent, the patch is
10458usually reviewed by the community at large. If somebody has concerns
10459with the patch, they will usually voice their concern over the mailing
10460list. If a patch does not receive any negative reviews, the maintainer
10461of the affected layer typically takes the patch, tests it, and then
10462based on successful testing, merges the patch.
10463
10464The "poky" repository, which is the Yocto Project's reference build
10465environment, is a hybrid repository that contains several individual
10466pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10467the combo-layer tool. The upstream location used for submitting changes
10468varies by component:
10469
10470- *Core Metadata:* Send your patch to the
10471 :oe_lists:`openembedded-core </g/openembedded-core>`
10472 mailing list. For example, a change to anything under the ``meta`` or
10473 ``scripts`` directories should be sent to this mailing list.
10474
10475- *BitBake:* For changes to BitBake (i.e. anything under the
10476 ``bitbake`` directory), send your patch to the
10477 :oe_lists:`bitbake-devel </g/bitbake-devel>`
10478 mailing list.
10479
10480- *"meta-\*" trees:* These trees contain Metadata. Use the
10481 :yocto_lists:`poky </g/poky>` mailing list.
10482
10483- *Documentation*: For changes to the Yocto Project documentation, use the
10484 :yocto_lists:`docs </g/docs>` mailing list.
10485
10486For changes to other layers hosted in the Yocto Project source
10487repositories (i.e. ``yoctoproject.org``) and tools use the
10488:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
10489
10490.. note::
10491
10492 Sometimes a layer's documentation specifies to use a particular
10493 mailing list. If so, use that list.
10494
10495For additional recipes that do not fit into the core Metadata, you
10496should determine which layer the recipe should go into and submit the
10497change in the manner recommended by the documentation (e.g. the
10498``README`` file) supplied with the layer. If in doubt, please ask on the
10499Yocto general mailing list or on the openembedded-devel mailing list.
10500
10501You can also push a change upstream and request a maintainer to pull the
10502change into the component's upstream repository. You do this by pushing
10503to a contribution repository that is upstream. See the
10504":ref:`overview-manual/development-environment:git workflows and the yocto project`"
10505section in the Yocto Project Overview and Concepts Manual for additional
10506concepts on working in the Yocto Project development environment.
10507
10508Maintainers commonly use ``-next`` branches to test submissions prior to
10509merging patches. Thus, you can get an idea of the status of a patch based on
10510whether the patch has been merged into one of these branches. The commonly
10511used testing branches for OpenEmbedded-Core are as follows:
10512
10513- *openembedded-core "master-next" branch:* This branch is part of the
10514 :oe_git:`openembedded-core </openembedded-core/>` repository and contains
10515 proposed changes to the core metadata.
10516
10517- *poky "master-next" branch:* This branch is part of the
10518 :yocto_git:`poky </poky/>` repository and combines proposed
10519 changes to BitBake, the core metadata and the poky distro.
10520
10521Similarly, stable branches maintained by the project may have corresponding
10522``-next`` branches which collect proposed changes. For example,
10523``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next``
10524branches in both the "openembdedded-core" and "poky" repositories.
10525
10526Other layers may have similar testing branches but there is no formal
10527requirement or standard for these so please check the documentation for the
10528layers you are contributing to.
10529
10530The following sections provide procedures for submitting a change.
10531
10532Preparing Changes for Submission
10533~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10534
105351. *Make Your Changes Locally:* Make your changes in your local Git
10536 repository. You should make small, controlled, isolated changes.
10537 Keeping changes small and isolated aids review, makes
10538 merging/rebasing easier and keeps the change history clean should
10539 anyone need to refer to it in future.
10540
105412. *Stage Your Changes:* Stage your changes by using the ``git add``
10542 command on each file you changed.
10543
105443. *Commit Your Changes:* Commit the change by using the ``git commit``
10545 command. Make sure your commit information follows standards by
10546 following these accepted conventions:
10547
10548 - Be sure to include a "Signed-off-by:" line in the same style as
10549 required by the Linux kernel. This can be done by using the
10550 ``git commit -s`` command. Adding this line signifies that you,
10551 the submitter, have agreed to the Developer's Certificate of
10552 Origin 1.1 as follows:
10553
10554 .. code-block:: none
10555
10556 Developer's Certificate of Origin 1.1
10557
10558 By making a contribution to this project, I certify that:
10559
10560 (a) The contribution was created in whole or in part by me and I
10561 have the right to submit it under the open source license
10562 indicated in the file; or
10563
10564 (b) The contribution is based upon previous work that, to the best
10565 of my knowledge, is covered under an appropriate open source
10566 license and I have the right under that license to submit that
10567 work with modifications, whether created in whole or in part
10568 by me, under the same open source license (unless I am
10569 permitted to submit under a different license), as indicated
10570 in the file; or
10571
10572 (c) The contribution was provided directly to me by some other
10573 person who certified (a), (b) or (c) and I have not modified
10574 it.
10575
10576 (d) I understand and agree that this project and the contribution
10577 are public and that a record of the contribution (including all
10578 personal information I submit with it, including my sign-off) is
10579 maintained indefinitely and may be redistributed consistent with
10580 this project or the open source license(s) involved.
10581
10582 - Provide a single-line summary of the change and, if more
10583 explanation is needed, provide more detail in the body of the
10584 commit. This summary is typically viewable in the "shortlist" of
10585 changes. Thus, providing something short and descriptive that
10586 gives the reader a summary of the change is useful when viewing a
10587 list of many commits. You should prefix this short description
10588 with the recipe name (if changing a recipe), or else with the
10589 short form path to the file being changed.
10590
10591 - For the body of the commit message, provide detailed information
10592 that describes what you changed, why you made the change, and the
10593 approach you used. It might also be helpful if you mention how you
10594 tested the change. Provide as much detail as you can in the body
10595 of the commit message.
10596
10597 .. note::
10598
10599 You do not need to provide a more detailed explanation of a
10600 change if the change is minor to the point of the single line
10601 summary providing all the information.
10602
10603 - If the change addresses a specific bug or issue that is associated
10604 with a bug-tracking ID, include a reference to that ID in your
10605 detailed description. For example, the Yocto Project uses a
10606 specific convention for bug references --- any commit that addresses
10607 a specific bug should use the following form for the detailed
10608 description. Be sure to use the actual bug-tracking ID from
10609 Bugzilla for bug-id::
10610
10611 Fixes [YOCTO #bug-id]
10612
10613 detailed description of change
10614
10615Using Email to Submit a Patch
10616~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10617
10618Depending on the components changed, you need to submit the email to a
10619specific mailing list. For some guidance on which mailing list to use,
10620see the
10621:ref:`list <dev-manual/common-tasks:submitting a change to the yocto project>`
10622at the beginning of this section. For a description of all the available
10623mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
10624Yocto Project Reference Manual.
10625
10626Here is the general procedure on how to submit a patch through email
10627without using the scripts once the steps in
10628:ref:`dev-manual/common-tasks:preparing changes for submission` have been followed:
10629
106301. *Format the Commit:* Format the commit into an email message. To
10631 format commits, use the ``git format-patch`` command. When you
10632 provide the command, you must include a revision list or a number of
10633 patches as part of the command. For example, either of these two
10634 commands takes your most recent single commit and formats it as an
10635 email message in the current directory::
10636
10637 $ git format-patch -1
10638
10639 or ::
10640
10641 $ git format-patch HEAD~
10642
10643 After the command is run, the current directory contains a numbered
10644 ``.patch`` file for the commit.
10645
10646 If you provide several commits as part of the command, the
10647 ``git format-patch`` command produces a series of numbered files in
10648 the current directory – one for each commit. If you have more than
10649 one patch, you should also use the ``--cover`` option with the
10650 command, which generates a cover letter as the first "patch" in the
10651 series. You can then edit the cover letter to provide a description
10652 for the series of patches. For information on the
10653 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
10654 using the ``man git-format-patch`` command.
10655
10656 .. note::
10657
10658 If you are or will be a frequent contributor to the Yocto Project
10659 or to OpenEmbedded, you might consider requesting a contrib area
10660 and the necessary associated rights.
10661
106622. *Send the patches via email:* Send the patches to the recipients and
10663 relevant mailing lists by using the ``git send-email`` command.
10664
10665 .. note::
10666
10667 In order to use ``git send-email``, you must have the proper Git packages
10668 installed on your host.
10669 For Ubuntu, Debian, and Fedora the package is ``git-email``.
10670
10671 The ``git send-email`` command sends email by using a local or remote
10672 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
10673 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
10674 file. If you are submitting patches through email only, it is very
10675 important that you submit them without any whitespace or HTML
10676 formatting that either you or your mailer introduces. The maintainer
10677 that receives your patches needs to be able to save and apply them
10678 directly from your emails. A good way to verify that what you are
10679 sending will be applicable by the maintainer is to do a dry run and
10680 send them to yourself and then save and apply them as the maintainer
10681 would.
10682
10683 The ``git send-email`` command is the preferred method for sending
10684 your patches using email since there is no risk of compromising
10685 whitespace in the body of the message, which can occur when you use
10686 your own mail client. The command also has several options that let
10687 you specify recipients and perform further editing of the email
10688 message. For information on how to use the ``git send-email``
10689 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
10690 ``man git-send-email`` command.
10691
10692The Yocto Project uses a `Patchwork instance <https://patchwork.openembedded.org/>`__
10693to track the status of patches submitted to the various mailing lists and to
10694support automated patch testing. Each submitted patch is checked for common
10695mistakes and deviations from the expected patch format and submitters are
10696notified by patchtest if such mistakes are found. This process helps to
10697reduce the burden of patch review on maintainers.
10698
10699.. note::
10700
10701 This system is imperfect and changes can sometimes get lost in the flow.
10702 Asking about the status of a patch or change is reasonable if the change
10703 has been idle for a while with no feedback.
10704
10705Using Scripts to Push a Change Upstream and Request a Pull
10706~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10707
10708For larger patch series it is preferable to send a pull request which not
10709only includes the patch but also a pointer to a branch that can be pulled
10710from. This involves making a local branch for your changes, pushing this
10711branch to an accessible repository and then using the ``create-pull-request``
10712and ``send-pull-request`` scripts from openembedded-core to create and send a
10713patch series with a link to the branch for review.
10714
10715Follow this procedure to push a change to an upstream "contrib" Git
10716repository once the steps in :ref:`dev-manual/common-tasks:preparing changes for submission` have
10717been followed:
10718
10719.. note::
10720
10721 You can find general Git information on how to push a change upstream
10722 in the
10723 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
10724
107251. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
10726 permissions to push to an upstream contrib repository, push the
10727 change to that repository::
10728
10729 $ git push upstream_remote_repo local_branch_name
10730
10731 For example, suppose you have permissions to push
10732 into the upstream ``meta-intel-contrib`` repository and you are
10733 working in a local branch named `your_name`\ ``/README``. The following
10734 command pushes your local commits to the ``meta-intel-contrib``
10735 upstream repository and puts the commit in a branch named
10736 `your_name`\ ``/README``::
10737
10738 $ git push meta-intel-contrib your_name/README
10739
107402. *Determine Who to Notify:* Determine the maintainer or the mailing
10741 list that you need to notify for the change.
10742
10743 Before submitting any change, you need to be sure who the maintainer
10744 is or what mailing list that you need to notify. Use either these
10745 methods to find out:
10746
10747 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10748 located in the :term:`Source Directory` at
10749 ``meta/conf/distro/include``, to see who is responsible for code.
10750
10751 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can
10752 enter the following command to bring up a short list of all
10753 commits against a specific file::
10754
10755 git shortlog -- filename
10756
10757 Just provide the name of the file for which you are interested. The
10758 information returned is not ordered by history but does include a
10759 list of everyone who has committed grouped by name. From the list,
10760 you can see who is responsible for the bulk of the changes against
10761 the file.
10762
10763 - *Examine the List of Mailing Lists:* For a list of the Yocto
10764 Project and related mailing lists, see the ":ref:`Mailing
10765 lists <resources-mailinglist>`" section in
10766 the Yocto Project Reference Manual.
10767
107683. *Make a Pull Request:* Notify the maintainer or the mailing list that
10769 you have pushed a change by making a pull request.
10770
10771 The Yocto Project provides two scripts that conveniently let you
10772 generate and send pull requests to the Yocto Project. These scripts
10773 are ``create-pull-request`` and ``send-pull-request``. You can find
10774 these scripts in the ``scripts`` directory within the
10775 :term:`Source Directory` (e.g.
10776 ``poky/scripts``).
10777
10778 Using these scripts correctly formats the requests without
10779 introducing any whitespace or HTML formatting. The maintainer that
10780 receives your patches either directly or through the mailing list
10781 needs to be able to save and apply them directly from your emails.
10782 Using these scripts is the preferred method for sending patches.
10783
10784 First, create the pull request. For example, the following command
10785 runs the script, specifies the upstream repository in the contrib
10786 directory into which you pushed the change, and provides a subject
10787 line in the created patch files::
10788
10789 $ poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
10790
10791 Running this script forms ``*.patch`` files in a folder named
10792 ``pull-``\ `PID` in the current directory. One of the patch files is a
10793 cover letter.
10794
10795 Before running the ``send-pull-request`` script, you must edit the
10796 cover letter patch to insert information about your change. After
10797 editing the cover letter, send the pull request. For example, the
10798 following command runs the script and specifies the patch directory
10799 and email address. In this example, the email address is a mailing
10800 list::
10801
10802 $ poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@lists.yoctoproject.org
10803
10804 You need to follow the prompts as the script is interactive.
10805
10806 .. note::
10807
10808 For help on using these scripts, simply provide the ``-h``
10809 argument as follows::
10810
10811 $ poky/scripts/create-pull-request -h
10812 $ poky/scripts/send-pull-request -h
10813
10814Responding to Patch Review
10815~~~~~~~~~~~~~~~~~~~~~~~~~~
10816
10817You may get feedback on your submitted patches from other community members
10818or from the automated patchtest service. If issues are identified in your
10819patch then it is usually necessary to address these before the patch will be
10820accepted into the project. In this case you should amend the patch according
10821to the feedback and submit an updated version to the relevant mailing list,
10822copying in the reviewers who provided feedback to the previous version of the
10823patch.
10824
10825The patch should be amended using ``git commit --amend`` or perhaps ``git
10826rebase`` for more expert git users. You should also modify the ``[PATCH]``
10827tag in the email subject line when sending the revised patch to mark the new
10828iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be
10829done by passing the ``-v`` argument to ``git format-patch`` with a version
10830number.
10831
10832Lastly please ensure that you also test your revised changes. In particular
10833please don't just edit the patch file written out by ``git format-patch`` and
10834resend it.
10835
10836Submitting Changes to Stable Release Branches
10837~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10838
10839The process for proposing changes to a Yocto Project stable branch differs
10840from the steps described above. Changes to a stable branch must address
10841identified bugs or CVEs and should be made carefully in order to avoid the
10842risk of introducing new bugs or breaking backwards compatibility. Typically
10843bug fixes must already be accepted into the master branch before they can be
10844backported to a stable branch unless the bug in question does not affect the
10845master branch or the fix on the master branch is unsuitable for backporting.
10846
10847The list of stable branches along with the status and maintainer for each
10848branch can be obtained from the
10849:yocto_wiki:`Releases wiki page </Releases>`.
10850
10851.. note::
10852
10853 Changes will not typically be accepted for branches which are marked as
10854 End-Of-Life (EOL).
10855
10856With this in mind, the steps to submit a change for a stable branch are as
10857follows:
10858
108591. *Identify the bug or CVE to be fixed:* This information should be
10860 collected so that it can be included in your submission.
10861
10862 See :ref:`dev-manual/common-tasks:checking for vulnerabilities`
10863 for details about CVE tracking.
10864
108652. *Check if the fix is already present in the master branch:* This will
10866 result in the most straightforward path into the stable branch for the
10867 fix.
10868
10869 a. *If the fix is present in the master branch --- submit a backport request
10870 by email:* You should send an email to the relevant stable branch
10871 maintainer and the mailing list with details of the bug or CVE to be
10872 fixed, the commit hash on the master branch that fixes the issue and
10873 the stable branches which you would like this fix to be backported to.
10874
10875 b. *If the fix is not present in the master branch --- submit the fix to the
10876 master branch first:* This will ensure that the fix passes through the
10877 project's usual patch review and test processes before being accepted.
10878 It will also ensure that bugs are not left unresolved in the master
10879 branch itself. Once the fix is accepted in the master branch a backport
10880 request can be submitted as above.
10881
10882 c. *If the fix is unsuitable for the master branch --- submit a patch
10883 directly for the stable branch:* This method should be considered as a
10884 last resort. It is typically necessary when the master branch is using
10885 a newer version of the software which includes an upstream fix for the
10886 issue or when the issue has been fixed on the master branch in a way
10887 that introduces backwards incompatible changes. In this case follow the
10888 steps in :ref:`dev-manual/common-tasks:preparing changes for submission` and
10889 :ref:`dev-manual/common-tasks:using email to submit a patch` but modify the subject header of your patch
10890 email to include the name of the stable branch which you are
10891 targetting. This can be done using the ``--subject-prefix`` argument to
10892 ``git format-patch``, for example to submit a patch to the dunfell
10893 branch use
10894 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``.
10895
10896Working With Licenses
10897=====================
10898
10899As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
10900section in the Yocto Project Overview and Concepts Manual, open source
10901projects are open to the public and they consequently have different
10902licensing structures in place. This section describes the mechanism by
10903which the :term:`OpenEmbedded Build System`
10904tracks changes to
10905licensing text and covers how to maintain open source license compliance
10906during your project's lifecycle. The section also describes how to
10907enable commercially licensed recipes, which by default are disabled.
10908
10909Tracking License Changes
10910------------------------
10911
10912The license of an upstream project might change in the future. In order
10913to prevent these changes going unnoticed, the
10914:term:`LIC_FILES_CHKSUM`
10915variable tracks changes to the license text. The checksums are validated
10916at the end of the configure step, and if the checksums do not match, the
10917build will fail.
10918
10919Specifying the ``LIC_FILES_CHKSUM`` Variable
10920~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10921
10922The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
10923in the source code for the recipe. Following is an example of how to
10924specify :term:`LIC_FILES_CHKSUM`::
10925
10926 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
10927 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
10928 file://licfile2.txt;endline=50;md5=zzzz \
10929 ..."
10930
10931.. note::
10932
10933 - When using "beginline" and "endline", realize that line numbering
10934 begins with one and not zero. Also, the included lines are
10935 inclusive (i.e. lines five through and including 29 in the
10936 previous example for ``licfile1.txt``).
10937
10938 - When a license check fails, the selected license text is included
10939 as part of the QA message. Using this output, you can determine
10940 the exact start and finish for the needed license text.
10941
10942The build system uses the :term:`S`
10943variable as the default directory when searching files listed in
10944:term:`LIC_FILES_CHKSUM`. The previous example employs the default
10945directory.
10946
10947Consider this next example::
10948
10949 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
10950 md5=bb14ed3c4cda583abc85401304b5cd4e"
10951 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
10952
10953The first line locates a file in ``${S}/src/ls.c`` and isolates lines
10954five through 16 as license text. The second line refers to a file in
10955:term:`WORKDIR`.
10956
10957Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
10958unless the :term:`LICENSE` variable is set to "CLOSED".
10959
10960Explanation of Syntax
10961~~~~~~~~~~~~~~~~~~~~~
10962
10963As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
10964lists all the important files that contain the license text for the
10965source code. It is possible to specify a checksum for an entire file, or
10966a specific section of a file (specified by beginning and ending line
10967numbers with the "beginline" and "endline" parameters, respectively).
10968The latter is useful for source files with a license notice header,
10969README documents, and so forth. If you do not use the "beginline"
10970parameter, then it is assumed that the text begins on the first line of
10971the file. Similarly, if you do not use the "endline" parameter, it is
10972assumed that the license text ends with the last line of the file.
10973
10974The "md5" parameter stores the md5 checksum of the license text. If the
10975license text changes in any way as compared to this parameter then a
10976mismatch occurs. This mismatch triggers a build failure and notifies the
10977developer. Notification allows the developer to review and address the
10978license text changes. Also note that if a mismatch occurs during the
10979build, the correct md5 checksum is placed in the build log and can be
10980easily copied to the recipe.
10981
10982There is no limit to how many files you can specify using the
10983:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
10984requires a few specifications for license tracking. Many projects have a
10985"COPYING" file that stores the license information for all the source
10986code files. This practice allows you to just track the "COPYING" file as
10987long as it is kept up to date.
10988
10989.. note::
10990
10991 - If you specify an empty or invalid "md5" parameter,
10992 :term:`BitBake` returns an md5
10993 mis-match error and displays the correct "md5" parameter value
10994 during the build. The correct parameter is also captured in the
10995 build log.
10996
10997 - If the whole file contains only license text, you do not need to
10998 use the "beginline" and "endline" parameters.
10999
11000Enabling Commercially Licensed Recipes
11001--------------------------------------
11002
11003By default, the OpenEmbedded build system disables components that have
11004commercial or other special licensing requirements. Such requirements
11005are defined on a recipe-by-recipe basis through the
11006:term:`LICENSE_FLAGS` variable
11007definition in the affected recipe. For instance, the
11008``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
11009contains the following statement::
11010
11011 LICENSE_FLAGS = "commercial"
11012
11013Here is a
11014slightly more complicated example that contains both an explicit recipe
11015name and version (after variable expansion)::
11016
11017 LICENSE_FLAGS = "license_${PN}_${PV}"
11018
11019In order for a component restricted by a
11020:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
11021needs to have a matching entry in the global
11022:term:`LICENSE_FLAGS_ACCEPTED`
11023variable, which is a variable typically defined in your ``local.conf``
11024file. For example, to enable the
11025``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
11026could add either the string "commercial_gst-plugins-ugly" or the more
11027general string "commercial" to :term:`LICENSE_FLAGS_ACCEPTED`. See the
11028":ref:`dev-manual/common-tasks:license flag matching`" section for a full
11029explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
11030example::
11031
11032 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly"
11033
11034Likewise, to additionally enable the package built from the recipe
11035containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
11036the actual recipe name was ``emgd_1.10.bb``, the following string would
11037enable that package as well as the original ``gst-plugins-ugly``
11038package::
11039
11040 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly license_emgd_1.10"
11041
11042As a convenience, you do not need to specify the
11043complete license string for every package. You can use
11044an abbreviated form, which consists of just the first portion or
11045portions of the license string before the initial underscore character
11046or characters. A partial string will match any license that contains the
11047given string as the first portion of its license. For example, the
11048following value will also match both of the packages
11049previously mentioned as well as any other packages that have licenses
11050starting with "commercial" or "license".
11051::
11052
11053 LICENSE_FLAGS_ACCEPTED = "commercial license"
11054
11055License Flag Matching
11056~~~~~~~~~~~~~~~~~~~~~
11057
11058License flag matching allows you to control what recipes the
11059OpenEmbedded build system includes in the build. Fundamentally, the
11060build system attempts to match :term:`LICENSE_FLAGS` strings found in
11061recipes against strings found in :term:`LICENSE_FLAGS_ACCEPTED`.
11062A match causes the build system to include a recipe in the
11063build, while failure to find a match causes the build system to exclude
11064a recipe.
11065
11066In general, license flag matching is simple. However, understanding some
11067concepts will help you correctly and effectively use matching.
11068
11069Before a flag defined by a particular recipe is tested against the
11070entries of :term:`LICENSE_FLAGS_ACCEPTED`, the expanded
11071string ``_${PN}`` is appended to the flag. This expansion makes each
11072:term:`LICENSE_FLAGS` value recipe-specific. After expansion, the
11073string is then matched against the entries. Thus, specifying
11074``LICENSE_FLAGS = "commercial"`` in recipe "foo", for example, results
11075in the string ``"commercial_foo"``. And, to create a match, that string
11076must appear among the entries of :term:`LICENSE_FLAGS_ACCEPTED`.
11077
11078Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
11079:term:`LICENSE_FLAGS_ACCEPTED` variable allows you a lot of flexibility for
11080including or excluding recipes based on licensing. For example, you can
11081broaden the matching capabilities by using license flags string subsets
11082in :term:`LICENSE_FLAGS_ACCEPTED`.
11083
11084.. note::
11085
11086 When using a string subset, be sure to use the part of the expanded
11087 string that precedes the appended underscore character (e.g.
11088 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
11089
11090For example, simply specifying the string "commercial" in the
11091:term:`LICENSE_FLAGS_ACCEPTED` variable matches any expanded
11092:term:`LICENSE_FLAGS` definition that starts with the string
11093"commercial" such as "commercial_foo" and "commercial_bar", which
11094are the strings the build system automatically generates for
11095hypothetical recipes named "foo" and "bar" assuming those recipes simply
11096specify the following::
11097
11098 LICENSE_FLAGS = "commercial"
11099
11100Thus, you can choose to exhaustively enumerate each license flag in the
11101list and allow only specific recipes into the image, or you can use a
11102string subset that causes a broader range of matches to allow a range of
11103recipes into the image.
11104
11105This scheme works even if the :term:`LICENSE_FLAGS` string already has
11106``_${PN}`` appended. For example, the build system turns the license
11107flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
11108both the general "commercial" and the specific "commercial_1.2_foo"
11109strings found in the :term:`LICENSE_FLAGS_ACCEPTED` variable, as expected.
11110
11111Here are some other scenarios:
11112
11113- You can specify a versioned string in the recipe such as
11114 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
11115 string to "commercial_foo_1.2_foo". Combine this license flag with a
11116 :term:`LICENSE_FLAGS_ACCEPTED` variable that has the string
11117 "commercial" and you match the flag along with any other flag that
11118 starts with the string "commercial".
11119
11120- Under the same circumstances, you can add "commercial_foo" in the
11121 :term:`LICENSE_FLAGS_ACCEPTED` variable and the build system not only
11122 matches "commercial_foo_1.2" but also matches any license flag with
11123 the string "commercial_foo", regardless of the version.
11124
11125- You can be very specific and use both the package and version parts
11126 in the :term:`LICENSE_FLAGS_ACCEPTED` list (e.g.
11127 "commercial_foo_1.2") to specifically match a versioned recipe.
11128
11129Other Variables Related to Commercial Licenses
11130~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11131
11132There are other helpful variables related to commercial license handling,
11133defined in the
11134``poky/meta/conf/distro/include/default-distrovars.inc`` file::
11135
11136 COMMERCIAL_AUDIO_PLUGINS ?= ""
11137 COMMERCIAL_VIDEO_PLUGINS ?= ""
11138
11139If you
11140want to enable these components, you can do so by making sure you have
11141statements similar to the following in your ``local.conf`` configuration
11142file::
11143
11144 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
11145 gst-plugins-ugly-mpegaudioparse"
11146 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
11147 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
11148 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
11149
11150
11151Of course, you could also create a matching list for those
11152components using the more general "commercial" in the
11153:term:`LICENSE_FLAGS_ACCEPTED` variable, but that would also enable all
11154the other packages with :term:`LICENSE_FLAGS`
11155containing "commercial", which you may or may not want::
11156
11157 LICENSE_FLAGS_ACCEPTED = "commercial"
11158
11159Specifying audio and video plugins as part of the
11160``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
11161(along with the enabling :term:`LICENSE_FLAGS_ACCEPTED`) includes the
11162plugins or components into built images, thus adding support for media
11163formats or components.
11164
11165Maintaining Open Source License Compliance During Your Product's Lifecycle
11166--------------------------------------------------------------------------
11167
11168One of the concerns for a development organization using open source
11169software is how to maintain compliance with various open source
11170licensing during the lifecycle of the product. While this section does
11171not provide legal advice or comprehensively cover all scenarios, it does
11172present methods that you can use to assist you in meeting the compliance
11173requirements during a software release.
11174
11175With hundreds of different open source licenses that the Yocto Project
11176tracks, it is difficult to know the requirements of each and every
11177license. However, the requirements of the major FLOSS licenses can begin
11178to be covered by assuming that there are three main areas of concern:
11179
11180- Source code must be provided.
11181
11182- License text for the software must be provided.
11183
11184- Compilation scripts and modifications to the source code must be
11185 provided.
11186
11187There are other requirements beyond the scope of these three and the
11188methods described in this section (e.g. the mechanism through which
11189source code is distributed).
11190
11191As different organizations have different methods of complying with open
11192source licensing, this section is not meant to imply that there is only
11193one single way to meet your compliance obligations, but rather to
11194describe one method of achieving compliance. The remainder of this
11195section describes methods supported to meet the previously mentioned
11196three requirements. Once you take steps to meet these requirements, and
11197prior to releasing images, sources, and the build system, you should
11198audit all artifacts to ensure completeness.
11199
11200.. note::
11201
11202 The Yocto Project generates a license manifest during image creation
11203 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
11204 to assist with any audits.
11205
11206Providing the Source Code
11207~~~~~~~~~~~~~~~~~~~~~~~~~
11208
11209Compliance activities should begin before you generate the final image.
11210The first thing you should look at is the requirement that tops the list
11211for most compliance groups --- providing the source. The Yocto Project has
11212a few ways of meeting this requirement.
11213
11214One of the easiest ways to meet this requirement is to provide the
11215entire :term:`DL_DIR` used by the
11216build. This method, however, has a few issues. The most obvious is the
11217size of the directory since it includes all sources used in the build
11218and not just the source used in the released image. It will include
11219toolchain source, and other artifacts, which you would not generally
11220release. However, the more serious issue for most companies is
11221accidental release of proprietary software. The Yocto Project provides
11222an :ref:`archiver <ref-classes-archiver>` class to
11223help avoid some of these concerns.
11224
11225Before you employ :term:`DL_DIR` or the :ref:`archiver <ref-classes-archiver>` class, you need to
11226decide how you choose to provide source. The source :ref:`archiver <ref-classes-archiver>` class
11227can generate tarballs and SRPMs and can create them with various levels
11228of compliance in mind.
11229
11230One way of doing this (but certainly not the only way) is to release
11231just the source as a tarball. You can do this by adding the following to
11232the ``local.conf`` file found in the :term:`Build Directory`::
11233
11234 INHERIT += "archiver"
11235 ARCHIVER_MODE[src] = "original"
11236
11237During the creation of your
11238image, the source from all recipes that deploy packages to the image is
11239placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
11240:term:`LICENSE` for each recipe.
11241Releasing the entire directory enables you to comply with requirements
11242concerning providing the unmodified source. It is important to note that
11243the size of the directory can get large.
11244
11245A way to help mitigate the size issue is to only release tarballs for
11246licenses that require the release of source. Let us assume you are only
11247concerned with GPL code as identified by running the following script:
11248
11249.. code-block:: shell
11250
11251 # Script to archive a subset of packages matching specific license(s)
11252 # Source and license files are copied into sub folders of package folder
11253 # Must be run from build folder
11254 #!/bin/bash
11255 src_release_dir="source-release"
11256 mkdir -p $src_release_dir
11257 for a in tmp/deploy/sources/*; do
11258 for d in $a/*; do
11259 # Get package name from path
11260 p=`basename $d`
11261 p=${p%-*}
11262 p=${p%-*}
11263 # Only archive GPL packages (update *GPL* regex for your license check)
11264 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
11265 if [ $numfiles -ge 1 ]; then
11266 echo Archiving $p
11267 mkdir -p $src_release_dir/$p/source
11268 cp $d/* $src_release_dir/$p/source 2> /dev/null
11269 mkdir -p $src_release_dir/$p/license
11270 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
11271 fi
11272 done
11273 done
11274
11275At this point, you
11276could create a tarball from the ``gpl_source_release`` directory and
11277provide that to the end user. This method would be a step toward
11278achieving compliance with section 3a of GPLv2 and with section 6 of
11279GPLv3.
11280
11281Providing License Text
11282~~~~~~~~~~~~~~~~~~~~~~
11283
11284One requirement that is often overlooked is inclusion of license text.
11285This requirement also needs to be dealt with prior to generating the
11286final image. Some licenses require the license text to accompany the
11287binary. You can achieve this by adding the following to your
11288``local.conf`` file::
11289
11290 COPY_LIC_MANIFEST = "1"
11291 COPY_LIC_DIRS = "1"
11292 LICENSE_CREATE_PACKAGE = "1"
11293
11294Adding these statements to the
11295configuration file ensures that the licenses collected during package
11296generation are included on your image.
11297
11298.. note::
11299
11300 Setting all three variables to "1" results in the image having two
11301 copies of the same license file. One copy resides in
11302 ``/usr/share/common-licenses`` and the other resides in
11303 ``/usr/share/license``.
11304
11305 The reason for this behavior is because
11306 :term:`COPY_LIC_DIRS` and
11307 :term:`COPY_LIC_MANIFEST`
11308 add a copy of the license when the image is built but do not offer a
11309 path for adding licenses for newly installed packages to an image.
11310 :term:`LICENSE_CREATE_PACKAGE`
11311 adds a separate package and an upgrade path for adding licenses to an
11312 image.
11313
11314As the source :ref:`archiver <ref-classes-archiver>` class has already archived the original
11315unmodified source that contains the license files, you would have
11316already met the requirements for inclusion of the license information
11317with source as defined by the GPL and other open source licenses.
11318
11319Providing Compilation Scripts and Source Code Modifications
11320~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11321
11322At this point, we have addressed all we need to prior to generating the
11323image. The next two requirements are addressed during the final
11324packaging of the release.
11325
11326By releasing the version of the OpenEmbedded build system and the layers
11327used during the build, you will be providing both compilation scripts
11328and the source code modifications in one step.
11329
11330If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
11331and a distro layer, and those
11332those layers are used to patch, compile, package, or modify (in any way)
11333any open source software included in your released images, you might be
11334required to release those layers under section 3 of GPLv2 or section 1
11335of GPLv3. One way of doing that is with a clean checkout of the version
11336of the Yocto Project and layers used during your build. Here is an
11337example:
11338
11339.. code-block:: shell
11340
11341 # We built using the dunfell branch of the poky repo
11342 $ git clone -b dunfell git://git.yoctoproject.org/poky
11343 $ cd poky
11344 # We built using the release_branch for our layers
11345 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11346 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11347 # clean up the .git repos
11348 $ find . -name ".git" -type d -exec rm -rf {} \;
11349
11350One thing a development organization might want to consider for end-user
11351convenience is to modify
11352``meta-poky/conf/templates/default/bblayers.conf.sample`` to ensure that when
11353the end user utilizes the released build system to build an image, the
11354development organization's layers are included in the ``bblayers.conf`` file
11355automatically::
11356
11357 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11358 # changes incompatibly
11359 POKY_BBLAYERS_CONF_VERSION = "2"
11360
11361 BBPATH = "${TOPDIR}"
11362 BBFILES ?= ""
11363
11364 BBLAYERS ?= " \
11365 ##OEROOT##/meta \
11366 ##OEROOT##/meta-poky \
11367 ##OEROOT##/meta-yocto-bsp \
11368 ##OEROOT##/meta-mylayer \
11369 "
11370
11371Creating and
11372providing an archive of the :term:`Metadata`
11373layers (recipes, configuration files, and so forth) enables you to meet
11374your requirements to include the scripts to control compilation as well
11375as any modifications to the original source.
11376
11377Compliance Limitations with Executables Built from Static Libraries
11378~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11379
11380When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS`
11381mechanisms as well as explicitly included in the image recipe with
11382:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B
11383(``DEPENDS += "B"``), package B will neither appear in the generated license
11384manifest nor in the generated source tarballs. This occurs as the
11385:ref:`license <ref-classes-license>` and :ref:`archiver <ref-classes-archiver>`
11386classes assume that only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS`
11387end up in the image.
11388
11389As a result, potential obligations regarding license compliance for package B
11390may not be met.
11391
11392The Yocto Project doesn't enable static libraries by default, in part because
11393of this issue. Before a solution to this limitation is found, you need to
11394keep in mind that if your root filesystem is built from static libraries,
11395you will need to manually ensure that your deliveries are compliant
11396with the licenses of these libraries.
11397
11398Copying Non Standard Licenses
11399-----------------------------
11400
11401Some packages, such as the linux-firmware package, have many licenses
11402that are not in any way common. You can avoid adding a lot of these
11403types of common license files, which are only applicable to a specific
11404package, by using the
11405:term:`NO_GENERIC_LICENSE`
11406variable. Using this variable also avoids QA errors when you use a
11407non-common, non-CLOSED license in a recipe.
11408
11409Here is an example that uses the ``LICENSE.Abilis.txt`` file as
11410the license from the fetched source::
11411
11412 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11413
11414Checking for Vulnerabilities
11415============================
11416
11417Vulnerabilities in Poky and OE-Core
11418-----------------------------------
11419
11420The Yocto Project has an infrastructure to track and address unfixed
11421known security vulnerabilities, as tracked by the public
11422:wikipedia:`Common Vulnerabilities and Exposures (CVE) <Common_Vulnerabilities_and_Exposures>`
11423database.
11424
11425The Yocto Project maintains a `list of known vulnerabilities
11426<https://autobuilder.yocto.io/pub/non-release/patchmetrics/>`__
11427for packages in Poky and OE-Core, tracking the evolution of the number of
11428unpatched CVEs and the status of patches. Such information is available for
11429the current development version and for each supported release.
11430
11431Security is a process, not a product, and thus at any time, a number of security
11432issues may be impacting Poky and OE-Core. It is up to the maintainers, users,
11433contributors and anyone interested in the issues to investigate and possibly fix them by
11434updating software components to newer versions or by applying patches to address them.
11435It is recommended to work with Poky and OE-Core upstream maintainers and submit
11436patches to fix them, see ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" for details.
11437
11438Vulnerability check at build time
11439---------------------------------
11440
11441To enable a check for CVE security vulnerabilities using :ref:`cve-check <ref-classes-cve-check>` in the specific image
11442or target you are building, add the following setting to your configuration::
11443
11444 INHERIT += "cve-check"
11445
11446The CVE database contains some old incomplete entries which have been
11447deemed not to impact Poky or OE-Core. These CVE entries can be excluded from the
11448check using build configuration::
11449
11450 include conf/distro/include/cve-extra-exclusions.inc
11451
11452With this CVE check enabled, BitBake build will try to map each compiled software component
11453recipe name and version information to the CVE database and generate recipe and
11454image specific reports. These reports will contain:
11455
11456- metadata about the software component like names and versions
11457
11458- metadata about the CVE issue such as description and NVD link
11459
11460- for each software component, a list of CVEs which are possibly impacting this version
11461
11462- status of each CVE: ``Patched``, ``Unpatched`` or ``Ignored``
11463
11464The status ``Patched`` means that a patch file to address the security issue has been
11465applied. ``Unpatched`` status means that no patches to address the issue have been
11466applied and that the issue needs to be investigated. ``Ignored`` means that after
11467analysis, it has been deemed to ignore the issue as it for example affects
11468the software component on a different operating system platform.
11469
11470After a build with CVE check enabled, reports for each compiled source recipe will be
11471found in ``build/tmp/deploy/cve``.
11472
11473For example the CVE check report for the ``flex-native`` recipe looks like::
11474
11475 $ cat poky/build/tmp/deploy/cve/flex-native
11476 LAYER: meta
11477 PACKAGE NAME: flex-native
11478 PACKAGE VERSION: 2.6.4
11479 CVE: CVE-2016-6354
11480 CVE STATUS: Patched
11481 CVE SUMMARY: Heap-based buffer overflow in the yy_get_next_buffer function in Flex before 2.6.1 might allow context-dependent attackers to cause a denial of service or possibly execute arbitrary code via vectors involving num_to_read.
11482 CVSS v2 BASE SCORE: 7.5
11483 CVSS v3 BASE SCORE: 9.8
11484 VECTOR: NETWORK
11485 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2016-6354
11486
11487 LAYER: meta
11488 PACKAGE NAME: flex-native
11489 PACKAGE VERSION: 2.6.4
11490 CVE: CVE-2019-6293
11491 CVE STATUS: Ignored
11492 CVE SUMMARY: An issue was discovered in the function mark_beginning_as_normal in nfa.c in flex 2.6.4. There is a stack exhaustion problem caused by the mark_beginning_as_normal function making recursive calls to itself in certain scenarios involving lots of '*' characters. Remote attackers could leverage this vulnerability to cause a denial-of-service.
11493 CVSS v2 BASE SCORE: 4.3
11494 CVSS v3 BASE SCORE: 5.5
11495 VECTOR: NETWORK
11496 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2019-6293
11497
11498For images, a summary of all recipes included in the image and their CVEs is also
11499generated in textual and JSON formats. These ``.cve`` and ``.json`` reports can be found
11500in the ``tmp/deploy/images`` directory for each compiled image.
11501
11502At build time CVE check will also throw warnings about ``Unpatched`` CVEs::
11503
11504 WARNING: flex-2.6.4-r0 do_cve_check: Found unpatched CVE (CVE-2019-6293), for more information check /poky/build/tmp/work/core2-64-poky-linux/flex/2.6.4-r0/temp/cve.log
11505 WARNING: libarchive-3.5.1-r0 do_cve_check: Found unpatched CVE (CVE-2021-36976), for more information check /poky/build/tmp/work/core2-64-poky-linux/libarchive/3.5.1-r0/temp/cve.log
11506
11507It is also possible to check the CVE status of individual packages as follows::
11508
11509 bitbake -c cve_check flex libarchive
11510
11511Fixing CVE product name and version mappings
11512--------------------------------------------
11513
11514By default, :ref:`cve-check <ref-classes-cve-check>` uses the recipe name :term:`BPN` as CVE
11515product name when querying the CVE database. If this mapping contains false positives, e.g.
11516some reported CVEs are not for the software component in question, or false negatives like
11517some CVEs are not found to impact the recipe when they should, then the problems can be
11518in the recipe name to CVE product mapping. These mapping issues can be fixed by setting
11519the :term:`CVE_PRODUCT` variable inside the recipe. This defines the name of the software component in the
11520upstream `NIST CVE database <https://nvd.nist.gov/>`__.
11521
11522The variable supports using vendor and product names like this::
11523
11524 CVE_PRODUCT = "flex_project:flex"
11525
11526In this example the vendor name used in the CVE database is ``flex_project`` and the
11527product is ``flex``. With this setting the ``flex`` recipe only maps to this specific
11528product and not products from other vendors with same name ``flex``.
11529
11530Similarly, when the recipe version :term:`PV` is not compatible with software versions used by
11531the upstream software component releases and the CVE database, these can be fixed using
11532the :term:`CVE_VERSION` variable.
11533
11534Note that if the CVE entries in the NVD database contain bugs or have missing or incomplete
11535information, it is recommended to fix the information there directly instead of working
11536around the issues possibly for a long time in Poky and OE-Core side recipes. Feedback to
11537NVD about CVE entries can be provided through the `NVD contact form <https://nvd.nist.gov/info/contact-form>`__.
11538
11539Fixing vulnerabilities in recipes
11540---------------------------------
11541
11542If a CVE security issue impacts a software component, it can be fixed by updating to a newer
11543version of the software component or by applying a patch. For Poky and OE-Core master branches, updating
11544to a newer software component release with fixes is the best option, but patches can be applied
11545if releases are not yet available.
11546
11547For stable branches, it is preferred to apply patches for the issues. For some software
11548components minor version updates can also be applied if they are backwards compatible.
11549
11550Here is an example of fixing CVE security issues with patch files,
11551an example from the :oe_layerindex:`ffmpeg recipe</layerindex/recipe/47350>`::
11552
11553 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
11554 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \
11555 file://fix-CVE-2020-20446.patch \
11556 file://fix-CVE-2020-20453.patch \
11557 file://fix-CVE-2020-22015.patch \
11558 file://fix-CVE-2020-22021.patch \
11559 file://fix-CVE-2020-22033-CVE-2020-22019.patch \
11560 file://fix-CVE-2021-33815.patch \
11561
11562A good practice is to include the CVE identifier in both the patch file name
11563and inside the patch file commit message using the format::
11564
11565 CVE: CVE-2020-22033
11566
11567CVE checker will then capture this information and change the CVE status to ``Patched``
11568in the generated reports.
11569
11570If analysis shows that the CVE issue does not impact the recipe due to configuration, platform,
11571version or other reasons, the CVE can be marked as ``Ignored`` using the :term:`CVE_CHECK_IGNORE` variable.
11572As mentioned previously, if data in the CVE database is wrong, it is recommend to fix those
11573issues in the CVE database directly.
11574
11575Recipes can be completely skipped by CVE check by including the recipe name in
11576the :term:`CVE_CHECK_SKIP_RECIPE` variable.
11577
11578Implementation details
11579----------------------
11580
11581Here's what the :ref:`cve-check <ref-classes-cve-check>` class does to
11582find unpatched CVE IDs.
11583
11584First the code goes through each patch file provided by a recipe. If a valid CVE ID
11585is found in the name of the file, the corresponding CVE is considered as patched.
11586Don't forget that if multiple CVE IDs are found in the filename, only the last
11587one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch
11588file. The found CVE IDs are also considered as patched.
11589
11590Then, the code looks up all the CVE IDs in the NIST database for all the
11591products defined in :term:`CVE_PRODUCT`. Then, for each found CVE:
11592
11593- If the package name (:term:`PN`) is part of
11594 :term:`CVE_CHECK_SKIP_RECIPE`, it is considered as ``Patched``.
11595
11596- If the CVE ID is part of :term:`CVE_CHECK_IGNORE`, it is
11597 set as ``Ignored``.
11598
11599- If the CVE ID is part of the patched CVE for the recipe, it is
11600 already considered as ``Patched``.
11601
11602- Otherwise, the code checks whether the recipe version (:term:`PV`)
11603 is within the range of versions impacted by the CVE. If so, the CVE
11604 is considered as ``Unpatched``.
11605
11606The CVE database is stored in :term:`DL_DIR` and can be inspected using
11607``sqlite3`` command as follows::
11608
11609 sqlite3 downloads/CVE_CHECK/nvdcve_1.1.db .dump | grep CVE-2021-37462
11610
11611When analyzing CVEs, it is recommended to:
11612
11613- study the latest information in `CVE database <https://nvd.nist.gov/vuln/search>`__.
11614
11615- check how upstream developers of the software component addressed the issue, e.g.
11616 what patch was applied, which upstream release contains the fix.
11617
11618- check what other Linux distributions like `Debian <https://security-tracker.debian.org/tracker/>`__
11619 did to analyze and address the issue.
11620
11621- follow security notices from other Linux distributions.
11622
11623- follow public `open source security mailing lists <https://oss-security.openwall.org/wiki/mailing-lists>`__ for
11624 discussions and advance notifications of CVE bugs and software releases with fixes.
11625
11626Creating a Software Bill of Materials
11627=====================================
11628
11629Once you are able to build an image for your project, once the licenses for
11630each software component are all identified (see
11631":ref:`dev-manual/common-tasks:working with licenses`") and once vulnerability
11632fixes are applied (see ":ref:`dev-manual/common-tasks:checking
11633for vulnerabilities`"), the OpenEmbedded build system can generate
11634a description of all the components you used, their licenses, their dependencies,
11635the changes that were applied and the known vulnerabilities that were fixed.
11636
11637This description is generated in the form of a *Software Bill of Materials*
11638(:term:`SBOM`), using the :term:`SPDX` standard.
11639
11640When you release software, this is the most standard way to provide information
11641about the Software Supply Chain of your software image and SDK. The
11642:term:`SBOM` tooling is often used to ensure open source license compliance by
11643providing the license texts used in the product which legal departments and end
11644users can read in standardized format.
11645
11646:term:`SBOM` information is also critical to performing vulnerability exposure
11647assessments, as all the components used in the Software Supply Chain are listed.
11648
11649The OpenEmbedded build system doesn't generate such information by default.
11650To make this happen, you must inherit the
11651:ref:`create-spdx <ref-classes-create-spdx>` class from a configuration file::
11652
11653 INHERIT += "create-spdx"
11654
11655You then get :term:`SPDX` output in JSON format as an
11656``IMAGE-MACHINE.spdx.json`` file in ``tmp/deploy/images/MACHINE/`` inside the
11657:term:`Build Directory`.
11658
11659This is a toplevel file accompanied by an ``IMAGE-MACHINE.spdx.index.json``
11660containing an index of JSON :term:`SPDX` files for individual recipes, together
11661with an ``IMAGE-MACHINE.spdx.tar.zst`` compressed archive containing all such
11662files.
11663
11664The :ref:`create-spdx <ref-classes-create-spdx>` class offers options to include
11665more information in the output :term:`SPDX` data, such as making the generated
11666files more human readable (:term:`SPDX_PRETTY`), adding compressed archives of
11667the files in the generated target packages (:term:`SPDX_ARCHIVE_PACKAGED`),
11668adding a description of the source files handled by the target recipes
11669(:term:`SPDX_INCLUDE_SOURCES`) and adding archives of these source files
11670themselves (:term:`SPDX_ARCHIVE_SOURCES`).
11671
11672Though the toplevel :term:`SPDX` output is available in
11673``tmp/deploy/images/MACHINE/`` inside the :term:`Build Directory`, ancillary
11674generated files are available in ``tmp/deploy/spdx/MACHINE`` too, such as:
11675
11676- The individual :term:`SPDX` JSON files in the ``IMAGE-MACHINE.spdx.tar.zst``
11677 archive.
11678
11679- Compressed archives of the files in the generated target packages,
11680 in ``packages/packagename.tar.zst`` (when :term:`SPDX_ARCHIVE_PACKAGED`
11681 is set).
11682
11683- Compressed archives of the source files used to build the host tools
11684 and the target packages in ``recipes/recipe-packagename.tar.zst``
11685 (when :term:`SPDX_ARCHIVE_SOURCES` is set). Those are needed to fulfill
11686 "source code access" license requirements.
11687
11688See the `tools page <https://spdx.dev/resources/tools/>`__ on the :term:`SPDX`
11689project website for a list of tools to consume and transform the :term:`SPDX`
11690data generated by the OpenEmbedded build system.
11691
11692Using the Error Reporting Tool
11693==============================
11694
11695The error reporting tool allows you to submit errors encountered during
11696builds to a central database. Outside of the build environment, you can
11697use a web interface to browse errors, view statistics, and query for
11698errors. The tool works using a client-server system where the client
11699portion is integrated with the installed Yocto Project
11700:term:`Source Directory` (e.g. ``poky``).
11701The server receives the information collected and saves it in a
11702database.
11703
11704There is a live instance of the error reporting server at
11705https://errors.yoctoproject.org.
11706When you want to get help with build failures, you can submit all of the
11707information on the failure easily and then point to the URL in your bug
11708report or send an email to the mailing list.
11709
11710.. note::
11711
11712 If you send error reports to this server, the reports become publicly
11713 visible.
11714
11715Enabling and Using the Tool
11716---------------------------
11717
11718By default, the error reporting tool is disabled. You can enable it by
11719inheriting the :ref:`report-error <ref-classes-report-error>`
11720class by adding the following statement to the end of your
11721``local.conf`` file in your :term:`Build Directory`::
11722
11723 INHERIT += "report-error"
11724
11725By default, the error reporting feature stores information in
11726``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11727However, you can specify a directory to use by adding the following to
11728your ``local.conf`` file::
11729
11730 ERR_REPORT_DIR = "path"
11731
11732Enabling error
11733reporting causes the build process to collect the errors and store them
11734in a file as previously described. When the build system encounters an
11735error, it includes a command as part of the console output. You can run
11736the command to send the error file to the server. For example, the
11737following command sends the errors to an upstream server::
11738
11739 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11740
11741In the previous example, the errors are sent to a public database
11742available at https://errors.yoctoproject.org, which is used by the
11743entire community. If you specify a particular server, you can send the
11744errors to a different database. Use the following command for more
11745information on available options::
11746
11747 $ send-error-report --help
11748
11749When sending the error file, you are prompted to review the data being
11750sent as well as to provide a name and optional email address. Once you
11751satisfy these prompts, the command returns a link from the server that
11752corresponds to your entry in the database. For example, here is a
11753typical link: https://errors.yoctoproject.org/Errors/Details/9522/
11754
11755Following the link takes you to a web interface where you can browse,
11756query the errors, and view statistics.
11757
11758Disabling the Tool
11759------------------
11760
11761To disable the error reporting feature, simply remove or comment out the
11762following statement from the end of your ``local.conf`` file in your
11763:term:`Build Directory`.
11764::
11765
11766 INHERIT += "report-error"
11767
11768Setting Up Your Own Error Reporting Server
11769------------------------------------------
11770
11771If you want to set up your own error reporting server, you can obtain
11772the code from the Git repository at :yocto_git:`/error-report-web/`.
11773Instructions on how to set it up are in the README document.
11774
11775Using Wayland and Weston
11776========================
11777
11778:wikipedia:`Wayland <Wayland_(display_server_protocol)>`
11779is a computer display server protocol that provides a method for
11780compositing window managers to communicate directly with applications
11781and video hardware and expects them to communicate with input hardware
11782using other libraries. Using Wayland with supporting targets can result
11783in better control over graphics frame rendering than an application
11784might otherwise achieve.
11785
11786The Yocto Project provides the Wayland protocol libraries and the
11787reference :wikipedia:`Weston <Wayland_(display_server_protocol)#Weston>`
11788compositor as part of its release. You can find the integrated packages
11789in the ``meta`` layer of the :term:`Source Directory`.
11790Specifically, you
11791can find the recipes that build both Wayland and Weston at
11792``meta/recipes-graphics/wayland``.
11793
11794You can build both the Wayland and Weston packages for use only with targets
11795that accept the :wikipedia:`Mesa 3D and Direct Rendering Infrastructure
11796<Mesa_(computer_graphics)>`, which is also known as Mesa DRI. This implies that
11797you cannot build and use the packages if your target uses, for example, the
11798Intel Embedded Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11799
11800.. note::
11801
11802 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11803 emulated QEMU hardware. However, this version of Weston will run
11804 under X emulation without issues.
11805
11806This section describes what you need to do to implement Wayland and use
11807the Weston compositor when building an image for a supporting target.
11808
11809Enabling Wayland in an Image
11810----------------------------
11811
11812To enable Wayland, you need to enable it to be built and enable it to be
11813included (installed) in the image.
11814
11815Building Wayland
11816~~~~~~~~~~~~~~~~
11817
11818To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11819Wayland with Kernel Mode Setting
11820(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11821support, include the "wayland" flag in the
11822:term:`DISTRO_FEATURES`
11823statement in your ``local.conf`` file::
11824
11825 DISTRO_FEATURES:append = " wayland"
11826
11827.. note::
11828
11829 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11830 support
11831
11832Installing Wayland and Weston
11833~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11834
11835To install the Wayland feature into an image, you must include the
11836following
11837:term:`CORE_IMAGE_EXTRA_INSTALL`
11838statement in your ``local.conf`` file::
11839
11840 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11841
11842Running Weston
11843--------------
11844
11845To run Weston inside X11, enabling it as described earlier and building
11846a Sato image is sufficient. If you are running your image under Sato, a
11847Weston Launcher appears in the "Utility" category.
11848
11849Alternatively, you can run Weston through the command-line interpretor
11850(CLI), which is better suited for development work. To run Weston under
11851the CLI, you need to do the following after your image is built:
11852
118531. Run these commands to export ``XDG_RUNTIME_DIR``::
11854
11855 mkdir -p /tmp/$USER-weston
11856 chmod 0700 /tmp/$USER-weston
11857 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11858
118592. Launch Weston in the shell::
11860
11861 weston