summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMichael Opdenacker <michael.opdenacker@bootlin.com>2022-11-24 17:50:52 +0100
committerRichard Purdie <richard.purdie@linuxfoundation.org>2022-12-01 19:20:29 +0000
commit945c669138a76be18c6b4da4f8f907d2a5cfd83f (patch)
treecebff3cae5021d4fcceb5aa51fce1c2aead97ed2
parent6fe3143800925463279d0664fc7f3372b53c6c52 (diff)
downloadpoky-945c669138a76be18c6b4da4f8f907d2a5cfd83f.tar.gz
manuals: split dev-manual/common-tasks.rst
A 500 KB source file is always harder to manage, and can have section title conflicts. So, the "Common Tasks" document is gone and all its constituents are moved up one level. You now have 40 chapters in the Development Tasks Manual. (From yocto-docs rev: 8a45bc469411410020b8e688c137395fcaf3761b) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-rw-r--r--documentation/brief-yoctoprojectqs/index.rst4
-rw-r--r--documentation/bsp-guide/bsp.rst22
-rw-r--r--documentation/dev-manual/bmaptool.rst59
-rw-r--r--documentation/dev-manual/build-quality.rst411
-rw-r--r--documentation/dev-manual/building.rst940
-rw-r--r--documentation/dev-manual/changes.rst525
-rw-r--r--documentation/dev-manual/common-tasks.rst11861
-rw-r--r--documentation/dev-manual/custom-distribution.rst108
-rw-r--r--documentation/dev-manual/custom-template-configuration-directory.rst52
-rw-r--r--documentation/dev-manual/customizing-images.rst223
-rw-r--r--documentation/dev-manual/debugging.rst1253
-rw-r--r--documentation/dev-manual/development-shell.rst82
-rw-r--r--documentation/dev-manual/device-manager.rst72
-rw-r--r--documentation/dev-manual/disk-space.rst40
-rw-r--r--documentation/dev-manual/efficiently-fetching-sources.rst68
-rw-r--r--documentation/dev-manual/error-reporting-tool.rst85
-rw-r--r--documentation/dev-manual/external-scm.rst67
-rw-r--r--documentation/dev-manual/external-toolchain.rst28
-rw-r--r--documentation/dev-manual/gobject-introspection.rst157
-rw-r--r--documentation/dev-manual/index.rst38
-rw-r--r--documentation/dev-manual/init-manager.rst89
-rw-r--r--documentation/dev-manual/layers.rst905
-rw-r--r--documentation/dev-manual/libraries.rst267
-rw-r--r--documentation/dev-manual/licenses.rst520
-rw-r--r--documentation/dev-manual/new-machine.rst118
-rw-r--r--documentation/dev-manual/new-recipe.rst1674
-rw-r--r--documentation/dev-manual/packages.rst1252
-rw-r--r--documentation/dev-manual/prebuilt-libraries.rst209
-rw-r--r--documentation/dev-manual/python-development-shell.rst50
-rw-r--r--documentation/dev-manual/quilt.rst90
-rw-r--r--documentation/dev-manual/read-only-rootfs.rst89
-rw-r--r--documentation/dev-manual/runtime-testing.rst599
-rw-r--r--documentation/dev-manual/sbom.rst68
-rw-r--r--documentation/dev-manual/securing-images.rst158
-rw-r--r--documentation/dev-manual/speeding-up-build.rst110
-rw-r--r--documentation/dev-manual/start.rst6
-rw-r--r--documentation/dev-manual/temporary-source-code.rst66
-rw-r--r--documentation/dev-manual/upgrading-recipes.rst400
-rw-r--r--documentation/dev-manual/vulnerabilities.rst214
-rw-r--r--documentation/dev-manual/wayland.rst90
-rw-r--r--documentation/dev-manual/wic.rst732
-rw-r--r--documentation/dev-manual/x32-psabi.rst54
-rw-r--r--documentation/kernel-dev/common.rst16
-rw-r--r--documentation/kernel-dev/faq.rst2
-rw-r--r--documentation/kernel-dev/intro.rst2
-rw-r--r--documentation/migration-guides/migration-1.4.rst2
-rw-r--r--documentation/migration-guides/migration-1.5.rst4
-rw-r--r--documentation/migration-guides/migration-1.6.rst6
-rw-r--r--documentation/migration-guides/migration-1.7.rst2
-rw-r--r--documentation/migration-guides/migration-2.1.rst2
-rw-r--r--documentation/migration-guides/migration-2.3.rst2
-rw-r--r--documentation/migration-guides/migration-2.5.rst2
-rw-r--r--documentation/migration-guides/migration-2.6.rst2
-rw-r--r--documentation/migration-guides/migration-general.rst2
-rw-r--r--documentation/overview-manual/concepts.rst20
-rw-r--r--documentation/overview-manual/development-environment.rst12
-rw-r--r--documentation/overview-manual/yp-intro.rst10
-rw-r--r--documentation/ref-manual/classes.rst38
-rw-r--r--documentation/ref-manual/devtool-reference.rst4
-rw-r--r--documentation/ref-manual/faq.rst6
-rw-r--r--documentation/ref-manual/features.rst8
-rw-r--r--documentation/ref-manual/images.rst4
-rw-r--r--documentation/ref-manual/kickstart.rst2
-rw-r--r--documentation/ref-manual/release-process.rst4
-rw-r--r--documentation/ref-manual/resources.rst4
-rw-r--r--documentation/ref-manual/structure.rst12
-rw-r--r--documentation/ref-manual/system-requirements.rst2
-rw-r--r--documentation/ref-manual/tasks.rst10
-rw-r--r--documentation/ref-manual/terms.rst8
-rw-r--r--documentation/ref-manual/variables.rst112
-rw-r--r--documentation/sdk-manual/extensible.rst4
-rw-r--r--documentation/test-manual/intro.rst2
-rw-r--r--documentation/test-manual/yocto-project-compatible.rst2
-rw-r--r--documentation/toaster-manual/reference.rst2
-rw-r--r--documentation/transitioning-to-a-custom-environment.rst8
-rw-r--r--documentation/what-i-wish-id-known.rst2
76 files changed, 12137 insertions, 12038 deletions
diff --git a/documentation/brief-yoctoprojectqs/index.rst b/documentation/brief-yoctoprojectqs/index.rst
index d322bbca6b..cfdd14ae4a 100644
--- a/documentation/brief-yoctoprojectqs/index.rst
+++ b/documentation/brief-yoctoprojectqs/index.rst
@@ -361,7 +361,7 @@ Follow these steps to add a hardware layer:
361 361
362 You can find 362 You can find
363 more information on adding layers in the 363 more information on adding layers in the
364 :ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script` 364 :ref:`dev-manual/layers:adding a layer using the \`\`bitbake-layers\`\` script`
365 section. 365 section.
366 366
367Completing these steps has added the ``meta-altera`` layer to your Yocto 367Completing these steps has added the ``meta-altera`` layer to your Yocto
@@ -396,7 +396,7 @@ The following commands run the tool to create a layer named
396 396
397For more information 397For more information
398on layers and how to create them, see the 398on layers and how to create them, see the
399:ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script` 399:ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`
400section in the Yocto Project Development Tasks Manual. 400section in the Yocto Project Development Tasks Manual.
401 401
402Where To Go Next 402Where To Go Next
diff --git a/documentation/bsp-guide/bsp.rst b/documentation/bsp-guide/bsp.rst
index 117056ebd1..23c70b5a30 100644
--- a/documentation/bsp-guide/bsp.rst
+++ b/documentation/bsp-guide/bsp.rst
@@ -127,7 +127,7 @@ you want to work with, such as::
127and so on. 127and so on.
128 128
129For more information on layers, see the 129For more information on layers, see the
130":ref:`dev-manual/common-tasks:understanding and creating layers`" 130":ref:`dev-manual/layers:understanding and creating layers`"
131section of the Yocto Project Development Tasks Manual. 131section of the Yocto Project Development Tasks Manual.
132 132
133Preparing Your Build Host to Work With BSP Layers 133Preparing Your Build Host to Work With BSP Layers
@@ -463,7 +463,7 @@ requirements are handled with the ``COPYING.MIT`` file.
463Licensing files can be MIT, BSD, GPLv*, and so forth. These files are 463Licensing files can be MIT, BSD, GPLv*, and so forth. These files are
464recommended for the BSP but are optional and totally up to the BSP 464recommended for the BSP but are optional and totally up to the BSP
465developer. For information on how to maintain license compliance, see 465developer. For information on how to maintain license compliance, see
466the ":ref:`dev-manual/common-tasks:maintaining open source license compliance during your product's lifecycle`" 466the ":ref:`dev-manual/licenses:maintaining open source license compliance during your product's lifecycle`"
467section in the Yocto Project Development Tasks Manual. 467section in the Yocto Project Development Tasks Manual.
468 468
469README File 469README File
@@ -589,7 +589,7 @@ filenames correspond to the values to which users have set the
589 589
590These files define things such as the kernel package to use 590These files define things such as the kernel package to use
591(:term:`PREFERRED_PROVIDER` of 591(:term:`PREFERRED_PROVIDER` of
592:ref:`virtual/kernel <dev-manual/common-tasks:using virtual providers>`), 592:ref:`virtual/kernel <dev-manual/new-recipe:using virtual providers>`),
593the hardware drivers to include in different types of images, any 593the hardware drivers to include in different types of images, any
594special software components that are needed, any bootloader information, 594special software components that are needed, any bootloader information,
595and also any special image format requirements. 595and also any special image format requirements.
@@ -757,7 +757,7 @@ workflow.
757 OpenEmbedded build system knows about. For more information on 757 OpenEmbedded build system knows about. For more information on
758 layers, see the ":ref:`overview-manual/yp-intro:the yocto project layer model`" 758 layers, see the ":ref:`overview-manual/yp-intro:the yocto project layer model`"
759 section in the Yocto Project Overview and Concepts Manual. You can also 759 section in the Yocto Project Overview and Concepts Manual. You can also
760 reference the ":ref:`dev-manual/common-tasks:understanding and creating layers`" 760 reference the ":ref:`dev-manual/layers:understanding and creating layers`"
761 section in the Yocto Project Development Tasks Manual. For more 761 section in the Yocto Project Development Tasks Manual. For more
762 information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`" 762 information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
763 section. 763 section.
@@ -816,7 +816,7 @@ workflow.
816 key configuration files are configured appropriately: the 816 key configuration files are configured appropriately: the
817 ``conf/local.conf`` and the ``conf/bblayers.conf`` file. You must 817 ``conf/local.conf`` and the ``conf/bblayers.conf`` file. You must
818 make the OpenEmbedded build system aware of your new layer. See the 818 make the OpenEmbedded build system aware of your new layer. See the
819 ":ref:`dev-manual/common-tasks:enabling your layer`" 819 ":ref:`dev-manual/layers:enabling your layer`"
820 section in the Yocto Project Development Tasks Manual for information 820 section in the Yocto Project Development Tasks Manual for information
821 on how to let the build system know about your new layer. 821 on how to let the build system know about your new layer.
822 822
@@ -845,7 +845,7 @@ Before looking at BSP requirements, you should consider the following:
845 layer that can be added to the Yocto Project. For guidelines on 845 layer that can be added to the Yocto Project. For guidelines on
846 creating a layer that meets these base requirements, see the 846 creating a layer that meets these base requirements, see the
847 ":ref:`bsp-guide/bsp:bsp layers`" section in this manual and the 847 ":ref:`bsp-guide/bsp:bsp layers`" section in this manual and the
848 ":ref:`dev-manual/common-tasks:understanding and creating layers`" 848 ":ref:`dev-manual/layers:understanding and creating layers`"
849 section in the Yocto Project Development Tasks Manual. 849 section in the Yocto Project Development Tasks Manual.
850 850
851- The requirements in this section apply regardless of how you package 851- The requirements in this section apply regardless of how you package
@@ -927,7 +927,7 @@ Yocto Project:
927 - The name and contact information for the BSP layer maintainer. 927 - The name and contact information for the BSP layer maintainer.
928 This is the person to whom patches and questions should be sent. 928 This is the person to whom patches and questions should be sent.
929 For information on how to find the right person, see the 929 For information on how to find the right person, see the
930 ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" 930 ":ref:`dev-manual/changes:submitting a change to the yocto project`"
931 section in the Yocto Project Development Tasks Manual. 931 section in the Yocto Project Development Tasks Manual.
932 932
933 - Instructions on how to build the BSP using the BSP layer. 933 - Instructions on how to build the BSP using the BSP layer.
@@ -1013,7 +1013,7 @@ the following:
1013 1013
1014- Create a ``*.bbappend`` file for the modified recipe. For information on using 1014- Create a ``*.bbappend`` file for the modified recipe. For information on using
1015 append files, see the 1015 append files, see the
1016 ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`" 1016 ":ref:`dev-manual/layers:appending other layers metadata with your layer`"
1017 section in the Yocto Project Development Tasks Manual. 1017 section in the Yocto Project Development Tasks Manual.
1018 1018
1019- Ensure your directory structure in the BSP layer that supports your 1019- Ensure your directory structure in the BSP layer that supports your
@@ -1117,7 +1117,7 @@ list describes them in order of preference:
1117 Specifying the matching license string signifies that you agree to 1117 Specifying the matching license string signifies that you agree to
1118 the license. Thus, the build system can build the corresponding 1118 the license. Thus, the build system can build the corresponding
1119 recipe and include the component in the image. See the 1119 recipe and include the component in the image. See the
1120 ":ref:`dev-manual/common-tasks:enabling commercially licensed recipes`" 1120 ":ref:`dev-manual/licenses:enabling commercially licensed recipes`"
1121 section in the Yocto Project Development Tasks Manual for details on 1121 section in the Yocto Project Development Tasks Manual for details on
1122 how to use these variables. 1122 how to use these variables.
1123 1123
@@ -1169,7 +1169,7 @@ Use these steps to create a BSP layer:
1169 ``create-layer`` subcommand to create a new general layer. For 1169 ``create-layer`` subcommand to create a new general layer. For
1170 instructions on how to create a general layer using the 1170 instructions on how to create a general layer using the
1171 ``bitbake-layers`` script, see the 1171 ``bitbake-layers`` script, see the
1172 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`" 1172 ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
1173 section in the Yocto Project Development Tasks Manual. 1173 section in the Yocto Project Development Tasks Manual.
1174 1174
1175- *Create a Layer Configuration File:* Every layer needs a layer 1175- *Create a Layer Configuration File:* Every layer needs a layer
@@ -1229,7 +1229,7 @@ configuration files is to examine various files for BSP from the
1229:yocto_git:`Source Repositories <>`. 1229:yocto_git:`Source Repositories <>`.
1230 1230
1231For a detailed description of this particular layer configuration file, 1231For a detailed description of this particular layer configuration file,
1232see ":ref:`step 3 <dev-manual/common-tasks:creating your own layer>`" 1232see ":ref:`step 3 <dev-manual/layers:creating your own layer>`"
1233in the discussion that describes how to create layers in the Yocto 1233in the discussion that describes how to create layers in the Yocto
1234Project Development Tasks Manual. 1234Project Development Tasks Manual.
1235 1235
diff --git a/documentation/dev-manual/bmaptool.rst b/documentation/dev-manual/bmaptool.rst
new file mode 100644
index 0000000000..4ee6f5e48b
--- /dev/null
+++ b/documentation/dev-manual/bmaptool.rst
@@ -0,0 +1,59 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Flashing Images Using ``bmaptool``
4**********************************
5
6A fast and easy way to flash an image to a bootable device is to use
7Bmaptool, which is integrated into the OpenEmbedded build system.
8Bmaptool is a generic tool that creates a file's block map (bmap) and
9then uses that map to copy the file. As compared to traditional tools
10such as dd or cp, Bmaptool can copy (or flash) large files like raw
11system image files much faster.
12
13.. note::
14
15 - If you are using Ubuntu or Debian distributions, you can install
16 the ``bmap-tools`` package using the following command and then
17 use the tool without specifying ``PATH`` even from the root
18 account::
19
20 $ sudo apt install bmap-tools
21
22 - If you are unable to install the ``bmap-tools`` package, you will
23 need to build Bmaptool before using it. Use the following command::
24
25 $ bitbake bmap-tools-native
26
27Following, is an example that shows how to flash a Wic image. Realize
28that while this example uses a Wic image, you can use Bmaptool to flash
29any type of image. Use these steps to flash an image using Bmaptool:
30
311. *Update your local.conf File:* You need to have the following set
32 in your ``local.conf`` file before building your image::
33
34 IMAGE_FSTYPES += "wic wic.bmap"
35
362. *Get Your Image:* Either have your image ready (pre-built with the
37 :term:`IMAGE_FSTYPES`
38 setting previously mentioned) or take the step to build the image::
39
40 $ bitbake image
41
423. *Flash the Device:* Flash the device with the image by using Bmaptool
43 depending on your particular setup. The following commands assume the
44 image resides in the :term:`Build Directory`'s ``deploy/images/`` area:
45
46 - If you have write access to the media, use this command form::
47
48 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
49
50 - If you do not have write access to the media, set your permissions
51 first and then use the same command form::
52
53 $ sudo chmod 666 /dev/sdX
54 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
55
56For help on the ``bmaptool`` command, use the following command::
57
58 $ bmaptool --help
59
diff --git a/documentation/dev-manual/build-quality.rst b/documentation/dev-manual/build-quality.rst
new file mode 100644
index 0000000000..03eee12bef
--- /dev/null
+++ b/documentation/dev-manual/build-quality.rst
@@ -0,0 +1,411 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Maintaining Build Output Quality
4********************************
5
6Many factors can influence the quality of a build. For example, if you
7upgrade a recipe to use a new version of an upstream software package or
8you experiment with some new configuration options, subtle changes can
9occur that you might not detect until later. Consider the case where
10your recipe is using a newer version of an upstream package. In this
11case, a new version of a piece of software might introduce an optional
12dependency on another library, which is auto-detected. If that library
13has already been built when the software is building, the software will
14link to the built library and that library will be pulled into your
15image along with the new software even if you did not want the library.
16
17The :ref:`buildhistory <ref-classes-buildhistory>`
18class helps you maintain the quality of your build output. You
19can use the class to highlight unexpected and possibly unwanted changes
20in the build output. When you enable build history, it records
21information about the contents of each package and image and then
22commits that information to a local Git repository where you can examine
23the information.
24
25The remainder of this section describes the following:
26
27- :ref:`How you can enable and disable build history <dev-manual/build-quality:enabling and disabling build history>`
28
29- :ref:`How to understand what the build history contains <dev-manual/build-quality:understanding what the build history contains>`
30
31- :ref:`How to limit the information used for build history <dev-manual/build-quality:using build history to gather image information only>`
32
33- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/build-quality:examining build history information>`
34
35Enabling and Disabling Build History
36====================================
37
38Build history is disabled by default. To enable it, add the following
39:term:`INHERIT` statement and set the :term:`BUILDHISTORY_COMMIT` variable to
40"1" at the end of your ``conf/local.conf`` file found in the
41:term:`Build Directory`::
42
43 INHERIT += "buildhistory"
44 BUILDHISTORY_COMMIT = "1"
45
46Enabling build history as
47previously described causes the OpenEmbedded build system to collect
48build output information and commit it as a single commit to a local
49:ref:`overview-manual/development-environment:git` repository.
50
51.. note::
52
53 Enabling build history increases your build times slightly,
54 particularly for images, and increases the amount of disk space used
55 during the build.
56
57You can disable build history by removing the previous statements from
58your ``conf/local.conf`` file.
59
60Understanding What the Build History Contains
61=============================================
62
63Build history information is kept in ``${``\ :term:`TOPDIR`\ ``}/buildhistory``
64in the :term:`Build Directory` as defined by the :term:`BUILDHISTORY_DIR`
65variable. Here is an example abbreviated listing:
66
67.. image:: figures/buildhistory.png
68 :align: center
69 :width: 50%
70
71At the top level, there is a ``metadata-revs`` file that lists the
72revisions of the repositories for the enabled layers when the build was
73produced. The rest of the data splits into separate ``packages``,
74``images`` and ``sdk`` directories, the contents of which are described
75as follows.
76
77Build History Package Information
78---------------------------------
79
80The history for each package contains a text file that has name-value
81pairs with information about the package. For example,
82``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
83contains the following:
84
85.. code-block:: none
86
87 PV = 1.22.1
88 PR = r32
89 RPROVIDES =
90 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
91 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
92 PKGSIZE = 540168
93 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
94 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
95 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
96 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
97 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
98 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
99 /etc/busybox.links.nosuid /etc/busybox.links.suid
100
101Most of these
102name-value pairs correspond to variables used to produce the package.
103The exceptions are ``FILELIST``, which is the actual list of files in
104the package, and ``PKGSIZE``, which is the total size of files in the
105package in bytes.
106
107There is also a file that corresponds to the recipe from which the package
108came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
109
110.. code-block:: none
111
112 PV = 1.22.1
113 PR = r32
114 DEPENDS = initscripts kern-tools-native update-rc.d-native \
115 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
116 virtual/libc virtual/update-alternatives
117 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
118 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
119 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
120
121Finally, for those recipes fetched from a version control system (e.g.,
122Git), there is a file that lists source revisions that are specified in
123the recipe and the actual revisions used during the build. Listed
124and actual revisions might differ when
125:term:`SRCREV` is set to
126${:term:`AUTOREV`}. Here is an
127example assuming
128``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
129
130 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
131 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
132 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
133 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
134
135You can use the
136``buildhistory-collect-srcrevs`` command with the ``-a`` option to
137collect the stored :term:`SRCREV` values from build history and report them
138in a format suitable for use in global configuration (e.g.,
139``local.conf`` or a distro include file) to override floating
140:term:`AUTOREV` values to a fixed set of revisions. Here is some example
141output from this command::
142
143 $ buildhistory-collect-srcrevs -a
144 # all-poky-linux
145 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8"
146 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac"
147 # core2-64-poky-linux
148 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66"
149 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8"
150 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0"
151 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0"
152 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1"
153 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c"
154 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7"
155 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee"
156 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1"
157 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100"
158 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3"
159 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf"
160 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced"
161 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8"
162 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f"
163 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa"
164 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70"
165 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
166 # qemux86_64-poky-linux
167 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c"
168 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0"
169 # x86_64-linux
170 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66"
171 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c"
172 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87"
173 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57"
174 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb"
175 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b"
176 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
177
178.. note::
179
180 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
181
182 - By default, only values where the :term:`SRCREV` was not hardcoded
183 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
184 option to see all :term:`SRCREV` values.
185
186 - The output statements might not have any effect if overrides are
187 applied elsewhere in the build system configuration. Use the
188 ``-f`` option to add the ``forcevariable`` override to each output
189 line if you need to work around this restriction.
190
191 - The script does apply special handling when building for multiple
192 machines. However, the script does place a comment before each set
193 of values that specifies which triplet to which they belong as
194 previously shown (e.g., ``i586-poky-linux``).
195
196Build History Image Information
197-------------------------------
198
199The files produced for each image are as follows:
200
201- ``image-files:`` A directory containing selected files from the root
202 filesystem. The files are defined by
203 :term:`BUILDHISTORY_IMAGE_FILES`.
204
205- ``build-id.txt:`` Human-readable information about the build
206 configuration and metadata source revisions. This file contains the
207 full build header as printed by BitBake.
208
209- ``*.dot:`` Dependency graphs for the image that are compatible with
210 ``graphviz``.
211
212- ``files-in-image.txt:`` A list of files in the image with
213 permissions, owner, group, size, and symlink information.
214
215- ``image-info.txt:`` A text file containing name-value pairs with
216 information about the image. See the following listing example for
217 more information.
218
219- ``installed-package-names.txt:`` A list of installed packages by name
220 only.
221
222- ``installed-package-sizes.txt:`` A list of installed packages ordered
223 by size.
224
225- ``installed-packages.txt:`` A list of installed packages with full
226 package filenames.
227
228.. note::
229
230 Installed package information is able to be gathered and produced
231 even if package management is disabled for the final image.
232
233Here is an example of ``image-info.txt``:
234
235.. code-block:: none
236
237 DISTRO = poky
238 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be
239 USER_CLASSES = buildstats
240 IMAGE_CLASSES = qemuboot qemuboot license_image
241 IMAGE_FEATURES = debug-tweaks
242 IMAGE_LINGUAS =
243 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp
244 BAD_RECOMMENDATIONS =
245 NO_RECOMMENDATIONS =
246 PACKAGE_EXCLUDE =
247 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible;
248 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
249 IMAGESIZE = 9265
250
251Other than ``IMAGESIZE``,
252which is the total size of the files in the image in Kbytes, the
253name-value pairs are variables that may have influenced the content of
254the image. This information is often useful when you are trying to
255determine why a change in the package or file listings has occurred.
256
257Using Build History to Gather Image Information Only
258----------------------------------------------------
259
260As you can see, build history produces image information, including
261dependency graphs, so you can see why something was pulled into the
262image. If you are just interested in this information and not interested
263in collecting specific package or SDK information, you can enable
264writing only image information without any history by adding the
265following to your ``conf/local.conf`` file found in the
266:term:`Build Directory`::
267
268 INHERIT += "buildhistory"
269 BUILDHISTORY_COMMIT = "0"
270 BUILDHISTORY_FEATURES = "image"
271
272Here, you set the
273:term:`BUILDHISTORY_FEATURES`
274variable to use the image feature only.
275
276Build History SDK Information
277-----------------------------
278
279Build history collects similar information on the contents of SDKs (e.g.
280``bitbake -c populate_sdk imagename``) as compared to information it
281collects for images. Furthermore, this information differs depending on
282whether an extensible or standard SDK is being produced.
283
284The following list shows the files produced for SDKs:
285
286- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
287 owner, group, size, and symlink information. This list includes both
288 the host and target parts of the SDK.
289
290- ``sdk-info.txt:`` A text file containing name-value pairs with
291 information about the SDK. See the following listing example for more
292 information.
293
294- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
295 with information about task group sizes (e.g. :ref:`ref-tasks-populate_sysroot`
296 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
297 only when an extensible SDK is created.
298
299- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
300 with information for the shared-state packages and sizes in the SDK.
301 The ``sstate-package-sizes.txt`` file exists only when an extensible
302 SDK is created.
303
304- ``sdk-files:`` A folder that contains copies of the files mentioned
305 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
306 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
307 specific to the extensible SDK although you can set it differently if
308 you would like to pull in specific files from the standard SDK.
309
310 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
311 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
312 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
313 copied into the ``sdk-files`` directory.
314
315- The following information appears under each of the ``host`` and
316 ``target`` directories for the portions of the SDK that run on the
317 host and on the target, respectively:
318
319 .. note::
320
321 The following files for the most part are empty when producing an
322 extensible SDK because this type of SDK is not constructed from
323 packages as is the standard SDK.
324
325 - ``depends.dot:`` Dependency graph for the SDK that is compatible
326 with ``graphviz``.
327
328 - ``installed-package-names.txt:`` A list of installed packages by
329 name only.
330
331 - ``installed-package-sizes.txt:`` A list of installed packages
332 ordered by size.
333
334 - ``installed-packages.txt:`` A list of installed packages with full
335 package filenames.
336
337Here is an example of ``sdk-info.txt``:
338
339.. code-block:: none
340
341 DISTRO = poky
342 DISTRO_VERSION = 1.3+snapshot-20130327
343 SDK_NAME = poky-glibc-i686-arm
344 SDK_VERSION = 1.3+snapshot
345 SDKMACHINE =
346 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
347 BAD_RECOMMENDATIONS =
348 SDKSIZE = 352712
349
350Other than ``SDKSIZE``, which is
351the total size of the files in the SDK in Kbytes, the name-value pairs
352are variables that might have influenced the content of the SDK. This
353information is often useful when you are trying to determine why a
354change in the package or file listings has occurred.
355
356Examining Build History Information
357-----------------------------------
358
359You can examine build history output from the command line or from a web
360interface.
361
362To see any changes that have occurred (assuming you have
363:term:`BUILDHISTORY_COMMIT` = "1"),
364you can simply use any Git command that allows you to view the history
365of a repository. Here is one method::
366
367 $ git log -p
368
369You need to realize,
370however, that this method does show changes that are not significant
371(e.g. a package's size changing by a few bytes).
372
373There is a command-line tool called ``buildhistory-diff``, though,
374that queries the Git repository and prints just the differences that
375might be significant in human-readable form. Here is an example::
376
377 $ poky/poky/scripts/buildhistory-diff . HEAD^
378 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
379 /etc/anotherpkg.conf was added
380 /sbin/anotherpkg was added
381 * (installed-package-names.txt):
382 * anotherpkg was added
383 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
384 anotherpkg was added
385 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
386 * PR changed from "r0" to "r1"
387 * PV changed from "0.1.10" to "0.1.12"
388 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
389 * PR changed from "r0" to "r1"
390 * PV changed from "0.1.10" to "0.1.12"
391
392.. note::
393
394 The ``buildhistory-diff`` tool requires the ``GitPython``
395 package. Be sure to install it using Pip3 as follows::
396
397 $ pip3 install GitPython --user
398
399
400 Alternatively, you can install ``python3-git`` using the appropriate
401 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``).
402
403To see changes to the build history using a web interface, follow the
404instruction in the ``README`` file
405:yocto_git:`here </buildhistory-web/>`.
406
407Here is a sample screenshot of the interface:
408
409.. image:: figures/buildhistory-web.png
410 :width: 100%
411
diff --git a/documentation/dev-manual/building.rst b/documentation/dev-manual/building.rst
new file mode 100644
index 0000000000..c2cf3a35f3
--- /dev/null
+++ b/documentation/dev-manual/building.rst
@@ -0,0 +1,940 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Building
4********
5
6This section describes various build procedures, such as the steps
7needed for a simple build, building a target for multiple configurations,
8generating an image for more than one machine, and so forth.
9
10Building a Simple Image
11=======================
12
13In the development environment, you need to build an image whenever you
14change hardware support, add or change system libraries, or add or
15change services that have dependencies. There are several methods that allow
16you to build an image within the Yocto Project. This section presents
17the basic steps you need to build a simple image using BitBake from a
18build host running Linux.
19
20.. note::
21
22 - For information on how to build an image using
23 :term:`Toaster`, see the
24 :doc:`/toaster-manual/index`.
25
26 - For information on how to use ``devtool`` to build images, see the
27 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
28 section in the Yocto Project Application Development and the
29 Extensible Software Development Kit (eSDK) manual.
30
31 - For a quick example on how to build an image using the
32 OpenEmbedded build system, see the
33 :doc:`/brief-yoctoprojectqs/index` document.
34
35The build process creates an entire Linux distribution from source and
36places it in your :term:`Build Directory` under ``tmp/deploy/images``. For
37detailed information on the build process using BitBake, see the
38":ref:`overview-manual/concepts:images`" section in the Yocto Project Overview
39and Concepts Manual.
40
41The following figure and list overviews the build process:
42
43.. image:: figures/bitbake-build-flow.png
44 :width: 100%
45
461. *Set up Your Host Development System to Support Development Using the
47 Yocto Project*: See the ":doc:`start`" section for options on how to get a
48 build host ready to use the Yocto Project.
49
502. *Initialize the Build Environment:* Initialize the build environment
51 by sourcing the build environment script (i.e.
52 :ref:`structure-core-script`)::
53
54 $ source oe-init-build-env [build_dir]
55
56 When you use the initialization script, the OpenEmbedded build system
57 uses ``build`` as the default :term:`Build Directory` in your current work
58 directory. You can use a `build_dir` argument with the script to
59 specify a different :term:`Build Directory`.
60
61 .. note::
62
63 A common practice is to use a different :term:`Build Directory` for
64 different targets; for example, ``~/build/x86`` for a ``qemux86``
65 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
66 event, it's typically cleaner to locate the :term:`Build Directory`
67 somewhere outside of your source directory.
68
693. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
70 ``conf/local.conf`` configuration file, which is found in the
71 :term:`Build Directory`, is set up how you want it. This file defines many
72 aspects of the build environment including the target machine architecture
73 through the :term:`MACHINE` variable, the packaging format used during
74 the build (:term:`PACKAGE_CLASSES`), and a centralized tarball download
75 directory through the :term:`DL_DIR` variable.
76
774. *Build the Image:* Build the image using the ``bitbake`` command::
78
79 $ bitbake target
80
81 .. note::
82
83 For information on BitBake, see the :doc:`bitbake:index`.
84
85 The target is the name of the recipe you want to build. Common
86 targets are the images in ``meta/recipes-core/images``,
87 ``meta/recipes-sato/images``, and so forth all found in the
88 :term:`Source Directory`. Alternatively, the target
89 can be the name of a recipe for a specific piece of software such as
90 BusyBox. For more details about the images the OpenEmbedded build
91 system supports, see the
92 ":ref:`ref-manual/images:Images`" chapter in the Yocto
93 Project Reference Manual.
94
95 As an example, the following command builds the
96 ``core-image-minimal`` image::
97
98 $ bitbake core-image-minimal
99
100 Once an
101 image has been built, it often needs to be installed. The images and
102 kernels built by the OpenEmbedded build system are placed in the
103 :term:`Build Directory` in ``tmp/deploy/images``. For information on how to
104 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
105 :doc:`/sdk-manual/index` manual. For
106 information about how to install these images, see the documentation
107 for your particular board or machine.
108
109Building Images for Multiple Targets Using Multiple Configurations
110==================================================================
111
112You can use a single ``bitbake`` command to build multiple images or
113packages for different targets where each image or package requires a
114different configuration (multiple configuration builds). The builds, in
115this scenario, are sometimes referred to as "multiconfigs", and this
116section uses that term throughout.
117
118This section describes how to set up for multiple configuration builds
119and how to account for cross-build dependencies between the
120multiconfigs.
121
122Setting Up and Running a Multiple Configuration Build
123-----------------------------------------------------
124
125To accomplish a multiple configuration build, you must define each
126target's configuration separately using a parallel configuration file in
127the :term:`Build Directory` or configuration directory within a layer, and you
128must follow a required file hierarchy. Additionally, you must enable the
129multiple configuration builds in your ``local.conf`` file.
130
131Follow these steps to set up and execute multiple configuration builds:
132
133- *Create Separate Configuration Files*: You need to create a single
134 configuration file for each build target (each multiconfig).
135 The configuration definitions are implementation dependent but often
136 each configuration file will define the machine and the
137 temporary directory BitBake uses for the build. Whether the same
138 temporary directory (:term:`TMPDIR`) can be shared will depend on what is
139 similar and what is different between the configurations. Multiple MACHINE
140 targets can share the same (:term:`TMPDIR`) as long as the rest of the
141 configuration is the same, multiple DISTRO settings would need separate
142 (:term:`TMPDIR`) directories.
143
144 For example, consider a scenario with two different multiconfigs for the same
145 :term:`MACHINE`: "qemux86" built
146 for two distributions such as "poky" and "poky-lsb". In this case,
147 you would need to use the different :term:`TMPDIR`.
148
149 Here is an example showing the minimal statements needed in a
150 configuration file for a "qemux86" target whose temporary build
151 directory is ``tmpmultix86``::
152
153 MACHINE = "qemux86"
154 TMPDIR = "${TOPDIR}/tmpmultix86"
155
156 The location for these multiconfig configuration files is specific.
157 They must reside in the current :term:`Build Directory` in a sub-directory of
158 ``conf`` named ``multiconfig`` or within a layer's ``conf`` directory
159 under a directory named ``multiconfig``. Following is an example that defines
160 two configuration files for the "x86" and "arm" multiconfigs:
161
162 .. image:: figures/multiconfig_files.png
163 :align: center
164 :width: 50%
165
166 The usual :term:`BBPATH` search path is used to locate multiconfig files in
167 a similar way to other conf files.
168
169- *Add the BitBake Multi-configuration Variable to the Local
170 Configuration File*: Use the
171 :term:`BBMULTICONFIG`
172 variable in your ``conf/local.conf`` configuration file to specify
173 each multiconfig. Continuing with the example from the previous
174 figure, the :term:`BBMULTICONFIG` variable needs to enable two
175 multiconfigs: "x86" and "arm" by specifying each configuration file::
176
177 BBMULTICONFIG = "x86 arm"
178
179 .. note::
180
181 A "default" configuration already exists by definition. This
182 configuration is named: "" (i.e. empty string) and is defined by
183 the variables coming from your ``local.conf``
184 file. Consequently, the previous example actually adds two
185 additional configurations to your build: "arm" and "x86" along
186 with "".
187
188- *Launch BitBake*: Use the following BitBake command form to launch
189 the multiple configuration build::
190
191 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
192
193 For the example in this section, the following command applies::
194
195 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
196
197 The previous BitBake command builds a ``core-image-minimal`` image
198 that is configured through the ``x86.conf`` configuration file, a
199 ``core-image-sato`` image that is configured through the ``arm.conf``
200 configuration file and a ``core-image-base`` that is configured
201 through your ``local.conf`` configuration file.
202
203.. note::
204
205 Support for multiple configuration builds in the Yocto Project &DISTRO;
206 (&DISTRO_NAME;) Release does not include Shared State (sstate)
207 optimizations. Consequently, if a build uses the same object twice
208 in, for example, two different :term:`TMPDIR`
209 directories, the build either loads from an existing sstate cache for
210 that build at the start or builds the object fresh.
211
212Enabling Multiple Configuration Build Dependencies
213--------------------------------------------------
214
215Sometimes dependencies can exist between targets (multiconfigs) in a
216multiple configuration build. For example, suppose that in order to
217build a ``core-image-sato`` image for an "x86" multiconfig, the root
218filesystem of an "arm" multiconfig must exist. This dependency is
219essentially that the
220:ref:`ref-tasks-image` task in the
221``core-image-sato`` recipe depends on the completion of the
222:ref:`ref-tasks-rootfs` task of the
223``core-image-minimal`` recipe.
224
225To enable dependencies in a multiple configuration build, you must
226declare the dependencies in the recipe using the following statement
227form::
228
229 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
230
231To better show how to use this statement, consider the example scenario
232from the first paragraph of this section. The following statement needs
233to be added to the recipe that builds the ``core-image-sato`` image::
234
235 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
236
237In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
238task on which the :ref:`ref-tasks-image` task in the recipe depends is the
239:ref:`ref-tasks-rootfs` task from the ``core-image-minimal`` recipe associated
240with the "arm" multiconfig.
241
242Once you set up this dependency, you can build the "x86" multiconfig
243using a BitBake command as follows::
244
245 $ bitbake mc:x86:core-image-sato
246
247This command executes all the tasks needed to create the
248``core-image-sato`` image for the "x86" multiconfig. Because of the
249dependency, BitBake also executes through the :ref:`ref-tasks-rootfs` task for the
250"arm" multiconfig build.
251
252Having a recipe depend on the root filesystem of another build might not
253seem that useful. Consider this change to the statement in the
254``core-image-sato`` recipe::
255
256 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
257
258In this case, BitBake must
259create the ``core-image-minimal`` image for the "arm" build since the
260"x86" build depends on it.
261
262Because "x86" and "arm" are enabled for multiple configuration builds
263and have separate configuration files, BitBake places the artifacts for
264each build in the respective temporary build directories (i.e.
265:term:`TMPDIR`).
266
267Building an Initial RAM Filesystem (Initramfs) Image
268====================================================
269
270An initial RAM filesystem (:term:`Initramfs`) image provides a temporary root
271filesystem used for early system initialization, typically providing tools and
272loading modules needed to locate and mount the final root filesystem.
273
274Follow these steps to create an :term:`Initramfs` image:
275
2761. *Create the Initramfs Image Recipe:* You can reference the
277 ``core-image-minimal-initramfs.bb`` recipe found in the
278 ``meta/recipes-core`` directory of the :term:`Source Directory`
279 as an example from which to work.
280
2812. *Decide if You Need to Bundle the Initramfs Image Into the Kernel
282 Image:* If you want the :term:`Initramfs` image that is built to be bundled
283 in with the kernel image, set the :term:`INITRAMFS_IMAGE_BUNDLE`
284 variable to ``"1"`` in your ``local.conf`` configuration file and set the
285 :term:`INITRAMFS_IMAGE` variable in the recipe that builds the kernel image.
286
287 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the :term:`Initramfs`
288 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
289 :term:`Initramfs` image is then passed to the kernel's ``Makefile`` using the
290 :term:`CONFIG_INITRAMFS_SOURCE` variable, allowing the :term:`Initramfs`
291 image to be built into the kernel normally.
292
2933. *Optionally Add Items to the Initramfs Image Through the Initramfs
294 Image Recipe:* If you add items to the :term:`Initramfs` image by way of its
295 recipe, you should use :term:`PACKAGE_INSTALL` rather than
296 :term:`IMAGE_INSTALL`. :term:`PACKAGE_INSTALL` gives more direct control of
297 what is added to the image as compared to the defaults you might not
298 necessarily want that are set by the :ref:`image <ref-classes-image>`
299 or :ref:`core-image <ref-classes-core-image>` classes.
300
3014. *Build the Kernel Image and the Initramfs Image:* Build your kernel
302 image using BitBake. Because the :term:`Initramfs` image recipe is a
303 dependency of the kernel image, the :term:`Initramfs` image is built as well
304 and bundled with the kernel image if you used the
305 :term:`INITRAMFS_IMAGE_BUNDLE` variable described earlier.
306
307Bundling an Initramfs Image From a Separate Multiconfig
308-------------------------------------------------------
309
310There may be a case where we want to build an :term:`Initramfs` image which does not
311inherit the same distro policy as our main image, for example, we may want
312our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our :term:`Initramfs`
313image to keep a smaller footprint. However, by performing the steps mentioned
314above the :term:`Initramfs` image will inherit ``TCLIBC="glibc"`` without allowing us
315to override it.
316
317To achieve this, you need to perform some additional steps:
318
3191. *Create a multiconfig for your Initramfs image:* You can perform the steps
320 on ":ref:`dev-manual/building:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
321 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
322 contains the variables::
323
324 TMPDIR="${TOPDIR}/tmp-initramfscfg"
325 TCLIBC="musl"
326
3272. *Set additional Initramfs variables on your main configuration:*
328 Additionally, on your main configuration (``local.conf``) you need to set the
329 variables::
330
331 INITRAMFS_MULTICONFIG = "initramfscfg"
332 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
333
334 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
335 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
336 to be built coming from the ``initramfscfg`` multiconfig, and to let the
337 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
338
339 Building a system with such configuration will build the kernel using the
340 main configuration but the :ref:`ref-tasks-bundle_initramfs` task will grab the
341 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
342 instead, resulting in a musl based :term:`Initramfs` image bundled in the kernel
343 but a glibc based main image.
344
345 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
346 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
347
348
349Building a Tiny System
350======================
351
352Very small distributions have some significant advantages such as
353requiring less on-die or in-package memory (cheaper), better performance
354through efficient cache usage, lower power requirements due to less
355memory, faster boot times, and reduced development overhead. Some
356real-world examples where a very small distribution gives you distinct
357advantages are digital cameras, medical devices, and small headless
358systems.
359
360This section presents information that shows you how you can trim your
361distribution to even smaller sizes than the ``poky-tiny`` distribution,
362which is around 5 Mbytes, that can be built out-of-the-box using the
363Yocto Project.
364
365Tiny System Overview
366--------------------
367
368The following list presents the overall steps you need to consider and
369perform to create distributions with smaller root filesystems, achieve
370faster boot times, maintain your critical functionality, and avoid
371initial RAM disks:
372
373- :ref:`Determine your goals and guiding principles
374 <dev-manual/building:goals and guiding principles>`
375
376- :ref:`dev-manual/building:understand what contributes to your image size`
377
378- :ref:`Reduce the size of the root filesystem
379 <dev-manual/building:trim the root filesystem>`
380
381- :ref:`Reduce the size of the kernel <dev-manual/building:trim the kernel>`
382
383- :ref:`dev-manual/building:remove package management requirements`
384
385- :ref:`dev-manual/building:look for other ways to minimize size`
386
387- :ref:`dev-manual/building:iterate on the process`
388
389Goals and Guiding Principles
390----------------------------
391
392Before you can reach your destination, you need to know where you are
393going. Here is an example list that you can use as a guide when creating
394very small distributions:
395
396- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
397 less and a root filesystem that is 3 Mbytes or less).
398
399- Find the areas that are currently taking 90% of the space and
400 concentrate on reducing those areas.
401
402- Do not create any difficult "hacks" to achieve your goals.
403
404- Leverage the device-specific options.
405
406- Work in a separate layer so that you keep changes isolated. For
407 information on how to create layers, see the
408 ":ref:`dev-manual/layers:understanding and creating layers`" section.
409
410Understand What Contributes to Your Image Size
411----------------------------------------------
412
413It is easiest to have something to start with when creating your own
414distribution. You can use the Yocto Project out-of-the-box to create the
415``poky-tiny`` distribution. Ultimately, you will want to make changes in
416your own distribution that are likely modeled after ``poky-tiny``.
417
418.. note::
419
420 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
421 ``local.conf`` file to "poky-tiny" as described in the
422 ":ref:`dev-manual/custom-distribution:creating your own distribution`"
423 section.
424
425Understanding some memory concepts will help you reduce the system size.
426Memory consists of static, dynamic, and temporary memory. Static memory
427is the TEXT (code), DATA (initialized data in the code), and BSS
428(uninitialized data) sections. Dynamic memory represents memory that is
429allocated at runtime: stacks, hash tables, and so forth. Temporary
430memory is recovered after the boot process. This memory consists of
431memory used for decompressing the kernel and for the ``__init__``
432functions.
433
434To help you see where you currently are with kernel and root filesystem
435sizes, you can use two tools found in the :term:`Source Directory`
436in the
437``scripts/tiny/`` directory:
438
439- ``ksize.py``: Reports component sizes for the kernel build objects.
440
441- ``dirsize.py``: Reports component sizes for the root filesystem.
442
443This next tool and command help you organize configuration fragments and
444view file dependencies in a human-readable form:
445
446- ``merge_config.sh``: Helps you manage configuration files and
447 fragments within the kernel. With this tool, you can merge individual
448 configuration fragments together. The tool allows you to make
449 overrides and warns you of any missing configuration options. The
450 tool is ideal for allowing you to iterate on configurations, create
451 minimal configurations, and create configuration files for different
452 machines without having to duplicate your process.
453
454 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
455 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
456 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
457 directory.
458
459 For more information on configuration fragments, see the
460 ":ref:`kernel-dev/common:creating configuration fragments`"
461 section in the Yocto Project Linux Kernel Development Manual.
462
463- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
464 with these options brings up a Dependency Explorer from which you can
465 view file dependencies. Understanding these dependencies allows you
466 to make informed decisions when cutting out various pieces of the
467 kernel and root filesystem.
468
469Trim the Root Filesystem
470------------------------
471
472The root filesystem is made up of packages for booting, libraries, and
473applications. To change things, you can configure how the packaging
474happens, which changes the way you build them. You can also modify the
475filesystem itself or select a different filesystem.
476
477First, find out what is hogging your root filesystem by running the
478``dirsize.py`` script from your root directory::
479
480 $ cd root-directory-of-image
481 $ dirsize.py 100000 > dirsize-100k.log
482 $ cat dirsize-100k.log
483
484You can apply a filter to the script to ignore files
485under a certain size. The previous example filters out any files below
486100 Kbytes. The sizes reported by the tool are uncompressed, and thus
487will be smaller by a relatively constant factor in a compressed root
488filesystem. When you examine your log file, you can focus on areas of
489the root filesystem that take up large amounts of memory.
490
491You need to be sure that what you eliminate does not cripple the
492functionality you need. One way to see how packages relate to each other
493is by using the Dependency Explorer UI with the BitBake command::
494
495 $ cd image-directory
496 $ bitbake -u taskexp -g image
497
498Use the interface to
499select potential packages you wish to eliminate and see their dependency
500relationships.
501
502When deciding how to reduce the size, get rid of packages that result in
503minimal impact on the feature set. For example, you might not need a VGA
504display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
505instead of ``udev``.
506
507Use your ``local.conf`` file to make changes. For example, to eliminate
508``udev`` and ``glib``, set the following in the local configuration
509file::
510
511 VIRTUAL-RUNTIME_dev_manager = ""
512
513Finally, you should consider exactly the type of root filesystem you
514need to meet your needs while also reducing its size. For example,
515consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
516:term:`Initramfs` using ``initramfs``. Be aware that ``ext3`` requires a 1
517Mbyte journal. If you are okay with running read-only, you do not need
518this journal.
519
520.. note::
521
522 After each round of elimination, you need to rebuild your system and
523 then use the tools to see the effects of your reductions.
524
525Trim the Kernel
526---------------
527
528The kernel is built by including policies for hardware-independent
529aspects. What subsystems do you enable? For what architecture are you
530building? Which drivers do you build by default?
531
532.. note::
533
534 You can modify the kernel source if you want to help with boot time.
535
536Run the ``ksize.py`` script from the top-level Linux build directory to
537get an idea of what is making up the kernel::
538
539 $ cd top-level-linux-build-directory
540 $ ksize.py > ksize.log
541 $ cat ksize.log
542
543When you examine the log, you will see how much space is taken up with
544the built-in ``.o`` files for drivers, networking, core kernel files,
545filesystem, sound, and so forth. The sizes reported by the tool are
546uncompressed, and thus will be smaller by a relatively constant factor
547in a compressed kernel image. Look to reduce the areas that are large
548and taking up around the "90% rule."
549
550To examine, or drill down, into any particular area, use the ``-d``
551option with the script::
552
553 $ ksize.py -d > ksize.log
554
555Using this option
556breaks out the individual file information for each area of the kernel
557(e.g. drivers, networking, and so forth).
558
559Use your log file to see what you can eliminate from the kernel based on
560features you can let go. For example, if you are not going to need
561sound, you do not need any drivers that support sound.
562
563After figuring out what to eliminate, you need to reconfigure the kernel
564to reflect those changes during the next build. You could run
565``menuconfig`` and make all your changes at once. However, that makes it
566difficult to see the effects of your individual eliminations and also
567makes it difficult to replicate the changes for perhaps another target
568device. A better method is to start with no configurations using
569``allnoconfig``, create configuration fragments for individual changes,
570and then manage the fragments into a single configuration file using
571``merge_config.sh``. The tool makes it easy for you to iterate using the
572configuration change and build cycle.
573
574Each time you make configuration changes, you need to rebuild the kernel
575and check to see what impact your changes had on the overall size.
576
577Remove Package Management Requirements
578--------------------------------------
579
580Packaging requirements add size to the image. One way to reduce the size
581of the image is to remove all the packaging requirements from the image.
582This reduction includes both removing the package manager and its unique
583dependencies as well as removing the package management data itself.
584
585To eliminate all the packaging requirements for an image, be sure that
586"package-management" is not part of your
587:term:`IMAGE_FEATURES`
588statement for the image. When you remove this feature, you are removing
589the package manager as well as its dependencies from the root
590filesystem.
591
592Look for Other Ways to Minimize Size
593------------------------------------
594
595Depending on your particular circumstances, other areas that you can
596trim likely exist. The key to finding these areas is through tools and
597methods described here combined with experimentation and iteration. Here
598are a couple of areas to experiment with:
599
600- ``glibc``: In general, follow this process:
601
602 1. Remove ``glibc`` features from
603 :term:`DISTRO_FEATURES`
604 that you think you do not need.
605
606 2. Build your distribution.
607
608 3. If the build fails due to missing symbols in a package, determine
609 if you can reconfigure the package to not need those features. For
610 example, change the configuration to not support wide character
611 support as is done for ``ncurses``. Or, if support for those
612 characters is needed, determine what ``glibc`` features provide
613 the support and restore the configuration.
614
615 4. Rebuild and repeat the process.
616
617- ``busybox``: For BusyBox, use a process similar as described for
618 ``glibc``. A difference is you will need to boot the resulting system
619 to see if you are able to do everything you expect from the running
620 system. You need to be sure to integrate configuration fragments into
621 Busybox because BusyBox handles its own core features and then allows
622 you to add configuration fragments on top.
623
624Iterate on the Process
625----------------------
626
627If you have not reached your goals on system size, you need to iterate
628on the process. The process is the same. Use the tools and see just what
629is taking up 90% of the root filesystem and the kernel. Decide what you
630can eliminate without limiting your device beyond what you need.
631
632Depending on your system, a good place to look might be Busybox, which
633provides a stripped down version of Unix tools in a single, executable
634file. You might be able to drop virtual terminal services or perhaps
635ipv6.
636
637Building Images for More than One Machine
638=========================================
639
640A common scenario developers face is creating images for several
641different machines that use the same software environment. In this
642situation, it is tempting to set the tunings and optimization flags for
643each build specifically for the targeted hardware (i.e. "maxing out" the
644tunings). Doing so can considerably add to build times and package feed
645maintenance collectively for the machines. For example, selecting tunes
646that are extremely specific to a CPU core used in a system might enable
647some micro optimizations in GCC for that particular system but would
648otherwise not gain you much of a performance difference across the other
649systems as compared to using a more general tuning across all the builds
650(e.g. setting :term:`DEFAULTTUNE`
651specifically for each machine's build). Rather than "max out" each
652build's tunings, you can take steps that cause the OpenEmbedded build
653system to reuse software across the various machines where it makes
654sense.
655
656If build speed and package feed maintenance are considerations, you
657should consider the points in this section that can help you optimize
658your tunings to best consider build times and package feed maintenance.
659
660- *Share the :term:`Build Directory`:* If at all possible, share the
661 :term:`TMPDIR` across builds. The Yocto Project supports switching between
662 different :term:`MACHINE` values in the same :term:`TMPDIR`. This practice
663 is well supported and regularly used by developers when building for
664 multiple machines. When you use the same :term:`TMPDIR` for multiple
665 machine builds, the OpenEmbedded build system can reuse the existing native
666 and often cross-recipes for multiple machines. Thus, build time decreases.
667
668 .. note::
669
670 If :term:`DISTRO` settings change or fundamental configuration settings
671 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
672 Sharing :term:`TMPDIR` under these circumstances might work but since it is
673 not guaranteed, you should use a clean :term:`TMPDIR`.
674
675- *Enable the Appropriate Package Architecture:* By default, the
676 OpenEmbedded build system enables three levels of package
677 architectures: "all", "tune" or "package", and "machine". Any given
678 recipe usually selects one of these package architectures (types) for
679 its output. Depending for what a given recipe creates packages,
680 making sure you enable the appropriate package architecture can
681 directly impact the build time.
682
683 A recipe that just generates scripts can enable "all" architecture
684 because there are no binaries to build. To specifically enable "all"
685 architecture, be sure your recipe inherits the
686 :ref:`allarch <ref-classes-allarch>` class.
687 This class is useful for "all" architectures because it configures
688 many variables so packages can be used across multiple architectures.
689
690 If your recipe needs to generate packages that are machine-specific
691 or when one of the build or runtime dependencies is already
692 machine-architecture dependent, which makes your recipe also
693 machine-architecture dependent, make sure your recipe enables the
694 "machine" package architecture through the
695 :term:`MACHINE_ARCH`
696 variable::
697
698 PACKAGE_ARCH = "${MACHINE_ARCH}"
699
700 When you do not
701 specifically enable a package architecture through the
702 :term:`PACKAGE_ARCH`, The
703 OpenEmbedded build system defaults to the
704 :term:`TUNE_PKGARCH` setting::
705
706 PACKAGE_ARCH = "${TUNE_PKGARCH}"
707
708- *Choose a Generic Tuning File if Possible:* Some tunes are more
709 generic and can run on multiple targets (e.g. an ``armv5`` set of
710 packages could run on ``armv6`` and ``armv7`` processors in most
711 cases). Similarly, ``i486`` binaries could work on ``i586`` and
712 higher processors. You should realize, however, that advances on
713 newer processor versions would not be used.
714
715 If you select the same tune for several different machines, the
716 OpenEmbedded build system reuses software previously built, thus
717 speeding up the overall build time. Realize that even though a new
718 sysroot for each machine is generated, the software is not recompiled
719 and only one package feed exists.
720
721- *Manage Granular Level Packaging:* Sometimes there are cases where
722 injecting another level of package architecture beyond the three
723 higher levels noted earlier can be useful. For example, consider how
724 NXP (formerly Freescale) allows for the easy reuse of binary packages
725 in their layer
726 :yocto_git:`meta-freescale </meta-freescale/>`.
727 In this example, the
728 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
729 class shares GPU packages for i.MX53 boards because all boards share
730 the AMD GPU. The i.MX6-based boards can do the same because all
731 boards share the Vivante GPU. This class inspects the BitBake
732 datastore to identify if the package provides or depends on one of
733 the sub-architecture values. If so, the class sets the
734 :term:`PACKAGE_ARCH` value
735 based on the ``MACHINE_SUBARCH`` value. If the package does not
736 provide or depend on one of the sub-architecture values but it
737 matches a value in the machine-specific filter, it sets
738 :term:`MACHINE_ARCH`. This
739 behavior reduces the number of packages built and saves build time by
740 reusing binaries.
741
742- *Use Tools to Debug Issues:* Sometimes you can run into situations
743 where software is being rebuilt when you think it should not be. For
744 example, the OpenEmbedded build system might not be using shared
745 state between machines when you think it should be. These types of
746 situations are usually due to references to machine-specific
747 variables such as :term:`MACHINE`,
748 :term:`SERIAL_CONSOLES`,
749 :term:`XSERVER`,
750 :term:`MACHINE_FEATURES`,
751 and so forth in code that is supposed to only be tune-specific or
752 when the recipe depends
753 (:term:`DEPENDS`,
754 :term:`RDEPENDS`,
755 :term:`RRECOMMENDS`,
756 :term:`RSUGGESTS`, and so forth)
757 on some other recipe that already has
758 :term:`PACKAGE_ARCH` defined
759 as "${MACHINE_ARCH}".
760
761 .. note::
762
763 Patches to fix any issues identified are most welcome as these
764 issues occasionally do occur.
765
766 For such cases, you can use some tools to help you sort out the
767 situation:
768
769 - ``state-diff-machines.sh``*:* You can find this tool in the
770 ``scripts`` directory of the Source Repositories. See the comments
771 in the script for information on how to use the tool.
772
773 - *BitBake's "-S printdiff" Option:* Using this option causes
774 BitBake to try to establish the closest signature match it can
775 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
776 over the matches to determine the stamps and delta where these two
777 stamp trees diverge.
778
779Building Software from an External Source
780=========================================
781
782By default, the OpenEmbedded build system uses the :term:`Build Directory`
783when building source code. The build process involves fetching the source
784files, unpacking them, and then patching them if necessary before the build
785takes place.
786
787There are situations where you might want to build software from source
788files that are external to and thus outside of the OpenEmbedded build
789system. For example, suppose you have a project that includes a new BSP
790with a heavily customized kernel. And, you want to minimize exposing the
791build system to the development team so that they can focus on their
792project and maintain everyone's workflow as much as possible. In this
793case, you want a kernel source directory on the development machine
794where the development occurs. You want the recipe's
795:term:`SRC_URI` variable to point to
796the external directory and use it as is, not copy it.
797
798To build from software that comes from an external source, all you need to do
799is inherit the :ref:`externalsrc <ref-classes-externalsrc>` class and then set
800the :term:`EXTERNALSRC` variable to point to your external source code. Here
801are the statements to put in your ``local.conf`` file::
802
803 INHERIT += "externalsrc"
804 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
805
806This next example shows how to accomplish the same thing by setting
807:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
808
809 EXTERNALSRC = "path"
810 EXTERNALSRC_BUILD = "path"
811
812.. note::
813
814 In order for these settings to take effect, you must globally or
815 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
816 class.
817
818By default, :ref:`ref-classes-externalsrc` builds the source code in a
819directory separate from the external source directory as specified by
820:term:`EXTERNALSRC`. If you need
821to have the source built in the same directory in which it resides, or
822some other nominated directory, you can set
823:term:`EXTERNALSRC_BUILD`
824to point to that directory::
825
826 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
827
828Replicating a Build Offline
829===========================
830
831It can be useful to take a "snapshot" of upstream sources used in a
832build and then use that "snapshot" later to replicate the build offline.
833To do so, you need to first prepare and populate your downloads
834directory your "snapshot" of files. Once your downloads directory is
835ready, you can use it at any time and from any machine to replicate your
836build.
837
838Follow these steps to populate your Downloads directory:
839
8401. *Create a Clean Downloads Directory:* Start with an empty downloads
841 directory (:term:`DL_DIR`). You
842 start with an empty downloads directory by either removing the files
843 in the existing directory or by setting :term:`DL_DIR` to point to either
844 an empty location or one that does not yet exist.
845
8462. *Generate Tarballs of the Source Git Repositories:* Edit your
847 ``local.conf`` configuration file as follows::
848
849 DL_DIR = "/home/your-download-dir/"
850 BB_GENERATE_MIRROR_TARBALLS = "1"
851
852 During
853 the fetch process in the next step, BitBake gathers the source files
854 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
855 the
856 :term:`BB_GENERATE_MIRROR_TARBALLS`
857 variable for more information.
858
8593. *Populate Your Downloads Directory Without Building:* Use BitBake to
860 fetch your sources but inhibit the build::
861
862 $ bitbake target --runonly=fetch
863
864 The downloads directory (i.e. ``${DL_DIR}``) now has
865 a "snapshot" of the source files in the form of tarballs, which can
866 be used for the build.
867
8684. *Optionally Remove Any Git or other SCM Subdirectories From the
869 Downloads Directory:* If you want, you can clean up your downloads
870 directory by removing any Git or other Source Control Management
871 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
872 already contain these subdirectories.
873
874Once your downloads directory has everything it needs regarding source
875files, you can create your "own-mirror" and build your target.
876Understand that you can use the files to build the target offline from
877any machine and at any time.
878
879Follow these steps to build your target using the files in the downloads
880directory:
881
8821. *Using Local Files Only:* Inside your ``local.conf`` file, add the
883 :term:`SOURCE_MIRROR_URL` variable, inherit the
884 :ref:`own-mirrors <ref-classes-own-mirrors>` class, and use the
885 :term:`BB_NO_NETWORK` variable to your ``local.conf``::
886
887 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
888 INHERIT += "own-mirrors"
889 BB_NO_NETWORK = "1"
890
891 The :term:`SOURCE_MIRROR_URL` and :ref:`own-mirrors <ref-classes-own-mirrors>`
892 class set up the system to use the downloads directory as your "own
893 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
894 BitBake's fetching process in step 3 stays local, which means files
895 from your "own-mirror" are used.
896
8972. *Start With a Clean Build:* You can start with a clean build by
898 removing the ``${``\ :term:`TMPDIR`\ ``}`` directory or using a new
899 :term:`Build Directory`.
900
9013. *Build Your Target:* Use BitBake to build your target::
902
903 $ bitbake target
904
905 The build completes using the known local "snapshot" of source
906 files from your mirror. The resulting tarballs for your "snapshot" of
907 source files are in the downloads directory.
908
909 .. note::
910
911 The offline build does not work if recipes attempt to find the
912 latest version of software by setting
913 :term:`SRCREV` to
914 ``${``\ :term:`AUTOREV`\ ``}``::
915
916 SRCREV = "${AUTOREV}"
917
918 When a recipe sets :term:`SRCREV` to
919 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
920 attempt to determine the latest version of software from the SCM.
921 Typically, recipes that use :term:`AUTOREV` are custom or modified
922 recipes. Recipes that reside in public repositories usually do not
923 use :term:`AUTOREV`.
924
925 If you do have recipes that use :term:`AUTOREV`, you can take steps to
926 still use the recipes in an offline build. Do the following:
927
928 1. Use a configuration generated by enabling :ref:`build
929 history <dev-manual/build-quality:maintaining build output quality>`.
930
931 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
932 stored :term:`SRCREV` values from the build's history. For more
933 information on collecting these values, see the
934 ":ref:`dev-manual/build-quality:build history package information`"
935 section.
936
937 3. Once you have the correct source revisions, you can modify
938 those recipes to set :term:`SRCREV` to specific versions of the
939 software.
940
diff --git a/documentation/dev-manual/changes.rst b/documentation/dev-manual/changes.rst
new file mode 100644
index 0000000000..8ccbf0d7ee
--- /dev/null
+++ b/documentation/dev-manual/changes.rst
@@ -0,0 +1,525 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Making Changes to the Yocto Project
4***********************************
5
6Because the Yocto Project is an open-source, community-based project,
7you can effect changes to the project. This section presents procedures
8that show you how to submit a defect against the project and how to
9submit a change.
10
11Submitting a Defect Against the Yocto Project
12=============================================
13
14Use the Yocto Project implementation of
15`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
16against the Yocto Project. For additional information on this
17implementation of Bugzilla see the ":ref:`Yocto Project
18Bugzilla <resources-bugtracker>`" section in the
19Yocto Project Reference Manual. For more detail on any of the following
20steps, see the Yocto Project
21:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`.
22
23Use the following general steps to submit a bug:
24
251. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
26
272. Click "File a Bug" to enter a new bug.
28
293. Choose the appropriate "Classification", "Product", and "Component"
30 for which the bug was found. Bugs for the Yocto Project fall into
31 one of several classifications, which in turn break down into
32 several products and components. For example, for a bug against the
33 ``meta-intel`` layer, you would choose "Build System, Metadata &
34 Runtime", "BSPs", and "bsps-meta-intel", respectively.
35
364. Choose the "Version" of the Yocto Project for which you found the
37 bug (e.g. &DISTRO;).
38
395. Determine and select the "Severity" of the bug. The severity
40 indicates how the bug impacted your work.
41
426. Choose the "Hardware" that the bug impacts.
43
447. Choose the "Architecture" that the bug impacts.
45
468. Choose a "Documentation change" item for the bug. Fixing a bug might
47 or might not affect the Yocto Project documentation. If you are
48 unsure of the impact to the documentation, select "Don't Know".
49
509. Provide a brief "Summary" of the bug. Try to limit your summary to
51 just a line or two and be sure to capture the essence of the bug.
52
5310. Provide a detailed "Description" of the bug. You should provide as
54 much detail as you can about the context, behavior, output, and so
55 forth that surrounds the bug. You can even attach supporting files
56 for output from logs by using the "Add an attachment" button.
57
5811. Click the "Submit Bug" button submit the bug. A new Bugzilla number
59 is assigned to the bug and the defect is logged in the bug tracking
60 system.
61
62Once you file a bug, the bug is processed by the Yocto Project Bug
63Triage Team and further details concerning the bug are assigned (e.g.
64priority and owner). You are the "Submitter" of the bug and any further
65categorization, progress, or comments on the bug result in Bugzilla
66sending you an automated email concerning the particular change or
67progress to the bug.
68
69Submitting a Change to the Yocto Project
70========================================
71
72Contributions to the Yocto Project and OpenEmbedded are very welcome.
73Because the system is extremely configurable and flexible, we recognize
74that developers will want to extend, configure or optimize it for their
75specific uses.
76
77The Yocto Project uses a mailing list and a patch-based workflow that is
78similar to the Linux kernel but contains important differences. In
79general, there is a mailing list through which you can submit patches. You
80should send patches to the appropriate mailing list so that they can be
81reviewed and merged by the appropriate maintainer. The specific mailing
82list you need to use depends on the location of the code you are
83changing. Each component (e.g. layer) should have a ``README`` file that
84indicates where to send the changes and which process to follow.
85
86You can send the patch to the mailing list using whichever approach you
87feel comfortable with to generate the patch. Once sent, the patch is
88usually reviewed by the community at large. If somebody has concerns
89with the patch, they will usually voice their concern over the mailing
90list. If a patch does not receive any negative reviews, the maintainer
91of the affected layer typically takes the patch, tests it, and then
92based on successful testing, merges the patch.
93
94The "poky" repository, which is the Yocto Project's reference build
95environment, is a hybrid repository that contains several individual
96pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
97the combo-layer tool. The upstream location used for submitting changes
98varies by component:
99
100- *Core Metadata:* Send your patch to the
101 :oe_lists:`openembedded-core </g/openembedded-core>`
102 mailing list. For example, a change to anything under the ``meta`` or
103 ``scripts`` directories should be sent to this mailing list.
104
105- *BitBake:* For changes to BitBake (i.e. anything under the
106 ``bitbake`` directory), send your patch to the
107 :oe_lists:`bitbake-devel </g/bitbake-devel>`
108 mailing list.
109
110- *"meta-\*" trees:* These trees contain Metadata. Use the
111 :yocto_lists:`poky </g/poky>` mailing list.
112
113- *Documentation*: For changes to the Yocto Project documentation, use the
114 :yocto_lists:`docs </g/docs>` mailing list.
115
116For changes to other layers hosted in the Yocto Project source
117repositories (i.e. ``yoctoproject.org``) and tools use the
118:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
119
120.. note::
121
122 Sometimes a layer's documentation specifies to use a particular
123 mailing list. If so, use that list.
124
125For additional recipes that do not fit into the core Metadata, you
126should determine which layer the recipe should go into and submit the
127change in the manner recommended by the documentation (e.g. the
128``README`` file) supplied with the layer. If in doubt, please ask on the
129Yocto general mailing list or on the openembedded-devel mailing list.
130
131You can also push a change upstream and request a maintainer to pull the
132change into the component's upstream repository. You do this by pushing
133to a contribution repository that is upstream. See the
134":ref:`overview-manual/development-environment:git workflows and the yocto project`"
135section in the Yocto Project Overview and Concepts Manual for additional
136concepts on working in the Yocto Project development environment.
137
138Maintainers commonly use ``-next`` branches to test submissions prior to
139merging patches. Thus, you can get an idea of the status of a patch based on
140whether the patch has been merged into one of these branches. The commonly
141used testing branches for OpenEmbedded-Core are as follows:
142
143- *openembedded-core "master-next" branch:* This branch is part of the
144 :oe_git:`openembedded-core </openembedded-core/>` repository and contains
145 proposed changes to the core metadata.
146
147- *poky "master-next" branch:* This branch is part of the
148 :yocto_git:`poky </poky/>` repository and combines proposed
149 changes to BitBake, the core metadata and the poky distro.
150
151Similarly, stable branches maintained by the project may have corresponding
152``-next`` branches which collect proposed changes. For example,
153``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next``
154branches in both the "openembdedded-core" and "poky" repositories.
155
156Other layers may have similar testing branches but there is no formal
157requirement or standard for these so please check the documentation for the
158layers you are contributing to.
159
160The following sections provide procedures for submitting a change.
161
162Preparing Changes for Submission
163--------------------------------
164
1651. *Make Your Changes Locally:* Make your changes in your local Git
166 repository. You should make small, controlled, isolated changes.
167 Keeping changes small and isolated aids review, makes
168 merging/rebasing easier and keeps the change history clean should
169 anyone need to refer to it in future.
170
1712. *Stage Your Changes:* Stage your changes by using the ``git add``
172 command on each file you changed.
173
1743. *Commit Your Changes:* Commit the change by using the ``git commit``
175 command. Make sure your commit information follows standards by
176 following these accepted conventions:
177
178 - Be sure to include a "Signed-off-by:" line in the same style as
179 required by the Linux kernel. This can be done by using the
180 ``git commit -s`` command. Adding this line signifies that you,
181 the submitter, have agreed to the Developer's Certificate of
182 Origin 1.1 as follows:
183
184 .. code-block:: none
185
186 Developer's Certificate of Origin 1.1
187
188 By making a contribution to this project, I certify that:
189
190 (a) The contribution was created in whole or in part by me and I
191 have the right to submit it under the open source license
192 indicated in the file; or
193
194 (b) The contribution is based upon previous work that, to the best
195 of my knowledge, is covered under an appropriate open source
196 license and I have the right under that license to submit that
197 work with modifications, whether created in whole or in part
198 by me, under the same open source license (unless I am
199 permitted to submit under a different license), as indicated
200 in the file; or
201
202 (c) The contribution was provided directly to me by some other
203 person who certified (a), (b) or (c) and I have not modified
204 it.
205
206 (d) I understand and agree that this project and the contribution
207 are public and that a record of the contribution (including all
208 personal information I submit with it, including my sign-off) is
209 maintained indefinitely and may be redistributed consistent with
210 this project or the open source license(s) involved.
211
212 - Provide a single-line summary of the change and, if more
213 explanation is needed, provide more detail in the body of the
214 commit. This summary is typically viewable in the "shortlist" of
215 changes. Thus, providing something short and descriptive that
216 gives the reader a summary of the change is useful when viewing a
217 list of many commits. You should prefix this short description
218 with the recipe name (if changing a recipe), or else with the
219 short form path to the file being changed.
220
221 - For the body of the commit message, provide detailed information
222 that describes what you changed, why you made the change, and the
223 approach you used. It might also be helpful if you mention how you
224 tested the change. Provide as much detail as you can in the body
225 of the commit message.
226
227 .. note::
228
229 You do not need to provide a more detailed explanation of a
230 change if the change is minor to the point of the single line
231 summary providing all the information.
232
233 - If the change addresses a specific bug or issue that is associated
234 with a bug-tracking ID, include a reference to that ID in your
235 detailed description. For example, the Yocto Project uses a
236 specific convention for bug references --- any commit that addresses
237 a specific bug should use the following form for the detailed
238 description. Be sure to use the actual bug-tracking ID from
239 Bugzilla for bug-id::
240
241 Fixes [YOCTO #bug-id]
242
243 detailed description of change
244
245Using Email to Submit a Patch
246-----------------------------
247
248Depending on the components changed, you need to submit the email to a
249specific mailing list. For some guidance on which mailing list to use,
250see the
251:ref:`list <dev-manual/changes:submitting a change to the yocto project>`
252at the beginning of this section. For a description of all the available
253mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
254Yocto Project Reference Manual.
255
256Here is the general procedure on how to submit a patch through email
257without using the scripts once the steps in
258:ref:`dev-manual/changes:preparing changes for submission` have been followed:
259
2601. *Format the Commit:* Format the commit into an email message. To
261 format commits, use the ``git format-patch`` command. When you
262 provide the command, you must include a revision list or a number of
263 patches as part of the command. For example, either of these two
264 commands takes your most recent single commit and formats it as an
265 email message in the current directory::
266
267 $ git format-patch -1
268
269 or ::
270
271 $ git format-patch HEAD~
272
273 After the command is run, the current directory contains a numbered
274 ``.patch`` file for the commit.
275
276 If you provide several commits as part of the command, the
277 ``git format-patch`` command produces a series of numbered files in
278 the current directory – one for each commit. If you have more than
279 one patch, you should also use the ``--cover`` option with the
280 command, which generates a cover letter as the first "patch" in the
281 series. You can then edit the cover letter to provide a description
282 for the series of patches. For information on the
283 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
284 using the ``man git-format-patch`` command.
285
286 .. note::
287
288 If you are or will be a frequent contributor to the Yocto Project
289 or to OpenEmbedded, you might consider requesting a contrib area
290 and the necessary associated rights.
291
2922. *Send the patches via email:* Send the patches to the recipients and
293 relevant mailing lists by using the ``git send-email`` command.
294
295 .. note::
296
297 In order to use ``git send-email``, you must have the proper Git packages
298 installed on your host.
299 For Ubuntu, Debian, and Fedora the package is ``git-email``.
300
301 The ``git send-email`` command sends email by using a local or remote
302 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
303 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
304 file. If you are submitting patches through email only, it is very
305 important that you submit them without any whitespace or HTML
306 formatting that either you or your mailer introduces. The maintainer
307 that receives your patches needs to be able to save and apply them
308 directly from your emails. A good way to verify that what you are
309 sending will be applicable by the maintainer is to do a dry run and
310 send them to yourself and then save and apply them as the maintainer
311 would.
312
313 The ``git send-email`` command is the preferred method for sending
314 your patches using email since there is no risk of compromising
315 whitespace in the body of the message, which can occur when you use
316 your own mail client. The command also has several options that let
317 you specify recipients and perform further editing of the email
318 message. For information on how to use the ``git send-email``
319 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
320 ``man git-send-email`` command.
321
322The Yocto Project uses a `Patchwork instance <https://patchwork.openembedded.org/>`__
323to track the status of patches submitted to the various mailing lists and to
324support automated patch testing. Each submitted patch is checked for common
325mistakes and deviations from the expected patch format and submitters are
326notified by patchtest if such mistakes are found. This process helps to
327reduce the burden of patch review on maintainers.
328
329.. note::
330
331 This system is imperfect and changes can sometimes get lost in the flow.
332 Asking about the status of a patch or change is reasonable if the change
333 has been idle for a while with no feedback.
334
335Using Scripts to Push a Change Upstream and Request a Pull
336----------------------------------------------------------
337
338For larger patch series it is preferable to send a pull request which not
339only includes the patch but also a pointer to a branch that can be pulled
340from. This involves making a local branch for your changes, pushing this
341branch to an accessible repository and then using the ``create-pull-request``
342and ``send-pull-request`` scripts from openembedded-core to create and send a
343patch series with a link to the branch for review.
344
345Follow this procedure to push a change to an upstream "contrib" Git
346repository once the steps in :ref:`dev-manual/changes:preparing changes for submission` have
347been followed:
348
349.. note::
350
351 You can find general Git information on how to push a change upstream
352 in the
353 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
354
3551. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
356 permissions to push to an upstream contrib repository, push the
357 change to that repository::
358
359 $ git push upstream_remote_repo local_branch_name
360
361 For example, suppose you have permissions to push
362 into the upstream ``meta-intel-contrib`` repository and you are
363 working in a local branch named `your_name`\ ``/README``. The following
364 command pushes your local commits to the ``meta-intel-contrib``
365 upstream repository and puts the commit in a branch named
366 `your_name`\ ``/README``::
367
368 $ git push meta-intel-contrib your_name/README
369
3702. *Determine Who to Notify:* Determine the maintainer or the mailing
371 list that you need to notify for the change.
372
373 Before submitting any change, you need to be sure who the maintainer
374 is or what mailing list that you need to notify. Use either these
375 methods to find out:
376
377 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
378 located in the :term:`Source Directory` at
379 ``meta/conf/distro/include``, to see who is responsible for code.
380
381 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can
382 enter the following command to bring up a short list of all
383 commits against a specific file::
384
385 git shortlog -- filename
386
387 Just provide the name of the file for which you are interested. The
388 information returned is not ordered by history but does include a
389 list of everyone who has committed grouped by name. From the list,
390 you can see who is responsible for the bulk of the changes against
391 the file.
392
393 - *Examine the List of Mailing Lists:* For a list of the Yocto
394 Project and related mailing lists, see the ":ref:`Mailing
395 lists <resources-mailinglist>`" section in
396 the Yocto Project Reference Manual.
397
3983. *Make a Pull Request:* Notify the maintainer or the mailing list that
399 you have pushed a change by making a pull request.
400
401 The Yocto Project provides two scripts that conveniently let you
402 generate and send pull requests to the Yocto Project. These scripts
403 are ``create-pull-request`` and ``send-pull-request``. You can find
404 these scripts in the ``scripts`` directory within the
405 :term:`Source Directory` (e.g.
406 ``poky/scripts``).
407
408 Using these scripts correctly formats the requests without
409 introducing any whitespace or HTML formatting. The maintainer that
410 receives your patches either directly or through the mailing list
411 needs to be able to save and apply them directly from your emails.
412 Using these scripts is the preferred method for sending patches.
413
414 First, create the pull request. For example, the following command
415 runs the script, specifies the upstream repository in the contrib
416 directory into which you pushed the change, and provides a subject
417 line in the created patch files::
418
419 $ poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
420
421 Running this script forms ``*.patch`` files in a folder named
422 ``pull-``\ `PID` in the current directory. One of the patch files is a
423 cover letter.
424
425 Before running the ``send-pull-request`` script, you must edit the
426 cover letter patch to insert information about your change. After
427 editing the cover letter, send the pull request. For example, the
428 following command runs the script and specifies the patch directory
429 and email address. In this example, the email address is a mailing
430 list::
431
432 $ poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@lists.yoctoproject.org
433
434 You need to follow the prompts as the script is interactive.
435
436 .. note::
437
438 For help on using these scripts, simply provide the ``-h``
439 argument as follows::
440
441 $ poky/scripts/create-pull-request -h
442 $ poky/scripts/send-pull-request -h
443
444Responding to Patch Review
445--------------------------
446
447You may get feedback on your submitted patches from other community members
448or from the automated patchtest service. If issues are identified in your
449patch then it is usually necessary to address these before the patch will be
450accepted into the project. In this case you should amend the patch according
451to the feedback and submit an updated version to the relevant mailing list,
452copying in the reviewers who provided feedback to the previous version of the
453patch.
454
455The patch should be amended using ``git commit --amend`` or perhaps ``git
456rebase`` for more expert git users. You should also modify the ``[PATCH]``
457tag in the email subject line when sending the revised patch to mark the new
458iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be
459done by passing the ``-v`` argument to ``git format-patch`` with a version
460number.
461
462Lastly please ensure that you also test your revised changes. In particular
463please don't just edit the patch file written out by ``git format-patch`` and
464resend it.
465
466Submitting Changes to Stable Release Branches
467---------------------------------------------
468
469The process for proposing changes to a Yocto Project stable branch differs
470from the steps described above. Changes to a stable branch must address
471identified bugs or CVEs and should be made carefully in order to avoid the
472risk of introducing new bugs or breaking backwards compatibility. Typically
473bug fixes must already be accepted into the master branch before they can be
474backported to a stable branch unless the bug in question does not affect the
475master branch or the fix on the master branch is unsuitable for backporting.
476
477The list of stable branches along with the status and maintainer for each
478branch can be obtained from the
479:yocto_wiki:`Releases wiki page </Releases>`.
480
481.. note::
482
483 Changes will not typically be accepted for branches which are marked as
484 End-Of-Life (EOL).
485
486With this in mind, the steps to submit a change for a stable branch are as
487follows:
488
4891. *Identify the bug or CVE to be fixed:* This information should be
490 collected so that it can be included in your submission.
491
492 See :ref:`dev-manual/vulnerabilities:checking for vulnerabilities`
493 for details about CVE tracking.
494
4952. *Check if the fix is already present in the master branch:* This will
496 result in the most straightforward path into the stable branch for the
497 fix.
498
499 a. *If the fix is present in the master branch --- submit a backport request
500 by email:* You should send an email to the relevant stable branch
501 maintainer and the mailing list with details of the bug or CVE to be
502 fixed, the commit hash on the master branch that fixes the issue and
503 the stable branches which you would like this fix to be backported to.
504
505 b. *If the fix is not present in the master branch --- submit the fix to the
506 master branch first:* This will ensure that the fix passes through the
507 project's usual patch review and test processes before being accepted.
508 It will also ensure that bugs are not left unresolved in the master
509 branch itself. Once the fix is accepted in the master branch a backport
510 request can be submitted as above.
511
512 c. *If the fix is unsuitable for the master branch --- submit a patch
513 directly for the stable branch:* This method should be considered as a
514 last resort. It is typically necessary when the master branch is using
515 a newer version of the software which includes an upstream fix for the
516 issue or when the issue has been fixed on the master branch in a way
517 that introduces backwards incompatible changes. In this case follow the
518 steps in :ref:`dev-manual/changes:preparing changes for submission` and
519 :ref:`dev-manual/changes:using email to submit a patch` but modify the subject header of your patch
520 email to include the name of the stable branch which you are
521 targetting. This can be done using the ``--subject-prefix`` argument to
522 ``git format-patch``, for example to submit a patch to the dunfell
523 branch use
524 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``.
525
diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
deleted file mode 100644
index 3ba64e1477..0000000000
--- a/documentation/dev-manual/common-tasks.rst
+++ /dev/null
@@ -1,11861 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
21":ref:`overview-manual/yp-intro:the yocto project layer model`"
22section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
27.. note::
28
29 It is very easy to create your own layers to use with the OpenEmbedded
30 build system, as the Yocto Project ships with tools that speed up creating
31 layers. This section describes the steps you perform by hand to create
32 layers so that you can better understand them. For information about the
33 layer-creation tools, see the
34 ":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
35 section in the Yocto Project Board Support Package (BSP) Developer's
36 Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
37 section further down in this manual.
38
39Follow these general steps to create your layer without using tools:
40
411. *Check Existing Layers:* Before creating a new layer, you should be
42 sure someone has not already created a layer containing the Metadata
43 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
44 for a list of layers from the OpenEmbedded community that can be used in
45 the Yocto Project. You could find a layer that is identical or close
46 to what you need.
47
482. *Create a Directory:* Create the directory for your layer. When you
49 create the layer, be sure to create the directory in an area not
50 associated with the Yocto Project :term:`Source Directory`
51 (e.g. the cloned ``poky`` repository).
52
53 While not strictly required, prepend the name of the directory with
54 the string "meta-". For example::
55
56 meta-mylayer
57 meta-GUI_xyz
58 meta-mymachine
59
60 With rare exceptions, a layer's name follows this form::
61
62 meta-root_name
63
64 Following this layer naming convention can save
65 you trouble later when tools, components, or variables "assume" your
66 layer name begins with "meta-". A notable example is in configuration
67 files as shown in the following step where layer names without the
68 "meta-" string are appended to several variables used in the
69 configuration.
70
713. *Create a Layer Configuration File:* Inside your new layer folder,
72 you need to create a ``conf/layer.conf`` file. It is easiest to take
73 an existing layer configuration file and copy that to your layer's
74 ``conf`` directory and then modify the file as needed.
75
76 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
77 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
78 demonstrates the required syntax. For your layer, you need to replace
79 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
80 for a layer named "meta-machinexyz")::
81
82 # We have a conf and classes directory, add to BBPATH
83 BBPATH .= ":${LAYERDIR}"
84
85 # We have recipes-* directories, add to BBFILES
86 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
87 ${LAYERDIR}/recipes-*/*/*.bbappend"
88
89 BBFILE_COLLECTIONS += "yoctobsp"
90 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
91 BBFILE_PRIORITY_yoctobsp = "5"
92 LAYERVERSION_yoctobsp = "4"
93 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
94
95 Following is an explanation of the layer configuration file:
96
97 - :term:`BBPATH`: Adds the layer's
98 root directory to BitBake's search path. Through the use of the
99 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
100 configuration files, and files that are included with ``include``
101 and ``require`` statements. For these cases, BitBake uses the
102 first file that matches the name found in :term:`BBPATH`. This is
103 similar to the way the ``PATH`` variable is used for binaries. It
104 is recommended, therefore, that you use unique class and
105 configuration filenames in your custom layer.
106
107 - :term:`BBFILES`: Defines the
108 location for all recipes in the layer.
109
110 - :term:`BBFILE_COLLECTIONS`:
111 Establishes the current layer through a unique identifier that is
112 used throughout the OpenEmbedded build system to refer to the
113 layer. In this example, the identifier "yoctobsp" is the
114 representation for the container layer named "meta-yocto-bsp".
115
116 - :term:`BBFILE_PATTERN`:
117 Expands immediately during parsing to provide the directory of the
118 layer.
119
120 - :term:`BBFILE_PRIORITY`:
121 Establishes a priority to use for recipes in the layer when the
122 OpenEmbedded build finds recipes of the same name in different
123 layers.
124
125 - :term:`LAYERVERSION`:
126 Establishes a version number for the layer. You can use this
127 version number to specify this exact version of the layer as a
128 dependency when using the
129 :term:`LAYERDEPENDS`
130 variable.
131
132 - :term:`LAYERDEPENDS`:
133 Lists all layers on which this layer depends (if any).
134
135 - :term:`LAYERSERIES_COMPAT`:
136 Lists the :yocto_wiki:`Yocto Project </Releases>`
137 releases for which the current version is compatible. This
138 variable is a good way to indicate if your particular layer is
139 current.
140
1414. *Add Content:* Depending on the type of layer, add the content. If
142 the layer adds support for a machine, add the machine configuration
143 in a ``conf/machine/`` file within the layer. If the layer adds
144 distro policy, add the distro configuration in a ``conf/distro/``
145 file within the layer. If the layer introduces new recipes, put the
146 recipes you need in ``recipes-*`` subdirectories within the layer.
147
148 .. note::
149
150 For an explanation of layer hierarchy that is compliant with the
151 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
152 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
153
1545. *Optionally Test for Compatibility:* If you want permission to use
155 the Yocto Project Compatibility logo with your layer or application
156 that uses your layer, perform the steps to apply for compatibility.
157 See the
158 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
159 section for more information.
160
161Following Best Practices When Creating Layers
162---------------------------------------------
163
164To create layers that are easier to maintain and that will not impact
165builds for other machines, you should consider the information in the
166following list:
167
168- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
169 Configuration:* In other words, do not copy an entire recipe into
170 your layer and then modify it. Rather, use an append file
171 (``.bbappend``) to override only those parts of the original recipe
172 you need to modify.
173
174- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
175 for each recipe that uses an include file. Or, if you are introducing
176 a new recipe that requires the included file, use the path relative
177 to the original layer directory to refer to the file. For example,
178 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
179 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
180 the include file, it could indicate a deficiency in the include file
181 in the layer to which it originally belongs. If this is the case, you
182 should try to address that deficiency instead of overlaying the
183 include file. For example, you could address this by getting the
184 maintainer of the include file to add a variable or variables to make
185 it easy to override the parts needing to be overridden.
186
187- *Structure Your Layers:* Proper use of overrides within append files
188 and placement of machine-specific files within your layer can ensure
189 that a build is not using the wrong Metadata and negatively impacting
190 a build for a different machine. Following are some examples:
191
192 - *Modify Variables to Support a Different Machine:* Suppose you
193 have a layer named ``meta-one`` that adds support for building
194 machine "one". To do so, you use an append file named
195 ``base-files.bbappend`` and create a dependency on "foo" by
196 altering the :term:`DEPENDS`
197 variable::
198
199 DEPENDS = "foo"
200
201 The dependency is created during any
202 build that includes the layer ``meta-one``. However, you might not
203 want this dependency for all machines. For example, suppose you
204 are building for machine "two" but your ``bblayers.conf`` file has
205 the ``meta-one`` layer included. During the build, the
206 ``base-files`` for machine "two" will also have the dependency on
207 ``foo``.
208
209 To make sure your changes apply only when building machine "one",
210 use a machine override with the :term:`DEPENDS` statement::
211
212 DEPENDS:one = "foo"
213
214 You should follow the same strategy when using ``:append``
215 and ``:prepend`` operations::
216
217 DEPENDS:append:one = " foo"
218 DEPENDS:prepend:one = "foo "
219
220 As an actual example, here's a
221 snippet from the generic kernel include file ``linux-yocto.inc``,
222 wherein the kernel compile and link options are adjusted in the
223 case of a subset of the supported architectures::
224
225 DEPENDS:append:aarch64 = " libgcc"
226 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
227 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
228
229 DEPENDS:append:nios2 = " libgcc"
230 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
231 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
232
233 DEPENDS:append:arc = " libgcc"
234 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
235 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
236
237 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
238
239 - *Place Machine-Specific Files in Machine-Specific Locations:* When
240 you have a base recipe, such as ``base-files.bb``, that contains a
241 :term:`SRC_URI` statement to a
242 file, you can use an append file to cause the build to use your
243 own version of the file. For example, an append file in your layer
244 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
245 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
246
247 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
248
249 The build for machine "one" will pick up your machine-specific file as
250 long as you have the file in
251 ``meta-one/recipes-core/base-files/base-files/``. However, if you
252 are building for a different machine and the ``bblayers.conf``
253 file includes the ``meta-one`` layer and the location of your
254 machine-specific file is the first location where that file is
255 found according to :term:`FILESPATH`, builds for all machines will
256 also use that machine-specific file.
257
258 You can make sure that a machine-specific file is used for a
259 particular machine by putting the file in a subdirectory specific
260 to the machine. For example, rather than placing the file in
261 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
262 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
263 Not only does this make sure the file is used only when building
264 for machine "one", but the build process locates the file more
265 quickly.
266
267 In summary, you need to place all files referenced from
268 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
269 order to restrict those files to machine-specific builds.
270
271- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
272 permission to use the Yocto Project Compatibility logo with your
273 layer or application that uses your layer, perform the steps to apply
274 for compatibility. See the
275 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
276 section for more information.
277
278- *Follow the Layer Naming Convention:* Store custom layers in a Git
279 repository that use the ``meta-layer_name`` format.
280
281- *Group Your Layers Locally:* Clone your repository alongside other
282 cloned ``meta`` directories from the :term:`Source Directory`.
283
284Making Sure Your Layer is Compatible With Yocto Project
285-------------------------------------------------------
286
287When you create a layer used with the Yocto Project, it is advantageous
288to make sure that the layer interacts well with existing Yocto Project
289layers (i.e. the layer is compatible with the Yocto Project). Ensuring
290compatibility makes the layer easy to be consumed by others in the Yocto
291Project community and could allow you permission to use the Yocto
292Project Compatible Logo.
293
294.. note::
295
296 Only Yocto Project member organizations are permitted to use the
297 Yocto Project Compatible Logo. The logo is not available for general
298 use. For information on how to become a Yocto Project member
299 organization, see the :yocto_home:`Yocto Project Website <>`.
300
301The Yocto Project Compatibility Program consists of a layer application
302process that requests permission to use the Yocto Project Compatibility
303Logo for your layer and application. The process consists of two parts:
304
3051. Successfully passing a script (``yocto-check-layer``) that when run
306 against your layer, tests it against constraints based on experiences
307 of how layers have worked in the real world and where pitfalls have
308 been found. Getting a "PASS" result from the script is required for
309 successful compatibility registration.
310
3112. Completion of an application acceptance form, which you can find at
312 :yocto_home:`/webform/yocto-project-compatible-registration`.
313
314To be granted permission to use the logo, you need to satisfy the
315following:
316
317- Be able to check the box indicating that you got a "PASS" when
318 running the script against your layer.
319
320- Answer "Yes" to the questions on the form or have an acceptable
321 explanation for any questions answered "No".
322
323- Be a Yocto Project Member Organization.
324
325The remainder of this section presents information on the registration
326form and on the ``yocto-check-layer`` script.
327
328Yocto Project Compatible Program Application
329~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
330
331Use the form to apply for your layer's approval. Upon successful
332application, you can use the Yocto Project Compatibility Logo with your
333layer and the application that uses your layer.
334
335To access the form, use this link:
336:yocto_home:`/webform/yocto-project-compatible-registration`.
337Follow the instructions on the form to complete your application.
338
339The application consists of the following sections:
340
341- *Contact Information:* Provide your contact information as the fields
342 require. Along with your information, provide the released versions
343 of the Yocto Project for which your layer is compatible.
344
345- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
346 items in the checklist. There is space at the bottom of the form for
347 any explanations for items for which you answered "No".
348
349- *Recommendations:* Provide answers for the questions regarding Linux
350 kernel use and build success.
351
352``yocto-check-layer`` Script
353~~~~~~~~~~~~~~~~~~~~~~~~~~~~
354
355The ``yocto-check-layer`` script provides you a way to assess how
356compatible your layer is with the Yocto Project. You should run this
357script prior to using the form to apply for compatibility as described
358in the previous section. You need to achieve a "PASS" result in order to
359have your application form successfully processed.
360
361The script divides tests into three areas: COMMON, BSP, and DISTRO. For
362example, given a distribution layer (DISTRO), the layer must pass both
363the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
364layer, the layer must pass the COMMON and BSP set of tests.
365
366To execute the script, enter the following commands from your build
367directory::
368
369 $ source oe-init-build-env
370 $ yocto-check-layer your_layer_directory
371
372Be sure to provide the actual directory for your
373layer as part of the command.
374
375Entering the command causes the script to determine the type of layer
376and then to execute a set of specific tests against the layer. The
377following list overviews the test:
378
379- ``common.test_readme``: Tests if a ``README`` file exists in the
380 layer and the file is not empty.
381
382- ``common.test_parse``: Tests to make sure that BitBake can parse the
383 files without error (i.e. ``bitbake -p``).
384
385- ``common.test_show_environment``: Tests that the global or per-recipe
386 environment is in order without errors (i.e. ``bitbake -e``).
387
388- ``common.test_world``: Verifies that ``bitbake world`` works.
389
390- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
391 layers do not come with recipes that change signatures.
392
393- ``common.test_layerseries_compat``: Verifies layer compatibility is
394 set properly.
395
396- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
397 configurations.
398
399- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
400 set the machine when the layer is added.
401
402- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
403 regardless of which machine is selected.
404
405- ``bsp.test_machine_signatures``: Verifies that building for a
406 particular machine affects only the signature of tasks specific to
407 that machine.
408
409- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
410 distro configurations.
411
412- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
413 does not set the distribution when the layer is added.
414
415Enabling Your Layer
416-------------------
417
418Before the OpenEmbedded build system can use your new layer, you need to
419enable it. To enable your layer, simply add your layer's path to the
420:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
421found in the :term:`Build Directory`. The following example shows how to
422enable your new ``meta-mylayer`` layer (note how your new layer exists
423outside of the official ``poky`` repository which you would have checked
424out earlier)::
425
426 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
427 # changes incompatibly
428 POKY_BBLAYERS_CONF_VERSION = "2"
429 BBPATH = "${TOPDIR}"
430 BBFILES ?= ""
431 BBLAYERS ?= " \
432 /home/user/poky/meta \
433 /home/user/poky/meta-poky \
434 /home/user/poky/meta-yocto-bsp \
435 /home/user/mystuff/meta-mylayer \
436 "
437
438BitBake parses each ``conf/layer.conf`` file from the top down as
439specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
440file. During the processing of each ``conf/layer.conf`` file, BitBake
441adds the recipes, classes and configurations contained within the
442particular layer to the source directory.
443
444Appending Other Layers Metadata With Your Layer
445-----------------------------------------------
446
447A recipe that appends Metadata to another recipe is called a BitBake
448append file. A BitBake append file uses the ``.bbappend`` file type
449suffix, while the corresponding recipe to which Metadata is being
450appended uses the ``.bb`` file type suffix.
451
452You can use a ``.bbappend`` file in your layer to make additions or
453changes to the content of another layer's recipe without having to copy
454the other layer's recipe into your layer. Your ``.bbappend`` file
455resides in your layer, while the main ``.bb`` recipe file to which you
456are appending Metadata resides in a different layer.
457
458Being able to append information to an existing recipe not only avoids
459duplication, but also automatically applies recipe changes from a
460different layer into your layer. If you were copying recipes, you would
461have to manually merge changes as they occur.
462
463When you create an append file, you must use the same root name as the
464corresponding recipe file. For example, the append file
465``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
466means the original recipe and append filenames are version
467number-specific. If the corresponding recipe is renamed to update to a
468newer version, you must also rename and possibly update the
469corresponding ``.bbappend`` as well. During the build process, BitBake
470displays an error on starting if it detects a ``.bbappend`` file that
471does not have a corresponding recipe with a matching name. See the
472:term:`BB_DANGLINGAPPENDS_WARNONLY`
473variable for information on how to handle this error.
474
475Overlaying a File Using Your Layer
476~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
477
478As an example, consider the main formfactor recipe and a corresponding
479formfactor append file both from the :term:`Source Directory`.
480Here is the main
481formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
482the "meta" layer at ``meta/recipes-bsp/formfactor``::
483
484 SUMMARY = "Device formfactor information"
485 DESCRIPTION = "A formfactor configuration file provides information about the \
486 target hardware for which the image is being built and information that the \
487 build system cannot obtain from other sources such as the kernel."
488 SECTION = "base"
489 LICENSE = "MIT"
490 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
491 PR = "r45"
492
493 SRC_URI = "file://config file://machconfig"
494 S = "${WORKDIR}"
495
496 PACKAGE_ARCH = "${MACHINE_ARCH}"
497 INHIBIT_DEFAULT_DEPS = "1"
498
499 do_install() {
500 # Install file only if it has contents
501 install -d ${D}${sysconfdir}/formfactor/
502 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
503 if [ -s "${S}/machconfig" ]; then
504 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
505 fi
506 }
507
508In the main recipe, note the :term:`SRC_URI`
509variable, which tells the OpenEmbedded build system where to find files
510during the build.
511
512Following is the append file, which is named ``formfactor_0.0.bbappend``
513and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
514file is in the layer at ``recipes-bsp/formfactor``::
515
516 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
517
518By default, the build system uses the
519:term:`FILESPATH` variable to
520locate files. This append file extends the locations by setting the
521:term:`FILESEXTRAPATHS`
522variable. Setting this variable in the ``.bbappend`` file is the most
523reliable and recommended method for adding directories to the search
524path used by the build system to find files.
525
526The statement in this example extends the directories to include
527``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
528which resolves to a directory named ``formfactor`` in the same directory
529in which the append file resides (i.e.
530``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
531have the supporting directory structure set up that will contain any
532files or patches you will be including from the layer.
533
534Using the immediate expansion assignment operator ``:=`` is important
535because of the reference to :term:`THISDIR`. The trailing colon character is
536important as it ensures that items in the list remain colon-separated.
537
538.. note::
539
540 BitBake automatically defines the :term:`THISDIR` variable. You should
541 never set this variable yourself. Using ":prepend" as part of the
542 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
543 paths in the final list.
544
545 Also, not all append files add extra files. Many append files simply
546 allow to add build options (e.g. ``systemd``). For these cases, your
547 append file would not even use the :term:`FILESEXTRAPATHS` statement.
548
549The end result of this ``.bbappend`` file is that on a Raspberry Pi, where
550``rpi`` will exist in the list of :term:`OVERRIDES`, the file
551``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be
552used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in
553:ref:`ref-tasks-install` will return true, and the file will be installed.
554
555Installing Additional Files Using Your Layer
556~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
557
558As another example, consider the main ``xserver-xf86-config`` recipe and a
559corresponding ``xserver-xf86-config`` append file both from the :term:`Source
560Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named
561``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at
562``meta/recipes-graphics/xorg-xserver``::
563
564 SUMMARY = "X.Org X server configuration file"
565 HOMEPAGE = "http://www.x.org"
566 SECTION = "x11/base"
567 LICENSE = "MIT"
568 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
569 PR = "r33"
570
571 SRC_URI = "file://xorg.conf"
572
573 S = "${WORKDIR}"
574
575 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf"
576
577 PACKAGE_ARCH = "${MACHINE_ARCH}"
578 ALLOW_EMPTY:${PN} = "1"
579
580 do_install () {
581 if test -s ${WORKDIR}/xorg.conf; then
582 install -d ${D}/${sysconfdir}/X11
583 install -m 0644 ${WORKDIR}/xorg.conf ${D}/${sysconfdir}/X11/
584 fi
585 }
586
587Following is the append file, which is named ``xserver-xf86-config_%.bbappend``
588and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
589file is in the layer at ``recipes-graphics/xorg-xserver``::
590
591 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
592
593 SRC_URI:append:rpi = " \
594 file://xorg.conf.d/98-pitft.conf \
595 file://xorg.conf.d/99-calibration.conf \
596 "
597 do_install:append:rpi () {
598 PITFT="${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}"
599 if [ "${PITFT}" = "1" ]; then
600 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/
601 install -m 0644 ${WORKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
602 install -m 0644 ${WORKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
603 fi
604 }
605
606 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*"
607
608Building off of the previous example, we once again are setting the
609:term:`FILESEXTRAPATHS` variable. In this case we are also using
610:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in
611the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a
612check for an additional :term:`MACHINE_FEATURES` that if set will cause these
613additional files to be installed. These additional files are listed in
614:term:`FILES` so that they will be packaged.
615
616Prioritizing Your Layer
617-----------------------
618
619Each layer is assigned a priority value. Priority values control which
620layer takes precedence if there are recipe files with the same name in
621multiple layers. For these cases, the recipe file from the layer with a
622higher priority number takes precedence. Priority values also affect the
623order in which multiple ``.bbappend`` files for the same recipe are
624applied. You can either specify the priority manually, or allow the
625build system to calculate it based on the layer's dependencies.
626
627To specify the layer's priority manually, use the
628:term:`BBFILE_PRIORITY`
629variable and append the layer's root name::
630
631 BBFILE_PRIORITY_mylayer = "1"
632
633.. note::
634
635 It is possible for a recipe with a lower version number
636 :term:`PV` in a layer that has a higher
637 priority to take precedence.
638
639 Also, the layer priority does not currently affect the precedence
640 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
641 might address this.
642
643Managing Layers
644---------------
645
646You can use the BitBake layer management tool ``bitbake-layers`` to
647provide a view into the structure of recipes across a multi-layer
648project. Being able to generate output that reports on configured layers
649with their paths and priorities and on ``.bbappend`` files and their
650applicable recipes can help to reveal potential problems.
651
652For help on the BitBake layer management tool, use the following
653command::
654
655 $ bitbake-layers --help
656
657The following list describes the available commands:
658
659- ``help:`` Displays general help or help on a specified command.
660
661- ``show-layers:`` Shows the current configured layers.
662
663- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
664 when a recipe with the same name exists in another layer that has a
665 higher layer priority.
666
667- ``show-recipes:`` Lists available recipes and the layers that
668 provide them.
669
670- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
671 which they apply.
672
673- ``show-cross-depends:`` Lists dependency relationships between
674 recipes that cross layer boundaries.
675
676- ``add-layer:`` Adds a layer to ``bblayers.conf``.
677
678- ``remove-layer:`` Removes a layer from ``bblayers.conf``
679
680- ``flatten:`` Flattens the layer configuration into a separate
681 output directory. Flattening your layer configuration builds a
682 "flattened" directory that contains the contents of all layers, with
683 any overlayed recipes removed and any ``.bbappend`` files appended to
684 the corresponding recipes. You might have to perform some manual
685 cleanup of the flattened layer as follows:
686
687 - Non-recipe files (such as patches) are overwritten. The flatten
688 command shows a warning for these files.
689
690 - Anything beyond the normal layer setup has been added to the
691 ``layer.conf`` file. Only the lowest priority layer's
692 ``layer.conf`` is used.
693
694 - Overridden and appended items from ``.bbappend`` files need to be
695 cleaned up. The contents of each ``.bbappend`` end up in the
696 flattened recipe. However, if there are appended or changed
697 variable values, you need to tidy these up yourself. Consider the
698 following example. Here, the ``bitbake-layers`` command adds the
699 line ``#### bbappended ...`` so that you know where the following
700 lines originate::
701
702 ...
703 DESCRIPTION = "A useful utility"
704 ...
705 EXTRA_OECONF = "--enable-something"
706 ...
707
708 #### bbappended from meta-anotherlayer ####
709
710 DESCRIPTION = "Customized utility"
711 EXTRA_OECONF += "--enable-somethingelse"
712
713
714 Ideally, you would tidy up these utilities as follows::
715
716 ...
717 DESCRIPTION = "Customized utility"
718 ...
719 EXTRA_OECONF = "--enable-something --enable-somethingelse"
720 ...
721
722- ``layerindex-fetch``: Fetches a layer from a layer index, along
723 with its dependent layers, and adds the layers to the
724 ``conf/bblayers.conf`` file.
725
726- ``layerindex-show-depends``: Finds layer dependencies from the
727 layer index.
728
729- ``save-build-conf``: Saves the currently active build configuration
730 (``conf/local.conf``, ``conf/bblayers.conf``) as a template into a layer.
731 This template can later be used for setting up builds via :term:``TEMPLATECONF``.
732 For information about saving and using configuration templates, see
733 ":ref:`dev-manual/common-tasks:creating a custom template configuration directory`".
734
735- ``create-layer``: Creates a basic layer.
736
737- ``create-layers-setup``: Writes out a configuration file and/or a script that
738 can replicate the directory structure and revisions of the layers in a current build.
739 For more information, see ":ref:`dev-manual/common-tasks:saving and restoring the layers setup`".
740
741Creating a General Layer Using the ``bitbake-layers`` Script
742------------------------------------------------------------
743
744The ``bitbake-layers`` script with the ``create-layer`` subcommand
745simplifies creating a new general layer.
746
747.. note::
748
749 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
750 section in the Yocto
751 Project Board Specific (BSP) Developer's Guide.
752
753 - In order to use a layer with the OpenEmbedded build system, you
754 need to add the layer to your ``bblayers.conf`` configuration
755 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
756 section for more information.
757
758The default mode of the script's operation with this subcommand is to
759create a layer with the following:
760
761- A layer priority of 6.
762
763- A ``conf`` subdirectory that contains a ``layer.conf`` file.
764
765- A ``recipes-example`` subdirectory that contains a further
766 subdirectory named ``example``, which contains an ``example.bb``
767 recipe file.
768
769- A ``COPYING.MIT``, which is the license statement for the layer. The
770 script assumes you want to use the MIT license, which is typical for
771 most layers, for the contents of the layer itself.
772
773- A ``README`` file, which is a file describing the contents of your
774 new layer.
775
776In its simplest form, you can use the following command form to create a
777layer. The command creates a layer whose name corresponds to
778"your_layer_name" in the current directory::
779
780 $ bitbake-layers create-layer your_layer_name
781
782As an example, the following command creates a layer named ``meta-scottrif``
783in your home directory::
784
785 $ cd /usr/home
786 $ bitbake-layers create-layer meta-scottrif
787 NOTE: Starting bitbake server...
788 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
789
790If you want to set the priority of the layer to other than the default
791value of "6", you can either use the ``--priority`` option or you
792can edit the
793:term:`BBFILE_PRIORITY` value
794in the ``conf/layer.conf`` after the script creates it. Furthermore, if
795you want to give the example recipe file some name other than the
796default, you can use the ``--example-recipe-name`` option.
797
798The easiest way to see how the ``bitbake-layers create-layer`` command
799works is to experiment with the script. You can also read the usage
800information by entering the following::
801
802 $ bitbake-layers create-layer --help
803 NOTE: Starting bitbake server...
804 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
805 [--example-recipe-name EXAMPLERECIPE]
806 layerdir
807
808 Create a basic layer
809
810 positional arguments:
811 layerdir Layer directory to create
812
813 optional arguments:
814 -h, --help show this help message and exit
815 --priority PRIORITY, -p PRIORITY
816 Layer directory to create
817 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
818 Filename of the example recipe
819
820Adding a Layer Using the ``bitbake-layers`` Script
821--------------------------------------------------
822
823Once you create your general layer, you must add it to your
824``bblayers.conf`` file. Adding the layer to this configuration file
825makes the OpenEmbedded build system aware of your layer so that it can
826search it for metadata.
827
828Add your layer by using the ``bitbake-layers add-layer`` command::
829
830 $ bitbake-layers add-layer your_layer_name
831
832Here is an example that adds a
833layer named ``meta-scottrif`` to the configuration file. Following the
834command that adds the layer is another ``bitbake-layers`` command that
835shows the layers that are in your ``bblayers.conf`` file::
836
837 $ bitbake-layers add-layer meta-scottrif
838 NOTE: Starting bitbake server...
839 Parsing recipes: 100% |##########################################################| Time: 0:00:49
840 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
841 $ bitbake-layers show-layers
842 NOTE: Starting bitbake server...
843 layer path priority
844 ==========================================================================
845 meta /home/scottrif/poky/meta 5
846 meta-poky /home/scottrif/poky/meta-poky 5
847 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
848 workspace /home/scottrif/poky/build/workspace 99
849 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
850
851
852Adding the layer to this file
853enables the build system to locate the layer during the build.
854
855.. note::
856
857 During a build, the OpenEmbedded build system looks in the layers
858 from the top of the list down to the bottom in that order.
859
860Saving and restoring the layers setup
861-------------------------------------
862
863Once you have a working build with the correct set of layers, it is beneficial
864to capture the layer setup --- what they are, which repositories they come from
865and which SCM revisions they're at --- into a configuration file, so that this
866setup can be easily replicated later, perhaps on a different machine. Here's
867how to do this::
868
869 $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/
870 NOTE: Starting bitbake server...
871 NOTE: Created /srv/work/alex/meta-alex/setup-layers.json
872 NOTE: Created /srv/work/alex/meta-alex/setup-layers
873
874The tool needs a single argument which tells where to place the output, consisting
875of a json formatted layer configuration, and a ``setup-layers`` script that can use that configuration
876to restore the layers in a different location, or on a different host machine. The argument
877can point to a custom layer (which is then deemed a "bootstrap" layer that needs to be
878checked out first), or into a completely independent location.
879
880The replication of the layers is performed by running the ``setup-layers`` script provided
881above:
882
8831. Clone the bootstrap layer or some other repository to obtain
884 the json config and the setup script that can use it.
885
8862. Run the script directly with no options::
887
888 alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers
889 Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override.
890
891 Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master
892 Running 'git init -q /srv/work/alex/my-build/meta-intel'
893 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel
894 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel
895 Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel
896
897 Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers
898 Running 'git init -q /srv/work/alex/my-build/poky'
899 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky
900 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky
901 Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky
902 Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky
903 Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky
904
905.. note::
906 This will work to update an existing checkout as well.
907
908.. note::
909 The script is self-sufficient and requires only python3
910 and git on the build machine.
911
912.. note::
913 Both the ``create-layers-setup`` and the ``setup-layers`` provided several additional options
914 that customize their behavior - you are welcome to study them via ``--help`` command line parameter.
915
916Customizing Images
917==================
918
919You can customize images to satisfy particular requirements. This
920section describes several methods and provides guidelines for each.
921
922Customizing Images Using ``local.conf``
923---------------------------------------
924
925Probably the easiest way to customize an image is to add a package by
926way of the ``local.conf`` configuration file. Because it is limited to
927local use, this method generally only allows you to add packages and is
928not as flexible as creating your own customized image. When you add
929packages using local variables this way, you need to realize that these
930variable changes are in effect for every build and consequently affect
931all images, which might not be what you require.
932
933To add a package to your image using the local configuration file, use
934the :term:`IMAGE_INSTALL` variable with the ``:append`` operator::
935
936 IMAGE_INSTALL:append = " strace"
937
938Use of the syntax is important; specifically, the leading space
939after the opening quote and before the package name, which is
940``strace`` in this example. This space is required since the ``:append``
941operator does not add the space.
942
943Furthermore, you must use ``:append`` instead of the ``+=`` operator if
944you want to avoid ordering issues. The reason for this is because doing
945so unconditionally appends to the variable and avoids ordering problems
946due to the variable being set in image recipes and ``.bbclass`` files
947with operators like ``?=``. Using ``:append`` ensures the operation
948takes effect.
949
950As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all
951images. It is possible to extend the syntax so that the variable applies
952to a specific image only. Here is an example::
953
954 IMAGE_INSTALL:append:pn-core-image-minimal = " strace"
955
956This example adds ``strace`` to the ``core-image-minimal`` image only.
957
958You can add packages using a similar approach through the
959:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
960``core-image-*`` images are affected.
961
962Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
963-------------------------------------------------------------------------------
964
965Another method for customizing your image is to enable or disable
966high-level image features by using the
967:term:`IMAGE_FEATURES` and
968:term:`EXTRA_IMAGE_FEATURES`
969variables. Although the functions for both variables are nearly
970equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
971a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
972``local.conf`` file, which is found in the :term:`Build Directory`.
973
974To understand how these features work, the best reference is
975:ref:`meta/classes-recipe/image.bbclass <ref-classes-image>`.
976This class lists out the available
977:term:`IMAGE_FEATURES` of which most map to package groups while some, such
978as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
979configuration settings.
980
981In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
982variable and then maps or configures the feature accordingly. Based on
983this information, the build system automatically adds the appropriate
984packages or configurations to the
985:term:`IMAGE_INSTALL` variable.
986Effectively, you are enabling extra features by extending the class or
987creating a custom class for use with specialized image ``.bb`` files.
988
989Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
990configuration file. Using a separate area from which to enable features
991with this variable helps you avoid overwriting the features in the image
992recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
993:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
994``meta/conf/bitbake.conf``.
995
996To illustrate how you can use these variables to modify your image,
997consider an example that selects the SSH server. The Yocto Project ships
998with two SSH servers you can use with your images: Dropbear and OpenSSH.
999Dropbear is a minimal SSH server appropriate for resource-constrained
1000environments, while OpenSSH is a well-known standard SSH server
1001implementation. By default, the ``core-image-sato`` image is configured
1002to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
1003images both include OpenSSH. The ``core-image-minimal`` image does not
1004contain an SSH server.
1005
1006You can customize your image and change these defaults. Edit the
1007:term:`IMAGE_FEATURES` variable in your recipe or use the
1008:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
1009configures the image you are working with to include
1010``ssh-server-dropbear`` or ``ssh-server-openssh``.
1011
1012.. note::
1013
1014 See the ":ref:`ref-manual/features:image features`" section in the Yocto
1015 Project Reference Manual for a complete list of image features that ship
1016 with the Yocto Project.
1017
1018Customizing Images Using Custom .bb Files
1019-----------------------------------------
1020
1021You can also customize an image by creating a custom recipe that defines
1022additional software as part of the image. The following example shows
1023the form for the two lines you need::
1024
1025 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
1026 inherit core-image
1027
1028Defining the software using a custom recipe gives you total control over
1029the contents of the image. It is important to use the correct names of
1030packages in the :term:`IMAGE_INSTALL` variable. You must use the
1031OpenEmbedded notation and not the Debian notation for the names (e.g.
1032``glibc-dev`` instead of ``libc6-dev``).
1033
1034The other method for creating a custom image is to base it on an
1035existing image. For example, if you want to create an image based on
1036``core-image-sato`` but add the additional package ``strace`` to the
1037image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
1038``.bb`` and add the following line to the end of the copy::
1039
1040 IMAGE_INSTALL += "strace"
1041
1042Customizing Images Using Custom Package Groups
1043----------------------------------------------
1044
1045For complex custom images, the best approach for customizing an image is
1046to create a custom package group recipe that is used to build the image
1047or images. A good example of a package group recipe is
1048``meta/recipes-core/packagegroups/packagegroup-base.bb``.
1049
1050If you examine that recipe, you see that the :term:`PACKAGES` variable lists
1051the package group packages to produce. The ``inherit packagegroup``
1052statement sets appropriate default values and automatically adds
1053``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
1054package specified in the :term:`PACKAGES` statement.
1055
1056.. note::
1057
1058 The ``inherit packagegroup`` line should be located near the top of the
1059 recipe, certainly before the :term:`PACKAGES` statement.
1060
1061For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
1062and :term:`RRECOMMENDS` entries to provide a list of packages the parent
1063task package should contain. You can see examples of these further down
1064in the ``packagegroup-base.bb`` recipe.
1065
1066Here is a short, fabricated example showing the same basic pieces for a
1067hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
1068the variable :term:`PN` is the standard way to abbreviate the reference to
1069the full packagegroup name ``packagegroup-custom``::
1070
1071 DESCRIPTION = "My Custom Package Groups"
1072
1073 inherit packagegroup
1074
1075 PACKAGES = "\
1076 ${PN}-apps \
1077 ${PN}-tools \
1078 "
1079
1080 RDEPENDS:${PN}-apps = "\
1081 dropbear \
1082 portmap \
1083 psplash"
1084
1085 RDEPENDS:${PN}-tools = "\
1086 oprofile \
1087 oprofileui-server \
1088 lttng-tools"
1089
1090 RRECOMMENDS:${PN}-tools = "\
1091 kernel-module-oprofile"
1092
1093In the previous example, two package group packages are created with
1094their dependencies and their recommended package dependencies listed:
1095``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1096build an image using these package group packages, you need to add
1097``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
1098:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
1099areas of this section.
1100
1101Customizing an Image Hostname
1102-----------------------------
1103
1104By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1105is the same as the machine name. For example, if
1106:term:`MACHINE` equals "qemux86", the
1107configured hostname written to ``/etc/hostname`` is "qemux86".
1108
1109You can customize this name by altering the value of the "hostname"
1110variable in the ``base-files`` recipe using either an append file or a
1111configuration file. Use the following in an append file::
1112
1113 hostname = "myhostname"
1114
1115Use the following in a configuration file::
1116
1117 hostname:pn-base-files = "myhostname"
1118
1119Changing the default value of the variable "hostname" can be useful in
1120certain situations. For example, suppose you need to do extensive
1121testing on an image and you would like to easily identify the image
1122under test from existing images with typical default hostnames. In this
1123situation, you could change the default hostname to "testme", which
1124results in all the images using the name "testme". Once testing is
1125complete and you do not need to rebuild the image for test any longer,
1126you can easily reset the default hostname.
1127
1128Another point of interest is that if you unset the variable, the image
1129will have no default hostname in the filesystem. Here is an example that
1130unsets the variable in a configuration file::
1131
1132 hostname:pn-base-files = ""
1133
1134Having no default hostname in the filesystem is suitable for
1135environments that use dynamic hostnames such as virtual machines.
1136
1137Writing a New Recipe
1138====================
1139
1140Recipes (``.bb`` files) are fundamental components in the Yocto Project
1141environment. Each software component built by the OpenEmbedded build
1142system requires a recipe to define the component. This section describes
1143how to create, write, and test a new recipe.
1144
1145.. note::
1146
1147 For information on variables that are useful for recipes and for
1148 information about recipe naming issues, see the
1149 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
1150 Reference Manual.
1151
1152Overview
1153--------
1154
1155The following figure shows the basic process for creating a new recipe.
1156The remainder of the section provides details for the steps.
1157
1158.. image:: figures/recipe-workflow.png
1159 :align: center
1160 :width: 50%
1161
1162Locate or Automatically Create a Base Recipe
1163--------------------------------------------
1164
1165You can always write a recipe from scratch. However, there are three choices
1166that can help you quickly get started with a new recipe:
1167
1168- ``devtool add``: A command that assists in creating a recipe and an
1169 environment conducive to development.
1170
1171- ``recipetool create``: A command provided by the Yocto Project that
1172 automates creation of a base recipe based on the source files.
1173
1174- *Existing Recipes:* Location and modification of an existing recipe
1175 that is similar in function to the recipe you need.
1176
1177.. note::
1178
1179 For information on recipe syntax, see the
1180 ":ref:`dev-manual/common-tasks:recipe syntax`" section.
1181
1182Creating the Base Recipe Using ``devtool add``
1183~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1184
1185The ``devtool add`` command uses the same logic for auto-creating the
1186recipe as ``recipetool create``, which is listed below. Additionally,
1187however, ``devtool add`` sets up an environment that makes it easy for
1188you to patch the source and to make changes to the recipe as is often
1189necessary when adding a recipe to build a new piece of software to be
1190included in a build.
1191
1192You can find a complete description of the ``devtool add`` command in
1193the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
1194in the Yocto Project Application Development and the Extensible Software
1195Development Kit (eSDK) manual.
1196
1197Creating the Base Recipe Using ``recipetool create``
1198~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1199
1200``recipetool create`` automates creation of a base recipe given a set of
1201source code files. As long as you can extract or point to the source
1202files, the tool will construct a recipe and automatically configure all
1203pre-build information into the recipe. For example, suppose you have an
1204application that builds using Autotools. Creating the base recipe using
1205``recipetool`` results in a recipe that has the pre-build dependencies,
1206license requirements, and checksums configured.
1207
1208To run the tool, you just need to be in your :term:`Build Directory` and
1209have sourced the build environment setup script (i.e.
1210:ref:`structure-core-script`). To get help on the tool, use the following
1211command::
1212
1213 $ recipetool -h
1214 NOTE: Starting bitbake server...
1215 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1216
1217 OpenEmbedded recipe tool
1218
1219 options:
1220 -d, --debug Enable debug output
1221 -q, --quiet Print only errors
1222 --color COLOR Colorize output (where COLOR is auto, always, never)
1223 -h, --help show this help message and exit
1224
1225 subcommands:
1226 create Create a new recipe
1227 newappend Create a bbappend for the specified target in the specified
1228 layer
1229 setvar Set a variable within a recipe
1230 appendfile Create/update a bbappend to replace a target file
1231 appendsrcfiles Create/update a bbappend to add or replace source files
1232 appendsrcfile Create/update a bbappend to add or replace a source file
1233 Use recipetool <subcommand> --help to get help on a specific command
1234
1235Running ``recipetool create -o OUTFILE`` creates the base recipe and
1236locates it properly in the layer that contains your source files.
1237Following are some syntax examples:
1238
1239 - Use this syntax to generate a recipe based on source. Once generated,
1240 the recipe resides in the existing source code layer::
1241
1242 recipetool create -o OUTFILE source
1243
1244 - Use this syntax to generate a recipe using code that
1245 you extract from source. The extracted code is placed in its own layer
1246 defined by :term:`EXTERNALSRC`.
1247 ::
1248
1249 recipetool create -o OUTFILE -x EXTERNALSRC source
1250
1251 - Use this syntax to generate a recipe based on source. The options
1252 direct ``recipetool`` to generate debugging information. Once generated,
1253 the recipe resides in the existing source code layer::
1254
1255 recipetool create -d -o OUTFILE source
1256
1257Locating and Using a Similar Recipe
1258~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1259
1260Before writing a recipe from scratch, it is often useful to discover
1261whether someone else has already written one that meets (or comes close
1262to meeting) your needs. The Yocto Project and OpenEmbedded communities
1263maintain many recipes that might be candidates for what you are doing.
1264You can find a good central index of these recipes in the
1265:oe_layerindex:`OpenEmbedded Layer Index <>`.
1266
1267Working from an existing recipe or a skeleton recipe is the best way to
1268get started. Here are some points on both methods:
1269
1270- *Locate and modify a recipe that is close to what you want to do:*
1271 This method works when you are familiar with the current recipe
1272 space. The method does not work so well for those new to the Yocto
1273 Project or writing recipes.
1274
1275 Some risks associated with this method are using a recipe that has
1276 areas totally unrelated to what you are trying to accomplish with
1277 your recipe, not recognizing areas of the recipe that you might have
1278 to add from scratch, and so forth. All these risks stem from
1279 unfamiliarity with the existing recipe space.
1280
1281- *Use and modify the following skeleton recipe:* If for some reason
1282 you do not want to use ``recipetool`` and you cannot find an existing
1283 recipe that is close to meeting your needs, you can use the following
1284 structure to provide the fundamental areas of a new recipe.
1285 ::
1286
1287 DESCRIPTION = ""
1288 HOMEPAGE = ""
1289 LICENSE = ""
1290 SECTION = ""
1291 DEPENDS = ""
1292 LIC_FILES_CHKSUM = ""
1293
1294 SRC_URI = ""
1295
1296Storing and Naming the Recipe
1297-----------------------------
1298
1299Once you have your base recipe, you should put it in your own layer and
1300name it appropriately. Locating it correctly ensures that the
1301OpenEmbedded build system can find it when you use BitBake to process
1302the recipe.
1303
1304- *Storing Your Recipe:* The OpenEmbedded build system locates your
1305 recipe through the layer's ``conf/layer.conf`` file and the
1306 :term:`BBFILES` variable. This
1307 variable sets up a path from which the build system can locate
1308 recipes. Here is the typical use::
1309
1310 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1311 ${LAYERDIR}/recipes-*/*/*.bbappend"
1312
1313 Consequently, you need to be sure you locate your new recipe inside
1314 your layer such that it can be found.
1315
1316 You can find more information on how layers are structured in the
1317 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
1318
1319- *Naming Your Recipe:* When you name your recipe, you need to follow
1320 this naming convention::
1321
1322 basename_version.bb
1323
1324 Use lower-cased characters and do not include the reserved suffixes
1325 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1326 them as part of your recipe name unless the string applies). Here are some
1327 examples:
1328
1329 .. code-block:: none
1330
1331 cups_1.7.0.bb
1332 gawk_4.0.2.bb
1333 irssi_0.8.16-rc1.bb
1334
1335Running a Build on the Recipe
1336-----------------------------
1337
1338Creating a new recipe is usually an iterative process that requires
1339using BitBake to process the recipe multiple times in order to
1340progressively discover and add information to the recipe file.
1341
1342Assuming you have sourced the build environment setup script (i.e.
1343:ref:`structure-core-script`) and you are in the :term:`Build Directory`, use
1344BitBake to process your recipe. All you need to provide is the
1345``basename`` of the recipe as described in the previous section::
1346
1347 $ bitbake basename
1348
1349During the build, the OpenEmbedded build system creates a temporary work
1350directory for each recipe
1351(``${``\ :term:`WORKDIR`\ ``}``)
1352where it keeps extracted source files, log files, intermediate
1353compilation and packaging files, and so forth.
1354
1355The path to the per-recipe temporary work directory depends on the
1356context in which it is being built. The quickest way to find this path
1357is to have BitBake return it by running the following::
1358
1359 $ bitbake -e basename | grep ^WORKDIR=
1360
1361As an example, assume a Source Directory
1362top-level folder named ``poky``, a default :term:`Build Directory` at
1363``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1364Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1365case, the work directory the build system uses to build the package
1366would be as follows::
1367
1368 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1369
1370Inside this directory you can find sub-directories such as ``image``,
1371``packages-split``, and ``temp``. After the build, you can examine these
1372to determine how well the build went.
1373
1374.. note::
1375
1376 You can find log files for each task in the recipe's ``temp``
1377 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1378 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1379 ``log.do_fetch``, and ``log.do_compile``).
1380
1381You can find more information about the build process in
1382":doc:`/overview-manual/development-environment`"
1383chapter of the Yocto Project Overview and Concepts Manual.
1384
1385Fetching Code
1386-------------
1387
1388The first thing your recipe must do is specify how to fetch the source
1389files. Fetching is controlled mainly through the
1390:term:`SRC_URI` variable. Your recipe
1391must have a :term:`SRC_URI` variable that points to where the source is
1392located. For a graphical representation of source locations, see the
1393":ref:`overview-manual/concepts:sources`" section in
1394the Yocto Project Overview and Concepts Manual.
1395
1396The :ref:`ref-tasks-fetch` task uses
1397the prefix of each entry in the :term:`SRC_URI` variable value to determine
1398which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
1399source files. It is the :term:`SRC_URI` variable that triggers the fetcher.
1400The :ref:`ref-tasks-patch` task uses
1401the variable after source is fetched to apply patches. The OpenEmbedded
1402build system uses
1403:term:`FILESOVERRIDES` for
1404scanning directory locations for local files in :term:`SRC_URI`.
1405
1406The :term:`SRC_URI` variable in your recipe must define each unique location
1407for your source files. It is good practice to not hard-code version
1408numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
1409values, use ``${``\ :term:`PV`\ ``}``,
1410which causes the fetch process to use the version specified in the
1411recipe filename. Specifying the version in this manner means that
1412upgrading the recipe to a future version is as simple as renaming the
1413recipe to match the new version.
1414
1415Here is a simple example from the
1416``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1417comes from a single tarball. Notice the use of the
1418:term:`PV` variable::
1419
1420 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
1421
1422Files mentioned in :term:`SRC_URI` whose names end in a typical archive
1423extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1424forth), are automatically extracted during the
1425:ref:`ref-tasks-unpack` task. For
1426another example that specifies these types of files, see the
1427":ref:`dev-manual/common-tasks:autotooled package`" section.
1428
1429Another way of specifying source is from an SCM. For Git repositories,
1430you must specify :term:`SRCREV` and you should specify :term:`PV` to include
1431the revision with :term:`SRCPV`. Here is an example from the recipe
1432``meta/recipes-core/musl/gcompat_git.bb``::
1433
1434 SRC_URI = "git://git.adelielinux.org/adelie/gcompat.git;protocol=https;branch=current"
1435
1436 PV = "1.0.0+1.1+git${SRCPV}"
1437 SRCREV = "af5a49e489fdc04b9cf02547650d7aeaccd43793"
1438
1439If your :term:`SRC_URI` statement includes URLs pointing to individual files
1440fetched from a remote server other than a version control system,
1441BitBake attempts to verify the files against checksums defined in your
1442recipe to ensure they have not been tampered with or otherwise modified
1443since the recipe was written. Two checksums are used:
1444``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1445
1446If your :term:`SRC_URI` variable points to more than a single URL (excluding
1447SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1448each URL. For these cases, you provide a name for each URL as part of
1449the :term:`SRC_URI` and then reference that name in the subsequent checksum
1450statements. Here is an example combining lines from the files
1451``git.inc`` and ``git_2.24.1.bb``::
1452
1453 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1454 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1455
1456 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1457 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1458 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1459 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1460
1461Proper values for ``md5`` and ``sha256`` checksums might be available
1462with other signatures on the download page for the upstream source (e.g.
1463``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1464OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1465you should verify all the signatures you find by hand.
1466
1467If no :term:`SRC_URI` checksums are specified when you attempt to build the
1468recipe, or you provide an incorrect checksum, the build will produce an
1469error for each missing or incorrect checksum. As part of the error
1470message, the build system provides the checksum string corresponding to
1471the fetched file. Once you have the correct checksums, you can copy and
1472paste them into your recipe and then run the build again to continue.
1473
1474.. note::
1475
1476 As mentioned, if the upstream source provides signatures for
1477 verifying the downloaded source code, you should verify those
1478 manually before setting the checksum values in the recipe and
1479 continuing with the build.
1480
1481This final example is a bit more complicated and is from the
1482``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
1483example's :term:`SRC_URI` statement identifies multiple files as the source
1484files for the recipe: a tarball, a patch file, a desktop file, and an
1485icon.
1486::
1487
1488 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1489 file://xwc.patch \
1490 file://rxvt.desktop \
1491 file://rxvt.png"
1492
1493When you specify local files using the ``file://`` URI protocol, the
1494build system fetches files from the local machine. The path is relative
1495to the :term:`FILESPATH` variable
1496and searches specific directories in a certain order:
1497``${``\ :term:`BP`\ ``}``,
1498``${``\ :term:`BPN`\ ``}``, and
1499``files``. The directories are assumed to be subdirectories of the
1500directory in which the recipe or append file resides. For another
1501example that specifies these types of files, see the
1502":ref:`dev-manual/common-tasks:single .c file package (hello world!)`" section.
1503
1504The previous example also specifies a patch file. Patch files are files
1505whose names usually end in ``.patch`` or ``.diff`` but can end with
1506compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1507The build system automatically applies patches as described in the
1508":ref:`dev-manual/common-tasks:patching code`" section.
1509
1510Fetching Code Through Firewalls
1511~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1512
1513Some users are behind firewalls and need to fetch code through a proxy.
1514See the ":doc:`/ref-manual/faq`" chapter for advice.
1515
1516Limiting the Number of Parallel Connections
1517~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1518
1519Some users are behind firewalls or use servers where the number of parallel
1520connections is limited. In such cases, you can limit the number of fetch
1521tasks being run in parallel by adding the following to your ``local.conf``
1522file::
1523
1524 do_fetch[number_threads] = "4"
1525
1526Unpacking Code
1527--------------
1528
1529During the build, the
1530:ref:`ref-tasks-unpack` task unpacks
1531the source with ``${``\ :term:`S`\ ``}``
1532pointing to where it is unpacked.
1533
1534If you are fetching your source files from an upstream source archived
1535tarball and the tarball's internal structure matches the common
1536convention of a top-level subdirectory named
1537``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
1538then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
1539fetch source from an archive that does not use this convention, or from
1540an SCM like Git or Subversion, your recipe needs to define :term:`S`.
1541
1542If processing your recipe using BitBake successfully unpacks the source
1543files, you need to be sure that the directory pointed to by ``${S}``
1544matches the structure of the source.
1545
1546Patching Code
1547-------------
1548
1549Sometimes it is necessary to patch code after it has been fetched. Any
1550files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
1551``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1552treated as patches. The
1553:ref:`ref-tasks-patch` task
1554automatically applies these patches.
1555
1556The build system should be able to apply patches with the "-p1" option
1557(i.e. one directory level in the path will be stripped off). If your
1558patch needs to have more directory levels stripped off, specify the
1559number of levels using the "striplevel" option in the :term:`SRC_URI` entry
1560for the patch. Alternatively, if your patch needs to be applied in a
1561specific subdirectory that is not specified in the patch file, use the
1562"patchdir" option in the entry.
1563
1564As with all local files referenced in
1565:term:`SRC_URI` using ``file://``,
1566you should place patch files in a directory next to the recipe either
1567named the same as the base name of the recipe
1568(:term:`BP` and
1569:term:`BPN`) or "files".
1570
1571Licensing
1572---------
1573
1574Your recipe needs to have both the
1575:term:`LICENSE` and
1576:term:`LIC_FILES_CHKSUM`
1577variables:
1578
1579- :term:`LICENSE`: This variable specifies the license for the software.
1580 If you do not know the license under which the software you are
1581 building is distributed, you should go to the source code and look
1582 for that information. Typical files containing this information
1583 include ``COPYING``, :term:`LICENSE`, and ``README`` files. You could
1584 also find the information near the top of a source file. For example,
1585 given a piece of software licensed under the GNU General Public
1586 License version 2, you would set :term:`LICENSE` as follows::
1587
1588 LICENSE = "GPL-2.0-only"
1589
1590 The licenses you specify within :term:`LICENSE` can have any name as long
1591 as you do not use spaces, since spaces are used as separators between
1592 license names. For standard licenses, use the names of the files in
1593 ``meta/files/common-licenses/`` or the :term:`SPDXLICENSEMAP` flag names
1594 defined in ``meta/conf/licenses.conf``.
1595
1596- :term:`LIC_FILES_CHKSUM`: The OpenEmbedded build system uses this
1597 variable to make sure the license text has not changed. If it has,
1598 the build produces an error and it affords you the chance to figure
1599 it out and correct the problem.
1600
1601 You need to specify all applicable licensing files for the software.
1602 At the end of the configuration step, the build process will compare
1603 the checksums of the files to be sure the text has not changed. Any
1604 differences result in an error with the message containing the
1605 current checksum. For more explanation and examples of how to set the
1606 :term:`LIC_FILES_CHKSUM` variable, see the
1607 ":ref:`dev-manual/common-tasks:tracking license changes`" section.
1608
1609 To determine the correct checksum string, you can list the
1610 appropriate files in the :term:`LIC_FILES_CHKSUM` variable with incorrect
1611 md5 strings, attempt to build the software, and then note the
1612 resulting error messages that will report the correct md5 strings.
1613 See the ":ref:`dev-manual/common-tasks:fetching code`" section for
1614 additional information.
1615
1616 Here is an example that assumes the software has a ``COPYING`` file::
1617
1618 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1619
1620 When you try to build the
1621 software, the build system will produce an error and give you the
1622 correct string that you can substitute into the recipe file for a
1623 subsequent build.
1624
1625Dependencies
1626------------
1627
1628Most software packages have a short list of other packages that they
1629require, which are called dependencies. These dependencies fall into two
1630main categories: build-time dependencies, which are required when the
1631software is built; and runtime dependencies, which are required to be
1632installed on the target in order for the software to run.
1633
1634Within a recipe, you specify build-time dependencies using the
1635:term:`DEPENDS` variable. Although there are nuances,
1636items specified in :term:`DEPENDS` should be names of other
1637recipes. It is important that you specify all build-time dependencies
1638explicitly.
1639
1640Another consideration is that configure scripts might automatically
1641check for optional dependencies and enable corresponding functionality
1642if those dependencies are found. If you wish to make a recipe that is
1643more generally useful (e.g. publish the recipe in a layer for others to
1644use), instead of hard-disabling the functionality, you can use the
1645:term:`PACKAGECONFIG` variable to allow functionality and the
1646corresponding dependencies to be enabled and disabled easily by other
1647users of the recipe.
1648
1649Similar to build-time dependencies, you specify runtime dependencies
1650through a variable -
1651:term:`RDEPENDS`, which is
1652package-specific. All variables that are package-specific need to have
1653the name of the package added to the end as an override. Since the main
1654package for a recipe has the same name as the recipe, and the recipe's
1655name can be found through the
1656``${``\ :term:`PN`\ ``}`` variable, then
1657you specify the dependencies for the main package by setting
1658``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you
1659would set ``RDEPENDS:${PN}-tools``, and so forth.
1660
1661Some runtime dependencies will be set automatically at packaging time.
1662These dependencies include any shared library dependencies (i.e. if a
1663package "example" contains "libexample" and another package "mypackage"
1664contains a binary that links to "libexample" then the OpenEmbedded build
1665system will automatically add a runtime dependency to "mypackage" on
1666"example"). See the
1667":ref:`overview-manual/concepts:automatically added runtime dependencies`"
1668section in the Yocto Project Overview and Concepts Manual for further
1669details.
1670
1671Configuring the Recipe
1672----------------------
1673
1674Most software provides some means of setting build-time configuration
1675options before compilation. Typically, setting these options is
1676accomplished by running a configure script with options, or by modifying
1677a build configuration file.
1678
1679.. note::
1680
1681 As of Yocto Project Release 1.7, some of the core recipes that
1682 package binary configuration scripts now disable the scripts due to
1683 the scripts previously requiring error-prone path substitution. The
1684 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1685 robust. You can find a list of the ``*-config`` scripts that are disabled
1686 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1687 in the Yocto Project Reference Manual.
1688
1689A major part of build-time configuration is about checking for
1690build-time dependencies and possibly enabling optional functionality as
1691a result. You need to specify any build-time dependencies for the
1692software you are building in your recipe's
1693:term:`DEPENDS` value, in terms of
1694other recipes that satisfy those dependencies. You can often find
1695build-time or runtime dependencies described in the software's
1696documentation.
1697
1698The following list provides configuration items of note based on how
1699your software is built:
1700
1701- *Autotools:* If your source files have a ``configure.ac`` file, then
1702 your software is built using Autotools. If this is the case, you just
1703 need to modify the configuration.
1704
1705 When using Autotools, your recipe needs to inherit the
1706 :ref:`autotools <ref-classes-autotools>` class and it does not have to
1707 contain a :ref:`ref-tasks-configure` task. However, you might still want to
1708 make some adjustments. For example, you can set :term:`EXTRA_OECONF` or
1709 :term:`PACKAGECONFIG_CONFARGS` to pass any needed configure options that
1710 are specific to the recipe.
1711
1712- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1713 your software is built using CMake. If this is the case, you just
1714 need to modify the configuration.
1715
1716 When you use CMake, your recipe needs to inherit the
1717 :ref:`cmake <ref-classes-cmake>` class and it does not have to contain a
1718 :ref:`ref-tasks-configure` task. You can make some adjustments by setting
1719 :term:`EXTRA_OECMAKE` to pass any needed configure options that are
1720 specific to the recipe.
1721
1722 .. note::
1723
1724 If you need to install one or more custom CMake toolchain files
1725 that are supplied by the application you are building, install the
1726 files to ``${D}${datadir}/cmake/Modules`` during :ref:`ref-tasks-install`.
1727
1728- *Other:* If your source files do not have a ``configure.ac`` or
1729 ``CMakeLists.txt`` file, then your software is built using some
1730 method other than Autotools or CMake. If this is the case, you
1731 normally need to provide a
1732 :ref:`ref-tasks-configure` task
1733 in your recipe unless, of course, there is nothing to configure.
1734
1735 Even if your software is not being built by Autotools or CMake, you
1736 still might not need to deal with any configuration issues. You need
1737 to determine if configuration is even a required step. You might need
1738 to modify a Makefile or some configuration file used for the build to
1739 specify necessary build options. Or, perhaps you might need to run a
1740 provided, custom configure script with the appropriate options.
1741
1742 For the case involving a custom configure script, you would run
1743 ``./configure --help`` and look for the options you need to set.
1744
1745Once configuration succeeds, it is always good practice to look at the
1746``log.do_configure`` file to ensure that the appropriate options have
1747been enabled and no additional build-time dependencies need to be added
1748to :term:`DEPENDS`. For example, if the configure script reports that it
1749found something not mentioned in :term:`DEPENDS`, or that it did not find
1750something that it needed for some desired optional functionality, then
1751you would need to add those to :term:`DEPENDS`. Looking at the log might
1752also reveal items being checked for, enabled, or both that you do not
1753want, or items not being found that are in :term:`DEPENDS`, in which case
1754you would need to look at passing extra options to the configure script
1755as needed. For reference information on configure options specific to
1756the software you are building, you can consult the output of the
1757``./configure --help`` command within ``${S}`` or consult the software's
1758upstream documentation.
1759
1760Using Headers to Interface with Devices
1761---------------------------------------
1762
1763If your recipe builds an application that needs to communicate with some
1764device or needs an API into a custom kernel, you will need to provide
1765appropriate header files. Under no circumstances should you ever modify
1766the existing
1767``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1768These headers are used to build ``libc`` and must not be compromised
1769with custom or machine-specific header information. If you customize
1770``libc`` through modified headers all other applications that use
1771``libc`` thus become affected.
1772
1773.. note::
1774
1775 Never copy and customize the ``libc`` header file (i.e.
1776 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
1777
1778The correct way to interface to a device or custom kernel is to use a
1779separate package that provides the additional headers for the driver or
1780other unique interfaces. When doing so, your application also becomes
1781responsible for creating a dependency on that specific provider.
1782
1783Consider the following:
1784
1785- Never modify ``linux-libc-headers.inc``. Consider that file to be
1786 part of the ``libc`` system, and not something you use to access the
1787 kernel directly. You should access ``libc`` through specific ``libc``
1788 calls.
1789
1790- Applications that must talk directly to devices should either provide
1791 necessary headers themselves, or establish a dependency on a special
1792 headers package that is specific to that driver.
1793
1794For example, suppose you want to modify an existing header that adds I/O
1795control or network support. If the modifications are used by a small
1796number programs, providing a unique version of a header is easy and has
1797little impact. When doing so, bear in mind the guidelines in the
1798previous list.
1799
1800.. note::
1801
1802 If for some reason your changes need to modify the behavior of the ``libc``,
1803 and subsequently all other applications on the system, use a ``.bbappend``
1804 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1805 make the changes machine specific.
1806
1807Consider a case where your kernel is older and you need an older
1808``libc`` ABI. The headers installed by your recipe should still be a
1809standard mainline kernel, not your own custom one.
1810
1811When you use custom kernel headers you need to get them from
1812:term:`STAGING_KERNEL_DIR`,
1813which is the directory with kernel headers that are required to build
1814out-of-tree modules. Your recipe will also need the following::
1815
1816 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1817
1818Compilation
1819-----------
1820
1821During a build, the :ref:`ref-tasks-compile` task happens after source is fetched,
1822unpacked, and configured. If the recipe passes through :ref:`ref-tasks-compile`
1823successfully, nothing needs to be done.
1824
1825However, if the compile step fails, you need to diagnose the failure.
1826Here are some common issues that cause failures.
1827
1828.. note::
1829
1830 For cases where improper paths are detected for configuration files
1831 or for when libraries/headers cannot be found, be sure you are using
1832 the more robust ``pkg-config``. See the note in section
1833 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information.
1834
1835- *Parallel build failures:* These failures manifest themselves as
1836 intermittent errors, or errors reporting that a file or directory
1837 that should be created by some other part of the build process could
1838 not be found. This type of failure can occur even if, upon
1839 inspection, the file or directory does exist after the build has
1840 failed, because that part of the build process happened in the wrong
1841 order.
1842
1843 To fix the problem, you need to either satisfy the missing dependency
1844 in the Makefile or whatever script produced the Makefile, or (as a
1845 workaround) set :term:`PARALLEL_MAKE` to an empty string::
1846
1847 PARALLEL_MAKE = ""
1848
1849 For information on parallel Makefile issues, see the
1850 ":ref:`dev-manual/common-tasks:debugging parallel make races`" section.
1851
1852- *Improper host path usage:* This failure applies to recipes building
1853 for the target or ":ref:`nativesdk <ref-classes-nativesdk>`" only. The
1854 failure occurs when the compilation process uses improper headers,
1855 libraries, or other files from the host system when cross-compiling for
1856 the target.
1857
1858 To fix the problem, examine the ``log.do_compile`` file to identify
1859 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1860 so forth) and then either add configure options, apply a patch, or do
1861 both.
1862
1863- *Failure to find required libraries/headers:* If a build-time
1864 dependency is missing because it has not been declared in
1865 :term:`DEPENDS`, or because the
1866 dependency exists but the path used by the build process to find the
1867 file is incorrect and the configure step did not detect it, the
1868 compilation process could fail. For either of these failures, the
1869 compilation process notes that files could not be found. In these
1870 cases, you need to go back and add additional options to the
1871 configure script as well as possibly add additional build-time
1872 dependencies to :term:`DEPENDS`.
1873
1874 Occasionally, it is necessary to apply a patch to the source to
1875 ensure the correct paths are used. If you need to specify paths to
1876 find files staged into the sysroot from other recipes, use the
1877 variables that the OpenEmbedded build system provides (e.g.
1878 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
1879 forth).
1880
1881Installing
1882----------
1883
1884During :ref:`ref-tasks-install`, the task copies the built files along with their
1885hierarchy to locations that would mirror their locations on the target
1886device. The installation process copies files from the
1887``${``\ :term:`S`\ ``}``,
1888``${``\ :term:`B`\ ``}``, and
1889``${``\ :term:`WORKDIR`\ ``}``
1890directories to the ``${``\ :term:`D`\ ``}``
1891directory to create the structure as it should appear on the target
1892system.
1893
1894How your software is built affects what you must do to be sure your
1895software is installed correctly. The following list describes what you
1896must do for installation depending on the type of build system used by
1897the software being built:
1898
1899- *Autotools and CMake:* If the software your recipe is building uses
1900 Autotools or CMake, the OpenEmbedded build system understands how to
1901 install the software. Consequently, you do not have to have a
1902 :ref:`ref-tasks-install` task as part of your recipe. You just need to make
1903 sure the install portion of the build completes with no issues.
1904 However, if you wish to install additional files not already being
1905 installed by ``make install``, you should do this using a
1906 ``do_install:append`` function using the install command as described
1907 in the "Manual" bulleted item later in this list.
1908
1909- *Other (using* ``make install``\ *)*: You need to define a :ref:`ref-tasks-install`
1910 function in your recipe. The function should call
1911 ``oe_runmake install`` and will likely need to pass in the
1912 destination directory as well. How you pass that path is dependent on
1913 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1914 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1915
1916 For an example recipe using ``make install``, see the
1917 ":ref:`dev-manual/common-tasks:makefile-based package`" section.
1918
1919- *Manual:* You need to define a :ref:`ref-tasks-install` function in your
1920 recipe. The function must first use ``install -d`` to create the
1921 directories under
1922 ``${``\ :term:`D`\ ``}``. Once the
1923 directories exist, your function can use ``install`` to manually
1924 install the built software into the directories.
1925
1926 You can find more information on ``install`` at
1927 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
1928
1929For the scenarios that do not use Autotools or CMake, you need to track
1930the installation and diagnose and fix any issues until everything
1931installs correctly. You need to look in the default location of
1932``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1933installed correctly.
1934
1935.. note::
1936
1937 - During the installation process, you might need to modify some of
1938 the installed files to suit the target layout. For example, you
1939 might need to replace hard-coded paths in an initscript with
1940 values of variables provided by the build system, such as
1941 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1942 modifications during :ref:`ref-tasks-install`, be sure to modify the
1943 destination file after copying rather than before copying.
1944 Modifying after copying ensures that the build system can
1945 re-execute :ref:`ref-tasks-install` if needed.
1946
1947 - ``oe_runmake install``, which can be run directly or can be run
1948 indirectly by the
1949 :ref:`autotools <ref-classes-autotools>` and
1950 :ref:`cmake <ref-classes-cmake>` classes,
1951 runs ``make install`` in parallel. Sometimes, a Makefile can have
1952 missing dependencies between targets that can result in race
1953 conditions. If you experience intermittent failures during
1954 :ref:`ref-tasks-install`, you might be able to work around them by disabling
1955 parallel Makefile installs by adding the following to the recipe::
1956
1957 PARALLEL_MAKEINST = ""
1958
1959 See :term:`PARALLEL_MAKEINST` for additional information.
1960
1961 - If you need to install one or more custom CMake toolchain files
1962 that are supplied by the application you are building, install the
1963 files to ``${D}${datadir}/cmake/Modules`` during
1964 :ref:`ref-tasks-install`.
1965
1966Enabling System Services
1967------------------------
1968
1969If you want to install a service, which is a process that usually starts
1970on boot and runs in the background, then you must include some
1971additional definitions in your recipe.
1972
1973If you are adding services and the service initialization script or the
1974service file itself is not installed, you must provide for that
1975installation in your recipe using a ``do_install:append`` function. If
1976your recipe already has a :ref:`ref-tasks-install` function, update the function
1977near its end rather than adding an additional ``do_install:append``
1978function.
1979
1980When you create the installation for your services, you need to
1981accomplish what is normally done by ``make install``. In other words,
1982make sure your installation arranges the output similar to how it is
1983arranged on the target system.
1984
1985The OpenEmbedded build system provides support for starting services two
1986different ways:
1987
1988- *SysVinit:* SysVinit is a system and service manager that manages the
1989 init system used to control the very basic functions of your system.
1990 The init program is the first program started by the Linux kernel
1991 when the system boots. Init then controls the startup, running and
1992 shutdown of all other programs.
1993
1994 To enable a service using SysVinit, your recipe needs to inherit the
1995 :ref:`update-rc.d <ref-classes-update-rc.d>` class. The class helps
1996 facilitate safely installing the package on the target.
1997
1998 You will need to set the
1999 :term:`INITSCRIPT_PACKAGES`,
2000 :term:`INITSCRIPT_NAME`,
2001 and
2002 :term:`INITSCRIPT_PARAMS`
2003 variables within your recipe.
2004
2005- *systemd:* System Management Daemon (systemd) was designed to replace
2006 SysVinit and to provide enhanced management of services. For more
2007 information on systemd, see the systemd homepage at
2008 https://freedesktop.org/wiki/Software/systemd/.
2009
2010 To enable a service using systemd, your recipe needs to inherit the
2011 :ref:`systemd <ref-classes-systemd>` class. See the ``systemd.bbclass`` file
2012 located in your :term:`Source Directory` section for more information.
2013
2014Packaging
2015---------
2016
2017Successful packaging is a combination of automated processes performed
2018by the OpenEmbedded build system and some specific steps you need to
2019take. The following list describes the process:
2020
2021- *Splitting Files*: The :ref:`ref-tasks-package` task splits the files produced
2022 by the recipe into logical components. Even software that produces a
2023 single binary might still have debug symbols, documentation, and
2024 other logical components that should be split out. The :ref:`ref-tasks-package`
2025 task ensures that files are split up and packaged correctly.
2026
2027- *Running QA Checks*: The
2028 :ref:`insane <ref-classes-insane>` class adds a
2029 step to the package generation process so that output quality
2030 assurance checks are generated by the OpenEmbedded build system. This
2031 step performs a range of checks to be sure the build's output is free
2032 of common problems that show up during runtime. For information on
2033 these checks, see the
2034 :ref:`insane <ref-classes-insane>` class and
2035 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
2036 chapter in the Yocto Project Reference Manual.
2037
2038- *Hand-Checking Your Packages*: After you build your software, you
2039 need to be sure your packages are correct. Examine the
2040 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
2041 directory and make sure files are where you expect them to be. If you
2042 discover problems, you can set
2043 :term:`PACKAGES`,
2044 :term:`FILES`,
2045 ``do_install(:append)``, and so forth as needed.
2046
2047- *Splitting an Application into Multiple Packages*: If you need to
2048 split an application into several packages, see the
2049 ":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
2050 section for an example.
2051
2052- *Installing a Post-Installation Script*: For an example showing how
2053 to install a post-installation script, see the
2054 ":ref:`dev-manual/common-tasks:post-installation scripts`" section.
2055
2056- *Marking Package Architecture*: Depending on what your recipe is
2057 building and how it is configured, it might be important to mark the
2058 packages produced as being specific to a particular machine, or to
2059 mark them as not being specific to a particular machine or
2060 architecture at all.
2061
2062 By default, packages apply to any machine with the same architecture
2063 as the target machine. When a recipe produces packages that are
2064 machine-specific (e.g. the
2065 :term:`MACHINE` value is passed
2066 into the configure script or a patch is applied only for a particular
2067 machine), you should mark them as such by adding the following to the
2068 recipe::
2069
2070 PACKAGE_ARCH = "${MACHINE_ARCH}"
2071
2072 On the other hand, if the recipe produces packages that do not
2073 contain anything specific to the target machine or architecture at
2074 all (e.g. recipes that simply package script files or configuration
2075 files), you should use the
2076 :ref:`allarch <ref-classes-allarch>` class to
2077 do this for you by adding this to your recipe::
2078
2079 inherit allarch
2080
2081 Ensuring that the package architecture is correct is not critical
2082 while you are doing the first few builds of your recipe. However, it
2083 is important in order to ensure that your recipe rebuilds (or does
2084 not rebuild) appropriately in response to changes in configuration,
2085 and to ensure that you get the appropriate packages installed on the
2086 target machine, particularly if you run separate builds for more than
2087 one target machine.
2088
2089Sharing Files Between Recipes
2090-----------------------------
2091
2092Recipes often need to use files provided by other recipes on the build
2093host. For example, an application linking to a common library needs
2094access to the library itself and its associated headers. The way this
2095access is accomplished is by populating a sysroot with files. Each
2096recipe has two sysroots in its work directory, one for target files
2097(``recipe-sysroot``) and one for files that are native to the build host
2098(``recipe-sysroot-native``).
2099
2100.. note::
2101
2102 You could find the term "staging" used within the Yocto project
2103 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
2104 variable).
2105
2106Recipes should never populate the sysroot directly (i.e. write files
2107into sysroot). Instead, files should be installed into standard
2108locations during the
2109:ref:`ref-tasks-install` task within
2110the ``${``\ :term:`D`\ ``}`` directory. The
2111reason for this limitation is that almost all files that populate the
2112sysroot are cataloged in manifests in order to ensure the files can be
2113removed later when a recipe is either modified or removed. Thus, the
2114sysroot is able to remain free from stale files.
2115
2116A subset of the files installed by the :ref:`ref-tasks-install` task are
2117used by the :ref:`ref-tasks-populate_sysroot` task as defined by the
2118:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
2119is possible to modify the list of directories that populate the sysroot.
2120The following example shows how you could add the ``/opt`` directory to
2121the list of directories within a recipe::
2122
2123 SYSROOT_DIRS += "/opt"
2124
2125.. note::
2126
2127 The `/sysroot-only` is to be used by recipes that generate artifacts
2128 that are not included in the target filesystem, allowing them to share
2129 these artifacts without needing to use the :term:`DEPLOY_DIR`.
2130
2131For a more complete description of the :ref:`ref-tasks-populate_sysroot`
2132task and its associated functions, see the
2133:ref:`staging <ref-classes-staging>` class.
2134
2135Using Virtual Providers
2136-----------------------
2137
2138Prior to a build, if you know that several different recipes provide the
2139same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2140as a placeholder for the actual provider. The actual provider is
2141determined at build-time.
2142
2143A common scenario where a virtual provider is used would be for the
2144kernel recipe. Suppose you have three kernel recipes whose
2145:term:`PN` values map to ``kernel-big``,
2146``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2147in some way uses a :term:`PROVIDES`
2148statement that essentially identifies itself as being able to provide
2149``virtual/kernel``. Here is one way through the
2150:ref:`kernel <ref-classes-kernel>` class::
2151
2152 PROVIDES += "virtual/kernel"
2153
2154Any recipe that inherits the :ref:`kernel <ref-classes-kernel>` class is
2155going to utilize a :term:`PROVIDES` statement that identifies that recipe as
2156being able to provide the ``virtual/kernel`` item.
2157
2158Now comes the time to actually build an image and you need a kernel
2159recipe, but which one? You can configure your build to call out the
2160kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
2161an example, consider the :yocto_git:`x86-base.inc
2162</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a
2163machine (i.e. :term:`MACHINE`) configuration file. This include file is the
2164reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
2165relevant lines from the include file::
2166
2167 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2168 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2169
2170When you use a virtual provider, you do not have to "hard code" a recipe
2171name as a build dependency. You can use the
2172:term:`DEPENDS` variable to state the
2173build is dependent on ``virtual/kernel`` for example::
2174
2175 DEPENDS = "virtual/kernel"
2176
2177During the build, the OpenEmbedded build system picks
2178the correct recipe needed for the ``virtual/kernel`` dependency based on
2179the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
2180mentioned at the beginning of this section, configure your build as
2181follows::
2182
2183 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
2184
2185.. note::
2186
2187 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
2188 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
2189 recipes from building is usually the desired behavior since this mechanism's
2190 purpose is to select between mutually exclusive alternative providers.
2191
2192The following lists specific examples of virtual providers:
2193
2194- ``virtual/kernel``: Provides the name of the kernel recipe to use
2195 when building a kernel image.
2196
2197- ``virtual/bootloader``: Provides the name of the bootloader to use
2198 when building an image.
2199
2200- ``virtual/libgbm``: Provides ``gbm.pc``.
2201
2202- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2203
2204- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2205
2206- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2207
2208- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2209
2210.. note::
2211
2212 Virtual providers only apply to build time dependencies specified with
2213 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2214 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2215
2216Properly Versioning Pre-Release Recipes
2217---------------------------------------
2218
2219Sometimes the name of a recipe can lead to versioning problems when the
2220recipe is upgraded to a final release. For example, consider the
2221``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
2222the ":ref:`dev-manual/common-tasks:storing and naming the recipe`" section.
2223This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
2224released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2225change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2226build system and package managers, so the resulting packages will not
2227correctly trigger an upgrade.
2228
2229In order to ensure the versions compare properly, the recommended
2230convention is to set :term:`PV` within the
2231recipe to "previous_version+current_version". You can use an additional
2232variable so that you can use the current version elsewhere. Here is an
2233example::
2234
2235 REALPV = "0.8.16-rc1"
2236 PV = "0.8.15+${REALPV}"
2237
2238Post-Installation Scripts
2239-------------------------
2240
2241Post-installation scripts run immediately after installing a package on
2242the target or during image creation when a package is included in an
2243image. To add a post-installation script to a package, add a
2244``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file
2245(``.bb``) and replace `PACKAGENAME` with the name of the package you want
2246to attach to the ``postinst`` script. To apply the post-installation
2247script to the main package for the recipe, which is usually what is
2248required, specify
2249``${``\ :term:`PN`\ ``}`` in place of
2250PACKAGENAME.
2251
2252A post-installation function has the following structure::
2253
2254 pkg_postinst:PACKAGENAME() {
2255 # Commands to carry out
2256 }
2257
2258The script defined in the post-installation function is called when the
2259root filesystem is created. If the script succeeds, the package is
2260marked as installed.
2261
2262.. note::
2263
2264 Any RPM post-installation script that runs on the target should
2265 return a 0 exit code. RPM does not allow non-zero exit codes for
2266 these scripts, and the RPM package manager will cause the package to
2267 fail installation on the target.
2268
2269Sometimes it is necessary for the execution of a post-installation
2270script to be delayed until the first boot. For example, the script might
2271need to be executed on the device itself. To delay script execution
2272until boot time, you must explicitly mark post installs to defer to the
2273target. You can use ``pkg_postinst_ontarget()`` or call
2274``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2275failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2276error during the
2277:ref:`ref-tasks-rootfs` task.
2278
2279If you have recipes that use ``pkg_postinst`` function and they require
2280the use of non-standard native tools that have dependencies during
2281root filesystem construction, you need to use the
2282:term:`PACKAGE_WRITE_DEPS`
2283variable in your recipe to list these tools. If you do not use this
2284variable, the tools might be missing and execution of the
2285post-installation script is deferred until first boot. Deferring the
2286script to the first boot is undesirable and impossible for read-only
2287root filesystems.
2288
2289.. note::
2290
2291 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
2292 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
2293 respectively. These scrips work in exactly the same way as does
2294 ``pkg_postinst`` with the exception that they run at different times. Also,
2295 because of when they run, they are not applicable to being run at image
2296 creation time like ``pkg_postinst``.
2297
2298Testing
2299-------
2300
2301The final step for completing your recipe is to be sure that the
2302software you built runs correctly. To accomplish runtime testing, add
2303the build's output packages to your image and test them on the target.
2304
2305For information on how to customize your image by adding specific
2306packages, see ":ref:`dev-manual/common-tasks:customizing images`" section.
2307
2308Examples
2309--------
2310
2311To help summarize how to write a recipe, this section provides some
2312examples given various scenarios:
2313
2314- Recipes that use local files
2315
2316- Using an Autotooled package
2317
2318- Using a Makefile-based package
2319
2320- Splitting an application into multiple packages
2321
2322- Adding binaries to an image
2323
2324Single .c File Package (Hello World!)
2325~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2326
2327Building an application from a single file that is stored locally (e.g.
2328under ``files``) requires a recipe that has the file listed in the
2329:term:`SRC_URI` variable. Additionally, you need to manually write the
2330:ref:`ref-tasks-compile` and :ref:`ref-tasks-install` tasks. The :term:`S` variable defines the
2331directory containing the source code, which is set to
2332:term:`WORKDIR` in this case --- the
2333directory BitBake uses for the build.
2334::
2335
2336 SUMMARY = "Simple helloworld application"
2337 SECTION = "examples"
2338 LICENSE = "MIT"
2339 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2340
2341 SRC_URI = "file://helloworld.c"
2342
2343 S = "${WORKDIR}"
2344
2345 do_compile() {
2346 ${CC} ${LDFLAGS} helloworld.c -o helloworld
2347 }
2348
2349 do_install() {
2350 install -d ${D}${bindir}
2351 install -m 0755 helloworld ${D}${bindir}
2352 }
2353
2354By default, the ``helloworld``, ``helloworld-dbg``, and
2355``helloworld-dev`` packages are built. For information on how to
2356customize the packaging process, see the
2357":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
2358section.
2359
2360Autotooled Package
2361~~~~~~~~~~~~~~~~~~
2362
2363Applications that use Autotools such as ``autoconf`` and ``automake``
2364require a recipe that has a source archive listed in :term:`SRC_URI` and
2365also inherit the :ref:`autotools <ref-classes-autotools>` class,
2366which contains the definitions of all the steps needed to build an
2367Autotool-based application. The result of the build is automatically
2368packaged. And, if the application uses NLS for localization, packages
2369with local information are generated (one package per language).
2370Following is one example: (``hello_2.3.bb``)
2371::
2372
2373 SUMMARY = "GNU Helloworld application"
2374 SECTION = "examples"
2375 LICENSE = "GPL-2.0-or-later"
2376 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2377
2378 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2379
2380 inherit autotools gettext
2381
2382The variable :term:`LIC_FILES_CHKSUM` is used to track source license
2383changes as described in the
2384":ref:`dev-manual/common-tasks:tracking license changes`" section in
2385the Yocto Project Overview and Concepts Manual. You can quickly create
2386Autotool-based recipes in a manner similar to the previous example.
2387
2388Makefile-Based Package
2389~~~~~~~~~~~~~~~~~~~~~~
2390
2391Applications that use GNU ``make`` also require a recipe that has the
2392source archive listed in :term:`SRC_URI`. You do not need to add a
2393:ref:`ref-tasks-compile` step since by default BitBake starts the ``make`` command
2394to compile the application. If you need additional ``make`` options, you
2395should store them in the
2396:term:`EXTRA_OEMAKE` or
2397:term:`PACKAGECONFIG_CONFARGS`
2398variables. BitBake passes these options into the GNU ``make``
2399invocation. Note that a :ref:`ref-tasks-install` task is still required.
2400Otherwise, BitBake runs an empty :ref:`ref-tasks-install` task by default.
2401
2402Some applications might require extra parameters to be passed to the
2403compiler. For example, the application might need an additional header
2404path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
2405following example shows this::
2406
2407 CFLAGS:prepend = "-I ${S}/include "
2408
2409In the following example, ``lz4`` is a makefile-based package::
2410
2411 SUMMARY = "Extremely Fast Compression algorithm"
2412 DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems."
2413 HOMEPAGE = "https://github.com/lz4/lz4"
2414
2415 LICENSE = "BSD-2-Clause | GPL-2.0-only"
2416 LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc \
2417 file://programs/COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
2418 file://LICENSE;md5=d57c0d21cb917fb4e0af2454aa48b956 \
2419 "
2420
2421 PE = "1"
2422
2423 SRCREV = "d44371841a2f1728a3f36839fd4b7e872d0927d3"
2424
2425 SRC_URI = "git://github.com/lz4/lz4.git;branch=release;protocol=https \
2426 file://CVE-2021-3520.patch \
2427 "
2428 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>.*)"
2429
2430 S = "${WORKDIR}/git"
2431
2432 # Fixed in r118, which is larger than the current version.
2433 CVE_CHECK_IGNORE += "CVE-2014-4715"
2434
2435 EXTRA_OEMAKE = "PREFIX=${prefix} CC='${CC}' CFLAGS='${CFLAGS}' DESTDIR=${D} LIBDIR=${libdir} INCLUDEDIR=${includedir} BUILD_STATIC=no"
2436
2437 do_install() {
2438 oe_runmake install
2439 }
2440
2441 BBCLASSEXTEND = "native nativesdk"
2442
2443Splitting an Application into Multiple Packages
2444~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2445
2446You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
2447application into multiple packages.
2448
2449Following is an example that uses the ``libxpm`` recipe. By default,
2450this recipe generates a single package that contains the library along
2451with a few binaries. You can modify the recipe to split the binaries
2452into separate packages::
2453
2454 require xorg-lib-common.inc
2455
2456 SUMMARY = "Xpm: X Pixmap extension library"
2457 LICENSE = "MIT"
2458 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2459 DEPENDS += "libxext libsm libxt"
2460 PE = "1"
2461
2462 XORG_PN = "libXpm"
2463
2464 PACKAGES =+ "sxpm cxpm"
2465 FILES:cxpm = "${bindir}/cxpm"
2466 FILES:sxpm = "${bindir}/sxpm"
2467
2468In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2469binaries in separate packages. Since ``bindir`` would be packaged into
2470the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
2471so additional package names are added to the start of list. This results
2472in the extra ``FILES:*`` variables then containing information that
2473define which files and directories go into which packages. Files
2474included by earlier packages are skipped by latter packages. Thus, the
2475main :term:`PN` package does not include the above listed files.
2476
2477Packaging Externally Produced Binaries
2478~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2479
2480Sometimes, you need to add pre-compiled binaries to an image. For
2481example, suppose that there are binaries for proprietary code,
2482created by a particular division of a company. Your part of the company
2483needs to use those binaries as part of an image that you are building
2484using the OpenEmbedded build system. Since you only have the binaries
2485and not the source code, you cannot use a typical recipe that expects to
2486fetch the source specified in
2487:term:`SRC_URI` and then compile it.
2488
2489One method is to package the binaries and then install them as part of
2490the image. Generally, it is not a good idea to package binaries since,
2491among other things, it can hinder the ability to reproduce builds and
2492could lead to compatibility problems with ABI in the future. However,
2493sometimes you have no choice.
2494
2495The easiest solution is to create a recipe that uses the
2496:ref:`bin_package <ref-classes-bin-package>` class
2497and to be sure that you are using default locations for build artifacts.
2498In most cases, the :ref:`bin_package <ref-classes-bin-package>` class handles "skipping" the
2499configure and compile steps as well as sets things up to grab packages
2500from the appropriate area. In particular, this class sets ``noexec`` on
2501both the :ref:`ref-tasks-configure`
2502and :ref:`ref-tasks-compile` tasks,
2503sets ``FILES:${PN}`` to "/" so that it picks up all files, and sets up a
2504:ref:`ref-tasks-install` task, which
2505effectively copies all files from ``${S}`` to ``${D}``. The
2506:ref:`bin_package <ref-classes-bin-package>` class works well when the files extracted into ``${S}``
2507are already laid out in the way they should be laid out on the target.
2508For more information on these variables, see the
2509:term:`FILES`,
2510:term:`PN`,
2511:term:`S`, and
2512:term:`D` variables in the Yocto Project
2513Reference Manual's variable glossary.
2514
2515.. note::
2516
2517 - Using :term:`DEPENDS` is a good
2518 idea even for components distributed in binary form, and is often
2519 necessary for shared libraries. For a shared library, listing the
2520 library dependencies in :term:`DEPENDS` makes sure that the libraries
2521 are available in the staging sysroot when other recipes link
2522 against the library, which might be necessary for successful
2523 linking.
2524
2525 - Using :term:`DEPENDS` also allows runtime dependencies between
2526 packages to be added automatically. See the
2527 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
2528 section in the Yocto Project Overview and Concepts Manual for more
2529 information.
2530
2531If you cannot use the :ref:`bin_package <ref-classes-bin-package>` class, you need to be sure you are
2532doing the following:
2533
2534- Create a recipe where the
2535 :ref:`ref-tasks-configure` and
2536 :ref:`ref-tasks-compile` tasks do
2537 nothing: It is usually sufficient to just not define these tasks in
2538 the recipe, because the default implementations do nothing unless a
2539 Makefile is found in
2540 ``${``\ :term:`S`\ ``}``.
2541
2542 If ``${S}`` might contain a Makefile, or if you inherit some class
2543 that replaces :ref:`ref-tasks-configure` and :ref:`ref-tasks-compile` with custom
2544 versions, then you can use the
2545 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2546 flag to turn the tasks into no-ops, as follows::
2547
2548 do_configure[noexec] = "1"
2549 do_compile[noexec] = "1"
2550
2551 Unlike
2552 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2553 using the flag preserves the dependency chain from the
2554 :ref:`ref-tasks-fetch`,
2555 :ref:`ref-tasks-unpack`, and
2556 :ref:`ref-tasks-patch` tasks to the
2557 :ref:`ref-tasks-install` task.
2558
2559- Make sure your :ref:`ref-tasks-install` task installs the binaries
2560 appropriately.
2561
2562- Ensure that you set up :term:`FILES`
2563 (usually
2564 ``FILES:${``\ :term:`PN`\ ``}``) to
2565 point to the files you have installed, which of course depends on
2566 where you have installed them and whether those files are in
2567 different locations than the defaults.
2568
2569Following Recipe Style Guidelines
2570---------------------------------
2571
2572When writing recipes, it is good to conform to existing style
2573guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
2574provides rough guidelines for preferred recipe style.
2575
2576It is common for existing recipes to deviate a bit from this style.
2577However, aiming for at least a consistent style is a good idea. Some
2578practices, such as omitting spaces around ``=`` operators in assignments
2579or ordering recipe components in an erratic way, are widely seen as poor
2580style.
2581
2582Recipe Syntax
2583-------------
2584
2585Understanding recipe file syntax is important for writing recipes. The
2586following list overviews the basic items that make up a BitBake recipe
2587file. For more complete BitBake syntax descriptions, see the
2588":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`"
2589chapter of the BitBake User Manual.
2590
2591- *Variable Assignments and Manipulations:* Variable assignments allow
2592 a value to be assigned to a variable. The assignment can be static
2593 text or might include the contents of other variables. In addition to
2594 the assignment, appending and prepending operations are also
2595 supported.
2596
2597 The following example shows some of the ways you can use variables in
2598 recipes::
2599
2600 S = "${WORKDIR}/postfix-${PV}"
2601 CFLAGS += "-DNO_ASM"
2602 CFLAGS:append = " --enable-important-feature"
2603
2604- *Functions:* Functions provide a series of actions to be performed.
2605 You usually use functions to override the default implementation of a
2606 task function or to complement a default function (i.e. append or
2607 prepend to an existing function). Standard functions use ``sh`` shell
2608 syntax, although access to OpenEmbedded variables and internal
2609 methods are also available.
2610
2611 Here is an example function from the ``sed`` recipe::
2612
2613 do_install () {
2614 autotools_do_install
2615 install -d ${D}${base_bindir}
2616 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2617 rmdir ${D}${bindir}/
2618 }
2619
2620 It is
2621 also possible to implement new functions that are called between
2622 existing tasks as long as the new functions are not replacing or
2623 complementing the default functions. You can implement functions in
2624 Python instead of shell. Both of these options are not seen in the
2625 majority of recipes.
2626
2627- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2628 to include common functions (``inherit``), load parts of a recipe
2629 from other files (``include`` and ``require``) and export variables
2630 to the environment (``export``).
2631
2632 The following example shows the use of some of these keywords::
2633
2634 export POSTCONF = "${STAGING_BINDIR}/postconf"
2635 inherit autoconf
2636 require otherfile.inc
2637
2638- *Comments (#):* Any lines that begin with the hash character (``#``)
2639 are treated as comment lines and are ignored::
2640
2641 # This is a comment
2642
2643This next list summarizes the most important and most commonly used
2644parts of the recipe syntax. For more information on these parts of the
2645syntax, you can reference the
2646":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
2647in the BitBake User Manual.
2648
2649- *Line Continuation (\\):* Use the backward slash (``\``) character to
2650 split a statement over multiple lines. Place the slash character at
2651 the end of the line that is to be continued on the next line::
2652
2653 VAR = "A really long \
2654 line"
2655
2656 .. note::
2657
2658 You cannot have any characters including spaces or tabs after the
2659 slash character.
2660
2661- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
2662 access the contents of a variable::
2663
2664 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2665
2666 .. note::
2667
2668 It is important to understand that the value of a variable
2669 expressed in this form does not get substituted automatically. The
2670 expansion of these expressions happens on-demand later (e.g.
2671 usually when a function that makes reference to the variable
2672 executes). This behavior ensures that the values are most
2673 appropriate for the context in which they are finally used. On the
2674 rare occasion that you do need the variable expression to be
2675 expanded immediately, you can use the
2676 :=
2677 operator instead of
2678 =
2679 when you make the assignment, but this is not generally needed.
2680
2681- *Quote All Assignments ("value"):* Use double quotes around values in
2682 all variable assignments (e.g. ``"value"``). Following is an example::
2683
2684 VAR1 = "${OTHERVAR}"
2685 VAR2 = "The version is ${PV}"
2686
2687- *Conditional Assignment (?=):* Conditional assignment is used to
2688 assign a value to a variable, but only when the variable is currently
2689 unset. Use the question mark followed by the equal sign (``?=``) to
2690 make a "soft" assignment used for conditional assignment. Typically,
2691 "soft" assignments are used in the ``local.conf`` file for variables
2692 that are allowed to come through from the external environment.
2693
2694 Here is an example where ``VAR1`` is set to "New value" if it is
2695 currently empty. However, if ``VAR1`` has already been set, it
2696 remains unchanged::
2697
2698 VAR1 ?= "New value"
2699
2700 In this next example, ``VAR1`` is left with the value "Original value"::
2701
2702 VAR1 = "Original value"
2703 VAR1 ?= "New value"
2704
2705- *Appending (+=):* Use the plus character followed by the equals sign
2706 (``+=``) to append values to existing variables.
2707
2708 .. note::
2709
2710 This operator adds a space between the existing content of the
2711 variable and the new content.
2712
2713 Here is an example::
2714
2715 SRC_URI += "file://fix-makefile.patch"
2716
2717- *Prepending (=+):* Use the equals sign followed by the plus character
2718 (``=+``) to prepend values to existing variables.
2719
2720 .. note::
2721
2722 This operator adds a space between the new content and the
2723 existing content of the variable.
2724
2725 Here is an example::
2726
2727 VAR =+ "Starts"
2728
2729- *Appending (:append):* Use the ``:append`` operator to append values
2730 to existing variables. This operator does not add any additional
2731 space. Also, the operator is applied after all the ``+=``, and ``=+``
2732 operators have been applied and after all ``=`` assignments have
2733 occurred. This means that if ``:append`` is used in a recipe, it can
2734 only be overridden by another layer using the special ``:remove``
2735 operator, which in turn will prevent further layers from adding it back.
2736
2737 The following example shows the space being explicitly added to the
2738 start to ensure the appended value is not merged with the existing
2739 value::
2740
2741 CFLAGS:append = " --enable-important-feature"
2742
2743 You can also use
2744 the ``:append`` operator with overrides, which results in the actions
2745 only being performed for the specified target or machine::
2746
2747 CFLAGS:append:sh4 = " --enable-important-sh4-specific-feature"
2748
2749- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend
2750 values to existing variables. This operator does not add any
2751 additional space. Also, the operator is applied after all the ``+=``,
2752 and ``=+`` operators have been applied and after all ``=``
2753 assignments have occurred.
2754
2755 The following example shows the space being explicitly added to the
2756 end to ensure the prepended value is not merged with the existing
2757 value::
2758
2759 CFLAGS:prepend = "-I${S}/myincludes "
2760
2761 You can also use the
2762 ``:prepend`` operator with overrides, which results in the actions
2763 only being performed for the specified target or machine::
2764
2765 CFLAGS:prepend:sh4 = "-I${S}/myincludes "
2766
2767- *Overrides:* You can use overrides to set a value conditionally,
2768 typically based on how the recipe is being built. For example, to set
2769 the :term:`KBRANCH` variable's
2770 value to "standard/base" for any target
2771 :term:`MACHINE`, except for
2772 qemuarm where it should be set to "standard/arm-versatile-926ejs",
2773 you would do the following::
2774
2775 KBRANCH = "standard/base"
2776 KBRANCH:qemuarm = "standard/arm-versatile-926ejs"
2777
2778 Overrides are also used to separate
2779 alternate values of a variable in other situations. For example, when
2780 setting variables such as
2781 :term:`FILES` and
2782 :term:`RDEPENDS` that are
2783 specific to individual packages produced by a recipe, you should
2784 always use an override that specifies the name of the package.
2785
2786- *Indentation:* Use spaces for indentation rather than tabs. For
2787 shell functions, both currently work. However, it is a policy
2788 decision of the Yocto Project to use tabs in shell functions. Realize
2789 that some layers have a policy to use spaces for all indentation.
2790
2791- *Using Python for Complex Operations:* For more advanced processing,
2792 it is possible to use Python code during variable assignments (e.g.
2793 search and replacement on a variable).
2794
2795 You indicate Python code using the ``${@python_code}`` syntax for the
2796 variable assignment::
2797
2798 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2799
2800- *Shell Function Syntax:* Write shell functions as if you were writing
2801 a shell script when you describe a list of actions to take. You
2802 should ensure that your script works with a generic ``sh`` and that
2803 it does not require any ``bash`` or other shell-specific
2804 functionality. The same considerations apply to various system
2805 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2806 might wish to use. If in doubt, you should check with multiple
2807 implementations --- including those from BusyBox.
2808
2809Adding a New Machine
2810====================
2811
2812Adding a new machine to the Yocto Project is a straightforward process.
2813This section describes how to add machines that are similar to those
2814that the Yocto Project already supports.
2815
2816.. note::
2817
2818 Although well within the capabilities of the Yocto Project, adding a
2819 totally new architecture might require changes to ``gcc``/``glibc``
2820 and to the site information, which is beyond the scope of this
2821 manual.
2822
2823For a complete example that shows how to add a new machine, see the
2824":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2825section in the Yocto Project Board Support Package (BSP) Developer's
2826Guide.
2827
2828Adding the Machine Configuration File
2829-------------------------------------
2830
2831To add a new machine, you need to add a new machine configuration file
2832to the layer's ``conf/machine`` directory. This configuration file
2833provides details about the device you are adding.
2834
2835The OpenEmbedded build system uses the root name of the machine
2836configuration file to reference the new machine. For example, given a
2837machine configuration file named ``crownbay.conf``, the build system
2838recognizes the machine as "crownbay".
2839
2840The most important variables you must set in your machine configuration
2841file or include from a lower-level configuration file are as follows:
2842
2843- :term:`TARGET_ARCH` (e.g. "arm")
2844
2845- ``PREFERRED_PROVIDER_virtual/kernel``
2846
2847- :term:`MACHINE_FEATURES` (e.g. "apm screen wifi")
2848
2849You might also need these variables:
2850
2851- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
2852
2853- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
2854
2855- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
2856
2857You can find full details on these variables in the reference section.
2858You can leverage existing machine ``.conf`` files from
2859``meta-yocto-bsp/conf/machine/``.
2860
2861Adding a Kernel for the Machine
2862-------------------------------
2863
2864The OpenEmbedded build system needs to be able to build a kernel for the
2865machine. You need to either create a new kernel recipe for this machine,
2866or extend an existing kernel recipe. You can find several kernel recipe
2867examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2868you can use as references.
2869
2870If you are creating a new kernel recipe, normal recipe-writing rules
2871apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
2872necessary patches and set :term:`S` to point at the source code. You need to
2873create a :ref:`ref-tasks-configure` task that configures the unpacked kernel with
2874a ``defconfig`` file. You can do this by using a ``make defconfig``
2875command or, more commonly, by copying in a suitable ``defconfig`` file
2876and then running ``make oldconfig``. By making use of ``inherit kernel``
2877and potentially some of the ``linux-*.inc`` files, most other
2878functionality is centralized and the defaults of the class normally work
2879well.
2880
2881If you are extending an existing kernel recipe, it is usually a matter
2882of adding a suitable ``defconfig`` file. The file needs to be added into
2883a location similar to ``defconfig`` files used for other machines in a
2884given kernel recipe. A possible way to do this is by listing the file in
2885the :term:`SRC_URI` and adding the machine to the expression in
2886:term:`COMPATIBLE_MACHINE`::
2887
2888 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2889
2890For more information on ``defconfig`` files, see the
2891":ref:`kernel-dev/common:changing the configuration`"
2892section in the Yocto Project Linux Kernel Development Manual.
2893
2894Adding a Formfactor Configuration File
2895--------------------------------------
2896
2897A formfactor configuration file provides information about the target
2898hardware for which the image is being built and information that the
2899build system cannot obtain from other sources such as the kernel. Some
2900examples of information contained in a formfactor configuration file
2901include framebuffer orientation, whether or not the system has a
2902keyboard, the positioning of the keyboard in relation to the screen, and
2903the screen resolution.
2904
2905The build system uses reasonable defaults in most cases. However, if
2906customization is necessary, you need to create a ``machconfig`` file in
2907the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2908contains directories for specific machines such as ``qemuarm`` and
2909``qemux86``. For information about the settings available and the
2910defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2911found in the same area.
2912
2913Following is an example for "qemuarm" machine::
2914
2915 HAVE_TOUCHSCREEN=1
2916 HAVE_KEYBOARD=1
2917 DISPLAY_CAN_ROTATE=0
2918 DISPLAY_ORIENTATION=0
2919 #DISPLAY_WIDTH_PIXELS=640
2920 #DISPLAY_HEIGHT_PIXELS=480
2921 #DISPLAY_BPP=16
2922 DISPLAY_DPI=150
2923 DISPLAY_SUBPIXEL_ORDER=vrgb
2924
2925Upgrading Recipes
2926=================
2927
2928Over time, upstream developers publish new versions for software built
2929by layer recipes. It is recommended to keep recipes up-to-date with
2930upstream version releases.
2931
2932While there are several methods to upgrade a recipe, you might
2933consider checking on the upgrade status of a recipe first. You can do so
2934using the ``devtool check-upgrade-status`` command. See the
2935":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2936section in the Yocto Project Reference Manual for more information.
2937
2938The remainder of this section describes three ways you can upgrade a
2939recipe. You can use the Automated Upgrade Helper (AUH) to set up
2940automatic version upgrades. Alternatively, you can use
2941``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
2942you can manually upgrade a recipe by editing the recipe itself.
2943
2944Using the Auto Upgrade Helper (AUH)
2945-----------------------------------
2946
2947The AUH utility works in conjunction with the OpenEmbedded build system
2948in order to automatically generate upgrades for recipes based on new
2949versions being published upstream. Use AUH when you want to create a
2950service that performs the upgrades automatically and optionally sends
2951you an email with the results.
2952
2953AUH allows you to update several recipes with a single use. You can also
2954optionally perform build and integration tests using images with the
2955results saved to your hard drive and emails of results optionally sent
2956to recipe maintainers. Finally, AUH creates Git commits with appropriate
2957commit messages in the layer's tree for the changes made to recipes.
2958
2959.. note::
2960
2961 In some conditions, you should not use AUH to upgrade recipes
2962 and should instead use either ``devtool upgrade`` or upgrade your
2963 recipes manually:
2964
2965 - When AUH cannot complete the upgrade sequence. This situation
2966 usually results because custom patches carried by the recipe
2967 cannot be automatically rebased to the new version. In this case,
2968 ``devtool upgrade`` allows you to manually resolve conflicts.
2969
2970 - When for any reason you want fuller control over the upgrade
2971 process. For example, when you want special arrangements for
2972 testing.
2973
2974The following steps describe how to set up the AUH utility:
2975
29761. *Be Sure the Development Host is Set Up:* You need to be sure that
2977 your development host is set up to use the Yocto Project. For
2978 information on how to set up your host, see the
2979 ":ref:`dev-manual/start:Preparing the Build Host`" section.
2980
29812. *Make Sure Git is Configured:* The AUH utility requires Git to be
2982 configured because AUH uses Git to save upgrades. Thus, you must have
2983 Git user and email configured. The following command shows your
2984 configurations::
2985
2986 $ git config --list
2987
2988 If you do not have the user and
2989 email configured, you can use the following commands to do so::
2990
2991 $ git config --global user.name some_name
2992 $ git config --global user.email username@domain.com
2993
29943. *Clone the AUH Repository:* To use AUH, you must clone the repository
2995 onto your development host. The following command uses Git to create
2996 a local copy of the repository on your system::
2997
2998 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
2999 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
3000 remote: Compressing objects: 100% (300/300), done.
3001 remote: Total 768 (delta 499), reused 703 (delta 434)
3002 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
3003 Resolving deltas: 100% (499/499), done.
3004 Checking connectivity... done.
3005
3006 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
3007 :term:`Poky` repositories.
3008
30094. *Create a Dedicated Build Directory:* Run the :ref:`structure-core-script`
3010 script to create a fresh :term:`Build Directory` that you use exclusively
3011 for running the AUH utility::
3012
3013 $ cd poky
3014 $ source oe-init-build-env your_AUH_build_directory
3015
3016 Re-using an existing :term:`Build Directory` and its configurations is not
3017 recommended as existing settings could cause AUH to fail or behave
3018 undesirably.
3019
30205. *Make Configurations in Your Local Configuration File:* Several
3021 settings are needed in the ``local.conf`` file in the build
3022 directory you just created for AUH. Make these following
3023 configurations:
3024
3025 - If you want to enable :ref:`Build
3026 History <dev-manual/common-tasks:maintaining build output quality>`,
3027 which is optional, you need the following lines in the
3028 ``conf/local.conf`` file::
3029
3030 INHERIT =+ "buildhistory"
3031 BUILDHISTORY_COMMIT = "1"
3032
3033 With this configuration and a successful
3034 upgrade, a build history "diff" file appears in the
3035 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3036 your :term:`Build Directory`.
3037
3038 - If you want to enable testing through the
3039 :ref:`testimage <ref-classes-testimage>`
3040 class, which is optional, you need to have the following set in
3041 your ``conf/local.conf`` file::
3042
3043 INHERIT += "testimage"
3044
3045 .. note::
3046
3047 If your distro does not enable by default ptest, which Poky
3048 does, you need the following in your ``local.conf`` file::
3049
3050 DISTRO_FEATURES:append = " ptest"
3051
3052
30536. *Optionally Start a vncserver:* If you are running in a server
3054 without an X11 session, you need to start a vncserver::
3055
3056 $ vncserver :1
3057 $ export DISPLAY=:1
3058
30597. *Create and Edit an AUH Configuration File:* You need to have the
3060 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
3061 :term:`Build Directory`. You can find a sample configuration file in the
3062 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
3063
3064 Read through the sample file and make configurations as needed. For
3065 example, if you enabled build history in your ``local.conf`` as
3066 described earlier, you must enable it in ``upgrade-helper.conf``.
3067
3068 Also, if you are using the default ``maintainers.inc`` file supplied
3069 with Poky and located in ``meta-yocto`` and you do not set a
3070 "maintainers_whitelist" or "global_maintainer_override" in the
3071 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3072 the AUH command-line, the utility automatically sends out emails to
3073 all the default maintainers. Please avoid this.
3074
3075This next set of examples describes how to use the AUH:
3076
3077- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
3078 following form::
3079
3080 $ upgrade-helper.py recipe_name
3081
3082 For example, this command upgrades the ``xmodmap`` recipe::
3083
3084 $ upgrade-helper.py xmodmap
3085
3086- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
3087 specific recipe to a particular version, use the following form::
3088
3089 $ upgrade-helper.py recipe_name -t version
3090
3091 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
3092
3093 $ upgrade-helper.py xmodmap -t 1.2.3
3094
3095- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3096 Notifications:* To upgrade all recipes to their most recent versions
3097 and suppress the email notifications, use the following command::
3098
3099 $ upgrade-helper.py all
3100
3101- *Upgrading all Recipes to the Latest Versions and Send Email
3102 Notifications:* To upgrade all recipes to their most recent versions
3103 and send email messages to maintainers for each attempted recipe as
3104 well as a status email, use the following command::
3105
3106 $ upgrade-helper.py -e all
3107
3108Once you have run the AUH utility, you can find the results in the AUH
3109:term:`Build Directory`::
3110
3111 ${BUILDDIR}/upgrade-helper/timestamp
3112
3113The AUH utility
3114also creates recipe update commits from successful upgrade attempts in
3115the layer tree.
3116
3117You can easily set up to run the AUH utility on a regular basis by using
3118a cron job. See the
3119:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
3120file distributed with the utility for an example.
3121
3122Using ``devtool upgrade``
3123-------------------------
3124
3125As mentioned earlier, an alternative method for upgrading recipes to
3126newer versions is to use
3127:doc:`devtool upgrade </ref-manual/devtool-reference>`.
3128You can read about ``devtool upgrade`` in general in the
3129":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
3130section in the Yocto Project Application Development and the Extensible
3131Software Development Kit (eSDK) Manual.
3132
3133To see all the command-line options available with ``devtool upgrade``,
3134use the following help command::
3135
3136 $ devtool upgrade -h
3137
3138If you want to find out what version a recipe is currently at upstream
3139without any attempt to upgrade your local version of the recipe, you can
3140use the following command::
3141
3142 $ devtool latest-version recipe_name
3143
3144As mentioned in the previous section describing AUH, ``devtool upgrade``
3145works in a less-automated manner than AUH. Specifically,
3146``devtool upgrade`` only works on a single recipe that you name on the
3147command line, cannot perform build and integration testing using images,
3148and does not automatically generate commits for changes in the source
3149tree. Despite all these "limitations", ``devtool upgrade`` updates the
3150recipe file to the new upstream version and attempts to rebase custom
3151patches contained by the recipe as needed.
3152
3153.. note::
3154
3155 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3156 of a "wrapper" application for ``devtool upgrade``.
3157
3158A typical scenario involves having used Git to clone an upstream
3159repository that you use during build operations. Because you have built the
3160recipe in the past, the layer is likely added to your
3161configuration already. If for some reason, the layer is not added, you
3162could add it easily using the
3163":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3164script. For example, suppose you use the ``nano.bb`` recipe from the
3165``meta-oe`` layer in the ``meta-openembedded`` repository. For this
3166example, assume that the layer has been cloned into following area::
3167
3168 /home/scottrif/meta-openembedded
3169
3170The following command from your :term:`Build Directory` adds the layer to
3171your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
3172
3173 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3174 NOTE: Starting bitbake server...
3175 Parsing recipes: 100% |##########################################| Time: 0:00:55
3176 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3177 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3178 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3179 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3180 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3181
3182For this example, assume that the ``nano.bb`` recipe that
3183is upstream has a 2.9.3 version number. However, the version in the
3184local repository is 2.7.4. The following command from your build
3185directory automatically upgrades the recipe for you:
3186
3187.. note::
3188
3189 Using the ``-V`` option is not necessary. Omitting the version number causes
3190 ``devtool upgrade`` to upgrade the recipe to the most recent version.
3191
3192::
3193
3194 $ devtool upgrade nano -V 2.9.3
3195 NOTE: Starting bitbake server...
3196 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3197 Parsing recipes: 100% |##########################################| Time: 0:00:46
3198 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3199 NOTE: Extracting current version source...
3200 NOTE: Resolving any missing task queue dependencies
3201 .
3202 .
3203 .
3204 NOTE: Executing SetScene Tasks
3205 NOTE: Executing RunQueue Tasks
3206 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3207 Adding changed files: 100% |#####################################| Time: 0:00:00
3208 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3209 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3210
3211Continuing with this example, you can use ``devtool build`` to build the
3212newly upgraded recipe::
3213
3214 $ devtool build nano
3215 NOTE: Starting bitbake server...
3216 Loading cache: 100% |################################################################################################| Time: 0:00:01
3217 Loaded 2040 entries from dependency cache.
3218 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3219 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3220 NOTE: Resolving any missing task queue dependencies
3221 .
3222 .
3223 .
3224 NOTE: Executing SetScene Tasks
3225 NOTE: Executing RunQueue Tasks
3226 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3227 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3228
3229Within the ``devtool upgrade`` workflow, you can
3230deploy and test your rebuilt software. For this example,
3231however, running ``devtool finish`` cleans up the workspace once the
3232source in your workspace is clean. This usually means using Git to stage
3233and submit commits for the changes generated by the upgrade process.
3234
3235Once the tree is clean, you can clean things up in this example with the
3236following command from the ``${BUILDDIR}/workspace/sources/nano``
3237directory::
3238
3239 $ devtool finish nano meta-oe
3240 NOTE: Starting bitbake server...
3241 Loading cache: 100% |################################################################################################| Time: 0:00:00
3242 Loaded 2040 entries from dependency cache.
3243 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3244 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3245 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3246 NOTE: Updating recipe nano_2.9.3.bb
3247 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3248 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3249 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3250
3251
3252Using the ``devtool finish`` command cleans up the workspace and creates a patch
3253file based on your commits. The tool puts all patch files back into the
3254source directory in a sub-directory named ``nano`` in this case.
3255
3256Manually Upgrading a Recipe
3257---------------------------
3258
3259If for some reason you choose not to upgrade recipes using
3260:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or
3261by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``,
3262you can manually edit the recipe files to upgrade the versions.
3263
3264.. note::
3265
3266 Manually updating multiple recipes scales poorly and involves many
3267 steps. The recommendation to upgrade recipe versions is through AUH
3268 or ``devtool upgrade``, both of which automate some steps and provide
3269 guidance for others needed for the manual process.
3270
3271To manually upgrade recipe versions, follow these general steps:
3272
32731. *Change the Version:* Rename the recipe such that the version (i.e.
3274 the :term:`PV` part of the recipe name)
3275 changes appropriately. If the version is not part of the recipe name,
3276 change the value as it is set for :term:`PV` within the recipe itself.
3277
32782. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
3279 is fetched from Git or some other version control system, update
3280 :term:`SRCREV` to point to the
3281 commit hash that matches the new version.
3282
32833. *Build the Software:* Try to build the recipe using BitBake. Typical
3284 build failures include the following:
3285
3286 - License statements were updated for the new version. For this
3287 case, you need to review any changes to the license and update the
3288 values of :term:`LICENSE` and
3289 :term:`LIC_FILES_CHKSUM`
3290 as needed.
3291
3292 .. note::
3293
3294 License changes are often inconsequential. For example, the
3295 license text's copyright year might have changed.
3296
3297 - Custom patches carried by the older version of the recipe might
3298 fail to apply to the new version. For these cases, you need to
3299 review the failures. Patches might not be necessary for the new
3300 version of the software if the upgraded version has fixed those
3301 issues. If a patch is necessary and failing, you need to rebase it
3302 into the new version.
3303
33044. *Optionally Attempt to Build for Several Architectures:* Once you
3305 successfully build the new software for a given architecture, you
3306 could test the build for other architectures by changing the
3307 :term:`MACHINE` variable and
3308 rebuilding the software. This optional step is especially important
3309 if the recipe is to be released publicly.
3310
33115. *Check the Upstream Change Log or Release Notes:* Checking both these
3312 reveals if there are new features that could break
3313 backwards-compatibility. If so, you need to take steps to mitigate or
3314 eliminate that situation.
3315
33166. *Optionally Create a Bootable Image and Test:* If you want, you can
3317 test the new software by booting it onto actual hardware.
3318
33197. *Create a Commit with the Change in the Layer Repository:* After all
3320 builds work and any testing is successful, you can create commits for
3321 any changes in the layer holding your upgraded recipe.
3322
3323Finding Temporary Source Code
3324=============================
3325
3326You might find it helpful during development to modify the temporary
3327source code used by recipes to build packages. For example, suppose you
3328are developing a patch and you need to experiment a bit to figure out
3329your solution. After you have initially built the package, you can
3330iteratively tweak the source code, which is located in the
3331:term:`Build Directory`, and then you can force a re-compile and quickly
3332test your altered code. Once you settle on a solution, you can then preserve
3333your changes in the form of patches.
3334
3335During a build, the unpacked temporary source code used by recipes to
3336build packages is available in the :term:`Build Directory` as defined by the
3337:term:`S` variable. Below is the default value for the :term:`S` variable as
3338defined in the ``meta/conf/bitbake.conf`` configuration file in the
3339:term:`Source Directory`::
3340
3341 S = "${WORKDIR}/${BP}"
3342
3343You should be aware that many recipes override the
3344:term:`S` variable. For example, recipes that fetch their source from Git
3345usually set :term:`S` to ``${WORKDIR}/git``.
3346
3347.. note::
3348
3349 The :term:`BP` represents the base recipe name, which consists of the name
3350 and version::
3351
3352 BP = "${BPN}-${PV}"
3353
3354
3355The path to the work directory for the recipe
3356(:term:`WORKDIR`) is defined as
3357follows::
3358
3359 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3360
3361The actual directory depends on several things:
3362
3363- :term:`TMPDIR`: The top-level build
3364 output directory.
3365
3366- :term:`MULTIMACH_TARGET_SYS`:
3367 The target system identifier.
3368
3369- :term:`PN`: The recipe name.
3370
3371- :term:`EXTENDPE`: The epoch --- if
3372 :term:`PE` is not specified, which is
3373 usually the case for most recipes, then :term:`EXTENDPE` is blank.
3374
3375- :term:`PV`: The recipe version.
3376
3377- :term:`PR`: The recipe revision.
3378
3379As an example, assume a Source Directory top-level folder named
3380``poky``, a default :term:`Build Directory` at ``poky/build``, and a
3381``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3382recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
3383build system uses to build the package would be as follows::
3384
3385 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3386
3387Using Quilt in Your Workflow
3388============================
3389
3390`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
3391that allows you to capture source code changes without having a clean
3392source tree. This section outlines the typical workflow you can use to
3393modify source code, test changes, and then preserve the changes in the
3394form of a patch all using Quilt.
3395
3396.. note::
3397
3398 With regard to preserving changes to source files, if you clean a
3399 recipe or have :ref:`rm_work <ref-classes-rm-work>` enabled, the
3400 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
3401 as described in the Yocto Project Application Development and the
3402 Extensible Software Development Kit (eSDK) manual is a safer
3403 development flow than the flow that uses Quilt.
3404
3405Follow these general steps:
3406
34071. *Find the Source Code:* Temporary source code used by the
3408 OpenEmbedded build system is kept in the :term:`Build Directory`. See the
3409 ":ref:`dev-manual/common-tasks:finding temporary source code`" section to
3410 learn how to locate the directory that has the temporary source code for a
3411 particular package.
3412
34132. *Change Your Working Directory:* You need to be in the directory that
3414 has the temporary source code. That directory is defined by the
3415 :term:`S` variable.
3416
34173. *Create a New Patch:* Before modifying source code, you need to
3418 create a new patch. To create a new patch file, use ``quilt new`` as
3419 below::
3420
3421 $ quilt new my_changes.patch
3422
34234. *Notify Quilt and Add Files:* After creating the patch, you need to
3424 notify Quilt about the files you plan to edit. You notify Quilt by
3425 adding the files to the patch you just created::
3426
3427 $ quilt add file1.c file2.c file3.c
3428
34295. *Edit the Files:* Make your changes in the source code to the files
3430 you added to the patch.
3431
34326. *Test Your Changes:* Once you have modified the source code, the
3433 easiest way to test your changes is by calling the :ref:`ref-tasks-compile`
3434 task as shown in the following example::
3435
3436 $ bitbake -c compile -f package
3437
3438 The ``-f`` or ``--force`` option forces the specified task to
3439 execute. If you find problems with your code, you can just keep
3440 editing and re-testing iteratively until things work as expected.
3441
3442 .. note::
3443
3444 All the modifications you make to the temporary source code disappear
3445 once you run the :ref:`ref-tasks-clean` or :ref:`ref-tasks-cleanall`
3446 tasks using BitBake (i.e. ``bitbake -c clean package`` and
3447 ``bitbake -c cleanall package``). Modifications will also disappear if
3448 you use the :ref:`rm_work <ref-classes-rm-work>` feature as described in
3449 the ":ref:`dev-manual/common-tasks:conserving disk space during builds`"
3450 section.
3451
34527. *Generate the Patch:* Once your changes work as expected, you need to
3453 use Quilt to generate the final patch that contains all your
3454 modifications.
3455 ::
3456
3457 $ quilt refresh
3458
3459 At this point, the
3460 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3461 ``file2.c``, and ``file3.c`` files.
3462
3463 You can find the resulting patch file in the ``patches/``
3464 subdirectory of the source (:term:`S`) directory.
3465
34668. *Copy the Patch File:* For simplicity, copy the patch file into a
3467 directory named ``files``, which you can create in the same directory
3468 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3469 file. Placing the patch here guarantees that the OpenEmbedded build
3470 system will find the patch. Next, add the patch into the :term:`SRC_URI`
3471 of the recipe. Here is an example::
3472
3473 SRC_URI += "file://my_changes.patch"
3474
3475Using a Development Shell
3476=========================
3477
3478When debugging certain commands or even when just editing packages,
3479``devshell`` can be a useful tool. When you invoke ``devshell``, all
3480tasks up to and including
3481:ref:`ref-tasks-patch` are run for the
3482specified target. Then, a new terminal is opened and you are placed in
3483``${``\ :term:`S`\ ``}``, the source
3484directory. In the new terminal, all the OpenEmbedded build-related
3485environment variables are still defined so you can use commands such as
3486``configure`` and ``make``. The commands execute just as if the
3487OpenEmbedded build system were executing them. Consequently, working
3488this way can be helpful when debugging a build or preparing software to
3489be used with the OpenEmbedded build system.
3490
3491Following is an example that uses ``devshell`` on a target named
3492``matchbox-desktop``::
3493
3494 $ bitbake matchbox-desktop -c devshell
3495
3496This command spawns a terminal with a shell prompt within the
3497OpenEmbedded build environment. The
3498:term:`OE_TERMINAL` variable
3499controls what type of shell is opened.
3500
3501For spawned terminals, the following occurs:
3502
3503- The ``PATH`` variable includes the cross-toolchain.
3504
3505- The ``pkgconfig`` variables find the correct ``.pc`` files.
3506
3507- The ``configure`` command finds the Yocto Project site files as well
3508 as any other necessary files.
3509
3510Within this environment, you can run configure or compile commands as if
3511they were being run by the OpenEmbedded build system itself. As noted
3512earlier, the working directory also automatically changes to the Source
3513Directory (:term:`S`).
3514
3515To manually run a specific task using ``devshell``, run the
3516corresponding ``run.*`` script in the
3517``${``\ :term:`WORKDIR`\ ``}/temp``
3518directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
3519not exist, which would be the case if the task was skipped by way of the
3520sstate cache, you can create the task by first running it outside of the
3521``devshell``::
3522
3523 $ bitbake -c task
3524
3525.. note::
3526
3527 - Execution of a task's ``run.*`` script and BitBake's execution of
3528 a task are identical. In other words, running the script re-runs
3529 the task just as it would be run using the ``bitbake -c`` command.
3530
3531 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3532 symbolic link (symlink) to the most recent version of that file.
3533
3534Remember, that the ``devshell`` is a mechanism that allows you to get
3535into the BitBake task execution environment. And as such, all commands
3536must be called just as BitBake would call them. That means you need to
3537provide the appropriate options for cross-compilation and so forth as
3538applicable.
3539
3540When you are finished using ``devshell``, exit the shell or close the
3541terminal window.
3542
3543.. note::
3544
3545 - It is worth remembering that when using ``devshell`` you need to
3546 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3547 instead of just using ``gcc``. The same applies to other
3548 applications such as ``binutils``, ``libtool`` and so forth.
3549 BitBake sets up environment variables such as :term:`CC` to assist
3550 applications, such as ``make`` to find the correct tools.
3551
3552 - It is also worth noting that ``devshell`` still works over X11
3553 forwarding and similar situations.
3554
3555Using a Python Development Shell
3556================================
3557
3558Similar to working within a development shell as described in the
3559previous section, you can also spawn and work within an interactive
3560Python development shell. When debugging certain commands or even when
3561just editing packages, ``pydevshell`` can be a useful tool. When you
3562invoke the ``pydevshell`` task, all tasks up to and including
3563:ref:`ref-tasks-patch` are run for the
3564specified target. Then a new terminal is opened. Additionally, key
3565Python objects and code are available in the same way they are to
3566BitBake tasks, in particular, the data store 'd'. So, commands such as
3567the following are useful when exploring the data store and running
3568functions::
3569
3570 pydevshell> d.getVar("STAGING_DIR")
3571 '/media/build1/poky/build/tmp/sysroots'
3572 pydevshell> d.getVar("STAGING_DIR", False)
3573 '${TMPDIR}/sysroots'
3574 pydevshell> d.setVar("FOO", "bar")
3575 pydevshell> d.getVar("FOO")
3576 'bar'
3577 pydevshell> d.delVar("FOO")
3578 pydevshell> d.getVar("FOO")
3579 pydevshell> bb.build.exec_func("do_unpack", d)
3580 pydevshell>
3581
3582See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:functions you can call from within python`"
3583section in the BitBake User Manual for details about available functions.
3584
3585The commands execute just as if the OpenEmbedded build
3586system were executing them. Consequently, working this way can be
3587helpful when debugging a build or preparing software to be used with the
3588OpenEmbedded build system.
3589
3590Following is an example that uses ``pydevshell`` on a target named
3591``matchbox-desktop``::
3592
3593 $ bitbake matchbox-desktop -c pydevshell
3594
3595This command spawns a terminal and places you in an interactive Python
3596interpreter within the OpenEmbedded build environment. The
3597:term:`OE_TERMINAL` variable
3598controls what type of shell is opened.
3599
3600When you are finished using ``pydevshell``, you can exit the shell
3601either by using Ctrl+d or closing the terminal window.
3602
3603Building
3604========
3605
3606This section describes various build procedures, such as the steps
3607needed for a simple build, building a target for multiple configurations,
3608generating an image for more than one machine, and so forth.
3609
3610Building a Simple Image
3611-----------------------
3612
3613In the development environment, you need to build an image whenever you
3614change hardware support, add or change system libraries, or add or
3615change services that have dependencies. There are several methods that allow
3616you to build an image within the Yocto Project. This section presents
3617the basic steps you need to build a simple image using BitBake from a
3618build host running Linux.
3619
3620.. note::
3621
3622 - For information on how to build an image using
3623 :term:`Toaster`, see the
3624 :doc:`/toaster-manual/index`.
3625
3626 - For information on how to use ``devtool`` to build images, see the
3627 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
3628 section in the Yocto Project Application Development and the
3629 Extensible Software Development Kit (eSDK) manual.
3630
3631 - For a quick example on how to build an image using the
3632 OpenEmbedded build system, see the
3633 :doc:`/brief-yoctoprojectqs/index` document.
3634
3635The build process creates an entire Linux distribution from source and
3636places it in your :term:`Build Directory` under ``tmp/deploy/images``. For
3637detailed information on the build process using BitBake, see the
3638":ref:`overview-manual/concepts:images`" section in the Yocto Project Overview
3639and Concepts Manual.
3640
3641The following figure and list overviews the build process:
3642
3643.. image:: figures/bitbake-build-flow.png
3644 :width: 100%
3645
36461. *Set up Your Host Development System to Support Development Using the
3647 Yocto Project*: See the ":doc:`start`" section for options on how to get a
3648 build host ready to use the Yocto Project.
3649
36502. *Initialize the Build Environment:* Initialize the build environment
3651 by sourcing the build environment script (i.e.
3652 :ref:`structure-core-script`)::
3653
3654 $ source oe-init-build-env [build_dir]
3655
3656 When you use the initialization script, the OpenEmbedded build system
3657 uses ``build`` as the default :term:`Build Directory` in your current work
3658 directory. You can use a `build_dir` argument with the script to
3659 specify a different :term:`Build Directory`.
3660
3661 .. note::
3662
3663 A common practice is to use a different :term:`Build Directory` for
3664 different targets; for example, ``~/build/x86`` for a ``qemux86``
3665 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
3666 event, it's typically cleaner to locate the :term:`Build Directory`
3667 somewhere outside of your source directory.
3668
36693. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
3670 ``conf/local.conf`` configuration file, which is found in the
3671 :term:`Build Directory`, is set up how you want it. This file defines many
3672 aspects of the build environment including the target machine architecture
3673 through the :term:`MACHINE` variable, the packaging format used during
3674 the build (:term:`PACKAGE_CLASSES`), and a centralized tarball download
3675 directory through the :term:`DL_DIR` variable.
3676
36774. *Build the Image:* Build the image using the ``bitbake`` command::
3678
3679 $ bitbake target
3680
3681 .. note::
3682
3683 For information on BitBake, see the :doc:`bitbake:index`.
3684
3685 The target is the name of the recipe you want to build. Common
3686 targets are the images in ``meta/recipes-core/images``,
3687 ``meta/recipes-sato/images``, and so forth all found in the
3688 :term:`Source Directory`. Alternatively, the target
3689 can be the name of a recipe for a specific piece of software such as
3690 BusyBox. For more details about the images the OpenEmbedded build
3691 system supports, see the
3692 ":ref:`ref-manual/images:Images`" chapter in the Yocto
3693 Project Reference Manual.
3694
3695 As an example, the following command builds the
3696 ``core-image-minimal`` image::
3697
3698 $ bitbake core-image-minimal
3699
3700 Once an
3701 image has been built, it often needs to be installed. The images and
3702 kernels built by the OpenEmbedded build system are placed in the
3703 :term:`Build Directory` in ``tmp/deploy/images``. For information on how to
3704 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
3705 :doc:`/sdk-manual/index` manual. For
3706 information about how to install these images, see the documentation
3707 for your particular board or machine.
3708
3709Building Images for Multiple Targets Using Multiple Configurations
3710------------------------------------------------------------------
3711
3712You can use a single ``bitbake`` command to build multiple images or
3713packages for different targets where each image or package requires a
3714different configuration (multiple configuration builds). The builds, in
3715this scenario, are sometimes referred to as "multiconfigs", and this
3716section uses that term throughout.
3717
3718This section describes how to set up for multiple configuration builds
3719and how to account for cross-build dependencies between the
3720multiconfigs.
3721
3722Setting Up and Running a Multiple Configuration Build
3723~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3724
3725To accomplish a multiple configuration build, you must define each
3726target's configuration separately using a parallel configuration file in
3727the :term:`Build Directory` or configuration directory within a layer, and you
3728must follow a required file hierarchy. Additionally, you must enable the
3729multiple configuration builds in your ``local.conf`` file.
3730
3731Follow these steps to set up and execute multiple configuration builds:
3732
3733- *Create Separate Configuration Files*: You need to create a single
3734 configuration file for each build target (each multiconfig).
3735 The configuration definitions are implementation dependent but often
3736 each configuration file will define the machine and the
3737 temporary directory BitBake uses for the build. Whether the same
3738 temporary directory (:term:`TMPDIR`) can be shared will depend on what is
3739 similar and what is different between the configurations. Multiple MACHINE
3740 targets can share the same (:term:`TMPDIR`) as long as the rest of the
3741 configuration is the same, multiple DISTRO settings would need separate
3742 (:term:`TMPDIR`) directories.
3743
3744 For example, consider a scenario with two different multiconfigs for the same
3745 :term:`MACHINE`: "qemux86" built
3746 for two distributions such as "poky" and "poky-lsb". In this case,
3747 you would need to use the different :term:`TMPDIR`.
3748
3749 Here is an example showing the minimal statements needed in a
3750 configuration file for a "qemux86" target whose temporary build
3751 directory is ``tmpmultix86``::
3752
3753 MACHINE = "qemux86"
3754 TMPDIR = "${TOPDIR}/tmpmultix86"
3755
3756 The location for these multiconfig configuration files is specific.
3757 They must reside in the current :term:`Build Directory` in a sub-directory of
3758 ``conf`` named ``multiconfig`` or within a layer's ``conf`` directory
3759 under a directory named ``multiconfig``. Following is an example that defines
3760 two configuration files for the "x86" and "arm" multiconfigs:
3761
3762 .. image:: figures/multiconfig_files.png
3763 :align: center
3764 :width: 50%
3765
3766 The usual :term:`BBPATH` search path is used to locate multiconfig files in
3767 a similar way to other conf files.
3768
3769- *Add the BitBake Multi-configuration Variable to the Local
3770 Configuration File*: Use the
3771 :term:`BBMULTICONFIG`
3772 variable in your ``conf/local.conf`` configuration file to specify
3773 each multiconfig. Continuing with the example from the previous
3774 figure, the :term:`BBMULTICONFIG` variable needs to enable two
3775 multiconfigs: "x86" and "arm" by specifying each configuration file::
3776
3777 BBMULTICONFIG = "x86 arm"
3778
3779 .. note::
3780
3781 A "default" configuration already exists by definition. This
3782 configuration is named: "" (i.e. empty string) and is defined by
3783 the variables coming from your ``local.conf``
3784 file. Consequently, the previous example actually adds two
3785 additional configurations to your build: "arm" and "x86" along
3786 with "".
3787
3788- *Launch BitBake*: Use the following BitBake command form to launch
3789 the multiple configuration build::
3790
3791 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3792
3793 For the example in this section, the following command applies::
3794
3795 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3796
3797 The previous BitBake command builds a ``core-image-minimal`` image
3798 that is configured through the ``x86.conf`` configuration file, a
3799 ``core-image-sato`` image that is configured through the ``arm.conf``
3800 configuration file and a ``core-image-base`` that is configured
3801 through your ``local.conf`` configuration file.
3802
3803.. note::
3804
3805 Support for multiple configuration builds in the Yocto Project &DISTRO;
3806 (&DISTRO_NAME;) Release does not include Shared State (sstate)
3807 optimizations. Consequently, if a build uses the same object twice
3808 in, for example, two different :term:`TMPDIR`
3809 directories, the build either loads from an existing sstate cache for
3810 that build at the start or builds the object fresh.
3811
3812Enabling Multiple Configuration Build Dependencies
3813~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3814
3815Sometimes dependencies can exist between targets (multiconfigs) in a
3816multiple configuration build. For example, suppose that in order to
3817build a ``core-image-sato`` image for an "x86" multiconfig, the root
3818filesystem of an "arm" multiconfig must exist. This dependency is
3819essentially that the
3820:ref:`ref-tasks-image` task in the
3821``core-image-sato`` recipe depends on the completion of the
3822:ref:`ref-tasks-rootfs` task of the
3823``core-image-minimal`` recipe.
3824
3825To enable dependencies in a multiple configuration build, you must
3826declare the dependencies in the recipe using the following statement
3827form::
3828
3829 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3830
3831To better show how to use this statement, consider the example scenario
3832from the first paragraph of this section. The following statement needs
3833to be added to the recipe that builds the ``core-image-sato`` image::
3834
3835 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3836
3837In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
3838task on which the :ref:`ref-tasks-image` task in the recipe depends is the
3839:ref:`ref-tasks-rootfs` task from the ``core-image-minimal`` recipe associated
3840with the "arm" multiconfig.
3841
3842Once you set up this dependency, you can build the "x86" multiconfig
3843using a BitBake command as follows::
3844
3845 $ bitbake mc:x86:core-image-sato
3846
3847This command executes all the tasks needed to create the
3848``core-image-sato`` image for the "x86" multiconfig. Because of the
3849dependency, BitBake also executes through the :ref:`ref-tasks-rootfs` task for the
3850"arm" multiconfig build.
3851
3852Having a recipe depend on the root filesystem of another build might not
3853seem that useful. Consider this change to the statement in the
3854``core-image-sato`` recipe::
3855
3856 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3857
3858In this case, BitBake must
3859create the ``core-image-minimal`` image for the "arm" build since the
3860"x86" build depends on it.
3861
3862Because "x86" and "arm" are enabled for multiple configuration builds
3863and have separate configuration files, BitBake places the artifacts for
3864each build in the respective temporary build directories (i.e.
3865:term:`TMPDIR`).
3866
3867Building an Initial RAM Filesystem (Initramfs) Image
3868----------------------------------------------------
3869
3870An initial RAM filesystem (:term:`Initramfs`) image provides a temporary root
3871filesystem used for early system initialization, typically providing tools and
3872loading modules needed to locate and mount the final root filesystem.
3873
3874Follow these steps to create an :term:`Initramfs` image:
3875
38761. *Create the Initramfs Image Recipe:* You can reference the
3877 ``core-image-minimal-initramfs.bb`` recipe found in the
3878 ``meta/recipes-core`` directory of the :term:`Source Directory`
3879 as an example from which to work.
3880
38812. *Decide if You Need to Bundle the Initramfs Image Into the Kernel
3882 Image:* If you want the :term:`Initramfs` image that is built to be bundled
3883 in with the kernel image, set the :term:`INITRAMFS_IMAGE_BUNDLE`
3884 variable to ``"1"`` in your ``local.conf`` configuration file and set the
3885 :term:`INITRAMFS_IMAGE` variable in the recipe that builds the kernel image.
3886
3887 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the :term:`Initramfs`
3888 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
3889 :term:`Initramfs` image is then passed to the kernel's ``Makefile`` using the
3890 :term:`CONFIG_INITRAMFS_SOURCE` variable, allowing the :term:`Initramfs`
3891 image to be built into the kernel normally.
3892
38933. *Optionally Add Items to the Initramfs Image Through the Initramfs
3894 Image Recipe:* If you add items to the :term:`Initramfs` image by way of its
3895 recipe, you should use :term:`PACKAGE_INSTALL` rather than
3896 :term:`IMAGE_INSTALL`. :term:`PACKAGE_INSTALL` gives more direct control of
3897 what is added to the image as compared to the defaults you might not
3898 necessarily want that are set by the :ref:`image <ref-classes-image>`
3899 or :ref:`core-image <ref-classes-core-image>` classes.
3900
39014. *Build the Kernel Image and the Initramfs Image:* Build your kernel
3902 image using BitBake. Because the :term:`Initramfs` image recipe is a
3903 dependency of the kernel image, the :term:`Initramfs` image is built as well
3904 and bundled with the kernel image if you used the
3905 :term:`INITRAMFS_IMAGE_BUNDLE` variable described earlier.
3906
3907Bundling an Initramfs Image From a Separate Multiconfig
3908~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3909
3910There may be a case where we want to build an :term:`Initramfs` image which does not
3911inherit the same distro policy as our main image, for example, we may want
3912our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our :term:`Initramfs`
3913image to keep a smaller footprint. However, by performing the steps mentioned
3914above the :term:`Initramfs` image will inherit ``TCLIBC="glibc"`` without allowing us
3915to override it.
3916
3917To achieve this, you need to perform some additional steps:
3918
39191. *Create a multiconfig for your Initramfs image:* You can perform the steps
3920 on ":ref:`dev-manual/common-tasks:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
3921 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
3922 contains the variables::
3923
3924 TMPDIR="${TOPDIR}/tmp-initramfscfg"
3925 TCLIBC="musl"
3926
39272. *Set additional Initramfs variables on your main configuration:*
3928 Additionally, on your main configuration (``local.conf``) you need to set the
3929 variables::
3930
3931 INITRAMFS_MULTICONFIG = "initramfscfg"
3932 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
3933
3934 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
3935 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
3936 to be built coming from the ``initramfscfg`` multiconfig, and to let the
3937 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
3938
3939 Building a system with such configuration will build the kernel using the
3940 main configuration but the :ref:`ref-tasks-bundle_initramfs` task will grab the
3941 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
3942 instead, resulting in a musl based :term:`Initramfs` image bundled in the kernel
3943 but a glibc based main image.
3944
3945 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
3946 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
3947
3948
3949Building a Tiny System
3950----------------------
3951
3952Very small distributions have some significant advantages such as
3953requiring less on-die or in-package memory (cheaper), better performance
3954through efficient cache usage, lower power requirements due to less
3955memory, faster boot times, and reduced development overhead. Some
3956real-world examples where a very small distribution gives you distinct
3957advantages are digital cameras, medical devices, and small headless
3958systems.
3959
3960This section presents information that shows you how you can trim your
3961distribution to even smaller sizes than the ``poky-tiny`` distribution,
3962which is around 5 Mbytes, that can be built out-of-the-box using the
3963Yocto Project.
3964
3965Tiny System Overview
3966~~~~~~~~~~~~~~~~~~~~
3967
3968The following list presents the overall steps you need to consider and
3969perform to create distributions with smaller root filesystems, achieve
3970faster boot times, maintain your critical functionality, and avoid
3971initial RAM disks:
3972
3973- :ref:`Determine your goals and guiding principles
3974 <dev-manual/common-tasks:goals and guiding principles>`
3975
3976- :ref:`dev-manual/common-tasks:understand what contributes to your image size`
3977
3978- :ref:`Reduce the size of the root filesystem
3979 <dev-manual/common-tasks:trim the root filesystem>`
3980
3981- :ref:`Reduce the size of the kernel <dev-manual/common-tasks:trim the kernel>`
3982
3983- :ref:`dev-manual/common-tasks:remove package management requirements`
3984
3985- :ref:`dev-manual/common-tasks:look for other ways to minimize size`
3986
3987- :ref:`dev-manual/common-tasks:iterate on the process`
3988
3989Goals and Guiding Principles
3990~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3991
3992Before you can reach your destination, you need to know where you are
3993going. Here is an example list that you can use as a guide when creating
3994very small distributions:
3995
3996- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
3997 less and a root filesystem that is 3 Mbytes or less).
3998
3999- Find the areas that are currently taking 90% of the space and
4000 concentrate on reducing those areas.
4001
4002- Do not create any difficult "hacks" to achieve your goals.
4003
4004- Leverage the device-specific options.
4005
4006- Work in a separate layer so that you keep changes isolated. For
4007 information on how to create layers, see the
4008 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
4009
4010Understand What Contributes to Your Image Size
4011~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4012
4013It is easiest to have something to start with when creating your own
4014distribution. You can use the Yocto Project out-of-the-box to create the
4015``poky-tiny`` distribution. Ultimately, you will want to make changes in
4016your own distribution that are likely modeled after ``poky-tiny``.
4017
4018.. note::
4019
4020 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
4021 ``local.conf`` file to "poky-tiny" as described in the
4022 ":ref:`dev-manual/common-tasks:creating your own distribution`"
4023 section.
4024
4025Understanding some memory concepts will help you reduce the system size.
4026Memory consists of static, dynamic, and temporary memory. Static memory
4027is the TEXT (code), DATA (initialized data in the code), and BSS
4028(uninitialized data) sections. Dynamic memory represents memory that is
4029allocated at runtime: stacks, hash tables, and so forth. Temporary
4030memory is recovered after the boot process. This memory consists of
4031memory used for decompressing the kernel and for the ``__init__``
4032functions.
4033
4034To help you see where you currently are with kernel and root filesystem
4035sizes, you can use two tools found in the :term:`Source Directory`
4036in the
4037``scripts/tiny/`` directory:
4038
4039- ``ksize.py``: Reports component sizes for the kernel build objects.
4040
4041- ``dirsize.py``: Reports component sizes for the root filesystem.
4042
4043This next tool and command help you organize configuration fragments and
4044view file dependencies in a human-readable form:
4045
4046- ``merge_config.sh``: Helps you manage configuration files and
4047 fragments within the kernel. With this tool, you can merge individual
4048 configuration fragments together. The tool allows you to make
4049 overrides and warns you of any missing configuration options. The
4050 tool is ideal for allowing you to iterate on configurations, create
4051 minimal configurations, and create configuration files for different
4052 machines without having to duplicate your process.
4053
4054 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4055 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4056 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4057 directory.
4058
4059 For more information on configuration fragments, see the
4060 ":ref:`kernel-dev/common:creating configuration fragments`"
4061 section in the Yocto Project Linux Kernel Development Manual.
4062
4063- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4064 with these options brings up a Dependency Explorer from which you can
4065 view file dependencies. Understanding these dependencies allows you
4066 to make informed decisions when cutting out various pieces of the
4067 kernel and root filesystem.
4068
4069Trim the Root Filesystem
4070~~~~~~~~~~~~~~~~~~~~~~~~
4071
4072The root filesystem is made up of packages for booting, libraries, and
4073applications. To change things, you can configure how the packaging
4074happens, which changes the way you build them. You can also modify the
4075filesystem itself or select a different filesystem.
4076
4077First, find out what is hogging your root filesystem by running the
4078``dirsize.py`` script from your root directory::
4079
4080 $ cd root-directory-of-image
4081 $ dirsize.py 100000 > dirsize-100k.log
4082 $ cat dirsize-100k.log
4083
4084You can apply a filter to the script to ignore files
4085under a certain size. The previous example filters out any files below
4086100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4087will be smaller by a relatively constant factor in a compressed root
4088filesystem. When you examine your log file, you can focus on areas of
4089the root filesystem that take up large amounts of memory.
4090
4091You need to be sure that what you eliminate does not cripple the
4092functionality you need. One way to see how packages relate to each other
4093is by using the Dependency Explorer UI with the BitBake command::
4094
4095 $ cd image-directory
4096 $ bitbake -u taskexp -g image
4097
4098Use the interface to
4099select potential packages you wish to eliminate and see their dependency
4100relationships.
4101
4102When deciding how to reduce the size, get rid of packages that result in
4103minimal impact on the feature set. For example, you might not need a VGA
4104display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4105instead of ``udev``.
4106
4107Use your ``local.conf`` file to make changes. For example, to eliminate
4108``udev`` and ``glib``, set the following in the local configuration
4109file::
4110
4111 VIRTUAL-RUNTIME_dev_manager = ""
4112
4113Finally, you should consider exactly the type of root filesystem you
4114need to meet your needs while also reducing its size. For example,
4115consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4116:term:`Initramfs` using ``initramfs``. Be aware that ``ext3`` requires a 1
4117Mbyte journal. If you are okay with running read-only, you do not need
4118this journal.
4119
4120.. note::
4121
4122 After each round of elimination, you need to rebuild your system and
4123 then use the tools to see the effects of your reductions.
4124
4125Trim the Kernel
4126~~~~~~~~~~~~~~~
4127
4128The kernel is built by including policies for hardware-independent
4129aspects. What subsystems do you enable? For what architecture are you
4130building? Which drivers do you build by default?
4131
4132.. note::
4133
4134 You can modify the kernel source if you want to help with boot time.
4135
4136Run the ``ksize.py`` script from the top-level Linux build directory to
4137get an idea of what is making up the kernel::
4138
4139 $ cd top-level-linux-build-directory
4140 $ ksize.py > ksize.log
4141 $ cat ksize.log
4142
4143When you examine the log, you will see how much space is taken up with
4144the built-in ``.o`` files for drivers, networking, core kernel files,
4145filesystem, sound, and so forth. The sizes reported by the tool are
4146uncompressed, and thus will be smaller by a relatively constant factor
4147in a compressed kernel image. Look to reduce the areas that are large
4148and taking up around the "90% rule."
4149
4150To examine, or drill down, into any particular area, use the ``-d``
4151option with the script::
4152
4153 $ ksize.py -d > ksize.log
4154
4155Using this option
4156breaks out the individual file information for each area of the kernel
4157(e.g. drivers, networking, and so forth).
4158
4159Use your log file to see what you can eliminate from the kernel based on
4160features you can let go. For example, if you are not going to need
4161sound, you do not need any drivers that support sound.
4162
4163After figuring out what to eliminate, you need to reconfigure the kernel
4164to reflect those changes during the next build. You could run
4165``menuconfig`` and make all your changes at once. However, that makes it
4166difficult to see the effects of your individual eliminations and also
4167makes it difficult to replicate the changes for perhaps another target
4168device. A better method is to start with no configurations using
4169``allnoconfig``, create configuration fragments for individual changes,
4170and then manage the fragments into a single configuration file using
4171``merge_config.sh``. The tool makes it easy for you to iterate using the
4172configuration change and build cycle.
4173
4174Each time you make configuration changes, you need to rebuild the kernel
4175and check to see what impact your changes had on the overall size.
4176
4177Remove Package Management Requirements
4178~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4179
4180Packaging requirements add size to the image. One way to reduce the size
4181of the image is to remove all the packaging requirements from the image.
4182This reduction includes both removing the package manager and its unique
4183dependencies as well as removing the package management data itself.
4184
4185To eliminate all the packaging requirements for an image, be sure that
4186"package-management" is not part of your
4187:term:`IMAGE_FEATURES`
4188statement for the image. When you remove this feature, you are removing
4189the package manager as well as its dependencies from the root
4190filesystem.
4191
4192Look for Other Ways to Minimize Size
4193~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4194
4195Depending on your particular circumstances, other areas that you can
4196trim likely exist. The key to finding these areas is through tools and
4197methods described here combined with experimentation and iteration. Here
4198are a couple of areas to experiment with:
4199
4200- ``glibc``: In general, follow this process:
4201
4202 1. Remove ``glibc`` features from
4203 :term:`DISTRO_FEATURES`
4204 that you think you do not need.
4205
4206 2. Build your distribution.
4207
4208 3. If the build fails due to missing symbols in a package, determine
4209 if you can reconfigure the package to not need those features. For
4210 example, change the configuration to not support wide character
4211 support as is done for ``ncurses``. Or, if support for those
4212 characters is needed, determine what ``glibc`` features provide
4213 the support and restore the configuration.
4214
4215 4. Rebuild and repeat the process.
4216
4217- ``busybox``: For BusyBox, use a process similar as described for
4218 ``glibc``. A difference is you will need to boot the resulting system
4219 to see if you are able to do everything you expect from the running
4220 system. You need to be sure to integrate configuration fragments into
4221 Busybox because BusyBox handles its own core features and then allows
4222 you to add configuration fragments on top.
4223
4224Iterate on the Process
4225~~~~~~~~~~~~~~~~~~~~~~
4226
4227If you have not reached your goals on system size, you need to iterate
4228on the process. The process is the same. Use the tools and see just what
4229is taking up 90% of the root filesystem and the kernel. Decide what you
4230can eliminate without limiting your device beyond what you need.
4231
4232Depending on your system, a good place to look might be Busybox, which
4233provides a stripped down version of Unix tools in a single, executable
4234file. You might be able to drop virtual terminal services or perhaps
4235ipv6.
4236
4237Building Images for More than One Machine
4238-----------------------------------------
4239
4240A common scenario developers face is creating images for several
4241different machines that use the same software environment. In this
4242situation, it is tempting to set the tunings and optimization flags for
4243each build specifically for the targeted hardware (i.e. "maxing out" the
4244tunings). Doing so can considerably add to build times and package feed
4245maintenance collectively for the machines. For example, selecting tunes
4246that are extremely specific to a CPU core used in a system might enable
4247some micro optimizations in GCC for that particular system but would
4248otherwise not gain you much of a performance difference across the other
4249systems as compared to using a more general tuning across all the builds
4250(e.g. setting :term:`DEFAULTTUNE`
4251specifically for each machine's build). Rather than "max out" each
4252build's tunings, you can take steps that cause the OpenEmbedded build
4253system to reuse software across the various machines where it makes
4254sense.
4255
4256If build speed and package feed maintenance are considerations, you
4257should consider the points in this section that can help you optimize
4258your tunings to best consider build times and package feed maintenance.
4259
4260- *Share the :term:`Build Directory`:* If at all possible, share the
4261 :term:`TMPDIR` across builds. The Yocto Project supports switching between
4262 different :term:`MACHINE` values in the same :term:`TMPDIR`. This practice
4263 is well supported and regularly used by developers when building for
4264 multiple machines. When you use the same :term:`TMPDIR` for multiple
4265 machine builds, the OpenEmbedded build system can reuse the existing native
4266 and often cross-recipes for multiple machines. Thus, build time decreases.
4267
4268 .. note::
4269
4270 If :term:`DISTRO` settings change or fundamental configuration settings
4271 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
4272 Sharing :term:`TMPDIR` under these circumstances might work but since it is
4273 not guaranteed, you should use a clean :term:`TMPDIR`.
4274
4275- *Enable the Appropriate Package Architecture:* By default, the
4276 OpenEmbedded build system enables three levels of package
4277 architectures: "all", "tune" or "package", and "machine". Any given
4278 recipe usually selects one of these package architectures (types) for
4279 its output. Depending for what a given recipe creates packages,
4280 making sure you enable the appropriate package architecture can
4281 directly impact the build time.
4282
4283 A recipe that just generates scripts can enable "all" architecture
4284 because there are no binaries to build. To specifically enable "all"
4285 architecture, be sure your recipe inherits the
4286 :ref:`allarch <ref-classes-allarch>` class.
4287 This class is useful for "all" architectures because it configures
4288 many variables so packages can be used across multiple architectures.
4289
4290 If your recipe needs to generate packages that are machine-specific
4291 or when one of the build or runtime dependencies is already
4292 machine-architecture dependent, which makes your recipe also
4293 machine-architecture dependent, make sure your recipe enables the
4294 "machine" package architecture through the
4295 :term:`MACHINE_ARCH`
4296 variable::
4297
4298 PACKAGE_ARCH = "${MACHINE_ARCH}"
4299
4300 When you do not
4301 specifically enable a package architecture through the
4302 :term:`PACKAGE_ARCH`, The
4303 OpenEmbedded build system defaults to the
4304 :term:`TUNE_PKGARCH` setting::
4305
4306 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4307
4308- *Choose a Generic Tuning File if Possible:* Some tunes are more
4309 generic and can run on multiple targets (e.g. an ``armv5`` set of
4310 packages could run on ``armv6`` and ``armv7`` processors in most
4311 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4312 higher processors. You should realize, however, that advances on
4313 newer processor versions would not be used.
4314
4315 If you select the same tune for several different machines, the
4316 OpenEmbedded build system reuses software previously built, thus
4317 speeding up the overall build time. Realize that even though a new
4318 sysroot for each machine is generated, the software is not recompiled
4319 and only one package feed exists.
4320
4321- *Manage Granular Level Packaging:* Sometimes there are cases where
4322 injecting another level of package architecture beyond the three
4323 higher levels noted earlier can be useful. For example, consider how
4324 NXP (formerly Freescale) allows for the easy reuse of binary packages
4325 in their layer
4326 :yocto_git:`meta-freescale </meta-freescale/>`.
4327 In this example, the
4328 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
4329 class shares GPU packages for i.MX53 boards because all boards share
4330 the AMD GPU. The i.MX6-based boards can do the same because all
4331 boards share the Vivante GPU. This class inspects the BitBake
4332 datastore to identify if the package provides or depends on one of
4333 the sub-architecture values. If so, the class sets the
4334 :term:`PACKAGE_ARCH` value
4335 based on the ``MACHINE_SUBARCH`` value. If the package does not
4336 provide or depend on one of the sub-architecture values but it
4337 matches a value in the machine-specific filter, it sets
4338 :term:`MACHINE_ARCH`. This
4339 behavior reduces the number of packages built and saves build time by
4340 reusing binaries.
4341
4342- *Use Tools to Debug Issues:* Sometimes you can run into situations
4343 where software is being rebuilt when you think it should not be. For
4344 example, the OpenEmbedded build system might not be using shared
4345 state between machines when you think it should be. These types of
4346 situations are usually due to references to machine-specific
4347 variables such as :term:`MACHINE`,
4348 :term:`SERIAL_CONSOLES`,
4349 :term:`XSERVER`,
4350 :term:`MACHINE_FEATURES`,
4351 and so forth in code that is supposed to only be tune-specific or
4352 when the recipe depends
4353 (:term:`DEPENDS`,
4354 :term:`RDEPENDS`,
4355 :term:`RRECOMMENDS`,
4356 :term:`RSUGGESTS`, and so forth)
4357 on some other recipe that already has
4358 :term:`PACKAGE_ARCH` defined
4359 as "${MACHINE_ARCH}".
4360
4361 .. note::
4362
4363 Patches to fix any issues identified are most welcome as these
4364 issues occasionally do occur.
4365
4366 For such cases, you can use some tools to help you sort out the
4367 situation:
4368
4369 - ``state-diff-machines.sh``*:* You can find this tool in the
4370 ``scripts`` directory of the Source Repositories. See the comments
4371 in the script for information on how to use the tool.
4372
4373 - *BitBake's "-S printdiff" Option:* Using this option causes
4374 BitBake to try to establish the closest signature match it can
4375 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4376 over the matches to determine the stamps and delta where these two
4377 stamp trees diverge.
4378
4379Building Software from an External Source
4380-----------------------------------------
4381
4382By default, the OpenEmbedded build system uses the :term:`Build Directory`
4383when building source code. The build process involves fetching the source
4384files, unpacking them, and then patching them if necessary before the build
4385takes place.
4386
4387There are situations where you might want to build software from source
4388files that are external to and thus outside of the OpenEmbedded build
4389system. For example, suppose you have a project that includes a new BSP
4390with a heavily customized kernel. And, you want to minimize exposing the
4391build system to the development team so that they can focus on their
4392project and maintain everyone's workflow as much as possible. In this
4393case, you want a kernel source directory on the development machine
4394where the development occurs. You want the recipe's
4395:term:`SRC_URI` variable to point to
4396the external directory and use it as is, not copy it.
4397
4398To build from software that comes from an external source, all you need to do
4399is inherit the :ref:`externalsrc <ref-classes-externalsrc>` class and then set
4400the :term:`EXTERNALSRC` variable to point to your external source code. Here
4401are the statements to put in your ``local.conf`` file::
4402
4403 INHERIT += "externalsrc"
4404 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
4405
4406This next example shows how to accomplish the same thing by setting
4407:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
4408
4409 EXTERNALSRC = "path"
4410 EXTERNALSRC_BUILD = "path"
4411
4412.. note::
4413
4414 In order for these settings to take effect, you must globally or
4415 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
4416 class.
4417
4418By default, :ref:`ref-classes-externalsrc` builds the source code in a
4419directory separate from the external source directory as specified by
4420:term:`EXTERNALSRC`. If you need
4421to have the source built in the same directory in which it resides, or
4422some other nominated directory, you can set
4423:term:`EXTERNALSRC_BUILD`
4424to point to that directory::
4425
4426 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
4427
4428Replicating a Build Offline
4429---------------------------
4430
4431It can be useful to take a "snapshot" of upstream sources used in a
4432build and then use that "snapshot" later to replicate the build offline.
4433To do so, you need to first prepare and populate your downloads
4434directory your "snapshot" of files. Once your downloads directory is
4435ready, you can use it at any time and from any machine to replicate your
4436build.
4437
4438Follow these steps to populate your Downloads directory:
4439
44401. *Create a Clean Downloads Directory:* Start with an empty downloads
4441 directory (:term:`DL_DIR`). You
4442 start with an empty downloads directory by either removing the files
4443 in the existing directory or by setting :term:`DL_DIR` to point to either
4444 an empty location or one that does not yet exist.
4445
44462. *Generate Tarballs of the Source Git Repositories:* Edit your
4447 ``local.conf`` configuration file as follows::
4448
4449 DL_DIR = "/home/your-download-dir/"
4450 BB_GENERATE_MIRROR_TARBALLS = "1"
4451
4452 During
4453 the fetch process in the next step, BitBake gathers the source files
4454 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
4455 the
4456 :term:`BB_GENERATE_MIRROR_TARBALLS`
4457 variable for more information.
4458
44593. *Populate Your Downloads Directory Without Building:* Use BitBake to
4460 fetch your sources but inhibit the build::
4461
4462 $ bitbake target --runonly=fetch
4463
4464 The downloads directory (i.e. ``${DL_DIR}``) now has
4465 a "snapshot" of the source files in the form of tarballs, which can
4466 be used for the build.
4467
44684. *Optionally Remove Any Git or other SCM Subdirectories From the
4469 Downloads Directory:* If you want, you can clean up your downloads
4470 directory by removing any Git or other Source Control Management
4471 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4472 already contain these subdirectories.
4473
4474Once your downloads directory has everything it needs regarding source
4475files, you can create your "own-mirror" and build your target.
4476Understand that you can use the files to build the target offline from
4477any machine and at any time.
4478
4479Follow these steps to build your target using the files in the downloads
4480directory:
4481
44821. *Using Local Files Only:* Inside your ``local.conf`` file, add the
4483 :term:`SOURCE_MIRROR_URL` variable, inherit the
4484 :ref:`own-mirrors <ref-classes-own-mirrors>` class, and use the
4485 :term:`BB_NO_NETWORK` variable to your ``local.conf``::
4486
4487 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4488 INHERIT += "own-mirrors"
4489 BB_NO_NETWORK = "1"
4490
4491 The :term:`SOURCE_MIRROR_URL` and :ref:`own-mirrors <ref-classes-own-mirrors>`
4492 class set up the system to use the downloads directory as your "own
4493 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
4494 BitBake's fetching process in step 3 stays local, which means files
4495 from your "own-mirror" are used.
4496
44972. *Start With a Clean Build:* You can start with a clean build by
4498 removing the ``${``\ :term:`TMPDIR`\ ``}`` directory or using a new
4499 :term:`Build Directory`.
4500
45013. *Build Your Target:* Use BitBake to build your target::
4502
4503 $ bitbake target
4504
4505 The build completes using the known local "snapshot" of source
4506 files from your mirror. The resulting tarballs for your "snapshot" of
4507 source files are in the downloads directory.
4508
4509 .. note::
4510
4511 The offline build does not work if recipes attempt to find the
4512 latest version of software by setting
4513 :term:`SRCREV` to
4514 ``${``\ :term:`AUTOREV`\ ``}``::
4515
4516 SRCREV = "${AUTOREV}"
4517
4518 When a recipe sets :term:`SRCREV` to
4519 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
4520 attempt to determine the latest version of software from the SCM.
4521 Typically, recipes that use :term:`AUTOREV` are custom or modified
4522 recipes. Recipes that reside in public repositories usually do not
4523 use :term:`AUTOREV`.
4524
4525 If you do have recipes that use :term:`AUTOREV`, you can take steps to
4526 still use the recipes in an offline build. Do the following:
4527
4528 1. Use a configuration generated by enabling :ref:`build
4529 history <dev-manual/common-tasks:maintaining build output quality>`.
4530
4531 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
4532 stored :term:`SRCREV` values from the build's history. For more
4533 information on collecting these values, see the
4534 ":ref:`dev-manual/common-tasks:build history package information`"
4535 section.
4536
4537 3. Once you have the correct source revisions, you can modify
4538 those recipes to set :term:`SRCREV` to specific versions of the
4539 software.
4540
4541Speeding Up a Build
4542===================
4543
4544Build time can be an issue. By default, the build system uses simple
4545controls to try and maximize build efficiency. In general, the default
4546settings for all the following variables result in the most efficient
4547build times when dealing with single socket systems (i.e. a single CPU).
4548If you have multiple CPUs, you might try increasing the default values
4549to gain more speed. See the descriptions in the glossary for each
4550variable for more information:
4551
4552- :term:`BB_NUMBER_THREADS`:
4553 The maximum number of threads BitBake simultaneously executes.
4554
4555- :term:`BB_NUMBER_PARSE_THREADS`:
4556 The number of threads BitBake uses during parsing.
4557
4558- :term:`PARALLEL_MAKE`: Extra
4559 options passed to the ``make`` command during the
4560 :ref:`ref-tasks-compile` task in
4561 order to specify parallel compilation on the local build host.
4562
4563- :term:`PARALLEL_MAKEINST`:
4564 Extra options passed to the ``make`` command during the
4565 :ref:`ref-tasks-install` task in
4566 order to specify parallel installation on the local build host.
4567
4568As mentioned, these variables all scale to the number of processor cores
4569available on the build system. For single socket systems, this
4570auto-scaling ensures that the build system fundamentally takes advantage
4571of potential parallel operations during the build based on the build
4572machine's capabilities.
4573
4574Following are additional factors that can affect build speed:
4575
4576- File system type: The file system type that the build is being
4577 performed on can also influence performance. Using ``ext4`` is
4578 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4579 improved features such as extents.
4580
4581- Disabling the updating of access time using ``noatime``: The
4582 ``noatime`` mount option prevents the build system from updating file
4583 and directory access times.
4584
4585- Setting a longer commit: Using the "commit=" mount option increases
4586 the interval in seconds between disk cache writes. Changing this
4587 interval from the five second default to something longer increases
4588 the risk of data loss but decreases the need to write to the disk,
4589 thus increasing the build performance.
4590
4591- Choosing the packaging backend: Of the available packaging backends,
4592 IPK is the fastest. Additionally, selecting a singular packaging
4593 backend also helps.
4594
4595- Using ``tmpfs`` for :term:`TMPDIR`
4596 as a temporary file system: While this can help speed up the build,
4597 the benefits are limited due to the compiler using ``-pipe``. The
4598 build system goes to some lengths to avoid ``sync()`` calls into the
4599 file system on the principle that if there was a significant failure,
4600 the :term:`Build Directory` contents could easily be rebuilt.
4601
4602- Inheriting the
4603 :ref:`rm_work <ref-classes-rm-work>` class:
4604 Inheriting this class has shown to speed up builds due to
4605 significantly lower amounts of data stored in the data cache as well
4606 as on disk. Inheriting this class also makes cleanup of
4607 :term:`TMPDIR` faster, at the
4608 expense of being easily able to dive into the source code. File
4609 system maintainers have recommended that the fastest way to clean up
4610 large numbers of files is to reformat partitions rather than delete
4611 files due to the linear nature of partitions. This, of course,
4612 assumes you structure the disk partitions and file systems in a way
4613 that this is practical.
4614
4615Aside from the previous list, you should keep some trade offs in mind
4616that can help you speed up the build:
4617
4618- Remove items from
4619 :term:`DISTRO_FEATURES`
4620 that you might not need.
4621
4622- Exclude debug symbols and other debug information: If you do not need
4623 these symbols and other debug information, disabling the ``*-dbg``
4624 package generation can speed up the build. You can disable this
4625 generation by setting the
4626 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4627 variable to "1".
4628
4629- Disable static library generation for recipes derived from
4630 ``autoconf`` or ``libtool``: Following is an example showing how to
4631 disable static libraries and still provide an override to handle
4632 exceptions::
4633
4634 STATICLIBCONF = "--disable-static"
4635 STATICLIBCONF:sqlite3-native = ""
4636 EXTRA_OECONF += "${STATICLIBCONF}"
4637
4638 .. note::
4639
4640 - Some recipes need static libraries in order to work correctly
4641 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4642 as in the previous example, account for these kinds of
4643 exceptions.
4644
4645 - Some packages have packaging code that assumes the presence of
4646 the static libraries. If so, you might need to exclude them as
4647 well.
4648
4649Working With Libraries
4650======================
4651
4652Libraries are an integral part of your system. This section describes
4653some common practices you might find helpful when working with libraries
4654to build your system:
4655
4656- :ref:`How to include static library files
4657 <dev-manual/common-tasks:including static library files>`
4658
4659- :ref:`How to use the Multilib feature to combine multiple versions of
4660 library files into a single image
4661 <dev-manual/common-tasks:combining multiple versions of library files into one image>`
4662
4663- :ref:`How to install multiple versions of the same library in parallel on
4664 the same system
4665 <dev-manual/common-tasks:installing multiple versions of the same library>`
4666
4667Including Static Library Files
4668------------------------------
4669
4670If you are building a library and the library offers static linking, you
4671can control which static library files (``*.a`` files) get included in
4672the built library.
4673
4674The :term:`PACKAGES` and
4675:term:`FILES:* <FILES>` variables in the
4676``meta/conf/bitbake.conf`` configuration file define how files installed
4677by the :ref:`ref-tasks-install` task are packaged. By default, the :term:`PACKAGES`
4678variable includes ``${PN}-staticdev``, which represents all static
4679library files.
4680
4681.. note::
4682
4683 Some previously released versions of the Yocto Project defined the
4684 static library files through ``${PN}-dev``.
4685
4686Following is part of the BitBake configuration file, where you can see
4687how the static library files are defined::
4688
4689 PACKAGE_BEFORE_PN ?= ""
4690 PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
4691 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4692 FILES = ""
4693
4694 FILES:${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
4695 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4696 ${base_bindir}/* ${base_sbindir}/* \
4697 ${base_libdir}/*${SOLIBS} \
4698 ${base_prefix}/lib/udev ${prefix}/lib/udev \
4699 ${base_libdir}/udev ${libdir}/udev \
4700 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4701 ${datadir}/pixmaps ${datadir}/applications \
4702 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4703 ${libdir}/bonobo/servers"
4704
4705 FILES:${PN}-bin = "${bindir}/* ${sbindir}/*"
4706
4707 FILES:${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
4708 ${datadir}/gnome/help"
4709 SECTION:${PN}-doc = "doc"
4710
4711 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
4712 FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
4713 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4714 ${datadir}/aclocal ${base_libdir}/*.o \
4715 ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
4716 ${libdir}/cmake ${datadir}/cmake"
4717 SECTION:${PN}-dev = "devel"
4718 ALLOW_EMPTY:${PN}-dev = "1"
4719 RDEPENDS:${PN}-dev = "${PN} (= ${EXTENDPKGV})"
4720
4721 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4722 SECTION:${PN}-staticdev = "devel"
4723 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
4724
4725Combining Multiple Versions of Library Files into One Image
4726-----------------------------------------------------------
4727
4728The build system offers the ability to build libraries with different
4729target optimizations or architecture formats and combine these together
4730into one system image. You can link different binaries in the image
4731against the different libraries as needed for specific use cases. This
4732feature is called "Multilib".
4733
4734An example would be where you have most of a system compiled in 32-bit
4735mode using 32-bit libraries, but you have something large, like a
4736database engine, that needs to be a 64-bit application and uses 64-bit
4737libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4738libraries.
4739
4740While the Multilib feature is most commonly used for 32 and 64-bit
4741differences, the approach the build system uses facilitates different
4742target optimizations. You could compile some binaries to use one set of
4743libraries and other binaries to use a different set of libraries. The
4744libraries could differ in architecture, compiler options, or other
4745optimizations.
4746
4747There are several examples in the ``meta-skeleton`` layer found in the
4748:term:`Source Directory`:
4749
4750- :oe_git:`conf/multilib-example.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example.conf>`
4751 configuration file.
4752
4753- :oe_git:`conf/multilib-example2.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example2.conf>`
4754 configuration file.
4755
4756- :oe_git:`recipes-multilib/images/core-image-multilib-example.bb </openembedded-core/tree/meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb>`
4757 recipe
4758
4759Preparing to Use Multilib
4760~~~~~~~~~~~~~~~~~~~~~~~~~
4761
4762User-specific requirements drive the Multilib feature. Consequently,
4763there is no one "out-of-the-box" configuration that would
4764meet your needs.
4765
4766In order to enable Multilib, you first need to ensure your recipe is
4767extended to support multiple libraries. Many standard recipes are
4768already extended and support multiple libraries. You can check in the
4769``meta/conf/multilib.conf`` configuration file in the
4770:term:`Source Directory` to see how this is
4771done using the
4772:term:`BBCLASSEXTEND` variable.
4773Eventually, all recipes will be covered and this list will not be
4774needed.
4775
4776For the most part, the :ref:`Multilib <ref-classes-multilib*>`
4777class extension works automatically to
4778extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
4779:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
4780Standard variables such as
4781:term:`DEPENDS`,
4782:term:`RDEPENDS`,
4783:term:`RPROVIDES`,
4784:term:`RRECOMMENDS`,
4785:term:`PACKAGES`, and
4786:term:`PACKAGES_DYNAMIC` are
4787automatically extended by the system. If you are extending any manual
4788code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
4789those names are extended correctly.
4790
4791Using Multilib
4792~~~~~~~~~~~~~~
4793
4794After you have set up the recipes, you need to define the actual
4795combination of multiple libraries you want to build. You accomplish this
4796through your ``local.conf`` configuration file in the
4797:term:`Build Directory`. An example configuration would be as follows::
4798
4799 MACHINE = "qemux86-64"
4800 require conf/multilib.conf
4801 MULTILIBS = "multilib:lib32"
4802 DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
4803 IMAGE_INSTALL:append = " lib32-glib-2.0"
4804
4805This example enables an additional library named
4806``lib32`` alongside the normal target packages. When combining these
4807"lib32" alternatives, the example uses "x86" for tuning. For information
4808on this particular tuning, see
4809``meta/conf/machine/include/ia32/arch-ia32.inc``.
4810
4811The example then includes ``lib32-glib-2.0`` in all the images, which
4812illustrates one method of including a multiple library dependency. You
4813can use a normal image build to include this dependency, for example::
4814
4815 $ bitbake core-image-sato
4816
4817You can also build Multilib packages
4818specifically with a command like this::
4819
4820 $ bitbake lib32-glib-2.0
4821
4822Additional Implementation Details
4823~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4824
4825There are generic implementation details as well as details that are specific to
4826package management systems. Following are implementation details
4827that exist regardless of the package management system:
4828
4829- The typical convention used for the class extension code as used by
4830 Multilib assumes that all package names specified in
4831 :term:`PACKAGES` that contain
4832 ``${PN}`` have ``${PN}`` at the start of the name. When that
4833 convention is not followed and ``${PN}`` appears at the middle or the
4834 end of a name, problems occur.
4835
4836- The :term:`TARGET_VENDOR`
4837 value under Multilib will be extended to "-vendormlmultilib" (e.g.
4838 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
4839 slightly unwieldy contraction is that any "-" characters in the
4840 vendor string presently break Autoconf's ``config.sub``, and other
4841 separators are problematic for different reasons.
4842
4843Here are the implementation details for the RPM Package Management System:
4844
4845- A unique architecture is defined for the Multilib packages, along
4846 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
4847 :term:`Build Directory`. For example, consider ``lib32`` in a
4848 ``qemux86-64`` image. The possible architectures in the system are "all",
4849 "qemux86_64", "lib32:qemux86_64", and "lib32:x86".
4850
4851- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
4852 packaging. The naming for a normal RPM package and a Multilib RPM
4853 package in a ``qemux86-64`` system resolves to something similar to
4854 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
4855 respectively.
4856
4857- When installing a Multilib image, the RPM backend first installs the
4858 base image and then installs the Multilib libraries.
4859
4860- The build system relies on RPM to resolve the identical files in the
4861 two (or more) Multilib packages.
4862
4863Here are the implementation details for the IPK Package Management System:
4864
4865- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
4866 packaging. The naming for a normal RPM package and a Multilib IPK
4867 package in a ``qemux86-64`` system resolves to something like
4868 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``,
4869 respectively.
4870
4871- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
4872 packages with and without the Multilib feature can exist in the same
4873 folder due to the ``${PN}`` differences.
4874
4875- IPK defines a sanity check for Multilib installation using certain
4876 rules for file comparison, overridden, etc.
4877
4878Installing Multiple Versions of the Same Library
4879------------------------------------------------
4880
4881There are be situations where you need to install and use multiple versions
4882of the same library on the same system at the same time. This
4883almost always happens when a library API changes and you have
4884multiple pieces of software that depend on the separate versions of the
4885library. To accommodate these situations, you can install multiple
4886versions of the same library in parallel on the same system.
4887
4888The process is straightforward as long as the libraries use proper
4889versioning. With properly versioned libraries, all you need to do to
4890individually specify the libraries is create separate, appropriately
4891named recipes where the :term:`PN` part of
4892the name includes a portion that differentiates each library version
4893(e.g. the major part of the version number). Thus, instead of having a
4894single recipe that loads one version of a library (e.g. ``clutter``),
4895you provide multiple recipes that result in different versions of the
4896libraries you want. As an example, the following two recipes would allow
4897the two separate versions of the ``clutter`` library to co-exist on the
4898same system:
4899
4900.. code-block:: none
4901
4902 clutter-1.6_1.6.20.bb
4903 clutter-1.8_1.8.4.bb
4904
4905Additionally, if
4906you have other recipes that depend on a given library, you need to use
4907the :term:`DEPENDS` variable to
4908create the dependency. Continuing with the same example, if you want to
4909have a recipe depend on the 1.8 version of the ``clutter`` library, use
4910the following in your recipe::
4911
4912 DEPENDS = "clutter-1.8"
4913
4914Working with Pre-Built Libraries
4915================================
4916
4917Introduction
4918-------------
4919
4920Some library vendors do not release source code for their software but do
4921release pre-built binaries. When shared libraries are built, they should
4922be versioned (see `this article
4923<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__
4924for some background), but sometimes this is not done.
4925
4926To summarize, a versioned library must meet two conditions:
4927
4928#. The filename must have the version appended, for example: ``libfoo.so.1.2.3``.
4929#. The library must have the ELF tag ``SONAME`` set to the major version
4930 of the library, for example: ``libfoo.so.1``. You can check this by
4931 running ``readelf -d filename | grep SONAME``.
4932
4933This section shows how to deal with both versioned and unversioned
4934pre-built libraries.
4935
4936Versioned Libraries
4937-------------------
4938
4939In this example we work with pre-built libraries for the FT4222H USB I/O chip.
4940Libraries are built for several target architecture variants and packaged in
4941an archive as follows::
4942
4943 ├── build-arm-hisiv300
4944 │   └── libft4222.so.1.4.4.44
4945 ├── build-arm-v5-sf
4946 │   └── libft4222.so.1.4.4.44
4947 ├── build-arm-v6-hf
4948 │   └── libft4222.so.1.4.4.44
4949 ├── build-arm-v7-hf
4950 │   └── libft4222.so.1.4.4.44
4951 ├── build-arm-v8
4952 │   └── libft4222.so.1.4.4.44
4953 ├── build-i386
4954 │   └── libft4222.so.1.4.4.44
4955 ├── build-i486
4956 │   └── libft4222.so.1.4.4.44
4957 ├── build-mips-eglibc-hf
4958 │   └── libft4222.so.1.4.4.44
4959 ├── build-pentium
4960 │   └── libft4222.so.1.4.4.44
4961 ├── build-x86_64
4962 │   └── libft4222.so.1.4.4.44
4963 ├── examples
4964 │   ├── get-version.c
4965 │   ├── i2cm.c
4966 │   ├── spim.c
4967 │   └── spis.c
4968 ├── ftd2xx.h
4969 ├── install4222.sh
4970 ├── libft4222.h
4971 ├── ReadMe.txt
4972 └── WinTypes.h
4973
4974To write a recipe to use such a library in your system:
4975
4976- The vendor will probably have a proprietary licence, so set
4977 :term:`LICENSE_FLAGS` in your recipe.
4978- The vendor provides a tarball containing libraries so set :term:`SRC_URI`
4979 appropriately.
4980- Set :term:`COMPATIBLE_HOST` so that the recipe cannot be used with an
4981 unsupported architecture. In the following example, we only support the 32
4982 and 64 bit variants of the ``x86`` architecture.
4983- As the vendor provides versioned libraries, we can use ``oe_soinstall``
4984 from :ref:`ref-classes-utils` to install the shared library and create
4985 symbolic links. If the vendor does not do this, we need to follow the
4986 non-versioned library guidelines in the next section.
4987- As the vendor likely used :term:`LDFLAGS` different from those in your Yocto
4988 Project build, disable the corresponding checks by adding ``ldflags``
4989 to :term:`INSANE_SKIP`.
4990- The vendor will typically ship release builds without debugging symbols.
4991 Avoid errors by preventing the packaging task from stripping out the symbols
4992 and adding them to a separate debug package. This is done by setting the
4993 ``INHIBIT_`` flags shown below.
4994
4995The complete recipe would look like this::
4996
4997 SUMMARY = "FTDI FT4222H Library"
4998 SECTION = "libs"
4999 LICENSE_FLAGS = "ftdi"
5000 LICENSE = "CLOSED"
5001
5002 COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
5003
5004 # Sources available in a .tgz file in .zip archive
5005 # at https://ftdichip.com/wp-content/uploads/2021/01/libft4222-linux-1.4.4.44.zip
5006 # Found on https://ftdichip.com/software-examples/ft4222h-software-examples/
5007 # Since dealing with this particular type of archive is out of topic here,
5008 # we use a local link.
5009 SRC_URI = "file://libft4222-linux-${PV}.tgz"
5010
5011 S = "${WORKDIR}"
5012
5013 ARCH_DIR:x86-64 = "build-x86_64"
5014 ARCH_DIR:i586 = "build-i386"
5015 ARCH_DIR:i686 = "build-i386"
5016
5017 INSANE_SKIP:${PN} = "ldflags"
5018 INHIBIT_PACKAGE_STRIP = "1"
5019 INHIBIT_SYSROOT_STRIP = "1"
5020 INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
5021
5022 do_install () {
5023 install -m 0755 -d ${D}${libdir}
5024 oe_soinstall ${S}/${ARCH_DIR}/libft4222.so.${PV} ${D}${libdir}
5025 install -d ${D}${includedir}
5026 install -m 0755 ${S}/*.h ${D}${includedir}
5027 }
5028
5029If the precompiled binaries are not statically linked and have dependencies on
5030other libraries, then by adding those libraries to :term:`DEPENDS`, the linking
5031can be examined and the appropriate :term:`RDEPENDS` automatically added.
5032
5033Non-Versioned Libraries
5034-----------------------
5035
5036Some Background
5037~~~~~~~~~~~~~~~
5038
5039Libraries in Linux systems are generally versioned so that it is possible
5040to have multiple versions of the same library installed, which eases upgrades
5041and support for older software. For example, suppose that in a versioned
5042library, an actual library is called ``libfoo.so.1.2``, a symbolic link named
5043``libfoo.so.1`` points to ``libfoo.so.1.2``, and a symbolic link named
5044``libfoo.so`` points to ``libfoo.so.1.2``. Given these conditions, when you
5045link a binary against a library, you typically provide the unversioned file
5046name (i.e. ``-lfoo`` to the linker). However, the linker follows the symbolic
5047link and actually links against the versioned filename. The unversioned symbolic
5048link is only used at development time. Consequently, the library is packaged
5049along with the headers in the development package ``${PN}-dev`` along with the
5050actual library and versioned symbolic links in ``${PN}``. Because versioned
5051libraries are far more common than unversioned libraries, the default packaging
5052rules assume versioned libraries.
5053
5054Yocto Library Packaging Overview
5055~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5056
5057It follows that packaging an unversioned library requires a bit of work in the
5058recipe. By default, ``libfoo.so`` gets packaged into ``${PN}-dev``, which
5059triggers a QA warning that a non-symlink library is in a ``-dev`` package,
5060and binaries in the same recipe link to the library in ``${PN}-dev``,
5061which triggers more QA warnings. To solve this problem, you need to package the
5062unversioned library into ``${PN}`` where it belongs. The following are the abridged
5063default :term:`FILES` variables in ``bitbake.conf``::
5064
5065 SOLIBS = ".so.*"
5066 SOLIBSDEV = ".so"
5067 FILES_${PN} = "... ${libdir}/lib*${SOLIBS} ..."
5068 FILES_SOLIBSDEV ?= "... ${libdir}/lib*${SOLIBSDEV} ..."
5069 FILES_${PN}-dev = "... ${FILES_SOLIBSDEV} ..."
5070
5071:term:`SOLIBS` defines a pattern that matches real shared object libraries.
5072:term:`SOLIBSDEV` matches the development form (unversioned symlink). These two
5073variables are then used in ``FILES:${PN}`` and ``FILES:${PN}-dev``, which puts
5074the real libraries into ``${PN}`` and the unversioned symbolic link into ``${PN}-dev``.
5075To package unversioned libraries, you need to modify the variables in the recipe
5076as follows::
5077
5078 SOLIBS = ".so"
5079 FILES_SOLIBSDEV = ""
5080
5081The modifications cause the ``.so`` file to be the real library
5082and unset :term:`FILES_SOLIBSDEV` so that no libraries get packaged into
5083``${PN}-dev``. The changes are required because unless :term:`PACKAGES` is changed,
5084``${PN}-dev`` collects files before `${PN}`. ``${PN}-dev`` must not collect any of
5085the files you want in ``${PN}``.
5086
5087Finally, loadable modules, essentially unversioned libraries that are linked
5088at runtime using ``dlopen()`` instead of at build time, should generally be
5089installed in a private directory. However, if they are installed in ``${libdir}``,
5090then the modules can be treated as unversioned libraries.
5091
5092Example
5093~~~~~~~
5094
5095The example below installs an unversioned x86-64 pre-built library named
5096``libfoo.so``. The :term:`COMPATIBLE_HOST` variable limits recipes to the
5097x86-64 architecture while the :term:`INSANE_SKIP`, :term:`INHIBIT_PACKAGE_STRIP`
5098and :term:`INHIBIT_SYSROOT_STRIP` variables are all set as in the above
5099versioned library example. The "magic" is setting the :term:`SOLIBS` and
5100:term:`FILES_SOLIBSDEV` variables as explained above::
5101
5102 SUMMARY = "libfoo sample recipe"
5103 SECTION = "libs"
5104 LICENSE = "CLOSED"
5105
5106 SRC_URI = "file://libfoo.so"
5107
5108 COMPATIBLE_HOST = "x86_64.*-linux"
5109
5110 INSANE_SKIP:${PN} = "ldflags"
5111 INHIBIT_PACKAGE_STRIP = "1"
5112 INHIBIT_SYSROOT_STRIP = "1"
5113 SOLIBS = ".so"
5114 FILES_SOLIBSDEV = ""
5115
5116 do_install () {
5117 install -d ${D}${libdir}
5118 install -m 0755 ${WORKDIR}/libfoo.so ${D}${libdir}
5119 }
5120
5121Using x32 psABI
5122===============
5123
5124x32 processor-specific Application Binary Interface (`x32
5125psABI <https://software.intel.com/en-us/node/628948>`__) is a native
512632-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
5127ABI defines the calling conventions between functions in a processing
5128environment. The interface determines what registers are used and what
5129the sizes are for various C data types.
5130
5131Some processing environments prefer using 32-bit applications even when
5132running on Intel 64-bit platforms. Consider the i386 psABI, which is a
5133very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
5134provide efficient use and access of the Intel 64-bit processor
5135resources, leaving the system underutilized. Now consider the x86_64
5136psABI. This ABI is newer and uses 64-bits for data sizes and program
5137pointers. The extra bits increase the footprint size of the programs,
5138libraries, and also increases the memory and file system size
5139requirements. Executing under the x32 psABI enables user programs to
5140utilize CPU and system resources more efficiently while keeping the
5141memory footprint of the applications low. Extra bits are used for
5142registers but not for addressing mechanisms.
5143
5144The Yocto Project supports the final specifications of x32 psABI as
5145follows:
5146
5147- You can create packages and images in x32 psABI format on x86_64
5148 architecture targets.
5149
5150- You can successfully build recipes with the x32 toolchain.
5151
5152- You can create and boot ``core-image-minimal`` and
5153 ``core-image-sato`` images.
5154
5155- There is RPM Package Manager (RPM) support for x32 binaries.
5156
5157- There is support for large images.
5158
5159To use the x32 psABI, you need to edit your ``conf/local.conf``
5160configuration file as follows::
5161
5162 MACHINE = "qemux86-64"
5163 DEFAULTTUNE = "x86-64-x32"
5164 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \
5165 or 'INVALID')) or 'lib'}"
5166
5167Once you have set
5168up your configuration file, use BitBake to build an image that supports
5169the x32 psABI. Here is an example::
5170
5171 $ bitbake core-image-sato
5172
5173Enabling GObject Introspection Support
5174======================================
5175
5176`GObject introspection <https://gi.readthedocs.io/en/latest/>`__
5177is the standard mechanism for accessing GObject-based software from
5178runtime environments. GObject is a feature of the GLib library that
5179provides an object framework for the GNOME desktop and related software.
5180GObject Introspection adds information to GObject that allows objects
5181created within it to be represented across different programming
5182languages. If you want to construct GStreamer pipelines using Python, or
5183control UPnP infrastructure using Javascript and GUPnP, GObject
5184introspection is the only way to do it.
5185
5186This section describes the Yocto Project support for generating and
5187packaging GObject introspection data. GObject introspection data is a
5188description of the API provided by libraries built on top of the GLib
5189framework, and, in particular, that framework's GObject mechanism.
5190GObject Introspection Repository (GIR) files go to ``-dev`` packages,
5191``typelib`` files go to main packages as they are packaged together with
5192libraries that are introspected.
5193
5194The data is generated when building such a library, by linking the
5195library with a small executable binary that asks the library to describe
5196itself, and then executing the binary and processing its output.
5197
5198Generating this data in a cross-compilation environment is difficult
5199because the library is produced for the target architecture, but its
5200code needs to be executed on the build host. This problem is solved with
5201the OpenEmbedded build system by running the code through QEMU, which
5202allows precisely that. Unfortunately, QEMU does not always work
5203perfectly as mentioned in the ":ref:`dev-manual/common-tasks:known issues`"
5204section.
5205
5206Enabling the Generation of Introspection Data
5207---------------------------------------------
5208
5209Enabling the generation of introspection data (GIR files) in your
5210library package involves the following:
5211
52121. Inherit the
5213 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
5214 class.
5215
52162. Make sure introspection is not disabled anywhere in the recipe or
5217 from anything the recipe includes. Also, make sure that
5218 "gobject-introspection-data" is not in
5219 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5220 and that "qemu-usermode" is not in
5221 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5222 In either of these conditions, nothing will happen.
5223
52243. Try to build the recipe. If you encounter build errors that look like
5225 something is unable to find ``.so`` libraries, check where these
5226 libraries are located in the source tree and add the following to the
5227 recipe::
5228
5229 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5230
5231 .. note::
5232
5233 See recipes in the ``oe-core`` repository that use that
5234 :term:`GIR_EXTRA_LIBS_PATH` variable as an example.
5235
52364. Look for any other errors, which probably mean that introspection
5237 support in a package is not entirely standard, and thus breaks down
5238 in a cross-compilation environment. For such cases, custom-made fixes
5239 are needed. A good place to ask and receive help in these cases is
5240 the :ref:`Yocto Project mailing
5241 lists <resources-mailinglist>`.
5242
5243.. note::
5244
5245 Using a library that no longer builds against the latest Yocto
5246 Project release and prints introspection related errors is a good
5247 candidate for the previous procedure.
5248
5249Disabling the Generation of Introspection Data
5250----------------------------------------------
5251
5252You might find that you do not want to generate introspection data. Or,
5253perhaps QEMU does not work on your build host and target architecture
5254combination. If so, you can use either of the following methods to
5255disable GIR file generations:
5256
5257- Add the following to your distro configuration::
5258
5259 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5260
5261 Adding this statement disables generating introspection data using
5262 QEMU but will still enable building introspection tools and libraries
5263 (i.e. building them does not require the use of QEMU).
5264
5265- Add the following to your machine configuration::
5266
5267 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5268
5269 Adding this statement disables the use of QEMU when building packages for your
5270 machine. Currently, this feature is used only by introspection
5271 recipes and has the same effect as the previously described option.
5272
5273 .. note::
5274
5275 Future releases of the Yocto Project might have other features
5276 affected by this option.
5277
5278If you disable introspection data, you can still obtain it through other
5279means such as copying the data from a suitable sysroot, or by generating
5280it on the target hardware. The OpenEmbedded build system does not
5281currently provide specific support for these techniques.
5282
5283Testing that Introspection Works in an Image
5284--------------------------------------------
5285
5286Use the following procedure to test if generating introspection data is
5287working in an image:
5288
52891. Make sure that "gobject-introspection-data" is not in
5290 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5291 and that "qemu-usermode" is not in
5292 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5293
52942. Build ``core-image-sato``.
5295
52963. Launch a Terminal and then start Python in the terminal.
5297
52984. Enter the following in the terminal::
5299
5300 >>> from gi.repository import GLib
5301 >>> GLib.get_host_name()
5302
53035. For something a little more advanced, enter the following see:
5304 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
5305
5306Known Issues
5307------------
5308
5309Here are know issues in GObject Introspection Support:
5310
5311- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5312 introspection data on that architecture.
5313
5314- x32 is not supported by QEMU. Consequently, introspection data is
5315 disabled.
5316
5317- musl causes transient GLib binaries to crash on assertion failures.
5318 Consequently, generating introspection data is disabled.
5319
5320- Because QEMU is not able to run the binaries correctly, introspection
5321 is disabled for some specific packages under specific architectures
5322 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5323
5324- QEMU usermode might not work properly when running 64-bit binaries
5325 under 32-bit host machines. In particular, "qemumips64" is known to
5326 not work under i686.
5327
5328Optionally Using an External Toolchain
5329======================================
5330
5331You might want to use an external toolchain as part of your development.
5332If this is the case, the fundamental steps you need to accomplish are as
5333follows:
5334
5335- Understand where the installed toolchain resides. For cases where you
5336 need to build the external toolchain, you would need to take separate
5337 steps to build and install the toolchain.
5338
5339- Make sure you add the layer that contains the toolchain to your
5340 ``bblayers.conf`` file through the
5341 :term:`BBLAYERS` variable.
5342
5343- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5344 to the location in which you installed the toolchain.
5345
5346A good example of an external toolchain used with the Yocto Project is
5347Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5348to use that particular layer in the ``README`` file at
5349https://github.com/MentorEmbedded/meta-sourcery/. You can find
5350further information by reading about the
5351:term:`TCMODE` variable in the Yocto
5352Project Reference Manual's variable glossary.
5353
5354Creating Partitioned Images Using Wic
5355=====================================
5356
5357Creating an image for a particular hardware target using the
5358OpenEmbedded build system does not necessarily mean you can boot that
5359image as is on your device. Physical devices accept and boot images in
5360various ways depending on the specifics of the device. Usually,
5361information about the hardware can tell you what image format the device
5362requires. Should your device require multiple partitions on an SD card,
5363flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5364create the properly partitioned image.
5365
5366The ``wic`` command generates partitioned images from existing
5367OpenEmbedded build artifacts. Image generation is driven by partitioning
5368commands contained in an OpenEmbedded kickstart file (``.wks``)
5369specified either directly on the command line or as one of a selection
5370of canned kickstart files as shown with the ``wic list images`` command
5371in the
5372":ref:`dev-manual/common-tasks:generate an image using an existing kickstart file`"
5373section. When you apply the command to a given set of build artifacts, the
5374result is an image or set of images that can be directly written onto media and
5375used on a particular system.
5376
5377.. note::
5378
5379 For a kickstart file reference, see the
5380 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
5381 Chapter in the Yocto Project Reference Manual.
5382
5383The ``wic`` command and the infrastructure it is based on is by
5384definition incomplete. The purpose of the command is to allow the
5385generation of customized images, and as such, was designed to be
5386completely extensible through a plugin interface. See the
5387":ref:`dev-manual/common-tasks:using the wic plugin interface`" section
5388for information on these plugins.
5389
5390This section provides some background information on Wic, describes what
5391you need to have in place to run the tool, provides instruction on how
5392to use the Wic utility, provides information on using the Wic plugins
5393interface, and provides several examples that show how to use Wic.
5394
5395Background
5396----------
5397
5398This section provides some background on the Wic utility. While none of
5399this information is required to use Wic, you might find it interesting.
5400
5401- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5402 "oe" diphthong in "oeic" was promoted to the letter "w", because
5403 "oeic" is both difficult to remember and to pronounce.
5404
5405- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5406 The Wic implementation has been heavily modified to make direct use
5407 of OpenEmbedded build artifacts instead of package installation and
5408 configuration, which are already incorporated within the OpenEmbedded
5409 artifacts.
5410
5411- Wic is a completely independent standalone utility that initially
5412 provides easier-to-use and more flexible replacements for an existing
5413 functionality in OE-Core's
5414 :ref:`image-live <ref-classes-image-live>`
5415 class. The difference between Wic and those examples is that with Wic
5416 the functionality of those scripts is implemented by a
5417 general-purpose partitioning language, which is based on Redhat
5418 kickstart syntax.
5419
5420Requirements
5421------------
5422
5423In order to use the Wic utility with the OpenEmbedded Build system, your
5424system needs to meet the following requirements:
5425
5426- The Linux distribution on your development host must support the
5427 Yocto Project. See the ":ref:`detailed-supported-distros`"
5428 section in the Yocto Project Reference Manual for the list of
5429 distributions that support the Yocto Project.
5430
5431- The standard system utilities, such as ``cp``, must be installed on
5432 your development host system.
5433
5434- You must have sourced the build environment setup script (i.e.
5435 :ref:`structure-core-script`) found in the :term:`Build Directory`.
5436
5437- You need to have the build artifacts already available, which
5438 typically means that you must have already created an image using the
5439 OpenEmbedded build system (e.g. ``core-image-minimal``). While it
5440 might seem redundant to generate an image in order to create an image
5441 using Wic, the current version of Wic requires the artifacts in the
5442 form generated by the OpenEmbedded build system.
5443
5444- You must build several native tools, which are built to run on the
5445 build system::
5446
5447 $ bitbake parted-native dosfstools-native mtools-native
5448
5449- Include "wic" as part of the
5450 :term:`IMAGE_FSTYPES`
5451 variable.
5452
5453- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
5454 as part of the :term:`WKS_FILE` variable. If multiple candidate files can
5455 be provided by different layers, specify all the possible names through the
5456 :term:`WKS_FILES` variable instead.
5457
5458Getting Help
5459------------
5460
5461You can get general help for the ``wic`` command by entering the ``wic``
5462command by itself or by entering the command with a help argument as
5463follows::
5464
5465 $ wic -h
5466 $ wic --help
5467 $ wic help
5468
5469Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5470``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
5471commands except "help" by using the following form::
5472
5473 $ wic help command
5474
5475For example, the following command returns help for the ``write``
5476command::
5477
5478 $ wic help write
5479
5480Wic supports help for three topics: ``overview``, ``plugins``, and
5481``kickstart``. You can get help for any topic using the following form::
5482
5483 $ wic help topic
5484
5485For example, the following returns overview help for Wic::
5486
5487 $ wic help overview
5488
5489There is one additional level of help for Wic. You can get help on
5490individual images through the ``list`` command. You can use the ``list``
5491command to return the available Wic images as follows::
5492
5493 $ wic list images
5494 genericx86 Create an EFI disk image for genericx86*
5495 edgerouter Create SD card image for Edgerouter
5496 beaglebone-yocto Create SD card image for Beaglebone
5497 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5498 systemd-bootdisk Create an EFI disk image with systemd-boot
5499 mkhybridiso Create a hybrid ISO image
5500 mkefidisk Create an EFI disk image
5501 sdimage-bootpart Create SD card image with a boot partition
5502 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5503 directdisk Create a 'pcbios' direct disk image
5504 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5505 qemuriscv Create qcow2 image for RISC-V QEMU machines
5506 directdisk-gpt Create a 'pcbios' direct disk image
5507 efi-bootdisk
5508
5509Once you know the list of available
5510Wic images, you can use ``help`` with the command to get help on a
5511particular image. For example, the following command returns help on the
5512"beaglebone-yocto" image::
5513
5514 $ wic list beaglebone-yocto help
5515
5516 Creates a partitioned SD card image for Beaglebone.
5517 Boot files are located in the first vfat partition.
5518
5519Operational Modes
5520-----------------
5521
5522You can use Wic in two different modes, depending on how much control
5523you need for specifying the OpenEmbedded build artifacts that are used
5524for creating the image: Raw and Cooked:
5525
5526- *Raw Mode:* You explicitly specify build artifacts through Wic
5527 command-line arguments.
5528
5529- *Cooked Mode:* The current
5530 :term:`MACHINE` setting and image
5531 name are used to automatically locate and provide the build
5532 artifacts. You just supply a kickstart file and the name of the image
5533 from which to use artifacts.
5534
5535Regardless of the mode you use, you need to have the build artifacts
5536ready and available.
5537
5538Raw Mode
5539~~~~~~~~
5540
5541Running Wic in raw mode allows you to specify all the partitions through
5542the ``wic`` command line. The primary use for raw mode is if you have
5543built your kernel outside of the Yocto Project :term:`Build Directory`.
5544In other words, you can point to arbitrary kernel, root filesystem locations,
5545and so forth. Contrast this behavior with cooked mode where Wic looks in the
5546:term:`Build Directory` (e.g. ``tmp/deploy/images/``\ machine).
5547
5548The general form of the ``wic`` command in raw mode is::
5549
5550 $ wic create wks_file options ...
5551
5552 Where:
5553
5554 wks_file:
5555 An OpenEmbedded kickstart file. You can provide
5556 your own custom file or use a file from a set of
5557 existing files as described by further options.
5558
5559 optional arguments:
5560 -h, --help show this help message and exit
5561 -o OUTDIR, --outdir OUTDIR
5562 name of directory to create image in
5563 -e IMAGE_NAME, --image-name IMAGE_NAME
5564 name of the image to use the artifacts from e.g. core-
5565 image-sato
5566 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5567 path to the /rootfs dir to use as the .wks rootfs
5568 source
5569 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5570 path to the dir containing the boot artifacts (e.g.
5571 /EFI or /syslinux dirs) to use as the .wks bootimg
5572 source
5573 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5574 path to the dir containing the kernel to use in the
5575 .wks bootimg
5576 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5577 path to the native sysroot containing the tools to use
5578 to build the image
5579 -s, --skip-build-check
5580 skip the build check
5581 -f, --build-rootfs build rootfs
5582 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5583 compress image with specified compressor
5584 -m, --bmap generate .bmap
5585 --no-fstab-update Do not change fstab file.
5586 -v VARS_DIR, --vars VARS_DIR
5587 directory with <image>.env files that store bitbake
5588 variables
5589 -D, --debug output debug information
5590
5591.. note::
5592
5593 You do not need root privileges to run Wic. In fact, you should not
5594 run as root when using the utility.
5595
5596Cooked Mode
5597~~~~~~~~~~~
5598
5599Running Wic in cooked mode leverages off artifacts in the
5600:term:`Build Directory`. In other words, you do not have to specify kernel or
5601root filesystem locations as part of the command. All you need to provide is
5602a kickstart file and the name of the image from which to use artifacts
5603by using the "-e" option. Wic looks in the :term:`Build Directory` (e.g.
5604``tmp/deploy/images/``\ machine) for artifacts.
5605
5606The general form of the ``wic`` command using Cooked Mode is as follows::
5607
5608 $ wic create wks_file -e IMAGE_NAME
5609
5610 Where:
5611
5612 wks_file:
5613 An OpenEmbedded kickstart file. You can provide
5614 your own custom file or use a file from a set of
5615 existing files provided with the Yocto Project
5616 release.
5617
5618 required argument:
5619 -e IMAGE_NAME, --image-name IMAGE_NAME
5620 name of the image to use the artifacts from e.g. core-
5621 image-sato
5622
5623Using an Existing Kickstart File
5624--------------------------------
5625
5626If you do not want to create your own kickstart file, you can use an
5627existing file provided by the Wic installation. As shipped, kickstart
5628files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
5629following two locations::
5630
5631 poky/meta-yocto-bsp/wic
5632 poky/scripts/lib/wic/canned-wks
5633
5634Use the following command to list the available kickstart files::
5635
5636 $ wic list images
5637 genericx86 Create an EFI disk image for genericx86*
5638 beaglebone-yocto Create SD card image for Beaglebone
5639 edgerouter Create SD card image for Edgerouter
5640 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
5641 directdisk-gpt Create a 'pcbios' direct disk image
5642 mkefidisk Create an EFI disk image
5643 directdisk Create a 'pcbios' direct disk image
5644 systemd-bootdisk Create an EFI disk image with systemd-boot
5645 mkhybridiso Create a hybrid ISO image
5646 sdimage-bootpart Create SD card image with a boot partition
5647 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5648 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5649
5650When you use an existing file, you
5651do not have to use the ``.wks`` extension. Here is an example in Raw
5652Mode that uses the ``directdisk`` file::
5653
5654 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5655 -k kernel_dir -n native_sysroot
5656
5657Here are the actual partition language commands used in the
5658``genericx86.wks`` file to generate an image::
5659
5660 # short-description: Create an EFI disk image for genericx86*
5661 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5662 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5663 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5664 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5665
5666 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5667
5668Using the Wic Plugin Interface
5669------------------------------
5670
5671You can extend and specialize Wic functionality by using Wic plugins.
5672This section explains the Wic plugin interface.
5673
5674.. note::
5675
5676 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5677 are beyond the scope of this section.
5678
5679Source plugins provide a mechanism to customize partition content during
5680the Wic image generation process. You can use source plugins to map
5681values that you specify using ``--source`` commands in kickstart files
5682(i.e. ``*.wks``) to a plugin implementation used to populate a given
5683partition.
5684
5685.. note::
5686
5687 If you use plugins that have build-time dependencies (e.g. native
5688 tools, bootloaders, and so forth) when building a Wic image, you need
5689 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
5690 variable.
5691
5692Source plugins are subclasses defined in plugin files. As shipped, the
5693Yocto Project provides several plugin files. You can see the source
5694plugin files that ship with the Yocto Project
5695:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
5696Each of these plugin files contains source plugins that are designed to
5697populate a specific Wic image partition.
5698
5699Source plugins are subclasses of the ``SourcePlugin`` class, which is
5700defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5701the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5702file is a subclass of the ``SourcePlugin`` class, which is found in the
5703``pluginbase.py`` file.
5704
5705You can also implement source plugins in a layer outside of the Source
5706Repositories (external layer). To do so, be sure that your plugin files
5707are located in a directory whose path is
5708``scripts/lib/wic/plugins/source/`` within your external layer. When the
5709plugin files are located there, the source plugins they contain are made
5710available to Wic.
5711
5712When the Wic implementation needs to invoke a partition-specific
5713implementation, it looks for the plugin with the same name as the
5714``--source`` parameter used in the kickstart file given to that
5715partition. For example, if the partition is set up using the following
5716command in a kickstart file::
5717
5718 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5719
5720The methods defined as class
5721members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5722``bootimg-pcbios.py`` plugin file are used.
5723
5724To be more concrete, here is the corresponding plugin definition from
5725the ``bootimg-pcbios.py`` file for the previous command along with an
5726example method called by the Wic implementation when it needs to prepare
5727a partition using an implementation-specific function::
5728
5729 .
5730 .
5731 .
5732 class BootimgPcbiosPlugin(SourcePlugin):
5733 """
5734 Create MBR boot partition and install syslinux on it.
5735 """
5736
5737 name = 'bootimg-pcbios'
5738 .
5739 .
5740 .
5741 @classmethod
5742 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5743 oe_builddir, bootimg_dir, kernel_dir,
5744 rootfs_dir, native_sysroot):
5745 """
5746 Called to do the actual content population for a partition i.e. it
5747 'prepares' the partition to be incorporated into the image.
5748 In this case, prepare content for legacy bios boot partition.
5749 """
5750 .
5751 .
5752 .
5753
5754If a
5755subclass (plugin) itself does not implement a particular function, Wic
5756locates and uses the default version in the superclass. It is for this
5757reason that all source plugins are derived from the ``SourcePlugin``
5758class.
5759
5760The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5761a set of methods that source plugins can implement or override. Any
5762plugins (subclass of ``SourcePlugin``) that do not implement a
5763particular method inherit the implementation of the method from the
5764``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5765class in the ``pluginbase.py`` file for details:
5766
5767The following list describes the methods implemented in the
5768``SourcePlugin`` class:
5769
5770- ``do_prepare_partition()``: Called to populate a partition with
5771 actual content. In other words, the method prepares the final
5772 partition image that is incorporated into the disk image.
5773
5774- ``do_configure_partition()``: Called before
5775 ``do_prepare_partition()`` to create custom configuration files for a
5776 partition (e.g. syslinux or grub configuration files).
5777
5778- ``do_install_disk()``: Called after all partitions have been
5779 prepared and assembled into a disk image. This method provides a hook
5780 to allow finalization of a disk image (e.g. writing an MBR).
5781
5782- ``do_stage_partition()``: Special content-staging hook called
5783 before ``do_prepare_partition()``. This method is normally empty.
5784
5785 Typically, a partition just uses the passed-in parameters (e.g. the
5786 unmodified value of ``bootimg_dir``). However, in some cases, things
5787 might need to be more tailored. As an example, certain files might
5788 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5789 allows those files to be staged in a customized fashion.
5790
5791 .. note::
5792
5793 ``get_bitbake_var()`` allows you to access non-standard variables that
5794 you might want to use for this behavior.
5795
5796You can extend the source plugin mechanism. To add more hooks, create
5797more source plugin methods within ``SourcePlugin`` and the corresponding
5798derived subclasses. The code that calls the plugin methods uses the
5799``plugin.get_source_plugin_methods()`` function to find the method or
5800methods needed by the call. Retrieval of those methods is accomplished
5801by filling up a dict with keys that contain the method names of
5802interest. On success, these will be filled in with the actual methods.
5803See the Wic implementation for examples and details.
5804
5805Wic Examples
5806------------
5807
5808This section provides several examples that show how to use the Wic
5809utility. All the examples assume the list of requirements in the
5810":ref:`dev-manual/common-tasks:requirements`" section have been met. The
5811examples assume the previously generated image is
5812``core-image-minimal``.
5813
5814Generate an Image using an Existing Kickstart File
5815~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5816
5817This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
5818file::
5819
5820 $ wic create mkefidisk -e core-image-minimal
5821 INFO: Building wic-tools...
5822 .
5823 .
5824 .
5825 INFO: The new image(s) can be found here:
5826 ./mkefidisk-201804191017-sda.direct
5827
5828 The following build artifacts were used to create the image(s):
5829 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5830 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5831 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5832 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5833
5834 INFO: The image(s) were created using OE kickstart file:
5835 /home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
5836
5837The previous example shows the easiest way to create an image by running
5838in cooked mode and supplying a kickstart file and the "-e" option to
5839point to the existing build artifacts. Your ``local.conf`` file needs to
5840have the :term:`MACHINE` variable set
5841to the machine you are using, which is "qemux86" in this example.
5842
5843Once the image builds, the output provides image location, artifact use,
5844and kickstart file information.
5845
5846.. note::
5847
5848 You should always verify the details provided in the output to make
5849 sure that the image was indeed created exactly as expected.
5850
5851Continuing with the example, you can now write the image from the
5852:term:`Build Directory` onto a USB stick, or whatever media for which you
5853built your image, and boot from the media. You can write the image by using
5854``bmaptool`` or ``dd``::
5855
5856 $ oe-run-native bmap-tools-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
5857
5858or ::
5859
5860 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5861
5862.. note::
5863
5864 For more information on how to use the ``bmaptool``
5865 to flash a device with an image, see the
5866 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``"
5867 section.
5868
5869Using a Modified Kickstart File
5870~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5871
5872Because partitioned image creation is driven by the kickstart file, it
5873is easy to affect image creation by changing the parameters in the file.
5874This next example demonstrates that through modification of the
5875``directdisk-gpt`` kickstart file.
5876
5877As mentioned earlier, you can use the command ``wic list images`` to
5878show the list of existing kickstart files. The directory in which the
5879``directdisk-gpt.wks`` file resides is
5880``scripts/lib/image/canned-wks/``, which is located in the
5881:term:`Source Directory` (e.g. ``poky``).
5882Because available files reside in this directory, you can create and add
5883your own custom files to the directory. Subsequent use of the
5884``wic list images`` command would then include your kickstart files.
5885
5886In this example, the existing ``directdisk-gpt`` file already does most
5887of what is needed. However, for the hardware in this example, the image
5888will need to boot from ``sdb`` instead of ``sda``, which is what the
5889``directdisk-gpt`` kickstart file uses.
5890
5891The example begins by making a copy of the ``directdisk-gpt.wks`` file
5892in the ``scripts/lib/image/canned-wks`` directory and then by changing
5893the lines that specify the target disk from which to boot.
5894::
5895
5896 $ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5897 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5898
5899Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5900changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5901example changes the following two lines and leaves the remaining lines
5902untouched::
5903
5904 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5905 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5906
5907Once the lines are changed, the
5908example generates the ``directdisksdb-gpt`` image. The command points
5909the process at the ``core-image-minimal`` artifacts for the Next Unit of
5910Computing (nuc) :term:`MACHINE` the
5911``local.conf``.
5912::
5913
5914 $ wic create directdisksdb-gpt -e core-image-minimal
5915 INFO: Building wic-tools...
5916 .
5917 .
5918 .
5919 Initialising tasks: 100% |#######################################| Time: 0:00:01
5920 NOTE: Executing SetScene Tasks
5921 NOTE: Executing RunQueue Tasks
5922 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5923 INFO: Creating image(s)...
5924
5925 INFO: The new image(s) can be found here:
5926 ./directdisksdb-gpt-201710090938-sdb.direct
5927
5928 The following build artifacts were used to create the image(s):
5929 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5930 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5931 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5932 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5933
5934 INFO: The image(s) were created using OE kickstart file:
5935 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5936
5937Continuing with the example, you can now directly ``dd`` the image to a
5938USB stick, or whatever media for which you built your image, and boot
5939the resulting media::
5940
5941 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5942 140966+0 records in
5943 140966+0 records out
5944 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5945 $ sudo eject /dev/sdb
5946
5947Using a Modified Kickstart File and Running in Raw Mode
5948~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5949
5950This next example manually specifies each build artifact (runs in Raw
5951Mode) and uses a modified kickstart file. The example also uses the
5952``-o`` option to cause Wic to create the output somewhere other than the
5953default output directory, which is the current directory::
5954
5955 $ wic create test.wks -o /home/stephano/testwic \
5956 --rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5957 --bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5958 --kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \
5959 --native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
5960
5961 INFO: Creating image(s)...
5962
5963 INFO: The new image(s) can be found here:
5964 /home/stephano/testwic/test-201710091445-sdb.direct
5965
5966 The following build artifacts were used to create the image(s):
5967 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5968 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5969 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5970 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5971
5972 INFO: The image(s) were created using OE kickstart file:
5973 test.wks
5974
5975For this example,
5976:term:`MACHINE` did not have to be
5977specified in the ``local.conf`` file since the artifact is manually
5978specified.
5979
5980Using Wic to Manipulate an Image
5981~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5982
5983Wic image manipulation allows you to shorten turnaround time during
5984image development. For example, you can use Wic to delete the kernel
5985partition of a Wic image and then insert a newly built kernel. This
5986saves you time from having to rebuild the entire image each time you
5987modify the kernel.
5988
5989.. note::
5990
5991 In order to use Wic to manipulate a Wic image as in this example,
5992 your development machine must have the ``mtools`` package installed.
5993
5994The following example examines the contents of the Wic image, deletes
5995the existing kernel, and then inserts a new kernel:
5996
59971. *List the Partitions:* Use the ``wic ls`` command to list all the
5998 partitions in the Wic image::
5999
6000 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
6001 Num Start End Size Fstype
6002 1 1048576 25041919 23993344 fat16
6003 2 25165824 72157183 46991360 ext4
6004
6005 The previous output shows two partitions in the
6006 ``core-image-minimal-qemux86.wic`` image.
6007
60082. *Examine a Particular Partition:* Use the ``wic ls`` command again
6009 but in a different form to examine a particular partition.
6010
6011 .. note::
6012
6013 You can get command usage on any Wic command using the following
6014 form::
6015
6016 $ wic help command
6017
6018
6019 For example, the following command shows you the various ways to
6020 use the
6021 wic ls
6022 command::
6023
6024 $ wic help ls
6025
6026
6027 The following command shows what is in partition one::
6028
6029 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
6030 Volume in drive : is boot
6031 Volume Serial Number is E894-1809
6032 Directory for ::/
6033
6034 libcom32 c32 186500 2017-10-09 16:06
6035 libutil c32 24148 2017-10-09 16:06
6036 syslinux cfg 220 2017-10-09 16:06
6037 vesamenu c32 27104 2017-10-09 16:06
6038 vmlinuz 6904608 2017-10-09 16:06
6039 5 files 7 142 580 bytes
6040 16 582 656 bytes free
6041
6042 The previous output shows five files, with the
6043 ``vmlinuz`` being the kernel.
6044
6045 .. note::
6046
6047 If you see the following error, you need to update or create a
6048 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
6049 in the file. Then, run the Wic command again::
6050
6051 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
6052 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
6053 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
6054
6055
60563. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
6057 ``vmlinuz`` file (kernel)::
6058
6059 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6060
60614. *Add In the New Kernel:* Use the ``wic cp`` command to add the
6062 updated kernel to the Wic image. Depending on how you built your
6063 kernel, it could be in different places. If you used ``devtool`` and
6064 an SDK to build your kernel, it resides in the ``tmp/work`` directory
6065 of the extensible SDK. If you used ``make`` to build the kernel, the
6066 kernel will be in the ``workspace/sources`` area.
6067
6068 The following example assumes ``devtool`` was used to build the
6069 kernel::
6070
6071 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
6072 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6073
6074 Once the new kernel is added back into the image, you can use the
6075 ``dd`` command or :ref:`bmaptool
6076 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>`
6077 to flash your wic image onto an SD card or USB stick and test your
6078 target.
6079
6080 .. note::
6081
6082 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
6083
6084Flashing Images Using ``bmaptool``
6085==================================
6086
6087A fast and easy way to flash an image to a bootable device is to use
6088Bmaptool, which is integrated into the OpenEmbedded build system.
6089Bmaptool is a generic tool that creates a file's block map (bmap) and
6090then uses that map to copy the file. As compared to traditional tools
6091such as dd or cp, Bmaptool can copy (or flash) large files like raw
6092system image files much faster.
6093
6094.. note::
6095
6096 - If you are using Ubuntu or Debian distributions, you can install
6097 the ``bmap-tools`` package using the following command and then
6098 use the tool without specifying ``PATH`` even from the root
6099 account::
6100
6101 $ sudo apt install bmap-tools
6102
6103 - If you are unable to install the ``bmap-tools`` package, you will
6104 need to build Bmaptool before using it. Use the following command::
6105
6106 $ bitbake bmap-tools-native
6107
6108Following, is an example that shows how to flash a Wic image. Realize
6109that while this example uses a Wic image, you can use Bmaptool to flash
6110any type of image. Use these steps to flash an image using Bmaptool:
6111
61121. *Update your local.conf File:* You need to have the following set
6113 in your ``local.conf`` file before building your image::
6114
6115 IMAGE_FSTYPES += "wic wic.bmap"
6116
61172. *Get Your Image:* Either have your image ready (pre-built with the
6118 :term:`IMAGE_FSTYPES`
6119 setting previously mentioned) or take the step to build the image::
6120
6121 $ bitbake image
6122
61233. *Flash the Device:* Flash the device with the image by using Bmaptool
6124 depending on your particular setup. The following commands assume the
6125 image resides in the :term:`Build Directory`'s ``deploy/images/`` area:
6126
6127 - If you have write access to the media, use this command form::
6128
6129 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6130
6131 - If you do not have write access to the media, set your permissions
6132 first and then use the same command form::
6133
6134 $ sudo chmod 666 /dev/sdX
6135 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6136
6137For help on the ``bmaptool`` command, use the following command::
6138
6139 $ bmaptool --help
6140
6141Making Images More Secure
6142=========================
6143
6144Security is of increasing concern for embedded devices. Consider the
6145issues and problems discussed in just this sampling of work found across
6146the Internet:
6147
6148- *"*\ `Security Risks of Embedded
6149 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
6150 by Bruce Schneier
6151
6152- *"*\ `Internet Census
6153 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
6154 Botnet
6155
6156- *"*\ `Security Issues for Embedded
6157 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
6158 by Jake Edge
6159
6160When securing your image is of concern, there are steps, tools, and
6161variables that you can consider to help you reach the security goals you
6162need for your particular device. Not all situations are identical when
6163it comes to making an image secure. Consequently, this section provides
6164some guidance and suggestions for consideration when you want to make
6165your image more secure.
6166
6167.. note::
6168
6169 Because the security requirements and risks are different for every
6170 type of device, this section cannot provide a complete reference on
6171 securing your custom OS. It is strongly recommended that you also
6172 consult other sources of information on embedded Linux system
6173 hardening and on security.
6174
6175General Considerations
6176----------------------
6177
6178There are general considerations that help you create more secure images.
6179You should consider the following suggestions to make your device
6180more secure:
6181
6182- Scan additional code you are adding to the system (e.g. application
6183 code) by using static analysis tools. Look for buffer overflows and
6184 other potential security problems.
6185
6186- Pay particular attention to the security for any web-based
6187 administration interface.
6188
6189 Web interfaces typically need to perform administrative functions and
6190 tend to need to run with elevated privileges. Thus, the consequences
6191 resulting from the interface's security becoming compromised can be
6192 serious. Look for common web vulnerabilities such as
6193 cross-site-scripting (XSS), unvalidated inputs, and so forth.
6194
6195 As with system passwords, the default credentials for accessing a
6196 web-based interface should not be the same across all devices. This
6197 is particularly true if the interface is enabled by default as it can
6198 be assumed that many end-users will not change the credentials.
6199
6200- Ensure you can update the software on the device to mitigate
6201 vulnerabilities discovered in the future. This consideration
6202 especially applies when your device is network-enabled.
6203
6204- Regularly scan and apply fixes for CVE security issues affecting
6205 all software components in the product, see ":ref:`dev-manual/common-tasks:checking for vulnerabilities`".
6206
6207- Regularly update your version of Poky and OE-Core from their upstream
6208 developers, e.g. to apply updates and security fixes from stable
6209 and LTS branches.
6210
6211- Ensure you remove or disable debugging functionality before producing
6212 the final image. For information on how to do this, see the
6213 ":ref:`dev-manual/common-tasks:considerations specific to the openembedded build system`"
6214 section.
6215
6216- Ensure you have no network services listening that are not needed.
6217
6218- Remove any software from the image that is not needed.
6219
6220- Enable hardware support for secure boot functionality when your
6221 device supports this functionality.
6222
6223Security Flags
6224--------------
6225
6226The Yocto Project has security flags that you can enable that help make
6227your build output more secure. The security flags are in the
6228``meta/conf/distro/include/security_flags.inc`` file in your
6229:term:`Source Directory` (e.g. ``poky``).
6230
6231.. note::
6232
6233 Depending on the recipe, certain security flags are enabled and
6234 disabled by default.
6235
6236Use the following line in your ``local.conf`` file or in your custom
6237distribution configuration file to enable the security compiler and
6238linker flags for your build::
6239
6240 require conf/distro/include/security_flags.inc
6241
6242Considerations Specific to the OpenEmbedded Build System
6243--------------------------------------------------------
6244
6245You can take some steps that are specific to the OpenEmbedded build
6246system to make your images more secure:
6247
6248- Ensure "debug-tweaks" is not one of your selected
6249 :term:`IMAGE_FEATURES`.
6250 When creating a new project, the default is to provide you with an
6251 initial ``local.conf`` file that enables this feature using the
6252 :term:`EXTRA_IMAGE_FEATURES`
6253 variable with the line::
6254
6255 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6256
6257 To disable that feature, simply comment out that line in your
6258 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
6259 "debug-tweaks" before producing your final image. Among other things,
6260 leaving this in place sets the root password as blank, which makes
6261 logging in for debugging or inspection easy during development but
6262 also means anyone can easily log in during production.
6263
6264- It is possible to set a root password for the image and also to set
6265 passwords for any extra users you might add (e.g. administrative or
6266 service type users). When you set up passwords for multiple images or
6267 users, you should not duplicate passwords.
6268
6269 To set up passwords, use the
6270 :ref:`extrausers <ref-classes-extrausers>`
6271 class, which is the preferred method. For an example on how to set up
6272 both root and user passwords, see the
6273 ":ref:`ref-classes-extrausers`" section.
6274
6275 .. note::
6276
6277 When adding extra user accounts or setting a root password, be
6278 cautious about setting the same password on every device. If you
6279 do this, and the password you have set is exposed, then every
6280 device is now potentially compromised. If you need this access but
6281 want to ensure security, consider setting a different, random
6282 password for each device. Typically, you do this as a separate
6283 step after you deploy the image onto the device.
6284
6285- Consider enabling a Mandatory Access Control (MAC) framework such as
6286 SMACK or SELinux and tuning it appropriately for your device's usage.
6287 You can find more information in the
6288 :yocto_git:`meta-selinux </meta-selinux/>` layer.
6289
6290Tools for Hardening Your Image
6291------------------------------
6292
6293The Yocto Project provides tools for making your image more secure. You
6294can find these tools in the ``meta-security`` layer of the
6295:yocto_git:`Yocto Project Source Repositories <>`.
6296
6297Creating Your Own Distribution
6298==============================
6299
6300When you build an image using the Yocto Project and do not alter any
6301distribution :term:`Metadata`, you are
6302creating a Poky distribution. If you wish to gain more control over
6303package alternative selections, compile-time options, and other
6304low-level configurations, you can create your own distribution.
6305
6306To create your own distribution, the basic steps consist of creating
6307your own distribution layer, creating your own distribution
6308configuration file, and then adding any needed code and Metadata to the
6309layer. The following steps provide some more detail:
6310
6311- *Create a layer for your new distro:* Create your distribution layer
6312 so that you can keep your Metadata and code for the distribution
6313 separate. It is strongly recommended that you create and use your own
6314 layer for configuration and code. Using your own layer as compared to
6315 just placing configurations in a ``local.conf`` configuration file
6316 makes it easier to reproduce the same build configuration when using
6317 multiple build machines. See the
6318 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
6319 section for information on how to quickly set up a layer.
6320
6321- *Create the distribution configuration file:* The distribution
6322 configuration file needs to be created in the ``conf/distro``
6323 directory of your layer. You need to name it using your distribution
6324 name (e.g. ``mydistro.conf``).
6325
6326 .. note::
6327
6328 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6329 name of your distribution.
6330
6331 You can split out parts of your configuration file into include files
6332 and then "require" them from within your distribution configuration
6333 file. Be sure to place the include files in the
6334 ``conf/distro/include`` directory of your layer. A common example
6335 usage of include files would be to separate out the selection of
6336 desired version and revisions for individual recipes.
6337
6338 Your configuration file needs to set the following required
6339 variables:
6340
6341 - :term:`DISTRO_NAME`
6342
6343 - :term:`DISTRO_VERSION`
6344
6345 These following variables are optional and you typically set them
6346 from the distribution configuration file:
6347
6348 - :term:`DISTRO_FEATURES`
6349
6350 - :term:`DISTRO_EXTRA_RDEPENDS`
6351
6352 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6353
6354 - :term:`TCLIBC`
6355
6356 .. tip::
6357
6358 If you want to base your distribution configuration file on the
6359 very basic configuration from OE-Core, you can use
6360 ``conf/distro/defaultsetup.conf`` as a reference and just include
6361 variables that differ as compared to ``defaultsetup.conf``.
6362 Alternatively, you can create a distribution configuration file
6363 from scratch using the ``defaultsetup.conf`` file or configuration files
6364 from another distribution such as Poky as a reference.
6365
6366- *Provide miscellaneous variables:* Be sure to define any other
6367 variables for which you want to create a default or enforce as part
6368 of the distribution configuration. You can include nearly any
6369 variable from the ``local.conf`` file. The variables you use are not
6370 limited to the list in the previous bulleted item.
6371
6372- *Point to Your distribution configuration file:* In your ``local.conf``
6373 file in the :term:`Build Directory`, set your :term:`DISTRO` variable to
6374 point to your distribution's configuration file. For example, if your
6375 distribution's configuration file is named ``mydistro.conf``, then
6376 you point to it as follows::
6377
6378 DISTRO = "mydistro"
6379
6380- *Add more to the layer if necessary:* Use your layer to hold other
6381 information needed for the distribution:
6382
6383 - Add recipes for installing distro-specific configuration files
6384 that are not already installed by another recipe. If you have
6385 distro-specific configuration files that are included by an
6386 existing recipe, you should add an append file (``.bbappend``) for
6387 those. For general information and recommendations on how to add
6388 recipes to your layer, see the
6389 ":ref:`dev-manual/common-tasks:creating your own layer`" and
6390 ":ref:`dev-manual/common-tasks:following best practices when creating layers`"
6391 sections.
6392
6393 - Add any image recipes that are specific to your distribution.
6394
6395 - Add a ``psplash`` append file for a branded splash screen. For
6396 information on append files, see the
6397 ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`"
6398 section.
6399
6400 - Add any other append files to make custom changes that are
6401 specific to individual recipes.
6402
6403Creating a Custom Template Configuration Directory
6404==================================================
6405
6406If you are producing your own customized version of the build system for
6407use by other users, you might want to provide a custom build configuration
6408that includes all the necessary settings and layers (i.e. ``local.conf`` and
6409``bblayers.conf`` that are created in a new :term:`Build Directory`) and a custom
6410message that is shown when setting up the build. This can be done by
6411creating one or more template configuration directories in your
6412custom distribution layer.
6413
6414This can be done by using ``bitbake-layers save-build-conf``::
6415
6416 $ bitbake-layers save-build-conf ../../meta-alex/ test-1
6417 NOTE: Starting bitbake server...
6418 NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
6419 Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
6420 You can try out the configuration with
6421 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
6422
6423The above command takes the config files from the currently active :term:`Build Directory` under ``conf``,
6424replaces site-specific paths in ``bblayers.conf`` with ``##OECORE##``-relative paths, and copies
6425the config files into a specified layer under a specified template name.
6426
6427To use those saved templates as a starting point for a build, users should point
6428to one of them with :term:`TEMPLATECONF` environment variable::
6429
6430 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
6431
6432The OpenEmbedded build system uses the environment variable
6433:term:`TEMPLATECONF` to locate the directory from which it gathers
6434configuration information that ultimately ends up in the
6435:term:`Build Directory` ``conf`` directory.
6436
6437If :term:`TEMPLATECONF` is not set, the default value is obtained
6438from ``.templateconf`` file that is read from the same directory as
6439``oe-init-build-env`` script. For the Poky reference distribution this
6440would be::
6441
6442 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf/templates/default}
6443
6444If you look at a configuration template directory, you will
6445see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6446``conf-notes.txt`` files. The build system uses these files to form the
6447respective ``bblayers.conf`` file, ``local.conf`` file, and show
6448users a note about the build they're setting up
6449when running the ``oe-init-build-env`` setup script. These can be
6450edited further if needed to improve or change the build configurations
6451available to the users.
6452
6453Conserving Disk Space
6454=====================
6455
6456Conserving Disk Space During Builds
6457-----------------------------------
6458
6459To help conserve disk space during builds, you can add the following
6460statement to your project's ``local.conf`` configuration file found in
6461the :term:`Build Directory`::
6462
6463 INHERIT += "rm_work"
6464
6465Adding this statement deletes the work directory used for
6466building a recipe once the recipe is built. For more information on
6467"rm_work", see the
6468:ref:`rm_work <ref-classes-rm-work>` class in the
6469Yocto Project Reference Manual.
6470
6471Purging Duplicate Shared State Cache Files
6472-------------------------------------------
6473
6474After multiple build iterations, the Shared State (sstate) cache can contain
6475duplicate cache files for a given package, while only the most recent one
6476is likely to be reusable. The following command purges all but the
6477newest sstate cache file for each package::
6478
6479 sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
6480
6481This command will ask you to confirm the deletions it identifies.
6482
6483.. note::
6484
6485 The duplicated sstate cache files of one package must have the same
6486 architecture, which means that sstate cache files with multiple
6487 architectures are not considered as duplicate.
6488
6489Run ``sstate-cache-management.sh`` for more details about this script.
6490
6491Working with Packages
6492=====================
6493
6494This section describes a few tasks that involve packages:
6495
6496- :ref:`dev-manual/common-tasks:excluding packages from an image`
6497
6498- :ref:`dev-manual/common-tasks:incrementing a package version`
6499
6500- :ref:`dev-manual/common-tasks:handling optional module packaging`
6501
6502- :ref:`dev-manual/common-tasks:using runtime package management`
6503
6504- :ref:`dev-manual/common-tasks:generating and using signed packages`
6505
6506- :ref:`Setting up and running package test
6507 (ptest) <dev-manual/common-tasks:testing packages with ptest>`
6508
6509- :ref:`dev-manual/common-tasks:creating node package manager (npm) packages`
6510
6511- :ref:`dev-manual/common-tasks:adding custom metadata to packages`
6512
6513Excluding Packages from an Image
6514--------------------------------
6515
6516You might find it necessary to prevent specific packages from being
6517installed into an image. If so, you can use several variables to direct
6518the build system to essentially ignore installing recommended packages
6519or to not install a package at all.
6520
6521The following list introduces variables you can use to prevent packages
6522from being installed into your image. Each of these variables only works
6523with IPK and RPM package types, not for Debian packages.
6524Also, you can use these variables from your ``local.conf`` file
6525or attach them to a specific image recipe by using a recipe name
6526override. For more detail on the variables, see the descriptions in the
6527Yocto Project Reference Manual's glossary chapter.
6528
6529- :term:`BAD_RECOMMENDATIONS`:
6530 Use this variable to specify "recommended-only" packages that you do
6531 not want installed.
6532
6533- :term:`NO_RECOMMENDATIONS`:
6534 Use this variable to prevent all "recommended-only" packages from
6535 being installed.
6536
6537- :term:`PACKAGE_EXCLUDE`:
6538 Use this variable to prevent specific packages from being installed
6539 regardless of whether they are "recommended-only" or not. You need to
6540 realize that the build process could fail with an error when you
6541 prevent the installation of a package whose presence is required by
6542 an installed package.
6543
6544Incrementing a Package Version
6545------------------------------
6546
6547This section provides some background on how binary package versioning
6548is accomplished and presents some of the services, variables, and
6549terminology involved.
6550
6551In order to understand binary package versioning, you need to consider
6552the following:
6553
6554- Binary Package: The binary package that is eventually built and
6555 installed into an image.
6556
6557- Binary Package Version: The binary package version is composed of two
6558 components --- a version and a revision.
6559
6560 .. note::
6561
6562 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
6563 but this discussion for the most part ignores :term:`PE`.
6564
6565 The version and revision are taken from the
6566 :term:`PV` and
6567 :term:`PR` variables, respectively.
6568
6569- :term:`PV`: The recipe version. :term:`PV` represents the version of the
6570 software being packaged. Do not confuse :term:`PV` with the binary
6571 package version.
6572
6573- :term:`PR`: The recipe revision.
6574
6575- :term:`SRCPV`: The OpenEmbedded
6576 build system uses this string to help define the value of :term:`PV` when
6577 the source code revision needs to be included in it.
6578
6579- :yocto_wiki:`PR Service </PR_Service>`: A
6580 network-based service that helps automate keeping package feeds
6581 compatible with existing package manager applications such as RPM,
6582 APT, and OPKG.
6583
6584Whenever the binary package content changes, the binary package version
6585must change. Changing the binary package version is accomplished by
6586changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
6587values occurs one of two ways:
6588
6589- Automatically using a Package Revision Service (PR Service).
6590
6591- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
6592
6593Given a primary challenge of any build system and its users is how to
6594maintain a package feed that is compatible with existing package manager
6595applications such as RPM, APT, and OPKG, using an automated system is
6596much preferred over a manual system. In either system, the main
6597requirement is that binary package version numbering increases in a
6598linear fashion and that there is a number of version components that
6599support that linear progression. For information on how to ensure
6600package revisioning remains linear, see the
6601":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
6602section.
6603
6604The following three sections provide related information on the PR
6605Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
6606how to ensure binary package revisioning remains linear.
6607
6608Working With a PR Service
6609~~~~~~~~~~~~~~~~~~~~~~~~~
6610
6611As mentioned, attempting to maintain revision numbers in the
6612:term:`Metadata` is error prone, inaccurate,
6613and causes problems for people submitting recipes. Conversely, the PR
6614Service automatically generates increasing numbers, particularly the
6615revision field, which removes the human element.
6616
6617.. note::
6618
6619 For additional information on using a PR Service, you can see the
6620 :yocto_wiki:`PR Service </PR_Service>` wiki page.
6621
6622The Yocto Project uses variables in order of decreasing priority to
6623facilitate revision numbering (i.e.
6624:term:`PE`,
6625:term:`PV`, and
6626:term:`PR` for epoch, version, and
6627revision, respectively). The values are highly dependent on the policies
6628and procedures of a given distribution and package feed.
6629
6630Because the OpenEmbedded build system uses
6631":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
6632unique to a given build, the build system knows when to rebuild
6633packages. All the inputs into a given task are represented by a
6634signature, which can trigger a rebuild when different. Thus, the build
6635system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
6636trigger a rebuild. The signatures, however, can be used to generate
6637these values.
6638
6639The PR Service works with both ``OEBasic`` and ``OEBasicHash``
6640generators. The value of :term:`PR` bumps when the checksum changes and the
6641different generator mechanisms change signatures under different
6642circumstances.
6643
6644As implemented, the build system includes values from the PR Service
6645into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
6646becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
6647:term:`PR` values to be used for whatever reasons, which include manual
6648:term:`PR` bumps, should it be necessary.
6649
6650By default, the PR Service is not enabled or running. Thus, the packages
6651generated are just "self consistent". The build system adds and removes
6652packages and there are no guarantees about upgrade paths but images will
6653be consistent and correct with the latest changes.
6654
6655The simplest form for a PR Service is for a single host development system
6656that builds the package feed (building system). For this scenario, you can
6657enable a local PR Service by setting :term:`PRSERV_HOST` in your
6658``local.conf`` file in the :term:`Build Directory`::
6659
6660 PRSERV_HOST = "localhost:0"
6661
6662Once the service is started, packages will automatically
6663get increasing :term:`PR` values and BitBake takes care of starting and
6664stopping the server.
6665
6666If you have a more complex setup where multiple host development systems
6667work against a common, shared package feed, you have a single PR Service
6668running and it is connected to each building system. For this scenario,
6669you need to start the PR Service using the ``bitbake-prserv`` command::
6670
6671 bitbake-prserv --host ip --port port --start
6672
6673In addition to
6674hand-starting the service, you need to update the ``local.conf`` file of
6675each building system as described earlier so each system points to the
6676server and port.
6677
6678It is also recommended you use build history, which adds some sanity
6679checks to binary package versions, in conjunction with the server that
6680is running the PR Service. To enable build history, add the following to
6681each building system's ``local.conf`` file::
6682
6683 # It is recommended to activate "buildhistory" for testing the PR service
6684 INHERIT += "buildhistory"
6685 BUILDHISTORY_COMMIT = "1"
6686
6687For information on build
6688history, see the
6689":ref:`dev-manual/common-tasks:maintaining build output quality`" section.
6690
6691.. note::
6692
6693 The OpenEmbedded build system does not maintain :term:`PR` information as
6694 part of the shared state (sstate) packages. If you maintain an sstate
6695 feed, it's expected that either all your building systems that
6696 contribute to the sstate feed use a shared PR Service, or you do not
6697 run a PR Service on any of your building systems. Having some systems
6698 use a PR Service while others do not leads to obvious problems.
6699
6700 For more information on shared state, see the
6701 ":ref:`overview-manual/concepts:shared state cache`"
6702 section in the Yocto Project Overview and Concepts Manual.
6703
6704Manually Bumping PR
6705~~~~~~~~~~~~~~~~~~~
6706
6707The alternative to setting up a PR Service is to manually "bump" the
6708:term:`PR` variable.
6709
6710If a committed change results in changing the package output, then the
6711value of the PR variable needs to be increased (or "bumped") as part of
6712that commit. For new recipes you should add the :term:`PR` variable and set
6713its initial value equal to "r0", which is the default. Even though the
6714default value is "r0", the practice of adding it to a new recipe makes
6715it harder to forget to bump the variable when you make changes to the
6716recipe in future.
6717
6718If you are sharing a common ``.inc`` file with multiple recipes, you can
6719also use the :term:`INC_PR` variable to ensure that the recipes sharing the
6720``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
6721``.inc`` file must set :term:`INC_PR` (initially to "r0"), and all recipes
6722referring to it should set :term:`PR` to "${INC_PR}.0" initially,
6723incrementing the last number when the recipe is changed. If the ``.inc``
6724file is changed then its :term:`INC_PR` should be incremented.
6725
6726When upgrading the version of a binary package, assuming the :term:`PV`
6727changes, the :term:`PR` variable should be reset to "r0" (or "${INC_PR}.0"
6728if you are using :term:`INC_PR`).
6729
6730Usually, version increases occur only to binary packages. However, if
6731for some reason :term:`PV` changes but does not increase, you can increase
6732the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
6733"0".
6734
6735Binary package version numbering strives to follow the `Debian Version
6736Field Policy
6737Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
6738These guidelines define how versions are compared and what "increasing"
6739a version means.
6740
6741Automatically Incrementing a Package Version Number
6742~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6743
6744When fetching a repository, BitBake uses the
6745:term:`SRCREV` variable to determine
6746the specific source code revision from which to build. You set the
6747:term:`SRCREV` variable to
6748:term:`AUTOREV` to cause the
6749OpenEmbedded build system to automatically use the latest revision of
6750the software::
6751
6752 SRCREV = "${AUTOREV}"
6753
6754Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
6755automatically update the version whenever the revision of the source
6756code changes. Here is an example::
6757
6758 PV = "1.0+git${SRCPV}"
6759
6760The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
6761
6762.. code-block:: none
6763
6764 AUTOINC+source_code_revision
6765
6766The build system replaces the ``AUTOINC``
6767with a number. The number used depends on the state of the PR Service:
6768
6769- If PR Service is enabled, the build system increments the number,
6770 which is similar to the behavior of
6771 :term:`PR`. This behavior results in
6772 linearly increasing package versions, which is desirable. Here is an
6773 example:
6774
6775 .. code-block:: none
6776
6777 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6778 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6779
6780- If PR Service is not enabled, the build system replaces the
6781 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6782 changing the package version since the source revision is included.
6783 However, package versions are not increased linearly. Here is an
6784 example:
6785
6786 .. code-block:: none
6787
6788 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6789 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6790
6791In summary, the OpenEmbedded build system does not track the history of
6792binary package versions for this purpose. ``AUTOINC``, in this case, is
6793comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
6794package version is simply replaced by "0". If PR server is enabled, the
6795build system keeps track of the package versions and bumps the number
6796when the package revision changes.
6797
6798Handling Optional Module Packaging
6799----------------------------------
6800
6801Many pieces of software split functionality into optional modules (or
6802plugins) and the plugins that are built might depend on configuration
6803options. To avoid having to duplicate the logic that determines what
6804modules are available in your recipe or to avoid having to package each
6805module by hand, the OpenEmbedded build system provides functionality to
6806handle module packaging dynamically.
6807
6808To handle optional module packaging, you need to do two things:
6809
6810- Ensure the module packaging is actually done.
6811
6812- Ensure that any dependencies on optional modules from other recipes
6813 are satisfied by your recipe.
6814
6815Making Sure the Packaging is Done
6816~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6817
6818To ensure the module packaging actually gets done, you use the
6819``do_split_packages`` function within the ``populate_packages`` Python
6820function in your recipe. The ``do_split_packages`` function searches for
6821a pattern of files or directories under a specified path and creates a
6822package for each one it finds by appending to the
6823:term:`PACKAGES` variable and
6824setting the appropriate values for ``FILES:packagename``,
6825``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
6826Here is an example from the ``lighttpd`` recipe::
6827
6828 python populate_packages:prepend () {
6829 lighttpd_libdir = d.expand('${libdir}')
6830 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6831 'lighttpd-module-%s', 'Lighttpd module for %s',
6832 extra_depends='')
6833 }
6834
6835The previous example specifies a number of things in the call to
6836``do_split_packages``.
6837
6838- A directory within the files installed by your recipe through
6839 :ref:`ref-tasks-install` in which to search.
6840
6841- A regular expression used to match module files in that directory. In
6842 the example, note the parentheses () that mark the part of the
6843 expression from which the module name should be derived.
6844
6845- A pattern to use for the package names.
6846
6847- A description for each package.
6848
6849- An empty string for ``extra_depends``, which disables the default
6850 dependency on the main ``lighttpd`` package. Thus, if a file in
6851 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6852 ``lighttpd-module-alias`` is created for it and the
6853 :term:`DESCRIPTION` is set to
6854 "Lighttpd module for alias".
6855
6856Often, packaging modules is as simple as the previous example. However,
6857there are more advanced options that you can use within
6858``do_split_packages`` to modify its behavior. And, if you need to, you
6859can add more logic by specifying a hook function that is called for each
6860package. It is also perfectly acceptable to call ``do_split_packages``
6861multiple times if you have more than one set of modules to package.
6862
6863For more examples that show how to use ``do_split_packages``, see the
6864``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
6865directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
6866also find examples in ``meta/classes-recipe/kernel.bbclass``.
6867
6868Following is a reference that shows ``do_split_packages`` mandatory and
6869optional arguments::
6870
6871 Mandatory arguments
6872
6873 root
6874 The path in which to search
6875 file_regex
6876 Regular expression to match searched files.
6877 Use parentheses () to mark the part of this
6878 expression that should be used to derive the
6879 module name (to be substituted where %s is
6880 used in other function arguments as noted below)
6881 output_pattern
6882 Pattern to use for the package names. Must
6883 include %s.
6884 description
6885 Description to set for each package. Must
6886 include %s.
6887
6888 Optional arguments
6889
6890 postinst
6891 Postinstall script to use for all packages
6892 (as a string)
6893 recursive
6894 True to perform a recursive search --- default
6895 False
6896 hook
6897 A hook function to be called for every match.
6898 The function will be called with the following
6899 arguments (in the order listed):
6900
6901 f
6902 Full path to the file/directory match
6903 pkg
6904 The package name
6905 file_regex
6906 As above
6907 output_pattern
6908 As above
6909 modulename
6910 The module name derived using file_regex
6911 extra_depends
6912 Extra runtime dependencies (RDEPENDS) to be
6913 set for all packages. The default value of None
6914 causes a dependency on the main package
6915 (${PN}) --- if you do not want this, pass empty
6916 string '' for this parameter.
6917 aux_files_pattern
6918 Extra item(s) to be added to FILES for each
6919 package. Can be a single string item or a list
6920 of strings for multiple items. Must include %s.
6921 postrm
6922 postrm script to use for all packages (as a
6923 string)
6924 allow_dirs
6925 True to allow directories to be matched -
6926 default False
6927 prepend
6928 If True, prepend created packages to PACKAGES
6929 instead of the default False which appends them
6930 match_path
6931 match file_regex on the whole relative path to
6932 the root rather than just the filename
6933 aux_files_pattern_verbatim
6934 Extra item(s) to be added to FILES for each
6935 package, using the actual derived module name
6936 rather than converting it to something legal
6937 for a package name. Can be a single string item
6938 or a list of strings for multiple items. Must
6939 include %s.
6940 allow_links
6941 True to allow symlinks to be matched --- default
6942 False
6943 summary
6944 Summary to set for each package. Must include %s;
6945 defaults to description if not set.
6946
6947
6948
6949Satisfying Dependencies
6950~~~~~~~~~~~~~~~~~~~~~~~
6951
6952The second part for handling optional module packaging is to ensure that
6953any dependencies on optional modules from other recipes are satisfied by
6954your recipe. You can be sure these dependencies are satisfied by using
6955the :term:`PACKAGES_DYNAMIC`
6956variable. Here is an example that continues with the ``lighttpd`` recipe
6957shown earlier::
6958
6959 PACKAGES_DYNAMIC = "lighttpd-module-.*"
6960
6961The name
6962specified in the regular expression can of course be anything. In this
6963example, it is ``lighttpd-module-`` and is specified as the prefix to
6964ensure that any :term:`RDEPENDS` and
6965:term:`RRECOMMENDS` on a package
6966name starting with the prefix are satisfied during build time. If you
6967are using ``do_split_packages`` as described in the previous section,
6968the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
6969pattern specified in the call to ``do_split_packages``.
6970
6971Using Runtime Package Management
6972--------------------------------
6973
6974During a build, BitBake always transforms a recipe into one or more
6975packages. For example, BitBake takes the ``bash`` recipe and produces a
6976number of packages (e.g. ``bash``, ``bash-bashbug``,
6977``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
6978``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
6979generated packages are included in an image.
6980
6981In several situations, you might need to update, add, remove, or query
6982the packages on a target device at runtime (i.e. without having to
6983generate a new image). Examples of such situations include:
6984
6985- You want to provide in-the-field updates to deployed devices (e.g.
6986 security updates).
6987
6988- You want to have a fast turn-around development cycle for one or more
6989 applications that run on your device.
6990
6991- You want to temporarily install the "debug" packages of various
6992 applications on your device so that debugging can be greatly improved
6993 by allowing access to symbols and source debugging.
6994
6995- You want to deploy a more minimal package selection of your device
6996 but allow in-the-field updates to add a larger selection for
6997 customization.
6998
6999In all these situations, you have something similar to a more
7000traditional Linux distribution in that in-field devices are able to
7001receive pre-compiled packages from a server for installation or update.
7002Being able to install these packages on a running, in-field device is
7003what is termed "runtime package management".
7004
7005In order to use runtime package management, you need a host or server
7006machine that serves up the pre-compiled packages plus the required
7007metadata. You also need package manipulation tools on the target. The
7008build machine is a likely candidate to act as the server. However, that
7009machine does not necessarily have to be the package server. The build
7010machine could push its artifacts to another machine that acts as the
7011server (e.g. Internet-facing). In fact, doing so is advantageous for a
7012production environment as getting the packages away from the development
7013system's :term:`Build Directory` prevents accidental overwrites.
7014
7015A simple build that targets just one device produces more than one
7016package database. In other words, the packages produced by a build are
7017separated out into a couple of different package groupings based on
7018criteria such as the target's CPU architecture, the target board, or the
7019C library used on the target. For example, a build targeting the
7020``qemux86`` device produces the following three package databases:
7021``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
7022device to be aware of all the packages that were available to it, you
7023would need to point it to each of these databases individually. In a
7024similar way, a traditional Linux distribution usually is configured to
7025be aware of a number of software repositories from which it retrieves
7026packages.
7027
7028Using runtime package management is completely optional and not required
7029for a successful build or deployment in any way. But if you want to make
7030use of runtime package management, you need to do a couple things above
7031and beyond the basics. The remainder of this section describes what you
7032need to do.
7033
7034Build Considerations
7035~~~~~~~~~~~~~~~~~~~~
7036
7037This section describes build considerations of which you need to be
7038aware in order to provide support for runtime package management.
7039
7040When BitBake generates packages, it needs to know what format or formats
7041to use. In your configuration, you use the
7042:term:`PACKAGE_CLASSES`
7043variable to specify the format:
7044
70451. Open the ``local.conf`` file inside your :term:`Build Directory` (e.g.
7046 ``poky/build/conf/local.conf``).
7047
70482. Select the desired package format as follows::
7049
7050 PACKAGE_CLASSES ?= "package_packageformat"
7051
7052 where packageformat can be "ipk", "rpm",
7053 "deb", or "tar" which are the supported package formats.
7054
7055 .. note::
7056
7057 Because the Yocto Project supports four different package formats,
7058 you can set the variable with more than one argument. However, the
7059 OpenEmbedded build system only uses the first argument when
7060 creating an image or Software Development Kit (SDK).
7061
7062If you would like your image to start off with a basic package database
7063containing the packages in your current build as well as to have the
7064relevant tools available on the target for runtime package management,
7065you can include "package-management" in the
7066:term:`IMAGE_FEATURES`
7067variable. Including "package-management" in this configuration variable
7068ensures that when the image is assembled for your target, the image
7069includes the currently-known package databases as well as the
7070target-specific tools required for runtime package management to be
7071performed on the target. However, this is not strictly necessary. You
7072could start your image off without any databases but only include the
7073required on-target package tool(s). As an example, you could include
7074"opkg" in your
7075:term:`IMAGE_INSTALL` variable
7076if you are using the IPK package format. You can then initialize your
7077target's package database(s) later once your image is up and running.
7078
7079Whenever you perform any sort of build step that can potentially
7080generate a package or modify existing package, it is always a good idea
7081to re-generate the package index after the build by using the following
7082command::
7083
7084 $ bitbake package-index
7085
7086It might be tempting to build the
7087package and the package index at the same time with a command such as
7088the following::
7089
7090 $ bitbake some-package package-index
7091
7092Do not do this as
7093BitBake does not schedule the package index for after the completion of
7094the package you are building. Consequently, you cannot be sure of the
7095package index including information for the package you just built.
7096Thus, be sure to run the package update step separately after building
7097any packages.
7098
7099You can use the
7100:term:`PACKAGE_FEED_ARCHS`,
7101:term:`PACKAGE_FEED_BASE_PATHS`,
7102and
7103:term:`PACKAGE_FEED_URIS`
7104variables to pre-configure target images to use a package feed. If you
7105do not define these variables, then manual steps as described in the
7106subsequent sections are necessary to configure the target. You should
7107set these variables before building the image in order to produce a
7108correctly configured image.
7109
7110When your build is complete, your packages reside in the
7111``${TMPDIR}/deploy/packageformat`` directory. For example, if
7112``${``\ :term:`TMPDIR`\ ``}`` is
7113``tmp`` and your selected package type is RPM, then your RPM packages
7114are available in ``tmp/deploy/rpm``.
7115
7116Host or Server Machine Setup
7117~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7118
7119Although other protocols are possible, a server using HTTP typically
7120serves packages. If you want to use HTTP, then set up and configure a
7121web server such as Apache 2, lighttpd, or Python web server on the
7122machine serving the packages.
7123
7124To keep things simple, this section describes how to set up a
7125Python web server to share package feeds from the developer's
7126machine. Although this server might not be the best for a production
7127environment, the setup is simple and straight forward. Should you want
7128to use a different server more suited for production (e.g. Apache 2,
7129Lighttpd, or Nginx), take the appropriate steps to do so.
7130
7131From within the :term:`Build Directory` where you have built an image based on
7132your packaging choice (i.e. the :term:`PACKAGE_CLASSES` setting), simply start
7133the server. The following example assumes a :term:`Build Directory` of ``poky/build``
7134and a :term:`PACKAGE_CLASSES` setting of
7135":ref:`package_rpm <ref-classes-package_rpm>`"::
7136
7137 $ cd poky/build/tmp/deploy/rpm
7138 $ python3 -m http.server
7139
7140Target Setup
7141~~~~~~~~~~~~
7142
7143Setting up the target differs depending on the package management
7144system. This section provides information for RPM, IPK, and DEB.
7145
7146Using RPM
7147^^^^^^^^^
7148
7149The :wikipedia:`Dandified Packaging <DNF_(software)>` (DNF) performs
7150runtime package management of RPM packages. In order to use DNF for
7151runtime package management, you must perform an initial setup on the
7152target machine for cases where the ``PACKAGE_FEED_*`` variables were not
7153set as part of the image that is running on the target. This means if
7154you built your image and did not use these variables as part of the
7155build and your image is now running on the target, you need to perform
7156the steps in this section if you want to use runtime package management.
7157
7158.. note::
7159
7160 For information on the ``PACKAGE_FEED_*`` variables, see
7161 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
7162 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
7163 glossary.
7164
7165On the target, you must inform DNF that package databases are available.
7166You do this by creating a file named
7167``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
7168
7169As an example, assume the target is able to use the following package
7170databases: ``all``, ``i586``, and ``qemux86`` from a server named
7171``my.server``. The specifics for setting up the web server are up to
7172you. The critical requirement is that the URIs in the target repository
7173configuration point to the correct remote location for the feeds.
7174
7175.. note::
7176
7177 For development purposes, you can point the web server to the build
7178 system's ``deploy`` directory. However, for production use, it is better to
7179 copy the package directories to a location outside of the build area and use
7180 that location. Doing so avoids situations where the build system
7181 overwrites or changes the ``deploy`` directory.
7182
7183When telling DNF where to look for the package databases, you must
7184declare individual locations per architecture or a single location used
7185for all architectures. You cannot do both:
7186
7187- *Create an Explicit List of Architectures:* Define individual base
7188 URLs to identify where each package database is located:
7189
7190 .. code-block:: none
7191
7192 [oe-packages]
7193 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
7194
7195 This example
7196 informs DNF about individual package databases for all three
7197 architectures.
7198
7199- *Create a Single (Full) Package Index:* Define a single base URL that
7200 identifies where a full package database is located::
7201
7202 [oe-packages]
7203 baseurl=http://my.server/rpm
7204
7205 This example informs DNF about a single
7206 package database that contains all the package index information for
7207 all supported architectures.
7208
7209Once you have informed DNF where to find the package databases, you need
7210to fetch them:
7211
7212.. code-block:: none
7213
7214 # dnf makecache
7215
7216DNF is now able to find, install, and
7217upgrade packages from the specified repository or repositories.
7218
7219.. note::
7220
7221 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
7222 additional information.
7223
7224Using IPK
7225^^^^^^^^^
7226
7227The ``opkg`` application performs runtime package management of IPK
7228packages. You must perform an initial setup for ``opkg`` on the target
7229machine if the
7230:term:`PACKAGE_FEED_ARCHS`,
7231:term:`PACKAGE_FEED_BASE_PATHS`,
7232and
7233:term:`PACKAGE_FEED_URIS`
7234variables have not been set or the target image was built before the
7235variables were set.
7236
7237The ``opkg`` application uses configuration files to find available
7238package databases. Thus, you need to create a configuration file inside
7239the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
7240you want to use.
7241
7242As an example, suppose you are serving packages from a ``ipk/``
7243directory containing the ``i586``, ``all``, and ``qemux86`` databases
7244through an HTTP server named ``my.server``. On the target, create a
7245configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7246directory containing the following:
7247
7248.. code-block:: none
7249
7250 src/gz all http://my.server/ipk/all
7251 src/gz i586 http://my.server/ipk/i586
7252 src/gz qemux86 http://my.server/ipk/qemux86
7253
7254Next, instruct ``opkg`` to fetch the
7255repository information:
7256
7257.. code-block:: none
7258
7259 # opkg update
7260
7261The ``opkg`` application is now able to find, install, and upgrade packages
7262from the specified repository.
7263
7264Using DEB
7265^^^^^^^^^
7266
7267The ``apt`` application performs runtime package management of DEB
7268packages. This application uses a source list file to find available
7269package databases. You must perform an initial setup for ``apt`` on the
7270target machine if the
7271:term:`PACKAGE_FEED_ARCHS`,
7272:term:`PACKAGE_FEED_BASE_PATHS`,
7273and
7274:term:`PACKAGE_FEED_URIS`
7275variables have not been set or the target image was built before the
7276variables were set.
7277
7278To inform ``apt`` of the repository you want to use, you might create a
7279list file (e.g. ``my_repo.list``) inside the
7280``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7281serving packages from a ``deb/`` directory containing the ``i586``,
7282``all``, and ``qemux86`` databases through an HTTP server named
7283``my.server``. The list file should contain:
7284
7285.. code-block:: none
7286
7287 deb http://my.server/deb/all ./
7288 deb http://my.server/deb/i586 ./
7289 deb http://my.server/deb/qemux86 ./
7290
7291Next, instruct the ``apt`` application
7292to fetch the repository information:
7293
7294.. code-block:: none
7295
7296 $ sudo apt update
7297
7298After this step,
7299``apt`` is able to find, install, and upgrade packages from the
7300specified repository.
7301
7302Generating and Using Signed Packages
7303------------------------------------
7304
7305In order to add security to RPM packages used during a build, you can
7306take steps to securely sign them. Once a signature is verified, the
7307OpenEmbedded build system can use the package in the build. If security
7308fails for a signed package, the build system stops the build.
7309
7310This section describes how to sign RPM packages during a build and how
7311to use signed package feeds (repositories) when doing a build.
7312
7313Signing RPM Packages
7314~~~~~~~~~~~~~~~~~~~~
7315
7316To enable signing RPM packages, you must set up the following
7317configurations in either your ``local.config`` or ``distro.config``
7318file::
7319
7320 # Inherit sign_rpm.bbclass to enable signing functionality
7321 INHERIT += " sign_rpm"
7322 # Define the GPG key that will be used for signing.
7323 RPM_GPG_NAME = "key_name"
7324 # Provide passphrase for the key
7325 RPM_GPG_PASSPHRASE = "passphrase"
7326
7327.. note::
7328
7329 Be sure to supply appropriate values for both `key_name` and
7330 `passphrase`.
7331
7332Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
7333the previous example, two optional variables related to signing are available:
7334
7335- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7336 when the package is signed.
7337
7338- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7339 package is signed.
7340
7341Processing Package Feeds
7342~~~~~~~~~~~~~~~~~~~~~~~~
7343
7344In addition to being able to sign RPM packages, you can also enable
7345signed package feeds for IPK and RPM packages.
7346
7347The steps you need to take to enable signed package feed use are similar
7348to the steps used to sign RPM packages. You must define the following in
7349your ``local.config`` or ``distro.config`` file::
7350
7351 INHERIT += "sign_package_feed"
7352 PACKAGE_FEED_GPG_NAME = "key_name"
7353 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7354
7355For signed package feeds, the passphrase must be specified in a separate file,
7356which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7357variable. Regarding security, keeping a plain text passphrase out of the
7358configuration is more secure.
7359
7360Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7361``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
7362related to signed package feeds are available:
7363
7364- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7365 when the package is signed.
7366
7367- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7368 package is signed.
7369
7370- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7371 signature. This variable applies only to RPM and IPK package feeds.
7372 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7373 "ASC", which is the default and specifies ascii armored, and "BIN",
7374 which specifies binary.
7375
7376Testing Packages With ptest
7377---------------------------
7378
7379A Package Test (ptest) runs tests against packages built by the
7380OpenEmbedded build system on the target machine. A ptest contains at
7381least two items: the actual test, and a shell script (``run-ptest``)
7382that starts the test. The shell script that starts the test must not
7383contain the actual test --- the script only starts the test. On the other
7384hand, the test can be anything from a simple shell script that runs a
7385binary and checks the output to an elaborate system of test binaries and
7386data files.
7387
7388The test generates output in the format used by Automake::
7389
7390 result: testname
7391
7392where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7393the testname can be any identifying string.
7394
7395For a list of Yocto Project recipes that are already enabled with ptest,
7396see the :yocto_wiki:`Ptest </Ptest>` wiki page.
7397
7398.. note::
7399
7400 A recipe is "ptest-enabled" if it inherits the
7401 :ref:`ptest <ref-classes-ptest>` class.
7402
7403Adding ptest to Your Build
7404~~~~~~~~~~~~~~~~~~~~~~~~~~
7405
7406To add package testing to your build, add the :term:`DISTRO_FEATURES` and
7407:term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which
7408is found in the :term:`Build Directory`::
7409
7410 DISTRO_FEATURES:append = " ptest"
7411 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7412
7413Once your build is complete, the ptest files are installed into the
7414``/usr/lib/package/ptest`` directory within the image, where ``package``
7415is the name of the package.
7416
7417Running ptest
7418~~~~~~~~~~~~~
7419
7420The ``ptest-runner`` package installs a shell script that loops through
7421all installed ptest test suites and runs them in sequence. Consequently,
7422you might want to add this package to your image.
7423
7424Getting Your Package Ready
7425~~~~~~~~~~~~~~~~~~~~~~~~~~
7426
7427In order to enable a recipe to run installed ptests on target hardware,
7428you need to prepare the recipes that build the packages you want to
7429test. Here is what you have to do for each recipe:
7430
7431- *Be sure the recipe inherits the* :ref:`ptest <ref-classes-ptest>` *class:*
7432 Include the following line in each recipe::
7433
7434 inherit ptest
7435
7436- *Create run-ptest:* This script starts your test. Locate the
7437 script where you will refer to it using
7438 :term:`SRC_URI`. Here is an
7439 example that starts a test for ``dbus``::
7440
7441 #!/bin/sh
7442 cd test
7443 make -k runtest-TESTS
7444
7445- *Ensure dependencies are met:* If the test adds build or runtime
7446 dependencies that normally do not exist for the package (such as
7447 requiring "make" to run the test suite), use the
7448 :term:`DEPENDS` and
7449 :term:`RDEPENDS` variables in
7450 your recipe in order for the package to meet the dependencies. Here
7451 is an example where the package has a runtime dependency on "make"::
7452
7453 RDEPENDS:${PN}-ptest += "make"
7454
7455- *Add a function to build the test suite:* Not many packages support
7456 cross-compilation of their test suites. Consequently, you usually
7457 need to add a cross-compilation function to the package.
7458
7459 Many packages based on Automake compile and run the test suite by
7460 using a single command such as ``make check``. However, the host
7461 ``make check`` builds and runs on the same computer, while
7462 cross-compiling requires that the package is built on the host but
7463 executed for the target architecture (though often, as in the case
7464 for ptest, the execution occurs on the host). The built version of
7465 Automake that ships with the Yocto Project includes a patch that
7466 separates building and execution. Consequently, packages that use the
7467 unaltered, patched version of ``make check`` automatically
7468 cross-compiles.
7469
7470 Regardless, you still must add a ``do_compile_ptest`` function to
7471 build the test suite. Add a function similar to the following to your
7472 recipe::
7473
7474 do_compile_ptest() {
7475 oe_runmake buildtest-TESTS
7476 }
7477
7478- *Ensure special configurations are set:* If the package requires
7479 special configurations prior to compiling the test code, you must
7480 insert a ``do_configure_ptest`` function into the recipe.
7481
7482- *Install the test suite:* The :ref:`ptest <ref-classes-ptest>` class
7483 automatically copies the file ``run-ptest`` to the target and then runs make
7484 ``install-ptest`` to run the tests. If this is not enough, you need
7485 to create a ``do_install_ptest`` function and make sure it gets
7486 called after the "make install-ptest" completes.
7487
7488Creating Node Package Manager (NPM) Packages
7489--------------------------------------------
7490
7491:wikipedia:`NPM <Npm_(software)>` is a package
7492manager for the JavaScript programming language. The Yocto Project
7493supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
7494use this fetcher in combination with
7495:doc:`devtool </ref-manual/devtool-reference>` to create
7496recipes that produce NPM packages.
7497
7498There are two workflows that allow you to create NPM packages using
7499``devtool``: the NPM registry modules method and the NPM project code
7500method.
7501
7502.. note::
7503
7504 While it is possible to create NPM recipes manually, using
7505 ``devtool`` is far simpler.
7506
7507Additionally, some requirements and caveats exist.
7508
7509Requirements and Caveats
7510~~~~~~~~~~~~~~~~~~~~~~~~
7511
7512You need to be aware of the following before using ``devtool`` to create
7513NPM packages:
7514
7515- Of the two methods that you can use ``devtool`` to create NPM
7516 packages, the registry approach is slightly simpler. However, you
7517 might consider the project approach because you do not have to
7518 publish your module in the `NPM registry <https://docs.npmjs.com/misc/registry>`__,
7519 which is NPM's public registry.
7520
7521- Be familiar with
7522 :doc:`devtool </ref-manual/devtool-reference>`.
7523
7524- The NPM host tools need the native ``nodejs-npm`` package, which is
7525 part of the OpenEmbedded environment. You need to get the package by
7526 cloning the :oe_git:`meta-openembedded </meta-openembedded>`
7527 repository. Be sure to add the path to your local copy
7528 to your ``bblayers.conf`` file.
7529
7530- ``devtool`` cannot detect native libraries in module dependencies.
7531 Consequently, you must manually add packages to your recipe.
7532
7533- While deploying NPM packages, ``devtool`` cannot determine which
7534 dependent packages are missing on the target (e.g. the node runtime
7535 ``nodejs``). Consequently, you need to find out what files are
7536 missing and be sure they are on the target.
7537
7538- Although you might not need NPM to run your node package, it is
7539 useful to have NPM on your target. The NPM package name is
7540 ``nodejs-npm``.
7541
7542Using the Registry Modules Method
7543~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7544
7545This section presents an example that uses the ``cute-files`` module,
7546which is a file browser web application.
7547
7548.. note::
7549
7550 You must know the ``cute-files`` module version.
7551
7552The first thing you need to do is use ``devtool`` and the NPM fetcher to
7553create the recipe::
7554
7555 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7556
7557The
7558``devtool add`` command runs ``recipetool create`` and uses the same
7559fetch URI to download each dependency and capture license details where
7560possible. The result is a generated recipe.
7561
7562After running for quite a long time, in particular building the
7563``nodejs-native`` package, the command should end as follows::
7564
7565 INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional
7566
7567The recipe file is fairly simple and contains every license that
7568``recipetool`` finds and includes the licenses in the recipe's
7569:term:`LIC_FILES_CHKSUM`
7570variables. You need to examine the variables and look for those with
7571"unknown" in the :term:`LICENSE`
7572field. You need to track down the license information for "unknown"
7573modules and manually add the information to the recipe.
7574
7575``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7576files capture the version of all dependent modules. Many packages do not
7577provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it
7578runs.
7579
7580.. note::
7581
7582 A package is created for each sub-module. This policy is the only
7583 practical way to have the licenses for all of the dependencies
7584 represented in the license manifest of the image.
7585
7586The ``devtool edit-recipe`` command lets you take a look at the recipe::
7587
7588 $ devtool edit-recipe cute-files
7589 # Recipe created by recipetool
7590 # This is the basis of a recipe and may need further editing in order to be fully functional.
7591 # (Feel free to remove these comments when editing.)
7592
7593 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7594 # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
7595 # your responsibility to verify that the values are complete and correct.
7596 #
7597 # NOTE: multiple licenses have been detected; they have been separated with &
7598 # in the LICENSE value for now since it is a reasonable assumption that all
7599 # of the licenses apply. If instead there is a choice between the multiple
7600 # licenses then you should change the value to separate the licenses with |
7601 # instead of &. If there is any doubt, check the accompanying documentation
7602 # to determine which situation is applicable.
7603
7604 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7605 LICENSE = "BSD-3-Clause & ISC & MIT"
7606 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
7607 file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \
7608 file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \
7609 ...
7610 file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a"
7611
7612 SRC_URI = " \
7613 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7614 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7615 "
7616
7617 S = "${WORKDIR}/npm"
7618
7619 inherit npm
7620
7621 LICENSE:${PN} = "MIT"
7622 LICENSE:${PN}-accepts = "MIT"
7623 LICENSE:${PN}-array-flatten = "MIT"
7624 ...
7625 LICENSE:${PN}-vary = "MIT"
7626
7627Here are three key points in the previous example:
7628
7629- :term:`SRC_URI` uses the NPM
7630 scheme so that the NPM fetcher is used.
7631
7632- ``recipetool`` collects all the license information. If a
7633 sub-module's license is unavailable, the sub-module's name appears in
7634 the comments.
7635
7636- The ``inherit npm`` statement causes the
7637 :ref:`npm <ref-classes-npm>` class to package
7638 up all the modules.
7639
7640You can run the following command to build the ``cute-files`` package::
7641
7642 $ devtool build cute-files
7643
7644Remember that ``nodejs`` must be installed on
7645the target before your package.
7646
7647Assuming 192.168.7.2 for the target's IP address, use the following
7648command to deploy your package::
7649
7650 $ devtool deploy-target -s cute-files root@192.168.7.2
7651
7652Once the package is installed on the target, you can
7653test the application to show the contents of any directory::
7654
7655 $ cd /usr/lib/node_modules/cute-files
7656 $ cute-files
7657
7658On a browser,
7659go to ``http://192.168.7.2:3000`` and you see the following:
7660
7661.. image:: figures/cute-files-npm-example.png
7662 :width: 100%
7663
7664You can find the recipe in ``workspace/recipes/cute-files``. You can use
7665the recipe in any layer you choose.
7666
7667Using the NPM Projects Code Method
7668~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7669
7670Although it is useful to package modules already in the NPM registry,
7671adding ``node.js`` projects under development is a more common developer
7672use case.
7673
7674This section covers the NPM projects code method, which is very similar
7675to the "registry" approach described in the previous section. In the NPM
7676projects method, you provide ``devtool`` with an URL that points to the
7677source files.
7678
7679Replicating the same example, (i.e. ``cute-files``) use the following
7680command::
7681
7682 $ devtool add https://github.com/martinaglv/cute-files.git
7683
7684The recipe this command generates is very similar to the recipe created in
7685the previous section. However, the :term:`SRC_URI` looks like the following::
7686
7687 SRC_URI = " \
7688 git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \
7689 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7690 "
7691
7692In this example,
7693the main module is taken from the Git repository and dependencies are
7694taken from the NPM registry. Other than those differences, the recipe is
7695basically the same between the two methods. You can build and deploy the
7696package exactly as described in the previous section that uses the
7697registry modules method.
7698
7699Adding custom metadata to packages
7700----------------------------------
7701
7702The variable
7703:term:`PACKAGE_ADD_METADATA`
7704can be used to add additional metadata to packages. This is reflected in
7705the package control/spec file. To take the ipk format for example, the
7706CONTROL file stored inside would contain the additional metadata as
7707additional lines.
7708
7709The variable can be used in multiple ways, including using suffixes to
7710set it for a specific package type and/or package. Note that the order
7711of precedence is the same as this list:
7712
7713- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
7714
7715- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7716
7717- ``PACKAGE_ADD_METADATA:<PN>``
7718
7719- :term:`PACKAGE_ADD_METADATA`
7720
7721`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
7722package type:
7723
7724- IPK for .ipk packages
7725
7726- DEB for .deb packages
7727
7728- RPM for .rpm packages
7729
7730`<PN>` is a parameter and expected to be a package name.
7731
7732The variable can contain multiple [one-line] metadata fields separated
7733by the literal sequence '\\n'. The separator can be redefined using the
7734variable flag ``separator``.
7735
7736Here is an example that adds two custom fields for ipk
7737packages::
7738
7739 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
7740
7741Efficiently Fetching Source Files During a Build
7742================================================
7743
7744The OpenEmbedded build system works with source files located through
7745the :term:`SRC_URI` variable. When
7746you build something using BitBake, a big part of the operation is
7747locating and downloading all the source tarballs. For images,
7748downloading all the source for various packages can take a significant
7749amount of time.
7750
7751This section shows you how you can use mirrors to speed up fetching
7752source files and how you can pre-fetch files all of which leads to more
7753efficient use of resources and time.
7754
7755Setting up Effective Mirrors
7756----------------------------
7757
7758A good deal that goes into a Yocto Project build is simply downloading
7759all of the source tarballs. Maybe you have been working with another
7760build system for which you have built up a
7761sizable directory of source tarballs. Or, perhaps someone else has such
7762a directory for which you have read access. If so, you can save time by
7763adding statements to your configuration file so that the build process
7764checks local directories first for existing tarballs before checking the
7765Internet.
7766
7767Here is an efficient way to set it up in your ``local.conf`` file::
7768
7769 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7770 INHERIT += "own-mirrors"
7771 BB_GENERATE_MIRROR_TARBALLS = "1"
7772 # BB_NO_NETWORK = "1"
7773
7774In the previous example, the
7775:term:`BB_GENERATE_MIRROR_TARBALLS`
7776variable causes the OpenEmbedded build system to generate tarballs of
7777the Git repositories and store them in the
7778:term:`DL_DIR` directory. Due to
7779performance reasons, generating and storing these tarballs is not the
7780build system's default behavior.
7781
7782You can also use the
7783:term:`PREMIRRORS` variable. For
7784an example, see the variable's glossary entry in the Yocto Project
7785Reference Manual.
7786
7787Getting Source Files and Suppressing the Build
7788----------------------------------------------
7789
7790Another technique you can use to ready yourself for a successive string
7791of build operations, is to pre-fetch all the source files without
7792actually starting a build. This technique lets you work through any
7793download issues and ultimately gathers all the source files into your
7794download directory :ref:`structure-build-downloads`,
7795which is located with :term:`DL_DIR`.
7796
7797Use the following BitBake command form to fetch all the necessary
7798sources without starting the build::
7799
7800 $ bitbake target --runall=fetch
7801
7802This
7803variation of the BitBake command guarantees that you have all the
7804sources for that BitBake target should you disconnect from the Internet
7805and want to do the build later offline.
7806
7807Selecting an Initialization Manager
7808===================================
7809
7810By default, the Yocto Project uses SysVinit as the initialization
7811manager. However, there is also support for systemd, which is a full
7812replacement for init with parallel starting of services, reduced shell
7813overhead and other features that are used by many distributions.
7814
7815Within the system, SysVinit treats system components as services. These
7816services are maintained as shell scripts stored in the ``/etc/init.d/``
7817directory. Services organize into different run levels. This
7818organization is maintained by putting links to the services in the
7819``/etc/rcN.d/`` directories, where `N/` is one of the following options:
7820"S", "0", "1", "2", "3", "4", "5", or "6".
7821
7822.. note::
7823
7824 Each runlevel has a dependency on the previous runlevel. This
7825 dependency allows the services to work properly.
7826
7827In comparison, systemd treats components as units. Using units is a
7828broader concept as compared to using a service. A unit includes several
7829different types of entities. Service is one of the types of entities.
7830The runlevel concept in SysVinit corresponds to the concept of a target
7831in systemd, where target is also a type of supported unit.
7832
7833In a SysVinit-based system, services load sequentially (i.e. one by one)
7834during init and parallelization is not supported. With systemd, services
7835start in parallel. Needless to say, the method can have an impact on
7836system startup performance.
7837
7838If you want to use SysVinit, you do not have to do anything. But, if you
7839want to use systemd, you must take some steps as described in the
7840following sections.
7841
7842Using systemd Exclusively
7843-------------------------
7844
7845Set these variables in your distribution configuration file as follows::
7846
7847 DISTRO_FEATURES:append = " systemd"
7848 VIRTUAL-RUNTIME_init_manager = "systemd"
7849
7850You can also prevent the SysVinit distribution feature from
7851being automatically enabled as follows::
7852
7853 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7854
7855Doing so removes any
7856redundant SysVinit scripts.
7857
7858To remove initscripts from your image altogether, set this variable
7859also::
7860
7861 VIRTUAL-RUNTIME_initscripts = ""
7862
7863For information on the backfill variable, see
7864:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7865
7866Using systemd for the Main Image and Using SysVinit for the Rescue Image
7867------------------------------------------------------------------------
7868
7869Set these variables in your distribution configuration file as follows::
7870
7871 DISTRO_FEATURES:append = " systemd"
7872 VIRTUAL-RUNTIME_init_manager = "systemd"
7873
7874Doing so causes your main image to use the
7875``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7876image cannot use this package group. However, it can install SysVinit
7877and the appropriate packages will have support for both systemd and
7878SysVinit.
7879
7880Using systemd-journald without a traditional syslog daemon
7881----------------------------------------------------------
7882
7883Counter-intuitively, ``systemd-journald`` is not a syslog runtime or provider,
7884and the proper way to use systemd-journald as your sole logging mechanism is to
7885effectively disable syslog entirely by setting these variables in your distribution
7886configuration file::
7887
7888 VIRTUAL-RUNTIME_syslog = ""
7889 VIRTUAL-RUNTIME_base-utils-syslog = ""
7890
7891Doing so will prevent ``rsyslog`` / ``busybox-syslog`` from being pulled in by
7892default, leaving only ``journald``.
7893
7894Selecting a Device Manager
7895==========================
7896
7897The Yocto Project provides multiple ways to manage the device manager
7898(``/dev``):
7899
7900- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev``
7901 directory is persistent and the required device nodes are created
7902 during the build.
7903
7904- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7905 directory is provided by the kernel as an in-memory file system and
7906 is automatically populated by the kernel at runtime. Additional
7907 configuration of device nodes is done in user space by a device
7908 manager like ``udev`` or ``busybox-mdev``.
7909
7910Using Persistent and Pre-Populated ``/dev``
7911--------------------------------------------
7912
7913To use the static method for device population, you need to set the
7914:term:`USE_DEVFS` variable to "0"
7915as follows::
7916
7917 USE_DEVFS = "0"
7918
7919The content of the resulting ``/dev`` directory is defined in a Device
7920Table file. The
7921:term:`IMAGE_DEVICE_TABLES`
7922variable defines the Device Table to use and should be set in the
7923machine or distro configuration file. Alternatively, you can set this
7924variable in your ``local.conf`` configuration file.
7925
7926If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
7927``device_table-minimal.txt`` is used::
7928
7929 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7930
7931The population is handled by the ``makedevs`` utility during image
7932creation:
7933
7934Using ``devtmpfs`` and a Device Manager
7935---------------------------------------
7936
7937To use the dynamic method for device population, you need to use (or be
7938sure to set) the :term:`USE_DEVFS`
7939variable to "1", which is the default::
7940
7941 USE_DEVFS = "1"
7942
7943With this
7944setting, the resulting ``/dev`` directory is populated by the kernel
7945using ``devtmpfs``. Make sure the corresponding kernel configuration
7946variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
7947kernel.
7948
7949All devices created by ``devtmpfs`` will be owned by ``root`` and have
7950permissions ``0600``.
7951
7952To have more control over the device nodes, you can use a device manager
7953like ``udev`` or ``busybox-mdev``. You choose the device manager by
7954defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
7955distro configuration file. Alternatively, you can set this variable in
7956your ``local.conf`` configuration file::
7957
7958 VIRTUAL-RUNTIME_dev_manager = "udev"
7959
7960 # Some alternative values
7961 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
7962 # VIRTUAL-RUNTIME_dev_manager = "systemd"
7963
7964Using an External SCM
7965=====================
7966
7967If you're working on a recipe that pulls from an external Source Code
7968Manager (SCM), it is possible to have the OpenEmbedded build system
7969notice new recipe changes added to the SCM and then build the resulting
7970packages that depend on the new recipes by using the latest versions.
7971This only works for SCMs from which it is possible to get a sensible
7972revision number for changes. Currently, you can do this with Apache
7973Subversion (SVN), Git, and Bazaar (BZR) repositories.
7974
7975To enable this behavior, the :term:`PV` of
7976the recipe needs to reference
7977:term:`SRCPV`. Here is an example::
7978
7979 PV = "1.2.3+git${SRCPV}"
7980
7981Then, you can add the following to your
7982``local.conf``::
7983
7984 SRCREV:pn-PN = "${AUTOREV}"
7985
7986:term:`PN` is the name of the recipe for
7987which you want to enable automatic source revision updating.
7988
7989If you do not want to update your local configuration file, you can add
7990the following directly to the recipe to finish enabling the feature::
7991
7992 SRCREV = "${AUTOREV}"
7993
7994The Yocto Project provides a distribution named ``poky-bleeding``, whose
7995configuration file contains the line::
7996
7997 require conf/distro/include/poky-floating-revisions.inc
7998
7999This line pulls in the
8000listed include file that contains numerous lines of exactly that form::
8001
8002 #SRCREV:pn-opkg-native ?= "${AUTOREV}"
8003 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}"
8004 #SRCREV:pn-opkg ?= "${AUTOREV}"
8005 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}"
8006 #SRCREV:pn-opkg-utils ?= "${AUTOREV}"
8007 SRCREV:pn-gconf-dbus ?= "${AUTOREV}"
8008 SRCREV:pn-matchbox-common ?= "${AUTOREV}"
8009 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}"
8010 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}"
8011 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}"
8012 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}"
8013 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}"
8014 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}"
8015 SRCREV:pn-matchbox-wm ?= "${AUTOREV}"
8016 SRCREV:pn-settings-daemon ?= "${AUTOREV}"
8017 SRCREV:pn-screenshot ?= "${AUTOREV}"
8018 . . .
8019
8020These lines allow you to
8021experiment with building a distribution that tracks the latest
8022development source for numerous packages.
8023
8024.. note::
8025
8026 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
8027 this in mind if you use it.
8028
8029Creating a Read-Only Root Filesystem
8030====================================
8031
8032Suppose, for security reasons, you need to disable your target device's
8033root filesystem's write permissions (i.e. you need a read-only root
8034filesystem). Or, perhaps you are running the device's operating system
8035from a read-only storage device. For either case, you can customize your
8036image for that behavior.
8037
8038.. note::
8039
8040 Supporting a read-only root filesystem requires that the system and
8041 applications do not try to write to the root filesystem. You must
8042 configure all parts of the target system to write elsewhere, or to
8043 gracefully fail in the event of attempting to write to the root
8044 filesystem.
8045
8046Creating the Root Filesystem
8047----------------------------
8048
8049To create the read-only root filesystem, simply add the
8050"read-only-rootfs" feature to your image, normally in one of two ways.
8051The first way is to add the "read-only-rootfs" image feature in the
8052image's recipe file via the :term:`IMAGE_FEATURES` variable::
8053
8054 IMAGE_FEATURES += "read-only-rootfs"
8055
8056As an alternative, you can add the same feature
8057from within your :term:`Build Directory`'s ``local.conf`` file with the
8058associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
8059
8060 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
8061
8062For more information on how to use these variables, see the
8063":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
8064section. For information on the variables, see
8065:term:`IMAGE_FEATURES` and
8066:term:`EXTRA_IMAGE_FEATURES`.
8067
8068Post-Installation Scripts and Read-Only Root Filesystem
8069-------------------------------------------------------
8070
8071It is very important that you make sure all post-Installation
8072(``pkg_postinst``) scripts for packages that are installed into the
8073image can be run at the time when the root filesystem is created during
8074the build on the host system. These scripts cannot attempt to run during
8075the first boot on the target device. With the "read-only-rootfs" feature
8076enabled, the build system makes sure that all post-installation scripts
8077succeed at file system creation time. If any of these scripts
8078still need to be run after the root filesystem is created, the build
8079immediately fails. These build-time checks ensure that the build fails
8080rather than the target device fails later during its initial boot
8081operation.
8082
8083Most of the common post-installation scripts generated by the build
8084system for the out-of-the-box Yocto Project are engineered so that they
8085can run during root filesystem creation (e.g. post-installation scripts
8086for caching fonts). However, if you create and add custom scripts, you
8087need to be sure they can be run during this file system creation.
8088
8089Here are some common problems that prevent post-installation scripts
8090from running during root filesystem creation:
8091
8092- *Not using $D in front of absolute paths:* The build system defines
8093 ``$``\ :term:`D` when the root
8094 filesystem is created. Furthermore, ``$D`` is blank when the script
8095 is run on the target device. This implies two purposes for ``$D``:
8096 ensuring paths are valid in both the host and target environments,
8097 and checking to determine which environment is being used as a method
8098 for taking appropriate actions.
8099
8100- *Attempting to run processes that are specific to or dependent on the
8101 target architecture:* You can work around these attempts by using
8102 native tools, which run on the host system, to accomplish the same
8103 tasks, or by alternatively running the processes under QEMU, which
8104 has the ``qemu_run_binary`` function. For more information, see the
8105 :ref:`qemu <ref-classes-qemu>` class.
8106
8107Areas With Write Access
8108-----------------------
8109
8110With the "read-only-rootfs" feature enabled, any attempt by the target
8111to write to the root filesystem at runtime fails. Consequently, you must
8112make sure that you configure processes and applications that attempt
8113these types of writes do so to directories with write access (e.g.
8114``/tmp`` or ``/var/run``).
8115
8116Maintaining Build Output Quality
8117================================
8118
8119Many factors can influence the quality of a build. For example, if you
8120upgrade a recipe to use a new version of an upstream software package or
8121you experiment with some new configuration options, subtle changes can
8122occur that you might not detect until later. Consider the case where
8123your recipe is using a newer version of an upstream package. In this
8124case, a new version of a piece of software might introduce an optional
8125dependency on another library, which is auto-detected. If that library
8126has already been built when the software is building, the software will
8127link to the built library and that library will be pulled into your
8128image along with the new software even if you did not want the library.
8129
8130The :ref:`buildhistory <ref-classes-buildhistory>`
8131class helps you maintain the quality of your build output. You
8132can use the class to highlight unexpected and possibly unwanted changes
8133in the build output. When you enable build history, it records
8134information about the contents of each package and image and then
8135commits that information to a local Git repository where you can examine
8136the information.
8137
8138The remainder of this section describes the following:
8139
8140- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>`
8141
8142- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>`
8143
8144- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>`
8145
8146- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>`
8147
8148Enabling and Disabling Build History
8149------------------------------------
8150
8151Build history is disabled by default. To enable it, add the following
8152:term:`INHERIT` statement and set the :term:`BUILDHISTORY_COMMIT` variable to
8153"1" at the end of your ``conf/local.conf`` file found in the
8154:term:`Build Directory`::
8155
8156 INHERIT += "buildhistory"
8157 BUILDHISTORY_COMMIT = "1"
8158
8159Enabling build history as
8160previously described causes the OpenEmbedded build system to collect
8161build output information and commit it as a single commit to a local
8162:ref:`overview-manual/development-environment:git` repository.
8163
8164.. note::
8165
8166 Enabling build history increases your build times slightly,
8167 particularly for images, and increases the amount of disk space used
8168 during the build.
8169
8170You can disable build history by removing the previous statements from
8171your ``conf/local.conf`` file.
8172
8173Understanding What the Build History Contains
8174---------------------------------------------
8175
8176Build history information is kept in ``${``\ :term:`TOPDIR`\ ``}/buildhistory``
8177in the :term:`Build Directory` as defined by the :term:`BUILDHISTORY_DIR`
8178variable. Here is an example abbreviated listing:
8179
8180.. image:: figures/buildhistory.png
8181 :align: center
8182 :width: 50%
8183
8184At the top level, there is a ``metadata-revs`` file that lists the
8185revisions of the repositories for the enabled layers when the build was
8186produced. The rest of the data splits into separate ``packages``,
8187``images`` and ``sdk`` directories, the contents of which are described
8188as follows.
8189
8190Build History Package Information
8191~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8192
8193The history for each package contains a text file that has name-value
8194pairs with information about the package. For example,
8195``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
8196contains the following:
8197
8198.. code-block:: none
8199
8200 PV = 1.22.1
8201 PR = r32
8202 RPROVIDES =
8203 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
8204 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
8205 PKGSIZE = 540168
8206 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
8207 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
8208 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
8209 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
8210 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
8211 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
8212 /etc/busybox.links.nosuid /etc/busybox.links.suid
8213
8214Most of these
8215name-value pairs correspond to variables used to produce the package.
8216The exceptions are ``FILELIST``, which is the actual list of files in
8217the package, and ``PKGSIZE``, which is the total size of files in the
8218package in bytes.
8219
8220There is also a file that corresponds to the recipe from which the package
8221came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
8222
8223.. code-block:: none
8224
8225 PV = 1.22.1
8226 PR = r32
8227 DEPENDS = initscripts kern-tools-native update-rc.d-native \
8228 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
8229 virtual/libc virtual/update-alternatives
8230 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
8231 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
8232 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
8233
8234Finally, for those recipes fetched from a version control system (e.g.,
8235Git), there is a file that lists source revisions that are specified in
8236the recipe and the actual revisions used during the build. Listed
8237and actual revisions might differ when
8238:term:`SRCREV` is set to
8239${:term:`AUTOREV`}. Here is an
8240example assuming
8241``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
8242
8243 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8244 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8245 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
8246 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
8247
8248You can use the
8249``buildhistory-collect-srcrevs`` command with the ``-a`` option to
8250collect the stored :term:`SRCREV` values from build history and report them
8251in a format suitable for use in global configuration (e.g.,
8252``local.conf`` or a distro include file) to override floating
8253:term:`AUTOREV` values to a fixed set of revisions. Here is some example
8254output from this command::
8255
8256 $ buildhistory-collect-srcrevs -a
8257 # all-poky-linux
8258 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8"
8259 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac"
8260 # core2-64-poky-linux
8261 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66"
8262 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8"
8263 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0"
8264 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0"
8265 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1"
8266 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c"
8267 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7"
8268 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee"
8269 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1"
8270 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100"
8271 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3"
8272 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf"
8273 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced"
8274 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8"
8275 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f"
8276 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa"
8277 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70"
8278 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
8279 # qemux86_64-poky-linux
8280 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c"
8281 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0"
8282 # x86_64-linux
8283 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66"
8284 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c"
8285 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87"
8286 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57"
8287 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb"
8288 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b"
8289 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
8290
8291.. note::
8292
8293 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
8294
8295 - By default, only values where the :term:`SRCREV` was not hardcoded
8296 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
8297 option to see all :term:`SRCREV` values.
8298
8299 - The output statements might not have any effect if overrides are
8300 applied elsewhere in the build system configuration. Use the
8301 ``-f`` option to add the ``forcevariable`` override to each output
8302 line if you need to work around this restriction.
8303
8304 - The script does apply special handling when building for multiple
8305 machines. However, the script does place a comment before each set
8306 of values that specifies which triplet to which they belong as
8307 previously shown (e.g., ``i586-poky-linux``).
8308
8309Build History Image Information
8310~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8311
8312The files produced for each image are as follows:
8313
8314- ``image-files:`` A directory containing selected files from the root
8315 filesystem. The files are defined by
8316 :term:`BUILDHISTORY_IMAGE_FILES`.
8317
8318- ``build-id.txt:`` Human-readable information about the build
8319 configuration and metadata source revisions. This file contains the
8320 full build header as printed by BitBake.
8321
8322- ``*.dot:`` Dependency graphs for the image that are compatible with
8323 ``graphviz``.
8324
8325- ``files-in-image.txt:`` A list of files in the image with
8326 permissions, owner, group, size, and symlink information.
8327
8328- ``image-info.txt:`` A text file containing name-value pairs with
8329 information about the image. See the following listing example for
8330 more information.
8331
8332- ``installed-package-names.txt:`` A list of installed packages by name
8333 only.
8334
8335- ``installed-package-sizes.txt:`` A list of installed packages ordered
8336 by size.
8337
8338- ``installed-packages.txt:`` A list of installed packages with full
8339 package filenames.
8340
8341.. note::
8342
8343 Installed package information is able to be gathered and produced
8344 even if package management is disabled for the final image.
8345
8346Here is an example of ``image-info.txt``:
8347
8348.. code-block:: none
8349
8350 DISTRO = poky
8351 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be
8352 USER_CLASSES = buildstats
8353 IMAGE_CLASSES = qemuboot qemuboot license_image
8354 IMAGE_FEATURES = debug-tweaks
8355 IMAGE_LINGUAS =
8356 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp
8357 BAD_RECOMMENDATIONS =
8358 NO_RECOMMENDATIONS =
8359 PACKAGE_EXCLUDE =
8360 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible;
8361 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8362 IMAGESIZE = 9265
8363
8364Other than ``IMAGESIZE``,
8365which is the total size of the files in the image in Kbytes, the
8366name-value pairs are variables that may have influenced the content of
8367the image. This information is often useful when you are trying to
8368determine why a change in the package or file listings has occurred.
8369
8370Using Build History to Gather Image Information Only
8371~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8372
8373As you can see, build history produces image information, including
8374dependency graphs, so you can see why something was pulled into the
8375image. If you are just interested in this information and not interested
8376in collecting specific package or SDK information, you can enable
8377writing only image information without any history by adding the
8378following to your ``conf/local.conf`` file found in the
8379:term:`Build Directory`::
8380
8381 INHERIT += "buildhistory"
8382 BUILDHISTORY_COMMIT = "0"
8383 BUILDHISTORY_FEATURES = "image"
8384
8385Here, you set the
8386:term:`BUILDHISTORY_FEATURES`
8387variable to use the image feature only.
8388
8389Build History SDK Information
8390~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8391
8392Build history collects similar information on the contents of SDKs (e.g.
8393``bitbake -c populate_sdk imagename``) as compared to information it
8394collects for images. Furthermore, this information differs depending on
8395whether an extensible or standard SDK is being produced.
8396
8397The following list shows the files produced for SDKs:
8398
8399- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8400 owner, group, size, and symlink information. This list includes both
8401 the host and target parts of the SDK.
8402
8403- ``sdk-info.txt:`` A text file containing name-value pairs with
8404 information about the SDK. See the following listing example for more
8405 information.
8406
8407- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8408 with information about task group sizes (e.g. :ref:`ref-tasks-populate_sysroot`
8409 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8410 only when an extensible SDK is created.
8411
8412- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8413 with information for the shared-state packages and sizes in the SDK.
8414 The ``sstate-package-sizes.txt`` file exists only when an extensible
8415 SDK is created.
8416
8417- ``sdk-files:`` A folder that contains copies of the files mentioned
8418 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8419 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8420 specific to the extensible SDK although you can set it differently if
8421 you would like to pull in specific files from the standard SDK.
8422
8423 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8424 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8425 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8426 copied into the ``sdk-files`` directory.
8427
8428- The following information appears under each of the ``host`` and
8429 ``target`` directories for the portions of the SDK that run on the
8430 host and on the target, respectively:
8431
8432 .. note::
8433
8434 The following files for the most part are empty when producing an
8435 extensible SDK because this type of SDK is not constructed from
8436 packages as is the standard SDK.
8437
8438 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8439 with ``graphviz``.
8440
8441 - ``installed-package-names.txt:`` A list of installed packages by
8442 name only.
8443
8444 - ``installed-package-sizes.txt:`` A list of installed packages
8445 ordered by size.
8446
8447 - ``installed-packages.txt:`` A list of installed packages with full
8448 package filenames.
8449
8450Here is an example of ``sdk-info.txt``:
8451
8452.. code-block:: none
8453
8454 DISTRO = poky
8455 DISTRO_VERSION = 1.3+snapshot-20130327
8456 SDK_NAME = poky-glibc-i686-arm
8457 SDK_VERSION = 1.3+snapshot
8458 SDKMACHINE =
8459 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8460 BAD_RECOMMENDATIONS =
8461 SDKSIZE = 352712
8462
8463Other than ``SDKSIZE``, which is
8464the total size of the files in the SDK in Kbytes, the name-value pairs
8465are variables that might have influenced the content of the SDK. This
8466information is often useful when you are trying to determine why a
8467change in the package or file listings has occurred.
8468
8469Examining Build History Information
8470~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8471
8472You can examine build history output from the command line or from a web
8473interface.
8474
8475To see any changes that have occurred (assuming you have
8476:term:`BUILDHISTORY_COMMIT` = "1"),
8477you can simply use any Git command that allows you to view the history
8478of a repository. Here is one method::
8479
8480 $ git log -p
8481
8482You need to realize,
8483however, that this method does show changes that are not significant
8484(e.g. a package's size changing by a few bytes).
8485
8486There is a command-line tool called ``buildhistory-diff``, though,
8487that queries the Git repository and prints just the differences that
8488might be significant in human-readable form. Here is an example::
8489
8490 $ poky/poky/scripts/buildhistory-diff . HEAD^
8491 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8492 /etc/anotherpkg.conf was added
8493 /sbin/anotherpkg was added
8494 * (installed-package-names.txt):
8495 * anotherpkg was added
8496 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8497 anotherpkg was added
8498 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8499 * PR changed from "r0" to "r1"
8500 * PV changed from "0.1.10" to "0.1.12"
8501 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8502 * PR changed from "r0" to "r1"
8503 * PV changed from "0.1.10" to "0.1.12"
8504
8505.. note::
8506
8507 The ``buildhistory-diff`` tool requires the ``GitPython``
8508 package. Be sure to install it using Pip3 as follows::
8509
8510 $ pip3 install GitPython --user
8511
8512
8513 Alternatively, you can install ``python3-git`` using the appropriate
8514 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``).
8515
8516To see changes to the build history using a web interface, follow the
8517instruction in the ``README`` file
8518:yocto_git:`here </buildhistory-web/>`.
8519
8520Here is a sample screenshot of the interface:
8521
8522.. image:: figures/buildhistory-web.png
8523 :width: 100%
8524
8525Performing Automated Runtime Testing
8526====================================
8527
8528The OpenEmbedded build system makes available a series of automated
8529tests for images to verify runtime functionality. You can run these
8530tests on either QEMU or actual target hardware. Tests are written in
8531Python making use of the ``unittest`` module, and the majority of them
8532run commands on the target system over SSH. This section describes how
8533you set up the environment to use these tests, run available tests, and
8534write and add your own tests.
8535
8536For information on the test and QA infrastructure available within the
8537Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
8538section in the Yocto Project Reference Manual.
8539
8540Enabling Tests
8541--------------
8542
8543Depending on whether you are planning to run tests using QEMU or on the
8544hardware, you have to take different steps to enable the tests. See the
8545following subsections for information on how to enable both types of
8546tests.
8547
8548Enabling Runtime Tests on QEMU
8549~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8550
8551In order to run tests, you need to do the following:
8552
8553- *Set up to avoid interaction with sudo for networking:* To
8554 accomplish this, you must do one of the following:
8555
8556 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8557 commands or just for ``runqemu-ifup``. You must provide the full
8558 path as that can change if you are using multiple clones of the
8559 source repository.
8560
8561 .. note::
8562
8563 On some distributions, you also need to comment out "Defaults
8564 requiretty" in ``/etc/sudoers``.
8565
8566 - Manually configure a tap interface for your system.
8567
8568 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8569 should generate a list of tap devices. This is the option
8570 typically chosen for Autobuilder-type environments.
8571
8572 .. note::
8573
8574 - Be sure to use an absolute path when calling this script
8575 with sudo.
8576
8577 - The package recipe ``qemu-helper-native`` is required to run
8578 this script. Build the package using the following command::
8579
8580 $ bitbake qemu-helper-native
8581
8582- *Set the DISPLAY variable:* You need to set this variable so that
8583 you have an X server available (e.g. start ``vncserver`` for a
8584 headless machine).
8585
8586- *Be sure your host's firewall accepts incoming connections from
8587 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8588 HTTP server on a random high number port, which is used to serve
8589 files to the target. The DNF module serves
8590 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8591 That means your host's firewall must accept incoming connections from
8592 192.168.7.0/24, which is the default IP range used for tap devices by
8593 ``runqemu``.
8594
8595- *Be sure your host has the correct packages installed:* Depending
8596 your host's distribution, you need to have the following packages
8597 installed:
8598
8599 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8600
8601 - openSUSE: ``sysstat`` and ``iproute2``
8602
8603 - Fedora: ``sysstat`` and ``iproute``
8604
8605 - CentOS: ``sysstat`` and ``iproute``
8606
8607Once you start running the tests, the following happens:
8608
86091. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8610
86112. The image is booted under QEMU using the standard ``runqemu`` script.
8612
86133. A default timeout of 500 seconds occurs to allow for the boot process
8614 to reach the login prompt. You can change the timeout period by
8615 setting
8616 :term:`TEST_QEMUBOOT_TIMEOUT`
8617 in the ``local.conf`` file.
8618
86194. Once the boot process is reached and the login prompt appears, the
8620 tests run. The full boot log is written to
8621 ``${WORKDIR}/testimage/qemu_boot_log``.
8622
86235. Each test module loads in the order found in :term:`TEST_SUITES`. You can
8624 find the full output of the commands run over SSH in
8625 ``${WORKDIR}/testimgage/ssh_target_log``.
8626
86276. If no failures occur, the task running the tests ends successfully.
8628 You can find the output from the ``unittest`` in the task log at
8629 ``${WORKDIR}/temp/log.do_testimage``.
8630
8631Enabling Runtime Tests on Hardware
8632~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8633
8634The OpenEmbedded build system can run tests on real hardware, and for
8635certain devices it can also deploy the image to be tested onto the
8636device beforehand.
8637
8638For automated deployment, a "controller image" is installed onto the
8639hardware once as part of setup. Then, each time tests are to be run, the
8640following occurs:
8641
86421. The controller image is booted into and used to write the image to be
8643 tested to a second partition.
8644
86452. The device is then rebooted using an external script that you need to
8646 provide.
8647
86483. The device boots into the image to be tested.
8649
8650When running tests (independent of whether the image has been deployed
8651automatically or not), the device is expected to be connected to a
8652network on a pre-determined IP address. You can either use static IP
8653addresses written into the image, or set the image to use DHCP and have
8654your DHCP server on the test network assign a known IP address based on
8655the MAC address of the device.
8656
8657In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
8658appropriate value. For QEMU, you do not have to change anything, the
8659default value is "qemu". For running tests on hardware, the following
8660options are available:
8661
8662- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8663 on a target system that is already running the image to be tested and
8664 is available on the network. You can use "simpleremote" in
8665 conjunction with either real hardware or an image running within a
8666 separately started QEMU or any other virtual machine manager.
8667
8668- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8669 an EFI-based machine with ``systemd-boot`` as bootloader and
8670 ``core-image-testmaster`` (or something similar) is installed. Also,
8671 your hardware under test must be in a DHCP-enabled network that gives
8672 it the same IP address for each reboot.
8673
8674 If you choose "SystemdbootTarget", there are additional requirements
8675 and considerations. See the
8676 ":ref:`dev-manual/common-tasks:selecting systemdboottarget`" section, which
8677 follows, for more information.
8678
8679- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8680 images and running tests on the BeagleBone "Black" or original
8681 "White" hardware. For information on how to use these tests, see the
8682 comments at the top of the BeagleBoneTarget
8683 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8684
8685- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
8686 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8687 For information on how to use these tests, see the comments at the
8688 top of the EdgeRouterTarget
8689 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8690
8691- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
8692 tests on any generic PC that boots using GRUB. For information on how
8693 to use these tests, see the comments at the top of the GrubTarget
8694 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8695
8696- *"your-target":* Create your own custom target if you want to run
8697 tests when you are deploying images and running tests on a custom
8698 machine within your BSP layer. To do this, you need to add a Python
8699 unit that defines the target class under ``lib/oeqa/controllers/``
8700 within your layer. You must also provide an empty ``__init__.py``.
8701 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8702
8703Selecting SystemdbootTarget
8704~~~~~~~~~~~~~~~~~~~~~~~~~~~
8705
8706If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
8707not need any information in this section. You can skip down to the
8708":ref:`dev-manual/common-tasks:running tests`" section.
8709
8710If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
8711perform a one-time setup of your controller image by doing the following:
8712
87131. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
8714
8715 EFI_PROVIDER = "systemd-boot"
8716
87172. *Build the controller image:* Build the ``core-image-testmaster`` image.
8718 The ``core-image-testmaster`` recipe is provided as an example for a
8719 "controller" image and you can customize the image recipe as you would
8720 any other recipe.
8721
8722 Here are the image recipe requirements:
8723
8724 - Inherits ``core-image`` so that kernel modules are installed.
8725
8726 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
8727 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8728
8729 - Uses a custom :term:`Initramfs` image with a custom
8730 installer. A normal image that you can install usually creates a
8731 single root filesystem partition. This image uses another installer that
8732 creates a specific partition layout. Not all Board Support
8733 Packages (BSPs) can use an installer. For such cases, you need to
8734 manually create the following partition layout on the target:
8735
8736 - First partition mounted under ``/boot``, labeled "boot".
8737
8738 - The main root filesystem partition where this image gets installed,
8739 which is mounted under ``/``.
8740
8741 - Another partition labeled "testrootfs" where test images get
8742 deployed.
8743
87443. *Install image:* Install the image that you just built on the target
8745 system.
8746
8747The final thing you need to do when setting :term:`TEST_TARGET` to
8748"SystemdbootTarget" is to set up the test image:
8749
87501. *Set up your local.conf file:* Make sure you have the following
8751 statements in your ``local.conf`` file::
8752
8753 IMAGE_FSTYPES += "tar.gz"
8754 INHERIT += "testimage"
8755 TEST_TARGET = "SystemdbootTarget"
8756 TEST_TARGET_IP = "192.168.2.3"
8757
87582. *Build your test image:* Use BitBake to build the image::
8759
8760 $ bitbake core-image-sato
8761
8762Power Control
8763~~~~~~~~~~~~~
8764
8765For most hardware targets other than "simpleremote", you can control
8766power:
8767
8768- You can use :term:`TEST_POWERCONTROL_CMD` together with
8769 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
8770 and does power cycling. The test code passes one argument to that
8771 command: off, on or cycle (off then on). Here is an example that
8772 could appear in your ``local.conf`` file::
8773
8774 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8775
8776 In this example, the expect
8777 script does the following:
8778
8779 .. code-block:: shell
8780
8781 ssh test@10.11.12.1 "pyctl nuc1 arg"
8782
8783 It then runs a Python script that controls power for a label called
8784 ``nuc1``.
8785
8786 .. note::
8787
8788 You need to customize :term:`TEST_POWERCONTROL_CMD` and
8789 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
8790 is that it accepts "on", "off", and "cycle" as the last argument.
8791
8792- When no command is defined, it connects to the device over SSH and
8793 uses the classic reboot command to reboot the device. Classic reboot
8794 is fine as long as the machine actually reboots (i.e. the SSH test
8795 has not failed). It is useful for scenarios where you have a simple
8796 setup, typically with a single board, and where some manual
8797 interaction is okay from time to time.
8798
8799If you have no hardware to automatically perform power control but still
8800wish to experiment with automated hardware testing, you can use the
8801``dialog-power-control`` script that shows a dialog prompting you to perform
8802the required power action. This script requires either KDialog or Zenity
8803to be installed. To use this script, set the
8804:term:`TEST_POWERCONTROL_CMD`
8805variable as follows::
8806
8807 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8808
8809Serial Console Connection
8810~~~~~~~~~~~~~~~~~~~~~~~~~
8811
8812For test target classes requiring a serial console to interact with the
8813bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8814you need to specify a command to use to connect to the serial console of
8815the target machine by using the
8816:term:`TEST_SERIALCONTROL_CMD`
8817variable and optionally the
8818:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8819variable.
8820
8821These cases could be a serial terminal program if the machine is
8822connected to a local serial port, or a ``telnet`` or ``ssh`` command
8823connecting to a remote console server. Regardless of the case, the
8824command simply needs to connect to the serial console and forward that
8825connection to standard input and output as any normal terminal program
8826does. For example, to use the picocom terminal program on serial device
8827``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
8828
8829 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8830
8831For local
8832devices where the serial port device disappears when the device reboots,
8833an additional "serdevtry" wrapper script is provided. To use this
8834wrapper, simply prefix the terminal command with
8835``${COREBASE}/scripts/contrib/serdevtry``::
8836
8837 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8838
8839Running Tests
8840-------------
8841
8842You can start the tests automatically or manually:
8843
8844- *Automatically running tests:* To run the tests automatically after the
8845 OpenEmbedded build system successfully creates an image, first set the
8846 :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the
8847 :term:`Build Directory`::
8848
8849 TESTIMAGE_AUTO = "1"
8850
8851 Next, build your image. If the image successfully builds, the
8852 tests run::
8853
8854 bitbake core-image-sato
8855
8856- *Manually running tests:* To manually run the tests, first globally
8857 inherit the :ref:`testimage <ref-classes-testimage>` class
8858 by editing your ``local.conf`` file::
8859
8860 INHERIT += "testimage"
8861
8862 Next, use BitBake to run the tests::
8863
8864 bitbake -c testimage image
8865
8866All test files reside in ``meta/lib/oeqa/runtime`` in the
8867:term:`Source Directory`. A test name maps
8868directly to a Python module. Each test module may contain a number of
8869individual tests. Tests are usually grouped together by the area tested
8870(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
8871
8872You can add tests to any layer provided you place them in the proper
8873area and you extend :term:`BBPATH` in
8874the ``local.conf`` file as normal. Be sure that tests reside in
8875``layer/lib/oeqa/runtime``.
8876
8877.. note::
8878
8879 Be sure that module names do not collide with module names used in
8880 the default set of test modules in ``meta/lib/oeqa/runtime``.
8881
8882You can change the set of tests run by appending or overriding
8883:term:`TEST_SUITES` variable in
8884``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
8885for the image. Test modules named within :term:`TEST_SUITES` cannot be
8886skipped even if a test is not suitable for an image (e.g. running the
8887RPM tests on an image without ``rpm``). Appending "auto" to
8888:term:`TEST_SUITES` causes the build system to try to run all tests that are
8889suitable for the image (i.e. each test module may elect to skip itself).
8890
8891The order you list tests in :term:`TEST_SUITES` is important and influences
8892test dependencies. Consequently, tests that depend on other tests should
8893be added after the test on which they depend. For example, since the
8894``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8895"ping" in the list. The test class provides no re-ordering or dependency
8896handling.
8897
8898.. note::
8899
8900 Each module can have multiple classes with multiple test methods.
8901 And, Python ``unittest`` rules apply.
8902
8903Here are some things to keep in mind when running tests:
8904
8905- The default tests for the image are defined as::
8906
8907 DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
8908
8909- Add your own test to the list of the by using the following::
8910
8911 TEST_SUITES:append = " mytest"
8912
8913- Run a specific list of tests as follows::
8914
8915 TEST_SUITES = "test1 test2 test3"
8916
8917 Remember, order is important. Be sure to place a test that is
8918 dependent on another test later in the order.
8919
8920Exporting Tests
8921---------------
8922
8923You can export tests so that they can run independently of the build
8924system. Exporting tests is required if you want to be able to hand the
8925test execution off to a scheduler. You can only export tests that are
8926defined in :term:`TEST_SUITES`.
8927
8928If your image is already built, make sure the following are set in your
8929``local.conf`` file::
8930
8931 INHERIT += "testexport"
8932 TEST_TARGET_IP = "IP-address-for-the-test-target"
8933 TEST_SERVER_IP = "IP-address-for-the-test-server"
8934
8935You can then export the tests with the
8936following BitBake command form::
8937
8938 $ bitbake image -c testexport
8939
8940Exporting the tests places them in the :term:`Build Directory` in
8941``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
8942variable.
8943
8944You can now run the tests outside of the build environment::
8945
8946 $ cd tmp/testexport/image
8947 $ ./runexported.py testdata.json
8948
8949Here is a complete example that shows IP addresses and uses the
8950``core-image-sato`` image::
8951
8952 INHERIT += "testexport"
8953 TEST_TARGET_IP = "192.168.7.2"
8954 TEST_SERVER_IP = "192.168.7.1"
8955
8956Use BitBake to export the tests::
8957
8958 $ bitbake core-image-sato -c testexport
8959
8960Run the tests outside of
8961the build environment using the following::
8962
8963 $ cd tmp/testexport/core-image-sato
8964 $ ./runexported.py testdata.json
8965
8966Writing New Tests
8967-----------------
8968
8969As mentioned previously, all new test files need to be in the proper
8970place for the build system to find them. New tests for additional
8971functionality outside of the core should be added to the layer that adds
8972the functionality, in ``layer/lib/oeqa/runtime`` (as long as
8973:term:`BBPATH` is extended in the
8974layer's ``layer.conf`` file as normal). Just remember the following:
8975
8976- Filenames need to map directly to test (module) names.
8977
8978- Do not use module names that collide with existing core tests.
8979
8980- Minimally, an empty ``__init__.py`` file must be present in the runtime
8981 directory.
8982
8983To create a new test, start by copying an existing module (e.g.
8984``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
8985code from ``meta/lib/oeqa/utils``, which are helper classes.
8986
8987.. note::
8988
8989 Structure shell commands such that you rely on them and they return a
8990 single code for success. Be aware that sometimes you will need to
8991 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
8992
8993You will notice that all test classes inherit ``oeRuntimeTest``, which
8994is found in ``meta/lib/oetest.py``. This base class offers some helper
8995attributes, which are described in the following sections:
8996
8997Class Methods
8998~~~~~~~~~~~~~
8999
9000Class methods are as follows:
9001
9002- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
9003 package list of the image, which is based on the manifest file that
9004 is generated during the :ref:`ref-tasks-rootfs` task.
9005
9006- *hasFeature(feature):* Returns "True" if the feature is in
9007 :term:`IMAGE_FEATURES` or
9008 :term:`DISTRO_FEATURES`.
9009
9010Class Attributes
9011~~~~~~~~~~~~~~~~
9012
9013Class attributes are as follows:
9014
9015- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
9016 Otherwise, ``pscmd`` equals "ps" (busybox).
9017
9018- *tc:* The called test context, which gives access to the
9019 following attributes:
9020
9021 - *d:* The BitBake datastore, which allows you to use stuff such
9022 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
9023
9024 - *testslist and testsrequired:* Used internally. The tests
9025 do not need these.
9026
9027 - *filesdir:* The absolute path to
9028 ``meta/lib/oeqa/runtime/files``, which contains helper files for
9029 tests meant for copying on the target such as small files written
9030 in C for compilation.
9031
9032 - *target:* The target controller object used to deploy and
9033 start an image on a particular target (e.g. Qemu, SimpleRemote,
9034 and SystemdbootTarget). Tests usually use the following:
9035
9036 - *ip:* The target's IP address.
9037
9038 - *server_ip:* The host's IP address, which is usually used
9039 by the DNF test suite.
9040
9041 - *run(cmd, timeout=None):* The single, most used method.
9042 This command is a wrapper for: ``ssh root@host "cmd"``. The
9043 command returns a tuple: (status, output), which are what their
9044 names imply - the return code of "cmd" and whatever output it
9045 produces. The optional timeout argument represents the number
9046 of seconds the test should wait for "cmd" to return. If the
9047 argument is "None", the test uses the default instance's
9048 timeout period, which is 300 seconds. If the argument is "0",
9049 the test runs until the command returns.
9050
9051 - *copy_to(localpath, remotepath):*
9052 ``scp localpath root@ip:remotepath``.
9053
9054 - *copy_from(remotepath, localpath):*
9055 ``scp root@host:remotepath localpath``.
9056
9057Instance Attributes
9058~~~~~~~~~~~~~~~~~~~
9059
9060There is a single instance attribute, which is ``target``. The ``target``
9061instance attribute is identical to the class attribute of the same name,
9062which is described in the previous section. This attribute exists as
9063both an instance and class attribute so tests can use
9064``self.target.run(cmd)`` in instance methods instead of
9065``oeRuntimeTest.tc.target.run(cmd)``.
9066
9067Installing Packages in the DUT Without the Package Manager
9068----------------------------------------------------------
9069
9070When a test requires a package built by BitBake, it is possible to
9071install that package. Installing the package does not require a package
9072manager be installed in the device under test (DUT). It does, however,
9073require an SSH connection and the target must be using the
9074``sshcontrol`` class.
9075
9076.. note::
9077
9078 This method uses ``scp`` to copy files from the host to the target, which
9079 causes permissions and special attributes to be lost.
9080
9081A JSON file is used to define the packages needed by a test. This file
9082must be in the same path as the file used to define the tests.
9083Furthermore, the filename must map directly to the test module name with
9084a ``.json`` extension.
9085
9086The JSON file must include an object with the test name as keys of an
9087object or an array. This object (or array of objects) uses the following
9088data:
9089
9090- "pkg" --- a mandatory string that is the name of the package to be
9091 installed.
9092
9093- "rm" --- an optional boolean, which defaults to "false", that specifies
9094 to remove the package after the test.
9095
9096- "extract" --- an optional boolean, which defaults to "false", that
9097 specifies if the package must be extracted from the package format.
9098 When set to "true", the package is not automatically installed into
9099 the DUT.
9100
9101Following is an example JSON file that handles test "foo" installing
9102package "bar" and test "foobar" installing packages "foo" and "bar".
9103Once the test is complete, the packages are removed from the DUT.
9104::
9105
9106 {
9107 "foo": {
9108 "pkg": "bar"
9109 },
9110 "foobar": [
9111 {
9112 "pkg": "foo",
9113 "rm": true
9114 },
9115 {
9116 "pkg": "bar",
9117 "rm": true
9118 }
9119 ]
9120 }
9121
9122Debugging Tools and Techniques
9123==============================
9124
9125The exact method for debugging build failures depends on the nature of
9126the problem and on the system's area from which the bug originates.
9127Standard debugging practices such as comparison against the last known
9128working version with examination of the changes and the re-application
9129of steps to identify the one causing the problem are valid for the Yocto
9130Project just as they are for any other system. Even though it is
9131impossible to detail every possible potential failure, this section
9132provides some general tips to aid in debugging given a variety of
9133situations.
9134
9135.. note::
9136
9137 A useful feature for debugging is the error reporting tool.
9138 Configuring the Yocto Project to use this tool causes the
9139 OpenEmbedded build system to produce error reporting commands as part
9140 of the console output. You can enter the commands after the build
9141 completes to log error information into a common database, that can
9142 help you figure out what might be going wrong. For information on how
9143 to enable and use this feature, see the
9144 ":ref:`dev-manual/common-tasks:using the error reporting tool`"
9145 section.
9146
9147The following list shows the debugging topics in the remainder of this
9148section:
9149
9150- ":ref:`dev-manual/common-tasks:viewing logs from failed tasks`" describes
9151 how to find and view logs from tasks that failed during the build
9152 process.
9153
9154- ":ref:`dev-manual/common-tasks:viewing variable values`" describes how to
9155 use the BitBake ``-e`` option to examine variable values after a
9156 recipe has been parsed.
9157
9158- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
9159 describes how to use the ``oe-pkgdata-util`` utility to query
9160 :term:`PKGDATA_DIR` and
9161 display package-related information for built packages.
9162
9163- ":ref:`dev-manual/common-tasks:viewing dependencies between recipes and tasks`"
9164 describes how to use the BitBake ``-g`` option to display recipe
9165 dependency information used during the build.
9166
9167- ":ref:`dev-manual/common-tasks:viewing task variable dependencies`" describes
9168 how to use the ``bitbake-dumpsig`` command in conjunction with key
9169 subdirectories in the :term:`Build Directory` to determine variable
9170 dependencies.
9171
9172- ":ref:`dev-manual/common-tasks:running specific tasks`" describes
9173 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
9174 to run specific tasks in the build chain. It can be useful to run
9175 tasks "out-of-order" when trying isolate build issues.
9176
9177- ":ref:`dev-manual/common-tasks:general BitBake problems`" describes how
9178 to use BitBake's ``-D`` debug output option to reveal more about what
9179 BitBake is doing during the build.
9180
9181- ":ref:`dev-manual/common-tasks:building with no dependencies`"
9182 describes how to use the BitBake ``-b`` option to build a recipe
9183 while ignoring dependencies.
9184
9185- ":ref:`dev-manual/common-tasks:recipe logging mechanisms`"
9186 describes how to use the many recipe logging functions to produce
9187 debugging output and report errors and warnings.
9188
9189- ":ref:`dev-manual/common-tasks:debugging parallel make races`"
9190 describes how to debug situations where the build consists of several
9191 parts that are run simultaneously and when the output or result of
9192 one part is not ready for use with a different part of the build that
9193 depends on that output.
9194
9195- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) remotely`"
9196 describes how to use GDB to allow you to examine running programs, which can
9197 help you fix problems.
9198
9199- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) on the target`"
9200 describes how to use GDB directly on target hardware for debugging.
9201
9202- ":ref:`dev-manual/common-tasks:other debugging tips`" describes
9203 miscellaneous debugging tips that can be useful.
9204
9205Viewing Logs from Failed Tasks
9206------------------------------
9207
9208You can find the log for a task in the file
9209``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
9210For example, the log for the
9211:ref:`ref-tasks-compile` task of the
9212QEMU minimal image for the x86 machine (``qemux86``) might be in
9213``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
9214To see the commands :term:`BitBake` ran
9215to generate a log, look at the corresponding ``run.do_``\ `taskname` file
9216in the same directory.
9217
9218``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
9219links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
9220``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
9221when it ran. The symlinks always point to the files corresponding to the
9222most recent run.
9223
9224Viewing Variable Values
9225-----------------------
9226
9227Sometimes you need to know the value of a variable as a result of
9228BitBake's parsing step. This could be because some unexpected behavior
9229occurred in your project. Perhaps an attempt to :ref:`modify a variable
9230<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
9231variables>` did not work out as expected.
9232
9233BitBake's ``-e`` option is used to display variable values after
9234parsing. The following command displays the variable values after the
9235configuration files (i.e. ``local.conf``, ``bblayers.conf``,
9236``bitbake.conf`` and so forth) have been parsed::
9237
9238 $ bitbake -e
9239
9240The following command displays variable values after a specific recipe has
9241been parsed. The variables include those from the configuration as well::
9242
9243 $ bitbake -e recipename
9244
9245.. note::
9246
9247 Each recipe has its own private set of variables (datastore).
9248 Internally, after parsing the configuration, a copy of the resulting
9249 datastore is made prior to parsing each recipe. This copying implies
9250 that variables set in one recipe will not be visible to other
9251 recipes.
9252
9253 Likewise, each task within a recipe gets a private datastore based on
9254 the recipe datastore, which means that variables set within one task
9255 will not be visible to other tasks.
9256
9257In the output of ``bitbake -e``, each variable is preceded by a
9258description of how the variable got its value, including temporary
9259values that were later overridden. This description also includes
9260variable flags (varflags) set on the variable. The output can be very
9261helpful during debugging.
9262
9263Variables that are exported to the environment are preceded by
9264``export`` in the output of ``bitbake -e``. See the following example::
9265
9266 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9267
9268In addition to variable values, the output of the ``bitbake -e`` and
9269``bitbake -e`` recipe commands includes the following information:
9270
9271- The output starts with a tree listing all configuration files and
9272 classes included globally, recursively listing the files they include
9273 or inherit in turn. Much of the behavior of the OpenEmbedded build
9274 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
9275 implemented in the :ref:`base <ref-classes-base>` class and the
9276 classes it inherits, rather than being built into BitBake itself.
9277
9278- After the variable values, all functions appear in the output. For
9279 shell functions, variables referenced within the function body are
9280 expanded. If a function has been modified using overrides or using
9281 override-style operators like ``:append`` and ``:prepend``, then the
9282 final assembled function body appears in the output.
9283
9284Viewing Package Information with ``oe-pkgdata-util``
9285----------------------------------------------------
9286
9287You can use the ``oe-pkgdata-util`` command-line utility to query
9288:term:`PKGDATA_DIR` and display
9289various package-related information. When you use the utility, you must
9290use it to view information on packages that have already been built.
9291
9292Following are a few of the available ``oe-pkgdata-util`` subcommands.
9293
9294.. note::
9295
9296 You can use the standard \* and ? globbing wildcards as part of
9297 package names and paths.
9298
9299- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9300 that have been built, optionally limiting the match to packages that
9301 match pattern.
9302
9303- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9304 files and directories contained in the given packages.
9305
9306 .. note::
9307
9308 A different way to view the contents of a package is to look at
9309 the
9310 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9311 directory of the recipe that generates the package. This directory
9312 is created by the
9313 :ref:`ref-tasks-package` task
9314 and has one subdirectory for each package the recipe generates,
9315 which contains the files stored in that package.
9316
9317 If you want to inspect the ``${WORKDIR}/packages-split``
9318 directory, make sure that
9319 :ref:`rm_work <ref-classes-rm-work>` is not
9320 enabled when you build the recipe.
9321
9322- ``oe-pkgdata-util find-path path ...``: Lists the names of
9323 the packages that contain the given paths. For example, the following
9324 tells us that ``/usr/share/man/man1/make.1`` is contained in the
9325 ``make-doc`` package::
9326
9327 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
9328 make-doc: /usr/share/man/man1/make.1
9329
9330- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9331 of the recipes that produce the given packages.
9332
9333For more information on the ``oe-pkgdata-util`` command, use the help
9334facility::
9335
9336 $ oe-pkgdata-util --help
9337 $ oe-pkgdata-util subcommand --help
9338
9339Viewing Dependencies Between Recipes and Tasks
9340----------------------------------------------
9341
9342Sometimes it can be hard to see why BitBake wants to build other recipes
9343before the one you have specified. Dependency information can help you
9344understand why a recipe is built.
9345
9346To generate dependency information for a recipe, run the following
9347command::
9348
9349 $ bitbake -g recipename
9350
9351This command writes the following files in the current directory:
9352
9353- ``pn-buildlist``: A list of recipes/targets involved in building
9354 `recipename`. "Involved" here means that at least one task from the
9355 recipe needs to run when building `recipename` from scratch. Targets
9356 that are in
9357 :term:`ASSUME_PROVIDED`
9358 are not listed.
9359
9360- ``task-depends.dot``: A graph showing dependencies between tasks.
9361
9362The graphs are in :wikipedia:`DOT <DOT_%28graph_description_language%29>`
9363format and can be converted to images (e.g. using the ``dot`` tool from
9364`Graphviz <https://www.graphviz.org/>`__).
9365
9366.. note::
9367
9368 - DOT files use a plain text format. The graphs generated using the
9369 ``bitbake -g`` command are often so large as to be difficult to
9370 read without special pruning (e.g. with BitBake's ``-I`` option)
9371 and processing. Despite the form and size of the graphs, the
9372 corresponding ``.dot`` files can still be possible to read and
9373 provide useful information.
9374
9375 As an example, the ``task-depends.dot`` file contains lines such
9376 as the following::
9377
9378 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9379
9380 The above example line reveals that the
9381 :ref:`ref-tasks-configure`
9382 task in ``libxslt`` depends on the
9383 :ref:`ref-tasks-populate_sysroot`
9384 task in ``libxml2``, which is a normal
9385 :term:`DEPENDS` dependency
9386 between the two recipes.
9387
9388 - For an example of how ``.dot`` files can be processed, see the
9389 ``scripts/contrib/graph-tool`` Python script, which finds and
9390 displays paths between graph nodes.
9391
9392You can use a different method to view dependency information by using
9393the following command::
9394
9395 $ bitbake -g -u taskexp recipename
9396
9397This command
9398displays a GUI window from which you can view build-time and runtime
9399dependencies for the recipes involved in building recipename.
9400
9401Viewing Task Variable Dependencies
9402----------------------------------
9403
9404As mentioned in the
9405":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9406User Manual, BitBake tries to automatically determine what variables a
9407task depends on so that it can rerun the task if any values of the
9408variables change. This determination is usually reliable. However, if
9409you do things like construct variable names at runtime, then you might
9410have to manually declare dependencies on those variables using
9411``vardeps`` as described in the
9412":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9413User Manual.
9414
9415If you are unsure whether a variable dependency is being picked up
9416automatically for a given task, you can list the variable dependencies
9417BitBake has determined by doing the following:
9418
94191. Build the recipe containing the task::
9420
9421 $ bitbake recipename
9422
94232. Inside the :term:`STAMPS_DIR`
9424 directory, find the signature data (``sigdata``) file that
9425 corresponds to the task. The ``sigdata`` files contain a pickled
9426 Python database of all the metadata that went into creating the input
9427 checksum for the task. As an example, for the
9428 :ref:`ref-tasks-fetch` task of the
9429 ``db`` recipe, the ``sigdata`` file might be found in the following
9430 location::
9431
9432 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9433
9434 For tasks that are accelerated through the shared state
9435 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
9436 additional ``siginfo`` file is written into
9437 :term:`SSTATE_DIR` along with
9438 the cached task output. The ``siginfo`` files contain exactly the
9439 same information as ``sigdata`` files.
9440
94413. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
9442 is an example::
9443
9444 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9445
9446 In the output of the above command, you will find a line like the
9447 following, which lists all the (inferred) variable dependencies for
9448 the task. This list also includes indirect dependencies from
9449 variables depending on other variables, recursively.
9450 ::
9451
9452 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9453
9454 .. note::
9455
9456 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9457 These functions in turn depend on the variables they reference.
9458
9459 The output of ``bitbake-dumpsig`` also includes the value each
9460 variable had, a list of dependencies for each variable, and
9461 :term:`BB_BASEHASH_IGNORE_VARS`
9462 information.
9463
9464There is also a ``bitbake-diffsigs`` command for comparing two
9465``siginfo`` or ``sigdata`` files. This command can be helpful when
9466trying to figure out what changed between two versions of a task. If you
9467call ``bitbake-diffsigs`` with just one file, the command behaves like
9468``bitbake-dumpsig``.
9469
9470You can also use BitBake to dump out the signature construction
9471information without executing tasks by using either of the following
9472BitBake command-line options::
9473
9474 ‐‐dump-signatures=SIGNATURE_HANDLER
9475 -S SIGNATURE_HANDLER
9476
9477
9478.. note::
9479
9480 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9481 dump only the signature or compare the dumped signature with the cached one,
9482 respectively.
9483
9484Using BitBake with either of these options causes BitBake to dump out
9485``sigdata`` files in the ``stamps`` directory for every task it would
9486have executed instead of building the specified target package.
9487
9488Viewing Metadata Used to Create the Input Signature of a Shared State Task
9489--------------------------------------------------------------------------
9490
9491Seeing what metadata went into creating the input signature of a shared
9492state (sstate) task can be a useful debugging aid. This information is
9493available in signature information (``siginfo``) files in
9494:term:`SSTATE_DIR`. For
9495information on how to view and interpret information in ``siginfo``
9496files, see the
9497":ref:`dev-manual/common-tasks:viewing task variable dependencies`" section.
9498
9499For conceptual information on shared state, see the
9500":ref:`overview-manual/concepts:shared state`"
9501section in the Yocto Project Overview and Concepts Manual.
9502
9503Invalidating Shared State to Force a Task to Run
9504------------------------------------------------
9505
9506The OpenEmbedded build system uses
9507:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
9508:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
9509rebuilding tasks. Collectively, this scheme is known as "shared state
9510code".
9511
9512As with all schemes, this one has some drawbacks. It is possible that
9513you could make implicit changes to your code that the checksum
9514calculations do not take into account. These implicit changes affect a
9515task's output but do not trigger the shared state code into rebuilding a
9516recipe. Consider an example during which a tool changes its output.
9517Assume that the output of ``rpmdeps`` changes. The result of the change
9518should be that all the ``package`` and ``package_write_rpm`` shared
9519state cache items become invalid. However, because the change to the
9520output is external to the code and therefore implicit, the associated
9521shared state cache items do not become invalidated. In this case, the
9522build process uses the cached items rather than running the task again.
9523Obviously, these types of implicit changes can cause problems.
9524
9525To avoid these problems during the build, you need to understand the
9526effects of any changes you make. Realize that changes you make directly
9527to a function are automatically factored into the checksum calculation.
9528Thus, these explicit changes invalidate the associated area of shared
9529state cache. However, you need to be aware of any implicit changes that
9530are not obvious changes to the code and could affect the output of a
9531given task.
9532
9533When you identify an implicit change, you can easily take steps to
9534invalidate the cache and force the tasks to run. The steps you can take
9535are as simple as changing a function's comments in the source code. For
9536example, to invalidate package shared state files, change the comment
9537statements of
9538:ref:`ref-tasks-package` or the
9539comments of one of the functions it calls. Even though the change is
9540purely cosmetic, it causes the checksum to be recalculated and forces
9541the build system to run the task again.
9542
9543.. note::
9544
9545 For an example of a commit that makes a cosmetic change to invalidate
9546 shared state, see this
9547 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
9548
9549Running Specific Tasks
9550----------------------
9551
9552Any given recipe consists of a set of tasks. The standard BitBake
9553behavior in most cases is: :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`,
9554:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, :ref:`ref-tasks-install`, :ref:`ref-tasks-package`,
9555:ref:`do_package_write_* <ref-tasks-package_write_deb>`, and :ref:`ref-tasks-build`. The default task is
9556:ref:`ref-tasks-build` and any tasks on which it depends build first. Some tasks,
9557such as :ref:`ref-tasks-devshell`, are not part of the default build chain. If you
9558wish to run a task that is not part of the default build chain, you can
9559use the ``-c`` option in BitBake. Here is an example::
9560
9561 $ bitbake matchbox-desktop -c devshell
9562
9563The ``-c`` option respects task dependencies, which means that all other
9564tasks (including tasks from other recipes) that the specified task
9565depends on will be run before the task. Even when you manually specify a
9566task to run with ``-c``, BitBake will only run the task if it considers
9567it "out of date". See the
9568":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
9569section in the Yocto Project Overview and Concepts Manual for how
9570BitBake determines whether a task is "out of date".
9571
9572If you want to force an up-to-date task to be rerun (e.g. because you
9573made manual modifications to the recipe's
9574:term:`WORKDIR` that you want to try
9575out), then you can use the ``-f`` option.
9576
9577.. note::
9578
9579 The reason ``-f`` is never required when running the
9580 :ref:`ref-tasks-devshell` task is because the
9581 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
9582 variable flag is already set for the task.
9583
9584The following example shows one way you can use the ``-f`` option::
9585
9586 $ bitbake matchbox-desktop
9587 .
9588 .
9589 make some changes to the source code in the work directory
9590 .
9591 .
9592 $ bitbake matchbox-desktop -c compile -f
9593 $ bitbake matchbox-desktop
9594
9595This sequence first builds and then recompiles ``matchbox-desktop``. The
9596last command reruns all tasks (basically the packaging tasks) after the
9597compile. BitBake recognizes that the :ref:`ref-tasks-compile` task was rerun and
9598therefore understands that the other tasks also need to be run again.
9599
9600Another, shorter way to rerun a task and all
9601:ref:`ref-manual/tasks:normal recipe build tasks`
9602that depend on it is to use the ``-C`` option.
9603
9604.. note::
9605
9606 This option is upper-cased and is separate from the ``-c``
9607 option, which is lower-cased.
9608
9609Using this option invalidates the given task and then runs the
9610:ref:`ref-tasks-build` task, which is
9611the default task if no task is given, and the tasks on which it depends.
9612You could replace the final two commands in the previous example with
9613the following single command::
9614
9615 $ bitbake matchbox-desktop -C compile
9616
9617Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9618the input checksum of the specified task. This tainting indirectly
9619causes the task and its dependent tasks to be rerun through the normal
9620task dependency mechanisms.
9621
9622.. note::
9623
9624 BitBake explicitly keeps track of which tasks have been tainted in
9625 this fashion, and will print warnings such as the following for
9626 builds involving such tasks:
9627
9628 .. code-block:: none
9629
9630 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9631
9632
9633 The purpose of the warning is to let you know that the work directory
9634 and build output might not be in the clean state they would be in for
9635 a "normal" build, depending on what actions you took. To get rid of
9636 such warnings, you can remove the work directory and rebuild the
9637 recipe, as follows::
9638
9639 $ bitbake matchbox-desktop -c clean
9640 $ bitbake matchbox-desktop
9641
9642
9643You can view a list of tasks in a given package by running the
9644:ref:`ref-tasks-listtasks` task as follows::
9645
9646 $ bitbake matchbox-desktop -c listtasks
9647
9648The results appear as output to the console and are also in
9649the file ``${WORKDIR}/temp/log.do_listtasks``.
9650
9651General BitBake Problems
9652------------------------
9653
9654You can see debug output from BitBake by using the ``-D`` option. The
9655debug output gives more information about what BitBake is doing and the
9656reason behind it. Each ``-D`` option you use increases the logging
9657level. The most common usage is ``-DDD``.
9658
9659The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
9660chose a certain version of a package or why BitBake picked a certain
9661provider. This command could also help you in a situation where you
9662think BitBake did something unexpected.
9663
9664Building with No Dependencies
9665-----------------------------
9666
9667To build a specific recipe (``.bb`` file), you can use the following
9668command form::
9669
9670 $ bitbake -b somepath/somerecipe.bb
9671
9672This command form does
9673not check for dependencies. Consequently, you should use it only when
9674you know existing dependencies have been met.
9675
9676.. note::
9677
9678 You can also specify fragments of the filename. In this case, BitBake
9679 checks for a unique match.
9680
9681Recipe Logging Mechanisms
9682-------------------------
9683
9684The Yocto Project provides several logging functions for producing
9685debugging output and reporting errors and warnings. For Python
9686functions, the following logging functions are available. All of these functions
9687log to ``${T}/log.do_``\ `task`, and can also log to standard output
9688(stdout) with the right settings:
9689
9690- ``bb.plain(msg)``: Writes msg as is to the log while also
9691 logging to stdout.
9692
9693- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9694 stdout if BitBake is called with "-v".
9695
9696- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9697 log. Also logs to stdout if the log level is greater than or equal to
9698 level. See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`" option
9699 in the BitBake User Manual for more information.
9700
9701- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9702 logging to stdout.
9703
9704- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9705 logging to standard out (stdout).
9706
9707 .. note::
9708
9709 Calling this function does not cause the task to fail.
9710
9711- ``bb.fatal(msg)``: This logging function is similar to
9712 ``bb.error(msg)`` but also causes the calling task to fail.
9713
9714 .. note::
9715
9716 ``bb.fatal()`` raises an exception, which means you do not need to put a
9717 "return" statement after the function.
9718
9719The same logging functions are also available in shell functions, under
9720the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9721and ``bbfatal``. The
9722:ref:`logging <ref-classes-logging>` class
9723implements these functions. See that class in the ``meta/classes``
9724folder of the :term:`Source Directory` for information.
9725
9726Logging With Python
9727~~~~~~~~~~~~~~~~~~~
9728
9729When creating recipes using Python and inserting code that handles build
9730logs, keep in mind the goal is to have informative logs while keeping
9731the console as "silent" as possible. Also, if you want status messages
9732in the log, use the "debug" loglevel.
9733
9734Following is an example written in Python. The code handles logging for
9735a function that determines the number of tasks needed to be run. See the
9736":ref:`ref-tasks-listtasks`"
9737section for additional information::
9738
9739 python do_listtasks() {
9740 bb.debug(2, "Starting to figure out the task list")
9741 if noteworthy_condition:
9742 bb.note("There are 47 tasks to run")
9743 bb.debug(2, "Got to point xyz")
9744 if warning_trigger:
9745 bb.warn("Detected warning_trigger, this might be a problem later.")
9746 if recoverable_error:
9747 bb.error("Hit recoverable_error, you really need to fix this!")
9748 if fatal_error:
9749 bb.fatal("fatal_error detected, unable to print the task list")
9750 bb.plain("The tasks present are abc")
9751 bb.debug(2, "Finished figuring out the tasklist")
9752 }
9753
9754Logging With Bash
9755~~~~~~~~~~~~~~~~~
9756
9757When creating recipes using Bash and inserting code that handles build
9758logs, you have the same goals --- informative with minimal console output.
9759The syntax you use for recipes written in Bash is similar to that of
9760recipes written in Python described in the previous section.
9761
9762Following is an example written in Bash. The code logs the progress of
9763the ``do_my_function`` function.
9764::
9765
9766 do_my_function() {
9767 bbdebug 2 "Running do_my_function"
9768 if [ exceptional_condition ]; then
9769 bbnote "Hit exceptional_condition"
9770 fi
9771 bbdebug 2 "Got to point xyz"
9772 if [ warning_trigger ]; then
9773 bbwarn "Detected warning_trigger, this might cause a problem later."
9774 fi
9775 if [ recoverable_error ]; then
9776 bberror "Hit recoverable_error, correcting"
9777 fi
9778 if [ fatal_error ]; then
9779 bbfatal "fatal_error detected"
9780 fi
9781 bbdebug 2 "Completed do_my_function"
9782 }
9783
9784
9785Debugging Parallel Make Races
9786-----------------------------
9787
9788A parallel ``make`` race occurs when the build consists of several parts
9789that are run simultaneously and a situation occurs when the output or
9790result of one part is not ready for use with a different part of the
9791build that depends on that output. Parallel make races are annoying and
9792can sometimes be difficult to reproduce and fix. However, there are some simple
9793tips and tricks that can help you debug and fix them. This section
9794presents a real-world example of an error encountered on the Yocto
9795Project autobuilder and the process used to fix it.
9796
9797.. note::
9798
9799 If you cannot properly fix a ``make`` race condition, you can work around it
9800 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
9801 variables.
9802
9803The Failure
9804~~~~~~~~~~~
9805
9806For this example, assume that you are building an image that depends on
9807the "neard" package. And, during the build, BitBake runs into problems
9808and creates the following output.
9809
9810.. note::
9811
9812 This example log file has longer lines artificially broken to make
9813 the listing easier to read.
9814
9815If you examine the output or the log file, you see the failure during
9816``make``:
9817
9818.. code-block:: none
9819
9820 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9821 | DEBUG: Executing shell function do_compile
9822 | NOTE: make -j 16
9823 | make --no-print-directory all-am
9824 | /bin/mkdir -p include/near
9825 | /bin/mkdir -p include/near
9826 | /bin/mkdir -p include/near
9827 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9828 0.14-r0/neard-0.14/include/types.h include/near/types.h
9829 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9830 0.14-r0/neard-0.14/include/log.h include/near/log.h
9831 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9832 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9833 | /bin/mkdir -p include/near
9834 | /bin/mkdir -p include/near
9835 | /bin/mkdir -p include/near
9836 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9837 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
9838 | /bin/mkdir -p include/near
9839 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9840 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
9841 | /bin/mkdir -p include/near
9842 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9843 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
9844 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9845 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
9846 | /bin/mkdir -p include/near
9847 | /bin/mkdir -p include/near
9848 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9849 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
9850 | /bin/mkdir -p include/near
9851 | /bin/mkdir -p include/near
9852 | /bin/mkdir -p include/near
9853 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9854 0.14-r0/neard-0.14/include/device.h include/near/device.h
9855 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9856 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
9857 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9858 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
9859 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9860 0.14-r0/neard-0.14/include/version.h include/near/version.h
9861 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9862 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
9863 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
9864 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/
9865 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
9866 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
9867 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
9868 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/
9869 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/
9870 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
9871 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
9872 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
9873 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
9874 -o tools/snep-send.o tools/snep-send.c
9875 | In file included from tools/snep-send.c:16:0:
9876 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9877 | #include <near/dbus.h>
9878 | ^
9879 | compilation terminated.
9880 | make[1]: *** [tools/snep-send.o] Error 1
9881 | make[1]: *** Waiting for unfinished jobs....
9882 | make: *** [all] Error 2
9883 | ERROR: oe_runmake failed
9884
9885Reproducing the Error
9886~~~~~~~~~~~~~~~~~~~~~
9887
9888Because race conditions are intermittent, they do not manifest
9889themselves every time you do the build. In fact, most times the build
9890will complete without problems even though the potential race condition
9891exists. Thus, once the error surfaces, you need a way to reproduce it.
9892
9893In this example, compiling the "neard" package is causing the problem.
9894So the first thing to do is build "neard" locally. Before you start the
9895build, set the
9896:term:`PARALLEL_MAKE` variable
9897in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
9898high value for :term:`PARALLEL_MAKE` increases the chances of the race
9899condition showing up::
9900
9901 $ bitbake neard
9902
9903Once the local build for "neard" completes, start a ``devshell`` build::
9904
9905 $ bitbake neard -c devshell
9906
9907For information on how to use a ``devshell``, see the
9908":ref:`dev-manual/common-tasks:using a development shell`" section.
9909
9910In the ``devshell``, do the following::
9911
9912 $ make clean
9913 $ make tools/snep-send.o
9914
9915The ``devshell`` commands cause the failure to clearly
9916be visible. In this case, there is a missing dependency for the ``neard``
9917Makefile target. Here is some abbreviated, sample output with the
9918missing dependency clearly visible at the end::
9919
9920 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
9921 .
9922 .
9923 .
9924 tools/snep-send.c
9925 In file included from tools/snep-send.c:16:0:
9926 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9927 #include <near/dbus.h>
9928 ^
9929 compilation terminated.
9930 make: *** [tools/snep-send.o] Error 1
9931 $
9932
9933
9934Creating a Patch for the Fix
9935~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9936
9937Because there is a missing dependency for the Makefile target, you need
9938to patch the ``Makefile.am`` file, which is generated from
9939``Makefile.in``. You can use Quilt to create the patch::
9940
9941 $ quilt new parallelmake.patch
9942 Patch patches/parallelmake.patch is now on top
9943 $ quilt add Makefile.am
9944 File Makefile.am added to patch patches/parallelmake.patch
9945
9946For more information on using Quilt, see the
9947":ref:`dev-manual/common-tasks:using quilt in your workflow`" section.
9948
9949At this point you need to make the edits to ``Makefile.am`` to add the
9950missing dependency. For our example, you have to add the following line
9951to the file::
9952
9953 tools/snep-send.$(OBJEXT): include/near/dbus.h
9954
9955Once you have edited the file, use the ``refresh`` command to create the
9956patch::
9957
9958 $ quilt refresh
9959 Refreshed patch patches/parallelmake.patch
9960
9961Once the patch file is created, you need to add it back to the originating
9962recipe folder. Here is an example assuming a top-level
9963:term:`Source Directory` named ``poky``::
9964
9965 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
9966
9967The final thing you need to do to implement the fix in the build is to
9968update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
9969:term:`SRC_URI` statement includes
9970the patch file. The recipe file is in the folder above the patch. Here
9971is what the edited :term:`SRC_URI` statement would look like::
9972
9973 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
9974 file://neard.in \
9975 file://neard.service.in \
9976 file://parallelmake.patch \
9977 "
9978
9979With the patch complete and moved to the correct folder and the
9980:term:`SRC_URI` statement updated, you can exit the ``devshell``::
9981
9982 $ exit
9983
9984Testing the Build
9985~~~~~~~~~~~~~~~~~
9986
9987With everything in place, you can get back to trying the build again
9988locally::
9989
9990 $ bitbake neard
9991
9992This build should succeed.
9993
9994Now you can open up a ``devshell`` again and repeat the clean and make
9995operations as follows::
9996
9997 $ bitbake neard -c devshell
9998 $ make clean
9999 $ make tools/snep-send.o
10000
10001The build should work without issue.
10002
10003As with all solved problems, if they originated upstream, you need to
10004submit the fix for the recipe in OE-Core and upstream so that the
10005problem is taken care of at its source. See the
10006":ref:`dev-manual/common-tasks:submitting a change to the yocto project`"
10007section for more information.
10008
10009Debugging With the GNU Project Debugger (GDB) Remotely
10010------------------------------------------------------
10011
10012GDB allows you to examine running programs, which in turn helps you to
10013understand and fix problems. It also allows you to perform post-mortem
10014style analysis of program crashes. GDB is available as a package within
10015the Yocto Project and is installed in SDK images by default. See the
10016":ref:`ref-manual/images:Images`" chapter in the Yocto
10017Project Reference Manual for a description of these images. You can find
10018information on GDB at https://sourceware.org/gdb/.
10019
10020.. note::
10021
10022 For best results, install debug (``-dbg``) packages for the applications you
10023 are going to debug. Doing so makes extra debug symbols available that give
10024 you more meaningful output.
10025
10026Sometimes, due to memory or disk space constraints, it is not possible
10027to use GDB directly on the remote target to debug applications. These
10028constraints arise because GDB needs to load the debugging information
10029and the binaries of the process being debugged. Additionally, GDB needs
10030to perform many computations to locate information such as function
10031names, variable names and values, stack traces and so forth --- even
10032before starting the debugging process. These extra computations place
10033more load on the target system and can alter the characteristics of the
10034program being debugged.
10035
10036To help get past the previously mentioned constraints, there are two
10037methods you can use: running a debuginfod server and using gdbserver.
10038
10039Using the debuginfod server method
10040~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10041
10042``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
10043Running a ``debuginfod`` server makes debug symbols readily available,
10044which means you don't need to download debugging information
10045and the binaries of the process being debugged. You can just fetch
10046debug symbols from the server.
10047
10048To run a ``debuginfod`` server, you need to do the following:
10049
10050- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
10051 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
10052 If not, set in your distro config file or in ``local.conf``::
10053
10054 DISTRO_FEATURES:append = " debuginfod"
10055
10056 This distro feature enables the server and client library in ``elfutils``,
10057 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
10058
10059- Run the following commands to launch the ``debuginfod`` server on the host::
10060
10061 $ oe-debuginfod
10062
10063- To use ``debuginfod`` on the target, you need to know the ip:port where
10064 ``debuginfod`` is listening on the host (port defaults to 8002), and export
10065 that into the shell environment, for example in ``qemu``::
10066
10067 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
10068
10069- Then debug info fetching should simply work when running the target ``gdb``,
10070 ``readelf`` or ``objdump``, for example::
10071
10072 root@qemux86-64:~# gdb /bin/cat
10073 ...
10074 Reading symbols from /bin/cat...
10075 Downloading separate debug info for /bin/cat...
10076 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
10077
10078- It's also possible to use ``debuginfod-find`` to just query the server::
10079
10080 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
10081 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
10082
10083
10084Using the gdbserver method
10085~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10086
10087gdbserver, which runs on the remote target and does not load any
10088debugging information from the debugged process. Instead, a GDB instance
10089processes the debugging information that is run on a remote computer -
10090the host GDB. The host GDB then sends control commands to gdbserver to
10091make it stop or start the debugged program, as well as read or write
10092memory regions of that debugged program. All the debugging information
10093loaded and processed as well as all the heavy debugging is done by the
10094host GDB. Offloading these processes gives the gdbserver running on the
10095target a chance to remain small and fast.
10096
10097Because the host GDB is responsible for loading the debugging
10098information and for doing the necessary processing to make actual
10099debugging happen, you have to make sure the host can access the
10100unstripped binaries complete with their debugging information and also
10101be sure the target is compiled with no optimizations. The host GDB must
10102also have local access to all the libraries used by the debugged
10103program. Because gdbserver does not need any local debugging
10104information, the binaries on the remote target can remain stripped.
10105However, the binaries must also be compiled without optimization so they
10106match the host's binaries.
10107
10108To remain consistent with GDB documentation and terminology, the binary
10109being debugged on the remote target machine is referred to as the
10110"inferior" binary. For documentation on GDB see the `GDB
10111site <https://sourceware.org/gdb/documentation/>`__.
10112
10113The following steps show you how to debug using the GNU project
10114debugger.
10115
101161. *Configure your build system to construct the companion debug
10117 filesystem:*
10118
10119 In your ``local.conf`` file, set the following::
10120
10121 IMAGE_GEN_DEBUGFS = "1"
10122 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
10123
10124 These options cause the
10125 OpenEmbedded build system to generate a special companion filesystem
10126 fragment, which contains the matching source and debug symbols to
10127 your deployable filesystem. The build system does this by looking at
10128 what is in the deployed filesystem, and pulling the corresponding
10129 ``-dbg`` packages.
10130
10131 The companion debug filesystem is not a complete filesystem, but only
10132 contains the debug fragments. This filesystem must be combined with
10133 the full filesystem for debugging. Subsequent steps in this procedure
10134 show how to combine the partial filesystem with the full filesystem.
10135
101362. *Configure the system to include gdbserver in the target filesystem:*
10137
10138 Make the following addition in your ``local.conf`` file::
10139
10140 EXTRA_IMAGE_FEATURES:append = " tools-debug"
10141
10142 The change makes
10143 sure the ``gdbserver`` package is included.
10144
101453. *Build the environment:*
10146
10147 Use the following command to construct the image and the companion
10148 Debug Filesystem::
10149
10150 $ bitbake image
10151
10152 Build the cross GDB component and
10153 make it available for debugging. Build the SDK that matches the
10154 image. Building the SDK is best for a production build that can be
10155 used later for debugging, especially during long term maintenance::
10156
10157 $ bitbake -c populate_sdk image
10158
10159 Alternatively, you can build the minimal toolchain components that
10160 match the target. Doing so creates a smaller than typical SDK and
10161 only contains a minimal set of components with which to build simple
10162 test applications, as well as run the debugger::
10163
10164 $ bitbake meta-toolchain
10165
10166 A final method is to build Gdb itself within the build system::
10167
10168 $ bitbake gdb-cross-<architecture>
10169
10170 Doing so produces a temporary copy of
10171 ``cross-gdb`` you can use for debugging during development. While
10172 this is the quickest approach, the two previous methods in this step
10173 are better when considering long-term maintenance strategies.
10174
10175 .. note::
10176
10177 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
10178 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
10179 actual name you want to use.
10180
101814. *Set up the* ``debugfs``\ *:*
10182
10183 Run the following commands to set up the ``debugfs``::
10184
10185 $ mkdir debugfs
10186 $ cd debugfs
10187 $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2
10188 $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2
10189
101905. *Set up GDB:*
10191
10192 Install the SDK (if you built one) and then source the correct
10193 environment file. Sourcing the environment file puts the SDK in your
10194 ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger.
10195
10196 If you are using the build system, Gdb is located in
10197 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
10198
101996. *Boot the target:*
10200
10201 For information on how to run QEMU, see the `QEMU
10202 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
10203
10204 .. note::
10205
10206 Be sure to verify that your host can access the target via TCP.
10207
102087. *Debug a program:*
10209
10210 Debugging a program involves running gdbserver on the target and then
10211 running Gdb on the host. The example in this step debugs ``gzip``:
10212
10213 .. code-block:: shell
10214
10215 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
10216
10217 For
10218 additional gdbserver options, see the `GDB Server
10219 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
10220
10221 After running gdbserver on the target, you need to run Gdb on the
10222 host and configure it and connect to the target. Use these commands::
10223
10224 $ cd directory-holding-the-debugfs-directory
10225 $ arch-gdb
10226 (gdb) set sysroot debugfs
10227 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
10228 (gdb) target remote IP-of-target:1234
10229
10230 At this
10231 point, everything should automatically load (i.e. matching binaries,
10232 symbols and headers).
10233
10234 .. note::
10235
10236 The Gdb ``set`` commands in the previous example can be placed into the
10237 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
10238 commands are in that file.
10239
102408. *Deploying without a full image rebuild:*
10241
10242 In many cases, during development you want a quick method to deploy a
10243 new binary to the target and debug it, without waiting for a full
10244 image build.
10245
10246 One approach to solving this situation is to just build the component
10247 you want to debug. Once you have built the component, copy the
10248 executable directly to both the target and the host ``debugfs``.
10249
10250 If the binary is processed through the debug splitting in
10251 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
10252 contents and corresponding ``/usr/src/debug`` files) from the work
10253 directory. Here is an example::
10254
10255 $ bitbake bash
10256 $ bitbake -c devshell bash
10257 $ cd ..
10258 $ scp packages-split/bash/bin/bash target:/bin/bash
10259 $ cp -a packages-split/bash-dbg/\* path/debugfs
10260
10261Debugging with the GNU Project Debugger (GDB) on the Target
10262-----------------------------------------------------------
10263
10264The previous section addressed using GDB remotely for debugging
10265purposes, which is the most usual case due to the inherent hardware
10266limitations on many embedded devices. However, debugging in the target
10267hardware itself is also possible with more powerful devices. This
10268section describes what you need to do in order to support using GDB to
10269debug on the target hardware.
10270
10271To support this kind of debugging, you need do the following:
10272
10273- Ensure that GDB is on the target. You can do this by making
10274 the following addition to your ``local.conf`` file::
10275
10276 EXTRA_IMAGE_FEATURES:append = " tools-debug"
10277
10278- Ensure that debug symbols are present. You can do so by adding the
10279 corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`::
10280
10281 IMAGE_INSTALL:append = " packagename-dbg"
10282
10283 Alternatively, you can add the following to ``local.conf`` to include
10284 all the debug symbols::
10285
10286 EXTRA_IMAGE_FEATURES:append = " dbg-pkgs"
10287
10288.. note::
10289
10290 To improve the debug information accuracy, you can reduce the level
10291 of optimization used by the compiler. For example, when adding the
10292 following line to your ``local.conf`` file, you will reduce optimization
10293 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
10294 of "-O -fno-omit-frame-pointer"::
10295
10296 DEBUG_BUILD = "1"
10297
10298 Consider that this will reduce the application's performance and is
10299 recommended only for debugging purposes.
10300
10301Other Debugging Tips
10302--------------------
10303
10304Here are some other tips that you might find useful:
10305
10306- When adding new packages, it is worth watching for undesirable items
10307 making their way into compiler command lines. For example, you do not
10308 want references to local system files like ``/usr/lib/`` or
10309 ``/usr/include/``.
10310
10311- If you want to remove the ``psplash`` boot splashscreen, add
10312 ``psplash=false`` to the kernel command line. Doing so prevents
10313 ``psplash`` from loading and thus allows you to see the console. It
10314 is also possible to switch out of the splashscreen by switching the
10315 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10316
10317- Removing :term:`TMPDIR` (usually ``tmp/``, within the
10318 :term:`Build Directory`) can often fix temporary build issues. Removing
10319 :term:`TMPDIR` is usually a relatively cheap operation, because task output
10320 will be cached in :term:`SSTATE_DIR` (usually ``sstate-cache/``, which is
10321 also in the :term:`Build Directory`).
10322
10323 .. note::
10324
10325 Removing :term:`TMPDIR` might be a workaround rather than a fix.
10326 Consequently, trying to determine the underlying cause of an issue before
10327 removing the directory is a good idea.
10328
10329- Understanding how a feature is used in practice within existing
10330 recipes can be very helpful. It is recommended that you configure
10331 some method that allows you to quickly search through files.
10332
10333 Using GNU Grep, you can use the following shell function to
10334 recursively search through common recipe-related files, skipping
10335 binary files, ``.git`` directories, and the :term:`Build Directory`
10336 (assuming its name starts with "build")::
10337
10338 g() {
10339 grep -Ir \
10340 --exclude-dir=.git \
10341 --exclude-dir='build*' \
10342 --include='*.bb*' \
10343 --include='*.inc*' \
10344 --include='*.conf*' \
10345 --include='*.py*' \
10346 "$@"
10347 }
10348
10349 Following are some usage examples::
10350
10351 $ g FOO # Search recursively for "FOO"
10352 $ g -i foo # Search recursively for "foo", ignoring case
10353 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10354
10355 If figuring
10356 out how some feature works requires a lot of searching, it might
10357 indicate that the documentation should be extended or improved. In
10358 such cases, consider filing a documentation bug using the Yocto
10359 Project implementation of
10360 :yocto_bugs:`Bugzilla <>`. For information on
10361 how to submit a bug against the Yocto Project, see the Yocto Project
10362 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
10363 and the
10364 ":ref:`dev-manual/common-tasks:submitting a defect against the yocto project`"
10365 section.
10366
10367 .. note::
10368
10369 The manuals might not be the right place to document variables
10370 that are purely internal and have a limited scope (e.g. internal
10371 variables used to implement a single ``.bbclass`` file).
10372
10373Making Changes to the Yocto Project
10374===================================
10375
10376Because the Yocto Project is an open-source, community-based project,
10377you can effect changes to the project. This section presents procedures
10378that show you how to submit a defect against the project and how to
10379submit a change.
10380
10381Submitting a Defect Against the Yocto Project
10382---------------------------------------------
10383
10384Use the Yocto Project implementation of
10385`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
10386against the Yocto Project. For additional information on this
10387implementation of Bugzilla see the ":ref:`Yocto Project
10388Bugzilla <resources-bugtracker>`" section in the
10389Yocto Project Reference Manual. For more detail on any of the following
10390steps, see the Yocto Project
10391:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`.
10392
10393Use the following general steps to submit a bug:
10394
103951. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10396
103972. Click "File a Bug" to enter a new bug.
10398
103993. Choose the appropriate "Classification", "Product", and "Component"
10400 for which the bug was found. Bugs for the Yocto Project fall into
10401 one of several classifications, which in turn break down into
10402 several products and components. For example, for a bug against the
10403 ``meta-intel`` layer, you would choose "Build System, Metadata &
10404 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10405
104064. Choose the "Version" of the Yocto Project for which you found the
10407 bug (e.g. &DISTRO;).
10408
104095. Determine and select the "Severity" of the bug. The severity
10410 indicates how the bug impacted your work.
10411
104126. Choose the "Hardware" that the bug impacts.
10413
104147. Choose the "Architecture" that the bug impacts.
10415
104168. Choose a "Documentation change" item for the bug. Fixing a bug might
10417 or might not affect the Yocto Project documentation. If you are
10418 unsure of the impact to the documentation, select "Don't Know".
10419
104209. Provide a brief "Summary" of the bug. Try to limit your summary to
10421 just a line or two and be sure to capture the essence of the bug.
10422
1042310. Provide a detailed "Description" of the bug. You should provide as
10424 much detail as you can about the context, behavior, output, and so
10425 forth that surrounds the bug. You can even attach supporting files
10426 for output from logs by using the "Add an attachment" button.
10427
1042811. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10429 is assigned to the bug and the defect is logged in the bug tracking
10430 system.
10431
10432Once you file a bug, the bug is processed by the Yocto Project Bug
10433Triage Team and further details concerning the bug are assigned (e.g.
10434priority and owner). You are the "Submitter" of the bug and any further
10435categorization, progress, or comments on the bug result in Bugzilla
10436sending you an automated email concerning the particular change or
10437progress to the bug.
10438
10439Submitting a Change to the Yocto Project
10440----------------------------------------
10441
10442Contributions to the Yocto Project and OpenEmbedded are very welcome.
10443Because the system is extremely configurable and flexible, we recognize
10444that developers will want to extend, configure or optimize it for their
10445specific uses.
10446
10447The Yocto Project uses a mailing list and a patch-based workflow that is
10448similar to the Linux kernel but contains important differences. In
10449general, there is a mailing list through which you can submit patches. You
10450should send patches to the appropriate mailing list so that they can be
10451reviewed and merged by the appropriate maintainer. The specific mailing
10452list you need to use depends on the location of the code you are
10453changing. Each component (e.g. layer) should have a ``README`` file that
10454indicates where to send the changes and which process to follow.
10455
10456You can send the patch to the mailing list using whichever approach you
10457feel comfortable with to generate the patch. Once sent, the patch is
10458usually reviewed by the community at large. If somebody has concerns
10459with the patch, they will usually voice their concern over the mailing
10460list. If a patch does not receive any negative reviews, the maintainer
10461of the affected layer typically takes the patch, tests it, and then
10462based on successful testing, merges the patch.
10463
10464The "poky" repository, which is the Yocto Project's reference build
10465environment, is a hybrid repository that contains several individual
10466pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10467the combo-layer tool. The upstream location used for submitting changes
10468varies by component:
10469
10470- *Core Metadata:* Send your patch to the
10471 :oe_lists:`openembedded-core </g/openembedded-core>`
10472 mailing list. For example, a change to anything under the ``meta`` or
10473 ``scripts`` directories should be sent to this mailing list.
10474
10475- *BitBake:* For changes to BitBake (i.e. anything under the
10476 ``bitbake`` directory), send your patch to the
10477 :oe_lists:`bitbake-devel </g/bitbake-devel>`
10478 mailing list.
10479
10480- *"meta-\*" trees:* These trees contain Metadata. Use the
10481 :yocto_lists:`poky </g/poky>` mailing list.
10482
10483- *Documentation*: For changes to the Yocto Project documentation, use the
10484 :yocto_lists:`docs </g/docs>` mailing list.
10485
10486For changes to other layers hosted in the Yocto Project source
10487repositories (i.e. ``yoctoproject.org``) and tools use the
10488:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
10489
10490.. note::
10491
10492 Sometimes a layer's documentation specifies to use a particular
10493 mailing list. If so, use that list.
10494
10495For additional recipes that do not fit into the core Metadata, you
10496should determine which layer the recipe should go into and submit the
10497change in the manner recommended by the documentation (e.g. the
10498``README`` file) supplied with the layer. If in doubt, please ask on the
10499Yocto general mailing list or on the openembedded-devel mailing list.
10500
10501You can also push a change upstream and request a maintainer to pull the
10502change into the component's upstream repository. You do this by pushing
10503to a contribution repository that is upstream. See the
10504":ref:`overview-manual/development-environment:git workflows and the yocto project`"
10505section in the Yocto Project Overview and Concepts Manual for additional
10506concepts on working in the Yocto Project development environment.
10507
10508Maintainers commonly use ``-next`` branches to test submissions prior to
10509merging patches. Thus, you can get an idea of the status of a patch based on
10510whether the patch has been merged into one of these branches. The commonly
10511used testing branches for OpenEmbedded-Core are as follows:
10512
10513- *openembedded-core "master-next" branch:* This branch is part of the
10514 :oe_git:`openembedded-core </openembedded-core/>` repository and contains
10515 proposed changes to the core metadata.
10516
10517- *poky "master-next" branch:* This branch is part of the
10518 :yocto_git:`poky </poky/>` repository and combines proposed
10519 changes to BitBake, the core metadata and the poky distro.
10520
10521Similarly, stable branches maintained by the project may have corresponding
10522``-next`` branches which collect proposed changes. For example,
10523``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next``
10524branches in both the "openembdedded-core" and "poky" repositories.
10525
10526Other layers may have similar testing branches but there is no formal
10527requirement or standard for these so please check the documentation for the
10528layers you are contributing to.
10529
10530The following sections provide procedures for submitting a change.
10531
10532Preparing Changes for Submission
10533~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10534
105351. *Make Your Changes Locally:* Make your changes in your local Git
10536 repository. You should make small, controlled, isolated changes.
10537 Keeping changes small and isolated aids review, makes
10538 merging/rebasing easier and keeps the change history clean should
10539 anyone need to refer to it in future.
10540
105412. *Stage Your Changes:* Stage your changes by using the ``git add``
10542 command on each file you changed.
10543
105443. *Commit Your Changes:* Commit the change by using the ``git commit``
10545 command. Make sure your commit information follows standards by
10546 following these accepted conventions:
10547
10548 - Be sure to include a "Signed-off-by:" line in the same style as
10549 required by the Linux kernel. This can be done by using the
10550 ``git commit -s`` command. Adding this line signifies that you,
10551 the submitter, have agreed to the Developer's Certificate of
10552 Origin 1.1 as follows:
10553
10554 .. code-block:: none
10555
10556 Developer's Certificate of Origin 1.1
10557
10558 By making a contribution to this project, I certify that:
10559
10560 (a) The contribution was created in whole or in part by me and I
10561 have the right to submit it under the open source license
10562 indicated in the file; or
10563
10564 (b) The contribution is based upon previous work that, to the best
10565 of my knowledge, is covered under an appropriate open source
10566 license and I have the right under that license to submit that
10567 work with modifications, whether created in whole or in part
10568 by me, under the same open source license (unless I am
10569 permitted to submit under a different license), as indicated
10570 in the file; or
10571
10572 (c) The contribution was provided directly to me by some other
10573 person who certified (a), (b) or (c) and I have not modified
10574 it.
10575
10576 (d) I understand and agree that this project and the contribution
10577 are public and that a record of the contribution (including all
10578 personal information I submit with it, including my sign-off) is
10579 maintained indefinitely and may be redistributed consistent with
10580 this project or the open source license(s) involved.
10581
10582 - Provide a single-line summary of the change and, if more
10583 explanation is needed, provide more detail in the body of the
10584 commit. This summary is typically viewable in the "shortlist" of
10585 changes. Thus, providing something short and descriptive that
10586 gives the reader a summary of the change is useful when viewing a
10587 list of many commits. You should prefix this short description
10588 with the recipe name (if changing a recipe), or else with the
10589 short form path to the file being changed.
10590
10591 - For the body of the commit message, provide detailed information
10592 that describes what you changed, why you made the change, and the
10593 approach you used. It might also be helpful if you mention how you
10594 tested the change. Provide as much detail as you can in the body
10595 of the commit message.
10596
10597 .. note::
10598
10599 You do not need to provide a more detailed explanation of a
10600 change if the change is minor to the point of the single line
10601 summary providing all the information.
10602
10603 - If the change addresses a specific bug or issue that is associated
10604 with a bug-tracking ID, include a reference to that ID in your
10605 detailed description. For example, the Yocto Project uses a
10606 specific convention for bug references --- any commit that addresses
10607 a specific bug should use the following form for the detailed
10608 description. Be sure to use the actual bug-tracking ID from
10609 Bugzilla for bug-id::
10610
10611 Fixes [YOCTO #bug-id]
10612
10613 detailed description of change
10614
10615Using Email to Submit a Patch
10616~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10617
10618Depending on the components changed, you need to submit the email to a
10619specific mailing list. For some guidance on which mailing list to use,
10620see the
10621:ref:`list <dev-manual/common-tasks:submitting a change to the yocto project>`
10622at the beginning of this section. For a description of all the available
10623mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
10624Yocto Project Reference Manual.
10625
10626Here is the general procedure on how to submit a patch through email
10627without using the scripts once the steps in
10628:ref:`dev-manual/common-tasks:preparing changes for submission` have been followed:
10629
106301. *Format the Commit:* Format the commit into an email message. To
10631 format commits, use the ``git format-patch`` command. When you
10632 provide the command, you must include a revision list or a number of
10633 patches as part of the command. For example, either of these two
10634 commands takes your most recent single commit and formats it as an
10635 email message in the current directory::
10636
10637 $ git format-patch -1
10638
10639 or ::
10640
10641 $ git format-patch HEAD~
10642
10643 After the command is run, the current directory contains a numbered
10644 ``.patch`` file for the commit.
10645
10646 If you provide several commits as part of the command, the
10647 ``git format-patch`` command produces a series of numbered files in
10648 the current directory – one for each commit. If you have more than
10649 one patch, you should also use the ``--cover`` option with the
10650 command, which generates a cover letter as the first "patch" in the
10651 series. You can then edit the cover letter to provide a description
10652 for the series of patches. For information on the
10653 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
10654 using the ``man git-format-patch`` command.
10655
10656 .. note::
10657
10658 If you are or will be a frequent contributor to the Yocto Project
10659 or to OpenEmbedded, you might consider requesting a contrib area
10660 and the necessary associated rights.
10661
106622. *Send the patches via email:* Send the patches to the recipients and
10663 relevant mailing lists by using the ``git send-email`` command.
10664
10665 .. note::
10666
10667 In order to use ``git send-email``, you must have the proper Git packages
10668 installed on your host.
10669 For Ubuntu, Debian, and Fedora the package is ``git-email``.
10670
10671 The ``git send-email`` command sends email by using a local or remote
10672 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
10673 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
10674 file. If you are submitting patches through email only, it is very
10675 important that you submit them without any whitespace or HTML
10676 formatting that either you or your mailer introduces. The maintainer
10677 that receives your patches needs to be able to save and apply them
10678 directly from your emails. A good way to verify that what you are
10679 sending will be applicable by the maintainer is to do a dry run and
10680 send them to yourself and then save and apply them as the maintainer
10681 would.
10682
10683 The ``git send-email`` command is the preferred method for sending
10684 your patches using email since there is no risk of compromising
10685 whitespace in the body of the message, which can occur when you use
10686 your own mail client. The command also has several options that let
10687 you specify recipients and perform further editing of the email
10688 message. For information on how to use the ``git send-email``
10689 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
10690 ``man git-send-email`` command.
10691
10692The Yocto Project uses a `Patchwork instance <https://patchwork.openembedded.org/>`__
10693to track the status of patches submitted to the various mailing lists and to
10694support automated patch testing. Each submitted patch is checked for common
10695mistakes and deviations from the expected patch format and submitters are
10696notified by patchtest if such mistakes are found. This process helps to
10697reduce the burden of patch review on maintainers.
10698
10699.. note::
10700
10701 This system is imperfect and changes can sometimes get lost in the flow.
10702 Asking about the status of a patch or change is reasonable if the change
10703 has been idle for a while with no feedback.
10704
10705Using Scripts to Push a Change Upstream and Request a Pull
10706~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10707
10708For larger patch series it is preferable to send a pull request which not
10709only includes the patch but also a pointer to a branch that can be pulled
10710from. This involves making a local branch for your changes, pushing this
10711branch to an accessible repository and then using the ``create-pull-request``
10712and ``send-pull-request`` scripts from openembedded-core to create and send a
10713patch series with a link to the branch for review.
10714
10715Follow this procedure to push a change to an upstream "contrib" Git
10716repository once the steps in :ref:`dev-manual/common-tasks:preparing changes for submission` have
10717been followed:
10718
10719.. note::
10720
10721 You can find general Git information on how to push a change upstream
10722 in the
10723 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
10724
107251. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
10726 permissions to push to an upstream contrib repository, push the
10727 change to that repository::
10728
10729 $ git push upstream_remote_repo local_branch_name
10730
10731 For example, suppose you have permissions to push
10732 into the upstream ``meta-intel-contrib`` repository and you are
10733 working in a local branch named `your_name`\ ``/README``. The following
10734 command pushes your local commits to the ``meta-intel-contrib``
10735 upstream repository and puts the commit in a branch named
10736 `your_name`\ ``/README``::
10737
10738 $ git push meta-intel-contrib your_name/README
10739
107402. *Determine Who to Notify:* Determine the maintainer or the mailing
10741 list that you need to notify for the change.
10742
10743 Before submitting any change, you need to be sure who the maintainer
10744 is or what mailing list that you need to notify. Use either these
10745 methods to find out:
10746
10747 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10748 located in the :term:`Source Directory` at
10749 ``meta/conf/distro/include``, to see who is responsible for code.
10750
10751 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can
10752 enter the following command to bring up a short list of all
10753 commits against a specific file::
10754
10755 git shortlog -- filename
10756
10757 Just provide the name of the file for which you are interested. The
10758 information returned is not ordered by history but does include a
10759 list of everyone who has committed grouped by name. From the list,
10760 you can see who is responsible for the bulk of the changes against
10761 the file.
10762
10763 - *Examine the List of Mailing Lists:* For a list of the Yocto
10764 Project and related mailing lists, see the ":ref:`Mailing
10765 lists <resources-mailinglist>`" section in
10766 the Yocto Project Reference Manual.
10767
107683. *Make a Pull Request:* Notify the maintainer or the mailing list that
10769 you have pushed a change by making a pull request.
10770
10771 The Yocto Project provides two scripts that conveniently let you
10772 generate and send pull requests to the Yocto Project. These scripts
10773 are ``create-pull-request`` and ``send-pull-request``. You can find
10774 these scripts in the ``scripts`` directory within the
10775 :term:`Source Directory` (e.g.
10776 ``poky/scripts``).
10777
10778 Using these scripts correctly formats the requests without
10779 introducing any whitespace or HTML formatting. The maintainer that
10780 receives your patches either directly or through the mailing list
10781 needs to be able to save and apply them directly from your emails.
10782 Using these scripts is the preferred method for sending patches.
10783
10784 First, create the pull request. For example, the following command
10785 runs the script, specifies the upstream repository in the contrib
10786 directory into which you pushed the change, and provides a subject
10787 line in the created patch files::
10788
10789 $ poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
10790
10791 Running this script forms ``*.patch`` files in a folder named
10792 ``pull-``\ `PID` in the current directory. One of the patch files is a
10793 cover letter.
10794
10795 Before running the ``send-pull-request`` script, you must edit the
10796 cover letter patch to insert information about your change. After
10797 editing the cover letter, send the pull request. For example, the
10798 following command runs the script and specifies the patch directory
10799 and email address. In this example, the email address is a mailing
10800 list::
10801
10802 $ poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@lists.yoctoproject.org
10803
10804 You need to follow the prompts as the script is interactive.
10805
10806 .. note::
10807
10808 For help on using these scripts, simply provide the ``-h``
10809 argument as follows::
10810
10811 $ poky/scripts/create-pull-request -h
10812 $ poky/scripts/send-pull-request -h
10813
10814Responding to Patch Review
10815~~~~~~~~~~~~~~~~~~~~~~~~~~
10816
10817You may get feedback on your submitted patches from other community members
10818or from the automated patchtest service. If issues are identified in your
10819patch then it is usually necessary to address these before the patch will be
10820accepted into the project. In this case you should amend the patch according
10821to the feedback and submit an updated version to the relevant mailing list,
10822copying in the reviewers who provided feedback to the previous version of the
10823patch.
10824
10825The patch should be amended using ``git commit --amend`` or perhaps ``git
10826rebase`` for more expert git users. You should also modify the ``[PATCH]``
10827tag in the email subject line when sending the revised patch to mark the new
10828iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be
10829done by passing the ``-v`` argument to ``git format-patch`` with a version
10830number.
10831
10832Lastly please ensure that you also test your revised changes. In particular
10833please don't just edit the patch file written out by ``git format-patch`` and
10834resend it.
10835
10836Submitting Changes to Stable Release Branches
10837~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10838
10839The process for proposing changes to a Yocto Project stable branch differs
10840from the steps described above. Changes to a stable branch must address
10841identified bugs or CVEs and should be made carefully in order to avoid the
10842risk of introducing new bugs or breaking backwards compatibility. Typically
10843bug fixes must already be accepted into the master branch before they can be
10844backported to a stable branch unless the bug in question does not affect the
10845master branch or the fix on the master branch is unsuitable for backporting.
10846
10847The list of stable branches along with the status and maintainer for each
10848branch can be obtained from the
10849:yocto_wiki:`Releases wiki page </Releases>`.
10850
10851.. note::
10852
10853 Changes will not typically be accepted for branches which are marked as
10854 End-Of-Life (EOL).
10855
10856With this in mind, the steps to submit a change for a stable branch are as
10857follows:
10858
108591. *Identify the bug or CVE to be fixed:* This information should be
10860 collected so that it can be included in your submission.
10861
10862 See :ref:`dev-manual/common-tasks:checking for vulnerabilities`
10863 for details about CVE tracking.
10864
108652. *Check if the fix is already present in the master branch:* This will
10866 result in the most straightforward path into the stable branch for the
10867 fix.
10868
10869 a. *If the fix is present in the master branch --- submit a backport request
10870 by email:* You should send an email to the relevant stable branch
10871 maintainer and the mailing list with details of the bug or CVE to be
10872 fixed, the commit hash on the master branch that fixes the issue and
10873 the stable branches which you would like this fix to be backported to.
10874
10875 b. *If the fix is not present in the master branch --- submit the fix to the
10876 master branch first:* This will ensure that the fix passes through the
10877 project's usual patch review and test processes before being accepted.
10878 It will also ensure that bugs are not left unresolved in the master
10879 branch itself. Once the fix is accepted in the master branch a backport
10880 request can be submitted as above.
10881
10882 c. *If the fix is unsuitable for the master branch --- submit a patch
10883 directly for the stable branch:* This method should be considered as a
10884 last resort. It is typically necessary when the master branch is using
10885 a newer version of the software which includes an upstream fix for the
10886 issue or when the issue has been fixed on the master branch in a way
10887 that introduces backwards incompatible changes. In this case follow the
10888 steps in :ref:`dev-manual/common-tasks:preparing changes for submission` and
10889 :ref:`dev-manual/common-tasks:using email to submit a patch` but modify the subject header of your patch
10890 email to include the name of the stable branch which you are
10891 targetting. This can be done using the ``--subject-prefix`` argument to
10892 ``git format-patch``, for example to submit a patch to the dunfell
10893 branch use
10894 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``.
10895
10896Working With Licenses
10897=====================
10898
10899As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
10900section in the Yocto Project Overview and Concepts Manual, open source
10901projects are open to the public and they consequently have different
10902licensing structures in place. This section describes the mechanism by
10903which the :term:`OpenEmbedded Build System`
10904tracks changes to
10905licensing text and covers how to maintain open source license compliance
10906during your project's lifecycle. The section also describes how to
10907enable commercially licensed recipes, which by default are disabled.
10908
10909Tracking License Changes
10910------------------------
10911
10912The license of an upstream project might change in the future. In order
10913to prevent these changes going unnoticed, the
10914:term:`LIC_FILES_CHKSUM`
10915variable tracks changes to the license text. The checksums are validated
10916at the end of the configure step, and if the checksums do not match, the
10917build will fail.
10918
10919Specifying the ``LIC_FILES_CHKSUM`` Variable
10920~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10921
10922The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
10923in the source code for the recipe. Following is an example of how to
10924specify :term:`LIC_FILES_CHKSUM`::
10925
10926 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
10927 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
10928 file://licfile2.txt;endline=50;md5=zzzz \
10929 ..."
10930
10931.. note::
10932
10933 - When using "beginline" and "endline", realize that line numbering
10934 begins with one and not zero. Also, the included lines are
10935 inclusive (i.e. lines five through and including 29 in the
10936 previous example for ``licfile1.txt``).
10937
10938 - When a license check fails, the selected license text is included
10939 as part of the QA message. Using this output, you can determine
10940 the exact start and finish for the needed license text.
10941
10942The build system uses the :term:`S`
10943variable as the default directory when searching files listed in
10944:term:`LIC_FILES_CHKSUM`. The previous example employs the default
10945directory.
10946
10947Consider this next example::
10948
10949 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
10950 md5=bb14ed3c4cda583abc85401304b5cd4e"
10951 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
10952
10953The first line locates a file in ``${S}/src/ls.c`` and isolates lines
10954five through 16 as license text. The second line refers to a file in
10955:term:`WORKDIR`.
10956
10957Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
10958unless the :term:`LICENSE` variable is set to "CLOSED".
10959
10960Explanation of Syntax
10961~~~~~~~~~~~~~~~~~~~~~
10962
10963As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
10964lists all the important files that contain the license text for the
10965source code. It is possible to specify a checksum for an entire file, or
10966a specific section of a file (specified by beginning and ending line
10967numbers with the "beginline" and "endline" parameters, respectively).
10968The latter is useful for source files with a license notice header,
10969README documents, and so forth. If you do not use the "beginline"
10970parameter, then it is assumed that the text begins on the first line of
10971the file. Similarly, if you do not use the "endline" parameter, it is
10972assumed that the license text ends with the last line of the file.
10973
10974The "md5" parameter stores the md5 checksum of the license text. If the
10975license text changes in any way as compared to this parameter then a
10976mismatch occurs. This mismatch triggers a build failure and notifies the
10977developer. Notification allows the developer to review and address the
10978license text changes. Also note that if a mismatch occurs during the
10979build, the correct md5 checksum is placed in the build log and can be
10980easily copied to the recipe.
10981
10982There is no limit to how many files you can specify using the
10983:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
10984requires a few specifications for license tracking. Many projects have a
10985"COPYING" file that stores the license information for all the source
10986code files. This practice allows you to just track the "COPYING" file as
10987long as it is kept up to date.
10988
10989.. note::
10990
10991 - If you specify an empty or invalid "md5" parameter,
10992 :term:`BitBake` returns an md5
10993 mis-match error and displays the correct "md5" parameter value
10994 during the build. The correct parameter is also captured in the
10995 build log.
10996
10997 - If the whole file contains only license text, you do not need to
10998 use the "beginline" and "endline" parameters.
10999
11000Enabling Commercially Licensed Recipes
11001--------------------------------------
11002
11003By default, the OpenEmbedded build system disables components that have
11004commercial or other special licensing requirements. Such requirements
11005are defined on a recipe-by-recipe basis through the
11006:term:`LICENSE_FLAGS` variable
11007definition in the affected recipe. For instance, the
11008``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
11009contains the following statement::
11010
11011 LICENSE_FLAGS = "commercial"
11012
11013Here is a
11014slightly more complicated example that contains both an explicit recipe
11015name and version (after variable expansion)::
11016
11017 LICENSE_FLAGS = "license_${PN}_${PV}"
11018
11019In order for a component restricted by a
11020:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
11021needs to have a matching entry in the global
11022:term:`LICENSE_FLAGS_ACCEPTED`
11023variable, which is a variable typically defined in your ``local.conf``
11024file. For example, to enable the
11025``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
11026could add either the string "commercial_gst-plugins-ugly" or the more
11027general string "commercial" to :term:`LICENSE_FLAGS_ACCEPTED`. See the
11028":ref:`dev-manual/common-tasks:license flag matching`" section for a full
11029explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
11030example::
11031
11032 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly"
11033
11034Likewise, to additionally enable the package built from the recipe
11035containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
11036the actual recipe name was ``emgd_1.10.bb``, the following string would
11037enable that package as well as the original ``gst-plugins-ugly``
11038package::
11039
11040 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly license_emgd_1.10"
11041
11042As a convenience, you do not need to specify the
11043complete license string for every package. You can use
11044an abbreviated form, which consists of just the first portion or
11045portions of the license string before the initial underscore character
11046or characters. A partial string will match any license that contains the
11047given string as the first portion of its license. For example, the
11048following value will also match both of the packages
11049previously mentioned as well as any other packages that have licenses
11050starting with "commercial" or "license".
11051::
11052
11053 LICENSE_FLAGS_ACCEPTED = "commercial license"
11054
11055License Flag Matching
11056~~~~~~~~~~~~~~~~~~~~~
11057
11058License flag matching allows you to control what recipes the
11059OpenEmbedded build system includes in the build. Fundamentally, the
11060build system attempts to match :term:`LICENSE_FLAGS` strings found in
11061recipes against strings found in :term:`LICENSE_FLAGS_ACCEPTED`.
11062A match causes the build system to include a recipe in the
11063build, while failure to find a match causes the build system to exclude
11064a recipe.
11065
11066In general, license flag matching is simple. However, understanding some
11067concepts will help you correctly and effectively use matching.
11068
11069Before a flag defined by a particular recipe is tested against the
11070entries of :term:`LICENSE_FLAGS_ACCEPTED`, the expanded
11071string ``_${PN}`` is appended to the flag. This expansion makes each
11072:term:`LICENSE_FLAGS` value recipe-specific. After expansion, the
11073string is then matched against the entries. Thus, specifying
11074``LICENSE_FLAGS = "commercial"`` in recipe "foo", for example, results
11075in the string ``"commercial_foo"``. And, to create a match, that string
11076must appear among the entries of :term:`LICENSE_FLAGS_ACCEPTED`.
11077
11078Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
11079:term:`LICENSE_FLAGS_ACCEPTED` variable allows you a lot of flexibility for
11080including or excluding recipes based on licensing. For example, you can
11081broaden the matching capabilities by using license flags string subsets
11082in :term:`LICENSE_FLAGS_ACCEPTED`.
11083
11084.. note::
11085
11086 When using a string subset, be sure to use the part of the expanded
11087 string that precedes the appended underscore character (e.g.
11088 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
11089
11090For example, simply specifying the string "commercial" in the
11091:term:`LICENSE_FLAGS_ACCEPTED` variable matches any expanded
11092:term:`LICENSE_FLAGS` definition that starts with the string
11093"commercial" such as "commercial_foo" and "commercial_bar", which
11094are the strings the build system automatically generates for
11095hypothetical recipes named "foo" and "bar" assuming those recipes simply
11096specify the following::
11097
11098 LICENSE_FLAGS = "commercial"
11099
11100Thus, you can choose to exhaustively enumerate each license flag in the
11101list and allow only specific recipes into the image, or you can use a
11102string subset that causes a broader range of matches to allow a range of
11103recipes into the image.
11104
11105This scheme works even if the :term:`LICENSE_FLAGS` string already has
11106``_${PN}`` appended. For example, the build system turns the license
11107flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
11108both the general "commercial" and the specific "commercial_1.2_foo"
11109strings found in the :term:`LICENSE_FLAGS_ACCEPTED` variable, as expected.
11110
11111Here are some other scenarios:
11112
11113- You can specify a versioned string in the recipe such as
11114 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
11115 string to "commercial_foo_1.2_foo". Combine this license flag with a
11116 :term:`LICENSE_FLAGS_ACCEPTED` variable that has the string
11117 "commercial" and you match the flag along with any other flag that
11118 starts with the string "commercial".
11119
11120- Under the same circumstances, you can add "commercial_foo" in the
11121 :term:`LICENSE_FLAGS_ACCEPTED` variable and the build system not only
11122 matches "commercial_foo_1.2" but also matches any license flag with
11123 the string "commercial_foo", regardless of the version.
11124
11125- You can be very specific and use both the package and version parts
11126 in the :term:`LICENSE_FLAGS_ACCEPTED` list (e.g.
11127 "commercial_foo_1.2") to specifically match a versioned recipe.
11128
11129Other Variables Related to Commercial Licenses
11130~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11131
11132There are other helpful variables related to commercial license handling,
11133defined in the
11134``poky/meta/conf/distro/include/default-distrovars.inc`` file::
11135
11136 COMMERCIAL_AUDIO_PLUGINS ?= ""
11137 COMMERCIAL_VIDEO_PLUGINS ?= ""
11138
11139If you
11140want to enable these components, you can do so by making sure you have
11141statements similar to the following in your ``local.conf`` configuration
11142file::
11143
11144 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
11145 gst-plugins-ugly-mpegaudioparse"
11146 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
11147 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
11148 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
11149
11150
11151Of course, you could also create a matching list for those
11152components using the more general "commercial" in the
11153:term:`LICENSE_FLAGS_ACCEPTED` variable, but that would also enable all
11154the other packages with :term:`LICENSE_FLAGS`
11155containing "commercial", which you may or may not want::
11156
11157 LICENSE_FLAGS_ACCEPTED = "commercial"
11158
11159Specifying audio and video plugins as part of the
11160``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
11161(along with the enabling :term:`LICENSE_FLAGS_ACCEPTED`) includes the
11162plugins or components into built images, thus adding support for media
11163formats or components.
11164
11165Maintaining Open Source License Compliance During Your Product's Lifecycle
11166--------------------------------------------------------------------------
11167
11168One of the concerns for a development organization using open source
11169software is how to maintain compliance with various open source
11170licensing during the lifecycle of the product. While this section does
11171not provide legal advice or comprehensively cover all scenarios, it does
11172present methods that you can use to assist you in meeting the compliance
11173requirements during a software release.
11174
11175With hundreds of different open source licenses that the Yocto Project
11176tracks, it is difficult to know the requirements of each and every
11177license. However, the requirements of the major FLOSS licenses can begin
11178to be covered by assuming that there are three main areas of concern:
11179
11180- Source code must be provided.
11181
11182- License text for the software must be provided.
11183
11184- Compilation scripts and modifications to the source code must be
11185 provided.
11186
11187There are other requirements beyond the scope of these three and the
11188methods described in this section (e.g. the mechanism through which
11189source code is distributed).
11190
11191As different organizations have different methods of complying with open
11192source licensing, this section is not meant to imply that there is only
11193one single way to meet your compliance obligations, but rather to
11194describe one method of achieving compliance. The remainder of this
11195section describes methods supported to meet the previously mentioned
11196three requirements. Once you take steps to meet these requirements, and
11197prior to releasing images, sources, and the build system, you should
11198audit all artifacts to ensure completeness.
11199
11200.. note::
11201
11202 The Yocto Project generates a license manifest during image creation
11203 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
11204 to assist with any audits.
11205
11206Providing the Source Code
11207~~~~~~~~~~~~~~~~~~~~~~~~~
11208
11209Compliance activities should begin before you generate the final image.
11210The first thing you should look at is the requirement that tops the list
11211for most compliance groups --- providing the source. The Yocto Project has
11212a few ways of meeting this requirement.
11213
11214One of the easiest ways to meet this requirement is to provide the
11215entire :term:`DL_DIR` used by the
11216build. This method, however, has a few issues. The most obvious is the
11217size of the directory since it includes all sources used in the build
11218and not just the source used in the released image. It will include
11219toolchain source, and other artifacts, which you would not generally
11220release. However, the more serious issue for most companies is
11221accidental release of proprietary software. The Yocto Project provides
11222an :ref:`archiver <ref-classes-archiver>` class to
11223help avoid some of these concerns.
11224
11225Before you employ :term:`DL_DIR` or the :ref:`archiver <ref-classes-archiver>` class, you need to
11226decide how you choose to provide source. The source :ref:`archiver <ref-classes-archiver>` class
11227can generate tarballs and SRPMs and can create them with various levels
11228of compliance in mind.
11229
11230One way of doing this (but certainly not the only way) is to release
11231just the source as a tarball. You can do this by adding the following to
11232the ``local.conf`` file found in the :term:`Build Directory`::
11233
11234 INHERIT += "archiver"
11235 ARCHIVER_MODE[src] = "original"
11236
11237During the creation of your
11238image, the source from all recipes that deploy packages to the image is
11239placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
11240:term:`LICENSE` for each recipe.
11241Releasing the entire directory enables you to comply with requirements
11242concerning providing the unmodified source. It is important to note that
11243the size of the directory can get large.
11244
11245A way to help mitigate the size issue is to only release tarballs for
11246licenses that require the release of source. Let us assume you are only
11247concerned with GPL code as identified by running the following script:
11248
11249.. code-block:: shell
11250
11251 # Script to archive a subset of packages matching specific license(s)
11252 # Source and license files are copied into sub folders of package folder
11253 # Must be run from build folder
11254 #!/bin/bash
11255 src_release_dir="source-release"
11256 mkdir -p $src_release_dir
11257 for a in tmp/deploy/sources/*; do
11258 for d in $a/*; do
11259 # Get package name from path
11260 p=`basename $d`
11261 p=${p%-*}
11262 p=${p%-*}
11263 # Only archive GPL packages (update *GPL* regex for your license check)
11264 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
11265 if [ $numfiles -ge 1 ]; then
11266 echo Archiving $p
11267 mkdir -p $src_release_dir/$p/source
11268 cp $d/* $src_release_dir/$p/source 2> /dev/null
11269 mkdir -p $src_release_dir/$p/license
11270 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
11271 fi
11272 done
11273 done
11274
11275At this point, you
11276could create a tarball from the ``gpl_source_release`` directory and
11277provide that to the end user. This method would be a step toward
11278achieving compliance with section 3a of GPLv2 and with section 6 of
11279GPLv3.
11280
11281Providing License Text
11282~~~~~~~~~~~~~~~~~~~~~~
11283
11284One requirement that is often overlooked is inclusion of license text.
11285This requirement also needs to be dealt with prior to generating the
11286final image. Some licenses require the license text to accompany the
11287binary. You can achieve this by adding the following to your
11288``local.conf`` file::
11289
11290 COPY_LIC_MANIFEST = "1"
11291 COPY_LIC_DIRS = "1"
11292 LICENSE_CREATE_PACKAGE = "1"
11293
11294Adding these statements to the
11295configuration file ensures that the licenses collected during package
11296generation are included on your image.
11297
11298.. note::
11299
11300 Setting all three variables to "1" results in the image having two
11301 copies of the same license file. One copy resides in
11302 ``/usr/share/common-licenses`` and the other resides in
11303 ``/usr/share/license``.
11304
11305 The reason for this behavior is because
11306 :term:`COPY_LIC_DIRS` and
11307 :term:`COPY_LIC_MANIFEST`
11308 add a copy of the license when the image is built but do not offer a
11309 path for adding licenses for newly installed packages to an image.
11310 :term:`LICENSE_CREATE_PACKAGE`
11311 adds a separate package and an upgrade path for adding licenses to an
11312 image.
11313
11314As the source :ref:`archiver <ref-classes-archiver>` class has already archived the original
11315unmodified source that contains the license files, you would have
11316already met the requirements for inclusion of the license information
11317with source as defined by the GPL and other open source licenses.
11318
11319Providing Compilation Scripts and Source Code Modifications
11320~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11321
11322At this point, we have addressed all we need to prior to generating the
11323image. The next two requirements are addressed during the final
11324packaging of the release.
11325
11326By releasing the version of the OpenEmbedded build system and the layers
11327used during the build, you will be providing both compilation scripts
11328and the source code modifications in one step.
11329
11330If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
11331and a distro layer, and those
11332those layers are used to patch, compile, package, or modify (in any way)
11333any open source software included in your released images, you might be
11334required to release those layers under section 3 of GPLv2 or section 1
11335of GPLv3. One way of doing that is with a clean checkout of the version
11336of the Yocto Project and layers used during your build. Here is an
11337example:
11338
11339.. code-block:: shell
11340
11341 # We built using the dunfell branch of the poky repo
11342 $ git clone -b dunfell git://git.yoctoproject.org/poky
11343 $ cd poky
11344 # We built using the release_branch for our layers
11345 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11346 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11347 # clean up the .git repos
11348 $ find . -name ".git" -type d -exec rm -rf {} \;
11349
11350One thing a development organization might want to consider for end-user
11351convenience is to modify
11352``meta-poky/conf/templates/default/bblayers.conf.sample`` to ensure that when
11353the end user utilizes the released build system to build an image, the
11354development organization's layers are included in the ``bblayers.conf`` file
11355automatically::
11356
11357 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11358 # changes incompatibly
11359 POKY_BBLAYERS_CONF_VERSION = "2"
11360
11361 BBPATH = "${TOPDIR}"
11362 BBFILES ?= ""
11363
11364 BBLAYERS ?= " \
11365 ##OEROOT##/meta \
11366 ##OEROOT##/meta-poky \
11367 ##OEROOT##/meta-yocto-bsp \
11368 ##OEROOT##/meta-mylayer \
11369 "
11370
11371Creating and
11372providing an archive of the :term:`Metadata`
11373layers (recipes, configuration files, and so forth) enables you to meet
11374your requirements to include the scripts to control compilation as well
11375as any modifications to the original source.
11376
11377Compliance Limitations with Executables Built from Static Libraries
11378~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11379
11380When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS`
11381mechanisms as well as explicitly included in the image recipe with
11382:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B
11383(``DEPENDS += "B"``), package B will neither appear in the generated license
11384manifest nor in the generated source tarballs. This occurs as the
11385:ref:`license <ref-classes-license>` and :ref:`archiver <ref-classes-archiver>`
11386classes assume that only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS`
11387end up in the image.
11388
11389As a result, potential obligations regarding license compliance for package B
11390may not be met.
11391
11392The Yocto Project doesn't enable static libraries by default, in part because
11393of this issue. Before a solution to this limitation is found, you need to
11394keep in mind that if your root filesystem is built from static libraries,
11395you will need to manually ensure that your deliveries are compliant
11396with the licenses of these libraries.
11397
11398Copying Non Standard Licenses
11399-----------------------------
11400
11401Some packages, such as the linux-firmware package, have many licenses
11402that are not in any way common. You can avoid adding a lot of these
11403types of common license files, which are only applicable to a specific
11404package, by using the
11405:term:`NO_GENERIC_LICENSE`
11406variable. Using this variable also avoids QA errors when you use a
11407non-common, non-CLOSED license in a recipe.
11408
11409Here is an example that uses the ``LICENSE.Abilis.txt`` file as
11410the license from the fetched source::
11411
11412 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11413
11414Checking for Vulnerabilities
11415============================
11416
11417Vulnerabilities in Poky and OE-Core
11418-----------------------------------
11419
11420The Yocto Project has an infrastructure to track and address unfixed
11421known security vulnerabilities, as tracked by the public
11422:wikipedia:`Common Vulnerabilities and Exposures (CVE) <Common_Vulnerabilities_and_Exposures>`
11423database.
11424
11425The Yocto Project maintains a `list of known vulnerabilities
11426<https://autobuilder.yocto.io/pub/non-release/patchmetrics/>`__
11427for packages in Poky and OE-Core, tracking the evolution of the number of
11428unpatched CVEs and the status of patches. Such information is available for
11429the current development version and for each supported release.
11430
11431Security is a process, not a product, and thus at any time, a number of security
11432issues may be impacting Poky and OE-Core. It is up to the maintainers, users,
11433contributors and anyone interested in the issues to investigate and possibly fix them by
11434updating software components to newer versions or by applying patches to address them.
11435It is recommended to work with Poky and OE-Core upstream maintainers and submit
11436patches to fix them, see ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" for details.
11437
11438Vulnerability check at build time
11439---------------------------------
11440
11441To enable a check for CVE security vulnerabilities using :ref:`cve-check <ref-classes-cve-check>` in the specific image
11442or target you are building, add the following setting to your configuration::
11443
11444 INHERIT += "cve-check"
11445
11446The CVE database contains some old incomplete entries which have been
11447deemed not to impact Poky or OE-Core. These CVE entries can be excluded from the
11448check using build configuration::
11449
11450 include conf/distro/include/cve-extra-exclusions.inc
11451
11452With this CVE check enabled, BitBake build will try to map each compiled software component
11453recipe name and version information to the CVE database and generate recipe and
11454image specific reports. These reports will contain:
11455
11456- metadata about the software component like names and versions
11457
11458- metadata about the CVE issue such as description and NVD link
11459
11460- for each software component, a list of CVEs which are possibly impacting this version
11461
11462- status of each CVE: ``Patched``, ``Unpatched`` or ``Ignored``
11463
11464The status ``Patched`` means that a patch file to address the security issue has been
11465applied. ``Unpatched`` status means that no patches to address the issue have been
11466applied and that the issue needs to be investigated. ``Ignored`` means that after
11467analysis, it has been deemed to ignore the issue as it for example affects
11468the software component on a different operating system platform.
11469
11470After a build with CVE check enabled, reports for each compiled source recipe will be
11471found in ``build/tmp/deploy/cve``.
11472
11473For example the CVE check report for the ``flex-native`` recipe looks like::
11474
11475 $ cat poky/build/tmp/deploy/cve/flex-native
11476 LAYER: meta
11477 PACKAGE NAME: flex-native
11478 PACKAGE VERSION: 2.6.4
11479 CVE: CVE-2016-6354
11480 CVE STATUS: Patched
11481 CVE SUMMARY: Heap-based buffer overflow in the yy_get_next_buffer function in Flex before 2.6.1 might allow context-dependent attackers to cause a denial of service or possibly execute arbitrary code via vectors involving num_to_read.
11482 CVSS v2 BASE SCORE: 7.5
11483 CVSS v3 BASE SCORE: 9.8
11484 VECTOR: NETWORK
11485 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2016-6354
11486
11487 LAYER: meta
11488 PACKAGE NAME: flex-native
11489 PACKAGE VERSION: 2.6.4
11490 CVE: CVE-2019-6293
11491 CVE STATUS: Ignored
11492 CVE SUMMARY: An issue was discovered in the function mark_beginning_as_normal in nfa.c in flex 2.6.4. There is a stack exhaustion problem caused by the mark_beginning_as_normal function making recursive calls to itself in certain scenarios involving lots of '*' characters. Remote attackers could leverage this vulnerability to cause a denial-of-service.
11493 CVSS v2 BASE SCORE: 4.3
11494 CVSS v3 BASE SCORE: 5.5
11495 VECTOR: NETWORK
11496 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2019-6293
11497
11498For images, a summary of all recipes included in the image and their CVEs is also
11499generated in textual and JSON formats. These ``.cve`` and ``.json`` reports can be found
11500in the ``tmp/deploy/images`` directory for each compiled image.
11501
11502At build time CVE check will also throw warnings about ``Unpatched`` CVEs::
11503
11504 WARNING: flex-2.6.4-r0 do_cve_check: Found unpatched CVE (CVE-2019-6293), for more information check /poky/build/tmp/work/core2-64-poky-linux/flex/2.6.4-r0/temp/cve.log
11505 WARNING: libarchive-3.5.1-r0 do_cve_check: Found unpatched CVE (CVE-2021-36976), for more information check /poky/build/tmp/work/core2-64-poky-linux/libarchive/3.5.1-r0/temp/cve.log
11506
11507It is also possible to check the CVE status of individual packages as follows::
11508
11509 bitbake -c cve_check flex libarchive
11510
11511Fixing CVE product name and version mappings
11512--------------------------------------------
11513
11514By default, :ref:`cve-check <ref-classes-cve-check>` uses the recipe name :term:`BPN` as CVE
11515product name when querying the CVE database. If this mapping contains false positives, e.g.
11516some reported CVEs are not for the software component in question, or false negatives like
11517some CVEs are not found to impact the recipe when they should, then the problems can be
11518in the recipe name to CVE product mapping. These mapping issues can be fixed by setting
11519the :term:`CVE_PRODUCT` variable inside the recipe. This defines the name of the software component in the
11520upstream `NIST CVE database <https://nvd.nist.gov/>`__.
11521
11522The variable supports using vendor and product names like this::
11523
11524 CVE_PRODUCT = "flex_project:flex"
11525
11526In this example the vendor name used in the CVE database is ``flex_project`` and the
11527product is ``flex``. With this setting the ``flex`` recipe only maps to this specific
11528product and not products from other vendors with same name ``flex``.
11529
11530Similarly, when the recipe version :term:`PV` is not compatible with software versions used by
11531the upstream software component releases and the CVE database, these can be fixed using
11532the :term:`CVE_VERSION` variable.
11533
11534Note that if the CVE entries in the NVD database contain bugs or have missing or incomplete
11535information, it is recommended to fix the information there directly instead of working
11536around the issues possibly for a long time in Poky and OE-Core side recipes. Feedback to
11537NVD about CVE entries can be provided through the `NVD contact form <https://nvd.nist.gov/info/contact-form>`__.
11538
11539Fixing vulnerabilities in recipes
11540---------------------------------
11541
11542If a CVE security issue impacts a software component, it can be fixed by updating to a newer
11543version of the software component or by applying a patch. For Poky and OE-Core master branches, updating
11544to a newer software component release with fixes is the best option, but patches can be applied
11545if releases are not yet available.
11546
11547For stable branches, it is preferred to apply patches for the issues. For some software
11548components minor version updates can also be applied if they are backwards compatible.
11549
11550Here is an example of fixing CVE security issues with patch files,
11551an example from the :oe_layerindex:`ffmpeg recipe</layerindex/recipe/47350>`::
11552
11553 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
11554 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \
11555 file://fix-CVE-2020-20446.patch \
11556 file://fix-CVE-2020-20453.patch \
11557 file://fix-CVE-2020-22015.patch \
11558 file://fix-CVE-2020-22021.patch \
11559 file://fix-CVE-2020-22033-CVE-2020-22019.patch \
11560 file://fix-CVE-2021-33815.patch \
11561
11562A good practice is to include the CVE identifier in both the patch file name
11563and inside the patch file commit message using the format::
11564
11565 CVE: CVE-2020-22033
11566
11567CVE checker will then capture this information and change the CVE status to ``Patched``
11568in the generated reports.
11569
11570If analysis shows that the CVE issue does not impact the recipe due to configuration, platform,
11571version or other reasons, the CVE can be marked as ``Ignored`` using the :term:`CVE_CHECK_IGNORE` variable.
11572As mentioned previously, if data in the CVE database is wrong, it is recommend to fix those
11573issues in the CVE database directly.
11574
11575Recipes can be completely skipped by CVE check by including the recipe name in
11576the :term:`CVE_CHECK_SKIP_RECIPE` variable.
11577
11578Implementation details
11579----------------------
11580
11581Here's what the :ref:`cve-check <ref-classes-cve-check>` class does to
11582find unpatched CVE IDs.
11583
11584First the code goes through each patch file provided by a recipe. If a valid CVE ID
11585is found in the name of the file, the corresponding CVE is considered as patched.
11586Don't forget that if multiple CVE IDs are found in the filename, only the last
11587one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch
11588file. The found CVE IDs are also considered as patched.
11589
11590Then, the code looks up all the CVE IDs in the NIST database for all the
11591products defined in :term:`CVE_PRODUCT`. Then, for each found CVE:
11592
11593- If the package name (:term:`PN`) is part of
11594 :term:`CVE_CHECK_SKIP_RECIPE`, it is considered as ``Patched``.
11595
11596- If the CVE ID is part of :term:`CVE_CHECK_IGNORE`, it is
11597 set as ``Ignored``.
11598
11599- If the CVE ID is part of the patched CVE for the recipe, it is
11600 already considered as ``Patched``.
11601
11602- Otherwise, the code checks whether the recipe version (:term:`PV`)
11603 is within the range of versions impacted by the CVE. If so, the CVE
11604 is considered as ``Unpatched``.
11605
11606The CVE database is stored in :term:`DL_DIR` and can be inspected using
11607``sqlite3`` command as follows::
11608
11609 sqlite3 downloads/CVE_CHECK/nvdcve_1.1.db .dump | grep CVE-2021-37462
11610
11611When analyzing CVEs, it is recommended to:
11612
11613- study the latest information in `CVE database <https://nvd.nist.gov/vuln/search>`__.
11614
11615- check how upstream developers of the software component addressed the issue, e.g.
11616 what patch was applied, which upstream release contains the fix.
11617
11618- check what other Linux distributions like `Debian <https://security-tracker.debian.org/tracker/>`__
11619 did to analyze and address the issue.
11620
11621- follow security notices from other Linux distributions.
11622
11623- follow public `open source security mailing lists <https://oss-security.openwall.org/wiki/mailing-lists>`__ for
11624 discussions and advance notifications of CVE bugs and software releases with fixes.
11625
11626Creating a Software Bill of Materials
11627=====================================
11628
11629Once you are able to build an image for your project, once the licenses for
11630each software component are all identified (see
11631":ref:`dev-manual/common-tasks:working with licenses`") and once vulnerability
11632fixes are applied (see ":ref:`dev-manual/common-tasks:checking
11633for vulnerabilities`"), the OpenEmbedded build system can generate
11634a description of all the components you used, their licenses, their dependencies,
11635the changes that were applied and the known vulnerabilities that were fixed.
11636
11637This description is generated in the form of a *Software Bill of Materials*
11638(:term:`SBOM`), using the :term:`SPDX` standard.
11639
11640When you release software, this is the most standard way to provide information
11641about the Software Supply Chain of your software image and SDK. The
11642:term:`SBOM` tooling is often used to ensure open source license compliance by
11643providing the license texts used in the product which legal departments and end
11644users can read in standardized format.
11645
11646:term:`SBOM` information is also critical to performing vulnerability exposure
11647assessments, as all the components used in the Software Supply Chain are listed.
11648
11649The OpenEmbedded build system doesn't generate such information by default.
11650To make this happen, you must inherit the
11651:ref:`create-spdx <ref-classes-create-spdx>` class from a configuration file::
11652
11653 INHERIT += "create-spdx"
11654
11655You then get :term:`SPDX` output in JSON format as an
11656``IMAGE-MACHINE.spdx.json`` file in ``tmp/deploy/images/MACHINE/`` inside the
11657:term:`Build Directory`.
11658
11659This is a toplevel file accompanied by an ``IMAGE-MACHINE.spdx.index.json``
11660containing an index of JSON :term:`SPDX` files for individual recipes, together
11661with an ``IMAGE-MACHINE.spdx.tar.zst`` compressed archive containing all such
11662files.
11663
11664The :ref:`create-spdx <ref-classes-create-spdx>` class offers options to include
11665more information in the output :term:`SPDX` data, such as making the generated
11666files more human readable (:term:`SPDX_PRETTY`), adding compressed archives of
11667the files in the generated target packages (:term:`SPDX_ARCHIVE_PACKAGED`),
11668adding a description of the source files handled by the target recipes
11669(:term:`SPDX_INCLUDE_SOURCES`) and adding archives of these source files
11670themselves (:term:`SPDX_ARCHIVE_SOURCES`).
11671
11672Though the toplevel :term:`SPDX` output is available in
11673``tmp/deploy/images/MACHINE/`` inside the :term:`Build Directory`, ancillary
11674generated files are available in ``tmp/deploy/spdx/MACHINE`` too, such as:
11675
11676- The individual :term:`SPDX` JSON files in the ``IMAGE-MACHINE.spdx.tar.zst``
11677 archive.
11678
11679- Compressed archives of the files in the generated target packages,
11680 in ``packages/packagename.tar.zst`` (when :term:`SPDX_ARCHIVE_PACKAGED`
11681 is set).
11682
11683- Compressed archives of the source files used to build the host tools
11684 and the target packages in ``recipes/recipe-packagename.tar.zst``
11685 (when :term:`SPDX_ARCHIVE_SOURCES` is set). Those are needed to fulfill
11686 "source code access" license requirements.
11687
11688See the `tools page <https://spdx.dev/resources/tools/>`__ on the :term:`SPDX`
11689project website for a list of tools to consume and transform the :term:`SPDX`
11690data generated by the OpenEmbedded build system.
11691
11692Using the Error Reporting Tool
11693==============================
11694
11695The error reporting tool allows you to submit errors encountered during
11696builds to a central database. Outside of the build environment, you can
11697use a web interface to browse errors, view statistics, and query for
11698errors. The tool works using a client-server system where the client
11699portion is integrated with the installed Yocto Project
11700:term:`Source Directory` (e.g. ``poky``).
11701The server receives the information collected and saves it in a
11702database.
11703
11704There is a live instance of the error reporting server at
11705https://errors.yoctoproject.org.
11706When you want to get help with build failures, you can submit all of the
11707information on the failure easily and then point to the URL in your bug
11708report or send an email to the mailing list.
11709
11710.. note::
11711
11712 If you send error reports to this server, the reports become publicly
11713 visible.
11714
11715Enabling and Using the Tool
11716---------------------------
11717
11718By default, the error reporting tool is disabled. You can enable it by
11719inheriting the :ref:`report-error <ref-classes-report-error>`
11720class by adding the following statement to the end of your
11721``local.conf`` file in your :term:`Build Directory`::
11722
11723 INHERIT += "report-error"
11724
11725By default, the error reporting feature stores information in
11726``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11727However, you can specify a directory to use by adding the following to
11728your ``local.conf`` file::
11729
11730 ERR_REPORT_DIR = "path"
11731
11732Enabling error
11733reporting causes the build process to collect the errors and store them
11734in a file as previously described. When the build system encounters an
11735error, it includes a command as part of the console output. You can run
11736the command to send the error file to the server. For example, the
11737following command sends the errors to an upstream server::
11738
11739 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11740
11741In the previous example, the errors are sent to a public database
11742available at https://errors.yoctoproject.org, which is used by the
11743entire community. If you specify a particular server, you can send the
11744errors to a different database. Use the following command for more
11745information on available options::
11746
11747 $ send-error-report --help
11748
11749When sending the error file, you are prompted to review the data being
11750sent as well as to provide a name and optional email address. Once you
11751satisfy these prompts, the command returns a link from the server that
11752corresponds to your entry in the database. For example, here is a
11753typical link: https://errors.yoctoproject.org/Errors/Details/9522/
11754
11755Following the link takes you to a web interface where you can browse,
11756query the errors, and view statistics.
11757
11758Disabling the Tool
11759------------------
11760
11761To disable the error reporting feature, simply remove or comment out the
11762following statement from the end of your ``local.conf`` file in your
11763:term:`Build Directory`.
11764::
11765
11766 INHERIT += "report-error"
11767
11768Setting Up Your Own Error Reporting Server
11769------------------------------------------
11770
11771If you want to set up your own error reporting server, you can obtain
11772the code from the Git repository at :yocto_git:`/error-report-web/`.
11773Instructions on how to set it up are in the README document.
11774
11775Using Wayland and Weston
11776========================
11777
11778:wikipedia:`Wayland <Wayland_(display_server_protocol)>`
11779is a computer display server protocol that provides a method for
11780compositing window managers to communicate directly with applications
11781and video hardware and expects them to communicate with input hardware
11782using other libraries. Using Wayland with supporting targets can result
11783in better control over graphics frame rendering than an application
11784might otherwise achieve.
11785
11786The Yocto Project provides the Wayland protocol libraries and the
11787reference :wikipedia:`Weston <Wayland_(display_server_protocol)#Weston>`
11788compositor as part of its release. You can find the integrated packages
11789in the ``meta`` layer of the :term:`Source Directory`.
11790Specifically, you
11791can find the recipes that build both Wayland and Weston at
11792``meta/recipes-graphics/wayland``.
11793
11794You can build both the Wayland and Weston packages for use only with targets
11795that accept the :wikipedia:`Mesa 3D and Direct Rendering Infrastructure
11796<Mesa_(computer_graphics)>`, which is also known as Mesa DRI. This implies that
11797you cannot build and use the packages if your target uses, for example, the
11798Intel Embedded Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11799
11800.. note::
11801
11802 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11803 emulated QEMU hardware. However, this version of Weston will run
11804 under X emulation without issues.
11805
11806This section describes what you need to do to implement Wayland and use
11807the Weston compositor when building an image for a supporting target.
11808
11809Enabling Wayland in an Image
11810----------------------------
11811
11812To enable Wayland, you need to enable it to be built and enable it to be
11813included (installed) in the image.
11814
11815Building Wayland
11816~~~~~~~~~~~~~~~~
11817
11818To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11819Wayland with Kernel Mode Setting
11820(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11821support, include the "wayland" flag in the
11822:term:`DISTRO_FEATURES`
11823statement in your ``local.conf`` file::
11824
11825 DISTRO_FEATURES:append = " wayland"
11826
11827.. note::
11828
11829 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11830 support
11831
11832Installing Wayland and Weston
11833~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11834
11835To install the Wayland feature into an image, you must include the
11836following
11837:term:`CORE_IMAGE_EXTRA_INSTALL`
11838statement in your ``local.conf`` file::
11839
11840 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11841
11842Running Weston
11843--------------
11844
11845To run Weston inside X11, enabling it as described earlier and building
11846a Sato image is sufficient. If you are running your image under Sato, a
11847Weston Launcher appears in the "Utility" category.
11848
11849Alternatively, you can run Weston through the command-line interpretor
11850(CLI), which is better suited for development work. To run Weston under
11851the CLI, you need to do the following after your image is built:
11852
118531. Run these commands to export ``XDG_RUNTIME_DIR``::
11854
11855 mkdir -p /tmp/$USER-weston
11856 chmod 0700 /tmp/$USER-weston
11857 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11858
118592. Launch Weston in the shell::
11860
11861 weston
diff --git a/documentation/dev-manual/custom-distribution.rst b/documentation/dev-manual/custom-distribution.rst
new file mode 100644
index 0000000000..e5b1ad777a
--- /dev/null
+++ b/documentation/dev-manual/custom-distribution.rst
@@ -0,0 +1,108 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating Your Own Distribution
4******************************
5
6When you build an image using the Yocto Project and do not alter any
7distribution :term:`Metadata`, you are
8creating a Poky distribution. If you wish to gain more control over
9package alternative selections, compile-time options, and other
10low-level configurations, you can create your own distribution.
11
12To create your own distribution, the basic steps consist of creating
13your own distribution layer, creating your own distribution
14configuration file, and then adding any needed code and Metadata to the
15layer. The following steps provide some more detail:
16
17- *Create a layer for your new distro:* Create your distribution layer
18 so that you can keep your Metadata and code for the distribution
19 separate. It is strongly recommended that you create and use your own
20 layer for configuration and code. Using your own layer as compared to
21 just placing configurations in a ``local.conf`` configuration file
22 makes it easier to reproduce the same build configuration when using
23 multiple build machines. See the
24 ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
25 section for information on how to quickly set up a layer.
26
27- *Create the distribution configuration file:* The distribution
28 configuration file needs to be created in the ``conf/distro``
29 directory of your layer. You need to name it using your distribution
30 name (e.g. ``mydistro.conf``).
31
32 .. note::
33
34 The :term:`DISTRO` variable in your ``local.conf`` file determines the
35 name of your distribution.
36
37 You can split out parts of your configuration file into include files
38 and then "require" them from within your distribution configuration
39 file. Be sure to place the include files in the
40 ``conf/distro/include`` directory of your layer. A common example
41 usage of include files would be to separate out the selection of
42 desired version and revisions for individual recipes.
43
44 Your configuration file needs to set the following required
45 variables:
46
47 - :term:`DISTRO_NAME`
48
49 - :term:`DISTRO_VERSION`
50
51 These following variables are optional and you typically set them
52 from the distribution configuration file:
53
54 - :term:`DISTRO_FEATURES`
55
56 - :term:`DISTRO_EXTRA_RDEPENDS`
57
58 - :term:`DISTRO_EXTRA_RRECOMMENDS`
59
60 - :term:`TCLIBC`
61
62 .. tip::
63
64 If you want to base your distribution configuration file on the
65 very basic configuration from OE-Core, you can use
66 ``conf/distro/defaultsetup.conf`` as a reference and just include
67 variables that differ as compared to ``defaultsetup.conf``.
68 Alternatively, you can create a distribution configuration file
69 from scratch using the ``defaultsetup.conf`` file or configuration files
70 from another distribution such as Poky as a reference.
71
72- *Provide miscellaneous variables:* Be sure to define any other
73 variables for which you want to create a default or enforce as part
74 of the distribution configuration. You can include nearly any
75 variable from the ``local.conf`` file. The variables you use are not
76 limited to the list in the previous bulleted item.
77
78- *Point to Your distribution configuration file:* In your ``local.conf``
79 file in the :term:`Build Directory`, set your :term:`DISTRO` variable to
80 point to your distribution's configuration file. For example, if your
81 distribution's configuration file is named ``mydistro.conf``, then
82 you point to it as follows::
83
84 DISTRO = "mydistro"
85
86- *Add more to the layer if necessary:* Use your layer to hold other
87 information needed for the distribution:
88
89 - Add recipes for installing distro-specific configuration files
90 that are not already installed by another recipe. If you have
91 distro-specific configuration files that are included by an
92 existing recipe, you should add an append file (``.bbappend``) for
93 those. For general information and recommendations on how to add
94 recipes to your layer, see the
95 ":ref:`dev-manual/layers:creating your own layer`" and
96 ":ref:`dev-manual/layers:following best practices when creating layers`"
97 sections.
98
99 - Add any image recipes that are specific to your distribution.
100
101 - Add a ``psplash`` append file for a branded splash screen. For
102 information on append files, see the
103 ":ref:`dev-manual/layers:appending other layers metadata with your layer`"
104 section.
105
106 - Add any other append files to make custom changes that are
107 specific to individual recipes.
108
diff --git a/documentation/dev-manual/custom-template-configuration-directory.rst b/documentation/dev-manual/custom-template-configuration-directory.rst
new file mode 100644
index 0000000000..9bffef35b4
--- /dev/null
+++ b/documentation/dev-manual/custom-template-configuration-directory.rst
@@ -0,0 +1,52 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Custom Template Configuration Directory
4**************************************************
5
6If you are producing your own customized version of the build system for
7use by other users, you might want to provide a custom build configuration
8that includes all the necessary settings and layers (i.e. ``local.conf`` and
9``bblayers.conf`` that are created in a new :term:`Build Directory`) and a custom
10message that is shown when setting up the build. This can be done by
11creating one or more template configuration directories in your
12custom distribution layer.
13
14This can be done by using ``bitbake-layers save-build-conf``::
15
16 $ bitbake-layers save-build-conf ../../meta-alex/ test-1
17 NOTE: Starting bitbake server...
18 NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
19 Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
20 You can try out the configuration with
21 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
22
23The above command takes the config files from the currently active :term:`Build Directory` under ``conf``,
24replaces site-specific paths in ``bblayers.conf`` with ``##OECORE##``-relative paths, and copies
25the config files into a specified layer under a specified template name.
26
27To use those saved templates as a starting point for a build, users should point
28to one of them with :term:`TEMPLATECONF` environment variable::
29
30 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
31
32The OpenEmbedded build system uses the environment variable
33:term:`TEMPLATECONF` to locate the directory from which it gathers
34configuration information that ultimately ends up in the
35:term:`Build Directory` ``conf`` directory.
36
37If :term:`TEMPLATECONF` is not set, the default value is obtained
38from ``.templateconf`` file that is read from the same directory as
39``oe-init-build-env`` script. For the Poky reference distribution this
40would be::
41
42 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf/templates/default}
43
44If you look at a configuration template directory, you will
45see the ``bblayers.conf.sample``, ``local.conf.sample``, and
46``conf-notes.txt`` files. The build system uses these files to form the
47respective ``bblayers.conf`` file, ``local.conf`` file, and show
48users a note about the build they're setting up
49when running the ``oe-init-build-env`` setup script. These can be
50edited further if needed to improve or change the build configurations
51available to the users.
52
diff --git a/documentation/dev-manual/customizing-images.rst b/documentation/dev-manual/customizing-images.rst
new file mode 100644
index 0000000000..5b18958ade
--- /dev/null
+++ b/documentation/dev-manual/customizing-images.rst
@@ -0,0 +1,223 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Customizing Images
4******************
5
6You can customize images to satisfy particular requirements. This
7section describes several methods and provides guidelines for each.
8
9Customizing Images Using ``local.conf``
10=======================================
11
12Probably the easiest way to customize an image is to add a package by
13way of the ``local.conf`` configuration file. Because it is limited to
14local use, this method generally only allows you to add packages and is
15not as flexible as creating your own customized image. When you add
16packages using local variables this way, you need to realize that these
17variable changes are in effect for every build and consequently affect
18all images, which might not be what you require.
19
20To add a package to your image using the local configuration file, use
21the :term:`IMAGE_INSTALL` variable with the ``:append`` operator::
22
23 IMAGE_INSTALL:append = " strace"
24
25Use of the syntax is important; specifically, the leading space
26after the opening quote and before the package name, which is
27``strace`` in this example. This space is required since the ``:append``
28operator does not add the space.
29
30Furthermore, you must use ``:append`` instead of the ``+=`` operator if
31you want to avoid ordering issues. The reason for this is because doing
32so unconditionally appends to the variable and avoids ordering problems
33due to the variable being set in image recipes and ``.bbclass`` files
34with operators like ``?=``. Using ``:append`` ensures the operation
35takes effect.
36
37As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all
38images. It is possible to extend the syntax so that the variable applies
39to a specific image only. Here is an example::
40
41 IMAGE_INSTALL:append:pn-core-image-minimal = " strace"
42
43This example adds ``strace`` to the ``core-image-minimal`` image only.
44
45You can add packages using a similar approach through the
46:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
47``core-image-*`` images are affected.
48
49Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
50===============================================================================
51
52Another method for customizing your image is to enable or disable
53high-level image features by using the
54:term:`IMAGE_FEATURES` and
55:term:`EXTRA_IMAGE_FEATURES`
56variables. Although the functions for both variables are nearly
57equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
58a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
59``local.conf`` file, which is found in the :term:`Build Directory`.
60
61To understand how these features work, the best reference is
62:ref:`meta/classes-recipe/image.bbclass <ref-classes-image>`.
63This class lists out the available
64:term:`IMAGE_FEATURES` of which most map to package groups while some, such
65as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
66configuration settings.
67
68In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
69variable and then maps or configures the feature accordingly. Based on
70this information, the build system automatically adds the appropriate
71packages or configurations to the
72:term:`IMAGE_INSTALL` variable.
73Effectively, you are enabling extra features by extending the class or
74creating a custom class for use with specialized image ``.bb`` files.
75
76Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
77configuration file. Using a separate area from which to enable features
78with this variable helps you avoid overwriting the features in the image
79recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
80:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
81``meta/conf/bitbake.conf``.
82
83To illustrate how you can use these variables to modify your image,
84consider an example that selects the SSH server. The Yocto Project ships
85with two SSH servers you can use with your images: Dropbear and OpenSSH.
86Dropbear is a minimal SSH server appropriate for resource-constrained
87environments, while OpenSSH is a well-known standard SSH server
88implementation. By default, the ``core-image-sato`` image is configured
89to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
90images both include OpenSSH. The ``core-image-minimal`` image does not
91contain an SSH server.
92
93You can customize your image and change these defaults. Edit the
94:term:`IMAGE_FEATURES` variable in your recipe or use the
95:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
96configures the image you are working with to include
97``ssh-server-dropbear`` or ``ssh-server-openssh``.
98
99.. note::
100
101 See the ":ref:`ref-manual/features:image features`" section in the Yocto
102 Project Reference Manual for a complete list of image features that ship
103 with the Yocto Project.
104
105Customizing Images Using Custom .bb Files
106=========================================
107
108You can also customize an image by creating a custom recipe that defines
109additional software as part of the image. The following example shows
110the form for the two lines you need::
111
112 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
113 inherit core-image
114
115Defining the software using a custom recipe gives you total control over
116the contents of the image. It is important to use the correct names of
117packages in the :term:`IMAGE_INSTALL` variable. You must use the
118OpenEmbedded notation and not the Debian notation for the names (e.g.
119``glibc-dev`` instead of ``libc6-dev``).
120
121The other method for creating a custom image is to base it on an
122existing image. For example, if you want to create an image based on
123``core-image-sato`` but add the additional package ``strace`` to the
124image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
125``.bb`` and add the following line to the end of the copy::
126
127 IMAGE_INSTALL += "strace"
128
129Customizing Images Using Custom Package Groups
130==============================================
131
132For complex custom images, the best approach for customizing an image is
133to create a custom package group recipe that is used to build the image
134or images. A good example of a package group recipe is
135``meta/recipes-core/packagegroups/packagegroup-base.bb``.
136
137If you examine that recipe, you see that the :term:`PACKAGES` variable lists
138the package group packages to produce. The ``inherit packagegroup``
139statement sets appropriate default values and automatically adds
140``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
141package specified in the :term:`PACKAGES` statement.
142
143.. note::
144
145 The ``inherit packagegroup`` line should be located near the top of the
146 recipe, certainly before the :term:`PACKAGES` statement.
147
148For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
149and :term:`RRECOMMENDS` entries to provide a list of packages the parent
150task package should contain. You can see examples of these further down
151in the ``packagegroup-base.bb`` recipe.
152
153Here is a short, fabricated example showing the same basic pieces for a
154hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
155the variable :term:`PN` is the standard way to abbreviate the reference to
156the full packagegroup name ``packagegroup-custom``::
157
158 DESCRIPTION = "My Custom Package Groups"
159
160 inherit packagegroup
161
162 PACKAGES = "\
163 ${PN}-apps \
164 ${PN}-tools \
165 "
166
167 RDEPENDS:${PN}-apps = "\
168 dropbear \
169 portmap \
170 psplash"
171
172 RDEPENDS:${PN}-tools = "\
173 oprofile \
174 oprofileui-server \
175 lttng-tools"
176
177 RRECOMMENDS:${PN}-tools = "\
178 kernel-module-oprofile"
179
180In the previous example, two package group packages are created with
181their dependencies and their recommended package dependencies listed:
182``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
183build an image using these package group packages, you need to add
184``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
185:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
186areas of this section.
187
188Customizing an Image Hostname
189=============================
190
191By default, the configured hostname (i.e. ``/etc/hostname``) in an image
192is the same as the machine name. For example, if
193:term:`MACHINE` equals "qemux86", the
194configured hostname written to ``/etc/hostname`` is "qemux86".
195
196You can customize this name by altering the value of the "hostname"
197variable in the ``base-files`` recipe using either an append file or a
198configuration file. Use the following in an append file::
199
200 hostname = "myhostname"
201
202Use the following in a configuration file::
203
204 hostname:pn-base-files = "myhostname"
205
206Changing the default value of the variable "hostname" can be useful in
207certain situations. For example, suppose you need to do extensive
208testing on an image and you would like to easily identify the image
209under test from existing images with typical default hostnames. In this
210situation, you could change the default hostname to "testme", which
211results in all the images using the name "testme". Once testing is
212complete and you do not need to rebuild the image for test any longer,
213you can easily reset the default hostname.
214
215Another point of interest is that if you unset the variable, the image
216will have no default hostname in the filesystem. Here is an example that
217unsets the variable in a configuration file::
218
219 hostname:pn-base-files = ""
220
221Having no default hostname in the filesystem is suitable for
222environments that use dynamic hostnames such as virtual machines.
223
diff --git a/documentation/dev-manual/debugging.rst b/documentation/dev-manual/debugging.rst
new file mode 100644
index 0000000000..ef296de7ac
--- /dev/null
+++ b/documentation/dev-manual/debugging.rst
@@ -0,0 +1,1253 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Debugging Tools and Techniques
4******************************
5
6The exact method for debugging build failures depends on the nature of
7the problem and on the system's area from which the bug originates.
8Standard debugging practices such as comparison against the last known
9working version with examination of the changes and the re-application
10of steps to identify the one causing the problem are valid for the Yocto
11Project just as they are for any other system. Even though it is
12impossible to detail every possible potential failure, this section
13provides some general tips to aid in debugging given a variety of
14situations.
15
16.. note::
17
18 A useful feature for debugging is the error reporting tool.
19 Configuring the Yocto Project to use this tool causes the
20 OpenEmbedded build system to produce error reporting commands as part
21 of the console output. You can enter the commands after the build
22 completes to log error information into a common database, that can
23 help you figure out what might be going wrong. For information on how
24 to enable and use this feature, see the
25 ":ref:`dev-manual/error-reporting-tool:using the error reporting tool`"
26 section.
27
28The following list shows the debugging topics in the remainder of this
29section:
30
31- ":ref:`dev-manual/debugging:viewing logs from failed tasks`" describes
32 how to find and view logs from tasks that failed during the build
33 process.
34
35- ":ref:`dev-manual/debugging:viewing variable values`" describes how to
36 use the BitBake ``-e`` option to examine variable values after a
37 recipe has been parsed.
38
39- ":ref:`dev-manual/debugging:viewing package information with \`\`oe-pkgdata-util\`\``"
40 describes how to use the ``oe-pkgdata-util`` utility to query
41 :term:`PKGDATA_DIR` and
42 display package-related information for built packages.
43
44- ":ref:`dev-manual/debugging:viewing dependencies between recipes and tasks`"
45 describes how to use the BitBake ``-g`` option to display recipe
46 dependency information used during the build.
47
48- ":ref:`dev-manual/debugging:viewing task variable dependencies`" describes
49 how to use the ``bitbake-dumpsig`` command in conjunction with key
50 subdirectories in the :term:`Build Directory` to determine variable
51 dependencies.
52
53- ":ref:`dev-manual/debugging:running specific tasks`" describes
54 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
55 to run specific tasks in the build chain. It can be useful to run
56 tasks "out-of-order" when trying isolate build issues.
57
58- ":ref:`dev-manual/debugging:general BitBake problems`" describes how
59 to use BitBake's ``-D`` debug output option to reveal more about what
60 BitBake is doing during the build.
61
62- ":ref:`dev-manual/debugging:building with no dependencies`"
63 describes how to use the BitBake ``-b`` option to build a recipe
64 while ignoring dependencies.
65
66- ":ref:`dev-manual/debugging:recipe logging mechanisms`"
67 describes how to use the many recipe logging functions to produce
68 debugging output and report errors and warnings.
69
70- ":ref:`dev-manual/debugging:debugging parallel make races`"
71 describes how to debug situations where the build consists of several
72 parts that are run simultaneously and when the output or result of
73 one part is not ready for use with a different part of the build that
74 depends on that output.
75
76- ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) remotely`"
77 describes how to use GDB to allow you to examine running programs, which can
78 help you fix problems.
79
80- ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) on the target`"
81 describes how to use GDB directly on target hardware for debugging.
82
83- ":ref:`dev-manual/debugging:other debugging tips`" describes
84 miscellaneous debugging tips that can be useful.
85
86Viewing Logs from Failed Tasks
87==============================
88
89You can find the log for a task in the file
90``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
91For example, the log for the
92:ref:`ref-tasks-compile` task of the
93QEMU minimal image for the x86 machine (``qemux86``) might be in
94``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
95To see the commands :term:`BitBake` ran
96to generate a log, look at the corresponding ``run.do_``\ `taskname` file
97in the same directory.
98
99``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
100links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
101``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
102when it ran. The symlinks always point to the files corresponding to the
103most recent run.
104
105Viewing Variable Values
106=======================
107
108Sometimes you need to know the value of a variable as a result of
109BitBake's parsing step. This could be because some unexpected behavior
110occurred in your project. Perhaps an attempt to :ref:`modify a variable
111<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
112variables>` did not work out as expected.
113
114BitBake's ``-e`` option is used to display variable values after
115parsing. The following command displays the variable values after the
116configuration files (i.e. ``local.conf``, ``bblayers.conf``,
117``bitbake.conf`` and so forth) have been parsed::
118
119 $ bitbake -e
120
121The following command displays variable values after a specific recipe has
122been parsed. The variables include those from the configuration as well::
123
124 $ bitbake -e recipename
125
126.. note::
127
128 Each recipe has its own private set of variables (datastore).
129 Internally, after parsing the configuration, a copy of the resulting
130 datastore is made prior to parsing each recipe. This copying implies
131 that variables set in one recipe will not be visible to other
132 recipes.
133
134 Likewise, each task within a recipe gets a private datastore based on
135 the recipe datastore, which means that variables set within one task
136 will not be visible to other tasks.
137
138In the output of ``bitbake -e``, each variable is preceded by a
139description of how the variable got its value, including temporary
140values that were later overridden. This description also includes
141variable flags (varflags) set on the variable. The output can be very
142helpful during debugging.
143
144Variables that are exported to the environment are preceded by
145``export`` in the output of ``bitbake -e``. See the following example::
146
147 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
148
149In addition to variable values, the output of the ``bitbake -e`` and
150``bitbake -e`` recipe commands includes the following information:
151
152- The output starts with a tree listing all configuration files and
153 classes included globally, recursively listing the files they include
154 or inherit in turn. Much of the behavior of the OpenEmbedded build
155 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
156 implemented in the :ref:`base <ref-classes-base>` class and the
157 classes it inherits, rather than being built into BitBake itself.
158
159- After the variable values, all functions appear in the output. For
160 shell functions, variables referenced within the function body are
161 expanded. If a function has been modified using overrides or using
162 override-style operators like ``:append`` and ``:prepend``, then the
163 final assembled function body appears in the output.
164
165Viewing Package Information with ``oe-pkgdata-util``
166====================================================
167
168You can use the ``oe-pkgdata-util`` command-line utility to query
169:term:`PKGDATA_DIR` and display
170various package-related information. When you use the utility, you must
171use it to view information on packages that have already been built.
172
173Following are a few of the available ``oe-pkgdata-util`` subcommands.
174
175.. note::
176
177 You can use the standard \* and ? globbing wildcards as part of
178 package names and paths.
179
180- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
181 that have been built, optionally limiting the match to packages that
182 match pattern.
183
184- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
185 files and directories contained in the given packages.
186
187 .. note::
188
189 A different way to view the contents of a package is to look at
190 the
191 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
192 directory of the recipe that generates the package. This directory
193 is created by the
194 :ref:`ref-tasks-package` task
195 and has one subdirectory for each package the recipe generates,
196 which contains the files stored in that package.
197
198 If you want to inspect the ``${WORKDIR}/packages-split``
199 directory, make sure that
200 :ref:`rm_work <ref-classes-rm-work>` is not
201 enabled when you build the recipe.
202
203- ``oe-pkgdata-util find-path path ...``: Lists the names of
204 the packages that contain the given paths. For example, the following
205 tells us that ``/usr/share/man/man1/make.1`` is contained in the
206 ``make-doc`` package::
207
208 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
209 make-doc: /usr/share/man/man1/make.1
210
211- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
212 of the recipes that produce the given packages.
213
214For more information on the ``oe-pkgdata-util`` command, use the help
215facility::
216
217 $ oe-pkgdata-util --help
218 $ oe-pkgdata-util subcommand --help
219
220Viewing Dependencies Between Recipes and Tasks
221==============================================
222
223Sometimes it can be hard to see why BitBake wants to build other recipes
224before the one you have specified. Dependency information can help you
225understand why a recipe is built.
226
227To generate dependency information for a recipe, run the following
228command::
229
230 $ bitbake -g recipename
231
232This command writes the following files in the current directory:
233
234- ``pn-buildlist``: A list of recipes/targets involved in building
235 `recipename`. "Involved" here means that at least one task from the
236 recipe needs to run when building `recipename` from scratch. Targets
237 that are in
238 :term:`ASSUME_PROVIDED`
239 are not listed.
240
241- ``task-depends.dot``: A graph showing dependencies between tasks.
242
243The graphs are in :wikipedia:`DOT <DOT_%28graph_description_language%29>`
244format and can be converted to images (e.g. using the ``dot`` tool from
245`Graphviz <https://www.graphviz.org/>`__).
246
247.. note::
248
249 - DOT files use a plain text format. The graphs generated using the
250 ``bitbake -g`` command are often so large as to be difficult to
251 read without special pruning (e.g. with BitBake's ``-I`` option)
252 and processing. Despite the form and size of the graphs, the
253 corresponding ``.dot`` files can still be possible to read and
254 provide useful information.
255
256 As an example, the ``task-depends.dot`` file contains lines such
257 as the following::
258
259 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
260
261 The above example line reveals that the
262 :ref:`ref-tasks-configure`
263 task in ``libxslt`` depends on the
264 :ref:`ref-tasks-populate_sysroot`
265 task in ``libxml2``, which is a normal
266 :term:`DEPENDS` dependency
267 between the two recipes.
268
269 - For an example of how ``.dot`` files can be processed, see the
270 ``scripts/contrib/graph-tool`` Python script, which finds and
271 displays paths between graph nodes.
272
273You can use a different method to view dependency information by using
274the following command::
275
276 $ bitbake -g -u taskexp recipename
277
278This command
279displays a GUI window from which you can view build-time and runtime
280dependencies for the recipes involved in building recipename.
281
282Viewing Task Variable Dependencies
283==================================
284
285As mentioned in the
286":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
287User Manual, BitBake tries to automatically determine what variables a
288task depends on so that it can rerun the task if any values of the
289variables change. This determination is usually reliable. However, if
290you do things like construct variable names at runtime, then you might
291have to manually declare dependencies on those variables using
292``vardeps`` as described in the
293":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
294User Manual.
295
296If you are unsure whether a variable dependency is being picked up
297automatically for a given task, you can list the variable dependencies
298BitBake has determined by doing the following:
299
3001. Build the recipe containing the task::
301
302 $ bitbake recipename
303
3042. Inside the :term:`STAMPS_DIR`
305 directory, find the signature data (``sigdata``) file that
306 corresponds to the task. The ``sigdata`` files contain a pickled
307 Python database of all the metadata that went into creating the input
308 checksum for the task. As an example, for the
309 :ref:`ref-tasks-fetch` task of the
310 ``db`` recipe, the ``sigdata`` file might be found in the following
311 location::
312
313 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
314
315 For tasks that are accelerated through the shared state
316 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
317 additional ``siginfo`` file is written into
318 :term:`SSTATE_DIR` along with
319 the cached task output. The ``siginfo`` files contain exactly the
320 same information as ``sigdata`` files.
321
3223. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
323 is an example::
324
325 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
326
327 In the output of the above command, you will find a line like the
328 following, which lists all the (inferred) variable dependencies for
329 the task. This list also includes indirect dependencies from
330 variables depending on other variables, recursively.
331 ::
332
333 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
334
335 .. note::
336
337 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
338 These functions in turn depend on the variables they reference.
339
340 The output of ``bitbake-dumpsig`` also includes the value each
341 variable had, a list of dependencies for each variable, and
342 :term:`BB_BASEHASH_IGNORE_VARS`
343 information.
344
345There is also a ``bitbake-diffsigs`` command for comparing two
346``siginfo`` or ``sigdata`` files. This command can be helpful when
347trying to figure out what changed between two versions of a task. If you
348call ``bitbake-diffsigs`` with just one file, the command behaves like
349``bitbake-dumpsig``.
350
351You can also use BitBake to dump out the signature construction
352information without executing tasks by using either of the following
353BitBake command-line options::
354
355 ‐‐dump-signatures=SIGNATURE_HANDLER
356 -S SIGNATURE_HANDLER
357
358
359.. note::
360
361 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
362 dump only the signature or compare the dumped signature with the cached one,
363 respectively.
364
365Using BitBake with either of these options causes BitBake to dump out
366``sigdata`` files in the ``stamps`` directory for every task it would
367have executed instead of building the specified target package.
368
369Viewing Metadata Used to Create the Input Signature of a Shared State Task
370==========================================================================
371
372Seeing what metadata went into creating the input signature of a shared
373state (sstate) task can be a useful debugging aid. This information is
374available in signature information (``siginfo``) files in
375:term:`SSTATE_DIR`. For
376information on how to view and interpret information in ``siginfo``
377files, see the
378":ref:`dev-manual/debugging:viewing task variable dependencies`" section.
379
380For conceptual information on shared state, see the
381":ref:`overview-manual/concepts:shared state`"
382section in the Yocto Project Overview and Concepts Manual.
383
384Invalidating Shared State to Force a Task to Run
385================================================
386
387The OpenEmbedded build system uses
388:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
389:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
390rebuilding tasks. Collectively, this scheme is known as "shared state
391code".
392
393As with all schemes, this one has some drawbacks. It is possible that
394you could make implicit changes to your code that the checksum
395calculations do not take into account. These implicit changes affect a
396task's output but do not trigger the shared state code into rebuilding a
397recipe. Consider an example during which a tool changes its output.
398Assume that the output of ``rpmdeps`` changes. The result of the change
399should be that all the ``package`` and ``package_write_rpm`` shared
400state cache items become invalid. However, because the change to the
401output is external to the code and therefore implicit, the associated
402shared state cache items do not become invalidated. In this case, the
403build process uses the cached items rather than running the task again.
404Obviously, these types of implicit changes can cause problems.
405
406To avoid these problems during the build, you need to understand the
407effects of any changes you make. Realize that changes you make directly
408to a function are automatically factored into the checksum calculation.
409Thus, these explicit changes invalidate the associated area of shared
410state cache. However, you need to be aware of any implicit changes that
411are not obvious changes to the code and could affect the output of a
412given task.
413
414When you identify an implicit change, you can easily take steps to
415invalidate the cache and force the tasks to run. The steps you can take
416are as simple as changing a function's comments in the source code. For
417example, to invalidate package shared state files, change the comment
418statements of
419:ref:`ref-tasks-package` or the
420comments of one of the functions it calls. Even though the change is
421purely cosmetic, it causes the checksum to be recalculated and forces
422the build system to run the task again.
423
424.. note::
425
426 For an example of a commit that makes a cosmetic change to invalidate
427 shared state, see this
428 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
429
430Running Specific Tasks
431======================
432
433Any given recipe consists of a set of tasks. The standard BitBake
434behavior in most cases is: :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`,
435:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, :ref:`ref-tasks-install`, :ref:`ref-tasks-package`,
436:ref:`do_package_write_* <ref-tasks-package_write_deb>`, and :ref:`ref-tasks-build`. The default task is
437:ref:`ref-tasks-build` and any tasks on which it depends build first. Some tasks,
438such as :ref:`ref-tasks-devshell`, are not part of the default build chain. If you
439wish to run a task that is not part of the default build chain, you can
440use the ``-c`` option in BitBake. Here is an example::
441
442 $ bitbake matchbox-desktop -c devshell
443
444The ``-c`` option respects task dependencies, which means that all other
445tasks (including tasks from other recipes) that the specified task
446depends on will be run before the task. Even when you manually specify a
447task to run with ``-c``, BitBake will only run the task if it considers
448it "out of date". See the
449":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
450section in the Yocto Project Overview and Concepts Manual for how
451BitBake determines whether a task is "out of date".
452
453If you want to force an up-to-date task to be rerun (e.g. because you
454made manual modifications to the recipe's
455:term:`WORKDIR` that you want to try
456out), then you can use the ``-f`` option.
457
458.. note::
459
460 The reason ``-f`` is never required when running the
461 :ref:`ref-tasks-devshell` task is because the
462 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
463 variable flag is already set for the task.
464
465The following example shows one way you can use the ``-f`` option::
466
467 $ bitbake matchbox-desktop
468 .
469 .
470 make some changes to the source code in the work directory
471 .
472 .
473 $ bitbake matchbox-desktop -c compile -f
474 $ bitbake matchbox-desktop
475
476This sequence first builds and then recompiles ``matchbox-desktop``. The
477last command reruns all tasks (basically the packaging tasks) after the
478compile. BitBake recognizes that the :ref:`ref-tasks-compile` task was rerun and
479therefore understands that the other tasks also need to be run again.
480
481Another, shorter way to rerun a task and all
482:ref:`ref-manual/tasks:normal recipe build tasks`
483that depend on it is to use the ``-C`` option.
484
485.. note::
486
487 This option is upper-cased and is separate from the ``-c``
488 option, which is lower-cased.
489
490Using this option invalidates the given task and then runs the
491:ref:`ref-tasks-build` task, which is
492the default task if no task is given, and the tasks on which it depends.
493You could replace the final two commands in the previous example with
494the following single command::
495
496 $ bitbake matchbox-desktop -C compile
497
498Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
499the input checksum of the specified task. This tainting indirectly
500causes the task and its dependent tasks to be rerun through the normal
501task dependency mechanisms.
502
503.. note::
504
505 BitBake explicitly keeps track of which tasks have been tainted in
506 this fashion, and will print warnings such as the following for
507 builds involving such tasks:
508
509 .. code-block:: none
510
511 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
512
513
514 The purpose of the warning is to let you know that the work directory
515 and build output might not be in the clean state they would be in for
516 a "normal" build, depending on what actions you took. To get rid of
517 such warnings, you can remove the work directory and rebuild the
518 recipe, as follows::
519
520 $ bitbake matchbox-desktop -c clean
521 $ bitbake matchbox-desktop
522
523
524You can view a list of tasks in a given package by running the
525:ref:`ref-tasks-listtasks` task as follows::
526
527 $ bitbake matchbox-desktop -c listtasks
528
529The results appear as output to the console and are also in
530the file ``${WORKDIR}/temp/log.do_listtasks``.
531
532General BitBake Problems
533========================
534
535You can see debug output from BitBake by using the ``-D`` option. The
536debug output gives more information about what BitBake is doing and the
537reason behind it. Each ``-D`` option you use increases the logging
538level. The most common usage is ``-DDD``.
539
540The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
541chose a certain version of a package or why BitBake picked a certain
542provider. This command could also help you in a situation where you
543think BitBake did something unexpected.
544
545Building with No Dependencies
546=============================
547
548To build a specific recipe (``.bb`` file), you can use the following
549command form::
550
551 $ bitbake -b somepath/somerecipe.bb
552
553This command form does
554not check for dependencies. Consequently, you should use it only when
555you know existing dependencies have been met.
556
557.. note::
558
559 You can also specify fragments of the filename. In this case, BitBake
560 checks for a unique match.
561
562Recipe Logging Mechanisms
563=========================
564
565The Yocto Project provides several logging functions for producing
566debugging output and reporting errors and warnings. For Python
567functions, the following logging functions are available. All of these functions
568log to ``${T}/log.do_``\ `task`, and can also log to standard output
569(stdout) with the right settings:
570
571- ``bb.plain(msg)``: Writes msg as is to the log while also
572 logging to stdout.
573
574- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
575 stdout if BitBake is called with "-v".
576
577- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
578 log. Also logs to stdout if the log level is greater than or equal to
579 level. See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`" option
580 in the BitBake User Manual for more information.
581
582- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
583 logging to stdout.
584
585- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
586 logging to standard out (stdout).
587
588 .. note::
589
590 Calling this function does not cause the task to fail.
591
592- ``bb.fatal(msg)``: This logging function is similar to
593 ``bb.error(msg)`` but also causes the calling task to fail.
594
595 .. note::
596
597 ``bb.fatal()`` raises an exception, which means you do not need to put a
598 "return" statement after the function.
599
600The same logging functions are also available in shell functions, under
601the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
602and ``bbfatal``. The
603:ref:`logging <ref-classes-logging>` class
604implements these functions. See that class in the ``meta/classes``
605folder of the :term:`Source Directory` for information.
606
607Logging With Python
608-------------------
609
610When creating recipes using Python and inserting code that handles build
611logs, keep in mind the goal is to have informative logs while keeping
612the console as "silent" as possible. Also, if you want status messages
613in the log, use the "debug" loglevel.
614
615Following is an example written in Python. The code handles logging for
616a function that determines the number of tasks needed to be run. See the
617":ref:`ref-tasks-listtasks`"
618section for additional information::
619
620 python do_listtasks() {
621 bb.debug(2, "Starting to figure out the task list")
622 if noteworthy_condition:
623 bb.note("There are 47 tasks to run")
624 bb.debug(2, "Got to point xyz")
625 if warning_trigger:
626 bb.warn("Detected warning_trigger, this might be a problem later.")
627 if recoverable_error:
628 bb.error("Hit recoverable_error, you really need to fix this!")
629 if fatal_error:
630 bb.fatal("fatal_error detected, unable to print the task list")
631 bb.plain("The tasks present are abc")
632 bb.debug(2, "Finished figuring out the tasklist")
633 }
634
635Logging With Bash
636-----------------
637
638When creating recipes using Bash and inserting code that handles build
639logs, you have the same goals --- informative with minimal console output.
640The syntax you use for recipes written in Bash is similar to that of
641recipes written in Python described in the previous section.
642
643Following is an example written in Bash. The code logs the progress of
644the ``do_my_function`` function.
645::
646
647 do_my_function() {
648 bbdebug 2 "Running do_my_function"
649 if [ exceptional_condition ]; then
650 bbnote "Hit exceptional_condition"
651 fi
652 bbdebug 2 "Got to point xyz"
653 if [ warning_trigger ]; then
654 bbwarn "Detected warning_trigger, this might cause a problem later."
655 fi
656 if [ recoverable_error ]; then
657 bberror "Hit recoverable_error, correcting"
658 fi
659 if [ fatal_error ]; then
660 bbfatal "fatal_error detected"
661 fi
662 bbdebug 2 "Completed do_my_function"
663 }
664
665
666Debugging Parallel Make Races
667=============================
668
669A parallel ``make`` race occurs when the build consists of several parts
670that are run simultaneously and a situation occurs when the output or
671result of one part is not ready for use with a different part of the
672build that depends on that output. Parallel make races are annoying and
673can sometimes be difficult to reproduce and fix. However, there are some simple
674tips and tricks that can help you debug and fix them. This section
675presents a real-world example of an error encountered on the Yocto
676Project autobuilder and the process used to fix it.
677
678.. note::
679
680 If you cannot properly fix a ``make`` race condition, you can work around it
681 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
682 variables.
683
684The Failure
685-----------
686
687For this example, assume that you are building an image that depends on
688the "neard" package. And, during the build, BitBake runs into problems
689and creates the following output.
690
691.. note::
692
693 This example log file has longer lines artificially broken to make
694 the listing easier to read.
695
696If you examine the output or the log file, you see the failure during
697``make``:
698
699.. code-block:: none
700
701 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
702 | DEBUG: Executing shell function do_compile
703 | NOTE: make -j 16
704 | make --no-print-directory all-am
705 | /bin/mkdir -p include/near
706 | /bin/mkdir -p include/near
707 | /bin/mkdir -p include/near
708 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
709 0.14-r0/neard-0.14/include/types.h include/near/types.h
710 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
711 0.14-r0/neard-0.14/include/log.h include/near/log.h
712 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
713 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
714 | /bin/mkdir -p include/near
715 | /bin/mkdir -p include/near
716 | /bin/mkdir -p include/near
717 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
718 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
719 | /bin/mkdir -p include/near
720 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
721 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
722 | /bin/mkdir -p include/near
723 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
724 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
725 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
726 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
727 | /bin/mkdir -p include/near
728 | /bin/mkdir -p include/near
729 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
730 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
731 | /bin/mkdir -p include/near
732 | /bin/mkdir -p include/near
733 | /bin/mkdir -p include/near
734 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
735 0.14-r0/neard-0.14/include/device.h include/near/device.h
736 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
737 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
738 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
739 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
740 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
741 0.14-r0/neard-0.14/include/version.h include/near/version.h
742 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
743 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
744 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
745 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/
746 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
747 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
748 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
749 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/
750 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/
751 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
752 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
753 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
754 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
755 -o tools/snep-send.o tools/snep-send.c
756 | In file included from tools/snep-send.c:16:0:
757 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
758 | #include <near/dbus.h>
759 | ^
760 | compilation terminated.
761 | make[1]: *** [tools/snep-send.o] Error 1
762 | make[1]: *** Waiting for unfinished jobs....
763 | make: *** [all] Error 2
764 | ERROR: oe_runmake failed
765
766Reproducing the Error
767---------------------
768
769Because race conditions are intermittent, they do not manifest
770themselves every time you do the build. In fact, most times the build
771will complete without problems even though the potential race condition
772exists. Thus, once the error surfaces, you need a way to reproduce it.
773
774In this example, compiling the "neard" package is causing the problem.
775So the first thing to do is build "neard" locally. Before you start the
776build, set the
777:term:`PARALLEL_MAKE` variable
778in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
779high value for :term:`PARALLEL_MAKE` increases the chances of the race
780condition showing up::
781
782 $ bitbake neard
783
784Once the local build for "neard" completes, start a ``devshell`` build::
785
786 $ bitbake neard -c devshell
787
788For information on how to use a ``devshell``, see the
789":ref:`dev-manual/development-shell:using a development shell`" section.
790
791In the ``devshell``, do the following::
792
793 $ make clean
794 $ make tools/snep-send.o
795
796The ``devshell`` commands cause the failure to clearly
797be visible. In this case, there is a missing dependency for the ``neard``
798Makefile target. Here is some abbreviated, sample output with the
799missing dependency clearly visible at the end::
800
801 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
802 .
803 .
804 .
805 tools/snep-send.c
806 In file included from tools/snep-send.c:16:0:
807 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
808 #include <near/dbus.h>
809 ^
810 compilation terminated.
811 make: *** [tools/snep-send.o] Error 1
812 $
813
814
815Creating a Patch for the Fix
816----------------------------
817
818Because there is a missing dependency for the Makefile target, you need
819to patch the ``Makefile.am`` file, which is generated from
820``Makefile.in``. You can use Quilt to create the patch::
821
822 $ quilt new parallelmake.patch
823 Patch patches/parallelmake.patch is now on top
824 $ quilt add Makefile.am
825 File Makefile.am added to patch patches/parallelmake.patch
826
827For more information on using Quilt, see the
828":ref:`dev-manual/quilt:using quilt in your workflow`" section.
829
830At this point you need to make the edits to ``Makefile.am`` to add the
831missing dependency. For our example, you have to add the following line
832to the file::
833
834 tools/snep-send.$(OBJEXT): include/near/dbus.h
835
836Once you have edited the file, use the ``refresh`` command to create the
837patch::
838
839 $ quilt refresh
840 Refreshed patch patches/parallelmake.patch
841
842Once the patch file is created, you need to add it back to the originating
843recipe folder. Here is an example assuming a top-level
844:term:`Source Directory` named ``poky``::
845
846 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
847
848The final thing you need to do to implement the fix in the build is to
849update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
850:term:`SRC_URI` statement includes
851the patch file. The recipe file is in the folder above the patch. Here
852is what the edited :term:`SRC_URI` statement would look like::
853
854 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
855 file://neard.in \
856 file://neard.service.in \
857 file://parallelmake.patch \
858 "
859
860With the patch complete and moved to the correct folder and the
861:term:`SRC_URI` statement updated, you can exit the ``devshell``::
862
863 $ exit
864
865Testing the Build
866-----------------
867
868With everything in place, you can get back to trying the build again
869locally::
870
871 $ bitbake neard
872
873This build should succeed.
874
875Now you can open up a ``devshell`` again and repeat the clean and make
876operations as follows::
877
878 $ bitbake neard -c devshell
879 $ make clean
880 $ make tools/snep-send.o
881
882The build should work without issue.
883
884As with all solved problems, if they originated upstream, you need to
885submit the fix for the recipe in OE-Core and upstream so that the
886problem is taken care of at its source. See the
887":ref:`dev-manual/changes:submitting a change to the yocto project`"
888section for more information.
889
890Debugging With the GNU Project Debugger (GDB) Remotely
891======================================================
892
893GDB allows you to examine running programs, which in turn helps you to
894understand and fix problems. It also allows you to perform post-mortem
895style analysis of program crashes. GDB is available as a package within
896the Yocto Project and is installed in SDK images by default. See the
897":ref:`ref-manual/images:Images`" chapter in the Yocto
898Project Reference Manual for a description of these images. You can find
899information on GDB at https://sourceware.org/gdb/.
900
901.. note::
902
903 For best results, install debug (``-dbg``) packages for the applications you
904 are going to debug. Doing so makes extra debug symbols available that give
905 you more meaningful output.
906
907Sometimes, due to memory or disk space constraints, it is not possible
908to use GDB directly on the remote target to debug applications. These
909constraints arise because GDB needs to load the debugging information
910and the binaries of the process being debugged. Additionally, GDB needs
911to perform many computations to locate information such as function
912names, variable names and values, stack traces and so forth --- even
913before starting the debugging process. These extra computations place
914more load on the target system and can alter the characteristics of the
915program being debugged.
916
917To help get past the previously mentioned constraints, there are two
918methods you can use: running a debuginfod server and using gdbserver.
919
920Using the debuginfod server method
921----------------------------------
922
923``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
924Running a ``debuginfod`` server makes debug symbols readily available,
925which means you don't need to download debugging information
926and the binaries of the process being debugged. You can just fetch
927debug symbols from the server.
928
929To run a ``debuginfod`` server, you need to do the following:
930
931- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
932 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
933 If not, set in your distro config file or in ``local.conf``::
934
935 DISTRO_FEATURES:append = " debuginfod"
936
937 This distro feature enables the server and client library in ``elfutils``,
938 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
939
940- Run the following commands to launch the ``debuginfod`` server on the host::
941
942 $ oe-debuginfod
943
944- To use ``debuginfod`` on the target, you need to know the ip:port where
945 ``debuginfod`` is listening on the host (port defaults to 8002), and export
946 that into the shell environment, for example in ``qemu``::
947
948 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
949
950- Then debug info fetching should simply work when running the target ``gdb``,
951 ``readelf`` or ``objdump``, for example::
952
953 root@qemux86-64:~# gdb /bin/cat
954 ...
955 Reading symbols from /bin/cat...
956 Downloading separate debug info for /bin/cat...
957 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
958
959- It's also possible to use ``debuginfod-find`` to just query the server::
960
961 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
962 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
963
964
965Using the gdbserver method
966--------------------------
967
968gdbserver, which runs on the remote target and does not load any
969debugging information from the debugged process. Instead, a GDB instance
970processes the debugging information that is run on a remote computer -
971the host GDB. The host GDB then sends control commands to gdbserver to
972make it stop or start the debugged program, as well as read or write
973memory regions of that debugged program. All the debugging information
974loaded and processed as well as all the heavy debugging is done by the
975host GDB. Offloading these processes gives the gdbserver running on the
976target a chance to remain small and fast.
977
978Because the host GDB is responsible for loading the debugging
979information and for doing the necessary processing to make actual
980debugging happen, you have to make sure the host can access the
981unstripped binaries complete with their debugging information and also
982be sure the target is compiled with no optimizations. The host GDB must
983also have local access to all the libraries used by the debugged
984program. Because gdbserver does not need any local debugging
985information, the binaries on the remote target can remain stripped.
986However, the binaries must also be compiled without optimization so they
987match the host's binaries.
988
989To remain consistent with GDB documentation and terminology, the binary
990being debugged on the remote target machine is referred to as the
991"inferior" binary. For documentation on GDB see the `GDB
992site <https://sourceware.org/gdb/documentation/>`__.
993
994The following steps show you how to debug using the GNU project
995debugger.
996
9971. *Configure your build system to construct the companion debug
998 filesystem:*
999
1000 In your ``local.conf`` file, set the following::
1001
1002 IMAGE_GEN_DEBUGFS = "1"
1003 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
1004
1005 These options cause the
1006 OpenEmbedded build system to generate a special companion filesystem
1007 fragment, which contains the matching source and debug symbols to
1008 your deployable filesystem. The build system does this by looking at
1009 what is in the deployed filesystem, and pulling the corresponding
1010 ``-dbg`` packages.
1011
1012 The companion debug filesystem is not a complete filesystem, but only
1013 contains the debug fragments. This filesystem must be combined with
1014 the full filesystem for debugging. Subsequent steps in this procedure
1015 show how to combine the partial filesystem with the full filesystem.
1016
10172. *Configure the system to include gdbserver in the target filesystem:*
1018
1019 Make the following addition in your ``local.conf`` file::
1020
1021 EXTRA_IMAGE_FEATURES:append = " tools-debug"
1022
1023 The change makes
1024 sure the ``gdbserver`` package is included.
1025
10263. *Build the environment:*
1027
1028 Use the following command to construct the image and the companion
1029 Debug Filesystem::
1030
1031 $ bitbake image
1032
1033 Build the cross GDB component and
1034 make it available for debugging. Build the SDK that matches the
1035 image. Building the SDK is best for a production build that can be
1036 used later for debugging, especially during long term maintenance::
1037
1038 $ bitbake -c populate_sdk image
1039
1040 Alternatively, you can build the minimal toolchain components that
1041 match the target. Doing so creates a smaller than typical SDK and
1042 only contains a minimal set of components with which to build simple
1043 test applications, as well as run the debugger::
1044
1045 $ bitbake meta-toolchain
1046
1047 A final method is to build Gdb itself within the build system::
1048
1049 $ bitbake gdb-cross-<architecture>
1050
1051 Doing so produces a temporary copy of
1052 ``cross-gdb`` you can use for debugging during development. While
1053 this is the quickest approach, the two previous methods in this step
1054 are better when considering long-term maintenance strategies.
1055
1056 .. note::
1057
1058 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
1059 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
1060 actual name you want to use.
1061
10624. *Set up the* ``debugfs``\ *:*
1063
1064 Run the following commands to set up the ``debugfs``::
1065
1066 $ mkdir debugfs
1067 $ cd debugfs
1068 $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2
1069 $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2
1070
10715. *Set up GDB:*
1072
1073 Install the SDK (if you built one) and then source the correct
1074 environment file. Sourcing the environment file puts the SDK in your
1075 ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger.
1076
1077 If you are using the build system, Gdb is located in
1078 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
1079
10806. *Boot the target:*
1081
1082 For information on how to run QEMU, see the `QEMU
1083 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
1084
1085 .. note::
1086
1087 Be sure to verify that your host can access the target via TCP.
1088
10897. *Debug a program:*
1090
1091 Debugging a program involves running gdbserver on the target and then
1092 running Gdb on the host. The example in this step debugs ``gzip``:
1093
1094 .. code-block:: shell
1095
1096 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
1097
1098 For
1099 additional gdbserver options, see the `GDB Server
1100 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
1101
1102 After running gdbserver on the target, you need to run Gdb on the
1103 host and configure it and connect to the target. Use these commands::
1104
1105 $ cd directory-holding-the-debugfs-directory
1106 $ arch-gdb
1107 (gdb) set sysroot debugfs
1108 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
1109 (gdb) target remote IP-of-target:1234
1110
1111 At this
1112 point, everything should automatically load (i.e. matching binaries,
1113 symbols and headers).
1114
1115 .. note::
1116
1117 The Gdb ``set`` commands in the previous example can be placed into the
1118 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
1119 commands are in that file.
1120
11218. *Deploying without a full image rebuild:*
1122
1123 In many cases, during development you want a quick method to deploy a
1124 new binary to the target and debug it, without waiting for a full
1125 image build.
1126
1127 One approach to solving this situation is to just build the component
1128 you want to debug. Once you have built the component, copy the
1129 executable directly to both the target and the host ``debugfs``.
1130
1131 If the binary is processed through the debug splitting in
1132 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
1133 contents and corresponding ``/usr/src/debug`` files) from the work
1134 directory. Here is an example::
1135
1136 $ bitbake bash
1137 $ bitbake -c devshell bash
1138 $ cd ..
1139 $ scp packages-split/bash/bin/bash target:/bin/bash
1140 $ cp -a packages-split/bash-dbg/\* path/debugfs
1141
1142Debugging with the GNU Project Debugger (GDB) on the Target
1143===========================================================
1144
1145The previous section addressed using GDB remotely for debugging
1146purposes, which is the most usual case due to the inherent hardware
1147limitations on many embedded devices. However, debugging in the target
1148hardware itself is also possible with more powerful devices. This
1149section describes what you need to do in order to support using GDB to
1150debug on the target hardware.
1151
1152To support this kind of debugging, you need do the following:
1153
1154- Ensure that GDB is on the target. You can do this by making
1155 the following addition to your ``local.conf`` file::
1156
1157 EXTRA_IMAGE_FEATURES:append = " tools-debug"
1158
1159- Ensure that debug symbols are present. You can do so by adding the
1160 corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`::
1161
1162 IMAGE_INSTALL:append = " packagename-dbg"
1163
1164 Alternatively, you can add the following to ``local.conf`` to include
1165 all the debug symbols::
1166
1167 EXTRA_IMAGE_FEATURES:append = " dbg-pkgs"
1168
1169.. note::
1170
1171 To improve the debug information accuracy, you can reduce the level
1172 of optimization used by the compiler. For example, when adding the
1173 following line to your ``local.conf`` file, you will reduce optimization
1174 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
1175 of "-O -fno-omit-frame-pointer"::
1176
1177 DEBUG_BUILD = "1"
1178
1179 Consider that this will reduce the application's performance and is
1180 recommended only for debugging purposes.
1181
1182Other Debugging Tips
1183====================
1184
1185Here are some other tips that you might find useful:
1186
1187- When adding new packages, it is worth watching for undesirable items
1188 making their way into compiler command lines. For example, you do not
1189 want references to local system files like ``/usr/lib/`` or
1190 ``/usr/include/``.
1191
1192- If you want to remove the ``psplash`` boot splashscreen, add
1193 ``psplash=false`` to the kernel command line. Doing so prevents
1194 ``psplash`` from loading and thus allows you to see the console. It
1195 is also possible to switch out of the splashscreen by switching the
1196 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
1197
1198- Removing :term:`TMPDIR` (usually ``tmp/``, within the
1199 :term:`Build Directory`) can often fix temporary build issues. Removing
1200 :term:`TMPDIR` is usually a relatively cheap operation, because task output
1201 will be cached in :term:`SSTATE_DIR` (usually ``sstate-cache/``, which is
1202 also in the :term:`Build Directory`).
1203
1204 .. note::
1205
1206 Removing :term:`TMPDIR` might be a workaround rather than a fix.
1207 Consequently, trying to determine the underlying cause of an issue before
1208 removing the directory is a good idea.
1209
1210- Understanding how a feature is used in practice within existing
1211 recipes can be very helpful. It is recommended that you configure
1212 some method that allows you to quickly search through files.
1213
1214 Using GNU Grep, you can use the following shell function to
1215 recursively search through common recipe-related files, skipping
1216 binary files, ``.git`` directories, and the :term:`Build Directory`
1217 (assuming its name starts with "build")::
1218
1219 g() {
1220 grep -Ir \
1221 --exclude-dir=.git \
1222 --exclude-dir='build*' \
1223 --include='*.bb*' \
1224 --include='*.inc*' \
1225 --include='*.conf*' \
1226 --include='*.py*' \
1227 "$@"
1228 }
1229
1230 Following are some usage examples::
1231
1232 $ g FOO # Search recursively for "FOO"
1233 $ g -i foo # Search recursively for "foo", ignoring case
1234 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
1235
1236 If figuring
1237 out how some feature works requires a lot of searching, it might
1238 indicate that the documentation should be extended or improved. In
1239 such cases, consider filing a documentation bug using the Yocto
1240 Project implementation of
1241 :yocto_bugs:`Bugzilla <>`. For information on
1242 how to submit a bug against the Yocto Project, see the Yocto Project
1243 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
1244 and the
1245 ":ref:`dev-manual/changes:submitting a defect against the yocto project`"
1246 section.
1247
1248 .. note::
1249
1250 The manuals might not be the right place to document variables
1251 that are purely internal and have a limited scope (e.g. internal
1252 variables used to implement a single ``.bbclass`` file).
1253
diff --git a/documentation/dev-manual/development-shell.rst b/documentation/dev-manual/development-shell.rst
new file mode 100644
index 0000000000..a18d792150
--- /dev/null
+++ b/documentation/dev-manual/development-shell.rst
@@ -0,0 +1,82 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using a Development Shell
4*************************
5
6When debugging certain commands or even when just editing packages,
7``devshell`` can be a useful tool. When you invoke ``devshell``, all
8tasks up to and including
9:ref:`ref-tasks-patch` are run for the
10specified target. Then, a new terminal is opened and you are placed in
11``${``\ :term:`S`\ ``}``, the source
12directory. In the new terminal, all the OpenEmbedded build-related
13environment variables are still defined so you can use commands such as
14``configure`` and ``make``. The commands execute just as if the
15OpenEmbedded build system were executing them. Consequently, working
16this way can be helpful when debugging a build or preparing software to
17be used with the OpenEmbedded build system.
18
19Following is an example that uses ``devshell`` on a target named
20``matchbox-desktop``::
21
22 $ bitbake matchbox-desktop -c devshell
23
24This command spawns a terminal with a shell prompt within the
25OpenEmbedded build environment. The
26:term:`OE_TERMINAL` variable
27controls what type of shell is opened.
28
29For spawned terminals, the following occurs:
30
31- The ``PATH`` variable includes the cross-toolchain.
32
33- The ``pkgconfig`` variables find the correct ``.pc`` files.
34
35- The ``configure`` command finds the Yocto Project site files as well
36 as any other necessary files.
37
38Within this environment, you can run configure or compile commands as if
39they were being run by the OpenEmbedded build system itself. As noted
40earlier, the working directory also automatically changes to the Source
41Directory (:term:`S`).
42
43To manually run a specific task using ``devshell``, run the
44corresponding ``run.*`` script in the
45``${``\ :term:`WORKDIR`\ ``}/temp``
46directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
47not exist, which would be the case if the task was skipped by way of the
48sstate cache, you can create the task by first running it outside of the
49``devshell``::
50
51 $ bitbake -c task
52
53.. note::
54
55 - Execution of a task's ``run.*`` script and BitBake's execution of
56 a task are identical. In other words, running the script re-runs
57 the task just as it would be run using the ``bitbake -c`` command.
58
59 - Any ``run.*`` file that does not have a ``.pid`` extension is a
60 symbolic link (symlink) to the most recent version of that file.
61
62Remember, that the ``devshell`` is a mechanism that allows you to get
63into the BitBake task execution environment. And as such, all commands
64must be called just as BitBake would call them. That means you need to
65provide the appropriate options for cross-compilation and so forth as
66applicable.
67
68When you are finished using ``devshell``, exit the shell or close the
69terminal window.
70
71.. note::
72
73 - It is worth remembering that when using ``devshell`` you need to
74 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
75 instead of just using ``gcc``. The same applies to other
76 applications such as ``binutils``, ``libtool`` and so forth.
77 BitBake sets up environment variables such as :term:`CC` to assist
78 applications, such as ``make`` to find the correct tools.
79
80 - It is also worth noting that ``devshell`` still works over X11
81 forwarding and similar situations.
82
diff --git a/documentation/dev-manual/device-manager.rst b/documentation/dev-manual/device-manager.rst
new file mode 100644
index 0000000000..4248c23b44
--- /dev/null
+++ b/documentation/dev-manual/device-manager.rst
@@ -0,0 +1,72 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Selecting a Device Manager
4**************************
5
6The Yocto Project provides multiple ways to manage the device manager
7(``/dev``):
8
9- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev``
10 directory is persistent and the required device nodes are created
11 during the build.
12
13- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
14 directory is provided by the kernel as an in-memory file system and
15 is automatically populated by the kernel at runtime. Additional
16 configuration of device nodes is done in user space by a device
17 manager like ``udev`` or ``busybox-mdev``.
18
19Using Persistent and Pre-Populated ``/dev``
20===========================================
21
22To use the static method for device population, you need to set the
23:term:`USE_DEVFS` variable to "0"
24as follows::
25
26 USE_DEVFS = "0"
27
28The content of the resulting ``/dev`` directory is defined in a Device
29Table file. The
30:term:`IMAGE_DEVICE_TABLES`
31variable defines the Device Table to use and should be set in the
32machine or distro configuration file. Alternatively, you can set this
33variable in your ``local.conf`` configuration file.
34
35If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
36``device_table-minimal.txt`` is used::
37
38 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
39
40The population is handled by the ``makedevs`` utility during image
41creation:
42
43Using ``devtmpfs`` and a Device Manager
44=======================================
45
46To use the dynamic method for device population, you need to use (or be
47sure to set) the :term:`USE_DEVFS`
48variable to "1", which is the default::
49
50 USE_DEVFS = "1"
51
52With this
53setting, the resulting ``/dev`` directory is populated by the kernel
54using ``devtmpfs``. Make sure the corresponding kernel configuration
55variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
56kernel.
57
58All devices created by ``devtmpfs`` will be owned by ``root`` and have
59permissions ``0600``.
60
61To have more control over the device nodes, you can use a device manager
62like ``udev`` or ``busybox-mdev``. You choose the device manager by
63defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
64distro configuration file. Alternatively, you can set this variable in
65your ``local.conf`` configuration file::
66
67 VIRTUAL-RUNTIME_dev_manager = "udev"
68
69 # Some alternative values
70 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
71 # VIRTUAL-RUNTIME_dev_manager = "systemd"
72
diff --git a/documentation/dev-manual/disk-space.rst b/documentation/dev-manual/disk-space.rst
new file mode 100644
index 0000000000..6fa48e1f4a
--- /dev/null
+++ b/documentation/dev-manual/disk-space.rst
@@ -0,0 +1,40 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Conserving Disk Space
4*********************
5
6Conserving Disk Space During Builds
7===================================
8
9To help conserve disk space during builds, you can add the following
10statement to your project's ``local.conf`` configuration file found in
11the :term:`Build Directory`::
12
13 INHERIT += "rm_work"
14
15Adding this statement deletes the work directory used for
16building a recipe once the recipe is built. For more information on
17"rm_work", see the
18:ref:`rm_work <ref-classes-rm-work>` class in the
19Yocto Project Reference Manual.
20
21Purging Duplicate Shared State Cache Files
22==========================================
23
24After multiple build iterations, the Shared State (sstate) cache can contain
25duplicate cache files for a given package, while only the most recent one
26is likely to be reusable. The following command purges all but the
27newest sstate cache file for each package::
28
29 sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
30
31This command will ask you to confirm the deletions it identifies.
32
33.. note::
34
35 The duplicated sstate cache files of one package must have the same
36 architecture, which means that sstate cache files with multiple
37 architectures are not considered as duplicate.
38
39Run ``sstate-cache-management.sh`` for more details about this script.
40
diff --git a/documentation/dev-manual/efficiently-fetching-sources.rst b/documentation/dev-manual/efficiently-fetching-sources.rst
new file mode 100644
index 0000000000..a15f0a92ce
--- /dev/null
+++ b/documentation/dev-manual/efficiently-fetching-sources.rst
@@ -0,0 +1,68 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Efficiently Fetching Source Files During a Build
4************************************************
5
6The OpenEmbedded build system works with source files located through
7the :term:`SRC_URI` variable. When
8you build something using BitBake, a big part of the operation is
9locating and downloading all the source tarballs. For images,
10downloading all the source for various packages can take a significant
11amount of time.
12
13This section shows you how you can use mirrors to speed up fetching
14source files and how you can pre-fetch files all of which leads to more
15efficient use of resources and time.
16
17Setting up Effective Mirrors
18============================
19
20A good deal that goes into a Yocto Project build is simply downloading
21all of the source tarballs. Maybe you have been working with another
22build system for which you have built up a
23sizable directory of source tarballs. Or, perhaps someone else has such
24a directory for which you have read access. If so, you can save time by
25adding statements to your configuration file so that the build process
26checks local directories first for existing tarballs before checking the
27Internet.
28
29Here is an efficient way to set it up in your ``local.conf`` file::
30
31 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
32 INHERIT += "own-mirrors"
33 BB_GENERATE_MIRROR_TARBALLS = "1"
34 # BB_NO_NETWORK = "1"
35
36In the previous example, the
37:term:`BB_GENERATE_MIRROR_TARBALLS`
38variable causes the OpenEmbedded build system to generate tarballs of
39the Git repositories and store them in the
40:term:`DL_DIR` directory. Due to
41performance reasons, generating and storing these tarballs is not the
42build system's default behavior.
43
44You can also use the
45:term:`PREMIRRORS` variable. For
46an example, see the variable's glossary entry in the Yocto Project
47Reference Manual.
48
49Getting Source Files and Suppressing the Build
50==============================================
51
52Another technique you can use to ready yourself for a successive string
53of build operations, is to pre-fetch all the source files without
54actually starting a build. This technique lets you work through any
55download issues and ultimately gathers all the source files into your
56download directory :ref:`structure-build-downloads`,
57which is located with :term:`DL_DIR`.
58
59Use the following BitBake command form to fetch all the necessary
60sources without starting the build::
61
62 $ bitbake target --runall=fetch
63
64This
65variation of the BitBake command guarantees that you have all the
66sources for that BitBake target should you disconnect from the Internet
67and want to do the build later offline.
68
diff --git a/documentation/dev-manual/error-reporting-tool.rst b/documentation/dev-manual/error-reporting-tool.rst
new file mode 100644
index 0000000000..a2636da37a
--- /dev/null
+++ b/documentation/dev-manual/error-reporting-tool.rst
@@ -0,0 +1,85 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using the Error Reporting Tool
4******************************
5
6The error reporting tool allows you to submit errors encountered during
7builds to a central database. Outside of the build environment, you can
8use a web interface to browse errors, view statistics, and query for
9errors. The tool works using a client-server system where the client
10portion is integrated with the installed Yocto Project
11:term:`Source Directory` (e.g. ``poky``).
12The server receives the information collected and saves it in a
13database.
14
15There is a live instance of the error reporting server at
16https://errors.yoctoproject.org.
17When you want to get help with build failures, you can submit all of the
18information on the failure easily and then point to the URL in your bug
19report or send an email to the mailing list.
20
21.. note::
22
23 If you send error reports to this server, the reports become publicly
24 visible.
25
26Enabling and Using the Tool
27===========================
28
29By default, the error reporting tool is disabled. You can enable it by
30inheriting the :ref:`report-error <ref-classes-report-error>`
31class by adding the following statement to the end of your
32``local.conf`` file in your :term:`Build Directory`::
33
34 INHERIT += "report-error"
35
36By default, the error reporting feature stores information in
37``${``\ :term:`LOG_DIR`\ ``}/error-report``.
38However, you can specify a directory to use by adding the following to
39your ``local.conf`` file::
40
41 ERR_REPORT_DIR = "path"
42
43Enabling error
44reporting causes the build process to collect the errors and store them
45in a file as previously described. When the build system encounters an
46error, it includes a command as part of the console output. You can run
47the command to send the error file to the server. For example, the
48following command sends the errors to an upstream server::
49
50 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
51
52In the previous example, the errors are sent to a public database
53available at https://errors.yoctoproject.org, which is used by the
54entire community. If you specify a particular server, you can send the
55errors to a different database. Use the following command for more
56information on available options::
57
58 $ send-error-report --help
59
60When sending the error file, you are prompted to review the data being
61sent as well as to provide a name and optional email address. Once you
62satisfy these prompts, the command returns a link from the server that
63corresponds to your entry in the database. For example, here is a
64typical link: https://errors.yoctoproject.org/Errors/Details/9522/
65
66Following the link takes you to a web interface where you can browse,
67query the errors, and view statistics.
68
69Disabling the Tool
70==================
71
72To disable the error reporting feature, simply remove or comment out the
73following statement from the end of your ``local.conf`` file in your
74:term:`Build Directory`.
75::
76
77 INHERIT += "report-error"
78
79Setting Up Your Own Error Reporting Server
80==========================================
81
82If you want to set up your own error reporting server, you can obtain
83the code from the Git repository at :yocto_git:`/error-report-web/`.
84Instructions on how to set it up are in the README document.
85
diff --git a/documentation/dev-manual/external-scm.rst b/documentation/dev-manual/external-scm.rst
new file mode 100644
index 0000000000..97a7e63e36
--- /dev/null
+++ b/documentation/dev-manual/external-scm.rst
@@ -0,0 +1,67 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using an External SCM
4*********************
5
6If you're working on a recipe that pulls from an external Source Code
7Manager (SCM), it is possible to have the OpenEmbedded build system
8notice new recipe changes added to the SCM and then build the resulting
9packages that depend on the new recipes by using the latest versions.
10This only works for SCMs from which it is possible to get a sensible
11revision number for changes. Currently, you can do this with Apache
12Subversion (SVN), Git, and Bazaar (BZR) repositories.
13
14To enable this behavior, the :term:`PV` of
15the recipe needs to reference
16:term:`SRCPV`. Here is an example::
17
18 PV = "1.2.3+git${SRCPV}"
19
20Then, you can add the following to your
21``local.conf``::
22
23 SRCREV:pn-PN = "${AUTOREV}"
24
25:term:`PN` is the name of the recipe for
26which you want to enable automatic source revision updating.
27
28If you do not want to update your local configuration file, you can add
29the following directly to the recipe to finish enabling the feature::
30
31 SRCREV = "${AUTOREV}"
32
33The Yocto Project provides a distribution named ``poky-bleeding``, whose
34configuration file contains the line::
35
36 require conf/distro/include/poky-floating-revisions.inc
37
38This line pulls in the
39listed include file that contains numerous lines of exactly that form::
40
41 #SRCREV:pn-opkg-native ?= "${AUTOREV}"
42 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}"
43 #SRCREV:pn-opkg ?= "${AUTOREV}"
44 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}"
45 #SRCREV:pn-opkg-utils ?= "${AUTOREV}"
46 SRCREV:pn-gconf-dbus ?= "${AUTOREV}"
47 SRCREV:pn-matchbox-common ?= "${AUTOREV}"
48 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}"
49 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}"
50 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}"
51 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}"
52 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}"
53 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}"
54 SRCREV:pn-matchbox-wm ?= "${AUTOREV}"
55 SRCREV:pn-settings-daemon ?= "${AUTOREV}"
56 SRCREV:pn-screenshot ?= "${AUTOREV}"
57 . . .
58
59These lines allow you to
60experiment with building a distribution that tracks the latest
61development source for numerous packages.
62
63.. note::
64
65 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
66 this in mind if you use it.
67
diff --git a/documentation/dev-manual/external-toolchain.rst b/documentation/dev-manual/external-toolchain.rst
new file mode 100644
index 0000000000..e1fabbed22
--- /dev/null
+++ b/documentation/dev-manual/external-toolchain.rst
@@ -0,0 +1,28 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Optionally Using an External Toolchain
4**************************************
5
6You might want to use an external toolchain as part of your development.
7If this is the case, the fundamental steps you need to accomplish are as
8follows:
9
10- Understand where the installed toolchain resides. For cases where you
11 need to build the external toolchain, you would need to take separate
12 steps to build and install the toolchain.
13
14- Make sure you add the layer that contains the toolchain to your
15 ``bblayers.conf`` file through the
16 :term:`BBLAYERS` variable.
17
18- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
19 to the location in which you installed the toolchain.
20
21A good example of an external toolchain used with the Yocto Project is
22Mentor Graphics Sourcery G++ Toolchain. You can see information on how
23to use that particular layer in the ``README`` file at
24https://github.com/MentorEmbedded/meta-sourcery/. You can find
25further information by reading about the
26:term:`TCMODE` variable in the Yocto
27Project Reference Manual's variable glossary.
28
diff --git a/documentation/dev-manual/gobject-introspection.rst b/documentation/dev-manual/gobject-introspection.rst
new file mode 100644
index 0000000000..89f21b7d10
--- /dev/null
+++ b/documentation/dev-manual/gobject-introspection.rst
@@ -0,0 +1,157 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Enabling GObject Introspection Support
4**************************************
5
6`GObject introspection <https://gi.readthedocs.io/en/latest/>`__
7is the standard mechanism for accessing GObject-based software from
8runtime environments. GObject is a feature of the GLib library that
9provides an object framework for the GNOME desktop and related software.
10GObject Introspection adds information to GObject that allows objects
11created within it to be represented across different programming
12languages. If you want to construct GStreamer pipelines using Python, or
13control UPnP infrastructure using Javascript and GUPnP, GObject
14introspection is the only way to do it.
15
16This section describes the Yocto Project support for generating and
17packaging GObject introspection data. GObject introspection data is a
18description of the API provided by libraries built on top of the GLib
19framework, and, in particular, that framework's GObject mechanism.
20GObject Introspection Repository (GIR) files go to ``-dev`` packages,
21``typelib`` files go to main packages as they are packaged together with
22libraries that are introspected.
23
24The data is generated when building such a library, by linking the
25library with a small executable binary that asks the library to describe
26itself, and then executing the binary and processing its output.
27
28Generating this data in a cross-compilation environment is difficult
29because the library is produced for the target architecture, but its
30code needs to be executed on the build host. This problem is solved with
31the OpenEmbedded build system by running the code through QEMU, which
32allows precisely that. Unfortunately, QEMU does not always work
33perfectly as mentioned in the ":ref:`dev-manual/gobject-introspection:known issues`"
34section.
35
36Enabling the Generation of Introspection Data
37=============================================
38
39Enabling the generation of introspection data (GIR files) in your
40library package involves the following:
41
421. Inherit the
43 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
44 class.
45
462. Make sure introspection is not disabled anywhere in the recipe or
47 from anything the recipe includes. Also, make sure that
48 "gobject-introspection-data" is not in
49 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
50 and that "qemu-usermode" is not in
51 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
52 In either of these conditions, nothing will happen.
53
543. Try to build the recipe. If you encounter build errors that look like
55 something is unable to find ``.so`` libraries, check where these
56 libraries are located in the source tree and add the following to the
57 recipe::
58
59 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
60
61 .. note::
62
63 See recipes in the ``oe-core`` repository that use that
64 :term:`GIR_EXTRA_LIBS_PATH` variable as an example.
65
664. Look for any other errors, which probably mean that introspection
67 support in a package is not entirely standard, and thus breaks down
68 in a cross-compilation environment. For such cases, custom-made fixes
69 are needed. A good place to ask and receive help in these cases is
70 the :ref:`Yocto Project mailing
71 lists <resources-mailinglist>`.
72
73.. note::
74
75 Using a library that no longer builds against the latest Yocto
76 Project release and prints introspection related errors is a good
77 candidate for the previous procedure.
78
79Disabling the Generation of Introspection Data
80==============================================
81
82You might find that you do not want to generate introspection data. Or,
83perhaps QEMU does not work on your build host and target architecture
84combination. If so, you can use either of the following methods to
85disable GIR file generations:
86
87- Add the following to your distro configuration::
88
89 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
90
91 Adding this statement disables generating introspection data using
92 QEMU but will still enable building introspection tools and libraries
93 (i.e. building them does not require the use of QEMU).
94
95- Add the following to your machine configuration::
96
97 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
98
99 Adding this statement disables the use of QEMU when building packages for your
100 machine. Currently, this feature is used only by introspection
101 recipes and has the same effect as the previously described option.
102
103 .. note::
104
105 Future releases of the Yocto Project might have other features
106 affected by this option.
107
108If you disable introspection data, you can still obtain it through other
109means such as copying the data from a suitable sysroot, or by generating
110it on the target hardware. The OpenEmbedded build system does not
111currently provide specific support for these techniques.
112
113Testing that Introspection Works in an Image
114============================================
115
116Use the following procedure to test if generating introspection data is
117working in an image:
118
1191. Make sure that "gobject-introspection-data" is not in
120 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
121 and that "qemu-usermode" is not in
122 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
123
1242. Build ``core-image-sato``.
125
1263. Launch a Terminal and then start Python in the terminal.
127
1284. Enter the following in the terminal::
129
130 >>> from gi.repository import GLib
131 >>> GLib.get_host_name()
132
1335. For something a little more advanced, enter the following see:
134 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
135
136Known Issues
137============
138
139Here are know issues in GObject Introspection Support:
140
141- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
142 introspection data on that architecture.
143
144- x32 is not supported by QEMU. Consequently, introspection data is
145 disabled.
146
147- musl causes transient GLib binaries to crash on assertion failures.
148 Consequently, generating introspection data is disabled.
149
150- Because QEMU is not able to run the binaries correctly, introspection
151 is disabled for some specific packages under specific architectures
152 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
153
154- QEMU usermode might not work properly when running 64-bit binaries
155 under 32-bit host machines. In particular, "qemumips64" is known to
156 not work under i686.
157
diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
index f16b135c4d..b0bb5576ad 100644
--- a/documentation/dev-manual/index.rst
+++ b/documentation/dev-manual/index.rst
@@ -12,7 +12,43 @@ Yocto Project Development Tasks Manual
12 12
13 intro 13 intro
14 start 14 start
15 common-tasks 15 layers
16 customizing-images
17 new-recipe
18 new-machine
19 upgrading-recipes
20 temporary-source-code
21 quilt.rst
22 development-shell
23 python-development-shell
24 building
25 speeding-up-build
26 libraries
27 prebuilt-libraries
28 x32-psabi
29 gobject-introspection
30 external-toolchain
31 wic
32 bmaptool
33 securing-images
34 custom-distribution
35 custom-template-configuration-directory
36 disk-space
37 packages
38 efficiently-fetching-sources
39 init-manager
40 device-manager
41 external-scm
42 read-only-rootfs
43 build-quality
44 runtime-testing
45 debugging
46 changes
47 licenses
48 vulnerabilities
49 sbom
50 error-reporting-tool
51 wayland
16 qemu 52 qemu
17 53
18.. include:: /boilerplate.rst 54.. include:: /boilerplate.rst
diff --git a/documentation/dev-manual/init-manager.rst b/documentation/dev-manual/init-manager.rst
new file mode 100644
index 0000000000..0617fed516
--- /dev/null
+++ b/documentation/dev-manual/init-manager.rst
@@ -0,0 +1,89 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Selecting an Initialization Manager
4***********************************
5
6By default, the Yocto Project uses SysVinit as the initialization
7manager. However, there is also support for systemd, which is a full
8replacement for init with parallel starting of services, reduced shell
9overhead and other features that are used by many distributions.
10
11Within the system, SysVinit treats system components as services. These
12services are maintained as shell scripts stored in the ``/etc/init.d/``
13directory. Services organize into different run levels. This
14organization is maintained by putting links to the services in the
15``/etc/rcN.d/`` directories, where `N/` is one of the following options:
16"S", "0", "1", "2", "3", "4", "5", or "6".
17
18.. note::
19
20 Each runlevel has a dependency on the previous runlevel. This
21 dependency allows the services to work properly.
22
23In comparison, systemd treats components as units. Using units is a
24broader concept as compared to using a service. A unit includes several
25different types of entities. Service is one of the types of entities.
26The runlevel concept in SysVinit corresponds to the concept of a target
27in systemd, where target is also a type of supported unit.
28
29In a SysVinit-based system, services load sequentially (i.e. one by one)
30during init and parallelization is not supported. With systemd, services
31start in parallel. Needless to say, the method can have an impact on
32system startup performance.
33
34If you want to use SysVinit, you do not have to do anything. But, if you
35want to use systemd, you must take some steps as described in the
36following sections.
37
38Using systemd Exclusively
39=========================
40
41Set these variables in your distribution configuration file as follows::
42
43 DISTRO_FEATURES:append = " systemd"
44 VIRTUAL-RUNTIME_init_manager = "systemd"
45
46You can also prevent the SysVinit distribution feature from
47being automatically enabled as follows::
48
49 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
50
51Doing so removes any
52redundant SysVinit scripts.
53
54To remove initscripts from your image altogether, set this variable
55also::
56
57 VIRTUAL-RUNTIME_initscripts = ""
58
59For information on the backfill variable, see
60:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
61
62Using systemd for the Main Image and Using SysVinit for the Rescue Image
63========================================================================
64
65Set these variables in your distribution configuration file as follows::
66
67 DISTRO_FEATURES:append = " systemd"
68 VIRTUAL-RUNTIME_init_manager = "systemd"
69
70Doing so causes your main image to use the
71``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
72image cannot use this package group. However, it can install SysVinit
73and the appropriate packages will have support for both systemd and
74SysVinit.
75
76Using systemd-journald without a traditional syslog daemon
77==========================================================
78
79Counter-intuitively, ``systemd-journald`` is not a syslog runtime or provider,
80and the proper way to use systemd-journald as your sole logging mechanism is to
81effectively disable syslog entirely by setting these variables in your distribution
82configuration file::
83
84 VIRTUAL-RUNTIME_syslog = ""
85 VIRTUAL-RUNTIME_base-utils-syslog = ""
86
87Doing so will prevent ``rsyslog`` / ``busybox-syslog`` from being pulled in by
88default, leaving only ``journald``.
89
diff --git a/documentation/dev-manual/layers.rst b/documentation/dev-manual/layers.rst
new file mode 100644
index 0000000000..ad22524833
--- /dev/null
+++ b/documentation/dev-manual/layers.rst
@@ -0,0 +1,905 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Understanding and Creating Layers
4*********************************
5
6The OpenEmbedded build system supports organizing
7:term:`Metadata` into multiple layers.
8Layers allow you to isolate different types of customizations from each
9other. For introductory information on the Yocto Project Layer Model,
10see the
11":ref:`overview-manual/yp-intro:the yocto project layer model`"
12section in the Yocto Project Overview and Concepts Manual.
13
14Creating Your Own Layer
15=======================
16
17.. note::
18
19 It is very easy to create your own layers to use with the OpenEmbedded
20 build system, as the Yocto Project ships with tools that speed up creating
21 layers. This section describes the steps you perform by hand to create
22 layers so that you can better understand them. For information about the
23 layer-creation tools, see the
24 ":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
25 section in the Yocto Project Board Support Package (BSP) Developer's
26 Guide and the ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
27 section further down in this manual.
28
29Follow these general steps to create your layer without using tools:
30
311. *Check Existing Layers:* Before creating a new layer, you should be
32 sure someone has not already created a layer containing the Metadata
33 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
34 for a list of layers from the OpenEmbedded community that can be used in
35 the Yocto Project. You could find a layer that is identical or close
36 to what you need.
37
382. *Create a Directory:* Create the directory for your layer. When you
39 create the layer, be sure to create the directory in an area not
40 associated with the Yocto Project :term:`Source Directory`
41 (e.g. the cloned ``poky`` repository).
42
43 While not strictly required, prepend the name of the directory with
44 the string "meta-". For example::
45
46 meta-mylayer
47 meta-GUI_xyz
48 meta-mymachine
49
50 With rare exceptions, a layer's name follows this form::
51
52 meta-root_name
53
54 Following this layer naming convention can save
55 you trouble later when tools, components, or variables "assume" your
56 layer name begins with "meta-". A notable example is in configuration
57 files as shown in the following step where layer names without the
58 "meta-" string are appended to several variables used in the
59 configuration.
60
613. *Create a Layer Configuration File:* Inside your new layer folder,
62 you need to create a ``conf/layer.conf`` file. It is easiest to take
63 an existing layer configuration file and copy that to your layer's
64 ``conf`` directory and then modify the file as needed.
65
66 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
67 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
68 demonstrates the required syntax. For your layer, you need to replace
69 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
70 for a layer named "meta-machinexyz")::
71
72 # We have a conf and classes directory, add to BBPATH
73 BBPATH .= ":${LAYERDIR}"
74
75 # We have recipes-* directories, add to BBFILES
76 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
77 ${LAYERDIR}/recipes-*/*/*.bbappend"
78
79 BBFILE_COLLECTIONS += "yoctobsp"
80 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
81 BBFILE_PRIORITY_yoctobsp = "5"
82 LAYERVERSION_yoctobsp = "4"
83 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
84
85 Following is an explanation of the layer configuration file:
86
87 - :term:`BBPATH`: Adds the layer's
88 root directory to BitBake's search path. Through the use of the
89 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
90 configuration files, and files that are included with ``include``
91 and ``require`` statements. For these cases, BitBake uses the
92 first file that matches the name found in :term:`BBPATH`. This is
93 similar to the way the ``PATH`` variable is used for binaries. It
94 is recommended, therefore, that you use unique class and
95 configuration filenames in your custom layer.
96
97 - :term:`BBFILES`: Defines the
98 location for all recipes in the layer.
99
100 - :term:`BBFILE_COLLECTIONS`:
101 Establishes the current layer through a unique identifier that is
102 used throughout the OpenEmbedded build system to refer to the
103 layer. In this example, the identifier "yoctobsp" is the
104 representation for the container layer named "meta-yocto-bsp".
105
106 - :term:`BBFILE_PATTERN`:
107 Expands immediately during parsing to provide the directory of the
108 layer.
109
110 - :term:`BBFILE_PRIORITY`:
111 Establishes a priority to use for recipes in the layer when the
112 OpenEmbedded build finds recipes of the same name in different
113 layers.
114
115 - :term:`LAYERVERSION`:
116 Establishes a version number for the layer. You can use this
117 version number to specify this exact version of the layer as a
118 dependency when using the
119 :term:`LAYERDEPENDS`
120 variable.
121
122 - :term:`LAYERDEPENDS`:
123 Lists all layers on which this layer depends (if any).
124
125 - :term:`LAYERSERIES_COMPAT`:
126 Lists the :yocto_wiki:`Yocto Project </Releases>`
127 releases for which the current version is compatible. This
128 variable is a good way to indicate if your particular layer is
129 current.
130
1314. *Add Content:* Depending on the type of layer, add the content. If
132 the layer adds support for a machine, add the machine configuration
133 in a ``conf/machine/`` file within the layer. If the layer adds
134 distro policy, add the distro configuration in a ``conf/distro/``
135 file within the layer. If the layer introduces new recipes, put the
136 recipes you need in ``recipes-*`` subdirectories within the layer.
137
138 .. note::
139
140 For an explanation of layer hierarchy that is compliant with the
141 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
142 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
143
1445. *Optionally Test for Compatibility:* If you want permission to use
145 the Yocto Project Compatibility logo with your layer or application
146 that uses your layer, perform the steps to apply for compatibility.
147 See the
148 ":ref:`dev-manual/layers:making sure your layer is compatible with yocto project`"
149 section for more information.
150
151Following Best Practices When Creating Layers
152=============================================
153
154To create layers that are easier to maintain and that will not impact
155builds for other machines, you should consider the information in the
156following list:
157
158- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
159 Configuration:* In other words, do not copy an entire recipe into
160 your layer and then modify it. Rather, use an append file
161 (``.bbappend``) to override only those parts of the original recipe
162 you need to modify.
163
164- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
165 for each recipe that uses an include file. Or, if you are introducing
166 a new recipe that requires the included file, use the path relative
167 to the original layer directory to refer to the file. For example,
168 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
169 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
170 the include file, it could indicate a deficiency in the include file
171 in the layer to which it originally belongs. If this is the case, you
172 should try to address that deficiency instead of overlaying the
173 include file. For example, you could address this by getting the
174 maintainer of the include file to add a variable or variables to make
175 it easy to override the parts needing to be overridden.
176
177- *Structure Your Layers:* Proper use of overrides within append files
178 and placement of machine-specific files within your layer can ensure
179 that a build is not using the wrong Metadata and negatively impacting
180 a build for a different machine. Following are some examples:
181
182 - *Modify Variables to Support a Different Machine:* Suppose you
183 have a layer named ``meta-one`` that adds support for building
184 machine "one". To do so, you use an append file named
185 ``base-files.bbappend`` and create a dependency on "foo" by
186 altering the :term:`DEPENDS`
187 variable::
188
189 DEPENDS = "foo"
190
191 The dependency is created during any
192 build that includes the layer ``meta-one``. However, you might not
193 want this dependency for all machines. For example, suppose you
194 are building for machine "two" but your ``bblayers.conf`` file has
195 the ``meta-one`` layer included. During the build, the
196 ``base-files`` for machine "two" will also have the dependency on
197 ``foo``.
198
199 To make sure your changes apply only when building machine "one",
200 use a machine override with the :term:`DEPENDS` statement::
201
202 DEPENDS:one = "foo"
203
204 You should follow the same strategy when using ``:append``
205 and ``:prepend`` operations::
206
207 DEPENDS:append:one = " foo"
208 DEPENDS:prepend:one = "foo "
209
210 As an actual example, here's a
211 snippet from the generic kernel include file ``linux-yocto.inc``,
212 wherein the kernel compile and link options are adjusted in the
213 case of a subset of the supported architectures::
214
215 DEPENDS:append:aarch64 = " libgcc"
216 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
217 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
218
219 DEPENDS:append:nios2 = " libgcc"
220 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
221 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
222
223 DEPENDS:append:arc = " libgcc"
224 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
225 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
226
227 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
228
229 - *Place Machine-Specific Files in Machine-Specific Locations:* When
230 you have a base recipe, such as ``base-files.bb``, that contains a
231 :term:`SRC_URI` statement to a
232 file, you can use an append file to cause the build to use your
233 own version of the file. For example, an append file in your layer
234 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
235 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
236
237 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
238
239 The build for machine "one" will pick up your machine-specific file as
240 long as you have the file in
241 ``meta-one/recipes-core/base-files/base-files/``. However, if you
242 are building for a different machine and the ``bblayers.conf``
243 file includes the ``meta-one`` layer and the location of your
244 machine-specific file is the first location where that file is
245 found according to :term:`FILESPATH`, builds for all machines will
246 also use that machine-specific file.
247
248 You can make sure that a machine-specific file is used for a
249 particular machine by putting the file in a subdirectory specific
250 to the machine. For example, rather than placing the file in
251 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
252 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
253 Not only does this make sure the file is used only when building
254 for machine "one", but the build process locates the file more
255 quickly.
256
257 In summary, you need to place all files referenced from
258 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
259 order to restrict those files to machine-specific builds.
260
261- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
262 permission to use the Yocto Project Compatibility logo with your
263 layer or application that uses your layer, perform the steps to apply
264 for compatibility. See the
265 ":ref:`dev-manual/layers:making sure your layer is compatible with yocto project`"
266 section for more information.
267
268- *Follow the Layer Naming Convention:* Store custom layers in a Git
269 repository that use the ``meta-layer_name`` format.
270
271- *Group Your Layers Locally:* Clone your repository alongside other
272 cloned ``meta`` directories from the :term:`Source Directory`.
273
274Making Sure Your Layer is Compatible With Yocto Project
275=======================================================
276
277When you create a layer used with the Yocto Project, it is advantageous
278to make sure that the layer interacts well with existing Yocto Project
279layers (i.e. the layer is compatible with the Yocto Project). Ensuring
280compatibility makes the layer easy to be consumed by others in the Yocto
281Project community and could allow you permission to use the Yocto
282Project Compatible Logo.
283
284.. note::
285
286 Only Yocto Project member organizations are permitted to use the
287 Yocto Project Compatible Logo. The logo is not available for general
288 use. For information on how to become a Yocto Project member
289 organization, see the :yocto_home:`Yocto Project Website <>`.
290
291The Yocto Project Compatibility Program consists of a layer application
292process that requests permission to use the Yocto Project Compatibility
293Logo for your layer and application. The process consists of two parts:
294
2951. Successfully passing a script (``yocto-check-layer``) that when run
296 against your layer, tests it against constraints based on experiences
297 of how layers have worked in the real world and where pitfalls have
298 been found. Getting a "PASS" result from the script is required for
299 successful compatibility registration.
300
3012. Completion of an application acceptance form, which you can find at
302 :yocto_home:`/webform/yocto-project-compatible-registration`.
303
304To be granted permission to use the logo, you need to satisfy the
305following:
306
307- Be able to check the box indicating that you got a "PASS" when
308 running the script against your layer.
309
310- Answer "Yes" to the questions on the form or have an acceptable
311 explanation for any questions answered "No".
312
313- Be a Yocto Project Member Organization.
314
315The remainder of this section presents information on the registration
316form and on the ``yocto-check-layer`` script.
317
318Yocto Project Compatible Program Application
319--------------------------------------------
320
321Use the form to apply for your layer's approval. Upon successful
322application, you can use the Yocto Project Compatibility Logo with your
323layer and the application that uses your layer.
324
325To access the form, use this link:
326:yocto_home:`/webform/yocto-project-compatible-registration`.
327Follow the instructions on the form to complete your application.
328
329The application consists of the following sections:
330
331- *Contact Information:* Provide your contact information as the fields
332 require. Along with your information, provide the released versions
333 of the Yocto Project for which your layer is compatible.
334
335- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
336 items in the checklist. There is space at the bottom of the form for
337 any explanations for items for which you answered "No".
338
339- *Recommendations:* Provide answers for the questions regarding Linux
340 kernel use and build success.
341
342``yocto-check-layer`` Script
343----------------------------
344
345The ``yocto-check-layer`` script provides you a way to assess how
346compatible your layer is with the Yocto Project. You should run this
347script prior to using the form to apply for compatibility as described
348in the previous section. You need to achieve a "PASS" result in order to
349have your application form successfully processed.
350
351The script divides tests into three areas: COMMON, BSP, and DISTRO. For
352example, given a distribution layer (DISTRO), the layer must pass both
353the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
354layer, the layer must pass the COMMON and BSP set of tests.
355
356To execute the script, enter the following commands from your build
357directory::
358
359 $ source oe-init-build-env
360 $ yocto-check-layer your_layer_directory
361
362Be sure to provide the actual directory for your
363layer as part of the command.
364
365Entering the command causes the script to determine the type of layer
366and then to execute a set of specific tests against the layer. The
367following list overviews the test:
368
369- ``common.test_readme``: Tests if a ``README`` file exists in the
370 layer and the file is not empty.
371
372- ``common.test_parse``: Tests to make sure that BitBake can parse the
373 files without error (i.e. ``bitbake -p``).
374
375- ``common.test_show_environment``: Tests that the global or per-recipe
376 environment is in order without errors (i.e. ``bitbake -e``).
377
378- ``common.test_world``: Verifies that ``bitbake world`` works.
379
380- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
381 layers do not come with recipes that change signatures.
382
383- ``common.test_layerseries_compat``: Verifies layer compatibility is
384 set properly.
385
386- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
387 configurations.
388
389- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
390 set the machine when the layer is added.
391
392- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
393 regardless of which machine is selected.
394
395- ``bsp.test_machine_signatures``: Verifies that building for a
396 particular machine affects only the signature of tasks specific to
397 that machine.
398
399- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
400 distro configurations.
401
402- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
403 does not set the distribution when the layer is added.
404
405Enabling Your Layer
406===================
407
408Before the OpenEmbedded build system can use your new layer, you need to
409enable it. To enable your layer, simply add your layer's path to the
410:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
411found in the :term:`Build Directory`. The following example shows how to
412enable your new ``meta-mylayer`` layer (note how your new layer exists
413outside of the official ``poky`` repository which you would have checked
414out earlier)::
415
416 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
417 # changes incompatibly
418 POKY_BBLAYERS_CONF_VERSION = "2"
419 BBPATH = "${TOPDIR}"
420 BBFILES ?= ""
421 BBLAYERS ?= " \
422 /home/user/poky/meta \
423 /home/user/poky/meta-poky \
424 /home/user/poky/meta-yocto-bsp \
425 /home/user/mystuff/meta-mylayer \
426 "
427
428BitBake parses each ``conf/layer.conf`` file from the top down as
429specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
430file. During the processing of each ``conf/layer.conf`` file, BitBake
431adds the recipes, classes and configurations contained within the
432particular layer to the source directory.
433
434Appending Other Layers Metadata With Your Layer
435===============================================
436
437A recipe that appends Metadata to another recipe is called a BitBake
438append file. A BitBake append file uses the ``.bbappend`` file type
439suffix, while the corresponding recipe to which Metadata is being
440appended uses the ``.bb`` file type suffix.
441
442You can use a ``.bbappend`` file in your layer to make additions or
443changes to the content of another layer's recipe without having to copy
444the other layer's recipe into your layer. Your ``.bbappend`` file
445resides in your layer, while the main ``.bb`` recipe file to which you
446are appending Metadata resides in a different layer.
447
448Being able to append information to an existing recipe not only avoids
449duplication, but also automatically applies recipe changes from a
450different layer into your layer. If you were copying recipes, you would
451have to manually merge changes as they occur.
452
453When you create an append file, you must use the same root name as the
454corresponding recipe file. For example, the append file
455``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
456means the original recipe and append filenames are version
457number-specific. If the corresponding recipe is renamed to update to a
458newer version, you must also rename and possibly update the
459corresponding ``.bbappend`` as well. During the build process, BitBake
460displays an error on starting if it detects a ``.bbappend`` file that
461does not have a corresponding recipe with a matching name. See the
462:term:`BB_DANGLINGAPPENDS_WARNONLY`
463variable for information on how to handle this error.
464
465Overlaying a File Using Your Layer
466----------------------------------
467
468As an example, consider the main formfactor recipe and a corresponding
469formfactor append file both from the :term:`Source Directory`.
470Here is the main
471formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
472the "meta" layer at ``meta/recipes-bsp/formfactor``::
473
474 SUMMARY = "Device formfactor information"
475 DESCRIPTION = "A formfactor configuration file provides information about the \
476 target hardware for which the image is being built and information that the \
477 build system cannot obtain from other sources such as the kernel."
478 SECTION = "base"
479 LICENSE = "MIT"
480 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
481 PR = "r45"
482
483 SRC_URI = "file://config file://machconfig"
484 S = "${WORKDIR}"
485
486 PACKAGE_ARCH = "${MACHINE_ARCH}"
487 INHIBIT_DEFAULT_DEPS = "1"
488
489 do_install() {
490 # Install file only if it has contents
491 install -d ${D}${sysconfdir}/formfactor/
492 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
493 if [ -s "${S}/machconfig" ]; then
494 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
495 fi
496 }
497
498In the main recipe, note the :term:`SRC_URI`
499variable, which tells the OpenEmbedded build system where to find files
500during the build.
501
502Following is the append file, which is named ``formfactor_0.0.bbappend``
503and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
504file is in the layer at ``recipes-bsp/formfactor``::
505
506 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
507
508By default, the build system uses the
509:term:`FILESPATH` variable to
510locate files. This append file extends the locations by setting the
511:term:`FILESEXTRAPATHS`
512variable. Setting this variable in the ``.bbappend`` file is the most
513reliable and recommended method for adding directories to the search
514path used by the build system to find files.
515
516The statement in this example extends the directories to include
517``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
518which resolves to a directory named ``formfactor`` in the same directory
519in which the append file resides (i.e.
520``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
521have the supporting directory structure set up that will contain any
522files or patches you will be including from the layer.
523
524Using the immediate expansion assignment operator ``:=`` is important
525because of the reference to :term:`THISDIR`. The trailing colon character is
526important as it ensures that items in the list remain colon-separated.
527
528.. note::
529
530 BitBake automatically defines the :term:`THISDIR` variable. You should
531 never set this variable yourself. Using ":prepend" as part of the
532 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
533 paths in the final list.
534
535 Also, not all append files add extra files. Many append files simply
536 allow to add build options (e.g. ``systemd``). For these cases, your
537 append file would not even use the :term:`FILESEXTRAPATHS` statement.
538
539The end result of this ``.bbappend`` file is that on a Raspberry Pi, where
540``rpi`` will exist in the list of :term:`OVERRIDES`, the file
541``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be
542used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in
543:ref:`ref-tasks-install` will return true, and the file will be installed.
544
545Installing Additional Files Using Your Layer
546--------------------------------------------
547
548As another example, consider the main ``xserver-xf86-config`` recipe and a
549corresponding ``xserver-xf86-config`` append file both from the :term:`Source
550Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named
551``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at
552``meta/recipes-graphics/xorg-xserver``::
553
554 SUMMARY = "X.Org X server configuration file"
555 HOMEPAGE = "http://www.x.org"
556 SECTION = "x11/base"
557 LICENSE = "MIT"
558 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
559 PR = "r33"
560
561 SRC_URI = "file://xorg.conf"
562
563 S = "${WORKDIR}"
564
565 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf"
566
567 PACKAGE_ARCH = "${MACHINE_ARCH}"
568 ALLOW_EMPTY:${PN} = "1"
569
570 do_install () {
571 if test -s ${WORKDIR}/xorg.conf; then
572 install -d ${D}/${sysconfdir}/X11
573 install -m 0644 ${WORKDIR}/xorg.conf ${D}/${sysconfdir}/X11/
574 fi
575 }
576
577Following is the append file, which is named ``xserver-xf86-config_%.bbappend``
578and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
579file is in the layer at ``recipes-graphics/xorg-xserver``::
580
581 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
582
583 SRC_URI:append:rpi = " \
584 file://xorg.conf.d/98-pitft.conf \
585 file://xorg.conf.d/99-calibration.conf \
586 "
587 do_install:append:rpi () {
588 PITFT="${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}"
589 if [ "${PITFT}" = "1" ]; then
590 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/
591 install -m 0644 ${WORKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
592 install -m 0644 ${WORKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
593 fi
594 }
595
596 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*"
597
598Building off of the previous example, we once again are setting the
599:term:`FILESEXTRAPATHS` variable. In this case we are also using
600:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in
601the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a
602check for an additional :term:`MACHINE_FEATURES` that if set will cause these
603additional files to be installed. These additional files are listed in
604:term:`FILES` so that they will be packaged.
605
606Prioritizing Your Layer
607=======================
608
609Each layer is assigned a priority value. Priority values control which
610layer takes precedence if there are recipe files with the same name in
611multiple layers. For these cases, the recipe file from the layer with a
612higher priority number takes precedence. Priority values also affect the
613order in which multiple ``.bbappend`` files for the same recipe are
614applied. You can either specify the priority manually, or allow the
615build system to calculate it based on the layer's dependencies.
616
617To specify the layer's priority manually, use the
618:term:`BBFILE_PRIORITY`
619variable and append the layer's root name::
620
621 BBFILE_PRIORITY_mylayer = "1"
622
623.. note::
624
625 It is possible for a recipe with a lower version number
626 :term:`PV` in a layer that has a higher
627 priority to take precedence.
628
629 Also, the layer priority does not currently affect the precedence
630 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
631 might address this.
632
633Managing Layers
634===============
635
636You can use the BitBake layer management tool ``bitbake-layers`` to
637provide a view into the structure of recipes across a multi-layer
638project. Being able to generate output that reports on configured layers
639with their paths and priorities and on ``.bbappend`` files and their
640applicable recipes can help to reveal potential problems.
641
642For help on the BitBake layer management tool, use the following
643command::
644
645 $ bitbake-layers --help
646
647The following list describes the available commands:
648
649- ``help:`` Displays general help or help on a specified command.
650
651- ``show-layers:`` Shows the current configured layers.
652
653- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
654 when a recipe with the same name exists in another layer that has a
655 higher layer priority.
656
657- ``show-recipes:`` Lists available recipes and the layers that
658 provide them.
659
660- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
661 which they apply.
662
663- ``show-cross-depends:`` Lists dependency relationships between
664 recipes that cross layer boundaries.
665
666- ``add-layer:`` Adds a layer to ``bblayers.conf``.
667
668- ``remove-layer:`` Removes a layer from ``bblayers.conf``
669
670- ``flatten:`` Flattens the layer configuration into a separate
671 output directory. Flattening your layer configuration builds a
672 "flattened" directory that contains the contents of all layers, with
673 any overlayed recipes removed and any ``.bbappend`` files appended to
674 the corresponding recipes. You might have to perform some manual
675 cleanup of the flattened layer as follows:
676
677 - Non-recipe files (such as patches) are overwritten. The flatten
678 command shows a warning for these files.
679
680 - Anything beyond the normal layer setup has been added to the
681 ``layer.conf`` file. Only the lowest priority layer's
682 ``layer.conf`` is used.
683
684 - Overridden and appended items from ``.bbappend`` files need to be
685 cleaned up. The contents of each ``.bbappend`` end up in the
686 flattened recipe. However, if there are appended or changed
687 variable values, you need to tidy these up yourself. Consider the
688 following example. Here, the ``bitbake-layers`` command adds the
689 line ``#### bbappended ...`` so that you know where the following
690 lines originate::
691
692 ...
693 DESCRIPTION = "A useful utility"
694 ...
695 EXTRA_OECONF = "--enable-something"
696 ...
697
698 #### bbappended from meta-anotherlayer ####
699
700 DESCRIPTION = "Customized utility"
701 EXTRA_OECONF += "--enable-somethingelse"
702
703
704 Ideally, you would tidy up these utilities as follows::
705
706 ...
707 DESCRIPTION = "Customized utility"
708 ...
709 EXTRA_OECONF = "--enable-something --enable-somethingelse"
710 ...
711
712- ``layerindex-fetch``: Fetches a layer from a layer index, along
713 with its dependent layers, and adds the layers to the
714 ``conf/bblayers.conf`` file.
715
716- ``layerindex-show-depends``: Finds layer dependencies from the
717 layer index.
718
719- ``save-build-conf``: Saves the currently active build configuration
720 (``conf/local.conf``, ``conf/bblayers.conf``) as a template into a layer.
721 This template can later be used for setting up builds via :term:``TEMPLATECONF``.
722 For information about saving and using configuration templates, see
723 ":ref:`dev-manual/custom-template-configuration-directory:creating a custom template configuration directory`".
724
725- ``create-layer``: Creates a basic layer.
726
727- ``create-layers-setup``: Writes out a configuration file and/or a script that
728 can replicate the directory structure and revisions of the layers in a current build.
729 For more information, see ":ref:`dev-manual/layers:saving and restoring the layers setup`".
730
731Creating a General Layer Using the ``bitbake-layers`` Script
732============================================================
733
734The ``bitbake-layers`` script with the ``create-layer`` subcommand
735simplifies creating a new general layer.
736
737.. note::
738
739 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
740 section in the Yocto
741 Project Board Specific (BSP) Developer's Guide.
742
743 - In order to use a layer with the OpenEmbedded build system, you
744 need to add the layer to your ``bblayers.conf`` configuration
745 file. See the ":ref:`dev-manual/layers:adding a layer using the \`\`bitbake-layers\`\` script`"
746 section for more information.
747
748The default mode of the script's operation with this subcommand is to
749create a layer with the following:
750
751- A layer priority of 6.
752
753- A ``conf`` subdirectory that contains a ``layer.conf`` file.
754
755- A ``recipes-example`` subdirectory that contains a further
756 subdirectory named ``example``, which contains an ``example.bb``
757 recipe file.
758
759- A ``COPYING.MIT``, which is the license statement for the layer. The
760 script assumes you want to use the MIT license, which is typical for
761 most layers, for the contents of the layer itself.
762
763- A ``README`` file, which is a file describing the contents of your
764 new layer.
765
766In its simplest form, you can use the following command form to create a
767layer. The command creates a layer whose name corresponds to
768"your_layer_name" in the current directory::
769
770 $ bitbake-layers create-layer your_layer_name
771
772As an example, the following command creates a layer named ``meta-scottrif``
773in your home directory::
774
775 $ cd /usr/home
776 $ bitbake-layers create-layer meta-scottrif
777 NOTE: Starting bitbake server...
778 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
779
780If you want to set the priority of the layer to other than the default
781value of "6", you can either use the ``--priority`` option or you
782can edit the
783:term:`BBFILE_PRIORITY` value
784in the ``conf/layer.conf`` after the script creates it. Furthermore, if
785you want to give the example recipe file some name other than the
786default, you can use the ``--example-recipe-name`` option.
787
788The easiest way to see how the ``bitbake-layers create-layer`` command
789works is to experiment with the script. You can also read the usage
790information by entering the following::
791
792 $ bitbake-layers create-layer --help
793 NOTE: Starting bitbake server...
794 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
795 [--example-recipe-name EXAMPLERECIPE]
796 layerdir
797
798 Create a basic layer
799
800 positional arguments:
801 layerdir Layer directory to create
802
803 optional arguments:
804 -h, --help show this help message and exit
805 --priority PRIORITY, -p PRIORITY
806 Layer directory to create
807 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
808 Filename of the example recipe
809
810Adding a Layer Using the ``bitbake-layers`` Script
811==================================================
812
813Once you create your general layer, you must add it to your
814``bblayers.conf`` file. Adding the layer to this configuration file
815makes the OpenEmbedded build system aware of your layer so that it can
816search it for metadata.
817
818Add your layer by using the ``bitbake-layers add-layer`` command::
819
820 $ bitbake-layers add-layer your_layer_name
821
822Here is an example that adds a
823layer named ``meta-scottrif`` to the configuration file. Following the
824command that adds the layer is another ``bitbake-layers`` command that
825shows the layers that are in your ``bblayers.conf`` file::
826
827 $ bitbake-layers add-layer meta-scottrif
828 NOTE: Starting bitbake server...
829 Parsing recipes: 100% |##########################################################| Time: 0:00:49
830 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
831 $ bitbake-layers show-layers
832 NOTE: Starting bitbake server...
833 layer path priority
834 ==========================================================================
835 meta /home/scottrif/poky/meta 5
836 meta-poky /home/scottrif/poky/meta-poky 5
837 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
838 workspace /home/scottrif/poky/build/workspace 99
839 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
840
841
842Adding the layer to this file
843enables the build system to locate the layer during the build.
844
845.. note::
846
847 During a build, the OpenEmbedded build system looks in the layers
848 from the top of the list down to the bottom in that order.
849
850Saving and restoring the layers setup
851=====================================
852
853Once you have a working build with the correct set of layers, it is beneficial
854to capture the layer setup --- what they are, which repositories they come from
855and which SCM revisions they're at --- into a configuration file, so that this
856setup can be easily replicated later, perhaps on a different machine. Here's
857how to do this::
858
859 $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/
860 NOTE: Starting bitbake server...
861 NOTE: Created /srv/work/alex/meta-alex/setup-layers.json
862 NOTE: Created /srv/work/alex/meta-alex/setup-layers
863
864The tool needs a single argument which tells where to place the output, consisting
865of a json formatted layer configuration, and a ``setup-layers`` script that can use that configuration
866to restore the layers in a different location, or on a different host machine. The argument
867can point to a custom layer (which is then deemed a "bootstrap" layer that needs to be
868checked out first), or into a completely independent location.
869
870The replication of the layers is performed by running the ``setup-layers`` script provided
871above:
872
8731. Clone the bootstrap layer or some other repository to obtain
874 the json config and the setup script that can use it.
875
8762. Run the script directly with no options::
877
878 alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers
879 Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override.
880
881 Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master
882 Running 'git init -q /srv/work/alex/my-build/meta-intel'
883 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel
884 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel
885 Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel
886
887 Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers
888 Running 'git init -q /srv/work/alex/my-build/poky'
889 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky
890 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky
891 Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky
892 Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky
893 Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky
894
895.. note::
896 This will work to update an existing checkout as well.
897
898.. note::
899 The script is self-sufficient and requires only python3
900 and git on the build machine.
901
902.. note::
903 Both the ``create-layers-setup`` and the ``setup-layers`` provided several additional options
904 that customize their behavior - you are welcome to study them via ``--help`` command line parameter.
905
diff --git a/documentation/dev-manual/libraries.rst b/documentation/dev-manual/libraries.rst
new file mode 100644
index 0000000000..ae4ca27209
--- /dev/null
+++ b/documentation/dev-manual/libraries.rst
@@ -0,0 +1,267 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working With Libraries
4**********************
5
6Libraries are an integral part of your system. This section describes
7some common practices you might find helpful when working with libraries
8to build your system:
9
10- :ref:`How to include static library files
11 <dev-manual/libraries:including static library files>`
12
13- :ref:`How to use the Multilib feature to combine multiple versions of
14 library files into a single image
15 <dev-manual/libraries:combining multiple versions of library files into one image>`
16
17- :ref:`How to install multiple versions of the same library in parallel on
18 the same system
19 <dev-manual/libraries:installing multiple versions of the same library>`
20
21Including Static Library Files
22==============================
23
24If you are building a library and the library offers static linking, you
25can control which static library files (``*.a`` files) get included in
26the built library.
27
28The :term:`PACKAGES` and
29:term:`FILES:* <FILES>` variables in the
30``meta/conf/bitbake.conf`` configuration file define how files installed
31by the :ref:`ref-tasks-install` task are packaged. By default, the :term:`PACKAGES`
32variable includes ``${PN}-staticdev``, which represents all static
33library files.
34
35.. note::
36
37 Some previously released versions of the Yocto Project defined the
38 static library files through ``${PN}-dev``.
39
40Following is part of the BitBake configuration file, where you can see
41how the static library files are defined::
42
43 PACKAGE_BEFORE_PN ?= ""
44 PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
45 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
46 FILES = ""
47
48 FILES:${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
49 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
50 ${base_bindir}/* ${base_sbindir}/* \
51 ${base_libdir}/*${SOLIBS} \
52 ${base_prefix}/lib/udev ${prefix}/lib/udev \
53 ${base_libdir}/udev ${libdir}/udev \
54 ${datadir}/${BPN} ${libdir}/${BPN}/* \
55 ${datadir}/pixmaps ${datadir}/applications \
56 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
57 ${libdir}/bonobo/servers"
58
59 FILES:${PN}-bin = "${bindir}/* ${sbindir}/*"
60
61 FILES:${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
62 ${datadir}/gnome/help"
63 SECTION:${PN}-doc = "doc"
64
65 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
66 FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
67 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
68 ${datadir}/aclocal ${base_libdir}/*.o \
69 ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
70 ${libdir}/cmake ${datadir}/cmake"
71 SECTION:${PN}-dev = "devel"
72 ALLOW_EMPTY:${PN}-dev = "1"
73 RDEPENDS:${PN}-dev = "${PN} (= ${EXTENDPKGV})"
74
75 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
76 SECTION:${PN}-staticdev = "devel"
77 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
78
79Combining Multiple Versions of Library Files into One Image
80===========================================================
81
82The build system offers the ability to build libraries with different
83target optimizations or architecture formats and combine these together
84into one system image. You can link different binaries in the image
85against the different libraries as needed for specific use cases. This
86feature is called "Multilib".
87
88An example would be where you have most of a system compiled in 32-bit
89mode using 32-bit libraries, but you have something large, like a
90database engine, that needs to be a 64-bit application and uses 64-bit
91libraries. Multilib allows you to get the best of both 32-bit and 64-bit
92libraries.
93
94While the Multilib feature is most commonly used for 32 and 64-bit
95differences, the approach the build system uses facilitates different
96target optimizations. You could compile some binaries to use one set of
97libraries and other binaries to use a different set of libraries. The
98libraries could differ in architecture, compiler options, or other
99optimizations.
100
101There are several examples in the ``meta-skeleton`` layer found in the
102:term:`Source Directory`:
103
104- :oe_git:`conf/multilib-example.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example.conf>`
105 configuration file.
106
107- :oe_git:`conf/multilib-example2.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example2.conf>`
108 configuration file.
109
110- :oe_git:`recipes-multilib/images/core-image-multilib-example.bb </openembedded-core/tree/meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb>`
111 recipe
112
113Preparing to Use Multilib
114-------------------------
115
116User-specific requirements drive the Multilib feature. Consequently,
117there is no one "out-of-the-box" configuration that would
118meet your needs.
119
120In order to enable Multilib, you first need to ensure your recipe is
121extended to support multiple libraries. Many standard recipes are
122already extended and support multiple libraries. You can check in the
123``meta/conf/multilib.conf`` configuration file in the
124:term:`Source Directory` to see how this is
125done using the
126:term:`BBCLASSEXTEND` variable.
127Eventually, all recipes will be covered and this list will not be
128needed.
129
130For the most part, the :ref:`Multilib <ref-classes-multilib*>`
131class extension works automatically to
132extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
133:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
134Standard variables such as
135:term:`DEPENDS`,
136:term:`RDEPENDS`,
137:term:`RPROVIDES`,
138:term:`RRECOMMENDS`,
139:term:`PACKAGES`, and
140:term:`PACKAGES_DYNAMIC` are
141automatically extended by the system. If you are extending any manual
142code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
143those names are extended correctly.
144
145Using Multilib
146--------------
147
148After you have set up the recipes, you need to define the actual
149combination of multiple libraries you want to build. You accomplish this
150through your ``local.conf`` configuration file in the
151:term:`Build Directory`. An example configuration would be as follows::
152
153 MACHINE = "qemux86-64"
154 require conf/multilib.conf
155 MULTILIBS = "multilib:lib32"
156 DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
157 IMAGE_INSTALL:append = " lib32-glib-2.0"
158
159This example enables an additional library named
160``lib32`` alongside the normal target packages. When combining these
161"lib32" alternatives, the example uses "x86" for tuning. For information
162on this particular tuning, see
163``meta/conf/machine/include/ia32/arch-ia32.inc``.
164
165The example then includes ``lib32-glib-2.0`` in all the images, which
166illustrates one method of including a multiple library dependency. You
167can use a normal image build to include this dependency, for example::
168
169 $ bitbake core-image-sato
170
171You can also build Multilib packages
172specifically with a command like this::
173
174 $ bitbake lib32-glib-2.0
175
176Additional Implementation Details
177---------------------------------
178
179There are generic implementation details as well as details that are specific to
180package management systems. Following are implementation details
181that exist regardless of the package management system:
182
183- The typical convention used for the class extension code as used by
184 Multilib assumes that all package names specified in
185 :term:`PACKAGES` that contain
186 ``${PN}`` have ``${PN}`` at the start of the name. When that
187 convention is not followed and ``${PN}`` appears at the middle or the
188 end of a name, problems occur.
189
190- The :term:`TARGET_VENDOR`
191 value under Multilib will be extended to "-vendormlmultilib" (e.g.
192 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
193 slightly unwieldy contraction is that any "-" characters in the
194 vendor string presently break Autoconf's ``config.sub``, and other
195 separators are problematic for different reasons.
196
197Here are the implementation details for the RPM Package Management System:
198
199- A unique architecture is defined for the Multilib packages, along
200 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
201 :term:`Build Directory`. For example, consider ``lib32`` in a
202 ``qemux86-64`` image. The possible architectures in the system are "all",
203 "qemux86_64", "lib32:qemux86_64", and "lib32:x86".
204
205- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
206 packaging. The naming for a normal RPM package and a Multilib RPM
207 package in a ``qemux86-64`` system resolves to something similar to
208 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
209 respectively.
210
211- When installing a Multilib image, the RPM backend first installs the
212 base image and then installs the Multilib libraries.
213
214- The build system relies on RPM to resolve the identical files in the
215 two (or more) Multilib packages.
216
217Here are the implementation details for the IPK Package Management System:
218
219- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
220 packaging. The naming for a normal RPM package and a Multilib IPK
221 package in a ``qemux86-64`` system resolves to something like
222 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``,
223 respectively.
224
225- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
226 packages with and without the Multilib feature can exist in the same
227 folder due to the ``${PN}`` differences.
228
229- IPK defines a sanity check for Multilib installation using certain
230 rules for file comparison, overridden, etc.
231
232Installing Multiple Versions of the Same Library
233================================================
234
235There are be situations where you need to install and use multiple versions
236of the same library on the same system at the same time. This
237almost always happens when a library API changes and you have
238multiple pieces of software that depend on the separate versions of the
239library. To accommodate these situations, you can install multiple
240versions of the same library in parallel on the same system.
241
242The process is straightforward as long as the libraries use proper
243versioning. With properly versioned libraries, all you need to do to
244individually specify the libraries is create separate, appropriately
245named recipes where the :term:`PN` part of
246the name includes a portion that differentiates each library version
247(e.g. the major part of the version number). Thus, instead of having a
248single recipe that loads one version of a library (e.g. ``clutter``),
249you provide multiple recipes that result in different versions of the
250libraries you want. As an example, the following two recipes would allow
251the two separate versions of the ``clutter`` library to co-exist on the
252same system:
253
254.. code-block:: none
255
256 clutter-1.6_1.6.20.bb
257 clutter-1.8_1.8.4.bb
258
259Additionally, if
260you have other recipes that depend on a given library, you need to use
261the :term:`DEPENDS` variable to
262create the dependency. Continuing with the same example, if you want to
263have a recipe depend on the 1.8 version of the ``clutter`` library, use
264the following in your recipe::
265
266 DEPENDS = "clutter-1.8"
267
diff --git a/documentation/dev-manual/licenses.rst b/documentation/dev-manual/licenses.rst
new file mode 100644
index 0000000000..0db193f7e4
--- /dev/null
+++ b/documentation/dev-manual/licenses.rst
@@ -0,0 +1,520 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working With Licenses
4*********************
5
6As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
7section in the Yocto Project Overview and Concepts Manual, open source
8projects are open to the public and they consequently have different
9licensing structures in place. This section describes the mechanism by
10which the :term:`OpenEmbedded Build System`
11tracks changes to
12licensing text and covers how to maintain open source license compliance
13during your project's lifecycle. The section also describes how to
14enable commercially licensed recipes, which by default are disabled.
15
16Tracking License Changes
17========================
18
19The license of an upstream project might change in the future. In order
20to prevent these changes going unnoticed, the
21:term:`LIC_FILES_CHKSUM`
22variable tracks changes to the license text. The checksums are validated
23at the end of the configure step, and if the checksums do not match, the
24build will fail.
25
26Specifying the ``LIC_FILES_CHKSUM`` Variable
27--------------------------------------------
28
29The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
30in the source code for the recipe. Following is an example of how to
31specify :term:`LIC_FILES_CHKSUM`::
32
33 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
34 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
35 file://licfile2.txt;endline=50;md5=zzzz \
36 ..."
37
38.. note::
39
40 - When using "beginline" and "endline", realize that line numbering
41 begins with one and not zero. Also, the included lines are
42 inclusive (i.e. lines five through and including 29 in the
43 previous example for ``licfile1.txt``).
44
45 - When a license check fails, the selected license text is included
46 as part of the QA message. Using this output, you can determine
47 the exact start and finish for the needed license text.
48
49The build system uses the :term:`S`
50variable as the default directory when searching files listed in
51:term:`LIC_FILES_CHKSUM`. The previous example employs the default
52directory.
53
54Consider this next example::
55
56 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
57 md5=bb14ed3c4cda583abc85401304b5cd4e"
58 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
59
60The first line locates a file in ``${S}/src/ls.c`` and isolates lines
61five through 16 as license text. The second line refers to a file in
62:term:`WORKDIR`.
63
64Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
65unless the :term:`LICENSE` variable is set to "CLOSED".
66
67Explanation of Syntax
68---------------------
69
70As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
71lists all the important files that contain the license text for the
72source code. It is possible to specify a checksum for an entire file, or
73a specific section of a file (specified by beginning and ending line
74numbers with the "beginline" and "endline" parameters, respectively).
75The latter is useful for source files with a license notice header,
76README documents, and so forth. If you do not use the "beginline"
77parameter, then it is assumed that the text begins on the first line of
78the file. Similarly, if you do not use the "endline" parameter, it is
79assumed that the license text ends with the last line of the file.
80
81The "md5" parameter stores the md5 checksum of the license text. If the
82license text changes in any way as compared to this parameter then a
83mismatch occurs. This mismatch triggers a build failure and notifies the
84developer. Notification allows the developer to review and address the
85license text changes. Also note that if a mismatch occurs during the
86build, the correct md5 checksum is placed in the build log and can be
87easily copied to the recipe.
88
89There is no limit to how many files you can specify using the
90:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
91requires a few specifications for license tracking. Many projects have a
92"COPYING" file that stores the license information for all the source
93code files. This practice allows you to just track the "COPYING" file as
94long as it is kept up to date.
95
96.. note::
97
98 - If you specify an empty or invalid "md5" parameter,
99 :term:`BitBake` returns an md5
100 mis-match error and displays the correct "md5" parameter value
101 during the build. The correct parameter is also captured in the
102 build log.
103
104 - If the whole file contains only license text, you do not need to
105 use the "beginline" and "endline" parameters.
106
107Enabling Commercially Licensed Recipes
108======================================
109
110By default, the OpenEmbedded build system disables components that have
111commercial or other special licensing requirements. Such requirements
112are defined on a recipe-by-recipe basis through the
113:term:`LICENSE_FLAGS` variable
114definition in the affected recipe. For instance, the
115``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
116contains the following statement::
117
118 LICENSE_FLAGS = "commercial"
119
120Here is a
121slightly more complicated example that contains both an explicit recipe
122name and version (after variable expansion)::
123
124 LICENSE_FLAGS = "license_${PN}_${PV}"
125
126In order for a component restricted by a
127:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
128needs to have a matching entry in the global
129:term:`LICENSE_FLAGS_ACCEPTED`
130variable, which is a variable typically defined in your ``local.conf``
131file. For example, to enable the
132``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
133could add either the string "commercial_gst-plugins-ugly" or the more
134general string "commercial" to :term:`LICENSE_FLAGS_ACCEPTED`. See the
135":ref:`dev-manual/licenses:license flag matching`" section for a full
136explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
137example::
138
139 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly"
140
141Likewise, to additionally enable the package built from the recipe
142containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
143the actual recipe name was ``emgd_1.10.bb``, the following string would
144enable that package as well as the original ``gst-plugins-ugly``
145package::
146
147 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly license_emgd_1.10"
148
149As a convenience, you do not need to specify the
150complete license string for every package. You can use
151an abbreviated form, which consists of just the first portion or
152portions of the license string before the initial underscore character
153or characters. A partial string will match any license that contains the
154given string as the first portion of its license. For example, the
155following value will also match both of the packages
156previously mentioned as well as any other packages that have licenses
157starting with "commercial" or "license".
158::
159
160 LICENSE_FLAGS_ACCEPTED = "commercial license"
161
162License Flag Matching
163---------------------
164
165License flag matching allows you to control what recipes the
166OpenEmbedded build system includes in the build. Fundamentally, the
167build system attempts to match :term:`LICENSE_FLAGS` strings found in
168recipes against strings found in :term:`LICENSE_FLAGS_ACCEPTED`.
169A match causes the build system to include a recipe in the
170build, while failure to find a match causes the build system to exclude
171a recipe.
172
173In general, license flag matching is simple. However, understanding some
174concepts will help you correctly and effectively use matching.
175
176Before a flag defined by a particular recipe is tested against the
177entries of :term:`LICENSE_FLAGS_ACCEPTED`, the expanded
178string ``_${PN}`` is appended to the flag. This expansion makes each
179:term:`LICENSE_FLAGS` value recipe-specific. After expansion, the
180string is then matched against the entries. Thus, specifying
181``LICENSE_FLAGS = "commercial"`` in recipe "foo", for example, results
182in the string ``"commercial_foo"``. And, to create a match, that string
183must appear among the entries of :term:`LICENSE_FLAGS_ACCEPTED`.
184
185Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
186:term:`LICENSE_FLAGS_ACCEPTED` variable allows you a lot of flexibility for
187including or excluding recipes based on licensing. For example, you can
188broaden the matching capabilities by using license flags string subsets
189in :term:`LICENSE_FLAGS_ACCEPTED`.
190
191.. note::
192
193 When using a string subset, be sure to use the part of the expanded
194 string that precedes the appended underscore character (e.g.
195 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
196
197For example, simply specifying the string "commercial" in the
198:term:`LICENSE_FLAGS_ACCEPTED` variable matches any expanded
199:term:`LICENSE_FLAGS` definition that starts with the string
200"commercial" such as "commercial_foo" and "commercial_bar", which
201are the strings the build system automatically generates for
202hypothetical recipes named "foo" and "bar" assuming those recipes simply
203specify the following::
204
205 LICENSE_FLAGS = "commercial"
206
207Thus, you can choose to exhaustively enumerate each license flag in the
208list and allow only specific recipes into the image, or you can use a
209string subset that causes a broader range of matches to allow a range of
210recipes into the image.
211
212This scheme works even if the :term:`LICENSE_FLAGS` string already has
213``_${PN}`` appended. For example, the build system turns the license
214flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
215both the general "commercial" and the specific "commercial_1.2_foo"
216strings found in the :term:`LICENSE_FLAGS_ACCEPTED` variable, as expected.
217
218Here are some other scenarios:
219
220- You can specify a versioned string in the recipe such as
221 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
222 string to "commercial_foo_1.2_foo". Combine this license flag with a
223 :term:`LICENSE_FLAGS_ACCEPTED` variable that has the string
224 "commercial" and you match the flag along with any other flag that
225 starts with the string "commercial".
226
227- Under the same circumstances, you can add "commercial_foo" in the
228 :term:`LICENSE_FLAGS_ACCEPTED` variable and the build system not only
229 matches "commercial_foo_1.2" but also matches any license flag with
230 the string "commercial_foo", regardless of the version.
231
232- You can be very specific and use both the package and version parts
233 in the :term:`LICENSE_FLAGS_ACCEPTED` list (e.g.
234 "commercial_foo_1.2") to specifically match a versioned recipe.
235
236Other Variables Related to Commercial Licenses
237----------------------------------------------
238
239There are other helpful variables related to commercial license handling,
240defined in the
241``poky/meta/conf/distro/include/default-distrovars.inc`` file::
242
243 COMMERCIAL_AUDIO_PLUGINS ?= ""
244 COMMERCIAL_VIDEO_PLUGINS ?= ""
245
246If you
247want to enable these components, you can do so by making sure you have
248statements similar to the following in your ``local.conf`` configuration
249file::
250
251 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
252 gst-plugins-ugly-mpegaudioparse"
253 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
254 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
255 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
256
257
258Of course, you could also create a matching list for those
259components using the more general "commercial" in the
260:term:`LICENSE_FLAGS_ACCEPTED` variable, but that would also enable all
261the other packages with :term:`LICENSE_FLAGS`
262containing "commercial", which you may or may not want::
263
264 LICENSE_FLAGS_ACCEPTED = "commercial"
265
266Specifying audio and video plugins as part of the
267``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
268(along with the enabling :term:`LICENSE_FLAGS_ACCEPTED`) includes the
269plugins or components into built images, thus adding support for media
270formats or components.
271
272Maintaining Open Source License Compliance During Your Product's Lifecycle
273==========================================================================
274
275One of the concerns for a development organization using open source
276software is how to maintain compliance with various open source
277licensing during the lifecycle of the product. While this section does
278not provide legal advice or comprehensively cover all scenarios, it does
279present methods that you can use to assist you in meeting the compliance
280requirements during a software release.
281
282With hundreds of different open source licenses that the Yocto Project
283tracks, it is difficult to know the requirements of each and every
284license. However, the requirements of the major FLOSS licenses can begin
285to be covered by assuming that there are three main areas of concern:
286
287- Source code must be provided.
288
289- License text for the software must be provided.
290
291- Compilation scripts and modifications to the source code must be
292 provided.
293
294There are other requirements beyond the scope of these three and the
295methods described in this section (e.g. the mechanism through which
296source code is distributed).
297
298As different organizations have different methods of complying with open
299source licensing, this section is not meant to imply that there is only
300one single way to meet your compliance obligations, but rather to
301describe one method of achieving compliance. The remainder of this
302section describes methods supported to meet the previously mentioned
303three requirements. Once you take steps to meet these requirements, and
304prior to releasing images, sources, and the build system, you should
305audit all artifacts to ensure completeness.
306
307.. note::
308
309 The Yocto Project generates a license manifest during image creation
310 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
311 to assist with any audits.
312
313Providing the Source Code
314-------------------------
315
316Compliance activities should begin before you generate the final image.
317The first thing you should look at is the requirement that tops the list
318for most compliance groups --- providing the source. The Yocto Project has
319a few ways of meeting this requirement.
320
321One of the easiest ways to meet this requirement is to provide the
322entire :term:`DL_DIR` used by the
323build. This method, however, has a few issues. The most obvious is the
324size of the directory since it includes all sources used in the build
325and not just the source used in the released image. It will include
326toolchain source, and other artifacts, which you would not generally
327release. However, the more serious issue for most companies is
328accidental release of proprietary software. The Yocto Project provides
329an :ref:`archiver <ref-classes-archiver>` class to
330help avoid some of these concerns.
331
332Before you employ :term:`DL_DIR` or the :ref:`archiver <ref-classes-archiver>` class, you need to
333decide how you choose to provide source. The source :ref:`archiver <ref-classes-archiver>` class
334can generate tarballs and SRPMs and can create them with various levels
335of compliance in mind.
336
337One way of doing this (but certainly not the only way) is to release
338just the source as a tarball. You can do this by adding the following to
339the ``local.conf`` file found in the :term:`Build Directory`::
340
341 INHERIT += "archiver"
342 ARCHIVER_MODE[src] = "original"
343
344During the creation of your
345image, the source from all recipes that deploy packages to the image is
346placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
347:term:`LICENSE` for each recipe.
348Releasing the entire directory enables you to comply with requirements
349concerning providing the unmodified source. It is important to note that
350the size of the directory can get large.
351
352A way to help mitigate the size issue is to only release tarballs for
353licenses that require the release of source. Let us assume you are only
354concerned with GPL code as identified by running the following script:
355
356.. code-block:: shell
357
358 # Script to archive a subset of packages matching specific license(s)
359 # Source and license files are copied into sub folders of package folder
360 # Must be run from build folder
361 #!/bin/bash
362 src_release_dir="source-release"
363 mkdir -p $src_release_dir
364 for a in tmp/deploy/sources/*; do
365 for d in $a/*; do
366 # Get package name from path
367 p=`basename $d`
368 p=${p%-*}
369 p=${p%-*}
370 # Only archive GPL packages (update *GPL* regex for your license check)
371 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
372 if [ $numfiles -ge 1 ]; then
373 echo Archiving $p
374 mkdir -p $src_release_dir/$p/source
375 cp $d/* $src_release_dir/$p/source 2> /dev/null
376 mkdir -p $src_release_dir/$p/license
377 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
378 fi
379 done
380 done
381
382At this point, you
383could create a tarball from the ``gpl_source_release`` directory and
384provide that to the end user. This method would be a step toward
385achieving compliance with section 3a of GPLv2 and with section 6 of
386GPLv3.
387
388Providing License Text
389----------------------
390
391One requirement that is often overlooked is inclusion of license text.
392This requirement also needs to be dealt with prior to generating the
393final image. Some licenses require the license text to accompany the
394binary. You can achieve this by adding the following to your
395``local.conf`` file::
396
397 COPY_LIC_MANIFEST = "1"
398 COPY_LIC_DIRS = "1"
399 LICENSE_CREATE_PACKAGE = "1"
400
401Adding these statements to the
402configuration file ensures that the licenses collected during package
403generation are included on your image.
404
405.. note::
406
407 Setting all three variables to "1" results in the image having two
408 copies of the same license file. One copy resides in
409 ``/usr/share/common-licenses`` and the other resides in
410 ``/usr/share/license``.
411
412 The reason for this behavior is because
413 :term:`COPY_LIC_DIRS` and
414 :term:`COPY_LIC_MANIFEST`
415 add a copy of the license when the image is built but do not offer a
416 path for adding licenses for newly installed packages to an image.
417 :term:`LICENSE_CREATE_PACKAGE`
418 adds a separate package and an upgrade path for adding licenses to an
419 image.
420
421As the source :ref:`archiver <ref-classes-archiver>` class has already archived the original
422unmodified source that contains the license files, you would have
423already met the requirements for inclusion of the license information
424with source as defined by the GPL and other open source licenses.
425
426Providing Compilation Scripts and Source Code Modifications
427-----------------------------------------------------------
428
429At this point, we have addressed all we need to prior to generating the
430image. The next two requirements are addressed during the final
431packaging of the release.
432
433By releasing the version of the OpenEmbedded build system and the layers
434used during the build, you will be providing both compilation scripts
435and the source code modifications in one step.
436
437If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
438and a distro layer, and those
439those layers are used to patch, compile, package, or modify (in any way)
440any open source software included in your released images, you might be
441required to release those layers under section 3 of GPLv2 or section 1
442of GPLv3. One way of doing that is with a clean checkout of the version
443of the Yocto Project and layers used during your build. Here is an
444example:
445
446.. code-block:: shell
447
448 # We built using the dunfell branch of the poky repo
449 $ git clone -b dunfell git://git.yoctoproject.org/poky
450 $ cd poky
451 # We built using the release_branch for our layers
452 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
453 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
454 # clean up the .git repos
455 $ find . -name ".git" -type d -exec rm -rf {} \;
456
457One thing a development organization might want to consider for end-user
458convenience is to modify
459``meta-poky/conf/templates/default/bblayers.conf.sample`` to ensure that when
460the end user utilizes the released build system to build an image, the
461development organization's layers are included in the ``bblayers.conf`` file
462automatically::
463
464 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
465 # changes incompatibly
466 POKY_BBLAYERS_CONF_VERSION = "2"
467
468 BBPATH = "${TOPDIR}"
469 BBFILES ?= ""
470
471 BBLAYERS ?= " \
472 ##OEROOT##/meta \
473 ##OEROOT##/meta-poky \
474 ##OEROOT##/meta-yocto-bsp \
475 ##OEROOT##/meta-mylayer \
476 "
477
478Creating and
479providing an archive of the :term:`Metadata`
480layers (recipes, configuration files, and so forth) enables you to meet
481your requirements to include the scripts to control compilation as well
482as any modifications to the original source.
483
484Compliance Limitations with Executables Built from Static Libraries
485-------------------------------------------------------------------
486
487When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS`
488mechanisms as well as explicitly included in the image recipe with
489:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B
490(``DEPENDS += "B"``), package B will neither appear in the generated license
491manifest nor in the generated source tarballs. This occurs as the
492:ref:`license <ref-classes-license>` and :ref:`archiver <ref-classes-archiver>`
493classes assume that only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS`
494end up in the image.
495
496As a result, potential obligations regarding license compliance for package B
497may not be met.
498
499The Yocto Project doesn't enable static libraries by default, in part because
500of this issue. Before a solution to this limitation is found, you need to
501keep in mind that if your root filesystem is built from static libraries,
502you will need to manually ensure that your deliveries are compliant
503with the licenses of these libraries.
504
505Copying Non Standard Licenses
506=============================
507
508Some packages, such as the linux-firmware package, have many licenses
509that are not in any way common. You can avoid adding a lot of these
510types of common license files, which are only applicable to a specific
511package, by using the
512:term:`NO_GENERIC_LICENSE`
513variable. Using this variable also avoids QA errors when you use a
514non-common, non-CLOSED license in a recipe.
515
516Here is an example that uses the ``LICENSE.Abilis.txt`` file as
517the license from the fetched source::
518
519 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
520
diff --git a/documentation/dev-manual/new-machine.rst b/documentation/dev-manual/new-machine.rst
new file mode 100644
index 0000000000..930dd7eac7
--- /dev/null
+++ b/documentation/dev-manual/new-machine.rst
@@ -0,0 +1,118 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Adding a New Machine
4********************
5
6Adding a new machine to the Yocto Project is a straightforward process.
7This section describes how to add machines that are similar to those
8that the Yocto Project already supports.
9
10.. note::
11
12 Although well within the capabilities of the Yocto Project, adding a
13 totally new architecture might require changes to ``gcc``/``glibc``
14 and to the site information, which is beyond the scope of this
15 manual.
16
17For a complete example that shows how to add a new machine, see the
18":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
19section in the Yocto Project Board Support Package (BSP) Developer's
20Guide.
21
22Adding the Machine Configuration File
23=====================================
24
25To add a new machine, you need to add a new machine configuration file
26to the layer's ``conf/machine`` directory. This configuration file
27provides details about the device you are adding.
28
29The OpenEmbedded build system uses the root name of the machine
30configuration file to reference the new machine. For example, given a
31machine configuration file named ``crownbay.conf``, the build system
32recognizes the machine as "crownbay".
33
34The most important variables you must set in your machine configuration
35file or include from a lower-level configuration file are as follows:
36
37- :term:`TARGET_ARCH` (e.g. "arm")
38
39- ``PREFERRED_PROVIDER_virtual/kernel``
40
41- :term:`MACHINE_FEATURES` (e.g. "apm screen wifi")
42
43You might also need these variables:
44
45- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
46
47- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
48
49- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
50
51You can find full details on these variables in the reference section.
52You can leverage existing machine ``.conf`` files from
53``meta-yocto-bsp/conf/machine/``.
54
55Adding a Kernel for the Machine
56===============================
57
58The OpenEmbedded build system needs to be able to build a kernel for the
59machine. You need to either create a new kernel recipe for this machine,
60or extend an existing kernel recipe. You can find several kernel recipe
61examples in the Source Directory at ``meta/recipes-kernel/linux`` that
62you can use as references.
63
64If you are creating a new kernel recipe, normal recipe-writing rules
65apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
66necessary patches and set :term:`S` to point at the source code. You need to
67create a :ref:`ref-tasks-configure` task that configures the unpacked kernel with
68a ``defconfig`` file. You can do this by using a ``make defconfig``
69command or, more commonly, by copying in a suitable ``defconfig`` file
70and then running ``make oldconfig``. By making use of ``inherit kernel``
71and potentially some of the ``linux-*.inc`` files, most other
72functionality is centralized and the defaults of the class normally work
73well.
74
75If you are extending an existing kernel recipe, it is usually a matter
76of adding a suitable ``defconfig`` file. The file needs to be added into
77a location similar to ``defconfig`` files used for other machines in a
78given kernel recipe. A possible way to do this is by listing the file in
79the :term:`SRC_URI` and adding the machine to the expression in
80:term:`COMPATIBLE_MACHINE`::
81
82 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
83
84For more information on ``defconfig`` files, see the
85":ref:`kernel-dev/common:changing the configuration`"
86section in the Yocto Project Linux Kernel Development Manual.
87
88Adding a Formfactor Configuration File
89======================================
90
91A formfactor configuration file provides information about the target
92hardware for which the image is being built and information that the
93build system cannot obtain from other sources such as the kernel. Some
94examples of information contained in a formfactor configuration file
95include framebuffer orientation, whether or not the system has a
96keyboard, the positioning of the keyboard in relation to the screen, and
97the screen resolution.
98
99The build system uses reasonable defaults in most cases. However, if
100customization is necessary, you need to create a ``machconfig`` file in
101the ``meta/recipes-bsp/formfactor/files`` directory. This directory
102contains directories for specific machines such as ``qemuarm`` and
103``qemux86``. For information about the settings available and the
104defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
105found in the same area.
106
107Following is an example for "qemuarm" machine::
108
109 HAVE_TOUCHSCREEN=1
110 HAVE_KEYBOARD=1
111 DISPLAY_CAN_ROTATE=0
112 DISPLAY_ORIENTATION=0
113 #DISPLAY_WIDTH_PIXELS=640
114 #DISPLAY_HEIGHT_PIXELS=480
115 #DISPLAY_BPP=16
116 DISPLAY_DPI=150
117 DISPLAY_SUBPIXEL_ORDER=vrgb
118
diff --git a/documentation/dev-manual/new-recipe.rst b/documentation/dev-manual/new-recipe.rst
new file mode 100644
index 0000000000..2e9c2089f7
--- /dev/null
+++ b/documentation/dev-manual/new-recipe.rst
@@ -0,0 +1,1674 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Writing a New Recipe
4********************
5
6Recipes (``.bb`` files) are fundamental components in the Yocto Project
7environment. Each software component built by the OpenEmbedded build
8system requires a recipe to define the component. This section describes
9how to create, write, and test a new recipe.
10
11.. note::
12
13 For information on variables that are useful for recipes and for
14 information about recipe naming issues, see the
15 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
16 Reference Manual.
17
18Overview
19========
20
21The following figure shows the basic process for creating a new recipe.
22The remainder of the section provides details for the steps.
23
24.. image:: figures/recipe-workflow.png
25 :align: center
26 :width: 50%
27
28Locate or Automatically Create a Base Recipe
29============================================
30
31You can always write a recipe from scratch. However, there are three choices
32that can help you quickly get started with a new recipe:
33
34- ``devtool add``: A command that assists in creating a recipe and an
35 environment conducive to development.
36
37- ``recipetool create``: A command provided by the Yocto Project that
38 automates creation of a base recipe based on the source files.
39
40- *Existing Recipes:* Location and modification of an existing recipe
41 that is similar in function to the recipe you need.
42
43.. note::
44
45 For information on recipe syntax, see the
46 ":ref:`dev-manual/new-recipe:recipe syntax`" section.
47
48Creating the Base Recipe Using ``devtool add``
49----------------------------------------------
50
51The ``devtool add`` command uses the same logic for auto-creating the
52recipe as ``recipetool create``, which is listed below. Additionally,
53however, ``devtool add`` sets up an environment that makes it easy for
54you to patch the source and to make changes to the recipe as is often
55necessary when adding a recipe to build a new piece of software to be
56included in a build.
57
58You can find a complete description of the ``devtool add`` command in
59the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
60in the Yocto Project Application Development and the Extensible Software
61Development Kit (eSDK) manual.
62
63Creating the Base Recipe Using ``recipetool create``
64----------------------------------------------------
65
66``recipetool create`` automates creation of a base recipe given a set of
67source code files. As long as you can extract or point to the source
68files, the tool will construct a recipe and automatically configure all
69pre-build information into the recipe. For example, suppose you have an
70application that builds using Autotools. Creating the base recipe using
71``recipetool`` results in a recipe that has the pre-build dependencies,
72license requirements, and checksums configured.
73
74To run the tool, you just need to be in your :term:`Build Directory` and
75have sourced the build environment setup script (i.e.
76:ref:`structure-core-script`). To get help on the tool, use the following
77command::
78
79 $ recipetool -h
80 NOTE: Starting bitbake server...
81 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
82
83 OpenEmbedded recipe tool
84
85 options:
86 -d, --debug Enable debug output
87 -q, --quiet Print only errors
88 --color COLOR Colorize output (where COLOR is auto, always, never)
89 -h, --help show this help message and exit
90
91 subcommands:
92 create Create a new recipe
93 newappend Create a bbappend for the specified target in the specified
94 layer
95 setvar Set a variable within a recipe
96 appendfile Create/update a bbappend to replace a target file
97 appendsrcfiles Create/update a bbappend to add or replace source files
98 appendsrcfile Create/update a bbappend to add or replace a source file
99 Use recipetool <subcommand> --help to get help on a specific command
100
101Running ``recipetool create -o OUTFILE`` creates the base recipe and
102locates it properly in the layer that contains your source files.
103Following are some syntax examples:
104
105 - Use this syntax to generate a recipe based on source. Once generated,
106 the recipe resides in the existing source code layer::
107
108 recipetool create -o OUTFILE source
109
110 - Use this syntax to generate a recipe using code that
111 you extract from source. The extracted code is placed in its own layer
112 defined by :term:`EXTERNALSRC`.
113 ::
114
115 recipetool create -o OUTFILE -x EXTERNALSRC source
116
117 - Use this syntax to generate a recipe based on source. The options
118 direct ``recipetool`` to generate debugging information. Once generated,
119 the recipe resides in the existing source code layer::
120
121 recipetool create -d -o OUTFILE source
122
123Locating and Using a Similar Recipe
124-----------------------------------
125
126Before writing a recipe from scratch, it is often useful to discover
127whether someone else has already written one that meets (or comes close
128to meeting) your needs. The Yocto Project and OpenEmbedded communities
129maintain many recipes that might be candidates for what you are doing.
130You can find a good central index of these recipes in the
131:oe_layerindex:`OpenEmbedded Layer Index <>`.
132
133Working from an existing recipe or a skeleton recipe is the best way to
134get started. Here are some points on both methods:
135
136- *Locate and modify a recipe that is close to what you want to do:*
137 This method works when you are familiar with the current recipe
138 space. The method does not work so well for those new to the Yocto
139 Project or writing recipes.
140
141 Some risks associated with this method are using a recipe that has
142 areas totally unrelated to what you are trying to accomplish with
143 your recipe, not recognizing areas of the recipe that you might have
144 to add from scratch, and so forth. All these risks stem from
145 unfamiliarity with the existing recipe space.
146
147- *Use and modify the following skeleton recipe:* If for some reason
148 you do not want to use ``recipetool`` and you cannot find an existing
149 recipe that is close to meeting your needs, you can use the following
150 structure to provide the fundamental areas of a new recipe.
151 ::
152
153 DESCRIPTION = ""
154 HOMEPAGE = ""
155 LICENSE = ""
156 SECTION = ""
157 DEPENDS = ""
158 LIC_FILES_CHKSUM = ""
159
160 SRC_URI = ""
161
162Storing and Naming the Recipe
163=============================
164
165Once you have your base recipe, you should put it in your own layer and
166name it appropriately. Locating it correctly ensures that the
167OpenEmbedded build system can find it when you use BitBake to process
168the recipe.
169
170- *Storing Your Recipe:* The OpenEmbedded build system locates your
171 recipe through the layer's ``conf/layer.conf`` file and the
172 :term:`BBFILES` variable. This
173 variable sets up a path from which the build system can locate
174 recipes. Here is the typical use::
175
176 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
177 ${LAYERDIR}/recipes-*/*/*.bbappend"
178
179 Consequently, you need to be sure you locate your new recipe inside
180 your layer such that it can be found.
181
182 You can find more information on how layers are structured in the
183 ":ref:`dev-manual/layers:understanding and creating layers`" section.
184
185- *Naming Your Recipe:* When you name your recipe, you need to follow
186 this naming convention::
187
188 basename_version.bb
189
190 Use lower-cased characters and do not include the reserved suffixes
191 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
192 them as part of your recipe name unless the string applies). Here are some
193 examples:
194
195 .. code-block:: none
196
197 cups_1.7.0.bb
198 gawk_4.0.2.bb
199 irssi_0.8.16-rc1.bb
200
201Running a Build on the Recipe
202=============================
203
204Creating a new recipe is usually an iterative process that requires
205using BitBake to process the recipe multiple times in order to
206progressively discover and add information to the recipe file.
207
208Assuming you have sourced the build environment setup script (i.e.
209:ref:`structure-core-script`) and you are in the :term:`Build Directory`, use
210BitBake to process your recipe. All you need to provide is the
211``basename`` of the recipe as described in the previous section::
212
213 $ bitbake basename
214
215During the build, the OpenEmbedded build system creates a temporary work
216directory for each recipe
217(``${``\ :term:`WORKDIR`\ ``}``)
218where it keeps extracted source files, log files, intermediate
219compilation and packaging files, and so forth.
220
221The path to the per-recipe temporary work directory depends on the
222context in which it is being built. The quickest way to find this path
223is to have BitBake return it by running the following::
224
225 $ bitbake -e basename | grep ^WORKDIR=
226
227As an example, assume a Source Directory
228top-level folder named ``poky``, a default :term:`Build Directory` at
229``poky/build``, and a ``qemux86-poky-linux`` machine target system.
230Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
231case, the work directory the build system uses to build the package
232would be as follows::
233
234 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
235
236Inside this directory you can find sub-directories such as ``image``,
237``packages-split``, and ``temp``. After the build, you can examine these
238to determine how well the build went.
239
240.. note::
241
242 You can find log files for each task in the recipe's ``temp``
243 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
244 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
245 ``log.do_fetch``, and ``log.do_compile``).
246
247You can find more information about the build process in
248":doc:`/overview-manual/development-environment`"
249chapter of the Yocto Project Overview and Concepts Manual.
250
251Fetching Code
252=============
253
254The first thing your recipe must do is specify how to fetch the source
255files. Fetching is controlled mainly through the
256:term:`SRC_URI` variable. Your recipe
257must have a :term:`SRC_URI` variable that points to where the source is
258located. For a graphical representation of source locations, see the
259":ref:`overview-manual/concepts:sources`" section in
260the Yocto Project Overview and Concepts Manual.
261
262The :ref:`ref-tasks-fetch` task uses
263the prefix of each entry in the :term:`SRC_URI` variable value to determine
264which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
265source files. It is the :term:`SRC_URI` variable that triggers the fetcher.
266The :ref:`ref-tasks-patch` task uses
267the variable after source is fetched to apply patches. The OpenEmbedded
268build system uses
269:term:`FILESOVERRIDES` for
270scanning directory locations for local files in :term:`SRC_URI`.
271
272The :term:`SRC_URI` variable in your recipe must define each unique location
273for your source files. It is good practice to not hard-code version
274numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
275values, use ``${``\ :term:`PV`\ ``}``,
276which causes the fetch process to use the version specified in the
277recipe filename. Specifying the version in this manner means that
278upgrading the recipe to a future version is as simple as renaming the
279recipe to match the new version.
280
281Here is a simple example from the
282``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
283comes from a single tarball. Notice the use of the
284:term:`PV` variable::
285
286 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
287
288Files mentioned in :term:`SRC_URI` whose names end in a typical archive
289extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
290forth), are automatically extracted during the
291:ref:`ref-tasks-unpack` task. For
292another example that specifies these types of files, see the
293":ref:`dev-manual/new-recipe:autotooled package`" section.
294
295Another way of specifying source is from an SCM. For Git repositories,
296you must specify :term:`SRCREV` and you should specify :term:`PV` to include
297the revision with :term:`SRCPV`. Here is an example from the recipe
298``meta/recipes-core/musl/gcompat_git.bb``::
299
300 SRC_URI = "git://git.adelielinux.org/adelie/gcompat.git;protocol=https;branch=current"
301
302 PV = "1.0.0+1.1+git${SRCPV}"
303 SRCREV = "af5a49e489fdc04b9cf02547650d7aeaccd43793"
304
305If your :term:`SRC_URI` statement includes URLs pointing to individual files
306fetched from a remote server other than a version control system,
307BitBake attempts to verify the files against checksums defined in your
308recipe to ensure they have not been tampered with or otherwise modified
309since the recipe was written. Two checksums are used:
310``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
311
312If your :term:`SRC_URI` variable points to more than a single URL (excluding
313SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
314each URL. For these cases, you provide a name for each URL as part of
315the :term:`SRC_URI` and then reference that name in the subsequent checksum
316statements. Here is an example combining lines from the files
317``git.inc`` and ``git_2.24.1.bb``::
318
319 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
320 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
321
322 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
323 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
324 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
325 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
326
327Proper values for ``md5`` and ``sha256`` checksums might be available
328with other signatures on the download page for the upstream source (e.g.
329``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
330OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
331you should verify all the signatures you find by hand.
332
333If no :term:`SRC_URI` checksums are specified when you attempt to build the
334recipe, or you provide an incorrect checksum, the build will produce an
335error for each missing or incorrect checksum. As part of the error
336message, the build system provides the checksum string corresponding to
337the fetched file. Once you have the correct checksums, you can copy and
338paste them into your recipe and then run the build again to continue.
339
340.. note::
341
342 As mentioned, if the upstream source provides signatures for
343 verifying the downloaded source code, you should verify those
344 manually before setting the checksum values in the recipe and
345 continuing with the build.
346
347This final example is a bit more complicated and is from the
348``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
349example's :term:`SRC_URI` statement identifies multiple files as the source
350files for the recipe: a tarball, a patch file, a desktop file, and an
351icon.
352::
353
354 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
355 file://xwc.patch \
356 file://rxvt.desktop \
357 file://rxvt.png"
358
359When you specify local files using the ``file://`` URI protocol, the
360build system fetches files from the local machine. The path is relative
361to the :term:`FILESPATH` variable
362and searches specific directories in a certain order:
363``${``\ :term:`BP`\ ``}``,
364``${``\ :term:`BPN`\ ``}``, and
365``files``. The directories are assumed to be subdirectories of the
366directory in which the recipe or append file resides. For another
367example that specifies these types of files, see the
368":ref:`dev-manual/new-recipe:single .c file package (hello world!)`" section.
369
370The previous example also specifies a patch file. Patch files are files
371whose names usually end in ``.patch`` or ``.diff`` but can end with
372compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
373The build system automatically applies patches as described in the
374":ref:`dev-manual/new-recipe:patching code`" section.
375
376Fetching Code Through Firewalls
377-------------------------------
378
379Some users are behind firewalls and need to fetch code through a proxy.
380See the ":doc:`/ref-manual/faq`" chapter for advice.
381
382Limiting the Number of Parallel Connections
383-------------------------------------------
384
385Some users are behind firewalls or use servers where the number of parallel
386connections is limited. In such cases, you can limit the number of fetch
387tasks being run in parallel by adding the following to your ``local.conf``
388file::
389
390 do_fetch[number_threads] = "4"
391
392Unpacking Code
393==============
394
395During the build, the
396:ref:`ref-tasks-unpack` task unpacks
397the source with ``${``\ :term:`S`\ ``}``
398pointing to where it is unpacked.
399
400If you are fetching your source files from an upstream source archived
401tarball and the tarball's internal structure matches the common
402convention of a top-level subdirectory named
403``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
404then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
405fetch source from an archive that does not use this convention, or from
406an SCM like Git or Subversion, your recipe needs to define :term:`S`.
407
408If processing your recipe using BitBake successfully unpacks the source
409files, you need to be sure that the directory pointed to by ``${S}``
410matches the structure of the source.
411
412Patching Code
413=============
414
415Sometimes it is necessary to patch code after it has been fetched. Any
416files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
417``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
418treated as patches. The
419:ref:`ref-tasks-patch` task
420automatically applies these patches.
421
422The build system should be able to apply patches with the "-p1" option
423(i.e. one directory level in the path will be stripped off). If your
424patch needs to have more directory levels stripped off, specify the
425number of levels using the "striplevel" option in the :term:`SRC_URI` entry
426for the patch. Alternatively, if your patch needs to be applied in a
427specific subdirectory that is not specified in the patch file, use the
428"patchdir" option in the entry.
429
430As with all local files referenced in
431:term:`SRC_URI` using ``file://``,
432you should place patch files in a directory next to the recipe either
433named the same as the base name of the recipe
434(:term:`BP` and
435:term:`BPN`) or "files".
436
437Licensing
438=========
439
440Your recipe needs to have both the
441:term:`LICENSE` and
442:term:`LIC_FILES_CHKSUM`
443variables:
444
445- :term:`LICENSE`: This variable specifies the license for the software.
446 If you do not know the license under which the software you are
447 building is distributed, you should go to the source code and look
448 for that information. Typical files containing this information
449 include ``COPYING``, :term:`LICENSE`, and ``README`` files. You could
450 also find the information near the top of a source file. For example,
451 given a piece of software licensed under the GNU General Public
452 License version 2, you would set :term:`LICENSE` as follows::
453
454 LICENSE = "GPL-2.0-only"
455
456 The licenses you specify within :term:`LICENSE` can have any name as long
457 as you do not use spaces, since spaces are used as separators between
458 license names. For standard licenses, use the names of the files in
459 ``meta/files/common-licenses/`` or the :term:`SPDXLICENSEMAP` flag names
460 defined in ``meta/conf/licenses.conf``.
461
462- :term:`LIC_FILES_CHKSUM`: The OpenEmbedded build system uses this
463 variable to make sure the license text has not changed. If it has,
464 the build produces an error and it affords you the chance to figure
465 it out and correct the problem.
466
467 You need to specify all applicable licensing files for the software.
468 At the end of the configuration step, the build process will compare
469 the checksums of the files to be sure the text has not changed. Any
470 differences result in an error with the message containing the
471 current checksum. For more explanation and examples of how to set the
472 :term:`LIC_FILES_CHKSUM` variable, see the
473 ":ref:`dev-manual/licenses:tracking license changes`" section.
474
475 To determine the correct checksum string, you can list the
476 appropriate files in the :term:`LIC_FILES_CHKSUM` variable with incorrect
477 md5 strings, attempt to build the software, and then note the
478 resulting error messages that will report the correct md5 strings.
479 See the ":ref:`dev-manual/new-recipe:fetching code`" section for
480 additional information.
481
482 Here is an example that assumes the software has a ``COPYING`` file::
483
484 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
485
486 When you try to build the
487 software, the build system will produce an error and give you the
488 correct string that you can substitute into the recipe file for a
489 subsequent build.
490
491Dependencies
492============
493
494Most software packages have a short list of other packages that they
495require, which are called dependencies. These dependencies fall into two
496main categories: build-time dependencies, which are required when the
497software is built; and runtime dependencies, which are required to be
498installed on the target in order for the software to run.
499
500Within a recipe, you specify build-time dependencies using the
501:term:`DEPENDS` variable. Although there are nuances,
502items specified in :term:`DEPENDS` should be names of other
503recipes. It is important that you specify all build-time dependencies
504explicitly.
505
506Another consideration is that configure scripts might automatically
507check for optional dependencies and enable corresponding functionality
508if those dependencies are found. If you wish to make a recipe that is
509more generally useful (e.g. publish the recipe in a layer for others to
510use), instead of hard-disabling the functionality, you can use the
511:term:`PACKAGECONFIG` variable to allow functionality and the
512corresponding dependencies to be enabled and disabled easily by other
513users of the recipe.
514
515Similar to build-time dependencies, you specify runtime dependencies
516through a variable -
517:term:`RDEPENDS`, which is
518package-specific. All variables that are package-specific need to have
519the name of the package added to the end as an override. Since the main
520package for a recipe has the same name as the recipe, and the recipe's
521name can be found through the
522``${``\ :term:`PN`\ ``}`` variable, then
523you specify the dependencies for the main package by setting
524``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you
525would set ``RDEPENDS:${PN}-tools``, and so forth.
526
527Some runtime dependencies will be set automatically at packaging time.
528These dependencies include any shared library dependencies (i.e. if a
529package "example" contains "libexample" and another package "mypackage"
530contains a binary that links to "libexample" then the OpenEmbedded build
531system will automatically add a runtime dependency to "mypackage" on
532"example"). See the
533":ref:`overview-manual/concepts:automatically added runtime dependencies`"
534section in the Yocto Project Overview and Concepts Manual for further
535details.
536
537Configuring the Recipe
538======================
539
540Most software provides some means of setting build-time configuration
541options before compilation. Typically, setting these options is
542accomplished by running a configure script with options, or by modifying
543a build configuration file.
544
545.. note::
546
547 As of Yocto Project Release 1.7, some of the core recipes that
548 package binary configuration scripts now disable the scripts due to
549 the scripts previously requiring error-prone path substitution. The
550 OpenEmbedded build system uses ``pkg-config`` now, which is much more
551 robust. You can find a list of the ``*-config`` scripts that are disabled
552 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
553 in the Yocto Project Reference Manual.
554
555A major part of build-time configuration is about checking for
556build-time dependencies and possibly enabling optional functionality as
557a result. You need to specify any build-time dependencies for the
558software you are building in your recipe's
559:term:`DEPENDS` value, in terms of
560other recipes that satisfy those dependencies. You can often find
561build-time or runtime dependencies described in the software's
562documentation.
563
564The following list provides configuration items of note based on how
565your software is built:
566
567- *Autotools:* If your source files have a ``configure.ac`` file, then
568 your software is built using Autotools. If this is the case, you just
569 need to modify the configuration.
570
571 When using Autotools, your recipe needs to inherit the
572 :ref:`autotools <ref-classes-autotools>` class and it does not have to
573 contain a :ref:`ref-tasks-configure` task. However, you might still want to
574 make some adjustments. For example, you can set :term:`EXTRA_OECONF` or
575 :term:`PACKAGECONFIG_CONFARGS` to pass any needed configure options that
576 are specific to the recipe.
577
578- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
579 your software is built using CMake. If this is the case, you just
580 need to modify the configuration.
581
582 When you use CMake, your recipe needs to inherit the
583 :ref:`cmake <ref-classes-cmake>` class and it does not have to contain a
584 :ref:`ref-tasks-configure` task. You can make some adjustments by setting
585 :term:`EXTRA_OECMAKE` to pass any needed configure options that are
586 specific to the recipe.
587
588 .. note::
589
590 If you need to install one or more custom CMake toolchain files
591 that are supplied by the application you are building, install the
592 files to ``${D}${datadir}/cmake/Modules`` during :ref:`ref-tasks-install`.
593
594- *Other:* If your source files do not have a ``configure.ac`` or
595 ``CMakeLists.txt`` file, then your software is built using some
596 method other than Autotools or CMake. If this is the case, you
597 normally need to provide a
598 :ref:`ref-tasks-configure` task
599 in your recipe unless, of course, there is nothing to configure.
600
601 Even if your software is not being built by Autotools or CMake, you
602 still might not need to deal with any configuration issues. You need
603 to determine if configuration is even a required step. You might need
604 to modify a Makefile or some configuration file used for the build to
605 specify necessary build options. Or, perhaps you might need to run a
606 provided, custom configure script with the appropriate options.
607
608 For the case involving a custom configure script, you would run
609 ``./configure --help`` and look for the options you need to set.
610
611Once configuration succeeds, it is always good practice to look at the
612``log.do_configure`` file to ensure that the appropriate options have
613been enabled and no additional build-time dependencies need to be added
614to :term:`DEPENDS`. For example, if the configure script reports that it
615found something not mentioned in :term:`DEPENDS`, or that it did not find
616something that it needed for some desired optional functionality, then
617you would need to add those to :term:`DEPENDS`. Looking at the log might
618also reveal items being checked for, enabled, or both that you do not
619want, or items not being found that are in :term:`DEPENDS`, in which case
620you would need to look at passing extra options to the configure script
621as needed. For reference information on configure options specific to
622the software you are building, you can consult the output of the
623``./configure --help`` command within ``${S}`` or consult the software's
624upstream documentation.
625
626Using Headers to Interface with Devices
627=======================================
628
629If your recipe builds an application that needs to communicate with some
630device or needs an API into a custom kernel, you will need to provide
631appropriate header files. Under no circumstances should you ever modify
632the existing
633``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
634These headers are used to build ``libc`` and must not be compromised
635with custom or machine-specific header information. If you customize
636``libc`` through modified headers all other applications that use
637``libc`` thus become affected.
638
639.. note::
640
641 Never copy and customize the ``libc`` header file (i.e.
642 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
643
644The correct way to interface to a device or custom kernel is to use a
645separate package that provides the additional headers for the driver or
646other unique interfaces. When doing so, your application also becomes
647responsible for creating a dependency on that specific provider.
648
649Consider the following:
650
651- Never modify ``linux-libc-headers.inc``. Consider that file to be
652 part of the ``libc`` system, and not something you use to access the
653 kernel directly. You should access ``libc`` through specific ``libc``
654 calls.
655
656- Applications that must talk directly to devices should either provide
657 necessary headers themselves, or establish a dependency on a special
658 headers package that is specific to that driver.
659
660For example, suppose you want to modify an existing header that adds I/O
661control or network support. If the modifications are used by a small
662number programs, providing a unique version of a header is easy and has
663little impact. When doing so, bear in mind the guidelines in the
664previous list.
665
666.. note::
667
668 If for some reason your changes need to modify the behavior of the ``libc``,
669 and subsequently all other applications on the system, use a ``.bbappend``
670 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
671 make the changes machine specific.
672
673Consider a case where your kernel is older and you need an older
674``libc`` ABI. The headers installed by your recipe should still be a
675standard mainline kernel, not your own custom one.
676
677When you use custom kernel headers you need to get them from
678:term:`STAGING_KERNEL_DIR`,
679which is the directory with kernel headers that are required to build
680out-of-tree modules. Your recipe will also need the following::
681
682 do_configure[depends] += "virtual/kernel:do_shared_workdir"
683
684Compilation
685===========
686
687During a build, the :ref:`ref-tasks-compile` task happens after source is fetched,
688unpacked, and configured. If the recipe passes through :ref:`ref-tasks-compile`
689successfully, nothing needs to be done.
690
691However, if the compile step fails, you need to diagnose the failure.
692Here are some common issues that cause failures.
693
694.. note::
695
696 For cases where improper paths are detected for configuration files
697 or for when libraries/headers cannot be found, be sure you are using
698 the more robust ``pkg-config``. See the note in section
699 ":ref:`dev-manual/new-recipe:Configuring the Recipe`" for additional information.
700
701- *Parallel build failures:* These failures manifest themselves as
702 intermittent errors, or errors reporting that a file or directory
703 that should be created by some other part of the build process could
704 not be found. This type of failure can occur even if, upon
705 inspection, the file or directory does exist after the build has
706 failed, because that part of the build process happened in the wrong
707 order.
708
709 To fix the problem, you need to either satisfy the missing dependency
710 in the Makefile or whatever script produced the Makefile, or (as a
711 workaround) set :term:`PARALLEL_MAKE` to an empty string::
712
713 PARALLEL_MAKE = ""
714
715 For information on parallel Makefile issues, see the
716 ":ref:`dev-manual/debugging:debugging parallel make races`" section.
717
718- *Improper host path usage:* This failure applies to recipes building
719 for the target or ":ref:`nativesdk <ref-classes-nativesdk>`" only. The
720 failure occurs when the compilation process uses improper headers,
721 libraries, or other files from the host system when cross-compiling for
722 the target.
723
724 To fix the problem, examine the ``log.do_compile`` file to identify
725 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
726 so forth) and then either add configure options, apply a patch, or do
727 both.
728
729- *Failure to find required libraries/headers:* If a build-time
730 dependency is missing because it has not been declared in
731 :term:`DEPENDS`, or because the
732 dependency exists but the path used by the build process to find the
733 file is incorrect and the configure step did not detect it, the
734 compilation process could fail. For either of these failures, the
735 compilation process notes that files could not be found. In these
736 cases, you need to go back and add additional options to the
737 configure script as well as possibly add additional build-time
738 dependencies to :term:`DEPENDS`.
739
740 Occasionally, it is necessary to apply a patch to the source to
741 ensure the correct paths are used. If you need to specify paths to
742 find files staged into the sysroot from other recipes, use the
743 variables that the OpenEmbedded build system provides (e.g.
744 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
745 forth).
746
747Installing
748==========
749
750During :ref:`ref-tasks-install`, the task copies the built files along with their
751hierarchy to locations that would mirror their locations on the target
752device. The installation process copies files from the
753``${``\ :term:`S`\ ``}``,
754``${``\ :term:`B`\ ``}``, and
755``${``\ :term:`WORKDIR`\ ``}``
756directories to the ``${``\ :term:`D`\ ``}``
757directory to create the structure as it should appear on the target
758system.
759
760How your software is built affects what you must do to be sure your
761software is installed correctly. The following list describes what you
762must do for installation depending on the type of build system used by
763the software being built:
764
765- *Autotools and CMake:* If the software your recipe is building uses
766 Autotools or CMake, the OpenEmbedded build system understands how to
767 install the software. Consequently, you do not have to have a
768 :ref:`ref-tasks-install` task as part of your recipe. You just need to make
769 sure the install portion of the build completes with no issues.
770 However, if you wish to install additional files not already being
771 installed by ``make install``, you should do this using a
772 ``do_install:append`` function using the install command as described
773 in the "Manual" bulleted item later in this list.
774
775- *Other (using* ``make install``\ *)*: You need to define a :ref:`ref-tasks-install`
776 function in your recipe. The function should call
777 ``oe_runmake install`` and will likely need to pass in the
778 destination directory as well. How you pass that path is dependent on
779 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
780 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
781
782 For an example recipe using ``make install``, see the
783 ":ref:`dev-manual/new-recipe:makefile-based package`" section.
784
785- *Manual:* You need to define a :ref:`ref-tasks-install` function in your
786 recipe. The function must first use ``install -d`` to create the
787 directories under
788 ``${``\ :term:`D`\ ``}``. Once the
789 directories exist, your function can use ``install`` to manually
790 install the built software into the directories.
791
792 You can find more information on ``install`` at
793 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
794
795For the scenarios that do not use Autotools or CMake, you need to track
796the installation and diagnose and fix any issues until everything
797installs correctly. You need to look in the default location of
798``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
799installed correctly.
800
801.. note::
802
803 - During the installation process, you might need to modify some of
804 the installed files to suit the target layout. For example, you
805 might need to replace hard-coded paths in an initscript with
806 values of variables provided by the build system, such as
807 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
808 modifications during :ref:`ref-tasks-install`, be sure to modify the
809 destination file after copying rather than before copying.
810 Modifying after copying ensures that the build system can
811 re-execute :ref:`ref-tasks-install` if needed.
812
813 - ``oe_runmake install``, which can be run directly or can be run
814 indirectly by the
815 :ref:`autotools <ref-classes-autotools>` and
816 :ref:`cmake <ref-classes-cmake>` classes,
817 runs ``make install`` in parallel. Sometimes, a Makefile can have
818 missing dependencies between targets that can result in race
819 conditions. If you experience intermittent failures during
820 :ref:`ref-tasks-install`, you might be able to work around them by disabling
821 parallel Makefile installs by adding the following to the recipe::
822
823 PARALLEL_MAKEINST = ""
824
825 See :term:`PARALLEL_MAKEINST` for additional information.
826
827 - If you need to install one or more custom CMake toolchain files
828 that are supplied by the application you are building, install the
829 files to ``${D}${datadir}/cmake/Modules`` during
830 :ref:`ref-tasks-install`.
831
832Enabling System Services
833========================
834
835If you want to install a service, which is a process that usually starts
836on boot and runs in the background, then you must include some
837additional definitions in your recipe.
838
839If you are adding services and the service initialization script or the
840service file itself is not installed, you must provide for that
841installation in your recipe using a ``do_install:append`` function. If
842your recipe already has a :ref:`ref-tasks-install` function, update the function
843near its end rather than adding an additional ``do_install:append``
844function.
845
846When you create the installation for your services, you need to
847accomplish what is normally done by ``make install``. In other words,
848make sure your installation arranges the output similar to how it is
849arranged on the target system.
850
851The OpenEmbedded build system provides support for starting services two
852different ways:
853
854- *SysVinit:* SysVinit is a system and service manager that manages the
855 init system used to control the very basic functions of your system.
856 The init program is the first program started by the Linux kernel
857 when the system boots. Init then controls the startup, running and
858 shutdown of all other programs.
859
860 To enable a service using SysVinit, your recipe needs to inherit the
861 :ref:`update-rc.d <ref-classes-update-rc.d>` class. The class helps
862 facilitate safely installing the package on the target.
863
864 You will need to set the
865 :term:`INITSCRIPT_PACKAGES`,
866 :term:`INITSCRIPT_NAME`,
867 and
868 :term:`INITSCRIPT_PARAMS`
869 variables within your recipe.
870
871- *systemd:* System Management Daemon (systemd) was designed to replace
872 SysVinit and to provide enhanced management of services. For more
873 information on systemd, see the systemd homepage at
874 https://freedesktop.org/wiki/Software/systemd/.
875
876 To enable a service using systemd, your recipe needs to inherit the
877 :ref:`systemd <ref-classes-systemd>` class. See the ``systemd.bbclass`` file
878 located in your :term:`Source Directory` section for more information.
879
880Packaging
881=========
882
883Successful packaging is a combination of automated processes performed
884by the OpenEmbedded build system and some specific steps you need to
885take. The following list describes the process:
886
887- *Splitting Files*: The :ref:`ref-tasks-package` task splits the files produced
888 by the recipe into logical components. Even software that produces a
889 single binary might still have debug symbols, documentation, and
890 other logical components that should be split out. The :ref:`ref-tasks-package`
891 task ensures that files are split up and packaged correctly.
892
893- *Running QA Checks*: The
894 :ref:`insane <ref-classes-insane>` class adds a
895 step to the package generation process so that output quality
896 assurance checks are generated by the OpenEmbedded build system. This
897 step performs a range of checks to be sure the build's output is free
898 of common problems that show up during runtime. For information on
899 these checks, see the
900 :ref:`insane <ref-classes-insane>` class and
901 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
902 chapter in the Yocto Project Reference Manual.
903
904- *Hand-Checking Your Packages*: After you build your software, you
905 need to be sure your packages are correct. Examine the
906 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
907 directory and make sure files are where you expect them to be. If you
908 discover problems, you can set
909 :term:`PACKAGES`,
910 :term:`FILES`,
911 ``do_install(:append)``, and so forth as needed.
912
913- *Splitting an Application into Multiple Packages*: If you need to
914 split an application into several packages, see the
915 ":ref:`dev-manual/new-recipe:splitting an application into multiple packages`"
916 section for an example.
917
918- *Installing a Post-Installation Script*: For an example showing how
919 to install a post-installation script, see the
920 ":ref:`dev-manual/new-recipe:post-installation scripts`" section.
921
922- *Marking Package Architecture*: Depending on what your recipe is
923 building and how it is configured, it might be important to mark the
924 packages produced as being specific to a particular machine, or to
925 mark them as not being specific to a particular machine or
926 architecture at all.
927
928 By default, packages apply to any machine with the same architecture
929 as the target machine. When a recipe produces packages that are
930 machine-specific (e.g. the
931 :term:`MACHINE` value is passed
932 into the configure script or a patch is applied only for a particular
933 machine), you should mark them as such by adding the following to the
934 recipe::
935
936 PACKAGE_ARCH = "${MACHINE_ARCH}"
937
938 On the other hand, if the recipe produces packages that do not
939 contain anything specific to the target machine or architecture at
940 all (e.g. recipes that simply package script files or configuration
941 files), you should use the
942 :ref:`allarch <ref-classes-allarch>` class to
943 do this for you by adding this to your recipe::
944
945 inherit allarch
946
947 Ensuring that the package architecture is correct is not critical
948 while you are doing the first few builds of your recipe. However, it
949 is important in order to ensure that your recipe rebuilds (or does
950 not rebuild) appropriately in response to changes in configuration,
951 and to ensure that you get the appropriate packages installed on the
952 target machine, particularly if you run separate builds for more than
953 one target machine.
954
955Sharing Files Between Recipes
956=============================
957
958Recipes often need to use files provided by other recipes on the build
959host. For example, an application linking to a common library needs
960access to the library itself and its associated headers. The way this
961access is accomplished is by populating a sysroot with files. Each
962recipe has two sysroots in its work directory, one for target files
963(``recipe-sysroot``) and one for files that are native to the build host
964(``recipe-sysroot-native``).
965
966.. note::
967
968 You could find the term "staging" used within the Yocto project
969 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
970 variable).
971
972Recipes should never populate the sysroot directly (i.e. write files
973into sysroot). Instead, files should be installed into standard
974locations during the
975:ref:`ref-tasks-install` task within
976the ``${``\ :term:`D`\ ``}`` directory. The
977reason for this limitation is that almost all files that populate the
978sysroot are cataloged in manifests in order to ensure the files can be
979removed later when a recipe is either modified or removed. Thus, the
980sysroot is able to remain free from stale files.
981
982A subset of the files installed by the :ref:`ref-tasks-install` task are
983used by the :ref:`ref-tasks-populate_sysroot` task as defined by the
984:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
985is possible to modify the list of directories that populate the sysroot.
986The following example shows how you could add the ``/opt`` directory to
987the list of directories within a recipe::
988
989 SYSROOT_DIRS += "/opt"
990
991.. note::
992
993 The `/sysroot-only` is to be used by recipes that generate artifacts
994 that are not included in the target filesystem, allowing them to share
995 these artifacts without needing to use the :term:`DEPLOY_DIR`.
996
997For a more complete description of the :ref:`ref-tasks-populate_sysroot`
998task and its associated functions, see the
999:ref:`staging <ref-classes-staging>` class.
1000
1001Using Virtual Providers
1002=======================
1003
1004Prior to a build, if you know that several different recipes provide the
1005same functionality, you can use a virtual provider (i.e. ``virtual/*``)
1006as a placeholder for the actual provider. The actual provider is
1007determined at build-time.
1008
1009A common scenario where a virtual provider is used would be for the
1010kernel recipe. Suppose you have three kernel recipes whose
1011:term:`PN` values map to ``kernel-big``,
1012``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
1013in some way uses a :term:`PROVIDES`
1014statement that essentially identifies itself as being able to provide
1015``virtual/kernel``. Here is one way through the
1016:ref:`kernel <ref-classes-kernel>` class::
1017
1018 PROVIDES += "virtual/kernel"
1019
1020Any recipe that inherits the :ref:`kernel <ref-classes-kernel>` class is
1021going to utilize a :term:`PROVIDES` statement that identifies that recipe as
1022being able to provide the ``virtual/kernel`` item.
1023
1024Now comes the time to actually build an image and you need a kernel
1025recipe, but which one? You can configure your build to call out the
1026kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
1027an example, consider the :yocto_git:`x86-base.inc
1028</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a
1029machine (i.e. :term:`MACHINE`) configuration file. This include file is the
1030reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
1031relevant lines from the include file::
1032
1033 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
1034 PREFERRED_VERSION_linux-yocto ??= "4.15%"
1035
1036When you use a virtual provider, you do not have to "hard code" a recipe
1037name as a build dependency. You can use the
1038:term:`DEPENDS` variable to state the
1039build is dependent on ``virtual/kernel`` for example::
1040
1041 DEPENDS = "virtual/kernel"
1042
1043During the build, the OpenEmbedded build system picks
1044the correct recipe needed for the ``virtual/kernel`` dependency based on
1045the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
1046mentioned at the beginning of this section, configure your build as
1047follows::
1048
1049 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
1050
1051.. note::
1052
1053 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
1054 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
1055 recipes from building is usually the desired behavior since this mechanism's
1056 purpose is to select between mutually exclusive alternative providers.
1057
1058The following lists specific examples of virtual providers:
1059
1060- ``virtual/kernel``: Provides the name of the kernel recipe to use
1061 when building a kernel image.
1062
1063- ``virtual/bootloader``: Provides the name of the bootloader to use
1064 when building an image.
1065
1066- ``virtual/libgbm``: Provides ``gbm.pc``.
1067
1068- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
1069
1070- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
1071
1072- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
1073
1074- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
1075
1076.. note::
1077
1078 Virtual providers only apply to build time dependencies specified with
1079 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
1080 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
1081
1082Properly Versioning Pre-Release Recipes
1083=======================================
1084
1085Sometimes the name of a recipe can lead to versioning problems when the
1086recipe is upgraded to a final release. For example, consider the
1087``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
1088the ":ref:`dev-manual/new-recipe:storing and naming the recipe`" section.
1089This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
1090released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
1091change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
1092build system and package managers, so the resulting packages will not
1093correctly trigger an upgrade.
1094
1095In order to ensure the versions compare properly, the recommended
1096convention is to set :term:`PV` within the
1097recipe to "previous_version+current_version". You can use an additional
1098variable so that you can use the current version elsewhere. Here is an
1099example::
1100
1101 REALPV = "0.8.16-rc1"
1102 PV = "0.8.15+${REALPV}"
1103
1104Post-Installation Scripts
1105=========================
1106
1107Post-installation scripts run immediately after installing a package on
1108the target or during image creation when a package is included in an
1109image. To add a post-installation script to a package, add a
1110``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file
1111(``.bb``) and replace `PACKAGENAME` with the name of the package you want
1112to attach to the ``postinst`` script. To apply the post-installation
1113script to the main package for the recipe, which is usually what is
1114required, specify
1115``${``\ :term:`PN`\ ``}`` in place of
1116PACKAGENAME.
1117
1118A post-installation function has the following structure::
1119
1120 pkg_postinst:PACKAGENAME() {
1121 # Commands to carry out
1122 }
1123
1124The script defined in the post-installation function is called when the
1125root filesystem is created. If the script succeeds, the package is
1126marked as installed.
1127
1128.. note::
1129
1130 Any RPM post-installation script that runs on the target should
1131 return a 0 exit code. RPM does not allow non-zero exit codes for
1132 these scripts, and the RPM package manager will cause the package to
1133 fail installation on the target.
1134
1135Sometimes it is necessary for the execution of a post-installation
1136script to be delayed until the first boot. For example, the script might
1137need to be executed on the device itself. To delay script execution
1138until boot time, you must explicitly mark post installs to defer to the
1139target. You can use ``pkg_postinst_ontarget()`` or call
1140``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
1141failure of a ``pkg_postinst()`` script (including exit 1) triggers an
1142error during the
1143:ref:`ref-tasks-rootfs` task.
1144
1145If you have recipes that use ``pkg_postinst`` function and they require
1146the use of non-standard native tools that have dependencies during
1147root filesystem construction, you need to use the
1148:term:`PACKAGE_WRITE_DEPS`
1149variable in your recipe to list these tools. If you do not use this
1150variable, the tools might be missing and execution of the
1151post-installation script is deferred until first boot. Deferring the
1152script to the first boot is undesirable and impossible for read-only
1153root filesystems.
1154
1155.. note::
1156
1157 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
1158 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
1159 respectively. These scrips work in exactly the same way as does
1160 ``pkg_postinst`` with the exception that they run at different times. Also,
1161 because of when they run, they are not applicable to being run at image
1162 creation time like ``pkg_postinst``.
1163
1164Testing
1165=======
1166
1167The final step for completing your recipe is to be sure that the
1168software you built runs correctly. To accomplish runtime testing, add
1169the build's output packages to your image and test them on the target.
1170
1171For information on how to customize your image by adding specific
1172packages, see ":ref:`dev-manual/customizing-images:customizing images`" section.
1173
1174Examples
1175========
1176
1177To help summarize how to write a recipe, this section provides some
1178examples given various scenarios:
1179
1180- Recipes that use local files
1181
1182- Using an Autotooled package
1183
1184- Using a Makefile-based package
1185
1186- Splitting an application into multiple packages
1187
1188- Adding binaries to an image
1189
1190Single .c File Package (Hello World!)
1191-------------------------------------
1192
1193Building an application from a single file that is stored locally (e.g.
1194under ``files``) requires a recipe that has the file listed in the
1195:term:`SRC_URI` variable. Additionally, you need to manually write the
1196:ref:`ref-tasks-compile` and :ref:`ref-tasks-install` tasks. The :term:`S` variable defines the
1197directory containing the source code, which is set to
1198:term:`WORKDIR` in this case --- the
1199directory BitBake uses for the build.
1200::
1201
1202 SUMMARY = "Simple helloworld application"
1203 SECTION = "examples"
1204 LICENSE = "MIT"
1205 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
1206
1207 SRC_URI = "file://helloworld.c"
1208
1209 S = "${WORKDIR}"
1210
1211 do_compile() {
1212 ${CC} ${LDFLAGS} helloworld.c -o helloworld
1213 }
1214
1215 do_install() {
1216 install -d ${D}${bindir}
1217 install -m 0755 helloworld ${D}${bindir}
1218 }
1219
1220By default, the ``helloworld``, ``helloworld-dbg``, and
1221``helloworld-dev`` packages are built. For information on how to
1222customize the packaging process, see the
1223":ref:`dev-manual/new-recipe:splitting an application into multiple packages`"
1224section.
1225
1226Autotooled Package
1227------------------
1228
1229Applications that use Autotools such as ``autoconf`` and ``automake``
1230require a recipe that has a source archive listed in :term:`SRC_URI` and
1231also inherit the :ref:`autotools <ref-classes-autotools>` class,
1232which contains the definitions of all the steps needed to build an
1233Autotool-based application. The result of the build is automatically
1234packaged. And, if the application uses NLS for localization, packages
1235with local information are generated (one package per language).
1236Following is one example: (``hello_2.3.bb``)
1237::
1238
1239 SUMMARY = "GNU Helloworld application"
1240 SECTION = "examples"
1241 LICENSE = "GPL-2.0-or-later"
1242 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
1243
1244 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
1245
1246 inherit autotools gettext
1247
1248The variable :term:`LIC_FILES_CHKSUM` is used to track source license
1249changes as described in the
1250":ref:`dev-manual/licenses:tracking license changes`" section in
1251the Yocto Project Overview and Concepts Manual. You can quickly create
1252Autotool-based recipes in a manner similar to the previous example.
1253
1254Makefile-Based Package
1255----------------------
1256
1257Applications that use GNU ``make`` also require a recipe that has the
1258source archive listed in :term:`SRC_URI`. You do not need to add a
1259:ref:`ref-tasks-compile` step since by default BitBake starts the ``make`` command
1260to compile the application. If you need additional ``make`` options, you
1261should store them in the
1262:term:`EXTRA_OEMAKE` or
1263:term:`PACKAGECONFIG_CONFARGS`
1264variables. BitBake passes these options into the GNU ``make``
1265invocation. Note that a :ref:`ref-tasks-install` task is still required.
1266Otherwise, BitBake runs an empty :ref:`ref-tasks-install` task by default.
1267
1268Some applications might require extra parameters to be passed to the
1269compiler. For example, the application might need an additional header
1270path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
1271following example shows this::
1272
1273 CFLAGS:prepend = "-I ${S}/include "
1274
1275In the following example, ``lz4`` is a makefile-based package::
1276
1277 SUMMARY = "Extremely Fast Compression algorithm"
1278 DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems."
1279 HOMEPAGE = "https://github.com/lz4/lz4"
1280
1281 LICENSE = "BSD-2-Clause | GPL-2.0-only"
1282 LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc \
1283 file://programs/COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
1284 file://LICENSE;md5=d57c0d21cb917fb4e0af2454aa48b956 \
1285 "
1286
1287 PE = "1"
1288
1289 SRCREV = "d44371841a2f1728a3f36839fd4b7e872d0927d3"
1290
1291 SRC_URI = "git://github.com/lz4/lz4.git;branch=release;protocol=https \
1292 file://CVE-2021-3520.patch \
1293 "
1294 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>.*)"
1295
1296 S = "${WORKDIR}/git"
1297
1298 # Fixed in r118, which is larger than the current version.
1299 CVE_CHECK_IGNORE += "CVE-2014-4715"
1300
1301 EXTRA_OEMAKE = "PREFIX=${prefix} CC='${CC}' CFLAGS='${CFLAGS}' DESTDIR=${D} LIBDIR=${libdir} INCLUDEDIR=${includedir} BUILD_STATIC=no"
1302
1303 do_install() {
1304 oe_runmake install
1305 }
1306
1307 BBCLASSEXTEND = "native nativesdk"
1308
1309Splitting an Application into Multiple Packages
1310-----------------------------------------------
1311
1312You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
1313application into multiple packages.
1314
1315Following is an example that uses the ``libxpm`` recipe. By default,
1316this recipe generates a single package that contains the library along
1317with a few binaries. You can modify the recipe to split the binaries
1318into separate packages::
1319
1320 require xorg-lib-common.inc
1321
1322 SUMMARY = "Xpm: X Pixmap extension library"
1323 LICENSE = "MIT"
1324 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
1325 DEPENDS += "libxext libsm libxt"
1326 PE = "1"
1327
1328 XORG_PN = "libXpm"
1329
1330 PACKAGES =+ "sxpm cxpm"
1331 FILES:cxpm = "${bindir}/cxpm"
1332 FILES:sxpm = "${bindir}/sxpm"
1333
1334In the previous example, we want to ship the ``sxpm`` and ``cxpm``
1335binaries in separate packages. Since ``bindir`` would be packaged into
1336the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
1337so additional package names are added to the start of list. This results
1338in the extra ``FILES:*`` variables then containing information that
1339define which files and directories go into which packages. Files
1340included by earlier packages are skipped by latter packages. Thus, the
1341main :term:`PN` package does not include the above listed files.
1342
1343Packaging Externally Produced Binaries
1344--------------------------------------
1345
1346Sometimes, you need to add pre-compiled binaries to an image. For
1347example, suppose that there are binaries for proprietary code,
1348created by a particular division of a company. Your part of the company
1349needs to use those binaries as part of an image that you are building
1350using the OpenEmbedded build system. Since you only have the binaries
1351and not the source code, you cannot use a typical recipe that expects to
1352fetch the source specified in
1353:term:`SRC_URI` and then compile it.
1354
1355One method is to package the binaries and then install them as part of
1356the image. Generally, it is not a good idea to package binaries since,
1357among other things, it can hinder the ability to reproduce builds and
1358could lead to compatibility problems with ABI in the future. However,
1359sometimes you have no choice.
1360
1361The easiest solution is to create a recipe that uses the
1362:ref:`bin_package <ref-classes-bin-package>` class
1363and to be sure that you are using default locations for build artifacts.
1364In most cases, the :ref:`bin_package <ref-classes-bin-package>` class handles "skipping" the
1365configure and compile steps as well as sets things up to grab packages
1366from the appropriate area. In particular, this class sets ``noexec`` on
1367both the :ref:`ref-tasks-configure`
1368and :ref:`ref-tasks-compile` tasks,
1369sets ``FILES:${PN}`` to "/" so that it picks up all files, and sets up a
1370:ref:`ref-tasks-install` task, which
1371effectively copies all files from ``${S}`` to ``${D}``. The
1372:ref:`bin_package <ref-classes-bin-package>` class works well when the files extracted into ``${S}``
1373are already laid out in the way they should be laid out on the target.
1374For more information on these variables, see the
1375:term:`FILES`,
1376:term:`PN`,
1377:term:`S`, and
1378:term:`D` variables in the Yocto Project
1379Reference Manual's variable glossary.
1380
1381.. note::
1382
1383 - Using :term:`DEPENDS` is a good
1384 idea even for components distributed in binary form, and is often
1385 necessary for shared libraries. For a shared library, listing the
1386 library dependencies in :term:`DEPENDS` makes sure that the libraries
1387 are available in the staging sysroot when other recipes link
1388 against the library, which might be necessary for successful
1389 linking.
1390
1391 - Using :term:`DEPENDS` also allows runtime dependencies between
1392 packages to be added automatically. See the
1393 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
1394 section in the Yocto Project Overview and Concepts Manual for more
1395 information.
1396
1397If you cannot use the :ref:`bin_package <ref-classes-bin-package>` class, you need to be sure you are
1398doing the following:
1399
1400- Create a recipe where the
1401 :ref:`ref-tasks-configure` and
1402 :ref:`ref-tasks-compile` tasks do
1403 nothing: It is usually sufficient to just not define these tasks in
1404 the recipe, because the default implementations do nothing unless a
1405 Makefile is found in
1406 ``${``\ :term:`S`\ ``}``.
1407
1408 If ``${S}`` might contain a Makefile, or if you inherit some class
1409 that replaces :ref:`ref-tasks-configure` and :ref:`ref-tasks-compile` with custom
1410 versions, then you can use the
1411 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
1412 flag to turn the tasks into no-ops, as follows::
1413
1414 do_configure[noexec] = "1"
1415 do_compile[noexec] = "1"
1416
1417 Unlike
1418 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
1419 using the flag preserves the dependency chain from the
1420 :ref:`ref-tasks-fetch`,
1421 :ref:`ref-tasks-unpack`, and
1422 :ref:`ref-tasks-patch` tasks to the
1423 :ref:`ref-tasks-install` task.
1424
1425- Make sure your :ref:`ref-tasks-install` task installs the binaries
1426 appropriately.
1427
1428- Ensure that you set up :term:`FILES`
1429 (usually
1430 ``FILES:${``\ :term:`PN`\ ``}``) to
1431 point to the files you have installed, which of course depends on
1432 where you have installed them and whether those files are in
1433 different locations than the defaults.
1434
1435Following Recipe Style Guidelines
1436=================================
1437
1438When writing recipes, it is good to conform to existing style
1439guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
1440provides rough guidelines for preferred recipe style.
1441
1442It is common for existing recipes to deviate a bit from this style.
1443However, aiming for at least a consistent style is a good idea. Some
1444practices, such as omitting spaces around ``=`` operators in assignments
1445or ordering recipe components in an erratic way, are widely seen as poor
1446style.
1447
1448Recipe Syntax
1449=============
1450
1451Understanding recipe file syntax is important for writing recipes. The
1452following list overviews the basic items that make up a BitBake recipe
1453file. For more complete BitBake syntax descriptions, see the
1454":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`"
1455chapter of the BitBake User Manual.
1456
1457- *Variable Assignments and Manipulations:* Variable assignments allow
1458 a value to be assigned to a variable. The assignment can be static
1459 text or might include the contents of other variables. In addition to
1460 the assignment, appending and prepending operations are also
1461 supported.
1462
1463 The following example shows some of the ways you can use variables in
1464 recipes::
1465
1466 S = "${WORKDIR}/postfix-${PV}"
1467 CFLAGS += "-DNO_ASM"
1468 CFLAGS:append = " --enable-important-feature"
1469
1470- *Functions:* Functions provide a series of actions to be performed.
1471 You usually use functions to override the default implementation of a
1472 task function or to complement a default function (i.e. append or
1473 prepend to an existing function). Standard functions use ``sh`` shell
1474 syntax, although access to OpenEmbedded variables and internal
1475 methods are also available.
1476
1477 Here is an example function from the ``sed`` recipe::
1478
1479 do_install () {
1480 autotools_do_install
1481 install -d ${D}${base_bindir}
1482 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
1483 rmdir ${D}${bindir}/
1484 }
1485
1486 It is
1487 also possible to implement new functions that are called between
1488 existing tasks as long as the new functions are not replacing or
1489 complementing the default functions. You can implement functions in
1490 Python instead of shell. Both of these options are not seen in the
1491 majority of recipes.
1492
1493- *Keywords:* BitBake recipes use only a few keywords. You use keywords
1494 to include common functions (``inherit``), load parts of a recipe
1495 from other files (``include`` and ``require``) and export variables
1496 to the environment (``export``).
1497
1498 The following example shows the use of some of these keywords::
1499
1500 export POSTCONF = "${STAGING_BINDIR}/postconf"
1501 inherit autoconf
1502 require otherfile.inc
1503
1504- *Comments (#):* Any lines that begin with the hash character (``#``)
1505 are treated as comment lines and are ignored::
1506
1507 # This is a comment
1508
1509This next list summarizes the most important and most commonly used
1510parts of the recipe syntax. For more information on these parts of the
1511syntax, you can reference the
1512":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
1513in the BitBake User Manual.
1514
1515- *Line Continuation (\\):* Use the backward slash (``\``) character to
1516 split a statement over multiple lines. Place the slash character at
1517 the end of the line that is to be continued on the next line::
1518
1519 VAR = "A really long \
1520 line"
1521
1522 .. note::
1523
1524 You cannot have any characters including spaces or tabs after the
1525 slash character.
1526
1527- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
1528 access the contents of a variable::
1529
1530 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
1531
1532 .. note::
1533
1534 It is important to understand that the value of a variable
1535 expressed in this form does not get substituted automatically. The
1536 expansion of these expressions happens on-demand later (e.g.
1537 usually when a function that makes reference to the variable
1538 executes). This behavior ensures that the values are most
1539 appropriate for the context in which they are finally used. On the
1540 rare occasion that you do need the variable expression to be
1541 expanded immediately, you can use the
1542 :=
1543 operator instead of
1544 =
1545 when you make the assignment, but this is not generally needed.
1546
1547- *Quote All Assignments ("value"):* Use double quotes around values in
1548 all variable assignments (e.g. ``"value"``). Following is an example::
1549
1550 VAR1 = "${OTHERVAR}"
1551 VAR2 = "The version is ${PV}"
1552
1553- *Conditional Assignment (?=):* Conditional assignment is used to
1554 assign a value to a variable, but only when the variable is currently
1555 unset. Use the question mark followed by the equal sign (``?=``) to
1556 make a "soft" assignment used for conditional assignment. Typically,
1557 "soft" assignments are used in the ``local.conf`` file for variables
1558 that are allowed to come through from the external environment.
1559
1560 Here is an example where ``VAR1`` is set to "New value" if it is
1561 currently empty. However, if ``VAR1`` has already been set, it
1562 remains unchanged::
1563
1564 VAR1 ?= "New value"
1565
1566 In this next example, ``VAR1`` is left with the value "Original value"::
1567
1568 VAR1 = "Original value"
1569 VAR1 ?= "New value"
1570
1571- *Appending (+=):* Use the plus character followed by the equals sign
1572 (``+=``) to append values to existing variables.
1573
1574 .. note::
1575
1576 This operator adds a space between the existing content of the
1577 variable and the new content.
1578
1579 Here is an example::
1580
1581 SRC_URI += "file://fix-makefile.patch"
1582
1583- *Prepending (=+):* Use the equals sign followed by the plus character
1584 (``=+``) to prepend values to existing variables.
1585
1586 .. note::
1587
1588 This operator adds a space between the new content and the
1589 existing content of the variable.
1590
1591 Here is an example::
1592
1593 VAR =+ "Starts"
1594
1595- *Appending (:append):* Use the ``:append`` operator to append values
1596 to existing variables. This operator does not add any additional
1597 space. Also, the operator is applied after all the ``+=``, and ``=+``
1598 operators have been applied and after all ``=`` assignments have
1599 occurred. This means that if ``:append`` is used in a recipe, it can
1600 only be overridden by another layer using the special ``:remove``
1601 operator, which in turn will prevent further layers from adding it back.
1602
1603 The following example shows the space being explicitly added to the
1604 start to ensure the appended value is not merged with the existing
1605 value::
1606
1607 CFLAGS:append = " --enable-important-feature"
1608
1609 You can also use
1610 the ``:append`` operator with overrides, which results in the actions
1611 only being performed for the specified target or machine::
1612
1613 CFLAGS:append:sh4 = " --enable-important-sh4-specific-feature"
1614
1615- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend
1616 values to existing variables. This operator does not add any
1617 additional space. Also, the operator is applied after all the ``+=``,
1618 and ``=+`` operators have been applied and after all ``=``
1619 assignments have occurred.
1620
1621 The following example shows the space being explicitly added to the
1622 end to ensure the prepended value is not merged with the existing
1623 value::
1624
1625 CFLAGS:prepend = "-I${S}/myincludes "
1626
1627 You can also use the
1628 ``:prepend`` operator with overrides, which results in the actions
1629 only being performed for the specified target or machine::
1630
1631 CFLAGS:prepend:sh4 = "-I${S}/myincludes "
1632
1633- *Overrides:* You can use overrides to set a value conditionally,
1634 typically based on how the recipe is being built. For example, to set
1635 the :term:`KBRANCH` variable's
1636 value to "standard/base" for any target
1637 :term:`MACHINE`, except for
1638 qemuarm where it should be set to "standard/arm-versatile-926ejs",
1639 you would do the following::
1640
1641 KBRANCH = "standard/base"
1642 KBRANCH:qemuarm = "standard/arm-versatile-926ejs"
1643
1644 Overrides are also used to separate
1645 alternate values of a variable in other situations. For example, when
1646 setting variables such as
1647 :term:`FILES` and
1648 :term:`RDEPENDS` that are
1649 specific to individual packages produced by a recipe, you should
1650 always use an override that specifies the name of the package.
1651
1652- *Indentation:* Use spaces for indentation rather than tabs. For
1653 shell functions, both currently work. However, it is a policy
1654 decision of the Yocto Project to use tabs in shell functions. Realize
1655 that some layers have a policy to use spaces for all indentation.
1656
1657- *Using Python for Complex Operations:* For more advanced processing,
1658 it is possible to use Python code during variable assignments (e.g.
1659 search and replacement on a variable).
1660
1661 You indicate Python code using the ``${@python_code}`` syntax for the
1662 variable assignment::
1663
1664 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
1665
1666- *Shell Function Syntax:* Write shell functions as if you were writing
1667 a shell script when you describe a list of actions to take. You
1668 should ensure that your script works with a generic ``sh`` and that
1669 it does not require any ``bash`` or other shell-specific
1670 functionality. The same considerations apply to various system
1671 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
1672 might wish to use. If in doubt, you should check with multiple
1673 implementations --- including those from BusyBox.
1674
diff --git a/documentation/dev-manual/packages.rst b/documentation/dev-manual/packages.rst
new file mode 100644
index 0000000000..5f7bd68b9a
--- /dev/null
+++ b/documentation/dev-manual/packages.rst
@@ -0,0 +1,1252 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working with Packages
4*********************
5
6This section describes a few tasks that involve packages:
7
8- :ref:`dev-manual/packages:excluding packages from an image`
9
10- :ref:`dev-manual/packages:incrementing a package version`
11
12- :ref:`dev-manual/packages:handling optional module packaging`
13
14- :ref:`dev-manual/packages:using runtime package management`
15
16- :ref:`dev-manual/packages:generating and using signed packages`
17
18- :ref:`Setting up and running package test
19 (ptest) <dev-manual/packages:testing packages with ptest>`
20
21- :ref:`dev-manual/packages:creating node package manager (npm) packages`
22
23- :ref:`dev-manual/packages:adding custom metadata to packages`
24
25Excluding Packages from an Image
26================================
27
28You might find it necessary to prevent specific packages from being
29installed into an image. If so, you can use several variables to direct
30the build system to essentially ignore installing recommended packages
31or to not install a package at all.
32
33The following list introduces variables you can use to prevent packages
34from being installed into your image. Each of these variables only works
35with IPK and RPM package types, not for Debian packages.
36Also, you can use these variables from your ``local.conf`` file
37or attach them to a specific image recipe by using a recipe name
38override. For more detail on the variables, see the descriptions in the
39Yocto Project Reference Manual's glossary chapter.
40
41- :term:`BAD_RECOMMENDATIONS`:
42 Use this variable to specify "recommended-only" packages that you do
43 not want installed.
44
45- :term:`NO_RECOMMENDATIONS`:
46 Use this variable to prevent all "recommended-only" packages from
47 being installed.
48
49- :term:`PACKAGE_EXCLUDE`:
50 Use this variable to prevent specific packages from being installed
51 regardless of whether they are "recommended-only" or not. You need to
52 realize that the build process could fail with an error when you
53 prevent the installation of a package whose presence is required by
54 an installed package.
55
56Incrementing a Package Version
57==============================
58
59This section provides some background on how binary package versioning
60is accomplished and presents some of the services, variables, and
61terminology involved.
62
63In order to understand binary package versioning, you need to consider
64the following:
65
66- Binary Package: The binary package that is eventually built and
67 installed into an image.
68
69- Binary Package Version: The binary package version is composed of two
70 components --- a version and a revision.
71
72 .. note::
73
74 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
75 but this discussion for the most part ignores :term:`PE`.
76
77 The version and revision are taken from the
78 :term:`PV` and
79 :term:`PR` variables, respectively.
80
81- :term:`PV`: The recipe version. :term:`PV` represents the version of the
82 software being packaged. Do not confuse :term:`PV` with the binary
83 package version.
84
85- :term:`PR`: The recipe revision.
86
87- :term:`SRCPV`: The OpenEmbedded
88 build system uses this string to help define the value of :term:`PV` when
89 the source code revision needs to be included in it.
90
91- :yocto_wiki:`PR Service </PR_Service>`: A
92 network-based service that helps automate keeping package feeds
93 compatible with existing package manager applications such as RPM,
94 APT, and OPKG.
95
96Whenever the binary package content changes, the binary package version
97must change. Changing the binary package version is accomplished by
98changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
99values occurs one of two ways:
100
101- Automatically using a Package Revision Service (PR Service).
102
103- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
104
105Given a primary challenge of any build system and its users is how to
106maintain a package feed that is compatible with existing package manager
107applications such as RPM, APT, and OPKG, using an automated system is
108much preferred over a manual system. In either system, the main
109requirement is that binary package version numbering increases in a
110linear fashion and that there is a number of version components that
111support that linear progression. For information on how to ensure
112package revisioning remains linear, see the
113":ref:`dev-manual/packages:automatically incrementing a package version number`"
114section.
115
116The following three sections provide related information on the PR
117Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
118how to ensure binary package revisioning remains linear.
119
120Working With a PR Service
121-------------------------
122
123As mentioned, attempting to maintain revision numbers in the
124:term:`Metadata` is error prone, inaccurate,
125and causes problems for people submitting recipes. Conversely, the PR
126Service automatically generates increasing numbers, particularly the
127revision field, which removes the human element.
128
129.. note::
130
131 For additional information on using a PR Service, you can see the
132 :yocto_wiki:`PR Service </PR_Service>` wiki page.
133
134The Yocto Project uses variables in order of decreasing priority to
135facilitate revision numbering (i.e.
136:term:`PE`,
137:term:`PV`, and
138:term:`PR` for epoch, version, and
139revision, respectively). The values are highly dependent on the policies
140and procedures of a given distribution and package feed.
141
142Because the OpenEmbedded build system uses
143":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
144unique to a given build, the build system knows when to rebuild
145packages. All the inputs into a given task are represented by a
146signature, which can trigger a rebuild when different. Thus, the build
147system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
148trigger a rebuild. The signatures, however, can be used to generate
149these values.
150
151The PR Service works with both ``OEBasic`` and ``OEBasicHash``
152generators. The value of :term:`PR` bumps when the checksum changes and the
153different generator mechanisms change signatures under different
154circumstances.
155
156As implemented, the build system includes values from the PR Service
157into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
158becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
159:term:`PR` values to be used for whatever reasons, which include manual
160:term:`PR` bumps, should it be necessary.
161
162By default, the PR Service is not enabled or running. Thus, the packages
163generated are just "self consistent". The build system adds and removes
164packages and there are no guarantees about upgrade paths but images will
165be consistent and correct with the latest changes.
166
167The simplest form for a PR Service is for a single host development system
168that builds the package feed (building system). For this scenario, you can
169enable a local PR Service by setting :term:`PRSERV_HOST` in your
170``local.conf`` file in the :term:`Build Directory`::
171
172 PRSERV_HOST = "localhost:0"
173
174Once the service is started, packages will automatically
175get increasing :term:`PR` values and BitBake takes care of starting and
176stopping the server.
177
178If you have a more complex setup where multiple host development systems
179work against a common, shared package feed, you have a single PR Service
180running and it is connected to each building system. For this scenario,
181you need to start the PR Service using the ``bitbake-prserv`` command::
182
183 bitbake-prserv --host ip --port port --start
184
185In addition to
186hand-starting the service, you need to update the ``local.conf`` file of
187each building system as described earlier so each system points to the
188server and port.
189
190It is also recommended you use build history, which adds some sanity
191checks to binary package versions, in conjunction with the server that
192is running the PR Service. To enable build history, add the following to
193each building system's ``local.conf`` file::
194
195 # It is recommended to activate "buildhistory" for testing the PR service
196 INHERIT += "buildhistory"
197 BUILDHISTORY_COMMIT = "1"
198
199For information on build
200history, see the
201":ref:`dev-manual/build-quality:maintaining build output quality`" section.
202
203.. note::
204
205 The OpenEmbedded build system does not maintain :term:`PR` information as
206 part of the shared state (sstate) packages. If you maintain an sstate
207 feed, it's expected that either all your building systems that
208 contribute to the sstate feed use a shared PR Service, or you do not
209 run a PR Service on any of your building systems. Having some systems
210 use a PR Service while others do not leads to obvious problems.
211
212 For more information on shared state, see the
213 ":ref:`overview-manual/concepts:shared state cache`"
214 section in the Yocto Project Overview and Concepts Manual.
215
216Manually Bumping PR
217-------------------
218
219The alternative to setting up a PR Service is to manually "bump" the
220:term:`PR` variable.
221
222If a committed change results in changing the package output, then the
223value of the PR variable needs to be increased (or "bumped") as part of
224that commit. For new recipes you should add the :term:`PR` variable and set
225its initial value equal to "r0", which is the default. Even though the
226default value is "r0", the practice of adding it to a new recipe makes
227it harder to forget to bump the variable when you make changes to the
228recipe in future.
229
230If you are sharing a common ``.inc`` file with multiple recipes, you can
231also use the :term:`INC_PR` variable to ensure that the recipes sharing the
232``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
233``.inc`` file must set :term:`INC_PR` (initially to "r0"), and all recipes
234referring to it should set :term:`PR` to "${INC_PR}.0" initially,
235incrementing the last number when the recipe is changed. If the ``.inc``
236file is changed then its :term:`INC_PR` should be incremented.
237
238When upgrading the version of a binary package, assuming the :term:`PV`
239changes, the :term:`PR` variable should be reset to "r0" (or "${INC_PR}.0"
240if you are using :term:`INC_PR`).
241
242Usually, version increases occur only to binary packages. However, if
243for some reason :term:`PV` changes but does not increase, you can increase
244the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
245"0".
246
247Binary package version numbering strives to follow the `Debian Version
248Field Policy
249Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
250These guidelines define how versions are compared and what "increasing"
251a version means.
252
253Automatically Incrementing a Package Version Number
254---------------------------------------------------
255
256When fetching a repository, BitBake uses the
257:term:`SRCREV` variable to determine
258the specific source code revision from which to build. You set the
259:term:`SRCREV` variable to
260:term:`AUTOREV` to cause the
261OpenEmbedded build system to automatically use the latest revision of
262the software::
263
264 SRCREV = "${AUTOREV}"
265
266Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
267automatically update the version whenever the revision of the source
268code changes. Here is an example::
269
270 PV = "1.0+git${SRCPV}"
271
272The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
273
274.. code-block:: none
275
276 AUTOINC+source_code_revision
277
278The build system replaces the ``AUTOINC``
279with a number. The number used depends on the state of the PR Service:
280
281- If PR Service is enabled, the build system increments the number,
282 which is similar to the behavior of
283 :term:`PR`. This behavior results in
284 linearly increasing package versions, which is desirable. Here is an
285 example:
286
287 .. code-block:: none
288
289 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
290 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
291
292- If PR Service is not enabled, the build system replaces the
293 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
294 changing the package version since the source revision is included.
295 However, package versions are not increased linearly. Here is an
296 example:
297
298 .. code-block:: none
299
300 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
301 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
302
303In summary, the OpenEmbedded build system does not track the history of
304binary package versions for this purpose. ``AUTOINC``, in this case, is
305comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
306package version is simply replaced by "0". If PR server is enabled, the
307build system keeps track of the package versions and bumps the number
308when the package revision changes.
309
310Handling Optional Module Packaging
311==================================
312
313Many pieces of software split functionality into optional modules (or
314plugins) and the plugins that are built might depend on configuration
315options. To avoid having to duplicate the logic that determines what
316modules are available in your recipe or to avoid having to package each
317module by hand, the OpenEmbedded build system provides functionality to
318handle module packaging dynamically.
319
320To handle optional module packaging, you need to do two things:
321
322- Ensure the module packaging is actually done.
323
324- Ensure that any dependencies on optional modules from other recipes
325 are satisfied by your recipe.
326
327Making Sure the Packaging is Done
328---------------------------------
329
330To ensure the module packaging actually gets done, you use the
331``do_split_packages`` function within the ``populate_packages`` Python
332function in your recipe. The ``do_split_packages`` function searches for
333a pattern of files or directories under a specified path and creates a
334package for each one it finds by appending to the
335:term:`PACKAGES` variable and
336setting the appropriate values for ``FILES:packagename``,
337``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
338Here is an example from the ``lighttpd`` recipe::
339
340 python populate_packages:prepend () {
341 lighttpd_libdir = d.expand('${libdir}')
342 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
343 'lighttpd-module-%s', 'Lighttpd module for %s',
344 extra_depends='')
345 }
346
347The previous example specifies a number of things in the call to
348``do_split_packages``.
349
350- A directory within the files installed by your recipe through
351 :ref:`ref-tasks-install` in which to search.
352
353- A regular expression used to match module files in that directory. In
354 the example, note the parentheses () that mark the part of the
355 expression from which the module name should be derived.
356
357- A pattern to use for the package names.
358
359- A description for each package.
360
361- An empty string for ``extra_depends``, which disables the default
362 dependency on the main ``lighttpd`` package. Thus, if a file in
363 ``${libdir}`` called ``mod_alias.so`` is found, a package called
364 ``lighttpd-module-alias`` is created for it and the
365 :term:`DESCRIPTION` is set to
366 "Lighttpd module for alias".
367
368Often, packaging modules is as simple as the previous example. However,
369there are more advanced options that you can use within
370``do_split_packages`` to modify its behavior. And, if you need to, you
371can add more logic by specifying a hook function that is called for each
372package. It is also perfectly acceptable to call ``do_split_packages``
373multiple times if you have more than one set of modules to package.
374
375For more examples that show how to use ``do_split_packages``, see the
376``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
377directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
378also find examples in ``meta/classes-recipe/kernel.bbclass``.
379
380Following is a reference that shows ``do_split_packages`` mandatory and
381optional arguments::
382
383 Mandatory arguments
384
385 root
386 The path in which to search
387 file_regex
388 Regular expression to match searched files.
389 Use parentheses () to mark the part of this
390 expression that should be used to derive the
391 module name (to be substituted where %s is
392 used in other function arguments as noted below)
393 output_pattern
394 Pattern to use for the package names. Must
395 include %s.
396 description
397 Description to set for each package. Must
398 include %s.
399
400 Optional arguments
401
402 postinst
403 Postinstall script to use for all packages
404 (as a string)
405 recursive
406 True to perform a recursive search --- default
407 False
408 hook
409 A hook function to be called for every match.
410 The function will be called with the following
411 arguments (in the order listed):
412
413 f
414 Full path to the file/directory match
415 pkg
416 The package name
417 file_regex
418 As above
419 output_pattern
420 As above
421 modulename
422 The module name derived using file_regex
423 extra_depends
424 Extra runtime dependencies (RDEPENDS) to be
425 set for all packages. The default value of None
426 causes a dependency on the main package
427 (${PN}) --- if you do not want this, pass empty
428 string '' for this parameter.
429 aux_files_pattern
430 Extra item(s) to be added to FILES for each
431 package. Can be a single string item or a list
432 of strings for multiple items. Must include %s.
433 postrm
434 postrm script to use for all packages (as a
435 string)
436 allow_dirs
437 True to allow directories to be matched -
438 default False
439 prepend
440 If True, prepend created packages to PACKAGES
441 instead of the default False which appends them
442 match_path
443 match file_regex on the whole relative path to
444 the root rather than just the filename
445 aux_files_pattern_verbatim
446 Extra item(s) to be added to FILES for each
447 package, using the actual derived module name
448 rather than converting it to something legal
449 for a package name. Can be a single string item
450 or a list of strings for multiple items. Must
451 include %s.
452 allow_links
453 True to allow symlinks to be matched --- default
454 False
455 summary
456 Summary to set for each package. Must include %s;
457 defaults to description if not set.
458
459
460
461Satisfying Dependencies
462-----------------------
463
464The second part for handling optional module packaging is to ensure that
465any dependencies on optional modules from other recipes are satisfied by
466your recipe. You can be sure these dependencies are satisfied by using
467the :term:`PACKAGES_DYNAMIC`
468variable. Here is an example that continues with the ``lighttpd`` recipe
469shown earlier::
470
471 PACKAGES_DYNAMIC = "lighttpd-module-.*"
472
473The name
474specified in the regular expression can of course be anything. In this
475example, it is ``lighttpd-module-`` and is specified as the prefix to
476ensure that any :term:`RDEPENDS` and
477:term:`RRECOMMENDS` on a package
478name starting with the prefix are satisfied during build time. If you
479are using ``do_split_packages`` as described in the previous section,
480the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
481pattern specified in the call to ``do_split_packages``.
482
483Using Runtime Package Management
484================================
485
486During a build, BitBake always transforms a recipe into one or more
487packages. For example, BitBake takes the ``bash`` recipe and produces a
488number of packages (e.g. ``bash``, ``bash-bashbug``,
489``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
490``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
491generated packages are included in an image.
492
493In several situations, you might need to update, add, remove, or query
494the packages on a target device at runtime (i.e. without having to
495generate a new image). Examples of such situations include:
496
497- You want to provide in-the-field updates to deployed devices (e.g.
498 security updates).
499
500- You want to have a fast turn-around development cycle for one or more
501 applications that run on your device.
502
503- You want to temporarily install the "debug" packages of various
504 applications on your device so that debugging can be greatly improved
505 by allowing access to symbols and source debugging.
506
507- You want to deploy a more minimal package selection of your device
508 but allow in-the-field updates to add a larger selection for
509 customization.
510
511In all these situations, you have something similar to a more
512traditional Linux distribution in that in-field devices are able to
513receive pre-compiled packages from a server for installation or update.
514Being able to install these packages on a running, in-field device is
515what is termed "runtime package management".
516
517In order to use runtime package management, you need a host or server
518machine that serves up the pre-compiled packages plus the required
519metadata. You also need package manipulation tools on the target. The
520build machine is a likely candidate to act as the server. However, that
521machine does not necessarily have to be the package server. The build
522machine could push its artifacts to another machine that acts as the
523server (e.g. Internet-facing). In fact, doing so is advantageous for a
524production environment as getting the packages away from the development
525system's :term:`Build Directory` prevents accidental overwrites.
526
527A simple build that targets just one device produces more than one
528package database. In other words, the packages produced by a build are
529separated out into a couple of different package groupings based on
530criteria such as the target's CPU architecture, the target board, or the
531C library used on the target. For example, a build targeting the
532``qemux86`` device produces the following three package databases:
533``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
534device to be aware of all the packages that were available to it, you
535would need to point it to each of these databases individually. In a
536similar way, a traditional Linux distribution usually is configured to
537be aware of a number of software repositories from which it retrieves
538packages.
539
540Using runtime package management is completely optional and not required
541for a successful build or deployment in any way. But if you want to make
542use of runtime package management, you need to do a couple things above
543and beyond the basics. The remainder of this section describes what you
544need to do.
545
546Build Considerations
547--------------------
548
549This section describes build considerations of which you need to be
550aware in order to provide support for runtime package management.
551
552When BitBake generates packages, it needs to know what format or formats
553to use. In your configuration, you use the
554:term:`PACKAGE_CLASSES`
555variable to specify the format:
556
5571. Open the ``local.conf`` file inside your :term:`Build Directory` (e.g.
558 ``poky/build/conf/local.conf``).
559
5602. Select the desired package format as follows::
561
562 PACKAGE_CLASSES ?= "package_packageformat"
563
564 where packageformat can be "ipk", "rpm",
565 "deb", or "tar" which are the supported package formats.
566
567 .. note::
568
569 Because the Yocto Project supports four different package formats,
570 you can set the variable with more than one argument. However, the
571 OpenEmbedded build system only uses the first argument when
572 creating an image or Software Development Kit (SDK).
573
574If you would like your image to start off with a basic package database
575containing the packages in your current build as well as to have the
576relevant tools available on the target for runtime package management,
577you can include "package-management" in the
578:term:`IMAGE_FEATURES`
579variable. Including "package-management" in this configuration variable
580ensures that when the image is assembled for your target, the image
581includes the currently-known package databases as well as the
582target-specific tools required for runtime package management to be
583performed on the target. However, this is not strictly necessary. You
584could start your image off without any databases but only include the
585required on-target package tool(s). As an example, you could include
586"opkg" in your
587:term:`IMAGE_INSTALL` variable
588if you are using the IPK package format. You can then initialize your
589target's package database(s) later once your image is up and running.
590
591Whenever you perform any sort of build step that can potentially
592generate a package or modify existing package, it is always a good idea
593to re-generate the package index after the build by using the following
594command::
595
596 $ bitbake package-index
597
598It might be tempting to build the
599package and the package index at the same time with a command such as
600the following::
601
602 $ bitbake some-package package-index
603
604Do not do this as
605BitBake does not schedule the package index for after the completion of
606the package you are building. Consequently, you cannot be sure of the
607package index including information for the package you just built.
608Thus, be sure to run the package update step separately after building
609any packages.
610
611You can use the
612:term:`PACKAGE_FEED_ARCHS`,
613:term:`PACKAGE_FEED_BASE_PATHS`,
614and
615:term:`PACKAGE_FEED_URIS`
616variables to pre-configure target images to use a package feed. If you
617do not define these variables, then manual steps as described in the
618subsequent sections are necessary to configure the target. You should
619set these variables before building the image in order to produce a
620correctly configured image.
621
622When your build is complete, your packages reside in the
623``${TMPDIR}/deploy/packageformat`` directory. For example, if
624``${``\ :term:`TMPDIR`\ ``}`` is
625``tmp`` and your selected package type is RPM, then your RPM packages
626are available in ``tmp/deploy/rpm``.
627
628Host or Server Machine Setup
629----------------------------
630
631Although other protocols are possible, a server using HTTP typically
632serves packages. If you want to use HTTP, then set up and configure a
633web server such as Apache 2, lighttpd, or Python web server on the
634machine serving the packages.
635
636To keep things simple, this section describes how to set up a
637Python web server to share package feeds from the developer's
638machine. Although this server might not be the best for a production
639environment, the setup is simple and straight forward. Should you want
640to use a different server more suited for production (e.g. Apache 2,
641Lighttpd, or Nginx), take the appropriate steps to do so.
642
643From within the :term:`Build Directory` where you have built an image based on
644your packaging choice (i.e. the :term:`PACKAGE_CLASSES` setting), simply start
645the server. The following example assumes a :term:`Build Directory` of ``poky/build``
646and a :term:`PACKAGE_CLASSES` setting of
647":ref:`package_rpm <ref-classes-package_rpm>`"::
648
649 $ cd poky/build/tmp/deploy/rpm
650 $ python3 -m http.server
651
652Target Setup
653------------
654
655Setting up the target differs depending on the package management
656system. This section provides information for RPM, IPK, and DEB.
657
658Using RPM
659~~~~~~~~~
660
661The :wikipedia:`Dandified Packaging <DNF_(software)>` (DNF) performs
662runtime package management of RPM packages. In order to use DNF for
663runtime package management, you must perform an initial setup on the
664target machine for cases where the ``PACKAGE_FEED_*`` variables were not
665set as part of the image that is running on the target. This means if
666you built your image and did not use these variables as part of the
667build and your image is now running on the target, you need to perform
668the steps in this section if you want to use runtime package management.
669
670.. note::
671
672 For information on the ``PACKAGE_FEED_*`` variables, see
673 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
674 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
675 glossary.
676
677On the target, you must inform DNF that package databases are available.
678You do this by creating a file named
679``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
680
681As an example, assume the target is able to use the following package
682databases: ``all``, ``i586``, and ``qemux86`` from a server named
683``my.server``. The specifics for setting up the web server are up to
684you. The critical requirement is that the URIs in the target repository
685configuration point to the correct remote location for the feeds.
686
687.. note::
688
689 For development purposes, you can point the web server to the build
690 system's ``deploy`` directory. However, for production use, it is better to
691 copy the package directories to a location outside of the build area and use
692 that location. Doing so avoids situations where the build system
693 overwrites or changes the ``deploy`` directory.
694
695When telling DNF where to look for the package databases, you must
696declare individual locations per architecture or a single location used
697for all architectures. You cannot do both:
698
699- *Create an Explicit List of Architectures:* Define individual base
700 URLs to identify where each package database is located:
701
702 .. code-block:: none
703
704 [oe-packages]
705 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
706
707 This example
708 informs DNF about individual package databases for all three
709 architectures.
710
711- *Create a Single (Full) Package Index:* Define a single base URL that
712 identifies where a full package database is located::
713
714 [oe-packages]
715 baseurl=http://my.server/rpm
716
717 This example informs DNF about a single
718 package database that contains all the package index information for
719 all supported architectures.
720
721Once you have informed DNF where to find the package databases, you need
722to fetch them:
723
724.. code-block:: none
725
726 # dnf makecache
727
728DNF is now able to find, install, and
729upgrade packages from the specified repository or repositories.
730
731.. note::
732
733 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
734 additional information.
735
736Using IPK
737~~~~~~~~~
738
739The ``opkg`` application performs runtime package management of IPK
740packages. You must perform an initial setup for ``opkg`` on the target
741machine if the
742:term:`PACKAGE_FEED_ARCHS`,
743:term:`PACKAGE_FEED_BASE_PATHS`,
744and
745:term:`PACKAGE_FEED_URIS`
746variables have not been set or the target image was built before the
747variables were set.
748
749The ``opkg`` application uses configuration files to find available
750package databases. Thus, you need to create a configuration file inside
751the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
752you want to use.
753
754As an example, suppose you are serving packages from a ``ipk/``
755directory containing the ``i586``, ``all``, and ``qemux86`` databases
756through an HTTP server named ``my.server``. On the target, create a
757configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
758directory containing the following:
759
760.. code-block:: none
761
762 src/gz all http://my.server/ipk/all
763 src/gz i586 http://my.server/ipk/i586
764 src/gz qemux86 http://my.server/ipk/qemux86
765
766Next, instruct ``opkg`` to fetch the
767repository information:
768
769.. code-block:: none
770
771 # opkg update
772
773The ``opkg`` application is now able to find, install, and upgrade packages
774from the specified repository.
775
776Using DEB
777~~~~~~~~~
778
779The ``apt`` application performs runtime package management of DEB
780packages. This application uses a source list file to find available
781package databases. You must perform an initial setup for ``apt`` on the
782target machine if the
783:term:`PACKAGE_FEED_ARCHS`,
784:term:`PACKAGE_FEED_BASE_PATHS`,
785and
786:term:`PACKAGE_FEED_URIS`
787variables have not been set or the target image was built before the
788variables were set.
789
790To inform ``apt`` of the repository you want to use, you might create a
791list file (e.g. ``my_repo.list``) inside the
792``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
793serving packages from a ``deb/`` directory containing the ``i586``,
794``all``, and ``qemux86`` databases through an HTTP server named
795``my.server``. The list file should contain:
796
797.. code-block:: none
798
799 deb http://my.server/deb/all ./
800 deb http://my.server/deb/i586 ./
801 deb http://my.server/deb/qemux86 ./
802
803Next, instruct the ``apt`` application
804to fetch the repository information:
805
806.. code-block:: none
807
808 $ sudo apt update
809
810After this step,
811``apt`` is able to find, install, and upgrade packages from the
812specified repository.
813
814Generating and Using Signed Packages
815====================================
816
817In order to add security to RPM packages used during a build, you can
818take steps to securely sign them. Once a signature is verified, the
819OpenEmbedded build system can use the package in the build. If security
820fails for a signed package, the build system stops the build.
821
822This section describes how to sign RPM packages during a build and how
823to use signed package feeds (repositories) when doing a build.
824
825Signing RPM Packages
826--------------------
827
828To enable signing RPM packages, you must set up the following
829configurations in either your ``local.config`` or ``distro.config``
830file::
831
832 # Inherit sign_rpm.bbclass to enable signing functionality
833 INHERIT += " sign_rpm"
834 # Define the GPG key that will be used for signing.
835 RPM_GPG_NAME = "key_name"
836 # Provide passphrase for the key
837 RPM_GPG_PASSPHRASE = "passphrase"
838
839.. note::
840
841 Be sure to supply appropriate values for both `key_name` and
842 `passphrase`.
843
844Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
845the previous example, two optional variables related to signing are available:
846
847- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
848 when the package is signed.
849
850- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
851 package is signed.
852
853Processing Package Feeds
854------------------------
855
856In addition to being able to sign RPM packages, you can also enable
857signed package feeds for IPK and RPM packages.
858
859The steps you need to take to enable signed package feed use are similar
860to the steps used to sign RPM packages. You must define the following in
861your ``local.config`` or ``distro.config`` file::
862
863 INHERIT += "sign_package_feed"
864 PACKAGE_FEED_GPG_NAME = "key_name"
865 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
866
867For signed package feeds, the passphrase must be specified in a separate file,
868which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
869variable. Regarding security, keeping a plain text passphrase out of the
870configuration is more secure.
871
872Aside from the ``PACKAGE_FEED_GPG_NAME`` and
873``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
874related to signed package feeds are available:
875
876- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
877 when the package is signed.
878
879- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
880 package is signed.
881
882- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
883 signature. This variable applies only to RPM and IPK package feeds.
884 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
885 "ASC", which is the default and specifies ascii armored, and "BIN",
886 which specifies binary.
887
888Testing Packages With ptest
889===========================
890
891A Package Test (ptest) runs tests against packages built by the
892OpenEmbedded build system on the target machine. A ptest contains at
893least two items: the actual test, and a shell script (``run-ptest``)
894that starts the test. The shell script that starts the test must not
895contain the actual test --- the script only starts the test. On the other
896hand, the test can be anything from a simple shell script that runs a
897binary and checks the output to an elaborate system of test binaries and
898data files.
899
900The test generates output in the format used by Automake::
901
902 result: testname
903
904where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
905the testname can be any identifying string.
906
907For a list of Yocto Project recipes that are already enabled with ptest,
908see the :yocto_wiki:`Ptest </Ptest>` wiki page.
909
910.. note::
911
912 A recipe is "ptest-enabled" if it inherits the
913 :ref:`ptest <ref-classes-ptest>` class.
914
915Adding ptest to Your Build
916--------------------------
917
918To add package testing to your build, add the :term:`DISTRO_FEATURES` and
919:term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which
920is found in the :term:`Build Directory`::
921
922 DISTRO_FEATURES:append = " ptest"
923 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
924
925Once your build is complete, the ptest files are installed into the
926``/usr/lib/package/ptest`` directory within the image, where ``package``
927is the name of the package.
928
929Running ptest
930-------------
931
932The ``ptest-runner`` package installs a shell script that loops through
933all installed ptest test suites and runs them in sequence. Consequently,
934you might want to add this package to your image.
935
936Getting Your Package Ready
937--------------------------
938
939In order to enable a recipe to run installed ptests on target hardware,
940you need to prepare the recipes that build the packages you want to
941test. Here is what you have to do for each recipe:
942
943- *Be sure the recipe inherits the* :ref:`ptest <ref-classes-ptest>` *class:*
944 Include the following line in each recipe::
945
946 inherit ptest
947
948- *Create run-ptest:* This script starts your test. Locate the
949 script where you will refer to it using
950 :term:`SRC_URI`. Here is an
951 example that starts a test for ``dbus``::
952
953 #!/bin/sh
954 cd test
955 make -k runtest-TESTS
956
957- *Ensure dependencies are met:* If the test adds build or runtime
958 dependencies that normally do not exist for the package (such as
959 requiring "make" to run the test suite), use the
960 :term:`DEPENDS` and
961 :term:`RDEPENDS` variables in
962 your recipe in order for the package to meet the dependencies. Here
963 is an example where the package has a runtime dependency on "make"::
964
965 RDEPENDS:${PN}-ptest += "make"
966
967- *Add a function to build the test suite:* Not many packages support
968 cross-compilation of their test suites. Consequently, you usually
969 need to add a cross-compilation function to the package.
970
971 Many packages based on Automake compile and run the test suite by
972 using a single command such as ``make check``. However, the host
973 ``make check`` builds and runs on the same computer, while
974 cross-compiling requires that the package is built on the host but
975 executed for the target architecture (though often, as in the case
976 for ptest, the execution occurs on the host). The built version of
977 Automake that ships with the Yocto Project includes a patch that
978 separates building and execution. Consequently, packages that use the
979 unaltered, patched version of ``make check`` automatically
980 cross-compiles.
981
982 Regardless, you still must add a ``do_compile_ptest`` function to
983 build the test suite. Add a function similar to the following to your
984 recipe::
985
986 do_compile_ptest() {
987 oe_runmake buildtest-TESTS
988 }
989
990- *Ensure special configurations are set:* If the package requires
991 special configurations prior to compiling the test code, you must
992 insert a ``do_configure_ptest`` function into the recipe.
993
994- *Install the test suite:* The :ref:`ptest <ref-classes-ptest>` class
995 automatically copies the file ``run-ptest`` to the target and then runs make
996 ``install-ptest`` to run the tests. If this is not enough, you need
997 to create a ``do_install_ptest`` function and make sure it gets
998 called after the "make install-ptest" completes.
999
1000Creating Node Package Manager (NPM) Packages
1001============================================
1002
1003:wikipedia:`NPM <Npm_(software)>` is a package
1004manager for the JavaScript programming language. The Yocto Project
1005supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
1006use this fetcher in combination with
1007:doc:`devtool </ref-manual/devtool-reference>` to create
1008recipes that produce NPM packages.
1009
1010There are two workflows that allow you to create NPM packages using
1011``devtool``: the NPM registry modules method and the NPM project code
1012method.
1013
1014.. note::
1015
1016 While it is possible to create NPM recipes manually, using
1017 ``devtool`` is far simpler.
1018
1019Additionally, some requirements and caveats exist.
1020
1021Requirements and Caveats
1022------------------------
1023
1024You need to be aware of the following before using ``devtool`` to create
1025NPM packages:
1026
1027- Of the two methods that you can use ``devtool`` to create NPM
1028 packages, the registry approach is slightly simpler. However, you
1029 might consider the project approach because you do not have to
1030 publish your module in the `NPM registry <https://docs.npmjs.com/misc/registry>`__,
1031 which is NPM's public registry.
1032
1033- Be familiar with
1034 :doc:`devtool </ref-manual/devtool-reference>`.
1035
1036- The NPM host tools need the native ``nodejs-npm`` package, which is
1037 part of the OpenEmbedded environment. You need to get the package by
1038 cloning the :oe_git:`meta-openembedded </meta-openembedded>`
1039 repository. Be sure to add the path to your local copy
1040 to your ``bblayers.conf`` file.
1041
1042- ``devtool`` cannot detect native libraries in module dependencies.
1043 Consequently, you must manually add packages to your recipe.
1044
1045- While deploying NPM packages, ``devtool`` cannot determine which
1046 dependent packages are missing on the target (e.g. the node runtime
1047 ``nodejs``). Consequently, you need to find out what files are
1048 missing and be sure they are on the target.
1049
1050- Although you might not need NPM to run your node package, it is
1051 useful to have NPM on your target. The NPM package name is
1052 ``nodejs-npm``.
1053
1054Using the Registry Modules Method
1055---------------------------------
1056
1057This section presents an example that uses the ``cute-files`` module,
1058which is a file browser web application.
1059
1060.. note::
1061
1062 You must know the ``cute-files`` module version.
1063
1064The first thing you need to do is use ``devtool`` and the NPM fetcher to
1065create the recipe::
1066
1067 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
1068
1069The
1070``devtool add`` command runs ``recipetool create`` and uses the same
1071fetch URI to download each dependency and capture license details where
1072possible. The result is a generated recipe.
1073
1074After running for quite a long time, in particular building the
1075``nodejs-native`` package, the command should end as follows::
1076
1077 INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional
1078
1079The recipe file is fairly simple and contains every license that
1080``recipetool`` finds and includes the licenses in the recipe's
1081:term:`LIC_FILES_CHKSUM`
1082variables. You need to examine the variables and look for those with
1083"unknown" in the :term:`LICENSE`
1084field. You need to track down the license information for "unknown"
1085modules and manually add the information to the recipe.
1086
1087``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
1088files capture the version of all dependent modules. Many packages do not
1089provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it
1090runs.
1091
1092.. note::
1093
1094 A package is created for each sub-module. This policy is the only
1095 practical way to have the licenses for all of the dependencies
1096 represented in the license manifest of the image.
1097
1098The ``devtool edit-recipe`` command lets you take a look at the recipe::
1099
1100 $ devtool edit-recipe cute-files
1101 # Recipe created by recipetool
1102 # This is the basis of a recipe and may need further editing in order to be fully functional.
1103 # (Feel free to remove these comments when editing.)
1104
1105 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
1106 # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
1107 # your responsibility to verify that the values are complete and correct.
1108 #
1109 # NOTE: multiple licenses have been detected; they have been separated with &
1110 # in the LICENSE value for now since it is a reasonable assumption that all
1111 # of the licenses apply. If instead there is a choice between the multiple
1112 # licenses then you should change the value to separate the licenses with |
1113 # instead of &. If there is any doubt, check the accompanying documentation
1114 # to determine which situation is applicable.
1115
1116 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
1117 LICENSE = "BSD-3-Clause & ISC & MIT"
1118 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
1119 file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \
1120 file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \
1121 ...
1122 file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a"
1123
1124 SRC_URI = " \
1125 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
1126 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
1127 "
1128
1129 S = "${WORKDIR}/npm"
1130
1131 inherit npm
1132
1133 LICENSE:${PN} = "MIT"
1134 LICENSE:${PN}-accepts = "MIT"
1135 LICENSE:${PN}-array-flatten = "MIT"
1136 ...
1137 LICENSE:${PN}-vary = "MIT"
1138
1139Here are three key points in the previous example:
1140
1141- :term:`SRC_URI` uses the NPM
1142 scheme so that the NPM fetcher is used.
1143
1144- ``recipetool`` collects all the license information. If a
1145 sub-module's license is unavailable, the sub-module's name appears in
1146 the comments.
1147
1148- The ``inherit npm`` statement causes the
1149 :ref:`npm <ref-classes-npm>` class to package
1150 up all the modules.
1151
1152You can run the following command to build the ``cute-files`` package::
1153
1154 $ devtool build cute-files
1155
1156Remember that ``nodejs`` must be installed on
1157the target before your package.
1158
1159Assuming 192.168.7.2 for the target's IP address, use the following
1160command to deploy your package::
1161
1162 $ devtool deploy-target -s cute-files root@192.168.7.2
1163
1164Once the package is installed on the target, you can
1165test the application to show the contents of any directory::
1166
1167 $ cd /usr/lib/node_modules/cute-files
1168 $ cute-files
1169
1170On a browser,
1171go to ``http://192.168.7.2:3000`` and you see the following:
1172
1173.. image:: figures/cute-files-npm-example.png
1174 :width: 100%
1175
1176You can find the recipe in ``workspace/recipes/cute-files``. You can use
1177the recipe in any layer you choose.
1178
1179Using the NPM Projects Code Method
1180----------------------------------
1181
1182Although it is useful to package modules already in the NPM registry,
1183adding ``node.js`` projects under development is a more common developer
1184use case.
1185
1186This section covers the NPM projects code method, which is very similar
1187to the "registry" approach described in the previous section. In the NPM
1188projects method, you provide ``devtool`` with an URL that points to the
1189source files.
1190
1191Replicating the same example, (i.e. ``cute-files``) use the following
1192command::
1193
1194 $ devtool add https://github.com/martinaglv/cute-files.git
1195
1196The recipe this command generates is very similar to the recipe created in
1197the previous section. However, the :term:`SRC_URI` looks like the following::
1198
1199 SRC_URI = " \
1200 git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \
1201 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
1202 "
1203
1204In this example,
1205the main module is taken from the Git repository and dependencies are
1206taken from the NPM registry. Other than those differences, the recipe is
1207basically the same between the two methods. You can build and deploy the
1208package exactly as described in the previous section that uses the
1209registry modules method.
1210
1211Adding custom metadata to packages
1212==================================
1213
1214The variable
1215:term:`PACKAGE_ADD_METADATA`
1216can be used to add additional metadata to packages. This is reflected in
1217the package control/spec file. To take the ipk format for example, the
1218CONTROL file stored inside would contain the additional metadata as
1219additional lines.
1220
1221The variable can be used in multiple ways, including using suffixes to
1222set it for a specific package type and/or package. Note that the order
1223of precedence is the same as this list:
1224
1225- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
1226
1227- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
1228
1229- ``PACKAGE_ADD_METADATA:<PN>``
1230
1231- :term:`PACKAGE_ADD_METADATA`
1232
1233`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
1234package type:
1235
1236- IPK for .ipk packages
1237
1238- DEB for .deb packages
1239
1240- RPM for .rpm packages
1241
1242`<PN>` is a parameter and expected to be a package name.
1243
1244The variable can contain multiple [one-line] metadata fields separated
1245by the literal sequence '\\n'. The separator can be redefined using the
1246variable flag ``separator``.
1247
1248Here is an example that adds two custom fields for ipk
1249packages::
1250
1251 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
1252
diff --git a/documentation/dev-manual/prebuilt-libraries.rst b/documentation/dev-manual/prebuilt-libraries.rst
new file mode 100644
index 0000000000..ca43463820
--- /dev/null
+++ b/documentation/dev-manual/prebuilt-libraries.rst
@@ -0,0 +1,209 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working with Pre-Built Libraries
4********************************
5
6Introduction
7============
8
9Some library vendors do not release source code for their software but do
10release pre-built binaries. When shared libraries are built, they should
11be versioned (see `this article
12<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__
13for some background), but sometimes this is not done.
14
15To summarize, a versioned library must meet two conditions:
16
17#. The filename must have the version appended, for example: ``libfoo.so.1.2.3``.
18#. The library must have the ELF tag ``SONAME`` set to the major version
19 of the library, for example: ``libfoo.so.1``. You can check this by
20 running ``readelf -d filename | grep SONAME``.
21
22This section shows how to deal with both versioned and unversioned
23pre-built libraries.
24
25Versioned Libraries
26===================
27
28In this example we work with pre-built libraries for the FT4222H USB I/O chip.
29Libraries are built for several target architecture variants and packaged in
30an archive as follows::
31
32 ├── build-arm-hisiv300
33 │   └── libft4222.so.1.4.4.44
34 ├── build-arm-v5-sf
35 │   └── libft4222.so.1.4.4.44
36 ├── build-arm-v6-hf
37 │   └── libft4222.so.1.4.4.44
38 ├── build-arm-v7-hf
39 │   └── libft4222.so.1.4.4.44
40 ├── build-arm-v8
41 │   └── libft4222.so.1.4.4.44
42 ├── build-i386
43 │   └── libft4222.so.1.4.4.44
44 ├── build-i486
45 │   └── libft4222.so.1.4.4.44
46 ├── build-mips-eglibc-hf
47 │   └── libft4222.so.1.4.4.44
48 ├── build-pentium
49 │   └── libft4222.so.1.4.4.44
50 ├── build-x86_64
51 │   └── libft4222.so.1.4.4.44
52 ├── examples
53 │   ├── get-version.c
54 │   ├── i2cm.c
55 │   ├── spim.c
56 │   └── spis.c
57 ├── ftd2xx.h
58 ├── install4222.sh
59 ├── libft4222.h
60 ├── ReadMe.txt
61 └── WinTypes.h
62
63To write a recipe to use such a library in your system:
64
65- The vendor will probably have a proprietary licence, so set
66 :term:`LICENSE_FLAGS` in your recipe.
67- The vendor provides a tarball containing libraries so set :term:`SRC_URI`
68 appropriately.
69- Set :term:`COMPATIBLE_HOST` so that the recipe cannot be used with an
70 unsupported architecture. In the following example, we only support the 32
71 and 64 bit variants of the ``x86`` architecture.
72- As the vendor provides versioned libraries, we can use ``oe_soinstall``
73 from :ref:`ref-classes-utils` to install the shared library and create
74 symbolic links. If the vendor does not do this, we need to follow the
75 non-versioned library guidelines in the next section.
76- As the vendor likely used :term:`LDFLAGS` different from those in your Yocto
77 Project build, disable the corresponding checks by adding ``ldflags``
78 to :term:`INSANE_SKIP`.
79- The vendor will typically ship release builds without debugging symbols.
80 Avoid errors by preventing the packaging task from stripping out the symbols
81 and adding them to a separate debug package. This is done by setting the
82 ``INHIBIT_`` flags shown below.
83
84The complete recipe would look like this::
85
86 SUMMARY = "FTDI FT4222H Library"
87 SECTION = "libs"
88 LICENSE_FLAGS = "ftdi"
89 LICENSE = "CLOSED"
90
91 COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
92
93 # Sources available in a .tgz file in .zip archive
94 # at https://ftdichip.com/wp-content/uploads/2021/01/libft4222-linux-1.4.4.44.zip
95 # Found on https://ftdichip.com/software-examples/ft4222h-software-examples/
96 # Since dealing with this particular type of archive is out of topic here,
97 # we use a local link.
98 SRC_URI = "file://libft4222-linux-${PV}.tgz"
99
100 S = "${WORKDIR}"
101
102 ARCH_DIR:x86-64 = "build-x86_64"
103 ARCH_DIR:i586 = "build-i386"
104 ARCH_DIR:i686 = "build-i386"
105
106 INSANE_SKIP:${PN} = "ldflags"
107 INHIBIT_PACKAGE_STRIP = "1"
108 INHIBIT_SYSROOT_STRIP = "1"
109 INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
110
111 do_install () {
112 install -m 0755 -d ${D}${libdir}
113 oe_soinstall ${S}/${ARCH_DIR}/libft4222.so.${PV} ${D}${libdir}
114 install -d ${D}${includedir}
115 install -m 0755 ${S}/*.h ${D}${includedir}
116 }
117
118If the precompiled binaries are not statically linked and have dependencies on
119other libraries, then by adding those libraries to :term:`DEPENDS`, the linking
120can be examined and the appropriate :term:`RDEPENDS` automatically added.
121
122Non-Versioned Libraries
123=======================
124
125Some Background
126---------------
127
128Libraries in Linux systems are generally versioned so that it is possible
129to have multiple versions of the same library installed, which eases upgrades
130and support for older software. For example, suppose that in a versioned
131library, an actual library is called ``libfoo.so.1.2``, a symbolic link named
132``libfoo.so.1`` points to ``libfoo.so.1.2``, and a symbolic link named
133``libfoo.so`` points to ``libfoo.so.1.2``. Given these conditions, when you
134link a binary against a library, you typically provide the unversioned file
135name (i.e. ``-lfoo`` to the linker). However, the linker follows the symbolic
136link and actually links against the versioned filename. The unversioned symbolic
137link is only used at development time. Consequently, the library is packaged
138along with the headers in the development package ``${PN}-dev`` along with the
139actual library and versioned symbolic links in ``${PN}``. Because versioned
140libraries are far more common than unversioned libraries, the default packaging
141rules assume versioned libraries.
142
143Yocto Library Packaging Overview
144--------------------------------
145
146It follows that packaging an unversioned library requires a bit of work in the
147recipe. By default, ``libfoo.so`` gets packaged into ``${PN}-dev``, which
148triggers a QA warning that a non-symlink library is in a ``-dev`` package,
149and binaries in the same recipe link to the library in ``${PN}-dev``,
150which triggers more QA warnings. To solve this problem, you need to package the
151unversioned library into ``${PN}`` where it belongs. The following are the abridged
152default :term:`FILES` variables in ``bitbake.conf``::
153
154 SOLIBS = ".so.*"
155 SOLIBSDEV = ".so"
156 FILES_${PN} = "... ${libdir}/lib*${SOLIBS} ..."
157 FILES_SOLIBSDEV ?= "... ${libdir}/lib*${SOLIBSDEV} ..."
158 FILES_${PN}-dev = "... ${FILES_SOLIBSDEV} ..."
159
160:term:`SOLIBS` defines a pattern that matches real shared object libraries.
161:term:`SOLIBSDEV` matches the development form (unversioned symlink). These two
162variables are then used in ``FILES:${PN}`` and ``FILES:${PN}-dev``, which puts
163the real libraries into ``${PN}`` and the unversioned symbolic link into ``${PN}-dev``.
164To package unversioned libraries, you need to modify the variables in the recipe
165as follows::
166
167 SOLIBS = ".so"
168 FILES_SOLIBSDEV = ""
169
170The modifications cause the ``.so`` file to be the real library
171and unset :term:`FILES_SOLIBSDEV` so that no libraries get packaged into
172``${PN}-dev``. The changes are required because unless :term:`PACKAGES` is changed,
173``${PN}-dev`` collects files before `${PN}`. ``${PN}-dev`` must not collect any of
174the files you want in ``${PN}``.
175
176Finally, loadable modules, essentially unversioned libraries that are linked
177at runtime using ``dlopen()`` instead of at build time, should generally be
178installed in a private directory. However, if they are installed in ``${libdir}``,
179then the modules can be treated as unversioned libraries.
180
181Example
182-------
183
184The example below installs an unversioned x86-64 pre-built library named
185``libfoo.so``. The :term:`COMPATIBLE_HOST` variable limits recipes to the
186x86-64 architecture while the :term:`INSANE_SKIP`, :term:`INHIBIT_PACKAGE_STRIP`
187and :term:`INHIBIT_SYSROOT_STRIP` variables are all set as in the above
188versioned library example. The "magic" is setting the :term:`SOLIBS` and
189:term:`FILES_SOLIBSDEV` variables as explained above::
190
191 SUMMARY = "libfoo sample recipe"
192 SECTION = "libs"
193 LICENSE = "CLOSED"
194
195 SRC_URI = "file://libfoo.so"
196
197 COMPATIBLE_HOST = "x86_64.*-linux"
198
199 INSANE_SKIP:${PN} = "ldflags"
200 INHIBIT_PACKAGE_STRIP = "1"
201 INHIBIT_SYSROOT_STRIP = "1"
202 SOLIBS = ".so"
203 FILES_SOLIBSDEV = ""
204
205 do_install () {
206 install -d ${D}${libdir}
207 install -m 0755 ${WORKDIR}/libfoo.so ${D}${libdir}
208 }
209
diff --git a/documentation/dev-manual/python-development-shell.rst b/documentation/dev-manual/python-development-shell.rst
new file mode 100644
index 0000000000..1b213318a5
--- /dev/null
+++ b/documentation/dev-manual/python-development-shell.rst
@@ -0,0 +1,50 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using a Python Development Shell
4********************************
5
6Similar to working within a development shell as described in the
7previous section, you can also spawn and work within an interactive
8Python development shell. When debugging certain commands or even when
9just editing packages, ``pydevshell`` can be a useful tool. When you
10invoke the ``pydevshell`` task, all tasks up to and including
11:ref:`ref-tasks-patch` are run for the
12specified target. Then a new terminal is opened. Additionally, key
13Python objects and code are available in the same way they are to
14BitBake tasks, in particular, the data store 'd'. So, commands such as
15the following are useful when exploring the data store and running
16functions::
17
18 pydevshell> d.getVar("STAGING_DIR")
19 '/media/build1/poky/build/tmp/sysroots'
20 pydevshell> d.getVar("STAGING_DIR", False)
21 '${TMPDIR}/sysroots'
22 pydevshell> d.setVar("FOO", "bar")
23 pydevshell> d.getVar("FOO")
24 'bar'
25 pydevshell> d.delVar("FOO")
26 pydevshell> d.getVar("FOO")
27 pydevshell> bb.build.exec_func("do_unpack", d)
28 pydevshell>
29
30See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:functions you can call from within python`"
31section in the BitBake User Manual for details about available functions.
32
33The commands execute just as if the OpenEmbedded build
34system were executing them. Consequently, working this way can be
35helpful when debugging a build or preparing software to be used with the
36OpenEmbedded build system.
37
38Following is an example that uses ``pydevshell`` on a target named
39``matchbox-desktop``::
40
41 $ bitbake matchbox-desktop -c pydevshell
42
43This command spawns a terminal and places you in an interactive Python
44interpreter within the OpenEmbedded build environment. The
45:term:`OE_TERMINAL` variable
46controls what type of shell is opened.
47
48When you are finished using ``pydevshell``, you can exit the shell
49either by using Ctrl+d or closing the terminal window.
50
diff --git a/documentation/dev-manual/quilt.rst b/documentation/dev-manual/quilt.rst
new file mode 100644
index 0000000000..d1b605bcd5
--- /dev/null
+++ b/documentation/dev-manual/quilt.rst
@@ -0,0 +1,90 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using Quilt in Your Workflow
4****************************
5
6`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
7that allows you to capture source code changes without having a clean
8source tree. This section outlines the typical workflow you can use to
9modify source code, test changes, and then preserve the changes in the
10form of a patch all using Quilt.
11
12.. note::
13
14 With regard to preserving changes to source files, if you clean a
15 recipe or have :ref:`rm_work <ref-classes-rm-work>` enabled, the
16 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
17 as described in the Yocto Project Application Development and the
18 Extensible Software Development Kit (eSDK) manual is a safer
19 development flow than the flow that uses Quilt.
20
21Follow these general steps:
22
231. *Find the Source Code:* Temporary source code used by the
24 OpenEmbedded build system is kept in the :term:`Build Directory`. See the
25 ":ref:`dev-manual/temporary-source-code:finding temporary source code`" section to
26 learn how to locate the directory that has the temporary source code for a
27 particular package.
28
292. *Change Your Working Directory:* You need to be in the directory that
30 has the temporary source code. That directory is defined by the
31 :term:`S` variable.
32
333. *Create a New Patch:* Before modifying source code, you need to
34 create a new patch. To create a new patch file, use ``quilt new`` as
35 below::
36
37 $ quilt new my_changes.patch
38
394. *Notify Quilt and Add Files:* After creating the patch, you need to
40 notify Quilt about the files you plan to edit. You notify Quilt by
41 adding the files to the patch you just created::
42
43 $ quilt add file1.c file2.c file3.c
44
455. *Edit the Files:* Make your changes in the source code to the files
46 you added to the patch.
47
486. *Test Your Changes:* Once you have modified the source code, the
49 easiest way to test your changes is by calling the :ref:`ref-tasks-compile`
50 task as shown in the following example::
51
52 $ bitbake -c compile -f package
53
54 The ``-f`` or ``--force`` option forces the specified task to
55 execute. If you find problems with your code, you can just keep
56 editing and re-testing iteratively until things work as expected.
57
58 .. note::
59
60 All the modifications you make to the temporary source code disappear
61 once you run the :ref:`ref-tasks-clean` or :ref:`ref-tasks-cleanall`
62 tasks using BitBake (i.e. ``bitbake -c clean package`` and
63 ``bitbake -c cleanall package``). Modifications will also disappear if
64 you use the :ref:`rm_work <ref-classes-rm-work>` feature as described in
65 the ":ref:`dev-manual/disk-space:conserving disk space during builds`"
66 section.
67
687. *Generate the Patch:* Once your changes work as expected, you need to
69 use Quilt to generate the final patch that contains all your
70 modifications.
71 ::
72
73 $ quilt refresh
74
75 At this point, the
76 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
77 ``file2.c``, and ``file3.c`` files.
78
79 You can find the resulting patch file in the ``patches/``
80 subdirectory of the source (:term:`S`) directory.
81
828. *Copy the Patch File:* For simplicity, copy the patch file into a
83 directory named ``files``, which you can create in the same directory
84 that holds the recipe (``.bb``) file or the append (``.bbappend``)
85 file. Placing the patch here guarantees that the OpenEmbedded build
86 system will find the patch. Next, add the patch into the :term:`SRC_URI`
87 of the recipe. Here is an example::
88
89 SRC_URI += "file://my_changes.patch"
90
diff --git a/documentation/dev-manual/read-only-rootfs.rst b/documentation/dev-manual/read-only-rootfs.rst
new file mode 100644
index 0000000000..e29659c678
--- /dev/null
+++ b/documentation/dev-manual/read-only-rootfs.rst
@@ -0,0 +1,89 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Read-Only Root Filesystem
4************************************
5
6Suppose, for security reasons, you need to disable your target device's
7root filesystem's write permissions (i.e. you need a read-only root
8filesystem). Or, perhaps you are running the device's operating system
9from a read-only storage device. For either case, you can customize your
10image for that behavior.
11
12.. note::
13
14 Supporting a read-only root filesystem requires that the system and
15 applications do not try to write to the root filesystem. You must
16 configure all parts of the target system to write elsewhere, or to
17 gracefully fail in the event of attempting to write to the root
18 filesystem.
19
20Creating the Root Filesystem
21============================
22
23To create the read-only root filesystem, simply add the
24"read-only-rootfs" feature to your image, normally in one of two ways.
25The first way is to add the "read-only-rootfs" image feature in the
26image's recipe file via the :term:`IMAGE_FEATURES` variable::
27
28 IMAGE_FEATURES += "read-only-rootfs"
29
30As an alternative, you can add the same feature
31from within your :term:`Build Directory`'s ``local.conf`` file with the
32associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
33
34 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
35
36For more information on how to use these variables, see the
37":ref:`dev-manual/customizing-images:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
38section. For information on the variables, see
39:term:`IMAGE_FEATURES` and
40:term:`EXTRA_IMAGE_FEATURES`.
41
42Post-Installation Scripts and Read-Only Root Filesystem
43=======================================================
44
45It is very important that you make sure all post-Installation
46(``pkg_postinst``) scripts for packages that are installed into the
47image can be run at the time when the root filesystem is created during
48the build on the host system. These scripts cannot attempt to run during
49the first boot on the target device. With the "read-only-rootfs" feature
50enabled, the build system makes sure that all post-installation scripts
51succeed at file system creation time. If any of these scripts
52still need to be run after the root filesystem is created, the build
53immediately fails. These build-time checks ensure that the build fails
54rather than the target device fails later during its initial boot
55operation.
56
57Most of the common post-installation scripts generated by the build
58system for the out-of-the-box Yocto Project are engineered so that they
59can run during root filesystem creation (e.g. post-installation scripts
60for caching fonts). However, if you create and add custom scripts, you
61need to be sure they can be run during this file system creation.
62
63Here are some common problems that prevent post-installation scripts
64from running during root filesystem creation:
65
66- *Not using $D in front of absolute paths:* The build system defines
67 ``$``\ :term:`D` when the root
68 filesystem is created. Furthermore, ``$D`` is blank when the script
69 is run on the target device. This implies two purposes for ``$D``:
70 ensuring paths are valid in both the host and target environments,
71 and checking to determine which environment is being used as a method
72 for taking appropriate actions.
73
74- *Attempting to run processes that are specific to or dependent on the
75 target architecture:* You can work around these attempts by using
76 native tools, which run on the host system, to accomplish the same
77 tasks, or by alternatively running the processes under QEMU, which
78 has the ``qemu_run_binary`` function. For more information, see the
79 :ref:`qemu <ref-classes-qemu>` class.
80
81Areas With Write Access
82=======================
83
84With the "read-only-rootfs" feature enabled, any attempt by the target
85to write to the root filesystem at runtime fails. Consequently, you must
86make sure that you configure processes and applications that attempt
87these types of writes do so to directories with write access (e.g.
88``/tmp`` or ``/var/run``).
89
diff --git a/documentation/dev-manual/runtime-testing.rst b/documentation/dev-manual/runtime-testing.rst
new file mode 100644
index 0000000000..77e1e6545f
--- /dev/null
+++ b/documentation/dev-manual/runtime-testing.rst
@@ -0,0 +1,599 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Performing Automated Runtime Testing
4************************************
5
6The OpenEmbedded build system makes available a series of automated
7tests for images to verify runtime functionality. You can run these
8tests on either QEMU or actual target hardware. Tests are written in
9Python making use of the ``unittest`` module, and the majority of them
10run commands on the target system over SSH. This section describes how
11you set up the environment to use these tests, run available tests, and
12write and add your own tests.
13
14For information on the test and QA infrastructure available within the
15Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
16section in the Yocto Project Reference Manual.
17
18Enabling Tests
19==============
20
21Depending on whether you are planning to run tests using QEMU or on the
22hardware, you have to take different steps to enable the tests. See the
23following subsections for information on how to enable both types of
24tests.
25
26Enabling Runtime Tests on QEMU
27------------------------------
28
29In order to run tests, you need to do the following:
30
31- *Set up to avoid interaction with sudo for networking:* To
32 accomplish this, you must do one of the following:
33
34 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
35 commands or just for ``runqemu-ifup``. You must provide the full
36 path as that can change if you are using multiple clones of the
37 source repository.
38
39 .. note::
40
41 On some distributions, you also need to comment out "Defaults
42 requiretty" in ``/etc/sudoers``.
43
44 - Manually configure a tap interface for your system.
45
46 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
47 should generate a list of tap devices. This is the option
48 typically chosen for Autobuilder-type environments.
49
50 .. note::
51
52 - Be sure to use an absolute path when calling this script
53 with sudo.
54
55 - The package recipe ``qemu-helper-native`` is required to run
56 this script. Build the package using the following command::
57
58 $ bitbake qemu-helper-native
59
60- *Set the DISPLAY variable:* You need to set this variable so that
61 you have an X server available (e.g. start ``vncserver`` for a
62 headless machine).
63
64- *Be sure your host's firewall accepts incoming connections from
65 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
66 HTTP server on a random high number port, which is used to serve
67 files to the target. The DNF module serves
68 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
69 That means your host's firewall must accept incoming connections from
70 192.168.7.0/24, which is the default IP range used for tap devices by
71 ``runqemu``.
72
73- *Be sure your host has the correct packages installed:* Depending
74 your host's distribution, you need to have the following packages
75 installed:
76
77 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
78
79 - openSUSE: ``sysstat`` and ``iproute2``
80
81 - Fedora: ``sysstat`` and ``iproute``
82
83 - CentOS: ``sysstat`` and ``iproute``
84
85Once you start running the tests, the following happens:
86
871. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
88
892. The image is booted under QEMU using the standard ``runqemu`` script.
90
913. A default timeout of 500 seconds occurs to allow for the boot process
92 to reach the login prompt. You can change the timeout period by
93 setting
94 :term:`TEST_QEMUBOOT_TIMEOUT`
95 in the ``local.conf`` file.
96
974. Once the boot process is reached and the login prompt appears, the
98 tests run. The full boot log is written to
99 ``${WORKDIR}/testimage/qemu_boot_log``.
100
1015. Each test module loads in the order found in :term:`TEST_SUITES`. You can
102 find the full output of the commands run over SSH in
103 ``${WORKDIR}/testimgage/ssh_target_log``.
104
1056. If no failures occur, the task running the tests ends successfully.
106 You can find the output from the ``unittest`` in the task log at
107 ``${WORKDIR}/temp/log.do_testimage``.
108
109Enabling Runtime Tests on Hardware
110----------------------------------
111
112The OpenEmbedded build system can run tests on real hardware, and for
113certain devices it can also deploy the image to be tested onto the
114device beforehand.
115
116For automated deployment, a "controller image" is installed onto the
117hardware once as part of setup. Then, each time tests are to be run, the
118following occurs:
119
1201. The controller image is booted into and used to write the image to be
121 tested to a second partition.
122
1232. The device is then rebooted using an external script that you need to
124 provide.
125
1263. The device boots into the image to be tested.
127
128When running tests (independent of whether the image has been deployed
129automatically or not), the device is expected to be connected to a
130network on a pre-determined IP address. You can either use static IP
131addresses written into the image, or set the image to use DHCP and have
132your DHCP server on the test network assign a known IP address based on
133the MAC address of the device.
134
135In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
136appropriate value. For QEMU, you do not have to change anything, the
137default value is "qemu". For running tests on hardware, the following
138options are available:
139
140- *"simpleremote":* Choose "simpleremote" if you are going to run tests
141 on a target system that is already running the image to be tested and
142 is available on the network. You can use "simpleremote" in
143 conjunction with either real hardware or an image running within a
144 separately started QEMU or any other virtual machine manager.
145
146- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
147 an EFI-based machine with ``systemd-boot`` as bootloader and
148 ``core-image-testmaster`` (or something similar) is installed. Also,
149 your hardware under test must be in a DHCP-enabled network that gives
150 it the same IP address for each reboot.
151
152 If you choose "SystemdbootTarget", there are additional requirements
153 and considerations. See the
154 ":ref:`dev-manual/runtime-testing:selecting systemdboottarget`" section, which
155 follows, for more information.
156
157- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
158 images and running tests on the BeagleBone "Black" or original
159 "White" hardware. For information on how to use these tests, see the
160 comments at the top of the BeagleBoneTarget
161 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
162
163- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
164 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
165 For information on how to use these tests, see the comments at the
166 top of the EdgeRouterTarget
167 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
168
169- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
170 tests on any generic PC that boots using GRUB. For information on how
171 to use these tests, see the comments at the top of the GrubTarget
172 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
173
174- *"your-target":* Create your own custom target if you want to run
175 tests when you are deploying images and running tests on a custom
176 machine within your BSP layer. To do this, you need to add a Python
177 unit that defines the target class under ``lib/oeqa/controllers/``
178 within your layer. You must also provide an empty ``__init__.py``.
179 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
180
181Selecting SystemdbootTarget
182---------------------------
183
184If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
185not need any information in this section. You can skip down to the
186":ref:`dev-manual/runtime-testing:running tests`" section.
187
188If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
189perform a one-time setup of your controller image by doing the following:
190
1911. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
192
193 EFI_PROVIDER = "systemd-boot"
194
1952. *Build the controller image:* Build the ``core-image-testmaster`` image.
196 The ``core-image-testmaster`` recipe is provided as an example for a
197 "controller" image and you can customize the image recipe as you would
198 any other recipe.
199
200 Here are the image recipe requirements:
201
202 - Inherits ``core-image`` so that kernel modules are installed.
203
204 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
205 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
206
207 - Uses a custom :term:`Initramfs` image with a custom
208 installer. A normal image that you can install usually creates a
209 single root filesystem partition. This image uses another installer that
210 creates a specific partition layout. Not all Board Support
211 Packages (BSPs) can use an installer. For such cases, you need to
212 manually create the following partition layout on the target:
213
214 - First partition mounted under ``/boot``, labeled "boot".
215
216 - The main root filesystem partition where this image gets installed,
217 which is mounted under ``/``.
218
219 - Another partition labeled "testrootfs" where test images get
220 deployed.
221
2223. *Install image:* Install the image that you just built on the target
223 system.
224
225The final thing you need to do when setting :term:`TEST_TARGET` to
226"SystemdbootTarget" is to set up the test image:
227
2281. *Set up your local.conf file:* Make sure you have the following
229 statements in your ``local.conf`` file::
230
231 IMAGE_FSTYPES += "tar.gz"
232 INHERIT += "testimage"
233 TEST_TARGET = "SystemdbootTarget"
234 TEST_TARGET_IP = "192.168.2.3"
235
2362. *Build your test image:* Use BitBake to build the image::
237
238 $ bitbake core-image-sato
239
240Power Control
241-------------
242
243For most hardware targets other than "simpleremote", you can control
244power:
245
246- You can use :term:`TEST_POWERCONTROL_CMD` together with
247 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
248 and does power cycling. The test code passes one argument to that
249 command: off, on or cycle (off then on). Here is an example that
250 could appear in your ``local.conf`` file::
251
252 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
253
254 In this example, the expect
255 script does the following:
256
257 .. code-block:: shell
258
259 ssh test@10.11.12.1 "pyctl nuc1 arg"
260
261 It then runs a Python script that controls power for a label called
262 ``nuc1``.
263
264 .. note::
265
266 You need to customize :term:`TEST_POWERCONTROL_CMD` and
267 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
268 is that it accepts "on", "off", and "cycle" as the last argument.
269
270- When no command is defined, it connects to the device over SSH and
271 uses the classic reboot command to reboot the device. Classic reboot
272 is fine as long as the machine actually reboots (i.e. the SSH test
273 has not failed). It is useful for scenarios where you have a simple
274 setup, typically with a single board, and where some manual
275 interaction is okay from time to time.
276
277If you have no hardware to automatically perform power control but still
278wish to experiment with automated hardware testing, you can use the
279``dialog-power-control`` script that shows a dialog prompting you to perform
280the required power action. This script requires either KDialog or Zenity
281to be installed. To use this script, set the
282:term:`TEST_POWERCONTROL_CMD`
283variable as follows::
284
285 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
286
287Serial Console Connection
288-------------------------
289
290For test target classes requiring a serial console to interact with the
291bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
292you need to specify a command to use to connect to the serial console of
293the target machine by using the
294:term:`TEST_SERIALCONTROL_CMD`
295variable and optionally the
296:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
297variable.
298
299These cases could be a serial terminal program if the machine is
300connected to a local serial port, or a ``telnet`` or ``ssh`` command
301connecting to a remote console server. Regardless of the case, the
302command simply needs to connect to the serial console and forward that
303connection to standard input and output as any normal terminal program
304does. For example, to use the picocom terminal program on serial device
305``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
306
307 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
308
309For local
310devices where the serial port device disappears when the device reboots,
311an additional "serdevtry" wrapper script is provided. To use this
312wrapper, simply prefix the terminal command with
313``${COREBASE}/scripts/contrib/serdevtry``::
314
315 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
316
317Running Tests
318=============
319
320You can start the tests automatically or manually:
321
322- *Automatically running tests:* To run the tests automatically after the
323 OpenEmbedded build system successfully creates an image, first set the
324 :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the
325 :term:`Build Directory`::
326
327 TESTIMAGE_AUTO = "1"
328
329 Next, build your image. If the image successfully builds, the
330 tests run::
331
332 bitbake core-image-sato
333
334- *Manually running tests:* To manually run the tests, first globally
335 inherit the :ref:`testimage <ref-classes-testimage>` class
336 by editing your ``local.conf`` file::
337
338 INHERIT += "testimage"
339
340 Next, use BitBake to run the tests::
341
342 bitbake -c testimage image
343
344All test files reside in ``meta/lib/oeqa/runtime`` in the
345:term:`Source Directory`. A test name maps
346directly to a Python module. Each test module may contain a number of
347individual tests. Tests are usually grouped together by the area tested
348(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
349
350You can add tests to any layer provided you place them in the proper
351area and you extend :term:`BBPATH` in
352the ``local.conf`` file as normal. Be sure that tests reside in
353``layer/lib/oeqa/runtime``.
354
355.. note::
356
357 Be sure that module names do not collide with module names used in
358 the default set of test modules in ``meta/lib/oeqa/runtime``.
359
360You can change the set of tests run by appending or overriding
361:term:`TEST_SUITES` variable in
362``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
363for the image. Test modules named within :term:`TEST_SUITES` cannot be
364skipped even if a test is not suitable for an image (e.g. running the
365RPM tests on an image without ``rpm``). Appending "auto" to
366:term:`TEST_SUITES` causes the build system to try to run all tests that are
367suitable for the image (i.e. each test module may elect to skip itself).
368
369The order you list tests in :term:`TEST_SUITES` is important and influences
370test dependencies. Consequently, tests that depend on other tests should
371be added after the test on which they depend. For example, since the
372``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
373"ping" in the list. The test class provides no re-ordering or dependency
374handling.
375
376.. note::
377
378 Each module can have multiple classes with multiple test methods.
379 And, Python ``unittest`` rules apply.
380
381Here are some things to keep in mind when running tests:
382
383- The default tests for the image are defined as::
384
385 DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
386
387- Add your own test to the list of the by using the following::
388
389 TEST_SUITES:append = " mytest"
390
391- Run a specific list of tests as follows::
392
393 TEST_SUITES = "test1 test2 test3"
394
395 Remember, order is important. Be sure to place a test that is
396 dependent on another test later in the order.
397
398Exporting Tests
399===============
400
401You can export tests so that they can run independently of the build
402system. Exporting tests is required if you want to be able to hand the
403test execution off to a scheduler. You can only export tests that are
404defined in :term:`TEST_SUITES`.
405
406If your image is already built, make sure the following are set in your
407``local.conf`` file::
408
409 INHERIT += "testexport"
410 TEST_TARGET_IP = "IP-address-for-the-test-target"
411 TEST_SERVER_IP = "IP-address-for-the-test-server"
412
413You can then export the tests with the
414following BitBake command form::
415
416 $ bitbake image -c testexport
417
418Exporting the tests places them in the :term:`Build Directory` in
419``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
420variable.
421
422You can now run the tests outside of the build environment::
423
424 $ cd tmp/testexport/image
425 $ ./runexported.py testdata.json
426
427Here is a complete example that shows IP addresses and uses the
428``core-image-sato`` image::
429
430 INHERIT += "testexport"
431 TEST_TARGET_IP = "192.168.7.2"
432 TEST_SERVER_IP = "192.168.7.1"
433
434Use BitBake to export the tests::
435
436 $ bitbake core-image-sato -c testexport
437
438Run the tests outside of
439the build environment using the following::
440
441 $ cd tmp/testexport/core-image-sato
442 $ ./runexported.py testdata.json
443
444Writing New Tests
445=================
446
447As mentioned previously, all new test files need to be in the proper
448place for the build system to find them. New tests for additional
449functionality outside of the core should be added to the layer that adds
450the functionality, in ``layer/lib/oeqa/runtime`` (as long as
451:term:`BBPATH` is extended in the
452layer's ``layer.conf`` file as normal). Just remember the following:
453
454- Filenames need to map directly to test (module) names.
455
456- Do not use module names that collide with existing core tests.
457
458- Minimally, an empty ``__init__.py`` file must be present in the runtime
459 directory.
460
461To create a new test, start by copying an existing module (e.g.
462``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
463code from ``meta/lib/oeqa/utils``, which are helper classes.
464
465.. note::
466
467 Structure shell commands such that you rely on them and they return a
468 single code for success. Be aware that sometimes you will need to
469 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
470
471You will notice that all test classes inherit ``oeRuntimeTest``, which
472is found in ``meta/lib/oetest.py``. This base class offers some helper
473attributes, which are described in the following sections:
474
475Class Methods
476-------------
477
478Class methods are as follows:
479
480- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
481 package list of the image, which is based on the manifest file that
482 is generated during the :ref:`ref-tasks-rootfs` task.
483
484- *hasFeature(feature):* Returns "True" if the feature is in
485 :term:`IMAGE_FEATURES` or
486 :term:`DISTRO_FEATURES`.
487
488Class Attributes
489----------------
490
491Class attributes are as follows:
492
493- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
494 Otherwise, ``pscmd`` equals "ps" (busybox).
495
496- *tc:* The called test context, which gives access to the
497 following attributes:
498
499 - *d:* The BitBake datastore, which allows you to use stuff such
500 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
501
502 - *testslist and testsrequired:* Used internally. The tests
503 do not need these.
504
505 - *filesdir:* The absolute path to
506 ``meta/lib/oeqa/runtime/files``, which contains helper files for
507 tests meant for copying on the target such as small files written
508 in C for compilation.
509
510 - *target:* The target controller object used to deploy and
511 start an image on a particular target (e.g. Qemu, SimpleRemote,
512 and SystemdbootTarget). Tests usually use the following:
513
514 - *ip:* The target's IP address.
515
516 - *server_ip:* The host's IP address, which is usually used
517 by the DNF test suite.
518
519 - *run(cmd, timeout=None):* The single, most used method.
520 This command is a wrapper for: ``ssh root@host "cmd"``. The
521 command returns a tuple: (status, output), which are what their
522 names imply - the return code of "cmd" and whatever output it
523 produces. The optional timeout argument represents the number
524 of seconds the test should wait for "cmd" to return. If the
525 argument is "None", the test uses the default instance's
526 timeout period, which is 300 seconds. If the argument is "0",
527 the test runs until the command returns.
528
529 - *copy_to(localpath, remotepath):*
530 ``scp localpath root@ip:remotepath``.
531
532 - *copy_from(remotepath, localpath):*
533 ``scp root@host:remotepath localpath``.
534
535Instance Attributes
536-------------------
537
538There is a single instance attribute, which is ``target``. The ``target``
539instance attribute is identical to the class attribute of the same name,
540which is described in the previous section. This attribute exists as
541both an instance and class attribute so tests can use
542``self.target.run(cmd)`` in instance methods instead of
543``oeRuntimeTest.tc.target.run(cmd)``.
544
545Installing Packages in the DUT Without the Package Manager
546==========================================================
547
548When a test requires a package built by BitBake, it is possible to
549install that package. Installing the package does not require a package
550manager be installed in the device under test (DUT). It does, however,
551require an SSH connection and the target must be using the
552``sshcontrol`` class.
553
554.. note::
555
556 This method uses ``scp`` to copy files from the host to the target, which
557 causes permissions and special attributes to be lost.
558
559A JSON file is used to define the packages needed by a test. This file
560must be in the same path as the file used to define the tests.
561Furthermore, the filename must map directly to the test module name with
562a ``.json`` extension.
563
564The JSON file must include an object with the test name as keys of an
565object or an array. This object (or array of objects) uses the following
566data:
567
568- "pkg" --- a mandatory string that is the name of the package to be
569 installed.
570
571- "rm" --- an optional boolean, which defaults to "false", that specifies
572 to remove the package after the test.
573
574- "extract" --- an optional boolean, which defaults to "false", that
575 specifies if the package must be extracted from the package format.
576 When set to "true", the package is not automatically installed into
577 the DUT.
578
579Following is an example JSON file that handles test "foo" installing
580package "bar" and test "foobar" installing packages "foo" and "bar".
581Once the test is complete, the packages are removed from the DUT.
582::
583
584 {
585 "foo": {
586 "pkg": "bar"
587 },
588 "foobar": [
589 {
590 "pkg": "foo",
591 "rm": true
592 },
593 {
594 "pkg": "bar",
595 "rm": true
596 }
597 ]
598 }
599
diff --git a/documentation/dev-manual/sbom.rst b/documentation/dev-manual/sbom.rst
new file mode 100644
index 0000000000..f80e81279a
--- /dev/null
+++ b/documentation/dev-manual/sbom.rst
@@ -0,0 +1,68 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Software Bill of Materials
4*************************************
5
6Once you are able to build an image for your project, once the licenses for
7each software component are all identified (see
8":ref:`dev-manual/licenses:working with licenses`") and once vulnerability
9fixes are applied (see ":ref:`dev-manual/vulnerabilities:checking
10for vulnerabilities`"), the OpenEmbedded build system can generate
11a description of all the components you used, their licenses, their dependencies,
12the changes that were applied and the known vulnerabilities that were fixed.
13
14This description is generated in the form of a *Software Bill of Materials*
15(:term:`SBOM`), using the :term:`SPDX` standard.
16
17When you release software, this is the most standard way to provide information
18about the Software Supply Chain of your software image and SDK. The
19:term:`SBOM` tooling is often used to ensure open source license compliance by
20providing the license texts used in the product which legal departments and end
21users can read in standardized format.
22
23:term:`SBOM` information is also critical to performing vulnerability exposure
24assessments, as all the components used in the Software Supply Chain are listed.
25
26The OpenEmbedded build system doesn't generate such information by default.
27To make this happen, you must inherit the
28:ref:`create-spdx <ref-classes-create-spdx>` class from a configuration file::
29
30 INHERIT += "create-spdx"
31
32You then get :term:`SPDX` output in JSON format as an
33``IMAGE-MACHINE.spdx.json`` file in ``tmp/deploy/images/MACHINE/`` inside the
34:term:`Build Directory`.
35
36This is a toplevel file accompanied by an ``IMAGE-MACHINE.spdx.index.json``
37containing an index of JSON :term:`SPDX` files for individual recipes, together
38with an ``IMAGE-MACHINE.spdx.tar.zst`` compressed archive containing all such
39files.
40
41The :ref:`create-spdx <ref-classes-create-spdx>` class offers options to include
42more information in the output :term:`SPDX` data, such as making the generated
43files more human readable (:term:`SPDX_PRETTY`), adding compressed archives of
44the files in the generated target packages (:term:`SPDX_ARCHIVE_PACKAGED`),
45adding a description of the source files handled by the target recipes
46(:term:`SPDX_INCLUDE_SOURCES`) and adding archives of these source files
47themselves (:term:`SPDX_ARCHIVE_SOURCES`).
48
49Though the toplevel :term:`SPDX` output is available in
50``tmp/deploy/images/MACHINE/`` inside the :term:`Build Directory`, ancillary
51generated files are available in ``tmp/deploy/spdx/MACHINE`` too, such as:
52
53- The individual :term:`SPDX` JSON files in the ``IMAGE-MACHINE.spdx.tar.zst``
54 archive.
55
56- Compressed archives of the files in the generated target packages,
57 in ``packages/packagename.tar.zst`` (when :term:`SPDX_ARCHIVE_PACKAGED`
58 is set).
59
60- Compressed archives of the source files used to build the host tools
61 and the target packages in ``recipes/recipe-packagename.tar.zst``
62 (when :term:`SPDX_ARCHIVE_SOURCES` is set). Those are needed to fulfill
63 "source code access" license requirements.
64
65See the `tools page <https://spdx.dev/resources/tools/>`__ on the :term:`SPDX`
66project website for a list of tools to consume and transform the :term:`SPDX`
67data generated by the OpenEmbedded build system.
68
diff --git a/documentation/dev-manual/securing-images.rst b/documentation/dev-manual/securing-images.rst
new file mode 100644
index 0000000000..f8dd572104
--- /dev/null
+++ b/documentation/dev-manual/securing-images.rst
@@ -0,0 +1,158 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Making Images More Secure
4*************************
5
6Security is of increasing concern for embedded devices. Consider the
7issues and problems discussed in just this sampling of work found across
8the Internet:
9
10- *"*\ `Security Risks of Embedded
11 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
12 by Bruce Schneier
13
14- *"*\ `Internet Census
15 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
16 Botnet
17
18- *"*\ `Security Issues for Embedded
19 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
20 by Jake Edge
21
22When securing your image is of concern, there are steps, tools, and
23variables that you can consider to help you reach the security goals you
24need for your particular device. Not all situations are identical when
25it comes to making an image secure. Consequently, this section provides
26some guidance and suggestions for consideration when you want to make
27your image more secure.
28
29.. note::
30
31 Because the security requirements and risks are different for every
32 type of device, this section cannot provide a complete reference on
33 securing your custom OS. It is strongly recommended that you also
34 consult other sources of information on embedded Linux system
35 hardening and on security.
36
37General Considerations
38======================
39
40There are general considerations that help you create more secure images.
41You should consider the following suggestions to make your device
42more secure:
43
44- Scan additional code you are adding to the system (e.g. application
45 code) by using static analysis tools. Look for buffer overflows and
46 other potential security problems.
47
48- Pay particular attention to the security for any web-based
49 administration interface.
50
51 Web interfaces typically need to perform administrative functions and
52 tend to need to run with elevated privileges. Thus, the consequences
53 resulting from the interface's security becoming compromised can be
54 serious. Look for common web vulnerabilities such as
55 cross-site-scripting (XSS), unvalidated inputs, and so forth.
56
57 As with system passwords, the default credentials for accessing a
58 web-based interface should not be the same across all devices. This
59 is particularly true if the interface is enabled by default as it can
60 be assumed that many end-users will not change the credentials.
61
62- Ensure you can update the software on the device to mitigate
63 vulnerabilities discovered in the future. This consideration
64 especially applies when your device is network-enabled.
65
66- Regularly scan and apply fixes for CVE security issues affecting
67 all software components in the product, see ":ref:`dev-manual/vulnerabilities:checking for vulnerabilities`".
68
69- Regularly update your version of Poky and OE-Core from their upstream
70 developers, e.g. to apply updates and security fixes from stable
71 and LTS branches.
72
73- Ensure you remove or disable debugging functionality before producing
74 the final image. For information on how to do this, see the
75 ":ref:`dev-manual/securing-images:considerations specific to the openembedded build system`"
76 section.
77
78- Ensure you have no network services listening that are not needed.
79
80- Remove any software from the image that is not needed.
81
82- Enable hardware support for secure boot functionality when your
83 device supports this functionality.
84
85Security Flags
86==============
87
88The Yocto Project has security flags that you can enable that help make
89your build output more secure. The security flags are in the
90``meta/conf/distro/include/security_flags.inc`` file in your
91:term:`Source Directory` (e.g. ``poky``).
92
93.. note::
94
95 Depending on the recipe, certain security flags are enabled and
96 disabled by default.
97
98Use the following line in your ``local.conf`` file or in your custom
99distribution configuration file to enable the security compiler and
100linker flags for your build::
101
102 require conf/distro/include/security_flags.inc
103
104Considerations Specific to the OpenEmbedded Build System
105========================================================
106
107You can take some steps that are specific to the OpenEmbedded build
108system to make your images more secure:
109
110- Ensure "debug-tweaks" is not one of your selected
111 :term:`IMAGE_FEATURES`.
112 When creating a new project, the default is to provide you with an
113 initial ``local.conf`` file that enables this feature using the
114 :term:`EXTRA_IMAGE_FEATURES`
115 variable with the line::
116
117 EXTRA_IMAGE_FEATURES = "debug-tweaks"
118
119 To disable that feature, simply comment out that line in your
120 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
121 "debug-tweaks" before producing your final image. Among other things,
122 leaving this in place sets the root password as blank, which makes
123 logging in for debugging or inspection easy during development but
124 also means anyone can easily log in during production.
125
126- It is possible to set a root password for the image and also to set
127 passwords for any extra users you might add (e.g. administrative or
128 service type users). When you set up passwords for multiple images or
129 users, you should not duplicate passwords.
130
131 To set up passwords, use the
132 :ref:`extrausers <ref-classes-extrausers>`
133 class, which is the preferred method. For an example on how to set up
134 both root and user passwords, see the
135 ":ref:`ref-classes-extrausers`" section.
136
137 .. note::
138
139 When adding extra user accounts or setting a root password, be
140 cautious about setting the same password on every device. If you
141 do this, and the password you have set is exposed, then every
142 device is now potentially compromised. If you need this access but
143 want to ensure security, consider setting a different, random
144 password for each device. Typically, you do this as a separate
145 step after you deploy the image onto the device.
146
147- Consider enabling a Mandatory Access Control (MAC) framework such as
148 SMACK or SELinux and tuning it appropriately for your device's usage.
149 You can find more information in the
150 :yocto_git:`meta-selinux </meta-selinux/>` layer.
151
152Tools for Hardening Your Image
153==============================
154
155The Yocto Project provides tools for making your image more secure. You
156can find these tools in the ``meta-security`` layer of the
157:yocto_git:`Yocto Project Source Repositories <>`.
158
diff --git a/documentation/dev-manual/speeding-up-build.rst b/documentation/dev-manual/speeding-up-build.rst
new file mode 100644
index 0000000000..696b1bdf76
--- /dev/null
+++ b/documentation/dev-manual/speeding-up-build.rst
@@ -0,0 +1,110 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Speeding Up a Build
4*******************
5
6Build time can be an issue. By default, the build system uses simple
7controls to try and maximize build efficiency. In general, the default
8settings for all the following variables result in the most efficient
9build times when dealing with single socket systems (i.e. a single CPU).
10If you have multiple CPUs, you might try increasing the default values
11to gain more speed. See the descriptions in the glossary for each
12variable for more information:
13
14- :term:`BB_NUMBER_THREADS`:
15 The maximum number of threads BitBake simultaneously executes.
16
17- :term:`BB_NUMBER_PARSE_THREADS`:
18 The number of threads BitBake uses during parsing.
19
20- :term:`PARALLEL_MAKE`: Extra
21 options passed to the ``make`` command during the
22 :ref:`ref-tasks-compile` task in
23 order to specify parallel compilation on the local build host.
24
25- :term:`PARALLEL_MAKEINST`:
26 Extra options passed to the ``make`` command during the
27 :ref:`ref-tasks-install` task in
28 order to specify parallel installation on the local build host.
29
30As mentioned, these variables all scale to the number of processor cores
31available on the build system. For single socket systems, this
32auto-scaling ensures that the build system fundamentally takes advantage
33of potential parallel operations during the build based on the build
34machine's capabilities.
35
36Following are additional factors that can affect build speed:
37
38- File system type: The file system type that the build is being
39 performed on can also influence performance. Using ``ext4`` is
40 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
41 improved features such as extents.
42
43- Disabling the updating of access time using ``noatime``: The
44 ``noatime`` mount option prevents the build system from updating file
45 and directory access times.
46
47- Setting a longer commit: Using the "commit=" mount option increases
48 the interval in seconds between disk cache writes. Changing this
49 interval from the five second default to something longer increases
50 the risk of data loss but decreases the need to write to the disk,
51 thus increasing the build performance.
52
53- Choosing the packaging backend: Of the available packaging backends,
54 IPK is the fastest. Additionally, selecting a singular packaging
55 backend also helps.
56
57- Using ``tmpfs`` for :term:`TMPDIR`
58 as a temporary file system: While this can help speed up the build,
59 the benefits are limited due to the compiler using ``-pipe``. The
60 build system goes to some lengths to avoid ``sync()`` calls into the
61 file system on the principle that if there was a significant failure,
62 the :term:`Build Directory` contents could easily be rebuilt.
63
64- Inheriting the
65 :ref:`rm_work <ref-classes-rm-work>` class:
66 Inheriting this class has shown to speed up builds due to
67 significantly lower amounts of data stored in the data cache as well
68 as on disk. Inheriting this class also makes cleanup of
69 :term:`TMPDIR` faster, at the
70 expense of being easily able to dive into the source code. File
71 system maintainers have recommended that the fastest way to clean up
72 large numbers of files is to reformat partitions rather than delete
73 files due to the linear nature of partitions. This, of course,
74 assumes you structure the disk partitions and file systems in a way
75 that this is practical.
76
77Aside from the previous list, you should keep some trade offs in mind
78that can help you speed up the build:
79
80- Remove items from
81 :term:`DISTRO_FEATURES`
82 that you might not need.
83
84- Exclude debug symbols and other debug information: If you do not need
85 these symbols and other debug information, disabling the ``*-dbg``
86 package generation can speed up the build. You can disable this
87 generation by setting the
88 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
89 variable to "1".
90
91- Disable static library generation for recipes derived from
92 ``autoconf`` or ``libtool``: Following is an example showing how to
93 disable static libraries and still provide an override to handle
94 exceptions::
95
96 STATICLIBCONF = "--disable-static"
97 STATICLIBCONF:sqlite3-native = ""
98 EXTRA_OECONF += "${STATICLIBCONF}"
99
100 .. note::
101
102 - Some recipes need static libraries in order to work correctly
103 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
104 as in the previous example, account for these kinds of
105 exceptions.
106
107 - Some packages have packaging code that assumes the presence of
108 the static libraries. If so, you might need to exclude them as
109 well.
110
diff --git a/documentation/dev-manual/start.rst b/documentation/dev-manual/start.rst
index 63d63fc27e..b02e961608 100644
--- a/documentation/dev-manual/start.rst
+++ b/documentation/dev-manual/start.rst
@@ -223,7 +223,7 @@ particular working environment and set of practices.
223 - Maintain your Metadata in layers that make sense for your 223 - Maintain your Metadata in layers that make sense for your
224 situation. See the ":ref:`overview-manual/yp-intro:the yocto project layer model`" 224 situation. See the ":ref:`overview-manual/yp-intro:the yocto project layer model`"
225 section in the Yocto Project Overview and Concepts Manual and the 225 section in the Yocto Project Overview and Concepts Manual and the
226 ":ref:`dev-manual/common-tasks:understanding and creating layers`" 226 ":ref:`dev-manual/layers:understanding and creating layers`"
227 section for more information on layers. 227 section for more information on layers.
228 228
229 - Separate the project's Metadata and code by using separate Git 229 - Separate the project's Metadata and code by using separate Git
@@ -247,13 +247,13 @@ particular working environment and set of practices.
247 project to fix bugs or add features. If you do submit patches, 247 project to fix bugs or add features. If you do submit patches,
248 follow the project commit guidelines for writing good commit 248 follow the project commit guidelines for writing good commit
249 messages. See the 249 messages. See the
250 ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" 250 ":ref:`dev-manual/changes:submitting a change to the yocto project`"
251 section. 251 section.
252 252
253 - Send changes to the core sooner than later as others are likely 253 - Send changes to the core sooner than later as others are likely
254 to run into the same issues. For some guidance on mailing lists 254 to run into the same issues. For some guidance on mailing lists
255 to use, see the list in the 255 to use, see the list in the
256 ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" 256 ":ref:`dev-manual/changes:submitting a change to the yocto project`"
257 section. For a description 257 section. For a description
258 of the available mailing lists, see the ":ref:`resources-mailinglist`" section in 258 of the available mailing lists, see the ":ref:`resources-mailinglist`" section in
259 the Yocto Project Reference Manual. 259 the Yocto Project Reference Manual.
diff --git a/documentation/dev-manual/temporary-source-code.rst b/documentation/dev-manual/temporary-source-code.rst
new file mode 100644
index 0000000000..08bf68d982
--- /dev/null
+++ b/documentation/dev-manual/temporary-source-code.rst
@@ -0,0 +1,66 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Finding Temporary Source Code
4*****************************
5
6You might find it helpful during development to modify the temporary
7source code used by recipes to build packages. For example, suppose you
8are developing a patch and you need to experiment a bit to figure out
9your solution. After you have initially built the package, you can
10iteratively tweak the source code, which is located in the
11:term:`Build Directory`, and then you can force a re-compile and quickly
12test your altered code. Once you settle on a solution, you can then preserve
13your changes in the form of patches.
14
15During a build, the unpacked temporary source code used by recipes to
16build packages is available in the :term:`Build Directory` as defined by the
17:term:`S` variable. Below is the default value for the :term:`S` variable as
18defined in the ``meta/conf/bitbake.conf`` configuration file in the
19:term:`Source Directory`::
20
21 S = "${WORKDIR}/${BP}"
22
23You should be aware that many recipes override the
24:term:`S` variable. For example, recipes that fetch their source from Git
25usually set :term:`S` to ``${WORKDIR}/git``.
26
27.. note::
28
29 The :term:`BP` represents the base recipe name, which consists of the name
30 and version::
31
32 BP = "${BPN}-${PV}"
33
34
35The path to the work directory for the recipe
36(:term:`WORKDIR`) is defined as
37follows::
38
39 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
40
41The actual directory depends on several things:
42
43- :term:`TMPDIR`: The top-level build
44 output directory.
45
46- :term:`MULTIMACH_TARGET_SYS`:
47 The target system identifier.
48
49- :term:`PN`: The recipe name.
50
51- :term:`EXTENDPE`: The epoch --- if
52 :term:`PE` is not specified, which is
53 usually the case for most recipes, then :term:`EXTENDPE` is blank.
54
55- :term:`PV`: The recipe version.
56
57- :term:`PR`: The recipe revision.
58
59As an example, assume a Source Directory top-level folder named
60``poky``, a default :term:`Build Directory` at ``poky/build``, and a
61``qemux86-poky-linux`` machine target system. Furthermore, suppose your
62recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
63build system uses to build the package would be as follows::
64
65 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
66
diff --git a/documentation/dev-manual/upgrading-recipes.rst b/documentation/dev-manual/upgrading-recipes.rst
new file mode 100644
index 0000000000..a0856f43da
--- /dev/null
+++ b/documentation/dev-manual/upgrading-recipes.rst
@@ -0,0 +1,400 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Upgrading Recipes
4*****************
5
6Over time, upstream developers publish new versions for software built
7by layer recipes. It is recommended to keep recipes up-to-date with
8upstream version releases.
9
10While there are several methods to upgrade a recipe, you might
11consider checking on the upgrade status of a recipe first. You can do so
12using the ``devtool check-upgrade-status`` command. See the
13":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
14section in the Yocto Project Reference Manual for more information.
15
16The remainder of this section describes three ways you can upgrade a
17recipe. You can use the Automated Upgrade Helper (AUH) to set up
18automatic version upgrades. Alternatively, you can use
19``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
20you can manually upgrade a recipe by editing the recipe itself.
21
22Using the Auto Upgrade Helper (AUH)
23===================================
24
25The AUH utility works in conjunction with the OpenEmbedded build system
26in order to automatically generate upgrades for recipes based on new
27versions being published upstream. Use AUH when you want to create a
28service that performs the upgrades automatically and optionally sends
29you an email with the results.
30
31AUH allows you to update several recipes with a single use. You can also
32optionally perform build and integration tests using images with the
33results saved to your hard drive and emails of results optionally sent
34to recipe maintainers. Finally, AUH creates Git commits with appropriate
35commit messages in the layer's tree for the changes made to recipes.
36
37.. note::
38
39 In some conditions, you should not use AUH to upgrade recipes
40 and should instead use either ``devtool upgrade`` or upgrade your
41 recipes manually:
42
43 - When AUH cannot complete the upgrade sequence. This situation
44 usually results because custom patches carried by the recipe
45 cannot be automatically rebased to the new version. In this case,
46 ``devtool upgrade`` allows you to manually resolve conflicts.
47
48 - When for any reason you want fuller control over the upgrade
49 process. For example, when you want special arrangements for
50 testing.
51
52The following steps describe how to set up the AUH utility:
53
541. *Be Sure the Development Host is Set Up:* You need to be sure that
55 your development host is set up to use the Yocto Project. For
56 information on how to set up your host, see the
57 ":ref:`dev-manual/start:Preparing the Build Host`" section.
58
592. *Make Sure Git is Configured:* The AUH utility requires Git to be
60 configured because AUH uses Git to save upgrades. Thus, you must have
61 Git user and email configured. The following command shows your
62 configurations::
63
64 $ git config --list
65
66 If you do not have the user and
67 email configured, you can use the following commands to do so::
68
69 $ git config --global user.name some_name
70 $ git config --global user.email username@domain.com
71
723. *Clone the AUH Repository:* To use AUH, you must clone the repository
73 onto your development host. The following command uses Git to create
74 a local copy of the repository on your system::
75
76 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
77 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
78 remote: Compressing objects: 100% (300/300), done.
79 remote: Total 768 (delta 499), reused 703 (delta 434)
80 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
81 Resolving deltas: 100% (499/499), done.
82 Checking connectivity... done.
83
84 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
85 :term:`Poky` repositories.
86
874. *Create a Dedicated Build Directory:* Run the :ref:`structure-core-script`
88 script to create a fresh :term:`Build Directory` that you use exclusively
89 for running the AUH utility::
90
91 $ cd poky
92 $ source oe-init-build-env your_AUH_build_directory
93
94 Re-using an existing :term:`Build Directory` and its configurations is not
95 recommended as existing settings could cause AUH to fail or behave
96 undesirably.
97
985. *Make Configurations in Your Local Configuration File:* Several
99 settings are needed in the ``local.conf`` file in the build
100 directory you just created for AUH. Make these following
101 configurations:
102
103 - If you want to enable :ref:`Build
104 History <dev-manual/build-quality:maintaining build output quality>`,
105 which is optional, you need the following lines in the
106 ``conf/local.conf`` file::
107
108 INHERIT =+ "buildhistory"
109 BUILDHISTORY_COMMIT = "1"
110
111 With this configuration and a successful
112 upgrade, a build history "diff" file appears in the
113 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
114 your :term:`Build Directory`.
115
116 - If you want to enable testing through the
117 :ref:`testimage <ref-classes-testimage>`
118 class, which is optional, you need to have the following set in
119 your ``conf/local.conf`` file::
120
121 INHERIT += "testimage"
122
123 .. note::
124
125 If your distro does not enable by default ptest, which Poky
126 does, you need the following in your ``local.conf`` file::
127
128 DISTRO_FEATURES:append = " ptest"
129
130
1316. *Optionally Start a vncserver:* If you are running in a server
132 without an X11 session, you need to start a vncserver::
133
134 $ vncserver :1
135 $ export DISPLAY=:1
136
1377. *Create and Edit an AUH Configuration File:* You need to have the
138 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
139 :term:`Build Directory`. You can find a sample configuration file in the
140 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
141
142 Read through the sample file and make configurations as needed. For
143 example, if you enabled build history in your ``local.conf`` as
144 described earlier, you must enable it in ``upgrade-helper.conf``.
145
146 Also, if you are using the default ``maintainers.inc`` file supplied
147 with Poky and located in ``meta-yocto`` and you do not set a
148 "maintainers_whitelist" or "global_maintainer_override" in the
149 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
150 the AUH command-line, the utility automatically sends out emails to
151 all the default maintainers. Please avoid this.
152
153This next set of examples describes how to use the AUH:
154
155- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
156 following form::
157
158 $ upgrade-helper.py recipe_name
159
160 For example, this command upgrades the ``xmodmap`` recipe::
161
162 $ upgrade-helper.py xmodmap
163
164- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
165 specific recipe to a particular version, use the following form::
166
167 $ upgrade-helper.py recipe_name -t version
168
169 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
170
171 $ upgrade-helper.py xmodmap -t 1.2.3
172
173- *Upgrading all Recipes to the Latest Versions and Suppressing Email
174 Notifications:* To upgrade all recipes to their most recent versions
175 and suppress the email notifications, use the following command::
176
177 $ upgrade-helper.py all
178
179- *Upgrading all Recipes to the Latest Versions and Send Email
180 Notifications:* To upgrade all recipes to their most recent versions
181 and send email messages to maintainers for each attempted recipe as
182 well as a status email, use the following command::
183
184 $ upgrade-helper.py -e all
185
186Once you have run the AUH utility, you can find the results in the AUH
187:term:`Build Directory`::
188
189 ${BUILDDIR}/upgrade-helper/timestamp
190
191The AUH utility
192also creates recipe update commits from successful upgrade attempts in
193the layer tree.
194
195You can easily set up to run the AUH utility on a regular basis by using
196a cron job. See the
197:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
198file distributed with the utility for an example.
199
200Using ``devtool upgrade``
201=========================
202
203As mentioned earlier, an alternative method for upgrading recipes to
204newer versions is to use
205:doc:`devtool upgrade </ref-manual/devtool-reference>`.
206You can read about ``devtool upgrade`` in general in the
207":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
208section in the Yocto Project Application Development and the Extensible
209Software Development Kit (eSDK) Manual.
210
211To see all the command-line options available with ``devtool upgrade``,
212use the following help command::
213
214 $ devtool upgrade -h
215
216If you want to find out what version a recipe is currently at upstream
217without any attempt to upgrade your local version of the recipe, you can
218use the following command::
219
220 $ devtool latest-version recipe_name
221
222As mentioned in the previous section describing AUH, ``devtool upgrade``
223works in a less-automated manner than AUH. Specifically,
224``devtool upgrade`` only works on a single recipe that you name on the
225command line, cannot perform build and integration testing using images,
226and does not automatically generate commits for changes in the source
227tree. Despite all these "limitations", ``devtool upgrade`` updates the
228recipe file to the new upstream version and attempts to rebase custom
229patches contained by the recipe as needed.
230
231.. note::
232
233 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
234 of a "wrapper" application for ``devtool upgrade``.
235
236A typical scenario involves having used Git to clone an upstream
237repository that you use during build operations. Because you have built the
238recipe in the past, the layer is likely added to your
239configuration already. If for some reason, the layer is not added, you
240could add it easily using the
241":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
242script. For example, suppose you use the ``nano.bb`` recipe from the
243``meta-oe`` layer in the ``meta-openembedded`` repository. For this
244example, assume that the layer has been cloned into following area::
245
246 /home/scottrif/meta-openembedded
247
248The following command from your :term:`Build Directory` adds the layer to
249your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
250
251 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
252 NOTE: Starting bitbake server...
253 Parsing recipes: 100% |##########################################| Time: 0:00:55
254 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
255 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
256 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
257 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
258 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
259
260For this example, assume that the ``nano.bb`` recipe that
261is upstream has a 2.9.3 version number. However, the version in the
262local repository is 2.7.4. The following command from your build
263directory automatically upgrades the recipe for you:
264
265.. note::
266
267 Using the ``-V`` option is not necessary. Omitting the version number causes
268 ``devtool upgrade`` to upgrade the recipe to the most recent version.
269
270::
271
272 $ devtool upgrade nano -V 2.9.3
273 NOTE: Starting bitbake server...
274 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
275 Parsing recipes: 100% |##########################################| Time: 0:00:46
276 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
277 NOTE: Extracting current version source...
278 NOTE: Resolving any missing task queue dependencies
279 .
280 .
281 .
282 NOTE: Executing SetScene Tasks
283 NOTE: Executing RunQueue Tasks
284 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
285 Adding changed files: 100% |#####################################| Time: 0:00:00
286 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
287 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
288
289Continuing with this example, you can use ``devtool build`` to build the
290newly upgraded recipe::
291
292 $ devtool build nano
293 NOTE: Starting bitbake server...
294 Loading cache: 100% |################################################################################################| Time: 0:00:01
295 Loaded 2040 entries from dependency cache.
296 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
297 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
298 NOTE: Resolving any missing task queue dependencies
299 .
300 .
301 .
302 NOTE: Executing SetScene Tasks
303 NOTE: Executing RunQueue Tasks
304 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
305 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
306
307Within the ``devtool upgrade`` workflow, you can
308deploy and test your rebuilt software. For this example,
309however, running ``devtool finish`` cleans up the workspace once the
310source in your workspace is clean. This usually means using Git to stage
311and submit commits for the changes generated by the upgrade process.
312
313Once the tree is clean, you can clean things up in this example with the
314following command from the ``${BUILDDIR}/workspace/sources/nano``
315directory::
316
317 $ devtool finish nano meta-oe
318 NOTE: Starting bitbake server...
319 Loading cache: 100% |################################################################################################| Time: 0:00:00
320 Loaded 2040 entries from dependency cache.
321 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
322 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
323 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
324 NOTE: Updating recipe nano_2.9.3.bb
325 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
326 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
327 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
328
329
330Using the ``devtool finish`` command cleans up the workspace and creates a patch
331file based on your commits. The tool puts all patch files back into the
332source directory in a sub-directory named ``nano`` in this case.
333
334Manually Upgrading a Recipe
335===========================
336
337If for some reason you choose not to upgrade recipes using
338:ref:`dev-manual/upgrading-recipes:Using the Auto Upgrade Helper (AUH)` or
339by :ref:`dev-manual/upgrading-recipes:Using \`\`devtool upgrade\`\``,
340you can manually edit the recipe files to upgrade the versions.
341
342.. note::
343
344 Manually updating multiple recipes scales poorly and involves many
345 steps. The recommendation to upgrade recipe versions is through AUH
346 or ``devtool upgrade``, both of which automate some steps and provide
347 guidance for others needed for the manual process.
348
349To manually upgrade recipe versions, follow these general steps:
350
3511. *Change the Version:* Rename the recipe such that the version (i.e.
352 the :term:`PV` part of the recipe name)
353 changes appropriately. If the version is not part of the recipe name,
354 change the value as it is set for :term:`PV` within the recipe itself.
355
3562. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
357 is fetched from Git or some other version control system, update
358 :term:`SRCREV` to point to the
359 commit hash that matches the new version.
360
3613. *Build the Software:* Try to build the recipe using BitBake. Typical
362 build failures include the following:
363
364 - License statements were updated for the new version. For this
365 case, you need to review any changes to the license and update the
366 values of :term:`LICENSE` and
367 :term:`LIC_FILES_CHKSUM`
368 as needed.
369
370 .. note::
371
372 License changes are often inconsequential. For example, the
373 license text's copyright year might have changed.
374
375 - Custom patches carried by the older version of the recipe might
376 fail to apply to the new version. For these cases, you need to
377 review the failures. Patches might not be necessary for the new
378 version of the software if the upgraded version has fixed those
379 issues. If a patch is necessary and failing, you need to rebase it
380 into the new version.
381
3824. *Optionally Attempt to Build for Several Architectures:* Once you
383 successfully build the new software for a given architecture, you
384 could test the build for other architectures by changing the
385 :term:`MACHINE` variable and
386 rebuilding the software. This optional step is especially important
387 if the recipe is to be released publicly.
388
3895. *Check the Upstream Change Log or Release Notes:* Checking both these
390 reveals if there are new features that could break
391 backwards-compatibility. If so, you need to take steps to mitigate or
392 eliminate that situation.
393
3946. *Optionally Create a Bootable Image and Test:* If you want, you can
395 test the new software by booting it onto actual hardware.
396
3977. *Create a Commit with the Change in the Layer Repository:* After all
398 builds work and any testing is successful, you can create commits for
399 any changes in the layer holding your upgraded recipe.
400
diff --git a/documentation/dev-manual/vulnerabilities.rst b/documentation/dev-manual/vulnerabilities.rst
new file mode 100644
index 0000000000..f8dac5edc6
--- /dev/null
+++ b/documentation/dev-manual/vulnerabilities.rst
@@ -0,0 +1,214 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Checking for Vulnerabilities
4****************************
5
6Vulnerabilities in Poky and OE-Core
7===================================
8
9The Yocto Project has an infrastructure to track and address unfixed
10known security vulnerabilities, as tracked by the public
11:wikipedia:`Common Vulnerabilities and Exposures (CVE) <Common_Vulnerabilities_and_Exposures>`
12database.
13
14The Yocto Project maintains a `list of known vulnerabilities
15<https://autobuilder.yocto.io/pub/non-release/patchmetrics/>`__
16for packages in Poky and OE-Core, tracking the evolution of the number of
17unpatched CVEs and the status of patches. Such information is available for
18the current development version and for each supported release.
19
20Security is a process, not a product, and thus at any time, a number of security
21issues may be impacting Poky and OE-Core. It is up to the maintainers, users,
22contributors and anyone interested in the issues to investigate and possibly fix them by
23updating software components to newer versions or by applying patches to address them.
24It is recommended to work with Poky and OE-Core upstream maintainers and submit
25patches to fix them, see ":ref:`dev-manual/changes:submitting a change to the yocto project`" for details.
26
27Vulnerability check at build time
28=================================
29
30To enable a check for CVE security vulnerabilities using :ref:`cve-check <ref-classes-cve-check>` in the specific image
31or target you are building, add the following setting to your configuration::
32
33 INHERIT += "cve-check"
34
35The CVE database contains some old incomplete entries which have been
36deemed not to impact Poky or OE-Core. These CVE entries can be excluded from the
37check using build configuration::
38
39 include conf/distro/include/cve-extra-exclusions.inc
40
41With this CVE check enabled, BitBake build will try to map each compiled software component
42recipe name and version information to the CVE database and generate recipe and
43image specific reports. These reports will contain:
44
45- metadata about the software component like names and versions
46
47- metadata about the CVE issue such as description and NVD link
48
49- for each software component, a list of CVEs which are possibly impacting this version
50
51- status of each CVE: ``Patched``, ``Unpatched`` or ``Ignored``
52
53The status ``Patched`` means that a patch file to address the security issue has been
54applied. ``Unpatched`` status means that no patches to address the issue have been
55applied and that the issue needs to be investigated. ``Ignored`` means that after
56analysis, it has been deemed to ignore the issue as it for example affects
57the software component on a different operating system platform.
58
59After a build with CVE check enabled, reports for each compiled source recipe will be
60found in ``build/tmp/deploy/cve``.
61
62For example the CVE check report for the ``flex-native`` recipe looks like::
63
64 $ cat poky/build/tmp/deploy/cve/flex-native
65 LAYER: meta
66 PACKAGE NAME: flex-native
67 PACKAGE VERSION: 2.6.4
68 CVE: CVE-2016-6354
69 CVE STATUS: Patched
70 CVE SUMMARY: Heap-based buffer overflow in the yy_get_next_buffer function in Flex before 2.6.1 might allow context-dependent attackers to cause a denial of service or possibly execute arbitrary code via vectors involving num_to_read.
71 CVSS v2 BASE SCORE: 7.5
72 CVSS v3 BASE SCORE: 9.8
73 VECTOR: NETWORK
74 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2016-6354
75
76 LAYER: meta
77 PACKAGE NAME: flex-native
78 PACKAGE VERSION: 2.6.4
79 CVE: CVE-2019-6293
80 CVE STATUS: Ignored
81 CVE SUMMARY: An issue was discovered in the function mark_beginning_as_normal in nfa.c in flex 2.6.4. There is a stack exhaustion problem caused by the mark_beginning_as_normal function making recursive calls to itself in certain scenarios involving lots of '*' characters. Remote attackers could leverage this vulnerability to cause a denial-of-service.
82 CVSS v2 BASE SCORE: 4.3
83 CVSS v3 BASE SCORE: 5.5
84 VECTOR: NETWORK
85 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2019-6293
86
87For images, a summary of all recipes included in the image and their CVEs is also
88generated in textual and JSON formats. These ``.cve`` and ``.json`` reports can be found
89in the ``tmp/deploy/images`` directory for each compiled image.
90
91At build time CVE check will also throw warnings about ``Unpatched`` CVEs::
92
93 WARNING: flex-2.6.4-r0 do_cve_check: Found unpatched CVE (CVE-2019-6293), for more information check /poky/build/tmp/work/core2-64-poky-linux/flex/2.6.4-r0/temp/cve.log
94 WARNING: libarchive-3.5.1-r0 do_cve_check: Found unpatched CVE (CVE-2021-36976), for more information check /poky/build/tmp/work/core2-64-poky-linux/libarchive/3.5.1-r0/temp/cve.log
95
96It is also possible to check the CVE status of individual packages as follows::
97
98 bitbake -c cve_check flex libarchive
99
100Fixing CVE product name and version mappings
101============================================
102
103By default, :ref:`cve-check <ref-classes-cve-check>` uses the recipe name :term:`BPN` as CVE
104product name when querying the CVE database. If this mapping contains false positives, e.g.
105some reported CVEs are not for the software component in question, or false negatives like
106some CVEs are not found to impact the recipe when they should, then the problems can be
107in the recipe name to CVE product mapping. These mapping issues can be fixed by setting
108the :term:`CVE_PRODUCT` variable inside the recipe. This defines the name of the software component in the
109upstream `NIST CVE database <https://nvd.nist.gov/>`__.
110
111The variable supports using vendor and product names like this::
112
113 CVE_PRODUCT = "flex_project:flex"
114
115In this example the vendor name used in the CVE database is ``flex_project`` and the
116product is ``flex``. With this setting the ``flex`` recipe only maps to this specific
117product and not products from other vendors with same name ``flex``.
118
119Similarly, when the recipe version :term:`PV` is not compatible with software versions used by
120the upstream software component releases and the CVE database, these can be fixed using
121the :term:`CVE_VERSION` variable.
122
123Note that if the CVE entries in the NVD database contain bugs or have missing or incomplete
124information, it is recommended to fix the information there directly instead of working
125around the issues possibly for a long time in Poky and OE-Core side recipes. Feedback to
126NVD about CVE entries can be provided through the `NVD contact form <https://nvd.nist.gov/info/contact-form>`__.
127
128Fixing vulnerabilities in recipes
129=================================
130
131If a CVE security issue impacts a software component, it can be fixed by updating to a newer
132version of the software component or by applying a patch. For Poky and OE-Core master branches, updating
133to a newer software component release with fixes is the best option, but patches can be applied
134if releases are not yet available.
135
136For stable branches, it is preferred to apply patches for the issues. For some software
137components minor version updates can also be applied if they are backwards compatible.
138
139Here is an example of fixing CVE security issues with patch files,
140an example from the :oe_layerindex:`ffmpeg recipe</layerindex/recipe/47350>`::
141
142 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
143 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \
144 file://fix-CVE-2020-20446.patch \
145 file://fix-CVE-2020-20453.patch \
146 file://fix-CVE-2020-22015.patch \
147 file://fix-CVE-2020-22021.patch \
148 file://fix-CVE-2020-22033-CVE-2020-22019.patch \
149 file://fix-CVE-2021-33815.patch \
150
151A good practice is to include the CVE identifier in both the patch file name
152and inside the patch file commit message using the format::
153
154 CVE: CVE-2020-22033
155
156CVE checker will then capture this information and change the CVE status to ``Patched``
157in the generated reports.
158
159If analysis shows that the CVE issue does not impact the recipe due to configuration, platform,
160version or other reasons, the CVE can be marked as ``Ignored`` using the :term:`CVE_CHECK_IGNORE` variable.
161As mentioned previously, if data in the CVE database is wrong, it is recommend to fix those
162issues in the CVE database directly.
163
164Recipes can be completely skipped by CVE check by including the recipe name in
165the :term:`CVE_CHECK_SKIP_RECIPE` variable.
166
167Implementation details
168======================
169
170Here's what the :ref:`cve-check <ref-classes-cve-check>` class does to
171find unpatched CVE IDs.
172
173First the code goes through each patch file provided by a recipe. If a valid CVE ID
174is found in the name of the file, the corresponding CVE is considered as patched.
175Don't forget that if multiple CVE IDs are found in the filename, only the last
176one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch
177file. The found CVE IDs are also considered as patched.
178
179Then, the code looks up all the CVE IDs in the NIST database for all the
180products defined in :term:`CVE_PRODUCT`. Then, for each found CVE:
181
182- If the package name (:term:`PN`) is part of
183 :term:`CVE_CHECK_SKIP_RECIPE`, it is considered as ``Patched``.
184
185- If the CVE ID is part of :term:`CVE_CHECK_IGNORE`, it is
186 set as ``Ignored``.
187
188- If the CVE ID is part of the patched CVE for the recipe, it is
189 already considered as ``Patched``.
190
191- Otherwise, the code checks whether the recipe version (:term:`PV`)
192 is within the range of versions impacted by the CVE. If so, the CVE
193 is considered as ``Unpatched``.
194
195The CVE database is stored in :term:`DL_DIR` and can be inspected using
196``sqlite3`` command as follows::
197
198 sqlite3 downloads/CVE_CHECK/nvdcve_1.1.db .dump | grep CVE-2021-37462
199
200When analyzing CVEs, it is recommended to:
201
202- study the latest information in `CVE database <https://nvd.nist.gov/vuln/search>`__.
203
204- check how upstream developers of the software component addressed the issue, e.g.
205 what patch was applied, which upstream release contains the fix.
206
207- check what other Linux distributions like `Debian <https://security-tracker.debian.org/tracker/>`__
208 did to analyze and address the issue.
209
210- follow security notices from other Linux distributions.
211
212- follow public `open source security mailing lists <https://oss-security.openwall.org/wiki/mailing-lists>`__ for
213 discussions and advance notifications of CVE bugs and software releases with fixes.
214
diff --git a/documentation/dev-manual/wayland.rst b/documentation/dev-manual/wayland.rst
new file mode 100644
index 0000000000..bcbf40acc5
--- /dev/null
+++ b/documentation/dev-manual/wayland.rst
@@ -0,0 +1,90 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using Wayland and Weston
4************************
5
6:wikipedia:`Wayland <Wayland_(display_server_protocol)>`
7is a computer display server protocol that provides a method for
8compositing window managers to communicate directly with applications
9and video hardware and expects them to communicate with input hardware
10using other libraries. Using Wayland with supporting targets can result
11in better control over graphics frame rendering than an application
12might otherwise achieve.
13
14The Yocto Project provides the Wayland protocol libraries and the
15reference :wikipedia:`Weston <Wayland_(display_server_protocol)#Weston>`
16compositor as part of its release. You can find the integrated packages
17in the ``meta`` layer of the :term:`Source Directory`.
18Specifically, you
19can find the recipes that build both Wayland and Weston at
20``meta/recipes-graphics/wayland``.
21
22You can build both the Wayland and Weston packages for use only with targets
23that accept the :wikipedia:`Mesa 3D and Direct Rendering Infrastructure
24<Mesa_(computer_graphics)>`, which is also known as Mesa DRI. This implies that
25you cannot build and use the packages if your target uses, for example, the
26Intel Embedded Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
27
28.. note::
29
30 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
31 emulated QEMU hardware. However, this version of Weston will run
32 under X emulation without issues.
33
34This section describes what you need to do to implement Wayland and use
35the Weston compositor when building an image for a supporting target.
36
37Enabling Wayland in an Image
38============================
39
40To enable Wayland, you need to enable it to be built and enable it to be
41included (installed) in the image.
42
43Building Wayland
44----------------
45
46To cause Mesa to build the ``wayland-egl`` platform and Weston to build
47Wayland with Kernel Mode Setting
48(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
49support, include the "wayland" flag in the
50:term:`DISTRO_FEATURES`
51statement in your ``local.conf`` file::
52
53 DISTRO_FEATURES:append = " wayland"
54
55.. note::
56
57 If X11 has been enabled elsewhere, Weston will build Wayland with X11
58 support
59
60Installing Wayland and Weston
61-----------------------------
62
63To install the Wayland feature into an image, you must include the
64following
65:term:`CORE_IMAGE_EXTRA_INSTALL`
66statement in your ``local.conf`` file::
67
68 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
69
70Running Weston
71==============
72
73To run Weston inside X11, enabling it as described earlier and building
74a Sato image is sufficient. If you are running your image under Sato, a
75Weston Launcher appears in the "Utility" category.
76
77Alternatively, you can run Weston through the command-line interpretor
78(CLI), which is better suited for development work. To run Weston under
79the CLI, you need to do the following after your image is built:
80
811. Run these commands to export ``XDG_RUNTIME_DIR``::
82
83 mkdir -p /tmp/$USER-weston
84 chmod 0700 /tmp/$USER-weston
85 export XDG_RUNTIME_DIR=/tmp/$USER-weston
86
872. Launch Weston in the shell::
88
89 weston
90
diff --git a/documentation/dev-manual/wic.rst b/documentation/dev-manual/wic.rst
new file mode 100644
index 0000000000..d2f7bd0130
--- /dev/null
+++ b/documentation/dev-manual/wic.rst
@@ -0,0 +1,732 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating Partitioned Images Using Wic
4*************************************
5
6Creating an image for a particular hardware target using the
7OpenEmbedded build system does not necessarily mean you can boot that
8image as is on your device. Physical devices accept and boot images in
9various ways depending on the specifics of the device. Usually,
10information about the hardware can tell you what image format the device
11requires. Should your device require multiple partitions on an SD card,
12flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
13create the properly partitioned image.
14
15The ``wic`` command generates partitioned images from existing
16OpenEmbedded build artifacts. Image generation is driven by partitioning
17commands contained in an OpenEmbedded kickstart file (``.wks``)
18specified either directly on the command line or as one of a selection
19of canned kickstart files as shown with the ``wic list images`` command
20in the
21":ref:`dev-manual/wic:generate an image using an existing kickstart file`"
22section. When you apply the command to a given set of build artifacts, the
23result is an image or set of images that can be directly written onto media and
24used on a particular system.
25
26.. note::
27
28 For a kickstart file reference, see the
29 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
30 Chapter in the Yocto Project Reference Manual.
31
32The ``wic`` command and the infrastructure it is based on is by
33definition incomplete. The purpose of the command is to allow the
34generation of customized images, and as such, was designed to be
35completely extensible through a plugin interface. See the
36":ref:`dev-manual/wic:using the wic plugin interface`" section
37for information on these plugins.
38
39This section provides some background information on Wic, describes what
40you need to have in place to run the tool, provides instruction on how
41to use the Wic utility, provides information on using the Wic plugins
42interface, and provides several examples that show how to use Wic.
43
44Background
45==========
46
47This section provides some background on the Wic utility. While none of
48this information is required to use Wic, you might find it interesting.
49
50- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
51 "oe" diphthong in "oeic" was promoted to the letter "w", because
52 "oeic" is both difficult to remember and to pronounce.
53
54- Wic is loosely based on the Meego Image Creator (``mic``) framework.
55 The Wic implementation has been heavily modified to make direct use
56 of OpenEmbedded build artifacts instead of package installation and
57 configuration, which are already incorporated within the OpenEmbedded
58 artifacts.
59
60- Wic is a completely independent standalone utility that initially
61 provides easier-to-use and more flexible replacements for an existing
62 functionality in OE-Core's
63 :ref:`image-live <ref-classes-image-live>`
64 class. The difference between Wic and those examples is that with Wic
65 the functionality of those scripts is implemented by a
66 general-purpose partitioning language, which is based on Redhat
67 kickstart syntax.
68
69Requirements
70============
71
72In order to use the Wic utility with the OpenEmbedded Build system, your
73system needs to meet the following requirements:
74
75- The Linux distribution on your development host must support the
76 Yocto Project. See the ":ref:`detailed-supported-distros`"
77 section in the Yocto Project Reference Manual for the list of
78 distributions that support the Yocto Project.
79
80- The standard system utilities, such as ``cp``, must be installed on
81 your development host system.
82
83- You must have sourced the build environment setup script (i.e.
84 :ref:`structure-core-script`) found in the :term:`Build Directory`.
85
86- You need to have the build artifacts already available, which
87 typically means that you must have already created an image using the
88 OpenEmbedded build system (e.g. ``core-image-minimal``). While it
89 might seem redundant to generate an image in order to create an image
90 using Wic, the current version of Wic requires the artifacts in the
91 form generated by the OpenEmbedded build system.
92
93- You must build several native tools, which are built to run on the
94 build system::
95
96 $ bitbake parted-native dosfstools-native mtools-native
97
98- Include "wic" as part of the
99 :term:`IMAGE_FSTYPES`
100 variable.
101
102- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
103 as part of the :term:`WKS_FILE` variable. If multiple candidate files can
104 be provided by different layers, specify all the possible names through the
105 :term:`WKS_FILES` variable instead.
106
107Getting Help
108============
109
110You can get general help for the ``wic`` command by entering the ``wic``
111command by itself or by entering the command with a help argument as
112follows::
113
114 $ wic -h
115 $ wic --help
116 $ wic help
117
118Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
119``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
120commands except "help" by using the following form::
121
122 $ wic help command
123
124For example, the following command returns help for the ``write``
125command::
126
127 $ wic help write
128
129Wic supports help for three topics: ``overview``, ``plugins``, and
130``kickstart``. You can get help for any topic using the following form::
131
132 $ wic help topic
133
134For example, the following returns overview help for Wic::
135
136 $ wic help overview
137
138There is one additional level of help for Wic. You can get help on
139individual images through the ``list`` command. You can use the ``list``
140command to return the available Wic images as follows::
141
142 $ wic list images
143 genericx86 Create an EFI disk image for genericx86*
144 edgerouter Create SD card image for Edgerouter
145 beaglebone-yocto Create SD card image for Beaglebone
146 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
147 systemd-bootdisk Create an EFI disk image with systemd-boot
148 mkhybridiso Create a hybrid ISO image
149 mkefidisk Create an EFI disk image
150 sdimage-bootpart Create SD card image with a boot partition
151 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
152 directdisk Create a 'pcbios' direct disk image
153 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
154 qemuriscv Create qcow2 image for RISC-V QEMU machines
155 directdisk-gpt Create a 'pcbios' direct disk image
156 efi-bootdisk
157
158Once you know the list of available
159Wic images, you can use ``help`` with the command to get help on a
160particular image. For example, the following command returns help on the
161"beaglebone-yocto" image::
162
163 $ wic list beaglebone-yocto help
164
165 Creates a partitioned SD card image for Beaglebone.
166 Boot files are located in the first vfat partition.
167
168Operational Modes
169=================
170
171You can use Wic in two different modes, depending on how much control
172you need for specifying the OpenEmbedded build artifacts that are used
173for creating the image: Raw and Cooked:
174
175- *Raw Mode:* You explicitly specify build artifacts through Wic
176 command-line arguments.
177
178- *Cooked Mode:* The current
179 :term:`MACHINE` setting and image
180 name are used to automatically locate and provide the build
181 artifacts. You just supply a kickstart file and the name of the image
182 from which to use artifacts.
183
184Regardless of the mode you use, you need to have the build artifacts
185ready and available.
186
187Raw Mode
188--------
189
190Running Wic in raw mode allows you to specify all the partitions through
191the ``wic`` command line. The primary use for raw mode is if you have
192built your kernel outside of the Yocto Project :term:`Build Directory`.
193In other words, you can point to arbitrary kernel, root filesystem locations,
194and so forth. Contrast this behavior with cooked mode where Wic looks in the
195:term:`Build Directory` (e.g. ``tmp/deploy/images/``\ machine).
196
197The general form of the ``wic`` command in raw mode is::
198
199 $ wic create wks_file options ...
200
201 Where:
202
203 wks_file:
204 An OpenEmbedded kickstart file. You can provide
205 your own custom file or use a file from a set of
206 existing files as described by further options.
207
208 optional arguments:
209 -h, --help show this help message and exit
210 -o OUTDIR, --outdir OUTDIR
211 name of directory to create image in
212 -e IMAGE_NAME, --image-name IMAGE_NAME
213 name of the image to use the artifacts from e.g. core-
214 image-sato
215 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
216 path to the /rootfs dir to use as the .wks rootfs
217 source
218 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
219 path to the dir containing the boot artifacts (e.g.
220 /EFI or /syslinux dirs) to use as the .wks bootimg
221 source
222 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
223 path to the dir containing the kernel to use in the
224 .wks bootimg
225 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
226 path to the native sysroot containing the tools to use
227 to build the image
228 -s, --skip-build-check
229 skip the build check
230 -f, --build-rootfs build rootfs
231 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
232 compress image with specified compressor
233 -m, --bmap generate .bmap
234 --no-fstab-update Do not change fstab file.
235 -v VARS_DIR, --vars VARS_DIR
236 directory with <image>.env files that store bitbake
237 variables
238 -D, --debug output debug information
239
240.. note::
241
242 You do not need root privileges to run Wic. In fact, you should not
243 run as root when using the utility.
244
245Cooked Mode
246-----------
247
248Running Wic in cooked mode leverages off artifacts in the
249:term:`Build Directory`. In other words, you do not have to specify kernel or
250root filesystem locations as part of the command. All you need to provide is
251a kickstart file and the name of the image from which to use artifacts
252by using the "-e" option. Wic looks in the :term:`Build Directory` (e.g.
253``tmp/deploy/images/``\ machine) for artifacts.
254
255The general form of the ``wic`` command using Cooked Mode is as follows::
256
257 $ wic create wks_file -e IMAGE_NAME
258
259 Where:
260
261 wks_file:
262 An OpenEmbedded kickstart file. You can provide
263 your own custom file or use a file from a set of
264 existing files provided with the Yocto Project
265 release.
266
267 required argument:
268 -e IMAGE_NAME, --image-name IMAGE_NAME
269 name of the image to use the artifacts from e.g. core-
270 image-sato
271
272Using an Existing Kickstart File
273================================
274
275If you do not want to create your own kickstart file, you can use an
276existing file provided by the Wic installation. As shipped, kickstart
277files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
278following two locations::
279
280 poky/meta-yocto-bsp/wic
281 poky/scripts/lib/wic/canned-wks
282
283Use the following command to list the available kickstart files::
284
285 $ wic list images
286 genericx86 Create an EFI disk image for genericx86*
287 beaglebone-yocto Create SD card image for Beaglebone
288 edgerouter Create SD card image for Edgerouter
289 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
290 directdisk-gpt Create a 'pcbios' direct disk image
291 mkefidisk Create an EFI disk image
292 directdisk Create a 'pcbios' direct disk image
293 systemd-bootdisk Create an EFI disk image with systemd-boot
294 mkhybridiso Create a hybrid ISO image
295 sdimage-bootpart Create SD card image with a boot partition
296 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
297 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
298
299When you use an existing file, you
300do not have to use the ``.wks`` extension. Here is an example in Raw
301Mode that uses the ``directdisk`` file::
302
303 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
304 -k kernel_dir -n native_sysroot
305
306Here are the actual partition language commands used in the
307``genericx86.wks`` file to generate an image::
308
309 # short-description: Create an EFI disk image for genericx86*
310 # long-description: Creates a partitioned EFI disk image for genericx86* machines
311 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
312 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
313 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
314
315 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
316
317Using the Wic Plugin Interface
318==============================
319
320You can extend and specialize Wic functionality by using Wic plugins.
321This section explains the Wic plugin interface.
322
323.. note::
324
325 Wic plugins consist of "source" and "imager" plugins. Imager plugins
326 are beyond the scope of this section.
327
328Source plugins provide a mechanism to customize partition content during
329the Wic image generation process. You can use source plugins to map
330values that you specify using ``--source`` commands in kickstart files
331(i.e. ``*.wks``) to a plugin implementation used to populate a given
332partition.
333
334.. note::
335
336 If you use plugins that have build-time dependencies (e.g. native
337 tools, bootloaders, and so forth) when building a Wic image, you need
338 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
339 variable.
340
341Source plugins are subclasses defined in plugin files. As shipped, the
342Yocto Project provides several plugin files. You can see the source
343plugin files that ship with the Yocto Project
344:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
345Each of these plugin files contains source plugins that are designed to
346populate a specific Wic image partition.
347
348Source plugins are subclasses of the ``SourcePlugin`` class, which is
349defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
350the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
351file is a subclass of the ``SourcePlugin`` class, which is found in the
352``pluginbase.py`` file.
353
354You can also implement source plugins in a layer outside of the Source
355Repositories (external layer). To do so, be sure that your plugin files
356are located in a directory whose path is
357``scripts/lib/wic/plugins/source/`` within your external layer. When the
358plugin files are located there, the source plugins they contain are made
359available to Wic.
360
361When the Wic implementation needs to invoke a partition-specific
362implementation, it looks for the plugin with the same name as the
363``--source`` parameter used in the kickstart file given to that
364partition. For example, if the partition is set up using the following
365command in a kickstart file::
366
367 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
368
369The methods defined as class
370members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
371``bootimg-pcbios.py`` plugin file are used.
372
373To be more concrete, here is the corresponding plugin definition from
374the ``bootimg-pcbios.py`` file for the previous command along with an
375example method called by the Wic implementation when it needs to prepare
376a partition using an implementation-specific function::
377
378 .
379 .
380 .
381 class BootimgPcbiosPlugin(SourcePlugin):
382 """
383 Create MBR boot partition and install syslinux on it.
384 """
385
386 name = 'bootimg-pcbios'
387 .
388 .
389 .
390 @classmethod
391 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
392 oe_builddir, bootimg_dir, kernel_dir,
393 rootfs_dir, native_sysroot):
394 """
395 Called to do the actual content population for a partition i.e. it
396 'prepares' the partition to be incorporated into the image.
397 In this case, prepare content for legacy bios boot partition.
398 """
399 .
400 .
401 .
402
403If a
404subclass (plugin) itself does not implement a particular function, Wic
405locates and uses the default version in the superclass. It is for this
406reason that all source plugins are derived from the ``SourcePlugin``
407class.
408
409The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
410a set of methods that source plugins can implement or override. Any
411plugins (subclass of ``SourcePlugin``) that do not implement a
412particular method inherit the implementation of the method from the
413``SourcePlugin`` class. For more information, see the ``SourcePlugin``
414class in the ``pluginbase.py`` file for details:
415
416The following list describes the methods implemented in the
417``SourcePlugin`` class:
418
419- ``do_prepare_partition()``: Called to populate a partition with
420 actual content. In other words, the method prepares the final
421 partition image that is incorporated into the disk image.
422
423- ``do_configure_partition()``: Called before
424 ``do_prepare_partition()`` to create custom configuration files for a
425 partition (e.g. syslinux or grub configuration files).
426
427- ``do_install_disk()``: Called after all partitions have been
428 prepared and assembled into a disk image. This method provides a hook
429 to allow finalization of a disk image (e.g. writing an MBR).
430
431- ``do_stage_partition()``: Special content-staging hook called
432 before ``do_prepare_partition()``. This method is normally empty.
433
434 Typically, a partition just uses the passed-in parameters (e.g. the
435 unmodified value of ``bootimg_dir``). However, in some cases, things
436 might need to be more tailored. As an example, certain files might
437 additionally need to be taken from ``bootimg_dir + /boot``. This hook
438 allows those files to be staged in a customized fashion.
439
440 .. note::
441
442 ``get_bitbake_var()`` allows you to access non-standard variables that
443 you might want to use for this behavior.
444
445You can extend the source plugin mechanism. To add more hooks, create
446more source plugin methods within ``SourcePlugin`` and the corresponding
447derived subclasses. The code that calls the plugin methods uses the
448``plugin.get_source_plugin_methods()`` function to find the method or
449methods needed by the call. Retrieval of those methods is accomplished
450by filling up a dict with keys that contain the method names of
451interest. On success, these will be filled in with the actual methods.
452See the Wic implementation for examples and details.
453
454Wic Examples
455============
456
457This section provides several examples that show how to use the Wic
458utility. All the examples assume the list of requirements in the
459":ref:`dev-manual/wic:requirements`" section have been met. The
460examples assume the previously generated image is
461``core-image-minimal``.
462
463Generate an Image using an Existing Kickstart File
464--------------------------------------------------
465
466This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
467file::
468
469 $ wic create mkefidisk -e core-image-minimal
470 INFO: Building wic-tools...
471 .
472 .
473 .
474 INFO: The new image(s) can be found here:
475 ./mkefidisk-201804191017-sda.direct
476
477 The following build artifacts were used to create the image(s):
478 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
479 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
480 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
481 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
482
483 INFO: The image(s) were created using OE kickstart file:
484 /home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
485
486The previous example shows the easiest way to create an image by running
487in cooked mode and supplying a kickstart file and the "-e" option to
488point to the existing build artifacts. Your ``local.conf`` file needs to
489have the :term:`MACHINE` variable set
490to the machine you are using, which is "qemux86" in this example.
491
492Once the image builds, the output provides image location, artifact use,
493and kickstart file information.
494
495.. note::
496
497 You should always verify the details provided in the output to make
498 sure that the image was indeed created exactly as expected.
499
500Continuing with the example, you can now write the image from the
501:term:`Build Directory` onto a USB stick, or whatever media for which you
502built your image, and boot from the media. You can write the image by using
503``bmaptool`` or ``dd``::
504
505 $ oe-run-native bmap-tools-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
506
507or ::
508
509 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
510
511.. note::
512
513 For more information on how to use the ``bmaptool``
514 to flash a device with an image, see the
515 ":ref:`dev-manual/bmaptool:flashing images using \`\`bmaptool\`\``"
516 section.
517
518Using a Modified Kickstart File
519-------------------------------
520
521Because partitioned image creation is driven by the kickstart file, it
522is easy to affect image creation by changing the parameters in the file.
523This next example demonstrates that through modification of the
524``directdisk-gpt`` kickstart file.
525
526As mentioned earlier, you can use the command ``wic list images`` to
527show the list of existing kickstart files. The directory in which the
528``directdisk-gpt.wks`` file resides is
529``scripts/lib/image/canned-wks/``, which is located in the
530:term:`Source Directory` (e.g. ``poky``).
531Because available files reside in this directory, you can create and add
532your own custom files to the directory. Subsequent use of the
533``wic list images`` command would then include your kickstart files.
534
535In this example, the existing ``directdisk-gpt`` file already does most
536of what is needed. However, for the hardware in this example, the image
537will need to boot from ``sdb`` instead of ``sda``, which is what the
538``directdisk-gpt`` kickstart file uses.
539
540The example begins by making a copy of the ``directdisk-gpt.wks`` file
541in the ``scripts/lib/image/canned-wks`` directory and then by changing
542the lines that specify the target disk from which to boot.
543::
544
545 $ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
546 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
547
548Next, the example modifies the ``directdisksdb-gpt.wks`` file and
549changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
550example changes the following two lines and leaves the remaining lines
551untouched::
552
553 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
554 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
555
556Once the lines are changed, the
557example generates the ``directdisksdb-gpt`` image. The command points
558the process at the ``core-image-minimal`` artifacts for the Next Unit of
559Computing (nuc) :term:`MACHINE` the
560``local.conf``.
561::
562
563 $ wic create directdisksdb-gpt -e core-image-minimal
564 INFO: Building wic-tools...
565 .
566 .
567 .
568 Initialising tasks: 100% |#######################################| Time: 0:00:01
569 NOTE: Executing SetScene Tasks
570 NOTE: Executing RunQueue Tasks
571 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
572 INFO: Creating image(s)...
573
574 INFO: The new image(s) can be found here:
575 ./directdisksdb-gpt-201710090938-sdb.direct
576
577 The following build artifacts were used to create the image(s):
578 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
579 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
580 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
581 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
582
583 INFO: The image(s) were created using OE kickstart file:
584 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
585
586Continuing with the example, you can now directly ``dd`` the image to a
587USB stick, or whatever media for which you built your image, and boot
588the resulting media::
589
590 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
591 140966+0 records in
592 140966+0 records out
593 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
594 $ sudo eject /dev/sdb
595
596Using a Modified Kickstart File and Running in Raw Mode
597-------------------------------------------------------
598
599This next example manually specifies each build artifact (runs in Raw
600Mode) and uses a modified kickstart file. The example also uses the
601``-o`` option to cause Wic to create the output somewhere other than the
602default output directory, which is the current directory::
603
604 $ wic create test.wks -o /home/stephano/testwic \
605 --rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
606 --bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
607 --kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \
608 --native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
609
610 INFO: Creating image(s)...
611
612 INFO: The new image(s) can be found here:
613 /home/stephano/testwic/test-201710091445-sdb.direct
614
615 The following build artifacts were used to create the image(s):
616 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
617 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
618 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
619 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
620
621 INFO: The image(s) were created using OE kickstart file:
622 test.wks
623
624For this example,
625:term:`MACHINE` did not have to be
626specified in the ``local.conf`` file since the artifact is manually
627specified.
628
629Using Wic to Manipulate an Image
630--------------------------------
631
632Wic image manipulation allows you to shorten turnaround time during
633image development. For example, you can use Wic to delete the kernel
634partition of a Wic image and then insert a newly built kernel. This
635saves you time from having to rebuild the entire image each time you
636modify the kernel.
637
638.. note::
639
640 In order to use Wic to manipulate a Wic image as in this example,
641 your development machine must have the ``mtools`` package installed.
642
643The following example examines the contents of the Wic image, deletes
644the existing kernel, and then inserts a new kernel:
645
6461. *List the Partitions:* Use the ``wic ls`` command to list all the
647 partitions in the Wic image::
648
649 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
650 Num Start End Size Fstype
651 1 1048576 25041919 23993344 fat16
652 2 25165824 72157183 46991360 ext4
653
654 The previous output shows two partitions in the
655 ``core-image-minimal-qemux86.wic`` image.
656
6572. *Examine a Particular Partition:* Use the ``wic ls`` command again
658 but in a different form to examine a particular partition.
659
660 .. note::
661
662 You can get command usage on any Wic command using the following
663 form::
664
665 $ wic help command
666
667
668 For example, the following command shows you the various ways to
669 use the
670 wic ls
671 command::
672
673 $ wic help ls
674
675
676 The following command shows what is in partition one::
677
678 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
679 Volume in drive : is boot
680 Volume Serial Number is E894-1809
681 Directory for ::/
682
683 libcom32 c32 186500 2017-10-09 16:06
684 libutil c32 24148 2017-10-09 16:06
685 syslinux cfg 220 2017-10-09 16:06
686 vesamenu c32 27104 2017-10-09 16:06
687 vmlinuz 6904608 2017-10-09 16:06
688 5 files 7 142 580 bytes
689 16 582 656 bytes free
690
691 The previous output shows five files, with the
692 ``vmlinuz`` being the kernel.
693
694 .. note::
695
696 If you see the following error, you need to update or create a
697 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
698 in the file. Then, run the Wic command again::
699
700 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
701 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
702 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
703
704
7053. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
706 ``vmlinuz`` file (kernel)::
707
708 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
709
7104. *Add In the New Kernel:* Use the ``wic cp`` command to add the
711 updated kernel to the Wic image. Depending on how you built your
712 kernel, it could be in different places. If you used ``devtool`` and
713 an SDK to build your kernel, it resides in the ``tmp/work`` directory
714 of the extensible SDK. If you used ``make`` to build the kernel, the
715 kernel will be in the ``workspace/sources`` area.
716
717 The following example assumes ``devtool`` was used to build the
718 kernel::
719
720 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
721 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
722
723 Once the new kernel is added back into the image, you can use the
724 ``dd`` command or :ref:`bmaptool
725 <dev-manual/bmaptool:flashing images using \`\`bmaptool\`\`>`
726 to flash your wic image onto an SD card or USB stick and test your
727 target.
728
729 .. note::
730
731 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
732
diff --git a/documentation/dev-manual/x32-psabi.rst b/documentation/dev-manual/x32-psabi.rst
new file mode 100644
index 0000000000..92b1f96fa4
--- /dev/null
+++ b/documentation/dev-manual/x32-psabi.rst
@@ -0,0 +1,54 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using x32 psABI
4***************
5
6x32 processor-specific Application Binary Interface (`x32
7psABI <https://software.intel.com/en-us/node/628948>`__) is a native
832-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
9ABI defines the calling conventions between functions in a processing
10environment. The interface determines what registers are used and what
11the sizes are for various C data types.
12
13Some processing environments prefer using 32-bit applications even when
14running on Intel 64-bit platforms. Consider the i386 psABI, which is a
15very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
16provide efficient use and access of the Intel 64-bit processor
17resources, leaving the system underutilized. Now consider the x86_64
18psABI. This ABI is newer and uses 64-bits for data sizes and program
19pointers. The extra bits increase the footprint size of the programs,
20libraries, and also increases the memory and file system size
21requirements. Executing under the x32 psABI enables user programs to
22utilize CPU and system resources more efficiently while keeping the
23memory footprint of the applications low. Extra bits are used for
24registers but not for addressing mechanisms.
25
26The Yocto Project supports the final specifications of x32 psABI as
27follows:
28
29- You can create packages and images in x32 psABI format on x86_64
30 architecture targets.
31
32- You can successfully build recipes with the x32 toolchain.
33
34- You can create and boot ``core-image-minimal`` and
35 ``core-image-sato`` images.
36
37- There is RPM Package Manager (RPM) support for x32 binaries.
38
39- There is support for large images.
40
41To use the x32 psABI, you need to edit your ``conf/local.conf``
42configuration file as follows::
43
44 MACHINE = "qemux86-64"
45 DEFAULTTUNE = "x86-64-x32"
46 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \
47 or 'INVALID')) or 'lib'}"
48
49Once you have set
50up your configuration file, use BitBake to build an image that supports
51the x32 psABI. Here is an example::
52
53 $ bitbake core-image-sato
54
diff --git a/documentation/kernel-dev/common.rst b/documentation/kernel-dev/common.rst
index 028b6af84c..e9660dd4e6 100644
--- a/documentation/kernel-dev/common.rst
+++ b/documentation/kernel-dev/common.rst
@@ -97,13 +97,13 @@ section:
97 97
98 For background information on working with common and BSP layers, 98 For background information on working with common and BSP layers,
99 see the 99 see the
100 ":ref:`dev-manual/common-tasks:understanding and creating layers`" 100 ":ref:`dev-manual/layers:understanding and creating layers`"
101 section in the Yocto Project Development Tasks Manual and the 101 section in the Yocto Project Development Tasks Manual and the
102 ":ref:`bsp-guide/bsp:bsp layers`" section in the Yocto Project Board 102 ":ref:`bsp-guide/bsp:bsp layers`" section in the Yocto Project Board
103 Support (BSP) Developer's Guide, respectively. For information on how to 103 Support (BSP) Developer's Guide, respectively. For information on how to
104 use the ``bitbake-layers create-layer`` command to quickly set up a layer, 104 use the ``bitbake-layers create-layer`` command to quickly set up a layer,
105 see the 105 see the
106 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`" 106 ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
107 section in the Yocto Project Development Tasks Manual. 107 section in the Yocto Project Development Tasks Manual.
108 108
1094. *Inform the BitBake Build Environment About Your Layer:* As directed 1094. *Inform the BitBake Build Environment About Your Layer:* As directed
@@ -213,13 +213,13 @@ section:
213 213
214 For background information on working with common and BSP layers, 214 For background information on working with common and BSP layers,
215 see the 215 see the
216 ":ref:`dev-manual/common-tasks:understanding and creating layers`" 216 ":ref:`dev-manual/layers:understanding and creating layers`"
217 section in the Yocto Project Development Tasks Manual and the 217 section in the Yocto Project Development Tasks Manual and the
218 ":ref:`bsp-guide/bsp:bsp layers`" section in the Yocto Project Board 218 ":ref:`bsp-guide/bsp:bsp layers`" section in the Yocto Project Board
219 Support (BSP) Developer's Guide, respectively. For information on how to 219 Support (BSP) Developer's Guide, respectively. For information on how to
220 use the ``bitbake-layers create-layer`` command to quickly set up a layer, 220 use the ``bitbake-layers create-layer`` command to quickly set up a layer,
221 see the 221 see the
222 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`" 222 ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
223 section in the Yocto Project Development Tasks Manual. 223 section in the Yocto Project Development Tasks Manual.
224 224
2254. *Inform the BitBake Build Environment About Your Layer:* As directed 2254. *Inform the BitBake Build Environment About Your Layer:* As directed
@@ -299,7 +299,7 @@ layer contains its own :term:`BitBake`
299append files (``.bbappend``) and provides a convenient mechanism to 299append files (``.bbappend``) and provides a convenient mechanism to
300create your own recipe files (``.bb``) as well as store and use kernel 300create your own recipe files (``.bb``) as well as store and use kernel
301patch files. For background information on working with layers, see the 301patch files. For background information on working with layers, see the
302":ref:`dev-manual/common-tasks:understanding and creating layers`" 302":ref:`dev-manual/layers:understanding and creating layers`"
303section in the Yocto Project Development Tasks Manual. 303section in the Yocto Project Development Tasks Manual.
304 304
305.. note:: 305.. note::
@@ -307,7 +307,7 @@ section in the Yocto Project Development Tasks Manual.
307 The Yocto Project comes with many tools that simplify tasks you need 307 The Yocto Project comes with many tools that simplify tasks you need
308 to perform. One such tool is the ``bitbake-layers create-layer`` 308 to perform. One such tool is the ``bitbake-layers create-layer``
309 command, which simplifies creating a new layer. See the 309 command, which simplifies creating a new layer. See the
310 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`" 310 ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
311 section in the Yocto Project Development Tasks Manual for 311 section in the Yocto Project Development Tasks Manual for
312 information on how to use this script to quick set up a new layer. 312 information on how to use this script to quick set up a new layer.
313 313
@@ -360,7 +360,7 @@ home directory:
360 The :term:`FILESEXTRAPATHS` and :term:`SRC_URI` statements 360 The :term:`FILESEXTRAPATHS` and :term:`SRC_URI` statements
361 enable the OpenEmbedded build system to find patch files. For more 361 enable the OpenEmbedded build system to find patch files. For more
362 information on using append files, see the 362 information on using append files, see the
363 ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`" 363 ":ref:`dev-manual/layers:appending other layers metadata with your layer`"
364 section in the Yocto Project Development Tasks Manual. 364 section in the Yocto Project Development Tasks Manual.
365 365
366Modifying an Existing Recipe 366Modifying an Existing Recipe
@@ -1002,7 +1002,7 @@ Section.
1002 For more information on append files and patches, see the 1002 For more information on append files and patches, see the
1003 ":ref:`kernel-dev/common:creating the append file`" and 1003 ":ref:`kernel-dev/common:creating the append file`" and
1004 ":ref:`kernel-dev/common:applying patches`" sections. You can also see the 1004 ":ref:`kernel-dev/common:applying patches`" sections. You can also see the
1005 ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`" 1005 ":ref:`dev-manual/layers:appending other layers metadata with your layer`"
1006 section in the Yocto Project Development Tasks Manual. 1006 section in the Yocto Project Development Tasks Manual.
1007 1007
1008 .. note:: 1008 .. note::
diff --git a/documentation/kernel-dev/faq.rst b/documentation/kernel-dev/faq.rst
index e40e3ff372..ceb5dda3f5 100644
--- a/documentation/kernel-dev/faq.rst
+++ b/documentation/kernel-dev/faq.rst
@@ -38,7 +38,7 @@ The kernel image (e.g. ``vmlinuz``) is provided by the
38specify whether or not the kernel image is installed in the generated 38specify whether or not the kernel image is installed in the generated
39root filesystem, override ``RDEPENDS:${KERNEL_PACKAGE_NAME}-base`` to include or not 39root filesystem, override ``RDEPENDS:${KERNEL_PACKAGE_NAME}-base`` to include or not
40include "kernel-image". See the 40include "kernel-image". See the
41":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`" 41":ref:`dev-manual/layers:appending other layers metadata with your layer`"
42section in the 42section in the
43Yocto Project Development Tasks Manual for information on how to use an 43Yocto Project Development Tasks Manual for information on how to use an
44append file to override metadata. 44append file to override metadata.
diff --git a/documentation/kernel-dev/intro.rst b/documentation/kernel-dev/intro.rst
index 267b7e7797..06cc884386 100644
--- a/documentation/kernel-dev/intro.rst
+++ b/documentation/kernel-dev/intro.rst
@@ -87,7 +87,7 @@ understand the following documentation:
87 as described in the Yocto Project Application Development and the 87 as described in the Yocto Project Application Development and the
88 Extensible Software Development Kit (eSDK) manual. 88 Extensible Software Development Kit (eSDK) manual.
89 89
90- The ":ref:`dev-manual/common-tasks:understanding and creating layers`" 90- The ":ref:`dev-manual/layers:understanding and creating layers`"
91 section in the Yocto Project Development Tasks Manual. 91 section in the Yocto Project Development Tasks Manual.
92 92
93- The ":ref:`kernel-dev/intro:kernel modification workflow`" section. 93- The ":ref:`kernel-dev/intro:kernel modification workflow`" section.
diff --git a/documentation/migration-guides/migration-1.4.rst b/documentation/migration-guides/migration-1.4.rst
index baf3c08379..471bfb0f2a 100644
--- a/documentation/migration-guides/migration-1.4.rst
+++ b/documentation/migration-guides/migration-1.4.rst
@@ -83,7 +83,7 @@ create an append file for the ``init-ifupdown`` recipe instead, which
83you can find in the :term:`Source Directory` at 83you can find in the :term:`Source Directory` at
84``meta/recipes-core/init-ifupdown``. For information on how to use 84``meta/recipes-core/init-ifupdown``. For information on how to use
85append files, see the 85append files, see the
86":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`" 86":ref:`dev-manual/layers:appending other layers metadata with your layer`"
87section in the Yocto Project Development Tasks Manual. 87section in the Yocto Project Development Tasks Manual.
88 88
89.. _migration-1.4-remote-debugging: 89.. _migration-1.4-remote-debugging:
diff --git a/documentation/migration-guides/migration-1.5.rst b/documentation/migration-guides/migration-1.5.rst
index ad7e239eaf..7ddbc01766 100644
--- a/documentation/migration-guides/migration-1.5.rst
+++ b/documentation/migration-guides/migration-1.5.rst
@@ -248,7 +248,7 @@ A new automated image testing framework has been added through the
248framework replaces the older ``imagetest-qemu`` framework. 248framework replaces the older ``imagetest-qemu`` framework.
249 249
250You can learn more about performing automated image tests in the 250You can learn more about performing automated image tests in the
251":ref:`dev-manual/common-tasks:performing automated runtime testing`" 251":ref:`dev-manual/runtime-testing:performing automated runtime testing`"
252section in the Yocto Project Development Tasks Manual. 252section in the Yocto Project Development Tasks Manual.
253 253
254.. _migration-1.5-build-history: 254.. _migration-1.5-build-history:
@@ -271,7 +271,7 @@ Following are changes to Build History:
271 option for each utility for more information on the new syntax. 271 option for each utility for more information on the new syntax.
272 272
273For more information on Build History, see the 273For more information on Build History, see the
274":ref:`dev-manual/common-tasks:maintaining build output quality`" 274":ref:`dev-manual/build-quality:maintaining build output quality`"
275section in the Yocto Project Development Tasks Manual. 275section in the Yocto Project Development Tasks Manual.
276 276
277.. _migration-1.5-udev: 277.. _migration-1.5-udev:
diff --git a/documentation/migration-guides/migration-1.6.rst b/documentation/migration-guides/migration-1.6.rst
index d07731dcb0..06414ba121 100644
--- a/documentation/migration-guides/migration-1.6.rst
+++ b/documentation/migration-guides/migration-1.6.rst
@@ -12,7 +12,7 @@ Project 1.6 Release (codename "daisy") from the prior release.
12The :ref:`archiver <ref-classes-archiver>` class has been rewritten 12The :ref:`archiver <ref-classes-archiver>` class has been rewritten
13and its configuration has been simplified. For more details on the 13and its configuration has been simplified. For more details on the
14source archiver, see the 14source archiver, see the
15":ref:`dev-manual/common-tasks:maintaining open source license compliance during your product's lifecycle`" 15":ref:`dev-manual/licenses:maintaining open source license compliance during your product's lifecycle`"
16section in the Yocto Project Development Tasks Manual. 16section in the Yocto Project Development Tasks Manual.
17 17
18.. _migration-1.6-packaging-changes: 18.. _migration-1.6-packaging-changes:
@@ -147,7 +147,7 @@ NFS mount, an error occurs.
147The ``PRINC`` variable has been deprecated and triggers a warning if 147The ``PRINC`` variable has been deprecated and triggers a warning if
148detected during a build. For :term:`PR` increments on changes, 148detected during a build. For :term:`PR` increments on changes,
149use the PR service instead. You can find out more about this service in 149use the PR service instead. You can find out more about this service in
150the ":ref:`dev-manual/common-tasks:working with a pr service`" 150the ":ref:`dev-manual/packages:working with a pr service`"
151section in the Yocto Project Development Tasks Manual. 151section in the Yocto Project Development Tasks Manual.
152 152
153.. _migration-1.6-variable-changes-IMAGE_TYPES: 153.. _migration-1.6-variable-changes-IMAGE_TYPES:
@@ -220,7 +220,7 @@ Package Test (ptest)
220 220
221Package Tests (ptest) are built but not installed by default. For 221Package Tests (ptest) are built but not installed by default. For
222information on using Package Tests, see the 222information on using Package Tests, see the
223":ref:`dev-manual/common-tasks:testing packages with ptest`" section in the 223":ref:`dev-manual/packages:testing packages with ptest`" section in the
224Yocto Project Development Tasks Manual. For information on the 224Yocto Project Development Tasks Manual. For information on the
225:ref:`ptest <ref-classes-ptest>` class, see the ":ref:`ref-classes-ptest`" 225:ref:`ptest <ref-classes-ptest>` class, see the ":ref:`ref-classes-ptest`"
226section. 226section.
diff --git a/documentation/migration-guides/migration-1.7.rst b/documentation/migration-guides/migration-1.7.rst
index e1a7594fb1..0c13e182cb 100644
--- a/documentation/migration-guides/migration-1.7.rst
+++ b/documentation/migration-guides/migration-1.7.rst
@@ -217,7 +217,7 @@ The following miscellaneous change occurred:
217 should manually remove old "build-id" files from your existing build 217 should manually remove old "build-id" files from your existing build
218 history repositories to avoid confusion. For information on the build 218 history repositories to avoid confusion. For information on the build
219 history feature, see the 219 history feature, see the
220 ":ref:`dev-manual/common-tasks:maintaining build output quality`" 220 ":ref:`dev-manual/build-quality:maintaining build output quality`"
221 section in the Yocto Project Development Tasks Manual. 221 section in the Yocto Project Development Tasks Manual.
222 222
223 223
diff --git a/documentation/migration-guides/migration-2.1.rst b/documentation/migration-guides/migration-2.1.rst
index f045feadff..0e266a7ff3 100644
--- a/documentation/migration-guides/migration-2.1.rst
+++ b/documentation/migration-guides/migration-2.1.rst
@@ -343,7 +343,7 @@ This release supports generation of GLib Introspective Repository (GIR)
343files through GObject introspection, which is the standard mechanism for 343files through GObject introspection, which is the standard mechanism for
344accessing GObject-based software from runtime environments. You can 344accessing GObject-based software from runtime environments. You can
345enable, disable, and test the generation of this data. See the 345enable, disable, and test the generation of this data. See the
346":ref:`dev-manual/common-tasks:enabling gobject introspection support`" 346":ref:`dev-manual/gobject-introspection:enabling gobject introspection support`"
347section in the Yocto Project Development Tasks Manual for more 347section in the Yocto Project Development Tasks Manual for more
348information. 348information.
349 349
diff --git a/documentation/migration-guides/migration-2.3.rst b/documentation/migration-guides/migration-2.3.rst
index 4a1f159e92..071a1945d3 100644
--- a/documentation/migration-guides/migration-2.3.rst
+++ b/documentation/migration-guides/migration-2.3.rst
@@ -363,7 +363,7 @@ The following changes have been made to Wic:
363.. note:: 363.. note::
364 364
365 For more information on Wic, see the 365 For more information on Wic, see the
366 ":ref:`dev-manual/common-tasks:creating partitioned images using wic`" 366 ":ref:`dev-manual/wic:creating partitioned images using wic`"
367 section in the Yocto Project Development Tasks Manual. 367 section in the Yocto Project Development Tasks Manual.
368 368
369- *Default Output Directory Changed:* Wic's default output directory is 369- *Default Output Directory Changed:* Wic's default output directory is
diff --git a/documentation/migration-guides/migration-2.5.rst b/documentation/migration-guides/migration-2.5.rst
index 04f4cd7e73..df117616bc 100644
--- a/documentation/migration-guides/migration-2.5.rst
+++ b/documentation/migration-guides/migration-2.5.rst
@@ -264,7 +264,7 @@ The following are additional changes:
264 will trigger a warning during :ref:`ref-tasks-rootfs`. 264 will trigger a warning during :ref:`ref-tasks-rootfs`.
265 265
266 For more information, see the 266 For more information, see the
267 ":ref:`dev-manual/common-tasks:post-installation scripts`" 267 ":ref:`dev-manual/new-recipe:post-installation scripts`"
268 section in the Yocto Project Development Tasks Manual. 268 section in the Yocto Project Development Tasks Manual.
269 269
270- The ``elf`` image type has been removed. This image type was removed 270- The ``elf`` image type has been removed. This image type was removed
diff --git a/documentation/migration-guides/migration-2.6.rst b/documentation/migration-guides/migration-2.6.rst
index 59592997ff..3d5ee18cbd 100644
--- a/documentation/migration-guides/migration-2.6.rst
+++ b/documentation/migration-guides/migration-2.6.rst
@@ -367,7 +367,7 @@ Any failure of a ``pkg_postinst()`` script (including exit 1) triggers
367an error during the :ref:`ref-tasks-rootfs` task. 367an error during the :ref:`ref-tasks-rootfs` task.
368 368
369For more information on post-installation behavior, see the 369For more information on post-installation behavior, see the
370":ref:`dev-manual/common-tasks:post-installation scripts`" 370":ref:`dev-manual/new-recipe:post-installation scripts`"
371section in the Yocto Project Development Tasks Manual. 371section in the Yocto Project Development Tasks Manual.
372 372
373.. _migration-2.6-python-3-profile-guided-optimizations: 373.. _migration-2.6-python-3-profile-guided-optimizations:
diff --git a/documentation/migration-guides/migration-general.rst b/documentation/migration-guides/migration-general.rst
index 8503017580..4d9dfbe617 100644
--- a/documentation/migration-guides/migration-general.rst
+++ b/documentation/migration-guides/migration-general.rst
@@ -103,4 +103,4 @@ any new Yocto Project release.
103 :ref:`buildhistory <ref-classes-buildhistory>` output using ``git diff`` or ``buildhistory-diff``. 103 :ref:`buildhistory <ref-classes-buildhistory>` output using ``git diff`` or ``buildhistory-diff``.
104 104
105 For more information on using :ref:`buildhistory <ref-classes-buildhistory>`, see 105 For more information on using :ref:`buildhistory <ref-classes-buildhistory>`, see
106 :ref:`dev-manual/common-tasks:maintaining build output quality`. 106 :ref:`dev-manual/build-quality:maintaining build output quality`.
diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst
index 386a9e09d5..562f460c33 100644
--- a/documentation/overview-manual/concepts.rst
+++ b/documentation/overview-manual/concepts.rst
@@ -34,7 +34,7 @@ itself is of various types:
34 34
35BitBake knows how to combine multiple data sources together and refers 35BitBake knows how to combine multiple data sources together and refers
36to each data source as a layer. For information on layers, see the 36to each data source as a layer. For information on layers, see the
37":ref:`dev-manual/common-tasks:understanding and creating layers`" 37":ref:`dev-manual/layers:understanding and creating layers`"
38section of the Yocto Project Development Tasks Manual. 38section of the Yocto Project Development Tasks Manual.
39 39
40Following are some brief details on these core components. For 40Following are some brief details on these core components. For
@@ -149,7 +149,7 @@ Conforming to a known structure allows BitBake to make assumptions
149during builds on where to find types of metadata. You can find 149during builds on where to find types of metadata. You can find
150procedures and learn about tools (i.e. ``bitbake-layers``) for creating 150procedures and learn about tools (i.e. ``bitbake-layers``) for creating
151layers suitable for the Yocto Project in the 151layers suitable for the Yocto Project in the
152":ref:`dev-manual/common-tasks:understanding and creating layers`" 152":ref:`dev-manual/layers:understanding and creating layers`"
153section of the Yocto Project Development Tasks Manual. 153section of the Yocto Project Development Tasks Manual.
154 154
155OpenEmbedded Build System Concepts 155OpenEmbedded Build System Concepts
@@ -307,7 +307,7 @@ during the build. By default, the layers listed in this file include
307layers minimally needed by the build system. However, you must manually 307layers minimally needed by the build system. However, you must manually
308add any custom layers you have created. You can find more information on 308add any custom layers you have created. You can find more information on
309working with the ``bblayers.conf`` file in the 309working with the ``bblayers.conf`` file in the
310":ref:`dev-manual/common-tasks:enabling your layer`" 310":ref:`dev-manual/layers:enabling your layer`"
311section in the Yocto Project Development Tasks Manual. 311section in the Yocto Project Development Tasks Manual.
312 312
313The files ``site.conf`` and ``auto.conf`` are not created by the 313The files ``site.conf`` and ``auto.conf`` are not created by the
@@ -398,7 +398,7 @@ a ``README`` file as good practice and especially if the layer is to be
398distributed, a configuration directory, and recipe directories. You can 398distributed, a configuration directory, and recipe directories. You can
399learn about the general structure for layers used with the Yocto Project 399learn about the general structure for layers used with the Yocto Project
400in the 400in the
401":ref:`dev-manual/common-tasks:creating your own layer`" 401":ref:`dev-manual/layers:creating your own layer`"
402section in the 402section in the
403Yocto Project Development Tasks Manual. For a general discussion on 403Yocto Project Development Tasks Manual. For a general discussion on
404layers and the many layers from which you can draw, see the 404layers and the many layers from which you can draw, see the
@@ -798,7 +798,7 @@ For more information on how the source directories are created, see the
798":ref:`overview-manual/concepts:source fetching`" section. For 798":ref:`overview-manual/concepts:source fetching`" section. For
799more information on how to create patches and how the build system 799more information on how to create patches and how the build system
800processes patches, see the 800processes patches, see the
801":ref:`dev-manual/common-tasks:patching code`" 801":ref:`dev-manual/new-recipe:patching code`"
802section in the 802section in the
803Yocto Project Development Tasks Manual. You can also see the 803Yocto Project Development Tasks Manual. You can also see the
804":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`" 804":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`"
@@ -999,7 +999,7 @@ stage of package installation, post installation scripts that are part
999of the packages are run. Any scripts that fail to run on the build host 999of the packages are run. Any scripts that fail to run on the build host
1000are run on the target when the target system is first booted. If you are 1000are run on the target when the target system is first booted. If you are
1001using a 1001using a
1002:ref:`read-only root filesystem <dev-manual/common-tasks:creating a read-only root filesystem>`, 1002:ref:`read-only root filesystem <dev-manual/read-only-rootfs:creating a read-only root filesystem>`,
1003all the post installation scripts must succeed on the build host during 1003all the post installation scripts must succeed on the build host during
1004the package installation phase since the root filesystem on the target 1004the package installation phase since the root filesystem on the target
1005is read-only. 1005is read-only.
@@ -1158,7 +1158,7 @@ varflag. If some other task depends on such a task, then that task will
1158also always be considered out of date, which might not be what you want. 1158also always be considered out of date, which might not be what you want.
1159 1159
1160For details on how to view information about a task's signature, see the 1160For details on how to view information about a task's signature, see the
1161":ref:`dev-manual/common-tasks:viewing task variable dependencies`" 1161":ref:`dev-manual/debugging:viewing task variable dependencies`"
1162section in the Yocto Project Development Tasks Manual. 1162section in the Yocto Project Development Tasks Manual.
1163 1163
1164Setscene Tasks and Shared State 1164Setscene Tasks and Shared State
@@ -1583,15 +1583,15 @@ them if they are deemed to be valid.
1583 the shared state packages. Consequently, there are considerations that 1583 the shared state packages. Consequently, there are considerations that
1584 affect maintaining shared state feeds. For information on how the 1584 affect maintaining shared state feeds. For information on how the
1585 build system works with packages and can track incrementing :term:`PR` 1585 build system works with packages and can track incrementing :term:`PR`
1586 information, see the ":ref:`dev-manual/common-tasks:automatically incrementing a package version number`" 1586 information, see the ":ref:`dev-manual/packages:automatically incrementing a package version number`"
1587 section in the Yocto Project Development Tasks Manual. 1587 section in the Yocto Project Development Tasks Manual.
1588 1588
1589 - The code in the build system that supports incremental builds is 1589 - The code in the build system that supports incremental builds is
1590 complex. For techniques that help you work around issues 1590 complex. For techniques that help you work around issues
1591 related to shared state code, see the 1591 related to shared state code, see the
1592 ":ref:`dev-manual/common-tasks:viewing metadata used to create the input signature of a shared state task`" 1592 ":ref:`dev-manual/debugging:viewing metadata used to create the input signature of a shared state task`"
1593 and 1593 and
1594 ":ref:`dev-manual/common-tasks:invalidating shared state to force a task to run`" 1594 ":ref:`dev-manual/debugging:invalidating shared state to force a task to run`"
1595 sections both in the Yocto Project Development Tasks Manual. 1595 sections both in the Yocto Project Development Tasks Manual.
1596 1596
1597The rest of this section goes into detail about the overall incremental 1597The rest of this section goes into detail about the overall incremental
diff --git a/documentation/overview-manual/development-environment.rst b/documentation/overview-manual/development-environment.rst
index 7d5953db33..0d931df3dc 100644
--- a/documentation/overview-manual/development-environment.rst
+++ b/documentation/overview-manual/development-environment.rst
@@ -93,7 +93,7 @@ are several ways of working in the Yocto Project environment:
93 through your Linux distribution and the Yocto Project. 93 through your Linux distribution and the Yocto Project.
94 94
95 For a general flow of the build procedures, see the 95 For a general flow of the build procedures, see the
96 ":ref:`dev-manual/common-tasks:building a simple image`" 96 ":ref:`dev-manual/building:building a simple image`"
97 section in the Yocto Project Development Tasks Manual. 97 section in the Yocto Project Development Tasks Manual.
98 98
99- *Board Support Package (BSP) Development:* Development of BSPs 99- *Board Support Package (BSP) Development:* Development of BSPs
@@ -244,7 +244,7 @@ and so forth.
244 244
245 For information on finding out who is responsible for (maintains) a 245 For information on finding out who is responsible for (maintains) a
246 particular area of code in the Yocto Project, see the 246 particular area of code in the Yocto Project, see the
247 ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" 247 ":ref:`dev-manual/changes:submitting a change to the yocto project`"
248 section of the Yocto Project Development Tasks Manual. 248 section of the Yocto Project Development Tasks Manual.
249 249
250The Yocto Project ``poky`` Git repository also has an upstream 250The Yocto Project ``poky`` Git repository also has an upstream
@@ -276,7 +276,7 @@ push them into the "contrib" area and subsequently request that the
276maintainer include them into an upstream branch. This process is called 276maintainer include them into an upstream branch. This process is called
277"submitting a patch" or "submitting a change." For information on 277"submitting a patch" or "submitting a change." For information on
278submitting patches and changes, see the 278submitting patches and changes, see the
279":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" 279":ref:`dev-manual/changes:submitting a change to the yocto project`"
280section in the Yocto Project Development Tasks Manual. 280section in the Yocto Project Development Tasks Manual.
281 281
282In summary, there is a single point of entry for changes into the 282In summary, there is a single point of entry for changes into the
@@ -343,7 +343,7 @@ Book <https://book.git-scm.com>`__.
343 the ``scripts`` folder of the 343 the ``scripts`` folder of the
344 :term:`Source Directory`. For information 344 :term:`Source Directory`. For information
345 on how to use these scripts, see the 345 on how to use these scripts, see the
346 ":ref:`dev-manual/common-tasks:using scripts to push a change upstream and request a pull`" 346 ":ref:`dev-manual/changes:using scripts to push a change upstream and request a pull`"
347 section in the Yocto Project Development Tasks Manual. 347 section in the Yocto Project Development Tasks Manual.
348 348
349- *Patch Workflow:* This workflow allows you to notify the maintainer 349- *Patch Workflow:* This workflow allows you to notify the maintainer
@@ -352,7 +352,7 @@ Book <https://book.git-scm.com>`__.
352 this type of change, you format the patch and then send the email 352 this type of change, you format the patch and then send the email
353 using the Git commands ``git format-patch`` and ``git send-email``. 353 using the Git commands ``git format-patch`` and ``git send-email``.
354 For information on how to use these scripts, see the 354 For information on how to use these scripts, see the
355 ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" 355 ":ref:`dev-manual/changes:submitting a change to the yocto project`"
356 section in the Yocto Project Development Tasks Manual. 356 section in the Yocto Project Development Tasks Manual.
357 357
358Git 358Git
@@ -647,5 +647,5 @@ Project uses in the ``meta/files/common-licenses`` directory in your
647For information that can help you maintain compliance with various open 647For information that can help you maintain compliance with various open
648source licensing during the lifecycle of a product created using the 648source licensing during the lifecycle of a product created using the
649Yocto Project, see the 649Yocto Project, see the
650":ref:`dev-manual/common-tasks:maintaining open source license compliance during your product's lifecycle`" 650":ref:`dev-manual/licenses:maintaining open source license compliance during your product's lifecycle`"
651section in the Yocto Project Development Tasks Manual. 651section in the Yocto Project Development Tasks Manual.
diff --git a/documentation/overview-manual/yp-intro.rst b/documentation/overview-manual/yp-intro.rst
index 8b476f43c4..600b46910e 100644
--- a/documentation/overview-manual/yp-intro.rst
+++ b/documentation/overview-manual/yp-intro.rst
@@ -129,7 +129,7 @@ Here are features and advantages of the Yocto Project:
129 arbitrarily include packages. 129 arbitrarily include packages.
130 130
131- *License Manifest:* The Yocto Project provides a :ref:`license 131- *License Manifest:* The Yocto Project provides a :ref:`license
132 manifest <dev-manual/common-tasks:maintaining open source license compliance during your product's lifecycle>` 132 manifest <dev-manual/licenses:maintaining open source license compliance during your product's lifecycle>`
133 for review by people who need to track the use of open source 133 for review by people who need to track the use of open source
134 licenses (e.g. legal teams). 134 licenses (e.g. legal teams).
135 135
@@ -225,7 +225,7 @@ your Metadata, the easier it is to cope with future changes.
225 225
226 - Layers support the inclusion of technologies, hardware components, 226 - Layers support the inclusion of technologies, hardware components,
227 and software components. The :ref:`Yocto Project 227 and software components. The :ref:`Yocto Project
228 Compatible <dev-manual/common-tasks:making sure your layer is compatible with yocto project>` 228 Compatible <dev-manual/layers:making sure your layer is compatible with yocto project>`
229 designation provides a minimum level of standardization that 229 designation provides a minimum level of standardization that
230 contributes to a strong ecosystem. "YP Compatible" is applied to 230 contributes to a strong ecosystem. "YP Compatible" is applied to
231 appropriate products and software components such as BSPs, other 231 appropriate products and software components such as BSPs, other
@@ -269,7 +269,7 @@ of the ``poky`` repository, you will see several layers: ``meta``,
269layer. 269layer.
270 270
271For procedures on how to create layers, see the 271For procedures on how to create layers, see the
272":ref:`dev-manual/common-tasks:understanding and creating layers`" 272":ref:`dev-manual/layers:understanding and creating layers`"
273section in the Yocto Project Development Tasks Manual. 273section in the Yocto Project Development Tasks Manual.
274 274
275Components and Tools 275Components and Tools
@@ -351,7 +351,7 @@ Yocto Project:
351 (BitBake and 351 (BitBake and
352 OE-Core) automatically generates upgrades for recipes that are based 352 OE-Core) automatically generates upgrades for recipes that are based
353 on new versions of the recipes published upstream. See 353 on new versions of the recipes published upstream. See
354 :ref:`dev-manual/common-tasks:using the auto upgrade helper (auh)` 354 :ref:`dev-manual/upgrading-recipes:using the auto upgrade helper (auh)`
355 for how to set it up. 355 for how to set it up.
356 356
357- *Recipe Reporting System:* The Recipe Reporting System tracks recipe 357- *Recipe Reporting System:* The Recipe Reporting System tracks recipe
@@ -776,7 +776,7 @@ helpful for getting started:
776 Yocto Project. 776 Yocto Project.
777 777
778 For more detailed information on layers, see the 778 For more detailed information on layers, see the
779 ":ref:`dev-manual/common-tasks:understanding and creating layers`" 779 ":ref:`dev-manual/layers:understanding and creating layers`"
780 section in the Yocto Project Development Tasks Manual. For a 780 section in the Yocto Project Development Tasks Manual. For a
781 discussion specifically on BSP Layers, see the 781 discussion specifically on BSP Layers, see the
782 ":ref:`bsp-guide/bsp:bsp layers`" section in the Yocto 782 ":ref:`bsp-guide/bsp:bsp layers`" section in the Yocto
diff --git a/documentation/ref-manual/classes.rst b/documentation/ref-manual/classes.rst
index aeab72a1e4..9f8593831f 100644
--- a/documentation/ref-manual/classes.rst
+++ b/documentation/ref-manual/classes.rst
@@ -74,7 +74,7 @@ The :ref:`archiver <ref-classes-archiver>` class supports releasing source code
74materials with the binaries. 74materials with the binaries.
75 75
76For more details on the source :ref:`archiver <ref-classes-archiver>`, see the 76For more details on the source :ref:`archiver <ref-classes-archiver>`, see the
77":ref:`dev-manual/common-tasks:maintaining open source license compliance during your product's lifecycle`" 77":ref:`dev-manual/licenses:maintaining open source license compliance during your product's lifecycle`"
78section in the Yocto Project Development Tasks Manual. You can also see 78section in the Yocto Project Development Tasks Manual. You can also see
79the :term:`ARCHIVER_MODE` variable for information 79the :term:`ARCHIVER_MODE` variable for information
80about the variable flags (varflags) that help control archive creation. 80about the variable flags (varflags) that help control archive creation.
@@ -93,7 +93,7 @@ standardization. This class defines a set of tasks (e.g. ``configure``,
93should usually be enough to define a few standard variables and then 93should usually be enough to define a few standard variables and then
94simply ``inherit autotools``. These classes can also work with software 94simply ``inherit autotools``. These classes can also work with software
95that emulates Autotools. For more information, see the 95that emulates Autotools. For more information, see the
96":ref:`dev-manual/common-tasks:autotooled package`" section 96":ref:`dev-manual/new-recipe:autotooled package`" section
97in the Yocto Project Development Tasks Manual. 97in the Yocto Project Development Tasks Manual.
98 98
99By default, the :ref:`autotools* <ref-classes-autotools>` classes use out-of-tree builds (i.e. 99By default, the :ref:`autotools* <ref-classes-autotools>` classes use out-of-tree builds (i.e.
@@ -222,7 +222,7 @@ The :ref:`buildhistory <ref-classes-buildhistory>` class records a history of bu
222which can be used to detect possible regressions as well as used for 222which can be used to detect possible regressions as well as used for
223analysis of the build output. For more information on using Build 223analysis of the build output. For more information on using Build
224History, see the 224History, see the
225":ref:`dev-manual/common-tasks:maintaining build output quality`" 225":ref:`dev-manual/build-quality:maintaining build output quality`"
226section in the Yocto Project Development Tasks Manual. 226section in the Yocto Project Development Tasks Manual.
227 227
228.. _ref-classes-buildstats: 228.. _ref-classes-buildstats:
@@ -390,7 +390,7 @@ by the :term:`SPDX_PRETTY`, :term:`SPDX_ARCHIVE_PACKAGED`,
390:term:`SPDX_ARCHIVE_SOURCES` and :term:`SPDX_INCLUDE_SOURCES` variables. 390:term:`SPDX_ARCHIVE_SOURCES` and :term:`SPDX_INCLUDE_SOURCES` variables.
391 391
392See the description of these variables and the 392See the description of these variables and the
393":ref:`dev-manual/common-tasks:creating a software bill of materials`" 393":ref:`dev-manual/sbom:creating a software bill of materials`"
394section in the Yocto Project Development Manual for more details. 394section in the Yocto Project Development Manual for more details.
395 395
396.. _ref-classes-cross: 396.. _ref-classes-cross:
@@ -484,7 +484,7 @@ These can only be detected by reviewing the details of the issues and iterating
484and following what happens in other Linux distributions and in the greater open source community. 484and following what happens in other Linux distributions and in the greater open source community.
485 485
486You will find some more details in the 486You will find some more details in the
487":ref:`dev-manual/common-tasks:checking for vulnerabilities`" 487":ref:`dev-manual/vulnerabilities:checking for vulnerabilities`"
488section in the Development Tasks Manual. 488section in the Development Tasks Manual.
489 489
490.. _ref-classes-debian: 490.. _ref-classes-debian:
@@ -524,7 +524,7 @@ staging the files from :term:`DEPLOYDIR` to :term:`DEPLOY_DIR_IMAGE`.
524==================== 524====================
525 525
526The :ref:`devshell <ref-classes-devshell>` class adds the :ref:`ref-tasks-devshell` task. Distribution 526The :ref:`devshell <ref-classes-devshell>` class adds the :ref:`ref-tasks-devshell` task. Distribution
527policy dictates whether to include this class. See the ":ref:`dev-manual/common-tasks:using a development shell`" 527policy dictates whether to include this class. See the ":ref:`dev-manual/development-shell:using a development shell`"
528section in the Yocto Project Development Tasks Manual for more 528section in the Yocto Project Development Tasks Manual for more
529information about using :ref:`devshell <ref-classes-devshell>`. 529information about using :ref:`devshell <ref-classes-devshell>`.
530 530
@@ -598,7 +598,7 @@ For more information on the :ref:`externalsrc <ref-classes-externalsrc>` class,
598``meta/classes/externalsrc.bbclass`` in the :term:`Source Directory`. 598``meta/classes/externalsrc.bbclass`` in the :term:`Source Directory`.
599For information on how to use the 599For information on how to use the
600:ref:`externalsrc <ref-classes-externalsrc>` class, see the 600:ref:`externalsrc <ref-classes-externalsrc>` class, see the
601":ref:`dev-manual/common-tasks:building software from an external source`" 601":ref:`dev-manual/building:building software from an external source`"
602section in the Yocto Project Development Tasks Manual. 602section in the Yocto Project Development Tasks Manual.
603 603
604.. _ref-classes-extrausers: 604.. _ref-classes-extrausers:
@@ -962,7 +962,7 @@ then one or more image files are created.
962 install into the image. 962 install into the image.
963 963
964For information on customizing images, see the 964For information on customizing images, see the
965":ref:`dev-manual/common-tasks:customizing images`" section 965":ref:`dev-manual/customizing-images:customizing images`" section
966in the Yocto Project Development Tasks Manual. For information on how 966in the Yocto Project Development Tasks Manual. For information on how
967images are created, see the 967images are created, see the
968":ref:`overview-manual/concepts:images`" section in the 968":ref:`overview-manual/concepts:images`" section in the
@@ -1364,7 +1364,7 @@ packages such as ``kernel-vmlinux``.
1364The :ref:`kernel <ref-classes-kernel>` class contains logic that allows you to embed an initial 1364The :ref:`kernel <ref-classes-kernel>` class contains logic that allows you to embed an initial
1365RAM filesystem (:term:`Initramfs`) image when you build the kernel image. For 1365RAM filesystem (:term:`Initramfs`) image when you build the kernel image. For
1366information on how to build an :term:`Initramfs`, see the 1366information on how to build an :term:`Initramfs`, see the
1367":ref:`dev-manual/common-tasks:building an initial ram filesystem (Initramfs) image`" section in 1367":ref:`dev-manual/building:building an initial ram filesystem (Initramfs) image`" section in
1368the Yocto Project Development Tasks Manual. 1368the Yocto Project Development Tasks Manual.
1369 1369
1370Various other classes are used by the :ref:`kernel <ref-classes-kernel>` and :ref:`module <ref-classes-module>` classes 1370Various other classes are used by the :ref:`kernel <ref-classes-kernel>` and :ref:`module <ref-classes-module>` classes
@@ -1674,7 +1674,7 @@ different target optimizations or target architectures and installing
1674them side-by-side in the same image. 1674them side-by-side in the same image.
1675 1675
1676For more information on using the Multilib feature, see the 1676For more information on using the Multilib feature, see the
1677":ref:`dev-manual/common-tasks:combining multiple versions of library files into one image`" 1677":ref:`dev-manual/libraries:combining multiple versions of library files into one image`"
1678section in the Yocto Project Development Tasks Manual. 1678section in the Yocto Project Development Tasks Manual.
1679 1679
1680.. _ref-classes-native: 1680.. _ref-classes-native:
@@ -1782,7 +1782,7 @@ Provides support for building Node.js software fetched using the
1782 fetcher to have dependencies fetched and packaged automatically. 1782 fetcher to have dependencies fetched and packaged automatically.
1783 1783
1784For information on how to create NPM packages, see the 1784For information on how to create NPM packages, see the
1785":ref:`dev-manual/common-tasks:creating node package manager (npm) packages`" 1785":ref:`dev-manual/packages:creating node package manager (npm) packages`"
1786section in the Yocto Project Development Tasks Manual. 1786section in the Yocto Project Development Tasks Manual.
1787 1787
1788.. _ref-classes-oelint: 1788.. _ref-classes-oelint:
@@ -1958,7 +1958,7 @@ If you take the optional step to set up a repository (package feed) on
1958the development host that can be used by DNF, you can install packages 1958the development host that can be used by DNF, you can install packages
1959from the feed while you are running the image on the target (i.e. 1959from the feed while you are running the image on the target (i.e.
1960runtime installation of packages). For more information, see the 1960runtime installation of packages). For more information, see the
1961":ref:`dev-manual/common-tasks:using runtime package management`" 1961":ref:`dev-manual/packages:using runtime package management`"
1962section in the Yocto Project Development Tasks Manual. 1962section in the Yocto Project Development Tasks Manual.
1963 1963
1964The package-specific class you choose can affect build-time performance 1964The package-specific class you choose can affect build-time performance
@@ -2077,7 +2077,7 @@ so forth). It is highly recommended that all package group recipes
2077inherit this class. 2077inherit this class.
2078 2078
2079For information on how to use this class, see the 2079For information on how to use this class, see the
2080":ref:`dev-manual/common-tasks:customizing images using custom package groups`" 2080":ref:`dev-manual/customizing-images:customizing images using custom package groups`"
2081section in the Yocto Project Development Tasks Manual. 2081section in the Yocto Project Development Tasks Manual.
2082 2082
2083Previously, this class was called the ``task`` class. 2083Previously, this class was called the ``task`` class.
@@ -2292,7 +2292,7 @@ The :ref:`primport <ref-classes-primport>` class provides functionality for impo
2292================== 2292==================
2293 2293
2294The :ref:`prserv <ref-classes-prserv>` class provides functionality for using a :ref:`PR 2294The :ref:`prserv <ref-classes-prserv>` class provides functionality for using a :ref:`PR
2295service <dev-manual/common-tasks:working with a pr service>` in order to 2295service <dev-manual/packages:working with a pr service>` in order to
2296automatically manage the incrementing of the :term:`PR` 2296automatically manage the incrementing of the :term:`PR`
2297variable for each recipe. 2297variable for each recipe.
2298 2298
@@ -2312,7 +2312,7 @@ runtime tests for recipes that build software that provides these tests.
2312This class is intended to be inherited by individual recipes. However, 2312This class is intended to be inherited by individual recipes. However,
2313the class' functionality is largely disabled unless "ptest" appears in 2313the class' functionality is largely disabled unless "ptest" appears in
2314:term:`DISTRO_FEATURES`. See the 2314:term:`DISTRO_FEATURES`. See the
2315":ref:`dev-manual/common-tasks:testing packages with ptest`" 2315":ref:`dev-manual/packages:testing packages with ptest`"
2316section in the Yocto Project Development Tasks Manual for more information 2316section in the Yocto Project Development Tasks Manual for more information
2317on ptest. 2317on ptest.
2318 2318
@@ -2325,7 +2325,7 @@ Enables package tests (ptests) specifically for GNOME packages, which
2325have tests intended to be executed with ``gnome-desktop-testing``. 2325have tests intended to be executed with ``gnome-desktop-testing``.
2326 2326
2327For information on setting up and running ptests, see the 2327For information on setting up and running ptests, see the
2328":ref:`dev-manual/common-tasks:testing packages with ptest`" 2328":ref:`dev-manual/packages:testing packages with ptest`"
2329section in the Yocto Project Development Tasks Manual. 2329section in the Yocto Project Development Tasks Manual.
2330 2330
2331.. _ref-classes-python3-dir: 2331.. _ref-classes-python3-dir:
@@ -2413,7 +2413,7 @@ override the removal by setting ``REMOVE_LIBTOOL_LA`` to "0" as follows::
2413======================== 2413========================
2414 2414
2415The :ref:`report-error <ref-classes-report-error>` class supports enabling the :ref:`error reporting 2415The :ref:`report-error <ref-classes-report-error>` class supports enabling the :ref:`error reporting
2416tool <dev-manual/common-tasks:using the error reporting tool>`", 2416tool <dev-manual/error-reporting-tool:using the error reporting tool>`",
2417which allows you to submit build error information to a central database. 2417which allows you to submit build error information to a central database.
2418 2418
2419The class collects debug information for recipe, recipe version, task, 2419The class collects debug information for recipe, recipe version, task,
@@ -2810,7 +2810,7 @@ unless you have set
2810:term:`SYSTEMD_AUTO_ENABLE` to "disable". 2810:term:`SYSTEMD_AUTO_ENABLE` to "disable".
2811 2811
2812For more information on :ref:`systemd <ref-classes-systemd>`, see the 2812For more information on :ref:`systemd <ref-classes-systemd>`, see the
2813":ref:`dev-manual/common-tasks:selecting an initialization manager`" 2813":ref:`dev-manual/init-manager:selecting an initialization manager`"
2814section in the Yocto Project Development Tasks Manual. 2814section in the Yocto Project Development Tasks Manual.
2815 2815
2816.. _ref-classes-systemd-boot: 2816.. _ref-classes-systemd-boot:
@@ -2885,7 +2885,7 @@ after it is built, you can set :term:`TESTIMAGE_AUTO`::
2885 TESTIMAGE_AUTO = "1" 2885 TESTIMAGE_AUTO = "1"
2886 2886
2887For information on how to enable, run, and create new tests, see the 2887For information on how to enable, run, and create new tests, see the
2888":ref:`dev-manual/common-tasks:performing automated runtime testing`" 2888":ref:`dev-manual/runtime-testing:performing automated runtime testing`"
2889section in the Yocto Project Development Tasks Manual. 2889section in the Yocto Project Development Tasks Manual.
2890 2890
2891.. _ref-classes-testsdk: 2891.. _ref-classes-testsdk:
diff --git a/documentation/ref-manual/devtool-reference.rst b/documentation/ref-manual/devtool-reference.rst
index 997ec0351c..c2e1c2c29b 100644
--- a/documentation/ref-manual/devtool-reference.rst
+++ b/documentation/ref-manual/devtool-reference.rst
@@ -410,7 +410,7 @@ Upgrading a Recipe
410As software matures, upstream recipes are upgraded to newer versions. As 410As software matures, upstream recipes are upgraded to newer versions. As
411a developer, you need to keep your local recipes up-to-date with the 411a developer, you need to keep your local recipes up-to-date with the
412upstream version releases. There are several ways of upgrading recipes. 412upstream version releases. There are several ways of upgrading recipes.
413You can read about them in the ":ref:`dev-manual/common-tasks:upgrading recipes`" 413You can read about them in the ":ref:`dev-manual/upgrading-recipes:upgrading recipes`"
414section of the Yocto Project Development Tasks Manual. This section 414section of the Yocto Project Development Tasks Manual. This section
415overviews the ``devtool upgrade`` command. 415overviews the ``devtool upgrade`` command.
416 416
@@ -438,7 +438,7 @@ You can read more on the ``devtool upgrade`` workflow in the
438":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`" 438":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
439section in the Yocto Project Application Development and the Extensible 439section in the Yocto Project Application Development and the Extensible
440Software Development Kit (eSDK) manual. You can also see an example of 440Software Development Kit (eSDK) manual. You can also see an example of
441how to use ``devtool upgrade`` in the ":ref:`dev-manual/common-tasks:using \`\`devtool upgrade\`\``" 441how to use ``devtool upgrade`` in the ":ref:`dev-manual/upgrading-recipes:using \`\`devtool upgrade\`\``"
442section in the Yocto Project Development Tasks Manual. 442section in the Yocto Project Development Tasks Manual.
443 443
444.. _devtool-resetting-a-recipe: 444.. _devtool-resetting-a-recipe:
diff --git a/documentation/ref-manual/faq.rst b/documentation/ref-manual/faq.rst
index d35ab78bff..8f38808a78 100644
--- a/documentation/ref-manual/faq.rst
+++ b/documentation/ref-manual/faq.rst
@@ -291,7 +291,7 @@ How do I make the Yocto Project support my board?
291Support for an additional board is added by creating a Board 291Support for an additional board is added by creating a Board
292Support Package (BSP) layer for it. For more information on how to 292Support Package (BSP) layer for it. For more information on how to
293create a BSP layer, see the 293create a BSP layer, see the
294":ref:`dev-manual/common-tasks:understanding and creating layers`" 294":ref:`dev-manual/layers:understanding and creating layers`"
295section in the Yocto Project Development Tasks Manual and the 295section in the Yocto Project Development Tasks Manual and the
296:doc:`/bsp-guide/index`. 296:doc:`/bsp-guide/index`.
297 297
@@ -303,7 +303,7 @@ How do I make the Yocto Project support my package?
303 303
304To add a package, you need to create a BitBake recipe. For 304To add a package, you need to create a BitBake recipe. For
305information on how to create a BitBake recipe, see the 305information on how to create a BitBake recipe, see the
306":ref:`dev-manual/common-tasks:writing a new recipe`" 306":ref:`dev-manual/new-recipe:writing a new recipe`"
307section in the Yocto Project Development Tasks Manual. 307section in the Yocto Project Development Tasks Manual.
308 308
309What do I need to ship for license compliance? 309What do I need to ship for license compliance?
@@ -320,7 +320,7 @@ configured and built.
320You can find more information on licensing in the 320You can find more information on licensing in the
321":ref:`overview-manual/development-environment:licensing`" 321":ref:`overview-manual/development-environment:licensing`"
322section in the Yocto Project Overview and Concepts Manual and also in the 322section in the Yocto Project Overview and Concepts Manual and also in the
323":ref:`dev-manual/common-tasks:maintaining open source license compliance during your product's lifecycle`" 323":ref:`dev-manual/licenses:maintaining open source license compliance during your product's lifecycle`"
324section in the Yocto Project Development Tasks Manual. 324section in the Yocto Project Development Tasks Manual.
325 325
326Do I have to make a full reflash after recompiling one package? 326Do I have to make a full reflash after recompiling one package?
diff --git a/documentation/ref-manual/features.rst b/documentation/ref-manual/features.rst
index 71d3c5e2aa..0229747650 100644
--- a/documentation/ref-manual/features.rst
+++ b/documentation/ref-manual/features.rst
@@ -143,7 +143,7 @@ metadata, as extra layers can define their own:
143- *cramfs:* Include CramFS support. 143- *cramfs:* Include CramFS support.
144 144
145- *debuginfod:* Include support for getting ELF debugging information through 145- *debuginfod:* Include support for getting ELF debugging information through
146 a :ref:`debuginfod <dev-manual/common-tasks:using the debuginfod server method>` 146 a :ref:`debuginfod <dev-manual/debugging:using the debuginfod server method>`
147 server. 147 server.
148 148
149- *directfb:* Include DirectFB support. 149- *directfb:* Include DirectFB support.
@@ -202,7 +202,7 @@ metadata, as extra layers can define their own:
202 202
203- *ptest:* Enables building the package tests where supported by 203- *ptest:* Enables building the package tests where supported by
204 individual recipes. For more information on package tests, see the 204 individual recipes. For more information on package tests, see the
205 ":ref:`dev-manual/common-tasks:testing packages with ptest`" section 205 ":ref:`dev-manual/packages:testing packages with ptest`" section
206 in the Yocto Project Development Tasks Manual. 206 in the Yocto Project Development Tasks Manual.
207 207
208- *pulseaudio:* Include support for 208- *pulseaudio:* Include support for
@@ -325,7 +325,7 @@ Here are the image features available for all images:
325 325
326- *read-only-rootfs:* Creates an image whose root filesystem is 326- *read-only-rootfs:* Creates an image whose root filesystem is
327 read-only. See the 327 read-only. See the
328 ":ref:`dev-manual/common-tasks:creating a read-only root filesystem`" 328 ":ref:`dev-manual/read-only-rootfs:creating a read-only root filesystem`"
329 section in the Yocto Project Development Tasks Manual for more 329 section in the Yocto Project Development Tasks Manual for more
330 information. 330 information.
331 331
@@ -394,7 +394,7 @@ these valid features is as follows:
394 394
395- *tools-debug:* Installs debugging tools such as ``strace`` and 395- *tools-debug:* Installs debugging tools such as ``strace`` and
396 ``gdb``. For information on GDB, see the 396 ``gdb``. For information on GDB, see the
397 ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) remotely`" section 397 ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) remotely`" section
398 in the Yocto Project Development Tasks Manual. For information on 398 in the Yocto Project Development Tasks Manual. For information on
399 tracing and profiling, see the :doc:`/profile-manual/index`. 399 tracing and profiling, see the :doc:`/profile-manual/index`.
400 400
diff --git a/documentation/ref-manual/images.rst b/documentation/ref-manual/images.rst
index 4b771d0601..d52b39f3cf 100644
--- a/documentation/ref-manual/images.rst
+++ b/documentation/ref-manual/images.rst
@@ -117,7 +117,7 @@ Following is a list of supported recipes:
117 deployed to a separate partition so that you can boot into it and use 117 deployed to a separate partition so that you can boot into it and use
118 it to deploy a second image to be tested. You can find more 118 it to deploy a second image to be tested. You can find more
119 information about runtime testing in the 119 information about runtime testing in the
120 ":ref:`dev-manual/common-tasks:performing automated runtime testing`" 120 ":ref:`dev-manual/runtime-testing:performing automated runtime testing`"
121 section in the Yocto Project Development Tasks Manual. 121 section in the Yocto Project Development Tasks Manual.
122 122
123- ``core-image-testmaster-initramfs``: A RAM-based Initial Root 123- ``core-image-testmaster-initramfs``: A RAM-based Initial Root
@@ -127,7 +127,7 @@ Following is a list of supported recipes:
127- ``core-image-weston``: A very basic Wayland image with a terminal. 127- ``core-image-weston``: A very basic Wayland image with a terminal.
128 This image provides the Wayland protocol libraries and the reference 128 This image provides the Wayland protocol libraries and the reference
129 Weston compositor. For more information, see the 129 Weston compositor. For more information, see the
130 ":ref:`dev-manual/common-tasks:using wayland and weston`" 130 ":ref:`dev-manual/wayland:using wayland and weston`"
131 section in the Yocto Project Development Tasks Manual. 131 section in the Yocto Project Development Tasks Manual.
132 132
133- ``core-image-x11``: A very basic X11 image with a terminal. 133- ``core-image-x11``: A very basic X11 image with a terminal.
diff --git a/documentation/ref-manual/kickstart.rst b/documentation/ref-manual/kickstart.rst
index 11bc373b3c..297887805c 100644
--- a/documentation/ref-manual/kickstart.rst
+++ b/documentation/ref-manual/kickstart.rst
@@ -82,7 +82,7 @@ the ``part`` and ``partition`` commands:
82 source of the data that populates the partition. The most common 82 source of the data that populates the partition. The most common
83 value for this option is "rootfs", but you can use any value that 83 value for this option is "rootfs", but you can use any value that
84 maps to a valid source plugin. For information on the source plugins, 84 maps to a valid source plugin. For information on the source plugins,
85 see the ":ref:`dev-manual/common-tasks:using the wic plugin interface`" 85 see the ":ref:`dev-manual/wic:using the wic plugin interface`"
86 section in the Yocto Project Development Tasks Manual. 86 section in the Yocto Project Development Tasks Manual.
87 87
88 If you use ``--source rootfs``, Wic creates a partition as large as 88 If you use ``--source rootfs``, Wic creates a partition as large as
diff --git a/documentation/ref-manual/release-process.rst b/documentation/ref-manual/release-process.rst
index c36fa557d8..19e8040638 100644
--- a/documentation/ref-manual/release-process.rst
+++ b/documentation/ref-manual/release-process.rst
@@ -107,7 +107,7 @@ Additionally, because the test strategies are visible to you as a
107developer, you can validate your projects. This section overviews the 107developer, you can validate your projects. This section overviews the
108available test infrastructure used in the Yocto Project. For information 108available test infrastructure used in the Yocto Project. For information
109on how to run available tests on your projects, see the 109on how to run available tests on your projects, see the
110":ref:`dev-manual/common-tasks:performing automated runtime testing`" 110":ref:`dev-manual/runtime-testing:performing automated runtime testing`"
111section in the Yocto Project Development Tasks Manual. 111section in the Yocto Project Development Tasks Manual.
112 112
113The QA/testing infrastructure is woven into the project to the point 113The QA/testing infrastructure is woven into the project to the point
@@ -134,7 +134,7 @@ consists of the following pieces:
134 operation and functions. However, the test can also use the IP 134 operation and functions. However, the test can also use the IP
135 address of a machine to test. 135 address of a machine to test.
136 136
137- :ref:`ptest <dev-manual/common-tasks:testing packages with ptest>`: 137- :ref:`ptest <dev-manual/packages:testing packages with ptest>`:
138 Runs tests against packages produced during the build for a given 138 Runs tests against packages produced during the build for a given
139 piece of software. The test allows the packages to be run within a 139 piece of software. The test allows the packages to be run within a
140 target image. 140 target image.
diff --git a/documentation/ref-manual/resources.rst b/documentation/ref-manual/resources.rst
index 292a9d6f61..c737cba912 100644
--- a/documentation/ref-manual/resources.rst
+++ b/documentation/ref-manual/resources.rst
@@ -23,7 +23,7 @@ The Yocto Project gladly accepts contributions. You can submit changes
23to the project either by creating and sending pull requests, or by 23to the project either by creating and sending pull requests, or by
24submitting patches through email. For information on how to do both as 24submitting patches through email. For information on how to do both as
25well as information on how to identify the maintainer for each area of 25well as information on how to identify the maintainer for each area of
26code, see the ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" section in the 26code, see the ":ref:`dev-manual/changes:submitting a change to the yocto project`" section in the
27Yocto Project Development Tasks Manual. 27Yocto Project Development Tasks Manual.
28 28
29.. _resources-bugtracker: 29.. _resources-bugtracker:
@@ -46,7 +46,7 @@ your expectations).
46For a general procedure and guidelines on how to use Bugzilla to submit a bug 46For a general procedure and guidelines on how to use Bugzilla to submit a bug
47against the Yocto Project, see the following: 47against the Yocto Project, see the following:
48 48
49- The ":ref:`dev-manual/common-tasks:submitting a defect against the yocto project`" 49- The ":ref:`dev-manual/changes:submitting a defect against the yocto project`"
50 section in the Yocto Project Development Tasks Manual. 50 section in the Yocto Project Development Tasks Manual.
51 51
52- The Yocto Project :yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>` 52- The Yocto Project :yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
diff --git a/documentation/ref-manual/structure.rst b/documentation/ref-manual/structure.rst
index 5436d94e1d..f3a52a19f3 100644
--- a/documentation/ref-manual/structure.rst
+++ b/documentation/ref-manual/structure.rst
@@ -177,7 +177,7 @@ within the :term:`Source Directory`. If you design a
177custom distribution, you can include your own version of this 177custom distribution, you can include your own version of this
178configuration file to mention the targets defined by your distribution. 178configuration file to mention the targets defined by your distribution.
179See the 179See the
180":ref:`dev-manual/common-tasks:creating a custom template configuration directory`" 180":ref:`dev-manual/custom-template-configuration-directory:creating a custom template configuration directory`"
181section in the Yocto Project Development Tasks Manual for more 181section in the Yocto Project Development Tasks Manual for more
182information. 182information.
183 183
@@ -194,7 +194,7 @@ your choice. For example, the following command creates a
194The OpenEmbedded build system uses the template configuration files, which 194The OpenEmbedded build system uses the template configuration files, which
195are found by default in the ``meta-poky/conf/templates/default`` directory in the Source 195are found by default in the ``meta-poky/conf/templates/default`` directory in the Source
196Directory. See the 196Directory. See the
197":ref:`dev-manual/common-tasks:creating a custom template configuration directory`" 197":ref:`dev-manual/custom-template-configuration-directory:creating a custom template configuration directory`"
198section in the Yocto Project Development Tasks Manual for more 198section in the Yocto Project Development Tasks Manual for more
199information. 199information.
200 200
@@ -236,7 +236,7 @@ The OpenEmbedded build system creates this directory when you enable
236build history via the :ref:`buildhistory <ref-classes-buildhistory>` class file. The directory 236build history via the :ref:`buildhistory <ref-classes-buildhistory>` class file. The directory
237organizes build information into image, packages, and SDK 237organizes build information into image, packages, and SDK
238subdirectories. For information on the build history feature, see the 238subdirectories. For information on the build history feature, see the
239":ref:`dev-manual/common-tasks:maintaining build output quality`" 239":ref:`dev-manual/build-quality:maintaining build output quality`"
240section in the Yocto Project Development Tasks Manual. 240section in the Yocto Project Development Tasks Manual.
241 241
242.. _structure-build-cache: 242.. _structure-build-cache:
@@ -303,7 +303,7 @@ file, it uses ``sed`` to substitute final
303---------------------------- 303----------------------------
304 304
305This configuration file defines 305This configuration file defines
306:ref:`layers <dev-manual/common-tasks:understanding and creating layers>`, 306:ref:`layers <dev-manual/layers:understanding and creating layers>`,
307which are directory trees, traversed (or walked) by BitBake. The 307which are directory trees, traversed (or walked) by BitBake. The
308``bblayers.conf`` file uses the :term:`BBLAYERS` 308``bblayers.conf`` file uses the :term:`BBLAYERS`
309variable to list the layers BitBake tries to find. 309variable to list the layers BitBake tries to find.
@@ -441,7 +441,7 @@ directory contains sub-directories for ``bash``, ``busybox``, and
441``glibc`` (among others) that in turn contain appropriate ``COPYING`` 441``glibc`` (among others) that in turn contain appropriate ``COPYING``
442license files with other licensing information. For information on 442license files with other licensing information. For information on
443licensing, see the 443licensing, see the
444":ref:`dev-manual/common-tasks:maintaining open source license compliance during your product's lifecycle`" 444":ref:`dev-manual/licenses:maintaining open source license compliance during your product's lifecycle`"
445section in the Yocto Project Development Tasks Manual. 445section in the Yocto Project Development Tasks Manual.
446 446
447.. _structure-build-tmp-deploy-images: 447.. _structure-build-tmp-deploy-images:
@@ -578,7 +578,7 @@ built within the Yocto Project. For this package, a work directory of
578``tmp/work/qemux86-poky-linux/linux-yocto/3.0+git1+<.....>``, referred 578``tmp/work/qemux86-poky-linux/linux-yocto/3.0+git1+<.....>``, referred
579to as the :term:`WORKDIR`, is created. Within this directory, the source is 579to as the :term:`WORKDIR`, is created. Within this directory, the source is
580unpacked to ``linux-qemux86-standard-build`` and then patched by Quilt. 580unpacked to ``linux-qemux86-standard-build`` and then patched by Quilt.
581(See the ":ref:`dev-manual/common-tasks:using quilt in your workflow`" section in 581(See the ":ref:`dev-manual/quilt:using quilt in your workflow`" section in
582the Yocto Project Development Tasks Manual for more information.) Within 582the Yocto Project Development Tasks Manual for more information.) Within
583the ``linux-qemux86-standard-build`` directory, standard Quilt 583the ``linux-qemux86-standard-build`` directory, standard Quilt
584directories ``linux-3.0/patches`` and ``linux-3.0/.pc`` are created, and 584directories ``linux-3.0/patches`` and ``linux-3.0/.pc`` are created, and
diff --git a/documentation/ref-manual/system-requirements.rst b/documentation/ref-manual/system-requirements.rst
index 1502633816..8dab359b69 100644
--- a/documentation/ref-manual/system-requirements.rst
+++ b/documentation/ref-manual/system-requirements.rst
@@ -86,7 +86,7 @@ distributions:
86 interested in hearing about your experience. For information on 86 interested in hearing about your experience. For information on
87 how to submit a bug, see the Yocto Project 87 how to submit a bug, see the Yocto Project
88 :yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>` 88 :yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
89 and the ":ref:`dev-manual/common-tasks:submitting a defect against the yocto project`" 89 and the ":ref:`dev-manual/changes:submitting a defect against the yocto project`"
90 section in the Yocto Project Development Tasks Manual. 90 section in the Yocto Project Development Tasks Manual.
91 91
92 92
diff --git a/documentation/ref-manual/tasks.rst b/documentation/ref-manual/tasks.rst
index b17c09cdc4..e626095b20 100644
--- a/documentation/ref-manual/tasks.rst
+++ b/documentation/ref-manual/tasks.rst
@@ -343,7 +343,7 @@ while ``file2.patch`` would not be applied.
343You can find out more about the patching process in the 343You can find out more about the patching process in the
344":ref:`overview-manual/concepts:patching`" section in 344":ref:`overview-manual/concepts:patching`" section in
345the Yocto Project Overview and Concepts Manual and the 345the Yocto Project Overview and Concepts Manual and the
346":ref:`dev-manual/common-tasks:patching code`" section in the 346":ref:`dev-manual/new-recipe:patching code`" section in the
347Yocto Project Development Tasks Manual. 347Yocto Project Development Tasks Manual.
348 348
349.. _ref-tasks-populate_lic: 349.. _ref-tasks-populate_lic:
@@ -522,7 +522,7 @@ scratch is guaranteed.
522Starts a shell in which an interactive Python interpreter allows you to 522Starts a shell in which an interactive Python interpreter allows you to
523interact with the BitBake build environment. From within this shell, you 523interact with the BitBake build environment. From within this shell, you
524can directly examine and set bits from the data store and execute 524can directly examine and set bits from the data store and execute
525functions as if within the BitBake environment. See the ":ref:`dev-manual/common-tasks:using a Python development shell`" section in 525functions as if within the BitBake environment. See the ":ref:`dev-manual/python-development-shell:using a Python development shell`" section in
526the Yocto Project Development Tasks Manual for more information about 526the Yocto Project Development Tasks Manual for more information about
527using ``pydevshell``. 527using ``pydevshell``.
528 528
@@ -532,7 +532,7 @@ using ``pydevshell``.
532--------------- 532---------------
533 533
534Starts a shell whose environment is set up for development, debugging, 534Starts a shell whose environment is set up for development, debugging,
535or both. See the ":ref:`dev-manual/common-tasks:using a development shell`" section in the 535or both. See the ":ref:`dev-manual/development-shell:using a development shell`" section in the
536Yocto Project Development Tasks Manual for more information about using 536Yocto Project Development Tasks Manual for more information about using
537``devshell``. 537``devshell``.
538 538
@@ -595,7 +595,7 @@ information on how the root filesystem is created.
595 595
596Boots an image and performs runtime tests within the image. For 596Boots an image and performs runtime tests within the image. For
597information on automatically testing images, see the 597information on automatically testing images, see the
598":ref:`dev-manual/common-tasks:performing automated runtime testing`" 598":ref:`dev-manual/runtime-testing:performing automated runtime testing`"
599section in the Yocto Project Development Tasks Manual. 599section in the Yocto Project Development Tasks Manual.
600 600
601.. _ref-tasks-testimage_auto: 601.. _ref-tasks-testimage_auto:
@@ -608,7 +608,7 @@ after it has been built. This task is enabled when you set
608:term:`TESTIMAGE_AUTO` equal to "1". 608:term:`TESTIMAGE_AUTO` equal to "1".
609 609
610For information on automatically testing images, see the 610For information on automatically testing images, see the
611":ref:`dev-manual/common-tasks:performing automated runtime testing`" 611":ref:`dev-manual/runtime-testing:performing automated runtime testing`"
612section in the Yocto Project Development Tasks Manual. 612section in the Yocto Project Development Tasks Manual.
613 613
614Kernel-Related Tasks 614Kernel-Related Tasks
diff --git a/documentation/ref-manual/terms.rst b/documentation/ref-manual/terms.rst
index 51f6e79200..4081dbebd7 100644
--- a/documentation/ref-manual/terms.rst
+++ b/documentation/ref-manual/terms.rst
@@ -21,7 +21,7 @@ universal, the list includes them just in case:
21 21
22 Information in append files extends or overrides the information in the 22 Information in append files extends or overrides the information in the
23 similarly-named recipe file. For an example of an append file in use, see 23 similarly-named recipe file. For an example of an append file in use, see
24 the ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`" 24 the ":ref:`dev-manual/layers:appending other layers metadata with your layer`"
25 section in the Yocto Project Development Tasks Manual. 25 section in the Yocto Project Development Tasks Manual.
26 26
27 When you name an append file, you can use the "``%``" wildcard character 27 When you name an append file, you can use the "``%``" wildcard character
@@ -203,7 +203,7 @@ universal, the list includes them just in case:
203 ":ref:`overview-manual/yp-intro:The Yocto Project Layer 203 ":ref:`overview-manual/yp-intro:The Yocto Project Layer
204 Model`" section in the Yocto Project Overview and Concepts Manual. For 204 Model`" section in the Yocto Project Overview and Concepts Manual. For
205 more detailed information on layers, see the 205 more detailed information on layers, see the
206 ":ref:`dev-manual/common-tasks:Understanding and Creating 206 ":ref:`dev-manual/layers:Understanding and Creating
207 Layers`" section in the Yocto Project Development Tasks Manual. For a 207 Layers`" section in the Yocto Project Development Tasks Manual. For a
208 discussion specifically on BSP Layers, see the ":ref:`bsp-guide/bsp:BSP 208 discussion specifically on BSP Layers, see the ":ref:`bsp-guide/bsp:BSP
209 Layers`" section in the Yocto Project Board Support Packages (BSP) 209 Layers`" section in the Yocto Project Board Support Packages (BSP)
@@ -335,7 +335,7 @@ universal, the list includes them just in case:
335 335
336 The OpenEmbedded Build System can generate such documentation for your 336 The OpenEmbedded Build System can generate such documentation for your
337 project, in :term:`SPDX` format, based on all the metadata it used to 337 project, in :term:`SPDX` format, based on all the metadata it used to
338 build the software images. See the ":ref:`dev-manual/common-tasks:creating 338 build the software images. See the ":ref:`dev-manual/sbom:creating
339 a software bill of materials`" section of the Development Tasks manual. 339 a software bill of materials`" section of the Development Tasks manual.
340 340
341 :term:`Source Directory` 341 :term:`Source Directory`
@@ -406,7 +406,7 @@ universal, the list includes them just in case:
406 provide an :term:`SBOM` associated to each a software image. 406 provide an :term:`SBOM` associated to each a software image.
407 407
408 For details, see Wikipedia's :wikipedia:`SPDX page <Software_Package_Data_Exchange>` 408 For details, see Wikipedia's :wikipedia:`SPDX page <Software_Package_Data_Exchange>`
409 and the ":ref:`dev-manual/common-tasks:creating a software bill of materials`" 409 and the ":ref:`dev-manual/sbom:creating a software bill of materials`"
410 section of the Development Tasks manual. 410 section of the Development Tasks manual.
411 411
412 :term:`Sysroot` 412 :term:`Sysroot`
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 8b54420d95..6bfacece07 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -221,7 +221,7 @@ system and gives an overview of their function and contents.
221 :term:`PV` in your recipe so that it does contain ``${SRCPV}``. 221 :term:`PV` in your recipe so that it does contain ``${SRCPV}``.
222 222
223 For more information see the 223 For more information see the
224 ":ref:`dev-manual/common-tasks:automatically incrementing a package version number`" 224 ":ref:`dev-manual/packages:automatically incrementing a package version number`"
225 section in the Yocto Project Development Tasks Manual. 225 section in the Yocto Project Development Tasks Manual.
226 226
227 :term:`AUTO_SYSLINUXMENU` 227 :term:`AUTO_SYSLINUXMENU`
@@ -237,7 +237,7 @@ system and gives an overview of their function and contents.
237 The list simply presents the tunes that are available. Not all tunes 237 The list simply presents the tunes that are available. Not all tunes
238 may be compatible with a particular machine configuration, or with 238 may be compatible with a particular machine configuration, or with
239 each other in a 239 each other in a
240 :ref:`Multilib <dev-manual/common-tasks:combining multiple versions of library files into one image>` 240 :ref:`Multilib <dev-manual/libraries:combining multiple versions of library files into one image>`
241 configuration. 241 configuration.
242 242
243 To add a tune to the list, be sure to append it with spaces using the 243 To add a tune to the list, be sure to append it with spaces using the
@@ -302,7 +302,7 @@ system and gives an overview of their function and contents.
302 :term:`BASE_LIB` 302 :term:`BASE_LIB`
303 The library directory name for the CPU or Application Binary 303 The library directory name for the CPU or Application Binary
304 Interface (ABI) tune. The :term:`BASE_LIB` applies only in the Multilib 304 Interface (ABI) tune. The :term:`BASE_LIB` applies only in the Multilib
305 context. See the ":ref:`dev-manual/common-tasks:combining multiple versions of library files into one image`" 305 context. See the ":ref:`dev-manual/libraries:combining multiple versions of library files into one image`"
306 section in the Yocto Project Development Tasks Manual for information 306 section in the Yocto Project Development Tasks Manual for information
307 on Multilib. 307 on Multilib.
308 308
@@ -526,7 +526,7 @@ system and gives an overview of their function and contents.
526 is not set higher than "20". 526 is not set higher than "20".
527 527
528 For more information on speeding up builds, see the 528 For more information on speeding up builds, see the
529 ":ref:`dev-manual/common-tasks:speeding up a build`" 529 ":ref:`dev-manual/speeding-up-build:speeding up a build`"
530 section in the Yocto Project Development Tasks Manual. 530 section in the Yocto Project Development Tasks Manual.
531 531
532 On the other hand, if your goal is to limit the amount of system 532 On the other hand, if your goal is to limit the amount of system
@@ -751,7 +751,7 @@ system and gives an overview of their function and contents.
751 751
752 For information on how to use :term:`BBMULTICONFIG` in an environment 752 For information on how to use :term:`BBMULTICONFIG` in an environment
753 that supports building targets with multiple configurations, see the 753 that supports building targets with multiple configurations, see the
754 ":ref:`dev-manual/common-tasks:building images for multiple targets using multiple configurations`" 754 ":ref:`dev-manual/building:building images for multiple targets using multiple configurations`"
755 section in the Yocto Project Development Tasks Manual. 755 section in the Yocto Project Development Tasks Manual.
756 756
757 :term:`BBSERVER` 757 :term:`BBSERVER`
@@ -980,7 +980,7 @@ system and gives an overview of their function and contents.
980 When inheriting the :ref:`buildhistory <ref-classes-buildhistory>` 980 When inheriting the :ref:`buildhistory <ref-classes-buildhistory>`
981 class, this variable specifies the build history features to be 981 class, this variable specifies the build history features to be
982 enabled. For more information on how build history works, see the 982 enabled. For more information on how build history works, see the
983 ":ref:`dev-manual/common-tasks:maintaining build output quality`" 983 ":ref:`dev-manual/build-quality:maintaining build output quality`"
984 section in the Yocto Project Development Tasks Manual. 984 section in the Yocto Project Development Tasks Manual.
985 985
986 You can specify these features in the form of a space-separated list: 986 You can specify these features in the form of a space-separated list:
@@ -1287,7 +1287,7 @@ system and gives an overview of their function and contents.
1287 will be the aggregate of all of them. 1287 will be the aggregate of all of them.
1288 1288
1289 For information on creating an :term:`Initramfs`, see the 1289 For information on creating an :term:`Initramfs`, see the
1290 ":ref:`dev-manual/common-tasks:building an initial ram filesystem (Initramfs) image`" section 1290 ":ref:`dev-manual/building:building an initial ram filesystem (Initramfs) image`" section
1291 in the Yocto Project Development Tasks Manual. 1291 in the Yocto Project Development Tasks Manual.
1292 1292
1293 :term:`CONFIG_SITE` 1293 :term:`CONFIG_SITE`
@@ -1335,7 +1335,7 @@ system and gives an overview of their function and contents.
1335 newly installed packages to an image, which might be most suitable for 1335 newly installed packages to an image, which might be most suitable for
1336 read-only filesystems that cannot be upgraded. See the 1336 read-only filesystems that cannot be upgraded. See the
1337 :term:`LICENSE_CREATE_PACKAGE` variable for additional information. 1337 :term:`LICENSE_CREATE_PACKAGE` variable for additional information.
1338 You can also reference the ":ref:`dev-manual/common-tasks:providing license text`" 1338 You can also reference the ":ref:`dev-manual/licenses:providing license text`"
1339 section in the Yocto Project Development Tasks Manual for 1339 section in the Yocto Project Development Tasks Manual for
1340 information on providing license text. 1340 information on providing license text.
1341 1341
@@ -1351,7 +1351,7 @@ system and gives an overview of their function and contents.
1351 newly installed packages to an image, which might be most suitable for 1351 newly installed packages to an image, which might be most suitable for
1352 read-only filesystems that cannot be upgraded. See the 1352 read-only filesystems that cannot be upgraded. See the
1353 :term:`LICENSE_CREATE_PACKAGE` variable for additional information. 1353 :term:`LICENSE_CREATE_PACKAGE` variable for additional information.
1354 You can also reference the ":ref:`dev-manual/common-tasks:providing license text`" 1354 You can also reference the ":ref:`dev-manual/licenses:providing license text`"
1355 section in the Yocto Project Development Tasks Manual for 1355 section in the Yocto Project Development Tasks Manual for
1356 information on providing license text. 1356 information on providing license text.
1357 1357
@@ -2118,7 +2118,7 @@ system and gives an overview of their function and contents.
2118 When used with the :ref:`report-error <ref-classes-report-error>` 2118 When used with the :ref:`report-error <ref-classes-report-error>`
2119 class, specifies the path used for storing the debug files created by 2119 class, specifies the path used for storing the debug files created by
2120 the :ref:`error reporting 2120 the :ref:`error reporting
2121 tool <dev-manual/common-tasks:using the error reporting tool>`, which 2121 tool <dev-manual/error-reporting-tool:using the error reporting tool>`, which
2122 allows you to submit build errors you encounter to a central 2122 allows you to submit build errors you encounter to a central
2123 database. By default, the value of this variable is 2123 database. By default, the value of this variable is
2124 ``${``\ :term:`LOG_DIR`\ ``}/error-report``. 2124 ``${``\ :term:`LOG_DIR`\ ``}/error-report``.
@@ -2276,7 +2276,7 @@ system and gives an overview of their function and contents.
2276 2276
2277 See the ":ref:`ref-classes-externalsrc`" section for details. You 2277 See the ":ref:`ref-classes-externalsrc`" section for details. You
2278 can also find information on how to use this variable in the 2278 can also find information on how to use this variable in the
2279 ":ref:`dev-manual/common-tasks:building software from an external source`" 2279 ":ref:`dev-manual/building:building software from an external source`"
2280 section in the Yocto Project Development Tasks Manual. 2280 section in the Yocto Project Development Tasks Manual.
2281 2281
2282 :term:`EXTERNALSRC_BUILD` 2282 :term:`EXTERNALSRC_BUILD`
@@ -2289,7 +2289,7 @@ system and gives an overview of their function and contents.
2289 2289
2290 See the ":ref:`ref-classes-externalsrc`" section for details. You 2290 See the ":ref:`ref-classes-externalsrc`" section for details. You
2291 can also find information on how to use this variable in the 2291 can also find information on how to use this variable in the
2292 ":ref:`dev-manual/common-tasks:building software from an external source`" 2292 ":ref:`dev-manual/building:building software from an external source`"
2293 section in the Yocto Project Development Tasks Manual. 2293 section in the Yocto Project Development Tasks Manual.
2294 2294
2295 :term:`EXTRA_AUTORECONF` 2295 :term:`EXTRA_AUTORECONF`
@@ -2326,7 +2326,7 @@ system and gives an overview of their function and contents.
2326 useful if you want to develop against the libraries in the image. 2326 useful if you want to develop against the libraries in the image.
2327 - "read-only-rootfs" --- creates an image whose root filesystem is 2327 - "read-only-rootfs" --- creates an image whose root filesystem is
2328 read-only. See the 2328 read-only. See the
2329 ":ref:`dev-manual/common-tasks:creating a read-only root filesystem`" 2329 ":ref:`dev-manual/read-only-rootfs:creating a read-only root filesystem`"
2330 section in the Yocto Project Development Tasks Manual for more 2330 section in the Yocto Project Development Tasks Manual for more
2331 information 2331 information
2332 - "tools-debug" --- adds debugging tools such as gdb and strace. 2332 - "tools-debug" --- adds debugging tools such as gdb and strace.
@@ -2339,7 +2339,7 @@ system and gives an overview of their function and contents.
2339 Project, see the ":ref:`ref-features-image`" section. 2339 Project, see the ":ref:`ref-features-image`" section.
2340 2340
2341 For an example that shows how to customize your image by using this 2341 For an example that shows how to customize your image by using this
2342 variable, see the ":ref:`dev-manual/common-tasks:customizing images using custom \`\`image_features\`\` and \`\`extra_image_features\`\``" 2342 variable, see the ":ref:`dev-manual/customizing-images:customizing images using custom \`\`image_features\`\` and \`\`extra_image_features\`\``"
2343 section in the Yocto Project Development Tasks Manual. 2343 section in the Yocto Project Development Tasks Manual.
2344 2344
2345 :term:`EXTRA_IMAGECMD` 2345 :term:`EXTRA_IMAGECMD`
@@ -2684,7 +2684,7 @@ system and gives an overview of their function and contents.
2684 You can find out more about the patching process in the 2684 You can find out more about the patching process in the
2685 ":ref:`overview-manual/concepts:patching`" section 2685 ":ref:`overview-manual/concepts:patching`" section
2686 in the Yocto Project Overview and Concepts Manual and the 2686 in the Yocto Project Overview and Concepts Manual and the
2687 ":ref:`dev-manual/common-tasks:patching code`" section in 2687 ":ref:`dev-manual/new-recipe:patching code`" section in
2688 the Yocto Project Development Tasks Manual. See the 2688 the Yocto Project Development Tasks Manual. See the
2689 :ref:`ref-tasks-patch` task as well. 2689 :ref:`ref-tasks-patch` task as well.
2690 2690
@@ -2819,7 +2819,7 @@ system and gives an overview of their function and contents.
2819 Allows to specify an extra search path for ``.so`` files 2819 Allows to specify an extra search path for ``.so`` files
2820 in GLib related recipes using GObject introspection, 2820 in GLib related recipes using GObject introspection,
2821 and which do not compile without this setting. 2821 and which do not compile without this setting.
2822 See the ":ref:`dev-manual/common-tasks:enabling gobject introspection support`" 2822 See the ":ref:`dev-manual/gobject-introspection:enabling gobject introspection support`"
2823 section for details. 2823 section for details.
2824 2824
2825 :term:`GITDIR` 2825 :term:`GITDIR`
@@ -3113,7 +3113,7 @@ system and gives an overview of their function and contents.
3113 the same files into a ``boot`` directory within the target partition. 3113 the same files into a ``boot`` directory within the target partition.
3114 3114
3115 You can find information on how to use the Wic tool in the 3115 You can find information on how to use the Wic tool in the
3116 ":ref:`dev-manual/common-tasks:creating partitioned images using wic`" 3116 ":ref:`dev-manual/wic:creating partitioned images using wic`"
3117 section of the Yocto Project Development Tasks Manual. Reference 3117 section of the Yocto Project Development Tasks Manual. Reference
3118 material for Wic is located in the 3118 material for Wic is located in the
3119 ":doc:`/ref-manual/kickstart`" chapter. 3119 ":doc:`/ref-manual/kickstart`" chapter.
@@ -3191,7 +3191,7 @@ system and gives an overview of their function and contents.
3191 the same files into a ``boot`` directory within the target partition. 3191 the same files into a ``boot`` directory within the target partition.
3192 3192
3193 You can find information on how to use the Wic tool in the 3193 You can find information on how to use the Wic tool in the
3194 ":ref:`dev-manual/common-tasks:creating partitioned images using wic`" 3194 ":ref:`dev-manual/wic:creating partitioned images using wic`"
3195 section of the Yocto Project Development Tasks Manual. Reference 3195 section of the Yocto Project Development Tasks Manual. Reference
3196 material for Wic is located in the 3196 material for Wic is located in the
3197 ":doc:`/ref-manual/kickstart`" chapter. 3197 ":doc:`/ref-manual/kickstart`" chapter.
@@ -3212,7 +3212,7 @@ system and gives an overview of their function and contents.
3212 the ":ref:`ref-features-image`" section. 3212 the ":ref:`ref-features-image`" section.
3213 3213
3214 For an example that shows how to customize your image by using this 3214 For an example that shows how to customize your image by using this
3215 variable, see the ":ref:`dev-manual/common-tasks:customizing images using custom \`\`image_features\`\` and \`\`extra_image_features\`\``" 3215 variable, see the ":ref:`dev-manual/customizing-images:customizing images using custom \`\`image_features\`\` and \`\`extra_image_features\`\``"
3216 section in the Yocto Project Development Tasks Manual. 3216 section in the Yocto Project Development Tasks Manual.
3217 3217
3218 :term:`IMAGE_FSTYPES` 3218 :term:`IMAGE_FSTYPES`
@@ -3268,7 +3268,7 @@ system and gives an overview of their function and contents.
3268 allows the initial RAM filesystem (:term:`Initramfs`) recipe to use a 3268 allows the initial RAM filesystem (:term:`Initramfs`) recipe to use a
3269 fixed set of packages and not be affected by :term:`IMAGE_INSTALL`. 3269 fixed set of packages and not be affected by :term:`IMAGE_INSTALL`.
3270 For information on creating an :term:`Initramfs`, see the 3270 For information on creating an :term:`Initramfs`, see the
3271 ":ref:`dev-manual/common-tasks:building an initial ram filesystem (Initramfs) image`" 3271 ":ref:`dev-manual/building:building an initial ram filesystem (Initramfs) image`"
3272 section in the Yocto Project Development Tasks Manual. 3272 section in the Yocto Project Development Tasks Manual.
3273 3273
3274 - Using :term:`IMAGE_INSTALL` with the 3274 - Using :term:`IMAGE_INSTALL` with the
@@ -3759,7 +3759,7 @@ system and gives an overview of their function and contents.
3759 or be included in the kernel binary. 3759 or be included in the kernel binary.
3760 3760
3761 For information on creating and using an :term:`Initramfs`, see the 3761 For information on creating and using an :term:`Initramfs`, see the
3762 ":ref:`dev-manual/common-tasks:building an initial ram filesystem (Initramfs) image`" 3762 ":ref:`dev-manual/building:building an initial ram filesystem (Initramfs) image`"
3763 section in the Yocto Project Development Tasks Manual. 3763 section in the Yocto Project Development Tasks Manual.
3764 3764
3765 :term:`INITRAMFS_DEPLOY_DIR_IMAGE` 3765 :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
@@ -3817,7 +3817,7 @@ system and gives an overview of their function and contents.
3817 :term:`INITRAMFS_IMAGE_BUNDLE` 3817 :term:`INITRAMFS_IMAGE_BUNDLE`
3818 variable, which allows the generated image to be bundled inside the 3818 variable, which allows the generated image to be bundled inside the
3819 kernel image. Additionally, for information on creating an :term:`Initramfs` 3819 kernel image. Additionally, for information on creating an :term:`Initramfs`
3820 image, see the ":ref:`dev-manual/common-tasks:building an initial ram filesystem (Initramfs) image`" section 3820 image, see the ":ref:`dev-manual/building:building an initial ram filesystem (Initramfs) image`" section
3821 in the Yocto Project Development Tasks Manual. 3821 in the Yocto Project Development Tasks Manual.
3822 3822
3823 :term:`INITRAMFS_IMAGE_BUNDLE` 3823 :term:`INITRAMFS_IMAGE_BUNDLE`
@@ -3869,7 +3869,7 @@ system and gives an overview of their function and contents.
3869 See the 3869 See the
3870 :yocto_git:`local.conf.sample.extended </poky/tree/meta-poky/conf/templates/default/local.conf.sample.extended>` 3870 :yocto_git:`local.conf.sample.extended </poky/tree/meta-poky/conf/templates/default/local.conf.sample.extended>`
3871 file for additional information. Also, for information on creating an 3871 file for additional information. Also, for information on creating an
3872 :term:`Initramfs`, see the ":ref:`dev-manual/common-tasks:building an initial ram filesystem (Initramfs) image`" section 3872 :term:`Initramfs`, see the ":ref:`dev-manual/building:building an initial ram filesystem (Initramfs) image`" section
3873 in the Yocto Project Development Tasks Manual. 3873 in the Yocto Project Development Tasks Manual.
3874 3874
3875 :term:`INITRAMFS_LINK_NAME` 3875 :term:`INITRAMFS_LINK_NAME`
@@ -3895,7 +3895,7 @@ system and gives an overview of their function and contents.
3895 a separate multiconfig, this is meant to be used in addition to :term:`INITRAMFS_DEPLOY_DIR_IMAGE`. 3895 a separate multiconfig, this is meant to be used in addition to :term:`INITRAMFS_DEPLOY_DIR_IMAGE`.
3896 3896
3897 For more information on how to bundle an :term:`Initramfs` image from a separate 3897 For more information on how to bundle an :term:`Initramfs` image from a separate
3898 multiconfig see the ":ref:`dev-manual/common-tasks:Bundling an Initramfs Image From a Separate Multiconfig`" 3898 multiconfig see the ":ref:`dev-manual/building:Bundling an Initramfs Image From a Separate Multiconfig`"
3899 section in the Yocto Project Development Tasks Manual. 3899 section in the Yocto Project Development Tasks Manual.
3900 3900
3901 :term:`INITRAMFS_NAME` 3901 :term:`INITRAMFS_NAME`
@@ -4497,7 +4497,7 @@ system and gives an overview of their function and contents.
4497 The OpenEmbedded build system produces a warning if the variable 4497 The OpenEmbedded build system produces a warning if the variable
4498 is not set for any given layer. 4498 is not set for any given layer.
4499 4499
4500 See the ":ref:`dev-manual/common-tasks:creating your own layer`" 4500 See the ":ref:`dev-manual/layers:creating your own layer`"
4501 section in the Yocto Project Development Tasks Manual. 4501 section in the Yocto Project Development Tasks Manual.
4502 4502
4503 :term:`LAYERVERSION` 4503 :term:`LAYERVERSION`
@@ -4546,7 +4546,7 @@ system and gives an overview of their function and contents.
4546 This variable must be defined for all recipes (unless 4546 This variable must be defined for all recipes (unless
4547 :term:`LICENSE` is set to "CLOSED"). 4547 :term:`LICENSE` is set to "CLOSED").
4548 4548
4549 For more information, see the ":ref:`dev-manual/common-tasks:tracking license changes`" 4549 For more information, see the ":ref:`dev-manual/licenses:tracking license changes`"
4550 section in the Yocto Project Development Tasks Manual. 4550 section in the Yocto Project Development Tasks Manual.
4551 4551
4552 :term:`LICENSE` 4552 :term:`LICENSE`
@@ -4610,7 +4610,7 @@ system and gives an overview of their function and contents.
4610 For related information on providing license text, see the 4610 For related information on providing license text, see the
4611 :term:`COPY_LIC_DIRS` variable, the 4611 :term:`COPY_LIC_DIRS` variable, the
4612 :term:`COPY_LIC_MANIFEST` variable, and the 4612 :term:`COPY_LIC_MANIFEST` variable, and the
4613 ":ref:`dev-manual/common-tasks:providing license text`" 4613 ":ref:`dev-manual/licenses:providing license text`"
4614 section in the Yocto Project Development Tasks Manual. 4614 section in the Yocto Project Development Tasks Manual.
4615 4615
4616 :term:`LICENSE_FLAGS` 4616 :term:`LICENSE_FLAGS`
@@ -4623,14 +4623,14 @@ system and gives an overview of their function and contents.
4623 typically used to mark recipes that might require additional licenses 4623 typically used to mark recipes that might require additional licenses
4624 in order to be used in a commercial product. For more information, 4624 in order to be used in a commercial product. For more information,
4625 see the 4625 see the
4626 ":ref:`dev-manual/common-tasks:enabling commercially licensed recipes`" 4626 ":ref:`dev-manual/licenses:enabling commercially licensed recipes`"
4627 section in the Yocto Project Development Tasks Manual. 4627 section in the Yocto Project Development Tasks Manual.
4628 4628
4629 :term:`LICENSE_FLAGS_ACCEPTED` 4629 :term:`LICENSE_FLAGS_ACCEPTED`
4630 Lists license flags that when specified in 4630 Lists license flags that when specified in
4631 :term:`LICENSE_FLAGS` within a recipe should not 4631 :term:`LICENSE_FLAGS` within a recipe should not
4632 prevent that recipe from being built. For more information, see the 4632 prevent that recipe from being built. For more information, see the
4633 ":ref:`dev-manual/common-tasks:enabling commercially licensed recipes`" 4633 ":ref:`dev-manual/licenses:enabling commercially licensed recipes`"
4634 section in the Yocto Project Development Tasks Manual. 4634 section in the Yocto Project Development Tasks Manual.
4635 4635
4636 :term:`LICENSE_PATH` 4636 :term:`LICENSE_PATH`
@@ -5193,7 +5193,7 @@ system and gives an overview of their function and contents.
5193 Controls how the OpenEmbedded build system spawns interactive 5193 Controls how the OpenEmbedded build system spawns interactive
5194 terminals on the host development system (e.g. using the BitBake 5194 terminals on the host development system (e.g. using the BitBake
5195 command with the ``-c devshell`` command-line option). For more 5195 command with the ``-c devshell`` command-line option). For more
5196 information, see the ":ref:`dev-manual/common-tasks:using a development shell`" section in 5196 information, see the ":ref:`dev-manual/development-shell:using a development shell`" section in
5197 the Yocto Project Development Tasks Manual. 5197 the Yocto Project Development Tasks Manual.
5198 5198
5199 You can use the following values for the :term:`OE_TERMINAL` variable: 5199 You can use the following values for the :term:`OE_TERMINAL` variable:
@@ -5341,7 +5341,7 @@ system and gives an overview of their function and contents.
5341 5341
5342 An easy way to see what overrides apply is to search for :term:`OVERRIDES` 5342 An easy way to see what overrides apply is to search for :term:`OVERRIDES`
5343 in the output of the ``bitbake -e`` command. See the 5343 in the output of the ``bitbake -e`` command. See the
5344 ":ref:`dev-manual/common-tasks:viewing variable values`" section in the Yocto 5344 ":ref:`dev-manual/debugging:viewing variable values`" section in the Yocto
5345 Project Development Tasks Manual for more information. 5345 Project Development Tasks Manual for more information.
5346 5346
5347 :term:`P` 5347 :term:`P`
@@ -5362,7 +5362,7 @@ system and gives an overview of their function and contents.
5362 specific by using the package name as a suffix. 5362 specific by using the package name as a suffix.
5363 5363
5364 You can find out more about applying this variable in the 5364 You can find out more about applying this variable in the
5365 ":ref:`dev-manual/common-tasks:adding custom metadata to packages`" 5365 ":ref:`dev-manual/packages:adding custom metadata to packages`"
5366 section in the Yocto Project Development Tasks Manual. 5366 section in the Yocto Project Development Tasks Manual.
5367 5367
5368 :term:`PACKAGE_ARCH` 5368 :term:`PACKAGE_ARCH`
@@ -5470,7 +5470,7 @@ system and gives an overview of their function and contents.
5470 use of the :term:`INHIBIT_PACKAGE_DEBUG_SPLIT` variable. 5470 use of the :term:`INHIBIT_PACKAGE_DEBUG_SPLIT` variable.
5471 5471
5472 You can find out more about debugging using GDB by reading the 5472 You can find out more about debugging using GDB by reading the
5473 ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) remotely`" section 5473 ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) remotely`" section
5474 in the Yocto Project Development Tasks Manual. 5474 in the Yocto Project Development Tasks Manual.
5475 5475
5476 :term:`PACKAGE_EXCLUDE` 5476 :term:`PACKAGE_EXCLUDE`
@@ -5629,7 +5629,7 @@ system and gives an overview of their function and contents.
5629 the :ref:`core-image-minimal-initramfs <ref-manual/images:images>` 5629 the :ref:`core-image-minimal-initramfs <ref-manual/images:images>`
5630 image. When working with an initial RAM filesystem (:term:`Initramfs`) image, 5630 image. When working with an initial RAM filesystem (:term:`Initramfs`) image,
5631 use the :term:`PACKAGE_INSTALL` variable. For information on creating an 5631 use the :term:`PACKAGE_INSTALL` variable. For information on creating an
5632 :term:`Initramfs`, see the ":ref:`dev-manual/common-tasks:building an initial ram filesystem (Initramfs) image`" section 5632 :term:`Initramfs`, see the ":ref:`dev-manual/building:building an initial ram filesystem (Initramfs) image`" section
5633 in the Yocto Project Development Tasks Manual. 5633 in the Yocto Project Development Tasks Manual.
5634 5634
5635 :term:`PACKAGE_INSTALL_ATTEMPTONLY` 5635 :term:`PACKAGE_INSTALL_ATTEMPTONLY`
@@ -5652,7 +5652,7 @@ system and gives an overview of their function and contents.
5652 :term:`PACKAGE_WRITE_DEPS`. 5652 :term:`PACKAGE_WRITE_DEPS`.
5653 5653
5654 For information on running post-installation scripts, see the 5654 For information on running post-installation scripts, see the
5655 ":ref:`dev-manual/common-tasks:post-installation scripts`" 5655 ":ref:`dev-manual/new-recipe:post-installation scripts`"
5656 section in the Yocto Project Development Tasks Manual. 5656 section in the Yocto Project Development Tasks Manual.
5657 5657
5658 :term:`PACKAGECONFIG` 5658 :term:`PACKAGECONFIG`
@@ -5803,7 +5803,7 @@ system and gives an overview of their function and contents.
5803 5803
5804 For an example of how to use the :term:`PACKAGES_DYNAMIC` variable when 5804 For an example of how to use the :term:`PACKAGES_DYNAMIC` variable when
5805 you are splitting packages, see the 5805 you are splitting packages, see the
5806 ":ref:`dev-manual/common-tasks:handling optional module packaging`" 5806 ":ref:`dev-manual/packages:handling optional module packaging`"
5807 section in the Yocto Project Development Tasks Manual. 5807 section in the Yocto Project Development Tasks Manual.
5808 5808
5809 :term:`PACKAGESPLITFUNCS` 5809 :term:`PACKAGESPLITFUNCS`
@@ -5841,7 +5841,7 @@ system and gives an overview of their function and contents.
5841 the :ref:`ref-tasks-compile` task that result in race conditions, you can clear 5841 the :ref:`ref-tasks-compile` task that result in race conditions, you can clear
5842 the :term:`PARALLEL_MAKE` variable within the recipe as a workaround. For 5842 the :term:`PARALLEL_MAKE` variable within the recipe as a workaround. For
5843 information on addressing race conditions, see the 5843 information on addressing race conditions, see the
5844 ":ref:`dev-manual/common-tasks:debugging parallel make races`" 5844 ":ref:`dev-manual/debugging:debugging parallel make races`"
5845 section in the Yocto Project Development Tasks Manual. 5845 section in the Yocto Project Development Tasks Manual.
5846 5846
5847 For single socket systems (i.e. one CPU), you should not have to 5847 For single socket systems (i.e. one CPU), you should not have to
@@ -5851,7 +5851,7 @@ system and gives an overview of their function and contents.
5851 not set higher than "-j 20". 5851 not set higher than "-j 20".
5852 5852
5853 For more information on speeding up builds, see the 5853 For more information on speeding up builds, see the
5854 ":ref:`dev-manual/common-tasks:speeding up a build`" 5854 ":ref:`dev-manual/speeding-up-build:speeding up a build`"
5855 section in the Yocto Project Development Tasks Manual. 5855 section in the Yocto Project Development Tasks Manual.
5856 5856
5857 :term:`PARALLEL_MAKEINST` 5857 :term:`PARALLEL_MAKEINST`
@@ -5872,7 +5872,7 @@ system and gives an overview of their function and contents.
5872 the :ref:`ref-tasks-install` task that result in race conditions, you can 5872 the :ref:`ref-tasks-install` task that result in race conditions, you can
5873 clear the :term:`PARALLEL_MAKEINST` variable within the recipe as a 5873 clear the :term:`PARALLEL_MAKEINST` variable within the recipe as a
5874 workaround. For information on addressing race conditions, see the 5874 workaround. For information on addressing race conditions, see the
5875 ":ref:`dev-manual/common-tasks:debugging parallel make races`" 5875 ":ref:`dev-manual/debugging:debugging parallel make races`"
5876 section in the Yocto Project Development Tasks Manual. 5876 section in the Yocto Project Development Tasks Manual.
5877 5877
5878 :term:`PATCHRESOLVE` 5878 :term:`PATCHRESOLVE`
@@ -5968,7 +5968,7 @@ system and gives an overview of their function and contents.
5968 For examples of how this data is used, see the 5968 For examples of how this data is used, see the
5969 ":ref:`overview-manual/concepts:automatically added runtime dependencies`" 5969 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
5970 section in the Yocto Project Overview and Concepts Manual and the 5970 section in the Yocto Project Overview and Concepts Manual and the
5971 ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``" 5971 ":ref:`dev-manual/debugging:viewing package information with \`\`oe-pkgdata-util\`\``"
5972 section in the Yocto Project Development Tasks Manual. For more 5972 section in the Yocto Project Development Tasks Manual. For more
5973 information on the shared, global-state directory, see 5973 information on the shared, global-state directory, see
5974 :term:`STAGING_DIR_HOST`. 5974 :term:`STAGING_DIR_HOST`.
@@ -6084,7 +6084,7 @@ system and gives an overview of their function and contents.
6084 6084
6085 Because manually managing :term:`PR` can be cumbersome and error-prone, 6085 Because manually managing :term:`PR` can be cumbersome and error-prone,
6086 an automated solution exists. See the 6086 an automated solution exists. See the
6087 ":ref:`dev-manual/common-tasks:working with a pr service`" section 6087 ":ref:`dev-manual/packages:working with a pr service`" section
6088 in the Yocto Project Development Tasks Manual for more information. 6088 in the Yocto Project Development Tasks Manual for more information.
6089 6089
6090 :term:`PREFERRED_PROVIDER` 6090 :term:`PREFERRED_PROVIDER`
@@ -6107,7 +6107,7 @@ system and gives an overview of their function and contents.
6107 PREFERRED_PROVIDER_virtual/libgl ?= "mesa" 6107 PREFERRED_PROVIDER_virtual/libgl ?= "mesa"
6108 6108
6109 For more 6109 For more
6110 information, see the ":ref:`dev-manual/common-tasks:using virtual providers`" 6110 information, see the ":ref:`dev-manual/new-recipe:using virtual providers`"
6111 section in the Yocto Project Development Tasks Manual. 6111 section in the Yocto Project Development Tasks Manual.
6112 6112
6113 .. note:: 6113 .. note::
@@ -6307,7 +6307,7 @@ system and gives an overview of their function and contents.
6307 6307
6308 You must 6308 You must
6309 set the variable if you want to automatically start a local :ref:`PR 6309 set the variable if you want to automatically start a local :ref:`PR
6310 service <dev-manual/common-tasks:working with a pr service>`. You can 6310 service <dev-manual/packages:working with a pr service>`. You can
6311 set :term:`PRSERV_HOST` to other values to use a remote PR service. 6311 set :term:`PRSERV_HOST` to other values to use a remote PR service.
6312 6312
6313 6313
@@ -6321,7 +6321,7 @@ system and gives an overview of their function and contents.
6321 6321
6322 :term:`PTEST_ENABLED` 6322 :term:`PTEST_ENABLED`
6323 Specifies whether or not :ref:`Package 6323 Specifies whether or not :ref:`Package
6324 Test <dev-manual/common-tasks:testing packages with ptest>` (ptest) 6324 Test <dev-manual/packages:testing packages with ptest>` (ptest)
6325 functionality is enabled when building a recipe. You should not set 6325 functionality is enabled when building a recipe. You should not set
6326 this variable directly. Enabling and disabling building Package Tests 6326 this variable directly. Enabling and disabling building Package Tests
6327 at build time should be done by adding "ptest" to (or removing it 6327 at build time should be done by adding "ptest" to (or removing it
@@ -7440,7 +7440,7 @@ system and gives an overview of their function and contents.
7440 various ``SPL_*`` variables used by the OpenEmbedded build system. 7440 various ``SPL_*`` variables used by the OpenEmbedded build system.
7441 7441
7442 See the BeagleBone machine configuration example in the 7442 See the BeagleBone machine configuration example in the
7443 ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`" 7443 ":ref:`dev-manual/layers:adding a layer using the \`\`bitbake-layers\`\` script`"
7444 section in the Yocto Project Board Support Package Developer's Guide 7444 section in the Yocto Project Board Support Package Developer's Guide
7445 for additional information. 7445 for additional information.
7446 7446
@@ -7532,7 +7532,7 @@ system and gives an overview of their function and contents.
7532 For information on limitations when inheriting the latest revision 7532 For information on limitations when inheriting the latest revision
7533 of software using :term:`SRCREV`, see the :term:`AUTOREV` variable 7533 of software using :term:`SRCREV`, see the :term:`AUTOREV` variable
7534 description and the 7534 description and the
7535 ":ref:`dev-manual/common-tasks:automatically incrementing a package version number`" 7535 ":ref:`dev-manual/packages:automatically incrementing a package version number`"
7536 section, which is in the Yocto Project Development Tasks Manual. 7536 section, which is in the Yocto Project Development Tasks Manual.
7537 7537
7538 :term:`SRCTREECOVEREDTASKS` 7538 :term:`SRCTREECOVEREDTASKS`
@@ -8058,7 +8058,7 @@ system and gives an overview of their function and contents.
8058 8058
8059 :term:`SYSVINIT_ENABLED_GETTYS` 8059 :term:`SYSVINIT_ENABLED_GETTYS`
8060 When using 8060 When using
8061 :ref:`SysVinit <dev-manual/common-tasks:enabling system services>`, 8061 :ref:`SysVinit <dev-manual/new-recipe:enabling system services>`,
8062 specifies a space-separated list of the virtual terminals that should 8062 specifies a space-separated list of the virtual terminals that should
8063 run a `getty <https://en.wikipedia.org/wiki/Getty_%28Unix%29>`__ 8063 run a `getty <https://en.wikipedia.org/wiki/Getty_%28Unix%29>`__
8064 (allowing login), assuming :term:`USE_VT` is not set to 8064 (allowing login), assuming :term:`USE_VT` is not set to
@@ -8305,7 +8305,7 @@ system and gives an overview of their function and contents.
8305 BitBake targets shown when sourcing the ``oe-init-build-env`` script. 8305 BitBake targets shown when sourcing the ``oe-init-build-env`` script.
8306 8306
8307 For details, see the 8307 For details, see the
8308 :ref:`dev-manual/common-tasks:creating a custom template configuration directory` 8308 :ref:`dev-manual/custom-template-configuration-directory:creating a custom template configuration directory`
8309 section in the Yocto Project Development Tasks manual. 8309 section in the Yocto Project Development Tasks manual.
8310 8310
8311 .. note:: 8311 .. note::
@@ -8360,7 +8360,7 @@ system and gives an overview of their function and contents.
8360 file. 8360 file.
8361 8361
8362 For more information on testing images, see the 8362 For more information on testing images, see the
8363 ":ref:`dev-manual/common-tasks:performing automated runtime testing`" 8363 ":ref:`dev-manual/runtime-testing:performing automated runtime testing`"
8364 section in the Yocto Project Development Tasks Manual. 8364 section in the Yocto Project Development Tasks Manual.
8365 8365
8366 :term:`TEST_SERIALCONTROL_CMD` 8366 :term:`TEST_SERIALCONTROL_CMD`
@@ -8433,7 +8433,7 @@ system and gives an overview of their function and contents.
8433 TEST_SUITES = "test_A test_B" 8433 TEST_SUITES = "test_A test_B"
8434 8434
8435 For more information on testing images, see the 8435 For more information on testing images, see the
8436 ":ref:`dev-manual/common-tasks:performing automated runtime testing`" 8436 ":ref:`dev-manual/runtime-testing:performing automated runtime testing`"
8437 section in the Yocto Project Development Tasks Manual. 8437 section in the Yocto Project Development Tasks Manual.
8438 8438
8439 :term:`TEST_TARGET` 8439 :term:`TEST_TARGET`
@@ -8452,7 +8452,7 @@ system and gives an overview of their function and contents.
8452 You can provide the following arguments with :term:`TEST_TARGET`: 8452 You can provide the following arguments with :term:`TEST_TARGET`:
8453 8453
8454 - *"qemu":* Boots a QEMU image and runs the tests. See the 8454 - *"qemu":* Boots a QEMU image and runs the tests. See the
8455 ":ref:`dev-manual/common-tasks:enabling runtime tests on qemu`" section 8455 ":ref:`dev-manual/runtime-testing:enabling runtime tests on qemu`" section
8456 in the Yocto Project Development Tasks Manual for more 8456 in the Yocto Project Development Tasks Manual for more
8457 information. 8457 information.
8458 8458
@@ -8468,7 +8468,7 @@ system and gives an overview of their function and contents.
8468 ``meta/lib/oeqa/controllers/simpleremote.py``. 8468 ``meta/lib/oeqa/controllers/simpleremote.py``.
8469 8469
8470 For information on running tests on hardware, see the 8470 For information on running tests on hardware, see the
8471 ":ref:`dev-manual/common-tasks:enabling runtime tests on hardware`" 8471 ":ref:`dev-manual/runtime-testing:enabling runtime tests on hardware`"
8472 section in the Yocto Project Development Tasks Manual. 8472 section in the Yocto Project Development Tasks Manual.
8473 8473
8474 :term:`TEST_TARGET_IP` 8474 :term:`TEST_TARGET_IP`
@@ -8505,7 +8505,7 @@ system and gives an overview of their function and contents.
8505 8505
8506 For more information 8506 For more information
8507 on enabling, running, and writing these tests, see the 8507 on enabling, running, and writing these tests, see the
8508 ":ref:`dev-manual/common-tasks:performing automated runtime testing`" 8508 ":ref:`dev-manual/runtime-testing:performing automated runtime testing`"
8509 section in the Yocto Project Development Tasks Manual and the 8509 section in the Yocto Project Development Tasks Manual and the
8510 ":ref:`ref-classes-testimage`" section. 8510 ":ref:`ref-classes-testimage`" section.
8511 8511
@@ -8969,13 +8969,13 @@ system and gives an overview of their function and contents.
8969 specifically set. Typically, you would set :term:`USE_DEVFS` to "0" for a 8969 specifically set. Typically, you would set :term:`USE_DEVFS` to "0" for a
8970 statically populated ``/dev`` directory. 8970 statically populated ``/dev`` directory.
8971 8971
8972 See the ":ref:`dev-manual/common-tasks:selecting a device manager`" section in 8972 See the ":ref:`dev-manual/device-manager:selecting a device manager`" section in
8973 the Yocto Project Development Tasks Manual for information on how to 8973 the Yocto Project Development Tasks Manual for information on how to
8974 use this variable. 8974 use this variable.
8975 8975
8976 :term:`USE_VT` 8976 :term:`USE_VT`
8977 When using 8977 When using
8978 :ref:`SysVinit <dev-manual/common-tasks:enabling system services>`, 8978 :ref:`SysVinit <dev-manual/new-recipe:enabling system services>`,
8979 determines whether or not to run a 8979 determines whether or not to run a
8980 `getty <https://en.wikipedia.org/wiki/Getty_%28Unix%29>`__ on any 8980 `getty <https://en.wikipedia.org/wiki/Getty_%28Unix%29>`__ on any
8981 virtual terminals in order to enable logging in through those 8981 virtual terminals in order to enable logging in through those
@@ -9156,7 +9156,7 @@ system and gives an overview of their function and contents.
9156 OpenEmbedded build system to create a partitioned image 9156 OpenEmbedded build system to create a partitioned image
9157 (``image.wic``). For information on how to create a partitioned 9157 (``image.wic``). For information on how to create a partitioned
9158 image, see the 9158 image, see the
9159 ":ref:`dev-manual/common-tasks:creating partitioned images using wic`" 9159 ":ref:`dev-manual/wic:creating partitioned images using wic`"
9160 section in the Yocto Project Development Tasks Manual. For details on 9160 section in the Yocto Project Development Tasks Manual. For details on
9161 the kickstart file format, see the ":doc:`/ref-manual/kickstart`" Chapter. 9161 the kickstart file format, see the ":doc:`/ref-manual/kickstart`" Chapter.
9162 9162
diff --git a/documentation/sdk-manual/extensible.rst b/documentation/sdk-manual/extensible.rst
index ed38ac3f3f..4d36270e64 100644
--- a/documentation/sdk-manual/extensible.rst
+++ b/documentation/sdk-manual/extensible.rst
@@ -652,7 +652,7 @@ counterparts.
652 ``devtool upgrade`` 652 ``devtool upgrade``
653 happens to be one. You can read about all the methods by which you 653 happens to be one. You can read about all the methods by which you
654 can upgrade recipes in the 654 can upgrade recipes in the
655 :ref:`dev-manual/common-tasks:upgrading recipes` section 655 :ref:`dev-manual/upgrading-recipes:upgrading recipes` section
656 of the Yocto Project Development Tasks Manual. 656 of the Yocto Project Development Tasks Manual.
657 657
658The ``devtool upgrade`` command is flexible enough to allow you to 658The ``devtool upgrade`` command is flexible enough to allow you to
@@ -1114,7 +1114,7 @@ links created within the source tree:
1114 - ``sysroot-destdir/``: Contains a subset of files installed within 1114 - ``sysroot-destdir/``: Contains a subset of files installed within
1115 :ref:`ref-tasks-install` that have been put into the shared sysroot. For 1115 :ref:`ref-tasks-install` that have been put into the shared sysroot. For
1116 more information, see the 1116 more information, see the
1117 ":ref:`dev-manual/common-tasks:sharing files between recipes`" section. 1117 ":ref:`dev-manual/new-recipe:sharing files between recipes`" section.
1118 1118
1119 - ``packages-split/``: Contains subdirectories for each package 1119 - ``packages-split/``: Contains subdirectories for each package
1120 produced by the recipe. For more information, see the 1120 produced by the recipe. For more information, see the
diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst
index 0831f80466..9afe108396 100644
--- a/documentation/test-manual/intro.rst
+++ b/documentation/test-manual/intro.rst
@@ -142,7 +142,7 @@ the following types of tests:
142- *Package Testing:* A Package Test (ptest) runs tests against packages 142- *Package Testing:* A Package Test (ptest) runs tests against packages
143 built by the OpenEmbedded build system on the target machine. See the 143 built by the OpenEmbedded build system on the target machine. See the
144 :ref:`Testing Packages With 144 :ref:`Testing Packages With
145 ptest <dev-manual/common-tasks:Testing Packages With ptest>` section 145 ptest <dev-manual/packages:Testing Packages With ptest>` section
146 in the Yocto Project Development Tasks Manual and the 146 in the Yocto Project Development Tasks Manual and the
147 ":yocto_wiki:`Ptest </Ptest>`" Wiki page for more 147 ":yocto_wiki:`Ptest </Ptest>`" Wiki page for more
148 information on Ptest. 148 information on Ptest.
diff --git a/documentation/test-manual/yocto-project-compatible.rst b/documentation/test-manual/yocto-project-compatible.rst
index 96c12ac083..65d924fad9 100644
--- a/documentation/test-manual/yocto-project-compatible.rst
+++ b/documentation/test-manual/yocto-project-compatible.rst
@@ -27,7 +27,7 @@ In the second version of the program, a script was added to make validation
27easier and clearer, the script is called ``yocto-check-layer`` and is 27easier and clearer, the script is called ``yocto-check-layer`` and is
28available in :term:`OpenEmbedded-Core (OE-Core)`. 28available in :term:`OpenEmbedded-Core (OE-Core)`.
29 29
30See :ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project` 30See :ref:`dev-manual/layers:making sure your layer is compatible with yocto project`
31for details. 31for details.
32 32
33======== 33========
diff --git a/documentation/toaster-manual/reference.rst b/documentation/toaster-manual/reference.rst
index f8aabeed04..e014d2f090 100644
--- a/documentation/toaster-manual/reference.rst
+++ b/documentation/toaster-manual/reference.rst
@@ -65,7 +65,7 @@ layers.
65For general information on layers, see the 65For general information on layers, see the
66":ref:`overview-manual/yp-intro:the yocto project layer model`" 66":ref:`overview-manual/yp-intro:the yocto project layer model`"
67section in the Yocto Project Overview and Concepts Manual. For information on how 67section in the Yocto Project Overview and Concepts Manual. For information on how
68to create layers, see the ":ref:`dev-manual/common-tasks:understanding and creating layers`" 68to create layers, see the ":ref:`dev-manual/layers:understanding and creating layers`"
69section in the Yocto Project Development Tasks Manual. 69section in the Yocto Project Development Tasks Manual.
70 70
71Configuring Toaster to Hook Into Your Layer Index 71Configuring Toaster to Hook Into Your Layer Index
diff --git a/documentation/transitioning-to-a-custom-environment.rst b/documentation/transitioning-to-a-custom-environment.rst
index ee1fc891ab..6ff55e5619 100644
--- a/documentation/transitioning-to-a-custom-environment.rst
+++ b/documentation/transitioning-to-a-custom-environment.rst
@@ -42,7 +42,7 @@ Transitioning to a custom environment for systems development
42 You might want to start with the build specification that Poky provides 42 You might want to start with the build specification that Poky provides
43 (which is reference embedded distribution) and then add your newly chosen 43 (which is reference embedded distribution) and then add your newly chosen
44 layers to that. Here is the information :ref:`about adding layers 44 layers to that. Here is the information :ref:`about adding layers
45 <dev-manual/common-tasks:Understanding and Creating Layers>`. 45 <dev-manual/layers:Understanding and Creating Layers>`.
46 46
47#. **Based on the layers you've chosen, make needed changes in your 47#. **Based on the layers you've chosen, make needed changes in your
48 configuration**. 48 configuration**.
@@ -58,7 +58,7 @@ Transitioning to a custom environment for systems development
58 releases. If you are using a Yocto Project release earlier than 2.4, use the 58 releases. If you are using a Yocto Project release earlier than 2.4, use the
59 ``yocto-layer create`` tool. The ``bitbake-layers`` tool also provides a number 59 ``yocto-layer create`` tool. The ``bitbake-layers`` tool also provides a number
60 of other useful layer-related commands. See 60 of other useful layer-related commands. See
61 :ref:`dev-manual/common-tasks:creating a general layer using the 61 :ref:`dev-manual/layers:creating a general layer using the
62 \`\`bitbake-layers\`\` script` section. 62 \`\`bitbake-layers\`\` script` section.
63 63
64#. **Create your own layer for the BSP you're going to use**. 64#. **Create your own layer for the BSP you're going to use**.
@@ -79,7 +79,7 @@ Transitioning to a custom environment for systems development
79 process of refinement. Start by getting each step of the build process 79 process of refinement. Start by getting each step of the build process
80 working beginning with fetching all the way through packaging. Next, run the 80 working beginning with fetching all the way through packaging. Next, run the
81 software on your target and refine further as needed. See :ref:`Writing a New 81 software on your target and refine further as needed. See :ref:`Writing a New
82 Recipe <dev-manual/common-tasks:writing a new recipe>` in the 82 Recipe <dev-manual/new-recipe:writing a new recipe>` in the
83 Yocto Project Development Tasks Manual for more information. 83 Yocto Project Development Tasks Manual for more information.
84 84
85#. **Now you're ready to create an image recipe**. 85#. **Now you're ready to create an image recipe**.
@@ -103,7 +103,7 @@ Transitioning to a custom environment for systems development
103 needs to change for your distribution. If you find yourself adding a lot of 103 needs to change for your distribution. If you find yourself adding a lot of
104 configuration to your local.conf file aside from paths and other typical 104 configuration to your local.conf file aside from paths and other typical
105 local settings, it's time to :ref:`consider creating your own distribution 105 local settings, it's time to :ref:`consider creating your own distribution
106 <dev-manual/common-tasks:creating your own distribution>`. 106 <dev-manual/custom-distribution:creating your own distribution>`.
107 107
108 You can add product specifications that can customize the distribution if 108 You can add product specifications that can customize the distribution if
109 needed in other layers. You can also add other functionality specific to the 109 needed in other layers. You can also add other functionality specific to the
diff --git a/documentation/what-i-wish-id-known.rst b/documentation/what-i-wish-id-known.rst
index 46c5cf19f9..10f746ff1f 100644
--- a/documentation/what-i-wish-id-known.rst
+++ b/documentation/what-i-wish-id-known.rst
@@ -132,7 +132,7 @@ contact us with other suggestions.
132 say "bitbake foo" where "foo" is the name for a specific recipe. As you 132 say "bitbake foo" where "foo" is the name for a specific recipe. As you
133 become more advanced using the Yocto Project, and if builds are failing, it 133 become more advanced using the Yocto Project, and if builds are failing, it
134 can be useful to make sure the fetch itself works as desired. Here are some 134 can be useful to make sure the fetch itself works as desired. Here are some
135 valuable links: :ref:`dev-manual/common-tasks:Using a Development 135 valuable links: :ref:`dev-manual/development-shell:Using a Development
136 Shell` for information on how to build and run a specific task using 136 Shell` for information on how to build and run a specific task using
137 devshell. Also, the :ref:`SDK manual shows how to build out a specific recipe 137 devshell. Also, the :ref:`SDK manual shows how to build out a specific recipe
138 <sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component>`. 138 <sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component>`.