summaryrefslogtreecommitdiffstats
path: root/documentation
diff options
context:
space:
mode:
Diffstat (limited to 'documentation')
-rw-r--r--documentation/dev-manual/bmaptool.rst59
-rw-r--r--documentation/dev-manual/build-quality.rst409
-rw-r--r--documentation/dev-manual/building.rst939
-rw-r--r--documentation/dev-manual/common-tasks.rst11414
-rw-r--r--documentation/dev-manual/custom-distribution.rst109
-rw-r--r--documentation/dev-manual/custom-template-configuration-directory.rst72
-rw-r--r--documentation/dev-manual/customizing-images.rst223
-rw-r--r--documentation/dev-manual/debugging.rst1245
-rw-r--r--documentation/dev-manual/development-shell.rst82
-rw-r--r--documentation/dev-manual/device-manager.rst74
-rw-r--r--documentation/dev-manual/disk-space.rst61
-rw-r--r--documentation/dev-manual/efficiently-fetching-sources.rst68
-rw-r--r--documentation/dev-manual/error-reporting-tool.rst84
-rw-r--r--documentation/dev-manual/external-scm.rst67
-rw-r--r--documentation/dev-manual/external-toolchain.rst40
-rw-r--r--documentation/dev-manual/gobject-introspection.rst155
-rw-r--r--documentation/dev-manual/index.rst39
-rw-r--r--documentation/dev-manual/init-manager.rst162
-rw-r--r--documentation/dev-manual/layers.rst839
-rw-r--r--documentation/dev-manual/libraries.rst267
-rw-r--r--documentation/dev-manual/licenses.rst524
-rw-r--r--documentation/dev-manual/new-machine.rst118
-rw-r--r--documentation/dev-manual/new-recipe.rst1634
-rw-r--r--documentation/dev-manual/packages.rst1238
-rw-r--r--documentation/dev-manual/prebuilt-libraries.rst209
-rw-r--r--documentation/dev-manual/python-development-shell.rst50
-rw-r--r--documentation/dev-manual/quilt.rst89
-rw-r--r--documentation/dev-manual/read-only-rootfs.rst89
-rw-r--r--documentation/dev-manual/runtime-testing.rst598
-rw-r--r--documentation/dev-manual/sbom.rst72
-rw-r--r--documentation/dev-manual/securing-images.rst156
-rw-r--r--documentation/dev-manual/speeding-up-build.rst109
-rw-r--r--documentation/dev-manual/temporary-source-code.rst66
-rw-r--r--documentation/dev-manual/upgrading-recipes.rst397
-rw-r--r--documentation/dev-manual/vulnerabilities.rst214
-rw-r--r--documentation/dev-manual/wayland.rst90
-rw-r--r--documentation/dev-manual/wic.rst729
-rw-r--r--documentation/dev-manual/x32-psabi.rst54
38 files changed, 11427 insertions, 11417 deletions
diff --git a/documentation/dev-manual/bmaptool.rst b/documentation/dev-manual/bmaptool.rst
new file mode 100644
index 0000000000..9add72cf3b
--- /dev/null
+++ b/documentation/dev-manual/bmaptool.rst
@@ -0,0 +1,59 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Flashing Images Using ``bmaptool``
4**********************************
5
6A fast and easy way to flash an image to a bootable device is to use
7Bmaptool, which is integrated into the OpenEmbedded build system.
8Bmaptool is a generic tool that creates a file's block map (bmap) and
9then uses that map to copy the file. As compared to traditional tools
10such as dd or cp, Bmaptool can copy (or flash) large files like raw
11system image files much faster.
12
13.. note::
14
15 - If you are using Ubuntu or Debian distributions, you can install
16 the ``bmap-tools`` package using the following command and then
17 use the tool without specifying ``PATH`` even from the root
18 account::
19
20 $ sudo apt install bmap-tools
21
22 - If you are unable to install the ``bmap-tools`` package, you will
23 need to build Bmaptool before using it. Use the following command::
24
25 $ bitbake bmap-tools-native
26
27Following, is an example that shows how to flash a Wic image. Realize
28that while this example uses a Wic image, you can use Bmaptool to flash
29any type of image. Use these steps to flash an image using Bmaptool:
30
31#. *Update your local.conf File:* You need to have the following set
32 in your ``local.conf`` file before building your image::
33
34 IMAGE_FSTYPES += "wic wic.bmap"
35
36#. *Get Your Image:* Either have your image ready (pre-built with the
37 :term:`IMAGE_FSTYPES`
38 setting previously mentioned) or take the step to build the image::
39
40 $ bitbake image
41
42#. *Flash the Device:* Flash the device with the image by using Bmaptool
43 depending on your particular setup. The following commands assume the
44 image resides in the :term:`Build Directory`'s ``deploy/images/`` area:
45
46 - If you have write access to the media, use this command form::
47
48 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
49
50 - If you do not have write access to the media, set your permissions
51 first and then use the same command form::
52
53 $ sudo chmod 666 /dev/sdX
54 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
55
56For help on the ``bmaptool`` command, use the following command::
57
58 $ bmaptool --help
59
diff --git a/documentation/dev-manual/build-quality.rst b/documentation/dev-manual/build-quality.rst
new file mode 100644
index 0000000000..713ea3a48e
--- /dev/null
+++ b/documentation/dev-manual/build-quality.rst
@@ -0,0 +1,409 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Maintaining Build Output Quality
4********************************
5
6Many factors can influence the quality of a build. For example, if you
7upgrade a recipe to use a new version of an upstream software package or
8you experiment with some new configuration options, subtle changes can
9occur that you might not detect until later. Consider the case where
10your recipe is using a newer version of an upstream package. In this
11case, a new version of a piece of software might introduce an optional
12dependency on another library, which is auto-detected. If that library
13has already been built when the software is building, the software will
14link to the built library and that library will be pulled into your
15image along with the new software even if you did not want the library.
16
17The :ref:`ref-classes-buildhistory` class helps you maintain the quality of
18your build output. You can use the class to highlight unexpected and possibly
19unwanted changes in the build output. When you enable build history, it records
20information about the contents of each package and image and then commits that
21information to a local Git repository where you can examine the information.
22
23The remainder of this section describes the following:
24
25- :ref:`How you can enable and disable build history <dev-manual/build-quality:enabling and disabling build history>`
26
27- :ref:`How to understand what the build history contains <dev-manual/build-quality:understanding what the build history contains>`
28
29- :ref:`How to limit the information used for build history <dev-manual/build-quality:using build history to gather image information only>`
30
31- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/build-quality:examining build history information>`
32
33Enabling and Disabling Build History
34====================================
35
36Build history is disabled by default. To enable it, add the following
37:term:`INHERIT` statement and set the :term:`BUILDHISTORY_COMMIT` variable to
38"1" at the end of your ``conf/local.conf`` file found in the
39:term:`Build Directory`::
40
41 INHERIT += "buildhistory"
42 BUILDHISTORY_COMMIT = "1"
43
44Enabling build history as
45previously described causes the OpenEmbedded build system to collect
46build output information and commit it as a single commit to a local
47:ref:`overview-manual/development-environment:git` repository.
48
49.. note::
50
51 Enabling build history increases your build times slightly,
52 particularly for images, and increases the amount of disk space used
53 during the build.
54
55You can disable build history by removing the previous statements from
56your ``conf/local.conf`` file.
57
58Understanding What the Build History Contains
59=============================================
60
61Build history information is kept in ``${``\ :term:`TOPDIR`\ ``}/buildhistory``
62in the :term:`Build Directory` as defined by the :term:`BUILDHISTORY_DIR`
63variable. Here is an example abbreviated listing:
64
65.. image:: figures/buildhistory.png
66 :align: center
67 :width: 50%
68
69At the top level, there is a ``metadata-revs`` file that lists the
70revisions of the repositories for the enabled layers when the build was
71produced. The rest of the data splits into separate ``packages``,
72``images`` and ``sdk`` directories, the contents of which are described
73as follows.
74
75Build History Package Information
76---------------------------------
77
78The history for each package contains a text file that has name-value
79pairs with information about the package. For example,
80``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
81contains the following:
82
83.. code-block:: none
84
85 PV = 1.22.1
86 PR = r32
87 RPROVIDES =
88 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
89 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
90 PKGSIZE = 540168
91 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
92 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
93 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
94 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
95 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
96 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
97 /etc/busybox.links.nosuid /etc/busybox.links.suid
98
99Most of these
100name-value pairs correspond to variables used to produce the package.
101The exceptions are ``FILELIST``, which is the actual list of files in
102the package, and ``PKGSIZE``, which is the total size of files in the
103package in bytes.
104
105There is also a file that corresponds to the recipe from which the package
106came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
107
108.. code-block:: none
109
110 PV = 1.22.1
111 PR = r32
112 DEPENDS = initscripts kern-tools-native update-rc.d-native \
113 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
114 virtual/libc virtual/update-alternatives
115 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
116 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
117 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
118
119Finally, for those recipes fetched from a version control system (e.g.,
120Git), there is a file that lists source revisions that are specified in
121the recipe and the actual revisions used during the build. Listed
122and actual revisions might differ when
123:term:`SRCREV` is set to
124${:term:`AUTOREV`}. Here is an
125example assuming
126``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
127
128 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
129 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
130 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
131 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
132
133You can use the
134``buildhistory-collect-srcrevs`` command with the ``-a`` option to
135collect the stored :term:`SRCREV` values from build history and report them
136in a format suitable for use in global configuration (e.g.,
137``local.conf`` or a distro include file) to override floating
138:term:`AUTOREV` values to a fixed set of revisions. Here is some example
139output from this command::
140
141 $ buildhistory-collect-srcrevs -a
142 # all-poky-linux
143 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8"
144 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac"
145 # core2-64-poky-linux
146 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66"
147 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8"
148 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0"
149 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0"
150 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1"
151 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c"
152 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7"
153 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee"
154 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1"
155 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100"
156 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3"
157 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf"
158 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced"
159 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8"
160 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f"
161 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa"
162 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70"
163 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
164 # qemux86_64-poky-linux
165 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c"
166 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0"
167 # x86_64-linux
168 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66"
169 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c"
170 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87"
171 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57"
172 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb"
173 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b"
174 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
175
176.. note::
177
178 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
179
180 - By default, only values where the :term:`SRCREV` was not hardcoded
181 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
182 option to see all :term:`SRCREV` values.
183
184 - The output statements might not have any effect if overrides are
185 applied elsewhere in the build system configuration. Use the
186 ``-f`` option to add the ``forcevariable`` override to each output
187 line if you need to work around this restriction.
188
189 - The script does apply special handling when building for multiple
190 machines. However, the script does place a comment before each set
191 of values that specifies which triplet to which they belong as
192 previously shown (e.g., ``i586-poky-linux``).
193
194Build History Image Information
195-------------------------------
196
197The files produced for each image are as follows:
198
199- ``image-files:`` A directory containing selected files from the root
200 filesystem. The files are defined by
201 :term:`BUILDHISTORY_IMAGE_FILES`.
202
203- ``build-id.txt:`` Human-readable information about the build
204 configuration and metadata source revisions. This file contains the
205 full build header as printed by BitBake.
206
207- ``*.dot:`` Dependency graphs for the image that are compatible with
208 ``graphviz``.
209
210- ``files-in-image.txt:`` A list of files in the image with
211 permissions, owner, group, size, and symlink information.
212
213- ``image-info.txt:`` A text file containing name-value pairs with
214 information about the image. See the following listing example for
215 more information.
216
217- ``installed-package-names.txt:`` A list of installed packages by name
218 only.
219
220- ``installed-package-sizes.txt:`` A list of installed packages ordered
221 by size.
222
223- ``installed-packages.txt:`` A list of installed packages with full
224 package filenames.
225
226.. note::
227
228 Installed package information is able to be gathered and produced
229 even if package management is disabled for the final image.
230
231Here is an example of ``image-info.txt``:
232
233.. code-block:: none
234
235 DISTRO = poky
236 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be
237 USER_CLASSES = buildstats
238 IMAGE_CLASSES = qemuboot qemuboot license_image
239 IMAGE_FEATURES = debug-tweaks
240 IMAGE_LINGUAS =
241 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp
242 BAD_RECOMMENDATIONS =
243 NO_RECOMMENDATIONS =
244 PACKAGE_EXCLUDE =
245 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible;
246 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
247 IMAGESIZE = 9265
248
249Other than ``IMAGESIZE``,
250which is the total size of the files in the image in Kbytes, the
251name-value pairs are variables that may have influenced the content of
252the image. This information is often useful when you are trying to
253determine why a change in the package or file listings has occurred.
254
255Using Build History to Gather Image Information Only
256----------------------------------------------------
257
258As you can see, build history produces image information, including
259dependency graphs, so you can see why something was pulled into the
260image. If you are just interested in this information and not interested
261in collecting specific package or SDK information, you can enable
262writing only image information without any history by adding the
263following to your ``conf/local.conf`` file found in the
264:term:`Build Directory`::
265
266 INHERIT += "buildhistory"
267 BUILDHISTORY_COMMIT = "0"
268 BUILDHISTORY_FEATURES = "image"
269
270Here, you set the
271:term:`BUILDHISTORY_FEATURES`
272variable to use the image feature only.
273
274Build History SDK Information
275-----------------------------
276
277Build history collects similar information on the contents of SDKs (e.g.
278``bitbake -c populate_sdk imagename``) as compared to information it
279collects for images. Furthermore, this information differs depending on
280whether an extensible or standard SDK is being produced.
281
282The following list shows the files produced for SDKs:
283
284- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
285 owner, group, size, and symlink information. This list includes both
286 the host and target parts of the SDK.
287
288- ``sdk-info.txt:`` A text file containing name-value pairs with
289 information about the SDK. See the following listing example for more
290 information.
291
292- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
293 with information about task group sizes (e.g. :ref:`ref-tasks-populate_sysroot`
294 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
295 only when an extensible SDK is created.
296
297- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
298 with information for the shared-state packages and sizes in the SDK.
299 The ``sstate-package-sizes.txt`` file exists only when an extensible
300 SDK is created.
301
302- ``sdk-files:`` A folder that contains copies of the files mentioned
303 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
304 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
305 specific to the extensible SDK although you can set it differently if
306 you would like to pull in specific files from the standard SDK.
307
308 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
309 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
310 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
311 copied into the ``sdk-files`` directory.
312
313- The following information appears under each of the ``host`` and
314 ``target`` directories for the portions of the SDK that run on the
315 host and on the target, respectively:
316
317 .. note::
318
319 The following files for the most part are empty when producing an
320 extensible SDK because this type of SDK is not constructed from
321 packages as is the standard SDK.
322
323 - ``depends.dot:`` Dependency graph for the SDK that is compatible
324 with ``graphviz``.
325
326 - ``installed-package-names.txt:`` A list of installed packages by
327 name only.
328
329 - ``installed-package-sizes.txt:`` A list of installed packages
330 ordered by size.
331
332 - ``installed-packages.txt:`` A list of installed packages with full
333 package filenames.
334
335Here is an example of ``sdk-info.txt``:
336
337.. code-block:: none
338
339 DISTRO = poky
340 DISTRO_VERSION = 1.3+snapshot-20130327
341 SDK_NAME = poky-glibc-i686-arm
342 SDK_VERSION = 1.3+snapshot
343 SDKMACHINE =
344 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
345 BAD_RECOMMENDATIONS =
346 SDKSIZE = 352712
347
348Other than ``SDKSIZE``, which is
349the total size of the files in the SDK in Kbytes, the name-value pairs
350are variables that might have influenced the content of the SDK. This
351information is often useful when you are trying to determine why a
352change in the package or file listings has occurred.
353
354Examining Build History Information
355-----------------------------------
356
357You can examine build history output from the command line or from a web
358interface.
359
360To see any changes that have occurred (assuming you have
361:term:`BUILDHISTORY_COMMIT` = "1"),
362you can simply use any Git command that allows you to view the history
363of a repository. Here is one method::
364
365 $ git log -p
366
367You need to realize,
368however, that this method does show changes that are not significant
369(e.g. a package's size changing by a few bytes).
370
371There is a command-line tool called ``buildhistory-diff``, though,
372that queries the Git repository and prints just the differences that
373might be significant in human-readable form. Here is an example::
374
375 $ poky/poky/scripts/buildhistory-diff . HEAD^
376 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
377 /etc/anotherpkg.conf was added
378 /sbin/anotherpkg was added
379 * (installed-package-names.txt):
380 * anotherpkg was added
381 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
382 anotherpkg was added
383 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
384 * PR changed from "r0" to "r1"
385 * PV changed from "0.1.10" to "0.1.12"
386 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
387 * PR changed from "r0" to "r1"
388 * PV changed from "0.1.10" to "0.1.12"
389
390.. note::
391
392 The ``buildhistory-diff`` tool requires the ``GitPython``
393 package. Be sure to install it using Pip3 as follows::
394
395 $ pip3 install GitPython --user
396
397
398 Alternatively, you can install ``python3-git`` using the appropriate
399 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``).
400
401To see changes to the build history using a web interface, follow the
402instruction in the ``README`` file
403:yocto_git:`here </buildhistory-web/>`.
404
405Here is a sample screenshot of the interface:
406
407.. image:: figures/buildhistory-web.png
408 :width: 100%
409
diff --git a/documentation/dev-manual/building.rst b/documentation/dev-manual/building.rst
new file mode 100644
index 0000000000..a395793493
--- /dev/null
+++ b/documentation/dev-manual/building.rst
@@ -0,0 +1,939 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Building
4********
5
6This section describes various build procedures, such as the steps
7needed for a simple build, building a target for multiple configurations,
8generating an image for more than one machine, and so forth.
9
10Building a Simple Image
11=======================
12
13In the development environment, you need to build an image whenever you
14change hardware support, add or change system libraries, or add or
15change services that have dependencies. There are several methods that allow
16you to build an image within the Yocto Project. This section presents
17the basic steps you need to build a simple image using BitBake from a
18build host running Linux.
19
20.. note::
21
22 - For information on how to build an image using
23 :term:`Toaster`, see the
24 :doc:`/toaster-manual/index`.
25
26 - For information on how to use ``devtool`` to build images, see the
27 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
28 section in the Yocto Project Application Development and the
29 Extensible Software Development Kit (eSDK) manual.
30
31 - For a quick example on how to build an image using the
32 OpenEmbedded build system, see the
33 :doc:`/brief-yoctoprojectqs/index` document.
34
35The build process creates an entire Linux distribution from source and
36places it in your :term:`Build Directory` under ``tmp/deploy/images``. For
37detailed information on the build process using BitBake, see the
38":ref:`overview-manual/concepts:images`" section in the Yocto Project Overview
39and Concepts Manual.
40
41The following figure and list overviews the build process:
42
43.. image:: figures/bitbake-build-flow.png
44 :width: 100%
45
46#. *Set up Your Host Development System to Support Development Using the
47 Yocto Project*: See the ":doc:`start`" section for options on how to get a
48 build host ready to use the Yocto Project.
49
50#. *Initialize the Build Environment:* Initialize the build environment
51 by sourcing the build environment script (i.e.
52 :ref:`structure-core-script`)::
53
54 $ source oe-init-build-env [build_dir]
55
56 When you use the initialization script, the OpenEmbedded build system
57 uses ``build`` as the default :term:`Build Directory` in your current work
58 directory. You can use a `build_dir` argument with the script to
59 specify a different :term:`Build Directory`.
60
61 .. note::
62
63 A common practice is to use a different :term:`Build Directory` for
64 different targets; for example, ``~/build/x86`` for a ``qemux86``
65 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
66 event, it's typically cleaner to locate the :term:`Build Directory`
67 somewhere outside of your source directory.
68
69#. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
70 ``conf/local.conf`` configuration file, which is found in the
71 :term:`Build Directory`, is set up how you want it. This file defines many
72 aspects of the build environment including the target machine architecture
73 through the :term:`MACHINE` variable, the packaging format used during
74 the build (:term:`PACKAGE_CLASSES`), and a centralized tarball download
75 directory through the :term:`DL_DIR` variable.
76
77#. *Build the Image:* Build the image using the ``bitbake`` command::
78
79 $ bitbake target
80
81 .. note::
82
83 For information on BitBake, see the :doc:`bitbake:index`.
84
85 The target is the name of the recipe you want to build. Common
86 targets are the images in ``meta/recipes-core/images``,
87 ``meta/recipes-sato/images``, and so forth all found in the
88 :term:`Source Directory`. Alternatively, the target
89 can be the name of a recipe for a specific piece of software such as
90 BusyBox. For more details about the images the OpenEmbedded build
91 system supports, see the
92 ":ref:`ref-manual/images:Images`" chapter in the Yocto
93 Project Reference Manual.
94
95 As an example, the following command builds the
96 ``core-image-minimal`` image::
97
98 $ bitbake core-image-minimal
99
100 Once an
101 image has been built, it often needs to be installed. The images and
102 kernels built by the OpenEmbedded build system are placed in the
103 :term:`Build Directory` in ``tmp/deploy/images``. For information on how to
104 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
105 :doc:`/sdk-manual/index` manual. For
106 information about how to install these images, see the documentation
107 for your particular board or machine.
108
109Building Images for Multiple Targets Using Multiple Configurations
110==================================================================
111
112You can use a single ``bitbake`` command to build multiple images or
113packages for different targets where each image or package requires a
114different configuration (multiple configuration builds). The builds, in
115this scenario, are sometimes referred to as "multiconfigs", and this
116section uses that term throughout.
117
118This section describes how to set up for multiple configuration builds
119and how to account for cross-build dependencies between the
120multiconfigs.
121
122Setting Up and Running a Multiple Configuration Build
123-----------------------------------------------------
124
125To accomplish a multiple configuration build, you must define each
126target's configuration separately using a parallel configuration file in
127the :term:`Build Directory` or configuration directory within a layer, and you
128must follow a required file hierarchy. Additionally, you must enable the
129multiple configuration builds in your ``local.conf`` file.
130
131Follow these steps to set up and execute multiple configuration builds:
132
133- *Create Separate Configuration Files*: You need to create a single
134 configuration file for each build target (each multiconfig).
135 The configuration definitions are implementation dependent but often
136 each configuration file will define the machine and the
137 temporary directory BitBake uses for the build. Whether the same
138 temporary directory (:term:`TMPDIR`) can be shared will depend on what is
139 similar and what is different between the configurations. Multiple MACHINE
140 targets can share the same (:term:`TMPDIR`) as long as the rest of the
141 configuration is the same, multiple :term:`DISTRO` settings would need separate
142 (:term:`TMPDIR`) directories.
143
144 For example, consider a scenario with two different multiconfigs for the same
145 :term:`MACHINE`: "qemux86" built
146 for two distributions such as "poky" and "poky-lsb". In this case,
147 you would need to use the different :term:`TMPDIR`.
148
149 Here is an example showing the minimal statements needed in a
150 configuration file for a "qemux86" target whose temporary build
151 directory is ``tmpmultix86``::
152
153 MACHINE = "qemux86"
154 TMPDIR = "${TOPDIR}/tmpmultix86"
155
156 The location for these multiconfig configuration files is specific.
157 They must reside in the current :term:`Build Directory` in a sub-directory of
158 ``conf`` named ``multiconfig`` or within a layer's ``conf`` directory
159 under a directory named ``multiconfig``. Following is an example that defines
160 two configuration files for the "x86" and "arm" multiconfigs:
161
162 .. image:: figures/multiconfig_files.png
163 :align: center
164 :width: 50%
165
166 The usual :term:`BBPATH` search path is used to locate multiconfig files in
167 a similar way to other conf files.
168
169- *Add the BitBake Multi-configuration Variable to the Local
170 Configuration File*: Use the
171 :term:`BBMULTICONFIG`
172 variable in your ``conf/local.conf`` configuration file to specify
173 each multiconfig. Continuing with the example from the previous
174 figure, the :term:`BBMULTICONFIG` variable needs to enable two
175 multiconfigs: "x86" and "arm" by specifying each configuration file::
176
177 BBMULTICONFIG = "x86 arm"
178
179 .. note::
180
181 A "default" configuration already exists by definition. This
182 configuration is named: "" (i.e. empty string) and is defined by
183 the variables coming from your ``local.conf``
184 file. Consequently, the previous example actually adds two
185 additional configurations to your build: "arm" and "x86" along
186 with "".
187
188- *Launch BitBake*: Use the following BitBake command form to launch
189 the multiple configuration build::
190
191 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
192
193 For the example in this section, the following command applies::
194
195 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
196
197 The previous BitBake command builds a ``core-image-minimal`` image
198 that is configured through the ``x86.conf`` configuration file, a
199 ``core-image-sato`` image that is configured through the ``arm.conf``
200 configuration file and a ``core-image-base`` that is configured
201 through your ``local.conf`` configuration file.
202
203.. note::
204
205 Support for multiple configuration builds in the Yocto Project &DISTRO;
206 (&DISTRO_NAME;) Release does not include Shared State (sstate)
207 optimizations. Consequently, if a build uses the same object twice
208 in, for example, two different :term:`TMPDIR`
209 directories, the build either loads from an existing sstate cache for
210 that build at the start or builds the object fresh.
211
212Enabling Multiple Configuration Build Dependencies
213--------------------------------------------------
214
215Sometimes dependencies can exist between targets (multiconfigs) in a
216multiple configuration build. For example, suppose that in order to
217build a ``core-image-sato`` image for an "x86" multiconfig, the root
218filesystem of an "arm" multiconfig must exist. This dependency is
219essentially that the
220:ref:`ref-tasks-image` task in the
221``core-image-sato`` recipe depends on the completion of the
222:ref:`ref-tasks-rootfs` task of the
223``core-image-minimal`` recipe.
224
225To enable dependencies in a multiple configuration build, you must
226declare the dependencies in the recipe using the following statement
227form::
228
229 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
230
231To better show how to use this statement, consider the example scenario
232from the first paragraph of this section. The following statement needs
233to be added to the recipe that builds the ``core-image-sato`` image::
234
235 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
236
237In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
238task on which the :ref:`ref-tasks-image` task in the recipe depends is the
239:ref:`ref-tasks-rootfs` task from the ``core-image-minimal`` recipe associated
240with the "arm" multiconfig.
241
242Once you set up this dependency, you can build the "x86" multiconfig
243using a BitBake command as follows::
244
245 $ bitbake mc:x86:core-image-sato
246
247This command executes all the tasks needed to create the
248``core-image-sato`` image for the "x86" multiconfig. Because of the
249dependency, BitBake also executes through the :ref:`ref-tasks-rootfs` task for the
250"arm" multiconfig build.
251
252Having a recipe depend on the root filesystem of another build might not
253seem that useful. Consider this change to the statement in the
254``core-image-sato`` recipe::
255
256 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
257
258In this case, BitBake must
259create the ``core-image-minimal`` image for the "arm" build since the
260"x86" build depends on it.
261
262Because "x86" and "arm" are enabled for multiple configuration builds
263and have separate configuration files, BitBake places the artifacts for
264each build in the respective temporary build directories (i.e.
265:term:`TMPDIR`).
266
267Building an Initial RAM Filesystem (Initramfs) Image
268====================================================
269
270An initial RAM filesystem (:term:`Initramfs`) image provides a temporary root
271filesystem used for early system initialization, typically providing tools and
272loading modules needed to locate and mount the final root filesystem.
273
274Follow these steps to create an :term:`Initramfs` image:
275
276#. *Create the Initramfs Image Recipe:* You can reference the
277 ``core-image-minimal-initramfs.bb`` recipe found in the
278 ``meta/recipes-core`` directory of the :term:`Source Directory`
279 as an example from which to work.
280
281#. *Decide if You Need to Bundle the Initramfs Image Into the Kernel
282 Image:* If you want the :term:`Initramfs` image that is built to be bundled
283 in with the kernel image, set the :term:`INITRAMFS_IMAGE_BUNDLE`
284 variable to ``"1"`` in your ``local.conf`` configuration file and set the
285 :term:`INITRAMFS_IMAGE` variable in the recipe that builds the kernel image.
286
287 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the :term:`Initramfs`
288 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
289 :term:`Initramfs` image is then passed to the kernel's ``Makefile`` using the
290 :term:`CONFIG_INITRAMFS_SOURCE` variable, allowing the :term:`Initramfs`
291 image to be built into the kernel normally.
292
293#. *Optionally Add Items to the Initramfs Image Through the Initramfs
294 Image Recipe:* If you add items to the :term:`Initramfs` image by way of its
295 recipe, you should use :term:`PACKAGE_INSTALL` rather than
296 :term:`IMAGE_INSTALL`. :term:`PACKAGE_INSTALL` gives more direct control of
297 what is added to the image as compared to the defaults you might not
298 necessarily want that are set by the :ref:`ref-classes-image`
299 or :ref:`ref-classes-core-image` classes.
300
301#. *Build the Kernel Image and the Initramfs Image:* Build your kernel
302 image using BitBake. Because the :term:`Initramfs` image recipe is a
303 dependency of the kernel image, the :term:`Initramfs` image is built as well
304 and bundled with the kernel image if you used the
305 :term:`INITRAMFS_IMAGE_BUNDLE` variable described earlier.
306
307Bundling an Initramfs Image From a Separate Multiconfig
308-------------------------------------------------------
309
310There may be a case where we want to build an :term:`Initramfs` image which does not
311inherit the same distro policy as our main image, for example, we may want
312our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our :term:`Initramfs`
313image to keep a smaller footprint. However, by performing the steps mentioned
314above the :term:`Initramfs` image will inherit ``TCLIBC="glibc"`` without allowing us
315to override it.
316
317To achieve this, you need to perform some additional steps:
318
319#. *Create a multiconfig for your Initramfs image:* You can perform the steps
320 on ":ref:`dev-manual/building:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
321 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
322 contains the variables::
323
324 TMPDIR="${TOPDIR}/tmp-initramfscfg"
325 TCLIBC="musl"
326
327#. *Set additional Initramfs variables on your main configuration:*
328 Additionally, on your main configuration (``local.conf``) you need to set the
329 variables::
330
331 INITRAMFS_MULTICONFIG = "initramfscfg"
332 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
333
334 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
335 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
336 to be built coming from the ``initramfscfg`` multiconfig, and to let the
337 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
338
339 Building a system with such configuration will build the kernel using the
340 main configuration but the :ref:`ref-tasks-bundle_initramfs` task will grab the
341 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
342 instead, resulting in a musl based :term:`Initramfs` image bundled in the kernel
343 but a glibc based main image.
344
345 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
346 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
347
348
349Building a Tiny System
350======================
351
352Very small distributions have some significant advantages such as
353requiring less on-die or in-package memory (cheaper), better performance
354through efficient cache usage, lower power requirements due to less
355memory, faster boot times, and reduced development overhead. Some
356real-world examples where a very small distribution gives you distinct
357advantages are digital cameras, medical devices, and small headless
358systems.
359
360This section presents information that shows you how you can trim your
361distribution to even smaller sizes than the ``poky-tiny`` distribution,
362which is around 5 Mbytes, that can be built out-of-the-box using the
363Yocto Project.
364
365Tiny System Overview
366--------------------
367
368The following list presents the overall steps you need to consider and
369perform to create distributions with smaller root filesystems, achieve
370faster boot times, maintain your critical functionality, and avoid
371initial RAM disks:
372
373- :ref:`Determine your goals and guiding principles
374 <dev-manual/building:goals and guiding principles>`
375
376- :ref:`dev-manual/building:understand what contributes to your image size`
377
378- :ref:`Reduce the size of the root filesystem
379 <dev-manual/building:trim the root filesystem>`
380
381- :ref:`Reduce the size of the kernel <dev-manual/building:trim the kernel>`
382
383- :ref:`dev-manual/building:remove package management requirements`
384
385- :ref:`dev-manual/building:look for other ways to minimize size`
386
387- :ref:`dev-manual/building:iterate on the process`
388
389Goals and Guiding Principles
390----------------------------
391
392Before you can reach your destination, you need to know where you are
393going. Here is an example list that you can use as a guide when creating
394very small distributions:
395
396- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
397 less and a root filesystem that is 3 Mbytes or less).
398
399- Find the areas that are currently taking 90% of the space and
400 concentrate on reducing those areas.
401
402- Do not create any difficult "hacks" to achieve your goals.
403
404- Leverage the device-specific options.
405
406- Work in a separate layer so that you keep changes isolated. For
407 information on how to create layers, see the
408 ":ref:`dev-manual/layers:understanding and creating layers`" section.
409
410Understand What Contributes to Your Image Size
411----------------------------------------------
412
413It is easiest to have something to start with when creating your own
414distribution. You can use the Yocto Project out-of-the-box to create the
415``poky-tiny`` distribution. Ultimately, you will want to make changes in
416your own distribution that are likely modeled after ``poky-tiny``.
417
418.. note::
419
420 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
421 ``local.conf`` file to "poky-tiny" as described in the
422 ":ref:`dev-manual/custom-distribution:creating your own distribution`"
423 section.
424
425Understanding some memory concepts will help you reduce the system size.
426Memory consists of static, dynamic, and temporary memory. Static memory
427is the TEXT (code), DATA (initialized data in the code), and BSS
428(uninitialized data) sections. Dynamic memory represents memory that is
429allocated at runtime: stacks, hash tables, and so forth. Temporary
430memory is recovered after the boot process. This memory consists of
431memory used for decompressing the kernel and for the ``__init__``
432functions.
433
434To help you see where you currently are with kernel and root filesystem
435sizes, you can use two tools found in the :term:`Source Directory`
436in the
437``scripts/tiny/`` directory:
438
439- ``ksize.py``: Reports component sizes for the kernel build objects.
440
441- ``dirsize.py``: Reports component sizes for the root filesystem.
442
443This next tool and command help you organize configuration fragments and
444view file dependencies in a human-readable form:
445
446- ``merge_config.sh``: Helps you manage configuration files and
447 fragments within the kernel. With this tool, you can merge individual
448 configuration fragments together. The tool allows you to make
449 overrides and warns you of any missing configuration options. The
450 tool is ideal for allowing you to iterate on configurations, create
451 minimal configurations, and create configuration files for different
452 machines without having to duplicate your process.
453
454 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
455 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
456 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
457 directory.
458
459 For more information on configuration fragments, see the
460 ":ref:`kernel-dev/common:creating configuration fragments`"
461 section in the Yocto Project Linux Kernel Development Manual.
462
463- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
464 with these options brings up a Dependency Explorer from which you can
465 view file dependencies. Understanding these dependencies allows you
466 to make informed decisions when cutting out various pieces of the
467 kernel and root filesystem.
468
469Trim the Root Filesystem
470------------------------
471
472The root filesystem is made up of packages for booting, libraries, and
473applications. To change things, you can configure how the packaging
474happens, which changes the way you build them. You can also modify the
475filesystem itself or select a different filesystem.
476
477First, find out what is hogging your root filesystem by running the
478``dirsize.py`` script from your root directory::
479
480 $ cd root-directory-of-image
481 $ dirsize.py 100000 > dirsize-100k.log
482 $ cat dirsize-100k.log
483
484You can apply a filter to the script to ignore files
485under a certain size. The previous example filters out any files below
486100 Kbytes. The sizes reported by the tool are uncompressed, and thus
487will be smaller by a relatively constant factor in a compressed root
488filesystem. When you examine your log file, you can focus on areas of
489the root filesystem that take up large amounts of memory.
490
491You need to be sure that what you eliminate does not cripple the
492functionality you need. One way to see how packages relate to each other
493is by using the Dependency Explorer UI with the BitBake command::
494
495 $ cd image-directory
496 $ bitbake -u taskexp -g image
497
498Use the interface to
499select potential packages you wish to eliminate and see their dependency
500relationships.
501
502When deciding how to reduce the size, get rid of packages that result in
503minimal impact on the feature set. For example, you might not need a VGA
504display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
505instead of ``udev``.
506
507Use your ``local.conf`` file to make changes. For example, to eliminate
508``udev`` and ``glib``, set the following in the local configuration
509file::
510
511 VIRTUAL-RUNTIME_dev_manager = ""
512
513Finally, you should consider exactly the type of root filesystem you
514need to meet your needs while also reducing its size. For example,
515consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
516:term:`Initramfs` using ``initramfs``. Be aware that ``ext3`` requires a 1
517Mbyte journal. If you are okay with running read-only, you do not need
518this journal.
519
520.. note::
521
522 After each round of elimination, you need to rebuild your system and
523 then use the tools to see the effects of your reductions.
524
525Trim the Kernel
526---------------
527
528The kernel is built by including policies for hardware-independent
529aspects. What subsystems do you enable? For what architecture are you
530building? Which drivers do you build by default?
531
532.. note::
533
534 You can modify the kernel source if you want to help with boot time.
535
536Run the ``ksize.py`` script from the top-level Linux build directory to
537get an idea of what is making up the kernel::
538
539 $ cd top-level-linux-build-directory
540 $ ksize.py > ksize.log
541 $ cat ksize.log
542
543When you examine the log, you will see how much space is taken up with
544the built-in ``.o`` files for drivers, networking, core kernel files,
545filesystem, sound, and so forth. The sizes reported by the tool are
546uncompressed, and thus will be smaller by a relatively constant factor
547in a compressed kernel image. Look to reduce the areas that are large
548and taking up around the "90% rule."
549
550To examine, or drill down, into any particular area, use the ``-d``
551option with the script::
552
553 $ ksize.py -d > ksize.log
554
555Using this option
556breaks out the individual file information for each area of the kernel
557(e.g. drivers, networking, and so forth).
558
559Use your log file to see what you can eliminate from the kernel based on
560features you can let go. For example, if you are not going to need
561sound, you do not need any drivers that support sound.
562
563After figuring out what to eliminate, you need to reconfigure the kernel
564to reflect those changes during the next build. You could run
565``menuconfig`` and make all your changes at once. However, that makes it
566difficult to see the effects of your individual eliminations and also
567makes it difficult to replicate the changes for perhaps another target
568device. A better method is to start with no configurations using
569``allnoconfig``, create configuration fragments for individual changes,
570and then manage the fragments into a single configuration file using
571``merge_config.sh``. The tool makes it easy for you to iterate using the
572configuration change and build cycle.
573
574Each time you make configuration changes, you need to rebuild the kernel
575and check to see what impact your changes had on the overall size.
576
577Remove Package Management Requirements
578--------------------------------------
579
580Packaging requirements add size to the image. One way to reduce the size
581of the image is to remove all the packaging requirements from the image.
582This reduction includes both removing the package manager and its unique
583dependencies as well as removing the package management data itself.
584
585To eliminate all the packaging requirements for an image, be sure that
586"package-management" is not part of your
587:term:`IMAGE_FEATURES`
588statement for the image. When you remove this feature, you are removing
589the package manager as well as its dependencies from the root
590filesystem.
591
592Look for Other Ways to Minimize Size
593------------------------------------
594
595Depending on your particular circumstances, other areas that you can
596trim likely exist. The key to finding these areas is through tools and
597methods described here combined with experimentation and iteration. Here
598are a couple of areas to experiment with:
599
600- ``glibc``: In general, follow this process:
601
602 #. Remove ``glibc`` features from
603 :term:`DISTRO_FEATURES`
604 that you think you do not need.
605
606 #. Build your distribution.
607
608 #. If the build fails due to missing symbols in a package, determine
609 if you can reconfigure the package to not need those features. For
610 example, change the configuration to not support wide character
611 support as is done for ``ncurses``. Or, if support for those
612 characters is needed, determine what ``glibc`` features provide
613 the support and restore the configuration.
614
615 4. Rebuild and repeat the process.
616
617- ``busybox``: For BusyBox, use a process similar as described for
618 ``glibc``. A difference is you will need to boot the resulting system
619 to see if you are able to do everything you expect from the running
620 system. You need to be sure to integrate configuration fragments into
621 Busybox because BusyBox handles its own core features and then allows
622 you to add configuration fragments on top.
623
624Iterate on the Process
625----------------------
626
627If you have not reached your goals on system size, you need to iterate
628on the process. The process is the same. Use the tools and see just what
629is taking up 90% of the root filesystem and the kernel. Decide what you
630can eliminate without limiting your device beyond what you need.
631
632Depending on your system, a good place to look might be Busybox, which
633provides a stripped down version of Unix tools in a single, executable
634file. You might be able to drop virtual terminal services or perhaps
635ipv6.
636
637Building Images for More than One Machine
638=========================================
639
640A common scenario developers face is creating images for several
641different machines that use the same software environment. In this
642situation, it is tempting to set the tunings and optimization flags for
643each build specifically for the targeted hardware (i.e. "maxing out" the
644tunings). Doing so can considerably add to build times and package feed
645maintenance collectively for the machines. For example, selecting tunes
646that are extremely specific to a CPU core used in a system might enable
647some micro optimizations in GCC for that particular system but would
648otherwise not gain you much of a performance difference across the other
649systems as compared to using a more general tuning across all the builds
650(e.g. setting :term:`DEFAULTTUNE`
651specifically for each machine's build). Rather than "max out" each
652build's tunings, you can take steps that cause the OpenEmbedded build
653system to reuse software across the various machines where it makes
654sense.
655
656If build speed and package feed maintenance are considerations, you
657should consider the points in this section that can help you optimize
658your tunings to best consider build times and package feed maintenance.
659
660- *Share the :term:`Build Directory`:* If at all possible, share the
661 :term:`TMPDIR` across builds. The Yocto Project supports switching between
662 different :term:`MACHINE` values in the same :term:`TMPDIR`. This practice
663 is well supported and regularly used by developers when building for
664 multiple machines. When you use the same :term:`TMPDIR` for multiple
665 machine builds, the OpenEmbedded build system can reuse the existing native
666 and often cross-recipes for multiple machines. Thus, build time decreases.
667
668 .. note::
669
670 If :term:`DISTRO` settings change or fundamental configuration settings
671 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
672 Sharing :term:`TMPDIR` under these circumstances might work but since it is
673 not guaranteed, you should use a clean :term:`TMPDIR`.
674
675- *Enable the Appropriate Package Architecture:* By default, the
676 OpenEmbedded build system enables three levels of package
677 architectures: "all", "tune" or "package", and "machine". Any given
678 recipe usually selects one of these package architectures (types) for
679 its output. Depending for what a given recipe creates packages,
680 making sure you enable the appropriate package architecture can
681 directly impact the build time.
682
683 A recipe that just generates scripts can enable "all" architecture
684 because there are no binaries to build. To specifically enable "all"
685 architecture, be sure your recipe inherits the
686 :ref:`ref-classes-allarch` class.
687 This class is useful for "all" architectures because it configures
688 many variables so packages can be used across multiple architectures.
689
690 If your recipe needs to generate packages that are machine-specific
691 or when one of the build or runtime dependencies is already
692 machine-architecture dependent, which makes your recipe also
693 machine-architecture dependent, make sure your recipe enables the
694 "machine" package architecture through the
695 :term:`MACHINE_ARCH`
696 variable::
697
698 PACKAGE_ARCH = "${MACHINE_ARCH}"
699
700 When you do not
701 specifically enable a package architecture through the
702 :term:`PACKAGE_ARCH`, The
703 OpenEmbedded build system defaults to the
704 :term:`TUNE_PKGARCH` setting::
705
706 PACKAGE_ARCH = "${TUNE_PKGARCH}"
707
708- *Choose a Generic Tuning File if Possible:* Some tunes are more
709 generic and can run on multiple targets (e.g. an ``armv5`` set of
710 packages could run on ``armv6`` and ``armv7`` processors in most
711 cases). Similarly, ``i486`` binaries could work on ``i586`` and
712 higher processors. You should realize, however, that advances on
713 newer processor versions would not be used.
714
715 If you select the same tune for several different machines, the
716 OpenEmbedded build system reuses software previously built, thus
717 speeding up the overall build time. Realize that even though a new
718 sysroot for each machine is generated, the software is not recompiled
719 and only one package feed exists.
720
721- *Manage Granular Level Packaging:* Sometimes there are cases where
722 injecting another level of package architecture beyond the three
723 higher levels noted earlier can be useful. For example, consider how
724 NXP (formerly Freescale) allows for the easy reuse of binary packages
725 in their layer
726 :yocto_git:`meta-freescale </meta-freescale/>`.
727 In this example, the
728 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
729 class shares GPU packages for i.MX53 boards because all boards share
730 the AMD GPU. The i.MX6-based boards can do the same because all
731 boards share the Vivante GPU. This class inspects the BitBake
732 datastore to identify if the package provides or depends on one of
733 the sub-architecture values. If so, the class sets the
734 :term:`PACKAGE_ARCH` value
735 based on the ``MACHINE_SUBARCH`` value. If the package does not
736 provide or depend on one of the sub-architecture values but it
737 matches a value in the machine-specific filter, it sets
738 :term:`MACHINE_ARCH`. This
739 behavior reduces the number of packages built and saves build time by
740 reusing binaries.
741
742- *Use Tools to Debug Issues:* Sometimes you can run into situations
743 where software is being rebuilt when you think it should not be. For
744 example, the OpenEmbedded build system might not be using shared
745 state between machines when you think it should be. These types of
746 situations are usually due to references to machine-specific
747 variables such as :term:`MACHINE`,
748 :term:`SERIAL_CONSOLES`,
749 :term:`XSERVER`,
750 :term:`MACHINE_FEATURES`,
751 and so forth in code that is supposed to only be tune-specific or
752 when the recipe depends
753 (:term:`DEPENDS`,
754 :term:`RDEPENDS`,
755 :term:`RRECOMMENDS`,
756 :term:`RSUGGESTS`, and so forth)
757 on some other recipe that already has
758 :term:`PACKAGE_ARCH` defined
759 as "${MACHINE_ARCH}".
760
761 .. note::
762
763 Patches to fix any issues identified are most welcome as these
764 issues occasionally do occur.
765
766 For such cases, you can use some tools to help you sort out the
767 situation:
768
769 - ``state-diff-machines.sh``*:* You can find this tool in the
770 ``scripts`` directory of the Source Repositories. See the comments
771 in the script for information on how to use the tool.
772
773 - *BitBake's "-S printdiff" Option:* Using this option causes
774 BitBake to try to establish the closest signature match it can
775 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
776 over the matches to determine the stamps and delta where these two
777 stamp trees diverge.
778
779Building Software from an External Source
780=========================================
781
782By default, the OpenEmbedded build system uses the :term:`Build Directory`
783when building source code. The build process involves fetching the source
784files, unpacking them, and then patching them if necessary before the build
785takes place.
786
787There are situations where you might want to build software from source
788files that are external to and thus outside of the OpenEmbedded build
789system. For example, suppose you have a project that includes a new BSP
790with a heavily customized kernel. And, you want to minimize exposing the
791build system to the development team so that they can focus on their
792project and maintain everyone's workflow as much as possible. In this
793case, you want a kernel source directory on the development machine
794where the development occurs. You want the recipe's
795:term:`SRC_URI` variable to point to
796the external directory and use it as is, not copy it.
797
798To build from software that comes from an external source, all you need to do
799is inherit the :ref:`ref-classes-externalsrc` class and then set
800the :term:`EXTERNALSRC` variable to point to your external source code. Here
801are the statements to put in your ``local.conf`` file::
802
803 INHERIT += "externalsrc"
804 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
805
806This next example shows how to accomplish the same thing by setting
807:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
808
809 EXTERNALSRC = "path"
810 EXTERNALSRC_BUILD = "path"
811
812.. note::
813
814 In order for these settings to take effect, you must globally or
815 locally inherit the :ref:`ref-classes-externalsrc` class.
816
817By default, :ref:`ref-classes-externalsrc` builds the source code in a
818directory separate from the external source directory as specified by
819:term:`EXTERNALSRC`. If you need
820to have the source built in the same directory in which it resides, or
821some other nominated directory, you can set
822:term:`EXTERNALSRC_BUILD`
823to point to that directory::
824
825 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
826
827Replicating a Build Offline
828===========================
829
830It can be useful to take a "snapshot" of upstream sources used in a
831build and then use that "snapshot" later to replicate the build offline.
832To do so, you need to first prepare and populate your downloads
833directory your "snapshot" of files. Once your downloads directory is
834ready, you can use it at any time and from any machine to replicate your
835build.
836
837Follow these steps to populate your Downloads directory:
838
839#. *Create a Clean Downloads Directory:* Start with an empty downloads
840 directory (:term:`DL_DIR`). You
841 start with an empty downloads directory by either removing the files
842 in the existing directory or by setting :term:`DL_DIR` to point to either
843 an empty location or one that does not yet exist.
844
845#. *Generate Tarballs of the Source Git Repositories:* Edit your
846 ``local.conf`` configuration file as follows::
847
848 DL_DIR = "/home/your-download-dir/"
849 BB_GENERATE_MIRROR_TARBALLS = "1"
850
851 During
852 the fetch process in the next step, BitBake gathers the source files
853 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
854 the
855 :term:`BB_GENERATE_MIRROR_TARBALLS`
856 variable for more information.
857
858#. *Populate Your Downloads Directory Without Building:* Use BitBake to
859 fetch your sources but inhibit the build::
860
861 $ bitbake target --runonly=fetch
862
863 The downloads directory (i.e. ``${DL_DIR}``) now has
864 a "snapshot" of the source files in the form of tarballs, which can
865 be used for the build.
866
867#. *Optionally Remove Any Git or other SCM Subdirectories From the
868 Downloads Directory:* If you want, you can clean up your downloads
869 directory by removing any Git or other Source Control Management
870 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
871 already contain these subdirectories.
872
873Once your downloads directory has everything it needs regarding source
874files, you can create your "own-mirror" and build your target.
875Understand that you can use the files to build the target offline from
876any machine and at any time.
877
878Follow these steps to build your target using the files in the downloads
879directory:
880
881#. *Using Local Files Only:* Inside your ``local.conf`` file, add the
882 :term:`SOURCE_MIRROR_URL` variable, inherit the
883 :ref:`ref-classes-own-mirrors` class, and use the
884 :term:`BB_NO_NETWORK` variable to your ``local.conf``::
885
886 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
887 INHERIT += "own-mirrors"
888 BB_NO_NETWORK = "1"
889
890 The :term:`SOURCE_MIRROR_URL` and :ref:`ref-classes-own-mirrors`
891 class set up the system to use the downloads directory as your "own
892 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
893 BitBake's fetching process in step 3 stays local, which means files
894 from your "own-mirror" are used.
895
896#. *Start With a Clean Build:* You can start with a clean build by
897 removing the ``${``\ :term:`TMPDIR`\ ``}`` directory or using a new
898 :term:`Build Directory`.
899
900#. *Build Your Target:* Use BitBake to build your target::
901
902 $ bitbake target
903
904 The build completes using the known local "snapshot" of source
905 files from your mirror. The resulting tarballs for your "snapshot" of
906 source files are in the downloads directory.
907
908 .. note::
909
910 The offline build does not work if recipes attempt to find the
911 latest version of software by setting
912 :term:`SRCREV` to
913 ``${``\ :term:`AUTOREV`\ ``}``::
914
915 SRCREV = "${AUTOREV}"
916
917 When a recipe sets :term:`SRCREV` to
918 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
919 attempt to determine the latest version of software from the SCM.
920 Typically, recipes that use :term:`AUTOREV` are custom or modified
921 recipes. Recipes that reside in public repositories usually do not
922 use :term:`AUTOREV`.
923
924 If you do have recipes that use :term:`AUTOREV`, you can take steps to
925 still use the recipes in an offline build. Do the following:
926
927 #. Use a configuration generated by enabling :ref:`build
928 history <dev-manual/build-quality:maintaining build output quality>`.
929
930 #. Use the ``buildhistory-collect-srcrevs`` command to collect the
931 stored :term:`SRCREV` values from the build's history. For more
932 information on collecting these values, see the
933 ":ref:`dev-manual/build-quality:build history package information`"
934 section.
935
936 #. Once you have the correct source revisions, you can modify
937 those recipes to set :term:`SRCREV` to specific versions of the
938 software.
939
diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
deleted file mode 100644
index a2404e5f4e..0000000000
--- a/documentation/dev-manual/common-tasks.rst
+++ /dev/null
@@ -1,11414 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
21":ref:`overview-manual/yp-intro:the yocto project layer model`"
22section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
27.. note::
28
29 It is very easy to create your own layers to use with the OpenEmbedded
30 build system, as the Yocto Project ships with tools that speed up creating
31 layers. This section describes the steps you perform by hand to create
32 layers so that you can better understand them. For information about the
33 layer-creation tools, see the
34 ":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
35 section in the Yocto Project Board Support Package (BSP) Developer's
36 Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
37 section further down in this manual.
38
39Follow these general steps to create your layer without using tools:
40
411. *Check Existing Layers:* Before creating a new layer, you should be
42 sure someone has not already created a layer containing the Metadata
43 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
44 for a list of layers from the OpenEmbedded community that can be used in
45 the Yocto Project. You could find a layer that is identical or close
46 to what you need.
47
482. *Create a Directory:* Create the directory for your layer. When you
49 create the layer, be sure to create the directory in an area not
50 associated with the Yocto Project :term:`Source Directory`
51 (e.g. the cloned ``poky`` repository).
52
53 While not strictly required, prepend the name of the directory with
54 the string "meta-". For example::
55
56 meta-mylayer
57 meta-GUI_xyz
58 meta-mymachine
59
60 With rare exceptions, a layer's name follows this form::
61
62 meta-root_name
63
64 Following this layer naming convention can save
65 you trouble later when tools, components, or variables "assume" your
66 layer name begins with "meta-". A notable example is in configuration
67 files as shown in the following step where layer names without the
68 "meta-" string are appended to several variables used in the
69 configuration.
70
713. *Create a Layer Configuration File:* Inside your new layer folder,
72 you need to create a ``conf/layer.conf`` file. It is easiest to take
73 an existing layer configuration file and copy that to your layer's
74 ``conf`` directory and then modify the file as needed.
75
76 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
77 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
78 demonstrates the required syntax. For your layer, you need to replace
79 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
80 for a layer named "meta-machinexyz")::
81
82 # We have a conf and classes directory, add to BBPATH
83 BBPATH .= ":${LAYERDIR}"
84
85 # We have recipes-* directories, add to BBFILES
86 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
87 ${LAYERDIR}/recipes-*/*/*.bbappend"
88
89 BBFILE_COLLECTIONS += "yoctobsp"
90 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
91 BBFILE_PRIORITY_yoctobsp = "5"
92 LAYERVERSION_yoctobsp = "4"
93 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
94
95 Following is an explanation of the layer configuration file:
96
97 - :term:`BBPATH`: Adds the layer's
98 root directory to BitBake's search path. Through the use of the
99 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
100 configuration files, and files that are included with ``include``
101 and ``require`` statements. For these cases, BitBake uses the
102 first file that matches the name found in :term:`BBPATH`. This is
103 similar to the way the ``PATH`` variable is used for binaries. It
104 is recommended, therefore, that you use unique class and
105 configuration filenames in your custom layer.
106
107 - :term:`BBFILES`: Defines the
108 location for all recipes in the layer.
109
110 - :term:`BBFILE_COLLECTIONS`:
111 Establishes the current layer through a unique identifier that is
112 used throughout the OpenEmbedded build system to refer to the
113 layer. In this example, the identifier "yoctobsp" is the
114 representation for the container layer named "meta-yocto-bsp".
115
116 - :term:`BBFILE_PATTERN`:
117 Expands immediately during parsing to provide the directory of the
118 layer.
119
120 - :term:`BBFILE_PRIORITY`:
121 Establishes a priority to use for recipes in the layer when the
122 OpenEmbedded build finds recipes of the same name in different
123 layers.
124
125 - :term:`LAYERVERSION`:
126 Establishes a version number for the layer. You can use this
127 version number to specify this exact version of the layer as a
128 dependency when using the
129 :term:`LAYERDEPENDS`
130 variable.
131
132 - :term:`LAYERDEPENDS`:
133 Lists all layers on which this layer depends (if any).
134
135 - :term:`LAYERSERIES_COMPAT`:
136 Lists the :yocto_wiki:`Yocto Project </Releases>`
137 releases for which the current version is compatible. This
138 variable is a good way to indicate if your particular layer is
139 current.
140
1414. *Add Content:* Depending on the type of layer, add the content. If
142 the layer adds support for a machine, add the machine configuration
143 in a ``conf/machine/`` file within the layer. If the layer adds
144 distro policy, add the distro configuration in a ``conf/distro/``
145 file within the layer. If the layer introduces new recipes, put the
146 recipes you need in ``recipes-*`` subdirectories within the layer.
147
148 .. note::
149
150 For an explanation of layer hierarchy that is compliant with the
151 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
152 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
153
1545. *Optionally Test for Compatibility:* If you want permission to use
155 the Yocto Project Compatibility logo with your layer or application
156 that uses your layer, perform the steps to apply for compatibility.
157 See the
158 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
159 section for more information.
160
161Following Best Practices When Creating Layers
162---------------------------------------------
163
164To create layers that are easier to maintain and that will not impact
165builds for other machines, you should consider the information in the
166following list:
167
168- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
169 Configuration:* In other words, do not copy an entire recipe into
170 your layer and then modify it. Rather, use an append file
171 (``.bbappend``) to override only those parts of the original recipe
172 you need to modify.
173
174- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
175 for each recipe that uses an include file. Or, if you are introducing
176 a new recipe that requires the included file, use the path relative
177 to the original layer directory to refer to the file. For example,
178 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
179 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
180 the include file, it could indicate a deficiency in the include file
181 in the layer to which it originally belongs. If this is the case, you
182 should try to address that deficiency instead of overlaying the
183 include file. For example, you could address this by getting the
184 maintainer of the include file to add a variable or variables to make
185 it easy to override the parts needing to be overridden.
186
187- *Structure Your Layers:* Proper use of overrides within append files
188 and placement of machine-specific files within your layer can ensure
189 that a build is not using the wrong Metadata and negatively impacting
190 a build for a different machine. Following are some examples:
191
192 - *Modify Variables to Support a Different Machine:* Suppose you
193 have a layer named ``meta-one`` that adds support for building
194 machine "one". To do so, you use an append file named
195 ``base-files.bbappend`` and create a dependency on "foo" by
196 altering the :term:`DEPENDS`
197 variable::
198
199 DEPENDS = "foo"
200
201 The dependency is created during any
202 build that includes the layer ``meta-one``. However, you might not
203 want this dependency for all machines. For example, suppose you
204 are building for machine "two" but your ``bblayers.conf`` file has
205 the ``meta-one`` layer included. During the build, the
206 ``base-files`` for machine "two" will also have the dependency on
207 ``foo``.
208
209 To make sure your changes apply only when building machine "one",
210 use a machine override with the :term:`DEPENDS` statement::
211
212 DEPENDS:one = "foo"
213
214 You should follow the same strategy when using ``:append``
215 and ``:prepend`` operations::
216
217 DEPENDS:append:one = " foo"
218 DEPENDS:prepend:one = "foo "
219
220 As an actual example, here's a
221 snippet from the generic kernel include file ``linux-yocto.inc``,
222 wherein the kernel compile and link options are adjusted in the
223 case of a subset of the supported architectures::
224
225 DEPENDS:append:aarch64 = " libgcc"
226 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
227 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
228
229 DEPENDS:append:nios2 = " libgcc"
230 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
231 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
232
233 DEPENDS:append:arc = " libgcc"
234 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
235 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
236
237 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
238
239 - *Place Machine-Specific Files in Machine-Specific Locations:* When
240 you have a base recipe, such as ``base-files.bb``, that contains a
241 :term:`SRC_URI` statement to a
242 file, you can use an append file to cause the build to use your
243 own version of the file. For example, an append file in your layer
244 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
245 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
246
247 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
248
249 The build for machine "one" will pick up your machine-specific file as
250 long as you have the file in
251 ``meta-one/recipes-core/base-files/base-files/``. However, if you
252 are building for a different machine and the ``bblayers.conf``
253 file includes the ``meta-one`` layer and the location of your
254 machine-specific file is the first location where that file is
255 found according to :term:`FILESPATH`, builds for all machines will
256 also use that machine-specific file.
257
258 You can make sure that a machine-specific file is used for a
259 particular machine by putting the file in a subdirectory specific
260 to the machine. For example, rather than placing the file in
261 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
262 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
263 Not only does this make sure the file is used only when building
264 for machine "one", but the build process locates the file more
265 quickly.
266
267 In summary, you need to place all files referenced from
268 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
269 order to restrict those files to machine-specific builds.
270
271- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
272 permission to use the Yocto Project Compatibility logo with your
273 layer or application that uses your layer, perform the steps to apply
274 for compatibility. See the
275 ":ref:`dev-manual/common-tasks:making sure your layer is compatible with yocto project`"
276 section for more information.
277
278- *Follow the Layer Naming Convention:* Store custom layers in a Git
279 repository that use the ``meta-layer_name`` format.
280
281- *Group Your Layers Locally:* Clone your repository alongside other
282 cloned ``meta`` directories from the :term:`Source Directory`.
283
284Making Sure Your Layer is Compatible With Yocto Project
285-------------------------------------------------------
286
287When you create a layer used with the Yocto Project, it is advantageous
288to make sure that the layer interacts well with existing Yocto Project
289layers (i.e. the layer is compatible with the Yocto Project). Ensuring
290compatibility makes the layer easy to be consumed by others in the Yocto
291Project community and could allow you permission to use the Yocto
292Project Compatible Logo.
293
294.. note::
295
296 Only Yocto Project member organizations are permitted to use the
297 Yocto Project Compatible Logo. The logo is not available for general
298 use. For information on how to become a Yocto Project member
299 organization, see the :yocto_home:`Yocto Project Website <>`.
300
301The Yocto Project Compatibility Program consists of a layer application
302process that requests permission to use the Yocto Project Compatibility
303Logo for your layer and application. The process consists of two parts:
304
3051. Successfully passing a script (``yocto-check-layer``) that when run
306 against your layer, tests it against constraints based on experiences
307 of how layers have worked in the real world and where pitfalls have
308 been found. Getting a "PASS" result from the script is required for
309 successful compatibility registration.
310
3112. Completion of an application acceptance form, which you can find at
312 :yocto_home:`/webform/yocto-project-compatible-registration`.
313
314To be granted permission to use the logo, you need to satisfy the
315following:
316
317- Be able to check the box indicating that you got a "PASS" when
318 running the script against your layer.
319
320- Answer "Yes" to the questions on the form or have an acceptable
321 explanation for any questions answered "No".
322
323- Be a Yocto Project Member Organization.
324
325The remainder of this section presents information on the registration
326form and on the ``yocto-check-layer`` script.
327
328Yocto Project Compatible Program Application
329~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
330
331Use the form to apply for your layer's approval. Upon successful
332application, you can use the Yocto Project Compatibility Logo with your
333layer and the application that uses your layer.
334
335To access the form, use this link:
336:yocto_home:`/webform/yocto-project-compatible-registration`.
337Follow the instructions on the form to complete your application.
338
339The application consists of the following sections:
340
341- *Contact Information:* Provide your contact information as the fields
342 require. Along with your information, provide the released versions
343 of the Yocto Project for which your layer is compatible.
344
345- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
346 items in the checklist. There is space at the bottom of the form for
347 any explanations for items for which you answered "No".
348
349- *Recommendations:* Provide answers for the questions regarding Linux
350 kernel use and build success.
351
352``yocto-check-layer`` Script
353~~~~~~~~~~~~~~~~~~~~~~~~~~~~
354
355The ``yocto-check-layer`` script provides you a way to assess how
356compatible your layer is with the Yocto Project. You should run this
357script prior to using the form to apply for compatibility as described
358in the previous section. You need to achieve a "PASS" result in order to
359have your application form successfully processed.
360
361The script divides tests into three areas: COMMON, BSP, and DISTRO. For
362example, given a distribution layer (DISTRO), the layer must pass both
363the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
364layer, the layer must pass the COMMON and BSP set of tests.
365
366To execute the script, enter the following commands from your build
367directory::
368
369 $ source oe-init-build-env
370 $ yocto-check-layer your_layer_directory
371
372Be sure to provide the actual directory for your
373layer as part of the command.
374
375Entering the command causes the script to determine the type of layer
376and then to execute a set of specific tests against the layer. The
377following list overviews the test:
378
379- ``common.test_readme``: Tests if a ``README`` file exists in the
380 layer and the file is not empty.
381
382- ``common.test_parse``: Tests to make sure that BitBake can parse the
383 files without error (i.e. ``bitbake -p``).
384
385- ``common.test_show_environment``: Tests that the global or per-recipe
386 environment is in order without errors (i.e. ``bitbake -e``).
387
388- ``common.test_world``: Verifies that ``bitbake world`` works.
389
390- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
391 layers do not come with recipes that change signatures.
392
393- ``common.test_layerseries_compat``: Verifies layer compatibility is
394 set properly.
395
396- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
397 configurations.
398
399- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
400 set the machine when the layer is added.
401
402- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
403 regardless of which machine is selected.
404
405- ``bsp.test_machine_signatures``: Verifies that building for a
406 particular machine affects only the signature of tasks specific to
407 that machine.
408
409- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
410 distro configurations.
411
412- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
413 does not set the distribution when the layer is added.
414
415Enabling Your Layer
416-------------------
417
418Before the OpenEmbedded build system can use your new layer, you need to
419enable it. To enable your layer, simply add your layer's path to the
420:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
421found in the :term:`Build Directory`.
422The following example shows how to enable your new
423``meta-mylayer`` layer (note how your new layer exists outside of
424the official ``poky`` repository which you would have checked out earlier)::
425
426 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
427 # changes incompatibly
428 POKY_BBLAYERS_CONF_VERSION = "2"
429 BBPATH = "${TOPDIR}"
430 BBFILES ?= ""
431 BBLAYERS ?= " \
432 /home/user/poky/meta \
433 /home/user/poky/meta-poky \
434 /home/user/poky/meta-yocto-bsp \
435 /home/user/mystuff/meta-mylayer \
436 "
437
438BitBake parses each ``conf/layer.conf`` file from the top down as
439specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
440file. During the processing of each ``conf/layer.conf`` file, BitBake
441adds the recipes, classes and configurations contained within the
442particular layer to the source directory.
443
444Appending Other Layers Metadata With Your Layer
445-----------------------------------------------
446
447A recipe that appends Metadata to another recipe is called a BitBake
448append file. A BitBake append file uses the ``.bbappend`` file type
449suffix, while the corresponding recipe to which Metadata is being
450appended uses the ``.bb`` file type suffix.
451
452You can use a ``.bbappend`` file in your layer to make additions or
453changes to the content of another layer's recipe without having to copy
454the other layer's recipe into your layer. Your ``.bbappend`` file
455resides in your layer, while the main ``.bb`` recipe file to which you
456are appending Metadata resides in a different layer.
457
458Being able to append information to an existing recipe not only avoids
459duplication, but also automatically applies recipe changes from a
460different layer into your layer. If you were copying recipes, you would
461have to manually merge changes as they occur.
462
463When you create an append file, you must use the same root name as the
464corresponding recipe file. For example, the append file
465``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
466means the original recipe and append filenames are version
467number-specific. If the corresponding recipe is renamed to update to a
468newer version, you must also rename and possibly update the
469corresponding ``.bbappend`` as well. During the build process, BitBake
470displays an error on starting if it detects a ``.bbappend`` file that
471does not have a corresponding recipe with a matching name. See the
472:term:`BB_DANGLINGAPPENDS_WARNONLY`
473variable for information on how to handle this error.
474
475Overlaying a File Using Your Layer
476~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
477
478As an example, consider the main formfactor recipe and a corresponding
479formfactor append file both from the :term:`Source Directory`.
480Here is the main
481formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
482the "meta" layer at ``meta/recipes-bsp/formfactor``::
483
484 SUMMARY = "Device formfactor information"
485 DESCRIPTION = "A formfactor configuration file provides information about the \
486 target hardware for which the image is being built and information that the \
487 build system cannot obtain from other sources such as the kernel."
488 SECTION = "base"
489 LICENSE = "MIT"
490 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
491 PR = "r45"
492
493 SRC_URI = "file://config file://machconfig"
494 S = "${WORKDIR}"
495
496 PACKAGE_ARCH = "${MACHINE_ARCH}"
497 INHIBIT_DEFAULT_DEPS = "1"
498
499 do_install() {
500 # Install file only if it has contents
501 install -d ${D}${sysconfdir}/formfactor/
502 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
503 if [ -s "${S}/machconfig" ]; then
504 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
505 fi
506 }
507
508In the main recipe, note the :term:`SRC_URI`
509variable, which tells the OpenEmbedded build system where to find files
510during the build.
511
512Following is the append file, which is named ``formfactor_0.0.bbappend``
513and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
514file is in the layer at ``recipes-bsp/formfactor``::
515
516 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
517
518By default, the build system uses the
519:term:`FILESPATH` variable to
520locate files. This append file extends the locations by setting the
521:term:`FILESEXTRAPATHS`
522variable. Setting this variable in the ``.bbappend`` file is the most
523reliable and recommended method for adding directories to the search
524path used by the build system to find files.
525
526The statement in this example extends the directories to include
527``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
528which resolves to a directory named ``formfactor`` in the same directory
529in which the append file resides (i.e.
530``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
531have the supporting directory structure set up that will contain any
532files or patches you will be including from the layer.
533
534Using the immediate expansion assignment operator ``:=`` is important
535because of the reference to :term:`THISDIR`. The trailing colon character is
536important as it ensures that items in the list remain colon-separated.
537
538.. note::
539
540 BitBake automatically defines the :term:`THISDIR` variable. You should
541 never set this variable yourself. Using ":prepend" as part of the
542 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
543 paths in the final list.
544
545 Also, not all append files add extra files. Many append files simply
546 allow to add build options (e.g. ``systemd``). For these cases, your
547 append file would not even use the :term:`FILESEXTRAPATHS` statement.
548
549The end result of this ``.bbappend`` file is that on a Raspberry Pi, where
550``rpi`` will exist in the list of :term:`OVERRIDES`, the file
551``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be
552used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in
553:ref:`ref-tasks-install` will return true, and the file will be installed.
554
555Installing Additional Files Using Your Layer
556~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
557
558As another example, consider the main ``xserver-xf86-config`` recipe and a
559corresponding ``xserver-xf86-config`` append file both from the :term:`Source
560Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named
561``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at
562``meta/recipes-graphics/xorg-xserver``::
563
564 SUMMARY = "X.Org X server configuration file"
565 HOMEPAGE = "http://www.x.org"
566 SECTION = "x11/base"
567 LICENSE = "MIT"
568 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
569 PR = "r33"
570
571 SRC_URI = "file://xorg.conf"
572
573 S = "${WORKDIR}"
574
575 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf"
576
577 PACKAGE_ARCH = "${MACHINE_ARCH}"
578 ALLOW_EMPTY:${PN} = "1"
579
580 do_install () {
581 if test -s ${WORKDIR}/xorg.conf; then
582 install -d ${D}/${sysconfdir}/X11
583 install -m 0644 ${WORKDIR}/xorg.conf ${D}/${sysconfdir}/X11/
584 fi
585 }
586
587Following is the append file, which is named ``xserver-xf86-config_%.bbappend``
588and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
589file is in the layer at ``recipes-graphics/xorg-xserver``::
590
591 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
592
593 SRC_URI:append:rpi = " \
594 file://xorg.conf.d/98-pitft.conf \
595 file://xorg.conf.d/99-calibration.conf \
596 "
597 do_install:append:rpi () {
598 PITFT="${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}"
599 if [ "${PITFT}" = "1" ]; then
600 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/
601 install -m 0644 ${WORKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
602 install -m 0644 ${WORKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
603 fi
604 }
605
606 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*"
607
608Building off of the previous example, we once again are setting the
609:term:`FILESEXTRAPATHS` variable. In this case we are also using
610:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in
611the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a
612check for an additional :term:`MACHINE_FEATURES` that if set will cause these
613additional files to be installed. These additional files are listed in
614:term:`FILES` so that they will be packaged.
615
616Prioritizing Your Layer
617-----------------------
618
619Each layer is assigned a priority value. Priority values control which
620layer takes precedence if there are recipe files with the same name in
621multiple layers. For these cases, the recipe file from the layer with a
622higher priority number takes precedence. Priority values also affect the
623order in which multiple ``.bbappend`` files for the same recipe are
624applied. You can either specify the priority manually, or allow the
625build system to calculate it based on the layer's dependencies.
626
627To specify the layer's priority manually, use the
628:term:`BBFILE_PRIORITY`
629variable and append the layer's root name::
630
631 BBFILE_PRIORITY_mylayer = "1"
632
633.. note::
634
635 It is possible for a recipe with a lower version number
636 :term:`PV` in a layer that has a higher
637 priority to take precedence.
638
639 Also, the layer priority does not currently affect the precedence
640 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
641 might address this.
642
643Managing Layers
644---------------
645
646You can use the BitBake layer management tool ``bitbake-layers`` to
647provide a view into the structure of recipes across a multi-layer
648project. Being able to generate output that reports on configured layers
649with their paths and priorities and on ``.bbappend`` files and their
650applicable recipes can help to reveal potential problems.
651
652For help on the BitBake layer management tool, use the following
653command::
654
655 $ bitbake-layers --help
656 NOTE: Starting bitbake server...
657 usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
658
659 BitBake layers utility
660
661 optional arguments:
662 -d, --debug Enable debug output
663 -q, --quiet Print only errors
664 -F, --force Force add without recipe parse verification
665 --color COLOR Colorize output (where COLOR is auto, always, never)
666 -h, --help show this help message and exit
667
668 subcommands:
669 <subcommand>
670 layerindex-fetch Fetches a layer from a layer index along with its
671 dependent layers, and adds them to conf/bblayers.conf.
672 layerindex-show-depends
673 Find layer dependencies from layer index.
674 add-layer Add one or more layers to bblayers.conf.
675 remove-layer Remove one or more layers from bblayers.conf.
676 flatten flatten layer configuration into a separate output
677 directory.
678 show-layers show current configured layers.
679 show-overlayed list overlayed recipes (where the same recipe exists
680 in another layer)
681 show-recipes list available recipes, showing the layer they are
682 provided by
683 show-appends list bbappend files and recipe files they apply to
684 show-cross-depends Show dependencies between recipes that cross layer
685 boundaries.
686 create-layer Create a basic layer
687
688 Use bitbake-layers <subcommand> --help to get help on a specific command
689
690The following list describes the available commands:
691
692- ``help:`` Displays general help or help on a specified command.
693
694- ``show-layers:`` Shows the current configured layers.
695
696- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
697 when a recipe with the same name exists in another layer that has a
698 higher layer priority.
699
700- ``show-recipes:`` Lists available recipes and the layers that
701 provide them.
702
703- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
704 which they apply.
705
706- ``show-cross-depends:`` Lists dependency relationships between
707 recipes that cross layer boundaries.
708
709- ``add-layer:`` Adds a layer to ``bblayers.conf``.
710
711- ``remove-layer:`` Removes a layer from ``bblayers.conf``
712
713- ``flatten:`` Flattens the layer configuration into a separate
714 output directory. Flattening your layer configuration builds a
715 "flattened" directory that contains the contents of all layers, with
716 any overlayed recipes removed and any ``.bbappend`` files appended to
717 the corresponding recipes. You might have to perform some manual
718 cleanup of the flattened layer as follows:
719
720 - Non-recipe files (such as patches) are overwritten. The flatten
721 command shows a warning for these files.
722
723 - Anything beyond the normal layer setup has been added to the
724 ``layer.conf`` file. Only the lowest priority layer's
725 ``layer.conf`` is used.
726
727 - Overridden and appended items from ``.bbappend`` files need to be
728 cleaned up. The contents of each ``.bbappend`` end up in the
729 flattened recipe. However, if there are appended or changed
730 variable values, you need to tidy these up yourself. Consider the
731 following example. Here, the ``bitbake-layers`` command adds the
732 line ``#### bbappended ...`` so that you know where the following
733 lines originate::
734
735 ...
736 DESCRIPTION = "A useful utility"
737 ...
738 EXTRA_OECONF = "--enable-something"
739 ...
740
741 #### bbappended from meta-anotherlayer ####
742
743 DESCRIPTION = "Customized utility"
744 EXTRA_OECONF += "--enable-somethingelse"
745
746
747 Ideally, you would tidy up these utilities as follows::
748
749 ...
750 DESCRIPTION = "Customized utility"
751 ...
752 EXTRA_OECONF = "--enable-something --enable-somethingelse"
753 ...
754
755- ``layerindex-fetch``: Fetches a layer from a layer index, along
756 with its dependent layers, and adds the layers to the
757 ``conf/bblayers.conf`` file.
758
759- ``layerindex-show-depends``: Finds layer dependencies from the
760 layer index.
761
762- ``create-layer``: Creates a basic layer.
763
764Creating a General Layer Using the ``bitbake-layers`` Script
765------------------------------------------------------------
766
767The ``bitbake-layers`` script with the ``create-layer`` subcommand
768simplifies creating a new general layer.
769
770.. note::
771
772 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
773 section in the Yocto
774 Project Board Specific (BSP) Developer's Guide.
775
776 - In order to use a layer with the OpenEmbedded build system, you
777 need to add the layer to your ``bblayers.conf`` configuration
778 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
779 section for more information.
780
781The default mode of the script's operation with this subcommand is to
782create a layer with the following:
783
784- A layer priority of 6.
785
786- A ``conf`` subdirectory that contains a ``layer.conf`` file.
787
788- A ``recipes-example`` subdirectory that contains a further
789 subdirectory named ``example``, which contains an ``example.bb``
790 recipe file.
791
792- A ``COPYING.MIT``, which is the license statement for the layer. The
793 script assumes you want to use the MIT license, which is typical for
794 most layers, for the contents of the layer itself.
795
796- A ``README`` file, which is a file describing the contents of your
797 new layer.
798
799In its simplest form, you can use the following command form to create a
800layer. The command creates a layer whose name corresponds to
801"your_layer_name" in the current directory::
802
803 $ bitbake-layers create-layer your_layer_name
804
805As an example, the following command creates a layer named ``meta-scottrif``
806in your home directory::
807
808 $ cd /usr/home
809 $ bitbake-layers create-layer meta-scottrif
810 NOTE: Starting bitbake server...
811 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
812
813If you want to set the priority of the layer to other than the default
814value of "6", you can either use the ``--priority`` option or you
815can edit the
816:term:`BBFILE_PRIORITY` value
817in the ``conf/layer.conf`` after the script creates it. Furthermore, if
818you want to give the example recipe file some name other than the
819default, you can use the ``--example-recipe-name`` option.
820
821The easiest way to see how the ``bitbake-layers create-layer`` command
822works is to experiment with the script. You can also read the usage
823information by entering the following::
824
825 $ bitbake-layers create-layer --help
826 NOTE: Starting bitbake server...
827 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
828 [--example-recipe-name EXAMPLERECIPE]
829 layerdir
830
831 Create a basic layer
832
833 positional arguments:
834 layerdir Layer directory to create
835
836 optional arguments:
837 -h, --help show this help message and exit
838 --priority PRIORITY, -p PRIORITY
839 Layer directory to create
840 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
841 Filename of the example recipe
842
843Adding a Layer Using the ``bitbake-layers`` Script
844--------------------------------------------------
845
846Once you create your general layer, you must add it to your
847``bblayers.conf`` file. Adding the layer to this configuration file
848makes the OpenEmbedded build system aware of your layer so that it can
849search it for metadata.
850
851Add your layer by using the ``bitbake-layers add-layer`` command::
852
853 $ bitbake-layers add-layer your_layer_name
854
855Here is an example that adds a
856layer named ``meta-scottrif`` to the configuration file. Following the
857command that adds the layer is another ``bitbake-layers`` command that
858shows the layers that are in your ``bblayers.conf`` file::
859
860 $ bitbake-layers add-layer meta-scottrif
861 NOTE: Starting bitbake server...
862 Parsing recipes: 100% |##########################################################| Time: 0:00:49
863 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
864 $ bitbake-layers show-layers
865 NOTE: Starting bitbake server...
866 layer path priority
867 ==========================================================================
868 meta /home/scottrif/poky/meta 5
869 meta-poky /home/scottrif/poky/meta-poky 5
870 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
871 workspace /home/scottrif/poky/build/workspace 99
872 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
873
874
875Adding the layer to this file
876enables the build system to locate the layer during the build.
877
878.. note::
879
880 During a build, the OpenEmbedded build system looks in the layers
881 from the top of the list down to the bottom in that order.
882
883Customizing Images
884==================
885
886You can customize images to satisfy particular requirements. This
887section describes several methods and provides guidelines for each.
888
889Customizing Images Using ``local.conf``
890---------------------------------------
891
892Probably the easiest way to customize an image is to add a package by
893way of the ``local.conf`` configuration file. Because it is limited to
894local use, this method generally only allows you to add packages and is
895not as flexible as creating your own customized image. When you add
896packages using local variables this way, you need to realize that these
897variable changes are in effect for every build and consequently affect
898all images, which might not be what you require.
899
900To add a package to your image using the local configuration file, use
901the :term:`IMAGE_INSTALL` variable with the ``:append`` operator::
902
903 IMAGE_INSTALL:append = " strace"
904
905Use of the syntax is important; specifically, the leading space
906after the opening quote and before the package name, which is
907``strace`` in this example. This space is required since the ``:append``
908operator does not add the space.
909
910Furthermore, you must use ``:append`` instead of the ``+=`` operator if
911you want to avoid ordering issues. The reason for this is because doing
912so unconditionally appends to the variable and avoids ordering problems
913due to the variable being set in image recipes and ``.bbclass`` files
914with operators like ``?=``. Using ``:append`` ensures the operation
915takes effect.
916
917As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all
918images. It is possible to extend the syntax so that the variable applies
919to a specific image only. Here is an example::
920
921 IMAGE_INSTALL:append:pn-core-image-minimal = " strace"
922
923This example adds ``strace`` to the ``core-image-minimal`` image only.
924
925You can add packages using a similar approach through the
926:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
927``core-image-*`` images are affected.
928
929Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
930-------------------------------------------------------------------------------
931
932Another method for customizing your image is to enable or disable
933high-level image features by using the
934:term:`IMAGE_FEATURES` and
935:term:`EXTRA_IMAGE_FEATURES`
936variables. Although the functions for both variables are nearly
937equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
938a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
939``local.conf`` file, which is found in the
940:term:`Build Directory`.
941
942To understand how these features work, the best reference is
943:ref:`meta/classes/image.bbclass <ref-classes-image>`.
944This class lists out the available
945:term:`IMAGE_FEATURES` of which most map to package groups while some, such
946as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
947configuration settings.
948
949In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
950variable and then maps or configures the feature accordingly. Based on
951this information, the build system automatically adds the appropriate
952packages or configurations to the
953:term:`IMAGE_INSTALL` variable.
954Effectively, you are enabling extra features by extending the class or
955creating a custom class for use with specialized image ``.bb`` files.
956
957Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
958configuration file. Using a separate area from which to enable features
959with this variable helps you avoid overwriting the features in the image
960recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
961:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
962``meta/conf/bitbake.conf``.
963
964To illustrate how you can use these variables to modify your image,
965consider an example that selects the SSH server. The Yocto Project ships
966with two SSH servers you can use with your images: Dropbear and OpenSSH.
967Dropbear is a minimal SSH server appropriate for resource-constrained
968environments, while OpenSSH is a well-known standard SSH server
969implementation. By default, the ``core-image-sato`` image is configured
970to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
971images both include OpenSSH. The ``core-image-minimal`` image does not
972contain an SSH server.
973
974You can customize your image and change these defaults. Edit the
975:term:`IMAGE_FEATURES` variable in your recipe or use the
976:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
977configures the image you are working with to include
978``ssh-server-dropbear`` or ``ssh-server-openssh``.
979
980.. note::
981
982 See the ":ref:`ref-manual/features:image features`" section in the Yocto
983 Project Reference Manual for a complete list of image features that ship
984 with the Yocto Project.
985
986Customizing Images Using Custom .bb Files
987-----------------------------------------
988
989You can also customize an image by creating a custom recipe that defines
990additional software as part of the image. The following example shows
991the form for the two lines you need::
992
993 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
994 inherit core-image
995
996Defining the software using a custom recipe gives you total control over
997the contents of the image. It is important to use the correct names of
998packages in the :term:`IMAGE_INSTALL` variable. You must use the
999OpenEmbedded notation and not the Debian notation for the names (e.g.
1000``glibc-dev`` instead of ``libc6-dev``).
1001
1002The other method for creating a custom image is to base it on an
1003existing image. For example, if you want to create an image based on
1004``core-image-sato`` but add the additional package ``strace`` to the
1005image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
1006``.bb`` and add the following line to the end of the copy::
1007
1008 IMAGE_INSTALL += "strace"
1009
1010Customizing Images Using Custom Package Groups
1011----------------------------------------------
1012
1013For complex custom images, the best approach for customizing an image is
1014to create a custom package group recipe that is used to build the image
1015or images. A good example of a package group recipe is
1016``meta/recipes-core/packagegroups/packagegroup-base.bb``.
1017
1018If you examine that recipe, you see that the :term:`PACKAGES` variable lists
1019the package group packages to produce. The ``inherit packagegroup``
1020statement sets appropriate default values and automatically adds
1021``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
1022package specified in the :term:`PACKAGES` statement.
1023
1024.. note::
1025
1026 The ``inherit packagegroup`` line should be located near the top of the
1027 recipe, certainly before the :term:`PACKAGES` statement.
1028
1029For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
1030and :term:`RRECOMMENDS` entries to provide a list of packages the parent
1031task package should contain. You can see examples of these further down
1032in the ``packagegroup-base.bb`` recipe.
1033
1034Here is a short, fabricated example showing the same basic pieces for a
1035hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
1036the variable :term:`PN` is the standard way to abbreviate the reference to
1037the full packagegroup name ``packagegroup-custom``::
1038
1039 DESCRIPTION = "My Custom Package Groups"
1040
1041 inherit packagegroup
1042
1043 PACKAGES = "\
1044 ${PN}-apps \
1045 ${PN}-tools \
1046 "
1047
1048 RDEPENDS:${PN}-apps = "\
1049 dropbear \
1050 portmap \
1051 psplash"
1052
1053 RDEPENDS:${PN}-tools = "\
1054 oprofile \
1055 oprofileui-server \
1056 lttng-tools"
1057
1058 RRECOMMENDS:${PN}-tools = "\
1059 kernel-module-oprofile"
1060
1061In the previous example, two package group packages are created with
1062their dependencies and their recommended package dependencies listed:
1063``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1064build an image using these package group packages, you need to add
1065``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
1066:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
1067areas of this section.
1068
1069Customizing an Image Hostname
1070-----------------------------
1071
1072By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1073is the same as the machine name. For example, if
1074:term:`MACHINE` equals "qemux86", the
1075configured hostname written to ``/etc/hostname`` is "qemux86".
1076
1077You can customize this name by altering the value of the "hostname"
1078variable in the ``base-files`` recipe using either an append file or a
1079configuration file. Use the following in an append file::
1080
1081 hostname = "myhostname"
1082
1083Use the following in a configuration file::
1084
1085 hostname:pn-base-files = "myhostname"
1086
1087Changing the default value of the variable "hostname" can be useful in
1088certain situations. For example, suppose you need to do extensive
1089testing on an image and you would like to easily identify the image
1090under test from existing images with typical default hostnames. In this
1091situation, you could change the default hostname to "testme", which
1092results in all the images using the name "testme". Once testing is
1093complete and you do not need to rebuild the image for test any longer,
1094you can easily reset the default hostname.
1095
1096Another point of interest is that if you unset the variable, the image
1097will have no default hostname in the filesystem. Here is an example that
1098unsets the variable in a configuration file::
1099
1100 hostname:pn-base-files = ""
1101
1102Having no default hostname in the filesystem is suitable for
1103environments that use dynamic hostnames such as virtual machines.
1104
1105Writing a New Recipe
1106====================
1107
1108Recipes (``.bb`` files) are fundamental components in the Yocto Project
1109environment. Each software component built by the OpenEmbedded build
1110system requires a recipe to define the component. This section describes
1111how to create, write, and test a new recipe.
1112
1113.. note::
1114
1115 For information on variables that are useful for recipes and for
1116 information about recipe naming issues, see the
1117 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
1118 Reference Manual.
1119
1120Overview
1121--------
1122
1123The following figure shows the basic process for creating a new recipe.
1124The remainder of the section provides details for the steps.
1125
1126.. image:: figures/recipe-workflow.png
1127 :align: center
1128
1129Locate or Automatically Create a Base Recipe
1130--------------------------------------------
1131
1132You can always write a recipe from scratch. However, there are three choices
1133that can help you quickly get started with a new recipe:
1134
1135- ``devtool add``: A command that assists in creating a recipe and an
1136 environment conducive to development.
1137
1138- ``recipetool create``: A command provided by the Yocto Project that
1139 automates creation of a base recipe based on the source files.
1140
1141- *Existing Recipes:* Location and modification of an existing recipe
1142 that is similar in function to the recipe you need.
1143
1144.. note::
1145
1146 For information on recipe syntax, see the
1147 ":ref:`dev-manual/common-tasks:recipe syntax`" section.
1148
1149Creating the Base Recipe Using ``devtool add``
1150~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1151
1152The ``devtool add`` command uses the same logic for auto-creating the
1153recipe as ``recipetool create``, which is listed below. Additionally,
1154however, ``devtool add`` sets up an environment that makes it easy for
1155you to patch the source and to make changes to the recipe as is often
1156necessary when adding a recipe to build a new piece of software to be
1157included in a build.
1158
1159You can find a complete description of the ``devtool add`` command in
1160the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
1161in the Yocto Project Application Development and the Extensible Software
1162Development Kit (eSDK) manual.
1163
1164Creating the Base Recipe Using ``recipetool create``
1165~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1166
1167``recipetool create`` automates creation of a base recipe given a set of
1168source code files. As long as you can extract or point to the source
1169files, the tool will construct a recipe and automatically configure all
1170pre-build information into the recipe. For example, suppose you have an
1171application that builds using Autotools. Creating the base recipe using
1172``recipetool`` results in a recipe that has the pre-build dependencies,
1173license requirements, and checksums configured.
1174
1175To run the tool, you just need to be in your
1176:term:`Build Directory` and have sourced the
1177build environment setup script (i.e.
1178:ref:`structure-core-script`).
1179To get help on the tool, use the following command::
1180
1181 $ recipetool -h
1182 NOTE: Starting bitbake server...
1183 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1184
1185 OpenEmbedded recipe tool
1186
1187 options:
1188 -d, --debug Enable debug output
1189 -q, --quiet Print only errors
1190 --color COLOR Colorize output (where COLOR is auto, always, never)
1191 -h, --help show this help message and exit
1192
1193 subcommands:
1194 create Create a new recipe
1195 newappend Create a bbappend for the specified target in the specified
1196 layer
1197 setvar Set a variable within a recipe
1198 appendfile Create/update a bbappend to replace a target file
1199 appendsrcfiles Create/update a bbappend to add or replace source files
1200 appendsrcfile Create/update a bbappend to add or replace a source file
1201 Use recipetool <subcommand> --help to get help on a specific command
1202
1203Running ``recipetool create -o OUTFILE`` creates the base recipe and
1204locates it properly in the layer that contains your source files.
1205Following are some syntax examples:
1206
1207 - Use this syntax to generate a recipe based on source. Once generated,
1208 the recipe resides in the existing source code layer::
1209
1210 recipetool create -o OUTFILE source
1211
1212 - Use this syntax to generate a recipe using code that
1213 you extract from source. The extracted code is placed in its own layer
1214 defined by :term:`EXTERNALSRC`.
1215 ::
1216
1217 recipetool create -o OUTFILE -x EXTERNALSRC source
1218
1219 - Use this syntax to generate a recipe based on source. The options
1220 direct ``recipetool`` to generate debugging information. Once generated,
1221 the recipe resides in the existing source code layer::
1222
1223 recipetool create -d -o OUTFILE source
1224
1225Locating and Using a Similar Recipe
1226~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1227
1228Before writing a recipe from scratch, it is often useful to discover
1229whether someone else has already written one that meets (or comes close
1230to meeting) your needs. The Yocto Project and OpenEmbedded communities
1231maintain many recipes that might be candidates for what you are doing.
1232You can find a good central index of these recipes in the
1233:oe_layerindex:`OpenEmbedded Layer Index <>`.
1234
1235Working from an existing recipe or a skeleton recipe is the best way to
1236get started. Here are some points on both methods:
1237
1238- *Locate and modify a recipe that is close to what you want to do:*
1239 This method works when you are familiar with the current recipe
1240 space. The method does not work so well for those new to the Yocto
1241 Project or writing recipes.
1242
1243 Some risks associated with this method are using a recipe that has
1244 areas totally unrelated to what you are trying to accomplish with
1245 your recipe, not recognizing areas of the recipe that you might have
1246 to add from scratch, and so forth. All these risks stem from
1247 unfamiliarity with the existing recipe space.
1248
1249- *Use and modify the following skeleton recipe:* If for some reason
1250 you do not want to use ``recipetool`` and you cannot find an existing
1251 recipe that is close to meeting your needs, you can use the following
1252 structure to provide the fundamental areas of a new recipe.
1253 ::
1254
1255 DESCRIPTION = ""
1256 HOMEPAGE = ""
1257 LICENSE = ""
1258 SECTION = ""
1259 DEPENDS = ""
1260 LIC_FILES_CHKSUM = ""
1261
1262 SRC_URI = ""
1263
1264Storing and Naming the Recipe
1265-----------------------------
1266
1267Once you have your base recipe, you should put it in your own layer and
1268name it appropriately. Locating it correctly ensures that the
1269OpenEmbedded build system can find it when you use BitBake to process
1270the recipe.
1271
1272- *Storing Your Recipe:* The OpenEmbedded build system locates your
1273 recipe through the layer's ``conf/layer.conf`` file and the
1274 :term:`BBFILES` variable. This
1275 variable sets up a path from which the build system can locate
1276 recipes. Here is the typical use::
1277
1278 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1279 ${LAYERDIR}/recipes-*/*/*.bbappend"
1280
1281 Consequently, you need to be sure you locate your new recipe inside
1282 your layer such that it can be found.
1283
1284 You can find more information on how layers are structured in the
1285 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
1286
1287- *Naming Your Recipe:* When you name your recipe, you need to follow
1288 this naming convention::
1289
1290 basename_version.bb
1291
1292 Use lower-cased characters and do not include the reserved suffixes
1293 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1294 them as part of your recipe name unless the string applies). Here are some
1295 examples:
1296
1297 .. code-block:: none
1298
1299 cups_1.7.0.bb
1300 gawk_4.0.2.bb
1301 irssi_0.8.16-rc1.bb
1302
1303Running a Build on the Recipe
1304-----------------------------
1305
1306Creating a new recipe is usually an iterative process that requires
1307using BitBake to process the recipe multiple times in order to
1308progressively discover and add information to the recipe file.
1309
1310Assuming you have sourced the build environment setup script (i.e.
1311:ref:`structure-core-script`) and you are in
1312the :term:`Build Directory`, use
1313BitBake to process your recipe. All you need to provide is the
1314``basename`` of the recipe as described in the previous section::
1315
1316 $ bitbake basename
1317
1318During the build, the OpenEmbedded build system creates a temporary work
1319directory for each recipe
1320(``${``\ :term:`WORKDIR`\ ``}``)
1321where it keeps extracted source files, log files, intermediate
1322compilation and packaging files, and so forth.
1323
1324The path to the per-recipe temporary work directory depends on the
1325context in which it is being built. The quickest way to find this path
1326is to have BitBake return it by running the following::
1327
1328 $ bitbake -e basename | grep ^WORKDIR=
1329
1330As an example, assume a Source Directory
1331top-level folder named ``poky``, a default Build Directory at
1332``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1333Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1334case, the work directory the build system uses to build the package
1335would be as follows::
1336
1337 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1338
1339Inside this directory you can find sub-directories such as ``image``,
1340``packages-split``, and ``temp``. After the build, you can examine these
1341to determine how well the build went.
1342
1343.. note::
1344
1345 You can find log files for each task in the recipe's ``temp``
1346 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1347 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1348 ``log.do_fetch``, and ``log.do_compile``).
1349
1350You can find more information about the build process in
1351":doc:`/overview-manual/development-environment`"
1352chapter of the Yocto Project Overview and Concepts Manual.
1353
1354Fetching Code
1355-------------
1356
1357The first thing your recipe must do is specify how to fetch the source
1358files. Fetching is controlled mainly through the
1359:term:`SRC_URI` variable. Your recipe
1360must have a :term:`SRC_URI` variable that points to where the source is
1361located. For a graphical representation of source locations, see the
1362":ref:`overview-manual/concepts:sources`" section in
1363the Yocto Project Overview and Concepts Manual.
1364
1365The :ref:`ref-tasks-fetch` task uses
1366the prefix of each entry in the :term:`SRC_URI` variable value to determine
1367which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
1368source files. It is the :term:`SRC_URI` variable that triggers the fetcher.
1369The :ref:`ref-tasks-patch` task uses
1370the variable after source is fetched to apply patches. The OpenEmbedded
1371build system uses
1372:term:`FILESOVERRIDES` for
1373scanning directory locations for local files in :term:`SRC_URI`.
1374
1375The :term:`SRC_URI` variable in your recipe must define each unique location
1376for your source files. It is good practice to not hard-code version
1377numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
1378values, use ``${``\ :term:`PV`\ ``}``,
1379which causes the fetch process to use the version specified in the
1380recipe filename. Specifying the version in this manner means that
1381upgrading the recipe to a future version is as simple as renaming the
1382recipe to match the new version.
1383
1384Here is a simple example from the
1385``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1386comes from a single tarball. Notice the use of the
1387:term:`PV` variable::
1388
1389 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
1390
1391Files mentioned in :term:`SRC_URI` whose names end in a typical archive
1392extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1393forth), are automatically extracted during the
1394:ref:`ref-tasks-unpack` task. For
1395another example that specifies these types of files, see the
1396":ref:`dev-manual/common-tasks:autotooled package`" section.
1397
1398Another way of specifying source is from an SCM. For Git repositories,
1399you must specify :term:`SRCREV` and you should specify :term:`PV` to include
1400the revision with :term:`SRCPV`. Here is an example from the recipe
1401``meta/recipes-core/musl/gcompat_git.bb``::
1402
1403 SRC_URI = "git://git.adelielinux.org/adelie/gcompat.git;protocol=https;branch=current"
1404
1405 PV = "1.0.0+1.1+git${SRCPV}"
1406 SRCREV = "af5a49e489fdc04b9cf02547650d7aeaccd43793"
1407
1408If your :term:`SRC_URI` statement includes URLs pointing to individual files
1409fetched from a remote server other than a version control system,
1410BitBake attempts to verify the files against checksums defined in your
1411recipe to ensure they have not been tampered with or otherwise modified
1412since the recipe was written. Two checksums are used:
1413``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1414
1415If your :term:`SRC_URI` variable points to more than a single URL (excluding
1416SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1417each URL. For these cases, you provide a name for each URL as part of
1418the :term:`SRC_URI` and then reference that name in the subsequent checksum
1419statements. Here is an example combining lines from the files
1420``git.inc`` and ``git_2.24.1.bb``::
1421
1422 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1423 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1424
1425 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1426 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1427 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1428 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1429
1430Proper values for ``md5`` and ``sha256`` checksums might be available
1431with other signatures on the download page for the upstream source (e.g.
1432``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1433OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1434you should verify all the signatures you find by hand.
1435
1436If no :term:`SRC_URI` checksums are specified when you attempt to build the
1437recipe, or you provide an incorrect checksum, the build will produce an
1438error for each missing or incorrect checksum. As part of the error
1439message, the build system provides the checksum string corresponding to
1440the fetched file. Once you have the correct checksums, you can copy and
1441paste them into your recipe and then run the build again to continue.
1442
1443.. note::
1444
1445 As mentioned, if the upstream source provides signatures for
1446 verifying the downloaded source code, you should verify those
1447 manually before setting the checksum values in the recipe and
1448 continuing with the build.
1449
1450This final example is a bit more complicated and is from the
1451``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
1452example's :term:`SRC_URI` statement identifies multiple files as the source
1453files for the recipe: a tarball, a patch file, a desktop file, and an
1454icon.
1455::
1456
1457 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1458 file://xwc.patch \
1459 file://rxvt.desktop \
1460 file://rxvt.png"
1461
1462When you specify local files using the ``file://`` URI protocol, the
1463build system fetches files from the local machine. The path is relative
1464to the :term:`FILESPATH` variable
1465and searches specific directories in a certain order:
1466``${``\ :term:`BP`\ ``}``,
1467``${``\ :term:`BPN`\ ``}``, and
1468``files``. The directories are assumed to be subdirectories of the
1469directory in which the recipe or append file resides. For another
1470example that specifies these types of files, see the
1471":ref:`dev-manual/common-tasks:single .c file package (hello world!)`" section.
1472
1473The previous example also specifies a patch file. Patch files are files
1474whose names usually end in ``.patch`` or ``.diff`` but can end with
1475compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1476The build system automatically applies patches as described in the
1477":ref:`dev-manual/common-tasks:patching code`" section.
1478
1479Fetching Code Through Firewalls
1480~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1481
1482Some users are behind firewalls and need to fetch code through a proxy.
1483See the ":doc:`/ref-manual/faq`" chapter for advice.
1484
1485Limiting the Number of Parallel Connections
1486~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1487
1488Some users are behind firewalls or use servers where the number of parallel
1489connections is limited. In such cases, you can limit the number of fetch
1490tasks being run in parallel by adding the following to your ``local.conf``
1491file::
1492
1493 do_fetch[number_threads] = "4"
1494
1495Unpacking Code
1496--------------
1497
1498During the build, the
1499:ref:`ref-tasks-unpack` task unpacks
1500the source with ``${``\ :term:`S`\ ``}``
1501pointing to where it is unpacked.
1502
1503If you are fetching your source files from an upstream source archived
1504tarball and the tarball's internal structure matches the common
1505convention of a top-level subdirectory named
1506``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
1507then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
1508fetch source from an archive that does not use this convention, or from
1509an SCM like Git or Subversion, your recipe needs to define :term:`S`.
1510
1511If processing your recipe using BitBake successfully unpacks the source
1512files, you need to be sure that the directory pointed to by ``${S}``
1513matches the structure of the source.
1514
1515Patching Code
1516-------------
1517
1518Sometimes it is necessary to patch code after it has been fetched. Any
1519files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
1520``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1521treated as patches. The
1522:ref:`ref-tasks-patch` task
1523automatically applies these patches.
1524
1525The build system should be able to apply patches with the "-p1" option
1526(i.e. one directory level in the path will be stripped off). If your
1527patch needs to have more directory levels stripped off, specify the
1528number of levels using the "striplevel" option in the :term:`SRC_URI` entry
1529for the patch. Alternatively, if your patch needs to be applied in a
1530specific subdirectory that is not specified in the patch file, use the
1531"patchdir" option in the entry.
1532
1533As with all local files referenced in
1534:term:`SRC_URI` using ``file://``,
1535you should place patch files in a directory next to the recipe either
1536named the same as the base name of the recipe
1537(:term:`BP` and
1538:term:`BPN`) or "files".
1539
1540Licensing
1541---------
1542
1543Your recipe needs to have both the
1544:term:`LICENSE` and
1545:term:`LIC_FILES_CHKSUM`
1546variables:
1547
1548- :term:`LICENSE`: This variable specifies the license for the software.
1549 If you do not know the license under which the software you are
1550 building is distributed, you should go to the source code and look
1551 for that information. Typical files containing this information
1552 include ``COPYING``, :term:`LICENSE`, and ``README`` files. You could
1553 also find the information near the top of a source file. For example,
1554 given a piece of software licensed under the GNU General Public
1555 License version 2, you would set :term:`LICENSE` as follows::
1556
1557 LICENSE = "GPL-2.0-only"
1558
1559 The licenses you specify within :term:`LICENSE` can have any name as long
1560 as you do not use spaces, since spaces are used as separators between
1561 license names. For standard licenses, use the names of the files in
1562 ``meta/files/common-licenses/`` or the :term:`SPDXLICENSEMAP` flag names
1563 defined in ``meta/conf/licenses.conf``.
1564
1565- :term:`LIC_FILES_CHKSUM`: The OpenEmbedded build system uses this
1566 variable to make sure the license text has not changed. If it has,
1567 the build produces an error and it affords you the chance to figure
1568 it out and correct the problem.
1569
1570 You need to specify all applicable licensing files for the software.
1571 At the end of the configuration step, the build process will compare
1572 the checksums of the files to be sure the text has not changed. Any
1573 differences result in an error with the message containing the
1574 current checksum. For more explanation and examples of how to set the
1575 :term:`LIC_FILES_CHKSUM` variable, see the
1576 ":ref:`dev-manual/common-tasks:tracking license changes`" section.
1577
1578 To determine the correct checksum string, you can list the
1579 appropriate files in the :term:`LIC_FILES_CHKSUM` variable with incorrect
1580 md5 strings, attempt to build the software, and then note the
1581 resulting error messages that will report the correct md5 strings.
1582 See the ":ref:`dev-manual/common-tasks:fetching code`" section for
1583 additional information.
1584
1585 Here is an example that assumes the software has a ``COPYING`` file::
1586
1587 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1588
1589 When you try to build the
1590 software, the build system will produce an error and give you the
1591 correct string that you can substitute into the recipe file for a
1592 subsequent build.
1593
1594Dependencies
1595------------
1596
1597Most software packages have a short list of other packages that they
1598require, which are called dependencies. These dependencies fall into two
1599main categories: build-time dependencies, which are required when the
1600software is built; and runtime dependencies, which are required to be
1601installed on the target in order for the software to run.
1602
1603Within a recipe, you specify build-time dependencies using the
1604:term:`DEPENDS` variable. Although there are nuances,
1605items specified in :term:`DEPENDS` should be names of other
1606recipes. It is important that you specify all build-time dependencies
1607explicitly.
1608
1609Another consideration is that configure scripts might automatically
1610check for optional dependencies and enable corresponding functionality
1611if those dependencies are found. If you wish to make a recipe that is
1612more generally useful (e.g. publish the recipe in a layer for others to
1613use), instead of hard-disabling the functionality, you can use the
1614:term:`PACKAGECONFIG` variable to allow functionality and the
1615corresponding dependencies to be enabled and disabled easily by other
1616users of the recipe.
1617
1618Similar to build-time dependencies, you specify runtime dependencies
1619through a variable -
1620:term:`RDEPENDS`, which is
1621package-specific. All variables that are package-specific need to have
1622the name of the package added to the end as an override. Since the main
1623package for a recipe has the same name as the recipe, and the recipe's
1624name can be found through the
1625``${``\ :term:`PN`\ ``}`` variable, then
1626you specify the dependencies for the main package by setting
1627``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you
1628would set ``RDEPENDS:${PN}-tools``, and so forth.
1629
1630Some runtime dependencies will be set automatically at packaging time.
1631These dependencies include any shared library dependencies (i.e. if a
1632package "example" contains "libexample" and another package "mypackage"
1633contains a binary that links to "libexample" then the OpenEmbedded build
1634system will automatically add a runtime dependency to "mypackage" on
1635"example"). See the
1636":ref:`overview-manual/concepts:automatically added runtime dependencies`"
1637section in the Yocto Project Overview and Concepts Manual for further
1638details.
1639
1640Configuring the Recipe
1641----------------------
1642
1643Most software provides some means of setting build-time configuration
1644options before compilation. Typically, setting these options is
1645accomplished by running a configure script with options, or by modifying
1646a build configuration file.
1647
1648.. note::
1649
1650 As of Yocto Project Release 1.7, some of the core recipes that
1651 package binary configuration scripts now disable the scripts due to
1652 the scripts previously requiring error-prone path substitution. The
1653 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1654 robust. You can find a list of the ``*-config`` scripts that are disabled
1655 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1656 in the Yocto Project Reference Manual.
1657
1658A major part of build-time configuration is about checking for
1659build-time dependencies and possibly enabling optional functionality as
1660a result. You need to specify any build-time dependencies for the
1661software you are building in your recipe's
1662:term:`DEPENDS` value, in terms of
1663other recipes that satisfy those dependencies. You can often find
1664build-time or runtime dependencies described in the software's
1665documentation.
1666
1667The following list provides configuration items of note based on how
1668your software is built:
1669
1670- *Autotools:* If your source files have a ``configure.ac`` file, then
1671 your software is built using Autotools. If this is the case, you just
1672 need to modify the configuration.
1673
1674 When using Autotools, your recipe needs to inherit the
1675 :ref:`autotools <ref-classes-autotools>` class
1676 and your recipe does not have to contain a
1677 :ref:`ref-tasks-configure` task.
1678 However, you might still want to make some adjustments. For example,
1679 you can set
1680 :term:`EXTRA_OECONF` or
1681 :term:`PACKAGECONFIG_CONFARGS`
1682 to pass any needed configure options that are specific to the recipe.
1683
1684- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1685 your software is built using CMake. If this is the case, you just
1686 need to modify the configuration.
1687
1688 When you use CMake, your recipe needs to inherit the
1689 :ref:`cmake <ref-classes-cmake>` class and your
1690 recipe does not have to contain a
1691 :ref:`ref-tasks-configure` task.
1692 You can make some adjustments by setting
1693 :term:`EXTRA_OECMAKE` to
1694 pass any needed configure options that are specific to the recipe.
1695
1696 .. note::
1697
1698 If you need to install one or more custom CMake toolchain files
1699 that are supplied by the application you are building, install the
1700 files to ``${D}${datadir}/cmake/Modules`` during ``do_install``.
1701
1702- *Other:* If your source files do not have a ``configure.ac`` or
1703 ``CMakeLists.txt`` file, then your software is built using some
1704 method other than Autotools or CMake. If this is the case, you
1705 normally need to provide a
1706 :ref:`ref-tasks-configure` task
1707 in your recipe unless, of course, there is nothing to configure.
1708
1709 Even if your software is not being built by Autotools or CMake, you
1710 still might not need to deal with any configuration issues. You need
1711 to determine if configuration is even a required step. You might need
1712 to modify a Makefile or some configuration file used for the build to
1713 specify necessary build options. Or, perhaps you might need to run a
1714 provided, custom configure script with the appropriate options.
1715
1716 For the case involving a custom configure script, you would run
1717 ``./configure --help`` and look for the options you need to set.
1718
1719Once configuration succeeds, it is always good practice to look at the
1720``log.do_configure`` file to ensure that the appropriate options have
1721been enabled and no additional build-time dependencies need to be added
1722to :term:`DEPENDS`. For example, if the configure script reports that it
1723found something not mentioned in :term:`DEPENDS`, or that it did not find
1724something that it needed for some desired optional functionality, then
1725you would need to add those to :term:`DEPENDS`. Looking at the log might
1726also reveal items being checked for, enabled, or both that you do not
1727want, or items not being found that are in :term:`DEPENDS`, in which case
1728you would need to look at passing extra options to the configure script
1729as needed. For reference information on configure options specific to
1730the software you are building, you can consult the output of the
1731``./configure --help`` command within ``${S}`` or consult the software's
1732upstream documentation.
1733
1734Using Headers to Interface with Devices
1735---------------------------------------
1736
1737If your recipe builds an application that needs to communicate with some
1738device or needs an API into a custom kernel, you will need to provide
1739appropriate header files. Under no circumstances should you ever modify
1740the existing
1741``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1742These headers are used to build ``libc`` and must not be compromised
1743with custom or machine-specific header information. If you customize
1744``libc`` through modified headers all other applications that use
1745``libc`` thus become affected.
1746
1747.. note::
1748
1749 Never copy and customize the ``libc`` header file (i.e.
1750 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
1751
1752The correct way to interface to a device or custom kernel is to use a
1753separate package that provides the additional headers for the driver or
1754other unique interfaces. When doing so, your application also becomes
1755responsible for creating a dependency on that specific provider.
1756
1757Consider the following:
1758
1759- Never modify ``linux-libc-headers.inc``. Consider that file to be
1760 part of the ``libc`` system, and not something you use to access the
1761 kernel directly. You should access ``libc`` through specific ``libc``
1762 calls.
1763
1764- Applications that must talk directly to devices should either provide
1765 necessary headers themselves, or establish a dependency on a special
1766 headers package that is specific to that driver.
1767
1768For example, suppose you want to modify an existing header that adds I/O
1769control or network support. If the modifications are used by a small
1770number programs, providing a unique version of a header is easy and has
1771little impact. When doing so, bear in mind the guidelines in the
1772previous list.
1773
1774.. note::
1775
1776 If for some reason your changes need to modify the behavior of the ``libc``,
1777 and subsequently all other applications on the system, use a ``.bbappend``
1778 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1779 make the changes machine specific.
1780
1781Consider a case where your kernel is older and you need an older
1782``libc`` ABI. The headers installed by your recipe should still be a
1783standard mainline kernel, not your own custom one.
1784
1785When you use custom kernel headers you need to get them from
1786:term:`STAGING_KERNEL_DIR`,
1787which is the directory with kernel headers that are required to build
1788out-of-tree modules. Your recipe will also need the following::
1789
1790 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1791
1792Compilation
1793-----------
1794
1795During a build, the ``do_compile`` task happens after source is fetched,
1796unpacked, and configured. If the recipe passes through ``do_compile``
1797successfully, nothing needs to be done.
1798
1799However, if the compile step fails, you need to diagnose the failure.
1800Here are some common issues that cause failures.
1801
1802.. note::
1803
1804 For cases where improper paths are detected for configuration files
1805 or for when libraries/headers cannot be found, be sure you are using
1806 the more robust ``pkg-config``. See the note in section
1807 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information.
1808
1809- *Parallel build failures:* These failures manifest themselves as
1810 intermittent errors, or errors reporting that a file or directory
1811 that should be created by some other part of the build process could
1812 not be found. This type of failure can occur even if, upon
1813 inspection, the file or directory does exist after the build has
1814 failed, because that part of the build process happened in the wrong
1815 order.
1816
1817 To fix the problem, you need to either satisfy the missing dependency
1818 in the Makefile or whatever script produced the Makefile, or (as a
1819 workaround) set :term:`PARALLEL_MAKE` to an empty string::
1820
1821 PARALLEL_MAKE = ""
1822
1823 For information on parallel Makefile issues, see the
1824 ":ref:`dev-manual/common-tasks:debugging parallel make races`" section.
1825
1826- *Improper host path usage:* This failure applies to recipes building
1827 for the target or ``nativesdk`` only. The failure occurs when the
1828 compilation process uses improper headers, libraries, or other files
1829 from the host system when cross-compiling for the target.
1830
1831 To fix the problem, examine the ``log.do_compile`` file to identify
1832 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1833 so forth) and then either add configure options, apply a patch, or do
1834 both.
1835
1836- *Failure to find required libraries/headers:* If a build-time
1837 dependency is missing because it has not been declared in
1838 :term:`DEPENDS`, or because the
1839 dependency exists but the path used by the build process to find the
1840 file is incorrect and the configure step did not detect it, the
1841 compilation process could fail. For either of these failures, the
1842 compilation process notes that files could not be found. In these
1843 cases, you need to go back and add additional options to the
1844 configure script as well as possibly add additional build-time
1845 dependencies to :term:`DEPENDS`.
1846
1847 Occasionally, it is necessary to apply a patch to the source to
1848 ensure the correct paths are used. If you need to specify paths to
1849 find files staged into the sysroot from other recipes, use the
1850 variables that the OpenEmbedded build system provides (e.g.
1851 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
1852 forth).
1853
1854Installing
1855----------
1856
1857During ``do_install``, the task copies the built files along with their
1858hierarchy to locations that would mirror their locations on the target
1859device. The installation process copies files from the
1860``${``\ :term:`S`\ ``}``,
1861``${``\ :term:`B`\ ``}``, and
1862``${``\ :term:`WORKDIR`\ ``}``
1863directories to the ``${``\ :term:`D`\ ``}``
1864directory to create the structure as it should appear on the target
1865system.
1866
1867How your software is built affects what you must do to be sure your
1868software is installed correctly. The following list describes what you
1869must do for installation depending on the type of build system used by
1870the software being built:
1871
1872- *Autotools and CMake:* If the software your recipe is building uses
1873 Autotools or CMake, the OpenEmbedded build system understands how to
1874 install the software. Consequently, you do not have to have a
1875 ``do_install`` task as part of your recipe. You just need to make
1876 sure the install portion of the build completes with no issues.
1877 However, if you wish to install additional files not already being
1878 installed by ``make install``, you should do this using a
1879 ``do_install:append`` function using the install command as described
1880 in the "Manual" bulleted item later in this list.
1881
1882- *Other (using* ``make install``\ *)*: You need to define a ``do_install``
1883 function in your recipe. The function should call
1884 ``oe_runmake install`` and will likely need to pass in the
1885 destination directory as well. How you pass that path is dependent on
1886 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1887 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1888
1889 For an example recipe using ``make install``, see the
1890 ":ref:`dev-manual/common-tasks:makefile-based package`" section.
1891
1892- *Manual:* You need to define a ``do_install`` function in your
1893 recipe. The function must first use ``install -d`` to create the
1894 directories under
1895 ``${``\ :term:`D`\ ``}``. Once the
1896 directories exist, your function can use ``install`` to manually
1897 install the built software into the directories.
1898
1899 You can find more information on ``install`` at
1900 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
1901
1902For the scenarios that do not use Autotools or CMake, you need to track
1903the installation and diagnose and fix any issues until everything
1904installs correctly. You need to look in the default location of
1905``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1906installed correctly.
1907
1908.. note::
1909
1910 - During the installation process, you might need to modify some of
1911 the installed files to suit the target layout. For example, you
1912 might need to replace hard-coded paths in an initscript with
1913 values of variables provided by the build system, such as
1914 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1915 modifications during ``do_install``, be sure to modify the
1916 destination file after copying rather than before copying.
1917 Modifying after copying ensures that the build system can
1918 re-execute ``do_install`` if needed.
1919
1920 - ``oe_runmake install``, which can be run directly or can be run
1921 indirectly by the
1922 :ref:`autotools <ref-classes-autotools>` and
1923 :ref:`cmake <ref-classes-cmake>` classes,
1924 runs ``make install`` in parallel. Sometimes, a Makefile can have
1925 missing dependencies between targets that can result in race
1926 conditions. If you experience intermittent failures during
1927 ``do_install``, you might be able to work around them by disabling
1928 parallel Makefile installs by adding the following to the recipe::
1929
1930 PARALLEL_MAKEINST = ""
1931
1932 See :term:`PARALLEL_MAKEINST` for additional information.
1933
1934 - If you need to install one or more custom CMake toolchain files
1935 that are supplied by the application you are building, install the
1936 files to ``${D}${datadir}/cmake/Modules`` during
1937 :ref:`ref-tasks-install`.
1938
1939Enabling System Services
1940------------------------
1941
1942If you want to install a service, which is a process that usually starts
1943on boot and runs in the background, then you must include some
1944additional definitions in your recipe.
1945
1946If you are adding services and the service initialization script or the
1947service file itself is not installed, you must provide for that
1948installation in your recipe using a ``do_install:append`` function. If
1949your recipe already has a ``do_install`` function, update the function
1950near its end rather than adding an additional ``do_install:append``
1951function.
1952
1953When you create the installation for your services, you need to
1954accomplish what is normally done by ``make install``. In other words,
1955make sure your installation arranges the output similar to how it is
1956arranged on the target system.
1957
1958The OpenEmbedded build system provides support for starting services two
1959different ways:
1960
1961- *SysVinit:* SysVinit is a system and service manager that manages the
1962 init system used to control the very basic functions of your system.
1963 The init program is the first program started by the Linux kernel
1964 when the system boots. Init then controls the startup, running and
1965 shutdown of all other programs.
1966
1967 To enable a service using SysVinit, your recipe needs to inherit the
1968 :ref:`update-rc.d <ref-classes-update-rc.d>`
1969 class. The class helps facilitate safely installing the package on
1970 the target.
1971
1972 You will need to set the
1973 :term:`INITSCRIPT_PACKAGES`,
1974 :term:`INITSCRIPT_NAME`,
1975 and
1976 :term:`INITSCRIPT_PARAMS`
1977 variables within your recipe.
1978
1979- *systemd:* System Management Daemon (systemd) was designed to replace
1980 SysVinit and to provide enhanced management of services. For more
1981 information on systemd, see the systemd homepage at
1982 https://freedesktop.org/wiki/Software/systemd/.
1983
1984 To enable a service using systemd, your recipe needs to inherit the
1985 :ref:`systemd <ref-classes-systemd>` class. See
1986 the ``systemd.bbclass`` file located in your :term:`Source Directory`
1987 section for
1988 more information.
1989
1990Packaging
1991---------
1992
1993Successful packaging is a combination of automated processes performed
1994by the OpenEmbedded build system and some specific steps you need to
1995take. The following list describes the process:
1996
1997- *Splitting Files*: The ``do_package`` task splits the files produced
1998 by the recipe into logical components. Even software that produces a
1999 single binary might still have debug symbols, documentation, and
2000 other logical components that should be split out. The ``do_package``
2001 task ensures that files are split up and packaged correctly.
2002
2003- *Running QA Checks*: The
2004 :ref:`insane <ref-classes-insane>` class adds a
2005 step to the package generation process so that output quality
2006 assurance checks are generated by the OpenEmbedded build system. This
2007 step performs a range of checks to be sure the build's output is free
2008 of common problems that show up during runtime. For information on
2009 these checks, see the
2010 :ref:`insane <ref-classes-insane>` class and
2011 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
2012 chapter in the Yocto Project Reference Manual.
2013
2014- *Hand-Checking Your Packages*: After you build your software, you
2015 need to be sure your packages are correct. Examine the
2016 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
2017 directory and make sure files are where you expect them to be. If you
2018 discover problems, you can set
2019 :term:`PACKAGES`,
2020 :term:`FILES`,
2021 ``do_install(:append)``, and so forth as needed.
2022
2023- *Splitting an Application into Multiple Packages*: If you need to
2024 split an application into several packages, see the
2025 ":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
2026 section for an example.
2027
2028- *Installing a Post-Installation Script*: For an example showing how
2029 to install a post-installation script, see the
2030 ":ref:`dev-manual/common-tasks:post-installation scripts`" section.
2031
2032- *Marking Package Architecture*: Depending on what your recipe is
2033 building and how it is configured, it might be important to mark the
2034 packages produced as being specific to a particular machine, or to
2035 mark them as not being specific to a particular machine or
2036 architecture at all.
2037
2038 By default, packages apply to any machine with the same architecture
2039 as the target machine. When a recipe produces packages that are
2040 machine-specific (e.g. the
2041 :term:`MACHINE` value is passed
2042 into the configure script or a patch is applied only for a particular
2043 machine), you should mark them as such by adding the following to the
2044 recipe::
2045
2046 PACKAGE_ARCH = "${MACHINE_ARCH}"
2047
2048 On the other hand, if the recipe produces packages that do not
2049 contain anything specific to the target machine or architecture at
2050 all (e.g. recipes that simply package script files or configuration
2051 files), you should use the
2052 :ref:`allarch <ref-classes-allarch>` class to
2053 do this for you by adding this to your recipe::
2054
2055 inherit allarch
2056
2057 Ensuring that the package architecture is correct is not critical
2058 while you are doing the first few builds of your recipe. However, it
2059 is important in order to ensure that your recipe rebuilds (or does
2060 not rebuild) appropriately in response to changes in configuration,
2061 and to ensure that you get the appropriate packages installed on the
2062 target machine, particularly if you run separate builds for more than
2063 one target machine.
2064
2065Sharing Files Between Recipes
2066-----------------------------
2067
2068Recipes often need to use files provided by other recipes on the build
2069host. For example, an application linking to a common library needs
2070access to the library itself and its associated headers. The way this
2071access is accomplished is by populating a sysroot with files. Each
2072recipe has two sysroots in its work directory, one for target files
2073(``recipe-sysroot``) and one for files that are native to the build host
2074(``recipe-sysroot-native``).
2075
2076.. note::
2077
2078 You could find the term "staging" used within the Yocto project
2079 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
2080 variable).
2081
2082Recipes should never populate the sysroot directly (i.e. write files
2083into sysroot). Instead, files should be installed into standard
2084locations during the
2085:ref:`ref-tasks-install` task within
2086the ``${``\ :term:`D`\ ``}`` directory. The
2087reason for this limitation is that almost all files that populate the
2088sysroot are cataloged in manifests in order to ensure the files can be
2089removed later when a recipe is either modified or removed. Thus, the
2090sysroot is able to remain free from stale files.
2091
2092A subset of the files installed by the :ref:`ref-tasks-install` task are
2093used by the :ref:`ref-tasks-populate_sysroot` task as defined by the the
2094:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
2095is possible to modify the list of directories that populate the sysroot.
2096The following example shows how you could add the ``/opt`` directory to
2097the list of directories within a recipe::
2098
2099 SYSROOT_DIRS += "/opt"
2100
2101.. note::
2102
2103 The `/sysroot-only` is to be used by recipes that generate artifacts
2104 that are not included in the target filesystem, allowing them to share
2105 these artifacts without needing to use the :term:`DEPLOY_DIR`.
2106
2107For a more complete description of the :ref:`ref-tasks-populate_sysroot`
2108task and its associated functions, see the
2109:ref:`staging <ref-classes-staging>` class.
2110
2111Using Virtual Providers
2112-----------------------
2113
2114Prior to a build, if you know that several different recipes provide the
2115same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2116as a placeholder for the actual provider. The actual provider is
2117determined at build-time.
2118
2119A common scenario where a virtual provider is used would be for the
2120kernel recipe. Suppose you have three kernel recipes whose
2121:term:`PN` values map to ``kernel-big``,
2122``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2123in some way uses a :term:`PROVIDES`
2124statement that essentially identifies itself as being able to provide
2125``virtual/kernel``. Here is one way through the
2126:ref:`kernel <ref-classes-kernel>` class::
2127
2128 PROVIDES += "virtual/kernel"
2129
2130Any recipe that inherits the :ref:`kernel <ref-classes-kernel>` class is
2131going to utilize a :term:`PROVIDES` statement that identifies that recipe as
2132being able to provide the ``virtual/kernel`` item.
2133
2134Now comes the time to actually build an image and you need a kernel
2135recipe, but which one? You can configure your build to call out the
2136kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
2137an example, consider the :yocto_git:`x86-base.inc
2138</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a
2139machine (i.e. :term:`MACHINE`) configuration file. This include file is the
2140reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
2141relevant lines from the include file::
2142
2143 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2144 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2145
2146When you use a virtual provider, you do not have to "hard code" a recipe
2147name as a build dependency. You can use the
2148:term:`DEPENDS` variable to state the
2149build is dependent on ``virtual/kernel`` for example::
2150
2151 DEPENDS = "virtual/kernel"
2152
2153During the build, the OpenEmbedded build system picks
2154the correct recipe needed for the ``virtual/kernel`` dependency based on
2155the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
2156mentioned at the beginning of this section, configure your build as
2157follows::
2158
2159 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
2160
2161.. note::
2162
2163 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
2164 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
2165 recipes from building is usually the desired behavior since this mechanism's
2166 purpose is to select between mutually exclusive alternative providers.
2167
2168The following lists specific examples of virtual providers:
2169
2170- ``virtual/kernel``: Provides the name of the kernel recipe to use
2171 when building a kernel image.
2172
2173- ``virtual/bootloader``: Provides the name of the bootloader to use
2174 when building an image.
2175
2176- ``virtual/libgbm``: Provides ``gbm.pc``.
2177
2178- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2179
2180- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2181
2182- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2183
2184- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2185
2186.. note::
2187
2188 Virtual providers only apply to build time dependencies specified with
2189 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2190 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2191
2192Properly Versioning Pre-Release Recipes
2193---------------------------------------
2194
2195Sometimes the name of a recipe can lead to versioning problems when the
2196recipe is upgraded to a final release. For example, consider the
2197``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
2198the ":ref:`dev-manual/common-tasks:storing and naming the recipe`" section.
2199This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
2200released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2201change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2202build system and package managers, so the resulting packages will not
2203correctly trigger an upgrade.
2204
2205In order to ensure the versions compare properly, the recommended
2206convention is to set :term:`PV` within the
2207recipe to "previous_version+current_version". You can use an additional
2208variable so that you can use the current version elsewhere. Here is an
2209example::
2210
2211 REALPV = "0.8.16-rc1"
2212 PV = "0.8.15+${REALPV}"
2213
2214Post-Installation Scripts
2215-------------------------
2216
2217Post-installation scripts run immediately after installing a package on
2218the target or during image creation when a package is included in an
2219image. To add a post-installation script to a package, add a
2220``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file
2221(``.bb``) and replace `PACKAGENAME` with the name of the package you want
2222to attach to the ``postinst`` script. To apply the post-installation
2223script to the main package for the recipe, which is usually what is
2224required, specify
2225``${``\ :term:`PN`\ ``}`` in place of
2226PACKAGENAME.
2227
2228A post-installation function has the following structure::
2229
2230 pkg_postinst:PACKAGENAME() {
2231 # Commands to carry out
2232 }
2233
2234The script defined in the post-installation function is called when the
2235root filesystem is created. If the script succeeds, the package is
2236marked as installed.
2237
2238.. note::
2239
2240 Any RPM post-installation script that runs on the target should
2241 return a 0 exit code. RPM does not allow non-zero exit codes for
2242 these scripts, and the RPM package manager will cause the package to
2243 fail installation on the target.
2244
2245Sometimes it is necessary for the execution of a post-installation
2246script to be delayed until the first boot. For example, the script might
2247need to be executed on the device itself. To delay script execution
2248until boot time, you must explicitly mark post installs to defer to the
2249target. You can use ``pkg_postinst_ontarget()`` or call
2250``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2251failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2252error during the
2253:ref:`ref-tasks-rootfs` task.
2254
2255If you have recipes that use ``pkg_postinst`` function and they require
2256the use of non-standard native tools that have dependencies during
2257root filesystem construction, you need to use the
2258:term:`PACKAGE_WRITE_DEPS`
2259variable in your recipe to list these tools. If you do not use this
2260variable, the tools might be missing and execution of the
2261post-installation script is deferred until first boot. Deferring the
2262script to the first boot is undesirable and impossible for read-only
2263root filesystems.
2264
2265.. note::
2266
2267 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
2268 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
2269 respectively. These scrips work in exactly the same way as does
2270 ``pkg_postinst`` with the exception that they run at different times. Also,
2271 because of when they run, they are not applicable to being run at image
2272 creation time like ``pkg_postinst``.
2273
2274Testing
2275-------
2276
2277The final step for completing your recipe is to be sure that the
2278software you built runs correctly. To accomplish runtime testing, add
2279the build's output packages to your image and test them on the target.
2280
2281For information on how to customize your image by adding specific
2282packages, see ":ref:`dev-manual/common-tasks:customizing images`" section.
2283
2284Examples
2285--------
2286
2287To help summarize how to write a recipe, this section provides some
2288examples given various scenarios:
2289
2290- Recipes that use local files
2291
2292- Using an Autotooled package
2293
2294- Using a Makefile-based package
2295
2296- Splitting an application into multiple packages
2297
2298- Adding binaries to an image
2299
2300Single .c File Package (Hello World!)
2301~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2302
2303Building an application from a single file that is stored locally (e.g.
2304under ``files``) requires a recipe that has the file listed in the
2305:term:`SRC_URI` variable. Additionally, you need to manually write the
2306``do_compile`` and ``do_install`` tasks. The :term:`S` variable defines the
2307directory containing the source code, which is set to
2308:term:`WORKDIR` in this case - the
2309directory BitBake uses for the build.
2310::
2311
2312 SUMMARY = "Simple helloworld application"
2313 SECTION = "examples"
2314 LICENSE = "MIT"
2315 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2316
2317 SRC_URI = "file://helloworld.c"
2318
2319 S = "${WORKDIR}"
2320
2321 do_compile() {
2322 ${CC} ${LDFLAGS} helloworld.c -o helloworld
2323 }
2324
2325 do_install() {
2326 install -d ${D}${bindir}
2327 install -m 0755 helloworld ${D}${bindir}
2328 }
2329
2330By default, the ``helloworld``, ``helloworld-dbg``, and
2331``helloworld-dev`` packages are built. For information on how to
2332customize the packaging process, see the
2333":ref:`dev-manual/common-tasks:splitting an application into multiple packages`"
2334section.
2335
2336Autotooled Package
2337~~~~~~~~~~~~~~~~~~
2338
2339Applications that use Autotools such as ``autoconf`` and ``automake``
2340require a recipe that has a source archive listed in :term:`SRC_URI` and
2341also inherit the
2342:ref:`autotools <ref-classes-autotools>` class,
2343which contains the definitions of all the steps needed to build an
2344Autotool-based application. The result of the build is automatically
2345packaged. And, if the application uses NLS for localization, packages
2346with local information are generated (one package per language).
2347Following is one example: (``hello_2.3.bb``)
2348::
2349
2350 SUMMARY = "GNU Helloworld application"
2351 SECTION = "examples"
2352 LICENSE = "GPL-2.0-or-later"
2353 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2354
2355 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2356
2357 inherit autotools gettext
2358
2359The variable :term:`LIC_FILES_CHKSUM` is used to track source license
2360changes as described in the
2361":ref:`dev-manual/common-tasks:tracking license changes`" section in
2362the Yocto Project Overview and Concepts Manual. You can quickly create
2363Autotool-based recipes in a manner similar to the previous example.
2364
2365Makefile-Based Package
2366~~~~~~~~~~~~~~~~~~~~~~
2367
2368Applications that use GNU ``make`` also require a recipe that has the
2369source archive listed in :term:`SRC_URI`. You do not need to add a
2370``do_compile`` step since by default BitBake starts the ``make`` command
2371to compile the application. If you need additional ``make`` options, you
2372should store them in the
2373:term:`EXTRA_OEMAKE` or
2374:term:`PACKAGECONFIG_CONFARGS`
2375variables. BitBake passes these options into the GNU ``make``
2376invocation. Note that a ``do_install`` task is still required.
2377Otherwise, BitBake runs an empty ``do_install`` task by default.
2378
2379Some applications might require extra parameters to be passed to the
2380compiler. For example, the application might need an additional header
2381path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
2382following example shows this::
2383
2384 CFLAGS:prepend = "-I ${S}/include "
2385
2386In the following example, ``lz4`` is a makefile-based package::
2387
2388 SUMMARY = "Extremely Fast Compression algorithm"
2389 DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems."
2390 HOMEPAGE = "https://github.com/lz4/lz4"
2391
2392 LICENSE = "BSD-2-Clause | GPL-2.0-only"
2393 LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc \
2394 file://programs/COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
2395 file://LICENSE;md5=d57c0d21cb917fb4e0af2454aa48b956 \
2396 "
2397
2398 PE = "1"
2399
2400 SRCREV = "d44371841a2f1728a3f36839fd4b7e872d0927d3"
2401
2402 SRC_URI = "git://github.com/lz4/lz4.git;branch=release;protocol=https \
2403 file://CVE-2021-3520.patch \
2404 "
2405 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>.*)"
2406
2407 S = "${WORKDIR}/git"
2408
2409 # Fixed in r118, which is larger than the current version.
2410 CVE_CHECK_IGNORE += "CVE-2014-4715"
2411
2412 EXTRA_OEMAKE = "PREFIX=${prefix} CC='${CC}' CFLAGS='${CFLAGS}' DESTDIR=${D} LIBDIR=${libdir} INCLUDEDIR=${includedir} BUILD_STATIC=no"
2413
2414 do_install() {
2415 oe_runmake install
2416 }
2417
2418 BBCLASSEXTEND = "native nativesdk"
2419
2420Splitting an Application into Multiple Packages
2421~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2422
2423You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
2424application into multiple packages.
2425
2426Following is an example that uses the ``libxpm`` recipe. By default,
2427this recipe generates a single package that contains the library along
2428with a few binaries. You can modify the recipe to split the binaries
2429into separate packages::
2430
2431 require xorg-lib-common.inc
2432
2433 SUMMARY = "Xpm: X Pixmap extension library"
2434 LICENSE = "MIT"
2435 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2436 DEPENDS += "libxext libsm libxt"
2437 PE = "1"
2438
2439 XORG_PN = "libXpm"
2440
2441 PACKAGES =+ "sxpm cxpm"
2442 FILES:cxpm = "${bindir}/cxpm"
2443 FILES:sxpm = "${bindir}/sxpm"
2444
2445In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2446binaries in separate packages. Since ``bindir`` would be packaged into
2447the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
2448so additional package names are added to the start of list. This results
2449in the extra ``FILES:*`` variables then containing information that
2450define which files and directories go into which packages. Files
2451included by earlier packages are skipped by latter packages. Thus, the
2452main :term:`PN` package does not include the above listed files.
2453
2454Packaging Externally Produced Binaries
2455~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2456
2457Sometimes, you need to add pre-compiled binaries to an image. For
2458example, suppose that there are binaries for proprietary code,
2459created by a particular division of a company. Your part of the company
2460needs to use those binaries as part of an image that you are building
2461using the OpenEmbedded build system. Since you only have the binaries
2462and not the source code, you cannot use a typical recipe that expects to
2463fetch the source specified in
2464:term:`SRC_URI` and then compile it.
2465
2466One method is to package the binaries and then install them as part of
2467the image. Generally, it is not a good idea to package binaries since,
2468among other things, it can hinder the ability to reproduce builds and
2469could lead to compatibility problems with ABI in the future. However,
2470sometimes you have no choice.
2471
2472The easiest solution is to create a recipe that uses the
2473:ref:`bin_package <ref-classes-bin-package>` class
2474and to be sure that you are using default locations for build artifacts.
2475In most cases, the :ref:`bin_package <ref-classes-bin-package>` class handles "skipping" the
2476configure and compile steps as well as sets things up to grab packages
2477from the appropriate area. In particular, this class sets ``noexec`` on
2478both the :ref:`ref-tasks-configure`
2479and :ref:`ref-tasks-compile` tasks,
2480sets ``FILES:${PN}`` to "/" so that it picks up all files, and sets up a
2481:ref:`ref-tasks-install` task, which
2482effectively copies all files from ``${S}`` to ``${D}``. The
2483:ref:`bin_package <ref-classes-bin-package>` class works well when the files extracted into ``${S}``
2484are already laid out in the way they should be laid out on the target.
2485For more information on these variables, see the
2486:term:`FILES`,
2487:term:`PN`,
2488:term:`S`, and
2489:term:`D` variables in the Yocto Project
2490Reference Manual's variable glossary.
2491
2492.. note::
2493
2494 - Using :term:`DEPENDS` is a good
2495 idea even for components distributed in binary form, and is often
2496 necessary for shared libraries. For a shared library, listing the
2497 library dependencies in :term:`DEPENDS` makes sure that the libraries
2498 are available in the staging sysroot when other recipes link
2499 against the library, which might be necessary for successful
2500 linking.
2501
2502 - Using :term:`DEPENDS` also allows runtime dependencies between
2503 packages to be added automatically. See the
2504 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
2505 section in the Yocto Project Overview and Concepts Manual for more
2506 information.
2507
2508If you cannot use the :ref:`bin_package <ref-classes-bin-package>` class, you need to be sure you are
2509doing the following:
2510
2511- Create a recipe where the
2512 :ref:`ref-tasks-configure` and
2513 :ref:`ref-tasks-compile` tasks do
2514 nothing: It is usually sufficient to just not define these tasks in
2515 the recipe, because the default implementations do nothing unless a
2516 Makefile is found in
2517 ``${``\ :term:`S`\ ``}``.
2518
2519 If ``${S}`` might contain a Makefile, or if you inherit some class
2520 that replaces ``do_configure`` and ``do_compile`` with custom
2521 versions, then you can use the
2522 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2523 flag to turn the tasks into no-ops, as follows::
2524
2525 do_configure[noexec] = "1"
2526 do_compile[noexec] = "1"
2527
2528 Unlike
2529 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2530 using the flag preserves the dependency chain from the
2531 :ref:`ref-tasks-fetch`,
2532 :ref:`ref-tasks-unpack`, and
2533 :ref:`ref-tasks-patch` tasks to the
2534 :ref:`ref-tasks-install` task.
2535
2536- Make sure your ``do_install`` task installs the binaries
2537 appropriately.
2538
2539- Ensure that you set up :term:`FILES`
2540 (usually
2541 ``FILES:${``\ :term:`PN`\ ``}``) to
2542 point to the files you have installed, which of course depends on
2543 where you have installed them and whether those files are in
2544 different locations than the defaults.
2545
2546Following Recipe Style Guidelines
2547---------------------------------
2548
2549When writing recipes, it is good to conform to existing style
2550guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
2551provides rough guidelines for preferred recipe style.
2552
2553It is common for existing recipes to deviate a bit from this style.
2554However, aiming for at least a consistent style is a good idea. Some
2555practices, such as omitting spaces around ``=`` operators in assignments
2556or ordering recipe components in an erratic way, are widely seen as poor
2557style.
2558
2559Recipe Syntax
2560-------------
2561
2562Understanding recipe file syntax is important for writing recipes. The
2563following list overviews the basic items that make up a BitBake recipe
2564file. For more complete BitBake syntax descriptions, see the
2565":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`"
2566chapter of the BitBake User Manual.
2567
2568- *Variable Assignments and Manipulations:* Variable assignments allow
2569 a value to be assigned to a variable. The assignment can be static
2570 text or might include the contents of other variables. In addition to
2571 the assignment, appending and prepending operations are also
2572 supported.
2573
2574 The following example shows some of the ways you can use variables in
2575 recipes::
2576
2577 S = "${WORKDIR}/postfix-${PV}"
2578 CFLAGS += "-DNO_ASM"
2579 SRC_URI:append = " file://fixup.patch"
2580
2581- *Functions:* Functions provide a series of actions to be performed.
2582 You usually use functions to override the default implementation of a
2583 task function or to complement a default function (i.e. append or
2584 prepend to an existing function). Standard functions use ``sh`` shell
2585 syntax, although access to OpenEmbedded variables and internal
2586 methods are also available.
2587
2588 Here is an example function from the ``sed`` recipe::
2589
2590 do_install () {
2591 autotools_do_install
2592 install -d ${D}${base_bindir}
2593 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2594 rmdir ${D}${bindir}/
2595 }
2596
2597 It is
2598 also possible to implement new functions that are called between
2599 existing tasks as long as the new functions are not replacing or
2600 complementing the default functions. You can implement functions in
2601 Python instead of shell. Both of these options are not seen in the
2602 majority of recipes.
2603
2604- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2605 to include common functions (``inherit``), load parts of a recipe
2606 from other files (``include`` and ``require``) and export variables
2607 to the environment (``export``).
2608
2609 The following example shows the use of some of these keywords::
2610
2611 export POSTCONF = "${STAGING_BINDIR}/postconf"
2612 inherit autoconf
2613 require otherfile.inc
2614
2615- *Comments (#):* Any lines that begin with the hash character (``#``)
2616 are treated as comment lines and are ignored::
2617
2618 # This is a comment
2619
2620This next list summarizes the most important and most commonly used
2621parts of the recipe syntax. For more information on these parts of the
2622syntax, you can reference the
2623:doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata` chapter
2624in the BitBake User Manual.
2625
2626- *Line Continuation (\\):* Use the backward slash (``\``) character to
2627 split a statement over multiple lines. Place the slash character at
2628 the end of the line that is to be continued on the next line::
2629
2630 VAR = "A really long \
2631 line"
2632
2633 .. note::
2634
2635 You cannot have any characters including spaces or tabs after the
2636 slash character.
2637
2638- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
2639 access the contents of a variable::
2640
2641 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2642
2643 .. note::
2644
2645 It is important to understand that the value of a variable
2646 expressed in this form does not get substituted automatically. The
2647 expansion of these expressions happens on-demand later (e.g.
2648 usually when a function that makes reference to the variable
2649 executes). This behavior ensures that the values are most
2650 appropriate for the context in which they are finally used. On the
2651 rare occasion that you do need the variable expression to be
2652 expanded immediately, you can use the
2653 :=
2654 operator instead of
2655 =
2656 when you make the assignment, but this is not generally needed.
2657
2658- *Quote All Assignments ("value"):* Use double quotes around values in
2659 all variable assignments (e.g. ``"value"``). Following is an example::
2660
2661 VAR1 = "${OTHERVAR}"
2662 VAR2 = "The version is ${PV}"
2663
2664- *Conditional Assignment (?=):* Conditional assignment is used to
2665 assign a value to a variable, but only when the variable is currently
2666 unset. Use the question mark followed by the equal sign (``?=``) to
2667 make a "soft" assignment used for conditional assignment. Typically,
2668 "soft" assignments are used in the ``local.conf`` file for variables
2669 that are allowed to come through from the external environment.
2670
2671 Here is an example where ``VAR1`` is set to "New value" if it is
2672 currently empty. However, if ``VAR1`` has already been set, it
2673 remains unchanged::
2674
2675 VAR1 ?= "New value"
2676
2677 In this next example, ``VAR1`` is left with the value "Original value"::
2678
2679 VAR1 = "Original value"
2680 VAR1 ?= "New value"
2681
2682- *Appending (+=):* Use the plus character followed by the equals sign
2683 (``+=``) to append values to existing variables.
2684
2685 .. note::
2686
2687 This operator adds a space between the existing content of the
2688 variable and the new content.
2689
2690 Here is an example::
2691
2692 SRC_URI += "file://fix-makefile.patch"
2693
2694- *Prepending (=+):* Use the equals sign followed by the plus character
2695 (``=+``) to prepend values to existing variables.
2696
2697 .. note::
2698
2699 This operator adds a space between the new content and the
2700 existing content of the variable.
2701
2702 Here is an example::
2703
2704 VAR =+ "Starts"
2705
2706- *Appending (:append):* Use the ``:append`` operator to append values
2707 to existing variables. This operator does not add any additional
2708 space. Also, the operator is applied after all the ``+=``, and ``=+``
2709 operators have been applied and after all ``=`` assignments have
2710 occurred.
2711
2712 The following example shows the space being explicitly added to the
2713 start to ensure the appended value is not merged with the existing
2714 value::
2715
2716 SRC_URI:append = " file://fix-makefile.patch"
2717
2718 You can also use
2719 the ``:append`` operator with overrides, which results in the actions
2720 only being performed for the specified target or machine::
2721
2722 SRC_URI:append:sh4 = " file://fix-makefile.patch"
2723
2724- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend
2725 values to existing variables. This operator does not add any
2726 additional space. Also, the operator is applied after all the ``+=``,
2727 and ``=+`` operators have been applied and after all ``=``
2728 assignments have occurred.
2729
2730 The following example shows the space being explicitly added to the
2731 end to ensure the prepended value is not merged with the existing
2732 value::
2733
2734 CFLAGS:prepend = "-I${S}/myincludes "
2735
2736 You can also use the
2737 ``:prepend`` operator with overrides, which results in the actions
2738 only being performed for the specified target or machine::
2739
2740 CFLAGS:prepend:sh4 = "-I${S}/myincludes "
2741
2742- *Overrides:* You can use overrides to set a value conditionally,
2743 typically based on how the recipe is being built. For example, to set
2744 the :term:`KBRANCH` variable's
2745 value to "standard/base" for any target
2746 :term:`MACHINE`, except for
2747 qemuarm where it should be set to "standard/arm-versatile-926ejs",
2748 you would do the following::
2749
2750 KBRANCH = "standard/base"
2751 KBRANCH:qemuarm = "standard/arm-versatile-926ejs"
2752
2753 Overrides are also used to separate
2754 alternate values of a variable in other situations. For example, when
2755 setting variables such as
2756 :term:`FILES` and
2757 :term:`RDEPENDS` that are
2758 specific to individual packages produced by a recipe, you should
2759 always use an override that specifies the name of the package.
2760
2761- *Indentation:* Use spaces for indentation rather than tabs. For
2762 shell functions, both currently work. However, it is a policy
2763 decision of the Yocto Project to use tabs in shell functions. Realize
2764 that some layers have a policy to use spaces for all indentation.
2765
2766- *Using Python for Complex Operations:* For more advanced processing,
2767 it is possible to use Python code during variable assignments (e.g.
2768 search and replacement on a variable).
2769
2770 You indicate Python code using the ``${@python_code}`` syntax for the
2771 variable assignment::
2772
2773 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2774
2775- *Shell Function Syntax:* Write shell functions as if you were writing
2776 a shell script when you describe a list of actions to take. You
2777 should ensure that your script works with a generic ``sh`` and that
2778 it does not require any ``bash`` or other shell-specific
2779 functionality. The same considerations apply to various system
2780 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2781 might wish to use. If in doubt, you should check with multiple
2782 implementations - including those from BusyBox.
2783
2784Adding a New Machine
2785====================
2786
2787Adding a new machine to the Yocto Project is a straightforward process.
2788This section describes how to add machines that are similar to those
2789that the Yocto Project already supports.
2790
2791.. note::
2792
2793 Although well within the capabilities of the Yocto Project, adding a
2794 totally new architecture might require changes to ``gcc``/``glibc``
2795 and to the site information, which is beyond the scope of this
2796 manual.
2797
2798For a complete example that shows how to add a new machine, see the
2799":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2800section in the Yocto Project Board Support Package (BSP) Developer's
2801Guide.
2802
2803Adding the Machine Configuration File
2804-------------------------------------
2805
2806To add a new machine, you need to add a new machine configuration file
2807to the layer's ``conf/machine`` directory. This configuration file
2808provides details about the device you are adding.
2809
2810The OpenEmbedded build system uses the root name of the machine
2811configuration file to reference the new machine. For example, given a
2812machine configuration file named ``crownbay.conf``, the build system
2813recognizes the machine as "crownbay".
2814
2815The most important variables you must set in your machine configuration
2816file or include from a lower-level configuration file are as follows:
2817
2818- :term:`TARGET_ARCH` (e.g. "arm")
2819
2820- ``PREFERRED_PROVIDER_virtual/kernel``
2821
2822- :term:`MACHINE_FEATURES` (e.g. "apm screen wifi")
2823
2824You might also need these variables:
2825
2826- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
2827
2828- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
2829
2830- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
2831
2832You can find full details on these variables in the reference section.
2833You can leverage existing machine ``.conf`` files from
2834``meta-yocto-bsp/conf/machine/``.
2835
2836Adding a Kernel for the Machine
2837-------------------------------
2838
2839The OpenEmbedded build system needs to be able to build a kernel for the
2840machine. You need to either create a new kernel recipe for this machine,
2841or extend an existing kernel recipe. You can find several kernel recipe
2842examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2843you can use as references.
2844
2845If you are creating a new kernel recipe, normal recipe-writing rules
2846apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
2847necessary patches and set :term:`S` to point at the source code. You need to
2848create a ``do_configure`` task that configures the unpacked kernel with
2849a ``defconfig`` file. You can do this by using a ``make defconfig``
2850command or, more commonly, by copying in a suitable ``defconfig`` file
2851and then running ``make oldconfig``. By making use of ``inherit kernel``
2852and potentially some of the ``linux-*.inc`` files, most other
2853functionality is centralized and the defaults of the class normally work
2854well.
2855
2856If you are extending an existing kernel recipe, it is usually a matter
2857of adding a suitable ``defconfig`` file. The file needs to be added into
2858a location similar to ``defconfig`` files used for other machines in a
2859given kernel recipe. A possible way to do this is by listing the file in
2860the :term:`SRC_URI` and adding the machine to the expression in
2861:term:`COMPATIBLE_MACHINE`::
2862
2863 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2864
2865For more information on ``defconfig`` files, see the
2866":ref:`kernel-dev/common:changing the configuration`"
2867section in the Yocto Project Linux Kernel Development Manual.
2868
2869Adding a Formfactor Configuration File
2870--------------------------------------
2871
2872A formfactor configuration file provides information about the target
2873hardware for which the image is being built and information that the
2874build system cannot obtain from other sources such as the kernel. Some
2875examples of information contained in a formfactor configuration file
2876include framebuffer orientation, whether or not the system has a
2877keyboard, the positioning of the keyboard in relation to the screen, and
2878the screen resolution.
2879
2880The build system uses reasonable defaults in most cases. However, if
2881customization is necessary, you need to create a ``machconfig`` file in
2882the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2883contains directories for specific machines such as ``qemuarm`` and
2884``qemux86``. For information about the settings available and the
2885defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2886found in the same area.
2887
2888Following is an example for "qemuarm" machine::
2889
2890 HAVE_TOUCHSCREEN=1
2891 HAVE_KEYBOARD=1
2892 DISPLAY_CAN_ROTATE=0
2893 DISPLAY_ORIENTATION=0
2894 #DISPLAY_WIDTH_PIXELS=640
2895 #DISPLAY_HEIGHT_PIXELS=480
2896 #DISPLAY_BPP=16
2897 DISPLAY_DPI=150
2898 DISPLAY_SUBPIXEL_ORDER=vrgb
2899
2900Upgrading Recipes
2901=================
2902
2903Over time, upstream developers publish new versions for software built
2904by layer recipes. It is recommended to keep recipes up-to-date with
2905upstream version releases.
2906
2907While there are several methods to upgrade a recipe, you might
2908consider checking on the upgrade status of a recipe first. You can do so
2909using the ``devtool check-upgrade-status`` command. See the
2910":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2911section in the Yocto Project Reference Manual for more information.
2912
2913The remainder of this section describes three ways you can upgrade a
2914recipe. You can use the Automated Upgrade Helper (AUH) to set up
2915automatic version upgrades. Alternatively, you can use
2916``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
2917you can manually upgrade a recipe by editing the recipe itself.
2918
2919Using the Auto Upgrade Helper (AUH)
2920-----------------------------------
2921
2922The AUH utility works in conjunction with the OpenEmbedded build system
2923in order to automatically generate upgrades for recipes based on new
2924versions being published upstream. Use AUH when you want to create a
2925service that performs the upgrades automatically and optionally sends
2926you an email with the results.
2927
2928AUH allows you to update several recipes with a single use. You can also
2929optionally perform build and integration tests using images with the
2930results saved to your hard drive and emails of results optionally sent
2931to recipe maintainers. Finally, AUH creates Git commits with appropriate
2932commit messages in the layer's tree for the changes made to recipes.
2933
2934.. note::
2935
2936 In some conditions, you should not use AUH to upgrade recipes
2937 and should instead use either ``devtool upgrade`` or upgrade your
2938 recipes manually:
2939
2940 - When AUH cannot complete the upgrade sequence. This situation
2941 usually results because custom patches carried by the recipe
2942 cannot be automatically rebased to the new version. In this case,
2943 ``devtool upgrade`` allows you to manually resolve conflicts.
2944
2945 - When for any reason you want fuller control over the upgrade
2946 process. For example, when you want special arrangements for
2947 testing.
2948
2949The following steps describe how to set up the AUH utility:
2950
29511. *Be Sure the Development Host is Set Up:* You need to be sure that
2952 your development host is set up to use the Yocto Project. For
2953 information on how to set up your host, see the
2954 ":ref:`dev-manual/start:Preparing the Build Host`" section.
2955
29562. *Make Sure Git is Configured:* The AUH utility requires Git to be
2957 configured because AUH uses Git to save upgrades. Thus, you must have
2958 Git user and email configured. The following command shows your
2959 configurations::
2960
2961 $ git config --list
2962
2963 If you do not have the user and
2964 email configured, you can use the following commands to do so::
2965
2966 $ git config --global user.name some_name
2967 $ git config --global user.email username@domain.com
2968
29693. *Clone the AUH Repository:* To use AUH, you must clone the repository
2970 onto your development host. The following command uses Git to create
2971 a local copy of the repository on your system::
2972
2973 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
2974 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
2975 remote: Compressing objects: 100% (300/300), done.
2976 remote: Total 768 (delta 499), reused 703 (delta 434)
2977 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
2978 Resolving deltas: 100% (499/499), done.
2979 Checking connectivity... done.
2980
2981 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
2982 :term:`Poky` repositories.
2983
29844. *Create a Dedicated Build Directory:* Run the
2985 :ref:`structure-core-script`
2986 script to create a fresh build directory that you use exclusively for
2987 running the AUH utility::
2988
2989 $ cd poky
2990 $ source oe-init-build-env your_AUH_build_directory
2991
2992 Re-using an existing build directory and its configurations is not
2993 recommended as existing settings could cause AUH to fail or behave
2994 undesirably.
2995
29965. *Make Configurations in Your Local Configuration File:* Several
2997 settings are needed in the ``local.conf`` file in the build
2998 directory you just created for AUH. Make these following
2999 configurations:
3000
3001 - If you want to enable :ref:`Build
3002 History <dev-manual/common-tasks:maintaining build output quality>`,
3003 which is optional, you need the following lines in the
3004 ``conf/local.conf`` file::
3005
3006 INHERIT =+ "buildhistory"
3007 BUILDHISTORY_COMMIT = "1"
3008
3009 With this configuration and a successful
3010 upgrade, a build history "diff" file appears in the
3011 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3012 your build directory.
3013
3014 - If you want to enable testing through the
3015 :ref:`testimage <ref-classes-testimage*>`
3016 class, which is optional, you need to have the following set in
3017 your ``conf/local.conf`` file::
3018
3019 INHERIT += "testimage"
3020
3021 .. note::
3022
3023 If your distro does not enable by default ptest, which Poky
3024 does, you need the following in your ``local.conf`` file::
3025
3026 DISTRO_FEATURES:append = " ptest"
3027
3028
30296. *Optionally Start a vncserver:* If you are running in a server
3030 without an X11 session, you need to start a vncserver::
3031
3032 $ vncserver :1
3033 $ export DISPLAY=:1
3034
30357. *Create and Edit an AUH Configuration File:* You need to have the
3036 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
3037 build directory. You can find a sample configuration file in the
3038 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
3039
3040 Read through the sample file and make configurations as needed. For
3041 example, if you enabled build history in your ``local.conf`` as
3042 described earlier, you must enable it in ``upgrade-helper.conf``.
3043
3044 Also, if you are using the default ``maintainers.inc`` file supplied
3045 with Poky and located in ``meta-yocto`` and you do not set a
3046 "maintainers_whitelist" or "global_maintainer_override" in the
3047 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3048 the AUH command-line, the utility automatically sends out emails to
3049 all the default maintainers. Please avoid this.
3050
3051This next set of examples describes how to use the AUH:
3052
3053- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
3054 following form::
3055
3056 $ upgrade-helper.py recipe_name
3057
3058 For example, this command upgrades the ``xmodmap`` recipe::
3059
3060 $ upgrade-helper.py xmodmap
3061
3062- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
3063 specific recipe to a particular version, use the following form::
3064
3065 $ upgrade-helper.py recipe_name -t version
3066
3067 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
3068
3069 $ upgrade-helper.py xmodmap -t 1.2.3
3070
3071- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3072 Notifications:* To upgrade all recipes to their most recent versions
3073 and suppress the email notifications, use the following command::
3074
3075 $ upgrade-helper.py all
3076
3077- *Upgrading all Recipes to the Latest Versions and Send Email
3078 Notifications:* To upgrade all recipes to their most recent versions
3079 and send email messages to maintainers for each attempted recipe as
3080 well as a status email, use the following command::
3081
3082 $ upgrade-helper.py -e all
3083
3084Once you have run the AUH utility, you can find the results in the AUH
3085build directory::
3086
3087 ${BUILDDIR}/upgrade-helper/timestamp
3088
3089The AUH utility
3090also creates recipe update commits from successful upgrade attempts in
3091the layer tree.
3092
3093You can easily set up to run the AUH utility on a regular basis by using
3094a cron job. See the
3095:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
3096file distributed with the utility for an example.
3097
3098Using ``devtool upgrade``
3099-------------------------
3100
3101As mentioned earlier, an alternative method for upgrading recipes to
3102newer versions is to use
3103:doc:`devtool upgrade </ref-manual/devtool-reference>`.
3104You can read about ``devtool upgrade`` in general in the
3105":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
3106section in the Yocto Project Application Development and the Extensible
3107Software Development Kit (eSDK) Manual.
3108
3109To see all the command-line options available with ``devtool upgrade``,
3110use the following help command::
3111
3112 $ devtool upgrade -h
3113
3114If you want to find out what version a recipe is currently at upstream
3115without any attempt to upgrade your local version of the recipe, you can
3116use the following command::
3117
3118 $ devtool latest-version recipe_name
3119
3120As mentioned in the previous section describing AUH, ``devtool upgrade``
3121works in a less-automated manner than AUH. Specifically,
3122``devtool upgrade`` only works on a single recipe that you name on the
3123command line, cannot perform build and integration testing using images,
3124and does not automatically generate commits for changes in the source
3125tree. Despite all these "limitations", ``devtool upgrade`` updates the
3126recipe file to the new upstream version and attempts to rebase custom
3127patches contained by the recipe as needed.
3128
3129.. note::
3130
3131 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3132 of a "wrapper" application for ``devtool upgrade``.
3133
3134A typical scenario involves having used Git to clone an upstream
3135repository that you use during build operations. Because you have built the
3136recipe in the past, the layer is likely added to your
3137configuration already. If for some reason, the layer is not added, you
3138could add it easily using the
3139":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3140script. For example, suppose you use the ``nano.bb`` recipe from the
3141``meta-oe`` layer in the ``meta-openembedded`` repository. For this
3142example, assume that the layer has been cloned into following area::
3143
3144 /home/scottrif/meta-openembedded
3145
3146The following command from your
3147:term:`Build Directory` adds the layer to
3148your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
3149
3150 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3151 NOTE: Starting bitbake server...
3152 Parsing recipes: 100% |##########################################| Time: 0:00:55
3153 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3154 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3155 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3156 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3157 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3158
3159For this example, assume that the ``nano.bb`` recipe that
3160is upstream has a 2.9.3 version number. However, the version in the
3161local repository is 2.7.4. The following command from your build
3162directory automatically upgrades the recipe for you:
3163
3164.. note::
3165
3166 Using the ``-V`` option is not necessary. Omitting the version number causes
3167 ``devtool upgrade`` to upgrade the recipe to the most recent version.
3168
3169::
3170
3171 $ devtool upgrade nano -V 2.9.3
3172 NOTE: Starting bitbake server...
3173 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3174 Parsing recipes: 100% |##########################################| Time: 0:00:46
3175 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3176 NOTE: Extracting current version source...
3177 NOTE: Resolving any missing task queue dependencies
3178 .
3179 .
3180 .
3181 NOTE: Executing SetScene Tasks
3182 NOTE: Executing RunQueue Tasks
3183 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3184 Adding changed files: 100% |#####################################| Time: 0:00:00
3185 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3186 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3187
3188Continuing with this example, you can use ``devtool build`` to build the
3189newly upgraded recipe::
3190
3191 $ devtool build nano
3192 NOTE: Starting bitbake server...
3193 Loading cache: 100% |################################################################################################| Time: 0:00:01
3194 Loaded 2040 entries from dependency cache.
3195 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3196 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3197 NOTE: Resolving any missing task queue dependencies
3198 .
3199 .
3200 .
3201 NOTE: Executing SetScene Tasks
3202 NOTE: Executing RunQueue Tasks
3203 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3204 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3205
3206Within the ``devtool upgrade`` workflow, you can
3207deploy and test your rebuilt software. For this example,
3208however, running ``devtool finish`` cleans up the workspace once the
3209source in your workspace is clean. This usually means using Git to stage
3210and submit commits for the changes generated by the upgrade process.
3211
3212Once the tree is clean, you can clean things up in this example with the
3213following command from the ``${BUILDDIR}/workspace/sources/nano``
3214directory::
3215
3216 $ devtool finish nano meta-oe
3217 NOTE: Starting bitbake server...
3218 Loading cache: 100% |################################################################################################| Time: 0:00:00
3219 Loaded 2040 entries from dependency cache.
3220 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3221 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3222 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3223 NOTE: Updating recipe nano_2.9.3.bb
3224 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3225 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3226 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3227
3228
3229Using the ``devtool finish`` command cleans up the workspace and creates a patch
3230file based on your commits. The tool puts all patch files back into the
3231source directory in a sub-directory named ``nano`` in this case.
3232
3233Manually Upgrading a Recipe
3234---------------------------
3235
3236If for some reason you choose not to upgrade recipes using
3237:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or
3238by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``,
3239you can manually edit the recipe files to upgrade the versions.
3240
3241.. note::
3242
3243 Manually updating multiple recipes scales poorly and involves many
3244 steps. The recommendation to upgrade recipe versions is through AUH
3245 or ``devtool upgrade``, both of which automate some steps and provide
3246 guidance for others needed for the manual process.
3247
3248To manually upgrade recipe versions, follow these general steps:
3249
32501. *Change the Version:* Rename the recipe such that the version (i.e.
3251 the :term:`PV` part of the recipe name)
3252 changes appropriately. If the version is not part of the recipe name,
3253 change the value as it is set for :term:`PV` within the recipe itself.
3254
32552. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
3256 is fetched from Git or some other version control system, update
3257 :term:`SRCREV` to point to the
3258 commit hash that matches the new version.
3259
32603. *Build the Software:* Try to build the recipe using BitBake. Typical
3261 build failures include the following:
3262
3263 - License statements were updated for the new version. For this
3264 case, you need to review any changes to the license and update the
3265 values of :term:`LICENSE` and
3266 :term:`LIC_FILES_CHKSUM`
3267 as needed.
3268
3269 .. note::
3270
3271 License changes are often inconsequential. For example, the
3272 license text's copyright year might have changed.
3273
3274 - Custom patches carried by the older version of the recipe might
3275 fail to apply to the new version. For these cases, you need to
3276 review the failures. Patches might not be necessary for the new
3277 version of the software if the upgraded version has fixed those
3278 issues. If a patch is necessary and failing, you need to rebase it
3279 into the new version.
3280
32814. *Optionally Attempt to Build for Several Architectures:* Once you
3282 successfully build the new software for a given architecture, you
3283 could test the build for other architectures by changing the
3284 :term:`MACHINE` variable and
3285 rebuilding the software. This optional step is especially important
3286 if the recipe is to be released publicly.
3287
32885. *Check the Upstream Change Log or Release Notes:* Checking both these
3289 reveals if there are new features that could break
3290 backwards-compatibility. If so, you need to take steps to mitigate or
3291 eliminate that situation.
3292
32936. *Optionally Create a Bootable Image and Test:* If you want, you can
3294 test the new software by booting it onto actual hardware.
3295
32967. *Create a Commit with the Change in the Layer Repository:* After all
3297 builds work and any testing is successful, you can create commits for
3298 any changes in the layer holding your upgraded recipe.
3299
3300Finding Temporary Source Code
3301=============================
3302
3303You might find it helpful during development to modify the temporary
3304source code used by recipes to build packages. For example, suppose you
3305are developing a patch and you need to experiment a bit to figure out
3306your solution. After you have initially built the package, you can
3307iteratively tweak the source code, which is located in the
3308:term:`Build Directory`, and then you can
3309force a re-compile and quickly test your altered code. Once you settle
3310on a solution, you can then preserve your changes in the form of
3311patches.
3312
3313During a build, the unpacked temporary source code used by recipes to
3314build packages is available in the Build Directory as defined by the
3315:term:`S` variable. Below is the default
3316value for the :term:`S` variable as defined in the
3317``meta/conf/bitbake.conf`` configuration file in the
3318:term:`Source Directory`::
3319
3320 S = "${WORKDIR}/${BP}"
3321
3322You should be aware that many recipes override the
3323:term:`S` variable. For example, recipes that fetch their source from Git
3324usually set :term:`S` to ``${WORKDIR}/git``.
3325
3326.. note::
3327
3328 The :term:`BP` represents the base recipe name, which consists of the name
3329 and version::
3330
3331 BP = "${BPN}-${PV}"
3332
3333
3334The path to the work directory for the recipe
3335(:term:`WORKDIR`) is defined as
3336follows::
3337
3338 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3339
3340The actual directory depends on several things:
3341
3342- :term:`TMPDIR`: The top-level build
3343 output directory.
3344
3345- :term:`MULTIMACH_TARGET_SYS`:
3346 The target system identifier.
3347
3348- :term:`PN`: The recipe name.
3349
3350- :term:`EXTENDPE`: The epoch - (if
3351 :term:`PE` is not specified, which is
3352 usually the case for most recipes, then :term:`EXTENDPE` is blank).
3353
3354- :term:`PV`: The recipe version.
3355
3356- :term:`PR`: The recipe revision.
3357
3358As an example, assume a Source Directory top-level folder named
3359``poky``, a default Build Directory at ``poky/build``, and a
3360``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3361recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
3362build system uses to build the package would be as follows::
3363
3364 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3365
3366Using Quilt in Your Workflow
3367============================
3368
3369`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
3370that allows you to capture source code changes without having a clean
3371source tree. This section outlines the typical workflow you can use to
3372modify source code, test changes, and then preserve the changes in the
3373form of a patch all using Quilt.
3374
3375.. note::
3376
3377 With regard to preserving changes to source files, if you clean a
3378 recipe or have ``rm_work`` enabled, the
3379 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
3380 as described in the Yocto Project Application Development and the
3381 Extensible Software Development Kit (eSDK) manual is a safer
3382 development flow than the flow that uses Quilt.
3383
3384Follow these general steps:
3385
33861. *Find the Source Code:* Temporary source code used by the
3387 OpenEmbedded build system is kept in the
3388 :term:`Build Directory`. See the
3389 ":ref:`dev-manual/common-tasks:finding temporary source code`" section to
3390 learn how to locate the directory that has the temporary source code for a
3391 particular package.
3392
33932. *Change Your Working Directory:* You need to be in the directory that
3394 has the temporary source code. That directory is defined by the
3395 :term:`S` variable.
3396
33973. *Create a New Patch:* Before modifying source code, you need to
3398 create a new patch. To create a new patch file, use ``quilt new`` as
3399 below::
3400
3401 $ quilt new my_changes.patch
3402
34034. *Notify Quilt and Add Files:* After creating the patch, you need to
3404 notify Quilt about the files you plan to edit. You notify Quilt by
3405 adding the files to the patch you just created::
3406
3407 $ quilt add file1.c file2.c file3.c
3408
34095. *Edit the Files:* Make your changes in the source code to the files
3410 you added to the patch.
3411
34126. *Test Your Changes:* Once you have modified the source code, the
3413 easiest way to test your changes is by calling the ``do_compile``
3414 task as shown in the following example::
3415
3416 $ bitbake -c compile -f package
3417
3418 The ``-f`` or ``--force`` option forces the specified task to
3419 execute. If you find problems with your code, you can just keep
3420 editing and re-testing iteratively until things work as expected.
3421
3422 .. note::
3423
3424 All the modifications you make to the temporary source code disappear
3425 once you run the ``do_clean`` or ``do_cleanall`` tasks using BitBake
3426 (i.e. ``bitbake -c clean package`` and ``bitbake -c cleanall package``).
3427 Modifications will also disappear if you use the ``rm_work`` feature as
3428 described in the
3429 ":ref:`dev-manual/common-tasks:conserving disk space during builds`"
3430 section.
3431
34327. *Generate the Patch:* Once your changes work as expected, you need to
3433 use Quilt to generate the final patch that contains all your
3434 modifications.
3435 ::
3436
3437 $ quilt refresh
3438
3439 At this point, the
3440 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3441 ``file2.c``, and ``file3.c`` files.
3442
3443 You can find the resulting patch file in the ``patches/``
3444 subdirectory of the source (:term:`S`) directory.
3445
34468. *Copy the Patch File:* For simplicity, copy the patch file into a
3447 directory named ``files``, which you can create in the same directory
3448 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3449 file. Placing the patch here guarantees that the OpenEmbedded build
3450 system will find the patch. Next, add the patch into the :term:`SRC_URI`
3451 of the recipe. Here is an example::
3452
3453 SRC_URI += "file://my_changes.patch"
3454
3455Using a Development Shell
3456=========================
3457
3458When debugging certain commands or even when just editing packages,
3459``devshell`` can be a useful tool. When you invoke ``devshell``, all
3460tasks up to and including
3461:ref:`ref-tasks-patch` are run for the
3462specified target. Then, a new terminal is opened and you are placed in
3463``${``\ :term:`S`\ ``}``, the source
3464directory. In the new terminal, all the OpenEmbedded build-related
3465environment variables are still defined so you can use commands such as
3466``configure`` and ``make``. The commands execute just as if the
3467OpenEmbedded build system were executing them. Consequently, working
3468this way can be helpful when debugging a build or preparing software to
3469be used with the OpenEmbedded build system.
3470
3471Following is an example that uses ``devshell`` on a target named
3472``matchbox-desktop``::
3473
3474 $ bitbake matchbox-desktop -c devshell
3475
3476This command spawns a terminal with a shell prompt within the
3477OpenEmbedded build environment. The
3478:term:`OE_TERMINAL` variable
3479controls what type of shell is opened.
3480
3481For spawned terminals, the following occurs:
3482
3483- The ``PATH`` variable includes the cross-toolchain.
3484
3485- The ``pkgconfig`` variables find the correct ``.pc`` files.
3486
3487- The ``configure`` command finds the Yocto Project site files as well
3488 as any other necessary files.
3489
3490Within this environment, you can run configure or compile commands as if
3491they were being run by the OpenEmbedded build system itself. As noted
3492earlier, the working directory also automatically changes to the Source
3493Directory (:term:`S`).
3494
3495To manually run a specific task using ``devshell``, run the
3496corresponding ``run.*`` script in the
3497``${``\ :term:`WORKDIR`\ ``}/temp``
3498directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
3499not exist, which would be the case if the task was skipped by way of the
3500sstate cache, you can create the task by first running it outside of the
3501``devshell``::
3502
3503 $ bitbake -c task
3504
3505.. note::
3506
3507 - Execution of a task's ``run.*`` script and BitBake's execution of
3508 a task are identical. In other words, running the script re-runs
3509 the task just as it would be run using the ``bitbake -c`` command.
3510
3511 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3512 symbolic link (symlink) to the most recent version of that file.
3513
3514Remember, that the ``devshell`` is a mechanism that allows you to get
3515into the BitBake task execution environment. And as such, all commands
3516must be called just as BitBake would call them. That means you need to
3517provide the appropriate options for cross-compilation and so forth as
3518applicable.
3519
3520When you are finished using ``devshell``, exit the shell or close the
3521terminal window.
3522
3523.. note::
3524
3525 - It is worth remembering that when using ``devshell`` you need to
3526 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3527 instead of just using ``gcc``. The same applies to other
3528 applications such as ``binutils``, ``libtool`` and so forth.
3529 BitBake sets up environment variables such as :term:`CC` to assist
3530 applications, such as ``make`` to find the correct tools.
3531
3532 - It is also worth noting that ``devshell`` still works over X11
3533 forwarding and similar situations.
3534
3535Using a Python Development Shell
3536================================
3537
3538Similar to working within a development shell as described in the
3539previous section, you can also spawn and work within an interactive
3540Python development shell. When debugging certain commands or even when
3541just editing packages, ``pydevshell`` can be a useful tool. When you
3542invoke the ``pydevshell`` task, all tasks up to and including
3543:ref:`ref-tasks-patch` are run for the
3544specified target. Then a new terminal is opened. Additionally, key
3545Python objects and code are available in the same way they are to
3546BitBake tasks, in particular, the data store 'd'. So, commands such as
3547the following are useful when exploring the data store and running
3548functions::
3549
3550 pydevshell> d.getVar("STAGING_DIR")
3551 '/media/build1/poky/build/tmp/sysroots'
3552 pydevshell> d.getVar("STAGING_DIR", False)
3553 '${TMPDIR}/sysroots'
3554 pydevshell> d.setVar("FOO", "bar")
3555 pydevshell> d.getVar("FOO")
3556 'bar'
3557 pydevshell> d.delVar("FOO")
3558 pydevshell> d.getVar("FOO")
3559 pydevshell> bb.build.exec_func("do_unpack", d)
3560 pydevshell>
3561
3562The commands execute just as if the OpenEmbedded build
3563system were executing them. Consequently, working this way can be
3564helpful when debugging a build or preparing software to be used with the
3565OpenEmbedded build system.
3566
3567Following is an example that uses ``pydevshell`` on a target named
3568``matchbox-desktop``::
3569
3570 $ bitbake matchbox-desktop -c pydevshell
3571
3572This command spawns a terminal and places you in an interactive Python
3573interpreter within the OpenEmbedded build environment. The
3574:term:`OE_TERMINAL` variable
3575controls what type of shell is opened.
3576
3577When you are finished using ``pydevshell``, you can exit the shell
3578either by using Ctrl+d or closing the terminal window.
3579
3580Building
3581========
3582
3583This section describes various build procedures, such as the steps
3584needed for a simple build, building a target for multiple configurations,
3585generating an image for more than one machine, and so forth.
3586
3587Building a Simple Image
3588-----------------------
3589
3590In the development environment, you need to build an image whenever you
3591change hardware support, add or change system libraries, or add or
3592change services that have dependencies. There are several methods that allow
3593you to build an image within the Yocto Project. This section presents
3594the basic steps you need to build a simple image using BitBake from a
3595build host running Linux.
3596
3597.. note::
3598
3599 - For information on how to build an image using
3600 :term:`Toaster`, see the
3601 :doc:`/toaster-manual/index`.
3602
3603 - For information on how to use ``devtool`` to build images, see the
3604 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
3605 section in the Yocto Project Application Development and the
3606 Extensible Software Development Kit (eSDK) manual.
3607
3608 - For a quick example on how to build an image using the
3609 OpenEmbedded build system, see the
3610 :doc:`/brief-yoctoprojectqs/index` document.
3611
3612The build process creates an entire Linux distribution from source and
3613places it in your :term:`Build Directory` under
3614``tmp/deploy/images``. For detailed information on the build process
3615using BitBake, see the ":ref:`overview-manual/concepts:images`" section in the
3616Yocto Project Overview and Concepts Manual.
3617
3618The following figure and list overviews the build process:
3619
3620.. image:: figures/bitbake-build-flow.png
3621 :align: center
3622
36231. *Set up Your Host Development System to Support Development Using the
3624 Yocto Project*: See the ":doc:`start`" section for options on how to get a
3625 build host ready to use the Yocto Project.
3626
36272. *Initialize the Build Environment:* Initialize the build environment
3628 by sourcing the build environment script (i.e.
3629 :ref:`structure-core-script`)::
3630
3631 $ source oe-init-build-env [build_dir]
3632
3633 When you use the initialization script, the OpenEmbedded build system
3634 uses ``build`` as the default :term:`Build Directory` in your current work
3635 directory. You can use a `build_dir` argument with the script to
3636 specify a different build directory.
3637
3638 .. note::
3639
3640 A common practice is to use a different Build Directory for
3641 different targets; for example, ``~/build/x86`` for a ``qemux86``
3642 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
3643 event, it's typically cleaner to locate the build directory
3644 somewhere outside of your source directory.
3645
36463. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
3647 ``conf/local.conf`` configuration file, which is found in the Build
3648 Directory, is set up how you want it. This file defines many aspects
3649 of the build environment including the target machine architecture
3650 through the :term:`MACHINE` variable, the packaging format used during
3651 the build
3652 (:term:`PACKAGE_CLASSES`),
3653 and a centralized tarball download directory through the
3654 :term:`DL_DIR` variable.
3655
36564. *Build the Image:* Build the image using the ``bitbake`` command::
3657
3658 $ bitbake target
3659
3660 .. note::
3661
3662 For information on BitBake, see the :doc:`bitbake:index`.
3663
3664 The target is the name of the recipe you want to build. Common
3665 targets are the images in ``meta/recipes-core/images``,
3666 ``meta/recipes-sato/images``, and so forth all found in the
3667 :term:`Source Directory`. Alternatively, the target
3668 can be the name of a recipe for a specific piece of software such as
3669 BusyBox. For more details about the images the OpenEmbedded build
3670 system supports, see the
3671 ":ref:`ref-manual/images:Images`" chapter in the Yocto
3672 Project Reference Manual.
3673
3674 As an example, the following command builds the
3675 ``core-image-minimal`` image::
3676
3677 $ bitbake core-image-minimal
3678
3679 Once an
3680 image has been built, it often needs to be installed. The images and
3681 kernels built by the OpenEmbedded build system are placed in the
3682 Build Directory in ``tmp/deploy/images``. For information on how to
3683 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
3684 :doc:`/sdk-manual/index` manual. For
3685 information about how to install these images, see the documentation
3686 for your particular board or machine.
3687
3688Building Images for Multiple Targets Using Multiple Configurations
3689------------------------------------------------------------------
3690
3691You can use a single ``bitbake`` command to build multiple images or
3692packages for different targets where each image or package requires a
3693different configuration (multiple configuration builds). The builds, in
3694this scenario, are sometimes referred to as "multiconfigs", and this
3695section uses that term throughout.
3696
3697This section describes how to set up for multiple configuration builds
3698and how to account for cross-build dependencies between the
3699multiconfigs.
3700
3701Setting Up and Running a Multiple Configuration Build
3702~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3703
3704To accomplish a multiple configuration build, you must define each
3705target's configuration separately using a parallel configuration file in
3706the :term:`Build Directory`, and you
3707must follow a required file hierarchy. Additionally, you must enable the
3708multiple configuration builds in your ``local.conf`` file.
3709
3710Follow these steps to set up and execute multiple configuration builds:
3711
3712- *Create Separate Configuration Files*: You need to create a single
3713 configuration file for each build target (each multiconfig).
3714 Minimally, each configuration file must define the machine and the
3715 temporary directory BitBake uses for the build. Suggested practice
3716 dictates that you do not overlap the temporary directories used
3717 during the builds. However, it is possible that you can share the
3718 temporary directory
3719 (:term:`TMPDIR`). For example,
3720 consider a scenario with two different multiconfigs for the same
3721 :term:`MACHINE`: "qemux86" built
3722 for two distributions such as "poky" and "poky-lsb". In this case,
3723 you might want to use the same :term:`TMPDIR`.
3724
3725 Here is an example showing the minimal statements needed in a
3726 configuration file for a "qemux86" target whose temporary build
3727 directory is ``tmpmultix86``::
3728
3729 MACHINE = "qemux86"
3730 TMPDIR = "${TOPDIR}/tmpmultix86"
3731
3732 The location for these multiconfig configuration files is specific.
3733 They must reside in the current build directory in a sub-directory of
3734 ``conf`` named ``multiconfig``. Following is an example that defines
3735 two configuration files for the "x86" and "arm" multiconfigs:
3736
3737 .. image:: figures/multiconfig_files.png
3738 :align: center
3739
3740 The reason for this required file hierarchy is because the :term:`BBPATH`
3741 variable is not constructed until the layers are parsed.
3742 Consequently, using the configuration file as a pre-configuration
3743 file is not possible unless it is located in the current working
3744 directory.
3745
3746- *Add the BitBake Multi-configuration Variable to the Local
3747 Configuration File*: Use the
3748 :term:`BBMULTICONFIG`
3749 variable in your ``conf/local.conf`` configuration file to specify
3750 each multiconfig. Continuing with the example from the previous
3751 figure, the :term:`BBMULTICONFIG` variable needs to enable two
3752 multiconfigs: "x86" and "arm" by specifying each configuration file::
3753
3754 BBMULTICONFIG = "x86 arm"
3755
3756 .. note::
3757
3758 A "default" configuration already exists by definition. This
3759 configuration is named: "" (i.e. empty string) and is defined by
3760 the variables coming from your ``local.conf``
3761 file. Consequently, the previous example actually adds two
3762 additional configurations to your build: "arm" and "x86" along
3763 with "".
3764
3765- *Launch BitBake*: Use the following BitBake command form to launch
3766 the multiple configuration build::
3767
3768 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3769
3770 For the example in this section, the following command applies::
3771
3772 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3773
3774 The previous BitBake command builds a ``core-image-minimal`` image
3775 that is configured through the ``x86.conf`` configuration file, a
3776 ``core-image-sato`` image that is configured through the ``arm.conf``
3777 configuration file and a ``core-image-base`` that is configured
3778 through your ``local.conf`` configuration file.
3779
3780.. note::
3781
3782 Support for multiple configuration builds in the Yocto Project &DISTRO;
3783 (&DISTRO_NAME;) Release does not include Shared State (sstate)
3784 optimizations. Consequently, if a build uses the same object twice
3785 in, for example, two different :term:`TMPDIR`
3786 directories, the build either loads from an existing sstate cache for
3787 that build at the start or builds the object fresh.
3788
3789Enabling Multiple Configuration Build Dependencies
3790~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3791
3792Sometimes dependencies can exist between targets (multiconfigs) in a
3793multiple configuration build. For example, suppose that in order to
3794build a ``core-image-sato`` image for an "x86" multiconfig, the root
3795filesystem of an "arm" multiconfig must exist. This dependency is
3796essentially that the
3797:ref:`ref-tasks-image` task in the
3798``core-image-sato`` recipe depends on the completion of the
3799:ref:`ref-tasks-rootfs` task of the
3800``core-image-minimal`` recipe.
3801
3802To enable dependencies in a multiple configuration build, you must
3803declare the dependencies in the recipe using the following statement
3804form::
3805
3806 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3807
3808To better show how to use this statement, consider the example scenario
3809from the first paragraph of this section. The following statement needs
3810to be added to the recipe that builds the ``core-image-sato`` image::
3811
3812 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3813
3814In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
3815task on which the ``do_image`` task in the recipe depends is the
3816``do_rootfs`` task from the ``core-image-minimal`` recipe associated
3817with the "arm" multiconfig.
3818
3819Once you set up this dependency, you can build the "x86" multiconfig
3820using a BitBake command as follows::
3821
3822 $ bitbake mc:x86:core-image-sato
3823
3824This command executes all the tasks needed to create the
3825``core-image-sato`` image for the "x86" multiconfig. Because of the
3826dependency, BitBake also executes through the ``do_rootfs`` task for the
3827"arm" multiconfig build.
3828
3829Having a recipe depend on the root filesystem of another build might not
3830seem that useful. Consider this change to the statement in the
3831``core-image-sato`` recipe::
3832
3833 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3834
3835In this case, BitBake must
3836create the ``core-image-minimal`` image for the "arm" build since the
3837"x86" build depends on it.
3838
3839Because "x86" and "arm" are enabled for multiple configuration builds
3840and have separate configuration files, BitBake places the artifacts for
3841each build in the respective temporary build directories (i.e.
3842:term:`TMPDIR`).
3843
3844Building an Initial RAM Filesystem (initramfs) Image
3845----------------------------------------------------
3846
3847An initial RAM filesystem (initramfs) image provides a temporary root
3848filesystem used for early system initialization (e.g. loading of modules
3849needed to locate and mount the "real" root filesystem).
3850
3851.. note::
3852
3853 The initramfs image is the successor of initial RAM disk (initrd). It
3854 is a "copy in and out" (cpio) archive of the initial filesystem that
3855 gets loaded into memory during the Linux startup process. Because
3856 Linux uses the contents of the archive during initialization, the
3857 initramfs image needs to contain all of the device drivers and tools
3858 needed to mount the final root filesystem.
3859
3860Follow these steps to create an initramfs image:
3861
38621. *Create the initramfs Image Recipe:* You can reference the
3863 ``core-image-minimal-initramfs.bb`` recipe found in the
3864 ``meta/recipes-core`` directory of the :term:`Source Directory`
3865 as an example
3866 from which to work.
3867
38682. *Decide if You Need to Bundle the initramfs Image Into the Kernel
3869 Image:* If you want the initramfs image that is built to be bundled
3870 in with the kernel image, set the
3871 :term:`INITRAMFS_IMAGE_BUNDLE`
3872 variable to "1" in your ``local.conf`` configuration file and set the
3873 :term:`INITRAMFS_IMAGE`
3874 variable in the recipe that builds the kernel image.
3875
3876 .. note::
3877
3878 It is recommended that you bundle the initramfs image with the
3879 kernel image to avoid circular dependencies between the kernel
3880 recipe and the initramfs recipe should the initramfs image include
3881 kernel modules.
3882
3883 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the initramfs
3884 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
3885 initramfs image is then passed to the kernel's ``Makefile`` using the
3886 :term:`CONFIG_INITRAMFS_SOURCE`
3887 variable, allowing the initramfs image to be built into the kernel
3888 normally.
3889
3890 .. note::
3891
3892 Bundling the initramfs with the kernel conflates the code in the initramfs
3893 with the GPLv2 licensed Linux kernel binary. Thus only GPLv2 compatible
3894 software may be part of a bundled initramfs.
3895
3896 .. note::
3897
3898 If you choose to not bundle the initramfs image with the kernel
3899 image, you are essentially using an
3900 `Initial RAM Disk (initrd) <https://en.wikipedia.org/wiki/Initrd>`__.
3901 Creating an initrd is handled primarily through the :term:`INITRD_IMAGE`,
3902 ``INITRD_LIVE``, and ``INITRD_IMAGE_LIVE`` variables. For more
3903 information, see the :ref:`ref-classes-image-live` file.
3904
39053. *Optionally Add Items to the initramfs Image Through the initramfs
3906 Image Recipe:* If you add items to the initramfs image by way of its
3907 recipe, you should use
3908 :term:`PACKAGE_INSTALL`
3909 rather than
3910 :term:`IMAGE_INSTALL`.
3911 :term:`PACKAGE_INSTALL` gives more direct control of what is added to the
3912 image as compared to the defaults you might not necessarily want that
3913 are set by the :ref:`image <ref-classes-image>`
3914 or :ref:`core-image <ref-classes-core-image>`
3915 classes.
3916
39174. *Build the Kernel Image and the initramfs Image:* Build your kernel
3918 image using BitBake. Because the initramfs image recipe is a
3919 dependency of the kernel image, the initramfs image is built as well
3920 and bundled with the kernel image if you used the
3921 :term:`INITRAMFS_IMAGE_BUNDLE`
3922 variable described earlier.
3923
3924Bundling an Initramfs Image From a Separate Multiconfig
3925~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3926
3927There may be a case where we want to build an initramfs image which does not
3928inherit the same distro policy as our main image, for example, we may want
3929our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our initramfs
3930image to keep a smaller footprint. However, by performing the steps mentioned
3931above the initramfs image will inherit ``TCLIBC="glibc"`` without allowing us
3932to override it.
3933
3934To achieve this, you need to perform some additional steps:
3935
39361. *Create a multiconfig for your initramfs image:* You can perform the steps
3937 on ":ref:`dev-manual/common-tasks:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
3938 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
3939 contains the variables::
3940
3941 TMPDIR="${TOPDIR}/tmp-initramfscfg"
3942 TCLIBC="musl"
3943
39442. *Set additional initramfs variables on your main configuration:*
3945 Additionally, on your main configuration (``local.conf``) you need to set the
3946 variables::
3947
3948 INITRAMFS_MULTICONFIG = "initramfscfg"
3949 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
3950
3951 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
3952 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
3953 to be built coming from the ``initramfscfg`` multiconfig, and to let the
3954 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
3955
3956 Building a system with such configuration will build the kernel using the
3957 main configuration but the ``do_bundle_initramfs`` task will grab the
3958 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
3959 instead, resulting in a musl based initramfs image bundled in the kernel
3960 but a glibc based main image.
3961
3962 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
3963 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
3964
3965
3966Building a Tiny System
3967----------------------
3968
3969Very small distributions have some significant advantages such as
3970requiring less on-die or in-package memory (cheaper), better performance
3971through efficient cache usage, lower power requirements due to less
3972memory, faster boot times, and reduced development overhead. Some
3973real-world examples where a very small distribution gives you distinct
3974advantages are digital cameras, medical devices, and small headless
3975systems.
3976
3977This section presents information that shows you how you can trim your
3978distribution to even smaller sizes than the ``poky-tiny`` distribution,
3979which is around 5 Mbytes, that can be built out-of-the-box using the
3980Yocto Project.
3981
3982Tiny System Overview
3983~~~~~~~~~~~~~~~~~~~~
3984
3985The following list presents the overall steps you need to consider and
3986perform to create distributions with smaller root filesystems, achieve
3987faster boot times, maintain your critical functionality, and avoid
3988initial RAM disks:
3989
3990- :ref:`Determine your goals and guiding principles
3991 <dev-manual/common-tasks:goals and guiding principles>`
3992
3993- :ref:`dev-manual/common-tasks:understand what contributes to your image size`
3994
3995- :ref:`Reduce the size of the root filesystem
3996 <dev-manual/common-tasks:trim the root filesystem>`
3997
3998- :ref:`Reduce the size of the kernel <dev-manual/common-tasks:trim the kernel>`
3999
4000- :ref:`dev-manual/common-tasks:remove package management requirements`
4001
4002- :ref:`dev-manual/common-tasks:look for other ways to minimize size`
4003
4004- :ref:`dev-manual/common-tasks:iterate on the process`
4005
4006Goals and Guiding Principles
4007~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4008
4009Before you can reach your destination, you need to know where you are
4010going. Here is an example list that you can use as a guide when creating
4011very small distributions:
4012
4013- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
4014 less and a root filesystem that is 3 Mbytes or less).
4015
4016- Find the areas that are currently taking 90% of the space and
4017 concentrate on reducing those areas.
4018
4019- Do not create any difficult "hacks" to achieve your goals.
4020
4021- Leverage the device-specific options.
4022
4023- Work in a separate layer so that you keep changes isolated. For
4024 information on how to create layers, see the
4025 ":ref:`dev-manual/common-tasks:understanding and creating layers`" section.
4026
4027Understand What Contributes to Your Image Size
4028~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4029
4030It is easiest to have something to start with when creating your own
4031distribution. You can use the Yocto Project out-of-the-box to create the
4032``poky-tiny`` distribution. Ultimately, you will want to make changes in
4033your own distribution that are likely modeled after ``poky-tiny``.
4034
4035.. note::
4036
4037 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
4038 ``local.conf`` file to "poky-tiny" as described in the
4039 ":ref:`dev-manual/common-tasks:creating your own distribution`"
4040 section.
4041
4042Understanding some memory concepts will help you reduce the system size.
4043Memory consists of static, dynamic, and temporary memory. Static memory
4044is the TEXT (code), DATA (initialized data in the code), and BSS
4045(uninitialized data) sections. Dynamic memory represents memory that is
4046allocated at runtime: stacks, hash tables, and so forth. Temporary
4047memory is recovered after the boot process. This memory consists of
4048memory used for decompressing the kernel and for the ``__init__``
4049functions.
4050
4051To help you see where you currently are with kernel and root filesystem
4052sizes, you can use two tools found in the :term:`Source Directory`
4053in the
4054``scripts/tiny/`` directory:
4055
4056- ``ksize.py``: Reports component sizes for the kernel build objects.
4057
4058- ``dirsize.py``: Reports component sizes for the root filesystem.
4059
4060This next tool and command help you organize configuration fragments and
4061view file dependencies in a human-readable form:
4062
4063- ``merge_config.sh``: Helps you manage configuration files and
4064 fragments within the kernel. With this tool, you can merge individual
4065 configuration fragments together. The tool allows you to make
4066 overrides and warns you of any missing configuration options. The
4067 tool is ideal for allowing you to iterate on configurations, create
4068 minimal configurations, and create configuration files for different
4069 machines without having to duplicate your process.
4070
4071 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4072 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4073 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4074 directory.
4075
4076 For more information on configuration fragments, see the
4077 ":ref:`kernel-dev/common:creating configuration fragments`"
4078 section in the Yocto Project Linux Kernel Development Manual.
4079
4080- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4081 with these options brings up a Dependency Explorer from which you can
4082 view file dependencies. Understanding these dependencies allows you
4083 to make informed decisions when cutting out various pieces of the
4084 kernel and root filesystem.
4085
4086Trim the Root Filesystem
4087~~~~~~~~~~~~~~~~~~~~~~~~
4088
4089The root filesystem is made up of packages for booting, libraries, and
4090applications. To change things, you can configure how the packaging
4091happens, which changes the way you build them. You can also modify the
4092filesystem itself or select a different filesystem.
4093
4094First, find out what is hogging your root filesystem by running the
4095``dirsize.py`` script from your root directory::
4096
4097 $ cd root-directory-of-image
4098 $ dirsize.py 100000 > dirsize-100k.log
4099 $ cat dirsize-100k.log
4100
4101You can apply a filter to the script to ignore files
4102under a certain size. The previous example filters out any files below
4103100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4104will be smaller by a relatively constant factor in a compressed root
4105filesystem. When you examine your log file, you can focus on areas of
4106the root filesystem that take up large amounts of memory.
4107
4108You need to be sure that what you eliminate does not cripple the
4109functionality you need. One way to see how packages relate to each other
4110is by using the Dependency Explorer UI with the BitBake command::
4111
4112 $ cd image-directory
4113 $ bitbake -u taskexp -g image
4114
4115Use the interface to
4116select potential packages you wish to eliminate and see their dependency
4117relationships.
4118
4119When deciding how to reduce the size, get rid of packages that result in
4120minimal impact on the feature set. For example, you might not need a VGA
4121display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4122instead of ``udev``.
4123
4124Use your ``local.conf`` file to make changes. For example, to eliminate
4125``udev`` and ``glib``, set the following in the local configuration
4126file::
4127
4128 VIRTUAL-RUNTIME_dev_manager = ""
4129
4130Finally, you should consider exactly the type of root filesystem you
4131need to meet your needs while also reducing its size. For example,
4132consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4133``initramfs`` using ``initramfs``. Be aware that ``ext3`` requires a 1
4134Mbyte journal. If you are okay with running read-only, you do not need
4135this journal.
4136
4137.. note::
4138
4139 After each round of elimination, you need to rebuild your system and
4140 then use the tools to see the effects of your reductions.
4141
4142Trim the Kernel
4143~~~~~~~~~~~~~~~
4144
4145The kernel is built by including policies for hardware-independent
4146aspects. What subsystems do you enable? For what architecture are you
4147building? Which drivers do you build by default?
4148
4149.. note::
4150
4151 You can modify the kernel source if you want to help with boot time.
4152
4153Run the ``ksize.py`` script from the top-level Linux build directory to
4154get an idea of what is making up the kernel::
4155
4156 $ cd top-level-linux-build-directory
4157 $ ksize.py > ksize.log
4158 $ cat ksize.log
4159
4160When you examine the log, you will see how much space is taken up with
4161the built-in ``.o`` files for drivers, networking, core kernel files,
4162filesystem, sound, and so forth. The sizes reported by the tool are
4163uncompressed, and thus will be smaller by a relatively constant factor
4164in a compressed kernel image. Look to reduce the areas that are large
4165and taking up around the "90% rule."
4166
4167To examine, or drill down, into any particular area, use the ``-d``
4168option with the script::
4169
4170 $ ksize.py -d > ksize.log
4171
4172Using this option
4173breaks out the individual file information for each area of the kernel
4174(e.g. drivers, networking, and so forth).
4175
4176Use your log file to see what you can eliminate from the kernel based on
4177features you can let go. For example, if you are not going to need
4178sound, you do not need any drivers that support sound.
4179
4180After figuring out what to eliminate, you need to reconfigure the kernel
4181to reflect those changes during the next build. You could run
4182``menuconfig`` and make all your changes at once. However, that makes it
4183difficult to see the effects of your individual eliminations and also
4184makes it difficult to replicate the changes for perhaps another target
4185device. A better method is to start with no configurations using
4186``allnoconfig``, create configuration fragments for individual changes,
4187and then manage the fragments into a single configuration file using
4188``merge_config.sh``. The tool makes it easy for you to iterate using the
4189configuration change and build cycle.
4190
4191Each time you make configuration changes, you need to rebuild the kernel
4192and check to see what impact your changes had on the overall size.
4193
4194Remove Package Management Requirements
4195~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4196
4197Packaging requirements add size to the image. One way to reduce the size
4198of the image is to remove all the packaging requirements from the image.
4199This reduction includes both removing the package manager and its unique
4200dependencies as well as removing the package management data itself.
4201
4202To eliminate all the packaging requirements for an image, be sure that
4203"package-management" is not part of your
4204:term:`IMAGE_FEATURES`
4205statement for the image. When you remove this feature, you are removing
4206the package manager as well as its dependencies from the root
4207filesystem.
4208
4209Look for Other Ways to Minimize Size
4210~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4211
4212Depending on your particular circumstances, other areas that you can
4213trim likely exist. The key to finding these areas is through tools and
4214methods described here combined with experimentation and iteration. Here
4215are a couple of areas to experiment with:
4216
4217- ``glibc``: In general, follow this process:
4218
4219 1. Remove ``glibc`` features from
4220 :term:`DISTRO_FEATURES`
4221 that you think you do not need.
4222
4223 2. Build your distribution.
4224
4225 3. If the build fails due to missing symbols in a package, determine
4226 if you can reconfigure the package to not need those features. For
4227 example, change the configuration to not support wide character
4228 support as is done for ``ncurses``. Or, if support for those
4229 characters is needed, determine what ``glibc`` features provide
4230 the support and restore the configuration.
4231
4232 4. Rebuild and repeat the process.
4233
4234- ``busybox``: For BusyBox, use a process similar as described for
4235 ``glibc``. A difference is you will need to boot the resulting system
4236 to see if you are able to do everything you expect from the running
4237 system. You need to be sure to integrate configuration fragments into
4238 Busybox because BusyBox handles its own core features and then allows
4239 you to add configuration fragments on top.
4240
4241Iterate on the Process
4242~~~~~~~~~~~~~~~~~~~~~~
4243
4244If you have not reached your goals on system size, you need to iterate
4245on the process. The process is the same. Use the tools and see just what
4246is taking up 90% of the root filesystem and the kernel. Decide what you
4247can eliminate without limiting your device beyond what you need.
4248
4249Depending on your system, a good place to look might be Busybox, which
4250provides a stripped down version of Unix tools in a single, executable
4251file. You might be able to drop virtual terminal services or perhaps
4252ipv6.
4253
4254Building Images for More than One Machine
4255-----------------------------------------
4256
4257A common scenario developers face is creating images for several
4258different machines that use the same software environment. In this
4259situation, it is tempting to set the tunings and optimization flags for
4260each build specifically for the targeted hardware (i.e. "maxing out" the
4261tunings). Doing so can considerably add to build times and package feed
4262maintenance collectively for the machines. For example, selecting tunes
4263that are extremely specific to a CPU core used in a system might enable
4264some micro optimizations in GCC for that particular system but would
4265otherwise not gain you much of a performance difference across the other
4266systems as compared to using a more general tuning across all the builds
4267(e.g. setting :term:`DEFAULTTUNE`
4268specifically for each machine's build). Rather than "max out" each
4269build's tunings, you can take steps that cause the OpenEmbedded build
4270system to reuse software across the various machines where it makes
4271sense.
4272
4273If build speed and package feed maintenance are considerations, you
4274should consider the points in this section that can help you optimize
4275your tunings to best consider build times and package feed maintenance.
4276
4277- *Share the Build Directory:* If at all possible, share the
4278 :term:`TMPDIR` across builds. The
4279 Yocto Project supports switching between different
4280 :term:`MACHINE` values in the same
4281 :term:`TMPDIR`. This practice is well supported and regularly used by
4282 developers when building for multiple machines. When you use the same
4283 :term:`TMPDIR` for multiple machine builds, the OpenEmbedded build system
4284 can reuse the existing native and often cross-recipes for multiple
4285 machines. Thus, build time decreases.
4286
4287 .. note::
4288
4289 If :term:`DISTRO` settings change or fundamental configuration settings
4290 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
4291 Sharing :term:`TMPDIR` under these circumstances might work but since it is
4292 not guaranteed, you should use a clean :term:`TMPDIR`.
4293
4294- *Enable the Appropriate Package Architecture:* By default, the
4295 OpenEmbedded build system enables three levels of package
4296 architectures: "all", "tune" or "package", and "machine". Any given
4297 recipe usually selects one of these package architectures (types) for
4298 its output. Depending for what a given recipe creates packages,
4299 making sure you enable the appropriate package architecture can
4300 directly impact the build time.
4301
4302 A recipe that just generates scripts can enable "all" architecture
4303 because there are no binaries to build. To specifically enable "all"
4304 architecture, be sure your recipe inherits the
4305 :ref:`allarch <ref-classes-allarch>` class.
4306 This class is useful for "all" architectures because it configures
4307 many variables so packages can be used across multiple architectures.
4308
4309 If your recipe needs to generate packages that are machine-specific
4310 or when one of the build or runtime dependencies is already
4311 machine-architecture dependent, which makes your recipe also
4312 machine-architecture dependent, make sure your recipe enables the
4313 "machine" package architecture through the
4314 :term:`MACHINE_ARCH`
4315 variable::
4316
4317 PACKAGE_ARCH = "${MACHINE_ARCH}"
4318
4319 When you do not
4320 specifically enable a package architecture through the
4321 :term:`PACKAGE_ARCH`, The
4322 OpenEmbedded build system defaults to the
4323 :term:`TUNE_PKGARCH` setting::
4324
4325 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4326
4327- *Choose a Generic Tuning File if Possible:* Some tunes are more
4328 generic and can run on multiple targets (e.g. an ``armv5`` set of
4329 packages could run on ``armv6`` and ``armv7`` processors in most
4330 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4331 higher processors. You should realize, however, that advances on
4332 newer processor versions would not be used.
4333
4334 If you select the same tune for several different machines, the
4335 OpenEmbedded build system reuses software previously built, thus
4336 speeding up the overall build time. Realize that even though a new
4337 sysroot for each machine is generated, the software is not recompiled
4338 and only one package feed exists.
4339
4340- *Manage Granular Level Packaging:* Sometimes there are cases where
4341 injecting another level of package architecture beyond the three
4342 higher levels noted earlier can be useful. For example, consider how
4343 NXP (formerly Freescale) allows for the easy reuse of binary packages
4344 in their layer
4345 :yocto_git:`meta-freescale </meta-freescale/>`.
4346 In this example, the
4347 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
4348 class shares GPU packages for i.MX53 boards because all boards share
4349 the AMD GPU. The i.MX6-based boards can do the same because all
4350 boards share the Vivante GPU. This class inspects the BitBake
4351 datastore to identify if the package provides or depends on one of
4352 the sub-architecture values. If so, the class sets the
4353 :term:`PACKAGE_ARCH` value
4354 based on the ``MACHINE_SUBARCH`` value. If the package does not
4355 provide or depend on one of the sub-architecture values but it
4356 matches a value in the machine-specific filter, it sets
4357 :term:`MACHINE_ARCH`. This
4358 behavior reduces the number of packages built and saves build time by
4359 reusing binaries.
4360
4361- *Use Tools to Debug Issues:* Sometimes you can run into situations
4362 where software is being rebuilt when you think it should not be. For
4363 example, the OpenEmbedded build system might not be using shared
4364 state between machines when you think it should be. These types of
4365 situations are usually due to references to machine-specific
4366 variables such as :term:`MACHINE`,
4367 :term:`SERIAL_CONSOLES`,
4368 :term:`XSERVER`,
4369 :term:`MACHINE_FEATURES`,
4370 and so forth in code that is supposed to only be tune-specific or
4371 when the recipe depends
4372 (:term:`DEPENDS`,
4373 :term:`RDEPENDS`,
4374 :term:`RRECOMMENDS`,
4375 :term:`RSUGGESTS`, and so forth)
4376 on some other recipe that already has
4377 :term:`PACKAGE_ARCH` defined
4378 as "${MACHINE_ARCH}".
4379
4380 .. note::
4381
4382 Patches to fix any issues identified are most welcome as these
4383 issues occasionally do occur.
4384
4385 For such cases, you can use some tools to help you sort out the
4386 situation:
4387
4388 - ``state-diff-machines.sh``*:* You can find this tool in the
4389 ``scripts`` directory of the Source Repositories. See the comments
4390 in the script for information on how to use the tool.
4391
4392 - *BitBake's "-S printdiff" Option:* Using this option causes
4393 BitBake to try to establish the closest signature match it can
4394 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4395 over the matches to determine the stamps and delta where these two
4396 stamp trees diverge.
4397
4398Building Software from an External Source
4399-----------------------------------------
4400
4401By default, the OpenEmbedded build system uses the
4402:term:`Build Directory` when building source
4403code. The build process involves fetching the source files, unpacking
4404them, and then patching them if necessary before the build takes place.
4405
4406There are situations where you might want to build software from source
4407files that are external to and thus outside of the OpenEmbedded build
4408system. For example, suppose you have a project that includes a new BSP
4409with a heavily customized kernel. And, you want to minimize exposing the
4410build system to the development team so that they can focus on their
4411project and maintain everyone's workflow as much as possible. In this
4412case, you want a kernel source directory on the development machine
4413where the development occurs. You want the recipe's
4414:term:`SRC_URI` variable to point to
4415the external directory and use it as is, not copy it.
4416
4417To build from software that comes from an external source, all you need
4418to do is inherit the
4419:ref:`externalsrc <ref-classes-externalsrc>` class
4420and then set the
4421:term:`EXTERNALSRC` variable to
4422point to your external source code. Here are the statements to put in
4423your ``local.conf`` file::
4424
4425 INHERIT += "externalsrc"
4426 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
4427
4428This next example shows how to accomplish the same thing by setting
4429:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
4430
4431 EXTERNALSRC = "path"
4432 EXTERNALSRC_BUILD = "path"
4433
4434.. note::
4435
4436 In order for these settings to take effect, you must globally or
4437 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
4438 class.
4439
4440By default, :ref:`ref-classes-externalsrc` builds the source code in a
4441directory separate from the external source directory as specified by
4442:term:`EXTERNALSRC`. If you need
4443to have the source built in the same directory in which it resides, or
4444some other nominated directory, you can set
4445:term:`EXTERNALSRC_BUILD`
4446to point to that directory::
4447
4448 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
4449
4450Replicating a Build Offline
4451---------------------------
4452
4453It can be useful to take a "snapshot" of upstream sources used in a
4454build and then use that "snapshot" later to replicate the build offline.
4455To do so, you need to first prepare and populate your downloads
4456directory your "snapshot" of files. Once your downloads directory is
4457ready, you can use it at any time and from any machine to replicate your
4458build.
4459
4460Follow these steps to populate your Downloads directory:
4461
44621. *Create a Clean Downloads Directory:* Start with an empty downloads
4463 directory (:term:`DL_DIR`). You
4464 start with an empty downloads directory by either removing the files
4465 in the existing directory or by setting :term:`DL_DIR` to point to either
4466 an empty location or one that does not yet exist.
4467
44682. *Generate Tarballs of the Source Git Repositories:* Edit your
4469 ``local.conf`` configuration file as follows::
4470
4471 DL_DIR = "/home/your-download-dir/"
4472 BB_GENERATE_MIRROR_TARBALLS = "1"
4473
4474 During
4475 the fetch process in the next step, BitBake gathers the source files
4476 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
4477 the
4478 :term:`BB_GENERATE_MIRROR_TARBALLS`
4479 variable for more information.
4480
44813. *Populate Your Downloads Directory Without Building:* Use BitBake to
4482 fetch your sources but inhibit the build::
4483
4484 $ bitbake target --runonly=fetch
4485
4486 The downloads directory (i.e. ``${DL_DIR}``) now has
4487 a "snapshot" of the source files in the form of tarballs, which can
4488 be used for the build.
4489
44904. *Optionally Remove Any Git or other SCM Subdirectories From the
4491 Downloads Directory:* If you want, you can clean up your downloads
4492 directory by removing any Git or other Source Control Management
4493 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4494 already contain these subdirectories.
4495
4496Once your downloads directory has everything it needs regarding source
4497files, you can create your "own-mirror" and build your target.
4498Understand that you can use the files to build the target offline from
4499any machine and at any time.
4500
4501Follow these steps to build your target using the files in the downloads
4502directory:
4503
45041. *Using Local Files Only:* Inside your ``local.conf`` file, add the
4505 :term:`SOURCE_MIRROR_URL` variable, inherit the
4506 :ref:`own-mirrors <ref-classes-own-mirrors>` class, and use the
4507 :term:`BB_NO_NETWORK` variable to your ``local.conf``.
4508 ::
4509
4510 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4511 INHERIT += "own-mirrors"
4512 BB_NO_NETWORK = "1"
4513
4514 The :term:`SOURCE_MIRROR_URL` and :ref:`own-mirrors <ref-classes-own-mirrors>`
4515 class set up the system to use the downloads directory as your "own
4516 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
4517 BitBake's fetching process in step 3 stays local, which means files
4518 from your "own-mirror" are used.
4519
45202. *Start With a Clean Build:* You can start with a clean build by
4521 removing the
4522 ``${``\ :term:`TMPDIR`\ ``}``
4523 directory or using a new :term:`Build Directory`.
4524
45253. *Build Your Target:* Use BitBake to build your target::
4526
4527 $ bitbake target
4528
4529 The build completes using the known local "snapshot" of source
4530 files from your mirror. The resulting tarballs for your "snapshot" of
4531 source files are in the downloads directory.
4532
4533 .. note::
4534
4535 The offline build does not work if recipes attempt to find the
4536 latest version of software by setting
4537 :term:`SRCREV` to
4538 ``${``\ :term:`AUTOREV`\ ``}``::
4539
4540 SRCREV = "${AUTOREV}"
4541
4542 When a recipe sets :term:`SRCREV` to
4543 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
4544 attempt to determine the latest version of software from the SCM.
4545 Typically, recipes that use :term:`AUTOREV` are custom or modified
4546 recipes. Recipes that reside in public repositories usually do not
4547 use :term:`AUTOREV`.
4548
4549 If you do have recipes that use :term:`AUTOREV`, you can take steps to
4550 still use the recipes in an offline build. Do the following:
4551
4552 1. Use a configuration generated by enabling :ref:`build
4553 history <dev-manual/common-tasks:maintaining build output quality>`.
4554
4555 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
4556 stored :term:`SRCREV` values from the build's history. For more
4557 information on collecting these values, see the
4558 ":ref:`dev-manual/common-tasks:build history package information`"
4559 section.
4560
4561 3. Once you have the correct source revisions, you can modify
4562 those recipes to set :term:`SRCREV` to specific versions of the
4563 software.
4564
4565Speeding Up a Build
4566===================
4567
4568Build time can be an issue. By default, the build system uses simple
4569controls to try and maximize build efficiency. In general, the default
4570settings for all the following variables result in the most efficient
4571build times when dealing with single socket systems (i.e. a single CPU).
4572If you have multiple CPUs, you might try increasing the default values
4573to gain more speed. See the descriptions in the glossary for each
4574variable for more information:
4575
4576- :term:`BB_NUMBER_THREADS`:
4577 The maximum number of threads BitBake simultaneously executes.
4578
4579- :term:`BB_NUMBER_PARSE_THREADS`:
4580 The number of threads BitBake uses during parsing.
4581
4582- :term:`PARALLEL_MAKE`: Extra
4583 options passed to the ``make`` command during the
4584 :ref:`ref-tasks-compile` task in
4585 order to specify parallel compilation on the local build host.
4586
4587- :term:`PARALLEL_MAKEINST`:
4588 Extra options passed to the ``make`` command during the
4589 :ref:`ref-tasks-install` task in
4590 order to specify parallel installation on the local build host.
4591
4592As mentioned, these variables all scale to the number of processor cores
4593available on the build system. For single socket systems, this
4594auto-scaling ensures that the build system fundamentally takes advantage
4595of potential parallel operations during the build based on the build
4596machine's capabilities.
4597
4598Following are additional factors that can affect build speed:
4599
4600- File system type: The file system type that the build is being
4601 performed on can also influence performance. Using ``ext4`` is
4602 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4603 improved features such as extents.
4604
4605- Disabling the updating of access time using ``noatime``: The
4606 ``noatime`` mount option prevents the build system from updating file
4607 and directory access times.
4608
4609- Setting a longer commit: Using the "commit=" mount option increases
4610 the interval in seconds between disk cache writes. Changing this
4611 interval from the five second default to something longer increases
4612 the risk of data loss but decreases the need to write to the disk,
4613 thus increasing the build performance.
4614
4615- Choosing the packaging backend: Of the available packaging backends,
4616 IPK is the fastest. Additionally, selecting a singular packaging
4617 backend also helps.
4618
4619- Using ``tmpfs`` for :term:`TMPDIR`
4620 as a temporary file system: While this can help speed up the build,
4621 the benefits are limited due to the compiler using ``-pipe``. The
4622 build system goes to some lengths to avoid ``sync()`` calls into the
4623 file system on the principle that if there was a significant failure,
4624 the :term:`Build Directory`
4625 contents could easily be rebuilt.
4626
4627- Inheriting the
4628 :ref:`rm_work <ref-classes-rm-work>` class:
4629 Inheriting this class has shown to speed up builds due to
4630 significantly lower amounts of data stored in the data cache as well
4631 as on disk. Inheriting this class also makes cleanup of
4632 :term:`TMPDIR` faster, at the
4633 expense of being easily able to dive into the source code. File
4634 system maintainers have recommended that the fastest way to clean up
4635 large numbers of files is to reformat partitions rather than delete
4636 files due to the linear nature of partitions. This, of course,
4637 assumes you structure the disk partitions and file systems in a way
4638 that this is practical.
4639
4640Aside from the previous list, you should keep some trade offs in mind
4641that can help you speed up the build:
4642
4643- Remove items from
4644 :term:`DISTRO_FEATURES`
4645 that you might not need.
4646
4647- Exclude debug symbols and other debug information: If you do not need
4648 these symbols and other debug information, disabling the ``*-dbg``
4649 package generation can speed up the build. You can disable this
4650 generation by setting the
4651 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4652 variable to "1".
4653
4654- Disable static library generation for recipes derived from
4655 ``autoconf`` or ``libtool``: Following is an example showing how to
4656 disable static libraries and still provide an override to handle
4657 exceptions::
4658
4659 STATICLIBCONF = "--disable-static"
4660 STATICLIBCONF:sqlite3-native = ""
4661 EXTRA_OECONF += "${STATICLIBCONF}"
4662
4663 .. note::
4664
4665 - Some recipes need static libraries in order to work correctly
4666 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4667 as in the previous example, account for these kinds of
4668 exceptions.
4669
4670 - Some packages have packaging code that assumes the presence of
4671 the static libraries. If so, you might need to exclude them as
4672 well.
4673
4674Working With Libraries
4675======================
4676
4677Libraries are an integral part of your system. This section describes
4678some common practices you might find helpful when working with libraries
4679to build your system:
4680
4681- :ref:`How to include static library files
4682 <dev-manual/common-tasks:including static library files>`
4683
4684- :ref:`How to use the Multilib feature to combine multiple versions of
4685 library files into a single image
4686 <dev-manual/common-tasks:combining multiple versions of library files into one image>`
4687
4688- :ref:`How to install multiple versions of the same library in parallel on
4689 the same system
4690 <dev-manual/common-tasks:installing multiple versions of the same library>`
4691
4692Including Static Library Files
4693------------------------------
4694
4695If you are building a library and the library offers static linking, you
4696can control which static library files (``*.a`` files) get included in
4697the built library.
4698
4699The :term:`PACKAGES` and
4700:term:`FILES:* <FILES>` variables in the
4701``meta/conf/bitbake.conf`` configuration file define how files installed
4702by the ``do_install`` task are packaged. By default, the :term:`PACKAGES`
4703variable includes ``${PN}-staticdev``, which represents all static
4704library files.
4705
4706.. note::
4707
4708 Some previously released versions of the Yocto Project defined the
4709 static library files through ``${PN}-dev``.
4710
4711Following is part of the BitBake configuration file, where you can see
4712how the static library files are defined::
4713
4714 PACKAGE_BEFORE_PN ?= ""
4715 PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
4716 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4717 FILES = ""
4718
4719 FILES:${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
4720 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4721 ${base_bindir}/* ${base_sbindir}/* \
4722 ${base_libdir}/*${SOLIBS} \
4723 ${base_prefix}/lib/udev ${prefix}/lib/udev \
4724 ${base_libdir}/udev ${libdir}/udev \
4725 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4726 ${datadir}/pixmaps ${datadir}/applications \
4727 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4728 ${libdir}/bonobo/servers"
4729
4730 FILES:${PN}-bin = "${bindir}/* ${sbindir}/*"
4731
4732 FILES:${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
4733 ${datadir}/gnome/help"
4734 SECTION:${PN}-doc = "doc"
4735
4736 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
4737 FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
4738 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4739 ${datadir}/aclocal ${base_libdir}/*.o \
4740 ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
4741 ${libdir}/cmake ${datadir}/cmake"
4742 SECTION:${PN}-dev = "devel"
4743 ALLOW_EMPTY:${PN}-dev = "1"
4744 RDEPENDS:${PN}-dev = "${PN} (= ${EXTENDPKGV})"
4745
4746 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4747 SECTION:${PN}-staticdev = "devel"
4748 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
4749
4750Combining Multiple Versions of Library Files into One Image
4751-----------------------------------------------------------
4752
4753The build system offers the ability to build libraries with different
4754target optimizations or architecture formats and combine these together
4755into one system image. You can link different binaries in the image
4756against the different libraries as needed for specific use cases. This
4757feature is called "Multilib".
4758
4759An example would be where you have most of a system compiled in 32-bit
4760mode using 32-bit libraries, but you have something large, like a
4761database engine, that needs to be a 64-bit application and uses 64-bit
4762libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4763libraries.
4764
4765While the Multilib feature is most commonly used for 32 and 64-bit
4766differences, the approach the build system uses facilitates different
4767target optimizations. You could compile some binaries to use one set of
4768libraries and other binaries to use a different set of libraries. The
4769libraries could differ in architecture, compiler options, or other
4770optimizations.
4771
4772There are several examples in the ``meta-skeleton`` layer found in the
4773:term:`Source Directory`:
4774
4775- :oe_git:`conf/multilib-example.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example.conf>`
4776 configuration file.
4777
4778- :oe_git:`conf/multilib-example2.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example2.conf>`
4779 configuration file.
4780
4781- :oe_git:`recipes-multilib/images/core-image-multilib-example.bb </openembedded-core/tree/meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb>`
4782 recipe
4783
4784Preparing to Use Multilib
4785~~~~~~~~~~~~~~~~~~~~~~~~~
4786
4787User-specific requirements drive the Multilib feature. Consequently,
4788there is no one "out-of-the-box" configuration that would
4789meet your needs.
4790
4791In order to enable Multilib, you first need to ensure your recipe is
4792extended to support multiple libraries. Many standard recipes are
4793already extended and support multiple libraries. You can check in the
4794``meta/conf/multilib.conf`` configuration file in the
4795:term:`Source Directory` to see how this is
4796done using the
4797:term:`BBCLASSEXTEND` variable.
4798Eventually, all recipes will be covered and this list will not be
4799needed.
4800
4801For the most part, the :ref:`Multilib <ref-classes-multilib*>`
4802class extension works automatically to
4803extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
4804:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
4805Standard variables such as
4806:term:`DEPENDS`,
4807:term:`RDEPENDS`,
4808:term:`RPROVIDES`,
4809:term:`RRECOMMENDS`,
4810:term:`PACKAGES`, and
4811:term:`PACKAGES_DYNAMIC` are
4812automatically extended by the system. If you are extending any manual
4813code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
4814those names are extended correctly.
4815
4816Using Multilib
4817~~~~~~~~~~~~~~
4818
4819After you have set up the recipes, you need to define the actual
4820combination of multiple libraries you want to build. You accomplish this
4821through your ``local.conf`` configuration file in the
4822:term:`Build Directory`. An example
4823configuration would be as follows::
4824
4825 MACHINE = "qemux86-64"
4826 require conf/multilib.conf
4827 MULTILIBS = "multilib:lib32"
4828 DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
4829 IMAGE_INSTALL:append = "lib32-glib-2.0"
4830
4831This example enables an additional library named
4832``lib32`` alongside the normal target packages. When combining these
4833"lib32" alternatives, the example uses "x86" for tuning. For information
4834on this particular tuning, see
4835``meta/conf/machine/include/ia32/arch-ia32.inc``.
4836
4837The example then includes ``lib32-glib-2.0`` in all the images, which
4838illustrates one method of including a multiple library dependency. You
4839can use a normal image build to include this dependency, for example::
4840
4841 $ bitbake core-image-sato
4842
4843You can also build Multilib packages
4844specifically with a command like this::
4845
4846 $ bitbake lib32-glib-2.0
4847
4848Additional Implementation Details
4849~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4850
4851There are generic implementation details as well as details that are specific to
4852package management systems. Following are implementation details
4853that exist regardless of the package management system:
4854
4855- The typical convention used for the class extension code as used by
4856 Multilib assumes that all package names specified in
4857 :term:`PACKAGES` that contain
4858 ``${PN}`` have ``${PN}`` at the start of the name. When that
4859 convention is not followed and ``${PN}`` appears at the middle or the
4860 end of a name, problems occur.
4861
4862- The :term:`TARGET_VENDOR`
4863 value under Multilib will be extended to "-vendormlmultilib" (e.g.
4864 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
4865 slightly unwieldy contraction is that any "-" characters in the
4866 vendor string presently break Autoconf's ``config.sub``, and other
4867 separators are problematic for different reasons.
4868
4869Here are the implementation details for the RPM Package Management System:
4870
4871- A unique architecture is defined for the Multilib packages, along
4872 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
4873 :term:`Build Directory`. For
4874 example, consider ``lib32`` in a ``qemux86-64`` image. The possible
4875 architectures in the system are "all", "qemux86_64",
4876 "lib32:qemux86_64", and "lib32:x86".
4877
4878- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
4879 packaging. The naming for a normal RPM package and a Multilib RPM
4880 package in a ``qemux86-64`` system resolves to something similar to
4881 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
4882 respectively.
4883
4884- When installing a Multilib image, the RPM backend first installs the
4885 base image and then installs the Multilib libraries.
4886
4887- The build system relies on RPM to resolve the identical files in the
4888 two (or more) Multilib packages.
4889
4890Here are the implementation details for the IPK Package Management System:
4891
4892- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
4893 packaging. The naming for a normal RPM package and a Multilib IPK
4894 package in a ``qemux86-64`` system resolves to something like
4895 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``,
4896 respectively.
4897
4898- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
4899 packages with and without the Multilib feature can exist in the same
4900 folder due to the ``${PN}`` differences.
4901
4902- IPK defines a sanity check for Multilib installation using certain
4903 rules for file comparison, overridden, etc.
4904
4905Installing Multiple Versions of the Same Library
4906------------------------------------------------
4907
4908There are be situations where you need to install and use multiple versions
4909of the same library on the same system at the same time. This
4910almost always happens when a library API changes and you have
4911multiple pieces of software that depend on the separate versions of the
4912library. To accommodate these situations, you can install multiple
4913versions of the same library in parallel on the same system.
4914
4915The process is straightforward as long as the libraries use proper
4916versioning. With properly versioned libraries, all you need to do to
4917individually specify the libraries is create separate, appropriately
4918named recipes where the :term:`PN` part of
4919the name includes a portion that differentiates each library version
4920(e.g. the major part of the version number). Thus, instead of having a
4921single recipe that loads one version of a library (e.g. ``clutter``),
4922you provide multiple recipes that result in different versions of the
4923libraries you want. As an example, the following two recipes would allow
4924the two separate versions of the ``clutter`` library to co-exist on the
4925same system:
4926
4927.. code-block:: none
4928
4929 clutter-1.6_1.6.20.bb
4930 clutter-1.8_1.8.4.bb
4931
4932Additionally, if
4933you have other recipes that depend on a given library, you need to use
4934the :term:`DEPENDS` variable to
4935create the dependency. Continuing with the same example, if you want to
4936have a recipe depend on the 1.8 version of the ``clutter`` library, use
4937the following in your recipe::
4938
4939 DEPENDS = "clutter-1.8"
4940
4941Working with Pre-Built Libraries
4942================================
4943
4944Introduction
4945-------------
4946
4947Some library vendors do not release source code for their software but do
4948release pre-built binaries. When shared libraries are built, they should
4949be versioned (see `this article
4950<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__
4951for some background), but sometimes this is not done.
4952
4953To summarize, a versioned library must meet two conditions:
4954
4955#. The filename must have the version appended, for example: ``libfoo.so.1.2.3``.
4956#. The library must have the ELF tag ``SONAME`` set to the major version
4957 of the library, for example: ``libfoo.so.1``. You can check this by
4958 running ``readelf -d filename | grep SONAME``.
4959
4960This section shows how to deal with both versioned and unversioned
4961pre-built libraries.
4962
4963Versioned Libraries
4964-------------------
4965
4966In this example we work with pre-built libraries for the FT4222H USB I/O chip.
4967Libraries are built for several target architecture variants and packaged in
4968an archive as follows::
4969
4970 ├── build-arm-hisiv300
4971 │   └── libft4222.so.1.4.4.44
4972 ├── build-arm-v5-sf
4973 │   └── libft4222.so.1.4.4.44
4974 ├── build-arm-v6-hf
4975 │   └── libft4222.so.1.4.4.44
4976 ├── build-arm-v7-hf
4977 │   └── libft4222.so.1.4.4.44
4978 ├── build-arm-v8
4979 │   └── libft4222.so.1.4.4.44
4980 ├── build-i386
4981 │   └── libft4222.so.1.4.4.44
4982 ├── build-i486
4983 │   └── libft4222.so.1.4.4.44
4984 ├── build-mips-eglibc-hf
4985 │   └── libft4222.so.1.4.4.44
4986 ├── build-pentium
4987 │   └── libft4222.so.1.4.4.44
4988 ├── build-x86_64
4989 │   └── libft4222.so.1.4.4.44
4990 ├── examples
4991 │   ├── get-version.c
4992 │   ├── i2cm.c
4993 │   ├── spim.c
4994 │   └── spis.c
4995 ├── ftd2xx.h
4996 ├── install4222.sh
4997 ├── libft4222.h
4998 ├── ReadMe.txt
4999 └── WinTypes.h
5000
5001To write a recipe to use such a library in your system:
5002
5003- The vendor will probably have a proprietary licence, so set
5004 :term:`LICENSE_FLAGS` in your recipe.
5005- The vendor provides a tarball containing libraries so set :term:`SRC_URI`
5006 appropriately.
5007- Set :term:`COMPATIBLE_HOST` so that the recipe cannot be used with an
5008 unsupported architecture. In the following example, we only support the 32
5009 and 64 bit variants of the ``x86`` architecture.
5010- As the vendor provides versioned libraries, we can use ``oe_soinstall``
5011 from :ref:`ref-classes-utils` to install the shared library and create
5012 symbolic links. If the vendor does not do this, we need to follow the
5013 non-versioned library guidelines in the next section.
5014- As the vendor likely used :term:`LDFLAGS` different from those in your Yocto
5015 Project build, disable the corresponding checks by adding ``ldflags``
5016 to :term:`INSANE_SKIP`.
5017- The vendor will typically ship release builds without debugging symbols.
5018 Avoid errors by preventing the packaging task from stripping out the symbols
5019 and adding them to a separate debug package. This is done by setting the
5020 ``INHIBIT_`` flags shown below.
5021
5022The complete recipe would look like this::
5023
5024 SUMMARY = "FTDI FT4222H Library"
5025 SECTION = "libs"
5026 LICENSE_FLAGS = "ftdi"
5027 LICENSE = "CLOSED"
5028
5029 COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
5030
5031 # Sources available in a .tgz file in .zip archive
5032 # at https://ftdichip.com/wp-content/uploads/2021/01/libft4222-linux-1.4.4.44.zip
5033 # Found on https://ftdichip.com/software-examples/ft4222h-software-examples/
5034 # Since dealing with this particular type of archive is out of topic here,
5035 # we use a local link.
5036 SRC_URI = "file://libft4222-linux-${PV}.tgz"
5037
5038 S = "${WORKDIR}"
5039
5040 ARCH_DIR:x86-64 = "build-x86_64"
5041 ARCH_DIR:i586 = "build-i386"
5042 ARCH_DIR:i686 = "build-i386"
5043
5044 INSANE_SKIP:${PN} = "ldflags"
5045 INHIBIT_PACKAGE_STRIP = "1"
5046 INHIBIT_SYSROOT_STRIP = "1"
5047 INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
5048
5049 do_install () {
5050 install -m 0755 -d ${D}${libdir}
5051 oe_soinstall ${S}/${ARCH_DIR}/libft4222.so.${PV} ${D}${libdir}
5052 install -d ${D}${includedir}
5053 install -m 0755 ${S}/*.h ${D}${includedir}
5054 }
5055
5056If the precompiled binaries are not statically linked and have dependencies on
5057other libraries, then by adding those libraries to :term:`DEPENDS`, the linking
5058can be examined and the appropriate :term:`RDEPENDS` automatically added.
5059
5060Non-Versioned Libraries
5061-----------------------
5062
5063Some Background
5064~~~~~~~~~~~~~~~
5065
5066Libraries in Linux systems are generally versioned so that it is possible
5067to have multiple versions of the same library installed, which eases upgrades
5068and support for older software. For example, suppose that in a versioned
5069library, an actual library is called ``libfoo.so.1.2``, a symbolic link named
5070``libfoo.so.1`` points to ``libfoo.so.1.2``, and a symbolic link named
5071``libfoo.so`` points to ``libfoo.so.1.2``. Given these conditions, when you
5072link a binary against a library, you typically provide the unversioned file
5073name (i.e. ``-lfoo`` to the linker). However, the linker follows the symbolic
5074link and actually links against the versioned filename. The unversioned symbolic
5075link is only used at development time. Consequently, the library is packaged
5076along with the headers in the development package ``${PN}-dev`` along with the
5077actual library and versioned symbolic links in ``${PN}``. Because versioned
5078libraries are far more common than unversioned libraries, the default packaging
5079rules assume versioned libraries.
5080
5081Yocto Library Packaging Overview
5082~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5083
5084It follows that packaging an unversioned library requires a bit of work in the
5085recipe. By default, ``libfoo.so`` gets packaged into ``${PN}-dev``, which
5086triggers a QA warning that a non-symlink library is in a ``-dev`` package,
5087and binaries in the same recipe link to the library in ``${PN}-dev``,
5088which triggers more QA warnings. To solve this problem, you need to package the
5089unversioned library into ``${PN}`` where it belongs. The following are the abridged
5090default :term:`FILES` variables in ``bitbake.conf``::
5091
5092 SOLIBS = ".so.*"
5093 SOLIBSDEV = ".so"
5094 FILES:${PN} = "... ${libdir}/lib*${SOLIBS} ..."
5095 FILES_SOLIBSDEV ?= "... ${libdir}/lib*${SOLIBSDEV} ..."
5096 FILES:${PN}-dev = "... ${FILES_SOLIBSDEV} ..."
5097
5098:term:`SOLIBS` defines a pattern that matches real shared object libraries.
5099:term:`SOLIBSDEV` matches the development form (unversioned symlink). These two
5100variables are then used in ``FILES:${PN}`` and ``FILES:${PN}-dev``, which puts
5101the real libraries into ``${PN}`` and the unversioned symbolic link into ``${PN}-dev``.
5102To package unversioned libraries, you need to modify the variables in the recipe
5103as follows::
5104
5105 SOLIBS = ".so"
5106 FILES_SOLIBSDEV = ""
5107
5108The modifications cause the ``.so`` file to be the real library
5109and unset :term:`FILES_SOLIBSDEV` so that no libraries get packaged into
5110``${PN}-dev``. The changes are required because unless :term:`PACKAGES` is changed,
5111``${PN}-dev`` collects files before `${PN}`. ``${PN}-dev`` must not collect any of
5112the files you want in ``${PN}``.
5113
5114Finally, loadable modules, essentially unversioned libraries that are linked
5115at runtime using ``dlopen()`` instead of at build time, should generally be
5116installed in a private directory. However, if they are installed in ``${libdir}``,
5117then the modules can be treated as unversioned libraries.
5118
5119Example
5120~~~~~~~
5121
5122The example below installs an unversioned x86-64 pre-built library named
5123``libfoo.so``. The :term:`COMPATIBLE_HOST` variable limits recipes to the
5124x86-64 architecture while the :term:`INSANE_SKIP`, :term:`INHIBIT_PACKAGE_STRIP`
5125and :term:`INHIBIT_SYSROOT_STRIP` variables are all set as in the above
5126versioned library example. The "magic" is setting the :term:`SOLIBS` and
5127:term:`FILES_SOLIBSDEV` variables as explained above::
5128
5129 SUMMARY = "libfoo sample recipe"
5130 SECTION = "libs"
5131 LICENSE = "CLOSED"
5132
5133 SRC_URI = "file://libfoo.so"
5134
5135 COMPATIBLE_HOST = "x86_64.*-linux"
5136
5137 INSANE_SKIP:${PN} = "ldflags"
5138 INHIBIT_PACKAGE_STRIP = "1"
5139 INHIBIT_SYSROOT_STRIP = "1"
5140 SOLIBS = ".so"
5141 FILES_SOLIBSDEV = ""
5142
5143 do_install () {
5144 install -d ${D}${libdir}
5145 install -m 0755 ${WORKDIR}/libfoo.so ${D}${libdir}
5146 }
5147
5148Using x32 psABI
5149===============
5150
5151x32 processor-specific Application Binary Interface (`x32
5152psABI <https://software.intel.com/en-us/node/628948>`__) is a native
515332-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
5154ABI defines the calling conventions between functions in a processing
5155environment. The interface determines what registers are used and what
5156the sizes are for various C data types.
5157
5158Some processing environments prefer using 32-bit applications even when
5159running on Intel 64-bit platforms. Consider the i386 psABI, which is a
5160very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
5161provide efficient use and access of the Intel 64-bit processor
5162resources, leaving the system underutilized. Now consider the x86_64
5163psABI. This ABI is newer and uses 64-bits for data sizes and program
5164pointers. The extra bits increase the footprint size of the programs,
5165libraries, and also increases the memory and file system size
5166requirements. Executing under the x32 psABI enables user programs to
5167utilize CPU and system resources more efficiently while keeping the
5168memory footprint of the applications low. Extra bits are used for
5169registers but not for addressing mechanisms.
5170
5171The Yocto Project supports the final specifications of x32 psABI as
5172follows:
5173
5174- You can create packages and images in x32 psABI format on x86_64
5175 architecture targets.
5176
5177- You can successfully build recipes with the x32 toolchain.
5178
5179- You can create and boot ``core-image-minimal`` and
5180 ``core-image-sato`` images.
5181
5182- There is RPM Package Manager (RPM) support for x32 binaries.
5183
5184- There is support for large images.
5185
5186To use the x32 psABI, you need to edit your ``conf/local.conf``
5187configuration file as follows::
5188
5189 MACHINE = "qemux86-64"
5190 DEFAULTTUNE = "x86-64-x32"
5191 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \
5192 or 'INVALID')) or 'lib'}"
5193
5194Once you have set
5195up your configuration file, use BitBake to build an image that supports
5196the x32 psABI. Here is an example::
5197
5198 $ bitbake core-image-sato
5199
5200Enabling GObject Introspection Support
5201======================================
5202
5203`GObject introspection <https://gi.readthedocs.io/en/latest/>`__
5204is the standard mechanism for accessing GObject-based software from
5205runtime environments. GObject is a feature of the GLib library that
5206provides an object framework for the GNOME desktop and related software.
5207GObject Introspection adds information to GObject that allows objects
5208created within it to be represented across different programming
5209languages. If you want to construct GStreamer pipelines using Python, or
5210control UPnP infrastructure using Javascript and GUPnP, GObject
5211introspection is the only way to do it.
5212
5213This section describes the Yocto Project support for generating and
5214packaging GObject introspection data. GObject introspection data is a
5215description of the API provided by libraries built on top of the GLib
5216framework, and, in particular, that framework's GObject mechanism.
5217GObject Introspection Repository (GIR) files go to ``-dev`` packages,
5218``typelib`` files go to main packages as they are packaged together with
5219libraries that are introspected.
5220
5221The data is generated when building such a library, by linking the
5222library with a small executable binary that asks the library to describe
5223itself, and then executing the binary and processing its output.
5224
5225Generating this data in a cross-compilation environment is difficult
5226because the library is produced for the target architecture, but its
5227code needs to be executed on the build host. This problem is solved with
5228the OpenEmbedded build system by running the code through QEMU, which
5229allows precisely that. Unfortunately, QEMU does not always work
5230perfectly as mentioned in the ":ref:`dev-manual/common-tasks:known issues`"
5231section.
5232
5233Enabling the Generation of Introspection Data
5234---------------------------------------------
5235
5236Enabling the generation of introspection data (GIR files) in your
5237library package involves the following:
5238
52391. Inherit the
5240 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
5241 class.
5242
52432. Make sure introspection is not disabled anywhere in the recipe or
5244 from anything the recipe includes. Also, make sure that
5245 "gobject-introspection-data" is not in
5246 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5247 and that "qemu-usermode" is not in
5248 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5249 In either of these conditions, nothing will happen.
5250
52513. Try to build the recipe. If you encounter build errors that look like
5252 something is unable to find ``.so`` libraries, check where these
5253 libraries are located in the source tree and add the following to the
5254 recipe::
5255
5256 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5257
5258 .. note::
5259
5260 See recipes in the ``oe-core`` repository that use that
5261 :term:`GIR_EXTRA_LIBS_PATH` variable as an example.
5262
52634. Look for any other errors, which probably mean that introspection
5264 support in a package is not entirely standard, and thus breaks down
5265 in a cross-compilation environment. For such cases, custom-made fixes
5266 are needed. A good place to ask and receive help in these cases is
5267 the :ref:`Yocto Project mailing
5268 lists <resources-mailinglist>`.
5269
5270.. note::
5271
5272 Using a library that no longer builds against the latest Yocto
5273 Project release and prints introspection related errors is a good
5274 candidate for the previous procedure.
5275
5276Disabling the Generation of Introspection Data
5277----------------------------------------------
5278
5279You might find that you do not want to generate introspection data. Or,
5280perhaps QEMU does not work on your build host and target architecture
5281combination. If so, you can use either of the following methods to
5282disable GIR file generations:
5283
5284- Add the following to your distro configuration::
5285
5286 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5287
5288 Adding this statement disables generating introspection data using
5289 QEMU but will still enable building introspection tools and libraries
5290 (i.e. building them does not require the use of QEMU).
5291
5292- Add the following to your machine configuration::
5293
5294 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5295
5296 Adding this statement disables the use of QEMU when building packages for your
5297 machine. Currently, this feature is used only by introspection
5298 recipes and has the same effect as the previously described option.
5299
5300 .. note::
5301
5302 Future releases of the Yocto Project might have other features
5303 affected by this option.
5304
5305If you disable introspection data, you can still obtain it through other
5306means such as copying the data from a suitable sysroot, or by generating
5307it on the target hardware. The OpenEmbedded build system does not
5308currently provide specific support for these techniques.
5309
5310Testing that Introspection Works in an Image
5311--------------------------------------------
5312
5313Use the following procedure to test if generating introspection data is
5314working in an image:
5315
53161. Make sure that "gobject-introspection-data" is not in
5317 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5318 and that "qemu-usermode" is not in
5319 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5320
53212. Build ``core-image-sato``.
5322
53233. Launch a Terminal and then start Python in the terminal.
5324
53254. Enter the following in the terminal::
5326
5327 >>> from gi.repository import GLib
5328 >>> GLib.get_host_name()
5329
53305. For something a little more advanced, enter the following see:
5331 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
5332
5333Known Issues
5334------------
5335
5336Here are know issues in GObject Introspection Support:
5337
5338- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5339 introspection data on that architecture.
5340
5341- x32 is not supported by QEMU. Consequently, introspection data is
5342 disabled.
5343
5344- musl causes transient GLib binaries to crash on assertion failures.
5345 Consequently, generating introspection data is disabled.
5346
5347- Because QEMU is not able to run the binaries correctly, introspection
5348 is disabled for some specific packages under specific architectures
5349 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5350
5351- QEMU usermode might not work properly when running 64-bit binaries
5352 under 32-bit host machines. In particular, "qemumips64" is known to
5353 not work under i686.
5354
5355Optionally Using an External Toolchain
5356======================================
5357
5358You might want to use an external toolchain as part of your development.
5359If this is the case, the fundamental steps you need to accomplish are as
5360follows:
5361
5362- Understand where the installed toolchain resides. For cases where you
5363 need to build the external toolchain, you would need to take separate
5364 steps to build and install the toolchain.
5365
5366- Make sure you add the layer that contains the toolchain to your
5367 ``bblayers.conf`` file through the
5368 :term:`BBLAYERS` variable.
5369
5370- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5371 to the location in which you installed the toolchain.
5372
5373A good example of an external toolchain used with the Yocto Project is
5374Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5375to use that particular layer in the ``README`` file at
5376https://github.com/MentorEmbedded/meta-sourcery/. You can find
5377further information by reading about the
5378:term:`TCMODE` variable in the Yocto
5379Project Reference Manual's variable glossary.
5380
5381Creating Partitioned Images Using Wic
5382=====================================
5383
5384Creating an image for a particular hardware target using the
5385OpenEmbedded build system does not necessarily mean you can boot that
5386image as is on your device. Physical devices accept and boot images in
5387various ways depending on the specifics of the device. Usually,
5388information about the hardware can tell you what image format the device
5389requires. Should your device require multiple partitions on an SD card,
5390flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5391create the properly partitioned image.
5392
5393The ``wic`` command generates partitioned images from existing
5394OpenEmbedded build artifacts. Image generation is driven by partitioning
5395commands contained in an Openembedded kickstart file (``.wks``)
5396specified either directly on the command line or as one of a selection
5397of canned kickstart files as shown with the ``wic list images`` command
5398in the
5399":ref:`dev-manual/common-tasks:generate an image using an existing kickstart file`"
5400section. When you apply the command to a given set of build artifacts, the
5401result is an image or set of images that can be directly written onto media and
5402used on a particular system.
5403
5404.. note::
5405
5406 For a kickstart file reference, see the
5407 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
5408 Chapter in the Yocto Project Reference Manual.
5409
5410The ``wic`` command and the infrastructure it is based on is by
5411definition incomplete. The purpose of the command is to allow the
5412generation of customized images, and as such, was designed to be
5413completely extensible through a plugin interface. See the
5414":ref:`dev-manual/common-tasks:using the wic plugin interface`" section
5415for information on these plugins.
5416
5417This section provides some background information on Wic, describes what
5418you need to have in place to run the tool, provides instruction on how
5419to use the Wic utility, provides information on using the Wic plugins
5420interface, and provides several examples that show how to use Wic.
5421
5422Background
5423----------
5424
5425This section provides some background on the Wic utility. While none of
5426this information is required to use Wic, you might find it interesting.
5427
5428- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5429 "oe" diphthong in "oeic" was promoted to the letter "w", because
5430 "oeic" is both difficult to remember and to pronounce.
5431
5432- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5433 The Wic implementation has been heavily modified to make direct use
5434 of OpenEmbedded build artifacts instead of package installation and
5435 configuration, which are already incorporated within the OpenEmbedded
5436 artifacts.
5437
5438- Wic is a completely independent standalone utility that initially
5439 provides easier-to-use and more flexible replacements for an existing
5440 functionality in OE-Core's
5441 :ref:`image-live <ref-classes-image-live>`
5442 class. The difference between Wic and those examples is that with Wic
5443 the functionality of those scripts is implemented by a
5444 general-purpose partitioning language, which is based on Redhat
5445 kickstart syntax.
5446
5447Requirements
5448------------
5449
5450In order to use the Wic utility with the OpenEmbedded Build system, your
5451system needs to meet the following requirements:
5452
5453- The Linux distribution on your development host must support the
5454 Yocto Project. See the ":ref:`detailed-supported-distros`"
5455 section in the Yocto Project Reference Manual for the list of
5456 distributions that support the Yocto Project.
5457
5458- The standard system utilities, such as ``cp``, must be installed on
5459 your development host system.
5460
5461- You must have sourced the build environment setup script (i.e.
5462 :ref:`structure-core-script`) found in the
5463 :term:`Build Directory`.
5464
5465- You need to have the build artifacts already available, which
5466 typically means that you must have already created an image using the
5467 Openembedded build system (e.g. ``core-image-minimal``). While it
5468 might seem redundant to generate an image in order to create an image
5469 using Wic, the current version of Wic requires the artifacts in the
5470 form generated by the OpenEmbedded build system.
5471
5472- You must build several native tools, which are built to run on the
5473 build system::
5474
5475 $ bitbake wic-tools
5476
5477- Include "wic" as part of the
5478 :term:`IMAGE_FSTYPES`
5479 variable.
5480
5481- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
5482 as part of the :term:`WKS_FILE` variable
5483
5484Getting Help
5485------------
5486
5487You can get general help for the ``wic`` command by entering the ``wic``
5488command by itself or by entering the command with a help argument as
5489follows::
5490
5491 $ wic -h
5492 $ wic --help
5493 $ wic help
5494
5495Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5496``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
5497commands except "help" by using the following form::
5498
5499 $ wic help command
5500
5501For example, the following command returns help for the ``write``
5502command::
5503
5504 $ wic help write
5505
5506Wic supports help for three topics: ``overview``, ``plugins``, and
5507``kickstart``. You can get help for any topic using the following form::
5508
5509 $ wic help topic
5510
5511For example, the following returns overview help for Wic::
5512
5513 $ wic help overview
5514
5515There is one additional level of help for Wic. You can get help on
5516individual images through the ``list`` command. You can use the ``list``
5517command to return the available Wic images as follows::
5518
5519 $ wic list images
5520 genericx86 Create an EFI disk image for genericx86*
5521 edgerouter Create SD card image for Edgerouter
5522 beaglebone-yocto Create SD card image for Beaglebone
5523 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5524 systemd-bootdisk Create an EFI disk image with systemd-boot
5525 mkhybridiso Create a hybrid ISO image
5526 mkefidisk Create an EFI disk image
5527 sdimage-bootpart Create SD card image with a boot partition
5528 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5529 directdisk Create a 'pcbios' direct disk image
5530 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5531 qemuriscv Create qcow2 image for RISC-V QEMU machines
5532 directdisk-gpt Create a 'pcbios' direct disk image
5533 efi-bootdisk
5534
5535Once you know the list of available
5536Wic images, you can use ``help`` with the command to get help on a
5537particular image. For example, the following command returns help on the
5538"beaglebone-yocto" image::
5539
5540 $ wic list beaglebone-yocto help
5541
5542 Creates a partitioned SD card image for Beaglebone.
5543 Boot files are located in the first vfat partition.
5544
5545Operational Modes
5546-----------------
5547
5548You can use Wic in two different modes, depending on how much control
5549you need for specifying the Openembedded build artifacts that are used
5550for creating the image: Raw and Cooked:
5551
5552- *Raw Mode:* You explicitly specify build artifacts through Wic
5553 command-line arguments.
5554
5555- *Cooked Mode:* The current
5556 :term:`MACHINE` setting and image
5557 name are used to automatically locate and provide the build
5558 artifacts. You just supply a kickstart file and the name of the image
5559 from which to use artifacts.
5560
5561Regardless of the mode you use, you need to have the build artifacts
5562ready and available.
5563
5564Raw Mode
5565~~~~~~~~
5566
5567Running Wic in raw mode allows you to specify all the partitions through
5568the ``wic`` command line. The primary use for raw mode is if you have
5569built your kernel outside of the Yocto Project
5570:term:`Build Directory`. In other words, you
5571can point to arbitrary kernel, root filesystem locations, and so forth.
5572Contrast this behavior with cooked mode where Wic looks in the Build
5573Directory (e.g. ``tmp/deploy/images/``\ machine).
5574
5575The general form of the ``wic`` command in raw mode is::
5576
5577 $ wic create wks_file options ...
5578
5579 Where:
5580
5581 wks_file:
5582 An OpenEmbedded kickstart file. You can provide
5583 your own custom file or use a file from a set of
5584 existing files as described by further options.
5585
5586 optional arguments:
5587 -h, --help show this help message and exit
5588 -o OUTDIR, --outdir OUTDIR
5589 name of directory to create image in
5590 -e IMAGE_NAME, --image-name IMAGE_NAME
5591 name of the image to use the artifacts from e.g. core-
5592 image-sato
5593 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5594 path to the /rootfs dir to use as the .wks rootfs
5595 source
5596 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5597 path to the dir containing the boot artifacts (e.g.
5598 /EFI or /syslinux dirs) to use as the .wks bootimg
5599 source
5600 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5601 path to the dir containing the kernel to use in the
5602 .wks bootimg
5603 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5604 path to the native sysroot containing the tools to use
5605 to build the image
5606 -s, --skip-build-check
5607 skip the build check
5608 -f, --build-rootfs build rootfs
5609 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5610 compress image with specified compressor
5611 -m, --bmap generate .bmap
5612 --no-fstab-update Do not change fstab file.
5613 -v VARS_DIR, --vars VARS_DIR
5614 directory with <image>.env files that store bitbake
5615 variables
5616 -D, --debug output debug information
5617
5618.. note::
5619
5620 You do not need root privileges to run Wic. In fact, you should not
5621 run as root when using the utility.
5622
5623Cooked Mode
5624~~~~~~~~~~~
5625
5626Running Wic in cooked mode leverages off artifacts in the Build
5627Directory. In other words, you do not have to specify kernel or root
5628filesystem locations as part of the command. All you need to provide is
5629a kickstart file and the name of the image from which to use artifacts
5630by using the "-e" option. Wic looks in the Build Directory (e.g.
5631``tmp/deploy/images/``\ machine) for artifacts.
5632
5633The general form of the ``wic`` command using Cooked Mode is as follows::
5634
5635 $ wic create wks_file -e IMAGE_NAME
5636
5637 Where:
5638
5639 wks_file:
5640 An OpenEmbedded kickstart file. You can provide
5641 your own custom file or use a file from a set of
5642 existing files provided with the Yocto Project
5643 release.
5644
5645 required argument:
5646 -e IMAGE_NAME, --image-name IMAGE_NAME
5647 name of the image to use the artifacts from e.g. core-
5648 image-sato
5649
5650Using an Existing Kickstart File
5651--------------------------------
5652
5653If you do not want to create your own kickstart file, you can use an
5654existing file provided by the Wic installation. As shipped, kickstart
5655files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
5656following two locations::
5657
5658 poky/meta-yocto-bsp/wic
5659 poky/scripts/lib/wic/canned-wks
5660
5661Use the following command to list the available kickstart files::
5662
5663 $ wic list images
5664 genericx86 Create an EFI disk image for genericx86*
5665 beaglebone-yocto Create SD card image for Beaglebone
5666 edgerouter Create SD card image for Edgerouter
5667 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
5668 directdisk-gpt Create a 'pcbios' direct disk image
5669 mkefidisk Create an EFI disk image
5670 directdisk Create a 'pcbios' direct disk image
5671 systemd-bootdisk Create an EFI disk image with systemd-boot
5672 mkhybridiso Create a hybrid ISO image
5673 sdimage-bootpart Create SD card image with a boot partition
5674 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5675 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5676
5677When you use an existing file, you
5678do not have to use the ``.wks`` extension. Here is an example in Raw
5679Mode that uses the ``directdisk`` file::
5680
5681 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5682 -k kernel_dir -n native_sysroot
5683
5684Here are the actual partition language commands used in the
5685``genericx86.wks`` file to generate an image::
5686
5687 # short-description: Create an EFI disk image for genericx86*
5688 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5689 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5690 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5691 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5692
5693 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5694
5695Using the Wic Plugin Interface
5696------------------------------
5697
5698You can extend and specialize Wic functionality by using Wic plugins.
5699This section explains the Wic plugin interface.
5700
5701.. note::
5702
5703 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5704 are beyond the scope of this section.
5705
5706Source plugins provide a mechanism to customize partition content during
5707the Wic image generation process. You can use source plugins to map
5708values that you specify using ``--source`` commands in kickstart files
5709(i.e. ``*.wks``) to a plugin implementation used to populate a given
5710partition.
5711
5712.. note::
5713
5714 If you use plugins that have build-time dependencies (e.g. native
5715 tools, bootloaders, and so forth) when building a Wic image, you need
5716 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
5717 variable.
5718
5719Source plugins are subclasses defined in plugin files. As shipped, the
5720Yocto Project provides several plugin files. You can see the source
5721plugin files that ship with the Yocto Project
5722:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
5723Each of these plugin files contains source plugins that are designed to
5724populate a specific Wic image partition.
5725
5726Source plugins are subclasses of the ``SourcePlugin`` class, which is
5727defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5728the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5729file is a subclass of the ``SourcePlugin`` class, which is found in the
5730``pluginbase.py`` file.
5731
5732You can also implement source plugins in a layer outside of the Source
5733Repositories (external layer). To do so, be sure that your plugin files
5734are located in a directory whose path is
5735``scripts/lib/wic/plugins/source/`` within your external layer. When the
5736plugin files are located there, the source plugins they contain are made
5737available to Wic.
5738
5739When the Wic implementation needs to invoke a partition-specific
5740implementation, it looks for the plugin with the same name as the
5741``--source`` parameter used in the kickstart file given to that
5742partition. For example, if the partition is set up using the following
5743command in a kickstart file::
5744
5745 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5746
5747The methods defined as class
5748members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5749``bootimg-pcbios.py`` plugin file are used.
5750
5751To be more concrete, here is the corresponding plugin definition from
5752the ``bootimg-pcbios.py`` file for the previous command along with an
5753example method called by the Wic implementation when it needs to prepare
5754a partition using an implementation-specific function::
5755
5756 .
5757 .
5758 .
5759 class BootimgPcbiosPlugin(SourcePlugin):
5760 """
5761 Create MBR boot partition and install syslinux on it.
5762 """
5763
5764 name = 'bootimg-pcbios'
5765 .
5766 .
5767 .
5768 @classmethod
5769 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5770 oe_builddir, bootimg_dir, kernel_dir,
5771 rootfs_dir, native_sysroot):
5772 """
5773 Called to do the actual content population for a partition i.e. it
5774 'prepares' the partition to be incorporated into the image.
5775 In this case, prepare content for legacy bios boot partition.
5776 """
5777 .
5778 .
5779 .
5780
5781If a
5782subclass (plugin) itself does not implement a particular function, Wic
5783locates and uses the default version in the superclass. It is for this
5784reason that all source plugins are derived from the ``SourcePlugin``
5785class.
5786
5787The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5788a set of methods that source plugins can implement or override. Any
5789plugins (subclass of ``SourcePlugin``) that do not implement a
5790particular method inherit the implementation of the method from the
5791``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5792class in the ``pluginbase.py`` file for details:
5793
5794The following list describes the methods implemented in the
5795``SourcePlugin`` class:
5796
5797- ``do_prepare_partition()``: Called to populate a partition with
5798 actual content. In other words, the method prepares the final
5799 partition image that is incorporated into the disk image.
5800
5801- ``do_configure_partition()``: Called before
5802 ``do_prepare_partition()`` to create custom configuration files for a
5803 partition (e.g. syslinux or grub configuration files).
5804
5805- ``do_install_disk()``: Called after all partitions have been
5806 prepared and assembled into a disk image. This method provides a hook
5807 to allow finalization of a disk image (e.g. writing an MBR).
5808
5809- ``do_stage_partition()``: Special content-staging hook called
5810 before ``do_prepare_partition()``. This method is normally empty.
5811
5812 Typically, a partition just uses the passed-in parameters (e.g. the
5813 unmodified value of ``bootimg_dir``). However, in some cases, things
5814 might need to be more tailored. As an example, certain files might
5815 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5816 allows those files to be staged in a customized fashion.
5817
5818 .. note::
5819
5820 ``get_bitbake_var()`` allows you to access non-standard variables that
5821 you might want to use for this behavior.
5822
5823You can extend the source plugin mechanism. To add more hooks, create
5824more source plugin methods within ``SourcePlugin`` and the corresponding
5825derived subclasses. The code that calls the plugin methods uses the
5826``plugin.get_source_plugin_methods()`` function to find the method or
5827methods needed by the call. Retrieval of those methods is accomplished
5828by filling up a dict with keys that contain the method names of
5829interest. On success, these will be filled in with the actual methods.
5830See the Wic implementation for examples and details.
5831
5832Wic Examples
5833------------
5834
5835This section provides several examples that show how to use the Wic
5836utility. All the examples assume the list of requirements in the
5837":ref:`dev-manual/common-tasks:requirements`" section have been met. The
5838examples assume the previously generated image is
5839``core-image-minimal``.
5840
5841Generate an Image using an Existing Kickstart File
5842~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5843
5844This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
5845file::
5846
5847 $ wic create mkefidisk -e core-image-minimal
5848 INFO: Building wic-tools...
5849 .
5850 .
5851 .
5852 INFO: The new image(s) can be found here:
5853 ./mkefidisk-201804191017-sda.direct
5854
5855 The following build artifacts were used to create the image(s):
5856 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5857 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5858 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5859 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5860
5861 INFO: The image(s) were created using OE kickstart file:
5862 /home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
5863
5864The previous example shows the easiest way to create an image by running
5865in cooked mode and supplying a kickstart file and the "-e" option to
5866point to the existing build artifacts. Your ``local.conf`` file needs to
5867have the :term:`MACHINE` variable set
5868to the machine you are using, which is "qemux86" in this example.
5869
5870Once the image builds, the output provides image location, artifact use,
5871and kickstart file information.
5872
5873.. note::
5874
5875 You should always verify the details provided in the output to make
5876 sure that the image was indeed created exactly as expected.
5877
5878Continuing with the example, you can now write the image from the Build
5879Directory onto a USB stick, or whatever media for which you built your
5880image, and boot from the media. You can write the image by using
5881``bmaptool`` or ``dd``::
5882
5883 $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
5884
5885or ::
5886
5887 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5888
5889.. note::
5890
5891 For more information on how to use the ``bmaptool``
5892 to flash a device with an image, see the
5893 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``"
5894 section.
5895
5896Using a Modified Kickstart File
5897~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5898
5899Because partitioned image creation is driven by the kickstart file, it
5900is easy to affect image creation by changing the parameters in the file.
5901This next example demonstrates that through modification of the
5902``directdisk-gpt`` kickstart file.
5903
5904As mentioned earlier, you can use the command ``wic list images`` to
5905show the list of existing kickstart files. The directory in which the
5906``directdisk-gpt.wks`` file resides is
5907``scripts/lib/image/canned-wks/``, which is located in the
5908:term:`Source Directory` (e.g. ``poky``).
5909Because available files reside in this directory, you can create and add
5910your own custom files to the directory. Subsequent use of the
5911``wic list images`` command would then include your kickstart files.
5912
5913In this example, the existing ``directdisk-gpt`` file already does most
5914of what is needed. However, for the hardware in this example, the image
5915will need to boot from ``sdb`` instead of ``sda``, which is what the
5916``directdisk-gpt`` kickstart file uses.
5917
5918The example begins by making a copy of the ``directdisk-gpt.wks`` file
5919in the ``scripts/lib/image/canned-wks`` directory and then by changing
5920the lines that specify the target disk from which to boot.
5921::
5922
5923 $ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5924 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5925
5926Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5927changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5928example changes the following two lines and leaves the remaining lines
5929untouched::
5930
5931 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5932 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5933
5934Once the lines are changed, the
5935example generates the ``directdisksdb-gpt`` image. The command points
5936the process at the ``core-image-minimal`` artifacts for the Next Unit of
5937Computing (nuc) :term:`MACHINE` the
5938``local.conf``.
5939::
5940
5941 $ wic create directdisksdb-gpt -e core-image-minimal
5942 INFO: Building wic-tools...
5943 .
5944 .
5945 .
5946 Initialising tasks: 100% |#######################################| Time: 0:00:01
5947 NOTE: Executing SetScene Tasks
5948 NOTE: Executing RunQueue Tasks
5949 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5950 INFO: Creating image(s)...
5951
5952 INFO: The new image(s) can be found here:
5953 ./directdisksdb-gpt-201710090938-sdb.direct
5954
5955 The following build artifacts were used to create the image(s):
5956 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5957 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5958 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5959 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5960
5961 INFO: The image(s) were created using OE kickstart file:
5962 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5963
5964Continuing with the example, you can now directly ``dd`` the image to a
5965USB stick, or whatever media for which you built your image, and boot
5966the resulting media::
5967
5968 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5969 140966+0 records in
5970 140966+0 records out
5971 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5972 $ sudo eject /dev/sdb
5973
5974Using a Modified Kickstart File and Running in Raw Mode
5975~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5976
5977This next example manually specifies each build artifact (runs in Raw
5978Mode) and uses a modified kickstart file. The example also uses the
5979``-o`` option to cause Wic to create the output somewhere other than the
5980default output directory, which is the current directory::
5981
5982 $ wic create test.wks -o /home/stephano/testwic \
5983 --rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5984 --bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5985 --kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \
5986 --native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
5987
5988 INFO: Creating image(s)...
5989
5990 INFO: The new image(s) can be found here:
5991 /home/stephano/testwic/test-201710091445-sdb.direct
5992
5993 The following build artifacts were used to create the image(s):
5994 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5995 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5996 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
5997 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5998
5999 INFO: The image(s) were created using OE kickstart file:
6000 test.wks
6001
6002For this example,
6003:term:`MACHINE` did not have to be
6004specified in the ``local.conf`` file since the artifact is manually
6005specified.
6006
6007Using Wic to Manipulate an Image
6008~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6009
6010Wic image manipulation allows you to shorten turnaround time during
6011image development. For example, you can use Wic to delete the kernel
6012partition of a Wic image and then insert a newly built kernel. This
6013saves you time from having to rebuild the entire image each time you
6014modify the kernel.
6015
6016.. note::
6017
6018 In order to use Wic to manipulate a Wic image as in this example,
6019 your development machine must have the ``mtools`` package installed.
6020
6021The following example examines the contents of the Wic image, deletes
6022the existing kernel, and then inserts a new kernel:
6023
60241. *List the Partitions:* Use the ``wic ls`` command to list all the
6025 partitions in the Wic image::
6026
6027 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
6028 Num Start End Size Fstype
6029 1 1048576 25041919 23993344 fat16
6030 2 25165824 72157183 46991360 ext4
6031
6032 The previous output shows two partitions in the
6033 ``core-image-minimal-qemux86.wic`` image.
6034
60352. *Examine a Particular Partition:* Use the ``wic ls`` command again
6036 but in a different form to examine a particular partition.
6037
6038 .. note::
6039
6040 You can get command usage on any Wic command using the following
6041 form::
6042
6043 $ wic help command
6044
6045
6046 For example, the following command shows you the various ways to
6047 use the
6048 wic ls
6049 command::
6050
6051 $ wic help ls
6052
6053
6054 The following command shows what is in partition one::
6055
6056 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
6057 Volume in drive : is boot
6058 Volume Serial Number is E894-1809
6059 Directory for ::/
6060
6061 libcom32 c32 186500 2017-10-09 16:06
6062 libutil c32 24148 2017-10-09 16:06
6063 syslinux cfg 220 2017-10-09 16:06
6064 vesamenu c32 27104 2017-10-09 16:06
6065 vmlinuz 6904608 2017-10-09 16:06
6066 5 files 7 142 580 bytes
6067 16 582 656 bytes free
6068
6069 The previous output shows five files, with the
6070 ``vmlinuz`` being the kernel.
6071
6072 .. note::
6073
6074 If you see the following error, you need to update or create a
6075 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
6076 in the file. Then, run the Wic command again::
6077
6078 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
6079 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
6080 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
6081
6082
60833. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
6084 ``vmlinuz`` file (kernel)::
6085
6086 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6087
60884. *Add In the New Kernel:* Use the ``wic cp`` command to add the
6089 updated kernel to the Wic image. Depending on how you built your
6090 kernel, it could be in different places. If you used ``devtool`` and
6091 an SDK to build your kernel, it resides in the ``tmp/work`` directory
6092 of the extensible SDK. If you used ``make`` to build the kernel, the
6093 kernel will be in the ``workspace/sources`` area.
6094
6095 The following example assumes ``devtool`` was used to build the
6096 kernel::
6097
6098 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
6099 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
6100
6101 Once the new kernel is added back into the image, you can use the
6102 ``dd`` command or :ref:`bmaptool
6103 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>`
6104 to flash your wic image onto an SD card or USB stick and test your
6105 target.
6106
6107 .. note::
6108
6109 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
6110
6111Flashing Images Using ``bmaptool``
6112==================================
6113
6114A fast and easy way to flash an image to a bootable device is to use
6115Bmaptool, which is integrated into the OpenEmbedded build system.
6116Bmaptool is a generic tool that creates a file's block map (bmap) and
6117then uses that map to copy the file. As compared to traditional tools
6118such as dd or cp, Bmaptool can copy (or flash) large files like raw
6119system image files much faster.
6120
6121.. note::
6122
6123 - If you are using Ubuntu or Debian distributions, you can install
6124 the ``bmap-tools`` package using the following command and then
6125 use the tool without specifying ``PATH`` even from the root
6126 account::
6127
6128 $ sudo apt install bmap-tools
6129
6130 - If you are unable to install the ``bmap-tools`` package, you will
6131 need to build Bmaptool before using it. Use the following command::
6132
6133 $ bitbake bmap-tools-native
6134
6135Following, is an example that shows how to flash a Wic image. Realize
6136that while this example uses a Wic image, you can use Bmaptool to flash
6137any type of image. Use these steps to flash an image using Bmaptool:
6138
61391. *Update your local.conf File:* You need to have the following set
6140 in your ``local.conf`` file before building your image::
6141
6142 IMAGE_FSTYPES += "wic wic.bmap"
6143
61442. *Get Your Image:* Either have your image ready (pre-built with the
6145 :term:`IMAGE_FSTYPES`
6146 setting previously mentioned) or take the step to build the image::
6147
6148 $ bitbake image
6149
61503. *Flash the Device:* Flash the device with the image by using Bmaptool
6151 depending on your particular setup. The following commands assume the
6152 image resides in the Build Directory's ``deploy/images/`` area:
6153
6154 - If you have write access to the media, use this command form::
6155
6156 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6157
6158 - If you do not have write access to the media, set your permissions
6159 first and then use the same command form::
6160
6161 $ sudo chmod 666 /dev/sdX
6162 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6163
6164For help on the ``bmaptool`` command, use the following command::
6165
6166 $ bmaptool --help
6167
6168Making Images More Secure
6169=========================
6170
6171Security is of increasing concern for embedded devices. Consider the
6172issues and problems discussed in just this sampling of work found across
6173the Internet:
6174
6175- *"*\ `Security Risks of Embedded
6176 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
6177 by Bruce Schneier
6178
6179- *"*\ `Internet Census
6180 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
6181 Botnet
6182
6183- *"*\ `Security Issues for Embedded
6184 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
6185 by Jake Edge
6186
6187When securing your image is of concern, there are steps, tools, and
6188variables that you can consider to help you reach the security goals you
6189need for your particular device. Not all situations are identical when
6190it comes to making an image secure. Consequently, this section provides
6191some guidance and suggestions for consideration when you want to make
6192your image more secure.
6193
6194.. note::
6195
6196 Because the security requirements and risks are different for every
6197 type of device, this section cannot provide a complete reference on
6198 securing your custom OS. It is strongly recommended that you also
6199 consult other sources of information on embedded Linux system
6200 hardening and on security.
6201
6202General Considerations
6203----------------------
6204
6205There are general considerations that help you create more secure images.
6206You should consider the following suggestions to make your device
6207more secure:
6208
6209- Scan additional code you are adding to the system (e.g. application
6210 code) by using static analysis tools. Look for buffer overflows and
6211 other potential security problems.
6212
6213- Pay particular attention to the security for any web-based
6214 administration interface.
6215
6216 Web interfaces typically need to perform administrative functions and
6217 tend to need to run with elevated privileges. Thus, the consequences
6218 resulting from the interface's security becoming compromised can be
6219 serious. Look for common web vulnerabilities such as
6220 cross-site-scripting (XSS), unvalidated inputs, and so forth.
6221
6222 As with system passwords, the default credentials for accessing a
6223 web-based interface should not be the same across all devices. This
6224 is particularly true if the interface is enabled by default as it can
6225 be assumed that many end-users will not change the credentials.
6226
6227- Ensure you can update the software on the device to mitigate
6228 vulnerabilities discovered in the future. This consideration
6229 especially applies when your device is network-enabled.
6230
6231- Ensure you remove or disable debugging functionality before producing
6232 the final image. For information on how to do this, see the
6233 ":ref:`dev-manual/common-tasks:considerations specific to the openembedded build system`"
6234 section.
6235
6236- Ensure you have no network services listening that are not needed.
6237
6238- Remove any software from the image that is not needed.
6239
6240- Enable hardware support for secure boot functionality when your
6241 device supports this functionality.
6242
6243Security Flags
6244--------------
6245
6246The Yocto Project has security flags that you can enable that help make
6247your build output more secure. The security flags are in the
6248``meta/conf/distro/include/security_flags.inc`` file in your
6249:term:`Source Directory` (e.g. ``poky``).
6250
6251.. note::
6252
6253 Depending on the recipe, certain security flags are enabled and
6254 disabled by default.
6255
6256Use the following line in your ``local.conf`` file or in your custom
6257distribution configuration file to enable the security compiler and
6258linker flags for your build::
6259
6260 require conf/distro/include/security_flags.inc
6261
6262Considerations Specific to the OpenEmbedded Build System
6263--------------------------------------------------------
6264
6265You can take some steps that are specific to the OpenEmbedded build
6266system to make your images more secure:
6267
6268- Ensure "debug-tweaks" is not one of your selected
6269 :term:`IMAGE_FEATURES`.
6270 When creating a new project, the default is to provide you with an
6271 initial ``local.conf`` file that enables this feature using the
6272 :term:`EXTRA_IMAGE_FEATURES`
6273 variable with the line::
6274
6275 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6276
6277 To disable that feature, simply comment out that line in your
6278 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
6279 "debug-tweaks" before producing your final image. Among other things,
6280 leaving this in place sets the root password as blank, which makes
6281 logging in for debugging or inspection easy during development but
6282 also means anyone can easily log in during production.
6283
6284- It is possible to set a root password for the image and also to set
6285 passwords for any extra users you might add (e.g. administrative or
6286 service type users). When you set up passwords for multiple images or
6287 users, you should not duplicate passwords.
6288
6289 To set up passwords, use the
6290 :ref:`extrausers <ref-classes-extrausers>`
6291 class, which is the preferred method. For an example on how to set up
6292 both root and user passwords, see the
6293 ":ref:`ref-classes-extrausers`" section.
6294
6295 .. note::
6296
6297 When adding extra user accounts or setting a root password, be
6298 cautious about setting the same password on every device. If you
6299 do this, and the password you have set is exposed, then every
6300 device is now potentially compromised. If you need this access but
6301 want to ensure security, consider setting a different, random
6302 password for each device. Typically, you do this as a separate
6303 step after you deploy the image onto the device.
6304
6305- Consider enabling a Mandatory Access Control (MAC) framework such as
6306 SMACK or SELinux and tuning it appropriately for your device's usage.
6307 You can find more information in the
6308 :yocto_git:`meta-selinux </meta-selinux/>` layer.
6309
6310Tools for Hardening Your Image
6311------------------------------
6312
6313The Yocto Project provides tools for making your image more secure. You
6314can find these tools in the ``meta-security`` layer of the
6315:yocto_git:`Yocto Project Source Repositories <>`.
6316
6317Creating Your Own Distribution
6318==============================
6319
6320When you build an image using the Yocto Project and do not alter any
6321distribution :term:`Metadata`, you are
6322creating a Poky distribution. If you wish to gain more control over
6323package alternative selections, compile-time options, and other
6324low-level configurations, you can create your own distribution.
6325
6326To create your own distribution, the basic steps consist of creating
6327your own distribution layer, creating your own distribution
6328configuration file, and then adding any needed code and Metadata to the
6329layer. The following steps provide some more detail:
6330
6331- *Create a layer for your new distro:* Create your distribution layer
6332 so that you can keep your Metadata and code for the distribution
6333 separate. It is strongly recommended that you create and use your own
6334 layer for configuration and code. Using your own layer as compared to
6335 just placing configurations in a ``local.conf`` configuration file
6336 makes it easier to reproduce the same build configuration when using
6337 multiple build machines. See the
6338 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
6339 section for information on how to quickly set up a layer.
6340
6341- *Create the distribution configuration file:* The distribution
6342 configuration file needs to be created in the ``conf/distro``
6343 directory of your layer. You need to name it using your distribution
6344 name (e.g. ``mydistro.conf``).
6345
6346 .. note::
6347
6348 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6349 name of your distribution.
6350
6351 You can split out parts of your configuration file into include files
6352 and then "require" them from within your distribution configuration
6353 file. Be sure to place the include files in the
6354 ``conf/distro/include`` directory of your layer. A common example
6355 usage of include files would be to separate out the selection of
6356 desired version and revisions for individual recipes.
6357
6358 Your configuration file needs to set the following required
6359 variables:
6360
6361 - :term:`DISTRO_NAME`
6362
6363 - :term:`DISTRO_VERSION`
6364
6365 These following variables are optional and you typically set them
6366 from the distribution configuration file:
6367
6368 - :term:`DISTRO_FEATURES`
6369
6370 - :term:`DISTRO_EXTRA_RDEPENDS`
6371
6372 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6373
6374 - :term:`TCLIBC`
6375
6376 .. tip::
6377
6378 If you want to base your distribution configuration file on the
6379 very basic configuration from OE-Core, you can use
6380 ``conf/distro/defaultsetup.conf`` as a reference and just include
6381 variables that differ as compared to ``defaultsetup.conf``.
6382 Alternatively, you can create a distribution configuration file
6383 from scratch using the ``defaultsetup.conf`` file or configuration files
6384 from another distribution such as Poky as a reference.
6385
6386- *Provide miscellaneous variables:* Be sure to define any other
6387 variables for which you want to create a default or enforce as part
6388 of the distribution configuration. You can include nearly any
6389 variable from the ``local.conf`` file. The variables you use are not
6390 limited to the list in the previous bulleted item.
6391
6392- *Point to Your distribution configuration file:* In your
6393 ``local.conf`` file in the :term:`Build Directory`,
6394 set your
6395 :term:`DISTRO` variable to point to
6396 your distribution's configuration file. For example, if your
6397 distribution's configuration file is named ``mydistro.conf``, then
6398 you point to it as follows::
6399
6400 DISTRO = "mydistro"
6401
6402- *Add more to the layer if necessary:* Use your layer to hold other
6403 information needed for the distribution:
6404
6405 - Add recipes for installing distro-specific configuration files
6406 that are not already installed by another recipe. If you have
6407 distro-specific configuration files that are included by an
6408 existing recipe, you should add an append file (``.bbappend``) for
6409 those. For general information and recommendations on how to add
6410 recipes to your layer, see the
6411 ":ref:`dev-manual/common-tasks:creating your own layer`" and
6412 ":ref:`dev-manual/common-tasks:following best practices when creating layers`"
6413 sections.
6414
6415 - Add any image recipes that are specific to your distribution.
6416
6417 - Add a ``psplash`` append file for a branded splash screen. For
6418 information on append files, see the
6419 ":ref:`dev-manual/common-tasks:appending other layers metadata with your layer`"
6420 section.
6421
6422 - Add any other append files to make custom changes that are
6423 specific to individual recipes.
6424
6425Creating a Custom Template Configuration Directory
6426==================================================
6427
6428If you are producing your own customized version of the build system for
6429use by other users, you might want to customize the message shown by the
6430setup script or you might want to change the template configuration
6431files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
6432new build directory.
6433
6434The OpenEmbedded build system uses the environment variable
6435``TEMPLATECONF`` to locate the directory from which it gathers
6436configuration information that ultimately ends up in the
6437:term:`Build Directory` ``conf`` directory.
6438By default, ``TEMPLATECONF`` is set as follows in the ``poky``
6439repository::
6440
6441 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
6442
6443This is the
6444directory used by the build system to find templates from which to build
6445some key configuration files. If you look at this directory, you will
6446see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6447``conf-notes.txt`` files. The build system uses these files to form the
6448respective ``bblayers.conf`` file, ``local.conf`` file, and display the
6449list of BitBake targets when running the setup script.
6450
6451To override these default configuration files with configurations you
6452want used within every new Build Directory, simply set the
6453``TEMPLATECONF`` variable to your directory. The ``TEMPLATECONF``
6454variable is set in the ``.templateconf`` file, which is in the top-level
6455:term:`Source Directory` folder
6456(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
6457directory.
6458
6459Best practices dictate that you should keep your template configuration
6460directory in your custom distribution layer. For example, suppose you
6461have a layer named ``meta-mylayer`` located in your home directory and
6462you want your template configuration directory named ``myconf``.
6463Changing the ``.templateconf`` as follows causes the OpenEmbedded build
6464system to look in your directory and base its configuration files on the
6465``*.sample`` configuration files it finds. The final configuration files
6466(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
6467your Build Directory, but they are based on your ``*.sample`` files.
6468::
6469
6470 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
6471
6472Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
6473also resides in the default ``meta-poky/conf`` directory. The script
6474that sets up the build environment (i.e.
6475:ref:`structure-core-script`) uses this file to
6476display BitBake targets as part of the script output. Customizing this
6477``conf-notes.txt`` file is a good way to make sure your list of custom
6478targets appears as part of the script's output.
6479
6480Here is the default list of targets displayed as a result of running
6481either of the setup scripts::
6482
6483 You can now run 'bitbake <target>'
6484
6485 Common targets are:
6486 core-image-minimal
6487 core-image-sato
6488 meta-toolchain
6489 meta-ide-support
6490
6491Changing the listed common targets is as easy as editing your version of
6492``conf-notes.txt`` in your custom template configuration directory and
6493making sure you have ``TEMPLATECONF`` set to your directory.
6494
6495Conserving Disk Space
6496=====================
6497
6498Conserving Disk Space During Builds
6499-----------------------------------
6500
6501To help conserve disk space during builds, you can add the following
6502statement to your project's ``local.conf`` configuration file found in
6503the :term:`Build Directory`::
6504
6505 INHERIT += "rm_work"
6506
6507Adding this statement deletes the work directory used for
6508building a recipe once the recipe is built. For more information on
6509"rm_work", see the
6510:ref:`rm_work <ref-classes-rm-work>` class in the
6511Yocto Project Reference Manual.
6512
6513Purging Duplicate Shared State Cache Files
6514------------------------------------------
6515
6516After multiple build iterations, the Shared State (sstate) cache can contain
6517duplicate cache files for a given package, consuming a substantial amount of
6518disk space. However, only the most recent cache files are likeky to be reusable.
6519
6520The following command is a quick way to purge all the cache files which
6521haven't been used for a least a specified number of days::
6522
6523 find build/sstate-cache -type f -mtime +$DAYS -delete
6524
6525The above command relies on the fact that BitBake touches the sstate cache
6526files as it accesses them, when it has write access to the cache.
6527
6528You could use ``-atime`` instead of ``-mtime`` if the partition isn't mounted
6529with the ``noatime`` option for a read only cache.
6530
6531For more advanced needs, OpenEmbedded-Core also offers a more elaborate
6532command. It has the ability to purge all but the newest cache files on each
6533architecture, and also to remove files that it considers unreachable by
6534exploring a set of build configurations. However, this command
6535requires a full build environment to be available and doesn't work well
6536covering multiple releases. It won't work either on limited environments
6537such as BSD based NAS::
6538
6539 sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
6540
6541This command will ask you to confirm the deletions it identifies.
6542Run ``sstate-cache-management.sh`` for more details about this script.
6543
6544.. note::
6545
6546 As this command is much more cautious and selective, removing only cache files,
6547 it will execute much slower than the simple ``find`` command described above.
6548 Therefore, it may not be your best option to trim huge cache directories.
6549
6550Working with Packages
6551=====================
6552
6553This section describes a few tasks that involve packages:
6554
6555- :ref:`dev-manual/common-tasks:excluding packages from an image`
6556
6557- :ref:`dev-manual/common-tasks:incrementing a package version`
6558
6559- :ref:`dev-manual/common-tasks:handling optional module packaging`
6560
6561- :ref:`dev-manual/common-tasks:using runtime package management`
6562
6563- :ref:`dev-manual/common-tasks:generating and using signed packages`
6564
6565- :ref:`Setting up and running package test
6566 (ptest) <dev-manual/common-tasks:testing packages with ptest>`
6567
6568- :ref:`dev-manual/common-tasks:creating node package manager (npm) packages`
6569
6570- :ref:`dev-manual/common-tasks:adding custom metadata to packages`
6571
6572Excluding Packages from an Image
6573--------------------------------
6574
6575You might find it necessary to prevent specific packages from being
6576installed into an image. If so, you can use several variables to direct
6577the build system to essentially ignore installing recommended packages
6578or to not install a package at all.
6579
6580The following list introduces variables you can use to prevent packages
6581from being installed into your image. Each of these variables only works
6582with IPK and RPM package types, not for Debian packages.
6583Also, you can use these variables from your ``local.conf`` file
6584or attach them to a specific image recipe by using a recipe name
6585override. For more detail on the variables, see the descriptions in the
6586Yocto Project Reference Manual's glossary chapter.
6587
6588- :term:`BAD_RECOMMENDATIONS`:
6589 Use this variable to specify "recommended-only" packages that you do
6590 not want installed.
6591
6592- :term:`NO_RECOMMENDATIONS`:
6593 Use this variable to prevent all "recommended-only" packages from
6594 being installed.
6595
6596- :term:`PACKAGE_EXCLUDE`:
6597 Use this variable to prevent specific packages from being installed
6598 regardless of whether they are "recommended-only" or not. You need to
6599 realize that the build process could fail with an error when you
6600 prevent the installation of a package whose presence is required by
6601 an installed package.
6602
6603Incrementing a Package Version
6604------------------------------
6605
6606This section provides some background on how binary package versioning
6607is accomplished and presents some of the services, variables, and
6608terminology involved.
6609
6610In order to understand binary package versioning, you need to consider
6611the following:
6612
6613- Binary Package: The binary package that is eventually built and
6614 installed into an image.
6615
6616- Binary Package Version: The binary package version is composed of two
6617 components - a version and a revision.
6618
6619 .. note::
6620
6621 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
6622 but this discussion for the most part ignores :term:`PE`.
6623
6624 The version and revision are taken from the
6625 :term:`PV` and
6626 :term:`PR` variables, respectively.
6627
6628- :term:`PV`: The recipe version. :term:`PV` represents the version of the
6629 software being packaged. Do not confuse :term:`PV` with the binary
6630 package version.
6631
6632- :term:`PR`: The recipe revision.
6633
6634- :term:`SRCPV`: The OpenEmbedded
6635 build system uses this string to help define the value of :term:`PV` when
6636 the source code revision needs to be included in it.
6637
6638- :yocto_wiki:`PR Service </PR_Service>`: A
6639 network-based service that helps automate keeping package feeds
6640 compatible with existing package manager applications such as RPM,
6641 APT, and OPKG.
6642
6643Whenever the binary package content changes, the binary package version
6644must change. Changing the binary package version is accomplished by
6645changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
6646values occurs one of two ways:
6647
6648- Automatically using a Package Revision Service (PR Service).
6649
6650- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
6651
6652Given a primary challenge of any build system and its users is how to
6653maintain a package feed that is compatible with existing package manager
6654applications such as RPM, APT, and OPKG, using an automated system is
6655much preferred over a manual system. In either system, the main
6656requirement is that binary package version numbering increases in a
6657linear fashion and that there is a number of version components that
6658support that linear progression. For information on how to ensure
6659package revisioning remains linear, see the
6660":ref:`dev-manual/common-tasks:automatically incrementing a package version number`"
6661section.
6662
6663The following three sections provide related information on the PR
6664Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
6665how to ensure binary package revisioning remains linear.
6666
6667Working With a PR Service
6668~~~~~~~~~~~~~~~~~~~~~~~~~
6669
6670As mentioned, attempting to maintain revision numbers in the
6671:term:`Metadata` is error prone, inaccurate,
6672and causes problems for people submitting recipes. Conversely, the PR
6673Service automatically generates increasing numbers, particularly the
6674revision field, which removes the human element.
6675
6676.. note::
6677
6678 For additional information on using a PR Service, you can see the
6679 :yocto_wiki:`PR Service </PR_Service>` wiki page.
6680
6681The Yocto Project uses variables in order of decreasing priority to
6682facilitate revision numbering (i.e.
6683:term:`PE`,
6684:term:`PV`, and
6685:term:`PR` for epoch, version, and
6686revision, respectively). The values are highly dependent on the policies
6687and procedures of a given distribution and package feed.
6688
6689Because the OpenEmbedded build system uses
6690":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
6691unique to a given build, the build system knows when to rebuild
6692packages. All the inputs into a given task are represented by a
6693signature, which can trigger a rebuild when different. Thus, the build
6694system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
6695trigger a rebuild. The signatures, however, can be used to generate
6696these values.
6697
6698The PR Service works with both ``OEBasic`` and ``OEBasicHash``
6699generators. The value of :term:`PR` bumps when the checksum changes and the
6700different generator mechanisms change signatures under different
6701circumstances.
6702
6703As implemented, the build system includes values from the PR Service
6704into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
6705becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
6706:term:`PR` values to be used for whatever reasons, which include manual
6707:term:`PR` bumps, should it be necessary.
6708
6709By default, the PR Service is not enabled or running. Thus, the packages
6710generated are just "self consistent". The build system adds and removes
6711packages and there are no guarantees about upgrade paths but images will
6712be consistent and correct with the latest changes.
6713
6714The simplest form for a PR Service is for a single host
6715development system that builds the package feed (building system). For
6716this scenario, you can enable a local PR Service by setting
6717:term:`PRSERV_HOST` in your
6718``local.conf`` file in the :term:`Build Directory`::
6719
6720 PRSERV_HOST = "localhost:0"
6721
6722Once the service is started, packages will automatically
6723get increasing :term:`PR` values and BitBake takes care of starting and
6724stopping the server.
6725
6726If you have a more complex setup where multiple host development systems
6727work against a common, shared package feed, you have a single PR Service
6728running and it is connected to each building system. For this scenario,
6729you need to start the PR Service using the ``bitbake-prserv`` command::
6730
6731 bitbake-prserv --host ip --port port --start
6732
6733In addition to
6734hand-starting the service, you need to update the ``local.conf`` file of
6735each building system as described earlier so each system points to the
6736server and port.
6737
6738It is also recommended you use build history, which adds some sanity
6739checks to binary package versions, in conjunction with the server that
6740is running the PR Service. To enable build history, add the following to
6741each building system's ``local.conf`` file::
6742
6743 # It is recommended to activate "buildhistory" for testing the PR service
6744 INHERIT += "buildhistory"
6745 BUILDHISTORY_COMMIT = "1"
6746
6747For information on build
6748history, see the
6749":ref:`dev-manual/common-tasks:maintaining build output quality`" section.
6750
6751.. note::
6752
6753 The OpenEmbedded build system does not maintain :term:`PR` information as
6754 part of the shared state (sstate) packages. If you maintain an sstate
6755 feed, it's expected that either all your building systems that
6756 contribute to the sstate feed use a shared PR Service, or you do not
6757 run a PR Service on any of your building systems. Having some systems
6758 use a PR Service while others do not leads to obvious problems.
6759
6760 For more information on shared state, see the
6761 ":ref:`overview-manual/concepts:shared state cache`"
6762 section in the Yocto Project Overview and Concepts Manual.
6763
6764Manually Bumping PR
6765~~~~~~~~~~~~~~~~~~~
6766
6767The alternative to setting up a PR Service is to manually "bump" the
6768:term:`PR` variable.
6769
6770If a committed change results in changing the package output, then the
6771value of the PR variable needs to be increased (or "bumped") as part of
6772that commit. For new recipes you should add the :term:`PR` variable and set
6773its initial value equal to "r0", which is the default. Even though the
6774default value is "r0", the practice of adding it to a new recipe makes
6775it harder to forget to bump the variable when you make changes to the
6776recipe in future.
6777
6778If you are sharing a common ``.inc`` file with multiple recipes, you can
6779also use the :term:`INC_PR` variable to ensure that the recipes sharing the
6780``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
6781``.inc`` file must set :term:`INC_PR` (initially to "r0"), and all recipes
6782referring to it should set :term:`PR` to "${INC_PR}.0" initially,
6783incrementing the last number when the recipe is changed. If the ``.inc``
6784file is changed then its :term:`INC_PR` should be incremented.
6785
6786When upgrading the version of a binary package, assuming the :term:`PV`
6787changes, the :term:`PR` variable should be reset to "r0" (or "${INC_PR}.0"
6788if you are using :term:`INC_PR`).
6789
6790Usually, version increases occur only to binary packages. However, if
6791for some reason :term:`PV` changes but does not increase, you can increase
6792the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
6793"0".
6794
6795Binary package version numbering strives to follow the `Debian Version
6796Field Policy
6797Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
6798These guidelines define how versions are compared and what "increasing"
6799a version means.
6800
6801Automatically Incrementing a Package Version Number
6802~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6803
6804When fetching a repository, BitBake uses the
6805:term:`SRCREV` variable to determine
6806the specific source code revision from which to build. You set the
6807:term:`SRCREV` variable to
6808:term:`AUTOREV` to cause the
6809OpenEmbedded build system to automatically use the latest revision of
6810the software::
6811
6812 SRCREV = "${AUTOREV}"
6813
6814Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
6815automatically update the version whenever the revision of the source
6816code changes. Here is an example::
6817
6818 PV = "1.0+git${SRCPV}"
6819
6820The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
6821
6822.. code-block:: none
6823
6824 AUTOINC+source_code_revision
6825
6826The build system replaces the ``AUTOINC``
6827with a number. The number used depends on the state of the PR Service:
6828
6829- If PR Service is enabled, the build system increments the number,
6830 which is similar to the behavior of
6831 :term:`PR`. This behavior results in
6832 linearly increasing package versions, which is desirable. Here is an
6833 example:
6834
6835 .. code-block:: none
6836
6837 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6838 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6839
6840- If PR Service is not enabled, the build system replaces the
6841 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6842 changing the package version since the source revision is included.
6843 However, package versions are not increased linearly. Here is an
6844 example:
6845
6846 .. code-block:: none
6847
6848 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6849 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6850
6851In summary, the OpenEmbedded build system does not track the history of
6852binary package versions for this purpose. ``AUTOINC``, in this case, is
6853comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
6854package version is simply replaced by "0". If PR server is enabled, the
6855build system keeps track of the package versions and bumps the number
6856when the package revision changes.
6857
6858Handling Optional Module Packaging
6859----------------------------------
6860
6861Many pieces of software split functionality into optional modules (or
6862plugins) and the plugins that are built might depend on configuration
6863options. To avoid having to duplicate the logic that determines what
6864modules are available in your recipe or to avoid having to package each
6865module by hand, the OpenEmbedded build system provides functionality to
6866handle module packaging dynamically.
6867
6868To handle optional module packaging, you need to do two things:
6869
6870- Ensure the module packaging is actually done.
6871
6872- Ensure that any dependencies on optional modules from other recipes
6873 are satisfied by your recipe.
6874
6875Making Sure the Packaging is Done
6876~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6877
6878To ensure the module packaging actually gets done, you use the
6879``do_split_packages`` function within the ``populate_packages`` Python
6880function in your recipe. The ``do_split_packages`` function searches for
6881a pattern of files or directories under a specified path and creates a
6882package for each one it finds by appending to the
6883:term:`PACKAGES` variable and
6884setting the appropriate values for ``FILES:packagename``,
6885``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
6886Here is an example from the ``lighttpd`` recipe::
6887
6888 python populate_packages:prepend () {
6889 lighttpd_libdir = d.expand('${libdir}')
6890 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6891 'lighttpd-module-%s', 'Lighttpd module for %s',
6892 extra_depends='')
6893 }
6894
6895The previous example specifies a number of things in the call to
6896``do_split_packages``.
6897
6898- A directory within the files installed by your recipe through
6899 ``do_install`` in which to search.
6900
6901- A regular expression used to match module files in that directory. In
6902 the example, note the parentheses () that mark the part of the
6903 expression from which the module name should be derived.
6904
6905- A pattern to use for the package names.
6906
6907- A description for each package.
6908
6909- An empty string for ``extra_depends``, which disables the default
6910 dependency on the main ``lighttpd`` package. Thus, if a file in
6911 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6912 ``lighttpd-module-alias`` is created for it and the
6913 :term:`DESCRIPTION` is set to
6914 "Lighttpd module for alias".
6915
6916Often, packaging modules is as simple as the previous example. However,
6917there are more advanced options that you can use within
6918``do_split_packages`` to modify its behavior. And, if you need to, you
6919can add more logic by specifying a hook function that is called for each
6920package. It is also perfectly acceptable to call ``do_split_packages``
6921multiple times if you have more than one set of modules to package.
6922
6923For more examples that show how to use ``do_split_packages``, see the
6924``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
6925directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
6926also find examples in ``meta/classes/kernel.bbclass``.
6927
6928Following is a reference that shows ``do_split_packages`` mandatory and
6929optional arguments::
6930
6931 Mandatory arguments
6932
6933 root
6934 The path in which to search
6935 file_regex
6936 Regular expression to match searched files.
6937 Use parentheses () to mark the part of this
6938 expression that should be used to derive the
6939 module name (to be substituted where %s is
6940 used in other function arguments as noted below)
6941 output_pattern
6942 Pattern to use for the package names. Must
6943 include %s.
6944 description
6945 Description to set for each package. Must
6946 include %s.
6947
6948 Optional arguments
6949
6950 postinst
6951 Postinstall script to use for all packages
6952 (as a string)
6953 recursive
6954 True to perform a recursive search - default
6955 False
6956 hook
6957 A hook function to be called for every match.
6958 The function will be called with the following
6959 arguments (in the order listed):
6960
6961 f
6962 Full path to the file/directory match
6963 pkg
6964 The package name
6965 file_regex
6966 As above
6967 output_pattern
6968 As above
6969 modulename
6970 The module name derived using file_regex
6971 extra_depends
6972 Extra runtime dependencies (RDEPENDS) to be
6973 set for all packages. The default value of None
6974 causes a dependency on the main package
6975 (${PN}) - if you do not want this, pass empty
6976 string '' for this parameter.
6977 aux_files_pattern
6978 Extra item(s) to be added to FILES for each
6979 package. Can be a single string item or a list
6980 of strings for multiple items. Must include %s.
6981 postrm
6982 postrm script to use for all packages (as a
6983 string)
6984 allow_dirs
6985 True to allow directories to be matched -
6986 default False
6987 prepend
6988 If True, prepend created packages to PACKAGES
6989 instead of the default False which appends them
6990 match_path
6991 match file_regex on the whole relative path to
6992 the root rather than just the filename
6993 aux_files_pattern_verbatim
6994 Extra item(s) to be added to FILES for each
6995 package, using the actual derived module name
6996 rather than converting it to something legal
6997 for a package name. Can be a single string item
6998 or a list of strings for multiple items. Must
6999 include %s.
7000 allow_links
7001 True to allow symlinks to be matched - default
7002 False
7003 summary
7004 Summary to set for each package. Must include %s;
7005 defaults to description if not set.
7006
7007
7008
7009Satisfying Dependencies
7010~~~~~~~~~~~~~~~~~~~~~~~
7011
7012The second part for handling optional module packaging is to ensure that
7013any dependencies on optional modules from other recipes are satisfied by
7014your recipe. You can be sure these dependencies are satisfied by using
7015the :term:`PACKAGES_DYNAMIC`
7016variable. Here is an example that continues with the ``lighttpd`` recipe
7017shown earlier::
7018
7019 PACKAGES_DYNAMIC = "lighttpd-module-.*"
7020
7021The name
7022specified in the regular expression can of course be anything. In this
7023example, it is ``lighttpd-module-`` and is specified as the prefix to
7024ensure that any :term:`RDEPENDS` and
7025:term:`RRECOMMENDS` on a package
7026name starting with the prefix are satisfied during build time. If you
7027are using ``do_split_packages`` as described in the previous section,
7028the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
7029pattern specified in the call to ``do_split_packages``.
7030
7031Using Runtime Package Management
7032--------------------------------
7033
7034During a build, BitBake always transforms a recipe into one or more
7035packages. For example, BitBake takes the ``bash`` recipe and produces a
7036number of packages (e.g. ``bash``, ``bash-bashbug``,
7037``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
7038``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
7039generated packages are included in an image.
7040
7041In several situations, you might need to update, add, remove, or query
7042the packages on a target device at runtime (i.e. without having to
7043generate a new image). Examples of such situations include:
7044
7045- You want to provide in-the-field updates to deployed devices (e.g.
7046 security updates).
7047
7048- You want to have a fast turn-around development cycle for one or more
7049 applications that run on your device.
7050
7051- You want to temporarily install the "debug" packages of various
7052 applications on your device so that debugging can be greatly improved
7053 by allowing access to symbols and source debugging.
7054
7055- You want to deploy a more minimal package selection of your device
7056 but allow in-the-field updates to add a larger selection for
7057 customization.
7058
7059In all these situations, you have something similar to a more
7060traditional Linux distribution in that in-field devices are able to
7061receive pre-compiled packages from a server for installation or update.
7062Being able to install these packages on a running, in-field device is
7063what is termed "runtime package management".
7064
7065In order to use runtime package management, you need a host or server
7066machine that serves up the pre-compiled packages plus the required
7067metadata. You also need package manipulation tools on the target. The
7068build machine is a likely candidate to act as the server. However, that
7069machine does not necessarily have to be the package server. The build
7070machine could push its artifacts to another machine that acts as the
7071server (e.g. Internet-facing). In fact, doing so is advantageous for a
7072production environment as getting the packages away from the development
7073system's build directory prevents accidental overwrites.
7074
7075A simple build that targets just one device produces more than one
7076package database. In other words, the packages produced by a build are
7077separated out into a couple of different package groupings based on
7078criteria such as the target's CPU architecture, the target board, or the
7079C library used on the target. For example, a build targeting the
7080``qemux86`` device produces the following three package databases:
7081``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
7082device to be aware of all the packages that were available to it, you
7083would need to point it to each of these databases individually. In a
7084similar way, a traditional Linux distribution usually is configured to
7085be aware of a number of software repositories from which it retrieves
7086packages.
7087
7088Using runtime package management is completely optional and not required
7089for a successful build or deployment in any way. But if you want to make
7090use of runtime package management, you need to do a couple things above
7091and beyond the basics. The remainder of this section describes what you
7092need to do.
7093
7094Build Considerations
7095~~~~~~~~~~~~~~~~~~~~
7096
7097This section describes build considerations of which you need to be
7098aware in order to provide support for runtime package management.
7099
7100When BitBake generates packages, it needs to know what format or formats
7101to use. In your configuration, you use the
7102:term:`PACKAGE_CLASSES`
7103variable to specify the format:
7104
71051. Open the ``local.conf`` file inside your
7106 :term:`Build Directory` (e.g.
7107 ``poky/build/conf/local.conf``).
7108
71092. Select the desired package format as follows::
7110
7111 PACKAGE_CLASSES ?= "package_packageformat"
7112
7113 where packageformat can be "ipk", "rpm",
7114 "deb", or "tar" which are the supported package formats.
7115
7116 .. note::
7117
7118 Because the Yocto Project supports four different package formats,
7119 you can set the variable with more than one argument. However, the
7120 OpenEmbedded build system only uses the first argument when
7121 creating an image or Software Development Kit (SDK).
7122
7123If you would like your image to start off with a basic package database
7124containing the packages in your current build as well as to have the
7125relevant tools available on the target for runtime package management,
7126you can include "package-management" in the
7127:term:`IMAGE_FEATURES`
7128variable. Including "package-management" in this configuration variable
7129ensures that when the image is assembled for your target, the image
7130includes the currently-known package databases as well as the
7131target-specific tools required for runtime package management to be
7132performed on the target. However, this is not strictly necessary. You
7133could start your image off without any databases but only include the
7134required on-target package tool(s). As an example, you could include
7135"opkg" in your
7136:term:`IMAGE_INSTALL` variable
7137if you are using the IPK package format. You can then initialize your
7138target's package database(s) later once your image is up and running.
7139
7140Whenever you perform any sort of build step that can potentially
7141generate a package or modify existing package, it is always a good idea
7142to re-generate the package index after the build by using the following
7143command::
7144
7145 $ bitbake package-index
7146
7147It might be tempting to build the
7148package and the package index at the same time with a command such as
7149the following::
7150
7151 $ bitbake some-package package-index
7152
7153Do not do this as
7154BitBake does not schedule the package index for after the completion of
7155the package you are building. Consequently, you cannot be sure of the
7156package index including information for the package you just built.
7157Thus, be sure to run the package update step separately after building
7158any packages.
7159
7160You can use the
7161:term:`PACKAGE_FEED_ARCHS`,
7162:term:`PACKAGE_FEED_BASE_PATHS`,
7163and
7164:term:`PACKAGE_FEED_URIS`
7165variables to pre-configure target images to use a package feed. If you
7166do not define these variables, then manual steps as described in the
7167subsequent sections are necessary to configure the target. You should
7168set these variables before building the image in order to produce a
7169correctly configured image.
7170
7171When your build is complete, your packages reside in the
7172``${TMPDIR}/deploy/packageformat`` directory. For example, if
7173``${``\ :term:`TMPDIR`\ ``}`` is
7174``tmp`` and your selected package type is RPM, then your RPM packages
7175are available in ``tmp/deploy/rpm``.
7176
7177Host or Server Machine Setup
7178~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7179
7180Although other protocols are possible, a server using HTTP typically
7181serves packages. If you want to use HTTP, then set up and configure a
7182web server such as Apache 2, lighttpd, or Python web server on the
7183machine serving the packages.
7184
7185To keep things simple, this section describes how to set up a
7186Python web server to share package feeds from the developer's
7187machine. Although this server might not be the best for a production
7188environment, the setup is simple and straight forward. Should you want
7189to use a different server more suited for production (e.g. Apache 2,
7190Lighttpd, or Nginx), take the appropriate steps to do so.
7191
7192From within the build directory where you have built an image based on
7193your packaging choice (i.e. the
7194:term:`PACKAGE_CLASSES`
7195setting), simply start the server. The following example assumes a build
7196directory of ``poky/build/tmp/deploy/rpm`` and a :term:`PACKAGE_CLASSES`
7197setting of "package_rpm"::
7198
7199 $ cd poky/build/tmp/deploy/rpm
7200 $ python3 -m http.server
7201
7202Target Setup
7203~~~~~~~~~~~~
7204
7205Setting up the target differs depending on the package management
7206system. This section provides information for RPM, IPK, and DEB.
7207
7208Using RPM
7209^^^^^^^^^
7210
7211The `Dandified Packaging
7212Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs
7213runtime package management of RPM packages. In order to use DNF for
7214runtime package management, you must perform an initial setup on the
7215target machine for cases where the ``PACKAGE_FEED_*`` variables were not
7216set as part of the image that is running on the target. This means if
7217you built your image and did not use these variables as part of the
7218build and your image is now running on the target, you need to perform
7219the steps in this section if you want to use runtime package management.
7220
7221.. note::
7222
7223 For information on the ``PACKAGE_FEED_*`` variables, see
7224 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
7225 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
7226 glossary.
7227
7228On the target, you must inform DNF that package databases are available.
7229You do this by creating a file named
7230``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
7231
7232As an example, assume the target is able to use the following package
7233databases: ``all``, ``i586``, and ``qemux86`` from a server named
7234``my.server``. The specifics for setting up the web server are up to
7235you. The critical requirement is that the URIs in the target repository
7236configuration point to the correct remote location for the feeds.
7237
7238.. note::
7239
7240 For development purposes, you can point the web server to the build
7241 system's ``deploy`` directory. However, for production use, it is better to
7242 copy the package directories to a location outside of the build area and use
7243 that location. Doing so avoids situations where the build system
7244 overwrites or changes the ``deploy`` directory.
7245
7246When telling DNF where to look for the package databases, you must
7247declare individual locations per architecture or a single location used
7248for all architectures. You cannot do both:
7249
7250- *Create an Explicit List of Architectures:* Define individual base
7251 URLs to identify where each package database is located:
7252
7253 .. code-block:: none
7254
7255 [oe-packages]
7256 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
7257
7258 This example
7259 informs DNF about individual package databases for all three
7260 architectures.
7261
7262- *Create a Single (Full) Package Index:* Define a single base URL that
7263 identifies where a full package database is located::
7264
7265 [oe-packages]
7266 baseurl=http://my.server/rpm
7267
7268 This example informs DNF about a single
7269 package database that contains all the package index information for
7270 all supported architectures.
7271
7272Once you have informed DNF where to find the package databases, you need
7273to fetch them:
7274
7275.. code-block:: none
7276
7277 # dnf makecache
7278
7279DNF is now able to find, install, and
7280upgrade packages from the specified repository or repositories.
7281
7282.. note::
7283
7284 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
7285 additional information.
7286
7287Using IPK
7288^^^^^^^^^
7289
7290The ``opkg`` application performs runtime package management of IPK
7291packages. You must perform an initial setup for ``opkg`` on the target
7292machine if the
7293:term:`PACKAGE_FEED_ARCHS`,
7294:term:`PACKAGE_FEED_BASE_PATHS`,
7295and
7296:term:`PACKAGE_FEED_URIS`
7297variables have not been set or the target image was built before the
7298variables were set.
7299
7300The ``opkg`` application uses configuration files to find available
7301package databases. Thus, you need to create a configuration file inside
7302the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
7303you want to use.
7304
7305As an example, suppose you are serving packages from a ``ipk/``
7306directory containing the ``i586``, ``all``, and ``qemux86`` databases
7307through an HTTP server named ``my.server``. On the target, create a
7308configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7309directory containing the following:
7310
7311.. code-block:: none
7312
7313 src/gz all http://my.server/ipk/all
7314 src/gz i586 http://my.server/ipk/i586
7315 src/gz qemux86 http://my.server/ipk/qemux86
7316
7317Next, instruct ``opkg`` to fetch the
7318repository information:
7319
7320.. code-block:: none
7321
7322 # opkg update
7323
7324The ``opkg`` application is now able to find, install, and upgrade packages
7325from the specified repository.
7326
7327Using DEB
7328^^^^^^^^^
7329
7330The ``apt`` application performs runtime package management of DEB
7331packages. This application uses a source list file to find available
7332package databases. You must perform an initial setup for ``apt`` on the
7333target machine if the
7334:term:`PACKAGE_FEED_ARCHS`,
7335:term:`PACKAGE_FEED_BASE_PATHS`,
7336and
7337:term:`PACKAGE_FEED_URIS`
7338variables have not been set or the target image was built before the
7339variables were set.
7340
7341To inform ``apt`` of the repository you want to use, you might create a
7342list file (e.g. ``my_repo.list``) inside the
7343``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7344serving packages from a ``deb/`` directory containing the ``i586``,
7345``all``, and ``qemux86`` databases through an HTTP server named
7346``my.server``. The list file should contain:
7347
7348.. code-block:: none
7349
7350 deb http://my.server/deb/all ./
7351 deb http://my.server/deb/i586 ./
7352 deb http://my.server/deb/qemux86 ./
7353
7354Next, instruct the ``apt`` application
7355to fetch the repository information:
7356
7357.. code-block:: none
7358
7359 $ sudo apt update
7360
7361After this step,
7362``apt`` is able to find, install, and upgrade packages from the
7363specified repository.
7364
7365Generating and Using Signed Packages
7366------------------------------------
7367
7368In order to add security to RPM packages used during a build, you can
7369take steps to securely sign them. Once a signature is verified, the
7370OpenEmbedded build system can use the package in the build. If security
7371fails for a signed package, the build system stops the build.
7372
7373This section describes how to sign RPM packages during a build and how
7374to use signed package feeds (repositories) when doing a build.
7375
7376Signing RPM Packages
7377~~~~~~~~~~~~~~~~~~~~
7378
7379To enable signing RPM packages, you must set up the following
7380configurations in either your ``local.config`` or ``distro.config``
7381file::
7382
7383 # Inherit sign_rpm.bbclass to enable signing functionality
7384 INHERIT += " sign_rpm"
7385 # Define the GPG key that will be used for signing.
7386 RPM_GPG_NAME = "key_name"
7387 # Provide passphrase for the key
7388 RPM_GPG_PASSPHRASE = "passphrase"
7389
7390.. note::
7391
7392 Be sure to supply appropriate values for both `key_name` and
7393 `passphrase`.
7394
7395Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
7396the previous example, two optional variables related to signing are available:
7397
7398- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7399 when the package is signed.
7400
7401- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7402 package is signed.
7403
7404Processing Package Feeds
7405~~~~~~~~~~~~~~~~~~~~~~~~
7406
7407In addition to being able to sign RPM packages, you can also enable
7408signed package feeds for IPK and RPM packages.
7409
7410The steps you need to take to enable signed package feed use are similar
7411to the steps used to sign RPM packages. You must define the following in
7412your ``local.config`` or ``distro.config`` file::
7413
7414 INHERIT += "sign_package_feed"
7415 PACKAGE_FEED_GPG_NAME = "key_name"
7416 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7417
7418For signed package feeds, the passphrase must be specified in a separate file,
7419which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7420variable. Regarding security, keeping a plain text passphrase out of the
7421configuration is more secure.
7422
7423Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7424``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
7425related to signed package feeds are available:
7426
7427- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7428 when the package is signed.
7429
7430- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7431 package is signed.
7432
7433- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7434 signature. This variable applies only to RPM and IPK package feeds.
7435 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7436 "ASC", which is the default and specifies ascii armored, and "BIN",
7437 which specifies binary.
7438
7439Testing Packages With ptest
7440---------------------------
7441
7442A Package Test (ptest) runs tests against packages built by the
7443OpenEmbedded build system on the target machine. A ptest contains at
7444least two items: the actual test, and a shell script (``run-ptest``)
7445that starts the test. The shell script that starts the test must not
7446contain the actual test - the script only starts the test. On the other
7447hand, the test can be anything from a simple shell script that runs a
7448binary and checks the output to an elaborate system of test binaries and
7449data files.
7450
7451The test generates output in the format used by Automake::
7452
7453 result: testname
7454
7455where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7456the testname can be any identifying string.
7457
7458For a list of Yocto Project recipes that are already enabled with ptest,
7459see the :yocto_wiki:`Ptest </Ptest>` wiki page.
7460
7461.. note::
7462
7463 A recipe is "ptest-enabled" if it inherits the
7464 :ref:`ptest <ref-classes-ptest>` class.
7465
7466Adding ptest to Your Build
7467~~~~~~~~~~~~~~~~~~~~~~~~~~
7468
7469To add package testing to your build, add the
7470:term:`DISTRO_FEATURES` and
7471:term:`EXTRA_IMAGE_FEATURES`
7472variables to your ``local.conf`` file, which is found in the
7473:term:`Build Directory`::
7474
7475 DISTRO_FEATURES:append = " ptest"
7476 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7477
7478Once your build is complete, the ptest files are installed into the
7479``/usr/lib/package/ptest`` directory within the image, where ``package``
7480is the name of the package.
7481
7482Running ptest
7483~~~~~~~~~~~~~
7484
7485The ``ptest-runner`` package installs a shell script that loops through
7486all installed ptest test suites and runs them in sequence. Consequently,
7487you might want to add this package to your image.
7488
7489Getting Your Package Ready
7490~~~~~~~~~~~~~~~~~~~~~~~~~~
7491
7492In order to enable a recipe to run installed ptests on target hardware,
7493you need to prepare the recipes that build the packages you want to
7494test. Here is what you have to do for each recipe:
7495
7496- *Be sure the recipe inherits
7497 the* :ref:`ptest <ref-classes-ptest>` *class:*
7498 Include the following line in each recipe::
7499
7500 inherit ptest
7501
7502- *Create run-ptest:* This script starts your test. Locate the
7503 script where you will refer to it using
7504 :term:`SRC_URI`. Here is an
7505 example that starts a test for ``dbus``::
7506
7507 #!/bin/sh
7508 cd test
7509 make -k runtest-TESTS
7510
7511- *Ensure dependencies are met:* If the test adds build or runtime
7512 dependencies that normally do not exist for the package (such as
7513 requiring "make" to run the test suite), use the
7514 :term:`DEPENDS` and
7515 :term:`RDEPENDS` variables in
7516 your recipe in order for the package to meet the dependencies. Here
7517 is an example where the package has a runtime dependency on "make"::
7518
7519 RDEPENDS:${PN}-ptest += "make"
7520
7521- *Add a function to build the test suite:* Not many packages support
7522 cross-compilation of their test suites. Consequently, you usually
7523 need to add a cross-compilation function to the package.
7524
7525 Many packages based on Automake compile and run the test suite by
7526 using a single command such as ``make check``. However, the host
7527 ``make check`` builds and runs on the same computer, while
7528 cross-compiling requires that the package is built on the host but
7529 executed for the target architecture (though often, as in the case
7530 for ptest, the execution occurs on the host). The built version of
7531 Automake that ships with the Yocto Project includes a patch that
7532 separates building and execution. Consequently, packages that use the
7533 unaltered, patched version of ``make check`` automatically
7534 cross-compiles.
7535
7536 Regardless, you still must add a ``do_compile_ptest`` function to
7537 build the test suite. Add a function similar to the following to your
7538 recipe::
7539
7540 do_compile_ptest() {
7541 oe_runmake buildtest-TESTS
7542 }
7543
7544- *Ensure special configurations are set:* If the package requires
7545 special configurations prior to compiling the test code, you must
7546 insert a ``do_configure_ptest`` function into the recipe.
7547
7548- *Install the test suite:* The ``ptest`` class automatically copies
7549 the file ``run-ptest`` to the target and then runs make
7550 ``install-ptest`` to run the tests. If this is not enough, you need
7551 to create a ``do_install_ptest`` function and make sure it gets
7552 called after the "make install-ptest" completes.
7553
7554Creating Node Package Manager (NPM) Packages
7555--------------------------------------------
7556
7557`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package
7558manager for the JavaScript programming language. The Yocto Project
7559supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
7560use this fetcher in combination with
7561:doc:`devtool </ref-manual/devtool-reference>` to create
7562recipes that produce NPM packages.
7563
7564There are two workflows that allow you to create NPM packages using
7565``devtool``: the NPM registry modules method and the NPM project code
7566method.
7567
7568.. note::
7569
7570 While it is possible to create NPM recipes manually, using
7571 ``devtool`` is far simpler.
7572
7573Additionally, some requirements and caveats exist.
7574
7575Requirements and Caveats
7576~~~~~~~~~~~~~~~~~~~~~~~~
7577
7578You need to be aware of the following before using ``devtool`` to create
7579NPM packages:
7580
7581- Of the two methods that you can use ``devtool`` to create NPM
7582 packages, the registry approach is slightly simpler. However, you
7583 might consider the project approach because you do not have to
7584 publish your module in the NPM registry
7585 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which
7586 is NPM's public registry.
7587
7588- Be familiar with
7589 :doc:`devtool </ref-manual/devtool-reference>`.
7590
7591- The NPM host tools need the native ``nodejs-npm`` package, which is
7592 part of the OpenEmbedded environment. You need to get the package by
7593 cloning the https://github.com/openembedded/meta-openembedded
7594 repository out of GitHub. Be sure to add the path to your local copy
7595 to your ``bblayers.conf`` file.
7596
7597- ``devtool`` cannot detect native libraries in module dependencies.
7598 Consequently, you must manually add packages to your recipe.
7599
7600- While deploying NPM packages, ``devtool`` cannot determine which
7601 dependent packages are missing on the target (e.g. the node runtime
7602 ``nodejs``). Consequently, you need to find out what files are
7603 missing and be sure they are on the target.
7604
7605- Although you might not need NPM to run your node package, it is
7606 useful to have NPM on your target. The NPM package name is
7607 ``nodejs-npm``.
7608
7609Using the Registry Modules Method
7610~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7611
7612This section presents an example that uses the ``cute-files`` module,
7613which is a file browser web application.
7614
7615.. note::
7616
7617 You must know the ``cute-files`` module version.
7618
7619The first thing you need to do is use ``devtool`` and the NPM fetcher to
7620create the recipe::
7621
7622 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7623
7624The
7625``devtool add`` command runs ``recipetool create`` and uses the same
7626fetch URI to download each dependency and capture license details where
7627possible. The result is a generated recipe.
7628
7629The recipe file is fairly simple and contains every license that
7630``recipetool`` finds and includes the licenses in the recipe's
7631:term:`LIC_FILES_CHKSUM`
7632variables. You need to examine the variables and look for those with
7633"unknown" in the :term:`LICENSE`
7634field. You need to track down the license information for "unknown"
7635modules and manually add the information to the recipe.
7636
7637``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7638files capture the version of all dependent modules. Many packages do not
7639provide shrinkwrap files. ``recipetool`` create a shrinkwrap file as it
7640runs.
7641
7642.. note::
7643
7644 A package is created for each sub-module. This policy is the only
7645 practical way to have the licenses for all of the dependencies
7646 represented in the license manifest of the image.
7647
7648The ``devtool edit-recipe`` command lets you take a look at the recipe::
7649
7650 $ devtool edit-recipe cute-files
7651 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7652 LICENSE = "MIT & ISC & Unknown"
7653 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
7654 file://node_modules/toidentifier/LICENSE;md5=1a261071a044d02eb6f2bb47f51a3502 \
7655 file://node_modules/debug/LICENSE;md5=ddd815a475e7338b0be7a14d8ee35a99 \
7656 ...
7657 SRC_URI = " \
7658 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7659 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7660 "
7661 S = "${WORKDIR}/npm"
7662 inherit npm
7663 LICENSE:${PN} = "MIT"
7664 LICENSE:${PN}-accepts = "MIT"
7665 LICENSE:${PN}-array-flatten = "MIT"
7666 ...
7667 LICENSE:${PN}-vary = "MIT"
7668
7669Here are three key points in the previous example:
7670
7671- :term:`SRC_URI` uses the NPM
7672 scheme so that the NPM fetcher is used.
7673
7674- ``recipetool`` collects all the license information. If a
7675 sub-module's license is unavailable, the sub-module's name appears in
7676 the comments.
7677
7678- The ``inherit npm`` statement causes the
7679 :ref:`npm <ref-classes-npm>` class to package
7680 up all the modules.
7681
7682You can run the following command to build the ``cute-files`` package::
7683
7684 $ devtool build cute-files
7685
7686Remember that ``nodejs`` must be installed on
7687the target before your package.
7688
7689Assuming 192.168.7.2 for the target's IP address, use the following
7690command to deploy your package::
7691
7692 $ devtool deploy-target -s cute-files root@192.168.7.2
7693
7694Once the package is installed on the target, you can
7695test the application:
7696
7697.. note::
7698
7699 Because of a known issue, you cannot simply run ``cute-files`` as you would
7700 if you had run ``npm install``.
7701
7702::
7703
7704 $ cd /usr/lib/node_modules/cute-files
7705 $ node cute-files.js
7706
7707On a browser,
7708go to ``http://192.168.7.2:3000`` and you see the following:
7709
7710.. image:: figures/cute-files-npm-example.png
7711 :align: center
7712
7713You can find the recipe in ``workspace/recipes/cute-files``. You can use
7714the recipe in any layer you choose.
7715
7716Using the NPM Projects Code Method
7717~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7718
7719Although it is useful to package modules already in the NPM registry,
7720adding ``node.js`` projects under development is a more common developer
7721use case.
7722
7723This section covers the NPM projects code method, which is very similar
7724to the "registry" approach described in the previous section. In the NPM
7725projects method, you provide ``devtool`` with an URL that points to the
7726source files.
7727
7728Replicating the same example, (i.e. ``cute-files``) use the following
7729command::
7730
7731 $ devtool add https://github.com/martinaglv/cute-files.git
7732
7733The
7734recipe this command generates is very similar to the recipe created in
7735the previous section. However, the :term:`SRC_URI` looks like the following::
7736
7737 SRC_URI = " \
7738 git://github.com/martinaglv/cute-files.git;protocol=https \
7739 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7740 "
7741
7742In this example,
7743the main module is taken from the Git repository and dependencies are
7744taken from the NPM registry. Other than those differences, the recipe is
7745basically the same between the two methods. You can build and deploy the
7746package exactly as described in the previous section that uses the
7747registry modules method.
7748
7749Adding custom metadata to packages
7750----------------------------------
7751
7752The variable
7753:term:`PACKAGE_ADD_METADATA`
7754can be used to add additional metadata to packages. This is reflected in
7755the package control/spec file. To take the ipk format for example, the
7756CONTROL file stored inside would contain the additional metadata as
7757additional lines.
7758
7759The variable can be used in multiple ways, including using suffixes to
7760set it for a specific package type and/or package. Note that the order
7761of precedence is the same as this list:
7762
7763- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
7764
7765- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7766
7767- ``PACKAGE_ADD_METADATA:<PN>``
7768
7769- :term:`PACKAGE_ADD_METADATA`
7770
7771`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
7772package type:
7773
7774- IPK for .ipk packages
7775
7776- DEB for .deb packages
7777
7778- RPM for .rpm packages
7779
7780`<PN>` is a parameter and expected to be a package name.
7781
7782The variable can contain multiple [one-line] metadata fields separated
7783by the literal sequence '\\n'. The separator can be redefined using the
7784variable flag ``separator``.
7785
7786Here is an example that adds two custom fields for ipk
7787packages::
7788
7789 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
7790
7791Efficiently Fetching Source Files During a Build
7792================================================
7793
7794The OpenEmbedded build system works with source files located through
7795the :term:`SRC_URI` variable. When
7796you build something using BitBake, a big part of the operation is
7797locating and downloading all the source tarballs. For images,
7798downloading all the source for various packages can take a significant
7799amount of time.
7800
7801This section shows you how you can use mirrors to speed up fetching
7802source files and how you can pre-fetch files all of which leads to more
7803efficient use of resources and time.
7804
7805Setting up Effective Mirrors
7806----------------------------
7807
7808A good deal that goes into a Yocto Project build is simply downloading
7809all of the source tarballs. Maybe you have been working with another
7810build system for which you have built up a
7811sizable directory of source tarballs. Or, perhaps someone else has such
7812a directory for which you have read access. If so, you can save time by
7813adding statements to your configuration file so that the build process
7814checks local directories first for existing tarballs before checking the
7815Internet.
7816
7817Here is an efficient way to set it up in your ``local.conf`` file::
7818
7819 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7820 INHERIT += "own-mirrors"
7821 BB_GENERATE_MIRROR_TARBALLS = "1"
7822 # BB_NO_NETWORK = "1"
7823
7824In the previous example, the
7825:term:`BB_GENERATE_MIRROR_TARBALLS`
7826variable causes the OpenEmbedded build system to generate tarballs of
7827the Git repositories and store them in the
7828:term:`DL_DIR` directory. Due to
7829performance reasons, generating and storing these tarballs is not the
7830build system's default behavior.
7831
7832You can also use the
7833:term:`PREMIRRORS` variable. For
7834an example, see the variable's glossary entry in the Yocto Project
7835Reference Manual.
7836
7837Getting Source Files and Suppressing the Build
7838----------------------------------------------
7839
7840Another technique you can use to ready yourself for a successive string
7841of build operations, is to pre-fetch all the source files without
7842actually starting a build. This technique lets you work through any
7843download issues and ultimately gathers all the source files into your
7844download directory :ref:`structure-build-downloads`,
7845which is located with :term:`DL_DIR`.
7846
7847Use the following BitBake command form to fetch all the necessary
7848sources without starting the build::
7849
7850 $ bitbake target --runall=fetch
7851
7852This
7853variation of the BitBake command guarantees that you have all the
7854sources for that BitBake target should you disconnect from the Internet
7855and want to do the build later offline.
7856
7857Selecting an Initialization Manager
7858===================================
7859
7860By default, the Yocto Project uses SysVinit as the initialization
7861manager. However, there is also support for systemd, which is a full
7862replacement for init with parallel starting of services, reduced shell
7863overhead and other features that are used by many distributions.
7864
7865Within the system, SysVinit treats system components as services. These
7866services are maintained as shell scripts stored in the ``/etc/init.d/``
7867directory. Services organize into different run levels. This
7868organization is maintained by putting links to the services in the
7869``/etc/rcN.d/`` directories, where `N/` is one of the following options:
7870"S", "0", "1", "2", "3", "4", "5", or "6".
7871
7872.. note::
7873
7874 Each runlevel has a dependency on the previous runlevel. This
7875 dependency allows the services to work properly.
7876
7877In comparison, systemd treats components as units. Using units is a
7878broader concept as compared to using a service. A unit includes several
7879different types of entities. Service is one of the types of entities.
7880The runlevel concept in SysVinit corresponds to the concept of a target
7881in systemd, where target is also a type of supported unit.
7882
7883In a SysVinit-based system, services load sequentially (i.e. one by one)
7884during init and parallelization is not supported. With systemd, services
7885start in parallel. Needless to say, the method can have an impact on
7886system startup performance.
7887
7888If you want to use SysVinit, you do not have to do anything. But, if you
7889want to use systemd, you must take some steps as described in the
7890following sections.
7891
7892Using systemd Exclusively
7893-------------------------
7894
7895Set these variables in your distribution configuration file as follows::
7896
7897 DISTRO_FEATURES:append = " systemd"
7898 VIRTUAL-RUNTIME_init_manager = "systemd"
7899
7900You can also prevent the SysVinit distribution feature from
7901being automatically enabled as follows::
7902
7903 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7904
7905Doing so removes any
7906redundant SysVinit scripts.
7907
7908To remove initscripts from your image altogether, set this variable
7909also::
7910
7911 VIRTUAL-RUNTIME_initscripts = ""
7912
7913For information on the backfill variable, see
7914:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7915
7916Using systemd for the Main Image and Using SysVinit for the Rescue Image
7917------------------------------------------------------------------------
7918
7919Set these variables in your distribution configuration file as follows::
7920
7921 DISTRO_FEATURES:append = " systemd"
7922 VIRTUAL-RUNTIME_init_manager = "systemd"
7923
7924Doing so causes your main image to use the
7925``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7926image cannot use this package group. However, it can install SysVinit
7927and the appropriate packages will have support for both systemd and
7928SysVinit.
7929
7930Using systemd-journald without a traditional syslog daemon
7931----------------------------------------------------------
7932
7933Counter-intuitively, ``systemd-journald`` is not a syslog runtime or provider,
7934and the proper way to use systemd-journald as your sole logging mechanism is to
7935effectively disable syslog entirely by setting these variables in your distribution
7936configuration file::
7937
7938 VIRTUAL-RUNTIME_syslog = ""
7939 VIRTUAL-RUNTIME_base-utils-syslog = ""
7940
7941Doing so will prevent ``rsyslog`` / ``busybox-syslog`` from being pulled in by
7942default, leaving only ``journald``.
7943
7944Selecting a Device Manager
7945==========================
7946
7947The Yocto Project provides multiple ways to manage the device manager
7948(``/dev``):
7949
7950- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev``
7951 directory is persistent and the required device nodes are created
7952 during the build.
7953
7954- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7955 directory is provided by the kernel as an in-memory file system and
7956 is automatically populated by the kernel at runtime. Additional
7957 configuration of device nodes is done in user space by a device
7958 manager like ``udev`` or ``busybox-mdev``.
7959
7960Using Persistent and Pre-Populated ``/dev``
7961--------------------------------------------
7962
7963To use the static method for device population, you need to set the
7964:term:`USE_DEVFS` variable to "0"
7965as follows::
7966
7967 USE_DEVFS = "0"
7968
7969The content of the resulting ``/dev`` directory is defined in a Device
7970Table file. The
7971:term:`IMAGE_DEVICE_TABLES`
7972variable defines the Device Table to use and should be set in the
7973machine or distro configuration file. Alternatively, you can set this
7974variable in your ``local.conf`` configuration file.
7975
7976If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
7977``device_table-minimal.txt`` is used::
7978
7979 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7980
7981The population is handled by the ``makedevs`` utility during image
7982creation:
7983
7984Using ``devtmpfs`` and a Device Manager
7985---------------------------------------
7986
7987To use the dynamic method for device population, you need to use (or be
7988sure to set) the :term:`USE_DEVFS`
7989variable to "1", which is the default::
7990
7991 USE_DEVFS = "1"
7992
7993With this
7994setting, the resulting ``/dev`` directory is populated by the kernel
7995using ``devtmpfs``. Make sure the corresponding kernel configuration
7996variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
7997kernel.
7998
7999All devices created by ``devtmpfs`` will be owned by ``root`` and have
8000permissions ``0600``.
8001
8002To have more control over the device nodes, you can use a device manager
8003like ``udev`` or ``busybox-mdev``. You choose the device manager by
8004defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
8005distro configuration file. Alternatively, you can set this variable in
8006your ``local.conf`` configuration file::
8007
8008 VIRTUAL-RUNTIME_dev_manager = "udev"
8009
8010 # Some alternative values
8011 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
8012 # VIRTUAL-RUNTIME_dev_manager = "systemd"
8013
8014Using an External SCM
8015=====================
8016
8017If you're working on a recipe that pulls from an external Source Code
8018Manager (SCM), it is possible to have the OpenEmbedded build system
8019notice new recipe changes added to the SCM and then build the resulting
8020packages that depend on the new recipes by using the latest versions.
8021This only works for SCMs from which it is possible to get a sensible
8022revision number for changes. Currently, you can do this with Apache
8023Subversion (SVN), Git, and Bazaar (BZR) repositories.
8024
8025To enable this behavior, the :term:`PV` of
8026the recipe needs to reference
8027:term:`SRCPV`. Here is an example::
8028
8029 PV = "1.2.3+git${SRCPV}"
8030
8031Then, you can add the following to your
8032``local.conf``::
8033
8034 SRCREV:pn-PN = "${AUTOREV}"
8035
8036:term:`PN` is the name of the recipe for
8037which you want to enable automatic source revision updating.
8038
8039If you do not want to update your local configuration file, you can add
8040the following directly to the recipe to finish enabling the feature::
8041
8042 SRCREV = "${AUTOREV}"
8043
8044The Yocto Project provides a distribution named ``poky-bleeding``, whose
8045configuration file contains the line::
8046
8047 require conf/distro/include/poky-floating-revisions.inc
8048
8049This line pulls in the
8050listed include file that contains numerous lines of exactly that form::
8051
8052 #SRCREV:pn-opkg-native ?= "${AUTOREV}"
8053 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}"
8054 #SRCREV:pn-opkg ?= "${AUTOREV}"
8055 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}"
8056 #SRCREV:pn-opkg-utils ?= "${AUTOREV}"
8057 SRCREV:pn-gconf-dbus ?= "${AUTOREV}"
8058 SRCREV:pn-matchbox-common ?= "${AUTOREV}"
8059 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}"
8060 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}"
8061 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}"
8062 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}"
8063 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}"
8064 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}"
8065 SRCREV:pn-matchbox-wm ?= "${AUTOREV}"
8066 SRCREV:pn-settings-daemon ?= "${AUTOREV}"
8067 SRCREV:pn-screenshot ?= "${AUTOREV}"
8068 . . .
8069
8070These lines allow you to
8071experiment with building a distribution that tracks the latest
8072development source for numerous packages.
8073
8074.. note::
8075
8076 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
8077 this in mind if you use it.
8078
8079Creating a Read-Only Root Filesystem
8080====================================
8081
8082Suppose, for security reasons, you need to disable your target device's
8083root filesystem's write permissions (i.e. you need a read-only root
8084filesystem). Or, perhaps you are running the device's operating system
8085from a read-only storage device. For either case, you can customize your
8086image for that behavior.
8087
8088.. note::
8089
8090 Supporting a read-only root filesystem requires that the system and
8091 applications do not try to write to the root filesystem. You must
8092 configure all parts of the target system to write elsewhere, or to
8093 gracefully fail in the event of attempting to write to the root
8094 filesystem.
8095
8096Creating the Root Filesystem
8097----------------------------
8098
8099To create the read-only root filesystem, simply add the
8100"read-only-rootfs" feature to your image, normally in one of two ways.
8101The first way is to add the "read-only-rootfs" image feature in the
8102image's recipe file via the :term:`IMAGE_FEATURES` variable::
8103
8104 IMAGE_FEATURES += "read-only-rootfs"
8105
8106As an alternative, you can add the same feature
8107from within your build directory's ``local.conf`` file with the
8108associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
8109
8110 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
8111
8112For more information on how to use these variables, see the
8113":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
8114section. For information on the variables, see
8115:term:`IMAGE_FEATURES` and
8116:term:`EXTRA_IMAGE_FEATURES`.
8117
8118Post-Installation Scripts and Read-Only Root Filesystem
8119-------------------------------------------------------
8120
8121It is very important that you make sure all post-Installation
8122(``pkg_postinst``) scripts for packages that are installed into the
8123image can be run at the time when the root filesystem is created during
8124the build on the host system. These scripts cannot attempt to run during
8125the first boot on the target device. With the "read-only-rootfs" feature
8126enabled, the build system makes sure that all post-installation scripts
8127succeed at file system creation time. If any of these scripts
8128still need to be run after the root filesystem is created, the build
8129immediately fails. These build-time checks ensure that the build fails
8130rather than the target device fails later during its initial boot
8131operation.
8132
8133Most of the common post-installation scripts generated by the build
8134system for the out-of-the-box Yocto Project are engineered so that they
8135can run during root filesystem creation (e.g. post-installation scripts
8136for caching fonts). However, if you create and add custom scripts, you
8137need to be sure they can be run during this file system creation.
8138
8139Here are some common problems that prevent post-installation scripts
8140from running during root filesystem creation:
8141
8142- *Not using $D in front of absolute paths:* The build system defines
8143 ``$``\ :term:`D` when the root
8144 filesystem is created. Furthermore, ``$D`` is blank when the script
8145 is run on the target device. This implies two purposes for ``$D``:
8146 ensuring paths are valid in both the host and target environments,
8147 and checking to determine which environment is being used as a method
8148 for taking appropriate actions.
8149
8150- *Attempting to run processes that are specific to or dependent on the
8151 target architecture:* You can work around these attempts by using
8152 native tools, which run on the host system, to accomplish the same
8153 tasks, or by alternatively running the processes under QEMU, which
8154 has the ``qemu_run_binary`` function. For more information, see the
8155 :ref:`qemu <ref-classes-qemu>` class.
8156
8157Areas With Write Access
8158-----------------------
8159
8160With the "read-only-rootfs" feature enabled, any attempt by the target
8161to write to the root filesystem at runtime fails. Consequently, you must
8162make sure that you configure processes and applications that attempt
8163these types of writes do so to directories with write access (e.g.
8164``/tmp`` or ``/var/run``).
8165
8166Maintaining Build Output Quality
8167================================
8168
8169Many factors can influence the quality of a build. For example, if you
8170upgrade a recipe to use a new version of an upstream software package or
8171you experiment with some new configuration options, subtle changes can
8172occur that you might not detect until later. Consider the case where
8173your recipe is using a newer version of an upstream package. In this
8174case, a new version of a piece of software might introduce an optional
8175dependency on another library, which is auto-detected. If that library
8176has already been built when the software is building, the software will
8177link to the built library and that library will be pulled into your
8178image along with the new software even if you did not want the library.
8179
8180The :ref:`buildhistory <ref-classes-buildhistory>`
8181class helps you maintain the quality of your build output. You
8182can use the class to highlight unexpected and possibly unwanted changes
8183in the build output. When you enable build history, it records
8184information about the contents of each package and image and then
8185commits that information to a local Git repository where you can examine
8186the information.
8187
8188The remainder of this section describes the following:
8189
8190- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>`
8191
8192- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>`
8193
8194- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>`
8195
8196- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>`
8197
8198Enabling and Disabling Build History
8199------------------------------------
8200
8201Build history is disabled by default. To enable it, add the following
8202:term:`INHERIT` statement and set the
8203:term:`BUILDHISTORY_COMMIT`
8204variable to "1" at the end of your ``conf/local.conf`` file found in the
8205:term:`Build Directory`::
8206
8207 INHERIT += "buildhistory"
8208 BUILDHISTORY_COMMIT = "1"
8209
8210Enabling build history as
8211previously described causes the OpenEmbedded build system to collect
8212build output information and commit it as a single commit to a local
8213:ref:`overview-manual/development-environment:git` repository.
8214
8215.. note::
8216
8217 Enabling build history increases your build times slightly,
8218 particularly for images, and increases the amount of disk space used
8219 during the build.
8220
8221You can disable build history by removing the previous statements from
8222your ``conf/local.conf`` file.
8223
8224Understanding What the Build History Contains
8225---------------------------------------------
8226
8227Build history information is kept in
8228``${``\ :term:`TOPDIR`\ ``}/buildhistory``
8229in the Build Directory as defined by the
8230:term:`BUILDHISTORY_DIR`
8231variable. Here is an example abbreviated listing:
8232
8233.. image:: figures/buildhistory.png
8234 :align: center
8235
8236At the top level, there is a ``metadata-revs`` file that lists the
8237revisions of the repositories for the enabled layers when the build was
8238produced. The rest of the data splits into separate ``packages``,
8239``images`` and ``sdk`` directories, the contents of which are described
8240as follows.
8241
8242Build History Package Information
8243~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8244
8245The history for each package contains a text file that has name-value
8246pairs with information about the package. For example,
8247``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
8248contains the following:
8249
8250.. code-block:: none
8251
8252 PV = 1.22.1
8253 PR = r32
8254 RPROVIDES =
8255 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
8256 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
8257 PKGSIZE = 540168
8258 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
8259 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
8260 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
8261 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
8262 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
8263 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
8264 /etc/busybox.links.nosuid /etc/busybox.links.suid
8265
8266Most of these
8267name-value pairs correspond to variables used to produce the package.
8268The exceptions are ``FILELIST``, which is the actual list of files in
8269the package, and ``PKGSIZE``, which is the total size of files in the
8270package in bytes.
8271
8272There is also a file that corresponds to the recipe from which the package
8273came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
8274
8275.. code-block:: none
8276
8277 PV = 1.22.1
8278 PR = r32
8279 DEPENDS = initscripts kern-tools-native update-rc.d-native \
8280 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
8281 virtual/libc virtual/update-alternatives
8282 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
8283 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
8284 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
8285
8286Finally, for those recipes fetched from a version control system (e.g.,
8287Git), there is a file that lists source revisions that are specified in
8288the recipe and the actual revisions used during the build. Listed
8289and actual revisions might differ when
8290:term:`SRCREV` is set to
8291${:term:`AUTOREV`}. Here is an
8292example assuming
8293``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
8294
8295 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8296 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8297 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
8298 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
8299
8300You can use the
8301``buildhistory-collect-srcrevs`` command with the ``-a`` option to
8302collect the stored :term:`SRCREV` values from build history and report them
8303in a format suitable for use in global configuration (e.g.,
8304``local.conf`` or a distro include file) to override floating
8305:term:`AUTOREV` values to a fixed set of revisions. Here is some example
8306output from this command::
8307
8308 $ buildhistory-collect-srcrevs -a
8309 # all-poky-linux
8310 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8"
8311 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac"
8312 # core2-64-poky-linux
8313 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66"
8314 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8"
8315 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0"
8316 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0"
8317 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1"
8318 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c"
8319 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7"
8320 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee"
8321 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1"
8322 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100"
8323 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3"
8324 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf"
8325 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced"
8326 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8"
8327 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f"
8328 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa"
8329 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70"
8330 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
8331 # qemux86_64-poky-linux
8332 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c"
8333 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0"
8334 # x86_64-linux
8335 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66"
8336 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c"
8337 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87"
8338 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57"
8339 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb"
8340 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b"
8341 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
8342
8343.. note::
8344
8345 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
8346
8347 - By default, only values where the :term:`SRCREV` was not hardcoded
8348 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
8349 option to see all :term:`SRCREV` values.
8350
8351 - The output statements might not have any effect if overrides are
8352 applied elsewhere in the build system configuration. Use the
8353 ``-f`` option to add the ``forcevariable`` override to each output
8354 line if you need to work around this restriction.
8355
8356 - The script does apply special handling when building for multiple
8357 machines. However, the script does place a comment before each set
8358 of values that specifies which triplet to which they belong as
8359 previously shown (e.g., ``i586-poky-linux``).
8360
8361Build History Image Information
8362~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8363
8364The files produced for each image are as follows:
8365
8366- ``image-files:`` A directory containing selected files from the root
8367 filesystem. The files are defined by
8368 :term:`BUILDHISTORY_IMAGE_FILES`.
8369
8370- ``build-id.txt:`` Human-readable information about the build
8371 configuration and metadata source revisions. This file contains the
8372 full build header as printed by BitBake.
8373
8374- ``*.dot:`` Dependency graphs for the image that are compatible with
8375 ``graphviz``.
8376
8377- ``files-in-image.txt:`` A list of files in the image with
8378 permissions, owner, group, size, and symlink information.
8379
8380- ``image-info.txt:`` A text file containing name-value pairs with
8381 information about the image. See the following listing example for
8382 more information.
8383
8384- ``installed-package-names.txt:`` A list of installed packages by name
8385 only.
8386
8387- ``installed-package-sizes.txt:`` A list of installed packages ordered
8388 by size.
8389
8390- ``installed-packages.txt:`` A list of installed packages with full
8391 package filenames.
8392
8393.. note::
8394
8395 Installed package information is able to be gathered and produced
8396 even if package management is disabled for the final image.
8397
8398Here is an example of ``image-info.txt``:
8399
8400.. code-block:: none
8401
8402 DISTRO = poky
8403 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be
8404 USER_CLASSES = buildstats
8405 IMAGE_CLASSES = qemuboot qemuboot license_image
8406 IMAGE_FEATURES = debug-tweaks
8407 IMAGE_LINGUAS =
8408 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp
8409 BAD_RECOMMENDATIONS =
8410 NO_RECOMMENDATIONS =
8411 PACKAGE_EXCLUDE =
8412 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible;
8413 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8414 IMAGESIZE = 9265
8415
8416Other than ``IMAGESIZE``,
8417which is the total size of the files in the image in Kbytes, the
8418name-value pairs are variables that may have influenced the content of
8419the image. This information is often useful when you are trying to
8420determine why a change in the package or file listings has occurred.
8421
8422Using Build History to Gather Image Information Only
8423~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8424
8425As you can see, build history produces image information, including
8426dependency graphs, so you can see why something was pulled into the
8427image. If you are just interested in this information and not interested
8428in collecting specific package or SDK information, you can enable
8429writing only image information without any history by adding the
8430following to your ``conf/local.conf`` file found in the
8431:term:`Build Directory`::
8432
8433 INHERIT += "buildhistory"
8434 BUILDHISTORY_COMMIT = "0"
8435 BUILDHISTORY_FEATURES = "image"
8436
8437Here, you set the
8438:term:`BUILDHISTORY_FEATURES`
8439variable to use the image feature only.
8440
8441Build History SDK Information
8442~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8443
8444Build history collects similar information on the contents of SDKs (e.g.
8445``bitbake -c populate_sdk imagename``) as compared to information it
8446collects for images. Furthermore, this information differs depending on
8447whether an extensible or standard SDK is being produced.
8448
8449The following list shows the files produced for SDKs:
8450
8451- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8452 owner, group, size, and symlink information. This list includes both
8453 the host and target parts of the SDK.
8454
8455- ``sdk-info.txt:`` A text file containing name-value pairs with
8456 information about the SDK. See the following listing example for more
8457 information.
8458
8459- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8460 with information about task group sizes (e.g. ``do_populate_sysroot``
8461 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8462 only when an extensible SDK is created.
8463
8464- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8465 with information for the shared-state packages and sizes in the SDK.
8466 The ``sstate-package-sizes.txt`` file exists only when an extensible
8467 SDK is created.
8468
8469- ``sdk-files:`` A folder that contains copies of the files mentioned
8470 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8471 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8472 specific to the extensible SDK although you can set it differently if
8473 you would like to pull in specific files from the standard SDK.
8474
8475 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8476 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8477 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8478 copied into the ``sdk-files`` directory.
8479
8480- The following information appears under each of the ``host`` and
8481 ``target`` directories for the portions of the SDK that run on the
8482 host and on the target, respectively:
8483
8484 .. note::
8485
8486 The following files for the most part are empty when producing an
8487 extensible SDK because this type of SDK is not constructed from
8488 packages as is the standard SDK.
8489
8490 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8491 with ``graphviz``.
8492
8493 - ``installed-package-names.txt:`` A list of installed packages by
8494 name only.
8495
8496 - ``installed-package-sizes.txt:`` A list of installed packages
8497 ordered by size.
8498
8499 - ``installed-packages.txt:`` A list of installed packages with full
8500 package filenames.
8501
8502Here is an example of ``sdk-info.txt``:
8503
8504.. code-block:: none
8505
8506 DISTRO = poky
8507 DISTRO_VERSION = 1.3+snapshot-20130327
8508 SDK_NAME = poky-glibc-i686-arm
8509 SDK_VERSION = 1.3+snapshot
8510 SDKMACHINE =
8511 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8512 BAD_RECOMMENDATIONS =
8513 SDKSIZE = 352712
8514
8515Other than ``SDKSIZE``, which is
8516the total size of the files in the SDK in Kbytes, the name-value pairs
8517are variables that might have influenced the content of the SDK. This
8518information is often useful when you are trying to determine why a
8519change in the package or file listings has occurred.
8520
8521Examining Build History Information
8522~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8523
8524You can examine build history output from the command line or from a web
8525interface.
8526
8527To see any changes that have occurred (assuming you have
8528:term:`BUILDHISTORY_COMMIT` = "1"),
8529you can simply use any Git command that allows you to view the history
8530of a repository. Here is one method::
8531
8532 $ git log -p
8533
8534You need to realize,
8535however, that this method does show changes that are not significant
8536(e.g. a package's size changing by a few bytes).
8537
8538There is a command-line tool called ``buildhistory-diff``, though,
8539that queries the Git repository and prints just the differences that
8540might be significant in human-readable form. Here is an example::
8541
8542 $ poky/poky/scripts/buildhistory-diff . HEAD^
8543 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8544 /etc/anotherpkg.conf was added
8545 /sbin/anotherpkg was added
8546 * (installed-package-names.txt):
8547 * anotherpkg was added
8548 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8549 anotherpkg was added
8550 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8551 * PR changed from "r0" to "r1"
8552 * PV changed from "0.1.10" to "0.1.12"
8553 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8554 * PR changed from "r0" to "r1"
8555 * PV changed from "0.1.10" to "0.1.12"
8556
8557.. note::
8558
8559 The ``buildhistory-diff`` tool requires the ``GitPython``
8560 package. Be sure to install it using Pip3 as follows::
8561
8562 $ pip3 install GitPython --user
8563
8564
8565 Alternatively, you can install ``python3-git`` using the appropriate
8566 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``).
8567
8568To see changes to the build history using a web interface, follow the
8569instruction in the ``README`` file
8570:yocto_git:`here </buildhistory-web/>`.
8571
8572Here is a sample screenshot of the interface:
8573
8574.. image:: figures/buildhistory-web.png
8575 :align: center
8576
8577Performing Automated Runtime Testing
8578====================================
8579
8580The OpenEmbedded build system makes available a series of automated
8581tests for images to verify runtime functionality. You can run these
8582tests on either QEMU or actual target hardware. Tests are written in
8583Python making use of the ``unittest`` module, and the majority of them
8584run commands on the target system over SSH. This section describes how
8585you set up the environment to use these tests, run available tests, and
8586write and add your own tests.
8587
8588For information on the test and QA infrastructure available within the
8589Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
8590section in the Yocto Project Reference Manual.
8591
8592Enabling Tests
8593--------------
8594
8595Depending on whether you are planning to run tests using QEMU or on the
8596hardware, you have to take different steps to enable the tests. See the
8597following subsections for information on how to enable both types of
8598tests.
8599
8600Enabling Runtime Tests on QEMU
8601~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8602
8603In order to run tests, you need to do the following:
8604
8605- *Set up to avoid interaction with sudo for networking:* To
8606 accomplish this, you must do one of the following:
8607
8608 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8609 commands or just for ``runqemu-ifup``. You must provide the full
8610 path as that can change if you are using multiple clones of the
8611 source repository.
8612
8613 .. note::
8614
8615 On some distributions, you also need to comment out "Defaults
8616 requiretty" in ``/etc/sudoers``.
8617
8618 - Manually configure a tap interface for your system.
8619
8620 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8621 should generate a list of tap devices. This is the option
8622 typically chosen for Autobuilder-type environments.
8623
8624 .. note::
8625
8626 - Be sure to use an absolute path when calling this script
8627 with sudo.
8628
8629 - The package recipe ``qemu-helper-native`` is required to run
8630 this script. Build the package using the following command::
8631
8632 $ bitbake qemu-helper-native
8633
8634- *Set the DISPLAY variable:* You need to set this variable so that
8635 you have an X server available (e.g. start ``vncserver`` for a
8636 headless machine).
8637
8638- *Be sure your host's firewall accepts incoming connections from
8639 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8640 HTTP server on a random high number port, which is used to serve
8641 files to the target. The DNF module serves
8642 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8643 That means your host's firewall must accept incoming connections from
8644 192.168.7.0/24, which is the default IP range used for tap devices by
8645 ``runqemu``.
8646
8647- *Be sure your host has the correct packages installed:* Depending
8648 your host's distribution, you need to have the following packages
8649 installed:
8650
8651 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8652
8653 - openSUSE: ``sysstat`` and ``iproute2``
8654
8655 - Fedora: ``sysstat`` and ``iproute``
8656
8657 - CentOS: ``sysstat`` and ``iproute``
8658
8659Once you start running the tests, the following happens:
8660
86611. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8662
86632. The image is booted under QEMU using the standard ``runqemu`` script.
8664
86653. A default timeout of 500 seconds occurs to allow for the boot process
8666 to reach the login prompt. You can change the timeout period by
8667 setting
8668 :term:`TEST_QEMUBOOT_TIMEOUT`
8669 in the ``local.conf`` file.
8670
86714. Once the boot process is reached and the login prompt appears, the
8672 tests run. The full boot log is written to
8673 ``${WORKDIR}/testimage/qemu_boot_log``.
8674
86755. Each test module loads in the order found in :term:`TEST_SUITES`. You can
8676 find the full output of the commands run over SSH in
8677 ``${WORKDIR}/testimgage/ssh_target_log``.
8678
86796. If no failures occur, the task running the tests ends successfully.
8680 You can find the output from the ``unittest`` in the task log at
8681 ``${WORKDIR}/temp/log.do_testimage``.
8682
8683Enabling Runtime Tests on Hardware
8684~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8685
8686The OpenEmbedded build system can run tests on real hardware, and for
8687certain devices it can also deploy the image to be tested onto the
8688device beforehand.
8689
8690For automated deployment, a "controller image" is installed onto the
8691hardware once as part of setup. Then, each time tests are to be run, the
8692following occurs:
8693
86941. The controller image is booted into and used to write the image to be
8695 tested to a second partition.
8696
86972. The device is then rebooted using an external script that you need to
8698 provide.
8699
87003. The device boots into the image to be tested.
8701
8702When running tests (independent of whether the image has been deployed
8703automatically or not), the device is expected to be connected to a
8704network on a pre-determined IP address. You can either use static IP
8705addresses written into the image, or set the image to use DHCP and have
8706your DHCP server on the test network assign a known IP address based on
8707the MAC address of the device.
8708
8709In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
8710appropriate value. For QEMU, you do not have to change anything, the
8711default value is "qemu". For running tests on hardware, the following
8712options are available:
8713
8714- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8715 on a target system that is already running the image to be tested and
8716 is available on the network. You can use "simpleremote" in
8717 conjunction with either real hardware or an image running within a
8718 separately started QEMU or any other virtual machine manager.
8719
8720- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8721 an EFI-based machine with ``systemd-boot`` as bootloader and
8722 ``core-image-testmaster`` (or something similar) is installed. Also,
8723 your hardware under test must be in a DHCP-enabled network that gives
8724 it the same IP address for each reboot.
8725
8726 If you choose "SystemdbootTarget", there are additional requirements
8727 and considerations. See the
8728 ":ref:`dev-manual/common-tasks:selecting systemdboottarget`" section, which
8729 follows, for more information.
8730
8731- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8732 images and running tests on the BeagleBone "Black" or original
8733 "White" hardware. For information on how to use these tests, see the
8734 comments at the top of the BeagleBoneTarget
8735 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8736
8737- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
8738 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8739 For information on how to use these tests, see the comments at the
8740 top of the EdgeRouterTarget
8741 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8742
8743- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
8744 tests on any generic PC that boots using GRUB. For information on how
8745 to use these tests, see the comments at the top of the GrubTarget
8746 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8747
8748- *"your-target":* Create your own custom target if you want to run
8749 tests when you are deploying images and running tests on a custom
8750 machine within your BSP layer. To do this, you need to add a Python
8751 unit that defines the target class under ``lib/oeqa/controllers/``
8752 within your layer. You must also provide an empty ``__init__.py``.
8753 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8754
8755Selecting SystemdbootTarget
8756~~~~~~~~~~~~~~~~~~~~~~~~~~~
8757
8758If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
8759not need any information in this section. You can skip down to the
8760":ref:`dev-manual/common-tasks:running tests`" section.
8761
8762If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
8763perform a one-time setup of your controller image by doing the following:
8764
87651. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
8766
8767 EFI_PROVIDER = "systemd-boot"
8768
87692. *Build the controller image:* Build the ``core-image-testmaster`` image.
8770 The ``core-image-testmaster`` recipe is provided as an example for a
8771 "controller" image and you can customize the image recipe as you would
8772 any other recipe.
8773
8774 Here are the image recipe requirements:
8775
8776 - Inherits ``core-image`` so that kernel modules are installed.
8777
8778 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
8779 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8780
8781 - Uses a custom Initial RAM Disk (initramfs) image with a custom
8782 installer. A normal image that you can install usually creates a
8783 single root filesystem partition. This image uses another installer that
8784 creates a specific partition layout. Not all Board Support
8785 Packages (BSPs) can use an installer. For such cases, you need to
8786 manually create the following partition layout on the target:
8787
8788 - First partition mounted under ``/boot``, labeled "boot".
8789
8790 - The main root filesystem partition where this image gets installed,
8791 which is mounted under ``/``.
8792
8793 - Another partition labeled "testrootfs" where test images get
8794 deployed.
8795
87963. *Install image:* Install the image that you just built on the target
8797 system.
8798
8799The final thing you need to do when setting :term:`TEST_TARGET` to
8800"SystemdbootTarget" is to set up the test image:
8801
88021. *Set up your local.conf file:* Make sure you have the following
8803 statements in your ``local.conf`` file::
8804
8805 IMAGE_FSTYPES += "tar.gz"
8806 INHERIT += "testimage"
8807 TEST_TARGET = "SystemdbootTarget"
8808 TEST_TARGET_IP = "192.168.2.3"
8809
88102. *Build your test image:* Use BitBake to build the image::
8811
8812 $ bitbake core-image-sato
8813
8814Power Control
8815~~~~~~~~~~~~~
8816
8817For most hardware targets other than "simpleremote", you can control
8818power:
8819
8820- You can use :term:`TEST_POWERCONTROL_CMD` together with
8821 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
8822 and does power cycling. The test code passes one argument to that
8823 command: off, on or cycle (off then on). Here is an example that
8824 could appear in your ``local.conf`` file::
8825
8826 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8827
8828 In this example, the expect
8829 script does the following:
8830
8831 .. code-block:: shell
8832
8833 ssh test@10.11.12.1 "pyctl nuc1 arg"
8834
8835 It then runs a Python script that controls power for a label called
8836 ``nuc1``.
8837
8838 .. note::
8839
8840 You need to customize :term:`TEST_POWERCONTROL_CMD` and
8841 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
8842 is that it accepts "on", "off", and "cycle" as the last argument.
8843
8844- When no command is defined, it connects to the device over SSH and
8845 uses the classic reboot command to reboot the device. Classic reboot
8846 is fine as long as the machine actually reboots (i.e. the SSH test
8847 has not failed). It is useful for scenarios where you have a simple
8848 setup, typically with a single board, and where some manual
8849 interaction is okay from time to time.
8850
8851If you have no hardware to automatically perform power control but still
8852wish to experiment with automated hardware testing, you can use the
8853``dialog-power-control`` script that shows a dialog prompting you to perform
8854the required power action. This script requires either KDialog or Zenity
8855to be installed. To use this script, set the
8856:term:`TEST_POWERCONTROL_CMD`
8857variable as follows::
8858
8859 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8860
8861Serial Console Connection
8862~~~~~~~~~~~~~~~~~~~~~~~~~
8863
8864For test target classes requiring a serial console to interact with the
8865bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8866you need to specify a command to use to connect to the serial console of
8867the target machine by using the
8868:term:`TEST_SERIALCONTROL_CMD`
8869variable and optionally the
8870:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8871variable.
8872
8873These cases could be a serial terminal program if the machine is
8874connected to a local serial port, or a ``telnet`` or ``ssh`` command
8875connecting to a remote console server. Regardless of the case, the
8876command simply needs to connect to the serial console and forward that
8877connection to standard input and output as any normal terminal program
8878does. For example, to use the picocom terminal program on serial device
8879``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
8880
8881 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8882
8883For local
8884devices where the serial port device disappears when the device reboots,
8885an additional "serdevtry" wrapper script is provided. To use this
8886wrapper, simply prefix the terminal command with
8887``${COREBASE}/scripts/contrib/serdevtry``::
8888
8889 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8890
8891Running Tests
8892-------------
8893
8894You can start the tests automatically or manually:
8895
8896- *Automatically running tests:* To run the tests automatically after
8897 the OpenEmbedded build system successfully creates an image, first
8898 set the
8899 :term:`TESTIMAGE_AUTO`
8900 variable to "1" in your ``local.conf`` file in the
8901 :term:`Build Directory`::
8902
8903 TESTIMAGE_AUTO = "1"
8904
8905 Next, build your image. If the image successfully builds, the
8906 tests run::
8907
8908 bitbake core-image-sato
8909
8910- *Manually running tests:* To manually run the tests, first globally
8911 inherit the
8912 :ref:`testimage <ref-classes-testimage*>` class
8913 by editing your ``local.conf`` file::
8914
8915 INHERIT += "testimage"
8916
8917 Next, use BitBake to run the tests::
8918
8919 bitbake -c testimage image
8920
8921All test files reside in ``meta/lib/oeqa/runtime/cases`` in the
8922:term:`Source Directory`. A test name maps
8923directly to a Python module. Each test module may contain a number of
8924individual tests. Tests are usually grouped together by the area tested
8925(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/cases/systemd.py``).
8926
8927You can add tests to any layer provided you place them in the proper
8928area and you extend :term:`BBPATH` in
8929the ``local.conf`` file as normal. Be sure that tests reside in
8930``layer/lib/oeqa/runtime/cases``.
8931
8932.. note::
8933
8934 Be sure that module names do not collide with module names used in
8935 the default set of test modules in ``meta/lib/oeqa/runtime/cases``.
8936
8937You can change the set of tests run by appending or overriding
8938:term:`TEST_SUITES` variable in
8939``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
8940for the image. Test modules named within :term:`TEST_SUITES` cannot be
8941skipped even if a test is not suitable for an image (e.g. running the
8942RPM tests on an image without ``rpm``). Appending "auto" to
8943:term:`TEST_SUITES` causes the build system to try to run all tests that are
8944suitable for the image (i.e. each test module may elect to skip itself).
8945
8946The order you list tests in :term:`TEST_SUITES` is important and influences
8947test dependencies. Consequently, tests that depend on other tests should
8948be added after the test on which they depend. For example, since the
8949``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8950"ping" in the list. The test class provides no re-ordering or dependency
8951handling.
8952
8953.. note::
8954
8955 Each module can have multiple classes with multiple test methods.
8956 And, Python ``unittest`` rules apply.
8957
8958Here are some things to keep in mind when running tests:
8959
8960- The default tests for the image are defined as::
8961
8962 DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
8963
8964- Add your own test to the list of the by using the following::
8965
8966 TEST_SUITES:append = " mytest"
8967
8968- Run a specific list of tests as follows::
8969
8970 TEST_SUITES = "test1 test2 test3"
8971
8972 Remember, order is important. Be sure to place a test that is
8973 dependent on another test later in the order.
8974
8975Exporting Tests
8976---------------
8977
8978You can export tests so that they can run independently of the build
8979system. Exporting tests is required if you want to be able to hand the
8980test execution off to a scheduler. You can only export tests that are
8981defined in :term:`TEST_SUITES`.
8982
8983If your image is already built, make sure the following are set in your
8984``local.conf`` file::
8985
8986 INHERIT += "testexport"
8987 TEST_TARGET_IP = "IP-address-for-the-test-target"
8988 TEST_SERVER_IP = "IP-address-for-the-test-server"
8989
8990You can then export the tests with the
8991following BitBake command form::
8992
8993 $ bitbake image -c testexport
8994
8995Exporting the tests places them in the
8996:term:`Build Directory` in
8997``tmp/testexport/``\ image, which is controlled by the
8998:term:`TEST_EXPORT_DIR` variable.
8999
9000You can now run the tests outside of the build environment::
9001
9002 $ cd tmp/testexport/image
9003 $ ./runexported.py testdata.json
9004
9005Here is a complete example that shows IP addresses and uses the
9006``core-image-sato`` image::
9007
9008 INHERIT += "testexport"
9009 TEST_TARGET_IP = "192.168.7.2"
9010 TEST_SERVER_IP = "192.168.7.1"
9011
9012Use BitBake to export the tests::
9013
9014 $ bitbake core-image-sato -c testexport
9015
9016Run the tests outside of
9017the build environment using the following::
9018
9019 $ cd tmp/testexport/core-image-sato
9020 $ ./runexported.py testdata.json
9021
9022Writing New Tests
9023-----------------
9024
9025As mentioned previously, all new test files need to be in the proper
9026place for the build system to find them. New tests for additional
9027functionality outside of the core should be added to the layer that adds
9028the functionality, in ``layer/lib/oeqa/runtime/cases`` (as long as
9029:term:`BBPATH` is extended in the
9030layer's ``layer.conf`` file as normal). Just remember the following:
9031
9032- Filenames need to map directly to test (module) names.
9033
9034- Do not use module names that collide with existing core tests.
9035
9036- Minimally, an empty ``__init__.py`` file must be present in the runtime
9037 directory.
9038
9039To create a new test, start by copying an existing module (e.g.
9040``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
9041code from ``meta/lib/oeqa/utils``, which are helper classes.
9042
9043.. note::
9044
9045 Structure shell commands such that you rely on them and they return a
9046 single code for success. Be aware that sometimes you will need to
9047 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
9048
9049You will notice that all test classes inherit ``oeRuntimeTest``, which
9050is found in ``meta/lib/oetest.py``. This base class offers some helper
9051attributes, which are described in the following sections:
9052
9053Class Methods
9054~~~~~~~~~~~~~
9055
9056Class methods are as follows:
9057
9058- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
9059 package list of the image, which is based on the manifest file that
9060 is generated during the ``do_rootfs`` task.
9061
9062- *hasFeature(feature):* Returns "True" if the feature is in
9063 :term:`IMAGE_FEATURES` or
9064 :term:`DISTRO_FEATURES`.
9065
9066Class Attributes
9067~~~~~~~~~~~~~~~~
9068
9069Class attributes are as follows:
9070
9071- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
9072 Otherwise, ``pscmd`` equals "ps" (busybox).
9073
9074- *tc:* The called test context, which gives access to the
9075 following attributes:
9076
9077 - *d:* The BitBake datastore, which allows you to use stuff such
9078 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
9079
9080 - *testslist and testsrequired:* Used internally. The tests
9081 do not need these.
9082
9083 - *filesdir:* The absolute path to
9084 ``meta/lib/oeqa/runtime/files``, which contains helper files for
9085 tests meant for copying on the target such as small files written
9086 in C for compilation.
9087
9088 - *target:* The target controller object used to deploy and
9089 start an image on a particular target (e.g. Qemu, SimpleRemote,
9090 and SystemdbootTarget). Tests usually use the following:
9091
9092 - *ip:* The target's IP address.
9093
9094 - *server_ip:* The host's IP address, which is usually used
9095 by the DNF test suite.
9096
9097 - *run(cmd, timeout=None):* The single, most used method.
9098 This command is a wrapper for: ``ssh root@host "cmd"``. The
9099 command returns a tuple: (status, output), which are what their
9100 names imply - the return code of "cmd" and whatever output it
9101 produces. The optional timeout argument represents the number
9102 of seconds the test should wait for "cmd" to return. If the
9103 argument is "None", the test uses the default instance's
9104 timeout period, which is 300 seconds. If the argument is "0",
9105 the test runs until the command returns.
9106
9107 - *copy_to(localpath, remotepath):*
9108 ``scp localpath root@ip:remotepath``.
9109
9110 - *copy_from(remotepath, localpath):*
9111 ``scp root@host:remotepath localpath``.
9112
9113Instance Attributes
9114~~~~~~~~~~~~~~~~~~~
9115
9116There is a single instance attribute, which is ``target``. The ``target``
9117instance attribute is identical to the class attribute of the same name,
9118which is described in the previous section. This attribute exists as
9119both an instance and class attribute so tests can use
9120``self.target.run(cmd)`` in instance methods instead of
9121``oeRuntimeTest.tc.target.run(cmd)``.
9122
9123Installing Packages in the DUT Without the Package Manager
9124----------------------------------------------------------
9125
9126When a test requires a package built by BitBake, it is possible to
9127install that package. Installing the package does not require a package
9128manager be installed in the device under test (DUT). It does, however,
9129require an SSH connection and the target must be using the
9130``sshcontrol`` class.
9131
9132.. note::
9133
9134 This method uses ``scp`` to copy files from the host to the target, which
9135 causes permissions and special attributes to be lost.
9136
9137A JSON file is used to define the packages needed by a test. This file
9138must be in the same path as the file used to define the tests.
9139Furthermore, the filename must map directly to the test module name with
9140a ``.json`` extension.
9141
9142The JSON file must include an object with the test name as keys of an
9143object or an array. This object (or array of objects) uses the following
9144data:
9145
9146- "pkg" - A mandatory string that is the name of the package to be
9147 installed.
9148
9149- "rm" - An optional boolean, which defaults to "false", that specifies
9150 to remove the package after the test.
9151
9152- "extract" - An optional boolean, which defaults to "false", that
9153 specifies if the package must be extracted from the package format.
9154 When set to "true", the package is not automatically installed into
9155 the DUT.
9156
9157Following is an example JSON file that handles test "foo" installing
9158package "bar" and test "foobar" installing packages "foo" and "bar".
9159Once the test is complete, the packages are removed from the DUT.
9160::
9161
9162 {
9163 "foo": {
9164 "pkg": "bar"
9165 },
9166 "foobar": [
9167 {
9168 "pkg": "foo",
9169 "rm": true
9170 },
9171 {
9172 "pkg": "bar",
9173 "rm": true
9174 }
9175 ]
9176 }
9177
9178Debugging Tools and Techniques
9179==============================
9180
9181The exact method for debugging build failures depends on the nature of
9182the problem and on the system's area from which the bug originates.
9183Standard debugging practices such as comparison against the last known
9184working version with examination of the changes and the re-application
9185of steps to identify the one causing the problem are valid for the Yocto
9186Project just as they are for any other system. Even though it is
9187impossible to detail every possible potential failure, this section
9188provides some general tips to aid in debugging given a variety of
9189situations.
9190
9191.. note::
9192
9193 A useful feature for debugging is the error reporting tool.
9194 Configuring the Yocto Project to use this tool causes the
9195 OpenEmbedded build system to produce error reporting commands as part
9196 of the console output. You can enter the commands after the build
9197 completes to log error information into a common database, that can
9198 help you figure out what might be going wrong. For information on how
9199 to enable and use this feature, see the
9200 ":ref:`dev-manual/common-tasks:using the error reporting tool`"
9201 section.
9202
9203The following list shows the debugging topics in the remainder of this
9204section:
9205
9206- ":ref:`dev-manual/common-tasks:viewing logs from failed tasks`" describes
9207 how to find and view logs from tasks that failed during the build
9208 process.
9209
9210- ":ref:`dev-manual/common-tasks:viewing variable values`" describes how to
9211 use the BitBake ``-e`` option to examine variable values after a
9212 recipe has been parsed.
9213
9214- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
9215 describes how to use the ``oe-pkgdata-util`` utility to query
9216 :term:`PKGDATA_DIR` and
9217 display package-related information for built packages.
9218
9219- ":ref:`dev-manual/common-tasks:viewing dependencies between recipes and tasks`"
9220 describes how to use the BitBake ``-g`` option to display recipe
9221 dependency information used during the build.
9222
9223- ":ref:`dev-manual/common-tasks:viewing task variable dependencies`" describes
9224 how to use the ``bitbake-dumpsig`` command in conjunction with key
9225 subdirectories in the
9226 :term:`Build Directory` to determine
9227 variable dependencies.
9228
9229- ":ref:`dev-manual/common-tasks:running specific tasks`" describes
9230 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
9231 to run specific tasks in the build chain. It can be useful to run
9232 tasks "out-of-order" when trying isolate build issues.
9233
9234- ":ref:`dev-manual/common-tasks:general bitbake problems`" describes how
9235 to use BitBake's ``-D`` debug output option to reveal more about what
9236 BitBake is doing during the build.
9237
9238- ":ref:`dev-manual/common-tasks:building with no dependencies`"
9239 describes how to use the BitBake ``-b`` option to build a recipe
9240 while ignoring dependencies.
9241
9242- ":ref:`dev-manual/common-tasks:recipe logging mechanisms`"
9243 describes how to use the many recipe logging functions to produce
9244 debugging output and report errors and warnings.
9245
9246- ":ref:`dev-manual/common-tasks:debugging parallel make races`"
9247 describes how to debug situations where the build consists of several
9248 parts that are run simultaneously and when the output or result of
9249 one part is not ready for use with a different part of the build that
9250 depends on that output.
9251
9252- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) remotely`"
9253 describes how to use GDB to allow you to examine running programs, which can
9254 help you fix problems.
9255
9256- ":ref:`dev-manual/common-tasks:debugging with the gnu project debugger (gdb) on the target`"
9257 describes how to use GDB directly on target hardware for debugging.
9258
9259- ":ref:`dev-manual/common-tasks:other debugging tips`" describes
9260 miscellaneous debugging tips that can be useful.
9261
9262Viewing Logs from Failed Tasks
9263------------------------------
9264
9265You can find the log for a task in the file
9266``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
9267For example, the log for the
9268:ref:`ref-tasks-compile` task of the
9269QEMU minimal image for the x86 machine (``qemux86``) might be in
9270``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
9271To see the commands :term:`BitBake` ran
9272to generate a log, look at the corresponding ``run.do_``\ `taskname` file
9273in the same directory.
9274
9275``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
9276links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
9277``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
9278when it ran. The symlinks always point to the files corresponding to the
9279most recent run.
9280
9281Viewing Variable Values
9282-----------------------
9283
9284Sometimes you need to know the value of a variable as a result of
9285BitBake's parsing step. This could be because some unexpected behavior
9286occurred in your project. Perhaps an attempt to :ref:`modify a variable
9287<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
9288variables>` did not work out as expected.
9289
9290BitBake's ``-e`` option is used to display variable values after
9291parsing. The following command displays the variable values after the
9292configuration files (i.e. ``local.conf``, ``bblayers.conf``,
9293``bitbake.conf`` and so forth) have been parsed::
9294
9295 $ bitbake -e
9296
9297The following command displays variable values after a specific recipe has
9298been parsed. The variables include those from the configuration as well::
9299
9300 $ bitbake -e recipename
9301
9302.. note::
9303
9304 Each recipe has its own private set of variables (datastore).
9305 Internally, after parsing the configuration, a copy of the resulting
9306 datastore is made prior to parsing each recipe. This copying implies
9307 that variables set in one recipe will not be visible to other
9308 recipes.
9309
9310 Likewise, each task within a recipe gets a private datastore based on
9311 the recipe datastore, which means that variables set within one task
9312 will not be visible to other tasks.
9313
9314In the output of ``bitbake -e``, each variable is preceded by a
9315description of how the variable got its value, including temporary
9316values that were later overridden. This description also includes
9317variable flags (varflags) set on the variable. The output can be very
9318helpful during debugging.
9319
9320Variables that are exported to the environment are preceded by
9321``export`` in the output of ``bitbake -e``. See the following example::
9322
9323 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9324
9325In addition to variable values, the output of the ``bitbake -e`` and
9326``bitbake -e`` recipe commands includes the following information:
9327
9328- The output starts with a tree listing all configuration files and
9329 classes included globally, recursively listing the files they include
9330 or inherit in turn. Much of the behavior of the OpenEmbedded build
9331 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
9332 implemented in the
9333 :ref:`base <ref-classes-base>` class and the
9334 classes it inherits, rather than being built into BitBake itself.
9335
9336- After the variable values, all functions appear in the output. For
9337 shell functions, variables referenced within the function body are
9338 expanded. If a function has been modified using overrides or using
9339 override-style operators like ``:append`` and ``:prepend``, then the
9340 final assembled function body appears in the output.
9341
9342Viewing Package Information with ``oe-pkgdata-util``
9343----------------------------------------------------
9344
9345You can use the ``oe-pkgdata-util`` command-line utility to query
9346:term:`PKGDATA_DIR` and display
9347various package-related information. When you use the utility, you must
9348use it to view information on packages that have already been built.
9349
9350Following are a few of the available ``oe-pkgdata-util`` subcommands.
9351
9352.. note::
9353
9354 You can use the standard \* and ? globbing wildcards as part of
9355 package names and paths.
9356
9357- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9358 that have been built, optionally limiting the match to packages that
9359 match pattern.
9360
9361- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9362 files and directories contained in the given packages.
9363
9364 .. note::
9365
9366 A different way to view the contents of a package is to look at
9367 the
9368 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9369 directory of the recipe that generates the package. This directory
9370 is created by the
9371 :ref:`ref-tasks-package` task
9372 and has one subdirectory for each package the recipe generates,
9373 which contains the files stored in that package.
9374
9375 If you want to inspect the ``${WORKDIR}/packages-split``
9376 directory, make sure that
9377 :ref:`rm_work <ref-classes-rm-work>` is not
9378 enabled when you build the recipe.
9379
9380- ``oe-pkgdata-util find-path path ...``: Lists the names of
9381 the packages that contain the given paths. For example, the following
9382 tells us that ``/usr/share/man/man1/make.1`` is contained in the
9383 ``make-doc`` package::
9384
9385 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
9386 make-doc: /usr/share/man/man1/make.1
9387
9388- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9389 of the recipes that produce the given packages.
9390
9391For more information on the ``oe-pkgdata-util`` command, use the help
9392facility::
9393
9394 $ oe-pkgdata-util --help
9395 $ oe-pkgdata-util subcommand --help
9396
9397Viewing Dependencies Between Recipes and Tasks
9398----------------------------------------------
9399
9400Sometimes it can be hard to see why BitBake wants to build other recipes
9401before the one you have specified. Dependency information can help you
9402understand why a recipe is built.
9403
9404To generate dependency information for a recipe, run the following
9405command::
9406
9407 $ bitbake -g recipename
9408
9409This command writes the following files in the current directory:
9410
9411- ``pn-buildlist``: A list of recipes/targets involved in building
9412 `recipename`. "Involved" here means that at least one task from the
9413 recipe needs to run when building `recipename` from scratch. Targets
9414 that are in
9415 :term:`ASSUME_PROVIDED`
9416 are not listed.
9417
9418- ``task-depends.dot``: A graph showing dependencies between tasks.
9419
9420The graphs are in
9421`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__
9422format and can be converted to images (e.g. using the ``dot`` tool from
9423`Graphviz <https://www.graphviz.org/>`__).
9424
9425.. note::
9426
9427 - DOT files use a plain text format. The graphs generated using the
9428 ``bitbake -g`` command are often so large as to be difficult to
9429 read without special pruning (e.g. with Bitbake's ``-I`` option)
9430 and processing. Despite the form and size of the graphs, the
9431 corresponding ``.dot`` files can still be possible to read and
9432 provide useful information.
9433
9434 As an example, the ``task-depends.dot`` file contains lines such
9435 as the following::
9436
9437 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9438
9439 The above example line reveals that the
9440 :ref:`ref-tasks-configure`
9441 task in ``libxslt`` depends on the
9442 :ref:`ref-tasks-populate_sysroot`
9443 task in ``libxml2``, which is a normal
9444 :term:`DEPENDS` dependency
9445 between the two recipes.
9446
9447 - For an example of how ``.dot`` files can be processed, see the
9448 ``scripts/contrib/graph-tool`` Python script, which finds and
9449 displays paths between graph nodes.
9450
9451You can use a different method to view dependency information by using
9452the following command::
9453
9454 $ bitbake -g -u taskexp recipename
9455
9456This command
9457displays a GUI window from which you can view build-time and runtime
9458dependencies for the recipes involved in building recipename.
9459
9460Viewing Task Variable Dependencies
9461----------------------------------
9462
9463As mentioned in the
9464":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9465User Manual, BitBake tries to automatically determine what variables a
9466task depends on so that it can rerun the task if any values of the
9467variables change. This determination is usually reliable. However, if
9468you do things like construct variable names at runtime, then you might
9469have to manually declare dependencies on those variables using
9470``vardeps`` as described in the
9471":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9472User Manual.
9473
9474If you are unsure whether a variable dependency is being picked up
9475automatically for a given task, you can list the variable dependencies
9476BitBake has determined by doing the following:
9477
94781. Build the recipe containing the task::
9479
9480 $ bitbake recipename
9481
94822. Inside the :term:`STAMPS_DIR`
9483 directory, find the signature data (``sigdata``) file that
9484 corresponds to the task. The ``sigdata`` files contain a pickled
9485 Python database of all the metadata that went into creating the input
9486 checksum for the task. As an example, for the
9487 :ref:`ref-tasks-fetch` task of the
9488 ``db`` recipe, the ``sigdata`` file might be found in the following
9489 location::
9490
9491 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9492
9493 For tasks that are accelerated through the shared state
9494 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
9495 additional ``siginfo`` file is written into
9496 :term:`SSTATE_DIR` along with
9497 the cached task output. The ``siginfo`` files contain exactly the
9498 same information as ``sigdata`` files.
9499
95003. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
9501 is an example::
9502
9503 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9504
9505 In the output of the above command, you will find a line like the
9506 following, which lists all the (inferred) variable dependencies for
9507 the task. This list also includes indirect dependencies from
9508 variables depending on other variables, recursively.
9509 ::
9510
9511 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9512
9513 .. note::
9514
9515 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9516 These functions in turn depend on the variables they reference.
9517
9518 The output of ``bitbake-dumpsig`` also includes the value each
9519 variable had, a list of dependencies for each variable, and
9520 :term:`BB_BASEHASH_IGNORE_VARS`
9521 information.
9522
9523There is also a ``bitbake-diffsigs`` command for comparing two
9524``siginfo`` or ``sigdata`` files. This command can be helpful when
9525trying to figure out what changed between two versions of a task. If you
9526call ``bitbake-diffsigs`` with just one file, the command behaves like
9527``bitbake-dumpsig``.
9528
9529You can also use BitBake to dump out the signature construction
9530information without executing tasks by using either of the following
9531BitBake command-line options::
9532
9533 ‐‐dump-signatures=SIGNATURE_HANDLER
9534 -S SIGNATURE_HANDLER
9535
9536
9537.. note::
9538
9539 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9540 dump only the signature or compare the dumped signature with the cached one,
9541 respectively.
9542
9543Using BitBake with either of these options causes BitBake to dump out
9544``sigdata`` files in the ``stamps`` directory for every task it would
9545have executed instead of building the specified target package.
9546
9547Viewing Metadata Used to Create the Input Signature of a Shared State Task
9548--------------------------------------------------------------------------
9549
9550Seeing what metadata went into creating the input signature of a shared
9551state (sstate) task can be a useful debugging aid. This information is
9552available in signature information (``siginfo``) files in
9553:term:`SSTATE_DIR`. For
9554information on how to view and interpret information in ``siginfo``
9555files, see the
9556":ref:`dev-manual/common-tasks:viewing task variable dependencies`" section.
9557
9558For conceptual information on shared state, see the
9559":ref:`overview-manual/concepts:shared state`"
9560section in the Yocto Project Overview and Concepts Manual.
9561
9562Invalidating Shared State to Force a Task to Run
9563------------------------------------------------
9564
9565The OpenEmbedded build system uses
9566:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
9567:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
9568rebuilding tasks. Collectively, this scheme is known as "shared state
9569code".
9570
9571As with all schemes, this one has some drawbacks. It is possible that
9572you could make implicit changes to your code that the checksum
9573calculations do not take into account. These implicit changes affect a
9574task's output but do not trigger the shared state code into rebuilding a
9575recipe. Consider an example during which a tool changes its output.
9576Assume that the output of ``rpmdeps`` changes. The result of the change
9577should be that all the ``package`` and ``package_write_rpm`` shared
9578state cache items become invalid. However, because the change to the
9579output is external to the code and therefore implicit, the associated
9580shared state cache items do not become invalidated. In this case, the
9581build process uses the cached items rather than running the task again.
9582Obviously, these types of implicit changes can cause problems.
9583
9584To avoid these problems during the build, you need to understand the
9585effects of any changes you make. Realize that changes you make directly
9586to a function are automatically factored into the checksum calculation.
9587Thus, these explicit changes invalidate the associated area of shared
9588state cache. However, you need to be aware of any implicit changes that
9589are not obvious changes to the code and could affect the output of a
9590given task.
9591
9592When you identify an implicit change, you can easily take steps to
9593invalidate the cache and force the tasks to run. The steps you can take
9594are as simple as changing a function's comments in the source code. For
9595example, to invalidate package shared state files, change the comment
9596statements of
9597:ref:`ref-tasks-package` or the
9598comments of one of the functions it calls. Even though the change is
9599purely cosmetic, it causes the checksum to be recalculated and forces
9600the build system to run the task again.
9601
9602.. note::
9603
9604 For an example of a commit that makes a cosmetic change to invalidate
9605 shared state, see this
9606 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
9607
9608Running Specific Tasks
9609----------------------
9610
9611Any given recipe consists of a set of tasks. The standard BitBake
9612behavior in most cases is: ``do_fetch``, ``do_unpack``, ``do_patch``,
9613``do_configure``, ``do_compile``, ``do_install``, ``do_package``,
9614``do_package_write_*``, and ``do_build``. The default task is
9615``do_build`` and any tasks on which it depends build first. Some tasks,
9616such as ``do_devshell``, are not part of the default build chain. If you
9617wish to run a task that is not part of the default build chain, you can
9618use the ``-c`` option in BitBake. Here is an example::
9619
9620 $ bitbake matchbox-desktop -c devshell
9621
9622The ``-c`` option respects task dependencies, which means that all other
9623tasks (including tasks from other recipes) that the specified task
9624depends on will be run before the task. Even when you manually specify a
9625task to run with ``-c``, BitBake will only run the task if it considers
9626it "out of date". See the
9627":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
9628section in the Yocto Project Overview and Concepts Manual for how
9629BitBake determines whether a task is "out of date".
9630
9631If you want to force an up-to-date task to be rerun (e.g. because you
9632made manual modifications to the recipe's
9633:term:`WORKDIR` that you want to try
9634out), then you can use the ``-f`` option.
9635
9636.. note::
9637
9638 The reason ``-f`` is never required when running the
9639 :ref:`ref-tasks-devshell` task is because the
9640 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
9641 variable flag is already set for the task.
9642
9643The following example shows one way you can use the ``-f`` option::
9644
9645 $ bitbake matchbox-desktop
9646 .
9647 .
9648 make some changes to the source code in the work directory
9649 .
9650 .
9651 $ bitbake matchbox-desktop -c compile -f
9652 $ bitbake matchbox-desktop
9653
9654This sequence first builds and then recompiles ``matchbox-desktop``. The
9655last command reruns all tasks (basically the packaging tasks) after the
9656compile. BitBake recognizes that the ``do_compile`` task was rerun and
9657therefore understands that the other tasks also need to be run again.
9658
9659Another, shorter way to rerun a task and all
9660:ref:`ref-manual/tasks:normal recipe build tasks`
9661that depend on it is to use the ``-C`` option.
9662
9663.. note::
9664
9665 This option is upper-cased and is separate from the ``-c``
9666 option, which is lower-cased.
9667
9668Using this option invalidates the given task and then runs the
9669:ref:`ref-tasks-build` task, which is
9670the default task if no task is given, and the tasks on which it depends.
9671You could replace the final two commands in the previous example with
9672the following single command::
9673
9674 $ bitbake matchbox-desktop -C compile
9675
9676Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9677the input checksum of the specified task. This tainting indirectly
9678causes the task and its dependent tasks to be rerun through the normal
9679task dependency mechanisms.
9680
9681.. note::
9682
9683 BitBake explicitly keeps track of which tasks have been tainted in
9684 this fashion, and will print warnings such as the following for
9685 builds involving such tasks:
9686
9687 .. code-block:: none
9688
9689 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9690
9691
9692 The purpose of the warning is to let you know that the work directory
9693 and build output might not be in the clean state they would be in for
9694 a "normal" build, depending on what actions you took. To get rid of
9695 such warnings, you can remove the work directory and rebuild the
9696 recipe, as follows::
9697
9698 $ bitbake matchbox-desktop -c clean
9699 $ bitbake matchbox-desktop
9700
9701
9702You can view a list of tasks in a given package by running the
9703``do_listtasks`` task as follows::
9704
9705 $ bitbake matchbox-desktop -c listtasks
9706
9707The results appear as output to the console and are also in
9708the file ``${WORKDIR}/temp/log.do_listtasks``.
9709
9710General BitBake Problems
9711------------------------
9712
9713You can see debug output from BitBake by using the ``-D`` option. The
9714debug output gives more information about what BitBake is doing and the
9715reason behind it. Each ``-D`` option you use increases the logging
9716level. The most common usage is ``-DDD``.
9717
9718The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
9719chose a certain version of a package or why BitBake picked a certain
9720provider. This command could also help you in a situation where you
9721think BitBake did something unexpected.
9722
9723Building with No Dependencies
9724-----------------------------
9725
9726To build a specific recipe (``.bb`` file), you can use the following
9727command form::
9728
9729 $ bitbake -b somepath/somerecipe.bb
9730
9731This command form does
9732not check for dependencies. Consequently, you should use it only when
9733you know existing dependencies have been met.
9734
9735.. note::
9736
9737 You can also specify fragments of the filename. In this case, BitBake
9738 checks for a unique match.
9739
9740Recipe Logging Mechanisms
9741-------------------------
9742
9743The Yocto Project provides several logging functions for producing
9744debugging output and reporting errors and warnings. For Python
9745functions, the following logging functions are available. All of these functions
9746log to ``${T}/log.do_``\ `task`, and can also log to standard output
9747(stdout) with the right settings:
9748
9749- ``bb.plain(msg)``: Writes msg as is to the log while also
9750 logging to stdout.
9751
9752- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9753 stdout if BitBake is called with "-v".
9754
9755- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9756 log. Also logs to stdout if the log level is greater than or equal to
9757 level. See the ":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`" option
9758 in the BitBake User Manual for more information.
9759
9760- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9761 logging to stdout.
9762
9763- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9764 logging to standard out (stdout).
9765
9766 .. note::
9767
9768 Calling this function does not cause the task to fail.
9769
9770- ``bb.fatal(msg)``: This logging function is similar to
9771 ``bb.error(msg)`` but also causes the calling task to fail.
9772
9773 .. note::
9774
9775 ``bb.fatal()`` raises an exception, which means you do not need to put a
9776 "return" statement after the function.
9777
9778The same logging functions are also available in shell functions, under
9779the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9780and ``bbfatal``. The
9781:ref:`logging <ref-classes-logging>` class
9782implements these functions. See that class in the ``meta/classes``
9783folder of the :term:`Source Directory` for information.
9784
9785Logging With Python
9786~~~~~~~~~~~~~~~~~~~
9787
9788When creating recipes using Python and inserting code that handles build
9789logs, keep in mind the goal is to have informative logs while keeping
9790the console as "silent" as possible. Also, if you want status messages
9791in the log, use the "debug" loglevel.
9792
9793Following is an example written in Python. The code handles logging for
9794a function that determines the number of tasks needed to be run. See the
9795":ref:`ref-tasks-listtasks`"
9796section for additional information::
9797
9798 python do_listtasks() {
9799 bb.debug(2, "Starting to figure out the task list")
9800 if noteworthy_condition:
9801 bb.note("There are 47 tasks to run")
9802 bb.debug(2, "Got to point xyz")
9803 if warning_trigger:
9804 bb.warn("Detected warning_trigger, this might be a problem later.")
9805 if recoverable_error:
9806 bb.error("Hit recoverable_error, you really need to fix this!")
9807 if fatal_error:
9808 bb.fatal("fatal_error detected, unable to print the task list")
9809 bb.plain("The tasks present are abc")
9810 bb.debug(2, "Finished figuring out the tasklist")
9811 }
9812
9813Logging With Bash
9814~~~~~~~~~~~~~~~~~
9815
9816When creating recipes using Bash and inserting code that handles build
9817logs, you have the same goals - informative with minimal console output.
9818The syntax you use for recipes written in Bash is similar to that of
9819recipes written in Python described in the previous section.
9820
9821Following is an example written in Bash. The code logs the progress of
9822the ``do_my_function`` function.
9823::
9824
9825 do_my_function() {
9826 bbdebug 2 "Running do_my_function"
9827 if [ exceptional_condition ]; then
9828 bbnote "Hit exceptional_condition"
9829 fi
9830 bbdebug 2 "Got to point xyz"
9831 if [ warning_trigger ]; then
9832 bbwarn "Detected warning_trigger, this might cause a problem later."
9833 fi
9834 if [ recoverable_error ]; then
9835 bberror "Hit recoverable_error, correcting"
9836 fi
9837 if [ fatal_error ]; then
9838 bbfatal "fatal_error detected"
9839 fi
9840 bbdebug 2 "Completed do_my_function"
9841 }
9842
9843
9844Debugging Parallel Make Races
9845-----------------------------
9846
9847A parallel ``make`` race occurs when the build consists of several parts
9848that are run simultaneously and a situation occurs when the output or
9849result of one part is not ready for use with a different part of the
9850build that depends on that output. Parallel make races are annoying and
9851can sometimes be difficult to reproduce and fix. However, there are some simple
9852tips and tricks that can help you debug and fix them. This section
9853presents a real-world example of an error encountered on the Yocto
9854Project autobuilder and the process used to fix it.
9855
9856.. note::
9857
9858 If you cannot properly fix a ``make`` race condition, you can work around it
9859 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
9860 variables.
9861
9862The Failure
9863~~~~~~~~~~~
9864
9865For this example, assume that you are building an image that depends on
9866the "neard" package. And, during the build, BitBake runs into problems
9867and creates the following output.
9868
9869.. note::
9870
9871 This example log file has longer lines artificially broken to make
9872 the listing easier to read.
9873
9874If you examine the output or the log file, you see the failure during
9875``make``:
9876
9877.. code-block:: none
9878
9879 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9880 | DEBUG: Executing shell function do_compile
9881 | NOTE: make -j 16
9882 | make --no-print-directory all-am
9883 | /bin/mkdir -p include/near
9884 | /bin/mkdir -p include/near
9885 | /bin/mkdir -p include/near
9886 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9887 0.14-r0/neard-0.14/include/types.h include/near/types.h
9888 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9889 0.14-r0/neard-0.14/include/log.h include/near/log.h
9890 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9891 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9892 | /bin/mkdir -p include/near
9893 | /bin/mkdir -p include/near
9894 | /bin/mkdir -p include/near
9895 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9896 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
9897 | /bin/mkdir -p include/near
9898 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9899 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
9900 | /bin/mkdir -p include/near
9901 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9902 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
9903 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9904 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
9905 | /bin/mkdir -p include/near
9906 | /bin/mkdir -p include/near
9907 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9908 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
9909 | /bin/mkdir -p include/near
9910 | /bin/mkdir -p include/near
9911 | /bin/mkdir -p include/near
9912 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9913 0.14-r0/neard-0.14/include/device.h include/near/device.h
9914 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9915 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
9916 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9917 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
9918 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9919 0.14-r0/neard-0.14/include/version.h include/near/version.h
9920 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9921 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
9922 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
9923 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/
9924 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
9925 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
9926 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
9927 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/
9928 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/
9929 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
9930 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
9931 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
9932 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
9933 -o tools/snep-send.o tools/snep-send.c
9934 | In file included from tools/snep-send.c:16:0:
9935 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9936 | #include <near/dbus.h>
9937 | ^
9938 | compilation terminated.
9939 | make[1]: *** [tools/snep-send.o] Error 1
9940 | make[1]: *** Waiting for unfinished jobs....
9941 | make: *** [all] Error 2
9942 | ERROR: oe_runmake failed
9943
9944Reproducing the Error
9945~~~~~~~~~~~~~~~~~~~~~
9946
9947Because race conditions are intermittent, they do not manifest
9948themselves every time you do the build. In fact, most times the build
9949will complete without problems even though the potential race condition
9950exists. Thus, once the error surfaces, you need a way to reproduce it.
9951
9952In this example, compiling the "neard" package is causing the problem.
9953So the first thing to do is build "neard" locally. Before you start the
9954build, set the
9955:term:`PARALLEL_MAKE` variable
9956in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
9957high value for :term:`PARALLEL_MAKE` increases the chances of the race
9958condition showing up::
9959
9960 $ bitbake neard
9961
9962Once the local build for "neard" completes, start a ``devshell`` build::
9963
9964 $ bitbake neard -c devshell
9965
9966For information on how to use a ``devshell``, see the
9967":ref:`dev-manual/common-tasks:using a development shell`" section.
9968
9969In the ``devshell``, do the following::
9970
9971 $ make clean
9972 $ make tools/snep-send.o
9973
9974The ``devshell`` commands cause the failure to clearly
9975be visible. In this case, there is a missing dependency for the ``neard``
9976Makefile target. Here is some abbreviated, sample output with the
9977missing dependency clearly visible at the end::
9978
9979 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
9980 .
9981 .
9982 .
9983 tools/snep-send.c
9984 In file included from tools/snep-send.c:16:0:
9985 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9986 #include <near/dbus.h>
9987 ^
9988 compilation terminated.
9989 make: *** [tools/snep-send.o] Error 1
9990 $
9991
9992
9993Creating a Patch for the Fix
9994~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9995
9996Because there is a missing dependency for the Makefile target, you need
9997to patch the ``Makefile.am`` file, which is generated from
9998``Makefile.in``. You can use Quilt to create the patch::
9999
10000 $ quilt new parallelmake.patch
10001 Patch patches/parallelmake.patch is now on top
10002 $ quilt add Makefile.am
10003 File Makefile.am added to patch patches/parallelmake.patch
10004
10005For more information on using Quilt, see the
10006":ref:`dev-manual/common-tasks:using quilt in your workflow`" section.
10007
10008At this point you need to make the edits to ``Makefile.am`` to add the
10009missing dependency. For our example, you have to add the following line
10010to the file::
10011
10012 tools/snep-send.$(OBJEXT): include/near/dbus.h
10013
10014Once you have edited the file, use the ``refresh`` command to create the
10015patch::
10016
10017 $ quilt refresh
10018 Refreshed patch patches/parallelmake.patch
10019
10020Once the patch file is created, you need to add it back to the originating
10021recipe folder. Here is an example assuming a top-level
10022:term:`Source Directory` named ``poky``::
10023
10024 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
10025
10026The final thing you need to do to implement the fix in the build is to
10027update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
10028:term:`SRC_URI` statement includes
10029the patch file. The recipe file is in the folder above the patch. Here
10030is what the edited :term:`SRC_URI` statement would look like::
10031
10032 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
10033 file://neard.in \
10034 file://neard.service.in \
10035 file://parallelmake.patch \
10036 "
10037
10038With the patch complete and moved to the correct folder and the
10039:term:`SRC_URI` statement updated, you can exit the ``devshell``::
10040
10041 $ exit
10042
10043Testing the Build
10044~~~~~~~~~~~~~~~~~
10045
10046With everything in place, you can get back to trying the build again
10047locally::
10048
10049 $ bitbake neard
10050
10051This build should succeed.
10052
10053Now you can open up a ``devshell`` again and repeat the clean and make
10054operations as follows::
10055
10056 $ bitbake neard -c devshell
10057 $ make clean
10058 $ make tools/snep-send.o
10059
10060The build should work without issue.
10061
10062As with all solved problems, if they originated upstream, you need to
10063submit the fix for the recipe in OE-Core and upstream so that the
10064problem is taken care of at its source. See the
10065:doc:`../contributor-guide/submit-changes` section for more information.
10066
10067Debugging With the GNU Project Debugger (GDB) Remotely
10068------------------------------------------------------
10069
10070GDB allows you to examine running programs, which in turn helps you to
10071understand and fix problems. It also allows you to perform post-mortem
10072style analysis of program crashes. GDB is available as a package within
10073the Yocto Project and is installed in SDK images by default. See the
10074":ref:`ref-manual/images:Images`" chapter in the Yocto
10075Project Reference Manual for a description of these images. You can find
10076information on GDB at https://sourceware.org/gdb/.
10077
10078.. note::
10079
10080 For best results, install debug (``-dbg``) packages for the applications you
10081 are going to debug. Doing so makes extra debug symbols available that give
10082 you more meaningful output.
10083
10084Sometimes, due to memory or disk space constraints, it is not possible
10085to use GDB directly on the remote target to debug applications. These
10086constraints arise because GDB needs to load the debugging information
10087and the binaries of the process being debugged. Additionally, GDB needs
10088to perform many computations to locate information such as function
10089names, variable names and values, stack traces and so forth - even
10090before starting the debugging process. These extra computations place
10091more load on the target system and can alter the characteristics of the
10092program being debugged.
10093
10094To help get past the previously mentioned constraints, there are two
10095methods you can use: running a debuginfod server and using gdbserver.
10096
10097Using the debuginfod server method
10098~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10099
10100``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
10101Running a ``debuginfod`` server makes debug symbols readily available,
10102which means you don't need to download debugging information
10103and the binaries of the process being debugged. You can just fetch
10104debug symbols from the server.
10105
10106To run a ``debuginfod`` server, you need to do the following:
10107
10108- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
10109 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
10110 If not, set in your distro config file or in ``local.conf``::
10111
10112 DISTRO_FEATURES:append = " debuginfod"
10113
10114 This distro feature enables the server and client library in ``elfutils``,
10115 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
10116
10117- Run the following commands to launch the ``debuginfod`` server on the host::
10118
10119 $ oe-debuginfod
10120
10121- To use ``debuginfod`` on the target, you need to know the ip:port where
10122 ``debuginfod`` is listening on the host (port defaults to 8002), and export
10123 that into the shell environment, for example in ``qemu``::
10124
10125 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
10126
10127- Then debug info fetching should simply work when running the target ``gdb``,
10128 ``readelf`` or ``objdump``, for example::
10129
10130 root@qemux86-64:~# gdb /bin/cat
10131 ...
10132 Reading symbols from /bin/cat...
10133 Downloading separate debug info for /bin/cat...
10134 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
10135
10136- It's also possible to use ``debuginfod-find`` to just query the server::
10137
10138 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
10139 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
10140
10141
10142Using the gdbserver method
10143~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10144
10145gdbserver, which runs on the remote target and does not load any
10146debugging information from the debugged process. Instead, a GDB instance
10147processes the debugging information that is run on a remote computer -
10148the host GDB. The host GDB then sends control commands to gdbserver to
10149make it stop or start the debugged program, as well as read or write
10150memory regions of that debugged program. All the debugging information
10151loaded and processed as well as all the heavy debugging is done by the
10152host GDB. Offloading these processes gives the gdbserver running on the
10153target a chance to remain small and fast.
10154
10155Because the host GDB is responsible for loading the debugging
10156information and for doing the necessary processing to make actual
10157debugging happen, you have to make sure the host can access the
10158unstripped binaries complete with their debugging information and also
10159be sure the target is compiled with no optimizations. The host GDB must
10160also have local access to all the libraries used by the debugged
10161program. Because gdbserver does not need any local debugging
10162information, the binaries on the remote target can remain stripped.
10163However, the binaries must also be compiled without optimization so they
10164match the host's binaries.
10165
10166To remain consistent with GDB documentation and terminology, the binary
10167being debugged on the remote target machine is referred to as the
10168"inferior" binary. For documentation on GDB see the `GDB
10169site <https://sourceware.org/gdb/documentation/>`__.
10170
10171The following steps show you how to debug using the GNU project
10172debugger.
10173
101741. *Configure your build system to construct the companion debug
10175 filesystem:*
10176
10177 In your ``local.conf`` file, set the following::
10178
10179 IMAGE_GEN_DEBUGFS = "1"
10180 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
10181
10182 These options cause the
10183 OpenEmbedded build system to generate a special companion filesystem
10184 fragment, which contains the matching source and debug symbols to
10185 your deployable filesystem. The build system does this by looking at
10186 what is in the deployed filesystem, and pulling the corresponding
10187 ``-dbg`` packages.
10188
10189 The companion debug filesystem is not a complete filesystem, but only
10190 contains the debug fragments. This filesystem must be combined with
10191 the full filesystem for debugging. Subsequent steps in this procedure
10192 show how to combine the partial filesystem with the full filesystem.
10193
101942. *Configure the system to include gdbserver in the target filesystem:*
10195
10196 Make the following addition in either your ``local.conf`` file or in
10197 an image recipe::
10198
10199 IMAGE_INSTALL:append = " gdbserver"
10200
10201 The change makes
10202 sure the ``gdbserver`` package is included.
10203
102043. *Build the environment:*
10205
10206 Use the following command to construct the image and the companion
10207 Debug Filesystem::
10208
10209 $ bitbake image
10210
10211 Build the cross GDB component and
10212 make it available for debugging. Build the SDK that matches the
10213 image. Building the SDK is best for a production build that can be
10214 used later for debugging, especially during long term maintenance::
10215
10216 $ bitbake -c populate_sdk image
10217
10218 Alternatively, you can build the minimal toolchain components that
10219 match the target. Doing so creates a smaller than typical SDK and
10220 only contains a minimal set of components with which to build simple
10221 test applications, as well as run the debugger::
10222
10223 $ bitbake meta-toolchain
10224
10225 A final method is to build Gdb itself within the build system::
10226
10227 $ bitbake gdb-cross-<architecture>
10228
10229 Doing so produces a temporary copy of
10230 ``cross-gdb`` you can use for debugging during development. While
10231 this is the quickest approach, the two previous methods in this step
10232 are better when considering long-term maintenance strategies.
10233
10234 .. note::
10235
10236 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
10237 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
10238 actual name you want to use.
10239
102404. *Set up the* ``debugfs``\ *:*
10241
10242 Run the following commands to set up the ``debugfs``::
10243
10244 $ mkdir debugfs
10245 $ cd debugfs
10246 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
10247 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2
10248
102495. *Set up GDB:*
10250
10251 Install the SDK (if you built one) and then source the correct
10252 environment file. Sourcing the environment file puts the SDK in your
10253 ``PATH`` environment variable.
10254
10255 If you are using the build system, Gdb is located in
10256 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
10257
102586. *Boot the target:*
10259
10260 For information on how to run QEMU, see the `QEMU
10261 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
10262
10263 .. note::
10264
10265 Be sure to verify that your host can access the target via TCP.
10266
102677. *Debug a program:*
10268
10269 Debugging a program involves running gdbserver on the target and then
10270 running Gdb on the host. The example in this step debugs ``gzip``:
10271
10272 .. code-block:: shell
10273
10274 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
10275
10276 For
10277 additional gdbserver options, see the `GDB Server
10278 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
10279
10280 After running gdbserver on the target, you need to run Gdb on the
10281 host and configure it and connect to the target. Use these commands::
10282
10283 $ cd directory-holding-the-debugfs-directory
10284 $ arch-gdb
10285 (gdb) set sysroot debugfs
10286 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
10287 (gdb) target remote IP-of-target:1234
10288
10289 At this
10290 point, everything should automatically load (i.e. matching binaries,
10291 symbols and headers).
10292
10293 .. note::
10294
10295 The Gdb ``set`` commands in the previous example can be placed into the
10296 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
10297 commands are in that file.
10298
102998. *Deploying without a full image rebuild:*
10300
10301 In many cases, during development you want a quick method to deploy a
10302 new binary to the target and debug it, without waiting for a full
10303 image build.
10304
10305 One approach to solving this situation is to just build the component
10306 you want to debug. Once you have built the component, copy the
10307 executable directly to both the target and the host ``debugfs``.
10308
10309 If the binary is processed through the debug splitting in
10310 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
10311 contents and corresponding ``/usr/src/debug`` files) from the work
10312 directory. Here is an example::
10313
10314 $ bitbake bash
10315 $ bitbake -c devshell bash
10316 $ cd ..
10317 $ scp packages-split/bash/bin/bash target:/bin/bash
10318 $ cp -a packages-split/bash-dbg/\* path/debugfs
10319
10320Debugging with the GNU Project Debugger (GDB) on the Target
10321-----------------------------------------------------------
10322
10323The previous section addressed using GDB remotely for debugging
10324purposes, which is the most usual case due to the inherent hardware
10325limitations on many embedded devices. However, debugging in the target
10326hardware itself is also possible with more powerful devices. This
10327section describes what you need to do in order to support using GDB to
10328debug on the target hardware.
10329
10330To support this kind of debugging, you need do the following:
10331
10332- Ensure that GDB is on the target. You can do this by adding "gdb" to
10333 :term:`IMAGE_INSTALL`::
10334
10335 IMAGE_INSTALL:append = " gdb"
10336
10337 Alternatively, you can add "tools-debug" to :term:`IMAGE_FEATURES`::
10338
10339 IMAGE_FEATURES:append = " tools-debug"
10340
10341- Ensure that debug symbols are present. You can make sure these
10342 symbols are present by installing ``-dbg``::
10343
10344 IMAGE_INSTALL:append = "packagename-dbg"
10345
10346 Alternatively, you can do the following to include
10347 all the debug symbols::
10348
10349 IMAGE_FEATURES:append = " dbg-pkgs"
10350
10351.. note::
10352
10353 To improve the debug information accuracy, you can reduce the level
10354 of optimization used by the compiler. For example, when adding the
10355 following line to your ``local.conf`` file, you will reduce optimization
10356 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
10357 of "-O -fno-omit-frame-pointer"::
10358
10359 DEBUG_BUILD = "1"
10360
10361 Consider that this will reduce the application's performance and is
10362 recommended only for debugging purposes.
10363
10364Other Debugging Tips
10365--------------------
10366
10367Here are some other tips that you might find useful:
10368
10369- When adding new packages, it is worth watching for undesirable items
10370 making their way into compiler command lines. For example, you do not
10371 want references to local system files like ``/usr/lib/`` or
10372 ``/usr/include/``.
10373
10374- If you want to remove the ``psplash`` boot splashscreen, add
10375 ``psplash=false`` to the kernel command line. Doing so prevents
10376 ``psplash`` from loading and thus allows you to see the console. It
10377 is also possible to switch out of the splashscreen by switching the
10378 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10379
10380- Removing :term:`TMPDIR` (usually
10381 ``tmp/``, within the
10382 :term:`Build Directory`) can often fix
10383 temporary build issues. Removing :term:`TMPDIR` is usually a relatively
10384 cheap operation, because task output will be cached in
10385 :term:`SSTATE_DIR` (usually
10386 ``sstate-cache/``, which is also in the Build Directory).
10387
10388 .. note::
10389
10390 Removing :term:`TMPDIR` might be a workaround rather than a fix.
10391 Consequently, trying to determine the underlying cause of an issue before
10392 removing the directory is a good idea.
10393
10394- Understanding how a feature is used in practice within existing
10395 recipes can be very helpful. It is recommended that you configure
10396 some method that allows you to quickly search through files.
10397
10398 Using GNU Grep, you can use the following shell function to
10399 recursively search through common recipe-related files, skipping
10400 binary files, ``.git`` directories, and the Build Directory (assuming
10401 its name starts with "build")::
10402
10403 g() {
10404 grep -Ir \
10405 --exclude-dir=.git \
10406 --exclude-dir='build*' \
10407 --include='*.bb*' \
10408 --include='*.inc*' \
10409 --include='*.conf*' \
10410 --include='*.py*' \
10411 "$@"
10412 }
10413
10414 Following are some usage examples::
10415
10416 $ g FOO # Search recursively for "FOO"
10417 $ g -i foo # Search recursively for "foo", ignoring case
10418 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10419
10420 If figuring
10421 out how some feature works requires a lot of searching, it might
10422 indicate that the documentation should be extended or improved. In
10423 such cases, consider filing a documentation bug using the Yocto
10424 Project implementation of
10425 :yocto_bugs:`Bugzilla <>`. For information on
10426 how to submit a bug against the Yocto Project, see the Yocto Project
10427 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
10428 and the ":doc:`../contributor-guide/report-defect`" section.
10429
10430 .. note::
10431
10432 The manuals might not be the right place to document variables
10433 that are purely internal and have a limited scope (e.g. internal
10434 variables used to implement a single ``.bbclass`` file).
10435
10436Working With Licenses
10437=====================
10438
10439As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
10440section in the Yocto Project Overview and Concepts Manual, open source
10441projects are open to the public and they consequently have different
10442licensing structures in place. This section describes the mechanism by
10443which the :term:`OpenEmbedded Build System`
10444tracks changes to
10445licensing text and covers how to maintain open source license compliance
10446during your project's lifecycle. The section also describes how to
10447enable commercially licensed recipes, which by default are disabled.
10448
10449Tracking License Changes
10450------------------------
10451
10452The license of an upstream project might change in the future. In order
10453to prevent these changes going unnoticed, the
10454:term:`LIC_FILES_CHKSUM`
10455variable tracks changes to the license text. The checksums are validated
10456at the end of the configure step, and if the checksums do not match, the
10457build will fail.
10458
10459Specifying the ``LIC_FILES_CHKSUM`` Variable
10460~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10461
10462The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
10463in the source code for the recipe. Following is an example of how to
10464specify :term:`LIC_FILES_CHKSUM`::
10465
10466 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
10467 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
10468 file://licfile2.txt;endline=50;md5=zzzz \
10469 ..."
10470
10471.. note::
10472
10473 - When using "beginline" and "endline", realize that line numbering
10474 begins with one and not zero. Also, the included lines are
10475 inclusive (i.e. lines five through and including 29 in the
10476 previous example for ``licfile1.txt``).
10477
10478 - When a license check fails, the selected license text is included
10479 as part of the QA message. Using this output, you can determine
10480 the exact start and finish for the needed license text.
10481
10482The build system uses the :term:`S`
10483variable as the default directory when searching files listed in
10484:term:`LIC_FILES_CHKSUM`. The previous example employs the default
10485directory.
10486
10487Consider this next example::
10488
10489 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
10490 md5=bb14ed3c4cda583abc85401304b5cd4e"
10491 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
10492
10493The first line locates a file in ``${S}/src/ls.c`` and isolates lines
10494five through 16 as license text. The second line refers to a file in
10495:term:`WORKDIR`.
10496
10497Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
10498unless the :term:`LICENSE` variable is set to "CLOSED".
10499
10500Explanation of Syntax
10501~~~~~~~~~~~~~~~~~~~~~
10502
10503As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
10504lists all the important files that contain the license text for the
10505source code. It is possible to specify a checksum for an entire file, or
10506a specific section of a file (specified by beginning and ending line
10507numbers with the "beginline" and "endline" parameters, respectively).
10508The latter is useful for source files with a license notice header,
10509README documents, and so forth. If you do not use the "beginline"
10510parameter, then it is assumed that the text begins on the first line of
10511the file. Similarly, if you do not use the "endline" parameter, it is
10512assumed that the license text ends with the last line of the file.
10513
10514The "md5" parameter stores the md5 checksum of the license text. If the
10515license text changes in any way as compared to this parameter then a
10516mismatch occurs. This mismatch triggers a build failure and notifies the
10517developer. Notification allows the developer to review and address the
10518license text changes. Also note that if a mismatch occurs during the
10519build, the correct md5 checksum is placed in the build log and can be
10520easily copied to the recipe.
10521
10522There is no limit to how many files you can specify using the
10523:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
10524requires a few specifications for license tracking. Many projects have a
10525"COPYING" file that stores the license information for all the source
10526code files. This practice allows you to just track the "COPYING" file as
10527long as it is kept up to date.
10528
10529.. note::
10530
10531 - If you specify an empty or invalid "md5" parameter,
10532 :term:`BitBake` returns an md5
10533 mis-match error and displays the correct "md5" parameter value
10534 during the build. The correct parameter is also captured in the
10535 build log.
10536
10537 - If the whole file contains only license text, you do not need to
10538 use the "beginline" and "endline" parameters.
10539
10540Enabling Commercially Licensed Recipes
10541--------------------------------------
10542
10543By default, the OpenEmbedded build system disables components that have
10544commercial or other special licensing requirements. Such requirements
10545are defined on a recipe-by-recipe basis through the
10546:term:`LICENSE_FLAGS` variable
10547definition in the affected recipe. For instance, the
10548``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
10549contains the following statement::
10550
10551 LICENSE_FLAGS = "commercial"
10552
10553Here is a
10554slightly more complicated example that contains both an explicit recipe
10555name and version (after variable expansion)::
10556
10557 LICENSE_FLAGS = "license_${PN}_${PV}"
10558
10559In order for a component restricted by a
10560:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
10561needs to have a matching entry in the global
10562:term:`LICENSE_FLAGS_ACCEPTED`
10563variable, which is a variable typically defined in your ``local.conf``
10564file. For example, to enable the
10565``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
10566could add either the string "commercial_gst-plugins-ugly" or the more
10567general string "commercial" to :term:`LICENSE_FLAGS_ACCEPTED`. See the
10568":ref:`dev-manual/common-tasks:license flag matching`" section for a full
10569explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
10570example::
10571
10572 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly"
10573
10574Likewise, to additionally enable the package built from the recipe
10575containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
10576the actual recipe name was ``emgd_1.10.bb``, the following string would
10577enable that package as well as the original ``gst-plugins-ugly``
10578package::
10579
10580 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly license_emgd_1.10"
10581
10582As a convenience, you do not need to specify the
10583complete license string for every package. You can use
10584an abbreviated form, which consists of just the first portion or
10585portions of the license string before the initial underscore character
10586or characters. A partial string will match any license that contains the
10587given string as the first portion of its license. For example, the
10588following value will also match both of the packages
10589previously mentioned as well as any other packages that have licenses
10590starting with "commercial" or "license".
10591::
10592
10593 LICENSE_FLAGS_ACCEPTED = "commercial license"
10594
10595License Flag Matching
10596~~~~~~~~~~~~~~~~~~~~~
10597
10598License flag matching allows you to control what recipes the
10599OpenEmbedded build system includes in the build. Fundamentally, the
10600build system attempts to match :term:`LICENSE_FLAGS` strings found in
10601recipes against strings found in :term:`LICENSE_FLAGS_ACCEPTED`.
10602A match causes the build system to include a recipe in the
10603build, while failure to find a match causes the build system to exclude
10604a recipe.
10605
10606In general, license flag matching is simple. However, understanding some
10607concepts will help you correctly and effectively use matching.
10608
10609Before a flag defined by a particular recipe is tested against the
10610entries of :term:`LICENSE_FLAGS_ACCEPTED`, the expanded
10611string ``_${PN}`` is appended to the flag. This expansion makes each
10612:term:`LICENSE_FLAGS` value recipe-specific. After expansion, the
10613string is then matched against the entries. Thus, specifying
10614``LICENSE_FLAGS = "commercial"`` in recipe "foo", for example, results
10615in the string ``"commercial_foo"``. And, to create a match, that string
10616must appear among the entries of :term:`LICENSE_FLAGS_ACCEPTED`.
10617
10618Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
10619:term:`LICENSE_FLAGS_ACCEPTED` variable allows you a lot of flexibility for
10620including or excluding recipes based on licensing. For example, you can
10621broaden the matching capabilities by using license flags string subsets
10622in :term:`LICENSE_FLAGS_ACCEPTED`.
10623
10624.. note::
10625
10626 When using a string subset, be sure to use the part of the expanded
10627 string that precedes the appended underscore character (e.g.
10628 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
10629
10630For example, simply specifying the string "commercial" in the
10631:term:`LICENSE_FLAGS_ACCEPTED` variable matches any expanded
10632:term:`LICENSE_FLAGS` definition that starts with the string
10633"commercial" such as "commercial_foo" and "commercial_bar", which
10634are the strings the build system automatically generates for
10635hypothetical recipes named "foo" and "bar" assuming those recipes simply
10636specify the following::
10637
10638 LICENSE_FLAGS = "commercial"
10639
10640Thus, you can choose to exhaustively enumerate each license flag in the
10641list and allow only specific recipes into the image, or you can use a
10642string subset that causes a broader range of matches to allow a range of
10643recipes into the image.
10644
10645This scheme works even if the :term:`LICENSE_FLAGS` string already has
10646``_${PN}`` appended. For example, the build system turns the license
10647flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
10648both the general "commercial" and the specific "commercial_1.2_foo"
10649strings found in the :term:`LICENSE_FLAGS_ACCEPTED` variable, as expected.
10650
10651Here are some other scenarios:
10652
10653- You can specify a versioned string in the recipe such as
10654 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
10655 string to "commercial_foo_1.2_foo". Combine this license flag with a
10656 :term:`LICENSE_FLAGS_ACCEPTED` variable that has the string
10657 "commercial" and you match the flag along with any other flag that
10658 starts with the string "commercial".
10659
10660- Under the same circumstances, you can add "commercial_foo" in the
10661 :term:`LICENSE_FLAGS_ACCEPTED` variable and the build system not only
10662 matches "commercial_foo_1.2" but also matches any license flag with
10663 the string "commercial_foo", regardless of the version.
10664
10665- You can be very specific and use both the package and version parts
10666 in the :term:`LICENSE_FLAGS_ACCEPTED` list (e.g.
10667 "commercial_foo_1.2") to specifically match a versioned recipe.
10668
10669Other Variables Related to Commercial Licenses
10670~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10671
10672There are other helpful variables related to commercial license handling,
10673defined in the
10674``poky/meta/conf/distro/include/default-distrovars.inc`` file::
10675
10676 COMMERCIAL_AUDIO_PLUGINS ?= ""
10677 COMMERCIAL_VIDEO_PLUGINS ?= ""
10678
10679If you
10680want to enable these components, you can do so by making sure you have
10681statements similar to the following in your ``local.conf`` configuration
10682file::
10683
10684 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
10685 gst-plugins-ugly-mpegaudioparse"
10686 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
10687 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
10688 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
10689
10690
10691Of course, you could also create a matching list for those
10692components using the more general "commercial" in the
10693:term:`LICENSE_FLAGS_ACCEPTED` variable, but that would also enable all
10694the other packages with :term:`LICENSE_FLAGS`
10695containing "commercial", which you may or may not want::
10696
10697 LICENSE_FLAGS_ACCEPTED = "commercial"
10698
10699Specifying audio and video plugins as part of the
10700``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
10701(along with the enabling :term:`LICENSE_FLAGS_ACCEPTED`) includes the
10702plugins or components into built images, thus adding support for media
10703formats or components.
10704
10705Maintaining Open Source License Compliance During Your Product's Lifecycle
10706--------------------------------------------------------------------------
10707
10708One of the concerns for a development organization using open source
10709software is how to maintain compliance with various open source
10710licensing during the lifecycle of the product. While this section does
10711not provide legal advice or comprehensively cover all scenarios, it does
10712present methods that you can use to assist you in meeting the compliance
10713requirements during a software release.
10714
10715With hundreds of different open source licenses that the Yocto Project
10716tracks, it is difficult to know the requirements of each and every
10717license. However, the requirements of the major FLOSS licenses can begin
10718to be covered by assuming that there are three main areas of concern:
10719
10720- Source code must be provided.
10721
10722- License text for the software must be provided.
10723
10724- Compilation scripts and modifications to the source code must be
10725 provided.
10726
10727- spdx files can be provided.
10728
10729There are other requirements beyond the scope of these three and the
10730methods described in this section (e.g. the mechanism through which
10731source code is distributed).
10732
10733As different organizations have different methods of complying with open
10734source licensing, this section is not meant to imply that there is only
10735one single way to meet your compliance obligations, but rather to
10736describe one method of achieving compliance. The remainder of this
10737section describes methods supported to meet the previously mentioned
10738three requirements. Once you take steps to meet these requirements, and
10739prior to releasing images, sources, and the build system, you should
10740audit all artifacts to ensure completeness.
10741
10742.. note::
10743
10744 The Yocto Project generates a license manifest during image creation
10745 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
10746 to assist with any audits.
10747
10748Providing the Source Code
10749~~~~~~~~~~~~~~~~~~~~~~~~~
10750
10751Compliance activities should begin before you generate the final image.
10752The first thing you should look at is the requirement that tops the list
10753for most compliance groups - providing the source. The Yocto Project has
10754a few ways of meeting this requirement.
10755
10756One of the easiest ways to meet this requirement is to provide the
10757entire :term:`DL_DIR` used by the
10758build. This method, however, has a few issues. The most obvious is the
10759size of the directory since it includes all sources used in the build
10760and not just the source used in the released image. It will include
10761toolchain source, and other artifacts, which you would not generally
10762release. However, the more serious issue for most companies is
10763accidental release of proprietary software. The Yocto Project provides
10764an :ref:`archiver <ref-classes-archiver>` class to
10765help avoid some of these concerns.
10766
10767Before you employ :term:`DL_DIR` or the :ref:`archiver <ref-classes-archiver>` class, you need to
10768decide how you choose to provide source. The source ``archiver`` class
10769can generate tarballs and SRPMs and can create them with various levels
10770of compliance in mind.
10771
10772One way of doing this (but certainly not the only way) is to release
10773just the source as a tarball. You can do this by adding the following to
10774the ``local.conf`` file found in the
10775:term:`Build Directory`::
10776
10777 INHERIT += "archiver"
10778 ARCHIVER_MODE[src] = "original"
10779
10780During the creation of your
10781image, the source from all recipes that deploy packages to the image is
10782placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
10783:term:`LICENSE` for each recipe.
10784Releasing the entire directory enables you to comply with requirements
10785concerning providing the unmodified source. It is important to note that
10786the size of the directory can get large.
10787
10788A way to help mitigate the size issue is to only release tarballs for
10789licenses that require the release of source. Let us assume you are only
10790concerned with GPL code as identified by running the following script:
10791
10792.. code-block:: shell
10793
10794 # Script to archive a subset of packages matching specific license(s)
10795 # Source and license files are copied into sub folders of package folder
10796 # Must be run from build folder
10797 #!/bin/bash
10798 src_release_dir="source-release"
10799 mkdir -p $src_release_dir
10800 for a in tmp/deploy/sources/*; do
10801 for d in $a/*; do
10802 # Get package name from path
10803 p=`basename $d`
10804 p=${p%-*}
10805 p=${p%-*}
10806 # Only archive GPL packages (update *GPL* regex for your license check)
10807 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
10808 if [ $numfiles -ge 1 ]; then
10809 echo Archiving $p
10810 mkdir -p $src_release_dir/$p/source
10811 cp $d/* $src_release_dir/$p/source 2> /dev/null
10812 mkdir -p $src_release_dir/$p/license
10813 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
10814 fi
10815 done
10816 done
10817
10818At this point, you
10819could create a tarball from the ``gpl_source_release`` directory and
10820provide that to the end user. This method would be a step toward
10821achieving compliance with section 3a of GPLv2 and with section 6 of
10822GPLv3.
10823
10824Providing License Text
10825~~~~~~~~~~~~~~~~~~~~~~
10826
10827One requirement that is often overlooked is inclusion of license text.
10828This requirement also needs to be dealt with prior to generating the
10829final image. Some licenses require the license text to accompany the
10830binary. You can achieve this by adding the following to your
10831``local.conf`` file::
10832
10833 COPY_LIC_MANIFEST = "1"
10834 COPY_LIC_DIRS = "1"
10835 LICENSE_CREATE_PACKAGE = "1"
10836
10837Adding these statements to the
10838configuration file ensures that the licenses collected during package
10839generation are included on your image.
10840
10841.. note::
10842
10843 Setting all three variables to "1" results in the image having two
10844 copies of the same license file. One copy resides in
10845 ``/usr/share/common-licenses`` and the other resides in
10846 ``/usr/share/license``.
10847
10848 The reason for this behavior is because
10849 :term:`COPY_LIC_DIRS` and
10850 :term:`COPY_LIC_MANIFEST`
10851 add a copy of the license when the image is built but do not offer a
10852 path for adding licenses for newly installed packages to an image.
10853 :term:`LICENSE_CREATE_PACKAGE`
10854 adds a separate package and an upgrade path for adding licenses to an
10855 image.
10856
10857As the source ``archiver`` class has already archived the original
10858unmodified source that contains the license files, you would have
10859already met the requirements for inclusion of the license information
10860with source as defined by the GPL and other open source licenses.
10861
10862Providing Compilation Scripts and Source Code Modifications
10863~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10864
10865At this point, we have addressed all we need to prior to generating the
10866image. The next two requirements are addressed during the final
10867packaging of the release.
10868
10869By releasing the version of the OpenEmbedded build system and the layers
10870used during the build, you will be providing both compilation scripts
10871and the source code modifications in one step.
10872
10873If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
10874and a distro layer, and those
10875those layers are used to patch, compile, package, or modify (in any way)
10876any open source software included in your released images, you might be
10877required to release those layers under section 3 of GPLv2 or section 1
10878of GPLv3. One way of doing that is with a clean checkout of the version
10879of the Yocto Project and layers used during your build. Here is an
10880example:
10881
10882.. code-block:: shell
10883
10884 # We built using the dunfell branch of the poky repo
10885 $ git clone -b dunfell git://git.yoctoproject.org/poky
10886 $ cd poky
10887 # We built using the release_branch for our layers
10888 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
10889 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
10890 # clean up the .git repos
10891 $ find . -name ".git" -type d -exec rm -rf {} \;
10892
10893One
10894thing a development organization might want to consider for end-user
10895convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to
10896ensure that when the end user utilizes the released build system to
10897build an image, the development organization's layers are included in
10898the ``bblayers.conf`` file automatically::
10899
10900 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
10901 # changes incompatibly
10902 POKY_BBLAYERS_CONF_VERSION = "2"
10903
10904 BBPATH = "${TOPDIR}"
10905 BBFILES ?= ""
10906
10907 BBLAYERS ?= " \
10908 ##OEROOT##/meta \
10909 ##OEROOT##/meta-poky \
10910 ##OEROOT##/meta-yocto-bsp \
10911 ##OEROOT##/meta-mylayer \
10912 "
10913
10914Creating and
10915providing an archive of the :term:`Metadata`
10916layers (recipes, configuration files, and so forth) enables you to meet
10917your requirements to include the scripts to control compilation as well
10918as any modifications to the original source.
10919
10920Compliance Limitations with Executables Built from Static Libraries
10921~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10922
10923When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS`
10924mechanisms as well as explicitly included in the image recipe with
10925:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B
10926(``DEPENDS += "B"``), package B will neither appear in the generated license
10927manifest nor in the generated source tarballs. This occurs as the
10928:ref:`license <ref-classes-license>` and :ref:`archiver <ref-classes-archiver>`
10929classes assume that only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS`
10930end up in the image.
10931
10932As a result, potential obligations regarding license compliance for package B
10933may not be met.
10934
10935The Yocto Project doesn't enable static libraries by default, in part because
10936of this issue. Before a solution to this limitation is found, you need to
10937keep in mind that if your root filesystem is built from static libraries,
10938you will need to manually ensure that your deliveries are compliant
10939with the licenses of these libraries.
10940
10941Copying Non Standard Licenses
10942-----------------------------
10943
10944Some packages, such as the linux-firmware package, have many licenses
10945that are not in any way common. You can avoid adding a lot of these
10946types of common license files, which are only applicable to a specific
10947package, by using the
10948:term:`NO_GENERIC_LICENSE`
10949variable. Using this variable also avoids QA errors when you use a
10950non-common, non-CLOSED license in a recipe.
10951
10952Here is an example that uses the ``LICENSE.Abilis.txt`` file as
10953the license from the fetched source::
10954
10955 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
10956
10957Checking for Vulnerabilities
10958============================
10959
10960Vulnerabilities in Poky and OE-Core
10961-----------------------------------
10962
10963The Yocto Project has an infrastructure to track and address unfixed
10964known security vulnerabilities, as tracked by the public
10965:wikipedia:`Common Vulnerabilities and Exposures (CVE) <Common_Vulnerabilities_and_Exposures>`
10966database.
10967
10968The Yocto Project maintains a `list of known vulnerabilities
10969<https://autobuilder.yocto.io/pub/non-release/patchmetrics/>`__
10970for packages in Poky and OE-Core, tracking the evolution of the number of
10971unpatched CVEs and the status of patches. Such information is available for
10972the current development version and for each supported release.
10973
10974Security is a process, not a product, and thus at any time, a number of security
10975issues may be impacting Poky and OE-Core. It is up to the maintainers, users,
10976contributors and anyone interested in the issues to investigate and possibly fix them by
10977updating software components to newer versions or by applying patches to address them.
10978It is recommended to work with Poky and OE-Core upstream maintainers and submit
10979patches to fix them, see :doc:`../contributor-guide/submit-changes` for details.
10980
10981Vulnerability check at build time
10982---------------------------------
10983
10984To enable a check for CVE security vulnerabilities using :ref:`cve-check <ref-classes-cve-check>` in the specific image
10985or target you are building, add the following setting to your configuration::
10986
10987 INHERIT += "cve-check"
10988
10989The CVE database contains some old incomplete entries which have been
10990deemed not to impact Poky or OE-Core. These CVE entries can be excluded from the
10991check using build configuration::
10992
10993 include conf/distro/include/cve-extra-exclusions.inc
10994
10995With this CVE check enabled, BitBake build will try to map each compiled software component
10996recipe name and version information to the CVE database and generate recipe and
10997image specific reports. These reports will contain:
10998
10999- metadata about the software component like names and versions
11000
11001- metadata about the CVE issue such as description and NVD link
11002
11003- for each software component, a list of CVEs which are possibly impacting this version
11004
11005- status of each CVE: ``Patched``, ``Unpatched`` or ``Ignored``
11006
11007The status ``Patched`` means that a patch file to address the security issue has been
11008applied. ``Unpatched`` status means that no patches to address the issue have been
11009applied and that the issue needs to be investigated. ``Ignored`` means that after
11010analysis, it has been deemed to ignore the issue as it for example affects
11011the software component on a different operating system platform.
11012
11013After a build with CVE check enabled, reports for each compiled source recipe will be
11014found in ``build/tmp/deploy/cve``.
11015
11016For example the CVE check report for the ``flex-native`` recipe looks like::
11017
11018 $ cat poky/build/tmp/deploy/cve/flex-native
11019 LAYER: meta
11020 PACKAGE NAME: flex-native
11021 PACKAGE VERSION: 2.6.4
11022 CVE: CVE-2016-6354
11023 CVE STATUS: Patched
11024 CVE SUMMARY: Heap-based buffer overflow in the yy_get_next_buffer function in Flex before 2.6.1 might allow context-dependent attackers to cause a denial of service or possibly execute arbitrary code via vectors involving num_to_read.
11025 CVSS v2 BASE SCORE: 7.5
11026 CVSS v3 BASE SCORE: 9.8
11027 VECTOR: NETWORK
11028 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2016-6354
11029
11030 LAYER: meta
11031 PACKAGE NAME: flex-native
11032 PACKAGE VERSION: 2.6.4
11033 CVE: CVE-2019-6293
11034 CVE STATUS: Ignored
11035 CVE SUMMARY: An issue was discovered in the function mark_beginning_as_normal in nfa.c in flex 2.6.4. There is a stack exhaustion problem caused by the mark_beginning_as_normal function making recursive calls to itself in certain scenarios involving lots of '*' characters. Remote attackers could leverage this vulnerability to cause a denial-of-service.
11036 CVSS v2 BASE SCORE: 4.3
11037 CVSS v3 BASE SCORE: 5.5
11038 VECTOR: NETWORK
11039 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2019-6293
11040
11041For images, a summary of all recipes included in the image and their CVEs is also
11042generated in textual and JSON formats. These ``.cve`` and ``.json`` reports can be found
11043in the ``tmp/deploy/images`` directory for each compiled image.
11044
11045At build time CVE check will also throw warnings about ``Unpatched`` CVEs::
11046
11047 WARNING: flex-2.6.4-r0 do_cve_check: Found unpatched CVE (CVE-2019-6293), for more information check /poky/build/tmp/work/core2-64-poky-linux/flex/2.6.4-r0/temp/cve.log
11048 WARNING: libarchive-3.5.1-r0 do_cve_check: Found unpatched CVE (CVE-2021-36976), for more information check /poky/build/tmp/work/core2-64-poky-linux/libarchive/3.5.1-r0/temp/cve.log
11049
11050It is also possible to check the CVE status of individual packages as follows::
11051
11052 bitbake -c cve_check flex libarchive
11053
11054Fixing CVE product name and version mappings
11055--------------------------------------------
11056
11057By default, :ref:`cve-check <ref-classes-cve-check>` uses the recipe name :term:`BPN` as CVE
11058product name when querying the CVE database. If this mapping contains false positives, e.g.
11059some reported CVEs are not for the software component in question, or false negatives like
11060some CVEs are not found to impact the recipe when they should, then the problems can be
11061in the recipe name to CVE product mapping. These mapping issues can be fixed by setting
11062the :term:`CVE_PRODUCT` variable inside the recipe. This defines the name of the software component in the
11063upstream `NIST CVE database <https://nvd.nist.gov/>`__.
11064
11065The variable supports using vendor and product names like this::
11066
11067 CVE_PRODUCT = "flex_project:flex"
11068
11069In this example the vendor name used in the CVE database is ``flex_project`` and the
11070product is ``flex``. With this setting the ``flex`` recipe only maps to this specific
11071product and not products from other vendors with same name ``flex``.
11072
11073Similarly, when the recipe version :term:`PV` is not compatible with software versions used by
11074the upstream software component releases and the CVE database, these can be fixed using
11075the :term:`CVE_VERSION` variable.
11076
11077Note that if the CVE entries in the NVD database contain bugs or have missing or incomplete
11078information, it is recommended to fix the information there directly instead of working
11079around the issues possibly for a long time in Poky and OE-Core side recipes. Feedback to
11080NVD about CVE entries can be provided through the `NVD contact form <https://nvd.nist.gov/info/contact-form>`__.
11081
11082Fixing vulnerabilities in recipes
11083---------------------------------
11084
11085If a CVE security issue impacts a software component, it can be fixed by updating to a newer
11086version of the software component or by applying a patch. For Poky and OE-Core master branches, updating
11087to a newer software component release with fixes is the best option, but patches can be applied
11088if releases are not yet available.
11089
11090For stable branches, it is preferred to apply patches for the issues. For some software
11091components minor version updates can also be applied if they are backwards compatible.
11092
11093Here is an example of fixing CVE security issues with patch files,
11094an example from the :oe_layerindex:`ffmpeg recipe</layerindex/recipe/47350>`::
11095
11096 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
11097 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \
11098 file://fix-CVE-2020-20446.patch \
11099 file://fix-CVE-2020-20453.patch \
11100 file://fix-CVE-2020-22015.patch \
11101 file://fix-CVE-2020-22021.patch \
11102 file://fix-CVE-2020-22033-CVE-2020-22019.patch \
11103 file://fix-CVE-2021-33815.patch \
11104
11105A good practice is to include the CVE identifier in both the patch file name
11106and inside the patch file commit message using the format::
11107
11108 CVE: CVE-2020-22033
11109
11110CVE checker will then capture this information and change the CVE status to ``Patched``
11111in the generated reports.
11112
11113If analysis shows that the CVE issue does not impact the recipe due to configuration, platform,
11114version or other reasons, the CVE can be marked as ``Ignored`` using the :term:`CVE_CHECK_IGNORE` variable.
11115As mentioned previously, if data in the CVE database is wrong, it is recommend to fix those
11116issues in the CVE database directly.
11117
11118Recipes can be completely skipped by CVE check by including the recipe name in
11119the :term:`CVE_CHECK_SKIP_RECIPE` variable.
11120
11121Implementation details
11122----------------------
11123
11124Here's what the :ref:`cve-check <ref-classes-cve-check>` class does to
11125find unpatched CVE IDs.
11126
11127First the code goes through each patch file provided by a recipe. If a valid CVE ID
11128is found in the name of the file, the corresponding CVE is considered as patched.
11129Don't forget that if multiple CVE IDs are found in the filename, only the last
11130one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch
11131file. The found CVE IDs are also considered as patched.
11132
11133Then, the code looks up all the CVE IDs in the NIST database for all the
11134products defined in :term:`CVE_PRODUCT`. Then, for each found CVE:
11135
11136- If the package name (:term:`PN`) is part of
11137 :term:`CVE_CHECK_SKIP_RECIPE`, it is considered as ``Patched``.
11138
11139- If the CVE ID is part of :term:`CVE_CHECK_IGNORE`, it is
11140 set as ``Ignored``.
11141
11142- If the CVE ID is part of the patched CVE for the recipe, it is
11143 already considered as ``Patched``.
11144
11145- Otherwise, the code checks whether the recipe version (:term:`PV`)
11146 is within the range of versions impacted by the CVE. If so, the CVE
11147 is considered as ``Unpatched``.
11148
11149The CVE database is stored in :term:`DL_DIR` and can be inspected using
11150``sqlite3`` command as follows::
11151
11152 sqlite3 downloads/CVE_CHECK/nvdcve_1.1.db .dump | grep CVE-2021-37462
11153
11154When analyzing CVEs, it is recommended to:
11155
11156- study the latest information in `CVE database <https://nvd.nist.gov/vuln/search>`__.
11157
11158- check how upstream developers of the software component addressed the issue, e.g.
11159 what patch was applied, which upstream release contains the fix.
11160
11161- check what other Linux distributions like `Debian <https://security-tracker.debian.org/tracker/>`__
11162 did to analyze and address the issue.
11163
11164- follow security notices from other Linux distributions.
11165
11166- follow public `open source security mailing lists <https://oss-security.openwall.org/wiki/mailing-lists>`__ for
11167 discussions and advance notifications of CVE bugs and software releases with fixes.
11168
11169Creating a Software Bill of Materials
11170=====================================
11171
11172Once you are able to build an image for your project, once the licenses for
11173each software component are all identified (see
11174":ref:`dev-manual/common-tasks:working with licenses`") and once vulnerability
11175fixes are applied (see ":ref:`dev-manual/common-tasks:checking
11176for vulnerabilities`"), the OpenEmbedded build system can generate
11177a description of all the components you used, their licenses, their dependencies,
11178the changes that were applied and the known vulnerabilities that were fixed.
11179
11180This description is generated in the form of a *Software Bill of Materials*
11181(:term:`SBOM`), using the :term:`SPDX` standard.
11182
11183When you release software, this is the most standard way to provide information
11184about the Software Supply Chain of your software image and SDK. The
11185:term:`SBOM` tooling is often used to ensure open source license compliance by
11186providing the license texts used in the product which legal departments and end
11187users can read in standardized format.
11188
11189:term:`SBOM` information is also critical to performing vulnerability exposure
11190assessments, as all the components used in the Software Supply Chain are listed.
11191
11192The OpenEmbedded build system doesn't generate such information by default.
11193To make this happen, you must inherit the
11194:ref:`create-spdx <ref-classes-create-spdx>` class from a configuration file::
11195
11196 INHERIT += "create-spdx"
11197
11198You then get :term:`SPDX` output in JSON format as an
11199``IMAGE-MACHINE.spdx.json`` file in ``tmp/deploy/images/MACHINE/`` inside the
11200:term:`Build Directory`.
11201
11202This is a toplevel file accompanied by an ``IMAGE-MACHINE.spdx.index.json``
11203containing an index of JSON :term:`SPDX` files for individual recipes, together
11204with an ``IMAGE-MACHINE.spdx.tar.zst`` compressed archive containing all such
11205files.
11206
11207The :ref:`ref-classes-create-spdx` class offers options to include
11208more information in the output :term:`SPDX` data, such as making the generated
11209files more human readable (:term:`SPDX_PRETTY`), adding compressed archives of
11210the files in the generated target packages (:term:`SPDX_ARCHIVE_PACKAGED`),
11211adding a description of the source files used to generate host tools and target
11212packages (:term:`SPDX_INCLUDE_SOURCES`) and adding archives of these source
11213files themselves (:term:`SPDX_ARCHIVE_SOURCES`).
11214
11215Though the toplevel :term:`SPDX` output is available in
11216``tmp/deploy/images/MACHINE/`` inside the :term:`Build Directory`, ancillary
11217generated files are available in ``tmp/deploy/spdx/MACHINE`` too, such as:
11218
11219- The individual :term:`SPDX` JSON files in the ``IMAGE-MACHINE.spdx.tar.zst``
11220 archive.
11221
11222- Compressed archives of the files in the generated target packages,
11223 in ``packages/packagename.tar.zst`` (when :term:`SPDX_ARCHIVE_PACKAGED`
11224 is set).
11225
11226- Compressed archives of the source files used to build the host tools
11227 and the target packages in ``recipes/recipe-packagename.tar.zst``
11228 (when :term:`SPDX_ARCHIVE_SOURCES` is set). Those are needed to fulfill
11229 "source code access" license requirements.
11230
11231See the `tools page <https://spdx.dev/resources/tools/>`__ on the :term:`SPDX`
11232project website for a list of tools to consume and transform the :term:`SPDX`
11233data generated by the OpenEmbedded build system.
11234
11235See also Joshua Watt's
11236`Automated SBoM generation with OpenEmbedded and the Yocto Project <https://youtu.be/Q5UQUM6zxVU>`__
11237presentation at FOSDEM 2023.
11238
11239
11240Using the Error Reporting Tool
11241==============================
11242
11243The error reporting tool allows you to submit errors encountered during
11244builds to a central database. Outside of the build environment, you can
11245use a web interface to browse errors, view statistics, and query for
11246errors. The tool works using a client-server system where the client
11247portion is integrated with the installed Yocto Project
11248:term:`Source Directory` (e.g. ``poky``).
11249The server receives the information collected and saves it in a
11250database.
11251
11252There is a live instance of the error reporting server at
11253https://errors.yoctoproject.org.
11254When you want to get help with build failures, you can submit all of the
11255information on the failure easily and then point to the URL in your bug
11256report or send an email to the mailing list.
11257
11258.. note::
11259
11260 If you send error reports to this server, the reports become publicly
11261 visible.
11262
11263Enabling and Using the Tool
11264---------------------------
11265
11266By default, the error reporting tool is disabled. You can enable it by
11267inheriting the
11268:ref:`report-error <ref-classes-report-error>`
11269class by adding the following statement to the end of your
11270``local.conf`` file in your
11271:term:`Build Directory`.
11272::
11273
11274 INHERIT += "report-error"
11275
11276By default, the error reporting feature stores information in
11277``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11278However, you can specify a directory to use by adding the following to
11279your ``local.conf`` file::
11280
11281 ERR_REPORT_DIR = "path"
11282
11283Enabling error
11284reporting causes the build process to collect the errors and store them
11285in a file as previously described. When the build system encounters an
11286error, it includes a command as part of the console output. You can run
11287the command to send the error file to the server. For example, the
11288following command sends the errors to an upstream server::
11289
11290 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11291
11292In the previous example, the errors are sent to a public database
11293available at https://errors.yoctoproject.org, which is used by the
11294entire community. If you specify a particular server, you can send the
11295errors to a different database. Use the following command for more
11296information on available options::
11297
11298 $ send-error-report --help
11299
11300When sending the error file, you are prompted to review the data being
11301sent as well as to provide a name and optional email address. Once you
11302satisfy these prompts, the command returns a link from the server that
11303corresponds to your entry in the database. For example, here is a
11304typical link: https://errors.yoctoproject.org/Errors/Details/9522/
11305
11306Following the link takes you to a web interface where you can browse,
11307query the errors, and view statistics.
11308
11309Disabling the Tool
11310------------------
11311
11312To disable the error reporting feature, simply remove or comment out the
11313following statement from the end of your ``local.conf`` file in your
11314:term:`Build Directory`.
11315::
11316
11317 INHERIT += "report-error"
11318
11319Setting Up Your Own Error Reporting Server
11320------------------------------------------
11321
11322If you want to set up your own error reporting server, you can obtain
11323the code from the Git repository at :yocto_git:`/error-report-web/`.
11324Instructions on how to set it up are in the README document.
11325
11326Using Wayland and Weston
11327========================
11328
11329`Wayland <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__
11330is a computer display server protocol that provides a method for
11331compositing window managers to communicate directly with applications
11332and video hardware and expects them to communicate with input hardware
11333using other libraries. Using Wayland with supporting targets can result
11334in better control over graphics frame rendering than an application
11335might otherwise achieve.
11336
11337The Yocto Project provides the Wayland protocol libraries and the
11338reference
11339`Weston <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__
11340compositor as part of its release. You can find the integrated packages
11341in the ``meta`` layer of the :term:`Source Directory`.
11342Specifically, you
11343can find the recipes that build both Wayland and Weston at
11344``meta/recipes-graphics/wayland``.
11345
11346You can build both the Wayland and Weston packages for use only with
11347targets that accept the `Mesa 3D and Direct Rendering
11348Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__,
11349which is also known as Mesa DRI. This implies that you cannot build and
11350use the packages if your target uses, for example, the Intel Embedded
11351Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11352
11353.. note::
11354
11355 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11356 emulated QEMU hardware. However, this version of Weston will run
11357 under X emulation without issues.
11358
11359This section describes what you need to do to implement Wayland and use
11360the Weston compositor when building an image for a supporting target.
11361
11362Enabling Wayland in an Image
11363----------------------------
11364
11365To enable Wayland, you need to enable it to be built and enable it to be
11366included (installed) in the image.
11367
11368Building Wayland
11369~~~~~~~~~~~~~~~~
11370
11371To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11372Wayland with Kernel Mode Setting
11373(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11374support, include the "wayland" flag in the
11375:term:`DISTRO_FEATURES`
11376statement in your ``local.conf`` file::
11377
11378 DISTRO_FEATURES:append = " wayland"
11379
11380.. note::
11381
11382 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11383 support
11384
11385Installing Wayland and Weston
11386~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11387
11388To install the Wayland feature into an image, you must include the
11389following
11390:term:`CORE_IMAGE_EXTRA_INSTALL`
11391statement in your ``local.conf`` file::
11392
11393 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11394
11395Running Weston
11396--------------
11397
11398To run Weston inside X11, enabling it as described earlier and building
11399a Sato image is sufficient. If you are running your image under Sato, a
11400Weston Launcher appears in the "Utility" category.
11401
11402Alternatively, you can run Weston through the command-line interpretor
11403(CLI), which is better suited for development work. To run Weston under
11404the CLI, you need to do the following after your image is built:
11405
114061. Run these commands to export ``XDG_RUNTIME_DIR``::
11407
11408 mkdir -p /tmp/$USER-weston
11409 chmod 0700 /tmp/$USER-weston
11410 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11411
114122. Launch Weston in the shell::
11413
11414 weston
diff --git a/documentation/dev-manual/custom-distribution.rst b/documentation/dev-manual/custom-distribution.rst
new file mode 100644
index 0000000000..47faed0d04
--- /dev/null
+++ b/documentation/dev-manual/custom-distribution.rst
@@ -0,0 +1,109 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating Your Own Distribution
4******************************
5
6When you build an image using the Yocto Project and do not alter any
7distribution :term:`Metadata`, you are
8creating a Poky distribution. If you wish to gain more control over
9package alternative selections, compile-time options, and other
10low-level configurations, you can create your own distribution.
11
12To create your own distribution, the basic steps consist of creating
13your own distribution layer, creating your own distribution
14configuration file, and then adding any needed code and Metadata to the
15layer. The following steps provide some more detail:
16
17- *Create a layer for your new distro:* Create your distribution layer
18 so that you can keep your Metadata and code for the distribution
19 separate. It is strongly recommended that you create and use your own
20 layer for configuration and code. Using your own layer as compared to
21 just placing configurations in a ``local.conf`` configuration file
22 makes it easier to reproduce the same build configuration when using
23 multiple build machines. See the
24 ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
25 section for information on how to quickly set up a layer.
26
27- *Create the distribution configuration file:* The distribution
28 configuration file needs to be created in the ``conf/distro``
29 directory of your layer. You need to name it using your distribution
30 name (e.g. ``mydistro.conf``).
31
32 .. note::
33
34 The :term:`DISTRO` variable in your ``local.conf`` file determines the
35 name of your distribution.
36
37 You can split out parts of your configuration file into include files
38 and then "require" them from within your distribution configuration
39 file. Be sure to place the include files in the
40 ``conf/distro/include`` directory of your layer. A common example
41 usage of include files would be to separate out the selection of
42 desired version and revisions for individual recipes.
43
44 Your configuration file needs to set the following required
45 variables:
46
47 - :term:`DISTRO_NAME`
48
49 - :term:`DISTRO_VERSION`
50
51 These following variables are optional and you typically set them
52 from the distribution configuration file:
53
54 - :term:`DISTRO_FEATURES`
55
56 - :term:`DISTRO_EXTRA_RDEPENDS`
57
58 - :term:`DISTRO_EXTRA_RRECOMMENDS`
59
60 - :term:`TCLIBC`
61
62 .. tip::
63
64 If you want to base your distribution configuration file on the
65 very basic configuration from OE-Core, you can use
66 ``conf/distro/defaultsetup.conf`` as a reference and just include
67 variables that differ as compared to ``defaultsetup.conf``.
68 Alternatively, you can create a distribution configuration file
69 from scratch using the ``defaultsetup.conf`` file or configuration files
70 from another distribution such as Poky as a reference.
71
72- *Provide miscellaneous variables:* Be sure to define any other
73 variables for which you want to create a default or enforce as part
74 of the distribution configuration. You can include nearly any
75 variable from the ``local.conf`` file. The variables you use are not
76 limited to the list in the previous bulleted item.
77
78- *Point to Your distribution configuration file:* In your ``local.conf``
79 file in the :term:`Build Directory`, set your :term:`DISTRO` variable to
80 point to your distribution's configuration file. For example, if your
81 distribution's configuration file is named ``mydistro.conf``, then
82 you point to it as follows::
83
84 DISTRO = "mydistro"
85
86- *Add more to the layer if necessary:* Use your layer to hold other
87 information needed for the distribution:
88
89 - Add recipes for installing distro-specific configuration files
90 that are not already installed by another recipe. If you have
91 distro-specific configuration files that are included by an
92 existing recipe, you should add an append file (``.bbappend``) for
93 those. For general information and recommendations on how to add
94 recipes to your layer, see the
95 ":ref:`dev-manual/layers:creating your own layer`" and
96 ":ref:`dev-manual/layers:following best practices when creating layers`"
97 sections.
98
99 - Add any image recipes that are specific to your distribution.
100
101 - Add a ``psplash`` append file for a branded splash screen, using
102 the :term:`SPLASH_IMAGES` variable.
103
104 - Add any other append files to make custom changes that are
105 specific to individual recipes.
106
107 For information on append files, see the
108 ":ref:`dev-manual/layers:appending other layers metadata with your layer`"
109 section.
diff --git a/documentation/dev-manual/custom-template-configuration-directory.rst b/documentation/dev-manual/custom-template-configuration-directory.rst
new file mode 100644
index 0000000000..90ed3ed92e
--- /dev/null
+++ b/documentation/dev-manual/custom-template-configuration-directory.rst
@@ -0,0 +1,72 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Custom Template Configuration Directory
4**************************************************
5
6If you are producing your own customized version of the build system for
7use by other users, you might want to customize the message shown by the
8setup script or you might want to change the template configuration
9files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
10new build directory.
11
12The OpenEmbedded build system uses the environment variable
13``TEMPLATECONF`` to locate the directory from which it gathers
14configuration information that ultimately ends up in the
15:term:`Build Directory` ``conf`` directory.
16By default, ``TEMPLATECONF`` is set as follows in the ``poky``
17repository::
18
19 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
20
21This is the
22directory used by the build system to find templates from which to build
23some key configuration files. If you look at this directory, you will
24see the ``bblayers.conf.sample``, ``local.conf.sample``, and
25``conf-notes.txt`` files. The build system uses these files to form the
26respective ``bblayers.conf`` file, ``local.conf`` file, and display the
27list of BitBake targets when running the setup script.
28
29To override these default configuration files with configurations you
30want used within every new Build Directory, simply set the
31``TEMPLATECONF`` variable to your directory. The ``TEMPLATECONF``
32variable is set in the ``.templateconf`` file, which is in the top-level
33:term:`Source Directory` folder
34(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
35directory.
36
37Best practices dictate that you should keep your template configuration
38directory in your custom distribution layer. For example, suppose you
39have a layer named ``meta-mylayer`` located in your home directory and
40you want your template configuration directory named ``myconf``.
41Changing the ``.templateconf`` as follows causes the OpenEmbedded build
42system to look in your directory and base its configuration files on the
43``*.sample`` configuration files it finds. The final configuration files
44(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
45your Build Directory, but they are based on your ``*.sample`` files.
46::
47
48 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
49
50Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
51also resides in the default ``meta-poky/conf`` directory. The script
52that sets up the build environment (i.e.
53:ref:`structure-core-script`) uses this file to
54display BitBake targets as part of the script output. Customizing this
55``conf-notes.txt`` file is a good way to make sure your list of custom
56targets appears as part of the script's output.
57
58Here is the default list of targets displayed as a result of running
59either of the setup scripts::
60
61 You can now run 'bitbake <target>'
62
63 Common targets are:
64 core-image-minimal
65 core-image-sato
66 meta-toolchain
67 meta-ide-support
68
69Changing the listed common targets is as easy as editing your version of
70``conf-notes.txt`` in your custom template configuration directory and
71making sure you have ``TEMPLATECONF`` set to your directory.
72
diff --git a/documentation/dev-manual/customizing-images.rst b/documentation/dev-manual/customizing-images.rst
new file mode 100644
index 0000000000..17dd2d2b0c
--- /dev/null
+++ b/documentation/dev-manual/customizing-images.rst
@@ -0,0 +1,223 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Customizing Images
4******************
5
6You can customize images to satisfy particular requirements. This
7section describes several methods and provides guidelines for each.
8
9Customizing Images Using ``local.conf``
10=======================================
11
12Probably the easiest way to customize an image is to add a package by
13way of the ``local.conf`` configuration file. Because it is limited to
14local use, this method generally only allows you to add packages and is
15not as flexible as creating your own customized image. When you add
16packages using local variables this way, you need to realize that these
17variable changes are in effect for every build and consequently affect
18all images, which might not be what you require.
19
20To add a package to your image using the local configuration file, use
21the :term:`IMAGE_INSTALL` variable with the ``:append`` operator::
22
23 IMAGE_INSTALL:append = " strace"
24
25Use of the syntax is important; specifically, the leading space
26after the opening quote and before the package name, which is
27``strace`` in this example. This space is required since the ``:append``
28operator does not add the space.
29
30Furthermore, you must use ``:append`` instead of the ``+=`` operator if
31you want to avoid ordering issues. The reason for this is because doing
32so unconditionally appends to the variable and avoids ordering problems
33due to the variable being set in image recipes and ``.bbclass`` files
34with operators like ``?=``. Using ``:append`` ensures the operation
35takes effect.
36
37As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all
38images. It is possible to extend the syntax so that the variable applies
39to a specific image only. Here is an example::
40
41 IMAGE_INSTALL:append:pn-core-image-minimal = " strace"
42
43This example adds ``strace`` to the ``core-image-minimal`` image only.
44
45You can add packages using a similar approach through the
46:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
47``core-image-*`` images are affected.
48
49Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
50===============================================================================
51
52Another method for customizing your image is to enable or disable
53high-level image features by using the
54:term:`IMAGE_FEATURES` and
55:term:`EXTRA_IMAGE_FEATURES`
56variables. Although the functions for both variables are nearly
57equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
58a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
59``local.conf`` file, which is found in the :term:`Build Directory`.
60
61To understand how these features work, the best reference is
62:ref:`meta/classes/image.bbclass <ref-classes-image>`.
63This class lists out the available
64:term:`IMAGE_FEATURES` of which most map to package groups while some, such
65as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
66configuration settings.
67
68In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
69variable and then maps or configures the feature accordingly. Based on
70this information, the build system automatically adds the appropriate
71packages or configurations to the
72:term:`IMAGE_INSTALL` variable.
73Effectively, you are enabling extra features by extending the class or
74creating a custom class for use with specialized image ``.bb`` files.
75
76Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
77configuration file. Using a separate area from which to enable features
78with this variable helps you avoid overwriting the features in the image
79recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
80:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
81``meta/conf/bitbake.conf``.
82
83To illustrate how you can use these variables to modify your image,
84consider an example that selects the SSH server. The Yocto Project ships
85with two SSH servers you can use with your images: Dropbear and OpenSSH.
86Dropbear is a minimal SSH server appropriate for resource-constrained
87environments, while OpenSSH is a well-known standard SSH server
88implementation. By default, the ``core-image-sato`` image is configured
89to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
90images both include OpenSSH. The ``core-image-minimal`` image does not
91contain an SSH server.
92
93You can customize your image and change these defaults. Edit the
94:term:`IMAGE_FEATURES` variable in your recipe or use the
95:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
96configures the image you are working with to include
97``ssh-server-dropbear`` or ``ssh-server-openssh``.
98
99.. note::
100
101 See the ":ref:`ref-manual/features:image features`" section in the Yocto
102 Project Reference Manual for a complete list of image features that ship
103 with the Yocto Project.
104
105Customizing Images Using Custom .bb Files
106=========================================
107
108You can also customize an image by creating a custom recipe that defines
109additional software as part of the image. The following example shows
110the form for the two lines you need::
111
112 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
113 inherit core-image
114
115Defining the software using a custom recipe gives you total control over
116the contents of the image. It is important to use the correct names of
117packages in the :term:`IMAGE_INSTALL` variable. You must use the
118OpenEmbedded notation and not the Debian notation for the names (e.g.
119``glibc-dev`` instead of ``libc6-dev``).
120
121The other method for creating a custom image is to base it on an
122existing image. For example, if you want to create an image based on
123``core-image-sato`` but add the additional package ``strace`` to the
124image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
125``.bb`` and add the following line to the end of the copy::
126
127 IMAGE_INSTALL += "strace"
128
129Customizing Images Using Custom Package Groups
130==============================================
131
132For complex custom images, the best approach for customizing an image is
133to create a custom package group recipe that is used to build the image
134or images. A good example of a package group recipe is
135``meta/recipes-core/packagegroups/packagegroup-base.bb``.
136
137If you examine that recipe, you see that the :term:`PACKAGES` variable lists
138the package group packages to produce. The ``inherit packagegroup``
139statement sets appropriate default values and automatically adds
140``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
141package specified in the :term:`PACKAGES` statement.
142
143.. note::
144
145 The ``inherit packagegroup`` line should be located near the top of the
146 recipe, certainly before the :term:`PACKAGES` statement.
147
148For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
149and :term:`RRECOMMENDS` entries to provide a list of packages the parent
150task package should contain. You can see examples of these further down
151in the ``packagegroup-base.bb`` recipe.
152
153Here is a short, fabricated example showing the same basic pieces for a
154hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
155the variable :term:`PN` is the standard way to abbreviate the reference to
156the full packagegroup name ``packagegroup-custom``::
157
158 DESCRIPTION = "My Custom Package Groups"
159
160 inherit packagegroup
161
162 PACKAGES = "\
163 ${PN}-apps \
164 ${PN}-tools \
165 "
166
167 RDEPENDS:${PN}-apps = "\
168 dropbear \
169 portmap \
170 psplash"
171
172 RDEPENDS:${PN}-tools = "\
173 oprofile \
174 oprofileui-server \
175 lttng-tools"
176
177 RRECOMMENDS:${PN}-tools = "\
178 kernel-module-oprofile"
179
180In the previous example, two package group packages are created with
181their dependencies and their recommended package dependencies listed:
182``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
183build an image using these package group packages, you need to add
184``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
185:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
186areas of this section.
187
188Customizing an Image Hostname
189=============================
190
191By default, the configured hostname (i.e. ``/etc/hostname``) in an image
192is the same as the machine name. For example, if
193:term:`MACHINE` equals "qemux86", the
194configured hostname written to ``/etc/hostname`` is "qemux86".
195
196You can customize this name by altering the value of the "hostname"
197variable in the ``base-files`` recipe using either an append file or a
198configuration file. Use the following in an append file::
199
200 hostname = "myhostname"
201
202Use the following in a configuration file::
203
204 hostname:pn-base-files = "myhostname"
205
206Changing the default value of the variable "hostname" can be useful in
207certain situations. For example, suppose you need to do extensive
208testing on an image and you would like to easily identify the image
209under test from existing images with typical default hostnames. In this
210situation, you could change the default hostname to "testme", which
211results in all the images using the name "testme". Once testing is
212complete and you do not need to rebuild the image for test any longer,
213you can easily reset the default hostname.
214
215Another point of interest is that if you unset the variable, the image
216will have no default hostname in the filesystem. Here is an example that
217unsets the variable in a configuration file::
218
219 hostname:pn-base-files = ""
220
221Having no default hostname in the filesystem is suitable for
222environments that use dynamic hostnames such as virtual machines.
223
diff --git a/documentation/dev-manual/debugging.rst b/documentation/dev-manual/debugging.rst
new file mode 100644
index 0000000000..fea2cb30a1
--- /dev/null
+++ b/documentation/dev-manual/debugging.rst
@@ -0,0 +1,1245 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Debugging Tools and Techniques
4******************************
5
6The exact method for debugging build failures depends on the nature of
7the problem and on the system's area from which the bug originates.
8Standard debugging practices such as comparison against the last known
9working version with examination of the changes and the re-application
10of steps to identify the one causing the problem are valid for the Yocto
11Project just as they are for any other system. Even though it is
12impossible to detail every possible potential failure, this section
13provides some general tips to aid in debugging given a variety of
14situations.
15
16.. note::
17
18 A useful feature for debugging is the error reporting tool.
19 Configuring the Yocto Project to use this tool causes the
20 OpenEmbedded build system to produce error reporting commands as part
21 of the console output. You can enter the commands after the build
22 completes to log error information into a common database, that can
23 help you figure out what might be going wrong. For information on how
24 to enable and use this feature, see the
25 ":ref:`dev-manual/error-reporting-tool:using the error reporting tool`"
26 section.
27
28The following list shows the debugging topics in the remainder of this
29section:
30
31- ":ref:`dev-manual/debugging:viewing logs from failed tasks`" describes
32 how to find and view logs from tasks that failed during the build
33 process.
34
35- ":ref:`dev-manual/debugging:viewing variable values`" describes how to
36 use the BitBake ``-e`` option to examine variable values after a
37 recipe has been parsed.
38
39- ":ref:`dev-manual/debugging:viewing package information with \`\`oe-pkgdata-util\`\``"
40 describes how to use the ``oe-pkgdata-util`` utility to query
41 :term:`PKGDATA_DIR` and
42 display package-related information for built packages.
43
44- ":ref:`dev-manual/debugging:viewing dependencies between recipes and tasks`"
45 describes how to use the BitBake ``-g`` option to display recipe
46 dependency information used during the build.
47
48- ":ref:`dev-manual/debugging:viewing task variable dependencies`" describes
49 how to use the ``bitbake-dumpsig`` command in conjunction with key
50 subdirectories in the :term:`Build Directory` to determine variable
51 dependencies.
52
53- ":ref:`dev-manual/debugging:running specific tasks`" describes
54 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
55 to run specific tasks in the build chain. It can be useful to run
56 tasks "out-of-order" when trying isolate build issues.
57
58- ":ref:`dev-manual/debugging:general BitBake problems`" describes how
59 to use BitBake's ``-D`` debug output option to reveal more about what
60 BitBake is doing during the build.
61
62- ":ref:`dev-manual/debugging:building with no dependencies`"
63 describes how to use the BitBake ``-b`` option to build a recipe
64 while ignoring dependencies.
65
66- ":ref:`dev-manual/debugging:recipe logging mechanisms`"
67 describes how to use the many recipe logging functions to produce
68 debugging output and report errors and warnings.
69
70- ":ref:`dev-manual/debugging:debugging parallel make races`"
71 describes how to debug situations where the build consists of several
72 parts that are run simultaneously and when the output or result of
73 one part is not ready for use with a different part of the build that
74 depends on that output.
75
76- ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) remotely`"
77 describes how to use GDB to allow you to examine running programs, which can
78 help you fix problems.
79
80- ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) on the target`"
81 describes how to use GDB directly on target hardware for debugging.
82
83- ":ref:`dev-manual/debugging:other debugging tips`" describes
84 miscellaneous debugging tips that can be useful.
85
86Viewing Logs from Failed Tasks
87==============================
88
89You can find the log for a task in the file
90``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
91For example, the log for the
92:ref:`ref-tasks-compile` task of the
93QEMU minimal image for the x86 machine (``qemux86``) might be in
94``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
95To see the commands :term:`BitBake` ran
96to generate a log, look at the corresponding ``run.do_``\ `taskname` file
97in the same directory.
98
99``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
100links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
101``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
102when it ran. The symlinks always point to the files corresponding to the
103most recent run.
104
105Viewing Variable Values
106=======================
107
108Sometimes you need to know the value of a variable as a result of
109BitBake's parsing step. This could be because some unexpected behavior
110occurred in your project. Perhaps an attempt to :ref:`modify a variable
111<bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
112variables>` did not work out as expected.
113
114BitBake's ``-e`` option is used to display variable values after
115parsing. The following command displays the variable values after the
116configuration files (i.e. ``local.conf``, ``bblayers.conf``,
117``bitbake.conf`` and so forth) have been parsed::
118
119 $ bitbake -e
120
121The following command displays variable values after a specific recipe has
122been parsed. The variables include those from the configuration as well::
123
124 $ bitbake -e recipename
125
126.. note::
127
128 Each recipe has its own private set of variables (datastore).
129 Internally, after parsing the configuration, a copy of the resulting
130 datastore is made prior to parsing each recipe. This copying implies
131 that variables set in one recipe will not be visible to other
132 recipes.
133
134 Likewise, each task within a recipe gets a private datastore based on
135 the recipe datastore, which means that variables set within one task
136 will not be visible to other tasks.
137
138In the output of ``bitbake -e``, each variable is preceded by a
139description of how the variable got its value, including temporary
140values that were later overridden. This description also includes
141variable flags (varflags) set on the variable. The output can be very
142helpful during debugging.
143
144Variables that are exported to the environment are preceded by
145``export`` in the output of ``bitbake -e``. See the following example::
146
147 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
148
149In addition to variable values, the output of the ``bitbake -e`` and
150``bitbake -e`` recipe commands includes the following information:
151
152- The output starts with a tree listing all configuration files and
153 classes included globally, recursively listing the files they include
154 or inherit in turn. Much of the behavior of the OpenEmbedded build
155 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
156 implemented in the :ref:`ref-classes-base` class and the
157 classes it inherits, rather than being built into BitBake itself.
158
159- After the variable values, all functions appear in the output. For
160 shell functions, variables referenced within the function body are
161 expanded. If a function has been modified using overrides or using
162 override-style operators like ``:append`` and ``:prepend``, then the
163 final assembled function body appears in the output.
164
165Viewing Package Information with ``oe-pkgdata-util``
166====================================================
167
168You can use the ``oe-pkgdata-util`` command-line utility to query
169:term:`PKGDATA_DIR` and display
170various package-related information. When you use the utility, you must
171use it to view information on packages that have already been built.
172
173Following are a few of the available ``oe-pkgdata-util`` subcommands.
174
175.. note::
176
177 You can use the standard \* and ? globbing wildcards as part of
178 package names and paths.
179
180- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
181 that have been built, optionally limiting the match to packages that
182 match pattern.
183
184- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
185 files and directories contained in the given packages.
186
187 .. note::
188
189 A different way to view the contents of a package is to look at
190 the
191 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
192 directory of the recipe that generates the package. This directory
193 is created by the
194 :ref:`ref-tasks-package` task
195 and has one subdirectory for each package the recipe generates,
196 which contains the files stored in that package.
197
198 If you want to inspect the ``${WORKDIR}/packages-split``
199 directory, make sure that :ref:`ref-classes-rm-work` is not
200 enabled when you build the recipe.
201
202- ``oe-pkgdata-util find-path path ...``: Lists the names of
203 the packages that contain the given paths. For example, the following
204 tells us that ``/usr/share/man/man1/make.1`` is contained in the
205 ``make-doc`` package::
206
207 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
208 make-doc: /usr/share/man/man1/make.1
209
210- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
211 of the recipes that produce the given packages.
212
213For more information on the ``oe-pkgdata-util`` command, use the help
214facility::
215
216 $ oe-pkgdata-util --help
217 $ oe-pkgdata-util subcommand --help
218
219Viewing Dependencies Between Recipes and Tasks
220==============================================
221
222Sometimes it can be hard to see why BitBake wants to build other recipes
223before the one you have specified. Dependency information can help you
224understand why a recipe is built.
225
226To generate dependency information for a recipe, run the following
227command::
228
229 $ bitbake -g recipename
230
231This command writes the following files in the current directory:
232
233- ``pn-buildlist``: A list of recipes/targets involved in building
234 `recipename`. "Involved" here means that at least one task from the
235 recipe needs to run when building `recipename` from scratch. Targets
236 that are in
237 :term:`ASSUME_PROVIDED`
238 are not listed.
239
240- ``task-depends.dot``: A graph showing dependencies between tasks.
241
242The graphs are in :wikipedia:`DOT <DOT_%28graph_description_language%29>`
243format and can be converted to images (e.g. using the ``dot`` tool from
244`Graphviz <https://www.graphviz.org/>`__).
245
246.. note::
247
248 - DOT files use a plain text format. The graphs generated using the
249 ``bitbake -g`` command are often so large as to be difficult to
250 read without special pruning (e.g. with BitBake's ``-I`` option)
251 and processing. Despite the form and size of the graphs, the
252 corresponding ``.dot`` files can still be possible to read and
253 provide useful information.
254
255 As an example, the ``task-depends.dot`` file contains lines such
256 as the following::
257
258 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
259
260 The above example line reveals that the
261 :ref:`ref-tasks-configure`
262 task in ``libxslt`` depends on the
263 :ref:`ref-tasks-populate_sysroot`
264 task in ``libxml2``, which is a normal
265 :term:`DEPENDS` dependency
266 between the two recipes.
267
268 - For an example of how ``.dot`` files can be processed, see the
269 ``scripts/contrib/graph-tool`` Python script, which finds and
270 displays paths between graph nodes.
271
272You can use a different method to view dependency information by using
273the following command::
274
275 $ bitbake -g -u taskexp recipename
276
277This command
278displays a GUI window from which you can view build-time and runtime
279dependencies for the recipes involved in building recipename.
280
281Viewing Task Variable Dependencies
282==================================
283
284As mentioned in the
285":ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`"
286section of the BitBake User Manual, BitBake tries to automatically determine
287what variables a task depends on so that it can rerun the task if any values of
288the variables change. This determination is usually reliable. However, if you
289do things like construct variable names at runtime, then you might have to
290manually declare dependencies on those variables using ``vardeps`` as described
291in the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
292section of the BitBake User Manual.
293
294If you are unsure whether a variable dependency is being picked up
295automatically for a given task, you can list the variable dependencies
296BitBake has determined by doing the following:
297
298#. Build the recipe containing the task::
299
300 $ bitbake recipename
301
302#. Inside the :term:`STAMPS_DIR`
303 directory, find the signature data (``sigdata``) file that
304 corresponds to the task. The ``sigdata`` files contain a pickled
305 Python database of all the metadata that went into creating the input
306 checksum for the task. As an example, for the
307 :ref:`ref-tasks-fetch` task of the
308 ``db`` recipe, the ``sigdata`` file might be found in the following
309 location::
310
311 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
312
313 For tasks that are accelerated through the shared state
314 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
315 additional ``siginfo`` file is written into
316 :term:`SSTATE_DIR` along with
317 the cached task output. The ``siginfo`` files contain exactly the
318 same information as ``sigdata`` files.
319
320#. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
321 is an example::
322
323 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
324
325 In the output of the above command, you will find a line like the
326 following, which lists all the (inferred) variable dependencies for
327 the task. This list also includes indirect dependencies from
328 variables depending on other variables, recursively::
329
330 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
331
332 .. note::
333
334 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
335 These functions in turn depend on the variables they reference.
336
337 The output of ``bitbake-dumpsig`` also includes the value each
338 variable had, a list of dependencies for each variable, and
339 :term:`BB_BASEHASH_IGNORE_VARS`
340 information.
341
342There is also a ``bitbake-diffsigs`` command for comparing two
343``siginfo`` or ``sigdata`` files. This command can be helpful when
344trying to figure out what changed between two versions of a task. If you
345call ``bitbake-diffsigs`` with just one file, the command behaves like
346``bitbake-dumpsig``.
347
348You can also use BitBake to dump out the signature construction
349information without executing tasks by using either of the following
350BitBake command-line options::
351
352 ‐‐dump-signatures=SIGNATURE_HANDLER
353 -S SIGNATURE_HANDLER
354
355
356.. note::
357
358 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
359 dump only the signature or compare the dumped signature with the cached one,
360 respectively.
361
362Using BitBake with either of these options causes BitBake to dump out
363``sigdata`` files in the ``stamps`` directory for every task it would
364have executed instead of building the specified target package.
365
366Viewing Metadata Used to Create the Input Signature of a Shared State Task
367==========================================================================
368
369Seeing what metadata went into creating the input signature of a shared
370state (sstate) task can be a useful debugging aid. This information is
371available in signature information (``siginfo``) files in
372:term:`SSTATE_DIR`. For
373information on how to view and interpret information in ``siginfo``
374files, see the
375":ref:`dev-manual/debugging:viewing task variable dependencies`" section.
376
377For conceptual information on shared state, see the
378":ref:`overview-manual/concepts:shared state`"
379section in the Yocto Project Overview and Concepts Manual.
380
381Invalidating Shared State to Force a Task to Run
382================================================
383
384The OpenEmbedded build system uses
385:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
386:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
387rebuilding tasks. Collectively, this scheme is known as "shared state
388code".
389
390As with all schemes, this one has some drawbacks. It is possible that
391you could make implicit changes to your code that the checksum
392calculations do not take into account. These implicit changes affect a
393task's output but do not trigger the shared state code into rebuilding a
394recipe. Consider an example during which a tool changes its output.
395Assume that the output of ``rpmdeps`` changes. The result of the change
396should be that all the ``package`` and ``package_write_rpm`` shared
397state cache items become invalid. However, because the change to the
398output is external to the code and therefore implicit, the associated
399shared state cache items do not become invalidated. In this case, the
400build process uses the cached items rather than running the task again.
401Obviously, these types of implicit changes can cause problems.
402
403To avoid these problems during the build, you need to understand the
404effects of any changes you make. Realize that changes you make directly
405to a function are automatically factored into the checksum calculation.
406Thus, these explicit changes invalidate the associated area of shared
407state cache. However, you need to be aware of any implicit changes that
408are not obvious changes to the code and could affect the output of a
409given task.
410
411When you identify an implicit change, you can easily take steps to
412invalidate the cache and force the tasks to run. The steps you can take
413are as simple as changing a function's comments in the source code. For
414example, to invalidate package shared state files, change the comment
415statements of
416:ref:`ref-tasks-package` or the
417comments of one of the functions it calls. Even though the change is
418purely cosmetic, it causes the checksum to be recalculated and forces
419the build system to run the task again.
420
421.. note::
422
423 For an example of a commit that makes a cosmetic change to invalidate
424 shared state, see this
425 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
426
427Running Specific Tasks
428======================
429
430Any given recipe consists of a set of tasks. The standard BitBake
431behavior in most cases is: :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`,
432:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, :ref:`ref-tasks-install`, :ref:`ref-tasks-package`,
433:ref:`do_package_write_* <ref-tasks-package_write_deb>`, and :ref:`ref-tasks-build`. The default task is
434:ref:`ref-tasks-build` and any tasks on which it depends build first. Some tasks,
435such as :ref:`ref-tasks-devshell`, are not part of the default build chain. If you
436wish to run a task that is not part of the default build chain, you can
437use the ``-c`` option in BitBake. Here is an example::
438
439 $ bitbake matchbox-desktop -c devshell
440
441The ``-c`` option respects task dependencies, which means that all other
442tasks (including tasks from other recipes) that the specified task
443depends on will be run before the task. Even when you manually specify a
444task to run with ``-c``, BitBake will only run the task if it considers
445it "out of date". See the
446":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
447section in the Yocto Project Overview and Concepts Manual for how
448BitBake determines whether a task is "out of date".
449
450If you want to force an up-to-date task to be rerun (e.g. because you
451made manual modifications to the recipe's
452:term:`WORKDIR` that you want to try
453out), then you can use the ``-f`` option.
454
455.. note::
456
457 The reason ``-f`` is never required when running the
458 :ref:`ref-tasks-devshell` task is because the
459 [\ :ref:`nostamp <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
460 variable flag is already set for the task.
461
462The following example shows one way you can use the ``-f`` option::
463
464 $ bitbake matchbox-desktop
465 .
466 .
467 make some changes to the source code in the work directory
468 .
469 .
470 $ bitbake matchbox-desktop -c compile -f
471 $ bitbake matchbox-desktop
472
473This sequence first builds and then recompiles ``matchbox-desktop``. The
474last command reruns all tasks (basically the packaging tasks) after the
475compile. BitBake recognizes that the :ref:`ref-tasks-compile` task was rerun and
476therefore understands that the other tasks also need to be run again.
477
478Another, shorter way to rerun a task and all
479:ref:`ref-manual/tasks:normal recipe build tasks`
480that depend on it is to use the ``-C`` option.
481
482.. note::
483
484 This option is upper-cased and is separate from the ``-c``
485 option, which is lower-cased.
486
487Using this option invalidates the given task and then runs the
488:ref:`ref-tasks-build` task, which is
489the default task if no task is given, and the tasks on which it depends.
490You could replace the final two commands in the previous example with
491the following single command::
492
493 $ bitbake matchbox-desktop -C compile
494
495Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
496the input checksum of the specified task. This tainting indirectly
497causes the task and its dependent tasks to be rerun through the normal
498task dependency mechanisms.
499
500.. note::
501
502 BitBake explicitly keeps track of which tasks have been tainted in
503 this fashion, and will print warnings such as the following for
504 builds involving such tasks:
505
506 .. code-block:: none
507
508 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
509
510
511 The purpose of the warning is to let you know that the work directory
512 and build output might not be in the clean state they would be in for
513 a "normal" build, depending on what actions you took. To get rid of
514 such warnings, you can remove the work directory and rebuild the
515 recipe, as follows::
516
517 $ bitbake matchbox-desktop -c clean
518 $ bitbake matchbox-desktop
519
520
521You can view a list of tasks in a given package by running the
522:ref:`ref-tasks-listtasks` task as follows::
523
524 $ bitbake matchbox-desktop -c listtasks
525
526The results appear as output to the console and are also in
527the file ``${WORKDIR}/temp/log.do_listtasks``.
528
529General BitBake Problems
530========================
531
532You can see debug output from BitBake by using the ``-D`` option. The
533debug output gives more information about what BitBake is doing and the
534reason behind it. Each ``-D`` option you use increases the logging
535level. The most common usage is ``-DDD``.
536
537The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
538chose a certain version of a package or why BitBake picked a certain
539provider. This command could also help you in a situation where you
540think BitBake did something unexpected.
541
542Building with No Dependencies
543=============================
544
545To build a specific recipe (``.bb`` file), you can use the following
546command form::
547
548 $ bitbake -b somepath/somerecipe.bb
549
550This command form does
551not check for dependencies. Consequently, you should use it only when
552you know existing dependencies have been met.
553
554.. note::
555
556 You can also specify fragments of the filename. In this case, BitBake
557 checks for a unique match.
558
559Recipe Logging Mechanisms
560=========================
561
562The Yocto Project provides several logging functions for producing
563debugging output and reporting errors and warnings. For Python
564functions, the following logging functions are available. All of these functions
565log to ``${T}/log.do_``\ `task`, and can also log to standard output
566(stdout) with the right settings:
567
568- ``bb.plain(msg)``: Writes msg as is to the log while also
569 logging to stdout.
570
571- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
572 stdout if BitBake is called with "-v".
573
574- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the log. Also logs to
575 stdout if the log level is greater than or equal to level. See the
576 ":ref:`bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`"
577 option in the BitBake User Manual for more information.
578
579- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
580 logging to stdout.
581
582- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
583 logging to standard out (stdout).
584
585 .. note::
586
587 Calling this function does not cause the task to fail.
588
589- ``bb.fatal(msg)``: This logging function is similar to
590 ``bb.error(msg)`` but also causes the calling task to fail.
591
592 .. note::
593
594 ``bb.fatal()`` raises an exception, which means you do not need to put a
595 "return" statement after the function.
596
597The same logging functions are also available in shell functions, under
598the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
599and ``bbfatal``. The :ref:`ref-classes-logging` class
600implements these functions. See that class in the ``meta/classes``
601folder of the :term:`Source Directory` for information.
602
603Logging With Python
604-------------------
605
606When creating recipes using Python and inserting code that handles build
607logs, keep in mind the goal is to have informative logs while keeping
608the console as "silent" as possible. Also, if you want status messages
609in the log, use the "debug" loglevel.
610
611Following is an example written in Python. The code handles logging for
612a function that determines the number of tasks needed to be run. See the
613":ref:`ref-tasks-listtasks`"
614section for additional information::
615
616 python do_listtasks() {
617 bb.debug(2, "Starting to figure out the task list")
618 if noteworthy_condition:
619 bb.note("There are 47 tasks to run")
620 bb.debug(2, "Got to point xyz")
621 if warning_trigger:
622 bb.warn("Detected warning_trigger, this might be a problem later.")
623 if recoverable_error:
624 bb.error("Hit recoverable_error, you really need to fix this!")
625 if fatal_error:
626 bb.fatal("fatal_error detected, unable to print the task list")
627 bb.plain("The tasks present are abc")
628 bb.debug(2, "Finished figuring out the tasklist")
629 }
630
631Logging With Bash
632-----------------
633
634When creating recipes using Bash and inserting code that handles build
635logs, you have the same goals --- informative with minimal console output.
636The syntax you use for recipes written in Bash is similar to that of
637recipes written in Python described in the previous section.
638
639Following is an example written in Bash. The code logs the progress of
640the ``do_my_function`` function::
641
642 do_my_function() {
643 bbdebug 2 "Running do_my_function"
644 if [ exceptional_condition ]; then
645 bbnote "Hit exceptional_condition"
646 fi
647 bbdebug 2 "Got to point xyz"
648 if [ warning_trigger ]; then
649 bbwarn "Detected warning_trigger, this might cause a problem later."
650 fi
651 if [ recoverable_error ]; then
652 bberror "Hit recoverable_error, correcting"
653 fi
654 if [ fatal_error ]; then
655 bbfatal "fatal_error detected"
656 fi
657 bbdebug 2 "Completed do_my_function"
658 }
659
660
661Debugging Parallel Make Races
662=============================
663
664A parallel ``make`` race occurs when the build consists of several parts
665that are run simultaneously and a situation occurs when the output or
666result of one part is not ready for use with a different part of the
667build that depends on that output. Parallel make races are annoying and
668can sometimes be difficult to reproduce and fix. However, there are some simple
669tips and tricks that can help you debug and fix them. This section
670presents a real-world example of an error encountered on the Yocto
671Project autobuilder and the process used to fix it.
672
673.. note::
674
675 If you cannot properly fix a ``make`` race condition, you can work around it
676 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
677 variables.
678
679The Failure
680-----------
681
682For this example, assume that you are building an image that depends on
683the "neard" package. And, during the build, BitBake runs into problems
684and creates the following output.
685
686.. note::
687
688 This example log file has longer lines artificially broken to make
689 the listing easier to read.
690
691If you examine the output or the log file, you see the failure during
692``make``:
693
694.. code-block:: none
695
696 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
697 | DEBUG: Executing shell function do_compile
698 | NOTE: make -j 16
699 | make --no-print-directory all-am
700 | /bin/mkdir -p include/near
701 | /bin/mkdir -p include/near
702 | /bin/mkdir -p include/near
703 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
704 0.14-r0/neard-0.14/include/types.h include/near/types.h
705 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
706 0.14-r0/neard-0.14/include/log.h include/near/log.h
707 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
708 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
709 | /bin/mkdir -p include/near
710 | /bin/mkdir -p include/near
711 | /bin/mkdir -p include/near
712 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
713 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
714 | /bin/mkdir -p include/near
715 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
716 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
717 | /bin/mkdir -p include/near
718 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
719 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
720 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
721 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
722 | /bin/mkdir -p include/near
723 | /bin/mkdir -p include/near
724 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
725 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
726 | /bin/mkdir -p include/near
727 | /bin/mkdir -p include/near
728 | /bin/mkdir -p include/near
729 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
730 0.14-r0/neard-0.14/include/device.h include/near/device.h
731 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
732 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
733 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
734 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
735 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
736 0.14-r0/neard-0.14/include/version.h include/near/version.h
737 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
738 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
739 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
740 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/
741 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
742 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
743 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
744 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/
745 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/
746 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
747 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
748 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
749 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
750 -o tools/snep-send.o tools/snep-send.c
751 | In file included from tools/snep-send.c:16:0:
752 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
753 | #include <near/dbus.h>
754 | ^
755 | compilation terminated.
756 | make[1]: *** [tools/snep-send.o] Error 1
757 | make[1]: *** Waiting for unfinished jobs....
758 | make: *** [all] Error 2
759 | ERROR: oe_runmake failed
760
761Reproducing the Error
762---------------------
763
764Because race conditions are intermittent, they do not manifest
765themselves every time you do the build. In fact, most times the build
766will complete without problems even though the potential race condition
767exists. Thus, once the error surfaces, you need a way to reproduce it.
768
769In this example, compiling the "neard" package is causing the problem.
770So the first thing to do is build "neard" locally. Before you start the
771build, set the
772:term:`PARALLEL_MAKE` variable
773in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
774high value for :term:`PARALLEL_MAKE` increases the chances of the race
775condition showing up::
776
777 $ bitbake neard
778
779Once the local build for "neard" completes, start a ``devshell`` build::
780
781 $ bitbake neard -c devshell
782
783For information on how to use a ``devshell``, see the
784":ref:`dev-manual/development-shell:using a development shell`" section.
785
786In the ``devshell``, do the following::
787
788 $ make clean
789 $ make tools/snep-send.o
790
791The ``devshell`` commands cause the failure to clearly
792be visible. In this case, there is a missing dependency for the ``neard``
793Makefile target. Here is some abbreviated, sample output with the
794missing dependency clearly visible at the end::
795
796 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
797 .
798 .
799 .
800 tools/snep-send.c
801 In file included from tools/snep-send.c:16:0:
802 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
803 #include <near/dbus.h>
804 ^
805 compilation terminated.
806 make: *** [tools/snep-send.o] Error 1
807 $
808
809
810Creating a Patch for the Fix
811----------------------------
812
813Because there is a missing dependency for the Makefile target, you need
814to patch the ``Makefile.am`` file, which is generated from
815``Makefile.in``. You can use Quilt to create the patch::
816
817 $ quilt new parallelmake.patch
818 Patch patches/parallelmake.patch is now on top
819 $ quilt add Makefile.am
820 File Makefile.am added to patch patches/parallelmake.patch
821
822For more information on using Quilt, see the
823":ref:`dev-manual/quilt:using quilt in your workflow`" section.
824
825At this point you need to make the edits to ``Makefile.am`` to add the
826missing dependency. For our example, you have to add the following line
827to the file::
828
829 tools/snep-send.$(OBJEXT): include/near/dbus.h
830
831Once you have edited the file, use the ``refresh`` command to create the
832patch::
833
834 $ quilt refresh
835 Refreshed patch patches/parallelmake.patch
836
837Once the patch file is created, you need to add it back to the originating
838recipe folder. Here is an example assuming a top-level
839:term:`Source Directory` named ``poky``::
840
841 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
842
843The final thing you need to do to implement the fix in the build is to
844update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
845:term:`SRC_URI` statement includes
846the patch file. The recipe file is in the folder above the patch. Here
847is what the edited :term:`SRC_URI` statement would look like::
848
849 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
850 file://neard.in \
851 file://neard.service.in \
852 file://parallelmake.patch \
853 "
854
855With the patch complete and moved to the correct folder and the
856:term:`SRC_URI` statement updated, you can exit the ``devshell``::
857
858 $ exit
859
860Testing the Build
861-----------------
862
863With everything in place, you can get back to trying the build again
864locally::
865
866 $ bitbake neard
867
868This build should succeed.
869
870Now you can open up a ``devshell`` again and repeat the clean and make
871operations as follows::
872
873 $ bitbake neard -c devshell
874 $ make clean
875 $ make tools/snep-send.o
876
877The build should work without issue.
878
879As with all solved problems, if they originated upstream, you need to
880submit the fix for the recipe in OE-Core and upstream so that the
881problem is taken care of at its source. See the
882":doc:`../contributor-guide/submit-changes`" section for more information.
883
884Debugging With the GNU Project Debugger (GDB) Remotely
885======================================================
886
887GDB allows you to examine running programs, which in turn helps you to
888understand and fix problems. It also allows you to perform post-mortem
889style analysis of program crashes. GDB is available as a package within
890the Yocto Project and is installed in SDK images by default. See the
891":ref:`ref-manual/images:Images`" chapter in the Yocto
892Project Reference Manual for a description of these images. You can find
893information on GDB at https://sourceware.org/gdb/.
894
895.. note::
896
897 For best results, install debug (``-dbg``) packages for the applications you
898 are going to debug. Doing so makes extra debug symbols available that give
899 you more meaningful output.
900
901Sometimes, due to memory or disk space constraints, it is not possible
902to use GDB directly on the remote target to debug applications. These
903constraints arise because GDB needs to load the debugging information
904and the binaries of the process being debugged. Additionally, GDB needs
905to perform many computations to locate information such as function
906names, variable names and values, stack traces and so forth --- even
907before starting the debugging process. These extra computations place
908more load on the target system and can alter the characteristics of the
909program being debugged.
910
911To help get past the previously mentioned constraints, there are two
912methods you can use: running a debuginfod server and using gdbserver.
913
914Using the debuginfod server method
915----------------------------------
916
917``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
918Running a ``debuginfod`` server makes debug symbols readily available,
919which means you don't need to download debugging information
920and the binaries of the process being debugged. You can just fetch
921debug symbols from the server.
922
923To run a ``debuginfod`` server, you need to do the following:
924
925- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
926 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
927 If not, set in your distro config file or in ``local.conf``::
928
929 DISTRO_FEATURES:append = " debuginfod"
930
931 This distro feature enables the server and client library in ``elfutils``,
932 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
933
934- Run the following commands to launch the ``debuginfod`` server on the host::
935
936 $ oe-debuginfod
937
938- To use ``debuginfod`` on the target, you need to know the ip:port where
939 ``debuginfod`` is listening on the host (port defaults to 8002), and export
940 that into the shell environment, for example in ``qemu``::
941
942 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
943
944- Then debug info fetching should simply work when running the target ``gdb``,
945 ``readelf`` or ``objdump``, for example::
946
947 root@qemux86-64:~# gdb /bin/cat
948 ...
949 Reading symbols from /bin/cat...
950 Downloading separate debug info for /bin/cat...
951 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
952
953- It's also possible to use ``debuginfod-find`` to just query the server::
954
955 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
956 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
957
958
959Using the gdbserver method
960--------------------------
961
962gdbserver, which runs on the remote target and does not load any
963debugging information from the debugged process. Instead, a GDB instance
964processes the debugging information that is run on a remote computer -
965the host GDB. The host GDB then sends control commands to gdbserver to
966make it stop or start the debugged program, as well as read or write
967memory regions of that debugged program. All the debugging information
968loaded and processed as well as all the heavy debugging is done by the
969host GDB. Offloading these processes gives the gdbserver running on the
970target a chance to remain small and fast.
971
972Because the host GDB is responsible for loading the debugging
973information and for doing the necessary processing to make actual
974debugging happen, you have to make sure the host can access the
975unstripped binaries complete with their debugging information and also
976be sure the target is compiled with no optimizations. The host GDB must
977also have local access to all the libraries used by the debugged
978program. Because gdbserver does not need any local debugging
979information, the binaries on the remote target can remain stripped.
980However, the binaries must also be compiled without optimization so they
981match the host's binaries.
982
983To remain consistent with GDB documentation and terminology, the binary
984being debugged on the remote target machine is referred to as the
985"inferior" binary. For documentation on GDB see the `GDB
986site <https://sourceware.org/gdb/documentation/>`__.
987
988The following steps show you how to debug using the GNU project
989debugger.
990
991#. *Configure your build system to construct the companion debug
992 filesystem:*
993
994 In your ``local.conf`` file, set the following::
995
996 IMAGE_GEN_DEBUGFS = "1"
997 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
998
999 These options cause the
1000 OpenEmbedded build system to generate a special companion filesystem
1001 fragment, which contains the matching source and debug symbols to
1002 your deployable filesystem. The build system does this by looking at
1003 what is in the deployed filesystem, and pulling the corresponding
1004 ``-dbg`` packages.
1005
1006 The companion debug filesystem is not a complete filesystem, but only
1007 contains the debug fragments. This filesystem must be combined with
1008 the full filesystem for debugging. Subsequent steps in this procedure
1009 show how to combine the partial filesystem with the full filesystem.
1010
1011#. *Configure the system to include gdbserver in the target filesystem:*
1012
1013 Make the following addition in your ``local.conf`` file::
1014
1015 EXTRA_IMAGE_FEATURES:append = " tools-debug"
1016
1017 The change makes
1018 sure the ``gdbserver`` package is included.
1019
1020#. *Build the environment:*
1021
1022 Use the following command to construct the image and the companion
1023 Debug Filesystem::
1024
1025 $ bitbake image
1026
1027 Build the cross GDB component and
1028 make it available for debugging. Build the SDK that matches the
1029 image. Building the SDK is best for a production build that can be
1030 used later for debugging, especially during long term maintenance::
1031
1032 $ bitbake -c populate_sdk image
1033
1034 Alternatively, you can build the minimal toolchain components that
1035 match the target. Doing so creates a smaller than typical SDK and
1036 only contains a minimal set of components with which to build simple
1037 test applications, as well as run the debugger::
1038
1039 $ bitbake meta-toolchain
1040
1041 A final method is to build Gdb itself within the build system::
1042
1043 $ bitbake gdb-cross-<architecture>
1044
1045 Doing so produces a temporary copy of
1046 ``cross-gdb`` you can use for debugging during development. While
1047 this is the quickest approach, the two previous methods in this step
1048 are better when considering long-term maintenance strategies.
1049
1050 .. note::
1051
1052 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
1053 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
1054 actual name you want to use.
1055
1056#. *Set up the* ``debugfs``\ *:*
1057
1058 Run the following commands to set up the ``debugfs``::
1059
1060 $ mkdir debugfs
1061 $ cd debugfs
1062 $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2
1063 $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2
1064
1065#. *Set up GDB:*
1066
1067 Install the SDK (if you built one) and then source the correct
1068 environment file. Sourcing the environment file puts the SDK in your
1069 ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger.
1070
1071 If you are using the build system, Gdb is located in
1072 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
1073
1074#. *Boot the target:*
1075
1076 For information on how to run QEMU, see the `QEMU
1077 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
1078
1079 .. note::
1080
1081 Be sure to verify that your host can access the target via TCP.
1082
1083#. *Debug a program:*
1084
1085 Debugging a program involves running gdbserver on the target and then
1086 running Gdb on the host. The example in this step debugs ``gzip``:
1087
1088 .. code-block:: shell
1089
1090 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
1091
1092 For
1093 additional gdbserver options, see the `GDB Server
1094 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
1095
1096 After running gdbserver on the target, you need to run Gdb on the
1097 host and configure it and connect to the target. Use these commands::
1098
1099 $ cd directory-holding-the-debugfs-directory
1100 $ arch-gdb
1101 (gdb) set sysroot debugfs
1102 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
1103 (gdb) target remote IP-of-target:1234
1104
1105 At this
1106 point, everything should automatically load (i.e. matching binaries,
1107 symbols and headers).
1108
1109 .. note::
1110
1111 The Gdb ``set`` commands in the previous example can be placed into the
1112 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
1113 commands are in that file.
1114
1115#. *Deploying without a full image rebuild:*
1116
1117 In many cases, during development you want a quick method to deploy a
1118 new binary to the target and debug it, without waiting for a full
1119 image build.
1120
1121 One approach to solving this situation is to just build the component
1122 you want to debug. Once you have built the component, copy the
1123 executable directly to both the target and the host ``debugfs``.
1124
1125 If the binary is processed through the debug splitting in
1126 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
1127 contents and corresponding ``/usr/src/debug`` files) from the work
1128 directory. Here is an example::
1129
1130 $ bitbake bash
1131 $ bitbake -c devshell bash
1132 $ cd ..
1133 $ scp packages-split/bash/bin/bash target:/bin/bash
1134 $ cp -a packages-split/bash-dbg/\* path/debugfs
1135
1136Debugging with the GNU Project Debugger (GDB) on the Target
1137===========================================================
1138
1139The previous section addressed using GDB remotely for debugging
1140purposes, which is the most usual case due to the inherent hardware
1141limitations on many embedded devices. However, debugging in the target
1142hardware itself is also possible with more powerful devices. This
1143section describes what you need to do in order to support using GDB to
1144debug on the target hardware.
1145
1146To support this kind of debugging, you need do the following:
1147
1148- Ensure that GDB is on the target. You can do this by making
1149 the following addition to your ``local.conf`` file::
1150
1151 EXTRA_IMAGE_FEATURES:append = " tools-debug"
1152
1153- Ensure that debug symbols are present. You can do so by adding the
1154 corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`::
1155
1156 IMAGE_INSTALL:append = " packagename-dbg"
1157
1158 Alternatively, you can add the following to ``local.conf`` to include
1159 all the debug symbols::
1160
1161 EXTRA_IMAGE_FEATURES:append = " dbg-pkgs"
1162
1163.. note::
1164
1165 To improve the debug information accuracy, you can reduce the level
1166 of optimization used by the compiler. For example, when adding the
1167 following line to your ``local.conf`` file, you will reduce optimization
1168 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
1169 of "-O -fno-omit-frame-pointer"::
1170
1171 DEBUG_BUILD = "1"
1172
1173 Consider that this will reduce the application's performance and is
1174 recommended only for debugging purposes.
1175
1176Other Debugging Tips
1177====================
1178
1179Here are some other tips that you might find useful:
1180
1181- When adding new packages, it is worth watching for undesirable items
1182 making their way into compiler command lines. For example, you do not
1183 want references to local system files like ``/usr/lib/`` or
1184 ``/usr/include/``.
1185
1186- If you want to remove the ``psplash`` boot splashscreen, add
1187 ``psplash=false`` to the kernel command line. Doing so prevents
1188 ``psplash`` from loading and thus allows you to see the console. It
1189 is also possible to switch out of the splashscreen by switching the
1190 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
1191
1192- Removing :term:`TMPDIR` (usually ``tmp/``, within the
1193 :term:`Build Directory`) can often fix temporary build issues. Removing
1194 :term:`TMPDIR` is usually a relatively cheap operation, because task output
1195 will be cached in :term:`SSTATE_DIR` (usually ``sstate-cache/``, which is
1196 also in the :term:`Build Directory`).
1197
1198 .. note::
1199
1200 Removing :term:`TMPDIR` might be a workaround rather than a fix.
1201 Consequently, trying to determine the underlying cause of an issue before
1202 removing the directory is a good idea.
1203
1204- Understanding how a feature is used in practice within existing
1205 recipes can be very helpful. It is recommended that you configure
1206 some method that allows you to quickly search through files.
1207
1208 Using GNU Grep, you can use the following shell function to
1209 recursively search through common recipe-related files, skipping
1210 binary files, ``.git`` directories, and the :term:`Build Directory`
1211 (assuming its name starts with "build")::
1212
1213 g() {
1214 grep -Ir \
1215 --exclude-dir=.git \
1216 --exclude-dir='build*' \
1217 --include='*.bb*' \
1218 --include='*.inc*' \
1219 --include='*.conf*' \
1220 --include='*.py*' \
1221 "$@"
1222 }
1223
1224 Following are some usage examples::
1225
1226 $ g FOO # Search recursively for "FOO"
1227 $ g -i foo # Search recursively for "foo", ignoring case
1228 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
1229
1230 If figuring
1231 out how some feature works requires a lot of searching, it might
1232 indicate that the documentation should be extended or improved. In
1233 such cases, consider filing a documentation bug using the Yocto
1234 Project implementation of
1235 :yocto_bugs:`Bugzilla <>`. For information on
1236 how to submit a bug against the Yocto Project, see the Yocto Project
1237 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
1238 and the ":doc:`../contributor-guide/report-defect`" section.
1239
1240 .. note::
1241
1242 The manuals might not be the right place to document variables
1243 that are purely internal and have a limited scope (e.g. internal
1244 variables used to implement a single ``.bbclass`` file).
1245
diff --git a/documentation/dev-manual/development-shell.rst b/documentation/dev-manual/development-shell.rst
new file mode 100644
index 0000000000..a18d792150
--- /dev/null
+++ b/documentation/dev-manual/development-shell.rst
@@ -0,0 +1,82 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using a Development Shell
4*************************
5
6When debugging certain commands or even when just editing packages,
7``devshell`` can be a useful tool. When you invoke ``devshell``, all
8tasks up to and including
9:ref:`ref-tasks-patch` are run for the
10specified target. Then, a new terminal is opened and you are placed in
11``${``\ :term:`S`\ ``}``, the source
12directory. In the new terminal, all the OpenEmbedded build-related
13environment variables are still defined so you can use commands such as
14``configure`` and ``make``. The commands execute just as if the
15OpenEmbedded build system were executing them. Consequently, working
16this way can be helpful when debugging a build or preparing software to
17be used with the OpenEmbedded build system.
18
19Following is an example that uses ``devshell`` on a target named
20``matchbox-desktop``::
21
22 $ bitbake matchbox-desktop -c devshell
23
24This command spawns a terminal with a shell prompt within the
25OpenEmbedded build environment. The
26:term:`OE_TERMINAL` variable
27controls what type of shell is opened.
28
29For spawned terminals, the following occurs:
30
31- The ``PATH`` variable includes the cross-toolchain.
32
33- The ``pkgconfig`` variables find the correct ``.pc`` files.
34
35- The ``configure`` command finds the Yocto Project site files as well
36 as any other necessary files.
37
38Within this environment, you can run configure or compile commands as if
39they were being run by the OpenEmbedded build system itself. As noted
40earlier, the working directory also automatically changes to the Source
41Directory (:term:`S`).
42
43To manually run a specific task using ``devshell``, run the
44corresponding ``run.*`` script in the
45``${``\ :term:`WORKDIR`\ ``}/temp``
46directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
47not exist, which would be the case if the task was skipped by way of the
48sstate cache, you can create the task by first running it outside of the
49``devshell``::
50
51 $ bitbake -c task
52
53.. note::
54
55 - Execution of a task's ``run.*`` script and BitBake's execution of
56 a task are identical. In other words, running the script re-runs
57 the task just as it would be run using the ``bitbake -c`` command.
58
59 - Any ``run.*`` file that does not have a ``.pid`` extension is a
60 symbolic link (symlink) to the most recent version of that file.
61
62Remember, that the ``devshell`` is a mechanism that allows you to get
63into the BitBake task execution environment. And as such, all commands
64must be called just as BitBake would call them. That means you need to
65provide the appropriate options for cross-compilation and so forth as
66applicable.
67
68When you are finished using ``devshell``, exit the shell or close the
69terminal window.
70
71.. note::
72
73 - It is worth remembering that when using ``devshell`` you need to
74 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
75 instead of just using ``gcc``. The same applies to other
76 applications such as ``binutils``, ``libtool`` and so forth.
77 BitBake sets up environment variables such as :term:`CC` to assist
78 applications, such as ``make`` to find the correct tools.
79
80 - It is also worth noting that ``devshell`` still works over X11
81 forwarding and similar situations.
82
diff --git a/documentation/dev-manual/device-manager.rst b/documentation/dev-manual/device-manager.rst
new file mode 100644
index 0000000000..0343d19b9c
--- /dev/null
+++ b/documentation/dev-manual/device-manager.rst
@@ -0,0 +1,74 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3.. _device-manager:
4
5Selecting a Device Manager
6**************************
7
8The Yocto Project provides multiple ways to manage the device manager
9(``/dev``):
10
11- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev``
12 directory is persistent and the required device nodes are created
13 during the build.
14
15- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
16 directory is provided by the kernel as an in-memory file system and
17 is automatically populated by the kernel at runtime. Additional
18 configuration of device nodes is done in user space by a device
19 manager like ``udev`` or ``busybox-mdev``.
20
21Using Persistent and Pre-Populated ``/dev``
22===========================================
23
24To use the static method for device population, you need to set the
25:term:`USE_DEVFS` variable to "0"
26as follows::
27
28 USE_DEVFS = "0"
29
30The content of the resulting ``/dev`` directory is defined in a Device
31Table file. The
32:term:`IMAGE_DEVICE_TABLES`
33variable defines the Device Table to use and should be set in the
34machine or distro configuration file. Alternatively, you can set this
35variable in your ``local.conf`` configuration file.
36
37If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
38``device_table-minimal.txt`` is used::
39
40 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
41
42The population is handled by the ``makedevs`` utility during image
43creation:
44
45Using ``devtmpfs`` and a Device Manager
46=======================================
47
48To use the dynamic method for device population, you need to use (or be
49sure to set) the :term:`USE_DEVFS`
50variable to "1", which is the default::
51
52 USE_DEVFS = "1"
53
54With this
55setting, the resulting ``/dev`` directory is populated by the kernel
56using ``devtmpfs``. Make sure the corresponding kernel configuration
57variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
58kernel.
59
60All devices created by ``devtmpfs`` will be owned by ``root`` and have
61permissions ``0600``.
62
63To have more control over the device nodes, you can use a device manager
64like ``udev`` or ``busybox-mdev``. You choose the device manager by
65defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
66distro configuration file. Alternatively, you can set this variable in
67your ``local.conf`` configuration file::
68
69 VIRTUAL-RUNTIME_dev_manager = "udev"
70
71 # Some alternative values
72 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
73 # VIRTUAL-RUNTIME_dev_manager = "systemd"
74
diff --git a/documentation/dev-manual/disk-space.rst b/documentation/dev-manual/disk-space.rst
new file mode 100644
index 0000000000..a84bef4511
--- /dev/null
+++ b/documentation/dev-manual/disk-space.rst
@@ -0,0 +1,61 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Conserving Disk Space
4*********************
5
6Conserving Disk Space During Builds
7===================================
8
9To help conserve disk space during builds, you can add the following
10statement to your project's ``local.conf`` configuration file found in
11the :term:`Build Directory`::
12
13 INHERIT += "rm_work"
14
15Adding this statement deletes the work directory used for
16building a recipe once the recipe is built. For more information on
17"rm_work", see the :ref:`ref-classes-rm-work` class in the
18Yocto Project Reference Manual.
19
20When you inherit this class and build a ``core-image-sato`` image for a
21``qemux86-64`` machine from an Ubuntu 22.04 x86-64 system, you end up with a
22final disk usage of 22 Gbytes instead of &MIN_DISK_SPACE; Gbytes. However,
23&MIN_DISK_SPACE_RM_WORK; Gbytes of initial free disk space are still needed to
24create temporary files before they can be deleted.
25
26Purging Duplicate Shared State Cache Files
27==========================================
28
29After multiple build iterations, the Shared State (sstate) cache can contain
30duplicate cache files for a given package, consuming a substantial amount of
31disk space. However, only the most recent cache files are likeky to be reusable.
32
33The following command is a quick way to purge all the cache files which
34haven't been used for a least a specified number of days::
35
36 find build/sstate-cache -type f -mtime +$DAYS -delete
37
38The above command relies on the fact that BitBake touches the sstate cache
39files as it accesses them, when it has write access to the cache.
40
41You could use ``-atime`` instead of ``-mtime`` if the partition isn't mounted
42with the ``noatime`` option for a read only cache.
43
44For more advanced needs, OpenEmbedded-Core also offers a more elaborate
45command. It has the ability to purge all but the newest cache files on each
46architecture, and also to remove files that it considers unreachable by
47exploring a set of build configurations. However, this command
48requires a full build environment to be available and doesn't work well
49covering multiple releases. It won't work either on limited environments
50such as BSD based NAS::
51
52 sstate-cache-management.sh --remove-duplicated --cache-dir=build/sstate-cache
53
54This command will ask you to confirm the deletions it identifies.
55Run ``sstate-cache-management.sh`` for more details about this script.
56
57.. note::
58
59 As this command is much more cautious and selective, removing only cache files,
60 it will execute much slower than the simple ``find`` command described above.
61 Therefore, it may not be your best option to trim huge cache directories.
diff --git a/documentation/dev-manual/efficiently-fetching-sources.rst b/documentation/dev-manual/efficiently-fetching-sources.rst
new file mode 100644
index 0000000000..fd9ad6aa69
--- /dev/null
+++ b/documentation/dev-manual/efficiently-fetching-sources.rst
@@ -0,0 +1,68 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Efficiently Fetching Source Files During a Build
4================================================
5
6The OpenEmbedded build system works with source files located through
7the :term:`SRC_URI` variable. When
8you build something using BitBake, a big part of the operation is
9locating and downloading all the source tarballs. For images,
10downloading all the source for various packages can take a significant
11amount of time.
12
13This section shows you how you can use mirrors to speed up fetching
14source files and how you can pre-fetch files all of which leads to more
15efficient use of resources and time.
16
17Setting up Effective Mirrors
18----------------------------
19
20A good deal that goes into a Yocto Project build is simply downloading
21all of the source tarballs. Maybe you have been working with another
22build system for which you have built up a
23sizable directory of source tarballs. Or, perhaps someone else has such
24a directory for which you have read access. If so, you can save time by
25adding statements to your configuration file so that the build process
26checks local directories first for existing tarballs before checking the
27Internet.
28
29Here is an efficient way to set it up in your ``local.conf`` file::
30
31 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
32 INHERIT += "own-mirrors"
33 BB_GENERATE_MIRROR_TARBALLS = "1"
34 # BB_NO_NETWORK = "1"
35
36In the previous example, the
37:term:`BB_GENERATE_MIRROR_TARBALLS`
38variable causes the OpenEmbedded build system to generate tarballs of
39the Git repositories and store them in the
40:term:`DL_DIR` directory. Due to
41performance reasons, generating and storing these tarballs is not the
42build system's default behavior.
43
44You can also use the
45:term:`PREMIRRORS` variable. For
46an example, see the variable's glossary entry in the Yocto Project
47Reference Manual.
48
49Getting Source Files and Suppressing the Build
50----------------------------------------------
51
52Another technique you can use to ready yourself for a successive string
53of build operations, is to pre-fetch all the source files without
54actually starting a build. This technique lets you work through any
55download issues and ultimately gathers all the source files into your
56download directory :ref:`structure-build-downloads`,
57which is located with :term:`DL_DIR`.
58
59Use the following BitBake command form to fetch all the necessary
60sources without starting the build::
61
62 $ bitbake target --runall=fetch
63
64This
65variation of the BitBake command guarantees that you have all the
66sources for that BitBake target should you disconnect from the Internet
67and want to do the build later offline.
68
diff --git a/documentation/dev-manual/error-reporting-tool.rst b/documentation/dev-manual/error-reporting-tool.rst
new file mode 100644
index 0000000000..84f3d9cd1e
--- /dev/null
+++ b/documentation/dev-manual/error-reporting-tool.rst
@@ -0,0 +1,84 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using the Error Reporting Tool
4******************************
5
6The error reporting tool allows you to submit errors encountered during
7builds to a central database. Outside of the build environment, you can
8use a web interface to browse errors, view statistics, and query for
9errors. The tool works using a client-server system where the client
10portion is integrated with the installed Yocto Project
11:term:`Source Directory` (e.g. ``poky``).
12The server receives the information collected and saves it in a
13database.
14
15There is a live instance of the error reporting server at
16https://errors.yoctoproject.org.
17When you want to get help with build failures, you can submit all of the
18information on the failure easily and then point to the URL in your bug
19report or send an email to the mailing list.
20
21.. note::
22
23 If you send error reports to this server, the reports become publicly
24 visible.
25
26Enabling and Using the Tool
27===========================
28
29By default, the error reporting tool is disabled. You can enable it by
30inheriting the :ref:`ref-classes-report-error` class by adding the
31following statement to the end of your ``local.conf`` file in your
32:term:`Build Directory`::
33
34 INHERIT += "report-error"
35
36By default, the error reporting feature stores information in
37``${``\ :term:`LOG_DIR`\ ``}/error-report``.
38However, you can specify a directory to use by adding the following to
39your ``local.conf`` file::
40
41 ERR_REPORT_DIR = "path"
42
43Enabling error
44reporting causes the build process to collect the errors and store them
45in a file as previously described. When the build system encounters an
46error, it includes a command as part of the console output. You can run
47the command to send the error file to the server. For example, the
48following command sends the errors to an upstream server::
49
50 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
51
52In the previous example, the errors are sent to a public database
53available at https://errors.yoctoproject.org, which is used by the
54entire community. If you specify a particular server, you can send the
55errors to a different database. Use the following command for more
56information on available options::
57
58 $ send-error-report --help
59
60When sending the error file, you are prompted to review the data being
61sent as well as to provide a name and optional email address. Once you
62satisfy these prompts, the command returns a link from the server that
63corresponds to your entry in the database. For example, here is a
64typical link: https://errors.yoctoproject.org/Errors/Details/9522/
65
66Following the link takes you to a web interface where you can browse,
67query the errors, and view statistics.
68
69Disabling the Tool
70==================
71
72To disable the error reporting feature, simply remove or comment out the
73following statement from the end of your ``local.conf`` file in your
74:term:`Build Directory`::
75
76 INHERIT += "report-error"
77
78Setting Up Your Own Error Reporting Server
79==========================================
80
81If you want to set up your own error reporting server, you can obtain
82the code from the Git repository at :yocto_git:`/error-report-web/`.
83Instructions on how to set it up are in the README document.
84
diff --git a/documentation/dev-manual/external-scm.rst b/documentation/dev-manual/external-scm.rst
new file mode 100644
index 0000000000..97a7e63e36
--- /dev/null
+++ b/documentation/dev-manual/external-scm.rst
@@ -0,0 +1,67 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using an External SCM
4*********************
5
6If you're working on a recipe that pulls from an external Source Code
7Manager (SCM), it is possible to have the OpenEmbedded build system
8notice new recipe changes added to the SCM and then build the resulting
9packages that depend on the new recipes by using the latest versions.
10This only works for SCMs from which it is possible to get a sensible
11revision number for changes. Currently, you can do this with Apache
12Subversion (SVN), Git, and Bazaar (BZR) repositories.
13
14To enable this behavior, the :term:`PV` of
15the recipe needs to reference
16:term:`SRCPV`. Here is an example::
17
18 PV = "1.2.3+git${SRCPV}"
19
20Then, you can add the following to your
21``local.conf``::
22
23 SRCREV:pn-PN = "${AUTOREV}"
24
25:term:`PN` is the name of the recipe for
26which you want to enable automatic source revision updating.
27
28If you do not want to update your local configuration file, you can add
29the following directly to the recipe to finish enabling the feature::
30
31 SRCREV = "${AUTOREV}"
32
33The Yocto Project provides a distribution named ``poky-bleeding``, whose
34configuration file contains the line::
35
36 require conf/distro/include/poky-floating-revisions.inc
37
38This line pulls in the
39listed include file that contains numerous lines of exactly that form::
40
41 #SRCREV:pn-opkg-native ?= "${AUTOREV}"
42 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}"
43 #SRCREV:pn-opkg ?= "${AUTOREV}"
44 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}"
45 #SRCREV:pn-opkg-utils ?= "${AUTOREV}"
46 SRCREV:pn-gconf-dbus ?= "${AUTOREV}"
47 SRCREV:pn-matchbox-common ?= "${AUTOREV}"
48 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}"
49 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}"
50 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}"
51 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}"
52 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}"
53 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}"
54 SRCREV:pn-matchbox-wm ?= "${AUTOREV}"
55 SRCREV:pn-settings-daemon ?= "${AUTOREV}"
56 SRCREV:pn-screenshot ?= "${AUTOREV}"
57 . . .
58
59These lines allow you to
60experiment with building a distribution that tracks the latest
61development source for numerous packages.
62
63.. note::
64
65 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
66 this in mind if you use it.
67
diff --git a/documentation/dev-manual/external-toolchain.rst b/documentation/dev-manual/external-toolchain.rst
new file mode 100644
index 0000000000..238f8cf467
--- /dev/null
+++ b/documentation/dev-manual/external-toolchain.rst
@@ -0,0 +1,40 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Optionally Using an External Toolchain
4**************************************
5
6You might want to use an external toolchain as part of your development.
7If this is the case, the fundamental steps you need to accomplish are as
8follows:
9
10- Understand where the installed toolchain resides. For cases where you
11 need to build the external toolchain, you would need to take separate
12 steps to build and install the toolchain.
13
14- Make sure you add the layer that contains the toolchain to your
15 ``bblayers.conf`` file through the
16 :term:`BBLAYERS` variable.
17
18- Set the :term:`EXTERNAL_TOOLCHAIN` variable in your ``local.conf`` file
19 to the location in which you installed the toolchain.
20
21The toolchain configuration is very flexible and customizable. It
22is primarily controlled with the :term:`TCMODE` variable. This variable
23controls which ``tcmode-*.inc`` file to include from the
24``meta/conf/distro/include`` directory within the :term:`Source Directory`.
25
26The default value of :term:`TCMODE` is "default", which tells the
27OpenEmbedded build system to use its internally built toolchain (i.e.
28``tcmode-default.inc``). However, other patterns are accepted. In
29particular, "external-\*" refers to external toolchains. One example is
30the Mentor Graphics Sourcery G++ Toolchain. Support for this toolchain resides
31in the separate ``meta-sourcery`` layer at
32https://github.com/MentorEmbedded/meta-sourcery/.
33See its ``README`` file for details about how to use this layer.
34
35Another example of external toolchain layer is
36:yocto_git:`meta-arm-toolchain </meta-arm/tree/meta-arm-toolchain/>`
37supporting GNU toolchains released by ARM.
38
39You can find further information by reading about the :term:`TCMODE` variable
40in the Yocto Project Reference Manual's variable glossary.
diff --git a/documentation/dev-manual/gobject-introspection.rst b/documentation/dev-manual/gobject-introspection.rst
new file mode 100644
index 0000000000..f7206e6fae
--- /dev/null
+++ b/documentation/dev-manual/gobject-introspection.rst
@@ -0,0 +1,155 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Enabling GObject Introspection Support
4**************************************
5
6`GObject introspection <https://gi.readthedocs.io/en/latest/>`__
7is the standard mechanism for accessing GObject-based software from
8runtime environments. GObject is a feature of the GLib library that
9provides an object framework for the GNOME desktop and related software.
10GObject Introspection adds information to GObject that allows objects
11created within it to be represented across different programming
12languages. If you want to construct GStreamer pipelines using Python, or
13control UPnP infrastructure using Javascript and GUPnP, GObject
14introspection is the only way to do it.
15
16This section describes the Yocto Project support for generating and
17packaging GObject introspection data. GObject introspection data is a
18description of the API provided by libraries built on top of the GLib
19framework, and, in particular, that framework's GObject mechanism.
20GObject Introspection Repository (GIR) files go to ``-dev`` packages,
21``typelib`` files go to main packages as they are packaged together with
22libraries that are introspected.
23
24The data is generated when building such a library, by linking the
25library with a small executable binary that asks the library to describe
26itself, and then executing the binary and processing its output.
27
28Generating this data in a cross-compilation environment is difficult
29because the library is produced for the target architecture, but its
30code needs to be executed on the build host. This problem is solved with
31the OpenEmbedded build system by running the code through QEMU, which
32allows precisely that. Unfortunately, QEMU does not always work
33perfectly as mentioned in the ":ref:`dev-manual/gobject-introspection:known issues`"
34section.
35
36Enabling the Generation of Introspection Data
37=============================================
38
39Enabling the generation of introspection data (GIR files) in your
40library package involves the following:
41
42#. Inherit the :ref:`ref-classes-gobject-introspection` class.
43
44#. Make sure introspection is not disabled anywhere in the recipe or
45 from anything the recipe includes. Also, make sure that
46 "gobject-introspection-data" is not in
47 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
48 and that "qemu-usermode" is not in
49 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
50 In either of these conditions, nothing will happen.
51
52#. Try to build the recipe. If you encounter build errors that look like
53 something is unable to find ``.so`` libraries, check where these
54 libraries are located in the source tree and add the following to the
55 recipe::
56
57 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
58
59 .. note::
60
61 See recipes in the ``oe-core`` repository that use that
62 :term:`GIR_EXTRA_LIBS_PATH` variable as an example.
63
64#. Look for any other errors, which probably mean that introspection
65 support in a package is not entirely standard, and thus breaks down
66 in a cross-compilation environment. For such cases, custom-made fixes
67 are needed. A good place to ask and receive help in these cases is
68 the :ref:`Yocto Project mailing
69 lists <resources-mailinglist>`.
70
71.. note::
72
73 Using a library that no longer builds against the latest Yocto
74 Project release and prints introspection related errors is a good
75 candidate for the previous procedure.
76
77Disabling the Generation of Introspection Data
78==============================================
79
80You might find that you do not want to generate introspection data. Or,
81perhaps QEMU does not work on your build host and target architecture
82combination. If so, you can use either of the following methods to
83disable GIR file generations:
84
85- Add the following to your distro configuration::
86
87 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
88
89 Adding this statement disables generating introspection data using
90 QEMU but will still enable building introspection tools and libraries
91 (i.e. building them does not require the use of QEMU).
92
93- Add the following to your machine configuration::
94
95 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
96
97 Adding this statement disables the use of QEMU when building packages for your
98 machine. Currently, this feature is used only by introspection
99 recipes and has the same effect as the previously described option.
100
101 .. note::
102
103 Future releases of the Yocto Project might have other features
104 affected by this option.
105
106If you disable introspection data, you can still obtain it through other
107means such as copying the data from a suitable sysroot, or by generating
108it on the target hardware. The OpenEmbedded build system does not
109currently provide specific support for these techniques.
110
111Testing that Introspection Works in an Image
112============================================
113
114Use the following procedure to test if generating introspection data is
115working in an image:
116
117#. Make sure that "gobject-introspection-data" is not in
118 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
119 and that "qemu-usermode" is not in
120 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
121
122#. Build ``core-image-sato``.
123
124#. Launch a Terminal and then start Python in the terminal.
125
126#. Enter the following in the terminal::
127
128 >>> from gi.repository import GLib
129 >>> GLib.get_host_name()
130
131#. For something a little more advanced, enter the following see:
132 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
133
134Known Issues
135============
136
137Here are know issues in GObject Introspection Support:
138
139- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
140 introspection data on that architecture.
141
142- x32 is not supported by QEMU. Consequently, introspection data is
143 disabled.
144
145- musl causes transient GLib binaries to crash on assertion failures.
146 Consequently, generating introspection data is disabled.
147
148- Because QEMU is not able to run the binaries correctly, introspection
149 is disabled for some specific packages under specific architectures
150 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
151
152- QEMU usermode might not work properly when running 64-bit binaries
153 under 32-bit host machines. In particular, "qemumips64" is known to
154 not work under i686.
155
diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
index f16b135c4d..3106b90a45 100644
--- a/documentation/dev-manual/index.rst
+++ b/documentation/dev-manual/index.rst
@@ -4,15 +4,48 @@
4Yocto Project Development Tasks Manual 4Yocto Project Development Tasks Manual
5====================================== 5======================================
6 6
7|
8
9.. toctree:: 7.. toctree::
10 :caption: Table of Contents 8 :caption: Table of Contents
11 :numbered: 9 :numbered:
12 10
13 intro 11 intro
14 start 12 start
15 common-tasks 13 layers
14 customizing-images
15 new-recipe
16 new-machine
17 upgrading-recipes
18 temporary-source-code
19 quilt.rst
20 development-shell
21 python-development-shell
22 building
23 speeding-up-build
24 libraries
25 prebuilt-libraries
26 x32-psabi
27 gobject-introspection
28 external-toolchain
29 wic
30 bmaptool
31 securing-images
32 custom-distribution
33 custom-template-configuration-directory
34 disk-space
35 packages
36 efficiently-fetching-sources
37 init-manager
38 device-manager
39 external-scm
40 read-only-rootfs
41 build-quality
42 runtime-testing
43 debugging
44 licenses
45 vulnerabilities
46 sbom
47 error-reporting-tool
48 wayland
16 qemu 49 qemu
17 50
18.. include:: /boilerplate.rst 51.. include:: /boilerplate.rst
diff --git a/documentation/dev-manual/init-manager.rst b/documentation/dev-manual/init-manager.rst
new file mode 100644
index 0000000000..20d61ea830
--- /dev/null
+++ b/documentation/dev-manual/init-manager.rst
@@ -0,0 +1,162 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3.. _init-manager:
4
5Selecting an Initialization Manager
6***********************************
7
8By default, the Yocto Project uses :wikipedia:`SysVinit <Init#SysV-style>` as
9the initialization manager. There is also support for BusyBox init, a simpler
10implementation, as well as support for :wikipedia:`systemd <Systemd>`, which
11is a full replacement for init with parallel starting of services, reduced
12shell overhead, increased security and resource limits for services, and other
13features that are used by many distributions.
14
15Within the system, SysVinit and BusyBox init treat system components as
16services. These services are maintained as shell scripts stored in the
17``/etc/init.d/`` directory.
18
19SysVinit is more elaborate than BusyBox init and organizes services in
20different run levels. This organization is maintained by putting links
21to the services in the ``/etc/rcN.d/`` directories, where `N/` is one
22of the following options: "S", "0", "1", "2", "3", "4", "5", or "6".
23
24.. note::
25
26 Each runlevel has a dependency on the previous runlevel. This
27 dependency allows the services to work properly.
28
29Both SysVinit and BusyBox init are configured through the ``/etc/inittab``
30file, with a very similar syntax, though of course BusyBox init features
31are more limited.
32
33In comparison, systemd treats components as units. Using units is a
34broader concept as compared to using a service. A unit includes several
35different types of entities. ``Service`` is one of the types of entities.
36The runlevel concept in SysVinit corresponds to the concept of a target
37in systemd, where target is also a type of supported unit.
38
39In systems with SysVinit or BusyBox init, services load sequentially (i.e. one
40by one) during init and parallelization is not supported. With systemd, services
41start in parallel. This method can have an impact on the startup performance
42of a given service, though systemd will also provide more services by default,
43therefore increasing the total system boot time. systemd also substantially
44increases system size because of its multiple components and the extra
45dependencies it pulls.
46
47On the contrary, BusyBox init is the simplest and the lightest solution and
48also comes with BusyBox mdev as device manager, a lighter replacement to
49:wikipedia:`udev <Udev>`, which SysVinit and systemd both use.
50
51The ":ref:`device-manager`" chapter has more details about device managers.
52
53Using SysVinit with udev
54=========================
55
56SysVinit with the udev device manager corresponds to the
57default setting in Poky. This corresponds to setting::
58
59 INIT_MANAGER = "sysvinit"
60
61Using BusyBox init with BusyBox mdev
62====================================
63
64BusyBox init with BusyBox mdev is the simplest and lightest solution
65for small root filesystems. All you need is BusyBox, which most systems
66have anyway::
67
68 INIT_MANAGER = "mdev-busybox"
69
70Using systemd
71=============
72
73The last option is to use systemd together with the udev device
74manager. This is the most powerful and versatile solution, especially
75for more complex systems::
76
77 INIT_MANAGER = "systemd"
78
79This will enable systemd and remove sysvinit components from the image.
80See :yocto_git:`meta/conf/distro/include/init-manager-systemd.inc
81</poky/tree/meta/conf/distro/include/init-manager-systemd.inc>` for exact
82details on what this does.
83
84Controling systemd from the target command line
85-----------------------------------------------
86
87Here is a quick reference for controling systemd from the command line on the
88target. Instead of opening and sometimes modifying files, most interaction
89happens through the ``systemctl`` and ``journalctl`` commands:
90
91- ``systemctl status``: show the status of all services
92- ``systemctl status <service>``: show the status of one service
93- ``systemctl [start|stop] <service>``: start or stop a service
94- ``systemctl [enable|disable] <service>``: enable or disable a service at boot time
95- ``systemctl list-units``: list all available units
96- ``journalctl -a``: show all logs for all services
97- ``journalctl -f``: show only the last log entries, and keep printing updates as they arrive
98- ``journalctl -u``: show only logs from a particular service
99
100Using systemd-journald without a traditional syslog daemon
101----------------------------------------------------------
102
103Counter-intuitively, ``systemd-journald`` is not a syslog runtime or provider,
104and the proper way to use ``systemd-journald`` as your sole logging mechanism is to
105effectively disable syslog entirely by setting these variables in your distribution
106configuration file::
107
108 VIRTUAL-RUNTIME_syslog = ""
109 VIRTUAL-RUNTIME_base-utils-syslog = ""
110
111Doing so will prevent ``rsyslog`` / ``busybox-syslog`` from being pulled in by
112default, leaving only ``systemd-journald``.
113
114Summary
115-------
116
117The Yocto Project supports three different initialization managers, offering
118increasing levels of complexity and functionality:
119
120.. list-table::
121 :widths: 40 20 20 20
122 :header-rows: 1
123
124 * -
125 - BusyBox init
126 - SysVinit
127 - systemd
128 * - Size
129 - Small
130 - Small
131 - Big [#footnote-systemd-size]_
132 * - Complexity
133 - Small
134 - Medium
135 - High
136 * - Support for boot profiles
137 - No
138 - Yes ("runlevels")
139 - Yes ("targets")
140 * - Services defined as
141 - Shell scripts
142 - Shell scripts
143 - Description files
144 * - Starting services in parallel
145 - No
146 - No
147 - Yes
148 * - Setting service resource limits
149 - No
150 - No
151 - Yes
152 * - Support service isolation
153 - No
154 - No
155 - Yes
156 * - Integrated logging
157 - No
158 - No
159 - Yes
160
161.. [#footnote-systemd-size] Using systemd increases the ``core-image-minimal``
162 image size by 160\% for ``qemux86-64`` on Mickledore (4.2), compared to SysVinit.
diff --git a/documentation/dev-manual/layers.rst b/documentation/dev-manual/layers.rst
new file mode 100644
index 0000000000..c5c34134b6
--- /dev/null
+++ b/documentation/dev-manual/layers.rst
@@ -0,0 +1,839 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Understanding and Creating Layers
4*********************************
5
6The OpenEmbedded build system supports organizing
7:term:`Metadata` into multiple layers.
8Layers allow you to isolate different types of customizations from each
9other. For introductory information on the Yocto Project Layer Model,
10see the
11":ref:`overview-manual/yp-intro:the yocto project layer model`"
12section in the Yocto Project Overview and Concepts Manual.
13
14Creating Your Own Layer
15=======================
16
17.. note::
18
19 It is very easy to create your own layers to use with the OpenEmbedded
20 build system, as the Yocto Project ships with tools that speed up creating
21 layers. This section describes the steps you perform by hand to create
22 layers so that you can better understand them. For information about the
23 layer-creation tools, see the
24 ":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
25 section in the Yocto Project Board Support Package (BSP) Developer's
26 Guide and the ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
27 section further down in this manual.
28
29Follow these general steps to create your layer without using tools:
30
31#. *Check Existing Layers:* Before creating a new layer, you should be
32 sure someone has not already created a layer containing the Metadata
33 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
34 for a list of layers from the OpenEmbedded community that can be used in
35 the Yocto Project. You could find a layer that is identical or close
36 to what you need.
37
38#. *Create a Directory:* Create the directory for your layer. When you
39 create the layer, be sure to create the directory in an area not
40 associated with the Yocto Project :term:`Source Directory`
41 (e.g. the cloned ``poky`` repository).
42
43 While not strictly required, prepend the name of the directory with
44 the string "meta-". For example::
45
46 meta-mylayer
47 meta-GUI_xyz
48 meta-mymachine
49
50 With rare exceptions, a layer's name follows this form::
51
52 meta-root_name
53
54 Following this layer naming convention can save
55 you trouble later when tools, components, or variables "assume" your
56 layer name begins with "meta-". A notable example is in configuration
57 files as shown in the following step where layer names without the
58 "meta-" string are appended to several variables used in the
59 configuration.
60
61#. *Create a Layer Configuration File:* Inside your new layer folder,
62 you need to create a ``conf/layer.conf`` file. It is easiest to take
63 an existing layer configuration file and copy that to your layer's
64 ``conf`` directory and then modify the file as needed.
65
66 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
67 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
68 demonstrates the required syntax. For your layer, you need to replace
69 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
70 for a layer named "meta-machinexyz")::
71
72 # We have a conf and classes directory, add to BBPATH
73 BBPATH .= ":${LAYERDIR}"
74
75 # We have recipes-* directories, add to BBFILES
76 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
77 ${LAYERDIR}/recipes-*/*/*.bbappend"
78
79 BBFILE_COLLECTIONS += "yoctobsp"
80 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
81 BBFILE_PRIORITY_yoctobsp = "5"
82 LAYERVERSION_yoctobsp = "4"
83 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
84
85 Following is an explanation of the layer configuration file:
86
87 - :term:`BBPATH`: Adds the layer's
88 root directory to BitBake's search path. Through the use of the
89 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
90 configuration files, and files that are included with ``include``
91 and ``require`` statements. For these cases, BitBake uses the
92 first file that matches the name found in :term:`BBPATH`. This is
93 similar to the way the ``PATH`` variable is used for binaries. It
94 is recommended, therefore, that you use unique class and
95 configuration filenames in your custom layer.
96
97 - :term:`BBFILES`: Defines the
98 location for all recipes in the layer.
99
100 - :term:`BBFILE_COLLECTIONS`:
101 Establishes the current layer through a unique identifier that is
102 used throughout the OpenEmbedded build system to refer to the
103 layer. In this example, the identifier "yoctobsp" is the
104 representation for the container layer named "meta-yocto-bsp".
105
106 - :term:`BBFILE_PATTERN`:
107 Expands immediately during parsing to provide the directory of the
108 layer.
109
110 - :term:`BBFILE_PRIORITY`:
111 Establishes a priority to use for recipes in the layer when the
112 OpenEmbedded build finds recipes of the same name in different
113 layers.
114
115 - :term:`LAYERVERSION`:
116 Establishes a version number for the layer. You can use this
117 version number to specify this exact version of the layer as a
118 dependency when using the
119 :term:`LAYERDEPENDS`
120 variable.
121
122 - :term:`LAYERDEPENDS`:
123 Lists all layers on which this layer depends (if any).
124
125 - :term:`LAYERSERIES_COMPAT`:
126 Lists the :yocto_wiki:`Yocto Project </Releases>`
127 releases for which the current version is compatible. This
128 variable is a good way to indicate if your particular layer is
129 current.
130
131#. *Add Content:* Depending on the type of layer, add the content. If
132 the layer adds support for a machine, add the machine configuration
133 in a ``conf/machine/`` file within the layer. If the layer adds
134 distro policy, add the distro configuration in a ``conf/distro/``
135 file within the layer. If the layer introduces new recipes, put the
136 recipes you need in ``recipes-*`` subdirectories within the layer.
137
138 .. note::
139
140 For an explanation of layer hierarchy that is compliant with the
141 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
142 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
143
144#. *Optionally Test for Compatibility:* If you want permission to use
145 the Yocto Project Compatibility logo with your layer or application
146 that uses your layer, perform the steps to apply for compatibility.
147 See the
148 ":ref:`dev-manual/layers:making sure your layer is compatible with yocto project`"
149 section for more information.
150
151Following Best Practices When Creating Layers
152=============================================
153
154To create layers that are easier to maintain and that will not impact
155builds for other machines, you should consider the information in the
156following list:
157
158- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
159 Configuration:* In other words, do not copy an entire recipe into
160 your layer and then modify it. Rather, use an append file
161 (``.bbappend``) to override only those parts of the original recipe
162 you need to modify.
163
164- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
165 for each recipe that uses an include file. Or, if you are introducing
166 a new recipe that requires the included file, use the path relative
167 to the original layer directory to refer to the file. For example,
168 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
169 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
170 the include file, it could indicate a deficiency in the include file
171 in the layer to which it originally belongs. If this is the case, you
172 should try to address that deficiency instead of overlaying the
173 include file. For example, you could address this by getting the
174 maintainer of the include file to add a variable or variables to make
175 it easy to override the parts needing to be overridden.
176
177- *Structure Your Layers:* Proper use of overrides within append files
178 and placement of machine-specific files within your layer can ensure
179 that a build is not using the wrong Metadata and negatively impacting
180 a build for a different machine. Following are some examples:
181
182 - *Modify Variables to Support a Different Machine:* Suppose you
183 have a layer named ``meta-one`` that adds support for building
184 machine "one". To do so, you use an append file named
185 ``base-files.bbappend`` and create a dependency on "foo" by
186 altering the :term:`DEPENDS`
187 variable::
188
189 DEPENDS = "foo"
190
191 The dependency is created during any
192 build that includes the layer ``meta-one``. However, you might not
193 want this dependency for all machines. For example, suppose you
194 are building for machine "two" but your ``bblayers.conf`` file has
195 the ``meta-one`` layer included. During the build, the
196 ``base-files`` for machine "two" will also have the dependency on
197 ``foo``.
198
199 To make sure your changes apply only when building machine "one",
200 use a machine override with the :term:`DEPENDS` statement::
201
202 DEPENDS:one = "foo"
203
204 You should follow the same strategy when using ``:append``
205 and ``:prepend`` operations::
206
207 DEPENDS:append:one = " foo"
208 DEPENDS:prepend:one = "foo "
209
210 As an actual example, here's a
211 snippet from the generic kernel include file ``linux-yocto.inc``,
212 wherein the kernel compile and link options are adjusted in the
213 case of a subset of the supported architectures::
214
215 DEPENDS:append:aarch64 = " libgcc"
216 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
217 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
218
219 DEPENDS:append:nios2 = " libgcc"
220 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
221 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
222
223 DEPENDS:append:arc = " libgcc"
224 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
225 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
226
227 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
228
229 - *Place Machine-Specific Files in Machine-Specific Locations:* When
230 you have a base recipe, such as ``base-files.bb``, that contains a
231 :term:`SRC_URI` statement to a
232 file, you can use an append file to cause the build to use your
233 own version of the file. For example, an append file in your layer
234 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
235 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
236
237 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
238
239 The build for machine "one" will pick up your machine-specific file as
240 long as you have the file in
241 ``meta-one/recipes-core/base-files/base-files/``. However, if you
242 are building for a different machine and the ``bblayers.conf``
243 file includes the ``meta-one`` layer and the location of your
244 machine-specific file is the first location where that file is
245 found according to :term:`FILESPATH`, builds for all machines will
246 also use that machine-specific file.
247
248 You can make sure that a machine-specific file is used for a
249 particular machine by putting the file in a subdirectory specific
250 to the machine. For example, rather than placing the file in
251 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
252 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
253 Not only does this make sure the file is used only when building
254 for machine "one", but the build process locates the file more
255 quickly.
256
257 In summary, you need to place all files referenced from
258 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
259 order to restrict those files to machine-specific builds.
260
261- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
262 permission to use the Yocto Project Compatibility logo with your
263 layer or application that uses your layer, perform the steps to apply
264 for compatibility. See the
265 ":ref:`dev-manual/layers:making sure your layer is compatible with yocto project`"
266 section for more information.
267
268- *Follow the Layer Naming Convention:* Store custom layers in a Git
269 repository that use the ``meta-layer_name`` format.
270
271- *Group Your Layers Locally:* Clone your repository alongside other
272 cloned ``meta`` directories from the :term:`Source Directory`.
273
274Making Sure Your Layer is Compatible With Yocto Project
275=======================================================
276
277When you create a layer used with the Yocto Project, it is advantageous
278to make sure that the layer interacts well with existing Yocto Project
279layers (i.e. the layer is compatible with the Yocto Project). Ensuring
280compatibility makes the layer easy to be consumed by others in the Yocto
281Project community and could allow you permission to use the Yocto
282Project Compatible Logo.
283
284.. note::
285
286 Only Yocto Project member organizations are permitted to use the
287 Yocto Project Compatible Logo. The logo is not available for general
288 use. For information on how to become a Yocto Project member
289 organization, see the :yocto_home:`Yocto Project Website <>`.
290
291The Yocto Project Compatibility Program consists of a layer application
292process that requests permission to use the Yocto Project Compatibility
293Logo for your layer and application. The process consists of two parts:
294
295#. Successfully passing a script (``yocto-check-layer``) that when run
296 against your layer, tests it against constraints based on experiences
297 of how layers have worked in the real world and where pitfalls have
298 been found. Getting a "PASS" result from the script is required for
299 successful compatibility registration.
300
301#. Completion of an application acceptance form, which you can find at
302 :yocto_home:`/webform/yocto-project-compatible-registration`.
303
304To be granted permission to use the logo, you need to satisfy the
305following:
306
307- Be able to check the box indicating that you got a "PASS" when
308 running the script against your layer.
309
310- Answer "Yes" to the questions on the form or have an acceptable
311 explanation for any questions answered "No".
312
313- Be a Yocto Project Member Organization.
314
315The remainder of this section presents information on the registration
316form and on the ``yocto-check-layer`` script.
317
318Yocto Project Compatible Program Application
319--------------------------------------------
320
321Use the form to apply for your layer's approval. Upon successful
322application, you can use the Yocto Project Compatibility Logo with your
323layer and the application that uses your layer.
324
325To access the form, use this link:
326:yocto_home:`/webform/yocto-project-compatible-registration`.
327Follow the instructions on the form to complete your application.
328
329The application consists of the following sections:
330
331- *Contact Information:* Provide your contact information as the fields
332 require. Along with your information, provide the released versions
333 of the Yocto Project for which your layer is compatible.
334
335- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
336 items in the checklist. There is space at the bottom of the form for
337 any explanations for items for which you answered "No".
338
339- *Recommendations:* Provide answers for the questions regarding Linux
340 kernel use and build success.
341
342``yocto-check-layer`` Script
343----------------------------
344
345The ``yocto-check-layer`` script provides you a way to assess how
346compatible your layer is with the Yocto Project. You should run this
347script prior to using the form to apply for compatibility as described
348in the previous section. You need to achieve a "PASS" result in order to
349have your application form successfully processed.
350
351The script divides tests into three areas: COMMON, BSP, and DISTRO. For
352example, given a distribution layer (DISTRO), the layer must pass both
353the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
354layer, the layer must pass the COMMON and BSP set of tests.
355
356To execute the script, enter the following commands from your build
357directory::
358
359 $ source oe-init-build-env
360 $ yocto-check-layer your_layer_directory
361
362Be sure to provide the actual directory for your
363layer as part of the command.
364
365Entering the command causes the script to determine the type of layer
366and then to execute a set of specific tests against the layer. The
367following list overviews the test:
368
369- ``common.test_readme``: Tests if a ``README`` file exists in the
370 layer and the file is not empty.
371
372- ``common.test_parse``: Tests to make sure that BitBake can parse the
373 files without error (i.e. ``bitbake -p``).
374
375- ``common.test_show_environment``: Tests that the global or per-recipe
376 environment is in order without errors (i.e. ``bitbake -e``).
377
378- ``common.test_world``: Verifies that ``bitbake world`` works.
379
380- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
381 layers do not come with recipes that change signatures.
382
383- ``common.test_layerseries_compat``: Verifies layer compatibility is
384 set properly.
385
386- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
387 configurations.
388
389- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
390 set the machine when the layer is added.
391
392- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
393 regardless of which machine is selected.
394
395- ``bsp.test_machine_signatures``: Verifies that building for a
396 particular machine affects only the signature of tasks specific to
397 that machine.
398
399- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
400 distro configurations.
401
402- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
403 does not set the distribution when the layer is added.
404
405Enabling Your Layer
406===================
407
408Before the OpenEmbedded build system can use your new layer, you need to
409enable it. To enable your layer, simply add your layer's path to the
410:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
411found in the :term:`Build Directory`. The following example shows how to
412enable your new ``meta-mylayer`` layer (note how your new layer exists
413outside of the official ``poky`` repository which you would have checked
414out earlier)::
415
416 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
417 # changes incompatibly
418 POKY_BBLAYERS_CONF_VERSION = "2"
419 BBPATH = "${TOPDIR}"
420 BBFILES ?= ""
421 BBLAYERS ?= " \
422 /home/user/poky/meta \
423 /home/user/poky/meta-poky \
424 /home/user/poky/meta-yocto-bsp \
425 /home/user/mystuff/meta-mylayer \
426 "
427
428BitBake parses each ``conf/layer.conf`` file from the top down as
429specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
430file. During the processing of each ``conf/layer.conf`` file, BitBake
431adds the recipes, classes and configurations contained within the
432particular layer to the source directory.
433
434Appending Other Layers Metadata With Your Layer
435===============================================
436
437A recipe that appends Metadata to another recipe is called a BitBake
438append file. A BitBake append file uses the ``.bbappend`` file type
439suffix, while the corresponding recipe to which Metadata is being
440appended uses the ``.bb`` file type suffix.
441
442You can use a ``.bbappend`` file in your layer to make additions or
443changes to the content of another layer's recipe without having to copy
444the other layer's recipe into your layer. Your ``.bbappend`` file
445resides in your layer, while the main ``.bb`` recipe file to which you
446are appending Metadata resides in a different layer.
447
448Being able to append information to an existing recipe not only avoids
449duplication, but also automatically applies recipe changes from a
450different layer into your layer. If you were copying recipes, you would
451have to manually merge changes as they occur.
452
453When you create an append file, you must use the same root name as the
454corresponding recipe file. For example, the append file
455``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
456means the original recipe and append filenames are version
457number-specific. If the corresponding recipe is renamed to update to a
458newer version, you must also rename and possibly update the
459corresponding ``.bbappend`` as well. During the build process, BitBake
460displays an error on starting if it detects a ``.bbappend`` file that
461does not have a corresponding recipe with a matching name. See the
462:term:`BB_DANGLINGAPPENDS_WARNONLY`
463variable for information on how to handle this error.
464
465Overlaying a File Using Your Layer
466----------------------------------
467
468As an example, consider the main formfactor recipe and a corresponding
469formfactor append file both from the :term:`Source Directory`.
470Here is the main
471formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
472the "meta" layer at ``meta/recipes-bsp/formfactor``::
473
474 SUMMARY = "Device formfactor information"
475 DESCRIPTION = "A formfactor configuration file provides information about the \
476 target hardware for which the image is being built and information that the \
477 build system cannot obtain from other sources such as the kernel."
478 SECTION = "base"
479 LICENSE = "MIT"
480 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
481 PR = "r45"
482
483 SRC_URI = "file://config file://machconfig"
484 S = "${WORKDIR}"
485
486 PACKAGE_ARCH = "${MACHINE_ARCH}"
487 INHIBIT_DEFAULT_DEPS = "1"
488
489 do_install() {
490 # Install file only if it has contents
491 install -d ${D}${sysconfdir}/formfactor/
492 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
493 if [ -s "${S}/machconfig" ]; then
494 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
495 fi
496 }
497
498In the main recipe, note the :term:`SRC_URI`
499variable, which tells the OpenEmbedded build system where to find files
500during the build.
501
502Following is the append file, which is named ``formfactor_0.0.bbappend``
503and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
504file is in the layer at ``recipes-bsp/formfactor``::
505
506 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
507
508By default, the build system uses the
509:term:`FILESPATH` variable to
510locate files. This append file extends the locations by setting the
511:term:`FILESEXTRAPATHS`
512variable. Setting this variable in the ``.bbappend`` file is the most
513reliable and recommended method for adding directories to the search
514path used by the build system to find files.
515
516The statement in this example extends the directories to include
517``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
518which resolves to a directory named ``formfactor`` in the same directory
519in which the append file resides (i.e.
520``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
521have the supporting directory structure set up that will contain any
522files or patches you will be including from the layer.
523
524Using the immediate expansion assignment operator ``:=`` is important
525because of the reference to :term:`THISDIR`. The trailing colon character is
526important as it ensures that items in the list remain colon-separated.
527
528.. note::
529
530 BitBake automatically defines the :term:`THISDIR` variable. You should
531 never set this variable yourself. Using ":prepend" as part of the
532 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
533 paths in the final list.
534
535 Also, not all append files add extra files. Many append files simply
536 allow to add build options (e.g. ``systemd``). For these cases, your
537 append file would not even use the :term:`FILESEXTRAPATHS` statement.
538
539The end result of this ``.bbappend`` file is that on a Raspberry Pi, where
540``rpi`` will exist in the list of :term:`OVERRIDES`, the file
541``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be
542used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in
543:ref:`ref-tasks-install` will return true, and the file will be installed.
544
545Installing Additional Files Using Your Layer
546--------------------------------------------
547
548As another example, consider the main ``xserver-xf86-config`` recipe and a
549corresponding ``xserver-xf86-config`` append file both from the :term:`Source
550Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named
551``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at
552``meta/recipes-graphics/xorg-xserver``::
553
554 SUMMARY = "X.Org X server configuration file"
555 HOMEPAGE = "http://www.x.org"
556 SECTION = "x11/base"
557 LICENSE = "MIT"
558 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
559 PR = "r33"
560
561 SRC_URI = "file://xorg.conf"
562
563 S = "${WORKDIR}"
564
565 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf"
566
567 PACKAGE_ARCH = "${MACHINE_ARCH}"
568 ALLOW_EMPTY:${PN} = "1"
569
570 do_install () {
571 if test -s ${WORKDIR}/xorg.conf; then
572 install -d ${D}/${sysconfdir}/X11
573 install -m 0644 ${WORKDIR}/xorg.conf ${D}/${sysconfdir}/X11/
574 fi
575 }
576
577Following is the append file, which is named ``xserver-xf86-config_%.bbappend``
578and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
579file is in the layer at ``recipes-graphics/xorg-xserver``::
580
581 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
582
583 SRC_URI:append:rpi = " \
584 file://xorg.conf.d/98-pitft.conf \
585 file://xorg.conf.d/99-calibration.conf \
586 "
587 do_install:append:rpi () {
588 PITFT="${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}"
589 if [ "${PITFT}" = "1" ]; then
590 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/
591 install -m 0644 ${WORKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
592 install -m 0644 ${WORKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
593 fi
594 }
595
596 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*"
597
598Building off of the previous example, we once again are setting the
599:term:`FILESEXTRAPATHS` variable. In this case we are also using
600:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in
601the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a
602check for an additional :term:`MACHINE_FEATURES` that if set will cause these
603additional files to be installed. These additional files are listed in
604:term:`FILES` so that they will be packaged.
605
606Prioritizing Your Layer
607=======================
608
609Each layer is assigned a priority value. Priority values control which
610layer takes precedence if there are recipe files with the same name in
611multiple layers. For these cases, the recipe file from the layer with a
612higher priority number takes precedence. Priority values also affect the
613order in which multiple ``.bbappend`` files for the same recipe are
614applied. You can either specify the priority manually, or allow the
615build system to calculate it based on the layer's dependencies.
616
617To specify the layer's priority manually, use the
618:term:`BBFILE_PRIORITY`
619variable and append the layer's root name::
620
621 BBFILE_PRIORITY_mylayer = "1"
622
623.. note::
624
625 It is possible for a recipe with a lower version number
626 :term:`PV` in a layer that has a higher
627 priority to take precedence.
628
629 Also, the layer priority does not currently affect the precedence
630 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
631 might address this.
632
633Managing Layers
634===============
635
636You can use the BitBake layer management tool ``bitbake-layers`` to
637provide a view into the structure of recipes across a multi-layer
638project. Being able to generate output that reports on configured layers
639with their paths and priorities and on ``.bbappend`` files and their
640applicable recipes can help to reveal potential problems.
641
642For help on the BitBake layer management tool, use the following
643command::
644
645 $ bitbake-layers --help
646
647The following list describes the available commands:
648
649- ``help:`` Displays general help or help on a specified command.
650
651- ``show-layers:`` Shows the current configured layers.
652
653- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
654 when a recipe with the same name exists in another layer that has a
655 higher layer priority.
656
657- ``show-recipes:`` Lists available recipes and the layers that
658 provide them.
659
660- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
661 which they apply.
662
663- ``show-cross-depends:`` Lists dependency relationships between
664 recipes that cross layer boundaries.
665
666- ``add-layer:`` Adds a layer to ``bblayers.conf``.
667
668- ``remove-layer:`` Removes a layer from ``bblayers.conf``
669
670- ``flatten:`` Flattens the layer configuration into a separate
671 output directory. Flattening your layer configuration builds a
672 "flattened" directory that contains the contents of all layers, with
673 any overlayed recipes removed and any ``.bbappend`` files appended to
674 the corresponding recipes. You might have to perform some manual
675 cleanup of the flattened layer as follows:
676
677 - Non-recipe files (such as patches) are overwritten. The flatten
678 command shows a warning for these files.
679
680 - Anything beyond the normal layer setup has been added to the
681 ``layer.conf`` file. Only the lowest priority layer's
682 ``layer.conf`` is used.
683
684 - Overridden and appended items from ``.bbappend`` files need to be
685 cleaned up. The contents of each ``.bbappend`` end up in the
686 flattened recipe. However, if there are appended or changed
687 variable values, you need to tidy these up yourself. Consider the
688 following example. Here, the ``bitbake-layers`` command adds the
689 line ``#### bbappended ...`` so that you know where the following
690 lines originate::
691
692 ...
693 DESCRIPTION = "A useful utility"
694 ...
695 EXTRA_OECONF = "--enable-something"
696 ...
697
698 #### bbappended from meta-anotherlayer ####
699
700 DESCRIPTION = "Customized utility"
701 EXTRA_OECONF += "--enable-somethingelse"
702
703
704 Ideally, you would tidy up these utilities as follows::
705
706 ...
707 DESCRIPTION = "Customized utility"
708 ...
709 EXTRA_OECONF = "--enable-something --enable-somethingelse"
710 ...
711
712- ``layerindex-fetch``: Fetches a layer from a layer index, along
713 with its dependent layers, and adds the layers to the
714 ``conf/bblayers.conf`` file.
715
716- ``layerindex-show-depends``: Finds layer dependencies from the
717 layer index.
718
719- ``create-layer``: Creates a basic layer.
720
721Creating a General Layer Using the ``bitbake-layers`` Script
722============================================================
723
724The ``bitbake-layers`` script with the ``create-layer`` subcommand
725simplifies creating a new general layer.
726
727.. note::
728
729 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
730 section in the Yocto
731 Project Board Specific (BSP) Developer's Guide.
732
733 - In order to use a layer with the OpenEmbedded build system, you
734 need to add the layer to your ``bblayers.conf`` configuration
735 file. See the ":ref:`dev-manual/layers:adding a layer using the \`\`bitbake-layers\`\` script`"
736 section for more information.
737
738The default mode of the script's operation with this subcommand is to
739create a layer with the following:
740
741- A layer priority of 6.
742
743- A ``conf`` subdirectory that contains a ``layer.conf`` file.
744
745- A ``recipes-example`` subdirectory that contains a further
746 subdirectory named ``example``, which contains an ``example.bb``
747 recipe file.
748
749- A ``COPYING.MIT``, which is the license statement for the layer. The
750 script assumes you want to use the MIT license, which is typical for
751 most layers, for the contents of the layer itself.
752
753- A ``README`` file, which is a file describing the contents of your
754 new layer.
755
756In its simplest form, you can use the following command form to create a
757layer. The command creates a layer whose name corresponds to
758"your_layer_name" in the current directory::
759
760 $ bitbake-layers create-layer your_layer_name
761
762As an example, the following command creates a layer named ``meta-scottrif``
763in your home directory::
764
765 $ cd /usr/home
766 $ bitbake-layers create-layer meta-scottrif
767 NOTE: Starting bitbake server...
768 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
769
770If you want to set the priority of the layer to other than the default
771value of "6", you can either use the ``--priority`` option or you
772can edit the
773:term:`BBFILE_PRIORITY` value
774in the ``conf/layer.conf`` after the script creates it. Furthermore, if
775you want to give the example recipe file some name other than the
776default, you can use the ``--example-recipe-name`` option.
777
778The easiest way to see how the ``bitbake-layers create-layer`` command
779works is to experiment with the script. You can also read the usage
780information by entering the following::
781
782 $ bitbake-layers create-layer --help
783 NOTE: Starting bitbake server...
784 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
785 [--example-recipe-name EXAMPLERECIPE]
786 layerdir
787
788 Create a basic layer
789
790 positional arguments:
791 layerdir Layer directory to create
792
793 optional arguments:
794 -h, --help show this help message and exit
795 --priority PRIORITY, -p PRIORITY
796 Layer directory to create
797 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
798 Filename of the example recipe
799
800Adding a Layer Using the ``bitbake-layers`` Script
801==================================================
802
803Once you create your general layer, you must add it to your
804``bblayers.conf`` file. Adding the layer to this configuration file
805makes the OpenEmbedded build system aware of your layer so that it can
806search it for metadata.
807
808Add your layer by using the ``bitbake-layers add-layer`` command::
809
810 $ bitbake-layers add-layer your_layer_name
811
812Here is an example that adds a
813layer named ``meta-scottrif`` to the configuration file. Following the
814command that adds the layer is another ``bitbake-layers`` command that
815shows the layers that are in your ``bblayers.conf`` file::
816
817 $ bitbake-layers add-layer meta-scottrif
818 NOTE: Starting bitbake server...
819 Parsing recipes: 100% |##########################################################| Time: 0:00:49
820 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
821 $ bitbake-layers show-layers
822 NOTE: Starting bitbake server...
823 layer path priority
824 ==========================================================================
825 meta /home/scottrif/poky/meta 5
826 meta-poky /home/scottrif/poky/meta-poky 5
827 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
828 workspace /home/scottrif/poky/build/workspace 99
829 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
830
831
832Adding the layer to this file
833enables the build system to locate the layer during the build.
834
835.. note::
836
837 During a build, the OpenEmbedded build system looks in the layers
838 from the top of the list down to the bottom in that order.
839
diff --git a/documentation/dev-manual/libraries.rst b/documentation/dev-manual/libraries.rst
new file mode 100644
index 0000000000..ae4ca27209
--- /dev/null
+++ b/documentation/dev-manual/libraries.rst
@@ -0,0 +1,267 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working With Libraries
4**********************
5
6Libraries are an integral part of your system. This section describes
7some common practices you might find helpful when working with libraries
8to build your system:
9
10- :ref:`How to include static library files
11 <dev-manual/libraries:including static library files>`
12
13- :ref:`How to use the Multilib feature to combine multiple versions of
14 library files into a single image
15 <dev-manual/libraries:combining multiple versions of library files into one image>`
16
17- :ref:`How to install multiple versions of the same library in parallel on
18 the same system
19 <dev-manual/libraries:installing multiple versions of the same library>`
20
21Including Static Library Files
22==============================
23
24If you are building a library and the library offers static linking, you
25can control which static library files (``*.a`` files) get included in
26the built library.
27
28The :term:`PACKAGES` and
29:term:`FILES:* <FILES>` variables in the
30``meta/conf/bitbake.conf`` configuration file define how files installed
31by the :ref:`ref-tasks-install` task are packaged. By default, the :term:`PACKAGES`
32variable includes ``${PN}-staticdev``, which represents all static
33library files.
34
35.. note::
36
37 Some previously released versions of the Yocto Project defined the
38 static library files through ``${PN}-dev``.
39
40Following is part of the BitBake configuration file, where you can see
41how the static library files are defined::
42
43 PACKAGE_BEFORE_PN ?= ""
44 PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
45 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
46 FILES = ""
47
48 FILES:${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
49 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
50 ${base_bindir}/* ${base_sbindir}/* \
51 ${base_libdir}/*${SOLIBS} \
52 ${base_prefix}/lib/udev ${prefix}/lib/udev \
53 ${base_libdir}/udev ${libdir}/udev \
54 ${datadir}/${BPN} ${libdir}/${BPN}/* \
55 ${datadir}/pixmaps ${datadir}/applications \
56 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
57 ${libdir}/bonobo/servers"
58
59 FILES:${PN}-bin = "${bindir}/* ${sbindir}/*"
60
61 FILES:${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
62 ${datadir}/gnome/help"
63 SECTION:${PN}-doc = "doc"
64
65 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
66 FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
67 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
68 ${datadir}/aclocal ${base_libdir}/*.o \
69 ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
70 ${libdir}/cmake ${datadir}/cmake"
71 SECTION:${PN}-dev = "devel"
72 ALLOW_EMPTY:${PN}-dev = "1"
73 RDEPENDS:${PN}-dev = "${PN} (= ${EXTENDPKGV})"
74
75 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
76 SECTION:${PN}-staticdev = "devel"
77 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
78
79Combining Multiple Versions of Library Files into One Image
80===========================================================
81
82The build system offers the ability to build libraries with different
83target optimizations or architecture formats and combine these together
84into one system image. You can link different binaries in the image
85against the different libraries as needed for specific use cases. This
86feature is called "Multilib".
87
88An example would be where you have most of a system compiled in 32-bit
89mode using 32-bit libraries, but you have something large, like a
90database engine, that needs to be a 64-bit application and uses 64-bit
91libraries. Multilib allows you to get the best of both 32-bit and 64-bit
92libraries.
93
94While the Multilib feature is most commonly used for 32 and 64-bit
95differences, the approach the build system uses facilitates different
96target optimizations. You could compile some binaries to use one set of
97libraries and other binaries to use a different set of libraries. The
98libraries could differ in architecture, compiler options, or other
99optimizations.
100
101There are several examples in the ``meta-skeleton`` layer found in the
102:term:`Source Directory`:
103
104- :oe_git:`conf/multilib-example.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example.conf>`
105 configuration file.
106
107- :oe_git:`conf/multilib-example2.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example2.conf>`
108 configuration file.
109
110- :oe_git:`recipes-multilib/images/core-image-multilib-example.bb </openembedded-core/tree/meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb>`
111 recipe
112
113Preparing to Use Multilib
114-------------------------
115
116User-specific requirements drive the Multilib feature. Consequently,
117there is no one "out-of-the-box" configuration that would
118meet your needs.
119
120In order to enable Multilib, you first need to ensure your recipe is
121extended to support multiple libraries. Many standard recipes are
122already extended and support multiple libraries. You can check in the
123``meta/conf/multilib.conf`` configuration file in the
124:term:`Source Directory` to see how this is
125done using the
126:term:`BBCLASSEXTEND` variable.
127Eventually, all recipes will be covered and this list will not be
128needed.
129
130For the most part, the :ref:`Multilib <ref-classes-multilib*>`
131class extension works automatically to
132extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
133:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
134Standard variables such as
135:term:`DEPENDS`,
136:term:`RDEPENDS`,
137:term:`RPROVIDES`,
138:term:`RRECOMMENDS`,
139:term:`PACKAGES`, and
140:term:`PACKAGES_DYNAMIC` are
141automatically extended by the system. If you are extending any manual
142code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
143those names are extended correctly.
144
145Using Multilib
146--------------
147
148After you have set up the recipes, you need to define the actual
149combination of multiple libraries you want to build. You accomplish this
150through your ``local.conf`` configuration file in the
151:term:`Build Directory`. An example configuration would be as follows::
152
153 MACHINE = "qemux86-64"
154 require conf/multilib.conf
155 MULTILIBS = "multilib:lib32"
156 DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
157 IMAGE_INSTALL:append = " lib32-glib-2.0"
158
159This example enables an additional library named
160``lib32`` alongside the normal target packages. When combining these
161"lib32" alternatives, the example uses "x86" for tuning. For information
162on this particular tuning, see
163``meta/conf/machine/include/ia32/arch-ia32.inc``.
164
165The example then includes ``lib32-glib-2.0`` in all the images, which
166illustrates one method of including a multiple library dependency. You
167can use a normal image build to include this dependency, for example::
168
169 $ bitbake core-image-sato
170
171You can also build Multilib packages
172specifically with a command like this::
173
174 $ bitbake lib32-glib-2.0
175
176Additional Implementation Details
177---------------------------------
178
179There are generic implementation details as well as details that are specific to
180package management systems. Following are implementation details
181that exist regardless of the package management system:
182
183- The typical convention used for the class extension code as used by
184 Multilib assumes that all package names specified in
185 :term:`PACKAGES` that contain
186 ``${PN}`` have ``${PN}`` at the start of the name. When that
187 convention is not followed and ``${PN}`` appears at the middle or the
188 end of a name, problems occur.
189
190- The :term:`TARGET_VENDOR`
191 value under Multilib will be extended to "-vendormlmultilib" (e.g.
192 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
193 slightly unwieldy contraction is that any "-" characters in the
194 vendor string presently break Autoconf's ``config.sub``, and other
195 separators are problematic for different reasons.
196
197Here are the implementation details for the RPM Package Management System:
198
199- A unique architecture is defined for the Multilib packages, along
200 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
201 :term:`Build Directory`. For example, consider ``lib32`` in a
202 ``qemux86-64`` image. The possible architectures in the system are "all",
203 "qemux86_64", "lib32:qemux86_64", and "lib32:x86".
204
205- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
206 packaging. The naming for a normal RPM package and a Multilib RPM
207 package in a ``qemux86-64`` system resolves to something similar to
208 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
209 respectively.
210
211- When installing a Multilib image, the RPM backend first installs the
212 base image and then installs the Multilib libraries.
213
214- The build system relies on RPM to resolve the identical files in the
215 two (or more) Multilib packages.
216
217Here are the implementation details for the IPK Package Management System:
218
219- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
220 packaging. The naming for a normal RPM package and a Multilib IPK
221 package in a ``qemux86-64`` system resolves to something like
222 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``,
223 respectively.
224
225- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
226 packages with and without the Multilib feature can exist in the same
227 folder due to the ``${PN}`` differences.
228
229- IPK defines a sanity check for Multilib installation using certain
230 rules for file comparison, overridden, etc.
231
232Installing Multiple Versions of the Same Library
233================================================
234
235There are be situations where you need to install and use multiple versions
236of the same library on the same system at the same time. This
237almost always happens when a library API changes and you have
238multiple pieces of software that depend on the separate versions of the
239library. To accommodate these situations, you can install multiple
240versions of the same library in parallel on the same system.
241
242The process is straightforward as long as the libraries use proper
243versioning. With properly versioned libraries, all you need to do to
244individually specify the libraries is create separate, appropriately
245named recipes where the :term:`PN` part of
246the name includes a portion that differentiates each library version
247(e.g. the major part of the version number). Thus, instead of having a
248single recipe that loads one version of a library (e.g. ``clutter``),
249you provide multiple recipes that result in different versions of the
250libraries you want. As an example, the following two recipes would allow
251the two separate versions of the ``clutter`` library to co-exist on the
252same system:
253
254.. code-block:: none
255
256 clutter-1.6_1.6.20.bb
257 clutter-1.8_1.8.4.bb
258
259Additionally, if
260you have other recipes that depend on a given library, you need to use
261the :term:`DEPENDS` variable to
262create the dependency. Continuing with the same example, if you want to
263have a recipe depend on the 1.8 version of the ``clutter`` library, use
264the following in your recipe::
265
266 DEPENDS = "clutter-1.8"
267
diff --git a/documentation/dev-manual/licenses.rst b/documentation/dev-manual/licenses.rst
new file mode 100644
index 0000000000..b485f77708
--- /dev/null
+++ b/documentation/dev-manual/licenses.rst
@@ -0,0 +1,524 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working With Licenses
4*********************
5
6As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
7section in the Yocto Project Overview and Concepts Manual, open source
8projects are open to the public and they consequently have different
9licensing structures in place. This section describes the mechanism by
10which the :term:`OpenEmbedded Build System`
11tracks changes to
12licensing text and covers how to maintain open source license compliance
13during your project's lifecycle. The section also describes how to
14enable commercially licensed recipes, which by default are disabled.
15
16Tracking License Changes
17========================
18
19The license of an upstream project might change in the future. In order
20to prevent these changes going unnoticed, the
21:term:`LIC_FILES_CHKSUM`
22variable tracks changes to the license text. The checksums are validated
23at the end of the configure step, and if the checksums do not match, the
24build will fail.
25
26Specifying the ``LIC_FILES_CHKSUM`` Variable
27--------------------------------------------
28
29The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
30in the source code for the recipe. Following is an example of how to
31specify :term:`LIC_FILES_CHKSUM`::
32
33 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
34 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
35 file://licfile2.txt;endline=50;md5=zzzz \
36 ..."
37
38.. note::
39
40 - When using "beginline" and "endline", realize that line numbering
41 begins with one and not zero. Also, the included lines are
42 inclusive (i.e. lines five through and including 29 in the
43 previous example for ``licfile1.txt``).
44
45 - When a license check fails, the selected license text is included
46 as part of the QA message. Using this output, you can determine
47 the exact start and finish for the needed license text.
48
49The build system uses the :term:`S`
50variable as the default directory when searching files listed in
51:term:`LIC_FILES_CHKSUM`. The previous example employs the default
52directory.
53
54Consider this next example::
55
56 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
57 md5=bb14ed3c4cda583abc85401304b5cd4e"
58 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
59
60The first line locates a file in ``${S}/src/ls.c`` and isolates lines
61five through 16 as license text. The second line refers to a file in
62:term:`WORKDIR`.
63
64Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
65unless the :term:`LICENSE` variable is set to "CLOSED".
66
67Explanation of Syntax
68---------------------
69
70As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
71lists all the important files that contain the license text for the
72source code. It is possible to specify a checksum for an entire file, or
73a specific section of a file (specified by beginning and ending line
74numbers with the "beginline" and "endline" parameters, respectively).
75The latter is useful for source files with a license notice header,
76README documents, and so forth. If you do not use the "beginline"
77parameter, then it is assumed that the text begins on the first line of
78the file. Similarly, if you do not use the "endline" parameter, it is
79assumed that the license text ends with the last line of the file.
80
81The "md5" parameter stores the md5 checksum of the license text. If the
82license text changes in any way as compared to this parameter then a
83mismatch occurs. This mismatch triggers a build failure and notifies the
84developer. Notification allows the developer to review and address the
85license text changes. Also note that if a mismatch occurs during the
86build, the correct md5 checksum is placed in the build log and can be
87easily copied to the recipe.
88
89There is no limit to how many files you can specify using the
90:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
91requires a few specifications for license tracking. Many projects have a
92"COPYING" file that stores the license information for all the source
93code files. This practice allows you to just track the "COPYING" file as
94long as it is kept up to date.
95
96.. note::
97
98 - If you specify an empty or invalid "md5" parameter,
99 :term:`BitBake` returns an md5
100 mis-match error and displays the correct "md5" parameter value
101 during the build. The correct parameter is also captured in the
102 build log.
103
104 - If the whole file contains only license text, you do not need to
105 use the "beginline" and "endline" parameters.
106
107Enabling Commercially Licensed Recipes
108======================================
109
110By default, the OpenEmbedded build system disables components that have
111commercial or other special licensing requirements. Such requirements
112are defined on a recipe-by-recipe basis through the
113:term:`LICENSE_FLAGS` variable
114definition in the affected recipe. For instance, the
115``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
116contains the following statement::
117
118 LICENSE_FLAGS = "commercial"
119
120Here is a
121slightly more complicated example that contains both an explicit recipe
122name and version (after variable expansion)::
123
124 LICENSE_FLAGS = "license_${PN}_${PV}"
125
126In order for a component restricted by a
127:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
128needs to have a matching entry in the global
129:term:`LICENSE_FLAGS_ACCEPTED`
130variable, which is a variable typically defined in your ``local.conf``
131file. For example, to enable the
132``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
133could add either the string "commercial_gst-plugins-ugly" or the more
134general string "commercial" to :term:`LICENSE_FLAGS_ACCEPTED`. See the
135":ref:`dev-manual/licenses:license flag matching`" section for a full
136explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
137example::
138
139 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly"
140
141Likewise, to additionally enable the package built from the recipe
142containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
143the actual recipe name was ``emgd_1.10.bb``, the following string would
144enable that package as well as the original ``gst-plugins-ugly``
145package::
146
147 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly license_emgd_1.10"
148
149As a convenience, you do not need to specify the
150complete license string for every package. You can use
151an abbreviated form, which consists of just the first portion or
152portions of the license string before the initial underscore character
153or characters. A partial string will match any license that contains the
154given string as the first portion of its license. For example, the
155following value will also match both of the packages
156previously mentioned as well as any other packages that have licenses
157starting with "commercial" or "license"::
158
159 LICENSE_FLAGS_ACCEPTED = "commercial license"
160
161License Flag Matching
162---------------------
163
164License flag matching allows you to control what recipes the
165OpenEmbedded build system includes in the build. Fundamentally, the
166build system attempts to match :term:`LICENSE_FLAGS` strings found in
167recipes against strings found in :term:`LICENSE_FLAGS_ACCEPTED`.
168A match causes the build system to include a recipe in the
169build, while failure to find a match causes the build system to exclude
170a recipe.
171
172In general, license flag matching is simple. However, understanding some
173concepts will help you correctly and effectively use matching.
174
175Before a flag defined by a particular recipe is tested against the
176entries of :term:`LICENSE_FLAGS_ACCEPTED`, the expanded
177string ``_${PN}`` is appended to the flag. This expansion makes each
178:term:`LICENSE_FLAGS` value recipe-specific. After expansion, the
179string is then matched against the entries. Thus, specifying
180``LICENSE_FLAGS = "commercial"`` in recipe "foo", for example, results
181in the string ``"commercial_foo"``. And, to create a match, that string
182must appear among the entries of :term:`LICENSE_FLAGS_ACCEPTED`.
183
184Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
185:term:`LICENSE_FLAGS_ACCEPTED` variable allows you a lot of flexibility for
186including or excluding recipes based on licensing. For example, you can
187broaden the matching capabilities by using license flags string subsets
188in :term:`LICENSE_FLAGS_ACCEPTED`.
189
190.. note::
191
192 When using a string subset, be sure to use the part of the expanded
193 string that precedes the appended underscore character (e.g.
194 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
195
196For example, simply specifying the string "commercial" in the
197:term:`LICENSE_FLAGS_ACCEPTED` variable matches any expanded
198:term:`LICENSE_FLAGS` definition that starts with the string
199"commercial" such as "commercial_foo" and "commercial_bar", which
200are the strings the build system automatically generates for
201hypothetical recipes named "foo" and "bar" assuming those recipes simply
202specify the following::
203
204 LICENSE_FLAGS = "commercial"
205
206Thus, you can choose to exhaustively enumerate each license flag in the
207list and allow only specific recipes into the image, or you can use a
208string subset that causes a broader range of matches to allow a range of
209recipes into the image.
210
211This scheme works even if the :term:`LICENSE_FLAGS` string already has
212``_${PN}`` appended. For example, the build system turns the license
213flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
214both the general "commercial" and the specific "commercial_1.2_foo"
215strings found in the :term:`LICENSE_FLAGS_ACCEPTED` variable, as expected.
216
217Here are some other scenarios:
218
219- You can specify a versioned string in the recipe such as
220 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
221 string to "commercial_foo_1.2_foo". Combine this license flag with a
222 :term:`LICENSE_FLAGS_ACCEPTED` variable that has the string
223 "commercial" and you match the flag along with any other flag that
224 starts with the string "commercial".
225
226- Under the same circumstances, you can add "commercial_foo" in the
227 :term:`LICENSE_FLAGS_ACCEPTED` variable and the build system not only
228 matches "commercial_foo_1.2" but also matches any license flag with
229 the string "commercial_foo", regardless of the version.
230
231- You can be very specific and use both the package and version parts
232 in the :term:`LICENSE_FLAGS_ACCEPTED` list (e.g.
233 "commercial_foo_1.2") to specifically match a versioned recipe.
234
235Other Variables Related to Commercial Licenses
236----------------------------------------------
237
238There are other helpful variables related to commercial license handling,
239defined in the
240``poky/meta/conf/distro/include/default-distrovars.inc`` file::
241
242 COMMERCIAL_AUDIO_PLUGINS ?= ""
243 COMMERCIAL_VIDEO_PLUGINS ?= ""
244
245If you want to enable these components, you can do so by making sure you have
246statements similar to the following in your ``local.conf`` configuration file::
247
248 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
249 gst-plugins-ugly-mpegaudioparse"
250 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
251 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
252 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
253
254Of course, you could also create a matching list for those components using the
255more general "commercial" string in the :term:`LICENSE_FLAGS_ACCEPTED` variable,
256but that would also enable all the other packages with :term:`LICENSE_FLAGS`
257containing "commercial", which you may or may not want::
258
259 LICENSE_FLAGS_ACCEPTED = "commercial"
260
261Specifying audio and video plugins as part of the
262:term:`COMMERCIAL_AUDIO_PLUGINS` and :term:`COMMERCIAL_VIDEO_PLUGINS` statements
263(along with :term:`LICENSE_FLAGS_ACCEPTED`) includes the plugins or
264components into built images, thus adding support for media formats or
265components.
266
267.. note::
268
269 GStreamer "ugly" and "bad" plugins are actually available through
270 open source licenses. However, the "ugly" ones can be subject to software
271 patents in some countries, making it necessary to pay licensing fees
272 to distribute them. The "bad" ones are just deemed unreliable by the
273 GStreamer community and should therefore be used with care.
274
275Maintaining Open Source License Compliance During Your Product's Lifecycle
276==========================================================================
277
278One of the concerns for a development organization using open source
279software is how to maintain compliance with various open source
280licensing during the lifecycle of the product. While this section does
281not provide legal advice or comprehensively cover all scenarios, it does
282present methods that you can use to assist you in meeting the compliance
283requirements during a software release.
284
285With hundreds of different open source licenses that the Yocto Project
286tracks, it is difficult to know the requirements of each and every
287license. However, the requirements of the major FLOSS licenses can begin
288to be covered by assuming that there are three main areas of concern:
289
290- Source code must be provided.
291
292- License text for the software must be provided.
293
294- Compilation scripts and modifications to the source code must be
295 provided.
296
297- spdx files can be provided.
298
299There are other requirements beyond the scope of these three and the
300methods described in this section (e.g. the mechanism through which
301source code is distributed).
302
303As different organizations have different methods of complying with open
304source licensing, this section is not meant to imply that there is only
305one single way to meet your compliance obligations, but rather to
306describe one method of achieving compliance. The remainder of this
307section describes methods supported to meet the previously mentioned
308three requirements. Once you take steps to meet these requirements, and
309prior to releasing images, sources, and the build system, you should
310audit all artifacts to ensure completeness.
311
312.. note::
313
314 The Yocto Project generates a license manifest during image creation
315 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
316 to assist with any audits.
317
318Providing the Source Code
319-------------------------
320
321Compliance activities should begin before you generate the final image.
322The first thing you should look at is the requirement that tops the list
323for most compliance groups --- providing the source. The Yocto Project has
324a few ways of meeting this requirement.
325
326One of the easiest ways to meet this requirement is to provide the
327entire :term:`DL_DIR` used by the
328build. This method, however, has a few issues. The most obvious is the
329size of the directory since it includes all sources used in the build
330and not just the source used in the released image. It will include
331toolchain source, and other artifacts, which you would not generally
332release. However, the more serious issue for most companies is
333accidental release of proprietary software. The Yocto Project provides
334an :ref:`ref-classes-archiver` class to help avoid some of these concerns.
335
336Before you employ :term:`DL_DIR` or the :ref:`ref-classes-archiver` class, you
337need to decide how you choose to provide source. The source
338:ref:`ref-classes-archiver` class can generate tarballs and SRPMs and can
339create them with various levels of compliance in mind.
340
341One way of doing this (but certainly not the only way) is to release
342just the source as a tarball. You can do this by adding the following to
343the ``local.conf`` file found in the :term:`Build Directory`::
344
345 INHERIT += "archiver"
346 ARCHIVER_MODE[src] = "original"
347
348During the creation of your
349image, the source from all recipes that deploy packages to the image is
350placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
351:term:`LICENSE` for each recipe.
352Releasing the entire directory enables you to comply with requirements
353concerning providing the unmodified source. It is important to note that
354the size of the directory can get large.
355
356A way to help mitigate the size issue is to only release tarballs for
357licenses that require the release of source. Let us assume you are only
358concerned with GPL code as identified by running the following script:
359
360.. code-block:: shell
361
362 # Script to archive a subset of packages matching specific license(s)
363 # Source and license files are copied into sub folders of package folder
364 # Must be run from build folder
365 #!/bin/bash
366 src_release_dir="source-release"
367 mkdir -p $src_release_dir
368 for a in tmp/deploy/sources/*; do
369 for d in $a/*; do
370 # Get package name from path
371 p=`basename $d`
372 p=${p%-*}
373 p=${p%-*}
374 # Only archive GPL packages (update *GPL* regex for your license check)
375 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
376 if [ $numfiles -ge 1 ]; then
377 echo Archiving $p
378 mkdir -p $src_release_dir/$p/source
379 cp $d/* $src_release_dir/$p/source 2> /dev/null
380 mkdir -p $src_release_dir/$p/license
381 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
382 fi
383 done
384 done
385
386At this point, you
387could create a tarball from the ``gpl_source_release`` directory and
388provide that to the end user. This method would be a step toward
389achieving compliance with section 3a of GPLv2 and with section 6 of
390GPLv3.
391
392Providing License Text
393~~~~~~~~~~~~~~~~~~~~~~
394
395One requirement that is often overlooked is inclusion of license text.
396This requirement also needs to be dealt with prior to generating the
397final image. Some licenses require the license text to accompany the
398binary. You can achieve this by adding the following to your
399``local.conf`` file::
400
401 COPY_LIC_MANIFEST = "1"
402 COPY_LIC_DIRS = "1"
403 LICENSE_CREATE_PACKAGE = "1"
404
405Adding these statements to the
406configuration file ensures that the licenses collected during package
407generation are included on your image.
408
409.. note::
410
411 Setting all three variables to "1" results in the image having two
412 copies of the same license file. One copy resides in
413 ``/usr/share/common-licenses`` and the other resides in
414 ``/usr/share/license``.
415
416 The reason for this behavior is because
417 :term:`COPY_LIC_DIRS` and
418 :term:`COPY_LIC_MANIFEST`
419 add a copy of the license when the image is built but do not offer a
420 path for adding licenses for newly installed packages to an image.
421 :term:`LICENSE_CREATE_PACKAGE`
422 adds a separate package and an upgrade path for adding licenses to an
423 image.
424
425As the source :ref:`ref-classes-archiver` class has already archived the
426original unmodified source that contains the license files, you would have
427already met the requirements for inclusion of the license information
428with source as defined by the GPL and other open source licenses.
429
430Providing Compilation Scripts and Source Code Modifications
431-----------------------------------------------------------
432
433At this point, we have addressed all we need to prior to generating the
434image. The next two requirements are addressed during the final
435packaging of the release.
436
437By releasing the version of the OpenEmbedded build system and the layers
438used during the build, you will be providing both compilation scripts
439and the source code modifications in one step.
440
441If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
442and a distro layer, and those
443those layers are used to patch, compile, package, or modify (in any way)
444any open source software included in your released images, you might be
445required to release those layers under section 3 of GPLv2 or section 1
446of GPLv3. One way of doing that is with a clean checkout of the version
447of the Yocto Project and layers used during your build. Here is an
448example:
449
450.. code-block:: shell
451
452 # We built using the dunfell branch of the poky repo
453 $ git clone -b dunfell git://git.yoctoproject.org/poky
454 $ cd poky
455 # We built using the release_branch for our layers
456 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
457 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
458 # clean up the .git repos
459 $ find . -name ".git" -type d -exec rm -rf {} \;
460
461One
462thing a development organization might want to consider for end-user
463convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to
464ensure that when the end user utilizes the released build system to
465build an image, the development organization's layers are included in
466the ``bblayers.conf`` file automatically::
467
468 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
469 # changes incompatibly
470 POKY_BBLAYERS_CONF_VERSION = "2"
471
472 BBPATH = "${TOPDIR}"
473 BBFILES ?= ""
474
475 BBLAYERS ?= " \
476 ##OEROOT##/meta \
477 ##OEROOT##/meta-poky \
478 ##OEROOT##/meta-yocto-bsp \
479 ##OEROOT##/meta-mylayer \
480 "
481
482Creating and
483providing an archive of the :term:`Metadata`
484layers (recipes, configuration files, and so forth) enables you to meet
485your requirements to include the scripts to control compilation as well
486as any modifications to the original source.
487
488Compliance Limitations with Executables Built from Static Libraries
489-------------------------------------------------------------------
490
491When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS`
492mechanisms as well as explicitly included in the image recipe with
493:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B
494(``DEPENDS += "B"``), package B will neither appear in the generated license
495manifest nor in the generated source tarballs. This occurs as the
496:ref:`ref-classes-license` and :ref:`ref-classes-archiver` classes assume that
497only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS`
498end up in the image.
499
500As a result, potential obligations regarding license compliance for package B
501may not be met.
502
503The Yocto Project doesn't enable static libraries by default, in part because
504of this issue. Before a solution to this limitation is found, you need to
505keep in mind that if your root filesystem is built from static libraries,
506you will need to manually ensure that your deliveries are compliant
507with the licenses of these libraries.
508
509Copying Non Standard Licenses
510=============================
511
512Some packages, such as the linux-firmware package, have many licenses
513that are not in any way common. You can avoid adding a lot of these
514types of common license files, which are only applicable to a specific
515package, by using the
516:term:`NO_GENERIC_LICENSE`
517variable. Using this variable also avoids QA errors when you use a
518non-common, non-CLOSED license in a recipe.
519
520Here is an example that uses the ``LICENSE.Abilis.txt`` file as
521the license from the fetched source::
522
523 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
524
diff --git a/documentation/dev-manual/new-machine.rst b/documentation/dev-manual/new-machine.rst
new file mode 100644
index 0000000000..b7978f4ba9
--- /dev/null
+++ b/documentation/dev-manual/new-machine.rst
@@ -0,0 +1,118 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Adding a New Machine
4====================
5
6Adding a new machine to the Yocto Project is a straightforward process.
7This section describes how to add machines that are similar to those
8that the Yocto Project already supports.
9
10.. note::
11
12 Although well within the capabilities of the Yocto Project, adding a
13 totally new architecture might require changes to ``gcc``/``glibc``
14 and to the site information, which is beyond the scope of this
15 manual.
16
17For a complete example that shows how to add a new machine, see the
18":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
19section in the Yocto Project Board Support Package (BSP) Developer's
20Guide.
21
22Adding the Machine Configuration File
23=====================================
24
25To add a new machine, you need to add a new machine configuration file
26to the layer's ``conf/machine`` directory. This configuration file
27provides details about the device you are adding.
28
29The OpenEmbedded build system uses the root name of the machine
30configuration file to reference the new machine. For example, given a
31machine configuration file named ``crownbay.conf``, the build system
32recognizes the machine as "crownbay".
33
34The most important variables you must set in your machine configuration
35file or include from a lower-level configuration file are as follows:
36
37- :term:`TARGET_ARCH` (e.g. "arm")
38
39- ``PREFERRED_PROVIDER_virtual/kernel``
40
41- :term:`MACHINE_FEATURES` (e.g. "apm screen wifi")
42
43You might also need these variables:
44
45- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
46
47- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
48
49- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
50
51You can find full details on these variables in the reference section.
52You can leverage existing machine ``.conf`` files from
53``meta-yocto-bsp/conf/machine/``.
54
55Adding a Kernel for the Machine
56===============================
57
58The OpenEmbedded build system needs to be able to build a kernel for the
59machine. You need to either create a new kernel recipe for this machine,
60or extend an existing kernel recipe. You can find several kernel recipe
61examples in the Source Directory at ``meta/recipes-kernel/linux`` that
62you can use as references.
63
64If you are creating a new kernel recipe, normal recipe-writing rules
65apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
66necessary patches and set :term:`S` to point at the source code. You need to
67create a :ref:`ref-tasks-configure` task that configures the unpacked kernel with
68a ``defconfig`` file. You can do this by using a ``make defconfig``
69command or, more commonly, by copying in a suitable ``defconfig`` file
70and then running ``make oldconfig``. By making use of ``inherit kernel``
71and potentially some of the ``linux-*.inc`` files, most other
72functionality is centralized and the defaults of the class normally work
73well.
74
75If you are extending an existing kernel recipe, it is usually a matter
76of adding a suitable ``defconfig`` file. The file needs to be added into
77a location similar to ``defconfig`` files used for other machines in a
78given kernel recipe. A possible way to do this is by listing the file in
79the :term:`SRC_URI` and adding the machine to the expression in
80:term:`COMPATIBLE_MACHINE`::
81
82 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
83
84For more information on ``defconfig`` files, see the
85":ref:`kernel-dev/common:changing the configuration`"
86section in the Yocto Project Linux Kernel Development Manual.
87
88Adding a Formfactor Configuration File
89======================================
90
91A formfactor configuration file provides information about the target
92hardware for which the image is being built and information that the
93build system cannot obtain from other sources such as the kernel. Some
94examples of information contained in a formfactor configuration file
95include framebuffer orientation, whether or not the system has a
96keyboard, the positioning of the keyboard in relation to the screen, and
97the screen resolution.
98
99The build system uses reasonable defaults in most cases. However, if
100customization is necessary, you need to create a ``machconfig`` file in
101the ``meta/recipes-bsp/formfactor/files`` directory. This directory
102contains directories for specific machines such as ``qemuarm`` and
103``qemux86``. For information about the settings available and the
104defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
105found in the same area.
106
107Following is an example for "qemuarm" machine::
108
109 HAVE_TOUCHSCREEN=1
110 HAVE_KEYBOARD=1
111 DISPLAY_CAN_ROTATE=0
112 DISPLAY_ORIENTATION=0
113 #DISPLAY_WIDTH_PIXELS=640
114 #DISPLAY_HEIGHT_PIXELS=480
115 #DISPLAY_BPP=16
116 DISPLAY_DPI=150
117 DISPLAY_SUBPIXEL_ORDER=vrgb
118
diff --git a/documentation/dev-manual/new-recipe.rst b/documentation/dev-manual/new-recipe.rst
new file mode 100644
index 0000000000..2c1101ec2b
--- /dev/null
+++ b/documentation/dev-manual/new-recipe.rst
@@ -0,0 +1,1634 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Writing a New Recipe
4********************
5
6Recipes (``.bb`` files) are fundamental components in the Yocto Project
7environment. Each software component built by the OpenEmbedded build
8system requires a recipe to define the component. This section describes
9how to create, write, and test a new recipe.
10
11.. note::
12
13 For information on variables that are useful for recipes and for
14 information about recipe naming issues, see the
15 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
16 Reference Manual.
17
18Overview
19========
20
21The following figure shows the basic process for creating a new recipe.
22The remainder of the section provides details for the steps.
23
24.. image:: figures/recipe-workflow.png
25 :align: center
26 :width: 50%
27
28Locate or Automatically Create a Base Recipe
29============================================
30
31You can always write a recipe from scratch. However, there are three choices
32that can help you quickly get started with a new recipe:
33
34- ``devtool add``: A command that assists in creating a recipe and an
35 environment conducive to development.
36
37- ``recipetool create``: A command provided by the Yocto Project that
38 automates creation of a base recipe based on the source files.
39
40- *Existing Recipes:* Location and modification of an existing recipe
41 that is similar in function to the recipe you need.
42
43.. note::
44
45 For information on recipe syntax, see the
46 ":ref:`dev-manual/new-recipe:recipe syntax`" section.
47
48Creating the Base Recipe Using ``devtool add``
49----------------------------------------------
50
51The ``devtool add`` command uses the same logic for auto-creating the
52recipe as ``recipetool create``, which is listed below. Additionally,
53however, ``devtool add`` sets up an environment that makes it easy for
54you to patch the source and to make changes to the recipe as is often
55necessary when adding a recipe to build a new piece of software to be
56included in a build.
57
58You can find a complete description of the ``devtool add`` command in
59the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
60in the Yocto Project Application Development and the Extensible Software
61Development Kit (eSDK) manual.
62
63Creating the Base Recipe Using ``recipetool create``
64----------------------------------------------------
65
66``recipetool create`` automates creation of a base recipe given a set of
67source code files. As long as you can extract or point to the source
68files, the tool will construct a recipe and automatically configure all
69pre-build information into the recipe. For example, suppose you have an
70application that builds using Autotools. Creating the base recipe using
71``recipetool`` results in a recipe that has the pre-build dependencies,
72license requirements, and checksums configured.
73
74To run the tool, you just need to be in your :term:`Build Directory` and
75have sourced the build environment setup script (i.e.
76:ref:`structure-core-script`). To get help on the tool, use the following
77command::
78
79 $ recipetool -h
80 NOTE: Starting bitbake server...
81 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
82
83 OpenEmbedded recipe tool
84
85 options:
86 -d, --debug Enable debug output
87 -q, --quiet Print only errors
88 --color COLOR Colorize output (where COLOR is auto, always, never)
89 -h, --help show this help message and exit
90
91 subcommands:
92 create Create a new recipe
93 newappend Create a bbappend for the specified target in the specified
94 layer
95 setvar Set a variable within a recipe
96 appendfile Create/update a bbappend to replace a target file
97 appendsrcfiles Create/update a bbappend to add or replace source files
98 appendsrcfile Create/update a bbappend to add or replace a source file
99 Use recipetool <subcommand> --help to get help on a specific command
100
101Running ``recipetool create -o OUTFILE`` creates the base recipe and
102locates it properly in the layer that contains your source files.
103Following are some syntax examples:
104
105 - Use this syntax to generate a recipe based on source. Once generated,
106 the recipe resides in the existing source code layer::
107
108 recipetool create -o OUTFILE source
109
110 - Use this syntax to generate a recipe using code that
111 you extract from source. The extracted code is placed in its own layer
112 defined by :term:`EXTERNALSRC`::
113
114 recipetool create -o OUTFILE -x EXTERNALSRC source
115
116 - Use this syntax to generate a recipe based on source. The options
117 direct ``recipetool`` to generate debugging information. Once generated,
118 the recipe resides in the existing source code layer::
119
120 recipetool create -d -o OUTFILE source
121
122Locating and Using a Similar Recipe
123-----------------------------------
124
125Before writing a recipe from scratch, it is often useful to discover
126whether someone else has already written one that meets (or comes close
127to meeting) your needs. The Yocto Project and OpenEmbedded communities
128maintain many recipes that might be candidates for what you are doing.
129You can find a good central index of these recipes in the
130:oe_layerindex:`OpenEmbedded Layer Index <>`.
131
132Working from an existing recipe or a skeleton recipe is the best way to
133get started. Here are some points on both methods:
134
135- *Locate and modify a recipe that is close to what you want to do:*
136 This method works when you are familiar with the current recipe
137 space. The method does not work so well for those new to the Yocto
138 Project or writing recipes.
139
140 Some risks associated with this method are using a recipe that has
141 areas totally unrelated to what you are trying to accomplish with
142 your recipe, not recognizing areas of the recipe that you might have
143 to add from scratch, and so forth. All these risks stem from
144 unfamiliarity with the existing recipe space.
145
146- *Use and modify the following skeleton recipe:* If for some reason
147 you do not want to use ``recipetool`` and you cannot find an existing
148 recipe that is close to meeting your needs, you can use the following
149 structure to provide the fundamental areas of a new recipe::
150
151 DESCRIPTION = ""
152 HOMEPAGE = ""
153 LICENSE = ""
154 SECTION = ""
155 DEPENDS = ""
156 LIC_FILES_CHKSUM = ""
157
158 SRC_URI = ""
159
160Storing and Naming the Recipe
161=============================
162
163Once you have your base recipe, you should put it in your own layer and
164name it appropriately. Locating it correctly ensures that the
165OpenEmbedded build system can find it when you use BitBake to process
166the recipe.
167
168- *Storing Your Recipe:* The OpenEmbedded build system locates your
169 recipe through the layer's ``conf/layer.conf`` file and the
170 :term:`BBFILES` variable. This
171 variable sets up a path from which the build system can locate
172 recipes. Here is the typical use::
173
174 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
175 ${LAYERDIR}/recipes-*/*/*.bbappend"
176
177 Consequently, you need to be sure you locate your new recipe inside
178 your layer such that it can be found.
179
180 You can find more information on how layers are structured in the
181 ":ref:`dev-manual/layers:understanding and creating layers`" section.
182
183- *Naming Your Recipe:* When you name your recipe, you need to follow
184 this naming convention::
185
186 basename_version.bb
187
188 Use lower-cased characters and do not include the reserved suffixes
189 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
190 them as part of your recipe name unless the string applies). Here are some
191 examples:
192
193 .. code-block:: none
194
195 cups_1.7.0.bb
196 gawk_4.0.2.bb
197 irssi_0.8.16-rc1.bb
198
199Running a Build on the Recipe
200=============================
201
202Creating a new recipe is usually an iterative process that requires
203using BitBake to process the recipe multiple times in order to
204progressively discover and add information to the recipe file.
205
206Assuming you have sourced the build environment setup script (i.e.
207:ref:`structure-core-script`) and you are in the :term:`Build Directory`, use
208BitBake to process your recipe. All you need to provide is the
209``basename`` of the recipe as described in the previous section::
210
211 $ bitbake basename
212
213During the build, the OpenEmbedded build system creates a temporary work
214directory for each recipe
215(``${``\ :term:`WORKDIR`\ ``}``)
216where it keeps extracted source files, log files, intermediate
217compilation and packaging files, and so forth.
218
219The path to the per-recipe temporary work directory depends on the
220context in which it is being built. The quickest way to find this path
221is to have BitBake return it by running the following::
222
223 $ bitbake -e basename | grep ^WORKDIR=
224
225As an example, assume a Source Directory
226top-level folder named ``poky``, a default :term:`Build Directory` at
227``poky/build``, and a ``qemux86-poky-linux`` machine target system.
228Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
229case, the work directory the build system uses to build the package
230would be as follows::
231
232 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
233
234Inside this directory you can find sub-directories such as ``image``,
235``packages-split``, and ``temp``. After the build, you can examine these
236to determine how well the build went.
237
238.. note::
239
240 You can find log files for each task in the recipe's ``temp``
241 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
242 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
243 ``log.do_fetch``, and ``log.do_compile``).
244
245You can find more information about the build process in
246":doc:`/overview-manual/development-environment`"
247chapter of the Yocto Project Overview and Concepts Manual.
248
249Fetching Code
250=============
251
252The first thing your recipe must do is specify how to fetch the source
253files. Fetching is controlled mainly through the
254:term:`SRC_URI` variable. Your recipe
255must have a :term:`SRC_URI` variable that points to where the source is
256located. For a graphical representation of source locations, see the
257":ref:`overview-manual/concepts:sources`" section in
258the Yocto Project Overview and Concepts Manual.
259
260The :ref:`ref-tasks-fetch` task uses the prefix of each entry in the
261:term:`SRC_URI` variable value to determine which
262:ref:`fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`
263to use to get your source files. It is the :term:`SRC_URI` variable that triggers
264the fetcher. The :ref:`ref-tasks-patch` task uses the variable after source is
265fetched to apply patches. The OpenEmbedded build system uses
266:term:`FILESOVERRIDES` for scanning directory locations for local files in
267:term:`SRC_URI`.
268
269The :term:`SRC_URI` variable in your recipe must define each unique location
270for your source files. It is good practice to not hard-code version
271numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
272values, use ``${``\ :term:`PV`\ ``}``,
273which causes the fetch process to use the version specified in the
274recipe filename. Specifying the version in this manner means that
275upgrading the recipe to a future version is as simple as renaming the
276recipe to match the new version.
277
278Here is a simple example from the
279``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
280comes from a single tarball. Notice the use of the
281:term:`PV` variable::
282
283 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
284
285Files mentioned in :term:`SRC_URI` whose names end in a typical archive
286extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
287forth), are automatically extracted during the
288:ref:`ref-tasks-unpack` task. For
289another example that specifies these types of files, see the
290":ref:`dev-manual/new-recipe:building an autotooled package`" section.
291
292Another way of specifying source is from an SCM. For Git repositories,
293you must specify :term:`SRCREV` and you should specify :term:`PV` to include
294the revision with :term:`SRCPV`. Here is an example from the recipe
295``meta/recipes-core/musl/gcompat_git.bb``::
296
297 SRC_URI = "git://git.adelielinux.org/adelie/gcompat.git;protocol=https;branch=current"
298
299 PV = "1.0.0+1.1+git${SRCPV}"
300 SRCREV = "af5a49e489fdc04b9cf02547650d7aeaccd43793"
301
302If your :term:`SRC_URI` statement includes URLs pointing to individual files
303fetched from a remote server other than a version control system,
304BitBake attempts to verify the files against checksums defined in your
305recipe to ensure they have not been tampered with or otherwise modified
306since the recipe was written. Two checksums are used:
307``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
308
309If your :term:`SRC_URI` variable points to more than a single URL (excluding
310SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
311each URL. For these cases, you provide a name for each URL as part of
312the :term:`SRC_URI` and then reference that name in the subsequent checksum
313statements. Here is an example combining lines from the files
314``git.inc`` and ``git_2.24.1.bb``::
315
316 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
317 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
318
319 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
320 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
321 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
322 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
323
324Proper values for ``md5`` and ``sha256`` checksums might be available
325with other signatures on the download page for the upstream source (e.g.
326``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
327OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
328you should verify all the signatures you find by hand.
329
330If no :term:`SRC_URI` checksums are specified when you attempt to build the
331recipe, or you provide an incorrect checksum, the build will produce an
332error for each missing or incorrect checksum. As part of the error
333message, the build system provides the checksum string corresponding to
334the fetched file. Once you have the correct checksums, you can copy and
335paste them into your recipe and then run the build again to continue.
336
337.. note::
338
339 As mentioned, if the upstream source provides signatures for
340 verifying the downloaded source code, you should verify those
341 manually before setting the checksum values in the recipe and
342 continuing with the build.
343
344This final example is a bit more complicated and is from the
345``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
346example's :term:`SRC_URI` statement identifies multiple files as the source
347files for the recipe: a tarball, a patch file, a desktop file, and an icon::
348
349 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
350 file://xwc.patch \
351 file://rxvt.desktop \
352 file://rxvt.png"
353
354When you specify local files using the ``file://`` URI protocol, the
355build system fetches files from the local machine. The path is relative
356to the :term:`FILESPATH` variable
357and searches specific directories in a certain order:
358``${``\ :term:`BP`\ ``}``,
359``${``\ :term:`BPN`\ ``}``, and
360``files``. The directories are assumed to be subdirectories of the
361directory in which the recipe or append file resides. For another
362example that specifies these types of files, see the
363"`building a single .c file package`_" section.
364
365The previous example also specifies a patch file. Patch files are files
366whose names usually end in ``.patch`` or ``.diff`` but can end with
367compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
368The build system automatically applies patches as described in the
369":ref:`dev-manual/new-recipe:patching code`" section.
370
371Fetching Code Through Firewalls
372-------------------------------
373
374Some users are behind firewalls and need to fetch code through a proxy.
375See the ":doc:`/ref-manual/faq`" chapter for advice.
376
377Limiting the Number of Parallel Connections
378-------------------------------------------
379
380Some users are behind firewalls or use servers where the number of parallel
381connections is limited. In such cases, you can limit the number of fetch
382tasks being run in parallel by adding the following to your ``local.conf``
383file::
384
385 do_fetch[number_threads] = "4"
386
387Unpacking Code
388==============
389
390During the build, the
391:ref:`ref-tasks-unpack` task unpacks
392the source with ``${``\ :term:`S`\ ``}``
393pointing to where it is unpacked.
394
395If you are fetching your source files from an upstream source archived
396tarball and the tarball's internal structure matches the common
397convention of a top-level subdirectory named
398``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
399then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
400fetch source from an archive that does not use this convention, or from
401an SCM like Git or Subversion, your recipe needs to define :term:`S`.
402
403If processing your recipe using BitBake successfully unpacks the source
404files, you need to be sure that the directory pointed to by ``${S}``
405matches the structure of the source.
406
407Patching Code
408=============
409
410Sometimes it is necessary to patch code after it has been fetched. Any
411files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
412``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
413treated as patches. The
414:ref:`ref-tasks-patch` task
415automatically applies these patches.
416
417The build system should be able to apply patches with the "-p1" option
418(i.e. one directory level in the path will be stripped off). If your
419patch needs to have more directory levels stripped off, specify the
420number of levels using the "striplevel" option in the :term:`SRC_URI` entry
421for the patch. Alternatively, if your patch needs to be applied in a
422specific subdirectory that is not specified in the patch file, use the
423"patchdir" option in the entry.
424
425As with all local files referenced in
426:term:`SRC_URI` using ``file://``,
427you should place patch files in a directory next to the recipe either
428named the same as the base name of the recipe
429(:term:`BP` and
430:term:`BPN`) or "files".
431
432Licensing
433=========
434
435Your recipe needs to define variables related to the license
436under whith the software is distributed. See the
437:ref:`contributor-guide/recipe-style-guide:recipe license fields`
438section in the Contributor Guide for details.
439
440Dependencies
441============
442
443Most software packages have a short list of other packages that they
444require, which are called dependencies. These dependencies fall into two
445main categories: build-time dependencies, which are required when the
446software is built; and runtime dependencies, which are required to be
447installed on the target in order for the software to run.
448
449Within a recipe, you specify build-time dependencies using the
450:term:`DEPENDS` variable. Although there are nuances,
451items specified in :term:`DEPENDS` should be names of other
452recipes. It is important that you specify all build-time dependencies
453explicitly.
454
455Another consideration is that configure scripts might automatically
456check for optional dependencies and enable corresponding functionality
457if those dependencies are found. If you wish to make a recipe that is
458more generally useful (e.g. publish the recipe in a layer for others to
459use), instead of hard-disabling the functionality, you can use the
460:term:`PACKAGECONFIG` variable to allow functionality and the
461corresponding dependencies to be enabled and disabled easily by other
462users of the recipe.
463
464Similar to build-time dependencies, you specify runtime dependencies
465through a variable -
466:term:`RDEPENDS`, which is
467package-specific. All variables that are package-specific need to have
468the name of the package added to the end as an override. Since the main
469package for a recipe has the same name as the recipe, and the recipe's
470name can be found through the
471``${``\ :term:`PN`\ ``}`` variable, then
472you specify the dependencies for the main package by setting
473``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you
474would set ``RDEPENDS:${PN}-tools``, and so forth.
475
476Some runtime dependencies will be set automatically at packaging time.
477These dependencies include any shared library dependencies (i.e. if a
478package "example" contains "libexample" and another package "mypackage"
479contains a binary that links to "libexample" then the OpenEmbedded build
480system will automatically add a runtime dependency to "mypackage" on
481"example"). See the
482":ref:`overview-manual/concepts:automatically added runtime dependencies`"
483section in the Yocto Project Overview and Concepts Manual for further
484details.
485
486Configuring the Recipe
487======================
488
489Most software provides some means of setting build-time configuration
490options before compilation. Typically, setting these options is
491accomplished by running a configure script with options, or by modifying
492a build configuration file.
493
494.. note::
495
496 As of Yocto Project Release 1.7, some of the core recipes that
497 package binary configuration scripts now disable the scripts due to
498 the scripts previously requiring error-prone path substitution. The
499 OpenEmbedded build system uses ``pkg-config`` now, which is much more
500 robust. You can find a list of the ``*-config`` scripts that are disabled
501 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
502 in the Yocto Project Reference Manual.
503
504A major part of build-time configuration is about checking for
505build-time dependencies and possibly enabling optional functionality as
506a result. You need to specify any build-time dependencies for the
507software you are building in your recipe's
508:term:`DEPENDS` value, in terms of
509other recipes that satisfy those dependencies. You can often find
510build-time or runtime dependencies described in the software's
511documentation.
512
513The following list provides configuration items of note based on how
514your software is built:
515
516- *Autotools:* If your source files have a ``configure.ac`` file, then
517 your software is built using Autotools. If this is the case, you just
518 need to modify the configuration.
519
520 When using Autotools, your recipe needs to inherit the
521 :ref:`ref-classes-autotools` class and it does not have to
522 contain a :ref:`ref-tasks-configure` task. However, you might still want to
523 make some adjustments. For example, you can set :term:`EXTRA_OECONF` or
524 :term:`PACKAGECONFIG_CONFARGS` to pass any needed configure options that
525 are specific to the recipe.
526
527- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
528 your software is built using CMake. If this is the case, you just
529 need to modify the configuration.
530
531 When you use CMake, your recipe needs to inherit the
532 :ref:`ref-classes-cmake` class and it does not have to contain a
533 :ref:`ref-tasks-configure` task. You can make some adjustments by setting
534 :term:`EXTRA_OECMAKE` to pass any needed configure options that are
535 specific to the recipe.
536
537 .. note::
538
539 If you need to install one or more custom CMake toolchain files
540 that are supplied by the application you are building, install the
541 files to ``${D}${datadir}/cmake/Modules`` during :ref:`ref-tasks-install`.
542
543- *Other:* If your source files do not have a ``configure.ac`` or
544 ``CMakeLists.txt`` file, then your software is built using some
545 method other than Autotools or CMake. If this is the case, you
546 normally need to provide a
547 :ref:`ref-tasks-configure` task
548 in your recipe unless, of course, there is nothing to configure.
549
550 Even if your software is not being built by Autotools or CMake, you
551 still might not need to deal with any configuration issues. You need
552 to determine if configuration is even a required step. You might need
553 to modify a Makefile or some configuration file used for the build to
554 specify necessary build options. Or, perhaps you might need to run a
555 provided, custom configure script with the appropriate options.
556
557 For the case involving a custom configure script, you would run
558 ``./configure --help`` and look for the options you need to set.
559
560Once configuration succeeds, it is always good practice to look at the
561``log.do_configure`` file to ensure that the appropriate options have
562been enabled and no additional build-time dependencies need to be added
563to :term:`DEPENDS`. For example, if the configure script reports that it
564found something not mentioned in :term:`DEPENDS`, or that it did not find
565something that it needed for some desired optional functionality, then
566you would need to add those to :term:`DEPENDS`. Looking at the log might
567also reveal items being checked for, enabled, or both that you do not
568want, or items not being found that are in :term:`DEPENDS`, in which case
569you would need to look at passing extra options to the configure script
570as needed. For reference information on configure options specific to
571the software you are building, you can consult the output of the
572``./configure --help`` command within ``${S}`` or consult the software's
573upstream documentation.
574
575Using Headers to Interface with Devices
576=======================================
577
578If your recipe builds an application that needs to communicate with some
579device or needs an API into a custom kernel, you will need to provide
580appropriate header files. Under no circumstances should you ever modify
581the existing
582``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
583These headers are used to build ``libc`` and must not be compromised
584with custom or machine-specific header information. If you customize
585``libc`` through modified headers all other applications that use
586``libc`` thus become affected.
587
588.. note::
589
590 Never copy and customize the ``libc`` header file (i.e.
591 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
592
593The correct way to interface to a device or custom kernel is to use a
594separate package that provides the additional headers for the driver or
595other unique interfaces. When doing so, your application also becomes
596responsible for creating a dependency on that specific provider.
597
598Consider the following:
599
600- Never modify ``linux-libc-headers.inc``. Consider that file to be
601 part of the ``libc`` system, and not something you use to access the
602 kernel directly. You should access ``libc`` through specific ``libc``
603 calls.
604
605- Applications that must talk directly to devices should either provide
606 necessary headers themselves, or establish a dependency on a special
607 headers package that is specific to that driver.
608
609For example, suppose you want to modify an existing header that adds I/O
610control or network support. If the modifications are used by a small
611number programs, providing a unique version of a header is easy and has
612little impact. When doing so, bear in mind the guidelines in the
613previous list.
614
615.. note::
616
617 If for some reason your changes need to modify the behavior of the ``libc``,
618 and subsequently all other applications on the system, use a ``.bbappend``
619 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
620 make the changes machine specific.
621
622Consider a case where your kernel is older and you need an older
623``libc`` ABI. The headers installed by your recipe should still be a
624standard mainline kernel, not your own custom one.
625
626When you use custom kernel headers you need to get them from
627:term:`STAGING_KERNEL_DIR`,
628which is the directory with kernel headers that are required to build
629out-of-tree modules. Your recipe will also need the following::
630
631 do_configure[depends] += "virtual/kernel:do_shared_workdir"
632
633Compilation
634===========
635
636During a build, the :ref:`ref-tasks-compile` task happens after source is fetched,
637unpacked, and configured. If the recipe passes through :ref:`ref-tasks-compile`
638successfully, nothing needs to be done.
639
640However, if the compile step fails, you need to diagnose the failure.
641Here are some common issues that cause failures.
642
643.. note::
644
645 For cases where improper paths are detected for configuration files
646 or for when libraries/headers cannot be found, be sure you are using
647 the more robust ``pkg-config``. See the note in section
648 ":ref:`dev-manual/new-recipe:Configuring the Recipe`" for additional information.
649
650- *Parallel build failures:* These failures manifest themselves as
651 intermittent errors, or errors reporting that a file or directory
652 that should be created by some other part of the build process could
653 not be found. This type of failure can occur even if, upon
654 inspection, the file or directory does exist after the build has
655 failed, because that part of the build process happened in the wrong
656 order.
657
658 To fix the problem, you need to either satisfy the missing dependency
659 in the Makefile or whatever script produced the Makefile, or (as a
660 workaround) set :term:`PARALLEL_MAKE` to an empty string::
661
662 PARALLEL_MAKE = ""
663
664 For information on parallel Makefile issues, see the
665 ":ref:`dev-manual/debugging:debugging parallel make races`" section.
666
667- *Improper host path usage:* This failure applies to recipes building
668 for the target or ":ref:`ref-classes-nativesdk`" only. The
669 failure occurs when the compilation process uses improper headers,
670 libraries, or other files from the host system when cross-compiling for
671 the target.
672
673 To fix the problem, examine the ``log.do_compile`` file to identify
674 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
675 so forth) and then either add configure options, apply a patch, or do
676 both.
677
678- *Failure to find required libraries/headers:* If a build-time
679 dependency is missing because it has not been declared in
680 :term:`DEPENDS`, or because the
681 dependency exists but the path used by the build process to find the
682 file is incorrect and the configure step did not detect it, the
683 compilation process could fail. For either of these failures, the
684 compilation process notes that files could not be found. In these
685 cases, you need to go back and add additional options to the
686 configure script as well as possibly add additional build-time
687 dependencies to :term:`DEPENDS`.
688
689 Occasionally, it is necessary to apply a patch to the source to
690 ensure the correct paths are used. If you need to specify paths to
691 find files staged into the sysroot from other recipes, use the
692 variables that the OpenEmbedded build system provides (e.g.
693 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
694 forth).
695
696Installing
697==========
698
699During :ref:`ref-tasks-install`, the task copies the built files along with their
700hierarchy to locations that would mirror their locations on the target
701device. The installation process copies files from the
702``${``\ :term:`S`\ ``}``,
703``${``\ :term:`B`\ ``}``, and
704``${``\ :term:`WORKDIR`\ ``}``
705directories to the ``${``\ :term:`D`\ ``}``
706directory to create the structure as it should appear on the target
707system.
708
709How your software is built affects what you must do to be sure your
710software is installed correctly. The following list describes what you
711must do for installation depending on the type of build system used by
712the software being built:
713
714- *Autotools and CMake:* If the software your recipe is building uses
715 Autotools or CMake, the OpenEmbedded build system understands how to
716 install the software. Consequently, you do not have to have a
717 :ref:`ref-tasks-install` task as part of your recipe. You just need to make
718 sure the install portion of the build completes with no issues.
719 However, if you wish to install additional files not already being
720 installed by ``make install``, you should do this using a
721 ``do_install:append`` function using the install command as described
722 in the "Manual" bulleted item later in this list.
723
724- *Other (using* ``make install``\ *)*: You need to define a :ref:`ref-tasks-install`
725 function in your recipe. The function should call
726 ``oe_runmake install`` and will likely need to pass in the
727 destination directory as well. How you pass that path is dependent on
728 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
729 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
730
731 For an example recipe using ``make install``, see the
732 ":ref:`dev-manual/new-recipe:building a makefile-based package`" section.
733
734- *Manual:* You need to define a :ref:`ref-tasks-install` function in your
735 recipe. The function must first use ``install -d`` to create the
736 directories under
737 ``${``\ :term:`D`\ ``}``. Once the
738 directories exist, your function can use ``install`` to manually
739 install the built software into the directories.
740
741 You can find more information on ``install`` at
742 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
743
744For the scenarios that do not use Autotools or CMake, you need to track
745the installation and diagnose and fix any issues until everything
746installs correctly. You need to look in the default location of
747``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
748installed correctly.
749
750.. note::
751
752 - During the installation process, you might need to modify some of
753 the installed files to suit the target layout. For example, you
754 might need to replace hard-coded paths in an initscript with
755 values of variables provided by the build system, such as
756 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
757 modifications during :ref:`ref-tasks-install`, be sure to modify the
758 destination file after copying rather than before copying.
759 Modifying after copying ensures that the build system can
760 re-execute :ref:`ref-tasks-install` if needed.
761
762 - ``oe_runmake install``, which can be run directly or can be run
763 indirectly by the :ref:`ref-classes-autotools` and
764 :ref:`ref-classes-cmake` classes, runs ``make install`` in parallel.
765 Sometimes, a Makefile can have missing dependencies between targets that
766 can result in race conditions. If you experience intermittent failures
767 during :ref:`ref-tasks-install`, you might be able to work around them by
768 disabling parallel Makefile installs by adding the following to the
769 recipe::
770
771 PARALLEL_MAKEINST = ""
772
773 See :term:`PARALLEL_MAKEINST` for additional information.
774
775 - If you need to install one or more custom CMake toolchain files
776 that are supplied by the application you are building, install the
777 files to ``${D}${datadir}/cmake/Modules`` during
778 :ref:`ref-tasks-install`.
779
780Enabling System Services
781========================
782
783If you want to install a service, which is a process that usually starts
784on boot and runs in the background, then you must include some
785additional definitions in your recipe.
786
787If you are adding services and the service initialization script or the
788service file itself is not installed, you must provide for that
789installation in your recipe using a ``do_install:append`` function. If
790your recipe already has a :ref:`ref-tasks-install` function, update the function
791near its end rather than adding an additional ``do_install:append``
792function.
793
794When you create the installation for your services, you need to
795accomplish what is normally done by ``make install``. In other words,
796make sure your installation arranges the output similar to how it is
797arranged on the target system.
798
799The OpenEmbedded build system provides support for starting services two
800different ways:
801
802- *SysVinit:* SysVinit is a system and service manager that manages the
803 init system used to control the very basic functions of your system.
804 The init program is the first program started by the Linux kernel
805 when the system boots. Init then controls the startup, running and
806 shutdown of all other programs.
807
808 To enable a service using SysVinit, your recipe needs to inherit the
809 :ref:`ref-classes-update-rc.d` class. The class helps
810 facilitate safely installing the package on the target.
811
812 You will need to set the
813 :term:`INITSCRIPT_PACKAGES`,
814 :term:`INITSCRIPT_NAME`,
815 and
816 :term:`INITSCRIPT_PARAMS`
817 variables within your recipe.
818
819- *systemd:* System Management Daemon (systemd) was designed to replace
820 SysVinit and to provide enhanced management of services. For more
821 information on systemd, see the systemd homepage at
822 https://freedesktop.org/wiki/Software/systemd/.
823
824 To enable a service using systemd, your recipe needs to inherit the
825 :ref:`ref-classes-systemd` class. See the ``systemd.bbclass`` file
826 located in your :term:`Source Directory` section for more information.
827
828Packaging
829=========
830
831Successful packaging is a combination of automated processes performed
832by the OpenEmbedded build system and some specific steps you need to
833take. The following list describes the process:
834
835- *Splitting Files*: The :ref:`ref-tasks-package` task splits the files produced
836 by the recipe into logical components. Even software that produces a
837 single binary might still have debug symbols, documentation, and
838 other logical components that should be split out. The :ref:`ref-tasks-package`
839 task ensures that files are split up and packaged correctly.
840
841- *Running QA Checks*: The :ref:`ref-classes-insane` class adds a
842 step to the package generation process so that output quality
843 assurance checks are generated by the OpenEmbedded build system. This
844 step performs a range of checks to be sure the build's output is free
845 of common problems that show up during runtime. For information on
846 these checks, see the :ref:`ref-classes-insane` class and
847 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
848 chapter in the Yocto Project Reference Manual.
849
850- *Hand-Checking Your Packages*: After you build your software, you
851 need to be sure your packages are correct. Examine the
852 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
853 directory and make sure files are where you expect them to be. If you
854 discover problems, you can set
855 :term:`PACKAGES`,
856 :term:`FILES`,
857 ``do_install(:append)``, and so forth as needed.
858
859- *Splitting an Application into Multiple Packages*: If you need to
860 split an application into several packages, see the
861 ":ref:`dev-manual/new-recipe:splitting an application into multiple packages`"
862 section for an example.
863
864- *Installing a Post-Installation Script*: For an example showing how
865 to install a post-installation script, see the
866 ":ref:`dev-manual/new-recipe:post-installation scripts`" section.
867
868- *Marking Package Architecture*: Depending on what your recipe is
869 building and how it is configured, it might be important to mark the
870 packages produced as being specific to a particular machine, or to
871 mark them as not being specific to a particular machine or
872 architecture at all.
873
874 By default, packages apply to any machine with the same architecture
875 as the target machine. When a recipe produces packages that are
876 machine-specific (e.g. the
877 :term:`MACHINE` value is passed
878 into the configure script or a patch is applied only for a particular
879 machine), you should mark them as such by adding the following to the
880 recipe::
881
882 PACKAGE_ARCH = "${MACHINE_ARCH}"
883
884 On the other hand, if the recipe produces packages that do not
885 contain anything specific to the target machine or architecture at
886 all (e.g. recipes that simply package script files or configuration
887 files), you should use the :ref:`ref-classes-allarch` class to
888 do this for you by adding this to your recipe::
889
890 inherit allarch
891
892 Ensuring that the package architecture is correct is not critical
893 while you are doing the first few builds of your recipe. However, it
894 is important in order to ensure that your recipe rebuilds (or does
895 not rebuild) appropriately in response to changes in configuration,
896 and to ensure that you get the appropriate packages installed on the
897 target machine, particularly if you run separate builds for more than
898 one target machine.
899
900Sharing Files Between Recipes
901=============================
902
903Recipes often need to use files provided by other recipes on the build
904host. For example, an application linking to a common library needs
905access to the library itself and its associated headers. The way this
906access is accomplished is by populating a sysroot with files. Each
907recipe has two sysroots in its work directory, one for target files
908(``recipe-sysroot``) and one for files that are native to the build host
909(``recipe-sysroot-native``).
910
911.. note::
912
913 You could find the term "staging" used within the Yocto project
914 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
915 variable).
916
917Recipes should never populate the sysroot directly (i.e. write files
918into sysroot). Instead, files should be installed into standard
919locations during the
920:ref:`ref-tasks-install` task within
921the ``${``\ :term:`D`\ ``}`` directory. The
922reason for this limitation is that almost all files that populate the
923sysroot are cataloged in manifests in order to ensure the files can be
924removed later when a recipe is either modified or removed. Thus, the
925sysroot is able to remain free from stale files.
926
927A subset of the files installed by the :ref:`ref-tasks-install` task are
928used by the :ref:`ref-tasks-populate_sysroot` task as defined by the
929:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
930is possible to modify the list of directories that populate the sysroot.
931The following example shows how you could add the ``/opt`` directory to
932the list of directories within a recipe::
933
934 SYSROOT_DIRS += "/opt"
935
936.. note::
937
938 The `/sysroot-only` is to be used by recipes that generate artifacts
939 that are not included in the target filesystem, allowing them to share
940 these artifacts without needing to use the :term:`DEPLOY_DIR`.
941
942For a more complete description of the :ref:`ref-tasks-populate_sysroot`
943task and its associated functions, see the
944:ref:`staging <ref-classes-staging>` class.
945
946Using Virtual Providers
947=======================
948
949Prior to a build, if you know that several different recipes provide the
950same functionality, you can use a virtual provider (i.e. ``virtual/*``)
951as a placeholder for the actual provider. The actual provider is
952determined at build-time.
953
954A common scenario where a virtual provider is used would be for the kernel
955recipe. Suppose you have three kernel recipes whose :term:`PN` values map to
956``kernel-big``, ``kernel-mid``, and ``kernel-small``. Furthermore, each of
957these recipes in some way uses a :term:`PROVIDES` statement that essentially
958identifies itself as being able to provide ``virtual/kernel``. Here is one way
959through the :ref:`ref-classes-kernel` class::
960
961 PROVIDES += "virtual/kernel"
962
963Any recipe that inherits the :ref:`ref-classes-kernel` class is
964going to utilize a :term:`PROVIDES` statement that identifies that recipe as
965being able to provide the ``virtual/kernel`` item.
966
967Now comes the time to actually build an image and you need a kernel
968recipe, but which one? You can configure your build to call out the
969kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
970an example, consider the :yocto_git:`x86-base.inc
971</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a
972machine (i.e. :term:`MACHINE`) configuration file. This include file is the
973reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
974relevant lines from the include file::
975
976 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
977 PREFERRED_VERSION_linux-yocto ??= "4.15%"
978
979When you use a virtual provider, you do not have to "hard code" a recipe
980name as a build dependency. You can use the
981:term:`DEPENDS` variable to state the
982build is dependent on ``virtual/kernel`` for example::
983
984 DEPENDS = "virtual/kernel"
985
986During the build, the OpenEmbedded build system picks
987the correct recipe needed for the ``virtual/kernel`` dependency based on
988the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
989mentioned at the beginning of this section, configure your build as
990follows::
991
992 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
993
994.. note::
995
996 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
997 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
998 recipes from building is usually the desired behavior since this mechanism's
999 purpose is to select between mutually exclusive alternative providers.
1000
1001The following lists specific examples of virtual providers:
1002
1003- ``virtual/kernel``: Provides the name of the kernel recipe to use
1004 when building a kernel image.
1005
1006- ``virtual/bootloader``: Provides the name of the bootloader to use
1007 when building an image.
1008
1009- ``virtual/libgbm``: Provides ``gbm.pc``.
1010
1011- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
1012
1013- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
1014
1015- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
1016
1017- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
1018
1019.. note::
1020
1021 Virtual providers only apply to build time dependencies specified with
1022 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
1023 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
1024
1025Properly Versioning Pre-Release Recipes
1026=======================================
1027
1028Sometimes the name of a recipe can lead to versioning problems when the
1029recipe is upgraded to a final release. For example, consider the
1030``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
1031the ":ref:`dev-manual/new-recipe:storing and naming the recipe`" section.
1032This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
1033released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
1034change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
1035build system and package managers, so the resulting packages will not
1036correctly trigger an upgrade.
1037
1038In order to ensure the versions compare properly, the recommended
1039convention is to set :term:`PV` within the
1040recipe to "previous_version+current_version". You can use an additional
1041variable so that you can use the current version elsewhere. Here is an
1042example::
1043
1044 REALPV = "0.8.16-rc1"
1045 PV = "0.8.15+${REALPV}"
1046
1047Post-Installation Scripts
1048=========================
1049
1050Post-installation scripts run immediately after installing a package on
1051the target or during image creation when a package is included in an
1052image. To add a post-installation script to a package, add a
1053``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file
1054(``.bb``) and replace `PACKAGENAME` with the name of the package you want
1055to attach to the ``postinst`` script. To apply the post-installation
1056script to the main package for the recipe, which is usually what is
1057required, specify
1058``${``\ :term:`PN`\ ``}`` in place of
1059PACKAGENAME.
1060
1061A post-installation function has the following structure::
1062
1063 pkg_postinst:PACKAGENAME() {
1064 # Commands to carry out
1065 }
1066
1067The script defined in the post-installation function is called when the
1068root filesystem is created. If the script succeeds, the package is
1069marked as installed.
1070
1071.. note::
1072
1073 Any RPM post-installation script that runs on the target should
1074 return a 0 exit code. RPM does not allow non-zero exit codes for
1075 these scripts, and the RPM package manager will cause the package to
1076 fail installation on the target.
1077
1078Sometimes it is necessary for the execution of a post-installation
1079script to be delayed until the first boot. For example, the script might
1080need to be executed on the device itself. To delay script execution
1081until boot time, you must explicitly mark post installs to defer to the
1082target. You can use ``pkg_postinst_ontarget()`` or call
1083``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
1084failure of a ``pkg_postinst()`` script (including exit 1) triggers an
1085error during the
1086:ref:`ref-tasks-rootfs` task.
1087
1088If you have recipes that use ``pkg_postinst`` function and they require
1089the use of non-standard native tools that have dependencies during
1090root filesystem construction, you need to use the
1091:term:`PACKAGE_WRITE_DEPS`
1092variable in your recipe to list these tools. If you do not use this
1093variable, the tools might be missing and execution of the
1094post-installation script is deferred until first boot. Deferring the
1095script to the first boot is undesirable and impossible for read-only
1096root filesystems.
1097
1098.. note::
1099
1100 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
1101 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
1102 respectively. These scrips work in exactly the same way as does
1103 ``pkg_postinst`` with the exception that they run at different times. Also,
1104 because of when they run, they are not applicable to being run at image
1105 creation time like ``pkg_postinst``.
1106
1107Testing
1108=======
1109
1110The final step for completing your recipe is to be sure that the
1111software you built runs correctly. To accomplish runtime testing, add
1112the build's output packages to your image and test them on the target.
1113
1114For information on how to customize your image by adding specific
1115packages, see ":ref:`dev-manual/customizing-images:customizing images`" section.
1116
1117Examples
1118========
1119
1120To help summarize how to write a recipe, this section provides some
1121recipe examples given various scenarios:
1122
1123- `Building a single .c file package`_
1124
1125- `Building a Makefile-based package`_
1126
1127- `Building an Autotooled package`_
1128
1129- `Building a Meson package`_
1130
1131- `Splitting an application into multiple packages`_
1132
1133- `Packaging externally produced binaries`_
1134
1135Building a Single .c File Package
1136---------------------------------
1137
1138Building an application from a single file that is stored locally (e.g. under
1139``files``) requires a recipe that has the file listed in the :term:`SRC_URI`
1140variable. Additionally, you need to manually write the :ref:`ref-tasks-compile`
1141and :ref:`ref-tasks-install` tasks. The :term:`S` variable defines the
1142directory containing the source code, which is set to :term:`WORKDIR` in this
1143case --- the directory BitBake uses for the build::
1144
1145 SUMMARY = "Simple helloworld application"
1146 SECTION = "examples"
1147 LICENSE = "MIT"
1148 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
1149
1150 SRC_URI = "file://helloworld.c"
1151
1152 S = "${WORKDIR}"
1153
1154 do_compile() {
1155 ${CC} ${LDFLAGS} helloworld.c -o helloworld
1156 }
1157
1158 do_install() {
1159 install -d ${D}${bindir}
1160 install -m 0755 helloworld ${D}${bindir}
1161 }
1162
1163By default, the ``helloworld``, ``helloworld-dbg``, and ``helloworld-dev`` packages
1164are built. For information on how to customize the packaging process, see the
1165":ref:`dev-manual/new-recipe:splitting an application into multiple packages`"
1166section.
1167
1168Building a Makefile-Based Package
1169---------------------------------
1170
1171Applications built with GNU ``make`` require a recipe that has the source archive
1172listed in :term:`SRC_URI`. You do not need to add a :ref:`ref-tasks-compile`
1173step since by default BitBake starts the ``make`` command to compile the
1174application. If you need additional ``make`` options, you should store them in
1175the :term:`EXTRA_OEMAKE` or :term:`PACKAGECONFIG_CONFARGS` variables. BitBake
1176passes these options into the GNU ``make`` invocation. Note that a
1177:ref:`ref-tasks-install` task is still required. Otherwise, BitBake runs an
1178empty :ref:`ref-tasks-install` task by default.
1179
1180Some applications might require extra parameters to be passed to the
1181compiler. For example, the application might need an additional header
1182path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
1183following example shows this::
1184
1185 CFLAGS:prepend = "-I ${S}/include "
1186
1187In the following example, ``lz4`` is a makefile-based package::
1188
1189 SUMMARY = "Extremely Fast Compression algorithm"
1190 DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems."
1191 HOMEPAGE = "https://github.com/lz4/lz4"
1192
1193 LICENSE = "BSD-2-Clause | GPL-2.0-only"
1194 LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc \
1195 file://programs/COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
1196 file://LICENSE;md5=d57c0d21cb917fb4e0af2454aa48b956 \
1197 "
1198
1199 PE = "1"
1200
1201 SRCREV = "d44371841a2f1728a3f36839fd4b7e872d0927d3"
1202
1203 SRC_URI = "git://github.com/lz4/lz4.git;branch=release;protocol=https \
1204 file://CVE-2021-3520.patch \
1205 "
1206 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>.*)"
1207
1208 S = "${WORKDIR}/git"
1209
1210 # Fixed in r118, which is larger than the current version.
1211 CVE_CHECK_IGNORE += "CVE-2014-4715"
1212
1213 EXTRA_OEMAKE = "PREFIX=${prefix} CC='${CC}' CFLAGS='${CFLAGS}' DESTDIR=${D} LIBDIR=${libdir} INCLUDEDIR=${includedir} BUILD_STATIC=no"
1214
1215 do_install() {
1216 oe_runmake install
1217 }
1218
1219 BBCLASSEXTEND = "native nativesdk"
1220
1221Building an Autotooled Package
1222------------------------------
1223
1224Applications built with the Autotools such as ``autoconf`` and ``automake``
1225require a recipe that has a source archive listed in :term:`SRC_URI` and also
1226inherit the :ref:`ref-classes-autotools` class, which contains the definitions
1227of all the steps needed to build an Autotool-based application. The result of
1228the build is automatically packaged. And, if the application uses NLS for
1229localization, packages with local information are generated (one package per
1230language). Following is one example: (``hello_2.3.bb``)::
1231
1232 SUMMARY = "GNU Helloworld application"
1233 SECTION = "examples"
1234 LICENSE = "GPL-2.0-or-later"
1235 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
1236
1237 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
1238
1239 inherit autotools gettext
1240
1241The variable :term:`LIC_FILES_CHKSUM` is used to track source license changes
1242as described in the ":ref:`dev-manual/licenses:tracking license changes`"
1243section in the Yocto Project Overview and Concepts Manual. You can quickly
1244create Autotool-based recipes in a manner similar to the previous example.
1245
1246.. _ref-building-meson-package:
1247
1248Building a Meson Package
1249------------------------
1250
1251Applications built with the `Meson build system <https://mesonbuild.com/>`__
1252just need a recipe that has sources described in :term:`SRC_URI` and inherits
1253the :ref:`ref-classes-meson` class.
1254
1255The :oe_git:`ipcalc recipe </meta-openembedded/tree/meta-networking/recipes-support/ipcalc>`
1256is a simple example of an application without dependencies::
1257
1258 SUMMARY = "Tool to assist in network address calculations for IPv4 and IPv6."
1259 HOMEPAGE = "https://gitlab.com/ipcalc/ipcalc"
1260
1261 SECTION = "net"
1262
1263 LICENSE = "GPL-2.0-only"
1264 LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
1265
1266 SRC_URI = "git://gitlab.com/ipcalc/ipcalc.git;protocol=https;branch=master"
1267 SRCREV = "4c4261a47f355946ee74013d4f5d0494487cc2d6"
1268
1269 S = "${WORKDIR}/git"
1270
1271 inherit meson
1272
1273Applications with dependencies are likely to inherit the
1274:ref:`ref-classes-pkgconfig` class, as ``pkg-config`` is the default method
1275used by Meson to find dependencies and compile applications against them.
1276
1277Splitting an Application into Multiple Packages
1278-----------------------------------------------
1279
1280You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
1281application into multiple packages.
1282
1283Following is an example that uses the ``libxpm`` recipe. By default,
1284this recipe generates a single package that contains the library along
1285with a few binaries. You can modify the recipe to split the binaries
1286into separate packages::
1287
1288 require xorg-lib-common.inc
1289
1290 SUMMARY = "Xpm: X Pixmap extension library"
1291 LICENSE = "MIT"
1292 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
1293 DEPENDS += "libxext libsm libxt"
1294 PE = "1"
1295
1296 XORG_PN = "libXpm"
1297
1298 PACKAGES =+ "sxpm cxpm"
1299 FILES:cxpm = "${bindir}/cxpm"
1300 FILES:sxpm = "${bindir}/sxpm"
1301
1302In the previous example, we want to ship the ``sxpm`` and ``cxpm``
1303binaries in separate packages. Since ``bindir`` would be packaged into
1304the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
1305so additional package names are added to the start of list. This results
1306in the extra ``FILES:*`` variables then containing information that
1307define which files and directories go into which packages. Files
1308included by earlier packages are skipped by latter packages. Thus, the
1309main :term:`PN` package does not include the above listed files.
1310
1311Packaging Externally Produced Binaries
1312--------------------------------------
1313
1314Sometimes, you need to add pre-compiled binaries to an image. For
1315example, suppose that there are binaries for proprietary code,
1316created by a particular division of a company. Your part of the company
1317needs to use those binaries as part of an image that you are building
1318using the OpenEmbedded build system. Since you only have the binaries
1319and not the source code, you cannot use a typical recipe that expects to
1320fetch the source specified in
1321:term:`SRC_URI` and then compile it.
1322
1323One method is to package the binaries and then install them as part of
1324the image. Generally, it is not a good idea to package binaries since,
1325among other things, it can hinder the ability to reproduce builds and
1326could lead to compatibility problems with ABI in the future. However,
1327sometimes you have no choice.
1328
1329The easiest solution is to create a recipe that uses the
1330:ref:`ref-classes-bin-package` class and to be sure that you are using default
1331locations for build artifacts. In most cases, the
1332:ref:`ref-classes-bin-package` class handles "skipping" the configure and
1333compile steps as well as sets things up to grab packages from the appropriate
1334area. In particular, this class sets ``noexec`` on both the
1335:ref:`ref-tasks-configure` and :ref:`ref-tasks-compile` tasks, sets
1336``FILES:${PN}`` to "/" so that it picks up all files, and sets up a
1337:ref:`ref-tasks-install` task, which effectively copies all files from ``${S}``
1338to ``${D}``. The :ref:`ref-classes-bin-package` class works well when the files
1339extracted into ``${S}`` are already laid out in the way they should be laid out
1340on the target. For more information on these variables, see the :term:`FILES`,
1341:term:`PN`, :term:`S`, and :term:`D` variables in the Yocto Project Reference
1342Manual's variable glossary.
1343
1344.. note::
1345
1346 - Using :term:`DEPENDS` is a good
1347 idea even for components distributed in binary form, and is often
1348 necessary for shared libraries. For a shared library, listing the
1349 library dependencies in :term:`DEPENDS` makes sure that the libraries
1350 are available in the staging sysroot when other recipes link
1351 against the library, which might be necessary for successful
1352 linking.
1353
1354 - Using :term:`DEPENDS` also allows runtime dependencies between
1355 packages to be added automatically. See the
1356 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
1357 section in the Yocto Project Overview and Concepts Manual for more
1358 information.
1359
1360If you cannot use the :ref:`ref-classes-bin-package` class, you need to be sure you are
1361doing the following:
1362
1363- Create a recipe where the
1364 :ref:`ref-tasks-configure` and
1365 :ref:`ref-tasks-compile` tasks do
1366 nothing: It is usually sufficient to just not define these tasks in
1367 the recipe, because the default implementations do nothing unless a
1368 Makefile is found in
1369 ``${``\ :term:`S`\ ``}``.
1370
1371 If ``${S}`` might contain a Makefile, or if you inherit some class
1372 that replaces :ref:`ref-tasks-configure` and :ref:`ref-tasks-compile` with custom
1373 versions, then you can use the
1374 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
1375 flag to turn the tasks into no-ops, as follows::
1376
1377 do_configure[noexec] = "1"
1378 do_compile[noexec] = "1"
1379
1380 Unlike :ref:`bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
1381 using the flag preserves the dependency chain from the :ref:`ref-tasks-fetch`,
1382 :ref:`ref-tasks-unpack`, and :ref:`ref-tasks-patch` tasks to the
1383 :ref:`ref-tasks-install` task.
1384
1385- Make sure your :ref:`ref-tasks-install` task installs the binaries
1386 appropriately.
1387
1388- Ensure that you set up :term:`FILES`
1389 (usually
1390 ``FILES:${``\ :term:`PN`\ ``}``) to
1391 point to the files you have installed, which of course depends on
1392 where you have installed them and whether those files are in
1393 different locations than the defaults.
1394
1395Following Recipe Style Guidelines
1396=================================
1397
1398When writing recipes, it is good to conform to existing style
1399guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
1400provides rough guidelines for preferred recipe style.
1401
1402It is common for existing recipes to deviate a bit from this style.
1403However, aiming for at least a consistent style is a good idea. Some
1404practices, such as omitting spaces around ``=`` operators in assignments
1405or ordering recipe components in an erratic way, are widely seen as poor
1406style.
1407
1408Recipe Syntax
1409=============
1410
1411Understanding recipe file syntax is important for writing recipes. The
1412following list overviews the basic items that make up a BitBake recipe
1413file. For more complete BitBake syntax descriptions, see the
1414":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`"
1415chapter of the BitBake User Manual.
1416
1417- *Variable Assignments and Manipulations:* Variable assignments allow
1418 a value to be assigned to a variable. The assignment can be static
1419 text or might include the contents of other variables. In addition to
1420 the assignment, appending and prepending operations are also
1421 supported.
1422
1423 The following example shows some of the ways you can use variables in
1424 recipes::
1425
1426 S = "${WORKDIR}/postfix-${PV}"
1427 CFLAGS += "-DNO_ASM"
1428 CFLAGS:append = " --enable-important-feature"
1429
1430- *Functions:* Functions provide a series of actions to be performed.
1431 You usually use functions to override the default implementation of a
1432 task function or to complement a default function (i.e. append or
1433 prepend to an existing function). Standard functions use ``sh`` shell
1434 syntax, although access to OpenEmbedded variables and internal
1435 methods are also available.
1436
1437 Here is an example function from the ``sed`` recipe::
1438
1439 do_install () {
1440 autotools_do_install
1441 install -d ${D}${base_bindir}
1442 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
1443 rmdir ${D}${bindir}/
1444 }
1445
1446 It is
1447 also possible to implement new functions that are called between
1448 existing tasks as long as the new functions are not replacing or
1449 complementing the default functions. You can implement functions in
1450 Python instead of shell. Both of these options are not seen in the
1451 majority of recipes.
1452
1453- *Keywords:* BitBake recipes use only a few keywords. You use keywords
1454 to include common functions (``inherit``), load parts of a recipe
1455 from other files (``include`` and ``require``) and export variables
1456 to the environment (``export``).
1457
1458 The following example shows the use of some of these keywords::
1459
1460 export POSTCONF = "${STAGING_BINDIR}/postconf"
1461 inherit autoconf
1462 require otherfile.inc
1463
1464- *Comments (#):* Any lines that begin with the hash character (``#``)
1465 are treated as comment lines and are ignored::
1466
1467 # This is a comment
1468
1469This next list summarizes the most important and most commonly used
1470parts of the recipe syntax. For more information on these parts of the
1471syntax, you can reference the
1472":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
1473in the BitBake User Manual.
1474
1475- *Line Continuation (\\):* Use the backward slash (``\``) character to
1476 split a statement over multiple lines. Place the slash character at
1477 the end of the line that is to be continued on the next line::
1478
1479 VAR = "A really long \
1480 line"
1481
1482 .. note::
1483
1484 You cannot have any characters including spaces or tabs after the
1485 slash character.
1486
1487- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
1488 access the contents of a variable::
1489
1490 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
1491
1492 .. note::
1493
1494 It is important to understand that the value of a variable
1495 expressed in this form does not get substituted automatically. The
1496 expansion of these expressions happens on-demand later (e.g.
1497 usually when a function that makes reference to the variable
1498 executes). This behavior ensures that the values are most
1499 appropriate for the context in which they are finally used. On the
1500 rare occasion that you do need the variable expression to be
1501 expanded immediately, you can use the
1502 :=
1503 operator instead of
1504 =
1505 when you make the assignment, but this is not generally needed.
1506
1507- *Quote All Assignments ("value"):* Use double quotes around values in
1508 all variable assignments (e.g. ``"value"``). Following is an example::
1509
1510 VAR1 = "${OTHERVAR}"
1511 VAR2 = "The version is ${PV}"
1512
1513- *Conditional Assignment (?=):* Conditional assignment is used to
1514 assign a value to a variable, but only when the variable is currently
1515 unset. Use the question mark followed by the equal sign (``?=``) to
1516 make a "soft" assignment used for conditional assignment. Typically,
1517 "soft" assignments are used in the ``local.conf`` file for variables
1518 that are allowed to come through from the external environment.
1519
1520 Here is an example where ``VAR1`` is set to "New value" if it is
1521 currently empty. However, if ``VAR1`` has already been set, it
1522 remains unchanged::
1523
1524 VAR1 ?= "New value"
1525
1526 In this next example, ``VAR1`` is left with the value "Original value"::
1527
1528 VAR1 = "Original value"
1529 VAR1 ?= "New value"
1530
1531- *Appending (+=):* Use the plus character followed by the equals sign
1532 (``+=``) to append values to existing variables.
1533
1534 .. note::
1535
1536 This operator adds a space between the existing content of the
1537 variable and the new content.
1538
1539 Here is an example::
1540
1541 SRC_URI += "file://fix-makefile.patch"
1542
1543- *Prepending (=+):* Use the equals sign followed by the plus character
1544 (``=+``) to prepend values to existing variables.
1545
1546 .. note::
1547
1548 This operator adds a space between the new content and the
1549 existing content of the variable.
1550
1551 Here is an example::
1552
1553 VAR =+ "Starts"
1554
1555- *Appending (:append):* Use the ``:append`` operator to append values
1556 to existing variables. This operator does not add any additional
1557 space. Also, the operator is applied after all the ``+=``, and ``=+``
1558 operators have been applied and after all ``=`` assignments have
1559 occurred. This means that if ``:append`` is used in a recipe, it can
1560 only be overridden by another layer using the special ``:remove``
1561 operator, which in turn will prevent further layers from adding it back.
1562
1563 The following example shows the space being explicitly added to the
1564 start to ensure the appended value is not merged with the existing
1565 value::
1566
1567 CFLAGS:append = " --enable-important-feature"
1568
1569 You can also use
1570 the ``:append`` operator with overrides, which results in the actions
1571 only being performed for the specified target or machine::
1572
1573 CFLAGS:append:sh4 = " --enable-important-sh4-specific-feature"
1574
1575- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend
1576 values to existing variables. This operator does not add any
1577 additional space. Also, the operator is applied after all the ``+=``,
1578 and ``=+`` operators have been applied and after all ``=``
1579 assignments have occurred.
1580
1581 The following example shows the space being explicitly added to the
1582 end to ensure the prepended value is not merged with the existing
1583 value::
1584
1585 CFLAGS:prepend = "-I${S}/myincludes "
1586
1587 You can also use the
1588 ``:prepend`` operator with overrides, which results in the actions
1589 only being performed for the specified target or machine::
1590
1591 CFLAGS:prepend:sh4 = "-I${S}/myincludes "
1592
1593- *Overrides:* You can use overrides to set a value conditionally,
1594 typically based on how the recipe is being built. For example, to set
1595 the :term:`KBRANCH` variable's
1596 value to "standard/base" for any target
1597 :term:`MACHINE`, except for
1598 qemuarm where it should be set to "standard/arm-versatile-926ejs",
1599 you would do the following::
1600
1601 KBRANCH = "standard/base"
1602 KBRANCH:qemuarm = "standard/arm-versatile-926ejs"
1603
1604 Overrides are also used to separate
1605 alternate values of a variable in other situations. For example, when
1606 setting variables such as
1607 :term:`FILES` and
1608 :term:`RDEPENDS` that are
1609 specific to individual packages produced by a recipe, you should
1610 always use an override that specifies the name of the package.
1611
1612- *Indentation:* Use spaces for indentation rather than tabs. For
1613 shell functions, both currently work. However, it is a policy
1614 decision of the Yocto Project to use tabs in shell functions. Realize
1615 that some layers have a policy to use spaces for all indentation.
1616
1617- *Using Python for Complex Operations:* For more advanced processing,
1618 it is possible to use Python code during variable assignments (e.g.
1619 search and replacement on a variable).
1620
1621 You indicate Python code using the ``${@python_code}`` syntax for the
1622 variable assignment::
1623
1624 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
1625
1626- *Shell Function Syntax:* Write shell functions as if you were writing
1627 a shell script when you describe a list of actions to take. You
1628 should ensure that your script works with a generic ``sh`` and that
1629 it does not require any ``bash`` or other shell-specific
1630 functionality. The same considerations apply to various system
1631 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
1632 might wish to use. If in doubt, you should check with multiple
1633 implementations --- including those from BusyBox.
1634
diff --git a/documentation/dev-manual/packages.rst b/documentation/dev-manual/packages.rst
new file mode 100644
index 0000000000..79f21d9f34
--- /dev/null
+++ b/documentation/dev-manual/packages.rst
@@ -0,0 +1,1238 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working with Packages
4*********************
5
6This section describes a few tasks that involve packages:
7
8- :ref:`dev-manual/packages:excluding packages from an image`
9
10- :ref:`dev-manual/packages:incrementing a package version`
11
12- :ref:`dev-manual/packages:handling optional module packaging`
13
14- :ref:`dev-manual/packages:using runtime package management`
15
16- :ref:`dev-manual/packages:generating and using signed packages`
17
18- :ref:`Setting up and running package test
19 (ptest) <dev-manual/packages:testing packages with ptest>`
20
21- :ref:`dev-manual/packages:creating node package manager (npm) packages`
22
23- :ref:`dev-manual/packages:adding custom metadata to packages`
24
25Excluding Packages from an Image
26================================
27
28You might find it necessary to prevent specific packages from being
29installed into an image. If so, you can use several variables to direct
30the build system to essentially ignore installing recommended packages
31or to not install a package at all.
32
33The following list introduces variables you can use to prevent packages
34from being installed into your image. Each of these variables only works
35with IPK and RPM package types, not for Debian packages.
36Also, you can use these variables from your ``local.conf`` file
37or attach them to a specific image recipe by using a recipe name
38override. For more detail on the variables, see the descriptions in the
39Yocto Project Reference Manual's glossary chapter.
40
41- :term:`BAD_RECOMMENDATIONS`:
42 Use this variable to specify "recommended-only" packages that you do
43 not want installed.
44
45- :term:`NO_RECOMMENDATIONS`:
46 Use this variable to prevent all "recommended-only" packages from
47 being installed.
48
49- :term:`PACKAGE_EXCLUDE`:
50 Use this variable to prevent specific packages from being installed
51 regardless of whether they are "recommended-only" or not. You need to
52 realize that the build process could fail with an error when you
53 prevent the installation of a package whose presence is required by
54 an installed package.
55
56Incrementing a Package Version
57==============================
58
59This section provides some background on how binary package versioning
60is accomplished and presents some of the services, variables, and
61terminology involved.
62
63In order to understand binary package versioning, you need to consider
64the following:
65
66- Binary Package: The binary package that is eventually built and
67 installed into an image.
68
69- Binary Package Version: The binary package version is composed of two
70 components --- a version and a revision.
71
72 .. note::
73
74 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
75 but this discussion for the most part ignores :term:`PE`.
76
77 The version and revision are taken from the
78 :term:`PV` and
79 :term:`PR` variables, respectively.
80
81- :term:`PV`: The recipe version. :term:`PV` represents the version of the
82 software being packaged. Do not confuse :term:`PV` with the binary
83 package version.
84
85- :term:`PR`: The recipe revision.
86
87- :term:`SRCPV`: The OpenEmbedded
88 build system uses this string to help define the value of :term:`PV` when
89 the source code revision needs to be included in it.
90
91- :yocto_wiki:`PR Service </PR_Service>`: A
92 network-based service that helps automate keeping package feeds
93 compatible with existing package manager applications such as RPM,
94 APT, and OPKG.
95
96Whenever the binary package content changes, the binary package version
97must change. Changing the binary package version is accomplished by
98changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
99values occurs one of two ways:
100
101- Automatically using a Package Revision Service (PR Service).
102
103- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
104
105Given a primary challenge of any build system and its users is how to
106maintain a package feed that is compatible with existing package manager
107applications such as RPM, APT, and OPKG, using an automated system is
108much preferred over a manual system. In either system, the main
109requirement is that binary package version numbering increases in a
110linear fashion and that there is a number of version components that
111support that linear progression. For information on how to ensure
112package revisioning remains linear, see the
113":ref:`dev-manual/packages:automatically incrementing a package version number`"
114section.
115
116The following three sections provide related information on the PR
117Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
118how to ensure binary package revisioning remains linear.
119
120Working With a PR Service
121-------------------------
122
123As mentioned, attempting to maintain revision numbers in the
124:term:`Metadata` is error prone, inaccurate,
125and causes problems for people submitting recipes. Conversely, the PR
126Service automatically generates increasing numbers, particularly the
127revision field, which removes the human element.
128
129.. note::
130
131 For additional information on using a PR Service, you can see the
132 :yocto_wiki:`PR Service </PR_Service>` wiki page.
133
134The Yocto Project uses variables in order of decreasing priority to
135facilitate revision numbering (i.e.
136:term:`PE`,
137:term:`PV`, and
138:term:`PR` for epoch, version, and
139revision, respectively). The values are highly dependent on the policies
140and procedures of a given distribution and package feed.
141
142Because the OpenEmbedded build system uses
143":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
144unique to a given build, the build system knows when to rebuild
145packages. All the inputs into a given task are represented by a
146signature, which can trigger a rebuild when different. Thus, the build
147system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
148trigger a rebuild. The signatures, however, can be used to generate
149these values.
150
151The PR Service works with both ``OEBasic`` and ``OEBasicHash``
152generators. The value of :term:`PR` bumps when the checksum changes and the
153different generator mechanisms change signatures under different
154circumstances.
155
156As implemented, the build system includes values from the PR Service
157into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
158becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
159:term:`PR` values to be used for whatever reasons, which include manual
160:term:`PR` bumps, should it be necessary.
161
162By default, the PR Service is not enabled or running. Thus, the packages
163generated are just "self consistent". The build system adds and removes
164packages and there are no guarantees about upgrade paths but images will
165be consistent and correct with the latest changes.
166
167The simplest form for a PR Service is for a single host development system
168that builds the package feed (building system). For this scenario, you can
169enable a local PR Service by setting :term:`PRSERV_HOST` in your
170``local.conf`` file in the :term:`Build Directory`::
171
172 PRSERV_HOST = "localhost:0"
173
174Once the service is started, packages will automatically
175get increasing :term:`PR` values and BitBake takes care of starting and
176stopping the server.
177
178If you have a more complex setup where multiple host development systems
179work against a common, shared package feed, you have a single PR Service
180running and it is connected to each building system. For this scenario,
181you need to start the PR Service using the ``bitbake-prserv`` command::
182
183 bitbake-prserv --host ip --port port --start
184
185In addition to
186hand-starting the service, you need to update the ``local.conf`` file of
187each building system as described earlier so each system points to the
188server and port.
189
190It is also recommended you use build history, which adds some sanity
191checks to binary package versions, in conjunction with the server that
192is running the PR Service. To enable build history, add the following to
193each building system's ``local.conf`` file::
194
195 # It is recommended to activate "buildhistory" for testing the PR service
196 INHERIT += "buildhistory"
197 BUILDHISTORY_COMMIT = "1"
198
199For information on build
200history, see the
201":ref:`dev-manual/build-quality:maintaining build output quality`" section.
202
203.. note::
204
205 The OpenEmbedded build system does not maintain :term:`PR` information as
206 part of the shared state (sstate) packages. If you maintain an sstate
207 feed, it's expected that either all your building systems that
208 contribute to the sstate feed use a shared PR Service, or you do not
209 run a PR Service on any of your building systems. Having some systems
210 use a PR Service while others do not leads to obvious problems.
211
212 For more information on shared state, see the
213 ":ref:`overview-manual/concepts:shared state cache`"
214 section in the Yocto Project Overview and Concepts Manual.
215
216Manually Bumping PR
217-------------------
218
219The alternative to setting up a PR Service is to manually "bump" the
220:term:`PR` variable.
221
222If a committed change results in changing the package output, then the
223value of the :term:`PR` variable needs to be increased (or "bumped") as part of
224that commit. For new recipes you should add the :term:`PR` variable and set
225its initial value equal to "r0", which is the default. Even though the
226default value is "r0", the practice of adding it to a new recipe makes
227it harder to forget to bump the variable when you make changes to the
228recipe in future.
229
230Usually, version increases occur only to binary packages. However, if
231for some reason :term:`PV` changes but does not increase, you can increase
232the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
233"0".
234
235Binary package version numbering strives to follow the `Debian Version
236Field Policy
237Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
238These guidelines define how versions are compared and what "increasing"
239a version means.
240
241Automatically Incrementing a Package Version Number
242---------------------------------------------------
243
244When fetching a repository, BitBake uses the
245:term:`SRCREV` variable to determine
246the specific source code revision from which to build. You set the
247:term:`SRCREV` variable to
248:term:`AUTOREV` to cause the
249OpenEmbedded build system to automatically use the latest revision of
250the software::
251
252 SRCREV = "${AUTOREV}"
253
254Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
255automatically update the version whenever the revision of the source
256code changes. Here is an example::
257
258 PV = "1.0+git${SRCPV}"
259
260The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
261
262.. code-block:: none
263
264 AUTOINC+source_code_revision
265
266The build system replaces the ``AUTOINC``
267with a number. The number used depends on the state of the PR Service:
268
269- If PR Service is enabled, the build system increments the number,
270 which is similar to the behavior of
271 :term:`PR`. This behavior results in
272 linearly increasing package versions, which is desirable. Here is an
273 example:
274
275 .. code-block:: none
276
277 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
278 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
279
280- If PR Service is not enabled, the build system replaces the
281 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
282 changing the package version since the source revision is included.
283 However, package versions are not increased linearly. Here is an
284 example:
285
286 .. code-block:: none
287
288 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
289 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
290
291In summary, the OpenEmbedded build system does not track the history of
292binary package versions for this purpose. ``AUTOINC``, in this case, is
293comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
294package version is simply replaced by "0". If PR server is enabled, the
295build system keeps track of the package versions and bumps the number
296when the package revision changes.
297
298Handling Optional Module Packaging
299==================================
300
301Many pieces of software split functionality into optional modules (or
302plugins) and the plugins that are built might depend on configuration
303options. To avoid having to duplicate the logic that determines what
304modules are available in your recipe or to avoid having to package each
305module by hand, the OpenEmbedded build system provides functionality to
306handle module packaging dynamically.
307
308To handle optional module packaging, you need to do two things:
309
310- Ensure the module packaging is actually done.
311
312- Ensure that any dependencies on optional modules from other recipes
313 are satisfied by your recipe.
314
315Making Sure the Packaging is Done
316---------------------------------
317
318To ensure the module packaging actually gets done, you use the
319``do_split_packages`` function within the ``populate_packages`` Python
320function in your recipe. The ``do_split_packages`` function searches for
321a pattern of files or directories under a specified path and creates a
322package for each one it finds by appending to the
323:term:`PACKAGES` variable and
324setting the appropriate values for ``FILES:packagename``,
325``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
326Here is an example from the ``lighttpd`` recipe::
327
328 python populate_packages:prepend () {
329 lighttpd_libdir = d.expand('${libdir}')
330 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
331 'lighttpd-module-%s', 'Lighttpd module for %s',
332 extra_depends='')
333 }
334
335The previous example specifies a number of things in the call to
336``do_split_packages``.
337
338- A directory within the files installed by your recipe through
339 :ref:`ref-tasks-install` in which to search.
340
341- A regular expression used to match module files in that directory. In
342 the example, note the parentheses () that mark the part of the
343 expression from which the module name should be derived.
344
345- A pattern to use for the package names.
346
347- A description for each package.
348
349- An empty string for ``extra_depends``, which disables the default
350 dependency on the main ``lighttpd`` package. Thus, if a file in
351 ``${libdir}`` called ``mod_alias.so`` is found, a package called
352 ``lighttpd-module-alias`` is created for it and the
353 :term:`DESCRIPTION` is set to
354 "Lighttpd module for alias".
355
356Often, packaging modules is as simple as the previous example. However,
357there are more advanced options that you can use within
358``do_split_packages`` to modify its behavior. And, if you need to, you
359can add more logic by specifying a hook function that is called for each
360package. It is also perfectly acceptable to call ``do_split_packages``
361multiple times if you have more than one set of modules to package.
362
363For more examples that show how to use ``do_split_packages``, see the
364``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
365directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
366also find examples in ``meta/classes-recipe/kernel.bbclass``.
367
368Following is a reference that shows ``do_split_packages`` mandatory and
369optional arguments::
370
371 Mandatory arguments
372
373 root
374 The path in which to search
375 file_regex
376 Regular expression to match searched files.
377 Use parentheses () to mark the part of this
378 expression that should be used to derive the
379 module name (to be substituted where %s is
380 used in other function arguments as noted below)
381 output_pattern
382 Pattern to use for the package names. Must
383 include %s.
384 description
385 Description to set for each package. Must
386 include %s.
387
388 Optional arguments
389
390 postinst
391 Postinstall script to use for all packages
392 (as a string)
393 recursive
394 True to perform a recursive search --- default
395 False
396 hook
397 A hook function to be called for every match.
398 The function will be called with the following
399 arguments (in the order listed):
400
401 f
402 Full path to the file/directory match
403 pkg
404 The package name
405 file_regex
406 As above
407 output_pattern
408 As above
409 modulename
410 The module name derived using file_regex
411 extra_depends
412 Extra runtime dependencies (RDEPENDS) to be
413 set for all packages. The default value of None
414 causes a dependency on the main package
415 (${PN}) --- if you do not want this, pass empty
416 string '' for this parameter.
417 aux_files_pattern
418 Extra item(s) to be added to FILES for each
419 package. Can be a single string item or a list
420 of strings for multiple items. Must include %s.
421 postrm
422 postrm script to use for all packages (as a
423 string)
424 allow_dirs
425 True to allow directories to be matched -
426 default False
427 prepend
428 If True, prepend created packages to PACKAGES
429 instead of the default False which appends them
430 match_path
431 match file_regex on the whole relative path to
432 the root rather than just the filename
433 aux_files_pattern_verbatim
434 Extra item(s) to be added to FILES for each
435 package, using the actual derived module name
436 rather than converting it to something legal
437 for a package name. Can be a single string item
438 or a list of strings for multiple items. Must
439 include %s.
440 allow_links
441 True to allow symlinks to be matched --- default
442 False
443 summary
444 Summary to set for each package. Must include %s;
445 defaults to description if not set.
446
447
448
449Satisfying Dependencies
450-----------------------
451
452The second part for handling optional module packaging is to ensure that
453any dependencies on optional modules from other recipes are satisfied by
454your recipe. You can be sure these dependencies are satisfied by using
455the :term:`PACKAGES_DYNAMIC`
456variable. Here is an example that continues with the ``lighttpd`` recipe
457shown earlier::
458
459 PACKAGES_DYNAMIC = "lighttpd-module-.*"
460
461The name
462specified in the regular expression can of course be anything. In this
463example, it is ``lighttpd-module-`` and is specified as the prefix to
464ensure that any :term:`RDEPENDS` and
465:term:`RRECOMMENDS` on a package
466name starting with the prefix are satisfied during build time. If you
467are using ``do_split_packages`` as described in the previous section,
468the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
469pattern specified in the call to ``do_split_packages``.
470
471Using Runtime Package Management
472================================
473
474During a build, BitBake always transforms a recipe into one or more
475packages. For example, BitBake takes the ``bash`` recipe and produces a
476number of packages (e.g. ``bash``, ``bash-bashbug``,
477``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
478``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
479generated packages are included in an image.
480
481In several situations, you might need to update, add, remove, or query
482the packages on a target device at runtime (i.e. without having to
483generate a new image). Examples of such situations include:
484
485- You want to provide in-the-field updates to deployed devices (e.g.
486 security updates).
487
488- You want to have a fast turn-around development cycle for one or more
489 applications that run on your device.
490
491- You want to temporarily install the "debug" packages of various
492 applications on your device so that debugging can be greatly improved
493 by allowing access to symbols and source debugging.
494
495- You want to deploy a more minimal package selection of your device
496 but allow in-the-field updates to add a larger selection for
497 customization.
498
499In all these situations, you have something similar to a more
500traditional Linux distribution in that in-field devices are able to
501receive pre-compiled packages from a server for installation or update.
502Being able to install these packages on a running, in-field device is
503what is termed "runtime package management".
504
505In order to use runtime package management, you need a host or server
506machine that serves up the pre-compiled packages plus the required
507metadata. You also need package manipulation tools on the target. The
508build machine is a likely candidate to act as the server. However, that
509machine does not necessarily have to be the package server. The build
510machine could push its artifacts to another machine that acts as the
511server (e.g. Internet-facing). In fact, doing so is advantageous for a
512production environment as getting the packages away from the development
513system's :term:`Build Directory` prevents accidental overwrites.
514
515A simple build that targets just one device produces more than one
516package database. In other words, the packages produced by a build are
517separated out into a couple of different package groupings based on
518criteria such as the target's CPU architecture, the target board, or the
519C library used on the target. For example, a build targeting the
520``qemux86`` device produces the following three package databases:
521``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
522device to be aware of all the packages that were available to it, you
523would need to point it to each of these databases individually. In a
524similar way, a traditional Linux distribution usually is configured to
525be aware of a number of software repositories from which it retrieves
526packages.
527
528Using runtime package management is completely optional and not required
529for a successful build or deployment in any way. But if you want to make
530use of runtime package management, you need to do a couple things above
531and beyond the basics. The remainder of this section describes what you
532need to do.
533
534Build Considerations
535--------------------
536
537This section describes build considerations of which you need to be
538aware in order to provide support for runtime package management.
539
540When BitBake generates packages, it needs to know what format or formats
541to use. In your configuration, you use the
542:term:`PACKAGE_CLASSES`
543variable to specify the format:
544
545#. Open the ``local.conf`` file inside your :term:`Build Directory` (e.g.
546 ``poky/build/conf/local.conf``).
547
548#. Select the desired package format as follows::
549
550 PACKAGE_CLASSES ?= "package_packageformat"
551
552 where packageformat can be "ipk", "rpm",
553 "deb", or "tar" which are the supported package formats.
554
555 .. note::
556
557 Because the Yocto Project supports four different package formats,
558 you can set the variable with more than one argument. However, the
559 OpenEmbedded build system only uses the first argument when
560 creating an image or Software Development Kit (SDK).
561
562If you would like your image to start off with a basic package database
563containing the packages in your current build as well as to have the
564relevant tools available on the target for runtime package management,
565you can include "package-management" in the
566:term:`IMAGE_FEATURES`
567variable. Including "package-management" in this configuration variable
568ensures that when the image is assembled for your target, the image
569includes the currently-known package databases as well as the
570target-specific tools required for runtime package management to be
571performed on the target. However, this is not strictly necessary. You
572could start your image off without any databases but only include the
573required on-target package tool(s). As an example, you could include
574"opkg" in your
575:term:`IMAGE_INSTALL` variable
576if you are using the IPK package format. You can then initialize your
577target's package database(s) later once your image is up and running.
578
579Whenever you perform any sort of build step that can potentially
580generate a package or modify existing package, it is always a good idea
581to re-generate the package index after the build by using the following
582command::
583
584 $ bitbake package-index
585
586It might be tempting to build the
587package and the package index at the same time with a command such as
588the following::
589
590 $ bitbake some-package package-index
591
592Do not do this as
593BitBake does not schedule the package index for after the completion of
594the package you are building. Consequently, you cannot be sure of the
595package index including information for the package you just built.
596Thus, be sure to run the package update step separately after building
597any packages.
598
599You can use the
600:term:`PACKAGE_FEED_ARCHS`,
601:term:`PACKAGE_FEED_BASE_PATHS`,
602and
603:term:`PACKAGE_FEED_URIS`
604variables to pre-configure target images to use a package feed. If you
605do not define these variables, then manual steps as described in the
606subsequent sections are necessary to configure the target. You should
607set these variables before building the image in order to produce a
608correctly configured image.
609
610When your build is complete, your packages reside in the
611``${TMPDIR}/deploy/packageformat`` directory. For example, if
612``${``\ :term:`TMPDIR`\ ``}`` is
613``tmp`` and your selected package type is RPM, then your RPM packages
614are available in ``tmp/deploy/rpm``.
615
616Host or Server Machine Setup
617----------------------------
618
619Although other protocols are possible, a server using HTTP typically
620serves packages. If you want to use HTTP, then set up and configure a
621web server such as Apache 2, lighttpd, or Python web server on the
622machine serving the packages.
623
624To keep things simple, this section describes how to set up a
625Python web server to share package feeds from the developer's
626machine. Although this server might not be the best for a production
627environment, the setup is simple and straight forward. Should you want
628to use a different server more suited for production (e.g. Apache 2,
629Lighttpd, or Nginx), take the appropriate steps to do so.
630
631From within the :term:`Build Directory` where you have built an image based on
632your packaging choice (i.e. the :term:`PACKAGE_CLASSES` setting), simply start
633the server. The following example assumes a :term:`Build Directory` of ``poky/build``
634and a :term:`PACKAGE_CLASSES` setting of ":ref:`ref-classes-package_rpm`"::
635
636 $ cd poky/build/tmp/deploy/rpm
637 $ python3 -m http.server
638
639Target Setup
640------------
641
642Setting up the target differs depending on the package management
643system. This section provides information for RPM, IPK, and DEB.
644
645Using RPM
646~~~~~~~~~
647
648The :wikipedia:`Dandified Packaging <DNF_(software)>` (DNF) performs
649runtime package management of RPM packages. In order to use DNF for
650runtime package management, you must perform an initial setup on the
651target machine for cases where the ``PACKAGE_FEED_*`` variables were not
652set as part of the image that is running on the target. This means if
653you built your image and did not use these variables as part of the
654build and your image is now running on the target, you need to perform
655the steps in this section if you want to use runtime package management.
656
657.. note::
658
659 For information on the ``PACKAGE_FEED_*`` variables, see
660 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
661 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
662 glossary.
663
664On the target, you must inform DNF that package databases are available.
665You do this by creating a file named
666``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
667
668As an example, assume the target is able to use the following package
669databases: ``all``, ``i586``, and ``qemux86`` from a server named
670``my.server``. The specifics for setting up the web server are up to
671you. The critical requirement is that the URIs in the target repository
672configuration point to the correct remote location for the feeds.
673
674.. note::
675
676 For development purposes, you can point the web server to the build
677 system's ``deploy`` directory. However, for production use, it is better to
678 copy the package directories to a location outside of the build area and use
679 that location. Doing so avoids situations where the build system
680 overwrites or changes the ``deploy`` directory.
681
682When telling DNF where to look for the package databases, you must
683declare individual locations per architecture or a single location used
684for all architectures. You cannot do both:
685
686- *Create an Explicit List of Architectures:* Define individual base
687 URLs to identify where each package database is located:
688
689 .. code-block:: none
690
691 [oe-packages]
692 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
693
694 This example
695 informs DNF about individual package databases for all three
696 architectures.
697
698- *Create a Single (Full) Package Index:* Define a single base URL that
699 identifies where a full package database is located::
700
701 [oe-packages]
702 baseurl=http://my.server/rpm
703
704 This example informs DNF about a single
705 package database that contains all the package index information for
706 all supported architectures.
707
708Once you have informed DNF where to find the package databases, you need
709to fetch them:
710
711.. code-block:: none
712
713 # dnf makecache
714
715DNF is now able to find, install, and
716upgrade packages from the specified repository or repositories.
717
718.. note::
719
720 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
721 additional information.
722
723Using IPK
724~~~~~~~~~
725
726The ``opkg`` application performs runtime package management of IPK
727packages. You must perform an initial setup for ``opkg`` on the target
728machine if the
729:term:`PACKAGE_FEED_ARCHS`,
730:term:`PACKAGE_FEED_BASE_PATHS`,
731and
732:term:`PACKAGE_FEED_URIS`
733variables have not been set or the target image was built before the
734variables were set.
735
736The ``opkg`` application uses configuration files to find available
737package databases. Thus, you need to create a configuration file inside
738the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
739you want to use.
740
741As an example, suppose you are serving packages from a ``ipk/``
742directory containing the ``i586``, ``all``, and ``qemux86`` databases
743through an HTTP server named ``my.server``. On the target, create a
744configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
745directory containing the following:
746
747.. code-block:: none
748
749 src/gz all http://my.server/ipk/all
750 src/gz i586 http://my.server/ipk/i586
751 src/gz qemux86 http://my.server/ipk/qemux86
752
753Next, instruct ``opkg`` to fetch the
754repository information:
755
756.. code-block:: none
757
758 # opkg update
759
760The ``opkg`` application is now able to find, install, and upgrade packages
761from the specified repository.
762
763Using DEB
764~~~~~~~~~
765
766The ``apt`` application performs runtime package management of DEB
767packages. This application uses a source list file to find available
768package databases. You must perform an initial setup for ``apt`` on the
769target machine if the
770:term:`PACKAGE_FEED_ARCHS`,
771:term:`PACKAGE_FEED_BASE_PATHS`,
772and
773:term:`PACKAGE_FEED_URIS`
774variables have not been set or the target image was built before the
775variables were set.
776
777To inform ``apt`` of the repository you want to use, you might create a
778list file (e.g. ``my_repo.list``) inside the
779``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
780serving packages from a ``deb/`` directory containing the ``i586``,
781``all``, and ``qemux86`` databases through an HTTP server named
782``my.server``. The list file should contain:
783
784.. code-block:: none
785
786 deb http://my.server/deb/all ./
787 deb http://my.server/deb/i586 ./
788 deb http://my.server/deb/qemux86 ./
789
790Next, instruct the ``apt`` application
791to fetch the repository information:
792
793.. code-block:: none
794
795 $ sudo apt update
796
797After this step,
798``apt`` is able to find, install, and upgrade packages from the
799specified repository.
800
801Generating and Using Signed Packages
802====================================
803
804In order to add security to RPM packages used during a build, you can
805take steps to securely sign them. Once a signature is verified, the
806OpenEmbedded build system can use the package in the build. If security
807fails for a signed package, the build system stops the build.
808
809This section describes how to sign RPM packages during a build and how
810to use signed package feeds (repositories) when doing a build.
811
812Signing RPM Packages
813--------------------
814
815To enable signing RPM packages, you must set up the following
816configurations in either your ``local.config`` or ``distro.config``
817file::
818
819 # Inherit sign_rpm.bbclass to enable signing functionality
820 INHERIT += " sign_rpm"
821 # Define the GPG key that will be used for signing.
822 RPM_GPG_NAME = "key_name"
823 # Provide passphrase for the key
824 RPM_GPG_PASSPHRASE = "passphrase"
825
826.. note::
827
828 Be sure to supply appropriate values for both `key_name` and
829 `passphrase`.
830
831Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
832the previous example, two optional variables related to signing are available:
833
834- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
835 when the package is signed.
836
837- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
838 package is signed.
839
840Processing Package Feeds
841------------------------
842
843In addition to being able to sign RPM packages, you can also enable
844signed package feeds for IPK and RPM packages.
845
846The steps you need to take to enable signed package feed use are similar
847to the steps used to sign RPM packages. You must define the following in
848your ``local.config`` or ``distro.config`` file::
849
850 INHERIT += "sign_package_feed"
851 PACKAGE_FEED_GPG_NAME = "key_name"
852 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
853
854For signed package feeds, the passphrase must be specified in a separate file,
855which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
856variable. Regarding security, keeping a plain text passphrase out of the
857configuration is more secure.
858
859Aside from the ``PACKAGE_FEED_GPG_NAME`` and
860``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
861related to signed package feeds are available:
862
863- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
864 when the package is signed.
865
866- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
867 package is signed.
868
869- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
870 signature. This variable applies only to RPM and IPK package feeds.
871 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
872 "ASC", which is the default and specifies ascii armored, and "BIN",
873 which specifies binary.
874
875Testing Packages With ptest
876===========================
877
878A Package Test (ptest) runs tests against packages built by the
879OpenEmbedded build system on the target machine. A ptest contains at
880least two items: the actual test, and a shell script (``run-ptest``)
881that starts the test. The shell script that starts the test must not
882contain the actual test --- the script only starts the test. On the other
883hand, the test can be anything from a simple shell script that runs a
884binary and checks the output to an elaborate system of test binaries and
885data files.
886
887The test generates output in the format used by Automake::
888
889 result: testname
890
891where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
892the testname can be any identifying string.
893
894For a list of Yocto Project recipes that are already enabled with ptest,
895see the :yocto_wiki:`Ptest </Ptest>` wiki page.
896
897.. note::
898
899 A recipe is "ptest-enabled" if it inherits the :ref:`ref-classes-ptest`
900 class.
901
902Adding ptest to Your Build
903--------------------------
904
905To add package testing to your build, add the :term:`DISTRO_FEATURES` and
906:term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which
907is found in the :term:`Build Directory`::
908
909 DISTRO_FEATURES:append = " ptest"
910 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
911
912Once your build is complete, the ptest files are installed into the
913``/usr/lib/package/ptest`` directory within the image, where ``package``
914is the name of the package.
915
916Running ptest
917-------------
918
919The ``ptest-runner`` package installs a shell script that loops through
920all installed ptest test suites and runs them in sequence. Consequently,
921you might want to add this package to your image.
922
923Getting Your Package Ready
924--------------------------
925
926In order to enable a recipe to run installed ptests on target hardware,
927you need to prepare the recipes that build the packages you want to
928test. Here is what you have to do for each recipe:
929
930- *Be sure the recipe inherits the* :ref:`ref-classes-ptest` *class:*
931 Include the following line in each recipe::
932
933 inherit ptest
934
935- *Create run-ptest:* This script starts your test. Locate the
936 script where you will refer to it using
937 :term:`SRC_URI`. Here is an
938 example that starts a test for ``dbus``::
939
940 #!/bin/sh
941 cd test
942 make -k runtest-TESTS
943
944- *Ensure dependencies are met:* If the test adds build or runtime
945 dependencies that normally do not exist for the package (such as
946 requiring "make" to run the test suite), use the
947 :term:`DEPENDS` and
948 :term:`RDEPENDS` variables in
949 your recipe in order for the package to meet the dependencies. Here
950 is an example where the package has a runtime dependency on "make"::
951
952 RDEPENDS:${PN}-ptest += "make"
953
954- *Add a function to build the test suite:* Not many packages support
955 cross-compilation of their test suites. Consequently, you usually
956 need to add a cross-compilation function to the package.
957
958 Many packages based on Automake compile and run the test suite by
959 using a single command such as ``make check``. However, the host
960 ``make check`` builds and runs on the same computer, while
961 cross-compiling requires that the package is built on the host but
962 executed for the target architecture (though often, as in the case
963 for ptest, the execution occurs on the host). The built version of
964 Automake that ships with the Yocto Project includes a patch that
965 separates building and execution. Consequently, packages that use the
966 unaltered, patched version of ``make check`` automatically
967 cross-compiles.
968
969 Regardless, you still must add a ``do_compile_ptest`` function to
970 build the test suite. Add a function similar to the following to your
971 recipe::
972
973 do_compile_ptest() {
974 oe_runmake buildtest-TESTS
975 }
976
977- *Ensure special configurations are set:* If the package requires
978 special configurations prior to compiling the test code, you must
979 insert a ``do_configure_ptest`` function into the recipe.
980
981- *Install the test suite:* The :ref:`ref-classes-ptest` class
982 automatically copies the file ``run-ptest`` to the target and then runs make
983 ``install-ptest`` to run the tests. If this is not enough, you need
984 to create a ``do_install_ptest`` function and make sure it gets
985 called after the "make install-ptest" completes.
986
987Creating Node Package Manager (NPM) Packages
988============================================
989
990:wikipedia:`NPM <Npm_(software)>` is a package manager for the JavaScript
991programming language. The Yocto Project supports the NPM
992:ref:`fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`.
993You can use this fetcher in combination with
994:doc:`devtool </ref-manual/devtool-reference>` to create recipes that produce
995NPM packages.
996
997There are two workflows that allow you to create NPM packages using
998``devtool``: the NPM registry modules method and the NPM project code
999method.
1000
1001.. note::
1002
1003 While it is possible to create NPM recipes manually, using
1004 ``devtool`` is far simpler.
1005
1006Additionally, some requirements and caveats exist.
1007
1008Requirements and Caveats
1009------------------------
1010
1011You need to be aware of the following before using ``devtool`` to create
1012NPM packages:
1013
1014- Of the two methods that you can use ``devtool`` to create NPM
1015 packages, the registry approach is slightly simpler. However, you
1016 might consider the project approach because you do not have to
1017 publish your module in the `NPM registry <https://docs.npmjs.com/misc/registry>`__,
1018 which is NPM's public registry.
1019
1020- Be familiar with
1021 :doc:`devtool </ref-manual/devtool-reference>`.
1022
1023- The NPM host tools need the native ``nodejs-npm`` package, which is
1024 part of the OpenEmbedded environment. You need to get the package by
1025 cloning the :oe_git:`meta-openembedded </meta-openembedded>`
1026 repository. Be sure to add the path to your local copy
1027 to your ``bblayers.conf`` file.
1028
1029- ``devtool`` cannot detect native libraries in module dependencies.
1030 Consequently, you must manually add packages to your recipe.
1031
1032- While deploying NPM packages, ``devtool`` cannot determine which
1033 dependent packages are missing on the target (e.g. the node runtime
1034 ``nodejs``). Consequently, you need to find out what files are
1035 missing and be sure they are on the target.
1036
1037- Although you might not need NPM to run your node package, it is
1038 useful to have NPM on your target. The NPM package name is
1039 ``nodejs-npm``.
1040
1041Using the Registry Modules Method
1042---------------------------------
1043
1044This section presents an example that uses the ``cute-files`` module,
1045which is a file browser web application.
1046
1047.. note::
1048
1049 You must know the ``cute-files`` module version.
1050
1051The first thing you need to do is use ``devtool`` and the NPM fetcher to
1052create the recipe::
1053
1054 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
1055
1056The
1057``devtool add`` command runs ``recipetool create`` and uses the same
1058fetch URI to download each dependency and capture license details where
1059possible. The result is a generated recipe.
1060
1061After running for quite a long time, in particular building the
1062``nodejs-native`` package, the command should end as follows::
1063
1064 INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional
1065
1066The recipe file is fairly simple and contains every license that
1067``recipetool`` finds and includes the licenses in the recipe's
1068:term:`LIC_FILES_CHKSUM`
1069variables. You need to examine the variables and look for those with
1070"unknown" in the :term:`LICENSE`
1071field. You need to track down the license information for "unknown"
1072modules and manually add the information to the recipe.
1073
1074``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
1075files capture the version of all dependent modules. Many packages do not
1076provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it
1077runs.
1078
1079.. note::
1080
1081 A package is created for each sub-module. This policy is the only
1082 practical way to have the licenses for all of the dependencies
1083 represented in the license manifest of the image.
1084
1085The ``devtool edit-recipe`` command lets you take a look at the recipe::
1086
1087 $ devtool edit-recipe cute-files
1088 # Recipe created by recipetool
1089 # This is the basis of a recipe and may need further editing in order to be fully functional.
1090 # (Feel free to remove these comments when editing.)
1091
1092 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
1093 # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
1094 # your responsibility to verify that the values are complete and correct.
1095 #
1096 # NOTE: multiple licenses have been detected; they have been separated with &
1097 # in the LICENSE value for now since it is a reasonable assumption that all
1098 # of the licenses apply. If instead there is a choice between the multiple
1099 # licenses then you should change the value to separate the licenses with |
1100 # instead of &. If there is any doubt, check the accompanying documentation
1101 # to determine which situation is applicable.
1102
1103 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
1104 LICENSE = "BSD-3-Clause & ISC & MIT"
1105 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
1106 file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \
1107 file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \
1108 ...
1109 file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a"
1110
1111 SRC_URI = " \
1112 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
1113 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
1114 "
1115
1116 S = "${WORKDIR}/npm"
1117
1118 inherit npm
1119
1120 LICENSE:${PN} = "MIT"
1121 LICENSE:${PN}-accepts = "MIT"
1122 LICENSE:${PN}-array-flatten = "MIT"
1123 ...
1124 LICENSE:${PN}-vary = "MIT"
1125
1126Here are three key points in the previous example:
1127
1128- :term:`SRC_URI` uses the NPM
1129 scheme so that the NPM fetcher is used.
1130
1131- ``recipetool`` collects all the license information. If a
1132 sub-module's license is unavailable, the sub-module's name appears in
1133 the comments.
1134
1135- The ``inherit npm`` statement causes the :ref:`ref-classes-npm` class to
1136 package up all the modules.
1137
1138You can run the following command to build the ``cute-files`` package::
1139
1140 $ devtool build cute-files
1141
1142Remember that ``nodejs`` must be installed on
1143the target before your package.
1144
1145Assuming 192.168.7.2 for the target's IP address, use the following
1146command to deploy your package::
1147
1148 $ devtool deploy-target -s cute-files root@192.168.7.2
1149
1150Once the package is installed on the target, you can
1151test the application to show the contents of any directory::
1152
1153 $ cd /usr/lib/node_modules/cute-files
1154 $ cute-files
1155
1156On a browser,
1157go to ``http://192.168.7.2:3000`` and you see the following:
1158
1159.. image:: figures/cute-files-npm-example.png
1160 :width: 100%
1161
1162You can find the recipe in ``workspace/recipes/cute-files``. You can use
1163the recipe in any layer you choose.
1164
1165Using the NPM Projects Code Method
1166----------------------------------
1167
1168Although it is useful to package modules already in the NPM registry,
1169adding ``node.js`` projects under development is a more common developer
1170use case.
1171
1172This section covers the NPM projects code method, which is very similar
1173to the "registry" approach described in the previous section. In the NPM
1174projects method, you provide ``devtool`` with an URL that points to the
1175source files.
1176
1177Replicating the same example, (i.e. ``cute-files``) use the following
1178command::
1179
1180 $ devtool add https://github.com/martinaglv/cute-files.git
1181
1182The recipe this command generates is very similar to the recipe created in
1183the previous section. However, the :term:`SRC_URI` looks like the following::
1184
1185 SRC_URI = " \
1186 git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \
1187 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
1188 "
1189
1190In this example,
1191the main module is taken from the Git repository and dependencies are
1192taken from the NPM registry. Other than those differences, the recipe is
1193basically the same between the two methods. You can build and deploy the
1194package exactly as described in the previous section that uses the
1195registry modules method.
1196
1197Adding custom metadata to packages
1198==================================
1199
1200The variable
1201:term:`PACKAGE_ADD_METADATA`
1202can be used to add additional metadata to packages. This is reflected in
1203the package control/spec file. To take the ipk format for example, the
1204CONTROL file stored inside would contain the additional metadata as
1205additional lines.
1206
1207The variable can be used in multiple ways, including using suffixes to
1208set it for a specific package type and/or package. Note that the order
1209of precedence is the same as this list:
1210
1211- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
1212
1213- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
1214
1215- ``PACKAGE_ADD_METADATA:<PN>``
1216
1217- :term:`PACKAGE_ADD_METADATA`
1218
1219`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
1220package type:
1221
1222- IPK for .ipk packages
1223
1224- DEB for .deb packages
1225
1226- RPM for .rpm packages
1227
1228`<PN>` is a parameter and expected to be a package name.
1229
1230The variable can contain multiple [one-line] metadata fields separated
1231by the literal sequence '\\n'. The separator can be redefined using the
1232variable flag ``separator``.
1233
1234Here is an example that adds two custom fields for ipk
1235packages::
1236
1237 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
1238
diff --git a/documentation/dev-manual/prebuilt-libraries.rst b/documentation/dev-manual/prebuilt-libraries.rst
new file mode 100644
index 0000000000..b80a844e93
--- /dev/null
+++ b/documentation/dev-manual/prebuilt-libraries.rst
@@ -0,0 +1,209 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working with Pre-Built Libraries
4********************************
5
6Introduction
7============
8
9Some library vendors do not release source code for their software but do
10release pre-built binaries. When shared libraries are built, they should
11be versioned (see `this article
12<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__
13for some background), but sometimes this is not done.
14
15To summarize, a versioned library must meet two conditions:
16
17#. The filename must have the version appended, for example: ``libfoo.so.1.2.3``.
18#. The library must have the ELF tag ``SONAME`` set to the major version
19 of the library, for example: ``libfoo.so.1``. You can check this by
20 running ``readelf -d filename | grep SONAME``.
21
22This section shows how to deal with both versioned and unversioned
23pre-built libraries.
24
25Versioned Libraries
26===================
27
28In this example we work with pre-built libraries for the FT4222H USB I/O chip.
29Libraries are built for several target architecture variants and packaged in
30an archive as follows::
31
32 ├── build-arm-hisiv300
33 │   └── libft4222.so.1.4.4.44
34 ├── build-arm-v5-sf
35 │   └── libft4222.so.1.4.4.44
36 ├── build-arm-v6-hf
37 │   └── libft4222.so.1.4.4.44
38 ├── build-arm-v7-hf
39 │   └── libft4222.so.1.4.4.44
40 ├── build-arm-v8
41 │   └── libft4222.so.1.4.4.44
42 ├── build-i386
43 │   └── libft4222.so.1.4.4.44
44 ├── build-i486
45 │   └── libft4222.so.1.4.4.44
46 ├── build-mips-eglibc-hf
47 │   └── libft4222.so.1.4.4.44
48 ├── build-pentium
49 │   └── libft4222.so.1.4.4.44
50 ├── build-x86_64
51 │   └── libft4222.so.1.4.4.44
52 ├── examples
53 │   ├── get-version.c
54 │   ├── i2cm.c
55 │   ├── spim.c
56 │   └── spis.c
57 ├── ftd2xx.h
58 ├── install4222.sh
59 ├── libft4222.h
60 ├── ReadMe.txt
61 └── WinTypes.h
62
63To write a recipe to use such a library in your system:
64
65- The vendor will probably have a proprietary licence, so set
66 :term:`LICENSE_FLAGS` in your recipe.
67- The vendor provides a tarball containing libraries so set :term:`SRC_URI`
68 appropriately.
69- Set :term:`COMPATIBLE_HOST` so that the recipe cannot be used with an
70 unsupported architecture. In the following example, we only support the 32
71 and 64 bit variants of the ``x86`` architecture.
72- As the vendor provides versioned libraries, we can use ``oe_soinstall``
73 from :ref:`ref-classes-utils` to install the shared library and create
74 symbolic links. If the vendor does not do this, we need to follow the
75 non-versioned library guidelines in the next section.
76- As the vendor likely used :term:`LDFLAGS` different from those in your Yocto
77 Project build, disable the corresponding checks by adding ``ldflags``
78 to :term:`INSANE_SKIP`.
79- The vendor will typically ship release builds without debugging symbols.
80 Avoid errors by preventing the packaging task from stripping out the symbols
81 and adding them to a separate debug package. This is done by setting the
82 ``INHIBIT_`` flags shown below.
83
84The complete recipe would look like this::
85
86 SUMMARY = "FTDI FT4222H Library"
87 SECTION = "libs"
88 LICENSE_FLAGS = "ftdi"
89 LICENSE = "CLOSED"
90
91 COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
92
93 # Sources available in a .tgz file in .zip archive
94 # at https://ftdichip.com/wp-content/uploads/2021/01/libft4222-linux-1.4.4.44.zip
95 # Found on https://ftdichip.com/software-examples/ft4222h-software-examples/
96 # Since dealing with this particular type of archive is out of topic here,
97 # we use a local link.
98 SRC_URI = "file://libft4222-linux-${PV}.tgz"
99
100 S = "${WORKDIR}"
101
102 ARCH_DIR:x86-64 = "build-x86_64"
103 ARCH_DIR:i586 = "build-i386"
104 ARCH_DIR:i686 = "build-i386"
105
106 INSANE_SKIP:${PN} = "ldflags"
107 INHIBIT_PACKAGE_STRIP = "1"
108 INHIBIT_SYSROOT_STRIP = "1"
109 INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
110
111 do_install () {
112 install -m 0755 -d ${D}${libdir}
113 oe_soinstall ${S}/${ARCH_DIR}/libft4222.so.${PV} ${D}${libdir}
114 install -d ${D}${includedir}
115 install -m 0755 ${S}/*.h ${D}${includedir}
116 }
117
118If the precompiled binaries are not statically linked and have dependencies on
119other libraries, then by adding those libraries to :term:`DEPENDS`, the linking
120can be examined and the appropriate :term:`RDEPENDS` automatically added.
121
122Non-Versioned Libraries
123=======================
124
125Some Background
126---------------
127
128Libraries in Linux systems are generally versioned so that it is possible
129to have multiple versions of the same library installed, which eases upgrades
130and support for older software. For example, suppose that in a versioned
131library, an actual library is called ``libfoo.so.1.2``, a symbolic link named
132``libfoo.so.1`` points to ``libfoo.so.1.2``, and a symbolic link named
133``libfoo.so`` points to ``libfoo.so.1.2``. Given these conditions, when you
134link a binary against a library, you typically provide the unversioned file
135name (i.e. ``-lfoo`` to the linker). However, the linker follows the symbolic
136link and actually links against the versioned filename. The unversioned symbolic
137link is only used at development time. Consequently, the library is packaged
138along with the headers in the development package ``${PN}-dev`` along with the
139actual library and versioned symbolic links in ``${PN}``. Because versioned
140libraries are far more common than unversioned libraries, the default packaging
141rules assume versioned libraries.
142
143Yocto Library Packaging Overview
144--------------------------------
145
146It follows that packaging an unversioned library requires a bit of work in the
147recipe. By default, ``libfoo.so`` gets packaged into ``${PN}-dev``, which
148triggers a QA warning that a non-symlink library is in a ``-dev`` package,
149and binaries in the same recipe link to the library in ``${PN}-dev``,
150which triggers more QA warnings. To solve this problem, you need to package the
151unversioned library into ``${PN}`` where it belongs. The following are the abridged
152default :term:`FILES` variables in ``bitbake.conf``::
153
154 SOLIBS = ".so.*"
155 SOLIBSDEV = ".so"
156 FILES:${PN} = "... ${libdir}/lib*${SOLIBS} ..."
157 FILES_SOLIBSDEV ?= "... ${libdir}/lib*${SOLIBSDEV} ..."
158 FILES:${PN}-dev = "... ${FILES_SOLIBSDEV} ..."
159
160:term:`SOLIBS` defines a pattern that matches real shared object libraries.
161:term:`SOLIBSDEV` matches the development form (unversioned symlink). These two
162variables are then used in ``FILES:${PN}`` and ``FILES:${PN}-dev``, which puts
163the real libraries into ``${PN}`` and the unversioned symbolic link into ``${PN}-dev``.
164To package unversioned libraries, you need to modify the variables in the recipe
165as follows::
166
167 SOLIBS = ".so"
168 FILES_SOLIBSDEV = ""
169
170The modifications cause the ``.so`` file to be the real library
171and unset :term:`FILES_SOLIBSDEV` so that no libraries get packaged into
172``${PN}-dev``. The changes are required because unless :term:`PACKAGES` is changed,
173``${PN}-dev`` collects files before `${PN}`. ``${PN}-dev`` must not collect any of
174the files you want in ``${PN}``.
175
176Finally, loadable modules, essentially unversioned libraries that are linked
177at runtime using ``dlopen()`` instead of at build time, should generally be
178installed in a private directory. However, if they are installed in ``${libdir}``,
179then the modules can be treated as unversioned libraries.
180
181Example
182-------
183
184The example below installs an unversioned x86-64 pre-built library named
185``libfoo.so``. The :term:`COMPATIBLE_HOST` variable limits recipes to the
186x86-64 architecture while the :term:`INSANE_SKIP`, :term:`INHIBIT_PACKAGE_STRIP`
187and :term:`INHIBIT_SYSROOT_STRIP` variables are all set as in the above
188versioned library example. The "magic" is setting the :term:`SOLIBS` and
189:term:`FILES_SOLIBSDEV` variables as explained above::
190
191 SUMMARY = "libfoo sample recipe"
192 SECTION = "libs"
193 LICENSE = "CLOSED"
194
195 SRC_URI = "file://libfoo.so"
196
197 COMPATIBLE_HOST = "x86_64.*-linux"
198
199 INSANE_SKIP:${PN} = "ldflags"
200 INHIBIT_PACKAGE_STRIP = "1"
201 INHIBIT_SYSROOT_STRIP = "1"
202 SOLIBS = ".so"
203 FILES_SOLIBSDEV = ""
204
205 do_install () {
206 install -d ${D}${libdir}
207 install -m 0755 ${WORKDIR}/libfoo.so ${D}${libdir}
208 }
209
diff --git a/documentation/dev-manual/python-development-shell.rst b/documentation/dev-manual/python-development-shell.rst
new file mode 100644
index 0000000000..2dc6a3f138
--- /dev/null
+++ b/documentation/dev-manual/python-development-shell.rst
@@ -0,0 +1,50 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using a Python Development Shell
4********************************
5
6Similar to working within a development shell as described in the
7previous section, you can also spawn and work within an interactive
8Python development shell. When debugging certain commands or even when
9just editing packages, ``pydevshell`` can be a useful tool. When you
10invoke the ``pydevshell`` task, all tasks up to and including
11:ref:`ref-tasks-patch` are run for the
12specified target. Then a new terminal is opened. Additionally, key
13Python objects and code are available in the same way they are to
14BitBake tasks, in particular, the data store 'd'. So, commands such as
15the following are useful when exploring the data store and running
16functions::
17
18 pydevshell> d.getVar("STAGING_DIR")
19 '/media/build1/poky/build/tmp/sysroots'
20 pydevshell> d.getVar("STAGING_DIR", False)
21 '${TMPDIR}/sysroots'
22 pydevshell> d.setVar("FOO", "bar")
23 pydevshell> d.getVar("FOO")
24 'bar'
25 pydevshell> d.delVar("FOO")
26 pydevshell> d.getVar("FOO")
27 pydevshell> bb.build.exec_func("do_unpack", d)
28 pydevshell>
29
30See the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:functions you can call from within python`"
31section in the BitBake User Manual for details about available functions.
32
33The commands execute just as if the OpenEmbedded build
34system were executing them. Consequently, working this way can be
35helpful when debugging a build or preparing software to be used with the
36OpenEmbedded build system.
37
38Following is an example that uses ``pydevshell`` on a target named
39``matchbox-desktop``::
40
41 $ bitbake matchbox-desktop -c pydevshell
42
43This command spawns a terminal and places you in an interactive Python
44interpreter within the OpenEmbedded build environment. The
45:term:`OE_TERMINAL` variable
46controls what type of shell is opened.
47
48When you are finished using ``pydevshell``, you can exit the shell
49either by using Ctrl+d or closing the terminal window.
50
diff --git a/documentation/dev-manual/quilt.rst b/documentation/dev-manual/quilt.rst
new file mode 100644
index 0000000000..59240705ad
--- /dev/null
+++ b/documentation/dev-manual/quilt.rst
@@ -0,0 +1,89 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using Quilt in Your Workflow
4****************************
5
6`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
7that allows you to capture source code changes without having a clean
8source tree. This section outlines the typical workflow you can use to
9modify source code, test changes, and then preserve the changes in the
10form of a patch all using Quilt.
11
12.. note::
13
14 With regard to preserving changes to source files, if you clean a
15 recipe or have :ref:`ref-classes-rm-work` enabled, the
16 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
17 as described in the Yocto Project Application Development and the
18 Extensible Software Development Kit (eSDK) manual is a safer
19 development flow than the flow that uses Quilt.
20
21Follow these general steps:
22
23#. *Find the Source Code:* Temporary source code used by the
24 OpenEmbedded build system is kept in the :term:`Build Directory`. See the
25 ":ref:`dev-manual/temporary-source-code:finding temporary source code`" section to
26 learn how to locate the directory that has the temporary source code for a
27 particular package.
28
29#. *Change Your Working Directory:* You need to be in the directory that
30 has the temporary source code. That directory is defined by the
31 :term:`S` variable.
32
33#. *Create a New Patch:* Before modifying source code, you need to
34 create a new patch. To create a new patch file, use ``quilt new`` as
35 below::
36
37 $ quilt new my_changes.patch
38
39#. *Notify Quilt and Add Files:* After creating the patch, you need to
40 notify Quilt about the files you plan to edit. You notify Quilt by
41 adding the files to the patch you just created::
42
43 $ quilt add file1.c file2.c file3.c
44
45#. *Edit the Files:* Make your changes in the source code to the files
46 you added to the patch.
47
48#. *Test Your Changes:* Once you have modified the source code, the
49 easiest way to test your changes is by calling the :ref:`ref-tasks-compile`
50 task as shown in the following example::
51
52 $ bitbake -c compile -f package
53
54 The ``-f`` or ``--force`` option forces the specified task to
55 execute. If you find problems with your code, you can just keep
56 editing and re-testing iteratively until things work as expected.
57
58 .. note::
59
60 All the modifications you make to the temporary source code disappear
61 once you run the :ref:`ref-tasks-clean` or :ref:`ref-tasks-cleanall`
62 tasks using BitBake (i.e. ``bitbake -c clean package`` and
63 ``bitbake -c cleanall package``). Modifications will also disappear if
64 you use the :ref:`ref-classes-rm-work` feature as described in
65 the ":ref:`dev-manual/disk-space:conserving disk space during builds`"
66 section.
67
68#. *Generate the Patch:* Once your changes work as expected, you need to
69 use Quilt to generate the final patch that contains all your
70 modifications::
71
72 $ quilt refresh
73
74 At this point, the
75 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
76 ``file2.c``, and ``file3.c`` files.
77
78 You can find the resulting patch file in the ``patches/``
79 subdirectory of the source (:term:`S`) directory.
80
81#. *Copy the Patch File:* For simplicity, copy the patch file into a
82 directory named ``files``, which you can create in the same directory
83 that holds the recipe (``.bb``) file or the append (``.bbappend``)
84 file. Placing the patch here guarantees that the OpenEmbedded build
85 system will find the patch. Next, add the patch into the :term:`SRC_URI`
86 of the recipe. Here is an example::
87
88 SRC_URI += "file://my_changes.patch"
89
diff --git a/documentation/dev-manual/read-only-rootfs.rst b/documentation/dev-manual/read-only-rootfs.rst
new file mode 100644
index 0000000000..251178ed54
--- /dev/null
+++ b/documentation/dev-manual/read-only-rootfs.rst
@@ -0,0 +1,89 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Read-Only Root Filesystem
4************************************
5
6Suppose, for security reasons, you need to disable your target device's
7root filesystem's write permissions (i.e. you need a read-only root
8filesystem). Or, perhaps you are running the device's operating system
9from a read-only storage device. For either case, you can customize your
10image for that behavior.
11
12.. note::
13
14 Supporting a read-only root filesystem requires that the system and
15 applications do not try to write to the root filesystem. You must
16 configure all parts of the target system to write elsewhere, or to
17 gracefully fail in the event of attempting to write to the root
18 filesystem.
19
20Creating the Root Filesystem
21============================
22
23To create the read-only root filesystem, simply add the
24"read-only-rootfs" feature to your image, normally in one of two ways.
25The first way is to add the "read-only-rootfs" image feature in the
26image's recipe file via the :term:`IMAGE_FEATURES` variable::
27
28 IMAGE_FEATURES += "read-only-rootfs"
29
30As an alternative, you can add the same feature
31from within your :term:`Build Directory`'s ``local.conf`` file with the
32associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
33
34 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
35
36For more information on how to use these variables, see the
37":ref:`dev-manual/customizing-images:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
38section. For information on the variables, see
39:term:`IMAGE_FEATURES` and
40:term:`EXTRA_IMAGE_FEATURES`.
41
42Post-Installation Scripts and Read-Only Root Filesystem
43=======================================================
44
45It is very important that you make sure all post-Installation
46(``pkg_postinst``) scripts for packages that are installed into the
47image can be run at the time when the root filesystem is created during
48the build on the host system. These scripts cannot attempt to run during
49the first boot on the target device. With the "read-only-rootfs" feature
50enabled, the build system makes sure that all post-installation scripts
51succeed at file system creation time. If any of these scripts
52still need to be run after the root filesystem is created, the build
53immediately fails. These build-time checks ensure that the build fails
54rather than the target device fails later during its initial boot
55operation.
56
57Most of the common post-installation scripts generated by the build
58system for the out-of-the-box Yocto Project are engineered so that they
59can run during root filesystem creation (e.g. post-installation scripts
60for caching fonts). However, if you create and add custom scripts, you
61need to be sure they can be run during this file system creation.
62
63Here are some common problems that prevent post-installation scripts
64from running during root filesystem creation:
65
66- *Not using $D in front of absolute paths:* The build system defines
67 ``$``\ :term:`D` when the root
68 filesystem is created. Furthermore, ``$D`` is blank when the script
69 is run on the target device. This implies two purposes for ``$D``:
70 ensuring paths are valid in both the host and target environments,
71 and checking to determine which environment is being used as a method
72 for taking appropriate actions.
73
74- *Attempting to run processes that are specific to or dependent on the
75 target architecture:* You can work around these attempts by using
76 native tools, which run on the host system, to accomplish the same
77 tasks, or by alternatively running the processes under QEMU, which
78 has the ``qemu_run_binary`` function. For more information, see the
79 :ref:`ref-classes-qemu` class.
80
81Areas With Write Access
82=======================
83
84With the "read-only-rootfs" feature enabled, any attempt by the target
85to write to the root filesystem at runtime fails. Consequently, you must
86make sure that you configure processes and applications that attempt
87these types of writes do so to directories with write access (e.g.
88``/tmp`` or ``/var/run``).
89
diff --git a/documentation/dev-manual/runtime-testing.rst b/documentation/dev-manual/runtime-testing.rst
new file mode 100644
index 0000000000..a5d8502fbb
--- /dev/null
+++ b/documentation/dev-manual/runtime-testing.rst
@@ -0,0 +1,598 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Performing Automated Runtime Testing
4************************************
5
6The OpenEmbedded build system makes available a series of automated
7tests for images to verify runtime functionality. You can run these
8tests on either QEMU or actual target hardware. Tests are written in
9Python making use of the ``unittest`` module, and the majority of them
10run commands on the target system over SSH. This section describes how
11you set up the environment to use these tests, run available tests, and
12write and add your own tests.
13
14For information on the test and QA infrastructure available within the
15Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
16section in the Yocto Project Reference Manual.
17
18Enabling Tests
19==============
20
21Depending on whether you are planning to run tests using QEMU or on the
22hardware, you have to take different steps to enable the tests. See the
23following subsections for information on how to enable both types of
24tests.
25
26Enabling Runtime Tests on QEMU
27------------------------------
28
29In order to run tests, you need to do the following:
30
31- *Set up to avoid interaction with sudo for networking:* To
32 accomplish this, you must do one of the following:
33
34 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
35 commands or just for ``runqemu-ifup``. You must provide the full
36 path as that can change if you are using multiple clones of the
37 source repository.
38
39 .. note::
40
41 On some distributions, you also need to comment out "Defaults
42 requiretty" in ``/etc/sudoers``.
43
44 - Manually configure a tap interface for your system.
45
46 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
47 should generate a list of tap devices. This is the option
48 typically chosen for Autobuilder-type environments.
49
50 .. note::
51
52 - Be sure to use an absolute path when calling this script
53 with sudo.
54
55 - The package recipe ``qemu-helper-native`` is required to run
56 this script. Build the package using the following command::
57
58 $ bitbake qemu-helper-native
59
60- *Set the DISPLAY variable:* You need to set this variable so that
61 you have an X server available (e.g. start ``vncserver`` for a
62 headless machine).
63
64- *Be sure your host's firewall accepts incoming connections from
65 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
66 HTTP server on a random high number port, which is used to serve
67 files to the target. The DNF module serves
68 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
69 That means your host's firewall must accept incoming connections from
70 192.168.7.0/24, which is the default IP range used for tap devices by
71 ``runqemu``.
72
73- *Be sure your host has the correct packages installed:* Depending
74 your host's distribution, you need to have the following packages
75 installed:
76
77 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
78
79 - openSUSE: ``sysstat`` and ``iproute2``
80
81 - Fedora: ``sysstat`` and ``iproute``
82
83 - CentOS: ``sysstat`` and ``iproute``
84
85Once you start running the tests, the following happens:
86
87#. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
88
89#. The image is booted under QEMU using the standard ``runqemu`` script.
90
91#. A default timeout of 500 seconds occurs to allow for the boot process
92 to reach the login prompt. You can change the timeout period by
93 setting
94 :term:`TEST_QEMUBOOT_TIMEOUT`
95 in the ``local.conf`` file.
96
97#. Once the boot process is reached and the login prompt appears, the
98 tests run. The full boot log is written to
99 ``${WORKDIR}/testimage/qemu_boot_log``.
100
101#. Each test module loads in the order found in :term:`TEST_SUITES`. You can
102 find the full output of the commands run over SSH in
103 ``${WORKDIR}/testimgage/ssh_target_log``.
104
105#. If no failures occur, the task running the tests ends successfully.
106 You can find the output from the ``unittest`` in the task log at
107 ``${WORKDIR}/temp/log.do_testimage``.
108
109Enabling Runtime Tests on Hardware
110----------------------------------
111
112The OpenEmbedded build system can run tests on real hardware, and for
113certain devices it can also deploy the image to be tested onto the
114device beforehand.
115
116For automated deployment, a "controller image" is installed onto the
117hardware once as part of setup. Then, each time tests are to be run, the
118following occurs:
119
120#. The controller image is booted into and used to write the image to be
121 tested to a second partition.
122
123#. The device is then rebooted using an external script that you need to
124 provide.
125
126#. The device boots into the image to be tested.
127
128When running tests (independent of whether the image has been deployed
129automatically or not), the device is expected to be connected to a
130network on a pre-determined IP address. You can either use static IP
131addresses written into the image, or set the image to use DHCP and have
132your DHCP server on the test network assign a known IP address based on
133the MAC address of the device.
134
135In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
136appropriate value. For QEMU, you do not have to change anything, the
137default value is "qemu". For running tests on hardware, the following
138options are available:
139
140- *"simpleremote":* Choose "simpleremote" if you are going to run tests
141 on a target system that is already running the image to be tested and
142 is available on the network. You can use "simpleremote" in
143 conjunction with either real hardware or an image running within a
144 separately started QEMU or any other virtual machine manager.
145
146- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
147 an EFI-based machine with ``systemd-boot`` as bootloader and
148 ``core-image-testmaster`` (or something similar) is installed. Also,
149 your hardware under test must be in a DHCP-enabled network that gives
150 it the same IP address for each reboot.
151
152 If you choose "SystemdbootTarget", there are additional requirements
153 and considerations. See the
154 ":ref:`dev-manual/runtime-testing:selecting systemdboottarget`" section, which
155 follows, for more information.
156
157- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
158 images and running tests on the BeagleBone "Black" or original
159 "White" hardware. For information on how to use these tests, see the
160 comments at the top of the BeagleBoneTarget
161 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
162
163- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
164 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
165 For information on how to use these tests, see the comments at the
166 top of the EdgeRouterTarget
167 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
168
169- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
170 tests on any generic PC that boots using GRUB. For information on how
171 to use these tests, see the comments at the top of the GrubTarget
172 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
173
174- *"your-target":* Create your own custom target if you want to run
175 tests when you are deploying images and running tests on a custom
176 machine within your BSP layer. To do this, you need to add a Python
177 unit that defines the target class under ``lib/oeqa/controllers/``
178 within your layer. You must also provide an empty ``__init__.py``.
179 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
180
181Selecting SystemdbootTarget
182---------------------------
183
184If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
185not need any information in this section. You can skip down to the
186":ref:`dev-manual/runtime-testing:running tests`" section.
187
188If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
189perform a one-time setup of your controller image by doing the following:
190
191#. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
192
193 EFI_PROVIDER = "systemd-boot"
194
195#. *Build the controller image:* Build the ``core-image-testmaster`` image.
196 The ``core-image-testmaster`` recipe is provided as an example for a
197 "controller" image and you can customize the image recipe as you would
198 any other recipe.
199
200 Here are the image recipe requirements:
201
202 - Inherits ``core-image`` so that kernel modules are installed.
203
204 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
205 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
206
207 - Uses a custom :term:`Initramfs` image with a custom
208 installer. A normal image that you can install usually creates a
209 single root filesystem partition. This image uses another installer that
210 creates a specific partition layout. Not all Board Support
211 Packages (BSPs) can use an installer. For such cases, you need to
212 manually create the following partition layout on the target:
213
214 - First partition mounted under ``/boot``, labeled "boot".
215
216 - The main root filesystem partition where this image gets installed,
217 which is mounted under ``/``.
218
219 - Another partition labeled "testrootfs" where test images get
220 deployed.
221
222#. *Install image:* Install the image that you just built on the target
223 system.
224
225The final thing you need to do when setting :term:`TEST_TARGET` to
226"SystemdbootTarget" is to set up the test image:
227
228#. *Set up your local.conf file:* Make sure you have the following
229 statements in your ``local.conf`` file::
230
231 IMAGE_FSTYPES += "tar.gz"
232 INHERIT += "testimage"
233 TEST_TARGET = "SystemdbootTarget"
234 TEST_TARGET_IP = "192.168.2.3"
235
236#. *Build your test image:* Use BitBake to build the image::
237
238 $ bitbake core-image-sato
239
240Power Control
241-------------
242
243For most hardware targets other than "simpleremote", you can control
244power:
245
246- You can use :term:`TEST_POWERCONTROL_CMD` together with
247 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
248 and does power cycling. The test code passes one argument to that
249 command: off, on or cycle (off then on). Here is an example that
250 could appear in your ``local.conf`` file::
251
252 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
253
254 In this example, the expect
255 script does the following:
256
257 .. code-block:: shell
258
259 ssh test@10.11.12.1 "pyctl nuc1 arg"
260
261 It then runs a Python script that controls power for a label called
262 ``nuc1``.
263
264 .. note::
265
266 You need to customize :term:`TEST_POWERCONTROL_CMD` and
267 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
268 is that it accepts "on", "off", and "cycle" as the last argument.
269
270- When no command is defined, it connects to the device over SSH and
271 uses the classic reboot command to reboot the device. Classic reboot
272 is fine as long as the machine actually reboots (i.e. the SSH test
273 has not failed). It is useful for scenarios where you have a simple
274 setup, typically with a single board, and where some manual
275 interaction is okay from time to time.
276
277If you have no hardware to automatically perform power control but still
278wish to experiment with automated hardware testing, you can use the
279``dialog-power-control`` script that shows a dialog prompting you to perform
280the required power action. This script requires either KDialog or Zenity
281to be installed. To use this script, set the
282:term:`TEST_POWERCONTROL_CMD`
283variable as follows::
284
285 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
286
287Serial Console Connection
288-------------------------
289
290For test target classes requiring a serial console to interact with the
291bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
292you need to specify a command to use to connect to the serial console of
293the target machine by using the
294:term:`TEST_SERIALCONTROL_CMD`
295variable and optionally the
296:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
297variable.
298
299These cases could be a serial terminal program if the machine is
300connected to a local serial port, or a ``telnet`` or ``ssh`` command
301connecting to a remote console server. Regardless of the case, the
302command simply needs to connect to the serial console and forward that
303connection to standard input and output as any normal terminal program
304does. For example, to use the picocom terminal program on serial device
305``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
306
307 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
308
309For local
310devices where the serial port device disappears when the device reboots,
311an additional "serdevtry" wrapper script is provided. To use this
312wrapper, simply prefix the terminal command with
313``${COREBASE}/scripts/contrib/serdevtry``::
314
315 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
316
317Running Tests
318=============
319
320You can start the tests automatically or manually:
321
322- *Automatically running tests:* To run the tests automatically after the
323 OpenEmbedded build system successfully creates an image, first set the
324 :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the
325 :term:`Build Directory`::
326
327 TESTIMAGE_AUTO = "1"
328
329 Next, build your image. If the image successfully builds, the
330 tests run::
331
332 bitbake core-image-sato
333
334- *Manually running tests:* To manually run the tests, first globally
335 inherit the :ref:`ref-classes-testimage*` class by editing your
336 ``local.conf`` file::
337
338 INHERIT += "testimage"
339
340 Next, use BitBake to run the tests::
341
342 bitbake -c testimage image
343
344All test files reside in ``meta/lib/oeqa/runtime/cases`` in the
345:term:`Source Directory`. A test name maps
346directly to a Python module. Each test module may contain a number of
347individual tests. Tests are usually grouped together by the area tested
348(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/cases/systemd.py``).
349
350You can add tests to any layer provided you place them in the proper
351area and you extend :term:`BBPATH` in
352the ``local.conf`` file as normal. Be sure that tests reside in
353``layer/lib/oeqa/runtime/cases``.
354
355.. note::
356
357 Be sure that module names do not collide with module names used in
358 the default set of test modules in ``meta/lib/oeqa/runtime/cases``.
359
360You can change the set of tests run by appending or overriding
361:term:`TEST_SUITES` variable in
362``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
363for the image. Test modules named within :term:`TEST_SUITES` cannot be
364skipped even if a test is not suitable for an image (e.g. running the
365RPM tests on an image without ``rpm``). Appending "auto" to
366:term:`TEST_SUITES` causes the build system to try to run all tests that are
367suitable for the image (i.e. each test module may elect to skip itself).
368
369The order you list tests in :term:`TEST_SUITES` is important and influences
370test dependencies. Consequently, tests that depend on other tests should
371be added after the test on which they depend. For example, since the
372``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
373"ping" in the list. The test class provides no re-ordering or dependency
374handling.
375
376.. note::
377
378 Each module can have multiple classes with multiple test methods.
379 And, Python ``unittest`` rules apply.
380
381Here are some things to keep in mind when running tests:
382
383- The default tests for the image are defined as::
384
385 DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
386
387- Add your own test to the list of the by using the following::
388
389 TEST_SUITES:append = " mytest"
390
391- Run a specific list of tests as follows::
392
393 TEST_SUITES = "test1 test2 test3"
394
395 Remember, order is important. Be sure to place a test that is
396 dependent on another test later in the order.
397
398Exporting Tests
399===============
400
401You can export tests so that they can run independently of the build
402system. Exporting tests is required if you want to be able to hand the
403test execution off to a scheduler. You can only export tests that are
404defined in :term:`TEST_SUITES`.
405
406If your image is already built, make sure the following are set in your
407``local.conf`` file::
408
409 INHERIT += "testexport"
410 TEST_TARGET_IP = "IP-address-for-the-test-target"
411 TEST_SERVER_IP = "IP-address-for-the-test-server"
412
413You can then export the tests with the
414following BitBake command form::
415
416 $ bitbake image -c testexport
417
418Exporting the tests places them in the :term:`Build Directory` in
419``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
420variable.
421
422You can now run the tests outside of the build environment::
423
424 $ cd tmp/testexport/image
425 $ ./runexported.py testdata.json
426
427Here is a complete example that shows IP addresses and uses the
428``core-image-sato`` image::
429
430 INHERIT += "testexport"
431 TEST_TARGET_IP = "192.168.7.2"
432 TEST_SERVER_IP = "192.168.7.1"
433
434Use BitBake to export the tests::
435
436 $ bitbake core-image-sato -c testexport
437
438Run the tests outside of
439the build environment using the following::
440
441 $ cd tmp/testexport/core-image-sato
442 $ ./runexported.py testdata.json
443
444Writing New Tests
445=================
446
447As mentioned previously, all new test files need to be in the proper
448place for the build system to find them. New tests for additional
449functionality outside of the core should be added to the layer that adds
450the functionality, in ``layer/lib/oeqa/runtime/cases`` (as long as
451:term:`BBPATH` is extended in the
452layer's ``layer.conf`` file as normal). Just remember the following:
453
454- Filenames need to map directly to test (module) names.
455
456- Do not use module names that collide with existing core tests.
457
458- Minimally, an empty ``__init__.py`` file must be present in the runtime
459 directory.
460
461To create a new test, start by copying an existing module (e.g.
462``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
463code from ``meta/lib/oeqa/utils``, which are helper classes.
464
465.. note::
466
467 Structure shell commands such that you rely on them and they return a
468 single code for success. Be aware that sometimes you will need to
469 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
470
471You will notice that all test classes inherit ``oeRuntimeTest``, which
472is found in ``meta/lib/oetest.py``. This base class offers some helper
473attributes, which are described in the following sections:
474
475Class Methods
476-------------
477
478Class methods are as follows:
479
480- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
481 package list of the image, which is based on the manifest file that
482 is generated during the :ref:`ref-tasks-rootfs` task.
483
484- *hasFeature(feature):* Returns "True" if the feature is in
485 :term:`IMAGE_FEATURES` or
486 :term:`DISTRO_FEATURES`.
487
488Class Attributes
489----------------
490
491Class attributes are as follows:
492
493- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
494 Otherwise, ``pscmd`` equals "ps" (busybox).
495
496- *tc:* The called test context, which gives access to the
497 following attributes:
498
499 - *d:* The BitBake datastore, which allows you to use stuff such
500 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
501
502 - *testslist and testsrequired:* Used internally. The tests
503 do not need these.
504
505 - *filesdir:* The absolute path to
506 ``meta/lib/oeqa/runtime/files``, which contains helper files for
507 tests meant for copying on the target such as small files written
508 in C for compilation.
509
510 - *target:* The target controller object used to deploy and
511 start an image on a particular target (e.g. Qemu, SimpleRemote,
512 and SystemdbootTarget). Tests usually use the following:
513
514 - *ip:* The target's IP address.
515
516 - *server_ip:* The host's IP address, which is usually used
517 by the DNF test suite.
518
519 - *run(cmd, timeout=None):* The single, most used method.
520 This command is a wrapper for: ``ssh root@host "cmd"``. The
521 command returns a tuple: (status, output), which are what their
522 names imply - the return code of "cmd" and whatever output it
523 produces. The optional timeout argument represents the number
524 of seconds the test should wait for "cmd" to return. If the
525 argument is "None", the test uses the default instance's
526 timeout period, which is 300 seconds. If the argument is "0",
527 the test runs until the command returns.
528
529 - *copy_to(localpath, remotepath):*
530 ``scp localpath root@ip:remotepath``.
531
532 - *copy_from(remotepath, localpath):*
533 ``scp root@host:remotepath localpath``.
534
535Instance Attributes
536-------------------
537
538There is a single instance attribute, which is ``target``. The ``target``
539instance attribute is identical to the class attribute of the same name,
540which is described in the previous section. This attribute exists as
541both an instance and class attribute so tests can use
542``self.target.run(cmd)`` in instance methods instead of
543``oeRuntimeTest.tc.target.run(cmd)``.
544
545Installing Packages in the DUT Without the Package Manager
546==========================================================
547
548When a test requires a package built by BitBake, it is possible to
549install that package. Installing the package does not require a package
550manager be installed in the device under test (DUT). It does, however,
551require an SSH connection and the target must be using the
552``sshcontrol`` class.
553
554.. note::
555
556 This method uses ``scp`` to copy files from the host to the target, which
557 causes permissions and special attributes to be lost.
558
559A JSON file is used to define the packages needed by a test. This file
560must be in the same path as the file used to define the tests.
561Furthermore, the filename must map directly to the test module name with
562a ``.json`` extension.
563
564The JSON file must include an object with the test name as keys of an
565object or an array. This object (or array of objects) uses the following
566data:
567
568- "pkg" --- a mandatory string that is the name of the package to be
569 installed.
570
571- "rm" --- an optional boolean, which defaults to "false", that specifies
572 to remove the package after the test.
573
574- "extract" --- an optional boolean, which defaults to "false", that
575 specifies if the package must be extracted from the package format.
576 When set to "true", the package is not automatically installed into
577 the DUT.
578
579Following is an example JSON file that handles test "foo" installing
580package "bar" and test "foobar" installing packages "foo" and "bar".
581Once the test is complete, the packages are removed from the DUT::
582
583 {
584 "foo": {
585 "pkg": "bar"
586 },
587 "foobar": [
588 {
589 "pkg": "foo",
590 "rm": true
591 },
592 {
593 "pkg": "bar",
594 "rm": true
595 }
596 ]
597 }
598
diff --git a/documentation/dev-manual/sbom.rst b/documentation/dev-manual/sbom.rst
new file mode 100644
index 0000000000..448c071c55
--- /dev/null
+++ b/documentation/dev-manual/sbom.rst
@@ -0,0 +1,72 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Software Bill of Materials
4*************************************
5
6Once you are able to build an image for your project, once the licenses for
7each software component are all identified (see
8":ref:`dev-manual/licenses:working with licenses`") and once vulnerability
9fixes are applied (see ":ref:`dev-manual/vulnerabilities:checking
10for vulnerabilities`"), the OpenEmbedded build system can generate
11a description of all the components you used, their licenses, their dependencies,
12their sources, the changes that were applied to them and the known
13vulnerabilities that were fixed.
14
15This description is generated in the form of a *Software Bill of Materials*
16(:term:`SBOM`), using the :term:`SPDX` standard.
17
18When you release software, this is the most standard way to provide information
19about the Software Supply Chain of your software image and SDK. The
20:term:`SBOM` tooling is often used to ensure open source license compliance by
21providing the license texts used in the product which legal departments and end
22users can read in standardized format.
23
24:term:`SBOM` information is also critical to performing vulnerability exposure
25assessments, as all the components used in the Software Supply Chain are listed.
26
27The OpenEmbedded build system doesn't generate such information by default.
28To make this happen, you must inherit the
29:ref:`ref-classes-create-spdx` class from a configuration file::
30
31 INHERIT += "create-spdx"
32
33You then get :term:`SPDX` output in JSON format as an
34``IMAGE-MACHINE.spdx.json`` file in ``tmp/deploy/images/MACHINE/`` inside the
35:term:`Build Directory`.
36
37This is a toplevel file accompanied by an ``IMAGE-MACHINE.spdx.index.json``
38containing an index of JSON :term:`SPDX` files for individual recipes, together
39with an ``IMAGE-MACHINE.spdx.tar.zst`` compressed archive containing all such
40files.
41
42The :ref:`ref-classes-create-spdx` class offers options to include
43more information in the output :term:`SPDX` data, such as making the generated
44files more human readable (:term:`SPDX_PRETTY`), adding compressed archives of
45the files in the generated target packages (:term:`SPDX_ARCHIVE_PACKAGED`),
46adding a description of the source files used to generate host tools and target
47packages (:term:`SPDX_INCLUDE_SOURCES`) and adding archives of these source
48files themselves (:term:`SPDX_ARCHIVE_SOURCES`).
49
50Though the toplevel :term:`SPDX` output is available in
51``tmp/deploy/images/MACHINE/`` inside the :term:`Build Directory`, ancillary
52generated files are available in ``tmp/deploy/spdx/MACHINE`` too, such as:
53
54- The individual :term:`SPDX` JSON files in the ``IMAGE-MACHINE.spdx.tar.zst``
55 archive.
56
57- Compressed archives of the files in the generated target packages,
58 in ``packages/packagename.tar.zst`` (when :term:`SPDX_ARCHIVE_PACKAGED`
59 is set).
60
61- Compressed archives of the source files used to build the host tools
62 and the target packages in ``recipes/recipe-packagename.tar.zst``
63 (when :term:`SPDX_ARCHIVE_SOURCES` is set). Those are needed to fulfill
64 "source code access" license requirements.
65
66See the `tools page <https://spdx.dev/resources/tools/>`__ on the :term:`SPDX`
67project website for a list of tools to consume and transform the :term:`SPDX`
68data generated by the OpenEmbedded build system.
69
70See also Joshua Watt's
71`Automated SBoM generation with OpenEmbedded and the Yocto Project <https://youtu.be/Q5UQUM6zxVU>`__
72presentation at FOSDEM 2023.
diff --git a/documentation/dev-manual/securing-images.rst b/documentation/dev-manual/securing-images.rst
new file mode 100644
index 0000000000..e5791d3d6d
--- /dev/null
+++ b/documentation/dev-manual/securing-images.rst
@@ -0,0 +1,156 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Making Images More Secure
4*************************
5
6Security is of increasing concern for embedded devices. Consider the
7issues and problems discussed in just this sampling of work found across
8the Internet:
9
10- *"*\ `Security Risks of Embedded
11 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
12 by Bruce Schneier
13
14- *"*\ `Internet Census
15 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
16 Botnet
17
18- *"*\ `Security Issues for Embedded
19 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
20 by Jake Edge
21
22When securing your image is of concern, there are steps, tools, and
23variables that you can consider to help you reach the security goals you
24need for your particular device. Not all situations are identical when
25it comes to making an image secure. Consequently, this section provides
26some guidance and suggestions for consideration when you want to make
27your image more secure.
28
29.. note::
30
31 Because the security requirements and risks are different for every
32 type of device, this section cannot provide a complete reference on
33 securing your custom OS. It is strongly recommended that you also
34 consult other sources of information on embedded Linux system
35 hardening and on security.
36
37General Considerations
38======================
39
40There are general considerations that help you create more secure images.
41You should consider the following suggestions to make your device
42more secure:
43
44- Scan additional code you are adding to the system (e.g. application
45 code) by using static analysis tools. Look for buffer overflows and
46 other potential security problems.
47
48- Pay particular attention to the security for any web-based
49 administration interface.
50
51 Web interfaces typically need to perform administrative functions and
52 tend to need to run with elevated privileges. Thus, the consequences
53 resulting from the interface's security becoming compromised can be
54 serious. Look for common web vulnerabilities such as
55 cross-site-scripting (XSS), unvalidated inputs, and so forth.
56
57 As with system passwords, the default credentials for accessing a
58 web-based interface should not be the same across all devices. This
59 is particularly true if the interface is enabled by default as it can
60 be assumed that many end-users will not change the credentials.
61
62- Ensure you can update the software on the device to mitigate
63 vulnerabilities discovered in the future. This consideration
64 especially applies when your device is network-enabled.
65
66- Regularly scan and apply fixes for CVE security issues affecting
67 all software components in the product, see ":ref:`dev-manual/vulnerabilities:checking for vulnerabilities`".
68
69- Regularly update your version of Poky and OE-Core from their upstream
70 developers, e.g. to apply updates and security fixes from stable
71 and :term:`LTS` branches.
72
73- Ensure you remove or disable debugging functionality before producing
74 the final image. For information on how to do this, see the
75 ":ref:`dev-manual/securing-images:considerations specific to the openembedded build system`"
76 section.
77
78- Ensure you have no network services listening that are not needed.
79
80- Remove any software from the image that is not needed.
81
82- Enable hardware support for secure boot functionality when your
83 device supports this functionality.
84
85Security Flags
86==============
87
88The Yocto Project has security flags that you can enable that help make
89your build output more secure. The security flags are in the
90``meta/conf/distro/include/security_flags.inc`` file in your
91:term:`Source Directory` (e.g. ``poky``).
92
93.. note::
94
95 Depending on the recipe, certain security flags are enabled and
96 disabled by default.
97
98Use the following line in your ``local.conf`` file or in your custom
99distribution configuration file to enable the security compiler and
100linker flags for your build::
101
102 require conf/distro/include/security_flags.inc
103
104Considerations Specific to the OpenEmbedded Build System
105========================================================
106
107You can take some steps that are specific to the OpenEmbedded build
108system to make your images more secure:
109
110- Ensure "debug-tweaks" is not one of your selected
111 :term:`IMAGE_FEATURES`.
112 When creating a new project, the default is to provide you with an
113 initial ``local.conf`` file that enables this feature using the
114 :term:`EXTRA_IMAGE_FEATURES`
115 variable with the line::
116
117 EXTRA_IMAGE_FEATURES = "debug-tweaks"
118
119 To disable that feature, simply comment out that line in your
120 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
121 "debug-tweaks" before producing your final image. Among other things,
122 leaving this in place sets the root password as blank, which makes
123 logging in for debugging or inspection easy during development but
124 also means anyone can easily log in during production.
125
126- It is possible to set a root password for the image and also to set
127 passwords for any extra users you might add (e.g. administrative or
128 service type users). When you set up passwords for multiple images or
129 users, you should not duplicate passwords.
130
131 To set up passwords, use the :ref:`ref-classes-extrausers` class, which
132 is the preferred method. For an example on how to set up both root and
133 user passwords, see the ":ref:`ref-classes-extrausers`" section.
134
135 .. note::
136
137 When adding extra user accounts or setting a root password, be
138 cautious about setting the same password on every device. If you
139 do this, and the password you have set is exposed, then every
140 device is now potentially compromised. If you need this access but
141 want to ensure security, consider setting a different, random
142 password for each device. Typically, you do this as a separate
143 step after you deploy the image onto the device.
144
145- Consider enabling a Mandatory Access Control (MAC) framework such as
146 SMACK or SELinux and tuning it appropriately for your device's usage.
147 You can find more information in the
148 :yocto_git:`meta-selinux </meta-selinux/>` layer.
149
150Tools for Hardening Your Image
151==============================
152
153The Yocto Project provides tools for making your image more secure. You
154can find these tools in the ``meta-security`` layer of the
155:yocto_git:`Yocto Project Source Repositories <>`.
156
diff --git a/documentation/dev-manual/speeding-up-build.rst b/documentation/dev-manual/speeding-up-build.rst
new file mode 100644
index 0000000000..31b6f75ab0
--- /dev/null
+++ b/documentation/dev-manual/speeding-up-build.rst
@@ -0,0 +1,109 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Speeding Up a Build
4*******************
5
6Build time can be an issue. By default, the build system uses simple
7controls to try and maximize build efficiency. In general, the default
8settings for all the following variables result in the most efficient
9build times when dealing with single socket systems (i.e. a single CPU).
10If you have multiple CPUs, you might try increasing the default values
11to gain more speed. See the descriptions in the glossary for each
12variable for more information:
13
14- :term:`BB_NUMBER_THREADS`:
15 The maximum number of threads BitBake simultaneously executes.
16
17- :term:`BB_NUMBER_PARSE_THREADS`:
18 The number of threads BitBake uses during parsing.
19
20- :term:`PARALLEL_MAKE`: Extra
21 options passed to the ``make`` command during the
22 :ref:`ref-tasks-compile` task in
23 order to specify parallel compilation on the local build host.
24
25- :term:`PARALLEL_MAKEINST`:
26 Extra options passed to the ``make`` command during the
27 :ref:`ref-tasks-install` task in
28 order to specify parallel installation on the local build host.
29
30As mentioned, these variables all scale to the number of processor cores
31available on the build system. For single socket systems, this
32auto-scaling ensures that the build system fundamentally takes advantage
33of potential parallel operations during the build based on the build
34machine's capabilities.
35
36Following are additional factors that can affect build speed:
37
38- File system type: The file system type that the build is being
39 performed on can also influence performance. Using ``ext4`` is
40 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
41 improved features such as extents.
42
43- Disabling the updating of access time using ``noatime``: The
44 ``noatime`` mount option prevents the build system from updating file
45 and directory access times.
46
47- Setting a longer commit: Using the "commit=" mount option increases
48 the interval in seconds between disk cache writes. Changing this
49 interval from the five second default to something longer increases
50 the risk of data loss but decreases the need to write to the disk,
51 thus increasing the build performance.
52
53- Choosing the packaging backend: Of the available packaging backends,
54 IPK is the fastest. Additionally, selecting a singular packaging
55 backend also helps.
56
57- Using ``tmpfs`` for :term:`TMPDIR`
58 as a temporary file system: While this can help speed up the build,
59 the benefits are limited due to the compiler using ``-pipe``. The
60 build system goes to some lengths to avoid ``sync()`` calls into the
61 file system on the principle that if there was a significant failure,
62 the :term:`Build Directory` contents could easily be rebuilt.
63
64- Inheriting the :ref:`ref-classes-rm-work` class:
65 Inheriting this class has shown to speed up builds due to
66 significantly lower amounts of data stored in the data cache as well
67 as on disk. Inheriting this class also makes cleanup of
68 :term:`TMPDIR` faster, at the
69 expense of being easily able to dive into the source code. File
70 system maintainers have recommended that the fastest way to clean up
71 large numbers of files is to reformat partitions rather than delete
72 files due to the linear nature of partitions. This, of course,
73 assumes you structure the disk partitions and file systems in a way
74 that this is practical.
75
76Aside from the previous list, you should keep some trade offs in mind
77that can help you speed up the build:
78
79- Remove items from
80 :term:`DISTRO_FEATURES`
81 that you might not need.
82
83- Exclude debug symbols and other debug information: If you do not need
84 these symbols and other debug information, disabling the ``*-dbg``
85 package generation can speed up the build. You can disable this
86 generation by setting the
87 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
88 variable to "1".
89
90- Disable static library generation for recipes derived from
91 ``autoconf`` or ``libtool``: Following is an example showing how to
92 disable static libraries and still provide an override to handle
93 exceptions::
94
95 STATICLIBCONF = "--disable-static"
96 STATICLIBCONF:sqlite3-native = ""
97 EXTRA_OECONF += "${STATICLIBCONF}"
98
99 .. note::
100
101 - Some recipes need static libraries in order to work correctly
102 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
103 as in the previous example, account for these kinds of
104 exceptions.
105
106 - Some packages have packaging code that assumes the presence of
107 the static libraries. If so, you might need to exclude them as
108 well.
109
diff --git a/documentation/dev-manual/temporary-source-code.rst b/documentation/dev-manual/temporary-source-code.rst
new file mode 100644
index 0000000000..08bf68d982
--- /dev/null
+++ b/documentation/dev-manual/temporary-source-code.rst
@@ -0,0 +1,66 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Finding Temporary Source Code
4*****************************
5
6You might find it helpful during development to modify the temporary
7source code used by recipes to build packages. For example, suppose you
8are developing a patch and you need to experiment a bit to figure out
9your solution. After you have initially built the package, you can
10iteratively tweak the source code, which is located in the
11:term:`Build Directory`, and then you can force a re-compile and quickly
12test your altered code. Once you settle on a solution, you can then preserve
13your changes in the form of patches.
14
15During a build, the unpacked temporary source code used by recipes to
16build packages is available in the :term:`Build Directory` as defined by the
17:term:`S` variable. Below is the default value for the :term:`S` variable as
18defined in the ``meta/conf/bitbake.conf`` configuration file in the
19:term:`Source Directory`::
20
21 S = "${WORKDIR}/${BP}"
22
23You should be aware that many recipes override the
24:term:`S` variable. For example, recipes that fetch their source from Git
25usually set :term:`S` to ``${WORKDIR}/git``.
26
27.. note::
28
29 The :term:`BP` represents the base recipe name, which consists of the name
30 and version::
31
32 BP = "${BPN}-${PV}"
33
34
35The path to the work directory for the recipe
36(:term:`WORKDIR`) is defined as
37follows::
38
39 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
40
41The actual directory depends on several things:
42
43- :term:`TMPDIR`: The top-level build
44 output directory.
45
46- :term:`MULTIMACH_TARGET_SYS`:
47 The target system identifier.
48
49- :term:`PN`: The recipe name.
50
51- :term:`EXTENDPE`: The epoch --- if
52 :term:`PE` is not specified, which is
53 usually the case for most recipes, then :term:`EXTENDPE` is blank.
54
55- :term:`PV`: The recipe version.
56
57- :term:`PR`: The recipe revision.
58
59As an example, assume a Source Directory top-level folder named
60``poky``, a default :term:`Build Directory` at ``poky/build``, and a
61``qemux86-poky-linux`` machine target system. Furthermore, suppose your
62recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
63build system uses to build the package would be as follows::
64
65 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
66
diff --git a/documentation/dev-manual/upgrading-recipes.rst b/documentation/dev-manual/upgrading-recipes.rst
new file mode 100644
index 0000000000..abcf7ca9fe
--- /dev/null
+++ b/documentation/dev-manual/upgrading-recipes.rst
@@ -0,0 +1,397 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Upgrading Recipes
4*****************
5
6Over time, upstream developers publish new versions for software built
7by layer recipes. It is recommended to keep recipes up-to-date with
8upstream version releases.
9
10While there are several methods to upgrade a recipe, you might
11consider checking on the upgrade status of a recipe first. You can do so
12using the ``devtool check-upgrade-status`` command. See the
13":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
14section in the Yocto Project Reference Manual for more information.
15
16The remainder of this section describes three ways you can upgrade a
17recipe. You can use the Automated Upgrade Helper (AUH) to set up
18automatic version upgrades. Alternatively, you can use
19``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
20you can manually upgrade a recipe by editing the recipe itself.
21
22Using the Auto Upgrade Helper (AUH)
23===================================
24
25The AUH utility works in conjunction with the OpenEmbedded build system
26in order to automatically generate upgrades for recipes based on new
27versions being published upstream. Use AUH when you want to create a
28service that performs the upgrades automatically and optionally sends
29you an email with the results.
30
31AUH allows you to update several recipes with a single use. You can also
32optionally perform build and integration tests using images with the
33results saved to your hard drive and emails of results optionally sent
34to recipe maintainers. Finally, AUH creates Git commits with appropriate
35commit messages in the layer's tree for the changes made to recipes.
36
37.. note::
38
39 In some conditions, you should not use AUH to upgrade recipes
40 and should instead use either ``devtool upgrade`` or upgrade your
41 recipes manually:
42
43 - When AUH cannot complete the upgrade sequence. This situation
44 usually results because custom patches carried by the recipe
45 cannot be automatically rebased to the new version. In this case,
46 ``devtool upgrade`` allows you to manually resolve conflicts.
47
48 - When for any reason you want fuller control over the upgrade
49 process. For example, when you want special arrangements for
50 testing.
51
52The following steps describe how to set up the AUH utility:
53
54#. *Be Sure the Development Host is Set Up:* You need to be sure that
55 your development host is set up to use the Yocto Project. For
56 information on how to set up your host, see the
57 ":ref:`dev-manual/start:Preparing the Build Host`" section.
58
59#. *Make Sure Git is Configured:* The AUH utility requires Git to be
60 configured because AUH uses Git to save upgrades. Thus, you must have
61 Git user and email configured. The following command shows your
62 configurations::
63
64 $ git config --list
65
66 If you do not have the user and
67 email configured, you can use the following commands to do so::
68
69 $ git config --global user.name some_name
70 $ git config --global user.email username@domain.com
71
72#. *Clone the AUH Repository:* To use AUH, you must clone the repository
73 onto your development host. The following command uses Git to create
74 a local copy of the repository on your system::
75
76 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
77 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
78 remote: Compressing objects: 100% (300/300), done.
79 remote: Total 768 (delta 499), reused 703 (delta 434)
80 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
81 Resolving deltas: 100% (499/499), done.
82 Checking connectivity... done.
83
84 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
85 :term:`Poky` repositories.
86
87#. *Create a Dedicated Build Directory:* Run the :ref:`structure-core-script`
88 script to create a fresh :term:`Build Directory` that you use exclusively
89 for running the AUH utility::
90
91 $ cd poky
92 $ source oe-init-build-env your_AUH_build_directory
93
94 Re-using an existing :term:`Build Directory` and its configurations is not
95 recommended as existing settings could cause AUH to fail or behave
96 undesirably.
97
98#. *Make Configurations in Your Local Configuration File:* Several
99 settings are needed in the ``local.conf`` file in the build
100 directory you just created for AUH. Make these following
101 configurations:
102
103 - If you want to enable :ref:`Build
104 History <dev-manual/build-quality:maintaining build output quality>`,
105 which is optional, you need the following lines in the
106 ``conf/local.conf`` file::
107
108 INHERIT =+ "buildhistory"
109 BUILDHISTORY_COMMIT = "1"
110
111 With this configuration and a successful
112 upgrade, a build history "diff" file appears in the
113 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
114 your :term:`Build Directory`.
115
116 - If you want to enable testing through the :ref:`ref-classes-testimage*`
117 class, which is optional, you need to have the following set in
118 your ``conf/local.conf`` file::
119
120 INHERIT += "testimage"
121
122 .. note::
123
124 If your distro does not enable by default ptest, which Poky
125 does, you need the following in your ``local.conf`` file::
126
127 DISTRO_FEATURES:append = " ptest"
128
129
130#. *Optionally Start a vncserver:* If you are running in a server
131 without an X11 session, you need to start a vncserver::
132
133 $ vncserver :1
134 $ export DISPLAY=:1
135
136#. *Create and Edit an AUH Configuration File:* You need to have the
137 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
138 :term:`Build Directory`. You can find a sample configuration file in the
139 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
140
141 Read through the sample file and make configurations as needed. For
142 example, if you enabled build history in your ``local.conf`` as
143 described earlier, you must enable it in ``upgrade-helper.conf``.
144
145 Also, if you are using the default ``maintainers.inc`` file supplied
146 with Poky and located in ``meta-yocto`` and you do not set a
147 "maintainers_whitelist" or "global_maintainer_override" in the
148 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
149 the AUH command-line, the utility automatically sends out emails to
150 all the default maintainers. Please avoid this.
151
152This next set of examples describes how to use the AUH:
153
154- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
155 following form::
156
157 $ upgrade-helper.py recipe_name
158
159 For example, this command upgrades the ``xmodmap`` recipe::
160
161 $ upgrade-helper.py xmodmap
162
163- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
164 specific recipe to a particular version, use the following form::
165
166 $ upgrade-helper.py recipe_name -t version
167
168 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
169
170 $ upgrade-helper.py xmodmap -t 1.2.3
171
172- *Upgrading all Recipes to the Latest Versions and Suppressing Email
173 Notifications:* To upgrade all recipes to their most recent versions
174 and suppress the email notifications, use the following command::
175
176 $ upgrade-helper.py all
177
178- *Upgrading all Recipes to the Latest Versions and Send Email
179 Notifications:* To upgrade all recipes to their most recent versions
180 and send email messages to maintainers for each attempted recipe as
181 well as a status email, use the following command::
182
183 $ upgrade-helper.py -e all
184
185Once you have run the AUH utility, you can find the results in the AUH
186:term:`Build Directory`::
187
188 ${BUILDDIR}/upgrade-helper/timestamp
189
190The AUH utility
191also creates recipe update commits from successful upgrade attempts in
192the layer tree.
193
194You can easily set up to run the AUH utility on a regular basis by using
195a cron job. See the
196:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
197file distributed with the utility for an example.
198
199Using ``devtool upgrade``
200=========================
201
202As mentioned earlier, an alternative method for upgrading recipes to
203newer versions is to use
204:doc:`devtool upgrade </ref-manual/devtool-reference>`.
205You can read about ``devtool upgrade`` in general in the
206":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
207section in the Yocto Project Application Development and the Extensible
208Software Development Kit (eSDK) Manual.
209
210To see all the command-line options available with ``devtool upgrade``,
211use the following help command::
212
213 $ devtool upgrade -h
214
215If you want to find out what version a recipe is currently at upstream
216without any attempt to upgrade your local version of the recipe, you can
217use the following command::
218
219 $ devtool latest-version recipe_name
220
221As mentioned in the previous section describing AUH, ``devtool upgrade``
222works in a less-automated manner than AUH. Specifically,
223``devtool upgrade`` only works on a single recipe that you name on the
224command line, cannot perform build and integration testing using images,
225and does not automatically generate commits for changes in the source
226tree. Despite all these "limitations", ``devtool upgrade`` updates the
227recipe file to the new upstream version and attempts to rebase custom
228patches contained by the recipe as needed.
229
230.. note::
231
232 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
233 of a "wrapper" application for ``devtool upgrade``.
234
235A typical scenario involves having used Git to clone an upstream
236repository that you use during build operations. Because you have built the
237recipe in the past, the layer is likely added to your
238configuration already. If for some reason, the layer is not added, you
239could add it easily using the
240":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
241script. For example, suppose you use the ``nano.bb`` recipe from the
242``meta-oe`` layer in the ``meta-openembedded`` repository. For this
243example, assume that the layer has been cloned into following area::
244
245 /home/scottrif/meta-openembedded
246
247The following command from your :term:`Build Directory` adds the layer to
248your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
249
250 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
251 NOTE: Starting bitbake server...
252 Parsing recipes: 100% |##########################################| Time: 0:00:55
253 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
254 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
255 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
256 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
257 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
258
259For this example, assume that the ``nano.bb`` recipe that
260is upstream has a 2.9.3 version number. However, the version in the
261local repository is 2.7.4. The following command from your build
262directory automatically upgrades the recipe for you::
263
264 $ devtool upgrade nano -V 2.9.3
265 NOTE: Starting bitbake server...
266 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
267 Parsing recipes: 100% |##########################################| Time: 0:00:46
268 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
269 NOTE: Extracting current version source...
270 NOTE: Resolving any missing task queue dependencies
271 .
272 .
273 .
274 NOTE: Executing SetScene Tasks
275 NOTE: Executing RunQueue Tasks
276 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
277 Adding changed files: 100% |#####################################| Time: 0:00:00
278 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
279 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
280
281.. note::
282
283 Using the ``-V`` option is not necessary. Omitting the version number causes
284 ``devtool upgrade`` to upgrade the recipe to the most recent version.
285
286Continuing with this example, you can use ``devtool build`` to build the
287newly upgraded recipe::
288
289 $ devtool build nano
290 NOTE: Starting bitbake server...
291 Loading cache: 100% |################################################################################################| Time: 0:00:01
292 Loaded 2040 entries from dependency cache.
293 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
294 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
295 NOTE: Resolving any missing task queue dependencies
296 .
297 .
298 .
299 NOTE: Executing SetScene Tasks
300 NOTE: Executing RunQueue Tasks
301 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
302 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
303
304Within the ``devtool upgrade`` workflow, you can
305deploy and test your rebuilt software. For this example,
306however, running ``devtool finish`` cleans up the workspace once the
307source in your workspace is clean. This usually means using Git to stage
308and submit commits for the changes generated by the upgrade process.
309
310Once the tree is clean, you can clean things up in this example with the
311following command from the ``${BUILDDIR}/workspace/sources/nano``
312directory::
313
314 $ devtool finish nano meta-oe
315 NOTE: Starting bitbake server...
316 Loading cache: 100% |################################################################################################| Time: 0:00:00
317 Loaded 2040 entries from dependency cache.
318 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
319 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
320 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
321 NOTE: Updating recipe nano_2.9.3.bb
322 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
323 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
324 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
325
326
327Using the ``devtool finish`` command cleans up the workspace and creates a patch
328file based on your commits. The tool puts all patch files back into the
329source directory in a sub-directory named ``nano`` in this case.
330
331Manually Upgrading a Recipe
332===========================
333
334If for some reason you choose not to upgrade recipes using
335:ref:`dev-manual/upgrading-recipes:Using the Auto Upgrade Helper (AUH)` or
336by :ref:`dev-manual/upgrading-recipes:Using \`\`devtool upgrade\`\``,
337you can manually edit the recipe files to upgrade the versions.
338
339.. note::
340
341 Manually updating multiple recipes scales poorly and involves many
342 steps. The recommendation to upgrade recipe versions is through AUH
343 or ``devtool upgrade``, both of which automate some steps and provide
344 guidance for others needed for the manual process.
345
346To manually upgrade recipe versions, follow these general steps:
347
348#. *Change the Version:* Rename the recipe such that the version (i.e.
349 the :term:`PV` part of the recipe name)
350 changes appropriately. If the version is not part of the recipe name,
351 change the value as it is set for :term:`PV` within the recipe itself.
352
353#. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
354 is fetched from Git or some other version control system, update
355 :term:`SRCREV` to point to the
356 commit hash that matches the new version.
357
358#. *Build the Software:* Try to build the recipe using BitBake. Typical
359 build failures include the following:
360
361 - License statements were updated for the new version. For this
362 case, you need to review any changes to the license and update the
363 values of :term:`LICENSE` and
364 :term:`LIC_FILES_CHKSUM`
365 as needed.
366
367 .. note::
368
369 License changes are often inconsequential. For example, the
370 license text's copyright year might have changed.
371
372 - Custom patches carried by the older version of the recipe might
373 fail to apply to the new version. For these cases, you need to
374 review the failures. Patches might not be necessary for the new
375 version of the software if the upgraded version has fixed those
376 issues. If a patch is necessary and failing, you need to rebase it
377 into the new version.
378
379#. *Optionally Attempt to Build for Several Architectures:* Once you
380 successfully build the new software for a given architecture, you
381 could test the build for other architectures by changing the
382 :term:`MACHINE` variable and
383 rebuilding the software. This optional step is especially important
384 if the recipe is to be released publicly.
385
386#. *Check the Upstream Change Log or Release Notes:* Checking both these
387 reveals if there are new features that could break
388 backwards-compatibility. If so, you need to take steps to mitigate or
389 eliminate that situation.
390
391#. *Optionally Create a Bootable Image and Test:* If you want, you can
392 test the new software by booting it onto actual hardware.
393
394#. *Create a Commit with the Change in the Layer Repository:* After all
395 builds work and any testing is successful, you can create commits for
396 any changes in the layer holding your upgraded recipe.
397
diff --git a/documentation/dev-manual/vulnerabilities.rst b/documentation/dev-manual/vulnerabilities.rst
new file mode 100644
index 0000000000..ac0ca249c1
--- /dev/null
+++ b/documentation/dev-manual/vulnerabilities.rst
@@ -0,0 +1,214 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Checking for Vulnerabilities
4****************************
5
6Vulnerabilities in Poky and OE-Core
7===================================
8
9The Yocto Project has an infrastructure to track and address unfixed
10known security vulnerabilities, as tracked by the public
11:wikipedia:`Common Vulnerabilities and Exposures (CVE) <Common_Vulnerabilities_and_Exposures>`
12database.
13
14The Yocto Project maintains a `list of known vulnerabilities
15<https://autobuilder.yocto.io/pub/non-release/patchmetrics/>`__
16for packages in Poky and OE-Core, tracking the evolution of the number of
17unpatched CVEs and the status of patches. Such information is available for
18the current development version and for each supported release.
19
20Security is a process, not a product, and thus at any time, a number of security
21issues may be impacting Poky and OE-Core. It is up to the maintainers, users,
22contributors and anyone interested in the issues to investigate and possibly fix them by
23updating software components to newer versions or by applying patches to address them.
24It is recommended to work with Poky and OE-Core upstream maintainers and submit
25patches to fix them, see ":doc:`../contributor-guide/submit-changes`" for details.
26
27Vulnerability check at build time
28=================================
29
30To enable a check for CVE security vulnerabilities using
31:ref:`ref-classes-cve-check` in the specific image or target you are building,
32add the following setting to your configuration::
33
34 INHERIT += "cve-check"
35
36The CVE database contains some old incomplete entries which have been
37deemed not to impact Poky or OE-Core. These CVE entries can be excluded from the
38check using build configuration::
39
40 include conf/distro/include/cve-extra-exclusions.inc
41
42With this CVE check enabled, BitBake build will try to map each compiled software component
43recipe name and version information to the CVE database and generate recipe and
44image specific reports. These reports will contain:
45
46- metadata about the software component like names and versions
47
48- metadata about the CVE issue such as description and NVD link
49
50- for each software component, a list of CVEs which are possibly impacting this version
51
52- status of each CVE: ``Patched``, ``Unpatched`` or ``Ignored``
53
54The status ``Patched`` means that a patch file to address the security issue has been
55applied. ``Unpatched`` status means that no patches to address the issue have been
56applied and that the issue needs to be investigated. ``Ignored`` means that after
57analysis, it has been deemed to ignore the issue as it for example affects
58the software component on a different operating system platform.
59
60After a build with CVE check enabled, reports for each compiled source recipe will be
61found in ``build/tmp/deploy/cve``.
62
63For example the CVE check report for the ``flex-native`` recipe looks like::
64
65 $ cat poky/build/tmp/deploy/cve/flex-native
66 LAYER: meta
67 PACKAGE NAME: flex-native
68 PACKAGE VERSION: 2.6.4
69 CVE: CVE-2016-6354
70 CVE STATUS: Patched
71 CVE SUMMARY: Heap-based buffer overflow in the yy_get_next_buffer function in Flex before 2.6.1 might allow context-dependent attackers to cause a denial of service or possibly execute arbitrary code via vectors involving num_to_read.
72 CVSS v2 BASE SCORE: 7.5
73 CVSS v3 BASE SCORE: 9.8
74 VECTOR: NETWORK
75 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2016-6354
76
77 LAYER: meta
78 PACKAGE NAME: flex-native
79 PACKAGE VERSION: 2.6.4
80 CVE: CVE-2019-6293
81 CVE STATUS: Ignored
82 CVE SUMMARY: An issue was discovered in the function mark_beginning_as_normal in nfa.c in flex 2.6.4. There is a stack exhaustion problem caused by the mark_beginning_as_normal function making recursive calls to itself in certain scenarios involving lots of '*' characters. Remote attackers could leverage this vulnerability to cause a denial-of-service.
83 CVSS v2 BASE SCORE: 4.3
84 CVSS v3 BASE SCORE: 5.5
85 VECTOR: NETWORK
86 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2019-6293
87
88For images, a summary of all recipes included in the image and their CVEs is also
89generated in textual and JSON formats. These ``.cve`` and ``.json`` reports can be found
90in the ``tmp/deploy/images`` directory for each compiled image.
91
92At build time CVE check will also throw warnings about ``Unpatched`` CVEs::
93
94 WARNING: flex-2.6.4-r0 do_cve_check: Found unpatched CVE (CVE-2019-6293), for more information check /poky/build/tmp/work/core2-64-poky-linux/flex/2.6.4-r0/temp/cve.log
95 WARNING: libarchive-3.5.1-r0 do_cve_check: Found unpatched CVE (CVE-2021-36976), for more information check /poky/build/tmp/work/core2-64-poky-linux/libarchive/3.5.1-r0/temp/cve.log
96
97It is also possible to check the CVE status of individual packages as follows::
98
99 bitbake -c cve_check flex libarchive
100
101Fixing CVE product name and version mappings
102============================================
103
104By default, :ref:`ref-classes-cve-check` uses the recipe name :term:`BPN` as CVE
105product name when querying the CVE database. If this mapping contains false positives, e.g.
106some reported CVEs are not for the software component in question, or false negatives like
107some CVEs are not found to impact the recipe when they should, then the problems can be
108in the recipe name to CVE product mapping. These mapping issues can be fixed by setting
109the :term:`CVE_PRODUCT` variable inside the recipe. This defines the name of the software component in the
110upstream `NIST CVE database <https://nvd.nist.gov/>`__.
111
112The variable supports using vendor and product names like this::
113
114 CVE_PRODUCT = "flex_project:flex"
115
116In this example the vendor name used in the CVE database is ``flex_project`` and the
117product is ``flex``. With this setting the ``flex`` recipe only maps to this specific
118product and not products from other vendors with same name ``flex``.
119
120Similarly, when the recipe version :term:`PV` is not compatible with software versions used by
121the upstream software component releases and the CVE database, these can be fixed using
122the :term:`CVE_VERSION` variable.
123
124Note that if the CVE entries in the NVD database contain bugs or have missing or incomplete
125information, it is recommended to fix the information there directly instead of working
126around the issues possibly for a long time in Poky and OE-Core side recipes. Feedback to
127NVD about CVE entries can be provided through the `NVD contact form <https://nvd.nist.gov/info/contact-form>`__.
128
129Fixing vulnerabilities in recipes
130=================================
131
132If a CVE security issue impacts a software component, it can be fixed by updating to a newer
133version of the software component or by applying a patch. For Poky and OE-Core master branches, updating
134to a newer software component release with fixes is the best option, but patches can be applied
135if releases are not yet available.
136
137For stable branches, it is preferred to apply patches for the issues. For some software
138components minor version updates can also be applied if they are backwards compatible.
139
140Here is an example of fixing CVE security issues with patch files,
141an example from the :oe_layerindex:`ffmpeg recipe</layerindex/recipe/47350>`::
142
143 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
144 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \
145 file://fix-CVE-2020-20446.patch \
146 file://fix-CVE-2020-20453.patch \
147 file://fix-CVE-2020-22015.patch \
148 file://fix-CVE-2020-22021.patch \
149 file://fix-CVE-2020-22033-CVE-2020-22019.patch \
150 file://fix-CVE-2021-33815.patch \
151
152A good practice is to include the CVE identifier in both the patch file name
153and inside the patch file commit message using the format::
154
155 CVE: CVE-2020-22033
156
157CVE checker will then capture this information and change the CVE status to ``Patched``
158in the generated reports.
159
160If analysis shows that the CVE issue does not impact the recipe due to configuration, platform,
161version or other reasons, the CVE can be marked as ``Ignored`` using the :term:`CVE_CHECK_IGNORE` variable.
162As mentioned previously, if data in the CVE database is wrong, it is recommend to fix those
163issues in the CVE database directly.
164
165Recipes can be completely skipped by CVE check by including the recipe name in
166the :term:`CVE_CHECK_SKIP_RECIPE` variable.
167
168Implementation details
169======================
170
171Here's what the :ref:`ref-classes-cve-check` class does to find unpatched CVE IDs.
172
173First the code goes through each patch file provided by a recipe. If a valid CVE ID
174is found in the name of the file, the corresponding CVE is considered as patched.
175Don't forget that if multiple CVE IDs are found in the filename, only the last
176one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch
177file. The found CVE IDs are also considered as patched.
178
179Then, the code looks up all the CVE IDs in the NIST database for all the
180products defined in :term:`CVE_PRODUCT`. Then, for each found CVE:
181
182- If the package name (:term:`PN`) is part of
183 :term:`CVE_CHECK_SKIP_RECIPE`, it is considered as ``Patched``.
184
185- If the CVE ID is part of :term:`CVE_CHECK_IGNORE`, it is
186 set as ``Ignored``.
187
188- If the CVE ID is part of the patched CVE for the recipe, it is
189 already considered as ``Patched``.
190
191- Otherwise, the code checks whether the recipe version (:term:`PV`)
192 is within the range of versions impacted by the CVE. If so, the CVE
193 is considered as ``Unpatched``.
194
195The CVE database is stored in :term:`DL_DIR` and can be inspected using
196``sqlite3`` command as follows::
197
198 sqlite3 downloads/CVE_CHECK/nvdcve_1.1.db .dump | grep CVE-2021-37462
199
200When analyzing CVEs, it is recommended to:
201
202- study the latest information in `CVE database <https://nvd.nist.gov/vuln/search>`__.
203
204- check how upstream developers of the software component addressed the issue, e.g.
205 what patch was applied, which upstream release contains the fix.
206
207- check what other Linux distributions like `Debian <https://security-tracker.debian.org/tracker/>`__
208 did to analyze and address the issue.
209
210- follow security notices from other Linux distributions.
211
212- follow public `open source security mailing lists <https://oss-security.openwall.org/wiki/mailing-lists>`__ for
213 discussions and advance notifications of CVE bugs and software releases with fixes.
214
diff --git a/documentation/dev-manual/wayland.rst b/documentation/dev-manual/wayland.rst
new file mode 100644
index 0000000000..097be9cbde
--- /dev/null
+++ b/documentation/dev-manual/wayland.rst
@@ -0,0 +1,90 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using Wayland and Weston
4************************
5
6:wikipedia:`Wayland <Wayland_(display_server_protocol)>`
7is a computer display server protocol that provides a method for
8compositing window managers to communicate directly with applications
9and video hardware and expects them to communicate with input hardware
10using other libraries. Using Wayland with supporting targets can result
11in better control over graphics frame rendering than an application
12might otherwise achieve.
13
14The Yocto Project provides the Wayland protocol libraries and the
15reference :wikipedia:`Weston <Wayland_(display_server_protocol)#Weston>`
16compositor as part of its release. You can find the integrated packages
17in the ``meta`` layer of the :term:`Source Directory`.
18Specifically, you
19can find the recipes that build both Wayland and Weston at
20``meta/recipes-graphics/wayland``.
21
22You can build both the Wayland and Weston packages for use only with targets
23that accept the :wikipedia:`Mesa 3D and Direct Rendering Infrastructure
24<Mesa_(computer_graphics)>`, which is also known as Mesa DRI. This implies that
25you cannot build and use the packages if your target uses, for example, the
26Intel Embedded Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
27
28.. note::
29
30 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
31 emulated QEMU hardware. However, this version of Weston will run
32 under X emulation without issues.
33
34This section describes what you need to do to implement Wayland and use
35the Weston compositor when building an image for a supporting target.
36
37Enabling Wayland in an Image
38============================
39
40To enable Wayland, you need to enable it to be built and enable it to be
41included (installed) in the image.
42
43Building Wayland
44----------------
45
46To cause Mesa to build the ``wayland-egl`` platform and Weston to build
47Wayland with Kernel Mode Setting
48(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
49support, include the "wayland" flag in the
50:term:`DISTRO_FEATURES`
51statement in your ``local.conf`` file::
52
53 DISTRO_FEATURES:append = " wayland"
54
55.. note::
56
57 If X11 has been enabled elsewhere, Weston will build Wayland with X11
58 support
59
60Installing Wayland and Weston
61-----------------------------
62
63To install the Wayland feature into an image, you must include the
64following
65:term:`CORE_IMAGE_EXTRA_INSTALL`
66statement in your ``local.conf`` file::
67
68 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
69
70Running Weston
71==============
72
73To run Weston inside X11, enabling it as described earlier and building
74a Sato image is sufficient. If you are running your image under Sato, a
75Weston Launcher appears in the "Utility" category.
76
77Alternatively, you can run Weston through the command-line interpretor
78(CLI), which is better suited for development work. To run Weston under
79the CLI, you need to do the following after your image is built:
80
81#. Run these commands to export ``XDG_RUNTIME_DIR``::
82
83 mkdir -p /tmp/$USER-weston
84 chmod 0700 /tmp/$USER-weston
85 export XDG_RUNTIME_DIR=/tmp/$USER-weston
86
87#. Launch Weston in the shell::
88
89 weston
90
diff --git a/documentation/dev-manual/wic.rst b/documentation/dev-manual/wic.rst
new file mode 100644
index 0000000000..213f143273
--- /dev/null
+++ b/documentation/dev-manual/wic.rst
@@ -0,0 +1,729 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating Partitioned Images Using Wic
4*************************************
5
6Creating an image for a particular hardware target using the
7OpenEmbedded build system does not necessarily mean you can boot that
8image as is on your device. Physical devices accept and boot images in
9various ways depending on the specifics of the device. Usually,
10information about the hardware can tell you what image format the device
11requires. Should your device require multiple partitions on an SD card,
12flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
13create the properly partitioned image.
14
15The ``wic`` command generates partitioned images from existing
16OpenEmbedded build artifacts. Image generation is driven by partitioning
17commands contained in an OpenEmbedded kickstart file (``.wks``)
18specified either directly on the command line or as one of a selection
19of canned kickstart files as shown with the ``wic list images`` command
20in the
21":ref:`dev-manual/wic:generate an image using an existing kickstart file`"
22section. When you apply the command to a given set of build artifacts, the
23result is an image or set of images that can be directly written onto media and
24used on a particular system.
25
26.. note::
27
28 For a kickstart file reference, see the
29 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
30 Chapter in the Yocto Project Reference Manual.
31
32The ``wic`` command and the infrastructure it is based on is by
33definition incomplete. The purpose of the command is to allow the
34generation of customized images, and as such, was designed to be
35completely extensible through a plugin interface. See the
36":ref:`dev-manual/wic:using the wic plugin interface`" section
37for information on these plugins.
38
39This section provides some background information on Wic, describes what
40you need to have in place to run the tool, provides instruction on how
41to use the Wic utility, provides information on using the Wic plugins
42interface, and provides several examples that show how to use Wic.
43
44Background
45==========
46
47This section provides some background on the Wic utility. While none of
48this information is required to use Wic, you might find it interesting.
49
50- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
51 "oe" diphthong in "oeic" was promoted to the letter "w", because
52 "oeic" is both difficult to remember and to pronounce.
53
54- Wic is loosely based on the Meego Image Creator (``mic``) framework.
55 The Wic implementation has been heavily modified to make direct use
56 of OpenEmbedded build artifacts instead of package installation and
57 configuration, which are already incorporated within the OpenEmbedded
58 artifacts.
59
60- Wic is a completely independent standalone utility that initially
61 provides easier-to-use and more flexible replacements for an existing
62 functionality in OE-Core's :ref:`ref-classes-image-live`
63 class. The difference between Wic and those examples is that with Wic
64 the functionality of those scripts is implemented by a
65 general-purpose partitioning language, which is based on Redhat
66 kickstart syntax.
67
68Requirements
69============
70
71In order to use the Wic utility with the OpenEmbedded Build system, your
72system needs to meet the following requirements:
73
74- The Linux distribution on your development host must support the
75 Yocto Project. See the ":ref:`detailed-supported-distros`"
76 section in the Yocto Project Reference Manual for the list of
77 distributions that support the Yocto Project.
78
79- The standard system utilities, such as ``cp``, must be installed on
80 your development host system.
81
82- You must have sourced the build environment setup script (i.e.
83 :ref:`structure-core-script`) found in the :term:`Build Directory`.
84
85- You need to have the build artifacts already available, which
86 typically means that you must have already created an image using the
87 OpenEmbedded build system (e.g. ``core-image-minimal``). While it
88 might seem redundant to generate an image in order to create an image
89 using Wic, the current version of Wic requires the artifacts in the
90 form generated by the OpenEmbedded build system.
91
92- You must build several native tools, which are built to run on the
93 build system::
94
95 $ bitbake wic-tools
96
97- Include "wic" as part of the
98 :term:`IMAGE_FSTYPES`
99 variable.
100
101- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
102 as part of the :term:`WKS_FILE` variable. If multiple candidate files can
103 be provided by different layers, specify all the possible names through the
104 :term:`WKS_FILES` variable instead.
105
106Getting Help
107============
108
109You can get general help for the ``wic`` command by entering the ``wic``
110command by itself or by entering the command with a help argument as
111follows::
112
113 $ wic -h
114 $ wic --help
115 $ wic help
116
117Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
118``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
119commands except "help" by using the following form::
120
121 $ wic help command
122
123For example, the following command returns help for the ``write``
124command::
125
126 $ wic help write
127
128Wic supports help for three topics: ``overview``, ``plugins``, and
129``kickstart``. You can get help for any topic using the following form::
130
131 $ wic help topic
132
133For example, the following returns overview help for Wic::
134
135 $ wic help overview
136
137There is one additional level of help for Wic. You can get help on
138individual images through the ``list`` command. You can use the ``list``
139command to return the available Wic images as follows::
140
141 $ wic list images
142 genericx86 Create an EFI disk image for genericx86*
143 edgerouter Create SD card image for Edgerouter
144 beaglebone-yocto Create SD card image for Beaglebone
145 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
146 systemd-bootdisk Create an EFI disk image with systemd-boot
147 mkhybridiso Create a hybrid ISO image
148 mkefidisk Create an EFI disk image
149 sdimage-bootpart Create SD card image with a boot partition
150 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
151 directdisk Create a 'pcbios' direct disk image
152 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
153 qemuriscv Create qcow2 image for RISC-V QEMU machines
154 directdisk-gpt Create a 'pcbios' direct disk image
155 efi-bootdisk
156
157Once you know the list of available
158Wic images, you can use ``help`` with the command to get help on a
159particular image. For example, the following command returns help on the
160"beaglebone-yocto" image::
161
162 $ wic list beaglebone-yocto help
163
164 Creates a partitioned SD card image for Beaglebone.
165 Boot files are located in the first vfat partition.
166
167Operational Modes
168=================
169
170You can use Wic in two different modes, depending on how much control
171you need for specifying the OpenEmbedded build artifacts that are used
172for creating the image: Raw and Cooked:
173
174- *Raw Mode:* You explicitly specify build artifacts through Wic
175 command-line arguments.
176
177- *Cooked Mode:* The current
178 :term:`MACHINE` setting and image
179 name are used to automatically locate and provide the build
180 artifacts. You just supply a kickstart file and the name of the image
181 from which to use artifacts.
182
183Regardless of the mode you use, you need to have the build artifacts
184ready and available.
185
186Raw Mode
187--------
188
189Running Wic in raw mode allows you to specify all the partitions through
190the ``wic`` command line. The primary use for raw mode is if you have
191built your kernel outside of the Yocto Project :term:`Build Directory`.
192In other words, you can point to arbitrary kernel, root filesystem locations,
193and so forth. Contrast this behavior with cooked mode where Wic looks in the
194:term:`Build Directory` (e.g. ``tmp/deploy/images/``\ machine).
195
196The general form of the ``wic`` command in raw mode is::
197
198 $ wic create wks_file options ...
199
200 Where:
201
202 wks_file:
203 An OpenEmbedded kickstart file. You can provide
204 your own custom file or use a file from a set of
205 existing files as described by further options.
206
207 optional arguments:
208 -h, --help show this help message and exit
209 -o OUTDIR, --outdir OUTDIR
210 name of directory to create image in
211 -e IMAGE_NAME, --image-name IMAGE_NAME
212 name of the image to use the artifacts from e.g. core-
213 image-sato
214 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
215 path to the /rootfs dir to use as the .wks rootfs
216 source
217 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
218 path to the dir containing the boot artifacts (e.g.
219 /EFI or /syslinux dirs) to use as the .wks bootimg
220 source
221 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
222 path to the dir containing the kernel to use in the
223 .wks bootimg
224 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
225 path to the native sysroot containing the tools to use
226 to build the image
227 -s, --skip-build-check
228 skip the build check
229 -f, --build-rootfs build rootfs
230 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
231 compress image with specified compressor
232 -m, --bmap generate .bmap
233 --no-fstab-update Do not change fstab file.
234 -v VARS_DIR, --vars VARS_DIR
235 directory with <image>.env files that store bitbake
236 variables
237 -D, --debug output debug information
238
239.. note::
240
241 You do not need root privileges to run Wic. In fact, you should not
242 run as root when using the utility.
243
244Cooked Mode
245-----------
246
247Running Wic in cooked mode leverages off artifacts in the
248:term:`Build Directory`. In other words, you do not have to specify kernel or
249root filesystem locations as part of the command. All you need to provide is
250a kickstart file and the name of the image from which to use artifacts
251by using the "-e" option. Wic looks in the :term:`Build Directory` (e.g.
252``tmp/deploy/images/``\ machine) for artifacts.
253
254The general form of the ``wic`` command using Cooked Mode is as follows::
255
256 $ wic create wks_file -e IMAGE_NAME
257
258 Where:
259
260 wks_file:
261 An OpenEmbedded kickstart file. You can provide
262 your own custom file or use a file from a set of
263 existing files provided with the Yocto Project
264 release.
265
266 required argument:
267 -e IMAGE_NAME, --image-name IMAGE_NAME
268 name of the image to use the artifacts from e.g. core-
269 image-sato
270
271Using an Existing Kickstart File
272================================
273
274If you do not want to create your own kickstart file, you can use an
275existing file provided by the Wic installation. As shipped, kickstart
276files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
277following two locations::
278
279 poky/meta-yocto-bsp/wic
280 poky/scripts/lib/wic/canned-wks
281
282Use the following command to list the available kickstart files::
283
284 $ wic list images
285 genericx86 Create an EFI disk image for genericx86*
286 beaglebone-yocto Create SD card image for Beaglebone
287 edgerouter Create SD card image for Edgerouter
288 qemux86-directdisk Create a QEMU machine 'pcbios' direct disk image
289 directdisk-gpt Create a 'pcbios' direct disk image
290 mkefidisk Create an EFI disk image
291 directdisk Create a 'pcbios' direct disk image
292 systemd-bootdisk Create an EFI disk image with systemd-boot
293 mkhybridiso Create a hybrid ISO image
294 sdimage-bootpart Create SD card image with a boot partition
295 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
296 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
297
298When you use an existing file, you
299do not have to use the ``.wks`` extension. Here is an example in Raw
300Mode that uses the ``directdisk`` file::
301
302 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
303 -k kernel_dir -n native_sysroot
304
305Here are the actual partition language commands used in the
306``genericx86.wks`` file to generate an image::
307
308 # short-description: Create an EFI disk image for genericx86*
309 # long-description: Creates a partitioned EFI disk image for genericx86* machines
310 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
311 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
312 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
313
314 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
315
316Using the Wic Plugin Interface
317==============================
318
319You can extend and specialize Wic functionality by using Wic plugins.
320This section explains the Wic plugin interface.
321
322.. note::
323
324 Wic plugins consist of "source" and "imager" plugins. Imager plugins
325 are beyond the scope of this section.
326
327Source plugins provide a mechanism to customize partition content during
328the Wic image generation process. You can use source plugins to map
329values that you specify using ``--source`` commands in kickstart files
330(i.e. ``*.wks``) to a plugin implementation used to populate a given
331partition.
332
333.. note::
334
335 If you use plugins that have build-time dependencies (e.g. native
336 tools, bootloaders, and so forth) when building a Wic image, you need
337 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
338 variable.
339
340Source plugins are subclasses defined in plugin files. As shipped, the
341Yocto Project provides several plugin files. You can see the source
342plugin files that ship with the Yocto Project
343:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
344Each of these plugin files contains source plugins that are designed to
345populate a specific Wic image partition.
346
347Source plugins are subclasses of the ``SourcePlugin`` class, which is
348defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
349the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
350file is a subclass of the ``SourcePlugin`` class, which is found in the
351``pluginbase.py`` file.
352
353You can also implement source plugins in a layer outside of the Source
354Repositories (external layer). To do so, be sure that your plugin files
355are located in a directory whose path is
356``scripts/lib/wic/plugins/source/`` within your external layer. When the
357plugin files are located there, the source plugins they contain are made
358available to Wic.
359
360When the Wic implementation needs to invoke a partition-specific
361implementation, it looks for the plugin with the same name as the
362``--source`` parameter used in the kickstart file given to that
363partition. For example, if the partition is set up using the following
364command in a kickstart file::
365
366 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
367
368The methods defined as class
369members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
370``bootimg-pcbios.py`` plugin file are used.
371
372To be more concrete, here is the corresponding plugin definition from
373the ``bootimg-pcbios.py`` file for the previous command along with an
374example method called by the Wic implementation when it needs to prepare
375a partition using an implementation-specific function::
376
377 .
378 .
379 .
380 class BootimgPcbiosPlugin(SourcePlugin):
381 """
382 Create MBR boot partition and install syslinux on it.
383 """
384
385 name = 'bootimg-pcbios'
386 .
387 .
388 .
389 @classmethod
390 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
391 oe_builddir, bootimg_dir, kernel_dir,
392 rootfs_dir, native_sysroot):
393 """
394 Called to do the actual content population for a partition i.e. it
395 'prepares' the partition to be incorporated into the image.
396 In this case, prepare content for legacy bios boot partition.
397 """
398 .
399 .
400 .
401
402If a
403subclass (plugin) itself does not implement a particular function, Wic
404locates and uses the default version in the superclass. It is for this
405reason that all source plugins are derived from the ``SourcePlugin``
406class.
407
408The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
409a set of methods that source plugins can implement or override. Any
410plugins (subclass of ``SourcePlugin``) that do not implement a
411particular method inherit the implementation of the method from the
412``SourcePlugin`` class. For more information, see the ``SourcePlugin``
413class in the ``pluginbase.py`` file for details:
414
415The following list describes the methods implemented in the
416``SourcePlugin`` class:
417
418- ``do_prepare_partition()``: Called to populate a partition with
419 actual content. In other words, the method prepares the final
420 partition image that is incorporated into the disk image.
421
422- ``do_configure_partition()``: Called before
423 ``do_prepare_partition()`` to create custom configuration files for a
424 partition (e.g. syslinux or grub configuration files).
425
426- ``do_install_disk()``: Called after all partitions have been
427 prepared and assembled into a disk image. This method provides a hook
428 to allow finalization of a disk image (e.g. writing an MBR).
429
430- ``do_stage_partition()``: Special content-staging hook called
431 before ``do_prepare_partition()``. This method is normally empty.
432
433 Typically, a partition just uses the passed-in parameters (e.g. the
434 unmodified value of ``bootimg_dir``). However, in some cases, things
435 might need to be more tailored. As an example, certain files might
436 additionally need to be taken from ``bootimg_dir + /boot``. This hook
437 allows those files to be staged in a customized fashion.
438
439 .. note::
440
441 ``get_bitbake_var()`` allows you to access non-standard variables that
442 you might want to use for this behavior.
443
444You can extend the source plugin mechanism. To add more hooks, create
445more source plugin methods within ``SourcePlugin`` and the corresponding
446derived subclasses. The code that calls the plugin methods uses the
447``plugin.get_source_plugin_methods()`` function to find the method or
448methods needed by the call. Retrieval of those methods is accomplished
449by filling up a dict with keys that contain the method names of
450interest. On success, these will be filled in with the actual methods.
451See the Wic implementation for examples and details.
452
453Wic Examples
454============
455
456This section provides several examples that show how to use the Wic
457utility. All the examples assume the list of requirements in the
458":ref:`dev-manual/wic:requirements`" section have been met. The
459examples assume the previously generated image is
460``core-image-minimal``.
461
462Generate an Image using an Existing Kickstart File
463--------------------------------------------------
464
465This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
466file::
467
468 $ wic create mkefidisk -e core-image-minimal
469 INFO: Building wic-tools...
470 .
471 .
472 .
473 INFO: The new image(s) can be found here:
474 ./mkefidisk-201804191017-sda.direct
475
476 The following build artifacts were used to create the image(s):
477 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
478 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
479 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
480 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
481
482 INFO: The image(s) were created using OE kickstart file:
483 /home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
484
485The previous example shows the easiest way to create an image by running
486in cooked mode and supplying a kickstart file and the "-e" option to
487point to the existing build artifacts. Your ``local.conf`` file needs to
488have the :term:`MACHINE` variable set
489to the machine you are using, which is "qemux86" in this example.
490
491Once the image builds, the output provides image location, artifact use,
492and kickstart file information.
493
494.. note::
495
496 You should always verify the details provided in the output to make
497 sure that the image was indeed created exactly as expected.
498
499Continuing with the example, you can now write the image from the
500:term:`Build Directory` onto a USB stick, or whatever media for which you
501built your image, and boot from the media. You can write the image by using
502``bmaptool`` or ``dd``::
503
504 $ oe-run-native bmap-tools-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
505
506or ::
507
508 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
509
510.. note::
511
512 For more information on how to use the ``bmaptool``
513 to flash a device with an image, see the
514 ":ref:`dev-manual/bmaptool:flashing images using \`\`bmaptool\`\``"
515 section.
516
517Using a Modified Kickstart File
518-------------------------------
519
520Because partitioned image creation is driven by the kickstart file, it
521is easy to affect image creation by changing the parameters in the file.
522This next example demonstrates that through modification of the
523``directdisk-gpt`` kickstart file.
524
525As mentioned earlier, you can use the command ``wic list images`` to
526show the list of existing kickstart files. The directory in which the
527``directdisk-gpt.wks`` file resides is
528``scripts/lib/image/canned-wks/``, which is located in the
529:term:`Source Directory` (e.g. ``poky``).
530Because available files reside in this directory, you can create and add
531your own custom files to the directory. Subsequent use of the
532``wic list images`` command would then include your kickstart files.
533
534In this example, the existing ``directdisk-gpt`` file already does most
535of what is needed. However, for the hardware in this example, the image
536will need to boot from ``sdb`` instead of ``sda``, which is what the
537``directdisk-gpt`` kickstart file uses.
538
539The example begins by making a copy of the ``directdisk-gpt.wks`` file
540in the ``scripts/lib/image/canned-wks`` directory and then by changing
541the lines that specify the target disk from which to boot::
542
543 $ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
544 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
545
546Next, the example modifies the ``directdisksdb-gpt.wks`` file and
547changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
548example changes the following two lines and leaves the remaining lines
549untouched::
550
551 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
552 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
553
554Once the lines are changed, the
555example generates the ``directdisksdb-gpt`` image. The command points
556the process at the ``core-image-minimal`` artifacts for the Next Unit of
557Computing (nuc) :term:`MACHINE` the
558``local.conf``::
559
560 $ wic create directdisksdb-gpt -e core-image-minimal
561 INFO: Building wic-tools...
562 .
563 .
564 .
565 Initialising tasks: 100% |#######################################| Time: 0:00:01
566 NOTE: Executing SetScene Tasks
567 NOTE: Executing RunQueue Tasks
568 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
569 INFO: Creating image(s)...
570
571 INFO: The new image(s) can be found here:
572 ./directdisksdb-gpt-201710090938-sdb.direct
573
574 The following build artifacts were used to create the image(s):
575 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
576 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
577 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
578 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
579
580 INFO: The image(s) were created using OE kickstart file:
581 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
582
583Continuing with the example, you can now directly ``dd`` the image to a
584USB stick, or whatever media for which you built your image, and boot
585the resulting media::
586
587 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
588 140966+0 records in
589 140966+0 records out
590 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
591 $ sudo eject /dev/sdb
592
593Using a Modified Kickstart File and Running in Raw Mode
594-------------------------------------------------------
595
596This next example manually specifies each build artifact (runs in Raw
597Mode) and uses a modified kickstart file. The example also uses the
598``-o`` option to cause Wic to create the output somewhere other than the
599default output directory, which is the current directory::
600
601 $ wic create test.wks -o /home/stephano/testwic \
602 --rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
603 --bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
604 --kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \
605 --native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
606
607 INFO: Creating image(s)...
608
609 INFO: The new image(s) can be found here:
610 /home/stephano/testwic/test-201710091445-sdb.direct
611
612 The following build artifacts were used to create the image(s):
613 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
614 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
615 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
616 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
617
618 INFO: The image(s) were created using OE kickstart file:
619 test.wks
620
621For this example,
622:term:`MACHINE` did not have to be
623specified in the ``local.conf`` file since the artifact is manually
624specified.
625
626Using Wic to Manipulate an Image
627--------------------------------
628
629Wic image manipulation allows you to shorten turnaround time during
630image development. For example, you can use Wic to delete the kernel
631partition of a Wic image and then insert a newly built kernel. This
632saves you time from having to rebuild the entire image each time you
633modify the kernel.
634
635.. note::
636
637 In order to use Wic to manipulate a Wic image as in this example,
638 your development machine must have the ``mtools`` package installed.
639
640The following example examines the contents of the Wic image, deletes
641the existing kernel, and then inserts a new kernel:
642
643#. *List the Partitions:* Use the ``wic ls`` command to list all the
644 partitions in the Wic image::
645
646 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
647 Num Start End Size Fstype
648 1 1048576 25041919 23993344 fat16
649 2 25165824 72157183 46991360 ext4
650
651 The previous output shows two partitions in the
652 ``core-image-minimal-qemux86.wic`` image.
653
654#. *Examine a Particular Partition:* Use the ``wic ls`` command again
655 but in a different form to examine a particular partition.
656
657 .. note::
658
659 You can get command usage on any Wic command using the following
660 form::
661
662 $ wic help command
663
664
665 For example, the following command shows you the various ways to
666 use the
667 wic ls
668 command::
669
670 $ wic help ls
671
672
673 The following command shows what is in partition one::
674
675 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
676 Volume in drive : is boot
677 Volume Serial Number is E894-1809
678 Directory for ::/
679
680 libcom32 c32 186500 2017-10-09 16:06
681 libutil c32 24148 2017-10-09 16:06
682 syslinux cfg 220 2017-10-09 16:06
683 vesamenu c32 27104 2017-10-09 16:06
684 vmlinuz 6904608 2017-10-09 16:06
685 5 files 7 142 580 bytes
686 16 582 656 bytes free
687
688 The previous output shows five files, with the
689 ``vmlinuz`` being the kernel.
690
691 .. note::
692
693 If you see the following error, you need to update or create a
694 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
695 in the file. Then, run the Wic command again::
696
697 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
698 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
699 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
700
701
702#. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
703 ``vmlinuz`` file (kernel)::
704
705 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
706
707#. *Add In the New Kernel:* Use the ``wic cp`` command to add the
708 updated kernel to the Wic image. Depending on how you built your
709 kernel, it could be in different places. If you used ``devtool`` and
710 an SDK to build your kernel, it resides in the ``tmp/work`` directory
711 of the extensible SDK. If you used ``make`` to build the kernel, the
712 kernel will be in the ``workspace/sources`` area.
713
714 The following example assumes ``devtool`` was used to build the
715 kernel::
716
717 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
718 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
719
720 Once the new kernel is added back into the image, you can use the
721 ``dd`` command or :ref:`bmaptool
722 <dev-manual/bmaptool:flashing images using \`\`bmaptool\`\`>`
723 to flash your wic image onto an SD card or USB stick and test your
724 target.
725
726 .. note::
727
728 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
729
diff --git a/documentation/dev-manual/x32-psabi.rst b/documentation/dev-manual/x32-psabi.rst
new file mode 100644
index 0000000000..92b1f96fa4
--- /dev/null
+++ b/documentation/dev-manual/x32-psabi.rst
@@ -0,0 +1,54 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using x32 psABI
4***************
5
6x32 processor-specific Application Binary Interface (`x32
7psABI <https://software.intel.com/en-us/node/628948>`__) is a native
832-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
9ABI defines the calling conventions between functions in a processing
10environment. The interface determines what registers are used and what
11the sizes are for various C data types.
12
13Some processing environments prefer using 32-bit applications even when
14running on Intel 64-bit platforms. Consider the i386 psABI, which is a
15very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
16provide efficient use and access of the Intel 64-bit processor
17resources, leaving the system underutilized. Now consider the x86_64
18psABI. This ABI is newer and uses 64-bits for data sizes and program
19pointers. The extra bits increase the footprint size of the programs,
20libraries, and also increases the memory and file system size
21requirements. Executing under the x32 psABI enables user programs to
22utilize CPU and system resources more efficiently while keeping the
23memory footprint of the applications low. Extra bits are used for
24registers but not for addressing mechanisms.
25
26The Yocto Project supports the final specifications of x32 psABI as
27follows:
28
29- You can create packages and images in x32 psABI format on x86_64
30 architecture targets.
31
32- You can successfully build recipes with the x32 toolchain.
33
34- You can create and boot ``core-image-minimal`` and
35 ``core-image-sato`` images.
36
37- There is RPM Package Manager (RPM) support for x32 binaries.
38
39- There is support for large images.
40
41To use the x32 psABI, you need to edit your ``conf/local.conf``
42configuration file as follows::
43
44 MACHINE = "qemux86-64"
45 DEFAULTTUNE = "x86-64-x32"
46 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \
47 or 'INVALID')) or 'lib'}"
48
49Once you have set
50up your configuration file, use BitBake to build an image that supports
51the x32 psABI. Here is an example::
52
53 $ bitbake core-image-sato
54