summaryrefslogtreecommitdiffstats
path: root/documentation/dev-manual
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/dev-manual')
-rw-r--r--documentation/dev-manual/bmaptool.rst59
-rw-r--r--documentation/dev-manual/build-quality.rst409
-rw-r--r--documentation/dev-manual/building.rst942
-rw-r--r--documentation/dev-manual/common-tasks.rst11587
-rw-r--r--documentation/dev-manual/custom-distribution.rst135
-rw-r--r--documentation/dev-manual/custom-template-configuration-directory.rst52
-rw-r--r--documentation/dev-manual/customizing-images.rst223
-rw-r--r--documentation/dev-manual/debugging.rst1271
-rw-r--r--documentation/dev-manual/development-shell.rst82
-rw-r--r--documentation/dev-manual/device-manager.rst74
-rw-r--r--documentation/dev-manual/disk-space.rst61
-rw-r--r--documentation/dev-manual/efficiently-fetching-sources.rst68
-rw-r--r--documentation/dev-manual/error-reporting-tool.rst84
-rw-r--r--documentation/dev-manual/external-scm.rst67
-rw-r--r--documentation/dev-manual/external-toolchain.rst40
-rw-r--r--documentation/dev-manual/figures/cute-files-npm-example.pngbin26248 -> 73191 bytes
-rw-r--r--documentation/dev-manual/gobject-introspection.rst155
-rw-r--r--documentation/dev-manual/history.rst67
-rw-r--r--documentation/dev-manual/index.rst41
-rw-r--r--documentation/dev-manual/init-manager.rst162
-rw-r--r--documentation/dev-manual/intro.rst24
-rw-r--r--documentation/dev-manual/layers.rst919
-rw-r--r--documentation/dev-manual/libraries.rst267
-rw-r--r--documentation/dev-manual/licenses.rst544
-rw-r--r--documentation/dev-manual/new-machine.rst118
-rw-r--r--documentation/dev-manual/new-recipe.rst1639
-rw-r--r--documentation/dev-manual/packages.rst1250
-rw-r--r--documentation/dev-manual/prebuilt-libraries.rst209
-rw-r--r--documentation/dev-manual/python-development-shell.rst50
-rw-r--r--documentation/dev-manual/qemu.rst118
-rw-r--r--documentation/dev-manual/quilt.rst89
-rw-r--r--documentation/dev-manual/read-only-rootfs.rst89
-rw-r--r--documentation/dev-manual/runtime-testing.rst594
-rw-r--r--documentation/dev-manual/sbom.rst83
-rw-r--r--documentation/dev-manual/securing-images.rst156
-rw-r--r--documentation/dev-manual/security-subjects.rst189
-rw-r--r--documentation/dev-manual/speeding-up-build.rst109
-rw-r--r--documentation/dev-manual/start.rst434
-rw-r--r--documentation/dev-manual/temporary-source-code.rst66
-rw-r--r--documentation/dev-manual/upgrading-recipes.rst397
-rw-r--r--documentation/dev-manual/vulnerabilities.rst293
-rw-r--r--documentation/dev-manual/wayland.rst90
-rw-r--r--documentation/dev-manual/wic.rst731
-rw-r--r--documentation/dev-manual/x32-psabi.rst54
44 files changed, 12119 insertions, 11972 deletions
diff --git a/documentation/dev-manual/bmaptool.rst b/documentation/dev-manual/bmaptool.rst
new file mode 100644
index 0000000000..f6f0e6afaf
--- /dev/null
+++ b/documentation/dev-manual/bmaptool.rst
@@ -0,0 +1,59 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Flashing Images Using ``bmaptool``
4**********************************
5
6A fast and easy way to flash an image to a bootable device is to use
7bmaptool, which is integrated into the OpenEmbedded build system.
8bmaptool is a generic tool that creates a file's block map (bmap) and
9then uses that map to copy the file. As compared to traditional tools
10such as dd or cp, bmaptool can copy (or flash) large files like raw
11system image files much faster.
12
13.. note::
14
15 - If you are using Ubuntu or Debian distributions, you can install
16 the ``bmap-tools`` package using the following command and then
17 use the tool without specifying ``PATH`` even from the root
18 account::
19
20 $ sudo apt install bmap-tools
21
22 - If you are unable to install the ``bmap-tools`` package, you will
23 need to build bmaptool before using it. Use the following command::
24
25 $ bitbake bmaptool-native
26
27Following, is an example that shows how to flash a Wic image. Realize
28that while this example uses a Wic image, you can use bmaptool to flash
29any type of image. Use these steps to flash an image using bmaptool:
30
31#. *Update your local.conf File:* You need to have the following set
32 in your ``local.conf`` file before building your image::
33
34 IMAGE_FSTYPES += "wic wic.bmap"
35
36#. *Get Your Image:* Either have your image ready (pre-built with the
37 :term:`IMAGE_FSTYPES`
38 setting previously mentioned) or take the step to build the image::
39
40 $ bitbake image
41
42#. *Flash the Device:* Flash the device with the image by using bmaptool
43 depending on your particular setup. The following commands assume the
44 image resides in the :term:`Build Directory`'s ``deploy/images/`` area:
45
46 - If you have write access to the media, use this command form::
47
48 $ oe-run-native bmaptool-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
49
50 - If you do not have write access to the media, set your permissions
51 first and then use the same command form::
52
53 $ sudo chmod 666 /dev/sdX
54 $ oe-run-native bmaptool-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
55
56For help on the ``bmaptool`` command, use the following command::
57
58 $ bmaptool --help
59
diff --git a/documentation/dev-manual/build-quality.rst b/documentation/dev-manual/build-quality.rst
new file mode 100644
index 0000000000..713ea3a48e
--- /dev/null
+++ b/documentation/dev-manual/build-quality.rst
@@ -0,0 +1,409 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Maintaining Build Output Quality
4********************************
5
6Many factors can influence the quality of a build. For example, if you
7upgrade a recipe to use a new version of an upstream software package or
8you experiment with some new configuration options, subtle changes can
9occur that you might not detect until later. Consider the case where
10your recipe is using a newer version of an upstream package. In this
11case, a new version of a piece of software might introduce an optional
12dependency on another library, which is auto-detected. If that library
13has already been built when the software is building, the software will
14link to the built library and that library will be pulled into your
15image along with the new software even if you did not want the library.
16
17The :ref:`ref-classes-buildhistory` class helps you maintain the quality of
18your build output. You can use the class to highlight unexpected and possibly
19unwanted changes in the build output. When you enable build history, it records
20information about the contents of each package and image and then commits that
21information to a local Git repository where you can examine the information.
22
23The remainder of this section describes the following:
24
25- :ref:`How you can enable and disable build history <dev-manual/build-quality:enabling and disabling build history>`
26
27- :ref:`How to understand what the build history contains <dev-manual/build-quality:understanding what the build history contains>`
28
29- :ref:`How to limit the information used for build history <dev-manual/build-quality:using build history to gather image information only>`
30
31- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/build-quality:examining build history information>`
32
33Enabling and Disabling Build History
34====================================
35
36Build history is disabled by default. To enable it, add the following
37:term:`INHERIT` statement and set the :term:`BUILDHISTORY_COMMIT` variable to
38"1" at the end of your ``conf/local.conf`` file found in the
39:term:`Build Directory`::
40
41 INHERIT += "buildhistory"
42 BUILDHISTORY_COMMIT = "1"
43
44Enabling build history as
45previously described causes the OpenEmbedded build system to collect
46build output information and commit it as a single commit to a local
47:ref:`overview-manual/development-environment:git` repository.
48
49.. note::
50
51 Enabling build history increases your build times slightly,
52 particularly for images, and increases the amount of disk space used
53 during the build.
54
55You can disable build history by removing the previous statements from
56your ``conf/local.conf`` file.
57
58Understanding What the Build History Contains
59=============================================
60
61Build history information is kept in ``${``\ :term:`TOPDIR`\ ``}/buildhistory``
62in the :term:`Build Directory` as defined by the :term:`BUILDHISTORY_DIR`
63variable. Here is an example abbreviated listing:
64
65.. image:: figures/buildhistory.png
66 :align: center
67 :width: 50%
68
69At the top level, there is a ``metadata-revs`` file that lists the
70revisions of the repositories for the enabled layers when the build was
71produced. The rest of the data splits into separate ``packages``,
72``images`` and ``sdk`` directories, the contents of which are described
73as follows.
74
75Build History Package Information
76---------------------------------
77
78The history for each package contains a text file that has name-value
79pairs with information about the package. For example,
80``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
81contains the following:
82
83.. code-block:: none
84
85 PV = 1.22.1
86 PR = r32
87 RPROVIDES =
88 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
89 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
90 PKGSIZE = 540168
91 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
92 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
93 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
94 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
95 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
96 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
97 /etc/busybox.links.nosuid /etc/busybox.links.suid
98
99Most of these
100name-value pairs correspond to variables used to produce the package.
101The exceptions are ``FILELIST``, which is the actual list of files in
102the package, and ``PKGSIZE``, which is the total size of files in the
103package in bytes.
104
105There is also a file that corresponds to the recipe from which the package
106came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
107
108.. code-block:: none
109
110 PV = 1.22.1
111 PR = r32
112 DEPENDS = initscripts kern-tools-native update-rc.d-native \
113 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
114 virtual/libc virtual/update-alternatives
115 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
116 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
117 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
118
119Finally, for those recipes fetched from a version control system (e.g.,
120Git), there is a file that lists source revisions that are specified in
121the recipe and the actual revisions used during the build. Listed
122and actual revisions might differ when
123:term:`SRCREV` is set to
124${:term:`AUTOREV`}. Here is an
125example assuming
126``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
127
128 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
129 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
130 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
131 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
132
133You can use the
134``buildhistory-collect-srcrevs`` command with the ``-a`` option to
135collect the stored :term:`SRCREV` values from build history and report them
136in a format suitable for use in global configuration (e.g.,
137``local.conf`` or a distro include file) to override floating
138:term:`AUTOREV` values to a fixed set of revisions. Here is some example
139output from this command::
140
141 $ buildhistory-collect-srcrevs -a
142 # all-poky-linux
143 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8"
144 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac"
145 # core2-64-poky-linux
146 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66"
147 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8"
148 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0"
149 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0"
150 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1"
151 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c"
152 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7"
153 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee"
154 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1"
155 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100"
156 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3"
157 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf"
158 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced"
159 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8"
160 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f"
161 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa"
162 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70"
163 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
164 # qemux86_64-poky-linux
165 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c"
166 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0"
167 # x86_64-linux
168 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66"
169 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c"
170 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87"
171 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57"
172 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb"
173 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b"
174 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
175
176.. note::
177
178 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
179
180 - By default, only values where the :term:`SRCREV` was not hardcoded
181 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
182 option to see all :term:`SRCREV` values.
183
184 - The output statements might not have any effect if overrides are
185 applied elsewhere in the build system configuration. Use the
186 ``-f`` option to add the ``forcevariable`` override to each output
187 line if you need to work around this restriction.
188
189 - The script does apply special handling when building for multiple
190 machines. However, the script does place a comment before each set
191 of values that specifies which triplet to which they belong as
192 previously shown (e.g., ``i586-poky-linux``).
193
194Build History Image Information
195-------------------------------
196
197The files produced for each image are as follows:
198
199- ``image-files:`` A directory containing selected files from the root
200 filesystem. The files are defined by
201 :term:`BUILDHISTORY_IMAGE_FILES`.
202
203- ``build-id.txt:`` Human-readable information about the build
204 configuration and metadata source revisions. This file contains the
205 full build header as printed by BitBake.
206
207- ``*.dot:`` Dependency graphs for the image that are compatible with
208 ``graphviz``.
209
210- ``files-in-image.txt:`` A list of files in the image with
211 permissions, owner, group, size, and symlink information.
212
213- ``image-info.txt:`` A text file containing name-value pairs with
214 information about the image. See the following listing example for
215 more information.
216
217- ``installed-package-names.txt:`` A list of installed packages by name
218 only.
219
220- ``installed-package-sizes.txt:`` A list of installed packages ordered
221 by size.
222
223- ``installed-packages.txt:`` A list of installed packages with full
224 package filenames.
225
226.. note::
227
228 Installed package information is able to be gathered and produced
229 even if package management is disabled for the final image.
230
231Here is an example of ``image-info.txt``:
232
233.. code-block:: none
234
235 DISTRO = poky
236 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be
237 USER_CLASSES = buildstats
238 IMAGE_CLASSES = qemuboot qemuboot license_image
239 IMAGE_FEATURES = debug-tweaks
240 IMAGE_LINGUAS =
241 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp
242 BAD_RECOMMENDATIONS =
243 NO_RECOMMENDATIONS =
244 PACKAGE_EXCLUDE =
245 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible;
246 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
247 IMAGESIZE = 9265
248
249Other than ``IMAGESIZE``,
250which is the total size of the files in the image in Kbytes, the
251name-value pairs are variables that may have influenced the content of
252the image. This information is often useful when you are trying to
253determine why a change in the package or file listings has occurred.
254
255Using Build History to Gather Image Information Only
256----------------------------------------------------
257
258As you can see, build history produces image information, including
259dependency graphs, so you can see why something was pulled into the
260image. If you are just interested in this information and not interested
261in collecting specific package or SDK information, you can enable
262writing only image information without any history by adding the
263following to your ``conf/local.conf`` file found in the
264:term:`Build Directory`::
265
266 INHERIT += "buildhistory"
267 BUILDHISTORY_COMMIT = "0"
268 BUILDHISTORY_FEATURES = "image"
269
270Here, you set the
271:term:`BUILDHISTORY_FEATURES`
272variable to use the image feature only.
273
274Build History SDK Information
275-----------------------------
276
277Build history collects similar information on the contents of SDKs (e.g.
278``bitbake -c populate_sdk imagename``) as compared to information it
279collects for images. Furthermore, this information differs depending on
280whether an extensible or standard SDK is being produced.
281
282The following list shows the files produced for SDKs:
283
284- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
285 owner, group, size, and symlink information. This list includes both
286 the host and target parts of the SDK.
287
288- ``sdk-info.txt:`` A text file containing name-value pairs with
289 information about the SDK. See the following listing example for more
290 information.
291
292- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
293 with information about task group sizes (e.g. :ref:`ref-tasks-populate_sysroot`
294 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
295 only when an extensible SDK is created.
296
297- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
298 with information for the shared-state packages and sizes in the SDK.
299 The ``sstate-package-sizes.txt`` file exists only when an extensible
300 SDK is created.
301
302- ``sdk-files:`` A folder that contains copies of the files mentioned
303 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
304 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
305 specific to the extensible SDK although you can set it differently if
306 you would like to pull in specific files from the standard SDK.
307
308 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
309 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
310 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
311 copied into the ``sdk-files`` directory.
312
313- The following information appears under each of the ``host`` and
314 ``target`` directories for the portions of the SDK that run on the
315 host and on the target, respectively:
316
317 .. note::
318
319 The following files for the most part are empty when producing an
320 extensible SDK because this type of SDK is not constructed from
321 packages as is the standard SDK.
322
323 - ``depends.dot:`` Dependency graph for the SDK that is compatible
324 with ``graphviz``.
325
326 - ``installed-package-names.txt:`` A list of installed packages by
327 name only.
328
329 - ``installed-package-sizes.txt:`` A list of installed packages
330 ordered by size.
331
332 - ``installed-packages.txt:`` A list of installed packages with full
333 package filenames.
334
335Here is an example of ``sdk-info.txt``:
336
337.. code-block:: none
338
339 DISTRO = poky
340 DISTRO_VERSION = 1.3+snapshot-20130327
341 SDK_NAME = poky-glibc-i686-arm
342 SDK_VERSION = 1.3+snapshot
343 SDKMACHINE =
344 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
345 BAD_RECOMMENDATIONS =
346 SDKSIZE = 352712
347
348Other than ``SDKSIZE``, which is
349the total size of the files in the SDK in Kbytes, the name-value pairs
350are variables that might have influenced the content of the SDK. This
351information is often useful when you are trying to determine why a
352change in the package or file listings has occurred.
353
354Examining Build History Information
355-----------------------------------
356
357You can examine build history output from the command line or from a web
358interface.
359
360To see any changes that have occurred (assuming you have
361:term:`BUILDHISTORY_COMMIT` = "1"),
362you can simply use any Git command that allows you to view the history
363of a repository. Here is one method::
364
365 $ git log -p
366
367You need to realize,
368however, that this method does show changes that are not significant
369(e.g. a package's size changing by a few bytes).
370
371There is a command-line tool called ``buildhistory-diff``, though,
372that queries the Git repository and prints just the differences that
373might be significant in human-readable form. Here is an example::
374
375 $ poky/poky/scripts/buildhistory-diff . HEAD^
376 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
377 /etc/anotherpkg.conf was added
378 /sbin/anotherpkg was added
379 * (installed-package-names.txt):
380 * anotherpkg was added
381 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
382 anotherpkg was added
383 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
384 * PR changed from "r0" to "r1"
385 * PV changed from "0.1.10" to "0.1.12"
386 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
387 * PR changed from "r0" to "r1"
388 * PV changed from "0.1.10" to "0.1.12"
389
390.. note::
391
392 The ``buildhistory-diff`` tool requires the ``GitPython``
393 package. Be sure to install it using Pip3 as follows::
394
395 $ pip3 install GitPython --user
396
397
398 Alternatively, you can install ``python3-git`` using the appropriate
399 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``).
400
401To see changes to the build history using a web interface, follow the
402instruction in the ``README`` file
403:yocto_git:`here </buildhistory-web/>`.
404
405Here is a sample screenshot of the interface:
406
407.. image:: figures/buildhistory-web.png
408 :width: 100%
409
diff --git a/documentation/dev-manual/building.rst b/documentation/dev-manual/building.rst
new file mode 100644
index 0000000000..fe502690dd
--- /dev/null
+++ b/documentation/dev-manual/building.rst
@@ -0,0 +1,942 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Building
4********
5
6This section describes various build procedures, such as the steps
7needed for a simple build, building a target for multiple configurations,
8generating an image for more than one machine, and so forth.
9
10Building a Simple Image
11=======================
12
13In the development environment, you need to build an image whenever you
14change hardware support, add or change system libraries, or add or
15change services that have dependencies. There are several methods that allow
16you to build an image within the Yocto Project. This section presents
17the basic steps you need to build a simple image using BitBake from a
18build host running Linux.
19
20.. note::
21
22 - For information on how to build an image using
23 :term:`Toaster`, see the
24 :doc:`/toaster-manual/index`.
25
26 - For information on how to use ``devtool`` to build images, see the
27 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
28 section in the Yocto Project Application Development and the
29 Extensible Software Development Kit (eSDK) manual.
30
31 - For a quick example on how to build an image using the
32 OpenEmbedded build system, see the
33 :doc:`/brief-yoctoprojectqs/index` document.
34
35 - You can also use the `Yocto Project BitBake
36 <https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__
37 extension for Visual Studio Code to build images.
38
39The build process creates an entire Linux distribution from source and
40places it in your :term:`Build Directory` under ``tmp/deploy/images``. For
41detailed information on the build process using BitBake, see the
42":ref:`overview-manual/concepts:images`" section in the Yocto Project Overview
43and Concepts Manual.
44
45The following figure and list overviews the build process:
46
47.. image:: figures/bitbake-build-flow.png
48 :width: 100%
49
50#. *Set up Your Host Development System to Support Development Using the
51 Yocto Project*: See the ":doc:`start`" section for options on how to get a
52 build host ready to use the Yocto Project.
53
54#. *Initialize the Build Environment:* Initialize the build environment
55 by sourcing the build environment script (i.e.
56 :ref:`structure-core-script`)::
57
58 $ source oe-init-build-env [build_dir]
59
60 When you use the initialization script, the OpenEmbedded build system
61 uses ``build`` as the default :term:`Build Directory` in your current work
62 directory. You can use a `build_dir` argument with the script to
63 specify a different :term:`Build Directory`.
64
65 .. note::
66
67 A common practice is to use a different :term:`Build Directory` for
68 different targets; for example, ``~/build/x86`` for a ``qemux86``
69 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
70 event, it's typically cleaner to locate the :term:`Build Directory`
71 somewhere outside of your source directory.
72
73#. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
74 ``conf/local.conf`` configuration file, which is found in the
75 :term:`Build Directory`, is set up how you want it. This file defines many
76 aspects of the build environment including the target machine architecture
77 through the :term:`MACHINE` variable, the packaging format used during
78 the build (:term:`PACKAGE_CLASSES`), and a centralized tarball download
79 directory through the :term:`DL_DIR` variable.
80
81#. *Build the Image:* Build the image using the ``bitbake`` command::
82
83 $ bitbake target
84
85 .. note::
86
87 For information on BitBake, see the :doc:`bitbake:index`.
88
89 The target is the name of the recipe you want to build. Common
90 targets are the images in ``meta/recipes-core/images``,
91 ``meta/recipes-sato/images``, and so forth all found in the
92 :term:`Source Directory`. Alternatively, the target
93 can be the name of a recipe for a specific piece of software such as
94 BusyBox. For more details about the images the OpenEmbedded build
95 system supports, see the
96 ":ref:`ref-manual/images:Images`" chapter in the Yocto
97 Project Reference Manual.
98
99 As an example, the following command builds the
100 ``core-image-minimal`` image::
101
102 $ bitbake core-image-minimal
103
104 Once an
105 image has been built, it often needs to be installed. The images and
106 kernels built by the OpenEmbedded build system are placed in the
107 :term:`Build Directory` in ``tmp/deploy/images``. For information on how to
108 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
109 :doc:`/sdk-manual/index` manual. For
110 information about how to install these images, see the documentation
111 for your particular board or machine.
112
113Building Images for Multiple Targets Using Multiple Configurations
114==================================================================
115
116You can use a single ``bitbake`` command to build multiple images or
117packages for different targets where each image or package requires a
118different configuration (multiple configuration builds). The builds, in
119this scenario, are sometimes referred to as "multiconfigs", and this
120section uses that term throughout.
121
122This section describes how to set up for multiple configuration builds
123and how to account for cross-build dependencies between the
124multiconfigs.
125
126Setting Up and Running a Multiple Configuration Build
127-----------------------------------------------------
128
129To accomplish a multiple configuration build, you must define each
130target's configuration separately using a parallel configuration file in
131the :term:`Build Directory` or configuration directory within a layer, and you
132must follow a required file hierarchy. Additionally, you must enable the
133multiple configuration builds in your ``local.conf`` file.
134
135Follow these steps to set up and execute multiple configuration builds:
136
137- *Create Separate Configuration Files*: You need to create a single
138 configuration file for each build target (each multiconfig).
139 The configuration definitions are implementation dependent but often
140 each configuration file will define the machine and the
141 temporary directory BitBake uses for the build. Whether the same
142 temporary directory (:term:`TMPDIR`) can be shared will depend on what is
143 similar and what is different between the configurations. Multiple MACHINE
144 targets can share the same (:term:`TMPDIR`) as long as the rest of the
145 configuration is the same, multiple :term:`DISTRO` settings would need separate
146 (:term:`TMPDIR`) directories.
147
148 For example, consider a scenario with two different multiconfigs for the same
149 :term:`MACHINE`: "qemux86" built
150 for two distributions such as "poky" and "poky-lsb". In this case,
151 you would need to use the different :term:`TMPDIR`.
152
153 Here is an example showing the minimal statements needed in a
154 configuration file for a "qemux86" target whose temporary build
155 directory is ``tmpmultix86``::
156
157 MACHINE = "qemux86"
158 TMPDIR = "${TOPDIR}/tmpmultix86"
159
160 The location for these multiconfig configuration files is specific.
161 They must reside in the current :term:`Build Directory` in a sub-directory of
162 ``conf`` named ``multiconfig`` or within a layer's ``conf`` directory
163 under a directory named ``multiconfig``. Here is an example that defines
164 two configuration files for the "x86" and "arm" multiconfigs:
165
166 .. image:: figures/multiconfig_files.png
167 :align: center
168 :width: 50%
169
170 The usual :term:`BBPATH` search path is used to locate multiconfig files in
171 a similar way to other conf files.
172
173- *Add the BitBake Multi-configuration Variable to the Local
174 Configuration File*: Use the
175 :term:`BBMULTICONFIG`
176 variable in your ``conf/local.conf`` configuration file to specify
177 each multiconfig. Continuing with the example from the previous
178 figure, the :term:`BBMULTICONFIG` variable needs to enable two
179 multiconfigs: "x86" and "arm" by specifying each configuration file::
180
181 BBMULTICONFIG = "x86 arm"
182
183 .. note::
184
185 A "default" configuration already exists by definition. This
186 configuration is named: "" (i.e. empty string) and is defined by
187 the variables coming from your ``local.conf``
188 file. Consequently, the previous example actually adds two
189 additional configurations to your build: "arm" and "x86" along
190 with "".
191
192- *Launch BitBake*: Use the following BitBake command form to launch
193 the multiple configuration build::
194
195 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
196
197 For the example in this section, the following command applies::
198
199 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
200
201 The previous BitBake command builds a ``core-image-minimal`` image
202 that is configured through the ``x86.conf`` configuration file, a
203 ``core-image-sato`` image that is configured through the ``arm.conf``
204 configuration file and a ``core-image-base`` that is configured
205 through your ``local.conf`` configuration file.
206
207.. note::
208
209 Support for multiple configuration builds in the Yocto Project &DISTRO;
210 (&DISTRO_NAME;) Release does not include Shared State (sstate)
211 optimizations. Consequently, if a build uses the same object twice
212 in, for example, two different :term:`TMPDIR`
213 directories, the build either loads from an existing sstate cache for
214 that build at the start or builds the object fresh.
215
216Enabling Multiple Configuration Build Dependencies
217--------------------------------------------------
218
219Sometimes dependencies can exist between targets (multiconfigs) in a
220multiple configuration build. For example, suppose that in order to
221build a ``core-image-sato`` image for an "x86" multiconfig, the root
222filesystem of an "arm" multiconfig must exist. This dependency is
223essentially that the
224:ref:`ref-tasks-image` task in the
225``core-image-sato`` recipe depends on the completion of the
226:ref:`ref-tasks-rootfs` task of the
227``core-image-minimal`` recipe.
228
229To enable dependencies in a multiple configuration build, you must
230declare the dependencies in the recipe using the following statement
231form::
232
233 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
234
235To better show how to use this statement, consider the example scenario
236from the first paragraph of this section. The following statement needs
237to be added to the recipe that builds the ``core-image-sato`` image::
238
239 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
240
241In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
242task on which the :ref:`ref-tasks-image` task in the recipe depends is the
243:ref:`ref-tasks-rootfs` task from the ``core-image-minimal`` recipe associated
244with the "arm" multiconfig.
245
246Once you set up this dependency, you can build the "x86" multiconfig
247using a BitBake command as follows::
248
249 $ bitbake mc:x86:core-image-sato
250
251This command executes all the tasks needed to create the
252``core-image-sato`` image for the "x86" multiconfig. Because of the
253dependency, BitBake also executes through the :ref:`ref-tasks-rootfs` task for the
254"arm" multiconfig build.
255
256Having a recipe depend on the root filesystem of another build might not
257seem that useful. Consider this change to the statement in the
258``core-image-sato`` recipe::
259
260 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
261
262In this case, BitBake must
263create the ``core-image-minimal`` image for the "arm" build since the
264"x86" build depends on it.
265
266Because "x86" and "arm" are enabled for multiple configuration builds
267and have separate configuration files, BitBake places the artifacts for
268each build in the respective temporary build directories (i.e.
269:term:`TMPDIR`).
270
271Building an Initial RAM Filesystem (Initramfs) Image
272====================================================
273
274An initial RAM filesystem (:term:`Initramfs`) image provides a temporary root
275filesystem used for early system initialization, typically providing tools and
276loading modules needed to locate and mount the final root filesystem.
277
278Follow these steps to create an :term:`Initramfs` image:
279
280#. *Create the Initramfs Image Recipe:* You can reference the
281 ``core-image-minimal-initramfs.bb`` recipe found in the
282 ``meta/recipes-core`` directory of the :term:`Source Directory`
283 as an example from which to work.
284
285#. *Decide if You Need to Bundle the Initramfs Image Into the Kernel
286 Image:* If you want the :term:`Initramfs` image that is built to be bundled
287 in with the kernel image, set the :term:`INITRAMFS_IMAGE_BUNDLE`
288 variable to ``"1"`` in your ``local.conf`` configuration file and set the
289 :term:`INITRAMFS_IMAGE` variable in the recipe that builds the kernel image.
290
291 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the :term:`Initramfs`
292 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
293 :term:`Initramfs` image is then passed to the kernel's ``Makefile`` using the
294 :term:`CONFIG_INITRAMFS_SOURCE` variable, allowing the :term:`Initramfs`
295 image to be built into the kernel normally.
296
297#. *Optionally Add Items to the Initramfs Image Through the Initramfs
298 Image Recipe:* If you add items to the :term:`Initramfs` image by way of its
299 recipe, you should use :term:`PACKAGE_INSTALL` rather than
300 :term:`IMAGE_INSTALL`. :term:`PACKAGE_INSTALL` gives more direct control of
301 what is added to the image as compared to the defaults you might not
302 necessarily want that are set by the :ref:`ref-classes-image`
303 or :ref:`ref-classes-core-image` classes.
304
305#. *Build the Kernel Image and the Initramfs Image:* Build your kernel
306 image using BitBake. Because the :term:`Initramfs` image recipe is a
307 dependency of the kernel image, the :term:`Initramfs` image is built as well
308 and bundled with the kernel image if you used the
309 :term:`INITRAMFS_IMAGE_BUNDLE` variable described earlier.
310
311Bundling an Initramfs Image From a Separate Multiconfig
312-------------------------------------------------------
313
314There may be a case where we want to build an :term:`Initramfs` image which does not
315inherit the same distro policy as our main image, for example, we may want
316our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our :term:`Initramfs`
317image to keep a smaller footprint. However, by performing the steps mentioned
318above the :term:`Initramfs` image will inherit ``TCLIBC="glibc"`` without allowing us
319to override it.
320
321To achieve this, you need to perform some additional steps:
322
323#. *Create a multiconfig for your Initramfs image:* You can perform the steps
324 on ":ref:`dev-manual/building:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
325 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
326 contains the variables::
327
328 TMPDIR="${TOPDIR}/tmp-initramfscfg"
329 TCLIBC="musl"
330
331#. *Set additional Initramfs variables on your main configuration:*
332 Additionally, on your main configuration (``local.conf``) you need to set the
333 variables::
334
335 INITRAMFS_MULTICONFIG = "initramfscfg"
336 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
337
338 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
339 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
340 to be built coming from the ``initramfscfg`` multiconfig, and to let the
341 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
342
343 Building a system with such configuration will build the kernel using the
344 main configuration but the :ref:`ref-tasks-bundle_initramfs` task will grab the
345 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
346 instead, resulting in a musl based :term:`Initramfs` image bundled in the kernel
347 but a glibc based main image.
348
349 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
350 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
351
352
353Building a Tiny System
354======================
355
356Very small distributions have some significant advantages such as
357requiring less on-die or in-package memory (cheaper), better performance
358through efficient cache usage, lower power requirements due to less
359memory, faster boot times, and reduced development overhead. Some
360real-world examples where a very small distribution gives you distinct
361advantages are digital cameras, medical devices, and small headless
362systems.
363
364This section presents information that shows you how you can trim your
365distribution to even smaller sizes than the ``poky-tiny`` distribution,
366which is around 5 Mbytes, that can be built out-of-the-box using the
367Yocto Project.
368
369Tiny System Overview
370--------------------
371
372The following list presents the overall steps you need to consider and
373perform to create distributions with smaller root filesystems, achieve
374faster boot times, maintain your critical functionality, and avoid
375initial RAM disks:
376
377- :ref:`Determine your goals and guiding principles
378 <dev-manual/building:goals and guiding principles>`
379
380- :ref:`dev-manual/building:understand what contributes to your image size`
381
382- :ref:`Reduce the size of the root filesystem
383 <dev-manual/building:trim the root filesystem>`
384
385- :ref:`Reduce the size of the kernel <dev-manual/building:trim the kernel>`
386
387- :ref:`dev-manual/building:remove package management requirements`
388
389- :ref:`dev-manual/building:look for other ways to minimize size`
390
391- :ref:`dev-manual/building:iterate on the process`
392
393Goals and Guiding Principles
394----------------------------
395
396Before you can reach your destination, you need to know where you are
397going. Here is an example list that you can use as a guide when creating
398very small distributions:
399
400- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
401 less and a root filesystem that is 3 Mbytes or less).
402
403- Find the areas that are currently taking 90% of the space and
404 concentrate on reducing those areas.
405
406- Do not create any difficult "hacks" to achieve your goals.
407
408- Leverage the device-specific options.
409
410- Work in a separate layer so that you keep changes isolated. For
411 information on how to create layers, see the
412 ":ref:`dev-manual/layers:understanding and creating layers`" section.
413
414Understand What Contributes to Your Image Size
415----------------------------------------------
416
417It is easiest to have something to start with when creating your own
418distribution. You can use the Yocto Project out-of-the-box to create the
419``poky-tiny`` distribution. Ultimately, you will want to make changes in
420your own distribution that are likely modeled after ``poky-tiny``.
421
422.. note::
423
424 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
425 ``local.conf`` file to "poky-tiny" as described in the
426 ":ref:`dev-manual/custom-distribution:creating your own distribution`"
427 section.
428
429Understanding some memory concepts will help you reduce the system size.
430Memory consists of static, dynamic, and temporary memory. Static memory
431is the TEXT (code), DATA (initialized data in the code), and BSS
432(uninitialized data) sections. Dynamic memory represents memory that is
433allocated at runtime: stacks, hash tables, and so forth. Temporary
434memory is recovered after the boot process. This memory consists of
435memory used for decompressing the kernel and for the ``__init__``
436functions.
437
438To help you see where you currently are with kernel and root filesystem
439sizes, you can use two tools found in the :term:`Source Directory`
440in the
441``scripts/tiny/`` directory:
442
443- ``ksize.py``: Reports component sizes for the kernel build objects.
444
445- ``dirsize.py``: Reports component sizes for the root filesystem.
446
447This next tool and command help you organize configuration fragments and
448view file dependencies in a human-readable form:
449
450- ``merge_config.sh``: Helps you manage configuration files and
451 fragments within the kernel. With this tool, you can merge individual
452 configuration fragments together. The tool allows you to make
453 overrides and warns you of any missing configuration options. The
454 tool is ideal for allowing you to iterate on configurations, create
455 minimal configurations, and create configuration files for different
456 machines without having to duplicate your process.
457
458 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
459 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
460 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
461 directory.
462
463 For more information on configuration fragments, see the
464 ":ref:`kernel-dev/common:creating configuration fragments`"
465 section in the Yocto Project Linux Kernel Development Manual.
466
467- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
468 with these options brings up a Dependency Explorer from which you can
469 view file dependencies. Understanding these dependencies allows you
470 to make informed decisions when cutting out various pieces of the
471 kernel and root filesystem.
472
473Trim the Root Filesystem
474------------------------
475
476The root filesystem is made up of packages for booting, libraries, and
477applications. To change things, you can configure how the packaging
478happens, which changes the way you build them. You can also modify the
479filesystem itself or select a different filesystem.
480
481First, find out what is hogging your root filesystem by running the
482``dirsize.py`` script from your root directory::
483
484 $ cd root-directory-of-image
485 $ dirsize.py 100000 > dirsize-100k.log
486 $ cat dirsize-100k.log
487
488You can apply a filter to the script to ignore files
489under a certain size. The previous example filters out any files below
490100 Kbytes. The sizes reported by the tool are uncompressed, and thus
491will be smaller by a relatively constant factor in a compressed root
492filesystem. When you examine your log file, you can focus on areas of
493the root filesystem that take up large amounts of memory.
494
495You need to be sure that what you eliminate does not cripple the
496functionality you need. One way to see how packages relate to each other
497is by using the Dependency Explorer UI with the BitBake command::
498
499 $ cd image-directory
500 $ bitbake -u taskexp -g image
501
502Use the interface to
503select potential packages you wish to eliminate and see their dependency
504relationships.
505
506When deciding how to reduce the size, get rid of packages that result in
507minimal impact on the feature set. For example, you might not need a VGA
508display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
509instead of ``udev``.
510
511Use your ``local.conf`` file to make changes. For example, to eliminate
512``udev`` and ``glib``, set the following in the local configuration
513file::
514
515 VIRTUAL-RUNTIME_dev_manager = ""
516
517Finally, you should consider exactly the type of root filesystem you
518need to meet your needs while also reducing its size. For example,
519consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
520:term:`Initramfs` using ``initramfs``. Be aware that ``ext3`` requires a 1
521Mbyte journal. If you are okay with running read-only, you do not need
522this journal.
523
524.. note::
525
526 After each round of elimination, you need to rebuild your system and
527 then use the tools to see the effects of your reductions.
528
529Trim the Kernel
530---------------
531
532The kernel is built by including policies for hardware-independent
533aspects. What subsystems do you enable? For what architecture are you
534building? Which drivers do you build by default?
535
536.. note::
537
538 You can modify the kernel source if you want to help with boot time.
539
540Run the ``ksize.py`` script from the top-level Linux build directory to
541get an idea of what is making up the kernel::
542
543 $ cd top-level-linux-build-directory
544 $ ksize.py > ksize.log
545 $ cat ksize.log
546
547When you examine the log, you will see how much space is taken up with
548the built-in ``.o`` files for drivers, networking, core kernel files,
549filesystem, sound, and so forth. The sizes reported by the tool are
550uncompressed, and thus will be smaller by a relatively constant factor
551in a compressed kernel image. Look to reduce the areas that are large
552and taking up around the "90% rule."
553
554To examine, or drill down, into any particular area, use the ``-d``
555option with the script::
556
557 $ ksize.py -d > ksize.log
558
559Using this option
560breaks out the individual file information for each area of the kernel
561(e.g. drivers, networking, and so forth).
562
563Use your log file to see what you can eliminate from the kernel based on
564features you can let go. For example, if you are not going to need
565sound, you do not need any drivers that support sound.
566
567After figuring out what to eliminate, you need to reconfigure the kernel
568to reflect those changes during the next build. You could run
569``menuconfig`` and make all your changes at once. However, that makes it
570difficult to see the effects of your individual eliminations and also
571makes it difficult to replicate the changes for perhaps another target
572device. A better method is to start with no configurations using
573``allnoconfig``, create configuration fragments for individual changes,
574and then manage the fragments into a single configuration file using
575``merge_config.sh``. The tool makes it easy for you to iterate using the
576configuration change and build cycle.
577
578Each time you make configuration changes, you need to rebuild the kernel
579and check to see what impact your changes had on the overall size.
580
581Remove Package Management Requirements
582--------------------------------------
583
584Packaging requirements add size to the image. One way to reduce the size
585of the image is to remove all the packaging requirements from the image.
586This reduction includes both removing the package manager and its unique
587dependencies as well as removing the package management data itself.
588
589To eliminate all the packaging requirements for an image, be sure that
590"package-management" is not part of your
591:term:`IMAGE_FEATURES`
592statement for the image. When you remove this feature, you are removing
593the package manager as well as its dependencies from the root
594filesystem.
595
596Look for Other Ways to Minimize Size
597------------------------------------
598
599Depending on your particular circumstances, other areas that you can
600trim likely exist. The key to finding these areas is through tools and
601methods described here combined with experimentation and iteration. Here
602are a couple of areas to experiment with:
603
604- ``glibc``: In general, follow this process:
605
606 #. Remove ``glibc`` features from
607 :term:`DISTRO_FEATURES`
608 that you think you do not need.
609
610 #. Build your distribution.
611
612 #. If the build fails due to missing symbols in a package, determine
613 if you can reconfigure the package to not need those features. For
614 example, change the configuration to not support wide character
615 support as is done for ``ncurses``. Or, if support for those
616 characters is needed, determine what ``glibc`` features provide
617 the support and restore the configuration.
618
619 4. Rebuild and repeat the process.
620
621- ``busybox``: For BusyBox, use a process similar as described for
622 ``glibc``. A difference is you will need to boot the resulting system
623 to see if you are able to do everything you expect from the running
624 system. You need to be sure to integrate configuration fragments into
625 Busybox because BusyBox handles its own core features and then allows
626 you to add configuration fragments on top.
627
628Iterate on the Process
629----------------------
630
631If you have not reached your goals on system size, you need to iterate
632on the process. The process is the same. Use the tools and see just what
633is taking up 90% of the root filesystem and the kernel. Decide what you
634can eliminate without limiting your device beyond what you need.
635
636Depending on your system, a good place to look might be Busybox, which
637provides a stripped down version of Unix tools in a single, executable
638file. You might be able to drop virtual terminal services or perhaps
639ipv6.
640
641Building Images for More than One Machine
642=========================================
643
644A common scenario developers face is creating images for several
645different machines that use the same software environment. In this
646situation, it is tempting to set the tunings and optimization flags for
647each build specifically for the targeted hardware (i.e. "maxing out" the
648tunings). Doing so can considerably add to build times and package feed
649maintenance collectively for the machines. For example, selecting tunes
650that are extremely specific to a CPU core used in a system might enable
651some micro optimizations in GCC for that particular system but would
652otherwise not gain you much of a performance difference across the other
653systems as compared to using a more general tuning across all the builds
654(e.g. setting :term:`DEFAULTTUNE`
655specifically for each machine's build). Rather than "max out" each
656build's tunings, you can take steps that cause the OpenEmbedded build
657system to reuse software across the various machines where it makes
658sense.
659
660If build speed and package feed maintenance are considerations, you
661should consider the points in this section that can help you optimize
662your tunings to best consider build times and package feed maintenance.
663
664- *Share the :term:`Build Directory`:* If at all possible, share the
665 :term:`TMPDIR` across builds. The Yocto Project supports switching between
666 different :term:`MACHINE` values in the same :term:`TMPDIR`. This practice
667 is well supported and regularly used by developers when building for
668 multiple machines. When you use the same :term:`TMPDIR` for multiple
669 machine builds, the OpenEmbedded build system can reuse the existing native
670 and often cross-recipes for multiple machines. Thus, build time decreases.
671
672 .. note::
673
674 If :term:`DISTRO` settings change or fundamental configuration settings
675 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
676 Sharing :term:`TMPDIR` under these circumstances might work but since it is
677 not guaranteed, you should use a clean :term:`TMPDIR`.
678
679- *Enable the Appropriate Package Architecture:* By default, the
680 OpenEmbedded build system enables three levels of package
681 architectures: "all", "tune" or "package", and "machine". Any given
682 recipe usually selects one of these package architectures (types) for
683 its output. Depending for what a given recipe creates packages,
684 making sure you enable the appropriate package architecture can
685 directly impact the build time.
686
687 A recipe that just generates scripts can enable "all" architecture
688 because there are no binaries to build. To specifically enable "all"
689 architecture, be sure your recipe inherits the
690 :ref:`ref-classes-allarch` class.
691 This class is useful for "all" architectures because it configures
692 many variables so packages can be used across multiple architectures.
693
694 If your recipe needs to generate packages that are machine-specific
695 or when one of the build or runtime dependencies is already
696 machine-architecture dependent, which makes your recipe also
697 machine-architecture dependent, make sure your recipe enables the
698 "machine" package architecture through the
699 :term:`MACHINE_ARCH`
700 variable::
701
702 PACKAGE_ARCH = "${MACHINE_ARCH}"
703
704 When you do not
705 specifically enable a package architecture through the
706 :term:`PACKAGE_ARCH`, The
707 OpenEmbedded build system defaults to the
708 :term:`TUNE_PKGARCH` setting::
709
710 PACKAGE_ARCH = "${TUNE_PKGARCH}"
711
712- *Choose a Generic Tuning File if Possible:* Some tunes are more
713 generic and can run on multiple targets (e.g. an ``armv5`` set of
714 packages could run on ``armv6`` and ``armv7`` processors in most
715 cases). Similarly, ``i486`` binaries could work on ``i586`` and
716 higher processors. You should realize, however, that advances on
717 newer processor versions would not be used.
718
719 If you select the same tune for several different machines, the
720 OpenEmbedded build system reuses software previously built, thus
721 speeding up the overall build time. Realize that even though a new
722 sysroot for each machine is generated, the software is not recompiled
723 and only one package feed exists.
724
725- *Manage Granular Level Packaging:* Sometimes there are cases where
726 injecting another level of package architecture beyond the three
727 higher levels noted earlier can be useful. For example, consider how
728 NXP (formerly Freescale) allows for the easy reuse of binary packages
729 in their layer
730 :yocto_git:`meta-freescale </meta-freescale/>`.
731 In this example, the
732 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
733 class shares GPU packages for i.MX53 boards because all boards share
734 the AMD GPU. The i.MX6-based boards can do the same because all
735 boards share the Vivante GPU. This class inspects the BitBake
736 datastore to identify if the package provides or depends on one of
737 the sub-architecture values. If so, the class sets the
738 :term:`PACKAGE_ARCH` value
739 based on the ``MACHINE_SUBARCH`` value. If the package does not
740 provide or depend on one of the sub-architecture values but it
741 matches a value in the machine-specific filter, it sets
742 :term:`MACHINE_ARCH`. This
743 behavior reduces the number of packages built and saves build time by
744 reusing binaries.
745
746- *Use Tools to Debug Issues:* Sometimes you can run into situations
747 where software is being rebuilt when you think it should not be. For
748 example, the OpenEmbedded build system might not be using shared
749 state between machines when you think it should be. These types of
750 situations are usually due to references to machine-specific
751 variables such as :term:`MACHINE`,
752 :term:`SERIAL_CONSOLES`,
753 :term:`XSERVER`,
754 :term:`MACHINE_FEATURES`,
755 and so forth in code that is supposed to only be tune-specific or
756 when the recipe depends
757 (:term:`DEPENDS`,
758 :term:`RDEPENDS`,
759 :term:`RRECOMMENDS`,
760 :term:`RSUGGESTS`, and so forth)
761 on some other recipe that already has
762 :term:`PACKAGE_ARCH` defined
763 as "${MACHINE_ARCH}".
764
765 .. note::
766
767 Patches to fix any issues identified are most welcome as these
768 issues occasionally do occur.
769
770 For such cases, you can use some tools to help you sort out the
771 situation:
772
773 - ``state-diff-machines.sh``*:* You can find this tool in the
774 ``scripts`` directory of the Source Repositories. See the comments
775 in the script for information on how to use the tool.
776
777 - *BitBake's "-S printdiff" Option:* Using this option causes
778 BitBake to try to establish the most recent signature match
779 (e.g. in the shared state cache) and then compare matched signatures
780 to determine the stamps and delta where these two stamp trees diverge.
781
782Building Software from an External Source
783=========================================
784
785By default, the OpenEmbedded build system uses the :term:`Build Directory`
786when building source code. The build process involves fetching the source
787files, unpacking them, and then patching them if necessary before the build
788takes place.
789
790There are situations where you might want to build software from source
791files that are external to and thus outside of the OpenEmbedded build
792system. For example, suppose you have a project that includes a new BSP
793with a heavily customized kernel. And, you want to minimize exposing the
794build system to the development team so that they can focus on their
795project and maintain everyone's workflow as much as possible. In this
796case, you want a kernel source directory on the development machine
797where the development occurs. You want the recipe's
798:term:`SRC_URI` variable to point to
799the external directory and use it as is, not copy it.
800
801To build from software that comes from an external source, all you need to do
802is inherit the :ref:`ref-classes-externalsrc` class and then set
803the :term:`EXTERNALSRC` variable to point to your external source code. Here
804are the statements to put in your ``local.conf`` file::
805
806 INHERIT += "externalsrc"
807 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
808
809This next example shows how to accomplish the same thing by setting
810:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
811
812 EXTERNALSRC = "path"
813 EXTERNALSRC_BUILD = "path"
814
815.. note::
816
817 In order for these settings to take effect, you must globally or
818 locally inherit the :ref:`ref-classes-externalsrc` class.
819
820By default, :ref:`ref-classes-externalsrc` builds the source code in a
821directory separate from the external source directory as specified by
822:term:`EXTERNALSRC`. If you need
823to have the source built in the same directory in which it resides, or
824some other nominated directory, you can set
825:term:`EXTERNALSRC_BUILD`
826to point to that directory::
827
828 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
829
830Replicating a Build Offline
831===========================
832
833It can be useful to take a "snapshot" of upstream sources used in a
834build and then use that "snapshot" later to replicate the build offline.
835To do so, you need to first prepare and populate your downloads
836directory your "snapshot" of files. Once your downloads directory is
837ready, you can use it at any time and from any machine to replicate your
838build.
839
840Follow these steps to populate your Downloads directory:
841
842#. *Create a Clean Downloads Directory:* Start with an empty downloads
843 directory (:term:`DL_DIR`). You
844 start with an empty downloads directory by either removing the files
845 in the existing directory or by setting :term:`DL_DIR` to point to either
846 an empty location or one that does not yet exist.
847
848#. *Generate Tarballs of the Source Git Repositories:* Edit your
849 ``local.conf`` configuration file as follows::
850
851 DL_DIR = "/home/your-download-dir/"
852 BB_GENERATE_MIRROR_TARBALLS = "1"
853
854 During
855 the fetch process in the next step, BitBake gathers the source files
856 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
857 the
858 :term:`BB_GENERATE_MIRROR_TARBALLS`
859 variable for more information.
860
861#. *Populate Your Downloads Directory Without Building:* Use BitBake to
862 fetch your sources but inhibit the build::
863
864 $ bitbake target --runonly=fetch
865
866 The downloads directory (i.e. ``${DL_DIR}``) now has
867 a "snapshot" of the source files in the form of tarballs, which can
868 be used for the build.
869
870#. *Optionally Remove Any Git or other SCM Subdirectories From the
871 Downloads Directory:* If you want, you can clean up your downloads
872 directory by removing any Git or other Source Control Management
873 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
874 already contain these subdirectories.
875
876Once your downloads directory has everything it needs regarding source
877files, you can create your "own-mirror" and build your target.
878Understand that you can use the files to build the target offline from
879any machine and at any time.
880
881Follow these steps to build your target using the files in the downloads
882directory:
883
884#. *Using Local Files Only:* Inside your ``local.conf`` file, add the
885 :term:`SOURCE_MIRROR_URL` variable, inherit the
886 :ref:`ref-classes-own-mirrors` class, and use the
887 :term:`BB_NO_NETWORK` variable to your ``local.conf``::
888
889 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
890 INHERIT += "own-mirrors"
891 BB_NO_NETWORK = "1"
892
893 The :term:`SOURCE_MIRROR_URL` and :ref:`ref-classes-own-mirrors`
894 class set up the system to use the downloads directory as your "own
895 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
896 BitBake's fetching process in step 3 stays local, which means files
897 from your "own-mirror" are used.
898
899#. *Start With a Clean Build:* You can start with a clean build by
900 removing the ``${``\ :term:`TMPDIR`\ ``}`` directory or using a new
901 :term:`Build Directory`.
902
903#. *Build Your Target:* Use BitBake to build your target::
904
905 $ bitbake target
906
907 The build completes using the known local "snapshot" of source
908 files from your mirror. The resulting tarballs for your "snapshot" of
909 source files are in the downloads directory.
910
911 .. note::
912
913 The offline build does not work if recipes attempt to find the
914 latest version of software by setting
915 :term:`SRCREV` to
916 ``${``\ :term:`AUTOREV`\ ``}``::
917
918 SRCREV = "${AUTOREV}"
919
920 When a recipe sets :term:`SRCREV` to
921 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
922 attempt to determine the latest version of software from the SCM.
923 Typically, recipes that use :term:`AUTOREV` are custom or modified
924 recipes. Recipes that reside in public repositories usually do not
925 use :term:`AUTOREV`.
926
927 If you do have recipes that use :term:`AUTOREV`, you can take steps to
928 still use the recipes in an offline build. Do the following:
929
930 #. Use a configuration generated by enabling :ref:`build
931 history <dev-manual/build-quality:maintaining build output quality>`.
932
933 #. Use the ``buildhistory-collect-srcrevs`` command to collect the
934 stored :term:`SRCREV` values from the build's history. For more
935 information on collecting these values, see the
936 ":ref:`dev-manual/build-quality:build history package information`"
937 section.
938
939 #. Once you have the correct source revisions, you can modify
940 those recipes to set :term:`SRCREV` to specific versions of the
941 software.
942
diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
deleted file mode 100644
index 65db4aed33..0000000000
--- a/documentation/dev-manual/common-tasks.rst
+++ /dev/null
@@ -1,11587 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3************
4Common Tasks
5************
6
7This chapter describes fundamental procedures such as creating layers,
8adding new software packages, extending or customizing images, porting
9work to new hardware (adding a new machine), and so forth. You will find
10that the procedures documented here occur often in the development cycle
11using the Yocto Project.
12
13Understanding and Creating Layers
14=================================
15
16The OpenEmbedded build system supports organizing
17:term:`Metadata` into multiple layers.
18Layers allow you to isolate different types of customizations from each
19other. For introductory information on the Yocto Project Layer Model,
20see the
21":ref:`overview-manual/yp-intro:the yocto project layer model`"
22section in the Yocto Project Overview and Concepts Manual.
23
24Creating Your Own Layer
25-----------------------
26
27It is very easy to create your own layers to use with the OpenEmbedded
28build system. The Yocto Project ships with tools that speed up creating
29layers. This section describes the steps you perform by hand to create
30layers so that you can better understand them. For information about the
31layer-creation tools, see the
32":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
33section in the Yocto Project Board Support Package (BSP) Developer's
34Guide and the ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
35section further down in this manual.
36
37Follow these general steps to create your layer without using tools:
38
391. *Check Existing Layers:* Before creating a new layer, you should be
40 sure someone has not already created a layer containing the Metadata
41 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
42 for a list of layers from the OpenEmbedded community that can be used in
43 the Yocto Project. You could find a layer that is identical or close
44 to what you need.
45
462. *Create a Directory:* Create the directory for your layer. When you
47 create the layer, be sure to create the directory in an area not
48 associated with the Yocto Project :term:`Source Directory`
49 (e.g. the cloned ``poky`` repository).
50
51 While not strictly required, prepend the name of the directory with
52 the string "meta-". For example:
53 ::
54
55 meta-mylayer
56 meta-GUI_xyz
57 meta-mymachine
58
59 With rare exceptions, a layer's name follows this form:
60 ::
61
62 meta-root_name
63
64 Following this layer naming convention can save
65 you trouble later when tools, components, or variables "assume" your
66 layer name begins with "meta-". A notable example is in configuration
67 files as shown in the following step where layer names without the
68 "meta-" string are appended to several variables used in the
69 configuration.
70
713. *Create a Layer Configuration File:* Inside your new layer folder,
72 you need to create a ``conf/layer.conf`` file. It is easiest to take
73 an existing layer configuration file and copy that to your layer's
74 ``conf`` directory and then modify the file as needed.
75
76 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
77 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
78 demonstrates the required syntax. For your layer, you need to replace
79 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
80 for a layer named "meta-machinexyz"):
81 ::
82
83 # We have a conf and classes directory, add to BBPATH
84 BBPATH .= ":${LAYERDIR}"
85
86 # We have recipes-* directories, add to BBFILES
87 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
88 ${LAYERDIR}/recipes-*/*/*.bbappend"
89
90 BBFILE_COLLECTIONS += "yoctobsp"
91 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
92 BBFILE_PRIORITY_yoctobsp = "5"
93 LAYERVERSION_yoctobsp = "4"
94 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
95
96 Following is an explanation of the layer configuration file:
97
98 - :term:`BBPATH`: Adds the layer's
99 root directory to BitBake's search path. Through the use of the
100 ``BBPATH`` variable, BitBake locates class files (``.bbclass``),
101 configuration files, and files that are included with ``include``
102 and ``require`` statements. For these cases, BitBake uses the
103 first file that matches the name found in ``BBPATH``. This is
104 similar to the way the ``PATH`` variable is used for binaries. It
105 is recommended, therefore, that you use unique class and
106 configuration filenames in your custom layer.
107
108 - :term:`BBFILES`: Defines the
109 location for all recipes in the layer.
110
111 - :term:`BBFILE_COLLECTIONS`:
112 Establishes the current layer through a unique identifier that is
113 used throughout the OpenEmbedded build system to refer to the
114 layer. In this example, the identifier "yoctobsp" is the
115 representation for the container layer named "meta-yocto-bsp".
116
117 - :term:`BBFILE_PATTERN`:
118 Expands immediately during parsing to provide the directory of the
119 layer.
120
121 - :term:`BBFILE_PRIORITY`:
122 Establishes a priority to use for recipes in the layer when the
123 OpenEmbedded build finds recipes of the same name in different
124 layers.
125
126 - :term:`LAYERVERSION`:
127 Establishes a version number for the layer. You can use this
128 version number to specify this exact version of the layer as a
129 dependency when using the
130 :term:`LAYERDEPENDS`
131 variable.
132
133 - :term:`LAYERDEPENDS`:
134 Lists all layers on which this layer depends (if any).
135
136 - :term:`LAYERSERIES_COMPAT`:
137 Lists the :yocto_wiki:`Yocto Project </Releases>`
138 releases for which the current version is compatible. This
139 variable is a good way to indicate if your particular layer is
140 current.
141
1424. *Add Content:* Depending on the type of layer, add the content. If
143 the layer adds support for a machine, add the machine configuration
144 in a ``conf/machine/`` file within the layer. If the layer adds
145 distro policy, add the distro configuration in a ``conf/distro/``
146 file within the layer. If the layer introduces new recipes, put the
147 recipes you need in ``recipes-*`` subdirectories within the layer.
148
149 .. note::
150
151 For an explanation of layer hierarchy that is compliant with the
152 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
153 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
154
1555. *Optionally Test for Compatibility:* If you want permission to use
156 the Yocto Project Compatibility logo with your layer or application
157 that uses your layer, perform the steps to apply for compatibility.
158 See the "`Making Sure Your Layer is Compatible With Yocto
159 Project <#making-sure-your-layer-is-compatible-with-yocto-project>`__"
160 section for more information.
161
162Following Best Practices When Creating Layers
163---------------------------------------------
164
165To create layers that are easier to maintain and that will not impact
166builds for other machines, you should consider the information in the
167following list:
168
169- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
170 Configuration:* In other words, do not copy an entire recipe into
171 your layer and then modify it. Rather, use an append file
172 (``.bbappend``) to override only those parts of the original recipe
173 you need to modify.
174
175- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
176 for each recipe that uses an include file. Or, if you are introducing
177 a new recipe that requires the included file, use the path relative
178 to the original layer directory to refer to the file. For example,
179 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
180 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
181 the include file, it could indicate a deficiency in the include file
182 in the layer to which it originally belongs. If this is the case, you
183 should try to address that deficiency instead of overlaying the
184 include file. For example, you could address this by getting the
185 maintainer of the include file to add a variable or variables to make
186 it easy to override the parts needing to be overridden.
187
188- *Structure Your Layers:* Proper use of overrides within append files
189 and placement of machine-specific files within your layer can ensure
190 that a build is not using the wrong Metadata and negatively impacting
191 a build for a different machine. Following are some examples:
192
193 - *Modify Variables to Support a Different Machine:* Suppose you
194 have a layer named ``meta-one`` that adds support for building
195 machine "one". To do so, you use an append file named
196 ``base-files.bbappend`` and create a dependency on "foo" by
197 altering the :term:`DEPENDS`
198 variable:
199 ::
200
201 DEPENDS = "foo"
202
203 The dependency is created during any
204 build that includes the layer ``meta-one``. However, you might not
205 want this dependency for all machines. For example, suppose you
206 are building for machine "two" but your ``bblayers.conf`` file has
207 the ``meta-one`` layer included. During the build, the
208 ``base-files`` for machine "two" will also have the dependency on
209 ``foo``.
210
211 To make sure your changes apply only when building machine "one",
212 use a machine override with the ``DEPENDS`` statement:
213 ::
214
215 DEPENDS_one = "foo"
216
217 You should follow the same strategy when using ``_append``
218 and ``_prepend`` operations:
219 ::
220
221 DEPENDS_append_one = " foo"
222 DEPENDS_prepend_one = "foo "
223
224 As an actual example, here's a
225 snippet from the generic kernel include file ``linux-yocto.inc``,
226 wherein the kernel compile and link options are adjusted in the
227 case of a subset of the supported architectures:
228 ::
229
230 DEPENDS_append_aarch64 = " libgcc"
231 KERNEL_CC_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
232 KERNEL_LD_append_aarch64 = " ${TOOLCHAIN_OPTIONS}"
233
234 DEPENDS_append_nios2 = " libgcc"
235 KERNEL_CC_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
236 KERNEL_LD_append_nios2 = " ${TOOLCHAIN_OPTIONS}"
237
238 DEPENDS_append_arc = " libgcc"
239 KERNEL_CC_append_arc = " ${TOOLCHAIN_OPTIONS}"
240 KERNEL_LD_append_arc = " ${TOOLCHAIN_OPTIONS}"
241
242 KERNEL_FEATURES_append_qemuall=" features/debug/printk.scc"
243
244 .. note::
245
246 Avoiding "+=" and "=+" and using machine-specific ``_append``
247 and ``_prepend`` operations is recommended as well.
248
249 - *Place Machine-Specific Files in Machine-Specific Locations:* When
250 you have a base recipe, such as ``base-files.bb``, that contains a
251 :term:`SRC_URI` statement to a
252 file, you can use an append file to cause the build to use your
253 own version of the file. For example, an append file in your layer
254 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
255 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows:
256 ::
257
258 FILESEXTRAPATHS_prepend := "${THISDIR}/${BPN}:"
259
260 The build for machine "one" will pick up your machine-specific file as
261 long as you have the file in
262 ``meta-one/recipes-core/base-files/base-files/``. However, if you
263 are building for a different machine and the ``bblayers.conf``
264 file includes the ``meta-one`` layer and the location of your
265 machine-specific file is the first location where that file is
266 found according to ``FILESPATH``, builds for all machines will
267 also use that machine-specific file.
268
269 You can make sure that a machine-specific file is used for a
270 particular machine by putting the file in a subdirectory specific
271 to the machine. For example, rather than placing the file in
272 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
273 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
274 Not only does this make sure the file is used only when building
275 for machine "one", but the build process locates the file more
276 quickly.
277
278 In summary, you need to place all files referenced from
279 ``SRC_URI`` in a machine-specific subdirectory within the layer in
280 order to restrict those files to machine-specific builds.
281
282- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
283 permission to use the Yocto Project Compatibility logo with your
284 layer or application that uses your layer, perform the steps to apply
285 for compatibility. See the "`Making Sure Your Layer is Compatible
286 With Yocto
287 Project <#making-sure-your-layer-is-compatible-with-yocto-project>`__"
288 section for more information.
289
290- *Follow the Layer Naming Convention:* Store custom layers in a Git
291 repository that use the ``meta-layer_name`` format.
292
293- *Group Your Layers Locally:* Clone your repository alongside other
294 cloned ``meta`` directories from the :term:`Source Directory`.
295
296Making Sure Your Layer is Compatible With Yocto Project
297-------------------------------------------------------
298
299When you create a layer used with the Yocto Project, it is advantageous
300to make sure that the layer interacts well with existing Yocto Project
301layers (i.e. the layer is compatible with the Yocto Project). Ensuring
302compatibility makes the layer easy to be consumed by others in the Yocto
303Project community and could allow you permission to use the Yocto
304Project Compatible Logo.
305
306.. note::
307
308 Only Yocto Project member organizations are permitted to use the
309 Yocto Project Compatible Logo. The logo is not available for general
310 use. For information on how to become a Yocto Project member
311 organization, see the :yocto_home:`Yocto Project Website <>`.
312
313The Yocto Project Compatibility Program consists of a layer application
314process that requests permission to use the Yocto Project Compatibility
315Logo for your layer and application. The process consists of two parts:
316
3171. Successfully passing a script (``yocto-check-layer``) that when run
318 against your layer, tests it against constraints based on experiences
319 of how layers have worked in the real world and where pitfalls have
320 been found. Getting a "PASS" result from the script is required for
321 successful compatibility registration.
322
3232. Completion of an application acceptance form, which you can find at
324 :yocto_home:`/webform/yocto-project-compatible-registration`.
325
326To be granted permission to use the logo, you need to satisfy the
327following:
328
329- Be able to check the box indicating that you got a "PASS" when
330 running the script against your layer.
331
332- Answer "Yes" to the questions on the form or have an acceptable
333 explanation for any questions answered "No".
334
335- Be a Yocto Project Member Organization.
336
337The remainder of this section presents information on the registration
338form and on the ``yocto-check-layer`` script.
339
340Yocto Project Compatible Program Application
341~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
342
343Use the form to apply for your layer's approval. Upon successful
344application, you can use the Yocto Project Compatibility Logo with your
345layer and the application that uses your layer.
346
347To access the form, use this link:
348:yocto_home:`/webform/yocto-project-compatible-registration`.
349Follow the instructions on the form to complete your application.
350
351The application consists of the following sections:
352
353- *Contact Information:* Provide your contact information as the fields
354 require. Along with your information, provide the released versions
355 of the Yocto Project for which your layer is compatible.
356
357- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
358 items in the checklist. Space exists at the bottom of the form for
359 any explanations for items for which you answered "No".
360
361- *Recommendations:* Provide answers for the questions regarding Linux
362 kernel use and build success.
363
364``yocto-check-layer`` Script
365~~~~~~~~~~~~~~~~~~~~~~~~~~~~
366
367The ``yocto-check-layer`` script provides you a way to assess how
368compatible your layer is with the Yocto Project. You should run this
369script prior to using the form to apply for compatibility as described
370in the previous section. You need to achieve a "PASS" result in order to
371have your application form successfully processed.
372
373The script divides tests into three areas: COMMON, BSP, and DISTRO. For
374example, given a distribution layer (DISTRO), the layer must pass both
375the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
376layer, the layer must pass the COMMON and BSP set of tests.
377
378To execute the script, enter the following commands from your build
379directory:
380::
381
382 $ source oe-init-build-env
383 $ yocto-check-layer your_layer_directory
384
385Be sure to provide the actual directory for your
386layer as part of the command.
387
388Entering the command causes the script to determine the type of layer
389and then to execute a set of specific tests against the layer. The
390following list overviews the test:
391
392- ``common.test_readme``: Tests if a ``README`` file exists in the
393 layer and the file is not empty.
394
395- ``common.test_parse``: Tests to make sure that BitBake can parse the
396 files without error (i.e. ``bitbake -p``).
397
398- ``common.test_show_environment``: Tests that the global or per-recipe
399 environment is in order without errors (i.e. ``bitbake -e``).
400
401- ``common.test_world``: Verifies that ``bitbake world`` works.
402
403- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
404 layers do not come with recipes that change signatures.
405
406- ``common.test_layerseries_compat``: Verifies layer compatibility is
407 set properly.
408
409- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
410 configurations.
411
412- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
413 set the machine when the layer is added.
414
415- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
416 regardless of which machine is selected.
417
418- ``bsp.test_machine_signatures``: Verifies that building for a
419 particular machine affects only the signature of tasks specific to
420 that machine.
421
422- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
423 distro configurations.
424
425- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
426 does not set the distribution when the layer is added.
427
428Enabling Your Layer
429-------------------
430
431Before the OpenEmbedded build system can use your new layer, you need to
432enable it. To enable your layer, simply add your layer's path to the
433``BBLAYERS`` variable in your ``conf/bblayers.conf`` file, which is
434found in the :term:`Build Directory`.
435The following example shows how to enable a layer named
436``meta-mylayer``:
437::
438
439 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
440 # changes incompatibly
441 POKY_BBLAYERS_CONF_VERSION = "2"
442 BBPATH = "${TOPDIR}"
443 BBFILES ?= ""
444 BBLAYERS ?= " \
445 /home/user/poky/meta \
446 /home/user/poky/meta-poky \
447 /home/user/poky/meta-yocto-bsp \
448 /home/user/poky/meta-mylayer \
449 "
450
451BitBake parses each ``conf/layer.conf`` file from the top down as
452specified in the ``BBLAYERS`` variable within the ``conf/bblayers.conf``
453file. During the processing of each ``conf/layer.conf`` file, BitBake
454adds the recipes, classes and configurations contained within the
455particular layer to the source directory.
456
457Using .bbappend Files in Your Layer
458-----------------------------------
459
460A recipe that appends Metadata to another recipe is called a BitBake
461append file. A BitBake append file uses the ``.bbappend`` file type
462suffix, while the corresponding recipe to which Metadata is being
463appended uses the ``.bb`` file type suffix.
464
465You can use a ``.bbappend`` file in your layer to make additions or
466changes to the content of another layer's recipe without having to copy
467the other layer's recipe into your layer. Your ``.bbappend`` file
468resides in your layer, while the main ``.bb`` recipe file to which you
469are appending Metadata resides in a different layer.
470
471Being able to append information to an existing recipe not only avoids
472duplication, but also automatically applies recipe changes from a
473different layer into your layer. If you were copying recipes, you would
474have to manually merge changes as they occur.
475
476When you create an append file, you must use the same root name as the
477corresponding recipe file. For example, the append file
478``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
479means the original recipe and append file names are version
480number-specific. If the corresponding recipe is renamed to update to a
481newer version, you must also rename and possibly update the
482corresponding ``.bbappend`` as well. During the build process, BitBake
483displays an error on starting if it detects a ``.bbappend`` file that
484does not have a corresponding recipe with a matching name. See the
485:term:`BB_DANGLINGAPPENDS_WARNONLY`
486variable for information on how to handle this error.
487
488As an example, consider the main formfactor recipe and a corresponding
489formfactor append file both from the :term:`Source Directory`.
490Here is the main
491formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
492the "meta" layer at ``meta/recipes-bsp/formfactor``:
493::
494
495 SUMMARY = "Device formfactor information"
496 DESCRIPTION = "A formfactor configuration file provides information about the \
497 target hardware for which the image is being built and information that the \
498 build system cannot obtain from other sources such as the kernel."
499 SECTION = "base"
500 LICENSE = "MIT"
501 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
502 PR = "r45"
503
504 SRC_URI = "file://config file://machconfig"
505 S = "${WORKDIR}"
506
507 PACKAGE_ARCH = "${MACHINE_ARCH}"
508 INHIBIT_DEFAULT_DEPS = "1"
509
510 do_install() {
511 # Install file only if it has contents
512 install -d ${D}${sysconfdir}/formfactor/
513 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
514 if [ -s "${S}/machconfig" ]; then
515 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
516 fi
517 }
518
519In the main recipe, note the :term:`SRC_URI`
520variable, which tells the OpenEmbedded build system where to find files
521during the build.
522
523Following is the append file, which is named ``formfactor_0.0.bbappend``
524and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
525file is in the layer at ``recipes-bsp/formfactor``:
526::
527
528 FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
529
530By default, the build system uses the
531:term:`FILESPATH` variable to
532locate files. This append file extends the locations by setting the
533:term:`FILESEXTRAPATHS`
534variable. Setting this variable in the ``.bbappend`` file is the most
535reliable and recommended method for adding directories to the search
536path used by the build system to find files.
537
538The statement in this example extends the directories to include
539``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
540which resolves to a directory named ``formfactor`` in the same directory
541in which the append file resides (i.e.
542``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
543have the supporting directory structure set up that will contain any
544files or patches you will be including from the layer.
545
546Using the immediate expansion assignment operator ``:=`` is important
547because of the reference to ``THISDIR``. The trailing colon character is
548important as it ensures that items in the list remain colon-separated.
549
550.. note::
551
552 BitBake automatically defines the ``THISDIR`` variable. You should
553 never set this variable yourself. Using "_prepend" as part of the
554 ``FILESEXTRAPATHS`` ensures your path will be searched prior to other
555 paths in the final list.
556
557 Also, not all append files add extra files. Many append files simply
558 exist to add build options (e.g. ``systemd``). For these cases, your
559 append file would not even use the ``FILESEXTRAPATHS`` statement.
560
561Prioritizing Your Layer
562-----------------------
563
564Each layer is assigned a priority value. Priority values control which
565layer takes precedence if there are recipe files with the same name in
566multiple layers. For these cases, the recipe file from the layer with a
567higher priority number takes precedence. Priority values also affect the
568order in which multiple ``.bbappend`` files for the same recipe are
569applied. You can either specify the priority manually, or allow the
570build system to calculate it based on the layer's dependencies.
571
572To specify the layer's priority manually, use the
573:term:`BBFILE_PRIORITY`
574variable and append the layer's root name:
575::
576
577 BBFILE_PRIORITY_mylayer = "1"
578
579.. note::
580
581 It is possible for a recipe with a lower version number
582 :term:`PV` in a layer that has a higher
583 priority to take precedence.
584
585 Also, the layer priority does not currently affect the precedence
586 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
587 might address this.
588
589Managing Layers
590---------------
591
592You can use the BitBake layer management tool ``bitbake-layers`` to
593provide a view into the structure of recipes across a multi-layer
594project. Being able to generate output that reports on configured layers
595with their paths and priorities and on ``.bbappend`` files and their
596applicable recipes can help to reveal potential problems.
597
598For help on the BitBake layer management tool, use the following
599command:
600::
601
602 $ bitbake-layers --help
603 NOTE: Starting bitbake server...
604 usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
605
606 BitBake layers utility
607
608 optional arguments:
609 -d, --debug Enable debug output
610 -q, --quiet Print only errors
611 -F, --force Force add without recipe parse verification
612 --color COLOR Colorize output (where COLOR is auto, always, never)
613 -h, --help show this help message and exit
614
615 subcommands:
616 <subcommand>
617 layerindex-fetch Fetches a layer from a layer index along with its
618 dependent layers, and adds them to conf/bblayers.conf.
619 layerindex-show-depends
620 Find layer dependencies from layer index.
621 add-layer Add one or more layers to bblayers.conf.
622 remove-layer Remove one or more layers from bblayers.conf.
623 flatten flatten layer configuration into a separate output
624 directory.
625 show-layers show current configured layers.
626 show-overlayed list overlayed recipes (where the same recipe exists
627 in another layer)
628 show-recipes list available recipes, showing the layer they are
629 provided by
630 show-appends list bbappend files and recipe files they apply to
631 show-cross-depends Show dependencies between recipes that cross layer
632 boundaries.
633 create-layer Create a basic layer
634
635 Use bitbake-layers <subcommand> --help to get help on a specific command
636
637The following list describes the available commands:
638
639- ``help:`` Displays general help or help on a specified command.
640
641- ``show-layers:`` Shows the current configured layers.
642
643- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
644 when a recipe with the same name exists in another layer that has a
645 higher layer priority.
646
647- ``show-recipes:`` Lists available recipes and the layers that
648 provide them.
649
650- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
651 which they apply.
652
653- ``show-cross-depends:`` Lists dependency relationships between
654 recipes that cross layer boundaries.
655
656- ``add-layer:`` Adds a layer to ``bblayers.conf``.
657
658- ``remove-layer:`` Removes a layer from ``bblayers.conf``
659
660- ``flatten:`` Flattens the layer configuration into a separate
661 output directory. Flattening your layer configuration builds a
662 "flattened" directory that contains the contents of all layers, with
663 any overlayed recipes removed and any ``.bbappend`` files appended to
664 the corresponding recipes. You might have to perform some manual
665 cleanup of the flattened layer as follows:
666
667 - Non-recipe files (such as patches) are overwritten. The flatten
668 command shows a warning for these files.
669
670 - Anything beyond the normal layer setup has been added to the
671 ``layer.conf`` file. Only the lowest priority layer's
672 ``layer.conf`` is used.
673
674 - Overridden and appended items from ``.bbappend`` files need to be
675 cleaned up. The contents of each ``.bbappend`` end up in the
676 flattened recipe. However, if there are appended or changed
677 variable values, you need to tidy these up yourself. Consider the
678 following example. Here, the ``bitbake-layers`` command adds the
679 line ``#### bbappended ...`` so that you know where the following
680 lines originate:
681 ::
682
683 ...
684 DESCRIPTION = "A useful utility"
685 ...
686 EXTRA_OECONF = "--enable-something"
687 ...
688
689 #### bbappended from meta-anotherlayer ####
690
691 DESCRIPTION = "Customized utility"
692 EXTRA_OECONF += "--enable-somethingelse"
693
694
695 Ideally, you would tidy up these utilities as follows:
696 ::
697
698 ...
699 DESCRIPTION = "Customized utility"
700 ...
701 EXTRA_OECONF = "--enable-something --enable-somethingelse"
702 ...
703
704- ``layerindex-fetch``: Fetches a layer from a layer index, along
705 with its dependent layers, and adds the layers to the
706 ``conf/bblayers.conf`` file.
707
708- ``layerindex-show-depends``: Finds layer dependencies from the
709 layer index.
710
711- ``create-layer``: Creates a basic layer.
712
713Creating a General Layer Using the ``bitbake-layers`` Script
714------------------------------------------------------------
715
716The ``bitbake-layers`` script with the ``create-layer`` subcommand
717simplifies creating a new general layer.
718
719.. note::
720
721 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
722 section in the Yocto
723 Project Board Specific (BSP) Developer's Guide.
724
725 - In order to use a layer with the OpenEmbedded build system, you
726 need to add the layer to your ``bblayers.conf`` configuration
727 file. See the ":ref:`dev-manual/common-tasks:adding a layer using the \`\`bitbake-layers\`\` script`"
728 section for more information.
729
730The default mode of the script's operation with this subcommand is to
731create a layer with the following:
732
733- A layer priority of 6.
734
735- A ``conf`` subdirectory that contains a ``layer.conf`` file.
736
737- A ``recipes-example`` subdirectory that contains a further
738 subdirectory named ``example``, which contains an ``example.bb``
739 recipe file.
740
741- A ``COPYING.MIT``, which is the license statement for the layer. The
742 script assumes you want to use the MIT license, which is typical for
743 most layers, for the contents of the layer itself.
744
745- A ``README`` file, which is a file describing the contents of your
746 new layer.
747
748In its simplest form, you can use the following command form to create a
749layer. The command creates a layer whose name corresponds to
750"your_layer_name" in the current directory:
751::
752
753 $ bitbake-layers create-layer your_layer_name
754
755As an example, the following command creates a layer named ``meta-scottrif``
756in your home directory:
757::
758
759 $ cd /usr/home
760 $ bitbake-layers create-layer meta-scottrif
761 NOTE: Starting bitbake server...
762 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
763
764If you want to set the priority of the layer to other than the default
765value of "6", you can either use the ``--priority`` option or you
766can edit the
767:term:`BBFILE_PRIORITY` value
768in the ``conf/layer.conf`` after the script creates it. Furthermore, if
769you want to give the example recipe file some name other than the
770default, you can use the ``--example-recipe-name`` option.
771
772The easiest way to see how the ``bitbake-layers create-layer`` command
773works is to experiment with the script. You can also read the usage
774information by entering the following:
775::
776
777 $ bitbake-layers create-layer --help
778 NOTE: Starting bitbake server...
779 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
780 [--example-recipe-name EXAMPLERECIPE]
781 layerdir
782
783 Create a basic layer
784
785 positional arguments:
786 layerdir Layer directory to create
787
788 optional arguments:
789 -h, --help show this help message and exit
790 --priority PRIORITY, -p PRIORITY
791 Layer directory to create
792 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
793 Filename of the example recipe
794
795Adding a Layer Using the ``bitbake-layers`` Script
796--------------------------------------------------
797
798Once you create your general layer, you must add it to your
799``bblayers.conf`` file. Adding the layer to this configuration file
800makes the OpenEmbedded build system aware of your layer so that it can
801search it for metadata.
802
803Add your layer by using the ``bitbake-layers add-layer`` command:
804::
805
806 $ bitbake-layers add-layer your_layer_name
807
808Here is an example that adds a
809layer named ``meta-scottrif`` to the configuration file. Following the
810command that adds the layer is another ``bitbake-layers`` command that
811shows the layers that are in your ``bblayers.conf`` file:
812::
813
814 $ bitbake-layers add-layer meta-scottrif
815 NOTE: Starting bitbake server...
816 Parsing recipes: 100% |##########################################################| Time: 0:00:49
817 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
818 $ bitbake-layers show-layers
819 NOTE: Starting bitbake server...
820 layer path priority
821 ==========================================================================
822 meta /home/scottrif/poky/meta 5
823 meta-poky /home/scottrif/poky/meta-poky 5
824 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
825 workspace /home/scottrif/poky/build/workspace 99
826 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
827
828
829Adding the layer to this file
830enables the build system to locate the layer during the build.
831
832.. note::
833
834 During a build, the OpenEmbedded build system looks in the layers
835 from the top of the list down to the bottom in that order.
836
837Customizing Images
838==================
839
840You can customize images to satisfy particular requirements. This
841section describes several methods and provides guidelines for each.
842
843Customizing Images Using ``local.conf``
844---------------------------------------
845
846Probably the easiest way to customize an image is to add a package by
847way of the ``local.conf`` configuration file. Because it is limited to
848local use, this method generally only allows you to add packages and is
849not as flexible as creating your own customized image. When you add
850packages using local variables this way, you need to realize that these
851variable changes are in effect for every build and consequently affect
852all images, which might not be what you require.
853
854To add a package to your image using the local configuration file, use
855the ``IMAGE_INSTALL`` variable with the ``_append`` operator:
856::
857
858 IMAGE_INSTALL_append = " strace"
859
860Use of the syntax is important -
861specifically, the space between the quote and the package name, which is
862``strace`` in this example. This space is required since the ``_append``
863operator does not add the space.
864
865Furthermore, you must use ``_append`` instead of the ``+=`` operator if
866you want to avoid ordering issues. The reason for this is because doing
867so unconditionally appends to the variable and avoids ordering problems
868due to the variable being set in image recipes and ``.bbclass`` files
869with operators like ``?=``. Using ``_append`` ensures the operation
870takes effect.
871
872As shown in its simplest use, ``IMAGE_INSTALL_append`` affects all
873images. It is possible to extend the syntax so that the variable applies
874to a specific image only. Here is an example:
875::
876
877 IMAGE_INSTALL_append_pn-core-image-minimal = " strace"
878
879This example adds ``strace`` to the ``core-image-minimal`` image only.
880
881You can add packages using a similar approach through the
882``CORE_IMAGE_EXTRA_INSTALL`` variable. If you use this variable, only
883``core-image-*`` images are affected.
884
885Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
886-------------------------------------------------------------------------------
887
888Another method for customizing your image is to enable or disable
889high-level image features by using the
890:term:`IMAGE_FEATURES` and
891:term:`EXTRA_IMAGE_FEATURES`
892variables. Although the functions for both variables are nearly
893equivalent, best practices dictate using ``IMAGE_FEATURES`` from within
894a recipe and using ``EXTRA_IMAGE_FEATURES`` from within your
895``local.conf`` file, which is found in the
896:term:`Build Directory`.
897
898To understand how these features work, the best reference is
899``meta/classes/core-image.bbclass``. This class lists out the available
900``IMAGE_FEATURES`` of which most map to package groups while some, such
901as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
902configuration settings.
903
904In summary, the file looks at the contents of the ``IMAGE_FEATURES``
905variable and then maps or configures the feature accordingly. Based on
906this information, the build system automatically adds the appropriate
907packages or configurations to the
908:term:`IMAGE_INSTALL` variable.
909Effectively, you are enabling extra features by extending the class or
910creating a custom class for use with specialized image ``.bb`` files.
911
912Use the ``EXTRA_IMAGE_FEATURES`` variable from within your local
913configuration file. Using a separate area from which to enable features
914with this variable helps you avoid overwriting the features in the image
915recipe that are enabled with ``IMAGE_FEATURES``. The value of
916``EXTRA_IMAGE_FEATURES`` is added to ``IMAGE_FEATURES`` within
917``meta/conf/bitbake.conf``.
918
919To illustrate how you can use these variables to modify your image,
920consider an example that selects the SSH server. The Yocto Project ships
921with two SSH servers you can use with your images: Dropbear and OpenSSH.
922Dropbear is a minimal SSH server appropriate for resource-constrained
923environments, while OpenSSH is a well-known standard SSH server
924implementation. By default, the ``core-image-sato`` image is configured
925to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
926images both include OpenSSH. The ``core-image-minimal`` image does not
927contain an SSH server.
928
929You can customize your image and change these defaults. Edit the
930``IMAGE_FEATURES`` variable in your recipe or use the
931``EXTRA_IMAGE_FEATURES`` in your ``local.conf`` file so that it
932configures the image you are working with to include
933``ssh-server-dropbear`` or ``ssh-server-openssh``.
934
935.. note::
936
937 See the ":ref:`ref-manual/features:image features`" section in the Yocto
938 Project Reference Manual for a complete list of image features that ship
939 with the Yocto Project.
940
941Customizing Images Using Custom .bb Files
942-----------------------------------------
943
944You can also customize an image by creating a custom recipe that defines
945additional software as part of the image. The following example shows
946the form for the two lines you need:
947::
948
949 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
950 inherit core-image
951
952Defining the software using a custom recipe gives you total control over
953the contents of the image. It is important to use the correct names of
954packages in the ``IMAGE_INSTALL`` variable. You must use the
955OpenEmbedded notation and not the Debian notation for the names (e.g.
956``glibc-dev`` instead of ``libc6-dev``).
957
958The other method for creating a custom image is to base it on an
959existing image. For example, if you want to create an image based on
960``core-image-sato`` but add the additional package ``strace`` to the
961image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
962``.bb`` and add the following line to the end of the copy:
963::
964
965 IMAGE_INSTALL += "strace"
966
967Customizing Images Using Custom Package Groups
968----------------------------------------------
969
970For complex custom images, the best approach for customizing an image is
971to create a custom package group recipe that is used to build the image
972or images. A good example of a package group recipe is
973``meta/recipes-core/packagegroups/packagegroup-base.bb``.
974
975If you examine that recipe, you see that the ``PACKAGES`` variable lists
976the package group packages to produce. The ``inherit packagegroup``
977statement sets appropriate default values and automatically adds
978``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
979package specified in the ``PACKAGES`` statement.
980
981.. note::
982
983 The ``inherit packagegroup`` line should be located near the top of the
984 recipe, certainly before the ``PACKAGES`` statement.
985
986For each package you specify in ``PACKAGES``, you can use ``RDEPENDS``
987and ``RRECOMMENDS`` entries to provide a list of packages the parent
988task package should contain. You can see examples of these further down
989in the ``packagegroup-base.bb`` recipe.
990
991Here is a short, fabricated example showing the same basic pieces for a
992hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
993the variable ``PN`` is the standard way to abbreviate the reference to
994the full packagegroup name ``packagegroup-custom``:
995::
996
997 DESCRIPTION = "My Custom Package Groups"
998
999 inherit packagegroup
1000
1001 PACKAGES = "\
1002 ${PN}-apps \
1003 ${PN}-tools \
1004 "
1005
1006 RDEPENDS_${PN}-apps = "\
1007 dropbear \
1008 portmap \
1009 psplash"
1010
1011 RDEPENDS_${PN}-tools = "\
1012 oprofile \
1013 oprofileui-server \
1014 lttng-tools"
1015
1016 RRECOMMENDS_${PN}-tools = "\
1017 kernel-module-oprofile"
1018
1019In the previous example, two package group packages are created with
1020their dependencies and their recommended package dependencies listed:
1021``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
1022build an image using these package group packages, you need to add
1023``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
1024``IMAGE_INSTALL``. For other forms of image dependencies see the other
1025areas of this section.
1026
1027Customizing an Image Hostname
1028-----------------------------
1029
1030By default, the configured hostname (i.e. ``/etc/hostname``) in an image
1031is the same as the machine name. For example, if
1032:term:`MACHINE` equals "qemux86", the
1033configured hostname written to ``/etc/hostname`` is "qemux86".
1034
1035You can customize this name by altering the value of the "hostname"
1036variable in the ``base-files`` recipe using either an append file or a
1037configuration file. Use the following in an append file:
1038::
1039
1040 hostname = "myhostname"
1041
1042Use the following in a configuration file:
1043::
1044
1045 hostname_pn-base-files = "myhostname"
1046
1047Changing the default value of the variable "hostname" can be useful in
1048certain situations. For example, suppose you need to do extensive
1049testing on an image and you would like to easily identify the image
1050under test from existing images with typical default hostnames. In this
1051situation, you could change the default hostname to "testme", which
1052results in all the images using the name "testme". Once testing is
1053complete and you do not need to rebuild the image for test any longer,
1054you can easily reset the default hostname.
1055
1056Another point of interest is that if you unset the variable, the image
1057will have no default hostname in the filesystem. Here is an example that
1058unsets the variable in a configuration file:
1059::
1060
1061 hostname_pn-base-files = ""
1062
1063Having no default hostname in the filesystem is suitable for
1064environments that use dynamic hostnames such as virtual machines.
1065
1066Writing a New Recipe
1067====================
1068
1069Recipes (``.bb`` files) are fundamental components in the Yocto Project
1070environment. Each software component built by the OpenEmbedded build
1071system requires a recipe to define the component. This section describes
1072how to create, write, and test a new recipe.
1073
1074.. note::
1075
1076 For information on variables that are useful for recipes and for
1077 information about recipe naming issues, see the
1078 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
1079 Reference Manual.
1080
1081Overview
1082--------
1083
1084The following figure shows the basic process for creating a new recipe.
1085The remainder of the section provides details for the steps.
1086
1087.. image:: figures/recipe-workflow.png
1088 :align: center
1089
1090Locate or Automatically Create a Base Recipe
1091--------------------------------------------
1092
1093You can always write a recipe from scratch. However, three choices exist
1094that can help you quickly get a start on a new recipe:
1095
1096- ``devtool add``: A command that assists in creating a recipe and an
1097 environment conducive to development.
1098
1099- ``recipetool create``: A command provided by the Yocto Project that
1100 automates creation of a base recipe based on the source files.
1101
1102- *Existing Recipes:* Location and modification of an existing recipe
1103 that is similar in function to the recipe you need.
1104
1105.. note::
1106
1107 For information on recipe syntax, see the
1108 ":ref:`dev-manual/common-tasks:recipe syntax`" section.
1109
1110Creating the Base Recipe Using ``devtool add``
1111~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1112
1113The ``devtool add`` command uses the same logic for auto-creating the
1114recipe as ``recipetool create``, which is listed below. Additionally,
1115however, ``devtool add`` sets up an environment that makes it easy for
1116you to patch the source and to make changes to the recipe as is often
1117necessary when adding a recipe to build a new piece of software to be
1118included in a build.
1119
1120You can find a complete description of the ``devtool add`` command in
1121the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
1122in the Yocto Project Application Development and the Extensible Software
1123Development Kit (eSDK) manual.
1124
1125Creating the Base Recipe Using ``recipetool create``
1126~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1127
1128``recipetool create`` automates creation of a base recipe given a set of
1129source code files. As long as you can extract or point to the source
1130files, the tool will construct a recipe and automatically configure all
1131pre-build information into the recipe. For example, suppose you have an
1132application that builds using Autotools. Creating the base recipe using
1133``recipetool`` results in a recipe that has the pre-build dependencies,
1134license requirements, and checksums configured.
1135
1136To run the tool, you just need to be in your
1137:term:`Build Directory` and have sourced the
1138build environment setup script (i.e.
1139:ref:`structure-core-script`).
1140To get help on the tool, use the following command:
1141::
1142
1143 $ recipetool -h
1144 NOTE: Starting bitbake server...
1145 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
1146
1147 OpenEmbedded recipe tool
1148
1149 options:
1150 -d, --debug Enable debug output
1151 -q, --quiet Print only errors
1152 --color COLOR Colorize output (where COLOR is auto, always, never)
1153 -h, --help show this help message and exit
1154
1155 subcommands:
1156 create Create a new recipe
1157 newappend Create a bbappend for the specified target in the specified
1158 layer
1159 setvar Set a variable within a recipe
1160 appendfile Create/update a bbappend to replace a target file
1161 appendsrcfiles Create/update a bbappend to add or replace source files
1162 appendsrcfile Create/update a bbappend to add or replace a source file
1163 Use recipetool <subcommand> --help to get help on a specific command
1164
1165Running ``recipetool create -o OUTFILE`` creates the base recipe and
1166locates it properly in the layer that contains your source files.
1167Following are some syntax examples:
1168
1169 - Use this syntax to generate a recipe based on source. Once generated,
1170 the recipe resides in the existing source code layer:
1171 ::
1172
1173 recipetool create -o OUTFILE source
1174
1175 - Use this syntax to generate a recipe using code that
1176 you extract from source. The extracted code is placed in its own layer
1177 defined by ``EXTERNALSRC``.
1178 ::
1179
1180 recipetool create -o OUTFILE -x EXTERNALSRC source
1181
1182 - Use this syntax to generate a recipe based on source. The options
1183 direct ``recipetool`` to generate debugging information. Once generated,
1184 the recipe resides in the existing source code layer:
1185 ::
1186
1187 recipetool create -d -o OUTFILE source
1188
1189Locating and Using a Similar Recipe
1190~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1191
1192Before writing a recipe from scratch, it is often useful to discover
1193whether someone else has already written one that meets (or comes close
1194to meeting) your needs. The Yocto Project and OpenEmbedded communities
1195maintain many recipes that might be candidates for what you are doing.
1196You can find a good central index of these recipes in the
1197:oe_layerindex:`OpenEmbedded Layer Index <>`.
1198
1199Working from an existing recipe or a skeleton recipe is the best way to
1200get started. Here are some points on both methods:
1201
1202- *Locate and modify a recipe that is close to what you want to do:*
1203 This method works when you are familiar with the current recipe
1204 space. The method does not work so well for those new to the Yocto
1205 Project or writing recipes.
1206
1207 Some risks associated with this method are using a recipe that has
1208 areas totally unrelated to what you are trying to accomplish with
1209 your recipe, not recognizing areas of the recipe that you might have
1210 to add from scratch, and so forth. All these risks stem from
1211 unfamiliarity with the existing recipe space.
1212
1213- *Use and modify the following skeleton recipe:* If for some reason
1214 you do not want to use ``recipetool`` and you cannot find an existing
1215 recipe that is close to meeting your needs, you can use the following
1216 structure to provide the fundamental areas of a new recipe.
1217 ::
1218
1219 DESCRIPTION = ""
1220 HOMEPAGE = ""
1221 LICENSE = ""
1222 SECTION = ""
1223 DEPENDS = ""
1224 LIC_FILES_CHKSUM = ""
1225
1226 SRC_URI = ""
1227
1228Storing and Naming the Recipe
1229-----------------------------
1230
1231Once you have your base recipe, you should put it in your own layer and
1232name it appropriately. Locating it correctly ensures that the
1233OpenEmbedded build system can find it when you use BitBake to process
1234the recipe.
1235
1236- *Storing Your Recipe:* The OpenEmbedded build system locates your
1237 recipe through the layer's ``conf/layer.conf`` file and the
1238 :term:`BBFILES` variable. This
1239 variable sets up a path from which the build system can locate
1240 recipes. Here is the typical use:
1241 ::
1242
1243 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
1244 ${LAYERDIR}/recipes-*/*/*.bbappend"
1245
1246 Consequently, you need to be sure you locate your new recipe inside
1247 your layer such that it can be found.
1248
1249 You can find more information on how layers are structured in the
1250 "`Understanding and Creating
1251 Layers <#understanding-and-creating-layers>`__" section.
1252
1253- *Naming Your Recipe:* When you name your recipe, you need to follow
1254 this naming convention:
1255 ::
1256
1257 basename_version.bb
1258
1259 Use lower-cased characters and do not include the reserved suffixes
1260 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
1261 them as part of your recipe name unless the string applies). Here are some
1262 examples:
1263
1264 .. code-block:: none
1265
1266 cups_1.7.0.bb
1267 gawk_4.0.2.bb
1268 irssi_0.8.16-rc1.bb
1269
1270Running a Build on the Recipe
1271-----------------------------
1272
1273Creating a new recipe is usually an iterative process that requires
1274using BitBake to process the recipe multiple times in order to
1275progressively discover and add information to the recipe file.
1276
1277Assuming you have sourced the build environment setup script (i.e.
1278:ref:`structure-core-script`) and you are in
1279the :term:`Build Directory`, use
1280BitBake to process your recipe. All you need to provide is the
1281``basename`` of the recipe as described in the previous section:
1282::
1283
1284 $ bitbake basename
1285
1286During the build, the OpenEmbedded build system creates a temporary work
1287directory for each recipe
1288(``${``\ :term:`WORKDIR`\ ``}``)
1289where it keeps extracted source files, log files, intermediate
1290compilation and packaging files, and so forth.
1291
1292The path to the per-recipe temporary work directory depends on the
1293context in which it is being built. The quickest way to find this path
1294is to have BitBake return it by running the following:
1295::
1296
1297 $ bitbake -e basename | grep ^WORKDIR=
1298
1299As an example, assume a Source Directory
1300top-level folder named ``poky``, a default Build Directory at
1301``poky/build``, and a ``qemux86-poky-linux`` machine target system.
1302Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
1303case, the work directory the build system uses to build the package
1304would be as follows:
1305::
1306
1307 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
1308
1309Inside this directory you can find sub-directories such as ``image``,
1310``packages-split``, and ``temp``. After the build, you can examine these
1311to determine how well the build went.
1312
1313.. note::
1314
1315 You can find log files for each task in the recipe's ``temp``
1316 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
1317 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
1318 ``log.do_fetch``, and ``log.do_compile``).
1319
1320You can find more information about the build process in
1321":doc:`/overview-manual/development-environment`"
1322chapter of the Yocto Project Overview and Concepts Manual.
1323
1324Fetching Code
1325-------------
1326
1327The first thing your recipe must do is specify how to fetch the source
1328files. Fetching is controlled mainly through the
1329:term:`SRC_URI` variable. Your recipe
1330must have a ``SRC_URI`` variable that points to where the source is
1331located. For a graphical representation of source locations, see the
1332":ref:`overview-manual/concepts:sources`" section in
1333the Yocto Project Overview and Concepts Manual.
1334
1335The :ref:`ref-tasks-fetch` task uses
1336the prefix of each entry in the ``SRC_URI`` variable value to determine
1337which :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` to use to get your
1338source files. It is the ``SRC_URI`` variable that triggers the fetcher.
1339The :ref:`ref-tasks-patch` task uses
1340the variable after source is fetched to apply patches. The OpenEmbedded
1341build system uses
1342:term:`FILESOVERRIDES` for
1343scanning directory locations for local files in ``SRC_URI``.
1344
1345The ``SRC_URI`` variable in your recipe must define each unique location
1346for your source files. It is good practice to not hard-code version
1347numbers in a URL used in ``SRC_URI``. Rather than hard-code these
1348values, use ``${``\ :term:`PV`\ ``}``,
1349which causes the fetch process to use the version specified in the
1350recipe filename. Specifying the version in this manner means that
1351upgrading the recipe to a future version is as simple as renaming the
1352recipe to match the new version.
1353
1354Here is a simple example from the
1355``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
1356comes from a single tarball. Notice the use of the
1357:term:`PV` variable:
1358::
1359
1360 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
1361
1362Files mentioned in ``SRC_URI`` whose names end in a typical archive
1363extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
1364forth), are automatically extracted during the
1365:ref:`ref-tasks-unpack` task. For
1366another example that specifies these types of files, see the
1367"`Autotooled Package <#new-recipe-autotooled-package>`__" section.
1368
1369Another way of specifying source is from an SCM. For Git repositories,
1370you must specify :term:`SRCREV` and
1371you should specify :term:`PV` to include
1372the revision with :term:`SRCPV`. Here
1373is an example from the recipe
1374``meta/recipes-kernel/blktrace/blktrace_git.bb``:
1375::
1376
1377 SRCREV = "d6918c8832793b4205ed3bfede78c2f915c23385"
1378
1379 PR = "r6"
1380 PV = "1.0.5+git${SRCPV}"
1381
1382 SRC_URI = "git://git.kernel.dk/blktrace.git \
1383 file://ldflags.patch"
1384
1385If your ``SRC_URI`` statement includes URLs pointing to individual files
1386fetched from a remote server other than a version control system,
1387BitBake attempts to verify the files against checksums defined in your
1388recipe to ensure they have not been tampered with or otherwise modified
1389since the recipe was written. Two checksums are used:
1390``SRC_URI[md5sum]`` and ``SRC_URI[sha256sum]``.
1391
1392If your ``SRC_URI`` variable points to more than a single URL (excluding
1393SCM URLs), you need to provide the ``md5`` and ``sha256`` checksums for
1394each URL. For these cases, you provide a name for each URL as part of
1395the ``SRC_URI`` and then reference that name in the subsequent checksum
1396statements. Here is an example combining lines from the files
1397``git.inc`` and ``git_2.24.1.bb``:
1398::
1399
1400 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
1401 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
1402
1403 SRC_URI[tarball.md5sum] = "166bde96adbbc11c8843d4f8f4f9811b"
1404 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
1405 SRC_URI[manpages.md5sum] = "31c2272a8979022497ba3d4202df145d"
1406 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
1407
1408Proper values for ``md5`` and ``sha256`` checksums might be available
1409with other signatures on the download page for the upstream source (e.g.
1410``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
1411OpenEmbedded build system only deals with ``sha256sum`` and ``md5sum``,
1412you should verify all the signatures you find by hand.
1413
1414If no ``SRC_URI`` checksums are specified when you attempt to build the
1415recipe, or you provide an incorrect checksum, the build will produce an
1416error for each missing or incorrect checksum. As part of the error
1417message, the build system provides the checksum string corresponding to
1418the fetched file. Once you have the correct checksums, you can copy and
1419paste them into your recipe and then run the build again to continue.
1420
1421.. note::
1422
1423 As mentioned, if the upstream source provides signatures for
1424 verifying the downloaded source code, you should verify those
1425 manually before setting the checksum values in the recipe and
1426 continuing with the build.
1427
1428This final example is a bit more complicated and is from the
1429``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
1430example's ``SRC_URI`` statement identifies multiple files as the source
1431files for the recipe: a tarball, a patch file, a desktop file, and an
1432icon.
1433::
1434
1435 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
1436 file://xwc.patch \
1437 file://rxvt.desktop \
1438 file://rxvt.png"
1439
1440When you specify local files using the ``file://`` URI protocol, the
1441build system fetches files from the local machine. The path is relative
1442to the :term:`FILESPATH` variable
1443and searches specific directories in a certain order:
1444``${``\ :term:`BP`\ ``}``,
1445``${``\ :term:`BPN`\ ``}``, and
1446``files``. The directories are assumed to be subdirectories of the
1447directory in which the recipe or append file resides. For another
1448example that specifies these types of files, see the "`Single .c File
1449Package (Hello
1450World!) <#new-recipe-single-c-file-package-hello-world>`__" section.
1451
1452The previous example also specifies a patch file. Patch files are files
1453whose names usually end in ``.patch`` or ``.diff`` but can end with
1454compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
1455The build system automatically applies patches as described in the
1456"`Patching Code <#new-recipe-patching-code>`__" section.
1457
1458Unpacking Code
1459--------------
1460
1461During the build, the
1462:ref:`ref-tasks-unpack` task unpacks
1463the source with ``${``\ :term:`S`\ ``}``
1464pointing to where it is unpacked.
1465
1466If you are fetching your source files from an upstream source archived
1467tarball and the tarball's internal structure matches the common
1468convention of a top-level subdirectory named
1469``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
1470then you do not need to set ``S``. However, if ``SRC_URI`` specifies to
1471fetch source from an archive that does not use this convention, or from
1472an SCM like Git or Subversion, your recipe needs to define ``S``.
1473
1474If processing your recipe using BitBake successfully unpacks the source
1475files, you need to be sure that the directory pointed to by ``${S}``
1476matches the structure of the source.
1477
1478Patching Code
1479-------------
1480
1481Sometimes it is necessary to patch code after it has been fetched. Any
1482files mentioned in ``SRC_URI`` whose names end in ``.patch`` or
1483``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz`` are
1484treated as patches. The
1485:ref:`ref-tasks-patch` task
1486automatically applies these patches.
1487
1488The build system should be able to apply patches with the "-p1" option
1489(i.e. one directory level in the path will be stripped off). If your
1490patch needs to have more directory levels stripped off, specify the
1491number of levels using the "striplevel" option in the ``SRC_URI`` entry
1492for the patch. Alternatively, if your patch needs to be applied in a
1493specific subdirectory that is not specified in the patch file, use the
1494"patchdir" option in the entry.
1495
1496As with all local files referenced in
1497:term:`SRC_URI` using ``file://``,
1498you should place patch files in a directory next to the recipe either
1499named the same as the base name of the recipe
1500(:term:`BP` and
1501:term:`BPN`) or "files".
1502
1503Licensing
1504---------
1505
1506Your recipe needs to have both the
1507:term:`LICENSE` and
1508:term:`LIC_FILES_CHKSUM`
1509variables:
1510
1511- ``LICENSE``: This variable specifies the license for the software.
1512 If you do not know the license under which the software you are
1513 building is distributed, you should go to the source code and look
1514 for that information. Typical files containing this information
1515 include ``COPYING``, ``LICENSE``, and ``README`` files. You could
1516 also find the information near the top of a source file. For example,
1517 given a piece of software licensed under the GNU General Public
1518 License version 2, you would set ``LICENSE`` as follows:
1519 ::
1520
1521 LICENSE = "GPLv2"
1522
1523 The licenses you specify within ``LICENSE`` can have any name as long
1524 as you do not use spaces, since spaces are used as separators between
1525 license names. For standard licenses, use the names of the files in
1526 ``meta/files/common-licenses/`` or the ``SPDXLICENSEMAP`` flag names
1527 defined in ``meta/conf/licenses.conf``.
1528
1529- ``LIC_FILES_CHKSUM``: The OpenEmbedded build system uses this
1530 variable to make sure the license text has not changed. If it has,
1531 the build produces an error and it affords you the chance to figure
1532 it out and correct the problem.
1533
1534 You need to specify all applicable licensing files for the software.
1535 At the end of the configuration step, the build process will compare
1536 the checksums of the files to be sure the text has not changed. Any
1537 differences result in an error with the message containing the
1538 current checksum. For more explanation and examples of how to set the
1539 ``LIC_FILES_CHKSUM`` variable, see the
1540 ":ref:`dev-manual/common-tasks:tracking license changes`" section.
1541
1542 To determine the correct checksum string, you can list the
1543 appropriate files in the ``LIC_FILES_CHKSUM`` variable with incorrect
1544 md5 strings, attempt to build the software, and then note the
1545 resulting error messages that will report the correct md5 strings.
1546 See the "`Fetching Code <#new-recipe-fetching-code>`__" section for
1547 additional information.
1548
1549 Here is an example that assumes the software has a ``COPYING`` file:
1550 ::
1551
1552 LIC_FILES_CHKSUM = "file://COPYING;md5=xxx"
1553
1554 When you try to build the
1555 software, the build system will produce an error and give you the
1556 correct string that you can substitute into the recipe file for a
1557 subsequent build.
1558
1559Dependencies
1560------------
1561
1562Most software packages have a short list of other packages that they
1563require, which are called dependencies. These dependencies fall into two
1564main categories: build-time dependencies, which are required when the
1565software is built; and runtime dependencies, which are required to be
1566installed on the target in order for the software to run.
1567
1568Within a recipe, you specify build-time dependencies using the
1569:term:`DEPENDS` variable. Although
1570nuances exist, items specified in ``DEPENDS`` should be names of other
1571recipes. It is important that you specify all build-time dependencies
1572explicitly.
1573
1574Another consideration is that configure scripts might automatically
1575check for optional dependencies and enable corresponding functionality
1576if those dependencies are found. If you wish to make a recipe that is
1577more generally useful (e.g. publish the recipe in a layer for others to
1578use), instead of hard-disabling the functionality, you can use the
1579:term:`PACKAGECONFIG` variable to allow functionality and the
1580corresponding dependencies to be enabled and disabled easily by other
1581users of the recipe.
1582
1583Similar to build-time dependencies, you specify runtime dependencies
1584through a variable -
1585:term:`RDEPENDS`, which is
1586package-specific. All variables that are package-specific need to have
1587the name of the package added to the end as an override. Since the main
1588package for a recipe has the same name as the recipe, and the recipe's
1589name can be found through the
1590``${``\ :term:`PN`\ ``}`` variable, then
1591you specify the dependencies for the main package by setting
1592``RDEPENDS_${PN}``. If the package were named ``${PN}-tools``, then you
1593would set ``RDEPENDS_${PN}-tools``, and so forth.
1594
1595Some runtime dependencies will be set automatically at packaging time.
1596These dependencies include any shared library dependencies (i.e. if a
1597package "example" contains "libexample" and another package "mypackage"
1598contains a binary that links to "libexample" then the OpenEmbedded build
1599system will automatically add a runtime dependency to "mypackage" on
1600"example"). See the
1601":ref:`overview-manual/concepts:automatically added runtime dependencies`"
1602section in the Yocto Project Overview and Concepts Manual for further
1603details.
1604
1605Configuring the Recipe
1606----------------------
1607
1608Most software provides some means of setting build-time configuration
1609options before compilation. Typically, setting these options is
1610accomplished by running a configure script with options, or by modifying
1611a build configuration file.
1612
1613.. note::
1614
1615 As of Yocto Project Release 1.7, some of the core recipes that
1616 package binary configuration scripts now disable the scripts due to
1617 the scripts previously requiring error-prone path substitution. The
1618 OpenEmbedded build system uses ``pkg-config`` now, which is much more
1619 robust. You can find a list of the ``*-config`` scripts that are disabled
1620 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
1621 in the Yocto Project Reference Manual.
1622
1623A major part of build-time configuration is about checking for
1624build-time dependencies and possibly enabling optional functionality as
1625a result. You need to specify any build-time dependencies for the
1626software you are building in your recipe's
1627:term:`DEPENDS` value, in terms of
1628other recipes that satisfy those dependencies. You can often find
1629build-time or runtime dependencies described in the software's
1630documentation.
1631
1632The following list provides configuration items of note based on how
1633your software is built:
1634
1635- *Autotools:* If your source files have a ``configure.ac`` file, then
1636 your software is built using Autotools. If this is the case, you just
1637 need to worry about modifying the configuration.
1638
1639 When using Autotools, your recipe needs to inherit the
1640 :ref:`autotools <ref-classes-autotools>` class
1641 and your recipe does not have to contain a
1642 :ref:`ref-tasks-configure` task.
1643 However, you might still want to make some adjustments. For example,
1644 you can set
1645 :term:`EXTRA_OECONF` or
1646 :term:`PACKAGECONFIG_CONFARGS`
1647 to pass any needed configure options that are specific to the recipe.
1648
1649- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
1650 your software is built using CMake. If this is the case, you just
1651 need to worry about modifying the configuration.
1652
1653 When you use CMake, your recipe needs to inherit the
1654 :ref:`cmake <ref-classes-cmake>` class and your
1655 recipe does not have to contain a
1656 :ref:`ref-tasks-configure` task.
1657 You can make some adjustments by setting
1658 :term:`EXTRA_OECMAKE` to
1659 pass any needed configure options that are specific to the recipe.
1660
1661 .. note::
1662
1663 If you need to install one or more custom CMake toolchain files
1664 that are supplied by the application you are building, install the
1665 files to ``${D}${datadir}/cmake/Modules`` during ``do_install``.
1666
1667- *Other:* If your source files do not have a ``configure.ac`` or
1668 ``CMakeLists.txt`` file, then your software is built using some
1669 method other than Autotools or CMake. If this is the case, you
1670 normally need to provide a
1671 :ref:`ref-tasks-configure` task
1672 in your recipe unless, of course, there is nothing to configure.
1673
1674 Even if your software is not being built by Autotools or CMake, you
1675 still might not need to deal with any configuration issues. You need
1676 to determine if configuration is even a required step. You might need
1677 to modify a Makefile or some configuration file used for the build to
1678 specify necessary build options. Or, perhaps you might need to run a
1679 provided, custom configure script with the appropriate options.
1680
1681 For the case involving a custom configure script, you would run
1682 ``./configure --help`` and look for the options you need to set.
1683
1684Once configuration succeeds, it is always good practice to look at the
1685``log.do_configure`` file to ensure that the appropriate options have
1686been enabled and no additional build-time dependencies need to be added
1687to ``DEPENDS``. For example, if the configure script reports that it
1688found something not mentioned in ``DEPENDS``, or that it did not find
1689something that it needed for some desired optional functionality, then
1690you would need to add those to ``DEPENDS``. Looking at the log might
1691also reveal items being checked for, enabled, or both that you do not
1692want, or items not being found that are in ``DEPENDS``, in which case
1693you would need to look at passing extra options to the configure script
1694as needed. For reference information on configure options specific to
1695the software you are building, you can consult the output of the
1696``./configure --help`` command within ``${S}`` or consult the software's
1697upstream documentation.
1698
1699Using Headers to Interface with Devices
1700---------------------------------------
1701
1702If your recipe builds an application that needs to communicate with some
1703device or needs an API into a custom kernel, you will need to provide
1704appropriate header files. Under no circumstances should you ever modify
1705the existing
1706``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
1707These headers are used to build ``libc`` and must not be compromised
1708with custom or machine-specific header information. If you customize
1709``libc`` through modified headers all other applications that use
1710``libc`` thus become affected.
1711
1712.. note::
1713
1714 Never copy and customize the ``libc`` header file (i.e.
1715 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
1716
1717The correct way to interface to a device or custom kernel is to use a
1718separate package that provides the additional headers for the driver or
1719other unique interfaces. When doing so, your application also becomes
1720responsible for creating a dependency on that specific provider.
1721
1722Consider the following:
1723
1724- Never modify ``linux-libc-headers.inc``. Consider that file to be
1725 part of the ``libc`` system, and not something you use to access the
1726 kernel directly. You should access ``libc`` through specific ``libc``
1727 calls.
1728
1729- Applications that must talk directly to devices should either provide
1730 necessary headers themselves, or establish a dependency on a special
1731 headers package that is specific to that driver.
1732
1733For example, suppose you want to modify an existing header that adds I/O
1734control or network support. If the modifications are used by a small
1735number programs, providing a unique version of a header is easy and has
1736little impact. When doing so, bear in mind the guidelines in the
1737previous list.
1738
1739.. note::
1740
1741 If for some reason your changes need to modify the behavior of the ``libc``,
1742 and subsequently all other applications on the system, use a ``.bbappend``
1743 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
1744 make the changes machine specific.
1745
1746Consider a case where your kernel is older and you need an older
1747``libc`` ABI. The headers installed by your recipe should still be a
1748standard mainline kernel, not your own custom one.
1749
1750When you use custom kernel headers you need to get them from
1751:term:`STAGING_KERNEL_DIR`,
1752which is the directory with kernel headers that are required to build
1753out-of-tree modules. Your recipe will also need the following:
1754::
1755
1756 do_configure[depends] += "virtual/kernel:do_shared_workdir"
1757
1758Compilation
1759-----------
1760
1761During a build, the ``do_compile`` task happens after source is fetched,
1762unpacked, and configured. If the recipe passes through ``do_compile``
1763successfully, nothing needs to be done.
1764
1765However, if the compile step fails, you need to diagnose the failure.
1766Here are some common issues that cause failures.
1767
1768.. note::
1769
1770 For cases where improper paths are detected for configuration files
1771 or for when libraries/headers cannot be found, be sure you are using
1772 the more robust ``pkg-config``. See the note in section
1773 ":ref:`dev-manual/common-tasks:Configuring the Recipe`" for additional information.
1774
1775- *Parallel build failures:* These failures manifest themselves as
1776 intermittent errors, or errors reporting that a file or directory
1777 that should be created by some other part of the build process could
1778 not be found. This type of failure can occur even if, upon
1779 inspection, the file or directory does exist after the build has
1780 failed, because that part of the build process happened in the wrong
1781 order.
1782
1783 To fix the problem, you need to either satisfy the missing dependency
1784 in the Makefile or whatever script produced the Makefile, or (as a
1785 workaround) set :term:`PARALLEL_MAKE` to an empty string:
1786 ::
1787
1788 PARALLEL_MAKE = ""
1789
1790 For information on parallel Makefile issues, see the "`Debugging
1791 Parallel Make Races <#debugging-parallel-make-races>`__" section.
1792
1793- *Improper host path usage:* This failure applies to recipes building
1794 for the target or ``nativesdk`` only. The failure occurs when the
1795 compilation process uses improper headers, libraries, or other files
1796 from the host system when cross-compiling for the target.
1797
1798 To fix the problem, examine the ``log.do_compile`` file to identify
1799 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
1800 so forth) and then either add configure options, apply a patch, or do
1801 both.
1802
1803- *Failure to find required libraries/headers:* If a build-time
1804 dependency is missing because it has not been declared in
1805 :term:`DEPENDS`, or because the
1806 dependency exists but the path used by the build process to find the
1807 file is incorrect and the configure step did not detect it, the
1808 compilation process could fail. For either of these failures, the
1809 compilation process notes that files could not be found. In these
1810 cases, you need to go back and add additional options to the
1811 configure script as well as possibly add additional build-time
1812 dependencies to ``DEPENDS``.
1813
1814 Occasionally, it is necessary to apply a patch to the source to
1815 ensure the correct paths are used. If you need to specify paths to
1816 find files staged into the sysroot from other recipes, use the
1817 variables that the OpenEmbedded build system provides (e.g.
1818 ``STAGING_BINDIR``, ``STAGING_INCDIR``, ``STAGING_DATADIR``, and so
1819 forth).
1820
1821Installing
1822----------
1823
1824During ``do_install``, the task copies the built files along with their
1825hierarchy to locations that would mirror their locations on the target
1826device. The installation process copies files from the
1827``${``\ :term:`S`\ ``}``,
1828``${``\ :term:`B`\ ``}``, and
1829``${``\ :term:`WORKDIR`\ ``}``
1830directories to the ``${``\ :term:`D`\ ``}``
1831directory to create the structure as it should appear on the target
1832system.
1833
1834How your software is built affects what you must do to be sure your
1835software is installed correctly. The following list describes what you
1836must do for installation depending on the type of build system used by
1837the software being built:
1838
1839- *Autotools and CMake:* If the software your recipe is building uses
1840 Autotools or CMake, the OpenEmbedded build system understands how to
1841 install the software. Consequently, you do not have to have a
1842 ``do_install`` task as part of your recipe. You just need to make
1843 sure the install portion of the build completes with no issues.
1844 However, if you wish to install additional files not already being
1845 installed by ``make install``, you should do this using a
1846 ``do_install_append`` function using the install command as described
1847 in the "Manual" bulleted item later in this list.
1848
1849- *Other (using* ``make install``\ *)*: You need to define a ``do_install``
1850 function in your recipe. The function should call
1851 ``oe_runmake install`` and will likely need to pass in the
1852 destination directory as well. How you pass that path is dependent on
1853 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
1854 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
1855
1856 For an example recipe using ``make install``, see the
1857 "`Makefile-Based Package <#new-recipe-makefile-based-package>`__"
1858 section.
1859
1860- *Manual:* You need to define a ``do_install`` function in your
1861 recipe. The function must first use ``install -d`` to create the
1862 directories under
1863 ``${``\ :term:`D`\ ``}``. Once the
1864 directories exist, your function can use ``install`` to manually
1865 install the built software into the directories.
1866
1867 You can find more information on ``install`` at
1868 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
1869
1870For the scenarios that do not use Autotools or CMake, you need to track
1871the installation and diagnose and fix any issues until everything
1872installs correctly. You need to look in the default location of
1873``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
1874installed correctly.
1875
1876.. note::
1877
1878 - During the installation process, you might need to modify some of
1879 the installed files to suit the target layout. For example, you
1880 might need to replace hard-coded paths in an initscript with
1881 values of variables provided by the build system, such as
1882 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
1883 modifications during ``do_install``, be sure to modify the
1884 destination file after copying rather than before copying.
1885 Modifying after copying ensures that the build system can
1886 re-execute ``do_install`` if needed.
1887
1888 - ``oe_runmake install``, which can be run directly or can be run
1889 indirectly by the
1890 :ref:`autotools <ref-classes-autotools>` and
1891 :ref:`cmake <ref-classes-cmake>` classes,
1892 runs ``make install`` in parallel. Sometimes, a Makefile can have
1893 missing dependencies between targets that can result in race
1894 conditions. If you experience intermittent failures during
1895 ``do_install``, you might be able to work around them by disabling
1896 parallel Makefile installs by adding the following to the recipe:
1897 ::
1898
1899 PARALLEL_MAKEINST = ""
1900
1901 See :term:`PARALLEL_MAKEINST` for additional information.
1902
1903 - If you need to install one or more custom CMake toolchain files
1904 that are supplied by the application you are building, install the
1905 files to ``${D}${datadir}/cmake/Modules`` during
1906 :ref:`ref-tasks-install`.
1907
1908Enabling System Services
1909------------------------
1910
1911If you want to install a service, which is a process that usually starts
1912on boot and runs in the background, then you must include some
1913additional definitions in your recipe.
1914
1915If you are adding services and the service initialization script or the
1916service file itself is not installed, you must provide for that
1917installation in your recipe using a ``do_install_append`` function. If
1918your recipe already has a ``do_install`` function, update the function
1919near its end rather than adding an additional ``do_install_append``
1920function.
1921
1922When you create the installation for your services, you need to
1923accomplish what is normally done by ``make install``. In other words,
1924make sure your installation arranges the output similar to how it is
1925arranged on the target system.
1926
1927The OpenEmbedded build system provides support for starting services two
1928different ways:
1929
1930- *SysVinit:* SysVinit is a system and service manager that manages the
1931 init system used to control the very basic functions of your system.
1932 The init program is the first program started by the Linux kernel
1933 when the system boots. Init then controls the startup, running and
1934 shutdown of all other programs.
1935
1936 To enable a service using SysVinit, your recipe needs to inherit the
1937 :ref:`update-rc.d <ref-classes-update-rc.d>`
1938 class. The class helps facilitate safely installing the package on
1939 the target.
1940
1941 You will need to set the
1942 :term:`INITSCRIPT_PACKAGES`,
1943 :term:`INITSCRIPT_NAME`,
1944 and
1945 :term:`INITSCRIPT_PARAMS`
1946 variables within your recipe.
1947
1948- *systemd:* System Management Daemon (systemd) was designed to replace
1949 SysVinit and to provide enhanced management of services. For more
1950 information on systemd, see the systemd homepage at
1951 https://freedesktop.org/wiki/Software/systemd/.
1952
1953 To enable a service using systemd, your recipe needs to inherit the
1954 :ref:`systemd <ref-classes-systemd>` class. See
1955 the ``systemd.bbclass`` file located in your :term:`Source Directory`
1956 section for
1957 more information.
1958
1959Packaging
1960---------
1961
1962Successful packaging is a combination of automated processes performed
1963by the OpenEmbedded build system and some specific steps you need to
1964take. The following list describes the process:
1965
1966- *Splitting Files*: The ``do_package`` task splits the files produced
1967 by the recipe into logical components. Even software that produces a
1968 single binary might still have debug symbols, documentation, and
1969 other logical components that should be split out. The ``do_package``
1970 task ensures that files are split up and packaged correctly.
1971
1972- *Running QA Checks*: The
1973 :ref:`insane <ref-classes-insane>` class adds a
1974 step to the package generation process so that output quality
1975 assurance checks are generated by the OpenEmbedded build system. This
1976 step performs a range of checks to be sure the build's output is free
1977 of common problems that show up during runtime. For information on
1978 these checks, see the
1979 :ref:`insane <ref-classes-insane>` class and
1980 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
1981 chapter in the Yocto Project Reference Manual.
1982
1983- *Hand-Checking Your Packages*: After you build your software, you
1984 need to be sure your packages are correct. Examine the
1985 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
1986 directory and make sure files are where you expect them to be. If you
1987 discover problems, you can set
1988 :term:`PACKAGES`,
1989 :term:`FILES`,
1990 ``do_install(_append)``, and so forth as needed.
1991
1992- *Splitting an Application into Multiple Packages*: If you need to
1993 split an application into several packages, see the "`Splitting an
1994 Application into Multiple
1995 Packages <#splitting-an-application-into-multiple-packages>`__"
1996 section for an example.
1997
1998- *Installing a Post-Installation Script*: For an example showing how
1999 to install a post-installation script, see the "`Post-Installation
2000 Scripts <#new-recipe-post-installation-scripts>`__" section.
2001
2002- *Marking Package Architecture*: Depending on what your recipe is
2003 building and how it is configured, it might be important to mark the
2004 packages produced as being specific to a particular machine, or to
2005 mark them as not being specific to a particular machine or
2006 architecture at all.
2007
2008 By default, packages apply to any machine with the same architecture
2009 as the target machine. When a recipe produces packages that are
2010 machine-specific (e.g. the
2011 :term:`MACHINE` value is passed
2012 into the configure script or a patch is applied only for a particular
2013 machine), you should mark them as such by adding the following to the
2014 recipe:
2015 ::
2016
2017 PACKAGE_ARCH = "${MACHINE_ARCH}"
2018
2019 On the other hand, if the recipe produces packages that do not
2020 contain anything specific to the target machine or architecture at
2021 all (e.g. recipes that simply package script files or configuration
2022 files), you should use the
2023 :ref:`allarch <ref-classes-allarch>` class to
2024 do this for you by adding this to your recipe:
2025 ::
2026
2027 inherit allarch
2028
2029 Ensuring that the package architecture is correct is not critical
2030 while you are doing the first few builds of your recipe. However, it
2031 is important in order to ensure that your recipe rebuilds (or does
2032 not rebuild) appropriately in response to changes in configuration,
2033 and to ensure that you get the appropriate packages installed on the
2034 target machine, particularly if you run separate builds for more than
2035 one target machine.
2036
2037Sharing Files Between Recipes
2038-----------------------------
2039
2040Recipes often need to use files provided by other recipes on the build
2041host. For example, an application linking to a common library needs
2042access to the library itself and its associated headers. The way this
2043access is accomplished is by populating a sysroot with files. Each
2044recipe has two sysroots in its work directory, one for target files
2045(``recipe-sysroot``) and one for files that are native to the build host
2046(``recipe-sysroot-native``).
2047
2048.. note::
2049
2050 You could find the term "staging" used within the Yocto project
2051 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
2052 variable).
2053
2054Recipes should never populate the sysroot directly (i.e. write files
2055into sysroot). Instead, files should be installed into standard
2056locations during the
2057:ref:`ref-tasks-install` task within
2058the ``${``\ :term:`D`\ ``}`` directory. The
2059reason for this limitation is that almost all files that populate the
2060sysroot are cataloged in manifests in order to ensure the files can be
2061removed later when a recipe is either modified or removed. Thus, the
2062sysroot is able to remain free from stale files.
2063
2064A subset of the files installed by the
2065:ref:`ref-tasks-install` task are
2066used by the
2067:ref:`ref-tasks-populate_sysroot`
2068task as defined by the the
2069:term:`SYSROOT_DIRS` variable to
2070automatically populate the sysroot. It is possible to modify the list of
2071directories that populate the sysroot. The following example shows how
2072you could add the ``/opt`` directory to the list of directories within a
2073recipe:
2074::
2075
2076 SYSROOT_DIRS += "/opt"
2077
2078.. note::
2079
2080 The `/sysroot-only` is to be used by recipes that generate artifacts
2081 that are not included in the target filesystem, allowing them to share
2082 these artifacts without needing to use the ``DEPLOY_DIR``.
2083
2084For a more complete description of the
2085:ref:`ref-tasks-populate_sysroot`
2086task and its associated functions, see the
2087:ref:`staging <ref-classes-staging>` class.
2088
2089Using Virtual Providers
2090-----------------------
2091
2092Prior to a build, if you know that several different recipes provide the
2093same functionality, you can use a virtual provider (i.e. ``virtual/*``)
2094as a placeholder for the actual provider. The actual provider is
2095determined at build-time.
2096
2097A common scenario where a virtual provider is used would be for the
2098kernel recipe. Suppose you have three kernel recipes whose
2099:term:`PN` values map to ``kernel-big``,
2100``kernel-mid``, and ``kernel-small``. Furthermore, each of these recipes
2101in some way uses a :term:`PROVIDES`
2102statement that essentially identifies itself as being able to provide
2103``virtual/kernel``. Here is one way through the
2104:ref:`kernel <ref-classes-kernel>` class:
2105::
2106
2107 PROVIDES += "${@ "virtual/kernel" if (d.getVar("KERNEL_PACKAGE_NAME") == "kernel") else "" }"
2108
2109Any recipe that inherits the ``kernel`` class is
2110going to utilize a ``PROVIDES`` statement that identifies that recipe as
2111being able to provide the ``virtual/kernel`` item.
2112
2113Now comes the time to actually build an image and you need a kernel
2114recipe, but which one? You can configure your build to call out the
2115kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
2116an example, consider the :yocto_git:`x86-base.inc
2117</poky/tree/meta/conf/machine/include/x86-base.inc>` include file, which is a
2118machine (i.e. :term:`MACHINE`) configuration file. This include file is the
2119reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
2120relevant lines from the include file:
2121::
2122
2123 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
2124 PREFERRED_VERSION_linux-yocto ??= "4.15%"
2125
2126When you use a virtual provider, you do not have to "hard code" a recipe
2127name as a build dependency. You can use the
2128:term:`DEPENDS` variable to state the
2129build is dependent on ``virtual/kernel`` for example:
2130::
2131
2132 DEPENDS = "virtual/kernel"
2133
2134During the build, the OpenEmbedded build system picks
2135the correct recipe needed for the ``virtual/kernel`` dependency based on
2136the ``PREFERRED_PROVIDER`` variable. If you want to use the small kernel
2137mentioned at the beginning of this section, configure your build as
2138follows:
2139::
2140
2141 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
2142
2143.. note::
2144
2145 Any recipe that ``PROVIDES`` a ``virtual/*`` item that is ultimately not
2146 selected through ``PREFERRED_PROVIDER`` does not get built. Preventing these
2147 recipes from building is usually the desired behavior since this mechanism's
2148 purpose is to select between mutually exclusive alternative providers.
2149
2150The following lists specific examples of virtual providers:
2151
2152- ``virtual/kernel``: Provides the name of the kernel recipe to use
2153 when building a kernel image.
2154
2155- ``virtual/bootloader``: Provides the name of the bootloader to use
2156 when building an image.
2157
2158- ``virtual/libgbm``: Provides ``gbm.pc``.
2159
2160- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
2161
2162- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
2163
2164- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
2165
2166- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
2167
2168.. note::
2169
2170 Virtual providers only apply to build time dependencies specified with
2171 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
2172 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
2173
2174Properly Versioning Pre-Release Recipes
2175---------------------------------------
2176
2177Sometimes the name of a recipe can lead to versioning problems when the
2178recipe is upgraded to a final release. For example, consider the
2179``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
2180the "`Storing and Naming the
2181Recipe <#new-recipe-storing-and-naming-the-recipe>`__" section. This
2182recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
2183released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
2184change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
2185build system and package managers, so the resulting packages will not
2186correctly trigger an upgrade.
2187
2188In order to ensure the versions compare properly, the recommended
2189convention is to set :term:`PV` within the
2190recipe to "previous_version+current_version". You can use an additional
2191variable so that you can use the current version elsewhere. Here is an
2192example:
2193::
2194
2195 REALPV = "0.8.16-rc1"
2196 PV = "0.8.15+${REALPV}"
2197
2198Post-Installation Scripts
2199-------------------------
2200
2201Post-installation scripts run immediately after installing a package on
2202the target or during image creation when a package is included in an
2203image. To add a post-installation script to a package, add a
2204``pkg_postinst_``\ `PACKAGENAME`\ ``()`` function to the recipe file
2205(``.bb``) and replace `PACKAGENAME` with the name of the package you want
2206to attach to the ``postinst`` script. To apply the post-installation
2207script to the main package for the recipe, which is usually what is
2208required, specify
2209``${``\ :term:`PN`\ ``}`` in place of
2210PACKAGENAME.
2211
2212A post-installation function has the following structure:
2213::
2214
2215 pkg_postinst_PACKAGENAME() {
2216 # Commands to carry out
2217 }
2218
2219The script defined in the post-installation function is called when the
2220root filesystem is created. If the script succeeds, the package is
2221marked as installed.
2222
2223.. note::
2224
2225 Any RPM post-installation script that runs on the target should
2226 return a 0 exit code. RPM does not allow non-zero exit codes for
2227 these scripts, and the RPM package manager will cause the package to
2228 fail installation on the target.
2229
2230Sometimes it is necessary for the execution of a post-installation
2231script to be delayed until the first boot. For example, the script might
2232need to be executed on the device itself. To delay script execution
2233until boot time, you must explicitly mark post installs to defer to the
2234target. You can use ``pkg_postinst_ontarget()`` or call
2235``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
2236failure of a ``pkg_postinst()`` script (including exit 1) triggers an
2237error during the
2238:ref:`ref-tasks-rootfs` task.
2239
2240If you have recipes that use ``pkg_postinst`` function and they require
2241the use of non-standard native tools that have dependencies during
2242rootfs construction, you need to use the
2243:term:`PACKAGE_WRITE_DEPS`
2244variable in your recipe to list these tools. If you do not use this
2245variable, the tools might be missing and execution of the
2246post-installation script is deferred until first boot. Deferring the
2247script to first boot is undesirable and for read-only rootfs impossible.
2248
2249.. note::
2250
2251 Equivalent support for pre-install, pre-uninstall, and post-uninstall
2252 scripts exist by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
2253 respectively. These scrips work in exactly the same way as does
2254 ``pkg_postinst`` with the exception that they run at different times. Also,
2255 because of when they run, they are not applicable to being run at image
2256 creation time like ``pkg_postinst``.
2257
2258Testing
2259-------
2260
2261The final step for completing your recipe is to be sure that the
2262software you built runs correctly. To accomplish runtime testing, add
2263the build's output packages to your image and test them on the target.
2264
2265For information on how to customize your image by adding specific
2266packages, see the "`Customizing
2267Images <#usingpoky-extend-customimage>`__" section.
2268
2269Examples
2270--------
2271
2272To help summarize how to write a recipe, this section provides some
2273examples given various scenarios:
2274
2275- Recipes that use local files
2276
2277- Using an Autotooled package
2278
2279- Using a Makefile-based package
2280
2281- Splitting an application into multiple packages
2282
2283- Adding binaries to an image
2284
2285Single .c File Package (Hello World!)
2286~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2287
2288Building an application from a single file that is stored locally (e.g.
2289under ``files``) requires a recipe that has the file listed in the
2290``SRC_URI`` variable. Additionally, you need to manually write the
2291``do_compile`` and ``do_install`` tasks. The ``S`` variable defines the
2292directory containing the source code, which is set to
2293:term:`WORKDIR` in this case - the
2294directory BitBake uses for the build.
2295::
2296
2297 SUMMARY = "Simple helloworld application"
2298 SECTION = "examples"
2299 LICENSE = "MIT"
2300 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
2301
2302 SRC_URI = "file://helloworld.c"
2303
2304 S = "${WORKDIR}"
2305
2306 do_compile() {
2307 ${CC} ${LDFLAGS} helloworld.c -o helloworld
2308 }
2309
2310 do_install() {
2311 install -d ${D}${bindir}
2312 install -m 0755 helloworld ${D}${bindir}
2313 }
2314
2315By default, the ``helloworld``, ``helloworld-dbg``, and
2316``helloworld-dev`` packages are built. For information on how to
2317customize the packaging process, see the "`Splitting an Application into
2318Multiple Packages <#splitting-an-application-into-multiple-packages>`__"
2319section.
2320
2321Autotooled Package
2322~~~~~~~~~~~~~~~~~~
2323
2324Applications that use Autotools such as ``autoconf`` and ``automake``
2325require a recipe that has a source archive listed in ``SRC_URI`` and
2326also inherit the
2327:ref:`autotools <ref-classes-autotools>` class,
2328which contains the definitions of all the steps needed to build an
2329Autotool-based application. The result of the build is automatically
2330packaged. And, if the application uses NLS for localization, packages
2331with local information are generated (one package per language).
2332Following is one example: (``hello_2.3.bb``)
2333::
2334
2335 SUMMARY = "GNU Helloworld application"
2336 SECTION = "examples"
2337 LICENSE = "GPLv2+"
2338 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
2339
2340 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
2341
2342 inherit autotools gettext
2343
2344The variable ``LIC_FILES_CHKSUM`` is used to track source license
2345changes as described in the
2346":ref:`dev-manual/common-tasks:tracking license changes`" section in
2347the Yocto Project Overview and Concepts Manual. You can quickly create
2348Autotool-based recipes in a manner similar to the previous example.
2349
2350Makefile-Based Package
2351~~~~~~~~~~~~~~~~~~~~~~
2352
2353Applications that use GNU ``make`` also require a recipe that has the
2354source archive listed in ``SRC_URI``. You do not need to add a
2355``do_compile`` step since by default BitBake starts the ``make`` command
2356to compile the application. If you need additional ``make`` options, you
2357should store them in the
2358:term:`EXTRA_OEMAKE` or
2359:term:`PACKAGECONFIG_CONFARGS`
2360variables. BitBake passes these options into the GNU ``make``
2361invocation. Note that a ``do_install`` task is still required.
2362Otherwise, BitBake runs an empty ``do_install`` task by default.
2363
2364Some applications might require extra parameters to be passed to the
2365compiler. For example, the application might need an additional header
2366path. You can accomplish this by adding to the ``CFLAGS`` variable. The
2367following example shows this:
2368::
2369
2370 CFLAGS_prepend = "-I ${S}/include "
2371
2372In the following example, ``mtd-utils`` is a makefile-based package:
2373::
2374
2375 SUMMARY = "Tools for managing memory technology devices"
2376 SECTION = "base"
2377 DEPENDS = "zlib lzo e2fsprogs util-linux"
2378 HOMEPAGE = "http://www.linux-mtd.infradead.org/"
2379 LICENSE = "GPLv2+"
2380 LIC_FILES_CHKSUM = "file://COPYING;md5=0636e73ff0215e8d672dc4c32c317bb3 \
2381 file://include/common.h;beginline=1;endline=17;md5=ba05b07912a44ea2bf81ce409380049c"
2382
2383 # Use the latest version at 26 Oct, 2013
2384 SRCREV = "9f107132a6a073cce37434ca9cda6917dd8d866b"
2385 SRC_URI = "git://git.infradead.org/mtd-utils.git \
2386 file://add-exclusion-to-mkfs-jffs2-git-2.patch \
2387 "
2388
2389 PV = "1.5.1+git${SRCPV}"
2390
2391 S = "${WORKDIR}/git"
2392
2393 EXTRA_OEMAKE = "'CC=${CC}' 'RANLIB=${RANLIB}' 'AR=${AR}' 'CFLAGS=${CFLAGS} -I${S}/include -DWITHOUT_XATTR' 'BUILDDIR=${S}'"
2394
2395 do_install () {
2396 oe_runmake install DESTDIR=${D} SBINDIR=${sbindir} MANDIR=${mandir} INCLUDEDIR=${includedir}
2397 }
2398
2399 PACKAGES =+ "mtd-utils-jffs2 mtd-utils-ubifs mtd-utils-misc"
2400
2401 FILES_mtd-utils-jffs2 = "${sbindir}/mkfs.jffs2 ${sbindir}/jffs2dump ${sbindir}/jffs2reader ${sbindir}/sumtool"
2402 FILES_mtd-utils-ubifs = "${sbindir}/mkfs.ubifs ${sbindir}/ubi*"
2403 FILES_mtd-utils-misc = "${sbindir}/nftl* ${sbindir}/ftl* ${sbindir}/rfd* ${sbindir}/doc* ${sbindir}/serve_image ${sbindir}/recv_image"
2404
2405 PARALLEL_MAKE = ""
2406
2407 BBCLASSEXTEND = "native"
2408
2409Splitting an Application into Multiple Packages
2410~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2411
2412You can use the variables ``PACKAGES`` and ``FILES`` to split an
2413application into multiple packages.
2414
2415Following is an example that uses the ``libxpm`` recipe. By default,
2416this recipe generates a single package that contains the library along
2417with a few binaries. You can modify the recipe to split the binaries
2418into separate packages:
2419::
2420
2421 require xorg-lib-common.inc
2422
2423 SUMMARY = "Xpm: X Pixmap extension library"
2424 LICENSE = "BSD"
2425 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
2426 DEPENDS += "libxext libsm libxt"
2427 PE = "1"
2428
2429 XORG_PN = "libXpm"
2430
2431 PACKAGES =+ "sxpm cxpm"
2432 FILES_cxpm = "${bindir}/cxpm"
2433 FILES_sxpm = "${bindir}/sxpm"
2434
2435In the previous example, we want to ship the ``sxpm`` and ``cxpm``
2436binaries in separate packages. Since ``bindir`` would be packaged into
2437the main ``PN`` package by default, we prepend the ``PACKAGES`` variable
2438so additional package names are added to the start of list. This results
2439in the extra ``FILES_*`` variables then containing information that
2440define which files and directories go into which packages. Files
2441included by earlier packages are skipped by latter packages. Thus, the
2442main ``PN`` package does not include the above listed files.
2443
2444Packaging Externally Produced Binaries
2445~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2446
2447Sometimes, you need to add pre-compiled binaries to an image. For
2448example, suppose that binaries for proprietary code exist, which are
2449created by a particular division of a company. Your part of the company
2450needs to use those binaries as part of an image that you are building
2451using the OpenEmbedded build system. Since you only have the binaries
2452and not the source code, you cannot use a typical recipe that expects to
2453fetch the source specified in
2454:term:`SRC_URI` and then compile it.
2455
2456One method is to package the binaries and then install them as part of
2457the image. Generally, it is not a good idea to package binaries since,
2458among other things, it can hinder the ability to reproduce builds and
2459could lead to compatibility problems with ABI in the future. However,
2460sometimes you have no choice.
2461
2462The easiest solution is to create a recipe that uses the
2463:ref:`bin_package <ref-classes-bin-package>` class
2464and to be sure that you are using default locations for build artifacts.
2465In most cases, the ``bin_package`` class handles "skipping" the
2466configure and compile steps as well as sets things up to grab packages
2467from the appropriate area. In particular, this class sets ``noexec`` on
2468both the :ref:`ref-tasks-configure`
2469and :ref:`ref-tasks-compile` tasks,
2470sets ``FILES_${PN}`` to "/" so that it picks up all files, and sets up a
2471:ref:`ref-tasks-install` task, which
2472effectively copies all files from ``${S}`` to ``${D}``. The
2473``bin_package`` class works well when the files extracted into ``${S}``
2474are already laid out in the way they should be laid out on the target.
2475For more information on these variables, see the
2476:term:`FILES`,
2477:term:`PN`,
2478:term:`S`, and
2479:term:`D` variables in the Yocto Project
2480Reference Manual's variable glossary.
2481
2482.. note::
2483
2484 - Using :term:`DEPENDS` is a good
2485 idea even for components distributed in binary form, and is often
2486 necessary for shared libraries. For a shared library, listing the
2487 library dependencies in ``DEPENDS`` makes sure that the libraries
2488 are available in the staging sysroot when other recipes link
2489 against the library, which might be necessary for successful
2490 linking.
2491
2492 - Using ``DEPENDS`` also allows runtime dependencies between
2493 packages to be added automatically. See the
2494 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
2495 section in the Yocto Project Overview and Concepts Manual for more
2496 information.
2497
2498If you cannot use the ``bin_package`` class, you need to be sure you are
2499doing the following:
2500
2501- Create a recipe where the
2502 :ref:`ref-tasks-configure` and
2503 :ref:`ref-tasks-compile` tasks do
2504 nothing: It is usually sufficient to just not define these tasks in
2505 the recipe, because the default implementations do nothing unless a
2506 Makefile is found in
2507 ``${``\ :term:`S`\ ``}``.
2508
2509 If ``${S}`` might contain a Makefile, or if you inherit some class
2510 that replaces ``do_configure`` and ``do_compile`` with custom
2511 versions, then you can use the
2512 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
2513 flag to turn the tasks into no-ops, as follows:
2514 ::
2515
2516 do_configure[noexec] = "1"
2517 do_compile[noexec] = "1"
2518
2519 Unlike
2520 :ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
2521 using the flag preserves the dependency chain from the
2522 :ref:`ref-tasks-fetch`,
2523 :ref:`ref-tasks-unpack`, and
2524 :ref:`ref-tasks-patch` tasks to the
2525 :ref:`ref-tasks-install` task.
2526
2527- Make sure your ``do_install`` task installs the binaries
2528 appropriately.
2529
2530- Ensure that you set up :term:`FILES`
2531 (usually
2532 ``FILES_${``\ :term:`PN`\ ``}``) to
2533 point to the files you have installed, which of course depends on
2534 where you have installed them and whether those files are in
2535 different locations than the defaults.
2536
2537.. note::
2538
2539 If image prelinking is enabled (e.g. "image-prelink" is in :term:`USER_CLASSES`
2540 which it is by default), prelink will change the binaries in the generated images
2541 and this often catches people out. Remove that class to ensure binaries are
2542 preserved exactly if that is necessary.
2543
2544Following Recipe Style Guidelines
2545---------------------------------
2546
2547When writing recipes, it is good to conform to existing style
2548guidelines. The :oe_wiki:`OpenEmbedded Styleguide </Styleguide>` wiki page
2549provides rough guidelines for preferred recipe style.
2550
2551It is common for existing recipes to deviate a bit from this style.
2552However, aiming for at least a consistent style is a good idea. Some
2553practices, such as omitting spaces around ``=`` operators in assignments
2554or ordering recipe components in an erratic way, are widely seen as poor
2555style.
2556
2557Recipe Syntax
2558-------------
2559
2560Understanding recipe file syntax is important for writing recipes. The
2561following list overviews the basic items that make up a BitBake recipe
2562file. For more complete BitBake syntax descriptions, see the
2563":doc:`bitbake-user-manual/bitbake-user-manual-metadata`"
2564chapter of the BitBake User Manual.
2565
2566- *Variable Assignments and Manipulations:* Variable assignments allow
2567 a value to be assigned to a variable. The assignment can be static
2568 text or might include the contents of other variables. In addition to
2569 the assignment, appending and prepending operations are also
2570 supported.
2571
2572 The following example shows some of the ways you can use variables in
2573 recipes:
2574 ::
2575
2576 S = "${WORKDIR}/postfix-${PV}"
2577 CFLAGS += "-DNO_ASM"
2578 SRC_URI_append = " file://fixup.patch"
2579
2580- *Functions:* Functions provide a series of actions to be performed.
2581 You usually use functions to override the default implementation of a
2582 task function or to complement a default function (i.e. append or
2583 prepend to an existing function). Standard functions use ``sh`` shell
2584 syntax, although access to OpenEmbedded variables and internal
2585 methods are also available.
2586
2587 The following is an example function from the ``sed`` recipe:
2588 ::
2589
2590 do_install () {
2591 autotools_do_install
2592 install -d ${D}${base_bindir}
2593 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
2594 rmdir ${D}${bindir}/
2595 }
2596
2597 It is
2598 also possible to implement new functions that are called between
2599 existing tasks as long as the new functions are not replacing or
2600 complementing the default functions. You can implement functions in
2601 Python instead of shell. Both of these options are not seen in the
2602 majority of recipes.
2603
2604- *Keywords:* BitBake recipes use only a few keywords. You use keywords
2605 to include common functions (``inherit``), load parts of a recipe
2606 from other files (``include`` and ``require``) and export variables
2607 to the environment (``export``).
2608
2609 The following example shows the use of some of these keywords:
2610 ::
2611
2612 export POSTCONF = "${STAGING_BINDIR}/postconf"
2613 inherit autoconf
2614 require otherfile.inc
2615
2616- *Comments (#):* Any lines that begin with the hash character (``#``)
2617 are treated as comment lines and are ignored:
2618 ::
2619
2620 # This is a comment
2621
2622This next list summarizes the most important and most commonly used
2623parts of the recipe syntax. For more information on these parts of the
2624syntax, you can reference the
2625:doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata` chapter
2626in the BitBake User Manual.
2627
2628- *Line Continuation (\\):* Use the backward slash (``\``) character to
2629 split a statement over multiple lines. Place the slash character at
2630 the end of the line that is to be continued on the next line:
2631 ::
2632
2633 VAR = "A really long \
2634 line"
2635
2636 .. note::
2637
2638 You cannot have any characters including spaces or tabs after the
2639 slash character.
2640
2641- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
2642 access the contents of a variable:
2643 ::
2644
2645 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
2646
2647 .. note::
2648
2649 It is important to understand that the value of a variable
2650 expressed in this form does not get substituted automatically. The
2651 expansion of these expressions happens on-demand later (e.g.
2652 usually when a function that makes reference to the variable
2653 executes). This behavior ensures that the values are most
2654 appropriate for the context in which they are finally used. On the
2655 rare occasion that you do need the variable expression to be
2656 expanded immediately, you can use the
2657 :=
2658 operator instead of
2659 =
2660 when you make the assignment, but this is not generally needed.
2661
2662- *Quote All Assignments ("value"):* Use double quotes around values in
2663 all variable assignments (e.g. ``"value"``). Following is an example:
2664 ::
2665
2666 VAR1 = "${OTHERVAR}"
2667 VAR2 = "The version is ${PV}"
2668
2669- *Conditional Assignment (?=):* Conditional assignment is used to
2670 assign a value to a variable, but only when the variable is currently
2671 unset. Use the question mark followed by the equal sign (``?=``) to
2672 make a "soft" assignment used for conditional assignment. Typically,
2673 "soft" assignments are used in the ``local.conf`` file for variables
2674 that are allowed to come through from the external environment.
2675
2676 Here is an example where ``VAR1`` is set to "New value" if it is
2677 currently empty. However, if ``VAR1`` has already been set, it
2678 remains unchanged:
2679 ::
2680
2681 VAR1 ?= "New value"
2682
2683 In this next example, ``VAR1`` is left with the value "Original value":
2684 ::
2685
2686 VAR1 = "Original value"
2687 VAR1 ?= "New value"
2688
2689- *Appending (+=):* Use the plus character followed by the equals sign
2690 (``+=``) to append values to existing variables.
2691
2692 .. note::
2693
2694 This operator adds a space between the existing content of the
2695 variable and the new content.
2696
2697 Here is an example:
2698 ::
2699
2700 SRC_URI += "file://fix-makefile.patch"
2701
2702- *Prepending (=+):* Use the equals sign followed by the plus character
2703 (``=+``) to prepend values to existing variables.
2704
2705 .. note::
2706
2707 This operator adds a space between the new content and the
2708 existing content of the variable.
2709
2710 Here is an example:
2711 ::
2712
2713 VAR =+ "Starts"
2714
2715- *Appending (_append):* Use the ``_append`` operator to append values
2716 to existing variables. This operator does not add any additional
2717 space. Also, the operator is applied after all the ``+=``, and ``=+``
2718 operators have been applied and after all ``=`` assignments have
2719 occurred.
2720
2721 The following example shows the space being explicitly added to the
2722 start to ensure the appended value is not merged with the existing
2723 value:
2724 ::
2725
2726 SRC_URI_append = " file://fix-makefile.patch"
2727
2728 You can also use
2729 the ``_append`` operator with overrides, which results in the actions
2730 only being performed for the specified target or machine:
2731 ::
2732
2733 SRC_URI_append_sh4 = " file://fix-makefile.patch"
2734
2735- *Prepending (_prepend):* Use the ``_prepend`` operator to prepend
2736 values to existing variables. This operator does not add any
2737 additional space. Also, the operator is applied after all the ``+=``,
2738 and ``=+`` operators have been applied and after all ``=``
2739 assignments have occurred.
2740
2741 The following example shows the space being explicitly added to the
2742 end to ensure the prepended value is not merged with the existing
2743 value:
2744 ::
2745
2746 CFLAGS_prepend = "-I${S}/myincludes "
2747
2748 You can also use the
2749 ``_prepend`` operator with overrides, which results in the actions
2750 only being performed for the specified target or machine:
2751 ::
2752
2753 CFLAGS_prepend_sh4 = "-I${S}/myincludes "
2754
2755- *Overrides:* You can use overrides to set a value conditionally,
2756 typically based on how the recipe is being built. For example, to set
2757 the :term:`KBRANCH` variable's
2758 value to "standard/base" for any target
2759 :term:`MACHINE`, except for
2760 qemuarm where it should be set to "standard/arm-versatile-926ejs",
2761 you would do the following:
2762 ::
2763
2764 KBRANCH = "standard/base"
2765 KBRANCH_qemuarm = "standard/arm-versatile-926ejs"
2766
2767 Overrides are also used to separate
2768 alternate values of a variable in other situations. For example, when
2769 setting variables such as
2770 :term:`FILES` and
2771 :term:`RDEPENDS` that are
2772 specific to individual packages produced by a recipe, you should
2773 always use an override that specifies the name of the package.
2774
2775- *Indentation:* Use spaces for indentation rather than tabs. For
2776 shell functions, both currently work. However, it is a policy
2777 decision of the Yocto Project to use tabs in shell functions. Realize
2778 that some layers have a policy to use spaces for all indentation.
2779
2780- *Using Python for Complex Operations:* For more advanced processing,
2781 it is possible to use Python code during variable assignments (e.g.
2782 search and replacement on a variable).
2783
2784 You indicate Python code using the ``${@python_code}`` syntax for the
2785 variable assignment:
2786 ::
2787
2788 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
2789
2790- *Shell Function Syntax:* Write shell functions as if you were writing
2791 a shell script when you describe a list of actions to take. You
2792 should ensure that your script works with a generic ``sh`` and that
2793 it does not require any ``bash`` or other shell-specific
2794 functionality. The same considerations apply to various system
2795 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
2796 might wish to use. If in doubt, you should check with multiple
2797 implementations - including those from BusyBox.
2798
2799Adding a New Machine
2800====================
2801
2802Adding a new machine to the Yocto Project is a straightforward process.
2803This section describes how to add machines that are similar to those
2804that the Yocto Project already supports.
2805
2806.. note::
2807
2808 Although well within the capabilities of the Yocto Project, adding a
2809 totally new architecture might require changes to ``gcc``/``glibc``
2810 and to the site information, which is beyond the scope of this
2811 manual.
2812
2813For a complete example that shows how to add a new machine, see the
2814":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
2815section in the Yocto Project Board Support Package (BSP) Developer's
2816Guide.
2817
2818Adding the Machine Configuration File
2819-------------------------------------
2820
2821To add a new machine, you need to add a new machine configuration file
2822to the layer's ``conf/machine`` directory. This configuration file
2823provides details about the device you are adding.
2824
2825The OpenEmbedded build system uses the root name of the machine
2826configuration file to reference the new machine. For example, given a
2827machine configuration file named ``crownbay.conf``, the build system
2828recognizes the machine as "crownbay".
2829
2830The most important variables you must set in your machine configuration
2831file or include from a lower-level configuration file are as follows:
2832
2833- ``TARGET_ARCH`` (e.g. "arm")
2834
2835- ``PREFERRED_PROVIDER_virtual/kernel``
2836
2837- ``MACHINE_FEATURES`` (e.g. "apm screen wifi")
2838
2839You might also need these variables:
2840
2841- ``SERIAL_CONSOLES`` (e.g. "115200;ttyS0 115200;ttyS1")
2842
2843- ``KERNEL_IMAGETYPE`` (e.g. "zImage")
2844
2845- ``IMAGE_FSTYPES`` (e.g. "tar.gz jffs2")
2846
2847You can find full details on these variables in the reference section.
2848You can leverage existing machine ``.conf`` files from
2849``meta-yocto-bsp/conf/machine/``.
2850
2851Adding a Kernel for the Machine
2852-------------------------------
2853
2854The OpenEmbedded build system needs to be able to build a kernel for the
2855machine. You need to either create a new kernel recipe for this machine,
2856or extend an existing kernel recipe. You can find several kernel recipe
2857examples in the Source Directory at ``meta/recipes-kernel/linux`` that
2858you can use as references.
2859
2860If you are creating a new kernel recipe, normal recipe-writing rules
2861apply for setting up a ``SRC_URI``. Thus, you need to specify any
2862necessary patches and set ``S`` to point at the source code. You need to
2863create a ``do_configure`` task that configures the unpacked kernel with
2864a ``defconfig`` file. You can do this by using a ``make defconfig``
2865command or, more commonly, by copying in a suitable ``defconfig`` file
2866and then running ``make oldconfig``. By making use of ``inherit kernel``
2867and potentially some of the ``linux-*.inc`` files, most other
2868functionality is centralized and the defaults of the class normally work
2869well.
2870
2871If you are extending an existing kernel recipe, it is usually a matter
2872of adding a suitable ``defconfig`` file. The file needs to be added into
2873a location similar to ``defconfig`` files used for other machines in a
2874given kernel recipe. A possible way to do this is by listing the file in
2875the ``SRC_URI`` and adding the machine to the expression in
2876``COMPATIBLE_MACHINE``:
2877::
2878
2879 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
2880
2881For more information on ``defconfig`` files, see the
2882":ref:`kernel-dev/common:changing the configuration`"
2883section in the Yocto Project Linux Kernel Development Manual.
2884
2885Adding a Formfactor Configuration File
2886--------------------------------------
2887
2888A formfactor configuration file provides information about the target
2889hardware for which the image is being built and information that the
2890build system cannot obtain from other sources such as the kernel. Some
2891examples of information contained in a formfactor configuration file
2892include framebuffer orientation, whether or not the system has a
2893keyboard, the positioning of the keyboard in relation to the screen, and
2894the screen resolution.
2895
2896The build system uses reasonable defaults in most cases. However, if
2897customization is necessary, you need to create a ``machconfig`` file in
2898the ``meta/recipes-bsp/formfactor/files`` directory. This directory
2899contains directories for specific machines such as ``qemuarm`` and
2900``qemux86``. For information about the settings available and the
2901defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
2902found in the same area.
2903
2904Following is an example for "qemuarm" machine:
2905::
2906
2907 HAVE_TOUCHSCREEN=1
2908 HAVE_KEYBOARD=1
2909 DISPLAY_CAN_ROTATE=0
2910 DISPLAY_ORIENTATION=0
2911 #DISPLAY_WIDTH_PIXELS=640
2912 #DISPLAY_HEIGHT_PIXELS=480
2913 #DISPLAY_BPP=16
2914 DISPLAY_DPI=150
2915 DISPLAY_SUBPIXEL_ORDER=vrgb
2916
2917Upgrading Recipes
2918=================
2919
2920Over time, upstream developers publish new versions for software built
2921by layer recipes. It is recommended to keep recipes up-to-date with
2922upstream version releases.
2923
2924While several methods exist that allow you upgrade a recipe, you might
2925consider checking on the upgrade status of a recipe first. You can do so
2926using the ``devtool check-upgrade-status`` command. See the
2927":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
2928section in the Yocto Project Reference Manual for more information.
2929
2930The remainder of this section describes three ways you can upgrade a
2931recipe. You can use the Automated Upgrade Helper (AUH) to set up
2932automatic version upgrades. Alternatively, you can use
2933``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
2934you can manually upgrade a recipe by editing the recipe itself.
2935
2936Using the Auto Upgrade Helper (AUH)
2937-----------------------------------
2938
2939The AUH utility works in conjunction with the OpenEmbedded build system
2940in order to automatically generate upgrades for recipes based on new
2941versions being published upstream. Use AUH when you want to create a
2942service that performs the upgrades automatically and optionally sends
2943you an email with the results.
2944
2945AUH allows you to update several recipes with a single use. You can also
2946optionally perform build and integration tests using images with the
2947results saved to your hard drive and emails of results optionally sent
2948to recipe maintainers. Finally, AUH creates Git commits with appropriate
2949commit messages in the layer's tree for the changes made to recipes.
2950
2951.. note::
2952
2953 Conditions do exist when you should not use AUH to upgrade recipes
2954 and you should instead use either ``devtool upgrade`` or upgrade your
2955 recipes manually:
2956
2957 - When AUH cannot complete the upgrade sequence. This situation
2958 usually results because custom patches carried by the recipe
2959 cannot be automatically rebased to the new version. In this case,
2960 ``devtool upgrade`` allows you to manually resolve conflicts.
2961
2962 - When for any reason you want fuller control over the upgrade
2963 process. For example, when you want special arrangements for
2964 testing.
2965
2966The following steps describe how to set up the AUH utility:
2967
29681. *Be Sure the Development Host is Set Up:* You need to be sure that
2969 your development host is set up to use the Yocto Project. For
2970 information on how to set up your host, see the
2971 ":ref:`dev-manual/start:Preparing the Build Host`" section.
2972
29732. *Make Sure Git is Configured:* The AUH utility requires Git to be
2974 configured because AUH uses Git to save upgrades. Thus, you must have
2975 Git user and email configured. The following command shows your
2976 configurations:
2977 ::
2978
2979 $ git config --list
2980
2981 If you do not have the user and
2982 email configured, you can use the following commands to do so:
2983 ::
2984
2985 $ git config --global user.name some_name
2986 $ git config --global user.email username@domain.com
2987
29883. *Clone the AUH Repository:* To use AUH, you must clone the repository
2989 onto your development host. The following command uses Git to create
2990 a local copy of the repository on your system:
2991 ::
2992
2993 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
2994 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
2995 remote: Compressing objects: 100% (300/300), done.
2996 remote: Total 768 (delta 499), reused 703 (delta 434)
2997 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
2998 Resolving deltas: 100% (499/499), done.
2999 Checking connectivity... done.
3000
3001 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
3002 :term:`Poky` repositories.
3003
30044. *Create a Dedicated Build Directory:* Run the
3005 :ref:`structure-core-script`
3006 script to create a fresh build directory that you use exclusively for
3007 running the AUH utility:
3008 ::
3009
3010 $ cd ~/poky
3011 $ source oe-init-build-env your_AUH_build_directory
3012
3013 Re-using an existing build directory and its configurations is not
3014 recommended as existing settings could cause AUH to fail or behave
3015 undesirably.
3016
30175. *Make Configurations in Your Local Configuration File:* Several
3018 settings need to exist in the ``local.conf`` file in the build
3019 directory you just created for AUH. Make these following
3020 configurations:
3021
3022 - If you want to enable :ref:`Build
3023 History <dev-manual/common-tasks:maintaining build output quality>`,
3024 which is optional, you need the following lines in the
3025 ``conf/local.conf`` file:
3026 ::
3027
3028 INHERIT =+ "buildhistory"
3029 BUILDHISTORY_COMMIT = "1"
3030
3031 With this configuration and a successful
3032 upgrade, a build history "diff" file appears in the
3033 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
3034 your build directory.
3035
3036 - If you want to enable testing through the
3037 :ref:`testimage <ref-classes-testimage*>`
3038 class, which is optional, you need to have the following set in
3039 your ``conf/local.conf`` file:
3040 ::
3041
3042 INHERIT += "testimage"
3043
3044 .. note::
3045
3046 If your distro does not enable by default ptest, which Poky
3047 does, you need the following in your ``local.conf`` file:
3048 ::
3049
3050 DISTRO_FEATURES_append = " ptest"
3051
3052
30536. *Optionally Start a vncserver:* If you are running in a server
3054 without an X11 session, you need to start a vncserver:
3055 ::
3056
3057 $ vncserver :1
3058 $ export DISPLAY=:1
3059
30607. *Create and Edit an AUH Configuration File:* You need to have the
3061 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
3062 build directory. You can find a sample configuration file in the
3063 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
3064
3065 Read through the sample file and make configurations as needed. For
3066 example, if you enabled build history in your ``local.conf`` as
3067 described earlier, you must enable it in ``upgrade-helper.conf``.
3068
3069 Also, if you are using the default ``maintainers.inc`` file supplied
3070 with Poky and located in ``meta-yocto`` and you do not set a
3071 "maintainers_whitelist" or "global_maintainer_override" in the
3072 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
3073 the AUH command-line, the utility automatically sends out emails to
3074 all the default maintainers. Please avoid this.
3075
3076This next set of examples describes how to use the AUH:
3077
3078- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
3079 following form:
3080 ::
3081
3082 $ upgrade-helper.py recipe_name
3083
3084 For example, this command upgrades the ``xmodmap`` recipe:
3085 ::
3086
3087 $ upgrade-helper.py xmodmap
3088
3089- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
3090 specific recipe to a particular version, use the following form:
3091 ::
3092
3093 $ upgrade-helper.py recipe_name -t version
3094
3095 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3:
3096 ::
3097
3098 $ upgrade-helper.py xmodmap -t 1.2.3
3099
3100- *Upgrading all Recipes to the Latest Versions and Suppressing Email
3101 Notifications:* To upgrade all recipes to their most recent versions
3102 and suppress the email notifications, use the following command:
3103 ::
3104
3105 $ upgrade-helper.py all
3106
3107- *Upgrading all Recipes to the Latest Versions and Send Email
3108 Notifications:* To upgrade all recipes to their most recent versions
3109 and send email messages to maintainers for each attempted recipe as
3110 well as a status email, use the following command:
3111 ::
3112
3113 $ upgrade-helper.py -e all
3114
3115Once you have run the AUH utility, you can find the results in the AUH
3116build directory:
3117::
3118
3119 ${BUILDDIR}/upgrade-helper/timestamp
3120
3121The AUH utility
3122also creates recipe update commits from successful upgrade attempts in
3123the layer tree.
3124
3125You can easily set up to run the AUH utility on a regular basis by using
3126a cron job. See the
3127:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
3128file distributed with the utility for an example.
3129
3130Using ``devtool upgrade``
3131-------------------------
3132
3133As mentioned earlier, an alternative method for upgrading recipes to
3134newer versions is to use
3135:doc:`devtool upgrade </ref-manual/devtool-reference>`.
3136You can read about ``devtool upgrade`` in general in the
3137":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
3138section in the Yocto Project Application Development and the Extensible
3139Software Development Kit (eSDK) Manual.
3140
3141To see all the command-line options available with ``devtool upgrade``,
3142use the following help command:
3143::
3144
3145 $ devtool upgrade -h
3146
3147If you want to find out what version a recipe is currently at upstream
3148without any attempt to upgrade your local version of the recipe, you can
3149use the following command:
3150::
3151
3152 $ devtool latest-version recipe_name
3153
3154As mentioned in the previous section describing AUH, ``devtool upgrade``
3155works in a less-automated manner than AUH. Specifically,
3156``devtool upgrade`` only works on a single recipe that you name on the
3157command line, cannot perform build and integration testing using images,
3158and does not automatically generate commits for changes in the source
3159tree. Despite all these "limitations", ``devtool upgrade`` updates the
3160recipe file to the new upstream version and attempts to rebase custom
3161patches contained by the recipe as needed.
3162
3163.. note::
3164
3165 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
3166 of a "wrapper" application for ``devtool upgrade``.
3167
3168A typical scenario involves having used Git to clone an upstream
3169repository that you use during build operations. Because you have built the
3170recipe in the past, the layer is likely added to your
3171configuration already. If for some reason, the layer is not added, you
3172could add it easily using the
3173":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
3174script. For example, suppose you use the ``nano.bb`` recipe from the
3175``meta-oe`` layer in the ``meta-openembedded`` repository. For this
3176example, assume that the layer has been cloned into following area:
3177::
3178
3179 /home/scottrif/meta-openembedded
3180
3181The following command from your
3182:term:`Build Directory` adds the layer to
3183your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``):
3184::
3185
3186 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
3187 NOTE: Starting bitbake server...
3188 Parsing recipes: 100% |##########################################| Time: 0:00:55
3189 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3190 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
3191 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
3192 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
3193 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
3194
3195For this example, assume that the ``nano.bb`` recipe that
3196is upstream has a 2.9.3 version number. However, the version in the
3197local repository is 2.7.4. The following command from your build
3198directory automatically upgrades the recipe for you:
3199
3200.. note::
3201
3202 Using the ``-V`` option is not necessary. Omitting the version number causes
3203 ``devtool upgrade`` to upgrade the recipe to the most recent version.
3204
3205::
3206
3207 $ devtool upgrade nano -V 2.9.3
3208 NOTE: Starting bitbake server...
3209 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
3210 Parsing recipes: 100% |##########################################| Time: 0:00:46
3211 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
3212 NOTE: Extracting current version source...
3213 NOTE: Resolving any missing task queue dependencies
3214 .
3215 .
3216 .
3217 NOTE: Executing SetScene Tasks
3218 NOTE: Executing RunQueue Tasks
3219 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
3220 Adding changed files: 100% |#####################################| Time: 0:00:00
3221 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
3222 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
3223
3224Continuing with this example, you can use ``devtool build`` to build the
3225newly upgraded recipe:
3226::
3227
3228 $ devtool build nano
3229 NOTE: Starting bitbake server...
3230 Loading cache: 100% |################################################################################################| Time: 0:00:01
3231 Loaded 2040 entries from dependency cache.
3232 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
3233 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3234 NOTE: Resolving any missing task queue dependencies
3235 .
3236 .
3237 .
3238 NOTE: Executing SetScene Tasks
3239 NOTE: Executing RunQueue Tasks
3240 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
3241 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
3242
3243Within the ``devtool upgrade`` workflow, opportunity
3244exists to deploy and test your rebuilt software. For this example,
3245however, running ``devtool finish`` cleans up the workspace once the
3246source in your workspace is clean. This usually means using Git to stage
3247and submit commits for the changes generated by the upgrade process.
3248
3249Once the tree is clean, you can clean things up in this example with the
3250following command from the ``${BUILDDIR}/workspace/sources/nano``
3251directory:
3252::
3253
3254 $ devtool finish nano meta-oe
3255 NOTE: Starting bitbake server...
3256 Loading cache: 100% |################################################################################################| Time: 0:00:00
3257 Loaded 2040 entries from dependency cache.
3258 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
3259 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
3260 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
3261 NOTE: Updating recipe nano_2.9.3.bb
3262 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
3263 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
3264 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
3265
3266
3267Using the ``devtool finish`` command cleans up the workspace and creates a patch
3268file based on your commits. The tool puts all patch files back into the
3269source directory in a sub-directory named ``nano`` in this case.
3270
3271Manually Upgrading a Recipe
3272---------------------------
3273
3274If for some reason you choose not to upgrade recipes using
3275:ref:`dev-manual/common-tasks:Using the Auto Upgrade Helper (AUH)` or
3276by :ref:`dev-manual/common-tasks:Using \`\`devtool upgrade\`\``,
3277you can manually edit the recipe files to upgrade the versions.
3278
3279.. note::
3280
3281 Manually updating multiple recipes scales poorly and involves many
3282 steps. The recommendation to upgrade recipe versions is through AUH
3283 or ``devtool upgrade``, both of which automate some steps and provide
3284 guidance for others needed for the manual process.
3285
3286To manually upgrade recipe versions, follow these general steps:
3287
32881. *Change the Version:* Rename the recipe such that the version (i.e.
3289 the :term:`PV` part of the recipe name)
3290 changes appropriately. If the version is not part of the recipe name,
3291 change the value as it is set for ``PV`` within the recipe itself.
3292
32932. *Update* ``SRCREV`` *if Needed*: If the source code your recipe builds
3294 is fetched from Git or some other version control system, update
3295 :term:`SRCREV` to point to the
3296 commit hash that matches the new version.
3297
32983. *Build the Software:* Try to build the recipe using BitBake. Typical
3299 build failures include the following:
3300
3301 - License statements were updated for the new version. For this
3302 case, you need to review any changes to the license and update the
3303 values of :term:`LICENSE` and
3304 :term:`LIC_FILES_CHKSUM`
3305 as needed.
3306
3307 .. note::
3308
3309 License changes are often inconsequential. For example, the
3310 license text's copyright year might have changed.
3311
3312 - Custom patches carried by the older version of the recipe might
3313 fail to apply to the new version. For these cases, you need to
3314 review the failures. Patches might not be necessary for the new
3315 version of the software if the upgraded version has fixed those
3316 issues. If a patch is necessary and failing, you need to rebase it
3317 into the new version.
3318
33194. *Optionally Attempt to Build for Several Architectures:* Once you
3320 successfully build the new software for a given architecture, you
3321 could test the build for other architectures by changing the
3322 :term:`MACHINE` variable and
3323 rebuilding the software. This optional step is especially important
3324 if the recipe is to be released publicly.
3325
33265. *Check the Upstream Change Log or Release Notes:* Checking both these
3327 reveals if new features exist that could break
3328 backwards-compatibility. If so, you need to take steps to mitigate or
3329 eliminate that situation.
3330
33316. *Optionally Create a Bootable Image and Test:* If you want, you can
3332 test the new software by booting it onto actual hardware.
3333
33347. *Create a Commit with the Change in the Layer Repository:* After all
3335 builds work and any testing is successful, you can create commits for
3336 any changes in the layer holding your upgraded recipe.
3337
3338Finding Temporary Source Code
3339=============================
3340
3341You might find it helpful during development to modify the temporary
3342source code used by recipes to build packages. For example, suppose you
3343are developing a patch and you need to experiment a bit to figure out
3344your solution. After you have initially built the package, you can
3345iteratively tweak the source code, which is located in the
3346:term:`Build Directory`, and then you can
3347force a re-compile and quickly test your altered code. Once you settle
3348on a solution, you can then preserve your changes in the form of
3349patches.
3350
3351During a build, the unpacked temporary source code used by recipes to
3352build packages is available in the Build Directory as defined by the
3353:term:`S` variable. Below is the default
3354value for the ``S`` variable as defined in the
3355``meta/conf/bitbake.conf`` configuration file in the
3356:term:`Source Directory`:
3357::
3358
3359 S = "${WORKDIR}/${BP}"
3360
3361You should be aware that many recipes override the
3362``S`` variable. For example, recipes that fetch their source from Git
3363usually set ``S`` to ``${WORKDIR}/git``.
3364
3365.. note::
3366
3367 The :term:`BP` represents the base recipe name, which consists of the name
3368 and version:
3369 ::
3370
3371 BP = "${BPN}-${PV}"
3372
3373
3374The path to the work directory for the recipe
3375(:term:`WORKDIR`) is defined as
3376follows:
3377::
3378
3379 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
3380
3381The actual directory depends on several things:
3382
3383- :term:`TMPDIR`: The top-level build
3384 output directory.
3385
3386- :term:`MULTIMACH_TARGET_SYS`:
3387 The target system identifier.
3388
3389- :term:`PN`: The recipe name.
3390
3391- :term:`EXTENDPE`: The epoch - (if
3392 :term:`PE` is not specified, which is
3393 usually the case for most recipes, then ``EXTENDPE`` is blank).
3394
3395- :term:`PV`: The recipe version.
3396
3397- :term:`PR`: The recipe revision.
3398
3399As an example, assume a Source Directory top-level folder named
3400``poky``, a default Build Directory at ``poky/build``, and a
3401``qemux86-poky-linux`` machine target system. Furthermore, suppose your
3402recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
3403build system uses to build the package would be as follows:
3404::
3405
3406 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
3407
3408Using Quilt in Your Workflow
3409============================
3410
3411`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
3412that allows you to capture source code changes without having a clean
3413source tree. This section outlines the typical workflow you can use to
3414modify source code, test changes, and then preserve the changes in the
3415form of a patch all using Quilt.
3416
3417.. note::
3418
3419 With regard to preserving changes to source files, if you clean a
3420 recipe or have ``rm_work`` enabled, the
3421 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
3422 as described in the Yocto Project Application Development and the
3423 Extensible Software Development Kit (eSDK) manual is a safer
3424 development flow than the flow that uses Quilt.
3425
3426Follow these general steps:
3427
34281. *Find the Source Code:* Temporary source code used by the
3429 OpenEmbedded build system is kept in the
3430 :term:`Build Directory`. See the
3431 "`Finding Temporary Source
3432 Code <#finding-the-temporary-source-code>`__" section to learn how to
3433 locate the directory that has the temporary source code for a
3434 particular package.
3435
34362. *Change Your Working Directory:* You need to be in the directory that
3437 has the temporary source code. That directory is defined by the
3438 :term:`S` variable.
3439
34403. *Create a New Patch:* Before modifying source code, you need to
3441 create a new patch. To create a new patch file, use ``quilt new`` as
3442 below:
3443 ::
3444
3445 $ quilt new my_changes.patch
3446
34474. *Notify Quilt and Add Files:* After creating the patch, you need to
3448 notify Quilt about the files you plan to edit. You notify Quilt by
3449 adding the files to the patch you just created:
3450 ::
3451
3452 $ quilt add file1.c file2.c file3.c
3453
34545. *Edit the Files:* Make your changes in the source code to the files
3455 you added to the patch.
3456
34576. *Test Your Changes:* Once you have modified the source code, the
3458 easiest way to test your changes is by calling the ``do_compile``
3459 task as shown in the following example:
3460 ::
3461
3462 $ bitbake -c compile -f package
3463
3464 The ``-f`` or ``--force`` option forces the specified task to
3465 execute. If you find problems with your code, you can just keep
3466 editing and re-testing iteratively until things work as expected.
3467
3468 .. note::
3469
3470 All the modifications you make to the temporary source code disappear
3471 once you run the ``do_clean`` or ``do_cleanall`` tasks using BitBake
3472 (i.e. ``bitbake -c clean package`` and ``bitbake -c cleanall package``).
3473 Modifications will also disappear if you use the ``rm_work`` feature as
3474 described in the
3475 ":ref:`dev-manual/common-tasks:conserving disk space during builds`"
3476 section.
3477
34787. *Generate the Patch:* Once your changes work as expected, you need to
3479 use Quilt to generate the final patch that contains all your
3480 modifications.
3481 ::
3482
3483 $ quilt refresh
3484
3485 At this point, the
3486 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
3487 ``file2.c``, and ``file3.c`` files.
3488
3489 You can find the resulting patch file in the ``patches/``
3490 subdirectory of the source (``S``) directory.
3491
34928. *Copy the Patch File:* For simplicity, copy the patch file into a
3493 directory named ``files``, which you can create in the same directory
3494 that holds the recipe (``.bb``) file or the append (``.bbappend``)
3495 file. Placing the patch here guarantees that the OpenEmbedded build
3496 system will find the patch. Next, add the patch into the ``SRC_URI``
3497 of the recipe. Here is an example:
3498 ::
3499
3500 SRC_URI += "file://my_changes.patch"
3501
3502Using a Development Shell
3503=========================
3504
3505When debugging certain commands or even when just editing packages,
3506``devshell`` can be a useful tool. When you invoke ``devshell``, all
3507tasks up to and including
3508:ref:`ref-tasks-patch` are run for the
3509specified target. Then, a new terminal is opened and you are placed in
3510``${``\ :term:`S`\ ``}``, the source
3511directory. In the new terminal, all the OpenEmbedded build-related
3512environment variables are still defined so you can use commands such as
3513``configure`` and ``make``. The commands execute just as if the
3514OpenEmbedded build system were executing them. Consequently, working
3515this way can be helpful when debugging a build or preparing software to
3516be used with the OpenEmbedded build system.
3517
3518Following is an example that uses ``devshell`` on a target named
3519``matchbox-desktop``:
3520::
3521
3522 $ bitbake matchbox-desktop -c devshell
3523
3524This command spawns a terminal with a shell prompt within the
3525OpenEmbedded build environment. The
3526:term:`OE_TERMINAL` variable
3527controls what type of shell is opened.
3528
3529For spawned terminals, the following occurs:
3530
3531- The ``PATH`` variable includes the cross-toolchain.
3532
3533- The ``pkgconfig`` variables find the correct ``.pc`` files.
3534
3535- The ``configure`` command finds the Yocto Project site files as well
3536 as any other necessary files.
3537
3538Within this environment, you can run configure or compile commands as if
3539they were being run by the OpenEmbedded build system itself. As noted
3540earlier, the working directory also automatically changes to the Source
3541Directory (:term:`S`).
3542
3543To manually run a specific task using ``devshell``, run the
3544corresponding ``run.*`` script in the
3545``${``\ :term:`WORKDIR`\ ``}/temp``
3546directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
3547not exist, which would be the case if the task was skipped by way of the
3548sstate cache, you can create the task by first running it outside of the
3549``devshell``:
3550::
3551
3552 $ bitbake -c task
3553
3554.. note::
3555
3556 - Execution of a task's ``run.*`` script and BitBake's execution of
3557 a task are identical. In other words, running the script re-runs
3558 the task just as it would be run using the ``bitbake -c`` command.
3559
3560 - Any ``run.*`` file that does not have a ``.pid`` extension is a
3561 symbolic link (symlink) to the most recent version of that file.
3562
3563Remember, that the ``devshell`` is a mechanism that allows you to get
3564into the BitBake task execution environment. And as such, all commands
3565must be called just as BitBake would call them. That means you need to
3566provide the appropriate options for cross-compilation and so forth as
3567applicable.
3568
3569When you are finished using ``devshell``, exit the shell or close the
3570terminal window.
3571
3572.. note::
3573
3574 - It is worth remembering that when using ``devshell`` you need to
3575 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
3576 instead of just using ``gcc``. The same applies to other
3577 applications such as ``binutils``, ``libtool`` and so forth.
3578 BitBake sets up environment variables such as ``CC`` to assist
3579 applications, such as ``make`` to find the correct tools.
3580
3581 - It is also worth noting that ``devshell`` still works over X11
3582 forwarding and similar situations.
3583
3584Using a Development Python Shell
3585================================
3586
3587Similar to working within a development shell as described in the
3588previous section, you can also spawn and work within an interactive
3589Python development shell. When debugging certain commands or even when
3590just editing packages, ``devpyshell`` can be a useful tool. When you
3591invoke ``devpyshell``, all tasks up to and including
3592:ref:`ref-tasks-patch` are run for the
3593specified target. Then a new terminal is opened. Additionally, key
3594Python objects and code are available in the same way they are to
3595BitBake tasks, in particular, the data store 'd'. So, commands such as
3596the following are useful when exploring the data store and running
3597functions:
3598::
3599
3600 pydevshell> d.getVar("STAGING_DIR")
3601 '/media/build1/poky/build/tmp/sysroots'
3602 pydevshell> d.getVar("STAGING_DIR")
3603 '${TMPDIR}/sysroots'
3604 pydevshell> d.setVar("FOO", "bar")
3605 pydevshell> d.getVar("FOO")
3606 'bar'
3607 pydevshell> d.delVar("FOO")
3608 pydevshell> d.getVar("FOO")
3609 pydevshell> bb.build.exec_func("do_unpack", d)
3610 pydevshell>
3611
3612The commands execute just as if the OpenEmbedded build
3613system were executing them. Consequently, working this way can be
3614helpful when debugging a build or preparing software to be used with the
3615OpenEmbedded build system.
3616
3617Following is an example that uses ``devpyshell`` on a target named
3618``matchbox-desktop``:
3619::
3620
3621 $ bitbake matchbox-desktop -c devpyshell
3622
3623This command spawns a terminal and places you in an interactive Python
3624interpreter within the OpenEmbedded build environment. The
3625:term:`OE_TERMINAL` variable
3626controls what type of shell is opened.
3627
3628When you are finished using ``devpyshell``, you can exit the shell
3629either by using Ctrl+d or closing the terminal window.
3630
3631Building
3632========
3633
3634This section describes various build procedures. For example, the steps
3635needed for a simple build, a target that uses multiple configurations,
3636building an image for more than one machine, and so forth.
3637
3638Building a Simple Image
3639-----------------------
3640
3641In the development environment, you need to build an image whenever you
3642change hardware support, add or change system libraries, or add or
3643change services that have dependencies. Several methods exist that allow
3644you to build an image within the Yocto Project. This section presents
3645the basic steps you need to build a simple image using BitBake from a
3646build host running Linux.
3647
3648.. note::
3649
3650 - For information on how to build an image using
3651 :term:`Toaster`, see the
3652 :doc:`/toaster-manual/index`.
3653
3654 - For information on how to use ``devtool`` to build images, see the
3655 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
3656 section in the Yocto Project Application Development and the
3657 Extensible Software Development Kit (eSDK) manual.
3658
3659 - For a quick example on how to build an image using the
3660 OpenEmbedded build system, see the
3661 :doc:`/brief-yoctoprojectqs/index` document.
3662
3663The build process creates an entire Linux distribution from source and
3664places it in your :term:`Build Directory` under
3665``tmp/deploy/images``. For detailed information on the build process
3666using BitBake, see the ":ref:`overview-manual/concepts:images`" section in the
3667Yocto Project Overview and Concepts Manual.
3668
3669The following figure and list overviews the build process:
3670
3671.. image:: figures/bitbake-build-flow.png
3672 :align: center
3673
36741. *Set up Your Host Development System to Support Development Using the
3675 Yocto Project*: See the ":doc:`start`" section for options on how to get a
3676 build host ready to use the Yocto Project.
3677
36782. *Initialize the Build Environment:* Initialize the build environment
3679 by sourcing the build environment script (i.e.
3680 :ref:`structure-core-script`):
3681 ::
3682
3683 $ source oe-init-build-env [build_dir]
3684
3685 When you use the initialization script, the OpenEmbedded build system
3686 uses ``build`` as the default :term:`Build Directory` in your current work
3687 directory. You can use a `build_dir` argument with the script to
3688 specify a different build directory.
3689
3690 .. note::
3691
3692 A common practice is to use a different Build Directory for
3693 different targets. For example, ``~/build/x86`` for a ``qemux86``
3694 target, and ``~/build/arm`` for a ``qemuarm`` target.
3695
36963. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
3697 ``conf/local.conf`` configuration file, which is found in the Build
3698 Directory, is set up how you want it. This file defines many aspects
3699 of the build environment including the target machine architecture
3700 through the ``MACHINE`` variable, the packaging format used during
3701 the build
3702 (:term:`PACKAGE_CLASSES`),
3703 and a centralized tarball download directory through the
3704 :term:`DL_DIR` variable.
3705
37064. *Build the Image:* Build the image using the ``bitbake`` command:
3707 ::
3708
3709 $ bitbake target
3710
3711 .. note::
3712
3713 For information on BitBake, see the :doc:`bitbake:index`.
3714
3715 The target is the name of the recipe you want to build. Common
3716 targets are the images in ``meta/recipes-core/images``,
3717 ``meta/recipes-sato/images``, and so forth all found in the
3718 :term:`Source Directory`. Or, the target
3719 can be the name of a recipe for a specific piece of software such as
3720 BusyBox. For more details about the images the OpenEmbedded build
3721 system supports, see the
3722 ":ref:`ref-manual/images:Images`" chapter in the Yocto
3723 Project Reference Manual.
3724
3725 As an example, the following command builds the
3726 ``core-image-minimal`` image:
3727 ::
3728
3729 $ bitbake core-image-minimal
3730
3731 Once an
3732 image has been built, it often needs to be installed. The images and
3733 kernels built by the OpenEmbedded build system are placed in the
3734 Build Directory in ``tmp/deploy/images``. For information on how to
3735 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
3736 :doc:`/sdk-manual/index` manual. For
3737 information about how to install these images, see the documentation
3738 for your particular board or machine.
3739
3740Building Images for Multiple Targets Using Multiple Configurations
3741------------------------------------------------------------------
3742
3743You can use a single ``bitbake`` command to build multiple images or
3744packages for different targets where each image or package requires a
3745different configuration (multiple configuration builds). The builds, in
3746this scenario, are sometimes referred to as "multiconfigs", and this
3747section uses that term throughout.
3748
3749This section describes how to set up for multiple configuration builds
3750and how to account for cross-build dependencies between the
3751multiconfigs.
3752
3753Setting Up and Running a Multiple Configuration Build
3754~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3755
3756To accomplish a multiple configuration build, you must define each
3757target's configuration separately using a parallel configuration file in
3758the :term:`Build Directory`, and you
3759must follow a required file hierarchy. Additionally, you must enable the
3760multiple configuration builds in your ``local.conf`` file.
3761
3762Follow these steps to set up and execute multiple configuration builds:
3763
3764- *Create Separate Configuration Files*: You need to create a single
3765 configuration file for each build target (each multiconfig).
3766 Minimally, each configuration file must define the machine and the
3767 temporary directory BitBake uses for the build. Suggested practice
3768 dictates that you do not overlap the temporary directories used
3769 during the builds. However, it is possible that you can share the
3770 temporary directory
3771 (:term:`TMPDIR`). For example,
3772 consider a scenario with two different multiconfigs for the same
3773 :term:`MACHINE`: "qemux86" built
3774 for two distributions such as "poky" and "poky-lsb". In this case,
3775 you might want to use the same ``TMPDIR``.
3776
3777 Here is an example showing the minimal statements needed in a
3778 configuration file for a "qemux86" target whose temporary build
3779 directory is ``tmpmultix86``:
3780 ::
3781
3782 MACHINE = "qemux86"
3783 TMPDIR = "${TOPDIR}/tmpmultix86"
3784
3785 The location for these multiconfig configuration files is specific.
3786 They must reside in the current build directory in a sub-directory of
3787 ``conf`` named ``multiconfig``. Following is an example that defines
3788 two configuration files for the "x86" and "arm" multiconfigs:
3789
3790 .. image:: figures/multiconfig_files.png
3791 :align: center
3792
3793 The reason for this required file hierarchy is because the ``BBPATH``
3794 variable is not constructed until the layers are parsed.
3795 Consequently, using the configuration file as a pre-configuration
3796 file is not possible unless it is located in the current working
3797 directory.
3798
3799- *Add the BitBake Multi-configuration Variable to the Local
3800 Configuration File*: Use the
3801 :term:`BBMULTICONFIG`
3802 variable in your ``conf/local.conf`` configuration file to specify
3803 each multiconfig. Continuing with the example from the previous
3804 figure, the ``BBMULTICONFIG`` variable needs to enable two
3805 multiconfigs: "x86" and "arm" by specifying each configuration file:
3806 ::
3807
3808 BBMULTICONFIG = "x86 arm"
3809
3810 .. note::
3811
3812 A "default" configuration already exists by definition. This
3813 configuration is named: "" (i.e. empty string) and is defined by
3814 the variables coming from your ``local.conf``
3815 file. Consequently, the previous example actually adds two
3816 additional configurations to your build: "arm" and "x86" along
3817 with "".
3818
3819- *Launch BitBake*: Use the following BitBake command form to launch
3820 the multiple configuration build:
3821 ::
3822
3823 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
3824
3825 For the example in this section, the following command applies:
3826 ::
3827
3828 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
3829
3830 The previous BitBake command builds a ``core-image-minimal`` image
3831 that is configured through the ``x86.conf`` configuration file, a
3832 ``core-image-sato`` image that is configured through the ``arm.conf``
3833 configuration file and a ``core-image-base`` that is configured
3834 through your ``local.conf`` configuration file.
3835
3836.. note::
3837
3838 Support for multiple configuration builds in the Yocto Project &DISTRO;
3839 (&DISTRO_NAME;) Release does not include Shared State (sstate)
3840 optimizations. Consequently, if a build uses the same object twice
3841 in, for example, two different ``TMPDIR``
3842 directories, the build either loads from an existing sstate cache for
3843 that build at the start or builds the object fresh.
3844
3845Enabling Multiple Configuration Build Dependencies
3846~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3847
3848Sometimes dependencies can exist between targets (multiconfigs) in a
3849multiple configuration build. For example, suppose that in order to
3850build a ``core-image-sato`` image for an "x86" multiconfig, the root
3851filesystem of an "arm" multiconfig must exist. This dependency is
3852essentially that the
3853:ref:`ref-tasks-image` task in the
3854``core-image-sato`` recipe depends on the completion of the
3855:ref:`ref-tasks-rootfs` task of the
3856``core-image-minimal`` recipe.
3857
3858To enable dependencies in a multiple configuration build, you must
3859declare the dependencies in the recipe using the following statement
3860form:
3861::
3862
3863 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
3864
3865To better show how to use this statement, consider the example scenario
3866from the first paragraph of this section. The following statement needs
3867to be added to the recipe that builds the ``core-image-sato`` image:
3868::
3869
3870 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
3871
3872In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
3873task on which the ``do_image`` task in the recipe depends is the
3874``do_rootfs`` task from the ``core-image-minimal`` recipe associated
3875with the "arm" multiconfig.
3876
3877Once you set up this dependency, you can build the "x86" multiconfig
3878using a BitBake command as follows:
3879::
3880
3881 $ bitbake mc:x86:core-image-sato
3882
3883This command executes all the tasks needed to create the
3884``core-image-sato`` image for the "x86" multiconfig. Because of the
3885dependency, BitBake also executes through the ``do_rootfs`` task for the
3886"arm" multiconfig build.
3887
3888Having a recipe depend on the root filesystem of another build might not
3889seem that useful. Consider this change to the statement in the
3890``core-image-sato`` recipe:
3891::
3892
3893 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
3894
3895In this case, BitBake must
3896create the ``core-image-minimal`` image for the "arm" build since the
3897"x86" build depends on it.
3898
3899Because "x86" and "arm" are enabled for multiple configuration builds
3900and have separate configuration files, BitBake places the artifacts for
3901each build in the respective temporary build directories (i.e.
3902:term:`TMPDIR`).
3903
3904Building an Initial RAM Filesystem (initramfs) Image
3905----------------------------------------------------
3906
3907An initial RAM filesystem (initramfs) image provides a temporary root
3908filesystem used for early system initialization (e.g. loading of modules
3909needed to locate and mount the "real" root filesystem).
3910
3911.. note::
3912
3913 The initramfs image is the successor of initial RAM disk (initrd). It
3914 is a "copy in and out" (cpio) archive of the initial filesystem that
3915 gets loaded into memory during the Linux startup process. Because
3916 Linux uses the contents of the archive during initialization, the
3917 initramfs image needs to contain all of the device drivers and tools
3918 needed to mount the final root filesystem.
3919
3920Follow these steps to create an initramfs image:
3921
39221. *Create the initramfs Image Recipe:* You can reference the
3923 ``core-image-minimal-initramfs.bb`` recipe found in the
3924 ``meta/recipes-core`` directory of the :term:`Source Directory`
3925 as an example
3926 from which to work.
3927
39282. *Decide if You Need to Bundle the initramfs Image Into the Kernel
3929 Image:* If you want the initramfs image that is built to be bundled
3930 in with the kernel image, set the
3931 :term:`INITRAMFS_IMAGE_BUNDLE`
3932 variable to "1" in your ``local.conf`` configuration file and set the
3933 :term:`INITRAMFS_IMAGE`
3934 variable in the recipe that builds the kernel image.
3935
3936 .. note::
3937
3938 It is recommended that you do bundle the initramfs image with the
3939 kernel image to avoid circular dependencies between the kernel
3940 recipe and the initramfs recipe should the initramfs image include
3941 kernel modules.
3942
3943 Setting the ``INITRAMFS_IMAGE_BUNDLE`` flag causes the initramfs
3944 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
3945 initramfs image is then passed to the kernel's ``Makefile`` using the
3946 :term:`CONFIG_INITRAMFS_SOURCE`
3947 variable, allowing the initramfs image to be built into the kernel
3948 normally.
3949
3950 .. note::
3951
3952 If you choose to not bundle the initramfs image with the kernel
3953 image, you are essentially using an
3954 `Initial RAM Disk (initrd) <https://en.wikipedia.org/wiki/Initrd>`__.
3955 Creating an initrd is handled primarily through the :term:`INITRD_IMAGE`,
3956 ``INITRD_LIVE``, and ``INITRD_IMAGE_LIVE`` variables. For more
3957 information, see the :ref:`ref-classes-image-live` file.
3958
39593. *Optionally Add Items to the initramfs Image Through the initramfs
3960 Image Recipe:* If you add items to the initramfs image by way of its
3961 recipe, you should use
3962 :term:`PACKAGE_INSTALL`
3963 rather than
3964 :term:`IMAGE_INSTALL`.
3965 ``PACKAGE_INSTALL`` gives more direct control of what is added to the
3966 image as compared to the defaults you might not necessarily want that
3967 are set by the :ref:`image <ref-classes-image>`
3968 or :ref:`core-image <ref-classes-core-image>`
3969 classes.
3970
39714. *Build the Kernel Image and the initramfs Image:* Build your kernel
3972 image using BitBake. Because the initramfs image recipe is a
3973 dependency of the kernel image, the initramfs image is built as well
3974 and bundled with the kernel image if you used the
3975 :term:`INITRAMFS_IMAGE_BUNDLE`
3976 variable described earlier.
3977
3978Building a Tiny System
3979----------------------
3980
3981Very small distributions have some significant advantages such as
3982requiring less on-die or in-package memory (cheaper), better performance
3983through efficient cache usage, lower power requirements due to less
3984memory, faster boot times, and reduced development overhead. Some
3985real-world examples where a very small distribution gives you distinct
3986advantages are digital cameras, medical devices, and small headless
3987systems.
3988
3989This section presents information that shows you how you can trim your
3990distribution to even smaller sizes than the ``poky-tiny`` distribution,
3991which is around 5 Mbytes, that can be built out-of-the-box using the
3992Yocto Project.
3993
3994Tiny System Overview
3995~~~~~~~~~~~~~~~~~~~~
3996
3997The following list presents the overall steps you need to consider and
3998perform to create distributions with smaller root filesystems, achieve
3999faster boot times, maintain your critical functionality, and avoid
4000initial RAM disks:
4001
4002- `Determine your goals and guiding
4003 principles. <#goals-and-guiding-principles>`__
4004
4005- `Understand what contributes to your image
4006 size. <#understand-what-gives-your-image-size>`__
4007
4008- `Reduce the size of the root
4009 filesystem. <#trim-the-root-filesystem>`__
4010
4011- `Reduce the size of the kernel. <#trim-the-kernel>`__
4012
4013- `Eliminate packaging
4014 requirements. <#remove-package-management-requirements>`__
4015
4016- `Look for other ways to minimize
4017 size. <#look-for-other-ways-to-minimize-size>`__
4018
4019- `Iterate on the process. <#iterate-on-the-process>`__
4020
4021Goals and Guiding Principles
4022~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4023
4024Before you can reach your destination, you need to know where you are
4025going. Here is an example list that you can use as a guide when creating
4026very small distributions:
4027
4028- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
4029 less and a root filesystem that is 3 Mbytes or less).
4030
4031- Find the areas that are currently taking 90% of the space and
4032 concentrate on reducing those areas.
4033
4034- Do not create any difficult "hacks" to achieve your goals.
4035
4036- Leverage the device-specific options.
4037
4038- Work in a separate layer so that you keep changes isolated. For
4039 information on how to create layers, see the "`Understanding and
4040 Creating Layers <#understanding-and-creating-layers>`__" section.
4041
4042Understand What Contributes to Your Image Size
4043~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4044
4045It is easiest to have something to start with when creating your own
4046distribution. You can use the Yocto Project out-of-the-box to create the
4047``poky-tiny`` distribution. Ultimately, you will want to make changes in
4048your own distribution that are likely modeled after ``poky-tiny``.
4049
4050.. note::
4051
4052 To use ``poky-tiny`` in your build, set the ``DISTRO`` variable in your
4053 ``local.conf`` file to "poky-tiny" as described in the
4054 ":ref:`dev-manual/common-tasks:creating your own distribution`"
4055 section.
4056
4057Understanding some memory concepts will help you reduce the system size.
4058Memory consists of static, dynamic, and temporary memory. Static memory
4059is the TEXT (code), DATA (initialized data in the code), and BSS
4060(uninitialized data) sections. Dynamic memory represents memory that is
4061allocated at runtime: stacks, hash tables, and so forth. Temporary
4062memory is recovered after the boot process. This memory consists of
4063memory used for decompressing the kernel and for the ``__init__``
4064functions.
4065
4066To help you see where you currently are with kernel and root filesystem
4067sizes, you can use two tools found in the :term:`Source Directory`
4068in the
4069``scripts/tiny/`` directory:
4070
4071- ``ksize.py``: Reports component sizes for the kernel build objects.
4072
4073- ``dirsize.py``: Reports component sizes for the root filesystem.
4074
4075This next tool and command help you organize configuration fragments and
4076view file dependencies in a human-readable form:
4077
4078- ``merge_config.sh``: Helps you manage configuration files and
4079 fragments within the kernel. With this tool, you can merge individual
4080 configuration fragments together. The tool allows you to make
4081 overrides and warns you of any missing configuration options. The
4082 tool is ideal for allowing you to iterate on configurations, create
4083 minimal configurations, and create configuration files for different
4084 machines without having to duplicate your process.
4085
4086 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
4087 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
4088 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
4089 directory.
4090
4091 For more information on configuration fragments, see the
4092 ":ref:`kernel-dev/common:creating configuration fragments`"
4093 section in the Yocto Project Linux Kernel Development Manual.
4094
4095- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
4096 with these options brings up a Dependency Explorer from which you can
4097 view file dependencies. Understanding these dependencies allows you
4098 to make informed decisions when cutting out various pieces of the
4099 kernel and root filesystem.
4100
4101Trim the Root Filesystem
4102~~~~~~~~~~~~~~~~~~~~~~~~
4103
4104The root filesystem is made up of packages for booting, libraries, and
4105applications. To change things, you can configure how the packaging
4106happens, which changes the way you build them. You can also modify the
4107filesystem itself or select a different filesystem.
4108
4109First, find out what is hogging your root filesystem by running the
4110``dirsize.py`` script from your root directory:
4111::
4112
4113 $ cd root-directory-of-image
4114 $ dirsize.py 100000 > dirsize-100k.log
4115 $ cat dirsize-100k.log
4116
4117You can apply a filter to the script to ignore files
4118under a certain size. The previous example filters out any files below
4119100 Kbytes. The sizes reported by the tool are uncompressed, and thus
4120will be smaller by a relatively constant factor in a compressed root
4121filesystem. When you examine your log file, you can focus on areas of
4122the root filesystem that take up large amounts of memory.
4123
4124You need to be sure that what you eliminate does not cripple the
4125functionality you need. One way to see how packages relate to each other
4126is by using the Dependency Explorer UI with the BitBake command:
4127::
4128
4129 $ cd image-directory
4130 $ bitbake -u taskexp -g image
4131
4132Use the interface to
4133select potential packages you wish to eliminate and see their dependency
4134relationships.
4135
4136When deciding how to reduce the size, get rid of packages that result in
4137minimal impact on the feature set. For example, you might not need a VGA
4138display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
4139instead of ``udev``.
4140
4141Use your ``local.conf`` file to make changes. For example, to eliminate
4142``udev`` and ``glib``, set the following in the local configuration
4143file:
4144::
4145
4146 VIRTUAL-RUNTIME_dev_manager = ""
4147
4148Finally, you should consider exactly the type of root filesystem you
4149need to meet your needs while also reducing its size. For example,
4150consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
4151``initramfs`` using ``initramfs``. Be aware that ``ext3`` requires a 1
4152Mbyte journal. If you are okay with running read-only, you do not need
4153this journal.
4154
4155.. note::
4156
4157 After each round of elimination, you need to rebuild your system and
4158 then use the tools to see the effects of your reductions.
4159
4160Trim the Kernel
4161~~~~~~~~~~~~~~~
4162
4163The kernel is built by including policies for hardware-independent
4164aspects. What subsystems do you enable? For what architecture are you
4165building? Which drivers do you build by default?
4166
4167.. note::
4168
4169 You can modify the kernel source if you want to help with boot time.
4170
4171Run the ``ksize.py`` script from the top-level Linux build directory to
4172get an idea of what is making up the kernel:
4173::
4174
4175 $ cd top-level-linux-build-directory
4176 $ ksize.py > ksize.log
4177 $ cat ksize.log
4178
4179When you examine the log, you will see how much space is taken up with
4180the built-in ``.o`` files for drivers, networking, core kernel files,
4181filesystem, sound, and so forth. The sizes reported by the tool are
4182uncompressed, and thus will be smaller by a relatively constant factor
4183in a compressed kernel image. Look to reduce the areas that are large
4184and taking up around the "90% rule."
4185
4186To examine, or drill down, into any particular area, use the ``-d``
4187option with the script:
4188::
4189
4190 $ ksize.py -d > ksize.log
4191
4192Using this option
4193breaks out the individual file information for each area of the kernel
4194(e.g. drivers, networking, and so forth).
4195
4196Use your log file to see what you can eliminate from the kernel based on
4197features you can let go. For example, if you are not going to need
4198sound, you do not need any drivers that support sound.
4199
4200After figuring out what to eliminate, you need to reconfigure the kernel
4201to reflect those changes during the next build. You could run
4202``menuconfig`` and make all your changes at once. However, that makes it
4203difficult to see the effects of your individual eliminations and also
4204makes it difficult to replicate the changes for perhaps another target
4205device. A better method is to start with no configurations using
4206``allnoconfig``, create configuration fragments for individual changes,
4207and then manage the fragments into a single configuration file using
4208``merge_config.sh``. The tool makes it easy for you to iterate using the
4209configuration change and build cycle.
4210
4211Each time you make configuration changes, you need to rebuild the kernel
4212and check to see what impact your changes had on the overall size.
4213
4214Remove Package Management Requirements
4215~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4216
4217Packaging requirements add size to the image. One way to reduce the size
4218of the image is to remove all the packaging requirements from the image.
4219This reduction includes both removing the package manager and its unique
4220dependencies as well as removing the package management data itself.
4221
4222To eliminate all the packaging requirements for an image, be sure that
4223"package-management" is not part of your
4224:term:`IMAGE_FEATURES`
4225statement for the image. When you remove this feature, you are removing
4226the package manager as well as its dependencies from the root
4227filesystem.
4228
4229Look for Other Ways to Minimize Size
4230~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4231
4232Depending on your particular circumstances, other areas that you can
4233trim likely exist. The key to finding these areas is through tools and
4234methods described here combined with experimentation and iteration. Here
4235are a couple of areas to experiment with:
4236
4237- ``glibc``: In general, follow this process:
4238
4239 1. Remove ``glibc`` features from
4240 :term:`DISTRO_FEATURES`
4241 that you think you do not need.
4242
4243 2. Build your distribution.
4244
4245 3. If the build fails due to missing symbols in a package, determine
4246 if you can reconfigure the package to not need those features. For
4247 example, change the configuration to not support wide character
4248 support as is done for ``ncurses``. Or, if support for those
4249 characters is needed, determine what ``glibc`` features provide
4250 the support and restore the configuration.
4251
4252 4. Rebuild and repeat the process.
4253
4254- ``busybox``: For BusyBox, use a process similar as described for
4255 ``glibc``. A difference is you will need to boot the resulting system
4256 to see if you are able to do everything you expect from the running
4257 system. You need to be sure to integrate configuration fragments into
4258 Busybox because BusyBox handles its own core features and then allows
4259 you to add configuration fragments on top.
4260
4261Iterate on the Process
4262~~~~~~~~~~~~~~~~~~~~~~
4263
4264If you have not reached your goals on system size, you need to iterate
4265on the process. The process is the same. Use the tools and see just what
4266is taking up 90% of the root filesystem and the kernel. Decide what you
4267can eliminate without limiting your device beyond what you need.
4268
4269Depending on your system, a good place to look might be Busybox, which
4270provides a stripped down version of Unix tools in a single, executable
4271file. You might be able to drop virtual terminal services or perhaps
4272ipv6.
4273
4274Building Images for More than One Machine
4275-----------------------------------------
4276
4277A common scenario developers face is creating images for several
4278different machines that use the same software environment. In this
4279situation, it is tempting to set the tunings and optimization flags for
4280each build specifically for the targeted hardware (i.e. "maxing out" the
4281tunings). Doing so can considerably add to build times and package feed
4282maintenance collectively for the machines. For example, selecting tunes
4283that are extremely specific to a CPU core used in a system might enable
4284some micro optimizations in GCC for that particular system but would
4285otherwise not gain you much of a performance difference across the other
4286systems as compared to using a more general tuning across all the builds
4287(e.g. setting :term:`DEFAULTTUNE`
4288specifically for each machine's build). Rather than "max out" each
4289build's tunings, you can take steps that cause the OpenEmbedded build
4290system to reuse software across the various machines where it makes
4291sense.
4292
4293If build speed and package feed maintenance are considerations, you
4294should consider the points in this section that can help you optimize
4295your tunings to best consider build times and package feed maintenance.
4296
4297- *Share the Build Directory:* If at all possible, share the
4298 :term:`TMPDIR` across builds. The
4299 Yocto Project supports switching between different
4300 :term:`MACHINE` values in the same
4301 ``TMPDIR``. This practice is well supported and regularly used by
4302 developers when building for multiple machines. When you use the same
4303 ``TMPDIR`` for multiple machine builds, the OpenEmbedded build system
4304 can reuse the existing native and often cross-recipes for multiple
4305 machines. Thus, build time decreases.
4306
4307 .. note::
4308
4309 If :term:`DISTRO` settings change or fundamental configuration settings
4310 such as the filesystem layout, you need to work with a clean ``TMPDIR``.
4311 Sharing ``TMPDIR`` under these circumstances might work but since it is
4312 not guaranteed, you should use a clean ``TMPDIR``.
4313
4314- *Enable the Appropriate Package Architecture:* By default, the
4315 OpenEmbedded build system enables three levels of package
4316 architectures: "all", "tune" or "package", and "machine". Any given
4317 recipe usually selects one of these package architectures (types) for
4318 its output. Depending for what a given recipe creates packages,
4319 making sure you enable the appropriate package architecture can
4320 directly impact the build time.
4321
4322 A recipe that just generates scripts can enable "all" architecture
4323 because there are no binaries to build. To specifically enable "all"
4324 architecture, be sure your recipe inherits the
4325 :ref:`allarch <ref-classes-allarch>` class.
4326 This class is useful for "all" architectures because it configures
4327 many variables so packages can be used across multiple architectures.
4328
4329 If your recipe needs to generate packages that are machine-specific
4330 or when one of the build or runtime dependencies is already
4331 machine-architecture dependent, which makes your recipe also
4332 machine-architecture dependent, make sure your recipe enables the
4333 "machine" package architecture through the
4334 :term:`MACHINE_ARCH`
4335 variable:
4336 ::
4337
4338 PACKAGE_ARCH = "${MACHINE_ARCH}"
4339
4340 When you do not
4341 specifically enable a package architecture through the
4342 :term:`PACKAGE_ARCH`, The
4343 OpenEmbedded build system defaults to the
4344 :term:`TUNE_PKGARCH` setting:
4345 ::
4346
4347 PACKAGE_ARCH = "${TUNE_PKGARCH}"
4348
4349- *Choose a Generic Tuning File if Possible:* Some tunes are more
4350 generic and can run on multiple targets (e.g. an ``armv5`` set of
4351 packages could run on ``armv6`` and ``armv7`` processors in most
4352 cases). Similarly, ``i486`` binaries could work on ``i586`` and
4353 higher processors. You should realize, however, that advances on
4354 newer processor versions would not be used.
4355
4356 If you select the same tune for several different machines, the
4357 OpenEmbedded build system reuses software previously built, thus
4358 speeding up the overall build time. Realize that even though a new
4359 sysroot for each machine is generated, the software is not recompiled
4360 and only one package feed exists.
4361
4362- *Manage Granular Level Packaging:* Sometimes cases exist where
4363 injecting another level of package architecture beyond the three
4364 higher levels noted earlier can be useful. For example, consider how
4365 NXP (formerly Freescale) allows for the easy reuse of binary packages
4366 in their layer
4367 :yocto_git:`meta-freescale </meta-freescale/>`.
4368 In this example, the
4369 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
4370 class shares GPU packages for i.MX53 boards because all boards share
4371 the AMD GPU. The i.MX6-based boards can do the same because all
4372 boards share the Vivante GPU. This class inspects the BitBake
4373 datastore to identify if the package provides or depends on one of
4374 the sub-architecture values. If so, the class sets the
4375 :term:`PACKAGE_ARCH` value
4376 based on the ``MACHINE_SUBARCH`` value. If the package does not
4377 provide or depend on one of the sub-architecture values but it
4378 matches a value in the machine-specific filter, it sets
4379 :term:`MACHINE_ARCH`. This
4380 behavior reduces the number of packages built and saves build time by
4381 reusing binaries.
4382
4383- *Use Tools to Debug Issues:* Sometimes you can run into situations
4384 where software is being rebuilt when you think it should not be. For
4385 example, the OpenEmbedded build system might not be using shared
4386 state between machines when you think it should be. These types of
4387 situations are usually due to references to machine-specific
4388 variables such as :term:`MACHINE`,
4389 :term:`SERIAL_CONSOLES`,
4390 :term:`XSERVER`,
4391 :term:`MACHINE_FEATURES`,
4392 and so forth in code that is supposed to only be tune-specific or
4393 when the recipe depends
4394 (:term:`DEPENDS`,
4395 :term:`RDEPENDS`,
4396 :term:`RRECOMMENDS`,
4397 :term:`RSUGGESTS`, and so forth)
4398 on some other recipe that already has
4399 :term:`PACKAGE_ARCH` defined
4400 as "${MACHINE_ARCH}".
4401
4402 .. note::
4403
4404 Patches to fix any issues identified are most welcome as these
4405 issues occasionally do occur.
4406
4407 For such cases, you can use some tools to help you sort out the
4408 situation:
4409
4410 - ``state-diff-machines.sh``*:* You can find this tool in the
4411 ``scripts`` directory of the Source Repositories. See the comments
4412 in the script for information on how to use the tool.
4413
4414 - *BitBake's "-S printdiff" Option:* Using this option causes
4415 BitBake to try to establish the closest signature match it can
4416 (e.g. in the shared state cache) and then run ``bitbake-diffsigs``
4417 over the matches to determine the stamps and delta where these two
4418 stamp trees diverge.
4419
4420Building Software from an External Source
4421-----------------------------------------
4422
4423By default, the OpenEmbedded build system uses the
4424:term:`Build Directory` when building source
4425code. The build process involves fetching the source files, unpacking
4426them, and then patching them if necessary before the build takes place.
4427
4428Situations exist where you might want to build software from source
4429files that are external to and thus outside of the OpenEmbedded build
4430system. For example, suppose you have a project that includes a new BSP
4431with a heavily customized kernel. And, you want to minimize exposing the
4432build system to the development team so that they can focus on their
4433project and maintain everyone's workflow as much as possible. In this
4434case, you want a kernel source directory on the development machine
4435where the development occurs. You want the recipe's
4436:term:`SRC_URI` variable to point to
4437the external directory and use it as is, not copy it.
4438
4439To build from software that comes from an external source, all you need
4440to do is inherit the
4441:ref:`externalsrc <ref-classes-externalsrc>` class
4442and then set the
4443:term:`EXTERNALSRC` variable to
4444point to your external source code. Here are the statements to put in
4445your ``local.conf`` file:
4446::
4447
4448 INHERIT += "externalsrc"
4449 EXTERNALSRC_pn-myrecipe = "path-to-your-source-tree"
4450
4451This next example shows how to accomplish the same thing by setting
4452``EXTERNALSRC`` in the recipe itself or in the recipe's append file:
4453::
4454
4455 EXTERNALSRC = "path"
4456 EXTERNALSRC_BUILD = "path"
4457
4458.. note::
4459
4460 In order for these settings to take effect, you must globally or
4461 locally inherit the :ref:`externalsrc <ref-classes-externalsrc>`
4462 class.
4463
4464By default, ``externalsrc.bbclass`` builds the source code in a
4465directory separate from the external source directory as specified by
4466:term:`EXTERNALSRC`. If you need
4467to have the source built in the same directory in which it resides, or
4468some other nominated directory, you can set
4469:term:`EXTERNALSRC_BUILD`
4470to point to that directory:
4471::
4472
4473 EXTERNALSRC_BUILD_pn-myrecipe = "path-to-your-source-tree"
4474
4475Replicating a Build Offline
4476---------------------------
4477
4478It can be useful to take a "snapshot" of upstream sources used in a
4479build and then use that "snapshot" later to replicate the build offline.
4480To do so, you need to first prepare and populate your downloads
4481directory your "snapshot" of files. Once your downloads directory is
4482ready, you can use it at any time and from any machine to replicate your
4483build.
4484
4485Follow these steps to populate your Downloads directory:
4486
44871. *Create a Clean Downloads Directory:* Start with an empty downloads
4488 directory (:term:`DL_DIR`). You
4489 start with an empty downloads directory by either removing the files
4490 in the existing directory or by setting ``DL_DIR`` to point to either
4491 an empty location or one that does not yet exist.
4492
44932. *Generate Tarballs of the Source Git Repositories:* Edit your
4494 ``local.conf`` configuration file as follows:
4495 ::
4496
4497 DL_DIR = "/home/your-download-dir/"
4498 BB_GENERATE_MIRROR_TARBALLS = "1"
4499
4500 During
4501 the fetch process in the next step, BitBake gathers the source files
4502 and creates tarballs in the directory pointed to by ``DL_DIR``. See
4503 the
4504 :term:`BB_GENERATE_MIRROR_TARBALLS`
4505 variable for more information.
4506
45073. *Populate Your Downloads Directory Without Building:* Use BitBake to
4508 fetch your sources but inhibit the build:
4509 ::
4510
4511 $ bitbake target --runonly=fetch
4512
4513 The downloads directory (i.e. ``${DL_DIR}``) now has
4514 a "snapshot" of the source files in the form of tarballs, which can
4515 be used for the build.
4516
45174. *Optionally Remove Any Git or other SCM Subdirectories From the
4518 Downloads Directory:* If you want, you can clean up your downloads
4519 directory by removing any Git or other Source Control Management
4520 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
4521 already contain these subdirectories.
4522
4523Once your downloads directory has everything it needs regarding source
4524files, you can create your "own-mirror" and build your target.
4525Understand that you can use the files to build the target offline from
4526any machine and at any time.
4527
4528Follow these steps to build your target using the files in the downloads
4529directory:
4530
45311. *Using Local Files Only:* Inside your ``local.conf`` file, add the
4532 :term:`SOURCE_MIRROR_URL`
4533 variable, inherit the
4534 :ref:`own-mirrors <ref-classes-own-mirrors>`
4535 class, and use the
4536 :term:`bitbake:BB_NO_NETWORK`
4537 variable to your ``local.conf``.
4538 ::
4539
4540 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
4541 INHERIT += "own-mirrors"
4542 BB_NO_NETWORK = "1"
4543
4544 The ``SOURCE_MIRROR_URL`` and ``own-mirror``
4545 class set up the system to use the downloads directory as your "own
4546 mirror". Using the ``BB_NO_NETWORK`` variable makes sure that
4547 BitBake's fetching process in step 3 stays local, which means files
4548 from your "own-mirror" are used.
4549
45502. *Start With a Clean Build:* You can start with a clean build by
4551 removing the
4552 ``${``\ :term:`TMPDIR`\ ``}``
4553 directory or using a new :term:`Build Directory`.
4554
45553. *Build Your Target:* Use BitBake to build your target:
4556 ::
4557
4558 $ bitbake target
4559
4560 The build completes using the known local "snapshot" of source
4561 files from your mirror. The resulting tarballs for your "snapshot" of
4562 source files are in the downloads directory.
4563
4564 .. note::
4565
4566 The offline build does not work if recipes attempt to find the
4567 latest version of software by setting
4568 :term:`SRCREV` to
4569 ``${``\ :term:`AUTOREV`\ ``}``:
4570 ::
4571
4572 SRCREV = "${AUTOREV}"
4573
4574 When a recipe sets ``SRCREV`` to
4575 ``${AUTOREV}``, the build system accesses the network in an
4576 attempt to determine the latest version of software from the SCM.
4577 Typically, recipes that use ``AUTOREV`` are custom or modified
4578 recipes. Recipes that reside in public repositories usually do not
4579 use ``AUTOREV``.
4580
4581 If you do have recipes that use ``AUTOREV``, you can take steps to
4582 still use the recipes in an offline build. Do the following:
4583
4584 1. Use a configuration generated by enabling `build
4585 history <#maintaining-build-output-quality>`__.
4586
4587 2. Use the ``buildhistory-collect-srcrevs`` command to collect the
4588 stored ``SRCREV`` values from the build's history. For more
4589 information on collecting these values, see the "`Build History
4590 Package Information <#build-history-package-information>`__"
4591 section.
4592
4593 3. Once you have the correct source revisions, you can modify
4594 those recipes to to set ``SRCREV`` to specific versions of the
4595 software.
4596
4597Speeding Up a Build
4598===================
4599
4600Build time can be an issue. By default, the build system uses simple
4601controls to try and maximize build efficiency. In general, the default
4602settings for all the following variables result in the most efficient
4603build times when dealing with single socket systems (i.e. a single CPU).
4604If you have multiple CPUs, you might try increasing the default values
4605to gain more speed. See the descriptions in the glossary for each
4606variable for more information:
4607
4608- :term:`BB_NUMBER_THREADS`:
4609 The maximum number of threads BitBake simultaneously executes.
4610
4611- :term:`bitbake:BB_NUMBER_PARSE_THREADS`:
4612 The number of threads BitBake uses during parsing.
4613
4614- :term:`PARALLEL_MAKE`: Extra
4615 options passed to the ``make`` command during the
4616 :ref:`ref-tasks-compile` task in
4617 order to specify parallel compilation on the local build host.
4618
4619- :term:`PARALLEL_MAKEINST`:
4620 Extra options passed to the ``make`` command during the
4621 :ref:`ref-tasks-install` task in
4622 order to specify parallel installation on the local build host.
4623
4624As mentioned, these variables all scale to the number of processor cores
4625available on the build system. For single socket systems, this
4626auto-scaling ensures that the build system fundamentally takes advantage
4627of potential parallel operations during the build based on the build
4628machine's capabilities.
4629
4630Following are additional factors that can affect build speed:
4631
4632- File system type: The file system type that the build is being
4633 performed on can also influence performance. Using ``ext4`` is
4634 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
4635 improved features such as extents.
4636
4637- Disabling the updating of access time using ``noatime``: The
4638 ``noatime`` mount option prevents the build system from updating file
4639 and directory access times.
4640
4641- Setting a longer commit: Using the "commit=" mount option increases
4642 the interval in seconds between disk cache writes. Changing this
4643 interval from the five second default to something longer increases
4644 the risk of data loss but decreases the need to write to the disk,
4645 thus increasing the build performance.
4646
4647- Choosing the packaging backend: Of the available packaging backends,
4648 IPK is the fastest. Additionally, selecting a singular packaging
4649 backend also helps.
4650
4651- Using ``tmpfs`` for :term:`TMPDIR`
4652 as a temporary file system: While this can help speed up the build,
4653 the benefits are limited due to the compiler using ``-pipe``. The
4654 build system goes to some lengths to avoid ``sync()`` calls into the
4655 file system on the principle that if there was a significant failure,
4656 the :term:`Build Directory`
4657 contents could easily be rebuilt.
4658
4659- Inheriting the
4660 :ref:`rm_work <ref-classes-rm-work>` class:
4661 Inheriting this class has shown to speed up builds due to
4662 significantly lower amounts of data stored in the data cache as well
4663 as on disk. Inheriting this class also makes cleanup of
4664 :term:`TMPDIR` faster, at the
4665 expense of being easily able to dive into the source code. File
4666 system maintainers have recommended that the fastest way to clean up
4667 large numbers of files is to reformat partitions rather than delete
4668 files due to the linear nature of partitions. This, of course,
4669 assumes you structure the disk partitions and file systems in a way
4670 that this is practical.
4671
4672Aside from the previous list, you should keep some trade offs in mind
4673that can help you speed up the build:
4674
4675- Remove items from
4676 :term:`DISTRO_FEATURES`
4677 that you might not need.
4678
4679- Exclude debug symbols and other debug information: If you do not need
4680 these symbols and other debug information, disabling the ``*-dbg``
4681 package generation can speed up the build. You can disable this
4682 generation by setting the
4683 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
4684 variable to "1".
4685
4686- Disable static library generation for recipes derived from
4687 ``autoconf`` or ``libtool``: Following is an example showing how to
4688 disable static libraries and still provide an override to handle
4689 exceptions:
4690 ::
4691
4692 STATICLIBCONF = "--disable-static"
4693 STATICLIBCONF_sqlite3-native = ""
4694 EXTRA_OECONF += "${STATICLIBCONF}"
4695
4696 .. note::
4697
4698 - Some recipes need static libraries in order to work correctly
4699 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
4700 as in the previous example, account for these kinds of
4701 exceptions.
4702
4703 - Some packages have packaging code that assumes the presence of
4704 the static libraries. If so, you might need to exclude them as
4705 well.
4706
4707Working With Libraries
4708======================
4709
4710Libraries are an integral part of your system. This section describes
4711some common practices you might find helpful when working with libraries
4712to build your system:
4713
4714- `How to include static library
4715 files <#including-static-library-files>`__
4716
4717- `How to use the Multilib feature to combine multiple versions of
4718 library files into a single
4719 image <#combining-multiple-versions-library-files-into-one-image>`__
4720
4721- `How to install multiple versions of the same library in parallel on
4722 the same
4723 system <#installing-multiple-versions-of-the-same-library>`__
4724
4725Including Static Library Files
4726------------------------------
4727
4728If you are building a library and the library offers static linking, you
4729can control which static library files (``*.a`` files) get included in
4730the built library.
4731
4732The :term:`PACKAGES` and
4733:term:`FILES_* <FILES>` variables in the
4734``meta/conf/bitbake.conf`` configuration file define how files installed
4735by the ``do_install`` task are packaged. By default, the ``PACKAGES``
4736variable includes ``${PN}-staticdev``, which represents all static
4737library files.
4738
4739.. note::
4740
4741 Some previously released versions of the Yocto Project defined the
4742 static library files through ``${PN}-dev``.
4743
4744Following is part of the BitBake configuration file, where you can see
4745how the static library files are defined:
4746::
4747
4748 PACKAGE_BEFORE_PN ?= ""
4749 PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
4750 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
4751 FILES = ""
4752
4753 FILES_${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
4754 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
4755 ${base_bindir}/* ${base_sbindir}/* \
4756 ${base_libdir}/*${SOLIBS} \
4757 ${base_prefix}/lib/udev/rules.d ${prefix}/lib/udev/rules.d \
4758 ${datadir}/${BPN} ${libdir}/${BPN}/* \
4759 ${datadir}/pixmaps ${datadir}/applications \
4760 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
4761 ${libdir}/bonobo/servers"
4762
4763 FILES_${PN}-bin = "${bindir}/* ${sbindir}/*"
4764
4765 FILES_${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
4766 ${datadir}/gnome/help"
4767 SECTION_${PN}-doc = "doc"
4768
4769 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
4770 FILES_${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
4771 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
4772 ${datadir}/aclocal ${base_libdir}/*.o \
4773 ${libdir}/${BPN}/*.la ${base_libdir}/*.la"
4774 SECTION_${PN}-dev = "devel"
4775 ALLOW_EMPTY_${PN}-dev = "1"
4776 RDEPENDS_${PN}-dev = "${PN} (= ${EXTENDPKGV})"
4777
4778 FILES_${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
4779 SECTION_${PN}-staticdev = "devel"
4780 RDEPENDS_${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
4781
4782Combining Multiple Versions of Library Files into One Image
4783-----------------------------------------------------------
4784
4785The build system offers the ability to build libraries with different
4786target optimizations or architecture formats and combine these together
4787into one system image. You can link different binaries in the image
4788against the different libraries as needed for specific use cases. This
4789feature is called "Multilib".
4790
4791An example would be where you have most of a system compiled in 32-bit
4792mode using 32-bit libraries, but you have something large, like a
4793database engine, that needs to be a 64-bit application and uses 64-bit
4794libraries. Multilib allows you to get the best of both 32-bit and 64-bit
4795libraries.
4796
4797While the Multilib feature is most commonly used for 32 and 64-bit
4798differences, the approach the build system uses facilitates different
4799target optimizations. You could compile some binaries to use one set of
4800libraries and other binaries to use a different set of libraries. The
4801libraries could differ in architecture, compiler options, or other
4802optimizations.
4803
4804Several examples exist in the ``meta-skeleton`` layer found in the
4805:term:`Source Directory`:
4806
4807- ``conf/multilib-example.conf`` configuration file
4808
4809- ``conf/multilib-example2.conf`` configuration file
4810
4811- ``recipes-multilib/images/core-image-multilib-example.bb`` recipe
4812
4813Preparing to Use Multilib
4814~~~~~~~~~~~~~~~~~~~~~~~~~
4815
4816User-specific requirements drive the Multilib feature. Consequently,
4817there is no one "out-of-the-box" configuration that likely exists to
4818meet your needs.
4819
4820In order to enable Multilib, you first need to ensure your recipe is
4821extended to support multiple libraries. Many standard recipes are
4822already extended and support multiple libraries. You can check in the
4823``meta/conf/multilib.conf`` configuration file in the
4824:term:`Source Directory` to see how this is
4825done using the
4826:term:`BBCLASSEXTEND` variable.
4827Eventually, all recipes will be covered and this list will not be
4828needed.
4829
4830For the most part, the Multilib class extension works automatically to
4831extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
4832``MLPREFIX`` is the particular multilib (e.g. "lib32-" or "lib64-").
4833Standard variables such as
4834:term:`DEPENDS`,
4835:term:`RDEPENDS`,
4836:term:`RPROVIDES`,
4837:term:`RRECOMMENDS`,
4838:term:`PACKAGES`, and
4839:term:`PACKAGES_DYNAMIC` are
4840automatically extended by the system. If you are extending any manual
4841code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
4842those names are extended correctly. This automatic extension code
4843resides in ``multilib.bbclass``.
4844
4845Using Multilib
4846~~~~~~~~~~~~~~
4847
4848After you have set up the recipes, you need to define the actual
4849combination of multiple libraries you want to build. You accomplish this
4850through your ``local.conf`` configuration file in the
4851:term:`Build Directory`. An example
4852configuration would be as follows:
4853::
4854
4855 MACHINE = "qemux86-64"
4856 require conf/multilib.conf
4857 MULTILIBS = "multilib:lib32"
4858 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
4859 IMAGE_INSTALL_append = "lib32-glib-2.0"
4860
4861This example enables an additional library named
4862``lib32`` alongside the normal target packages. When combining these
4863"lib32" alternatives, the example uses "x86" for tuning. For information
4864on this particular tuning, see
4865``meta/conf/machine/include/ia32/arch-ia32.inc``.
4866
4867The example then includes ``lib32-glib-2.0`` in all the images, which
4868illustrates one method of including a multiple library dependency. You
4869can use a normal image build to include this dependency, for example:
4870::
4871
4872 $ bitbake core-image-sato
4873
4874You can also build Multilib packages
4875specifically with a command like this:
4876::
4877
4878 $ bitbake lib32-glib-2.0
4879
4880Additional Implementation Details
4881~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4882
4883Generic implementation details as well as details that are specific to
4884package management systems exist. Following are implementation details
4885that exist regardless of the package management system:
4886
4887- The typical convention used for the class extension code as used by
4888 Multilib assumes that all package names specified in
4889 :term:`PACKAGES` that contain
4890 ``${PN}`` have ``${PN}`` at the start of the name. When that
4891 convention is not followed and ``${PN}`` appears at the middle or the
4892 end of a name, problems occur.
4893
4894- The :term:`TARGET_VENDOR`
4895 value under Multilib will be extended to "-vendormlmultilib" (e.g.
4896 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
4897 slightly unwieldy contraction is that any "-" characters in the
4898 vendor string presently break Autoconf's ``config.sub``, and other
4899 separators are problematic for different reasons.
4900
4901For the RPM Package Management System, the following implementation
4902details exist:
4903
4904- A unique architecture is defined for the Multilib packages, along
4905 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
4906 :term:`Build Directory`. For
4907 example, consider ``lib32`` in a ``qemux86-64`` image. The possible
4908 architectures in the system are "all", "qemux86_64",
4909 "lib32_qemux86_64", and "lib32_x86".
4910
4911- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
4912 packaging. The naming for a normal RPM package and a Multilib RPM
4913 package in a ``qemux86-64`` system resolves to something similar to
4914 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
4915 respectively.
4916
4917- When installing a Multilib image, the RPM backend first installs the
4918 base image and then installs the Multilib libraries.
4919
4920- The build system relies on RPM to resolve the identical files in the
4921 two (or more) Multilib packages.
4922
4923For the IPK Package Management System, the following implementation
4924details exist:
4925
4926- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
4927 packaging. The naming for a normal RPM package and a Multilib IPK
4928 package in a ``qemux86-64`` system resolves to something like
4929 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw_x86.ipk``,
4930 respectively.
4931
4932- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
4933 packages with and without the Multilib feature can exist in the same
4934 folder due to the ``${PN}`` differences.
4935
4936- IPK defines a sanity check for Multilib installation using certain
4937 rules for file comparison, overridden, etc.
4938
4939Installing Multiple Versions of the Same Library
4940------------------------------------------------
4941
4942Situations can exist where you need to install and use multiple versions
4943of the same library on the same system at the same time. These
4944situations almost always exist when a library API changes and you have
4945multiple pieces of software that depend on the separate versions of the
4946library. To accommodate these situations, you can install multiple
4947versions of the same library in parallel on the same system.
4948
4949The process is straightforward as long as the libraries use proper
4950versioning. With properly versioned libraries, all you need to do to
4951individually specify the libraries is create separate, appropriately
4952named recipes where the :term:`PN` part of
4953the name includes a portion that differentiates each library version
4954(e.g. the major part of the version number). Thus, instead of having a
4955single recipe that loads one version of a library (e.g. ``clutter``),
4956you provide multiple recipes that result in different versions of the
4957libraries you want. As an example, the following two recipes would allow
4958the two separate versions of the ``clutter`` library to co-exist on the
4959same system:
4960
4961.. code-block:: none
4962
4963 clutter-1.6_1.6.20.bb
4964 clutter-1.8_1.8.4.bb
4965
4966Additionally, if
4967you have other recipes that depend on a given library, you need to use
4968the :term:`DEPENDS` variable to
4969create the dependency. Continuing with the same example, if you want to
4970have a recipe depend on the 1.8 version of the ``clutter`` library, use
4971the following in your recipe:
4972::
4973
4974 DEPENDS = "clutter-1.8"
4975
4976Using x32 psABI
4977===============
4978
4979x32 processor-specific Application Binary Interface (`x32
4980psABI <https://software.intel.com/en-us/node/628948>`__) is a native
498132-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
4982ABI defines the calling conventions between functions in a processing
4983environment. The interface determines what registers are used and what
4984the sizes are for various C data types.
4985
4986Some processing environments prefer using 32-bit applications even when
4987running on Intel 64-bit platforms. Consider the i386 psABI, which is a
4988very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
4989provide efficient use and access of the Intel 64-bit processor
4990resources, leaving the system underutilized. Now consider the x86_64
4991psABI. This ABI is newer and uses 64-bits for data sizes and program
4992pointers. The extra bits increase the footprint size of the programs,
4993libraries, and also increases the memory and file system size
4994requirements. Executing under the x32 psABI enables user programs to
4995utilize CPU and system resources more efficiently while keeping the
4996memory footprint of the applications low. Extra bits are used for
4997registers but not for addressing mechanisms.
4998
4999The Yocto Project supports the final specifications of x32 psABI as
5000follows:
5001
5002- You can create packages and images in x32 psABI format on x86_64
5003 architecture targets.
5004
5005- You can successfully build recipes with the x32 toolchain.
5006
5007- You can create and boot ``core-image-minimal`` and
5008 ``core-image-sato`` images.
5009
5010- RPM Package Manager (RPM) support exists for x32 binaries.
5011
5012- Support for large images exists.
5013
5014To use the x32 psABI, you need to edit your ``conf/local.conf``
5015configuration file as follows:
5016::
5017
5018 MACHINE = "qemux86-64"
5019 DEFAULTTUNE = "x86-64-x32"
5020 baselib = "${@d.getVar('BASE_LIB_tune-' + (d.getVar('DEFAULTTUNE') \
5021 or 'INVALID')) or 'lib'}"
5022
5023Once you have set
5024up your configuration file, use BitBake to build an image that supports
5025the x32 psABI. Here is an example:
5026::
5027
5028 $ bitbake core-image-sato
5029
5030Enabling GObject Introspection Support
5031======================================
5032
5033`GObject
5034introspection <https://wiki.gnome.org/Projects/GObjectIntrospection>`__
5035is the standard mechanism for accessing GObject-based software from
5036runtime environments. GObject is a feature of the GLib library that
5037provides an object framework for the GNOME desktop and related software.
5038GObject Introspection adds information to GObject that allows objects
5039created within it to be represented across different programming
5040languages. If you want to construct GStreamer pipelines using Python, or
5041control UPnP infrastructure using Javascript and GUPnP, GObject
5042introspection is the only way to do it.
5043
5044This section describes the Yocto Project support for generating and
5045packaging GObject introspection data. GObject introspection data is a
5046description of the API provided by libraries built on top of GLib
5047framework, and, in particular, that framework's GObject mechanism.
5048GObject Introspection Repository (GIR) files go to ``-dev`` packages,
5049``typelib`` files go to main packages as they are packaged together with
5050libraries that are introspected.
5051
5052The data is generated when building such a library, by linking the
5053library with a small executable binary that asks the library to describe
5054itself, and then executing the binary and processing its output.
5055
5056Generating this data in a cross-compilation environment is difficult
5057because the library is produced for the target architecture, but its
5058code needs to be executed on the build host. This problem is solved with
5059the OpenEmbedded build system by running the code through QEMU, which
5060allows precisely that. Unfortunately, QEMU does not always work
5061perfectly as mentioned in the "`Known Issues <#known-issues>`__"
5062section.
5063
5064Enabling the Generation of Introspection Data
5065---------------------------------------------
5066
5067Enabling the generation of introspection data (GIR files) in your
5068library package involves the following:
5069
50701. Inherit the
5071 :ref:`gobject-introspection <ref-classes-gobject-introspection>`
5072 class.
5073
50742. Make sure introspection is not disabled anywhere in the recipe or
5075 from anything the recipe includes. Also, make sure that
5076 "gobject-introspection-data" is not in
5077 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5078 and that "qemu-usermode" is not in
5079 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5080 If either of these conditions exist, nothing will happen.
5081
50823. Try to build the recipe. If you encounter build errors that look like
5083 something is unable to find ``.so`` libraries, check where these
5084 libraries are located in the source tree and add the following to the
5085 recipe:
5086 ::
5087
5088 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
5089
5090 .. note::
5091
5092 See recipes in the ``oe-core`` repository that use that
5093 ``GIR_EXTRA_LIBS_PATH`` variable as an example.
5094
50954. Look for any other errors, which probably mean that introspection
5096 support in a package is not entirely standard, and thus breaks down
5097 in a cross-compilation environment. For such cases, custom-made fixes
5098 are needed. A good place to ask and receive help in these cases is
5099 the :ref:`Yocto Project mailing
5100 lists <resources-mailinglist>`.
5101
5102.. note::
5103
5104 Using a library that no longer builds against the latest Yocto
5105 Project release and prints introspection related errors is a good
5106 candidate for the previous procedure.
5107
5108Disabling the Generation of Introspection Data
5109----------------------------------------------
5110
5111You might find that you do not want to generate introspection data. Or,
5112perhaps QEMU does not work on your build host and target architecture
5113combination. If so, you can use either of the following methods to
5114disable GIR file generations:
5115
5116- Add the following to your distro configuration:
5117 ::
5118
5119 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
5120
5121 Adding this statement disables generating introspection data using
5122 QEMU but will still enable building introspection tools and libraries
5123 (i.e. building them does not require the use of QEMU).
5124
5125- Add the following to your machine configuration:
5126 ::
5127
5128 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
5129
5130 Adding this statement disables the use of QEMU when building packages for your
5131 machine. Currently, this feature is used only by introspection
5132 recipes and has the same effect as the previously described option.
5133
5134 .. note::
5135
5136 Future releases of the Yocto Project might have other features
5137 affected by this option.
5138
5139If you disable introspection data, you can still obtain it through other
5140means such as copying the data from a suitable sysroot, or by generating
5141it on the target hardware. The OpenEmbedded build system does not
5142currently provide specific support for these techniques.
5143
5144Testing that Introspection Works in an Image
5145--------------------------------------------
5146
5147Use the following procedure to test if generating introspection data is
5148working in an image:
5149
51501. Make sure that "gobject-introspection-data" is not in
5151 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
5152 and that "qemu-usermode" is not in
5153 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
5154
51552. Build ``core-image-sato``.
5156
51573. Launch a Terminal and then start Python in the terminal.
5158
51594. Enter the following in the terminal:
5160 ::
5161
5162 >>> from gi.repository import GLib
5163 >>> GLib.get_host_name()
5164
51655. For something a little more advanced, enter the following see:
5166 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
5167
5168Known Issues
5169------------
5170
5171The following know issues exist for GObject Introspection Support:
5172
5173- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
5174 introspection data on that architecture.
5175
5176- x32 is not supported by QEMU. Consequently, introspection data is
5177 disabled.
5178
5179- musl causes transient GLib binaries to crash on assertion failures.
5180 Consequently, generating introspection data is disabled.
5181
5182- Because QEMU is not able to run the binaries correctly, introspection
5183 is disabled for some specific packages under specific architectures
5184 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
5185
5186- QEMU usermode might not work properly when running 64-bit binaries
5187 under 32-bit host machines. In particular, "qemumips64" is known to
5188 not work under i686.
5189
5190Optionally Using an External Toolchain
5191======================================
5192
5193You might want to use an external toolchain as part of your development.
5194If this is the case, the fundamental steps you need to accomplish are as
5195follows:
5196
5197- Understand where the installed toolchain resides. For cases where you
5198 need to build the external toolchain, you would need to take separate
5199 steps to build and install the toolchain.
5200
5201- Make sure you add the layer that contains the toolchain to your
5202 ``bblayers.conf`` file through the
5203 :term:`BBLAYERS` variable.
5204
5205- Set the ``EXTERNAL_TOOLCHAIN`` variable in your ``local.conf`` file
5206 to the location in which you installed the toolchain.
5207
5208A good example of an external toolchain used with the Yocto Project is
5209Mentor Graphics Sourcery G++ Toolchain. You can see information on how
5210to use that particular layer in the ``README`` file at
5211https://github.com/MentorEmbedded/meta-sourcery/. You can find
5212further information by reading about the
5213:term:`TCMODE` variable in the Yocto
5214Project Reference Manual's variable glossary.
5215
5216Creating Partitioned Images Using Wic
5217=====================================
5218
5219Creating an image for a particular hardware target using the
5220OpenEmbedded build system does not necessarily mean you can boot that
5221image as is on your device. Physical devices accept and boot images in
5222various ways depending on the specifics of the device. Usually,
5223information about the hardware can tell you what image format the device
5224requires. Should your device require multiple partitions on an SD card,
5225flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
5226create the properly partitioned image.
5227
5228The ``wic`` command generates partitioned images from existing
5229OpenEmbedded build artifacts. Image generation is driven by partitioning
5230commands contained in an Openembedded kickstart file (``.wks``)
5231specified either directly on the command line or as one of a selection
5232of canned kickstart files as shown with the ``wic list images`` command
5233in the "`Using an Existing Kickstart
5234File <#using-a-provided-kickstart-file>`__" section. When you apply the
5235command to a given set of build artifacts, the result is an image or set
5236of images that can be directly written onto media and used on a
5237particular system.
5238
5239.. note::
5240
5241 For a kickstart file reference, see the
5242 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
5243 Chapter in the Yocto Project Reference Manual.
5244
5245The ``wic`` command and the infrastructure it is based on is by
5246definition incomplete. The purpose of the command is to allow the
5247generation of customized images, and as such, was designed to be
5248completely extensible through a plugin interface. See the "`Using the
5249Wic PlugIn Interface <#wic-using-the-wic-plugin-interface>`__" section
5250for information on these plugins.
5251
5252This section provides some background information on Wic, describes what
5253you need to have in place to run the tool, provides instruction on how
5254to use the Wic utility, provides information on using the Wic plugins
5255interface, and provides several examples that show how to use Wic.
5256
5257Background
5258----------
5259
5260This section provides some background on the Wic utility. While none of
5261this information is required to use Wic, you might find it interesting.
5262
5263- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
5264 "oe" diphthong in "oeic" was promoted to the letter "w", because
5265 "oeic" is both difficult to remember and to pronounce.
5266
5267- Wic is loosely based on the Meego Image Creator (``mic``) framework.
5268 The Wic implementation has been heavily modified to make direct use
5269 of OpenEmbedded build artifacts instead of package installation and
5270 configuration, which are already incorporated within the OpenEmbedded
5271 artifacts.
5272
5273- Wic is a completely independent standalone utility that initially
5274 provides easier-to-use and more flexible replacements for an existing
5275 functionality in OE-Core's
5276 :ref:`image-live <ref-classes-image-live>`
5277 class. The difference between Wic and those examples is that with Wic
5278 the functionality of those scripts is implemented by a
5279 general-purpose partitioning language, which is based on Redhat
5280 kickstart syntax.
5281
5282Requirements
5283------------
5284
5285In order to use the Wic utility with the OpenEmbedded Build system, your
5286system needs to meet the following requirements:
5287
5288- The Linux distribution on your development host must support the
5289 Yocto Project. See the ":ref:`detailed-supported-distros`"
5290 section in the Yocto Project Reference Manual for the list of
5291 distributions that support the Yocto Project.
5292
5293- The standard system utilities, such as ``cp``, must be installed on
5294 your development host system.
5295
5296- You must have sourced the build environment setup script (i.e.
5297 :ref:`structure-core-script`) found in the
5298 :term:`Build Directory`.
5299
5300- You need to have the build artifacts already available, which
5301 typically means that you must have already created an image using the
5302 Openembedded build system (e.g. ``core-image-minimal``). While it
5303 might seem redundant to generate an image in order to create an image
5304 using Wic, the current version of Wic requires the artifacts in the
5305 form generated by the OpenEmbedded build system.
5306
5307- You must build several native tools, which are built to run on the
5308 build system:
5309 ::
5310
5311 $ bitbake parted-native dosfstools-native mtools-native
5312
5313- Include "wic" as part of the
5314 :term:`IMAGE_FSTYPES`
5315 variable.
5316
5317- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
5318 as part of the :term:`WKS_FILE` variable
5319
5320Getting Help
5321------------
5322
5323You can get general help for the ``wic`` command by entering the ``wic``
5324command by itself or by entering the command with a help argument as
5325follows:
5326::
5327
5328 $ wic -h
5329 $ wic --help
5330 $ wic help
5331
5332Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
5333``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
5334commands except "help" by using the following form:
5335::
5336
5337 $ wic help command
5338
5339For example, the following command returns help for the ``write``
5340command:
5341::
5342
5343 $ wic help write
5344
5345Wic supports help for three topics: ``overview``, ``plugins``, and
5346``kickstart``. You can get help for any topic using the following form:
5347::
5348
5349 $ wic help topic
5350
5351For example, the following returns overview help for Wic:
5352::
5353
5354 $ wic help overview
5355
5356One additional level of help exists for Wic. You can get help on
5357individual images through the ``list`` command. You can use the ``list``
5358command to return the available Wic images as follows:
5359::
5360
5361 $ wic list images
5362 genericx86 Create an EFI disk image for genericx86*
5363 beaglebone-yocto Create SD card image for Beaglebone
5364 edgerouter Create SD card image for Edgerouter
5365 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5366 directdisk-gpt Create a 'pcbios' direct disk image
5367 mkefidisk Create an EFI disk image
5368 directdisk Create a 'pcbios' direct disk image
5369 systemd-bootdisk Create an EFI disk image with systemd-boot
5370 mkhybridiso Create a hybrid ISO image
5371 sdimage-bootpart Create SD card image with a boot partition
5372 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5373 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5374
5375Once you know the list of available
5376Wic images, you can use ``help`` with the command to get help on a
5377particular image. For example, the following command returns help on the
5378"beaglebone-yocto" image:
5379::
5380
5381 $ wic list beaglebone-yocto help
5382
5383 Creates a partitioned SD card image for Beaglebone.
5384 Boot files are located in the first vfat partition.
5385
5386Operational Modes
5387-----------------
5388
5389You can use Wic in two different modes, depending on how much control
5390you need for specifying the Openembedded build artifacts that are used
5391for creating the image: Raw and Cooked:
5392
5393- *Raw Mode:* You explicitly specify build artifacts through Wic
5394 command-line arguments.
5395
5396- *Cooked Mode:* The current
5397 :term:`MACHINE` setting and image
5398 name are used to automatically locate and provide the build
5399 artifacts. You just supply a kickstart file and the name of the image
5400 from which to use artifacts.
5401
5402Regardless of the mode you use, you need to have the build artifacts
5403ready and available.
5404
5405Raw Mode
5406~~~~~~~~
5407
5408Running Wic in raw mode allows you to specify all the partitions through
5409the ``wic`` command line. The primary use for raw mode is if you have
5410built your kernel outside of the Yocto Project
5411:term:`Build Directory`. In other words, you
5412can point to arbitrary kernel, root filesystem locations, and so forth.
5413Contrast this behavior with cooked mode where Wic looks in the Build
5414Directory (e.g. ``tmp/deploy/images/``\ machine).
5415
5416The general form of the ``wic`` command in raw mode is:
5417::
5418
5419 $ wic create wks_file options ...
5420
5421 Where:
5422
5423 wks_file:
5424 An OpenEmbedded kickstart file. You can provide
5425 your own custom file or use a file from a set of
5426 existing files as described by further options.
5427
5428 optional arguments:
5429 -h, --help show this help message and exit
5430 -o OUTDIR, --outdir OUTDIR
5431 name of directory to create image in
5432 -e IMAGE_NAME, --image-name IMAGE_NAME
5433 name of the image to use the artifacts from e.g. core-
5434 image-sato
5435 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
5436 path to the /rootfs dir to use as the .wks rootfs
5437 source
5438 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
5439 path to the dir containing the boot artifacts (e.g.
5440 /EFI or /syslinux dirs) to use as the .wks bootimg
5441 source
5442 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
5443 path to the dir containing the kernel to use in the
5444 .wks bootimg
5445 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
5446 path to the native sysroot containing the tools to use
5447 to build the image
5448 -s, --skip-build-check
5449 skip the build check
5450 -f, --build-rootfs build rootfs
5451 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
5452 compress image with specified compressor
5453 -m, --bmap generate .bmap
5454 --no-fstab-update Do not change fstab file.
5455 -v VARS_DIR, --vars VARS_DIR
5456 directory with <image>.env files that store bitbake
5457 variables
5458 -D, --debug output debug information
5459
5460.. note::
5461
5462 You do not need root privileges to run Wic. In fact, you should not
5463 run as root when using the utility.
5464
5465Cooked Mode
5466~~~~~~~~~~~
5467
5468Running Wic in cooked mode leverages off artifacts in the Build
5469Directory. In other words, you do not have to specify kernel or root
5470filesystem locations as part of the command. All you need to provide is
5471a kickstart file and the name of the image from which to use artifacts
5472by using the "-e" option. Wic looks in the Build Directory (e.g.
5473``tmp/deploy/images/``\ machine) for artifacts.
5474
5475The general form of the ``wic`` command using Cooked Mode is as follows:
5476::
5477
5478 $ wic create wks_file -e IMAGE_NAME
5479
5480 Where:
5481
5482 wks_file:
5483 An OpenEmbedded kickstart file. You can provide
5484 your own custom file or use a file from a set of
5485 existing files provided with the Yocto Project
5486 release.
5487
5488 required argument:
5489 -e IMAGE_NAME, --image-name IMAGE_NAME
5490 name of the image to use the artifacts from e.g. core-
5491 image-sato
5492
5493Using an Existing Kickstart File
5494--------------------------------
5495
5496If you do not want to create your own kickstart file, you can use an
5497existing file provided by the Wic installation. As shipped, kickstart
5498files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
5499following two locations:
5500::
5501
5502 poky/meta-yocto-bsp/wic
5503 poky/scripts/lib/wic/canned-wks
5504
5505Use the following command to list the available kickstart files:
5506::
5507
5508 $ wic list images
5509 genericx86 Create an EFI disk image for genericx86*
5510 beaglebone-yocto Create SD card image for Beaglebone
5511 edgerouter Create SD card image for Edgerouter
5512 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
5513 directdisk-gpt Create a 'pcbios' direct disk image
5514 mkefidisk Create an EFI disk image
5515 directdisk Create a 'pcbios' direct disk image
5516 systemd-bootdisk Create an EFI disk image with systemd-boot
5517 mkhybridiso Create a hybrid ISO image
5518 sdimage-bootpart Create SD card image with a boot partition
5519 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
5520 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
5521
5522When you use an existing file, you
5523do not have to use the ``.wks`` extension. Here is an example in Raw
5524Mode that uses the ``directdisk`` file:
5525::
5526
5527 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
5528 -k kernel_dir -n native_sysroot
5529
5530Here are the actual partition language commands used in the
5531``genericx86.wks`` file to generate an image:
5532::
5533
5534 # short-description: Create an EFI disk image for genericx86*
5535 # long-description: Creates a partitioned EFI disk image for genericx86* machines
5536 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
5537 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
5538 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
5539
5540 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
5541
5542Using the Wic Plugin Interface
5543------------------------------
5544
5545You can extend and specialize Wic functionality by using Wic plugins.
5546This section explains the Wic plugin interface.
5547
5548.. note::
5549
5550 Wic plugins consist of "source" and "imager" plugins. Imager plugins
5551 are beyond the scope of this section.
5552
5553Source plugins provide a mechanism to customize partition content during
5554the Wic image generation process. You can use source plugins to map
5555values that you specify using ``--source`` commands in kickstart files
5556(i.e. ``*.wks``) to a plugin implementation used to populate a given
5557partition.
5558
5559.. note::
5560
5561 If you use plugins that have build-time dependencies (e.g. native
5562 tools, bootloaders, and so forth) when building a Wic image, you need
5563 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
5564 variable.
5565
5566Source plugins are subclasses defined in plugin files. As shipped, the
5567Yocto Project provides several plugin files. You can see the source
5568plugin files that ship with the Yocto Project
5569:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
5570Each of these plugin files contains source plugins that are designed to
5571populate a specific Wic image partition.
5572
5573Source plugins are subclasses of the ``SourcePlugin`` class, which is
5574defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
5575the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
5576file is a subclass of the ``SourcePlugin`` class, which is found in the
5577``pluginbase.py`` file.
5578
5579You can also implement source plugins in a layer outside of the Source
5580Repositories (external layer). To do so, be sure that your plugin files
5581are located in a directory whose path is
5582``scripts/lib/wic/plugins/source/`` within your external layer. When the
5583plugin files are located there, the source plugins they contain are made
5584available to Wic.
5585
5586When the Wic implementation needs to invoke a partition-specific
5587implementation, it looks for the plugin with the same name as the
5588``--source`` parameter used in the kickstart file given to that
5589partition. For example, if the partition is set up using the following
5590command in a kickstart file:
5591::
5592
5593 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
5594
5595The methods defined as class
5596members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
5597``bootimg-pcbios.py`` plugin file are used.
5598
5599To be more concrete, here is the corresponding plugin definition from
5600the ``bootimg-pcbios.py`` file for the previous command along with an
5601example method called by the Wic implementation when it needs to prepare
5602a partition using an implementation-specific function:
5603::
5604
5605 .
5606 .
5607 .
5608 class BootimgPcbiosPlugin(SourcePlugin):
5609 """
5610 Create MBR boot partition and install syslinux on it.
5611 """
5612
5613 name = 'bootimg-pcbios'
5614 .
5615 .
5616 .
5617 @classmethod
5618 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
5619 oe_builddir, bootimg_dir, kernel_dir,
5620 rootfs_dir, native_sysroot):
5621 """
5622 Called to do the actual content population for a partition i.e. it
5623 'prepares' the partition to be incorporated into the image.
5624 In this case, prepare content for legacy bios boot partition.
5625 """
5626 .
5627 .
5628 .
5629
5630If a
5631subclass (plugin) itself does not implement a particular function, Wic
5632locates and uses the default version in the superclass. It is for this
5633reason that all source plugins are derived from the ``SourcePlugin``
5634class.
5635
5636The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
5637a set of methods that source plugins can implement or override. Any
5638plugins (subclass of ``SourcePlugin``) that do not implement a
5639particular method inherit the implementation of the method from the
5640``SourcePlugin`` class. For more information, see the ``SourcePlugin``
5641class in the ``pluginbase.py`` file for details:
5642
5643The following list describes the methods implemented in the
5644``SourcePlugin`` class:
5645
5646- ``do_prepare_partition()``: Called to populate a partition with
5647 actual content. In other words, the method prepares the final
5648 partition image that is incorporated into the disk image.
5649
5650- ``do_configure_partition()``: Called before
5651 ``do_prepare_partition()`` to create custom configuration files for a
5652 partition (e.g. syslinux or grub configuration files).
5653
5654- ``do_install_disk()``: Called after all partitions have been
5655 prepared and assembled into a disk image. This method provides a hook
5656 to allow finalization of a disk image (e.g. writing an MBR).
5657
5658- ``do_stage_partition()``: Special content-staging hook called
5659 before ``do_prepare_partition()``. This method is normally empty.
5660
5661 Typically, a partition just uses the passed-in parameters (e.g. the
5662 unmodified value of ``bootimg_dir``). However, in some cases, things
5663 might need to be more tailored. As an example, certain files might
5664 additionally need to be taken from ``bootimg_dir + /boot``. This hook
5665 allows those files to be staged in a customized fashion.
5666
5667 .. note::
5668
5669 ``get_bitbake_var()`` allows you to access non-standard variables that
5670 you might want to use for this behavior.
5671
5672You can extend the source plugin mechanism. To add more hooks, create
5673more source plugin methods within ``SourcePlugin`` and the corresponding
5674derived subclasses. The code that calls the plugin methods uses the
5675``plugin.get_source_plugin_methods()`` function to find the method or
5676methods needed by the call. Retrieval of those methods is accomplished
5677by filling up a dict with keys that contain the method names of
5678interest. On success, these will be filled in with the actual methods.
5679See the Wic implementation for examples and details.
5680
5681Wic Examples
5682------------
5683
5684This section provides several examples that show how to use the Wic
5685utility. All the examples assume the list of requirements in the
5686"`Requirements <#wic-requirements>`__" section have been met. The
5687examples assume the previously generated image is
5688``core-image-minimal``.
5689
5690Generate an Image using an Existing Kickstart File
5691~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5692
5693This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
5694file:
5695::
5696
5697 $ wic create mkefidisk -e core-image-minimal
5698 INFO: Building wic-tools...
5699 .
5700 .
5701 .
5702 INFO: The new image(s) can be found here:
5703 ./mkefidisk-201804191017-sda.direct
5704
5705 The following build artifacts were used to create the image(s):
5706 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5707 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5708 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5709 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5710
5711 INFO: The image(s) were created using OE kickstart file:
5712 /home/stephano/build/master/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
5713
5714The previous example shows the easiest way to create an image by running
5715in cooked mode and supplying a kickstart file and the "-e" option to
5716point to the existing build artifacts. Your ``local.conf`` file needs to
5717have the :term:`MACHINE` variable set
5718to the machine you are using, which is "qemux86" in this example.
5719
5720Once the image builds, the output provides image location, artifact use,
5721and kickstart file information.
5722
5723.. note::
5724
5725 You should always verify the details provided in the output to make
5726 sure that the image was indeed created exactly as expected.
5727
5728Continuing with the example, you can now write the image from the Build
5729Directory onto a USB stick, or whatever media for which you built your
5730image, and boot from the media. You can write the image by using
5731``bmaptool`` or ``dd``:
5732::
5733
5734 $ oe-run-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
5735
5736or ::
5737
5738 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
5739
5740.. note::
5741
5742 For more information on how to use the ``bmaptool``
5743 to flash a device with an image, see the
5744 ":ref:`dev-manual/common-tasks:flashing images using \`\`bmaptool\`\``"
5745 section.
5746
5747Using a Modified Kickstart File
5748~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5749
5750Because partitioned image creation is driven by the kickstart file, it
5751is easy to affect image creation by changing the parameters in the file.
5752This next example demonstrates that through modification of the
5753``directdisk-gpt`` kickstart file.
5754
5755As mentioned earlier, you can use the command ``wic list images`` to
5756show the list of existing kickstart files. The directory in which the
5757``directdisk-gpt.wks`` file resides is
5758``scripts/lib/image/canned-wks/``, which is located in the
5759:term:`Source Directory` (e.g. ``poky``).
5760Because available files reside in this directory, you can create and add
5761your own custom files to the directory. Subsequent use of the
5762``wic list images`` command would then include your kickstart files.
5763
5764In this example, the existing ``directdisk-gpt`` file already does most
5765of what is needed. However, for the hardware in this example, the image
5766will need to boot from ``sdb`` instead of ``sda``, which is what the
5767``directdisk-gpt`` kickstart file uses.
5768
5769The example begins by making a copy of the ``directdisk-gpt.wks`` file
5770in the ``scripts/lib/image/canned-wks`` directory and then by changing
5771the lines that specify the target disk from which to boot.
5772::
5773
5774 $ cp /home/stephano/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
5775 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5776
5777Next, the example modifies the ``directdisksdb-gpt.wks`` file and
5778changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
5779example changes the following two lines and leaves the remaining lines
5780untouched:
5781::
5782
5783 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
5784 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
5785
5786Once the lines are changed, the
5787example generates the ``directdisksdb-gpt`` image. The command points
5788the process at the ``core-image-minimal`` artifacts for the Next Unit of
5789Computing (nuc) :term:`MACHINE` the
5790``local.conf``.
5791::
5792
5793 $ wic create directdisksdb-gpt -e core-image-minimal
5794 INFO: Building wic-tools...
5795 .
5796 .
5797 .
5798 Initialising tasks: 100% |#######################################| Time: 0:00:01
5799 NOTE: Executing SetScene Tasks
5800 NOTE: Executing RunQueue Tasks
5801 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
5802 INFO: Creating image(s)...
5803
5804 INFO: The new image(s) can be found here:
5805 ./directdisksdb-gpt-201710090938-sdb.direct
5806
5807 The following build artifacts were used to create the image(s):
5808 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5809 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5810 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5811 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5812
5813 INFO: The image(s) were created using OE kickstart file:
5814 /home/stephano/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
5815
5816Continuing with the example, you can now directly ``dd`` the image to a
5817USB stick, or whatever media for which you built your image, and boot
5818the resulting media:
5819::
5820
5821 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
5822 140966+0 records in
5823 140966+0 records out
5824 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
5825 $ sudo eject /dev/sdb
5826
5827Using a Modified Kickstart File and Running in Raw Mode
5828~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5829
5830This next example manually specifies each build artifact (runs in Raw
5831Mode) and uses a modified kickstart file. The example also uses the
5832``-o`` option to cause Wic to create the output somewhere other than the
5833default output directory, which is the current directory:
5834::
5835
5836 $ wic create /home/stephano/my_yocto/test.wks -o /home/stephano/testwic \
5837 --rootfs-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
5838 --bootimg-dir /home/stephano/build/master/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
5839 --kernel-dir /home/stephano/build/master/build/tmp/deploy/images/qemux86 \
5840 --native-sysroot /home/stephano/build/master/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
5841
5842 INFO: Creating image(s)...
5843
5844 INFO: The new image(s) can be found here:
5845 /home/stephano/testwic/test-201710091445-sdb.direct
5846
5847 The following build artifacts were used to create the image(s):
5848 ROOTFS_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
5849 BOOTIMG_DIR: /home/stephano/build/master/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
5850 KERNEL_DIR: /home/stephano/build/master/build/tmp-glibc/deploy/images/qemux86
5851 NATIVE_SYSROOT: /home/stephano/build/master/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
5852
5853 INFO: The image(s) were created using OE kickstart file:
5854 /home/stephano/my_yocto/test.wks
5855
5856For this example,
5857:term:`MACHINE` did not have to be
5858specified in the ``local.conf`` file since the artifact is manually
5859specified.
5860
5861Using Wic to Manipulate an Image
5862~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5863
5864Wic image manipulation allows you to shorten turnaround time during
5865image development. For example, you can use Wic to delete the kernel
5866partition of a Wic image and then insert a newly built kernel. This
5867saves you time from having to rebuild the entire image each time you
5868modify the kernel.
5869
5870.. note::
5871
5872 In order to use Wic to manipulate a Wic image as in this example,
5873 your development machine must have the ``mtools`` package installed.
5874
5875The following example examines the contents of the Wic image, deletes
5876the existing kernel, and then inserts a new kernel:
5877
58781. *List the Partitions:* Use the ``wic ls`` command to list all the
5879 partitions in the Wic image:
5880 ::
5881
5882 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
5883 Num Start End Size Fstype
5884 1 1048576 25041919 23993344 fat16
5885 2 25165824 72157183 46991360 ext4
5886
5887 The previous output shows two partitions in the
5888 ``core-image-minimal-qemux86.wic`` image.
5889
58902. *Examine a Particular Partition:* Use the ``wic ls`` command again
5891 but in a different form to examine a particular partition.
5892
5893 .. note::
5894
5895 You can get command usage on any Wic command using the following
5896 form:
5897 ::
5898
5899 $ wic help command
5900
5901
5902 For example, the following command shows you the various ways to
5903 use the
5904 wic ls
5905 command:
5906 ::
5907
5908 $ wic help ls
5909
5910
5911 The following command shows what is in Partition one:
5912 ::
5913
5914 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
5915 Volume in drive : is boot
5916 Volume Serial Number is E894-1809
5917 Directory for ::/
5918
5919 libcom32 c32 186500 2017-10-09 16:06
5920 libutil c32 24148 2017-10-09 16:06
5921 syslinux cfg 220 2017-10-09 16:06
5922 vesamenu c32 27104 2017-10-09 16:06
5923 vmlinuz 6904608 2017-10-09 16:06
5924 5 files 7 142 580 bytes
5925 16 582 656 bytes free
5926
5927 The previous output shows five files, with the
5928 ``vmlinuz`` being the kernel.
5929
5930 .. note::
5931
5932 If you see the following error, you need to update or create a
5933 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
5934 in the file. Then, run the Wic command again:
5935 ::
5936
5937 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
5938 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
5939 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
5940
5941
59423. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
5943 ``vmlinuz`` file (kernel):
5944 ::
5945
5946 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
5947
59484. *Add In the New Kernel:* Use the ``wic cp`` command to add the
5949 updated kernel to the Wic image. Depending on how you built your
5950 kernel, it could be in different places. If you used ``devtool`` and
5951 an SDK to build your kernel, it resides in the ``tmp/work`` directory
5952 of the extensible SDK. If you used ``make`` to build the kernel, the
5953 kernel will be in the ``workspace/sources`` area.
5954
5955 The following example assumes ``devtool`` was used to build the
5956 kernel:
5957 ::
5958
5959 cp ~/poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
5960 ~/poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
5961
5962 Once the new kernel is added back into the image, you can use the
5963 ``dd`` command or :ref:`bmaptool
5964 <dev-manual/common-tasks:flashing images using \`\`bmaptool\`\`>`
5965 to flash your wic image onto an SD card or USB stick and test your
5966 target.
5967
5968 .. note::
5969
5970 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
5971
5972Flashing Images Using ``bmaptool``
5973==================================
5974
5975A fast and easy way to flash an image to a bootable device is to use
5976Bmaptool, which is integrated into the OpenEmbedded build system.
5977Bmaptool is a generic tool that creates a file's block map (bmap) and
5978then uses that map to copy the file. As compared to traditional tools
5979such as dd or cp, Bmaptool can copy (or flash) large files like raw
5980system image files much faster.
5981
5982.. note::
5983
5984 - If you are using Ubuntu or Debian distributions, you can install
5985 the ``bmap-tools`` package using the following command and then
5986 use the tool without specifying ``PATH`` even from the root
5987 account:
5988 ::
5989
5990 $ sudo apt-get install bmap-tools
5991
5992 - If you are unable to install the ``bmap-tools`` package, you will
5993 need to build Bmaptool before using it. Use the following command:
5994 ::
5995
5996 $ bitbake bmap-tools-native
5997
5998Following, is an example that shows how to flash a Wic image. Realize
5999that while this example uses a Wic image, you can use Bmaptool to flash
6000any type of image. Use these steps to flash an image using Bmaptool:
6001
60021. *Update your local.conf File:* You need to have the following set
6003 in your ``local.conf`` file before building your image:
6004 ::
6005
6006 IMAGE_FSTYPES += "wic wic.bmap"
6007
60082. *Get Your Image:* Either have your image ready (pre-built with the
6009 :term:`IMAGE_FSTYPES`
6010 setting previously mentioned) or take the step to build the image:
6011 ::
6012
6013 $ bitbake image
6014
60153. *Flash the Device:* Flash the device with the image by using Bmaptool
6016 depending on your particular setup. The following commands assume the
6017 image resides in the Build Directory's ``deploy/images/`` area:
6018
6019 - If you have write access to the media, use this command form:
6020 ::
6021
6022 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6023
6024 - If you do not have write access to the media, set your permissions
6025 first and then use the same command form:
6026 ::
6027
6028 $ sudo chmod 666 /dev/sdX
6029 $ oe-run-native bmap-tools-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
6030
6031For help on the ``bmaptool`` command, use the following command:
6032::
6033
6034 $ bmaptool --help
6035
6036Making Images More Secure
6037=========================
6038
6039Security is of increasing concern for embedded devices. Consider the
6040issues and problems discussed in just this sampling of work found across
6041the Internet:
6042
6043- *"*\ `Security Risks of Embedded
6044 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
6045 by Bruce Schneier
6046
6047- *"*\ `Internet Census
6048 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
6049 Botnet
6050
6051- *"*\ `Security Issues for Embedded
6052 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
6053 by Jake Edge
6054
6055When securing your image is of concern, there are steps, tools, and
6056variables that you can consider to help you reach the security goals you
6057need for your particular device. Not all situations are identical when
6058it comes to making an image secure. Consequently, this section provides
6059some guidance and suggestions for consideration when you want to make
6060your image more secure.
6061
6062.. note::
6063
6064 Because the security requirements and risks are different for every
6065 type of device, this section cannot provide a complete reference on
6066 securing your custom OS. It is strongly recommended that you also
6067 consult other sources of information on embedded Linux system
6068 hardening and on security.
6069
6070General Considerations
6071----------------------
6072
6073General considerations exist that help you create more secure images.
6074You should consider the following suggestions to help make your device
6075more secure:
6076
6077- Scan additional code you are adding to the system (e.g. application
6078 code) by using static analysis tools. Look for buffer overflows and
6079 other potential security problems.
6080
6081- Pay particular attention to the security for any web-based
6082 administration interface.
6083
6084 Web interfaces typically need to perform administrative functions and
6085 tend to need to run with elevated privileges. Thus, the consequences
6086 resulting from the interface's security becoming compromised can be
6087 serious. Look for common web vulnerabilities such as
6088 cross-site-scripting (XSS), unvalidated inputs, and so forth.
6089
6090 As with system passwords, the default credentials for accessing a
6091 web-based interface should not be the same across all devices. This
6092 is particularly true if the interface is enabled by default as it can
6093 be assumed that many end-users will not change the credentials.
6094
6095- Ensure you can update the software on the device to mitigate
6096 vulnerabilities discovered in the future. This consideration
6097 especially applies when your device is network-enabled.
6098
6099- Ensure you remove or disable debugging functionality before producing
6100 the final image. For information on how to do this, see the
6101 "`Considerations Specific to the OpenEmbedded Build
6102 System <#considerations-specific-to-the-openembedded-build-system>`__"
6103 section.
6104
6105- Ensure you have no network services listening that are not needed.
6106
6107- Remove any software from the image that is not needed.
6108
6109- Enable hardware support for secure boot functionality when your
6110 device supports this functionality.
6111
6112Security Flags
6113--------------
6114
6115The Yocto Project has security flags that you can enable that help make
6116your build output more secure. The security flags are in the
6117``meta/conf/distro/include/security_flags.inc`` file in your
6118:term:`Source Directory` (e.g. ``poky``).
6119
6120.. note::
6121
6122 Depending on the recipe, certain security flags are enabled and
6123 disabled by default.
6124
6125Use the following line in your ``local.conf`` file or in your custom
6126distribution configuration file to enable the security compiler and
6127linker flags for your build:
6128::
6129
6130 require conf/distro/include/security_flags.inc
6131
6132Considerations Specific to the OpenEmbedded Build System
6133--------------------------------------------------------
6134
6135You can take some steps that are specific to the OpenEmbedded build
6136system to make your images more secure:
6137
6138- Ensure "debug-tweaks" is not one of your selected
6139 :term:`IMAGE_FEATURES`.
6140 When creating a new project, the default is to provide you with an
6141 initial ``local.conf`` file that enables this feature using the
6142 :term:`EXTRA_IMAGE_FEATURES`
6143 variable with the line:
6144 ::
6145
6146 EXTRA_IMAGE_FEATURES = "debug-tweaks"
6147
6148 To disable that feature, simply comment out that line in your
6149 ``local.conf`` file, or make sure ``IMAGE_FEATURES`` does not contain
6150 "debug-tweaks" before producing your final image. Among other things,
6151 leaving this in place sets the root password as blank, which makes
6152 logging in for debugging or inspection easy during development but
6153 also means anyone can easily log in during production.
6154
6155- It is possible to set a root password for the image and also to set
6156 passwords for any extra users you might add (e.g. administrative or
6157 service type users). When you set up passwords for multiple images or
6158 users, you should not duplicate passwords.
6159
6160 To set up passwords, use the
6161 :ref:`extrausers <ref-classes-extrausers>`
6162 class, which is the preferred method. For an example on how to set up
6163 both root and user passwords, see the
6164 ":ref:`extrausers.bbclass <ref-classes-extrausers>`"
6165 section.
6166
6167 .. note::
6168
6169 When adding extra user accounts or setting a root password, be
6170 cautious about setting the same password on every device. If you
6171 do this, and the password you have set is exposed, then every
6172 device is now potentially compromised. If you need this access but
6173 want to ensure security, consider setting a different, random
6174 password for each device. Typically, you do this as a separate
6175 step after you deploy the image onto the device.
6176
6177- Consider enabling a Mandatory Access Control (MAC) framework such as
6178 SMACK or SELinux and tuning it appropriately for your device's usage.
6179 You can find more information in the
6180 :yocto_git:`meta-selinux </meta-selinux/>` layer.
6181
6182Tools for Hardening Your Image
6183------------------------------
6184
6185The Yocto Project provides tools for making your image more secure. You
6186can find these tools in the ``meta-security`` layer of the
6187:yocto_git:`Yocto Project Source Repositories <>`.
6188
6189Creating Your Own Distribution
6190==============================
6191
6192When you build an image using the Yocto Project and do not alter any
6193distribution :term:`Metadata`, you are
6194creating a Poky distribution. If you wish to gain more control over
6195package alternative selections, compile-time options, and other
6196low-level configurations, you can create your own distribution.
6197
6198To create your own distribution, the basic steps consist of creating
6199your own distribution layer, creating your own distribution
6200configuration file, and then adding any needed code and Metadata to the
6201layer. The following steps provide some more detail:
6202
6203- *Create a layer for your new distro:* Create your distribution layer
6204 so that you can keep your Metadata and code for the distribution
6205 separate. It is strongly recommended that you create and use your own
6206 layer for configuration and code. Using your own layer as compared to
6207 just placing configurations in a ``local.conf`` configuration file
6208 makes it easier to reproduce the same build configuration when using
6209 multiple build machines. See the
6210 ":ref:`dev-manual/common-tasks:creating a general layer using the \`\`bitbake-layers\`\` script`"
6211 section for information on how to quickly set up a layer.
6212
6213- *Create the distribution configuration file:* The distribution
6214 configuration file needs to be created in the ``conf/distro``
6215 directory of your layer. You need to name it using your distribution
6216 name (e.g. ``mydistro.conf``).
6217
6218 .. note::
6219
6220 The :term:`DISTRO` variable in your ``local.conf`` file determines the
6221 name of your distribution.
6222
6223 You can split out parts of your configuration file into include files
6224 and then "require" them from within your distribution configuration
6225 file. Be sure to place the include files in the
6226 ``conf/distro/include`` directory of your layer. A common example
6227 usage of include files would be to separate out the selection of
6228 desired version and revisions for individual recipes.
6229
6230 Your configuration file needs to set the following required
6231 variables:
6232
6233 - :term:`DISTRO_NAME`
6234
6235 - :term:`DISTRO_VERSION`
6236
6237 These following variables are optional and you typically set them
6238 from the distribution configuration file:
6239
6240 - :term:`DISTRO_FEATURES`
6241
6242 - :term:`DISTRO_EXTRA_RDEPENDS`
6243
6244 - :term:`DISTRO_EXTRA_RRECOMMENDS`
6245
6246 - :term:`TCLIBC`
6247
6248 .. tip::
6249
6250 If you want to base your distribution configuration file on the
6251 very basic configuration from OE-Core, you can use
6252 ``conf/distro/defaultsetup.conf`` as a reference and just include
6253 variables that differ as compared to ``defaultsetup.conf``.
6254 Alternatively, you can create a distribution configuration file
6255 from scratch using the ``defaultsetup.conf`` file or configuration files
6256 from other distributions such as Poky or Angstrom as references.
6257
6258- *Provide miscellaneous variables:* Be sure to define any other
6259 variables for which you want to create a default or enforce as part
6260 of the distribution configuration. You can include nearly any
6261 variable from the ``local.conf`` file. The variables you use are not
6262 limited to the list in the previous bulleted item.
6263
6264- *Point to Your distribution configuration file:* In your
6265 ``local.conf`` file in the :term:`Build Directory`,
6266 set your
6267 :term:`DISTRO` variable to point to
6268 your distribution's configuration file. For example, if your
6269 distribution's configuration file is named ``mydistro.conf``, then
6270 you point to it as follows:
6271 ::
6272
6273 DISTRO = "mydistro"
6274
6275- *Add more to the layer if necessary:* Use your layer to hold other
6276 information needed for the distribution:
6277
6278 - Add recipes for installing distro-specific configuration files
6279 that are not already installed by another recipe. If you have
6280 distro-specific configuration files that are included by an
6281 existing recipe, you should add an append file (``.bbappend``) for
6282 those. For general information and recommendations on how to add
6283 recipes to your layer, see the "`Creating Your Own
6284 Layer <#creating-your-own-layer>`__" and "`Following Best
6285 Practices When Creating
6286 Layers <#best-practices-to-follow-when-creating-layers>`__"
6287 sections.
6288
6289 - Add any image recipes that are specific to your distribution.
6290
6291 - Add a ``psplash`` append file for a branded splash screen. For
6292 information on append files, see the "`Using .bbappend Files in
6293 Your Layer <#using-bbappend-files>`__" section.
6294
6295 - Add any other append files to make custom changes that are
6296 specific to individual recipes.
6297
6298Creating a Custom Template Configuration Directory
6299==================================================
6300
6301If you are producing your own customized version of the build system for
6302use by other users, you might want to customize the message shown by the
6303setup script or you might want to change the template configuration
6304files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
6305new build directory.
6306
6307The OpenEmbedded build system uses the environment variable
6308``TEMPLATECONF`` to locate the directory from which it gathers
6309configuration information that ultimately ends up in the
6310:term:`Build Directory` ``conf`` directory.
6311By default, ``TEMPLATECONF`` is set as follows in the ``poky``
6312repository:
6313::
6314
6315 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
6316
6317This is the
6318directory used by the build system to find templates from which to build
6319some key configuration files. If you look at this directory, you will
6320see the ``bblayers.conf.sample``, ``local.conf.sample``, and
6321``conf-notes.txt`` files. The build system uses these files to form the
6322respective ``bblayers.conf`` file, ``local.conf`` file, and display the
6323list of BitBake targets when running the setup script.
6324
6325To override these default configuration files with configurations you
6326want used within every new Build Directory, simply set the
6327``TEMPLATECONF`` variable to your directory. The ``TEMPLATECONF``
6328variable is set in the ``.templateconf`` file, which is in the top-level
6329:term:`Source Directory` folder
6330(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
6331directory.
6332
6333Best practices dictate that you should keep your template configuration
6334directory in your custom distribution layer. For example, suppose you
6335have a layer named ``meta-mylayer`` located in your home directory and
6336you want your template configuration directory named ``myconf``.
6337Changing the ``.templateconf`` as follows causes the OpenEmbedded build
6338system to look in your directory and base its configuration files on the
6339``*.sample`` configuration files it finds. The final configuration files
6340(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
6341your Build Directory, but they are based on your ``*.sample`` files.
6342::
6343
6344 TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
6345
6346Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
6347also resides in the default ``meta-poky/conf`` directory. The script
6348that sets up the build environment (i.e.
6349:ref:`structure-core-script`) uses this file to
6350display BitBake targets as part of the script output. Customizing this
6351``conf-notes.txt`` file is a good way to make sure your list of custom
6352targets appears as part of the script's output.
6353
6354Here is the default list of targets displayed as a result of running
6355either of the setup scripts:
6356::
6357
6358 You can now run 'bitbake <target>'
6359
6360 Common targets are:
6361 core-image-minimal
6362 core-image-sato
6363 meta-toolchain
6364 meta-ide-support
6365
6366Changing the listed common targets is as easy as editing your version of
6367``conf-notes.txt`` in your custom template configuration directory and
6368making sure you have ``TEMPLATECONF`` set to your directory.
6369
6370Conserving Disk Space During Builds
6371===================================
6372
6373To help conserve disk space during builds, you can add the following
6374statement to your project's ``local.conf`` configuration file found in
6375the :term:`Build Directory`:
6376::
6377
6378 INHERIT += "rm_work"
6379
6380Adding this statement deletes the work directory used for
6381building a recipe once the recipe is built. For more information on
6382"rm_work", see the
6383:ref:`rm_work <ref-classes-rm-work>` class in the
6384Yocto Project Reference Manual.
6385
6386Working with Packages
6387=====================
6388
6389This section describes a few tasks that involve packages:
6390
6391- `Excluding packages from an
6392 image <#excluding-packages-from-an-image>`__
6393
6394- `Incrementing a binary package
6395 version <#incrementing-a-binary-package-version>`__
6396
6397- `Handling optional module
6398 packaging <#handling-optional-module-packaging>`__
6399
6400- `Using runtime package
6401 management <#using-runtime-package-management>`__
6402
6403- `Generating and using signed
6404 packages <#generating-and-using-signed-packages>`__
6405
6406- `Setting up and running package test
6407 (ptest) <#testing-packages-with-ptest>`__
6408
6409- `Creating node package manager (NPM)
6410 packages <#creating-node-package-manager-npm-packages>`__
6411
6412- `Adding custom metadata to
6413 packages <#adding-custom-metadata-to-packages>`__
6414
6415Excluding Packages from an Image
6416--------------------------------
6417
6418You might find it necessary to prevent specific packages from being
6419installed into an image. If so, you can use several variables to direct
6420the build system to essentially ignore installing recommended packages
6421or to not install a package at all.
6422
6423The following list introduces variables you can use to prevent packages
6424from being installed into your image. Each of these variables only works
6425with IPK and RPM package types. Support for Debian packages does not
6426exist. Also, you can use these variables from your ``local.conf`` file
6427or attach them to a specific image recipe by using a recipe name
6428override. For more detail on the variables, see the descriptions in the
6429Yocto Project Reference Manual's glossary chapter.
6430
6431- :term:`BAD_RECOMMENDATIONS`:
6432 Use this variable to specify "recommended-only" packages that you do
6433 not want installed.
6434
6435- :term:`NO_RECOMMENDATIONS`:
6436 Use this variable to prevent all "recommended-only" packages from
6437 being installed.
6438
6439- :term:`PACKAGE_EXCLUDE`:
6440 Use this variable to prevent specific packages from being installed
6441 regardless of whether they are "recommended-only" or not. You need to
6442 realize that the build process could fail with an error when you
6443 prevent the installation of a package whose presence is required by
6444 an installed package.
6445
6446Incrementing a Package Version
6447------------------------------
6448
6449This section provides some background on how binary package versioning
6450is accomplished and presents some of the services, variables, and
6451terminology involved.
6452
6453In order to understand binary package versioning, you need to consider
6454the following:
6455
6456- Binary Package: The binary package that is eventually built and
6457 installed into an image.
6458
6459- Binary Package Version: The binary package version is composed of two
6460 components - a version and a revision.
6461
6462 .. note::
6463
6464 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
6465 but this discussion for the most part ignores ``PE``.
6466
6467 The version and revision are taken from the
6468 :term:`PV` and
6469 :term:`PR` variables, respectively.
6470
6471- ``PV``: The recipe version. ``PV`` represents the version of the
6472 software being packaged. Do not confuse ``PV`` with the binary
6473 package version.
6474
6475- ``PR``: The recipe revision.
6476
6477- :term:`SRCPV`: The OpenEmbedded
6478 build system uses this string to help define the value of ``PV`` when
6479 the source code revision needs to be included in it.
6480
6481- :yocto_wiki:`PR Service </PR_Service>`: A
6482 network-based service that helps automate keeping package feeds
6483 compatible with existing package manager applications such as RPM,
6484 APT, and OPKG.
6485
6486Whenever the binary package content changes, the binary package version
6487must change. Changing the binary package version is accomplished by
6488changing or "bumping" the ``PR`` and/or ``PV`` values. Increasing these
6489values occurs one of two ways:
6490
6491- Automatically using a Package Revision Service (PR Service).
6492
6493- Manually incrementing the ``PR`` and/or ``PV`` variables.
6494
6495Given a primary challenge of any build system and its users is how to
6496maintain a package feed that is compatible with existing package manager
6497applications such as RPM, APT, and OPKG, using an automated system is
6498much preferred over a manual system. In either system, the main
6499requirement is that binary package version numbering increases in a
6500linear fashion and that a number of version components exist that
6501support that linear progression. For information on how to ensure
6502package revisioning remains linear, see the "`Automatically Incrementing
6503a Binary Package Revision
6504Number <#automatically-incrementing-a-binary-package-revision-number>`__"
6505section.
6506
6507The following three sections provide related information on the PR
6508Service, the manual method for "bumping" ``PR`` and/or ``PV``, and on
6509how to ensure binary package revisioning remains linear.
6510
6511Working With a PR Service
6512~~~~~~~~~~~~~~~~~~~~~~~~~
6513
6514As mentioned, attempting to maintain revision numbers in the
6515:term:`Metadata` is error prone, inaccurate,
6516and causes problems for people submitting recipes. Conversely, the PR
6517Service automatically generates increasing numbers, particularly the
6518revision field, which removes the human element.
6519
6520.. note::
6521
6522 For additional information on using a PR Service, you can see the
6523 :yocto_wiki:`PR Service </PR_Service>` wiki page.
6524
6525The Yocto Project uses variables in order of decreasing priority to
6526facilitate revision numbering (i.e.
6527:term:`PE`,
6528:term:`PV`, and
6529:term:`PR` for epoch, version, and
6530revision, respectively). The values are highly dependent on the policies
6531and procedures of a given distribution and package feed.
6532
6533Because the OpenEmbedded build system uses
6534":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
6535unique to a given build, the build system knows when to rebuild
6536packages. All the inputs into a given task are represented by a
6537signature, which can trigger a rebuild when different. Thus, the build
6538system itself does not rely on the ``PR``, ``PV``, and ``PE`` numbers to
6539trigger a rebuild. The signatures, however, can be used to generate
6540these values.
6541
6542The PR Service works with both ``OEBasic`` and ``OEBasicHash``
6543generators. The value of ``PR`` bumps when the checksum changes and the
6544different generator mechanisms change signatures under different
6545circumstances.
6546
6547As implemented, the build system includes values from the PR Service
6548into the ``PR`` field as an addition using the form "``.x``" so ``r0``
6549becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
6550``PR`` values to be used for whatever reasons, which include manual
6551``PR`` bumps, should it be necessary.
6552
6553By default, the PR Service is not enabled or running. Thus, the packages
6554generated are just "self consistent". The build system adds and removes
6555packages and there are no guarantees about upgrade paths but images will
6556be consistent and correct with the latest changes.
6557
6558The simplest form for a PR Service is for it to exist for a single host
6559development system that builds the package feed (building system). For
6560this scenario, you can enable a local PR Service by setting
6561:term:`PRSERV_HOST` in your
6562``local.conf`` file in the :term:`Build Directory`:
6563::
6564
6565 PRSERV_HOST = "localhost:0"
6566
6567Once the service is started, packages will automatically
6568get increasing ``PR`` values and BitBake takes care of starting and
6569stopping the server.
6570
6571If you have a more complex setup where multiple host development systems
6572work against a common, shared package feed, you have a single PR Service
6573running and it is connected to each building system. For this scenario,
6574you need to start the PR Service using the ``bitbake-prserv`` command:
6575::
6576
6577 bitbake-prserv --host ip --port port --start
6578
6579In addition to
6580hand-starting the service, you need to update the ``local.conf`` file of
6581each building system as described earlier so each system points to the
6582server and port.
6583
6584It is also recommended you use build history, which adds some sanity
6585checks to binary package versions, in conjunction with the server that
6586is running the PR Service. To enable build history, add the following to
6587each building system's ``local.conf`` file:
6588::
6589
6590 # It is recommended to activate "buildhistory" for testing the PR service
6591 INHERIT += "buildhistory"
6592 BUILDHISTORY_COMMIT = "1"
6593
6594For information on build
6595history, see the "`Maintaining Build Output
6596Quality <#maintaining-build-output-quality>`__" section.
6597
6598.. note::
6599
6600 The OpenEmbedded build system does not maintain ``PR`` information as
6601 part of the shared state (sstate) packages. If you maintain an sstate
6602 feed, its expected that either all your building systems that
6603 contribute to the sstate feed use a shared PR Service, or you do not
6604 run a PR Service on any of your building systems. Having some systems
6605 use a PR Service while others do not leads to obvious problems.
6606
6607 For more information on shared state, see the
6608 ":ref:`overview-manual/concepts:shared state cache`"
6609 section in the Yocto Project Overview and Concepts Manual.
6610
6611Manually Bumping PR
6612~~~~~~~~~~~~~~~~~~~
6613
6614The alternative to setting up a PR Service is to manually "bump" the
6615:term:`PR` variable.
6616
6617If a committed change results in changing the package output, then the
6618value of the PR variable needs to be increased (or "bumped") as part of
6619that commit. For new recipes you should add the ``PR`` variable and set
6620its initial value equal to "r0", which is the default. Even though the
6621default value is "r0", the practice of adding it to a new recipe makes
6622it harder to forget to bump the variable when you make changes to the
6623recipe in future.
6624
6625If you are sharing a common ``.inc`` file with multiple recipes, you can
6626also use the ``INC_PR`` variable to ensure that the recipes sharing the
6627``.inc`` file are rebuilt when the ``.inc`` file itself is changed. The
6628``.inc`` file must set ``INC_PR`` (initially to "r0"), and all recipes
6629referring to it should set ``PR`` to "${INC_PR}.0" initially,
6630incrementing the last number when the recipe is changed. If the ``.inc``
6631file is changed then its ``INC_PR`` should be incremented.
6632
6633When upgrading the version of a binary package, assuming the ``PV``
6634changes, the ``PR`` variable should be reset to "r0" (or "${INC_PR}.0"
6635if you are using ``INC_PR``).
6636
6637Usually, version increases occur only to binary packages. However, if
6638for some reason ``PV`` changes but does not increase, you can increase
6639the ``PE`` variable (Package Epoch). The ``PE`` variable defaults to
6640"0".
6641
6642Binary package version numbering strives to follow the `Debian Version
6643Field Policy
6644Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
6645These guidelines define how versions are compared and what "increasing"
6646a version means.
6647
6648Automatically Incrementing a Package Version Number
6649~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6650
6651When fetching a repository, BitBake uses the
6652:term:`SRCREV` variable to determine
6653the specific source code revision from which to build. You set the
6654``SRCREV`` variable to
6655:term:`AUTOREV` to cause the
6656OpenEmbedded build system to automatically use the latest revision of
6657the software:
6658::
6659
6660 SRCREV = "${AUTOREV}"
6661
6662Furthermore, you need to reference ``SRCPV`` in ``PV`` in order to
6663automatically update the version whenever the revision of the source
6664code changes. Here is an example:
6665::
6666
6667 PV = "1.0+git${SRCPV}"
6668
6669The OpenEmbedded build system substitutes ``SRCPV`` with the following:
6670
6671.. code-block:: none
6672
6673 AUTOINC+source_code_revision
6674
6675The build system replaces the ``AUTOINC``
6676with a number. The number used depends on the state of the PR Service:
6677
6678- If PR Service is enabled, the build system increments the number,
6679 which is similar to the behavior of
6680 :term:`PR`. This behavior results in
6681 linearly increasing package versions, which is desirable. Here is an
6682 example:
6683
6684 .. code-block:: none
6685
6686 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6687 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
6688
6689- If PR Service is not enabled, the build system replaces the
6690 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
6691 changing the package version since the source revision is included.
6692 However, package versions are not increased linearly. Here is an
6693 example:
6694
6695 .. code-block:: none
6696
6697 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
6698 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
6699
6700In summary, the OpenEmbedded build system does not track the history of
6701binary package versions for this purpose. ``AUTOINC``, in this case, is
6702comparable to ``PR``. If PR server is not enabled, ``AUTOINC`` in the
6703package version is simply replaced by "0". If PR server is enabled, the
6704build system keeps track of the package versions and bumps the number
6705when the package revision changes.
6706
6707Handling Optional Module Packaging
6708----------------------------------
6709
6710Many pieces of software split functionality into optional modules (or
6711plugins) and the plugins that are built might depend on configuration
6712options. To avoid having to duplicate the logic that determines what
6713modules are available in your recipe or to avoid having to package each
6714module by hand, the OpenEmbedded build system provides functionality to
6715handle module packaging dynamically.
6716
6717To handle optional module packaging, you need to do two things:
6718
6719- Ensure the module packaging is actually done.
6720
6721- Ensure that any dependencies on optional modules from other recipes
6722 are satisfied by your recipe.
6723
6724Making Sure the Packaging is Done
6725~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6726
6727To ensure the module packaging actually gets done, you use the
6728``do_split_packages`` function within the ``populate_packages`` Python
6729function in your recipe. The ``do_split_packages`` function searches for
6730a pattern of files or directories under a specified path and creates a
6731package for each one it finds by appending to the
6732:term:`PACKAGES` variable and
6733setting the appropriate values for ``FILES_packagename``,
6734``RDEPENDS_packagename``, ``DESCRIPTION_packagename``, and so forth.
6735Here is an example from the ``lighttpd`` recipe:
6736::
6737
6738 python populate_packages_prepend () {
6739 lighttpd_libdir = d.expand('${libdir}')
6740 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
6741 'lighttpd-module-%s', 'Lighttpd module for %s',
6742 extra_depends='')
6743 }
6744
6745The previous example specifies a number of things in the call to
6746``do_split_packages``.
6747
6748- A directory within the files installed by your recipe through
6749 ``do_install`` in which to search.
6750
6751- A regular expression used to match module files in that directory. In
6752 the example, note the parentheses () that mark the part of the
6753 expression from which the module name should be derived.
6754
6755- A pattern to use for the package names.
6756
6757- A description for each package.
6758
6759- An empty string for ``extra_depends``, which disables the default
6760 dependency on the main ``lighttpd`` package. Thus, if a file in
6761 ``${libdir}`` called ``mod_alias.so`` is found, a package called
6762 ``lighttpd-module-alias`` is created for it and the
6763 :term:`DESCRIPTION` is set to
6764 "Lighttpd module for alias".
6765
6766Often, packaging modules is as simple as the previous example. However,
6767more advanced options exist that you can use within
6768``do_split_packages`` to modify its behavior. And, if you need to, you
6769can add more logic by specifying a hook function that is called for each
6770package. It is also perfectly acceptable to call ``do_split_packages``
6771multiple times if you have more than one set of modules to package.
6772
6773For more examples that show how to use ``do_split_packages``, see the
6774``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
6775directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
6776also find examples in ``meta/classes/kernel.bbclass``.
6777
6778Following is a reference that shows ``do_split_packages`` mandatory and
6779optional arguments:
6780::
6781
6782 Mandatory arguments
6783
6784 root
6785 The path in which to search
6786 file_regex
6787 Regular expression to match searched files.
6788 Use parentheses () to mark the part of this
6789 expression that should be used to derive the
6790 module name (to be substituted where %s is
6791 used in other function arguments as noted below)
6792 output_pattern
6793 Pattern to use for the package names. Must
6794 include %s.
6795 description
6796 Description to set for each package. Must
6797 include %s.
6798
6799 Optional arguments
6800
6801 postinst
6802 Postinstall script to use for all packages
6803 (as a string)
6804 recursive
6805 True to perform a recursive search - default
6806 False
6807 hook
6808 A hook function to be called for every match.
6809 The function will be called with the following
6810 arguments (in the order listed):
6811
6812 f
6813 Full path to the file/directory match
6814 pkg
6815 The package name
6816 file_regex
6817 As above
6818 output_pattern
6819 As above
6820 modulename
6821 The module name derived using file_regex
6822 extra_depends
6823 Extra runtime dependencies (RDEPENDS) to be
6824 set for all packages. The default value of None
6825 causes a dependency on the main package
6826 (${PN}) - if you do not want this, pass empty
6827 string '' for this parameter.
6828 aux_files_pattern
6829 Extra item(s) to be added to FILES for each
6830 package. Can be a single string item or a list
6831 of strings for multiple items. Must include %s.
6832 postrm
6833 postrm script to use for all packages (as a
6834 string)
6835 allow_dirs
6836 True to allow directories to be matched -
6837 default False
6838 prepend
6839 If True, prepend created packages to PACKAGES
6840 instead of the default False which appends them
6841 match_path
6842 match file_regex on the whole relative path to
6843 the root rather than just the file name
6844 aux_files_pattern_verbatim
6845 Extra item(s) to be added to FILES for each
6846 package, using the actual derived module name
6847 rather than converting it to something legal
6848 for a package name. Can be a single string item
6849 or a list of strings for multiple items. Must
6850 include %s.
6851 allow_links
6852 True to allow symlinks to be matched - default
6853 False
6854 summary
6855 Summary to set for each package. Must include %s;
6856 defaults to description if not set.
6857
6858
6859
6860Satisfying Dependencies
6861~~~~~~~~~~~~~~~~~~~~~~~
6862
6863The second part for handling optional module packaging is to ensure that
6864any dependencies on optional modules from other recipes are satisfied by
6865your recipe. You can be sure these dependencies are satisfied by using
6866the :term:`PACKAGES_DYNAMIC`
6867variable. Here is an example that continues with the ``lighttpd`` recipe
6868shown earlier:
6869::
6870
6871 PACKAGES_DYNAMIC = "lighttpd-module-.*"
6872
6873The name
6874specified in the regular expression can of course be anything. In this
6875example, it is ``lighttpd-module-`` and is specified as the prefix to
6876ensure that any :term:`RDEPENDS` and
6877:term:`RRECOMMENDS` on a package
6878name starting with the prefix are satisfied during build time. If you
6879are using ``do_split_packages`` as described in the previous section,
6880the value you put in ``PACKAGES_DYNAMIC`` should correspond to the name
6881pattern specified in the call to ``do_split_packages``.
6882
6883Using Runtime Package Management
6884--------------------------------
6885
6886During a build, BitBake always transforms a recipe into one or more
6887packages. For example, BitBake takes the ``bash`` recipe and produces a
6888number of packages (e.g. ``bash``, ``bash-bashbug``,
6889``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
6890``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
6891generated packages are included in an image.
6892
6893In several situations, you might need to update, add, remove, or query
6894the packages on a target device at runtime (i.e. without having to
6895generate a new image). Examples of such situations include:
6896
6897- You want to provide in-the-field updates to deployed devices (e.g.
6898 security updates).
6899
6900- You want to have a fast turn-around development cycle for one or more
6901 applications that run on your device.
6902
6903- You want to temporarily install the "debug" packages of various
6904 applications on your device so that debugging can be greatly improved
6905 by allowing access to symbols and source debugging.
6906
6907- You want to deploy a more minimal package selection of your device
6908 but allow in-the-field updates to add a larger selection for
6909 customization.
6910
6911In all these situations, you have something similar to a more
6912traditional Linux distribution in that in-field devices are able to
6913receive pre-compiled packages from a server for installation or update.
6914Being able to install these packages on a running, in-field device is
6915what is termed "runtime package management".
6916
6917In order to use runtime package management, you need a host or server
6918machine that serves up the pre-compiled packages plus the required
6919metadata. You also need package manipulation tools on the target. The
6920build machine is a likely candidate to act as the server. However, that
6921machine does not necessarily have to be the package server. The build
6922machine could push its artifacts to another machine that acts as the
6923server (e.g. Internet-facing). In fact, doing so is advantageous for a
6924production environment as getting the packages away from the development
6925system's build directory prevents accidental overwrites.
6926
6927A simple build that targets just one device produces more than one
6928package database. In other words, the packages produced by a build are
6929separated out into a couple of different package groupings based on
6930criteria such as the target's CPU architecture, the target board, or the
6931C library used on the target. For example, a build targeting the
6932``qemux86`` device produces the following three package databases:
6933``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
6934device to be aware of all the packages that were available to it, you
6935would need to point it to each of these databases individually. In a
6936similar way, a traditional Linux distribution usually is configured to
6937be aware of a number of software repositories from which it retrieves
6938packages.
6939
6940Using runtime package management is completely optional and not required
6941for a successful build or deployment in any way. But if you want to make
6942use of runtime package management, you need to do a couple things above
6943and beyond the basics. The remainder of this section describes what you
6944need to do.
6945
6946Build Considerations
6947~~~~~~~~~~~~~~~~~~~~
6948
6949This section describes build considerations of which you need to be
6950aware in order to provide support for runtime package management.
6951
6952When BitBake generates packages, it needs to know what format or formats
6953to use. In your configuration, you use the
6954:term:`PACKAGE_CLASSES`
6955variable to specify the format:
6956
69571. Open the ``local.conf`` file inside your
6958 :term:`Build Directory` (e.g.
6959 ``~/poky/build/conf/local.conf``).
6960
69612. Select the desired package format as follows:
6962 ::
6963
6964 PACKAGE_CLASSES ?= "package_packageformat"
6965
6966 where packageformat can be "ipk", "rpm",
6967 "deb", or "tar" which are the supported package formats.
6968
6969 .. note::
6970
6971 Because the Yocto Project supports four different package formats,
6972 you can set the variable with more than one argument. However, the
6973 OpenEmbedded build system only uses the first argument when
6974 creating an image or Software Development Kit (SDK).
6975
6976If you would like your image to start off with a basic package database
6977containing the packages in your current build as well as to have the
6978relevant tools available on the target for runtime package management,
6979you can include "package-management" in the
6980:term:`IMAGE_FEATURES`
6981variable. Including "package-management" in this configuration variable
6982ensures that when the image is assembled for your target, the image
6983includes the currently-known package databases as well as the
6984target-specific tools required for runtime package management to be
6985performed on the target. However, this is not strictly necessary. You
6986could start your image off without any databases but only include the
6987required on-target package tool(s). As an example, you could include
6988"opkg" in your
6989:term:`IMAGE_INSTALL` variable
6990if you are using the IPK package format. You can then initialize your
6991target's package database(s) later once your image is up and running.
6992
6993Whenever you perform any sort of build step that can potentially
6994generate a package or modify existing package, it is always a good idea
6995to re-generate the package index after the build by using the following
6996command:
6997::
6998
6999 $ bitbake package-index
7000
7001It might be tempting to build the
7002package and the package index at the same time with a command such as
7003the following:
7004::
7005
7006 $ bitbake some-package package-index
7007
7008Do not do this as
7009BitBake does not schedule the package index for after the completion of
7010the package you are building. Consequently, you cannot be sure of the
7011package index including information for the package you just built.
7012Thus, be sure to run the package update step separately after building
7013any packages.
7014
7015You can use the
7016:term:`PACKAGE_FEED_ARCHS`,
7017:term:`PACKAGE_FEED_BASE_PATHS`,
7018and
7019:term:`PACKAGE_FEED_URIS`
7020variables to pre-configure target images to use a package feed. If you
7021do not define these variables, then manual steps as described in the
7022subsequent sections are necessary to configure the target. You should
7023set these variables before building the image in order to produce a
7024correctly configured image.
7025
7026When your build is complete, your packages reside in the
7027``${TMPDIR}/deploy/packageformat`` directory. For example, if
7028``${``\ :term:`TMPDIR`\ ``}`` is
7029``tmp`` and your selected package type is RPM, then your RPM packages
7030are available in ``tmp/deploy/rpm``.
7031
7032Host or Server Machine Setup
7033~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7034
7035Although other protocols are possible, a server using HTTP typically
7036serves packages. If you want to use HTTP, then set up and configure a
7037web server such as Apache 2, lighttpd, or Python web server on the
7038machine serving the packages.
7039
7040To keep things simple, this section describes how to set up a
7041Python web server to share package feeds from the developer's
7042machine. Although this server might not be the best for a production
7043environment, the setup is simple and straight forward. Should you want
7044to use a different server more suited for production (e.g. Apache 2,
7045Lighttpd, or Nginx), take the appropriate steps to do so.
7046
7047From within the build directory where you have built an image based on
7048your packaging choice (i.e. the
7049:term:`PACKAGE_CLASSES`
7050setting), simply start the server. The following example assumes a build
7051directory of ``~/poky/build/tmp/deploy/rpm`` and a ``PACKAGE_CLASSES``
7052setting of "package_rpm":
7053::
7054
7055 $ cd ~/poky/build/tmp/deploy/rpm
7056 $ python3 -m http.server
7057
7058Target Setup
7059~~~~~~~~~~~~
7060
7061Setting up the target differs depending on the package management
7062system. This section provides information for RPM, IPK, and DEB.
7063
7064Using RPM
7065^^^^^^^^^
7066
7067The `Dandified Packaging
7068Tool <https://en.wikipedia.org/wiki/DNF_(software)>`__ (DNF) performs
7069runtime package management of RPM packages. In order to use DNF for
7070runtime package management, you must perform an initial setup on the
7071target machine for cases where the ``PACKAGE_FEED_*`` variables were not
7072set as part of the image that is running on the target. This means if
7073you built your image and did not not use these variables as part of the
7074build and your image is now running on the target, you need to perform
7075the steps in this section if you want to use runtime package management.
7076
7077.. note::
7078
7079 For information on the ``PACKAGE_FEED_*`` variables, see
7080 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
7081 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
7082 glossary.
7083
7084On the target, you must inform DNF that package databases are available.
7085You do this by creating a file named
7086``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
7087
7088As an example, assume the target is able to use the following package
7089databases: ``all``, ``i586``, and ``qemux86`` from a server named
7090``my.server``. The specifics for setting up the web server are up to
7091you. The critical requirement is that the URIs in the target repository
7092configuration point to the correct remote location for the feeds.
7093
7094.. note::
7095
7096 For development purposes, you can point the web server to the build
7097 system's ``deploy`` directory. However, for production use, it is better to
7098 copy the package directories to a location outside of the build area and use
7099 that location. Doing so avoids situations where the build system
7100 overwrites or changes the ``deploy`` directory.
7101
7102When telling DNF where to look for the package databases, you must
7103declare individual locations per architecture or a single location used
7104for all architectures. You cannot do both:
7105
7106- *Create an Explicit List of Architectures:* Define individual base
7107 URLs to identify where each package database is located:
7108
7109 .. code-block:: none
7110
7111 [oe-packages]
7112 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
7113
7114 This example
7115 informs DNF about individual package databases for all three
7116 architectures.
7117
7118- *Create a Single (Full) Package Index:* Define a single base URL that
7119 identifies where a full package database is located:
7120 ::
7121
7122 [oe-packages]
7123 baseurl=http://my.server/rpm
7124
7125 This example informs DNF about a single
7126 package database that contains all the package index information for
7127 all supported architectures.
7128
7129Once you have informed DNF where to find the package databases, you need
7130to fetch them:
7131
7132.. code-block:: none
7133
7134 # dnf makecache
7135
7136DNF is now able to find, install, and
7137upgrade packages from the specified repository or repositories.
7138
7139.. note::
7140
7141 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
7142 additional information.
7143
7144Using IPK
7145^^^^^^^^^
7146
7147The ``opkg`` application performs runtime package management of IPK
7148packages. You must perform an initial setup for ``opkg`` on the target
7149machine if the
7150:term:`PACKAGE_FEED_ARCHS`,
7151:term:`PACKAGE_FEED_BASE_PATHS`,
7152and
7153:term:`PACKAGE_FEED_URIS`
7154variables have not been set or the target image was built before the
7155variables were set.
7156
7157The ``opkg`` application uses configuration files to find available
7158package databases. Thus, you need to create a configuration file inside
7159the ``/etc/opkg/`` direction, which informs ``opkg`` of any repository
7160you want to use.
7161
7162As an example, suppose you are serving packages from a ``ipk/``
7163directory containing the ``i586``, ``all``, and ``qemux86`` databases
7164through an HTTP server named ``my.server``. On the target, create a
7165configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
7166directory containing the following:
7167
7168.. code-block:: none
7169
7170 src/gz all http://my.server/ipk/all
7171 src/gz i586 http://my.server/ipk/i586
7172 src/gz qemux86 http://my.server/ipk/qemux86
7173
7174Next, instruct ``opkg`` to fetch the
7175repository information:
7176
7177.. code-block:: none
7178
7179 # opkg update
7180
7181The ``opkg`` application is now able to find, install, and upgrade packages
7182from the specified repository.
7183
7184Using DEB
7185^^^^^^^^^
7186
7187The ``apt`` application performs runtime package management of DEB
7188packages. This application uses a source list file to find available
7189package databases. You must perform an initial setup for ``apt`` on the
7190target machine if the
7191:term:`PACKAGE_FEED_ARCHS`,
7192:term:`PACKAGE_FEED_BASE_PATHS`,
7193and
7194:term:`PACKAGE_FEED_URIS`
7195variables have not been set or the target image was built before the
7196variables were set.
7197
7198To inform ``apt`` of the repository you want to use, you might create a
7199list file (e.g. ``my_repo.list``) inside the
7200``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
7201serving packages from a ``deb/`` directory containing the ``i586``,
7202``all``, and ``qemux86`` databases through an HTTP server named
7203``my.server``. The list file should contain:
7204
7205.. code-block:: none
7206
7207 deb http://my.server/deb/all ./
7208 deb http://my.server/deb/i586 ./
7209 deb http://my.server/deb/qemux86 ./
7210
7211Next, instruct the ``apt`` application
7212to fetch the repository information:
7213
7214.. code-block:: none
7215
7216 # apt-get update
7217
7218After this step,
7219``apt`` is able to find, install, and upgrade packages from the
7220specified repository.
7221
7222Generating and Using Signed Packages
7223------------------------------------
7224
7225In order to add security to RPM packages used during a build, you can
7226take steps to securely sign them. Once a signature is verified, the
7227OpenEmbedded build system can use the package in the build. If security
7228fails for a signed package, the build system aborts the build.
7229
7230This section describes how to sign RPM packages during a build and how
7231to use signed package feeds (repositories) when doing a build.
7232
7233Signing RPM Packages
7234~~~~~~~~~~~~~~~~~~~~
7235
7236To enable signing RPM packages, you must set up the following
7237configurations in either your ``local.config`` or ``distro.config``
7238file:
7239::
7240
7241 # Inherit sign_rpm.bbclass to enable signing functionality
7242 INHERIT += " sign_rpm"
7243 # Define the GPG key that will be used for signing.
7244 RPM_GPG_NAME = "key_name"
7245 # Provide passphrase for the key
7246 RPM_GPG_PASSPHRASE = "passphrase"
7247
7248.. note::
7249
7250 Be sure to supply appropriate values for both `key_name` and
7251 `passphrase`.
7252
7253Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
7254the previous example, two optional variables related to signing exist:
7255
7256- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
7257 when the package is signed.
7258
7259- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7260 package is signed.
7261
7262Processing Package Feeds
7263~~~~~~~~~~~~~~~~~~~~~~~~
7264
7265In addition to being able to sign RPM packages, you can also enable
7266signed package feeds for IPK and RPM packages.
7267
7268The steps you need to take to enable signed package feed use are similar
7269to the steps used to sign RPM packages. You must define the following in
7270your ``local.config`` or ``distro.config`` file:
7271::
7272
7273 INHERIT += "sign_package_feed"
7274 PACKAGE_FEED_GPG_NAME = "key_name"
7275 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
7276
7277For signed package feeds, the passphrase must exist in a separate file,
7278which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
7279variable. Regarding security, keeping a plain text passphrase out of the
7280configuration is more secure.
7281
7282Aside from the ``PACKAGE_FEED_GPG_NAME`` and
7283``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
7284related to signed package feeds exist:
7285
7286- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
7287 when the package is signed.
7288
7289- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
7290 package is signed.
7291
7292- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
7293 signature. This variable applies only to RPM and IPK package feeds.
7294 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
7295 "ASC", which is the default and specifies ascii armored, and "BIN",
7296 which specifies binary.
7297
7298Testing Packages With ptest
7299---------------------------
7300
7301A Package Test (ptest) runs tests against packages built by the
7302OpenEmbedded build system on the target machine. A ptest contains at
7303least two items: the actual test, and a shell script (``run-ptest``)
7304that starts the test. The shell script that starts the test must not
7305contain the actual test - the script only starts the test. On the other
7306hand, the test can be anything from a simple shell script that runs a
7307binary and checks the output to an elaborate system of test binaries and
7308data files.
7309
7310The test generates output in the format used by Automake:
7311::
7312
7313 result: testname
7314
7315where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
7316the testname can be any identifying string.
7317
7318For a list of Yocto Project recipes that are already enabled with ptest,
7319see the :yocto_wiki:`Ptest </Ptest>` wiki page.
7320
7321.. note::
7322
7323 A recipe is "ptest-enabled" if it inherits the
7324 :ref:`ptest <ref-classes-ptest>` class.
7325
7326Adding ptest to Your Build
7327~~~~~~~~~~~~~~~~~~~~~~~~~~
7328
7329To add package testing to your build, add the
7330:term:`DISTRO_FEATURES` and
7331:term:`EXTRA_IMAGE_FEATURES`
7332variables to your ``local.conf`` file, which is found in the
7333:term:`Build Directory`:
7334::
7335
7336 DISTRO_FEATURES_append = " ptest"
7337 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
7338
7339Once your build is complete, the ptest files are installed into the
7340``/usr/lib/package/ptest`` directory within the image, where ``package``
7341is the name of the package.
7342
7343Running ptest
7344~~~~~~~~~~~~~
7345
7346The ``ptest-runner`` package installs a shell script that loops through
7347all installed ptest test suites and runs them in sequence. Consequently,
7348you might want to add this package to your image.
7349
7350Getting Your Package Ready
7351~~~~~~~~~~~~~~~~~~~~~~~~~~
7352
7353In order to enable a recipe to run installed ptests on target hardware,
7354you need to prepare the recipes that build the packages you want to
7355test. Here is what you have to do for each recipe:
7356
7357- *Be sure the recipe inherits
7358 the* :ref:`ptest <ref-classes-ptest>` *class:*
7359 Include the following line in each recipe:
7360 ::
7361
7362 inherit ptest
7363
7364- *Create run-ptest:* This script starts your test. Locate the
7365 script where you will refer to it using
7366 :term:`SRC_URI`. Here is an
7367 example that starts a test for ``dbus``:
7368 ::
7369
7370 #!/bin/sh
7371 cd test
7372 make -k runtest-TESTS
7373
7374- *Ensure dependencies are met:* If the test adds build or runtime
7375 dependencies that normally do not exist for the package (such as
7376 requiring "make" to run the test suite), use the
7377 :term:`DEPENDS` and
7378 :term:`RDEPENDS` variables in
7379 your recipe in order for the package to meet the dependencies. Here
7380 is an example where the package has a runtime dependency on "make":
7381 ::
7382
7383 RDEPENDS_${PN}-ptest += "make"
7384
7385- *Add a function to build the test suite:* Not many packages support
7386 cross-compilation of their test suites. Consequently, you usually
7387 need to add a cross-compilation function to the package.
7388
7389 Many packages based on Automake compile and run the test suite by
7390 using a single command such as ``make check``. However, the host
7391 ``make check`` builds and runs on the same computer, while
7392 cross-compiling requires that the package is built on the host but
7393 executed for the target architecture (though often, as in the case
7394 for ptest, the execution occurs on the host). The built version of
7395 Automake that ships with the Yocto Project includes a patch that
7396 separates building and execution. Consequently, packages that use the
7397 unaltered, patched version of ``make check`` automatically
7398 cross-compiles.
7399
7400 Regardless, you still must add a ``do_compile_ptest`` function to
7401 build the test suite. Add a function similar to the following to your
7402 recipe:
7403 ::
7404
7405 do_compile_ptest() {
7406 oe_runmake buildtest-TESTS
7407 }
7408
7409- *Ensure special configurations are set:* If the package requires
7410 special configurations prior to compiling the test code, you must
7411 insert a ``do_configure_ptest`` function into the recipe.
7412
7413- *Install the test suite:* The ``ptest`` class automatically copies
7414 the file ``run-ptest`` to the target and then runs make
7415 ``install-ptest`` to run the tests. If this is not enough, you need
7416 to create a ``do_install_ptest`` function and make sure it gets
7417 called after the "make install-ptest" completes.
7418
7419Creating Node Package Manager (NPM) Packages
7420--------------------------------------------
7421
7422`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ is a package
7423manager for the JavaScript programming language. The Yocto Project
7424supports the NPM :ref:`fetcher <bitbake:bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`. You can
7425use this fetcher in combination with
7426:doc:`devtool </ref-manual/devtool-reference>` to create
7427recipes that produce NPM packages.
7428
7429Two workflows exist that allow you to create NPM packages using
7430``devtool``: the NPM registry modules method and the NPM project code
7431method.
7432
7433.. note::
7434
7435 While it is possible to create NPM recipes manually, using
7436 ``devtool`` is far simpler.
7437
7438Additionally, some requirements and caveats exist.
7439
7440Requirements and Caveats
7441~~~~~~~~~~~~~~~~~~~~~~~~
7442
7443You need to be aware of the following before using ``devtool`` to create
7444NPM packages:
7445
7446- Of the two methods that you can use ``devtool`` to create NPM
7447 packages, the registry approach is slightly simpler. However, you
7448 might consider the project approach because you do not have to
7449 publish your module in the NPM registry
7450 (`npm-registry <https://docs.npmjs.com/misc/registry>`_), which
7451 is NPM's public registry.
7452
7453- Be familiar with
7454 :doc:`devtool </ref-manual/devtool-reference>`.
7455
7456- The NPM host tools need the native ``nodejs-npm`` package, which is
7457 part of the OpenEmbedded environment. You need to get the package by
7458 cloning the https://github.com/openembedded/meta-openembedded
7459 repository out of GitHub. Be sure to add the path to your local copy
7460 to your ``bblayers.conf`` file.
7461
7462- ``devtool`` cannot detect native libraries in module dependencies.
7463 Consequently, you must manually add packages to your recipe.
7464
7465- While deploying NPM packages, ``devtool`` cannot determine which
7466 dependent packages are missing on the target (e.g. the node runtime
7467 ``nodejs``). Consequently, you need to find out what files are
7468 missing and be sure they are on the target.
7469
7470- Although you might not need NPM to run your node package, it is
7471 useful to have NPM on your target. The NPM package name is
7472 ``nodejs-npm``.
7473
7474Using the Registry Modules Method
7475~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7476
7477This section presents an example that uses the ``cute-files`` module,
7478which is a file browser web application.
7479
7480.. note::
7481
7482 You must know the ``cute-files`` module version.
7483
7484The first thing you need to do is use ``devtool`` and the NPM fetcher to
7485create the recipe:
7486::
7487
7488 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
7489
7490The
7491``devtool add`` command runs ``recipetool create`` and uses the same
7492fetch URI to download each dependency and capture license details where
7493possible. The result is a generated recipe.
7494
7495The recipe file is fairly simple and contains every license that
7496``recipetool`` finds and includes the licenses in the recipe's
7497:term:`LIC_FILES_CHKSUM`
7498variables. You need to examine the variables and look for those with
7499"unknown" in the :term:`LICENSE`
7500field. You need to track down the license information for "unknown"
7501modules and manually add the information to the recipe.
7502
7503``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
7504files capture the version of all dependent modules. Many packages do not
7505provide shrinkwrap files. ``recipetool`` create a shrinkwrap file as it
7506runs.
7507
7508.. note::
7509
7510 A package is created for each sub-module. This policy is the only
7511 practical way to have the licenses for all of the dependencies
7512 represented in the license manifest of the image.
7513
7514The ``devtool edit-recipe`` command lets you take a look at the recipe:
7515::
7516
7517 $ devtool edit-recipe cute-files
7518 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
7519 LICENSE = "MIT & ISC & Unknown"
7520 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
7521 file://node_modules/toidentifier/LICENSE;md5=1a261071a044d02eb6f2bb47f51a3502 \
7522 file://node_modules/debug/LICENSE;md5=ddd815a475e7338b0be7a14d8ee35a99 \
7523 ...
7524 SRC_URI = " \
7525 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
7526 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7527 "
7528 S = "${WORKDIR}/npm"
7529 inherit npm LICENSE_${PN} = "MIT"
7530 LICENSE_${PN}-accepts = "MIT"
7531 LICENSE_${PN}-array-flatten = "MIT"
7532 ...
7533 LICENSE_${PN}-vary = "MIT"
7534
7535Three key points exist in the previous example:
7536
7537- :term:`SRC_URI` uses the NPM
7538 scheme so that the NPM fetcher is used.
7539
7540- ``recipetool`` collects all the license information. If a
7541 sub-module's license is unavailable, the sub-module's name appears in
7542 the comments.
7543
7544- The ``inherit npm`` statement causes the
7545 :ref:`npm <ref-classes-npm>` class to package
7546 up all the modules.
7547
7548You can run the following command to build the ``cute-files`` package:
7549::
7550
7551 $ devtool build cute-files
7552
7553Remember that ``nodejs`` must be installed on
7554the target before your package.
7555
7556Assuming 192.168.7.2 for the target's IP address, use the following
7557command to deploy your package:
7558::
7559
7560 $ devtool deploy-target -s cute-files root@192.168.7.2
7561
7562Once the package is installed on the target, you can
7563test the application:
7564
7565.. note::
7566
7567 Because of a known issue, you cannot simply run ``cute-files`` as you would
7568 if you had run ``npm install``.
7569
7570::
7571
7572 $ cd /usr/lib/node_modules/cute-files
7573 $ node cute-files.js
7574
7575On a browser,
7576go to ``http://192.168.7.2:3000`` and you see the following:
7577
7578.. image:: figures/cute-files-npm-example.png
7579 :align: center
7580
7581You can find the recipe in ``workspace/recipes/cute-files``. You can use
7582the recipe in any layer you choose.
7583
7584Using the NPM Projects Code Method
7585~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7586
7587Although it is useful to package modules already in the NPM registry,
7588adding ``node.js`` projects under development is a more common developer
7589use case.
7590
7591This section covers the NPM projects code method, which is very similar
7592to the "registry" approach described in the previous section. In the NPM
7593projects method, you provide ``devtool`` with an URL that points to the
7594source files.
7595
7596Replicating the same example, (i.e. ``cute-files``) use the following
7597command:
7598::
7599
7600 $ devtool add https://github.com/martinaglv/cute-files.git
7601
7602The
7603recipe this command generates is very similar to the recipe created in
7604the previous section. However, the ``SRC_URI`` looks like the following:
7605::
7606
7607 SRC_URI = " \
7608 git://github.com/martinaglv/cute-files.git;protocol=https \
7609 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
7610 "
7611
7612In this example,
7613the main module is taken from the Git repository and dependencies are
7614taken from the NPM registry. Other than those differences, the recipe is
7615basically the same between the two methods. You can build and deploy the
7616package exactly as described in the previous section that uses the
7617registry modules method.
7618
7619Adding custom metadata to packages
7620----------------------------------
7621
7622The variable
7623:term:`PACKAGE_ADD_METADATA`
7624can be used to add additional metadata to packages. This is reflected in
7625the package control/spec file. To take the ipk format for example, the
7626CONTROL file stored inside would contain the additional metadata as
7627additional lines.
7628
7629The variable can be used in multiple ways, including using suffixes to
7630set it for a specific package type and/or package. Note that the order
7631of precedence is the same as this list:
7632
7633- ``PACKAGE_ADD_METADATA_<PKGTYPE>_<PN>``
7634
7635- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
7636
7637- ``PACKAGE_ADD_METADATA_<PN>``
7638
7639- ``PACKAGE_ADD_METADATA``
7640
7641`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
7642package type:
7643
7644- IPK for .ipk packages
7645
7646- DEB for .deb packages
7647
7648- RPM for .rpm packages
7649
7650`<PN>` is a parameter and expected to be a package name.
7651
7652The variable can contain multiple [one-line] metadata fields separated
7653by the literal sequence '\\n'. The separator can be redefined using the
7654variable flag ``separator``.
7655
7656The following is an example that adds two custom fields for ipk
7657packages:
7658::
7659
7660 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
7661
7662Efficiently Fetching Source Files During a Build
7663================================================
7664
7665The OpenEmbedded build system works with source files located through
7666the :term:`SRC_URI` variable. When
7667you build something using BitBake, a big part of the operation is
7668locating and downloading all the source tarballs. For images,
7669downloading all the source for various packages can take a significant
7670amount of time.
7671
7672This section shows you how you can use mirrors to speed up fetching
7673source files and how you can pre-fetch files all of which leads to more
7674efficient use of resources and time.
7675
7676Setting up Effective Mirrors
7677----------------------------
7678
7679A good deal that goes into a Yocto Project build is simply downloading
7680all of the source tarballs. Maybe you have been working with another
7681build system (OpenEmbedded or Angstrom) for which you have built up a
7682sizable directory of source tarballs. Or, perhaps someone else has such
7683a directory for which you have read access. If so, you can save time by
7684adding statements to your configuration file so that the build process
7685checks local directories first for existing tarballs before checking the
7686Internet.
7687
7688Here is an efficient way to set it up in your ``local.conf`` file:
7689::
7690
7691 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
7692 INHERIT += "own-mirrors"
7693 BB_GENERATE_MIRROR_TARBALLS = "1"
7694 # BB_NO_NETWORK = "1"
7695
7696In the previous example, the
7697:term:`BB_GENERATE_MIRROR_TARBALLS`
7698variable causes the OpenEmbedded build system to generate tarballs of
7699the Git repositories and store them in the
7700:term:`DL_DIR` directory. Due to
7701performance reasons, generating and storing these tarballs is not the
7702build system's default behavior.
7703
7704You can also use the
7705:term:`PREMIRRORS` variable. For
7706an example, see the variable's glossary entry in the Yocto Project
7707Reference Manual.
7708
7709Getting Source Files and Suppressing the Build
7710----------------------------------------------
7711
7712Another technique you can use to ready yourself for a successive string
7713of build operations, is to pre-fetch all the source files without
7714actually starting a build. This technique lets you work through any
7715download issues and ultimately gathers all the source files into your
7716download directory :ref:`structure-build-downloads`,
7717which is located with :term:`DL_DIR`.
7718
7719Use the following BitBake command form to fetch all the necessary
7720sources without starting the build:
7721::
7722
7723 $ bitbake target --runall=fetch
7724
7725This
7726variation of the BitBake command guarantees that you have all the
7727sources for that BitBake target should you disconnect from the Internet
7728and want to do the build later offline.
7729
7730Selecting an Initialization Manager
7731===================================
7732
7733By default, the Yocto Project uses SysVinit as the initialization
7734manager. However, support also exists for systemd, which is a full
7735replacement for init with parallel starting of services, reduced shell
7736overhead and other features that are used by many distributions.
7737
7738Within the system, SysVinit treats system components as services. These
7739services are maintained as shell scripts stored in the ``/etc/init.d/``
7740directory. Services organize into different run levels. This
7741organization is maintained by putting links to the services in the
7742``/etc/rcN.d/`` directories, where `N/` is one of the following options:
7743"S", "0", "1", "2", "3", "4", "5", or "6".
7744
7745.. note::
7746
7747 Each runlevel has a dependency on the previous runlevel. This
7748 dependency allows the services to work properly.
7749
7750In comparison, systemd treats components as units. Using units is a
7751broader concept as compared to using a service. A unit includes several
7752different types of entities. Service is one of the types of entities.
7753The runlevel concept in SysVinit corresponds to the concept of a target
7754in systemd, where target is also a type of supported unit.
7755
7756In a SysVinit-based system, services load sequentially (i.e. one by one)
7757during init and parallelization is not supported. With systemd, services
7758start in parallel. Needless to say, the method can have an impact on
7759system startup performance.
7760
7761If you want to use SysVinit, you do not have to do anything. But, if you
7762want to use systemd, you must take some steps as described in the
7763following sections.
7764
7765Using systemd Exclusively
7766-------------------------
7767
7768Set these variables in your distribution configuration file as follows:
7769::
7770
7771 DISTRO_FEATURES_append = " systemd"
7772 VIRTUAL-RUNTIME_init_manager = "systemd"
7773
7774You can also prevent the SysVinit distribution feature from
7775being automatically enabled as follows:
7776::
7777
7778 DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
7779
7780Doing so removes any
7781redundant SysVinit scripts.
7782
7783To remove initscripts from your image altogether, set this variable
7784also:
7785::
7786
7787 VIRTUAL-RUNTIME_initscripts = ""
7788
7789For information on the backfill variable, see
7790:term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`.
7791
7792Using systemd for the Main Image and Using SysVinit for the Rescue Image
7793------------------------------------------------------------------------
7794
7795Set these variables in your distribution configuration file as follows:
7796::
7797
7798 DISTRO_FEATURES_append = " systemd"
7799 VIRTUAL-RUNTIME_init_manager = "systemd"
7800
7801Doing so causes your main image to use the
7802``packagegroup-core-boot.bb`` recipe and systemd. The rescue/minimal
7803image cannot use this package group. However, it can install SysVinit
7804and the appropriate packages will have support for both systemd and
7805SysVinit.
7806
7807Selecting a Device Manager
7808==========================
7809
7810The Yocto Project provides multiple ways to manage the device manager
7811(``/dev``):
7812
7813- Persistent and Pre-Populated\ ``/dev``: For this case, the ``/dev``
7814 directory is persistent and the required device nodes are created
7815 during the build.
7816
7817- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
7818 directory is provided by the kernel as an in-memory file system and
7819 is automatically populated by the kernel at runtime. Additional
7820 configuration of device nodes is done in user space by a device
7821 manager like ``udev`` or ``busybox-mdev``.
7822
7823Using Persistent and Pre-Populated\ ``/dev``
7824--------------------------------------------
7825
7826To use the static method for device population, you need to set the
7827:term:`USE_DEVFS` variable to "0"
7828as follows:
7829::
7830
7831 USE_DEVFS = "0"
7832
7833The content of the resulting ``/dev`` directory is defined in a Device
7834Table file. The
7835:term:`IMAGE_DEVICE_TABLES`
7836variable defines the Device Table to use and should be set in the
7837machine or distro configuration file. Alternatively, you can set this
7838variable in your ``local.conf`` configuration file.
7839
7840If you do not define the ``IMAGE_DEVICE_TABLES`` variable, the default
7841``device_table-minimal.txt`` is used:
7842::
7843
7844 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
7845
7846The population is handled by the ``makedevs`` utility during image
7847creation:
7848
7849Using ``devtmpfs`` and a Device Manager
7850---------------------------------------
7851
7852To use the dynamic method for device population, you need to use (or be
7853sure to set) the :term:`USE_DEVFS`
7854variable to "1", which is the default:
7855::
7856
7857 USE_DEVFS = "1"
7858
7859With this
7860setting, the resulting ``/dev`` directory is populated by the kernel
7861using ``devtmpfs``. Make sure the corresponding kernel configuration
7862variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
7863kernel.
7864
7865All devices created by ``devtmpfs`` will be owned by ``root`` and have
7866permissions ``0600``.
7867
7868To have more control over the device nodes, you can use a device manager
7869like ``udev`` or ``busybox-mdev``. You choose the device manager by
7870defining the ``VIRTUAL-RUNTIME_dev_manager`` variable in your machine or
7871distro configuration file. Alternatively, you can set this variable in
7872your ``local.conf`` configuration file:
7873::
7874
7875 VIRTUAL-RUNTIME_dev_manager = "udev"
7876
7877 # Some alternative values
7878 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
7879 # VIRTUAL-RUNTIME_dev_manager = "systemd"
7880
7881Using an External SCM
7882=====================
7883
7884If you're working on a recipe that pulls from an external Source Code
7885Manager (SCM), it is possible to have the OpenEmbedded build system
7886notice new recipe changes added to the SCM and then build the resulting
7887packages that depend on the new recipes by using the latest versions.
7888This only works for SCMs from which it is possible to get a sensible
7889revision number for changes. Currently, you can do this with Apache
7890Subversion (SVN), Git, and Bazaar (BZR) repositories.
7891
7892To enable this behavior, the :term:`PV` of
7893the recipe needs to reference
7894:term:`SRCPV`. Here is an example:
7895::
7896
7897 PV = "1.2.3+git${SRCPV}"
7898
7899Then, you can add the following to your
7900``local.conf``:
7901::
7902
7903 SRCREV_pn-PN = "${AUTOREV}"
7904
7905:term:`PN` is the name of the recipe for
7906which you want to enable automatic source revision updating.
7907
7908If you do not want to update your local configuration file, you can add
7909the following directly to the recipe to finish enabling the feature:
7910::
7911
7912 SRCREV = "${AUTOREV}"
7913
7914The Yocto Project provides a distribution named ``poky-bleeding``, whose
7915configuration file contains the line:
7916::
7917
7918 require conf/distro/include/poky-floating-revisions.inc
7919
7920This line pulls in the
7921listed include file that contains numerous lines of exactly that form:
7922::
7923
7924 #SRCREV_pn-opkg-native ?= "${AUTOREV}"
7925 #SRCREV_pn-opkg-sdk ?= "${AUTOREV}"
7926 #SRCREV_pn-opkg ?= "${AUTOREV}"
7927 #SRCREV_pn-opkg-utils-native ?= "${AUTOREV}"
7928 #SRCREV_pn-opkg-utils ?= "${AUTOREV}"
7929 SRCREV_pn-gconf-dbus ?= "${AUTOREV}"
7930 SRCREV_pn-matchbox-common ?= "${AUTOREV}"
7931 SRCREV_pn-matchbox-config-gtk ?= "${AUTOREV}"
7932 SRCREV_pn-matchbox-desktop ?= "${AUTOREV}"
7933 SRCREV_pn-matchbox-keyboard ?= "${AUTOREV}"
7934 SRCREV_pn-matchbox-panel-2 ?= "${AUTOREV}"
7935 SRCREV_pn-matchbox-themes-extra ?= "${AUTOREV}"
7936 SRCREV_pn-matchbox-terminal ?= "${AUTOREV}"
7937 SRCREV_pn-matchbox-wm ?= "${AUTOREV}"
7938 SRCREV_pn-settings-daemon ?= "${AUTOREV}"
7939 SRCREV_pn-screenshot ?= "${AUTOREV}"
7940 . . .
7941
7942These lines allow you to
7943experiment with building a distribution that tracks the latest
7944development source for numerous packages.
7945
7946.. note::
7947
7948 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
7949 this in mind if you use it.
7950
7951Creating a Read-Only Root Filesystem
7952====================================
7953
7954Suppose, for security reasons, you need to disable your target device's
7955root filesystem's write permissions (i.e. you need a read-only root
7956filesystem). Or, perhaps you are running the device's operating system
7957from a read-only storage device. For either case, you can customize your
7958image for that behavior.
7959
7960.. note::
7961
7962 Supporting a read-only root filesystem requires that the system and
7963 applications do not try to write to the root filesystem. You must
7964 configure all parts of the target system to write elsewhere, or to
7965 gracefully fail in the event of attempting to write to the root
7966 filesystem.
7967
7968Creating the Root Filesystem
7969----------------------------
7970
7971To create the read-only root filesystem, simply add the
7972"read-only-rootfs" feature to your image, normally in one of two ways.
7973The first way is to add the "read-only-rootfs" image feature in the
7974image's recipe file via the ``IMAGE_FEATURES`` variable:
7975::
7976
7977 IMAGE_FEATURES += "read-only-rootfs"
7978
7979As an alternative, you can add the same feature
7980from within your build directory's ``local.conf`` file with the
7981associated ``EXTRA_IMAGE_FEATURES`` variable, as in:
7982::
7983
7984 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
7985
7986For more information on how to use these variables, see the
7987":ref:`dev-manual/common-tasks:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
7988section. For information on the variables, see
7989:term:`IMAGE_FEATURES` and
7990:term:`EXTRA_IMAGE_FEATURES`.
7991
7992Post-Installation Scripts and Read-Only Root Filesystem
7993-------------------------------------------------------
7994
7995It is very important that you make sure all post-Installation
7996(``pkg_postinst``) scripts for packages that are installed into the
7997image can be run at the time when the root filesystem is created during
7998the build on the host system. These scripts cannot attempt to run during
7999first-boot on the target device. With the "read-only-rootfs" feature
8000enabled, the build system checks during root filesystem creation to make
8001sure all post-installation scripts succeed. If any of these scripts
8002still need to be run after the root filesystem is created, the build
8003immediately fails. These build-time checks ensure that the build fails
8004rather than the target device fails later during its initial boot
8005operation.
8006
8007Most of the common post-installation scripts generated by the build
8008system for the out-of-the-box Yocto Project are engineered so that they
8009can run during root filesystem creation (e.g. post-installation scripts
8010for caching fonts). However, if you create and add custom scripts, you
8011need to be sure they can be run during this file system creation.
8012
8013Here are some common problems that prevent post-installation scripts
8014from running during root filesystem creation:
8015
8016- *Not using $D in front of absolute paths:* The build system defines
8017 ``$``\ :term:`D` when the root
8018 filesystem is created. Furthermore, ``$D`` is blank when the script
8019 is run on the target device. This implies two purposes for ``$D``:
8020 ensuring paths are valid in both the host and target environments,
8021 and checking to determine which environment is being used as a method
8022 for taking appropriate actions.
8023
8024- *Attempting to run processes that are specific to or dependent on the
8025 target architecture:* You can work around these attempts by using
8026 native tools, which run on the host system, to accomplish the same
8027 tasks, or by alternatively running the processes under QEMU, which
8028 has the ``qemu_run_binary`` function. For more information, see the
8029 :ref:`qemu <ref-classes-qemu>` class.
8030
8031Areas With Write Access
8032-----------------------
8033
8034With the "read-only-rootfs" feature enabled, any attempt by the target
8035to write to the root filesystem at runtime fails. Consequently, you must
8036make sure that you configure processes and applications that attempt
8037these types of writes do so to directories with write access (e.g.
8038``/tmp`` or ``/var/run``).
8039
8040Maintaining Build Output Quality
8041================================
8042
8043Many factors can influence the quality of a build. For example, if you
8044upgrade a recipe to use a new version of an upstream software package or
8045you experiment with some new configuration options, subtle changes can
8046occur that you might not detect until later. Consider the case where
8047your recipe is using a newer version of an upstream package. In this
8048case, a new version of a piece of software might introduce an optional
8049dependency on another library, which is auto-detected. If that library
8050has already been built when the software is building, the software will
8051link to the built library and that library will be pulled into your
8052image along with the new software even if you did not want the library.
8053
8054The :ref:`buildhistory <ref-classes-buildhistory>`
8055class exists to help you maintain the quality of your build output. You
8056can use the class to highlight unexpected and possibly unwanted changes
8057in the build output. When you enable build history, it records
8058information about the contents of each package and image and then
8059commits that information to a local Git repository where you can examine
8060the information.
8061
8062The remainder of this section describes the following:
8063
8064- :ref:`How you can enable and disable build history <dev-manual/common-tasks:enabling and disabling build history>`
8065
8066- :ref:`How to understand what the build history contains <dev-manual/common-tasks:understanding what the build history contains>`
8067
8068- :ref:`How to limit the information used for build history <dev-manual/common-tasks:using build history to gather image information only>`
8069
8070- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/common-tasks:examining build history information>`
8071
8072Enabling and Disabling Build History
8073------------------------------------
8074
8075Build history is disabled by default. To enable it, add the following
8076``INHERIT`` statement and set the
8077:term:`BUILDHISTORY_COMMIT`
8078variable to "1" at the end of your ``conf/local.conf`` file found in the
8079:term:`Build Directory`:
8080::
8081
8082 INHERIT += "buildhistory"
8083 BUILDHISTORY_COMMIT = "1"
8084
8085Enabling build history as
8086previously described causes the OpenEmbedded build system to collect
8087build output information and commit it as a single commit to a local
8088:ref:`overview-manual/development-environment:git` repository.
8089
8090.. note::
8091
8092 Enabling build history increases your build times slightly,
8093 particularly for images, and increases the amount of disk space used
8094 during the build.
8095
8096You can disable build history by removing the previous statements from
8097your ``conf/local.conf`` file.
8098
8099Understanding What the Build History Contains
8100---------------------------------------------
8101
8102Build history information is kept in
8103``${``\ :term:`TOPDIR`\ ``}/buildhistory``
8104in the Build Directory as defined by the
8105:term:`BUILDHISTORY_DIR`
8106variable. The following is an example abbreviated listing:
8107
8108.. image:: figures/buildhistory.png
8109 :align: center
8110
8111At the top level, a ``metadata-revs`` file exists that lists the
8112revisions of the repositories for the enabled layers when the build was
8113produced. The rest of the data splits into separate ``packages``,
8114``images`` and ``sdk`` directories, the contents of which are described
8115as follows.
8116
8117Build History Package Information
8118~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8119
8120The history for each package contains a text file that has name-value
8121pairs with information about the package. For example,
8122``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
8123contains the following:
8124
8125.. code-block:: none
8126
8127 PV = 1.22.1
8128 PR = r32
8129 RPROVIDES =
8130 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
8131 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
8132 PKGSIZE = 540168
8133 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
8134 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
8135 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
8136 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
8137 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
8138 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
8139 /etc/busybox.links.nosuid /etc/busybox.links.suid
8140
8141Most of these
8142name-value pairs correspond to variables used to produce the package.
8143The exceptions are ``FILELIST``, which is the actual list of files in
8144the package, and ``PKGSIZE``, which is the total size of files in the
8145package in bytes.
8146
8147A file also exists that corresponds to the recipe from which the package
8148came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
8149
8150.. code-block:: none
8151
8152 PV = 1.22.1
8153 PR = r32
8154 DEPENDS = initscripts kern-tools-native update-rc.d-native \
8155 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
8156 virtual/libc virtual/update-alternatives
8157 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
8158 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
8159 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
8160
8161Finally, for those recipes fetched from a version control system (e.g.,
8162Git), a file exists that lists source revisions that are specified in
8163the recipe and lists the actual revisions used during the build. Listed
8164and actual revisions might differ when
8165:term:`SRCREV` is set to
8166${:term:`AUTOREV`}. Here is an
8167example assuming
8168``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``):
8169::
8170
8171 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8172 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8173 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
8174 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
8175
8176You can use the
8177``buildhistory-collect-srcrevs`` command with the ``-a`` option to
8178collect the stored ``SRCREV`` values from build history and report them
8179in a format suitable for use in global configuration (e.g.,
8180``local.conf`` or a distro include file) to override floating
8181``AUTOREV`` values to a fixed set of revisions. Here is some example
8182output from this command:
8183::
8184
8185 $ buildhistory-collect-srcrevs -a
8186 # i586-poky-linux
8187 SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
8188 SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
8189 SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
8190 SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
8191 # x86_64-linux
8192 SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
8193 SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
8194 SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
8195 SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
8196 SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
8197 SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
8198 SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
8199 SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
8200 SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
8201 # qemux86-poky-linux
8202 SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
8203 SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
8204 # all-poky-linux
8205 SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
8206
8207.. note::
8208
8209 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
8210
8211 - By default, only values where the ``SRCREV`` was not hardcoded
8212 (usually when ``AUTOREV`` is used) are reported. Use the ``-a``
8213 option to see all ``SRCREV`` values.
8214
8215 - The output statements might not have any effect if overrides are
8216 applied elsewhere in the build system configuration. Use the
8217 ``-f`` option to add the ``forcevariable`` override to each output
8218 line if you need to work around this restriction.
8219
8220 - The script does apply special handling when building for multiple
8221 machines. However, the script does place a comment before each set
8222 of values that specifies which triplet to which they belong as
8223 previously shown (e.g., ``i586-poky-linux``).
8224
8225Build History Image Information
8226~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8227
8228The files produced for each image are as follows:
8229
8230- ``image-files:`` A directory containing selected files from the root
8231 filesystem. The files are defined by
8232 :term:`BUILDHISTORY_IMAGE_FILES`.
8233
8234- ``build-id.txt:`` Human-readable information about the build
8235 configuration and metadata source revisions. This file contains the
8236 full build header as printed by BitBake.
8237
8238- ``*.dot:`` Dependency graphs for the image that are compatible with
8239 ``graphviz``.
8240
8241- ``files-in-image.txt:`` A list of files in the image with
8242 permissions, owner, group, size, and symlink information.
8243
8244- ``image-info.txt:`` A text file containing name-value pairs with
8245 information about the image. See the following listing example for
8246 more information.
8247
8248- ``installed-package-names.txt:`` A list of installed packages by name
8249 only.
8250
8251- ``installed-package-sizes.txt:`` A list of installed packages ordered
8252 by size.
8253
8254- ``installed-packages.txt:`` A list of installed packages with full
8255 package filenames.
8256
8257.. note::
8258
8259 Installed package information is able to be gathered and produced
8260 even if package management is disabled for the final image.
8261
8262Here is an example of ``image-info.txt``:
8263
8264.. code-block:: none
8265
8266 DISTRO = poky
8267 DISTRO_VERSION = 1.7
8268 USER_CLASSES = buildstats image-mklibs image-prelink
8269 IMAGE_CLASSES = image_types
8270 IMAGE_FEATURES = debug-tweaks
8271 IMAGE_LINGUAS =
8272 IMAGE_INSTALL = packagegroup-core-boot run-postinsts
8273 BAD_RECOMMENDATIONS =
8274 NO_RECOMMENDATIONS =
8275 PACKAGE_EXCLUDE =
8276 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
8277 write_image_manifest ; buildhistory_list_installed_image ; \
8278 buildhistory_get_image_installed ; ssh_allow_empty_password; \
8279 postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
8280 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
8281 IMAGESIZE = 6900
8282
8283Other than ``IMAGESIZE``,
8284which is the total size of the files in the image in Kbytes, the
8285name-value pairs are variables that may have influenced the content of
8286the image. This information is often useful when you are trying to
8287determine why a change in the package or file listings has occurred.
8288
8289Using Build History to Gather Image Information Only
8290~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8291
8292As you can see, build history produces image information, including
8293dependency graphs, so you can see why something was pulled into the
8294image. If you are just interested in this information and not interested
8295in collecting specific package or SDK information, you can enable
8296writing only image information without any history by adding the
8297following to your ``conf/local.conf`` file found in the
8298:term:`Build Directory`:
8299::
8300
8301 INHERIT += "buildhistory"
8302 BUILDHISTORY_COMMIT = "0"
8303 BUILDHISTORY_FEATURES = "image"
8304
8305Here, you set the
8306:term:`BUILDHISTORY_FEATURES`
8307variable to use the image feature only.
8308
8309Build History SDK Information
8310~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8311
8312Build history collects similar information on the contents of SDKs (e.g.
8313``bitbake -c populate_sdk imagename``) as compared to information it
8314collects for images. Furthermore, this information differs depending on
8315whether an extensible or standard SDK is being produced.
8316
8317The following list shows the files produced for SDKs:
8318
8319- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
8320 owner, group, size, and symlink information. This list includes both
8321 the host and target parts of the SDK.
8322
8323- ``sdk-info.txt:`` A text file containing name-value pairs with
8324 information about the SDK. See the following listing example for more
8325 information.
8326
8327- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
8328 with information about task group sizes (e.g. ``do_populate_sysroot``
8329 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
8330 only when an extensible SDK is created.
8331
8332- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
8333 with information for the shared-state packages and sizes in the SDK.
8334 The ``sstate-package-sizes.txt`` file exists only when an extensible
8335 SDK is created.
8336
8337- ``sdk-files:`` A folder that contains copies of the files mentioned
8338 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
8339 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
8340 specific to the extensible SDK although you can set it differently if
8341 you would like to pull in specific files from the standard SDK.
8342
8343 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
8344 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
8345 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
8346 copied into the ``sdk-files`` directory.
8347
8348- The following information appears under each of the ``host`` and
8349 ``target`` directories for the portions of the SDK that run on the
8350 host and on the target, respectively:
8351
8352 .. note::
8353
8354 The following files for the most part are empty when producing an
8355 extensible SDK because this type of SDK is not constructed from
8356 packages as is the standard SDK.
8357
8358 - ``depends.dot:`` Dependency graph for the SDK that is compatible
8359 with ``graphviz``.
8360
8361 - ``installed-package-names.txt:`` A list of installed packages by
8362 name only.
8363
8364 - ``installed-package-sizes.txt:`` A list of installed packages
8365 ordered by size.
8366
8367 - ``installed-packages.txt:`` A list of installed packages with full
8368 package filenames.
8369
8370Here is an example of ``sdk-info.txt``:
8371
8372.. code-block:: none
8373
8374 DISTRO = poky
8375 DISTRO_VERSION = 1.3+snapshot-20130327
8376 SDK_NAME = poky-glibc-i686-arm
8377 SDK_VERSION = 1.3+snapshot
8378 SDKMACHINE =
8379 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
8380 BAD_RECOMMENDATIONS =
8381 SDKSIZE = 352712
8382
8383Other than ``SDKSIZE``, which is
8384the total size of the files in the SDK in Kbytes, the name-value pairs
8385are variables that might have influenced the content of the SDK. This
8386information is often useful when you are trying to determine why a
8387change in the package or file listings has occurred.
8388
8389Examining Build History Information
8390~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8391
8392You can examine build history output from the command line or from a web
8393interface.
8394
8395To see any changes that have occurred (assuming you have
8396:term:`BUILDHISTORY_COMMIT` = "1"),
8397you can simply use any Git command that allows you to view the history
8398of a repository. Here is one method:
8399::
8400
8401 $ git log -p
8402
8403You need to realize,
8404however, that this method does show changes that are not significant
8405(e.g. a package's size changing by a few bytes).
8406
8407A command-line tool called ``buildhistory-diff`` does exist, though,
8408that queries the Git repository and prints just the differences that
8409might be significant in human-readable form. Here is an example:
8410::
8411
8412 $ ~/poky/poky/scripts/buildhistory-diff . HEAD^
8413 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
8414 /etc/anotherpkg.conf was added
8415 /sbin/anotherpkg was added
8416 * (installed-package-names.txt):
8417 * anotherpkg was added
8418 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
8419 anotherpkg was added
8420 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
8421 * PR changed from "r0" to "r1"
8422 * PV changed from "0.1.10" to "0.1.12"
8423 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
8424 * PR changed from "r0" to "r1"
8425 * PV changed from "0.1.10" to "0.1.12"
8426
8427.. note::
8428
8429 The ``buildhistory-diff`` tool requires the ``GitPython``
8430 package. Be sure to install it using Pip3 as follows:
8431 ::
8432
8433 $ pip3 install GitPython --user
8434
8435
8436 Alternatively, you can install ``python3-git`` using the appropriate
8437 distribution package manager (e.g. ``apt-get``, ``dnf``, or ``zipper``).
8438
8439To see changes to the build history using a web interface, follow the
8440instruction in the ``README`` file
8441:yocto_git:`here </buildhistory-web/>`.
8442
8443Here is a sample screenshot of the interface:
8444
8445.. image:: figures/buildhistory-web.png
8446 :align: center
8447
8448Performing Automated Runtime Testing
8449====================================
8450
8451The OpenEmbedded build system makes available a series of automated
8452tests for images to verify runtime functionality. You can run these
8453tests on either QEMU or actual target hardware. Tests are written in
8454Python making use of the ``unittest`` module, and the majority of them
8455run commands on the target system over SSH. This section describes how
8456you set up the environment to use these tests, run available tests, and
8457write and add your own tests.
8458
8459For information on the test and QA infrastructure available within the
8460Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
8461section in the Yocto Project Reference Manual.
8462
8463Enabling Tests
8464--------------
8465
8466Depending on whether you are planning to run tests using QEMU or on the
8467hardware, you have to take different steps to enable the tests. See the
8468following subsections for information on how to enable both types of
8469tests.
8470
8471Enabling Runtime Tests on QEMU
8472~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8473
8474In order to run tests, you need to do the following:
8475
8476- *Set up to avoid interaction with sudo for networking:* To
8477 accomplish this, you must do one of the following:
8478
8479 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
8480 commands or just for ``runqemu-ifup``. You must provide the full
8481 path as that can change if you are using multiple clones of the
8482 source repository.
8483
8484 .. note::
8485
8486 On some distributions, you also need to comment out "Defaults
8487 requiretty" in ``/etc/sudoers``.
8488
8489 - Manually configure a tap interface for your system.
8490
8491 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
8492 should generate a list of tap devices. This is the option
8493 typically chosen for Autobuilder-type environments.
8494
8495 .. note::
8496
8497 - Be sure to use an absolute path when calling this script
8498 with sudo.
8499
8500 - The package recipe ``qemu-helper-native`` is required to run
8501 this script. Build the package using the following command:
8502 ::
8503
8504 $ bitbake qemu-helper-native
8505
8506- *Set the DISPLAY variable:* You need to set this variable so that
8507 you have an X server available (e.g. start ``vncserver`` for a
8508 headless machine).
8509
8510- *Be sure your host's firewall accepts incoming connections from
8511 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
8512 HTTP server on a random high number port, which is used to serve
8513 files to the target. The DNF module serves
8514 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
8515 That means your host's firewall must accept incoming connections from
8516 192.168.7.0/24, which is the default IP range used for tap devices by
8517 ``runqemu``.
8518
8519- *Be sure your host has the correct packages installed:* Depending
8520 your host's distribution, you need to have the following packages
8521 installed:
8522
8523 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
8524
8525 - OpenSUSE: ``sysstat`` and ``iproute2``
8526
8527 - Fedora: ``sysstat`` and ``iproute``
8528
8529 - CentOS: ``sysstat`` and ``iproute``
8530
8531Once you start running the tests, the following happens:
8532
85331. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
8534
85352. The image is booted under QEMU using the standard ``runqemu`` script.
8536
85373. A default timeout of 500 seconds occurs to allow for the boot process
8538 to reach the login prompt. You can change the timeout period by
8539 setting
8540 :term:`TEST_QEMUBOOT_TIMEOUT`
8541 in the ``local.conf`` file.
8542
85434. Once the boot process is reached and the login prompt appears, the
8544 tests run. The full boot log is written to
8545 ``${WORKDIR}/testimage/qemu_boot_log``.
8546
85475. Each test module loads in the order found in ``TEST_SUITES``. You can
8548 find the full output of the commands run over SSH in
8549 ``${WORKDIR}/testimgage/ssh_target_log``.
8550
85516. If no failures occur, the task running the tests ends successfully.
8552 You can find the output from the ``unittest`` in the task log at
8553 ``${WORKDIR}/temp/log.do_testimage``.
8554
8555Enabling Runtime Tests on Hardware
8556~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
8557
8558The OpenEmbedded build system can run tests on real hardware, and for
8559certain devices it can also deploy the image to be tested onto the
8560device beforehand.
8561
8562For automated deployment, a "master image" is installed onto the
8563hardware once as part of setup. Then, each time tests are to be run, the
8564following occurs:
8565
85661. The master image is booted into and used to write the image to be
8567 tested to a second partition.
8568
85692. The device is then rebooted using an external script that you need to
8570 provide.
8571
85723. The device boots into the image to be tested.
8573
8574When running tests (independent of whether the image has been deployed
8575automatically or not), the device is expected to be connected to a
8576network on a pre-determined IP address. You can either use static IP
8577addresses written into the image, or set the image to use DHCP and have
8578your DHCP server on the test network assign a known IP address based on
8579the MAC address of the device.
8580
8581In order to run tests on hardware, you need to set ``TEST_TARGET`` to an
8582appropriate value. For QEMU, you do not have to change anything, the
8583default value is "qemu". For running tests on hardware, the following
8584options exist:
8585
8586- *"simpleremote":* Choose "simpleremote" if you are going to run tests
8587 on a target system that is already running the image to be tested and
8588 is available on the network. You can use "simpleremote" in
8589 conjunction with either real hardware or an image running within a
8590 separately started QEMU or any other virtual machine manager.
8591
8592- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
8593 an EFI-based machine with ``systemd-boot`` as bootloader and
8594 ``core-image-testmaster`` (or something similar) is installed. Also,
8595 your hardware under test must be in a DHCP-enabled network that gives
8596 it the same IP address for each reboot.
8597
8598 If you choose "SystemdbootTarget", there are additional requirements
8599 and considerations. See the "`Selecting
8600 SystemdbootTarget <#selecting-systemdboottarget>`__" section, which
8601 follows, for more information.
8602
8603- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
8604 images and running tests on the BeagleBone "Black" or original
8605 "White" hardware. For information on how to use these tests, see the
8606 comments at the top of the BeagleBoneTarget
8607 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
8608
8609- *"EdgeRouterTarget":* Choose "EdgeRouterTarget" if you are deploying
8610 images and running tests on the Ubiquiti Networks EdgeRouter Lite.
8611 For information on how to use these tests, see the comments at the
8612 top of the EdgeRouterTarget
8613 ``meta-yocto-bsp/lib/oeqa/controllers/edgeroutertarget.py`` file.
8614
8615- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
8616 tests on any generic PC that boots using GRUB. For information on how
8617 to use these tests, see the comments at the top of the GrubTarget
8618 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
8619
8620- *"your-target":* Create your own custom target if you want to run
8621 tests when you are deploying images and running tests on a custom
8622 machine within your BSP layer. To do this, you need to add a Python
8623 unit that defines the target class under ``lib/oeqa/controllers/``
8624 within your layer. You must also provide an empty ``__init__.py``.
8625 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
8626
8627Selecting SystemdbootTarget
8628~~~~~~~~~~~~~~~~~~~~~~~~~~~
8629
8630If you did not set ``TEST_TARGET`` to "SystemdbootTarget", then you do
8631not need any information in this section. You can skip down to the
8632"`Running Tests <#qemu-image-running-tests>`__" section.
8633
8634If you did set ``TEST_TARGET`` to "SystemdbootTarget", you also need to
8635perform a one-time setup of your master image by doing the following:
8636
86371. *Set EFI_PROVIDER:* Be sure that ``EFI_PROVIDER`` is as follows:
8638 ::
8639
8640 EFI_PROVIDER = "systemd-boot"
8641
86422. *Build the master image:* Build the ``core-image-testmaster`` image.
8643 The ``core-image-testmaster`` recipe is provided as an example for a
8644 "master" image and you can customize the image recipe as you would
8645 any other recipe.
8646
8647 Here are the image recipe requirements:
8648
8649 - Inherits ``core-image`` so that kernel modules are installed.
8650
8651 - Installs normal linux utilities not busybox ones (e.g. ``bash``,
8652 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
8653
8654 - Uses a custom Initial RAM Disk (initramfs) image with a custom
8655 installer. A normal image that you can install usually creates a
8656 single rootfs partition. This image uses another installer that
8657 creates a specific partition layout. Not all Board Support
8658 Packages (BSPs) can use an installer. For such cases, you need to
8659 manually create the following partition layout on the target:
8660
8661 - First partition mounted under ``/boot``, labeled "boot".
8662
8663 - The main rootfs partition where this image gets installed,
8664 which is mounted under ``/``.
8665
8666 - Another partition labeled "testrootfs" where test images get
8667 deployed.
8668
86693. *Install image:* Install the image that you just built on the target
8670 system.
8671
8672The final thing you need to do when setting ``TEST_TARGET`` to
8673"SystemdbootTarget" is to set up the test image:
8674
86751. *Set up your local.conf file:* Make sure you have the following
8676 statements in your ``local.conf`` file:
8677 ::
8678
8679 IMAGE_FSTYPES += "tar.gz"
8680 INHERIT += "testimage"
8681 TEST_TARGET = "SystemdbootTarget"
8682 TEST_TARGET_IP = "192.168.2.3"
8683
86842. *Build your test image:* Use BitBake to build the image:
8685 ::
8686
8687 $ bitbake core-image-sato
8688
8689Power Control
8690~~~~~~~~~~~~~
8691
8692For most hardware targets other than "simpleremote", you can control
8693power:
8694
8695- You can use ``TEST_POWERCONTROL_CMD`` together with
8696 ``TEST_POWERCONTROL_EXTRA_ARGS`` as a command that runs on the host
8697 and does power cycling. The test code passes one argument to that
8698 command: off, on or cycle (off then on). Here is an example that
8699 could appear in your ``local.conf`` file:
8700 ::
8701
8702 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
8703
8704 In this example, the expect
8705 script does the following:
8706
8707 .. code-block:: shell
8708
8709 ssh test@10.11.12.1 "pyctl nuc1 arg"
8710
8711 It then runs a Python script that controls power for a label called
8712 ``nuc1``.
8713
8714 .. note::
8715
8716 You need to customize ``TEST_POWERCONTROL_CMD`` and
8717 ``TEST_POWERCONTROL_EXTRA_ARGS`` for your own setup. The one requirement
8718 is that it accepts "on", "off", and "cycle" as the last argument.
8719
8720- When no command is defined, it connects to the device over SSH and
8721 uses the classic reboot command to reboot the device. Classic reboot
8722 is fine as long as the machine actually reboots (i.e. the SSH test
8723 has not failed). It is useful for scenarios where you have a simple
8724 setup, typically with a single board, and where some manual
8725 interaction is okay from time to time.
8726
8727If you have no hardware to automatically perform power control but still
8728wish to experiment with automated hardware testing, you can use the
8729``dialog-power-control`` script that shows a dialog prompting you to perform
8730the required power action. This script requires either KDialog or Zenity
8731to be installed. To use this script, set the
8732:term:`TEST_POWERCONTROL_CMD`
8733variable as follows:
8734::
8735
8736 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
8737
8738Serial Console Connection
8739~~~~~~~~~~~~~~~~~~~~~~~~~
8740
8741For test target classes requiring a serial console to interact with the
8742bootloader (e.g. BeagleBoneTarget, EdgeRouterTarget, and GrubTarget),
8743you need to specify a command to use to connect to the serial console of
8744the target machine by using the
8745:term:`TEST_SERIALCONTROL_CMD`
8746variable and optionally the
8747:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
8748variable.
8749
8750These cases could be a serial terminal program if the machine is
8751connected to a local serial port, or a ``telnet`` or ``ssh`` command
8752connecting to a remote console server. Regardless of the case, the
8753command simply needs to connect to the serial console and forward that
8754connection to standard input and output as any normal terminal program
8755does. For example, to use the picocom terminal program on serial device
8756``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows:
8757::
8758
8759 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
8760
8761For local
8762devices where the serial port device disappears when the device reboots,
8763an additional "serdevtry" wrapper script is provided. To use this
8764wrapper, simply prefix the terminal command with
8765``${COREBASE}/scripts/contrib/serdevtry``:
8766::
8767
8768 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
8769
8770Running Tests
8771-------------
8772
8773You can start the tests automatically or manually:
8774
8775- *Automatically running tests:* To run the tests automatically after
8776 the OpenEmbedded build system successfully creates an image, first
8777 set the
8778 :term:`TESTIMAGE_AUTO`
8779 variable to "1" in your ``local.conf`` file in the
8780 :term:`Build Directory`:
8781 ::
8782
8783 TESTIMAGE_AUTO = "1"
8784
8785 Next, build your image. If the image successfully builds, the
8786 tests run:
8787 ::
8788
8789 bitbake core-image-sato
8790
8791- *Manually running tests:* To manually run the tests, first globally
8792 inherit the
8793 :ref:`testimage <ref-classes-testimage*>` class
8794 by editing your ``local.conf`` file:
8795 ::
8796
8797 INHERIT += "testimage"
8798
8799 Next, use BitBake to run the tests:
8800 ::
8801
8802 bitbake -c testimage image
8803
8804All test files reside in ``meta/lib/oeqa/runtime`` in the
8805:term:`Source Directory`. A test name maps
8806directly to a Python module. Each test module may contain a number of
8807individual tests. Tests are usually grouped together by the area tested
8808(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/systemd.py``).
8809
8810You can add tests to any layer provided you place them in the proper
8811area and you extend :term:`BBPATH` in
8812the ``local.conf`` file as normal. Be sure that tests reside in
8813``layer/lib/oeqa/runtime``.
8814
8815.. note::
8816
8817 Be sure that module names do not collide with module names used in
8818 the default set of test modules in ``meta/lib/oeqa/runtime``.
8819
8820You can change the set of tests run by appending or overriding
8821:term:`TEST_SUITES` variable in
8822``local.conf``. Each name in ``TEST_SUITES`` represents a required test
8823for the image. Test modules named within ``TEST_SUITES`` cannot be
8824skipped even if a test is not suitable for an image (e.g. running the
8825RPM tests on an image without ``rpm``). Appending "auto" to
8826``TEST_SUITES`` causes the build system to try to run all tests that are
8827suitable for the image (i.e. each test module may elect to skip itself).
8828
8829The order you list tests in ``TEST_SUITES`` is important and influences
8830test dependencies. Consequently, tests that depend on other tests should
8831be added after the test on which they depend. For example, since the
8832``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
8833"ping" in the list. The test class provides no re-ordering or dependency
8834handling.
8835
8836.. note::
8837
8838 Each module can have multiple classes with multiple test methods.
8839 And, Python ``unittest`` rules apply.
8840
8841Here are some things to keep in mind when running tests:
8842
8843- The default tests for the image are defined as:
8844 ::
8845
8846 DEFAULT_TEST_SUITES_pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
8847
8848- Add your own test to the list of the by using the following:
8849 ::
8850
8851 TEST_SUITES_append = " mytest"
8852
8853- Run a specific list of tests as follows:
8854 ::
8855
8856 TEST_SUITES = "test1 test2 test3"
8857
8858 Remember, order is important. Be sure to place a test that is
8859 dependent on another test later in the order.
8860
8861Exporting Tests
8862---------------
8863
8864You can export tests so that they can run independently of the build
8865system. Exporting tests is required if you want to be able to hand the
8866test execution off to a scheduler. You can only export tests that are
8867defined in :term:`TEST_SUITES`.
8868
8869If your image is already built, make sure the following are set in your
8870``local.conf`` file:
8871::
8872
8873 INHERIT += "testexport"
8874 TEST_TARGET_IP = "IP-address-for-the-test-target"
8875 TEST_SERVER_IP = "IP-address-for-the-test-server"
8876
8877You can then export the tests with the
8878following BitBake command form:
8879::
8880
8881 $ bitbake image -c testexport
8882
8883Exporting the tests places them in the
8884:term:`Build Directory` in
8885``tmp/testexport/``\ image, which is controlled by the
8886``TEST_EXPORT_DIR`` variable.
8887
8888You can now run the tests outside of the build environment:
8889::
8890
8891 $ cd tmp/testexport/image
8892 $ ./runexported.py testdata.json
8893
8894Here is a complete example that shows IP addresses and uses the
8895``core-image-sato`` image:
8896::
8897
8898 INHERIT += "testexport"
8899 TEST_TARGET_IP = "192.168.7.2"
8900 TEST_SERVER_IP = "192.168.7.1"
8901
8902Use BitBake to export the tests:
8903::
8904
8905 $ bitbake core-image-sato -c testexport
8906
8907Run the tests outside of
8908the build environment using the following:
8909::
8910
8911 $ cd tmp/testexport/core-image-sato
8912 $ ./runexported.py testdata.json
8913
8914Writing New Tests
8915-----------------
8916
8917As mentioned previously, all new test files need to be in the proper
8918place for the build system to find them. New tests for additional
8919functionality outside of the core should be added to the layer that adds
8920the functionality, in ``layer/lib/oeqa/runtime`` (as long as
8921:term:`BBPATH` is extended in the
8922layer's ``layer.conf`` file as normal). Just remember the following:
8923
8924- Filenames need to map directly to test (module) names.
8925
8926- Do not use module names that collide with existing core tests.
8927
8928- Minimally, an empty ``__init__.py`` file must exist in the runtime
8929 directory.
8930
8931To create a new test, start by copying an existing module (e.g.
8932``syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
8933code from ``meta/lib/oeqa/utils``, which are helper classes.
8934
8935.. note::
8936
8937 Structure shell commands such that you rely on them and they return a
8938 single code for success. Be aware that sometimes you will need to
8939 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
8940
8941You will notice that all test classes inherit ``oeRuntimeTest``, which
8942is found in ``meta/lib/oetest.py``. This base class offers some helper
8943attributes, which are described in the following sections:
8944
8945Class Methods
8946~~~~~~~~~~~~~
8947
8948Class methods are as follows:
8949
8950- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
8951 package list of the image, which is based on the manifest file that
8952 is generated during the ``do_rootfs`` task.
8953
8954- *hasFeature(feature):* Returns "True" if the feature is in
8955 :term:`IMAGE_FEATURES` or
8956 :term:`DISTRO_FEATURES`.
8957
8958Class Attributes
8959~~~~~~~~~~~~~~~~
8960
8961Class attributes are as follows:
8962
8963- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
8964 Otherwise, ``pscmd`` equals "ps" (busybox).
8965
8966- *tc:* The called test context, which gives access to the
8967 following attributes:
8968
8969 - *d:* The BitBake datastore, which allows you to use stuff such
8970 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
8971
8972 - *testslist and testsrequired:* Used internally. The tests
8973 do not need these.
8974
8975 - *filesdir:* The absolute path to
8976 ``meta/lib/oeqa/runtime/files``, which contains helper files for
8977 tests meant for copying on the target such as small files written
8978 in C for compilation.
8979
8980 - *target:* The target controller object used to deploy and
8981 start an image on a particular target (e.g. Qemu, SimpleRemote,
8982 and SystemdbootTarget). Tests usually use the following:
8983
8984 - *ip:* The target's IP address.
8985
8986 - *server_ip:* The host's IP address, which is usually used
8987 by the DNF test suite.
8988
8989 - *run(cmd, timeout=None):* The single, most used method.
8990 This command is a wrapper for: ``ssh root@host "cmd"``. The
8991 command returns a tuple: (status, output), which are what their
8992 names imply - the return code of "cmd" and whatever output it
8993 produces. The optional timeout argument represents the number
8994 of seconds the test should wait for "cmd" to return. If the
8995 argument is "None", the test uses the default instance's
8996 timeout period, which is 300 seconds. If the argument is "0",
8997 the test runs until the command returns.
8998
8999 - *copy_to(localpath, remotepath):*
9000 ``scp localpath root@ip:remotepath``.
9001
9002 - *copy_from(remotepath, localpath):*
9003 ``scp root@host:remotepath localpath``.
9004
9005Instance Attributes
9006~~~~~~~~~~~~~~~~~~~
9007
9008A single instance attribute exists, which is ``target``. The ``target``
9009instance attribute is identical to the class attribute of the same name,
9010which is described in the previous section. This attribute exists as
9011both an instance and class attribute so tests can use
9012``self.target.run(cmd)`` in instance methods instead of
9013``oeRuntimeTest.tc.target.run(cmd)``.
9014
9015Installing Packages in the DUT Without the Package Manager
9016----------------------------------------------------------
9017
9018When a test requires a package built by BitBake, it is possible to
9019install that package. Installing the package does not require a package
9020manager be installed in the device under test (DUT). It does, however,
9021require an SSH connection and the target must be using the
9022``sshcontrol`` class.
9023
9024.. note::
9025
9026 This method uses ``scp`` to copy files from the host to the target, which
9027 causes permissions and special attributes to be lost.
9028
9029A JSON file is used to define the packages needed by a test. This file
9030must be in the same path as the file used to define the tests.
9031Furthermore, the filename must map directly to the test module name with
9032a ``.json`` extension.
9033
9034The JSON file must include an object with the test name as keys of an
9035object or an array. This object (or array of objects) uses the following
9036data:
9037
9038- "pkg" - A mandatory string that is the name of the package to be
9039 installed.
9040
9041- "rm" - An optional boolean, which defaults to "false", that specifies
9042 to remove the package after the test.
9043
9044- "extract" - An optional boolean, which defaults to "false", that
9045 specifies if the package must be extracted from the package format.
9046 When set to "true", the package is not automatically installed into
9047 the DUT.
9048
9049Following is an example JSON file that handles test "foo" installing
9050package "bar" and test "foobar" installing packages "foo" and "bar".
9051Once the test is complete, the packages are removed from the DUT.
9052::
9053
9054 {
9055 "foo": {
9056 "pkg": "bar"
9057 },
9058 "foobar": [
9059 {
9060 "pkg": "foo",
9061 "rm": true
9062 },
9063 {
9064 "pkg": "bar",
9065 "rm": true
9066 }
9067 ]
9068 }
9069
9070Debugging Tools and Techniques
9071==============================
9072
9073The exact method for debugging build failures depends on the nature of
9074the problem and on the system's area from which the bug originates.
9075Standard debugging practices such as comparison against the last known
9076working version with examination of the changes and the re-application
9077of steps to identify the one causing the problem are valid for the Yocto
9078Project just as they are for any other system. Even though it is
9079impossible to detail every possible potential failure, this section
9080provides some general tips to aid in debugging given a variety of
9081situations.
9082
9083.. note::
9084
9085 A useful feature for debugging is the error reporting tool.
9086 Configuring the Yocto Project to use this tool causes the
9087 OpenEmbedded build system to produce error reporting commands as part
9088 of the console output. You can enter the commands after the build
9089 completes to log error information into a common database, that can
9090 help you figure out what might be going wrong. For information on how
9091 to enable and use this feature, see the
9092 ":ref:`dev-manual/common-tasks:using the error reporting tool`"
9093 section.
9094
9095The following list shows the debugging topics in the remainder of this
9096section:
9097
9098- "`Viewing Logs from Failed
9099 Tasks <#dev-debugging-viewing-logs-from-failed-tasks>`__" describes
9100 how to find and view logs from tasks that failed during the build
9101 process.
9102
9103- "`Viewing Variable
9104 Values <#dev-debugging-viewing-variable-values>`__" describes how to
9105 use the BitBake ``-e`` option to examine variable values after a
9106 recipe has been parsed.
9107
9108- ":ref:`dev-manual/common-tasks:viewing package information with \`\`oe-pkgdata-util\`\``"
9109 describes how to use the ``oe-pkgdata-util`` utility to query
9110 :term:`PKGDATA_DIR` and
9111 display package-related information for built packages.
9112
9113- "`Viewing Dependencies Between Recipes and
9114 Tasks <#dev-viewing-dependencies-between-recipes-and-tasks>`__"
9115 describes how to use the BitBake ``-g`` option to display recipe
9116 dependency information used during the build.
9117
9118- "`Viewing Task Variable
9119 Dependencies <#dev-viewing-task-variable-dependencies>`__" describes
9120 how to use the ``bitbake-dumpsig`` command in conjunction with key
9121 subdirectories in the
9122 :term:`Build Directory` to determine
9123 variable dependencies.
9124
9125- "`Running Specific Tasks <#dev-debugging-taskrunning>`__" describes
9126 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
9127 to run specific tasks in the build chain. It can be useful to run
9128 tasks "out-of-order" when trying isolate build issues.
9129
9130- "`General BitBake Problems <#dev-debugging-bitbake>`__" describes how
9131 to use BitBake's ``-D`` debug output option to reveal more about what
9132 BitBake is doing during the build.
9133
9134- "`Building with No Dependencies <#dev-debugging-buildfile>`__"
9135 describes how to use the BitBake ``-b`` option to build a recipe
9136 while ignoring dependencies.
9137
9138- "`Recipe Logging Mechanisms <#recipe-logging-mechanisms>`__"
9139 describes how to use the many recipe logging functions to produce
9140 debugging output and report errors and warnings.
9141
9142- "`Debugging Parallel Make Races <#debugging-parallel-make-races>`__"
9143 describes how to debug situations where the build consists of several
9144 parts that are run simultaneously and when the output or result of
9145 one part is not ready for use with a different part of the build that
9146 depends on that output.
9147
9148- "`Debugging With the GNU Project Debugger (GDB)
9149 Remotely <#platdev-gdb-remotedebug>`__" describes how to use GDB to
9150 allow you to examine running programs, which can help you fix
9151 problems.
9152
9153- "`Debugging with the GNU Project Debugger (GDB) on the
9154 Target <#debugging-with-the-gnu-project-debugger-gdb-on-the-target>`__"
9155 describes how to use GDB directly on target hardware for debugging.
9156
9157- "`Other Debugging Tips <#dev-other-debugging-others>`__" describes
9158 miscellaneous debugging tips that can be useful.
9159
9160Viewing Logs from Failed Tasks
9161------------------------------
9162
9163You can find the log for a task in the file
9164``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
9165For example, the log for the
9166:ref:`ref-tasks-compile` task of the
9167QEMU minimal image for the x86 machine (``qemux86``) might be in
9168``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
9169To see the commands :term:`BitBake` ran
9170to generate a log, look at the corresponding ``run.do_``\ `taskname` file
9171in the same directory.
9172
9173``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
9174links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
9175``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
9176when it ran. The symlinks always point to the files corresponding to the
9177most recent run.
9178
9179Viewing Variable Values
9180-----------------------
9181
9182Sometimes you need to know the value of a variable as a result of
9183BitBake's parsing step. This could be because some unexpected behavior
9184occurred in your project. Perhaps an attempt to :ref:`modify a variable
9185<bitbake:bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
9186variables>` did not work out as expected.
9187
9188BitBake's ``-e`` option is used to display variable values after
9189parsing. The following command displays the variable values after the
9190configuration files (i.e. ``local.conf``, ``bblayers.conf``,
9191``bitbake.conf`` and so forth) have been parsed:
9192::
9193
9194 $ bitbake -e
9195
9196The following command displays variable values after a specific recipe has
9197been parsed. The variables include those from the configuration as well:
9198::
9199
9200 $ bitbake -e recipename
9201
9202.. note::
9203
9204 Each recipe has its own private set of variables (datastore).
9205 Internally, after parsing the configuration, a copy of the resulting
9206 datastore is made prior to parsing each recipe. This copying implies
9207 that variables set in one recipe will not be visible to other
9208 recipes.
9209
9210 Likewise, each task within a recipe gets a private datastore based on
9211 the recipe datastore, which means that variables set within one task
9212 will not be visible to other tasks.
9213
9214In the output of ``bitbake -e``, each variable is preceded by a
9215description of how the variable got its value, including temporary
9216values that were later overridden. This description also includes
9217variable flags (varflags) set on the variable. The output can be very
9218helpful during debugging.
9219
9220Variables that are exported to the environment are preceded by
9221``export`` in the output of ``bitbake -e``. See the following example:
9222::
9223
9224 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
9225
9226In addition to variable values, the output of the ``bitbake -e`` and
9227``bitbake -e`` recipe commands includes the following information:
9228
9229- The output starts with a tree listing all configuration files and
9230 classes included globally, recursively listing the files they include
9231 or inherit in turn. Much of the behavior of the OpenEmbedded build
9232 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
9233 implemented in the
9234 :ref:`base <ref-classes-base>` class and the
9235 classes it inherits, rather than being built into BitBake itself.
9236
9237- After the variable values, all functions appear in the output. For
9238 shell functions, variables referenced within the function body are
9239 expanded. If a function has been modified using overrides or using
9240 override-style operators like ``_append`` and ``_prepend``, then the
9241 final assembled function body appears in the output.
9242
9243Viewing Package Information with ``oe-pkgdata-util``
9244----------------------------------------------------
9245
9246You can use the ``oe-pkgdata-util`` command-line utility to query
9247:term:`PKGDATA_DIR` and display
9248various package-related information. When you use the utility, you must
9249use it to view information on packages that have already been built.
9250
9251Following are a few of the available ``oe-pkgdata-util`` subcommands.
9252
9253.. note::
9254
9255 You can use the standard \* and ? globbing wildcards as part of
9256 package names and paths.
9257
9258- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
9259 that have been built, optionally limiting the match to packages that
9260 match pattern.
9261
9262- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
9263 files and directories contained in the given packages.
9264
9265 .. note::
9266
9267 A different way to view the contents of a package is to look at
9268 the
9269 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
9270 directory of the recipe that generates the package. This directory
9271 is created by the
9272 :ref:`ref-tasks-package` task
9273 and has one subdirectory for each package the recipe generates,
9274 which contains the files stored in that package.
9275
9276 If you want to inspect the ``${WORKDIR}/packages-split``
9277 directory, make sure that
9278 :ref:`rm_work <ref-classes-rm-work>` is not
9279 enabled when you build the recipe.
9280
9281- ``oe-pkgdata-util find-path path ...``: Lists the names of
9282 the packages that contain the given paths. For example, the following
9283 tells us that ``/usr/share/man/man1/make.1`` is contained in the
9284 ``make-doc`` package:
9285 ::
9286
9287 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
9288 make-doc: /usr/share/man/man1/make.1
9289
9290- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
9291 of the recipes that produce the given packages.
9292
9293For more information on the ``oe-pkgdata-util`` command, use the help
9294facility:
9295::
9296
9297 $ oe-pkgdata-util --help
9298 $ oe-pkgdata-util subcommand --help
9299
9300Viewing Dependencies Between Recipes and Tasks
9301----------------------------------------------
9302
9303Sometimes it can be hard to see why BitBake wants to build other recipes
9304before the one you have specified. Dependency information can help you
9305understand why a recipe is built.
9306
9307To generate dependency information for a recipe, run the following
9308command:
9309::
9310
9311 $ bitbake -g recipename
9312
9313This command writes the following files in the current directory:
9314
9315- ``pn-buildlist``: A list of recipes/targets involved in building
9316 `recipename`. "Involved" here means that at least one task from the
9317 recipe needs to run when building `recipename` from scratch. Targets
9318 that are in
9319 :term:`ASSUME_PROVIDED`
9320 are not listed.
9321
9322- ``task-depends.dot``: A graph showing dependencies between tasks.
9323
9324The graphs are in
9325`DOT <https://en.wikipedia.org/wiki/DOT_%28graph_description_language%29>`__
9326format and can be converted to images (e.g. using the ``dot`` tool from
9327`Graphviz <https://www.graphviz.org/>`__).
9328
9329.. note::
9330
9331 - DOT files use a plain text format. The graphs generated using the
9332 ``bitbake -g`` command are often so large as to be difficult to
9333 read without special pruning (e.g. with Bitbake's ``-I`` option)
9334 and processing. Despite the form and size of the graphs, the
9335 corresponding ``.dot`` files can still be possible to read and
9336 provide useful information.
9337
9338 As an example, the ``task-depends.dot`` file contains lines such
9339 as the following:
9340 ::
9341
9342 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
9343
9344 The above example line reveals that the
9345 :ref:`ref-tasks-configure`
9346 task in ``libxslt`` depends on the
9347 :ref:`ref-tasks-populate_sysroot`
9348 task in ``libxml2``, which is a normal
9349 :term:`DEPENDS` dependency
9350 between the two recipes.
9351
9352 - For an example of how ``.dot`` files can be processed, see the
9353 ``scripts/contrib/graph-tool`` Python script, which finds and
9354 displays paths between graph nodes.
9355
9356You can use a different method to view dependency information by using
9357the following command:
9358::
9359
9360 $ bitbake -g -u taskexp recipename
9361
9362This command
9363displays a GUI window from which you can view build-time and runtime
9364dependencies for the recipes involved in building recipename.
9365
9366Viewing Task Variable Dependencies
9367----------------------------------
9368
9369As mentioned in the
9370":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" section of the BitBake
9371User Manual, BitBake tries to automatically determine what variables a
9372task depends on so that it can rerun the task if any values of the
9373variables change. This determination is usually reliable. However, if
9374you do things like construct variable names at runtime, then you might
9375have to manually declare dependencies on those variables using
9376``vardeps`` as described in the
9377":ref:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" section of the BitBake
9378User Manual.
9379
9380If you are unsure whether a variable dependency is being picked up
9381automatically for a given task, you can list the variable dependencies
9382BitBake has determined by doing the following:
9383
93841. Build the recipe containing the task:
9385::
9386
9387 $ bitbake recipename
9388
93892. Inside the :term:`STAMPS_DIR`
9390 directory, find the signature data (``sigdata``) file that
9391 corresponds to the task. The ``sigdata`` files contain a pickled
9392 Python database of all the metadata that went into creating the input
9393 checksum for the task. As an example, for the
9394 :ref:`ref-tasks-fetch` task of the
9395 ``db`` recipe, the ``sigdata`` file might be found in the following
9396 location:
9397 ::
9398
9399 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9400
9401 For tasks that are accelerated through the shared state
9402 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
9403 additional ``siginfo`` file is written into
9404 :term:`SSTATE_DIR` along with
9405 the cached task output. The ``siginfo`` files contain exactly the
9406 same information as ``sigdata`` files.
9407
94083. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
9409 is an example:
9410 ::
9411
9412 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
9413
9414 In the output of the above command, you will find a line like the
9415 following, which lists all the (inferred) variable dependencies for
9416 the task. This list also includes indirect dependencies from
9417 variables depending on other variables, recursively.
9418 ::
9419
9420 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
9421
9422 .. note::
9423
9424 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
9425 These functions in turn depend on the variables they reference.
9426
9427 The output of ``bitbake-dumpsig`` also includes the value each
9428 variable had, a list of dependencies for each variable, and
9429 :term:`bitbake:BB_HASHBASE_WHITELIST`
9430 information.
9431
9432There is also a ``bitbake-diffsigs`` command for comparing two
9433``siginfo`` or ``sigdata`` files. This command can be helpful when
9434trying to figure out what changed between two versions of a task. If you
9435call ``bitbake-diffsigs`` with just one file, the command behaves like
9436``bitbake-dumpsig``.
9437
9438You can also use BitBake to dump out the signature construction
9439information without executing tasks by using either of the following
9440BitBake command-line options:
9441::
9442
9443 ‐‐dump-signatures=SIGNATURE_HANDLER
9444 -S SIGNATURE_HANDLER
9445
9446
9447.. note::
9448
9449 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
9450 dump only the signature or compare the dumped signature with the cached one,
9451 respectively.
9452
9453Using BitBake with either of these options causes BitBake to dump out
9454``sigdata`` files in the ``stamps`` directory for every task it would
9455have executed instead of building the specified target package.
9456
9457Viewing Metadata Used to Create the Input Signature of a Shared State Task
9458--------------------------------------------------------------------------
9459
9460Seeing what metadata went into creating the input signature of a shared
9461state (sstate) task can be a useful debugging aid. This information is
9462available in signature information (``siginfo``) files in
9463:term:`SSTATE_DIR`. For
9464information on how to view and interpret information in ``siginfo``
9465files, see the "`Viewing Task Variable
9466Dependencies <#dev-viewing-task-variable-dependencies>`__" section.
9467
9468For conceptual information on shared state, see the
9469":ref:`overview-manual/concepts:shared state`"
9470section in the Yocto Project Overview and Concepts Manual.
9471
9472Invalidating Shared State to Force a Task to Run
9473------------------------------------------------
9474
9475The OpenEmbedded build system uses
9476:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
9477:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
9478rebuilding tasks. Collectively, this scheme is known as "shared state
9479code".
9480
9481As with all schemes, this one has some drawbacks. It is possible that
9482you could make implicit changes to your code that the checksum
9483calculations do not take into account. These implicit changes affect a
9484task's output but do not trigger the shared state code into rebuilding a
9485recipe. Consider an example during which a tool changes its output.
9486Assume that the output of ``rpmdeps`` changes. The result of the change
9487should be that all the ``package`` and ``package_write_rpm`` shared
9488state cache items become invalid. However, because the change to the
9489output is external to the code and therefore implicit, the associated
9490shared state cache items do not become invalidated. In this case, the
9491build process uses the cached items rather than running the task again.
9492Obviously, these types of implicit changes can cause problems.
9493
9494To avoid these problems during the build, you need to understand the
9495effects of any changes you make. Realize that changes you make directly
9496to a function are automatically factored into the checksum calculation.
9497Thus, these explicit changes invalidate the associated area of shared
9498state cache. However, you need to be aware of any implicit changes that
9499are not obvious changes to the code and could affect the output of a
9500given task.
9501
9502When you identify an implicit change, you can easily take steps to
9503invalidate the cache and force the tasks to run. The steps you can take
9504are as simple as changing a function's comments in the source code. For
9505example, to invalidate package shared state files, change the comment
9506statements of
9507:ref:`ref-tasks-package` or the
9508comments of one of the functions it calls. Even though the change is
9509purely cosmetic, it causes the checksum to be recalculated and forces
9510the build system to run the task again.
9511
9512.. note::
9513
9514 For an example of a commit that makes a cosmetic change to invalidate
9515 shared state, see this
9516 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
9517
9518Running Specific Tasks
9519----------------------
9520
9521Any given recipe consists of a set of tasks. The standard BitBake
9522behavior in most cases is: ``do_fetch``, ``do_unpack``, ``do_patch``,
9523``do_configure``, ``do_compile``, ``do_install``, ``do_package``,
9524``do_package_write_*``, and ``do_build``. The default task is
9525``do_build`` and any tasks on which it depends build first. Some tasks,
9526such as ``do_devshell``, are not part of the default build chain. If you
9527wish to run a task that is not part of the default build chain, you can
9528use the ``-c`` option in BitBake. Here is an example:
9529::
9530
9531 $ bitbake matchbox-desktop -c devshell
9532
9533The ``-c`` option respects task dependencies, which means that all other
9534tasks (including tasks from other recipes) that the specified task
9535depends on will be run before the task. Even when you manually specify a
9536task to run with ``-c``, BitBake will only run the task if it considers
9537it "out of date". See the
9538":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
9539section in the Yocto Project Overview and Concepts Manual for how
9540BitBake determines whether a task is "out of date".
9541
9542If you want to force an up-to-date task to be rerun (e.g. because you
9543made manual modifications to the recipe's
9544:term:`WORKDIR` that you want to try
9545out), then you can use the ``-f`` option.
9546
9547.. note::
9548
9549 The reason ``-f`` is never required when running the
9550 :ref:`ref-tasks-devshell` task is because the
9551 [\ :ref:`nostamp <bitbake:bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
9552 variable flag is already set for the task.
9553
9554The following example shows one way you can use the ``-f`` option:
9555::
9556
9557 $ bitbake matchbox-desktop
9558 .
9559 .
9560 make some changes to the source code in the work directory
9561 .
9562 .
9563 $ bitbake matchbox-desktop -c compile -f
9564 $ bitbake matchbox-desktop
9565
9566This sequence first builds and then recompiles ``matchbox-desktop``. The
9567last command reruns all tasks (basically the packaging tasks) after the
9568compile. BitBake recognizes that the ``do_compile`` task was rerun and
9569therefore understands that the other tasks also need to be run again.
9570
9571Another, shorter way to rerun a task and all
9572:ref:`ref-manual/tasks:normal recipe build tasks`
9573that depend on it is to use the ``-C`` option.
9574
9575.. note::
9576
9577 This option is upper-cased and is separate from the ``-c``
9578 option, which is lower-cased.
9579
9580Using this option invalidates the given task and then runs the
9581:ref:`ref-tasks-build` task, which is
9582the default task if no task is given, and the tasks on which it depends.
9583You could replace the final two commands in the previous example with
9584the following single command:
9585::
9586
9587 $ bitbake matchbox-desktop -C compile
9588
9589Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
9590the input checksum of the specified task. This tainting indirectly
9591causes the task and its dependent tasks to be rerun through the normal
9592task dependency mechanisms.
9593
9594.. note::
9595
9596 BitBake explicitly keeps track of which tasks have been tainted in
9597 this fashion, and will print warnings such as the following for
9598 builds involving such tasks:
9599
9600 .. code-block:: none
9601
9602 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
9603
9604
9605 The purpose of the warning is to let you know that the work directory
9606 and build output might not be in the clean state they would be in for
9607 a "normal" build, depending on what actions you took. To get rid of
9608 such warnings, you can remove the work directory and rebuild the
9609 recipe, as follows:
9610 ::
9611
9612 $ bitbake matchbox-desktop -c clean
9613 $ bitbake matchbox-desktop
9614
9615
9616You can view a list of tasks in a given package by running the
9617``do_listtasks`` task as follows:
9618::
9619
9620 $ bitbake matchbox-desktop -c listtasks
9621
9622The results appear as output to the console and are also in
9623the file ``${WORKDIR}/temp/log.do_listtasks``.
9624
9625General BitBake Problems
9626------------------------
9627
9628You can see debug output from BitBake by using the ``-D`` option. The
9629debug output gives more information about what BitBake is doing and the
9630reason behind it. Each ``-D`` option you use increases the logging
9631level. The most common usage is ``-DDD``.
9632
9633The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
9634chose a certain version of a package or why BitBake picked a certain
9635provider. This command could also help you in a situation where you
9636think BitBake did something unexpected.
9637
9638Building with No Dependencies
9639-----------------------------
9640
9641To build a specific recipe (``.bb`` file), you can use the following
9642command form:
9643::
9644
9645 $ bitbake -b somepath/somerecipe.bb
9646
9647This command form does
9648not check for dependencies. Consequently, you should use it only when
9649you know existing dependencies have been met.
9650
9651.. note::
9652
9653 You can also specify fragments of the filename. In this case, BitBake
9654 checks for a unique match.
9655
9656Recipe Logging Mechanisms
9657-------------------------
9658
9659The Yocto Project provides several logging functions for producing
9660debugging output and reporting errors and warnings. For Python
9661functions, the following logging functions exist. All of these functions
9662log to ``${T}/log.do_``\ `task`, and can also log to standard output
9663(stdout) with the right settings:
9664
9665- ``bb.plain(msg)``: Writes msg as is to the log while also
9666 logging to stdout.
9667
9668- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
9669 stdout if BitBake is called with "-v".
9670
9671- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the
9672 log. Also logs to stdout if the log level is greater than or equal to
9673 level. See the ":ref:`-D <bitbake:bitbake-user-manual/bitbake-user-manual-intro:usage and syntax>`" option
9674 in the BitBake User Manual for more information.
9675
9676- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
9677 logging to stdout.
9678
9679- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
9680 logging to standard out (stdout).
9681
9682 .. note::
9683
9684 Calling this function does not cause the task to fail.
9685
9686- ``bb.fatal(``\ msg\ ``)``: This logging function is similar to
9687 ``bb.error(``\ msg\ ``)`` but also causes the calling task to fail.
9688
9689 .. note::
9690
9691 ``bb.fatal()`` raises an exception, which means you do not need to put a
9692 "return" statement after the function.
9693
9694The same logging functions are also available in shell functions, under
9695the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
9696and ``bbfatal``. The
9697:ref:`logging <ref-classes-logging>` class
9698implements these functions. See that class in the ``meta/classes``
9699folder of the :term:`Source Directory` for information.
9700
9701Logging With Python
9702~~~~~~~~~~~~~~~~~~~
9703
9704When creating recipes using Python and inserting code that handles build
9705logs, keep in mind the goal is to have informative logs while keeping
9706the console as "silent" as possible. Also, if you want status messages
9707in the log, use the "debug" loglevel.
9708
9709Following is an example written in Python. The code handles logging for
9710a function that determines the number of tasks needed to be run. See the
9711":ref:`ref-tasks-listtasks`"
9712section for additional information:
9713::
9714
9715 python do_listtasks() {
9716 bb.debug(2, "Starting to figure out the task list")
9717 if noteworthy_condition:
9718 bb.note("There are 47 tasks to run")
9719 bb.debug(2, "Got to point xyz")
9720 if warning_trigger:
9721 bb.warn("Detected warning_trigger, this might be a problem later.")
9722 if recoverable_error:
9723 bb.error("Hit recoverable_error, you really need to fix this!")
9724 if fatal_error:
9725 bb.fatal("fatal_error detected, unable to print the task list")
9726 bb.plain("The tasks present are abc")
9727 bb.debug(2, "Finished figuring out the tasklist")
9728 }
9729
9730Logging With Bash
9731~~~~~~~~~~~~~~~~~
9732
9733When creating recipes using Bash and inserting code that handles build
9734logs, you have the same goals - informative with minimal console output.
9735The syntax you use for recipes written in Bash is similar to that of
9736recipes written in Python described in the previous section.
9737
9738Following is an example written in Bash. The code logs the progress of
9739the ``do_my_function`` function.
9740::
9741
9742 do_my_function() {
9743 bbdebug 2 "Running do_my_function"
9744 if [ exceptional_condition ]; then
9745 bbnote "Hit exceptional_condition"
9746 fi
9747 bbdebug 2 "Got to point xyz"
9748 if [ warning_trigger ]; then
9749 bbwarn "Detected warning_trigger, this might cause a problem later."
9750 fi
9751 if [ recoverable_error ]; then
9752 bberror "Hit recoverable_error, correcting"
9753 fi
9754 if [ fatal_error ]; then
9755 bbfatal "fatal_error detected"
9756 fi
9757 bbdebug 2 "Completed do_my_function"
9758 }
9759
9760
9761Debugging Parallel Make Races
9762-----------------------------
9763
9764A parallel ``make`` race occurs when the build consists of several parts
9765that are run simultaneously and a situation occurs when the output or
9766result of one part is not ready for use with a different part of the
9767build that depends on that output. Parallel make races are annoying and
9768can sometimes be difficult to reproduce and fix. However, some simple
9769tips and tricks exist that can help you debug and fix them. This section
9770presents a real-world example of an error encountered on the Yocto
9771Project autobuilder and the process used to fix it.
9772
9773.. note::
9774
9775 If you cannot properly fix a ``make`` race condition, you can work around it
9776 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
9777 variables.
9778
9779The Failure
9780~~~~~~~~~~~
9781
9782For this example, assume that you are building an image that depends on
9783the "neard" package. And, during the build, BitBake runs into problems
9784and creates the following output.
9785
9786.. note::
9787
9788 This example log file has longer lines artificially broken to make
9789 the listing easier to read.
9790
9791If you examine the output or the log file, you see the failure during
9792``make``:
9793
9794.. code-block:: none
9795
9796 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
9797 | DEBUG: Executing shell function do_compile
9798 | NOTE: make -j 16
9799 | make --no-print-directory all-am
9800 | /bin/mkdir -p include/near
9801 | /bin/mkdir -p include/near
9802 | /bin/mkdir -p include/near
9803 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9804 0.14-r0/neard-0.14/include/types.h include/near/types.h
9805 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9806 0.14-r0/neard-0.14/include/log.h include/near/log.h
9807 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9808 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
9809 | /bin/mkdir -p include/near
9810 | /bin/mkdir -p include/near
9811 | /bin/mkdir -p include/near
9812 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9813 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
9814 | /bin/mkdir -p include/near
9815 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9816 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
9817 | /bin/mkdir -p include/near
9818 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9819 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
9820 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9821 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
9822 | /bin/mkdir -p include/near
9823 | /bin/mkdir -p include/near
9824 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9825 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
9826 | /bin/mkdir -p include/near
9827 | /bin/mkdir -p include/near
9828 | /bin/mkdir -p include/near
9829 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9830 0.14-r0/neard-0.14/include/device.h include/near/device.h
9831 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9832 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
9833 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9834 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
9835 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9836 0.14-r0/neard-0.14/include/version.h include/near/version.h
9837 | ln -s /home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
9838 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
9839 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
9840 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/
9841 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
9842 yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
9843 -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
9844 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/yocto-slave/nightly-x86/build/build/
9845 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/yocto-slave/
9846 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
9847 yocto-slave/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
9848 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
9849 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
9850 -o tools/snep-send.o tools/snep-send.c
9851 | In file included from tools/snep-send.c:16:0:
9852 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9853 | #include <near/dbus.h>
9854 | ^
9855 | compilation terminated.
9856 | make[1]: *** [tools/snep-send.o] Error 1
9857 | make[1]: *** Waiting for unfinished jobs....
9858 | make: *** [all] Error 2
9859 | ERROR: oe_runmake failed
9860
9861Reproducing the Error
9862~~~~~~~~~~~~~~~~~~~~~
9863
9864Because race conditions are intermittent, they do not manifest
9865themselves every time you do the build. In fact, most times the build
9866will complete without problems even though the potential race condition
9867exists. Thus, once the error surfaces, you need a way to reproduce it.
9868
9869In this example, compiling the "neard" package is causing the problem.
9870So the first thing to do is build "neard" locally. Before you start the
9871build, set the
9872:term:`PARALLEL_MAKE` variable
9873in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
9874high value for ``PARALLEL_MAKE`` increases the chances of the race
9875condition showing up:
9876::
9877
9878 $ bitbake neard
9879
9880Once the local build for "neard" completes, start a ``devshell`` build:
9881::
9882
9883 $ bitbake neard -c devshell
9884
9885For information on how to use a
9886``devshell``, see the "`Using a Development
9887Shell <#platdev-appdev-devshell>`__" section.
9888
9889In the ``devshell``, do the following:
9890::
9891
9892 $ make clean
9893 $ make tools/snep-send.o
9894
9895The ``devshell`` commands cause the failure to clearly
9896be visible. In this case, a missing dependency exists for the "neard"
9897Makefile target. Here is some abbreviated, sample output with the
9898missing dependency clearly visible at the end:
9899::
9900
9901 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
9902 .
9903 .
9904 .
9905 tools/snep-send.c
9906 In file included from tools/snep-send.c:16:0:
9907 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
9908 #include <near/dbus.h>
9909 ^
9910 compilation terminated.
9911 make: *** [tools/snep-send.o] Error 1
9912 $
9913
9914
9915Creating a Patch for the Fix
9916~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9917
9918Because there is a missing dependency for the Makefile target, you need
9919to patch the ``Makefile.am`` file, which is generated from
9920``Makefile.in``. You can use Quilt to create the patch:
9921::
9922
9923 $ quilt new parallelmake.patch
9924 Patch patches/parallelmake.patch is now on top
9925 $ quilt add Makefile.am
9926 File Makefile.am added to patch patches/parallelmake.patch
9927
9928For more information on using Quilt, see the
9929"`Using Quilt in Your Workflow <#using-a-quilt-workflow>`__" section.
9930
9931At this point you need to make the edits to ``Makefile.am`` to add the
9932missing dependency. For our example, you have to add the following line
9933to the file:
9934::
9935
9936 tools/snep-send.$(OBJEXT): include/near/dbus.h
9937
9938Once you have edited the file, use the ``refresh`` command to create the
9939patch:
9940::
9941
9942 $ quilt refresh
9943 Refreshed patch patches/parallelmake.patch
9944
9945Once
9946the patch file exists, you need to add it back to the originating recipe
9947folder. Here is an example assuming a top-level
9948:term:`Source Directory` named ``poky``:
9949::
9950
9951 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
9952
9953The final thing you need to do to implement the fix in the build is to
9954update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
9955:term:`SRC_URI` statement includes
9956the patch file. The recipe file is in the folder above the patch. Here
9957is what the edited ``SRC_URI`` statement would look like:
9958::
9959
9960 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
9961 file://neard.in \
9962 file://neard.service.in \
9963 file://parallelmake.patch \
9964 "
9965
9966With the patch complete and moved to the correct folder and the
9967``SRC_URI`` statement updated, you can exit the ``devshell``:
9968::
9969
9970 $ exit
9971
9972Testing the Build
9973~~~~~~~~~~~~~~~~~
9974
9975With everything in place, you can get back to trying the build again
9976locally:
9977::
9978
9979 $ bitbake neard
9980
9981This build should succeed.
9982
9983Now you can open up a ``devshell`` again and repeat the clean and make
9984operations as follows:
9985::
9986
9987 $ bitbake neard -c devshell
9988 $ make clean
9989 $ make tools/snep-send.o
9990
9991The build should work without issue.
9992
9993As with all solved problems, if they originated upstream, you need to
9994submit the fix for the recipe in OE-Core and upstream so that the
9995problem is taken care of at its source. See the "`Submitting a Change to
9996the Yocto Project <#how-to-submit-a-change>`__" section for more
9997information.
9998
9999Debugging With the GNU Project Debugger (GDB) Remotely
10000------------------------------------------------------
10001
10002GDB allows you to examine running programs, which in turn helps you to
10003understand and fix problems. It also allows you to perform post-mortem
10004style analysis of program crashes. GDB is available as a package within
10005the Yocto Project and is installed in SDK images by default. See the
10006":ref:`ref-manual/images:Images`" chapter in the Yocto
10007Project Reference Manual for a description of these images. You can find
10008information on GDB at https://sourceware.org/gdb/.
10009
10010.. note::
10011
10012 For best results, install debug (``-dbg``) packages for the applications you
10013 are going to debug. Doing so makes extra debug symbols available that give
10014 you more meaningful output.
10015
10016Sometimes, due to memory or disk space constraints, it is not possible
10017to use GDB directly on the remote target to debug applications. These
10018constraints arise because GDB needs to load the debugging information
10019and the binaries of the process being debugged. Additionally, GDB needs
10020to perform many computations to locate information such as function
10021names, variable names and values, stack traces and so forth - even
10022before starting the debugging process. These extra computations place
10023more load on the target system and can alter the characteristics of the
10024program being debugged.
10025
10026To help get past the previously mentioned constraints, you can use
10027gdbserver, which runs on the remote target and does not load any
10028debugging information from the debugged process. Instead, a GDB instance
10029processes the debugging information that is run on a remote computer -
10030the host GDB. The host GDB then sends control commands to gdbserver to
10031make it stop or start the debugged program, as well as read or write
10032memory regions of that debugged program. All the debugging information
10033loaded and processed as well as all the heavy debugging is done by the
10034host GDB. Offloading these processes gives the gdbserver running on the
10035target a chance to remain small and fast.
10036
10037Because the host GDB is responsible for loading the debugging
10038information and for doing the necessary processing to make actual
10039debugging happen, you have to make sure the host can access the
10040unstripped binaries complete with their debugging information and also
10041be sure the target is compiled with no optimizations. The host GDB must
10042also have local access to all the libraries used by the debugged
10043program. Because gdbserver does not need any local debugging
10044information, the binaries on the remote target can remain stripped.
10045However, the binaries must also be compiled without optimization so they
10046match the host's binaries.
10047
10048To remain consistent with GDB documentation and terminology, the binary
10049being debugged on the remote target machine is referred to as the
10050"inferior" binary. For documentation on GDB see the `GDB
10051site <https://sourceware.org/gdb/documentation/>`__.
10052
10053The following steps show you how to debug using the GNU project
10054debugger.
10055
100561. *Configure your build system to construct the companion debug
10057 filesystem:*
10058
10059 In your ``local.conf`` file, set the following:
10060 ::
10061
10062 IMAGE_GEN_DEBUGFS = "1"
10063 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
10064
10065 These options cause the
10066 OpenEmbedded build system to generate a special companion filesystem
10067 fragment, which contains the matching source and debug symbols to
10068 your deployable filesystem. The build system does this by looking at
10069 what is in the deployed filesystem, and pulling the corresponding
10070 ``-dbg`` packages.
10071
10072 The companion debug filesystem is not a complete filesystem, but only
10073 contains the debug fragments. This filesystem must be combined with
10074 the full filesystem for debugging. Subsequent steps in this procedure
10075 show how to combine the partial filesystem with the full filesystem.
10076
100772. *Configure the system to include gdbserver in the target filesystem:*
10078
10079 Make the following addition in either your ``local.conf`` file or in
10080 an image recipe:
10081 ::
10082
10083 IMAGE_INSTALL_append = " gdbserver"
10084
10085 The change makes
10086 sure the ``gdbserver`` package is included.
10087
100883. *Build the environment:*
10089
10090 Use the following command to construct the image and the companion
10091 Debug Filesystem:
10092 ::
10093
10094 $ bitbake image
10095
10096 Build the cross GDB component and
10097 make it available for debugging. Build the SDK that matches the
10098 image. Building the SDK is best for a production build that can be
10099 used later for debugging, especially during long term maintenance:
10100 ::
10101
10102 $ bitbake -c populate_sdk image
10103
10104 Alternatively, you can build the minimal toolchain components that
10105 match the target. Doing so creates a smaller than typical SDK and
10106 only contains a minimal set of components with which to build simple
10107 test applications, as well as run the debugger:
10108 ::
10109
10110 $ bitbake meta-toolchain
10111
10112 A final method is to build Gdb itself within the build system:
10113 ::
10114
10115 $ bitbake gdb-cross-<architecture>
10116
10117 Doing so produces a temporary copy of
10118 ``cross-gdb`` you can use for debugging during development. While
10119 this is the quickest approach, the two previous methods in this step
10120 are better when considering long-term maintenance strategies.
10121
10122 .. note::
10123
10124 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
10125 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
10126 actual name you want to use.
10127
101284. *Set up the* ``debugfs``\ *:*
10129
10130 Run the following commands to set up the ``debugfs``:
10131 ::
10132
10133 $ mkdir debugfs
10134 $ cd debugfs
10135 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image.rootfs.tar.bz2
10136 $ tar xvfj build-dir/tmp-glibc/deploy/images/machine/image-dbg.rootfs.tar.bz2
10137
101385. *Set up GDB:*
10139
10140 Install the SDK (if you built one) and then source the correct
10141 environment file. Sourcing the environment file puts the SDK in your
10142 ``PATH`` environment variable.
10143
10144 If you are using the build system, Gdb is located in
10145 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
10146
101476. *Boot the target:*
10148
10149 For information on how to run QEMU, see the `QEMU
10150 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
10151
10152 .. note::
10153
10154 Be sure to verify that your host can access the target via TCP.
10155
101567. *Debug a program:*
10157
10158 Debugging a program involves running gdbserver on the target and then
10159 running Gdb on the host. The example in this step debugs ``gzip``:
10160
10161 .. code-block:: shell
10162
10163 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
10164
10165 For
10166 additional gdbserver options, see the `GDB Server
10167 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
10168
10169 After running gdbserver on the target, you need to run Gdb on the
10170 host and configure it and connect to the target. Use these commands:
10171 ::
10172
10173 $ cd directory-holding-the-debugfs-directory
10174 $ arch-gdb
10175 (gdb) set sysroot debugfs
10176 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
10177 (gdb) target remote IP-of-target:1234
10178
10179 At this
10180 point, everything should automatically load (i.e. matching binaries,
10181 symbols and headers).
10182
10183 .. note::
10184
10185 The Gdb ``set`` commands in the previous example can be placed into the
10186 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
10187 commands are in that file.
10188
101898. *Deploying without a full image rebuild:*
10190
10191 In many cases, during development you want a quick method to deploy a
10192 new binary to the target and debug it, without waiting for a full
10193 image build.
10194
10195 One approach to solving this situation is to just build the component
10196 you want to debug. Once you have built the component, copy the
10197 executable directly to both the target and the host ``debugfs``.
10198
10199 If the binary is processed through the debug splitting in
10200 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
10201 contents and corresponding ``/usr/src/debug`` files) from the work
10202 directory. Here is an example:
10203 ::
10204
10205 $ bitbake bash
10206 $ bitbake -c devshell bash
10207 $ cd ..
10208 $ scp packages-split/bash/bin/bash target:/bin/bash
10209 $ cp -a packages-split/bash-dbg/\* path/debugfs
10210
10211Debugging with the GNU Project Debugger (GDB) on the Target
10212-----------------------------------------------------------
10213
10214The previous section addressed using GDB remotely for debugging
10215purposes, which is the most usual case due to the inherent hardware
10216limitations on many embedded devices. However, debugging in the target
10217hardware itself is also possible with more powerful devices. This
10218section describes what you need to do in order to support using GDB to
10219debug on the target hardware.
10220
10221To support this kind of debugging, you need do the following:
10222
10223- Ensure that GDB is on the target. You can do this by adding "gdb" to
10224 :term:`IMAGE_INSTALL`:
10225 ::
10226
10227 IMAGE_INSTALL_append = " gdb"
10228
10229 Alternatively, you can add "tools-debug" to :term:`IMAGE_FEATURES`:
10230 ::
10231
10232 IMAGE_FEATURES_append = " tools-debug"
10233
10234- Ensure that debug symbols are present. You can make sure these
10235 symbols are present by installing ``-dbg``:
10236 ::
10237
10238 IMAGE_INSTALL_append = "packagename-dbg"
10239
10240 Alternatively, you can do the following to include
10241 all the debug symbols:
10242 ::
10243
10244 IMAGE_FEATURES_append = " dbg-pkgs"
10245
10246.. note::
10247
10248 To improve the debug information accuracy, you can reduce the level
10249 of optimization used by the compiler. For example, when adding the
10250 following line to your ``local.conf`` file, you will reduce optimization
10251 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
10252 of "-O -fno-omit-frame-pointer":
10253 ::
10254
10255 DEBUG_BUILD = "1"
10256
10257 Consider that this will reduce the application's performance and is
10258 recommended only for debugging purposes.
10259
10260Other Debugging Tips
10261--------------------
10262
10263Here are some other tips that you might find useful:
10264
10265- When adding new packages, it is worth watching for undesirable items
10266 making their way into compiler command lines. For example, you do not
10267 want references to local system files like ``/usr/lib/`` or
10268 ``/usr/include/``.
10269
10270- If you want to remove the ``psplash`` boot splashscreen, add
10271 ``psplash=false`` to the kernel command line. Doing so prevents
10272 ``psplash`` from loading and thus allows you to see the console. It
10273 is also possible to switch out of the splashscreen by switching the
10274 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
10275
10276- Removing :term:`TMPDIR` (usually
10277 ``tmp/``, within the
10278 :term:`Build Directory`) can often fix
10279 temporary build issues. Removing ``TMPDIR`` is usually a relatively
10280 cheap operation, because task output will be cached in
10281 :term:`SSTATE_DIR` (usually
10282 ``sstate-cache/``, which is also in the Build Directory).
10283
10284 .. note::
10285
10286 Removing ``TMPDIR`` might be a workaround rather than a fix.
10287 Consequently, trying to determine the underlying cause of an issue before
10288 removing the directory is a good idea.
10289
10290- Understanding how a feature is used in practice within existing
10291 recipes can be very helpful. It is recommended that you configure
10292 some method that allows you to quickly search through files.
10293
10294 Using GNU Grep, you can use the following shell function to
10295 recursively search through common recipe-related files, skipping
10296 binary files, ``.git`` directories, and the Build Directory (assuming
10297 its name starts with "build"):
10298 ::
10299
10300 g() {
10301 grep -Ir \
10302 --exclude-dir=.git \
10303 --exclude-dir='build*' \
10304 --include='*.bb*' \
10305 --include='*.inc*' \
10306 --include='*.conf*' \
10307 --include='*.py*' \
10308 "$@"
10309 }
10310
10311 Following are some usage examples:
10312 ::
10313
10314 $ g FOO # Search recursively for "FOO"
10315 $ g -i foo # Search recursively for "foo", ignoring case
10316 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
10317
10318 If figuring
10319 out how some feature works requires a lot of searching, it might
10320 indicate that the documentation should be extended or improved. In
10321 such cases, consider filing a documentation bug using the Yocto
10322 Project implementation of
10323 :yocto_bugs:`Bugzilla <>`. For information on
10324 how to submit a bug against the Yocto Project, see the Yocto Project
10325 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
10326 and the "`Submitting a Defect Against the Yocto
10327 Project <#submitting-a-defect-against-the-yocto-project>`__" section.
10328
10329 .. note::
10330
10331 The manuals might not be the right place to document variables
10332 that are purely internal and have a limited scope (e.g. internal
10333 variables used to implement a single ``.bbclass`` file).
10334
10335Making Changes to the Yocto Project
10336===================================
10337
10338Because the Yocto Project is an open-source, community-based project,
10339you can effect changes to the project. This section presents procedures
10340that show you how to submit a defect against the project and how to
10341submit a change.
10342
10343Submitting a Defect Against the Yocto Project
10344---------------------------------------------
10345
10346Use the Yocto Project implementation of
10347`Bugzilla <https://www.bugzilla.org/about/>`__ to submit a defect (bug)
10348against the Yocto Project. For additional information on this
10349implementation of Bugzilla see the ":ref:`Yocto Project
10350Bugzilla <resources-bugtracker>`" section in the
10351Yocto Project Reference Manual. For more detail on any of the following
10352steps, see the Yocto Project
10353:yocto_wiki:`Bugzilla wiki page </Bugzilla_Configuration_and_Bug_Tracking>`.
10354
10355Use the following general steps to submit a bug:
10356
103571. Open the Yocto Project implementation of :yocto_bugs:`Bugzilla <>`.
10358
103592. Click "File a Bug" to enter a new bug.
10360
103613. Choose the appropriate "Classification", "Product", and "Component"
10362 for which the bug was found. Bugs for the Yocto Project fall into
10363 one of several classifications, which in turn break down into
10364 several products and components. For example, for a bug against the
10365 ``meta-intel`` layer, you would choose "Build System, Metadata &
10366 Runtime", "BSPs", and "bsps-meta-intel", respectively.
10367
103684. Choose the "Version" of the Yocto Project for which you found the
10369 bug (e.g. &DISTRO;).
10370
103715. Determine and select the "Severity" of the bug. The severity
10372 indicates how the bug impacted your work.
10373
103746. Choose the "Hardware" that the bug impacts.
10375
103767. Choose the "Architecture" that the bug impacts.
10377
103788. Choose a "Documentation change" item for the bug. Fixing a bug might
10379 or might not affect the Yocto Project documentation. If you are
10380 unsure of the impact to the documentation, select "Don't Know".
10381
103829. Provide a brief "Summary" of the bug. Try to limit your summary to
10383 just a line or two and be sure to capture the essence of the bug.
10384
1038510. Provide a detailed "Description" of the bug. You should provide as
10386 much detail as you can about the context, behavior, output, and so
10387 forth that surrounds the bug. You can even attach supporting files
10388 for output from logs by using the "Add an attachment" button.
10389
1039011. Click the "Submit Bug" button submit the bug. A new Bugzilla number
10391 is assigned to the bug and the defect is logged in the bug tracking
10392 system.
10393
10394Once you file a bug, the bug is processed by the Yocto Project Bug
10395Triage Team and further details concerning the bug are assigned (e.g.
10396priority and owner). You are the "Submitter" of the bug and any further
10397categorization, progress, or comments on the bug result in Bugzilla
10398sending you an automated email concerning the particular change or
10399progress to the bug.
10400
10401Submitting a Change to the Yocto Project
10402----------------------------------------
10403
10404Contributions to the Yocto Project and OpenEmbedded are very welcome.
10405Because the system is extremely configurable and flexible, we recognize
10406that developers will want to extend, configure or optimize it for their
10407specific uses.
10408
10409The Yocto Project uses a mailing list and a patch-based workflow that is
10410similar to the Linux kernel but contains important differences. In
10411general, a mailing list exists through which you can submit patches. You
10412should send patches to the appropriate mailing list so that they can be
10413reviewed and merged by the appropriate maintainer. The specific mailing
10414list you need to use depends on the location of the code you are
10415changing. Each component (e.g. layer) should have a ``README`` file that
10416indicates where to send the changes and which process to follow.
10417
10418You can send the patch to the mailing list using whichever approach you
10419feel comfortable with to generate the patch. Once sent, the patch is
10420usually reviewed by the community at large. If somebody has concerns
10421with the patch, they will usually voice their concern over the mailing
10422list. If a patch does not receive any negative reviews, the maintainer
10423of the affected layer typically takes the patch, tests it, and then
10424based on successful testing, merges the patch.
10425
10426The "poky" repository, which is the Yocto Project's reference build
10427environment, is a hybrid repository that contains several individual
10428pieces (e.g. BitBake, Metadata, documentation, and so forth) built using
10429the combo-layer tool. The upstream location used for submitting changes
10430varies by component:
10431
10432- *Core Metadata:* Send your patch to the
10433 :oe_lists:`openembedded-core </g/openembedded-core>`
10434 mailing list. For example, a change to anything under the ``meta`` or
10435 ``scripts`` directories should be sent to this mailing list.
10436
10437- *BitBake:* For changes to BitBake (i.e. anything under the
10438 ``bitbake`` directory), send your patch to the
10439 :oe_lists:`bitbake-devel </g/bitbake-devel>`
10440 mailing list.
10441
10442- *"meta-\*" trees:* These trees contain Metadata. Use the
10443 :yocto_lists:`poky </g/poky>` mailing list.
10444
10445- *Documentation*: For changes to the Yocto Project documentation, use the
10446 :yocto_lists:`docs </g/docs>` mailing list.
10447
10448For changes to other layers hosted in the Yocto Project source
10449repositories (i.e. ``yoctoproject.org``) and tools use the
10450:yocto_lists:`Yocto Project </g/yocto/>` general mailing list.
10451
10452.. note::
10453
10454 Sometimes a layer's documentation specifies to use a particular
10455 mailing list. If so, use that list.
10456
10457For additional recipes that do not fit into the core Metadata, you
10458should determine which layer the recipe should go into and submit the
10459change in the manner recommended by the documentation (e.g. the
10460``README`` file) supplied with the layer. If in doubt, please ask on the
10461Yocto general mailing list or on the openembedded-devel mailing list.
10462
10463You can also push a change upstream and request a maintainer to pull the
10464change into the component's upstream repository. You do this by pushing
10465to a contribution repository that is upstream. See the
10466":ref:`overview-manual/development-environment:git workflows and the yocto project`"
10467section in the Yocto Project Overview and Concepts Manual for additional
10468concepts on working in the Yocto Project development environment.
10469
10470Maintainers commonly use ``-next`` branches to test submissions prior to
10471merging patches. Thus, you can get an idea of the status of a patch based on
10472whether the patch has been merged into one of these branches. The commonly
10473used testing branches for OpenEmbedded-Core are as follows:
10474
10475- *openembedded-core "master-next" branch:* This branch is part of the
10476 :oe_git:`openembedded-core </openembedded-core/>` repository and contains
10477 proposed changes to the core metadata.
10478
10479- *poky "master-next" branch:* This branch is part of the
10480 :yocto_git:`poky </poky/>` repository and combines proposed
10481 changes to bitbake, the core metadata and the poky distro.
10482
10483Similarly, stable branches maintained by the project may have corresponding
10484``-next`` branches which collect proposed changes. For example,
10485``&DISTRO_NAME_NO_CAP;-next`` and ``&DISTRO_NAME_NO_CAP_MINUS_ONE;-next``
10486branches in both the "openembdedded-core" and "poky" repositories.
10487
10488Other layers may have similar testing branches but there is no formal
10489requirement or standard for these so please check the documentation for the
10490layers you are contributing to.
10491
10492The following sections provide procedures for submitting a change.
10493
10494Preparing Changes for Submission
10495~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10496
104971. *Make Your Changes Locally:* Make your changes in your local Git
10498 repository. You should make small, controlled, isolated changes.
10499 Keeping changes small and isolated aids review, makes
10500 merging/rebasing easier and keeps the change history clean should
10501 anyone need to refer to it in future.
10502
105032. *Stage Your Changes:* Stage your changes by using the ``git add``
10504 command on each file you changed.
10505
105063. *Commit Your Changes:* Commit the change by using the ``git commit``
10507 command. Make sure your commit information follows standards by
10508 following these accepted conventions:
10509
10510 - Be sure to include a "Signed-off-by:" line in the same style as
10511 required by the Linux kernel. Adding this line signifies that you,
10512 the submitter, have agreed to the Developer's Certificate of
10513 Origin 1.1 as follows:
10514
10515 .. code-block:: none
10516
10517 Developer's Certificate of Origin 1.1
10518
10519 By making a contribution to this project, I certify that:
10520
10521 (a) The contribution was created in whole or in part by me and I
10522 have the right to submit it under the open source license
10523 indicated in the file; or
10524
10525 (b) The contribution is based upon previous work that, to the best
10526 of my knowledge, is covered under an appropriate open source
10527 license and I have the right under that license to submit that
10528 work with modifications, whether created in whole or in part
10529 by me, under the same open source license (unless I am
10530 permitted to submit under a different license), as indicated
10531 in the file; or
10532
10533 (c) The contribution was provided directly to me by some other
10534 person who certified (a), (b) or (c) and I have not modified
10535 it.
10536
10537 (d) I understand and agree that this project and the contribution
10538 are public and that a record of the contribution (including all
10539 personal information I submit with it, including my sign-off) is
10540 maintained indefinitely and may be redistributed consistent with
10541 this project or the open source license(s) involved.
10542
10543 - Provide a single-line summary of the change and, if more
10544 explanation is needed, provide more detail in the body of the
10545 commit. This summary is typically viewable in the "shortlist" of
10546 changes. Thus, providing something short and descriptive that
10547 gives the reader a summary of the change is useful when viewing a
10548 list of many commits. You should prefix this short description
10549 with the recipe name (if changing a recipe), or else with the
10550 short form path to the file being changed.
10551
10552 - For the body of the commit message, provide detailed information
10553 that describes what you changed, why you made the change, and the
10554 approach you used. It might also be helpful if you mention how you
10555 tested the change. Provide as much detail as you can in the body
10556 of the commit message.
10557
10558 .. note::
10559
10560 You do not need to provide a more detailed explanation of a
10561 change if the change is minor to the point of the single line
10562 summary providing all the information.
10563
10564 - If the change addresses a specific bug or issue that is associated
10565 with a bug-tracking ID, include a reference to that ID in your
10566 detailed description. For example, the Yocto Project uses a
10567 specific convention for bug references - any commit that addresses
10568 a specific bug should use the following form for the detailed
10569 description. Be sure to use the actual bug-tracking ID from
10570 Bugzilla for bug-id:
10571 ::
10572
10573 Fixes [YOCTO #bug-id]
10574
10575 detailed description of change
10576
10577Using Email to Submit a Patch
10578~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10579
10580Depending on the components changed, you need to submit the email to a
10581specific mailing list. For some guidance on which mailing list to use,
10582see the `list <#figuring-out-the-mailing-list-to-use>`__ at the
10583beginning of this section. For a description of all the available
10584mailing lists, see the ":ref:`Mailing Lists <resources-mailinglist>`" section in the
10585Yocto Project Reference Manual.
10586
10587Here is the general procedure on how to submit a patch through email
10588without using the scripts once the steps in
10589:ref:`dev-manual/common-tasks:preparing changes for submission` have been followed:
10590
105911. *Format the Commit:* Format the commit into an email message. To
10592 format commits, use the ``git format-patch`` command. When you
10593 provide the command, you must include a revision list or a number of
10594 patches as part of the command. For example, either of these two
10595 commands takes your most recent single commit and formats it as an
10596 email message in the current directory:
10597 ::
10598
10599 $ git format-patch -1
10600
10601 or ::
10602
10603 $ git format-patch HEAD~
10604
10605 After the command is run, the current directory contains a numbered
10606 ``.patch`` file for the commit.
10607
10608 If you provide several commits as part of the command, the
10609 ``git format-patch`` command produces a series of numbered files in
10610 the current directory – one for each commit. If you have more than
10611 one patch, you should also use the ``--cover`` option with the
10612 command, which generates a cover letter as the first "patch" in the
10613 series. You can then edit the cover letter to provide a description
10614 for the series of patches. For information on the
10615 ``git format-patch`` command, see ``GIT_FORMAT_PATCH(1)`` displayed
10616 using the ``man git-format-patch`` command.
10617
10618 .. note::
10619
10620 If you are or will be a frequent contributor to the Yocto Project
10621 or to OpenEmbedded, you might consider requesting a contrib area
10622 and the necessary associated rights.
10623
106242. *Send the patches via email:* Send the patches to the recipients and
10625 relevant mailing lists by using the ``git send-email`` command.
10626
10627 .. note::
10628
10629 In order to use ``git send-email``, you must have the proper Git packages
10630 installed on your host.
10631 For Ubuntu, Debian, and Fedora the package is ``git-email``.
10632
10633 The ``git send-email`` command sends email by using a local or remote
10634 Mail Transport Agent (MTA) such as ``msmtp``, ``sendmail``, or
10635 through a direct ``smtp`` configuration in your Git ``~/.gitconfig``
10636 file. If you are submitting patches through email only, it is very
10637 important that you submit them without any whitespace or HTML
10638 formatting that either you or your mailer introduces. The maintainer
10639 that receives your patches needs to be able to save and apply them
10640 directly from your emails. A good way to verify that what you are
10641 sending will be applicable by the maintainer is to do a dry run and
10642 send them to yourself and then save and apply them as the maintainer
10643 would.
10644
10645 The ``git send-email`` command is the preferred method for sending
10646 your patches using email since there is no risk of compromising
10647 whitespace in the body of the message, which can occur when you use
10648 your own mail client. The command also has several options that let
10649 you specify recipients and perform further editing of the email
10650 message. For information on how to use the ``git send-email``
10651 command, see ``GIT-SEND-EMAIL(1)`` displayed using the
10652 ``man git-send-email`` command.
10653
10654The Yocto Project uses a `Patchwork instance <https://patchwork.openembedded.org/>`__
10655to track the status of patches submitted to the various mailing lists and to
10656support automated patch testing. Each submitted patch is checked for common
10657mistakes and deviations from the expected patch format and submitters are
10658notified by patchtest if such mistakes are found. This process helps to
10659reduce the burden of patch review on maintainers.
10660
10661.. note::
10662
10663 This system is imperfect and changes can sometimes get lost in the flow.
10664 Asking about the status of a patch or change is reasonable if the change
10665 has been idle for a while with no feedback.
10666
10667Using Scripts to Push a Change Upstream and Request a Pull
10668~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10669
10670For larger patch series it is preferable to send a pull request which not
10671only includes the patch but also a pointer to a branch that can be pulled
10672from. This involves making a local branch for your changes, pushing this
10673branch to an accessible repository and then using the ``create-pull-request``
10674and ``send-pull-request`` scripts from openembedded-core to create and send a
10675patch series with a link to the branch for review.
10676
10677Follow this procedure to push a change to an upstream "contrib" Git
10678repository once the steps in :ref:`dev-manual/common-tasks:preparing changes for submission` have
10679been followed:
10680
10681.. note::
10682
10683 You can find general Git information on how to push a change upstream
10684 in the
10685 `Git Community Book <https://git-scm.com/book/en/v2/Distributed-Git-Distributed-Workflows>`__.
10686
106871. *Push Your Commits to a "Contrib" Upstream:* If you have arranged for
10688 permissions to push to an upstream contrib repository, push the
10689 change to that repository:
10690 ::
10691
10692 $ git push upstream_remote_repo local_branch_name
10693
10694 For example, suppose you have permissions to push
10695 into the upstream ``meta-intel-contrib`` repository and you are
10696 working in a local branch named `your_name`\ ``/README``. The following
10697 command pushes your local commits to the ``meta-intel-contrib``
10698 upstream repository and puts the commit in a branch named
10699 `your_name`\ ``/README``:
10700 ::
10701
10702 $ git push meta-intel-contrib your_name/README
10703
107042. *Determine Who to Notify:* Determine the maintainer or the mailing
10705 list that you need to notify for the change.
10706
10707 Before submitting any change, you need to be sure who the maintainer
10708 is or what mailing list that you need to notify. Use either these
10709 methods to find out:
10710
10711 - *Maintenance File:* Examine the ``maintainers.inc`` file, which is
10712 located in the :term:`Source Directory` at
10713 ``meta/conf/distro/include``, to see who is responsible for code.
10714
10715 - *Search by File:* Using :ref:`overview-manual/development-environment:git`, you can
10716 enter the following command to bring up a short list of all
10717 commits against a specific file:
10718 ::
10719
10720 git shortlog -- filename
10721
10722 Just provide the name of the file for which you are interested. The
10723 information returned is not ordered by history but does include a
10724 list of everyone who has committed grouped by name. From the list,
10725 you can see who is responsible for the bulk of the changes against
10726 the file.
10727
10728 - *Examine the List of Mailing Lists:* For a list of the Yocto
10729 Project and related mailing lists, see the ":ref:`Mailing
10730 lists <resources-mailinglist>`" section in
10731 the Yocto Project Reference Manual.
10732
107333. *Make a Pull Request:* Notify the maintainer or the mailing list that
10734 you have pushed a change by making a pull request.
10735
10736 The Yocto Project provides two scripts that conveniently let you
10737 generate and send pull requests to the Yocto Project. These scripts
10738 are ``create-pull-request`` and ``send-pull-request``. You can find
10739 these scripts in the ``scripts`` directory within the
10740 :term:`Source Directory` (e.g.
10741 ``~/poky/scripts``).
10742
10743 Using these scripts correctly formats the requests without
10744 introducing any whitespace or HTML formatting. The maintainer that
10745 receives your patches either directly or through the mailing list
10746 needs to be able to save and apply them directly from your emails.
10747 Using these scripts is the preferred method for sending patches.
10748
10749 First, create the pull request. For example, the following command
10750 runs the script, specifies the upstream repository in the contrib
10751 directory into which you pushed the change, and provides a subject
10752 line in the created patch files:
10753 ::
10754
10755 $ ~/poky/scripts/create-pull-request -u meta-intel-contrib -s "Updated Manual Section Reference in README"
10756
10757 Running this script forms ``*.patch`` files in a folder named
10758 ``pull-``\ `PID` in the current directory. One of the patch files is a
10759 cover letter.
10760
10761 Before running the ``send-pull-request`` script, you must edit the
10762 cover letter patch to insert information about your change. After
10763 editing the cover letter, send the pull request. For example, the
10764 following command runs the script and specifies the patch directory
10765 and email address. In this example, the email address is a mailing
10766 list:
10767 ::
10768
10769 $ ~/poky/scripts/send-pull-request -p ~/meta-intel/pull-10565 -t meta-intel@yoctoproject.org
10770
10771 You need to follow the prompts as the script is interactive.
10772
10773 .. note::
10774
10775 For help on using these scripts, simply provide the ``-h``
10776 argument as follows:
10777 ::
10778
10779 $ poky/scripts/create-pull-request -h
10780 $ poky/scripts/send-pull-request -h
10781
10782Responding to Patch Review
10783~~~~~~~~~~~~~~~~~~~~~~~~~~
10784
10785You may get feedback on your submitted patches from other community members
10786or from the automated patchtest service. If issues are identified in your
10787patch then it is usually necessary to address these before the patch will be
10788accepted into the project. In this case you should amend the patch according
10789to the feedback and submit an updated version to the relevant mailing list,
10790copying in the reviewers who provided feedback to the previous version of the
10791patch.
10792
10793The patch should be amended using ``git commit --amend`` or perhaps ``git
10794rebase`` for more expert git users. You should also modify the ``[PATCH]``
10795tag in the email subject line when sending the revised patch to mark the new
10796iteration as ``[PATCH v2]``, ``[PATCH v3]``, etc as appropriate. This can be
10797done by passing the ``-v`` argument to ``git format-patch`` with a version
10798number.
10799
10800Lastly please ensure that you also test your revised changes. In particular
10801please don't just edit the patch file written out by ``git format-patch`` and
10802resend it.
10803
10804Submitting Changes to Stable Release Branches
10805~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10806
10807The process for proposing changes to a Yocto Project stable branch differs
10808from the steps described above. Changes to a stable branch must address
10809identified bugs or CVEs and should be made carefully in order to avoid the
10810risk of introducing new bugs or breaking backwards compatibility. Typically
10811bug fixes must already be accepted into the master branch before they can be
10812backported to a stable branch unless the bug in question does not affect the
10813master branch or the fix on the master branch is unsuitable for backporting.
10814
10815The list of stable branches along with the status and maintainer for each
10816branch can be obtained from the
10817:yocto_wiki:`Releases wiki page </Releases>`.
10818
10819.. note::
10820
10821 Changes will not typically be accepted for branches which are marked as
10822 End-Of-Life (EOL).
10823
10824With this in mind, the steps to submit a change for a stable branch are as
10825follows:
10826
108271. *Identify the bug or CVE to be fixed:* This information should be
10828 collected so that it can be included in your submission.
10829
108302. *Check if the fix is already present in the master branch:* This will
10831 result in the most straightforward path into the stable branch for the
10832 fix.
10833
10834 a. *If the fix is present in the master branch - Submit a backport request
10835 by email:* You should send an email to the relevant stable branch
10836 maintainer and the mailing list with details of the bug or CVE to be
10837 fixed, the commit hash on the master branch that fixes the issue and
10838 the stable branches which you would like this fix to be backported to.
10839
10840 b. *If the fix is not present in the master branch - Submit the fix to the
10841 master branch first:* This will ensure that the fix passes through the
10842 project's usual patch review and test processes before being accepted.
10843 It will also ensure that bugs are not left unresolved in the master
10844 branch itself. Once the fix is accepted in the master branch a backport
10845 request can be submitted as above.
10846
10847 c. *If the fix is unsuitable for the master branch - Submit a patch
10848 directly for the stable branch:* This method should be considered as a
10849 last resort. It is typically necessary when the master branch is using
10850 a newer version of the software which includes an upstream fix for the
10851 issue or when the issue has been fixed on the master branch in a way
10852 that introduces backwards incompatible changes. In this case follow the
10853 steps in :ref:`dev-manual/common-tasks:preparing changes for submission` and
10854 :ref:`dev-manual/common-tasks:using email to submit a patch` but modify the subject header of your patch
10855 email to include the name of the stable branch which you are
10856 targetting. This can be done using the ``--subject-prefix`` argument to
10857 ``git format-patch``, for example to submit a patch to the dunfell
10858 branch use
10859 ``git format-patch --subject-prefix='&DISTRO_NAME_NO_CAP_MINUS_ONE;][PATCH' ...``.
10860
10861Working With Licenses
10862=====================
10863
10864As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
10865section in the Yocto Project Overview and Concepts Manual, open source
10866projects are open to the public and they consequently have different
10867licensing structures in place. This section describes the mechanism by
10868which the :term:`OpenEmbedded Build System`
10869tracks changes to
10870licensing text and covers how to maintain open source license compliance
10871during your project's lifecycle. The section also describes how to
10872enable commercially licensed recipes, which by default are disabled.
10873
10874Tracking License Changes
10875------------------------
10876
10877The license of an upstream project might change in the future. In order
10878to prevent these changes going unnoticed, the
10879:term:`LIC_FILES_CHKSUM`
10880variable tracks changes to the license text. The checksums are validated
10881at the end of the configure step, and if the checksums do not match, the
10882build will fail.
10883
10884Specifying the ``LIC_FILES_CHKSUM`` Variable
10885~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
10886
10887The ``LIC_FILES_CHKSUM`` variable contains checksums of the license text
10888in the source code for the recipe. Following is an example of how to
10889specify ``LIC_FILES_CHKSUM``:
10890::
10891
10892 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
10893 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
10894 file://licfile2.txt;endline=50;md5=zzzz \
10895 ..."
10896
10897.. note::
10898
10899 - When using "beginline" and "endline", realize that line numbering
10900 begins with one and not zero. Also, the included lines are
10901 inclusive (i.e. lines five through and including 29 in the
10902 previous example for ``licfile1.txt``).
10903
10904 - When a license check fails, the selected license text is included
10905 as part of the QA message. Using this output, you can determine
10906 the exact start and finish for the needed license text.
10907
10908The build system uses the :term:`S`
10909variable as the default directory when searching files listed in
10910``LIC_FILES_CHKSUM``. The previous example employs the default
10911directory.
10912
10913Consider this next example:
10914::
10915
10916 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
10917 md5=bb14ed3c4cda583abc85401304b5cd4e"
10918 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
10919
10920The first line locates a file in ``${S}/src/ls.c`` and isolates lines
10921five through 16 as license text. The second line refers to a file in
10922:term:`WORKDIR`.
10923
10924Note that ``LIC_FILES_CHKSUM`` variable is mandatory for all recipes,
10925unless the ``LICENSE`` variable is set to "CLOSED".
10926
10927Explanation of Syntax
10928~~~~~~~~~~~~~~~~~~~~~
10929
10930As mentioned in the previous section, the ``LIC_FILES_CHKSUM`` variable
10931lists all the important files that contain the license text for the
10932source code. It is possible to specify a checksum for an entire file, or
10933a specific section of a file (specified by beginning and ending line
10934numbers with the "beginline" and "endline" parameters, respectively).
10935The latter is useful for source files with a license notice header,
10936README documents, and so forth. If you do not use the "beginline"
10937parameter, then it is assumed that the text begins on the first line of
10938the file. Similarly, if you do not use the "endline" parameter, it is
10939assumed that the license text ends with the last line of the file.
10940
10941The "md5" parameter stores the md5 checksum of the license text. If the
10942license text changes in any way as compared to this parameter then a
10943mismatch occurs. This mismatch triggers a build failure and notifies the
10944developer. Notification allows the developer to review and address the
10945license text changes. Also note that if a mismatch occurs during the
10946build, the correct md5 checksum is placed in the build log and can be
10947easily copied to the recipe.
10948
10949There is no limit to how many files you can specify using the
10950``LIC_FILES_CHKSUM`` variable. Generally, however, every project
10951requires a few specifications for license tracking. Many projects have a
10952"COPYING" file that stores the license information for all the source
10953code files. This practice allows you to just track the "COPYING" file as
10954long as it is kept up to date.
10955
10956.. note::
10957
10958 - If you specify an empty or invalid "md5" parameter,
10959 :term:`BitBake` returns an md5
10960 mis-match error and displays the correct "md5" parameter value
10961 during the build. The correct parameter is also captured in the
10962 build log.
10963
10964 - If the whole file contains only license text, you do not need to
10965 use the "beginline" and "endline" parameters.
10966
10967Enabling Commercially Licensed Recipes
10968--------------------------------------
10969
10970By default, the OpenEmbedded build system disables components that have
10971commercial or other special licensing requirements. Such requirements
10972are defined on a recipe-by-recipe basis through the
10973:term:`LICENSE_FLAGS` variable
10974definition in the affected recipe. For instance, the
10975``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
10976contains the following statement:
10977::
10978
10979 LICENSE_FLAGS = "commercial"
10980
10981Here is a
10982slightly more complicated example that contains both an explicit recipe
10983name and version (after variable expansion):
10984::
10985
10986 LICENSE_FLAGS = "license_${PN}_${PV}"
10987
10988In order for a component restricted by a
10989``LICENSE_FLAGS`` definition to be enabled and included in an image, it
10990needs to have a matching entry in the global
10991:term:`LICENSE_FLAGS_WHITELIST`
10992variable, which is a variable typically defined in your ``local.conf``
10993file. For example, to enable the
10994``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
10995could add either the string "commercial_gst-plugins-ugly" or the more
10996general string "commercial" to ``LICENSE_FLAGS_WHITELIST``. See the
10997"`License Flag Matching <#license-flag-matching>`__" section for a full
10998explanation of how ``LICENSE_FLAGS`` matching works. Here is the
10999example:
11000::
11001
11002 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly"
11003
11004Likewise, to additionally enable the package built from the recipe
11005containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
11006the actual recipe name was ``emgd_1.10.bb``, the following string would
11007enable that package as well as the original ``gst-plugins-ugly``
11008package:
11009::
11010
11011 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly license_emgd_1.10"
11012
11013As a convenience, you do not need to specify the
11014complete license string in the whitelist for every package. You can use
11015an abbreviated form, which consists of just the first portion or
11016portions of the license string before the initial underscore character
11017or characters. A partial string will match any license that contains the
11018given string as the first portion of its license. For example, the
11019following whitelist string will also match both of the packages
11020previously mentioned as well as any other packages that have licenses
11021starting with "commercial" or "license".
11022::
11023
11024 LICENSE_FLAGS_WHITELIST = "commercial license"
11025
11026License Flag Matching
11027~~~~~~~~~~~~~~~~~~~~~
11028
11029License flag matching allows you to control what recipes the
11030OpenEmbedded build system includes in the build. Fundamentally, the
11031build system attempts to match ``LICENSE_FLAGS`` strings found in
11032recipes against ``LICENSE_FLAGS_WHITELIST`` strings found in the
11033whitelist. A match causes the build system to include a recipe in the
11034build, while failure to find a match causes the build system to exclude
11035a recipe.
11036
11037In general, license flag matching is simple. However, understanding some
11038concepts will help you correctly and effectively use matching.
11039
11040Before a flag defined by a particular recipe is tested against the
11041contents of the whitelist, the expanded string ``_${PN}`` is appended to
11042the flag. This expansion makes each ``LICENSE_FLAGS`` value
11043recipe-specific. After expansion, the string is then matched against the
11044whitelist. Thus, specifying ``LICENSE_FLAGS = "commercial"`` in recipe
11045"foo", for example, results in the string ``"commercial_foo"``. And, to
11046create a match, that string must appear in the whitelist.
11047
11048Judicious use of the ``LICENSE_FLAGS`` strings and the contents of the
11049``LICENSE_FLAGS_WHITELIST`` variable allows you a lot of flexibility for
11050including or excluding recipes based on licensing. For example, you can
11051broaden the matching capabilities by using license flags string subsets
11052in the whitelist.
11053
11054.. note::
11055
11056 When using a string subset, be sure to use the part of the expanded
11057 string that precedes the appended underscore character (e.g.
11058 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
11059
11060For example, simply specifying the string "commercial" in the whitelist
11061matches any expanded ``LICENSE_FLAGS`` definition that starts with the
11062string "commercial" such as "commercial_foo" and "commercial_bar", which
11063are the strings the build system automatically generates for
11064hypothetical recipes named "foo" and "bar" assuming those recipes simply
11065specify the following:
11066::
11067
11068 LICENSE_FLAGS = "commercial"
11069
11070Thus, you can choose
11071to exhaustively enumerate each license flag in the whitelist and allow
11072only specific recipes into the image, or you can use a string subset
11073that causes a broader range of matches to allow a range of recipes into
11074the image.
11075
11076This scheme works even if the ``LICENSE_FLAGS`` string already has
11077``_${PN}`` appended. For example, the build system turns the license
11078flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
11079both the general "commercial" and the specific "commercial_1.2_foo"
11080strings found in the whitelist, as expected.
11081
11082Here are some other scenarios:
11083
11084- You can specify a versioned string in the recipe such as
11085 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
11086 string to "commercial_foo_1.2_foo". Combine this license flag with a
11087 whitelist that has the string "commercial" and you match the flag
11088 along with any other flag that starts with the string "commercial".
11089
11090- Under the same circumstances, you can use "commercial_foo" in the
11091 whitelist and the build system not only matches "commercial_foo_1.2"
11092 but also matches any license flag with the string "commercial_foo",
11093 regardless of the version.
11094
11095- You can be very specific and use both the package and version parts
11096 in the whitelist (e.g. "commercial_foo_1.2") to specifically match a
11097 versioned recipe.
11098
11099Other Variables Related to Commercial Licenses
11100~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11101
11102Other helpful variables related to commercial license handling exist and
11103are defined in the
11104``poky/meta/conf/distro/include/default-distrovars.inc`` file:
11105::
11106
11107 COMMERCIAL_AUDIO_PLUGINS ?= ""
11108 COMMERCIAL_VIDEO_PLUGINS ?= ""
11109
11110If you
11111want to enable these components, you can do so by making sure you have
11112statements similar to the following in your ``local.conf`` configuration
11113file:
11114::
11115
11116 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
11117 gst-plugins-ugly-mpegaudioparse"
11118 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
11119 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
11120 LICENSE_FLAGS_WHITELIST = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
11121
11122
11123Of course, you could also create a matching whitelist for those
11124components using the more general "commercial" in the whitelist, but
11125that would also enable all the other packages with ``LICENSE_FLAGS``
11126containing "commercial", which you may or may not want:
11127::
11128
11129 LICENSE_FLAGS_WHITELIST = "commercial"
11130
11131Specifying audio and video plugins as part of the
11132``COMMERCIAL_AUDIO_PLUGINS`` and ``COMMERCIAL_VIDEO_PLUGINS`` statements
11133(along with the enabling ``LICENSE_FLAGS_WHITELIST``) includes the
11134plugins or components into built images, thus adding support for media
11135formats or components.
11136
11137Maintaining Open Source License Compliance During Your Product's Lifecycle
11138--------------------------------------------------------------------------
11139
11140One of the concerns for a development organization using open source
11141software is how to maintain compliance with various open source
11142licensing during the lifecycle of the product. While this section does
11143not provide legal advice or comprehensively cover all scenarios, it does
11144present methods that you can use to assist you in meeting the compliance
11145requirements during a software release.
11146
11147With hundreds of different open source licenses that the Yocto Project
11148tracks, it is difficult to know the requirements of each and every
11149license. However, the requirements of the major FLOSS licenses can begin
11150to be covered by assuming that three main areas of concern exist:
11151
11152- Source code must be provided.
11153
11154- License text for the software must be provided.
11155
11156- Compilation scripts and modifications to the source code must be
11157 provided.
11158
11159- spdx files can be provided.
11160
11161There are other requirements beyond the scope of these three and the
11162methods described in this section (e.g. the mechanism through which
11163source code is distributed).
11164
11165As different organizations have different methods of complying with open
11166source licensing, this section is not meant to imply that there is only
11167one single way to meet your compliance obligations, but rather to
11168describe one method of achieving compliance. The remainder of this
11169section describes methods supported to meet the previously mentioned
11170three requirements. Once you take steps to meet these requirements, and
11171prior to releasing images, sources, and the build system, you should
11172audit all artifacts to ensure completeness.
11173
11174.. note::
11175
11176 The Yocto Project generates a license manifest during image creation
11177 that is located in ``${DEPLOY_DIR}/licenses/``\ `image_name`\ ``-``\ `datestamp`
11178 to assist with any audits.
11179
11180Providing the Source Code
11181~~~~~~~~~~~~~~~~~~~~~~~~~
11182
11183Compliance activities should begin before you generate the final image.
11184The first thing you should look at is the requirement that tops the list
11185for most compliance groups - providing the source. The Yocto Project has
11186a few ways of meeting this requirement.
11187
11188One of the easiest ways to meet this requirement is to provide the
11189entire :term:`DL_DIR` used by the
11190build. This method, however, has a few issues. The most obvious is the
11191size of the directory since it includes all sources used in the build
11192and not just the source used in the released image. It will include
11193toolchain source, and other artifacts, which you would not generally
11194release. However, the more serious issue for most companies is
11195accidental release of proprietary software. The Yocto Project provides
11196an :ref:`archiver <ref-classes-archiver>` class to
11197help avoid some of these concerns.
11198
11199Before you employ ``DL_DIR`` or the ``archiver`` class, you need to
11200decide how you choose to provide source. The source ``archiver`` class
11201can generate tarballs and SRPMs and can create them with various levels
11202of compliance in mind.
11203
11204One way of doing this (but certainly not the only way) is to release
11205just the source as a tarball. You can do this by adding the following to
11206the ``local.conf`` file found in the
11207:term:`Build Directory`:
11208::
11209
11210 INHERIT += "archiver"
11211 ARCHIVER_MODE[src] = "original"
11212
11213During the creation of your
11214image, the source from all recipes that deploy packages to the image is
11215placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
11216:term:`LICENSE` for each recipe.
11217Releasing the entire directory enables you to comply with requirements
11218concerning providing the unmodified source. It is important to note that
11219the size of the directory can get large.
11220
11221A way to help mitigate the size issue is to only release tarballs for
11222licenses that require the release of source. Let us assume you are only
11223concerned with GPL code as identified by running the following script:
11224
11225.. code-block:: shell
11226
11227 # Script to archive a subset of packages matching specific license(s)
11228 # Source and license files are copied into sub folders of package folder
11229 # Must be run from build folder
11230 #!/bin/bash
11231 src_release_dir="source-release"
11232 mkdir -p $src_release_dir
11233 for a in tmp/deploy/sources/*; do
11234 for d in $a/*; do
11235 # Get package name from path
11236 p=`basename $d`
11237 p=${p%-*}
11238 p=${p%-*}
11239 # Only archive GPL packages (update *GPL* regex for your license check)
11240 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
11241 if [ $numfiles -gt 1 ]; then
11242 echo Archiving $p
11243 mkdir -p $src_release_dir/$p/source
11244 cp $d/* $src_release_dir/$p/source 2> /dev/null
11245 mkdir -p $src_release_dir/$p/license
11246 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
11247 fi
11248 done
11249 done
11250
11251At this point, you
11252could create a tarball from the ``gpl_source_release`` directory and
11253provide that to the end user. This method would be a step toward
11254achieving compliance with section 3a of GPLv2 and with section 6 of
11255GPLv3.
11256
11257Providing License Text
11258~~~~~~~~~~~~~~~~~~~~~~
11259
11260One requirement that is often overlooked is inclusion of license text.
11261This requirement also needs to be dealt with prior to generating the
11262final image. Some licenses require the license text to accompany the
11263binary. You can achieve this by adding the following to your
11264``local.conf`` file:
11265::
11266
11267 COPY_LIC_MANIFEST = "1"
11268 COPY_LIC_DIRS = "1"
11269 LICENSE_CREATE_PACKAGE = "1"
11270
11271Adding these statements to the
11272configuration file ensures that the licenses collected during package
11273generation are included on your image.
11274
11275.. note::
11276
11277 Setting all three variables to "1" results in the image having two
11278 copies of the same license file. One copy resides in
11279 ``/usr/share/common-licenses`` and the other resides in
11280 ``/usr/share/license``.
11281
11282 The reason for this behavior is because
11283 :term:`COPY_LIC_DIRS` and
11284 :term:`COPY_LIC_MANIFEST`
11285 add a copy of the license when the image is built but do not offer a
11286 path for adding licenses for newly installed packages to an image.
11287 :term:`LICENSE_CREATE_PACKAGE`
11288 adds a separate package and an upgrade path for adding licenses to an
11289 image.
11290
11291As the source ``archiver`` class has already archived the original
11292unmodified source that contains the license files, you would have
11293already met the requirements for inclusion of the license information
11294with source as defined by the GPL and other open source licenses.
11295
11296Providing Compilation Scripts and Source Code Modifications
11297~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11298
11299At this point, we have addressed all we need to prior to generating the
11300image. The next two requirements are addressed during the final
11301packaging of the release.
11302
11303By releasing the version of the OpenEmbedded build system and the layers
11304used during the build, you will be providing both compilation scripts
11305and the source code modifications in one step.
11306
11307If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
11308and a distro layer, and those
11309those layers are used to patch, compile, package, or modify (in any way)
11310any open source software included in your released images, you might be
11311required to release those layers under section 3 of GPLv2 or section 1
11312of GPLv3. One way of doing that is with a clean checkout of the version
11313of the Yocto Project and layers used during your build. Here is an
11314example:
11315
11316.. code-block:: shell
11317
11318 # We built using the dunfell branch of the poky repo
11319 $ git clone -b dunfell git://git.yoctoproject.org/poky
11320 $ cd poky
11321 # We built using the release_branch for our layers
11322 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
11323 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
11324 # clean up the .git repos
11325 $ find . -name ".git" -type d -exec rm -rf {} \;
11326
11327One
11328thing a development organization might want to consider for end-user
11329convenience is to modify ``meta-poky/conf/bblayers.conf.sample`` to
11330ensure that when the end user utilizes the released build system to
11331build an image, the development organization's layers are included in
11332the ``bblayers.conf`` file automatically:
11333::
11334
11335 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
11336 # changes incompatibly
11337 POKY_BBLAYERS_CONF_VERSION = "2"
11338
11339 BBPATH = "${TOPDIR}"
11340 BBFILES ?= ""
11341
11342 BBLAYERS ?= " \
11343 ##OEROOT##/meta \
11344 ##OEROOT##/meta-poky \
11345 ##OEROOT##/meta-yocto-bsp \
11346 ##OEROOT##/meta-mylayer \
11347 "
11348
11349Creating and
11350providing an archive of the :term:`Metadata`
11351layers (recipes, configuration files, and so forth) enables you to meet
11352your requirements to include the scripts to control compilation as well
11353as any modifications to the original source.
11354
11355Providing spdx files
11356~~~~~~~~~~~~~~~~~~~~~~~~~
11357
11358The spdx module has been integrated to a layer named meta-spdxscanner.
11359meta-spdxscanner provides several kinds of scanner. If you want to enable
11360this function, you have to follow the following steps:
11361
113621. Add meta-spdxscanner layer into ``bblayers.conf``.
11363
113642. Refer to the README in meta-spdxscanner to setup the environment (e.g,
11365 setup a fossology server) needed for the scanner.
11366
113673. Meta-spdxscanner provides several methods within the bbclass to create spdx files.
11368 Please choose one that you want to use and enable the spdx task. You have to
11369 add some config options in ``local.conf`` file in your :term:`Build
11370 Directory`. The following is an example showing how to generate spdx files
11371 during bitbake using the fossology-python.bbclass::
11372
11373 # Select fossology-python.bbclass.
11374 INHERIT += "fossology-python"
11375 # For fossology-python.bbclass, TOKEN is necessary, so, after setup a
11376 # Fossology server, you have to create a token.
11377 TOKEN = "eyJ0eXAiO..."
11378 # The fossology server is necessary for fossology-python.bbclass.
11379 FOSSOLOGY_SERVER = "http://xx.xx.xx.xx:8081/repo"
11380 # If you want to upload the source code to a special folder:
11381 FOLDER_NAME = "xxxx" //Optional
11382 # If you don't want to put spdx files in tmp/deploy/spdx, you can enable:
11383 SPDX_DEPLOY_DIR = "${DEPLOY_DIR}" //Optional
11384
11385For more usage information refer to :yocto_git:`the meta-spdxscanner repository
11386</meta-spdxscanner/>`.
11387
11388
11389Copying Licenses that Do Not Exist
11390----------------------------------
11391
11392Some packages, such as the linux-firmware package, have many licenses
11393that are not in any way common. You can avoid adding a lot of these
11394types of common license files, which are only applicable to a specific
11395package, by using the
11396:term:`NO_GENERIC_LICENSE`
11397variable. Using this variable also avoids QA errors when you use a
11398non-common, non-CLOSED license in a recipe.
11399
11400The following is an example that uses the ``LICENSE.Abilis.txt`` file as
11401the license from the fetched source:
11402::
11403
11404 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
11405
11406Using the Error Reporting Tool
11407==============================
11408
11409The error reporting tool allows you to submit errors encountered during
11410builds to a central database. Outside of the build environment, you can
11411use a web interface to browse errors, view statistics, and query for
11412errors. The tool works using a client-server system where the client
11413portion is integrated with the installed Yocto Project
11414:term:`Source Directory` (e.g. ``poky``).
11415The server receives the information collected and saves it in a
11416database.
11417
11418A live instance of the error reporting server exists at
11419https://errors.yoctoproject.org. This server exists so that when
11420you want to get help with build failures, you can submit all of the
11421information on the failure easily and then point to the URL in your bug
11422report or send an email to the mailing list.
11423
11424.. note::
11425
11426 If you send error reports to this server, the reports become publicly
11427 visible.
11428
11429Enabling and Using the Tool
11430---------------------------
11431
11432By default, the error reporting tool is disabled. You can enable it by
11433inheriting the
11434:ref:`report-error <ref-classes-report-error>`
11435class by adding the following statement to the end of your
11436``local.conf`` file in your
11437:term:`Build Directory`.
11438::
11439
11440 INHERIT += "report-error"
11441
11442By default, the error reporting feature stores information in
11443``${``\ :term:`LOG_DIR`\ ``}/error-report``.
11444However, you can specify a directory to use by adding the following to
11445your ``local.conf`` file:
11446::
11447
11448 ERR_REPORT_DIR = "path"
11449
11450Enabling error
11451reporting causes the build process to collect the errors and store them
11452in a file as previously described. When the build system encounters an
11453error, it includes a command as part of the console output. You can run
11454the command to send the error file to the server. For example, the
11455following command sends the errors to an upstream server:
11456::
11457
11458 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
11459
11460In the previous example, the errors are sent to a public database
11461available at https://errors.yoctoproject.org, which is used by the
11462entire community. If you specify a particular server, you can send the
11463errors to a different database. Use the following command for more
11464information on available options:
11465::
11466
11467 $ send-error-report --help
11468
11469When sending the error file, you are prompted to review the data being
11470sent as well as to provide a name and optional email address. Once you
11471satisfy these prompts, the command returns a link from the server that
11472corresponds to your entry in the database. For example, here is a
11473typical link: https://errors.yoctoproject.org/Errors/Details/9522/
11474
11475Following the link takes you to a web interface where you can browse,
11476query the errors, and view statistics.
11477
11478Disabling the Tool
11479------------------
11480
11481To disable the error reporting feature, simply remove or comment out the
11482following statement from the end of your ``local.conf`` file in your
11483:term:`Build Directory`.
11484::
11485
11486 INHERIT += "report-error"
11487
11488Setting Up Your Own Error Reporting Server
11489------------------------------------------
11490
11491If you want to set up your own error reporting server, you can obtain
11492the code from the Git repository at :yocto_git:`/error-report-web/`.
11493Instructions on how to set it up are in the README document.
11494
11495Using Wayland and Weston
11496========================
11497
11498`Wayland <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)>`__
11499is a computer display server protocol that provides a method for
11500compositing window managers to communicate directly with applications
11501and video hardware and expects them to communicate with input hardware
11502using other libraries. Using Wayland with supporting targets can result
11503in better control over graphics frame rendering than an application
11504might otherwise achieve.
11505
11506The Yocto Project provides the Wayland protocol libraries and the
11507reference
11508`Weston <https://en.wikipedia.org/wiki/Wayland_(display_server_protocol)#Weston>`__
11509compositor as part of its release. You can find the integrated packages
11510in the ``meta`` layer of the :term:`Source Directory`.
11511Specifically, you
11512can find the recipes that build both Wayland and Weston at
11513``meta/recipes-graphics/wayland``.
11514
11515You can build both the Wayland and Weston packages for use only with
11516targets that accept the `Mesa 3D and Direct Rendering
11517Infrastructure <https://en.wikipedia.org/wiki/Mesa_(computer_graphics)>`__,
11518which is also known as Mesa DRI. This implies that you cannot build and
11519use the packages if your target uses, for example, the Intel Embedded
11520Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
11521
11522.. note::
11523
11524 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
11525 emulated QEMU hardware. However, this version of Weston will run
11526 under X emulation without issues.
11527
11528This section describes what you need to do to implement Wayland and use
11529the Weston compositor when building an image for a supporting target.
11530
11531Enabling Wayland in an Image
11532----------------------------
11533
11534To enable Wayland, you need to enable it to be built and enable it to be
11535included (installed) in the image.
11536
11537Building Wayland
11538~~~~~~~~~~~~~~~~
11539
11540To cause Mesa to build the ``wayland-egl`` platform and Weston to build
11541Wayland with Kernel Mode Setting
11542(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
11543support, include the "wayland" flag in the
11544:term:`DISTRO_FEATURES`
11545statement in your ``local.conf`` file:
11546::
11547
11548 DISTRO_FEATURES_append = " wayland"
11549
11550.. note::
11551
11552 If X11 has been enabled elsewhere, Weston will build Wayland with X11
11553 support
11554
11555Installing Wayland and Weston
11556~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11557
11558To install the Wayland feature into an image, you must include the
11559following
11560:term:`CORE_IMAGE_EXTRA_INSTALL`
11561statement in your ``local.conf`` file:
11562::
11563
11564 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
11565
11566Running Weston
11567--------------
11568
11569To run Weston inside X11, enabling it as described earlier and building
11570a Sato image is sufficient. If you are running your image under Sato, a
11571Weston Launcher appears in the "Utility" category.
11572
11573Alternatively, you can run Weston through the command-line interpretor
11574(CLI), which is better suited for development work. To run Weston under
11575the CLI, you need to do the following after your image is built:
11576
115771. Run these commands to export ``XDG_RUNTIME_DIR``:
11578 ::
11579
11580 mkdir -p /tmp/$USER-weston
11581 chmod 0700 /tmp/$USER-weston
11582 export XDG_RUNTIME_DIR=/tmp/$USER-weston
11583
115842. Launch Weston in the shell:
11585 ::
11586
11587 weston
diff --git a/documentation/dev-manual/custom-distribution.rst b/documentation/dev-manual/custom-distribution.rst
new file mode 100644
index 0000000000..0bc386d606
--- /dev/null
+++ b/documentation/dev-manual/custom-distribution.rst
@@ -0,0 +1,135 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating Your Own Distribution
4******************************
5
6When you build an image using the Yocto Project and do not alter any
7distribution :term:`Metadata`, you are using the Poky distribution.
8Poky is explicitly a *reference* distribution for testing and
9development purposes. It enables most hardware and software features
10so that they can be tested, but this also means that from a security
11point of view the attack surface is very large. Additionally, at some
12point it is likely that you will want to gain more control over package
13alternative selections, compile-time options, and other low-level
14configurations. For both of these reasons, if you are using the Yocto
15Project for production use then you are strongly encouraged to create
16your own distribution.
17
18To create your own distribution, the basic steps consist of creating
19your own distribution layer, creating your own distribution
20configuration file, and then adding any needed code and Metadata to the
21layer. The following steps provide some more detail:
22
23- *Create a layer for your new distro:* Create your distribution layer
24 so that you can keep your Metadata and code for the distribution
25 separate. It is strongly recommended that you create and use your own
26 layer for configuration and code. Using your own layer as compared to
27 just placing configurations in a ``local.conf`` configuration file
28 makes it easier to reproduce the same build configuration when using
29 multiple build machines. See the
30 ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
31 section for information on how to quickly set up a layer.
32
33- *Create the distribution configuration file:* The distribution
34 configuration file needs to be created in the ``conf/distro``
35 directory of your layer. You need to name it using your distribution
36 name (e.g. ``mydistro.conf``).
37
38 .. note::
39
40 The :term:`DISTRO` variable in your ``local.conf`` file determines the
41 name of your distribution.
42
43 You can split out parts of your configuration file into include files
44 and then "require" them from within your distribution configuration
45 file. Be sure to place the include files in the
46 ``conf/distro/include`` directory of your layer. A common example
47 usage of include files would be to separate out the selection of
48 desired version and revisions for individual recipes.
49
50 Your configuration file needs to set the following required
51 variables:
52
53 - :term:`DISTRO_NAME`
54
55 - :term:`DISTRO_VERSION`
56
57 These following variables are optional and you typically set them
58 from the distribution configuration file:
59
60 - :term:`DISTRO_FEATURES`
61
62 - :term:`DISTRO_EXTRA_RDEPENDS`
63
64 - :term:`DISTRO_EXTRA_RRECOMMENDS`
65
66 - :term:`TCLIBC`
67
68 .. tip::
69
70 If you want to base your distribution configuration file on the
71 very basic configuration from OE-Core, you can use
72 ``conf/distro/defaultsetup.conf`` as a reference and just include
73 variables that differ as compared to ``defaultsetup.conf``.
74 Alternatively, you can create a distribution configuration file
75 from scratch using the ``defaultsetup.conf`` file or configuration files
76 from another distribution such as Poky as a reference.
77
78- *Provide miscellaneous variables:* Be sure to define any other
79 variables for which you want to create a default or enforce as part
80 of the distribution configuration. You can include nearly any
81 variable from the ``local.conf`` file. The variables you use are not
82 limited to the list in the previous bulleted item.
83
84- *Point to Your distribution configuration file:* In your ``local.conf``
85 file in the :term:`Build Directory`, set your :term:`DISTRO` variable to
86 point to your distribution's configuration file. For example, if your
87 distribution's configuration file is named ``mydistro.conf``, then
88 you point to it as follows::
89
90 DISTRO = "mydistro"
91
92- *Add more to the layer if necessary:* Use your layer to hold other
93 information needed for the distribution:
94
95 - Add recipes for installing distro-specific configuration files
96 that are not already installed by another recipe. If you have
97 distro-specific configuration files that are included by an
98 existing recipe, you should add an append file (``.bbappend``) for
99 those. For general information and recommendations on how to add
100 recipes to your layer, see the
101 ":ref:`dev-manual/layers:creating your own layer`" and
102 ":ref:`dev-manual/layers:following best practices when creating layers`"
103 sections.
104
105 - Add any image recipes that are specific to your distribution.
106
107 - Add a ``psplash`` append file for a branded splash screen, using
108 the :term:`SPLASH_IMAGES` variable.
109
110 - Add any other append files to make custom changes that are
111 specific to individual recipes.
112
113 For information on append files, see the
114 ":ref:`dev-manual/layers:appending other layers metadata with your layer`"
115 section.
116
117Copying and modifying the Poky distribution
118===========================================
119
120Instead of creating a custom distribution from scratch as per above, you may
121wish to start your custom distribution configuration by copying the Poky
122distribution provided within the ``meta-poky`` layer and then modifying it.
123This is fine, however if you do this you should keep the following in mind:
124
125- Every reference to Poky needs to be updated in your copy so that it
126 will still apply. This includes override usage within files (e.g. ``:poky``)
127 and in directory names. This is a good opportunity to evaluate each one of
128 these customizations to see if they are needed for your use case.
129
130- Unless you also intend to use them, the ``poky-tiny``, ``poky-altcfg`` and
131 ``poky-bleeding`` variants and any references to them can be removed.
132
133- More generally, the Poky distribution configuration enables a lot more
134 than you likely need for your production use case. You should evaluate *every*
135 configuration choice made in your copy to determine if it is needed.
diff --git a/documentation/dev-manual/custom-template-configuration-directory.rst b/documentation/dev-manual/custom-template-configuration-directory.rst
new file mode 100644
index 0000000000..06fcada822
--- /dev/null
+++ b/documentation/dev-manual/custom-template-configuration-directory.rst
@@ -0,0 +1,52 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Custom Template Configuration Directory
4**************************************************
5
6If you are producing your own customized version of the build system for
7use by other users, you might want to provide a custom build configuration
8that includes all the necessary settings and layers (i.e. ``local.conf`` and
9``bblayers.conf`` that are created in a new :term:`Build Directory`) and a custom
10message that is shown when setting up the build. This can be done by
11creating one or more template configuration directories in your
12custom distribution layer.
13
14This can be done by using ``bitbake-layers save-build-conf``::
15
16 $ bitbake-layers save-build-conf ../../meta-alex/ test-1
17 NOTE: Starting bitbake server...
18 NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
19 Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
20 You can try out the configuration with
21 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
22
23The above command takes the config files from the currently active :term:`Build Directory` under ``conf``,
24replaces site-specific paths in ``bblayers.conf`` with ``##OECORE##``-relative paths, and copies
25the config files into a specified layer under a specified template name.
26
27To use those saved templates as a starting point for a build, users should point
28to one of them with :term:`TEMPLATECONF` environment variable::
29
30 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
31
32The OpenEmbedded build system uses the environment variable
33:term:`TEMPLATECONF` to locate the directory from which it gathers
34configuration information that ultimately ends up in the
35:term:`Build Directory` ``conf`` directory.
36
37If :term:`TEMPLATECONF` is not set, the default value is obtained
38from ``.templateconf`` file that is read from the same directory as
39``oe-init-build-env`` script. For the Poky reference distribution this
40would be::
41
42 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf/templates/default}
43
44If you look at a configuration template directory, you will
45see the ``bblayers.conf.sample``, ``local.conf.sample``, ``conf-summary.txt`` and
46``conf-notes.txt`` files. The build system uses these files to form the
47respective ``bblayers.conf`` file, ``local.conf`` file, and show
48users usage information about the build they're setting up
49when running the ``oe-init-build-env`` setup script. These can be
50edited further if needed to improve or change the build configurations
51available to the users, and provide useful summaries and detailed usage notes.
52
diff --git a/documentation/dev-manual/customizing-images.rst b/documentation/dev-manual/customizing-images.rst
new file mode 100644
index 0000000000..5b18958ade
--- /dev/null
+++ b/documentation/dev-manual/customizing-images.rst
@@ -0,0 +1,223 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Customizing Images
4******************
5
6You can customize images to satisfy particular requirements. This
7section describes several methods and provides guidelines for each.
8
9Customizing Images Using ``local.conf``
10=======================================
11
12Probably the easiest way to customize an image is to add a package by
13way of the ``local.conf`` configuration file. Because it is limited to
14local use, this method generally only allows you to add packages and is
15not as flexible as creating your own customized image. When you add
16packages using local variables this way, you need to realize that these
17variable changes are in effect for every build and consequently affect
18all images, which might not be what you require.
19
20To add a package to your image using the local configuration file, use
21the :term:`IMAGE_INSTALL` variable with the ``:append`` operator::
22
23 IMAGE_INSTALL:append = " strace"
24
25Use of the syntax is important; specifically, the leading space
26after the opening quote and before the package name, which is
27``strace`` in this example. This space is required since the ``:append``
28operator does not add the space.
29
30Furthermore, you must use ``:append`` instead of the ``+=`` operator if
31you want to avoid ordering issues. The reason for this is because doing
32so unconditionally appends to the variable and avoids ordering problems
33due to the variable being set in image recipes and ``.bbclass`` files
34with operators like ``?=``. Using ``:append`` ensures the operation
35takes effect.
36
37As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all
38images. It is possible to extend the syntax so that the variable applies
39to a specific image only. Here is an example::
40
41 IMAGE_INSTALL:append:pn-core-image-minimal = " strace"
42
43This example adds ``strace`` to the ``core-image-minimal`` image only.
44
45You can add packages using a similar approach through the
46:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
47``core-image-*`` images are affected.
48
49Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
50===============================================================================
51
52Another method for customizing your image is to enable or disable
53high-level image features by using the
54:term:`IMAGE_FEATURES` and
55:term:`EXTRA_IMAGE_FEATURES`
56variables. Although the functions for both variables are nearly
57equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
58a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
59``local.conf`` file, which is found in the :term:`Build Directory`.
60
61To understand how these features work, the best reference is
62:ref:`meta/classes-recipe/image.bbclass <ref-classes-image>`.
63This class lists out the available
64:term:`IMAGE_FEATURES` of which most map to package groups while some, such
65as ``debug-tweaks`` and ``read-only-rootfs``, resolve as general
66configuration settings.
67
68In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
69variable and then maps or configures the feature accordingly. Based on
70this information, the build system automatically adds the appropriate
71packages or configurations to the
72:term:`IMAGE_INSTALL` variable.
73Effectively, you are enabling extra features by extending the class or
74creating a custom class for use with specialized image ``.bb`` files.
75
76Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
77configuration file. Using a separate area from which to enable features
78with this variable helps you avoid overwriting the features in the image
79recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
80:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
81``meta/conf/bitbake.conf``.
82
83To illustrate how you can use these variables to modify your image,
84consider an example that selects the SSH server. The Yocto Project ships
85with two SSH servers you can use with your images: Dropbear and OpenSSH.
86Dropbear is a minimal SSH server appropriate for resource-constrained
87environments, while OpenSSH is a well-known standard SSH server
88implementation. By default, the ``core-image-sato`` image is configured
89to use Dropbear. The ``core-image-full-cmdline`` and ``core-image-lsb``
90images both include OpenSSH. The ``core-image-minimal`` image does not
91contain an SSH server.
92
93You can customize your image and change these defaults. Edit the
94:term:`IMAGE_FEATURES` variable in your recipe or use the
95:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
96configures the image you are working with to include
97``ssh-server-dropbear`` or ``ssh-server-openssh``.
98
99.. note::
100
101 See the ":ref:`ref-manual/features:image features`" section in the Yocto
102 Project Reference Manual for a complete list of image features that ship
103 with the Yocto Project.
104
105Customizing Images Using Custom .bb Files
106=========================================
107
108You can also customize an image by creating a custom recipe that defines
109additional software as part of the image. The following example shows
110the form for the two lines you need::
111
112 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
113 inherit core-image
114
115Defining the software using a custom recipe gives you total control over
116the contents of the image. It is important to use the correct names of
117packages in the :term:`IMAGE_INSTALL` variable. You must use the
118OpenEmbedded notation and not the Debian notation for the names (e.g.
119``glibc-dev`` instead of ``libc6-dev``).
120
121The other method for creating a custom image is to base it on an
122existing image. For example, if you want to create an image based on
123``core-image-sato`` but add the additional package ``strace`` to the
124image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
125``.bb`` and add the following line to the end of the copy::
126
127 IMAGE_INSTALL += "strace"
128
129Customizing Images Using Custom Package Groups
130==============================================
131
132For complex custom images, the best approach for customizing an image is
133to create a custom package group recipe that is used to build the image
134or images. A good example of a package group recipe is
135``meta/recipes-core/packagegroups/packagegroup-base.bb``.
136
137If you examine that recipe, you see that the :term:`PACKAGES` variable lists
138the package group packages to produce. The ``inherit packagegroup``
139statement sets appropriate default values and automatically adds
140``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
141package specified in the :term:`PACKAGES` statement.
142
143.. note::
144
145 The ``inherit packagegroup`` line should be located near the top of the
146 recipe, certainly before the :term:`PACKAGES` statement.
147
148For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
149and :term:`RRECOMMENDS` entries to provide a list of packages the parent
150task package should contain. You can see examples of these further down
151in the ``packagegroup-base.bb`` recipe.
152
153Here is a short, fabricated example showing the same basic pieces for a
154hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
155the variable :term:`PN` is the standard way to abbreviate the reference to
156the full packagegroup name ``packagegroup-custom``::
157
158 DESCRIPTION = "My Custom Package Groups"
159
160 inherit packagegroup
161
162 PACKAGES = "\
163 ${PN}-apps \
164 ${PN}-tools \
165 "
166
167 RDEPENDS:${PN}-apps = "\
168 dropbear \
169 portmap \
170 psplash"
171
172 RDEPENDS:${PN}-tools = "\
173 oprofile \
174 oprofileui-server \
175 lttng-tools"
176
177 RRECOMMENDS:${PN}-tools = "\
178 kernel-module-oprofile"
179
180In the previous example, two package group packages are created with
181their dependencies and their recommended package dependencies listed:
182``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
183build an image using these package group packages, you need to add
184``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
185:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
186areas of this section.
187
188Customizing an Image Hostname
189=============================
190
191By default, the configured hostname (i.e. ``/etc/hostname``) in an image
192is the same as the machine name. For example, if
193:term:`MACHINE` equals "qemux86", the
194configured hostname written to ``/etc/hostname`` is "qemux86".
195
196You can customize this name by altering the value of the "hostname"
197variable in the ``base-files`` recipe using either an append file or a
198configuration file. Use the following in an append file::
199
200 hostname = "myhostname"
201
202Use the following in a configuration file::
203
204 hostname:pn-base-files = "myhostname"
205
206Changing the default value of the variable "hostname" can be useful in
207certain situations. For example, suppose you need to do extensive
208testing on an image and you would like to easily identify the image
209under test from existing images with typical default hostnames. In this
210situation, you could change the default hostname to "testme", which
211results in all the images using the name "testme". Once testing is
212complete and you do not need to rebuild the image for test any longer,
213you can easily reset the default hostname.
214
215Another point of interest is that if you unset the variable, the image
216will have no default hostname in the filesystem. Here is an example that
217unsets the variable in a configuration file::
218
219 hostname:pn-base-files = ""
220
221Having no default hostname in the filesystem is suitable for
222environments that use dynamic hostnames such as virtual machines.
223
diff --git a/documentation/dev-manual/debugging.rst b/documentation/dev-manual/debugging.rst
new file mode 100644
index 0000000000..92458a0c37
--- /dev/null
+++ b/documentation/dev-manual/debugging.rst
@@ -0,0 +1,1271 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Debugging Tools and Techniques
4******************************
5
6The exact method for debugging build failures depends on the nature of
7the problem and on the system's area from which the bug originates.
8Standard debugging practices such as comparison against the last known
9working version with examination of the changes and the re-application
10of steps to identify the one causing the problem are valid for the Yocto
11Project just as they are for any other system. Even though it is
12impossible to detail every possible potential failure, this section
13provides some general tips to aid in debugging given a variety of
14situations.
15
16.. note::
17
18 A useful feature for debugging is the error reporting tool.
19 Configuring the Yocto Project to use this tool causes the
20 OpenEmbedded build system to produce error reporting commands as part
21 of the console output. You can enter the commands after the build
22 completes to log error information into a common database, that can
23 help you figure out what might be going wrong. For information on how
24 to enable and use this feature, see the
25 ":ref:`dev-manual/error-reporting-tool:using the error reporting tool`"
26 section.
27
28The following list shows the debugging topics in the remainder of this
29section:
30
31- ":ref:`dev-manual/debugging:viewing logs from failed tasks`" describes
32 how to find and view logs from tasks that failed during the build
33 process.
34
35- ":ref:`dev-manual/debugging:viewing variable values`" describes how to
36 use the BitBake ``-e`` option to examine variable values after a
37 recipe has been parsed.
38
39- ":ref:`dev-manual/debugging:viewing package information with \`\`oe-pkgdata-util\`\``"
40 describes how to use the ``oe-pkgdata-util`` utility to query
41 :term:`PKGDATA_DIR` and
42 display package-related information for built packages.
43
44- ":ref:`dev-manual/debugging:viewing dependencies between recipes and tasks`"
45 describes how to use the BitBake ``-g`` option to display recipe
46 dependency information used during the build.
47
48- ":ref:`dev-manual/debugging:viewing task variable dependencies`" describes
49 how to use the ``bitbake-dumpsig`` command in conjunction with key
50 subdirectories in the :term:`Build Directory` to determine variable
51 dependencies.
52
53- ":ref:`dev-manual/debugging:running specific tasks`" describes
54 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
55 to run specific tasks in the build chain. It can be useful to run
56 tasks "out-of-order" when trying isolate build issues.
57
58- ":ref:`dev-manual/debugging:general BitBake problems`" describes how
59 to use BitBake's ``-D`` debug output option to reveal more about what
60 BitBake is doing during the build.
61
62- ":ref:`dev-manual/debugging:building with no dependencies`"
63 describes how to use the BitBake ``-b`` option to build a recipe
64 while ignoring dependencies.
65
66- ":ref:`dev-manual/debugging:recipe logging mechanisms`"
67 describes how to use the many recipe logging functions to produce
68 debugging output and report errors and warnings.
69
70- ":ref:`dev-manual/debugging:debugging parallel make races`"
71 describes how to debug situations where the build consists of several
72 parts that are run simultaneously and when the output or result of
73 one part is not ready for use with a different part of the build that
74 depends on that output.
75
76- ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) remotely`"
77 describes how to use GDB to allow you to examine running programs, which can
78 help you fix problems.
79
80- ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) on the target`"
81 describes how to use GDB directly on target hardware for debugging.
82
83- ":ref:`dev-manual/debugging:other debugging tips`" describes
84 miscellaneous debugging tips that can be useful.
85
86Viewing Logs from Failed Tasks
87==============================
88
89You can find the log for a task in the file
90``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
91For example, the log for the
92:ref:`ref-tasks-compile` task of the
93QEMU minimal image for the x86 machine (``qemux86``) might be in
94``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
95To see the commands :term:`BitBake` ran
96to generate a log, look at the corresponding ``run.do_``\ `taskname` file
97in the same directory.
98
99``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
100links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
101``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
102when it ran. The symlinks always point to the files corresponding to the
103most recent run.
104
105Viewing Variable Values
106=======================
107
108Sometimes you need to know the value of a variable as a result of
109BitBake's parsing step. This could be because some unexpected behavior
110occurred in your project. Perhaps an attempt to :ref:`modify a variable
111<bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
112variables>` did not work out as expected.
113
114BitBake's ``-e`` option is used to display variable values after
115parsing. The following command displays the variable values after the
116configuration files (i.e. ``local.conf``, ``bblayers.conf``,
117``bitbake.conf`` and so forth) have been parsed::
118
119 $ bitbake -e
120
121The following command displays variable values after a specific recipe has
122been parsed. The variables include those from the configuration as well::
123
124 $ bitbake -e recipename
125
126.. note::
127
128 Each recipe has its own private set of variables (datastore).
129 Internally, after parsing the configuration, a copy of the resulting
130 datastore is made prior to parsing each recipe. This copying implies
131 that variables set in one recipe will not be visible to other
132 recipes.
133
134 Likewise, each task within a recipe gets a private datastore based on
135 the recipe datastore, which means that variables set within one task
136 will not be visible to other tasks.
137
138In the output of ``bitbake -e``, each variable is preceded by a
139description of how the variable got its value, including temporary
140values that were later overridden. This description also includes
141variable flags (varflags) set on the variable. The output can be very
142helpful during debugging.
143
144Variables that are exported to the environment are preceded by
145``export`` in the output of ``bitbake -e``. See the following example::
146
147 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
148
149In addition to variable values, the output of the ``bitbake -e`` and
150``bitbake -e`` recipe commands includes the following information:
151
152- The output starts with a tree listing all configuration files and
153 classes included globally, recursively listing the files they include
154 or inherit in turn. Much of the behavior of the OpenEmbedded build
155 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
156 implemented in the :ref:`ref-classes-base` class and the
157 classes it inherits, rather than being built into BitBake itself.
158
159- After the variable values, all functions appear in the output. For
160 shell functions, variables referenced within the function body are
161 expanded. If a function has been modified using overrides or using
162 override-style operators like ``:append`` and ``:prepend``, then the
163 final assembled function body appears in the output.
164
165Viewing Package Information with ``oe-pkgdata-util``
166====================================================
167
168You can use the ``oe-pkgdata-util`` command-line utility to query
169:term:`PKGDATA_DIR` and display
170various package-related information. When you use the utility, you must
171use it to view information on packages that have already been built.
172
173Here are a few of the available ``oe-pkgdata-util`` subcommands.
174
175.. note::
176
177 You can use the standard \* and ? globbing wildcards as part of
178 package names and paths.
179
180- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
181 that have been built, optionally limiting the match to packages that
182 match pattern.
183
184- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
185 files and directories contained in the given packages.
186
187 .. note::
188
189 A different way to view the contents of a package is to look at
190 the
191 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
192 directory of the recipe that generates the package. This directory
193 is created by the
194 :ref:`ref-tasks-package` task
195 and has one subdirectory for each package the recipe generates,
196 which contains the files stored in that package.
197
198 If you want to inspect the ``${WORKDIR}/packages-split``
199 directory, make sure that :ref:`ref-classes-rm-work` is not
200 enabled when you build the recipe.
201
202- ``oe-pkgdata-util find-path path ...``: Lists the names of
203 the packages that contain the given paths. For example, the following
204 tells us that ``/usr/share/man/man1/make.1`` is contained in the
205 ``make-doc`` package::
206
207 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
208 make-doc: /usr/share/man/man1/make.1
209
210- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
211 of the recipes that produce the given packages.
212
213For more information on the ``oe-pkgdata-util`` command, use the help
214facility::
215
216 $ oe-pkgdata-util --help
217 $ oe-pkgdata-util subcommand --help
218
219Viewing Dependencies Between Recipes and Tasks
220==============================================
221
222Sometimes it can be hard to see why BitBake wants to build other recipes
223before the one you have specified. Dependency information can help you
224understand why a recipe is built.
225
226To generate dependency information for a recipe, run the following
227command::
228
229 $ bitbake -g recipename
230
231This command writes the following files in the current directory:
232
233- ``pn-buildlist``: A list of recipes/targets involved in building
234 `recipename`. "Involved" here means that at least one task from the
235 recipe needs to run when building `recipename` from scratch. Targets
236 that are in
237 :term:`ASSUME_PROVIDED`
238 are not listed.
239
240- ``task-depends.dot``: A graph showing dependencies between tasks.
241
242The graphs are in :wikipedia:`DOT <DOT_%28graph_description_language%29>`
243format and can be converted to images (e.g. using the ``dot`` tool from
244`Graphviz <https://www.graphviz.org/>`__).
245
246.. note::
247
248 - DOT files use a plain text format. The graphs generated using the
249 ``bitbake -g`` command are often so large as to be difficult to
250 read without special pruning (e.g. with BitBake's ``-I`` option)
251 and processing. Despite the form and size of the graphs, the
252 corresponding ``.dot`` files can still be possible to read and
253 provide useful information.
254
255 As an example, the ``task-depends.dot`` file contains lines such
256 as the following::
257
258 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
259
260 The above example line reveals that the
261 :ref:`ref-tasks-configure`
262 task in ``libxslt`` depends on the
263 :ref:`ref-tasks-populate_sysroot`
264 task in ``libxml2``, which is a normal
265 :term:`DEPENDS` dependency
266 between the two recipes.
267
268 - For an example of how ``.dot`` files can be processed, see the
269 ``scripts/contrib/graph-tool`` Python script, which finds and
270 displays paths between graph nodes.
271
272You can use a different method to view dependency information by using
273either::
274
275 $ bitbake -g -u taskexp recipename
276
277or::
278
279 $ bitbake -g -u taskexp_ncurses recipename
280
281The ``-u taskdep`` option GUI window from which you can view build-time and
282runtime dependencies for the recipes involved in building recipename. The
283``-u taskexp_ncurses`` option uses ncurses instead of GTK to render the UI.
284
285Viewing Task Variable Dependencies
286==================================
287
288As mentioned in the
289":ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`"
290section of the BitBake User Manual, BitBake tries to automatically determine
291what variables a task depends on so that it can rerun the task if any values of
292the variables change. This determination is usually reliable. However, if you
293do things like construct variable names at runtime, then you might have to
294manually declare dependencies on those variables using ``vardeps`` as described
295in the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
296section of the BitBake User Manual.
297
298If you are unsure whether a variable dependency is being picked up
299automatically for a given task, you can list the variable dependencies
300BitBake has determined by doing the following:
301
302#. Build the recipe containing the task::
303
304 $ bitbake recipename
305
306#. Inside the :term:`STAMPS_DIR`
307 directory, find the signature data (``sigdata``) file that
308 corresponds to the task. The ``sigdata`` files contain a pickled
309 Python database of all the metadata that went into creating the input
310 checksum for the task. As an example, for the
311 :ref:`ref-tasks-fetch` task of the
312 ``db`` recipe, the ``sigdata`` file might be found in the following
313 location::
314
315 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
316
317 For tasks that are accelerated through the shared state
318 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
319 additional ``siginfo`` file is written into
320 :term:`SSTATE_DIR` along with
321 the cached task output. The ``siginfo`` files contain exactly the
322 same information as ``sigdata`` files.
323
324#. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
325 is an example::
326
327 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
328
329 In the output of the above command, you will find a line like the
330 following, which lists all the (inferred) variable dependencies for
331 the task. This list also includes indirect dependencies from
332 variables depending on other variables, recursively::
333
334 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[sha256sum]', 'base_do_fetch']
335
336 .. note::
337
338 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
339 These functions in turn depend on the variables they reference.
340
341 The output of ``bitbake-dumpsig`` also includes the value each
342 variable had, a list of dependencies for each variable, and
343 :term:`BB_BASEHASH_IGNORE_VARS`
344 information.
345
346Debugging signature construction and unexpected task executions
347===============================================================
348
349There is a ``bitbake-diffsigs`` command for comparing two
350``siginfo`` or ``sigdata`` files. This command can be helpful when
351trying to figure out what changed between two versions of a task. If you
352call ``bitbake-diffsigs`` with just one file, the command behaves like
353``bitbake-dumpsig``.
354
355You can also use BitBake to dump out the signature construction
356information without executing tasks by using either of the following
357BitBake command-line options::
358
359 ‐‐dump-signatures=SIGNATURE_HANDLER
360 -S SIGNATURE_HANDLER
361
362
363.. note::
364
365 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
366 dump only the signature or compare the dumped signature with the most recent one,
367 respectively. "printdiff" will try to establish the most recent
368 signature match (e.g. in the sstate cache) and then
369 compare the matched signatures to determine the stamps and delta
370 where these two stamp trees diverge. This can be used to determine why
371 tasks need to be re-run in situations where that is not expected.
372
373Using BitBake with either of these options causes BitBake to dump out
374``sigdata`` files in the ``stamps`` directory for every task it would
375have executed instead of building the specified target package.
376
377Viewing Metadata Used to Create the Input Signature of a Shared State Task
378==========================================================================
379
380Seeing what metadata went into creating the input signature of a shared
381state (sstate) task can be a useful debugging aid. This information is
382available in signature information (``siginfo``) files in
383:term:`SSTATE_DIR`. For
384information on how to view and interpret information in ``siginfo``
385files, see the
386":ref:`dev-manual/debugging:viewing task variable dependencies`" section.
387
388For conceptual information on shared state, see the
389":ref:`overview-manual/concepts:shared state`"
390section in the Yocto Project Overview and Concepts Manual.
391
392Invalidating Shared State to Force a Task to Run
393================================================
394
395The OpenEmbedded build system uses
396:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
397:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
398rebuilding tasks. Collectively, this scheme is known as "shared state
399code".
400
401As with all schemes, this one has some drawbacks. It is possible that
402you could make implicit changes to your code that the checksum
403calculations do not take into account. These implicit changes affect a
404task's output but do not trigger the shared state code into rebuilding a
405recipe. Consider an example during which a tool changes its output.
406Assume that the output of ``rpmdeps`` changes. The result of the change
407should be that all the ``package`` and ``package_write_rpm`` shared
408state cache items become invalid. However, because the change to the
409output is external to the code and therefore implicit, the associated
410shared state cache items do not become invalidated. In this case, the
411build process uses the cached items rather than running the task again.
412Obviously, these types of implicit changes can cause problems.
413
414To avoid these problems during the build, you need to understand the
415effects of any changes you make. Realize that changes you make directly
416to a function are automatically factored into the checksum calculation.
417Thus, these explicit changes invalidate the associated area of shared
418state cache. However, you need to be aware of any implicit changes that
419are not obvious changes to the code and could affect the output of a
420given task.
421
422When you identify an implicit change, you can easily take steps to
423invalidate the cache and force the tasks to run. The steps you can take
424are as simple as changing a function's comments in the source code. For
425example, to invalidate package shared state files, change the comment
426statements of
427:ref:`ref-tasks-package` or the
428comments of one of the functions it calls. Even though the change is
429purely cosmetic, it causes the checksum to be recalculated and forces
430the build system to run the task again.
431
432.. note::
433
434 For an example of a commit that makes a cosmetic change to invalidate
435 shared state, see this
436 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
437
438Running Specific Tasks
439======================
440
441Any given recipe consists of a set of tasks. The standard BitBake
442behavior in most cases is: :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`,
443:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, :ref:`ref-tasks-install`, :ref:`ref-tasks-package`,
444:ref:`do_package_write_* <ref-tasks-package_write_deb>`, and :ref:`ref-tasks-build`. The default task is
445:ref:`ref-tasks-build` and any tasks on which it depends build first. Some tasks,
446such as :ref:`ref-tasks-devshell`, are not part of the default build chain. If you
447wish to run a task that is not part of the default build chain, you can
448use the ``-c`` option in BitBake. Here is an example::
449
450 $ bitbake matchbox-desktop -c devshell
451
452The ``-c`` option respects task dependencies, which means that all other
453tasks (including tasks from other recipes) that the specified task
454depends on will be run before the task. Even when you manually specify a
455task to run with ``-c``, BitBake will only run the task if it considers
456it "out of date". See the
457":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
458section in the Yocto Project Overview and Concepts Manual for how
459BitBake determines whether a task is "out of date".
460
461If you want to force an up-to-date task to be rerun (e.g. because you
462made manual modifications to the recipe's
463:term:`WORKDIR` that you want to try
464out), then you can use the ``-f`` option.
465
466.. note::
467
468 The reason ``-f`` is never required when running the
469 :ref:`ref-tasks-devshell` task is because the
470 [\ :ref:`nostamp <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
471 variable flag is already set for the task.
472
473The following example shows one way you can use the ``-f`` option::
474
475 $ bitbake matchbox-desktop
476 .
477 .
478 make some changes to the source code in the work directory
479 .
480 .
481 $ bitbake matchbox-desktop -c compile -f
482 $ bitbake matchbox-desktop
483
484This sequence first builds and then recompiles ``matchbox-desktop``. The
485last command reruns all tasks (basically the packaging tasks) after the
486compile. BitBake recognizes that the :ref:`ref-tasks-compile` task was rerun and
487therefore understands that the other tasks also need to be run again.
488
489Another, shorter way to rerun a task and all
490:ref:`ref-manual/tasks:normal recipe build tasks`
491that depend on it is to use the ``-C`` option.
492
493.. note::
494
495 This option is upper-cased and is separate from the ``-c``
496 option, which is lower-cased.
497
498Using this option invalidates the given task and then runs the
499:ref:`ref-tasks-build` task, which is
500the default task if no task is given, and the tasks on which it depends.
501You could replace the final two commands in the previous example with
502the following single command::
503
504 $ bitbake matchbox-desktop -C compile
505
506Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
507the input checksum of the specified task. This tainting indirectly
508causes the task and its dependent tasks to be rerun through the normal
509task dependency mechanisms.
510
511.. note::
512
513 BitBake explicitly keeps track of which tasks have been tainted in
514 this fashion, and will print warnings such as the following for
515 builds involving such tasks:
516
517 .. code-block:: none
518
519 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
520
521
522 The purpose of the warning is to let you know that the work directory
523 and build output might not be in the clean state they would be in for
524 a "normal" build, depending on what actions you took. To get rid of
525 such warnings, you can remove the work directory and rebuild the
526 recipe, as follows::
527
528 $ bitbake matchbox-desktop -c clean
529 $ bitbake matchbox-desktop
530
531
532You can view a list of tasks in a given package by running the
533:ref:`ref-tasks-listtasks` task as follows::
534
535 $ bitbake matchbox-desktop -c listtasks
536
537The results appear as output to the console and are also in
538the file ``${WORKDIR}/temp/log.do_listtasks``.
539
540General BitBake Problems
541========================
542
543You can see debug output from BitBake by using the ``-D`` option. The
544debug output gives more information about what BitBake is doing and the
545reason behind it. Each ``-D`` option you use increases the logging
546level. The most common usage is ``-DDD``.
547
548The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
549chose a certain version of a package or why BitBake picked a certain
550provider. This command could also help you in a situation where you
551think BitBake did something unexpected.
552
553Building with No Dependencies
554=============================
555
556To build a specific recipe (``.bb`` file), you can use the following
557command form::
558
559 $ bitbake -b somepath/somerecipe.bb
560
561This command form does
562not check for dependencies. Consequently, you should use it only when
563you know existing dependencies have been met.
564
565.. note::
566
567 You can also specify fragments of the filename. In this case, BitBake
568 checks for a unique match.
569
570Recipe Logging Mechanisms
571=========================
572
573The Yocto Project provides several logging functions for producing
574debugging output and reporting errors and warnings. For Python
575functions, the following logging functions are available. All of these functions
576log to ``${T}/log.do_``\ `task`, and can also log to standard output
577(stdout) with the right settings:
578
579- ``bb.plain(msg)``: Writes msg as is to the log while also
580 logging to stdout.
581
582- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
583 stdout if BitBake is called with "-v".
584
585- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the log. Also logs to
586 stdout if the log level is greater than or equal to level. See the
587 ":ref:`bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`"
588 option in the BitBake User Manual for more information.
589
590- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
591 logging to stdout.
592
593- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
594 logging to standard out (stdout).
595
596 .. note::
597
598 Calling this function does not cause the task to fail.
599
600- ``bb.fatal(msg)``: This logging function is similar to
601 ``bb.error(msg)`` but also causes the calling task to fail.
602
603 .. note::
604
605 ``bb.fatal()`` raises an exception, which means you do not need to put a
606 "return" statement after the function.
607
608The same logging functions are also available in shell functions, under
609the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
610and ``bbfatal``. The :ref:`ref-classes-logging` class
611implements these functions. See that class in the ``meta/classes``
612folder of the :term:`Source Directory` for information.
613
614Logging With Python
615-------------------
616
617When creating recipes using Python and inserting code that handles build
618logs, keep in mind the goal is to have informative logs while keeping
619the console as "silent" as possible. Also, if you want status messages
620in the log, use the "debug" loglevel.
621
622Here is an example written in Python. The code handles logging for
623a function that determines the number of tasks needed to be run. See the
624":ref:`ref-tasks-listtasks`"
625section for additional information::
626
627 python do_listtasks() {
628 bb.debug(2, "Starting to figure out the task list")
629 if noteworthy_condition:
630 bb.note("There are 47 tasks to run")
631 bb.debug(2, "Got to point xyz")
632 if warning_trigger:
633 bb.warn("Detected warning_trigger, this might be a problem later.")
634 if recoverable_error:
635 bb.error("Hit recoverable_error, you really need to fix this!")
636 if fatal_error:
637 bb.fatal("fatal_error detected, unable to print the task list")
638 bb.plain("The tasks present are abc")
639 bb.debug(2, "Finished figuring out the tasklist")
640 }
641
642Logging With Bash
643-----------------
644
645When creating recipes using Bash and inserting code that handles build
646logs, you have the same goals --- informative with minimal console output.
647The syntax you use for recipes written in Bash is similar to that of
648recipes written in Python described in the previous section.
649
650Here is an example written in Bash. The code logs the progress of
651the ``do_my_function`` function::
652
653 do_my_function() {
654 bbdebug 2 "Running do_my_function"
655 if [ exceptional_condition ]; then
656 bbnote "Hit exceptional_condition"
657 fi
658 bbdebug 2 "Got to point xyz"
659 if [ warning_trigger ]; then
660 bbwarn "Detected warning_trigger, this might cause a problem later."
661 fi
662 if [ recoverable_error ]; then
663 bberror "Hit recoverable_error, correcting"
664 fi
665 if [ fatal_error ]; then
666 bbfatal "fatal_error detected"
667 fi
668 bbdebug 2 "Completed do_my_function"
669 }
670
671
672Debugging Parallel Make Races
673=============================
674
675A parallel ``make`` race occurs when the build consists of several parts
676that are run simultaneously and a situation occurs when the output or
677result of one part is not ready for use with a different part of the
678build that depends on that output. Parallel make races are annoying and
679can sometimes be difficult to reproduce and fix. However, there are some simple
680tips and tricks that can help you debug and fix them. This section
681presents a real-world example of an error encountered on the Yocto
682Project autobuilder and the process used to fix it.
683
684.. note::
685
686 If you cannot properly fix a ``make`` race condition, you can work around it
687 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
688 variables.
689
690The Failure
691-----------
692
693For this example, assume that you are building an image that depends on
694the "neard" package. And, during the build, BitBake runs into problems
695and creates the following output.
696
697.. note::
698
699 This example log file has longer lines artificially broken to make
700 the listing easier to read.
701
702If you examine the output or the log file, you see the failure during
703``make``:
704
705.. code-block:: none
706
707 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
708 | DEBUG: Executing shell function do_compile
709 | NOTE: make -j 16
710 | make --no-print-directory all-am
711 | /bin/mkdir -p include/near
712 | /bin/mkdir -p include/near
713 | /bin/mkdir -p include/near
714 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
715 0.14-r0/neard-0.14/include/types.h include/near/types.h
716 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
717 0.14-r0/neard-0.14/include/log.h include/near/log.h
718 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
719 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
720 | /bin/mkdir -p include/near
721 | /bin/mkdir -p include/near
722 | /bin/mkdir -p include/near
723 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
724 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
725 | /bin/mkdir -p include/near
726 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
727 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
728 | /bin/mkdir -p include/near
729 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
730 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
731 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
732 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
733 | /bin/mkdir -p include/near
734 | /bin/mkdir -p include/near
735 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
736 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
737 | /bin/mkdir -p include/near
738 | /bin/mkdir -p include/near
739 | /bin/mkdir -p include/near
740 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
741 0.14-r0/neard-0.14/include/device.h include/near/device.h
742 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
743 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
744 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
745 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
746 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
747 0.14-r0/neard-0.14/include/version.h include/near/version.h
748 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
749 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
750 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
751 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/
752 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
753 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
754 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
755 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/
756 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/
757 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
758 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
759 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
760 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
761 -o tools/snep-send.o tools/snep-send.c
762 | In file included from tools/snep-send.c:16:0:
763 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
764 | #include <near/dbus.h>
765 | ^
766 | compilation terminated.
767 | make[1]: *** [tools/snep-send.o] Error 1
768 | make[1]: *** Waiting for unfinished jobs....
769 | make: *** [all] Error 2
770 | ERROR: oe_runmake failed
771
772Reproducing the Error
773---------------------
774
775Because race conditions are intermittent, they do not manifest
776themselves every time you do the build. In fact, most times the build
777will complete without problems even though the potential race condition
778exists. Thus, once the error surfaces, you need a way to reproduce it.
779
780In this example, compiling the "neard" package is causing the problem.
781So the first thing to do is build "neard" locally. Before you start the
782build, set the
783:term:`PARALLEL_MAKE` variable
784in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
785high value for :term:`PARALLEL_MAKE` increases the chances of the race
786condition showing up::
787
788 $ bitbake neard
789
790Once the local build for "neard" completes, start a ``devshell`` build::
791
792 $ bitbake neard -c devshell
793
794For information on how to use a ``devshell``, see the
795":ref:`dev-manual/development-shell:using a development shell`" section.
796
797In the ``devshell``, do the following::
798
799 $ make clean
800 $ make tools/snep-send.o
801
802The ``devshell`` commands cause the failure to clearly
803be visible. In this case, there is a missing dependency for the ``neard``
804Makefile target. Here is some abbreviated, sample output with the
805missing dependency clearly visible at the end::
806
807 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
808 .
809 .
810 .
811 tools/snep-send.c
812 In file included from tools/snep-send.c:16:0:
813 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
814 #include <near/dbus.h>
815 ^
816 compilation terminated.
817 make: *** [tools/snep-send.o] Error 1
818 $
819
820
821Creating a Patch for the Fix
822----------------------------
823
824Because there is a missing dependency for the Makefile target, you need
825to patch the ``Makefile.am`` file, which is generated from
826``Makefile.in``. You can use Quilt to create the patch::
827
828 $ quilt new parallelmake.patch
829 Patch patches/parallelmake.patch is now on top
830 $ quilt add Makefile.am
831 File Makefile.am added to patch patches/parallelmake.patch
832
833For more information on using Quilt, see the
834":ref:`dev-manual/quilt:using quilt in your workflow`" section.
835
836At this point you need to make the edits to ``Makefile.am`` to add the
837missing dependency. For our example, you have to add the following line
838to the file::
839
840 tools/snep-send.$(OBJEXT): include/near/dbus.h
841
842Once you have edited the file, use the ``refresh`` command to create the
843patch::
844
845 $ quilt refresh
846 Refreshed patch patches/parallelmake.patch
847
848Once the patch file is created, you need to add it back to the originating
849recipe folder. Here is an example assuming a top-level
850:term:`Source Directory` named ``poky``::
851
852 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
853
854The final thing you need to do to implement the fix in the build is to
855update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
856:term:`SRC_URI` statement includes
857the patch file. The recipe file is in the folder above the patch. Here
858is what the edited :term:`SRC_URI` statement would look like::
859
860 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
861 file://neard.in \
862 file://neard.service.in \
863 file://parallelmake.patch \
864 "
865
866With the patch complete and moved to the correct folder and the
867:term:`SRC_URI` statement updated, you can exit the ``devshell``::
868
869 $ exit
870
871Testing the Build
872-----------------
873
874With everything in place, you can get back to trying the build again
875locally::
876
877 $ bitbake neard
878
879This build should succeed.
880
881Now you can open up a ``devshell`` again and repeat the clean and make
882operations as follows::
883
884 $ bitbake neard -c devshell
885 $ make clean
886 $ make tools/snep-send.o
887
888The build should work without issue.
889
890As with all solved problems, if they originated upstream, you need to
891submit the fix for the recipe in OE-Core and upstream so that the
892problem is taken care of at its source. See the
893":doc:`../contributor-guide/submit-changes`" section for more information.
894
895Debugging With the GNU Project Debugger (GDB) Remotely
896======================================================
897
898GDB allows you to examine running programs, which in turn helps you to
899understand and fix problems. It also allows you to perform post-mortem
900style analysis of program crashes. GDB is available as a package within
901the Yocto Project and is installed in SDK images by default. See the
902":ref:`ref-manual/images:Images`" chapter in the Yocto
903Project Reference Manual for a description of these images. You can find
904information on GDB at https://sourceware.org/gdb/.
905
906.. note::
907
908 For best results, install debug (``-dbg``) packages for the applications you
909 are going to debug. Doing so makes extra debug symbols available that give
910 you more meaningful output.
911
912Sometimes, due to memory or disk space constraints, it is not possible
913to use GDB directly on the remote target to debug applications. These
914constraints arise because GDB needs to load the debugging information
915and the binaries of the process being debugged. Additionally, GDB needs
916to perform many computations to locate information such as function
917names, variable names and values, stack traces and so forth --- even
918before starting the debugging process. These extra computations place
919more load on the target system and can alter the characteristics of the
920program being debugged.
921
922To help get past the previously mentioned constraints, there are two
923methods you can use: running a debuginfod server and using gdbserver.
924
925Using the debuginfod server method
926----------------------------------
927
928``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
929Running a ``debuginfod`` server makes debug symbols readily available,
930which means you don't need to download debugging information
931and the binaries of the process being debugged. You can just fetch
932debug symbols from the server.
933
934To run a ``debuginfod`` server, you need to do the following:
935
936- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
937 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
938 If not, set in your distro config file or in ``local.conf``::
939
940 DISTRO_FEATURES:append = " debuginfod"
941
942 This distro feature enables the server and client library in ``elfutils``,
943 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
944
945- Run the following commands to launch the ``debuginfod`` server on the host::
946
947 $ oe-debuginfod
948
949- To use ``debuginfod`` on the target, you need to know the ip:port where
950 ``debuginfod`` is listening on the host (port defaults to 8002), and export
951 that into the shell environment, for example in ``qemu``::
952
953 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
954
955- Then debug info fetching should simply work when running the target ``gdb``,
956 ``readelf`` or ``objdump``, for example::
957
958 root@qemux86-64:~# gdb /bin/cat
959 ...
960 Reading symbols from /bin/cat...
961 Downloading separate debug info for /bin/cat...
962 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
963
964- It's also possible to use ``debuginfod-find`` to just query the server::
965
966 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
967 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
968
969
970Using the gdbserver method
971--------------------------
972
973gdbserver, which runs on the remote target and does not load any
974debugging information from the debugged process. Instead, a GDB instance
975processes the debugging information that is run on a remote computer -
976the host GDB. The host GDB then sends control commands to gdbserver to
977make it stop or start the debugged program, as well as read or write
978memory regions of that debugged program. All the debugging information
979loaded and processed as well as all the heavy debugging is done by the
980host GDB. Offloading these processes gives the gdbserver running on the
981target a chance to remain small and fast.
982
983Because the host GDB is responsible for loading the debugging
984information and for doing the necessary processing to make actual
985debugging happen, you have to make sure the host can access the
986unstripped binaries complete with their debugging information and also
987be sure the target is compiled with no optimizations. The host GDB must
988also have local access to all the libraries used by the debugged
989program. Because gdbserver does not need any local debugging
990information, the binaries on the remote target can remain stripped.
991However, the binaries must also be compiled without optimization so they
992match the host's binaries.
993
994To remain consistent with GDB documentation and terminology, the binary
995being debugged on the remote target machine is referred to as the
996"inferior" binary. For documentation on GDB see the `GDB
997site <https://sourceware.org/gdb/documentation/>`__.
998
999The following steps show you how to debug using the GNU project
1000debugger.
1001
1002#. *Configure your build system to construct the companion debug
1003 filesystem:*
1004
1005 In your ``local.conf`` file, set the following::
1006
1007 IMAGE_GEN_DEBUGFS = "1"
1008 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
1009
1010 These options cause the
1011 OpenEmbedded build system to generate a special companion filesystem
1012 fragment, which contains the matching source and debug symbols to
1013 your deployable filesystem. The build system does this by looking at
1014 what is in the deployed filesystem, and pulling the corresponding
1015 ``-dbg`` packages.
1016
1017 The companion debug filesystem is not a complete filesystem, but only
1018 contains the debug fragments. This filesystem must be combined with
1019 the full filesystem for debugging. Subsequent steps in this procedure
1020 show how to combine the partial filesystem with the full filesystem.
1021
1022#. *Configure the system to include gdbserver in the target filesystem:*
1023
1024 Make the following addition in your ``local.conf`` file::
1025
1026 EXTRA_IMAGE_FEATURES:append = " tools-debug"
1027
1028 The change makes
1029 sure the ``gdbserver`` package is included.
1030
1031#. *Build the environment:*
1032
1033 Use the following command to construct the image and the companion
1034 Debug Filesystem::
1035
1036 $ bitbake image
1037
1038 Build the cross GDB component and
1039 make it available for debugging. Build the SDK that matches the
1040 image. Building the SDK is best for a production build that can be
1041 used later for debugging, especially during long term maintenance::
1042
1043 $ bitbake -c populate_sdk image
1044
1045 Alternatively, you can build the minimal toolchain components that
1046 match the target. Doing so creates a smaller than typical SDK and
1047 only contains a minimal set of components with which to build simple
1048 test applications, as well as run the debugger::
1049
1050 $ bitbake meta-toolchain
1051
1052 A final method is to build Gdb itself within the build system::
1053
1054 $ bitbake gdb-cross-<architecture>
1055
1056 Doing so produces a temporary copy of
1057 ``cross-gdb`` you can use for debugging during development. While
1058 this is the quickest approach, the two previous methods in this step
1059 are better when considering long-term maintenance strategies.
1060
1061 .. note::
1062
1063 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
1064 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
1065 actual name you want to use.
1066
1067#. *Set up the* ``debugfs``\ *:*
1068
1069 Run the following commands to set up the ``debugfs``::
1070
1071 $ mkdir debugfs
1072 $ cd debugfs
1073 $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2
1074 $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2
1075
1076#. *Set up GDB:*
1077
1078 Install the SDK (if you built one) and then source the correct
1079 environment file. Sourcing the environment file puts the SDK in your
1080 ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger.
1081
1082 If you are using the build system, Gdb is located in
1083 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
1084
1085#. *Boot the target:*
1086
1087 For information on how to run QEMU, see the `QEMU
1088 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
1089
1090 .. note::
1091
1092 Be sure to verify that your host can access the target via TCP.
1093
1094#. *Debug a program:*
1095
1096 Debugging a program involves running gdbserver on the target and then
1097 running Gdb on the host. The example in this step debugs ``gzip``:
1098
1099 .. code-block:: shell
1100
1101 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
1102
1103 For
1104 additional gdbserver options, see the `GDB Server
1105 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
1106
1107 After running gdbserver on the target, you need to run Gdb on the
1108 host and configure it and connect to the target. Use these commands::
1109
1110 $ cd directory-holding-the-debugfs-directory
1111 $ arch-gdb
1112 (gdb) set sysroot debugfs
1113 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
1114 (gdb) target remote IP-of-target:1234
1115
1116 At this
1117 point, everything should automatically load (i.e. matching binaries,
1118 symbols and headers).
1119
1120 .. note::
1121
1122 The Gdb ``set`` commands in the previous example can be placed into the
1123 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
1124 commands are in that file.
1125
1126#. *Deploying without a full image rebuild:*
1127
1128 In many cases, during development you want a quick method to deploy a
1129 new binary to the target and debug it, without waiting for a full
1130 image build.
1131
1132 One approach to solving this situation is to just build the component
1133 you want to debug. Once you have built the component, copy the
1134 executable directly to both the target and the host ``debugfs``.
1135
1136 If the binary is processed through the debug splitting in
1137 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
1138 contents and corresponding ``/usr/src/debug`` files) from the work
1139 directory. Here is an example::
1140
1141 $ bitbake bash
1142 $ bitbake -c devshell bash
1143 $ cd ..
1144 $ scp packages-split/bash/bin/bash target:/bin/bash
1145 $ cp -a packages-split/bash-dbg/\* path/debugfs
1146
1147Debugging with the GNU Project Debugger (GDB) on the Target
1148===========================================================
1149
1150The previous section addressed using GDB remotely for debugging
1151purposes, which is the most usual case due to the inherent hardware
1152limitations on many embedded devices. However, debugging in the target
1153hardware itself is also possible with more powerful devices. This
1154section describes what you need to do in order to support using GDB to
1155debug on the target hardware.
1156
1157To support this kind of debugging, you need do the following:
1158
1159- Ensure that GDB is on the target. You can do this by making
1160 the following addition to your ``local.conf`` file::
1161
1162 EXTRA_IMAGE_FEATURES:append = " tools-debug"
1163
1164- Ensure that debug symbols are present. You can do so by adding the
1165 corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`::
1166
1167 IMAGE_INSTALL:append = " packagename-dbg"
1168
1169 Alternatively, you can add the following to ``local.conf`` to include
1170 all the debug symbols::
1171
1172 EXTRA_IMAGE_FEATURES:append = " dbg-pkgs"
1173
1174.. note::
1175
1176 To improve the debug information accuracy, you can reduce the level
1177 of optimization used by the compiler. For example, when adding the
1178 following line to your ``local.conf`` file, you will reduce optimization
1179 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
1180 of "-O -fno-omit-frame-pointer"::
1181
1182 DEBUG_BUILD = "1"
1183
1184 Consider that this will reduce the application's performance and is
1185 recommended only for debugging purposes.
1186
1187Enabling Minidebuginfo
1188======================
1189
1190Enabling the :term:`DISTRO_FEATURES` minidebuginfo adds a compressed ELF section ``.gnu_debugdata``
1191to all binary files, containing only function names, and thus increasing the size of the
1192binaries only by 5 to 10%. For comparison, full debug symbols can be 10 times as big as
1193a stripped binary, and it is thus not always possible to deploy full debug symbols.
1194Minidebuginfo data allows, on the one side, to retrieve a call-stack using
1195GDB (command backtrace) without deploying full debug symbols to the target. It also
1196allows to retrieve a symbolicated call-stack when using ``systemd-coredump`` to manage
1197coredumps (commands ``coredumpctl list`` and ``coredumpctl info``).
1198
1199This feature was created by Fedora, see https://fedoraproject.org/wiki/Features/MiniDebugInfo for
1200more details.
1201
1202Other Debugging Tips
1203====================
1204
1205Here are some other tips that you might find useful:
1206
1207- When adding new packages, it is worth watching for undesirable items
1208 making their way into compiler command lines. For example, you do not
1209 want references to local system files like ``/usr/lib/`` or
1210 ``/usr/include/``.
1211
1212- If you want to remove the ``psplash`` boot splashscreen, add
1213 ``psplash=false`` to the kernel command line. Doing so prevents
1214 ``psplash`` from loading and thus allows you to see the console. It
1215 is also possible to switch out of the splashscreen by switching the
1216 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
1217
1218- Removing :term:`TMPDIR` (usually ``tmp/``, within the
1219 :term:`Build Directory`) can often fix temporary build issues. Removing
1220 :term:`TMPDIR` is usually a relatively cheap operation, because task output
1221 will be cached in :term:`SSTATE_DIR` (usually ``sstate-cache/``, which is
1222 also in the :term:`Build Directory`).
1223
1224 .. note::
1225
1226 Removing :term:`TMPDIR` might be a workaround rather than a fix.
1227 Consequently, trying to determine the underlying cause of an issue before
1228 removing the directory is a good idea.
1229
1230- Understanding how a feature is used in practice within existing
1231 recipes can be very helpful. It is recommended that you configure
1232 some method that allows you to quickly search through files.
1233
1234 Using GNU Grep, you can use the following shell function to
1235 recursively search through common recipe-related files, skipping
1236 binary files, ``.git`` directories, and the :term:`Build Directory`
1237 (assuming its name starts with "build")::
1238
1239 g() {
1240 grep -Ir \
1241 --exclude-dir=.git \
1242 --exclude-dir='build*' \
1243 --include='*.bb*' \
1244 --include='*.inc*' \
1245 --include='*.conf*' \
1246 --include='*.py*' \
1247 "$@"
1248 }
1249
1250 Here are some usage examples::
1251
1252 $ g FOO # Search recursively for "FOO"
1253 $ g -i foo # Search recursively for "foo", ignoring case
1254 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
1255
1256 If figuring
1257 out how some feature works requires a lot of searching, it might
1258 indicate that the documentation should be extended or improved. In
1259 such cases, consider filing a documentation bug using the Yocto
1260 Project implementation of
1261 :yocto_bugs:`Bugzilla <>`. For information on
1262 how to submit a bug against the Yocto Project, see the Yocto Project
1263 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
1264 and the ":doc:`../contributor-guide/report-defect`" section.
1265
1266 .. note::
1267
1268 The manuals might not be the right place to document variables
1269 that are purely internal and have a limited scope (e.g. internal
1270 variables used to implement a single ``.bbclass`` file).
1271
diff --git a/documentation/dev-manual/development-shell.rst b/documentation/dev-manual/development-shell.rst
new file mode 100644
index 0000000000..be26bcffc7
--- /dev/null
+++ b/documentation/dev-manual/development-shell.rst
@@ -0,0 +1,82 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using a Development Shell
4*************************
5
6When debugging certain commands or even when just editing packages,
7``devshell`` can be a useful tool. When you invoke ``devshell``, all
8tasks up to and including
9:ref:`ref-tasks-patch` are run for the
10specified target. Then, a new terminal is opened and you are placed in
11``${``\ :term:`S`\ ``}``, the source
12directory. In the new terminal, all the OpenEmbedded build-related
13environment variables are still defined so you can use commands such as
14``configure`` and ``make``. The commands execute just as if the
15OpenEmbedded build system were executing them. Consequently, working
16this way can be helpful when debugging a build or preparing software to
17be used with the OpenEmbedded build system.
18
19Here is an example that uses ``devshell`` on a target named
20``matchbox-desktop``::
21
22 $ bitbake matchbox-desktop -c devshell
23
24This command spawns a terminal with a shell prompt within the
25OpenEmbedded build environment. The
26:term:`OE_TERMINAL` variable
27controls what type of shell is opened.
28
29For spawned terminals, the following occurs:
30
31- The ``PATH`` variable includes the cross-toolchain.
32
33- The ``pkgconfig`` variables find the correct ``.pc`` files.
34
35- The ``configure`` command finds the Yocto Project site files as well
36 as any other necessary files.
37
38Within this environment, you can run configure or compile commands as if
39they were being run by the OpenEmbedded build system itself. As noted
40earlier, the working directory also automatically changes to the Source
41Directory (:term:`S`).
42
43To manually run a specific task using ``devshell``, run the
44corresponding ``run.*`` script in the
45``${``\ :term:`WORKDIR`\ ``}/temp``
46directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
47not exist, which would be the case if the task was skipped by way of the
48sstate cache, you can create the task by first running it outside of the
49``devshell``::
50
51 $ bitbake -c task
52
53.. note::
54
55 - Execution of a task's ``run.*`` script and BitBake's execution of
56 a task are identical. In other words, running the script re-runs
57 the task just as it would be run using the ``bitbake -c`` command.
58
59 - Any ``run.*`` file that does not have a ``.pid`` extension is a
60 symbolic link (symlink) to the most recent version of that file.
61
62Remember, that the ``devshell`` is a mechanism that allows you to get
63into the BitBake task execution environment. And as such, all commands
64must be called just as BitBake would call them. That means you need to
65provide the appropriate options for cross-compilation and so forth as
66applicable.
67
68When you are finished using ``devshell``, exit the shell or close the
69terminal window.
70
71.. note::
72
73 - It is worth remembering that when using ``devshell`` you need to
74 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
75 instead of just using ``gcc``. The same applies to other
76 applications such as ``binutils``, ``libtool`` and so forth.
77 BitBake sets up environment variables such as :term:`CC` to assist
78 applications, such as ``make`` to find the correct tools.
79
80 - It is also worth noting that ``devshell`` still works over X11
81 forwarding and similar situations.
82
diff --git a/documentation/dev-manual/device-manager.rst b/documentation/dev-manual/device-manager.rst
new file mode 100644
index 0000000000..49fc785fec
--- /dev/null
+++ b/documentation/dev-manual/device-manager.rst
@@ -0,0 +1,74 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3.. _device-manager:
4
5Selecting a Device Manager
6**************************
7
8The Yocto Project provides multiple ways to manage the device manager
9(``/dev``):
10
11- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev``
12 directory is persistent and the required device nodes are created
13 during the build.
14
15- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
16 directory is provided by the kernel as an in-memory file system and
17 is automatically populated by the kernel at runtime. Additional
18 configuration of device nodes is done in user space by a device
19 manager like ``udev`` or ``busybox-mdev``.
20
21Using Persistent and Pre-Populated ``/dev``
22===========================================
23
24To use the static method for device population, you need to set the
25:term:`USE_DEVFS` variable to "0"
26as follows::
27
28 USE_DEVFS = "0"
29
30The content of the resulting ``/dev`` directory is defined in a Device
31Table file. The
32:term:`IMAGE_DEVICE_TABLES`
33variable defines the Device Table to use and should be set in the
34machine or distro configuration file. Alternatively, you can set this
35variable in your ``local.conf`` configuration file.
36
37If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
38``device_table-minimal.txt`` is used::
39
40 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
41
42The population is handled by the ``makedevs`` utility during image
43creation:
44
45Using ``devtmpfs`` and a Device Manager
46=======================================
47
48To use the dynamic method for device population, you need to use (or be
49sure to set) the :term:`USE_DEVFS`
50variable to "1", which is the default::
51
52 USE_DEVFS = "1"
53
54With this
55setting, the resulting ``/dev`` directory is populated by the kernel
56using ``devtmpfs``. Make sure the corresponding kernel configuration
57variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
58kernel.
59
60All devices created by ``devtmpfs`` will be owned by ``root`` and have
61permissions ``0600``.
62
63To have more control over the device nodes, you can use a device manager like
64``udev`` or ``busybox-mdev``. You choose the device manager by defining the
65:term:`VIRTUAL-RUNTIME_dev_manager <VIRTUAL-RUNTIME>` variable in your machine
66or distro configuration file. Alternatively, you can set this variable in
67your ``local.conf`` configuration file::
68
69 VIRTUAL-RUNTIME_dev_manager = "udev"
70
71 # Some alternative values
72 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
73 # VIRTUAL-RUNTIME_dev_manager = "systemd"
74
diff --git a/documentation/dev-manual/disk-space.rst b/documentation/dev-manual/disk-space.rst
new file mode 100644
index 0000000000..efca82601d
--- /dev/null
+++ b/documentation/dev-manual/disk-space.rst
@@ -0,0 +1,61 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Conserving Disk Space
4*********************
5
6Conserving Disk Space During Builds
7===================================
8
9To help conserve disk space during builds, you can add the following
10statement to your project's ``local.conf`` configuration file found in
11the :term:`Build Directory`::
12
13 INHERIT += "rm_work"
14
15Adding this statement deletes the work directory used for
16building a recipe once the recipe is built. For more information on
17"rm_work", see the :ref:`ref-classes-rm-work` class in the
18Yocto Project Reference Manual.
19
20When you inherit this class and build a ``core-image-sato`` image for a
21``qemux86-64`` machine from an Ubuntu 22.04 x86-64 system, you end up with a
22final disk usage of 22 Gbytes instead of &MIN_DISK_SPACE; Gbytes. However,
23&MIN_DISK_SPACE_RM_WORK; Gbytes of initial free disk space are still needed to
24create temporary files before they can be deleted.
25
26Purging Obsolete Shared State Cache Files
27=========================================
28
29After multiple build iterations, the Shared State (sstate) cache can contain
30multiple cache files for a given package, consuming a substantial amount of
31disk space. However, only the most recent ones are likely to be reused.
32
33The following command is a quick way to purge all the cache files which
34haven't been used for a least a specified number of days::
35
36 find build/sstate-cache -type f -mtime +$DAYS -delete
37
38The above command relies on the fact that BitBake touches the sstate cache
39files as it accesses them, when it has write access to the cache.
40
41You could use ``-atime`` instead of ``-mtime`` if the partition isn't mounted
42with the ``noatime`` option for a read only cache.
43
44For more advanced needs, OpenEmbedded-Core also offers a more elaborate
45command. It has the ability to purge all but the newest cache files on each
46architecture, and also to remove files that it considers unreachable by
47exploring a set of build configurations. However, this command
48requires a full build environment to be available and doesn't work well
49covering multiple releases. It won't work either on limited environments
50such as BSD based NAS::
51
52 sstate-cache-management.py --remove-duplicated --cache-dir=sstate-cache
53
54This command will ask you to confirm the deletions it identifies.
55Run ``sstate-cache-management.sh`` for more details about this script.
56
57.. note::
58
59 As this command is much more cautious and selective, removing only cache files,
60 it will execute much slower than the simple ``find`` command described above.
61 Therefore, it may not be your best option to trim huge cache directories.
diff --git a/documentation/dev-manual/efficiently-fetching-sources.rst b/documentation/dev-manual/efficiently-fetching-sources.rst
new file mode 100644
index 0000000000..a15f0a92ce
--- /dev/null
+++ b/documentation/dev-manual/efficiently-fetching-sources.rst
@@ -0,0 +1,68 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Efficiently Fetching Source Files During a Build
4************************************************
5
6The OpenEmbedded build system works with source files located through
7the :term:`SRC_URI` variable. When
8you build something using BitBake, a big part of the operation is
9locating and downloading all the source tarballs. For images,
10downloading all the source for various packages can take a significant
11amount of time.
12
13This section shows you how you can use mirrors to speed up fetching
14source files and how you can pre-fetch files all of which leads to more
15efficient use of resources and time.
16
17Setting up Effective Mirrors
18============================
19
20A good deal that goes into a Yocto Project build is simply downloading
21all of the source tarballs. Maybe you have been working with another
22build system for which you have built up a
23sizable directory of source tarballs. Or, perhaps someone else has such
24a directory for which you have read access. If so, you can save time by
25adding statements to your configuration file so that the build process
26checks local directories first for existing tarballs before checking the
27Internet.
28
29Here is an efficient way to set it up in your ``local.conf`` file::
30
31 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
32 INHERIT += "own-mirrors"
33 BB_GENERATE_MIRROR_TARBALLS = "1"
34 # BB_NO_NETWORK = "1"
35
36In the previous example, the
37:term:`BB_GENERATE_MIRROR_TARBALLS`
38variable causes the OpenEmbedded build system to generate tarballs of
39the Git repositories and store them in the
40:term:`DL_DIR` directory. Due to
41performance reasons, generating and storing these tarballs is not the
42build system's default behavior.
43
44You can also use the
45:term:`PREMIRRORS` variable. For
46an example, see the variable's glossary entry in the Yocto Project
47Reference Manual.
48
49Getting Source Files and Suppressing the Build
50==============================================
51
52Another technique you can use to ready yourself for a successive string
53of build operations, is to pre-fetch all the source files without
54actually starting a build. This technique lets you work through any
55download issues and ultimately gathers all the source files into your
56download directory :ref:`structure-build-downloads`,
57which is located with :term:`DL_DIR`.
58
59Use the following BitBake command form to fetch all the necessary
60sources without starting the build::
61
62 $ bitbake target --runall=fetch
63
64This
65variation of the BitBake command guarantees that you have all the
66sources for that BitBake target should you disconnect from the Internet
67and want to do the build later offline.
68
diff --git a/documentation/dev-manual/error-reporting-tool.rst b/documentation/dev-manual/error-reporting-tool.rst
new file mode 100644
index 0000000000..84f3d9cd1e
--- /dev/null
+++ b/documentation/dev-manual/error-reporting-tool.rst
@@ -0,0 +1,84 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using the Error Reporting Tool
4******************************
5
6The error reporting tool allows you to submit errors encountered during
7builds to a central database. Outside of the build environment, you can
8use a web interface to browse errors, view statistics, and query for
9errors. The tool works using a client-server system where the client
10portion is integrated with the installed Yocto Project
11:term:`Source Directory` (e.g. ``poky``).
12The server receives the information collected and saves it in a
13database.
14
15There is a live instance of the error reporting server at
16https://errors.yoctoproject.org.
17When you want to get help with build failures, you can submit all of the
18information on the failure easily and then point to the URL in your bug
19report or send an email to the mailing list.
20
21.. note::
22
23 If you send error reports to this server, the reports become publicly
24 visible.
25
26Enabling and Using the Tool
27===========================
28
29By default, the error reporting tool is disabled. You can enable it by
30inheriting the :ref:`ref-classes-report-error` class by adding the
31following statement to the end of your ``local.conf`` file in your
32:term:`Build Directory`::
33
34 INHERIT += "report-error"
35
36By default, the error reporting feature stores information in
37``${``\ :term:`LOG_DIR`\ ``}/error-report``.
38However, you can specify a directory to use by adding the following to
39your ``local.conf`` file::
40
41 ERR_REPORT_DIR = "path"
42
43Enabling error
44reporting causes the build process to collect the errors and store them
45in a file as previously described. When the build system encounters an
46error, it includes a command as part of the console output. You can run
47the command to send the error file to the server. For example, the
48following command sends the errors to an upstream server::
49
50 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
51
52In the previous example, the errors are sent to a public database
53available at https://errors.yoctoproject.org, which is used by the
54entire community. If you specify a particular server, you can send the
55errors to a different database. Use the following command for more
56information on available options::
57
58 $ send-error-report --help
59
60When sending the error file, you are prompted to review the data being
61sent as well as to provide a name and optional email address. Once you
62satisfy these prompts, the command returns a link from the server that
63corresponds to your entry in the database. For example, here is a
64typical link: https://errors.yoctoproject.org/Errors/Details/9522/
65
66Following the link takes you to a web interface where you can browse,
67query the errors, and view statistics.
68
69Disabling the Tool
70==================
71
72To disable the error reporting feature, simply remove or comment out the
73following statement from the end of your ``local.conf`` file in your
74:term:`Build Directory`::
75
76 INHERIT += "report-error"
77
78Setting Up Your Own Error Reporting Server
79==========================================
80
81If you want to set up your own error reporting server, you can obtain
82the code from the Git repository at :yocto_git:`/error-report-web/`.
83Instructions on how to set it up are in the README document.
84
diff --git a/documentation/dev-manual/external-scm.rst b/documentation/dev-manual/external-scm.rst
new file mode 100644
index 0000000000..97a7e63e36
--- /dev/null
+++ b/documentation/dev-manual/external-scm.rst
@@ -0,0 +1,67 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using an External SCM
4*********************
5
6If you're working on a recipe that pulls from an external Source Code
7Manager (SCM), it is possible to have the OpenEmbedded build system
8notice new recipe changes added to the SCM and then build the resulting
9packages that depend on the new recipes by using the latest versions.
10This only works for SCMs from which it is possible to get a sensible
11revision number for changes. Currently, you can do this with Apache
12Subversion (SVN), Git, and Bazaar (BZR) repositories.
13
14To enable this behavior, the :term:`PV` of
15the recipe needs to reference
16:term:`SRCPV`. Here is an example::
17
18 PV = "1.2.3+git${SRCPV}"
19
20Then, you can add the following to your
21``local.conf``::
22
23 SRCREV:pn-PN = "${AUTOREV}"
24
25:term:`PN` is the name of the recipe for
26which you want to enable automatic source revision updating.
27
28If you do not want to update your local configuration file, you can add
29the following directly to the recipe to finish enabling the feature::
30
31 SRCREV = "${AUTOREV}"
32
33The Yocto Project provides a distribution named ``poky-bleeding``, whose
34configuration file contains the line::
35
36 require conf/distro/include/poky-floating-revisions.inc
37
38This line pulls in the
39listed include file that contains numerous lines of exactly that form::
40
41 #SRCREV:pn-opkg-native ?= "${AUTOREV}"
42 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}"
43 #SRCREV:pn-opkg ?= "${AUTOREV}"
44 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}"
45 #SRCREV:pn-opkg-utils ?= "${AUTOREV}"
46 SRCREV:pn-gconf-dbus ?= "${AUTOREV}"
47 SRCREV:pn-matchbox-common ?= "${AUTOREV}"
48 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}"
49 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}"
50 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}"
51 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}"
52 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}"
53 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}"
54 SRCREV:pn-matchbox-wm ?= "${AUTOREV}"
55 SRCREV:pn-settings-daemon ?= "${AUTOREV}"
56 SRCREV:pn-screenshot ?= "${AUTOREV}"
57 . . .
58
59These lines allow you to
60experiment with building a distribution that tracks the latest
61development source for numerous packages.
62
63.. note::
64
65 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
66 this in mind if you use it.
67
diff --git a/documentation/dev-manual/external-toolchain.rst b/documentation/dev-manual/external-toolchain.rst
new file mode 100644
index 0000000000..238f8cf467
--- /dev/null
+++ b/documentation/dev-manual/external-toolchain.rst
@@ -0,0 +1,40 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Optionally Using an External Toolchain
4**************************************
5
6You might want to use an external toolchain as part of your development.
7If this is the case, the fundamental steps you need to accomplish are as
8follows:
9
10- Understand where the installed toolchain resides. For cases where you
11 need to build the external toolchain, you would need to take separate
12 steps to build and install the toolchain.
13
14- Make sure you add the layer that contains the toolchain to your
15 ``bblayers.conf`` file through the
16 :term:`BBLAYERS` variable.
17
18- Set the :term:`EXTERNAL_TOOLCHAIN` variable in your ``local.conf`` file
19 to the location in which you installed the toolchain.
20
21The toolchain configuration is very flexible and customizable. It
22is primarily controlled with the :term:`TCMODE` variable. This variable
23controls which ``tcmode-*.inc`` file to include from the
24``meta/conf/distro/include`` directory within the :term:`Source Directory`.
25
26The default value of :term:`TCMODE` is "default", which tells the
27OpenEmbedded build system to use its internally built toolchain (i.e.
28``tcmode-default.inc``). However, other patterns are accepted. In
29particular, "external-\*" refers to external toolchains. One example is
30the Mentor Graphics Sourcery G++ Toolchain. Support for this toolchain resides
31in the separate ``meta-sourcery`` layer at
32https://github.com/MentorEmbedded/meta-sourcery/.
33See its ``README`` file for details about how to use this layer.
34
35Another example of external toolchain layer is
36:yocto_git:`meta-arm-toolchain </meta-arm/tree/meta-arm-toolchain/>`
37supporting GNU toolchains released by ARM.
38
39You can find further information by reading about the :term:`TCMODE` variable
40in the Yocto Project Reference Manual's variable glossary.
diff --git a/documentation/dev-manual/figures/cute-files-npm-example.png b/documentation/dev-manual/figures/cute-files-npm-example.png
index 1ebe74f535..a02cca097f 100644
--- a/documentation/dev-manual/figures/cute-files-npm-example.png
+++ b/documentation/dev-manual/figures/cute-files-npm-example.png
Binary files differ
diff --git a/documentation/dev-manual/gobject-introspection.rst b/documentation/dev-manual/gobject-introspection.rst
new file mode 100644
index 0000000000..f7206e6fae
--- /dev/null
+++ b/documentation/dev-manual/gobject-introspection.rst
@@ -0,0 +1,155 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Enabling GObject Introspection Support
4**************************************
5
6`GObject introspection <https://gi.readthedocs.io/en/latest/>`__
7is the standard mechanism for accessing GObject-based software from
8runtime environments. GObject is a feature of the GLib library that
9provides an object framework for the GNOME desktop and related software.
10GObject Introspection adds information to GObject that allows objects
11created within it to be represented across different programming
12languages. If you want to construct GStreamer pipelines using Python, or
13control UPnP infrastructure using Javascript and GUPnP, GObject
14introspection is the only way to do it.
15
16This section describes the Yocto Project support for generating and
17packaging GObject introspection data. GObject introspection data is a
18description of the API provided by libraries built on top of the GLib
19framework, and, in particular, that framework's GObject mechanism.
20GObject Introspection Repository (GIR) files go to ``-dev`` packages,
21``typelib`` files go to main packages as they are packaged together with
22libraries that are introspected.
23
24The data is generated when building such a library, by linking the
25library with a small executable binary that asks the library to describe
26itself, and then executing the binary and processing its output.
27
28Generating this data in a cross-compilation environment is difficult
29because the library is produced for the target architecture, but its
30code needs to be executed on the build host. This problem is solved with
31the OpenEmbedded build system by running the code through QEMU, which
32allows precisely that. Unfortunately, QEMU does not always work
33perfectly as mentioned in the ":ref:`dev-manual/gobject-introspection:known issues`"
34section.
35
36Enabling the Generation of Introspection Data
37=============================================
38
39Enabling the generation of introspection data (GIR files) in your
40library package involves the following:
41
42#. Inherit the :ref:`ref-classes-gobject-introspection` class.
43
44#. Make sure introspection is not disabled anywhere in the recipe or
45 from anything the recipe includes. Also, make sure that
46 "gobject-introspection-data" is not in
47 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
48 and that "qemu-usermode" is not in
49 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
50 In either of these conditions, nothing will happen.
51
52#. Try to build the recipe. If you encounter build errors that look like
53 something is unable to find ``.so`` libraries, check where these
54 libraries are located in the source tree and add the following to the
55 recipe::
56
57 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
58
59 .. note::
60
61 See recipes in the ``oe-core`` repository that use that
62 :term:`GIR_EXTRA_LIBS_PATH` variable as an example.
63
64#. Look for any other errors, which probably mean that introspection
65 support in a package is not entirely standard, and thus breaks down
66 in a cross-compilation environment. For such cases, custom-made fixes
67 are needed. A good place to ask and receive help in these cases is
68 the :ref:`Yocto Project mailing
69 lists <resources-mailinglist>`.
70
71.. note::
72
73 Using a library that no longer builds against the latest Yocto
74 Project release and prints introspection related errors is a good
75 candidate for the previous procedure.
76
77Disabling the Generation of Introspection Data
78==============================================
79
80You might find that you do not want to generate introspection data. Or,
81perhaps QEMU does not work on your build host and target architecture
82combination. If so, you can use either of the following methods to
83disable GIR file generations:
84
85- Add the following to your distro configuration::
86
87 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
88
89 Adding this statement disables generating introspection data using
90 QEMU but will still enable building introspection tools and libraries
91 (i.e. building them does not require the use of QEMU).
92
93- Add the following to your machine configuration::
94
95 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
96
97 Adding this statement disables the use of QEMU when building packages for your
98 machine. Currently, this feature is used only by introspection
99 recipes and has the same effect as the previously described option.
100
101 .. note::
102
103 Future releases of the Yocto Project might have other features
104 affected by this option.
105
106If you disable introspection data, you can still obtain it through other
107means such as copying the data from a suitable sysroot, or by generating
108it on the target hardware. The OpenEmbedded build system does not
109currently provide specific support for these techniques.
110
111Testing that Introspection Works in an Image
112============================================
113
114Use the following procedure to test if generating introspection data is
115working in an image:
116
117#. Make sure that "gobject-introspection-data" is not in
118 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
119 and that "qemu-usermode" is not in
120 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
121
122#. Build ``core-image-sato``.
123
124#. Launch a Terminal and then start Python in the terminal.
125
126#. Enter the following in the terminal::
127
128 >>> from gi.repository import GLib
129 >>> GLib.get_host_name()
130
131#. For something a little more advanced, enter the following see:
132 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
133
134Known Issues
135============
136
137Here are know issues in GObject Introspection Support:
138
139- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
140 introspection data on that architecture.
141
142- x32 is not supported by QEMU. Consequently, introspection data is
143 disabled.
144
145- musl causes transient GLib binaries to crash on assertion failures.
146 Consequently, generating introspection data is disabled.
147
148- Because QEMU is not able to run the binaries correctly, introspection
149 is disabled for some specific packages under specific architectures
150 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
151
152- QEMU usermode might not work properly when running 64-bit binaries
153 under 32-bit host machines. In particular, "qemumips64" is known to
154 not work under i686.
155
diff --git a/documentation/dev-manual/history.rst b/documentation/dev-manual/history.rst
deleted file mode 100644
index 1ba312422c..0000000000
--- a/documentation/dev-manual/history.rst
+++ /dev/null
@@ -1,67 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3***********************
4Manual Revision History
5***********************
6
7.. list-table::
8 :widths: 10 15 40
9 :header-rows: 1
10
11 * - Revision
12 - Date
13 - Note
14 * - 1.1
15 - October 2011
16 - The initial document released with the Yocto Project 1.1 Release
17 * - 1.2
18 - April 2012
19 - Released with the Yocto Project 1.2 Release.
20 * - 1.3
21 - October 2012
22 - Released with the Yocto Project 1.3 Release.
23 * - 1.4
24 - April 2013
25 - Released with the Yocto Project 1.4 Release.
26 * - 1.5
27 - October 2013
28 - Released with the Yocto Project 1.5 Release.
29 * - 1.6
30 - April 2014
31 - Released with the Yocto Project 1.6 Release.
32 * - 1.7
33 - October 2014
34 - Released with the Yocto Project 1.7 Release.
35 * - 1.8
36 - April 2015
37 - Released with the Yocto Project 1.8 Release.
38 * - 2.0
39 - October 2015
40 - Released with the Yocto Project 2.0 Release.
41 * - 2.1
42 - April 2016
43 - Released with the Yocto Project 2.1 Release.
44 * - 2.2
45 - October 2016
46 - Released with the Yocto Project 2.2 Release.
47 * - 2.3
48 - May 2017
49 - Released with the Yocto Project 2.3 Release.
50 * - 2.4
51 - October 2017
52 - Released with the Yocto Project 2.4 Release.
53 * - 2.5
54 - May 2018
55 - Released with the Yocto Project 2.5 Release.
56 * - 2.6
57 - November 2018
58 - Released with the Yocto Project 2.6 Release.
59 * - 2.7
60 - May 2019
61 - Released with the Yocto Project 2.7 Release.
62 * - 3.0
63 - October 2019
64 - Released with the Yocto Project 3.0 Release.
65 * - 3.1
66 - April 2020
67 - Released with the Yocto Project 3.1 Release.
diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
index 941db2df8c..9ccf60f701 100644
--- a/documentation/dev-manual/index.rst
+++ b/documentation/dev-manual/index.rst
@@ -4,16 +4,49 @@
4Yocto Project Development Tasks Manual 4Yocto Project Development Tasks Manual
5====================================== 5======================================
6 6
7|
8
9.. toctree:: 7.. toctree::
10 :caption: Table of Contents 8 :caption: Table of Contents
11 :numbered: 9 :numbered:
12 10
13 intro 11 intro
14 start 12 start
15 common-tasks 13 layers
14 customizing-images
15 new-recipe
16 new-machine
17 upgrading-recipes
18 temporary-source-code
19 quilt.rst
20 development-shell
21 python-development-shell
22 building
23 speeding-up-build
24 libraries
25 prebuilt-libraries
26 x32-psabi
27 gobject-introspection
28 external-toolchain
29 wic
30 bmaptool
31 securing-images
32 custom-distribution
33 custom-template-configuration-directory
34 disk-space
35 packages
36 efficiently-fetching-sources
37 init-manager
38 device-manager
39 external-scm
40 read-only-rootfs
41 build-quality
42 runtime-testing
43 debugging
44 licenses
45 security-subjects
46 vulnerabilities
47 sbom
48 error-reporting-tool
49 wayland
16 qemu 50 qemu
17 history
18 51
19.. include:: /boilerplate.rst 52.. include:: /boilerplate.rst
diff --git a/documentation/dev-manual/init-manager.rst b/documentation/dev-manual/init-manager.rst
new file mode 100644
index 0000000000..ddce82b81f
--- /dev/null
+++ b/documentation/dev-manual/init-manager.rst
@@ -0,0 +1,162 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3.. _init-manager:
4
5Selecting an Initialization Manager
6***********************************
7
8By default, the Yocto Project uses :wikipedia:`SysVinit <Init#SysV-style>` as
9the initialization manager. There is also support for BusyBox init, a simpler
10implementation, as well as support for :wikipedia:`systemd <Systemd>`, which
11is a full replacement for init with parallel starting of services, reduced
12shell overhead, increased security and resource limits for services, and other
13features that are used by many distributions.
14
15Within the system, SysVinit and BusyBox init treat system components as
16services. These services are maintained as shell scripts stored in the
17``/etc/init.d/`` directory.
18
19SysVinit is more elaborate than BusyBox init and organizes services in
20different run levels. This organization is maintained by putting links
21to the services in the ``/etc/rcN.d/`` directories, where `N/` is one
22of the following options: "S", "0", "1", "2", "3", "4", "5", or "6".
23
24.. note::
25
26 Each runlevel has a dependency on the previous runlevel. This
27 dependency allows the services to work properly.
28
29Both SysVinit and BusyBox init are configured through the ``/etc/inittab``
30file, with a very similar syntax, though of course BusyBox init features
31are more limited.
32
33In comparison, systemd treats components as units. Using units is a
34broader concept as compared to using a service. A unit includes several
35different types of entities. ``Service`` is one of the types of entities.
36The runlevel concept in SysVinit corresponds to the concept of a target
37in systemd, where target is also a type of supported unit.
38
39In systems with SysVinit or BusyBox init, services load sequentially (i.e. one
40by one) during init and parallelization is not supported. With systemd, services
41start in parallel. This method can have an impact on the startup performance
42of a given service, though systemd will also provide more services by default,
43therefore increasing the total system boot time. systemd also substantially
44increases system size because of its multiple components and the extra
45dependencies it pulls.
46
47On the contrary, BusyBox init is the simplest and the lightest solution and
48also comes with BusyBox mdev as device manager, a lighter replacement to
49:wikipedia:`udev <Udev>`, which SysVinit and systemd both use.
50
51The ":ref:`device-manager`" chapter has more details about device managers.
52
53Using SysVinit with udev
54=========================
55
56SysVinit with the udev device manager corresponds to the
57default setting in Poky. This corresponds to setting::
58
59 INIT_MANAGER = "sysvinit"
60
61Using BusyBox init with BusyBox mdev
62====================================
63
64BusyBox init with BusyBox mdev is the simplest and lightest solution
65for small root filesystems. All you need is BusyBox, which most systems
66have anyway::
67
68 INIT_MANAGER = "mdev-busybox"
69
70Using systemd
71=============
72
73The last option is to use systemd together with the udev device
74manager. This is the most powerful and versatile solution, especially
75for more complex systems::
76
77 INIT_MANAGER = "systemd"
78
79This will enable systemd and remove sysvinit components from the image.
80See :yocto_git:`meta/conf/distro/include/init-manager-systemd.inc
81</poky/tree/meta/conf/distro/include/init-manager-systemd.inc>` for exact
82details on what this does.
83
84Controling systemd from the target command line
85-----------------------------------------------
86
87Here is a quick reference for controling systemd from the command line on the
88target. Instead of opening and sometimes modifying files, most interaction
89happens through the ``systemctl`` and ``journalctl`` commands:
90
91- ``systemctl status``: show the status of all services
92- ``systemctl status <service>``: show the status of one service
93- ``systemctl [start|stop] <service>``: start or stop a service
94- ``systemctl [enable|disable] <service>``: enable or disable a service at boot time
95- ``systemctl list-units``: list all available units
96- ``journalctl -a``: show all logs for all services
97- ``journalctl -f``: show only the last log entries, and keep printing updates as they arrive
98- ``journalctl -u``: show only logs from a particular service
99
100Using systemd-journald without a traditional syslog daemon
101----------------------------------------------------------
102
103Counter-intuitively, ``systemd-journald`` is not a syslog runtime or provider,
104and the proper way to use ``systemd-journald`` as your sole logging mechanism is to
105effectively disable syslog entirely by setting these variables in your distribution
106configuration file::
107
108 VIRTUAL-RUNTIME_syslog = ""
109 VIRTUAL-RUNTIME_base-utils-syslog = ""
110
111Doing so will prevent ``rsyslog`` / ``busybox-syslog`` from being pulled in by
112default, leaving only ``systemd-journald``.
113
114Summary
115-------
116
117The Yocto Project supports three different initialization managers, offering
118increasing levels of complexity and functionality:
119
120.. list-table::
121 :widths: 40 20 20 20
122 :header-rows: 1
123
124 * -
125 - BusyBox init
126 - SysVinit
127 - systemd
128 * - Size
129 - Small
130 - Small
131 - Big [#footnote-systemd-size]_
132 * - Complexity
133 - Small
134 - Medium
135 - High
136 * - Support for boot profiles
137 - No
138 - Yes ("runlevels")
139 - Yes ("targets")
140 * - Services defined as
141 - Shell scripts
142 - Shell scripts
143 - Description files
144 * - Starting services in parallel
145 - No
146 - No
147 - Yes
148 * - Setting service resource limits
149 - No
150 - No
151 - Yes
152 * - Support service isolation
153 - No
154 - No
155 - Yes
156 * - Integrated logging
157 - No
158 - No
159 - Yes
160
161.. [#footnote-systemd-size] Using systemd increases the ``core-image-minimal``
162 image size by 160\% for ``qemux86-64`` on Mickledore (4.2), compared to SysVinit.
diff --git a/documentation/dev-manual/intro.rst b/documentation/dev-manual/intro.rst
index 23c848e870..0f7370a96d 100644
--- a/documentation/dev-manual/intro.rst
+++ b/documentation/dev-manual/intro.rst
@@ -7,16 +7,16 @@ The Yocto Project Development Tasks Manual
7Welcome 7Welcome
8======= 8=======
9 9
10Welcome to the Yocto Project Development Tasks Manual! This manual 10Welcome to the Yocto Project Development Tasks Manual. This manual
11provides relevant procedures necessary for developing in the Yocto 11provides relevant procedures necessary for developing in the Yocto
12Project environment (i.e. developing embedded Linux images and 12Project environment (i.e. developing embedded Linux images and
13user-space applications that run on targeted devices). The manual groups 13user-space applications that run on targeted devices). This manual groups
14related procedures into higher-level sections. Procedures can consist of 14related procedures into higher-level sections. Procedures can consist of
15high-level steps or low-level steps depending on the topic. 15high-level steps or low-level steps depending on the topic.
16 16
17This manual provides the following: 17This manual provides the following:
18 18
19- Procedures that help you get going with the Yocto Project. For 19- Procedures that help you get going with the Yocto Project; for
20 example, procedures that show you how to set up a build host and work 20 example, procedures that show you how to set up a build host and work
21 with the Yocto Project source repositories. 21 with the Yocto Project source repositories.
22 22
@@ -24,25 +24,25 @@ This manual provides the following:
24 Changes can be improvements, new features, or bug fixes. 24 Changes can be improvements, new features, or bug fixes.
25 25
26- Procedures related to "everyday" tasks you perform while developing 26- Procedures related to "everyday" tasks you perform while developing
27 images and applications using the Yocto Project. For example, 27 images and applications using the Yocto Project, such as
28 procedures to create a layer, customize an image, write a new recipe, 28 creating a new layer, customizing an image, writing a new recipe,
29 and so forth. 29 and so forth.
30 30
31This manual does not provide the following: 31This manual does not provide the following:
32 32
33- Redundant Step-by-step Instructions: For example, the 33- Redundant step-by-step instructions: For example, the
34 :doc:`/sdk-manual/index` manual contains detailed 34 :doc:`/sdk-manual/index` manual contains detailed
35 instructions on how to install an SDK, which is used to develop 35 instructions on how to install an SDK, which is used to develop
36 applications for target hardware. 36 applications for target hardware.
37 37
38- Reference or Conceptual Material: This type of material resides in an 38- Reference or conceptual material: This type of material resides in an
39 appropriate reference manual. For example, system variables are 39 appropriate reference manual. As an example, system variables are
40 documented in the :doc:`/ref-manual/index`. 40 documented in the :doc:`/ref-manual/index`.
41 41
42- Detailed Public Information Not Specific to the Yocto Project: For 42- Detailed public information not specific to the Yocto Project: For
43 example, exhaustive information on how to use the Source Control 43 example, exhaustive information on how to use the Git version
44 Manager Git is better covered with Internet searches and official Git 44 control system is better covered with Internet searches and official Git
45 Documentation than through the Yocto Project documentation. 45 documentation than through the Yocto Project documentation.
46 46
47Other Information 47Other Information
48================= 48=================
diff --git a/documentation/dev-manual/layers.rst b/documentation/dev-manual/layers.rst
new file mode 100644
index 0000000000..91889bd0ae
--- /dev/null
+++ b/documentation/dev-manual/layers.rst
@@ -0,0 +1,919 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Understanding and Creating Layers
4*********************************
5
6The OpenEmbedded build system supports organizing
7:term:`Metadata` into multiple layers.
8Layers allow you to isolate different types of customizations from each
9other. For introductory information on the Yocto Project Layer Model,
10see the
11":ref:`overview-manual/yp-intro:the yocto project layer model`"
12section in the Yocto Project Overview and Concepts Manual.
13
14Creating Your Own Layer
15=======================
16
17.. note::
18
19 It is very easy to create your own layers to use with the OpenEmbedded
20 build system, as the Yocto Project ships with tools that speed up creating
21 layers. This section describes the steps you perform by hand to create
22 layers so that you can better understand them. For information about the
23 layer-creation tools, see the
24 ":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
25 section in the Yocto Project Board Support Package (BSP) Developer's
26 Guide and the ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
27 section further down in this manual.
28
29Follow these general steps to create your layer without using tools:
30
31#. *Check Existing Layers:* Before creating a new layer, you should be
32 sure someone has not already created a layer containing the Metadata
33 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
34 for a list of layers from the OpenEmbedded community that can be used in
35 the Yocto Project. You could find a layer that is identical or close
36 to what you need.
37
38#. *Create a Directory:* Create the directory for your layer. When you
39 create the layer, be sure to create the directory in an area not
40 associated with the Yocto Project :term:`Source Directory`
41 (e.g. the cloned ``poky`` repository).
42
43 While not strictly required, prepend the name of the directory with
44 the string "meta-". For example::
45
46 meta-mylayer
47 meta-GUI_xyz
48 meta-mymachine
49
50 With rare exceptions, a layer's name follows this form::
51
52 meta-root_name
53
54 Following this layer naming convention can save
55 you trouble later when tools, components, or variables "assume" your
56 layer name begins with "meta-". A notable example is in configuration
57 files as shown in the following step where layer names without the
58 "meta-" string are appended to several variables used in the
59 configuration.
60
61#. *Create a Layer Configuration File:* Inside your new layer folder,
62 you need to create a ``conf/layer.conf`` file. It is easiest to take
63 an existing layer configuration file and copy that to your layer's
64 ``conf`` directory and then modify the file as needed.
65
66 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
67 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
68 demonstrates the required syntax. For your layer, you need to replace
69 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
70 for a layer named "meta-machinexyz")::
71
72 # We have a conf and classes directory, add to BBPATH
73 BBPATH .= ":${LAYERDIR}"
74
75 # We have recipes-* directories, add to BBFILES
76 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
77 ${LAYERDIR}/recipes-*/*/*.bbappend"
78
79 BBFILE_COLLECTIONS += "yoctobsp"
80 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
81 BBFILE_PRIORITY_yoctobsp = "5"
82 LAYERVERSION_yoctobsp = "4"
83 LAYERSERIES_COMPAT_yoctobsp = "dunfell"
84
85 Here is an explanation of the layer configuration file:
86
87 - :term:`BBPATH`: Adds the layer's
88 root directory to BitBake's search path. Through the use of the
89 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
90 configuration files, and files that are included with ``include``
91 and ``require`` statements. For these cases, BitBake uses the
92 first file that matches the name found in :term:`BBPATH`. This is
93 similar to the way the ``PATH`` variable is used for binaries. It
94 is recommended, therefore, that you use unique class and
95 configuration filenames in your custom layer.
96
97 - :term:`BBFILES`: Defines the
98 location for all recipes in the layer.
99
100 - :term:`BBFILE_COLLECTIONS`:
101 Establishes the current layer through a unique identifier that is
102 used throughout the OpenEmbedded build system to refer to the
103 layer. In this example, the identifier "yoctobsp" is the
104 representation for the container layer named "meta-yocto-bsp".
105
106 - :term:`BBFILE_PATTERN`:
107 Expands immediately during parsing to provide the directory of the
108 layer.
109
110 - :term:`BBFILE_PRIORITY`:
111 Establishes a priority to use for recipes in the layer when the
112 OpenEmbedded build finds recipes of the same name in different
113 layers.
114
115 - :term:`LAYERVERSION`:
116 Establishes a version number for the layer. You can use this
117 version number to specify this exact version of the layer as a
118 dependency when using the
119 :term:`LAYERDEPENDS`
120 variable.
121
122 - :term:`LAYERDEPENDS`:
123 Lists all layers on which this layer depends (if any).
124
125 - :term:`LAYERSERIES_COMPAT`:
126 Lists the :yocto_wiki:`Yocto Project </Releases>`
127 releases for which the current version is compatible. This
128 variable is a good way to indicate if your particular layer is
129 current.
130
131
132 .. note::
133
134 A layer does not have to contain only recipes ``.bb`` or append files
135 ``.bbappend``. Generally, developers create layers using
136 ``bitbake-layers create-layer``.
137 See ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`",
138 explaining how the ``layer.conf`` file is created from a template located in
139 ``meta/lib/bblayers/templates/layer.conf``.
140 In fact, none of the variables set in ``layer.conf`` are mandatory,
141 except when :term:`BBFILE_COLLECTIONS` is present. In this case
142 :term:`LAYERSERIES_COMPAT` and :term:`BBFILE_PATTERN` have to be
143 defined too.
144
145#. *Add Content:* Depending on the type of layer, add the content. If
146 the layer adds support for a machine, add the machine configuration
147 in a ``conf/machine/`` file within the layer. If the layer adds
148 distro policy, add the distro configuration in a ``conf/distro/``
149 file within the layer. If the layer introduces new recipes, put the
150 recipes you need in ``recipes-*`` subdirectories within the layer.
151
152 .. note::
153
154 For an explanation of layer hierarchy that is compliant with the
155 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
156 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
157
158#. *Optionally Test for Compatibility:* If you want permission to use
159 the Yocto Project Compatibility logo with your layer or application
160 that uses your layer, perform the steps to apply for compatibility.
161 See the
162 ":ref:`dev-manual/layers:making sure your layer is compatible with yocto project`"
163 section for more information.
164
165Following Best Practices When Creating Layers
166=============================================
167
168To create layers that are easier to maintain and that will not impact
169builds for other machines, you should consider the information in the
170following list:
171
172- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
173 Configuration:* In other words, do not copy an entire recipe into
174 your layer and then modify it. Rather, use an append file
175 (``.bbappend``) to override only those parts of the original recipe
176 you need to modify.
177
178- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
179 for each recipe that uses an include file. Or, if you are introducing
180 a new recipe that requires the included file, use the path relative
181 to the original layer directory to refer to the file. For example,
182 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
183 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
184 the include file, it could indicate a deficiency in the include file
185 in the layer to which it originally belongs. If this is the case, you
186 should try to address that deficiency instead of overlaying the
187 include file. For example, you could address this by getting the
188 maintainer of the include file to add a variable or variables to make
189 it easy to override the parts needing to be overridden.
190
191- *Structure Your Layers:* Proper use of overrides within append files
192 and placement of machine-specific files within your layer can ensure
193 that a build is not using the wrong Metadata and negatively impacting
194 a build for a different machine. Here are some examples:
195
196 - *Modify Variables to Support a Different Machine:* Suppose you
197 have a layer named ``meta-one`` that adds support for building
198 machine "one". To do so, you use an append file named
199 ``base-files.bbappend`` and create a dependency on "foo" by
200 altering the :term:`DEPENDS`
201 variable::
202
203 DEPENDS = "foo"
204
205 The dependency is created during any
206 build that includes the layer ``meta-one``. However, you might not
207 want this dependency for all machines. For example, suppose you
208 are building for machine "two" but your ``bblayers.conf`` file has
209 the ``meta-one`` layer included. During the build, the
210 ``base-files`` for machine "two" will also have the dependency on
211 ``foo``.
212
213 To make sure your changes apply only when building machine "one",
214 use a machine override with the :term:`DEPENDS` statement::
215
216 DEPENDS:one = "foo"
217
218 You should follow the same strategy when using ``:append``
219 and ``:prepend`` operations::
220
221 DEPENDS:append:one = " foo"
222 DEPENDS:prepend:one = "foo "
223
224 As an actual example, here's a
225 snippet from the generic kernel include file ``linux-yocto.inc``,
226 wherein the kernel compile and link options are adjusted in the
227 case of a subset of the supported architectures::
228
229 DEPENDS:append:aarch64 = " libgcc"
230 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
231 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
232
233 DEPENDS:append:nios2 = " libgcc"
234 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
235 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
236
237 DEPENDS:append:arc = " libgcc"
238 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
239 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
240
241 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
242
243 - *Place Machine-Specific Files in Machine-Specific Locations:* When
244 you have a base recipe, such as ``base-files.bb``, that contains a
245 :term:`SRC_URI` statement to a
246 file, you can use an append file to cause the build to use your
247 own version of the file. For example, an append file in your layer
248 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
249 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
250
251 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
252
253 The build for machine "one" will pick up your machine-specific file as
254 long as you have the file in
255 ``meta-one/recipes-core/base-files/base-files/``. However, if you
256 are building for a different machine and the ``bblayers.conf``
257 file includes the ``meta-one`` layer and the location of your
258 machine-specific file is the first location where that file is
259 found according to :term:`FILESPATH`, builds for all machines will
260 also use that machine-specific file.
261
262 You can make sure that a machine-specific file is used for a
263 particular machine by putting the file in a subdirectory specific
264 to the machine. For example, rather than placing the file in
265 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
266 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
267 Not only does this make sure the file is used only when building
268 for machine "one", but the build process locates the file more
269 quickly.
270
271 In summary, you need to place all files referenced from
272 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
273 order to restrict those files to machine-specific builds.
274
275- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
276 permission to use the Yocto Project Compatibility logo with your
277 layer or application that uses your layer, perform the steps to apply
278 for compatibility. See the
279 ":ref:`dev-manual/layers:making sure your layer is compatible with yocto project`"
280 section for more information.
281
282- *Follow the Layer Naming Convention:* Store custom layers in a Git
283 repository that use the ``meta-layer_name`` format.
284
285- *Group Your Layers Locally:* Clone your repository alongside other
286 cloned ``meta`` directories from the :term:`Source Directory`.
287
288Making Sure Your Layer is Compatible With Yocto Project
289=======================================================
290
291When you create a layer used with the Yocto Project, it is advantageous
292to make sure that the layer interacts well with existing Yocto Project
293layers (i.e. the layer is compatible with the Yocto Project). Ensuring
294compatibility makes the layer easy to be consumed by others in the Yocto
295Project community and could allow you permission to use the Yocto
296Project Compatible Logo.
297
298.. note::
299
300 Only Yocto Project member organizations are permitted to use the
301 Yocto Project Compatible Logo. The logo is not available for general
302 use. For information on how to become a Yocto Project member
303 organization, see the :yocto_home:`Yocto Project Website <>`.
304
305The Yocto Project Compatibility Program consists of a layer application
306process that requests permission to use the Yocto Project Compatibility
307Logo for your layer and application. The process consists of two parts:
308
309#. Successfully passing a script (``yocto-check-layer``) that when run
310 against your layer, tests it against constraints based on experiences
311 of how layers have worked in the real world and where pitfalls have
312 been found. Getting a "PASS" result from the script is required for
313 successful compatibility registration.
314
315#. Completion of an application acceptance form, which you can find at
316 :yocto_home:`/compatible-registration/`.
317
318To be granted permission to use the logo, you need to satisfy the
319following:
320
321- Be able to check the box indicating that you got a "PASS" when
322 running the script against your layer.
323
324- Answer "Yes" to the questions on the form or have an acceptable
325 explanation for any questions answered "No".
326
327- Be a Yocto Project Member Organization.
328
329The remainder of this section presents information on the registration
330form and on the ``yocto-check-layer`` script.
331
332Yocto Project Compatible Program Application
333--------------------------------------------
334
335Use the form to apply for your layer's approval. Upon successful
336application, you can use the Yocto Project Compatibility Logo with your
337layer and the application that uses your layer.
338
339To access the form, use this link:
340:yocto_home:`/compatible-registration`.
341Follow the instructions on the form to complete your application.
342
343The application consists of the following sections:
344
345- *Contact Information:* Provide your contact information as the fields
346 require. Along with your information, provide the released versions
347 of the Yocto Project for which your layer is compatible.
348
349- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
350 items in the checklist. There is space at the bottom of the form for
351 any explanations for items for which you answered "No".
352
353- *Recommendations:* Provide answers for the questions regarding Linux
354 kernel use and build success.
355
356``yocto-check-layer`` Script
357----------------------------
358
359The ``yocto-check-layer`` script provides you a way to assess how
360compatible your layer is with the Yocto Project. You should run this
361script prior to using the form to apply for compatibility as described
362in the previous section. You need to achieve a "PASS" result in order to
363have your application form successfully processed.
364
365The script divides tests into three areas: COMMON, BSP, and DISTRO. For
366example, given a distribution layer (DISTRO), the layer must pass both
367the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
368layer, the layer must pass the COMMON and BSP set of tests.
369
370To execute the script, enter the following commands from your build
371directory::
372
373 $ source oe-init-build-env
374 $ yocto-check-layer your_layer_directory
375
376Be sure to provide the actual directory for your
377layer as part of the command.
378
379Entering the command causes the script to determine the type of layer
380and then to execute a set of specific tests against the layer. The
381following list overviews the test:
382
383- ``common.test_readme``: Tests if a ``README`` file exists in the
384 layer and the file is not empty.
385
386- ``common.test_parse``: Tests to make sure that BitBake can parse the
387 files without error (i.e. ``bitbake -p``).
388
389- ``common.test_show_environment``: Tests that the global or per-recipe
390 environment is in order without errors (i.e. ``bitbake -e``).
391
392- ``common.test_world``: Verifies that ``bitbake world`` works.
393
394- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
395 layers do not come with recipes that change signatures.
396
397- ``common.test_layerseries_compat``: Verifies layer compatibility is
398 set properly.
399
400- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
401 configurations.
402
403- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
404 set the machine when the layer is added.
405
406- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
407 regardless of which machine is selected.
408
409- ``bsp.test_machine_signatures``: Verifies that building for a
410 particular machine affects only the signature of tasks specific to
411 that machine.
412
413- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
414 distro configurations.
415
416- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
417 does not set the distribution when the layer is added.
418
419Enabling Your Layer
420===================
421
422Before the OpenEmbedded build system can use your new layer, you need to
423enable it. To enable your layer, simply add your layer's path to the
424:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
425found in the :term:`Build Directory`. The following example shows how to
426enable your new ``meta-mylayer`` layer (note how your new layer exists
427outside of the official ``poky`` repository which you would have checked
428out earlier)::
429
430 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
431 # changes incompatibly
432 POKY_BBLAYERS_CONF_VERSION = "2"
433 BBPATH = "${TOPDIR}"
434 BBFILES ?= ""
435 BBLAYERS ?= " \
436 /home/user/poky/meta \
437 /home/user/poky/meta-poky \
438 /home/user/poky/meta-yocto-bsp \
439 /home/user/mystuff/meta-mylayer \
440 "
441
442BitBake parses each ``conf/layer.conf`` file from the top down as
443specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
444file. During the processing of each ``conf/layer.conf`` file, BitBake
445adds the recipes, classes and configurations contained within the
446particular layer to the source directory.
447
448Appending Other Layers Metadata With Your Layer
449===============================================
450
451A recipe that appends Metadata to another recipe is called a BitBake
452append file. A BitBake append file uses the ``.bbappend`` file type
453suffix, while the corresponding recipe to which Metadata is being
454appended uses the ``.bb`` file type suffix.
455
456You can use a ``.bbappend`` file in your layer to make additions or
457changes to the content of another layer's recipe without having to copy
458the other layer's recipe into your layer. Your ``.bbappend`` file
459resides in your layer, while the main ``.bb`` recipe file to which you
460are appending Metadata resides in a different layer.
461
462Being able to append information to an existing recipe not only avoids
463duplication, but also automatically applies recipe changes from a
464different layer into your layer. If you were copying recipes, you would
465have to manually merge changes as they occur.
466
467When you create an append file, you must use the same root name as the
468corresponding recipe file. For example, the append file
469``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
470means the original recipe and append filenames are version
471number-specific. If the corresponding recipe is renamed to update to a
472newer version, you must also rename and possibly update the
473corresponding ``.bbappend`` as well. During the build process, BitBake
474displays an error on starting if it detects a ``.bbappend`` file that
475does not have a corresponding recipe with a matching name. See the
476:term:`BB_DANGLINGAPPENDS_WARNONLY`
477variable for information on how to handle this error.
478
479Overlaying a File Using Your Layer
480----------------------------------
481
482As an example, consider the main formfactor recipe and a corresponding
483formfactor append file both from the :term:`Source Directory`.
484Here is the main
485formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
486the "meta" layer at ``meta/recipes-bsp/formfactor``::
487
488 SUMMARY = "Device formfactor information"
489 DESCRIPTION = "A formfactor configuration file provides information about the \
490 target hardware for which the image is being built and information that the \
491 build system cannot obtain from other sources such as the kernel."
492 SECTION = "base"
493 LICENSE = "MIT"
494 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
495 PR = "r45"
496
497 SRC_URI = "file://config file://machconfig"
498 S = "${WORKDIR}"
499
500 PACKAGE_ARCH = "${MACHINE_ARCH}"
501 INHIBIT_DEFAULT_DEPS = "1"
502
503 do_install() {
504 # Install file only if it has contents
505 install -d ${D}${sysconfdir}/formfactor/
506 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
507 if [ -s "${S}/machconfig" ]; then
508 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
509 fi
510 }
511
512In the main recipe, note the :term:`SRC_URI`
513variable, which tells the OpenEmbedded build system where to find files
514during the build.
515
516Here is the append file, which is named ``formfactor_0.0.bbappend``
517and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
518file is in the layer at ``recipes-bsp/formfactor``::
519
520 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
521
522By default, the build system uses the
523:term:`FILESPATH` variable to
524locate files. This append file extends the locations by setting the
525:term:`FILESEXTRAPATHS`
526variable. Setting this variable in the ``.bbappend`` file is the most
527reliable and recommended method for adding directories to the search
528path used by the build system to find files.
529
530The statement in this example extends the directories to include
531``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
532which resolves to a directory named ``formfactor`` in the same directory
533in which the append file resides (i.e.
534``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
535have the supporting directory structure set up that will contain any
536files or patches you will be including from the layer.
537
538Using the immediate expansion assignment operator ``:=`` is important
539because of the reference to :term:`THISDIR`. The trailing colon character is
540important as it ensures that items in the list remain colon-separated.
541
542.. note::
543
544 BitBake automatically defines the :term:`THISDIR` variable. You should
545 never set this variable yourself. Using ":prepend" as part of the
546 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
547 paths in the final list.
548
549 Also, not all append files add extra files. Many append files simply
550 allow to add build options (e.g. ``systemd``). For these cases, your
551 append file would not even use the :term:`FILESEXTRAPATHS` statement.
552
553The end result of this ``.bbappend`` file is that on a Raspberry Pi, where
554``rpi`` will exist in the list of :term:`OVERRIDES`, the file
555``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be
556used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in
557:ref:`ref-tasks-install` will return true, and the file will be installed.
558
559Installing Additional Files Using Your Layer
560--------------------------------------------
561
562As another example, consider the main ``xserver-xf86-config`` recipe and a
563corresponding ``xserver-xf86-config`` append file both from the :term:`Source
564Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named
565``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at
566``meta/recipes-graphics/xorg-xserver``::
567
568 SUMMARY = "X.Org X server configuration file"
569 HOMEPAGE = "http://www.x.org"
570 SECTION = "x11/base"
571 LICENSE = "MIT"
572 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
573 PR = "r33"
574
575 SRC_URI = "file://xorg.conf"
576
577 S = "${WORKDIR}"
578
579 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf"
580
581 PACKAGE_ARCH = "${MACHINE_ARCH}"
582 ALLOW_EMPTY:${PN} = "1"
583
584 do_install () {
585 if test -s ${WORKDIR}/xorg.conf; then
586 install -d ${D}/${sysconfdir}/X11
587 install -m 0644 ${WORKDIR}/xorg.conf ${D}/${sysconfdir}/X11/
588 fi
589 }
590
591Here is the append file, which is named ``xserver-xf86-config_%.bbappend``
592and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
593file is in the layer at ``recipes-graphics/xorg-xserver``::
594
595 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
596
597 SRC_URI:append:rpi = " \
598 file://xorg.conf.d/98-pitft.conf \
599 file://xorg.conf.d/99-calibration.conf \
600 "
601 do_install:append:rpi () {
602 PITFT="${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}"
603 if [ "${PITFT}" = "1" ]; then
604 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/
605 install -m 0644 ${WORKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
606 install -m 0644 ${WORKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
607 fi
608 }
609
610 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*"
611
612Building off of the previous example, we once again are setting the
613:term:`FILESEXTRAPATHS` variable. In this case we are also using
614:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in
615the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a
616check for an additional :term:`MACHINE_FEATURES` that if set will cause these
617additional files to be installed. These additional files are listed in
618:term:`FILES` so that they will be packaged.
619
620Prioritizing Your Layer
621=======================
622
623Each layer is assigned a priority value. Priority values control which
624layer takes precedence if there are recipe files with the same name in
625multiple layers. For these cases, the recipe file from the layer with a
626higher priority number takes precedence. Priority values also affect the
627order in which multiple ``.bbappend`` files for the same recipe are
628applied. You can either specify the priority manually, or allow the
629build system to calculate it based on the layer's dependencies.
630
631To specify the layer's priority manually, use the
632:term:`BBFILE_PRIORITY`
633variable and append the layer's root name::
634
635 BBFILE_PRIORITY_mylayer = "1"
636
637.. note::
638
639 It is possible for a recipe with a lower version number
640 :term:`PV` in a layer that has a higher
641 priority to take precedence.
642
643 Also, the layer priority does not currently affect the precedence
644 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
645 might address this.
646
647Managing Layers
648===============
649
650You can use the BitBake layer management tool ``bitbake-layers`` to
651provide a view into the structure of recipes across a multi-layer
652project. Being able to generate output that reports on configured layers
653with their paths and priorities and on ``.bbappend`` files and their
654applicable recipes can help to reveal potential problems.
655
656For help on the BitBake layer management tool, use the following
657command::
658
659 $ bitbake-layers --help
660
661The following list describes the available commands:
662
663- ``help:`` Displays general help or help on a specified command.
664
665- ``show-layers:`` Shows the current configured layers.
666
667- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
668 when a recipe with the same name exists in another layer that has a
669 higher layer priority.
670
671- ``show-recipes:`` Lists available recipes and the layers that
672 provide them.
673
674- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
675 which they apply.
676
677- ``show-cross-depends:`` Lists dependency relationships between
678 recipes that cross layer boundaries.
679
680- ``add-layer:`` Adds a layer to ``bblayers.conf``.
681
682- ``remove-layer:`` Removes a layer from ``bblayers.conf``
683
684- ``flatten:`` Flattens the layer configuration into a separate
685 output directory. Flattening your layer configuration builds a
686 "flattened" directory that contains the contents of all layers, with
687 any overlayed recipes removed and any ``.bbappend`` files appended to
688 the corresponding recipes. You might have to perform some manual
689 cleanup of the flattened layer as follows:
690
691 - Non-recipe files (such as patches) are overwritten. The flatten
692 command shows a warning for these files.
693
694 - Anything beyond the normal layer setup has been added to the
695 ``layer.conf`` file. Only the lowest priority layer's
696 ``layer.conf`` is used.
697
698 - Overridden and appended items from ``.bbappend`` files need to be
699 cleaned up. The contents of each ``.bbappend`` end up in the
700 flattened recipe. However, if there are appended or changed
701 variable values, you need to tidy these up yourself. Consider the
702 following example. Here, the ``bitbake-layers`` command adds the
703 line ``#### bbappended ...`` so that you know where the following
704 lines originate::
705
706 ...
707 DESCRIPTION = "A useful utility"
708 ...
709 EXTRA_OECONF = "--enable-something"
710 ...
711
712 #### bbappended from meta-anotherlayer ####
713
714 DESCRIPTION = "Customized utility"
715 EXTRA_OECONF += "--enable-somethingelse"
716
717
718 Ideally, you would tidy up these utilities as follows::
719
720 ...
721 DESCRIPTION = "Customized utility"
722 ...
723 EXTRA_OECONF = "--enable-something --enable-somethingelse"
724 ...
725
726- ``layerindex-fetch``: Fetches a layer from a layer index, along
727 with its dependent layers, and adds the layers to the
728 ``conf/bblayers.conf`` file.
729
730- ``layerindex-show-depends``: Finds layer dependencies from the
731 layer index.
732
733- ``save-build-conf``: Saves the currently active build configuration
734 (``conf/local.conf``, ``conf/bblayers.conf``) as a template into a layer.
735 This template can later be used for setting up builds via :term:`TEMPLATECONF`.
736 For information about saving and using configuration templates, see
737 ":ref:`dev-manual/custom-template-configuration-directory:creating a custom template configuration directory`".
738
739- ``create-layer``: Creates a basic layer.
740
741- ``create-layers-setup``: Writes out a configuration file and/or a script that
742 can replicate the directory structure and revisions of the layers in a current build.
743 For more information, see ":ref:`dev-manual/layers:saving and restoring the layers setup`".
744
745Creating a General Layer Using the ``bitbake-layers`` Script
746============================================================
747
748The ``bitbake-layers`` script with the ``create-layer`` subcommand
749simplifies creating a new general layer.
750
751.. note::
752
753 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
754 section in the Yocto
755 Project Board Specific (BSP) Developer's Guide.
756
757 - In order to use a layer with the OpenEmbedded build system, you
758 need to add the layer to your ``bblayers.conf`` configuration
759 file. See the ":ref:`dev-manual/layers:adding a layer using the \`\`bitbake-layers\`\` script`"
760 section for more information.
761
762The default mode of the script's operation with this subcommand is to
763create a layer with the following:
764
765- A layer priority of 6.
766
767- A ``conf`` subdirectory that contains a ``layer.conf`` file.
768
769- A ``recipes-example`` subdirectory that contains a further
770 subdirectory named ``example``, which contains an ``example.bb``
771 recipe file.
772
773- A ``COPYING.MIT``, which is the license statement for the layer. The
774 script assumes you want to use the MIT license, which is typical for
775 most layers, for the contents of the layer itself.
776
777- A ``README`` file, which is a file describing the contents of your
778 new layer.
779
780In its simplest form, you can use the following command form to create a
781layer. The command creates a layer whose name corresponds to
782"your_layer_name" in the current directory::
783
784 $ bitbake-layers create-layer your_layer_name
785
786As an example, the following command creates a layer named ``meta-scottrif``
787in your home directory::
788
789 $ cd /usr/home
790 $ bitbake-layers create-layer meta-scottrif
791 NOTE: Starting bitbake server...
792 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
793
794If you want to set the priority of the layer to other than the default
795value of "6", you can either use the ``--priority`` option or you
796can edit the
797:term:`BBFILE_PRIORITY` value
798in the ``conf/layer.conf`` after the script creates it. Furthermore, if
799you want to give the example recipe file some name other than the
800default, you can use the ``--example-recipe-name`` option.
801
802The easiest way to see how the ``bitbake-layers create-layer`` command
803works is to experiment with the script. You can also read the usage
804information by entering the following::
805
806 $ bitbake-layers create-layer --help
807 NOTE: Starting bitbake server...
808 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
809 [--example-recipe-name EXAMPLERECIPE]
810 layerdir
811
812 Create a basic layer
813
814 positional arguments:
815 layerdir Layer directory to create
816
817 optional arguments:
818 -h, --help show this help message and exit
819 --priority PRIORITY, -p PRIORITY
820 Layer directory to create
821 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
822 Filename of the example recipe
823
824Adding a Layer Using the ``bitbake-layers`` Script
825==================================================
826
827Once you create your general layer, you must add it to your
828``bblayers.conf`` file. Adding the layer to this configuration file
829makes the OpenEmbedded build system aware of your layer so that it can
830search it for metadata.
831
832Add your layer by using the ``bitbake-layers add-layer`` command::
833
834 $ bitbake-layers add-layer your_layer_name
835
836Here is an example that adds a
837layer named ``meta-scottrif`` to the configuration file. Following the
838command that adds the layer is another ``bitbake-layers`` command that
839shows the layers that are in your ``bblayers.conf`` file::
840
841 $ bitbake-layers add-layer meta-scottrif
842 NOTE: Starting bitbake server...
843 Parsing recipes: 100% |##########################################################| Time: 0:00:49
844 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
845 $ bitbake-layers show-layers
846 NOTE: Starting bitbake server...
847 layer path priority
848 ==========================================================================
849 meta /home/scottrif/poky/meta 5
850 meta-poky /home/scottrif/poky/meta-poky 5
851 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
852 workspace /home/scottrif/poky/build/workspace 99
853 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
854
855
856Adding the layer to this file
857enables the build system to locate the layer during the build.
858
859.. note::
860
861 During a build, the OpenEmbedded build system looks in the layers
862 from the top of the list down to the bottom in that order.
863
864Saving and restoring the layers setup
865=====================================
866
867Once you have a working build with the correct set of layers, it is beneficial
868to capture the layer setup --- what they are, which repositories they come from
869and which SCM revisions they're at --- into a configuration file, so that this
870setup can be easily replicated later, perhaps on a different machine. Here's
871how to do this::
872
873 $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/
874 NOTE: Starting bitbake server...
875 NOTE: Created /srv/work/alex/meta-alex/setup-layers.json
876 NOTE: Created /srv/work/alex/meta-alex/setup-layers
877
878The tool needs a single argument which tells where to place the output, consisting
879of a json formatted layer configuration, and a ``setup-layers`` script that can use that configuration
880to restore the layers in a different location, or on a different host machine. The argument
881can point to a custom layer (which is then deemed a "bootstrap" layer that needs to be
882checked out first), or into a completely independent location.
883
884The replication of the layers is performed by running the ``setup-layers`` script provided
885above:
886
887#. Clone the bootstrap layer or some other repository to obtain
888 the json config and the setup script that can use it.
889
890#. Run the script directly with no options::
891
892 alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers
893 Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override.
894
895 Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master
896 Running 'git init -q /srv/work/alex/my-build/meta-intel'
897 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel
898 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel
899 Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel
900
901 Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers
902 Running 'git init -q /srv/work/alex/my-build/poky'
903 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky
904 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky
905 Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky
906 Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky
907 Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky
908
909.. note::
910 This will work to update an existing checkout as well.
911
912.. note::
913 The script is self-sufficient and requires only python3
914 and git on the build machine.
915
916.. note::
917 Both the ``create-layers-setup`` and the ``setup-layers`` provided several additional options
918 that customize their behavior - you are welcome to study them via ``--help`` command line parameter.
919
diff --git a/documentation/dev-manual/libraries.rst b/documentation/dev-manual/libraries.rst
new file mode 100644
index 0000000000..521dbb9a7c
--- /dev/null
+++ b/documentation/dev-manual/libraries.rst
@@ -0,0 +1,267 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working With Libraries
4**********************
5
6Libraries are an integral part of your system. This section describes
7some common practices you might find helpful when working with libraries
8to build your system:
9
10- :ref:`How to include static library files
11 <dev-manual/libraries:including static library files>`
12
13- :ref:`How to use the Multilib feature to combine multiple versions of
14 library files into a single image
15 <dev-manual/libraries:combining multiple versions of library files into one image>`
16
17- :ref:`How to install multiple versions of the same library in parallel on
18 the same system
19 <dev-manual/libraries:installing multiple versions of the same library>`
20
21Including Static Library Files
22==============================
23
24If you are building a library and the library offers static linking, you
25can control which static library files (``*.a`` files) get included in
26the built library.
27
28The :term:`PACKAGES` and
29:term:`FILES:* <FILES>` variables in the
30``meta/conf/bitbake.conf`` configuration file define how files installed
31by the :ref:`ref-tasks-install` task are packaged. By default, the :term:`PACKAGES`
32variable includes ``${PN}-staticdev``, which represents all static
33library files.
34
35.. note::
36
37 Some previously released versions of the Yocto Project defined the
38 static library files through ``${PN}-dev``.
39
40Here is the part of the BitBake configuration file, where you can see
41how the static library files are defined::
42
43 PACKAGE_BEFORE_PN ?= ""
44 PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
45 PACKAGES_DYNAMIC = "^${PN}-locale-.*"
46 FILES = ""
47
48 FILES:${PN} = "${bindir}/* ${sbindir}/* ${libexecdir}/* ${libdir}/lib*${SOLIBS} \
49 ${sysconfdir} ${sharedstatedir} ${localstatedir} \
50 ${base_bindir}/* ${base_sbindir}/* \
51 ${base_libdir}/*${SOLIBS} \
52 ${base_prefix}/lib/udev ${prefix}/lib/udev \
53 ${base_libdir}/udev ${libdir}/udev \
54 ${datadir}/${BPN} ${libdir}/${BPN}/* \
55 ${datadir}/pixmaps ${datadir}/applications \
56 ${datadir}/idl ${datadir}/omf ${datadir}/sounds \
57 ${libdir}/bonobo/servers"
58
59 FILES:${PN}-bin = "${bindir}/* ${sbindir}/*"
60
61 FILES:${PN}-doc = "${docdir} ${mandir} ${infodir} ${datadir}/gtk-doc \
62 ${datadir}/gnome/help"
63 SECTION:${PN}-doc = "doc"
64
65 FILES_SOLIBSDEV ?= "${base_libdir}/lib*${SOLIBSDEV} ${libdir}/lib*${SOLIBSDEV}"
66 FILES:${PN}-dev = "${includedir} ${FILES_SOLIBSDEV} ${libdir}/*.la \
67 ${libdir}/*.o ${libdir}/pkgconfig ${datadir}/pkgconfig \
68 ${datadir}/aclocal ${base_libdir}/*.o \
69 ${libdir}/${BPN}/*.la ${base_libdir}/*.la \
70 ${libdir}/cmake ${datadir}/cmake"
71 SECTION:${PN}-dev = "devel"
72 ALLOW_EMPTY:${PN}-dev = "1"
73 RDEPENDS:${PN}-dev = "${PN} (= ${EXTENDPKGV})"
74
75 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
76 SECTION:${PN}-staticdev = "devel"
77 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
78
79Combining Multiple Versions of Library Files into One Image
80===========================================================
81
82The build system offers the ability to build libraries with different
83target optimizations or architecture formats and combine these together
84into one system image. You can link different binaries in the image
85against the different libraries as needed for specific use cases. This
86feature is called "Multilib".
87
88An example would be where you have most of a system compiled in 32-bit
89mode using 32-bit libraries, but you have something large, like a
90database engine, that needs to be a 64-bit application and uses 64-bit
91libraries. Multilib allows you to get the best of both 32-bit and 64-bit
92libraries.
93
94While the Multilib feature is most commonly used for 32 and 64-bit
95differences, the approach the build system uses facilitates different
96target optimizations. You could compile some binaries to use one set of
97libraries and other binaries to use a different set of libraries. The
98libraries could differ in architecture, compiler options, or other
99optimizations.
100
101There are several examples in the ``meta-skeleton`` layer found in the
102:term:`Source Directory`:
103
104- :oe_git:`conf/multilib-example.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example.conf>`
105 configuration file.
106
107- :oe_git:`conf/multilib-example2.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example2.conf>`
108 configuration file.
109
110- :oe_git:`recipes-multilib/images/core-image-multilib-example.bb </openembedded-core/tree/meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb>`
111 recipe
112
113Preparing to Use Multilib
114-------------------------
115
116User-specific requirements drive the Multilib feature. Consequently,
117there is no one "out-of-the-box" configuration that would
118meet your needs.
119
120In order to enable Multilib, you first need to ensure your recipe is
121extended to support multiple libraries. Many standard recipes are
122already extended and support multiple libraries. You can check in the
123``meta/conf/multilib.conf`` configuration file in the
124:term:`Source Directory` to see how this is
125done using the
126:term:`BBCLASSEXTEND` variable.
127Eventually, all recipes will be covered and this list will not be
128needed.
129
130For the most part, the :ref:`Multilib <ref-classes-multilib*>`
131class extension works automatically to
132extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
133:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
134Standard variables such as
135:term:`DEPENDS`,
136:term:`RDEPENDS`,
137:term:`RPROVIDES`,
138:term:`RRECOMMENDS`,
139:term:`PACKAGES`, and
140:term:`PACKAGES_DYNAMIC` are
141automatically extended by the system. If you are extending any manual
142code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
143those names are extended correctly.
144
145Using Multilib
146--------------
147
148After you have set up the recipes, you need to define the actual
149combination of multiple libraries you want to build. You accomplish this
150through your ``local.conf`` configuration file in the
151:term:`Build Directory`. An example configuration would be as follows::
152
153 MACHINE = "qemux86-64"
154 require conf/multilib.conf
155 MULTILIBS = "multilib:lib32"
156 DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
157 IMAGE_INSTALL:append = " lib32-glib-2.0"
158
159This example enables an additional library named
160``lib32`` alongside the normal target packages. When combining these
161"lib32" alternatives, the example uses "x86" for tuning. For information
162on this particular tuning, see
163``meta/conf/machine/include/ia32/arch-ia32.inc``.
164
165The example then includes ``lib32-glib-2.0`` in all the images, which
166illustrates one method of including a multiple library dependency. You
167can use a normal image build to include this dependency, for example::
168
169 $ bitbake core-image-sato
170
171You can also build Multilib packages
172specifically with a command like this::
173
174 $ bitbake lib32-glib-2.0
175
176Additional Implementation Details
177---------------------------------
178
179There are generic implementation details as well as details that are specific to
180package management systems. Here are implementation details
181that exist regardless of the package management system:
182
183- The typical convention used for the class extension code as used by
184 Multilib assumes that all package names specified in
185 :term:`PACKAGES` that contain
186 ``${PN}`` have ``${PN}`` at the start of the name. When that
187 convention is not followed and ``${PN}`` appears at the middle or the
188 end of a name, problems occur.
189
190- The :term:`TARGET_VENDOR`
191 value under Multilib will be extended to "-vendormlmultilib" (e.g.
192 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
193 slightly unwieldy contraction is that any "-" characters in the
194 vendor string presently break Autoconf's ``config.sub``, and other
195 separators are problematic for different reasons.
196
197Here are the implementation details for the RPM Package Management System:
198
199- A unique architecture is defined for the Multilib packages, along
200 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
201 :term:`Build Directory`. For example, consider ``lib32`` in a
202 ``qemux86-64`` image. The possible architectures in the system are "all",
203 "qemux86_64", "lib32:qemux86_64", and "lib32:x86".
204
205- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
206 packaging. The naming for a normal RPM package and a Multilib RPM
207 package in a ``qemux86-64`` system resolves to something similar to
208 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
209 respectively.
210
211- When installing a Multilib image, the RPM backend first installs the
212 base image and then installs the Multilib libraries.
213
214- The build system relies on RPM to resolve the identical files in the
215 two (or more) Multilib packages.
216
217Here are the implementation details for the IPK Package Management System:
218
219- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
220 packaging. The naming for a normal RPM package and a Multilib IPK
221 package in a ``qemux86-64`` system resolves to something like
222 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``,
223 respectively.
224
225- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
226 packages with and without the Multilib feature can exist in the same
227 folder due to the ``${PN}`` differences.
228
229- IPK defines a sanity check for Multilib installation using certain
230 rules for file comparison, overridden, etc.
231
232Installing Multiple Versions of the Same Library
233================================================
234
235There are be situations where you need to install and use multiple versions
236of the same library on the same system at the same time. This
237almost always happens when a library API changes and you have
238multiple pieces of software that depend on the separate versions of the
239library. To accommodate these situations, you can install multiple
240versions of the same library in parallel on the same system.
241
242The process is straightforward as long as the libraries use proper
243versioning. With properly versioned libraries, all you need to do to
244individually specify the libraries is create separate, appropriately
245named recipes where the :term:`PN` part of
246the name includes a portion that differentiates each library version
247(e.g. the major part of the version number). Thus, instead of having a
248single recipe that loads one version of a library (e.g. ``clutter``),
249you provide multiple recipes that result in different versions of the
250libraries you want. As an example, the following two recipes would allow
251the two separate versions of the ``clutter`` library to co-exist on the
252same system:
253
254.. code-block:: none
255
256 clutter-1.6_1.6.20.bb
257 clutter-1.8_1.8.4.bb
258
259Additionally, if
260you have other recipes that depend on a given library, you need to use
261the :term:`DEPENDS` variable to
262create the dependency. Continuing with the same example, if you want to
263have a recipe depend on the 1.8 version of the ``clutter`` library, use
264the following in your recipe::
265
266 DEPENDS = "clutter-1.8"
267
diff --git a/documentation/dev-manual/licenses.rst b/documentation/dev-manual/licenses.rst
new file mode 100644
index 0000000000..bffff3675f
--- /dev/null
+++ b/documentation/dev-manual/licenses.rst
@@ -0,0 +1,544 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working With Licenses
4*********************
5
6As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
7section in the Yocto Project Overview and Concepts Manual, open source
8projects are open to the public and they consequently have different
9licensing structures in place. This section describes the mechanism by
10which the :term:`OpenEmbedded Build System`
11tracks changes to
12licensing text and covers how to maintain open source license compliance
13during your project's lifecycle. The section also describes how to
14enable commercially licensed recipes, which by default are disabled.
15
16Tracking License Changes
17========================
18
19The license of an upstream project might change in the future. In order
20to prevent these changes going unnoticed, the
21:term:`LIC_FILES_CHKSUM`
22variable tracks changes to the license text. The checksums are validated
23at the end of the configure step, and if the checksums do not match, the
24build will fail.
25
26Specifying the ``LIC_FILES_CHKSUM`` Variable
27--------------------------------------------
28
29The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
30in the source code for the recipe. Here is an example of how to
31specify :term:`LIC_FILES_CHKSUM`::
32
33 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
34 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
35 file://licfile2.txt;endline=50;md5=zzzz \
36 ..."
37
38.. note::
39
40 - When using "beginline" and "endline", realize that line numbering
41 begins with one and not zero. Also, the included lines are
42 inclusive (i.e. lines five through and including 29 in the
43 previous example for ``licfile1.txt``).
44
45 - When a license check fails, the selected license text is included
46 as part of the QA message. Using this output, you can determine
47 the exact start and finish for the needed license text.
48
49The build system uses the :term:`S`
50variable as the default directory when searching files listed in
51:term:`LIC_FILES_CHKSUM`. The previous example employs the default
52directory.
53
54Consider this next example::
55
56 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
57 md5=bb14ed3c4cda583abc85401304b5cd4e"
58 LIC_FILES_CHKSUM = "file://${WORKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
59
60The first line locates a file in ``${S}/src/ls.c`` and isolates lines
61five through 16 as license text. The second line refers to a file in
62:term:`WORKDIR`.
63
64Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
65unless the :term:`LICENSE` variable is set to "CLOSED".
66
67Explanation of Syntax
68---------------------
69
70As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
71lists all the important files that contain the license text for the
72source code. It is possible to specify a checksum for an entire file, or
73a specific section of a file (specified by beginning and ending line
74numbers with the "beginline" and "endline" parameters, respectively).
75The latter is useful for source files with a license notice header,
76README documents, and so forth. If you do not use the "beginline"
77parameter, then it is assumed that the text begins on the first line of
78the file. Similarly, if you do not use the "endline" parameter, it is
79assumed that the license text ends with the last line of the file.
80
81The "md5" parameter stores the md5 checksum of the license text. If the
82license text changes in any way as compared to this parameter then a
83mismatch occurs. This mismatch triggers a build failure and notifies the
84developer. Notification allows the developer to review and address the
85license text changes. Also note that if a mismatch occurs during the
86build, the correct md5 checksum is placed in the build log and can be
87easily copied to the recipe.
88
89There is no limit to how many files you can specify using the
90:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
91requires a few specifications for license tracking. Many projects have a
92"COPYING" file that stores the license information for all the source
93code files. This practice allows you to just track the "COPYING" file as
94long as it is kept up to date.
95
96.. note::
97
98 - If you specify an empty or invalid "md5" parameter,
99 :term:`BitBake` returns an md5
100 mis-match error and displays the correct "md5" parameter value
101 during the build. The correct parameter is also captured in the
102 build log.
103
104 - If the whole file contains only license text, you do not need to
105 use the "beginline" and "endline" parameters.
106
107Enabling Commercially Licensed Recipes
108======================================
109
110By default, the OpenEmbedded build system disables components that have
111commercial or other special licensing requirements. Such requirements
112are defined on a recipe-by-recipe basis through the
113:term:`LICENSE_FLAGS` variable
114definition in the affected recipe. For instance, the
115``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
116contains the following statement::
117
118 LICENSE_FLAGS = "commercial"
119
120Here is a
121slightly more complicated example that contains both an explicit recipe
122name and version (after variable expansion)::
123
124 LICENSE_FLAGS = "license_${PN}_${PV}"
125
126It is possible to give more details about a specific license
127using flags on the :term:`LICENSE_FLAGS_DETAILS` variable::
128
129 LICENSE_FLAGS_DETAILS[my-eula-license] = "For further details, see https://example.com/eula."
130
131If set, this will be displayed to the user if the license hasn't been accepted.
132
133In order for a component restricted by a
134:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
135needs to have a matching entry in the global
136:term:`LICENSE_FLAGS_ACCEPTED`
137variable, which is a variable typically defined in your ``local.conf``
138file. For example, to enable the
139``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
140could add either the string "commercial_gst-plugins-ugly" or the more
141general string "commercial" to :term:`LICENSE_FLAGS_ACCEPTED`. See the
142":ref:`dev-manual/licenses:license flag matching`" section for a full
143explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
144example::
145
146 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly"
147
148Likewise, to additionally enable the package built from the recipe
149containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
150the actual recipe name was ``emgd_1.10.bb``, the following string would
151enable that package as well as the original ``gst-plugins-ugly``
152package::
153
154 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly license_emgd_1.10"
155
156As a convenience, you do not need to specify the
157complete license string for every package. You can use
158an abbreviated form, which consists of just the first portion or
159portions of the license string before the initial underscore character
160or characters. A partial string will match any license that contains the
161given string as the first portion of its license. For example, the
162following value will also match both of the packages
163previously mentioned as well as any other packages that have licenses
164starting with "commercial" or "license"::
165
166 LICENSE_FLAGS_ACCEPTED = "commercial license"
167
168License Flag Matching
169---------------------
170
171License flag matching allows you to control what recipes the
172OpenEmbedded build system includes in the build. Fundamentally, the
173build system attempts to match :term:`LICENSE_FLAGS` strings found in
174recipes against strings found in :term:`LICENSE_FLAGS_ACCEPTED`.
175A match causes the build system to include a recipe in the
176build, while failure to find a match causes the build system to exclude
177a recipe.
178
179In general, license flag matching is simple. However, understanding some
180concepts will help you correctly and effectively use matching.
181
182Before a flag defined by a particular recipe is tested against the
183entries of :term:`LICENSE_FLAGS_ACCEPTED`, the expanded
184string ``_${PN}`` is appended to the flag. This expansion makes each
185:term:`LICENSE_FLAGS` value recipe-specific. After expansion, the
186string is then matched against the entries. Thus, specifying
187``LICENSE_FLAGS = "commercial"`` in recipe "foo", for example, results
188in the string ``"commercial_foo"``. And, to create a match, that string
189must appear among the entries of :term:`LICENSE_FLAGS_ACCEPTED`.
190
191Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
192:term:`LICENSE_FLAGS_ACCEPTED` variable allows you a lot of flexibility for
193including or excluding recipes based on licensing. For example, you can
194broaden the matching capabilities by using license flags string subsets
195in :term:`LICENSE_FLAGS_ACCEPTED`.
196
197.. note::
198
199 When using a string subset, be sure to use the part of the expanded
200 string that precedes the appended underscore character (e.g.
201 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
202
203For example, simply specifying the string "commercial" in the
204:term:`LICENSE_FLAGS_ACCEPTED` variable matches any expanded
205:term:`LICENSE_FLAGS` definition that starts with the string
206"commercial" such as "commercial_foo" and "commercial_bar", which
207are the strings the build system automatically generates for
208hypothetical recipes named "foo" and "bar" assuming those recipes simply
209specify the following::
210
211 LICENSE_FLAGS = "commercial"
212
213Thus, you can choose to exhaustively enumerate each license flag in the
214list and allow only specific recipes into the image, or you can use a
215string subset that causes a broader range of matches to allow a range of
216recipes into the image.
217
218This scheme works even if the :term:`LICENSE_FLAGS` string already has
219``_${PN}`` appended. For example, the build system turns the license
220flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
221both the general "commercial" and the specific "commercial_1.2_foo"
222strings found in the :term:`LICENSE_FLAGS_ACCEPTED` variable, as expected.
223
224Here are some other scenarios:
225
226- You can specify a versioned string in the recipe such as
227 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
228 string to "commercial_foo_1.2_foo". Combine this license flag with a
229 :term:`LICENSE_FLAGS_ACCEPTED` variable that has the string
230 "commercial" and you match the flag along with any other flag that
231 starts with the string "commercial".
232
233- Under the same circumstances, you can add "commercial_foo" in the
234 :term:`LICENSE_FLAGS_ACCEPTED` variable and the build system not only
235 matches "commercial_foo_1.2" but also matches any license flag with
236 the string "commercial_foo", regardless of the version.
237
238- You can be very specific and use both the package and version parts
239 in the :term:`LICENSE_FLAGS_ACCEPTED` list (e.g.
240 "commercial_foo_1.2") to specifically match a versioned recipe.
241
242Other Variables Related to Commercial Licenses
243----------------------------------------------
244
245There are other helpful variables related to commercial license handling,
246defined in the
247``poky/meta/conf/distro/include/default-distrovars.inc`` file::
248
249 COMMERCIAL_AUDIO_PLUGINS ?= ""
250 COMMERCIAL_VIDEO_PLUGINS ?= ""
251
252If you want to enable these components, you can do so by making sure you have
253statements similar to the following in your ``local.conf`` configuration file::
254
255 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
256 gst-plugins-ugly-mpegaudioparse"
257 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
258 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
259 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
260
261Of course, you could also create a matching list for those components using the
262more general "commercial" string in the :term:`LICENSE_FLAGS_ACCEPTED` variable,
263but that would also enable all the other packages with :term:`LICENSE_FLAGS`
264containing "commercial", which you may or may not want::
265
266 LICENSE_FLAGS_ACCEPTED = "commercial"
267
268Specifying audio and video plugins as part of the
269:term:`COMMERCIAL_AUDIO_PLUGINS` and :term:`COMMERCIAL_VIDEO_PLUGINS` statements
270(along with :term:`LICENSE_FLAGS_ACCEPTED`) includes the plugins or
271components into built images, thus adding support for media formats or
272components.
273
274.. note::
275
276 GStreamer "ugly" and "bad" plugins are actually available through
277 open source licenses. However, the "ugly" ones can be subject to software
278 patents in some countries, making it necessary to pay licensing fees
279 to distribute them. The "bad" ones are just deemed unreliable by the
280 GStreamer community and should therefore be used with care.
281
282Maintaining Open Source License Compliance During Your Product's Lifecycle
283==========================================================================
284
285One of the concerns for a development organization using open source
286software is how to maintain compliance with various open source
287licensing during the lifecycle of the product. While this section does
288not provide legal advice or comprehensively cover all scenarios, it does
289present methods that you can use to assist you in meeting the compliance
290requirements during a software release.
291
292With hundreds of different open source licenses that the Yocto Project
293tracks, it is difficult to know the requirements of each and every
294license. However, the requirements of the major FLOSS licenses can begin
295to be covered by assuming that there are three main areas of concern:
296
297- Source code must be provided.
298
299- License text for the software must be provided.
300
301- Compilation scripts and modifications to the source code must be
302 provided.
303
304There are other requirements beyond the scope of these three and the
305methods described in this section (e.g. the mechanism through which
306source code is distributed).
307
308As different organizations have different ways of releasing software,
309there can be multiple ways of meeting license obligations. At
310least, we describe here two methods for achieving compliance:
311
312- The first method is to use OpenEmbedded's ability to provide
313 the source code, provide a list of licenses, as well as
314 compilation scripts and source code modifications.
315
316 The remainder of this section describes supported methods to meet
317 the previously mentioned three requirements.
318
319- The second method is to generate a *Software Bill of Materials*
320 (:term:`SBoM`), as described in the ":doc:`/dev-manual/sbom`" section.
321 Not only do you generate :term:`SPDX` output which can be used meet
322 license compliance requirements (except for sharing the build system
323 and layers sources for the time being), but this output also includes
324 component version and patch information which can be used
325 for vulnerability assessment.
326
327Whatever method you choose, prior to releasing images, sources,
328and the build system, you should audit all artifacts to ensure
329completeness.
330
331.. note::
332
333 The Yocto Project generates a license manifest during image creation
334 that is located in
335 ``${DEPLOY_DIR}/licenses/${SSTATE_PKGARCH}/<image-name>-<machine>.rootfs-<datestamp>/``
336 to assist with any audits.
337
338Providing the Source Code
339-------------------------
340
341Compliance activities should begin before you generate the final image.
342The first thing you should look at is the requirement that tops the list
343for most compliance groups --- providing the source. The Yocto Project has
344a few ways of meeting this requirement.
345
346One of the easiest ways to meet this requirement is to provide the
347entire :term:`DL_DIR` used by the
348build. This method, however, has a few issues. The most obvious is the
349size of the directory since it includes all sources used in the build
350and not just the source used in the released image. It will include
351toolchain source, and other artifacts, which you would not generally
352release. However, the more serious issue for most companies is
353accidental release of proprietary software. The Yocto Project provides
354an :ref:`ref-classes-archiver` class to help avoid some of these concerns.
355
356Before you employ :term:`DL_DIR` or the :ref:`ref-classes-archiver` class, you
357need to decide how you choose to provide source. The source
358:ref:`ref-classes-archiver` class can generate tarballs and SRPMs and can
359create them with various levels of compliance in mind.
360
361One way of doing this (but certainly not the only way) is to release
362just the source as a tarball. You can do this by adding the following to
363the ``local.conf`` file found in the :term:`Build Directory`::
364
365 INHERIT += "archiver"
366 ARCHIVER_MODE[src] = "original"
367
368During the creation of your
369image, the source from all recipes that deploy packages to the image is
370placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
371:term:`LICENSE` for each recipe.
372Releasing the entire directory enables you to comply with requirements
373concerning providing the unmodified source. It is important to note that
374the size of the directory can get large.
375
376A way to help mitigate the size issue is to only release tarballs for
377licenses that require the release of source. Let us assume you are only
378concerned with GPL code as identified by running the following script:
379
380.. code-block:: shell
381
382 # Script to archive a subset of packages matching specific license(s)
383 # Source and license files are copied into sub folders of package folder
384 # Must be run from build folder
385 #!/bin/bash
386 src_release_dir="source-release"
387 mkdir -p $src_release_dir
388 for a in tmp/deploy/sources/*; do
389 for d in $a/*; do
390 # Get package name from path
391 p=`basename $d`
392 p=${p%-*}
393 p=${p%-*}
394 # Only archive GPL packages (update *GPL* regex for your license check)
395 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
396 if [ $numfiles -ge 1 ]; then
397 echo Archiving $p
398 mkdir -p $src_release_dir/$p/source
399 cp $d/* $src_release_dir/$p/source 2> /dev/null
400 mkdir -p $src_release_dir/$p/license
401 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
402 fi
403 done
404 done
405
406At this point, you
407could create a tarball from the ``gpl_source_release`` directory and
408provide that to the end user. This method would be a step toward
409achieving compliance with section 3a of GPLv2 and with section 6 of
410GPLv3.
411
412Providing License Text
413----------------------
414
415One requirement that is often overlooked is inclusion of license text.
416This requirement also needs to be dealt with prior to generating the
417final image. Some licenses require the license text to accompany the
418binary. You can achieve this by adding the following to your
419``local.conf`` file::
420
421 COPY_LIC_MANIFEST = "1"
422 COPY_LIC_DIRS = "1"
423 LICENSE_CREATE_PACKAGE = "1"
424
425Adding these statements to the
426configuration file ensures that the licenses collected during package
427generation are included on your image.
428
429.. note::
430
431 Setting all three variables to "1" results in the image having two
432 copies of the same license file. One copy resides in
433 ``/usr/share/common-licenses`` and the other resides in
434 ``/usr/share/license``.
435
436 The reason for this behavior is because
437 :term:`COPY_LIC_DIRS` and
438 :term:`COPY_LIC_MANIFEST`
439 add a copy of the license when the image is built but do not offer a
440 path for adding licenses for newly installed packages to an image.
441 :term:`LICENSE_CREATE_PACKAGE`
442 adds a separate package and an upgrade path for adding licenses to an
443 image.
444
445As the source :ref:`ref-classes-archiver` class has already archived the
446original unmodified source that contains the license files, you would have
447already met the requirements for inclusion of the license information
448with source as defined by the GPL and other open source licenses.
449
450Providing Compilation Scripts and Source Code Modifications
451-----------------------------------------------------------
452
453At this point, we have addressed all we need prior to generating the
454image. The next two requirements are addressed during the final
455packaging of the release.
456
457By releasing the version of the OpenEmbedded build system and the layers
458used during the build, you will be providing both compilation scripts
459and the source code modifications in one step.
460
461If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
462and a distro layer, and those
463those layers are used to patch, compile, package, or modify (in any way)
464any open source software included in your released images, you might be
465required to release those layers under section 3 of GPLv2 or section 1
466of GPLv3. One way of doing that is with a clean checkout of the version
467of the Yocto Project and layers used during your build. Here is an
468example:
469
470.. code-block:: shell
471
472 # We built using the dunfell branch of the poky repo
473 $ git clone -b dunfell git://git.yoctoproject.org/poky
474 $ cd poky
475 # We built using the release_branch for our layers
476 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
477 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
478 # clean up the .git repos
479 $ find . -name ".git" -type d -exec rm -rf {} \;
480
481One thing a development organization might want to consider for end-user
482convenience is to modify
483``meta-poky/conf/templates/default/bblayers.conf.sample`` to ensure that when
484the end user utilizes the released build system to build an image, the
485development organization's layers are included in the ``bblayers.conf`` file
486automatically::
487
488 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
489 # changes incompatibly
490 POKY_BBLAYERS_CONF_VERSION = "2"
491
492 BBPATH = "${TOPDIR}"
493 BBFILES ?= ""
494
495 BBLAYERS ?= " \
496 ##OEROOT##/meta \
497 ##OEROOT##/meta-poky \
498 ##OEROOT##/meta-yocto-bsp \
499 ##OEROOT##/meta-mylayer \
500 "
501
502Creating and
503providing an archive of the :term:`Metadata`
504layers (recipes, configuration files, and so forth) enables you to meet
505your requirements to include the scripts to control compilation as well
506as any modifications to the original source.
507
508Compliance Limitations with Executables Built from Static Libraries
509-------------------------------------------------------------------
510
511When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS`
512mechanisms as well as explicitly included in the image recipe with
513:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B
514(``DEPENDS += "B"``), package B will neither appear in the generated license
515manifest nor in the generated source tarballs. This occurs as the
516:ref:`ref-classes-license` and :ref:`ref-classes-archiver` classes assume that
517only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS`
518end up in the image.
519
520As a result, potential obligations regarding license compliance for package B
521may not be met.
522
523The Yocto Project doesn't enable static libraries by default, in part because
524of this issue. Before a solution to this limitation is found, you need to
525keep in mind that if your root filesystem is built from static libraries,
526you will need to manually ensure that your deliveries are compliant
527with the licenses of these libraries.
528
529Copying Non Standard Licenses
530=============================
531
532Some packages, such as the linux-firmware package, have many licenses
533that are not in any way common. You can avoid adding a lot of these
534types of common license files, which are only applicable to a specific
535package, by using the
536:term:`NO_GENERIC_LICENSE`
537variable. Using this variable also avoids QA errors when you use a
538non-common, non-CLOSED license in a recipe.
539
540Here is an example that uses the ``LICENSE.Abilis.txt`` file as
541the license from the fetched source::
542
543 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
544
diff --git a/documentation/dev-manual/new-machine.rst b/documentation/dev-manual/new-machine.rst
new file mode 100644
index 0000000000..469b2d395a
--- /dev/null
+++ b/documentation/dev-manual/new-machine.rst
@@ -0,0 +1,118 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Adding a New Machine
4********************
5
6Adding a new machine to the Yocto Project is a straightforward process.
7This section describes how to add machines that are similar to those
8that the Yocto Project already supports.
9
10.. note::
11
12 Although well within the capabilities of the Yocto Project, adding a
13 totally new architecture might require changes to ``gcc``/``glibc``
14 and to the site information, which is beyond the scope of this
15 manual.
16
17For a complete example that shows how to add a new machine, see the
18":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
19section in the Yocto Project Board Support Package (BSP) Developer's
20Guide.
21
22Adding the Machine Configuration File
23=====================================
24
25To add a new machine, you need to add a new machine configuration file
26to the layer's ``conf/machine`` directory. This configuration file
27provides details about the device you are adding.
28
29The OpenEmbedded build system uses the root name of the machine
30configuration file to reference the new machine. For example, given a
31machine configuration file named ``crownbay.conf``, the build system
32recognizes the machine as "crownbay".
33
34The most important variables you must set in your machine configuration
35file or include from a lower-level configuration file are as follows:
36
37- :term:`TARGET_ARCH` (e.g. "arm")
38
39- ``PREFERRED_PROVIDER_virtual/kernel``
40
41- :term:`MACHINE_FEATURES` (e.g. "screen wifi")
42
43You might also need these variables:
44
45- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
46
47- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
48
49- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
50
51You can find full details on these variables in the reference section.
52You can leverage existing machine ``.conf`` files from
53``meta-yocto-bsp/conf/machine/``.
54
55Adding a Kernel for the Machine
56===============================
57
58The OpenEmbedded build system needs to be able to build a kernel for the
59machine. You need to either create a new kernel recipe for this machine,
60or extend an existing kernel recipe. You can find several kernel recipe
61examples in the Source Directory at ``meta/recipes-kernel/linux`` that
62you can use as references.
63
64If you are creating a new kernel recipe, normal recipe-writing rules
65apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
66necessary patches and set :term:`S` to point at the source code. You need to
67create a :ref:`ref-tasks-configure` task that configures the unpacked kernel with
68a ``defconfig`` file. You can do this by using a ``make defconfig``
69command or, more commonly, by copying in a suitable ``defconfig`` file
70and then running ``make oldconfig``. By making use of ``inherit kernel``
71and potentially some of the ``linux-*.inc`` files, most other
72functionality is centralized and the defaults of the class normally work
73well.
74
75If you are extending an existing kernel recipe, it is usually a matter
76of adding a suitable ``defconfig`` file. The file needs to be added into
77a location similar to ``defconfig`` files used for other machines in a
78given kernel recipe. A possible way to do this is by listing the file in
79the :term:`SRC_URI` and adding the machine to the expression in
80:term:`COMPATIBLE_MACHINE`::
81
82 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
83
84For more information on ``defconfig`` files, see the
85":ref:`kernel-dev/common:changing the configuration`"
86section in the Yocto Project Linux Kernel Development Manual.
87
88Adding a Formfactor Configuration File
89======================================
90
91A formfactor configuration file provides information about the target
92hardware for which the image is being built and information that the
93build system cannot obtain from other sources such as the kernel. Some
94examples of information contained in a formfactor configuration file
95include framebuffer orientation, whether or not the system has a
96keyboard, the positioning of the keyboard in relation to the screen, and
97the screen resolution.
98
99The build system uses reasonable defaults in most cases. However, if
100customization is necessary, you need to create a ``machconfig`` file in
101the ``meta/recipes-bsp/formfactor/files`` directory. This directory
102contains directories for specific machines such as ``qemuarm`` and
103``qemux86``. For information about the settings available and the
104defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
105found in the same area.
106
107Here is an example for "qemuarm" machine::
108
109 HAVE_TOUCHSCREEN=1
110 HAVE_KEYBOARD=1
111 DISPLAY_CAN_ROTATE=0
112 DISPLAY_ORIENTATION=0
113 #DISPLAY_WIDTH_PIXELS=640
114 #DISPLAY_HEIGHT_PIXELS=480
115 #DISPLAY_BPP=16
116 DISPLAY_DPI=150
117 DISPLAY_SUBPIXEL_ORDER=vrgb
118
diff --git a/documentation/dev-manual/new-recipe.rst b/documentation/dev-manual/new-recipe.rst
new file mode 100644
index 0000000000..61fc2eb122
--- /dev/null
+++ b/documentation/dev-manual/new-recipe.rst
@@ -0,0 +1,1639 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Writing a New Recipe
4********************
5
6Recipes (``.bb`` files) are fundamental components in the Yocto Project
7environment. Each software component built by the OpenEmbedded build
8system requires a recipe to define the component. This section describes
9how to create, write, and test a new recipe.
10
11.. note::
12
13 For information on variables that are useful for recipes and for
14 information about recipe naming issues, see the
15 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
16 Reference Manual.
17
18Overview
19========
20
21The following figure shows the basic process for creating a new recipe.
22The remainder of the section provides details for the steps.
23
24.. image:: figures/recipe-workflow.png
25 :align: center
26 :width: 50%
27
28Locate or Automatically Create a Base Recipe
29============================================
30
31You can always write a recipe from scratch. However, there are three choices
32that can help you quickly get started with a new recipe:
33
34- ``devtool add``: A command that assists in creating a recipe and an
35 environment conducive to development.
36
37- ``recipetool create``: A command provided by the Yocto Project that
38 automates creation of a base recipe based on the source files.
39
40- *Existing Recipes:* Location and modification of an existing recipe
41 that is similar in function to the recipe you need.
42
43.. note::
44
45 For information on recipe syntax, see the
46 ":ref:`dev-manual/new-recipe:recipe syntax`" section.
47
48Creating the Base Recipe Using ``devtool add``
49----------------------------------------------
50
51The ``devtool add`` command uses the same logic for auto-creating the
52recipe as ``recipetool create``, which is listed below. Additionally,
53however, ``devtool add`` sets up an environment that makes it easy for
54you to patch the source and to make changes to the recipe as is often
55necessary when adding a recipe to build a new piece of software to be
56included in a build.
57
58You can find a complete description of the ``devtool add`` command in
59the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section
60in the Yocto Project Application Development and the Extensible Software
61Development Kit (eSDK) manual.
62
63Creating the Base Recipe Using ``recipetool create``
64----------------------------------------------------
65
66``recipetool create`` automates creation of a base recipe given a set of
67source code files. As long as you can extract or point to the source
68files, the tool will construct a recipe and automatically configure all
69pre-build information into the recipe. For example, suppose you have an
70application that builds using Autotools. Creating the base recipe using
71``recipetool`` results in a recipe that has the pre-build dependencies,
72license requirements, and checksums configured.
73
74To run the tool, you just need to be in your :term:`Build Directory` and
75have sourced the build environment setup script (i.e.
76:ref:`structure-core-script`). To get help on the tool, use the following
77command::
78
79 $ recipetool -h
80 NOTE: Starting bitbake server...
81 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
82
83 OpenEmbedded recipe tool
84
85 options:
86 -d, --debug Enable debug output
87 -q, --quiet Print only errors
88 --color COLOR Colorize output (where COLOR is auto, always, never)
89 -h, --help show this help message and exit
90
91 subcommands:
92 create Create a new recipe
93 newappend Create a bbappend for the specified target in the specified
94 layer
95 setvar Set a variable within a recipe
96 appendfile Create/update a bbappend to replace a target file
97 appendsrcfiles Create/update a bbappend to add or replace source files
98 appendsrcfile Create/update a bbappend to add or replace a source file
99 Use recipetool <subcommand> --help to get help on a specific command
100
101Running ``recipetool create -o OUTFILE`` creates the base recipe and
102locates it properly in the layer that contains your source files.
103Here are some syntax examples:
104
105 - Use this syntax to generate a recipe based on source. Once generated,
106 the recipe resides in the existing source code layer::
107
108 recipetool create -o OUTFILE source
109
110 - Use this syntax to generate a recipe using code that
111 you extract from source. The extracted code is placed in its own layer
112 defined by :term:`EXTERNALSRC`::
113
114 recipetool create -o OUTFILE -x EXTERNALSRC source
115
116 - Use this syntax to generate a recipe based on source. The options
117 direct ``recipetool`` to generate debugging information. Once generated,
118 the recipe resides in the existing source code layer::
119
120 recipetool create -d -o OUTFILE source
121
122Locating and Using a Similar Recipe
123-----------------------------------
124
125Before writing a recipe from scratch, it is often useful to discover
126whether someone else has already written one that meets (or comes close
127to meeting) your needs. The Yocto Project and OpenEmbedded communities
128maintain many recipes that might be candidates for what you are doing.
129You can find a good central index of these recipes in the
130:oe_layerindex:`OpenEmbedded Layer Index <>`.
131
132Working from an existing recipe or a skeleton recipe is the best way to
133get started. Here are some points on both methods:
134
135- *Locate and modify a recipe that is close to what you want to do:*
136 This method works when you are familiar with the current recipe
137 space. The method does not work so well for those new to the Yocto
138 Project or writing recipes.
139
140 Some risks associated with this method are using a recipe that has
141 areas totally unrelated to what you are trying to accomplish with
142 your recipe, not recognizing areas of the recipe that you might have
143 to add from scratch, and so forth. All these risks stem from
144 unfamiliarity with the existing recipe space.
145
146- *Use and modify the following skeleton recipe:* If for some reason
147 you do not want to use ``recipetool`` and you cannot find an existing
148 recipe that is close to meeting your needs, you can use the following
149 structure to provide the fundamental areas of a new recipe::
150
151 DESCRIPTION = ""
152 HOMEPAGE = ""
153 LICENSE = ""
154 SECTION = ""
155 DEPENDS = ""
156 LIC_FILES_CHKSUM = ""
157
158 SRC_URI = ""
159
160Storing and Naming the Recipe
161=============================
162
163Once you have your base recipe, you should put it in your own layer and
164name it appropriately. Locating it correctly ensures that the
165OpenEmbedded build system can find it when you use BitBake to process
166the recipe.
167
168- *Storing Your Recipe:* The OpenEmbedded build system locates your
169 recipe through the layer's ``conf/layer.conf`` file and the
170 :term:`BBFILES` variable. This
171 variable sets up a path from which the build system can locate
172 recipes. Here is the typical use::
173
174 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
175 ${LAYERDIR}/recipes-*/*/*.bbappend"
176
177 Consequently, you need to be sure you locate your new recipe inside
178 your layer such that it can be found.
179
180 You can find more information on how layers are structured in the
181 ":ref:`dev-manual/layers:understanding and creating layers`" section.
182
183- *Naming Your Recipe:* When you name your recipe, you need to follow
184 this naming convention::
185
186 basename_version.bb
187
188 Use lower-cased characters and do not include the reserved suffixes
189 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
190 them as part of your recipe name unless the string applies). Here are some
191 examples:
192
193 .. code-block:: none
194
195 cups_1.7.0.bb
196 gawk_4.0.2.bb
197 irssi_0.8.16-rc1.bb
198
199Running a Build on the Recipe
200=============================
201
202Creating a new recipe is usually an iterative process that requires
203using BitBake to process the recipe multiple times in order to
204progressively discover and add information to the recipe file.
205
206Assuming you have sourced the build environment setup script (i.e.
207:ref:`structure-core-script`) and you are in the :term:`Build Directory`, use
208BitBake to process your recipe. All you need to provide is the
209``basename`` of the recipe as described in the previous section::
210
211 $ bitbake basename
212
213During the build, the OpenEmbedded build system creates a temporary work
214directory for each recipe
215(``${``\ :term:`WORKDIR`\ ``}``)
216where it keeps extracted source files, log files, intermediate
217compilation and packaging files, and so forth.
218
219The path to the per-recipe temporary work directory depends on the
220context in which it is being built. The quickest way to find this path
221is to have BitBake return it by running the following::
222
223 $ bitbake -e basename | grep ^WORKDIR=
224
225As an example, assume a Source Directory
226top-level folder named ``poky``, a default :term:`Build Directory` at
227``poky/build``, and a ``qemux86-poky-linux`` machine target system.
228Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
229case, the work directory the build system uses to build the package
230would be as follows::
231
232 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
233
234Inside this directory you can find sub-directories such as ``image``,
235``packages-split``, and ``temp``. After the build, you can examine these
236to determine how well the build went.
237
238.. note::
239
240 You can find log files for each task in the recipe's ``temp``
241 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
242 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
243 ``log.do_fetch``, and ``log.do_compile``).
244
245You can find more information about the build process in
246":doc:`/overview-manual/development-environment`"
247chapter of the Yocto Project Overview and Concepts Manual.
248
249Fetching Code
250=============
251
252The first thing your recipe must do is specify how to fetch the source
253files. Fetching is controlled mainly through the
254:term:`SRC_URI` variable. Your recipe
255must have a :term:`SRC_URI` variable that points to where the source is
256located. For a graphical representation of source locations, see the
257":ref:`overview-manual/concepts:sources`" section in
258the Yocto Project Overview and Concepts Manual.
259
260The :ref:`ref-tasks-fetch` task uses the prefix of each entry in the
261:term:`SRC_URI` variable value to determine which
262:ref:`fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`
263to use to get your source files. It is the :term:`SRC_URI` variable that triggers
264the fetcher. The :ref:`ref-tasks-patch` task uses the variable after source is
265fetched to apply patches. The OpenEmbedded build system uses
266:term:`FILESOVERRIDES` for scanning directory locations for local files in
267:term:`SRC_URI`.
268
269The :term:`SRC_URI` variable in your recipe must define each unique location
270for your source files. It is good practice to not hard-code version
271numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
272values, use ``${``\ :term:`PV`\ ``}``,
273which causes the fetch process to use the version specified in the
274recipe filename. Specifying the version in this manner means that
275upgrading the recipe to a future version is as simple as renaming the
276recipe to match the new version.
277
278Here is a simple example from the
279``meta/recipes-devtools/strace/strace_5.5.bb`` recipe where the source
280comes from a single tarball. Notice the use of the
281:term:`PV` variable::
282
283 SRC_URI = "https://strace.io/files/${PV}/strace-${PV}.tar.xz \
284
285Files mentioned in :term:`SRC_URI` whose names end in a typical archive
286extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
287forth), are automatically extracted during the
288:ref:`ref-tasks-unpack` task. For
289another example that specifies these types of files, see the
290":ref:`dev-manual/new-recipe:building an autotooled package`" section.
291
292Another way of specifying source is from an SCM. For Git repositories,
293you must specify :term:`SRCREV` and you should specify :term:`PV` to include
294the revision with :term:`SRCPV`. Here is an example from the recipe
295``meta/recipes-core/musl/gcompat_git.bb``::
296
297 SRC_URI = "git://git.adelielinux.org/adelie/gcompat.git;protocol=https;branch=current"
298
299 PV = "1.0.0+1.1+git${SRCPV}"
300 SRCREV = "af5a49e489fdc04b9cf02547650d7aeaccd43793"
301
302If your :term:`SRC_URI` statement includes URLs pointing to individual files
303fetched from a remote server other than a version control system,
304BitBake attempts to verify the files against checksums defined in your
305recipe to ensure they have not been tampered with or otherwise modified
306since the recipe was written. Multiple checksums are supported:
307``SRC_URI[md5sum]``, ``SRC_URI[sha1sum]``, ``SRC_URI[sha256sum]``.
308``SRC_URI[sha384sum]`` and ``SRC_URI[sha512sum]``, but only
309``SRC_URI[sha256sum]`` is commonly used.
310
311.. note::
312
313 ``SRC_URI[md5sum]`` used to also be commonly used, but it is deprecated
314 and should be replaced by ``SRC_URI[sha256sum]`` when updating existing
315 recipes.
316
317If your :term:`SRC_URI` variable points to more than a single URL (excluding
318SCM URLs), you need to provide the ``sha256`` checksum for each URL. For these
319cases, you provide a name for each URL as part of the :term:`SRC_URI` and then
320reference that name in the subsequent checksum statements. Here is an example
321combining lines from the files ``git.inc`` and ``git_2.24.1.bb``::
322
323 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
324 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
325
326 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
327 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
328
329The proper value for the ``sha256`` checksum might be available together
330with other signatures on the download page for the upstream source (e.g.
331``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
332OpenEmbedded build system typically only deals with ``sha256sum``,
333you should verify all the signatures you find by hand.
334
335If no :term:`SRC_URI` checksums are specified when you attempt to build the
336recipe, or you provide an incorrect checksum, the build will produce an
337error for each missing or incorrect checksum. As part of the error
338message, the build system provides the checksum string corresponding to
339the fetched file. Once you have the correct checksums, you can copy and
340paste them into your recipe and then run the build again to continue.
341
342.. note::
343
344 As mentioned, if the upstream source provides signatures for
345 verifying the downloaded source code, you should verify those
346 manually before setting the checksum values in the recipe and
347 continuing with the build.
348
349This final example is a bit more complicated and is from the
350``meta/recipes-sato/rxvt-unicode/rxvt-unicode_9.20.bb`` recipe. The
351example's :term:`SRC_URI` statement identifies multiple files as the source
352files for the recipe: a tarball, a patch file, a desktop file, and an icon::
353
354 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
355 file://xwc.patch \
356 file://rxvt.desktop \
357 file://rxvt.png"
358
359When you specify local files using the ``file://`` URI protocol, the
360build system fetches files from the local machine. The path is relative
361to the :term:`FILESPATH` variable
362and searches specific directories in a certain order:
363``${``\ :term:`BP`\ ``}``,
364``${``\ :term:`BPN`\ ``}``, and
365``files``. The directories are assumed to be subdirectories of the
366directory in which the recipe or append file resides. For another
367example that specifies these types of files, see the
368"`building a single .c file package`_" section.
369
370The previous example also specifies a patch file. Patch files are files
371whose names usually end in ``.patch`` or ``.diff`` but can end with
372compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
373The build system automatically applies patches as described in the
374":ref:`dev-manual/new-recipe:patching code`" section.
375
376Fetching Code Through Firewalls
377-------------------------------
378
379Some users are behind firewalls and need to fetch code through a proxy.
380See the ":doc:`/ref-manual/faq`" chapter for advice.
381
382Limiting the Number of Parallel Connections
383-------------------------------------------
384
385Some users are behind firewalls or use servers where the number of parallel
386connections is limited. In such cases, you can limit the number of fetch
387tasks being run in parallel by adding the following to your ``local.conf``
388file::
389
390 do_fetch[number_threads] = "4"
391
392Unpacking Code
393==============
394
395During the build, the
396:ref:`ref-tasks-unpack` task unpacks
397the source with ``${``\ :term:`S`\ ``}``
398pointing to where it is unpacked.
399
400If you are fetching your source files from an upstream source archived
401tarball and the tarball's internal structure matches the common
402convention of a top-level subdirectory named
403``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
404then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
405fetch source from an archive that does not use this convention, or from
406an SCM like Git or Subversion, your recipe needs to define :term:`S`.
407
408If processing your recipe using BitBake successfully unpacks the source
409files, you need to be sure that the directory pointed to by ``${S}``
410matches the structure of the source.
411
412Patching Code
413=============
414
415Sometimes it is necessary to patch code after it has been fetched. Any
416files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
417``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz``,
418``patch.bz2``, etc.) are treated as patches. The
419:ref:`ref-tasks-patch` task
420automatically applies these patches.
421
422The build system should be able to apply patches with the "-p1" option
423(i.e. one directory level in the path will be stripped off). If your
424patch needs to have more directory levels stripped off, specify the
425number of levels using the "striplevel" option in the :term:`SRC_URI` entry
426for the patch. Alternatively, if your patch needs to be applied in a
427specific subdirectory that is not specified in the patch file, use the
428"patchdir" option in the entry.
429
430As with all local files referenced in
431:term:`SRC_URI` using ``file://``,
432you should place patch files in a directory next to the recipe either
433named the same as the base name of the recipe
434(:term:`BP` and
435:term:`BPN`) or "files".
436
437Licensing
438=========
439
440Your recipe needs to define variables related to the license
441under whith the software is distributed. See the
442:ref:`contributor-guide/recipe-style-guide:recipe license fields`
443section in the Contributor Guide for details.
444
445Dependencies
446============
447
448Most software packages have a short list of other packages that they
449require, which are called dependencies. These dependencies fall into two
450main categories: build-time dependencies, which are required when the
451software is built; and runtime dependencies, which are required to be
452installed on the target in order for the software to run.
453
454Within a recipe, you specify build-time dependencies using the
455:term:`DEPENDS` variable. Although there are nuances,
456items specified in :term:`DEPENDS` should be names of other
457recipes. It is important that you specify all build-time dependencies
458explicitly.
459
460Another consideration is that configure scripts might automatically
461check for optional dependencies and enable corresponding functionality
462if those dependencies are found. If you wish to make a recipe that is
463more generally useful (e.g. publish the recipe in a layer for others to
464use), instead of hard-disabling the functionality, you can use the
465:term:`PACKAGECONFIG` variable to allow functionality and the
466corresponding dependencies to be enabled and disabled easily by other
467users of the recipe.
468
469Similar to build-time dependencies, you specify runtime dependencies
470through a variable -
471:term:`RDEPENDS`, which is
472package-specific. All variables that are package-specific need to have
473the name of the package added to the end as an override. Since the main
474package for a recipe has the same name as the recipe, and the recipe's
475name can be found through the
476``${``\ :term:`PN`\ ``}`` variable, then
477you specify the dependencies for the main package by setting
478``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you
479would set ``RDEPENDS:${PN}-tools``, and so forth.
480
481Some runtime dependencies will be set automatically at packaging time.
482These dependencies include any shared library dependencies (i.e. if a
483package "example" contains "libexample" and another package "mypackage"
484contains a binary that links to "libexample" then the OpenEmbedded build
485system will automatically add a runtime dependency to "mypackage" on
486"example"). See the
487":ref:`overview-manual/concepts:automatically added runtime dependencies`"
488section in the Yocto Project Overview and Concepts Manual for further
489details.
490
491Configuring the Recipe
492======================
493
494Most software provides some means of setting build-time configuration
495options before compilation. Typically, setting these options is
496accomplished by running a configure script with options, or by modifying
497a build configuration file.
498
499.. note::
500
501 As of Yocto Project Release 1.7, some of the core recipes that
502 package binary configuration scripts now disable the scripts due to
503 the scripts previously requiring error-prone path substitution. The
504 OpenEmbedded build system uses ``pkg-config`` now, which is much more
505 robust. You can find a list of the ``*-config`` scripts that are disabled
506 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
507 in the Yocto Project Reference Manual.
508
509A major part of build-time configuration is about checking for
510build-time dependencies and possibly enabling optional functionality as
511a result. You need to specify any build-time dependencies for the
512software you are building in your recipe's
513:term:`DEPENDS` value, in terms of
514other recipes that satisfy those dependencies. You can often find
515build-time or runtime dependencies described in the software's
516documentation.
517
518The following list provides configuration items of note based on how
519your software is built:
520
521- *Autotools:* If your source files have a ``configure.ac`` file, then
522 your software is built using Autotools. If this is the case, you just
523 need to modify the configuration.
524
525 When using Autotools, your recipe needs to inherit the
526 :ref:`ref-classes-autotools` class and it does not have to
527 contain a :ref:`ref-tasks-configure` task. However, you might still want to
528 make some adjustments. For example, you can set :term:`EXTRA_OECONF` or
529 :term:`PACKAGECONFIG_CONFARGS` to pass any needed configure options that
530 are specific to the recipe.
531
532- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
533 your software is built using CMake. If this is the case, you just
534 need to modify the configuration.
535
536 When you use CMake, your recipe needs to inherit the
537 :ref:`ref-classes-cmake` class and it does not have to contain a
538 :ref:`ref-tasks-configure` task. You can make some adjustments by setting
539 :term:`EXTRA_OECMAKE` to pass any needed configure options that are
540 specific to the recipe.
541
542 .. note::
543
544 If you need to install one or more custom CMake toolchain files
545 that are supplied by the application you are building, install the
546 files to ``${D}${datadir}/cmake/Modules`` during :ref:`ref-tasks-install`.
547
548- *Other:* If your source files do not have a ``configure.ac`` or
549 ``CMakeLists.txt`` file, then your software is built using some
550 method other than Autotools or CMake. If this is the case, you
551 normally need to provide a
552 :ref:`ref-tasks-configure` task
553 in your recipe unless, of course, there is nothing to configure.
554
555 Even if your software is not being built by Autotools or CMake, you
556 still might not need to deal with any configuration issues. You need
557 to determine if configuration is even a required step. You might need
558 to modify a Makefile or some configuration file used for the build to
559 specify necessary build options. Or, perhaps you might need to run a
560 provided, custom configure script with the appropriate options.
561
562 For the case involving a custom configure script, you would run
563 ``./configure --help`` and look for the options you need to set.
564
565Once configuration succeeds, it is always good practice to look at the
566``log.do_configure`` file to ensure that the appropriate options have
567been enabled and no additional build-time dependencies need to be added
568to :term:`DEPENDS`. For example, if the configure script reports that it
569found something not mentioned in :term:`DEPENDS`, or that it did not find
570something that it needed for some desired optional functionality, then
571you would need to add those to :term:`DEPENDS`. Looking at the log might
572also reveal items being checked for, enabled, or both that you do not
573want, or items not being found that are in :term:`DEPENDS`, in which case
574you would need to look at passing extra options to the configure script
575as needed. For reference information on configure options specific to
576the software you are building, you can consult the output of the
577``./configure --help`` command within ``${S}`` or consult the software's
578upstream documentation.
579
580Using Headers to Interface with Devices
581=======================================
582
583If your recipe builds an application that needs to communicate with some
584device or needs an API into a custom kernel, you will need to provide
585appropriate header files. Under no circumstances should you ever modify
586the existing
587``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
588These headers are used to build ``libc`` and must not be compromised
589with custom or machine-specific header information. If you customize
590``libc`` through modified headers all other applications that use
591``libc`` thus become affected.
592
593.. note::
594
595 Never copy and customize the ``libc`` header file (i.e.
596 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
597
598The correct way to interface to a device or custom kernel is to use a
599separate package that provides the additional headers for the driver or
600other unique interfaces. When doing so, your application also becomes
601responsible for creating a dependency on that specific provider.
602
603Consider the following:
604
605- Never modify ``linux-libc-headers.inc``. Consider that file to be
606 part of the ``libc`` system, and not something you use to access the
607 kernel directly. You should access ``libc`` through specific ``libc``
608 calls.
609
610- Applications that must talk directly to devices should either provide
611 necessary headers themselves, or establish a dependency on a special
612 headers package that is specific to that driver.
613
614For example, suppose you want to modify an existing header that adds I/O
615control or network support. If the modifications are used by a small
616number programs, providing a unique version of a header is easy and has
617little impact. When doing so, bear in mind the guidelines in the
618previous list.
619
620.. note::
621
622 If for some reason your changes need to modify the behavior of the ``libc``,
623 and subsequently all other applications on the system, use a ``.bbappend``
624 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
625 make the changes machine specific.
626
627Consider a case where your kernel is older and you need an older
628``libc`` ABI. The headers installed by your recipe should still be a
629standard mainline kernel, not your own custom one.
630
631When you use custom kernel headers you need to get them from
632:term:`STAGING_KERNEL_DIR`,
633which is the directory with kernel headers that are required to build
634out-of-tree modules. Your recipe will also need the following::
635
636 do_configure[depends] += "virtual/kernel:do_shared_workdir"
637
638Compilation
639===========
640
641During a build, the :ref:`ref-tasks-compile` task happens after source is fetched,
642unpacked, and configured. If the recipe passes through :ref:`ref-tasks-compile`
643successfully, nothing needs to be done.
644
645However, if the compile step fails, you need to diagnose the failure.
646Here are some common issues that cause failures.
647
648.. note::
649
650 For cases where improper paths are detected for configuration files
651 or for when libraries/headers cannot be found, be sure you are using
652 the more robust ``pkg-config``. See the note in section
653 ":ref:`dev-manual/new-recipe:Configuring the Recipe`" for additional information.
654
655- *Parallel build failures:* These failures manifest themselves as
656 intermittent errors, or errors reporting that a file or directory
657 that should be created by some other part of the build process could
658 not be found. This type of failure can occur even if, upon
659 inspection, the file or directory does exist after the build has
660 failed, because that part of the build process happened in the wrong
661 order.
662
663 To fix the problem, you need to either satisfy the missing dependency
664 in the Makefile or whatever script produced the Makefile, or (as a
665 workaround) set :term:`PARALLEL_MAKE` to an empty string::
666
667 PARALLEL_MAKE = ""
668
669 For information on parallel Makefile issues, see the
670 ":ref:`dev-manual/debugging:debugging parallel make races`" section.
671
672- *Improper host path usage:* This failure applies to recipes building
673 for the target or ":ref:`ref-classes-nativesdk`" only. The
674 failure occurs when the compilation process uses improper headers,
675 libraries, or other files from the host system when cross-compiling for
676 the target.
677
678 To fix the problem, examine the ``log.do_compile`` file to identify
679 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
680 so forth) and then either add configure options, apply a patch, or do
681 both.
682
683- *Failure to find required libraries/headers:* If a build-time
684 dependency is missing because it has not been declared in
685 :term:`DEPENDS`, or because the
686 dependency exists but the path used by the build process to find the
687 file is incorrect and the configure step did not detect it, the
688 compilation process could fail. For either of these failures, the
689 compilation process notes that files could not be found. In these
690 cases, you need to go back and add additional options to the
691 configure script as well as possibly add additional build-time
692 dependencies to :term:`DEPENDS`.
693
694 Occasionally, it is necessary to apply a patch to the source to
695 ensure the correct paths are used. If you need to specify paths to
696 find files staged into the sysroot from other recipes, use the
697 variables that the OpenEmbedded build system provides (e.g.
698 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
699 forth).
700
701Installing
702==========
703
704During :ref:`ref-tasks-install`, the task copies the built files along with their
705hierarchy to locations that would mirror their locations on the target
706device. The installation process copies files from the
707``${``\ :term:`S`\ ``}``,
708``${``\ :term:`B`\ ``}``, and
709``${``\ :term:`WORKDIR`\ ``}``
710directories to the ``${``\ :term:`D`\ ``}``
711directory to create the structure as it should appear on the target
712system.
713
714How your software is built affects what you must do to be sure your
715software is installed correctly. The following list describes what you
716must do for installation depending on the type of build system used by
717the software being built:
718
719- *Autotools and CMake:* If the software your recipe is building uses
720 Autotools or CMake, the OpenEmbedded build system understands how to
721 install the software. Consequently, you do not have to have a
722 :ref:`ref-tasks-install` task as part of your recipe. You just need to make
723 sure the install portion of the build completes with no issues.
724 However, if you wish to install additional files not already being
725 installed by ``make install``, you should do this using a
726 ``do_install:append`` function using the install command as described
727 in the "Manual" bulleted item later in this list.
728
729- *Other (using* ``make install``\ *)*: You need to define a :ref:`ref-tasks-install`
730 function in your recipe. The function should call
731 ``oe_runmake install`` and will likely need to pass in the
732 destination directory as well. How you pass that path is dependent on
733 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
734 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
735
736 For an example recipe using ``make install``, see the
737 ":ref:`dev-manual/new-recipe:building a makefile-based package`" section.
738
739- *Manual:* You need to define a :ref:`ref-tasks-install` function in your
740 recipe. The function must first use ``install -d`` to create the
741 directories under
742 ``${``\ :term:`D`\ ``}``. Once the
743 directories exist, your function can use ``install`` to manually
744 install the built software into the directories.
745
746 You can find more information on ``install`` at
747 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
748
749For the scenarios that do not use Autotools or CMake, you need to track
750the installation and diagnose and fix any issues until everything
751installs correctly. You need to look in the default location of
752``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
753installed correctly.
754
755.. note::
756
757 - During the installation process, you might need to modify some of
758 the installed files to suit the target layout. For example, you
759 might need to replace hard-coded paths in an initscript with
760 values of variables provided by the build system, such as
761 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
762 modifications during :ref:`ref-tasks-install`, be sure to modify the
763 destination file after copying rather than before copying.
764 Modifying after copying ensures that the build system can
765 re-execute :ref:`ref-tasks-install` if needed.
766
767 - ``oe_runmake install``, which can be run directly or can be run
768 indirectly by the :ref:`ref-classes-autotools` and
769 :ref:`ref-classes-cmake` classes, runs ``make install`` in parallel.
770 Sometimes, a Makefile can have missing dependencies between targets that
771 can result in race conditions. If you experience intermittent failures
772 during :ref:`ref-tasks-install`, you might be able to work around them by
773 disabling parallel Makefile installs by adding the following to the
774 recipe::
775
776 PARALLEL_MAKEINST = ""
777
778 See :term:`PARALLEL_MAKEINST` for additional information.
779
780 - If you need to install one or more custom CMake toolchain files
781 that are supplied by the application you are building, install the
782 files to ``${D}${datadir}/cmake/Modules`` during
783 :ref:`ref-tasks-install`.
784
785Enabling System Services
786========================
787
788If you want to install a service, which is a process that usually starts
789on boot and runs in the background, then you must include some
790additional definitions in your recipe.
791
792If you are adding services and the service initialization script or the
793service file itself is not installed, you must provide for that
794installation in your recipe using a ``do_install:append`` function. If
795your recipe already has a :ref:`ref-tasks-install` function, update the function
796near its end rather than adding an additional ``do_install:append``
797function.
798
799When you create the installation for your services, you need to
800accomplish what is normally done by ``make install``. In other words,
801make sure your installation arranges the output similar to how it is
802arranged on the target system.
803
804The OpenEmbedded build system provides support for starting services two
805different ways:
806
807- *SysVinit:* SysVinit is a system and service manager that manages the
808 init system used to control the very basic functions of your system.
809 The init program is the first program started by the Linux kernel
810 when the system boots. Init then controls the startup, running and
811 shutdown of all other programs.
812
813 To enable a service using SysVinit, your recipe needs to inherit the
814 :ref:`ref-classes-update-rc.d` class. The class helps
815 facilitate safely installing the package on the target.
816
817 You will need to set the
818 :term:`INITSCRIPT_PACKAGES`,
819 :term:`INITSCRIPT_NAME`,
820 and
821 :term:`INITSCRIPT_PARAMS`
822 variables within your recipe.
823
824- *systemd:* System Management Daemon (systemd) was designed to replace
825 SysVinit and to provide enhanced management of services. For more
826 information on systemd, see the systemd homepage at
827 https://freedesktop.org/wiki/Software/systemd/.
828
829 To enable a service using systemd, your recipe needs to inherit the
830 :ref:`ref-classes-systemd` class. See the ``systemd.bbclass`` file
831 located in your :term:`Source Directory` section for more information.
832
833Packaging
834=========
835
836Successful packaging is a combination of automated processes performed
837by the OpenEmbedded build system and some specific steps you need to
838take. The following list describes the process:
839
840- *Splitting Files*: The :ref:`ref-tasks-package` task splits the files produced
841 by the recipe into logical components. Even software that produces a
842 single binary might still have debug symbols, documentation, and
843 other logical components that should be split out. The :ref:`ref-tasks-package`
844 task ensures that files are split up and packaged correctly.
845
846- *Running QA Checks*: The :ref:`ref-classes-insane` class adds a
847 step to the package generation process so that output quality
848 assurance checks are generated by the OpenEmbedded build system. This
849 step performs a range of checks to be sure the build's output is free
850 of common problems that show up during runtime. For information on
851 these checks, see the :ref:`ref-classes-insane` class and
852 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
853 chapter in the Yocto Project Reference Manual.
854
855- *Hand-Checking Your Packages*: After you build your software, you
856 need to be sure your packages are correct. Examine the
857 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
858 directory and make sure files are where you expect them to be. If you
859 discover problems, you can set
860 :term:`PACKAGES`,
861 :term:`FILES`,
862 ``do_install(:append)``, and so forth as needed.
863
864- *Splitting an Application into Multiple Packages*: If you need to
865 split an application into several packages, see the
866 ":ref:`dev-manual/new-recipe:splitting an application into multiple packages`"
867 section for an example.
868
869- *Installing a Post-Installation Script*: For an example showing how
870 to install a post-installation script, see the
871 ":ref:`dev-manual/new-recipe:post-installation scripts`" section.
872
873- *Marking Package Architecture*: Depending on what your recipe is
874 building and how it is configured, it might be important to mark the
875 packages produced as being specific to a particular machine, or to
876 mark them as not being specific to a particular machine or
877 architecture at all.
878
879 By default, packages apply to any machine with the same architecture
880 as the target machine. When a recipe produces packages that are
881 machine-specific (e.g. the
882 :term:`MACHINE` value is passed
883 into the configure script or a patch is applied only for a particular
884 machine), you should mark them as such by adding the following to the
885 recipe::
886
887 PACKAGE_ARCH = "${MACHINE_ARCH}"
888
889 On the other hand, if the recipe produces packages that do not
890 contain anything specific to the target machine or architecture at
891 all (e.g. recipes that simply package script files or configuration
892 files), you should use the :ref:`ref-classes-allarch` class to
893 do this for you by adding this to your recipe::
894
895 inherit allarch
896
897 Ensuring that the package architecture is correct is not critical
898 while you are doing the first few builds of your recipe. However, it
899 is important in order to ensure that your recipe rebuilds (or does
900 not rebuild) appropriately in response to changes in configuration,
901 and to ensure that you get the appropriate packages installed on the
902 target machine, particularly if you run separate builds for more than
903 one target machine.
904
905Sharing Files Between Recipes
906=============================
907
908Recipes often need to use files provided by other recipes on the build
909host. For example, an application linking to a common library needs
910access to the library itself and its associated headers. The way this
911access is accomplished is by populating a sysroot with files. Each
912recipe has two sysroots in its work directory, one for target files
913(``recipe-sysroot``) and one for files that are native to the build host
914(``recipe-sysroot-native``).
915
916.. note::
917
918 You could find the term "staging" used within the Yocto project
919 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
920 variable).
921
922Recipes should never populate the sysroot directly (i.e. write files
923into sysroot). Instead, files should be installed into standard
924locations during the
925:ref:`ref-tasks-install` task within
926the ``${``\ :term:`D`\ ``}`` directory. The
927reason for this limitation is that almost all files that populate the
928sysroot are cataloged in manifests in order to ensure the files can be
929removed later when a recipe is either modified or removed. Thus, the
930sysroot is able to remain free from stale files.
931
932A subset of the files installed by the :ref:`ref-tasks-install` task are
933used by the :ref:`ref-tasks-populate_sysroot` task as defined by the
934:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
935is possible to modify the list of directories that populate the sysroot.
936The following example shows how you could add the ``/opt`` directory to
937the list of directories within a recipe::
938
939 SYSROOT_DIRS += "/opt"
940
941.. note::
942
943 The `/sysroot-only` is to be used by recipes that generate artifacts
944 that are not included in the target filesystem, allowing them to share
945 these artifacts without needing to use the :term:`DEPLOY_DIR`.
946
947For a more complete description of the :ref:`ref-tasks-populate_sysroot`
948task and its associated functions, see the
949:ref:`staging <ref-classes-staging>` class.
950
951Using Virtual Providers
952=======================
953
954Prior to a build, if you know that several different recipes provide the
955same functionality, you can use a virtual provider (i.e. ``virtual/*``)
956as a placeholder for the actual provider. The actual provider is
957determined at build-time.
958
959A common scenario where a virtual provider is used would be for the kernel
960recipe. Suppose you have three kernel recipes whose :term:`PN` values map to
961``kernel-big``, ``kernel-mid``, and ``kernel-small``. Furthermore, each of
962these recipes in some way uses a :term:`PROVIDES` statement that essentially
963identifies itself as being able to provide ``virtual/kernel``. Here is one way
964through the :ref:`ref-classes-kernel` class::
965
966 PROVIDES += "virtual/kernel"
967
968Any recipe that inherits the :ref:`ref-classes-kernel` class is
969going to utilize a :term:`PROVIDES` statement that identifies that recipe as
970being able to provide the ``virtual/kernel`` item.
971
972Now comes the time to actually build an image and you need a kernel
973recipe, but which one? You can configure your build to call out the
974kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
975an example, consider the :yocto_git:`x86-base.inc
976</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a
977machine (i.e. :term:`MACHINE`) configuration file. This include file is the
978reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
979relevant lines from the include file::
980
981 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
982 PREFERRED_VERSION_linux-yocto ??= "4.15%"
983
984When you use a virtual provider, you do not have to "hard code" a recipe
985name as a build dependency. You can use the
986:term:`DEPENDS` variable to state the
987build is dependent on ``virtual/kernel`` for example::
988
989 DEPENDS = "virtual/kernel"
990
991During the build, the OpenEmbedded build system picks
992the correct recipe needed for the ``virtual/kernel`` dependency based on
993the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
994mentioned at the beginning of this section, configure your build as
995follows::
996
997 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
998
999.. note::
1000
1001 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
1002 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
1003 recipes from building is usually the desired behavior since this mechanism's
1004 purpose is to select between mutually exclusive alternative providers.
1005
1006The following lists specific examples of virtual providers:
1007
1008- ``virtual/kernel``: Provides the name of the kernel recipe to use
1009 when building a kernel image.
1010
1011- ``virtual/bootloader``: Provides the name of the bootloader to use
1012 when building an image.
1013
1014- ``virtual/libgbm``: Provides ``gbm.pc``.
1015
1016- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
1017
1018- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
1019
1020- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
1021
1022- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
1023
1024.. note::
1025
1026 Virtual providers only apply to build time dependencies specified with
1027 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
1028 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
1029
1030Properly Versioning Pre-Release Recipes
1031=======================================
1032
1033Sometimes the name of a recipe can lead to versioning problems when the
1034recipe is upgraded to a final release. For example, consider the
1035``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
1036the ":ref:`dev-manual/new-recipe:storing and naming the recipe`" section.
1037This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
1038released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
1039change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
1040build system and package managers, so the resulting packages will not
1041correctly trigger an upgrade.
1042
1043In order to ensure the versions compare properly, the recommended
1044convention is to use a tilde (``~``) character as follows::
1045
1046 PV = 0.8.16~rc1
1047
1048This way ``0.8.16~rc1`` sorts before ``0.8.16``. See the
1049":ref:`contributor-guide/recipe-style-guide:version policy`" section in the
1050Yocto Project and OpenEmbedded Contributor Guide for more details about
1051versioning code corresponding to a pre-release or to a specific Git commit.
1052
1053Post-Installation Scripts
1054=========================
1055
1056Post-installation scripts run immediately after installing a package on
1057the target or during image creation when a package is included in an
1058image. To add a post-installation script to a package, add a
1059``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file
1060(``.bb``) and replace `PACKAGENAME` with the name of the package you want
1061to attach to the ``postinst`` script. To apply the post-installation
1062script to the main package for the recipe, which is usually what is
1063required, specify
1064``${``\ :term:`PN`\ ``}`` in place of
1065PACKAGENAME.
1066
1067A post-installation function has the following structure::
1068
1069 pkg_postinst:PACKAGENAME() {
1070 # Commands to carry out
1071 }
1072
1073The script defined in the post-installation function is called when the
1074root filesystem is created. If the script succeeds, the package is
1075marked as installed.
1076
1077.. note::
1078
1079 Any RPM post-installation script that runs on the target should
1080 return a 0 exit code. RPM does not allow non-zero exit codes for
1081 these scripts, and the RPM package manager will cause the package to
1082 fail installation on the target.
1083
1084Sometimes it is necessary for the execution of a post-installation
1085script to be delayed until the first boot. For example, the script might
1086need to be executed on the device itself. To delay script execution
1087until boot time, you must explicitly mark post installs to defer to the
1088target. You can use ``pkg_postinst_ontarget()`` or call
1089``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
1090failure of a ``pkg_postinst()`` script (including exit 1) triggers an
1091error during the
1092:ref:`ref-tasks-rootfs` task.
1093
1094If you have recipes that use ``pkg_postinst`` function and they require
1095the use of non-standard native tools that have dependencies during
1096root filesystem construction, you need to use the
1097:term:`PACKAGE_WRITE_DEPS`
1098variable in your recipe to list these tools. If you do not use this
1099variable, the tools might be missing and execution of the
1100post-installation script is deferred until first boot. Deferring the
1101script to the first boot is undesirable and impossible for read-only
1102root filesystems.
1103
1104.. note::
1105
1106 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
1107 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
1108 respectively. These scrips work in exactly the same way as does
1109 ``pkg_postinst`` with the exception that they run at different times. Also,
1110 because of when they run, they are not applicable to being run at image
1111 creation time like ``pkg_postinst``.
1112
1113Testing
1114=======
1115
1116The final step for completing your recipe is to be sure that the
1117software you built runs correctly. To accomplish runtime testing, add
1118the build's output packages to your image and test them on the target.
1119
1120For information on how to customize your image by adding specific
1121packages, see ":ref:`dev-manual/customizing-images:customizing images`" section.
1122
1123Examples
1124========
1125
1126To help summarize how to write a recipe, this section provides some
1127recipe examples given various scenarios:
1128
1129- `Building a single .c file package`_
1130
1131- `Building a Makefile-based package`_
1132
1133- `Building an Autotooled package`_
1134
1135- `Building a Meson package`_
1136
1137- `Splitting an application into multiple packages`_
1138
1139- `Packaging externally produced binaries`_
1140
1141Building a Single .c File Package
1142---------------------------------
1143
1144Building an application from a single file that is stored locally (e.g. under
1145``files``) requires a recipe that has the file listed in the :term:`SRC_URI`
1146variable. Additionally, you need to manually write the :ref:`ref-tasks-compile`
1147and :ref:`ref-tasks-install` tasks. The :term:`S` variable defines the
1148directory containing the source code, which is set to :term:`WORKDIR` in this
1149case --- the directory BitBake uses for the build::
1150
1151 SUMMARY = "Simple helloworld application"
1152 SECTION = "examples"
1153 LICENSE = "MIT"
1154 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
1155
1156 SRC_URI = "file://helloworld.c"
1157
1158 S = "${WORKDIR}"
1159
1160 do_compile() {
1161 ${CC} ${LDFLAGS} helloworld.c -o helloworld
1162 }
1163
1164 do_install() {
1165 install -d ${D}${bindir}
1166 install -m 0755 helloworld ${D}${bindir}
1167 }
1168
1169By default, the ``helloworld``, ``helloworld-dbg``, and ``helloworld-dev`` packages
1170are built. For information on how to customize the packaging process, see the
1171":ref:`dev-manual/new-recipe:splitting an application into multiple packages`"
1172section.
1173
1174Building a Makefile-Based Package
1175---------------------------------
1176
1177Applications built with GNU ``make`` require a recipe that has the source archive
1178listed in :term:`SRC_URI`. You do not need to add a :ref:`ref-tasks-compile`
1179step since by default BitBake starts the ``make`` command to compile the
1180application. If you need additional ``make`` options, you should store them in
1181the :term:`EXTRA_OEMAKE` or :term:`PACKAGECONFIG_CONFARGS` variables. BitBake
1182passes these options into the GNU ``make`` invocation. Note that a
1183:ref:`ref-tasks-install` task is still required. Otherwise, BitBake runs an
1184empty :ref:`ref-tasks-install` task by default.
1185
1186Some applications might require extra parameters to be passed to the
1187compiler. For example, the application might need an additional header
1188path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
1189following example shows this::
1190
1191 CFLAGS:prepend = "-I ${S}/include "
1192
1193In the following example, ``lz4`` is a makefile-based package::
1194
1195 SUMMARY = "Extremely Fast Compression algorithm"
1196 DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems."
1197 HOMEPAGE = "https://github.com/lz4/lz4"
1198
1199 LICENSE = "BSD-2-Clause | GPL-2.0-only"
1200 LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc \
1201 file://programs/COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
1202 file://LICENSE;md5=d57c0d21cb917fb4e0af2454aa48b956 \
1203 "
1204
1205 PE = "1"
1206
1207 SRCREV = "d44371841a2f1728a3f36839fd4b7e872d0927d3"
1208
1209 SRC_URI = "git://github.com/lz4/lz4.git;branch=release;protocol=https \
1210 file://CVE-2021-3520.patch \
1211 "
1212 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>.*)"
1213
1214 S = "${WORKDIR}/git"
1215
1216 CVE_STATUS[CVE-2014-4715] = "fixed-version: Fixed in r118, which is larger than the current version"
1217
1218 EXTRA_OEMAKE = "PREFIX=${prefix} CC='${CC}' CFLAGS='${CFLAGS}' DESTDIR=${D} LIBDIR=${libdir} INCLUDEDIR=${includedir} BUILD_STATIC=no"
1219
1220 do_install() {
1221 oe_runmake install
1222 }
1223
1224 BBCLASSEXTEND = "native nativesdk"
1225
1226Building an Autotooled Package
1227------------------------------
1228
1229Applications built with the Autotools such as ``autoconf`` and ``automake``
1230require a recipe that has a source archive listed in :term:`SRC_URI` and also
1231inherit the :ref:`ref-classes-autotools` class, which contains the definitions
1232of all the steps needed to build an Autotool-based application. The result of
1233the build is automatically packaged. And, if the application uses NLS for
1234localization, packages with local information are generated (one package per
1235language). Here is one example: (``hello_2.3.bb``)::
1236
1237 SUMMARY = "GNU Helloworld application"
1238 SECTION = "examples"
1239 LICENSE = "GPL-2.0-or-later"
1240 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
1241
1242 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
1243
1244 inherit autotools gettext
1245
1246The variable :term:`LIC_FILES_CHKSUM` is used to track source license changes
1247as described in the ":ref:`dev-manual/licenses:tracking license changes`"
1248section in the Yocto Project Overview and Concepts Manual. You can quickly
1249create Autotool-based recipes in a manner similar to the previous example.
1250
1251.. _ref-building-meson-package:
1252
1253Building a Meson Package
1254------------------------
1255
1256Applications built with the `Meson build system <https://mesonbuild.com/>`__
1257just need a recipe that has sources described in :term:`SRC_URI` and inherits
1258the :ref:`ref-classes-meson` class.
1259
1260The :oe_git:`ipcalc recipe </meta-openembedded/tree/meta-networking/recipes-support/ipcalc>`
1261is a simple example of an application without dependencies::
1262
1263 SUMMARY = "Tool to assist in network address calculations for IPv4 and IPv6."
1264 HOMEPAGE = "https://gitlab.com/ipcalc/ipcalc"
1265
1266 SECTION = "net"
1267
1268 LICENSE = "GPL-2.0-only"
1269 LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
1270
1271 SRC_URI = "git://gitlab.com/ipcalc/ipcalc.git;protocol=https;branch=master"
1272 SRCREV = "4c4261a47f355946ee74013d4f5d0494487cc2d6"
1273
1274 S = "${WORKDIR}/git"
1275
1276 inherit meson
1277
1278Applications with dependencies are likely to inherit the
1279:ref:`ref-classes-pkgconfig` class, as ``pkg-config`` is the default method
1280used by Meson to find dependencies and compile applications against them.
1281
1282Splitting an Application into Multiple Packages
1283-----------------------------------------------
1284
1285You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
1286application into multiple packages.
1287
1288Here is an example that uses the ``libxpm`` recipe. By default,
1289this recipe generates a single package that contains the library along
1290with a few binaries. You can modify the recipe to split the binaries
1291into separate packages::
1292
1293 require xorg-lib-common.inc
1294
1295 SUMMARY = "Xpm: X Pixmap extension library"
1296 LICENSE = "MIT"
1297 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
1298 DEPENDS += "libxext libsm libxt"
1299 PE = "1"
1300
1301 XORG_PN = "libXpm"
1302
1303 PACKAGES =+ "sxpm cxpm"
1304 FILES:cxpm = "${bindir}/cxpm"
1305 FILES:sxpm = "${bindir}/sxpm"
1306
1307In the previous example, we want to ship the ``sxpm`` and ``cxpm``
1308binaries in separate packages. Since ``bindir`` would be packaged into
1309the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
1310so additional package names are added to the start of list. This results
1311in the extra ``FILES:*`` variables then containing information that
1312define which files and directories go into which packages. Files
1313included by earlier packages are skipped by latter packages. Thus, the
1314main :term:`PN` package does not include the above listed files.
1315
1316Packaging Externally Produced Binaries
1317--------------------------------------
1318
1319Sometimes, you need to add pre-compiled binaries to an image. For
1320example, suppose that there are binaries for proprietary code,
1321created by a particular division of a company. Your part of the company
1322needs to use those binaries as part of an image that you are building
1323using the OpenEmbedded build system. Since you only have the binaries
1324and not the source code, you cannot use a typical recipe that expects to
1325fetch the source specified in
1326:term:`SRC_URI` and then compile it.
1327
1328One method is to package the binaries and then install them as part of
1329the image. Generally, it is not a good idea to package binaries since,
1330among other things, it can hinder the ability to reproduce builds and
1331could lead to compatibility problems with ABI in the future. However,
1332sometimes you have no choice.
1333
1334The easiest solution is to create a recipe that uses the
1335:ref:`ref-classes-bin-package` class and to be sure that you are using default
1336locations for build artifacts. In most cases, the
1337:ref:`ref-classes-bin-package` class handles "skipping" the configure and
1338compile steps as well as sets things up to grab packages from the appropriate
1339area. In particular, this class sets ``noexec`` on both the
1340:ref:`ref-tasks-configure` and :ref:`ref-tasks-compile` tasks, sets
1341``FILES:${PN}`` to "/" so that it picks up all files, and sets up a
1342:ref:`ref-tasks-install` task, which effectively copies all files from ``${S}``
1343to ``${D}``. The :ref:`ref-classes-bin-package` class works well when the files
1344extracted into ``${S}`` are already laid out in the way they should be laid out
1345on the target. For more information on these variables, see the :term:`FILES`,
1346:term:`PN`, :term:`S`, and :term:`D` variables in the Yocto Project Reference
1347Manual's variable glossary.
1348
1349.. note::
1350
1351 - Using :term:`DEPENDS` is a good
1352 idea even for components distributed in binary form, and is often
1353 necessary for shared libraries. For a shared library, listing the
1354 library dependencies in :term:`DEPENDS` makes sure that the libraries
1355 are available in the staging sysroot when other recipes link
1356 against the library, which might be necessary for successful
1357 linking.
1358
1359 - Using :term:`DEPENDS` also allows runtime dependencies between
1360 packages to be added automatically. See the
1361 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
1362 section in the Yocto Project Overview and Concepts Manual for more
1363 information.
1364
1365If you cannot use the :ref:`ref-classes-bin-package` class, you need to be sure you are
1366doing the following:
1367
1368- Create a recipe where the
1369 :ref:`ref-tasks-configure` and
1370 :ref:`ref-tasks-compile` tasks do
1371 nothing: It is usually sufficient to just not define these tasks in
1372 the recipe, because the default implementations do nothing unless a
1373 Makefile is found in
1374 ``${``\ :term:`S`\ ``}``.
1375
1376 If ``${S}`` might contain a Makefile, or if you inherit some class
1377 that replaces :ref:`ref-tasks-configure` and :ref:`ref-tasks-compile` with custom
1378 versions, then you can use the
1379 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
1380 flag to turn the tasks into no-ops, as follows::
1381
1382 do_configure[noexec] = "1"
1383 do_compile[noexec] = "1"
1384
1385 Unlike :ref:`bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
1386 using the flag preserves the dependency chain from the :ref:`ref-tasks-fetch`,
1387 :ref:`ref-tasks-unpack`, and :ref:`ref-tasks-patch` tasks to the
1388 :ref:`ref-tasks-install` task.
1389
1390- Make sure your :ref:`ref-tasks-install` task installs the binaries
1391 appropriately.
1392
1393- Ensure that you set up :term:`FILES`
1394 (usually
1395 ``FILES:${``\ :term:`PN`\ ``}``) to
1396 point to the files you have installed, which of course depends on
1397 where you have installed them and whether those files are in
1398 different locations than the defaults.
1399
1400Following Recipe Style Guidelines
1401=================================
1402
1403When writing recipes, it is good to conform to existing style guidelines.
1404See the ":doc:`../contributor-guide/recipe-style-guide`" in the Yocto Project
1405and OpenEmbedded Contributor Guide for reference.
1406
1407It is common for existing recipes to deviate a bit from this style.
1408However, aiming for at least a consistent style is a good idea. Some
1409practices, such as omitting spaces around ``=`` operators in assignments
1410or ordering recipe components in an erratic way, are widely seen as poor
1411style.
1412
1413Recipe Syntax
1414=============
1415
1416Understanding recipe file syntax is important for writing recipes. The
1417following list overviews the basic items that make up a BitBake recipe
1418file. For more complete BitBake syntax descriptions, see the
1419":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`"
1420chapter of the BitBake User Manual.
1421
1422- *Variable Assignments and Manipulations:* Variable assignments allow
1423 a value to be assigned to a variable. The assignment can be static
1424 text or might include the contents of other variables. In addition to
1425 the assignment, appending and prepending operations are also
1426 supported.
1427
1428 The following example shows some of the ways you can use variables in
1429 recipes::
1430
1431 S = "${WORKDIR}/postfix-${PV}"
1432 CFLAGS += "-DNO_ASM"
1433 CFLAGS:append = " --enable-important-feature"
1434
1435- *Functions:* Functions provide a series of actions to be performed.
1436 You usually use functions to override the default implementation of a
1437 task function or to complement a default function (i.e. append or
1438 prepend to an existing function). Standard functions use ``sh`` shell
1439 syntax, although access to OpenEmbedded variables and internal
1440 methods are also available.
1441
1442 Here is an example function from the ``sed`` recipe::
1443
1444 do_install () {
1445 autotools_do_install
1446 install -d ${D}${base_bindir}
1447 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
1448 rmdir ${D}${bindir}/
1449 }
1450
1451 It is
1452 also possible to implement new functions that are called between
1453 existing tasks as long as the new functions are not replacing or
1454 complementing the default functions. You can implement functions in
1455 Python instead of shell. Both of these options are not seen in the
1456 majority of recipes.
1457
1458- *Keywords:* BitBake recipes use only a few keywords. You use keywords
1459 to include common functions (``inherit``), load parts of a recipe
1460 from other files (``include`` and ``require``) and export variables
1461 to the environment (``export``).
1462
1463 The following example shows the use of some of these keywords::
1464
1465 export POSTCONF = "${STAGING_BINDIR}/postconf"
1466 inherit autoconf
1467 require otherfile.inc
1468
1469- *Comments (#):* Any lines that begin with the hash character (``#``)
1470 are treated as comment lines and are ignored::
1471
1472 # This is a comment
1473
1474This next list summarizes the most important and most commonly used
1475parts of the recipe syntax. For more information on these parts of the
1476syntax, you can reference the
1477":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
1478in the BitBake User Manual.
1479
1480- *Line Continuation (\\):* Use the backward slash (``\``) character to
1481 split a statement over multiple lines. Place the slash character at
1482 the end of the line that is to be continued on the next line::
1483
1484 VAR = "A really long \
1485 line"
1486
1487 .. note::
1488
1489 You cannot have any characters including spaces or tabs after the
1490 slash character.
1491
1492- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
1493 access the contents of a variable::
1494
1495 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
1496
1497 .. note::
1498
1499 It is important to understand that the value of a variable
1500 expressed in this form does not get substituted automatically. The
1501 expansion of these expressions happens on-demand later (e.g.
1502 usually when a function that makes reference to the variable
1503 executes). This behavior ensures that the values are most
1504 appropriate for the context in which they are finally used. On the
1505 rare occasion that you do need the variable expression to be
1506 expanded immediately, you can use the
1507 :=
1508 operator instead of
1509 =
1510 when you make the assignment, but this is not generally needed.
1511
1512- *Quote All Assignments ("value"):* Use double quotes around values in
1513 all variable assignments (e.g. ``"value"``). Here is an example::
1514
1515 VAR1 = "${OTHERVAR}"
1516 VAR2 = "The version is ${PV}"
1517
1518- *Conditional Assignment (?=):* Conditional assignment is used to
1519 assign a value to a variable, but only when the variable is currently
1520 unset. Use the question mark followed by the equal sign (``?=``) to
1521 make a "soft" assignment used for conditional assignment. Typically,
1522 "soft" assignments are used in the ``local.conf`` file for variables
1523 that are allowed to come through from the external environment.
1524
1525 Here is an example where ``VAR1`` is set to "New value" if it is
1526 currently empty. However, if ``VAR1`` has already been set, it
1527 remains unchanged::
1528
1529 VAR1 ?= "New value"
1530
1531 In this next example, ``VAR1`` is left with the value "Original value"::
1532
1533 VAR1 = "Original value"
1534 VAR1 ?= "New value"
1535
1536- *Appending (+=):* Use the plus character followed by the equals sign
1537 (``+=``) to append values to existing variables.
1538
1539 .. note::
1540
1541 This operator adds a space between the existing content of the
1542 variable and the new content.
1543
1544 Here is an example::
1545
1546 SRC_URI += "file://fix-makefile.patch"
1547
1548- *Prepending (=+):* Use the equals sign followed by the plus character
1549 (``=+``) to prepend values to existing variables.
1550
1551 .. note::
1552
1553 This operator adds a space between the new content and the
1554 existing content of the variable.
1555
1556 Here is an example::
1557
1558 VAR =+ "Starts"
1559
1560- *Appending (:append):* Use the ``:append`` operator to append values
1561 to existing variables. This operator does not add any additional
1562 space. Also, the operator is applied after all the ``+=``, and ``=+``
1563 operators have been applied and after all ``=`` assignments have
1564 occurred. This means that if ``:append`` is used in a recipe, it can
1565 only be overridden by another layer using the special ``:remove``
1566 operator, which in turn will prevent further layers from adding it back.
1567
1568 The following example shows the space being explicitly added to the
1569 start to ensure the appended value is not merged with the existing
1570 value::
1571
1572 CFLAGS:append = " --enable-important-feature"
1573
1574 You can also use
1575 the ``:append`` operator with overrides, which results in the actions
1576 only being performed for the specified target or machine::
1577
1578 CFLAGS:append:sh4 = " --enable-important-sh4-specific-feature"
1579
1580- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend
1581 values to existing variables. This operator does not add any
1582 additional space. Also, the operator is applied after all the ``+=``,
1583 and ``=+`` operators have been applied and after all ``=``
1584 assignments have occurred.
1585
1586 The following example shows the space being explicitly added to the
1587 end to ensure the prepended value is not merged with the existing
1588 value::
1589
1590 CFLAGS:prepend = "-I${S}/myincludes "
1591
1592 You can also use the
1593 ``:prepend`` operator with overrides, which results in the actions
1594 only being performed for the specified target or machine::
1595
1596 CFLAGS:prepend:sh4 = "-I${S}/myincludes "
1597
1598- *Overrides:* You can use overrides to set a value conditionally,
1599 typically based on how the recipe is being built. For example, to set
1600 the :term:`KBRANCH` variable's
1601 value to "standard/base" for any target
1602 :term:`MACHINE`, except for
1603 qemuarm where it should be set to "standard/arm-versatile-926ejs",
1604 you would do the following::
1605
1606 KBRANCH = "standard/base"
1607 KBRANCH:qemuarm = "standard/arm-versatile-926ejs"
1608
1609 Overrides are also used to separate
1610 alternate values of a variable in other situations. For example, when
1611 setting variables such as
1612 :term:`FILES` and
1613 :term:`RDEPENDS` that are
1614 specific to individual packages produced by a recipe, you should
1615 always use an override that specifies the name of the package.
1616
1617- *Indentation:* Use spaces for indentation rather than tabs. For
1618 shell functions, both currently work. However, it is a policy
1619 decision of the Yocto Project to use tabs in shell functions. Realize
1620 that some layers have a policy to use spaces for all indentation.
1621
1622- *Using Python for Complex Operations:* For more advanced processing,
1623 it is possible to use Python code during variable assignments (e.g.
1624 search and replacement on a variable).
1625
1626 You indicate Python code using the ``${@python_code}`` syntax for the
1627 variable assignment::
1628
1629 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
1630
1631- *Shell Function Syntax:* Write shell functions as if you were writing
1632 a shell script when you describe a list of actions to take. You
1633 should ensure that your script works with a generic ``sh`` and that
1634 it does not require any ``bash`` or other shell-specific
1635 functionality. The same considerations apply to various system
1636 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
1637 might wish to use. If in doubt, you should check with multiple
1638 implementations --- including those from BusyBox.
1639
diff --git a/documentation/dev-manual/packages.rst b/documentation/dev-manual/packages.rst
new file mode 100644
index 0000000000..e5028fffdc
--- /dev/null
+++ b/documentation/dev-manual/packages.rst
@@ -0,0 +1,1250 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working with Packages
4*********************
5
6This section describes a few tasks that involve packages:
7
8- :ref:`dev-manual/packages:excluding packages from an image`
9
10- :ref:`dev-manual/packages:incrementing a package version`
11
12- :ref:`dev-manual/packages:handling optional module packaging`
13
14- :ref:`dev-manual/packages:using runtime package management`
15
16- :ref:`dev-manual/packages:generating and using signed packages`
17
18- :ref:`Setting up and running package test
19 (ptest) <dev-manual/packages:testing packages with ptest>`
20
21- :ref:`dev-manual/packages:creating node package manager (npm) packages`
22
23- :ref:`dev-manual/packages:adding custom metadata to packages`
24
25Excluding Packages from an Image
26================================
27
28You might find it necessary to prevent specific packages from being
29installed into an image. If so, you can use several variables to direct
30the build system to essentially ignore installing recommended packages
31or to not install a package at all.
32
33The following list introduces variables you can use to prevent packages
34from being installed into your image. Each of these variables only works
35with IPK and RPM package types, not for Debian packages.
36Also, you can use these variables from your ``local.conf`` file
37or attach them to a specific image recipe by using a recipe name
38override. For more detail on the variables, see the descriptions in the
39Yocto Project Reference Manual's glossary chapter.
40
41- :term:`BAD_RECOMMENDATIONS`:
42 Use this variable to specify "recommended-only" packages that you do
43 not want installed.
44
45- :term:`NO_RECOMMENDATIONS`:
46 Use this variable to prevent all "recommended-only" packages from
47 being installed.
48
49- :term:`PACKAGE_EXCLUDE`:
50 Use this variable to prevent specific packages from being installed
51 regardless of whether they are "recommended-only" or not. You need to
52 realize that the build process could fail with an error when you
53 prevent the installation of a package whose presence is required by
54 an installed package.
55
56Incrementing a Package Version
57==============================
58
59This section provides some background on how binary package versioning
60is accomplished and presents some of the services, variables, and
61terminology involved.
62
63In order to understand binary package versioning, you need to consider
64the following:
65
66- Binary Package: The binary package that is eventually built and
67 installed into an image.
68
69- Binary Package Version: The binary package version is composed of two
70 components --- a version and a revision.
71
72 .. note::
73
74 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
75 but this discussion for the most part ignores :term:`PE`.
76
77 The version and revision are taken from the
78 :term:`PV` and
79 :term:`PR` variables, respectively.
80
81- :term:`PV`: The recipe version. :term:`PV` represents the version of the
82 software being packaged. Do not confuse :term:`PV` with the binary
83 package version.
84
85- :term:`PR`: The recipe revision.
86
87- :term:`SRCPV`: The OpenEmbedded
88 build system uses this string to help define the value of :term:`PV` when
89 the source code revision needs to be included in it.
90
91- :yocto_wiki:`PR Service </PR_Service>`: A
92 network-based service that helps automate keeping package feeds
93 compatible with existing package manager applications such as RPM,
94 APT, and OPKG.
95
96Whenever the binary package content changes, the binary package version
97must change. Changing the binary package version is accomplished by
98changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
99values occurs one of two ways:
100
101- Automatically using a Package Revision Service (PR Service).
102
103- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
104
105Given a primary challenge of any build system and its users is how to
106maintain a package feed that is compatible with existing package manager
107applications such as RPM, APT, and OPKG, using an automated system is
108much preferred over a manual system. In either system, the main
109requirement is that binary package version numbering increases in a
110linear fashion and that there is a number of version components that
111support that linear progression. For information on how to ensure
112package revisioning remains linear, see the
113":ref:`dev-manual/packages:automatically incrementing a package version number`"
114section.
115
116The following three sections provide related information on the PR
117Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
118how to ensure binary package revisioning remains linear.
119
120Working With a PR Service
121-------------------------
122
123As mentioned, attempting to maintain revision numbers in the
124:term:`Metadata` is error prone, inaccurate,
125and causes problems for people submitting recipes. Conversely, the PR
126Service automatically generates increasing numbers, particularly the
127revision field, which removes the human element.
128
129.. note::
130
131 For additional information on using a PR Service, you can see the
132 :yocto_wiki:`PR Service </PR_Service>` wiki page.
133
134The Yocto Project uses variables in order of decreasing priority to
135facilitate revision numbering (i.e.
136:term:`PE`,
137:term:`PV`, and
138:term:`PR` for epoch, version, and
139revision, respectively). The values are highly dependent on the policies
140and procedures of a given distribution and package feed.
141
142Because the OpenEmbedded build system uses
143":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
144unique to a given build, the build system knows when to rebuild
145packages. All the inputs into a given task are represented by a
146signature, which can trigger a rebuild when different. Thus, the build
147system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
148trigger a rebuild. The signatures, however, can be used to generate
149these values.
150
151The PR Service works with both ``OEBasic`` and ``OEBasicHash``
152generators. The value of :term:`PR` bumps when the checksum changes and the
153different generator mechanisms change signatures under different
154circumstances.
155
156As implemented, the build system includes values from the PR Service
157into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
158becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
159:term:`PR` values to be used for whatever reasons, which include manual
160:term:`PR` bumps, should it be necessary.
161
162By default, the PR Service is not enabled or running. Thus, the packages
163generated are just "self consistent". The build system adds and removes
164packages and there are no guarantees about upgrade paths but images will
165be consistent and correct with the latest changes.
166
167The simplest form for a PR Service is for a single host development system
168that builds the package feed (building system). For this scenario, you can
169enable a local PR Service by setting :term:`PRSERV_HOST` in your
170``local.conf`` file in the :term:`Build Directory`::
171
172 PRSERV_HOST = "localhost:0"
173
174Once the service is started, packages will automatically
175get increasing :term:`PR` values and BitBake takes care of starting and
176stopping the server.
177
178If you have a more complex setup where multiple host development systems
179work against a common, shared package feed, you have a single PR Service
180running and it is connected to each building system. For this scenario,
181you need to start the PR Service using the ``bitbake-prserv`` command::
182
183 bitbake-prserv --host ip --port port --start
184
185In addition to
186hand-starting the service, you need to update the ``local.conf`` file of
187each building system as described earlier so each system points to the
188server and port.
189
190It is also recommended you use build history, which adds some sanity
191checks to binary package versions, in conjunction with the server that
192is running the PR Service. To enable build history, add the following to
193each building system's ``local.conf`` file::
194
195 # It is recommended to activate "buildhistory" for testing the PR service
196 INHERIT += "buildhistory"
197 BUILDHISTORY_COMMIT = "1"
198
199For information on build
200history, see the
201":ref:`dev-manual/build-quality:maintaining build output quality`" section.
202
203.. note::
204
205 The OpenEmbedded build system does not maintain :term:`PR` information as
206 part of the shared state (sstate) packages. If you maintain an sstate
207 feed, it's expected that either all your building systems that
208 contribute to the sstate feed use a shared PR service, or you do not
209 run a PR service on any of your building systems.
210
211 That's because if you had multiple machines sharing a PR service but
212 not their sstate feed, you could end up with "diverging" hashes for
213 the same output artefacts. When presented to the share PR service,
214 each would be considered as new and would increase the revision
215 number, causing many unnecessary package upgrades.
216
217 For more information on shared state, see the
218 ":ref:`overview-manual/concepts:shared state cache`"
219 section in the Yocto Project Overview and Concepts Manual.
220
221Manually Bumping PR
222-------------------
223
224The alternative to setting up a PR Service is to manually "bump" the
225:term:`PR` variable.
226
227If a committed change results in changing the package output, then the
228value of the :term:`PR` variable needs to be increased (or "bumped") as part of
229that commit. For new recipes you should add the :term:`PR` variable and set
230its initial value equal to "r0", which is the default. Even though the
231default value is "r0", the practice of adding it to a new recipe makes
232it harder to forget to bump the variable when you make changes to the
233recipe in future.
234
235Usually, version increases occur only to binary packages. However, if
236for some reason :term:`PV` changes but does not increase, you can increase
237the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
238"0".
239
240Binary package version numbering strives to follow the `Debian Version
241Field Policy
242Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
243These guidelines define how versions are compared and what "increasing"
244a version means.
245
246Automatically Incrementing a Package Version Number
247---------------------------------------------------
248
249When fetching a repository, BitBake uses the
250:term:`SRCREV` variable to determine
251the specific source code revision from which to build. You set the
252:term:`SRCREV` variable to
253:term:`AUTOREV` to cause the
254OpenEmbedded build system to automatically use the latest revision of
255the software::
256
257 SRCREV = "${AUTOREV}"
258
259Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
260automatically update the version whenever the revision of the source
261code changes. Here is an example::
262
263 PV = "1.0+git${SRCPV}"
264
265The OpenEmbedded build system substitutes :term:`SRCPV` with the following:
266
267.. code-block:: none
268
269 AUTOINC+source_code_revision
270
271The build system replaces the ``AUTOINC``
272with a number. The number used depends on the state of the PR Service:
273
274- If PR Service is enabled, the build system increments the number,
275 which is similar to the behavior of
276 :term:`PR`. This behavior results in
277 linearly increasing package versions, which is desirable. Here is an
278 example:
279
280 .. code-block:: none
281
282 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
283 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
284
285- If PR Service is not enabled, the build system replaces the
286 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
287 changing the package version since the source revision is included.
288 However, package versions are not increased linearly. Here is an
289 example:
290
291 .. code-block:: none
292
293 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
294 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
295
296In summary, the OpenEmbedded build system does not track the history of
297binary package versions for this purpose. ``AUTOINC``, in this case, is
298comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
299package version is simply replaced by "0". If PR server is enabled, the
300build system keeps track of the package versions and bumps the number
301when the package revision changes.
302
303Handling Optional Module Packaging
304==================================
305
306Many pieces of software split functionality into optional modules (or
307plugins) and the plugins that are built might depend on configuration
308options. To avoid having to duplicate the logic that determines what
309modules are available in your recipe or to avoid having to package each
310module by hand, the OpenEmbedded build system provides functionality to
311handle module packaging dynamically.
312
313To handle optional module packaging, you need to do two things:
314
315- Ensure the module packaging is actually done.
316
317- Ensure that any dependencies on optional modules from other recipes
318 are satisfied by your recipe.
319
320Making Sure the Packaging is Done
321---------------------------------
322
323To ensure the module packaging actually gets done, you use the
324``do_split_packages`` function within the ``populate_packages`` Python
325function in your recipe. The ``do_split_packages`` function searches for
326a pattern of files or directories under a specified path and creates a
327package for each one it finds by appending to the
328:term:`PACKAGES` variable and
329setting the appropriate values for ``FILES:packagename``,
330``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
331Here is an example from the ``lighttpd`` recipe::
332
333 python populate_packages:prepend () {
334 lighttpd_libdir = d.expand('${libdir}')
335 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
336 'lighttpd-module-%s', 'Lighttpd module for %s',
337 extra_depends='')
338 }
339
340The previous example specifies a number of things in the call to
341``do_split_packages``.
342
343- A directory within the files installed by your recipe through
344 :ref:`ref-tasks-install` in which to search.
345
346- A regular expression used to match module files in that directory. In
347 the example, note the parentheses () that mark the part of the
348 expression from which the module name should be derived.
349
350- A pattern to use for the package names.
351
352- A description for each package.
353
354- An empty string for ``extra_depends``, which disables the default
355 dependency on the main ``lighttpd`` package. Thus, if a file in
356 ``${libdir}`` called ``mod_alias.so`` is found, a package called
357 ``lighttpd-module-alias`` is created for it and the
358 :term:`DESCRIPTION` is set to
359 "Lighttpd module for alias".
360
361Often, packaging modules is as simple as the previous example. However,
362there are more advanced options that you can use within
363``do_split_packages`` to modify its behavior. And, if you need to, you
364can add more logic by specifying a hook function that is called for each
365package. It is also perfectly acceptable to call ``do_split_packages``
366multiple times if you have more than one set of modules to package.
367
368For more examples that show how to use ``do_split_packages``, see the
369``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
370directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
371also find examples in ``meta/classes-recipe/kernel.bbclass``.
372
373Here is a reference that shows ``do_split_packages`` mandatory and
374optional arguments::
375
376 Mandatory arguments
377
378 root
379 The path in which to search
380 file_regex
381 Regular expression to match searched files.
382 Use parentheses () to mark the part of this
383 expression that should be used to derive the
384 module name (to be substituted where %s is
385 used in other function arguments as noted below)
386 output_pattern
387 Pattern to use for the package names. Must
388 include %s.
389 description
390 Description to set for each package. Must
391 include %s.
392
393 Optional arguments
394
395 postinst
396 Postinstall script to use for all packages
397 (as a string)
398 recursive
399 True to perform a recursive search --- default
400 False
401 hook
402 A hook function to be called for every match.
403 The function will be called with the following
404 arguments (in the order listed):
405
406 f
407 Full path to the file/directory match
408 pkg
409 The package name
410 file_regex
411 As above
412 output_pattern
413 As above
414 modulename
415 The module name derived using file_regex
416 extra_depends
417 Extra runtime dependencies (RDEPENDS) to be
418 set for all packages. The default value of None
419 causes a dependency on the main package
420 (${PN}) --- if you do not want this, pass empty
421 string '' for this parameter.
422 aux_files_pattern
423 Extra item(s) to be added to FILES for each
424 package. Can be a single string item or a list
425 of strings for multiple items. Must include %s.
426 postrm
427 postrm script to use for all packages (as a
428 string)
429 allow_dirs
430 True to allow directories to be matched -
431 default False
432 prepend
433 If True, prepend created packages to PACKAGES
434 instead of the default False which appends them
435 match_path
436 match file_regex on the whole relative path to
437 the root rather than just the filename
438 aux_files_pattern_verbatim
439 Extra item(s) to be added to FILES for each
440 package, using the actual derived module name
441 rather than converting it to something legal
442 for a package name. Can be a single string item
443 or a list of strings for multiple items. Must
444 include %s.
445 allow_links
446 True to allow symlinks to be matched --- default
447 False
448 summary
449 Summary to set for each package. Must include %s;
450 defaults to description if not set.
451
452
453
454Satisfying Dependencies
455-----------------------
456
457The second part for handling optional module packaging is to ensure that
458any dependencies on optional modules from other recipes are satisfied by
459your recipe. You can be sure these dependencies are satisfied by using
460the :term:`PACKAGES_DYNAMIC`
461variable. Here is an example that continues with the ``lighttpd`` recipe
462shown earlier::
463
464 PACKAGES_DYNAMIC = "lighttpd-module-.*"
465
466The name
467specified in the regular expression can of course be anything. In this
468example, it is ``lighttpd-module-`` and is specified as the prefix to
469ensure that any :term:`RDEPENDS` and
470:term:`RRECOMMENDS` on a package
471name starting with the prefix are satisfied during build time. If you
472are using ``do_split_packages`` as described in the previous section,
473the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
474pattern specified in the call to ``do_split_packages``.
475
476Using Runtime Package Management
477================================
478
479During a build, BitBake always transforms a recipe into one or more
480packages. For example, BitBake takes the ``bash`` recipe and produces a
481number of packages (e.g. ``bash``, ``bash-bashbug``,
482``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
483``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
484generated packages are included in an image.
485
486In several situations, you might need to update, add, remove, or query
487the packages on a target device at runtime (i.e. without having to
488generate a new image). Examples of such situations include:
489
490- You want to provide in-the-field updates to deployed devices (e.g.
491 security updates).
492
493- You want to have a fast turn-around development cycle for one or more
494 applications that run on your device.
495
496- You want to temporarily install the "debug" packages of various
497 applications on your device so that debugging can be greatly improved
498 by allowing access to symbols and source debugging.
499
500- You want to deploy a more minimal package selection of your device
501 but allow in-the-field updates to add a larger selection for
502 customization.
503
504In all these situations, you have something similar to a more
505traditional Linux distribution in that in-field devices are able to
506receive pre-compiled packages from a server for installation or update.
507Being able to install these packages on a running, in-field device is
508what is termed "runtime package management".
509
510In order to use runtime package management, you need a host or server
511machine that serves up the pre-compiled packages plus the required
512metadata. You also need package manipulation tools on the target. The
513build machine is a likely candidate to act as the server. However, that
514machine does not necessarily have to be the package server. The build
515machine could push its artifacts to another machine that acts as the
516server (e.g. Internet-facing). In fact, doing so is advantageous for a
517production environment as getting the packages away from the development
518system's :term:`Build Directory` prevents accidental overwrites.
519
520A simple build that targets just one device produces more than one
521package database. In other words, the packages produced by a build are
522separated out into a couple of different package groupings based on
523criteria such as the target's CPU architecture, the target board, or the
524C library used on the target. For example, a build targeting the
525``qemux86`` device produces the following three package databases:
526``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
527device to be aware of all the packages that were available to it, you
528would need to point it to each of these databases individually. In a
529similar way, a traditional Linux distribution usually is configured to
530be aware of a number of software repositories from which it retrieves
531packages.
532
533Using runtime package management is completely optional and not required
534for a successful build or deployment in any way. But if you want to make
535use of runtime package management, you need to do a couple things above
536and beyond the basics. The remainder of this section describes what you
537need to do.
538
539Build Considerations
540--------------------
541
542This section describes build considerations of which you need to be
543aware in order to provide support for runtime package management.
544
545When BitBake generates packages, it needs to know what format or formats
546to use. In your configuration, you use the
547:term:`PACKAGE_CLASSES`
548variable to specify the format:
549
550#. Open the ``local.conf`` file inside your :term:`Build Directory` (e.g.
551 ``poky/build/conf/local.conf``).
552
553#. Select the desired package format as follows::
554
555 PACKAGE_CLASSES ?= "package_packageformat"
556
557 where packageformat can be "ipk", "rpm",
558 "deb", or "tar" which are the supported package formats.
559
560 .. note::
561
562 Because the Yocto Project supports four different package formats,
563 you can set the variable with more than one argument. However, the
564 OpenEmbedded build system only uses the first argument when
565 creating an image or Software Development Kit (SDK).
566
567If you would like your image to start off with a basic package database
568containing the packages in your current build as well as to have the
569relevant tools available on the target for runtime package management,
570you can include "package-management" in the
571:term:`IMAGE_FEATURES`
572variable. Including "package-management" in this configuration variable
573ensures that when the image is assembled for your target, the image
574includes the currently-known package databases as well as the
575target-specific tools required for runtime package management to be
576performed on the target. However, this is not strictly necessary. You
577could start your image off without any databases but only include the
578required on-target package tool(s). As an example, you could include
579"opkg" in your
580:term:`IMAGE_INSTALL` variable
581if you are using the IPK package format. You can then initialize your
582target's package database(s) later once your image is up and running.
583
584Whenever you perform any sort of build step that can potentially
585generate a package or modify existing package, it is always a good idea
586to re-generate the package index after the build by using the following
587command::
588
589 $ bitbake package-index
590
591It might be tempting to build the
592package and the package index at the same time with a command such as
593the following::
594
595 $ bitbake some-package package-index
596
597Do not do this as
598BitBake does not schedule the package index for after the completion of
599the package you are building. Consequently, you cannot be sure of the
600package index including information for the package you just built.
601Thus, be sure to run the package update step separately after building
602any packages.
603
604You can use the
605:term:`PACKAGE_FEED_ARCHS`,
606:term:`PACKAGE_FEED_BASE_PATHS`,
607and
608:term:`PACKAGE_FEED_URIS`
609variables to pre-configure target images to use a package feed. If you
610do not define these variables, then manual steps as described in the
611subsequent sections are necessary to configure the target. You should
612set these variables before building the image in order to produce a
613correctly configured image.
614
615.. note::
616
617 Your image will need enough free storage space to run package upgrades,
618 especially if many of them need to be downloaded at the same time.
619 You should make sure images are created with enough free space
620 by setting the :term:`IMAGE_ROOTFS_EXTRA_SPACE` variable.
621
622When your build is complete, your packages reside in the
623``${TMPDIR}/deploy/packageformat`` directory. For example, if
624``${``\ :term:`TMPDIR`\ ``}`` is
625``tmp`` and your selected package type is RPM, then your RPM packages
626are available in ``tmp/deploy/rpm``.
627
628Host or Server Machine Setup
629----------------------------
630
631Although other protocols are possible, a server using HTTP typically
632serves packages. If you want to use HTTP, then set up and configure a
633web server such as Apache 2, lighttpd, or Python web server on the
634machine serving the packages.
635
636To keep things simple, this section describes how to set up a
637Python web server to share package feeds from the developer's
638machine. Although this server might not be the best for a production
639environment, the setup is simple and straight forward. Should you want
640to use a different server more suited for production (e.g. Apache 2,
641Lighttpd, or Nginx), take the appropriate steps to do so.
642
643From within the :term:`Build Directory` where you have built an image based on
644your packaging choice (i.e. the :term:`PACKAGE_CLASSES` setting), simply start
645the server. The following example assumes a :term:`Build Directory` of ``poky/build``
646and a :term:`PACKAGE_CLASSES` setting of ":ref:`ref-classes-package_rpm`"::
647
648 $ cd poky/build/tmp/deploy/rpm
649 $ python3 -m http.server
650
651Target Setup
652------------
653
654Setting up the target differs depending on the package management
655system. This section provides information for RPM, IPK, and DEB.
656
657Using RPM
658~~~~~~~~~
659
660The :wikipedia:`Dandified Packaging <DNF_(software)>` (DNF) performs
661runtime package management of RPM packages. In order to use DNF for
662runtime package management, you must perform an initial setup on the
663target machine for cases where the ``PACKAGE_FEED_*`` variables were not
664set as part of the image that is running on the target. This means if
665you built your image and did not use these variables as part of the
666build and your image is now running on the target, you need to perform
667the steps in this section if you want to use runtime package management.
668
669.. note::
670
671 For information on the ``PACKAGE_FEED_*`` variables, see
672 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
673 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
674 glossary.
675
676On the target, you must inform DNF that package databases are available.
677You do this by creating a file named
678``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
679
680As an example, assume the target is able to use the following package
681databases: ``all``, ``i586``, and ``qemux86`` from a server named
682``my.server``. The specifics for setting up the web server are up to
683you. The critical requirement is that the URIs in the target repository
684configuration point to the correct remote location for the feeds.
685
686.. note::
687
688 For development purposes, you can point the web server to the build
689 system's ``deploy`` directory. However, for production use, it is better to
690 copy the package directories to a location outside of the build area and use
691 that location. Doing so avoids situations where the build system
692 overwrites or changes the ``deploy`` directory.
693
694When telling DNF where to look for the package databases, you must
695declare individual locations per architecture or a single location used
696for all architectures. You cannot do both:
697
698- *Create an Explicit List of Architectures:* Define individual base
699 URLs to identify where each package database is located:
700
701 .. code-block:: none
702
703 [oe-packages]
704 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
705
706 This example
707 informs DNF about individual package databases for all three
708 architectures.
709
710- *Create a Single (Full) Package Index:* Define a single base URL that
711 identifies where a full package database is located::
712
713 [oe-packages]
714 baseurl=http://my.server/rpm
715
716 This example informs DNF about a single
717 package database that contains all the package index information for
718 all supported architectures.
719
720Once you have informed DNF where to find the package databases, you need
721to fetch them:
722
723.. code-block:: none
724
725 # dnf makecache
726
727DNF is now able to find, install, and
728upgrade packages from the specified repository or repositories.
729
730.. note::
731
732 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
733 additional information.
734
735Using IPK
736~~~~~~~~~
737
738The ``opkg`` application performs runtime package management of IPK
739packages. You must perform an initial setup for ``opkg`` on the target
740machine if the
741:term:`PACKAGE_FEED_ARCHS`,
742:term:`PACKAGE_FEED_BASE_PATHS`,
743and
744:term:`PACKAGE_FEED_URIS`
745variables have not been set or the target image was built before the
746variables were set.
747
748The ``opkg`` application uses configuration files to find available
749package databases. Thus, you need to create a configuration file inside
750the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
751you want to use.
752
753As an example, suppose you are serving packages from a ``ipk/``
754directory containing the ``i586``, ``all``, and ``qemux86`` databases
755through an HTTP server named ``my.server``. On the target, create a
756configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
757directory containing the following:
758
759.. code-block:: none
760
761 src/gz all http://my.server/ipk/all
762 src/gz i586 http://my.server/ipk/i586
763 src/gz qemux86 http://my.server/ipk/qemux86
764
765Next, instruct ``opkg`` to fetch the
766repository information:
767
768.. code-block:: none
769
770 # opkg update
771
772The ``opkg`` application is now able to find, install, and upgrade packages
773from the specified repository.
774
775Using DEB
776~~~~~~~~~
777
778The ``apt`` application performs runtime package management of DEB
779packages. This application uses a source list file to find available
780package databases. You must perform an initial setup for ``apt`` on the
781target machine if the
782:term:`PACKAGE_FEED_ARCHS`,
783:term:`PACKAGE_FEED_BASE_PATHS`,
784and
785:term:`PACKAGE_FEED_URIS`
786variables have not been set or the target image was built before the
787variables were set.
788
789To inform ``apt`` of the repository you want to use, you might create a
790list file (e.g. ``my_repo.list``) inside the
791``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
792serving packages from a ``deb/`` directory containing the ``i586``,
793``all``, and ``qemux86`` databases through an HTTP server named
794``my.server``. The list file should contain:
795
796.. code-block:: none
797
798 deb http://my.server/deb/all ./
799 deb http://my.server/deb/i586 ./
800 deb http://my.server/deb/qemux86 ./
801
802Next, instruct the ``apt`` application
803to fetch the repository information:
804
805.. code-block:: none
806
807 $ sudo apt update
808
809After this step,
810``apt`` is able to find, install, and upgrade packages from the
811specified repository.
812
813Generating and Using Signed Packages
814====================================
815
816In order to add security to RPM packages used during a build, you can
817take steps to securely sign them. Once a signature is verified, the
818OpenEmbedded build system can use the package in the build. If security
819fails for a signed package, the build system stops the build.
820
821This section describes how to sign RPM packages during a build and how
822to use signed package feeds (repositories) when doing a build.
823
824Signing RPM Packages
825--------------------
826
827To enable signing RPM packages, you must set up the following
828configurations in either your ``local.config`` or ``distro.config``
829file::
830
831 # Inherit sign_rpm.bbclass to enable signing functionality
832 INHERIT += " sign_rpm"
833 # Define the GPG key that will be used for signing.
834 RPM_GPG_NAME = "key_name"
835 # Provide passphrase for the key
836 RPM_GPG_PASSPHRASE = "passphrase"
837
838.. note::
839
840 Be sure to supply appropriate values for both `key_name` and
841 `passphrase`.
842
843Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
844the previous example, two optional variables related to signing are available:
845
846- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
847 when the package is signed.
848
849- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
850 package is signed.
851
852Processing Package Feeds
853------------------------
854
855In addition to being able to sign RPM packages, you can also enable
856signed package feeds for IPK and RPM packages.
857
858The steps you need to take to enable signed package feed use are similar
859to the steps used to sign RPM packages. You must define the following in
860your ``local.config`` or ``distro.config`` file::
861
862 INHERIT += "sign_package_feed"
863 PACKAGE_FEED_GPG_NAME = "key_name"
864 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
865
866For signed package feeds, the passphrase must be specified in a separate file,
867which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
868variable. Regarding security, keeping a plain text passphrase out of the
869configuration is more secure.
870
871Aside from the ``PACKAGE_FEED_GPG_NAME`` and
872``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
873related to signed package feeds are available:
874
875- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
876 when the package is signed.
877
878- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
879 package is signed.
880
881- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
882 signature. This variable applies only to RPM and IPK package feeds.
883 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
884 "ASC", which is the default and specifies ascii armored, and "BIN",
885 which specifies binary.
886
887Testing Packages With ptest
888===========================
889
890A Package Test (ptest) runs tests against packages built by the
891OpenEmbedded build system on the target machine. A ptest contains at
892least two items: the actual test, and a shell script (``run-ptest``)
893that starts the test. The shell script that starts the test must not
894contain the actual test --- the script only starts the test. On the other
895hand, the test can be anything from a simple shell script that runs a
896binary and checks the output to an elaborate system of test binaries and
897data files.
898
899The test generates output in the format used by Automake::
900
901 result: testname
902
903where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
904the testname can be any identifying string.
905
906For a list of Yocto Project recipes that are already enabled with ptest,
907see the :yocto_wiki:`Ptest </Ptest>` wiki page.
908
909.. note::
910
911 A recipe is "ptest-enabled" if it inherits the :ref:`ref-classes-ptest`
912 class.
913
914Adding ptest to Your Build
915--------------------------
916
917To add package testing to your build, add the :term:`DISTRO_FEATURES` and
918:term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which
919is found in the :term:`Build Directory`::
920
921 DISTRO_FEATURES:append = " ptest"
922 EXTRA_IMAGE_FEATURES += "ptest-pkgs"
923
924Once your build is complete, the ptest files are installed into the
925``/usr/lib/package/ptest`` directory within the image, where ``package``
926is the name of the package.
927
928Running ptest
929-------------
930
931The ``ptest-runner`` package installs a shell script that loops through
932all installed ptest test suites and runs them in sequence. Consequently,
933you might want to add this package to your image.
934
935Getting Your Package Ready
936--------------------------
937
938In order to enable a recipe to run installed ptests on target hardware,
939you need to prepare the recipes that build the packages you want to
940test. Here is what you have to do for each recipe:
941
942- *Be sure the recipe inherits the* :ref:`ref-classes-ptest` *class:*
943 Include the following line in each recipe::
944
945 inherit ptest
946
947- *Create run-ptest:* This script starts your test. Locate the
948 script where you will refer to it using
949 :term:`SRC_URI`. Here is an
950 example that starts a test for ``dbus``::
951
952 #!/bin/sh
953 cd test
954 make -k runtest-TESTS
955
956- *Ensure dependencies are met:* If the test adds build or runtime
957 dependencies that normally do not exist for the package (such as
958 requiring "make" to run the test suite), use the
959 :term:`DEPENDS` and
960 :term:`RDEPENDS` variables in
961 your recipe in order for the package to meet the dependencies. Here
962 is an example where the package has a runtime dependency on "make"::
963
964 RDEPENDS:${PN}-ptest += "make"
965
966- *Add a function to build the test suite:* Not many packages support
967 cross-compilation of their test suites. Consequently, you usually
968 need to add a cross-compilation function to the package.
969
970 Many packages based on Automake compile and run the test suite by
971 using a single command such as ``make check``. However, the host
972 ``make check`` builds and runs on the same computer, while
973 cross-compiling requires that the package is built on the host but
974 executed for the target architecture (though often, as in the case
975 for ptest, the execution occurs on the host). The built version of
976 Automake that ships with the Yocto Project includes a patch that
977 separates building and execution. Consequently, packages that use the
978 unaltered, patched version of ``make check`` automatically
979 cross-compiles.
980
981 Regardless, you still must add a ``do_compile_ptest`` function to
982 build the test suite. Add a function similar to the following to your
983 recipe::
984
985 do_compile_ptest() {
986 oe_runmake buildtest-TESTS
987 }
988
989- *Ensure special configurations are set:* If the package requires
990 special configurations prior to compiling the test code, you must
991 insert a ``do_configure_ptest`` function into the recipe.
992
993- *Install the test suite:* The :ref:`ref-classes-ptest` class
994 automatically copies the file ``run-ptest`` to the target and then runs make
995 ``install-ptest`` to run the tests. If this is not enough, you need
996 to create a ``do_install_ptest`` function and make sure it gets
997 called after the "make install-ptest" completes.
998
999Creating Node Package Manager (NPM) Packages
1000============================================
1001
1002:wikipedia:`NPM <Npm_(software)>` is a package manager for the JavaScript
1003programming language. The Yocto Project supports the NPM
1004:ref:`fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`.
1005You can use this fetcher in combination with
1006:doc:`devtool </ref-manual/devtool-reference>` to create recipes that produce
1007NPM packages.
1008
1009There are two workflows that allow you to create NPM packages using
1010``devtool``: the NPM registry modules method and the NPM project code
1011method.
1012
1013.. note::
1014
1015 While it is possible to create NPM recipes manually, using
1016 ``devtool`` is far simpler.
1017
1018Additionally, some requirements and caveats exist.
1019
1020Requirements and Caveats
1021------------------------
1022
1023You need to be aware of the following before using ``devtool`` to create
1024NPM packages:
1025
1026- Of the two methods that you can use ``devtool`` to create NPM
1027 packages, the registry approach is slightly simpler. However, you
1028 might consider the project approach because you do not have to
1029 publish your module in the `NPM registry <https://docs.npmjs.com/misc/registry>`__,
1030 which is NPM's public registry.
1031
1032- Be familiar with
1033 :doc:`devtool </ref-manual/devtool-reference>`.
1034
1035- The NPM host tools need the native ``nodejs-npm`` package, which is
1036 part of the OpenEmbedded environment. You need to get the package by
1037 cloning the :oe_git:`meta-openembedded </meta-openembedded>`
1038 repository. Be sure to add the path to your local copy
1039 to your ``bblayers.conf`` file.
1040
1041- ``devtool`` cannot detect native libraries in module dependencies.
1042 Consequently, you must manually add packages to your recipe.
1043
1044- While deploying NPM packages, ``devtool`` cannot determine which
1045 dependent packages are missing on the target (e.g. the node runtime
1046 ``nodejs``). Consequently, you need to find out what files are
1047 missing and be sure they are on the target.
1048
1049- Although you might not need NPM to run your node package, it is
1050 useful to have NPM on your target. The NPM package name is
1051 ``nodejs-npm``.
1052
1053Using the Registry Modules Method
1054---------------------------------
1055
1056This section presents an example that uses the ``cute-files`` module,
1057which is a file browser web application.
1058
1059.. note::
1060
1061 You must know the ``cute-files`` module version.
1062
1063The first thing you need to do is use ``devtool`` and the NPM fetcher to
1064create the recipe::
1065
1066 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
1067
1068The
1069``devtool add`` command runs ``recipetool create`` and uses the same
1070fetch URI to download each dependency and capture license details where
1071possible. The result is a generated recipe.
1072
1073After running for quite a long time, in particular building the
1074``nodejs-native`` package, the command should end as follows::
1075
1076 INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional
1077
1078The recipe file is fairly simple and contains every license that
1079``recipetool`` finds and includes the licenses in the recipe's
1080:term:`LIC_FILES_CHKSUM`
1081variables. You need to examine the variables and look for those with
1082"unknown" in the :term:`LICENSE`
1083field. You need to track down the license information for "unknown"
1084modules and manually add the information to the recipe.
1085
1086``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
1087files capture the version of all dependent modules. Many packages do not
1088provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it
1089runs.
1090
1091.. note::
1092
1093 A package is created for each sub-module. This policy is the only
1094 practical way to have the licenses for all of the dependencies
1095 represented in the license manifest of the image.
1096
1097The ``devtool edit-recipe`` command lets you take a look at the recipe::
1098
1099 $ devtool edit-recipe cute-files
1100 # Recipe created by recipetool
1101 # This is the basis of a recipe and may need further editing in order to be fully functional.
1102 # (Feel free to remove these comments when editing.)
1103
1104 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
1105 # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
1106 # your responsibility to verify that the values are complete and correct.
1107 #
1108 # NOTE: multiple licenses have been detected; they have been separated with &
1109 # in the LICENSE value for now since it is a reasonable assumption that all
1110 # of the licenses apply. If instead there is a choice between the multiple
1111 # licenses then you should change the value to separate the licenses with |
1112 # instead of &. If there is any doubt, check the accompanying documentation
1113 # to determine which situation is applicable.
1114
1115 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
1116 LICENSE = "BSD-3-Clause & ISC & MIT"
1117 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
1118 file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \
1119 file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \
1120 ...
1121 file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a"
1122
1123 SRC_URI = " \
1124 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
1125 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
1126 "
1127
1128 S = "${WORKDIR}/npm"
1129
1130 inherit npm
1131
1132 LICENSE:${PN} = "MIT"
1133 LICENSE:${PN}-accepts = "MIT"
1134 LICENSE:${PN}-array-flatten = "MIT"
1135 ...
1136 LICENSE:${PN}-vary = "MIT"
1137
1138Three key points in the previous example are:
1139
1140- :term:`SRC_URI` uses the NPM
1141 scheme so that the NPM fetcher is used.
1142
1143- ``recipetool`` collects all the license information. If a
1144 sub-module's license is unavailable, the sub-module's name appears in
1145 the comments.
1146
1147- The ``inherit npm`` statement causes the :ref:`ref-classes-npm` class to
1148 package up all the modules.
1149
1150You can run the following command to build the ``cute-files`` package::
1151
1152 $ devtool build cute-files
1153
1154Remember that ``nodejs`` must be installed on
1155the target before your package.
1156
1157Assuming 192.168.7.2 for the target's IP address, use the following
1158command to deploy your package::
1159
1160 $ devtool deploy-target -s cute-files root@192.168.7.2
1161
1162Once the package is installed on the target, you can
1163test the application to show the contents of any directory::
1164
1165 $ cd /usr/lib/node_modules/cute-files
1166 $ cute-files
1167
1168On a browser,
1169go to ``http://192.168.7.2:3000`` and you see the following:
1170
1171.. image:: figures/cute-files-npm-example.png
1172 :width: 100%
1173
1174You can find the recipe in ``workspace/recipes/cute-files``. You can use
1175the recipe in any layer you choose.
1176
1177Using the NPM Projects Code Method
1178----------------------------------
1179
1180Although it is useful to package modules already in the NPM registry,
1181adding ``node.js`` projects under development is a more common developer
1182use case.
1183
1184This section covers the NPM projects code method, which is very similar
1185to the "registry" approach described in the previous section. In the NPM
1186projects method, you provide ``devtool`` with an URL that points to the
1187source files.
1188
1189Replicating the same example, (i.e. ``cute-files``) use the following
1190command::
1191
1192 $ devtool add https://github.com/martinaglv/cute-files.git
1193
1194The recipe this command generates is very similar to the recipe created in
1195the previous section. However, the :term:`SRC_URI` looks like the following::
1196
1197 SRC_URI = " \
1198 git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \
1199 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
1200 "
1201
1202In this example,
1203the main module is taken from the Git repository and dependencies are
1204taken from the NPM registry. Other than those differences, the recipe is
1205basically the same between the two methods. You can build and deploy the
1206package exactly as described in the previous section that uses the
1207registry modules method.
1208
1209Adding custom metadata to packages
1210==================================
1211
1212The variable
1213:term:`PACKAGE_ADD_METADATA`
1214can be used to add additional metadata to packages. This is reflected in
1215the package control/spec file. To take the ipk format for example, the
1216CONTROL file stored inside would contain the additional metadata as
1217additional lines.
1218
1219The variable can be used in multiple ways, including using suffixes to
1220set it for a specific package type and/or package. Note that the order
1221of precedence is the same as this list:
1222
1223- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
1224
1225- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
1226
1227- ``PACKAGE_ADD_METADATA:<PN>``
1228
1229- :term:`PACKAGE_ADD_METADATA`
1230
1231`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
1232package type:
1233
1234- IPK for .ipk packages
1235
1236- DEB for .deb packages
1237
1238- RPM for .rpm packages
1239
1240`<PN>` is a parameter and expected to be a package name.
1241
1242The variable can contain multiple [one-line] metadata fields separated
1243by the literal sequence '\\n'. The separator can be redefined using the
1244variable flag ``separator``.
1245
1246Here is an example that adds two custom fields for ipk
1247packages::
1248
1249 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
1250
diff --git a/documentation/dev-manual/prebuilt-libraries.rst b/documentation/dev-manual/prebuilt-libraries.rst
new file mode 100644
index 0000000000..a05f39ca1e
--- /dev/null
+++ b/documentation/dev-manual/prebuilt-libraries.rst
@@ -0,0 +1,209 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working with Pre-Built Libraries
4********************************
5
6Introduction
7============
8
9Some library vendors do not release source code for their software but do
10release pre-built binaries. When shared libraries are built, they should
11be versioned (see `this article
12<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__
13for some background), but sometimes this is not done.
14
15To summarize, a versioned library must meet two conditions:
16
17#. The filename must have the version appended, for example: ``libfoo.so.1.2.3``.
18#. The library must have the ELF tag ``SONAME`` set to the major version
19 of the library, for example: ``libfoo.so.1``. You can check this by
20 running ``readelf -d filename | grep SONAME``.
21
22This section shows how to deal with both versioned and unversioned
23pre-built libraries.
24
25Versioned Libraries
26===================
27
28In this example we work with pre-built libraries for the FT4222H USB I/O chip.
29Libraries are built for several target architecture variants and packaged in
30an archive as follows::
31
32 ├── build-arm-hisiv300
33 │   └── libft4222.so.1.4.4.44
34 ├── build-arm-v5-sf
35 │   └── libft4222.so.1.4.4.44
36 ├── build-arm-v6-hf
37 │   └── libft4222.so.1.4.4.44
38 ├── build-arm-v7-hf
39 │   └── libft4222.so.1.4.4.44
40 ├── build-arm-v8
41 │   └── libft4222.so.1.4.4.44
42 ├── build-i386
43 │   └── libft4222.so.1.4.4.44
44 ├── build-i486
45 │   └── libft4222.so.1.4.4.44
46 ├── build-mips-eglibc-hf
47 │   └── libft4222.so.1.4.4.44
48 ├── build-pentium
49 │   └── libft4222.so.1.4.4.44
50 ├── build-x86_64
51 │   └── libft4222.so.1.4.4.44
52 ├── examples
53 │   ├── get-version.c
54 │   ├── i2cm.c
55 │   ├── spim.c
56 │   └── spis.c
57 ├── ftd2xx.h
58 ├── install4222.sh
59 ├── libft4222.h
60 ├── ReadMe.txt
61 └── WinTypes.h
62
63To write a recipe to use such a library in your system:
64
65- The vendor will probably have a proprietary licence, so set
66 :term:`LICENSE_FLAGS` in your recipe.
67- The vendor provides a tarball containing libraries so set :term:`SRC_URI`
68 appropriately.
69- Set :term:`COMPATIBLE_HOST` so that the recipe cannot be used with an
70 unsupported architecture. In the following example, we only support the 32
71 and 64 bit variants of the ``x86`` architecture.
72- As the vendor provides versioned libraries, we can use ``oe_soinstall``
73 from :ref:`ref-classes-utils` to install the shared library and create
74 symbolic links. If the vendor does not do this, we need to follow the
75 non-versioned library guidelines in the next section.
76- As the vendor likely used :term:`LDFLAGS` different from those in your Yocto
77 Project build, disable the corresponding checks by adding ``ldflags``
78 to :term:`INSANE_SKIP`.
79- The vendor will typically ship release builds without debugging symbols.
80 Avoid errors by preventing the packaging task from stripping out the symbols
81 and adding them to a separate debug package. This is done by setting the
82 ``INHIBIT_`` flags shown below.
83
84The complete recipe would look like this::
85
86 SUMMARY = "FTDI FT4222H Library"
87 SECTION = "libs"
88 LICENSE_FLAGS = "ftdi"
89 LICENSE = "CLOSED"
90
91 COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
92
93 # Sources available in a .tgz file in .zip archive
94 # at https://ftdichip.com/wp-content/uploads/2021/01/libft4222-linux-1.4.4.44.zip
95 # Found on https://ftdichip.com/software-examples/ft4222h-software-examples/
96 # Since dealing with this particular type of archive is out of topic here,
97 # we use a local link.
98 SRC_URI = "file://libft4222-linux-${PV}.tgz"
99
100 S = "${WORKDIR}"
101
102 ARCH_DIR:x86-64 = "build-x86_64"
103 ARCH_DIR:i586 = "build-i386"
104 ARCH_DIR:i686 = "build-i386"
105
106 INSANE_SKIP:${PN} = "ldflags"
107 INHIBIT_PACKAGE_STRIP = "1"
108 INHIBIT_SYSROOT_STRIP = "1"
109 INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
110
111 do_install () {
112 install -m 0755 -d ${D}${libdir}
113 oe_soinstall ${S}/${ARCH_DIR}/libft4222.so.${PV} ${D}${libdir}
114 install -d ${D}${includedir}
115 install -m 0755 ${S}/*.h ${D}${includedir}
116 }
117
118If the precompiled binaries are not statically linked and have dependencies on
119other libraries, then by adding those libraries to :term:`DEPENDS`, the linking
120can be examined and the appropriate :term:`RDEPENDS` automatically added.
121
122Non-Versioned Libraries
123=======================
124
125Some Background
126---------------
127
128Libraries in Linux systems are generally versioned so that it is possible
129to have multiple versions of the same library installed, which eases upgrades
130and support for older software. For example, suppose that in a versioned
131library, an actual library is called ``libfoo.so.1.2``, a symbolic link named
132``libfoo.so.1`` points to ``libfoo.so.1.2``, and a symbolic link named
133``libfoo.so`` points to ``libfoo.so.1.2``. Given these conditions, when you
134link a binary against a library, you typically provide the unversioned file
135name (i.e. ``-lfoo`` to the linker). However, the linker follows the symbolic
136link and actually links against the versioned filename. The unversioned symbolic
137link is only used at development time. Consequently, the library is packaged
138along with the headers in the development package ``${PN}-dev`` along with the
139actual library and versioned symbolic links in ``${PN}``. Because versioned
140libraries are far more common than unversioned libraries, the default packaging
141rules assume versioned libraries.
142
143Yocto Library Packaging Overview
144--------------------------------
145
146It follows that packaging an unversioned library requires a bit of work in the
147recipe. By default, ``libfoo.so`` gets packaged into ``${PN}-dev``, which
148triggers a QA warning that a non-symlink library is in a ``-dev`` package,
149and binaries in the same recipe link to the library in ``${PN}-dev``,
150which triggers more QA warnings. To solve this problem, you need to package the
151unversioned library into ``${PN}`` where it belongs. The abridged
152default :term:`FILES` variables in ``bitbake.conf`` are::
153
154 SOLIBS = ".so.*"
155 SOLIBSDEV = ".so"
156 FILES:${PN} = "... ${libdir}/lib*${SOLIBS} ..."
157 FILES_SOLIBSDEV ?= "... ${libdir}/lib*${SOLIBSDEV} ..."
158 FILES:${PN}-dev = "... ${FILES_SOLIBSDEV} ..."
159
160:term:`SOLIBS` defines a pattern that matches real shared object libraries.
161:term:`SOLIBSDEV` matches the development form (unversioned symlink). These two
162variables are then used in ``FILES:${PN}`` and ``FILES:${PN}-dev``, which puts
163the real libraries into ``${PN}`` and the unversioned symbolic link into ``${PN}-dev``.
164To package unversioned libraries, you need to modify the variables in the recipe
165as follows::
166
167 SOLIBS = ".so"
168 FILES_SOLIBSDEV = ""
169
170The modifications cause the ``.so`` file to be the real library
171and unset :term:`FILES_SOLIBSDEV` so that no libraries get packaged into
172``${PN}-dev``. The changes are required because unless :term:`PACKAGES` is changed,
173``${PN}-dev`` collects files before `${PN}`. ``${PN}-dev`` must not collect any of
174the files you want in ``${PN}``.
175
176Finally, loadable modules, essentially unversioned libraries that are linked
177at runtime using ``dlopen()`` instead of at build time, should generally be
178installed in a private directory. However, if they are installed in ``${libdir}``,
179then the modules can be treated as unversioned libraries.
180
181Example
182-------
183
184The example below installs an unversioned x86-64 pre-built library named
185``libfoo.so``. The :term:`COMPATIBLE_HOST` variable limits recipes to the
186x86-64 architecture while the :term:`INSANE_SKIP`, :term:`INHIBIT_PACKAGE_STRIP`
187and :term:`INHIBIT_SYSROOT_STRIP` variables are all set as in the above
188versioned library example. The "magic" is setting the :term:`SOLIBS` and
189:term:`FILES_SOLIBSDEV` variables as explained above::
190
191 SUMMARY = "libfoo sample recipe"
192 SECTION = "libs"
193 LICENSE = "CLOSED"
194
195 SRC_URI = "file://libfoo.so"
196
197 COMPATIBLE_HOST = "x86_64.*-linux"
198
199 INSANE_SKIP:${PN} = "ldflags"
200 INHIBIT_PACKAGE_STRIP = "1"
201 INHIBIT_SYSROOT_STRIP = "1"
202 SOLIBS = ".so"
203 FILES_SOLIBSDEV = ""
204
205 do_install () {
206 install -d ${D}${libdir}
207 install -m 0755 ${WORKDIR}/libfoo.so ${D}${libdir}
208 }
209
diff --git a/documentation/dev-manual/python-development-shell.rst b/documentation/dev-manual/python-development-shell.rst
new file mode 100644
index 0000000000..81a5c43472
--- /dev/null
+++ b/documentation/dev-manual/python-development-shell.rst
@@ -0,0 +1,50 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using a Python Development Shell
4********************************
5
6Similar to working within a development shell as described in the
7previous section, you can also spawn and work within an interactive
8Python development shell. When debugging certain commands or even when
9just editing packages, ``pydevshell`` can be a useful tool. When you
10invoke the ``pydevshell`` task, all tasks up to and including
11:ref:`ref-tasks-patch` are run for the
12specified target. Then a new terminal is opened. Additionally, key
13Python objects and code are available in the same way they are to
14BitBake tasks, in particular, the data store 'd'. So, commands such as
15the following are useful when exploring the data store and running
16functions::
17
18 pydevshell> d.getVar("STAGING_DIR")
19 '/media/build1/poky/build/tmp/sysroots'
20 pydevshell> d.getVar("STAGING_DIR", False)
21 '${TMPDIR}/sysroots'
22 pydevshell> d.setVar("FOO", "bar")
23 pydevshell> d.getVar("FOO")
24 'bar'
25 pydevshell> d.delVar("FOO")
26 pydevshell> d.getVar("FOO")
27 pydevshell> bb.build.exec_func("do_unpack", d)
28 pydevshell>
29
30See the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:functions you can call from within python`"
31section in the BitBake User Manual for details about available functions.
32
33The commands execute just as if the OpenEmbedded build
34system were executing them. Consequently, working this way can be
35helpful when debugging a build or preparing software to be used with the
36OpenEmbedded build system.
37
38Here is an example that uses ``pydevshell`` on a target named
39``matchbox-desktop``::
40
41 $ bitbake matchbox-desktop -c pydevshell
42
43This command spawns a terminal and places you in an interactive Python
44interpreter within the OpenEmbedded build environment. The
45:term:`OE_TERMINAL` variable
46controls what type of shell is opened.
47
48When you are finished using ``pydevshell``, you can exit the shell
49either by using Ctrl+d or closing the terminal window.
50
diff --git a/documentation/dev-manual/qemu.rst b/documentation/dev-manual/qemu.rst
index 766691b97b..19f3e40d63 100644
--- a/documentation/dev-manual/qemu.rst
+++ b/documentation/dev-manual/qemu.rst
@@ -44,31 +44,29 @@ To use QEMU, you need to have QEMU installed and initialized as well as
44have the proper artifacts (i.e. image files and root filesystems) 44have the proper artifacts (i.e. image files and root filesystems)
45available. Follow these general steps to run QEMU: 45available. Follow these general steps to run QEMU:
46 46
471. *Install QEMU:* QEMU is made available with the Yocto Project a 47#. *Install QEMU:* QEMU is made available with the Yocto Project a
48 number of ways. One method is to install a Software Development Kit 48 number of ways. One method is to install a Software Development Kit
49 (SDK). See ":ref:`sdk-manual/intro:the qemu emulator`" section in the 49 (SDK). See ":ref:`sdk-manual/intro:the qemu emulator`" section in the
50 Yocto Project Application Development and the Extensible Software 50 Yocto Project Application Development and the Extensible Software
51 Development Kit (eSDK) manual for information on how to install QEMU. 51 Development Kit (eSDK) manual for information on how to install QEMU.
52 52
532. *Setting Up the Environment:* How you set up the QEMU environment 53#. *Setting Up the Environment:* How you set up the QEMU environment
54 depends on how you installed QEMU: 54 depends on how you installed QEMU:
55 55
56 - If you cloned the ``poky`` repository or you downloaded and 56 - If you cloned the ``poky`` repository or you downloaded and
57 unpacked a Yocto Project release tarball, you can source the build 57 unpacked a Yocto Project release tarball, you can source the build
58 environment script (i.e. :ref:`structure-core-script`): 58 environment script (i.e. :ref:`structure-core-script`)::
59 ::
60 59
61 $ cd ~/poky 60 $ cd poky
62 $ source oe-init-build-env 61 $ source oe-init-build-env
63 62
64 - If you installed a cross-toolchain, you can run the script that 63 - If you installed a cross-toolchain, you can run the script that
65 initializes the toolchain. For example, the following commands run 64 initializes the toolchain. For example, the following commands run
66 the initialization script from the default ``poky_sdk`` directory: 65 the initialization script from the default ``poky_sdk`` directory::
67 ::
68 66
69 . ~/poky_sdk/environment-setup-core2-64-poky-linux 67 . poky_sdk/environment-setup-core2-64-poky-linux
70 68
713. *Ensure the Artifacts are in Place:* You need to be sure you have a 69#. *Ensure the Artifacts are in Place:* You need to be sure you have a
72 pre-built kernel that will boot in QEMU. You also need the target 70 pre-built kernel that will boot in QEMU. You also need the target
73 root filesystem for your target machine's architecture: 71 root filesystem for your target machine's architecture:
74 72
@@ -86,8 +84,7 @@ available. Follow these general steps to run QEMU:
86 Extensible Software Development Kit (eSDK) manual for information on 84 Extensible Software Development Kit (eSDK) manual for information on
87 how to extract a root filesystem. 85 how to extract a root filesystem.
88 86
894. *Run QEMU:* The basic ``runqemu`` command syntax is as follows: 87#. *Run QEMU:* The basic ``runqemu`` command syntax is as follows::
90 ::
91 88
92 $ runqemu [option ] [...] 89 $ runqemu [option ] [...]
93 90
@@ -102,12 +99,13 @@ available. Follow these general steps to run QEMU:
102 Here are some additional examples to help illustrate further QEMU: 99 Here are some additional examples to help illustrate further QEMU:
103 100
104 - This example starts QEMU with MACHINE set to "qemux86-64". 101 - This example starts QEMU with MACHINE set to "qemux86-64".
105 Assuming a standard 102 Assuming a standard :term:`Build Directory`, ``runqemu``
106 :term:`Build Directory`, ``runqemu``
107 automatically finds the ``bzImage-qemux86-64.bin`` image file and 103 automatically finds the ``bzImage-qemux86-64.bin`` image file and
108 the ``core-image-minimal-qemux86-64-20200218002850.rootfs.ext4`` 104 the ``core-image-minimal-qemux86-64-20200218002850.rootfs.ext4``
109 (assuming the current build created a ``core-image-minimal`` 105 (assuming the current build created a ``core-image-minimal``
110 image). 106 image)::
107
108 $ runqemu qemux86-64
111 109
112 .. note:: 110 .. note::
113 111
@@ -115,38 +113,31 @@ available. Follow these general steps to run QEMU:
115 and uses the most recently built image according to the 113 and uses the most recently built image according to the
116 timestamp. 114 timestamp.
117 115
118 ::
119
120 $ runqemu qemux86-64
121
122 - This example produces the exact same results as the previous 116 - This example produces the exact same results as the previous
123 example. This command, however, specifically provides the image 117 example. This command, however, specifically provides the image
124 and root filesystem type. 118 and root filesystem type::
125 ::
126 119
127 $ runqemu qemux86-64 core-image-minimal ext4 120 $ runqemu qemux86-64 core-image-minimal ext4
128 121
129 - This example specifies to boot an initial RAM disk image and to 122 - This example specifies to boot an :term:`Initramfs` image and to
130 enable audio in QEMU. For this case, ``runqemu`` set the internal 123 enable audio in QEMU. For this case, ``runqemu`` sets the internal
131 variable ``FSTYPE`` to "cpio.gz". Also, for audio to be enabled, 124 variable ``FSTYPE`` to ``cpio.gz``. Also, for audio to be enabled,
132 an appropriate driver must be installed (see the previous 125 an appropriate driver must be installed (see the ``audio`` option
133 description for the ``audio`` option for more information). 126 in :ref:`dev-manual/qemu:\`\`runqemu\`\` command-line options`
134 :: 127 for more information)::
135 128
136 $ runqemu qemux86-64 ramfs audio 129 $ runqemu qemux86-64 ramfs audio
137 130
138 - This example does not provide enough information for QEMU to 131 - This example does not provide enough information for QEMU to
139 launch. While the command does provide a root filesystem type, it 132 launch. While the command does provide a root filesystem type, it
140 must also minimally provide a `MACHINE`, `KERNEL`, or `VM` option. 133 must also minimally provide a `MACHINE`, `KERNEL`, or `VM` option::
141 ::
142 134
143 $ runqemu ext4 135 $ runqemu ext4
144 136
145 - This example specifies to boot a virtual machine image 137 - This example specifies to boot a virtual machine image
146 (``.wic.vmdk`` file). From the ``.wic.vmdk``, ``runqemu`` 138 (``.wic.vmdk`` file). From the ``.wic.vmdk``, ``runqemu``
147 determines the QEMU architecture (`MACHINE`) to be "qemux86-64" and 139 determines the QEMU architecture (`MACHINE`) to be "qemux86-64" and
148 the root filesystem type to be "vmdk". 140 the root filesystem type to be "vmdk"::
149 ::
150 141
151 $ runqemu /home/scott-lenovo/vm/core-image-minimal-qemux86-64.wic.vmdk 142 $ runqemu /home/scott-lenovo/vm/core-image-minimal-qemux86-64.wic.vmdk
152 143
@@ -193,7 +184,7 @@ the system does not need root privileges to run. It uses a user space
193NFS server to avoid that. Follow these steps to set up for running QEMU 184NFS server to avoid that. Follow these steps to set up for running QEMU
194using an NFS server. 185using an NFS server.
195 186
1961. *Extract a Root Filesystem:* Once you are able to run QEMU in your 187#. *Extract a Root Filesystem:* Once you are able to run QEMU in your
197 environment, you can use the ``runqemu-extract-sdk`` script, which is 188 environment, you can use the ``runqemu-extract-sdk`` script, which is
198 located in the ``scripts`` directory along with the ``runqemu`` 189 located in the ``scripts`` directory along with the ``runqemu``
199 script. 190 script.
@@ -207,7 +198,7 @@ using an NFS server.
207 198
208 runqemu-extract-sdk ./tmp/deploy/images/qemux86-64/core-image-sato-qemux86-64.tar.bz2 test-nfs 199 runqemu-extract-sdk ./tmp/deploy/images/qemux86-64/core-image-sato-qemux86-64.tar.bz2 test-nfs
209 200
2102. *Start QEMU:* Once you have extracted the file system, you can run 201#. *Start QEMU:* Once you have extracted the file system, you can run
211 ``runqemu`` normally with the additional location of the file system. 202 ``runqemu`` normally with the additional location of the file system.
212 You can then also make changes to the files within ``./test-nfs`` and 203 You can then also make changes to the files within ``./test-nfs`` and
213 see those changes appear in the image in real time. Here is an 204 see those changes appear in the image in real time. Here is an
@@ -222,18 +213,15 @@ using an NFS server.
222 Should you need to start, stop, or restart the NFS share, you can use 213 Should you need to start, stop, or restart the NFS share, you can use
223 the following commands: 214 the following commands:
224 215
225 - The following command starts the NFS share: 216 - To start the NFS share::
226 ::
227 217
228 runqemu-export-rootfs start file-system-location 218 runqemu-export-rootfs start file-system-location
229 219
230 - The following command stops the NFS share: 220 - To stop the NFS share::
231 ::
232 221
233 runqemu-export-rootfs stop file-system-location 222 runqemu-export-rootfs stop file-system-location
234 223
235 - The following command restarts the NFS share: 224 - To restart the NFS share::
236 ::
237 225
238 runqemu-export-rootfs restart file-system-location 226 runqemu-export-rootfs restart file-system-location
239 227
@@ -252,11 +240,10 @@ be a problem when QEMU is running with KVM enabled. Specifically,
252software compiled with a certain CPU feature crashes when run on a CPU 240software compiled with a certain CPU feature crashes when run on a CPU
253under KVM that does not support that feature. To work around this 241under KVM that does not support that feature. To work around this
254problem, you can override QEMU's runtime CPU setting by changing the 242problem, you can override QEMU's runtime CPU setting by changing the
255``QB_CPU_KVM`` variable in ``qemuboot.conf`` in the 243``QB_CPU_KVM`` variable in ``qemuboot.conf`` in the :term:`Build Directory`
256:term:`Build Directory` ``deploy/image`` 244``deploy/image`` directory. This setting specifies a ``-cpu`` option passed
257directory. This setting specifies a ``-cpu`` option passed into QEMU in 245into QEMU in the ``runqemu`` script. Running ``qemu -cpu help`` returns a
258the ``runqemu`` script. Running ``qemu -cpu help`` returns a list of 246list of available supported CPU types.
259available supported CPU types.
260 247
261QEMU Performance 248QEMU Performance
262================ 249================
@@ -281,7 +268,7 @@ present, the toolchain is also automatically used.
281 268
282.. note:: 269.. note::
283 270
284 Several mechanisms exist that let you connect to the system running 271 There are several mechanisms to connect to the system running
285 on the QEMU emulator: 272 on the QEMU emulator:
286 273
287 - QEMU provides a framebuffer interface that makes standard consoles 274 - QEMU provides a framebuffer interface that makes standard consoles
@@ -292,7 +279,7 @@ present, the toolchain is also automatically used.
292 that port to run a console. The connection uses standard IP 279 that port to run a console. The connection uses standard IP
293 networking. 280 networking.
294 281
295 - SSH servers exist in some QEMU images. The ``core-image-sato`` 282 - SSH servers are available in some QEMU images. The ``core-image-sato``
296 QEMU image has a Dropbear secure shell (SSH) server that runs with 283 QEMU image has a Dropbear secure shell (SSH) server that runs with
297 the root password disabled. The ``core-image-full-cmdline`` and 284 the root password disabled. The ``core-image-full-cmdline`` and
298 ``core-image-lsb`` QEMU images have OpenSSH instead of Dropbear. 285 ``core-image-lsb`` QEMU images have OpenSSH instead of Dropbear.
@@ -306,15 +293,14 @@ present, the toolchain is also automatically used.
306 tarball by using the ``runqemu-extract-sdk`` command. After 293 tarball by using the ``runqemu-extract-sdk`` command. After
307 running the command, you must then point the ``runqemu`` script to 294 running the command, you must then point the ``runqemu`` script to
308 the extracted directory instead of a root filesystem image file. 295 the extracted directory instead of a root filesystem image file.
309 See the "`Running Under a Network File System (NFS) 296 See the
310 Server <#qemu-running-under-a-network-file-system-nfs-server>`__" 297 ":ref:`dev-manual/qemu:running under a network file system (nfs) server`"
311 section for more information. 298 section for more information.
312 299
313QEMU Command-Line Syntax 300QEMU Command-Line Syntax
314======================== 301========================
315 302
316The basic ``runqemu`` command syntax is as follows: 303The basic ``runqemu`` command syntax is as follows::
317::
318 304
319 $ runqemu [option ] [...] 305 $ runqemu [option ] [...]
320 306
@@ -325,8 +311,7 @@ timestamp when it needs to look for an image. Minimally, through the use
325of options, you must provide either a machine name, a virtual machine 311of options, you must provide either a machine name, a virtual machine
326image (``*wic.vmdk``), or a kernel image (``*.bin``). 312image (``*wic.vmdk``), or a kernel image (``*.bin``).
327 313
328Following is the command-line help output for the ``runqemu`` command: 314Here is the command-line help output for the ``runqemu`` command::
329::
330 315
331 $ runqemu --help 316 $ runqemu --help
332 317
@@ -338,7 +323,7 @@ Following is the command-line help output for the ``runqemu`` command:
338 Simplified QEMU command-line options can be passed with: 323 Simplified QEMU command-line options can be passed with:
339 nographic - disable video console 324 nographic - disable video console
340 serial - enable a serial console on /dev/ttyS0 325 serial - enable a serial console on /dev/ttyS0
341 slirp - enable user networking, no root privileges is required 326 slirp - enable user networking, no root privileges required
342 kvm - enable KVM when running x86/x86_64 (VT-capable CPU required) 327 kvm - enable KVM when running x86/x86_64 (VT-capable CPU required)
343 kvm-vhost - enable KVM with vhost when running x86/x86_64 (VT-capable CPU required) 328 kvm-vhost - enable KVM with vhost when running x86/x86_64 (VT-capable CPU required)
344 publicvnc - enable a VNC server open to all hosts 329 publicvnc - enable a VNC server open to all hosts
@@ -368,7 +353,7 @@ Following is the command-line help output for the ``runqemu`` command:
368``runqemu`` Command-Line Options 353``runqemu`` Command-Line Options
369================================ 354================================
370 355
371Following is a description of ``runqemu`` options you can provide on the 356Here is a description of ``runqemu`` options you can provide on the
372command line: 357command line:
373 358
374.. note:: 359.. note::
@@ -402,7 +387,7 @@ command line:
402 options are basically identical. If you do not provide a MACHINE 387 options are basically identical. If you do not provide a MACHINE
403 option, ``runqemu`` tries to determine it based on other options. 388 option, ``runqemu`` tries to determine it based on other options.
404 389
405- ``ramfs``: Indicates you are booting an initial RAM disk (initramfs) 390- ``ramfs``: Indicates you are booting an :term:`Initramfs`
406 image, which means the ``FSTYPE`` is ``cpio.gz``. 391 image, which means the ``FSTYPE`` is ``cpio.gz``.
407 392
408- ``iso``: Indicates you are booting an ISO image, which means the 393- ``iso``: Indicates you are booting an ISO image, which means the
@@ -436,6 +421,29 @@ command line:
436 networking that does not need root access but also is not as easy to 421 networking that does not need root access but also is not as easy to
437 use or comprehensive as the default. 422 use or comprehensive as the default.
438 423
424 Using ``slirp`` by default will forward the guest machine's
425 22 and 23 TCP ports to host machine's 2222 and 2323 ports
426 (or the next free ports). Specific forwarding rules can be configured
427 by setting ``QB_SLIRP_OPT`` as environment variable or in ``qemuboot.conf``
428 in the :term:`Build Directory` ``deploy/image`` directory.
429 Examples::
430
431 QB_SLIRP_OPT="-netdev user,id=net0,hostfwd=tcp::8080-:80"
432
433 QB_SLIRP_OPT="-netdev user,id=net0,hostfwd=tcp::8080-:80,hostfwd=tcp::2222-:22"
434
435 The first example forwards TCP port 80 from the emulated system to
436 port 8080 (or the next free port) on the host system,
437 allowing access to an http server running in QEMU from
438 ``http://<host ip>:8080/``.
439
440 The second example does the same, but also forwards TCP port 22 on the
441 guest system to 2222 (or the next free port) on the host system,
442 allowing ssh access to the emulated system using
443 ``ssh -P 2222 <user>@<host ip>``.
444
445 Keep in mind that proper configuration of firewall software is required.
446
439- ``kvm``: Enables KVM when running "qemux86" or "qemux86-64" QEMU 447- ``kvm``: Enables KVM when running "qemux86" or "qemux86-64" QEMU
440 architectures. For KVM to work, all the following conditions must be 448 architectures. For KVM to work, all the following conditions must be
441 met: 449 met:
@@ -452,7 +460,7 @@ command line:
452 or "qemux86-64" QEMU architectures. For KVM with VHOST to work, the 460 or "qemux86-64" QEMU architectures. For KVM with VHOST to work, the
453 following conditions must be met: 461 following conditions must be met:
454 462
455 - `kvm <#kvm-cond>`__ option conditions must be met. 463 - ``kvm`` option conditions defined above must be met.
456 464
457 - Your build host has to have virtio net device, which are 465 - Your build host has to have virtio net device, which are
458 ``/dev/vhost-net``. 466 ``/dev/vhost-net``.
diff --git a/documentation/dev-manual/quilt.rst b/documentation/dev-manual/quilt.rst
new file mode 100644
index 0000000000..59240705ad
--- /dev/null
+++ b/documentation/dev-manual/quilt.rst
@@ -0,0 +1,89 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using Quilt in Your Workflow
4****************************
5
6`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
7that allows you to capture source code changes without having a clean
8source tree. This section outlines the typical workflow you can use to
9modify source code, test changes, and then preserve the changes in the
10form of a patch all using Quilt.
11
12.. note::
13
14 With regard to preserving changes to source files, if you clean a
15 recipe or have :ref:`ref-classes-rm-work` enabled, the
16 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
17 as described in the Yocto Project Application Development and the
18 Extensible Software Development Kit (eSDK) manual is a safer
19 development flow than the flow that uses Quilt.
20
21Follow these general steps:
22
23#. *Find the Source Code:* Temporary source code used by the
24 OpenEmbedded build system is kept in the :term:`Build Directory`. See the
25 ":ref:`dev-manual/temporary-source-code:finding temporary source code`" section to
26 learn how to locate the directory that has the temporary source code for a
27 particular package.
28
29#. *Change Your Working Directory:* You need to be in the directory that
30 has the temporary source code. That directory is defined by the
31 :term:`S` variable.
32
33#. *Create a New Patch:* Before modifying source code, you need to
34 create a new patch. To create a new patch file, use ``quilt new`` as
35 below::
36
37 $ quilt new my_changes.patch
38
39#. *Notify Quilt and Add Files:* After creating the patch, you need to
40 notify Quilt about the files you plan to edit. You notify Quilt by
41 adding the files to the patch you just created::
42
43 $ quilt add file1.c file2.c file3.c
44
45#. *Edit the Files:* Make your changes in the source code to the files
46 you added to the patch.
47
48#. *Test Your Changes:* Once you have modified the source code, the
49 easiest way to test your changes is by calling the :ref:`ref-tasks-compile`
50 task as shown in the following example::
51
52 $ bitbake -c compile -f package
53
54 The ``-f`` or ``--force`` option forces the specified task to
55 execute. If you find problems with your code, you can just keep
56 editing and re-testing iteratively until things work as expected.
57
58 .. note::
59
60 All the modifications you make to the temporary source code disappear
61 once you run the :ref:`ref-tasks-clean` or :ref:`ref-tasks-cleanall`
62 tasks using BitBake (i.e. ``bitbake -c clean package`` and
63 ``bitbake -c cleanall package``). Modifications will also disappear if
64 you use the :ref:`ref-classes-rm-work` feature as described in
65 the ":ref:`dev-manual/disk-space:conserving disk space during builds`"
66 section.
67
68#. *Generate the Patch:* Once your changes work as expected, you need to
69 use Quilt to generate the final patch that contains all your
70 modifications::
71
72 $ quilt refresh
73
74 At this point, the
75 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
76 ``file2.c``, and ``file3.c`` files.
77
78 You can find the resulting patch file in the ``patches/``
79 subdirectory of the source (:term:`S`) directory.
80
81#. *Copy the Patch File:* For simplicity, copy the patch file into a
82 directory named ``files``, which you can create in the same directory
83 that holds the recipe (``.bb``) file or the append (``.bbappend``)
84 file. Placing the patch here guarantees that the OpenEmbedded build
85 system will find the patch. Next, add the patch into the :term:`SRC_URI`
86 of the recipe. Here is an example::
87
88 SRC_URI += "file://my_changes.patch"
89
diff --git a/documentation/dev-manual/read-only-rootfs.rst b/documentation/dev-manual/read-only-rootfs.rst
new file mode 100644
index 0000000000..251178ed54
--- /dev/null
+++ b/documentation/dev-manual/read-only-rootfs.rst
@@ -0,0 +1,89 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Read-Only Root Filesystem
4************************************
5
6Suppose, for security reasons, you need to disable your target device's
7root filesystem's write permissions (i.e. you need a read-only root
8filesystem). Or, perhaps you are running the device's operating system
9from a read-only storage device. For either case, you can customize your
10image for that behavior.
11
12.. note::
13
14 Supporting a read-only root filesystem requires that the system and
15 applications do not try to write to the root filesystem. You must
16 configure all parts of the target system to write elsewhere, or to
17 gracefully fail in the event of attempting to write to the root
18 filesystem.
19
20Creating the Root Filesystem
21============================
22
23To create the read-only root filesystem, simply add the
24"read-only-rootfs" feature to your image, normally in one of two ways.
25The first way is to add the "read-only-rootfs" image feature in the
26image's recipe file via the :term:`IMAGE_FEATURES` variable::
27
28 IMAGE_FEATURES += "read-only-rootfs"
29
30As an alternative, you can add the same feature
31from within your :term:`Build Directory`'s ``local.conf`` file with the
32associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
33
34 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
35
36For more information on how to use these variables, see the
37":ref:`dev-manual/customizing-images:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
38section. For information on the variables, see
39:term:`IMAGE_FEATURES` and
40:term:`EXTRA_IMAGE_FEATURES`.
41
42Post-Installation Scripts and Read-Only Root Filesystem
43=======================================================
44
45It is very important that you make sure all post-Installation
46(``pkg_postinst``) scripts for packages that are installed into the
47image can be run at the time when the root filesystem is created during
48the build on the host system. These scripts cannot attempt to run during
49the first boot on the target device. With the "read-only-rootfs" feature
50enabled, the build system makes sure that all post-installation scripts
51succeed at file system creation time. If any of these scripts
52still need to be run after the root filesystem is created, the build
53immediately fails. These build-time checks ensure that the build fails
54rather than the target device fails later during its initial boot
55operation.
56
57Most of the common post-installation scripts generated by the build
58system for the out-of-the-box Yocto Project are engineered so that they
59can run during root filesystem creation (e.g. post-installation scripts
60for caching fonts). However, if you create and add custom scripts, you
61need to be sure they can be run during this file system creation.
62
63Here are some common problems that prevent post-installation scripts
64from running during root filesystem creation:
65
66- *Not using $D in front of absolute paths:* The build system defines
67 ``$``\ :term:`D` when the root
68 filesystem is created. Furthermore, ``$D`` is blank when the script
69 is run on the target device. This implies two purposes for ``$D``:
70 ensuring paths are valid in both the host and target environments,
71 and checking to determine which environment is being used as a method
72 for taking appropriate actions.
73
74- *Attempting to run processes that are specific to or dependent on the
75 target architecture:* You can work around these attempts by using
76 native tools, which run on the host system, to accomplish the same
77 tasks, or by alternatively running the processes under QEMU, which
78 has the ``qemu_run_binary`` function. For more information, see the
79 :ref:`ref-classes-qemu` class.
80
81Areas With Write Access
82=======================
83
84With the "read-only-rootfs" feature enabled, any attempt by the target
85to write to the root filesystem at runtime fails. Consequently, you must
86make sure that you configure processes and applications that attempt
87these types of writes do so to directories with write access (e.g.
88``/tmp`` or ``/var/run``).
89
diff --git a/documentation/dev-manual/runtime-testing.rst b/documentation/dev-manual/runtime-testing.rst
new file mode 100644
index 0000000000..7a2b42f25a
--- /dev/null
+++ b/documentation/dev-manual/runtime-testing.rst
@@ -0,0 +1,594 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Performing Automated Runtime Testing
4************************************
5
6The OpenEmbedded build system makes available a series of automated
7tests for images to verify runtime functionality. You can run these
8tests on either QEMU or actual target hardware. Tests are written in
9Python making use of the ``unittest`` module, and the majority of them
10run commands on the target system over SSH. This section describes how
11you set up the environment to use these tests, run available tests, and
12write and add your own tests.
13
14For information on the test and QA infrastructure available within the
15Yocto Project, see the ":ref:`ref-manual/release-process:testing and quality assurance`"
16section in the Yocto Project Reference Manual.
17
18Enabling Tests
19==============
20
21Depending on whether you are planning to run tests using QEMU or on the
22hardware, you have to take different steps to enable the tests. See the
23following subsections for information on how to enable both types of
24tests.
25
26Enabling Runtime Tests on QEMU
27------------------------------
28
29In order to run tests, you need to do the following:
30
31- *Set up to avoid interaction with sudo for networking:* To
32 accomplish this, you must do one of the following:
33
34 - Add ``NOPASSWD`` for your user in ``/etc/sudoers`` either for all
35 commands or just for ``runqemu-ifup``. You must provide the full
36 path as that can change if you are using multiple clones of the
37 source repository.
38
39 .. note::
40
41 On some distributions, you also need to comment out "Defaults
42 requiretty" in ``/etc/sudoers``.
43
44 - Manually configure a tap interface for your system.
45
46 - Run as root the script in ``scripts/runqemu-gen-tapdevs``, which
47 should generate a list of tap devices. This is the option
48 typically chosen for Autobuilder-type environments.
49
50 .. note::
51
52 - Be sure to use an absolute path when calling this script
53 with sudo.
54
55 - Ensure that your host has the package ``iptables`` installed.
56
57 - The package recipe ``qemu-helper-native`` is required to run
58 this script. Build the package using the following command::
59
60 $ bitbake qemu-helper-native
61
62- *Set the DISPLAY variable:* You need to set this variable so that
63 you have an X server available (e.g. start ``vncserver`` for a
64 headless machine).
65
66- *Be sure your host's firewall accepts incoming connections from
67 192.168.7.0/24:* Some of the tests (in particular DNF tests) start an
68 HTTP server on a random high number port, which is used to serve
69 files to the target. The DNF module serves
70 ``${WORKDIR}/oe-rootfs-repo`` so it can run DNF channel commands.
71 That means your host's firewall must accept incoming connections from
72 192.168.7.0/24, which is the default IP range used for tap devices by
73 ``runqemu``.
74
75- *Be sure your host has the correct packages installed:* Depending
76 your host's distribution, you need to have the following packages
77 installed:
78
79 - Ubuntu and Debian: ``sysstat`` and ``iproute2``
80
81 - openSUSE: ``sysstat`` and ``iproute2``
82
83 - Fedora: ``sysstat`` and ``iproute``
84
85 - CentOS: ``sysstat`` and ``iproute``
86
87Once you start running the tests, the following happens:
88
89#. A copy of the root filesystem is written to ``${WORKDIR}/testimage``.
90
91#. The image is booted under QEMU using the standard ``runqemu`` script.
92
93#. A default timeout of 500 seconds occurs to allow for the boot process
94 to reach the login prompt. You can change the timeout period by
95 setting
96 :term:`TEST_QEMUBOOT_TIMEOUT`
97 in the ``local.conf`` file.
98
99#. Once the boot process is reached and the login prompt appears, the
100 tests run. The full boot log is written to
101 ``${WORKDIR}/testimage/qemu_boot_log``.
102
103#. Each test module loads in the order found in :term:`TEST_SUITES`. You can
104 find the full output of the commands run over SSH in
105 ``${WORKDIR}/testimgage/ssh_target_log``.
106
107#. If no failures occur, the task running the tests ends successfully.
108 You can find the output from the ``unittest`` in the task log at
109 ``${WORKDIR}/temp/log.do_testimage``.
110
111Enabling Runtime Tests on Hardware
112----------------------------------
113
114The OpenEmbedded build system can run tests on real hardware, and for
115certain devices it can also deploy the image to be tested onto the
116device beforehand.
117
118For automated deployment, a "controller image" is installed onto the
119hardware once as part of setup. Then, each time tests are to be run, the
120following occurs:
121
122#. The controller image is booted into and used to write the image to be
123 tested to a second partition.
124
125#. The device is then rebooted using an external script that you need to
126 provide.
127
128#. The device boots into the image to be tested.
129
130When running tests (independent of whether the image has been deployed
131automatically or not), the device is expected to be connected to a
132network on a pre-determined IP address. You can either use static IP
133addresses written into the image, or set the image to use DHCP and have
134your DHCP server on the test network assign a known IP address based on
135the MAC address of the device.
136
137In order to run tests on hardware, you need to set :term:`TEST_TARGET` to an
138appropriate value. For QEMU, you do not have to change anything, the
139default value is "qemu". For running tests on hardware, the following
140options are available:
141
142- *"simpleremote":* Choose "simpleremote" if you are going to run tests
143 on a target system that is already running the image to be tested and
144 is available on the network. You can use "simpleremote" in
145 conjunction with either real hardware or an image running within a
146 separately started QEMU or any other virtual machine manager.
147
148- *"SystemdbootTarget":* Choose "SystemdbootTarget" if your hardware is
149 an EFI-based machine with ``systemd-boot`` as bootloader and
150 ``core-image-testmaster`` (or something similar) is installed. Also,
151 your hardware under test must be in a DHCP-enabled network that gives
152 it the same IP address for each reboot.
153
154 If you choose "SystemdbootTarget", there are additional requirements
155 and considerations. See the
156 ":ref:`dev-manual/runtime-testing:selecting systemdboottarget`" section, which
157 follows, for more information.
158
159- *"BeagleBoneTarget":* Choose "BeagleBoneTarget" if you are deploying
160 images and running tests on the BeagleBone "Black" or original
161 "White" hardware. For information on how to use these tests, see the
162 comments at the top of the BeagleBoneTarget
163 ``meta-yocto-bsp/lib/oeqa/controllers/beaglebonetarget.py`` file.
164
165- *"GrubTarget":* Choose "GrubTarget" if you are deploying images and running
166 tests on any generic PC that boots using GRUB. For information on how
167 to use these tests, see the comments at the top of the GrubTarget
168 ``meta-yocto-bsp/lib/oeqa/controllers/grubtarget.py`` file.
169
170- *"your-target":* Create your own custom target if you want to run
171 tests when you are deploying images and running tests on a custom
172 machine within your BSP layer. To do this, you need to add a Python
173 unit that defines the target class under ``lib/oeqa/controllers/``
174 within your layer. You must also provide an empty ``__init__.py``.
175 For examples, see files in ``meta-yocto-bsp/lib/oeqa/controllers/``.
176
177Selecting SystemdbootTarget
178---------------------------
179
180If you did not set :term:`TEST_TARGET` to "SystemdbootTarget", then you do
181not need any information in this section. You can skip down to the
182":ref:`dev-manual/runtime-testing:running tests`" section.
183
184If you did set :term:`TEST_TARGET` to "SystemdbootTarget", you also need to
185perform a one-time setup of your controller image by doing the following:
186
187#. *Set EFI_PROVIDER:* Be sure that :term:`EFI_PROVIDER` is as follows::
188
189 EFI_PROVIDER = "systemd-boot"
190
191#. *Build the controller image:* Build the ``core-image-testmaster`` image.
192 The ``core-image-testmaster`` recipe is provided as an example for a
193 "controller" image and you can customize the image recipe as you would
194 any other recipe.
195
196 Image recipe requirements are:
197
198 - Inherits ``core-image`` so that kernel modules are installed.
199
200 - Installs normal linux utilities not BusyBox ones (e.g. ``bash``,
201 ``coreutils``, ``tar``, ``gzip``, and ``kmod``).
202
203 - Uses a custom :term:`Initramfs` image with a custom
204 installer. A normal image that you can install usually creates a
205 single root filesystem partition. This image uses another installer that
206 creates a specific partition layout. Not all Board Support
207 Packages (BSPs) can use an installer. For such cases, you need to
208 manually create the following partition layout on the target:
209
210 - First partition mounted under ``/boot``, labeled "boot".
211
212 - The main root filesystem partition where this image gets installed,
213 which is mounted under ``/``.
214
215 - Another partition labeled "testrootfs" where test images get
216 deployed.
217
218#. *Install image:* Install the image that you just built on the target
219 system.
220
221The final thing you need to do when setting :term:`TEST_TARGET` to
222"SystemdbootTarget" is to set up the test image:
223
224#. *Set up your local.conf file:* Make sure you have the following
225 statements in your ``local.conf`` file::
226
227 IMAGE_FSTYPES += "tar.gz"
228 IMAGE_CLASSES += "testimage"
229 TEST_TARGET = "SystemdbootTarget"
230 TEST_TARGET_IP = "192.168.2.3"
231
232#. *Build your test image:* Use BitBake to build the image::
233
234 $ bitbake core-image-sato
235
236Power Control
237-------------
238
239For most hardware targets other than "simpleremote", you can control
240power:
241
242- You can use :term:`TEST_POWERCONTROL_CMD` together with
243 :term:`TEST_POWERCONTROL_EXTRA_ARGS` as a command that runs on the host
244 and does power cycling. The test code passes one argument to that
245 command: off, on or cycle (off then on). Here is an example that
246 could appear in your ``local.conf`` file::
247
248 TEST_POWERCONTROL_CMD = "powercontrol.exp test 10.11.12.1 nuc1"
249
250 In this example, the expect
251 script does the following:
252
253 .. code-block:: shell
254
255 ssh test@10.11.12.1 "pyctl nuc1 arg"
256
257 It then runs a Python script that controls power for a label called
258 ``nuc1``.
259
260 .. note::
261
262 You need to customize :term:`TEST_POWERCONTROL_CMD` and
263 :term:`TEST_POWERCONTROL_EXTRA_ARGS` for your own setup. The one requirement
264 is that it accepts "on", "off", and "cycle" as the last argument.
265
266- When no command is defined, it connects to the device over SSH and
267 uses the classic reboot command to reboot the device. Classic reboot
268 is fine as long as the machine actually reboots (i.e. the SSH test
269 has not failed). It is useful for scenarios where you have a simple
270 setup, typically with a single board, and where some manual
271 interaction is okay from time to time.
272
273If you have no hardware to automatically perform power control but still
274wish to experiment with automated hardware testing, you can use the
275``dialog-power-control`` script that shows a dialog prompting you to perform
276the required power action. This script requires either KDialog or Zenity
277to be installed. To use this script, set the
278:term:`TEST_POWERCONTROL_CMD`
279variable as follows::
280
281 TEST_POWERCONTROL_CMD = "${COREBASE}/scripts/contrib/dialog-power-control"
282
283Serial Console Connection
284-------------------------
285
286For test target classes requiring a serial console to interact with the
287bootloader (e.g. BeagleBoneTarget and GrubTarget),
288you need to specify a command to use to connect to the serial console of
289the target machine by using the
290:term:`TEST_SERIALCONTROL_CMD`
291variable and optionally the
292:term:`TEST_SERIALCONTROL_EXTRA_ARGS`
293variable.
294
295These cases could be a serial terminal program if the machine is
296connected to a local serial port, or a ``telnet`` or ``ssh`` command
297connecting to a remote console server. Regardless of the case, the
298command simply needs to connect to the serial console and forward that
299connection to standard input and output as any normal terminal program
300does. For example, to use the picocom terminal program on serial device
301``/dev/ttyUSB0`` at 115200bps, you would set the variable as follows::
302
303 TEST_SERIALCONTROL_CMD = "picocom /dev/ttyUSB0 -b 115200"
304
305For local
306devices where the serial port device disappears when the device reboots,
307an additional "serdevtry" wrapper script is provided. To use this
308wrapper, simply prefix the terminal command with
309``${COREBASE}/scripts/contrib/serdevtry``::
310
311 TEST_SERIALCONTROL_CMD = "${COREBASE}/scripts/contrib/serdevtry picocom -b 115200 /dev/ttyUSB0"
312
313Running Tests
314=============
315
316You can start the tests automatically or manually:
317
318- *Automatically running tests:* To run the tests automatically after the
319 OpenEmbedded build system successfully creates an image, first set the
320 :term:`TESTIMAGE_AUTO` variable to "1" in your ``local.conf`` file in the
321 :term:`Build Directory`::
322
323 TESTIMAGE_AUTO = "1"
324
325 Next, build your image. If the image successfully builds, the
326 tests run::
327
328 bitbake core-image-sato
329
330- *Manually running tests:* To manually run the tests, first globally
331 inherit the :ref:`ref-classes-testimage` class by editing your
332 ``local.conf`` file::
333
334 IMAGE_CLASSES += "testimage"
335
336 Next, use BitBake to run the tests::
337
338 bitbake -c testimage image
339
340All test files reside in ``meta/lib/oeqa/runtime/cases`` in the
341:term:`Source Directory`. A test name maps
342directly to a Python module. Each test module may contain a number of
343individual tests. Tests are usually grouped together by the area tested
344(e.g tests for systemd reside in ``meta/lib/oeqa/runtime/cases/systemd.py``).
345
346You can add tests to any layer provided you place them in the proper
347area and you extend :term:`BBPATH` in
348the ``local.conf`` file as normal. Be sure that tests reside in
349``layer/lib/oeqa/runtime/cases``.
350
351.. note::
352
353 Be sure that module names do not collide with module names used in
354 the default set of test modules in ``meta/lib/oeqa/runtime/cases``.
355
356You can change the set of tests run by appending or overriding
357:term:`TEST_SUITES` variable in
358``local.conf``. Each name in :term:`TEST_SUITES` represents a required test
359for the image. Test modules named within :term:`TEST_SUITES` cannot be
360skipped even if a test is not suitable for an image (e.g. running the
361RPM tests on an image without ``rpm``). Appending "auto" to
362:term:`TEST_SUITES` causes the build system to try to run all tests that are
363suitable for the image (i.e. each test module may elect to skip itself).
364
365The order you list tests in :term:`TEST_SUITES` is important and influences
366test dependencies. Consequently, tests that depend on other tests should
367be added after the test on which they depend. For example, since the
368``ssh`` test depends on the ``ping`` test, "ssh" needs to come after
369"ping" in the list. The test class provides no re-ordering or dependency
370handling.
371
372.. note::
373
374 Each module can have multiple classes with multiple test methods.
375 And, Python ``unittest`` rules apply.
376
377Here are some things to keep in mind when running tests:
378
379- The default tests for the image are defined as::
380
381 DEFAULT_TEST_SUITES:pn-image = "ping ssh df connman syslog xorg scp vnc date rpm dnf dmesg"
382
383- Add your own test to the list of the by using the following::
384
385 TEST_SUITES:append = " mytest"
386
387- Run a specific list of tests as follows::
388
389 TEST_SUITES = "test1 test2 test3"
390
391 Remember, order is important. Be sure to place a test that is
392 dependent on another test later in the order.
393
394Exporting Tests
395===============
396
397You can export tests so that they can run independently of the build
398system. Exporting tests is required if you want to be able to hand the
399test execution off to a scheduler. You can only export tests that are
400defined in :term:`TEST_SUITES`.
401
402If your image is already built, make sure the following are set in your
403``local.conf`` file::
404
405 INHERIT += "testexport"
406 TEST_TARGET_IP = "IP-address-for-the-test-target"
407 TEST_SERVER_IP = "IP-address-for-the-test-server"
408
409You can then export the tests with the
410following BitBake command form::
411
412 $ bitbake image -c testexport
413
414Exporting the tests places them in the :term:`Build Directory` in
415``tmp/testexport/``\ image, which is controlled by the :term:`TEST_EXPORT_DIR`
416variable.
417
418You can now run the tests outside of the build environment::
419
420 $ cd tmp/testexport/image
421 $ ./runexported.py testdata.json
422
423Here is a complete example that shows IP addresses and uses the
424``core-image-sato`` image::
425
426 INHERIT += "testexport"
427 TEST_TARGET_IP = "192.168.7.2"
428 TEST_SERVER_IP = "192.168.7.1"
429
430Use BitBake to export the tests::
431
432 $ bitbake core-image-sato -c testexport
433
434Run the tests outside of
435the build environment using the following::
436
437 $ cd tmp/testexport/core-image-sato
438 $ ./runexported.py testdata.json
439
440Writing New Tests
441=================
442
443As mentioned previously, all new test files need to be in the proper
444place for the build system to find them. New tests for additional
445functionality outside of the core should be added to the layer that adds
446the functionality, in ``layer/lib/oeqa/runtime/cases`` (as long as
447:term:`BBPATH` is extended in the
448layer's ``layer.conf`` file as normal). Just remember the following:
449
450- Filenames need to map directly to test (module) names.
451
452- Do not use module names that collide with existing core tests.
453
454- Minimally, an empty ``__init__.py`` file must be present in the runtime
455 directory.
456
457To create a new test, start by copying an existing module (e.g.
458``oe_syslog.py`` or ``gcc.py`` are good ones to use). Test modules can use
459code from ``meta/lib/oeqa/utils``, which are helper classes.
460
461.. note::
462
463 Structure shell commands such that you rely on them and they return a
464 single code for success. Be aware that sometimes you will need to
465 parse the output. See the ``df.py`` and ``date.py`` modules for examples.
466
467You will notice that all test classes inherit ``oeRuntimeTest``, which
468is found in ``meta/lib/oetest.py``. This base class offers some helper
469attributes, which are described in the following sections:
470
471Class Methods
472-------------
473
474Class methods are as follows:
475
476- *hasPackage(pkg):* Returns "True" if ``pkg`` is in the installed
477 package list of the image, which is based on the manifest file that
478 is generated during the :ref:`ref-tasks-rootfs` task.
479
480- *hasFeature(feature):* Returns "True" if the feature is in
481 :term:`IMAGE_FEATURES` or
482 :term:`DISTRO_FEATURES`.
483
484Class Attributes
485----------------
486
487Class attributes are as follows:
488
489- *pscmd:* Equals "ps -ef" if ``procps`` is installed in the image.
490 Otherwise, ``pscmd`` equals "ps" (busybox).
491
492- *tc:* The called test context, which gives access to the
493 following attributes:
494
495 - *d:* The BitBake datastore, which allows you to use stuff such
496 as ``oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager")``.
497
498 - *testslist and testsrequired:* Used internally. The tests
499 do not need these.
500
501 - *filesdir:* The absolute path to
502 ``meta/lib/oeqa/runtime/files``, which contains helper files for
503 tests meant for copying on the target such as small files written
504 in C for compilation.
505
506 - *target:* The target controller object used to deploy and
507 start an image on a particular target (e.g. Qemu, SimpleRemote,
508 and SystemdbootTarget). Tests usually use the following:
509
510 - *ip:* The target's IP address.
511
512 - *server_ip:* The host's IP address, which is usually used
513 by the DNF test suite.
514
515 - *run(cmd, timeout=None):* The single, most used method.
516 This command is a wrapper for: ``ssh root@host "cmd"``. The
517 command returns a tuple: (status, output), which are what their
518 names imply - the return code of "cmd" and whatever output it
519 produces. The optional timeout argument represents the number
520 of seconds the test should wait for "cmd" to return. If the
521 argument is "None", the test uses the default instance's
522 timeout period, which is 300 seconds. If the argument is "0",
523 the test runs until the command returns.
524
525 - *copy_to(localpath, remotepath):*
526 ``scp localpath root@ip:remotepath``.
527
528 - *copy_from(remotepath, localpath):*
529 ``scp root@host:remotepath localpath``.
530
531Instance Attributes
532-------------------
533
534There is a single instance attribute, which is ``target``. The ``target``
535instance attribute is identical to the class attribute of the same name,
536which is described in the previous section. This attribute exists as
537both an instance and class attribute so tests can use
538``self.target.run(cmd)`` in instance methods instead of
539``oeRuntimeTest.tc.target.run(cmd)``.
540
541Installing Packages in the DUT Without the Package Manager
542==========================================================
543
544When a test requires a package built by BitBake, it is possible to
545install that package. Installing the package does not require a package
546manager be installed in the device under test (DUT). It does, however,
547require an SSH connection and the target must be using the
548``sshcontrol`` class.
549
550.. note::
551
552 This method uses ``scp`` to copy files from the host to the target, which
553 causes permissions and special attributes to be lost.
554
555A JSON file is used to define the packages needed by a test. This file
556must be in the same path as the file used to define the tests.
557Furthermore, the filename must map directly to the test module name with
558a ``.json`` extension.
559
560The JSON file must include an object with the test name as keys of an
561object or an array. This object (or array of objects) uses the following
562data:
563
564- "pkg" --- a mandatory string that is the name of the package to be
565 installed.
566
567- "rm" --- an optional boolean, which defaults to "false", that specifies
568 to remove the package after the test.
569
570- "extract" --- an optional boolean, which defaults to "false", that
571 specifies if the package must be extracted from the package format.
572 When set to "true", the package is not automatically installed into
573 the DUT.
574
575Here is an example JSON file that handles test "foo" installing
576package "bar" and test "foobar" installing packages "foo" and "bar".
577Once the test is complete, the packages are removed from the DUT::
578
579 {
580 "foo": {
581 "pkg": "bar"
582 },
583 "foobar": [
584 {
585 "pkg": "foo",
586 "rm": true
587 },
588 {
589 "pkg": "bar",
590 "rm": true
591 }
592 ]
593 }
594
diff --git a/documentation/dev-manual/sbom.rst b/documentation/dev-manual/sbom.rst
new file mode 100644
index 0000000000..b72bad1554
--- /dev/null
+++ b/documentation/dev-manual/sbom.rst
@@ -0,0 +1,83 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Software Bill of Materials
4*************************************
5
6Once you are able to build an image for your project, once the licenses for
7each software component are all identified (see
8":ref:`dev-manual/licenses:working with licenses`") and once vulnerability
9fixes are applied (see ":ref:`dev-manual/vulnerabilities:checking
10for vulnerabilities`"), the OpenEmbedded build system can generate
11a description of all the components you used, their licenses, their dependencies,
12their sources, the changes that were applied to them and the known
13vulnerabilities that were fixed.
14
15This description is generated in the form of a *Software Bill of Materials*
16(:term:`SBOM`), using the :term:`SPDX` standard.
17
18When you release software, this is the most standard way to provide information
19about the Software Supply Chain of your software image and SDK. The
20:term:`SBOM` tooling is often used to ensure open source license compliance by
21providing the license texts used in the product which legal departments and end
22users can read in standardized format.
23
24:term:`SBOM` information is also critical to performing vulnerability exposure
25assessments, as all the components used in the Software Supply Chain are listed.
26
27The OpenEmbedded build system doesn't generate such information by default.
28To make this happen, you must inherit the
29:ref:`ref-classes-create-spdx` class from a configuration file::
30
31 INHERIT += "create-spdx"
32
33Upon building an image, you will then get:
34
35- :term:`SPDX` output in JSON format as an ``IMAGE-MACHINE.spdx.json`` file in
36 ``tmp/deploy/images/MACHINE/`` inside the :term:`Build Directory`.
37
38- This toplevel file is accompanied by an ``IMAGE-MACHINE.spdx.index.json``
39 containing an index of JSON :term:`SPDX` files for individual recipes.
40
41- The compressed archive ``IMAGE-MACHINE.spdx.tar.zst`` contains the index
42 and the files for the single recipes.
43
44The :ref:`ref-classes-create-spdx` class offers options to include
45more information in the output :term:`SPDX` data:
46
47- Make the json files more human readable by setting (:term:`SPDX_PRETTY`).
48
49- Add compressed archives of the files in the generated target packages by
50 setting (:term:`SPDX_ARCHIVE_PACKAGED`).
51
52- Add a description of the source files used to generate host tools and target
53 packages (:term:`SPDX_INCLUDE_SOURCES`)
54
55- Add archives of these source files themselves (:term:`SPDX_ARCHIVE_SOURCES`).
56
57Though the toplevel :term:`SPDX` output is available in
58``tmp/deploy/images/MACHINE/`` inside the :term:`Build Directory`, ancillary
59generated files are available in ``tmp/deploy/spdx/MACHINE`` too, such as:
60
61- The individual :term:`SPDX` JSON files in the ``IMAGE-MACHINE.spdx.tar.zst``
62 archive.
63
64- Compressed archives of the files in the generated target packages,
65 in ``packages/packagename.tar.zst`` (when :term:`SPDX_ARCHIVE_PACKAGED`
66 is set).
67
68- Compressed archives of the source files used to build the host tools
69 and the target packages in ``recipes/recipe-packagename.tar.zst``
70 (when :term:`SPDX_ARCHIVE_SOURCES` is set). Those are needed to fulfill
71 "source code access" license requirements.
72
73See also the :term:`SPDX_CUSTOM_ANNOTATION_VARS` variable which allows
74to associate custom notes to a recipe.
75See the `tools page <https://spdx.dev/resources/tools/>`__ on the :term:`SPDX`
76project website for a list of tools to consume and transform the :term:`SPDX`
77data generated by the OpenEmbedded build system.
78
79See also Joshua Watt's presentations
80`Automated SBoM generation with OpenEmbedded and the Yocto Project <https://youtu.be/Q5UQUM6zxVU>`__
81at FOSDEM 2023 and
82`SPDX in the Yocto Project <https://fosdem.org/2024/schedule/event/fosdem-2024-3318-spdx-in-the-yocto-project/>`__
83at FOSDEM 2024.
diff --git a/documentation/dev-manual/securing-images.rst b/documentation/dev-manual/securing-images.rst
new file mode 100644
index 0000000000..e5791d3d6d
--- /dev/null
+++ b/documentation/dev-manual/securing-images.rst
@@ -0,0 +1,156 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Making Images More Secure
4*************************
5
6Security is of increasing concern for embedded devices. Consider the
7issues and problems discussed in just this sampling of work found across
8the Internet:
9
10- *"*\ `Security Risks of Embedded
11 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
12 by Bruce Schneier
13
14- *"*\ `Internet Census
15 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
16 Botnet
17
18- *"*\ `Security Issues for Embedded
19 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
20 by Jake Edge
21
22When securing your image is of concern, there are steps, tools, and
23variables that you can consider to help you reach the security goals you
24need for your particular device. Not all situations are identical when
25it comes to making an image secure. Consequently, this section provides
26some guidance and suggestions for consideration when you want to make
27your image more secure.
28
29.. note::
30
31 Because the security requirements and risks are different for every
32 type of device, this section cannot provide a complete reference on
33 securing your custom OS. It is strongly recommended that you also
34 consult other sources of information on embedded Linux system
35 hardening and on security.
36
37General Considerations
38======================
39
40There are general considerations that help you create more secure images.
41You should consider the following suggestions to make your device
42more secure:
43
44- Scan additional code you are adding to the system (e.g. application
45 code) by using static analysis tools. Look for buffer overflows and
46 other potential security problems.
47
48- Pay particular attention to the security for any web-based
49 administration interface.
50
51 Web interfaces typically need to perform administrative functions and
52 tend to need to run with elevated privileges. Thus, the consequences
53 resulting from the interface's security becoming compromised can be
54 serious. Look for common web vulnerabilities such as
55 cross-site-scripting (XSS), unvalidated inputs, and so forth.
56
57 As with system passwords, the default credentials for accessing a
58 web-based interface should not be the same across all devices. This
59 is particularly true if the interface is enabled by default as it can
60 be assumed that many end-users will not change the credentials.
61
62- Ensure you can update the software on the device to mitigate
63 vulnerabilities discovered in the future. This consideration
64 especially applies when your device is network-enabled.
65
66- Regularly scan and apply fixes for CVE security issues affecting
67 all software components in the product, see ":ref:`dev-manual/vulnerabilities:checking for vulnerabilities`".
68
69- Regularly update your version of Poky and OE-Core from their upstream
70 developers, e.g. to apply updates and security fixes from stable
71 and :term:`LTS` branches.
72
73- Ensure you remove or disable debugging functionality before producing
74 the final image. For information on how to do this, see the
75 ":ref:`dev-manual/securing-images:considerations specific to the openembedded build system`"
76 section.
77
78- Ensure you have no network services listening that are not needed.
79
80- Remove any software from the image that is not needed.
81
82- Enable hardware support for secure boot functionality when your
83 device supports this functionality.
84
85Security Flags
86==============
87
88The Yocto Project has security flags that you can enable that help make
89your build output more secure. The security flags are in the
90``meta/conf/distro/include/security_flags.inc`` file in your
91:term:`Source Directory` (e.g. ``poky``).
92
93.. note::
94
95 Depending on the recipe, certain security flags are enabled and
96 disabled by default.
97
98Use the following line in your ``local.conf`` file or in your custom
99distribution configuration file to enable the security compiler and
100linker flags for your build::
101
102 require conf/distro/include/security_flags.inc
103
104Considerations Specific to the OpenEmbedded Build System
105========================================================
106
107You can take some steps that are specific to the OpenEmbedded build
108system to make your images more secure:
109
110- Ensure "debug-tweaks" is not one of your selected
111 :term:`IMAGE_FEATURES`.
112 When creating a new project, the default is to provide you with an
113 initial ``local.conf`` file that enables this feature using the
114 :term:`EXTRA_IMAGE_FEATURES`
115 variable with the line::
116
117 EXTRA_IMAGE_FEATURES = "debug-tweaks"
118
119 To disable that feature, simply comment out that line in your
120 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
121 "debug-tweaks" before producing your final image. Among other things,
122 leaving this in place sets the root password as blank, which makes
123 logging in for debugging or inspection easy during development but
124 also means anyone can easily log in during production.
125
126- It is possible to set a root password for the image and also to set
127 passwords for any extra users you might add (e.g. administrative or
128 service type users). When you set up passwords for multiple images or
129 users, you should not duplicate passwords.
130
131 To set up passwords, use the :ref:`ref-classes-extrausers` class, which
132 is the preferred method. For an example on how to set up both root and
133 user passwords, see the ":ref:`ref-classes-extrausers`" section.
134
135 .. note::
136
137 When adding extra user accounts or setting a root password, be
138 cautious about setting the same password on every device. If you
139 do this, and the password you have set is exposed, then every
140 device is now potentially compromised. If you need this access but
141 want to ensure security, consider setting a different, random
142 password for each device. Typically, you do this as a separate
143 step after you deploy the image onto the device.
144
145- Consider enabling a Mandatory Access Control (MAC) framework such as
146 SMACK or SELinux and tuning it appropriately for your device's usage.
147 You can find more information in the
148 :yocto_git:`meta-selinux </meta-selinux/>` layer.
149
150Tools for Hardening Your Image
151==============================
152
153The Yocto Project provides tools for making your image more secure. You
154can find these tools in the ``meta-security`` layer of the
155:yocto_git:`Yocto Project Source Repositories <>`.
156
diff --git a/documentation/dev-manual/security-subjects.rst b/documentation/dev-manual/security-subjects.rst
new file mode 100644
index 0000000000..1b02b6a9e9
--- /dev/null
+++ b/documentation/dev-manual/security-subjects.rst
@@ -0,0 +1,189 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Dealing with Vulnerability Reports
4**********************************
5
6The Yocto Project and OpenEmbedded are open-source, community-based projects
7used in numerous products. They assemble multiple other open-source projects,
8and need to handle security issues and practices both internal (in the code
9maintained by both projects), and external (maintained by other projects and
10organizations).
11
12This manual assembles security-related information concerning the whole
13ecosystem. It includes information on reporting a potential security issue,
14the operation of the YP Security team and how to contribute in the
15related code. It is written to be useful for both security researchers and
16YP developers.
17
18How to report a potential security vulnerability?
19=================================================
20
21If you would like to report a public issue (for example, one with a released
22CVE number), please report it using the
23:yocto_bugs:`Security Bugzilla </enter_bug.cgi?product=Security>`.
24
25If you are dealing with a not-yet-released issue, or an urgent one, please send
26a message to security AT yoctoproject DOT org, including as many details as
27possible: the layer or software module affected, the recipe and its version,
28and any example code, if available. This mailing list is monitored by the
29Yocto Project Security team.
30
31For each layer, you might also look for specific instructions (if any) for
32reporting potential security issues in the specific ``SECURITY.md`` file at the
33root of the repository. Instructions on how and where submit a patch are
34usually available in ``README.md``. If this is your first patch to the
35Yocto Project/OpenEmbedded, you might want to have a look into the
36Contributor's Manual section
37":ref:`contributor-guide/submit-changes:preparing changes for submission`".
38
39Branches maintained with security fixes
40---------------------------------------
41
42See the
43:ref:`Release process <ref-manual/release-process:Stable Release Process>`
44documentation for details regarding the policies and maintenance of stable
45branches.
46
47The :yocto_wiki:`Releases page </Releases>` contains a list
48of all releases of the Yocto Project. Versions in gray are no longer actively
49maintained with security patches, but well-tested patches may still be accepted
50for them for significant issues.
51
52Security-related discussions at the Yocto Project
53-------------------------------------------------
54
55We have set up two security-related mailing lists:
56
57 - Public List: yocto [dash] security [at] yoctoproject[dot] org
58
59 This is a public mailing list for anyone to subscribe to. This list is an
60 open list to discuss public security issues/patches and security-related
61 initiatives. For more information, including subscription information,
62 please see the :yocto_lists:`yocto-security mailing list info page </g/yocto-security>`.
63
64 - Private List: security [at] yoctoproject [dot] org
65
66 This is a private mailing list for reporting non-published potential
67 vulnerabilities. The list is monitored by the Yocto Project Security team.
68
69
70What you should do if you find a security vulnerability
71-------------------------------------------------------
72
73If you find a security flaw: a crash, an information leakage, or anything that
74can have a security impact if exploited in any Open Source software built or
75used by the Yocto Project, please report this to the Yocto Project Security
76Team. If you prefer to contact the upstream project directly, please send a
77copy to the security team at the Yocto Project as well. If you believe this is
78highly sensitive information, please report the vulnerability in a secure way,
79i.e. encrypt the email and send it to the private list. This ensures that
80the exploit is not leaked and exploited before a response/fix has been generated.
81
82Security team
83=============
84
85The Yocto Project/OpenEmbedded security team coordinates the work on security
86subjects in the project. All general discussion takes place publicly. The
87Security Team only uses confidential communication tools to deal with private
88vulnerability reports before they are released.
89
90Security team appointment
91-------------------------
92
93The Yocto Project Security Team consists of at least three members. When new
94members are needed, the Yocto Project Technical Steering Committee (YP TSC)
95asks for nominations by public channels including a nomination deadline.
96Self-nominations are possible. When the limit time is
97reached, the YP TSC posts the list of candidates for the comments of project
98participants and developers. Comments may be sent publicly or privately to the
99YP and OE TSCs. The candidates are approved by both YP TSC and OpenEmbedded
100Technical Steering Committee (OE TSC) and the final list of the team members
101is announced publicly. The aim is to have people representing technical
102leadership, security knowledge and infrastructure present with enough people
103to provide backup/coverage but keep the notification list small enough to
104minimize information risk and maintain trust.
105
106YP Security Team members may resign at any time.
107
108Security Team Operations
109------------------------
110
111The work of the Security Team might require high confidentiality. Team members
112are individuals selected by merit and do not represent the companies they work
113for. They do not share information about confidential issues outside of the team
114and do not hint about ongoing embargoes.
115
116Team members can bring in domain experts as needed. Those people should be
117added to individual issues only and adhere to the same standards as the YP
118Security Team.
119
120The YP security team organizes its meetings and communication as needed.
121
122When the YP Security team receives a report about a potential security
123vulnerability, they quickly analyze and notify the reporter of the result.
124They might also request more information.
125
126If the issue is confirmed and affects the code maintained by the YP, they
127confidentially notify maintainers of that code and work with them to prepare
128a fix.
129
130If the issue is confirmed and affects an upstream project, the YP security team
131notifies the project. Usually, the upstream project analyzes the problem again.
132If they deem it a real security problem in their software, they develop and
133release a fix following their security policy. They may want to include the
134original reporter in the loop. There is also sometimes some coordination for
135handling patches, backporting patches etc, or just understanding the problem
136or what caused it.
137
138When the fix is publicly available, the YP security team member or the
139package maintainer sends patches against the YP code base, following usual
140procedures, including public code review.
141
142What Yocto Security Team does when it receives a security vulnerability
143-----------------------------------------------------------------------
144
145The YP Security Team team performs a quick analysis and would usually report
146the flaw to the upstream project. Normally the upstream project analyzes the
147problem. If they deem it a real security problem in their software, they
148develop and release a fix following their own security policy. They may want
149to include the original reporter in the loop. There is also sometimes some
150coordination for handling patches, backporting patches etc, or just
151understanding the problem or what caused it.
152
153The security policy of the upstream project might include a notification to
154Linux distributions or other important downstream projects in advance to
155discuss coordinated disclosure. These mailing lists are normally non-public.
156
157When the upstream project releases a version with the fix, they are responsible
158for contacting `Mitre <https://www.cve.org/>`__ to get a CVE number assigned and
159the CVE record published.
160
161If an upstream project does not respond quickly
162-----------------------------------------------
163
164If an upstream project does not fix the problem in a reasonable time,
165the Yocto's Security Team will contact other interested parties (usually
166other distributions) in the community and together try to solve the
167vulnerability as quickly as possible.
168
169The Yocto Project Security team adheres to the 90 days disclosure policy
170by default. An increase of the embargo time is possible when necessary.
171
172Current Security Team members
173-----------------------------
174
175For secure communications, please send your messages encrypted using the GPG
176keys. Remember, message headers are not encrypted so do not include sensitive
177information in the subject line.
178
179 - Ross Burton: <ross@burtonini.com> `Public key <https://keys.openpgp.org/search?q=ross%40burtonini.com>`__
180
181 - Michael Halstead: <mhalstead [at] linuxfoundation [dot] org>
182 `Public key <https://pgp.mit.edu/pks/lookup?op=vindex&search=0x3373170601861969>`__
183 or `Public key <https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xd1f2407285e571ed12a407a73373170601861969>`__
184
185 - Richard Purdie: <richard.purdie@linuxfoundation.org> `Public key <https://keys.openpgp.org/search?q=richard.purdie%40linuxfoundation.org>`__
186
187 - Marta Rybczynska: <marta DOT rybczynska [at] syslinbit [dot] com> `Public key <https://keys.openpgp.org/search?q=marta.rybczynska@syslinbit.com>`__
188
189 - Steve Sakoman: <steve [at] sakoman [dot] com> `Public key <https://keys.openpgp.org/search?q=steve%40sakoman.com>`__
diff --git a/documentation/dev-manual/speeding-up-build.rst b/documentation/dev-manual/speeding-up-build.rst
new file mode 100644
index 0000000000..6e0d7873ac
--- /dev/null
+++ b/documentation/dev-manual/speeding-up-build.rst
@@ -0,0 +1,109 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Speeding Up a Build
4*******************
5
6Build time can be an issue. By default, the build system uses simple
7controls to try and maximize build efficiency. In general, the default
8settings for all the following variables result in the most efficient
9build times when dealing with single socket systems (i.e. a single CPU).
10If you have multiple CPUs, you might try increasing the default values
11to gain more speed. See the descriptions in the glossary for each
12variable for more information:
13
14- :term:`BB_NUMBER_THREADS`:
15 The maximum number of threads BitBake simultaneously executes.
16
17- :term:`BB_NUMBER_PARSE_THREADS`:
18 The number of threads BitBake uses during parsing.
19
20- :term:`PARALLEL_MAKE`: Extra
21 options passed to the ``make`` command during the
22 :ref:`ref-tasks-compile` task in
23 order to specify parallel compilation on the local build host.
24
25- :term:`PARALLEL_MAKEINST`:
26 Extra options passed to the ``make`` command during the
27 :ref:`ref-tasks-install` task in
28 order to specify parallel installation on the local build host.
29
30As mentioned, these variables all scale to the number of processor cores
31available on the build system. For single socket systems, this
32auto-scaling ensures that the build system fundamentally takes advantage
33of potential parallel operations during the build based on the build
34machine's capabilities.
35
36Additional factors that can affect build speed are:
37
38- File system type: The file system type that the build is being
39 performed on can also influence performance. Using ``ext4`` is
40 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
41 improved features such as extents.
42
43- Disabling the updating of access time using ``noatime``: The
44 ``noatime`` mount option prevents the build system from updating file
45 and directory access times.
46
47- Setting a longer commit: Using the "commit=" mount option increases
48 the interval in seconds between disk cache writes. Changing this
49 interval from the five second default to something longer increases
50 the risk of data loss but decreases the need to write to the disk,
51 thus increasing the build performance.
52
53- Choosing the packaging backend: Of the available packaging backends,
54 IPK is the fastest. Additionally, selecting a singular packaging
55 backend also helps.
56
57- Using ``tmpfs`` for :term:`TMPDIR`
58 as a temporary file system: While this can help speed up the build,
59 the benefits are limited due to the compiler using ``-pipe``. The
60 build system goes to some lengths to avoid ``sync()`` calls into the
61 file system on the principle that if there was a significant failure,
62 the :term:`Build Directory` contents could easily be rebuilt.
63
64- Inheriting the :ref:`ref-classes-rm-work` class:
65 Inheriting this class has shown to speed up builds due to
66 significantly lower amounts of data stored in the data cache as well
67 as on disk. Inheriting this class also makes cleanup of
68 :term:`TMPDIR` faster, at the
69 expense of being easily able to dive into the source code. File
70 system maintainers have recommended that the fastest way to clean up
71 large numbers of files is to reformat partitions rather than delete
72 files due to the linear nature of partitions. This, of course,
73 assumes you structure the disk partitions and file systems in a way
74 that this is practical.
75
76Aside from the previous list, you should keep some trade offs in mind
77that can help you speed up the build:
78
79- Remove items from
80 :term:`DISTRO_FEATURES`
81 that you might not need.
82
83- Exclude debug symbols and other debug information: If you do not need
84 these symbols and other debug information, disabling the ``*-dbg``
85 package generation can speed up the build. You can disable this
86 generation by setting the
87 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
88 variable to "1".
89
90- Disable static library generation for recipes derived from
91 ``autoconf`` or ``libtool``: Here is an example showing how to
92 disable static libraries and still provide an override to handle
93 exceptions::
94
95 STATICLIBCONF = "--disable-static"
96 STATICLIBCONF:sqlite3-native = ""
97 EXTRA_OECONF += "${STATICLIBCONF}"
98
99 .. note::
100
101 - Some recipes need static libraries in order to work correctly
102 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
103 as in the previous example, account for these kinds of
104 exceptions.
105
106 - Some packages have packaging code that assumes the presence of
107 the static libraries. If so, you might need to exclude them as
108 well.
109
diff --git a/documentation/dev-manual/start.rst b/documentation/dev-manual/start.rst
index 03061a79f3..386e5f5d29 100644
--- a/documentation/dev-manual/start.rst
+++ b/documentation/dev-manual/start.rst
@@ -29,14 +29,14 @@ however, keep in mind, the procedure here is simply a starting point.
29You can build off these steps and customize the procedure to fit any 29You can build off these steps and customize the procedure to fit any
30particular working environment and set of practices. 30particular working environment and set of practices.
31 31
321. *Determine Who is Going to be Developing:* You first need to 32#. *Determine Who is Going to be Developing:* You first need to
33 understand who is going to be doing anything related to the Yocto 33 understand who is going to be doing anything related to the Yocto
34 Project and determine their roles. Making this determination is 34 Project and determine their roles. Making this determination is
35 essential to completing subsequent steps, which are to get your 35 essential to completing subsequent steps, which are to get your
36 equipment together and set up your development environment's 36 equipment together and set up your development environment's
37 hardware topology. 37 hardware topology.
38 38
39 The following roles exist: 39 Possible roles are:
40 40
41 - *Application Developer:* This type of developer does application 41 - *Application Developer:* This type of developer does application
42 level work on top of an existing software stack. 42 level work on top of an existing software stack.
@@ -52,7 +52,7 @@ particular working environment and set of practices.
52 automated tests that are used to ensure all application and core 52 automated tests that are used to ensure all application and core
53 system development meets desired quality standards. 53 system development meets desired quality standards.
54 54
552. *Gather the Hardware:* Based on the size and make-up of the team, 55#. *Gather the Hardware:* Based on the size and make-up of the team,
56 get the hardware together. Ideally, any development, build, or test 56 get the hardware together. Ideally, any development, build, or test
57 engineer uses a system that runs a supported Linux distribution. 57 engineer uses a system that runs a supported Linux distribution.
58 These systems, in general, should be high performance (e.g. dual, 58 These systems, in general, should be high performance (e.g. dual,
@@ -66,13 +66,13 @@ particular working environment and set of practices.
66 building Yocto Project development containers to be run under 66 building Yocto Project development containers to be run under
67 Docker, which is described later. 67 Docker, which is described later.
68 68
693. *Understand the Hardware Topology of the Environment:* Once you 69#. *Understand the Hardware Topology of the Environment:* Once you
70 understand the hardware involved and the make-up of the team, you 70 understand the hardware involved and the make-up of the team, you
71 can understand the hardware topology of the development environment. 71 can understand the hardware topology of the development environment.
72 You can get a visual idea of the machines and their roles across the 72 You can get a visual idea of the machines and their roles across the
73 development environment. 73 development environment.
74 74
754. *Use Git as Your Source Control Manager (SCM):* Keeping your 75#. *Use Git as Your Source Control Manager (SCM):* Keeping your
76 :term:`Metadata` (i.e. recipes, 76 :term:`Metadata` (i.e. recipes,
77 configuration files, classes, and so forth) and any software you are 77 configuration files, classes, and so forth) and any software you are
78 developing under the control of an SCM system that is compatible 78 developing under the control of an SCM system that is compatible
@@ -88,31 +88,18 @@ particular working environment and set of practices.
88 For information about BitBake, see the 88 For information about BitBake, see the
89 :doc:`bitbake:index`. 89 :doc:`bitbake:index`.
90 90
91 It is relatively easy to set up Git services and create 91 It is relatively easy to set up Git services and create infrastructure like
92 infrastructure like :yocto_git:`/`, which is based on 92 :yocto_git:`/`, which is based on server software called
93 server software called ``gitolite`` with ``cgit`` being used to 93 `Gitolite <https://gitolite.com>`__
94 generate the web interface that lets you view the repositories. The 94 with `cgit <https://git.zx2c4.com/cgit/about/>`__ being used to
95 ``gitolite`` software identifies users using SSH keys and allows 95 generate the web interface that lets you view the repositories.
96 ``gitolite`` identifies users using SSH keys and allows
96 branch-based access controls to repositories that you can control as 97 branch-based access controls to repositories that you can control as
97 little or as much as necessary. 98 little or as much as necessary.
98 99
99 .. note:: 100#. *Set up the Application Development Machines:* As mentioned earlier,
100
101 The setup of these services is beyond the scope of this manual.
102 However, sites such as the following exist that describe how to
103 perform setup:
104
105 - `Gitolite <https://gitolite.com>`__: Information for
106 ``gitolite``.
107
108 - `Interfaces, frontends, and
109 tools <https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools>`__:
110 Documentation on how to create interfaces and frontends for
111 Git.
112
1135. *Set up the Application Development Machines:* As mentioned earlier,
114 application developers are creating applications on top of existing 101 application developers are creating applications on top of existing
115 software stacks. Following are some best practices for setting up 102 software stacks. Here are some best practices for setting up
116 machines used for application development: 103 machines used for application development:
117 104
118 - Use a pre-built toolchain that contains the software stack 105 - Use a pre-built toolchain that contains the software stack
@@ -129,9 +116,9 @@ particular working environment and set of practices.
129 - Use multiple toolchains installed locally into different 116 - Use multiple toolchains installed locally into different
130 locations to allow development across versions. 117 locations to allow development across versions.
131 118
1326. *Set up the Core Development Machines:* As mentioned earlier, core 119#. *Set up the Core Development Machines:* As mentioned earlier, core
133 developers work on the contents of the operating system itself. 120 developers work on the contents of the operating system itself.
134 Following are some best practices for setting up machines used for 121 Here are some best practices for setting up machines used for
135 developing images: 122 developing images:
136 123
137 - Have the :term:`OpenEmbedded Build System` available on 124 - Have the :term:`OpenEmbedded Build System` available on
@@ -146,7 +133,7 @@ particular working environment and set of practices.
146 - Share layers amongst the developers of a particular project and 133 - Share layers amongst the developers of a particular project and
147 contain the policy configuration that defines the project. 134 contain the policy configuration that defines the project.
148 135
1497. *Set up an Autobuilder:* Autobuilders are often the core of the 136#. *Set up an Autobuilder:* Autobuilders are often the core of the
150 development environment. It is here that changes from individual 137 development environment. It is here that changes from individual
151 developers are brought together and centrally tested. Based on this 138 developers are brought together and centrally tested. Based on this
152 automated build and test environment, subsequent decisions about 139 automated build and test environment, subsequent decisions about
@@ -184,13 +171,13 @@ particular working environment and set of practices.
184 - Allows scheduling of builds so that resources can be used 171 - Allows scheduling of builds so that resources can be used
185 efficiently. 172 efficiently.
186 173
1878. *Set up Test Machines:* Use a small number of shared, high 174#. *Set up Test Machines:* Use a small number of shared, high
188 performance systems for testing purposes. Developers can use these 175 performance systems for testing purposes. Developers can use these
189 systems for wider, more extensive testing while they continue to 176 systems for wider, more extensive testing while they continue to
190 develop locally using their primary development system. 177 develop locally using their primary development system.
191 178
1929. *Document Policies and Change Flow:* The Yocto Project uses a 179#. *Document Policies and Change Flow:* The Yocto Project uses a
193 hierarchical structure and a pull model. Scripts exist to create and 180 hierarchical structure and a pull model. There are scripts to create and
194 send pull requests (i.e. ``create-pull-request`` and 181 send pull requests (i.e. ``create-pull-request`` and
195 ``send-pull-request``). This model is in line with other open source 182 ``send-pull-request``). This model is in line with other open source
196 projects where maintainers are responsible for specific areas of the 183 projects where maintainers are responsible for specific areas of the
@@ -214,9 +201,9 @@ particular working environment and set of practices.
214 possible. Chances are if you have discovered the need for changes, 201 possible. Chances are if you have discovered the need for changes,
215 someone else in the community needs them also. 202 someone else in the community needs them also.
216 203
21710. *Development Environment Summary:* Aside from the previous steps, 204#. *Development Environment Summary:* Aside from the previous steps,
218 some best practices exist within the Yocto Project development 205 here are best practices within the Yocto Project development
219 environment. Consider the following: 206 environment:
220 207
221 - Use :ref:`overview-manual/development-environment:git` as the source control 208 - Use :ref:`overview-manual/development-environment:git` as the source control
222 system. 209 system.
@@ -224,14 +211,14 @@ particular working environment and set of practices.
224 - Maintain your Metadata in layers that make sense for your 211 - Maintain your Metadata in layers that make sense for your
225 situation. See the ":ref:`overview-manual/yp-intro:the yocto project layer model`" 212 situation. See the ":ref:`overview-manual/yp-intro:the yocto project layer model`"
226 section in the Yocto Project Overview and Concepts Manual and the 213 section in the Yocto Project Overview and Concepts Manual and the
227 ":ref:`dev-manual/common-tasks:understanding and creating layers`" 214 ":ref:`dev-manual/layers:understanding and creating layers`"
228 section for more information on layers. 215 section for more information on layers.
229 216
230 - Separate the project's Metadata and code by using separate Git 217 - Separate the project's Metadata and code by using separate Git
231 repositories. See the ":ref:`overview-manual/development-environment:yocto project source repositories`" 218 repositories. See the ":ref:`overview-manual/development-environment:yocto project source repositories`"
232 section in the Yocto Project Overview and Concepts Manual for 219 section in the Yocto Project Overview and Concepts Manual for
233 information on these repositories. See the "`Locating Yocto 220 information on these repositories. See the
234 Project Source Files <#locating-yocto-project-source-files>`__" 221 ":ref:`dev-manual/start:locating yocto project source files`"
235 section for information on how to set up local Git repositories 222 section for information on how to set up local Git repositories
236 for related upstream Yocto Project Git repositories. 223 for related upstream Yocto Project Git repositories.
237 224
@@ -247,14 +234,13 @@ particular working environment and set of practices.
247 - The Yocto Project community encourages you to send patches to the 234 - The Yocto Project community encourages you to send patches to the
248 project to fix bugs or add features. If you do submit patches, 235 project to fix bugs or add features. If you do submit patches,
249 follow the project commit guidelines for writing good commit 236 follow the project commit guidelines for writing good commit
250 messages. See the 237 messages. See the ":doc:`../contributor-guide/submit-changes`"
251 ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" 238 section in the Yocto Project and OpenEmbedded Contributor Guide.
252 section.
253 239
254 - Send changes to the core sooner than later as others are likely 240 - Send changes to the core sooner than later as others are likely
255 to run into the same issues. For some guidance on mailing lists 241 to run into the same issues. For some guidance on mailing lists
256 to use, see the list in the 242 to use, see the lists in the
257 ":ref:`dev-manual/common-tasks:submitting a change to the yocto project`" 243 ":ref:`contributor-guide/submit-changes:finding a suitable mailing list`"
258 section. For a description 244 section. For a description
259 of the available mailing lists, see the ":ref:`resources-mailinglist`" section in 245 of the available mailing lists, see the ":ref:`resources-mailinglist`" section in
260 the Yocto Project Reference Manual. 246 the Yocto Project Reference Manual.
@@ -268,16 +254,16 @@ development using the Yocto Project. Your build host can be a native
268Linux machine (recommended), it can be a machine (Linux, Mac, or 254Linux machine (recommended), it can be a machine (Linux, Mac, or
269Windows) that uses `CROPS <https://github.com/crops/poky-container>`__, 255Windows) that uses `CROPS <https://github.com/crops/poky-container>`__,
270which leverages `Docker Containers <https://www.docker.com/>`__ or it 256which leverages `Docker Containers <https://www.docker.com/>`__ or it
271can be a Windows machine capable of running Windows Subsystem For Linux 257can be a Windows machine capable of running version 2 of Windows Subsystem
272v2 (WSL). 258For Linux (WSL 2).
273 259
274.. note:: 260.. note::
275 261
276 The Yocto Project is not compatible with 262 The Yocto Project is not compatible with version 1 of
277 `Windows Subsystem for Linux v1 <https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux>`__. 263 :wikipedia:`Windows Subsystem for Linux <Windows_Subsystem_for_Linux>`.
278 It is compatible but not officially supported nor validated with 264 It is compatible but neither officially supported nor validated with
279 WSLv2. If you still decide to use WSL please upgrade to 265 WSL 2. If you still decide to use WSL please upgrade to
280 `WSLv2 <https://docs.microsoft.com/en-us/windows/wsl/install-win10>`__. 266 `WSL 2 <https://learn.microsoft.com/en-us/windows/wsl/install>`__.
281 267
282Once your build host is set up to use the Yocto Project, further steps 268Once your build host is set up to use the Yocto Project, further steps
283are necessary depending on what you want to accomplish. See the 269are necessary depending on what you want to accomplish. See the
@@ -297,38 +283,40 @@ Setting Up a Native Linux Host
297Follow these steps to prepare a native Linux machine as your Yocto 283Follow these steps to prepare a native Linux machine as your Yocto
298Project Build Host: 284Project Build Host:
299 285
3001. *Use a Supported Linux Distribution:* You should have a reasonably 286#. *Use a Supported Linux Distribution:* You should have a reasonably
301 current Linux-based host system. You will have the best results with 287 current Linux-based host system. You will have the best results with
302 a recent release of Fedora, openSUSE, Debian, Ubuntu, RHEL or CentOS 288 a recent release of Fedora, openSUSE, Debian, Ubuntu, RHEL or CentOS
303 as these releases are frequently tested against the Yocto Project and 289 as these releases are frequently tested against the Yocto Project and
304 officially supported. For a list of the distributions under 290 officially supported. For a list of the distributions under
305 validation and their status, see the ":ref:`Supported Linux 291 validation and their status, see the ":ref:`Supported Linux
306 Distributions <detailed-supported-distros>`" 292 Distributions <system-requirements-supported-distros>`"
307 section in the Yocto Project Reference Manual and the wiki page at 293 section in the Yocto Project Reference Manual and the wiki page at
308 :yocto_wiki:`Distribution Support </Distribution_Support>`. 294 :yocto_wiki:`Distribution Support </Distribution_Support>`.
309 295
3102. *Have Enough Free Memory:* Your system should have at least 50 Gbytes 296#. *Have Enough Free Memory:* Your system should have at least 50 Gbytes
311 of free disk space for building images. 297 of free disk space for building images.
312 298
3133. *Meet Minimal Version Requirements:* The OpenEmbedded build system 299#. *Meet Minimal Version Requirements:* The OpenEmbedded build system
314 should be able to run on any modern distribution that has the 300 should be able to run on any modern distribution that has the
315 following versions for Git, tar, Python and gcc. 301 following versions for Git, tar, Python, gcc and make.
302
303 - Git &MIN_GIT_VERSION; or greater
316 304
317 - Git 1.8.3.1 or greater 305 - tar &MIN_TAR_VERSION; or greater
318 306
319 - tar 1.28 or greater 307 - Python &MIN_PYTHON_VERSION; or greater.
320 308
321 - Python 3.5.0 or greater. 309 - gcc &MIN_GCC_VERSION; or greater.
322 310
323 - gcc 5.0 or greater. 311 - GNU make &MIN_MAKE_VERSION; or greater
324 312
325 If your build host does not meet any of these three listed version 313 If your build host does not meet any of these listed version
326 requirements, you can take steps to prepare the system so that you 314 requirements, you can take steps to prepare the system so that you
327 can still use the Yocto Project. See the 315 can still use the Yocto Project. See the
328 ":ref:`ref-manual/system-requirements:required git, tar, python and gcc versions`" 316 ":ref:`ref-manual/system-requirements:required git, tar, python, make and gcc versions`"
329 section in the Yocto Project Reference Manual for information. 317 section in the Yocto Project Reference Manual for information.
330 318
3314. *Install Development Host Packages:* Required development host 319#. *Install Development Host Packages:* Required development host
332 packages vary depending on your build host and what you want to do 320 packages vary depending on your build host and what you want to do
333 with the Yocto Project. Collectively, the number of required packages 321 with the Yocto Project. Collectively, the number of required packages
334 is large if you want to be able to cover all cases. 322 is large if you want to be able to cover all cases.
@@ -346,7 +334,10 @@ to use the Extensible SDK, see the ":doc:`/sdk-manual/extensible`" Chapter in th
346Project Application Development and the Extensible Software Development 334Project Application Development and the Extensible Software Development
347Kit (eSDK) manual. If you want to work on the kernel, see the :doc:`/kernel-dev/index`. If you are going to use 335Kit (eSDK) manual. If you want to work on the kernel, see the :doc:`/kernel-dev/index`. If you are going to use
348Toaster, see the ":doc:`/toaster-manual/setup-and-use`" 336Toaster, see the ":doc:`/toaster-manual/setup-and-use`"
349section in the Toaster User Manual. 337section in the Toaster User Manual. If you are a VSCode user, you can configure
338the `Yocto Project BitBake
339<https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__
340extension accordingly.
350 341
351Setting Up to Use CROss PlatformS (CROPS) 342Setting Up to Use CROss PlatformS (CROPS)
352----------------------------------------- 343-----------------------------------------
@@ -360,7 +351,7 @@ Yocto Project on a Windows, Mac, or Linux machine.
360Follow these general steps to prepare a Windows, Mac, or Linux machine 351Follow these general steps to prepare a Windows, Mac, or Linux machine
361as your Yocto Project build host: 352as your Yocto Project build host:
362 353
3631. *Determine What Your Build Host Needs:* 354#. *Determine What Your Build Host Needs:*
364 `Docker <https://www.docker.com/what-docker>`__ is a software 355 `Docker <https://www.docker.com/what-docker>`__ is a software
365 container platform that you need to install on the build host. 356 container platform that you need to install on the build host.
366 Depending on your build host, you might have to install different 357 Depending on your build host, you might have to install different
@@ -369,65 +360,57 @@ as your Yocto Project build host:
369 Platforms <https://docs.docker.com/engine/install/#supported-platforms>`__" 360 Platforms <https://docs.docker.com/engine/install/#supported-platforms>`__"
370 your build host needs to run containers. 361 your build host needs to run containers.
371 362
3722. *Choose What To Install:* Depending on whether or not your build host 363#. *Choose What To Install:* Depending on whether or not your build host
373 meets system requirements, you need to install "Docker CE Stable" or 364 meets system requirements, you need to install "Docker CE Stable" or
374 the "Docker Toolbox". Most situations call for Docker CE. However, if 365 the "Docker Toolbox". Most situations call for Docker CE. However, if
375 you have a build host that does not meet requirements (e.g. 366 you have a build host that does not meet requirements (e.g.
376 Pre-Windows 10 or Windows 10 "Home" version), you must install Docker 367 Pre-Windows 10 or Windows 10 "Home" version), you must install Docker
377 Toolbox instead. 368 Toolbox instead.
378 369
3793. *Go to the Install Site for Your Platform:* Click the link for the 370#. *Go to the Install Site for Your Platform:* Click the link for the
380 Docker edition associated with your build host's native software. For 371 Docker edition associated with your build host's native software. For
381 example, if your build host is running Microsoft Windows Version 10 372 example, if your build host is running Microsoft Windows Version 10
382 and you want the Docker CE Stable edition, click that link under 373 and you want the Docker CE Stable edition, click that link under
383 "Supported Platforms". 374 "Supported Platforms".
384 375
3854. *Install the Software:* Once you have understood all the 376#. *Install the Software:* Once you have understood all the
386 pre-requisites, you can download and install the appropriate 377 pre-requisites, you can download and install the appropriate
387 software. Follow the instructions for your specific machine and the 378 software. Follow the instructions for your specific machine and the
388 type of the software you need to install: 379 type of the software you need to install:
389 380
390 - Install `Docker CE for 381 - Install `Docker Desktop on
391 Windows <https://docs.docker.com/docker-for-windows/install/#install-docker-desktop-on-windows>`__ 382 Windows <https://docs.docker.com/docker-for-windows/install/#install-docker-desktop-on-windows>`__
392 for Windows build hosts that meet requirements. 383 for Windows build hosts that meet requirements.
393 384
394 - Install `Docker CE for 385 - Install `Docker Desktop on
395 MacOs <https://docs.docker.com/docker-for-mac/install/#install-and-run-docker-desktop-on-mac>`__ 386 MacOs <https://docs.docker.com/docker-for-mac/install/#install-and-run-docker-desktop-on-mac>`__
396 for Mac build hosts that meet requirements. 387 for Mac build hosts that meet requirements.
397 388
398 - Install `Docker Toolbox for 389 - Install `Docker Engine on
399 Windows <https://docs.docker.com/toolbox/toolbox_install_windows/>`__ 390 CentOS <https://docs.docker.com/engine/install/centos/>`__
400 for Windows build hosts that do not meet Docker requirements.
401
402 - Install `Docker Toolbox for
403 MacOS <https://docs.docker.com/toolbox/toolbox_install_mac/>`__
404 for Mac build hosts that do not meet Docker requirements.
405
406 - Install `Docker CE for
407 CentOS <https://docs.docker.com/install/linux/docker-ce/centos/>`__
408 for Linux build hosts running the CentOS distribution. 391 for Linux build hosts running the CentOS distribution.
409 392
410 - Install `Docker CE for 393 - Install `Docker Engine on
411 Debian <https://docs.docker.com/install/linux/docker-ce/debian/>`__ 394 Debian <https://docs.docker.com/engine/install/debian/>`__
412 for Linux build hosts running the Debian distribution. 395 for Linux build hosts running the Debian distribution.
413 396
414 - Install `Docker CE for 397 - Install `Docker Engine for
415 Fedora <https://docs.docker.com/install/linux/docker-ce/fedora/>`__ 398 Fedora <https://docs.docker.com/engine/install/fedora/>`__
416 for Linux build hosts running the Fedora distribution. 399 for Linux build hosts running the Fedora distribution.
417 400
418 - Install `Docker CE for 401 - Install `Docker Engine for
419 Ubuntu <https://docs.docker.com/install/linux/docker-ce/ubuntu/>`__ 402 Ubuntu <https://docs.docker.com/engine/install/ubuntu/>`__
420 for Linux build hosts running the Ubuntu distribution. 403 for Linux build hosts running the Ubuntu distribution.
421 404
4225. *Optionally Orient Yourself With Docker:* If you are unfamiliar with 405#. *Optionally Orient Yourself With Docker:* If you are unfamiliar with
423 Docker and the container concept, you can learn more here - 406 Docker and the container concept, you can learn more here -
424 https://docs.docker.com/get-started/. 407 https://docs.docker.com/get-started/.
425 408
4266. *Launch Docker or Docker Toolbox:* You should be able to launch 409#. *Launch Docker or Docker Toolbox:* You should be able to launch
427 Docker or the Docker Toolbox and have a terminal shell on your 410 Docker or the Docker Toolbox and have a terminal shell on your
428 development host. 411 development host.
429 412
4307. *Set Up the Containers to Use the Yocto Project:* Go to 413#. *Set Up the Containers to Use the Yocto Project:* Go to
431 https://github.com/crops/docker-win-mac-docs/wiki and follow 414 https://github.com/crops/docker-win-mac-docs/wiki and follow
432 the directions for your particular build host (i.e. Linux, Mac, or 415 the directions for your particular build host (i.e. Linux, Mac, or
433 Windows). 416 Windows).
@@ -446,37 +429,41 @@ section. If you are going to use the Extensible SDK container, see the
446Project Application Development and the Extensible Software Development 429Project Application Development and the Extensible Software Development
447Kit (eSDK) manual. If you are going to use the Toaster container, see 430Kit (eSDK) manual. If you are going to use the Toaster container, see
448the ":doc:`/toaster-manual/setup-and-use`" 431the ":doc:`/toaster-manual/setup-and-use`"
449section in the Toaster User Manual. 432section in the Toaster User Manual. If you are a VSCode user, you can configure
433the `Yocto Project BitBake
434<https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__
435extension accordingly.
450 436
451Setting Up to Use Windows Subsystem For Linux (WSLv2) 437Setting Up to Use Windows Subsystem For Linux (WSL 2)
452----------------------------------------------------- 438-----------------------------------------------------
453 439
454With `Windows Subsystem for Linux 440With `Windows Subsystem for Linux (WSL 2)
455(WSLv2) <https://docs.microsoft.com/en-us/windows/wsl/wsl2-about>`__, 441<https://learn.microsoft.com/en-us/windows/wsl/>`__,
456you can create a Yocto Project development environment that allows you 442you can create a Yocto Project development environment that allows you
457to build on Windows. You can set up a Linux distribution inside Windows 443to build on Windows. You can set up a Linux distribution inside Windows
458in which you can develop using the Yocto Project. 444in which you can develop using the Yocto Project.
459 445
460Follow these general steps to prepare a Windows machine using WSLv2 as 446Follow these general steps to prepare a Windows machine using WSL 2 as
461your Yocto Project build host: 447your Yocto Project build host:
462 448
4631. *Make sure your Windows 10 machine is capable of running WSLv2:* 449#. *Make sure your Windows machine is capable of running WSL 2:*
464 WSLv2 is only available for Windows 10 builds > 18917. To check which 450
465 build version you are running, you may open a command prompt on 451 While all Windows 11 and Windows Server 2022 builds support WSL 2,
466 Windows and execute the command "ver". 452 the first versions of Windows 10 and Windows Server 2019 didn't.
467 :: 453 Check the minimum build numbers for `Windows 10
454 <https://learn.microsoft.com/en-us/windows/wsl/install-manual#step-2---check-requirements-for-running-wsl-2>`__
455 and for `Windows Server 2019
456 <https://learn.microsoft.com/en-us/windows/wsl/install-on-server>`__.
457
458 To check which build version you are running, you may open a command
459 prompt on Windows and execute the command "ver"::
468 460
469 C:\Users\myuser> ver 461 C:\Users\myuser> ver
470 462
471 Microsoft Windows [Version 10.0.19041.153] 463 Microsoft Windows [Version 10.0.19041.153]
472 464
473 If your build is capable of running 465#. *Install the Linux distribution of your choice inside WSL 2:*
474 WSLv2 you may continue, for more information on this subject or 466 Once you know your version of Windows supports WSL 2, you can
475 instructions on how to upgrade to WSLv2 visit `Windows 10
476 WSLv2 <https://docs.microsoft.com/en-us/windows/wsl/wsl2-install>`__
477
4782. *Install the Linux distribution of your choice inside Windows 10:*
479 Once you know your version of Windows 10 supports WSLv2, you can
480 install the distribution of your choice from the Microsoft Store. 467 install the distribution of your choice from the Microsoft Store.
481 Open the Microsoft Store and search for Linux. While there are 468 Open the Microsoft Store and search for Linux. While there are
482 several Linux distributions available, the assumption is that your 469 several Linux distributions available, the assumption is that your
@@ -485,37 +472,33 @@ your Yocto Project build host:
485 making your selection, simply click "Get" to download and install the 472 making your selection, simply click "Get" to download and install the
486 distribution. 473 distribution.
487 474
4883. *Check your Linux distribution is using WSLv2:* Open a Windows 475#. *Check which Linux distribution WSL 2 is using:* Open a Windows
489 PowerShell and run: 476 PowerShell and run::
490 ::
491 477
492 C:\WINDOWS\system32> wsl -l -v 478 C:\WINDOWS\system32> wsl -l -v
493 NAME STATE VERSION 479 NAME STATE VERSION
494 *Ubuntu Running 2 480 *Ubuntu Running 2
495 481
496 Note the version column which says the WSL version 482 Note that WSL 2 supports running as many different Linux distributions
497 being used by your distribution, on compatible systems, this can be 483 as you want to install.
498 changed back at any point in time.
499 484
5004. *Optionally Orient Yourself on WSL:* If you are unfamiliar with WSL, 485#. *Optionally Get Familiar with WSL:* You can learn more on
501 you can learn more here -
502 https://docs.microsoft.com/en-us/windows/wsl/wsl2-about. 486 https://docs.microsoft.com/en-us/windows/wsl/wsl2-about.
503 487
5045. *Launch your WSL Distibution:* From the Windows start menu simply 488#. *Launch your WSL Distibution:* From the Windows start menu simply
505 launch your WSL distribution just like any other application. 489 launch your WSL distribution just like any other application.
506 490
5076. *Optimize your WSLv2 storage often:* Due to the way storage is 491#. *Optimize your WSL 2 storage often:* Due to the way storage is
508 handled on WSLv2, the storage space used by the undelying Linux 492 handled on WSL 2, the storage space used by the underlying Linux
509 distribution is not reflected immedately, and since bitbake heavily 493 distribution is not reflected immediately, and since BitBake heavily
510 uses storage, after several builds, you may be unaware you are 494 uses storage, after several builds, you may be unaware you are
511 running out of space. WSLv2 uses a VHDX file for storage, this issue 495 running out of space. As WSL 2 uses a VHDX file for storage, this issue
512 can be easily avoided by manually optimizing this file often, this 496 can be easily avoided by regularly optimizing this file in a manual way:
513 can be done in the following way:
514 497
515 1. *Find the location of your VHDX file:* First you need to find the 498 1. *Find the location of your VHDX file:*
516 distro app package directory, to achieve this open a Windows 499
517 Powershell as Administrator and run: 500 First you need to find the distro app package directory, to achieve this
518 :: 501 open a Windows Powershell as Administrator and run::
519 502
520 C:\WINDOWS\system32> Get-AppxPackage -Name "*Ubuntu*" | Select PackageFamilyName 503 C:\WINDOWS\system32> Get-AppxPackage -Name "*Ubuntu*" | Select PackageFamilyName
521 PackageFamilyName 504 PackageFamilyName
@@ -525,43 +508,62 @@ your Yocto Project build host:
525 508
526 You should now 509 You should now
527 replace the PackageFamilyName and your user on the following path 510 replace the PackageFamilyName and your user on the following path
528 to find your VHDX file: 511 to find your VHDX file::
529 ::
530 512
531 ls C:\Users\myuser\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79abcdefgh\LocalState\ 513 ls C:\Users\myuser\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79abcdefgh\LocalState\
532 Mode LastWriteTime Length Name 514 Mode LastWriteTime Length Name
533 -a---- 3/14/2020 9:52 PM 57418973184 ext4.vhdx 515 -a---- 3/14/2020 9:52 PM 57418973184 ext4.vhdx
534 516
535 Your VHDX file path is: 517 Your VHDX file path is:
536 ``C:\Users\myuser\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79abcdefgh\LocalState\ext4.vhdx`` 518 ``C:\Users\myuser\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79abcdefgh\LocalState\ext4.vhdx``
537 519
538 2. *Optimize your VHDX file:* Open a Windows Powershell as 520 2a. *Optimize your VHDX file using Windows Powershell:*
539 Administrator to optimize your VHDX file, shutting down WSL first: 521
540 :: 522 To use the ``optimize-vhd`` cmdlet below, first install the Hyper-V
523 option on Windows. Then, open a Windows Powershell as Administrator to
524 optimize your VHDX file, shutting down WSL first::
541 525
542 C:\WINDOWS\system32> wsl --shutdown 526 C:\WINDOWS\system32> wsl --shutdown
543 C:\WINDOWS\system32> optimize-vhd -Path C:\Users\myuser\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79abcdefgh\LocalState\ext4.vhdx -Mode full 527 C:\WINDOWS\system32> optimize-vhd -Path C:\Users\myuser\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79abcdefgh\LocalState\ext4.vhdx -Mode full
544 528
545 A progress bar should be shown while optimizing the 529 A progress bar should be shown while optimizing the
546 VHDX file, and storage should now be reflected correctly on the 530 VHDX file, and storage should now be reflected correctly on the
547 Windows Explorer. 531 Windows Explorer.
532
533 2b. *Optimize your VHDX file using DiskPart:*
534
535 The ``optimize-vhd`` cmdlet noted in step 2a above is provided by
536 Hyper-V. Not all SKUs of Windows can install Hyper-V. As an alternative,
537 use the DiskPart tool. To start, open a Windows command prompt as
538 Administrator to optimize your VHDX file, shutting down WSL first::
539
540 C:\WINDOWS\system32> wsl --shutdown
541 C:\WINDOWS\system32> diskpart
542
543 DISKPART> select vdisk file="<path_to_VHDX_file>"
544 DISKPART> attach vdisk readonly
545 DISKPART> compact vdisk
546 DISKPART> exit
548 547
549.. note:: 548.. note::
550 549
551 The current implementation of WSLv2 does not have out-of-the-box 550 The current implementation of WSL 2 does not have out-of-the-box
552 access to external devices such as those connected through a USB 551 access to external devices such as those connected through a USB
553 port, but it automatically mounts your ``C:`` drive on ``/mnt/c/`` 552 port, but it automatically mounts your ``C:`` drive on ``/mnt/c/``
554 (and others), which you can use to share deploy artifacts to be later 553 (and others), which you can use to share deploy artifacts to be later
555 flashed on hardware through Windows, but your build directory should 554 flashed on hardware through Windows, but your :term:`Build Directory`
556 not reside inside this mountpoint. 555 should not reside inside this mountpoint.
557 556
558Once you have WSLv2 set up, everything is in place to develop just as if 557Once you have WSL 2 set up, everything is in place to develop just as if
559you were running on a native Linux machine. If you are going to use the 558you were running on a native Linux machine. If you are going to use the
560Extensible SDK container, see the ":doc:`/sdk-manual/extensible`" Chapter in the Yocto 559Extensible SDK container, see the ":doc:`/sdk-manual/extensible`" Chapter in the Yocto
561Project Application Development and the Extensible Software Development 560Project Application Development and the Extensible Software Development
562Kit (eSDK) manual. If you are going to use the Toaster container, see 561Kit (eSDK) manual. If you are going to use the Toaster container, see
563the ":doc:`/toaster-manual/setup-and-use`" 562the ":doc:`/toaster-manual/setup-and-use`"
564section in the Toaster User Manual. 563section in the Toaster User Manual. If you are a VSCode user, you can configure
564the `Yocto Project BitBake
565<https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__
566extension accordingly.
565 567
566Locating Yocto Project Source Files 568Locating Yocto Project Source Files
567=================================== 569===================================
@@ -591,14 +593,14 @@ repository at :yocto_git:`/poky`.
591Use the following procedure to locate the latest upstream copy of the 593Use the following procedure to locate the latest upstream copy of the
592``poky`` Git repository: 594``poky`` Git repository:
593 595
5941. *Access Repositories:* Open a browser and go to 596#. *Access Repositories:* Open a browser and go to
595 :yocto_git:`/` to access the GUI-based interface into the 597 :yocto_git:`/` to access the GUI-based interface into the
596 Yocto Project source repositories. 598 Yocto Project source repositories.
597 599
5982. *Select the Repository:* Click on the repository in which you are 600#. *Select the Repository:* Click on the repository in which you are
599 interested (e.g. ``poky``). 601 interested (e.g. ``poky``).
600 602
6013. *Find the URL Used to Clone the Repository:* At the bottom of the 603#. *Find the URL Used to Clone the Repository:* At the bottom of the
602 page, note the URL used to clone that repository 604 page, note the URL used to clone that repository
603 (e.g. :yocto_git:`/poky`). 605 (e.g. :yocto_git:`/poky`).
604 606
@@ -607,73 +609,43 @@ Use the following procedure to locate the latest upstream copy of the
607 For information on cloning a repository, see the 609 For information on cloning a repository, see the
608 ":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`" section. 610 ":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`" section.
609 611
610Accessing Index of Releases 612Accessing Source Archives
611--------------------------- 613-------------------------
614
615The Yocto Project also provides source archives of its releases, which
616are available on :yocto_dl:`/releases/yocto/`. Then, choose the subdirectory
617containing the release you wish to use, for example
618:yocto_dl:`yocto-&DISTRO; </releases/yocto/yocto-&DISTRO;/>`.
612 619
613Yocto Project maintains an Index of Releases area that contains related 620You will find there source archives of individual components (if you wish
614files that contribute to the Yocto Project. Rather than Git 621to use them individually), and of the corresponding Poky release bundling
615repositories, these files are tarballs that represent snapshots in time 622a selection of these components.
616of a given component.
617 623
618.. note:: 624.. note::
619 625
620 The recommended method for accessing Yocto Project components is to 626 The recommended method for accessing Yocto Project components is to
621 use Git to clone the upstream repository and work from within that 627 use Git to clone the upstream repository and work from within that
622 locally cloned repository. The procedure in this section exists 628 locally cloned repository.
623 should you desire a tarball snapshot of any given component.
624
625Follow these steps to locate and download a particular tarball:
626
6271. *Access the Index of Releases:* Open a browser and go to
628 :yocto_dl:`Index of Releases </releases>`. The
629 list represents released components (e.g. ``bitbake``, ``sato``, and
630 so on).
631
632 .. note::
633
634 The ``yocto`` directory contains the full array of released Poky
635 tarballs. The ``poky`` directory in the Index of Releases was
636 historically used for very early releases and exists now only for
637 retroactive completeness.
638
6392. *Select a Component:* Click on any released component in which you
640 are interested (e.g. ``yocto``).
641
6423. *Find the Tarball:* Drill down to find the associated tarball. For
643 example, click on ``yocto-&DISTRO;`` to view files associated with the
644 Yocto Project &DISTRO; release (e.g.
645 ``&YOCTO_POKY;.tar.bz2``, which is the
646 released Poky tarball).
647
6484. *Download the Tarball:* Click the tarball to download and save a
649 snapshot of the given component.
650 629
651Using the Downloads Page 630Using the Downloads Page
652------------------------ 631------------------------
653 632
654The :yocto_home:`Yocto Project Website <>` uses a "DOWNLOADS" page 633The :yocto_home:`Yocto Project Website <>` uses a "RELEASES" page
655from which you can locate and download tarballs of any Yocto Project 634from which you can locate and download tarballs of any Yocto Project
656release. Rather than Git repositories, these files represent snapshot 635release. Rather than Git repositories, these files represent snapshot
657tarballs similar to the tarballs located in the Index of Releases 636tarballs similar to the tarballs located in the Index of Releases
658described in the "`Accessing Index of 637described in the ":ref:`dev-manual/start:accessing source archives`" section.
659Releases <#accessing-index-of-releases>`__" section.
660
661.. note::
662
663 The recommended method for accessing Yocto Project components is to
664 use Git to clone a repository and work from within that local
665 repository. The procedure in this section exists should you desire a
666 tarball snapshot of any given component.
667 638
6681. *Go to the Yocto Project Website:* Open The 639#. *Go to the Yocto Project Website:* Open The
669 :yocto_home:`Yocto Project Website <>` in your browser. 640 :yocto_home:`Yocto Project Website <>` in your browser.
670 641
6712. *Get to the Downloads Area:* Select the "DOWNLOADS" item from the 642#. *Get to the Downloads Area:* Select the "RELEASES" item from the
672 pull-down "SOFTWARE" tab menu near the top of the page. 643 pull-down "DEVELOPMENT" tab menu near the top of the page.
673 644
6743. *Select a Yocto Project Release:* Use the menu next to "RELEASE" to 645#. *Select a Yocto Project Release:* On the top of the "RELEASE" page currently
675 display and choose a recent or past supported Yocto Project release 646 supported releases are displayed, further down past supported Yocto Project
676 (e.g. &DISTRO_NAME_NO_CAP;, &DISTRO_NAME_NO_CAP_MINUS_ONE;, and so forth). 647 releases are visible. The "Download" links in the rows of the table there
648 will lead to the download tarballs for the release.
677 649
678 .. note:: 650 .. note::
679 651
@@ -683,35 +655,9 @@ Releases <#accessing-index-of-releases>`__" section.
683 You can use the "RELEASE ARCHIVE" link to reveal a menu of all Yocto 655 You can use the "RELEASE ARCHIVE" link to reveal a menu of all Yocto
684 Project releases. 656 Project releases.
685 657
6864. *Download Tools or Board Support Packages (BSPs):* From the 658#. *Download Tools or Board Support Packages (BSPs):* Next to the tarballs you
687 "DOWNLOADS" page, you can download tools or BSPs as well. Just scroll 659 will find download tools or BSPs as well. Just select a Yocto Project
688 down the page and look for what you need. 660 release and look for what you need.
689
690Accessing Nightly Builds
691------------------------
692
693Yocto Project maintains an area for nightly builds that contains tarball
694releases at https://autobuilder.yocto.io//pub/nightly/. These builds include Yocto
695Project releases ("poky"), toolchains, and builds for supported
696machines.
697
698Should you ever want to access a nightly build of a particular Yocto
699Project component, use the following procedure:
700
7011. *Locate the Index of Nightly Builds:* Open a browser and go to
702 https://autobuilder.yocto.io//pub/nightly/ to access the Nightly Builds.
703
7042. *Select a Date:* Click on the date in which you are interested. If
705 you want the latest builds, use "CURRENT".
706
7073. *Select a Build:* Choose the area in which you are interested. For
708 example, if you are looking for the most recent toolchains, select
709 the "toolchain" link.
710
7114. *Find the Tarball:* Drill down to find the associated tarball.
712
7135. *Download the Tarball:* Click the tarball to download and save a
714 snapshot of the given component.
715 661
716Cloning and Checking Out Branches 662Cloning and Checking Out Branches
717================================= 663=================================
@@ -737,13 +683,12 @@ Cloning the ``poky`` Repository
737Follow these steps to create a local version of the upstream 683Follow these steps to create a local version of the upstream
738:term:`Poky` Git repository. 684:term:`Poky` Git repository.
739 685
7401. *Set Your Directory:* Change your working directory to where you want 686#. *Set Your Directory:* Change your working directory to where you want
741 to create your local copy of ``poky``. 687 to create your local copy of ``poky``.
742 688
7432. *Clone the Repository:* The following example command clones the 689#. *Clone the Repository:* The following example command clones the
744 ``poky`` repository and uses the default name "poky" for your local 690 ``poky`` repository and uses the default name "poky" for your local
745 repository: 691 repository::
746 ::
747 692
748 $ git clone git://git.yoctoproject.org/poky 693 $ git clone git://git.yoctoproject.org/poky
749 Cloning into 'poky'... 694 Cloning into 'poky'...
@@ -759,16 +704,15 @@ Follow these steps to create a local version of the upstream
759 "master" branch, which results in a snapshot of the latest 704 "master" branch, which results in a snapshot of the latest
760 development changes for "master". For information on how to check out 705 development changes for "master". For information on how to check out
761 a specific development branch or on how to check out a local branch 706 a specific development branch or on how to check out a local branch
762 based on a tag name, see the "`Checking Out By Branch in 707 based on a tag name, see the
763 Poky <#checking-out-by-branch-in-poky>`__" and `Checking Out By Tag 708 ":ref:`dev-manual/start:checking out by branch in poky`" and
764 in Poky <#checkout-out-by-tag-in-poky>`__" sections, respectively. 709 ":ref:`dev-manual/start:checking out by tag in poky`" sections, respectively.
765 710
766 Once the local repository is created, you can change to that 711 Once the local repository is created, you can change to that
767 directory and check its status. Here, the single "master" branch 712 directory and check its status. The ``master`` branch is checked out
768 exists on your system and by default, it is checked out: 713 by default::
769 ::
770 714
771 $ cd ~/poky 715 $ cd poky
772 $ git status 716 $ git status
773 On branch master 717 On branch master
774 Your branch is up-to-date with 'origin/master'. 718 Your branch is up-to-date with 'origin/master'.
@@ -798,13 +742,13 @@ and then specifically check out that development branch.
798 Further development on top of the branch that occurs after check it 742 Further development on top of the branch that occurs after check it
799 out can occur. 743 out can occur.
800 744
8011. *Switch to the Poky Directory:* If you have a local poky Git 745#. *Switch to the Poky Directory:* If you have a local poky Git
802 repository, switch to that directory. If you do not have the local 746 repository, switch to that directory. If you do not have the local
803 copy of poky, see the 747 copy of poky, see the
804 ":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`" 748 ":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`"
805 section. 749 section.
806 750
8072. *Determine Existing Branch Names:* 751#. *Determine Existing Branch Names:*
808 :: 752 ::
809 753
810 $ git branch -a 754 $ git branch -a
@@ -825,10 +769,9 @@ and then specifically check out that development branch.
825 remotes/origin/zeus-next 769 remotes/origin/zeus-next
826 ... and so on ... 770 ... and so on ...
827 771
8283. *Check out the Branch:* Check out the development branch in which you 772#. *Check out the Branch:* Check out the development branch in which you
829 want to work. For example, to access the files for the Yocto Project 773 want to work. For example, to access the files for the Yocto Project
830 &DISTRO; Release (&DISTRO_NAME;), use the following command: 774 &DISTRO; Release (&DISTRO_NAME;), use the following command::
831 ::
832 775
833 $ git checkout -b &DISTRO_NAME_NO_CAP; origin/&DISTRO_NAME_NO_CAP; 776 $ git checkout -b &DISTRO_NAME_NO_CAP; origin/&DISTRO_NAME_NO_CAP;
834 Branch &DISTRO_NAME_NO_CAP; set up to track remote branch &DISTRO_NAME_NO_CAP; from origin. 777 Branch &DISTRO_NAME_NO_CAP; set up to track remote branch &DISTRO_NAME_NO_CAP; from origin.
@@ -840,8 +783,7 @@ and then specifically check out that development branch.
840 783
841 The following command displays the branches that are now part of your 784 The following command displays the branches that are now part of your
842 local poky repository. The asterisk character indicates the branch 785 local poky repository. The asterisk character indicates the branch
843 that is currently checked out for work: 786 that is currently checked out for work::
844 ::
845 787
846 $ git branch 788 $ git branch
847 master 789 master
@@ -861,21 +803,19 @@ similar to checking out by branch name except you use tag names.
861 Checking out a branch based on a tag gives you a stable set of files 803 Checking out a branch based on a tag gives you a stable set of files
862 not affected by development on the branch above the tag. 804 not affected by development on the branch above the tag.
863 805
8641. *Switch to the Poky Directory:* If you have a local poky Git 806#. *Switch to the Poky Directory:* If you have a local poky Git
865 repository, switch to that directory. If you do not have the local 807 repository, switch to that directory. If you do not have the local
866 copy of poky, see the 808 copy of poky, see the
867 ":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`" 809 ":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`"
868 section. 810 section.
869 811
8702. *Fetch the Tag Names:* To checkout the branch based on a tag name, 812#. *Fetch the Tag Names:* To checkout the branch based on a tag name,
871 you need to fetch the upstream tags into your local repository: 813 you need to fetch the upstream tags into your local repository::
872 ::
873 814
874 $ git fetch --tags 815 $ git fetch --tags
875 $ 816 $
876 817
8773. *List the Tag Names:* You can list the tag names now: 818#. *List the Tag Names:* You can list the tag names now::
878 ::
879 819
880 $ git tag 820 $ git tag
881 1.1_M1.final 821 1.1_M1.final
@@ -897,7 +837,7 @@ similar to checking out by branch name except you use tag names.
897 yocto_1.5_M5.rc8 837 yocto_1.5_M5.rc8
898 838
899 839
9004. *Check out the Branch:* 840#. *Check out the Branch:*
901 :: 841 ::
902 842
903 $ git checkout tags/yocto-&DISTRO; -b my_yocto_&DISTRO; 843 $ git checkout tags/yocto-&DISTRO; -b my_yocto_&DISTRO;
diff --git a/documentation/dev-manual/temporary-source-code.rst b/documentation/dev-manual/temporary-source-code.rst
new file mode 100644
index 0000000000..08bf68d982
--- /dev/null
+++ b/documentation/dev-manual/temporary-source-code.rst
@@ -0,0 +1,66 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Finding Temporary Source Code
4*****************************
5
6You might find it helpful during development to modify the temporary
7source code used by recipes to build packages. For example, suppose you
8are developing a patch and you need to experiment a bit to figure out
9your solution. After you have initially built the package, you can
10iteratively tweak the source code, which is located in the
11:term:`Build Directory`, and then you can force a re-compile and quickly
12test your altered code. Once you settle on a solution, you can then preserve
13your changes in the form of patches.
14
15During a build, the unpacked temporary source code used by recipes to
16build packages is available in the :term:`Build Directory` as defined by the
17:term:`S` variable. Below is the default value for the :term:`S` variable as
18defined in the ``meta/conf/bitbake.conf`` configuration file in the
19:term:`Source Directory`::
20
21 S = "${WORKDIR}/${BP}"
22
23You should be aware that many recipes override the
24:term:`S` variable. For example, recipes that fetch their source from Git
25usually set :term:`S` to ``${WORKDIR}/git``.
26
27.. note::
28
29 The :term:`BP` represents the base recipe name, which consists of the name
30 and version::
31
32 BP = "${BPN}-${PV}"
33
34
35The path to the work directory for the recipe
36(:term:`WORKDIR`) is defined as
37follows::
38
39 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
40
41The actual directory depends on several things:
42
43- :term:`TMPDIR`: The top-level build
44 output directory.
45
46- :term:`MULTIMACH_TARGET_SYS`:
47 The target system identifier.
48
49- :term:`PN`: The recipe name.
50
51- :term:`EXTENDPE`: The epoch --- if
52 :term:`PE` is not specified, which is
53 usually the case for most recipes, then :term:`EXTENDPE` is blank.
54
55- :term:`PV`: The recipe version.
56
57- :term:`PR`: The recipe revision.
58
59As an example, assume a Source Directory top-level folder named
60``poky``, a default :term:`Build Directory` at ``poky/build``, and a
61``qemux86-poky-linux`` machine target system. Furthermore, suppose your
62recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
63build system uses to build the package would be as follows::
64
65 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
66
diff --git a/documentation/dev-manual/upgrading-recipes.rst b/documentation/dev-manual/upgrading-recipes.rst
new file mode 100644
index 0000000000..4fac78bdfb
--- /dev/null
+++ b/documentation/dev-manual/upgrading-recipes.rst
@@ -0,0 +1,397 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Upgrading Recipes
4*****************
5
6Over time, upstream developers publish new versions for software built
7by layer recipes. It is recommended to keep recipes up-to-date with
8upstream version releases.
9
10While there are several methods to upgrade a recipe, you might
11consider checking on the upgrade status of a recipe first. You can do so
12using the ``devtool check-upgrade-status`` command. See the
13":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
14section in the Yocto Project Reference Manual for more information.
15
16The remainder of this section describes three ways you can upgrade a
17recipe. You can use the Automated Upgrade Helper (AUH) to set up
18automatic version upgrades. Alternatively, you can use
19``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
20you can manually upgrade a recipe by editing the recipe itself.
21
22Using the Auto Upgrade Helper (AUH)
23===================================
24
25The AUH utility works in conjunction with the OpenEmbedded build system
26in order to automatically generate upgrades for recipes based on new
27versions being published upstream. Use AUH when you want to create a
28service that performs the upgrades automatically and optionally sends
29you an email with the results.
30
31AUH allows you to update several recipes with a single use. You can also
32optionally perform build and integration tests using images with the
33results saved to your hard drive and emails of results optionally sent
34to recipe maintainers. Finally, AUH creates Git commits with appropriate
35commit messages in the layer's tree for the changes made to recipes.
36
37.. note::
38
39 In some conditions, you should not use AUH to upgrade recipes
40 and should instead use either ``devtool upgrade`` or upgrade your
41 recipes manually:
42
43 - When AUH cannot complete the upgrade sequence. This situation
44 usually results because custom patches carried by the recipe
45 cannot be automatically rebased to the new version. In this case,
46 ``devtool upgrade`` allows you to manually resolve conflicts.
47
48 - When for any reason you want fuller control over the upgrade
49 process. For example, when you want special arrangements for
50 testing.
51
52The following steps describe how to set up the AUH utility:
53
54#. *Be Sure the Development Host is Set Up:* You need to be sure that
55 your development host is set up to use the Yocto Project. For
56 information on how to set up your host, see the
57 ":ref:`dev-manual/start:Preparing the Build Host`" section.
58
59#. *Make Sure Git is Configured:* The AUH utility requires Git to be
60 configured because AUH uses Git to save upgrades. Thus, you must have
61 Git user and email configured. The following command shows your
62 configurations::
63
64 $ git config --list
65
66 If you do not have the user and
67 email configured, you can use the following commands to do so::
68
69 $ git config --global user.name some_name
70 $ git config --global user.email username@domain.com
71
72#. *Clone the AUH Repository:* To use AUH, you must clone the repository
73 onto your development host. The following command uses Git to create
74 a local copy of the repository on your system::
75
76 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
77 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
78 remote: Compressing objects: 100% (300/300), done.
79 remote: Total 768 (delta 499), reused 703 (delta 434)
80 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
81 Resolving deltas: 100% (499/499), done.
82 Checking connectivity... done.
83
84 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
85 :term:`Poky` repositories.
86
87#. *Create a Dedicated Build Directory:* Run the :ref:`structure-core-script`
88 script to create a fresh :term:`Build Directory` that you use exclusively
89 for running the AUH utility::
90
91 $ cd poky
92 $ source oe-init-build-env your_AUH_build_directory
93
94 Re-using an existing :term:`Build Directory` and its configurations is not
95 recommended as existing settings could cause AUH to fail or behave
96 undesirably.
97
98#. *Make Configurations in Your Local Configuration File:* Several
99 settings are needed in the ``local.conf`` file in the build
100 directory you just created for AUH. Make these following
101 configurations:
102
103 - If you want to enable :ref:`Build
104 History <dev-manual/build-quality:maintaining build output quality>`,
105 which is optional, you need the following lines in the
106 ``conf/local.conf`` file::
107
108 INHERIT =+ "buildhistory"
109 BUILDHISTORY_COMMIT = "1"
110
111 With this configuration and a successful
112 upgrade, a build history "diff" file appears in the
113 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
114 your :term:`Build Directory`.
115
116 - If you want to enable testing through the :ref:`ref-classes-testimage`
117 class, which is optional, you need to have the following set in
118 your ``conf/local.conf`` file::
119
120 IMAGE_CLASSES += "testimage"
121
122 .. note::
123
124 If your distro does not enable by default ptest, which Poky
125 does, you need the following in your ``local.conf`` file::
126
127 DISTRO_FEATURES:append = " ptest"
128
129
130#. *Optionally Start a vncserver:* If you are running in a server
131 without an X11 session, you need to start a vncserver::
132
133 $ vncserver :1
134 $ export DISPLAY=:1
135
136#. *Create and Edit an AUH Configuration File:* You need to have the
137 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
138 :term:`Build Directory`. You can find a sample configuration file in the
139 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
140
141 Read through the sample file and make configurations as needed. For
142 example, if you enabled build history in your ``local.conf`` as
143 described earlier, you must enable it in ``upgrade-helper.conf``.
144
145 Also, if you are using the default ``maintainers.inc`` file supplied
146 with Poky and located in ``meta-yocto`` and you do not set a
147 "maintainers_whitelist" or "global_maintainer_override" in the
148 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
149 the AUH command-line, the utility automatically sends out emails to
150 all the default maintainers. Please avoid this.
151
152This next set of examples describes how to use the AUH:
153
154- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
155 following form::
156
157 $ upgrade-helper.py recipe_name
158
159 For example, this command upgrades the ``xmodmap`` recipe::
160
161 $ upgrade-helper.py xmodmap
162
163- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
164 specific recipe to a particular version, use the following form::
165
166 $ upgrade-helper.py recipe_name -t version
167
168 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
169
170 $ upgrade-helper.py xmodmap -t 1.2.3
171
172- *Upgrading all Recipes to the Latest Versions and Suppressing Email
173 Notifications:* To upgrade all recipes to their most recent versions
174 and suppress the email notifications, use the following command::
175
176 $ upgrade-helper.py all
177
178- *Upgrading all Recipes to the Latest Versions and Send Email
179 Notifications:* To upgrade all recipes to their most recent versions
180 and send email messages to maintainers for each attempted recipe as
181 well as a status email, use the following command::
182
183 $ upgrade-helper.py -e all
184
185Once you have run the AUH utility, you can find the results in the AUH
186:term:`Build Directory`::
187
188 ${BUILDDIR}/upgrade-helper/timestamp
189
190The AUH utility
191also creates recipe update commits from successful upgrade attempts in
192the layer tree.
193
194You can easily set up to run the AUH utility on a regular basis by using
195a cron job. See the
196:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
197file distributed with the utility for an example.
198
199Using ``devtool upgrade``
200=========================
201
202As mentioned earlier, an alternative method for upgrading recipes to
203newer versions is to use
204:doc:`devtool upgrade </ref-manual/devtool-reference>`.
205You can read about ``devtool upgrade`` in general in the
206":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
207section in the Yocto Project Application Development and the Extensible
208Software Development Kit (eSDK) Manual.
209
210To see all the command-line options available with ``devtool upgrade``,
211use the following help command::
212
213 $ devtool upgrade -h
214
215If you want to find out what version a recipe is currently at upstream
216without any attempt to upgrade your local version of the recipe, you can
217use the following command::
218
219 $ devtool latest-version recipe_name
220
221As mentioned in the previous section describing AUH, ``devtool upgrade``
222works in a less-automated manner than AUH. Specifically,
223``devtool upgrade`` only works on a single recipe that you name on the
224command line, cannot perform build and integration testing using images,
225and does not automatically generate commits for changes in the source
226tree. Despite all these "limitations", ``devtool upgrade`` updates the
227recipe file to the new upstream version and attempts to rebase custom
228patches contained by the recipe as needed.
229
230.. note::
231
232 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
233 of a "wrapper" application for ``devtool upgrade``.
234
235A typical scenario involves having used Git to clone an upstream
236repository that you use during build operations. Because you have built the
237recipe in the past, the layer is likely added to your
238configuration already. If for some reason, the layer is not added, you
239could add it easily using the
240":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
241script. For example, suppose you use the ``nano.bb`` recipe from the
242``meta-oe`` layer in the ``meta-openembedded`` repository. For this
243example, assume that the layer has been cloned into following area::
244
245 /home/scottrif/meta-openembedded
246
247The following command from your :term:`Build Directory` adds the layer to
248your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
249
250 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
251 NOTE: Starting bitbake server...
252 Parsing recipes: 100% |##########################################| Time: 0:00:55
253 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
254 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
255 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
256 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
257 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
258
259For this example, assume that the ``nano.bb`` recipe that
260is upstream has a 2.9.3 version number. However, the version in the
261local repository is 2.7.4. The following command from your build
262directory automatically upgrades the recipe for you::
263
264 $ devtool upgrade nano -V 2.9.3
265 NOTE: Starting bitbake server...
266 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
267 Parsing recipes: 100% |##########################################| Time: 0:00:46
268 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
269 NOTE: Extracting current version source...
270 NOTE: Resolving any missing task queue dependencies
271 .
272 .
273 .
274 NOTE: Executing SetScene Tasks
275 NOTE: Executing RunQueue Tasks
276 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
277 Adding changed files: 100% |#####################################| Time: 0:00:00
278 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
279 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
280
281.. note::
282
283 Using the ``-V`` option is not necessary. Omitting the version number causes
284 ``devtool upgrade`` to upgrade the recipe to the most recent version.
285
286Continuing with this example, you can use ``devtool build`` to build the
287newly upgraded recipe::
288
289 $ devtool build nano
290 NOTE: Starting bitbake server...
291 Loading cache: 100% |################################################################################################| Time: 0:00:01
292 Loaded 2040 entries from dependency cache.
293 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
294 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
295 NOTE: Resolving any missing task queue dependencies
296 .
297 .
298 .
299 NOTE: Executing SetScene Tasks
300 NOTE: Executing RunQueue Tasks
301 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
302 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
303
304Within the ``devtool upgrade`` workflow, you can
305deploy and test your rebuilt software. For this example,
306however, running ``devtool finish`` cleans up the workspace once the
307source in your workspace is clean. This usually means using Git to stage
308and submit commits for the changes generated by the upgrade process.
309
310Once the tree is clean, you can clean things up in this example with the
311following command from the ``${BUILDDIR}/workspace/sources/nano``
312directory::
313
314 $ devtool finish nano meta-oe
315 NOTE: Starting bitbake server...
316 Loading cache: 100% |################################################################################################| Time: 0:00:00
317 Loaded 2040 entries from dependency cache.
318 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
319 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
320 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
321 NOTE: Updating recipe nano_2.9.3.bb
322 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
323 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
324 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
325
326
327Using the ``devtool finish`` command cleans up the workspace and creates a patch
328file based on your commits. The tool puts all patch files back into the
329source directory in a sub-directory named ``nano`` in this case.
330
331Manually Upgrading a Recipe
332===========================
333
334If for some reason you choose not to upgrade recipes using
335:ref:`dev-manual/upgrading-recipes:Using the Auto Upgrade Helper (AUH)` or
336by :ref:`dev-manual/upgrading-recipes:Using \`\`devtool upgrade\`\``,
337you can manually edit the recipe files to upgrade the versions.
338
339.. note::
340
341 Manually updating multiple recipes scales poorly and involves many
342 steps. The recommendation to upgrade recipe versions is through AUH
343 or ``devtool upgrade``, both of which automate some steps and provide
344 guidance for others needed for the manual process.
345
346To manually upgrade recipe versions, follow these general steps:
347
348#. *Change the Version:* Rename the recipe such that the version (i.e.
349 the :term:`PV` part of the recipe name)
350 changes appropriately. If the version is not part of the recipe name,
351 change the value as it is set for :term:`PV` within the recipe itself.
352
353#. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
354 is fetched from Git or some other version control system, update
355 :term:`SRCREV` to point to the
356 commit hash that matches the new version.
357
358#. *Build the Software:* Try to build the recipe using BitBake. Typical
359 build failures include the following:
360
361 - License statements were updated for the new version. For this
362 case, you need to review any changes to the license and update the
363 values of :term:`LICENSE` and
364 :term:`LIC_FILES_CHKSUM`
365 as needed.
366
367 .. note::
368
369 License changes are often inconsequential. For example, the
370 license text's copyright year might have changed.
371
372 - Custom patches carried by the older version of the recipe might
373 fail to apply to the new version. For these cases, you need to
374 review the failures. Patches might not be necessary for the new
375 version of the software if the upgraded version has fixed those
376 issues. If a patch is necessary and failing, you need to rebase it
377 into the new version.
378
379#. *Optionally Attempt to Build for Several Architectures:* Once you
380 successfully build the new software for a given architecture, you
381 could test the build for other architectures by changing the
382 :term:`MACHINE` variable and
383 rebuilding the software. This optional step is especially important
384 if the recipe is to be released publicly.
385
386#. *Check the Upstream Change Log or Release Notes:* Checking both these
387 reveals if there are new features that could break
388 backwards-compatibility. If so, you need to take steps to mitigate or
389 eliminate that situation.
390
391#. *Optionally Create a Bootable Image and Test:* If you want, you can
392 test the new software by booting it onto actual hardware.
393
394#. *Create a Commit with the Change in the Layer Repository:* After all
395 builds work and any testing is successful, you can create commits for
396 any changes in the layer holding your upgraded recipe.
397
diff --git a/documentation/dev-manual/vulnerabilities.rst b/documentation/dev-manual/vulnerabilities.rst
new file mode 100644
index 0000000000..1bc2a85929
--- /dev/null
+++ b/documentation/dev-manual/vulnerabilities.rst
@@ -0,0 +1,293 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Checking for Vulnerabilities
4****************************
5
6Vulnerabilities in Poky and OE-Core
7===================================
8
9The Yocto Project has an infrastructure to track and address unfixed
10known security vulnerabilities, as tracked by the public
11:wikipedia:`Common Vulnerabilities and Exposures (CVE) <Common_Vulnerabilities_and_Exposures>`
12database.
13
14The Yocto Project maintains a `list of known vulnerabilities
15<https://autobuilder.yocto.io/pub/non-release/patchmetrics/>`__
16for packages in Poky and OE-Core, tracking the evolution of the number of
17unpatched CVEs and the status of patches. Such information is available for
18the current development version and for each supported release.
19
20Security is a process, not a product, and thus at any time, a number of security
21issues may be impacting Poky and OE-Core. It is up to the maintainers, users,
22contributors and anyone interested in the issues to investigate and possibly fix them by
23updating software components to newer versions or by applying patches to address them.
24It is recommended to work with Poky and OE-Core upstream maintainers and submit
25patches to fix them, see ":doc:`../contributor-guide/submit-changes`" for details.
26
27Vulnerability check at build time
28=================================
29
30To enable a check for CVE security vulnerabilities using
31:ref:`ref-classes-cve-check` in the specific image or target you are building,
32add the following setting to your configuration::
33
34 INHERIT += "cve-check"
35
36The CVE database contains some old incomplete entries which have been
37deemed not to impact Poky or OE-Core. These CVE entries can be excluded from the
38check using build configuration::
39
40 include conf/distro/include/cve-extra-exclusions.inc
41
42With this CVE check enabled, BitBake build will try to map each compiled software component
43recipe name and version information to the CVE database and generate recipe and
44image specific reports. These reports will contain:
45
46- metadata about the software component like names and versions
47
48- metadata about the CVE issue such as description and NVD link
49
50- for each software component, a list of CVEs which are possibly impacting this version
51
52- status of each CVE: ``Patched``, ``Unpatched`` or ``Ignored``
53
54The status ``Patched`` means that a patch file to address the security issue has been
55applied. ``Unpatched`` status means that no patches to address the issue have been
56applied and that the issue needs to be investigated. ``Ignored`` means that after
57analysis, it has been deemed to ignore the issue as it for example affects
58the software component on a different operating system platform.
59
60After a build with CVE check enabled, reports for each compiled source recipe will be
61found in ``build/tmp/deploy/cve``.
62
63For example the CVE check report for the ``flex-native`` recipe looks like::
64
65 $ cat poky/build/tmp/deploy/cve/flex-native
66 LAYER: meta
67 PACKAGE NAME: flex-native
68 PACKAGE VERSION: 2.6.4
69 CVE: CVE-2016-6354
70 CVE STATUS: Patched
71 CVE SUMMARY: Heap-based buffer overflow in the yy_get_next_buffer function in Flex before 2.6.1 might allow context-dependent attackers to cause a denial of service or possibly execute arbitrary code via vectors involving num_to_read.
72 CVSS v2 BASE SCORE: 7.5
73 CVSS v3 BASE SCORE: 9.8
74 VECTOR: NETWORK
75 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2016-6354
76
77 LAYER: meta
78 PACKAGE NAME: flex-native
79 PACKAGE VERSION: 2.6.4
80 CVE: CVE-2019-6293
81 CVE STATUS: Ignored
82 CVE SUMMARY: An issue was discovered in the function mark_beginning_as_normal in nfa.c in flex 2.6.4. There is a stack exhaustion problem caused by the mark_beginning_as_normal function making recursive calls to itself in certain scenarios involving lots of '*' characters. Remote attackers could leverage this vulnerability to cause a denial-of-service.
83 CVSS v2 BASE SCORE: 4.3
84 CVSS v3 BASE SCORE: 5.5
85 VECTOR: NETWORK
86 MORE INFORMATION: https://nvd.nist.gov/vuln/detail/CVE-2019-6293
87
88For images, a summary of all recipes included in the image and their CVEs is also
89generated in textual and JSON formats. These ``.cve`` and ``.json`` reports can be found
90in the ``tmp/deploy/images`` directory for each compiled image.
91
92At build time CVE check will also throw warnings about ``Unpatched`` CVEs::
93
94 WARNING: flex-2.6.4-r0 do_cve_check: Found unpatched CVE (CVE-2019-6293), for more information check /poky/build/tmp/work/core2-64-poky-linux/flex/2.6.4-r0/temp/cve.log
95 WARNING: libarchive-3.5.1-r0 do_cve_check: Found unpatched CVE (CVE-2021-36976), for more information check /poky/build/tmp/work/core2-64-poky-linux/libarchive/3.5.1-r0/temp/cve.log
96
97It is also possible to check the CVE status of individual packages as follows::
98
99 bitbake -c cve_check flex libarchive
100
101Fixing CVE product name and version mappings
102============================================
103
104By default, :ref:`ref-classes-cve-check` uses the recipe name :term:`BPN` as CVE
105product name when querying the CVE database. If this mapping contains false positives, e.g.
106some reported CVEs are not for the software component in question, or false negatives like
107some CVEs are not found to impact the recipe when they should, then the problems can be
108in the recipe name to CVE product mapping. These mapping issues can be fixed by setting
109the :term:`CVE_PRODUCT` variable inside the recipe. This defines the name of the software component in the
110upstream `NIST CVE database <https://nvd.nist.gov/>`__.
111
112The variable supports using vendor and product names like this::
113
114 CVE_PRODUCT = "flex_project:flex"
115
116In this example the vendor name used in the CVE database is ``flex_project`` and the
117product is ``flex``. With this setting the ``flex`` recipe only maps to this specific
118product and not products from other vendors with same name ``flex``.
119
120Similarly, when the recipe version :term:`PV` is not compatible with software versions used by
121the upstream software component releases and the CVE database, these can be fixed using
122the :term:`CVE_VERSION` variable.
123
124Note that if the CVE entries in the NVD database contain bugs or have missing or incomplete
125information, it is recommended to fix the information there directly instead of working
126around the issues possibly for a long time in Poky and OE-Core side recipes. Feedback to
127NVD about CVE entries can be provided through the `NVD contact form <https://nvd.nist.gov/info/contact-form>`__.
128
129Fixing vulnerabilities in recipes
130=================================
131
132Suppose a CVE security issue impacts a software component. In that case, it can
133be fixed by updating to a newer version, by applying a patch, or by marking it
134as patched via :term:`CVE_STATUS` variable flag. For Poky and OE-Core master
135branches, updating to a more recent software component release with fixes is
136the best option, but patches can be applied if releases are not yet available.
137
138For stable branches, we want to avoid API (Application Programming Interface)
139or ABI (Application Binary Interface) breakages. When submitting an update,
140a minor version update of a component is preferred if the version is
141backward-compatible. Many software components have backward-compatible stable
142versions, with a notable example of the Linux kernel. However, if the new
143version does or likely might introduce incompatibilities, extracting and
144backporting patches is preferred.
145
146Here is an example of fixing CVE security issues with patch files,
147an example from the :oe_layerindex:`ffmpeg recipe for dunfell </layerindex/recipe/122174>`::
148
149 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
150 file://mips64_cpu_detection.patch \
151 file://CVE-2020-12284.patch \
152 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \
153 file://CVE-2021-3566.patch \
154 file://CVE-2021-38291.patch \
155 file://CVE-2022-1475.patch \
156 file://CVE-2022-3109.patch \
157 file://CVE-2022-3341.patch \
158 file://CVE-2022-48434.patch \
159 "
160
161The recipe has both generic and security-related fixes. The CVE patch files are named
162according to the CVE they fix.
163
164When preparing the patch file, take the original patch from the upstream repository.
165Do not use patches from different distributions, except if it is the only available source.
166
167Modify the patch adding OE-related metadata. We will follow the example of the
168``CVE-2022-3341.patch``.
169
170The original `commit message <https://github.com/FFmpeg/FFmpeg/commit/9cf652cef49d74afe3d454f27d49eb1a1394951e.patch/>`__
171is::
172
173 From 9cf652cef49d74afe3d454f27d49eb1a1394951e Mon Sep 17 00:00:00 2001
174 From: Jiasheng Jiang <jiasheng@iscas.ac.cn>
175 Date: Wed, 23 Feb 2022 10:31:59 +0800
176 Subject: [PATCH] avformat/nutdec: Add check for avformat_new_stream
177
178 Check for failure of avformat_new_stream() and propagate
179 the error code.
180
181 Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
182 ---
183 libavformat/nutdec.c | 16 ++++++++++++----
184 1 file changed, 12 insertions(+), 4 deletions(-)
185
186
187For the correct operations of the ``cve-check``, it requires the CVE
188identification in a ``CVE:`` tag of the patch file commit message using
189the format::
190
191 CVE: CVE-2022-3341
192
193It is also recommended to add the ``Upstream-Status:`` tag with a link
194to the original patch and sign-off by people working on the backport.
195If there are any modifications to the original patch, note them in
196the ``Comments:`` tag.
197
198With the additional information, the header of the patch file in OE-core becomes::
199
200 From 9cf652cef49d74afe3d454f27d49eb1a1394951e Mon Sep 17 00:00:00 2001
201 From: Jiasheng Jiang <jiasheng@iscas.ac.cn>
202 Date: Wed, 23 Feb 2022 10:31:59 +0800
203 Subject: [PATCH] avformat/nutdec: Add check for avformat_new_stream
204
205 Check for failure of avformat_new_stream() and propagate
206 the error code.
207
208 Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
209
210 CVE: CVE-2022-3341
211
212 Upstream-Status: Backport [https://github.com/FFmpeg/FFmpeg/commit/9cf652cef49d74afe3d454f27d49eb1a1394951e]
213
214 Comments: Refreshed Hunk
215 Signed-off-by: Narpat Mali <narpat.mali@windriver.com>
216 Signed-off-by: Bhabu Bindu <bhabu.bindu@kpit.com>
217 ---
218 libavformat/nutdec.c | 16 ++++++++++++----
219 1 file changed, 12 insertions(+), 4 deletions(-)
220
221A good practice is to include the CVE identifier in the patch file name, the patch file
222commit message and optionally in the recipe commit message.
223
224CVE checker will then capture this information and change the CVE status to ``Patched``
225in the generated reports.
226
227If analysis shows that the CVE issue does not impact the recipe due to configuration, platform,
228version or other reasons, the CVE can be marked as ``Ignored`` by using
229the :term:`CVE_STATUS` variable flag with appropriate reason which is mapped to ``Ignored``.
230The entry should have the format like::
231
232 CVE_STATUS[CVE-2016-10642] = "cpe-incorrect: This is specific to the npm package that installs cmake, so isn't relevant to OpenEmbedded"
233
234As mentioned previously, if data in the CVE database is wrong, it is recommended
235to fix those issues in the CVE database (NVD in the case of OE-core and Poky)
236directly.
237
238Note that if there are many CVEs with the same status and reason, those can be
239shared by using the :term:`CVE_STATUS_GROUPS` variable.
240
241Recipes can be completely skipped by CVE check by including the recipe name in
242the :term:`CVE_CHECK_SKIP_RECIPE` variable.
243
244Implementation details
245======================
246
247Here's what the :ref:`ref-classes-cve-check` class does to find unpatched CVE IDs.
248
249First the code goes through each patch file provided by a recipe. If a valid CVE ID
250is found in the name of the file, the corresponding CVE is considered as patched.
251Don't forget that if multiple CVE IDs are found in the filename, only the last
252one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch
253file. The found CVE IDs are also considered as patched.
254Additionally ``CVE_STATUS`` variable flags are parsed for reasons mapped to ``Patched``
255and these are also considered as patched.
256
257Then, the code looks up all the CVE IDs in the NIST database for all the
258products defined in :term:`CVE_PRODUCT`. Then, for each found CVE:
259
260- If the package name (:term:`PN`) is part of
261 :term:`CVE_CHECK_SKIP_RECIPE`, it is considered as ``Patched``.
262
263- If the CVE ID has status ``CVE_STATUS[<CVE ID>] = "ignored"`` or if it's set to
264 any reason which is mapped to status ``Ignored`` via ``CVE_CHECK_STATUSMAP``,
265 it is set as ``Ignored``.
266
267- If the CVE ID is part of the patched CVE for the recipe, it is
268 already considered as ``Patched``.
269
270- Otherwise, the code checks whether the recipe version (:term:`PV`)
271 is within the range of versions impacted by the CVE. If so, the CVE
272 is considered as ``Unpatched``.
273
274The CVE database is stored in :term:`DL_DIR` and can be inspected using
275``sqlite3`` command as follows::
276
277 sqlite3 downloads/CVE_CHECK/nvdcve_1.1.db .dump | grep CVE-2021-37462
278
279When analyzing CVEs, it is recommended to:
280
281- study the latest information in `CVE database <https://nvd.nist.gov/vuln/search>`__.
282
283- check how upstream developers of the software component addressed the issue, e.g.
284 what patch was applied, which upstream release contains the fix.
285
286- check what other Linux distributions like `Debian <https://security-tracker.debian.org/tracker/>`__
287 did to analyze and address the issue.
288
289- follow security notices from other Linux distributions.
290
291- follow public `open source security mailing lists <https://oss-security.openwall.org/wiki/mailing-lists>`__ for
292 discussions and advance notifications of CVE bugs and software releases with fixes.
293
diff --git a/documentation/dev-manual/wayland.rst b/documentation/dev-manual/wayland.rst
new file mode 100644
index 0000000000..097be9cbde
--- /dev/null
+++ b/documentation/dev-manual/wayland.rst
@@ -0,0 +1,90 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using Wayland and Weston
4************************
5
6:wikipedia:`Wayland <Wayland_(display_server_protocol)>`
7is a computer display server protocol that provides a method for
8compositing window managers to communicate directly with applications
9and video hardware and expects them to communicate with input hardware
10using other libraries. Using Wayland with supporting targets can result
11in better control over graphics frame rendering than an application
12might otherwise achieve.
13
14The Yocto Project provides the Wayland protocol libraries and the
15reference :wikipedia:`Weston <Wayland_(display_server_protocol)#Weston>`
16compositor as part of its release. You can find the integrated packages
17in the ``meta`` layer of the :term:`Source Directory`.
18Specifically, you
19can find the recipes that build both Wayland and Weston at
20``meta/recipes-graphics/wayland``.
21
22You can build both the Wayland and Weston packages for use only with targets
23that accept the :wikipedia:`Mesa 3D and Direct Rendering Infrastructure
24<Mesa_(computer_graphics)>`, which is also known as Mesa DRI. This implies that
25you cannot build and use the packages if your target uses, for example, the
26Intel Embedded Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
27
28.. note::
29
30 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
31 emulated QEMU hardware. However, this version of Weston will run
32 under X emulation without issues.
33
34This section describes what you need to do to implement Wayland and use
35the Weston compositor when building an image for a supporting target.
36
37Enabling Wayland in an Image
38============================
39
40To enable Wayland, you need to enable it to be built and enable it to be
41included (installed) in the image.
42
43Building Wayland
44----------------
45
46To cause Mesa to build the ``wayland-egl`` platform and Weston to build
47Wayland with Kernel Mode Setting
48(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
49support, include the "wayland" flag in the
50:term:`DISTRO_FEATURES`
51statement in your ``local.conf`` file::
52
53 DISTRO_FEATURES:append = " wayland"
54
55.. note::
56
57 If X11 has been enabled elsewhere, Weston will build Wayland with X11
58 support
59
60Installing Wayland and Weston
61-----------------------------
62
63To install the Wayland feature into an image, you must include the
64following
65:term:`CORE_IMAGE_EXTRA_INSTALL`
66statement in your ``local.conf`` file::
67
68 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
69
70Running Weston
71==============
72
73To run Weston inside X11, enabling it as described earlier and building
74a Sato image is sufficient. If you are running your image under Sato, a
75Weston Launcher appears in the "Utility" category.
76
77Alternatively, you can run Weston through the command-line interpretor
78(CLI), which is better suited for development work. To run Weston under
79the CLI, you need to do the following after your image is built:
80
81#. Run these commands to export ``XDG_RUNTIME_DIR``::
82
83 mkdir -p /tmp/$USER-weston
84 chmod 0700 /tmp/$USER-weston
85 export XDG_RUNTIME_DIR=/tmp/$USER-weston
86
87#. Launch Weston in the shell::
88
89 weston
90
diff --git a/documentation/dev-manual/wic.rst b/documentation/dev-manual/wic.rst
new file mode 100644
index 0000000000..a3880f3a1c
--- /dev/null
+++ b/documentation/dev-manual/wic.rst
@@ -0,0 +1,731 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating Partitioned Images Using Wic
4*************************************
5
6Creating an image for a particular hardware target using the
7OpenEmbedded build system does not necessarily mean you can boot that
8image as is on your device. Physical devices accept and boot images in
9various ways depending on the specifics of the device. Usually,
10information about the hardware can tell you what image format the device
11requires. Should your device require multiple partitions on an SD card,
12flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
13create the properly partitioned image.
14
15The ``wic`` command generates partitioned images from existing
16OpenEmbedded build artifacts. Image generation is driven by partitioning
17commands contained in an OpenEmbedded kickstart file (``.wks``)
18specified either directly on the command line or as one of a selection
19of canned kickstart files as shown with the ``wic list images`` command
20in the
21":ref:`dev-manual/wic:generate an image using an existing kickstart file`"
22section. When you apply the command to a given set of build artifacts, the
23result is an image or set of images that can be directly written onto media and
24used on a particular system.
25
26.. note::
27
28 For a kickstart file reference, see the
29 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
30 Chapter in the Yocto Project Reference Manual.
31
32The ``wic`` command and the infrastructure it is based on is by
33definition incomplete. The purpose of the command is to allow the
34generation of customized images, and as such, was designed to be
35completely extensible through a plugin interface. See the
36":ref:`dev-manual/wic:using the wic plugin interface`" section
37for information on these plugins.
38
39This section provides some background information on Wic, describes what
40you need to have in place to run the tool, provides instruction on how
41to use the Wic utility, provides information on using the Wic plugins
42interface, and provides several examples that show how to use Wic.
43
44Background
45==========
46
47This section provides some background on the Wic utility. While none of
48this information is required to use Wic, you might find it interesting.
49
50- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
51 "oe" diphthong in "oeic" was promoted to the letter "w", because
52 "oeic" is both difficult to remember and to pronounce.
53
54- Wic is loosely based on the Meego Image Creator (``mic``) framework.
55 The Wic implementation has been heavily modified to make direct use
56 of OpenEmbedded build artifacts instead of package installation and
57 configuration, which are already incorporated within the OpenEmbedded
58 artifacts.
59
60- Wic is a completely independent standalone utility that initially
61 provides easier-to-use and more flexible replacements for an existing
62 functionality in OE-Core's :ref:`ref-classes-image-live`
63 class. The difference between Wic and those examples is that with Wic
64 the functionality of those scripts is implemented by a
65 general-purpose partitioning language, which is based on Redhat
66 kickstart syntax.
67
68Requirements
69============
70
71In order to use the Wic utility with the OpenEmbedded Build system, your
72system needs to meet the following requirements:
73
74- The Linux distribution on your development host must support the
75 Yocto Project. See the ":ref:`system-requirements-supported-distros`"
76 section in the Yocto Project Reference Manual for the list of
77 distributions that support the Yocto Project.
78
79- The standard system utilities, such as ``cp``, must be installed on
80 your development host system.
81
82- You must have sourced the build environment setup script (i.e.
83 :ref:`structure-core-script`) found in the :term:`Build Directory`.
84
85- You need to have the build artifacts already available, which
86 typically means that you must have already created an image using the
87 OpenEmbedded build system (e.g. ``core-image-minimal``). While it
88 might seem redundant to generate an image in order to create an image
89 using Wic, the current version of Wic requires the artifacts in the
90 form generated by the OpenEmbedded build system.
91
92- You must build several native tools, which are built to run on the
93 build system::
94
95 $ bitbake wic-tools
96
97- Include "wic" as part of the
98 :term:`IMAGE_FSTYPES`
99 variable.
100
101- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
102 as part of the :term:`WKS_FILE` variable. If multiple candidate files can
103 be provided by different layers, specify all the possible names through the
104 :term:`WKS_FILES` variable instead.
105
106Getting Help
107============
108
109You can get general help for the ``wic`` command by entering the ``wic``
110command by itself or by entering the command with a help argument as
111follows::
112
113 $ wic -h
114 $ wic --help
115 $ wic help
116
117Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
118``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
119commands except "help" by using the following form::
120
121 $ wic help command
122
123For example, the following command returns help for the ``write``
124command::
125
126 $ wic help write
127
128Wic supports help for three topics: ``overview``, ``plugins``, and
129``kickstart``. You can get help for any topic using the following form::
130
131 $ wic help topic
132
133For example, the following returns overview help for Wic::
134
135 $ wic help overview
136
137There is one additional level of help for Wic. You can get help on
138individual images through the ``list`` command. You can use the ``list``
139command to return the available Wic images as follows::
140
141 $ wic list images
142 genericx86 Create an EFI disk image for genericx86*
143 beaglebone-yocto Create SD card image for Beaglebone
144 qemuriscv Create qcow2 image for RISC-V QEMU machines
145 mkefidisk Create an EFI disk image
146 qemuloongarch Create qcow2 image for LoongArch QEMU machines
147 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
148 directdisk Create a 'pcbios' direct disk image
149 efi-bootdisk
150 mkhybridiso Create a hybrid ISO image
151 directdisk-gpt Create a 'pcbios' direct disk image
152 systemd-bootdisk Create an EFI disk image with systemd-boot
153 sdimage-bootpart Create SD card image with a boot partition
154 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
155 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
156
157Once you know the list of available
158Wic images, you can use ``help`` with the command to get help on a
159particular image. For example, the following command returns help on the
160"beaglebone-yocto" image::
161
162 $ wic list beaglebone-yocto help
163
164 Creates a partitioned SD card image for Beaglebone.
165 Boot files are located in the first vfat partition.
166
167Operational Modes
168=================
169
170You can use Wic in two different modes, depending on how much control
171you need for specifying the OpenEmbedded build artifacts that are used
172for creating the image: Raw and Cooked:
173
174- *Raw Mode:* You explicitly specify build artifacts through Wic
175 command-line arguments.
176
177- *Cooked Mode:* The current
178 :term:`MACHINE` setting and image
179 name are used to automatically locate and provide the build
180 artifacts. You just supply a kickstart file and the name of the image
181 from which to use artifacts.
182
183Regardless of the mode you use, you need to have the build artifacts
184ready and available.
185
186Raw Mode
187--------
188
189Running Wic in raw mode allows you to specify all the partitions through
190the ``wic`` command line. The primary use for raw mode is if you have
191built your kernel outside of the Yocto Project :term:`Build Directory`.
192In other words, you can point to arbitrary kernel, root filesystem locations,
193and so forth. Contrast this behavior with cooked mode where Wic looks in the
194:term:`Build Directory` (e.g. ``tmp/deploy/images/``\ machine).
195
196The general form of the ``wic`` command in raw mode is::
197
198 $ wic create wks_file options ...
199
200 Where:
201
202 wks_file:
203 An OpenEmbedded kickstart file. You can provide
204 your own custom file or use a file from a set of
205 existing files as described by further options.
206
207 optional arguments:
208 -h, --help show this help message and exit
209 -o OUTDIR, --outdir OUTDIR
210 name of directory to create image in
211 -e IMAGE_NAME, --image-name IMAGE_NAME
212 name of the image to use the artifacts from e.g. core-
213 image-sato
214 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
215 path to the /rootfs dir to use as the .wks rootfs
216 source
217 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
218 path to the dir containing the boot artifacts (e.g.
219 /EFI or /syslinux dirs) to use as the .wks bootimg
220 source
221 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
222 path to the dir containing the kernel to use in the
223 .wks bootimg
224 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
225 path to the native sysroot containing the tools to use
226 to build the image
227 -s, --skip-build-check
228 skip the build check
229 -f, --build-rootfs build rootfs
230 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
231 compress image with specified compressor
232 -m, --bmap generate .bmap
233 --no-fstab-update Do not change fstab file.
234 -v VARS_DIR, --vars VARS_DIR
235 directory with <image>.env files that store bitbake
236 variables
237 -D, --debug output debug information
238
239.. note::
240
241 You do not need root privileges to run Wic. In fact, you should not
242 run as root when using the utility.
243
244Cooked Mode
245-----------
246
247Running Wic in cooked mode leverages off artifacts in the
248:term:`Build Directory`. In other words, you do not have to specify kernel or
249root filesystem locations as part of the command. All you need to provide is
250a kickstart file and the name of the image from which to use artifacts
251by using the "-e" option. Wic looks in the :term:`Build Directory` (e.g.
252``tmp/deploy/images/``\ machine) for artifacts.
253
254The general form of the ``wic`` command using Cooked Mode is as follows::
255
256 $ wic create wks_file -e IMAGE_NAME
257
258 Where:
259
260 wks_file:
261 An OpenEmbedded kickstart file. You can provide
262 your own custom file or use a file from a set of
263 existing files provided with the Yocto Project
264 release.
265
266 required argument:
267 -e IMAGE_NAME, --image-name IMAGE_NAME
268 name of the image to use the artifacts from e.g. core-
269 image-sato
270
271Using an Existing Kickstart File
272================================
273
274If you do not want to create your own kickstart file, you can use an
275existing file provided by the Wic installation. As shipped, kickstart
276files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
277following two locations::
278
279 poky/meta-yocto-bsp/wic
280 poky/scripts/lib/wic/canned-wks
281
282Use the following command to list the available kickstart files::
283
284 $ wic list images
285 genericx86 Create an EFI disk image for genericx86*
286 beaglebone-yocto Create SD card image for Beaglebone
287 qemuriscv Create qcow2 image for RISC-V QEMU machines
288 mkefidisk Create an EFI disk image
289 qemuloongarch Create qcow2 image for LoongArch QEMU machines
290 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
291 directdisk Create a 'pcbios' direct disk image
292 efi-bootdisk
293 mkhybridiso Create a hybrid ISO image
294 directdisk-gpt Create a 'pcbios' direct disk image
295 systemd-bootdisk Create an EFI disk image with systemd-boot
296 sdimage-bootpart Create SD card image with a boot partition
297 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
298 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
299
300When you use an existing file, you
301do not have to use the ``.wks`` extension. Here is an example in Raw
302Mode that uses the ``directdisk`` file::
303
304 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
305 -k kernel_dir -n native_sysroot
306
307Here are the actual partition language commands used in the
308``genericx86.wks`` file to generate an image::
309
310 # short-description: Create an EFI disk image for genericx86*
311 # long-description: Creates a partitioned EFI disk image for genericx86* machines
312 part /boot --source bootimg-efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
313 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
314 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
315
316 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
317
318Using the Wic Plugin Interface
319==============================
320
321You can extend and specialize Wic functionality by using Wic plugins.
322This section explains the Wic plugin interface.
323
324.. note::
325
326 Wic plugins consist of "source" and "imager" plugins. Imager plugins
327 are beyond the scope of this section.
328
329Source plugins provide a mechanism to customize partition content during
330the Wic image generation process. You can use source plugins to map
331values that you specify using ``--source`` commands in kickstart files
332(i.e. ``*.wks``) to a plugin implementation used to populate a given
333partition.
334
335.. note::
336
337 If you use plugins that have build-time dependencies (e.g. native
338 tools, bootloaders, and so forth) when building a Wic image, you need
339 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
340 variable.
341
342Source plugins are subclasses defined in plugin files. As shipped, the
343Yocto Project provides several plugin files. You can see the source
344plugin files that ship with the Yocto Project
345:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
346Each of these plugin files contains source plugins that are designed to
347populate a specific Wic image partition.
348
349Source plugins are subclasses of the ``SourcePlugin`` class, which is
350defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
351the ``BootimgEFIPlugin`` source plugin found in the ``bootimg-efi.py``
352file is a subclass of the ``SourcePlugin`` class, which is found in the
353``pluginbase.py`` file.
354
355You can also implement source plugins in a layer outside of the Source
356Repositories (external layer). To do so, be sure that your plugin files
357are located in a directory whose path is
358``scripts/lib/wic/plugins/source/`` within your external layer. When the
359plugin files are located there, the source plugins they contain are made
360available to Wic.
361
362When the Wic implementation needs to invoke a partition-specific
363implementation, it looks for the plugin with the same name as the
364``--source`` parameter used in the kickstart file given to that
365partition. For example, if the partition is set up using the following
366command in a kickstart file::
367
368 part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
369
370The methods defined as class
371members of the matching source plugin (i.e. ``bootimg-pcbios``) in the
372``bootimg-pcbios.py`` plugin file are used.
373
374To be more concrete, here is the corresponding plugin definition from
375the ``bootimg-pcbios.py`` file for the previous command along with an
376example method called by the Wic implementation when it needs to prepare
377a partition using an implementation-specific function::
378
379 .
380 .
381 .
382 class BootimgPcbiosPlugin(SourcePlugin):
383 """
384 Create MBR boot partition and install syslinux on it.
385 """
386
387 name = 'bootimg-pcbios'
388 .
389 .
390 .
391 @classmethod
392 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
393 oe_builddir, bootimg_dir, kernel_dir,
394 rootfs_dir, native_sysroot):
395 """
396 Called to do the actual content population for a partition i.e. it
397 'prepares' the partition to be incorporated into the image.
398 In this case, prepare content for legacy bios boot partition.
399 """
400 .
401 .
402 .
403
404If a
405subclass (plugin) itself does not implement a particular function, Wic
406locates and uses the default version in the superclass. It is for this
407reason that all source plugins are derived from the ``SourcePlugin``
408class.
409
410The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
411a set of methods that source plugins can implement or override. Any
412plugins (subclass of ``SourcePlugin``) that do not implement a
413particular method inherit the implementation of the method from the
414``SourcePlugin`` class. For more information, see the ``SourcePlugin``
415class in the ``pluginbase.py`` file for details:
416
417The following list describes the methods implemented in the
418``SourcePlugin`` class:
419
420- ``do_prepare_partition()``: Called to populate a partition with
421 actual content. In other words, the method prepares the final
422 partition image that is incorporated into the disk image.
423
424- ``do_configure_partition()``: Called before
425 ``do_prepare_partition()`` to create custom configuration files for a
426 partition (e.g. syslinux or grub configuration files).
427
428- ``do_install_disk()``: Called after all partitions have been
429 prepared and assembled into a disk image. This method provides a hook
430 to allow finalization of a disk image (e.g. writing an MBR).
431
432- ``do_stage_partition()``: Special content-staging hook called
433 before ``do_prepare_partition()``. This method is normally empty.
434
435 Typically, a partition just uses the passed-in parameters (e.g. the
436 unmodified value of ``bootimg_dir``). However, in some cases, things
437 might need to be more tailored. As an example, certain files might
438 additionally need to be taken from ``bootimg_dir + /boot``. This hook
439 allows those files to be staged in a customized fashion.
440
441 .. note::
442
443 ``get_bitbake_var()`` allows you to access non-standard variables that
444 you might want to use for this behavior.
445
446You can extend the source plugin mechanism. To add more hooks, create
447more source plugin methods within ``SourcePlugin`` and the corresponding
448derived subclasses. The code that calls the plugin methods uses the
449``plugin.get_source_plugin_methods()`` function to find the method or
450methods needed by the call. Retrieval of those methods is accomplished
451by filling up a dict with keys that contain the method names of
452interest. On success, these will be filled in with the actual methods.
453See the Wic implementation for examples and details.
454
455Wic Examples
456============
457
458This section provides several examples that show how to use the Wic
459utility. All the examples assume the list of requirements in the
460":ref:`dev-manual/wic:requirements`" section have been met. The
461examples assume the previously generated image is
462``core-image-minimal``.
463
464Generate an Image using an Existing Kickstart File
465--------------------------------------------------
466
467This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
468file::
469
470 $ wic create mkefidisk -e core-image-minimal
471 INFO: Building wic-tools...
472 .
473 .
474 .
475 INFO: The new image(s) can be found here:
476 ./mkefidisk-201804191017-sda.direct
477
478 The following build artifacts were used to create the image(s):
479 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
480 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
481 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
482 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
483
484 INFO: The image(s) were created using OE kickstart file:
485 /home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
486
487The previous example shows the easiest way to create an image by running
488in cooked mode and supplying a kickstart file and the "-e" option to
489point to the existing build artifacts. Your ``local.conf`` file needs to
490have the :term:`MACHINE` variable set
491to the machine you are using, which is "qemux86" in this example.
492
493Once the image builds, the output provides image location, artifact use,
494and kickstart file information.
495
496.. note::
497
498 You should always verify the details provided in the output to make
499 sure that the image was indeed created exactly as expected.
500
501Continuing with the example, you can now write the image from the
502:term:`Build Directory` onto a USB stick, or whatever media for which you
503built your image, and boot from the media. You can write the image by using
504``bmaptool`` or ``dd``::
505
506 $ oe-run-native bmaptool-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
507
508or ::
509
510 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
511
512.. note::
513
514 For more information on how to use the ``bmaptool``
515 to flash a device with an image, see the
516 ":ref:`dev-manual/bmaptool:flashing images using \`\`bmaptool\`\``"
517 section.
518
519Using a Modified Kickstart File
520-------------------------------
521
522Because partitioned image creation is driven by the kickstart file, it
523is easy to affect image creation by changing the parameters in the file.
524This next example demonstrates that through modification of the
525``directdisk-gpt`` kickstart file.
526
527As mentioned earlier, you can use the command ``wic list images`` to
528show the list of existing kickstart files. The directory in which the
529``directdisk-gpt.wks`` file resides is
530``scripts/lib/image/canned-wks/``, which is located in the
531:term:`Source Directory` (e.g. ``poky``).
532Because available files reside in this directory, you can create and add
533your own custom files to the directory. Subsequent use of the
534``wic list images`` command would then include your kickstart files.
535
536In this example, the existing ``directdisk-gpt`` file already does most
537of what is needed. However, for the hardware in this example, the image
538will need to boot from ``sdb`` instead of ``sda``, which is what the
539``directdisk-gpt`` kickstart file uses.
540
541The example begins by making a copy of the ``directdisk-gpt.wks`` file
542in the ``scripts/lib/image/canned-wks`` directory and then by changing
543the lines that specify the target disk from which to boot::
544
545 $ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
546 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
547
548Next, the example modifies the ``directdisksdb-gpt.wks`` file and
549changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
550example changes the following two lines and leaves the remaining lines
551untouched::
552
553 part /boot --source bootimg-pcbios --ondisk sdb --label boot --active --align 1024
554 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
555
556Once the lines are changed, the
557example generates the ``directdisksdb-gpt`` image. The command points
558the process at the ``core-image-minimal`` artifacts for the Next Unit of
559Computing (nuc) :term:`MACHINE` the
560``local.conf``::
561
562 $ wic create directdisksdb-gpt -e core-image-minimal
563 INFO: Building wic-tools...
564 .
565 .
566 .
567 Initialising tasks: 100% |#######################################| Time: 0:00:01
568 NOTE: Executing SetScene Tasks
569 NOTE: Executing RunQueue Tasks
570 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
571 INFO: Creating image(s)...
572
573 INFO: The new image(s) can be found here:
574 ./directdisksdb-gpt-201710090938-sdb.direct
575
576 The following build artifacts were used to create the image(s):
577 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
578 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
579 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
580 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
581
582 INFO: The image(s) were created using OE kickstart file:
583 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
584
585Continuing with the example, you can now directly ``dd`` the image to a
586USB stick, or whatever media for which you built your image, and boot
587the resulting media::
588
589 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
590 140966+0 records in
591 140966+0 records out
592 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
593 $ sudo eject /dev/sdb
594
595Using a Modified Kickstart File and Running in Raw Mode
596-------------------------------------------------------
597
598This next example manually specifies each build artifact (runs in Raw
599Mode) and uses a modified kickstart file. The example also uses the
600``-o`` option to cause Wic to create the output somewhere other than the
601default output directory, which is the current directory::
602
603 $ wic create test.wks -o /home/stephano/testwic \
604 --rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
605 --bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
606 --kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \
607 --native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
608
609 INFO: Creating image(s)...
610
611 INFO: The new image(s) can be found here:
612 /home/stephano/testwic/test-201710091445-sdb.direct
613
614 The following build artifacts were used to create the image(s):
615 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
616 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
617 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
618 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
619
620 INFO: The image(s) were created using OE kickstart file:
621 test.wks
622
623For this example,
624:term:`MACHINE` did not have to be
625specified in the ``local.conf`` file since the artifact is manually
626specified.
627
628Using Wic to Manipulate an Image
629--------------------------------
630
631Wic image manipulation allows you to shorten turnaround time during
632image development. For example, you can use Wic to delete the kernel
633partition of a Wic image and then insert a newly built kernel. This
634saves you time from having to rebuild the entire image each time you
635modify the kernel.
636
637.. note::
638
639 In order to use Wic to manipulate a Wic image as in this example,
640 your development machine must have the ``mtools`` package installed.
641
642The following example examines the contents of the Wic image, deletes
643the existing kernel, and then inserts a new kernel:
644
645#. *List the Partitions:* Use the ``wic ls`` command to list all the
646 partitions in the Wic image::
647
648 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
649 Num Start End Size Fstype
650 1 1048576 25041919 23993344 fat16
651 2 25165824 72157183 46991360 ext4
652
653 The previous output shows two partitions in the
654 ``core-image-minimal-qemux86.wic`` image.
655
656#. *Examine a Particular Partition:* Use the ``wic ls`` command again
657 but in a different form to examine a particular partition.
658
659 .. note::
660
661 You can get command usage on any Wic command using the following
662 form::
663
664 $ wic help command
665
666
667 For example, the following command shows you the various ways to
668 use the
669 wic ls
670 command::
671
672 $ wic help ls
673
674
675 The following command shows what is in partition one::
676
677 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
678 Volume in drive : is boot
679 Volume Serial Number is E894-1809
680 Directory for ::/
681
682 libcom32 c32 186500 2017-10-09 16:06
683 libutil c32 24148 2017-10-09 16:06
684 syslinux cfg 220 2017-10-09 16:06
685 vesamenu c32 27104 2017-10-09 16:06
686 vmlinuz 6904608 2017-10-09 16:06
687 5 files 7 142 580 bytes
688 16 582 656 bytes free
689
690 The previous output shows five files, with the
691 ``vmlinuz`` being the kernel.
692
693 .. note::
694
695 If you see the following error, you need to update or create a
696 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
697 in the file. Then, run the Wic command again::
698
699 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
700 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
701 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
702
703
704#. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
705 ``vmlinuz`` file (kernel)::
706
707 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
708
709#. *Add In the New Kernel:* Use the ``wic cp`` command to add the
710 updated kernel to the Wic image. Depending on how you built your
711 kernel, it could be in different places. If you used ``devtool`` and
712 an SDK to build your kernel, it resides in the ``tmp/work`` directory
713 of the extensible SDK. If you used ``make`` to build the kernel, the
714 kernel will be in the ``workspace/sources`` area.
715
716 The following example assumes ``devtool`` was used to build the
717 kernel::
718
719 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
720 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
721
722 Once the new kernel is added back into the image, you can use the
723 ``dd`` command or :ref:`bmaptool
724 <dev-manual/bmaptool:flashing images using \`\`bmaptool\`\`>`
725 to flash your wic image onto an SD card or USB stick and test your
726 target.
727
728 .. note::
729
730 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
731
diff --git a/documentation/dev-manual/x32-psabi.rst b/documentation/dev-manual/x32-psabi.rst
new file mode 100644
index 0000000000..92b1f96fa4
--- /dev/null
+++ b/documentation/dev-manual/x32-psabi.rst
@@ -0,0 +1,54 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using x32 psABI
4***************
5
6x32 processor-specific Application Binary Interface (`x32
7psABI <https://software.intel.com/en-us/node/628948>`__) is a native
832-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
9ABI defines the calling conventions between functions in a processing
10environment. The interface determines what registers are used and what
11the sizes are for various C data types.
12
13Some processing environments prefer using 32-bit applications even when
14running on Intel 64-bit platforms. Consider the i386 psABI, which is a
15very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
16provide efficient use and access of the Intel 64-bit processor
17resources, leaving the system underutilized. Now consider the x86_64
18psABI. This ABI is newer and uses 64-bits for data sizes and program
19pointers. The extra bits increase the footprint size of the programs,
20libraries, and also increases the memory and file system size
21requirements. Executing under the x32 psABI enables user programs to
22utilize CPU and system resources more efficiently while keeping the
23memory footprint of the applications low. Extra bits are used for
24registers but not for addressing mechanisms.
25
26The Yocto Project supports the final specifications of x32 psABI as
27follows:
28
29- You can create packages and images in x32 psABI format on x86_64
30 architecture targets.
31
32- You can successfully build recipes with the x32 toolchain.
33
34- You can create and boot ``core-image-minimal`` and
35 ``core-image-sato`` images.
36
37- There is RPM Package Manager (RPM) support for x32 binaries.
38
39- There is support for large images.
40
41To use the x32 psABI, you need to edit your ``conf/local.conf``
42configuration file as follows::
43
44 MACHINE = "qemux86-64"
45 DEFAULTTUNE = "x86-64-x32"
46 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \
47 or 'INVALID')) or 'lib'}"
48
49Once you have set
50up your configuration file, use BitBake to build an image that supports
51the x32 psABI. Here is an example::
52
53 $ bitbake core-image-sato
54