summaryrefslogtreecommitdiffstats
path: root/documentation/dev-manual
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/dev-manual')
-rw-r--r--documentation/dev-manual/bblock.rst129
-rw-r--r--documentation/dev-manual/bmaptool.rst58
-rw-r--r--documentation/dev-manual/build-quality.rst409
-rw-r--r--documentation/dev-manual/building.rst877
-rw-r--r--documentation/dev-manual/creating-fragments.rst146
-rw-r--r--documentation/dev-manual/custom-distribution.rst135
-rw-r--r--documentation/dev-manual/custom-template-configuration-directory.rst52
-rw-r--r--documentation/dev-manual/customizing-images.rst221
-rw-r--r--documentation/dev-manual/debugging.rst1271
-rw-r--r--documentation/dev-manual/development-shell.rst82
-rw-r--r--documentation/dev-manual/device-manager.rst74
-rw-r--r--documentation/dev-manual/devtool.rst1322
-rw-r--r--documentation/dev-manual/disk-space.rst61
-rw-r--r--documentation/dev-manual/efficiently-fetching-sources.rst68
-rw-r--r--documentation/dev-manual/error-reporting-tool.rst84
-rw-r--r--documentation/dev-manual/external-scm.rst70
-rw-r--r--documentation/dev-manual/external-toolchain.rst40
-rw-r--r--documentation/dev-manual/figures/bitbake-build-flow.pngbin50338 -> 0 bytes
-rw-r--r--documentation/dev-manual/figures/buildhistory-web.pngbin49966 -> 0 bytes
-rw-r--r--documentation/dev-manual/figures/buildhistory.pngbin44900 -> 0 bytes
-rw-r--r--documentation/dev-manual/figures/cute-files-npm-example.pngbin73191 -> 0 bytes
-rw-r--r--documentation/dev-manual/figures/dev-title.pngbin15950 -> 0 bytes
-rw-r--r--documentation/dev-manual/figures/devtool-add-flow.pngbin181699 -> 0 bytes
-rw-r--r--documentation/dev-manual/figures/devtool-modify-flow.pngbin171676 -> 0 bytes
-rw-r--r--documentation/dev-manual/figures/devtool-upgrade-flow.pngbin138917 -> 0 bytes
-rw-r--r--documentation/dev-manual/figures/multiconfig_files.pngbin18611 -> 0 bytes
-rw-r--r--documentation/dev-manual/figures/recipe-workflow.pngbin48276 -> 0 bytes
-rw-r--r--documentation/dev-manual/gobject-introspection.rst155
-rw-r--r--documentation/dev-manual/hashequivserver.rst129
-rw-r--r--documentation/dev-manual/index.rst58
-rw-r--r--documentation/dev-manual/init-manager.rst162
-rw-r--r--documentation/dev-manual/intro.rst59
-rw-r--r--documentation/dev-manual/layers.rst1050
-rw-r--r--documentation/dev-manual/libraries.rst237
-rw-r--r--documentation/dev-manual/licenses.rst544
-rw-r--r--documentation/dev-manual/limiting-resources.rst138
-rw-r--r--documentation/dev-manual/multiconfig.rst312
-rw-r--r--documentation/dev-manual/new-machine.rst118
-rw-r--r--documentation/dev-manual/new-recipe.rst1656
-rw-r--r--documentation/dev-manual/packages.rst1149
-rw-r--r--documentation/dev-manual/poky-manual-setup.rst128
-rw-r--r--documentation/dev-manual/prebuilt-libraries.rst209
-rw-r--r--documentation/dev-manual/python-development-shell.rst50
-rw-r--r--documentation/dev-manual/qemu.rst467
-rw-r--r--documentation/dev-manual/quilt.rst89
-rw-r--r--documentation/dev-manual/read-only-rootfs.rst89
-rw-r--r--documentation/dev-manual/sbom.rst94
-rw-r--r--documentation/dev-manual/securing-images.rst156
-rw-r--r--documentation/dev-manual/security-subjects.rst194
-rw-r--r--documentation/dev-manual/speeding-up-build.rst109
-rw-r--r--documentation/dev-manual/start.rst877
-rw-r--r--documentation/dev-manual/temporary-source-code.rst73
-rw-r--r--documentation/dev-manual/upgrading-recipes.rst397
-rw-r--r--documentation/dev-manual/vulnerabilities.rst337
-rw-r--r--documentation/dev-manual/wayland.rst90
-rw-r--r--documentation/dev-manual/wic.rst731
-rw-r--r--documentation/dev-manual/x32-psabi.rst54
57 files changed, 0 insertions, 15010 deletions
diff --git a/documentation/dev-manual/bblock.rst b/documentation/dev-manual/bblock.rst
deleted file mode 100644
index 605bb75655..0000000000
--- a/documentation/dev-manual/bblock.rst
+++ /dev/null
@@ -1,129 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Locking and Unlocking Recipes Using ``bblock``
4**********************************************
5
6By design, the OpenEmbedded build system builds everything from scratch
7unless BitBake determines that specific tasks do not require rebuilding.
8At startup, it computes a signature for all tasks, based on the task's input.
9Then, it compares these signatures with the ones from the sstate cache (if they
10exist). Any changes cause the task to rerun.
11
12During development, changes might trigger BitBake to rebuild certain
13recipes, even when we know they do not require rebuilding at that stage.
14For example, modifying a recipe can lead to rebuilding its native
15counterpart, which might prove unnecessary. Editing the ``python3`` recipe,
16for instance, can prompt BitBake to rebuild ``python3-native`` along with any
17recipes that depend on it.
18
19To prevent this, use ``bblock`` to lock specific tasks or recipes to
20specific signatures, forcing BitBake to use the sstate cache for them.
21
22.. warning::
23
24 Use ``bblock`` only during the development phase.
25
26 Forcing BitBake to use the sstate cache, regardless of input changes, means
27 the recipe metadata no longer directly reflect the output. Use this feature
28 with caution. If you do not understand why signatures change, see the section
29 on :yocto_wiki:`understanding what changed </TipsAndTricks/Understanding_what_changed_(diffsigs_etc)>`.
30
31
32Locking tasks and recipes
33-------------------------
34
35To lock a recipe, use::
36
37 $ bblock recipe
38
39You can also use a space-separated list of recipes to lock multiple recipes::
40
41 $ bblock recipe1 recipe2
42
43Locking a recipe means locking all tasks of the recipe. If you need to
44lock only particular tasks, use the `-t` option with a comma-separated
45list of tasks::
46
47 $ bblock -t task1,task2 recipe
48
49
50Unlocking tasks and recipes
51---------------------------
52
53To unlock a recipe, use the ``-r`` option::
54
55 $ bblock -r recipe
56
57You can also use a space-separated list of recipes to unlock multiple recipes::
58
59 $ bblock -r recipe1 recipe2
60
61Unlocking a recipe means unlocking all tasks of the recipe. If you need to
62unlock only particular tasks use the ``-t`` option with a comma-separated
63list of tasks::
64
65 $ bblock -r -t task1,task2 recipe
66
67To unlock all recipes, do not specify any recipe::
68
69 $ bblock -r
70
71
72Configuration file
73------------------
74
75``bblock`` will dump the signatures in the ``build/conf/bblock.conf`` file,
76included by default in :oe_git:`meta/conf/bitbake.conf </openembedded-core/tree/meta/conf/bitbake.conf>`.
77
78To dump the file, use the ``-d`` option::
79
80 $ bblock -d
81
82
83Locking mechanism
84-----------------
85
86``bblock`` computes the signature(s) of the task(s) and sets the 3 following
87variables: :term:`SIGGEN_LOCKEDSIGS`, :term:`SIGGEN_LOCKEDSIGS_TYPES`
88and :term:`SIGGEN_LOCKEDSIGS_TASKSIG_CHECK`.
89
90In particular, ``bblock`` sets::
91
92 SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"
93 SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"
94
95 SIGGEN_LOCKEDSIGS_<package_arch> += "<recipe>:<task>:<signature>"
96
97This produces architecture specific locks and reminds user that some tasks
98have locked signatures.
99
100Example
101-------
102
103When working on the ``python3`` recipe, we can lock ``python3-native`` with
104the following::
105
106 $ bblock python3-native
107 $ bblock -d
108 # Generated by bblock
109 SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"
110 SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"
111
112 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_patch:865859c27e603ba42025b7bb766c3cd4c0f477e4962cfd39128c0619d695fce7"
113 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_populate_sysroot:f8fa5d3194cef638416000252b959e86d0a19f6b7898e1f56b643c588cdd8605"
114 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_prepare_recipe_sysroot:fe295ac505d9d1143313424b201c6f3f2a0a90da40a13a905b86b874705f226a"
115 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_fetch:1b6e4728fee631bc7a8a7006855c5b8182a8224579e32e3d0a2db77c26459f25"
116 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_unpack:2ad74d6f865ef75c35c0e6bbe3f9a90923a6b2c62c18a3ddef514ea31fbc588f"
117 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_deploy_source_date_epoch:15f89b8483c1ad7507480f337619bb98c26e231227785eb3543db163593e7b42"
118 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_configure:7960c13d23270fdb12b3a7c426ce1da0d2f5c7cf5e5d3f5bdce5fa330eb7d482"
119 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_compile:012e1d4a63f1a78fc2143bd90d704dbcf5865c5257d6272aa7540ec1cd3063d9"
120 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_install:d3401cc2afa4c996beb154beaad3e45fa0272b9c56fb86e9db14ec3544c68f9d"
121 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_build:fa88bb7afb9046c0417c24a3fa98a058653805a8b00eda2c2d7fea68fc42f882"
122 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_collect_spdx_deps:cc9c53ba7c495567e9a38ec4801830c425c0d1f895aa2fc66930a2edd510d9b4"
123 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_create_spdx:766a1d09368438b7b5a1a8e2a8f823b2b731db44b57e67d8b3196de91966f9c5"
124 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_create_package_spdx:46f80faeab25575e9977ba3bf14c819489c3d489432ae5145255635108c21020"
125 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_recipe_qa:cb960cdb074e7944e894958db58f3dc2a0436ecf87c247feb3e095e214fec0e4"
126 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_populate_lic:15657441621ee83f15c2e650e7edbb036870b56f55e72e046c6142da3c5783fd"
127 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_create_manifest:24f0abbec221d27bbb2909b6e846288b12cab419f1faf9f5006ed80423d37e28"
128 SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_addto_recipe_sysroot:bcb6a1905f113128de3f88d702b706befd6a786267c045ee82532759a7c214d7"
129
diff --git a/documentation/dev-manual/bmaptool.rst b/documentation/dev-manual/bmaptool.rst
deleted file mode 100644
index 87162a49c9..0000000000
--- a/documentation/dev-manual/bmaptool.rst
+++ /dev/null
@@ -1,58 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Flashing Images Using `bmaptool`
4********************************
5
6A fast and easy way to flash an image to a bootable device is to use
7`bmaptool`, which is integrated into the OpenEmbedded build system.
8`bmaptool` is a generic tool that creates a file's block map (bmap) and
9then uses that map to copy the file. As compared to traditional tools
10such as `dd` or `cp`, `bmaptool` can copy (or flash) large files like raw
11system image files much faster.
12
13.. note::
14
15 - If you are using Ubuntu or Debian distributions, you can install
16 the ``bmap-tools`` package using the following command and then
17 use the tool without specifying ``PATH`` even from the root
18 account::
19
20 $ sudo apt install bmap-tools
21
22 - If you are unable to install the ``bmap-tools`` package, you will
23 need to build `bmaptool` before using it. Use the following command::
24
25 $ bitbake bmaptool-native -caddto_recipe_sysroot
26
27Following, is an example that shows how to flash a Wic image. Realize
28that while this example uses a Wic image, you can use `bmaptool` to flash
29any type of image. Use these steps to flash an image using `bmaptool`:
30
31#. *Update your local.conf File:* You need to have the following set
32 in your ``local.conf`` file before building your image::
33
34 IMAGE_FSTYPES += "wic wic.bmap"
35
36#. *Get Your Image:* Either have your image ready (pre-built with the
37 :term:`IMAGE_FSTYPES`
38 setting previously mentioned) or take the step to build the image::
39
40 $ bitbake image
41
42#. *Flash the Device:* Flash the device with the image by using `bmaptool`
43 depending on your particular setup. The following commands assume the
44 image resides in the :term:`Build Directory`'s ``deploy/images/`` area:
45
46 - If you installed the package for `bmaptool`, you can directly run::
47
48 $ sudo bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
49
50 - Otherwise, if you built `bmaptool` with BitBake, run::
51
52 $ sudo chmod a+w /dev/sdX # get write access to the media, needed only once after booting
53 $ oe-run-native bmaptool-native bmaptool copy build-directory/tmp/deploy/images/machine/image.wic /dev/sdX
54
55For help on the ``bmaptool`` command, use the following command::
56
57 $ bmaptool --help
58
diff --git a/documentation/dev-manual/build-quality.rst b/documentation/dev-manual/build-quality.rst
deleted file mode 100644
index fbe5fb6f0b..0000000000
--- a/documentation/dev-manual/build-quality.rst
+++ /dev/null
@@ -1,409 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Maintaining Build Output Quality
4********************************
5
6Many factors can influence the quality of a build. For example, if you
7upgrade a recipe to use a new version of an upstream software package or
8you experiment with some new configuration options, subtle changes can
9occur that you might not detect until later. Consider the case where
10your recipe is using a newer version of an upstream package. In this
11case, a new version of a piece of software might introduce an optional
12dependency on another library, which is auto-detected. If that library
13has already been built when the software is building, the software will
14link to the built library and that library will be pulled into your
15image along with the new software even if you did not want the library.
16
17The :ref:`ref-classes-buildhistory` class helps you maintain the quality of
18your build output. You can use the class to highlight unexpected and possibly
19unwanted changes in the build output. When you enable build history, it records
20information about the contents of each package and image and then commits that
21information to a local Git repository where you can examine the information.
22
23The remainder of this section describes the following:
24
25- :ref:`How you can enable and disable build history <dev-manual/build-quality:enabling and disabling build history>`
26
27- :ref:`How to understand what the build history contains <dev-manual/build-quality:understanding what the build history contains>`
28
29- :ref:`How to limit the information used for build history <dev-manual/build-quality:using build history to gather image information only>`
30
31- :ref:`How to examine the build history from both a command-line and web interface <dev-manual/build-quality:examining build history information>`
32
33Enabling and Disabling Build History
34====================================
35
36Build history is disabled by default. To enable it, add the following
37:term:`INHERIT` statement and set the :term:`BUILDHISTORY_COMMIT` variable to
38"1" at the end of your ``conf/local.conf`` file found in the
39:term:`Build Directory`::
40
41 INHERIT += "buildhistory"
42 BUILDHISTORY_COMMIT = "1"
43
44Enabling build history as
45previously described causes the OpenEmbedded build system to collect
46build output information and commit it as a single commit to a local
47:ref:`overview-manual/development-environment:git` repository.
48
49.. note::
50
51 Enabling build history increases your build times slightly,
52 particularly for images, and increases the amount of disk space used
53 during the build.
54
55You can disable build history by removing the previous statements from
56your ``conf/local.conf`` file.
57
58Understanding What the Build History Contains
59=============================================
60
61Build history information is kept in ``${``\ :term:`TOPDIR`\ ``}/buildhistory``
62in the :term:`Build Directory` as defined by the :term:`BUILDHISTORY_DIR`
63variable. Here is an example abbreviated listing:
64
65.. image:: figures/buildhistory.png
66 :align: center
67 :width: 50%
68
69At the top level, there is a ``metadata-revs`` file that lists the
70revisions of the repositories for the enabled layers when the build was
71produced. The rest of the data splits into separate ``packages``,
72``images`` and ``sdk`` directories, the contents of which are described
73as follows.
74
75Build History Package Information
76---------------------------------
77
78The history for each package contains a text file that has name-value
79pairs with information about the package. For example,
80``buildhistory/packages/i586-poky-linux/busybox/busybox/latest``
81contains the following:
82
83.. code-block:: none
84
85 PV = 1.22.1
86 PR = r32
87 RPROVIDES =
88 RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
89 RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
90 PKGSIZE = 540168
91 FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
92 /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
93 /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
94 /usr/share/pixmaps /usr/share/applications /usr/share/idl \
95 /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
96 FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
97 /etc/busybox.links.nosuid /etc/busybox.links.suid
98
99Most of these
100name-value pairs correspond to variables used to produce the package.
101The exceptions are ``FILELIST``, which is the actual list of files in
102the package, and ``PKGSIZE``, which is the total size of files in the
103package in bytes.
104
105There is also a file that corresponds to the recipe from which the package
106came (e.g. ``buildhistory/packages/i586-poky-linux/busybox/latest``):
107
108.. code-block:: none
109
110 PV = 1.22.1
111 PR = r32
112 DEPENDS = initscripts kern-tools-native update-rc.d-native \
113 virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
114 virtual/libc virtual/update-alternatives
115 PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
116 busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
117 busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
118
119Finally, for those recipes fetched from a version control system (e.g.,
120Git), there is a file that lists source revisions that are specified in
121the recipe and the actual revisions used during the build. Listed
122and actual revisions might differ when
123:term:`SRCREV` is set to
124${:term:`AUTOREV`}. Here is an
125example assuming
126``buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev``)::
127
128 # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
129 SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
130 # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
131 SRCREV_meta ="a227f20eff056e511d504b2e490f3774ab260d6f"
132
133You can use the
134``buildhistory-collect-srcrevs`` command with the ``-a`` option to
135collect the stored :term:`SRCREV` values from build history and report them
136in a format suitable for use in global configuration (e.g.,
137``local.conf`` or a distro include file) to override floating
138:term:`AUTOREV` values to a fixed set of revisions. Here is some example
139output from this command::
140
141 $ buildhistory-collect-srcrevs -a
142 # all-poky-linux
143 SRCREV:pn-ca-certificates = "07de54fdcc5806bde549e1edf60738c6bccf50e8"
144 SRCREV:pn-update-rc.d = "8636cf478d426b568c1be11dbd9346f67e03adac"
145 # core2-64-poky-linux
146 SRCREV:pn-binutils = "87d4632d36323091e731eb07b8aa65f90293da66"
147 SRCREV:pn-btrfs-tools = "8ad326b2f28c044cb6ed9016d7c3285e23b673c8"
148 SRCREV_bzip2-tests:pn-bzip2 = "f9061c030a25de5b6829e1abf373057309c734c0"
149 SRCREV:pn-e2fsprogs = "02540dedd3ddc52c6ae8aaa8a95ce75c3f8be1c0"
150 SRCREV:pn-file = "504206e53a89fd6eed71aeaf878aa3512418eab1"
151 SRCREV_glibc:pn-glibc = "24962427071fa532c3c48c918e9d64d719cc8a6c"
152 SRCREV:pn-gnome-desktop-testing = "e346cd4ed2e2102c9b195b614f3c642d23f5f6e7"
153 SRCREV:pn-init-system-helpers = "dbd9197569c0935029acd5c9b02b84c68fd937ee"
154 SRCREV:pn-kmod = "b6ecfc916a17eab8f93be5b09f4e4f845aabd3d1"
155 SRCREV:pn-libnsl2 = "82245c0c58add79a8e34ab0917358217a70e5100"
156 SRCREV:pn-libseccomp = "57357d2741a3b3d3e8425889a6b79a130e0fa2f3"
157 SRCREV:pn-libxcrypt = "50cf2b6dd4fdf04309445f2eec8de7051d953abf"
158 SRCREV:pn-ncurses = "51d0fd9cc3edb975f04224f29f777f8f448e8ced"
159 SRCREV:pn-procps = "19a508ea121c0c4ac6d0224575a036de745eaaf8"
160 SRCREV:pn-psmisc = "5fab6b7ab385080f1db725d6803136ec1841a15f"
161 SRCREV:pn-ptest-runner = "bcb82804daa8f725b6add259dcef2067e61a75aa"
162 SRCREV:pn-shared-mime-info = "18e558fa1c8b90b86757ade09a4ba4d6a6cf8f70"
163 SRCREV:pn-zstd = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
164 # qemux86_64-poky-linux
165 SRCREV_machine:pn-linux-yocto = "20301aeb1a64164b72bc72af58802b315e025c9c"
166 SRCREV_meta:pn-linux-yocto = "2d38a472b21ae343707c8bd64ac68a9eaca066a0"
167 # x86_64-linux
168 SRCREV:pn-binutils-cross-x86_64 = "87d4632d36323091e731eb07b8aa65f90293da66"
169 SRCREV_glibc:pn-cross-localedef-native = "24962427071fa532c3c48c918e9d64d719cc8a6c"
170 SRCREV_localedef:pn-cross-localedef-native = "794da69788cbf9bf57b59a852f9f11307663fa87"
171 SRCREV:pn-debianutils-native = "de14223e5bffe15e374a441302c528ffc1cbed57"
172 SRCREV:pn-libmodulemd-native = "ee80309bc766d781a144e6879419b29f444d94eb"
173 SRCREV:pn-virglrenderer-native = "363915595e05fb252e70d6514be2f0c0b5ca312b"
174 SRCREV:pn-zstd-native = "e47e674cd09583ff0503f0f6defd6d23d8b718d3"
175
176.. note::
177
178 Here are some notes on using the ``buildhistory-collect-srcrevs`` command:
179
180 - By default, only values where the :term:`SRCREV` was not hardcoded
181 (usually when :term:`AUTOREV` is used) are reported. Use the ``-a``
182 option to see all :term:`SRCREV` values.
183
184 - The output statements might not have any effect if overrides are
185 applied elsewhere in the build system configuration. Use the
186 ``-f`` option to add the ``forcevariable`` override to each output
187 line if you need to work around this restriction.
188
189 - The script does apply special handling when building for multiple
190 machines. However, the script does place a comment before each set
191 of values that specifies which triplet to which they belong as
192 previously shown (e.g., ``i586-poky-linux``).
193
194Build History Image Information
195-------------------------------
196
197The files produced for each image are as follows:
198
199- ``image-files:`` A directory containing selected files from the root
200 filesystem. The files are defined by
201 :term:`BUILDHISTORY_IMAGE_FILES`.
202
203- ``build-id.txt:`` Human-readable information about the build
204 configuration and metadata source revisions. This file contains the
205 full build header as printed by BitBake.
206
207- ``*.dot:`` Dependency graphs for the image that are compatible with
208 ``graphviz``.
209
210- ``files-in-image.txt:`` A list of files in the image with
211 permissions, owner, group, size, and symlink information.
212
213- ``image-info.txt:`` A text file containing name-value pairs with
214 information about the image. See the following listing example for
215 more information.
216
217- ``installed-package-names.txt:`` A list of installed packages by name
218 only.
219
220- ``installed-package-sizes.txt:`` A list of installed packages ordered
221 by size.
222
223- ``installed-packages.txt:`` A list of installed packages with full
224 package filenames.
225
226.. note::
227
228 Installed package information is able to be gathered and produced
229 even if package management is disabled for the final image.
230
231Here is an example of ``image-info.txt``:
232
233.. code-block:: none
234
235 DISTRO = poky
236 DISTRO_VERSION = 3.4+snapshot-a0245d7be08f3d24ea1875e9f8872aa6bbff93be
237 USER_CLASSES = buildstats
238 IMAGE_CLASSES = qemuboot qemuboot license_image
239 IMAGE_FEATURES = allow-empty-password empty-root-password allow-root-login post-install-logging
240 IMAGE_LINGUAS =
241 IMAGE_INSTALL = packagegroup-core-boot speex speexdsp
242 BAD_RECOMMENDATIONS =
243 NO_RECOMMENDATIONS =
244 PACKAGE_EXCLUDE =
245 ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; cve_check_write_rootfs_manifest; ssh_allow_empty_password; ssh_allow_root_login; postinst_enable_logging; rootfs_update_timestamp; write_image_test_data; empty_var_volatile; sort_passwd; rootfs_reproducible;
246 IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ;
247 IMAGESIZE = 9265
248
249Other than ``IMAGESIZE``,
250which is the total size of the files in the image in Kbytes, the
251name-value pairs are variables that may have influenced the content of
252the image. This information is often useful when you are trying to
253determine why a change in the package or file listings has occurred.
254
255Using Build History to Gather Image Information Only
256----------------------------------------------------
257
258As you can see, build history produces image information, including
259dependency graphs, so you can see why something was pulled into the
260image. If you are just interested in this information and not interested
261in collecting specific package or SDK information, you can enable
262writing only image information without any history by adding the
263following to your ``conf/local.conf`` file found in the
264:term:`Build Directory`::
265
266 INHERIT += "buildhistory"
267 BUILDHISTORY_COMMIT = "0"
268 BUILDHISTORY_FEATURES = "image"
269
270Here, you set the
271:term:`BUILDHISTORY_FEATURES`
272variable to use the image feature only.
273
274Build History SDK Information
275-----------------------------
276
277Build history collects similar information on the contents of SDKs (e.g.
278``bitbake -c populate_sdk imagename``) as compared to information it
279collects for images. Furthermore, this information differs depending on
280whether an extensible or standard SDK is being produced.
281
282The following list shows the files produced for SDKs:
283
284- ``files-in-sdk.txt:`` A list of files in the SDK with permissions,
285 owner, group, size, and symlink information. This list includes both
286 the host and target parts of the SDK.
287
288- ``sdk-info.txt:`` A text file containing name-value pairs with
289 information about the SDK. See the following listing example for more
290 information.
291
292- ``sstate-task-sizes.txt:`` A text file containing name-value pairs
293 with information about task group sizes (e.g. :ref:`ref-tasks-populate_sysroot`
294 tasks have a total size). The ``sstate-task-sizes.txt`` file exists
295 only when an extensible SDK is created.
296
297- ``sstate-package-sizes.txt:`` A text file containing name-value pairs
298 with information for the shared-state packages and sizes in the SDK.
299 The ``sstate-package-sizes.txt`` file exists only when an extensible
300 SDK is created.
301
302- ``sdk-files:`` A folder that contains copies of the files mentioned
303 in ``BUILDHISTORY_SDK_FILES`` if the files are present in the output.
304 Additionally, the default value of ``BUILDHISTORY_SDK_FILES`` is
305 specific to the extensible SDK although you can set it differently if
306 you would like to pull in specific files from the standard SDK.
307
308 The default files are ``conf/local.conf``, ``conf/bblayers.conf``,
309 ``conf/auto.conf``, ``conf/locked-sigs.inc``, and
310 ``conf/devtool.conf``. Thus, for an extensible SDK, these files get
311 copied into the ``sdk-files`` directory.
312
313- The following information appears under each of the ``host`` and
314 ``target`` directories for the portions of the SDK that run on the
315 host and on the target, respectively:
316
317 .. note::
318
319 The following files for the most part are empty when producing an
320 extensible SDK because this type of SDK is not constructed from
321 packages as is the standard SDK.
322
323 - ``depends.dot:`` Dependency graph for the SDK that is compatible
324 with ``graphviz``.
325
326 - ``installed-package-names.txt:`` A list of installed packages by
327 name only.
328
329 - ``installed-package-sizes.txt:`` A list of installed packages
330 ordered by size.
331
332 - ``installed-packages.txt:`` A list of installed packages with full
333 package filenames.
334
335Here is an example of ``sdk-info.txt``:
336
337.. code-block:: none
338
339 DISTRO = poky
340 DISTRO_VERSION = 1.3+snapshot-20130327
341 SDK_NAME = poky-glibc-i686-arm
342 SDK_VERSION = 1.3+snapshot
343 SDKMACHINE =
344 SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
345 BAD_RECOMMENDATIONS =
346 SDKSIZE = 352712
347
348Other than ``SDKSIZE``, which is
349the total size of the files in the SDK in Kbytes, the name-value pairs
350are variables that might have influenced the content of the SDK. This
351information is often useful when you are trying to determine why a
352change in the package or file listings has occurred.
353
354Examining Build History Information
355-----------------------------------
356
357You can examine build history output from the command line or from a web
358interface.
359
360To see any changes that have occurred (assuming you have
361:term:`BUILDHISTORY_COMMIT` = "1"),
362you can simply use any Git command that allows you to view the history
363of a repository. Here is one method::
364
365 $ git log -p
366
367You need to realize,
368however, that this method does show changes that are not significant
369(e.g. a package's size changing by a few bytes).
370
371There is a command-line tool called ``buildhistory-diff``, though,
372that queries the Git repository and prints just the differences that
373might be significant in human-readable form. Here is an example::
374
375 $ poky/poky/scripts/buildhistory-diff . HEAD^
376 Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
377 /etc/anotherpkg.conf was added
378 /sbin/anotherpkg was added
379 * (installed-package-names.txt):
380 * anotherpkg was added
381 Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
382 anotherpkg was added
383 packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
384 * PR changed from "r0" to "r1"
385 * PV changed from "0.1.10" to "0.1.12"
386 packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
387 * PR changed from "r0" to "r1"
388 * PV changed from "0.1.10" to "0.1.12"
389
390.. note::
391
392 The ``buildhistory-diff`` tool requires the ``GitPython``
393 package. Be sure to install it using Pip3 as follows::
394
395 $ pip3 install GitPython --user
396
397
398 Alternatively, you can install ``python3-git`` using the appropriate
399 distribution package manager (e.g. ``apt``, ``dnf``, or ``zipper``).
400
401To see changes to the build history using a web interface, follow the
402instruction in the ``README`` file
403:yocto_git:`here </buildhistory-web/>`.
404
405Here is a sample screenshot of the interface:
406
407.. image:: figures/buildhistory-web.png
408 :width: 100%
409
diff --git a/documentation/dev-manual/building.rst b/documentation/dev-manual/building.rst
deleted file mode 100644
index 60ad11f52f..0000000000
--- a/documentation/dev-manual/building.rst
+++ /dev/null
@@ -1,877 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Building
4********
5
6This section describes various build procedures, such as the steps
7needed for a simple build, building a target for multiple configurations,
8generating an image for more than one machine, and so forth.
9
10Building a Simple Image
11=======================
12
13In the development environment, you need to build an image whenever you
14change hardware support, add or change system libraries, or add or
15change services that have dependencies. There are several methods that allow
16you to build an image within the Yocto Project. This section presents
17the basic steps you need to build a simple image using BitBake from a
18build host running Linux.
19
20.. note::
21
22 - For information on how to build an image using
23 :term:`Toaster`, see the
24 :doc:`/toaster-manual/index`.
25
26 - For information on how to use ``devtool`` to build images, see the
27 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
28 section in the Yocto Project Application Development and the
29 Extensible Software Development Kit (eSDK) manual.
30
31 - For a quick example on how to build an image using the
32 OpenEmbedded build system, see the
33 :doc:`/brief-yoctoprojectqs/index` document.
34
35 - You can also use the `Yocto Project BitBake
36 <https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__
37 extension for Visual Studio Code to build images.
38
39The build process creates an entire Linux distribution from source and
40places it in your :term:`Build Directory` under ``tmp/deploy/images``. For
41detailed information on the build process using BitBake, see the
42":ref:`overview-manual/concepts:images`" section in the Yocto Project Overview
43and Concepts Manual.
44
45The following figure and list overviews the build process:
46
47.. image:: figures/bitbake-build-flow.png
48 :width: 100%
49
50#. *Set up Your Host Development System to Support Development Using the
51 Yocto Project*: See the ":doc:`/dev-manual/start`" section for options on how to get a
52 build host ready to use the Yocto Project.
53
54#. *Initialize the Build Environment:* Initialize the build environment
55 by sourcing the build environment script (i.e.
56 :ref:`structure-core-script`)::
57
58 $ source oe-init-build-env [build_dir]
59
60 When you use the initialization script, the OpenEmbedded build system
61 uses ``build`` as the default :term:`Build Directory` in your current work
62 directory. You can use a `build_dir` argument with the script to
63 specify a different :term:`Build Directory`.
64
65 .. note::
66
67 A common practice is to use a different :term:`Build Directory` for
68 different targets; for example, ``~/build/x86`` for a ``qemux86``
69 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
70 event, it's typically cleaner to locate the :term:`Build Directory`
71 somewhere outside of your source directory.
72
73#. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
74 ``conf/local.conf`` configuration file, which is found in the
75 :term:`Build Directory`, is set up how you want it. This file defines many
76 aspects of the build environment including the target machine architecture
77 through the :term:`MACHINE` variable, the packaging format used during
78 the build (:term:`PACKAGE_CLASSES`), and a centralized tarball download
79 directory through the :term:`DL_DIR` variable.
80
81#. *Build the Image:* Build the image using the ``bitbake`` command::
82
83 $ bitbake target
84
85 .. note::
86
87 For information on BitBake, see the :doc:`bitbake:index`.
88
89 The target is the name of the recipe you want to build. Common
90 targets are the images in ``meta/recipes-core/images``,
91 ``meta/recipes-sato/images``, and so forth all found in the
92 :term:`Source Directory`. Alternatively, the target
93 can be the name of a recipe for a specific piece of software such as
94 BusyBox. For more details about the images the OpenEmbedded build
95 system supports, see the
96 ":ref:`ref-manual/images:Images`" chapter in the Yocto
97 Project Reference Manual.
98
99 As an example, the following command builds the
100 ``core-image-minimal`` image::
101
102 $ bitbake core-image-minimal
103
104 Once an
105 image has been built, it often needs to be installed. The images and
106 kernels built by the OpenEmbedded build system are placed in the
107 :term:`Build Directory` in ``tmp/deploy/images``. For information on how to
108 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
109 :doc:`/sdk-manual/index` manual. For
110 information about how to install these images, see the documentation
111 for your particular board or machine.
112
113Building Images for Multiple Targets Using Multiple Configurations
114==================================================================
115
116See the :doc:`/dev-manual/multiconfig` section of the Yocto Project Development Tasks
117Manual.
118
119Building an Initial RAM Filesystem (Initramfs) Image
120====================================================
121
122An initial RAM filesystem (:term:`Initramfs`) image provides a temporary root
123filesystem used for early system initialization, typically providing tools and
124loading modules needed to locate and mount the final root filesystem.
125
126Follow these steps to create an :term:`Initramfs` image:
127
128#. *Create the Initramfs Image Recipe:* You can reference the
129 ``core-image-minimal-initramfs.bb`` recipe found in the
130 ``meta/recipes-core`` directory of the :term:`Source Directory`
131 as an example from which to work. The ``core-image-minimal-initramfs`` recipe
132 is based on the :ref:`initramfs-framework <dev-manual/building:Customizing an
133 Initramfs using \`\`initramfs-framework\`\`>` recipe described below.
134
135#. *Decide if You Need to Bundle the Initramfs Image Into the Kernel
136 Image:* If you want the :term:`Initramfs` image that is built to be bundled
137 in with the kernel image, set the :term:`INITRAMFS_IMAGE_BUNDLE`
138 variable to ``"1"`` in your ``local.conf`` configuration file and set the
139 :term:`INITRAMFS_IMAGE` variable in the recipe that builds the kernel image.
140
141 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the :term:`Initramfs`
142 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
143 :term:`Initramfs` image is then passed to the kernel's ``Makefile`` using the
144 :term:`CONFIG_INITRAMFS_SOURCE` variable, allowing the :term:`Initramfs`
145 image to be built into the kernel normally.
146
147#. *Optionally Add Items to the Initramfs Image Through the Initramfs
148 Image Recipe:* If you add items to the :term:`Initramfs` image by way of its
149 recipe, you should use :term:`PACKAGE_INSTALL` rather than
150 :term:`IMAGE_INSTALL`. :term:`PACKAGE_INSTALL` gives more direct control of
151 what is added to the image as compared to the defaults you might not
152 necessarily want that are set by the :ref:`ref-classes-image`
153 or :ref:`ref-classes-core-image` classes.
154
155#. *Build the Kernel Image and the Initramfs Image:* Build your kernel
156 image using BitBake. Because the :term:`Initramfs` image recipe is a
157 dependency of the kernel image, the :term:`Initramfs` image is built as well
158 and bundled with the kernel image if you used the
159 :term:`INITRAMFS_IMAGE_BUNDLE` variable described earlier.
160
161Customizing an Initramfs using ``initramfs-framework``
162------------------------------------------------------
163
164The ``core-image-minimal-initramfs.bb`` recipe found in
165:oe_git:`meta/recipes-core/images
166</openembedded-core/tree/meta/recipes-core/images>` uses the
167:oe_git:`initramfs-framework_1.0.bb
168</openembedded-core/tree/meta/recipes-core/initrdscripts/initramfs-framework_1.0.bb>`
169recipe as its base component. The goal of the ``initramfs-framework`` recipe is
170to provide the building blocks to build a customized :term:`Initramfs`.
171
172The ``initramfs-framework`` recipe relies on shell initialization scripts
173defined in :oe_git:`meta/recipes-core/initrdscripts/initramfs-framework
174</openembedded-core/tree/meta/recipes-core/initrdscripts/initramfs-framework>`. Since some of
175these scripts do not apply for all use cases, the ``initramfs-framework`` recipe
176defines different packages:
177
178- ``initramfs-framework-base``: this package installs the basic components of
179 an :term:`Initramfs`, such as the ``init`` script or the ``/dev/console``
180 character special file. As this package is a runtime dependency of all
181 modules listed below, it is automatically pulled in when one of the modules
182 is installed in the image.
183- ``initramfs-module-exec``: support for execution of applications.
184- ``initramfs-module-mdev``: support for `mdev
185 <https://wiki.gentoo.org/wiki/Mdev>`__.
186- ``initramfs-module-udev``: support for :wikipedia:`Udev <Udev>`.
187- ``initramfs-module-e2fs``: support for :wikipedia:`ext4/ext3/ext2
188 <Extended_file_system>` filesystems.
189- ``initramfs-module-nfsrootfs``: support for locating and mounting the root
190 partition via :wikipedia:`NFS <Network_File_System>`.
191- ``initramfs-module-rootfs``: support for locating and mounting the root
192 partition.
193- ``initramfs-module-debug``: dynamic debug support.
194- ``initramfs-module-lvm``: :wikipedia:`LVM <Logical_volume_management>` rootfs support.
195- ``initramfs-module-overlayroot``: support for mounting a read-write overlay
196 on top of a read-only root filesystem.
197
198In addition to the packages defined by the ``initramfs-framework`` recipe
199itself, the following packages are defined by the recipes present in
200:oe_git:`meta/recipes-core/initrdscripts </openembedded-core/tree/meta/recipes-core/initrdscripts>`:
201
202- ``initramfs-module-install``: module to create and install a partition layout
203 on a selected block device.
204- ``initramfs-module-install-efi``: module to create and install an EFI
205 partition layout on a selected block device.
206- ``initramfs-module-setup-live``: module to start a shell in the
207 :term:`Initramfs` if ``root=/dev/ram0`` in passed in the `Kernel command-line
208 <https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html>`__
209 or the ``root=`` parameter was not passed.
210
211To customize the :term:`Initramfs`, you can add or remove packages listed
212earlier from the :term:`PACKAGE_INSTALL` variable with a :ref:`bbappend
213<dev-manual/layers:Appending Other Layers Metadata With Your Layer>` on the
214``core-image-minimal-initramfs`` recipe, or create a custom recipe for the
215:term:`Initramfs` taking ``core-image-minimal-initramfs`` as example.
216
217Custom scripts can be added to the :term:`Initramfs` by writing your own
218recipes. The recipes are conventionally named ``initramfs-module-<module name>``
219where ``<module name>`` is the name of the module. The recipe should set its
220:term:`RDEPENDS` package-specific variables to include
221``initramfs-framework-base`` and the other packages on which the module depends
222at runtime.
223
224The recipe must install shell initialization scripts in :term:`${D} <D>`\
225``/init.d`` and must follow the ``<number>-<script name>`` naming scheme where:
226
227- ``<number>`` is a *two-digit* number that affects the execution order of the
228 script compared to others. For example, the script ``80-setup-live`` would be
229 executed after ``01-udev`` because 80 is greater than 01.
230
231 This number being two-digits is important here as the scripts are executed
232 alphabetically. For example, the script ``10-script`` would be executed
233 before the script ``8-script``, because ``1`` is inferior to ``8``.
234 Therefore, the script should be named ``08-script``.
235
236- ``<script name>`` is the script name which you can choose freely.
237
238 If two script use the same ``<number>``, they are sorted alphabetically based
239 on ``<script name>``.
240
241Bundling an Initramfs Image From a Separate Multiconfig
242-------------------------------------------------------
243
244There may be a case where we want to build an :term:`Initramfs` image which does not
245inherit the same distro policy as our main image, for example, we may want
246our main image to use ``TCLIBC = "glibc"``, but to use ``TCLIBC = "musl"`` in our :term:`Initramfs`
247image to keep a smaller footprint. However, by performing the steps mentioned
248above the :term:`Initramfs` image will inherit ``TCLIBC = "glibc"`` without allowing us
249to override it.
250
251To achieve this, you need to perform some additional steps:
252
253#. *Create a multiconfig for your Initramfs image:* You can perform the steps
254 on ":ref:`dev-manual/building:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
255 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
256 contains the variables::
257
258 TMPDIR = "${TOPDIR}/tmp-initramfscfg"
259 TCLIBC = "musl"
260
261#. *Set additional Initramfs variables on your main configuration:*
262 Additionally, on your main configuration (``local.conf``) you need to set the
263 variables::
264
265 INITRAMFS_MULTICONFIG = "initramfscfg"
266 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
267
268 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
269 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
270 to be built coming from the ``initramfscfg`` multiconfig, and to let the
271 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
272
273 Building a system with such configuration will build the kernel using the
274 main configuration but the :ref:`ref-tasks-bundle_initramfs` task will grab the
275 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
276 instead, resulting in a musl based :term:`Initramfs` image bundled in the kernel
277 but a glibc based main image.
278
279 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
280 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
281
282
283Building a Tiny System
284======================
285
286Very small distributions have some significant advantages such as
287requiring less on-die or in-package memory (cheaper), better performance
288through efficient cache usage, lower power requirements due to less
289memory, faster boot times, and reduced development overhead. Some
290real-world examples where a very small distribution gives you distinct
291advantages are digital cameras, medical devices, and small headless
292systems.
293
294This section presents information that shows you how you can trim your
295distribution to even smaller sizes than the ``poky-tiny`` distribution,
296which is around 5 Mbytes, that can be built out-of-the-box using the
297Yocto Project.
298
299Tiny System Overview
300--------------------
301
302The following list presents the overall steps you need to consider and
303perform to create distributions with smaller root filesystems, achieve
304faster boot times, maintain your critical functionality, and avoid
305initial RAM disks:
306
307- :ref:`Determine your goals and guiding principles
308 <dev-manual/building:goals and guiding principles>`
309
310- :ref:`dev-manual/building:understand what contributes to your image size`
311
312- :ref:`Reduce the size of the root filesystem
313 <dev-manual/building:trim the root filesystem>`
314
315- :ref:`Reduce the size of the kernel <dev-manual/building:trim the kernel>`
316
317- :ref:`dev-manual/building:remove package management requirements`
318
319- :ref:`dev-manual/building:look for other ways to minimize size`
320
321- :ref:`dev-manual/building:iterate on the process`
322
323Goals and Guiding Principles
324----------------------------
325
326Before you can reach your destination, you need to know where you are
327going. Here is an example list that you can use as a guide when creating
328very small distributions:
329
330- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
331 less and a root filesystem that is 3 Mbytes or less).
332
333- Find the areas that are currently taking 90% of the space and
334 concentrate on reducing those areas.
335
336- Do not create any difficult "hacks" to achieve your goals.
337
338- Leverage the device-specific options.
339
340- Work in a separate layer so that you keep changes isolated. For
341 information on how to create layers, see the
342 ":ref:`dev-manual/layers:understanding and creating layers`" section.
343
344Understand What Contributes to Your Image Size
345----------------------------------------------
346
347It is easiest to have something to start with when creating your own
348distribution. You can use the Yocto Project out-of-the-box to create the
349``poky-tiny`` distribution. Ultimately, you will want to make changes in
350your own distribution that are likely modeled after ``poky-tiny``.
351
352.. note::
353
354 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
355 ``local.conf`` file to "poky-tiny" as described in the
356 ":ref:`dev-manual/custom-distribution:creating your own distribution`"
357 section.
358
359Understanding some memory concepts will help you reduce the system size.
360Memory consists of static, dynamic, and temporary memory. Static memory
361is the TEXT (code), DATA (initialized data in the code), and BSS
362(uninitialized data) sections. Dynamic memory represents memory that is
363allocated at runtime: stacks, hash tables, and so forth. Temporary
364memory is recovered after the boot process. This memory consists of
365memory used for decompressing the kernel and for the ``__init__``
366functions.
367
368To help you see where you currently are with kernel and root filesystem
369sizes, you can use two tools found in the :term:`Source Directory`
370in the
371``scripts/tiny/`` directory:
372
373- ``ksize.py``: Reports component sizes for the kernel build objects.
374
375- ``dirsize.py``: Reports component sizes for the root filesystem.
376
377This next tool and command help you organize configuration fragments and
378view file dependencies in a human-readable form:
379
380- ``merge_config.sh``: Helps you manage configuration files and
381 fragments within the kernel. With this tool, you can merge individual
382 configuration fragments together. The tool allows you to make
383 overrides and warns you of any missing configuration options. The
384 tool is ideal for allowing you to iterate on configurations, create
385 minimal configurations, and create configuration files for different
386 machines without having to duplicate your process.
387
388 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
389 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
390 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
391 directory.
392
393 For more information on configuration fragments, see the
394 ":ref:`kernel-dev/common:creating configuration fragments`"
395 section in the Yocto Project Linux Kernel Development Manual.
396
397- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
398 with these options brings up a Dependency Explorer from which you can
399 view file dependencies. Understanding these dependencies allows you
400 to make informed decisions when cutting out various pieces of the
401 kernel and root filesystem.
402
403Trim the Root Filesystem
404------------------------
405
406The root filesystem is made up of packages for booting, libraries, and
407applications. To change things, you can configure how the packaging
408happens, which changes the way you build them. You can also modify the
409filesystem itself or select a different filesystem.
410
411First, find out what is hogging your root filesystem by running the
412``dirsize.py`` script from your root directory::
413
414 $ cd root-directory-of-image
415 $ dirsize.py 100000 > dirsize-100k.log
416 $ cat dirsize-100k.log
417
418You can apply a filter to the script to ignore files
419under a certain size. The previous example filters out any files below
420100 Kbytes. The sizes reported by the tool are uncompressed, and thus
421will be smaller by a relatively constant factor in a compressed root
422filesystem. When you examine your log file, you can focus on areas of
423the root filesystem that take up large amounts of memory.
424
425You need to be sure that what you eliminate does not cripple the
426functionality you need. One way to see how packages relate to each other
427is by using the Dependency Explorer UI with the BitBake command::
428
429 $ cd image-directory
430 $ bitbake -u taskexp -g image
431
432Use the interface to
433select potential packages you wish to eliminate and see their dependency
434relationships.
435
436When deciding how to reduce the size, get rid of packages that result in
437minimal impact on the feature set. For example, you might not need a VGA
438display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
439instead of ``udev``.
440
441Use your ``local.conf`` file to make changes. For example, to eliminate
442``udev`` and ``glib``, set the following in the local configuration
443file::
444
445 VIRTUAL-RUNTIME_dev_manager = ""
446
447Finally, you should consider exactly the type of root filesystem you
448need to meet your needs while also reducing its size. For example,
449consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
450:term:`Initramfs` using ``initramfs``. Be aware that ``ext3`` requires a 1
451Mbyte journal. If you are okay with running read-only, you do not need
452this journal.
453
454.. note::
455
456 After each round of elimination, you need to rebuild your system and
457 then use the tools to see the effects of your reductions.
458
459Trim the Kernel
460---------------
461
462The kernel is built by including policies for hardware-independent
463aspects. What subsystems do you enable? For what architecture are you
464building? Which drivers do you build by default?
465
466.. note::
467
468 You can modify the kernel source if you want to help with boot time.
469
470Run the ``ksize.py`` script from the top-level Linux build directory to
471get an idea of what is making up the kernel::
472
473 $ cd top-level-linux-build-directory
474 $ ksize.py > ksize.log
475 $ cat ksize.log
476
477When you examine the log, you will see how much space is taken up with
478the built-in ``.o`` files for drivers, networking, core kernel files,
479filesystem, sound, and so forth. The sizes reported by the tool are
480uncompressed, and thus will be smaller by a relatively constant factor
481in a compressed kernel image. Look to reduce the areas that are large
482and taking up around the "90% rule."
483
484To examine, or drill down, into any particular area, use the ``-d``
485option with the script::
486
487 $ ksize.py -d > ksize.log
488
489Using this option
490breaks out the individual file information for each area of the kernel
491(e.g. drivers, networking, and so forth).
492
493Use your log file to see what you can eliminate from the kernel based on
494features you can let go. For example, if you are not going to need
495sound, you do not need any drivers that support sound.
496
497After figuring out what to eliminate, you need to reconfigure the kernel
498to reflect those changes during the next build. You could run
499``menuconfig`` and make all your changes at once. However, that makes it
500difficult to see the effects of your individual eliminations and also
501makes it difficult to replicate the changes for perhaps another target
502device. A better method is to start with no configurations using
503``allnoconfig``, create configuration fragments for individual changes,
504and then manage the fragments into a single configuration file using
505``merge_config.sh``. The tool makes it easy for you to iterate using the
506configuration change and build cycle.
507
508Each time you make configuration changes, you need to rebuild the kernel
509and check to see what impact your changes had on the overall size.
510
511Remove Package Management Requirements
512--------------------------------------
513
514Packaging requirements add size to the image. One way to reduce the size
515of the image is to remove all the packaging requirements from the image.
516This reduction includes both removing the package manager and its unique
517dependencies as well as removing the package management data itself.
518
519To eliminate all the packaging requirements for an image, be sure that
520"package-management" is not part of your
521:term:`IMAGE_FEATURES`
522statement for the image. When you remove this feature, you are removing
523the package manager as well as its dependencies from the root
524filesystem.
525
526Look for Other Ways to Minimize Size
527------------------------------------
528
529Depending on your particular circumstances, other areas that you can
530trim likely exist. The key to finding these areas is through tools and
531methods described here combined with experimentation and iteration. Here
532are a couple of areas to experiment with:
533
534- ``glibc``: In general, follow this process:
535
536 #. Remove ``glibc`` features from
537 :term:`DISTRO_FEATURES`
538 that you think you do not need.
539
540 #. Build your distribution.
541
542 #. If the build fails due to missing symbols in a package, determine
543 if you can reconfigure the package to not need those features. For
544 example, change the configuration to not support wide character
545 support as is done for ``ncurses``. Or, if support for those
546 characters is needed, determine what ``glibc`` features provide
547 the support and restore the configuration.
548
549 4. Rebuild and repeat the process.
550
551- ``busybox``: For BusyBox, use a process similar as described for
552 ``glibc``. A difference is you will need to boot the resulting system
553 to see if you are able to do everything you expect from the running
554 system. You need to be sure to integrate configuration fragments into
555 Busybox because BusyBox handles its own core features and then allows
556 you to add configuration fragments on top.
557
558Iterate on the Process
559----------------------
560
561If you have not reached your goals on system size, you need to iterate
562on the process. The process is the same. Use the tools and see just what
563is taking up 90% of the root filesystem and the kernel. Decide what you
564can eliminate without limiting your device beyond what you need.
565
566Depending on your system, a good place to look might be Busybox, which
567provides a stripped down version of Unix tools in a single, executable
568file. You might be able to drop virtual terminal services or perhaps
569ipv6.
570
571Building Images for More than One Machine
572=========================================
573
574A common scenario developers face is creating images for several
575different machines that use the same software environment. In this
576situation, it is tempting to set the tunings and optimization flags for
577each build specifically for the targeted hardware (i.e. "maxing out" the
578tunings). Doing so can considerably add to build times and package feed
579maintenance collectively for the machines. For example, selecting tunes
580that are extremely specific to a CPU core used in a system might enable
581some micro optimizations in GCC for that particular system but would
582otherwise not gain you much of a performance difference across the other
583systems as compared to using a more general tuning across all the builds
584(e.g. setting :term:`DEFAULTTUNE`
585specifically for each machine's build). Rather than "max out" each
586build's tunings, you can take steps that cause the OpenEmbedded build
587system to reuse software across the various machines where it makes
588sense.
589
590If build speed and package feed maintenance are considerations, you
591should consider the points in this section that can help you optimize
592your tunings to best consider build times and package feed maintenance.
593
594- *Share the* :term:`Build Directory` *:* If at all possible, share the
595 :term:`TMPDIR` across builds. The Yocto Project supports switching between
596 different :term:`MACHINE` values in the same :term:`TMPDIR`. This practice
597 is well supported and regularly used by developers when building for
598 multiple machines. When you use the same :term:`TMPDIR` for multiple
599 machine builds, the OpenEmbedded build system can reuse the existing native
600 and often cross-recipes for multiple machines. Thus, build time decreases.
601
602 .. note::
603
604 If :term:`DISTRO` settings change or fundamental configuration settings
605 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
606 Sharing :term:`TMPDIR` under these circumstances might work but since it is
607 not guaranteed, you should use a clean :term:`TMPDIR`.
608
609- *Enable the Appropriate Package Architecture:* By default, the
610 OpenEmbedded build system enables three levels of package
611 architectures: "all", "tune" or "package", and "machine". Any given
612 recipe usually selects one of these package architectures (types) for
613 its output. Depending for what a given recipe creates packages,
614 making sure you enable the appropriate package architecture can
615 directly impact the build time.
616
617 A recipe that just generates scripts can enable "all" architecture
618 because there are no binaries to build. To specifically enable "all"
619 architecture, be sure your recipe inherits the
620 :ref:`ref-classes-allarch` class.
621 This class is useful for "all" architectures because it configures
622 many variables so packages can be used across multiple architectures.
623
624 If your recipe needs to generate packages that are machine-specific
625 or when one of the build or runtime dependencies is already
626 machine-architecture dependent, which makes your recipe also
627 machine-architecture dependent, make sure your recipe enables the
628 "machine" package architecture through the
629 :term:`MACHINE_ARCH`
630 variable::
631
632 PACKAGE_ARCH = "${MACHINE_ARCH}"
633
634 When you do not
635 specifically enable a package architecture through the
636 :term:`PACKAGE_ARCH`, The
637 OpenEmbedded build system defaults to the
638 :term:`TUNE_PKGARCH` setting::
639
640 PACKAGE_ARCH = "${TUNE_PKGARCH}"
641
642- *Choose a Generic Tuning File if Possible:* Some tunes are more
643 generic and can run on multiple targets (e.g. an ``armv5`` set of
644 packages could run on ``armv6`` and ``armv7`` processors in most
645 cases). Similarly, ``i486`` binaries could work on ``i586`` and
646 higher processors. You should realize, however, that advances on
647 newer processor versions would not be used.
648
649 If you select the same tune for several different machines, the
650 OpenEmbedded build system reuses software previously built, thus
651 speeding up the overall build time. Realize that even though a new
652 sysroot for each machine is generated, the software is not recompiled
653 and only one package feed exists.
654
655- *Manage Granular Level Packaging:* Sometimes there are cases where
656 injecting another level of package architecture beyond the three
657 higher levels noted earlier can be useful. For example, consider how
658 NXP (formerly Freescale) allows for the easy reuse of binary packages
659 in their layer
660 :yocto_git:`meta-freescale </meta-freescale/>`.
661 In this example, the
662 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
663 class shares GPU packages for i.MX53 boards because all boards share
664 the AMD GPU. The i.MX6-based boards can do the same because all
665 boards share the Vivante GPU. This class inspects the BitBake
666 datastore to identify if the package provides or depends on one of
667 the sub-architecture values. If so, the class sets the
668 :term:`PACKAGE_ARCH` value
669 based on the ``MACHINE_SUBARCH`` value. If the package does not
670 provide or depend on one of the sub-architecture values but it
671 matches a value in the machine-specific filter, it sets
672 :term:`MACHINE_ARCH`. This
673 behavior reduces the number of packages built and saves build time by
674 reusing binaries.
675
676- *Use Tools to Debug Issues:* Sometimes you can run into situations
677 where software is being rebuilt when you think it should not be. For
678 example, the OpenEmbedded build system might not be using shared
679 state between machines when you think it should be. These types of
680 situations are usually due to references to machine-specific
681 variables such as :term:`MACHINE`,
682 :term:`SERIAL_CONSOLES`,
683 :term:`XSERVER`,
684 :term:`MACHINE_FEATURES`,
685 and so forth in code that is supposed to only be tune-specific or
686 when the recipe depends
687 (:term:`DEPENDS`,
688 :term:`RDEPENDS`,
689 :term:`RRECOMMENDS`,
690 :term:`RSUGGESTS`, and so forth)
691 on some other recipe that already has
692 :term:`PACKAGE_ARCH` defined
693 as "${MACHINE_ARCH}".
694
695 .. note::
696
697 Patches to fix any issues identified are most welcome as these
698 issues occasionally do occur.
699
700 For such cases, you can use some tools to help you sort out the
701 situation:
702
703 - ``state-diff-machines.sh``*:* You can find this tool in the
704 ``scripts`` directory of the Source Repositories. See the comments
705 in the script for information on how to use the tool.
706
707 - *BitBake's "-S printdiff" Option:* Using this option causes
708 BitBake to try to establish the most recent signature match
709 (e.g. in the shared state cache) and then compare matched signatures
710 to determine the stamps and delta where these two stamp trees diverge.
711
712Building Software from an External Source
713=========================================
714
715By default, the OpenEmbedded build system uses the :term:`Build Directory`
716when building source code. The build process involves fetching the source
717files, unpacking them, and then patching them if necessary before the build
718takes place.
719
720There are situations where you might want to build software from source
721files that are external to and thus outside of the OpenEmbedded build
722system. For example, suppose you have a project that includes a new BSP
723with a heavily customized kernel. And, you want to minimize exposing the
724build system to the development team so that they can focus on their
725project and maintain everyone's workflow as much as possible. In this
726case, you want a kernel source directory on the development machine
727where the development occurs. You want the recipe's
728:term:`SRC_URI` variable to point to
729the external directory and use it as is, not copy it.
730
731To build from software that comes from an external source, all you need to do
732is inherit the :ref:`ref-classes-externalsrc` class and then set
733the :term:`EXTERNALSRC` variable to point to your external source code. Here
734are the statements to put in your ``local.conf`` file::
735
736 INHERIT += "externalsrc"
737 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
738
739This next example shows how to accomplish the same thing by setting
740:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
741
742 EXTERNALSRC = "path"
743 EXTERNALSRC_BUILD = "path"
744
745.. note::
746
747 In order for these settings to take effect, you must globally or
748 locally inherit the :ref:`ref-classes-externalsrc` class.
749
750By default, :ref:`ref-classes-externalsrc` builds the source code in a
751directory separate from the external source directory as specified by
752:term:`EXTERNALSRC`. If you need
753to have the source built in the same directory in which it resides, or
754some other nominated directory, you can set
755:term:`EXTERNALSRC_BUILD`
756to point to that directory::
757
758 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
759
760.. note::
761
762 The values of :term:`EXTERNALSRC` and :term:`EXTERNALSRC_BUILD`
763 must be absolute paths.
764
765Replicating a Build Offline
766===========================
767
768It can be useful to take a "snapshot" of upstream sources used in a
769build and then use that "snapshot" later to replicate the build offline.
770To do so, you need to first prepare and populate your downloads
771directory your "snapshot" of files. Once your downloads directory is
772ready, you can use it at any time and from any machine to replicate your
773build.
774
775Follow these steps to populate your Downloads directory:
776
777#. *Create a Clean Downloads Directory:* Start with an empty downloads
778 directory (:term:`DL_DIR`). You
779 start with an empty downloads directory by either removing the files
780 in the existing directory or by setting :term:`DL_DIR` to point to either
781 an empty location or one that does not yet exist.
782
783#. *Generate Tarballs of the Source Git Repositories:* Edit your
784 ``local.conf`` configuration file as follows::
785
786 DL_DIR = "/home/your-download-dir/"
787 BB_GENERATE_MIRROR_TARBALLS = "1"
788
789 During
790 the fetch process in the next step, BitBake gathers the source files
791 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
792 the
793 :term:`BB_GENERATE_MIRROR_TARBALLS`
794 variable for more information.
795
796#. *Populate Your Downloads Directory Without Building:* Use BitBake to
797 fetch your sources but inhibit the build::
798
799 $ bitbake target --runonly=fetch
800
801 The downloads directory (i.e. ``${DL_DIR}``) now has
802 a "snapshot" of the source files in the form of tarballs, which can
803 be used for the build.
804
805#. *Optionally Remove Any Git or other SCM Subdirectories From the
806 Downloads Directory:* If you want, you can clean up your downloads
807 directory by removing any Git or other Source Control Management
808 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
809 already contain these subdirectories.
810
811Once your downloads directory has everything it needs regarding source
812files, you can create your "own-mirror" and build your target.
813Understand that you can use the files to build the target offline from
814any machine and at any time.
815
816Follow these steps to build your target using the files in the downloads
817directory:
818
819#. *Using Local Files Only:* Inside your ``local.conf`` file, add the
820 :term:`SOURCE_MIRROR_URL` variable, inherit the
821 :ref:`ref-classes-own-mirrors` class, and add the
822 :term:`BB_NO_NETWORK` variable to your ``local.conf``::
823
824 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
825 INHERIT += "own-mirrors"
826 BB_NO_NETWORK = "1"
827
828 The :term:`SOURCE_MIRROR_URL` and :ref:`ref-classes-own-mirrors`
829 class set up the system to use the downloads directory as your "own
830 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
831 BitBake's fetching process in step 3 stays local, which means files
832 from your "own-mirror" are used.
833
834#. *Start With a Clean Build:* You can start with a clean build by
835 removing the ``${``\ :term:`TMPDIR`\ ``}`` directory or using a new
836 :term:`Build Directory`.
837
838#. *Build Your Target:* Use BitBake to build your target::
839
840 $ bitbake target
841
842 The build completes using the known local "snapshot" of source
843 files from your mirror. The resulting tarballs for your "snapshot" of
844 source files are in the downloads directory.
845
846 .. note::
847
848 The offline build does not work if recipes attempt to find the
849 latest version of software by setting
850 :term:`SRCREV` to
851 ``${``\ :term:`AUTOREV`\ ``}``::
852
853 SRCREV = "${AUTOREV}"
854
855 When a recipe sets :term:`SRCREV` to
856 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
857 attempt to determine the latest version of software from the SCM.
858 Typically, recipes that use :term:`AUTOREV` are custom or modified
859 recipes. Recipes that reside in public repositories usually do not
860 use :term:`AUTOREV`.
861
862 If you do have recipes that use :term:`AUTOREV`, you can take steps to
863 still use the recipes in an offline build. Do the following:
864
865 #. Use a configuration generated by enabling :ref:`build
866 history <dev-manual/build-quality:maintaining build output quality>`.
867
868 #. Use the ``buildhistory-collect-srcrevs`` command to collect the
869 stored :term:`SRCREV` values from the build's history. For more
870 information on collecting these values, see the
871 ":ref:`dev-manual/build-quality:build history package information`"
872 section.
873
874 #. Once you have the correct source revisions, you can modify
875 those recipes to set :term:`SRCREV` to specific versions of the
876 software.
877
diff --git a/documentation/dev-manual/creating-fragments.rst b/documentation/dev-manual/creating-fragments.rst
deleted file mode 100644
index 7f437d7c94..0000000000
--- a/documentation/dev-manual/creating-fragments.rst
+++ /dev/null
@@ -1,146 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating New Configuration Fragments In Your Build
4**************************************************
5
6:term:`Configuration Fragments <Configuration Fragment>` define top level build
7configuration features that can be independently enabled and disabled using
8standard tooling. Such features are made of one or several build configuration
9statements that are either contained in a fragment file, or are set indirectly
10using the :term:`Built-in Fragment` mechanism.
11
12This section will describe how to create new fragments for your builds.
13
14There are two kinds of configuration fragments:
15
16- Standard :term:`Configuration Fragments <Configuration Fragment>` which a
17 stored in a file. These fragments include a summary and a description,
18 following by configuration statements.
19
20- :term:`Built-in Fragments <Built-in Fragment>` which can be used to assign a
21 value to a single variable and do not require a separate definition file.
22 They are especially useful when a list of possible values is very long (or
23 infinite).
24
25Creating A Standard Configuration Fragment
26==========================================
27
28By default, all configuration fragments are located within the
29``conf/fragments`` directory of a :term:`layer`. This location is defined by the
30:term:`OE_FRAGMENTS_PREFIX` variable which, in turn, is used as a parameter in an
31:ref:`addfragments <bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\`
32directive>` directive in :oe_git:`bitbake.conf </openembedded-core/tree/meta/conf/bitbake.conf>`.
33
34You can create one or more :term:`configuration fragment` files in your
35:term:`layer` in this directory. Let's take the following example, where
36``custom-fragment.conf`` is our custom fragment file::
37
38 meta-custom
39 ├── conf
40 │   ├── fragments
41 │   │   └── custom-fragment.conf
42 │   └── layer.conf
43 ...
44
45For our ``custom-fragment.conf`` file, the following variables **must** be set
46for our fragment to be considered a valid fragment by the :term:`OpenEmbedded
47Build System`:
48
49- :term:`BB_CONF_FRAGMENT_SUMMARY`: a one-line summary of this fragment.
50
51- :term:`BB_CONF_FRAGMENT_DESCRIPTION`: a description of this fragment.
52
53.. note::
54
55 The :term:`BB_CONF_FRAGMENT_SUMMARY` and :term:`BB_CONF_FRAGMENT_DESCRIPTION`
56 variables are also passed as parameters in an :ref:`addfragments
57 <bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\`
58 directive>` directive in :oe_git:`bitbake.conf
59 </openembedded-core/tree/meta/conf/bitbake.conf>`.
60
61After creating these variables, our custom fragment should look like the
62following:
63
64.. code-block::
65 :caption: custom-fragment.conf
66
67 BB_CONF_FRAGMENT_SUMMARY = "This fragment sets a limit of 4 bitbake threads and 4 parsing threads"
68 BB_CONF_FRAGMENT_DESCRIPTION = "This fragment is useful to constrain resource consumption when the Yocto default \
69 is causing an overload of host machine's memory and CPU resources."
70
71For now, our fragment does not have any additional configuration statement.
72Let's add the following assignments to our fragment:
73
74.. code-block::
75 :caption: custom-fragment.conf (continued)
76
77 BB_NUMBER_THREADS = "4"
78 BB_NUMBER_PARSE_THREADS = "4"
79
80This means that our fragment can be enabled to set a limit on the number of
81threads :term:`BitBake` will use with the :term:`BB_NUMBER_THREADS` and
82:term:`BB_NUMBER_PARSE_THREADS` variables.
83
84For now, our fragment exists and is listed by the
85:ref:`ref-bitbake-config-build-list-fragments` command, but is not enabled. To
86enable this fragment, use the :ref:`ref-bitbake-config-build-enable-fragment`
87command::
88
89 bitbake-config-build enable-fragment meta-custom/custom-fragment
90
91.. note::
92
93 The ``meta-custom`` prefix in the above command depends on the name of your
94 layer. This name is defined by the :term:`BBFILE_COLLECTIONS` variable in
95 the ``conf/layer.conf`` file of your layer.
96
97Standard Configuration fragments can be organized in a more complex way. For
98example, it's possible to create sub-directories to organize your fragments::
99
100 meta-custom
101 ├── conf
102 │   ├── fragments
103 │   │   ├── networking
104 │   │   │   └── mirrors.conf
105 │   │   └── resources
106 │   │   └── numberthreads.conf
107 │   └── layer.conf
108 ...
109
110In the above example, the ``meta-custom/networking/mirrors`` and
111``meta-custom/resources/numberthreads`` fragments will be available in your
112build.
113
114Creating A Built-in Fragment
115============================
116
117Within the :term:`OpenEmbedded Build System`, Built-in Fragments are defined
118with the :term:`OE_FRAGMENTS_BUILTIN` variable, which is passed as a
119parameter in an :ref:`addfragments <bitbake-user-manual/bitbake-user-manual-metadata:\`\`addfragments\`\`
120directive>` directive in :oe_git:`bitbake.conf </openembedded-core/tree/meta/conf/bitbake.conf>`.
121
122Adding new :term:`Built-in Fragments <Built-in Fragment>` can be done by
123appending the :term:`OE_FRAGMENTS_BUILTIN` variable from your :term:`layer`
124configuration file:
125
126.. code-block::
127 :caption: layer.conf
128
129 OE_FRAGMENTS_BUILTIN:append = " custom-builtin-fragment:CUSTOM_VARIABLE"
130
131.. warning::
132
133 Make sure to use the ``:append`` override in the above assignment, as using
134 ``+=`` can lead to unexpected behavior.
135
136.. warning::
137
138 Due to the way :term:`BitBake` parses files, it is not possible to modify
139 :term:`OE_FRAGMENTS_BUILTIN` from any kind of :term:`configuration file`.
140 Setting it from the :term:`layer` configuration file (``conf/layer.conf``) is
141 the retained solution to create new built-in fragments.
142
143You can then use the :ref:`ref-bitbake-config-build-enable-fragment` command to
144set a value to the ``CUSTOM_VARIABLE`` variable::
145
146 bitbake-config-build enable-fragment custom-builtin-fragment/somevalue
diff --git a/documentation/dev-manual/custom-distribution.rst b/documentation/dev-manual/custom-distribution.rst
deleted file mode 100644
index 0bc386d606..0000000000
--- a/documentation/dev-manual/custom-distribution.rst
+++ /dev/null
@@ -1,135 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating Your Own Distribution
4******************************
5
6When you build an image using the Yocto Project and do not alter any
7distribution :term:`Metadata`, you are using the Poky distribution.
8Poky is explicitly a *reference* distribution for testing and
9development purposes. It enables most hardware and software features
10so that they can be tested, but this also means that from a security
11point of view the attack surface is very large. Additionally, at some
12point it is likely that you will want to gain more control over package
13alternative selections, compile-time options, and other low-level
14configurations. For both of these reasons, if you are using the Yocto
15Project for production use then you are strongly encouraged to create
16your own distribution.
17
18To create your own distribution, the basic steps consist of creating
19your own distribution layer, creating your own distribution
20configuration file, and then adding any needed code and Metadata to the
21layer. The following steps provide some more detail:
22
23- *Create a layer for your new distro:* Create your distribution layer
24 so that you can keep your Metadata and code for the distribution
25 separate. It is strongly recommended that you create and use your own
26 layer for configuration and code. Using your own layer as compared to
27 just placing configurations in a ``local.conf`` configuration file
28 makes it easier to reproduce the same build configuration when using
29 multiple build machines. See the
30 ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
31 section for information on how to quickly set up a layer.
32
33- *Create the distribution configuration file:* The distribution
34 configuration file needs to be created in the ``conf/distro``
35 directory of your layer. You need to name it using your distribution
36 name (e.g. ``mydistro.conf``).
37
38 .. note::
39
40 The :term:`DISTRO` variable in your ``local.conf`` file determines the
41 name of your distribution.
42
43 You can split out parts of your configuration file into include files
44 and then "require" them from within your distribution configuration
45 file. Be sure to place the include files in the
46 ``conf/distro/include`` directory of your layer. A common example
47 usage of include files would be to separate out the selection of
48 desired version and revisions for individual recipes.
49
50 Your configuration file needs to set the following required
51 variables:
52
53 - :term:`DISTRO_NAME`
54
55 - :term:`DISTRO_VERSION`
56
57 These following variables are optional and you typically set them
58 from the distribution configuration file:
59
60 - :term:`DISTRO_FEATURES`
61
62 - :term:`DISTRO_EXTRA_RDEPENDS`
63
64 - :term:`DISTRO_EXTRA_RRECOMMENDS`
65
66 - :term:`TCLIBC`
67
68 .. tip::
69
70 If you want to base your distribution configuration file on the
71 very basic configuration from OE-Core, you can use
72 ``conf/distro/defaultsetup.conf`` as a reference and just include
73 variables that differ as compared to ``defaultsetup.conf``.
74 Alternatively, you can create a distribution configuration file
75 from scratch using the ``defaultsetup.conf`` file or configuration files
76 from another distribution such as Poky as a reference.
77
78- *Provide miscellaneous variables:* Be sure to define any other
79 variables for which you want to create a default or enforce as part
80 of the distribution configuration. You can include nearly any
81 variable from the ``local.conf`` file. The variables you use are not
82 limited to the list in the previous bulleted item.
83
84- *Point to Your distribution configuration file:* In your ``local.conf``
85 file in the :term:`Build Directory`, set your :term:`DISTRO` variable to
86 point to your distribution's configuration file. For example, if your
87 distribution's configuration file is named ``mydistro.conf``, then
88 you point to it as follows::
89
90 DISTRO = "mydistro"
91
92- *Add more to the layer if necessary:* Use your layer to hold other
93 information needed for the distribution:
94
95 - Add recipes for installing distro-specific configuration files
96 that are not already installed by another recipe. If you have
97 distro-specific configuration files that are included by an
98 existing recipe, you should add an append file (``.bbappend``) for
99 those. For general information and recommendations on how to add
100 recipes to your layer, see the
101 ":ref:`dev-manual/layers:creating your own layer`" and
102 ":ref:`dev-manual/layers:following best practices when creating layers`"
103 sections.
104
105 - Add any image recipes that are specific to your distribution.
106
107 - Add a ``psplash`` append file for a branded splash screen, using
108 the :term:`SPLASH_IMAGES` variable.
109
110 - Add any other append files to make custom changes that are
111 specific to individual recipes.
112
113 For information on append files, see the
114 ":ref:`dev-manual/layers:appending other layers metadata with your layer`"
115 section.
116
117Copying and modifying the Poky distribution
118===========================================
119
120Instead of creating a custom distribution from scratch as per above, you may
121wish to start your custom distribution configuration by copying the Poky
122distribution provided within the ``meta-poky`` layer and then modifying it.
123This is fine, however if you do this you should keep the following in mind:
124
125- Every reference to Poky needs to be updated in your copy so that it
126 will still apply. This includes override usage within files (e.g. ``:poky``)
127 and in directory names. This is a good opportunity to evaluate each one of
128 these customizations to see if they are needed for your use case.
129
130- Unless you also intend to use them, the ``poky-tiny``, ``poky-altcfg`` and
131 ``poky-bleeding`` variants and any references to them can be removed.
132
133- More generally, the Poky distribution configuration enables a lot more
134 than you likely need for your production use case. You should evaluate *every*
135 configuration choice made in your copy to determine if it is needed.
diff --git a/documentation/dev-manual/custom-template-configuration-directory.rst b/documentation/dev-manual/custom-template-configuration-directory.rst
deleted file mode 100644
index 06fcada822..0000000000
--- a/documentation/dev-manual/custom-template-configuration-directory.rst
+++ /dev/null
@@ -1,52 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Custom Template Configuration Directory
4**************************************************
5
6If you are producing your own customized version of the build system for
7use by other users, you might want to provide a custom build configuration
8that includes all the necessary settings and layers (i.e. ``local.conf`` and
9``bblayers.conf`` that are created in a new :term:`Build Directory`) and a custom
10message that is shown when setting up the build. This can be done by
11creating one or more template configuration directories in your
12custom distribution layer.
13
14This can be done by using ``bitbake-layers save-build-conf``::
15
16 $ bitbake-layers save-build-conf ../../meta-alex/ test-1
17 NOTE: Starting bitbake server...
18 NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
19 Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
20 You can try out the configuration with
21 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
22
23The above command takes the config files from the currently active :term:`Build Directory` under ``conf``,
24replaces site-specific paths in ``bblayers.conf`` with ``##OECORE##``-relative paths, and copies
25the config files into a specified layer under a specified template name.
26
27To use those saved templates as a starting point for a build, users should point
28to one of them with :term:`TEMPLATECONF` environment variable::
29
30 TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
31
32The OpenEmbedded build system uses the environment variable
33:term:`TEMPLATECONF` to locate the directory from which it gathers
34configuration information that ultimately ends up in the
35:term:`Build Directory` ``conf`` directory.
36
37If :term:`TEMPLATECONF` is not set, the default value is obtained
38from ``.templateconf`` file that is read from the same directory as
39``oe-init-build-env`` script. For the Poky reference distribution this
40would be::
41
42 TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf/templates/default}
43
44If you look at a configuration template directory, you will
45see the ``bblayers.conf.sample``, ``local.conf.sample``, ``conf-summary.txt`` and
46``conf-notes.txt`` files. The build system uses these files to form the
47respective ``bblayers.conf`` file, ``local.conf`` file, and show
48users usage information about the build they're setting up
49when running the ``oe-init-build-env`` setup script. These can be
50edited further if needed to improve or change the build configurations
51available to the users, and provide useful summaries and detailed usage notes.
52
diff --git a/documentation/dev-manual/customizing-images.rst b/documentation/dev-manual/customizing-images.rst
deleted file mode 100644
index 6eed9ef56a..0000000000
--- a/documentation/dev-manual/customizing-images.rst
+++ /dev/null
@@ -1,221 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Customizing Images
4******************
5
6You can customize images to satisfy particular requirements. This
7section describes several methods and provides guidelines for each.
8
9Customizing Images Using ``local.conf``
10=======================================
11
12Probably the easiest way to customize an image is to add a package by
13way of the ``local.conf`` configuration file. Because it is limited to
14local use, this method generally only allows you to add packages and is
15not as flexible as creating your own customized image. When you add
16packages using local variables this way, you need to realize that these
17variable changes are in effect for every build and consequently affect
18all images, which might not be what you require.
19
20To add a package to your image using the local configuration file, use
21the :term:`IMAGE_INSTALL` variable with the ``:append`` operator::
22
23 IMAGE_INSTALL:append = " strace"
24
25Use of the syntax is important; specifically, the leading space
26after the opening quote and before the package name, which is
27``strace`` in this example. This space is required since the ``:append``
28operator does not add the space.
29
30Furthermore, you must use ``:append`` instead of the ``+=`` operator if
31you want to avoid ordering issues. The reason for this is because doing
32so unconditionally appends to the variable and avoids ordering problems
33due to the variable being set in image recipes and ``.bbclass`` files
34with operators like ``?=``. Using ``:append`` ensures the operation
35takes effect.
36
37As shown in its simplest use, ``IMAGE_INSTALL:append`` affects all
38images. It is possible to extend the syntax so that the variable applies
39to a specific image only. Here is an example::
40
41 IMAGE_INSTALL:append:pn-core-image-minimal = " strace"
42
43This example adds ``strace`` to the ``core-image-minimal`` image only.
44
45You can add packages using a similar approach through the
46:term:`CORE_IMAGE_EXTRA_INSTALL` variable. If you use this variable, only
47``core-image-*`` images are affected.
48
49Customizing Images Using Custom ``IMAGE_FEATURES`` and ``EXTRA_IMAGE_FEATURES``
50===============================================================================
51
52Another method for customizing your image is to enable or disable
53high-level image features by using the
54:term:`IMAGE_FEATURES` and
55:term:`EXTRA_IMAGE_FEATURES`
56variables. Although the functions for both variables are nearly
57equivalent, best practices dictate using :term:`IMAGE_FEATURES` from within
58a recipe and using :term:`EXTRA_IMAGE_FEATURES` from within your
59``local.conf`` file, which is found in the :term:`Build Directory`.
60
61To understand how these features work, the best reference is
62:ref:`meta/classes-recipe/image.bbclass <ref-classes-image>`.
63This class lists out the available
64:term:`IMAGE_FEATURES` of which most map to package groups while some, such
65as ``read-only-rootfs``, resolve as general configuration settings.
66
67In summary, the file looks at the contents of the :term:`IMAGE_FEATURES`
68variable and then maps or configures the feature accordingly. Based on
69this information, the build system automatically adds the appropriate
70packages or configurations to the
71:term:`IMAGE_INSTALL` variable.
72Effectively, you are enabling extra features by extending the class or
73creating a custom class for use with specialized image ``.bb`` files.
74
75Use the :term:`EXTRA_IMAGE_FEATURES` variable from within your local
76configuration file. Using a separate area from which to enable features
77with this variable helps you avoid overwriting the features in the image
78recipe that are enabled with :term:`IMAGE_FEATURES`. The value of
79:term:`EXTRA_IMAGE_FEATURES` is added to :term:`IMAGE_FEATURES` within
80``meta/conf/bitbake.conf``.
81
82To illustrate how you can use these variables to modify your image, consider an
83example that selects the SSH server. The Yocto Project ships with two SSH
84servers you can use with your images: Dropbear and OpenSSH. Dropbear is a
85minimal SSH server appropriate for resource-constrained environments, while
86OpenSSH is a well-known standard SSH server implementation. By default, the
87``core-image-sato`` image is configured to use Dropbear. The
88``core-image-full-cmdline`` image includes OpenSSH. The ``core-image-minimal``
89image does not contain an SSH server.
90
91You can customize your image and change these defaults. Edit the
92:term:`IMAGE_FEATURES` variable in your recipe or use the
93:term:`EXTRA_IMAGE_FEATURES` in your ``local.conf`` file so that it
94configures the image you are working with to include
95``ssh-server-dropbear`` or ``ssh-server-openssh``.
96
97.. note::
98
99 See the ":ref:`ref-manual/features:image features`" section in the Yocto
100 Project Reference Manual for a complete list of image features that ship
101 with the Yocto Project.
102
103Customizing Images Using Custom .bb Files
104=========================================
105
106You can also customize an image by creating a custom recipe that defines
107additional software as part of the image. The following example shows
108the form for the two lines you need::
109
110 IMAGE_INSTALL = "packagegroup-core-x11-base package1 package2"
111 inherit core-image
112
113Defining the software using a custom recipe gives you total control over
114the contents of the image. It is important to use the correct names of
115packages in the :term:`IMAGE_INSTALL` variable. You must use the
116OpenEmbedded notation and not the Debian notation for the names (e.g.
117``glibc-dev`` instead of ``libc6-dev``).
118
119The other method for creating a custom image is to base it on an
120existing image. For example, if you want to create an image based on
121``core-image-sato`` but add the additional package ``strace`` to the
122image, copy the ``meta/recipes-sato/images/core-image-sato.bb`` to a new
123``.bb`` and add the following line to the end of the copy::
124
125 IMAGE_INSTALL += "strace"
126
127Customizing Images Using Custom Package Groups
128==============================================
129
130For complex custom images, the best approach for customizing an image is
131to create a custom package group recipe that is used to build the image
132or images. A good example of a package group recipe is
133``meta/recipes-core/packagegroups/packagegroup-base.bb``.
134
135If you examine that recipe, you see that the :term:`PACKAGES` variable lists
136the package group packages to produce. The ``inherit packagegroup``
137statement sets appropriate default values and automatically adds
138``-dev``, ``-dbg``, and ``-ptest`` complementary packages for each
139package specified in the :term:`PACKAGES` statement.
140
141.. note::
142
143 The ``inherit packagegroup`` line should be located near the top of the
144 recipe, certainly before the :term:`PACKAGES` statement.
145
146For each package you specify in :term:`PACKAGES`, you can use :term:`RDEPENDS`
147and :term:`RRECOMMENDS` entries to provide a list of packages the parent
148task package should contain. You can see examples of these further down
149in the ``packagegroup-base.bb`` recipe.
150
151Here is a short, fabricated example showing the same basic pieces for a
152hypothetical packagegroup defined in ``packagegroup-custom.bb``, where
153the variable :term:`PN` is the standard way to abbreviate the reference to
154the full packagegroup name ``packagegroup-custom``::
155
156 DESCRIPTION = "My Custom Package Groups"
157
158 inherit packagegroup
159
160 PACKAGES = "\
161 ${PN}-apps \
162 ${PN}-tools \
163 "
164
165 RDEPENDS:${PN}-apps = "\
166 dropbear \
167 portmap \
168 psplash"
169
170 RDEPENDS:${PN}-tools = "\
171 oprofile \
172 oprofileui-server \
173 lttng-tools"
174
175 RRECOMMENDS:${PN}-tools = "\
176 kernel-module-oprofile"
177
178In the previous example, two package group packages are created with
179their dependencies and their recommended package dependencies listed:
180``packagegroup-custom-apps``, and ``packagegroup-custom-tools``. To
181build an image using these package group packages, you need to add
182``packagegroup-custom-apps`` and/or ``packagegroup-custom-tools`` to
183:term:`IMAGE_INSTALL`. For other forms of image dependencies see the other
184areas of this section.
185
186Customizing an Image Hostname
187=============================
188
189By default, the configured hostname (i.e. ``/etc/hostname``) in an image
190is the same as the machine name. For example, if
191:term:`MACHINE` equals "qemux86", the
192configured hostname written to ``/etc/hostname`` is "qemux86".
193
194You can customize this name by altering the value of the "hostname"
195variable in the ``base-files`` recipe using either an append file or a
196configuration file. Use the following in an append file::
197
198 hostname = "myhostname"
199
200Use the following in a configuration file::
201
202 hostname:pn-base-files = "myhostname"
203
204Changing the default value of the variable "hostname" can be useful in
205certain situations. For example, suppose you need to do extensive
206testing on an image and you would like to easily identify the image
207under test from existing images with typical default hostnames. In this
208situation, you could change the default hostname to "testme", which
209results in all the images using the name "testme". Once testing is
210complete and you do not need to rebuild the image for test any longer,
211you can easily reset the default hostname.
212
213Another point of interest is that if you unset the variable, the image
214will have no default hostname in the filesystem. Here is an example that
215unsets the variable in a configuration file::
216
217 hostname:pn-base-files = ""
218
219Having no default hostname in the filesystem is suitable for
220environments that use dynamic hostnames such as virtual machines.
221
diff --git a/documentation/dev-manual/debugging.rst b/documentation/dev-manual/debugging.rst
deleted file mode 100644
index 6c45ccf652..0000000000
--- a/documentation/dev-manual/debugging.rst
+++ /dev/null
@@ -1,1271 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Debugging Tools and Techniques
4******************************
5
6The exact method for debugging build failures depends on the nature of
7the problem and on the system's area from which the bug originates.
8Standard debugging practices such as comparison against the last known
9working version with examination of the changes and the re-application
10of steps to identify the one causing the problem are valid for the Yocto
11Project just as they are for any other system. Even though it is
12impossible to detail every possible potential failure, this section
13provides some general tips to aid in debugging given a variety of
14situations.
15
16.. note::
17
18 A useful feature for debugging is the error reporting tool.
19 Configuring the Yocto Project to use this tool causes the
20 OpenEmbedded build system to produce error reporting commands as part
21 of the console output. You can enter the commands after the build
22 completes to log error information into a common database, that can
23 help you figure out what might be going wrong. For information on how
24 to enable and use this feature, see the
25 ":ref:`dev-manual/error-reporting-tool:using the error reporting tool`"
26 section.
27
28The following list shows the debugging topics in the remainder of this
29section:
30
31- ":ref:`dev-manual/debugging:viewing logs from failed tasks`" describes
32 how to find and view logs from tasks that failed during the build
33 process.
34
35- ":ref:`dev-manual/debugging:viewing variable values`" describes how to
36 use the BitBake ``-e`` option to examine variable values after a
37 recipe has been parsed.
38
39- ":ref:`dev-manual/debugging:viewing package information with ``oe-pkgdata-util```"
40 describes how to use the ``oe-pkgdata-util`` utility to query
41 :term:`PKGDATA_DIR` and
42 display package-related information for built packages.
43
44- ":ref:`dev-manual/debugging:viewing dependencies between recipes and tasks`"
45 describes how to use the BitBake ``-g`` option to display recipe
46 dependency information used during the build.
47
48- ":ref:`dev-manual/debugging:viewing task variable dependencies`" describes
49 how to use the ``bitbake-dumpsig`` command in conjunction with key
50 subdirectories in the :term:`Build Directory` to determine variable
51 dependencies.
52
53- ":ref:`dev-manual/debugging:running specific tasks`" describes
54 how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``)
55 to run specific tasks in the build chain. It can be useful to run
56 tasks "out-of-order" when trying isolate build issues.
57
58- ":ref:`dev-manual/debugging:general BitBake problems`" describes how
59 to use BitBake's ``-D`` debug output option to reveal more about what
60 BitBake is doing during the build.
61
62- ":ref:`dev-manual/debugging:building with no dependencies`"
63 describes how to use the BitBake ``-b`` option to build a recipe
64 while ignoring dependencies.
65
66- ":ref:`dev-manual/debugging:recipe logging mechanisms`"
67 describes how to use the many recipe logging functions to produce
68 debugging output and report errors and warnings.
69
70- ":ref:`dev-manual/debugging:debugging parallel make races`"
71 describes how to debug situations where the build consists of several
72 parts that are run simultaneously and when the output or result of
73 one part is not ready for use with a different part of the build that
74 depends on that output.
75
76- ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) remotely`"
77 describes how to use GDB to allow you to examine running programs, which can
78 help you fix problems.
79
80- ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) on the target`"
81 describes how to use GDB directly on target hardware for debugging.
82
83- ":ref:`dev-manual/debugging:other debugging tips`" describes
84 miscellaneous debugging tips that can be useful.
85
86Viewing Logs from Failed Tasks
87==============================
88
89You can find the log for a task in the file
90``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`.
91For example, the log for the
92:ref:`ref-tasks-compile` task of the
93QEMU minimal image for the x86 machine (``qemux86``) might be in
94``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``.
95To see the commands :term:`BitBake` ran
96to generate a log, look at the corresponding ``run.do_``\ `taskname` file
97in the same directory.
98
99``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic
100links to ``log.do_``\ `taskname`\ ``.``\ `pid` and
101``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had
102when it ran. The symlinks always point to the files corresponding to the
103most recent run.
104
105Viewing Variable Values
106=======================
107
108Sometimes you need to know the value of a variable as a result of
109BitBake's parsing step. This could be because some unexpected behavior
110occurred in your project. Perhaps an attempt to :ref:`modify a variable
111<bitbake-user-manual/bitbake-user-manual-metadata:modifying existing
112variables>` did not work out as expected.
113
114BitBake's ``-e`` option is used to display variable values after
115parsing. The following command displays the variable values after the
116configuration files (i.e. ``local.conf``, ``bblayers.conf``,
117``bitbake.conf`` and so forth) have been parsed::
118
119 $ bitbake -e
120
121The following command displays variable values after a specific recipe has
122been parsed. The variables include those from the configuration as well::
123
124 $ bitbake -e recipename
125
126.. note::
127
128 Each recipe has its own private set of variables (datastore).
129 Internally, after parsing the configuration, a copy of the resulting
130 datastore is made prior to parsing each recipe. This copying implies
131 that variables set in one recipe will not be visible to other
132 recipes.
133
134 Likewise, each task within a recipe gets a private datastore based on
135 the recipe datastore, which means that variables set within one task
136 will not be visible to other tasks.
137
138In the output of ``bitbake -e``, each variable is preceded by a
139description of how the variable got its value, including temporary
140values that were later overridden. This description also includes
141variable flags (varflags) set on the variable. The output can be very
142helpful during debugging.
143
144Variables that are exported to the environment are preceded by
145``export`` in the output of ``bitbake -e``. See the following example::
146
147 export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86"
148
149In addition to variable values, the output of the ``bitbake -e`` and
150``bitbake -e`` recipe commands includes the following information:
151
152- The output starts with a tree listing all configuration files and
153 classes included globally, recursively listing the files they include
154 or inherit in turn. Much of the behavior of the OpenEmbedded build
155 system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is
156 implemented in the :ref:`ref-classes-base` class and the
157 classes it inherits, rather than being built into BitBake itself.
158
159- After the variable values, all functions appear in the output. For
160 shell functions, variables referenced within the function body are
161 expanded. If a function has been modified using overrides or using
162 override-style operators like ``:append`` and ``:prepend``, then the
163 final assembled function body appears in the output.
164
165Viewing Package Information with ``oe-pkgdata-util``
166====================================================
167
168You can use the ``oe-pkgdata-util`` command-line utility to query
169:term:`PKGDATA_DIR` and display
170various package-related information. When you use the utility, you must
171use it to view information on packages that have already been built.
172
173Here are a few of the available ``oe-pkgdata-util`` subcommands.
174
175.. note::
176
177 You can use the standard \* and ? globbing wildcards as part of
178 package names and paths.
179
180- ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages
181 that have been built, optionally limiting the match to packages that
182 match pattern.
183
184- ``oe-pkgdata-util list-pkg-files package ...``: Lists the
185 files and directories contained in the given packages.
186
187 .. note::
188
189 A different way to view the contents of a package is to look at
190 the
191 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
192 directory of the recipe that generates the package. This directory
193 is created by the
194 :ref:`ref-tasks-package` task
195 and has one subdirectory for each package the recipe generates,
196 which contains the files stored in that package.
197
198 If you want to inspect the ``${WORKDIR}/packages-split``
199 directory, make sure that :ref:`ref-classes-rm-work` is not
200 enabled when you build the recipe.
201
202- ``oe-pkgdata-util find-path path ...``: Lists the names of
203 the packages that contain the given paths. For example, the following
204 tells us that ``/usr/share/man/man1/make.1`` is contained in the
205 ``make-doc`` package::
206
207 $ oe-pkgdata-util find-path /usr/share/man/man1/make.1
208 make-doc: /usr/share/man/man1/make.1
209
210- ``oe-pkgdata-util lookup-recipe package ...``: Lists the name
211 of the recipes that produce the given packages.
212
213For more information on the ``oe-pkgdata-util`` command, use the help
214facility::
215
216 $ oe-pkgdata-util --help
217 $ oe-pkgdata-util subcommand --help
218
219Viewing Dependencies Between Recipes and Tasks
220==============================================
221
222Sometimes it can be hard to see why BitBake wants to build other recipes
223before the one you have specified. Dependency information can help you
224understand why a recipe is built.
225
226To generate dependency information for a recipe, run the following
227command::
228
229 $ bitbake -g recipename
230
231This command writes the following files in the current directory:
232
233- ``pn-buildlist``: A list of recipes/targets involved in building
234 `recipename`. "Involved" here means that at least one task from the
235 recipe needs to run when building `recipename` from scratch. Targets
236 that are in
237 :term:`ASSUME_PROVIDED`
238 are not listed.
239
240- ``task-depends.dot``: A graph showing dependencies between tasks.
241
242The graphs are in :wikipedia:`DOT <DOT_%28graph_description_language%29>`
243format and can be converted to images (e.g. using the ``dot`` tool from
244`Graphviz <https://www.graphviz.org/>`__).
245
246.. note::
247
248 - DOT files use a plain text format. The graphs generated using the
249 ``bitbake -g`` command are often so large as to be difficult to
250 read without special pruning (e.g. with BitBake's ``-I`` option)
251 and processing. Despite the form and size of the graphs, the
252 corresponding ``.dot`` files can still be possible to read and
253 provide useful information.
254
255 As an example, the ``task-depends.dot`` file contains lines such
256 as the following::
257
258 "libxslt.do_configure" -> "libxml2.do_populate_sysroot"
259
260 The above example line reveals that the
261 :ref:`ref-tasks-configure`
262 task in ``libxslt`` depends on the
263 :ref:`ref-tasks-populate_sysroot`
264 task in ``libxml2``, which is a normal
265 :term:`DEPENDS` dependency
266 between the two recipes.
267
268 - For an example of how ``.dot`` files can be processed, see the
269 ``scripts/contrib/graph-tool`` Python script, which finds and
270 displays paths between graph nodes.
271
272You can use a different method to view dependency information by using
273either::
274
275 $ bitbake -g -u taskexp recipename
276
277or::
278
279 $ bitbake -g -u taskexp_ncurses recipename
280
281The ``-u taskdep`` option GUI window from which you can view build-time and
282runtime dependencies for the recipes involved in building recipename. The
283``-u taskexp_ncurses`` option uses ncurses instead of GTK to render the UI.
284
285Viewing Task Variable Dependencies
286==================================
287
288As mentioned in the
289":ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`"
290section of the BitBake User Manual, BitBake tries to automatically determine
291what variables a task depends on so that it can rerun the task if any values of
292the variables change. This determination is usually reliable. However, if you
293do things like construct variable names at runtime, then you might have to
294manually declare dependencies on those variables using ``vardeps`` as described
295in the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flags`"
296section of the BitBake User Manual.
297
298If you are unsure whether a variable dependency is being picked up
299automatically for a given task, you can list the variable dependencies
300BitBake has determined by doing the following:
301
302#. Build the recipe containing the task::
303
304 $ bitbake recipename
305
306#. Inside the :term:`STAMPS_DIR`
307 directory, find the signature data (``sigdata``) file that
308 corresponds to the task. The ``sigdata`` files contain a pickled
309 Python database of all the metadata that went into creating the input
310 checksum for the task. As an example, for the
311 :ref:`ref-tasks-fetch` task of the
312 ``db`` recipe, the ``sigdata`` file might be found in the following
313 location::
314
315 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
316
317 For tasks that are accelerated through the shared state
318 (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an
319 additional ``siginfo`` file is written into
320 :term:`SSTATE_DIR` along with
321 the cached task output. The ``siginfo`` files contain exactly the
322 same information as ``sigdata`` files.
323
324#. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here
325 is an example::
326
327 $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
328
329 In the output of the above command, you will find a line like the
330 following, which lists all the (inferred) variable dependencies for
331 the task. This list also includes indirect dependencies from
332 variables depending on other variables, recursively::
333
334 Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[sha256sum]', 'base_do_fetch']
335
336 .. note::
337
338 Functions (e.g. ``base_do_fetch``) also count as variable dependencies.
339 These functions in turn depend on the variables they reference.
340
341 The output of ``bitbake-dumpsig`` also includes the value each
342 variable had, a list of dependencies for each variable, and
343 :term:`BB_BASEHASH_IGNORE_VARS`
344 information.
345
346Debugging signature construction and unexpected task executions
347===============================================================
348
349There is a ``bitbake-diffsigs`` command for comparing two
350``siginfo`` or ``sigdata`` files. This command can be helpful when
351trying to figure out what changed between two versions of a task. If you
352call ``bitbake-diffsigs`` with just one file, the command behaves like
353``bitbake-dumpsig``.
354
355You can also use BitBake to dump out the signature construction
356information without executing tasks by using either of the following
357BitBake command-line options::
358
359 ‐‐dump-signatures=SIGNATURE_HANDLER
360 -S SIGNATURE_HANDLER
361
362
363.. note::
364
365 Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which
366 dump only the signature or compare the dumped signature with the most recent one,
367 respectively. "printdiff" will try to establish the most recent
368 signature match (e.g. in the sstate cache) and then
369 compare the matched signatures to determine the stamps and delta
370 where these two stamp trees diverge. This can be used to determine why
371 tasks need to be re-run in situations where that is not expected.
372
373Using BitBake with either of these options causes BitBake to dump out
374``sigdata`` files in the ``stamps`` directory for every task it would
375have executed instead of building the specified target package.
376
377Viewing Metadata Used to Create the Input Signature of a Shared State Task
378==========================================================================
379
380Seeing what metadata went into creating the input signature of a shared
381state (sstate) task can be a useful debugging aid. This information is
382available in signature information (``siginfo``) files in
383:term:`SSTATE_DIR`. For
384information on how to view and interpret information in ``siginfo``
385files, see the
386":ref:`dev-manual/debugging:viewing task variable dependencies`" section.
387
388For conceptual information on shared state, see the
389":ref:`overview-manual/concepts:shared state`"
390section in the Yocto Project Overview and Concepts Manual.
391
392Invalidating Shared State to Force a Task to Run
393================================================
394
395The OpenEmbedded build system uses
396:ref:`checksums <overview-manual/concepts:checksums (signatures)>` and
397:ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily
398rebuilding tasks. Collectively, this scheme is known as "shared state
399code".
400
401As with all schemes, this one has some drawbacks. It is possible that
402you could make implicit changes to your code that the checksum
403calculations do not take into account. These implicit changes affect a
404task's output but do not trigger the shared state code into rebuilding a
405recipe. Consider an example during which a tool changes its output.
406Assume that the output of ``rpmdeps`` changes. The result of the change
407should be that all the ``package`` and ``package_write_rpm`` shared
408state cache items become invalid. However, because the change to the
409output is external to the code and therefore implicit, the associated
410shared state cache items do not become invalidated. In this case, the
411build process uses the cached items rather than running the task again.
412Obviously, these types of implicit changes can cause problems.
413
414To avoid these problems during the build, you need to understand the
415effects of any changes you make. Realize that changes you make directly
416to a function are automatically factored into the checksum calculation.
417Thus, these explicit changes invalidate the associated area of shared
418state cache. However, you need to be aware of any implicit changes that
419are not obvious changes to the code and could affect the output of a
420given task.
421
422When you identify an implicit change, you can easily take steps to
423invalidate the cache and force the tasks to run. The steps you can take
424are as simple as changing a function's comments in the source code. For
425example, to invalidate package shared state files, change the comment
426statements of
427:ref:`ref-tasks-package` or the
428comments of one of the functions it calls. Even though the change is
429purely cosmetic, it causes the checksum to be recalculated and forces
430the build system to run the task again.
431
432.. note::
433
434 For an example of a commit that makes a cosmetic change to invalidate
435 shared state, see this
436 :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`.
437
438Running Specific Tasks
439======================
440
441Any given recipe consists of a set of tasks. The standard BitBake
442behavior in most cases is: :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`,
443:ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, :ref:`ref-tasks-install`, :ref:`ref-tasks-package`,
444:ref:`do_package_write_* <ref-tasks-package_write_deb>`, and :ref:`ref-tasks-build`. The default task is
445:ref:`ref-tasks-build` and any tasks on which it depends build first. Some tasks,
446such as :ref:`ref-tasks-devshell`, are not part of the default build chain. If you
447wish to run a task that is not part of the default build chain, you can
448use the ``-c`` option in BitBake. Here is an example::
449
450 $ bitbake matchbox-desktop -c devshell
451
452The ``-c`` option respects task dependencies, which means that all other
453tasks (including tasks from other recipes) that the specified task
454depends on will be run before the task. Even when you manually specify a
455task to run with ``-c``, BitBake will only run the task if it considers
456it "out of date". See the
457":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`"
458section in the Yocto Project Overview and Concepts Manual for how
459BitBake determines whether a task is "out of date".
460
461If you want to force an up-to-date task to be rerun (e.g. because you
462made manual modifications to the recipe's
463:term:`WORKDIR` that you want to try
464out), then you can use the ``-f`` option.
465
466.. note::
467
468 The reason ``-f`` is never required when running the
469 :ref:`ref-tasks-devshell` task is because the
470 [\ :ref:`nostamp <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ]
471 variable flag is already set for the task.
472
473The following example shows one way you can use the ``-f`` option::
474
475 $ bitbake matchbox-desktop
476 .
477 .
478 make some changes to the source code in the work directory
479 .
480 .
481 $ bitbake matchbox-desktop -c compile -f
482 $ bitbake matchbox-desktop
483
484This sequence first builds and then recompiles ``matchbox-desktop``. The
485last command reruns all tasks (basically the packaging tasks) after the
486compile. BitBake recognizes that the :ref:`ref-tasks-compile` task was rerun and
487therefore understands that the other tasks also need to be run again.
488
489Another, shorter way to rerun a task and all
490:ref:`ref-manual/tasks:normal recipe build tasks`
491that depend on it is to use the ``-C`` option.
492
493.. note::
494
495 This option is upper-cased and is separate from the ``-c``
496 option, which is lower-cased.
497
498Using this option invalidates the given task and then runs the
499:ref:`ref-tasks-build` task, which is
500the default task if no task is given, and the tasks on which it depends.
501You could replace the final two commands in the previous example with
502the following single command::
503
504 $ bitbake matchbox-desktop -C compile
505
506Internally, the ``-f`` and ``-C`` options work by tainting (modifying)
507the input checksum of the specified task. This tainting indirectly
508causes the task and its dependent tasks to be rerun through the normal
509task dependency mechanisms.
510
511.. note::
512
513 BitBake explicitly keeps track of which tasks have been tainted in
514 this fashion, and will print warnings such as the following for
515 builds involving such tasks:
516
517 .. code-block:: none
518
519 WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run
520
521
522 The purpose of the warning is to let you know that the work directory
523 and build output might not be in the clean state they would be in for
524 a "normal" build, depending on what actions you took. To get rid of
525 such warnings, you can remove the work directory and rebuild the
526 recipe, as follows::
527
528 $ bitbake matchbox-desktop -c clean
529 $ bitbake matchbox-desktop
530
531
532You can view a list of tasks in a given package by running the
533:ref:`ref-tasks-listtasks` task as follows::
534
535 $ bitbake matchbox-desktop -c listtasks
536
537The results appear as output to the console and are also in
538the file ``${WORKDIR}/temp/log.do_listtasks``.
539
540General BitBake Problems
541========================
542
543You can see debug output from BitBake by using the ``-D`` option. The
544debug output gives more information about what BitBake is doing and the
545reason behind it. Each ``-D`` option you use increases the logging
546level. The most common usage is ``-DDD``.
547
548The output from ``bitbake -DDD -v targetname`` can reveal why BitBake
549chose a certain version of a package or why BitBake picked a certain
550provider. This command could also help you in a situation where you
551think BitBake did something unexpected.
552
553Building with No Dependencies
554=============================
555
556To build a specific recipe (``.bb`` file), you can use the following
557command form::
558
559 $ bitbake -b somepath/somerecipe.bb
560
561This command form does
562not check for dependencies. Consequently, you should use it only when
563you know existing dependencies have been met.
564
565.. note::
566
567 You can also specify fragments of the filename. In this case, BitBake
568 checks for a unique match.
569
570Recipe Logging Mechanisms
571=========================
572
573The Yocto Project provides several logging functions for producing
574debugging output and reporting errors and warnings. For Python
575functions, the following logging functions are available. All of these functions
576log to ``${T}/log.do_``\ `task`, and can also log to standard output
577(stdout) with the right settings:
578
579- ``bb.plain(msg)``: Writes msg as is to the log while also
580 logging to stdout.
581
582- ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to
583 stdout if BitBake is called with "-v".
584
585- ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the log. Also logs to
586 stdout if the log level is greater than or equal to level. See the
587 ":ref:`bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`"
588 option in the BitBake User Manual for more information.
589
590- ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also
591 logging to stdout.
592
593- ``bb.error(msg)``: Writes "ERROR: msg" to the log while also
594 logging to standard out (stdout).
595
596 .. note::
597
598 Calling this function does not cause the task to fail.
599
600- ``bb.fatal(msg)``: This logging function is similar to
601 ``bb.error(msg)`` but also causes the calling task to fail.
602
603 .. note::
604
605 ``bb.fatal()`` raises an exception, which means you do not need to put a
606 "return" statement after the function.
607
608The same logging functions are also available in shell functions, under
609the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``,
610and ``bbfatal``. The :ref:`ref-classes-logging` class
611implements these functions. See that class in the ``meta/classes``
612folder of the :term:`Source Directory` for information.
613
614Logging With Python
615-------------------
616
617When creating recipes using Python and inserting code that handles build
618logs, keep in mind the goal is to have informative logs while keeping
619the console as "silent" as possible. Also, if you want status messages
620in the log, use the "debug" loglevel.
621
622Here is an example written in Python. The code handles logging for
623a function that determines the number of tasks needed to be run. See the
624":ref:`ref-tasks-listtasks`"
625section for additional information::
626
627 python do_listtasks() {
628 bb.debug(2, "Starting to figure out the task list")
629 if noteworthy_condition:
630 bb.note("There are 47 tasks to run")
631 bb.debug(2, "Got to point xyz")
632 if warning_trigger:
633 bb.warn("Detected warning_trigger, this might be a problem later.")
634 if recoverable_error:
635 bb.error("Hit recoverable_error, you really need to fix this!")
636 if fatal_error:
637 bb.fatal("fatal_error detected, unable to print the task list")
638 bb.plain("The tasks present are abc")
639 bb.debug(2, "Finished figuring out the tasklist")
640 }
641
642Logging With Bash
643-----------------
644
645When creating recipes using Bash and inserting code that handles build
646logs, you have the same goals --- informative with minimal console output.
647The syntax you use for recipes written in Bash is similar to that of
648recipes written in Python described in the previous section.
649
650Here is an example written in Bash. The code logs the progress of
651the ``do_my_function`` function::
652
653 do_my_function() {
654 bbdebug 2 "Running do_my_function"
655 if [ exceptional_condition ]; then
656 bbnote "Hit exceptional_condition"
657 fi
658 bbdebug 2 "Got to point xyz"
659 if [ warning_trigger ]; then
660 bbwarn "Detected warning_trigger, this might cause a problem later."
661 fi
662 if [ recoverable_error ]; then
663 bberror "Hit recoverable_error, correcting"
664 fi
665 if [ fatal_error ]; then
666 bbfatal "fatal_error detected"
667 fi
668 bbdebug 2 "Completed do_my_function"
669 }
670
671
672Debugging Parallel Make Races
673=============================
674
675A parallel ``make`` race occurs when the build consists of several parts
676that are run simultaneously and a situation occurs when the output or
677result of one part is not ready for use with a different part of the
678build that depends on that output. Parallel make races are annoying and
679can sometimes be difficult to reproduce and fix. However, there are some simple
680tips and tricks that can help you debug and fix them. This section
681presents a real-world example of an error encountered on the Yocto
682Project autobuilder and the process used to fix it.
683
684.. note::
685
686 If you cannot properly fix a ``make`` race condition, you can work around it
687 by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST`
688 variables.
689
690The Failure
691-----------
692
693For this example, assume that you are building an image that depends on
694the "neard" package. And, during the build, BitBake runs into problems
695and creates the following output.
696
697.. note::
698
699 This example log file has longer lines artificially broken to make
700 the listing easier to read.
701
702If you examine the output or the log file, you see the failure during
703``make``:
704
705.. code-block:: none
706
707 | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common']
708 | DEBUG: Executing shell function do_compile
709 | NOTE: make -j 16
710 | make --no-print-directory all-am
711 | /bin/mkdir -p include/near
712 | /bin/mkdir -p include/near
713 | /bin/mkdir -p include/near
714 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
715 0.14-r0/neard-0.14/include/types.h include/near/types.h
716 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
717 0.14-r0/neard-0.14/include/log.h include/near/log.h
718 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
719 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h
720 | /bin/mkdir -p include/near
721 | /bin/mkdir -p include/near
722 | /bin/mkdir -p include/near
723 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
724 0.14-r0/neard-0.14/include/tag.h include/near/tag.h
725 | /bin/mkdir -p include/near
726 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
727 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h
728 | /bin/mkdir -p include/near
729 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
730 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h
731 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
732 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h
733 | /bin/mkdir -p include/near
734 | /bin/mkdir -p include/near
735 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
736 0.14-r0/neard-0.14/include/setting.h include/near/setting.h
737 | /bin/mkdir -p include/near
738 | /bin/mkdir -p include/near
739 | /bin/mkdir -p include/near
740 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
741 0.14-r0/neard-0.14/include/device.h include/near/device.h
742 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
743 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h
744 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
745 0.14-r0/neard-0.14/include/snep.h include/near/snep.h
746 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
747 0.14-r0/neard-0.14/include/version.h include/near/version.h
748 | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/
749 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h
750 | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h
751 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/
752 build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/
753 yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0
754 -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/
755 lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/
756 tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/
757 nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/
758 nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3
759 -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\"
760 -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c
761 -o tools/snep-send.o tools/snep-send.c
762 | In file included from tools/snep-send.c:16:0:
763 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
764 | #include <near/dbus.h>
765 | ^
766 | compilation terminated.
767 | make[1]: *** [tools/snep-send.o] Error 1
768 | make[1]: *** Waiting for unfinished jobs....
769 | make: *** [all] Error 2
770 | ERROR: oe_runmake failed
771
772Reproducing the Error
773---------------------
774
775Because race conditions are intermittent, they do not manifest
776themselves every time you do the build. In fact, most times the build
777will complete without problems even though the potential race condition
778exists. Thus, once the error surfaces, you need a way to reproduce it.
779
780In this example, compiling the "neard" package is causing the problem.
781So the first thing to do is build "neard" locally. Before you start the
782build, set the
783:term:`PARALLEL_MAKE` variable
784in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a
785high value for :term:`PARALLEL_MAKE` increases the chances of the race
786condition showing up::
787
788 $ bitbake neard
789
790Once the local build for "neard" completes, start a ``devshell`` build::
791
792 $ bitbake neard -c devshell
793
794For information on how to use a ``devshell``, see the
795":ref:`dev-manual/development-shell:using a development shell`" section.
796
797In the ``devshell``, do the following::
798
799 $ make clean
800 $ make tools/snep-send.o
801
802The ``devshell`` commands cause the failure to clearly
803be visible. In this case, there is a missing dependency for the ``neard``
804Makefile target. Here is some abbreviated, sample output with the
805missing dependency clearly visible at the end::
806
807 i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/......
808 .
809 .
810 .
811 tools/snep-send.c
812 In file included from tools/snep-send.c:16:0:
813 tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory
814 #include <near/dbus.h>
815 ^
816 compilation terminated.
817 make: *** [tools/snep-send.o] Error 1
818 $
819
820
821Creating a Patch for the Fix
822----------------------------
823
824Because there is a missing dependency for the Makefile target, you need
825to patch the ``Makefile.am`` file, which is generated from
826``Makefile.in``. You can use Quilt to create the patch::
827
828 $ quilt new parallelmake.patch
829 Patch patches/parallelmake.patch is now on top
830 $ quilt add Makefile.am
831 File Makefile.am added to patch patches/parallelmake.patch
832
833For more information on using Quilt, see the
834":ref:`dev-manual/quilt:using quilt in your workflow`" section.
835
836At this point you need to make the edits to ``Makefile.am`` to add the
837missing dependency. For our example, you have to add the following line
838to the file::
839
840 tools/snep-send.$(OBJEXT): include/near/dbus.h
841
842Once you have edited the file, use the ``refresh`` command to create the
843patch::
844
845 $ quilt refresh
846 Refreshed patch patches/parallelmake.patch
847
848Once the patch file is created, you need to add it back to the originating
849recipe folder. Here is an example assuming a top-level
850:term:`Source Directory` named ``poky``::
851
852 $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard
853
854The final thing you need to do to implement the fix in the build is to
855update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the
856:term:`SRC_URI` statement includes
857the patch file. The recipe file is in the folder above the patch. Here
858is what the edited :term:`SRC_URI` statement would look like::
859
860 SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \
861 file://neard.in \
862 file://neard.service.in \
863 file://parallelmake.patch \
864 "
865
866With the patch complete and moved to the correct folder and the
867:term:`SRC_URI` statement updated, you can exit the ``devshell``::
868
869 $ exit
870
871Testing the Build
872-----------------
873
874With everything in place, you can get back to trying the build again
875locally::
876
877 $ bitbake neard
878
879This build should succeed.
880
881Now you can open up a ``devshell`` again and repeat the clean and make
882operations as follows::
883
884 $ bitbake neard -c devshell
885 $ make clean
886 $ make tools/snep-send.o
887
888The build should work without issue.
889
890As with all solved problems, if they originated upstream, you need to
891submit the fix for the recipe in OE-Core and upstream so that the
892problem is taken care of at its source. See the
893":doc:`/contributor-guide/submit-changes`" section for more information.
894
895Debugging With the GNU Project Debugger (GDB) Remotely
896======================================================
897
898GDB allows you to examine running programs, which in turn helps you to
899understand and fix problems. It also allows you to perform post-mortem
900style analysis of program crashes. GDB is available as a package within
901the Yocto Project and is installed in SDK images by default. See the
902":ref:`ref-manual/images:Images`" chapter in the Yocto
903Project Reference Manual for a description of these images. You can find
904information on GDB at https://sourceware.org/gdb/.
905
906.. note::
907
908 For best results, install debug (``-dbg``) packages for the applications you
909 are going to debug. Doing so makes extra debug symbols available that give
910 you more meaningful output.
911
912Sometimes, due to memory or disk space constraints, it is not possible
913to use GDB directly on the remote target to debug applications. These
914constraints arise because GDB needs to load the debugging information
915and the binaries of the process being debugged. Additionally, GDB needs
916to perform many computations to locate information such as function
917names, variable names and values, stack traces and so forth --- even
918before starting the debugging process. These extra computations place
919more load on the target system and can alter the characteristics of the
920program being debugged.
921
922To help get past the previously mentioned constraints, there are two
923methods you can use: running a debuginfod server and using gdbserver.
924
925Using the debuginfod server method
926----------------------------------
927
928``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files.
929Running a ``debuginfod`` server makes debug symbols readily available,
930which means you don't need to download debugging information
931and the binaries of the process being debugged. You can just fetch
932debug symbols from the server.
933
934To run a ``debuginfod`` server, you need to do the following:
935
936- Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES`
937 (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution).
938 If not, set in your distro config file or in ``local.conf``::
939
940 DISTRO_FEATURES:append = " debuginfod"
941
942 This distro feature enables the server and client library in ``elfutils``,
943 and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``).
944
945- Run the following commands to launch the ``debuginfod`` server on the host::
946
947 $ oe-debuginfod
948
949- To use ``debuginfod`` on the target, you need to know the ip:port where
950 ``debuginfod`` is listening on the host (port defaults to 8002), and export
951 that into the shell environment, for example in ``qemu``::
952
953 root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/"
954
955- Then debug info fetching should simply work when running the target ``gdb``,
956 ``readelf`` or ``objdump``, for example::
957
958 root@qemux86-64:~# gdb /bin/cat
959 ...
960 Reading symbols from /bin/cat...
961 Downloading separate debug info for /bin/cat...
962 Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo...
963
964- It's also possible to use ``debuginfod-find`` to just query the server::
965
966 root@qemux86-64:~# debuginfod-find debuginfo /bin/ls
967 /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo
968
969
970Using the gdbserver method
971--------------------------
972
973gdbserver, which runs on the remote target and does not load any
974debugging information from the debugged process. Instead, a GDB instance
975processes the debugging information that is run on a remote computer -
976the host GDB. The host GDB then sends control commands to gdbserver to
977make it stop or start the debugged program, as well as read or write
978memory regions of that debugged program. All the debugging information
979loaded and processed as well as all the heavy debugging is done by the
980host GDB. Offloading these processes gives the gdbserver running on the
981target a chance to remain small and fast.
982
983Because the host GDB is responsible for loading the debugging
984information and for doing the necessary processing to make actual
985debugging happen, you have to make sure the host can access the
986unstripped binaries complete with their debugging information and also
987be sure the target is compiled with no optimizations. The host GDB must
988also have local access to all the libraries used by the debugged
989program. Because gdbserver does not need any local debugging
990information, the binaries on the remote target can remain stripped.
991However, the binaries must also be compiled without optimization so they
992match the host's binaries.
993
994To remain consistent with GDB documentation and terminology, the binary
995being debugged on the remote target machine is referred to as the
996"inferior" binary. For documentation on GDB see the `GDB
997site <https://sourceware.org/gdb/documentation/>`__.
998
999The following steps show you how to debug using the GNU project
1000debugger.
1001
1002#. *Configure your build system to construct the companion debug
1003 filesystem:*
1004
1005 In your ``local.conf`` file, set the following::
1006
1007 IMAGE_GEN_DEBUGFS = "1"
1008 IMAGE_FSTYPES_DEBUGFS = "tar.bz2"
1009
1010 These options cause the
1011 OpenEmbedded build system to generate a special companion filesystem
1012 fragment, which contains the matching source and debug symbols to
1013 your deployable filesystem. The build system does this by looking at
1014 what is in the deployed filesystem, and pulling the corresponding
1015 ``-dbg`` packages.
1016
1017 The companion debug filesystem is not a complete filesystem, but only
1018 contains the debug fragments. This filesystem must be combined with
1019 the full filesystem for debugging. Subsequent steps in this procedure
1020 show how to combine the partial filesystem with the full filesystem.
1021
1022#. *Configure the system to include gdbserver in the target filesystem:*
1023
1024 Make the following addition in your ``local.conf`` file::
1025
1026 EXTRA_IMAGE_FEATURES:append = " tools-debug"
1027
1028 The change makes
1029 sure the ``gdbserver`` package is included.
1030
1031#. *Build the environment:*
1032
1033 Use the following command to construct the image and the companion
1034 Debug Filesystem::
1035
1036 $ bitbake image
1037
1038 Build the cross GDB component and
1039 make it available for debugging. Build the SDK that matches the
1040 image. Building the SDK is best for a production build that can be
1041 used later for debugging, especially during long term maintenance::
1042
1043 $ bitbake -c populate_sdk image
1044
1045 Alternatively, you can build the minimal toolchain components that
1046 match the target. Doing so creates a smaller than typical SDK and
1047 only contains a minimal set of components with which to build simple
1048 test applications, as well as run the debugger::
1049
1050 $ bitbake meta-toolchain
1051
1052 A final method is to build Gdb itself within the build system::
1053
1054 $ bitbake gdb-cross-<architecture>
1055
1056 Doing so produces a temporary copy of
1057 ``cross-gdb`` you can use for debugging during development. While
1058 this is the quickest approach, the two previous methods in this step
1059 are better when considering long-term maintenance strategies.
1060
1061 .. note::
1062
1063 If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests
1064 the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the
1065 actual name you want to use.
1066
1067#. *Set up the* ``debugfs``\ *:*
1068
1069 Run the following commands to set up the ``debugfs``::
1070
1071 $ mkdir debugfs
1072 $ cd debugfs
1073 $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2
1074 $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2
1075
1076#. *Set up GDB:*
1077
1078 Install the SDK (if you built one) and then source the correct
1079 environment file. Sourcing the environment file puts the SDK in your
1080 ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger.
1081
1082 If you are using the build system, Gdb is located in
1083 `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb``
1084
1085#. *Boot the target:*
1086
1087 For information on how to run QEMU, see the `QEMU
1088 Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__.
1089
1090 .. note::
1091
1092 Be sure to verify that your host can access the target via TCP.
1093
1094#. *Debug a program:*
1095
1096 Debugging a program involves running gdbserver on the target and then
1097 running Gdb on the host. The example in this step debugs ``gzip``:
1098
1099 .. code-block:: shell
1100
1101 root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help
1102
1103 For
1104 additional gdbserver options, see the `GDB Server
1105 Documentation <https://www.gnu.org/software/gdb/documentation/>`__.
1106
1107 After running gdbserver on the target, you need to run Gdb on the
1108 host and configure it and connect to the target. Use these commands::
1109
1110 $ cd directory-holding-the-debugfs-directory
1111 $ arch-gdb
1112 (gdb) set sysroot debugfs
1113 (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug
1114 (gdb) target remote IP-of-target:1234
1115
1116 At this
1117 point, everything should automatically load (i.e. matching binaries,
1118 symbols and headers).
1119
1120 .. note::
1121
1122 The Gdb ``set`` commands in the previous example can be placed into the
1123 users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever
1124 commands are in that file.
1125
1126#. *Deploying without a full image rebuild:*
1127
1128 In many cases, during development you want a quick method to deploy a
1129 new binary to the target and debug it, without waiting for a full
1130 image build.
1131
1132 One approach to solving this situation is to just build the component
1133 you want to debug. Once you have built the component, copy the
1134 executable directly to both the target and the host ``debugfs``.
1135
1136 If the binary is processed through the debug splitting in
1137 OpenEmbedded, you should also copy the debug items (i.e. ``.debug``
1138 contents and corresponding ``/usr/src/debug`` files) from the work
1139 directory. Here is an example::
1140
1141 $ bitbake bash
1142 $ bitbake -c devshell bash
1143 $ cd ..
1144 $ scp packages-split/bash/bin/bash target:/bin/bash
1145 $ cp -a packages-split/bash-dbg/\* path/debugfs
1146
1147Debugging with the GNU Project Debugger (GDB) on the Target
1148===========================================================
1149
1150The previous section addressed using GDB remotely for debugging
1151purposes, which is the most usual case due to the inherent hardware
1152limitations on many embedded devices. However, debugging in the target
1153hardware itself is also possible with more powerful devices. This
1154section describes what you need to do in order to support using GDB to
1155debug on the target hardware.
1156
1157To support this kind of debugging, you need do the following:
1158
1159- Ensure that GDB is on the target. You can do this by making
1160 the following addition to your ``local.conf`` file::
1161
1162 EXTRA_IMAGE_FEATURES:append = " tools-debug"
1163
1164- Ensure that debug symbols are present. You can do so by adding the
1165 corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`::
1166
1167 IMAGE_INSTALL:append = " packagename-dbg"
1168
1169 Alternatively, you can add the following to ``local.conf`` to include
1170 all the debug symbols::
1171
1172 EXTRA_IMAGE_FEATURES:append = " dbg-pkgs"
1173
1174.. note::
1175
1176 To improve the debug information accuracy, you can reduce the level
1177 of optimization used by the compiler. For example, when adding the
1178 following line to your ``local.conf`` file, you will reduce optimization
1179 from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION`
1180 of "-O -fno-omit-frame-pointer"::
1181
1182 DEBUG_BUILD = "1"
1183
1184 Consider that this will reduce the application's performance and is
1185 recommended only for debugging purposes.
1186
1187Enabling Minidebuginfo
1188======================
1189
1190Enabling the :term:`DISTRO_FEATURES` minidebuginfo adds a compressed ELF section ``.gnu_debugdata``
1191to all binary files, containing only function names, and thus increasing the size of the
1192binaries only by 5 to 10%. For comparison, full debug symbols can be 10 times as big as
1193a stripped binary, and it is thus not always possible to deploy full debug symbols.
1194Minidebuginfo data allows, on the one side, to retrieve a call-stack using
1195GDB (command backtrace) without deploying full debug symbols to the target. It also
1196allows to retrieve a symbolicated call-stack when using ``systemd-coredump`` to manage
1197coredumps (commands ``coredumpctl list`` and ``coredumpctl info``).
1198
1199This feature was created by Fedora, see https://fedoraproject.org/wiki/Features/MiniDebugInfo for
1200more details.
1201
1202Other Debugging Tips
1203====================
1204
1205Here are some other tips that you might find useful:
1206
1207- When adding new packages, it is worth watching for undesirable items
1208 making their way into compiler command lines. For example, you do not
1209 want references to local system files like ``/usr/lib/`` or
1210 ``/usr/include/``.
1211
1212- If you want to remove the ``psplash`` boot splashscreen, add
1213 ``psplash=false`` to the kernel command line. Doing so prevents
1214 ``psplash`` from loading and thus allows you to see the console. It
1215 is also possible to switch out of the splashscreen by switching the
1216 virtual console (e.g. Fn+Left or Fn+Right on a Zaurus).
1217
1218- Removing :term:`TMPDIR` (usually ``tmp/``, within the
1219 :term:`Build Directory`) can often fix temporary build issues. Removing
1220 :term:`TMPDIR` is usually a relatively cheap operation, because task output
1221 will be cached in :term:`SSTATE_DIR` (usually ``sstate-cache/``, which is
1222 also in the :term:`Build Directory`).
1223
1224 .. note::
1225
1226 Removing :term:`TMPDIR` might be a workaround rather than a fix.
1227 Consequently, trying to determine the underlying cause of an issue before
1228 removing the directory is a good idea.
1229
1230- Understanding how a feature is used in practice within existing
1231 recipes can be very helpful. It is recommended that you configure
1232 some method that allows you to quickly search through files.
1233
1234 Using GNU Grep, you can use the following shell function to
1235 recursively search through common recipe-related files, skipping
1236 binary files, ``.git`` directories, and the :term:`Build Directory`
1237 (assuming its name starts with "build")::
1238
1239 g() {
1240 grep -Ir \
1241 --exclude-dir=.git \
1242 --exclude-dir='build*' \
1243 --include='*.bb*' \
1244 --include='*.inc*' \
1245 --include='*.conf*' \
1246 --include='*.py*' \
1247 "$@"
1248 }
1249
1250 Here are some usage examples::
1251
1252 $ g FOO # Search recursively for "FOO"
1253 $ g -i foo # Search recursively for "foo", ignoring case
1254 $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR"
1255
1256 If figuring
1257 out how some feature works requires a lot of searching, it might
1258 indicate that the documentation should be extended or improved. In
1259 such cases, consider filing a documentation bug using the Yocto
1260 Project implementation of
1261 :yocto_bugs:`Bugzilla <>`. For information on
1262 how to submit a bug against the Yocto Project, see the Yocto Project
1263 Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>`
1264 and the ":doc:`/contributor-guide/report-defect`" section.
1265
1266 .. note::
1267
1268 The manuals might not be the right place to document variables
1269 that are purely internal and have a limited scope (e.g. internal
1270 variables used to implement a single ``.bbclass`` file).
1271
diff --git a/documentation/dev-manual/development-shell.rst b/documentation/dev-manual/development-shell.rst
deleted file mode 100644
index be26bcffc7..0000000000
--- a/documentation/dev-manual/development-shell.rst
+++ /dev/null
@@ -1,82 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using a Development Shell
4*************************
5
6When debugging certain commands or even when just editing packages,
7``devshell`` can be a useful tool. When you invoke ``devshell``, all
8tasks up to and including
9:ref:`ref-tasks-patch` are run for the
10specified target. Then, a new terminal is opened and you are placed in
11``${``\ :term:`S`\ ``}``, the source
12directory. In the new terminal, all the OpenEmbedded build-related
13environment variables are still defined so you can use commands such as
14``configure`` and ``make``. The commands execute just as if the
15OpenEmbedded build system were executing them. Consequently, working
16this way can be helpful when debugging a build or preparing software to
17be used with the OpenEmbedded build system.
18
19Here is an example that uses ``devshell`` on a target named
20``matchbox-desktop``::
21
22 $ bitbake matchbox-desktop -c devshell
23
24This command spawns a terminal with a shell prompt within the
25OpenEmbedded build environment. The
26:term:`OE_TERMINAL` variable
27controls what type of shell is opened.
28
29For spawned terminals, the following occurs:
30
31- The ``PATH`` variable includes the cross-toolchain.
32
33- The ``pkgconfig`` variables find the correct ``.pc`` files.
34
35- The ``configure`` command finds the Yocto Project site files as well
36 as any other necessary files.
37
38Within this environment, you can run configure or compile commands as if
39they were being run by the OpenEmbedded build system itself. As noted
40earlier, the working directory also automatically changes to the Source
41Directory (:term:`S`).
42
43To manually run a specific task using ``devshell``, run the
44corresponding ``run.*`` script in the
45``${``\ :term:`WORKDIR`\ ``}/temp``
46directory (e.g., ``run.do_configure.``\ `pid`). If a task's script does
47not exist, which would be the case if the task was skipped by way of the
48sstate cache, you can create the task by first running it outside of the
49``devshell``::
50
51 $ bitbake -c task
52
53.. note::
54
55 - Execution of a task's ``run.*`` script and BitBake's execution of
56 a task are identical. In other words, running the script re-runs
57 the task just as it would be run using the ``bitbake -c`` command.
58
59 - Any ``run.*`` file that does not have a ``.pid`` extension is a
60 symbolic link (symlink) to the most recent version of that file.
61
62Remember, that the ``devshell`` is a mechanism that allows you to get
63into the BitBake task execution environment. And as such, all commands
64must be called just as BitBake would call them. That means you need to
65provide the appropriate options for cross-compilation and so forth as
66applicable.
67
68When you are finished using ``devshell``, exit the shell or close the
69terminal window.
70
71.. note::
72
73 - It is worth remembering that when using ``devshell`` you need to
74 use the full compiler name such as ``arm-poky-linux-gnueabi-gcc``
75 instead of just using ``gcc``. The same applies to other
76 applications such as ``binutils``, ``libtool`` and so forth.
77 BitBake sets up environment variables such as :term:`CC` to assist
78 applications, such as ``make`` to find the correct tools.
79
80 - It is also worth noting that ``devshell`` still works over X11
81 forwarding and similar situations.
82
diff --git a/documentation/dev-manual/device-manager.rst b/documentation/dev-manual/device-manager.rst
deleted file mode 100644
index 49fc785fec..0000000000
--- a/documentation/dev-manual/device-manager.rst
+++ /dev/null
@@ -1,74 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3.. _device-manager:
4
5Selecting a Device Manager
6**************************
7
8The Yocto Project provides multiple ways to manage the device manager
9(``/dev``):
10
11- Persistent and Pre-Populated ``/dev``: For this case, the ``/dev``
12 directory is persistent and the required device nodes are created
13 during the build.
14
15- Use ``devtmpfs`` with a Device Manager: For this case, the ``/dev``
16 directory is provided by the kernel as an in-memory file system and
17 is automatically populated by the kernel at runtime. Additional
18 configuration of device nodes is done in user space by a device
19 manager like ``udev`` or ``busybox-mdev``.
20
21Using Persistent and Pre-Populated ``/dev``
22===========================================
23
24To use the static method for device population, you need to set the
25:term:`USE_DEVFS` variable to "0"
26as follows::
27
28 USE_DEVFS = "0"
29
30The content of the resulting ``/dev`` directory is defined in a Device
31Table file. The
32:term:`IMAGE_DEVICE_TABLES`
33variable defines the Device Table to use and should be set in the
34machine or distro configuration file. Alternatively, you can set this
35variable in your ``local.conf`` configuration file.
36
37If you do not define the :term:`IMAGE_DEVICE_TABLES` variable, the default
38``device_table-minimal.txt`` is used::
39
40 IMAGE_DEVICE_TABLES = "device_table-mymachine.txt"
41
42The population is handled by the ``makedevs`` utility during image
43creation:
44
45Using ``devtmpfs`` and a Device Manager
46=======================================
47
48To use the dynamic method for device population, you need to use (or be
49sure to set) the :term:`USE_DEVFS`
50variable to "1", which is the default::
51
52 USE_DEVFS = "1"
53
54With this
55setting, the resulting ``/dev`` directory is populated by the kernel
56using ``devtmpfs``. Make sure the corresponding kernel configuration
57variable ``CONFIG_DEVTMPFS`` is set when building you build a Linux
58kernel.
59
60All devices created by ``devtmpfs`` will be owned by ``root`` and have
61permissions ``0600``.
62
63To have more control over the device nodes, you can use a device manager like
64``udev`` or ``busybox-mdev``. You choose the device manager by defining the
65:term:`VIRTUAL-RUNTIME_dev_manager <VIRTUAL-RUNTIME>` variable in your machine
66or distro configuration file. Alternatively, you can set this variable in
67your ``local.conf`` configuration file::
68
69 VIRTUAL-RUNTIME_dev_manager = "udev"
70
71 # Some alternative values
72 # VIRTUAL-RUNTIME_dev_manager = "busybox-mdev"
73 # VIRTUAL-RUNTIME_dev_manager = "systemd"
74
diff --git a/documentation/dev-manual/devtool.rst b/documentation/dev-manual/devtool.rst
deleted file mode 100644
index c82dc9c333..0000000000
--- a/documentation/dev-manual/devtool.rst
+++ /dev/null
@@ -1,1322 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using the ``devtool`` command-line tool
4***************************************
5
6The ``devtool`` command-line tool provides a number of features that
7help you build, test, and package software. This command is available
8alongside the ``bitbake`` command. Additionally, the ``devtool`` command
9is a key part of the :term:`Extensible Software Development Kit (eSDK)`.
10
11Use ``devtool add`` to Add an Application
12=========================================
13
14The ``devtool add`` command generates a new recipe based on existing
15source code. This command takes advantage of the
16:ref:`devtool-the-workspace-layer-structure`
17layer that many ``devtool`` commands use. The command is flexible enough
18to allow you to extract source code into both the workspace or a
19separate local Git repository and to use existing code that does not
20need to be extracted.
21
22Depending on your particular scenario, the arguments and options you use
23with ``devtool add`` form different combinations. The following diagram
24shows common development flows you would use with the ``devtool add``
25command:
26
27.. image:: figures/devtool-add-flow.png
28 :width: 100%
29
30#. *Generating the New Recipe*: The top part of the flow shows three
31 scenarios by which you could use ``devtool add`` to generate a recipe
32 based on existing source code.
33
34 In a shared development environment, it is typical for other
35 developers to be responsible for various areas of source code. As a
36 developer, you are probably interested in using that source code as
37 part of your development within the Yocto Project. All you need is
38 access to the code, a recipe, and a controlled area in which to do
39 your work.
40
41 Within the diagram, three possible scenarios feed into the
42 ``devtool add`` workflow:
43
44 - *Left*: The left scenario in the figure represents a common
45 situation where the source code does not exist locally and needs
46 to be extracted. In this situation, the source code is extracted
47 to the default workspace --- you do not want the files in some
48 specific location outside of the workspace. Thus, everything you
49 need will be located in the workspace::
50
51 $ devtool add recipe fetchuri
52
53 With this command, ``devtool`` extracts the upstream
54 source files into a local Git repository within the ``sources``
55 folder. The command then creates a recipe named recipe and a
56 corresponding append file in the workspace. If you do not provide
57 recipe, the command makes an attempt to determine the recipe name.
58
59 - *Middle*: The middle scenario in the figure also represents a
60 situation where the source code does not exist locally. In this
61 case, the code is again upstream and needs to be extracted to some
62 local area --- this time outside of the default workspace.
63
64 .. note::
65
66 If required, ``devtool`` always creates a Git repository locally
67 during the extraction.
68
69 Furthermore, the first positional argument ``srctree`` in this case
70 identifies where the ``devtool add`` command will locate the
71 extracted code outside of the workspace. You need to specify an
72 empty directory::
73
74 $ devtool add recipe srctree fetchuri
75
76 In summary, the source code is pulled from fetchuri and extracted into the
77 location defined by ``srctree`` as a local Git repository.
78
79 Within workspace, ``devtool`` creates a recipe named recipe along
80 with an associated append file.
81
82 - *Right*: The right scenario in the figure represents a situation
83 where the ``srctree`` has been previously prepared outside of the
84 ``devtool`` workspace.
85
86 The following command provides a new recipe name and identifies
87 the existing source tree location::
88
89 $ devtool add recipe srctree
90
91 The command examines the source code and creates a recipe named
92 recipe for the code and places the recipe into the workspace.
93
94 Because the extracted source code already exists, ``devtool`` does
95 not try to relocate the source code into the workspace --- only the
96 new recipe is placed in the workspace.
97
98 Aside from a recipe folder, the command also creates an associated
99 append folder and places an initial ``*.bbappend`` file within.
100
101#. *Edit the Recipe*: You can use ``devtool edit-recipe`` to open up the
102 editor as defined by the ``$EDITOR`` environment variable and modify
103 the file::
104
105 $ devtool edit-recipe recipe
106
107 From within the editor, you can make modifications to the recipe that
108 take effect when you build it later.
109
110#. *Build the Recipe or Rebuild the Image*: The next step you take
111 depends on what you are going to do with the new code.
112
113 If you need to eventually move the build output to the target
114 hardware, use the following ``devtool`` command::
115
116 $ devtool build recipe
117
118 On the other hand, if you want an image to contain the recipe's
119 packages from the workspace for immediate deployment onto a device
120 (e.g. for testing purposes), you can use the ``devtool build-image``
121 command::
122
123 $ devtool build-image image
124
125#. *Deploy the Build Output*: When you use the ``devtool build`` command
126 to build out your recipe, you probably want to see if the resulting
127 build output works as expected on the target hardware.
128
129 .. note::
130
131 This step assumes you have a previously built image that is
132 already either running in QEMU or is running on actual hardware.
133 Also, it is assumed that for deployment of the image to the
134 target, SSH is installed in the image and, if the image is running
135 on real hardware, you have network access to and from your
136 development machine.
137
138 You can deploy your build output to that target hardware by using the
139 ``devtool deploy-target`` command::
140
141 $ devtool deploy-target recipe target
142
143 The target is a live target machine running as an SSH server.
144
145 You can, of course, also deploy the image you build to actual
146 hardware by using the ``devtool build-image`` command. However,
147 ``devtool`` does not provide a specific command that allows you to
148 deploy the image to actual hardware.
149
150#. *Finish Your Work With the Recipe*: The ``devtool finish`` command
151 creates any patches corresponding to commits in the local Git
152 repository, moves the new recipe to a more permanent layer, and then
153 resets the recipe so that the recipe is built normally rather than
154 from the workspace::
155
156 $ devtool finish recipe layer
157
158 .. note::
159
160 Any changes you want to turn into patches must be committed to the
161 Git repository in the source tree.
162
163 As mentioned, the ``devtool finish`` command moves the final recipe
164 to its permanent layer.
165
166 As a final process of the ``devtool finish`` command, the state of
167 the standard layers and the upstream source is restored so that you
168 can build the recipe from those areas rather than the workspace.
169
170 .. note::
171
172 You can use the ``devtool reset`` command to put things back should you
173 decide you do not want to proceed with your work. If you do use this
174 command, realize that the source tree is preserved.
175
176Use ``devtool modify`` to Modify the Source of an Existing Component
177====================================================================
178
179The ``devtool modify`` command prepares the way to work on existing code
180that already has a local recipe in place that is used to build the
181software. The command is flexible enough to allow you to extract code
182from an upstream source, specify the existing recipe, and keep track of
183and gather any patch files from other developers that are associated
184with the code.
185
186Depending on your particular scenario, the arguments and options you use
187with ``devtool modify`` form different combinations. The following
188diagram shows common development flows for the ``devtool modify``
189command:
190
191.. image:: figures/devtool-modify-flow.png
192 :width: 100%
193
194#. *Preparing to Modify the Code*: The top part of the flow shows three
195 scenarios by which you could use ``devtool modify`` to prepare to
196 work on source files. Each scenario assumes the following:
197
198 - The recipe exists locally in a layer external to the ``devtool``
199 workspace.
200
201 - The source files exist either upstream in an un-extracted state or
202 locally in a previously extracted state.
203
204 The typical situation is where another developer has created a layer
205 for use with the Yocto Project and their recipe already resides in
206 that layer. Furthermore, their source code is readily available
207 either upstream or locally.
208
209 - *Left*: The left scenario in the figure represents a common
210 situation where the source code does not exist locally and it
211 needs to be extracted from an upstream source. In this situation,
212 the source is extracted into the default ``devtool`` workspace
213 location. The recipe, in this scenario, is in its own layer
214 outside the workspace (i.e. ``meta-``\ layername).
215
216 The following command identifies the recipe and, by default,
217 extracts the source files::
218
219 $ devtool modify recipe
220
221 Once ``devtool`` locates the recipe, ``devtool`` uses the recipe's
222 :term:`SRC_URI` statements to locate the source code and any local
223 patch files from other developers.
224
225 With this scenario, there is no ``srctree`` argument. Consequently, the
226 default behavior of the ``devtool modify`` command is to extract
227 the source files pointed to by the :term:`SRC_URI` statements into a
228 local Git structure. Furthermore, the location for the extracted
229 source is the default area within the ``devtool`` workspace. The
230 result is that the command sets up both the source code and an
231 append file within the workspace while the recipe remains in its
232 original location.
233
234 Additionally, if you have any non-patch local files (i.e. files
235 referred to with ``file://`` entries in :term:`SRC_URI` statement
236 excluding ``*.patch/`` or ``*.diff``), these files are copied to
237 an ``oe-local-files`` folder under the newly created source tree.
238 Copying the files here gives you a convenient area from which you
239 can modify the files. Any changes or additions you make to those
240 files are incorporated into the build the next time you build the
241 software just as are other changes you might have made to the
242 source.
243
244 - *Middle*: The middle scenario in the figure represents a situation
245 where the source code also does not exist locally. In this case,
246 the code is again upstream and needs to be extracted to some local
247 area as a Git repository. The recipe, in this scenario, is again
248 local and in its own layer outside the workspace.
249
250 The following command tells ``devtool`` the recipe with which to
251 work and, in this case, identifies a local area for the extracted
252 source files that exists outside of the default ``devtool``
253 workspace::
254
255 $ devtool modify recipe srctree
256
257 .. note::
258
259 You cannot provide a URL for ``srctree`` using the ``devtool`` command.
260
261 As with all extractions, the command uses the recipe's :term:`SRC_URI`
262 statements to locate the source files and any associated patch
263 files. Non-patch files are copied to an ``oe-local-files`` folder
264 under the newly created source tree.
265
266 Once the files are located, the command by default extracts them
267 into ``srctree``.
268
269 Within workspace, ``devtool`` creates an append file for the
270 recipe. The recipe remains in its original location but the source
271 files are extracted to the location you provide with ``srctree``.
272
273 - *Right*: The right scenario in the figure represents a situation
274 where the source tree (``srctree``) already exists locally as a
275 previously extracted Git structure outside of the ``devtool``
276 workspace. In this example, the recipe also exists elsewhere
277 locally in its own layer.
278
279 The following command tells ``devtool`` the recipe with which to
280 work, uses the "-n" option to indicate source does not need to be
281 extracted, and uses ``srctree`` to point to the previously extracted
282 source files::
283
284 $ devtool modify -n recipe srctree
285
286 If an ``oe-local-files`` subdirectory happens to exist and it
287 contains non-patch files, the files are used. However, if the
288 subdirectory does not exist and you run the ``devtool finish``
289 command, any non-patch files that might exist next to the recipe
290 are removed because it appears to ``devtool`` that you have
291 deleted those files.
292
293 Once the ``devtool modify`` command finishes, it creates only an
294 append file for the recipe in the ``devtool`` workspace. The
295 recipe and the source code remain in their original locations.
296
297#. *Edit the Source*: Once you have used the ``devtool modify`` command,
298 you are free to make changes to the source files. You can use any
299 editor you like to make and save your source code modifications.
300
301#. *Build the Recipe or Rebuild the Image*: The next step you take
302 depends on what you are going to do with the new code.
303
304 If you need to eventually move the build output to the target
305 hardware, use the following ``devtool`` command::
306
307 $ devtool build recipe
308
309 On the other hand, if you want an image to contain the recipe's
310 packages from the workspace for immediate deployment onto a device
311 (e.g. for testing purposes), you can use the ``devtool build-image``
312 command::
313
314 $ devtool build-image image
315
316#. *Deploy the Build Output*: When you use the ``devtool build`` command
317 to build out your recipe, you probably want to see if the resulting
318 build output works as expected on target hardware.
319
320 .. note::
321
322 This step assumes you have a previously built image that is
323 already either running in QEMU or running on actual hardware.
324 Also, it is assumed that for deployment of the image to the
325 target, SSH is installed in the image and if the image is running
326 on real hardware that you have network access to and from your
327 development machine.
328
329 You can deploy your build output to that target hardware by using the
330 ``devtool deploy-target`` command::
331
332 $ devtool deploy-target recipe target
333
334 The target is a live target machine running as an SSH server.
335
336 You can, of course, use other methods to deploy the image you built
337 using the ``devtool build-image`` command to actual hardware.
338 ``devtool`` does not provide a specific command to deploy the image
339 to actual hardware.
340
341#. *Finish Your Work With the Recipe*: The ``devtool finish`` command
342 creates any patches corresponding to commits in the local Git
343 repository, updates the recipe to point to them (or creates a
344 ``.bbappend`` file to do so, depending on the specified destination
345 layer), and then resets the recipe so that the recipe is built
346 normally rather than from the workspace::
347
348 $ devtool finish recipe layer
349
350 .. note::
351
352 Any changes you want to turn into patches must be staged and
353 committed within the local Git repository before you use the
354 ``devtool finish`` command.
355
356 Because there is no need to move the recipe, ``devtool finish``
357 either updates the original recipe in the original layer or the
358 command creates a ``.bbappend`` file in a different layer as provided
359 by layer. Any work you did in the ``oe-local-files`` directory is
360 preserved in the original files next to the recipe during the
361 ``devtool finish`` command.
362
363 As a final process of the ``devtool finish`` command, the state of
364 the standard layers and the upstream source is restored so that you
365 can build the recipe from those areas rather than from the workspace.
366
367 .. note::
368
369 You can use the ``devtool reset`` command to put things back should you
370 decide you do not want to proceed with your work. If you do use this
371 command, realize that the source tree is preserved.
372
373``devtool ide-sdk`` configures IDEs and bootstraps SDKs
374=======================================================
375
376The ``devtool ide-sdk`` command can provide an IDE configuration for IDEs when
377working on the source code of one or more recipes.
378Depending on the programming language, and the build system used by the recipe,
379the tools required for cross-development and remote debugging are different.
380For example:
381
382- A C/C++ project usually uses CMake or Meson.
383
384- A Python project uses setuptools or one of its successors.
385
386- A Rust project uses Cargo.
387
388Also, the IDE plugins needed for the integration of a build system with the
389IDE and the corresponding settings are usually specific to these build-systems.
390To hide all these details from the user, ``devtool ide-sdk`` does two things:
391
392- It generates any kind of SDK needed for cross-development and remote
393 debugging of the specified recipes.
394
395- It generates the configuration for the IDE (and the IDE plugins) for using
396 the cross-toolchain and remote debugging tools provided by the SDK directly
397 from the IDE.
398
399For supported build systems the configurations generated by ``devtool ide-sdk``
400combine the advantages of the ``devtool modify`` based workflow
401(see :ref:`using_devtool`) with the advantages of the simple Environment Setup
402script based workflow (see :ref:`running_the_ext_sdk_env`) provided by Yocto's
403SDK or eSDK:
404
405- The source code of the recipe is in the workspace created by
406 ``devtool modify`` or ``devtool add``.
407 Using ``devtool build``, ``devtool build-image``,
408 ``devtool deploy-target`` or ``bitbake`` is possible.
409 Also ``devtool ide-sdk`` can be used to update the SDK and the IDE
410 configuration at any time.
411
412- ``devtool ide-sdk`` aims to support multiple programming languages and
413 multiple IDEs natively. "Natively" means that the IDE is configured to call
414 the build tool (e.g. ``cmake`` or ``meson``) directly. This has several
415 advantages.
416 First of all, it is usually much faster to call for example ``cmake`` than
417 ``devtool build``.
418 It also allows to benefit from the very good integration that IDEs like
419 VSCode offer for tools like CMake or GDB.
420
421 However, supporting many programming languages and multiple
422 IDEs is quite an elaborate and constantly evolving thing. Support for IDEs
423 is therefore implemented as plugins. Plugins can also be provided by
424 optional layers.
425
426So much about the introduction to the default mode of ``devtool sdk-ide`` which
427is called the "modified" mode because it uses the workspace created by
428``devtool modify`` and the per recipe :term:`Sysroots <Sysroot>` of BitBake.
429
430For some recipes and use cases, this default behavior of ``devtool ide-sdk``
431with full ``devtool`` and ``bitbake`` integration might not be suitable.
432To offer full feature parity with the SDK and the eSDK, ``devtool ide-sdk`` has
433a second mode called "shared" mode.
434If ``devtool ide-sdk`` is called with the ``--mode=shared`` option, it
435bootstraps an SDK directly from the BitBake environment, which offers the same
436Environment Setup script as described in :ref:`running_the_ext_sdk_env`.
437In addition to the (e)SDK installer-based setup, the IDE gets configured
438to use the shared :term:`Sysroots <Sysroot>` and the tools from the SDK.
439``devtool ide-sdk --mode=shared`` is basically a wrapper for the setup of the
440extensible SDK as described in :ref:`setting_up_ext_sdk_in_build`.
441
442The use of ``devtool ide-sdk`` is an alternative to using one of the SDK
443installers.
444``devtool ide-sdk`` allows the creation of SDKs that offer all the
445functionality of the SDK and the eSDK installers. Compared to the installers,
446however, the SDK created with ``devtool ide-sdk`` is much more flexible.
447For example, it is very easy to change the :term:`MACHINE` in the
448``local.conf`` file, update the layer meta data and then regenerate the SDK.
449
450Let's take a look at an example of how to use ``devtool ide-sdk`` in each of
451the two modes:
452
453#. *Modified mode*:
454
455 In order to use the ``devtool ide-sdk``, a few settings are needed. As a
456 starting example, the following lines of code can be added to the
457 ``local.conf`` file::
458
459 # Build the companion debug file system
460 IMAGE_GEN_DEBUGFS = "1"
461 # Optimize build time: with devtool ide-sdk the dbg tar is not needed
462 IMAGE_FSTYPES_DEBUGFS = ""
463 # Without copying the binaries into roofs-dbg, GDB does not find all source files.
464 IMAGE_CLASSES += "image-combined-dbg"
465
466 # SSH is mandatory, no password simplifies the usage
467 EXTRA_IMAGE_FEATURES += "\
468 ssh-server-openssh \
469 allow-empty-password \
470 allow-root-login \
471 empty-root-password \
472 "
473
474 # Remote debugging needs gdbserver on the target device
475 IMAGE_INSTALL:append = " gdbserver"
476
477 # Add the recipes which should be modified to the image
478 # Otherwise some dependencies might be missing.
479 IMAGE_INSTALL:append = " my-recipe"
480
481 Assuming the BitBake environment is set up correctly and a workspace has
482 been created for the recipe using ``devtool modify my-recipe`` or probably
483 even better by using ``devtool modify my-recipe --debug-build``, the
484 following command can create the SDK and the configuration for VSCode in
485 the recipe workspace::
486
487 $ devtool ide-sdk my-recipe core-image-minimal --target root@192.168.7.2
488
489 The command requires an image recipe (``core-image-minimal`` for this
490 example) that is used to create the SDK.
491 This firmware image should also be installed on the target device.
492 It is possible to pass multiple package recipes::
493
494 $ devtool ide-sdk my-recipe-1 my-recipe-2 core-image-minimal --target root@192.168.7.2
495
496 ``devtool ide-sdk`` tries to create an IDE configuration for all package
497 recipes.
498
499 What this command does exactly depends on the recipe, more precisely on the
500 build tool used by the recipe. The basic idea is to configure the IDE so
501 that it calls the build tool exactly as ``bitbake`` does.
502
503 For example, a CMake preset is created for a recipe that inherits
504 :ref:`ref-classes-cmake`. In the case of VSCode, CMake presets are supported
505 by the CMake Tools plugin. This is an example of how the build configuration
506 used by ``bitbake`` is exported to an IDE configuration that gives exactly
507 the same build results.
508
509 Support for remote debugging with seamless integration into the IDE is
510 important for a cross-SDK. ``devtool ide-sdk`` automatically generates the
511 necessary helper scripts for deploying the compiled artifacts to the target
512 device as well as the necessary configuration for the debugger and the IDE.
513
514 .. note::
515
516 To ensure that the debug symbols on the build machine match the binaries
517 running on the target device, it is essential that the image built by
518 ``devtool ide-sdk`` is running on the target device.
519
520 The default IDE is VSCode. Some hints about using VSCode:
521
522 - VSCode can be used to work on the BitBake recipes or the application
523 source code.
524 Usually there is one instance of VSCode running in the folder where the
525 BitBake recipes are. This instance has the
526 `Yocto Project BitBake plugin <https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`_
527 running.
528
529 .. warning::
530
531 Some VSCode plugins (Python, BitBake and others) need a reasonable
532 configuration to work as expected. Otherwise, some plugins try to
533 index the build directory of BitBake, which keeps your system quite
534 busy until an out of memory exception stops this nonsense.
535 Other plugins, such as the BitBake plugin, do not behave as expected.
536
537 To work around such issues, the ``oe-init-build-env`` script creates
538 an initial ``.vscode/settings.json`` file if ``code`` can be found
539 and the ``.vscode`` folder does not yet exist.
540 It is best to run ``oe-init-build-env`` once before starting VSCode.
541 An alternative approach is to use a build folder outside the layers,
542 e.g. ``oe-init-build-env ../build``.
543
544 The BitBake plugin also offers to create devtool workspaces and run
545 ``devtool ide-sdk`` with a few mouse clicks.
546 Of course, issuing commands in the terminal works as well.
547
548 - To work on the source code of a recipe another instance of VSCode is
549 started in the recipe's workspace. Example::
550
551 code build/workspace/sources/my-recipe
552
553 This instance of VSCode uses plugins that are useful for the development
554 of the application. ``devtool ide-sdk`` generates the necessary
555 ``extensions.json``, ``settings.json``, ``tasks.json`` and ``launch.json``
556 configuration files for all the involved plugins.
557
558 When the source code folder present in the workspace folder is opened in
559 VSCode for the first time, a pop-up message recommends installing the
560 required plugins.
561 After accepting the installation of the plugins, working with the source
562 code or some debugging tasks should work as usual with VSCode.
563
564 Starting the VSCode instances in the recipe workspace folders can also be
565 done by a mouse click on the recipe workspaces in the first VSCode
566 instance.
567
568 - To work with CMake press ``Ctrl + Shift + p``, type ``cmake``. This will
569 show some possible commands like selecting a CMake preset, compiling or
570 running CTest.
571
572 For recipes inheriting :ref:`ref-classes-cmake-qemu` rather than
573 :ref:`ref-classes-cmake`, executing cross-compiled unit tests on the host
574 can be supported transparently with QEMU user-mode.
575
576 - To work with Meson press ``Ctrl + Shift + p``, type ``meson``. This will
577 show some possible commands like compiling or executing the unit tests.
578
579 A note on running cross-compiled unit tests on the host: Meson enables
580 support for QEMU user mode by default. It is expected that the execution
581 of the unit tests from the IDE will work without any additional steps,
582 given that the code is suitable for the execution on the host machine.
583
584 - For the deployment to the target device, just press ``Ctrl + Shift + p``,
585 type ``task``. Select ``install && deploy-target``.
586
587 - For remote debugging, switch to the debugging view by pressing the "play"
588 button with the ``bug icon`` on the left side. This will provide a green
589 play button with a drop-down list where a debug configuration can be
590 selected. After selecting one of the generated configurations, press the
591 "play" button.
592
593 Starting a remote debugging session automatically initiates the
594 deployment to the target device. If this is not desired, the
595 ``"dependsOn": ["install && deploy-target...]`` parameter of the tasks
596 with ``"label": "gdbserver start...`` can be removed from the
597 ``tasks.json`` file.
598
599 VSCode supports GDB with many different setups and configurations for
600 many different use cases. However, most of these setups have some
601 limitations when it comes to cross-development, support only a few target
602 architectures or require a high performance target device. Therefore
603 ``devtool ide-sdk`` supports the classic, generic setup with GDB on the
604 development host and gdbserver on the target device.
605
606 Roughly summarized, this means:
607
608 - The binaries are copied via SSH to the remote target device by a
609 script referred by ``tasks.json``.
610
611 - gdbserver is started on the remote target device via SSH by a script
612 referred by ``tasks.json``.
613
614 Changing the parameters that are passed to the debugging executable
615 requires modifying the generated script. The script is located at
616 ``oe-scripts/gdbserver_*``. Defining the parameters in the ``args``
617 field in the ``launch.json`` file does not work.
618
619 - VSCode connects to gdbserver as documented in
620 `Remote debugging or debugging with a local debugger server
621 <https://code.visualstudio.com/docs/cpp/launch-json-reference#_remote-debugging-or-debugging-with-a-local-debugger-server>`__.
622
623 Additionally ``--ide=none`` is supported. With the ``none`` IDE parameter,
624 some generic configuration files like ``gdbinit`` files and some helper
625 scripts starting gdbserver remotely on the target device as well as the GDB
626 client on the host are generated.
627
628 Here is a usage example for the ``cmake-example`` recipe from the
629 ``meta-selftest`` layer which inherits :ref:`ref-classes-cmake-qemu`:
630
631 .. code-block:: sh
632
633 # Create the SDK
634 devtool modify cmake-example --debug-build
635 devtool ide-sdk cmake-example core-image-minimal -c --ide=none
636
637 # Install the firmware on a target device or start QEMU
638 runqemu
639
640 # From exploring the workspace of cmake-example
641 cd build/workspace/sources/cmake-example
642
643 # Find cmake-native and save the path into a variable
644 # Note: using just cmake instead of $CMAKE_NATIVE would work in many cases
645 CMAKE_NATIVE="$(jq -r '.configurePresets[0] | "\(.cmakeExecutable)"' CMakeUserPresets.json)"
646
647 # List available CMake presets
648 "$CMAKE_NATIVE" --list-presets
649 Available configure presets:
650
651 "cmake-example-cortexa57" - cmake-example: cortexa57
652
653 # Re-compile the already compiled sources
654 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57
655 ninja: no work to do.
656 # Do a clean re-build
657 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target clean
658 [1/1] Cleaning all built files...
659 Cleaning... 8 files.
660 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target all
661 [7/7] Linking CXX executable cmake-example
662
663 # Run the cross-compiled unit tests with QEMU user-mode
664 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target test
665 [0/1] Running tests...
666 Test project .../build/tmp/work/cortexa57-poky-linux/cmake-example/1.0/cmake-example-1.0
667 Start 1: test-cmake-example
668 1/1 Test #1: test-cmake-example ............... Passed 0.03 sec
669
670 100% tests passed, 0 tests failed out of 1
671
672 Total Test time (real) = 0.03 sec
673
674 # Using CTest directly is possible as well
675 CTEST_NATIVE="$(dirname "$CMAKE_NATIVE")/ctest"
676
677 # List available CMake presets
678 "$CTEST_NATIVE" --list-presets
679 Available test presets:
680
681 "cmake-example-cortexa57" - cmake-example: cortexa57
682
683 # Run the cross-compiled unit tests with QEMU user-mode
684 "$CTEST_NATIVE" --preset "cmake-example-cortexa57"
685 Test project ...build/tmp/work/cortexa57-poky-linux/cmake-example/1.0/cmake-example-1.0
686 Start 1: test-cmake-example
687 1/1 Test #1: test-cmake-example ............... Passed 0.03 sec
688
689 100% tests passed, 0 tests failed out of 1
690
691 Total Test time (real) = 0.03 sec
692
693 # Deploying the new build to the target device (default is QEUM at 192.168.7.2)
694 oe-scripts/install_and_deploy_cmake-example-cortexa57
695
696 # Start a remote debugging session with gdbserver on the target and GDB on the host
697 oe-scripts/gdbserver_1234_usr-bin-cmake-example_m
698 oe-scripts/gdb_1234_usr-bin-cmake-example
699 break main
700 run
701 step
702 stepi
703 continue
704 quit
705
706 # Stop gdbserver on the target device
707 oe-scripts/gdbserver_1234_usr-bin-cmake-example_m stop
708
709#. *Shared sysroots mode*
710
711 Creating an SDK with shared :term:`Sysroots <Sysroot>` that contains all the
712 dependencies needed to work with ``my-recipe`` is possible with the following
713 example command::
714
715 $ devtool ide-sdk --mode=shared my-recipe
716
717 For VSCode the cross-toolchain is exposed as a CMake kit. CMake kits are
718 defined in ``~/.local/share/CMakeTools/cmake-tools-kits.json``.
719 The following example shows how the cross-toolchain can be selected in
720 VSCode. First of all we need a folder containing a CMake project.
721 For this example, let's create a CMake project and start VSCode::
722
723 mkdir kit-test
724 echo "project(foo VERSION 1.0)" > kit-test/CMakeLists.txt
725 code kit-test
726
727 If there is a CMake project in the workspace, cross-compilation is
728 supported:
729
730 - Press ``Ctrl + Shift + P``, type ``CMake: Scan for Kits``
731 - Press ``Ctrl + Shift + P``, type ``CMake: Select a Kit``
732
733 Finally most of the features provided by CMake and the IDE should be
734 available.
735
736 Other IDEs than VSCode are supported as well. However,
737 ``devtool ide-sdk --mode=shared --ide=none my-recipe`` is currently
738 just a simple wrapper for the setup of the extensible SDK, as described in
739 :ref:`setting_up_ext_sdk_in_build`.
740
741Use ``devtool upgrade`` to Create a Version of the Recipe that Supports a Newer Version of the Software
742=======================================================================================================
743
744The ``devtool upgrade`` command upgrades an existing recipe to that of a
745more up-to-date version found upstream. Throughout the life of software,
746recipes continually undergo version upgrades by their upstream
747publishers. You can use the ``devtool upgrade`` workflow to make sure
748your recipes you are using for builds are up-to-date with their upstream
749counterparts.
750
751.. note::
752
753 Several methods exist by which you can upgrade recipes ---
754 ``devtool upgrade`` happens to be one. You can read about all the methods by
755 which you can upgrade recipes in the
756 :ref:`dev-manual/upgrading-recipes:upgrading recipes` section of the Yocto
757 Project Development Tasks Manual.
758
759The ``devtool upgrade`` command is flexible enough to allow you to specify
760source code revision and versioning schemes, extract code into or out of the
761``devtool`` :ref:`devtool-the-workspace-layer-structure`, and work with any
762source file forms that the
763:ref:`bitbake-user-manual/bitbake-user-manual-fetching:fetchers` support.
764
765The following diagram shows the common development flow used with the
766``devtool upgrade`` command:
767
768.. image:: figures/devtool-upgrade-flow.png
769 :width: 100%
770
771#. *Initiate the Upgrade*: The top part of the flow shows the typical
772 scenario by which you use the ``devtool upgrade`` command. The
773 following conditions exist:
774
775 - The recipe exists in a local layer external to the ``devtool``
776 workspace.
777
778 - The source files for the new release exist in the same location
779 pointed to by :term:`SRC_URI`
780 in the recipe (e.g. a tarball with the new version number in the
781 name, or as a different revision in the upstream Git repository).
782
783 A common situation is where third-party software has undergone a
784 revision so that it has been upgraded. The recipe you have access to
785 is likely in your own layer. Thus, you need to upgrade the recipe to
786 use the newer version of the software::
787
788 $ devtool upgrade -V version recipe
789
790 By default, the ``devtool upgrade`` command extracts source
791 code into the ``sources`` directory in the
792 :ref:`devtool-the-workspace-layer-structure`.
793 If you want the code extracted to any other location, you need to
794 provide the ``srctree`` positional argument with the command as follows::
795
796 $ devtool upgrade -V version recipe srctree
797
798 .. note::
799
800 In this example, the "-V" option specifies the new version. If you
801 don't use "-V", the command upgrades the recipe to the latest
802 version.
803
804 If the source files pointed to by the :term:`SRC_URI` statement in the
805 recipe are in a Git repository, you must provide the "-S" option and
806 specify a revision for the software.
807
808 Once ``devtool`` locates the recipe, it uses the :term:`SRC_URI` variable
809 to locate the source code and any local patch files from other
810 developers. The result is that the command sets up the source code,
811 the new version of the recipe, and an append file all within the
812 workspace.
813
814 Additionally, if you have any non-patch local files (i.e. files
815 referred to with ``file://`` entries in :term:`SRC_URI` statement
816 excluding ``*.patch/`` or ``*.diff``), these files are copied to an
817 ``oe-local-files`` folder under the newly created source tree.
818 Copying the files here gives you a convenient area from which you can
819 modify the files. Any changes or additions you make to those files
820 are incorporated into the build the next time you build the software
821 just as are other changes you might have made to the source.
822
823#. *Resolve any Conflicts created by the Upgrade*: Conflicts could happen
824 after upgrading the software to a new version. Conflicts occur
825 if your recipe specifies some patch files in :term:`SRC_URI` that
826 conflict with changes made in the new version of the software. For
827 such cases, you need to resolve the conflicts by editing the source
828 and following the normal ``git rebase`` conflict resolution process.
829
830 Before moving onto the next step, be sure to resolve any such
831 conflicts created through use of a newer or different version of the
832 software.
833
834#. *Build the Recipe or Rebuild the Image*: The next step you take
835 depends on what you are going to do with the new code.
836
837 If you need to eventually move the build output to the target
838 hardware, use the following ``devtool`` command::
839
840 $ devtool build recipe
841
842 On the other hand, if you want an image to contain the recipe's
843 packages from the workspace for immediate deployment onto a device
844 (e.g. for testing purposes), you can use the ``devtool build-image``
845 command::
846
847 $ devtool build-image image
848
849#. *Deploy the Build Output*: When you use the ``devtool build`` command
850 or ``bitbake`` to build your recipe, you probably want to see if the
851 resulting build output works as expected on target hardware.
852
853 .. note::
854
855 This step assumes you have a previously built image that is
856 already either running in QEMU or running on actual hardware.
857 Also, it is assumed that for deployment of the image to the
858 target, SSH is installed in the image and if the image is running
859 on real hardware that you have network access to and from your
860 development machine.
861
862 You can deploy your build output to that target hardware by using the
863 ``devtool deploy-target`` command::
864
865 $ devtool deploy-target recipe target
866
867 The target is a live target machine running as an SSH server.
868
869 You can, of course, also deploy the image you build using the
870 ``devtool build-image`` command to actual hardware. However,
871 ``devtool`` does not provide a specific command that allows you to do
872 this.
873
874#. *Finish Your Work With the Recipe*: The ``devtool finish`` command
875 creates any patches corresponding to commits in the local Git
876 repository, moves the new recipe to a more permanent layer, and then
877 resets the recipe so that the recipe is built normally rather than
878 from the workspace.
879
880 Any work you did in the ``oe-local-files`` directory is preserved in
881 the original files next to the recipe during the ``devtool finish``
882 command.
883
884 If you specify a destination layer that is the same as the original
885 source, then the old version of the recipe and associated files are
886 removed prior to adding the new version::
887
888 $ devtool finish recipe layer
889
890 .. note::
891
892 Any changes you want to turn into patches must be committed to the
893 Git repository in the source tree.
894
895 As a final process of the ``devtool finish`` command, the state of
896 the standard layers and the upstream source is restored so that you
897 can build the recipe from those areas rather than the workspace.
898
899 .. note::
900
901 You can use the ``devtool reset`` command to put things back should you
902 decide you do not want to proceed with your work. If you do use this
903 command, realize that the source tree is preserved.
904
905A Closer Look at ``devtool add``
906================================
907
908The ``devtool add`` command automatically creates a recipe based on the
909source tree you provide with the command. Currently, the command has
910support for the following:
911
912- Autotools (``autoconf`` and ``automake``)
913
914- CMake
915
916- Scons
917
918- ``qmake``
919
920- Plain ``Makefile``
921
922- Out-of-tree kernel module
923
924- Binary package (i.e. "-b" option)
925
926- Node.js module
927
928- Python modules that use ``setuptools`` or ``distutils``
929
930Apart from binary packages, the determination of how a source tree
931should be treated is automatic based on the files present within that
932source tree. For example, if a ``CMakeLists.txt`` file is found, then
933the source tree is assumed to be using CMake and is treated accordingly.
934
935.. note::
936
937 In most cases, you need to edit the automatically generated recipe in
938 order to make it build properly. Typically, you would go through
939 several edit and build cycles until the recipe successfully builds.
940 Once the recipe builds, you could use possible further iterations to
941 test the recipe on the target device.
942
943The remainder of this section covers specifics regarding how parts of
944the recipe are generated.
945
946Name and Version
947----------------
948
949If you do not specify a name and version on the command line,
950``devtool add`` uses various metadata within the source tree in an
951attempt to determine the name and version of the software being built.
952Based on what the tool determines, ``devtool`` sets the name of the
953created recipe file accordingly.
954
955If ``devtool`` cannot determine the name and version, the command prints
956an error. For such cases, you must re-run the command and provide the
957name and version, just the name, or just the version as part of the
958command line.
959
960Sometimes the name or version determined from the source tree might be
961incorrect. For such a case, you must reset the recipe::
962
963 $ devtool reset -n recipename
964
965After running the ``devtool reset`` command, you need to
966run ``devtool add`` again and provide the name or the version.
967
968Dependency Detection and Mapping
969--------------------------------
970
971The ``devtool add`` command attempts to detect build-time dependencies and map
972them to other recipes in the system. During this mapping, the command fills in
973the names of those recipes as part of the :term:`DEPENDS` variable within the
974recipe. If a dependency cannot be mapped, ``devtool`` places a comment
975in the recipe indicating such. The inability to map a dependency can
976result from naming not being recognized or because the dependency simply
977is not available. For cases where the dependency is not available, you
978must use the ``devtool add`` command to add an additional recipe that
979satisfies the dependency. Once you add that recipe, you need to update
980the :term:`DEPENDS` variable in the original recipe to include the new
981recipe.
982
983If you need to add runtime dependencies, you can do so by adding the
984following to your recipe::
985
986 RDEPENDS:${PN} += "dependency1 dependency2 ..."
987
988.. note::
989
990 The ``devtool add`` command often cannot distinguish between mandatory and
991 optional dependencies. Consequently, some of the detected dependencies might
992 in fact be optional. When in doubt, consult the documentation or the
993 configure script for the software the recipe is building for further
994 details. In some cases, you might find you can substitute the
995 dependency with an option that disables the associated functionality
996 passed to the configure script.
997
998License Detection
999-----------------
1000
1001The ``devtool add`` command attempts to determine if the software you are
1002adding is able to be distributed under a common, open-source license. If
1003so, the command sets the :term:`LICENSE` value accordingly.
1004You should double-check the value added by the command against the
1005documentation or source files for the software you are building and, if
1006necessary, update that :term:`LICENSE` value.
1007
1008The ``devtool add`` command also sets the :term:`LIC_FILES_CHKSUM`
1009value to point to all files that appear to be license-related. Realize
1010that license statements often appear in comments at the top of source
1011files or within the documentation. In such cases, the command does not
1012recognize those license statements. Consequently, you might need to
1013amend the :term:`LIC_FILES_CHKSUM` variable to point to one or more of those
1014comments if present. Setting :term:`LIC_FILES_CHKSUM` is particularly
1015important for third-party software. The mechanism attempts to ensure
1016correct licensing should you upgrade the recipe to a newer upstream
1017version in future. Any change in licensing is detected and you receive
1018an error prompting you to check the license text again.
1019
1020If the ``devtool add`` command cannot determine licensing information,
1021``devtool`` sets the :term:`LICENSE` value to "CLOSED" and leaves the
1022:term:`LIC_FILES_CHKSUM` value unset. This behavior allows you to continue
1023with development even though the settings are unlikely to be correct in
1024all cases. You should check the documentation or source files for the
1025software you are building to determine the actual license.
1026
1027Adding Makefile-Only Software
1028-----------------------------
1029
1030The use of Make by itself is very common in both proprietary and
1031open-source software. Unfortunately, Makefiles are often not written
1032with cross-compilation in mind. Thus, ``devtool add`` often cannot do
1033very much to ensure that these Makefiles build correctly. It is very
1034common, for example, to explicitly call ``gcc`` instead of using the
1035:term:`CC` variable. Usually, in a
1036cross-compilation environment, ``gcc`` is the compiler for the build
1037host and the cross-compiler is named something similar to
1038``arm-poky-linux-gnueabi-gcc`` and might require arguments (e.g. to
1039point to the associated sysroot for the target machine).
1040
1041When writing a recipe for Makefile-only software, keep the following in
1042mind:
1043
1044- You probably need to patch the Makefile to use variables instead of
1045 hardcoding tools within the toolchain such as ``gcc`` and ``g++``.
1046
1047- The environment in which Make runs is set up with various standard
1048 variables for compilation (e.g. :term:`CC`, :term:`CXX`, and so forth) in a
1049 similar manner to the environment set up by an :ref:`SDK
1050 <overview-manual/concepts:Application Development SDK>`'s environment
1051 setup script. One easy way to see these variables is to run the
1052 ``devtool build`` command on the recipe and then look in
1053 ``oe-logs/run.do_compile``. Towards the top of this file, there is
1054 a list of environment variables that are set. You can take
1055 advantage of these variables within the Makefile.
1056
1057- If the Makefile sets a default for a variable using "=", that default
1058 overrides the value set in the environment, which is usually not
1059 desirable. For this case, you can either patch the Makefile so it
1060 sets the default using the "?=" operator, or you can alternatively
1061 force the value on the ``make`` command line. To force the value on
1062 the command line, add the variable setting to
1063 :term:`EXTRA_OEMAKE` or
1064 :term:`PACKAGECONFIG_CONFARGS`
1065 within the recipe. Here is an example using :term:`EXTRA_OEMAKE`::
1066
1067 EXTRA_OEMAKE += "'CC=${CC}' 'CXX=${CXX}'"
1068
1069 In the above example,
1070 single quotes are used around the variable settings as the values are
1071 likely to contain spaces because required default options are passed
1072 to the compiler.
1073
1074- Hardcoding paths inside Makefiles is often problematic in a
1075 cross-compilation environment. This is particularly true because
1076 those hardcoded paths often point to locations on the build host and
1077 thus will either be read-only or will introduce contamination into
1078 the cross-compilation because they are specific to the build host
1079 rather than the target. Patching the Makefile to use prefix variables
1080 or other path variables is usually the way to handle this situation.
1081
1082- Sometimes a Makefile runs target-specific commands such as
1083 ``ldconfig``. For such cases, you might be able to apply patches that
1084 remove these commands from the Makefile.
1085
1086Adding Native Tools
1087-------------------
1088
1089Often, you need to build additional tools that run on the :term:`Build Host`
1090as opposed to the target. You should indicate this requirement by using one of
1091the following methods when you run ``devtool add``:
1092
1093- Specify the name of the recipe such that it ends with "-native".
1094 Specifying the name like this produces a recipe that only builds for
1095 the build host.
1096
1097- Specify the "--also-native" option with the ``devtool add``
1098 command. Specifying this option creates a recipe file that still
1099 builds for the target but also creates a variant with a "-native"
1100 suffix that builds for the build host.
1101
1102.. note::
1103
1104 If you need to add a tool that is shipped as part of a source tree
1105 that builds code for the target, you can typically accomplish this by
1106 building the native and target parts separately rather than within
1107 the same compilation process. Realize though that with the
1108 "--also-native" option, you can add the tool using just one
1109 recipe file.
1110
1111Adding Node.js Modules
1112----------------------
1113
1114You can use the ``devtool add`` command two different ways to add
1115Node.js modules: through ``npm`` or from a repository or local source.
1116
1117Use the following form to add Node.js modules through ``npm``::
1118
1119 $ devtool add "npm://registry.npmjs.org;name=forever;version=0.15.1"
1120
1121The name and
1122version parameters are mandatory. Lockdown and shrinkwrap files are
1123generated and pointed to by the recipe in order to freeze the version
1124that is fetched for the dependencies according to the first time. This
1125also saves checksums that are verified on future fetches. Together,
1126these behaviors ensure the reproducibility and integrity of the build.
1127
1128.. note::
1129
1130 - You must use quotes around the URL. ``devtool add`` does not
1131 require the quotes, but the shell considers ";" as a splitter
1132 between multiple commands. Thus, without the quotes,
1133 ``devtool add`` does not receive the other parts, which results in
1134 several "command not found" errors.
1135
1136As mentioned earlier, you can also add Node.js modules directly from a
1137repository or local source tree. To add modules this way, use
1138``devtool add`` in the following form::
1139
1140 $ devtool add https://github.com/diversario/node-ssdp
1141
1142In this example, ``devtool`` fetches the specified Git repository, detects the
1143code as Node.js code, fetches dependencies using ``npm``, and sets
1144:term:`SRC_URI` accordingly.
1145
1146Working With Recipes
1147====================
1148
1149When building a recipe using the ``devtool build`` command, the typical
1150build progresses as follows:
1151
1152#. Fetch the source
1153
1154#. Unpack the source
1155
1156#. Configure the source
1157
1158#. Compile the source
1159
1160#. Install the build output
1161
1162#. Package the installed output
1163
1164For recipes in the workspace, fetching and unpacking is disabled as the
1165source tree has already been prepared and is persistent. Each of these
1166build steps is defined as a function (task), usually with a "do\_" prefix
1167(e.g. :ref:`ref-tasks-fetch`,
1168:ref:`ref-tasks-unpack`, and so
1169forth). These functions are typically shell scripts but can instead be
1170written in Python.
1171
1172If you look at the contents of a recipe, you will see that the recipe
1173does not include complete instructions for building the software.
1174Instead, common functionality is encapsulated in classes inherited with
1175the ``inherit`` directive. This technique leaves the recipe to describe
1176just the things that are specific to the software being built. There is
1177a :ref:`ref-classes-base` class that is implicitly inherited by all recipes
1178and provides the functionality that most recipes typically need.
1179
1180The remainder of this section presents information useful when working
1181with recipes.
1182
1183Finding Logs and Work Files
1184---------------------------
1185
1186After the first run of the ``devtool build`` command, recipes that were
1187previously created using the ``devtool add`` command or whose sources
1188were modified using the ``devtool modify`` command contain symbolic
1189links created within the source tree:
1190
1191- ``oe-logs``: This link points to the directory in which log files and
1192 run scripts for each build step are created.
1193
1194- ``oe-workdir``: This link points to the temporary work area for the
1195 recipe. The following locations under ``oe-workdir`` are particularly
1196 useful:
1197
1198 - ``image/``: Contains all of the files installed during the
1199 :ref:`ref-tasks-install` stage.
1200 Within a recipe, this directory is referred to by the expression
1201 ``${``\ :term:`D`\ ``}``.
1202
1203 - ``sysroot-destdir/``: Contains a subset of files installed within
1204 :ref:`ref-tasks-install` that have been put into the shared sysroot. For
1205 more information, see the
1206 ":ref:`dev-manual/new-recipe:sharing files between recipes`" section.
1207
1208 - ``packages-split/``: Contains subdirectories for each package
1209 produced by the recipe. For more information, see the
1210 ":ref:`dev-manual/devtool:packaging`" section.
1211
1212You can use these links to get more information on what is happening at
1213each build step.
1214
1215Setting Configure Arguments
1216---------------------------
1217
1218If the software your recipe is building uses GNU autoconf, then a fixed
1219set of arguments is passed to it to enable cross-compilation plus any
1220extras specified by :term:`EXTRA_OECONF` or :term:`PACKAGECONFIG_CONFARGS`
1221set within the recipe. If you wish to pass additional options, add them
1222to :term:`EXTRA_OECONF` or :term:`PACKAGECONFIG_CONFARGS`. Other supported build
1223tools have similar variables (e.g. :term:`EXTRA_OECMAKE` for CMake,
1224:term:`EXTRA_OESCONS` for Scons, and so forth). If you need to pass anything on
1225the ``make`` command line, you can use :term:`EXTRA_OEMAKE` or the
1226:term:`PACKAGECONFIG_CONFARGS` variables to do so.
1227
1228You can use the ``devtool configure-help`` command to help you set the
1229arguments listed in the previous paragraph. The command determines the
1230exact options being passed, and shows them to you along with any custom
1231arguments specified through :term:`EXTRA_OECONF` or
1232:term:`PACKAGECONFIG_CONFARGS`. If applicable, the command also shows you
1233the output of the configure script's "--help" option as a
1234reference.
1235
1236Sharing Files Between Recipes
1237-----------------------------
1238
1239Recipes often need to use files provided by other recipes on the
1240:term:`Build Host`. For example,
1241an application linking to a common library needs access to the library
1242itself and its associated headers. The way this access is accomplished
1243is through the :term:`Sysroot`. There is a sysroot for the target machine, and a
1244sysroot for the build host.
1245
1246Recipes should never write files directly into the sysroot. Instead,
1247files should be installed into standard locations during the
1248:ref:`ref-tasks-install` task within the ``${``\ :term:`D`\ ``}`` directory. A
1249subset of these files automatically goes into the sysroot. The reason
1250for this limitation is that almost all files that go into the sysroot
1251are cataloged in manifests in order to ensure they can be removed later
1252when a recipe is modified or removed. Thus, the sysroot is able to
1253remain free from stale files.
1254
1255Packaging
1256---------
1257
1258If you examine how build output gets into the final image on
1259the target device, it is important to understand packaging because the
1260contents of the image are expressed in terms of packages and not
1261recipes.
1262
1263During the :ref:`ref-tasks-package` task, files installed during the
1264:ref:`ref-tasks-install` task are split into one main package, which is almost
1265always named the same as the recipe, and into several other packages. This
1266separation exists because not all of those installed files are useful in every
1267image. For example, you probably do not need any of the documentation installed
1268in a production image. Consequently, for each recipe the documentation
1269files are separated into a ``-doc`` package. Recipes that package
1270software containing optional modules or plugins might undergo additional
1271package splitting as well.
1272
1273After building a recipe, you can see where files have gone by looking in
1274the ``oe-workdir/packages-split`` directory, which contains a
1275subdirectory for each package. Apart from some advanced cases, the
1276:term:`PACKAGES` and :term:`FILES` variables controls
1277splitting. The :term:`PACKAGES` variable lists all of the packages to be
1278produced, while the :term:`FILES` variable specifies which files to include
1279in each package by using an override to specify the package. For
1280example, ``FILES:${PN}`` specifies the files to go into the main package
1281(i.e. the main package has the same name as the recipe and
1282``${``\ :term:`PN`\ ``}`` evaluates to the
1283recipe name). The order of the :term:`PACKAGES` value is significant. For
1284each installed file, the first package whose :term:`FILES` value matches the
1285file is the package into which the file goes. Both the :term:`PACKAGES` and
1286:term:`FILES` variables have default values. Consequently, you might find
1287you do not even need to set these variables in your recipe unless the
1288software the recipe is building installs files into non-standard
1289locations.
1290
1291Restoring the Target Device to its Original State
1292=================================================
1293
1294If you use the ``devtool deploy-target`` command to write a recipe's
1295build output to the target, and you are working on an existing component
1296of the system, then you might find yourself in a situation where you
1297need to restore the original files that existed prior to running the
1298``devtool deploy-target`` command. Because the ``devtool deploy-target``
1299command backs up any files it overwrites, you can use the
1300``devtool undeploy-target`` command to restore those files and remove
1301any other files the recipe deployed. Consider the following example::
1302
1303 $ devtool undeploy-target lighttpd root@192.168.7.2
1304
1305If you have deployed
1306multiple applications, you can remove them all using the "-a" option
1307thus restoring the target device to its original state::
1308
1309 $ devtool undeploy-target -a root@192.168.7.2
1310
1311Information about files deployed to
1312the target as well as any backed up files are stored on the target
1313itself. This storage, of course, requires some additional space on the
1314target machine.
1315
1316.. note::
1317
1318 The ``devtool deploy-target`` and ``devtool undeploy-target`` commands do
1319 not currently interact with any package management system on the target
1320 device (e.g. RPM or OPKG). Consequently, you should not intermingle
1321 ``devtool deploy-target`` and package manager operations on the target
1322 device. Doing so could result in a conflicting set of files.
diff --git a/documentation/dev-manual/disk-space.rst b/documentation/dev-manual/disk-space.rst
deleted file mode 100644
index ba3afa5a2c..0000000000
--- a/documentation/dev-manual/disk-space.rst
+++ /dev/null
@@ -1,61 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Conserving Disk Space
4*********************
5
6Conserving Disk Space During Builds
7===================================
8
9To help conserve disk space during builds, you can add the following
10statement to your project's ``local.conf`` configuration file found in
11the :term:`Build Directory`::
12
13 INHERIT += "rm_work"
14
15Adding this statement deletes the work directory used for
16building a recipe once the recipe is built. For more information on
17"rm_work", see the :ref:`ref-classes-rm-work` class in the
18Yocto Project Reference Manual.
19
20When you inherit this class and build a ``core-image-sato`` image for a
21``qemux86-64`` machine from an Ubuntu 22.04 x86-64 system, you end up with a
22final disk usage of 22 Gbytes instead of &MIN_DISK_SPACE; Gbytes. However,
23&MIN_DISK_SPACE_RM_WORK; Gbytes of initial free disk space are still needed to
24create temporary files before they can be deleted.
25
26Purging Obsolete Shared State Cache Files
27=========================================
28
29After multiple build iterations, the Shared State (sstate) cache can contain
30multiple cache files for a given package, consuming a substantial amount of
31disk space. However, only the most recent ones are likely to be reused.
32
33The following command is a quick way to purge all the cache files which
34haven't been used for a least a specified number of days::
35
36 find build/sstate-cache -type f -mtime +$DAYS -delete
37
38The above command relies on the fact that BitBake touches the sstate cache
39files as it accesses them, when it has write access to the cache.
40
41You could use ``-atime`` instead of ``-mtime`` if the partition isn't mounted
42with the ``noatime`` option for a read only cache.
43
44For more advanced needs, OpenEmbedded-Core also offers a more elaborate
45command. It has the ability to purge all but the newest cache files on each
46architecture, and also to remove files that it considers unreachable by
47exploring a set of build configurations. However, this command
48requires a full build environment to be available and doesn't work well
49covering multiple releases. It won't work either on limited environments
50such as BSD based NAS::
51
52 sstate-cache-management.py --remove-duplicated --cache-dir=sstate-cache
53
54This command will ask you to confirm the deletions it identifies.
55Run ``sstate-cache-management.py`` for more details about this script.
56
57.. note::
58
59 As this command is much more cautious and selective, removing only cache files,
60 it will execute much slower than the simple ``find`` command described above.
61 Therefore, it may not be your best option to trim huge cache directories.
diff --git a/documentation/dev-manual/efficiently-fetching-sources.rst b/documentation/dev-manual/efficiently-fetching-sources.rst
deleted file mode 100644
index a3366226c0..0000000000
--- a/documentation/dev-manual/efficiently-fetching-sources.rst
+++ /dev/null
@@ -1,68 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Efficiently Fetching Source Files During a Build
4************************************************
5
6The OpenEmbedded build system works with source files located through
7the :term:`SRC_URI` variable. When
8you build something using BitBake, a big part of the operation is
9locating and downloading all of the source code. For images,
10downloading all the source for various packages can take a significant
11amount of time.
12
13This section shows you how you can use mirrors to speed up fetching
14source files and how you can pre-fetch files all of which leads to more
15efficient use of resources and time.
16
17Setting up Effective Mirrors
18============================
19
20A good deal that goes into a Yocto Project build is simply downloading
21source code. Maybe you have been working with another
22build system for which you have built up a
23sizable directory of source tarballs. Or, perhaps someone else has such
24a directory for which you have read access. If so, you can save time by
25adding statements to your configuration file so that the build process
26checks local directories first for existing tarballs before checking the
27Internet.
28
29Here is an efficient way to set it up in your ``local.conf`` file::
30
31 SOURCE_MIRROR_URL ?= "file:///home/you/your-download-dir/"
32 INHERIT += "own-mirrors"
33 BB_GENERATE_MIRROR_TARBALLS = "1"
34 # BB_NO_NETWORK = "1"
35
36In the previous example, the
37:term:`BB_GENERATE_MIRROR_TARBALLS`
38variable causes the OpenEmbedded build system to generate tarballs of
39the Git repositories and store them in the
40:term:`DL_DIR` directory. Due to
41performance reasons, generating and storing these tarballs is not the
42build system's default behavior.
43
44You can also use the
45:term:`PREMIRRORS` variable. For
46an example, see the variable's glossary entry in the Yocto Project
47Reference Manual.
48
49Getting Source Files and Suppressing the Build
50==============================================
51
52Another technique you can use to ready yourself for a successive string
53of build operations, is to pre-fetch all the source files without
54actually starting a build. This technique lets you work through any
55download issues and ultimately gathers all the source files into your
56download directory :ref:`structure-build-downloads`,
57which is located with :term:`DL_DIR`.
58
59Use the following BitBake command form to fetch all the necessary
60sources without starting the build::
61
62 $ bitbake target --runall=fetch
63
64This
65variation of the BitBake command guarantees that you have all the
66sources for that BitBake target should you disconnect from the Internet
67and want to do the build later offline.
68
diff --git a/documentation/dev-manual/error-reporting-tool.rst b/documentation/dev-manual/error-reporting-tool.rst
deleted file mode 100644
index 84f3d9cd1e..0000000000
--- a/documentation/dev-manual/error-reporting-tool.rst
+++ /dev/null
@@ -1,84 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using the Error Reporting Tool
4******************************
5
6The error reporting tool allows you to submit errors encountered during
7builds to a central database. Outside of the build environment, you can
8use a web interface to browse errors, view statistics, and query for
9errors. The tool works using a client-server system where the client
10portion is integrated with the installed Yocto Project
11:term:`Source Directory` (e.g. ``poky``).
12The server receives the information collected and saves it in a
13database.
14
15There is a live instance of the error reporting server at
16https://errors.yoctoproject.org.
17When you want to get help with build failures, you can submit all of the
18information on the failure easily and then point to the URL in your bug
19report or send an email to the mailing list.
20
21.. note::
22
23 If you send error reports to this server, the reports become publicly
24 visible.
25
26Enabling and Using the Tool
27===========================
28
29By default, the error reporting tool is disabled. You can enable it by
30inheriting the :ref:`ref-classes-report-error` class by adding the
31following statement to the end of your ``local.conf`` file in your
32:term:`Build Directory`::
33
34 INHERIT += "report-error"
35
36By default, the error reporting feature stores information in
37``${``\ :term:`LOG_DIR`\ ``}/error-report``.
38However, you can specify a directory to use by adding the following to
39your ``local.conf`` file::
40
41 ERR_REPORT_DIR = "path"
42
43Enabling error
44reporting causes the build process to collect the errors and store them
45in a file as previously described. When the build system encounters an
46error, it includes a command as part of the console output. You can run
47the command to send the error file to the server. For example, the
48following command sends the errors to an upstream server::
49
50 $ send-error-report /home/brandusa/project/poky/build/tmp/log/error-report/error_report_201403141617.txt
51
52In the previous example, the errors are sent to a public database
53available at https://errors.yoctoproject.org, which is used by the
54entire community. If you specify a particular server, you can send the
55errors to a different database. Use the following command for more
56information on available options::
57
58 $ send-error-report --help
59
60When sending the error file, you are prompted to review the data being
61sent as well as to provide a name and optional email address. Once you
62satisfy these prompts, the command returns a link from the server that
63corresponds to your entry in the database. For example, here is a
64typical link: https://errors.yoctoproject.org/Errors/Details/9522/
65
66Following the link takes you to a web interface where you can browse,
67query the errors, and view statistics.
68
69Disabling the Tool
70==================
71
72To disable the error reporting feature, simply remove or comment out the
73following statement from the end of your ``local.conf`` file in your
74:term:`Build Directory`::
75
76 INHERIT += "report-error"
77
78Setting Up Your Own Error Reporting Server
79==========================================
80
81If you want to set up your own error reporting server, you can obtain
82the code from the Git repository at :yocto_git:`/error-report-web/`.
83Instructions on how to set it up are in the README document.
84
diff --git a/documentation/dev-manual/external-scm.rst b/documentation/dev-manual/external-scm.rst
deleted file mode 100644
index 896b1b5ac7..0000000000
--- a/documentation/dev-manual/external-scm.rst
+++ /dev/null
@@ -1,70 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using an External SCM
4*********************
5
6If you're working on a recipe that pulls from an external Source Code
7Manager (SCM), it is possible to have the OpenEmbedded build system
8notice new recipe changes added to the SCM and then build the resulting
9packages that depend on the new recipes by using the latest versions.
10This only works for SCMs from which it is possible to get a sensible
11revision number for changes. Currently, you can do this with Apache
12Subversion (SVN), Git, and Bazaar (BZR) repositories.
13
14To enable this behavior, the :term:`PV` of
15the recipe needs to include a ``+`` sign in its assignment.
16Here is an example::
17
18 PV = "1.2.3+git"
19
20:term:`Bitbake` later includes the source control information in :term:`PKGV`
21during the packaging phase.
22
23Then, you can add the following to your
24``local.conf``::
25
26 SRCREV:pn-PN = "${AUTOREV}"
27
28:term:`PN` is the name of the recipe for
29which you want to enable automatic source revision updating.
30
31If you do not want to update your local configuration file, you can add
32the following directly to the recipe to finish enabling the feature::
33
34 SRCREV = "${AUTOREV}"
35
36The Yocto Project provides a distribution named ``poky-bleeding``, whose
37configuration file contains the line::
38
39 require conf/distro/include/poky-floating-revisions.inc
40
41This line pulls in the
42listed include file that contains numerous lines of exactly that form::
43
44 #SRCREV:pn-opkg-native ?= "${AUTOREV}"
45 #SRCREV:pn-opkg-sdk ?= "${AUTOREV}"
46 #SRCREV:pn-opkg ?= "${AUTOREV}"
47 #SRCREV:pn-opkg-utils-native ?= "${AUTOREV}"
48 #SRCREV:pn-opkg-utils ?= "${AUTOREV}"
49 SRCREV:pn-gconf-dbus ?= "${AUTOREV}"
50 SRCREV:pn-matchbox-common ?= "${AUTOREV}"
51 SRCREV:pn-matchbox-config-gtk ?= "${AUTOREV}"
52 SRCREV:pn-matchbox-desktop ?= "${AUTOREV}"
53 SRCREV:pn-matchbox-keyboard ?= "${AUTOREV}"
54 SRCREV:pn-matchbox-panel-2 ?= "${AUTOREV}"
55 SRCREV:pn-matchbox-themes-extra ?= "${AUTOREV}"
56 SRCREV:pn-matchbox-terminal ?= "${AUTOREV}"
57 SRCREV:pn-matchbox-wm ?= "${AUTOREV}"
58 SRCREV:pn-settings-daemon ?= "${AUTOREV}"
59 SRCREV:pn-screenshot ?= "${AUTOREV}"
60 . . .
61
62These lines allow you to
63experiment with building a distribution that tracks the latest
64development source for numerous packages.
65
66.. note::
67
68 The ``poky-bleeding`` distribution is not tested on a regular basis. Keep
69 this in mind if you use it.
70
diff --git a/documentation/dev-manual/external-toolchain.rst b/documentation/dev-manual/external-toolchain.rst
deleted file mode 100644
index 238f8cf467..0000000000
--- a/documentation/dev-manual/external-toolchain.rst
+++ /dev/null
@@ -1,40 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Optionally Using an External Toolchain
4**************************************
5
6You might want to use an external toolchain as part of your development.
7If this is the case, the fundamental steps you need to accomplish are as
8follows:
9
10- Understand where the installed toolchain resides. For cases where you
11 need to build the external toolchain, you would need to take separate
12 steps to build and install the toolchain.
13
14- Make sure you add the layer that contains the toolchain to your
15 ``bblayers.conf`` file through the
16 :term:`BBLAYERS` variable.
17
18- Set the :term:`EXTERNAL_TOOLCHAIN` variable in your ``local.conf`` file
19 to the location in which you installed the toolchain.
20
21The toolchain configuration is very flexible and customizable. It
22is primarily controlled with the :term:`TCMODE` variable. This variable
23controls which ``tcmode-*.inc`` file to include from the
24``meta/conf/distro/include`` directory within the :term:`Source Directory`.
25
26The default value of :term:`TCMODE` is "default", which tells the
27OpenEmbedded build system to use its internally built toolchain (i.e.
28``tcmode-default.inc``). However, other patterns are accepted. In
29particular, "external-\*" refers to external toolchains. One example is
30the Mentor Graphics Sourcery G++ Toolchain. Support for this toolchain resides
31in the separate ``meta-sourcery`` layer at
32https://github.com/MentorEmbedded/meta-sourcery/.
33See its ``README`` file for details about how to use this layer.
34
35Another example of external toolchain layer is
36:yocto_git:`meta-arm-toolchain </meta-arm/tree/meta-arm-toolchain/>`
37supporting GNU toolchains released by ARM.
38
39You can find further information by reading about the :term:`TCMODE` variable
40in the Yocto Project Reference Manual's variable glossary.
diff --git a/documentation/dev-manual/figures/bitbake-build-flow.png b/documentation/dev-manual/figures/bitbake-build-flow.png
deleted file mode 100644
index 108265ae0c..0000000000
--- a/documentation/dev-manual/figures/bitbake-build-flow.png
+++ /dev/null
Binary files differ
diff --git a/documentation/dev-manual/figures/buildhistory-web.png b/documentation/dev-manual/figures/buildhistory-web.png
deleted file mode 100644
index f6db86c977..0000000000
--- a/documentation/dev-manual/figures/buildhistory-web.png
+++ /dev/null
Binary files differ
diff --git a/documentation/dev-manual/figures/buildhistory.png b/documentation/dev-manual/figures/buildhistory.png
deleted file mode 100644
index bd5f8a4908..0000000000
--- a/documentation/dev-manual/figures/buildhistory.png
+++ /dev/null
Binary files differ
diff --git a/documentation/dev-manual/figures/cute-files-npm-example.png b/documentation/dev-manual/figures/cute-files-npm-example.png
deleted file mode 100644
index a02cca097f..0000000000
--- a/documentation/dev-manual/figures/cute-files-npm-example.png
+++ /dev/null
Binary files differ
diff --git a/documentation/dev-manual/figures/dev-title.png b/documentation/dev-manual/figures/dev-title.png
deleted file mode 100644
index 15e67d0744..0000000000
--- a/documentation/dev-manual/figures/dev-title.png
+++ /dev/null
Binary files differ
diff --git a/documentation/dev-manual/figures/devtool-add-flow.png b/documentation/dev-manual/figures/devtool-add-flow.png
deleted file mode 100644
index e7d6173d2d..0000000000
--- a/documentation/dev-manual/figures/devtool-add-flow.png
+++ /dev/null
Binary files differ
diff --git a/documentation/dev-manual/figures/devtool-modify-flow.png b/documentation/dev-manual/figures/devtool-modify-flow.png
deleted file mode 100644
index 18ba8b7e65..0000000000
--- a/documentation/dev-manual/figures/devtool-modify-flow.png
+++ /dev/null
Binary files differ
diff --git a/documentation/dev-manual/figures/devtool-upgrade-flow.png b/documentation/dev-manual/figures/devtool-upgrade-flow.png
deleted file mode 100644
index 7d4f395e24..0000000000
--- a/documentation/dev-manual/figures/devtool-upgrade-flow.png
+++ /dev/null
Binary files differ
diff --git a/documentation/dev-manual/figures/multiconfig_files.png b/documentation/dev-manual/figures/multiconfig_files.png
deleted file mode 100644
index 0b59338b3a..0000000000
--- a/documentation/dev-manual/figures/multiconfig_files.png
+++ /dev/null
Binary files differ
diff --git a/documentation/dev-manual/figures/recipe-workflow.png b/documentation/dev-manual/figures/recipe-workflow.png
deleted file mode 100644
index c0e960b13b..0000000000
--- a/documentation/dev-manual/figures/recipe-workflow.png
+++ /dev/null
Binary files differ
diff --git a/documentation/dev-manual/gobject-introspection.rst b/documentation/dev-manual/gobject-introspection.rst
deleted file mode 100644
index f7206e6fae..0000000000
--- a/documentation/dev-manual/gobject-introspection.rst
+++ /dev/null
@@ -1,155 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Enabling GObject Introspection Support
4**************************************
5
6`GObject introspection <https://gi.readthedocs.io/en/latest/>`__
7is the standard mechanism for accessing GObject-based software from
8runtime environments. GObject is a feature of the GLib library that
9provides an object framework for the GNOME desktop and related software.
10GObject Introspection adds information to GObject that allows objects
11created within it to be represented across different programming
12languages. If you want to construct GStreamer pipelines using Python, or
13control UPnP infrastructure using Javascript and GUPnP, GObject
14introspection is the only way to do it.
15
16This section describes the Yocto Project support for generating and
17packaging GObject introspection data. GObject introspection data is a
18description of the API provided by libraries built on top of the GLib
19framework, and, in particular, that framework's GObject mechanism.
20GObject Introspection Repository (GIR) files go to ``-dev`` packages,
21``typelib`` files go to main packages as they are packaged together with
22libraries that are introspected.
23
24The data is generated when building such a library, by linking the
25library with a small executable binary that asks the library to describe
26itself, and then executing the binary and processing its output.
27
28Generating this data in a cross-compilation environment is difficult
29because the library is produced for the target architecture, but its
30code needs to be executed on the build host. This problem is solved with
31the OpenEmbedded build system by running the code through QEMU, which
32allows precisely that. Unfortunately, QEMU does not always work
33perfectly as mentioned in the ":ref:`dev-manual/gobject-introspection:known issues`"
34section.
35
36Enabling the Generation of Introspection Data
37=============================================
38
39Enabling the generation of introspection data (GIR files) in your
40library package involves the following:
41
42#. Inherit the :ref:`ref-classes-gobject-introspection` class.
43
44#. Make sure introspection is not disabled anywhere in the recipe or
45 from anything the recipe includes. Also, make sure that
46 "gobject-introspection-data" is not in
47 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
48 and that "qemu-usermode" is not in
49 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
50 In either of these conditions, nothing will happen.
51
52#. Try to build the recipe. If you encounter build errors that look like
53 something is unable to find ``.so`` libraries, check where these
54 libraries are located in the source tree and add the following to the
55 recipe::
56
57 GIR_EXTRA_LIBS_PATH = "${B}/something/.libs"
58
59 .. note::
60
61 See recipes in the ``oe-core`` repository that use that
62 :term:`GIR_EXTRA_LIBS_PATH` variable as an example.
63
64#. Look for any other errors, which probably mean that introspection
65 support in a package is not entirely standard, and thus breaks down
66 in a cross-compilation environment. For such cases, custom-made fixes
67 are needed. A good place to ask and receive help in these cases is
68 the :ref:`Yocto Project mailing
69 lists <resources-mailinglist>`.
70
71.. note::
72
73 Using a library that no longer builds against the latest Yocto
74 Project release and prints introspection related errors is a good
75 candidate for the previous procedure.
76
77Disabling the Generation of Introspection Data
78==============================================
79
80You might find that you do not want to generate introspection data. Or,
81perhaps QEMU does not work on your build host and target architecture
82combination. If so, you can use either of the following methods to
83disable GIR file generations:
84
85- Add the following to your distro configuration::
86
87 DISTRO_FEATURES_BACKFILL_CONSIDERED = "gobject-introspection-data"
88
89 Adding this statement disables generating introspection data using
90 QEMU but will still enable building introspection tools and libraries
91 (i.e. building them does not require the use of QEMU).
92
93- Add the following to your machine configuration::
94
95 MACHINE_FEATURES_BACKFILL_CONSIDERED = "qemu-usermode"
96
97 Adding this statement disables the use of QEMU when building packages for your
98 machine. Currently, this feature is used only by introspection
99 recipes and has the same effect as the previously described option.
100
101 .. note::
102
103 Future releases of the Yocto Project might have other features
104 affected by this option.
105
106If you disable introspection data, you can still obtain it through other
107means such as copying the data from a suitable sysroot, or by generating
108it on the target hardware. The OpenEmbedded build system does not
109currently provide specific support for these techniques.
110
111Testing that Introspection Works in an Image
112============================================
113
114Use the following procedure to test if generating introspection data is
115working in an image:
116
117#. Make sure that "gobject-introspection-data" is not in
118 :term:`DISTRO_FEATURES_BACKFILL_CONSIDERED`
119 and that "qemu-usermode" is not in
120 :term:`MACHINE_FEATURES_BACKFILL_CONSIDERED`.
121
122#. Build ``core-image-sato``.
123
124#. Launch a Terminal and then start Python in the terminal.
125
126#. Enter the following in the terminal::
127
128 >>> from gi.repository import GLib
129 >>> GLib.get_host_name()
130
131#. For something a little more advanced, enter the following see:
132 https://python-gtk-3-tutorial.readthedocs.io/en/latest/introduction.html
133
134Known Issues
135============
136
137Here are know issues in GObject Introspection Support:
138
139- ``qemu-ppc64`` immediately crashes. Consequently, you cannot build
140 introspection data on that architecture.
141
142- x32 is not supported by QEMU. Consequently, introspection data is
143 disabled.
144
145- musl causes transient GLib binaries to crash on assertion failures.
146 Consequently, generating introspection data is disabled.
147
148- Because QEMU is not able to run the binaries correctly, introspection
149 is disabled for some specific packages under specific architectures
150 (e.g. ``gcr``, ``libsecret``, and ``webkit``).
151
152- QEMU usermode might not work properly when running 64-bit binaries
153 under 32-bit host machines. In particular, "qemumips64" is known to
154 not work under i686.
155
diff --git a/documentation/dev-manual/hashequivserver.rst b/documentation/dev-manual/hashequivserver.rst
deleted file mode 100644
index c866f5001f..0000000000
--- a/documentation/dev-manual/hashequivserver.rst
+++ /dev/null
@@ -1,129 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Setting up a Hash Equivalence Server
4************************************
5
6A :ref:`overview-manual/concepts:Hash Equivalence` server can help reduce build
7times by using the mechanism described in the :ref:`overview-manual/concepts:Hash Equivalence`
8section of the Yocto Project Overview and Concepts Manual.
9
10This document will guide you through the steps required to set up the reference
11Hash Equivalence server provided by the :oe_git:`bitbake-hashserv
12</bitbake/tree/bin/bitbake-hashserv>` script in :term:`BitBake`.
13
14This guide will explain how to setup a local Hash Equivalence server and does
15not explain how to setup the surrounding infrastructure to secure this server.
16
17Hash Equivalence Server Setup
18=============================
19
20From this point onwards, the commands displayed below are assumed to be run from
21the :term:`BitBake` repository, which can be found :oe_git:`here </bitbake>`.
22
23To start a basic Hash Equivalence server, one could simply run::
24
25 ./bin/bitbake-hashserv
26
27This will take all of the default options of the script, which are already
28sufficient to start a local server.
29
30Run ``./bin/bitbake-hashserv --help`` to see what options the script can take.
31Some of the important ones are:
32
33- ``--database`` controls the location of the hash server database (default:
34 ``./hashserver.db``).
35
36- ``--bind`` controls the bind address of the server (default:
37 ``unix://./hashserver.sock``).
38
39 You can specify three types of addresses:
40
41 - ``unix://PATH``: will bind to a Unix socket at ``PATH``.
42
43 - ``wss://ADDRESS:PORT``: will bind to a Websocket at ``ADDRESS:PORT``.
44
45 - ``ADDRESS:PORT``: will bind to a raw TCP socket at ``ADDRESS:PORT``.
46
47- ``--log`` can be used to control the logging level of the server (e.g.
48 ``INFO`` will print information about clients connection to the server).
49
50- ``--upstream`` can be used to specify an upstream server to pull hashes from.
51 This has no default value, meaning no upstream server is used.
52
53- ``--db-username`` and ``--db-password`` can be used to control the access to
54 the database.
55
56- ``--read-only`` will disable hash reports from clients.
57
58These variables can also be set from the environment from where it is being run.
59Run ``./bin/bitbake-hashserv --help`` to get the variable names that you can
60export.
61
62.. warning::
63
64 The shared Hash Equivalence server needs to be maintained together with the
65 :ref:`Shared State <overview-manual/concepts:Shared State>` cache. Otherwise,
66 the server could report Shared State hashes that only exist on specific
67 clients.
68
69 We therefore recommend that one Hash Equivalence server be set up to
70 correspond with a given Shared State cache, and to start this server
71 in *read-only mode* (``--read-only`` option), so that it doesn't store
72 equivalences for Shared State caches that are local to clients.
73
74 If there is no pre-existing Shared State Cache, the server should allow
75 hashes to be reported (no ``--read-only`` option) to create the initial
76 Hash Equivalence database.
77
78Yocto Project Build Setup
79=========================
80
81To use the server started in the previous section, set the following variables
82in a :term:`configuration file`::
83
84 BB_HASHSERVE = "<bind address>"
85 BB_SIGNATURE_HANDLER = "OEEquivHash"
86
87The ``<bind address>`` value should be replaced to point to the server started
88in the previous section.
89
90See the documentation of :term:`BB_SIGNATURE_HANDLER` for more details on this
91variable.
92
93You can optionally specify an upstream server with :term:`BB_HASHSERVE_UPSTREAM`
94variable. For example::
95
96 BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws"
97
98This will make the local server pull hashes from the upstream server. The
99:term:`BB_HASHSERVE_UPSTREAM` only works when a server is running
100(:term:`BB_HASHSERVE` is set).
101
102To output debugging information on what is happening with Hash Equivalence when
103builds are started, you can configure :term:`BitBake` logging as follows from a
104:term:`configuration file`::
105
106 BB_LOGCONFIG = "hashequiv.json"
107
108With ``hashequiv.json`` containing the following content:
109
110.. code-block:: json
111
112 {
113 "version": 1,
114 "loggers": {
115 "BitBake.SigGen.HashEquiv": {
116 "level": "VERBOSE",
117 "handlers": ["BitBake.verbconsole"]
118 },
119 "BitBake.RunQueue.HashEquiv": {
120 "level": "VERBOSE",
121 "handlers": ["BitBake.verbconsole"]
122 }
123 }
124 }
125
126This will make Hash Equivalence related changes be printed on the console, such
127as::
128
129 NOTE: Task <recipe.bb>:do_<task> unihash changed to dc0da29c62a2d78d8d853fbb9c173778fe7d6fa4a68c67494b17afffe8ca1894
diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
deleted file mode 100644
index adf776e00b..0000000000
--- a/documentation/dev-manual/index.rst
+++ /dev/null
@@ -1,58 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3======================================
4Yocto Project Development Tasks Manual
5======================================
6
7.. toctree::
8 :caption: Table of Contents
9 :numbered:
10
11 intro
12 start
13 layers
14 customizing-images
15 new-recipe
16 new-machine
17 upgrading-recipes
18 temporary-source-code
19 creating-fragments
20 poky-manual-setup
21 quilt.rst
22 development-shell
23 python-development-shell
24 building
25 multiconfig
26 speeding-up-build
27 limiting-resources
28 libraries
29 prebuilt-libraries
30 devtool
31 x32-psabi
32 gobject-introspection
33 external-toolchain
34 wic
35 bmaptool
36 securing-images
37 custom-distribution
38 custom-template-configuration-directory
39 disk-space
40 packages
41 efficiently-fetching-sources
42 init-manager
43 device-manager
44 external-scm
45 read-only-rootfs
46 build-quality
47 debugging
48 licenses
49 security-subjects
50 vulnerabilities
51 sbom
52 error-reporting-tool
53 wayland
54 qemu
55 bblock
56 hashequivserver
57
58.. include:: /boilerplate.rst
diff --git a/documentation/dev-manual/init-manager.rst b/documentation/dev-manual/init-manager.rst
deleted file mode 100644
index d0fdfdf773..0000000000
--- a/documentation/dev-manual/init-manager.rst
+++ /dev/null
@@ -1,162 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3.. _init-manager:
4
5Selecting an Initialization Manager
6***********************************
7
8By default, the Yocto Project uses :wikipedia:`SysVinit <Init#SysV-style>` as
9the initialization manager. There is also support for BusyBox init, a simpler
10implementation, as well as support for :wikipedia:`systemd <Systemd>`, which
11is a full replacement for init with parallel starting of services, reduced
12shell overhead, increased security and resource limits for services, and other
13features that are used by many distributions.
14
15Within the system, SysVinit and BusyBox init treat system components as
16services. These services are maintained as shell scripts stored in the
17``/etc/init.d/`` directory.
18
19SysVinit is more elaborate than BusyBox init and organizes services in
20different run levels. This organization is maintained by putting links
21to the services in the ``/etc/rcN.d/`` directories, where `N/` is one
22of the following options: "S", "0", "1", "2", "3", "4", "5", or "6".
23
24.. note::
25
26 Each runlevel has a dependency on the previous runlevel. This
27 dependency allows the services to work properly.
28
29Both SysVinit and BusyBox init are configured through the ``/etc/inittab``
30file, with a very similar syntax, though of course BusyBox init features
31are more limited.
32
33In comparison, systemd treats components as units. Using units is a
34broader concept as compared to using a service. A unit includes several
35different types of entities. ``Service`` is one of the types of entities.
36The runlevel concept in SysVinit corresponds to the concept of a target
37in systemd, where target is also a type of supported unit.
38
39In systems with SysVinit or BusyBox init, services load sequentially (i.e. one
40by one) during init and parallelization is not supported. With systemd, services
41start in parallel. This method can have an impact on the startup performance
42of a given service, though systemd will also provide more services by default,
43therefore increasing the total system boot time. systemd also substantially
44increases system size because of its multiple components and the extra
45dependencies it pulls.
46
47By contrast, BusyBox init is the simplest and the lightest solution and
48also comes with BusyBox mdev as device manager, a lighter replacement to
49:wikipedia:`udev <Udev>`, which SysVinit and systemd both use.
50
51The ":ref:`device-manager`" chapter has more details about device managers.
52
53Using SysVinit with udev
54=========================
55
56SysVinit with the udev device manager corresponds to the
57default setting in Poky. This corresponds to setting::
58
59 INIT_MANAGER = "sysvinit"
60
61Using BusyBox init with BusyBox mdev
62====================================
63
64BusyBox init with BusyBox mdev is the simplest and lightest solution
65for small root filesystems. All you need is BusyBox, which most systems
66have anyway::
67
68 INIT_MANAGER = "mdev-busybox"
69
70Using systemd
71=============
72
73The last option is to use systemd together with the udev device
74manager. This is the most powerful and versatile solution, especially
75for more complex systems::
76
77 INIT_MANAGER = "systemd"
78
79This will enable systemd and remove sysvinit components from the image.
80See :yocto_git:`meta/conf/distro/include/init-manager-systemd.inc
81</poky/tree/meta/conf/distro/include/init-manager-systemd.inc>` for exact
82details on what this does.
83
84Controling systemd from the target command line
85-----------------------------------------------
86
87Here is a quick reference for controling systemd from the command line on the
88target. Instead of opening and sometimes modifying files, most interaction
89happens through the ``systemctl`` and ``journalctl`` commands:
90
91- ``systemctl status``: show the status of all services
92- ``systemctl status <service>``: show the status of one service
93- ``systemctl [start|stop] <service>``: start or stop a service
94- ``systemctl [enable|disable] <service>``: enable or disable a service at boot time
95- ``systemctl list-units``: list all available units
96- ``journalctl -a``: show all logs for all services
97- ``journalctl -f``: show only the last log entries, and keep printing updates as they arrive
98- ``journalctl -u``: show only logs from a particular service
99
100Using systemd-journald without a traditional syslog daemon
101----------------------------------------------------------
102
103Counter-intuitively, ``systemd-journald`` is not a syslog runtime or provider,
104and the proper way to use ``systemd-journald`` as your sole logging mechanism is to
105effectively disable syslog entirely by setting these variables in your distribution
106configuration file::
107
108 VIRTUAL-RUNTIME_syslog = ""
109 VIRTUAL-RUNTIME_base-utils-syslog = ""
110
111Doing so will prevent ``rsyslog`` / ``busybox-syslog`` from being pulled in by
112default, leaving only ``systemd-journald``.
113
114Summary
115-------
116
117The Yocto Project supports three different initialization managers, offering
118increasing levels of complexity and functionality:
119
120.. list-table::
121 :widths: 40 20 20 20
122 :header-rows: 1
123
124 * -
125 - BusyBox init
126 - SysVinit
127 - systemd
128 * - Size
129 - Small
130 - Small
131 - Big [#footnote-systemd-size]_
132 * - Complexity
133 - Small
134 - Medium
135 - High
136 * - Support for boot profiles
137 - No
138 - Yes ("runlevels")
139 - Yes ("targets")
140 * - Services defined as
141 - Shell scripts
142 - Shell scripts
143 - Description files
144 * - Starting services in parallel
145 - No
146 - No
147 - Yes
148 * - Setting service resource limits
149 - No
150 - No
151 - Yes
152 * - Support service isolation
153 - No
154 - No
155 - Yes
156 * - Integrated logging
157 - No
158 - No
159 - Yes
160
161.. [#footnote-systemd-size] Using systemd increases the ``core-image-minimal``
162 image size by 160\% for ``qemux86-64`` on Mickledore (4.2), compared to SysVinit.
diff --git a/documentation/dev-manual/intro.rst b/documentation/dev-manual/intro.rst
deleted file mode 100644
index 0f7370a96d..0000000000
--- a/documentation/dev-manual/intro.rst
+++ /dev/null
@@ -1,59 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3******************************************
4The Yocto Project Development Tasks Manual
5******************************************
6
7Welcome
8=======
9
10Welcome to the Yocto Project Development Tasks Manual. This manual
11provides relevant procedures necessary for developing in the Yocto
12Project environment (i.e. developing embedded Linux images and
13user-space applications that run on targeted devices). This manual groups
14related procedures into higher-level sections. Procedures can consist of
15high-level steps or low-level steps depending on the topic.
16
17This manual provides the following:
18
19- Procedures that help you get going with the Yocto Project; for
20 example, procedures that show you how to set up a build host and work
21 with the Yocto Project source repositories.
22
23- Procedures that show you how to submit changes to the Yocto Project.
24 Changes can be improvements, new features, or bug fixes.
25
26- Procedures related to "everyday" tasks you perform while developing
27 images and applications using the Yocto Project, such as
28 creating a new layer, customizing an image, writing a new recipe,
29 and so forth.
30
31This manual does not provide the following:
32
33- Redundant step-by-step instructions: For example, the
34 :doc:`/sdk-manual/index` manual contains detailed
35 instructions on how to install an SDK, which is used to develop
36 applications for target hardware.
37
38- Reference or conceptual material: This type of material resides in an
39 appropriate reference manual. As an example, system variables are
40 documented in the :doc:`/ref-manual/index`.
41
42- Detailed public information not specific to the Yocto Project: For
43 example, exhaustive information on how to use the Git version
44 control system is better covered with Internet searches and official Git
45 documentation than through the Yocto Project documentation.
46
47Other Information
48=================
49
50Because this manual presents information for many different topics,
51supplemental information is recommended for full comprehension. For
52introductory information on the Yocto Project, see the
53:yocto_home:`Yocto Project Website <>`. If you want to build an image with no
54knowledge of Yocto Project as a way of quickly testing it out, see the
55:doc:`/brief-yoctoprojectqs/index` document.
56
57For a comprehensive list of links and other documentation, see the
58":ref:`ref-manual/resources:links and related documentation`"
59section in the Yocto Project Reference Manual.
diff --git a/documentation/dev-manual/layers.rst b/documentation/dev-manual/layers.rst
deleted file mode 100644
index cc7103ba61..0000000000
--- a/documentation/dev-manual/layers.rst
+++ /dev/null
@@ -1,1050 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Understanding and Creating Layers
4*********************************
5
6The OpenEmbedded build system supports organizing
7:term:`Metadata` into multiple layers.
8Layers allow you to isolate different types of customizations from each
9other. For introductory information on the Yocto Project Layer Model,
10see the
11":ref:`overview-manual/yp-intro:the yocto project layer model`"
12section in the Yocto Project Overview and Concepts Manual.
13
14Creating Your Own Layer
15=======================
16
17.. note::
18
19 It is very easy to create your own layers to use with the OpenEmbedded
20 build system, as the Yocto Project ships with tools that speed up creating
21 layers. This section describes the steps you perform by hand to create
22 layers so that you can better understand them. For information about the
23 layer-creation tools, see the
24 ":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
25 section in the Yocto Project Board Support Package (BSP) Developer's
26 Guide and the ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`"
27 section further down in this manual.
28
29Follow these general steps to create your layer without using tools:
30
31#. *Check Existing Layers:* Before creating a new layer, you should be
32 sure someone has not already created a layer containing the Metadata
33 you need. You can see the :oe_layerindex:`OpenEmbedded Metadata Index <>`
34 for a list of layers from the OpenEmbedded community that can be used in
35 the Yocto Project. You could find a layer that is identical or close
36 to what you need.
37
38#. *Create a Directory:* Create the directory for your layer. When you
39 create the layer, be sure to create the directory in an area not
40 associated with the Yocto Project :term:`Source Directory`
41 (e.g. the cloned ``poky`` repository).
42
43 While not strictly required, prepend the name of the directory with
44 the string "meta-". For example::
45
46 meta-mylayer
47 meta-GUI_xyz
48 meta-mymachine
49
50 With rare exceptions, a layer's name follows this form::
51
52 meta-root_name
53
54 Following this layer naming convention can save
55 you trouble later when tools, components, or variables "assume" your
56 layer name begins with "meta-". A notable example is in configuration
57 files as shown in the following step where layer names without the
58 "meta-" string are appended to several variables used in the
59 configuration.
60
61#. *Create a Layer Configuration File:* Inside your new layer folder,
62 you need to create a ``conf/layer.conf`` file. It is easiest to take
63 an existing layer configuration file and copy that to your layer's
64 ``conf`` directory and then modify the file as needed.
65
66 The ``meta-yocto-bsp/conf/layer.conf`` file in the Yocto Project
67 :yocto_git:`Source Repositories </poky/tree/meta-yocto-bsp/conf>`
68 demonstrates the required syntax. For your layer, you need to replace
69 "yoctobsp" with a unique identifier for your layer (e.g. "machinexyz"
70 for a layer named "meta-machinexyz")::
71
72 # We have a conf and classes directory, add to BBPATH
73 BBPATH .= ":${LAYERDIR}"
74
75 # We have recipes-* directories, add to BBFILES
76 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
77 ${LAYERDIR}/recipes-*/*/*.bbappend"
78
79 BBFILE_COLLECTIONS += "yoctobsp"
80 BBFILE_PATTERN_yoctobsp = "^${LAYERDIR}/"
81 BBFILE_PRIORITY_yoctobsp = "5"
82 LAYERVERSION_yoctobsp = "4"
83 LAYERSERIES_COMPAT_yoctobsp = "walnascar"
84
85 Here is an explanation of the layer configuration file:
86
87 - :term:`BBPATH`: Adds the layer's
88 root directory to BitBake's search path. Through the use of the
89 :term:`BBPATH` variable, BitBake locates class files (``.bbclass``),
90 configuration files, and files that are included with ``include``
91 and ``require`` statements. For these cases, BitBake uses the
92 first file that matches the name found in :term:`BBPATH`. This is
93 similar to the way the ``PATH`` variable is used for binaries. It
94 is recommended, therefore, that you use unique class and
95 configuration filenames in your custom layer.
96
97 - :term:`BBFILES`: Defines the
98 location for all recipes in the layer.
99
100 - :term:`BBFILE_COLLECTIONS`:
101 Establishes the current layer through a unique identifier that is
102 used throughout the OpenEmbedded build system to refer to the
103 layer. In this example, the identifier "yoctobsp" is the
104 representation for the container layer named "meta-yocto-bsp".
105
106 - :term:`BBFILE_PATTERN`:
107 Expands immediately during parsing to provide the directory of the
108 layer.
109
110 - :term:`BBFILE_PRIORITY`:
111 Establishes a priority to use for recipes in the layer when the
112 OpenEmbedded build finds recipes of the same name in different
113 layers.
114
115 - :term:`LAYERVERSION`:
116 Establishes a version number for the layer. You can use this
117 version number to specify this exact version of the layer as a
118 dependency when using the
119 :term:`LAYERDEPENDS`
120 variable.
121
122 - :term:`LAYERDEPENDS`:
123 Lists all layers on which this layer depends (if any).
124
125 - :term:`LAYERSERIES_COMPAT`:
126 Lists the :yocto_wiki:`Yocto Project </Releases>`
127 releases for which the current version is compatible. This
128 variable is a good way to indicate if your particular layer is
129 current.
130
131
132 .. note::
133
134 A layer does not have to contain only recipes ``.bb`` or append files
135 ``.bbappend``. Generally, developers create layers using
136 ``bitbake-layers create-layer``.
137 See ":ref:`dev-manual/layers:creating a general layer using the \`\`bitbake-layers\`\` script`",
138 explaining how the ``layer.conf`` file is created from a template located in
139 ``meta/lib/bblayers/templates/layer.conf``.
140 In fact, none of the variables set in ``layer.conf`` are mandatory,
141 except when :term:`BBFILE_COLLECTIONS` is present. In this case
142 :term:`LAYERSERIES_COMPAT` and :term:`BBFILE_PATTERN` have to be
143 defined too.
144
145#. *Add Content:* Depending on the type of layer, add the content. If
146 the layer adds support for a machine, add the machine configuration
147 in a ``conf/machine/`` file within the layer. If the layer adds
148 distro policy, add the distro configuration in a ``conf/distro/``
149 file within the layer. If the layer introduces new recipes, put the
150 recipes you need in ``recipes-*`` subdirectories within the layer.
151
152 .. note::
153
154 For an explanation of layer hierarchy that is compliant with the
155 Yocto Project, see the ":ref:`bsp-guide/bsp:example filesystem layout`"
156 section in the Yocto Project Board Support Package (BSP) Developer's Guide.
157
158#. *Optionally Test for Compatibility:* If you want permission to use
159 the Yocto Project Compatibility logo with your layer or application
160 that uses your layer, perform the steps to apply for compatibility.
161 See the
162 ":ref:`dev-manual/layers:making sure your layer is compatible with yocto project`"
163 section for more information.
164
165Following Best Practices When Creating Layers
166=============================================
167
168To create layers that are easier to maintain and that will not impact
169builds for other machines, you should consider the information in the
170following list:
171
172- *Avoid "Overlaying" Entire Recipes from Other Layers in Your
173 Configuration:* In other words, do not copy an entire recipe into
174 your layer and then modify it. Rather, use an append file
175 (``.bbappend``) to override only those parts of the original recipe
176 you need to modify.
177
178- *Avoid Duplicating Include Files:* Use append files (``.bbappend``)
179 for each recipe that uses an include file. Or, if you are introducing
180 a new recipe that requires the included file, use the path relative
181 to the original layer directory to refer to the file. For example,
182 use ``require recipes-core/``\ `package`\ ``/``\ `file`\ ``.inc`` instead
183 of ``require`` `file`\ ``.inc``. If you're finding you have to overlay
184 the include file, it could indicate a deficiency in the include file
185 in the layer to which it originally belongs. If this is the case, you
186 should try to address that deficiency instead of overlaying the
187 include file. For example, you could address this by getting the
188 maintainer of the include file to add a variable or variables to make
189 it easy to override the parts needing to be overridden.
190
191- *Structure Your Layers:* Proper use of overrides within append files
192 and placement of machine-specific files within your layer can ensure
193 that a build is not using the wrong Metadata and negatively impacting
194 a build for a different machine. Here are some examples:
195
196 - *Modify Variables to Support a Different Machine:* Suppose you
197 have a layer named ``meta-one`` that adds support for building
198 machine "one". To do so, you use an append file named
199 ``base-files.bbappend`` and create a dependency on "foo" by
200 altering the :term:`DEPENDS`
201 variable::
202
203 DEPENDS = "foo"
204
205 The dependency is created during any
206 build that includes the layer ``meta-one``. However, you might not
207 want this dependency for all machines. For example, suppose you
208 are building for machine "two" but your ``bblayers.conf`` file has
209 the ``meta-one`` layer included. During the build, the
210 ``base-files`` for machine "two" will also have the dependency on
211 ``foo``.
212
213 To make sure your changes apply only when building machine "one",
214 use a machine override with the :term:`DEPENDS` statement::
215
216 DEPENDS:one = "foo"
217
218 You should follow the same strategy when using ``:append``
219 and ``:prepend`` operations::
220
221 DEPENDS:append:one = " foo"
222 DEPENDS:prepend:one = "foo "
223
224 As an actual example, here's a
225 snippet from the generic kernel include file ``linux-yocto.inc``,
226 wherein the kernel compile and link options are adjusted in the
227 case of a subset of the supported architectures::
228
229 DEPENDS:append:aarch64 = " libgcc"
230 KERNEL_CC:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
231 KERNEL_LD:append:aarch64 = " ${TOOLCHAIN_OPTIONS}"
232
233 DEPENDS:append:nios2 = " libgcc"
234 KERNEL_CC:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
235 KERNEL_LD:append:nios2 = " ${TOOLCHAIN_OPTIONS}"
236
237 DEPENDS:append:arc = " libgcc"
238 KERNEL_CC:append:arc = " ${TOOLCHAIN_OPTIONS}"
239 KERNEL_LD:append:arc = " ${TOOLCHAIN_OPTIONS}"
240
241 KERNEL_FEATURES:append:qemuall=" features/debug/printk.scc"
242
243 - *Place Machine-Specific Files in Machine-Specific Locations:* When
244 you have a base recipe, such as ``base-files.bb``, that contains a
245 :term:`SRC_URI` statement to a
246 file, you can use an append file to cause the build to use your
247 own version of the file. For example, an append file in your layer
248 at ``meta-one/recipes-core/base-files/base-files.bbappend`` could
249 extend :term:`FILESPATH` using :term:`FILESEXTRAPATHS` as follows::
250
251 FILESEXTRAPATHS:prepend := "${THISDIR}/${BPN}:"
252
253 The build for machine "one" will pick up your machine-specific file as
254 long as you have the file in
255 ``meta-one/recipes-core/base-files/base-files/``. However, if you
256 are building for a different machine and the ``bblayers.conf``
257 file includes the ``meta-one`` layer and the location of your
258 machine-specific file is the first location where that file is
259 found according to :term:`FILESPATH`, builds for all machines will
260 also use that machine-specific file.
261
262 You can make sure that a machine-specific file is used for a
263 particular machine by putting the file in a subdirectory specific
264 to the machine. For example, rather than placing the file in
265 ``meta-one/recipes-core/base-files/base-files/`` as shown above,
266 put it in ``meta-one/recipes-core/base-files/base-files/one/``.
267 Not only does this make sure the file is used only when building
268 for machine "one", but the build process locates the file more
269 quickly.
270
271 In summary, you need to place all files referenced from
272 :term:`SRC_URI` in a machine-specific subdirectory within the layer in
273 order to restrict those files to machine-specific builds.
274
275- *Perform Steps to Apply for Yocto Project Compatibility:* If you want
276 permission to use the Yocto Project Compatibility logo with your
277 layer or application that uses your layer, perform the steps to apply
278 for compatibility. See the
279 ":ref:`dev-manual/layers:making sure your layer is compatible with yocto project`"
280 section for more information.
281
282- *Follow the Layer Naming Convention:* Store custom layers in a Git
283 repository that use the ``meta-layer_name`` format.
284
285- *Group Your Layers Locally:* Clone your repository alongside other
286 cloned ``meta`` directories from the :term:`Source Directory`.
287
288Making Sure Your Layer is Compatible With Yocto Project
289=======================================================
290
291When you create a layer used with the Yocto Project, it is advantageous
292to make sure that the layer interacts well with existing Yocto Project
293layers (i.e. the layer is compatible with the Yocto Project). Ensuring
294compatibility makes the layer easy to be consumed by others in the Yocto
295Project community and could allow you permission to use the Yocto
296Project Compatible Logo.
297
298.. note::
299
300 Only Yocto Project member organizations are permitted to use the
301 Yocto Project Compatible Logo. The logo is not available for general
302 use. For information on how to become a Yocto Project member
303 organization, see the :yocto_home:`Yocto Project Website <>`.
304
305The Yocto Project Compatibility Program consists of a layer application
306process that requests permission to use the Yocto Project Compatibility
307Logo for your layer and application. The process consists of two parts:
308
309#. Successfully passing a script (``yocto-check-layer``) that, when run
310 against your layer, tests it against constraints based on experiences
311 of how layers have worked in the real world and where pitfalls have
312 been found. Getting a "PASS" result from the script is required for
313 successful compatibility registration.
314
315#. Completion of an application acceptance form, which you can find at
316 :yocto_home:`/compatible-registration/`.
317
318To be granted permission to use the logo, you need to satisfy the
319following:
320
321- Be able to check the box indicating that you got a "PASS" when
322 running the script against your layer.
323
324- Answer "Yes" to the questions on the form or have an acceptable
325 explanation for any questions answered "No".
326
327- Be a Yocto Project Member Organization.
328
329The remainder of this section presents information on the registration
330form and on the ``yocto-check-layer`` script.
331
332Yocto Project Compatible Program Application
333--------------------------------------------
334
335Use the form to apply for your layer's approval. Upon successful
336application, you can use the Yocto Project Compatibility Logo with your
337layer and the application that uses your layer.
338
339To access the form, use this link:
340:yocto_home:`/compatible-registration`.
341Follow the instructions on the form to complete your application.
342
343The application consists of the following sections:
344
345- *Contact Information:* Provide your contact information as the fields
346 require. Along with your information, provide the released versions
347 of the Yocto Project for which your layer is compatible.
348
349- *Acceptance Criteria:* Provide "Yes" or "No" answers for each of the
350 items in the checklist. There is space at the bottom of the form for
351 any explanations for items for which you answered "No".
352
353- *Recommendations:* Provide answers for the questions regarding Linux
354 kernel use and build success.
355
356``yocto-check-layer`` Script
357----------------------------
358
359The ``yocto-check-layer`` script provides you a way to assess how
360compatible your layer is with the Yocto Project. You should run this
361script prior to using the form to apply for compatibility as described
362in the previous section. You need to achieve a "PASS" result in order to
363have your application form successfully processed.
364
365The script divides tests into three areas: COMMON, BSP, and DISTRO. For
366example, given a distribution layer (DISTRO), the layer must pass both
367the COMMON and DISTRO related tests. Furthermore, if your layer is a BSP
368layer, the layer must pass the COMMON and BSP set of tests.
369
370To execute the script, enter the following commands from your build
371directory::
372
373 $ source oe-init-build-env
374 $ yocto-check-layer your_layer_directory
375
376Be sure to provide the actual directory for your
377layer as part of the command.
378
379You should pass the dependencies of your layer using the ``--dependency``
380argument::
381
382 $ source oe-init-build-env
383 $ yocto-check-layer --dependency <layer1_directory> <layer2_directory> ... -- your_layer_directory
384
385For :term:`BSP <Board Support Package (BSP)>` layers, the ``--machines``
386argument should be passed as part of the command::
387
388 $ source oe-init-build-env
389 $ yocto-check-layer --machines <machine1> <machine2> ... -- your_layer_directory
390
391These machines are the ones present in your BSP layer, in the ``conf/machine/``
392directory.
393
394Entering the command causes the script to determine the type of layer
395and then to execute a set of specific tests against the layer.
396
397The following list overviews the test:
398
399- ``common.test_readme``: Tests if a ``README`` file exists in the
400 layer and the file is not empty.
401
402- ``common.test_security``: Tests that the layer has a ``SECURITY.md``
403 (or similar) file, either in the layer itself or at the top of the containing
404 git repository.
405
406- ``common.test_parse``: Tests to make sure that BitBake can parse the
407 files without error (i.e. ``bitbake -p``).
408
409- ``common.test_show_environment``: Tests that the global or per-recipe
410 environment is in order without errors (i.e. ``bitbake -e``).
411
412- ``common.test_world``: Verifies that ``bitbake world`` works.
413
414- ``common.test_world_inherit_class``: Verifies that ``bitbake world`` works
415 when the :ref:`ref-classes-yocto-check-layer` class is inherited.
416
417- ``common.test_patches_upstream_status``: Verifies that all the patch files
418 included in the layer contain a
419 :ref:`contributor-guide/recipe-style-guide:Patch Upstream Status`.
420
421- ``common.test_signatures``: Tests to be sure that BSP and DISTRO
422 layers do not come with recipes that change signatures.
423
424- ``common.test_layerseries_compat``: Verifies layer compatibility is
425 set properly.
426
427- ``bsp.test_bsp_defines_machines``: Tests if a BSP layer has machine
428 configurations.
429
430- ``bsp.test_bsp_no_set_machine``: Tests to ensure a BSP layer does not
431 set the machine when the layer is added.
432
433- ``bsp.test_machine_world``: Verifies that ``bitbake world`` works
434 regardless of which machine is selected.
435
436- ``bsp.test_machine_signatures``: Verifies that building for a
437 particular machine affects only the signature of tasks specific to
438 that machine.
439
440- ``distro.test_distro_defines_distros``: Tests if a DISTRO layer has
441 distro configurations.
442
443- ``distro.test_distro_no_set_distros``: Tests to ensure a DISTRO layer
444 does not set the distribution when the layer is added.
445
446For a complete list of tests, see the :oe_git:`scripts/lib/checklayer/cases
447</openembedded-core/tree/scripts/lib/checklayer/cases>` directory in
448:term:`OpenEmbedded-Core (OE-Core)`.
449
450Enabling Your Layer
451===================
452
453Before the OpenEmbedded build system can use your new layer, you need to
454enable it. To enable your layer, simply add your layer's path to the
455:term:`BBLAYERS` variable in your ``conf/bblayers.conf`` file, which is
456found in the :term:`Build Directory`. The following example shows how to
457enable your new ``meta-mylayer`` layer (note how your new layer exists
458outside of the official ``poky`` repository which you would have checked
459out earlier)::
460
461 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
462 # changes incompatibly
463 POKY_BBLAYERS_CONF_VERSION = "2"
464 BBPATH = "${TOPDIR}"
465 BBFILES ?= ""
466 BBLAYERS ?= " \
467 /home/user/poky/meta \
468 /home/user/poky/meta-poky \
469 /home/user/poky/meta-yocto-bsp \
470 /home/user/mystuff/meta-mylayer \
471 "
472
473BitBake parses each ``conf/layer.conf`` file from the top down as
474specified in the :term:`BBLAYERS` variable within the ``conf/bblayers.conf``
475file. During the processing of each ``conf/layer.conf`` file, BitBake
476adds the recipes, classes and configurations contained within the
477particular layer to the source directory.
478
479Appending Other Layers Metadata With Your Layer
480===============================================
481
482A recipe that appends Metadata to another recipe is called a BitBake
483append file. A BitBake append file uses the ``.bbappend`` file type
484suffix, while the corresponding recipe to which Metadata is being
485appended uses the ``.bb`` file type suffix.
486
487You can use a ``.bbappend`` file in your layer to make additions or
488changes to the content of another layer's recipe without having to copy
489the other layer's recipe into your layer. Your ``.bbappend`` file
490resides in your layer, while the main ``.bb`` recipe file to which you
491are appending Metadata resides in a different layer.
492
493Being able to append information to an existing recipe not only avoids
494duplication, but also automatically applies recipe changes from a
495different layer into your layer. If you were copying recipes, you would
496have to manually merge changes as they occur.
497
498When you create an append file, you must use the same root name as the
499corresponding recipe file. For example, the append file
500``someapp_3.1.bbappend`` must apply to ``someapp_3.1.bb``. This
501means the original recipe and append filenames are version
502number-specific. If the corresponding recipe is renamed to update to a
503newer version, you must also rename and possibly update the
504corresponding ``.bbappend`` as well.
505
506During the build process, BitBake displays an error on startup if it detects a
507``.bbappend`` file that does not have a corresponding recipe with a matching
508name. To handle these errors, the best practice is to rename the ``.bbappend``
509to match the original recipe version. This also gives you the opportunity to see
510if the ``.bbappend`` is still relevant for the new version of the recipe.
511
512Another method is to use the character ``%`` in the ``.bbappend`` filename. For
513example, to append information to every ``6.*`` minor versions of the recipe
514``someapp``, the ``someapp_6.%.bbappend`` file can be created. This way, an
515error will only be triggered if the ``someapp`` recipe has a major version
516update.
517
518Finally, another method to deal with these errors is to use the variable
519:term:`BBMASK`, especially in cases where modifying the ``.bbappend`` is not
520possible.
521
522Overlaying a File Using Your Layer
523----------------------------------
524
525As an example, consider the main formfactor recipe and a corresponding
526formfactor append file both from the :term:`Source Directory`.
527Here is the main
528formfactor recipe, which is named ``formfactor_0.0.bb`` and located in
529the "meta" layer at ``meta/recipes-bsp/formfactor``::
530
531 SUMMARY = "Device formfactor information"
532 DESCRIPTION = "A formfactor configuration file provides information about the \
533 target hardware for which the image is being built and information that the \
534 build system cannot obtain from other sources such as the kernel."
535 SECTION = "base"
536 LICENSE = "MIT"
537 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
538
539 SRC_URI = "file://config file://machconfig"
540 S = "${UNPACKDIR}"
541
542 PACKAGE_ARCH = "${MACHINE_ARCH}"
543 INHIBIT_DEFAULT_DEPS = "1"
544
545 do_install() {
546 # Install file only if it has contents
547 install -d ${D}${sysconfdir}/formfactor/
548 install -m 0644 ${S}/config ${D}${sysconfdir}/formfactor/
549 if [ -s "${S}/machconfig" ]; then
550 install -m 0644 ${S}/machconfig ${D}${sysconfdir}/formfactor/
551 fi
552 }
553
554In the main recipe, note the :term:`SRC_URI`
555variable, which tells the OpenEmbedded build system where to find files
556during the build.
557
558Here is the append file, which is named ``formfactor_0.0.bbappend``
559and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
560file is in the layer at ``recipes-bsp/formfactor``::
561
562 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
563
564By default, the build system uses the
565:term:`FILESPATH` variable to
566locate files. This append file extends the locations by setting the
567:term:`FILESEXTRAPATHS`
568variable. Setting this variable in the ``.bbappend`` file is the most
569reliable and recommended method for adding directories to the search
570path used by the build system to find files.
571
572The statement in this example extends the directories to include
573``${``\ :term:`THISDIR`\ ``}/${``\ :term:`PN`\ ``}``,
574which resolves to a directory named ``formfactor`` in the same directory
575in which the append file resides (i.e.
576``meta-raspberrypi/recipes-bsp/formfactor``. This implies that you must
577have the supporting directory structure set up that will contain any
578files or patches you will be including from the layer.
579
580Using the immediate expansion assignment operator ``:=`` is important
581because of the reference to :term:`THISDIR`. The trailing colon character is
582important as it ensures that items in the list remain colon-separated.
583
584.. note::
585
586 BitBake automatically defines the :term:`THISDIR` variable. You should
587 never set this variable yourself. Using ":prepend" as part of the
588 :term:`FILESEXTRAPATHS` ensures your path will be searched prior to other
589 paths in the final list.
590
591 Also, not all append files add extra files. Many append files simply
592 allow to add build options (e.g. ``systemd``). For these cases, your
593 append file would not even use the :term:`FILESEXTRAPATHS` statement.
594
595The end result of this ``.bbappend`` file is that on a Raspberry Pi, where
596``rpi`` will exist in the list of :term:`OVERRIDES`, the file
597``meta-raspberrypi/recipes-bsp/formfactor/formfactor/rpi/machconfig`` will be
598used during :ref:`ref-tasks-fetch` and the test for a non-zero file size in
599:ref:`ref-tasks-install` will return true, and the file will be installed.
600
601Installing Additional Files Using Your Layer
602--------------------------------------------
603
604As another example, consider the main ``xserver-xf86-config`` recipe and a
605corresponding ``xserver-xf86-config`` append file both from the :term:`Source
606Directory`. Here is the main ``xserver-xf86-config`` recipe, which is named
607``xserver-xf86-config_0.1.bb`` and located in the "meta" layer at
608``meta/recipes-graphics/xorg-xserver``::
609
610 SUMMARY = "X.Org X server configuration file"
611 HOMEPAGE = "http://www.x.org"
612 SECTION = "x11/base"
613 LICENSE = "MIT"
614 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
615
616 SRC_URI = "file://xorg.conf"
617
618 S = "${UNPACKDIR}"
619
620 CONFFILES:${PN} = "${sysconfdir}/X11/xorg.conf"
621
622 PACKAGE_ARCH = "${MACHINE_ARCH}"
623 ALLOW_EMPTY:${PN} = "1"
624
625 do_install () {
626 if test -s ${UNPACKDIR}/xorg.conf; then
627 install -d ${D}/${sysconfdir}/X11
628 install -m 0644 ${UNPACKDIR}/xorg.conf ${D}/${sysconfdir}/X11/
629 fi
630 }
631
632Here is the append file, which is named ``xserver-xf86-config_%.bbappend``
633and is from the Raspberry Pi BSP Layer named ``meta-raspberrypi``. The
634file is in the layer at ``recipes-graphics/xorg-xserver``::
635
636 FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"
637
638 SRC_URI:append:rpi = " \
639 file://xorg.conf.d/98-pitft.conf \
640 file://xorg.conf.d/99-calibration.conf \
641 "
642 do_install:append:rpi () {
643 PITFT = "${@bb.utils.contains("MACHINE_FEATURES", "pitft", "1", "0", d)}"
644 if [ "${PITFT}" = "1" ]; then
645 install -d ${D}/${sysconfdir}/X11/xorg.conf.d/
646 install -m 0644 ${UNPACKDIR}/xorg.conf.d/98-pitft.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
647 install -m 0644 ${UNPACKDIR}/xorg.conf.d/99-calibration.conf ${D}/${sysconfdir}/X11/xorg.conf.d/
648 fi
649 }
650
651 FILES:${PN}:append:rpi = " ${sysconfdir}/X11/xorg.conf.d/*"
652
653Building off of the previous example, we once again are setting the
654:term:`FILESEXTRAPATHS` variable. In this case we are also using
655:term:`SRC_URI` to list additional source files to use when ``rpi`` is found in
656the list of :term:`OVERRIDES`. The :ref:`ref-tasks-install` task will then perform a
657check for an additional :term:`MACHINE_FEATURES` that if set will cause these
658additional files to be installed. These additional files are listed in
659:term:`FILES` so that they will be packaged.
660
661Prioritizing Your Layer
662=======================
663
664Each layer is assigned a priority value. Priority values control which
665layer takes precedence if there are recipe files with the same name in
666multiple layers. For these cases, the recipe file from the layer with a
667higher priority number takes precedence. Priority values also affect the
668order in which multiple ``.bbappend`` files for the same recipe are
669applied. You can either specify the priority manually, or allow the
670build system to calculate it based on the layer's dependencies.
671
672To specify the layer's priority manually, use the
673:term:`BBFILE_PRIORITY`
674variable and append the layer's root name::
675
676 BBFILE_PRIORITY_mylayer = "1"
677
678.. note::
679
680 It is possible for a recipe with a lower version number
681 :term:`PV` in a layer that has a higher
682 priority to take precedence.
683
684 Also, the layer priority does not currently affect the precedence
685 order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
686 might address this.
687
688Providing Global-level Configurations With Your Layer
689-----------------------------------------------------
690
691When creating a layer, you may need to define configurations that should take
692effect globally in your build environment when the layer is part of the build.
693The ``layer.conf`` file is a :term:`configuration file` that affects the build
694system globally, so it is a candidate for this use-case.
695
696.. warning::
697
698 Providing unconditional global level configuration from the ``layer.conf``
699 file is *not* a good practice, and should be avoided. For this reason, the
700 section :ref:`ref-conditional-layer-confs` below shows how the ``layer.conf``
701 file can be used to provide configurations only if a certain condition is
702 met.
703
704For example, if your layer provides a Linux kernel recipe named
705``linux-custom``, you may want to make :term:`PREFERRED_PROVIDER_virtual/kernel
706<PREFERRED_PROVIDER>` point to ``linux-custom``::
707
708 PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
709
710This can be defined in the ``layer.conf`` file. If your layer is at the last
711position in the :term:`BBLAYERS` list, it will take precedence over previous
712``PREFERRED_PROVIDER_virtual/kernel`` assignments (unless one is set from a
713:term:`configuration file` that is parsed later, such as machine or distro
714configuration files).
715
716.. _ref-conditional-layer-confs:
717
718Conditionally Provide Global-level Configurations With Your Layer
719^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
720
721In some cases, your layer may provide global configurations only if some
722features it provides are enabled. Since the ``layer.conf`` file is parsed at an
723earlier stage in the parsing process, the :term:`DISTRO_FEATURES` and
724:term:`MACHINE_FEATURES` variables are not yet available to ``layer.conf``, and
725declaring conditional assignments based on these variables is not possible. The
726following technique shows a way to bypass this limitation by using the
727:term:`USER_CLASSES` variable and a conditional ``require`` command.
728
729In the following steps, let's assume our layer is named ``meta-mylayer`` and
730that this layer defines a custom :ref:`distro feature <ref-features-distro>`
731named ``mylayer-kernel``. We will set the :term:`PREFERRED_PROVIDER` variable
732for the kernel only if our feature ``mylayer-kernel`` is part of the
733:term:`DISTRO_FEATURES`:
734
735#. Create an include file in the directory
736 ``meta-mylayer/conf/distro/include/``, for example a file named
737 ``mylayer-kernel-provider.inc`` that sets the kernel provider to
738 ``linux-custom``::
739
740 PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
741
742#. Provide a path to this include file in your ``layer.conf``::
743
744 META_MYLAYER_KERNEL_PROVIDER_PATH = "${LAYERDIR}/conf/distro/include/mylayer-kernel-provider.inc"
745
746#. Create a new class in ``meta-mylayer/classes-global/``, for example a class
747 ``meta-mylayer-cfg.bbclass``. Make it conditionally require the file
748 ``mylayer-kernel-provider.inc`` defined above, using the variable
749 ``META_MYLAYER_KERNEL_PROVIDER_PATH`` defined in ``layer.conf``::
750
751 require ${@bb.utils.contains('DISTRO_FEATURES', 'mylayer-kernel', '${META_MYLAYER_KERNEL_PROVIDER_PATH}', '', d)}
752
753 For details on the ``bb.utils.contains`` function, see its definition in
754 :bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`.
755
756 .. note::
757
758 The ``require`` command is designed to not fail if the function
759 ``bb.utils.contains`` returns an empty string.
760
761#. Back to your ``layer.conf`` file, add the class ``meta-mylayer-cfg`` class to
762 the :term:`USER_CLASSES` variable::
763
764 USER_CLASSES:append = " meta-mylayer-cfg"
765
766 This will add the class ``meta-mylayer-cfg`` to the list of classes to
767 globally inherit. Since the ``require`` command is conditional in
768 ``meta-mylayer-cfg.bbclass``, even though inherited the class will have no
769 effect unless the feature ``mylayer-kernel`` is enabled through
770 :term:`DISTRO_FEATURES`.
771
772This technique can also be used for :ref:`Machine features
773<ref-features-machine>` by following the same steps. Though not mandatory, it is
774recommended to put include files for :term:`DISTRO_FEATURES` in your layer's
775``conf/distro/include`` and the ones for :term:`MACHINE_FEATURES` in your
776layer's ``conf/machine/include``.
777
778Managing Layers
779===============
780
781You can use the BitBake layer management tool ``bitbake-layers`` to
782provide a view into the structure of recipes across a multi-layer
783project. Being able to generate output that reports on configured layers
784with their paths and priorities and on ``.bbappend`` files and their
785applicable recipes can help to reveal potential problems.
786
787For help on the BitBake layer management tool, use the following
788command::
789
790 $ bitbake-layers --help
791
792The following list describes the available commands:
793
794- ``help:`` Displays general help or help on a specified command.
795
796- ``show-layers:`` Shows the current configured layers.
797
798- ``show-overlayed:`` Lists overlayed recipes. A recipe is overlayed
799 when a recipe with the same name exists in another layer that has a
800 higher layer priority.
801
802- ``show-recipes:`` Lists available recipes and the layers that
803 provide them.
804
805- ``show-appends:`` Lists ``.bbappend`` files and the recipe files to
806 which they apply.
807
808- ``show-cross-depends:`` Lists dependency relationships between
809 recipes that cross layer boundaries.
810
811- ``add-layer:`` Adds a layer to ``bblayers.conf``.
812
813- ``remove-layer:`` Removes a layer from ``bblayers.conf``
814
815- ``flatten:`` Flattens the layer configuration into a separate
816 output directory. Flattening your layer configuration builds a
817 "flattened" directory that contains the contents of all layers, with
818 any overlayed recipes removed and any ``.bbappend`` files appended to
819 the corresponding recipes. You might have to perform some manual
820 cleanup of the flattened layer as follows:
821
822 - Non-recipe files (such as patches) are overwritten. The flatten
823 command shows a warning for these files.
824
825 - Anything beyond the normal layer setup has been added to the
826 ``layer.conf`` file. Only the lowest priority layer's
827 ``layer.conf`` is used.
828
829 - Overridden and appended items from ``.bbappend`` files need to be
830 cleaned up. The contents of each ``.bbappend`` end up in the
831 flattened recipe. However, if there are appended or changed
832 variable values, you need to tidy these up yourself. Consider the
833 following example. Here, the ``bitbake-layers`` command adds the
834 line ``#### bbappended ...`` so that you know where the following
835 lines originate::
836
837 ...
838 DESCRIPTION = "A useful utility"
839 ...
840 EXTRA_OECONF = "--enable-something"
841 ...
842
843 #### bbappended from meta-anotherlayer ####
844
845 DESCRIPTION = "Customized utility"
846 EXTRA_OECONF += "--enable-somethingelse"
847
848
849 Ideally, you would tidy up these utilities as follows::
850
851 ...
852 DESCRIPTION = "Customized utility"
853 ...
854 EXTRA_OECONF = "--enable-something --enable-somethingelse"
855 ...
856
857- ``layerindex-fetch``: Fetches a layer from a layer index, along
858 with its dependent layers, and adds the layers to the
859 ``conf/bblayers.conf`` file.
860
861- ``layerindex-show-depends``: Finds layer dependencies from the
862 layer index.
863
864- ``save-build-conf``: Saves the currently active build configuration
865 (``conf/local.conf``, ``conf/bblayers.conf``) as a template into a layer.
866 This template can later be used for setting up builds via :term:`TEMPLATECONF`.
867 For information about saving and using configuration templates, see
868 ":ref:`dev-manual/custom-template-configuration-directory:creating a custom template configuration directory`".
869
870- ``create-layer``: Creates a basic layer.
871
872- ``create-layers-setup``: Writes out a configuration file and/or a script that
873 can replicate the directory structure and revisions of the layers in a current build.
874 For more information, see ":ref:`dev-manual/layers:saving and restoring the layers setup`".
875
876Creating a General Layer Using the ``bitbake-layers`` Script
877============================================================
878
879The ``bitbake-layers`` script with the ``create-layer`` subcommand
880simplifies creating a new general layer.
881
882.. note::
883
884 - For information on BSP layers, see the ":ref:`bsp-guide/bsp:bsp layers`"
885 section in the Yocto
886 Project Board Specific (BSP) Developer's Guide.
887
888 - In order to use a layer with the OpenEmbedded build system, you
889 need to add the layer to your ``bblayers.conf`` configuration
890 file. See the ":ref:`dev-manual/layers:adding a layer using the \`\`bitbake-layers\`\` script`"
891 section for more information.
892
893The default mode of the script's operation with this subcommand is to
894create a layer with the following:
895
896- A layer priority of 6.
897
898- A ``conf`` subdirectory that contains a ``layer.conf`` file.
899
900- A ``recipes-example`` subdirectory that contains a further
901 subdirectory named ``example``, which contains an ``example.bb``
902 recipe file.
903
904- A ``COPYING.MIT``, which is the license statement for the layer. The
905 script assumes you want to use the MIT license, which is typical for
906 most layers, for the contents of the layer itself.
907
908- A ``README`` file, which is a file describing the contents of your
909 new layer.
910
911In its simplest form, you can use the following command form to create a
912layer. The command creates a layer whose name corresponds to
913"your_layer_name" in the current directory::
914
915 $ bitbake-layers create-layer your_layer_name
916
917As an example, the following command creates a layer named ``meta-scottrif``
918in your home directory::
919
920 $ cd /usr/home
921 $ bitbake-layers create-layer meta-scottrif
922 NOTE: Starting bitbake server...
923 Add your new layer with 'bitbake-layers add-layer meta-scottrif'
924
925If you want to set the priority of the layer to other than the default
926value of "6", you can either use the ``--priority`` option or you
927can edit the
928:term:`BBFILE_PRIORITY` value
929in the ``conf/layer.conf`` after the script creates it. Furthermore, if
930you want to give the example recipe file some name other than the
931default, you can use the ``--example-recipe-name`` option.
932
933The easiest way to see how the ``bitbake-layers create-layer`` command
934works is to experiment with the script. You can also read the usage
935information by entering the following::
936
937 $ bitbake-layers create-layer --help
938 NOTE: Starting bitbake server...
939 usage: bitbake-layers create-layer [-h] [--priority PRIORITY]
940 [--example-recipe-name EXAMPLERECIPE]
941 layerdir
942
943 Create a basic layer
944
945 positional arguments:
946 layerdir Layer directory to create
947
948 optional arguments:
949 -h, --help show this help message and exit
950 --priority PRIORITY, -p PRIORITY
951 Layer directory to create
952 --example-recipe-name EXAMPLERECIPE, -e EXAMPLERECIPE
953 Filename of the example recipe
954
955Adding a Layer Using the ``bitbake-layers`` Script
956==================================================
957
958Once you create your general layer, you must add it to your
959``bblayers.conf`` file. Adding the layer to this configuration file
960makes the OpenEmbedded build system aware of your layer so that it can
961search it for metadata.
962
963Add your layer by using the ``bitbake-layers add-layer`` command::
964
965 $ bitbake-layers add-layer your_layer_name
966
967Here is an example that adds a
968layer named ``meta-scottrif`` to the configuration file. Following the
969command that adds the layer is another ``bitbake-layers`` command that
970shows the layers that are in your ``bblayers.conf`` file::
971
972 $ bitbake-layers add-layer meta-scottrif
973 NOTE: Starting bitbake server...
974 Parsing recipes: 100% |##########################################################| Time: 0:00:49
975 Parsing of 1441 .bb files complete (0 cached, 1441 parsed). 2055 targets, 56 skipped, 0 masked, 0 errors.
976 $ bitbake-layers show-layers
977 NOTE: Starting bitbake server...
978 layer path priority
979 ==========================================================================
980 meta /home/scottrif/poky/meta 5
981 meta-poky /home/scottrif/poky/meta-poky 5
982 meta-yocto-bsp /home/scottrif/poky/meta-yocto-bsp 5
983 workspace /home/scottrif/poky/build/workspace 99
984 meta-scottrif /home/scottrif/poky/build/meta-scottrif 6
985
986
987Adding the layer to this file
988enables the build system to locate the layer during the build.
989
990.. note::
991
992 During a build, the OpenEmbedded build system looks in the layers
993 from the top of the list down to the bottom in that order.
994
995Saving and restoring the layers setup
996=====================================
997
998Once you have a working build with the correct set of layers, it is beneficial
999to capture the layer setup --- what they are, which repositories they come from
1000and which SCM revisions they're at --- into a configuration file, so that this
1001setup can be easily replicated later, perhaps on a different machine. Here's
1002how to do this::
1003
1004 $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/
1005 NOTE: Starting bitbake server...
1006 NOTE: Created /srv/work/alex/meta-alex/setup-layers.json
1007 NOTE: Created /srv/work/alex/meta-alex/setup-layers
1008
1009The tool needs a single argument which tells where to place the output, consisting
1010of a json formatted layer configuration, and a ``setup-layers`` script that can use that configuration
1011to restore the layers in a different location, or on a different host machine. The argument
1012can point to a custom layer (which is then deemed a "bootstrap" layer that needs to be
1013checked out first), or into a completely independent location.
1014
1015The replication of the layers is performed by running the ``setup-layers`` script provided
1016above:
1017
1018#. Clone the bootstrap layer or some other repository to obtain
1019 the json config and the setup script that can use it.
1020
1021#. Run the script directly with no options::
1022
1023 alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers
1024 Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override.
1025
1026 Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master
1027 Running 'git init -q /srv/work/alex/my-build/meta-intel'
1028 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel
1029 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel
1030 Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel
1031
1032 Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers
1033 Running 'git init -q /srv/work/alex/my-build/poky'
1034 Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky
1035 Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky
1036 Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky
1037 Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky
1038 Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky
1039
1040.. note::
1041 This will work to update an existing checkout as well.
1042
1043.. note::
1044 The script is self-sufficient and requires only python3
1045 and git on the build machine.
1046
1047.. note::
1048 Both the ``create-layers-setup`` and the ``setup-layers`` provided several additional options
1049 that customize their behavior - you are welcome to study them via ``--help`` command line parameter.
1050
diff --git a/documentation/dev-manual/libraries.rst b/documentation/dev-manual/libraries.rst
deleted file mode 100644
index a8c38128ea..0000000000
--- a/documentation/dev-manual/libraries.rst
+++ /dev/null
@@ -1,237 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working With Libraries
4**********************
5
6Libraries are an integral part of your system. This section describes
7some common practices you might find helpful when working with libraries
8to build your system:
9
10- :ref:`How to include static library files
11 <dev-manual/libraries:including static library files>`
12
13- :ref:`How to use the Multilib feature to combine multiple versions of
14 library files into a single image
15 <dev-manual/libraries:combining multiple versions of library files into one image>`
16
17- :ref:`How to install multiple versions of the same library in parallel on
18 the same system
19 <dev-manual/libraries:installing multiple versions of the same library>`
20
21Including Static Library Files
22==============================
23
24If you are building a library and the library offers static linking, you
25can control which static library files (``*.a`` files) get included in
26the built library.
27
28The :term:`PACKAGES` and
29:term:`FILES:* <FILES>` variables in the
30``meta/conf/bitbake.conf`` configuration file define how files installed
31by the :ref:`ref-tasks-install` task are packaged. By default, the :term:`PACKAGES`
32variable includes ``${PN}-staticdev``, which represents all static
33library files.
34
35.. note::
36
37 Some previously released versions of the Yocto Project defined the
38 static library files through ``${PN}-dev``.
39
40Here is a small part of the BitBake configuration file, where you can see
41how the static library files are defined::
42
43 PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
44
45 FILES:${PN}-staticdev = "${libdir}/*.a ${base_libdir}/*.a ${libdir}/${BPN}/*.a"
46 SECTION:${PN}-staticdev = "devel"
47 RDEPENDS:${PN}-staticdev = "${PN}-dev (= ${EXTENDPKGV})"
48
49Combining Multiple Versions of Library Files into One Image
50===========================================================
51
52The build system offers the ability to build libraries with different
53target optimizations or architecture formats and combine these together
54into one system image. You can link different binaries in the image
55against the different libraries as needed for specific use cases. This
56feature is called "Multilib".
57
58An example would be where you have most of a system compiled in 32-bit
59mode using 32-bit libraries, but you have something large, like a
60database engine, that needs to be a 64-bit application and uses 64-bit
61libraries. Multilib allows you to get the best of both 32-bit and 64-bit
62libraries.
63
64While the Multilib feature is most commonly used for 32 and 64-bit
65differences, the approach the build system uses facilitates different
66target optimizations. You could compile some binaries to use one set of
67libraries and other binaries to use a different set of libraries. The
68libraries could differ in architecture, compiler options, or other
69optimizations.
70
71There are several examples in the ``meta-skeleton`` layer found in the
72:term:`Source Directory`:
73
74- :oe_git:`conf/multilib-example.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example.conf>`
75 configuration file.
76
77- :oe_git:`conf/multilib-example2.conf </openembedded-core/tree/meta-skeleton/conf/multilib-example2.conf>`
78 configuration file.
79
80- :oe_git:`recipes-multilib/images/core-image-multilib-example.bb </openembedded-core/tree/meta-skeleton/recipes-multilib/images/core-image-multilib-example.bb>`
81 recipe
82
83Preparing to Use Multilib
84-------------------------
85
86User-specific requirements drive the Multilib feature. Consequently,
87there is no one "out-of-the-box" configuration that would
88meet your needs.
89
90In order to enable Multilib, you first need to ensure your recipe is
91extended to support multiple libraries. Many standard recipes are
92already extended and support multiple libraries. You can check in the
93``meta/conf/multilib.conf`` configuration file in the
94:term:`Source Directory` to see how this is
95done using the
96:term:`BBCLASSEXTEND` variable.
97Eventually, all recipes will be covered and this list will not be
98needed.
99
100For the most part, the :ref:`Multilib <ref-classes-multilib*>`
101class extension works automatically to
102extend the package name from ``${PN}`` to ``${MLPREFIX}${PN}``, where
103:term:`MLPREFIX` is the particular multilib (e.g. "lib32-" or "lib64-").
104Standard variables such as
105:term:`DEPENDS`,
106:term:`RDEPENDS`,
107:term:`RPROVIDES`,
108:term:`RRECOMMENDS`,
109:term:`PACKAGES`, and
110:term:`PACKAGES_DYNAMIC` are
111automatically extended by the system. If you are extending any manual
112code in the recipe, you can use the ``${MLPREFIX}`` variable to ensure
113those names are extended correctly.
114
115Using Multilib
116--------------
117
118After you have set up the recipes, you need to define the actual
119combination of multiple libraries you want to build. You accomplish this
120through your ``local.conf`` configuration file in the
121:term:`Build Directory`. An example configuration would be as follows::
122
123 MACHINE = "qemux86-64"
124 require conf/multilib.conf
125 MULTILIBS = "multilib:lib32"
126 DEFAULTTUNE:virtclass-multilib-lib32 = "x86"
127 IMAGE_INSTALL:append = " lib32-glib-2.0"
128
129This example enables an additional library named
130``lib32`` alongside the normal target packages. When combining these
131"lib32" alternatives, the example uses "x86" for tuning. For information
132on this particular tuning, see
133``meta/conf/machine/include/ia32/arch-ia32.inc``.
134
135The example then includes ``lib32-glib-2.0`` in all the images, which
136illustrates one method of including a multiple library dependency. You
137can use a normal image build to include this dependency, for example::
138
139 $ bitbake core-image-sato
140
141You can also build Multilib packages
142specifically with a command like this::
143
144 $ bitbake lib32-glib-2.0
145
146Additional Implementation Details
147---------------------------------
148
149There are generic implementation details as well as details that are specific to
150package management systems. Here are implementation details
151that exist regardless of the package management system:
152
153- The typical convention used for the class extension code as used by
154 Multilib assumes that all package names specified in
155 :term:`PACKAGES` that contain
156 ``${PN}`` have ``${PN}`` at the start of the name. When that
157 convention is not followed and ``${PN}`` appears at the middle or the
158 end of a name, problems occur.
159
160- The :term:`TARGET_VENDOR`
161 value under Multilib will be extended to "-vendormlmultilib" (e.g.
162 "-pokymllib32" for a "lib32" Multilib with Poky). The reason for this
163 slightly unwieldy contraction is that any "-" characters in the
164 vendor string presently break Autoconf's ``config.sub``, and other
165 separators are problematic for different reasons.
166
167Here are the implementation details for the RPM Package Management System:
168
169- A unique architecture is defined for the Multilib packages, along
170 with creating a unique deploy folder under ``tmp/deploy/rpm`` in the
171 :term:`Build Directory`. For example, consider ``lib32`` in a
172 ``qemux86-64`` image. The possible architectures in the system are "all",
173 "qemux86_64", "lib32:qemux86_64", and "lib32:x86".
174
175- The ``${MLPREFIX}`` variable is stripped from ``${PN}`` during RPM
176 packaging. The naming for a normal RPM package and a Multilib RPM
177 package in a ``qemux86-64`` system resolves to something similar to
178 ``bash-4.1-r2.x86_64.rpm`` and ``bash-4.1.r2.lib32_x86.rpm``,
179 respectively.
180
181- When installing a Multilib image, the RPM backend first installs the
182 base image and then installs the Multilib libraries.
183
184- The build system relies on RPM to resolve the identical files in the
185 two (or more) Multilib packages.
186
187Here are the implementation details for the IPK Package Management System:
188
189- The ``${MLPREFIX}`` is not stripped from ``${PN}`` during IPK
190 packaging. The naming for a normal RPM package and a Multilib IPK
191 package in a ``qemux86-64`` system resolves to something like
192 ``bash_4.1-r2.x86_64.ipk`` and ``lib32-bash_4.1-rw:x86.ipk``,
193 respectively.
194
195- The IPK deploy folder is not modified with ``${MLPREFIX}`` because
196 packages with and without the Multilib feature can exist in the same
197 folder due to the ``${PN}`` differences.
198
199- IPK defines a sanity check for Multilib installation using certain
200 rules for file comparison, overridden, etc.
201
202Installing Multiple Versions of the Same Library
203================================================
204
205There are be situations where you need to install and use multiple versions
206of the same library on the same system at the same time. This
207almost always happens when a library API changes and you have
208multiple pieces of software that depend on the separate versions of the
209library. To accommodate these situations, you can install multiple
210versions of the same library in parallel on the same system.
211
212The process is straightforward as long as the libraries use proper
213versioning. With properly versioned libraries, all you need to do to
214individually specify the libraries is create separate, appropriately
215named recipes where the :term:`PN` part of
216the name includes a portion that differentiates each library version
217(e.g. the major part of the version number). Thus, instead of having a
218single recipe that loads one version of a library (e.g. ``clutter``),
219you provide multiple recipes that result in different versions of the
220libraries you want. As an example, the following two recipes would allow
221the two separate versions of the ``clutter`` library to co-exist on the
222same system:
223
224.. code-block:: none
225
226 clutter-1.6_1.6.20.bb
227 clutter-1.8_1.8.4.bb
228
229Additionally, if
230you have other recipes that depend on a given library, you need to use
231the :term:`DEPENDS` variable to
232create the dependency. Continuing with the same example, if you want to
233have a recipe depend on the 1.8 version of the ``clutter`` library, use
234the following in your recipe::
235
236 DEPENDS = "clutter-1.8"
237
diff --git a/documentation/dev-manual/licenses.rst b/documentation/dev-manual/licenses.rst
deleted file mode 100644
index 7d6636eeff..0000000000
--- a/documentation/dev-manual/licenses.rst
+++ /dev/null
@@ -1,544 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working With Licenses
4*********************
5
6As mentioned in the ":ref:`overview-manual/development-environment:licensing`"
7section in the Yocto Project Overview and Concepts Manual, open source
8projects are open to the public and they consequently have different
9licensing structures in place. This section describes the mechanism by
10which the :term:`OpenEmbedded Build System`
11tracks changes to
12licensing text and covers how to maintain open source license compliance
13during your project's lifecycle. The section also describes how to
14enable commercially licensed recipes, which by default are disabled.
15
16Tracking License Changes
17========================
18
19The license of an upstream project might change in the future. In order
20to prevent these changes going unnoticed, the
21:term:`LIC_FILES_CHKSUM`
22variable tracks changes to the license text. The checksums are validated
23at the end of the configure step, and if the checksums do not match, the
24build will fail.
25
26Specifying the ``LIC_FILES_CHKSUM`` Variable
27--------------------------------------------
28
29The :term:`LIC_FILES_CHKSUM` variable contains checksums of the license text
30in the source code for the recipe. Here is an example of how to
31specify :term:`LIC_FILES_CHKSUM`::
32
33 LIC_FILES_CHKSUM = "file://COPYING;md5=xxxx \
34 file://licfile1.txt;beginline=5;endline=29;md5=yyyy \
35 file://licfile2.txt;endline=50;md5=zzzz \
36 ..."
37
38.. note::
39
40 - When using "beginline" and "endline", realize that line numbering
41 begins with one and not zero. Also, the included lines are
42 inclusive (i.e. lines five through and including 29 in the
43 previous example for ``licfile1.txt``).
44
45 - When a license check fails, the selected license text is included
46 as part of the QA message. Using this output, you can determine
47 the exact start and finish for the needed license text.
48
49The build system uses the :term:`S`
50variable as the default directory when searching files listed in
51:term:`LIC_FILES_CHKSUM`. The previous example employs the default
52directory.
53
54Consider this next example::
55
56 LIC_FILES_CHKSUM = "file://src/ls.c;beginline=5;endline=16;\
57 md5=bb14ed3c4cda583abc85401304b5cd4e"
58 LIC_FILES_CHKSUM = "file://${UNPACKDIR}/license.html;md5=5c94767cedb5d6987c902ac850ded2c6"
59
60The first line locates a file in ``${S}/src/ls.c`` and isolates lines
61five through 16 as license text. The second line refers to a file in
62:term:`UNPACKDIR`.
63
64Note that :term:`LIC_FILES_CHKSUM` variable is mandatory for all recipes,
65unless the :term:`LICENSE` variable is set to "CLOSED".
66
67Explanation of Syntax
68---------------------
69
70As mentioned in the previous section, the :term:`LIC_FILES_CHKSUM` variable
71lists all the important files that contain the license text for the
72source code. It is possible to specify a checksum for an entire file, or
73a specific section of a file (specified by beginning and ending line
74numbers with the "beginline" and "endline" parameters, respectively).
75The latter is useful for source files with a license notice header,
76README documents, and so forth. If you do not use the "beginline"
77parameter, then it is assumed that the text begins on the first line of
78the file. Similarly, if you do not use the "endline" parameter, it is
79assumed that the license text ends with the last line of the file.
80
81The "md5" parameter stores the md5 checksum of the license text. If the
82license text changes in any way as compared to this parameter then a
83mismatch occurs. This mismatch triggers a build failure and notifies the
84developer. Notification allows the developer to review and address the
85license text changes. Also note that if a mismatch occurs during the
86build, the correct md5 checksum is placed in the build log and can be
87easily copied to the recipe.
88
89There is no limit to how many files you can specify using the
90:term:`LIC_FILES_CHKSUM` variable. Generally, however, every project
91requires a few specifications for license tracking. Many projects have a
92"COPYING" file that stores the license information for all the source
93code files. This practice allows you to just track the "COPYING" file as
94long as it is kept up to date.
95
96.. note::
97
98 - If you specify an empty or invalid "md5" parameter,
99 :term:`BitBake` returns an md5
100 mis-match error and displays the correct "md5" parameter value
101 during the build. The correct parameter is also captured in the
102 build log.
103
104 - If the whole file contains only license text, you do not need to
105 use the "beginline" and "endline" parameters.
106
107Enabling Commercially Licensed Recipes
108======================================
109
110By default, the OpenEmbedded build system disables components that have
111commercial or other special licensing requirements. Such requirements
112are defined on a recipe-by-recipe basis through the
113:term:`LICENSE_FLAGS` variable
114definition in the affected recipe. For instance, the
115``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` recipe
116contains the following statement::
117
118 LICENSE_FLAGS = "commercial"
119
120Here is a
121slightly more complicated example that contains both an explicit recipe
122name and version (after variable expansion)::
123
124 LICENSE_FLAGS = "license_${PN}_${PV}"
125
126It is possible to give more details about a specific license
127using flags on the :term:`LICENSE_FLAGS_DETAILS` variable::
128
129 LICENSE_FLAGS_DETAILS[my-eula-license] = "For further details, see https://example.com/eula."
130
131If set, this will be displayed to the user if the license hasn't been accepted.
132
133In order for a component restricted by a
134:term:`LICENSE_FLAGS` definition to be enabled and included in an image, it
135needs to have a matching entry in the global
136:term:`LICENSE_FLAGS_ACCEPTED`
137variable, which is a variable typically defined in your ``local.conf``
138file. For example, to enable the
139``poky/meta/recipes-multimedia/gstreamer/gst-plugins-ugly`` package, you
140could add either the string "commercial_gst-plugins-ugly" or the more
141general string "commercial" to :term:`LICENSE_FLAGS_ACCEPTED`. See the
142":ref:`dev-manual/licenses:license flag matching`" section for a full
143explanation of how :term:`LICENSE_FLAGS` matching works. Here is the
144example::
145
146 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly"
147
148Likewise, to additionally enable the package built from the recipe
149containing ``LICENSE_FLAGS = "license_${PN}_${PV}"``, and assuming that
150the actual recipe name was ``emgd_1.10.bb``, the following string would
151enable that package as well as the original ``gst-plugins-ugly``
152package::
153
154 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly license_emgd_1.10"
155
156As a convenience, you do not need to specify the
157complete license string for every package. You can use
158an abbreviated form, which consists of just the first portion or
159portions of the license string before the initial underscore character
160or characters. A partial string will match any license that contains the
161given string as the first portion of its license. For example, the
162following value will also match both of the packages
163previously mentioned as well as any other packages that have licenses
164starting with "commercial" or "license"::
165
166 LICENSE_FLAGS_ACCEPTED = "commercial license"
167
168License Flag Matching
169---------------------
170
171License flag matching allows you to control what recipes the
172OpenEmbedded build system includes in the build. Fundamentally, the
173build system attempts to match :term:`LICENSE_FLAGS` strings found in
174recipes against strings found in :term:`LICENSE_FLAGS_ACCEPTED`.
175A match causes the build system to include a recipe in the
176build, while failure to find a match causes the build system to exclude
177a recipe.
178
179In general, license flag matching is simple. However, understanding some
180concepts will help you correctly and effectively use matching.
181
182Before a flag defined by a particular recipe is tested against the
183entries of :term:`LICENSE_FLAGS_ACCEPTED`, the expanded
184string ``_${PN}`` is appended to the flag. This expansion makes each
185:term:`LICENSE_FLAGS` value recipe-specific. After expansion, the
186string is then matched against the entries. Thus, specifying
187``LICENSE_FLAGS = "commercial"`` in recipe "foo", for example, results
188in the string ``"commercial_foo"``. And, to create a match, that string
189must appear among the entries of :term:`LICENSE_FLAGS_ACCEPTED`.
190
191Judicious use of the :term:`LICENSE_FLAGS` strings and the contents of the
192:term:`LICENSE_FLAGS_ACCEPTED` variable allows you a lot of flexibility for
193including or excluding recipes based on licensing. For example, you can
194broaden the matching capabilities by using license flags string subsets
195in :term:`LICENSE_FLAGS_ACCEPTED`.
196
197.. note::
198
199 When using a string subset, be sure to use the part of the expanded
200 string that precedes the appended underscore character (e.g.
201 ``usethispart_1.3``, ``usethispart_1.4``, and so forth).
202
203For example, simply specifying the string "commercial" in the
204:term:`LICENSE_FLAGS_ACCEPTED` variable matches any expanded
205:term:`LICENSE_FLAGS` definition that starts with the string
206"commercial" such as "commercial_foo" and "commercial_bar", which
207are the strings the build system automatically generates for
208hypothetical recipes named "foo" and "bar" assuming those recipes simply
209specify the following::
210
211 LICENSE_FLAGS = "commercial"
212
213Thus, you can choose to exhaustively enumerate each license flag in the
214list and allow only specific recipes into the image, or you can use a
215string subset that causes a broader range of matches to allow a range of
216recipes into the image.
217
218This scheme works even if the :term:`LICENSE_FLAGS` string already has
219``_${PN}`` appended. For example, the build system turns the license
220flag "commercial_1.2_foo" into "commercial_1.2_foo_foo" and would match
221both the general "commercial" and the specific "commercial_1.2_foo"
222strings found in the :term:`LICENSE_FLAGS_ACCEPTED` variable, as expected.
223
224Here are some other scenarios:
225
226- You can specify a versioned string in the recipe such as
227 "commercial_foo_1.2" in a "foo" recipe. The build system expands this
228 string to "commercial_foo_1.2_foo". Combine this license flag with a
229 :term:`LICENSE_FLAGS_ACCEPTED` variable that has the string
230 "commercial" and you match the flag along with any other flag that
231 starts with the string "commercial".
232
233- Under the same circumstances, you can add "commercial_foo" in the
234 :term:`LICENSE_FLAGS_ACCEPTED` variable and the build system not only
235 matches "commercial_foo_1.2" but also matches any license flag with
236 the string "commercial_foo", regardless of the version.
237
238- You can be very specific and use both the package and version parts
239 in the :term:`LICENSE_FLAGS_ACCEPTED` list (e.g.
240 "commercial_foo_1.2") to specifically match a versioned recipe.
241
242Other Variables Related to Commercial Licenses
243----------------------------------------------
244
245There are other helpful variables related to commercial license handling,
246defined in the
247``poky/meta/conf/distro/include/default-distrovars.inc`` file::
248
249 COMMERCIAL_AUDIO_PLUGINS ?= ""
250 COMMERCIAL_VIDEO_PLUGINS ?= ""
251
252If you want to enable these components, you can do so by making sure you have
253statements similar to the following in your ``local.conf`` configuration file::
254
255 COMMERCIAL_AUDIO_PLUGINS = "gst-plugins-ugly-mad \
256 gst-plugins-ugly-mpegaudioparse"
257 COMMERCIAL_VIDEO_PLUGINS = "gst-plugins-ugly-mpeg2dec \
258 gst-plugins-ugly-mpegstream gst-plugins-bad-mpegvideoparse"
259 LICENSE_FLAGS_ACCEPTED = "commercial_gst-plugins-ugly commercial_gst-plugins-bad commercial_qmmp"
260
261Of course, you could also create a matching list for those components using the
262more general "commercial" string in the :term:`LICENSE_FLAGS_ACCEPTED` variable,
263but that would also enable all the other packages with :term:`LICENSE_FLAGS`
264containing "commercial", which you may or may not want::
265
266 LICENSE_FLAGS_ACCEPTED = "commercial"
267
268Specifying audio and video plugins as part of the
269:term:`COMMERCIAL_AUDIO_PLUGINS` and :term:`COMMERCIAL_VIDEO_PLUGINS` statements
270(along with :term:`LICENSE_FLAGS_ACCEPTED`) includes the plugins or
271components into built images, thus adding support for media formats or
272components.
273
274.. note::
275
276 GStreamer "ugly" and "bad" plugins are actually available through
277 open source licenses. However, the "ugly" ones can be subject to software
278 patents in some countries, making it necessary to pay licensing fees
279 to distribute them. The "bad" ones are just deemed unreliable by the
280 GStreamer community and should therefore be used with care.
281
282Maintaining Open Source License Compliance During Your Product's Lifecycle
283==========================================================================
284
285One of the concerns for a development organization using open source
286software is how to maintain compliance with various open source
287licensing during the lifecycle of the product. While this section does
288not provide legal advice or comprehensively cover all scenarios, it does
289present methods that you can use to assist you in meeting the compliance
290requirements during a software release.
291
292With hundreds of different open source licenses that the Yocto Project
293tracks, it is difficult to know the requirements of each and every
294license. However, the requirements of the major FLOSS licenses can begin
295to be covered by assuming that there are three main areas of concern:
296
297- Source code must be provided.
298
299- License text for the software must be provided.
300
301- Compilation scripts and modifications to the source code must be
302 provided.
303
304There are other requirements beyond the scope of these three and the
305methods described in this section (e.g. the mechanism through which
306source code is distributed).
307
308As different organizations have different ways of releasing software,
309there can be multiple ways of meeting license obligations. At
310least, we describe here two methods for achieving compliance:
311
312- The first method is to use OpenEmbedded's ability to provide
313 the source code, provide a list of licenses, as well as
314 compilation scripts and source code modifications.
315
316 The remainder of this section describes supported methods to meet
317 the previously mentioned three requirements.
318
319- The second method is to generate a *Software Bill of Materials*
320 (:term:`SBoM`), as described in the ":doc:`/dev-manual/sbom`" section.
321 Not only do you generate :term:`SPDX` output which can be used meet
322 license compliance requirements (except for sharing the build system
323 and layers sources for the time being), but this output also includes
324 component version and patch information which can be used
325 for vulnerability assessment.
326
327Whatever method you choose, prior to releasing images, sources,
328and the build system, you should audit all artifacts to ensure
329completeness.
330
331.. note::
332
333 The Yocto Project generates a license manifest during image creation
334 that is located in
335 ``${DEPLOY_DIR}/licenses/${SSTATE_PKGARCH}/<image-name>-<machine>.rootfs-<datestamp>/``
336 to assist with any audits.
337
338Providing the Source Code
339-------------------------
340
341Compliance activities should begin before you generate the final image.
342The first thing you should look at is the requirement that tops the list
343for most compliance groups --- providing the source. The Yocto Project has
344a few ways of meeting this requirement.
345
346One of the easiest ways to meet this requirement is to provide the
347entire :term:`DL_DIR` used by the
348build. This method, however, has a few issues. The most obvious is the
349size of the directory since it includes all sources used in the build
350and not just the source used in the released image. It will include
351toolchain source, and other artifacts, which you would not generally
352release. However, the more serious issue for most companies is
353accidental release of proprietary software. The Yocto Project provides
354an :ref:`ref-classes-archiver` class to help avoid some of these concerns.
355
356Before you employ :term:`DL_DIR` or the :ref:`ref-classes-archiver` class, you
357need to decide how you choose to provide source. The source
358:ref:`ref-classes-archiver` class can generate tarballs and SRPMs and can
359create them with various levels of compliance in mind.
360
361One way of doing this (but certainly not the only way) is to release
362just the source as a tarball. You can do this by adding the following to
363the ``local.conf`` file found in the :term:`Build Directory`::
364
365 INHERIT += "archiver"
366 ARCHIVER_MODE[src] = "original"
367
368During the creation of your
369image, the source from all recipes that deploy packages to the image is
370placed within subdirectories of ``DEPLOY_DIR/sources`` based on the
371:term:`LICENSE` for each recipe.
372Releasing the entire directory enables you to comply with requirements
373concerning providing the unmodified source. It is important to note that
374the size of the directory can get large.
375
376A way to help mitigate the size issue is to only release tarballs for
377licenses that require the release of source. Let us assume you are only
378concerned with GPL code as identified by running the following script:
379
380.. code-block:: shell
381
382 # Script to archive a subset of packages matching specific license(s)
383 # Source and license files are copied into sub folders of package folder
384 # Must be run from build folder
385 #!/bin/bash
386 src_release_dir="source-release"
387 mkdir -p $src_release_dir
388 for a in tmp/deploy/sources/*; do
389 for d in $a/*; do
390 # Get package name from path
391 p=`basename $d`
392 p=${p%-*}
393 p=${p%-*}
394 # Only archive GPL packages (update *GPL* regex for your license check)
395 numfiles=`ls tmp/deploy/licenses/$p/*GPL* 2> /dev/null | wc -l`
396 if [ $numfiles -ge 1 ]; then
397 echo Archiving $p
398 mkdir -p $src_release_dir/$p/source
399 cp $d/* $src_release_dir/$p/source 2> /dev/null
400 mkdir -p $src_release_dir/$p/license
401 cp tmp/deploy/licenses/$p/* $src_release_dir/$p/license 2> /dev/null
402 fi
403 done
404 done
405
406At this point, you
407could create a tarball from the ``gpl_source_release`` directory and
408provide that to the end user. This method would be a step toward
409achieving compliance with section 3a of GPLv2 and with section 6 of
410GPLv3.
411
412Providing License Text
413----------------------
414
415One requirement that is often overlooked is inclusion of license text.
416This requirement also needs to be dealt with prior to generating the
417final image. Some licenses require the license text to accompany the
418binary. You can achieve this by adding the following to your
419``local.conf`` file::
420
421 COPY_LIC_MANIFEST = "1"
422 COPY_LIC_DIRS = "1"
423 LICENSE_CREATE_PACKAGE = "1"
424
425Adding these statements to the
426configuration file ensures that the licenses collected during package
427generation are included on your image.
428
429.. note::
430
431 Setting all three variables to "1" results in the image having two
432 copies of the same license file. One copy resides in
433 ``/usr/share/common-licenses`` and the other resides in
434 ``/usr/share/license``.
435
436 The reason for this behavior is because
437 :term:`COPY_LIC_DIRS` and
438 :term:`COPY_LIC_MANIFEST`
439 add a copy of the license when the image is built but do not offer a
440 path for adding licenses for newly installed packages to an image.
441 :term:`LICENSE_CREATE_PACKAGE`
442 adds a separate package and an upgrade path for adding licenses to an
443 image.
444
445As the source :ref:`ref-classes-archiver` class has already archived the
446original unmodified source that contains the license files, you would have
447already met the requirements for inclusion of the license information
448with source as defined by the GPL and other open source licenses.
449
450Providing Compilation Scripts and Source Code Modifications
451-----------------------------------------------------------
452
453At this point, we have addressed all we need prior to generating the
454image. The next two requirements are addressed during the final
455packaging of the release.
456
457By releasing the version of the OpenEmbedded build system and the layers
458used during the build, you will be providing both compilation scripts
459and the source code modifications in one step.
460
461If the deployment team has a :ref:`overview-manual/concepts:bsp layer`
462and a distro layer, and those
463those layers are used to patch, compile, package, or modify (in any way)
464any open source software included in your released images, you might be
465required to release those layers under section 3 of GPLv2 or section 1
466of GPLv3. One way of doing that is with a clean checkout of the version
467of the Yocto Project and layers used during your build. Here is an
468example:
469
470.. code-block:: shell
471
472 # We built using the dunfell branch of the poky repo
473 $ git clone -b dunfell git://git.yoctoproject.org/poky
474 $ cd poky
475 # We built using the release_branch for our layers
476 $ git clone -b release_branch git://git.mycompany.com/meta-my-bsp-layer
477 $ git clone -b release_branch git://git.mycompany.com/meta-my-software-layer
478 # clean up the .git repos
479 $ find . -name ".git" -type d -exec rm -rf {} \;
480
481One thing a development organization might want to consider for end-user
482convenience is to modify
483``meta-poky/conf/templates/default/bblayers.conf.sample`` to ensure that when
484the end user utilizes the released build system to build an image, the
485development organization's layers are included in the ``bblayers.conf`` file
486automatically::
487
488 # POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
489 # changes incompatibly
490 POKY_BBLAYERS_CONF_VERSION = "2"
491
492 BBPATH = "${TOPDIR}"
493 BBFILES ?= ""
494
495 BBLAYERS ?= " \
496 ##OEROOT##/meta \
497 ##OEROOT##/meta-poky \
498 ##OEROOT##/meta-yocto-bsp \
499 ##OEROOT##/meta-mylayer \
500 "
501
502Creating and
503providing an archive of the :term:`Metadata`
504layers (recipes, configuration files, and so forth) enables you to meet
505your requirements to include the scripts to control compilation as well
506as any modifications to the original source.
507
508Compliance Limitations with Executables Built from Static Libraries
509-------------------------------------------------------------------
510
511When package A is added to an image via the :term:`RDEPENDS` or :term:`RRECOMMENDS`
512mechanisms as well as explicitly included in the image recipe with
513:term:`IMAGE_INSTALL`, and depends on a static linked library recipe B
514(``DEPENDS += "B"``), package B will neither appear in the generated license
515manifest nor in the generated source tarballs. This occurs as the
516:ref:`ref-classes-license` and :ref:`ref-classes-archiver` classes assume that
517only packages included via :term:`RDEPENDS` or :term:`RRECOMMENDS`
518end up in the image.
519
520As a result, potential obligations regarding license compliance for package B
521may not be met.
522
523The Yocto Project doesn't enable static libraries by default, in part because
524of this issue. Before a solution to this limitation is found, you need to
525keep in mind that if your root filesystem is built from static libraries,
526you will need to manually ensure that your deliveries are compliant
527with the licenses of these libraries.
528
529Copying Non Standard Licenses
530=============================
531
532Some packages, such as the linux-firmware package, have many licenses
533that are not in any way common. You can avoid adding a lot of these
534types of common license files, which are only applicable to a specific
535package, by using the
536:term:`NO_GENERIC_LICENSE`
537variable. Using this variable also avoids QA errors when you use a
538non-common, non-CLOSED license in a recipe.
539
540Here is an example that uses the ``LICENSE.Abilis.txt`` file as
541the license from the fetched source::
542
543 NO_GENERIC_LICENSE[Firmware-Abilis] = "LICENSE.Abilis.txt"
544
diff --git a/documentation/dev-manual/limiting-resources.rst b/documentation/dev-manual/limiting-resources.rst
deleted file mode 100644
index 0a792689c3..0000000000
--- a/documentation/dev-manual/limiting-resources.rst
+++ /dev/null
@@ -1,138 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Limiting the Host Resources Usage
4*********************************
5
6While you sometimes need to :doc:`speed up a build
7</dev-manual/speeding-up-build>`, you may also need to limit the resources used
8by the :term:`OpenEmbedded Build System`, especially on shared infrastructures
9where multiple users start heavy-load builds, or when building on low-power
10machines.
11
12This document aims at giving the different configuration variables available to
13limit the resources used by the build system. These variables should be set from
14a :term:`configuration file` and thus take effect over the entire build environment.
15For each variable, also see the variable description in the glossary for more
16details.
17
18- :term:`BB_NUMBER_THREADS`:
19
20 This sets a hard limit on the number of threads :term:`BitBake` can run at the
21 same time. Lowering this value will set a limit to the number of
22 :term:`BitBake` threads, but will not prevent a single task from starting more
23 compilation threads (see :term:`PARALLEL_MAKE`).
24
25- :term:`BB_NUMBER_PARSE_THREADS`:
26
27 Like :term:`BB_NUMBER_THREADS`, but this variable sets a limit on the number
28 of threads during the parsing of the environment (before executing tasks).
29
30- :term:`PARALLEL_MAKE`:
31
32 This variable should be set in the form of ``-jN``, where ``N`` is a positive
33 integer. This integer controls the number of threads used when starting
34 ``make``. Note that this variable is not limited to the usage of ``make``,
35 but extends to the compilation (:ref:`ref-tasks-compile` task) commands
36 defined by the :ref:`ref-classes-meson`, :ref:`ref-classes-cmake` and such
37 classes.
38
39 If you want to have a different limit from the rest of the build for a
40 recipe, it is also possible to achieve with the following line added to your
41 ``local.conf`` :term:`configuration file`::
42
43 PARALLEL_MAKE:pn-linux-yocto = "-j4"
44
45 The above example will limit the number of threads used by ``make`` for the
46 ``linux-yocto`` recipe to 4.
47
48- :term:`PARALLEL_MAKEINST`:
49
50 Like :term:`PARALLEL_MAKE`, but this variable controls the number of threads
51 used during the :ref:`ref-tasks-install` task.
52
53 The default value of :term:`PARALLEL_MAKEINST` is the value of
54 :term:`PARALLEL_MAKE`.
55
56.. note::
57
58 While most of the variables in this document help to limit the CPU load, it
59 is also possible that the host system runs out of physical RAM when running
60 builds. This can trigger the out-of-memory killer and stop the related
61 processes abruptly. This can create strange looking failures in the output
62 log of the tasks in question. The out-of-memory killer only logs in the
63 kernel dmesg logs, so it is advised to monitor it closely with the ``dmesg``
64 command when encountering unexpected failures during builds.
65
66 In these situations, lowering the value of :term:`PARALLEL_MAKE` and
67 :term:`BB_NUMBER_THREADS` is recommended.
68
69- :term:`BB_PRESSURE_MAX_CPU`, :term:`BB_PRESSURE_MAX_IO` and
70 :term:`BB_PRESSURE_MAX_MEMORY`:
71
72 These variables control the limit of pressure (PSI as defined by
73 https://docs.kernel.org/accounting/psi.html) on the system, and will
74 limit the number of :term:`BitBake` threads dynamically depending on the
75 current pressure of the system. This also means that your host must support
76 the PSI kernel feature (otherwise see :term:`BB_LOADFACTOR_MAX` below).
77
78 These variables take a positive integer between 1 (extremely low limit) and
79 1000000 (value unlikely ever reached). Setting an extremely low value, such
80 as 2, is not desirable as it will result in :term:`BitBake` limiting the
81 number of threads to 1 most of the time.
82
83 To determine a reasonable value to set for your host, follow the steps below:
84
85 #. In a Bash shell, start the following script, which will provide an
86 estimate of the current pressure on your host:
87
88 .. code-block:: bash
89
90 pressure="0"
91 while true; do
92 prev_pressure="$pressure"
93 pressure=$(head -1 /proc/pressure/cpu | cut -d' ' -f5 | cut -d'=' -f2)
94 echo $(( $pressure - $prev_pressure ))
95 sleep 1
96 done
97
98 .. note::
99
100 Change ``/proc/pressure/cpu`` to ``/proc/pressure/io`` or
101 ``/proc/pressure/memory`` to change the pressure type to monitor.
102
103 This script can be stopped by pressing Control + C.
104
105 #. Then, start a heavy-load build, for example::
106
107 bitbake virtual/kernel -c compile -f
108
109 You can stop the build at anytime with Control + C.
110
111 #. Monitor the values printed on the console. These should indicate how the
112 pressure evolves during the build. You can take a value below the maximum
113 printed value as a starting point.
114
115 After setting initial values, :term:`BitBake` will print messages on the
116 console in the following format each time the current pressure exceeds of the
117 limit set by the above variables::
118
119 Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads
120
121 Take a look at the value between parenthesis: ``CPU: 1105.9/2.0, IO: 0.0/2.0,
122 Mem: 0.0/2.0``. They correspond to the current pressure value for the CPU, IO
123 and memory respectively. If :term:`BitBake` prints these messages a lot, it
124 is likely that your pressure limit is too low, and thus can be raised to a
125 higher value.
126
127- :term:`BB_LOADFACTOR_MAX`:
128
129 This variable will limit the number of threads :term:`BitBake` will start
130 by monitoring the current CPU load of the host system. :term:`BitBake` will
131 print the following when the limit set by :term:`BB_LOADFACTOR_MAX` is
132 reached::
133
134 Load average limiting set to True as load average: 0.7188262939453125 - using 37/64 bitbake threads
135
136 This variable has no effect when any of :term:`BB_PRESSURE_MAX_CPU`,
137 :term:`BB_PRESSURE_MAX_IO` or :term:`BB_PRESSURE_MAX_MEMORY` is set, as it
138 was designed for systems that do not have pressure information available.
diff --git a/documentation/dev-manual/multiconfig.rst b/documentation/dev-manual/multiconfig.rst
deleted file mode 100644
index 71fe542efb..0000000000
--- a/documentation/dev-manual/multiconfig.rst
+++ /dev/null
@@ -1,312 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Building Images for Multiple Targets With Multiconfig
4*****************************************************
5
6You can use a single ``bitbake`` command to build multiple images or
7packages for different targets where each image or package requires a
8different configuration (multiple configuration builds). The builds, in
9this scenario, are sometimes referred to as "multiconfigs", and this
10section uses that term throughout.
11
12This section describes how to set up for multiple configuration builds
13and how to account for cross-build dependencies between the
14multiconfigs.
15
16Setting Up and Running a Multiple Configuration Build
17=====================================================
18
19To accomplish a multiple configuration build, you must define each
20target's configuration separately using a parallel :term:`configuration file` in
21the :term:`Build Directory` or configuration directory within a layer, and you
22must follow a required file hierarchy. Additionally, you must enable the
23multiple configuration builds in your ``local.conf`` file.
24
25Follow these steps to set up and execute multiple configuration builds:
26
27- *Create Separate Configuration Files*: You need to create a single
28 :term:`Configuration File` for each build target (each multiconfig).
29 The configuration definitions are implementation dependent but often
30 each configuration file will define the :term:`MACHINE` and the
31 temporary directory (:term:`TMPDIR`) BitBake uses for the build.
32
33 .. note::
34
35 Whether the same temporary directory (:term:`TMPDIR`) can be shared will
36 depend on what is similar and what is different between the
37 configurations. Multiple :term:`MACHINE` targets can share the same
38 :term:`TMPDIR` as long as the rest of the configuration is the same,
39 multiple :term:`DISTRO` settings would need separate :term:`TMPDIR`
40 directories.
41
42 For example, consider a scenario with two different multiconfigs for the same
43 :term:`MACHINE`: "qemux86" built for two distributions such as "poky" and
44 "poky-lsb". In this case, you would need to use two different :term:`TMPDIR`.
45
46 In the general case, using separate :term:`TMPDIR` for the different
47 multiconfigs is strongly recommended.
48
49 The location for these multiconfig configuration files is specific.
50 They must reside in the current :term:`Build Directory` in a sub-directory of
51 ``conf`` named ``multiconfig`` or within a :term:`Layer`'s ``conf`` directory
52 under a directory named ``multiconfig``. Here is an example that defines
53 two configuration files for the "x86" and "arm" multiconfigs:
54
55 .. image:: figures/multiconfig_files.png
56 :align: center
57 :width: 50%
58
59 The usual :term:`BBPATH` search path is used to locate multiconfig files in
60 a similar way to other configuration files.
61
62 Here is an example showing the minimal statements needed in a
63 :term:`configuration file` named ``qemux86.conf`` for a ``qemux86`` target
64 whose temporary build directory is ``tmp-qemux86``::
65
66 MACHINE = "qemux86"
67 TMPDIR .= "-${BB_CURRENT_MC}"
68
69 BitBake will expand the :term:`BB_CURRENT_MC` variable to the value of the
70 current multiconfig in use. We append this value to :term:`TMPDIR` so that
71 any change on the definition of :term:`TMPDIR` will automatically affect the
72 value of :term:`TMPDIR` for each multiconfig.
73
74- *Add the BitBake Multi-configuration Variable to the Local
75 Configuration File*: Use the
76 :term:`BBMULTICONFIG`
77 variable in your ``conf/local.conf`` configuration file to specify
78 each multiconfig. Continuing with the example from the previous
79 figure, the :term:`BBMULTICONFIG` variable needs to enable two
80 multiconfigs: "x86" and "arm" by specifying each configuration file::
81
82 BBMULTICONFIG = "x86 arm"
83
84 .. note::
85
86 A "default" configuration already exists by definition. This
87 configuration is named: "" (i.e. empty string) and is defined by
88 the variables coming from your ``local.conf``
89 file. Consequently, the previous example actually adds two
90 additional configurations to your build: "arm" and "x86" along
91 with "".
92
93- *Launch BitBake*: Use the following BitBake command form to launch
94 the multiple configuration build::
95
96 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
97
98 For the example in this section, the following command applies::
99
100 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
101
102 The previous BitBake command builds several components:
103
104 - A ``core-image-minimal`` image that is configured through the ``x86.conf``
105 configuration file
106
107 - A ``core-image-sato`` image that is configured through the ``arm.conf``
108 configuration file
109
110 - A ``core-image-base`` that is configured through your ``local.conf``
111 configuration file
112
113.. note::
114
115 Support for multiple configuration builds in the Yocto Project &DISTRO;
116 (&DISTRO_NAME;) Release does not include Shared State (sstate)
117 optimizations. Consequently, if a build uses the same object twice
118 in, for example, two different :term:`TMPDIR`
119 directories, the build either loads from an existing sstate cache for
120 that build at the start or builds the object fresh.
121
122Enabling Multiple Configuration Build Dependencies
123==================================================
124
125Sometimes dependencies can exist between targets (multiconfigs) in a
126multiple configuration build. For example, suppose that in order to
127build a ``core-image-sato`` image for an "x86" multiconfig, the root
128filesystem of an "arm" multiconfig must exist. This dependency is
129essentially that the
130:ref:`ref-tasks-image` task in the
131``core-image-sato`` recipe depends on the completion of the
132:ref:`ref-tasks-rootfs` task of the
133``core-image-minimal`` recipe.
134
135To enable dependencies in a multiple configuration build, you must
136declare the dependencies in the recipe using the following statement
137form::
138
139 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
140
141To better show how to use this statement, consider the example scenario
142from the first paragraph of this section. The following statement needs
143to be added to the recipe that builds the ``core-image-sato`` image::
144
145 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
146
147In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
148task on which the :ref:`ref-tasks-image` task in the recipe depends is the
149:ref:`ref-tasks-rootfs` task from the ``core-image-minimal`` recipe associated
150with the "arm" multiconfig.
151
152Once you set up this dependency, you can build the "x86" multiconfig
153using a BitBake command as follows::
154
155 $ bitbake mc:x86:core-image-sato
156
157This command executes all the tasks needed to create the
158``core-image-sato`` image for the "x86" multiconfig. Because of the
159dependency, BitBake also executes through the :ref:`ref-tasks-rootfs` task for the
160"arm" multiconfig build.
161
162Having a recipe depend on the root filesystem of another build might not
163seem that useful. Consider this change to the statement in the
164``core-image-sato`` recipe::
165
166 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
167
168In this case, BitBake must
169create the ``core-image-minimal`` image for the "arm" build since the
170"x86" build depends on it.
171
172Because "x86" and "arm" are enabled for multiple configuration builds
173and have separate configuration files, BitBake places the artifacts for
174each build in the respective temporary build directories (i.e.
175:term:`TMPDIR`).
176
177Suggested best practices
178========================
179
180- :term:`TMPDIR` (other than the default set in bitbake.conf) is only set in
181 ``local.conf`` by the user. This means that we should **not** manipulate
182 :term:`TMPDIR` in any way within the Machine or Distro :term:`configuration
183 file`.
184
185- A multiconfig should specify a :term:`TMPDIR`, and should specify it by
186 appending the multiconfig name with :term:`BB_CURRENT_MC`.
187
188- Recipes that are used to transfer the output from a multiconfig build to
189 another should use ``do_task[mcdepends]`` to trigger the build of the
190 component, and then transfer the item to the current configuration in
191 :ref:`ref-tasks-install` and :ref:`ref-tasks-deploy`, assuming the value of
192 the deployed item based on :term:`TMPDIR`.
193
194 The :ref:`ref-tasks-install` and :ref:`ref-tasks-deploy` tasks should look
195 like this::
196
197 do_install() {
198 install -m 0644 ${TMPDIR}-<multiconfig>/tmp/deploy/images/<machine>/somefile ${D}/some/path
199 }
200
201 do_deploy() {
202 install -m 0644 ${TMPDIR}-<multiconfig>/tmp/deploy/images/<machine>/somefile ${DEPLOYDIR}/somefile
203 }
204
205 In the example above:
206
207 - ``<multiconfig>`` is the multiconfig name as set by the multiconfig
208 :term:`configuration file` (see the :ref:`dev-manual/multiconfig:Setting
209 Up and Running a Multiple Configuration Build` section above).
210
211 - ``<machine>`` must be the :term:`MACHINE` for which ``somefile`` was built
212 and deployed. This value may differ from the current :term:`MACHINE` if
213 the multiconfig :term:`configuration file` overrides it.
214
215- Firmware recipes can set the :term:`INHIBIT_DEFAULT_DEPS` variable to ``1``
216 if they don't rely on default dependencies such as the standard C library.
217
218Common use case: building baremetal firmware alongside a Linux build
219====================================================================
220
221A common use case for multiconfig is to use the default configuration as the
222regular Linux build, while one or more multiconfigs can be used to build special
223components, such as baremetal firmware. It would also apply to a scenario where
224a microcontroller, for example, is companion to a main processor where Linux is
225running. This section details how one can achieve these kinds of scenarios with
226a multiconfig build.
227
228Adding a multiconfig configuration file and recipe for a baremetal firmware
229---------------------------------------------------------------------------
230
231As described in :ref:`dev-manual/multiconfig:Setting Up and Running a Multiple
232Configuration Build`, each multiconfig will require a separate
233:term:`Configuration File`. In addition, we will define a separate
234:term:`TMPDIR` for our baremetal firmware build configuration.
235
236For example, we will define a new ``conf/multiconfig/baremetal-firmware.conf``
237as follows::
238
239 TMPDIR .= "-${BB_CURRENT_MC}"
240 TCLIBC = "newlib"
241
242The ``baremetal-firmware.conf`` file configures a separate :term:`TMPDIR` for
243holding binaries compiled with the `newlib <https://sourceware.org/newlib/>`__
244toolchain (see :term:`TCLIBC`).
245
246.. note::
247
248 Here, the default :term:`MACHINE` is not overridden by the multiconfig
249 configuration file. As a consequence, the architecture of the built baremetal
250 binaries will be the same. In other cases, where the firmware runs on a
251 completely different architecture, the :term:`MACHINE` must be overridden.
252
253We then create a recipe ``my-firmware.bb`` that defines how the baremetal
254firmware is built. The recipe should contain enough information for the
255:term:`OpenEmbedded build system` to properly compile the firmware with our
256toolchain. The building tasks may vary depending on the nature of the firmware.
257However, the recipe should define a :ref:`ref-classes-deploy` task that deploys
258the output into the :term:`DEPLOYDIR` directory. We will consider in the
259following that the file is named ``my-firmware.elf``.
260
261Building the firmware
262---------------------
263
264The firmware can be built with BitBake with the following command::
265
266 $ bitbake mc:baremetal-firmware:my-firmware
267
268However, we would prefer for ``my-firmware`` to be automatically built when
269triggering a normal Linux build.
270
271Using a ``mcdepend``, a recipe belonging to the Linux build can trigger the
272build of ``my-firmware``. For example, let's consider that our Linux build needs
273to assemble a "special" firmware that uses the output of our ``my-firmware``
274recipe - let's call it ``my-parent-firmware.bb``. Then, we should specify this
275dependency in ``my-parent-firmware.bb`` with::
276
277 do_compile[mcdepends] = "mc::baremetal-firmware:my-firmware:do_deploy"
278
279The above will ensure that when the :ref:`ref-tasks-compile` task of
280``my-parent-firmware`` is triggered, the :ref:`ref-tasks-deploy` task of
281``my-firmware`` will already have run successfully.
282
283Using the output of ``my-firmware``
284-----------------------------------
285
286After ``my-firmware`` recipe has deployed ``my-firmware.elf``, we need to use
287the output in some way. We can make a series of assumptions, based on the
288default Yocto Project variables in order to get the binary for packaging.
289
290First, we can set the following in ``my-parent-firmware.bb``::
291
292 FIRMWARE_FILE ??= "${TMPDIR}-baremetal-firmware/deploy/images/<machine>/my-firmware.elf"
293 FIRMWARE_FILE[vardepsexclude] += "TMPDIR"
294
295The first assignment stores the value of the path to the firmware built and
296deployed by the ``my-firmware.bb`` recipe. The second assignment excludes the
297:term:`TMPDIR` variable from being part of ``FIRMWARE_FILE``'s dependencies ---
298meaning that changing the value of :term:`TMPDIR` (for example, changing the
299host on which the firmware is built) will not invalidate the :ref:`shared state
300cache <overview-manual/concepts:shared state cache>`.
301
302Additionally, ``<machine>`` should be replaced by the :term:`MACHINE` for which
303we are building in the baremetal-firmware context.
304
305We can then add a :ref:`ref-tasks-install` task to ``my-parent-firmware``::
306
307 do_install() {
308 install -Dm 0644 ${FIRMWARE_FILE} ${D}/lib/firmware/my-firmware.elf
309 }
310
311Doing the above will allow the firmware binary to be transferred and packaged
312into the Linux context and rootfs.
diff --git a/documentation/dev-manual/new-machine.rst b/documentation/dev-manual/new-machine.rst
deleted file mode 100644
index 469b2d395a..0000000000
--- a/documentation/dev-manual/new-machine.rst
+++ /dev/null
@@ -1,118 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Adding a New Machine
4********************
5
6Adding a new machine to the Yocto Project is a straightforward process.
7This section describes how to add machines that are similar to those
8that the Yocto Project already supports.
9
10.. note::
11
12 Although well within the capabilities of the Yocto Project, adding a
13 totally new architecture might require changes to ``gcc``/``glibc``
14 and to the site information, which is beyond the scope of this
15 manual.
16
17For a complete example that shows how to add a new machine, see the
18":ref:`bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script`"
19section in the Yocto Project Board Support Package (BSP) Developer's
20Guide.
21
22Adding the Machine Configuration File
23=====================================
24
25To add a new machine, you need to add a new machine configuration file
26to the layer's ``conf/machine`` directory. This configuration file
27provides details about the device you are adding.
28
29The OpenEmbedded build system uses the root name of the machine
30configuration file to reference the new machine. For example, given a
31machine configuration file named ``crownbay.conf``, the build system
32recognizes the machine as "crownbay".
33
34The most important variables you must set in your machine configuration
35file or include from a lower-level configuration file are as follows:
36
37- :term:`TARGET_ARCH` (e.g. "arm")
38
39- ``PREFERRED_PROVIDER_virtual/kernel``
40
41- :term:`MACHINE_FEATURES` (e.g. "screen wifi")
42
43You might also need these variables:
44
45- :term:`SERIAL_CONSOLES` (e.g. "115200;ttyS0 115200;ttyS1")
46
47- :term:`KERNEL_IMAGETYPE` (e.g. "zImage")
48
49- :term:`IMAGE_FSTYPES` (e.g. "tar.gz jffs2")
50
51You can find full details on these variables in the reference section.
52You can leverage existing machine ``.conf`` files from
53``meta-yocto-bsp/conf/machine/``.
54
55Adding a Kernel for the Machine
56===============================
57
58The OpenEmbedded build system needs to be able to build a kernel for the
59machine. You need to either create a new kernel recipe for this machine,
60or extend an existing kernel recipe. You can find several kernel recipe
61examples in the Source Directory at ``meta/recipes-kernel/linux`` that
62you can use as references.
63
64If you are creating a new kernel recipe, normal recipe-writing rules
65apply for setting up a :term:`SRC_URI`. Thus, you need to specify any
66necessary patches and set :term:`S` to point at the source code. You need to
67create a :ref:`ref-tasks-configure` task that configures the unpacked kernel with
68a ``defconfig`` file. You can do this by using a ``make defconfig``
69command or, more commonly, by copying in a suitable ``defconfig`` file
70and then running ``make oldconfig``. By making use of ``inherit kernel``
71and potentially some of the ``linux-*.inc`` files, most other
72functionality is centralized and the defaults of the class normally work
73well.
74
75If you are extending an existing kernel recipe, it is usually a matter
76of adding a suitable ``defconfig`` file. The file needs to be added into
77a location similar to ``defconfig`` files used for other machines in a
78given kernel recipe. A possible way to do this is by listing the file in
79the :term:`SRC_URI` and adding the machine to the expression in
80:term:`COMPATIBLE_MACHINE`::
81
82 COMPATIBLE_MACHINE = '(qemux86|qemumips)'
83
84For more information on ``defconfig`` files, see the
85":ref:`kernel-dev/common:changing the configuration`"
86section in the Yocto Project Linux Kernel Development Manual.
87
88Adding a Formfactor Configuration File
89======================================
90
91A formfactor configuration file provides information about the target
92hardware for which the image is being built and information that the
93build system cannot obtain from other sources such as the kernel. Some
94examples of information contained in a formfactor configuration file
95include framebuffer orientation, whether or not the system has a
96keyboard, the positioning of the keyboard in relation to the screen, and
97the screen resolution.
98
99The build system uses reasonable defaults in most cases. However, if
100customization is necessary, you need to create a ``machconfig`` file in
101the ``meta/recipes-bsp/formfactor/files`` directory. This directory
102contains directories for specific machines such as ``qemuarm`` and
103``qemux86``. For information about the settings available and the
104defaults, see the ``meta/recipes-bsp/formfactor/files/config`` file
105found in the same area.
106
107Here is an example for "qemuarm" machine::
108
109 HAVE_TOUCHSCREEN=1
110 HAVE_KEYBOARD=1
111 DISPLAY_CAN_ROTATE=0
112 DISPLAY_ORIENTATION=0
113 #DISPLAY_WIDTH_PIXELS=640
114 #DISPLAY_HEIGHT_PIXELS=480
115 #DISPLAY_BPP=16
116 DISPLAY_DPI=150
117 DISPLAY_SUBPIXEL_ORDER=vrgb
118
diff --git a/documentation/dev-manual/new-recipe.rst b/documentation/dev-manual/new-recipe.rst
deleted file mode 100644
index aa4fb97a4b..0000000000
--- a/documentation/dev-manual/new-recipe.rst
+++ /dev/null
@@ -1,1656 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Writing a New Recipe
4********************
5
6Recipes (``.bb`` files) are fundamental components in the Yocto Project
7environment. Each software component built by the OpenEmbedded build
8system requires a recipe to define the component. This section describes
9how to create, write, and test a new recipe.
10
11.. note::
12
13 For information on variables that are useful for recipes and for
14 information about recipe naming issues, see the
15 ":ref:`ref-manual/varlocality:recipes`" section of the Yocto Project
16 Reference Manual.
17
18Overview
19========
20
21The following figure shows the basic process for creating a new recipe.
22The remainder of the section provides details for the steps.
23
24.. image:: figures/recipe-workflow.png
25 :align: center
26 :width: 50%
27
28Locate or Automatically Create a Base Recipe
29============================================
30
31You can always write a recipe from scratch. However, there are three choices
32that can help you quickly get started with a new recipe:
33
34- ``devtool add``: A command that assists in creating a recipe and an
35 environment conducive to development.
36
37- ``recipetool create``: A command provided by the Yocto Project that
38 automates creation of a base recipe based on the source files.
39
40- *Existing Recipes:* Location and modification of an existing recipe
41 that is similar in function to the recipe you need.
42
43.. note::
44
45 For information on recipe syntax, see the
46 ":ref:`dev-manual/new-recipe:recipe syntax`" section.
47
48Creating the Base Recipe Using ``devtool add``
49----------------------------------------------
50
51The ``devtool add`` command uses the same logic for auto-creating the
52recipe as ``recipetool create``, which is listed below. Additionally,
53however, ``devtool add`` sets up an environment that makes it easy for
54you to patch the source and to make changes to the recipe as is often
55necessary when adding a recipe to build a new piece of software to be
56included in a build.
57
58You can find a complete description of the ``devtool add`` command in
59the ":ref:`dev-manual/devtool:a closer look at ``devtool add```" section
60in the Yocto Project Application Development and the Extensible Software
61Development Kit (eSDK) manual.
62
63Creating the Base Recipe Using ``recipetool create``
64----------------------------------------------------
65
66``recipetool create`` automates creation of a base recipe given a set of
67source code files. As long as you can extract or point to the source
68files, the tool will construct a recipe and automatically configure all
69pre-build information into the recipe. For example, suppose you have an
70application that builds using Autotools. Creating the base recipe using
71``recipetool`` results in a recipe that has the pre-build dependencies,
72license requirements, and checksums configured.
73
74To run the tool, you just need to be in your :term:`Build Directory` and
75have sourced the build environment setup script (i.e.
76:ref:`structure-core-script`). To get help on the tool, use the following
77command::
78
79 $ recipetool -h
80 NOTE: Starting bitbake server...
81 usage: recipetool [-d] [-q] [--color COLOR] [-h] <subcommand> ...
82
83 OpenEmbedded recipe tool
84
85 options:
86 -d, --debug Enable debug output
87 -q, --quiet Print only errors
88 --color COLOR Colorize output (where COLOR is auto, always, never)
89 -h, --help show this help message and exit
90
91 subcommands:
92 create Create a new recipe
93 newappend Create a bbappend for the specified target in the specified
94 layer
95 setvar Set a variable within a recipe
96 appendfile Create/update a bbappend to replace a target file
97 appendsrcfiles Create/update a bbappend to add or replace source files
98 appendsrcfile Create/update a bbappend to add or replace a source file
99 Use recipetool <subcommand> --help to get help on a specific command
100
101Running ``recipetool create -o OUTFILE`` creates the base recipe and
102locates it properly in the layer that contains your source files.
103Here are some syntax examples:
104
105 - Use this syntax to generate a recipe based on source. Once generated,
106 the recipe resides in the existing source code layer::
107
108 recipetool create -o OUTFILE source
109
110 - Use this syntax to generate a recipe using code that
111 you extract from source. The extracted code is placed in its own layer
112 defined by :term:`EXTERNALSRC`::
113
114 recipetool create -o OUTFILE -x EXTERNALSRC source
115
116 - Use this syntax to generate a recipe based on source. The options
117 direct ``recipetool`` to generate debugging information. Once generated,
118 the recipe resides in the existing source code layer::
119
120 recipetool create -d -o OUTFILE source
121
122Locating and Using a Similar Recipe
123-----------------------------------
124
125Before writing a recipe from scratch, it is often useful to discover
126whether someone else has already written one that meets (or comes close
127to meeting) your needs. The Yocto Project and OpenEmbedded communities
128maintain many recipes that might be candidates for what you are doing.
129You can find a good central index of these recipes in the
130:oe_layerindex:`OpenEmbedded Layer Index <>`.
131
132Working from an existing recipe or a skeleton recipe is the best way to
133get started. Here are some points on both methods:
134
135- *Locate and modify a recipe that is close to what you want to do:*
136 This method works when you are familiar with the current recipe
137 space. The method does not work so well for those new to the Yocto
138 Project or writing recipes.
139
140 Some risks associated with this method are using a recipe that has
141 areas totally unrelated to what you are trying to accomplish with
142 your recipe, not recognizing areas of the recipe that you might have
143 to add from scratch, and so forth. All these risks stem from
144 unfamiliarity with the existing recipe space.
145
146- *Use and modify the following skeleton recipe:* If for some reason
147 you do not want to use ``recipetool`` and you cannot find an existing
148 recipe that is close to meeting your needs, you can use the following
149 structure to provide the fundamental areas of a new recipe::
150
151 DESCRIPTION = ""
152 HOMEPAGE = ""
153 LICENSE = ""
154 SECTION = ""
155 DEPENDS = ""
156 LIC_FILES_CHKSUM = ""
157
158 SRC_URI = ""
159
160Storing and Naming the Recipe
161=============================
162
163Once you have your base recipe, you should put it in your own layer and
164name it appropriately. Locating it correctly ensures that the
165OpenEmbedded build system can find it when you use BitBake to process
166the recipe.
167
168- *Storing Your Recipe:* The OpenEmbedded build system locates your
169 recipe through the layer's ``conf/layer.conf`` file and the
170 :term:`BBFILES` variable. This
171 variable sets up a path from which the build system can locate
172 recipes. Here is the typical use::
173
174 BBFILES += "${LAYERDIR}/recipes-*/*/*.bb \
175 ${LAYERDIR}/recipes-*/*/*.bbappend"
176
177 Consequently, you need to be sure you locate your new recipe inside
178 your layer such that it can be found.
179
180 You can find more information on how layers are structured in the
181 ":ref:`dev-manual/layers:understanding and creating layers`" section.
182
183- *Naming Your Recipe:* When you name your recipe, you need to follow
184 this naming convention::
185
186 basename_version.bb
187
188 Use lower-cased characters and do not include the reserved suffixes
189 ``-native``, ``-cross``, ``-initial``, or ``-dev`` casually (i.e. do not use
190 them as part of your recipe name unless the string applies). Here are some
191 examples (which includes the use of the string "git" as a special case of a
192 version identifier):
193
194 .. code-block:: none
195
196 cups_2.4.12.bb
197 gawk_5.3.2.bb
198 psplash_git.bb
199
200Running a Build on the Recipe
201=============================
202
203Creating a new recipe is usually an iterative process that requires
204using BitBake to process the recipe multiple times in order to
205progressively discover and add information to the recipe file.
206
207Assuming you have sourced the build environment setup script (i.e.
208:ref:`structure-core-script`) and you are in the :term:`Build Directory`, use
209BitBake to process your recipe. All you need to provide is the
210``basename`` of the recipe as described in the previous section::
211
212 $ bitbake basename
213
214During the build, the OpenEmbedded build system creates a temporary work
215directory for each recipe
216(``${``\ :term:`WORKDIR`\ ``}``)
217where it keeps extracted source files, log files, intermediate
218compilation and packaging files, and so forth.
219
220The path to the per-recipe temporary work directory depends on the
221context in which it is being built. The quickest way to find this path
222is to have BitBake return it by running the following::
223
224 $ bitbake -e basename | grep ^WORKDIR=
225
226As an example, assume a Source Directory
227top-level folder named ``poky``, a default :term:`Build Directory` at
228``poky/build``, and a ``qemux86-poky-linux`` machine target system.
229Furthermore, suppose your recipe is named ``foo_1.3.0.bb``. In this
230case, the work directory the build system uses to build the package
231would be as follows::
232
233 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
234
235Inside this directory you can find sub-directories such as ``image``,
236``packages-split``, and ``temp``. After the build, you can examine these
237to determine how well the build went.
238
239.. note::
240
241 You can find log files for each task in the recipe's ``temp``
242 directory (e.g. ``poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0/temp``).
243 Log files are named ``log.taskname`` (e.g. ``log.do_configure``,
244 ``log.do_fetch``, and ``log.do_compile``).
245
246You can find more information about the build process in
247":doc:`/overview-manual/development-environment`"
248chapter of the Yocto Project Overview and Concepts Manual.
249
250Fetching Code
251=============
252
253The first thing your recipe must do is specify how to fetch the source
254files. Fetching is controlled mainly through the
255:term:`SRC_URI` variable. Your recipe
256must have a :term:`SRC_URI` variable that points to where the source is
257located. For a graphical representation of source locations, see the
258":ref:`overview-manual/concepts:sources`" section in
259the Yocto Project Overview and Concepts Manual.
260
261The :ref:`ref-tasks-fetch` task uses the prefix of each entry in the
262:term:`SRC_URI` variable value to determine which
263:ref:`fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`
264to use to get your source files. It is the :term:`SRC_URI` variable that triggers
265the fetcher. The :ref:`ref-tasks-patch` task uses the variable after source is
266fetched to apply patches. The OpenEmbedded build system uses
267:term:`FILESOVERRIDES` for scanning directory locations for local files in
268:term:`SRC_URI`.
269
270The :term:`SRC_URI` variable in your recipe must define each unique location
271for your source files. It is good practice to not hard-code version
272numbers in a URL used in :term:`SRC_URI`. Rather than hard-code these
273values, use ``${``\ :term:`PV`\ ``}``,
274which causes the fetch process to use the version specified in the
275recipe filename. Specifying the version in this manner means that
276upgrading the recipe to a future version is as simple as renaming the
277recipe to match the new version.
278
279Here is a simple example from the
280:oe_git:`strace recipe </openembedded-core/tree/meta/recipes-devtools/strace>`
281where the source comes from a single tarball. Notice the use of the
282:term:`PV` variable::
283
284 SRC_URI = "${GITHUB_BASE_URI}/download/v${PV}/strace-${PV}.tar.xz \
285
286Files mentioned in :term:`SRC_URI` whose names end in a typical archive
287extension (e.g. ``.tar``, ``.tar.gz``, ``.tar.bz2``, ``.zip``, and so
288forth), are automatically extracted during the
289:ref:`ref-tasks-unpack` task. For
290another example that specifies these types of files, see the
291":ref:`dev-manual/new-recipe:building an autotooled package`" section.
292
293Another way of specifying source is from an SCM. For Git repositories,
294you must specify :term:`SRCREV` and you should specify :term:`PV` to include
295a ``+`` sign in its definition. Here is an example from the recipe
296:oe_git:`l3afpad_git.bb </openembedded-core/tree/meta/recipes-sato/l3afpad/l3afpad_git.bb>`::
297
298 SRC_URI = "git://github.com/stevenhoneyman/l3afpad.git;branch=master;protocol=https"
299
300 PV = "0.8.18.1.11+git"
301 SRCREV = "3cdccdc9505643e50f8208171d9eee5de11a42ff"
302
303If your :term:`SRC_URI` statement includes URLs pointing to individual files
304fetched from a remote server other than a version control system,
305BitBake attempts to verify the files against checksums defined in your
306recipe to ensure they have not been tampered with or otherwise modified
307since the recipe was written. Multiple checksums are supported:
308``SRC_URI[md5sum]``, ``SRC_URI[sha1sum]``, ``SRC_URI[sha256sum]``.
309``SRC_URI[sha384sum]`` and ``SRC_URI[sha512sum]``, but only
310``SRC_URI[sha256sum]`` is commonly used.
311
312.. note::
313
314 ``SRC_URI[md5sum]`` used to also be commonly used, but it is deprecated
315 and should be replaced by ``SRC_URI[sha256sum]`` when updating existing
316 recipes.
317
318If your :term:`SRC_URI` variable points to more than a single URL (excluding
319SCM URLs), you need to provide the ``sha256`` checksum for each URL. For these
320cases, you provide a name for each URL as part of the :term:`SRC_URI` and then
321reference that name in the subsequent checksum statements. Here is an example
322combining lines from the files ``git.inc`` and ``git_2.24.1.bb``::
323
324 SRC_URI = "${KERNELORG_MIRROR}/software/scm/git/git-${PV}.tar.gz;name=tarball \
325 ${KERNELORG_MIRROR}/software/scm/git/git-manpages-${PV}.tar.gz;name=manpages"
326
327 SRC_URI[tarball.sha256sum] = "ad5334956301c86841eb1e5b1bb20884a6bad89a10a6762c958220c7cf64da02"
328 SRC_URI[manpages.sha256sum] = "9a7ae3a093bea39770eb96ca3e5b40bff7af0b9f6123f089d7821d0e5b8e1230"
329
330The proper value for the ``sha256`` checksum might be available together
331with other signatures on the download page for the upstream source (e.g.
332``md5``, ``sha1``, ``sha256``, ``GPG``, and so forth). Because the
333OpenEmbedded build system typically only deals with ``sha256sum``,
334you should verify all the signatures you find by hand.
335
336If no :term:`SRC_URI` checksums are specified when you attempt to build the
337recipe, or you provide an incorrect checksum, the build will produce an
338error for each missing or incorrect checksum. As part of the error
339message, the build system provides the checksum string corresponding to
340the fetched file. Once you have the correct checksums, you can copy and
341paste them into your recipe and then run the build again to continue.
342
343.. note::
344
345 As mentioned, if the upstream source provides signatures for
346 verifying the downloaded source code, you should verify those
347 manually before setting the checksum values in the recipe and
348 continuing with the build.
349
350This final example is a bit more complicated and is from the
351:oe_git:`rxvt-unicode </openembedded-core/tree/meta/recipes-sato/rxvt-unicode>`
352recipe. The example's :term:`SRC_URI` statement identifies multiple files as the source
353files for the recipe: a tarball, a patch file, a desktop file, and an icon::
354
355 SRC_URI = "http://dist.schmorp.de/rxvt-unicode/Attic/rxvt-unicode-${PV}.tar.bz2 \
356 file://xwc.patch \
357 file://rxvt.desktop \
358 file://rxvt.png"
359
360When you specify local files using the ``file://`` URI protocol, the
361build system fetches files from the local machine. The path is relative
362to the :term:`FILESPATH` variable
363and searches specific directories in a certain order:
364``${``\ :term:`BP`\ ``}``,
365``${``\ :term:`BPN`\ ``}``, and
366``files``. The directories are assumed to be subdirectories of the
367directory in which the recipe or append file resides. For another
368example that specifies these types of files, see the
369"`building a single .c file package`_" section.
370
371The previous example also specifies a patch file. Patch files are files
372whose names usually end in ``.patch`` or ``.diff`` but can end with
373compressed suffixes such as ``diff.gz`` and ``patch.bz2``, for example.
374The build system automatically applies patches as described in the
375":ref:`dev-manual/new-recipe:patching code`" section.
376
377Fetching Code Through Firewalls
378-------------------------------
379
380Some users are behind firewalls and need to fetch code through a proxy.
381See the ":doc:`/ref-manual/faq`" chapter for advice.
382
383Limiting the Number of Parallel Connections
384-------------------------------------------
385
386Some users are behind firewalls or use servers where the number of parallel
387connections is limited. In such cases, you can limit the number of fetch
388tasks being run in parallel by adding the following to your ``local.conf``
389file::
390
391 do_fetch[number_threads] = "4"
392
393Unpacking Code
394==============
395
396During the build, the
397:ref:`ref-tasks-unpack` task unpacks
398the source with ``${``\ :term:`S`\ ``}``
399pointing to where it is unpacked.
400
401If you are fetching your source files from an upstream source archived
402tarball and the tarball's internal structure matches the common
403convention of a top-level subdirectory named
404``${``\ :term:`BPN`\ ``}-${``\ :term:`PV`\ ``}``,
405then you do not need to set :term:`S`. However, if :term:`SRC_URI` specifies to
406fetch source from an archive that does not use this convention, or from
407an SCM like Git or Subversion, your recipe needs to define :term:`S`.
408
409If processing your recipe using BitBake successfully unpacks the source
410files, you need to be sure that the directory pointed to by ``${S}``
411matches the structure of the source.
412
413Patching Code
414=============
415
416Sometimes it is necessary to patch code after it has been fetched. Any
417files mentioned in :term:`SRC_URI` whose names end in ``.patch`` or
418``.diff`` or compressed versions of these suffixes (e.g. ``diff.gz``,
419``patch.bz2``, etc.) are treated as patches. The
420:ref:`ref-tasks-patch` task
421automatically applies these patches.
422
423The build system should be able to apply patches with the "-p1" option
424(i.e. one directory level in the path will be stripped off). If your
425patch needs to have more directory levels stripped off, specify the
426number of levels using the "striplevel" option in the :term:`SRC_URI` entry
427for the patch. Alternatively, if your patch needs to be applied in a
428specific subdirectory that is not specified in the patch file, use the
429"patchdir" option in the entry.
430
431As with all local files referenced in
432:term:`SRC_URI` using ``file://``,
433you should place patch files in a directory next to the recipe either
434named the same as the base name of the recipe
435(:term:`BP` and
436:term:`BPN`) or "files".
437
438Licensing
439=========
440
441Your recipe needs to define variables related to the license
442under whith the software is distributed. See the
443:ref:`contributor-guide/recipe-style-guide:recipe license fields`
444section in the Contributor Guide for details.
445
446Dependencies
447============
448
449Most software packages have a short list of other packages that they
450require, which are called dependencies. These dependencies fall into two
451main categories: build-time dependencies, which are required when the
452software is built; and runtime dependencies, which are required to be
453installed on the target in order for the software to run.
454
455Within a recipe, you specify build-time dependencies using the
456:term:`DEPENDS` variable. Although there are nuances,
457items specified in :term:`DEPENDS` should be names of other
458recipes. It is important that you specify all build-time dependencies
459explicitly.
460
461Another consideration is that configure scripts might automatically
462check for optional dependencies and enable corresponding functionality
463if those dependencies are found. If you wish to make a recipe that is
464more generally useful (e.g. publish the recipe in a layer for others to
465use), instead of hard-disabling the functionality, you can use the
466:term:`PACKAGECONFIG` variable to allow functionality and the
467corresponding dependencies to be enabled and disabled easily by other
468users of the recipe.
469
470Similar to build-time dependencies, you specify runtime dependencies
471through a variable -
472:term:`RDEPENDS`, which is
473package-specific. All variables that are package-specific need to have
474the name of the package added to the end as an override. Since the main
475package for a recipe has the same name as the recipe, and the recipe's
476name can be found through the
477``${``\ :term:`PN`\ ``}`` variable, then
478you specify the dependencies for the main package by setting
479``RDEPENDS:${PN}``. If the package were named ``${PN}-tools``, then you
480would set ``RDEPENDS:${PN}-tools``, and so forth.
481
482Some runtime dependencies will be set automatically at packaging time.
483These dependencies include any shared library dependencies (i.e. if a
484package "example" contains "libexample" and another package "mypackage"
485contains a binary that links to "libexample" then the OpenEmbedded build
486system will automatically add a runtime dependency to "mypackage" on
487"example"). See the
488":ref:`overview-manual/concepts:automatically added runtime dependencies`"
489section in the Yocto Project Overview and Concepts Manual for further
490details.
491
492Configuring the Recipe
493======================
494
495Most software provides some means of setting build-time configuration
496options before compilation. Typically, setting these options is
497accomplished by running a configure script with options, or by modifying
498a build configuration file.
499
500.. note::
501
502 As of Yocto Project Release 1.7, some of the core recipes that
503 package binary configuration scripts now disable the scripts due to
504 the scripts previously requiring error-prone path substitution. The
505 OpenEmbedded build system uses ``pkg-config`` now, which is much more
506 robust. You can find a list of the ``*-config`` scripts that are disabled
507 in the ":ref:`migration-1.7-binary-configuration-scripts-disabled`" section
508 in the Yocto Project Reference Manual.
509
510A major part of build-time configuration is about checking for
511build-time dependencies and possibly enabling optional functionality as
512a result. You need to specify any build-time dependencies for the
513software you are building in your recipe's
514:term:`DEPENDS` value, in terms of
515other recipes that satisfy those dependencies. You can often find
516build-time or runtime dependencies described in the software's
517documentation.
518
519The following list provides configuration items of note based on how
520your software is built:
521
522- *Autotools:* If your source files have a ``configure.ac`` file, then
523 your software is built using Autotools. If this is the case, you just
524 need to modify the configuration.
525
526 When using Autotools, your recipe needs to inherit the
527 :ref:`ref-classes-autotools` class and it does not have to
528 contain a :ref:`ref-tasks-configure` task. However, you might still want to
529 make some adjustments. For example, you can set :term:`EXTRA_OECONF` or
530 :term:`PACKAGECONFIG_CONFARGS` to pass any needed configure options that
531 are specific to the recipe.
532
533- *CMake:* If your source files have a ``CMakeLists.txt`` file, then
534 your software is built using CMake. If this is the case, you just
535 need to modify the configuration.
536
537 When you use CMake, your recipe needs to inherit the
538 :ref:`ref-classes-cmake` class and it does not have to contain a
539 :ref:`ref-tasks-configure` task. You can make some adjustments by setting
540 :term:`EXTRA_OECMAKE` to pass any needed configure options that are
541 specific to the recipe.
542
543 .. note::
544
545 If you need to install one or more custom CMake toolchain files
546 that are supplied by the application you are building, install the
547 files to ``${D}${datadir}/cmake/Modules`` during :ref:`ref-tasks-install`.
548
549- *Other:* If your source files do not have a ``configure.ac`` or
550 ``CMakeLists.txt`` file, then your software is built using some
551 method other than Autotools or CMake. If this is the case, you
552 normally need to provide a
553 :ref:`ref-tasks-configure` task
554 in your recipe unless, of course, there is nothing to configure.
555
556 Even if your software is not being built by Autotools or CMake, you
557 still might not need to deal with any configuration issues. You need
558 to determine if configuration is even a required step. You might need
559 to modify a Makefile or some configuration file used for the build to
560 specify necessary build options. Or, perhaps you might need to run a
561 provided, custom configure script with the appropriate options.
562
563 For the case involving a custom configure script, you would run
564 ``./configure --help`` and look for the options you need to set.
565
566Once configuration succeeds, it is always good practice to look at the
567``log.do_configure`` file to ensure that the appropriate options have
568been enabled and no additional build-time dependencies need to be added
569to :term:`DEPENDS`. For example, if the configure script reports that it
570found something not mentioned in :term:`DEPENDS`, or that it did not find
571something that it needed for some desired optional functionality, then
572you would need to add those to :term:`DEPENDS`. Looking at the log might
573also reveal items being checked for, enabled, or both that you do not
574want, or items not being found that are in :term:`DEPENDS`, in which case
575you would need to look at passing extra options to the configure script
576as needed. For reference information on configure options specific to
577the software you are building, you can consult the output of the
578``./configure --help`` command within ``${S}`` or consult the software's
579upstream documentation.
580
581Using Headers to Interface with Devices
582=======================================
583
584If your recipe builds an application that needs to communicate with some
585device or needs an API into a custom kernel, you will need to provide
586appropriate header files. Under no circumstances should you ever modify
587the existing
588``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc`` file.
589These headers are used to build ``libc`` and must not be compromised
590with custom or machine-specific header information. If you customize
591``libc`` through modified headers all other applications that use
592``libc`` thus become affected.
593
594.. note::
595
596 Never copy and customize the ``libc`` header file (i.e.
597 ``meta/recipes-kernel/linux-libc-headers/linux-libc-headers.inc``).
598
599The correct way to interface to a device or custom kernel is to use a
600separate package that provides the additional headers for the driver or
601other unique interfaces. When doing so, your application also becomes
602responsible for creating a dependency on that specific provider.
603
604Consider the following:
605
606- Never modify ``linux-libc-headers.inc``. Consider that file to be
607 part of the ``libc`` system, and not something you use to access the
608 kernel directly. You should access ``libc`` through specific ``libc``
609 calls.
610
611- Applications that must talk directly to devices should either provide
612 necessary headers themselves, or establish a dependency on a special
613 headers package that is specific to that driver.
614
615For example, suppose you want to modify an existing header that adds I/O
616control or network support. If the modifications are used by a small
617number programs, providing a unique version of a header is easy and has
618little impact. When doing so, bear in mind the guidelines in the
619previous list.
620
621.. note::
622
623 If for some reason your changes need to modify the behavior of the ``libc``,
624 and subsequently all other applications on the system, use a ``.bbappend``
625 to modify the ``linux-kernel-headers.inc`` file. However, take care to not
626 make the changes machine specific.
627
628Consider a case where your kernel is older and you need an older
629``libc`` ABI. The headers installed by your recipe should still be a
630standard mainline kernel, not your own custom one.
631
632When you use custom kernel headers you need to get them from
633:term:`STAGING_KERNEL_DIR`,
634which is the directory with kernel headers that are required to build
635out-of-tree modules. Your recipe will also need the following::
636
637 do_configure[depends] += "virtual/kernel:do_shared_workdir"
638
639Compilation
640===========
641
642During a build, the :ref:`ref-tasks-compile` task happens after source is fetched,
643unpacked, and configured. If the recipe passes through :ref:`ref-tasks-compile`
644successfully, nothing needs to be done.
645
646However, if the compile step fails, you need to diagnose the failure.
647Here are some common issues that cause failures.
648
649.. note::
650
651 For cases where improper paths are detected for configuration files
652 or for when libraries/headers cannot be found, be sure you are using
653 the more robust ``pkg-config``. See the note in section
654 ":ref:`dev-manual/new-recipe:Configuring the Recipe`" for additional information.
655
656- *Parallel build failures:* These failures manifest themselves as
657 intermittent errors, or errors reporting that a file or directory
658 that should be created by some other part of the build process could
659 not be found. This type of failure can occur even if, upon
660 inspection, the file or directory does exist after the build has
661 failed, because that part of the build process happened in the wrong
662 order.
663
664 To fix the problem, you need to either satisfy the missing dependency
665 in the Makefile or whatever script produced the Makefile, or (as a
666 workaround) set :term:`PARALLEL_MAKE` to an empty string::
667
668 PARALLEL_MAKE = ""
669
670 For information on parallel Makefile issues, see the
671 ":ref:`dev-manual/debugging:debugging parallel make races`" section.
672
673- *Improper host path usage:* This failure applies to recipes building
674 for the target or ":ref:`ref-classes-nativesdk`" only. The
675 failure occurs when the compilation process uses improper headers,
676 libraries, or other files from the host system when cross-compiling for
677 the target.
678
679 To fix the problem, examine the ``log.do_compile`` file to identify
680 the host paths being used (e.g. ``/usr/include``, ``/usr/lib``, and
681 so forth) and then either add configure options, apply a patch, or do
682 both.
683
684- *Failure to find required libraries/headers:* If a build-time
685 dependency is missing because it has not been declared in
686 :term:`DEPENDS`, or because the
687 dependency exists but the path used by the build process to find the
688 file is incorrect and the configure step did not detect it, the
689 compilation process could fail. For either of these failures, the
690 compilation process notes that files could not be found. In these
691 cases, you need to go back and add additional options to the
692 configure script as well as possibly add additional build-time
693 dependencies to :term:`DEPENDS`.
694
695 Occasionally, it is necessary to apply a patch to the source to
696 ensure the correct paths are used. If you need to specify paths to
697 find files staged into the sysroot from other recipes, use the
698 variables that the OpenEmbedded build system provides (e.g.
699 :term:`STAGING_BINDIR`, :term:`STAGING_INCDIR`, :term:`STAGING_DATADIR`, and so
700 forth).
701
702Installing
703==========
704
705During :ref:`ref-tasks-install`, the task copies the built files along with their
706hierarchy to locations that would mirror their locations on the target
707device. The installation process copies files from the
708``${``\ :term:`S`\ ``}``,
709``${``\ :term:`B`\ ``}``, and
710``${``\ :term:`UNPACKDIR`\ ``}``
711directories to the ``${``\ :term:`D`\ ``}``
712directory to create the structure as it should appear on the target
713system.
714
715How your software is built affects what you must do to be sure your
716software is installed correctly. The following list describes what you
717must do for installation depending on the type of build system used by
718the software being built:
719
720- *Autotools and CMake:* If the software your recipe is building uses
721 Autotools or CMake, the OpenEmbedded build system understands how to
722 install the software. Consequently, you do not have to have a
723 :ref:`ref-tasks-install` task as part of your recipe. You just need to make
724 sure the install portion of the build completes with no issues.
725 However, if you wish to install additional files not already being
726 installed by ``make install``, you should do this using a
727 ``do_install:append`` function using the install command as described
728 in the "Manual" bulleted item later in this list.
729
730- *Other (using* ``make install``\ *)*: You need to define a :ref:`ref-tasks-install`
731 function in your recipe. The function should call
732 ``oe_runmake install`` and will likely need to pass in the
733 destination directory as well. How you pass that path is dependent on
734 how the ``Makefile`` being run is written (e.g. ``DESTDIR=${D}``,
735 ``PREFIX=${D}``, ``INSTALLROOT=${D}``, and so forth).
736
737 For an example recipe using ``make install``, see the
738 ":ref:`dev-manual/new-recipe:building a makefile-based package`" section.
739
740- *Manual:* You need to define a :ref:`ref-tasks-install` function in your
741 recipe. The function must first use ``install -d`` to create the
742 directories under
743 ``${``\ :term:`D`\ ``}``. Once the
744 directories exist, your function can use ``install`` to manually
745 install the built software into the directories.
746
747 You can find more information on ``install`` at
748 https://www.gnu.org/software/coreutils/manual/html_node/install-invocation.html.
749
750For the scenarios that do not use Autotools or CMake, you need to track
751the installation and diagnose and fix any issues until everything
752installs correctly. You need to look in the default location of
753``${D}``, which is ``${WORKDIR}/image``, to be sure your files have been
754installed correctly.
755
756.. note::
757
758 - During the installation process, you might need to modify some of
759 the installed files to suit the target layout. For example, you
760 might need to replace hard-coded paths in an initscript with
761 values of variables provided by the build system, such as
762 replacing ``/usr/bin/`` with ``${bindir}``. If you do perform such
763 modifications during :ref:`ref-tasks-install`, be sure to modify the
764 destination file after copying rather than before copying.
765 Modifying after copying ensures that the build system can
766 re-execute :ref:`ref-tasks-install` if needed.
767
768 - ``oe_runmake install``, which can be run directly or can be run
769 indirectly by the :ref:`ref-classes-autotools` and
770 :ref:`ref-classes-cmake` classes, runs ``make install`` in parallel.
771 Sometimes, a Makefile can have missing dependencies between targets that
772 can result in race conditions. If you experience intermittent failures
773 during :ref:`ref-tasks-install`, you might be able to work around them by
774 disabling parallel Makefile installs by adding the following to the
775 recipe::
776
777 PARALLEL_MAKEINST = ""
778
779 See :term:`PARALLEL_MAKEINST` for additional information.
780
781 - If you need to install one or more custom CMake toolchain files
782 that are supplied by the application you are building, install the
783 files to ``${D}${datadir}/cmake/Modules`` during
784 :ref:`ref-tasks-install`.
785
786Enabling System Services
787========================
788
789If you want to install a service, which is a process that usually starts
790on boot and runs in the background, then you must include some
791additional definitions in your recipe.
792
793If you are adding services and the service initialization script or the
794service file itself is not installed, you must provide for that
795installation in your recipe using a ``do_install:append`` function. If
796your recipe already has a :ref:`ref-tasks-install` function, update the function
797near its end rather than adding an additional ``do_install:append``
798function.
799
800When you create the installation for your services, you need to
801accomplish what is normally done by ``make install``. In other words,
802make sure your installation arranges the output similar to how it is
803arranged on the target system.
804
805The OpenEmbedded build system provides support for starting services two
806different ways:
807
808- *SysVinit:* SysVinit is a system and service manager that manages the
809 init system used to control the very basic functions of your system.
810 The init program is the first program started by the Linux kernel
811 when the system boots. Init then controls the startup, running and
812 shutdown of all other programs.
813
814 To enable a service using SysVinit, your recipe needs to inherit the
815 :ref:`ref-classes-update-rc.d` class. The class helps
816 facilitate safely installing the package on the target.
817
818 You will need to set the
819 :term:`INITSCRIPT_PACKAGES`,
820 :term:`INITSCRIPT_NAME`,
821 and
822 :term:`INITSCRIPT_PARAMS`
823 variables within your recipe.
824
825- *systemd:* System Management Daemon (systemd) was designed to replace
826 SysVinit and to provide enhanced management of services. For more
827 information on systemd, see the systemd homepage at
828 https://freedesktop.org/wiki/Software/systemd/.
829
830 To enable a service using systemd, your recipe needs to inherit the
831 :ref:`ref-classes-systemd` class. See the ``systemd.bbclass`` file
832 located in your :term:`Source Directory` section for more information.
833
834Packaging
835=========
836
837Successful packaging is a combination of automated processes performed
838by the OpenEmbedded build system and some specific steps you need to
839take. The following list describes the process:
840
841- *Splitting Files*: The :ref:`ref-tasks-package` task splits the files produced
842 by the recipe into logical components. Even software that produces a
843 single binary might still have debug symbols, documentation, and
844 other logical components that should be split out. The :ref:`ref-tasks-package`
845 task ensures that files are split up and packaged correctly.
846
847- *Running QA Checks*: The :ref:`ref-classes-insane` class adds a
848 step to the package generation process so that output quality
849 assurance checks are generated by the OpenEmbedded build system. This
850 step performs a range of checks to be sure the build's output is free
851 of common problems that show up during runtime. For information on
852 these checks, see the :ref:`ref-classes-insane` class and
853 the ":ref:`ref-manual/qa-checks:qa error and warning messages`"
854 chapter in the Yocto Project Reference Manual.
855
856- *Hand-Checking Your Packages*: After you build your software, you
857 need to be sure your packages are correct. Examine the
858 ``${``\ :term:`WORKDIR`\ ``}/packages-split``
859 directory and make sure files are where you expect them to be. If you
860 discover problems, you can set
861 :term:`PACKAGES`,
862 :term:`FILES`,
863 ``do_install(:append)``, and so forth as needed.
864
865- *Splitting an Application into Multiple Packages*: If you need to
866 split an application into several packages, see the
867 ":ref:`dev-manual/new-recipe:splitting an application into multiple packages`"
868 section for an example.
869
870- *Installing a Post-Installation Script*: For an example showing how
871 to install a post-installation script, see the
872 ":ref:`dev-manual/new-recipe:post-installation scripts`" section.
873
874- *Marking Package Architecture*: Depending on what your recipe is
875 building and how it is configured, it might be important to mark the
876 packages produced as being specific to a particular machine, or to
877 mark them as not being specific to a particular machine or
878 architecture at all.
879
880 By default, packages apply to any machine with the same architecture
881 as the target machine. When a recipe produces packages that are
882 machine-specific (e.g. the
883 :term:`MACHINE` value is passed
884 into the configure script or a patch is applied only for a particular
885 machine), you should mark them as such by adding the following to the
886 recipe::
887
888 PACKAGE_ARCH = "${MACHINE_ARCH}"
889
890 On the other hand, if the recipe produces packages that do not
891 contain anything specific to the target machine or architecture at
892 all (e.g. recipes that simply package script files or configuration
893 files), you should use the :ref:`ref-classes-allarch` class to
894 do this for you by adding this to your recipe::
895
896 inherit allarch
897
898 Ensuring that the package architecture is correct is not critical
899 while you are doing the first few builds of your recipe. However, it
900 is important in order to ensure that your recipe rebuilds (or does
901 not rebuild) appropriately in response to changes in configuration,
902 and to ensure that you get the appropriate packages installed on the
903 target machine, particularly if you run separate builds for more than
904 one target machine.
905
906Sharing Files Between Recipes
907=============================
908
909Recipes often need to use files provided by other recipes on the build
910host. For example, an application linking to a common library needs
911access to the library itself and its associated headers. The way this
912access is accomplished is by populating a sysroot with files. Each
913recipe has two sysroots in its work directory, one for target files
914(``recipe-sysroot``) and one for files that are native to the build host
915(``recipe-sysroot-native``).
916
917.. note::
918
919 You could find the term "staging" used within the Yocto project
920 regarding files populating sysroots (e.g. the :term:`STAGING_DIR`
921 variable).
922
923Recipes should never populate the sysroot directly (i.e. write files
924into sysroot). Instead, files should be installed into standard
925locations during the
926:ref:`ref-tasks-install` task within
927the ``${``\ :term:`D`\ ``}`` directory. The
928reason for this limitation is that almost all files that populate the
929sysroot are cataloged in manifests in order to ensure the files can be
930removed later when a recipe is either modified or removed. Thus, the
931sysroot is able to remain free from stale files.
932
933A subset of the files installed by the :ref:`ref-tasks-install` task are
934used by the :ref:`ref-tasks-populate_sysroot` task as defined by the
935:term:`SYSROOT_DIRS` variable to automatically populate the sysroot. It
936is possible to modify the list of directories that populate the sysroot.
937The following example shows how you could add the ``/opt`` directory to
938the list of directories within a recipe::
939
940 SYSROOT_DIRS += "/opt"
941
942.. note::
943
944 The `/sysroot-only` is to be used by recipes that generate artifacts
945 that are not included in the target filesystem, allowing them to share
946 these artifacts without needing to use the :term:`DEPLOY_DIR`.
947
948For a more complete description of the :ref:`ref-tasks-populate_sysroot`
949task and its associated functions, see the
950:ref:`staging <ref-classes-staging>` class.
951
952Using Virtual Providers
953=======================
954
955Prior to a build, if you know that several different recipes provide the
956same functionality, you can use a virtual provider (i.e. ``virtual/*``)
957as a placeholder for the actual provider. The actual provider is
958determined at build-time.
959
960A common scenario where a virtual provider is used would be for the kernel
961recipe. Suppose you have three kernel recipes whose :term:`PN` values map to
962``kernel-big``, ``kernel-mid``, and ``kernel-small``. Furthermore, each of
963these recipes in some way uses a :term:`PROVIDES` statement that essentially
964identifies itself as being able to provide ``virtual/kernel``. Here is one way
965through the :ref:`ref-classes-kernel` class::
966
967 PROVIDES += "virtual/kernel"
968
969Any recipe that inherits the :ref:`ref-classes-kernel` class is
970going to utilize a :term:`PROVIDES` statement that identifies that recipe as
971being able to provide the ``virtual/kernel`` item.
972
973Now comes the time to actually build an image and you need a kernel
974recipe, but which one? You can configure your build to call out the
975kernel recipe you want by using the :term:`PREFERRED_PROVIDER` variable. As
976an example, consider the :yocto_git:`x86-base.inc
977</poky/tree/meta/conf/machine/include/x86/x86-base.inc>` include file, which is a
978machine (i.e. :term:`MACHINE`) configuration file. This include file is the
979reason all x86-based machines use the ``linux-yocto`` kernel. Here are the
980relevant lines from the include file::
981
982 PREFERRED_PROVIDER_virtual/kernel ??= "linux-yocto"
983 PREFERRED_VERSION_linux-yocto ??= "4.15%"
984
985When you use a virtual provider, you do not have to "hard code" a recipe
986name as a build dependency. You can use the
987:term:`DEPENDS` variable to state the
988build is dependent on ``virtual/kernel`` for example::
989
990 DEPENDS = "virtual/kernel"
991
992During the build, the OpenEmbedded build system picks
993the correct recipe needed for the ``virtual/kernel`` dependency based on
994the :term:`PREFERRED_PROVIDER` variable. If you want to use the small kernel
995mentioned at the beginning of this section, configure your build as
996follows::
997
998 PREFERRED_PROVIDER_virtual/kernel ??= "kernel-small"
999
1000.. note::
1001
1002 Any recipe that :term:`PROVIDES` a ``virtual/*`` item that is ultimately not
1003 selected through :term:`PREFERRED_PROVIDER` does not get built. Preventing these
1004 recipes from building is usually the desired behavior since this mechanism's
1005 purpose is to select between mutually exclusive alternative providers.
1006
1007The following lists specific examples of virtual providers:
1008
1009- ``virtual/kernel``: Provides the name of the kernel recipe to use
1010 when building a kernel image.
1011
1012- ``virtual/bootloader``: Provides the name of the bootloader to use
1013 when building an image.
1014
1015- ``virtual/libgbm``: Provides ``gbm.pc``.
1016
1017- ``virtual/egl``: Provides ``egl.pc`` and possibly ``wayland-egl.pc``.
1018
1019- ``virtual/libgl``: Provides ``gl.pc`` (i.e. libGL).
1020
1021- ``virtual/libgles1``: Provides ``glesv1_cm.pc`` (i.e. libGLESv1_CM).
1022
1023- ``virtual/libgles2``: Provides ``glesv2.pc`` (i.e. libGLESv2).
1024
1025.. note::
1026
1027 Virtual providers only apply to build time dependencies specified with
1028 :term:`PROVIDES` and :term:`DEPENDS`. They do not apply to runtime
1029 dependencies specified with :term:`RPROVIDES` and :term:`RDEPENDS`.
1030
1031Properly Versioning Pre-Release Recipes
1032=======================================
1033
1034Sometimes the name of a recipe can lead to versioning problems when the
1035recipe is upgraded to a final release. For example, consider the
1036``irssi_0.8.16-rc1.bb`` recipe file in the list of example recipes in
1037the ":ref:`dev-manual/new-recipe:storing and naming the recipe`" section.
1038This recipe is at a release candidate stage (i.e. "rc1"). When the recipe is
1039released, the recipe filename becomes ``irssi_0.8.16.bb``. The version
1040change from ``0.8.16-rc1`` to ``0.8.16`` is seen as a decrease by the
1041build system and package managers, so the resulting packages will not
1042correctly trigger an upgrade.
1043
1044In order to ensure the versions compare properly, the recommended
1045convention is to use a tilde (``~``) character as follows::
1046
1047 PV = 0.8.16~rc1
1048
1049This way ``0.8.16~rc1`` sorts before ``0.8.16``. See the
1050":ref:`contributor-guide/recipe-style-guide:version policy`" section in the
1051Yocto Project and OpenEmbedded Contributor Guide for more details about
1052versioning code corresponding to a pre-release or to a specific Git commit.
1053
1054Post-Installation Scripts
1055=========================
1056
1057Post-installation scripts run immediately after installing a package on
1058the target or during image creation when a package is included in an
1059image. To add a post-installation script to a package, add a
1060``pkg_postinst:``\ `PACKAGENAME`\ ``()`` function to the recipe file
1061(``.bb``) and replace `PACKAGENAME` with the name of the package you want
1062to attach to the ``postinst`` script. To apply the post-installation
1063script to the main package for the recipe, which is usually what is
1064required, specify
1065``${``\ :term:`PN`\ ``}`` in place of
1066PACKAGENAME.
1067
1068A post-installation function has the following structure::
1069
1070 pkg_postinst:PACKAGENAME() {
1071 # Commands to carry out
1072 }
1073
1074The script defined in the post-installation function is called when the
1075root filesystem is created. If the script succeeds, the package is
1076marked as installed.
1077
1078.. note::
1079
1080 Any RPM post-installation script that runs on the target should
1081 return a 0 exit code. RPM does not allow non-zero exit codes for
1082 these scripts, and the RPM package manager will cause the package to
1083 fail installation on the target.
1084
1085Sometimes it is necessary for the execution of a post-installation
1086script to be delayed until the first boot. For example, the script might
1087need to be executed on the device itself. To delay script execution
1088until boot time, you must explicitly mark post installs to defer to the
1089target. You can use ``pkg_postinst_ontarget()`` or call
1090``postinst_intercept delay_to_first_boot`` from ``pkg_postinst()``. Any
1091failure of a ``pkg_postinst()`` script (including exit 1) triggers an
1092error during the
1093:ref:`ref-tasks-rootfs` task.
1094
1095If you have recipes that use ``pkg_postinst`` function and they require
1096the use of non-standard native tools that have dependencies during
1097root filesystem construction, you need to use the
1098:term:`PACKAGE_WRITE_DEPS`
1099variable in your recipe to list these tools. If you do not use this
1100variable, the tools might be missing and execution of the
1101post-installation script is deferred until first boot. Deferring the
1102script to the first boot is undesirable and impossible for read-only
1103root filesystems.
1104
1105.. note::
1106
1107 There is equivalent support for pre-install, pre-uninstall, and post-uninstall
1108 scripts by way of ``pkg_preinst``, ``pkg_prerm``, and ``pkg_postrm``,
1109 respectively. These scrips work in exactly the same way as does
1110 ``pkg_postinst`` with the exception that they run at different times. Also,
1111 because of when they run, they are not applicable to being run at image
1112 creation time like ``pkg_postinst``.
1113
1114Testing
1115=======
1116
1117The final step for completing your recipe is to be sure that the
1118software you built runs correctly. To accomplish runtime testing, add
1119the build's output packages to your image and test them on the target.
1120
1121For information on how to customize your image by adding specific
1122packages, see ":ref:`dev-manual/customizing-images:customizing images`" section.
1123
1124Examples
1125========
1126
1127To help summarize how to write a recipe, this section provides some
1128recipe examples given various scenarios:
1129
1130- `Building a single .c file package`_
1131
1132- `Building a Makefile-based package`_
1133
1134- `Building an Autotooled package`_
1135
1136- `Building a Meson package`_
1137
1138- `Splitting an application into multiple packages`_
1139
1140- `Packaging externally produced binaries`_
1141
1142Building a Single .c File Package
1143---------------------------------
1144
1145Building an application from a single file that is stored locally (e.g. under
1146``files``) requires a recipe that has the file listed in the :term:`SRC_URI`
1147variable. Additionally, you need to manually write the :ref:`ref-tasks-compile`
1148and :ref:`ref-tasks-install` tasks. The :term:`S` variable defines the
1149directory containing the source code, which is set to :term:`UNPACKDIR` in this
1150case --- the directory BitBake uses for the build::
1151
1152 SUMMARY = "Simple helloworld application"
1153 SECTION = "examples"
1154 LICENSE = "MIT"
1155 LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
1156
1157 SRC_URI = "file://helloworld.c"
1158
1159 S = "${UNPACKDIR}"
1160
1161 do_compile() {
1162 ${CC} ${LDFLAGS} helloworld.c -o helloworld
1163 }
1164
1165 do_install() {
1166 install -d ${D}${bindir}
1167 install -m 0755 helloworld ${D}${bindir}
1168 }
1169
1170By default, the ``helloworld``, ``helloworld-dbg``, and ``helloworld-dev`` packages
1171are built. For information on how to customize the packaging process, see the
1172":ref:`dev-manual/new-recipe:splitting an application into multiple packages`"
1173section.
1174
1175Building a Makefile-Based Package
1176---------------------------------
1177
1178Applications built with GNU ``make`` require a recipe that has the source archive
1179listed in :term:`SRC_URI`. You do not need to add a :ref:`ref-tasks-compile`
1180step since by default BitBake starts the ``make`` command to compile the
1181application. If you need additional ``make`` options, you should store them in
1182the :term:`EXTRA_OEMAKE` or :term:`PACKAGECONFIG_CONFARGS` variables. BitBake
1183passes these options into the GNU ``make`` invocation. Note that a
1184:ref:`ref-tasks-install` task is still required. Otherwise, BitBake runs an
1185empty :ref:`ref-tasks-install` task by default.
1186
1187Some applications might require extra parameters to be passed to the
1188compiler. For example, the application might need an additional header
1189path. You can accomplish this by adding to the :term:`CFLAGS` variable. The
1190following example shows this::
1191
1192 CFLAGS:prepend = "-I ${S}/include "
1193
1194In the following example, ``lz4`` is a makefile-based package::
1195
1196 SUMMARY = "Extremely Fast Compression algorithm"
1197 DESCRIPTION = "LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems."
1198 HOMEPAGE = "https://github.com/lz4/lz4"
1199
1200 LICENSE = "BSD-2-Clause | GPL-2.0-only"
1201 LIC_FILES_CHKSUM = "file://lib/LICENSE;md5=ebc2ea4814a64de7708f1571904b32cc \
1202 file://programs/COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263 \
1203 file://LICENSE;md5=d57c0d21cb917fb4e0af2454aa48b956 \
1204 "
1205
1206 PE = "1"
1207
1208 SRCREV = "d44371841a2f1728a3f36839fd4b7e872d0927d3"
1209
1210 SRC_URI = "git://github.com/lz4/lz4.git;branch=release;protocol=https \
1211 file://CVE-2021-3520.patch \
1212 "
1213 UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>.*)"
1214
1215 CVE_STATUS[CVE-2014-4715] = "fixed-version: Fixed in r118, which is larger than the current version"
1216
1217 EXTRA_OEMAKE = "PREFIX=${prefix} CC='${CC}' CFLAGS='${CFLAGS}' DESTDIR=${D} LIBDIR=${libdir} INCLUDEDIR=${includedir} BUILD_STATIC=no"
1218
1219 do_install() {
1220 oe_runmake install
1221 }
1222
1223 BBCLASSEXTEND = "native nativesdk"
1224
1225Building an Autotooled Package
1226------------------------------
1227
1228Applications built with the Autotools such as ``autoconf`` and ``automake``
1229require a recipe that has a source archive listed in :term:`SRC_URI` and also
1230inherit the :ref:`ref-classes-autotools` class, which contains the definitions
1231of all the steps needed to build an Autotool-based application. The result of
1232the build is automatically packaged. And, if the application uses NLS for
1233localization, packages with local information are generated (one package per
1234language). Here is one example: (``hello_2.3.bb``)::
1235
1236 SUMMARY = "GNU Helloworld application"
1237 SECTION = "examples"
1238 LICENSE = "GPL-2.0-or-later"
1239 LIC_FILES_CHKSUM = "file://COPYING;md5=751419260aa954499f7abaabaa882bbe"
1240
1241 SRC_URI = "${GNU_MIRROR}/hello/hello-${PV}.tar.gz"
1242
1243 inherit autotools gettext
1244
1245The variable :term:`LIC_FILES_CHKSUM` is used to track source license changes
1246as described in the ":ref:`dev-manual/licenses:tracking license changes`"
1247section in the Yocto Project Overview and Concepts Manual. You can quickly
1248create Autotool-based recipes in a manner similar to the previous example.
1249
1250.. _ref-building-meson-package:
1251
1252Building a Meson Package
1253------------------------
1254
1255Applications built with the `Meson build system <https://mesonbuild.com/>`__
1256just need a recipe that has sources described in :term:`SRC_URI` and inherits
1257the :ref:`ref-classes-meson` class.
1258
1259The :oe_git:`ipcalc recipe </meta-openembedded/tree/meta-networking/recipes-support/ipcalc>`
1260is a simple example of an application without dependencies::
1261
1262 SUMMARY = "Tool to assist in network address calculations for IPv4 and IPv6."
1263 HOMEPAGE = "https://gitlab.com/ipcalc/ipcalc"
1264
1265 SECTION = "net"
1266
1267 LICENSE = "GPL-2.0-only"
1268 LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
1269
1270 SRC_URI = "git://gitlab.com/ipcalc/ipcalc.git;protocol=https;branch=master"
1271 SRCREV = "4c4261a47f355946ee74013d4f5d0494487cc2d6"
1272
1273 inherit meson
1274
1275Applications with dependencies are likely to inherit the
1276:ref:`ref-classes-pkgconfig` class, as ``pkg-config`` is the default method
1277used by Meson to find dependencies and compile applications against them.
1278
1279Splitting an Application into Multiple Packages
1280-----------------------------------------------
1281
1282You can use the variables :term:`PACKAGES` and :term:`FILES` to split an
1283application into multiple packages.
1284
1285Here is an example that uses the ``libxpm`` recipe. By default,
1286this recipe generates a single package that contains the library along
1287with a few binaries. You can modify the recipe to split the binaries
1288into separate packages::
1289
1290 require xorg-lib-common.inc
1291
1292 SUMMARY = "Xpm: X Pixmap extension library"
1293 LICENSE = "MIT"
1294 LIC_FILES_CHKSUM = "file://COPYING;md5=51f4270b012ecd4ab1a164f5f4ed6cf7"
1295 DEPENDS += "libxext libsm libxt"
1296 PE = "1"
1297
1298 XORG_PN = "libXpm"
1299
1300 PACKAGES =+ "sxpm cxpm"
1301 FILES:cxpm = "${bindir}/cxpm"
1302 FILES:sxpm = "${bindir}/sxpm"
1303
1304In the previous example, we want to ship the ``sxpm`` and ``cxpm``
1305binaries in separate packages. Since ``bindir`` would be packaged into
1306the main :term:`PN` package by default, we prepend the :term:`PACKAGES` variable
1307so additional package names are added to the start of list. This results
1308in the extra ``FILES:*`` variables then containing information that
1309define which files and directories go into which packages. Files
1310included by earlier packages are skipped by latter packages. Thus, the
1311main :term:`PN` package does not include the above listed files.
1312
1313Packaging Externally Produced Binaries
1314--------------------------------------
1315
1316Sometimes, you need to add pre-compiled binaries to an image. For
1317example, suppose that there are binaries for proprietary code,
1318created by a particular division of a company. Your part of the company
1319needs to use those binaries as part of an image that you are building
1320using the OpenEmbedded build system. Since you only have the binaries
1321and not the source code, you cannot use a typical recipe that expects to
1322fetch the source specified in
1323:term:`SRC_URI` and then compile it.
1324
1325One method is to package the binaries and then install them as part of
1326the image. Generally, it is not a good idea to package binaries since,
1327among other things, it can hinder the ability to reproduce builds and
1328could lead to compatibility problems with ABI in the future. However,
1329sometimes you have no choice.
1330
1331The easiest solution is to create a recipe that uses the
1332:ref:`ref-classes-bin-package` class and to be sure that you are using default
1333locations for build artifacts. In most cases, the
1334:ref:`ref-classes-bin-package` class handles "skipping" the configure and
1335compile steps as well as sets things up to grab packages from the appropriate
1336area. In particular, this class sets ``noexec`` on both the
1337:ref:`ref-tasks-configure` and :ref:`ref-tasks-compile` tasks, sets
1338``FILES:${PN}`` to "/" so that it picks up all files, and sets up a
1339:ref:`ref-tasks-install` task, which effectively copies all files from ``${S}``
1340to ``${D}``. The :ref:`ref-classes-bin-package` class works well when the files
1341extracted into ``${S}`` are already laid out in the way they should be laid out
1342on the target. For more information on these variables, see the :term:`FILES`,
1343:term:`PN`, :term:`S`, and :term:`D` variables in the Yocto Project Reference
1344Manual's variable glossary.
1345
1346.. note::
1347
1348 - Using :term:`DEPENDS` is a good
1349 idea even for components distributed in binary form, and is often
1350 necessary for shared libraries. For a shared library, listing the
1351 library dependencies in :term:`DEPENDS` makes sure that the libraries
1352 are available in the staging sysroot when other recipes link
1353 against the library, which might be necessary for successful
1354 linking.
1355
1356 - Using :term:`DEPENDS` also allows runtime dependencies between
1357 packages to be added automatically. See the
1358 ":ref:`overview-manual/concepts:automatically added runtime dependencies`"
1359 section in the Yocto Project Overview and Concepts Manual for more
1360 information.
1361
1362If you cannot use the :ref:`ref-classes-bin-package` class, you need to be sure you are
1363doing the following:
1364
1365- Create a recipe where the
1366 :ref:`ref-tasks-configure` and
1367 :ref:`ref-tasks-compile` tasks do
1368 nothing: It is usually sufficient to just not define these tasks in
1369 the recipe, because the default implementations do nothing unless a
1370 Makefile is found in
1371 ``${``\ :term:`S`\ ``}``.
1372
1373 If ``${S}`` might contain a Makefile, or if you inherit some class
1374 that replaces :ref:`ref-tasks-configure` and :ref:`ref-tasks-compile` with custom
1375 versions, then you can use the
1376 ``[``\ :ref:`noexec <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ``]``
1377 flag to turn the tasks into no-ops, as follows::
1378
1379 do_configure[noexec] = "1"
1380 do_compile[noexec] = "1"
1381
1382 Unlike :ref:`bitbake-user-manual/bitbake-user-manual-metadata:deleting a task`,
1383 using the flag preserves the dependency chain from the :ref:`ref-tasks-fetch`,
1384 :ref:`ref-tasks-unpack`, and :ref:`ref-tasks-patch` tasks to the
1385 :ref:`ref-tasks-install` task.
1386
1387- Make sure your :ref:`ref-tasks-install` task installs the binaries
1388 appropriately.
1389
1390- Ensure that you set up :term:`FILES`
1391 (usually
1392 ``FILES:${``\ :term:`PN`\ ``}``) to
1393 point to the files you have installed, which of course depends on
1394 where you have installed them and whether those files are in
1395 different locations than the defaults.
1396
1397As a basic example of a :ref:`ref-classes-bin-package`-style recipe, consider
1398this snippet from the
1399:oe_git:`wireless-regdb </openembedded-core/tree/meta/recipes-kernel/wireless-regdb>`
1400recipe file, which fetches a single tarball of binary content and manually
1401installs with no need for any configuration or compilation::
1402
1403 SRC_URI = "https://www.kernel.org/pub/software/network/${BPN}/${BP}.tar.xz"
1404 SRC_URI[sha256sum] = "57f8e7721cf5a880c13ae0c202edbb21092a060d45f9e9c59bcd2a8272bfa456"
1405
1406 inherit bin_package allarch
1407
1408 do_install() {
1409 install -d -m0755 ${D}${nonarch_libdir}/crda
1410 install -d -m0755 ${D}${sysconfdir}/wireless-regdb/pubkeys
1411 install -m 0644 regulatory.bin ${D}${nonarch_libdir}/crda/regulatory.bin
1412 install -m 0644 wens.key.pub.pem ${D}${sysconfdir}/wireless-regdb/pubkeys/wens.key.pub.pem
1413 install -m 0644 -D regulatory.db ${D}${nonarch_base_libdir}/firmware/regulatory.db
1414 install -m 0644 regulatory.db.p7s ${D}${nonarch_base_libdir}/firmware/regulatory.db.p7s
1415 }
1416
1417Following Recipe Style Guidelines
1418=================================
1419
1420When writing recipes, it is good to conform to existing style guidelines.
1421See the ":doc:`/contributor-guide/recipe-style-guide`" in the Yocto Project
1422and OpenEmbedded Contributor Guide for reference.
1423
1424It is common for existing recipes to deviate a bit from this style.
1425However, aiming for at least a consistent style is a good idea. Some
1426practices, such as omitting spaces around ``=`` operators in assignments
1427or ordering recipe components in an erratic way, are widely seen as poor
1428style.
1429
1430Recipe Syntax
1431=============
1432
1433Understanding recipe file syntax is important for writing recipes. The
1434following list overviews the basic items that make up a BitBake recipe
1435file. For more complete BitBake syntax descriptions, see the
1436":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`"
1437chapter of the BitBake User Manual.
1438
1439- *Variable Assignments and Manipulations:* Variable assignments allow
1440 a value to be assigned to a variable. The assignment can be static
1441 text or might include the contents of other variables. In addition to
1442 the assignment, appending and prepending operations are also
1443 supported.
1444
1445 The following example shows some of the ways you can use variables in
1446 recipes::
1447
1448 S = "${UNPACKDIR}/postfix-${PV}"
1449 CFLAGS += "-DNO_ASM"
1450 CFLAGS:append = " --enable-important-feature"
1451
1452- *Functions:* Functions provide a series of actions to be performed.
1453 You usually use functions to override the default implementation of a
1454 task function or to complement a default function (i.e. append or
1455 prepend to an existing function). Standard functions use ``sh`` shell
1456 syntax, although access to OpenEmbedded variables and internal
1457 methods are also available.
1458
1459 Here is an example function from the ``sed`` recipe::
1460
1461 do_install () {
1462 autotools_do_install
1463 install -d ${D}${base_bindir}
1464 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
1465 rmdir ${D}${bindir}/
1466 }
1467
1468 It is
1469 also possible to implement new functions that are called between
1470 existing tasks as long as the new functions are not replacing or
1471 complementing the default functions. You can implement functions in
1472 Python instead of shell. Both of these options are not seen in the
1473 majority of recipes.
1474
1475- *Keywords:* BitBake recipes use only a few keywords. You use keywords
1476 to include common functions (``inherit``), load parts of a recipe
1477 from other files (``include`` and ``require``) and export variables
1478 to the environment (``export``).
1479
1480 The following example shows the use of some of these keywords::
1481
1482 export POSTCONF = "${STAGING_BINDIR}/postconf"
1483 inherit autoconf
1484 require otherfile.inc
1485
1486- *Comments (#):* Any lines that begin with the hash character (``#``)
1487 are treated as comment lines and are ignored::
1488
1489 # This is a comment
1490
1491This next list summarizes the most important and most commonly used
1492parts of the recipe syntax. For more information on these parts of the
1493syntax, you can reference the
1494":doc:`bitbake:bitbake-user-manual/bitbake-user-manual-metadata`" chapter
1495in the BitBake User Manual.
1496
1497- *Line Continuation (\\):* Use the backward slash (``\``) character to
1498 split a statement over multiple lines. Place the slash character at
1499 the end of the line that is to be continued on the next line::
1500
1501 VAR = "A really long \
1502 line"
1503
1504 .. note::
1505
1506 You cannot have any characters including spaces or tabs after the
1507 slash character.
1508
1509- *Using Variables (${VARNAME}):* Use the ``${VARNAME}`` syntax to
1510 access the contents of a variable::
1511
1512 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
1513
1514 .. note::
1515
1516 It is important to understand that the value of a variable
1517 expressed in this form does not get substituted automatically. The
1518 expansion of these expressions happens on-demand later (e.g.
1519 usually when a function that makes reference to the variable
1520 executes). This behavior ensures that the values are most
1521 appropriate for the context in which they are finally used. On the
1522 rare occasion that you do need the variable expression to be
1523 expanded immediately, you can use the
1524 :=
1525 operator instead of
1526 =
1527 when you make the assignment, but this is not generally needed.
1528
1529- *Quote All Assignments ("value"):* Use double quotes around values in
1530 all variable assignments (e.g. ``"value"``). Here is an example::
1531
1532 VAR1 = "${OTHERVAR}"
1533 VAR2 = "The version is ${PV}"
1534
1535- *Conditional Assignment (?=):* Conditional assignment is used to
1536 assign a value to a variable, but only when the variable is currently
1537 unset. Use the question mark followed by the equal sign (``?=``) to
1538 make a "soft" assignment used for conditional assignment. Typically,
1539 "soft" assignments are used in the ``local.conf`` file for variables
1540 that are allowed to come through from the external environment.
1541
1542 Here is an example where ``VAR1`` is set to "New value" if it is
1543 currently empty. However, if ``VAR1`` has already been set, it
1544 remains unchanged::
1545
1546 VAR1 ?= "New value"
1547
1548 In this next example, ``VAR1`` is left with the value "Original value"::
1549
1550 VAR1 = "Original value"
1551 VAR1 ?= "New value"
1552
1553- *Appending (+=):* Use the plus character followed by the equals sign
1554 (``+=``) to append values to existing variables.
1555
1556 .. note::
1557
1558 This operator adds a space between the existing content of the
1559 variable and the new content.
1560
1561 Here is an example::
1562
1563 SRC_URI += "file://fix-makefile.patch"
1564
1565- *Prepending (=+):* Use the equals sign followed by the plus character
1566 (``=+``) to prepend values to existing variables.
1567
1568 .. note::
1569
1570 This operator adds a space between the new content and the
1571 existing content of the variable.
1572
1573 Here is an example::
1574
1575 VAR =+ "Starts"
1576
1577- *Appending (:append):* Use the ``:append`` operator to append values
1578 to existing variables. This operator does not add any additional
1579 space. Also, the operator is applied after all the ``+=``, and ``=+``
1580 operators have been applied and after all ``=`` assignments have
1581 occurred. This means that if ``:append`` is used in a recipe, it can
1582 only be overridden by another layer using the special ``:remove``
1583 operator, which in turn will prevent further layers from adding it back.
1584
1585 The following example shows the space being explicitly added to the
1586 start to ensure the appended value is not merged with the existing
1587 value::
1588
1589 CFLAGS:append = " --enable-important-feature"
1590
1591 You can also use
1592 the ``:append`` operator with overrides, which results in the actions
1593 only being performed for the specified target or machine::
1594
1595 CFLAGS:append:sh4 = " --enable-important-sh4-specific-feature"
1596
1597- *Prepending (:prepend):* Use the ``:prepend`` operator to prepend
1598 values to existing variables. This operator does not add any
1599 additional space. Also, the operator is applied after all the ``+=``,
1600 and ``=+`` operators have been applied and after all ``=``
1601 assignments have occurred.
1602
1603 The following example shows the space being explicitly added to the
1604 end to ensure the prepended value is not merged with the existing
1605 value::
1606
1607 CFLAGS:prepend = "-I${S}/myincludes "
1608
1609 You can also use the
1610 ``:prepend`` operator with overrides, which results in the actions
1611 only being performed for the specified target or machine::
1612
1613 CFLAGS:prepend:sh4 = "-I${S}/myincludes "
1614
1615- *Overrides:* You can use overrides to set a value conditionally,
1616 typically based on how the recipe is being built. For example, to set
1617 the :term:`KBRANCH` variable's
1618 value to "standard/base" for any target
1619 :term:`MACHINE`, except for
1620 qemuarm where it should be set to "standard/arm-versatile-926ejs",
1621 you would do the following::
1622
1623 KBRANCH = "standard/base"
1624 KBRANCH:qemuarm = "standard/arm-versatile-926ejs"
1625
1626 Overrides are also used to separate
1627 alternate values of a variable in other situations. For example, when
1628 setting variables such as
1629 :term:`FILES` and
1630 :term:`RDEPENDS` that are
1631 specific to individual packages produced by a recipe, you should
1632 always use an override that specifies the name of the package.
1633
1634- *Indentation:* Use spaces for indentation rather than tabs. For
1635 shell functions, both currently work. However, it is a policy
1636 decision of the Yocto Project to use tabs in shell functions. Realize
1637 that some layers have a policy to use spaces for all indentation.
1638
1639- *Using Python for Complex Operations:* For more advanced processing,
1640 it is possible to use Python code during variable assignments (e.g.
1641 search and replacement on a variable).
1642
1643 You indicate Python code using the ``${@python_code}`` syntax for the
1644 variable assignment::
1645
1646 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
1647
1648- *Shell Function Syntax:* Write shell functions as if you were writing
1649 a shell script when you describe a list of actions to take. You
1650 should ensure that your script works with a generic ``sh`` and that
1651 it does not require any ``bash`` or other shell-specific
1652 functionality. The same considerations apply to various system
1653 utilities (e.g. ``sed``, ``grep``, ``awk``, and so forth) that you
1654 might wish to use. If in doubt, you should check with multiple
1655 implementations --- including those from BusyBox.
1656
diff --git a/documentation/dev-manual/packages.rst b/documentation/dev-manual/packages.rst
deleted file mode 100644
index 8bd48c8e8f..0000000000
--- a/documentation/dev-manual/packages.rst
+++ /dev/null
@@ -1,1149 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working with Packages
4*********************
5
6This section describes a few tasks that involve packages:
7
8- :ref:`dev-manual/packages:excluding packages from an image`
9
10- :ref:`dev-manual/packages:incrementing a package version`
11
12- :ref:`dev-manual/packages:handling optional module packaging`
13
14- :ref:`dev-manual/packages:using runtime package management`
15
16- :ref:`dev-manual/packages:generating and using signed packages`
17
18- :ref:`Setting up and running package test
19 (ptest) <test-manual/ptest:testing packages with ptest>`
20
21- :ref:`dev-manual/packages:creating node package manager (npm) packages`
22
23- :ref:`dev-manual/packages:adding custom metadata to packages`
24
25Excluding Packages from an Image
26================================
27
28You might find it necessary to prevent specific packages from being
29installed into an image. If so, you can use several variables to direct
30the build system to essentially ignore installing recommended packages
31or to not install a package at all.
32
33The following list introduces variables you can use to prevent packages
34from being installed into your image. Each of these variables only works
35with IPK and RPM package types, not for Debian packages.
36Also, you can use these variables from your ``local.conf`` file
37or attach them to a specific image recipe by using a recipe name
38override. For more detail on the variables, see the descriptions in the
39Yocto Project Reference Manual's glossary chapter.
40
41- :term:`BAD_RECOMMENDATIONS`:
42 Use this variable to specify "recommended-only" packages that you do
43 not want installed.
44
45- :term:`NO_RECOMMENDATIONS`:
46 Use this variable to prevent all "recommended-only" packages from
47 being installed.
48
49- :term:`PACKAGE_EXCLUDE`:
50 Use this variable to prevent specific packages from being installed
51 regardless of whether they are "recommended-only" or not. You need to
52 realize that the build process could fail with an error when you
53 prevent the installation of a package whose presence is required by
54 an installed package.
55
56Incrementing a Package Version
57==============================
58
59This section provides some background on how binary package versioning
60is accomplished and presents some of the services, variables, and
61terminology involved.
62
63In order to understand binary package versioning, you need to consider
64the following:
65
66- Binary Package: The binary package that is eventually built and
67 installed into an image.
68
69- Binary Package Version: The binary package version is composed of two
70 components --- a version and a revision.
71
72 .. note::
73
74 Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
75 but this discussion for the most part ignores :term:`PE`.
76
77 The version and revision are taken from the
78 :term:`PV` and
79 :term:`PR` variables, respectively.
80
81- :term:`PV`: The recipe version. :term:`PV` represents the version of the
82 software being packaged. Do not confuse :term:`PV` with the binary
83 package version.
84
85- :term:`PR`: The recipe revision.
86
87- :yocto_wiki:`PR Service </PR_Service>`: A
88 network-based service that helps automate keeping package feeds
89 compatible with existing package manager applications such as RPM,
90 APT, and OPKG.
91
92Whenever the binary package content changes, the binary package version
93must change. Changing the binary package version is accomplished by
94changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
95values occurs one of two ways:
96
97- Automatically using a Package Revision Service (PR Service).
98
99- Manually incrementing the :term:`PR` and/or :term:`PV` variables.
100
101Given a primary challenge of any build system and its users is how to
102maintain a package feed that is compatible with existing package manager
103applications such as RPM, APT, and OPKG, using an automated system is
104much preferred over a manual system. In either system, the main
105requirement is that binary package version numbering increases in a
106linear fashion and that there is a number of version components that
107support that linear progression. For information on how to ensure
108package revisioning remains linear, see the
109":ref:`dev-manual/packages:automatically incrementing a package version number`"
110section.
111
112The following three sections provide related information on the PR
113Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
114how to ensure binary package revisioning remains linear.
115
116Working With a PR Service
117-------------------------
118
119As mentioned, attempting to maintain revision numbers in the
120:term:`Metadata` is error prone, inaccurate,
121and causes problems for people submitting recipes. Conversely, the PR
122Service automatically generates increasing numbers, particularly the
123revision field, which removes the human element.
124
125.. note::
126
127 For additional information on using a PR Service, you can see the
128 :yocto_wiki:`PR Service </PR_Service>` wiki page.
129
130The Yocto Project uses variables in order of decreasing priority to
131facilitate revision numbering (i.e.
132:term:`PE`,
133:term:`PV`, and
134:term:`PR` for epoch, version, and
135revision, respectively). The values are highly dependent on the policies
136and procedures of a given distribution and package feed.
137
138Because the OpenEmbedded build system uses
139":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
140unique to a given build, the build system knows when to rebuild
141packages. All the inputs into a given task are represented by a
142signature, which can trigger a rebuild when different. Thus, the build
143system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
144trigger a rebuild. The signatures, however, can be used to generate
145these values.
146
147The PR Service works with both ``OEBasic`` and ``OEBasicHash``
148generators. The value of :term:`PR` bumps when the checksum changes and the
149different generator mechanisms change signatures under different
150circumstances.
151
152As implemented, the build system includes values from the PR Service
153into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
154becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
155:term:`PR` values to be used for whatever reasons, which include manual
156:term:`PR` bumps, should it be necessary.
157
158By default, the PR Service is not enabled or running. Thus, the packages
159generated are just "self consistent". The build system adds and removes
160packages and there are no guarantees about upgrade paths but images will
161be consistent and correct with the latest changes.
162
163The simplest form for a PR Service is for a single host development system
164that builds the package feed (building system). For this scenario, you can
165enable a local PR Service by setting :term:`PRSERV_HOST` in your
166``local.conf`` file in the :term:`Build Directory`::
167
168 PRSERV_HOST = "localhost:0"
169
170Once the service is started, packages will automatically
171get increasing :term:`PR` values and BitBake takes care of starting and
172stopping the server.
173
174If you have a more complex setup where multiple host development systems
175work against a common, shared package feed, you have a single PR Service
176running and it is connected to each building system. For this scenario,
177you need to start the PR Service using the ``bitbake-prserv`` command::
178
179 bitbake-prserv --host ip --port port --start
180
181In addition to
182hand-starting the service, you need to update the ``local.conf`` file of
183each building system as described earlier so each system points to the
184server and port.
185
186It is also recommended you use build history, which adds some sanity
187checks to binary package versions, in conjunction with the server that
188is running the PR Service. To enable build history, add the following to
189each building system's ``local.conf`` file::
190
191 # It is recommended to activate "buildhistory" for testing the PR service
192 INHERIT += "buildhistory"
193 BUILDHISTORY_COMMIT = "1"
194
195For information on build
196history, see the
197":ref:`dev-manual/build-quality:maintaining build output quality`" section.
198
199.. note::
200
201 The OpenEmbedded build system does not maintain :term:`PR` information as
202 part of the shared state (sstate) packages. If you maintain an sstate
203 feed, it's expected that either all your building systems that
204 contribute to the sstate feed use a shared PR service, or you do not
205 run a PR service on any of your building systems.
206
207 That's because if you had multiple machines sharing a PR service but
208 not their sstate feed, you could end up with "diverging" hashes for
209 the same output artefacts. When presented to the share PR service,
210 each would be considered as new and would increase the revision
211 number, causing many unnecessary package upgrades.
212
213 For more information on shared state, see the
214 ":ref:`overview-manual/concepts:shared state cache`"
215 section in the Yocto Project Overview and Concepts Manual.
216
217Manually Bumping PR
218-------------------
219
220The alternative to setting up a PR Service is to manually "bump" the
221:term:`PR` variable.
222
223If a committed change results in changing the package output, then the
224value of the :term:`PR` variable needs to be increased (or "bumped") as part of
225that commit. For new recipes you should add the :term:`PR` variable and set
226its initial value equal to "r0", which is the default. Even though the
227default value is "r0", the practice of adding it to a new recipe makes
228it harder to forget to bump the variable when you make changes to the
229recipe in future.
230
231Usually, version increases occur only to binary packages. However, if
232for some reason :term:`PV` changes but does not increase, you can increase
233the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
234"0".
235
236Binary package version numbering strives to follow the `Debian Version
237Field Policy
238Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
239These guidelines define how versions are compared and what "increasing"
240a version means.
241
242Automatically Incrementing a Package Version Number
243---------------------------------------------------
244
245When fetching a repository, BitBake uses the
246:term:`SRCREV` variable to determine
247the specific source code revision from which to build. You set the
248:term:`SRCREV` variable to
249:term:`AUTOREV` to cause the
250OpenEmbedded build system to automatically use the latest revision of
251the software::
252
253 SRCREV = "${AUTOREV}"
254
255Furthermore, you need to include a ``+`` sign in :term:`PV` in order to
256automatically update the version whenever the revision of the source
257code changes. Here is an example::
258
259 PV = "1.0+git"
260
261The OpenEmbedded build system will automatically add the source control
262information to the end of the variable :term:`PKGV`, in this format::
263
264 AUTOINC+source_code_revision
265
266The build system replaces the ``AUTOINC``
267with a number. The number used depends on the state of the PR Service:
268
269- If PR Service is enabled, the build system increments the number,
270 which is similar to the behavior of
271 :term:`PR`. This behavior results in
272 linearly increasing package versions, which is desirable. Here is an
273 example:
274
275 .. code-block:: none
276
277 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
278 hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk
279
280- If PR Service is not enabled, the build system replaces the
281 ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
282 changing the package version since the source revision is included.
283 However, package versions are not increased linearly. Here is an
284 example:
285
286 .. code-block:: none
287
288 hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
289 hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk
290
291In summary, the OpenEmbedded build system does not track the history of
292binary package versions for this purpose. ``AUTOINC``, in this case, is
293comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
294package version is simply replaced by "0". If PR server is enabled, the
295build system keeps track of the package versions and bumps the number
296when the package revision changes.
297
298Handling Optional Module Packaging
299==================================
300
301Many pieces of software split functionality into optional modules (or
302plugins) and the plugins that are built might depend on configuration
303options. To avoid having to duplicate the logic that determines what
304modules are available in your recipe or to avoid having to package each
305module by hand, the OpenEmbedded build system provides functionality to
306handle module packaging dynamically.
307
308To handle optional module packaging, you need to do two things:
309
310- Ensure the module packaging is actually done.
311
312- Ensure that any dependencies on optional modules from other recipes
313 are satisfied by your recipe.
314
315Making Sure the Packaging is Done
316---------------------------------
317
318To ensure the module packaging actually gets done, you use the
319``do_split_packages`` function within the ``populate_packages`` Python
320function in your recipe. The ``do_split_packages`` function searches for
321a pattern of files or directories under a specified path and creates a
322package for each one it finds by appending to the
323:term:`PACKAGES` variable and
324setting the appropriate values for ``FILES:packagename``,
325``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
326Here is an example from the ``lighttpd`` recipe::
327
328 python populate_packages:prepend () {
329 lighttpd_libdir = d.expand('${libdir}')
330 do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
331 'lighttpd-module-%s', 'Lighttpd module for %s',
332 extra_depends='')
333 }
334
335The previous example specifies a number of things in the call to
336``do_split_packages``.
337
338- A directory within the files installed by your recipe through
339 :ref:`ref-tasks-install` in which to search.
340
341- A regular expression used to match module files in that directory. In
342 the example, note the parentheses () that mark the part of the
343 expression from which the module name should be derived.
344
345- A pattern to use for the package names.
346
347- A description for each package.
348
349- An empty string for ``extra_depends``, which disables the default
350 dependency on the main ``lighttpd`` package. Thus, if a file in
351 ``${libdir}`` called ``mod_alias.so`` is found, a package called
352 ``lighttpd-module-alias`` is created for it and the
353 :term:`DESCRIPTION` is set to
354 "Lighttpd module for alias".
355
356Often, packaging modules is as simple as the previous example. However,
357there are more advanced options that you can use within
358``do_split_packages`` to modify its behavior. And, if you need to, you
359can add more logic by specifying a hook function that is called for each
360package. It is also perfectly acceptable to call ``do_split_packages``
361multiple times if you have more than one set of modules to package.
362
363For more examples that show how to use ``do_split_packages``, see the
364``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
365directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
366also find examples in ``meta/classes-recipe/kernel.bbclass``.
367
368Here is a reference that shows ``do_split_packages`` mandatory and
369optional arguments::
370
371 Mandatory arguments
372
373 root
374 The path in which to search
375 file_regex
376 Regular expression to match searched files.
377 Use parentheses () to mark the part of this
378 expression that should be used to derive the
379 module name (to be substituted where %s is
380 used in other function arguments as noted below)
381 output_pattern
382 Pattern to use for the package names. Must
383 include %s.
384 description
385 Description to set for each package. Must
386 include %s.
387
388 Optional arguments
389
390 postinst
391 Postinstall script to use for all packages
392 (as a string)
393 recursive
394 True to perform a recursive search --- default
395 False
396 hook
397 A hook function to be called for every match.
398 The function will be called with the following
399 arguments (in the order listed):
400
401 f
402 Full path to the file/directory match
403 pkg
404 The package name
405 file_regex
406 As above
407 output_pattern
408 As above
409 modulename
410 The module name derived using file_regex
411 extra_depends
412 Extra runtime dependencies (RDEPENDS) to be
413 set for all packages. The default value of None
414 causes a dependency on the main package
415 (${PN}) --- if you do not want this, pass empty
416 string '' for this parameter.
417 aux_files_pattern
418 Extra item(s) to be added to FILES for each
419 package. Can be a single string item or a list
420 of strings for multiple items. Must include %s.
421 postrm
422 postrm script to use for all packages (as a
423 string)
424 allow_dirs
425 True to allow directories to be matched -
426 default False
427 prepend
428 If True, prepend created packages to PACKAGES
429 instead of the default False which appends them
430 match_path
431 match file_regex on the whole relative path to
432 the root rather than just the filename
433 aux_files_pattern_verbatim
434 Extra item(s) to be added to FILES for each
435 package, using the actual derived module name
436 rather than converting it to something legal
437 for a package name. Can be a single string item
438 or a list of strings for multiple items. Must
439 include %s.
440 allow_links
441 True to allow symlinks to be matched --- default
442 False
443 summary
444 Summary to set for each package. Must include %s;
445 defaults to description if not set.
446
447
448
449Satisfying Dependencies
450-----------------------
451
452The second part for handling optional module packaging is to ensure that
453any dependencies on optional modules from other recipes are satisfied by
454your recipe. You can be sure these dependencies are satisfied by using
455the :term:`PACKAGES_DYNAMIC`
456variable. Here is an example that continues with the ``lighttpd`` recipe
457shown earlier::
458
459 PACKAGES_DYNAMIC = "lighttpd-module-.*"
460
461The name
462specified in the regular expression can of course be anything. In this
463example, it is ``lighttpd-module-`` and is specified as the prefix to
464ensure that any :term:`RDEPENDS` and
465:term:`RRECOMMENDS` on a package
466name starting with the prefix are satisfied during build time. If you
467are using ``do_split_packages`` as described in the previous section,
468the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
469pattern specified in the call to ``do_split_packages``.
470
471Using Runtime Package Management
472================================
473
474During a build, BitBake always transforms a recipe into one or more
475packages. For example, BitBake takes the ``bash`` recipe and produces a
476number of packages (e.g. ``bash``, ``bash-bashbug``,
477``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
478``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
479generated packages are included in an image.
480
481In several situations, you might need to update, add, remove, or query
482the packages on a target device at runtime (i.e. without having to
483generate a new image). Examples of such situations include:
484
485- You want to provide in-the-field updates to deployed devices (e.g.
486 security updates).
487
488- You want to have a fast turn-around development cycle for one or more
489 applications that run on your device.
490
491- You want to temporarily install the "debug" packages of various
492 applications on your device so that debugging can be greatly improved
493 by allowing access to symbols and source debugging.
494
495- You want to deploy a more minimal package selection of your device
496 but allow in-the-field updates to add a larger selection for
497 customization.
498
499In all these situations, you have something similar to a more
500traditional Linux distribution in that in-field devices are able to
501receive pre-compiled packages from a server for installation or update.
502Being able to install these packages on a running, in-field device is
503what is termed "runtime package management".
504
505In order to use runtime package management, you need a host or server
506machine that serves up the pre-compiled packages plus the required
507metadata. You also need package manipulation tools on the target. The
508build machine is a likely candidate to act as the server. However, that
509machine does not necessarily have to be the package server. The build
510machine could push its artifacts to another machine that acts as the
511server (e.g. Internet-facing). In fact, doing so is advantageous for a
512production environment as getting the packages away from the development
513system's :term:`Build Directory` prevents accidental overwrites.
514
515A simple build that targets just one device produces more than one
516package database. In other words, the packages produced by a build are
517separated out into a couple of different package groupings based on
518criteria such as the target's CPU architecture, the target board, or the
519C library used on the target. For example, a build targeting the
520``qemux86`` device produces the following three package databases:
521``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
522device to be aware of all the packages that were available to it, you
523would need to point it to each of these databases individually. In a
524similar way, a traditional Linux distribution usually is configured to
525be aware of a number of software repositories from which it retrieves
526packages.
527
528Using runtime package management is completely optional and not required
529for a successful build or deployment in any way. But if you want to make
530use of runtime package management, you need to do a couple things above
531and beyond the basics. The remainder of this section describes what you
532need to do.
533
534Build Considerations
535--------------------
536
537This section describes build considerations of which you need to be
538aware in order to provide support for runtime package management.
539
540When BitBake generates packages, it needs to know what format or formats
541to use. In your configuration, you use the
542:term:`PACKAGE_CLASSES`
543variable to specify the format:
544
545#. Open the ``local.conf`` file inside your :term:`Build Directory` (e.g.
546 ``poky/build/conf/local.conf``).
547
548#. Select the desired package format as follows::
549
550 PACKAGE_CLASSES ?= "package_packageformat"
551
552 where packageformat can be "ipk", "rpm",
553 "deb", or "tar" which are the supported package formats.
554
555 .. note::
556
557 Because the Yocto Project supports four different package formats,
558 you can set the variable with more than one argument. However, the
559 OpenEmbedded build system only uses the first argument when
560 creating an image or Software Development Kit (SDK).
561
562If you would like your image to start off with a basic package database
563containing the packages in your current build as well as to have the
564relevant tools available on the target for runtime package management,
565you can include "package-management" in the
566:term:`IMAGE_FEATURES`
567variable. Including "package-management" in this configuration variable
568ensures that when the image is assembled for your target, the image
569includes the currently-known package databases as well as the
570target-specific tools required for runtime package management to be
571performed on the target. However, this is not strictly necessary. You
572could start your image off without any databases but only include the
573required on-target package tool(s). As an example, you could include
574"opkg" in your
575:term:`IMAGE_INSTALL` variable
576if you are using the IPK package format. You can then initialize your
577target's package database(s) later once your image is up and running.
578
579Whenever you perform any sort of build step that can potentially
580generate a package or modify existing package, it is always a good idea
581to re-generate the package index after the build by using the following
582command::
583
584 $ bitbake package-index
585
586It might be tempting to build the
587package and the package index at the same time with a command such as
588the following::
589
590 $ bitbake some-package package-index
591
592Do not do this as
593BitBake does not schedule the package index for after the completion of
594the package you are building. Consequently, you cannot be sure of the
595package index including information for the package you just built.
596Thus, be sure to run the package update step separately after building
597any packages.
598
599You can use the
600:term:`PACKAGE_FEED_ARCHS`,
601:term:`PACKAGE_FEED_BASE_PATHS`,
602and
603:term:`PACKAGE_FEED_URIS`
604variables to pre-configure target images to use a package feed. If you
605do not define these variables, then manual steps as described in the
606subsequent sections are necessary to configure the target. You should
607set these variables before building the image in order to produce a
608correctly configured image.
609
610.. note::
611
612 Your image will need enough free storage space to run package upgrades,
613 especially if many of them need to be downloaded at the same time.
614 You should make sure images are created with enough free space
615 by setting the :term:`IMAGE_ROOTFS_EXTRA_SPACE` variable.
616
617When your build is complete, your packages reside in the
618``${TMPDIR}/deploy/packageformat`` directory. For example, if
619``${``\ :term:`TMPDIR`\ ``}`` is
620``tmp`` and your selected package type is RPM, then your RPM packages
621are available in ``tmp/deploy/rpm``.
622
623Host or Server Machine Setup
624----------------------------
625
626Although other protocols are possible, a server using HTTP typically
627serves packages. If you want to use HTTP, then set up and configure a
628web server such as Apache 2, lighttpd, or Python web server on the
629machine serving the packages.
630
631To keep things simple, this section describes how to set up a
632Python web server to share package feeds from the developer's
633machine. Although this server might not be the best for a production
634environment, the setup is simple and straight forward. Should you want
635to use a different server more suited for production (e.g. Apache 2,
636Lighttpd, or Nginx), take the appropriate steps to do so.
637
638From within the :term:`Build Directory` where you have built an image based on
639your packaging choice (i.e. the :term:`PACKAGE_CLASSES` setting), simply start
640the server. The following example assumes a :term:`Build Directory` of ``poky/build``
641and a :term:`PACKAGE_CLASSES` setting of ":ref:`ref-classes-package_rpm`"::
642
643 $ cd poky/build/tmp/deploy/rpm
644 $ python3 -m http.server
645
646Target Setup
647------------
648
649Setting up the target differs depending on the package management
650system. This section provides information for RPM, IPK, and DEB.
651
652Using RPM
653~~~~~~~~~
654
655The :wikipedia:`Dandified Packaging <DNF_(software)>` (DNF) performs
656runtime package management of RPM packages. In order to use DNF for
657runtime package management, you must perform an initial setup on the
658target machine for cases where the ``PACKAGE_FEED_*`` variables were not
659set as part of the image that is running on the target. This means if
660you built your image and did not use these variables as part of the
661build and your image is now running on the target, you need to perform
662the steps in this section if you want to use runtime package management.
663
664.. note::
665
666 For information on the ``PACKAGE_FEED_*`` variables, see
667 :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
668 :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
669 glossary.
670
671On the target, you must inform DNF that package databases are available.
672You do this by creating a file named
673``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.
674
675As an example, assume the target is able to use the following package
676databases: ``all``, ``i586``, and ``qemux86`` from a server named
677``my.server``. The specifics for setting up the web server are up to
678you. The critical requirement is that the URIs in the target repository
679configuration point to the correct remote location for the feeds.
680
681.. note::
682
683 For development purposes, you can point the web server to the build
684 system's ``deploy`` directory. However, for production use, it is better to
685 copy the package directories to a location outside of the build area and use
686 that location. Doing so avoids situations where the build system
687 overwrites or changes the ``deploy`` directory.
688
689When telling DNF where to look for the package databases, you must
690declare individual locations per architecture or a single location used
691for all architectures. You cannot do both:
692
693- *Create an Explicit List of Architectures:* Define individual base
694 URLs to identify where each package database is located:
695
696 .. code-block:: none
697
698 [oe-packages]
699 baseurl=http://my.server/rpm/i586 http://my.server/rpm/qemux86 http://my.server/rpm/all
700
701 This example
702 informs DNF about individual package databases for all three
703 architectures.
704
705- *Create a Single (Full) Package Index:* Define a single base URL that
706 identifies where a full package database is located::
707
708 [oe-packages]
709 baseurl=http://my.server/rpm
710
711 This example informs DNF about a single
712 package database that contains all the package index information for
713 all supported architectures.
714
715Once you have informed DNF where to find the package databases, you need
716to fetch them:
717
718.. code-block:: none
719
720 # dnf makecache
721
722DNF is now able to find, install, and
723upgrade packages from the specified repository or repositories.
724
725.. note::
726
727 See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
728 additional information.
729
730Using IPK
731~~~~~~~~~
732
733The ``opkg`` application performs runtime package management of IPK
734packages. You must perform an initial setup for ``opkg`` on the target
735machine if the
736:term:`PACKAGE_FEED_ARCHS`,
737:term:`PACKAGE_FEED_BASE_PATHS`,
738and
739:term:`PACKAGE_FEED_URIS`
740variables have not been set or the target image was built before the
741variables were set.
742
743The ``opkg`` application uses configuration files to find available
744package databases. Thus, you need to create a configuration file inside
745the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
746you want to use.
747
748As an example, suppose you are serving packages from a ``ipk/``
749directory containing the ``i586``, ``all``, and ``qemux86`` databases
750through an HTTP server named ``my.server``. On the target, create a
751configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
752directory containing the following:
753
754.. code-block:: none
755
756 src/gz all http://my.server/ipk/all
757 src/gz i586 http://my.server/ipk/i586
758 src/gz qemux86 http://my.server/ipk/qemux86
759
760Next, instruct ``opkg`` to fetch the
761repository information:
762
763.. code-block:: none
764
765 # opkg update
766
767The ``opkg`` application is now able to find, install, and upgrade packages
768from the specified repository.
769
770Using DEB
771~~~~~~~~~
772
773The ``apt`` application performs runtime package management of DEB
774packages. This application uses a source list file to find available
775package databases. You must perform an initial setup for ``apt`` on the
776target machine if the
777:term:`PACKAGE_FEED_ARCHS`,
778:term:`PACKAGE_FEED_BASE_PATHS`,
779and
780:term:`PACKAGE_FEED_URIS`
781variables have not been set or the target image was built before the
782variables were set.
783
784To inform ``apt`` of the repository you want to use, you might create a
785list file (e.g. ``my_repo.list``) inside the
786``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
787serving packages from a ``deb/`` directory containing the ``i586``,
788``all``, and ``qemux86`` databases through an HTTP server named
789``my.server``. The list file should contain:
790
791.. code-block:: none
792
793 deb http://my.server/deb/all ./
794 deb http://my.server/deb/i586 ./
795 deb http://my.server/deb/qemux86 ./
796
797Next, instruct the ``apt`` application
798to fetch the repository information:
799
800.. code-block:: none
801
802 $ sudo apt update
803
804After this step,
805``apt`` is able to find, install, and upgrade packages from the
806specified repository.
807
808Generating and Using Signed Packages
809====================================
810
811In order to add security to RPM packages used during a build, you can
812take steps to securely sign them. Once a signature is verified, the
813OpenEmbedded build system can use the package in the build. If security
814fails for a signed package, the build system stops the build.
815
816This section describes how to sign RPM packages during a build and how
817to use signed package feeds (repositories) when doing a build.
818
819Signing RPM Packages
820--------------------
821
822To enable signing RPM packages, you must modify the ``rpm``
823recipe configuration to include support for OpenPGP signing.
824That may be done either in a ``.bbappend`` for the ``rpm`` recipe::
825
826 PACKAGECONFIG:append = " sequoia"
827
828or in a :term:`Configuration File`::
829
830 PACKAGECONFIG:append:pn-rpm-native = " sequoia"
831 PACKAGECONFIG:append:pn-rpm = " sequoia"
832
833You must also set up the following settings in a
834:term:`Configuration File`::
835
836 # Inherit sign_rpm.bbclass to enable signing functionality
837 INHERIT += " sign_rpm"
838 # Define the GPG key that will be used for signing.
839 RPM_GPG_NAME = "key_name"
840 # Provide passphrase for the key
841 RPM_GPG_PASSPHRASE = "passphrase"
842
843.. note::
844
845 Be sure to supply appropriate values for both `key_name` and
846 `passphrase`.
847
848Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
849the previous example, two optional variables related to signing are available:
850
851- *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
852 when the package is signed.
853
854- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
855 package is signed.
856
857Processing Package Feeds
858------------------------
859
860In addition to being able to sign RPM packages, you can also enable
861signed package feeds for IPK and RPM packages.
862
863The steps you need to take to enable signed package feed use are similar
864to the steps used to sign RPM packages. You must define the following in
865your ``local.config`` or ``distro.config`` file::
866
867 INHERIT += "sign_package_feed"
868 PACKAGE_FEED_GPG_NAME = "key_name"
869 PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"
870
871For signed package feeds, the passphrase must be specified in a separate file,
872which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
873variable. Regarding security, keeping a plain text passphrase out of the
874configuration is more secure.
875
876Aside from the ``PACKAGE_FEED_GPG_NAME`` and
877``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
878related to signed package feeds are available:
879
880- *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
881 when the package is signed.
882
883- *GPG_PATH:* Specifies the ``gpg`` home directory used when the
884 package is signed.
885
886- *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
887 signature. This variable applies only to RPM and IPK package feeds.
888 Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
889 "ASC", which is the default and specifies ascii armored, and "BIN",
890 which specifies binary.
891
892Testing Packages With ptest
893===========================
894
895See the :ref:`test-manual/ptest:Testing Packages With ptest` section of the
896Yocto Project Test Environment Manual.
897
898Creating Node Package Manager (NPM) Packages
899============================================
900
901:wikipedia:`NPM <Npm_(software)>` is a package manager for the JavaScript
902programming language. The Yocto Project supports the NPM
903:ref:`fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`.
904You can use this fetcher in combination with
905:doc:`devtool </ref-manual/devtool-reference>` to create recipes that produce
906NPM packages.
907
908There are two workflows that allow you to create NPM packages using
909``devtool``: the NPM registry modules method and the NPM project code
910method.
911
912.. note::
913
914 While it is possible to create NPM recipes manually, using
915 ``devtool`` is far simpler.
916
917Additionally, some requirements and caveats exist.
918
919Requirements and Caveats
920------------------------
921
922You need to be aware of the following before using ``devtool`` to create
923NPM packages:
924
925- Of the two methods that you can use ``devtool`` to create NPM
926 packages, the registry approach is slightly simpler. However, you
927 might consider the project approach because you do not have to
928 publish your module in the `NPM registry <https://docs.npmjs.com/misc/registry>`__,
929 which is NPM's public registry.
930
931- Be familiar with
932 :doc:`devtool </ref-manual/devtool-reference>`.
933
934- The NPM host tools need the native ``nodejs-npm`` package, which is
935 part of the OpenEmbedded environment. You need to get the package by
936 cloning the :oe_git:`meta-openembedded </meta-openembedded>`
937 repository. Be sure to add the path to your local copy
938 to your ``bblayers.conf`` file.
939
940- ``devtool`` cannot detect native libraries in module dependencies.
941 Consequently, you must manually add packages to your recipe.
942
943- While deploying NPM packages, ``devtool`` cannot determine which
944 dependent packages are missing on the target (e.g. the node runtime
945 ``nodejs``). Consequently, you need to find out what files are
946 missing and be sure they are on the target.
947
948- Although you might not need NPM to run your node package, it is
949 useful to have NPM on your target. The NPM package name is
950 ``nodejs-npm``.
951
952Using the Registry Modules Method
953---------------------------------
954
955This section presents an example that uses the ``cute-files`` module,
956which is a file browser web application.
957
958.. note::
959
960 You must know the ``cute-files`` module version.
961
962The first thing you need to do is use ``devtool`` and the NPM fetcher to
963create the recipe::
964
965 $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"
966
967The
968``devtool add`` command runs ``recipetool create`` and uses the same
969fetch URI to download each dependency and capture license details where
970possible. The result is a generated recipe.
971
972After running for quite a long time, in particular building the
973``nodejs-native`` package, the command should end as follows::
974
975 INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional
976
977The recipe file is fairly simple and contains every license that
978``recipetool`` finds and includes the licenses in the recipe's
979:term:`LIC_FILES_CHKSUM`
980variables. You need to examine the variables and look for those with
981"unknown" in the :term:`LICENSE`
982field. You need to track down the license information for "unknown"
983modules and manually add the information to the recipe.
984
985``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
986files capture the version of all dependent modules. Many packages do not
987provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it
988runs.
989
990.. note::
991
992 A package is created for each sub-module. This policy is the only
993 practical way to have the licenses for all of the dependencies
994 represented in the license manifest of the image.
995
996The ``devtool edit-recipe`` command lets you take a look at the recipe::
997
998 $ devtool edit-recipe cute-files
999 # Recipe created by recipetool
1000 # This is the basis of a recipe and may need further editing in order to be fully functional.
1001 # (Feel free to remove these comments when editing.)
1002
1003 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
1004 # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
1005 # your responsibility to verify that the values are complete and correct.
1006 #
1007 # NOTE: multiple licenses have been detected; they have been separated with &
1008 # in the LICENSE value for now since it is a reasonable assumption that all
1009 # of the licenses apply. If instead there is a choice between the multiple
1010 # licenses then you should change the value to separate the licenses with |
1011 # instead of &. If there is any doubt, check the accompanying documentation
1012 # to determine which situation is applicable.
1013
1014 SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
1015 LICENSE = "BSD-3-Clause & ISC & MIT"
1016 LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
1017 file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \
1018 file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \
1019 ...
1020 file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a"
1021
1022 SRC_URI = " \
1023 npm://registry.npmjs.org/;package=cute-files;version=${PV} \
1024 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
1025 "
1026
1027 S = "${UNPACKDIR}/npm"
1028
1029 inherit npm
1030
1031 LICENSE:${PN} = "MIT"
1032 LICENSE:${PN}-accepts = "MIT"
1033 LICENSE:${PN}-array-flatten = "MIT"
1034 ...
1035 LICENSE:${PN}-vary = "MIT"
1036
1037Three key points in the previous example are:
1038
1039- :term:`SRC_URI` uses the NPM
1040 scheme so that the NPM fetcher is used.
1041
1042- ``recipetool`` collects all the license information. If a
1043 sub-module's license is unavailable, the sub-module's name appears in
1044 the comments.
1045
1046- The ``inherit npm`` statement causes the :ref:`ref-classes-npm` class to
1047 package up all the modules.
1048
1049You can run the following command to build the ``cute-files`` package::
1050
1051 $ devtool build cute-files
1052
1053Remember that ``nodejs`` must be installed on
1054the target before your package.
1055
1056Assuming 192.168.7.2 for the target's IP address, use the following
1057command to deploy your package::
1058
1059 $ devtool deploy-target -s cute-files root@192.168.7.2
1060
1061Once the package is installed on the target, you can
1062test the application to show the contents of any directory::
1063
1064 $ cd /usr/lib/node_modules/cute-files
1065 $ cute-files
1066
1067On a browser,
1068go to ``http://192.168.7.2:3000`` and you see the following:
1069
1070.. image:: figures/cute-files-npm-example.png
1071 :width: 100%
1072
1073You can find the recipe in ``workspace/recipes/cute-files``. You can use
1074the recipe in any layer you choose.
1075
1076Using the NPM Projects Code Method
1077----------------------------------
1078
1079Although it is useful to package modules already in the NPM registry,
1080adding ``node.js`` projects under development is a more common developer
1081use case.
1082
1083This section covers the NPM projects code method, which is very similar
1084to the "registry" approach described in the previous section. In the NPM
1085projects method, you provide ``devtool`` with an URL that points to the
1086source files.
1087
1088Replicating the same example, (i.e. ``cute-files``) use the following
1089command::
1090
1091 $ devtool add https://github.com/martinaglv/cute-files.git
1092
1093The recipe this command generates is very similar to the recipe created in
1094the previous section. However, the :term:`SRC_URI` looks like the following::
1095
1096 SRC_URI = " \
1097 git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \
1098 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
1099 "
1100
1101In this example,
1102the main module is taken from the Git repository and dependencies are
1103taken from the NPM registry. Other than those differences, the recipe is
1104basically the same between the two methods. You can build and deploy the
1105package exactly as described in the previous section that uses the
1106registry modules method.
1107
1108Adding custom metadata to packages
1109==================================
1110
1111The variable
1112:term:`PACKAGE_ADD_METADATA`
1113can be used to add additional metadata to packages. This is reflected in
1114the package control/spec file. To take the ipk format for example, the
1115CONTROL file stored inside would contain the additional metadata as
1116additional lines.
1117
1118The variable can be used in multiple ways, including using suffixes to
1119set it for a specific package type and/or package. Note that the order
1120of precedence is the same as this list:
1121
1122- ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``
1123
1124- ``PACKAGE_ADD_METADATA_<PKGTYPE>``
1125
1126- ``PACKAGE_ADD_METADATA:<PN>``
1127
1128- :term:`PACKAGE_ADD_METADATA`
1129
1130`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
1131package type:
1132
1133- IPK for .ipk packages
1134
1135- DEB for .deb packages
1136
1137- RPM for .rpm packages
1138
1139`<PN>` is a parameter and expected to be a package name.
1140
1141The variable can contain multiple [one-line] metadata fields separated
1142by the literal sequence '\\n'. The separator can be redefined using the
1143variable flag ``separator``.
1144
1145Here is an example that adds two custom fields for ipk
1146packages::
1147
1148 PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"
1149
diff --git a/documentation/dev-manual/poky-manual-setup.rst b/documentation/dev-manual/poky-manual-setup.rst
deleted file mode 100644
index 3e7c7d5066..0000000000
--- a/documentation/dev-manual/poky-manual-setup.rst
+++ /dev/null
@@ -1,128 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Setting Up the Poky Reference Distro Manually
4*********************************************
5
6While the default way to setup the :term:`Poky` reference distro is to use
7``bitbake-setup``, it is also possible to manually setup the environment. This
8document guides through this setup step-by-step.
9
10.. note::
11
12 This document will produce a setup similar to what is described in
13 :doc:`/brief-yoctoprojectqs/index`, which shows how to setup :term:`Poky`
14 with ``bitbake-setup``.
15
16Use Git to Clone The Layers
17===========================
18
19Go to :yocto_wiki:`Releases wiki page </Releases>`, and choose a release
20(such as ``&DISTRO_REL_LATEST_TAG;``), corresponding to either the latest stable
21release or a Long Term Support release.
22
23Once you complete the setup instructions for your machine (see the
24:doc:`/ref-manual/system-requirements` section of the Yocto Project Reference
25Manual), you need to get a copy of the different :term:`layers <Layer>` needed
26to setup the :term:`Poky` reference distribution on your build host. Use the
27following commands:
28
29.. code-block:: shell
30
31 $ mkdir layers/
32 $ git clone -b &DISTRO_REL_LATEST_TAG; https://git.openembedded.org/bitbake ./layers/bitbake
33 $ git clone -b &DISTRO_REL_LATEST_TAG; https://git.openembedded.org/openembedded-core ./layers/openembedded-core
34 $ git clone -b &DISTRO_REL_LATEST_TAG; https://git.yoctoproject.org/meta-yocto ./layers/meta-yocto
35
36Building Your Image
37===================
38
39Use the following steps to build your image. The build process creates
40an entire Linux distribution, including the toolchain, from source.
41
42.. note::
43
44 - If you are working behind a firewall and your build host is not
45 set up for proxies, you could encounter problems with the build
46 process when fetching source code (e.g. fetcher failures or Git
47 failures).
48
49 - If you do not know your proxy settings, consult your local network
50 infrastructure resources and get that information. A good starting
51 point could also be to check your web browser settings. Finally,
52 you can find more information on the
53 ":yocto_wiki:`Working Behind a Network Proxy </Working_Behind_a_Network_Proxy>`"
54 page of the Yocto Project Wiki.
55
56#. **Initialize the Build Environment:** From your current working directory,
57 setup a build environment with the following command:
58
59 .. code-block:: shell
60
61 $ TEMPLATECONF=$PWD/layers/meta-yocto/meta-poky/conf/templates/default source ./layers/openembedded-core/oe-init-build-env
62
63 Among other things, the script creates the :term:`Build Directory`, which is
64 ``build`` in this case and is located in the :term:`Source Directory`.
65 After the script runs, your current working directory is set to the
66 :term:`Build Directory`. Later, when the build completes, the :term:`Build
67 Directory` contains all the files created during the build.
68
69#. **Examine Your Local Configuration File:** When you set up the build
70 environment, a local configuration file named ``local.conf`` becomes
71 available in a ``conf`` sub-directory of the :term:`Build Directory`. For
72 this example, the defaults are set to build for a ``qemux86-64`` target,
73 which is suitable for emulation. The package manager used is set to the RPM
74 package manager.
75
76 .. tip::
77
78 You can significantly speed up your build and guard against fetcher
79 failures by using :ref:`overview-manual/concepts:shared state cache`
80 mirrors and enabling :ref:`overview-manual/concepts:hash equivalence`.
81 This way, you can use pre-built artifacts rather than building them.
82 This is relevant only when your network and the server that you use
83 can download these artifacts faster than you would be able to build them.
84
85 To use such mirrors, uncomment the below lines in your ``conf/local.conf``
86 file in the :term:`Build Directory`::
87
88 BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws"
89 SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"
90 BB_HASHSERVE = "auto"
91 BB_SIGNATURE_HANDLER = "OEEquivHash"
92
93 The hash equivalence server needs the websockets python module version 9.1
94 or later. Debian GNU/Linux 12 (Bookworm) and later, Fedora, CentOS Stream
95 9 and later, and Ubuntu 22.04 (LTS) and later, all have a recent enough
96 package. Other supported distributions need to get the module some other
97 place than their package feed, e.g. via ``pip``. You can otherwise
98 install a :term:`Buildtools` tarball by following the instructions in
99 the :ref:`system-requirements-buildtools` section of the Yocto Project
100 Reference Manual.
101
102#. **Start the Build:** Continue with the following command to build an OS
103 image for the target, which is ``core-image-sato`` in this example:
104
105 .. code-block:: shell
106
107 $ bitbake core-image-sato
108
109 For information on using the ``bitbake`` command, see the
110 :ref:`overview-manual/concepts:bitbake` section in the Yocto Project Overview and
111 Concepts Manual, or see
112 :ref:`bitbake-user-manual/bitbake-user-manual-intro:the bitbake command`
113 in the BitBake User Manual.
114
115#. **Simulate Your Image Using QEMU:** Once this particular image is
116 built, you can start QEMU, which is a Quick EMUlator that ships with
117 the Yocto Project:
118
119 .. code-block:: shell
120
121 $ runqemu qemux86-64
122
123 If you want to learn more about running QEMU, see the
124 :ref:`dev-manual/qemu:using the quick emulator (qemu)` chapter in
125 the Yocto Project Development Tasks Manual.
126
127#. **Exit QEMU:** Exit QEMU by either clicking on the shutdown icon or by typing
128 ``Ctrl-C`` in the QEMU transcript window from which you evoked QEMU.
diff --git a/documentation/dev-manual/prebuilt-libraries.rst b/documentation/dev-manual/prebuilt-libraries.rst
deleted file mode 100644
index 59621ca16d..0000000000
--- a/documentation/dev-manual/prebuilt-libraries.rst
+++ /dev/null
@@ -1,209 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Working with Pre-Built Libraries
4********************************
5
6Introduction
7============
8
9Some library vendors do not release source code for their software but do
10release pre-built binaries. When shared libraries are built, they should
11be versioned (see `this article
12<https://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html>`__
13for some background), but sometimes this is not done.
14
15To summarize, a versioned library must meet two conditions:
16
17#. The filename must have the version appended, for example: ``libfoo.so.1.2.3``.
18#. The library must have the ELF tag ``SONAME`` set to the major version
19 of the library, for example: ``libfoo.so.1``. You can check this by
20 running ``readelf -d filename | grep SONAME``.
21
22This section shows how to deal with both versioned and unversioned
23pre-built libraries.
24
25Versioned Libraries
26===================
27
28In this example we work with pre-built libraries for the FT4222H USB I/O chip.
29Libraries are built for several target architecture variants and packaged in
30an archive as follows::
31
32 ├── build-arm-hisiv300
33 │   └── libft4222.so.1.4.4.44
34 ├── build-arm-v5-sf
35 │   └── libft4222.so.1.4.4.44
36 ├── build-arm-v6-hf
37 │   └── libft4222.so.1.4.4.44
38 ├── build-arm-v7-hf
39 │   └── libft4222.so.1.4.4.44
40 ├── build-arm-v8
41 │   └── libft4222.so.1.4.4.44
42 ├── build-i386
43 │   └── libft4222.so.1.4.4.44
44 ├── build-i486
45 │   └── libft4222.so.1.4.4.44
46 ├── build-mips-eglibc-hf
47 │   └── libft4222.so.1.4.4.44
48 ├── build-pentium
49 │   └── libft4222.so.1.4.4.44
50 ├── build-x86_64
51 │   └── libft4222.so.1.4.4.44
52 ├── examples
53 │   ├── get-version.c
54 │   ├── i2cm.c
55 │   ├── spim.c
56 │   └── spis.c
57 ├── ftd2xx.h
58 ├── install4222.sh
59 ├── libft4222.h
60 ├── ReadMe.txt
61 └── WinTypes.h
62
63To write a recipe to use such a library in your system:
64
65- The vendor will probably have a proprietary licence, so set
66 :term:`LICENSE_FLAGS` in your recipe.
67- The vendor provides a tarball containing libraries so set :term:`SRC_URI`
68 appropriately.
69- Set :term:`COMPATIBLE_HOST` so that the recipe cannot be used with an
70 unsupported architecture. In the following example, we only support the 32
71 and 64 bit variants of the ``x86`` architecture.
72- As the vendor provides versioned libraries, we can use ``oe_soinstall``
73 from :ref:`ref-classes-utils` to install the shared library and create
74 symbolic links. If the vendor does not do this, we need to follow the
75 non-versioned library guidelines in the next section.
76- As the vendor likely used :term:`LDFLAGS` different from those in your Yocto
77 Project build, disable the corresponding checks by adding ``ldflags``
78 to :term:`INSANE_SKIP`.
79- The vendor will typically ship release builds without debugging symbols.
80 Avoid errors by preventing the packaging task from stripping out the symbols
81 and adding them to a separate debug package. This is done by setting the
82 ``INHIBIT_`` flags shown below.
83
84The complete recipe would look like this::
85
86 SUMMARY = "FTDI FT4222H Library"
87 SECTION = "libs"
88 LICENSE_FLAGS = "ftdi"
89 LICENSE = "CLOSED"
90
91 COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
92
93 # Sources available in a .tgz file in .zip archive
94 # at https://ftdichip.com/wp-content/uploads/2021/01/libft4222-linux-1.4.4.44.zip
95 # Found on https://ftdichip.com/software-examples/ft4222h-software-examples/
96 # Since dealing with this particular type of archive is out of topic here,
97 # we use a local link.
98 SRC_URI = "file://libft4222-linux-${PV}.tgz"
99
100 S = "${UNPACKDIR}"
101
102 ARCH_DIR:x86-64 = "build-x86_64"
103 ARCH_DIR:i586 = "build-i386"
104 ARCH_DIR:i686 = "build-i386"
105
106 INSANE_SKIP:${PN} = "ldflags"
107 INHIBIT_PACKAGE_STRIP = "1"
108 INHIBIT_SYSROOT_STRIP = "1"
109 INHIBIT_PACKAGE_DEBUG_SPLIT = "1"
110
111 do_install () {
112 install -m 0755 -d ${D}${libdir}
113 oe_soinstall ${S}/${ARCH_DIR}/libft4222.so.${PV} ${D}${libdir}
114 install -d ${D}${includedir}
115 install -m 0755 ${S}/*.h ${D}${includedir}
116 }
117
118If the precompiled binaries are not statically linked and have dependencies on
119other libraries, then by adding those libraries to :term:`DEPENDS`, the linking
120can be examined and the appropriate :term:`RDEPENDS` automatically added.
121
122Non-Versioned Libraries
123=======================
124
125Some Background
126---------------
127
128Libraries in Linux systems are generally versioned so that it is possible
129to have multiple versions of the same library installed, which eases upgrades
130and support for older software. For example, suppose that in a versioned
131library, an actual library is called ``libfoo.so.1.2``, a symbolic link named
132``libfoo.so.1`` points to ``libfoo.so.1.2``, and a symbolic link named
133``libfoo.so`` points to ``libfoo.so.1.2``. Given these conditions, when you
134link a binary against a library, you typically provide the unversioned file
135name (i.e. ``-lfoo`` to the linker). However, the linker follows the symbolic
136link and actually links against the versioned filename. The unversioned symbolic
137link is only used at development time. Consequently, the library is packaged
138along with the headers in the development package ``${PN}-dev`` along with the
139actual library and versioned symbolic links in ``${PN}``. Because versioned
140libraries are far more common than unversioned libraries, the default packaging
141rules assume versioned libraries.
142
143Yocto Library Packaging Overview
144--------------------------------
145
146It follows that packaging an unversioned library requires a bit of work in the
147recipe. By default, ``libfoo.so`` gets packaged into ``${PN}-dev``, which
148triggers a QA warning that a non-symlink library is in a ``-dev`` package,
149and binaries in the same recipe link to the library in ``${PN}-dev``,
150which triggers more QA warnings. To solve this problem, you need to package the
151unversioned library into ``${PN}`` where it belongs. The abridged
152default :term:`FILES` variables in ``bitbake.conf`` are::
153
154 SOLIBS = ".so.*"
155 SOLIBSDEV = ".so"
156 FILES:${PN} = "... ${libdir}/lib*${SOLIBS} ..."
157 FILES_SOLIBSDEV ?= "... ${libdir}/lib*${SOLIBSDEV} ..."
158 FILES:${PN}-dev = "... ${FILES_SOLIBSDEV} ..."
159
160:term:`SOLIBS` defines a pattern that matches real shared object libraries.
161:term:`SOLIBSDEV` matches the development form (unversioned symlink). These two
162variables are then used in ``FILES:${PN}`` and ``FILES:${PN}-dev``, which puts
163the real libraries into ``${PN}`` and the unversioned symbolic link into ``${PN}-dev``.
164To package unversioned libraries, you need to modify the variables in the recipe
165as follows::
166
167 SOLIBS = ".so"
168 FILES_SOLIBSDEV = ""
169
170The modifications cause the ``.so`` file to be the real library
171and unset :term:`FILES_SOLIBSDEV` so that no libraries get packaged into
172``${PN}-dev``. The changes are required because unless :term:`PACKAGES` is changed,
173``${PN}-dev`` collects files before ``${PN}``. ``${PN}-dev`` must not collect any of
174the files you want in ``${PN}``.
175
176Finally, loadable modules, essentially unversioned libraries that are linked
177at runtime using ``dlopen()`` instead of at build time, should generally be
178installed in a private directory. However, if they are installed in ``${libdir}``,
179then the modules can be treated as unversioned libraries.
180
181Example
182-------
183
184The example below installs an unversioned x86-64 pre-built library named
185``libfoo.so``. The :term:`COMPATIBLE_HOST` variable limits recipes to the
186x86-64 architecture while the :term:`INSANE_SKIP`, :term:`INHIBIT_PACKAGE_STRIP`
187and :term:`INHIBIT_SYSROOT_STRIP` variables are all set as in the above
188versioned library example. The "magic" is setting the :term:`SOLIBS` and
189:term:`FILES_SOLIBSDEV` variables as explained above::
190
191 SUMMARY = "libfoo sample recipe"
192 SECTION = "libs"
193 LICENSE = "CLOSED"
194
195 SRC_URI = "file://libfoo.so"
196
197 COMPATIBLE_HOST = "x86_64.*-linux"
198
199 INSANE_SKIP:${PN} = "ldflags"
200 INHIBIT_PACKAGE_STRIP = "1"
201 INHIBIT_SYSROOT_STRIP = "1"
202 SOLIBS = ".so"
203 FILES_SOLIBSDEV = ""
204
205 do_install () {
206 install -d ${D}${libdir}
207 install -m 0755 ${UNPACKDIR}/libfoo.so ${D}${libdir}
208 }
209
diff --git a/documentation/dev-manual/python-development-shell.rst b/documentation/dev-manual/python-development-shell.rst
deleted file mode 100644
index 81a5c43472..0000000000
--- a/documentation/dev-manual/python-development-shell.rst
+++ /dev/null
@@ -1,50 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using a Python Development Shell
4********************************
5
6Similar to working within a development shell as described in the
7previous section, you can also spawn and work within an interactive
8Python development shell. When debugging certain commands or even when
9just editing packages, ``pydevshell`` can be a useful tool. When you
10invoke the ``pydevshell`` task, all tasks up to and including
11:ref:`ref-tasks-patch` are run for the
12specified target. Then a new terminal is opened. Additionally, key
13Python objects and code are available in the same way they are to
14BitBake tasks, in particular, the data store 'd'. So, commands such as
15the following are useful when exploring the data store and running
16functions::
17
18 pydevshell> d.getVar("STAGING_DIR")
19 '/media/build1/poky/build/tmp/sysroots'
20 pydevshell> d.getVar("STAGING_DIR", False)
21 '${TMPDIR}/sysroots'
22 pydevshell> d.setVar("FOO", "bar")
23 pydevshell> d.getVar("FOO")
24 'bar'
25 pydevshell> d.delVar("FOO")
26 pydevshell> d.getVar("FOO")
27 pydevshell> bb.build.exec_func("do_unpack", d)
28 pydevshell>
29
30See the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:functions you can call from within python`"
31section in the BitBake User Manual for details about available functions.
32
33The commands execute just as if the OpenEmbedded build
34system were executing them. Consequently, working this way can be
35helpful when debugging a build or preparing software to be used with the
36OpenEmbedded build system.
37
38Here is an example that uses ``pydevshell`` on a target named
39``matchbox-desktop``::
40
41 $ bitbake matchbox-desktop -c pydevshell
42
43This command spawns a terminal and places you in an interactive Python
44interpreter within the OpenEmbedded build environment. The
45:term:`OE_TERMINAL` variable
46controls what type of shell is opened.
47
48When you are finished using ``pydevshell``, you can exit the shell
49either by using Ctrl+d or closing the terminal window.
50
diff --git a/documentation/dev-manual/qemu.rst b/documentation/dev-manual/qemu.rst
deleted file mode 100644
index 9e0b8b8136..0000000000
--- a/documentation/dev-manual/qemu.rst
+++ /dev/null
@@ -1,467 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3*******************************
4Using the Quick EMUlator (QEMU)
5*******************************
6
7The Yocto Project uses an implementation of the Quick EMUlator (QEMU)
8Open Source project as part of the Yocto Project development "tool set".
9This chapter provides both procedures that show you how to use the Quick
10EMUlator (QEMU) and other QEMU information helpful for development
11purposes.
12
13Overview
14========
15
16Within the context of the Yocto Project, QEMU is an emulator and
17virtualization machine that allows you to run a complete image you have
18built using the Yocto Project as just another task on your build system.
19QEMU is useful for running and testing images and applications on
20supported Yocto Project architectures without having actual hardware.
21Among other things, the Yocto Project uses QEMU to run automated Quality
22Assurance (QA) tests on final images shipped with each release.
23
24.. note::
25
26 This implementation is not the same as QEMU in general.
27
28This section provides a brief reference for the Yocto Project
29implementation of QEMU.
30
31For official information and documentation on QEMU in general, see the
32following references:
33
34- `QEMU Website <https://wiki.qemu.org/Main_Page>`__\ *:* The official
35 website for the QEMU Open Source project.
36
37- `Documentation <https://wiki.qemu.org/Manual>`__\ *:* The QEMU user
38 manual.
39
40Running QEMU
41============
42
43To use QEMU, you need to have QEMU installed and initialized as well as
44have the proper artifacts (i.e. image files and root filesystems)
45available. Follow these general steps to run QEMU:
46
47#. *Install QEMU:* QEMU is made available with the Yocto Project a
48 number of ways. One method is to install a Software Development Kit
49 (SDK). See ":ref:`sdk-manual/intro:the qemu emulator`" section in the
50 Yocto Project Application Development and the Extensible Software
51 Development Kit (eSDK) manual for information on how to install QEMU.
52
53#. *Setting Up the Environment:* How you set up the QEMU environment
54 depends on how you installed QEMU:
55
56 - If you cloned the ``poky`` repository or you downloaded and
57 unpacked a Yocto Project release tarball, you can source the build
58 environment script (i.e. :ref:`structure-core-script`)::
59
60 $ cd poky
61 $ source oe-init-build-env
62
63 - If you installed a cross-toolchain, you can run the script that
64 initializes the toolchain. For example, the following commands run
65 the initialization script from the default ``poky_sdk`` directory::
66
67 . poky_sdk/environment-setup-core2-64-poky-linux
68
69#. *Ensure the Artifacts are in Place:* You need to be sure you have a
70 pre-built kernel that will boot in QEMU. You also need the target
71 root filesystem for your target machine's architecture:
72
73 - If you have previously built an image for QEMU (e.g. ``qemux86``,
74 ``qemuarm``, and so forth), then the artifacts are in place in
75 your :term:`Build Directory`.
76
77 - If you have not built an image, you can go to the
78 :yocto_dl:`machines/qemu </releases/yocto/&DISTRO_REL_LATEST_TAG;/machines/qemu/>` area and download a
79 pre-built image that matches your architecture and can be run on
80 QEMU.
81
82 See the ":ref:`sdk-manual/appendix-obtain:extracting the root filesystem`"
83 section in the Yocto Project Application Development and the
84 Extensible Software Development Kit (eSDK) manual for information on
85 how to extract a root filesystem.
86
87#. *Run QEMU:* The basic ``runqemu`` command syntax is as follows::
88
89 $ runqemu [option ] [...]
90
91 Based on what you provide on the command
92 line, ``runqemu`` does a good job of figuring out what you are trying
93 to do. For example, by default, QEMU looks for the most recently
94 built image according to the timestamp when it needs to look for an
95 image. Minimally, through the use of options, you must provide either
96 a machine name, a virtual machine image (``*wic.vmdk``), or a kernel
97 image (``*.bin``).
98
99 Here are some additional examples to help illustrate further QEMU:
100
101 - This example starts QEMU with MACHINE set to "qemux86-64".
102 Assuming a standard :term:`Build Directory`, ``runqemu``
103 automatically finds the ``bzImage-qemux86-64.bin`` image file and
104 the ``core-image-minimal-qemux86-64-20200218002850.rootfs.ext4``
105 (assuming the current build created a ``core-image-minimal``
106 image)::
107
108 $ runqemu qemux86-64
109
110 .. note::
111
112 When more than one image with the same name exists, QEMU finds
113 and uses the most recently built image according to the
114 timestamp.
115
116 - This example produces the exact same results as the previous
117 example. This command, however, specifically provides the image
118 and root filesystem type::
119
120 $ runqemu qemux86-64 core-image-minimal ext4
121
122 - This example specifies to boot an :term:`Initramfs` image and to
123 enable audio in QEMU. For this case, ``runqemu`` sets the internal
124 variable ``FSTYPE`` to ``cpio.gz``. Also, for audio to be enabled,
125 an appropriate driver must be installed (see the ``audio`` option
126 in :ref:`dev-manual/qemu:\`\`runqemu\`\` command-line options`
127 for more information)::
128
129 $ runqemu qemux86-64 ramfs audio
130
131 - This example does not provide enough information for QEMU to
132 launch. While the command does provide a root filesystem type, it
133 must also minimally provide a `MACHINE`, `KERNEL`, or `VM` option::
134
135 $ runqemu ext4
136
137 - This example specifies to boot a virtual machine image
138 (``.wic.vmdk`` file). From the ``.wic.vmdk``, ``runqemu``
139 determines the QEMU architecture (`MACHINE`) to be "qemux86-64" and
140 the root filesystem type to be "vmdk"::
141
142 $ runqemu /home/scott-lenovo/vm/core-image-minimal-qemux86-64.wic.vmdk
143
144Switching Between Consoles
145==========================
146
147When booting or running QEMU, you can switch between supported consoles
148by using Ctrl+Alt+number. For example, Ctrl+Alt+3 switches you to the
149serial console as long as that console is enabled. Being able to switch
150consoles is helpful, for example, if the main QEMU console breaks for
151some reason.
152
153.. note::
154
155 Usually, "2" gets you to the main console and "3" gets you to the
156 serial console.
157
158Removing the Splash Screen
159==========================
160
161You can remove the splash screen when QEMU is booting by using Alt+left.
162Removing the splash screen allows you to see what is happening in the
163background.
164
165Disabling the Cursor Grab
166=========================
167
168The default QEMU integration captures the cursor within the main window.
169It does this since standard mouse devices only provide relative input
170and not absolute coordinates. You then have to break out of the grab
171using the "Ctrl+Alt" key combination. However, the Yocto Project's
172integration of QEMU enables the wacom USB touch pad driver by default to
173allow input of absolute coordinates. This default means that the mouse
174can enter and leave the main window without the grab taking effect
175leading to a better user experience.
176
177Running Under a Network File System (NFS) Server
178================================================
179
180One method for running QEMU is to run it on an NFS server. This is
181useful when you need to access the same file system from both the build
182and the emulated system at the same time. It is also worth noting that
183the system does not need root privileges to run. It uses a user space
184NFS server to avoid that. Follow these steps to set up for running QEMU
185using an NFS server.
186
187#. *Extract a Root Filesystem:* Once you are able to run QEMU in your
188 environment, you can use the ``runqemu-extract-sdk`` script, which is
189 located in the ``scripts`` directory along with the ``runqemu``
190 script.
191
192 The ``runqemu-extract-sdk`` takes a root filesystem tarball and
193 extracts it into a location that you specify. Here is an example that
194 takes a file system and extracts it to a directory named
195 ``test-nfs``:
196
197 .. code-block:: none
198
199 runqemu-extract-sdk ./tmp/deploy/images/qemux86-64/core-image-sato-qemux86-64.tar.bz2 test-nfs
200
201#. *Start QEMU:* Once you have extracted the file system, you can run
202 ``runqemu`` normally with the additional location of the file system.
203 You can then also make changes to the files within ``./test-nfs`` and
204 see those changes appear in the image in real time. Here is an
205 example using the ``qemux86`` image:
206
207 .. code-block:: none
208
209 runqemu qemux86-64 ./test-nfs
210
211.. note::
212
213 Should you need to start, stop, or restart the NFS share, you can use
214 the following commands:
215
216 - To start the NFS share::
217
218 runqemu-export-rootfs start file-system-location
219
220 - To stop the NFS share::
221
222 runqemu-export-rootfs stop file-system-location
223
224 - To restart the NFS share::
225
226 runqemu-export-rootfs restart file-system-location
227
228QEMU CPU Compatibility Under KVM
229================================
230
231By default, the QEMU build compiles for and targets 64-bit and x86 Intel
232Core2 Duo processors and 32-bit x86 Intel Pentium II processors. QEMU
233builds for and targets these CPU types because they display a broad
234range of CPU feature compatibility with many commonly used CPUs.
235
236Despite this broad range of compatibility, the CPUs could support a
237feature that your host CPU does not support. Although this situation is
238not a problem when QEMU uses software emulation of the feature, it can
239be a problem when QEMU is running with KVM enabled. Specifically,
240software compiled with a certain CPU feature crashes when run on a CPU
241under KVM that does not support that feature. To work around this
242problem, you can override QEMU's runtime CPU setting by changing the
243``QB_CPU_KVM`` variable in ``qemuboot.conf`` in the :term:`Build Directory`
244``deploy/image`` directory. This setting specifies a ``-cpu`` option passed
245into QEMU in the ``runqemu`` script. Running ``qemu -cpu help`` returns a
246list of available supported CPU types.
247
248QEMU Performance
249================
250
251Using QEMU to emulate your hardware can result in speed issues depending
252on the target and host architecture mix. For example, using the
253``qemux86`` image in the emulator on an Intel-based 32-bit (x86) host
254machine is fast because the target and host architectures match. On the
255other hand, using the ``qemuarm`` image on the same Intel-based host can
256be slower. But, you still achieve faithful emulation of ARM-specific
257issues.
258
259To speed things up, the QEMU images support using ``distcc`` to call a
260cross-compiler outside the emulated system. If you used ``runqemu`` to
261start QEMU, and the ``distccd`` application is present on the host
262system, any BitBake cross-compiling toolchain available from the build
263system is automatically used from within QEMU simply by calling
264``distcc``. You can accomplish this by defining the cross-compiler
265variable (e.g. ``export CC="distcc"``). Alternatively, if you are using
266a suitable SDK image or the appropriate stand-alone toolchain is
267present, the toolchain is also automatically used.
268
269.. note::
270
271 There are several mechanisms to connect to the system running
272 on the QEMU emulator:
273
274 - QEMU provides a framebuffer interface that makes standard consoles
275 available.
276
277 - Generally, headless embedded devices have a serial port. If so,
278 you can configure the operating system of the running image to use
279 that port to run a console. The connection uses standard IP
280 networking.
281
282 - SSH servers are available in some QEMU images. The ``core-image-sato``
283 QEMU image has a Dropbear secure shell (SSH) server that runs with the
284 root password disabled. The ``core-image-full-cmdline`` QEMU image has
285 OpenSSH instead of Dropbear. Including these SSH servers allow you to use
286 standard ``ssh`` and ``scp`` commands. The ``core-image-minimal`` QEMU
287 image, however, contains no SSH server.
288
289 - You can use a provided, user-space NFS server to boot the QEMU
290 session using a local copy of the root filesystem on the host. In
291 order to make this connection, you must extract a root filesystem
292 tarball by using the ``runqemu-extract-sdk`` command. After
293 running the command, you must then point the ``runqemu`` script to
294 the extracted directory instead of a root filesystem image file.
295 See the
296 ":ref:`dev-manual/qemu:running under a network file system (nfs) server`"
297 section for more information.
298
299QEMU Command-Line Syntax
300========================
301
302The basic ``runqemu`` command syntax is as follows::
303
304 $ runqemu [option ] [...]
305
306Based on what you provide on the command line, ``runqemu`` does a
307good job of figuring out what you are trying to do. For example, by
308default, QEMU looks for the most recently built image according to the
309timestamp when it needs to look for an image. Minimally, through the use
310of options, you must provide either a machine name, a virtual machine
311image (``*wic.vmdk``), or a kernel image (``*.bin``).
312
313Here is the command-line help output for the ``runqemu`` command::
314
315 $ runqemu --help
316
317 Usage: you can run this script with any valid combination
318 of the following environment variables (in any order):
319 KERNEL - the kernel image file to use
320 ROOTFS - the rootfs image file or nfsroot directory to use
321 MACHINE - the machine name (optional, autodetected from KERNEL filename if unspecified)
322 Simplified QEMU command-line options can be passed with:
323 nographic - disable video console
324 serial - enable a serial console on /dev/ttyS0
325 slirp - enable user networking, no root privileges required
326 kvm - enable KVM when running x86/x86_64 (VT-capable CPU required)
327 kvm-vhost - enable KVM with vhost when running x86/x86_64 (VT-capable CPU required)
328 publicvnc - enable a VNC server open to all hosts
329 audio - enable audio
330 [*/]ovmf* - OVMF firmware file or base name for booting with UEFI
331 tcpserial=<port> - specify tcp serial port number
332 biosdir=<dir> - specify custom bios dir
333 biosfilename=<filename> - specify bios filename
334 qemuparams=<xyz> - specify custom parameters to QEMU
335 bootparams=<xyz> - specify custom kernel parameters during boot
336 help, -h, --help: print this text
337
338 Examples:
339 runqemu
340 runqemu qemuarm
341 runqemu tmp/deploy/images/qemuarm
342 runqemu tmp/deploy/images/qemux86/<qemuboot.conf>
343 runqemu qemux86-64 core-image-sato ext4
344 runqemu qemux86-64 wic-image-minimal wic
345 runqemu path/to/bzImage-qemux86.bin path/to/nfsrootdir/ serial
346 runqemu qemux86 iso/hddimg/wic.vmdk/wic.qcow2/wic.vdi/ramfs/cpio.gz...
347 runqemu qemux86 qemuparams="-m 256"
348 runqemu qemux86 bootparams="psplash=false"
349 runqemu path/to/<image>-<machine>.wic
350 runqemu path/to/<image>-<machine>.wic.vmdk
351
352``runqemu`` Command-Line Options
353================================
354
355Here is a description of ``runqemu`` options you can provide on the
356command line:
357
358.. note::
359
360 If you do provide some "illegal" option combination or perhaps you do
361 not provide enough in the way of options, ``runqemu``
362 provides appropriate error messaging to help you correct the problem.
363
364- `QEMUARCH`: The QEMU machine architecture, which must be "qemuarm",
365 "qemuarm64", "qemumips", "qemumips64", "qemuppc", "qemux86", or
366 "qemux86-64".
367
368- `VM`: The virtual machine image, which must be a ``.wic.vmdk``
369 file. Use this option when you want to boot a ``.wic.vmdk`` image.
370
371- `ROOTFS`: A root filesystem that has one of the following filetype
372 extensions: "ext2", "ext3", "ext4", "jffs2", "nfs", or "btrfs". If
373 the filename you provide for this option uses "nfs", it must provide
374 an explicit root filesystem path.
375
376- `KERNEL`: A kernel image, which is a ``.bin`` file. When you provide a
377 ``.bin`` file, ``runqemu`` detects it and assumes the file is a
378 kernel image.
379
380- `MACHINE`: The architecture of the QEMU machine, which must be one of
381 the following: "qemux86", "qemux86-64", "qemuarm", "qemuarm64",
382 "qemumips", "qemumips64", or "qemuppc". The MACHINE and QEMUARCH
383 options are basically identical. If you do not provide a MACHINE
384 option, ``runqemu`` tries to determine it based on other options.
385
386- ``ramfs``: Indicates you are booting an :term:`Initramfs`
387 image, which means the ``FSTYPE`` is ``cpio.gz``.
388
389- ``iso``: Indicates you are booting an ISO image, which means the
390 ``FSTYPE`` is ``.iso``.
391
392- ``nographic``: Disables the video console, which sets the console to
393 "ttys0". This option is useful when you have logged into a server and
394 you do not want to disable forwarding from the X Window System (X11)
395 to your workstation or laptop.
396
397- ``serial``: Enables a serial console on ``/dev/ttyS0``.
398
399- ``biosdir``: Establishes a custom directory for BIOS, VGA BIOS and
400 keymaps.
401
402- ``biosfilename``: Establishes a custom BIOS name.
403
404- ``qemuparams=\"xyz\"``: Specifies custom QEMU parameters. Use this
405 option to pass options other than the simple "kvm" and "serial"
406 options.
407
408- ``bootparams=\"xyz\"``: Specifies custom boot parameters for the
409 kernel.
410
411- ``audio``: Enables audio in QEMU. The MACHINE option must be either
412 "qemux86" or "qemux86-64" in order for audio to be enabled.
413 Additionally, the ``snd_intel8x0`` or ``snd_ens1370`` driver must be
414 installed in linux guest.
415
416- ``slirp``: Enables "slirp" networking, which is a different way of
417 networking that does not need root access but also is not as easy to
418 use or comprehensive as the default.
419
420 Using ``slirp`` by default will forward the guest machine's
421 22 and 23 TCP ports to host machine's 2222 and 2323 ports
422 (or the next free ports). Specific forwarding rules can be configured
423 by setting ``QB_SLIRP_OPT`` as environment variable or in ``qemuboot.conf``
424 in the :term:`Build Directory` ``deploy/image`` directory.
425 Examples::
426
427 QB_SLIRP_OPT = "-netdev user,id=net0,hostfwd=tcp::8080-:80"
428
429 QB_SLIRP_OPT = "-netdev user,id=net0,hostfwd=tcp::8080-:80,hostfwd=tcp::2222-:22"
430
431 The first example forwards TCP port 80 from the emulated system to
432 port 8080 (or the next free port) on the host system,
433 allowing access to an http server running in QEMU from
434 ``http://<host ip>:8080/``.
435
436 The second example does the same, but also forwards TCP port 22 on the
437 guest system to 2222 (or the next free port) on the host system,
438 allowing ssh access to the emulated system using
439 ``ssh -P 2222 <user>@<host ip>``.
440
441 Keep in mind that proper configuration of firewall software is required.
442
443- ``kvm``: Enables KVM when running "qemux86" or "qemux86-64" QEMU
444 architectures. For KVM to work, all the following conditions must be
445 met:
446
447 - Your MACHINE must be either qemux86" or "qemux86-64".
448
449 - Your build host has to have the KVM modules installed, which are
450 ``/dev/kvm``.
451
452 - The build host ``/dev/kvm`` directory has to be both writable and
453 readable.
454
455- ``kvm-vhost``: Enables KVM with VHOST support when running "qemux86"
456 or "qemux86-64" QEMU architectures. For KVM with VHOST to work, the
457 following conditions must be met:
458
459 - ``kvm`` option conditions defined above must be met.
460
461 - Your build host has to have virtio net device, which are
462 ``/dev/vhost-net``.
463
464 - The build host ``/dev/vhost-net`` directory has to be either
465 readable or writable and "slirp-enabled".
466
467- ``publicvnc``: Enables a VNC server open to all hosts.
diff --git a/documentation/dev-manual/quilt.rst b/documentation/dev-manual/quilt.rst
deleted file mode 100644
index 59240705ad..0000000000
--- a/documentation/dev-manual/quilt.rst
+++ /dev/null
@@ -1,89 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using Quilt in Your Workflow
4****************************
5
6`Quilt <https://savannah.nongnu.org/projects/quilt>`__ is a powerful tool
7that allows you to capture source code changes without having a clean
8source tree. This section outlines the typical workflow you can use to
9modify source code, test changes, and then preserve the changes in the
10form of a patch all using Quilt.
11
12.. note::
13
14 With regard to preserving changes to source files, if you clean a
15 recipe or have :ref:`ref-classes-rm-work` enabled, the
16 :ref:`devtool workflow <sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow>`
17 as described in the Yocto Project Application Development and the
18 Extensible Software Development Kit (eSDK) manual is a safer
19 development flow than the flow that uses Quilt.
20
21Follow these general steps:
22
23#. *Find the Source Code:* Temporary source code used by the
24 OpenEmbedded build system is kept in the :term:`Build Directory`. See the
25 ":ref:`dev-manual/temporary-source-code:finding temporary source code`" section to
26 learn how to locate the directory that has the temporary source code for a
27 particular package.
28
29#. *Change Your Working Directory:* You need to be in the directory that
30 has the temporary source code. That directory is defined by the
31 :term:`S` variable.
32
33#. *Create a New Patch:* Before modifying source code, you need to
34 create a new patch. To create a new patch file, use ``quilt new`` as
35 below::
36
37 $ quilt new my_changes.patch
38
39#. *Notify Quilt and Add Files:* After creating the patch, you need to
40 notify Quilt about the files you plan to edit. You notify Quilt by
41 adding the files to the patch you just created::
42
43 $ quilt add file1.c file2.c file3.c
44
45#. *Edit the Files:* Make your changes in the source code to the files
46 you added to the patch.
47
48#. *Test Your Changes:* Once you have modified the source code, the
49 easiest way to test your changes is by calling the :ref:`ref-tasks-compile`
50 task as shown in the following example::
51
52 $ bitbake -c compile -f package
53
54 The ``-f`` or ``--force`` option forces the specified task to
55 execute. If you find problems with your code, you can just keep
56 editing and re-testing iteratively until things work as expected.
57
58 .. note::
59
60 All the modifications you make to the temporary source code disappear
61 once you run the :ref:`ref-tasks-clean` or :ref:`ref-tasks-cleanall`
62 tasks using BitBake (i.e. ``bitbake -c clean package`` and
63 ``bitbake -c cleanall package``). Modifications will also disappear if
64 you use the :ref:`ref-classes-rm-work` feature as described in
65 the ":ref:`dev-manual/disk-space:conserving disk space during builds`"
66 section.
67
68#. *Generate the Patch:* Once your changes work as expected, you need to
69 use Quilt to generate the final patch that contains all your
70 modifications::
71
72 $ quilt refresh
73
74 At this point, the
75 ``my_changes.patch`` file has all your edits made to the ``file1.c``,
76 ``file2.c``, and ``file3.c`` files.
77
78 You can find the resulting patch file in the ``patches/``
79 subdirectory of the source (:term:`S`) directory.
80
81#. *Copy the Patch File:* For simplicity, copy the patch file into a
82 directory named ``files``, which you can create in the same directory
83 that holds the recipe (``.bb``) file or the append (``.bbappend``)
84 file. Placing the patch here guarantees that the OpenEmbedded build
85 system will find the patch. Next, add the patch into the :term:`SRC_URI`
86 of the recipe. Here is an example::
87
88 SRC_URI += "file://my_changes.patch"
89
diff --git a/documentation/dev-manual/read-only-rootfs.rst b/documentation/dev-manual/read-only-rootfs.rst
deleted file mode 100644
index 251178ed54..0000000000
--- a/documentation/dev-manual/read-only-rootfs.rst
+++ /dev/null
@@ -1,89 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Read-Only Root Filesystem
4************************************
5
6Suppose, for security reasons, you need to disable your target device's
7root filesystem's write permissions (i.e. you need a read-only root
8filesystem). Or, perhaps you are running the device's operating system
9from a read-only storage device. For either case, you can customize your
10image for that behavior.
11
12.. note::
13
14 Supporting a read-only root filesystem requires that the system and
15 applications do not try to write to the root filesystem. You must
16 configure all parts of the target system to write elsewhere, or to
17 gracefully fail in the event of attempting to write to the root
18 filesystem.
19
20Creating the Root Filesystem
21============================
22
23To create the read-only root filesystem, simply add the
24"read-only-rootfs" feature to your image, normally in one of two ways.
25The first way is to add the "read-only-rootfs" image feature in the
26image's recipe file via the :term:`IMAGE_FEATURES` variable::
27
28 IMAGE_FEATURES += "read-only-rootfs"
29
30As an alternative, you can add the same feature
31from within your :term:`Build Directory`'s ``local.conf`` file with the
32associated :term:`EXTRA_IMAGE_FEATURES` variable, as in::
33
34 EXTRA_IMAGE_FEATURES = "read-only-rootfs"
35
36For more information on how to use these variables, see the
37":ref:`dev-manual/customizing-images:Customizing Images Using Custom \`\`IMAGE_FEATURES\`\` and \`\`EXTRA_IMAGE_FEATURES\`\``"
38section. For information on the variables, see
39:term:`IMAGE_FEATURES` and
40:term:`EXTRA_IMAGE_FEATURES`.
41
42Post-Installation Scripts and Read-Only Root Filesystem
43=======================================================
44
45It is very important that you make sure all post-Installation
46(``pkg_postinst``) scripts for packages that are installed into the
47image can be run at the time when the root filesystem is created during
48the build on the host system. These scripts cannot attempt to run during
49the first boot on the target device. With the "read-only-rootfs" feature
50enabled, the build system makes sure that all post-installation scripts
51succeed at file system creation time. If any of these scripts
52still need to be run after the root filesystem is created, the build
53immediately fails. These build-time checks ensure that the build fails
54rather than the target device fails later during its initial boot
55operation.
56
57Most of the common post-installation scripts generated by the build
58system for the out-of-the-box Yocto Project are engineered so that they
59can run during root filesystem creation (e.g. post-installation scripts
60for caching fonts). However, if you create and add custom scripts, you
61need to be sure they can be run during this file system creation.
62
63Here are some common problems that prevent post-installation scripts
64from running during root filesystem creation:
65
66- *Not using $D in front of absolute paths:* The build system defines
67 ``$``\ :term:`D` when the root
68 filesystem is created. Furthermore, ``$D`` is blank when the script
69 is run on the target device. This implies two purposes for ``$D``:
70 ensuring paths are valid in both the host and target environments,
71 and checking to determine which environment is being used as a method
72 for taking appropriate actions.
73
74- *Attempting to run processes that are specific to or dependent on the
75 target architecture:* You can work around these attempts by using
76 native tools, which run on the host system, to accomplish the same
77 tasks, or by alternatively running the processes under QEMU, which
78 has the ``qemu_run_binary`` function. For more information, see the
79 :ref:`ref-classes-qemu` class.
80
81Areas With Write Access
82=======================
83
84With the "read-only-rootfs" feature enabled, any attempt by the target
85to write to the root filesystem at runtime fails. Consequently, you must
86make sure that you configure processes and applications that attempt
87these types of writes do so to directories with write access (e.g.
88``/tmp`` or ``/var/run``).
89
diff --git a/documentation/dev-manual/sbom.rst b/documentation/dev-manual/sbom.rst
deleted file mode 100644
index ca0fc8b9d6..0000000000
--- a/documentation/dev-manual/sbom.rst
+++ /dev/null
@@ -1,94 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating a Software Bill of Materials
4*************************************
5
6Once you are able to build an image for your project, once the licenses for
7each software component are all identified (see
8":ref:`dev-manual/licenses:working with licenses`") and once vulnerability
9fixes are applied (see ":ref:`dev-manual/vulnerabilities:checking
10for vulnerabilities`"), the OpenEmbedded build system can generate
11a description of all the components you used, their licenses, their dependencies,
12their sources, the changes that were applied to them and the known
13vulnerabilities that were fixed.
14
15This description is generated in the form of a *Software Bill of Materials*
16(:term:`SBOM`), using the :term:`SPDX` standard.
17
18When you release software, this is the most standard way to provide information
19about the Software Supply Chain of your software image and SDK. The
20:term:`SBOM` tooling is often used to ensure open source license compliance by
21providing the license texts used in the product which legal departments and end
22users can read in standardized format.
23
24:term:`SBOM` information is also critical to performing vulnerability exposure
25assessments, as all the components used in the Software Supply Chain are listed.
26
27The OpenEmbedded build system doesn't generate such information by default,
28though the :term:`Poky` reference distribution has it enabled out of the box.
29
30To enable it, inherit the :ref:`ref-classes-create-spdx` class from a
31configuration file::
32
33 INHERIT += "create-spdx"
34
35In the :term:`Poky` reference distribution, :term:`SPDX` generation does
36consume some build time resources and thus if needed it can be disabled from a
37:term:`configuration file`::
38
39 INHERIT:remove = "create-spdx"
40
41Upon building an image, you will then get:
42
43- :term:`SPDX` output in JSON format as an ``IMAGE-MACHINE.spdx.json`` file in
44 ``tmp/deploy/images/MACHINE/`` inside the :term:`Build Directory`.
45
46- This toplevel file is accompanied by an ``IMAGE-MACHINE.spdx.index.json``
47 containing an index of JSON :term:`SPDX` files for individual recipes.
48
49- The compressed archive ``IMAGE-MACHINE.spdx.tar.zst`` contains the index
50 and the files for the single recipes.
51
52The :ref:`ref-classes-create-spdx` class offers options to include
53more information in the output :term:`SPDX` data:
54
55- Make the json files more human readable by setting (:term:`SPDX_PRETTY`).
56
57- Add compressed archives of the files in the generated target packages by
58 setting (:term:`SPDX_ARCHIVE_PACKAGED`).
59
60- Add a description of the source files used to generate host tools and target
61 packages (:term:`SPDX_INCLUDE_SOURCES`)
62
63- Add a description of the **compiled** source files used to generate host tools
64 and target packages (:term:`SPDX_INCLUDE_COMPILED_SOURCES`)
65
66- Add archives of these source files themselves (:term:`SPDX_ARCHIVE_SOURCES`).
67
68Though the toplevel :term:`SPDX` output is available in
69``tmp/deploy/images/MACHINE/`` inside the :term:`Build Directory`, ancillary
70generated files are available in ``tmp/deploy/spdx/MACHINE`` too, such as:
71
72- The individual :term:`SPDX` JSON files in the ``IMAGE-MACHINE.spdx.tar.zst``
73 archive.
74
75- Compressed archives of the files in the generated target packages,
76 in ``packages/packagename.tar.zst`` (when :term:`SPDX_ARCHIVE_PACKAGED`
77 is set).
78
79- Compressed archives of the source files used to build the host tools
80 and the target packages in ``recipes/recipe-packagename.tar.zst``
81 (when :term:`SPDX_ARCHIVE_SOURCES` is set). Those are needed to fulfill
82 "source code access" license requirements.
83
84See also the :term:`SPDX_CUSTOM_ANNOTATION_VARS` variable which allows
85to associate custom notes to a recipe.
86See the `tools page <https://spdx.dev/resources/tools/>`__ on the :term:`SPDX`
87project website for a list of tools to consume and transform the :term:`SPDX`
88data generated by the OpenEmbedded build system.
89
90See also Joshua Watt's presentations
91`Automated SBoM generation with OpenEmbedded and the Yocto Project <https://youtu.be/Q5UQUM6zxVU>`__
92at FOSDEM 2023 and
93`SPDX in the Yocto Project <https://fosdem.org/2024/schedule/event/fosdem-2024-3318-spdx-in-the-yocto-project/>`__
94at FOSDEM 2024.
diff --git a/documentation/dev-manual/securing-images.rst b/documentation/dev-manual/securing-images.rst
deleted file mode 100644
index f4b528e559..0000000000
--- a/documentation/dev-manual/securing-images.rst
+++ /dev/null
@@ -1,156 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Making Images More Secure
4*************************
5
6Security is of increasing concern for embedded devices. Consider the
7issues and problems discussed in just this sampling of work found across
8the Internet:
9
10- *"*\ `Security Risks of Embedded
11 Systems <https://www.schneier.com/blog/archives/2014/01/security_risks_9.html>`__\ *"*
12 by Bruce Schneier
13
14- *"*\ `Internet Census
15 2012 <http://census2012.sourceforge.net/paper.html>`__\ *"* by Carna
16 Botnet
17
18- *"*\ `Security Issues for Embedded
19 Devices <https://elinux.org/images/6/6f/Security-issues.pdf>`__\ *"*
20 by Jake Edge
21
22When securing your image is of concern, there are steps, tools, and
23variables that you can consider to help you reach the security goals you
24need for your particular device. Not all situations are identical when
25it comes to making an image secure. Consequently, this section provides
26some guidance and suggestions for consideration when you want to make
27your image more secure.
28
29.. note::
30
31 Because the security requirements and risks are different for every
32 type of device, this section cannot provide a complete reference on
33 securing your custom OS. It is strongly recommended that you also
34 consult other sources of information on embedded Linux system
35 hardening and on security.
36
37General Considerations
38======================
39
40There are general considerations that help you create more secure images.
41You should consider the following suggestions to make your device
42more secure:
43
44- Scan additional code you are adding to the system (e.g. application
45 code) by using static analysis tools. Look for buffer overflows and
46 other potential security problems.
47
48- Pay particular attention to the security for any web-based
49 administration interface.
50
51 Web interfaces typically need to perform administrative functions and
52 tend to need to run with elevated privileges. Thus, the consequences
53 resulting from the interface's security becoming compromised can be
54 serious. Look for common web vulnerabilities such as
55 cross-site-scripting (XSS), unvalidated inputs, and so forth.
56
57 As with system passwords, the default credentials for accessing a
58 web-based interface should not be the same across all devices. This
59 is particularly true if the interface is enabled by default as it can
60 be assumed that many end-users will not change the credentials.
61
62- Ensure you can update the software on the device to mitigate
63 vulnerabilities discovered in the future. This consideration
64 especially applies when your device is network-enabled.
65
66- Regularly scan and apply fixes for CVE security issues affecting
67 all software components in the product, see ":ref:`dev-manual/vulnerabilities:checking for vulnerabilities`".
68
69- Regularly update your version of Poky and OE-Core from their upstream
70 developers, e.g. to apply updates and security fixes from stable
71 and :term:`LTS` branches.
72
73- Ensure you remove or disable debugging functionality before producing
74 the final image. For information on how to do this, see the
75 ":ref:`dev-manual/securing-images:considerations specific to the openembedded build system`"
76 section.
77
78- Ensure you have no network services listening that are not needed.
79
80- Remove any software from the image that is not needed.
81
82- Enable hardware support for secure boot functionality when your
83 device supports this functionality.
84
85Security Flags
86==============
87
88The Yocto Project has security flags that you can enable that help make
89your build output more secure. The security flags are in the
90``meta/conf/distro/include/security_flags.inc`` file in your
91:term:`Source Directory` (e.g. ``poky``).
92
93.. note::
94
95 Depending on the recipe, certain security flags are enabled and
96 disabled by default.
97
98Use the following line in your ``local.conf`` file or in your custom
99distribution configuration file to enable the security compiler and
100linker flags for your build::
101
102 require conf/distro/include/security_flags.inc
103
104Considerations Specific to the OpenEmbedded Build System
105========================================================
106
107You can take some steps that are specific to the OpenEmbedded build
108system to make your images more secure:
109
110- Ensure that "allow-empty-password", "allow-root-login", or
111 "empty-root-password" are not one of your selected :term:`IMAGE_FEATURES`.
112 When creating a new project, the default is to provide you with an
113 initial ``local.conf`` file that enables these features using the
114 :term:`EXTRA_IMAGE_FEATURES`
115 variable with the line::
116
117 EXTRA_IMAGE_FEATURES = "allow-empty-password empty-root-password allow-root-login"
118
119 To disable these features, simply comment out that line in your
120 ``local.conf`` file, or make sure :term:`IMAGE_FEATURES` does not contain
121 any of these features before producing your final image. Among other things,
122 leaving this in place sets the root password as blank, which makes
123 logging in for debugging or inspection easy during development but
124 also means anyone can easily log in during production.
125
126- It is possible to set a root password for the image and also to set
127 passwords for any extra users you might add (e.g. administrative or
128 service type users). When you set up passwords for multiple images or
129 users, you should not duplicate passwords.
130
131 To set up passwords, use the :ref:`ref-classes-extrausers` class, which
132 is the preferred method. For an example on how to set up both root and
133 user passwords, see the ":ref:`ref-classes-extrausers`" section.
134
135 .. note::
136
137 When adding extra user accounts or setting a root password, be
138 cautious about setting the same password on every device. If you
139 do this, and the password you have set is exposed, then every
140 device is now potentially compromised. If you need this access but
141 want to ensure security, consider setting a different, random
142 password for each device. Typically, you do this as a separate
143 step after you deploy the image onto the device.
144
145- Consider enabling a Mandatory Access Control (MAC) framework such as
146 SMACK or SELinux and tuning it appropriately for your device's usage.
147 You can find more information in the
148 :yocto_git:`meta-selinux </meta-selinux/>` layer.
149
150Tools for Hardening Your Image
151==============================
152
153The Yocto Project provides tools for making your image more secure. You
154can find these tools in the ``meta-security`` layer of the
155:yocto_git:`Yocto Project Source Repositories <>`.
156
diff --git a/documentation/dev-manual/security-subjects.rst b/documentation/dev-manual/security-subjects.rst
deleted file mode 100644
index 6785b5a16a..0000000000
--- a/documentation/dev-manual/security-subjects.rst
+++ /dev/null
@@ -1,194 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Dealing with Vulnerability Reports
4**********************************
5
6The Yocto Project and OpenEmbedded are open-source, community-based projects
7used in numerous products. They assemble multiple other open-source projects,
8and need to handle security issues and practices both internal (in the code
9maintained by both projects), and external (maintained by other projects and
10organizations).
11
12This manual assembles security-related information concerning the whole
13ecosystem. It includes information on reporting a potential security issue,
14the operation of the YP Security team and how to contribute in the
15related code. It is written to be useful for both security researchers and
16YP developers.
17
18How to report a potential security vulnerability?
19=================================================
20
21If you would like to report a public issue (for example, one with a released
22CVE number), please report it using the
23:yocto_bugs:`Security Bugzilla </enter_bug.cgi?product=Security>`.
24
25If you are dealing with a not-yet-released issue, or an urgent one, please send
26a message to security AT yoctoproject DOT org, including as many details as
27possible: the layer or software module affected, the recipe and its version,
28and any example code, if available. This mailing list is monitored by the
29Yocto Project Security team.
30
31For each layer, you might also look for specific instructions (if any) for
32reporting potential security issues in the specific ``SECURITY.md`` file at the
33root of the repository. Instructions on how and where submit a patch are
34usually available in ``README.md``. If this is your first patch to the
35Yocto Project/OpenEmbedded, you might want to have a look into the
36Contributor's Manual section
37":ref:`contributor-guide/submit-changes:preparing changes for submission`".
38
39Branches maintained with security fixes
40---------------------------------------
41
42See the
43:ref:`Release process <ref-manual/release-process:Stable Release Process>`
44documentation for details regarding the policies and maintenance of stable
45branches.
46
47The :yocto_wiki:`Releases page </Releases>` contains a list
48of all releases of the Yocto Project. Versions in gray are no longer actively
49maintained with security patches, but well-tested patches may still be accepted
50for them for significant issues.
51
52Security-related discussions at the Yocto Project
53-------------------------------------------------
54
55We have set up two security-related emails/mailing lists:
56
57 - Public Mailing List: yocto [dash] security [at] yoctoproject[dot] org
58
59 This is a public mailing list for anyone to subscribe to. This list is an
60 open list to discuss public security issues/patches and security-related
61 initiatives. For more information, including subscription information,
62 please see the :yocto_lists:`yocto-security mailing list info page
63 </g/yocto-security>`.
64
65 This list requires moderator approval for new topics to be posted, to avoid
66 private security reports to be posted by mistake.
67
68 - Yocto Project Security Team: security [at] yoctoproject [dot] org
69
70 This is an email for reporting non-published potential vulnerabilities.
71 Emails sent to this address are forwarded to the Yocto Project Security
72 Team members.
73
74
75What you should do if you find a security vulnerability
76-------------------------------------------------------
77
78If you find a security flaw: a crash, an information leakage, or anything that
79can have a security impact if exploited in any Open Source software built or
80used by the Yocto Project, please report this to the Yocto Project Security
81Team. If you prefer to contact the upstream project directly, please send a
82copy to the security team at the Yocto Project as well. If you believe this is
83highly sensitive information, please report the vulnerability in a secure way,
84i.e. encrypt the email and send it to the private list. This ensures that
85the exploit is not leaked and exploited before a response/fix has been generated.
86
87Security team
88=============
89
90The Yocto Project/OpenEmbedded security team coordinates the work on security
91subjects in the project. All general discussion takes place publicly. The
92Security Team only uses confidential communication tools to deal with private
93vulnerability reports before they are released.
94
95Security team appointment
96-------------------------
97
98The Yocto Project Security Team consists of at least three members. When new
99members are needed, the Yocto Project Technical Steering Committee (YP TSC)
100asks for nominations by public channels including a nomination deadline.
101Self-nominations are possible. When the limit time is
102reached, the YP TSC posts the list of candidates for the comments of project
103participants and developers. Comments may be sent publicly or privately to the
104YP and OE TSCs. The candidates are approved by both YP TSC and OpenEmbedded
105Technical Steering Committee (OE TSC) and the final list of the team members
106is announced publicly. The aim is to have people representing technical
107leadership, security knowledge and infrastructure present with enough people
108to provide backup/coverage but keep the notification list small enough to
109minimize information risk and maintain trust.
110
111YP Security Team members may resign at any time.
112
113Security Team Operations
114------------------------
115
116The work of the Security Team might require high confidentiality. Team members
117are individuals selected by merit and do not represent the companies they work
118for. They do not share information about confidential issues outside of the team
119and do not hint about ongoing embargoes.
120
121Team members can bring in domain experts as needed. Those people should be
122added to individual issues only and adhere to the same standards as the YP
123Security Team.
124
125The YP security team organizes its meetings and communication as needed.
126
127When the YP Security team receives a report about a potential security
128vulnerability, they quickly analyze and notify the reporter of the result.
129They might also request more information.
130
131If the issue is confirmed and affects the code maintained by the YP, they
132confidentially notify maintainers of that code and work with them to prepare
133a fix.
134
135If the issue is confirmed and affects an upstream project, the YP security team
136notifies the project. Usually, the upstream project analyzes the problem again.
137If they deem it a real security problem in their software, they develop and
138release a fix following their security policy. They may want to include the
139original reporter in the loop. There is also sometimes some coordination for
140handling patches, backporting patches etc, or just understanding the problem
141or what caused it.
142
143When the fix is publicly available, the YP security team member or the
144package maintainer sends patches against the YP code base, following usual
145procedures, including public code review.
146
147What Yocto Security Team does when it receives a security vulnerability
148-----------------------------------------------------------------------
149
150The YP Security Team team performs a quick analysis and would usually report
151the flaw to the upstream project. Normally the upstream project analyzes the
152problem. If they deem it a real security problem in their software, they
153develop and release a fix following their own security policy. They may want
154to include the original reporter in the loop. There is also sometimes some
155coordination for handling patches, backporting patches etc, or just
156understanding the problem or what caused it.
157
158The security policy of the upstream project might include a notification to
159Linux distributions or other important downstream projects in advance to
160discuss coordinated disclosure. These mailing lists are normally non-public.
161
162When the upstream project releases a version with the fix, they are responsible
163for contacting `Mitre <https://www.cve.org/>`__ to get a CVE number assigned and
164the CVE record published.
165
166If an upstream project does not respond quickly
167-----------------------------------------------
168
169If an upstream project does not fix the problem in a reasonable time,
170the Yocto's Security Team will contact other interested parties (usually
171other distributions) in the community and together try to solve the
172vulnerability as quickly as possible.
173
174The Yocto Project Security team adheres to the 90 days disclosure policy
175by default. An increase of the embargo time is possible when necessary.
176
177Current Security Team members
178-----------------------------
179
180For secure communications, please send your messages encrypted using the GPG
181keys. Remember, message headers are not encrypted so do not include sensitive
182information in the subject line.
183
184 - Ross Burton: <ross@burtonini.com> `Public key <https://keys.openpgp.org/search?q=ross%40burtonini.com>`__
185
186 - Michael Halstead: <mhalstead [at] linuxfoundation [dot] org>
187 `Public key <https://pgp.mit.edu/pks/lookup?op=vindex&search=0x3373170601861969>`__
188 or `Public key <https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xd1f2407285e571ed12a407a73373170601861969>`__
189
190 - Richard Purdie: <richard.purdie@linuxfoundation.org> `Public key <https://keys.openpgp.org/search?q=richard.purdie%40linuxfoundation.org>`__
191
192 - Marta Rybczynska: <marta DOT rybczynska [at] syslinbit [dot] com> `Public key <https://keys.openpgp.org/search?q=marta.rybczynska@syslinbit.com>`__
193
194 - Steve Sakoman: <steve [at] sakoman [dot] com> `Public key <https://keys.openpgp.org/search?q=steve%40sakoman.com>`__
diff --git a/documentation/dev-manual/speeding-up-build.rst b/documentation/dev-manual/speeding-up-build.rst
deleted file mode 100644
index 6e0d7873ac..0000000000
--- a/documentation/dev-manual/speeding-up-build.rst
+++ /dev/null
@@ -1,109 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Speeding Up a Build
4*******************
5
6Build time can be an issue. By default, the build system uses simple
7controls to try and maximize build efficiency. In general, the default
8settings for all the following variables result in the most efficient
9build times when dealing with single socket systems (i.e. a single CPU).
10If you have multiple CPUs, you might try increasing the default values
11to gain more speed. See the descriptions in the glossary for each
12variable for more information:
13
14- :term:`BB_NUMBER_THREADS`:
15 The maximum number of threads BitBake simultaneously executes.
16
17- :term:`BB_NUMBER_PARSE_THREADS`:
18 The number of threads BitBake uses during parsing.
19
20- :term:`PARALLEL_MAKE`: Extra
21 options passed to the ``make`` command during the
22 :ref:`ref-tasks-compile` task in
23 order to specify parallel compilation on the local build host.
24
25- :term:`PARALLEL_MAKEINST`:
26 Extra options passed to the ``make`` command during the
27 :ref:`ref-tasks-install` task in
28 order to specify parallel installation on the local build host.
29
30As mentioned, these variables all scale to the number of processor cores
31available on the build system. For single socket systems, this
32auto-scaling ensures that the build system fundamentally takes advantage
33of potential parallel operations during the build based on the build
34machine's capabilities.
35
36Additional factors that can affect build speed are:
37
38- File system type: The file system type that the build is being
39 performed on can also influence performance. Using ``ext4`` is
40 recommended as compared to ``ext2`` and ``ext3`` due to ``ext4``
41 improved features such as extents.
42
43- Disabling the updating of access time using ``noatime``: The
44 ``noatime`` mount option prevents the build system from updating file
45 and directory access times.
46
47- Setting a longer commit: Using the "commit=" mount option increases
48 the interval in seconds between disk cache writes. Changing this
49 interval from the five second default to something longer increases
50 the risk of data loss but decreases the need to write to the disk,
51 thus increasing the build performance.
52
53- Choosing the packaging backend: Of the available packaging backends,
54 IPK is the fastest. Additionally, selecting a singular packaging
55 backend also helps.
56
57- Using ``tmpfs`` for :term:`TMPDIR`
58 as a temporary file system: While this can help speed up the build,
59 the benefits are limited due to the compiler using ``-pipe``. The
60 build system goes to some lengths to avoid ``sync()`` calls into the
61 file system on the principle that if there was a significant failure,
62 the :term:`Build Directory` contents could easily be rebuilt.
63
64- Inheriting the :ref:`ref-classes-rm-work` class:
65 Inheriting this class has shown to speed up builds due to
66 significantly lower amounts of data stored in the data cache as well
67 as on disk. Inheriting this class also makes cleanup of
68 :term:`TMPDIR` faster, at the
69 expense of being easily able to dive into the source code. File
70 system maintainers have recommended that the fastest way to clean up
71 large numbers of files is to reformat partitions rather than delete
72 files due to the linear nature of partitions. This, of course,
73 assumes you structure the disk partitions and file systems in a way
74 that this is practical.
75
76Aside from the previous list, you should keep some trade offs in mind
77that can help you speed up the build:
78
79- Remove items from
80 :term:`DISTRO_FEATURES`
81 that you might not need.
82
83- Exclude debug symbols and other debug information: If you do not need
84 these symbols and other debug information, disabling the ``*-dbg``
85 package generation can speed up the build. You can disable this
86 generation by setting the
87 :term:`INHIBIT_PACKAGE_DEBUG_SPLIT`
88 variable to "1".
89
90- Disable static library generation for recipes derived from
91 ``autoconf`` or ``libtool``: Here is an example showing how to
92 disable static libraries and still provide an override to handle
93 exceptions::
94
95 STATICLIBCONF = "--disable-static"
96 STATICLIBCONF:sqlite3-native = ""
97 EXTRA_OECONF += "${STATICLIBCONF}"
98
99 .. note::
100
101 - Some recipes need static libraries in order to work correctly
102 (e.g. ``pseudo-native`` needs ``sqlite3-native``). Overrides,
103 as in the previous example, account for these kinds of
104 exceptions.
105
106 - Some packages have packaging code that assumes the presence of
107 the static libraries. If so, you might need to exclude them as
108 well.
109
diff --git a/documentation/dev-manual/start.rst b/documentation/dev-manual/start.rst
deleted file mode 100644
index 2a1be7260e..0000000000
--- a/documentation/dev-manual/start.rst
+++ /dev/null
@@ -1,877 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3***********************************
4Setting Up to Use the Yocto Project
5***********************************
6
7This chapter provides guidance on how to prepare to use the Yocto
8Project. You can learn about creating a team environment to develop
9using the Yocto Project, how to set up a :ref:`build
10host <dev-manual/start:preparing the build host>`, how to locate
11Yocto Project source repositories, and how to create local Git
12repositories.
13
14Creating a Team Development Environment
15=======================================
16
17It might not be immediately clear how you can use the Yocto Project in a
18team development environment, or how to scale it for a large team of
19developers. You can adapt the Yocto Project to many different use cases
20and scenarios; however, this flexibility could cause difficulties if you
21are trying to create a working setup that scales effectively.
22
23To help you understand how to set up this type of environment, this
24section presents a procedure that gives you information that can help
25you get the results you want. The procedure is high-level and presents
26some of the project's most successful experiences, practices, solutions,
27and available technologies that have proved to work well in the past;
28however, keep in mind, the procedure here is simply a starting point.
29You can build off these steps and customize the procedure to fit any
30particular working environment and set of practices.
31
32#. *Determine Who is Going to be Developing:* You first need to
33 understand who is going to be doing anything related to the Yocto
34 Project and determine their roles. Making this determination is
35 essential to completing subsequent steps, which are to get your
36 equipment together and set up your development environment's
37 hardware topology.
38
39 Possible roles are:
40
41 - *Application Developer:* This type of developer does application
42 level work on top of an existing software stack.
43
44 - *Core System Developer:* This type of developer works on the
45 contents of the operating system image itself.
46
47 - *Build Engineer:* This type of developer manages Autobuilders and
48 releases. Depending on the specifics of the environment, not all
49 situations might need a Build Engineer.
50
51 - *Test Engineer:* This type of developer creates and manages
52 automated tests that are used to ensure all application and core
53 system development meets desired quality standards.
54
55#. *Gather the Hardware:* Based on the size and make-up of the team,
56 get the hardware together. Ideally, any development, build, or test
57 engineer uses a system that runs a supported Linux distribution.
58 These systems, in general, should be high performance (e.g. dual,
59 six-core Xeons with 24 Gbytes of RAM and plenty of disk space). You
60 can help ensure efficiency by having any machines used for testing
61 or that run Autobuilders be as high performance as possible.
62
63 .. note::
64
65 Given sufficient processing power, you might also consider
66 building Yocto Project development containers to be run under
67 Docker, which is described later.
68
69#. *Understand the Hardware Topology of the Environment:* Once you
70 understand the hardware involved and the make-up of the team, you
71 can understand the hardware topology of the development environment.
72 You can get a visual idea of the machines and their roles across the
73 development environment.
74
75#. *Use Git as Your Source Control Manager (SCM):* Keeping your
76 :term:`Metadata` (i.e. recipes,
77 configuration files, classes, and so forth) and any software you are
78 developing under the control of an SCM system that is compatible
79 with the OpenEmbedded build system is advisable. Of all of the SCMs
80 supported by BitBake, the Yocto Project team strongly recommends using
81 :ref:`overview-manual/development-environment:git`.
82 Git is a distributed system
83 that is easy to back up, allows you to work remotely, and then
84 connects back to the infrastructure.
85
86 .. note::
87
88 For information about BitBake, see the
89 :doc:`bitbake:index`.
90
91 It is relatively easy to set up Git services and create infrastructure like
92 :yocto_git:`/`, which is based on server software called
93 `Gitolite <https://gitolite.com>`__
94 with `cgit <https://git.zx2c4.com/cgit/about/>`__ being used to
95 generate the web interface that lets you view the repositories.
96 ``gitolite`` identifies users using SSH keys and allows
97 branch-based access controls to repositories that you can control as
98 little or as much as necessary.
99
100#. *Set up the Application Development Machines:* As mentioned earlier,
101 application developers are creating applications on top of existing
102 software stacks. Here are some best practices for setting up
103 machines used for application development:
104
105 - Use a pre-built toolchain that contains the software stack
106 itself. Then, develop the application code on top of the stack.
107 This method works well for small numbers of relatively isolated
108 applications.
109
110 - Keep your cross-development toolchains updated. You can do this
111 through provisioning either as new toolchain downloads or as
112 updates through a package update mechanism to
113 provide updates to an existing toolchain. The exact mechanics of
114 how and when to do this depend on local policy.
115
116 - Use multiple toolchains installed locally into different
117 locations to allow development across versions.
118
119#. *Set up the Core Development Machines:* As mentioned earlier, core
120 developers work on the contents of the operating system itself.
121 Here are some best practices for setting up machines used for
122 developing images:
123
124 - Have the :term:`OpenEmbedded Build System` available on
125 the developer workstations so developers can run their own builds
126 and directly rebuild the software stack.
127
128 - Keep the core system unchanged as much as possible and do your
129 work in layers on top of the core system. Doing so gives you a
130 greater level of portability when upgrading to new versions of
131 the core system or Board Support Packages (BSPs).
132
133 - Share layers amongst the developers of a particular project and
134 contain the policy configuration that defines the project.
135
136#. *Set up an Autobuilder:* Autobuilders are often the core of the
137 development environment. It is here that changes from individual
138 developers are brought together and centrally tested. Based on this
139 automated build and test environment, subsequent decisions about
140 releases can be made. Autobuilders also allow for "continuous
141 integration" style testing of software components and regression
142 identification and tracking.
143
144 See ":yocto_ab:`Yocto Project Autobuilder <>`" for more
145 information and links to buildbot. The Yocto Project team has found
146 this implementation works well in this role. A public example of
147 this is the Yocto Project Autobuilders, which the Yocto Project team
148 uses to test the overall health of the project.
149
150 The features of this system are:
151
152 - Highlights when commits break the build.
153
154 - Populates an :ref:`sstate
155 cache <overview-manual/concepts:shared state cache>` from which
156 developers can pull rather than requiring local builds.
157
158 - Allows commit hook triggers, which trigger builds when commits
159 are made.
160
161 - Allows triggering of automated image booting and testing under
162 the Quick EMUlator (QEMU).
163
164 - Supports incremental build testing and from-scratch builds.
165
166 - Shares output that allows developer testing and historical
167 regression investigation.
168
169 - Creates output that can be used for releases.
170
171 - Allows scheduling of builds so that resources can be used
172 efficiently.
173
174#. *Set up Test Machines:* Use a small number of shared, high
175 performance systems for testing purposes. Developers can use these
176 systems for wider, more extensive testing while they continue to
177 develop locally using their primary development system.
178
179#. *Document Policies and Change Flow:* The Yocto Project uses a
180 hierarchical structure and a pull model. There are scripts to create and
181 send pull requests (i.e. ``create-pull-request`` and
182 ``send-pull-request``). This model is in line with other open source
183 projects where maintainers are responsible for specific areas of the
184 project and a single maintainer handles the final "top-of-tree"
185 merges.
186
187 .. note::
188
189 You can also use a more collective push model. The ``gitolite``
190 software supports both the push and pull models quite easily.
191
192 As with any development environment, it is important to document the
193 policy used as well as any main project guidelines so they are
194 understood by everyone. It is also a good idea to have
195 well-structured commit messages, which are usually a part of a
196 project's guidelines. Good commit messages are essential when
197 looking back in time and trying to understand why changes were made.
198
199 If you discover that changes are needed to the core layer of the
200 project, it is worth sharing those with the community as soon as
201 possible. Chances are if you have discovered the need for changes,
202 someone else in the community needs them also.
203
204#. *Development Environment Summary:* Aside from the previous steps,
205 here are best practices within the Yocto Project development
206 environment:
207
208 - Use :ref:`overview-manual/development-environment:git` as the source control
209 system.
210
211 - Maintain your Metadata in layers that make sense for your
212 situation. See the ":ref:`overview-manual/yp-intro:the yocto project layer model`"
213 section in the Yocto Project Overview and Concepts Manual and the
214 ":ref:`dev-manual/layers:understanding and creating layers`"
215 section for more information on layers.
216
217 - Separate the project's Metadata and code by using separate Git
218 repositories. See the ":ref:`overview-manual/development-environment:yocto project source repositories`"
219 section in the Yocto Project Overview and Concepts Manual for
220 information on these repositories. See the
221 ":ref:`dev-manual/start:locating yocto project source files`"
222 section for information on how to set up local Git repositories
223 for related upstream Yocto Project Git repositories.
224
225 - Set up the directory for the shared state cache
226 (:term:`SSTATE_DIR`) where
227 it makes sense. For example, set up the sstate cache on a system
228 used by developers in the same organization and share the same
229 source directories on their machines.
230
231 - Set up a local :ref:`overview-manual/concepts:Hash Equivalence` server.
232
233 See the :ref:`overview-manual/concepts:Hash Equivalence` section of the
234 Yocto Project Overview and Concepts Manual for more details on the hash
235 equivalence feature.
236
237 See the :doc:`/dev-manual/hashequivserver` section of the Yocto Project
238 Development Tasks Manual for details on how to setup a
239 :ref:`overview-manual/concepts:Hash Equivalence` server.
240
241 - Set up an Autobuilder and have it populate the sstate cache and
242 source directories.
243
244 - The Yocto Project community encourages you to send patches to the
245 project to fix bugs or add features. If you do submit patches,
246 follow the project commit guidelines for writing good commit
247 messages. See the ":doc:`/contributor-guide/submit-changes`"
248 section in the Yocto Project and OpenEmbedded Contributor Guide.
249
250 - Send changes to the core sooner than later as others are likely
251 to run into the same issues. For some guidance on mailing lists
252 to use, see the lists in the
253 ":ref:`contributor-guide/submit-changes:finding a suitable mailing list`"
254 section. For a description
255 of the available mailing lists, see the ":ref:`resources-mailinglist`" section in
256 the Yocto Project Reference Manual.
257
258Preparing the Build Host
259========================
260
261This section provides procedures to set up a system to be used as your
262:term:`Build Host` for
263development using the Yocto Project. Your build host can be a native
264Linux machine (recommended), it can be a machine (Linux, Mac, or
265Windows) that uses `CROPS <https://github.com/crops/poky-container>`__,
266which leverages `Docker Containers <https://www.docker.com/>`__ or it
267can be a Windows machine capable of running version 2 of Windows Subsystem
268For Linux (WSL 2).
269
270.. note::
271
272 The Yocto Project is not compatible with version 1 of
273 :wikipedia:`Windows Subsystem for Linux <Windows_Subsystem_for_Linux>`.
274 It is compatible but neither officially supported nor validated with
275 WSL 2. If you still decide to use WSL please upgrade to
276 `WSL 2 <https://learn.microsoft.com/en-us/windows/wsl/install>`__.
277
278Once your build host is set up to use the Yocto Project, further steps
279are necessary depending on what you want to accomplish. See the
280following references for information on how to prepare for Board Support
281Package (BSP) development and kernel development:
282
283- *BSP Development:* See the ":ref:`bsp-guide/bsp:preparing your build host to work with bsp layers`"
284 section in the Yocto Project Board Support Package (BSP) Developer's
285 Guide.
286
287- *Kernel Development:* See the ":ref:`kernel-dev/common:preparing the build host to work on the kernel`"
288 section in the Yocto Project Linux Kernel Development Manual.
289
290Setting Up a Native Linux Host
291------------------------------
292
293Follow these steps to prepare a native Linux machine as your Yocto
294Project Build Host:
295
296#. *Use a Supported Linux Distribution:* You should have a reasonably
297 current Linux-based host system. You will have the best results with
298 a recent release of Fedora, openSUSE, Debian, Ubuntu, RHEL or CentOS
299 as these releases are frequently tested against the Yocto Project and
300 officially supported. For a list of the distributions under
301 validation and their status, see the ":ref:`Supported Linux
302 Distributions <system-requirements-supported-distros>`"
303 section in the Yocto Project Reference Manual and the wiki page at
304 :yocto_wiki:`Distribution Support </Distribution_Support>`.
305
306#. *Have Enough Free Memory:* Your system should have at least 50 Gbytes
307 of free disk space for building images.
308
309#. *Meet Minimal Version Requirements:* The OpenEmbedded build system
310 should be able to run on any modern distribution that has the
311 following versions for Git, tar, Python, gcc and make.
312
313 - Git &MIN_GIT_VERSION; or greater
314
315 - tar &MIN_TAR_VERSION; or greater
316
317 - Python &MIN_PYTHON_VERSION; or greater.
318
319 - gcc &MIN_GCC_VERSION; or greater.
320
321 - GNU make &MIN_MAKE_VERSION; or greater
322
323 If your build host does not satisfy all of these listed version
324 requirements, you can take steps to prepare the system so that you
325 can still use the Yocto Project. See the
326 ":ref:`ref-manual/system-requirements:required git, tar, python, make and gcc versions`"
327 section in the Yocto Project Reference Manual for information.
328
329#. *Install Development Host Packages:* Required development host
330 packages vary depending on your build host and what you want to do
331 with the Yocto Project. Collectively, the number of required packages
332 is large if you want to be able to cover all cases.
333
334 For lists of required packages for all scenarios, see the
335 ":ref:`ref-manual/system-requirements:required packages for the build host`"
336 section in the Yocto Project Reference Manual.
337
338Once you have completed the previous steps, you are ready to continue
339using a given development path on your native Linux machine. If you are
340going to use BitBake, see the
341":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`"
342section. If you are going
343to use the Extensible SDK, see the ":doc:`/sdk-manual/extensible`" Chapter in the Yocto
344Project Application Development and the Extensible Software Development
345Kit (eSDK) manual. If you want to work on the kernel, see the :doc:`/kernel-dev/index`. If you are going to use
346Toaster, see the ":doc:`/toaster-manual/setup-and-use`"
347section in the Toaster User Manual. If you are a VSCode user, you can configure
348the `Yocto Project BitBake
349<https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__
350extension accordingly.
351
352Setting Up to Use CROss PlatformS (CROPS)
353-----------------------------------------
354
355With `CROPS <https://github.com/crops/poky-container>`__, which
356leverages `Docker Containers <https://www.docker.com/>`__, you can
357create a Yocto Project development environment that is operating system
358agnostic. You can set up a container in which you can develop using the
359Yocto Project on a Windows, Mac, or Linux machine.
360
361Follow these general steps to prepare a Windows, Mac, or Linux machine
362as your Yocto Project build host:
363
364#. *Determine What Your Build Host Needs:*
365 `Docker <https://www.docker.com/what-docker>`__ is a software
366 container platform that you need to install on the build host.
367 Depending on your build host, you might have to install different
368 software to support Docker containers. Go to the Docker installation
369 page and read about the platform requirements in "`Supported
370 Platforms <https://docs.docker.com/engine/install/#supported-platforms>`__"
371 your build host needs to run containers.
372
373#. *Choose What To Install:* Depending on whether or not your build host
374 meets system requirements, you need to install "Docker CE Stable" or
375 the "Docker Toolbox". Most situations call for Docker CE. However, if
376 you have a build host that does not meet requirements (e.g.
377 Pre-Windows 10 or Windows 10 "Home" version), you must install Docker
378 Toolbox instead.
379
380#. *Go to the Install Site for Your Platform:* Click the link for the
381 Docker edition associated with your build host's native software. For
382 example, if your build host is running Microsoft Windows Version 10
383 and you want the Docker CE Stable edition, click that link under
384 "Supported Platforms".
385
386#. *Install the Software:* Once you have understood all the
387 pre-requisites, you can download and install the appropriate
388 software. Follow the instructions for your specific machine and the
389 type of the software you need to install:
390
391 - Install `Docker Desktop on
392 Windows <https://docs.docker.com/docker-for-windows/install/#install-docker-desktop-on-windows>`__
393 for Windows build hosts that meet requirements.
394
395 - Install `Docker Desktop on
396 MacOs <https://docs.docker.com/docker-for-mac/install/#install-and-run-docker-desktop-on-mac>`__
397 for Mac build hosts that meet requirements.
398
399 - Install `Docker Engine on
400 CentOS <https://docs.docker.com/engine/install/centos/>`__
401 for Linux build hosts running the CentOS distribution.
402
403 - Install `Docker Engine on
404 Debian <https://docs.docker.com/engine/install/debian/>`__
405 for Linux build hosts running the Debian distribution.
406
407 - Install `Docker Engine for
408 Fedora <https://docs.docker.com/engine/install/fedora/>`__
409 for Linux build hosts running the Fedora distribution.
410
411 - Install `Docker Engine for
412 Ubuntu <https://docs.docker.com/engine/install/ubuntu/>`__
413 for Linux build hosts running the Ubuntu distribution.
414
415#. *Optionally Orient Yourself With Docker:* If you are unfamiliar with
416 Docker and the container concept, you can learn more here -
417 https://docs.docker.com/get-started/.
418
419#. *Launch Docker or Docker Toolbox:* You should be able to launch
420 Docker or the Docker Toolbox and have a terminal shell on your
421 development host.
422
423#. *Set Up the Containers to Use the Yocto Project:* Go to
424 https://github.com/crops/docker-win-mac-docs/wiki and follow
425 the directions for your particular build host (i.e. Linux, Mac, or
426 Windows).
427
428 Once you complete the setup instructions for your machine, you have
429 the Poky, Extensible SDK, and Toaster containers available. You can
430 click those links from the page and learn more about using each of
431 those containers.
432
433Once you have a container set up, everything is in place to develop just
434as if you were running on a native Linux machine. If you are going to
435use the Poky container, see the
436":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`"
437section. If you are going to use the Extensible SDK container, see the
438":doc:`/sdk-manual/extensible`" Chapter in the Yocto
439Project Application Development and the Extensible Software Development
440Kit (eSDK) manual. If you are going to use the Toaster container, see
441the ":doc:`/toaster-manual/setup-and-use`"
442section in the Toaster User Manual. If you are a VSCode user, you can configure
443the `Yocto Project BitBake
444<https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__
445extension accordingly.
446
447Setting Up to Use Windows Subsystem For Linux (WSL 2)
448-----------------------------------------------------
449
450With `Windows Subsystem for Linux (WSL 2)
451<https://learn.microsoft.com/en-us/windows/wsl/>`__,
452you can create a Yocto Project development environment that allows you
453to build on Windows. You can set up a Linux distribution inside Windows
454in which you can develop using the Yocto Project.
455
456Follow these general steps to prepare a Windows machine using WSL 2 as
457your Yocto Project build host:
458
459#. *Make sure your Windows machine is capable of running WSL 2:*
460
461 While all Windows 11 and Windows Server 2022 builds support WSL 2,
462 the first versions of Windows 10 and Windows Server 2019 didn't.
463 Check the minimum build numbers for `Windows 10
464 <https://learn.microsoft.com/en-us/windows/wsl/install-manual#step-2---check-requirements-for-running-wsl-2>`__
465 and for `Windows Server 2019
466 <https://learn.microsoft.com/en-us/windows/wsl/install-on-server>`__.
467
468 To check which build version you are running, you may open a command
469 prompt on Windows and execute the command "ver"::
470
471 C:\Users\myuser> ver
472
473 Microsoft Windows [Version 10.0.19041.153]
474
475#. *Install the Linux distribution of your choice inside WSL 2:*
476 Once you know your version of Windows supports WSL 2, you can
477 install the distribution of your choice from the Microsoft Store.
478 Open the Microsoft Store and search for Linux. While there are
479 several Linux distributions available, the assumption is that your
480 pick will be one of the distributions supported by the Yocto Project
481 as stated on the instructions for using a native Linux host. After
482 making your selection, simply click "Get" to download and install the
483 distribution.
484
485#. *Check which Linux distribution WSL 2 is using:* Open a Windows
486 PowerShell and run::
487
488 C:\WINDOWS\system32> wsl -l -v
489 NAME STATE VERSION
490 *Ubuntu Running 2
491
492 Note that WSL 2 supports running as many different Linux distributions
493 as you want to install.
494
495#. *Optionally Get Familiar with WSL:* You can learn more on
496 https://docs.microsoft.com/en-us/windows/wsl/wsl2-about.
497
498#. *Launch your WSL Distibution:* From the Windows start menu simply
499 launch your WSL distribution just like any other application.
500
501#. *Optimize your WSL 2 storage often:* Due to the way storage is
502 handled on WSL 2, the storage space used by the underlying Linux
503 distribution is not reflected immediately, and since BitBake heavily
504 uses storage, after several builds, you may be unaware you are
505 running out of space. As WSL 2 uses a VHDX file for storage, this issue
506 can be easily avoided by regularly optimizing this file in a manual way:
507
508 1. *Find the location of your VHDX file:*
509
510 First you need to find the distro app package directory, to achieve this
511 open a Windows Powershell as Administrator and run::
512
513 C:\WINDOWS\system32> Get-AppxPackage -Name "*Ubuntu*" | Select PackageFamilyName
514 PackageFamilyName
515 -----------------
516 CanonicalGroupLimited.UbuntuonWindows_79abcdefgh
517
518
519 You should now
520 replace the PackageFamilyName and your user on the following path
521 to find your VHDX file::
522
523 ls C:\Users\myuser\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79abcdefgh\LocalState\
524 Mode LastWriteTime Length Name
525 -a---- 3/14/2020 9:52 PM 57418973184 ext4.vhdx
526
527 Your VHDX file path is:
528 ``C:\Users\myuser\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79abcdefgh\LocalState\ext4.vhdx``
529
530 2a. *Optimize your VHDX file using Windows Powershell:*
531
532 To use the ``optimize-vhd`` cmdlet below, first install the Hyper-V
533 option on Windows. Then, open a Windows Powershell as Administrator to
534 optimize your VHDX file, shutting down WSL first::
535
536 C:\WINDOWS\system32> wsl --shutdown
537 C:\WINDOWS\system32> optimize-vhd -Path C:\Users\myuser\AppData\Local\Packages\CanonicalGroupLimited.UbuntuonWindows_79abcdefgh\LocalState\ext4.vhdx -Mode full
538
539 A progress bar should be shown while optimizing the
540 VHDX file, and storage should now be reflected correctly on the
541 Windows Explorer.
542
543 2b. *Optimize your VHDX file using DiskPart:*
544
545 The ``optimize-vhd`` cmdlet noted in step 2a above is provided by
546 Hyper-V. Not all SKUs of Windows can install Hyper-V. As an alternative,
547 use the DiskPart tool. To start, open a Windows command prompt as
548 Administrator to optimize your VHDX file, shutting down WSL first::
549
550 C:\WINDOWS\system32> wsl --shutdown
551 C:\WINDOWS\system32> diskpart
552
553 DISKPART> select vdisk file="<path_to_VHDX_file>"
554 DISKPART> attach vdisk readonly
555 DISKPART> compact vdisk
556 DISKPART> detach
557 DISKPART> exit
558
559.. note::
560
561 The current implementation of WSL 2 does not have out-of-the-box
562 access to external devices such as those connected through a USB
563 port, but it automatically mounts your ``C:`` drive on ``/mnt/c/``
564 (and others), which you can use to share deploy artifacts to be later
565 flashed on hardware through Windows, but your :term:`Build Directory`
566 should not reside inside this mountpoint.
567
568Once you have WSL 2 set up, everything is in place to develop just as if
569you were running on a native Linux machine. If you are going to use the
570Extensible SDK container, see the ":doc:`/sdk-manual/extensible`" Chapter in the Yocto
571Project Application Development and the Extensible Software Development
572Kit (eSDK) manual. If you are going to use the Toaster container, see
573the ":doc:`/toaster-manual/setup-and-use`"
574section in the Toaster User Manual. If you are a VSCode user, you can configure
575the `Yocto Project BitBake
576<https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__
577extension accordingly.
578
579Locating Yocto Project Source Files
580===================================
581
582This section shows you how to locate, fetch, unpack, patch and configure the source
583files you'll need to work with the Yocto Project.
584
585.. note::
586
587 - For concepts and introductory information about Git as it is used
588 in the Yocto Project, see the ":ref:`overview-manual/development-environment:git`"
589 section in the Yocto Project Overview and Concepts Manual.
590
591 - For concepts on Yocto Project source repositories, see the
592 ":ref:`overview-manual/development-environment:yocto project source repositories`"
593 section in the Yocto Project Overview and Concepts Manual."
594
595Accessing Source Repositories
596-----------------------------
597
598Working from a copy of the upstream :ref:`dev-manual/start:accessing source repositories` is the
599preferred method for obtaining and using a Yocto Project release. You
600can view the Yocto Project Source Repositories at
601:yocto_git:`/`. In particular, you can find the ``poky``
602repository at :yocto_git:`/poky`.
603
604Use the following procedure to locate the latest upstream copy of the
605``poky`` Git repository:
606
607#. *Access Repositories:* Open a browser and go to
608 :yocto_git:`/` to access the GUI-based interface into the
609 Yocto Project source repositories.
610
611#. *Select the Repository:* Click on the repository in which you are
612 interested (e.g. ``poky``).
613
614#. *Find the URL Used to Clone the Repository:* At the bottom of the
615 page, note the URL used to clone that repository
616 (e.g. :yocto_git:`/poky`).
617
618 .. note::
619
620 For information on cloning a repository, see the
621 ":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`" section.
622
623Accessing Source Archives
624-------------------------
625
626The Yocto Project also provides source archives of its releases, which
627are available on :yocto_dl:`/releases/yocto/`. Then, choose the subdirectory
628containing the release you wish to use, for example
629:yocto_dl:`&DISTRO_REL_LATEST_TAG; </releases/yocto/&DISTRO_REL_LATEST_TAG;/>`.
630
631You will find there source archives of individual components (if you wish
632to use them individually), and of the corresponding Poky release bundling
633a selection of these components.
634
635.. note::
636
637 The recommended method for accessing Yocto Project components is to
638 use Git to clone the upstream repository and work from within that
639 locally cloned repository.
640
641Using the Downloads Page
642------------------------
643
644The :yocto_home:`Yocto Project Website <>` uses a "RELEASES" page
645from which you can locate and download tarballs of any Yocto Project
646release. Rather than Git repositories, these files represent snapshot
647tarballs similar to the tarballs located in the Index of Releases
648described in the ":ref:`dev-manual/start:accessing source archives`" section.
649
650#. *Go to the Yocto Project Website:* Open The
651 :yocto_home:`Yocto Project Website <>` in your browser.
652
653#. *Get to the Downloads Area:* Select the "RELEASES" item from the
654 pull-down "DEVELOPMENT" tab menu near the top of the page.
655
656#. *Select a Yocto Project Release:* On the top of the "RELEASES" page, currently
657 supported releases are displayed; further down, past supported Yocto Project
658 releases are visible. The "Download" links in the rows of the table there
659 will lead to the download tarballs for the release.
660
661 .. note::
662
663 For a "map" of Yocto Project releases to version numbers, see the
664 :yocto_wiki:`Releases </Releases>` wiki page.
665
666 You can use the "RELEASE ARCHIVE" link to reveal a menu of all Yocto
667 Project releases.
668
669#. *Download Tools or Board Support Packages (BSPs):* Next to the tarballs you
670 will find download tools or BSPs as well. Just select a Yocto Project
671 release and look for what you need.
672
673Cloning and Checking Out Branches
674=================================
675
676To use the Yocto Project for development, you need a release locally
677installed on your development system. This locally installed set of
678files is referred to as the :term:`Source Directory`
679in the Yocto Project documentation.
680
681The preferred method of creating your Source Directory is by using
682:ref:`overview-manual/development-environment:git` to clone a local copy of the upstream
683``poky`` repository. Working from a cloned copy of the upstream
684repository allows you to contribute back into the Yocto Project or to
685simply work with the latest software on a development branch. Because
686Git maintains and creates an upstream repository with a complete history
687of changes and you are working with a local clone of that repository,
688you have access to all the Yocto Project development branches and tag
689names used in the upstream repository.
690
691Cloning the ``poky`` Repository
692-------------------------------
693
694Follow these steps to create a local version of the upstream
695:term:`Poky` Git repository.
696
697#. *Set Your Directory:* Change your working directory to where you want
698 to create your local copy of ``poky``.
699
700#. *Clone the Repository:* The following example command clones the
701 ``poky`` repository and uses the default name "poky" for your local
702 repository::
703
704 $ git clone git://git.yoctoproject.org/poky
705 Cloning into 'poky'...
706 remote: Counting objects: 432160, done.
707 remote: Compressing objects: 100% (102056/102056), done.
708 remote: Total 432160 (delta 323116), reused 432037 (delta 323000)
709 Receiving objects: 100% (432160/432160), 153.81 MiB | 8.54 MiB/s, done.
710 Resolving deltas: 100% (323116/323116), done.
711 Checking connectivity... done.
712
713 Unless you
714 specify a specific development branch or tag name, Git clones the
715 "master" branch, which results in a snapshot of the latest
716 development changes for "master". For information on how to check out
717 a specific development branch or on how to check out a local branch
718 based on a tag name, see the
719 ":ref:`dev-manual/start:checking out by branch in poky`" and
720 ":ref:`dev-manual/start:checking out by tag in poky`" sections, respectively.
721
722 Once the local repository is created, you can change to that
723 directory and check its status. The ``master`` branch is checked out
724 by default::
725
726 $ cd poky
727 $ git status
728 On branch master
729 Your branch is up-to-date with 'origin/master'.
730 nothing to commit, working directory clean
731 $ git branch
732 * master
733
734 Your local repository of poky is initially identical to the
735 upstream poky repository from which it was cloned. As you
736 work with the local branch, you can periodically use the
737 ``git pull`` command to be sure you stay up-to-date
738 with the upstream poky branch.
739
740Checking Out by Branch in Poky
741------------------------------
742
743When you clone the upstream poky repository, you have access to all its
744development branches. Each development branch in a repository is unique
745as it forks off the "master" branch. To see and use the files of a
746particular development branch locally, you need to know the branch name
747and then specifically check out that development branch.
748
749.. note::
750
751 Checking out an active development branch by branch name gives you a
752 snapshot of that particular branch at the time you check it out.
753 Further development on top of the branch that occurs after check it
754 out can occur.
755
756#. *Switch to the Poky Directory:* If you have a local poky Git
757 repository, switch to that directory. If you do not have the local
758 copy of poky, see the
759 ":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`"
760 section.
761
762#. *Determine Existing Branch Names:*
763 ::
764
765 $ git branch -a
766 * master
767 remotes/origin/1.1_M1
768 remotes/origin/1.1_M2
769 remotes/origin/1.1_M3
770 remotes/origin/1.1_M4
771 remotes/origin/1.2_M1
772 remotes/origin/1.2_M2
773 remotes/origin/1.2_M3
774 . . .
775 remotes/origin/thud
776 remotes/origin/thud-next
777 remotes/origin/warrior
778 remotes/origin/warrior-next
779 remotes/origin/zeus
780 remotes/origin/zeus-next
781 ... and so on ...
782
783#. *Check out the Branch:* Check out the development branch in which you
784 want to work. For example, to access the files for the Yocto Project
785 &DISTRO; Release (&DISTRO_NAME;), use the following command::
786
787 $ git checkout -b &DISTRO_NAME_NO_CAP; origin/&DISTRO_NAME_NO_CAP;
788 Branch &DISTRO_NAME_NO_CAP; set up to track remote branch &DISTRO_NAME_NO_CAP; from origin.
789 Switched to a new branch '&DISTRO_NAME_NO_CAP;'
790
791 The previous command checks out the "&DISTRO_NAME_NO_CAP;" development
792 branch and reports that the branch is tracking the upstream
793 "origin/&DISTRO_NAME_NO_CAP;" branch.
794
795 The following command displays the branches that are now part of your
796 local poky repository. The asterisk character indicates the branch
797 that is currently checked out for work::
798
799 $ git branch
800 master
801 * &DISTRO_NAME_NO_CAP;
802
803Checking Out by Tag in Poky
804---------------------------
805
806Similar to branches, the upstream repository uses tags to mark specific
807commits associated with significant points in a development branch (i.e.
808a release point or stage of a release). You might want to set up a local
809branch based on one of those points in the repository. The process is
810similar to checking out by branch name except you use tag names.
811
812.. note::
813
814 Checking out a branch based on a tag gives you a stable set of files
815 not affected by development on the branch above the tag.
816
817#. *Switch to the Poky Directory:* If you have a local poky Git
818 repository, switch to that directory. If you do not have the local
819 copy of poky, see the
820 ":ref:`dev-manual/start:cloning the \`\`poky\`\` repository`"
821 section.
822
823#. *Fetch the Tag Names:* To checkout the branch based on a tag name,
824 you need to fetch the upstream tags into your local repository::
825
826 $ git fetch --tags
827 $
828
829#. *List the Tag Names:* You can list the tag names now::
830
831 $ git tag
832 1.1_M1.final
833 1.1_M1.rc1
834 1.1_M1.rc2
835 1.1_M2.final
836 1.1_M2.rc1
837 .
838 .
839 .
840 yocto-2.5
841 yocto-2.5.1
842 yocto-2.5.2
843 yocto-2.5.3
844 yocto-2.6
845 yocto-2.6.1
846 yocto-2.6.2
847 yocto-2.7
848 yocto_1.5_M5.rc8
849
850
851#. *Check out the Branch:*
852 ::
853
854 $ git checkout tags/yocto-&DISTRO; -b my_yocto_&DISTRO;
855 Switched to a new branch 'my_yocto_&DISTRO;'
856 $ git branch
857 master
858 * my_yocto_&DISTRO;
859
860 The previous command creates and
861 checks out a local branch named "my_yocto_&DISTRO;", which is based on
862 the commit in the upstream poky repository that has the same tag. In
863 this example, the files you have available locally as a result of the
864 ``checkout`` command are a snapshot of the "&DISTRO_NAME_NO_CAP;"
865 development branch at the point where Yocto Project &DISTRO; was
866 released.
867
868Initializing the Build Environment
869==================================
870
871Before you can use Yocto you need to setup the build environment.
872From within the ``poky`` directory, source the :ref:`ref-manual/structure:``oe-init-build-env``` environment
873setup script to define Yocto Project's build environment on your build host::
874
875 $ source oe-init-build-env
876
877Note, that this step will have to be repeated every time you open a new shell.
diff --git a/documentation/dev-manual/temporary-source-code.rst b/documentation/dev-manual/temporary-source-code.rst
deleted file mode 100644
index 9a7cd0f771..0000000000
--- a/documentation/dev-manual/temporary-source-code.rst
+++ /dev/null
@@ -1,73 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Finding Temporary Source Code
4*****************************
5
6You might find it helpful during development to modify the temporary
7source code used by recipes to build packages. For example, suppose you
8are developing a patch and you need to experiment a bit to figure out
9your solution. After you have initially built the package, you can
10iteratively tweak the source code, which is located in the
11:term:`Build Directory`, and then you can force a re-compile and quickly
12test your altered code. Once you settle on a solution, you can then preserve
13your changes in the form of patches.
14
15During a build, the unpacked temporary source code used by recipes to
16build packages is available in the :term:`Build Directory` as defined by the
17:term:`S` variable. Below is the default value for the :term:`S` variable as
18defined in the ``meta/conf/bitbake.conf`` configuration file in the
19:term:`Source Directory`::
20
21 S = "${UNPACKDIR}/${BP}"
22
23You should be aware that many recipes override the
24:term:`S` variable when the default isn't accurate.
25
26.. note::
27
28 The :term:`BP` represents the base recipe name, which consists of the name
29 and version::
30
31 BP = "${BPN}-${PV}"
32
33 This matches the location that the git fetcher unpacks to, and usually
34 matches unpacked content of release tarballs (e.g. they contain a single
35 directory which matches value of ${BP} exactly).
36
37The path to the unpack directory for the recipe
38(:term:`UNPACKDIR`) is defined as follows::
39
40 ${WORKDIR}/sources
41
42In turn, the path to the work directory for the recipe
43(:term:`WORKDIR`) is defined as
44follows::
45
46 ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
47
48The actual directory depends on several things:
49
50- :term:`TMPDIR`: The top-level build
51 output directory.
52
53- :term:`MULTIMACH_TARGET_SYS`:
54 The target system identifier.
55
56- :term:`PN`: The recipe name.
57
58- :term:`EXTENDPE`: The epoch --- if
59 :term:`PE` is not specified, which is
60 usually the case for most recipes, then :term:`EXTENDPE` is blank.
61
62- :term:`PV`: The recipe version.
63
64- :term:`PR`: The recipe revision.
65
66As an example, assume a Source Directory top-level folder named
67``poky``, a default :term:`Build Directory` at ``poky/build``, and a
68``qemux86-poky-linux`` machine target system. Furthermore, suppose your
69recipe is named ``foo_1.3.0.bb``. In this case, the work directory the
70build system uses to build the package would be as follows::
71
72 poky/build/tmp/work/qemux86-poky-linux/foo/1.3.0-r0
73
diff --git a/documentation/dev-manual/upgrading-recipes.rst b/documentation/dev-manual/upgrading-recipes.rst
deleted file mode 100644
index 2e65678679..0000000000
--- a/documentation/dev-manual/upgrading-recipes.rst
+++ /dev/null
@@ -1,397 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Upgrading Recipes
4*****************
5
6Over time, upstream developers publish new versions for software built
7by layer recipes. It is recommended to keep recipes up-to-date with
8upstream version releases.
9
10While there are several methods to upgrade a recipe, you might
11consider checking on the upgrade status of a recipe first. You can do so
12using the ``devtool check-upgrade-status`` command. See the
13":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
14section in the Yocto Project Reference Manual for more information.
15
16The remainder of this section describes three ways you can upgrade a
17recipe. You can use the Automated Upgrade Helper (AUH) to set up
18automatic version upgrades. Alternatively, you can use
19``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
20you can manually upgrade a recipe by editing the recipe itself.
21
22Using the Auto Upgrade Helper (AUH)
23===================================
24
25The AUH utility works in conjunction with the OpenEmbedded build system
26in order to automatically generate upgrades for recipes based on new
27versions being published upstream. Use AUH when you want to create a
28service that performs the upgrades automatically and optionally sends
29you an email with the results.
30
31AUH allows you to update several recipes with a single use. You can also
32optionally perform build and integration tests using images with the
33results saved to your hard drive and emails of results optionally sent
34to recipe maintainers. Finally, AUH creates Git commits with appropriate
35commit messages in the layer's tree for the changes made to recipes.
36
37.. note::
38
39 In some conditions, you should not use AUH to upgrade recipes
40 and should instead use either ``devtool upgrade`` or upgrade your
41 recipes manually:
42
43 - When AUH cannot complete the upgrade sequence. This situation
44 usually results because custom patches carried by the recipe
45 cannot be automatically rebased to the new version. In this case,
46 ``devtool upgrade`` allows you to manually resolve conflicts.
47
48 - When for any reason you want fuller control over the upgrade
49 process. For example, when you want special arrangements for
50 testing.
51
52The following steps describe how to set up the AUH utility:
53
54#. *Be Sure the Development Host is Set Up:* You need to be sure that
55 your development host is set up to use the Yocto Project. For
56 information on how to set up your host, see the
57 ":ref:`dev-manual/start:Preparing the Build Host`" section.
58
59#. *Make Sure Git is Configured:* The AUH utility requires Git to be
60 configured because AUH uses Git to save upgrades. Thus, you must have
61 Git user and email configured. The following command shows your
62 configurations::
63
64 $ git config --list
65
66 If you do not have the user and
67 email configured, you can use the following commands to do so::
68
69 $ git config --global user.name some_name
70 $ git config --global user.email username@domain.com
71
72#. *Clone the AUH Repository:* To use AUH, you must clone the repository
73 onto your development host. The following command uses Git to create
74 a local copy of the repository on your system::
75
76 $ git clone git://git.yoctoproject.org/auto-upgrade-helper
77 Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
78 remote: Compressing objects: 100% (300/300), done.
79 remote: Total 768 (delta 499), reused 703 (delta 434)
80 Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
81 Resolving deltas: 100% (499/499), done.
82 Checking connectivity... done.
83
84 AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
85 :term:`Poky` repositories.
86
87#. *Create a Dedicated Build Directory:* Run the :ref:`structure-core-script`
88 script to create a fresh :term:`Build Directory` that you use exclusively
89 for running the AUH utility::
90
91 $ cd poky
92 $ source oe-init-build-env your_AUH_build_directory
93
94 Re-using an existing :term:`Build Directory` and its configurations is not
95 recommended as existing settings could cause AUH to fail or behave
96 undesirably.
97
98#. *Make Configurations in Your Local Configuration File:* Several
99 settings are needed in the ``local.conf`` file in the build
100 directory you just created for AUH. Make these following
101 configurations:
102
103 - If you want to enable :ref:`Build
104 History <dev-manual/build-quality:maintaining build output quality>`,
105 which is optional, you need the following lines in the
106 ``conf/local.conf`` file::
107
108 INHERIT =+ "buildhistory"
109 BUILDHISTORY_COMMIT = "1"
110
111 With this configuration and a successful
112 upgrade, a build history "diff" file appears in the
113 ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
114 your :term:`Build Directory`.
115
116 - If you want to enable testing through the :ref:`ref-classes-testimage`
117 class, which is optional, you need to have the following set in
118 your ``conf/local.conf`` file::
119
120 IMAGE_CLASSES += "testimage"
121
122 .. note::
123
124 If your distro does not enable by default ptest, which Poky
125 does, you need the following in your ``local.conf`` file::
126
127 DISTRO_FEATURES:append = " ptest"
128
129
130#. *Optionally Start a vncserver:* If you are running in a server
131 without an X11 session, you need to start a vncserver::
132
133 $ vncserver :1
134 $ export DISPLAY=:1
135
136#. *Create and Edit an AUH Configuration File:* You need to have the
137 ``upgrade-helper/upgrade-helper.conf`` configuration file in your
138 :term:`Build Directory`. You can find a sample configuration file in the
139 :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
140
141 Read through the sample file and make configurations as needed. For
142 example, if you enabled build history in your ``local.conf`` as
143 described earlier, you must enable it in ``upgrade-helper.conf``.
144
145 Also, if you are using the default ``maintainers.inc`` file supplied
146 with Poky and located in ``meta-yocto`` and you do not set a
147 "maintainers_whitelist" or "global_maintainer_override" in the
148 ``upgrade-helper.conf`` configuration, and you specify "-e all" on
149 the AUH command-line, the utility automatically sends out emails to
150 all the default maintainers. Please avoid this.
151
152This next set of examples describes how to use the AUH:
153
154- *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
155 following form::
156
157 $ upgrade-helper.py recipe_name
158
159 For example, this command upgrades the ``xmodmap`` recipe::
160
161 $ upgrade-helper.py xmodmap
162
163- *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
164 specific recipe to a particular version, use the following form::
165
166 $ upgrade-helper.py recipe_name -t version
167
168 For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
169
170 $ upgrade-helper.py xmodmap -t 1.2.3
171
172- *Upgrading all Recipes to the Latest Versions and Suppressing Email
173 Notifications:* To upgrade all recipes to their most recent versions
174 and suppress the email notifications, use the following command::
175
176 $ upgrade-helper.py all
177
178- *Upgrading all Recipes to the Latest Versions and Send Email
179 Notifications:* To upgrade all recipes to their most recent versions
180 and send email messages to maintainers for each attempted recipe as
181 well as a status email, use the following command::
182
183 $ upgrade-helper.py -e all
184
185Once you have run the AUH utility, you can find the results in the AUH
186:term:`Build Directory`::
187
188 ${BUILDDIR}/upgrade-helper/timestamp
189
190The AUH utility
191also creates recipe update commits from successful upgrade attempts in
192the layer tree.
193
194You can easily set up to run the AUH utility on a regular basis by using
195a cron job. See the
196:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
197file distributed with the utility for an example.
198
199Using ``devtool upgrade``
200=========================
201
202As mentioned earlier, an alternative method for upgrading recipes to
203newer versions is to use
204:doc:`devtool upgrade </ref-manual/devtool-reference>`.
205You can read about ``devtool upgrade`` in general in the
206":ref:`dev-manual/devtool:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
207section in the Yocto Project Application Development and the Extensible
208Software Development Kit (eSDK) Manual.
209
210To see all the command-line options available with ``devtool upgrade``,
211use the following help command::
212
213 $ devtool upgrade -h
214
215If you want to find out what version a recipe is currently at upstream
216without any attempt to upgrade your local version of the recipe, you can
217use the following command::
218
219 $ devtool latest-version recipe_name
220
221As mentioned in the previous section describing AUH, ``devtool upgrade``
222works in a less-automated manner than AUH. Specifically,
223``devtool upgrade`` only works on a single recipe that you name on the
224command line, cannot perform build and integration testing using images,
225and does not automatically generate commits for changes in the source
226tree. Despite all these "limitations", ``devtool upgrade`` updates the
227recipe file to the new upstream version and attempts to rebase custom
228patches contained by the recipe as needed.
229
230.. note::
231
232 AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
233 of a "wrapper" application for ``devtool upgrade``.
234
235A typical scenario involves having used Git to clone an upstream
236repository that you use during build operations. Because you have built the
237recipe in the past, the layer is likely added to your
238configuration already. If for some reason, the layer is not added, you
239could add it easily using the
240":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
241script. For example, suppose you use the ``nano.bb`` recipe from the
242``meta-oe`` layer in the ``meta-openembedded`` repository. For this
243example, assume that the layer has been cloned into following area::
244
245 /home/scottrif/meta-openembedded
246
247The following command from your :term:`Build Directory` adds the layer to
248your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
249
250 $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
251 NOTE: Starting bitbake server...
252 Parsing recipes: 100% |##########################################| Time: 0:00:55
253 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
254 Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
255 Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
256 Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
257 Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
258
259For this example, assume that the ``nano.bb`` recipe that
260is upstream has a 2.9.3 version number. However, the version in the
261local repository is 2.7.4. The following command from your build
262directory automatically upgrades the recipe for you::
263
264 $ devtool upgrade nano -V 2.9.3
265 NOTE: Starting bitbake server...
266 NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
267 Parsing recipes: 100% |##########################################| Time: 0:00:46
268 Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
269 NOTE: Extracting current version source...
270 NOTE: Resolving any missing task queue dependencies
271 .
272 .
273 .
274 NOTE: Executing SetScene Tasks
275 NOTE: Executing RunQueue Tasks
276 NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
277 Adding changed files: 100% |#####################################| Time: 0:00:00
278 NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
279 NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
280
281.. note::
282
283 Using the ``-V`` option is not necessary. Omitting the version number causes
284 ``devtool upgrade`` to upgrade the recipe to the most recent version.
285
286Continuing with this example, you can use ``devtool build`` to build the
287newly upgraded recipe::
288
289 $ devtool build nano
290 NOTE: Starting bitbake server...
291 Loading cache: 100% |################################################################################################| Time: 0:00:01
292 Loaded 2040 entries from dependency cache.
293 Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
294 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
295 NOTE: Resolving any missing task queue dependencies
296 .
297 .
298 .
299 NOTE: Executing SetScene Tasks
300 NOTE: Executing RunQueue Tasks
301 NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
302 NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
303
304Within the ``devtool upgrade`` workflow, you can
305deploy and test your rebuilt software. For this example,
306however, running ``devtool finish`` cleans up the workspace once the
307source in your workspace is clean. This usually means using Git to stage
308and submit commits for the changes generated by the upgrade process.
309
310Once the tree is clean, you can clean things up in this example with the
311following command from the ``${BUILDDIR}/workspace/sources/nano``
312directory::
313
314 $ devtool finish nano meta-oe
315 NOTE: Starting bitbake server...
316 Loading cache: 100% |################################################################################################| Time: 0:00:00
317 Loaded 2040 entries from dependency cache.
318 Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
319 Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
320 NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
321 NOTE: Updating recipe nano_2.9.3.bb
322 NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
323 NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
324 NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
325
326
327Using the ``devtool finish`` command cleans up the workspace and creates a patch
328file based on your commits. The tool puts all patch files back into the
329source directory in a sub-directory named ``nano`` in this case.
330
331Manually Upgrading a Recipe
332===========================
333
334If for some reason you choose not to upgrade recipes using
335:ref:`dev-manual/upgrading-recipes:Using the Auto Upgrade Helper (AUH)` or
336by :ref:`dev-manual/upgrading-recipes:Using ``devtool upgrade```,
337you can manually edit the recipe files to upgrade the versions.
338
339.. note::
340
341 Manually updating multiple recipes scales poorly and involves many
342 steps. The recommendation to upgrade recipe versions is through AUH
343 or ``devtool upgrade``, both of which automate some steps and provide
344 guidance for others needed for the manual process.
345
346To manually upgrade recipe versions, follow these general steps:
347
348#. *Change the Version:* Rename the recipe such that the version (i.e.
349 the :term:`PV` part of the recipe name)
350 changes appropriately. If the version is not part of the recipe name,
351 change the value as it is set for :term:`PV` within the recipe itself.
352
353#. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
354 is fetched from Git or some other version control system, update
355 :term:`SRCREV` to point to the
356 commit hash that matches the new version.
357
358#. *Build the Software:* Try to build the recipe using BitBake. Typical
359 build failures include the following:
360
361 - License statements were updated for the new version. For this
362 case, you need to review any changes to the license and update the
363 values of :term:`LICENSE` and
364 :term:`LIC_FILES_CHKSUM`
365 as needed.
366
367 .. note::
368
369 License changes are often inconsequential. For example, the
370 license text's copyright year might have changed.
371
372 - Custom patches carried by the older version of the recipe might
373 fail to apply to the new version. For these cases, you need to
374 review the failures. Patches might not be necessary for the new
375 version of the software if the upgraded version has fixed those
376 issues. If a patch is necessary and failing, you need to rebase it
377 into the new version.
378
379#. *Optionally Attempt to Build for Several Architectures:* Once you
380 successfully build the new software for a given architecture, you
381 could test the build for other architectures by changing the
382 :term:`MACHINE` variable and
383 rebuilding the software. This optional step is especially important
384 if the recipe is to be released publicly.
385
386#. *Check the Upstream Change Log or Release Notes:* Checking both these
387 reveals if there are new features that could break
388 backwards-compatibility. If so, you need to take steps to mitigate or
389 eliminate that situation.
390
391#. *Optionally Create a Bootable Image and Test:* If you want, you can
392 test the new software by booting it onto actual hardware.
393
394#. *Create a Commit with the Change in the Layer Repository:* After all
395 builds work and any testing is successful, you can create commits for
396 any changes in the layer holding your upgraded recipe.
397
diff --git a/documentation/dev-manual/vulnerabilities.rst b/documentation/dev-manual/vulnerabilities.rst
deleted file mode 100644
index 3a83a40f31..0000000000
--- a/documentation/dev-manual/vulnerabilities.rst
+++ /dev/null
@@ -1,337 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Checking for Vulnerabilities
4****************************
5
6Vulnerabilities in Poky and OE-Core
7===================================
8
9The Yocto Project has an infrastructure to track and address unfixed
10known security vulnerabilities, as tracked by the public
11:wikipedia:`Common Vulnerabilities and Exposures (CVE) <Common_Vulnerabilities_and_Exposures>`
12database.
13
14The Yocto Project maintains a `list of known vulnerabilities
15<https://valkyrie.yocto.io/pub/non-release/patchmetrics/>`__
16for packages in Poky and OE-Core, tracking the evolution of the number of
17unpatched CVEs and the status of patches. Such information is available for
18the current development version and for each supported release.
19
20Security is a process, not a product, and thus at any time, a number of security
21issues may be impacting Poky and OE-Core. It is up to the maintainers, users,
22contributors and anyone interested in the issues to investigate and possibly fix them by
23updating software components to newer versions or by applying patches to address them.
24It is recommended to work with Poky and OE-Core upstream maintainers and submit
25patches to fix them, see ":doc:`/contributor-guide/submit-changes`" for details.
26
27Vulnerability check at build time
28=================================
29
30To enable a check for CVE security vulnerabilities using
31:ref:`ref-classes-cve-check` in the specific image or target you are building,
32add the following setting to your configuration::
33
34 INHERIT += "cve-check"
35
36The CVE database contains some old incomplete entries which have been
37deemed not to impact Poky or OE-Core. These CVE entries can be excluded from the
38check using build configuration::
39
40 include conf/distro/include/cve-extra-exclusions.inc
41
42With this CVE check enabled, BitBake build will try to map each compiled software component
43recipe name and version information to the CVE database and generate recipe and
44image specific reports. These reports will contain:
45
46- metadata about the software component like names and versions
47
48- metadata about the CVE issue such as description and NVD link
49
50- for each software component, a list of CVEs which are possibly impacting this version
51
52- status of each CVE: ``Patched``, ``Unpatched`` or ``Ignored``
53
54The status ``Patched`` means that a patch file to address the security issue has been
55applied. ``Unpatched`` status means that no patches to address the issue have been
56applied and that the issue needs to be investigated. ``Ignored`` means that after
57analysis, it has been deemed to ignore the issue as it for example affects
58the software component on a different operating system platform.
59
60By default, no NVD API key is used to retrieve data from the CVE database, which
61results in larger delays between NVD API requests. See the :term:`NVDCVE_API_KEY`
62documentation on how to request and set a NVD API key.
63
64After a build with CVE check enabled, reports for each compiled source recipe will be
65found in ``build/tmp/deploy/cve``.
66
67For example the CVE check report for the ``flex-native`` recipe looks like::
68
69 $ cat ./tmp/deploy/cve/flex-native_cve.json
70 {
71 "version": "1",
72 "package": [
73 {
74 "name": "flex-native",
75 "layer": "meta",
76 "version": "2.6.4",
77 "products": [
78 {
79 "product": "flex",
80 "cvesInRecord": "No"
81 },
82 {
83 "product": "flex",
84 "cvesInRecord": "Yes"
85 }
86 ],
87 "issue": [
88 {
89 "id": "CVE-2006-0459",
90 "status": "Patched",
91 "link": "https://nvd.nist.gov/vuln/detail/CVE-2006-0459",
92 "summary": "flex.skl in Will Estes and John Millaway Fast Lexical Analyzer Generator (flex) before 2.5.33 does not allocate enough memory for grammars containing (1) REJECT statements or (2) trailing context rules, which causes flex to generate code that contains a buffer overflow that might allow context-dependent attackers to execute arbitrary code.",
93 "scorev2": "7.5",
94 "scorev3": "0.0",
95 "scorev4": "0.0",
96 "modified": "2024-11-21T00:06Z",
97 "vector": "NETWORK",
98 "vectorString": "AV:N/AC:L/Au:N/C:P/I:P/A:P",
99 "detail": "version-not-in-range"
100 },
101 {
102 "id": "CVE-2016-6354",
103 "status": "Patched",
104 "link": "https://nvd.nist.gov/vuln/detail/CVE-2016-6354",
105 "summary": "Heap-based buffer overflow in the yy_get_next_buffer function in Flex before 2.6.1 might allow context-dependent attackers to cause a denial of service or possibly execute arbitrary code via vectors involving num_to_read.",
106 "scorev2": "7.5",
107 "scorev3": "9.8",
108 "scorev4": "0.0",
109 "modified": "2024-11-21T02:55Z",
110 "vector": "NETWORK",
111 "vectorString": "AV:N/AC:L/Au:N/C:P/I:P/A:P",
112 "detail": "version-not-in-range"
113 },
114 {
115 "id": "CVE-2019-6293",
116 "status": "Ignored",
117 "link": "https://nvd.nist.gov/vuln/detail/CVE-2019-6293",
118 "summary": "An issue was discovered in the function mark_beginning_as_normal in nfa.c in flex 2.6.4. There is a stack exhaustion problem caused by the mark_beginning_as_normal function making recursive calls to itself in certain scenarios involving lots of '*' characters. Remote attackers could leverage this vulnerability to cause a denial-of-service.",
119 "scorev2": "4.3",
120 "scorev3": "5.5",
121 "scorev4": "0.0",
122 "modified": "2024-11-21T04:46Z",
123 "vector": "NETWORK",
124 "vectorString": "AV:N/AC:M/Au:N/C:N/I:N/A:P",
125 "detail": "upstream-wontfix",
126 "description": "there is stack exhaustion but no bug and it is building the parser, not running it, effectively similar to a compiler ICE. Upstream no plans to address this."
127 }
128 ]
129 }
130 ]
131 }
132
133For images, a summary of all recipes included in the image and their CVEs is also
134generated in the JSON format. These ``.json`` reports can be found
135in the ``tmp/deploy/images`` directory for each compiled image.
136
137At build time CVE check will also throw warnings about ``Unpatched`` CVEs::
138
139 WARNING: qemu-native-9.2.0-r0 do_cve_check: Found unpatched CVE (CVE-2023-1386)
140
141It is also possible to check the CVE status of individual packages as follows::
142
143 bitbake -c cve_check flex libarchive
144
145Fixing CVE product name and version mappings
146============================================
147
148By default, :ref:`ref-classes-cve-check` uses the recipe name :term:`BPN` as CVE
149product name when querying the CVE database. If this mapping contains false positives, e.g.
150some reported CVEs are not for the software component in question, or false negatives like
151some CVEs are not found to impact the recipe when they should, then the problems can be
152in the recipe name to CVE product mapping. These mapping issues can be fixed by setting
153the :term:`CVE_PRODUCT` variable inside the recipe. This defines the name of the software component in the
154upstream `NIST CVE database <https://nvd.nist.gov/>`__.
155
156The variable supports using vendor and product names like this::
157
158 CVE_PRODUCT = "flex_project:flex westes:flex"
159
160In this example we have two possible vendors names, ``flex_project`` and ``westes``,
161with the product name ``flex``. With this setting the ``flex`` recipe only maps to this specific
162product and not products from other vendors with same name ``flex``.
163
164Similarly, when the recipe version :term:`PV` is not compatible with software versions used by
165the upstream software component releases and the CVE database, these can be fixed using
166the :term:`CVE_VERSION` variable.
167
168Note that if the CVE entries in the NVD database contain bugs or have missing or incomplete
169information, it is recommended to fix the information there directly instead of working
170around the issues possibly for a long time in Poky and OE-Core side recipes. Feedback to
171NVD about CVE entries can be provided through the `NVD contact form <https://nvd.nist.gov/info/contact-form>`__.
172
173Fixing vulnerabilities in recipes
174=================================
175
176Suppose a CVE security issue impacts a software component. In that case, it can
177be fixed by updating to a newer version, by applying a patch, or by marking it
178as patched via :term:`CVE_STATUS` variable flag. For Poky and OE-Core master
179branches, updating to a more recent software component release with fixes is
180the best option, but patches can be applied if releases are not yet available.
181
182For stable branches, we want to avoid API (Application Programming Interface)
183or ABI (Application Binary Interface) breakages. When submitting an update,
184a minor version update of a component is preferred if the version is
185backward-compatible. Many software components have backward-compatible stable
186versions, with a notable example of the Linux kernel. However, if the new
187version does or likely might introduce incompatibilities, extracting and
188backporting patches is preferred.
189
190Here is an example of fixing CVE security issues with patch files,
191an example from the :oe_layerindex:`ffmpeg recipe for dunfell </layerindex/recipe/122174>`::
192
193 SRC_URI = "https://www.ffmpeg.org/releases/${BP}.tar.xz \
194 file://mips64_cpu_detection.patch \
195 file://CVE-2020-12284.patch \
196 file://0001-libavutil-include-assembly-with-full-path-from-sourc.patch \
197 file://CVE-2021-3566.patch \
198 file://CVE-2021-38291.patch \
199 file://CVE-2022-1475.patch \
200 file://CVE-2022-3109.patch \
201 file://CVE-2022-3341.patch \
202 file://CVE-2022-48434.patch \
203 "
204
205The recipe has both generic and security-related fixes. The CVE patch files are named
206according to the CVE they fix.
207
208When preparing the patch file, take the original patch from the upstream repository.
209Do not use patches from different distributions, except if it is the only available source.
210
211Modify the patch adding OE-related metadata. We will follow the example of the
212``CVE-2022-3341.patch``.
213
214The original `commit message <https://github.com/FFmpeg/FFmpeg/commit/9cf652cef49d74afe3d454f27d49eb1a1394951e.patch/>`__
215is::
216
217 From 9cf652cef49d74afe3d454f27d49eb1a1394951e Mon Sep 17 00:00:00 2001
218 From: Jiasheng Jiang <jiasheng@iscas.ac.cn>
219 Date: Wed, 23 Feb 2022 10:31:59 +0800
220 Subject: [PATCH] avformat/nutdec: Add check for avformat_new_stream
221
222 Check for failure of avformat_new_stream() and propagate
223 the error code.
224
225 Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
226 ---
227 libavformat/nutdec.c | 16 ++++++++++++----
228 1 file changed, 12 insertions(+), 4 deletions(-)
229
230
231For the correct operations of the ``cve-check``, it requires the CVE
232identification in a ``CVE:`` tag of the patch file commit message using
233the format::
234
235 CVE: CVE-2022-3341
236
237It is also recommended to add the ``Upstream-Status:`` tag with a link
238to the original patch and sign-off by people working on the backport.
239If there are any modifications to the original patch, note them in
240the ``Comments:`` tag.
241
242With the additional information, the header of the patch file in OE-core becomes::
243
244 From 9cf652cef49d74afe3d454f27d49eb1a1394951e Mon Sep 17 00:00:00 2001
245 From: Jiasheng Jiang <jiasheng@iscas.ac.cn>
246 Date: Wed, 23 Feb 2022 10:31:59 +0800
247 Subject: [PATCH] avformat/nutdec: Add check for avformat_new_stream
248
249 Check for failure of avformat_new_stream() and propagate
250 the error code.
251
252 Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
253
254 CVE: CVE-2022-3341
255
256 Upstream-Status: Backport [https://github.com/FFmpeg/FFmpeg/commit/9cf652cef49d74afe3d454f27d49eb1a1394951e]
257
258 Comments: Refreshed Hunk
259 Signed-off-by: Narpat Mali <narpat.mali@windriver.com>
260 Signed-off-by: Bhabu Bindu <bhabu.bindu@kpit.com>
261 ---
262 libavformat/nutdec.c | 16 ++++++++++++----
263 1 file changed, 12 insertions(+), 4 deletions(-)
264
265A good practice is to include the CVE identifier in the patch file name, the patch file
266commit message and optionally in the recipe commit message.
267
268CVE checker will then capture this information and change the CVE status to ``Patched``
269in the generated reports.
270
271If analysis shows that the CVE issue does not impact the recipe due to configuration, platform,
272version or other reasons, the CVE can be marked as ``Ignored`` by using
273the :term:`CVE_STATUS` variable flag with appropriate reason which is mapped to ``Ignored``.
274The entry should have the format like::
275
276 CVE_STATUS[CVE-2016-10642] = "cpe-incorrect: This is specific to the npm package that installs cmake, so isn't relevant to OpenEmbedded"
277
278As mentioned previously, if data in the CVE database is wrong, it is recommended
279to fix those issues in the CVE database (NVD in the case of OE-core and Poky)
280directly.
281
282Note that if there are many CVEs with the same status and reason, those can be
283shared by using the :term:`CVE_STATUS_GROUPS` variable.
284
285Recipes can be completely skipped by CVE check by including the recipe name in
286the :term:`CVE_CHECK_SKIP_RECIPE` variable.
287
288Implementation details
289======================
290
291Here's what the :ref:`ref-classes-cve-check` class does to find unpatched CVE IDs.
292
293First the code goes through each patch file provided by a recipe. If a valid CVE ID
294is found in the name of the file, the corresponding CVE is considered as patched.
295Don't forget that if multiple CVE IDs are found in the filename, only the last
296one is considered. Then, the code looks for ``CVE: CVE-ID`` lines in the patch
297file. The found CVE IDs are also considered as patched.
298Additionally ``CVE_STATUS`` variable flags are parsed for reasons mapped to ``Patched``
299and these are also considered as patched.
300
301Then, the code looks up all the CVE IDs in the NIST database for all the
302products defined in :term:`CVE_PRODUCT`. Then, for each found CVE:
303
304- If the package name (:term:`PN`) is part of
305 :term:`CVE_CHECK_SKIP_RECIPE`, it is considered as ``Patched``.
306
307- If the CVE ID has status ``CVE_STATUS[<CVE ID>] = "ignored"`` or if it's set to
308 any reason which is mapped to status ``Ignored`` via ``CVE_CHECK_STATUSMAP``,
309 it is set as ``Ignored``.
310
311- If the CVE ID is part of the patched CVE for the recipe, it is
312 already considered as ``Patched``.
313
314- Otherwise, the code checks whether the recipe version (:term:`PV`)
315 is within the range of versions impacted by the CVE. If so, the CVE
316 is considered as ``Unpatched``.
317
318The CVE database is stored in :term:`DL_DIR` and can be inspected using
319``sqlite3`` command as follows::
320
321 sqlite3 downloads/CVE_CHECK2/nvd*.db .dump | grep CVE-2021-37462
322
323When analyzing CVEs, it is recommended to:
324
325- study the latest information in `CVE database <https://nvd.nist.gov/vuln/search>`__.
326
327- check how upstream developers of the software component addressed the issue, e.g.
328 what patch was applied, which upstream release contains the fix.
329
330- check what other Linux distributions like `Debian <https://security-tracker.debian.org/tracker/>`__
331 did to analyze and address the issue.
332
333- follow security notices from other Linux distributions.
334
335- follow public `open source security mailing lists <https://oss-security.openwall.org/wiki/mailing-lists>`__ for
336 discussions and advance notifications of CVE bugs and software releases with fixes.
337
diff --git a/documentation/dev-manual/wayland.rst b/documentation/dev-manual/wayland.rst
deleted file mode 100644
index 097be9cbde..0000000000
--- a/documentation/dev-manual/wayland.rst
+++ /dev/null
@@ -1,90 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using Wayland and Weston
4************************
5
6:wikipedia:`Wayland <Wayland_(display_server_protocol)>`
7is a computer display server protocol that provides a method for
8compositing window managers to communicate directly with applications
9and video hardware and expects them to communicate with input hardware
10using other libraries. Using Wayland with supporting targets can result
11in better control over graphics frame rendering than an application
12might otherwise achieve.
13
14The Yocto Project provides the Wayland protocol libraries and the
15reference :wikipedia:`Weston <Wayland_(display_server_protocol)#Weston>`
16compositor as part of its release. You can find the integrated packages
17in the ``meta`` layer of the :term:`Source Directory`.
18Specifically, you
19can find the recipes that build both Wayland and Weston at
20``meta/recipes-graphics/wayland``.
21
22You can build both the Wayland and Weston packages for use only with targets
23that accept the :wikipedia:`Mesa 3D and Direct Rendering Infrastructure
24<Mesa_(computer_graphics)>`, which is also known as Mesa DRI. This implies that
25you cannot build and use the packages if your target uses, for example, the
26Intel Embedded Media and Graphics Driver (Intel EMGD) that overrides Mesa DRI.
27
28.. note::
29
30 Due to lack of EGL support, Weston 1.0.3 will not run directly on the
31 emulated QEMU hardware. However, this version of Weston will run
32 under X emulation without issues.
33
34This section describes what you need to do to implement Wayland and use
35the Weston compositor when building an image for a supporting target.
36
37Enabling Wayland in an Image
38============================
39
40To enable Wayland, you need to enable it to be built and enable it to be
41included (installed) in the image.
42
43Building Wayland
44----------------
45
46To cause Mesa to build the ``wayland-egl`` platform and Weston to build
47Wayland with Kernel Mode Setting
48(`KMS <https://wiki.archlinux.org/index.php/Kernel_Mode_Setting>`__)
49support, include the "wayland" flag in the
50:term:`DISTRO_FEATURES`
51statement in your ``local.conf`` file::
52
53 DISTRO_FEATURES:append = " wayland"
54
55.. note::
56
57 If X11 has been enabled elsewhere, Weston will build Wayland with X11
58 support
59
60Installing Wayland and Weston
61-----------------------------
62
63To install the Wayland feature into an image, you must include the
64following
65:term:`CORE_IMAGE_EXTRA_INSTALL`
66statement in your ``local.conf`` file::
67
68 CORE_IMAGE_EXTRA_INSTALL += "wayland weston"
69
70Running Weston
71==============
72
73To run Weston inside X11, enabling it as described earlier and building
74a Sato image is sufficient. If you are running your image under Sato, a
75Weston Launcher appears in the "Utility" category.
76
77Alternatively, you can run Weston through the command-line interpretor
78(CLI), which is better suited for development work. To run Weston under
79the CLI, you need to do the following after your image is built:
80
81#. Run these commands to export ``XDG_RUNTIME_DIR``::
82
83 mkdir -p /tmp/$USER-weston
84 chmod 0700 /tmp/$USER-weston
85 export XDG_RUNTIME_DIR=/tmp/$USER-weston
86
87#. Launch Weston in the shell::
88
89 weston
90
diff --git a/documentation/dev-manual/wic.rst b/documentation/dev-manual/wic.rst
deleted file mode 100644
index 07170537b4..0000000000
--- a/documentation/dev-manual/wic.rst
+++ /dev/null
@@ -1,731 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Creating Partitioned Images Using Wic
4*************************************
5
6Creating an image for a particular hardware target using the
7OpenEmbedded build system does not necessarily mean you can boot that
8image as is on your device. Physical devices accept and boot images in
9various ways depending on the specifics of the device. Usually,
10information about the hardware can tell you what image format the device
11requires. Should your device require multiple partitions on an SD card,
12flash, or an HDD, you can use the OpenEmbedded Image Creator, Wic, to
13create the properly partitioned image.
14
15The ``wic`` command generates partitioned images from existing
16OpenEmbedded build artifacts. Image generation is driven by partitioning
17commands contained in an OpenEmbedded kickstart file (``.wks``)
18specified either directly on the command line or as one of a selection
19of canned kickstart files as shown with the ``wic list images`` command
20in the
21":ref:`dev-manual/wic:generate an image using an existing kickstart file`"
22section. When you apply the command to a given set of build artifacts, the
23result is an image or set of images that can be directly written onto media and
24used on a particular system.
25
26.. note::
27
28 For a kickstart file reference, see the
29 ":ref:`ref-manual/kickstart:openembedded kickstart (\`\`.wks\`\`) reference`"
30 Chapter in the Yocto Project Reference Manual.
31
32The ``wic`` command and the infrastructure it is based on is by
33definition incomplete. The purpose of the command is to allow the
34generation of customized images, and as such, was designed to be
35completely extensible through a plugin interface. See the
36":ref:`dev-manual/wic:using the wic plugin interface`" section
37for information on these plugins.
38
39This section provides some background information on Wic, describes what
40you need to have in place to run the tool, provides instruction on how
41to use the Wic utility, provides information on using the Wic plugins
42interface, and provides several examples that show how to use Wic.
43
44Background
45==========
46
47This section provides some background on the Wic utility. While none of
48this information is required to use Wic, you might find it interesting.
49
50- The name "Wic" is derived from OpenEmbedded Image Creator (oeic). The
51 "oe" diphthong in "oeic" was promoted to the letter "w", because
52 "oeic" is both difficult to remember and to pronounce.
53
54- Wic is loosely based on the Meego Image Creator (``mic``) framework.
55 The Wic implementation has been heavily modified to make direct use
56 of OpenEmbedded build artifacts instead of package installation and
57 configuration, which are already incorporated within the OpenEmbedded
58 artifacts.
59
60- Wic is a completely independent standalone utility that initially
61 provides easier-to-use and more flexible replacements for an existing
62 functionality in OE-Core's :ref:`ref-classes-image-live`
63 class. The difference between Wic and those examples is that with Wic
64 the functionality of those scripts is implemented by a
65 general-purpose partitioning language, which is based on Redhat
66 kickstart syntax.
67
68Requirements
69============
70
71In order to use the Wic utility with the OpenEmbedded Build system, your
72system needs to meet the following requirements:
73
74- The Linux distribution on your development host must support the
75 Yocto Project. See the ":ref:`system-requirements-supported-distros`"
76 section in the Yocto Project Reference Manual for the list of
77 distributions that support the Yocto Project.
78
79- The standard system utilities, such as ``cp``, must be installed on
80 your development host system.
81
82- You must have sourced the build environment setup script (i.e.
83 :ref:`structure-core-script`) found in the :term:`Build Directory`.
84
85- You need to have the build artifacts already available, which
86 typically means that you must have already created an image using the
87 OpenEmbedded build system (e.g. ``core-image-minimal``). While it
88 might seem redundant to generate an image in order to create an image
89 using Wic, the current version of Wic requires the artifacts in the
90 form generated by the OpenEmbedded build system.
91
92- You must build several native tools, which are built to run on the
93 build system::
94
95 $ bitbake wic-tools
96
97- Include "wic" as part of the
98 :term:`IMAGE_FSTYPES`
99 variable.
100
101- Include the name of the :ref:`wic kickstart file <openembedded-kickstart-wks-reference>`
102 as part of the :term:`WKS_FILE` variable. If multiple candidate files can
103 be provided by different layers, specify all the possible names through the
104 :term:`WKS_FILES` variable instead.
105
106Getting Help
107============
108
109You can get general help for the ``wic`` command by entering the ``wic``
110command by itself or by entering the command with a help argument as
111follows::
112
113 $ wic -h
114 $ wic --help
115 $ wic help
116
117Currently, Wic supports seven commands: ``cp``, ``create``, ``help``,
118``list``, ``ls``, ``rm``, and ``write``. You can get help for all these
119commands except "help" by using the following form::
120
121 $ wic help command
122
123For example, the following command returns help for the ``write``
124command::
125
126 $ wic help write
127
128Wic supports help for three topics: ``overview``, ``plugins``, and
129``kickstart``. You can get help for any topic using the following form::
130
131 $ wic help topic
132
133For example, the following returns overview help for Wic::
134
135 $ wic help overview
136
137There is one additional level of help for Wic. You can get help on
138individual images through the ``list`` command. You can use the ``list``
139command to return the available Wic images as follows::
140
141 $ wic list images
142 genericx86 Create an EFI disk image for genericx86*
143 beaglebone-yocto Create SD card image for Beaglebone
144 qemuriscv Create qcow2 image for RISC-V QEMU machines
145 mkefidisk Create an EFI disk image
146 qemuloongarch Create qcow2 image for LoongArch QEMU machines
147 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
148 directdisk Create a 'pcbios' direct disk image
149 efi-bootdisk
150 mkhybridiso Create a hybrid ISO image
151 directdisk-gpt Create a 'pcbios' direct disk image
152 systemd-bootdisk Create an EFI disk image with systemd-boot
153 sdimage-bootpart Create SD card image with a boot partition
154 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
155 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
156
157Once you know the list of available
158Wic images, you can use ``help`` with the command to get help on a
159particular image. For example, the following command returns help on the
160"beaglebone-yocto" image::
161
162 $ wic list beaglebone-yocto help
163
164 Creates a partitioned SD card image for Beaglebone.
165 Boot files are located in the first vfat partition.
166
167Operational Modes
168=================
169
170You can use Wic in two different modes, depending on how much control
171you need for specifying the OpenEmbedded build artifacts that are used
172for creating the image: Raw and Cooked:
173
174- *Raw Mode:* You explicitly specify build artifacts through Wic
175 command-line arguments.
176
177- *Cooked Mode:* The current
178 :term:`MACHINE` setting and image
179 name are used to automatically locate and provide the build
180 artifacts. You just supply a kickstart file and the name of the image
181 from which to use artifacts.
182
183Regardless of the mode you use, you need to have the build artifacts
184ready and available.
185
186Raw Mode
187--------
188
189Running Wic in raw mode allows you to specify all the partitions through
190the ``wic`` command line. The primary use for raw mode is if you have
191built your kernel outside of the Yocto Project :term:`Build Directory`.
192In other words, you can point to arbitrary kernel, root filesystem locations,
193and so forth. Contrast this behavior with cooked mode where Wic looks in the
194:term:`Build Directory` (e.g. ``tmp/deploy/images/``\ machine).
195
196The general form of the ``wic`` command in raw mode is::
197
198 $ wic create wks_file options ...
199
200 Where:
201
202 wks_file:
203 An OpenEmbedded kickstart file. You can provide
204 your own custom file or use a file from a set of
205 existing files as described by further options.
206
207 optional arguments:
208 -h, --help show this help message and exit
209 -o OUTDIR, --outdir OUTDIR
210 name of directory to create image in
211 -e IMAGE_NAME, --image-name IMAGE_NAME
212 name of the image to use the artifacts from e.g. core-
213 image-sato
214 -r ROOTFS_DIR, --rootfs-dir ROOTFS_DIR
215 path to the /rootfs dir to use as the .wks rootfs
216 source
217 -b BOOTIMG_DIR, --bootimg-dir BOOTIMG_DIR
218 path to the dir containing the boot artifacts (e.g.
219 /EFI or /syslinux dirs) to use as the .wks bootimg
220 source
221 -k KERNEL_DIR, --kernel-dir KERNEL_DIR
222 path to the dir containing the kernel to use in the
223 .wks bootimg
224 -n NATIVE_SYSROOT, --native-sysroot NATIVE_SYSROOT
225 path to the native sysroot containing the tools to use
226 to build the image
227 -s, --skip-build-check
228 skip the build check
229 -f, --build-rootfs build rootfs
230 -c {gzip,bzip2,xz}, --compress-with {gzip,bzip2,xz}
231 compress image with specified compressor
232 -m, --bmap generate .bmap
233 --no-fstab-update Do not change fstab file.
234 -v VARS_DIR, --vars VARS_DIR
235 directory with <image>.env files that store bitbake
236 variables
237 -D, --debug output debug information
238
239.. note::
240
241 You do not need root privileges to run Wic. In fact, you should not
242 run as root when using the utility.
243
244Cooked Mode
245-----------
246
247Running Wic in cooked mode leverages off artifacts in the
248:term:`Build Directory`. In other words, you do not have to specify kernel or
249root filesystem locations as part of the command. All you need to provide is
250a kickstart file and the name of the image from which to use artifacts
251by using the "-e" option. Wic looks in the :term:`Build Directory` (e.g.
252``tmp/deploy/images/``\ machine) for artifacts.
253
254The general form of the ``wic`` command using Cooked Mode is as follows::
255
256 $ wic create wks_file -e IMAGE_NAME
257
258 Where:
259
260 wks_file:
261 An OpenEmbedded kickstart file. You can provide
262 your own custom file or use a file from a set of
263 existing files provided with the Yocto Project
264 release.
265
266 required argument:
267 -e IMAGE_NAME, --image-name IMAGE_NAME
268 name of the image to use the artifacts from e.g. core-
269 image-sato
270
271Using an Existing Kickstart File
272================================
273
274If you do not want to create your own kickstart file, you can use an
275existing file provided by the Wic installation. As shipped, kickstart
276files can be found in the :ref:`overview-manual/development-environment:yocto project source repositories` in the
277following two locations::
278
279 poky/meta-yocto-bsp/wic
280 poky/scripts/lib/wic/canned-wks
281
282Use the following command to list the available kickstart files::
283
284 $ wic list images
285 genericx86 Create an EFI disk image for genericx86*
286 beaglebone-yocto Create SD card image for Beaglebone
287 qemuriscv Create qcow2 image for RISC-V QEMU machines
288 mkefidisk Create an EFI disk image
289 qemuloongarch Create qcow2 image for LoongArch QEMU machines
290 directdisk-multi-rootfs Create multi rootfs image using rootfs plugin
291 directdisk Create a 'pcbios' direct disk image
292 efi-bootdisk
293 mkhybridiso Create a hybrid ISO image
294 directdisk-gpt Create a 'pcbios' direct disk image
295 systemd-bootdisk Create an EFI disk image with systemd-boot
296 sdimage-bootpart Create SD card image with a boot partition
297 qemux86-directdisk Create a qemu machine 'pcbios' direct disk image
298 directdisk-bootloader-config Create a 'pcbios' direct disk image with custom bootloader config
299
300When you use an existing file, you
301do not have to use the ``.wks`` extension. Here is an example in Raw
302Mode that uses the ``directdisk`` file::
303
304 $ wic create directdisk -r rootfs_dir -b bootimg_dir \
305 -k kernel_dir -n native_sysroot
306
307Here are the actual partition language commands used in the
308``genericx86.wks`` file to generate an image::
309
310 # short-description: Create an EFI disk image for genericx86*
311 # long-description: Creates a partitioned EFI disk image for genericx86* machines
312 part /boot --source bootimg_efi --sourceparams="loader=grub-efi" --ondisk sda --label msdos --active --align 1024
313 part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 --use-uuid
314 part swap --ondisk sda --size 44 --label swap1 --fstype=swap
315
316 bootloader --ptable gpt --timeout=5 --append="rootfstype=ext4 console=ttyS0,115200 console=tty0"
317
318Using the Wic Plugin Interface
319==============================
320
321You can extend and specialize Wic functionality by using Wic plugins.
322This section explains the Wic plugin interface.
323
324.. note::
325
326 Wic plugins consist of "source" and "imager" plugins. Imager plugins
327 are beyond the scope of this section.
328
329Source plugins provide a mechanism to customize partition content during
330the Wic image generation process. You can use source plugins to map
331values that you specify using ``--source`` commands in kickstart files
332(i.e. ``*.wks``) to a plugin implementation used to populate a given
333partition.
334
335.. note::
336
337 If you use plugins that have build-time dependencies (e.g. native
338 tools, bootloaders, and so forth) when building a Wic image, you need
339 to specify those dependencies using the :term:`WKS_FILE_DEPENDS`
340 variable.
341
342Source plugins are subclasses defined in plugin files. As shipped, the
343Yocto Project provides several plugin files. You can see the source
344plugin files that ship with the Yocto Project
345:yocto_git:`here </poky/tree/scripts/lib/wic/plugins/source>`.
346Each of these plugin files contains source plugins that are designed to
347populate a specific Wic image partition.
348
349Source plugins are subclasses of the ``SourcePlugin`` class, which is
350defined in the ``poky/scripts/lib/wic/pluginbase.py`` file. For example,
351the ``BootimgEFIPlugin`` source plugin found in the ``bootimg_efi.py``
352file is a subclass of the ``SourcePlugin`` class, which is found in the
353``pluginbase.py`` file.
354
355You can also implement source plugins in a layer outside of the Source
356Repositories (external layer). To do so, be sure that your plugin files
357are located in a directory whose path is
358``scripts/lib/wic/plugins/source/`` within your external layer. When the
359plugin files are located there, the source plugins they contain are made
360available to Wic.
361
362When the Wic implementation needs to invoke a partition-specific
363implementation, it looks for the plugin with the same name as the
364``--source`` parameter used in the kickstart file given to that
365partition. For example, if the partition is set up using the following
366command in a kickstart file::
367
368 part /boot --source bootimg_pcbios --ondisk sda --label boot --active --align 1024
369
370The methods defined as class
371members of the matching source plugin (i.e. ``bootimg_pcbios``) in the
372``bootimg_pcbios.py`` plugin file are used.
373
374To be more concrete, here is the corresponding plugin definition from
375the ``bootimg_pcbios.py`` file for the previous command along with an
376example method called by the Wic implementation when it needs to prepare
377a partition using an implementation-specific function::
378
379 .
380 .
381 .
382 class BootimgPcbiosPlugin(SourcePlugin):
383 """
384 Create MBR boot partition and install syslinux on it.
385 """
386
387 name = 'bootimg_pcbios'
388 .
389 .
390 .
391 @classmethod
392 def do_prepare_partition(cls, part, source_params, creator, cr_workdir,
393 oe_builddir, bootimg_dir, kernel_dir,
394 rootfs_dir, native_sysroot):
395 """
396 Called to do the actual content population for a partition i.e. it
397 'prepares' the partition to be incorporated into the image.
398 In this case, prepare content for legacy bios boot partition.
399 """
400 .
401 .
402 .
403
404If a
405subclass (plugin) itself does not implement a particular function, Wic
406locates and uses the default version in the superclass. It is for this
407reason that all source plugins are derived from the ``SourcePlugin``
408class.
409
410The ``SourcePlugin`` class defined in the ``pluginbase.py`` file defines
411a set of methods that source plugins can implement or override. Any
412plugins (subclass of ``SourcePlugin``) that do not implement a
413particular method inherit the implementation of the method from the
414``SourcePlugin`` class. For more information, see the ``SourcePlugin``
415class in the ``pluginbase.py`` file for details:
416
417The following list describes the methods implemented in the
418``SourcePlugin`` class:
419
420- ``do_prepare_partition()``: Called to populate a partition with
421 actual content. In other words, the method prepares the final
422 partition image that is incorporated into the disk image.
423
424- ``do_configure_partition()``: Called before
425 ``do_prepare_partition()`` to create custom configuration files for a
426 partition (e.g. syslinux or grub configuration files).
427
428- ``do_install_disk()``: Called after all partitions have been
429 prepared and assembled into a disk image. This method provides a hook
430 to allow finalization of a disk image (e.g. writing an MBR).
431
432- ``do_stage_partition()``: Special content-staging hook called
433 before ``do_prepare_partition()``. This method is normally empty.
434
435 Typically, a partition just uses the passed-in parameters (e.g. the
436 unmodified value of ``bootimg_dir``). However, in some cases, things
437 might need to be more tailored. As an example, certain files might
438 additionally need to be taken from ``bootimg_dir + /boot``. This hook
439 allows those files to be staged in a customized fashion.
440
441 .. note::
442
443 ``get_bitbake_var()`` allows you to access non-standard variables that
444 you might want to use for this behavior.
445
446You can extend the source plugin mechanism. To add more hooks, create
447more source plugin methods within ``SourcePlugin`` and the corresponding
448derived subclasses. The code that calls the plugin methods uses the
449``plugin.get_source_plugin_methods()`` function to find the method or
450methods needed by the call. Retrieval of those methods is accomplished
451by filling up a dict with keys that contain the method names of
452interest. On success, these will be filled in with the actual methods.
453See the Wic implementation for examples and details.
454
455Wic Examples
456============
457
458This section provides several examples that show how to use the Wic
459utility. All the examples assume the list of requirements in the
460":ref:`dev-manual/wic:requirements`" section have been met. The
461examples assume the previously generated image is
462``core-image-minimal``.
463
464Generate an Image using an Existing Kickstart File
465--------------------------------------------------
466
467This example runs in Cooked Mode and uses the ``mkefidisk`` kickstart
468file::
469
470 $ wic create mkefidisk -e core-image-minimal
471 INFO: Building wic-tools...
472 .
473 .
474 .
475 INFO: The new image(s) can be found here:
476 ./mkefidisk-201804191017-sda.direct
477
478 The following build artifacts were used to create the image(s):
479 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
480 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
481 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
482 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
483
484 INFO: The image(s) were created using OE kickstart file:
485 /home/stephano/yocto/openembedded-core/scripts/lib/wic/canned-wks/mkefidisk.wks
486
487The previous example shows the easiest way to create an image by running
488in cooked mode and supplying a kickstart file and the "-e" option to
489point to the existing build artifacts. Your ``local.conf`` file needs to
490have the :term:`MACHINE` variable set
491to the machine you are using, which is "qemux86" in this example.
492
493Once the image builds, the output provides image location, artifact use,
494and kickstart file information.
495
496.. note::
497
498 You should always verify the details provided in the output to make
499 sure that the image was indeed created exactly as expected.
500
501Continuing with the example, you can now write the image from the
502:term:`Build Directory` onto a USB stick, or whatever media for which you
503built your image, and boot from the media. You can write the image by using
504``bmaptool`` or ``dd``::
505
506 $ oe-run-native bmaptool-native bmaptool copy mkefidisk-201804191017-sda.direct /dev/sdX
507
508or ::
509
510 $ sudo dd if=mkefidisk-201804191017-sda.direct of=/dev/sdX
511
512.. note::
513
514 For more information on how to use the ``bmaptool``
515 to flash a device with an image, see the
516 ":ref:`dev-manual/bmaptool:flashing images using \`bmaptool\``"
517 section.
518
519Using a Modified Kickstart File
520-------------------------------
521
522Because partitioned image creation is driven by the kickstart file, it
523is easy to affect image creation by changing the parameters in the file.
524This next example demonstrates that through modification of the
525``directdisk-gpt`` kickstart file.
526
527As mentioned earlier, you can use the command ``wic list images`` to
528show the list of existing kickstart files. The directory in which the
529``directdisk-gpt.wks`` file resides is
530``scripts/lib/image/canned-wks/``, which is located in the
531:term:`Source Directory` (e.g. ``poky``).
532Because available files reside in this directory, you can create and add
533your own custom files to the directory. Subsequent use of the
534``wic list images`` command would then include your kickstart files.
535
536In this example, the existing ``directdisk-gpt`` file already does most
537of what is needed. However, for the hardware in this example, the image
538will need to boot from ``sdb`` instead of ``sda``, which is what the
539``directdisk-gpt`` kickstart file uses.
540
541The example begins by making a copy of the ``directdisk-gpt.wks`` file
542in the ``scripts/lib/image/canned-wks`` directory and then by changing
543the lines that specify the target disk from which to boot::
544
545 $ cp /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisk-gpt.wks \
546 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
547
548Next, the example modifies the ``directdisksdb-gpt.wks`` file and
549changes all instances of "``--ondisk sda``" to "``--ondisk sdb``". The
550example changes the following two lines and leaves the remaining lines
551untouched::
552
553 part /boot --source bootimg_pcbios --ondisk sdb --label boot --active --align 1024
554 part / --source rootfs --ondisk sdb --fstype=ext4 --label platform --align 1024 --use-uuid
555
556Once the lines are changed, the
557example generates the ``directdisksdb-gpt`` image. The command points
558the process at the ``core-image-minimal`` artifacts for the Next Unit of
559Computing (nuc) :term:`MACHINE` the
560``local.conf``::
561
562 $ wic create directdisksdb-gpt -e core-image-minimal
563 INFO: Building wic-tools...
564 .
565 .
566 .
567 Initialising tasks: 100% |#######################################| Time: 0:00:01
568 NOTE: Executing SetScene Tasks
569 NOTE: Executing RunQueue Tasks
570 NOTE: Tasks Summary: Attempted 1161 tasks of which 1157 didn't need to be rerun and all succeeded.
571 INFO: Creating image(s)...
572
573 INFO: The new image(s) can be found here:
574 ./directdisksdb-gpt-201710090938-sdb.direct
575
576 The following build artifacts were used to create the image(s):
577 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
578 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
579 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
580 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
581
582 INFO: The image(s) were created using OE kickstart file:
583 /home/stephano/yocto/poky/scripts/lib/wic/canned-wks/directdisksdb-gpt.wks
584
585Continuing with the example, you can now directly ``dd`` the image to a
586USB stick, or whatever media for which you built your image, and boot
587the resulting media::
588
589 $ sudo dd if=directdisksdb-gpt-201710090938-sdb.direct of=/dev/sdb
590 140966+0 records in
591 140966+0 records out
592 72174592 bytes (72 MB, 69 MiB) copied, 78.0282 s, 925 kB/s
593 $ sudo eject /dev/sdb
594
595Using a Modified Kickstart File and Running in Raw Mode
596-------------------------------------------------------
597
598This next example manually specifies each build artifact (runs in Raw
599Mode) and uses a modified kickstart file. The example also uses the
600``-o`` option to cause Wic to create the output somewhere other than the
601default output directory, which is the current directory::
602
603 $ wic create test.wks -o /home/stephano/testwic \
604 --rootfs-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs \
605 --bootimg-dir /home/stephano/yocto/build/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share \
606 --kernel-dir /home/stephano/yocto/build/tmp/deploy/images/qemux86 \
607 --native-sysroot /home/stephano/yocto/build/tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native
608
609 INFO: Creating image(s)...
610
611 INFO: The new image(s) can be found here:
612 /home/stephano/testwic/test-201710091445-sdb.direct
613
614 The following build artifacts were used to create the image(s):
615 ROOTFS_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/rootfs
616 BOOTIMG_DIR: /home/stephano/yocto/build/tmp-glibc/work/qemux86-oe-linux/core-image-minimal/1.0-r0/recipe-sysroot/usr/share
617 KERNEL_DIR: /home/stephano/yocto/build/tmp-glibc/deploy/images/qemux86
618 NATIVE_SYSROOT: /home/stephano/yocto/build/tmp-glibc/work/i586-oe-linux/wic-tools/1.0-r0/recipe-sysroot-native
619
620 INFO: The image(s) were created using OE kickstart file:
621 test.wks
622
623For this example,
624:term:`MACHINE` did not have to be
625specified in the ``local.conf`` file since the artifact is manually
626specified.
627
628Using Wic to Manipulate an Image
629--------------------------------
630
631Wic image manipulation allows you to shorten turnaround time during
632image development. For example, you can use Wic to delete the kernel
633partition of a Wic image and then insert a newly built kernel. This
634saves you time from having to rebuild the entire image each time you
635modify the kernel.
636
637.. note::
638
639 In order to use Wic to manipulate a Wic image as in this example,
640 your development machine must have the ``mtools`` package installed.
641
642The following example examines the contents of the Wic image, deletes
643the existing kernel, and then inserts a new kernel:
644
645#. *List the Partitions:* Use the ``wic ls`` command to list all the
646 partitions in the Wic image::
647
648 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic
649 Num Start End Size Fstype
650 1 1048576 25041919 23993344 fat16
651 2 25165824 72157183 46991360 ext4
652
653 The previous output shows two partitions in the
654 ``core-image-minimal-qemux86.wic`` image.
655
656#. *Examine a Particular Partition:* Use the ``wic ls`` command again
657 but in a different form to examine a particular partition.
658
659 .. note::
660
661 You can get command usage on any Wic command using the following
662 form::
663
664 $ wic help command
665
666
667 For example, the following command shows you the various ways to
668 use the
669 wic ls
670 command::
671
672 $ wic help ls
673
674
675 The following command shows what is in partition one::
676
677 $ wic ls tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1
678 Volume in drive : is boot
679 Volume Serial Number is E894-1809
680 Directory for ::/
681
682 libcom32 c32 186500 2017-10-09 16:06
683 libutil c32 24148 2017-10-09 16:06
684 syslinux cfg 220 2017-10-09 16:06
685 vesamenu c32 27104 2017-10-09 16:06
686 vmlinuz 6904608 2017-10-09 16:06
687 5 files 7 142 580 bytes
688 16 582 656 bytes free
689
690 The previous output shows five files, with the
691 ``vmlinuz`` being the kernel.
692
693 .. note::
694
695 If you see the following error, you need to update or create a
696 ``~/.mtoolsrc`` file and be sure to have the line "mtools_skip_check=1"
697 in the file. Then, run the Wic command again::
698
699 ERROR: _exec_cmd: /usr/bin/mdir -i /tmp/wic-parttfokuwra ::/ returned '1' instead of 0
700 output: Total number of sectors (47824) not a multiple of sectors per track (32)!
701 Add mtools_skip_check=1 to your .mtoolsrc file to skip this test
702
703
704#. *Remove the Old Kernel:* Use the ``wic rm`` command to remove the
705 ``vmlinuz`` file (kernel)::
706
707 $ wic rm tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
708
709#. *Add In the New Kernel:* Use the ``wic cp`` command to add the
710 updated kernel to the Wic image. Depending on how you built your
711 kernel, it could be in different places. If you used ``devtool`` and
712 an SDK to build your kernel, it resides in the ``tmp/work`` directory
713 of the extensible SDK. If you used ``make`` to build the kernel, the
714 kernel will be in the ``workspace/sources`` area.
715
716 The following example assumes ``devtool`` was used to build the
717 kernel::
718
719 $ wic cp poky_sdk/tmp/work/qemux86-poky-linux/linux-yocto/4.12.12+git999-r0/linux-yocto-4.12.12+git999/arch/x86/boot/bzImage \
720 poky/build/tmp/deploy/images/qemux86/core-image-minimal-qemux86.wic:1/vmlinuz
721
722 Once the new kernel is added back into the image, you can use the
723 ``dd`` command or :ref:`bmaptool
724 <dev-manual/bmaptool:flashing images using \`bmaptool\`>` commands
725 to flash your wic image onto an SD card or USB stick and test your
726 target.
727
728 .. note::
729
730 Using ``bmaptool`` is generally 10 to 20 times faster than using ``dd``.
731
diff --git a/documentation/dev-manual/x32-psabi.rst b/documentation/dev-manual/x32-psabi.rst
deleted file mode 100644
index 92b1f96fa4..0000000000
--- a/documentation/dev-manual/x32-psabi.rst
+++ /dev/null
@@ -1,54 +0,0 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using x32 psABI
4***************
5
6x32 processor-specific Application Binary Interface (`x32
7psABI <https://software.intel.com/en-us/node/628948>`__) is a native
832-bit processor-specific ABI for Intel 64 (x86-64) architectures. An
9ABI defines the calling conventions between functions in a processing
10environment. The interface determines what registers are used and what
11the sizes are for various C data types.
12
13Some processing environments prefer using 32-bit applications even when
14running on Intel 64-bit platforms. Consider the i386 psABI, which is a
15very old 32-bit ABI for Intel 64-bit platforms. The i386 psABI does not
16provide efficient use and access of the Intel 64-bit processor
17resources, leaving the system underutilized. Now consider the x86_64
18psABI. This ABI is newer and uses 64-bits for data sizes and program
19pointers. The extra bits increase the footprint size of the programs,
20libraries, and also increases the memory and file system size
21requirements. Executing under the x32 psABI enables user programs to
22utilize CPU and system resources more efficiently while keeping the
23memory footprint of the applications low. Extra bits are used for
24registers but not for addressing mechanisms.
25
26The Yocto Project supports the final specifications of x32 psABI as
27follows:
28
29- You can create packages and images in x32 psABI format on x86_64
30 architecture targets.
31
32- You can successfully build recipes with the x32 toolchain.
33
34- You can create and boot ``core-image-minimal`` and
35 ``core-image-sato`` images.
36
37- There is RPM Package Manager (RPM) support for x32 binaries.
38
39- There is support for large images.
40
41To use the x32 psABI, you need to edit your ``conf/local.conf``
42configuration file as follows::
43
44 MACHINE = "qemux86-64"
45 DEFAULTTUNE = "x86-64-x32"
46 baselib = "${@d.getVar('BASE_LIB:tune-' + (d.getVar('DEFAULTTUNE') \
47 or 'INVALID')) or 'lib'}"
48
49Once you have set
50up your configuration file, use BitBake to build an image that supports
51the x32 psABI. Here is an example::
52
53 $ bitbake core-image-sato
54