summaryrefslogtreecommitdiffstats
path: root/documentation/dev-manual/building.rst
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/dev-manual/building.rst')
-rw-r--r--documentation/dev-manual/building.rst872
1 files changed, 872 insertions, 0 deletions
diff --git a/documentation/dev-manual/building.rst b/documentation/dev-manual/building.rst
new file mode 100644
index 0000000000..32c7aa5da0
--- /dev/null
+++ b/documentation/dev-manual/building.rst
@@ -0,0 +1,872 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Building
4********
5
6This section describes various build procedures, such as the steps
7needed for a simple build, building a target for multiple configurations,
8generating an image for more than one machine, and so forth.
9
10Building a Simple Image
11=======================
12
13In the development environment, you need to build an image whenever you
14change hardware support, add or change system libraries, or add or
15change services that have dependencies. There are several methods that allow
16you to build an image within the Yocto Project. This section presents
17the basic steps you need to build a simple image using BitBake from a
18build host running Linux.
19
20.. note::
21
22 - For information on how to build an image using
23 :term:`Toaster`, see the
24 :doc:`/toaster-manual/index`.
25
26 - For information on how to use ``devtool`` to build images, see the
27 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
28 section in the Yocto Project Application Development and the
29 Extensible Software Development Kit (eSDK) manual.
30
31 - For a quick example on how to build an image using the
32 OpenEmbedded build system, see the
33 :doc:`/brief-yoctoprojectqs/index` document.
34
35 - You can also use the `Yocto Project BitBake
36 <https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__
37 extension for Visual Studio Code to build images.
38
39The build process creates an entire Linux distribution from source and
40places it in your :term:`Build Directory` under ``tmp/deploy/images``. For
41detailed information on the build process using BitBake, see the
42":ref:`overview-manual/concepts:images`" section in the Yocto Project Overview
43and Concepts Manual.
44
45The following figure and list overviews the build process:
46
47.. image:: figures/bitbake-build-flow.png
48 :width: 100%
49
50#. *Set up Your Host Development System to Support Development Using the
51 Yocto Project*: See the ":doc:`/dev-manual/start`" section for options on how to get a
52 build host ready to use the Yocto Project.
53
54#. *Initialize the Build Environment:* Initialize the build environment
55 by sourcing the build environment script (i.e.
56 :ref:`structure-core-script`)::
57
58 $ source oe-init-build-env [build_dir]
59
60 When you use the initialization script, the OpenEmbedded build system
61 uses ``build`` as the default :term:`Build Directory` in your current work
62 directory. You can use a `build_dir` argument with the script to
63 specify a different :term:`Build Directory`.
64
65 .. note::
66
67 A common practice is to use a different :term:`Build Directory` for
68 different targets; for example, ``~/build/x86`` for a ``qemux86``
69 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
70 event, it's typically cleaner to locate the :term:`Build Directory`
71 somewhere outside of your source directory.
72
73#. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
74 ``conf/local.conf`` configuration file, which is found in the
75 :term:`Build Directory`, is set up how you want it. This file defines many
76 aspects of the build environment including the target machine architecture
77 through the :term:`MACHINE` variable, the packaging format used during
78 the build (:term:`PACKAGE_CLASSES`), and a centralized tarball download
79 directory through the :term:`DL_DIR` variable.
80
81#. *Build the Image:* Build the image using the ``bitbake`` command::
82
83 $ bitbake target
84
85 .. note::
86
87 For information on BitBake, see the :doc:`bitbake:index`.
88
89 The target is the name of the recipe you want to build. Common
90 targets are the images in ``meta/recipes-core/images``,
91 ``meta/recipes-sato/images``, and so forth all found in the
92 :term:`Source Directory`. Alternatively, the target
93 can be the name of a recipe for a specific piece of software such as
94 BusyBox. For more details about the images the OpenEmbedded build
95 system supports, see the
96 ":ref:`ref-manual/images:Images`" chapter in the Yocto
97 Project Reference Manual.
98
99 As an example, the following command builds the
100 ``core-image-minimal`` image::
101
102 $ bitbake core-image-minimal
103
104 Once an
105 image has been built, it often needs to be installed. The images and
106 kernels built by the OpenEmbedded build system are placed in the
107 :term:`Build Directory` in ``tmp/deploy/images``. For information on how to
108 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
109 :doc:`/sdk-manual/index` manual. For
110 information about how to install these images, see the documentation
111 for your particular board or machine.
112
113Building Images for Multiple Targets Using Multiple Configurations
114==================================================================
115
116See the :doc:`/dev-manual/multiconfig` section of the Yocto Project Development Tasks
117Manual.
118
119Building an Initial RAM Filesystem (Initramfs) Image
120====================================================
121
122An initial RAM filesystem (:term:`Initramfs`) image provides a temporary root
123filesystem used for early system initialization, typically providing tools and
124loading modules needed to locate and mount the final root filesystem.
125
126Follow these steps to create an :term:`Initramfs` image:
127
128#. *Create the Initramfs Image Recipe:* You can reference the
129 ``core-image-minimal-initramfs.bb`` recipe found in the
130 ``meta/recipes-core`` directory of the :term:`Source Directory`
131 as an example from which to work. The ``core-image-minimal-initramfs`` recipe
132 is based on the :ref:`initramfs-framework <dev-manual/building:Customizing an
133 Initramfs using \`\`initramfs-framework\`\`>` recipe described below.
134
135#. *Decide if You Need to Bundle the Initramfs Image Into the Kernel
136 Image:* If you want the :term:`Initramfs` image that is built to be bundled
137 in with the kernel image, set the :term:`INITRAMFS_IMAGE_BUNDLE`
138 variable to ``"1"`` in your ``local.conf`` configuration file and set the
139 :term:`INITRAMFS_IMAGE` variable in the recipe that builds the kernel image.
140
141 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the :term:`Initramfs`
142 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
143 :term:`Initramfs` image is then passed to the kernel's ``Makefile`` using the
144 :term:`CONFIG_INITRAMFS_SOURCE` variable, allowing the :term:`Initramfs`
145 image to be built into the kernel normally.
146
147#. *Optionally Add Items to the Initramfs Image Through the Initramfs
148 Image Recipe:* If you add items to the :term:`Initramfs` image by way of its
149 recipe, you should use :term:`PACKAGE_INSTALL` rather than
150 :term:`IMAGE_INSTALL`. :term:`PACKAGE_INSTALL` gives more direct control of
151 what is added to the image as compared to the defaults you might not
152 necessarily want that are set by the :ref:`ref-classes-image`
153 or :ref:`ref-classes-core-image` classes.
154
155#. *Build the Kernel Image and the Initramfs Image:* Build your kernel
156 image using BitBake. Because the :term:`Initramfs` image recipe is a
157 dependency of the kernel image, the :term:`Initramfs` image is built as well
158 and bundled with the kernel image if you used the
159 :term:`INITRAMFS_IMAGE_BUNDLE` variable described earlier.
160
161Customizing an Initramfs using ``initramfs-framework``
162------------------------------------------------------
163
164The ``core-image-minimal-initramfs.bb`` recipe found in
165:oe_git:`meta/recipes-core/images
166</openembedded-core/tree/meta/recipes-core/images>` uses the
167:oe_git:`initramfs-framework_1.0.bb
168</openembedded-core/tree/meta/recipes-core/initrdscripts/initramfs-framework_1.0.bb>`
169recipe as its base component. The goal of the ``initramfs-framework`` recipe is
170to provide the building blocks to build a customized :term:`Initramfs`.
171
172The ``initramfs-framework`` recipe relies on shell initialization scripts
173defined in :oe_git:`meta/recipes-core/initrdscripts/initramfs-framework
174</openembedded-core/tree/meta/recipes-core/initrdscripts/initramfs-framework>`. Since some of
175these scripts do not apply for all use cases, the ``initramfs-framework`` recipe
176defines different packages:
177
178- ``initramfs-framework-base``: this package installs the basic components of
179 an :term:`Initramfs`, such as the ``init`` script or the ``/dev/console``
180 character special file. As this package is a runtime dependency of all
181 modules listed below, it is automatically pulled in when one of the modules
182 is installed in the image.
183- ``initramfs-module-exec``: support for execution of applications.
184- ``initramfs-module-mdev``: support for `mdev
185 <https://wiki.gentoo.org/wiki/Mdev>`__.
186- ``initramfs-module-udev``: support for :wikipedia:`Udev <Udev>`.
187- ``initramfs-module-e2fs``: support for :wikipedia:`ext4/ext3/ext2
188 <Extended_file_system>` filesystems.
189- ``initramfs-module-nfsrootfs``: support for locating and mounting the root
190 partition via :wikipedia:`NFS <Network_File_System>`.
191- ``initramfs-module-rootfs``: support for locating and mounting the root
192 partition.
193- ``initramfs-module-debug``: dynamic debug support.
194- ``initramfs-module-lvm``: :wikipedia:`LVM <Logical_volume_management>` rootfs support.
195- ``initramfs-module-overlayroot``: support for mounting a read-write overlay
196 on top of a read-only root filesystem.
197
198In addition to the packages defined by the ``initramfs-framework`` recipe
199itself, the following packages are defined by the recipes present in
200:oe_git:`meta/recipes-core/initrdscripts </openembedded-core/tree/meta/recipes-core/initrdscripts>`:
201
202- ``initramfs-module-install``: module to create and install a partition layout
203 on a selected block device.
204- ``initramfs-module-install-efi``: module to create and install an EFI
205 partition layout on a selected block device.
206- ``initramfs-module-setup-live``: module to start a shell in the
207 :term:`Initramfs` if ``root=/dev/ram0`` in passed in the `Kernel command-line
208 <https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html>`__
209 or the ``root=`` parameter was not passed.
210
211To customize the :term:`Initramfs`, you can add or remove packages listed
212earlier from the :term:`PACKAGE_INSTALL` variable with a :ref:`bbappend
213<dev-manual/layers:Appending Other Layers Metadata With Your Layer>` on the
214``core-image-minimal-initramfs`` recipe, or create a custom recipe for the
215:term:`Initramfs` taking ``core-image-minimal-initramfs`` as example.
216
217Custom scripts can be added to the :term:`Initramfs` by writing your own
218recipes. The recipes are conventionally named ``initramfs-module-<module name>``
219where ``<module name>`` is the name of the module. The recipe should set its
220:term:`RDEPENDS` package-specific variables to include
221``initramfs-framework-base`` and the other packages on which the module depends
222at runtime.
223
224The recipe must install shell initialization scripts in :term:`${D} <D>`\
225``/init.d`` and must follow the ``<number>-<script name>`` naming scheme where:
226
227- ``<number>`` is a *two-digit* number that affects the execution order of the
228 script compared to others. For example, the script ``80-setup-live`` would be
229 executed after ``01-udev`` because 80 is greater than 01.
230
231 This number being two-digits is important here as the scripts are executed
232 alphabetically. For example, the script ``10-script`` would be executed
233 before the script ``8-script``, because ``1`` is inferior to ``8``.
234 Therefore, the script should be named ``08-script``.
235
236- ``<script name>`` is the script name which you can choose freely.
237
238 If two script use the same ``<number>``, they are sorted alphabetically based
239 on ``<script name>``.
240
241Bundling an Initramfs Image From a Separate Multiconfig
242-------------------------------------------------------
243
244There may be a case where we want to build an :term:`Initramfs` image which does not
245inherit the same distro policy as our main image, for example, we may want
246our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our :term:`Initramfs`
247image to keep a smaller footprint. However, by performing the steps mentioned
248above the :term:`Initramfs` image will inherit ``TCLIBC="glibc"`` without allowing us
249to override it.
250
251To achieve this, you need to perform some additional steps:
252
253#. *Create a multiconfig for your Initramfs image:* You can perform the steps
254 on ":ref:`dev-manual/building:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
255 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
256 contains the variables::
257
258 TMPDIR="${TOPDIR}/tmp-initramfscfg"
259 TCLIBC="musl"
260
261#. *Set additional Initramfs variables on your main configuration:*
262 Additionally, on your main configuration (``local.conf``) you need to set the
263 variables::
264
265 INITRAMFS_MULTICONFIG = "initramfscfg"
266 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
267
268 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
269 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
270 to be built coming from the ``initramfscfg`` multiconfig, and to let the
271 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
272
273 Building a system with such configuration will build the kernel using the
274 main configuration but the :ref:`ref-tasks-bundle_initramfs` task will grab the
275 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
276 instead, resulting in a musl based :term:`Initramfs` image bundled in the kernel
277 but a glibc based main image.
278
279 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
280 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
281
282
283Building a Tiny System
284======================
285
286Very small distributions have some significant advantages such as
287requiring less on-die or in-package memory (cheaper), better performance
288through efficient cache usage, lower power requirements due to less
289memory, faster boot times, and reduced development overhead. Some
290real-world examples where a very small distribution gives you distinct
291advantages are digital cameras, medical devices, and small headless
292systems.
293
294This section presents information that shows you how you can trim your
295distribution to even smaller sizes than the ``poky-tiny`` distribution,
296which is around 5 Mbytes, that can be built out-of-the-box using the
297Yocto Project.
298
299Tiny System Overview
300--------------------
301
302The following list presents the overall steps you need to consider and
303perform to create distributions with smaller root filesystems, achieve
304faster boot times, maintain your critical functionality, and avoid
305initial RAM disks:
306
307- :ref:`Determine your goals and guiding principles
308 <dev-manual/building:goals and guiding principles>`
309
310- :ref:`dev-manual/building:understand what contributes to your image size`
311
312- :ref:`Reduce the size of the root filesystem
313 <dev-manual/building:trim the root filesystem>`
314
315- :ref:`Reduce the size of the kernel <dev-manual/building:trim the kernel>`
316
317- :ref:`dev-manual/building:remove package management requirements`
318
319- :ref:`dev-manual/building:look for other ways to minimize size`
320
321- :ref:`dev-manual/building:iterate on the process`
322
323Goals and Guiding Principles
324----------------------------
325
326Before you can reach your destination, you need to know where you are
327going. Here is an example list that you can use as a guide when creating
328very small distributions:
329
330- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
331 less and a root filesystem that is 3 Mbytes or less).
332
333- Find the areas that are currently taking 90% of the space and
334 concentrate on reducing those areas.
335
336- Do not create any difficult "hacks" to achieve your goals.
337
338- Leverage the device-specific options.
339
340- Work in a separate layer so that you keep changes isolated. For
341 information on how to create layers, see the
342 ":ref:`dev-manual/layers:understanding and creating layers`" section.
343
344Understand What Contributes to Your Image Size
345----------------------------------------------
346
347It is easiest to have something to start with when creating your own
348distribution. You can use the Yocto Project out-of-the-box to create the
349``poky-tiny`` distribution. Ultimately, you will want to make changes in
350your own distribution that are likely modeled after ``poky-tiny``.
351
352.. note::
353
354 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
355 ``local.conf`` file to "poky-tiny" as described in the
356 ":ref:`dev-manual/custom-distribution:creating your own distribution`"
357 section.
358
359Understanding some memory concepts will help you reduce the system size.
360Memory consists of static, dynamic, and temporary memory. Static memory
361is the TEXT (code), DATA (initialized data in the code), and BSS
362(uninitialized data) sections. Dynamic memory represents memory that is
363allocated at runtime: stacks, hash tables, and so forth. Temporary
364memory is recovered after the boot process. This memory consists of
365memory used for decompressing the kernel and for the ``__init__``
366functions.
367
368To help you see where you currently are with kernel and root filesystem
369sizes, you can use two tools found in the :term:`Source Directory`
370in the
371``scripts/tiny/`` directory:
372
373- ``ksize.py``: Reports component sizes for the kernel build objects.
374
375- ``dirsize.py``: Reports component sizes for the root filesystem.
376
377This next tool and command help you organize configuration fragments and
378view file dependencies in a human-readable form:
379
380- ``merge_config.sh``: Helps you manage configuration files and
381 fragments within the kernel. With this tool, you can merge individual
382 configuration fragments together. The tool allows you to make
383 overrides and warns you of any missing configuration options. The
384 tool is ideal for allowing you to iterate on configurations, create
385 minimal configurations, and create configuration files for different
386 machines without having to duplicate your process.
387
388 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
389 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
390 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
391 directory.
392
393 For more information on configuration fragments, see the
394 ":ref:`kernel-dev/common:creating configuration fragments`"
395 section in the Yocto Project Linux Kernel Development Manual.
396
397- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
398 with these options brings up a Dependency Explorer from which you can
399 view file dependencies. Understanding these dependencies allows you
400 to make informed decisions when cutting out various pieces of the
401 kernel and root filesystem.
402
403Trim the Root Filesystem
404------------------------
405
406The root filesystem is made up of packages for booting, libraries, and
407applications. To change things, you can configure how the packaging
408happens, which changes the way you build them. You can also modify the
409filesystem itself or select a different filesystem.
410
411First, find out what is hogging your root filesystem by running the
412``dirsize.py`` script from your root directory::
413
414 $ cd root-directory-of-image
415 $ dirsize.py 100000 > dirsize-100k.log
416 $ cat dirsize-100k.log
417
418You can apply a filter to the script to ignore files
419under a certain size. The previous example filters out any files below
420100 Kbytes. The sizes reported by the tool are uncompressed, and thus
421will be smaller by a relatively constant factor in a compressed root
422filesystem. When you examine your log file, you can focus on areas of
423the root filesystem that take up large amounts of memory.
424
425You need to be sure that what you eliminate does not cripple the
426functionality you need. One way to see how packages relate to each other
427is by using the Dependency Explorer UI with the BitBake command::
428
429 $ cd image-directory
430 $ bitbake -u taskexp -g image
431
432Use the interface to
433select potential packages you wish to eliminate and see their dependency
434relationships.
435
436When deciding how to reduce the size, get rid of packages that result in
437minimal impact on the feature set. For example, you might not need a VGA
438display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
439instead of ``udev``.
440
441Use your ``local.conf`` file to make changes. For example, to eliminate
442``udev`` and ``glib``, set the following in the local configuration
443file::
444
445 VIRTUAL-RUNTIME_dev_manager = ""
446
447Finally, you should consider exactly the type of root filesystem you
448need to meet your needs while also reducing its size. For example,
449consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
450:term:`Initramfs` using ``initramfs``. Be aware that ``ext3`` requires a 1
451Mbyte journal. If you are okay with running read-only, you do not need
452this journal.
453
454.. note::
455
456 After each round of elimination, you need to rebuild your system and
457 then use the tools to see the effects of your reductions.
458
459Trim the Kernel
460---------------
461
462The kernel is built by including policies for hardware-independent
463aspects. What subsystems do you enable? For what architecture are you
464building? Which drivers do you build by default?
465
466.. note::
467
468 You can modify the kernel source if you want to help with boot time.
469
470Run the ``ksize.py`` script from the top-level Linux build directory to
471get an idea of what is making up the kernel::
472
473 $ cd top-level-linux-build-directory
474 $ ksize.py > ksize.log
475 $ cat ksize.log
476
477When you examine the log, you will see how much space is taken up with
478the built-in ``.o`` files for drivers, networking, core kernel files,
479filesystem, sound, and so forth. The sizes reported by the tool are
480uncompressed, and thus will be smaller by a relatively constant factor
481in a compressed kernel image. Look to reduce the areas that are large
482and taking up around the "90% rule."
483
484To examine, or drill down, into any particular area, use the ``-d``
485option with the script::
486
487 $ ksize.py -d > ksize.log
488
489Using this option
490breaks out the individual file information for each area of the kernel
491(e.g. drivers, networking, and so forth).
492
493Use your log file to see what you can eliminate from the kernel based on
494features you can let go. For example, if you are not going to need
495sound, you do not need any drivers that support sound.
496
497After figuring out what to eliminate, you need to reconfigure the kernel
498to reflect those changes during the next build. You could run
499``menuconfig`` and make all your changes at once. However, that makes it
500difficult to see the effects of your individual eliminations and also
501makes it difficult to replicate the changes for perhaps another target
502device. A better method is to start with no configurations using
503``allnoconfig``, create configuration fragments for individual changes,
504and then manage the fragments into a single configuration file using
505``merge_config.sh``. The tool makes it easy for you to iterate using the
506configuration change and build cycle.
507
508Each time you make configuration changes, you need to rebuild the kernel
509and check to see what impact your changes had on the overall size.
510
511Remove Package Management Requirements
512--------------------------------------
513
514Packaging requirements add size to the image. One way to reduce the size
515of the image is to remove all the packaging requirements from the image.
516This reduction includes both removing the package manager and its unique
517dependencies as well as removing the package management data itself.
518
519To eliminate all the packaging requirements for an image, be sure that
520"package-management" is not part of your
521:term:`IMAGE_FEATURES`
522statement for the image. When you remove this feature, you are removing
523the package manager as well as its dependencies from the root
524filesystem.
525
526Look for Other Ways to Minimize Size
527------------------------------------
528
529Depending on your particular circumstances, other areas that you can
530trim likely exist. The key to finding these areas is through tools and
531methods described here combined with experimentation and iteration. Here
532are a couple of areas to experiment with:
533
534- ``glibc``: In general, follow this process:
535
536 #. Remove ``glibc`` features from
537 :term:`DISTRO_FEATURES`
538 that you think you do not need.
539
540 #. Build your distribution.
541
542 #. If the build fails due to missing symbols in a package, determine
543 if you can reconfigure the package to not need those features. For
544 example, change the configuration to not support wide character
545 support as is done for ``ncurses``. Or, if support for those
546 characters is needed, determine what ``glibc`` features provide
547 the support and restore the configuration.
548
549 4. Rebuild and repeat the process.
550
551- ``busybox``: For BusyBox, use a process similar as described for
552 ``glibc``. A difference is you will need to boot the resulting system
553 to see if you are able to do everything you expect from the running
554 system. You need to be sure to integrate configuration fragments into
555 Busybox because BusyBox handles its own core features and then allows
556 you to add configuration fragments on top.
557
558Iterate on the Process
559----------------------
560
561If you have not reached your goals on system size, you need to iterate
562on the process. The process is the same. Use the tools and see just what
563is taking up 90% of the root filesystem and the kernel. Decide what you
564can eliminate without limiting your device beyond what you need.
565
566Depending on your system, a good place to look might be Busybox, which
567provides a stripped down version of Unix tools in a single, executable
568file. You might be able to drop virtual terminal services or perhaps
569ipv6.
570
571Building Images for More than One Machine
572=========================================
573
574A common scenario developers face is creating images for several
575different machines that use the same software environment. In this
576situation, it is tempting to set the tunings and optimization flags for
577each build specifically for the targeted hardware (i.e. "maxing out" the
578tunings). Doing so can considerably add to build times and package feed
579maintenance collectively for the machines. For example, selecting tunes
580that are extremely specific to a CPU core used in a system might enable
581some micro optimizations in GCC for that particular system but would
582otherwise not gain you much of a performance difference across the other
583systems as compared to using a more general tuning across all the builds
584(e.g. setting :term:`DEFAULTTUNE`
585specifically for each machine's build). Rather than "max out" each
586build's tunings, you can take steps that cause the OpenEmbedded build
587system to reuse software across the various machines where it makes
588sense.
589
590If build speed and package feed maintenance are considerations, you
591should consider the points in this section that can help you optimize
592your tunings to best consider build times and package feed maintenance.
593
594- *Share the* :term:`Build Directory` *:* If at all possible, share the
595 :term:`TMPDIR` across builds. The Yocto Project supports switching between
596 different :term:`MACHINE` values in the same :term:`TMPDIR`. This practice
597 is well supported and regularly used by developers when building for
598 multiple machines. When you use the same :term:`TMPDIR` for multiple
599 machine builds, the OpenEmbedded build system can reuse the existing native
600 and often cross-recipes for multiple machines. Thus, build time decreases.
601
602 .. note::
603
604 If :term:`DISTRO` settings change or fundamental configuration settings
605 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
606 Sharing :term:`TMPDIR` under these circumstances might work but since it is
607 not guaranteed, you should use a clean :term:`TMPDIR`.
608
609- *Enable the Appropriate Package Architecture:* By default, the
610 OpenEmbedded build system enables three levels of package
611 architectures: "all", "tune" or "package", and "machine". Any given
612 recipe usually selects one of these package architectures (types) for
613 its output. Depending for what a given recipe creates packages,
614 making sure you enable the appropriate package architecture can
615 directly impact the build time.
616
617 A recipe that just generates scripts can enable "all" architecture
618 because there are no binaries to build. To specifically enable "all"
619 architecture, be sure your recipe inherits the
620 :ref:`ref-classes-allarch` class.
621 This class is useful for "all" architectures because it configures
622 many variables so packages can be used across multiple architectures.
623
624 If your recipe needs to generate packages that are machine-specific
625 or when one of the build or runtime dependencies is already
626 machine-architecture dependent, which makes your recipe also
627 machine-architecture dependent, make sure your recipe enables the
628 "machine" package architecture through the
629 :term:`MACHINE_ARCH`
630 variable::
631
632 PACKAGE_ARCH = "${MACHINE_ARCH}"
633
634 When you do not
635 specifically enable a package architecture through the
636 :term:`PACKAGE_ARCH`, The
637 OpenEmbedded build system defaults to the
638 :term:`TUNE_PKGARCH` setting::
639
640 PACKAGE_ARCH = "${TUNE_PKGARCH}"
641
642- *Choose a Generic Tuning File if Possible:* Some tunes are more
643 generic and can run on multiple targets (e.g. an ``armv5`` set of
644 packages could run on ``armv6`` and ``armv7`` processors in most
645 cases). Similarly, ``i486`` binaries could work on ``i586`` and
646 higher processors. You should realize, however, that advances on
647 newer processor versions would not be used.
648
649 If you select the same tune for several different machines, the
650 OpenEmbedded build system reuses software previously built, thus
651 speeding up the overall build time. Realize that even though a new
652 sysroot for each machine is generated, the software is not recompiled
653 and only one package feed exists.
654
655- *Manage Granular Level Packaging:* Sometimes there are cases where
656 injecting another level of package architecture beyond the three
657 higher levels noted earlier can be useful. For example, consider how
658 NXP (formerly Freescale) allows for the easy reuse of binary packages
659 in their layer
660 :yocto_git:`meta-freescale </meta-freescale/>`.
661 In this example, the
662 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
663 class shares GPU packages for i.MX53 boards because all boards share
664 the AMD GPU. The i.MX6-based boards can do the same because all
665 boards share the Vivante GPU. This class inspects the BitBake
666 datastore to identify if the package provides or depends on one of
667 the sub-architecture values. If so, the class sets the
668 :term:`PACKAGE_ARCH` value
669 based on the ``MACHINE_SUBARCH`` value. If the package does not
670 provide or depend on one of the sub-architecture values but it
671 matches a value in the machine-specific filter, it sets
672 :term:`MACHINE_ARCH`. This
673 behavior reduces the number of packages built and saves build time by
674 reusing binaries.
675
676- *Use Tools to Debug Issues:* Sometimes you can run into situations
677 where software is being rebuilt when you think it should not be. For
678 example, the OpenEmbedded build system might not be using shared
679 state between machines when you think it should be. These types of
680 situations are usually due to references to machine-specific
681 variables such as :term:`MACHINE`,
682 :term:`SERIAL_CONSOLES`,
683 :term:`XSERVER`,
684 :term:`MACHINE_FEATURES`,
685 and so forth in code that is supposed to only be tune-specific or
686 when the recipe depends
687 (:term:`DEPENDS`,
688 :term:`RDEPENDS`,
689 :term:`RRECOMMENDS`,
690 :term:`RSUGGESTS`, and so forth)
691 on some other recipe that already has
692 :term:`PACKAGE_ARCH` defined
693 as "${MACHINE_ARCH}".
694
695 .. note::
696
697 Patches to fix any issues identified are most welcome as these
698 issues occasionally do occur.
699
700 For such cases, you can use some tools to help you sort out the
701 situation:
702
703 - ``state-diff-machines.sh``*:* You can find this tool in the
704 ``scripts`` directory of the Source Repositories. See the comments
705 in the script for information on how to use the tool.
706
707 - *BitBake's "-S printdiff" Option:* Using this option causes
708 BitBake to try to establish the most recent signature match
709 (e.g. in the shared state cache) and then compare matched signatures
710 to determine the stamps and delta where these two stamp trees diverge.
711
712Building Software from an External Source
713=========================================
714
715By default, the OpenEmbedded build system uses the :term:`Build Directory`
716when building source code. The build process involves fetching the source
717files, unpacking them, and then patching them if necessary before the build
718takes place.
719
720There are situations where you might want to build software from source
721files that are external to and thus outside of the OpenEmbedded build
722system. For example, suppose you have a project that includes a new BSP
723with a heavily customized kernel. And, you want to minimize exposing the
724build system to the development team so that they can focus on their
725project and maintain everyone's workflow as much as possible. In this
726case, you want a kernel source directory on the development machine
727where the development occurs. You want the recipe's
728:term:`SRC_URI` variable to point to
729the external directory and use it as is, not copy it.
730
731To build from software that comes from an external source, all you need to do
732is inherit the :ref:`ref-classes-externalsrc` class and then set
733the :term:`EXTERNALSRC` variable to point to your external source code. Here
734are the statements to put in your ``local.conf`` file::
735
736 INHERIT += "externalsrc"
737 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
738
739This next example shows how to accomplish the same thing by setting
740:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
741
742 EXTERNALSRC = "path"
743 EXTERNALSRC_BUILD = "path"
744
745.. note::
746
747 In order for these settings to take effect, you must globally or
748 locally inherit the :ref:`ref-classes-externalsrc` class.
749
750By default, :ref:`ref-classes-externalsrc` builds the source code in a
751directory separate from the external source directory as specified by
752:term:`EXTERNALSRC`. If you need
753to have the source built in the same directory in which it resides, or
754some other nominated directory, you can set
755:term:`EXTERNALSRC_BUILD`
756to point to that directory::
757
758 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
759
760Replicating a Build Offline
761===========================
762
763It can be useful to take a "snapshot" of upstream sources used in a
764build and then use that "snapshot" later to replicate the build offline.
765To do so, you need to first prepare and populate your downloads
766directory your "snapshot" of files. Once your downloads directory is
767ready, you can use it at any time and from any machine to replicate your
768build.
769
770Follow these steps to populate your Downloads directory:
771
772#. *Create a Clean Downloads Directory:* Start with an empty downloads
773 directory (:term:`DL_DIR`). You
774 start with an empty downloads directory by either removing the files
775 in the existing directory or by setting :term:`DL_DIR` to point to either
776 an empty location or one that does not yet exist.
777
778#. *Generate Tarballs of the Source Git Repositories:* Edit your
779 ``local.conf`` configuration file as follows::
780
781 DL_DIR = "/home/your-download-dir/"
782 BB_GENERATE_MIRROR_TARBALLS = "1"
783
784 During
785 the fetch process in the next step, BitBake gathers the source files
786 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
787 the
788 :term:`BB_GENERATE_MIRROR_TARBALLS`
789 variable for more information.
790
791#. *Populate Your Downloads Directory Without Building:* Use BitBake to
792 fetch your sources but inhibit the build::
793
794 $ bitbake target --runonly=fetch
795
796 The downloads directory (i.e. ``${DL_DIR}``) now has
797 a "snapshot" of the source files in the form of tarballs, which can
798 be used for the build.
799
800#. *Optionally Remove Any Git or other SCM Subdirectories From the
801 Downloads Directory:* If you want, you can clean up your downloads
802 directory by removing any Git or other Source Control Management
803 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
804 already contain these subdirectories.
805
806Once your downloads directory has everything it needs regarding source
807files, you can create your "own-mirror" and build your target.
808Understand that you can use the files to build the target offline from
809any machine and at any time.
810
811Follow these steps to build your target using the files in the downloads
812directory:
813
814#. *Using Local Files Only:* Inside your ``local.conf`` file, add the
815 :term:`SOURCE_MIRROR_URL` variable, inherit the
816 :ref:`ref-classes-own-mirrors` class, and add the
817 :term:`BB_NO_NETWORK` variable to your ``local.conf``::
818
819 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
820 INHERIT += "own-mirrors"
821 BB_NO_NETWORK = "1"
822
823 The :term:`SOURCE_MIRROR_URL` and :ref:`ref-classes-own-mirrors`
824 class set up the system to use the downloads directory as your "own
825 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
826 BitBake's fetching process in step 3 stays local, which means files
827 from your "own-mirror" are used.
828
829#. *Start With a Clean Build:* You can start with a clean build by
830 removing the ``${``\ :term:`TMPDIR`\ ``}`` directory or using a new
831 :term:`Build Directory`.
832
833#. *Build Your Target:* Use BitBake to build your target::
834
835 $ bitbake target
836
837 The build completes using the known local "snapshot" of source
838 files from your mirror. The resulting tarballs for your "snapshot" of
839 source files are in the downloads directory.
840
841 .. note::
842
843 The offline build does not work if recipes attempt to find the
844 latest version of software by setting
845 :term:`SRCREV` to
846 ``${``\ :term:`AUTOREV`\ ``}``::
847
848 SRCREV = "${AUTOREV}"
849
850 When a recipe sets :term:`SRCREV` to
851 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
852 attempt to determine the latest version of software from the SCM.
853 Typically, recipes that use :term:`AUTOREV` are custom or modified
854 recipes. Recipes that reside in public repositories usually do not
855 use :term:`AUTOREV`.
856
857 If you do have recipes that use :term:`AUTOREV`, you can take steps to
858 still use the recipes in an offline build. Do the following:
859
860 #. Use a configuration generated by enabling :ref:`build
861 history <dev-manual/build-quality:maintaining build output quality>`.
862
863 #. Use the ``buildhistory-collect-srcrevs`` command to collect the
864 stored :term:`SRCREV` values from the build's history. For more
865 information on collecting these values, see the
866 ":ref:`dev-manual/build-quality:build history package information`"
867 section.
868
869 #. Once you have the correct source revisions, you can modify
870 those recipes to set :term:`SRCREV` to specific versions of the
871 software.
872