summaryrefslogtreecommitdiffstats
path: root/documentation/dev-manual/building.rst
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/dev-manual/building.rst')
-rw-r--r--documentation/dev-manual/building.rst942
1 files changed, 942 insertions, 0 deletions
diff --git a/documentation/dev-manual/building.rst b/documentation/dev-manual/building.rst
new file mode 100644
index 0000000000..fe502690dd
--- /dev/null
+++ b/documentation/dev-manual/building.rst
@@ -0,0 +1,942 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Building
4********
5
6This section describes various build procedures, such as the steps
7needed for a simple build, building a target for multiple configurations,
8generating an image for more than one machine, and so forth.
9
10Building a Simple Image
11=======================
12
13In the development environment, you need to build an image whenever you
14change hardware support, add or change system libraries, or add or
15change services that have dependencies. There are several methods that allow
16you to build an image within the Yocto Project. This section presents
17the basic steps you need to build a simple image using BitBake from a
18build host running Linux.
19
20.. note::
21
22 - For information on how to build an image using
23 :term:`Toaster`, see the
24 :doc:`/toaster-manual/index`.
25
26 - For information on how to use ``devtool`` to build images, see the
27 ":ref:`sdk-manual/extensible:using \`\`devtool\`\` in your sdk workflow`"
28 section in the Yocto Project Application Development and the
29 Extensible Software Development Kit (eSDK) manual.
30
31 - For a quick example on how to build an image using the
32 OpenEmbedded build system, see the
33 :doc:`/brief-yoctoprojectqs/index` document.
34
35 - You can also use the `Yocto Project BitBake
36 <https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`__
37 extension for Visual Studio Code to build images.
38
39The build process creates an entire Linux distribution from source and
40places it in your :term:`Build Directory` under ``tmp/deploy/images``. For
41detailed information on the build process using BitBake, see the
42":ref:`overview-manual/concepts:images`" section in the Yocto Project Overview
43and Concepts Manual.
44
45The following figure and list overviews the build process:
46
47.. image:: figures/bitbake-build-flow.png
48 :width: 100%
49
50#. *Set up Your Host Development System to Support Development Using the
51 Yocto Project*: See the ":doc:`start`" section for options on how to get a
52 build host ready to use the Yocto Project.
53
54#. *Initialize the Build Environment:* Initialize the build environment
55 by sourcing the build environment script (i.e.
56 :ref:`structure-core-script`)::
57
58 $ source oe-init-build-env [build_dir]
59
60 When you use the initialization script, the OpenEmbedded build system
61 uses ``build`` as the default :term:`Build Directory` in your current work
62 directory. You can use a `build_dir` argument with the script to
63 specify a different :term:`Build Directory`.
64
65 .. note::
66
67 A common practice is to use a different :term:`Build Directory` for
68 different targets; for example, ``~/build/x86`` for a ``qemux86``
69 target, and ``~/build/arm`` for a ``qemuarm`` target. In any
70 event, it's typically cleaner to locate the :term:`Build Directory`
71 somewhere outside of your source directory.
72
73#. *Make Sure Your* ``local.conf`` *File is Correct*: Ensure the
74 ``conf/local.conf`` configuration file, which is found in the
75 :term:`Build Directory`, is set up how you want it. This file defines many
76 aspects of the build environment including the target machine architecture
77 through the :term:`MACHINE` variable, the packaging format used during
78 the build (:term:`PACKAGE_CLASSES`), and a centralized tarball download
79 directory through the :term:`DL_DIR` variable.
80
81#. *Build the Image:* Build the image using the ``bitbake`` command::
82
83 $ bitbake target
84
85 .. note::
86
87 For information on BitBake, see the :doc:`bitbake:index`.
88
89 The target is the name of the recipe you want to build. Common
90 targets are the images in ``meta/recipes-core/images``,
91 ``meta/recipes-sato/images``, and so forth all found in the
92 :term:`Source Directory`. Alternatively, the target
93 can be the name of a recipe for a specific piece of software such as
94 BusyBox. For more details about the images the OpenEmbedded build
95 system supports, see the
96 ":ref:`ref-manual/images:Images`" chapter in the Yocto
97 Project Reference Manual.
98
99 As an example, the following command builds the
100 ``core-image-minimal`` image::
101
102 $ bitbake core-image-minimal
103
104 Once an
105 image has been built, it often needs to be installed. The images and
106 kernels built by the OpenEmbedded build system are placed in the
107 :term:`Build Directory` in ``tmp/deploy/images``. For information on how to
108 run pre-built images such as ``qemux86`` and ``qemuarm``, see the
109 :doc:`/sdk-manual/index` manual. For
110 information about how to install these images, see the documentation
111 for your particular board or machine.
112
113Building Images for Multiple Targets Using Multiple Configurations
114==================================================================
115
116You can use a single ``bitbake`` command to build multiple images or
117packages for different targets where each image or package requires a
118different configuration (multiple configuration builds). The builds, in
119this scenario, are sometimes referred to as "multiconfigs", and this
120section uses that term throughout.
121
122This section describes how to set up for multiple configuration builds
123and how to account for cross-build dependencies between the
124multiconfigs.
125
126Setting Up and Running a Multiple Configuration Build
127-----------------------------------------------------
128
129To accomplish a multiple configuration build, you must define each
130target's configuration separately using a parallel configuration file in
131the :term:`Build Directory` or configuration directory within a layer, and you
132must follow a required file hierarchy. Additionally, you must enable the
133multiple configuration builds in your ``local.conf`` file.
134
135Follow these steps to set up and execute multiple configuration builds:
136
137- *Create Separate Configuration Files*: You need to create a single
138 configuration file for each build target (each multiconfig).
139 The configuration definitions are implementation dependent but often
140 each configuration file will define the machine and the
141 temporary directory BitBake uses for the build. Whether the same
142 temporary directory (:term:`TMPDIR`) can be shared will depend on what is
143 similar and what is different between the configurations. Multiple MACHINE
144 targets can share the same (:term:`TMPDIR`) as long as the rest of the
145 configuration is the same, multiple :term:`DISTRO` settings would need separate
146 (:term:`TMPDIR`) directories.
147
148 For example, consider a scenario with two different multiconfigs for the same
149 :term:`MACHINE`: "qemux86" built
150 for two distributions such as "poky" and "poky-lsb". In this case,
151 you would need to use the different :term:`TMPDIR`.
152
153 Here is an example showing the minimal statements needed in a
154 configuration file for a "qemux86" target whose temporary build
155 directory is ``tmpmultix86``::
156
157 MACHINE = "qemux86"
158 TMPDIR = "${TOPDIR}/tmpmultix86"
159
160 The location for these multiconfig configuration files is specific.
161 They must reside in the current :term:`Build Directory` in a sub-directory of
162 ``conf`` named ``multiconfig`` or within a layer's ``conf`` directory
163 under a directory named ``multiconfig``. Here is an example that defines
164 two configuration files for the "x86" and "arm" multiconfigs:
165
166 .. image:: figures/multiconfig_files.png
167 :align: center
168 :width: 50%
169
170 The usual :term:`BBPATH` search path is used to locate multiconfig files in
171 a similar way to other conf files.
172
173- *Add the BitBake Multi-configuration Variable to the Local
174 Configuration File*: Use the
175 :term:`BBMULTICONFIG`
176 variable in your ``conf/local.conf`` configuration file to specify
177 each multiconfig. Continuing with the example from the previous
178 figure, the :term:`BBMULTICONFIG` variable needs to enable two
179 multiconfigs: "x86" and "arm" by specifying each configuration file::
180
181 BBMULTICONFIG = "x86 arm"
182
183 .. note::
184
185 A "default" configuration already exists by definition. This
186 configuration is named: "" (i.e. empty string) and is defined by
187 the variables coming from your ``local.conf``
188 file. Consequently, the previous example actually adds two
189 additional configurations to your build: "arm" and "x86" along
190 with "".
191
192- *Launch BitBake*: Use the following BitBake command form to launch
193 the multiple configuration build::
194
195 $ bitbake [mc:multiconfigname:]target [[[mc:multiconfigname:]target] ... ]
196
197 For the example in this section, the following command applies::
198
199 $ bitbake mc:x86:core-image-minimal mc:arm:core-image-sato mc::core-image-base
200
201 The previous BitBake command builds a ``core-image-minimal`` image
202 that is configured through the ``x86.conf`` configuration file, a
203 ``core-image-sato`` image that is configured through the ``arm.conf``
204 configuration file and a ``core-image-base`` that is configured
205 through your ``local.conf`` configuration file.
206
207.. note::
208
209 Support for multiple configuration builds in the Yocto Project &DISTRO;
210 (&DISTRO_NAME;) Release does not include Shared State (sstate)
211 optimizations. Consequently, if a build uses the same object twice
212 in, for example, two different :term:`TMPDIR`
213 directories, the build either loads from an existing sstate cache for
214 that build at the start or builds the object fresh.
215
216Enabling Multiple Configuration Build Dependencies
217--------------------------------------------------
218
219Sometimes dependencies can exist between targets (multiconfigs) in a
220multiple configuration build. For example, suppose that in order to
221build a ``core-image-sato`` image for an "x86" multiconfig, the root
222filesystem of an "arm" multiconfig must exist. This dependency is
223essentially that the
224:ref:`ref-tasks-image` task in the
225``core-image-sato`` recipe depends on the completion of the
226:ref:`ref-tasks-rootfs` task of the
227``core-image-minimal`` recipe.
228
229To enable dependencies in a multiple configuration build, you must
230declare the dependencies in the recipe using the following statement
231form::
232
233 task_or_package[mcdepends] = "mc:from_multiconfig:to_multiconfig:recipe_name:task_on_which_to_depend"
234
235To better show how to use this statement, consider the example scenario
236from the first paragraph of this section. The following statement needs
237to be added to the recipe that builds the ``core-image-sato`` image::
238
239 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_rootfs"
240
241In this example, the `from_multiconfig` is "x86". The `to_multiconfig` is "arm". The
242task on which the :ref:`ref-tasks-image` task in the recipe depends is the
243:ref:`ref-tasks-rootfs` task from the ``core-image-minimal`` recipe associated
244with the "arm" multiconfig.
245
246Once you set up this dependency, you can build the "x86" multiconfig
247using a BitBake command as follows::
248
249 $ bitbake mc:x86:core-image-sato
250
251This command executes all the tasks needed to create the
252``core-image-sato`` image for the "x86" multiconfig. Because of the
253dependency, BitBake also executes through the :ref:`ref-tasks-rootfs` task for the
254"arm" multiconfig build.
255
256Having a recipe depend on the root filesystem of another build might not
257seem that useful. Consider this change to the statement in the
258``core-image-sato`` recipe::
259
260 do_image[mcdepends] = "mc:x86:arm:core-image-minimal:do_image"
261
262In this case, BitBake must
263create the ``core-image-minimal`` image for the "arm" build since the
264"x86" build depends on it.
265
266Because "x86" and "arm" are enabled for multiple configuration builds
267and have separate configuration files, BitBake places the artifacts for
268each build in the respective temporary build directories (i.e.
269:term:`TMPDIR`).
270
271Building an Initial RAM Filesystem (Initramfs) Image
272====================================================
273
274An initial RAM filesystem (:term:`Initramfs`) image provides a temporary root
275filesystem used for early system initialization, typically providing tools and
276loading modules needed to locate and mount the final root filesystem.
277
278Follow these steps to create an :term:`Initramfs` image:
279
280#. *Create the Initramfs Image Recipe:* You can reference the
281 ``core-image-minimal-initramfs.bb`` recipe found in the
282 ``meta/recipes-core`` directory of the :term:`Source Directory`
283 as an example from which to work.
284
285#. *Decide if You Need to Bundle the Initramfs Image Into the Kernel
286 Image:* If you want the :term:`Initramfs` image that is built to be bundled
287 in with the kernel image, set the :term:`INITRAMFS_IMAGE_BUNDLE`
288 variable to ``"1"`` in your ``local.conf`` configuration file and set the
289 :term:`INITRAMFS_IMAGE` variable in the recipe that builds the kernel image.
290
291 Setting the :term:`INITRAMFS_IMAGE_BUNDLE` flag causes the :term:`Initramfs`
292 image to be unpacked into the ``${B}/usr/`` directory. The unpacked
293 :term:`Initramfs` image is then passed to the kernel's ``Makefile`` using the
294 :term:`CONFIG_INITRAMFS_SOURCE` variable, allowing the :term:`Initramfs`
295 image to be built into the kernel normally.
296
297#. *Optionally Add Items to the Initramfs Image Through the Initramfs
298 Image Recipe:* If you add items to the :term:`Initramfs` image by way of its
299 recipe, you should use :term:`PACKAGE_INSTALL` rather than
300 :term:`IMAGE_INSTALL`. :term:`PACKAGE_INSTALL` gives more direct control of
301 what is added to the image as compared to the defaults you might not
302 necessarily want that are set by the :ref:`ref-classes-image`
303 or :ref:`ref-classes-core-image` classes.
304
305#. *Build the Kernel Image and the Initramfs Image:* Build your kernel
306 image using BitBake. Because the :term:`Initramfs` image recipe is a
307 dependency of the kernel image, the :term:`Initramfs` image is built as well
308 and bundled with the kernel image if you used the
309 :term:`INITRAMFS_IMAGE_BUNDLE` variable described earlier.
310
311Bundling an Initramfs Image From a Separate Multiconfig
312-------------------------------------------------------
313
314There may be a case where we want to build an :term:`Initramfs` image which does not
315inherit the same distro policy as our main image, for example, we may want
316our main image to use ``TCLIBC="glibc"``, but to use ``TCLIBC="musl"`` in our :term:`Initramfs`
317image to keep a smaller footprint. However, by performing the steps mentioned
318above the :term:`Initramfs` image will inherit ``TCLIBC="glibc"`` without allowing us
319to override it.
320
321To achieve this, you need to perform some additional steps:
322
323#. *Create a multiconfig for your Initramfs image:* You can perform the steps
324 on ":ref:`dev-manual/building:building images for multiple targets using multiple configurations`" to create a separate multiconfig.
325 For the sake of simplicity let's assume such multiconfig is called: ``initramfscfg.conf`` and
326 contains the variables::
327
328 TMPDIR="${TOPDIR}/tmp-initramfscfg"
329 TCLIBC="musl"
330
331#. *Set additional Initramfs variables on your main configuration:*
332 Additionally, on your main configuration (``local.conf``) you need to set the
333 variables::
334
335 INITRAMFS_MULTICONFIG = "initramfscfg"
336 INITRAMFS_DEPLOY_DIR_IMAGE = "${TOPDIR}/tmp-initramfscfg/deploy/images/${MACHINE}"
337
338 The variables :term:`INITRAMFS_MULTICONFIG` and :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
339 are used to create a multiconfig dependency from the kernel to the :term:`INITRAMFS_IMAGE`
340 to be built coming from the ``initramfscfg`` multiconfig, and to let the
341 buildsystem know where the :term:`INITRAMFS_IMAGE` will be located.
342
343 Building a system with such configuration will build the kernel using the
344 main configuration but the :ref:`ref-tasks-bundle_initramfs` task will grab the
345 selected :term:`INITRAMFS_IMAGE` from :term:`INITRAMFS_DEPLOY_DIR_IMAGE`
346 instead, resulting in a musl based :term:`Initramfs` image bundled in the kernel
347 but a glibc based main image.
348
349 The same is applicable to avoid inheriting :term:`DISTRO_FEATURES` on :term:`INITRAMFS_IMAGE`
350 or to build a different :term:`DISTRO` for it such as ``poky-tiny``.
351
352
353Building a Tiny System
354======================
355
356Very small distributions have some significant advantages such as
357requiring less on-die or in-package memory (cheaper), better performance
358through efficient cache usage, lower power requirements due to less
359memory, faster boot times, and reduced development overhead. Some
360real-world examples where a very small distribution gives you distinct
361advantages are digital cameras, medical devices, and small headless
362systems.
363
364This section presents information that shows you how you can trim your
365distribution to even smaller sizes than the ``poky-tiny`` distribution,
366which is around 5 Mbytes, that can be built out-of-the-box using the
367Yocto Project.
368
369Tiny System Overview
370--------------------
371
372The following list presents the overall steps you need to consider and
373perform to create distributions with smaller root filesystems, achieve
374faster boot times, maintain your critical functionality, and avoid
375initial RAM disks:
376
377- :ref:`Determine your goals and guiding principles
378 <dev-manual/building:goals and guiding principles>`
379
380- :ref:`dev-manual/building:understand what contributes to your image size`
381
382- :ref:`Reduce the size of the root filesystem
383 <dev-manual/building:trim the root filesystem>`
384
385- :ref:`Reduce the size of the kernel <dev-manual/building:trim the kernel>`
386
387- :ref:`dev-manual/building:remove package management requirements`
388
389- :ref:`dev-manual/building:look for other ways to minimize size`
390
391- :ref:`dev-manual/building:iterate on the process`
392
393Goals and Guiding Principles
394----------------------------
395
396Before you can reach your destination, you need to know where you are
397going. Here is an example list that you can use as a guide when creating
398very small distributions:
399
400- Determine how much space you need (e.g. a kernel that is 1 Mbyte or
401 less and a root filesystem that is 3 Mbytes or less).
402
403- Find the areas that are currently taking 90% of the space and
404 concentrate on reducing those areas.
405
406- Do not create any difficult "hacks" to achieve your goals.
407
408- Leverage the device-specific options.
409
410- Work in a separate layer so that you keep changes isolated. For
411 information on how to create layers, see the
412 ":ref:`dev-manual/layers:understanding and creating layers`" section.
413
414Understand What Contributes to Your Image Size
415----------------------------------------------
416
417It is easiest to have something to start with when creating your own
418distribution. You can use the Yocto Project out-of-the-box to create the
419``poky-tiny`` distribution. Ultimately, you will want to make changes in
420your own distribution that are likely modeled after ``poky-tiny``.
421
422.. note::
423
424 To use ``poky-tiny`` in your build, set the :term:`DISTRO` variable in your
425 ``local.conf`` file to "poky-tiny" as described in the
426 ":ref:`dev-manual/custom-distribution:creating your own distribution`"
427 section.
428
429Understanding some memory concepts will help you reduce the system size.
430Memory consists of static, dynamic, and temporary memory. Static memory
431is the TEXT (code), DATA (initialized data in the code), and BSS
432(uninitialized data) sections. Dynamic memory represents memory that is
433allocated at runtime: stacks, hash tables, and so forth. Temporary
434memory is recovered after the boot process. This memory consists of
435memory used for decompressing the kernel and for the ``__init__``
436functions.
437
438To help you see where you currently are with kernel and root filesystem
439sizes, you can use two tools found in the :term:`Source Directory`
440in the
441``scripts/tiny/`` directory:
442
443- ``ksize.py``: Reports component sizes for the kernel build objects.
444
445- ``dirsize.py``: Reports component sizes for the root filesystem.
446
447This next tool and command help you organize configuration fragments and
448view file dependencies in a human-readable form:
449
450- ``merge_config.sh``: Helps you manage configuration files and
451 fragments within the kernel. With this tool, you can merge individual
452 configuration fragments together. The tool allows you to make
453 overrides and warns you of any missing configuration options. The
454 tool is ideal for allowing you to iterate on configurations, create
455 minimal configurations, and create configuration files for different
456 machines without having to duplicate your process.
457
458 The ``merge_config.sh`` script is part of the Linux Yocto kernel Git
459 repositories (i.e. ``linux-yocto-3.14``, ``linux-yocto-3.10``,
460 ``linux-yocto-3.8``, and so forth) in the ``scripts/kconfig``
461 directory.
462
463 For more information on configuration fragments, see the
464 ":ref:`kernel-dev/common:creating configuration fragments`"
465 section in the Yocto Project Linux Kernel Development Manual.
466
467- ``bitbake -u taskexp -g bitbake_target``: Using the BitBake command
468 with these options brings up a Dependency Explorer from which you can
469 view file dependencies. Understanding these dependencies allows you
470 to make informed decisions when cutting out various pieces of the
471 kernel and root filesystem.
472
473Trim the Root Filesystem
474------------------------
475
476The root filesystem is made up of packages for booting, libraries, and
477applications. To change things, you can configure how the packaging
478happens, which changes the way you build them. You can also modify the
479filesystem itself or select a different filesystem.
480
481First, find out what is hogging your root filesystem by running the
482``dirsize.py`` script from your root directory::
483
484 $ cd root-directory-of-image
485 $ dirsize.py 100000 > dirsize-100k.log
486 $ cat dirsize-100k.log
487
488You can apply a filter to the script to ignore files
489under a certain size. The previous example filters out any files below
490100 Kbytes. The sizes reported by the tool are uncompressed, and thus
491will be smaller by a relatively constant factor in a compressed root
492filesystem. When you examine your log file, you can focus on areas of
493the root filesystem that take up large amounts of memory.
494
495You need to be sure that what you eliminate does not cripple the
496functionality you need. One way to see how packages relate to each other
497is by using the Dependency Explorer UI with the BitBake command::
498
499 $ cd image-directory
500 $ bitbake -u taskexp -g image
501
502Use the interface to
503select potential packages you wish to eliminate and see their dependency
504relationships.
505
506When deciding how to reduce the size, get rid of packages that result in
507minimal impact on the feature set. For example, you might not need a VGA
508display. Or, you might be able to get by with ``devtmpfs`` and ``mdev``
509instead of ``udev``.
510
511Use your ``local.conf`` file to make changes. For example, to eliminate
512``udev`` and ``glib``, set the following in the local configuration
513file::
514
515 VIRTUAL-RUNTIME_dev_manager = ""
516
517Finally, you should consider exactly the type of root filesystem you
518need to meet your needs while also reducing its size. For example,
519consider ``cramfs``, ``squashfs``, ``ubifs``, ``ext2``, or an
520:term:`Initramfs` using ``initramfs``. Be aware that ``ext3`` requires a 1
521Mbyte journal. If you are okay with running read-only, you do not need
522this journal.
523
524.. note::
525
526 After each round of elimination, you need to rebuild your system and
527 then use the tools to see the effects of your reductions.
528
529Trim the Kernel
530---------------
531
532The kernel is built by including policies for hardware-independent
533aspects. What subsystems do you enable? For what architecture are you
534building? Which drivers do you build by default?
535
536.. note::
537
538 You can modify the kernel source if you want to help with boot time.
539
540Run the ``ksize.py`` script from the top-level Linux build directory to
541get an idea of what is making up the kernel::
542
543 $ cd top-level-linux-build-directory
544 $ ksize.py > ksize.log
545 $ cat ksize.log
546
547When you examine the log, you will see how much space is taken up with
548the built-in ``.o`` files for drivers, networking, core kernel files,
549filesystem, sound, and so forth. The sizes reported by the tool are
550uncompressed, and thus will be smaller by a relatively constant factor
551in a compressed kernel image. Look to reduce the areas that are large
552and taking up around the "90% rule."
553
554To examine, or drill down, into any particular area, use the ``-d``
555option with the script::
556
557 $ ksize.py -d > ksize.log
558
559Using this option
560breaks out the individual file information for each area of the kernel
561(e.g. drivers, networking, and so forth).
562
563Use your log file to see what you can eliminate from the kernel based on
564features you can let go. For example, if you are not going to need
565sound, you do not need any drivers that support sound.
566
567After figuring out what to eliminate, you need to reconfigure the kernel
568to reflect those changes during the next build. You could run
569``menuconfig`` and make all your changes at once. However, that makes it
570difficult to see the effects of your individual eliminations and also
571makes it difficult to replicate the changes for perhaps another target
572device. A better method is to start with no configurations using
573``allnoconfig``, create configuration fragments for individual changes,
574and then manage the fragments into a single configuration file using
575``merge_config.sh``. The tool makes it easy for you to iterate using the
576configuration change and build cycle.
577
578Each time you make configuration changes, you need to rebuild the kernel
579and check to see what impact your changes had on the overall size.
580
581Remove Package Management Requirements
582--------------------------------------
583
584Packaging requirements add size to the image. One way to reduce the size
585of the image is to remove all the packaging requirements from the image.
586This reduction includes both removing the package manager and its unique
587dependencies as well as removing the package management data itself.
588
589To eliminate all the packaging requirements for an image, be sure that
590"package-management" is not part of your
591:term:`IMAGE_FEATURES`
592statement for the image. When you remove this feature, you are removing
593the package manager as well as its dependencies from the root
594filesystem.
595
596Look for Other Ways to Minimize Size
597------------------------------------
598
599Depending on your particular circumstances, other areas that you can
600trim likely exist. The key to finding these areas is through tools and
601methods described here combined with experimentation and iteration. Here
602are a couple of areas to experiment with:
603
604- ``glibc``: In general, follow this process:
605
606 #. Remove ``glibc`` features from
607 :term:`DISTRO_FEATURES`
608 that you think you do not need.
609
610 #. Build your distribution.
611
612 #. If the build fails due to missing symbols in a package, determine
613 if you can reconfigure the package to not need those features. For
614 example, change the configuration to not support wide character
615 support as is done for ``ncurses``. Or, if support for those
616 characters is needed, determine what ``glibc`` features provide
617 the support and restore the configuration.
618
619 4. Rebuild and repeat the process.
620
621- ``busybox``: For BusyBox, use a process similar as described for
622 ``glibc``. A difference is you will need to boot the resulting system
623 to see if you are able to do everything you expect from the running
624 system. You need to be sure to integrate configuration fragments into
625 Busybox because BusyBox handles its own core features and then allows
626 you to add configuration fragments on top.
627
628Iterate on the Process
629----------------------
630
631If you have not reached your goals on system size, you need to iterate
632on the process. The process is the same. Use the tools and see just what
633is taking up 90% of the root filesystem and the kernel. Decide what you
634can eliminate without limiting your device beyond what you need.
635
636Depending on your system, a good place to look might be Busybox, which
637provides a stripped down version of Unix tools in a single, executable
638file. You might be able to drop virtual terminal services or perhaps
639ipv6.
640
641Building Images for More than One Machine
642=========================================
643
644A common scenario developers face is creating images for several
645different machines that use the same software environment. In this
646situation, it is tempting to set the tunings and optimization flags for
647each build specifically for the targeted hardware (i.e. "maxing out" the
648tunings). Doing so can considerably add to build times and package feed
649maintenance collectively for the machines. For example, selecting tunes
650that are extremely specific to a CPU core used in a system might enable
651some micro optimizations in GCC for that particular system but would
652otherwise not gain you much of a performance difference across the other
653systems as compared to using a more general tuning across all the builds
654(e.g. setting :term:`DEFAULTTUNE`
655specifically for each machine's build). Rather than "max out" each
656build's tunings, you can take steps that cause the OpenEmbedded build
657system to reuse software across the various machines where it makes
658sense.
659
660If build speed and package feed maintenance are considerations, you
661should consider the points in this section that can help you optimize
662your tunings to best consider build times and package feed maintenance.
663
664- *Share the :term:`Build Directory`:* If at all possible, share the
665 :term:`TMPDIR` across builds. The Yocto Project supports switching between
666 different :term:`MACHINE` values in the same :term:`TMPDIR`. This practice
667 is well supported and regularly used by developers when building for
668 multiple machines. When you use the same :term:`TMPDIR` for multiple
669 machine builds, the OpenEmbedded build system can reuse the existing native
670 and often cross-recipes for multiple machines. Thus, build time decreases.
671
672 .. note::
673
674 If :term:`DISTRO` settings change or fundamental configuration settings
675 such as the filesystem layout, you need to work with a clean :term:`TMPDIR`.
676 Sharing :term:`TMPDIR` under these circumstances might work but since it is
677 not guaranteed, you should use a clean :term:`TMPDIR`.
678
679- *Enable the Appropriate Package Architecture:* By default, the
680 OpenEmbedded build system enables three levels of package
681 architectures: "all", "tune" or "package", and "machine". Any given
682 recipe usually selects one of these package architectures (types) for
683 its output. Depending for what a given recipe creates packages,
684 making sure you enable the appropriate package architecture can
685 directly impact the build time.
686
687 A recipe that just generates scripts can enable "all" architecture
688 because there are no binaries to build. To specifically enable "all"
689 architecture, be sure your recipe inherits the
690 :ref:`ref-classes-allarch` class.
691 This class is useful for "all" architectures because it configures
692 many variables so packages can be used across multiple architectures.
693
694 If your recipe needs to generate packages that are machine-specific
695 or when one of the build or runtime dependencies is already
696 machine-architecture dependent, which makes your recipe also
697 machine-architecture dependent, make sure your recipe enables the
698 "machine" package architecture through the
699 :term:`MACHINE_ARCH`
700 variable::
701
702 PACKAGE_ARCH = "${MACHINE_ARCH}"
703
704 When you do not
705 specifically enable a package architecture through the
706 :term:`PACKAGE_ARCH`, The
707 OpenEmbedded build system defaults to the
708 :term:`TUNE_PKGARCH` setting::
709
710 PACKAGE_ARCH = "${TUNE_PKGARCH}"
711
712- *Choose a Generic Tuning File if Possible:* Some tunes are more
713 generic and can run on multiple targets (e.g. an ``armv5`` set of
714 packages could run on ``armv6`` and ``armv7`` processors in most
715 cases). Similarly, ``i486`` binaries could work on ``i586`` and
716 higher processors. You should realize, however, that advances on
717 newer processor versions would not be used.
718
719 If you select the same tune for several different machines, the
720 OpenEmbedded build system reuses software previously built, thus
721 speeding up the overall build time. Realize that even though a new
722 sysroot for each machine is generated, the software is not recompiled
723 and only one package feed exists.
724
725- *Manage Granular Level Packaging:* Sometimes there are cases where
726 injecting another level of package architecture beyond the three
727 higher levels noted earlier can be useful. For example, consider how
728 NXP (formerly Freescale) allows for the easy reuse of binary packages
729 in their layer
730 :yocto_git:`meta-freescale </meta-freescale/>`.
731 In this example, the
732 :yocto_git:`fsl-dynamic-packagearch </meta-freescale/tree/classes/fsl-dynamic-packagearch.bbclass>`
733 class shares GPU packages for i.MX53 boards because all boards share
734 the AMD GPU. The i.MX6-based boards can do the same because all
735 boards share the Vivante GPU. This class inspects the BitBake
736 datastore to identify if the package provides or depends on one of
737 the sub-architecture values. If so, the class sets the
738 :term:`PACKAGE_ARCH` value
739 based on the ``MACHINE_SUBARCH`` value. If the package does not
740 provide or depend on one of the sub-architecture values but it
741 matches a value in the machine-specific filter, it sets
742 :term:`MACHINE_ARCH`. This
743 behavior reduces the number of packages built and saves build time by
744 reusing binaries.
745
746- *Use Tools to Debug Issues:* Sometimes you can run into situations
747 where software is being rebuilt when you think it should not be. For
748 example, the OpenEmbedded build system might not be using shared
749 state between machines when you think it should be. These types of
750 situations are usually due to references to machine-specific
751 variables such as :term:`MACHINE`,
752 :term:`SERIAL_CONSOLES`,
753 :term:`XSERVER`,
754 :term:`MACHINE_FEATURES`,
755 and so forth in code that is supposed to only be tune-specific or
756 when the recipe depends
757 (:term:`DEPENDS`,
758 :term:`RDEPENDS`,
759 :term:`RRECOMMENDS`,
760 :term:`RSUGGESTS`, and so forth)
761 on some other recipe that already has
762 :term:`PACKAGE_ARCH` defined
763 as "${MACHINE_ARCH}".
764
765 .. note::
766
767 Patches to fix any issues identified are most welcome as these
768 issues occasionally do occur.
769
770 For such cases, you can use some tools to help you sort out the
771 situation:
772
773 - ``state-diff-machines.sh``*:* You can find this tool in the
774 ``scripts`` directory of the Source Repositories. See the comments
775 in the script for information on how to use the tool.
776
777 - *BitBake's "-S printdiff" Option:* Using this option causes
778 BitBake to try to establish the most recent signature match
779 (e.g. in the shared state cache) and then compare matched signatures
780 to determine the stamps and delta where these two stamp trees diverge.
781
782Building Software from an External Source
783=========================================
784
785By default, the OpenEmbedded build system uses the :term:`Build Directory`
786when building source code. The build process involves fetching the source
787files, unpacking them, and then patching them if necessary before the build
788takes place.
789
790There are situations where you might want to build software from source
791files that are external to and thus outside of the OpenEmbedded build
792system. For example, suppose you have a project that includes a new BSP
793with a heavily customized kernel. And, you want to minimize exposing the
794build system to the development team so that they can focus on their
795project and maintain everyone's workflow as much as possible. In this
796case, you want a kernel source directory on the development machine
797where the development occurs. You want the recipe's
798:term:`SRC_URI` variable to point to
799the external directory and use it as is, not copy it.
800
801To build from software that comes from an external source, all you need to do
802is inherit the :ref:`ref-classes-externalsrc` class and then set
803the :term:`EXTERNALSRC` variable to point to your external source code. Here
804are the statements to put in your ``local.conf`` file::
805
806 INHERIT += "externalsrc"
807 EXTERNALSRC:pn-myrecipe = "path-to-your-source-tree"
808
809This next example shows how to accomplish the same thing by setting
810:term:`EXTERNALSRC` in the recipe itself or in the recipe's append file::
811
812 EXTERNALSRC = "path"
813 EXTERNALSRC_BUILD = "path"
814
815.. note::
816
817 In order for these settings to take effect, you must globally or
818 locally inherit the :ref:`ref-classes-externalsrc` class.
819
820By default, :ref:`ref-classes-externalsrc` builds the source code in a
821directory separate from the external source directory as specified by
822:term:`EXTERNALSRC`. If you need
823to have the source built in the same directory in which it resides, or
824some other nominated directory, you can set
825:term:`EXTERNALSRC_BUILD`
826to point to that directory::
827
828 EXTERNALSRC_BUILD:pn-myrecipe = "path-to-your-source-tree"
829
830Replicating a Build Offline
831===========================
832
833It can be useful to take a "snapshot" of upstream sources used in a
834build and then use that "snapshot" later to replicate the build offline.
835To do so, you need to first prepare and populate your downloads
836directory your "snapshot" of files. Once your downloads directory is
837ready, you can use it at any time and from any machine to replicate your
838build.
839
840Follow these steps to populate your Downloads directory:
841
842#. *Create a Clean Downloads Directory:* Start with an empty downloads
843 directory (:term:`DL_DIR`). You
844 start with an empty downloads directory by either removing the files
845 in the existing directory or by setting :term:`DL_DIR` to point to either
846 an empty location or one that does not yet exist.
847
848#. *Generate Tarballs of the Source Git Repositories:* Edit your
849 ``local.conf`` configuration file as follows::
850
851 DL_DIR = "/home/your-download-dir/"
852 BB_GENERATE_MIRROR_TARBALLS = "1"
853
854 During
855 the fetch process in the next step, BitBake gathers the source files
856 and creates tarballs in the directory pointed to by :term:`DL_DIR`. See
857 the
858 :term:`BB_GENERATE_MIRROR_TARBALLS`
859 variable for more information.
860
861#. *Populate Your Downloads Directory Without Building:* Use BitBake to
862 fetch your sources but inhibit the build::
863
864 $ bitbake target --runonly=fetch
865
866 The downloads directory (i.e. ``${DL_DIR}``) now has
867 a "snapshot" of the source files in the form of tarballs, which can
868 be used for the build.
869
870#. *Optionally Remove Any Git or other SCM Subdirectories From the
871 Downloads Directory:* If you want, you can clean up your downloads
872 directory by removing any Git or other Source Control Management
873 (SCM) subdirectories such as ``${DL_DIR}/git2/*``. The tarballs
874 already contain these subdirectories.
875
876Once your downloads directory has everything it needs regarding source
877files, you can create your "own-mirror" and build your target.
878Understand that you can use the files to build the target offline from
879any machine and at any time.
880
881Follow these steps to build your target using the files in the downloads
882directory:
883
884#. *Using Local Files Only:* Inside your ``local.conf`` file, add the
885 :term:`SOURCE_MIRROR_URL` variable, inherit the
886 :ref:`ref-classes-own-mirrors` class, and use the
887 :term:`BB_NO_NETWORK` variable to your ``local.conf``::
888
889 SOURCE_MIRROR_URL ?= "file:///home/your-download-dir/"
890 INHERIT += "own-mirrors"
891 BB_NO_NETWORK = "1"
892
893 The :term:`SOURCE_MIRROR_URL` and :ref:`ref-classes-own-mirrors`
894 class set up the system to use the downloads directory as your "own
895 mirror". Using the :term:`BB_NO_NETWORK` variable makes sure that
896 BitBake's fetching process in step 3 stays local, which means files
897 from your "own-mirror" are used.
898
899#. *Start With a Clean Build:* You can start with a clean build by
900 removing the ``${``\ :term:`TMPDIR`\ ``}`` directory or using a new
901 :term:`Build Directory`.
902
903#. *Build Your Target:* Use BitBake to build your target::
904
905 $ bitbake target
906
907 The build completes using the known local "snapshot" of source
908 files from your mirror. The resulting tarballs for your "snapshot" of
909 source files are in the downloads directory.
910
911 .. note::
912
913 The offline build does not work if recipes attempt to find the
914 latest version of software by setting
915 :term:`SRCREV` to
916 ``${``\ :term:`AUTOREV`\ ``}``::
917
918 SRCREV = "${AUTOREV}"
919
920 When a recipe sets :term:`SRCREV` to
921 ``${``\ :term:`AUTOREV`\ ``}``, the build system accesses the network in an
922 attempt to determine the latest version of software from the SCM.
923 Typically, recipes that use :term:`AUTOREV` are custom or modified
924 recipes. Recipes that reside in public repositories usually do not
925 use :term:`AUTOREV`.
926
927 If you do have recipes that use :term:`AUTOREV`, you can take steps to
928 still use the recipes in an offline build. Do the following:
929
930 #. Use a configuration generated by enabling :ref:`build
931 history <dev-manual/build-quality:maintaining build output quality>`.
932
933 #. Use the ``buildhistory-collect-srcrevs`` command to collect the
934 stored :term:`SRCREV` values from the build's history. For more
935 information on collecting these values, see the
936 ":ref:`dev-manual/build-quality:build history package information`"
937 section.
938
939 #. Once you have the correct source revisions, you can modify
940 those recipes to set :term:`SRCREV` to specific versions of the
941 software.
942