dev-manual, ref-manual: Moved build history info to dev-manual

Fixes [YOCTO #12370]

The section in the ref-manual on build history has been moved to
the dev-manual.  It is more of a "how-to" piece of information than
a reference.

(From yocto-docs rev: 9634bd8dc51e2972e6a5f3a3d3b4256c8ca8749c)

Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

1 files changed, 562 insertions, 2 deletions

diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml
index 13a84b9c22..0c2101fde9 100644
--- a/documentation/dev-manual/dev-manual-common-tasks.xml
+++ b/documentation/dev-manual/dev-manual-common-tasks.xml
@@ -7432,8 +7432,8 @@ Some notes from Cal:
      BUILDHISTORY_COMMIT = "1"
                     </literallayout>
                     For information on build history, see the
-                    "<ulink url='&YOCTO_DOCS_REF_URL;#maintaining-build-output-quality'>Maintaining Build Output Quality</ulink>"
-                    section in the Yocto Project Reference Manual.
+                    "<link linkend='maintaining-build-output-quality'>Maintaining Build Output Quality</link>"
+                    section.
                 </para>
 
                 <note>
@@ -9089,6 +9089,566 @@ Some notes from Cal:
         </section>
     </section>
 
+
+
+
+    <section id='maintaining-build-output-quality'>
+        <title>Maintaining Build Output Quality</title>
+
+        <para>
+            Many factors can influence the quality of a build.
+            For example, if you upgrade a recipe to use a new version of an
+            upstream software package or you experiment with some new
+            configuration options, subtle changes can occur that you might
+            not detect until later.
+            Consider the case where your recipe is using a newer version of
+            an upstream package.
+            In this case, a new version of a piece of software might
+            introduce an optional dependency on another library, which is
+            auto-detected.
+            If that library has already been built when the software is
+            building, the software will link to the built library and that
+            library will be pulled into your image along with the new
+            software even if you did not want the library.
+        </para>
+
+        <para>
+            The
+            <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-buildhistory'><filename>buildhistory</filename></ulink>
+            class exists to help you maintain the quality of your build
+            output.
+            You can use the class to highlight unexpected and possibly
+            unwanted changes in the build output.
+            When you enable build history, it records information about the
+            contents of each package and image and then commits that
+            information to a local Git repository where you can examine
+            the information.
+        </para>
+
+        <para>
+            The remainder of this section describes the following:
+            <itemizedlist>
+               <listitem><para>
+                   How you can enable and disable build history
+                   </para></listitem>
+               <listitem><para>
+                   How to understand what the build history contains
+                   </para></listitem>
+               <listitem><para>
+                   How to limit the information used for build history
+                   </para></listitem>
+               <listitem><para>
+                   How to examine the build history from both a
+                   command-line and web interface
+                   </para></listitem>
+           </itemizedlist>
+        </para>
+
+        <section id='enabling-and-disabling-build-history'>
+            <title>Enabling and Disabling Build History</title>
+
+            <para>
+                Build history is disabled by default.
+                To enable it, add the following <filename>INHERIT</filename>
+                statement and set the
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></ulink>
+                variable to "1" at the end of your
+                <filename>conf/local.conf</filename> file found in the
+                <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>:
+                <literallayout class='monospaced'>
+     INHERIT += "buildhistory"
+     BUILDHISTORY_COMMIT = "1"
+                </literallayout>
+                Enabling build history as previously described causes the
+                OpenEmbedded build system to collect build output information
+                and commit it as a single commit to a local
+                <ulink url='&YOCTO_DOCS_OVERVIEW_URL;#git'>Git</ulink>
+                repository.
+                <note>
+                    Enabling build history increases your build times slightly,
+                    particularly for images, and increases the amount of disk
+                    space used during the build.
+                </note>
+            </para>
+
+            <para>
+                You can disable build history by removing the previous
+                statements from your <filename>conf/local.conf</filename>
+                file.
+            </para>
+        </section>
+
+        <section id='understanding-what-the-build-history-contains'>
+            <title>Understanding What the Build History Contains</title>
+
+            <para>
+                Build history information is kept in
+                <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-TOPDIR'><filename>TOPDIR</filename></ulink><filename>}/buildhistory</filename>
+                in the Build Directory as defined by the
+                <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_DIR'><filename>BUILDHISTORY_DIR</filename></ulink>
+                variable.
+                The following is an example abbreviated listing:
+                <imagedata fileref="figures/buildhistory.png" align="center" width="6in" depth="4in" />
+            </para>
+
+            <para>
+                At the top level, a <filename>metadata-revs</filename>
+                file exists that lists the revisions of the repositories for
+                the enabled layers when the build was produced.
+                The rest of the data splits into separate
+                <filename>packages</filename>, <filename>images</filename>
+                and <filename>sdk</filename> directories, the contents of
+                which are described as follows.
+            </para>
+
+            <section id='build-history-package-information'>
+                <title>Build History Package Information</title>
+
+                <para>
+                    The history for each package contains a text file that has
+                    name-value pairs with information about the package.
+                    For example,
+                    <filename>buildhistory/packages/i586-poky-linux/busybox/busybox/latest</filename>
+                    contains the following:
+                    <literallayout class='monospaced'>
+     PV = 1.22.1
+     PR = r32
+     RPROVIDES =
+     RDEPENDS = glibc (>= 2.20) update-alternatives-opkg
+     RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d
+     PKGSIZE = 540168
+     FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \
+        /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \
+        /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \
+        /usr/share/pixmaps /usr/share/applications /usr/share/idl \
+        /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers
+     FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \
+        /etc/busybox.links.nosuid /etc/busybox.links.suid
+                    </literallayout>
+                    Most of these name-value pairs correspond to variables
+                    used to produce the package.
+                    The exceptions are <filename>FILELIST</filename>, which
+                    is the actual list of files in the package, and
+                    <filename>PKGSIZE</filename>, which is the total size of
+                    files in the package in bytes.
+                </para>
+
+                <para>
+                    A file also exists that corresponds to the recipe from
+                    which the package came (e.g.
+                    <filename>buildhistory/packages/i586-poky-linux/busybox/latest</filename>):
+                    <literallayout class='monospaced'>
+     PV = 1.22.1
+     PR = r32
+     DEPENDS = initscripts kern-tools-native update-rc.d-native \
+        virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \
+        virtual/libc virtual/update-alternatives
+     PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \
+        busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \
+        busybox-staticdev busybox-dev busybox-doc busybox-locale busybox
+                    </literallayout>
+                </para>
+
+                <para>
+                    Finally, for those recipes fetched from a version control
+                    system (e.g., Git), a file exists that lists source
+                    revisions that are specified in the recipe and lists
+                    the actual revisions used during the build.
+                    Listed and actual revisions might differ when
+                    <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink>
+                    is set to
+                    ${<ulink url='&YOCTO_DOCS_REF_URL;#var-AUTOREV'><filename>AUTOREV</filename></ulink>}.
+                    Here is an example assuming
+                    <filename>buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev</filename>):
+                    <literallayout class='monospaced'>
+     # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
+     SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
+     # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
+     SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f"
+                    </literallayout>
+                    You can use the
+                    <filename>buildhistory-collect-srcrevs</filename>
+                    command with the <filename>-a</filename> option to
+                    collect the stored <filename>SRCREV</filename> values
+                    from build history and report them in a format suitable for
+                    use in global configuration (e.g.,
+                    <filename>local.conf</filename> or a distro include file)
+                    to override floating <filename>AUTOREV</filename> values
+                    to a fixed set of revisions.
+                    Here is some example output from this command:
+                    <literallayout class='monospaced'>
+     $ buildhistory-collect-srcrevs -a
+     # i586-poky-linux
+     SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072"
+     SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072"
+     SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a"
+     SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
+     # x86_64-linux
+     SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa"
+     SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf"
+     SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11"
+     SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072"
+     SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3"
+     SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca"
+     SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a"
+     SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff"
+     SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4"
+     # qemux86-poky-linux
+     SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1"
+     SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f"
+     # all-poky-linux
+     SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11"
+                    </literallayout>
+                    <note>
+                        Here are some notes on using the
+                        <filename>buildhistory-collect-srcrevs</filename>
+                        command:
+                        <itemizedlist>
+                            <listitem><para>
+                                By default, only values where the
+                                <filename>SRCREV</filename> was not hardcoded
+                                (usually when <filename>AUTOREV</filename>
+                                is used) are reported.
+                                Use the <filename>-a</filename> option to
+                                see all <filename>SRCREV</filename> values.
+                                </para></listitem>
+                            <listitem><para>
+                                The output statements might not have any effect
+                                if overrides are applied elsewhere in the
+                                build system configuration.
+                                Use the <filename>-f</filename> option to add
+                                the <filename>forcevariable</filename> override
+                                to each output line if you need to work around
+                                this restriction.
+                                </para></listitem>
+                            <listitem><para>
+                                The script does apply special handling when
+                                building for multiple machines.
+                                However, the script does place a comment before
+                                each set of values that specifies which
+                                triplet to which they belong as previously
+                                shown (e.g.,
+                                <filename>i586-poky-linux</filename>).
+                                </para></listitem>
+                        </itemizedlist>
+                    </note>
+                </para>
+            </section>
+
+            <section id='build-history-image-information'>
+                <title>Build History Image Information</title>
+
+                <para>
+                    The files produced for each image are as follows:
+                    <itemizedlist>
+                        <listitem><para>
+                            <filename>image-files:</filename>
+                            A directory containing selected files from the root
+                            filesystem.
+                            The files are defined by
+                            <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_IMAGE_FILES'><filename>BUILDHISTORY_IMAGE_FILES</filename></ulink>.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>build-id.txt:</filename>
+                            Human-readable information about the build
+                            configuration and metadata source revisions.
+                            This file contains the full build header as printed
+                            by BitBake.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>*.dot:</filename>
+                            Dependency graphs for the image that are
+                            compatible with <filename>graphviz</filename>.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>files-in-image.txt:</filename>
+                                A list of files in the image with permissions,
+                            owner, group, size, and symlink information.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>image-info.txt:</filename>
+                            A text file containing name-value pairs with
+                            information about the image.
+                            See the following listing example for more
+                            information.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>installed-package-names.txt:</filename>
+                            A list of installed packages by name only.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>installed-package-sizes.txt:</filename>
+                            A list of installed packages ordered by size.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>installed-packages.txt:</filename>
+                            A list of installed packages with full package
+                            filenames.
+                            </para></listitem>
+                    </itemizedlist>
+                    <note>
+                        Installed package information is able to be gathered
+                        and produced even if package management is disabled
+                        for the final image.
+                    </note>
+                </para>
+
+                <para>
+                    Here is an example of <filename>image-info.txt</filename>:
+                    <literallayout class='monospaced'>
+     DISTRO = poky
+     DISTRO_VERSION = 1.7
+     USER_CLASSES = buildstats image-mklibs image-prelink
+     IMAGE_CLASSES = image_types
+     IMAGE_FEATURES = debug-tweaks
+     IMAGE_LINGUAS =
+     IMAGE_INSTALL = packagegroup-core-boot run-postinsts
+     BAD_RECOMMENDATIONS =
+     NO_RECOMMENDATIONS =
+     PACKAGE_EXCLUDE =
+     ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \
+        write_image_manifest ; buildhistory_list_installed_image ; \
+        buildhistory_get_image_installed ; ssh_allow_empty_password;  \
+        postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ;
+     IMAGE_POSTPROCESS_COMMAND =   buildhistory_get_imageinfo ;
+     IMAGESIZE = 6900
+                    </literallayout>
+                    Other than <filename>IMAGESIZE</filename>, which is the
+                    total size of the files in the image in Kbytes, the
+                    name-value pairs are variables that may have influenced the
+                    content of the image.
+                    This information is often useful when you are trying to
+                    determine why a change in the package or file
+                    listings has occurred.
+                </para>
+            </section>
+
+            <section id='using-build-history-to-gather-image-information-only'>
+                <title>Using Build History to Gather Image Information Only</title>
+
+                <para>
+                    As you can see, build history produces image information,
+                    including dependency graphs, so you can see why something
+                    was pulled into the image.
+                    If you are just interested in this information and not
+                    interested in collecting specific package or SDK
+                    information, you can enable writing only image information
+                    without any history by adding the following to your
+                    <filename>conf/local.conf</filename> file found in the
+                    <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>:
+                    <literallayout class='monospaced'>
+     INHERIT += "buildhistory"
+     BUILDHISTORY_COMMIT = "0"
+     BUILDHISTORY_FEATURES = "image"
+                    </literallayout>
+                    Here, you set the
+                    <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_FEATURES'><filename>BUILDHISTORY_FEATURES</filename></ulink>
+                    variable to use the image feature only.
+                </para>
+            </section>
+
+            <section id='build-history-sdk-information'>
+                <title>Build History SDK Information</title>
+
+                <para>
+                    Build history collects similar information on the contents
+                    of SDKs
+                    (e.g. <filename>bitbake -c populate_sdk imagename</filename>)
+                    as compared to information it collects for images.
+                    Furthermore, this information differs depending on whether
+                    an extensible or standard SDK is being produced.
+                </para>
+
+                <para>
+                    The following list shows the files produced for SDKs:
+                    <itemizedlist>
+                        <listitem><para>
+                            <filename>files-in-sdk.txt:</filename>
+                            A list of files in the SDK with permissions,
+                            owner, group, size, and symlink information.
+                            This list includes both the host and target parts
+                            of the SDK.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>sdk-info.txt:</filename>
+                            A text file containing name-value pairs with
+                            information about the SDK.
+                            See the following listing example for more
+                            information.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>sstate-task-sizes.txt:</filename>
+                            A text file containing name-value pairs with
+                            information about task group sizes
+                            (e.g. <filename>do_populate_sysroot</filename>
+                            tasks have a total size).
+                            The <filename>sstate-task-sizes.txt</filename> file
+                            exists only when an extensible SDK is created.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>sstate-package-sizes.txt:</filename>
+                            A text file containing name-value pairs with
+                            information for the shared-state packages and
+                            sizes in the SDK.
+                            The <filename>sstate-package-sizes.txt</filename>
+                            file exists only when an extensible SDK is created.
+                            </para></listitem>
+                        <listitem><para>
+                            <filename>sdk-files:</filename>
+                            A folder that contains copies of the files
+                            mentioned in
+                            <filename>BUILDHISTORY_SDK_FILES</filename> if the
+                            files are present in the output.
+                            Additionally, the default value of
+                            <filename>BUILDHISTORY_SDK_FILES</filename> is
+                            specific to the extensible SDK although you can
+                            set it differently if you would like to pull in
+                            specific files from the standard SDK.</para>
+
+                            <para>The default files are
+                            <filename>conf/local.conf</filename>,
+                            <filename>conf/bblayers.conf</filename>,
+                            <filename>conf/auto.conf</filename>,
+                            <filename>conf/locked-sigs.inc</filename>, and
+                            <filename>conf/devtool.conf</filename>.
+                            Thus, for an extensible SDK, these files get
+                            copied into the <filename>sdk-files</filename>
+                            directory.
+                            </para></listitem>
+                        <listitem><para>
+                            The following information appears under
+                            each of the <filename>host</filename>
+                            and <filename>target</filename> directories
+                            for the portions of the SDK that run on the host
+                            and on the target, respectively:
+                            <note>
+                                The following files for the most part are empty
+                                when producing an extensible SDK because this
+                                type of SDK is not constructed from packages
+                                as is the standard SDK.
+                            </note>
+                            <itemizedlist>
+                                <listitem><para>
+                                    <filename>depends.dot:</filename>
+                                    Dependency graph for the SDK that is
+                                    compatible with
+                                    <filename>graphviz</filename>.
+                                    </para></listitem>
+                                <listitem><para>
+                                    <filename>installed-package-names.txt:</filename>
+                                    A list of installed packages by name only.
+                                    </para></listitem>
+                                <listitem><para>
+                                    <filename>installed-package-sizes.txt:</filename>
+                                    A list of installed packages ordered by size.
+                                    </para></listitem>
+                                <listitem><para>
+                                    <filename>installed-packages.txt:</filename>
+                                    A list of installed packages with full
+                                    package filenames.
+                                    </para></listitem>
+                                </itemizedlist>
+                            </para></listitem>
+                    </itemizedlist>
+                </para>
+
+                <para>
+                    Here is an example of <filename>sdk-info.txt</filename>:
+                    <literallayout class='monospaced'>
+     DISTRO = poky
+     DISTRO_VERSION = 1.3+snapshot-20130327
+     SDK_NAME = poky-glibc-i686-arm
+     SDK_VERSION = 1.3+snapshot
+     SDKMACHINE =
+     SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs
+     BAD_RECOMMENDATIONS =
+     SDKSIZE = 352712
+                    </literallayout>
+                    Other than <filename>SDKSIZE</filename>, which is the
+                    total size of the files in the SDK in Kbytes, the
+                    name-value pairs are variables that might have influenced
+                    the content of the SDK.
+                    This information is often useful when you are trying to
+                    determine why a change in the package or file listings
+                    has occurred.
+                </para>
+            </section>
+
+            <section id='examining-build-history-information'>
+                <title>Examining Build History Information</title>
+
+                <para>
+                    You can examine build history output from the command
+                    line or from a web interface.
+                </para>
+
+                <para>
+                    To see any changes that have occurred (assuming you have
+                    <ulink url='&YOCTO_DOCS_REF_URL;#var-BUILDHISTORY_COMMIT'><filename>BUILDHISTORY_COMMIT</filename></ulink><filename>&nbsp;= "1"</filename>),
+                    you can simply use any Git command that allows you to
+                    view the history of a repository.
+                    Here is one method:
+                    <literallayout class='monospaced'>
+      $ git log -p
+                    </literallayout>
+                    You need to realize, however, that this method does show
+                    changes that are not significant (e.g. a package's size
+                    changing by a few bytes).
+                </para>
+
+                <para>
+                    A command-line tool called
+                    <filename>buildhistory-diff</filename> does exist, though,
+                    that queries the Git repository and prints just the
+                    differences that might be significant in human-readable
+                    form.
+                    Here is an example:
+                    <literallayout class='monospaced'>
+     $ ~/poky/poky/scripts/buildhistory-diff . HEAD^
+     Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt):
+        /etc/anotherpkg.conf was added
+        /sbin/anotherpkg was added
+        * (installed-package-names.txt):
+        *   anotherpkg was added
+     Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt):
+        anotherpkg was added
+     packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras"
+        * PR changed from "r0" to "r1"
+        * PV changed from "0.1.10" to "0.1.12"
+     packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%)
+        * PR changed from "r0" to "r1"
+        * PV changed from "0.1.10" to "0.1.12"
+                    </literallayout>
+                    <note>
+                        The <filename>buildhistory-diff</filename> tool
+                        requires the <filename>GitPython</filename> package.
+                        Be sure to install it using Pip3 as follows:
+                        <literallayout class='monospaced'>
+   $ pip3 install GitPython --user
+                        </literallayout>
+                        Alternatively, you can install
+                        <filename>python3-git</filename> using the appropriate
+                        distribution package manager (e.g.
+                        <filename>apt-get</filename>, <filename>dnf</filename>,
+                        or <filename>zipper</filename>).
+                    </note>
+                </para>
+
+                <para>
+                    To see changes to the build history using a web interface,
+                    follow the instruction in the <filename>README</filename>
+                    file here.
+                    <ulink url='http://git.yoctoproject.org/cgit/cgit.cgi/buildhistory-web/'></ulink>.
+                </para>
+
+                <para>
+                    Here is a sample screenshot of the interface:
+                    <imagedata fileref="figures/buildhistory-web.png" align="center" scalefit="1" width="130%" contentdepth="130%" />
+                </para>
+            </section>
+        </section>
+    </section>
+
     <section id="performing-automated-runtime-testing">
         <title>Performing Automated Runtime Testing</title>