From 355103f8cf2295fb2de91ce25a084e286bdeadeb Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Tue, 23 Jan 2018 11:19:45 -0800 Subject: dev-manual, ref-manual: Moved build history info to dev-manual Fixes [YOCTO #12370] The section in the ref-manual on build history has been moved to the dev-manual. It is more of a "how-to" piece of information than a reference. (From yocto-docs rev: 9634bd8dc51e2972e6a5f3a3d3b4256c8ca8749c) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- .../dev-manual/dev-manual-common-tasks.xml | 564 ++++++++++++++++++++- .../dev-manual/figures/buildhistory-web.png | Bin 0 -> 49966 bytes documentation/dev-manual/figures/buildhistory.png | Bin 0 -> 44900 bytes 3 files changed, 562 insertions(+), 2 deletions(-) create mode 100644 documentation/dev-manual/figures/buildhistory-web.png create mode 100644 documentation/dev-manual/figures/buildhistory.png (limited to 'documentation/dev-manual') diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml index 13a84b9c22..0c2101fde9 100644 --- a/documentation/dev-manual/dev-manual-common-tasks.xml +++ b/documentation/dev-manual/dev-manual-common-tasks.xml @@ -7432,8 +7432,8 @@ Some notes from Cal: BUILDHISTORY_COMMIT = "1" For information on build history, see the - "Maintaining Build Output Quality" - section in the Yocto Project Reference Manual. + "Maintaining Build Output Quality" + section. @@ -9089,6 +9089,566 @@ Some notes from Cal: + + + +
+ Maintaining Build Output Quality + + + Many factors can influence the quality of a build. + For example, if you upgrade a recipe to use a new version of an + upstream software package or you experiment with some new + configuration options, subtle changes can occur that you might + not detect until later. + Consider the case where your recipe is using a newer version of + an upstream package. + In this case, a new version of a piece of software might + introduce an optional dependency on another library, which is + auto-detected. + If that library has already been built when the software is + building, the software will link to the built library and that + library will be pulled into your image along with the new + software even if you did not want the library. + + + + The + buildhistory + class exists to help you maintain the quality of your build + output. + You can use the class to highlight unexpected and possibly + unwanted changes in the build output. + When you enable build history, it records information about the + contents of each package and image and then commits that + information to a local Git repository where you can examine + the information. + + + + The remainder of this section describes the following: + + + How you can enable and disable build history + + + How to understand what the build history contains + + + How to limit the information used for build history + + + How to examine the build history from both a + command-line and web interface + + + + +
+ Enabling and Disabling Build History + + + Build history is disabled by default. + To enable it, add the following INHERIT + statement and set the + BUILDHISTORY_COMMIT + variable to "1" at the end of your + conf/local.conf file found in the + Build Directory: + + INHERIT += "buildhistory" + BUILDHISTORY_COMMIT = "1" + + Enabling build history as previously described causes the + OpenEmbedded build system to collect build output information + and commit it as a single commit to a local + Git + repository. + + Enabling build history increases your build times slightly, + particularly for images, and increases the amount of disk + space used during the build. + + + + + You can disable build history by removing the previous + statements from your conf/local.conf + file. + +
+ +
+ Understanding What the Build History Contains + + + Build history information is kept in + ${TOPDIR}/buildhistory + in the Build Directory as defined by the + BUILDHISTORY_DIR + variable. + The following is an example abbreviated listing: + + + + + At the top level, a metadata-revs + file exists that lists the revisions of the repositories for + the enabled layers when the build was produced. + The rest of the data splits into separate + packages, images + and sdk directories, the contents of + which are described as follows. + + +
+ Build History Package Information + + + The history for each package contains a text file that has + name-value pairs with information about the package. + For example, + buildhistory/packages/i586-poky-linux/busybox/busybox/latest + contains the following: + + PV = 1.22.1 + PR = r32 + RPROVIDES = + RDEPENDS = glibc (>= 2.20) update-alternatives-opkg + RRECOMMENDS = busybox-syslog busybox-udhcpc update-rc.d + PKGSIZE = 540168 + FILES = /usr/bin/* /usr/sbin/* /usr/lib/busybox/* /usr/lib/lib*.so.* \ + /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d \ + /usr/lib/udev/rules.d /usr/share/busybox /usr/lib/busybox/* \ + /usr/share/pixmaps /usr/share/applications /usr/share/idl \ + /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers + FILELIST = /bin/busybox /bin/busybox.nosuid /bin/busybox.suid /bin/sh \ + /etc/busybox.links.nosuid /etc/busybox.links.suid + + Most of these name-value pairs correspond to variables + used to produce the package. + The exceptions are FILELIST, which + is the actual list of files in the package, and + PKGSIZE, which is the total size of + files in the package in bytes. + + + + A file also exists that corresponds to the recipe from + which the package came (e.g. + buildhistory/packages/i586-poky-linux/busybox/latest): + + PV = 1.22.1 + PR = r32 + DEPENDS = initscripts kern-tools-native update-rc.d-native \ + virtual/i586-poky-linux-compilerlibs virtual/i586-poky-linux-gcc \ + virtual/libc virtual/update-alternatives + PACKAGES = busybox-ptest busybox-httpd busybox-udhcpd busybox-udhcpc \ + busybox-syslog busybox-mdev busybox-hwclock busybox-dbg \ + busybox-staticdev busybox-dev busybox-doc busybox-locale busybox + + + + + Finally, for those recipes fetched from a version control + system (e.g., Git), a file exists that lists source + revisions that are specified in the recipe and lists + the actual revisions used during the build. + Listed and actual revisions might differ when + SRCREV + is set to + ${AUTOREV}. + Here is an example assuming + buildhistory/packages/qemux86-poky-linux/linux-yocto/latest_srcrev): + + # SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" + SRCREV_machine = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" + # SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f" + SRCREV_meta = "a227f20eff056e511d504b2e490f3774ab260d6f" + + You can use the + buildhistory-collect-srcrevs + command with the -a option to + collect the stored SRCREV values + from build history and report them in a format suitable for + use in global configuration (e.g., + local.conf or a distro include file) + to override floating AUTOREV values + to a fixed set of revisions. + Here is some example output from this command: + + $ buildhistory-collect-srcrevs -a + # i586-poky-linux + SRCREV_pn-glibc = "b8079dd0d360648e4e8de48656c5c38972621072" + SRCREV_pn-glibc-initial = "b8079dd0d360648e4e8de48656c5c38972621072" + SRCREV_pn-opkg-utils = "53274f087565fd45d8452c5367997ba6a682a37a" + SRCREV_pn-kmod = "fd56638aed3fe147015bfa10ed4a5f7491303cb4" + # x86_64-linux + SRCREV_pn-gtk-doc-stub-native = "1dea266593edb766d6d898c79451ef193eb17cfa" + SRCREV_pn-dtc-native = "65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf" + SRCREV_pn-update-rc.d-native = "eca680ddf28d024954895f59a241a622dd575c11" + SRCREV_glibc_pn-cross-localedef-native = "b8079dd0d360648e4e8de48656c5c38972621072" + SRCREV_localedef_pn-cross-localedef-native = "c833367348d39dad7ba018990bfdaffaec8e9ed3" + SRCREV_pn-prelink-native = "faa069deec99bf61418d0bab831c83d7c1b797ca" + SRCREV_pn-opkg-utils-native = "53274f087565fd45d8452c5367997ba6a682a37a" + SRCREV_pn-kern-tools-native = "23345b8846fe4bd167efdf1bd8a1224b2ba9a5ff" + SRCREV_pn-kmod-native = "fd56638aed3fe147015bfa10ed4a5f7491303cb4" + # qemux86-poky-linux + SRCREV_machine_pn-linux-yocto = "38cd560d5022ed2dbd1ab0dca9642e47c98a0aa1" + SRCREV_meta_pn-linux-yocto = "a227f20eff056e511d504b2e490f3774ab260d6f" + # all-poky-linux + SRCREV_pn-update-rc.d = "eca680ddf28d024954895f59a241a622dd575c11" + + + Here are some notes on using the + buildhistory-collect-srcrevs + command: + + + By default, only values where the + SRCREV was not hardcoded + (usually when AUTOREV + is used) are reported. + Use the -a option to + see all SRCREV values. + + + The output statements might not have any effect + if overrides are applied elsewhere in the + build system configuration. + Use the -f option to add + the forcevariable override + to each output line if you need to work around + this restriction. + + + The script does apply special handling when + building for multiple machines. + However, the script does place a comment before + each set of values that specifies which + triplet to which they belong as previously + shown (e.g., + i586-poky-linux). + + + + +
+ +
+ Build History Image Information + + + The files produced for each image are as follows: + + + image-files: + A directory containing selected files from the root + filesystem. + The files are defined by + BUILDHISTORY_IMAGE_FILES. + + + build-id.txt: + Human-readable information about the build + configuration and metadata source revisions. + This file contains the full build header as printed + by BitBake. + + + *.dot: + Dependency graphs for the image that are + compatible with graphviz. + + + files-in-image.txt: + A list of files in the image with permissions, + owner, group, size, and symlink information. + + + image-info.txt: + A text file containing name-value pairs with + information about the image. + See the following listing example for more + information. + + + installed-package-names.txt: + A list of installed packages by name only. + + + installed-package-sizes.txt: + A list of installed packages ordered by size. + + + installed-packages.txt: + A list of installed packages with full package + filenames. + + + + Installed package information is able to be gathered + and produced even if package management is disabled + for the final image. + + + + + Here is an example of image-info.txt: + + DISTRO = poky + DISTRO_VERSION = 1.7 + USER_CLASSES = buildstats image-mklibs image-prelink + IMAGE_CLASSES = image_types + IMAGE_FEATURES = debug-tweaks + IMAGE_LINGUAS = + IMAGE_INSTALL = packagegroup-core-boot run-postinsts + BAD_RECOMMENDATIONS = + NO_RECOMMENDATIONS = + PACKAGE_EXCLUDE = + ROOTFS_POSTPROCESS_COMMAND = write_package_manifest; license_create_manifest; \ + write_image_manifest ; buildhistory_list_installed_image ; \ + buildhistory_get_image_installed ; ssh_allow_empty_password; \ + postinst_enable_logging; rootfs_update_timestamp ; ssh_disable_dns_lookup ; + IMAGE_POSTPROCESS_COMMAND = buildhistory_get_imageinfo ; + IMAGESIZE = 6900 + + Other than IMAGESIZE, which is the + total size of the files in the image in Kbytes, the + name-value pairs are variables that may have influenced the + content of the image. + This information is often useful when you are trying to + determine why a change in the package or file + listings has occurred. + +
+ +
+ Using Build History to Gather Image Information Only + + + As you can see, build history produces image information, + including dependency graphs, so you can see why something + was pulled into the image. + If you are just interested in this information and not + interested in collecting specific package or SDK + information, you can enable writing only image information + without any history by adding the following to your + conf/local.conf file found in the + Build Directory: + + INHERIT += "buildhistory" + BUILDHISTORY_COMMIT = "0" + BUILDHISTORY_FEATURES = "image" + + Here, you set the + BUILDHISTORY_FEATURES + variable to use the image feature only. + +
+ +
+ Build History SDK Information + + + Build history collects similar information on the contents + of SDKs + (e.g. bitbake -c populate_sdk imagename) + as compared to information it collects for images. + Furthermore, this information differs depending on whether + an extensible or standard SDK is being produced. + + + + The following list shows the files produced for SDKs: + + + files-in-sdk.txt: + A list of files in the SDK with permissions, + owner, group, size, and symlink information. + This list includes both the host and target parts + of the SDK. + + + sdk-info.txt: + A text file containing name-value pairs with + information about the SDK. + See the following listing example for more + information. + + + sstate-task-sizes.txt: + A text file containing name-value pairs with + information about task group sizes + (e.g. do_populate_sysroot + tasks have a total size). + The sstate-task-sizes.txt file + exists only when an extensible SDK is created. + + + sstate-package-sizes.txt: + A text file containing name-value pairs with + information for the shared-state packages and + sizes in the SDK. + The sstate-package-sizes.txt + file exists only when an extensible SDK is created. + + + sdk-files: + A folder that contains copies of the files + mentioned in + BUILDHISTORY_SDK_FILES if the + files are present in the output. + Additionally, the default value of + BUILDHISTORY_SDK_FILES is + specific to the extensible SDK although you can + set it differently if you would like to pull in + specific files from the standard SDK. + + The default files are + conf/local.conf, + conf/bblayers.conf, + conf/auto.conf, + conf/locked-sigs.inc, and + conf/devtool.conf. + Thus, for an extensible SDK, these files get + copied into the sdk-files + directory. + + + The following information appears under + each of the host + and target directories + for the portions of the SDK that run on the host + and on the target, respectively: + + The following files for the most part are empty + when producing an extensible SDK because this + type of SDK is not constructed from packages + as is the standard SDK. + + + + depends.dot: + Dependency graph for the SDK that is + compatible with + graphviz. + + + installed-package-names.txt: + A list of installed packages by name only. + + + installed-package-sizes.txt: + A list of installed packages ordered by size. + + + installed-packages.txt: + A list of installed packages with full + package filenames. + + + + + + + + Here is an example of sdk-info.txt: + + DISTRO = poky + DISTRO_VERSION = 1.3+snapshot-20130327 + SDK_NAME = poky-glibc-i686-arm + SDK_VERSION = 1.3+snapshot + SDKMACHINE = + SDKIMAGE_FEATURES = dev-pkgs dbg-pkgs + BAD_RECOMMENDATIONS = + SDKSIZE = 352712 + + Other than SDKSIZE, which is the + total size of the files in the SDK in Kbytes, the + name-value pairs are variables that might have influenced + the content of the SDK. + This information is often useful when you are trying to + determine why a change in the package or file listings + has occurred. + +
+ +
+ Examining Build History Information + + + You can examine build history output from the command + line or from a web interface. + + + + To see any changes that have occurred (assuming you have + BUILDHISTORY_COMMIT = "1"), + you can simply use any Git command that allows you to + view the history of a repository. + Here is one method: + + $ git log -p + + You need to realize, however, that this method does show + changes that are not significant (e.g. a package's size + changing by a few bytes). + + + + A command-line tool called + buildhistory-diff does exist, though, + that queries the Git repository and prints just the + differences that might be significant in human-readable + form. + Here is an example: + + $ ~/poky/poky/scripts/buildhistory-diff . HEAD^ + Changes to images/qemux86_64/glibc/core-image-minimal (files-in-image.txt): + /etc/anotherpkg.conf was added + /sbin/anotherpkg was added + * (installed-package-names.txt): + * anotherpkg was added + Changes to images/qemux86_64/glibc/core-image-minimal (installed-package-names.txt): + anotherpkg was added + packages/qemux86_64-poky-linux/v86d: PACKAGES: added "v86d-extras" + * PR changed from "r0" to "r1" + * PV changed from "0.1.10" to "0.1.12" + packages/qemux86_64-poky-linux/v86d/v86d: PKGSIZE changed from 110579 to 144381 (+30%) + * PR changed from "r0" to "r1" + * PV changed from "0.1.10" to "0.1.12" + + + The buildhistory-diff tool + requires the GitPython package. + Be sure to install it using Pip3 as follows: + + $ pip3 install GitPython --user + + Alternatively, you can install + python3-git using the appropriate + distribution package manager (e.g. + apt-get, dnf, + or zipper). + + + + + To see changes to the build history using a web interface, + follow the instruction in the README + file here. + . + + + + Here is a sample screenshot of the interface: + + +
+
+
+
Performing Automated Runtime Testing diff --git a/documentation/dev-manual/figures/buildhistory-web.png b/documentation/dev-manual/figures/buildhistory-web.png new file mode 100644 index 0000000000..f6db86c977 Binary files /dev/null and b/documentation/dev-manual/figures/buildhistory-web.png differ diff --git a/documentation/dev-manual/figures/buildhistory.png b/documentation/dev-manual/figures/buildhistory.png new file mode 100644 index 0000000000..bd5f8a4908 Binary files /dev/null and b/documentation/dev-manual/figures/buildhistory.png differ -- cgit v1.2.3-54-g00ecf