summaryrefslogtreecommitdiffstats
path: root/bitbake/doc
diff options
context:
space:
mode:
authorScott Rifenbark <scott.m.rifenbark@intel.com>2014-01-29 12:06:15 -0600
committerRichard Purdie <richard.purdie@linuxfoundation.org>2014-02-11 12:17:06 +0000
commit3cdf800334bf7f6297e6c15bfc334814e8416380 (patch)
tree6a64a133528164fe04e2be45049c821e74cf6d8f /bitbake/doc
parent89058e1ef7142765e29f5c403470d5bd2932e298 (diff)
downloadpoky-3cdf800334bf7f6297e6c15bfc334814e8416380.tar.gz
bitbake: user-manual-metadata.xml: Added "Checksums (Signatures)" section.
Added this section to the end of the Metadata chapter. (Bitbake rev: b653c58284cafd0b79991520543ca6239705d36b) Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'bitbake/doc')
-rw-r--r--bitbake/doc/user-manual/user-manual-metadata.xml183
1 files changed, 183 insertions, 0 deletions
diff --git a/bitbake/doc/user-manual/user-manual-metadata.xml b/bitbake/doc/user-manual/user-manual-metadata.xml
index d191d3589a..abfc89d75c 100644
--- a/bitbake/doc/user-manual/user-manual-metadata.xml
+++ b/bitbake/doc/user-manual/user-manual-metadata.xml
@@ -1092,4 +1092,187 @@
1092 deletes all the flags for a variable. 1092 deletes all the flags for a variable.
1093 </para> 1093 </para>
1094 </section> 1094 </section>
1095
1096 <section id='task-checksums-and-setscene'>
1097 <title>Task Checksums and Setscene</title>
1098
1099 <para>
1100 This list is a place holder of content that needs explanation here.
1101 Items should be moved to appropriate sections below as completed.
1102 <itemizedlist>
1103 <listitem><para><filename>STAMP</filename></para></listitem>
1104 <listitem><para><filename>STAMPCLEAN</filename></para></listitem>
1105 <listitem><para><filename>BB_STAMP_WHITELIST</filename></para></listitem>
1106 <listitem><para><filename>BB_STAMP_POLICY</filename></para></listitem>
1107 <listitem><para><filename>BB_HASHCHECK_FUNCTION</filename></para></listitem>
1108 <listitem><para><filename>BB_SETSCENE_VERIFY_FUNCTION</filename></para></listitem>
1109 <listitem><para><filename>BB_SETSCENE_DEPVALID</filename></para></listitem>
1110 <listitem><para><filename>BB_TASKHASH</filename></para></listitem>
1111 </itemizedlist>
1112 </para>
1113
1114 <section id='checksums'>
1115 <title>Checksums (Signatures)</title>
1116
1117 <para>
1118 BitBake uses checksums (or signatures) along with the setscene
1119 to determine if a task needs to be run.
1120 This section describes the process.
1121 To help understand how BitBake does this, the section assumes an
1122 OpenEmbedded metadata-based example.
1123 </para>
1124
1125 <para>
1126 The setscene code uses a checksum, which is a unique signature of a task's
1127 inputs, to determine if a task needs to be run again.
1128 Because it is a change in a task's inputs that triggers a rerun, the process
1129 needs to detect all the inputs to a given task.
1130 For shell tasks, this turns out to be fairly easy because
1131 BitBake generates a "run" shell script for each task and
1132 it is possible to create a checksum that gives you a good idea of when
1133 the task's data changes.
1134 </para>
1135
1136 <para>
1137 To complicate the problem, some things should not be included in
1138 the checksum.
1139 First, there is the actual specific build path of a given task -
1140 the working directory.
1141 It does not matter if the work directory changes because it should not
1142 affect the output for target packages.
1143 The simplistic approach for excluding the work directory is to set
1144 it to some fixed value and create the checksum for the "run" script.
1145 </para>
1146
1147 <para>
1148 Another problem results from the "run" scripts containing functions that
1149 might or might not get called.
1150 The incremental build solution contains code that figures out dependencies
1151 between shell functions.
1152 This code is used to prune the "run" scripts down to the minimum set,
1153 thereby alleviating this problem and making the "run" scripts much more
1154 readable as a bonus.
1155 </para>
1156
1157 <para>
1158 So far we have solutions for shell scripts.
1159 What about Python tasks?
1160 The same approach applies even though these tasks are more difficult.
1161 The process needs to figure out what variables a Python function accesses
1162 and what functions it calls.
1163 Again, the incremental build solution contains code that first figures out
1164 the variable and function dependencies, and then creates a checksum for the data
1165 used as the input to the task.
1166 </para>
1167
1168 <para>
1169 Like the working directory case, situations exist where dependencies
1170 should be ignored.
1171 For these cases, you can instruct the build process to ignore a dependency
1172 by using a line like the following:
1173 <literallayout class='monospaced'>
1174 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
1175 </literallayout>
1176 This example ensures that the <filename>PACKAGE_ARCHS</filename> variable does not
1177 depend on the value of <filename>MACHINE</filename>, even if it does reference it.
1178 </para>
1179
1180 <para>
1181 Equally, there are cases where we need to add dependencies BitBake
1182 is not able to find.
1183 You can accomplish this by using a line like the following:
1184 <literallayout class='monospaced'>
1185 PACKAGE_ARCHS[vardeps] = "MACHINE"
1186 </literallayout>
1187 This example explicitly adds the <filename>MACHINE</filename> variable as a
1188 dependency for <filename>PACKAGE_ARCHS</filename>.
1189 </para>
1190
1191 <para>
1192 Consider a case with in-line Python, for example, where BitBake is not
1193 able to figure out dependencies.
1194 When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
1195 produces output when it discovers something for which it cannot figure out
1196 dependencies.
1197 </para>
1198
1199 <para>
1200 Thus far, this section has limited discussion to the direct inputs into a task.
1201 Information based on direct inputs is referred to as the "basehash" in the
1202 code.
1203 However, there is still the question of a task's indirect inputs - the
1204 things that were already built and present in the build directory.
1205 The checksum (or signature) for a particular task needs to add the hashes
1206 of all the tasks on which the particular task depends.
1207 Choosing which dependencies to add is a policy decision.
1208 However, the effect is to generate a master checksum that combines the basehash
1209 and the hashes of the task's dependencies.
1210 </para>
1211
1212 <para>
1213 At the code level, there are a variety of ways both the basehash and the
1214 dependent task hashes can be influenced.
1215 Within the BitBake configuration file, we can give BitBake some extra information
1216 to help it construct the basehash.
1217 The following statement effectively results in a list of global variable
1218 dependency excludes - variables never included in any checksum.
1219 This example uses variables from OpenEmbedded to help illustrate
1220 the concept:
1221 <literallayout class='monospaced'>
1222 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
1223 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
1224 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
1225 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
1226 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
1227 </literallayout>
1228 The previous example excludes the work directory, which is part of
1229 <filename>TMPDIR</filename>.
1230 </para>
1231
1232 <para>
1233 The rules for deciding which hashes of dependent tasks to include through
1234 dependency chains are more complex and are generally accomplished with a
1235 Python function.
1236 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
1237 of this and also illustrates how you can insert your own policy into the system
1238 if so desired.
1239 This file defines the two basic signature generators OpenEmbedded Core
1240 uses: "OEBasic" and "OEBasicHash".
1241 By default, there is a dummy "noop" signature handler enabled in BitBake.
1242 This means that behavior is unchanged from previous versions.
1243 <filename>OE-Core</filename> uses the "OEBasicHash" signature handler by default
1244 through this setting in the <filename>bitbake.conf</filename> file:
1245 <literallayout class='monospaced'>
1246 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
1247 </literallayout>
1248 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
1249 "OEBasic" version but adds the task hash to the stamp files.
1250 This results in any metadata change that changes the task hash, automatically
1251 causing the task to be run again.
1252 This removes the need to bump
1253 <link linkend='var-PR'><filename>PR</filename></link>
1254 values, and changes to metadata automatically ripple across the build.
1255 </para>
1256
1257 <para>
1258 It is also worth noting that the end result of these signature generators is to
1259 make some dependency and hash information available to the build.
1260 This information includes:
1261 <itemizedlist>
1262 <listitem><para><filename>BB_BASEHASH_task-&lt;taskname&gt;</filename>:
1263 The base hashes for each task in the recipe.
1264 </para></listitem>
1265 <listitem><para><filename>BB_BASEHASH_&lt;filename:taskname&gt;</filename>:
1266 The base hashes for each dependent task.
1267 </para></listitem>
1268 <listitem><para><filename>BBHASHDEPS_&lt;filename:taskname&gt;</filename>:
1269 The task dependencies for each task.
1270 </para></listitem>
1271 <listitem><para><filename>BB_TASKHASH</filename>:
1272 The hash of the currently running task.
1273 </para></listitem>
1274 </itemizedlist>
1275 </para>
1276 </section>
1277 </section>
1095</chapter> 1278</chapter>