summaryrefslogtreecommitdiffstats
path: root/documentation/getting-started/eclipse/html/getting-started/overview-checksums.html
diff options
context:
space:
mode:
Diffstat (limited to 'documentation/getting-started/eclipse/html/getting-started/overview-checksums.html')
-rw-r--r--documentation/getting-started/eclipse/html/getting-started/overview-checksums.html209
1 files changed, 209 insertions, 0 deletions
diff --git a/documentation/getting-started/eclipse/html/getting-started/overview-checksums.html b/documentation/getting-started/eclipse/html/getting-started/overview-checksums.html
new file mode 100644
index 0000000000..09ad110f71
--- /dev/null
+++ b/documentation/getting-started/eclipse/html/getting-started/overview-checksums.html
@@ -0,0 +1,209 @@
1<html>
2<head>
3<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
4<title>3.3.2. Checksums (Signatures)</title>
5<link rel="stylesheet" type="text/css" href="../book.css">
6<meta name="generator" content="DocBook XSL Stylesheets V1.76.1">
7<link rel="home" href="index.html" title="Getting Started With Yocto Project">
8<link rel="up" href="shared-state-cache.html" title="3.3. Shared State Cache">
9<link rel="prev" href="overall-architecture.html" title="3.3.1. Overall Architecture">
10<link rel="next" href="shared-state.html" title="3.3.3. Shared State">
11</head>
12<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="section" title="3.3.2. Checksums (Signatures)">
13<div class="titlepage"><div><div><h3 class="title">
14<a name="overview-checksums"></a>3.3.2. Checksums (Signatures)</h3></div></div></div>
15<p>
16 The shared state code uses a checksum, which is a unique
17 signature of a task's inputs, to determine if a task needs to
18 be run again.
19 Because it is a change in a task's inputs that triggers a
20 rerun, the process needs to detect all the inputs to a given
21 task.
22 For shell tasks, this turns out to be fairly easy because
23 the build process generates a "run" shell script for each task
24 and it is possible to create a checksum that gives you a good
25 idea of when the task's data changes.
26 </p>
27<p>
28 To complicate the problem, there are things that should not be
29 included in the checksum.
30 First, there is the actual specific build path of a given
31 task - the
32 <a class="link" href="../ref-manual/var-WORKDIR.html" target="_self"><code class="filename">WORKDIR</code></a>.
33 It does not matter if the work directory changes because it
34 should not affect the output for target packages.
35 Also, the build process has the objective of making native
36 or cross packages relocatable.
37 </p>
38<div class="note" title="Note" style="margin-left: 0.5in; margin-right: 0.5in;">
39<h3 class="title">Note</h3>
40 Both native and cross packages run on the build host.
41 However, cross packages generate output for the target
42 architecture.
43 </div>
44<p>
45 The checksum therefore needs to exclude
46 <code class="filename">WORKDIR</code>.
47 The simplistic approach for excluding the work directory is to
48 set <code class="filename">WORKDIR</code> to some fixed value and
49 create the checksum for the "run" script.
50 </p>
51<p>
52 Another problem results from the "run" scripts containing
53 functions that might or might not get called.
54 The incremental build solution contains code that figures out
55 dependencies between shell functions.
56 This code is used to prune the "run" scripts down to the
57 minimum set, thereby alleviating this problem and making the
58 "run" scripts much more readable as a bonus.
59 </p>
60<p>
61 So far we have solutions for shell scripts.
62 What about Python tasks?
63 The same approach applies even though these tasks are more
64 difficult.
65 The process needs to figure out what variables a Python
66 function accesses and what functions it calls.
67 Again, the incremental build solution contains code that first
68 figures out the variable and function dependencies, and then
69 creates a checksum for the data used as the input to the task.
70 </p>
71<p>
72 Like the <code class="filename">WORKDIR</code> case, situations exist
73 where dependencies should be ignored.
74 For these cases, you can instruct the build process to
75 ignore a dependency by using a line like the following:
76 </p>
77<pre class="literallayout">
78 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
79 </pre>
80<p>
81 This example ensures that the
82 <a class="link" href="../ref-manual/var-PACKAGE_ARCHS.html" target="_self"><code class="filename">PACKAGE_ARCHS</code></a>
83 variable does not depend on the value of
84 <a class="link" href="../ref-manual/var-MACHINE.html" target="_self"><code class="filename">MACHINE</code></a>,
85 even if it does reference it.
86 </p>
87<p>
88 Equally, there are cases where we need to add dependencies
89 BitBake is not able to find.
90 You can accomplish this by using a line like the following:
91 </p>
92<pre class="literallayout">
93 PACKAGE_ARCHS[vardeps] = "MACHINE"
94 </pre>
95<p>
96 This example explicitly adds the <code class="filename">MACHINE</code>
97 variable as a dependency for
98 <code class="filename">PACKAGE_ARCHS</code>.
99 </p>
100<p>
101 Consider a case with in-line Python, for example, where
102 BitBake is not able to figure out dependencies.
103 When running in debug mode (i.e. using
104 <code class="filename">-DDD</code>), BitBake produces output when it
105 discovers something for which it cannot figure out dependencies.
106 The Yocto Project team has currently not managed to cover
107 those dependencies in detail and is aware of the need to fix
108 this situation.
109 </p>
110<p>
111 Thus far, this section has limited discussion to the direct
112 inputs into a task.
113 Information based on direct inputs is referred to as the
114 "basehash" in the code.
115 However, there is still the question of a task's indirect
116 inputs - the things that were already built and present in the
117 <a class="link" href="../ref-manual/build-directory.html" target="_self">Build Directory</a>.
118 The checksum (or signature) for a particular task needs to add
119 the hashes of all the tasks on which the particular task
120 depends.
121 Choosing which dependencies to add is a policy decision.
122 However, the effect is to generate a master checksum that
123 combines the basehash and the hashes of the task's
124 dependencies.
125 </p>
126<p>
127 At the code level, there are a variety of ways both the
128 basehash and the dependent task hashes can be influenced.
129 Within the BitBake configuration file, we can give BitBake
130 some extra information to help it construct the basehash.
131 The following statement effectively results in a list of
132 global variable dependency excludes - variables never
133 included in any checksum:
134 </p>
135<pre class="literallayout">
136 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
137 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
138 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
139 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
140 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
141 </pre>
142<p>
143 The previous example excludes
144 <a class="link" href="../ref-manual/var-WORKDIR.html" target="_self"><code class="filename">WORKDIR</code></a>
145 since that variable is actually constructed as a path within
146 <a class="link" href="../ref-manual/var-TMPDIR.html" target="_self"><code class="filename">TMPDIR</code></a>,
147 which is on the whitelist.
148 </p>
149<p>
150 The rules for deciding which hashes of dependent tasks to
151 include through dependency chains are more complex and are
152 generally accomplished with a Python function.
153 The code in <code class="filename">meta/lib/oe/sstatesig.py</code> shows
154 two examples of this and also illustrates how you can insert
155 your own policy into the system if so desired.
156 This file defines the two basic signature generators
157 <a class="link" href="../ref-manual/oe-core.html" target="_self">OE-Core</a>
158 uses: "OEBasic" and "OEBasicHash".
159 By default, there is a dummy "noop" signature handler enabled
160 in BitBake.
161 This means that behavior is unchanged from previous versions.
162 OE-Core uses the "OEBasicHash" signature handler by default
163 through this setting in the <code class="filename">bitbake.conf</code>
164 file:
165 </p>
166<pre class="literallayout">
167 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
168 </pre>
169<p>
170 The "OEBasicHash" <code class="filename">BB_SIGNATURE_HANDLER</code>
171 is the same as the "OEBasic" version but adds the task hash to
172 the stamp files.
173 This results in any
174 <a class="link" href="../ref-manual/metadata.html" target="_self">Metadata</a>
175 change that changes the task hash, automatically
176 causing the task to be run again.
177 This removes the need to bump
178 <a class="link" href="../ref-manual/var-PR.html" target="_self"><code class="filename">PR</code></a>
179 values, and changes to Metadata automatically ripple across
180 the build.
181 </p>
182<p>
183 It is also worth noting that the end result of these
184 signature generators is to make some dependency and hash
185 information available to the build.
186 This information includes:
187 </p>
188<div class="itemizedlist"><ul class="itemizedlist" type="disc">
189<li class="listitem"><p>
190 <code class="filename">BB_BASEHASH_task-</code><em class="replaceable"><code>taskname</code></em>:
191 The base hashes for each task in the recipe.
192 </p></li>
193<li class="listitem"><p>
194 <code class="filename">BB_BASEHASH_</code><em class="replaceable"><code>filename</code></em><code class="filename">:</code><em class="replaceable"><code>taskname</code></em>:
195 The base hashes for each dependent task.
196 </p></li>
197<li class="listitem"><p>
198 <code class="filename">BBHASHDEPS_</code><em class="replaceable"><code>filename</code></em><code class="filename">:</code><em class="replaceable"><code>taskname</code></em>:
199 The task dependencies for each task.
200 </p></li>
201<li class="listitem"><p>
202 <code class="filename">BB_TASKHASH</code>:
203 The hash of the currently running task.
204 </p></li>
205</ul></div>
206<p>
207 </p>
208</div></body>
209</html>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-04-30 21:40:26 +0000