diff options
Diffstat (limited to 'documentation/ref-manual/eclipse/html/poky-ref-manual/checksums.html')
-rw-r--r-- | documentation/ref-manual/eclipse/html/poky-ref-manual/checksums.html | 164 |
1 files changed, 164 insertions, 0 deletions
diff --git a/documentation/ref-manual/eclipse/html/poky-ref-manual/checksums.html b/documentation/ref-manual/eclipse/html/poky-ref-manual/checksums.html new file mode 100644 index 0000000000..5dccce93b9 --- /dev/null +++ b/documentation/ref-manual/eclipse/html/poky-ref-manual/checksums.html | |||
@@ -0,0 +1,164 @@ | |||
1 | <html> | ||
2 | <head> | ||
3 | <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> | ||
4 | <title>3.2.2. Checksums (Signatures)</title> | ||
5 | <link rel="stylesheet" type="text/css" href="../book.css"> | ||
6 | <meta name="generator" content="DocBook XSL Stylesheets V1.76.1"> | ||
7 | <link rel="home" href="index.html" title="The Yocto Project Reference Manual"> | ||
8 | <link rel="up" href="shared-state-cache.html" title="3.2. Shared State Cache"> | ||
9 | <link rel="prev" href="overall-architecture.html" title="3.2.1. Overall Architecture"> | ||
10 | <link rel="next" href="shared-state.html" title="3.2.3. Shared State"> | ||
11 | </head> | ||
12 | <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="section" title="3.2.2. Checksums (Signatures)"> | ||
13 | <div class="titlepage"><div><div><h3 class="title"> | ||
14 | <a name="checksums"></a>3.2.2. Checksums (Signatures)</h3></div></div></div> | ||
15 | <p> | ||
16 | The shared state code uses a checksum, which is a unique signature of a task's | ||
17 | inputs, to determine if a task needs to be run again. | ||
18 | Because it is a change in a task's inputs that triggers a rerun, the process | ||
19 | needs to detect all the inputs to a given task. | ||
20 | For shell tasks, this turns out to be fairly easy because | ||
21 | the build process generates a "run" shell script for each task and | ||
22 | it is possible to create a checksum that gives you a good idea of when | ||
23 | the task's data changes. | ||
24 | </p> | ||
25 | <p> | ||
26 | To complicate the problem, there are things that should not be included in | ||
27 | the checksum. | ||
28 | First, there is the actual specific build path of a given task - | ||
29 | the <code class="filename">WORKDIR</code>. | ||
30 | It does not matter if the working directory changes because it should not | ||
31 | affect the output for target packages. | ||
32 | Also, the build process has the objective of making native/cross packages relocatable. | ||
33 | The checksum therefore needs to exclude <code class="filename">WORKDIR</code>. | ||
34 | The simplistic approach for excluding the working directory is to set | ||
35 | <code class="filename">WORKDIR</code> to some fixed value and create the checksum | ||
36 | for the "run" script. | ||
37 | </p> | ||
38 | <p> | ||
39 | Another problem results from the "run" scripts containing functions that | ||
40 | might or might not get called. | ||
41 | The incremental build solution contains code that figures out dependencies | ||
42 | between shell functions. | ||
43 | This code is used to prune the "run" scripts down to the minimum set, | ||
44 | thereby alleviating this problem and making the "run" scripts much more | ||
45 | readable as a bonus. | ||
46 | </p> | ||
47 | <p> | ||
48 | So far we have solutions for shell scripts. | ||
49 | What about python tasks? | ||
50 | The same approach applies even though these tasks are more difficult. | ||
51 | The process needs to figure out what variables a python function accesses | ||
52 | and what functions it calls. | ||
53 | Again, the incremental build solution contains code that first figures out | ||
54 | the variable and function dependencies, and then creates a checksum for the data | ||
55 | used as the input to the task. | ||
56 | </p> | ||
57 | <p> | ||
58 | Like the <code class="filename">WORKDIR</code> case, situations exist where dependencies | ||
59 | should be ignored. | ||
60 | For these cases, you can instruct the build process to ignore a dependency | ||
61 | by using a line like the following: | ||
62 | </p> | ||
63 | <pre class="literallayout"> | ||
64 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" | ||
65 | </pre> | ||
66 | <p> | ||
67 | This example ensures that the <code class="filename">PACKAGE_ARCHS</code> variable does not | ||
68 | depend on the value of <code class="filename">MACHINE</code>, even if it does reference it. | ||
69 | </p> | ||
70 | <p> | ||
71 | Equally, there are cases where we need to add dependencies BitBake is not able to find. | ||
72 | You can accomplish this by using a line like the following: | ||
73 | </p> | ||
74 | <pre class="literallayout"> | ||
75 | PACKAGE_ARCHS[vardeps] = "MACHINE" | ||
76 | </pre> | ||
77 | <p> | ||
78 | This example explicitly adds the <code class="filename">MACHINE</code> variable as a | ||
79 | dependency for <code class="filename">PACKAGE_ARCHS</code>. | ||
80 | </p> | ||
81 | <p> | ||
82 | Consider a case with inline python, for example, where BitBake is not | ||
83 | able to figure out dependencies. | ||
84 | When running in debug mode (i.e. using <code class="filename">-DDD</code>), BitBake | ||
85 | produces output when it discovers something for which it cannot figure out | ||
86 | dependencies. | ||
87 | The Yocto Project team has currently not managed to cover those dependencies | ||
88 | in detail and is aware of the need to fix this situation. | ||
89 | </p> | ||
90 | <p> | ||
91 | Thus far, this section has limited discussion to the direct inputs into a task. | ||
92 | Information based on direct inputs is referred to as the "basehash" in the | ||
93 | code. | ||
94 | However, there is still the question of a task's indirect inputs - the | ||
95 | things that were already built and present in the Build Directory. | ||
96 | The checksum (or signature) for a particular task needs to add the hashes | ||
97 | of all the tasks on which the particular task depends. | ||
98 | Choosing which dependencies to add is a policy decision. | ||
99 | However, the effect is to generate a master checksum that combines the basehash | ||
100 | and the hashes of the task's dependencies. | ||
101 | </p> | ||
102 | <p> | ||
103 | At the code level, there are a variety of ways both the basehash and the | ||
104 | dependent task hashes can be influenced. | ||
105 | Within the BitBake configuration file, we can give BitBake some extra information | ||
106 | to help it construct the basehash. | ||
107 | The following statements effectively result in a list of global variable | ||
108 | dependency excludes - variables never included in any checksum: | ||
109 | </p> | ||
110 | <pre class="literallayout"> | ||
111 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH" | ||
112 | BB_HASHBASE_WHITELIST += "DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS" | ||
113 | BB_HASHBASE_WHITELIST += "FILE_DIRNAME HOME LOGNAME SHELL TERM USER" | ||
114 | BB_HASHBASE_WHITELIST += "FILESPATH USERNAME STAGING_DIR_HOST STAGING_DIR_TARGET" | ||
115 | </pre> | ||
116 | <p> | ||
117 | The previous example actually excludes | ||
118 | <a class="link" href="ref-variables-glos.html#var-WORKDIR" title="WORKDIR"><code class="filename">WORKDIR</code></a> | ||
119 | since it is actually constructed as a path within | ||
120 | <a class="link" href="ref-variables-glos.html#var-TMPDIR" title="TMPDIR"><code class="filename">TMPDIR</code></a>, which is on | ||
121 | the whitelist. | ||
122 | </p> | ||
123 | <p> | ||
124 | The rules for deciding which hashes of dependent tasks to include through | ||
125 | dependency chains are more complex and are generally accomplished with a | ||
126 | python function. | ||
127 | The code in <code class="filename">meta/lib/oe/sstatesig.py</code> shows two examples | ||
128 | of this and also illustrates how you can insert your own policy into the system | ||
129 | if so desired. | ||
130 | This file defines the two basic signature generators <code class="filename">OE-Core</code> | ||
131 | uses: "OEBasic" and "OEBasicHash". | ||
132 | By default, there is a dummy "noop" signature handler enabled in BitBake. | ||
133 | This means that behavior is unchanged from previous versions. | ||
134 | <code class="filename">OE-Core</code> uses the "OEBasic" signature handler by default | ||
135 | through this setting in the <code class="filename">bitbake.conf</code> file: | ||
136 | </p> | ||
137 | <pre class="literallayout"> | ||
138 | BB_SIGNATURE_HANDLER ?= "OEBasic" | ||
139 | </pre> | ||
140 | <p> | ||
141 | The "OEBasicHash" <code class="filename">BB_SIGNATURE_HANDLER</code> is the same as the | ||
142 | "OEBasic" version but adds the task hash to the stamp files. | ||
143 | This results in any metadata change that changes the task hash, automatically | ||
144 | causing the task to be run again. | ||
145 | This removes the need to bump <a class="link" href="ref-variables-glos.html#var-PR" title="PR"><code class="filename">PR</code></a> | ||
146 | values and changes to metadata automatically ripple across the build. | ||
147 | Currently, this behavior is not the default behavior for <code class="filename">OE-Core</code> | ||
148 | but is the default in <code class="filename">poky</code>. | ||
149 | </p> | ||
150 | <p> | ||
151 | It is also worth noting that the end result of these signature generators is to | ||
152 | make some dependency and hash information available to the build. | ||
153 | This information includes: | ||
154 | </p> | ||
155 | <pre class="literallayout"> | ||
156 | BB_BASEHASH_task-<taskname> - the base hashes for each task in the recipe | ||
157 | BB_BASEHASH_<filename:taskname> - the base hashes for each dependent task | ||
158 | BBHASHDEPS_<filename:taskname> - The task dependencies for each task | ||
159 | BB_TASKHASH - the hash of the currently running task | ||
160 | </pre> | ||
161 | <p> | ||
162 | </p> | ||
163 | </div></body> | ||
164 | </html> | ||