summaryrefslogtreecommitdiffstats
path: root/documentation/poky-ref-manual
diff options
context:
space:
mode:
authorScott Rifenbark <scott.m.rifenbark@intel.com>2011-12-07 13:57:53 -0800
committerRichard Purdie <richard.purdie@linuxfoundation.org>2012-01-30 16:05:47 +0000
commitc2494d30148a04ed625e422c4fbc416031cfb178 (patch)
treec093fa7d582be3f7d79f08b40d841aea0c7bb1f8 /documentation/poky-ref-manual
parent9d87cd99521f4eaf5bc980c990639cc5a0a3038f (diff)
downloadpoky-c2494d30148a04ed625e422c4fbc416031cfb178.tar.gz
documentation/poky-ref-manual/usingpoky.xml: partial for YOCTO #1500
First draft of a re-write to the "Running a Build" section to try and satisfy YOCTO #1500. I segmented the section into three areas rather than a single area. This allowed me to create a sub-section for the sstate stuff where it could be addressed on its own. I sent the draft out to Richard and Mark H. and got feedback from RP that is going to cause further changes. Thus, I am committing this partial change. (From yocto-docs rev: f040ed6979e988968863016103aa3ad4e7365159) Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/poky-ref-manual')
-rw-r--r--documentation/poky-ref-manual/usingpoky.xml282
1 files changed, 234 insertions, 48 deletions
diff --git a/documentation/poky-ref-manual/usingpoky.xml b/documentation/poky-ref-manual/usingpoky.xml
index 7b24841ac0..eb1f9cb045 100644
--- a/documentation/poky-ref-manual/usingpoky.xml
+++ b/documentation/poky-ref-manual/usingpoky.xml
@@ -152,67 +152,253 @@
152 <title>Running a Build</title> 152 <title>Running a Build</title>
153 153
154 <para> 154 <para>
155 You can find information on how to build an image using the Yocto Project in the 155 You can find general information on how to build an image using the
156 Yocto Project in the
156 <ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html#building-image'> 157 <ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html#building-image'>
157 Building an Image</ulink> section of the 158 Building an Image</ulink> section of the
158 <ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html'> 159 <ulink url='http://www.yoctoproject.org/docs/1.1/yocto-project-qs/yocto-project-qs.html'>
159 Yocto Project Quick Start</ulink>. 160 Yocto Project Quick Start</ulink>.
160 This section provides a quick overview. 161 This section provides a summary of the build process and provides information
162 for less obvious aspects of the build process.
161 </para> 163 </para>
162 164
163 <para> 165 <section id='build-overview'>
164 The first thing you need to do is set up the Yocto Project build environment by sourcing 166 <title>Build Overview</title>
165 the environment setup script as follows:
166 <literallayout class='monospaced'>
167 $ source oe-init-build-env [build_dir];
168 </literallayout>
169 </para>
170 167
171 <para> 168 <para>
172 The <filename>build_dir</filename> is optional and specifies the directory Yocto Project 169 The first thing you need to do is set up the Yocto Project build environment by sourcing
173 uses for the build. 170 the environment setup script as follows:
174 If you do not specify a build directory it defaults to <filename>build</filename> 171 <literallayout class='monospaced'>
175 in the Yocto Project files directory structure. 172 $ source oe-init-build-env [build_dir]
176 A common practice is to use a different build directory for different targets. 173 </literallayout>
177 For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename> 174 </para>
178 target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
179 See <link linkend="structure-core-script">oe-init-build-env</link>
180 for more information on this script.
181 </para>
182 175
183 <para> 176 <para>
184 Once the Yocto Project build environment is set up, you can build a target using: 177 The <filename>build_dir</filename> is optional and specifies the directory Yocto Project
185 <literallayout class='monospaced'> 178 uses for the build.
179 If you do not specify a build directory it defaults to <filename>build</filename>
180 in your current working directory.
181 A common practice is to use a different build directory for different targets.
182 For example, <filename>~/build/x86</filename> for a <filename>qemux86</filename>
183 target, and <filename>~/build/arm</filename> for a <filename>qemuarm</filename> target.
184 See <link linkend="structure-core-script">oe-init-build-env</link>
185 for more information on this script.
186 </para>
187
188 <para>
189 Once the Yocto Project build environment is set up, you can build a target using:
190 <literallayout class='monospaced'>
186 $ bitbake &lt;target&gt; 191 $ bitbake &lt;target&gt;
187 </literallayout> 192 </literallayout>
188 </para> 193 </para>
189 194
190 <para> 195 <para>
191 The <filename>target</filename> is the name of the recipe you want to build. 196 The <filename>target</filename> is the name of the recipe you want to build.
192 Common targets are the images in <filename>meta/recipes-core/images</filename>, 197 Common targets are the images in <filename>meta/recipes-core/images</filename>,
193 <filename>/meta/recipes-sato/images</filename>, etc. all found in the Yocto Project 198 <filename>/meta/recipes-sato/images</filename>, etc. all found in the Yocto Project
194 files. 199 files.
195 Or, the target can be the name of a recipe for a specific piece of software such as 200 Or, the target can be the name of a recipe for a specific piece of software such as
196 <application>busybox</application>. 201 <application>busybox</application>.
197 For more details about the images Yocto Project supports, see the 202 For more details about the images Yocto Project supports, see the
198 <link linkend="ref-images">'Reference: Images'</link> appendix. 203 <link linkend="ref-images">'Reference: Images'</link> appendix.
199 </para> 204 </para>
205
206 <note>
207 Building an image without GNU Public License Version 3 (GPLv3) components is
208 only supported for minimal and base images.
209 See <link linkend='ref-images'>'Reference: Images'</link> for more information.
210 </note>
211 </section>
212
213 <section id='building-an-image-using-gpl-components'>
214 <title>Building an Image Using GPL Components</title>
215
216 <para>
217 When building an image using GPL components, you need to maintain your original
218 settings and not switch back and forth applying different versions of the GNU
219 Public License.
220 If you rebuild using different versions of GPL, dependency errors might occur
221 due to some components not being rebuilt.
222 </para>
223 </section>
200 224
201 <note> 225 <section id="considering-shared-state-cache">
202 Building an image without GNU Public License Version 3 (GPLv3) components is 226 <title>Considering Shared State Cache</title>
203 only supported for minimal and base images. 227
204 See <link linkend='ref-images'>'Reference: Images'</link> for more information. 228 <para>
205 </note> 229 By design, the Yocto Project builds everything from scratch unless it can determine that
206 230 a given task's inputs have not changed.
207 <note> 231 While building from scratch ensures that everything is current, it does also
208 When building an image using GPL components, you need to maintain your original 232 mean that a lot of time could be spent rebuiding things that don't necessarily need built.
209 settings and not switch back and forth applying different versions of the GNU 233 </para>
210 Public License. 234
211 If you rebuild using different versions of GPL, dependency errors might occur 235 <para>
212 due to some components not being rebuilt. 236 The Yocto Project build process uses a shared state caching scheme to avoid having to
213 </note> 237 rebuild software when it is not necessary.
238 Because the build time for a Yocto image can be significant, it is helpful to try and
239 determine what really needs built and what can be skipped given a particular project's
240 build process.
241 </para>
242
243 <para>
244 The scheme that the Yocto Project uses involves checksum generation and comparison for
245 a task's inputs.
246 The scheme also employs an area of memory called the shared state cache that is
247 pointed to by the <filename>SSTATE_DIR</filename> variable.
248 This area contains task output generated from a previous build.
249 If a given task's checksum matches the checksum of a previous build for the same
250 task, the build process uses the state of the cache rather than rerunning that
251 task.
252 </para>
253
254 <para>
255 The previous paragraph is a simplistic explanation of how the build process
256 uses checksums and shared state memory cache to avoide building tasks that
257 don't need built.
258 If you want a bit more explanation on the topic,
259 see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
260 State - What does it mean and why should I care?</ulink>" from the Yocto
261 Project discussion archives.
262 </para>
263
264 <para>
265 As with all schemes, this one has some drawbacks.
266 It is possible that you could make implicit changes that are not factored into the checksum
267 calculation, but do affect a task's output.
268 A good example is perhaps when a tool changes its output.
269 Let's say that the output of <filename>rpmdeps</filename> needed to change.
270 The result of the change should be that all the "package", "package_write_rpm",
271 and "package_deploy-rpm" sstate-cache items would become invalid.
272 But, because this is a change that is external to the code and therefore implicit,
273 the associated sstate-cache items do not become invalidated.
274 In this case, the build process would use the cache items rather than running the
275 task again.
276 Obviously, these types of implicit changes can cause problems.
277 </para>
278
279 <para>
280 To avoid these problems during the build, you need to understand the effects of any
281 change you make.
282 Note that any changes you make directly to a function automatically are factored into
283 the checksum calculation and thus, will invalidate the associated area of sstate cache.
284 You need to be aware of any implicit changes that are not obvious changes to the
285 code and could affect the output of a given task.
286 Once you are aware of such a change, you can take steps to invalidate the cache
287 and force the task to run.
288 The step to take is as simple as changing a function's comments in the source code.
289 For example, to invalidate package sstate files, change the comment statments
290 of <filename>do_package</filename> or one of the functions it calls.
291 The change is purely cosmetic, but it causes the checksum to be recalculated and
292 forces the task to be run again.
293 </para>
294
295 <note>
296 For an example of a commit that makes a cosmetic change to invalidate an sstate,
297 see this
298 <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
299 </note>
300 </section>
301
302
303<!--
304
305 <section id="considering-shared-state-cache">
306 <title>Considering Shared State Cache</title>
307
308 <para>
309 What is shared state in general.
310 Benefits?
311 How we handle things
312 (reference https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.htm),
313 which is RP's dissertation on how YP solved it.
314 We need to talk a bit about checksum generation for tasks and how the
315 sstate code uses them to figure out what needs rebuilt and what can be re-loaded
316 from the sstate cache.
317 Need to tell about cases where an implicit change can mess things up and under
318 normal situations the state in the sstate cache would be used but it shouldn't be.
319 This is the scenario described by bug 1500 - typical case.
320 Then we talk about how we can invalidate parts of the cache on a per-class basis
321 if needed.
322
323 there is a discussion at
324 https://lists.yoctoproject.org/pipermail/yocto/2011-March/001157.htm
325 that talks about sstate and how the YP team attacked and solved the problem.
326 This is probably a good place to get information from to broach the whole
327 sstate concept.
328
329 YP, by default, builds from scratch.
330 This is good but it means spending a lot of time rebuilding things that don't
331 necessarily need rebuilding.
332
333 The SSTATE_DIR variable points to the directory for the shared state cache that
334 is used during a build.
335
336 A task's inputs have a checksum or signature associated with them.
337 If the checksum changes on an input as compared to a prior build, the task must be rerun.
338 The shared state (sstate) code keeps track of what output is generated by which tasks.
339 So if a task's inputs have not changed then the output associated with the task can
340 be yanked from some place and re-used. No re-build required for that particular task.
341
342 A "run" shell script is created for each task.
343 You can create a checksum for the task based on the inputs to the task.
344 When you have this checksum, the code will look at it and compare it to the previous
345 checksum to see if the task's inputs have changed.
346 If so, the task needs to be re-run.
347
348 Python tasks have python functions that access variables.
349 Python functions will call other python functions as well.
350 The solution was to figure out the variable and function dependencies and create
351 a checksum value for the data coming into the python task.
352
353 Here is a conversation with Mark Hatle regarding bug 1500 (638 is related):
354
355(01:23:34 PM) scottrif: mark - you have a minute?
356(01:34:05 PM) Mark Hatle: sure..
357(01:34:11 PM) Mark Hatle: might be a bit slow to respond, but I'm here
358(01:34:45 PM) scottrif: Hi - I am looking at bug 1500 and trying to get a bit of better understanding. Here is the link to the bug - http://bugzilla.pokylinux.org/show_bug.cgi?id=1500
359(01:35:25 PM) scottrif: It seems that the key for the user here is to when to "Know" when to put some comments into a function to invalidate certain areas of sstate.
360(01:35:49 PM) Mark Hatle: what the issue is, if you make changes to something that is not normally calculated in the checksums for sstate, then you can get package mismatches..
361(01:35:50 PM) scottrif: This trick of "knowing"... does it need to be explained?
362(01:36:25 PM) Mark Hatle: The developer will have to know they made such a change.. Then to deal with this, they can use a patch like what is referenced to cause specific steps to be invalidated and various steps re-run..
363(01:37:01 PM) scottrif: so my question is will the developer know when they make a change like this?
364(01:37:04 PM) Mark Hatle: In this case, we change part of the back-end packaging mechanisms.. which changed internal dependency generation. The sstate code does not checksum the internal dependency generation, it assumes that is code that doesn't change behavior
365(01:37:24 PM) Mark Hatle: They should understand the ramifications of their changes — and thus know they need to do this.
366(01:37:46 PM) Mark Hatle: Examples of times you need to do this. Back end packaging changes occur — i.e. you change the format of dependency generation..
367(01:38:38 PM) scottrif: do you have any other examples?
368(01:38:39 PM) Mark Hatle: when you change a recipe itself, source code.. it is -not- necessary to do this
369(01:38:49 PM) Mark Hatle: RP might be better at examples of when to do it..
370(01:39:11 PM) scottrif: right - If I change a recipe then every thing dependent further down the line gets regenerated right?
371(01:39:17 PM) Mark Hatle: This should never be necessary when a recipe changes.. it will only be necessary when some classes or back-end (packaging frameworks) change..
372(01:39:21 PM) Mark Hatle: ya
373(01:39:33 PM) Mark Hatle: Another way to think of this is implicit dependencies..
374(01:40:01 PM) Mark Hatle: I change RPM.. If you build something that produces an RPM package.. the assumption is the RPM package won't change, even if the RPM binary changes..
375(01:40:10 PM) Mark Hatle: If the format of the package changes.. you would need to do this
376(01:40:53 PM) Mark Hatle: RP can probably give you an idea of the various implicit dependencies, and which ones this type of change is needed for
377(01:41:26 PM) scottrif: okay. I am struggling a bit with how to word it. what I will do is write something up and send it out to you and RP for a look
378(01:41:47 PM) Mark Hatle: ya, I understand.. it's an odd set of situations that can cause this — but we definitely need to document it
379(01:42:01 PM) scottrif: I just want the information to help the user understand the conditions when they will want to invalidate parts of the sstate
380(01:42:18 PM) scottrif: I will likely use the RPM example as the case to illustrate it
381(01:42:26 PM) scottrif: as it seems pretty straight forward
382(01:42:28 PM) Mark Hatle: yup. Key thing is it's only needed on implicit dependencies.. Normal case is back end packaging format changes..
383(01:42:31 PM) Mark Hatle: yup
384(01:42:47 PM) scottrif: ok - thanks Mark
385
386 Here is what RP wants to address 1500:
387
388If its desired to change the checksum of a given subset of tasks, maybe
389due to a change which isn't directly visible in the code itself (e.g. a
390tool changed its output) its possible to do this by changing a function
391comments since the sstate checksums include the body of functions. To
392invalidate package sstate files for example, do_package or one of the
393functions it calls can be changed, even if its just a cosmetic change to
394the commends.
395http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54
396is an example of a commit which does this.
397
398-->
214</section> 399</section>
215 400
401
216<section id='usingpoky-install'> 402<section id='usingpoky-install'>
217 <title>Installing and Using the Result</title> 403 <title>Installing and Using the Result</title>
218 404