summaryrefslogtreecommitdiffstats
path: root/documentation/overview-manual/overview-development-environment.xml
diff options
context:
space:
mode:
authorScott Rifenbark <srifenbark@gmail.com>2018-01-05 11:00:59 -0800
committerRichard Purdie <richard.purdie@linuxfoundation.org>2018-02-14 15:25:27 +0000
commitb797db0db4b472edbee96cd891c9523818d13346 (patch)
tree219fa940ca3407006357c3fc90acb5c252906235 /documentation/overview-manual/overview-development-environment.xml
parent64c37b5400b489a3b52bbffac59689b392e0e1f0 (diff)
downloadpoky-b797db0db4b472edbee96cd891c9523818d13346.tar.gz
documentation: Created new YP Overview Manual
Fixes [YOCTO #12370] Added a new chapter to the newly created Yocto Project Overview Manual. This chapter originated from the YP Reference Manual and was the old chapter 3, which talked a lot about various concepts. This information is better suited for the new overview manual. The change involved moving the entire chapter and renaming it. This move of the content affected many, many external references and links into the old area. Consequently, I had to recast all these links from the different manual. Changes also included fixing the mega-manual.xml file so that it would include the new overview manual as part of it. Many figures had to be relocated as part of the move as well. This meant deleting them from the ref-manual/figures folder and putting them into the overview-manual/figures folder. (From yocto-docs rev: 87b81358f2bbd02b4a0d966d86c4d7b006d4d78f) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation/overview-manual/overview-development-environment.xml')
-rw-r--r--documentation/overview-manual/overview-development-environment.xml2773
1 files changed, 2773 insertions, 0 deletions
diff --git a/documentation/overview-manual/overview-development-environment.xml b/documentation/overview-manual/overview-development-environment.xml
new file mode 100644
index 0000000000..32fe1816ab
--- /dev/null
+++ b/documentation/overview-manual/overview-development-environment.xml
@@ -0,0 +1,2773 @@
1<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
2"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"
3[<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] >
4
5<chapter id='overview-development-environment'>
6<title>The Yocto Project Development Environment</title>
7
8<para>
9 This chapter takes a look at the Yocto Project development
10 environment and also provides a detailed look at what goes on during
11 development in that environment.
12 The chapter provides Yocto Project Development environment concepts that
13 help you understand how work is accomplished in an open source environment,
14 which is very different as compared to work accomplished in a closed,
15 proprietary environment.
16</para>
17
18<para>
19 Specifically, this chapter addresses open source philosophy, workflows,
20 Git, source repositories, licensing, recipe syntax, and development
21 syntax.
22</para>
23
24<section id='open-source-philosophy'>
25 <title>Open Source Philosophy</title>
26
27 <para>
28 Open source philosophy is characterized by software development
29 directed by peer production and collaboration through an active
30 community of developers.
31 Contrast this to the more standard centralized development models
32 used by commercial software companies where a finite set of developers
33 produces a product for sale using a defined set of procedures that
34 ultimately result in an end product whose architecture and source
35 material are closed to the public.
36 </para>
37
38 <para>
39 Open source projects conceptually have differing concurrent agendas,
40 approaches, and production.
41 These facets of the development process can come from anyone in the
42 public (community) that has a stake in the software project.
43 The open source environment contains new copyright, licensing, domain,
44 and consumer issues that differ from the more traditional development
45 environment.
46 In an open source environment, the end product, source material,
47 and documentation are all available to the public at no cost.
48 </para>
49
50 <para>
51 A benchmark example of an open source project is the Linux kernel,
52 which was initially conceived and created by Finnish computer science
53 student Linus Torvalds in 1991.
54 Conversely, a good example of a non-open source project is the
55 <trademark class='registered'>Windows</trademark> family of operating
56 systems developed by
57 <trademark class='registered'>Microsoft</trademark> Corporation.
58 </para>
59
60 <para>
61 Wikipedia has a good historical description of the Open Source
62 Philosophy
63 <ulink url='http://en.wikipedia.org/wiki/Open_source'>here</ulink>.
64 You can also find helpful information on how to participate in the
65 Linux Community
66 <ulink url='http://ldn.linuxfoundation.org/book/how-participate-linux-community'>here</ulink>.
67 </para>
68</section>
69
70<section id='workflows'>
71 <title>Workflows</title>
72
73 <para>
74 This section provides workflow concepts using the Yocto Project and
75 Git.
76 In particular, the information covers basic practices that describe
77 roles and actions in a collaborative development environment.
78 <note>
79 If you are familiar with this type of development environment, you
80 might not want to read this section.
81 </note>
82 </para>
83
84 <para>
85 The Yocto Project files are maintained using Git in "master"
86 branches whose Git histories track every change and whose structures
87 provides branches for all diverging functionality.
88 Although there is no need to use Git, many open source projects do so.
89 <para>
90
91 </para>
92 For the Yocto Project, a key individual called the "maintainer" is
93 responsible for the "master" branch of a given Git repository.
94 The "master" branch is the “upstream” repository from which final or
95 most recent builds of the project occur.
96 The maintainer is responsible for accepting changes from other
97 developers and for organizing the underlying branch structure to
98 reflect release strategies and so forth.
99 <note>For information on finding out who is responsible for (maintains)
100 a particular area of code, see the
101 "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>"
102 section of the Yocto Project Development Tasks Manual.
103 </note>
104 </para>
105
106 <para>
107 The Yocto Project <filename>poky</filename> Git repository also has an
108 upstream contribution Git repository named
109 <filename>poky-contrib</filename>.
110 You can see all the branches in this repository using the web interface
111 of the
112 <ulink url='&YOCTO_GIT_URL;'>Source Repositories</ulink> organized
113 within the "Poky Support" area.
114 These branches temporarily hold changes to the project that have been
115 submitted or committed by the Yocto Project development team and by
116 community members who contribute to the project.
117 The maintainer determines if the changes are qualified to be moved
118 from the "contrib" branches into the "master" branch of the Git
119 repository.
120 </para>
121
122 <para>
123 Developers (including contributing community members) create and
124 maintain cloned repositories of the upstream "master" branch.
125 The cloned repositories are local to their development platforms and
126 are used to develop changes.
127 When a developer is satisfied with a particular feature or change,
128 they "push" the changes to the appropriate "contrib" repository.
129 </para>
130
131 <para>
132 Developers are responsible for keeping their local repository
133 up-to-date with "master".
134 They are also responsible for straightening out any conflicts that
135 might arise within files that are being worked on simultaneously by
136 more than one person.
137 All this work is done locally on the developer’s machine before
138 anything is pushed to a "contrib" area and examined at the maintainer’s
139 level.
140 </para>
141
142 <para>
143 A somewhat formal method exists by which developers commit changes
144 and push them into the "contrib" area and subsequently request that
145 the maintainer include them into "master".
146 This process is called “submitting a patch” or "submitting a change."
147 For information on submitting patches and changes, see the
148 "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>"
149 section in the Yocto Project Development Tasks Manual.
150 </para>
151
152 <para>
153 To summarize the development workflow: a single point of entry
154 exists for changes into the project’s "master" branch of the
155 Git repository, which is controlled by the project’s maintainer.
156 And, a set of developers exist who independently develop, test, and
157 submit changes to "contrib" areas for the maintainer to examine.
158 The maintainer then chooses which changes are going to become a
159 permanent part of the project.
160 </para>
161
162 <para>
163 <imagedata fileref="figures/git-workflow.png" width="6in" depth="3in" align="left" scalefit="1" />
164 </para>
165
166 <para>
167 While each development environment is unique, there are some best
168 practices or methods that help development run smoothly.
169 The following list describes some of these practices.
170 For more information about Git workflows, see the workflow topics in
171 the
172 <ulink url='http://book.git-scm.com'>Git Community Book</ulink>.
173 <itemizedlist>
174 <listitem><para>
175 <emphasis>Make Small Changes:</emphasis>
176 It is best to keep the changes you commit small as compared to
177 bundling many disparate changes into a single commit.
178 This practice not only keeps things manageable but also allows
179 the maintainer to more easily include or refuse changes.</para>
180
181 <para>It is also good practice to leave the repository in a
182 state that allows you to still successfully build your project.
183 In other words, do not commit half of a feature,
184 then add the other half as a separate, later commit.
185 Each commit should take you from one buildable project state
186 to another buildable state.
187 </para></listitem>
188 <listitem><para>
189 <emphasis>Use Branches Liberally:</emphasis>
190 It is very easy to create, use, and delete local branches in
191 your working Git repository.
192 You can name these branches anything you like.
193 It is helpful to give them names associated with the particular
194 feature or change on which you are working.
195 Once you are done with a feature or change and have merged it
196 into your local master branch, simply discard the temporary
197 branch.
198 </para></listitem>
199 <listitem><para>
200 <emphasis>Merge Changes:</emphasis>
201 The <filename>git merge</filename> command allows you to take
202 the changes from one branch and fold them into another branch.
203 This process is especially helpful when more than a single
204 developer might be working on different parts of the same
205 feature.
206 Merging changes also automatically identifies any collisions
207 or "conflicts" that might happen as a result of the same lines
208 of code being altered by two different developers.
209 </para></listitem>
210 <listitem><para>
211 <emphasis>Manage Branches:</emphasis>
212 Because branches are easy to use, you should use a system
213 where branches indicate varying levels of code readiness.
214 For example, you can have a "work" branch to develop in, a
215 "test" branch where the code or change is tested, a "stage"
216 branch where changes are ready to be committed, and so forth.
217 As your project develops, you can merge code across the
218 branches to reflect ever-increasing stable states of the
219 development.
220 </para></listitem>
221 <listitem><para>
222 <emphasis>Use Push and Pull:</emphasis>
223 The push-pull workflow is based on the concept of developers
224 "pushing" local commits to a remote repository, which is
225 usually a contribution repository.
226 This workflow is also based on developers "pulling" known
227 states of the project down into their local development
228 repositories.
229 The workflow easily allows you to pull changes submitted by
230 other developers from the upstream repository into your
231 work area ensuring that you have the most recent software
232 on which to develop.
233 The Yocto Project has two scripts named
234 <filename>create-pull-request</filename> and
235 <filename>send-pull-request</filename> that ship with the
236 release to facilitate this workflow.
237 You can find these scripts in the <filename>scripts</filename>
238 folder of the
239 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
240 For information on how to use these scripts, see the
241 "<ulink url='&YOCTO_DOCS_DEV_URL;#pushing-a-change-upstream'>Using Scripts to Push a Change Upstream and Request a Pull</ulink>"
242 section in the Yocto Project Development Tasks Manual.
243 </para></listitem>
244 <listitem><para>
245 <emphasis>Patch Workflow:</emphasis>
246 This workflow allows you to notify the maintainer through an
247 email that you have a change (or patch) you would like
248 considered for the "master" branch of the Git repository.
249 To send this type of change, you format the patch and then
250 send the email using the Git commands
251 <filename>git format-patch</filename> and
252 <filename>git send-email</filename>.
253 For information on how to use these scripts, see the
254 "<ulink url='&YOCTO_DOCS_DEV_URL;#how-to-submit-a-change'>Submitting a Change to the Yocto Project</ulink>"
255 section in the Yocto Project Development Tasks Manual.
256 </para></listitem>
257 </itemizedlist>
258 </para>
259</section>
260
261<section id='git'>
262 <title>Git</title>
263
264 <para>
265 The Yocto Project makes extensive use of Git, which is a
266 free, open source distributed version control system.
267 Git supports distributed development, non-linear development,
268 and can handle large projects.
269 It is best that you have some fundamental understanding
270 of how Git tracks projects and how to work with Git if
271 you are going to use the Yocto Project for development.
272 This section provides a quick overview of how Git works and
273 provides you with a summary of some essential Git commands.
274 <note><title>Notes</title>
275 <itemizedlist>
276 <listitem><para>
277 For more information on Git, see
278 <ulink url='http://git-scm.com/documentation'></ulink>.
279 </para></listitem>
280 <listitem><para>
281 If you need to download Git, it is recommended that you add
282 Git to your system through your distribution's "software
283 store" (e.g. for Ubuntu, use the Ubuntu Software feature).
284 For the Git download page, see
285 <ulink url='http://git-scm.com/download'></ulink>.
286 </para></listitem>
287 <listitem><para>
288 For examples beyond the limited few in this section on how
289 to use Git with the Yocto Project, see the
290 "<ulink url='&YOCTO_DOCS_DEV_URL;#working-with-yocto-project-source-files'>Working With Yocto Project Source Files</ulink>"
291 section in the Yocto Project Development Tasks Manual.
292 </para></listitem>
293 </itemizedlist>
294 </note>
295 </para>
296
297 <section id='repositories-tags-and-branches'>
298 <title>Repositories, Tags, and Branches</title>
299
300 <para>
301 As mentioned briefly in the previous section and also in the
302 "<link linkend='workflows'>Workflows</link>" section,
303 the Yocto Project maintains source repositories at
304 <ulink url='&YOCTO_GIT_URL;/cgit.cgi'></ulink>.
305 If you look at this web-interface of the repositories, each item
306 is a separate Git repository.
307 </para>
308
309 <para>
310 Git repositories use branching techniques that track content
311 change (not files) within a project (e.g. a new feature or updated
312 documentation).
313 Creating a tree-like structure based on project divergence allows
314 for excellent historical information over the life of a project.
315 This methodology also allows for an environment from which you can
316 do lots of local experimentation on projects as you develop
317 changes or new features.
318 </para>
319
320 <para>
321 A Git repository represents all development efforts for a given
322 project.
323 For example, the Git repository <filename>poky</filename> contains
324 all changes and developments for Poky over the course of its
325 entire life.
326 That means that all changes that make up all releases are captured.
327 The repository maintains a complete history of changes.
328 </para>
329
330 <para>
331 You can create a local copy of any repository by "cloning" it
332 with the <filename>git clone</filename> command.
333 When you clone a Git repository, you end up with an identical
334 copy of the repository on your development system.
335 Once you have a local copy of a repository, you can take steps to
336 develop locally.
337 For examples on how to clone Git repositories, see the
338 "<ulink url='&YOCTO_DOCS_DEV_URL;#working-with-yocto-project-source-files'>Working With Yocto Project Source Files</ulink>"
339 section in the Yocto Project Development Tasks Manual.
340 </para>
341
342 <para>
343 It is important to understand that Git tracks content change and
344 not files.
345 Git uses "branches" to organize different development efforts.
346 For example, the <filename>poky</filename> repository has
347 several branches that include the current "&DISTRO_NAME_NO_CAP;"
348 branch, the "master" branch, and many branches for past
349 Yocto Project releases.
350 You can see all the branches by going to
351 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and
352 clicking on the
353 <filename><ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/refs/heads'>[...]</ulink></filename>
354 link beneath the "Branch" heading.
355 </para>
356
357 <para>
358 Each of these branches represents a specific area of development.
359 The "master" branch represents the current or most recent
360 development.
361 All other branches represent offshoots of the "master" branch.
362 </para>
363
364 <para>
365 When you create a local copy of a Git repository, the copy has
366 the same set of branches as the original.
367 This means you can use Git to create a local working area
368 (also called a branch) that tracks a specific development branch
369 from the upstream source Git repository.
370 in other words, you can define your local Git environment to
371 work on any development branch in the repository.
372 To help illustrate, consider the following example Git commands:
373 <literallayout class='monospaced'>
374 $ cd ~
375 $ git clone git://git.yoctoproject.org/poky
376 $ cd poky
377 $ git checkout -b &DISTRO_NAME_NO_CAP; origin/&DISTRO_NAME_NO_CAP;
378 </literallayout>
379 In the previous example after moving to the home directory, the
380 <filename>git clone</filename> command creates a
381 local copy of the upstream <filename>poky</filename> Git repository.
382 By default, Git checks out the "master" branch for your work.
383 After changing the working directory to the new local repository
384 (i.e. <filename>poky</filename>), the
385 <filename>git checkout</filename> command creates
386 and checks out a local branch named "&DISTRO_NAME_NO_CAP;", which
387 tracks the upstream "origin/&DISTRO_NAME_NO_CAP;" branch.
388 Changes you make while in this branch would ultimately affect
389 the upstream "&DISTRO_NAME_NO_CAP;" branch of the
390 <filename>poky</filename> repository.
391 </para>
392
393 <para>
394 It is important to understand that when you create and checkout a
395 local working branch based on a branch name,
396 your local environment matches the "tip" of that particular
397 development branch at the time you created your local branch,
398 which could be different from the files in the "master" branch
399 of the upstream repository.
400 In other words, creating and checking out a local branch based on
401 the "&DISTRO_NAME_NO_CAP;" branch name is not the same as
402 cloning and checking out the "master" branch if the repository.
403 Keep reading to see how you create a local snapshot of a Yocto
404 Project Release.
405 </para>
406
407 <para>
408 Git uses "tags" to mark specific changes in a repository.
409 Typically, a tag is used to mark a special point such as the final
410 change before a project is released.
411 You can see the tags used with the <filename>poky</filename> Git
412 repository by going to
413 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/'></ulink> and
414 clicking on the
415 <filename><ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/refs/tags'>[...]</ulink></filename>
416 link beneath the "Tag" heading.
417 </para>
418
419 <para>
420 Some key tags for the <filename>poky</filename> are
421 <filename>jethro-14.0.3</filename>,
422 <filename>morty-16.0.1</filename>,
423 <filename>pyro-17.0.0</filename>, and
424 <filename>&DISTRO_NAME_NO_CAP;-&POKYVERSION;</filename>.
425 These tags represent Yocto Project releases.
426 </para>
427
428 <para>
429 When you create a local copy of the Git repository, you also
430 have access to all the tags in the upstream repository.
431 Similar to branches, you can create and checkout a local working
432 Git branch based on a tag name.
433 When you do this, you get a snapshot of the Git repository that
434 reflects the state of the files when the change was made associated
435 with that tag.
436 The most common use is to checkout a working branch that matches
437 a specific Yocto Project release.
438 Here is an example:
439 <literallayout class='monospaced'>
440 $ cd ~
441 $ git clone git://git.yoctoproject.org/poky
442 $ cd poky
443 $ git fetch --all --tags --prune
444 $ git checkout tags/pyro-17.0.0 -b my-pyro-17.0.0
445 </literallayout>
446 In this example, the name of the top-level directory of your
447 local Yocto Project repository is <filename>poky</filename>.
448 After moving to the <filename>poky</filename> directory, the
449 <filename>git fetch</filename> command makes all the upstream
450 tags available locally in your repository.
451 Finally, the <filename>git checkout</filename> command
452 creates and checks out a branch named "my-pyro-17.0.0" that is
453 based on the specific change upstream in the repository
454 associated with the "pyro-17.0.0" tag.
455 The files in your repository now exactly match that particular
456 Yocto Project release as it is tagged in the upstream Git
457 repository.
458 It is important to understand that when you create and
459 checkout a local working branch based on a tag, your environment
460 matches a specific point in time and not the entire development
461 branch (i.e. the "tip" of the branch).
462 </para>
463 </section>
464
465 <section id='basic-commands'>
466 <title>Basic Commands</title>
467
468 <para>
469 Git has an extensive set of commands that lets you manage changes
470 and perform collaboration over the life of a project.
471 Conveniently though, you can manage with a small set of basic
472 operations and workflows once you understand the basic
473 philosophy behind Git.
474 You do not have to be an expert in Git to be functional.
475 A good place to look for instruction on a minimal set of Git
476 commands is
477 <ulink url='http://git-scm.com/documentation'>here</ulink>.
478 </para>
479
480 <para>
481 If you do not know much about Git, you should educate
482 yourself by visiting the links previously mentioned.
483 </para>
484
485 <para>
486 The following list of Git commands briefly describes some basic
487 Git operations as a way to get started.
488 As with any set of commands, this list (in most cases) simply shows
489 the base command and omits the many arguments they support.
490 See the Git documentation for complete descriptions and strategies
491 on how to use these commands:
492 <itemizedlist>
493 <listitem><para>
494 <emphasis><filename>git init</filename>:</emphasis>
495 Initializes an empty Git repository.
496 You cannot use Git commands unless you have a
497 <filename>.git</filename> repository.
498 </para></listitem>
499 <listitem><para id='git-commands-clone'>
500 <emphasis><filename>git clone</filename>:</emphasis>
501 Creates a local clone of a Git repository that is on
502 equal footing with a fellow developer’s Git repository
503 or an upstream repository.
504 </para></listitem>
505 <listitem><para>
506 <emphasis><filename>git add</filename>:</emphasis>
507 Locally stages updated file contents to the index that
508 Git uses to track changes.
509 You must stage all files that have changed before you
510 can commit them.
511 </para></listitem>
512 <listitem><para>
513 <emphasis><filename>git commit</filename>:</emphasis>
514 Creates a local "commit" that documents the changes you
515 made.
516 Only changes that have been staged can be committed.
517 Commits are used for historical purposes, for determining
518 if a maintainer of a project will allow the change,
519 and for ultimately pushing the change from your local
520 Git repository into the project’s upstream repository.
521 </para></listitem>
522 <listitem><para>
523 <emphasis><filename>git status</filename>:</emphasis>
524 Reports any modified files that possibly need to be
525 staged and gives you a status of where you stand regarding
526 local commits as compared to the upstream repository.
527 </para></listitem>
528 <listitem><para>
529 <emphasis><filename>git checkout</filename> <replaceable>branch-name</replaceable>:</emphasis>
530 Changes your working branch.
531 This command is analogous to "cd".
532 </para></listitem>
533 <listitem><para><emphasis><filename>git checkout –b</filename> <replaceable>working-branch</replaceable>:</emphasis>
534 Creates and checks out a working branch on your local
535 machine that you can use to isolate your work.
536 It is a good idea to use local branches when adding
537 specific features or changes.
538 Using isolated branches facilitates easy removal of
539 changes if they do not work out.
540 </para></listitem>
541 <listitem><para><emphasis><filename>git branch</filename>:</emphasis>
542 Displays the existing local branches associated with your
543 local repository.
544 The branch that you have currently checked out is noted
545 with an asterisk character.
546 </para></listitem>
547 <listitem><para>
548 <emphasis><filename>git branch -D</filename> <replaceable>branch-name</replaceable>:</emphasis>
549 Deletes an existing local branch.
550 You need to be in a local branch other than the one you
551 are deleting in order to delete
552 <replaceable>branch-name</replaceable>.
553 </para></listitem>
554 <listitem><para>
555 <emphasis><filename>git pull</filename>:</emphasis>
556 Retrieves information from an upstream Git repository
557 and places it in your local Git repository.
558 You use this command to make sure you are synchronized with
559 the repository from which you are basing changes
560 (.e.g. the "master" branch).
561 </para></listitem>
562 <listitem><para>
563 <emphasis><filename>git push</filename>:</emphasis>
564 Sends all your committed local changes to the upstream Git
565 repository that your local repository is tracking
566 (e.g. a contribution repository).
567 The maintainer of the project draws from these repositories
568 to merge changes (commits) into the appropriate branch
569 of project's upstream repository.
570 </para></listitem>
571 <listitem><para>
572 <emphasis><filename>git merge</filename>:</emphasis>
573 Combines or adds changes from one
574 local branch of your repository with another branch.
575 When you create a local Git repository, the default branch
576 is named "master".
577 A typical workflow is to create a temporary branch that is
578 based off "master" that you would use for isolated work.
579 You would make your changes in that isolated branch,
580 stage and commit them locally, switch to the "master"
581 branch, and then use the <filename>git merge</filename>
582 command to apply the changes from your isolated branch
583 into the currently checked out branch (e.g. "master").
584 After the merge is complete and if you are done with
585 working in that isolated branch, you can safely delete
586 the isolated branch.
587 </para></listitem>
588 <listitem><para>
589 <emphasis><filename>git cherry-pick</filename>:</emphasis>
590 Choose and apply specific commits from one branch
591 into another branch.
592 There are times when you might not be able to merge
593 all the changes in one branch with
594 another but need to pick out certain ones.
595 </para></listitem>
596 <listitem><para>
597 <emphasis><filename>gitk</filename>:</emphasis>
598 Provides a GUI view of the branches and changes in your
599 local Git repository.
600 This command is a good way to graphically see where things
601 have diverged in your local repository.
602 <note>
603 You need to install the <filename>gitk</filename>
604 package on your development system to use this
605 command.
606 </note>
607 </para></listitem>
608 <listitem><para>
609 <emphasis><filename>git log</filename>:</emphasis>
610 Reports a history of your commits to the repository.
611 This report lists all commits regardless of whether you
612 have pushed them upstream or not.
613 </para></listitem>
614 <listitem><para>
615 <emphasis><filename>git diff</filename>:</emphasis>
616 Displays line-by-line differences between a local
617 working file and the same file as understood by Git.
618 This command is useful to see what you have changed
619 in any given file.
620 </para></listitem>
621 </itemizedlist>
622 </para>
623 </section>
624</section>
625
626<section id='yocto-project-repositories'>
627 <title>Yocto Project Source Repositories</title>
628
629 <para>
630 The Yocto Project team maintains complete source repositories for all
631 Yocto Project files at
632 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi'></ulink>.
633 This web-based source code browser is organized into categories by
634 function such as IDE Plugins, Matchbox, Poky, Yocto Linux Kernel, and
635 so forth.
636 From the interface, you can click on any particular item in the "Name"
637 column and see the URL at the bottom of the page that you need to clone
638 a Git repository for that particular item.
639 Having a local Git repository of the
640 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>,
641 which is usually named "poky", allows
642 you to make changes, contribute to the history, and ultimately enhance
643 the Yocto Project's tools, Board Support Packages, and so forth.
644 </para>
645
646 <para>
647 For any supported release of Yocto Project, you can also go to the
648 <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink> and
649 select the "Downloads" tab and get a released tarball of the
650 <filename>poky</filename> repository or any supported BSP tarballs.
651 Unpacking these tarballs gives you a snapshot of the released
652 files.
653 <note><title>Notes</title>
654 <itemizedlist>
655 <listitem><para>
656 The recommended method for setting up the Yocto Project
657 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>
658 and the files for supported BSPs
659 (e.g., <filename>meta-intel</filename>) is to use
660 <link linkend='git'>Git</link> to create a local copy of
661 the upstream repositories.
662 </para></listitem>
663 <listitem><para>
664 Be sure to always work in matching branches for both
665 the selected BSP repository and the
666 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>
667 (i.e. <filename>poky</filename>) repository.
668 For example, if you have checked out the "master" branch
669 of <filename>poky</filename> and you are going to use
670 <filename>meta-intel</filename>, be sure to checkout the
671 "master" branch of <filename>meta-intel</filename>.
672 </para></listitem>
673 </itemizedlist>
674 </note>
675 </para>
676
677 <para>
678 In summary, here is where you can get the project files needed for
679 development:
680 <itemizedlist>
681 <listitem><para id='source-repositories'>
682 <emphasis>
683 <ulink url='&YOCTO_GIT_URL;/cgit/cgit.cgi'>Source Repositories:</ulink>
684 </emphasis>
685 This area contains IDE Plugins, Matchbox, Poky, Poky Support,
686 Tools, Yocto Linux Kernel, and Yocto Metadata Layers.
687 You can create local copies of Git repositories for each of
688 these areas.</para>
689
690 <para>
691 <imagedata fileref="figures/source-repos.png" align="center" width="6in" depth="4in" />
692 For steps on how to view and access these upstream Git
693 repositories, see the
694 "<ulink url='&YOCTO_DOCS_DEV_URL;#accessing-source-repositories'>Accessing Source Repositories</ulink>"
695 Section in the Yocto Project Development Tasks Manual.
696 </para></listitem>
697 <listitem><para><anchor id='index-downloads' />
698 <emphasis>
699 <ulink url='&YOCTO_DL_URL;/releases/'>Index of /releases:</ulink>
700 </emphasis>
701 This is an index of releases such as
702 the <trademark class='trade'>Eclipse</trademark>
703 Yocto Plug-in, miscellaneous support, Poky, Pseudo, installers
704 for cross-development toolchains, and all released versions of
705 Yocto Project in the form of images or tarballs.
706 Downloading and extracting these files does not produce a local
707 copy of the Git repository but rather a snapshot of a
708 particular release or image.</para>
709
710 <para>
711 <imagedata fileref="figures/index-downloads.png" align="center" width="6in" depth="3.5in" />
712 For steps on how to view and access these files, see the
713 "<ulink url='&YOCTO_DOCS_DEV_URL;#accessing-index-of-releases'>Accessing Index of Releases</ulink>"
714 section in the Yocto Project Development Tasks Manual.
715 </para></listitem>
716 <listitem><para id='downloads-page'>
717 <emphasis>"Downloads" page for the
718 <ulink url='&YOCTO_HOME_URL;'>Yocto Project Website</ulink>:
719 </emphasis></para>
720
721 <para role="writernotes">This section will change due to
722 reworking of the YP Website.</para>
723
724 <para>The Yocto Project website includes a "Downloads" tab
725 that allows you to download any Yocto Project
726 release and Board Support Package (BSP) in tarball form.
727 The tarballs are similar to those found in the
728 <ulink url='&YOCTO_DL_URL;/releases/'>Index of /releases:</ulink> area.</para>
729
730 <para>
731 <imagedata fileref="figures/yp-download.png" align="center" width="6in" depth="4in" />
732 For steps on how to use the "Downloads" page, see the
733 "<ulink url='&YOCTO_DOCS_DEV_URL;#using-the-downloads-page'>Using the Downloads Page</ulink>"
734 section in the Yocto Project Development Tasks Manual.
735 </para></listitem>
736 </itemizedlist>
737 </para>
738</section>
739
740<section id='licensing'>
741 <title>Licensing</title>
742
743 <para>
744 Because open source projects are open to the public, they have
745 different licensing structures in place.
746 License evolution for both Open Source and Free Software has an
747 interesting history.
748 If you are interested in this history, you can find basic information
749 here:
750 <itemizedlist>
751 <listitem><para>
752 <ulink url='http://en.wikipedia.org/wiki/Open-source_license'>Open source license history</ulink>
753 </para></listitem>
754 <listitem><para>
755 <ulink url='http://en.wikipedia.org/wiki/Free_software_license'>Free software license history</ulink>
756 </para></listitem>
757 </itemizedlist>
758 </para>
759
760 <para>
761 In general, the Yocto Project is broadly licensed under the
762 Massachusetts Institute of Technology (MIT) License.
763 MIT licensing permits the reuse of software within proprietary
764 software as long as the license is distributed with that software.
765 MIT is also compatible with the GNU General Public License (GPL).
766 Patches to the Yocto Project follow the upstream licensing scheme.
767 You can find information on the MIT license
768 <ulink url='http://www.opensource.org/licenses/mit-license.php'>here</ulink>.
769 You can find information on the GNU GPL
770 <ulink url='http://www.opensource.org/licenses/LGPL-3.0'>here</ulink>.
771 </para>
772
773 <para>
774 When you build an image using the Yocto Project, the build process
775 uses a known list of licenses to ensure compliance.
776 You can find this list in the
777 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>
778 at <filename>meta/files/common-licenses</filename>.
779 Once the build completes, the list of all licenses found and used
780 during that build are kept in the
781 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
782 at <filename>tmp/deploy/licenses</filename>.
783 </para>
784
785 <para>
786 If a module requires a license that is not in the base list, the
787 build process generates a warning during the build.
788 These tools make it easier for a developer to be certain of the
789 licenses with which their shipped products must comply.
790 However, even with these tools it is still up to the developer to
791 resolve potential licensing issues.
792 </para>
793
794 <para>
795 The base list of licenses used by the build process is a combination
796 of the Software Package Data Exchange (SPDX) list and the Open
797 Source Initiative (OSI) projects.
798 <ulink url='http://spdx.org'>SPDX Group</ulink> is a working group of
799 the Linux Foundation that maintains a specification for a standard
800 format for communicating the components, licenses, and copyrights
801 associated with a software package.
802 <ulink url='http://opensource.org'>OSI</ulink> is a corporation
803 dedicated to the Open Source Definition and the effort for reviewing
804 and approving licenses that conform to the Open Source Definition
805 (OSD).
806 </para>
807
808 <para>
809 You can find a list of the combined SPDX and OSI licenses that the
810 Yocto Project uses in the
811 <filename>meta/files/common-licenses</filename> directory in your
812 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
813 </para>
814
815 <para>
816 For information that can help you maintain compliance with various
817 open source licensing during the lifecycle of a product created using
818 the Yocto Project, see the
819 "<ulink url='&YOCTO_DOCS_DEV_URL;#maintaining-open-source-license-compliance-during-your-products-lifecycle'>Maintaining Open Source License Compliance During Your Product's Lifecycle</ulink>"
820 section in the Yocto Project Development Tasks Manual.
821 </para>
822</section>
823
824<section id='recipe-syntax'>
825 <title>Recipe Syntax</title>
826
827 <para>
828 Understanding recipe file syntax is important for
829 writing recipes.
830 The following list overviews the basic items that make up a
831 BitBake recipe file.
832 For more complete BitBake syntax descriptions, see the
833 "<ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>"
834 chapter of the BitBake User Manual.
835 <itemizedlist>
836 <listitem><para><emphasis>Variable Assignments and Manipulations:</emphasis>
837 Variable assignments allow a value to be assigned to a
838 variable.
839 The assignment can be static text or might include
840 the contents of other variables.
841 In addition to the assignment, appending and prepending
842 operations are also supported.</para>
843 <para>The following example shows some of the ways
844 you can use variables in recipes:
845 <literallayout class='monospaced'>
846 S = "${WORKDIR}/postfix-${PV}"
847 CFLAGS += "-DNO_ASM"
848 SRC_URI_append = " file://fixup.patch"
849 </literallayout>
850 </para></listitem>
851 <listitem><para><emphasis>Functions:</emphasis>
852 Functions provide a series of actions to be performed.
853 You usually use functions to override the default
854 implementation of a task function or to complement
855 a default function (i.e. append or prepend to an
856 existing function).
857 Standard functions use <filename>sh</filename> shell
858 syntax, although access to OpenEmbedded variables and
859 internal methods are also available.</para>
860 <para>The following is an example function from the
861 <filename>sed</filename> recipe:
862 <literallayout class='monospaced'>
863 do_install () {
864 autotools_do_install
865 install -d ${D}${base_bindir}
866 mv ${D}${bindir}/sed ${D}${base_bindir}/sed
867 rmdir ${D}${bindir}/
868 }
869 </literallayout>
870 It is also possible to implement new functions that
871 are called between existing tasks as long as the
872 new functions are not replacing or complementing the
873 default functions.
874 You can implement functions in Python
875 instead of shell.
876 Both of these options are not seen in the majority of
877 recipes.</para></listitem>
878 <listitem><para><emphasis>Keywords:</emphasis>
879 BitBake recipes use only a few keywords.
880 You use keywords to include common
881 functions (<filename>inherit</filename>), load parts
882 of a recipe from other files
883 (<filename>include</filename> and
884 <filename>require</filename>) and export variables
885 to the environment (<filename>export</filename>).</para>
886 <para>The following example shows the use of some of
887 these keywords:
888 <literallayout class='monospaced'>
889 export POSTCONF = "${STAGING_BINDIR}/postconf"
890 inherit autoconf
891 require otherfile.inc
892 </literallayout>
893 </para></listitem>
894 <listitem><para><emphasis>Comments:</emphasis>
895 Any lines that begin with the hash character
896 (<filename>#</filename>) are treated as comment lines
897 and are ignored:
898 <literallayout class='monospaced'>
899 # This is a comment
900 </literallayout>
901 </para></listitem>
902 </itemizedlist>
903 </para>
904
905 <para>
906 This next list summarizes the most important and most commonly
907 used parts of the recipe syntax.
908 For more information on these parts of the syntax, you can
909 reference the
910 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual-metadata'>Syntax and Operators</ulink>
911 chapter in the BitBake User Manual.
912 <itemizedlist>
913 <listitem><para><emphasis>Line Continuation: <filename>\</filename></emphasis> -
914 Use the backward slash (<filename>\</filename>)
915 character to split a statement over multiple lines.
916 Place the slash character at the end of the line that
917 is to be continued on the next line:
918 <literallayout class='monospaced'>
919 VAR = "A really long \
920 line"
921 </literallayout>
922 <note>
923 You cannot have any characters including spaces
924 or tabs after the slash character.
925 </note>
926 </para></listitem>
927 <listitem><para>
928 <emphasis>Using Variables: <filename>${...}</filename></emphasis> -
929 Use the <filename>${<replaceable>VARNAME</replaceable>}</filename> syntax to
930 access the contents of a variable:
931 <literallayout class='monospaced'>
932 SRC_URI = "${SOURCEFORGE_MIRROR}/libpng/zlib-${PV}.tar.gz"
933 </literallayout>
934 <note>
935 It is important to understand that the value of a
936 variable expressed in this form does not get
937 substituted automatically.
938 The expansion of these expressions happens
939 on-demand later (e.g. usually when a function that
940 makes reference to the variable executes).
941 This behavior ensures that the values are most
942 appropriate for the context in which they are
943 finally used.
944 On the rare occasion that you do need the variable
945 expression to be expanded immediately, you can use
946 the <filename>:=</filename> operator instead of
947 <filename>=</filename> when you make the
948 assignment, but this is not generally needed.
949 </note>
950 </para></listitem>
951 <listitem><para><emphasis>Quote All Assignments: <filename>"<replaceable>value</replaceable>"</filename></emphasis> -
952 Use double quotes around the value in all variable
953 assignments.
954 <literallayout class='monospaced'>
955 VAR1 = "${OTHERVAR}"
956 VAR2 = "The version is ${PV}"
957 </literallayout>
958 </para></listitem>
959 <listitem><para><emphasis>Conditional Assignment: <filename>?=</filename></emphasis> -
960 Conditional assignment is used to assign a value to
961 a variable, but only when the variable is currently
962 unset.
963 Use the question mark followed by the equal sign
964 (<filename>?=</filename>) to make a "soft" assignment
965 used for conditional assignment.
966 Typically, "soft" assignments are used in the
967 <filename>local.conf</filename> file for variables
968 that are allowed to come through from the external
969 environment.
970 </para>
971 <para>Here is an example where
972 <filename>VAR1</filename> is set to "New value" if
973 it is currently empty.
974 However, if <filename>VAR1</filename> has already been
975 set, it remains unchanged:
976 <literallayout class='monospaced'>
977 VAR1 ?= "New value"
978 </literallayout>
979 In this next example, <filename>VAR1</filename>
980 is left with the value "Original value":
981 <literallayout class='monospaced'>
982 VAR1 = "Original value"
983 VAR1 ?= "New value"
984 </literallayout>
985 </para></listitem>
986 <listitem><para><emphasis>Appending: <filename>+=</filename></emphasis> -
987 Use the plus character followed by the equals sign
988 (<filename>+=</filename>) to append values to existing
989 variables.
990 <note>
991 This operator adds a space between the existing
992 content of the variable and the new content.
993 </note></para>
994 <para>Here is an example:
995 <literallayout class='monospaced'>
996 SRC_URI += "file://fix-makefile.patch"
997 </literallayout>
998 </para></listitem>
999 <listitem><para><emphasis>Prepending: <filename>=+</filename></emphasis> -
1000 Use the equals sign followed by the plus character
1001 (<filename>=+</filename>) to prepend values to existing
1002 variables.
1003 <note>
1004 This operator adds a space between the new content
1005 and the existing content of the variable.
1006 </note></para>
1007 <para>Here is an example:
1008 <literallayout class='monospaced'>
1009 VAR =+ "Starts"
1010 </literallayout>
1011 </para></listitem>
1012 <listitem><para><emphasis>Appending: <filename>_append</filename></emphasis> -
1013 Use the <filename>_append</filename> operator to
1014 append values to existing variables.
1015 This operator does not add any additional space.
1016 Also, the operator is applied after all the
1017 <filename>+=</filename>, and
1018 <filename>=+</filename> operators have been applied and
1019 after all <filename>=</filename> assignments have
1020 occurred.
1021 </para>
1022 <para>The following example shows the space being
1023 explicitly added to the start to ensure the appended
1024 value is not merged with the existing value:
1025 <literallayout class='monospaced'>
1026 SRC_URI_append = " file://fix-makefile.patch"
1027 </literallayout>
1028 You can also use the <filename>_append</filename>
1029 operator with overrides, which results in the actions
1030 only being performed for the specified target or
1031 machine:
1032 <literallayout class='monospaced'>
1033 SRC_URI_append_sh4 = " file://fix-makefile.patch"
1034 </literallayout>
1035 </para></listitem>
1036 <listitem><para><emphasis>Prepending: <filename>_prepend</filename></emphasis> -
1037 Use the <filename>_prepend</filename> operator to
1038 prepend values to existing variables.
1039 This operator does not add any additional space.
1040 Also, the operator is applied after all the
1041 <filename>+=</filename>, and
1042 <filename>=+</filename> operators have been applied and
1043 after all <filename>=</filename> assignments have
1044 occurred.
1045 </para>
1046 <para>The following example shows the space being
1047 explicitly added to the end to ensure the prepended
1048 value is not merged with the existing value:
1049 <literallayout class='monospaced'>
1050 CFLAGS_prepend = "-I${S}/myincludes "
1051 </literallayout>
1052 You can also use the <filename>_prepend</filename>
1053 operator with overrides, which results in the actions
1054 only being performed for the specified target or
1055 machine:
1056 <literallayout class='monospaced'>
1057 CFLAGS_prepend_sh4 = "-I${S}/myincludes "
1058 </literallayout>
1059 </para></listitem>
1060 <listitem><para><emphasis>Overrides:</emphasis> -
1061 You can use overrides to set a value conditionally,
1062 typically based on how the recipe is being built.
1063 For example, to set the
1064 <ulink url='&YOCTO_DOCS_REF_URL;#var-KBRANCH'><filename>KBRANCH</filename></ulink>
1065 variable's value to "standard/base" for any target
1066 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
1067 except for qemuarm where it should be set to
1068 "standard/arm-versatile-926ejs", you would do the
1069 following:
1070 <literallayout class='monospaced'>
1071 KBRANCH = "standard/base"
1072 KBRANCH_qemuarm = "standard/arm-versatile-926ejs"
1073 </literallayout>
1074 Overrides are also used to separate alternate values
1075 of a variable in other situations.
1076 For example, when setting variables such as
1077 <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>
1078 and
1079 <ulink url='&YOCTO_DOCS_REF_URL;#var-RDEPENDS'><filename>RDEPENDS</filename></ulink>
1080 that are specific to individual packages produced by
1081 a recipe, you should always use an override that
1082 specifies the name of the package.
1083 </para></listitem>
1084 <listitem><para><emphasis>Indentation:</emphasis>
1085 Use spaces for indentation rather than than tabs.
1086 For shell functions, both currently work.
1087 However, it is a policy decision of the Yocto Project
1088 to use tabs in shell functions.
1089 Realize that some layers have a policy to use spaces
1090 for all indentation.
1091 </para></listitem>
1092 <listitem><para><emphasis>Using Python for Complex Operations: <filename>${@<replaceable>python_code</replaceable>}</filename></emphasis> -
1093 For more advanced processing, it is possible to use
1094 Python code during variable assignments (e.g.
1095 search and replacement on a variable).</para>
1096 <para>You indicate Python code using the
1097 <filename>${@<replaceable>python_code</replaceable>}</filename>
1098 syntax for the variable assignment:
1099 <literallayout class='monospaced'>
1100 SRC_URI = "ftp://ftp.info-zip.org/pub/infozip/src/zip${@d.getVar('PV',1).replace('.', '')}.tgz
1101 </literallayout>
1102 </para></listitem>
1103 <listitem><para><emphasis>Shell Function Syntax:</emphasis>
1104 Write shell functions as if you were writing a shell
1105 script when you describe a list of actions to take.
1106 You should ensure that your script works with a generic
1107 <filename>sh</filename> and that it does not require
1108 any <filename>bash</filename> or other shell-specific
1109 functionality.
1110 The same considerations apply to various system
1111 utilities (e.g. <filename>sed</filename>,
1112 <filename>grep</filename>, <filename>awk</filename>,
1113 and so forth) that you might wish to use.
1114 If in doubt, you should check with multiple
1115 implementations - including those from BusyBox.
1116 </para></listitem>
1117 </itemizedlist>
1118 </para>
1119</section>
1120
1121<section id="development-concepts">
1122 <title>Development Concepts</title>
1123
1124 <para>
1125 This section takes a more detailed look inside the development
1126 process.
1127 The following diagram represents development at a high level.
1128 The remainder of this chapter expands on the fundamental input, output,
1129 process, and
1130 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>) blocks
1131 that make up development in the Yocto Project environment.
1132 </para>
1133
1134 <para id='general-yocto-environment-figure'>
1135 <imagedata fileref="figures/yocto-environment-ref.png" align="center" width="8in" depth="4.25in" />
1136 </para>
1137
1138 <para>
1139 In general, development consists of several functional areas:
1140 <itemizedlist>
1141 <listitem><para><emphasis>User Configuration:</emphasis>
1142 Metadata you can use to control the build process.
1143 </para></listitem>
1144 <listitem><para><emphasis>Metadata Layers:</emphasis>
1145 Various layers that provide software, machine, and
1146 distro Metadata.</para></listitem>
1147 <listitem><para><emphasis>Source Files:</emphasis>
1148 Upstream releases, local projects, and SCMs.</para></listitem>
1149 <listitem><para><emphasis>Build System:</emphasis>
1150 Processes under the control of
1151 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>.
1152 This block expands on how BitBake fetches source, applies
1153 patches, completes compilation, analyzes output for package
1154 generation, creates and tests packages, generates images, and
1155 generates cross-development tools.</para></listitem>
1156 <listitem><para><emphasis>Package Feeds:</emphasis>
1157 Directories containing output packages (RPM, DEB or IPK),
1158 which are subsequently used in the construction of an image or
1159 SDK, produced by the build system.
1160 These feeds can also be copied and shared using a web server or
1161 other means to facilitate extending or updating existing
1162 images on devices at runtime if runtime package management is
1163 enabled.</para></listitem>
1164 <listitem><para><emphasis>Images:</emphasis>
1165 Images produced by the development process.
1166 </para></listitem>
1167 <listitem><para><emphasis>Application Development SDK:</emphasis>
1168 Cross-development tools that are produced along with an image
1169 or separately with BitBake.</para></listitem>
1170 </itemizedlist>
1171 </para>
1172
1173 <section id="user-configuration">
1174 <title>User Configuration</title>
1175
1176 <para>
1177 User configuration helps define the build.
1178 Through user configuration, you can tell BitBake the
1179 target architecture for which you are building the image,
1180 where to store downloaded source, and other build properties.
1181 </para>
1182
1183 <para>
1184 The following figure shows an expanded representation of the
1185 "User Configuration" box of the
1186 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>:
1187 </para>
1188
1189 <para>
1190 <imagedata fileref="figures/user-configuration.png" align="center" width="8in" depth="4.5in" />
1191 </para>
1192
1193 <para>
1194 BitBake needs some basic configuration files in order to complete
1195 a build.
1196 These files are <filename>*.conf</filename> files.
1197 The minimally necessary ones reside as example files in the
1198 <ulink url='&YOCTO_DOCS_REF_URL;#source-directory'>Source Directory</ulink>.
1199 For simplicity, this section refers to the Source Directory as
1200 the "Poky Directory."
1201 </para>
1202
1203 <para>
1204 When you clone the <filename>poky</filename> Git repository or you
1205 download and unpack a Yocto Project release, you can set up the
1206 Source Directory to be named anything you want.
1207 For this discussion, the cloned repository uses the default
1208 name <filename>poky</filename>.
1209 <note>
1210 The Poky repository is primarily an aggregation of existing
1211 repositories.
1212 It is not a canonical upstream source.
1213 </note>
1214 </para>
1215
1216 <para>
1217 The <filename>meta-poky</filename> layer inside Poky contains
1218 a <filename>conf</filename> directory that has example
1219 configuration files.
1220 These example files are used as a basis for creating actual
1221 configuration files when you source the build environment
1222 script
1223 (i.e.
1224 <ulink url='&YOCTO_DOCS_REF_URL;#structure-core-script'><filename>&OE_INIT_FILE;</filename></ulink>).
1225 </para>
1226
1227 <para>
1228 Sourcing the build environment script creates a
1229 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
1230 if one does not already exist.
1231 BitBake uses the Build Directory for all its work during builds.
1232 The Build Directory has a <filename>conf</filename> directory that
1233 contains default versions of your <filename>local.conf</filename>
1234 and <filename>bblayers.conf</filename> configuration files.
1235 These default configuration files are created only if versions
1236 do not already exist in the Build Directory at the time you
1237 source the build environment setup script.
1238 </para>
1239
1240 <para>
1241 Because the Poky repository is fundamentally an aggregation of
1242 existing repositories, some users might be familiar with running
1243 the <filename>&OE_INIT_FILE;</filename> script in the context
1244 of separate OpenEmbedded-Core and BitBake repositories rather than a
1245 single Poky repository.
1246 This discussion assumes the script is executed from within a cloned
1247 or unpacked version of Poky.
1248 </para>
1249
1250 <para>
1251 Depending on where the script is sourced, different sub-scripts
1252 are called to set up the Build Directory (Yocto or OpenEmbedded).
1253 Specifically, the script
1254 <filename>scripts/oe-setup-builddir</filename> inside the
1255 poky directory sets up the Build Directory and seeds the directory
1256 (if necessary) with configuration files appropriate for the
1257 Yocto Project development environment.
1258 <note>
1259 The <filename>scripts/oe-setup-builddir</filename> script
1260 uses the <filename>$TEMPLATECONF</filename> variable to
1261 determine which sample configuration files to locate.
1262 </note>
1263 </para>
1264
1265 <para>
1266 The <filename>local.conf</filename> file provides many
1267 basic variables that define a build environment.
1268 Here is a list of a few.
1269 To see the default configurations in a <filename>local.conf</filename>
1270 file created by the build environment script, see the
1271 <filename>local.conf.sample</filename> in the
1272 <filename>meta-poky</filename> layer:
1273 <itemizedlist>
1274 <listitem><para><emphasis>Parallelism Options:</emphasis>
1275 Controlled by the
1276 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_NUMBER_THREADS'><filename>BB_NUMBER_THREADS</filename></ulink>,
1277 <ulink url='&YOCTO_DOCS_REF_URL;#var-PARALLEL_MAKE'><filename>PARALLEL_MAKE</filename></ulink>,
1278 and
1279 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_NUMBER_PARSE_THREADS'><filename>BB_NUMBER_PARSE_THREADS</filename></ulink>
1280 variables.</para></listitem>
1281 <listitem><para><emphasis>Target Machine Selection:</emphasis>
1282 Controlled by the
1283 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>
1284 variable.</para></listitem>
1285 <listitem><para><emphasis>Download Directory:</emphasis>
1286 Controlled by the
1287 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
1288 variable.</para></listitem>
1289 <listitem><para><emphasis>Shared State Directory:</emphasis>
1290 Controlled by the
1291 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
1292 variable.</para></listitem>
1293 <listitem><para><emphasis>Build Output:</emphasis>
1294 Controlled by the
1295 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
1296 variable.</para></listitem>
1297 </itemizedlist>
1298 <note>
1299 Configurations set in the <filename>conf/local.conf</filename>
1300 file can also be set in the
1301 <filename>conf/site.conf</filename> and
1302 <filename>conf/auto.conf</filename> configuration files.
1303 </note>
1304 </para>
1305
1306 <para>
1307 The <filename>bblayers.conf</filename> file tells BitBake what
1308 layers you want considered during the build.
1309 By default, the layers listed in this file include layers
1310 minimally needed by the build system.
1311 However, you must manually add any custom layers you have created.
1312 You can find more information on working with the
1313 <filename>bblayers.conf</filename> file in the
1314 "<ulink url='&YOCTO_DOCS_DEV_URL;#enabling-your-layer'>Enabling Your Layer</ulink>"
1315 section in the Yocto Project Development Tasks Manual.
1316 </para>
1317
1318 <para>
1319 The files <filename>site.conf</filename> and
1320 <filename>auto.conf</filename> are not created by the environment
1321 initialization script.
1322 If you want the <filename>site.conf</filename> file, you need to
1323 create that yourself.
1324 The <filename>auto.conf</filename> file is typically created by
1325 an autobuilder:
1326 <itemizedlist>
1327 <listitem><para><emphasis><filename>site.conf</filename>:</emphasis>
1328 You can use the <filename>conf/site.conf</filename>
1329 configuration file to configure multiple build directories.
1330 For example, suppose you had several build environments and
1331 they shared some common features.
1332 You can set these default build properties here.
1333 A good example is perhaps the packaging format to use
1334 through the
1335 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
1336 variable.</para>
1337 <para>One useful scenario for using the
1338 <filename>conf/site.conf</filename> file is to extend your
1339 <ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
1340 variable to include the path to a
1341 <filename>conf/site.conf</filename>.
1342 Then, when BitBake looks for Metadata using
1343 <filename>BBPATH</filename>, it finds the
1344 <filename>conf/site.conf</filename> file and applies your
1345 common configurations found in the file.
1346 To override configurations in a particular build directory,
1347 alter the similar configurations within that build
1348 directory's <filename>conf/local.conf</filename> file.
1349 </para></listitem>
1350 <listitem><para><emphasis><filename>auto.conf</filename>:</emphasis>
1351 The file is usually created and written to by
1352 an autobuilder.
1353 The settings put into the file are typically the same as
1354 you would find in the <filename>conf/local.conf</filename>
1355 or the <filename>conf/site.conf</filename> files.
1356 </para></listitem>
1357 </itemizedlist>
1358 </para>
1359
1360 <para>
1361 You can edit all configuration files to further define
1362 any particular build environment.
1363 This process is represented by the "User Configuration Edits"
1364 box in the figure.
1365 </para>
1366
1367 <para>
1368 When you launch your build with the
1369 <filename>bitbake <replaceable>target</replaceable></filename>
1370 command, BitBake sorts out the configurations to ultimately
1371 define your build environment.
1372 It is important to understand that the OpenEmbedded build system
1373 reads the configuration files in a specific order:
1374 <filename>site.conf</filename>, <filename>auto.conf</filename>,
1375 and <filename>local.conf</filename>.
1376 And, the build system applies the normal assignment statement
1377 rules.
1378 Because the files are parsed in a specific order, variable
1379 assignments for the same variable could be affected.
1380 For example, if the <filename>auto.conf</filename> file and
1381 the <filename>local.conf</filename> set
1382 <replaceable>variable1</replaceable> to different values, because
1383 the build system parses <filename>local.conf</filename> after
1384 <filename>auto.conf</filename>,
1385 <replaceable>variable1</replaceable> is assigned the value from
1386 the <filename>local.conf</filename> file.
1387 </para>
1388 </section>
1389
1390 <section id="metadata-machine-configuration-and-policy-configuration">
1391 <title>Metadata, Machine Configuration, and Policy Configuration</title>
1392
1393 <para>
1394 The previous section described the user configurations that
1395 define BitBake's global behavior.
1396 This section takes a closer look at the layers the build system
1397 uses to further control the build.
1398 These layers provide Metadata for the software, machine, and
1399 policy.
1400 </para>
1401
1402 <para>
1403 In general, three types of layer input exist:
1404 <itemizedlist>
1405 <listitem><para><emphasis>Policy Configuration:</emphasis>
1406 Distribution Layers provide top-level or general
1407 policies for the image or SDK being built.
1408 For example, this layer would dictate whether BitBake
1409 produces RPM or IPK packages.</para></listitem>
1410 <listitem><para><emphasis>Machine Configuration:</emphasis>
1411 Board Support Package (BSP) layers provide machine
1412 configurations.
1413 This type of information is specific to a particular
1414 target architecture.</para></listitem>
1415 <listitem><para><emphasis>Metadata:</emphasis>
1416 Software layers contain user-supplied recipe files,
1417 patches, and append files.
1418 </para></listitem>
1419 </itemizedlist>
1420 </para>
1421
1422 <para>
1423 The following figure shows an expanded representation of the
1424 Metadata, Machine Configuration, and Policy Configuration input
1425 (layers) boxes of the
1426 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>:
1427 </para>
1428
1429 <para>
1430 <imagedata fileref="figures/layer-input.png" align="center" width="8in" depth="7.5in" />
1431 </para>
1432
1433 <para>
1434 In general, all layers have a similar structure.
1435 They all contain a licensing file
1436 (e.g. <filename>COPYING</filename>) if the layer is to be
1437 distributed, a <filename>README</filename> file as good practice
1438 and especially if the layer is to be distributed, a
1439 configuration directory, and recipe directories.
1440 </para>
1441
1442 <para>
1443 The Yocto Project has many layers that can be used.
1444 You can see a web-interface listing of them on the
1445 <ulink url="http://git.yoctoproject.org/">Source Repositories</ulink>
1446 page.
1447 The layers are shown at the bottom categorized under
1448 "Yocto Metadata Layers."
1449 These layers are fundamentally a subset of the
1450 <ulink url="http://layers.openembedded.org/layerindex/layers/">OpenEmbedded Metadata Index</ulink>,
1451 which lists all layers provided by the OpenEmbedded community.
1452 <note>
1453 Layers exist in the Yocto Project Source Repositories that
1454 cannot be found in the OpenEmbedded Metadata Index.
1455 These layers are either deprecated or experimental in nature.
1456 </note>
1457 </para>
1458
1459 <para>
1460 BitBake uses the <filename>conf/bblayers.conf</filename> file,
1461 which is part of the user configuration, to find what layers it
1462 should be using as part of the build.
1463 </para>
1464
1465 <para>
1466 For more information on layers, see the
1467 "<ulink url='&YOCTO_DOCS_DEV_URL;#understanding-and-creating-layers'>Understanding and Creating Layers</ulink>"
1468 section in the Yocto Project Development Tasks Manual.
1469 </para>
1470
1471 <section id="distro-layer">
1472 <title>Distro Layer</title>
1473
1474 <para>
1475 The distribution layer provides policy configurations for your
1476 distribution.
1477 Best practices dictate that you isolate these types of
1478 configurations into their own layer.
1479 Settings you provide in
1480 <filename>conf/distro/<replaceable>distro</replaceable>.conf</filename> override
1481 similar
1482 settings that BitBake finds in your
1483 <filename>conf/local.conf</filename> file in the Build
1484 Directory.
1485 </para>
1486
1487 <para>
1488 The following list provides some explanation and references
1489 for what you typically find in the distribution layer:
1490 <itemizedlist>
1491 <listitem><para><emphasis>classes:</emphasis>
1492 Class files (<filename>.bbclass</filename>) hold
1493 common functionality that can be shared among
1494 recipes in the distribution.
1495 When your recipes inherit a class, they take on the
1496 settings and functions for that class.
1497 You can read more about class files in the
1498 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes'>Classes</ulink>"
1499 section of the Yocto Reference Manual.
1500 </para></listitem>
1501 <listitem><para><emphasis>conf:</emphasis>
1502 This area holds configuration files for the
1503 layer (<filename>conf/layer.conf</filename>),
1504 the distribution
1505 (<filename>conf/distro/<replaceable>distro</replaceable>.conf</filename>),
1506 and any distribution-wide include files.
1507 </para></listitem>
1508 <listitem><para><emphasis>recipes-*:</emphasis>
1509 Recipes and append files that affect common
1510 functionality across the distribution.
1511 This area could include recipes and append files
1512 to add distribution-specific configuration,
1513 initialization scripts, custom image recipes,
1514 and so forth.</para></listitem>
1515 </itemizedlist>
1516 </para>
1517 </section>
1518
1519 <section id="bsp-layer">
1520 <title>BSP Layer</title>
1521
1522 <para>
1523 The BSP Layer provides machine configurations.
1524 Everything in this layer is specific to the machine for which
1525 you are building the image or the SDK.
1526 A common structure or form is defined for BSP layers.
1527 You can learn more about this structure in the
1528 <ulink url='&YOCTO_DOCS_BSP_URL;'>Yocto Project Board Support Package (BSP) Developer's Guide</ulink>.
1529 <note>
1530 In order for a BSP layer to be considered compliant with the
1531 Yocto Project, it must meet some structural requirements.
1532 </note>
1533 </para>
1534
1535 <para>
1536 The BSP Layer's configuration directory contains
1537 configuration files for the machine
1538 (<filename>conf/machine/<replaceable>machine</replaceable>.conf</filename>) and,
1539 of course, the layer (<filename>conf/layer.conf</filename>).
1540 </para>
1541
1542 <para>
1543 The remainder of the layer is dedicated to specific recipes
1544 by function: <filename>recipes-bsp</filename>,
1545 <filename>recipes-core</filename>,
1546 <filename>recipes-graphics</filename>, and
1547 <filename>recipes-kernel</filename>.
1548 Metadata can exist for multiple formfactors, graphics
1549 support systems, and so forth.
1550 <note>
1551 While the figure shows several <filename>recipes-*</filename>
1552 directories, not all these directories appear in all
1553 BSP layers.
1554 </note>
1555 </para>
1556 </section>
1557
1558 <section id="software-layer">
1559 <title>Software Layer</title>
1560
1561 <para>
1562 The software layer provides the Metadata for additional
1563 software packages used during the build.
1564 This layer does not include Metadata that is specific to the
1565 distribution or the machine, which are found in their
1566 respective layers.
1567 </para>
1568
1569 <para>
1570 This layer contains any new recipes that your project needs
1571 in the form of recipe files.
1572 </para>
1573 </section>
1574 </section>
1575
1576 <section id="sources-dev-environment">
1577 <title>Sources</title>
1578
1579 <para>
1580 In order for the OpenEmbedded build system to create an image or
1581 any target, it must be able to access source files.
1582 The
1583 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
1584 represents source files using the "Upstream Project Releases",
1585 "Local Projects", and "SCMs (optional)" boxes.
1586 The figure represents mirrors, which also play a role in locating
1587 source files, with the "Source Mirror(s)" box.
1588 </para>
1589
1590 <para>
1591 The method by which source files are ultimately organized is
1592 a function of the project.
1593 For example, for released software, projects tend to use tarballs
1594 or other archived files that can capture the state of a release
1595 guaranteeing that it is statically represented.
1596 On the other hand, for a project that is more dynamic or
1597 experimental in nature, a project might keep source files in a
1598 repository controlled by a Source Control Manager (SCM) such as
1599 Git.
1600 Pulling source from a repository allows you to control
1601 the point in the repository (the revision) from which you want to
1602 build software.
1603 Finally, a combination of the two might exist, which would give the
1604 consumer a choice when deciding where to get source files.
1605 </para>
1606
1607 <para>
1608 BitBake uses the
1609 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1610 variable to point to source files regardless of their location.
1611 Each recipe must have a <filename>SRC_URI</filename> variable
1612 that points to the source.
1613 </para>
1614
1615 <para>
1616 Another area that plays a significant role in where source files
1617 come from is pointed to by the
1618 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
1619 variable.
1620 This area is a cache that can hold previously downloaded source.
1621 You can also instruct the OpenEmbedded build system to create
1622 tarballs from Git repositories, which is not the default behavior,
1623 and store them in the <filename>DL_DIR</filename> by using the
1624 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
1625 variable.
1626 </para>
1627
1628 <para>
1629 Judicious use of a <filename>DL_DIR</filename> directory can
1630 save the build system a trip across the Internet when looking
1631 for files.
1632 A good method for using a download directory is to have
1633 <filename>DL_DIR</filename> point to an area outside of your
1634 Build Directory.
1635 Doing so allows you to safely delete the Build Directory
1636 if needed without fear of removing any downloaded source file.
1637 </para>
1638
1639 <para>
1640 The remainder of this section provides a deeper look into the
1641 source files and the mirrors.
1642 Here is a more detailed look at the source file area of the
1643 base figure:
1644 <imagedata fileref="figures/source-input.png" align="center" width="7in" depth="7.5in" />
1645 </para>
1646
1647 <section id='upstream-project-releases'>
1648 <title>Upstream Project Releases</title>
1649
1650 <para>
1651 Upstream project releases exist anywhere in the form of an
1652 archived file (e.g. tarball or zip file).
1653 These files correspond to individual recipes.
1654 For example, the figure uses specific releases each for
1655 BusyBox, Qt, and Dbus.
1656 An archive file can be for any released product that can be
1657 built using a recipe.
1658 </para>
1659 </section>
1660
1661 <section id='local-projects'>
1662 <title>Local Projects</title>
1663
1664 <para>
1665 Local projects are custom bits of software the user provides.
1666 These bits reside somewhere local to a project - perhaps
1667 a directory into which the user checks in items (e.g.
1668 a local directory containing a development source tree
1669 used by the group).
1670 </para>
1671
1672 <para>
1673 The canonical method through which to include a local project
1674 is to use the
1675 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc</filename></ulink>
1676 class to include that local project.
1677 You use either the <filename>local.conf</filename> or a
1678 recipe's append file to override or set the
1679 recipe to point to the local directory on your disk to pull
1680 in the whole source tree.
1681 </para>
1682
1683 <para>
1684 For information on how to use the
1685 <filename>externalsrc</filename> class, see the
1686 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-externalsrc'><filename>externalsrc.bbclass</filename></ulink>"
1687 section.
1688 </para>
1689 </section>
1690
1691 <section id='scms'>
1692 <title>Source Control Managers (Optional)</title>
1693
1694 <para>
1695 Another place the build system can get source files from is
1696 through an SCM such as Git or Subversion.
1697 In this case, a repository is cloned or checked out.
1698 The
1699 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
1700 task inside BitBake uses
1701 the <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1702 variable and the argument's prefix to determine the correct
1703 fetcher module.
1704 </para>
1705
1706 <note>
1707 For information on how to have the OpenEmbedded build system
1708 generate tarballs for Git repositories and place them in the
1709 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
1710 directory, see the
1711 <ulink url='&YOCTO_DOCS_REF_URL;#var-BB_GENERATE_MIRROR_TARBALLS'><filename>BB_GENERATE_MIRROR_TARBALLS</filename></ulink>
1712 variable.
1713 </note>
1714
1715 <para>
1716 When fetching a repository, BitBake uses the
1717 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRCREV'><filename>SRCREV</filename></ulink>
1718 variable to determine the specific revision from which to
1719 build.
1720 </para>
1721 </section>
1722
1723 <section id='source-mirrors'>
1724 <title>Source Mirror(s)</title>
1725
1726 <para>
1727 Two kinds of mirrors exist: pre-mirrors and regular mirrors.
1728 The
1729 <ulink url='&YOCTO_DOCS_REF_URL;#var-PREMIRRORS'><filename>PREMIRRORS</filename></ulink>
1730 and
1731 <ulink url='&YOCTO_DOCS_REF_URL;#var-MIRRORS'><filename>MIRRORS</filename></ulink>
1732 variables point to these, respectively.
1733 BitBake checks pre-mirrors before looking upstream for any
1734 source files.
1735 Pre-mirrors are appropriate when you have a shared directory
1736 that is not a directory defined by the
1737 <ulink url='&YOCTO_DOCS_REF_URL;#var-DL_DIR'><filename>DL_DIR</filename></ulink>
1738 variable.
1739 A Pre-mirror typically points to a shared directory that is
1740 local to your organization.
1741 </para>
1742
1743 <para>
1744 Regular mirrors can be any site across the Internet that is
1745 used as an alternative location for source code should the
1746 primary site not be functioning for some reason or another.
1747 </para>
1748 </section>
1749 </section>
1750
1751 <section id="package-feeds-dev-environment">
1752 <title>Package Feeds</title>
1753
1754 <para>
1755 When the OpenEmbedded build system generates an image or an SDK,
1756 it gets the packages from a package feed area located in the
1757 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
1758 The
1759 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
1760 shows this package feeds area in the upper-right corner.
1761 </para>
1762
1763 <para>
1764 This section looks a little closer into the package feeds area used
1765 by the build system.
1766 Here is a more detailed look at the area:
1767 <imagedata fileref="figures/package-feeds.png" align="center" width="7in" depth="6in" />
1768 </para>
1769
1770 <para>
1771 Package feeds are an intermediary step in the build process.
1772 The OpenEmbedded build system provides classes to generate
1773 different package types, and you specify which classes to enable
1774 through the
1775 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>
1776 variable.
1777 Before placing the packages into package feeds,
1778 the build process validates them with generated output quality
1779 assurance checks through the
1780 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-insane'><filename>insane</filename></ulink>
1781 class.
1782 </para>
1783
1784 <para>
1785 The package feed area resides in the Build Directory.
1786 The directory the build system uses to temporarily store packages
1787 is determined by a combination of variables and the particular
1788 package manager in use.
1789 See the "Package Feeds" box in the illustration and note the
1790 information to the right of that area.
1791 In particular, the following defines where package files are
1792 kept:
1793 <itemizedlist>
1794 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
1795 Defined as <filename>tmp/deploy</filename> in the Build
1796 Directory.
1797 </para></listitem>
1798 <listitem><para><filename>DEPLOY_DIR_*</filename>:
1799 Depending on the package manager used, the package type
1800 sub-folder.
1801 Given RPM, IPK, or DEB packaging and tarball creation, the
1802 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_RPM'><filename>DEPLOY_DIR_RPM</filename></ulink>,
1803 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IPK'><filename>DEPLOY_DIR_IPK</filename></ulink>,
1804 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_DEB'><filename>DEPLOY_DIR_DEB</filename></ulink>,
1805 or
1806 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_TAR'><filename>DEPLOY_DIR_TAR</filename></ulink>,
1807 variables are used, respectively.
1808 </para></listitem>
1809 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink>:
1810 Defines architecture-specific sub-folders.
1811 For example, packages could exist for the i586 or qemux86
1812 architectures.
1813 </para></listitem>
1814 </itemizedlist>
1815 </para>
1816
1817 <para>
1818 BitBake uses the <filename>do_package_write_*</filename> tasks to
1819 generate packages and place them into the package holding area (e.g.
1820 <filename>do_package_write_ipk</filename> for IPK packages).
1821 See the
1822 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_deb'><filename>do_package_write_deb</filename></ulink>",
1823 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>",
1824 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_rpm'><filename>do_package_write_rpm</filename></ulink>",
1825 and
1826 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_tar'><filename>do_package_write_tar</filename></ulink>"
1827 sections for additional information.
1828 As an example, consider a scenario where an IPK packaging manager
1829 is being used and package architecture support for both i586
1830 and qemux86 exist.
1831 Packages for the i586 architecture are placed in
1832 <filename>build/tmp/deploy/ipk/i586</filename>, while packages for
1833 the qemux86 architecture are placed in
1834 <filename>build/tmp/deploy/ipk/qemux86</filename>.
1835 </para>
1836 </section>
1837
1838 <section id='bitbake-dev-environment'>
1839 <title>BitBake</title>
1840
1841 <para>
1842 The OpenEmbedded build system uses
1843 <ulink url='&YOCTO_DOCS_REF_URL;#bitbake-term'>BitBake</ulink>
1844 to produce images.
1845 You can see from the
1846 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
1847 the BitBake area consists of several functional areas.
1848 This section takes a closer look at each of those areas.
1849 </para>
1850
1851 <para>
1852 Separate documentation exists for the BitBake tool.
1853 See the
1854 <ulink url='&YOCTO_DOCS_BB_URL;#bitbake-user-manual'>BitBake User Manual</ulink>
1855 for reference material on BitBake.
1856 </para>
1857
1858 <section id='source-fetching-dev-environment'>
1859 <title>Source Fetching</title>
1860
1861 <para>
1862 The first stages of building a recipe are to fetch and unpack
1863 the source code:
1864 <imagedata fileref="figures/source-fetching.png" align="center" width="6.5in" depth="5in" />
1865 </para>
1866
1867 <para>
1868 The
1869 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-fetch'><filename>do_fetch</filename></ulink>
1870 and
1871 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
1872 tasks fetch the source files and unpack them into the work
1873 directory.
1874 <note>
1875 For every local file (e.g. <filename>file://</filename>)
1876 that is part of a recipe's
1877 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1878 statement, the OpenEmbedded build system takes a checksum
1879 of the file for the recipe and inserts the checksum into
1880 the signature for the <filename>do_fetch</filename>.
1881 If any local file has been modified, the
1882 <filename>do_fetch</filename> task and all tasks that
1883 depend on it are re-executed.
1884 </note>
1885 By default, everything is accomplished in the
1886 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>,
1887 which has a defined structure.
1888 For additional general information on the Build Directory,
1889 see the
1890 "<ulink url='&YOCTO_DOCS_REF_URL;#structure-core-build'><filename>build/</filename></ulink>"
1891 section in the Yocto Project Reference Manual.
1892 </para>
1893
1894 <para>
1895 Unpacked source files are pointed to by the
1896 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1897 variable.
1898 Each recipe has an area in the Build Directory where the
1899 unpacked source code resides.
1900 The name of that directory for any given recipe is defined from
1901 several different variables.
1902 You can see the variables that define these directories
1903 by looking at the figure:
1904 <itemizedlist>
1905 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink> -
1906 The base directory where the OpenEmbedded build system
1907 performs all its work during the build.
1908 </para></listitem>
1909 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCH'><filename>PACKAGE_ARCH</filename></ulink> -
1910 The architecture of the built package or packages.
1911 </para></listitem>
1912 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-TARGET_OS'><filename>TARGET_OS</filename></ulink> -
1913 The operating system of the target device.
1914 </para></listitem>
1915 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PN'><filename>PN</filename></ulink> -
1916 The name of the built package.
1917 </para></listitem>
1918 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PV'><filename>PV</filename></ulink> -
1919 The version of the recipe used to build the package.
1920 </para></listitem>
1921 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> -
1922 The revision of the recipe used to build the package.
1923 </para></listitem>
1924 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> -
1925 The location within <filename>TMPDIR</filename> where
1926 a specific package is built.
1927 </para></listitem>
1928 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink> -
1929 Contains the unpacked source files for a given recipe.
1930 </para></listitem>
1931 </itemizedlist>
1932 </para>
1933 </section>
1934
1935 <section id='patching-dev-environment'>
1936 <title>Patching</title>
1937
1938 <para>
1939 Once source code is fetched and unpacked, BitBake locates
1940 patch files and applies them to the source files:
1941 <imagedata fileref="figures/patching.png" align="center" width="6in" depth="5in" />
1942 </para>
1943
1944 <para>
1945 The
1946 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
1947 task processes recipes by
1948 using the
1949 <ulink url='&YOCTO_DOCS_REF_URL;#var-SRC_URI'><filename>SRC_URI</filename></ulink>
1950 variable to locate applicable patch files, which by default
1951 are <filename>*.patch</filename> or
1952 <filename>*.diff</filename> files, or any file if
1953 "apply=yes" is specified for the file in
1954 <filename>SRC_URI</filename>.
1955 </para>
1956
1957 <para>
1958 BitBake finds and applies multiple patches for a single recipe
1959 in the order in which it finds the patches.
1960 Patches are applied to the recipe's source files located in the
1961 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
1962 directory.
1963 </para>
1964
1965 <para>
1966 For more information on how the source directories are
1967 created, see the
1968 "<link linkend='source-fetching-dev-environment'>Source Fetching</link>"
1969 section.
1970 </para>
1971 </section>
1972
1973 <section id='configuration-and-compilation-dev-environment'>
1974 <title>Configuration and Compilation</title>
1975
1976 <para>
1977 After source code is patched, BitBake executes tasks that
1978 configure and compile the source code:
1979 <imagedata fileref="figures/configuration-compile-autoreconf.png" align="center" width="7in" depth="5in" />
1980 </para>
1981
1982 <para>
1983 This step in the build process consists of three tasks:
1984 <itemizedlist>
1985 <listitem><para>
1986 <emphasis><ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-prepare_recipe_sysroot'><filename>do_prepare_recipe_sysroot</filename></ulink>:</emphasis>
1987 This task sets up the two sysroots in
1988 <filename>${</filename><ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink><filename>}</filename>
1989 (i.e. <filename>recipe-sysroot</filename> and
1990 <filename>recipe-sysroot-native</filename>) so that
1991 the sysroots contain the contents of the
1992 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>
1993 tasks of the recipes on which the recipe
1994 containing the tasks depends.
1995 A sysroot exists for both the target and for the native
1996 binaries, which run on the host system.
1997 </para></listitem>
1998 <listitem><para><emphasis><filename>do_configure</filename>:</emphasis>
1999 This task configures the source by enabling and
2000 disabling any build-time and configuration options for
2001 the software being built.
2002 Configurations can come from the recipe itself as well
2003 as from an inherited class.
2004 Additionally, the software itself might configure itself
2005 depending on the target for which it is being built.
2006 </para>
2007
2008 <para>The configurations handled by the
2009 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-configure'><filename>do_configure</filename></ulink>
2010 task are specific
2011 to source code configuration for the source code
2012 being built by the recipe.</para>
2013
2014 <para>If you are using the
2015 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-autotools'><filename>autotools</filename></ulink>
2016 class,
2017 you can add additional configuration options by using
2018 the
2019 <ulink url='&YOCTO_DOCS_REF_URL;#var-EXTRA_OECONF'><filename>EXTRA_OECONF</filename></ulink>
2020 or
2021 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGECONFIG_CONFARGS'><filename>PACKAGECONFIG_CONFARGS</filename></ulink>
2022 variables.
2023 For information on how this variable works within
2024 that class, see the
2025 <filename>meta/classes/autotools.bbclass</filename> file.
2026 </para></listitem>
2027 <listitem><para><emphasis><filename>do_compile</filename>:</emphasis>
2028 Once a configuration task has been satisfied, BitBake
2029 compiles the source using the
2030 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-compile'><filename>do_compile</filename></ulink>
2031 task.
2032 Compilation occurs in the directory pointed to by the
2033 <ulink url='&YOCTO_DOCS_REF_URL;#var-B'><filename>B</filename></ulink>
2034 variable.
2035 Realize that the <filename>B</filename> directory is, by
2036 default, the same as the
2037 <ulink url='&YOCTO_DOCS_REF_URL;#var-S'><filename>S</filename></ulink>
2038 directory.</para></listitem>
2039 <listitem><para><emphasis><filename>do_install</filename>:</emphasis>
2040 Once compilation is done, BitBake executes the
2041 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
2042 task.
2043 This task copies files from the <filename>B</filename>
2044 directory and places them in a holding area pointed to
2045 by the
2046 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink>
2047 variable.</para></listitem>
2048 </itemizedlist>
2049 </para>
2050 </section>
2051
2052 <section id='package-splitting-dev-environment'>
2053 <title>Package Splitting</title>
2054
2055 <para>
2056 After source code is configured and compiled, the
2057 OpenEmbedded build system analyzes
2058 the results and splits the output into packages:
2059 <imagedata fileref="figures/analysis-for-package-splitting.png" align="center" width="7in" depth="7in" />
2060 </para>
2061
2062 <para>
2063 The
2064 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
2065 and
2066 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>
2067 tasks combine to analyze
2068 the files found in the
2069 <ulink url='&YOCTO_DOCS_REF_URL;#var-D'><filename>D</filename></ulink> directory
2070 and split them into subsets based on available packages and
2071 files.
2072 The analyzing process involves the following as well as other
2073 items: splitting out debugging symbols,
2074 looking at shared library dependencies between packages,
2075 and looking at package relationships.
2076 The <filename>do_packagedata</filename> task creates package
2077 metadata based on the analysis such that the
2078 OpenEmbedded build system can generate the final packages.
2079 Working, staged, and intermediate results of the analysis
2080 and package splitting process use these areas:
2081 <itemizedlist>
2082 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PKGD'><filename>PKGD</filename></ulink> -
2083 The destination directory for packages before they are
2084 split.
2085 </para></listitem>
2086 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDATA_DIR'><filename>PKGDATA_DIR</filename></ulink> -
2087 A shared, global-state directory that holds data
2088 generated during the packaging process.
2089 </para></listitem>
2090 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDESTWORK'><filename>PKGDESTWORK</filename></ulink> -
2091 A temporary work area used by the
2092 <filename>do_package</filename> task.
2093 </para></listitem>
2094 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PKGDEST'><filename>PKGDEST</filename></ulink> -
2095 The parent directory for packages after they have
2096 been split.
2097 </para></listitem>
2098 </itemizedlist>
2099 The <ulink url='&YOCTO_DOCS_REF_URL;#var-FILES'><filename>FILES</filename></ulink>
2100 variable defines the files that go into each package in
2101 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGES'><filename>PACKAGES</filename></ulink>.
2102 If you want details on how this is accomplished, you can
2103 look at the
2104 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-package'><filename>package</filename></ulink>
2105 class.
2106 </para>
2107
2108 <para>
2109 Depending on the type of packages being created (RPM, DEB, or
2110 IPK), the <filename>do_package_write_*</filename> task
2111 creates the actual packages and places them in the
2112 Package Feed area, which is
2113 <filename>${TMPDIR}/deploy</filename>.
2114 You can see the
2115 "<link linkend='package-feeds-dev-environment'>Package Feeds</link>"
2116 section for more detail on that part of the build process.
2117 <note>
2118 Support for creating feeds directly from the
2119 <filename>deploy/*</filename> directories does not exist.
2120 Creating such feeds usually requires some kind of feed
2121 maintenance mechanism that would upload the new packages
2122 into an official package feed (e.g. the
2123 Ångström distribution).
2124 This functionality is highly distribution-specific
2125 and thus is not provided out of the box.
2126 </note>
2127 </para>
2128 </section>
2129
2130 <section id='image-generation-dev-environment'>
2131 <title>Image Generation</title>
2132
2133 <para>
2134 Once packages are split and stored in the Package Feeds area,
2135 the OpenEmbedded build system uses BitBake to generate the
2136 root filesystem image:
2137 <imagedata fileref="figures/image-generation.png" align="center" width="6in" depth="7in" />
2138 </para>
2139
2140 <para>
2141 The image generation process consists of several stages and
2142 depends on several tasks and variables.
2143 The
2144 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-rootfs'><filename>do_rootfs</filename></ulink>
2145 task creates the root filesystem (file and directory structure)
2146 for an image.
2147 This task uses several key variables to help create the list
2148 of packages to actually install:
2149 <itemizedlist>
2150 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_INSTALL'><filename>IMAGE_INSTALL</filename></ulink>:
2151 Lists out the base set of packages to install from
2152 the Package Feeds area.</para></listitem>
2153 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_EXCLUDE'><filename>PACKAGE_EXCLUDE</filename></ulink>:
2154 Specifies packages that should not be installed.
2155 </para></listitem>
2156 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FEATURES'><filename>IMAGE_FEATURES</filename></ulink>:
2157 Specifies features to include in the image.
2158 Most of these features map to additional packages for
2159 installation.</para></listitem>
2160 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_CLASSES'><filename>PACKAGE_CLASSES</filename></ulink>:
2161 Specifies the package backend to use and consequently
2162 helps determine where to locate packages within the
2163 Package Feeds area.</para></listitem>
2164 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_LINGUAS'><filename>IMAGE_LINGUAS</filename></ulink>:
2165 Determines the language(s) for which additional
2166 language support packages are installed.
2167 </para></listitem>
2168 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_INSTALL'><filename>PACKAGE_INSTALL</filename></ulink>:
2169 The final list of packages passed to the package manager
2170 for installation into the image.
2171 </para></listitem>
2172 </itemizedlist>
2173 </para>
2174
2175 <para>
2176 With
2177 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_ROOTFS'><filename>IMAGE_ROOTFS</filename></ulink>
2178 pointing to the location of the filesystem under construction and
2179 the <filename>PACKAGE_INSTALL</filename> variable providing the
2180 final list of packages to install, the root file system is
2181 created.
2182 </para>
2183
2184 <para>
2185 Package installation is under control of the package manager
2186 (e.g. dnf/rpm, opkg, or apt/dpkg) regardless of whether or
2187 not package management is enabled for the target.
2188 At the end of the process, if package management is not
2189 enabled for the target, the package manager's data files
2190 are deleted from the root filesystem.
2191 As part of the final stage of package installation, postinstall
2192 scripts that are part of the packages are run.
2193 Any scripts that fail to run
2194 on the build host are run on the target when the target system
2195 is first booted.
2196 If you are using a
2197 <ulink url='&YOCTO_DOCS_DEV_URL;#creating-a-read-only-root-filesystem'>read-only root filesystem</ulink>,
2198 all the post installation scripts must succeed during the
2199 package installation phase since the root filesystem is
2200 read-only.
2201 </para>
2202
2203 <para>
2204 The final stages of the <filename>do_rootfs</filename> task
2205 handle post processing.
2206 Post processing includes creation of a manifest file and
2207 optimizations.
2208 </para>
2209
2210 <para>
2211 The manifest file (<filename>.manifest</filename>) resides
2212 in the same directory as the root filesystem image.
2213 This file lists out, line-by-line, the installed packages.
2214 The manifest file is useful for the
2215 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage*'><filename>testimage</filename></ulink>
2216 class, for example, to determine whether or not to run
2217 specific tests.
2218 See the
2219 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_MANIFEST'><filename>IMAGE_MANIFEST</filename></ulink>
2220 variable for additional information.
2221 </para>
2222
2223 <para>
2224 Optimizing processes run across the image include
2225 <filename>mklibs</filename>, <filename>prelink</filename>,
2226 and any other post-processing commands as defined by the
2227 <ulink url='&YOCTO_DOCS_REF_URL;#var-ROOTFS_POSTPROCESS_COMMAND'><filename>ROOTFS_POSTPROCESS_COMMAND</filename></ulink>
2228 variable.
2229 The <filename>mklibs</filename> process optimizes the size
2230 of the libraries, while the
2231 <filename>prelink</filename> process optimizes the dynamic
2232 linking of shared libraries to reduce start up time of
2233 executables.
2234 </para>
2235
2236 <para>
2237 After the root filesystem is built, processing begins on
2238 the image through the
2239 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image'><filename>do_image</filename></ulink>
2240 task.
2241 The build system runs any pre-processing commands as defined
2242 by the
2243 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_PREPROCESS_COMMAND'><filename>IMAGE_PREPROCESS_COMMAND</filename></ulink>
2244 variable.
2245 This variable specifies a list of functions to call before
2246 the OpenEmbedded build system creates the final image output
2247 files.
2248 </para>
2249
2250 <para>
2251 The OpenEmbedded build system dynamically creates
2252 <filename>do_image_*</filename> tasks as needed, based
2253 on the image types specified in the
2254 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
2255 variable.
2256 The process turns everything into an image file or a set of
2257 image files and compresses the root filesystem image to reduce
2258 the overall size of the image.
2259 The formats used for the root filesystem depend on the
2260 <filename>IMAGE_FSTYPES</filename> variable.
2261 </para>
2262
2263 <para>
2264 As an example, a dynamically created task when creating a
2265 particular image <replaceable>type</replaceable> would take the
2266 following form:
2267 <literallayout class='monospaced'>
2268 do_image_<replaceable>type</replaceable>[depends]
2269 </literallayout>
2270 So, if the <replaceable>type</replaceable> as specified by the
2271 <filename>IMAGE_FSTYPES</filename> were
2272 <filename>ext4</filename>, the dynamically generated task
2273 would be as follows:
2274 <literallayout class='monospaced'>
2275 do_image_ext4[depends]
2276 </literallayout>
2277 </para>
2278
2279 <para>
2280 The final task involved in image creation is the
2281 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-image-complete'><filename>do_image_complete</filename></ulink>
2282 task.
2283 This task completes the image by applying any image
2284 post processing as defined through the
2285 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_POSTPROCESS_COMMAND'><filename>IMAGE_POSTPROCESS_COMMAND</filename></ulink>
2286 variable.
2287 The variable specifies a list of functions to call once the
2288 OpenEmbedded build system has created the final image output
2289 files.
2290 </para>
2291
2292 <note>
2293 The entire image generation process is run under Pseudo.
2294 Running under Pseudo ensures that the files in the root
2295 filesystem have correct ownership.
2296 </note>
2297 </section>
2298
2299 <section id='sdk-generation-dev-environment'>
2300 <title>SDK Generation</title>
2301
2302 <para>
2303 The OpenEmbedded build system uses BitBake to generate the
2304 Software Development Kit (SDK) installer script for both the
2305 standard and extensible SDKs:
2306 <imagedata fileref="figures/sdk-generation.png" align="center" />
2307 </para>
2308
2309 <note>
2310 For more information on the cross-development toolchain
2311 generation, see the
2312 "<ulink url='&YOCTO_DOCS_REF_URL;#cross-development-toolchain-generation'>Cross-Development Toolchain Generation</ulink>"
2313 section in the Yocto Project Reference Manual.
2314 For information on advantages gained when building a
2315 cross-development toolchain using the
2316 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sdk'><filename>do_populate_sdk</filename></ulink>
2317 task, see the
2318 "<ulink url='&YOCTO_DOCS_SDK_URL;#sdk-building-an-sdk-installer'>Building an SDK Installer</ulink>"
2319 section in the Yocto Project Application Development and the
2320 Extensible Software Development Kit (SDK) manual.
2321 </note>
2322
2323 <para>
2324 Like image generation, the SDK script process consists of
2325 several stages and depends on many variables.
2326 The <filename>do_populate_sdk</filename> and
2327 <filename>do_populate_sdk_ext</filename> tasks use these
2328 key variables to help create the list of packages to actually
2329 install.
2330 For information on the variables listed in the figure, see the
2331 "<link linkend='sdk-dev-environment'>Application Development SDK</link>"
2332 section.
2333 </para>
2334
2335 <para>
2336 The <filename>do_populate_sdk</filename> task helps create
2337 the standard SDK and handles two parts: a target part and a
2338 host part.
2339 The target part is the part built for the target hardware and
2340 includes libraries and headers.
2341 The host part is the part of the SDK that runs on the
2342 <ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>.
2343 </para>
2344
2345 <para>
2346 The <filename>do_populate_sdk_ext</filename> task helps create
2347 the extensible SDK and handles host and target parts
2348 differently than its counter part does for the standard SDK.
2349 For the extensible SDK, the task encapsulates the build system,
2350 which includes everything needed (host and target) for the SDK.
2351 </para>
2352
2353 <para>
2354 Regardless of the type of SDK being constructed, the
2355 tasks perform some cleanup after which a cross-development
2356 environment setup script and any needed configuration files
2357 are created.
2358 The final output is the Cross-development
2359 toolchain installation script (<filename>.sh</filename> file),
2360 which includes the environment setup script.
2361 </para>
2362 </section>
2363
2364 <section id='stamp-files-and-the-rerunning-of-tasks'>
2365 <title>Stamp Files and the Rerunning of Tasks</title>
2366
2367 <para>
2368 For each task that completes successfully, BitBake writes a
2369 stamp file into the
2370 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMPS_DIR'><filename>STAMPS_DIR</filename></ulink>
2371 directory.
2372 The beginning of the stamp file's filename is determined by the
2373 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
2374 variable, and the end of the name consists of the task's name
2375 and current
2376 <ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksum</ulink>.
2377 <note>
2378 This naming scheme assumes that
2379 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
2380 is "OEBasicHash", which is almost always the case in
2381 current OpenEmbedded.
2382 </note>
2383 To determine if a task needs to be rerun, BitBake checks if a
2384 stamp file with a matching input checksum exists for the task.
2385 If such a stamp file exists, the task's output is assumed to
2386 exist and still be valid.
2387 If the file does not exist, the task is rerun.
2388 <note>
2389 <para>The stamp mechanism is more general than the shared
2390 state (sstate) cache mechanism described in the
2391 "<link linkend='setscene-tasks-and-shared-state'>Setscene Tasks and Shared State</link>"
2392 section.
2393 BitBake avoids rerunning any task that has a valid
2394 stamp file, not just tasks that can be accelerated through
2395 the sstate cache.</para>
2396 <para>However, you should realize that stamp files only
2397 serve as a marker that some work has been done and that
2398 these files do not record task output.
2399 The actual task output would usually be somewhere in
2400 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>
2401 (e.g. in some recipe's
2402 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.)
2403 What the sstate cache mechanism adds is a way to cache task
2404 output that can then be shared between build machines.
2405 </para>
2406 </note>
2407 Since <filename>STAMPS_DIR</filename> is usually a subdirectory
2408 of <filename>TMPDIR</filename>, removing
2409 <filename>TMPDIR</filename> will also remove
2410 <filename>STAMPS_DIR</filename>, which means tasks will
2411 properly be rerun to repopulate <filename>TMPDIR</filename>.
2412 </para>
2413
2414 <para>
2415 If you want some task to always be considered "out of date",
2416 you can mark it with the
2417 <ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'><filename>nostamp</filename></ulink>
2418 varflag.
2419 If some other task depends on such a task, then that task will
2420 also always be considered out of date, which might not be what
2421 you want.
2422 </para>
2423
2424 <para>
2425 For details on how to view information about a task's
2426 signature, see the
2427 "<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
2428 section in the Yocto Project Reference Manual.
2429 </para>
2430 </section>
2431
2432 <section id='setscene-tasks-and-shared-state'>
2433 <title>Setscene Tasks and Shared State</title>
2434
2435 <para>
2436 The description of tasks so far assumes that BitBake needs to
2437 build everything and there are no prebuilt objects available.
2438 BitBake does support skipping tasks if prebuilt objects are
2439 available.
2440 These objects are usually made available in the form of a
2441 shared state (sstate) cache.
2442 <note>
2443 For information on variables affecting sstate, see the
2444 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
2445 and
2446 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
2447 variables.
2448 </note>
2449 </para>
2450
2451 <para>
2452 The idea of a setscene task (i.e
2453 <filename>do_</filename><replaceable>taskname</replaceable><filename>_setscene</filename>)
2454 is a version of the task where
2455 instead of building something, BitBake can skip to the end
2456 result and simply place a set of files into specific locations
2457 as needed.
2458 In some cases, it makes sense to have a setscene task variant
2459 (e.g. generating package files in the
2460 <filename>do_package_write_*</filename> task).
2461 In other cases, it does not make sense, (e.g. a
2462 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-patch'><filename>do_patch</filename></ulink>
2463 task or
2464 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-unpack'><filename>do_unpack</filename></ulink>
2465 task) since the work involved would be equal to or greater than
2466 the underlying task.
2467 </para>
2468
2469 <para>
2470 In the OpenEmbedded build system, the common tasks that have
2471 setscene variants are
2472 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>,
2473 <filename>do_package_write_*</filename>,
2474 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>,
2475 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-packagedata'><filename>do_packagedata</filename></ulink>,
2476 and
2477 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-populate_sysroot'><filename>do_populate_sysroot</filename></ulink>.
2478 Notice that these are most of the tasks whose output is an
2479 end result.
2480 </para>
2481
2482 <para>
2483 The OpenEmbedded build system has knowledge of the relationship
2484 between these tasks and other tasks that precede them.
2485 For example, if BitBake runs
2486 <filename>do_populate_sysroot_setscene</filename> for
2487 something, there is little point in running any of the
2488 <filename>do_fetch</filename>, <filename>do_unpack</filename>,
2489 <filename>do_patch</filename>,
2490 <filename>do_configure</filename>,
2491 <filename>do_compile</filename>, and
2492 <filename>do_install</filename> tasks.
2493 However, if <filename>do_package</filename> needs to be run,
2494 BitBake would need to run those other tasks.
2495 </para>
2496
2497 <para>
2498 It becomes more complicated if everything can come from an
2499 sstate cache because some objects are simply not required at
2500 all.
2501 For example, you do not need a compiler or native tools, such
2502 as quilt, if there is nothing to compile or patch.
2503 If the <filename>do_package_write_*</filename> packages are
2504 available from sstate, BitBake does not need the
2505 <filename>do_package</filename> task data.
2506 </para>
2507
2508 <para>
2509 To handle all these complexities, BitBake runs in two phases.
2510 The first is the "setscene" stage.
2511 During this stage, BitBake first checks the sstate cache for
2512 any targets it is planning to build.
2513 BitBake does a fast check to see if the object exists rather
2514 than a complete download.
2515 If nothing exists, the second phase, which is the setscene
2516 stage, completes and the main build proceeds.
2517 </para>
2518
2519 <para>
2520 If objects are found in the sstate cache, the OpenEmbedded
2521 build system works backwards from the end targets specified
2522 by the user.
2523 For example, if an image is being built, the OpenEmbedded build
2524 system first looks for the packages needed for that image and
2525 the tools needed to construct an image.
2526 If those are available, the compiler is not needed.
2527 Thus, the compiler is not even downloaded.
2528 If something was found to be unavailable, or the download or
2529 setscene task fails, the OpenEmbedded build system then tries
2530 to install dependencies, such as the compiler, from the cache.
2531 </para>
2532
2533 <para>
2534 The availability of objects in the sstate cache is handled by
2535 the function specified by the
2536 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_HASHCHECK_FUNCTION'><filename>BB_HASHCHECK_FUNCTION</filename></ulink>
2537 variable and returns a list of the objects that are available.
2538 The function specified by the
2539 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SETSCENE_DEPVALID'><filename>BB_SETSCENE_DEPVALID</filename></ulink>
2540 variable is the function that determines whether a given
2541 dependency needs to be followed, and whether for any given
2542 relationship the function needs to be passed.
2543 The function returns a True or False value.
2544 </para>
2545 </section>
2546 </section>
2547
2548 <section id='images-dev-environment'>
2549 <title>Images</title>
2550
2551 <para>
2552 The images produced by the OpenEmbedded build system
2553 are compressed forms of the
2554 root filesystem that are ready to boot on a target device.
2555 You can see from the
2556 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>
2557 that BitBake output, in part, consists of images.
2558 This section is going to look more closely at this output:
2559 <imagedata fileref="figures/images.png" align="center" width="5.5in" depth="5.5in" />
2560 </para>
2561
2562 <para>
2563 For a list of example images that the Yocto Project provides,
2564 see the
2565 "<ulink url='&YOCTO_DOCS_REF_URL;#ref-images'>Images</ulink>"
2566 chapter in the Yocto Project Reference Manual.
2567 </para>
2568
2569 <para>
2570 Images are written out to the
2571 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
2572 inside the
2573 <filename>tmp/deploy/images/<replaceable>machine</replaceable>/</filename>
2574 folder as shown in the figure.
2575 This folder contains any files expected to be loaded on the
2576 target device.
2577 The
2578 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>
2579 variable points to the <filename>deploy</filename> directory,
2580 while the
2581 <ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR_IMAGE'><filename>DEPLOY_DIR_IMAGE</filename></ulink>
2582 variable points to the appropriate directory containing images for
2583 the current configuration.
2584 <itemizedlist>
2585 <listitem><para><filename><replaceable>kernel-image</replaceable></filename>:
2586 A kernel binary file.
2587 The
2588 <ulink url='&YOCTO_DOCS_REF_URL;#var-KERNEL_IMAGETYPE'><filename>KERNEL_IMAGETYPE</filename></ulink>
2589 variable setting determines the naming scheme for the
2590 kernel image file.
2591 Depending on that variable, the file could begin with
2592 a variety of naming strings.
2593 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
2594 directory can contain multiple image files for the
2595 machine.</para></listitem>
2596 <listitem><para><filename><replaceable>root-filesystem-image</replaceable></filename>:
2597 Root filesystems for the target device (e.g.
2598 <filename>*.ext3</filename> or <filename>*.bz2</filename>
2599 files).
2600 The
2601 <ulink url='&YOCTO_DOCS_REF_URL;#var-IMAGE_FSTYPES'><filename>IMAGE_FSTYPES</filename></ulink>
2602 variable setting determines the root filesystem image
2603 type.
2604 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
2605 directory can contain multiple root filesystems for the
2606 machine.</para></listitem>
2607 <listitem><para><filename><replaceable>kernel-modules</replaceable></filename>:
2608 Tarballs that contain all the modules built for the kernel.
2609 Kernel module tarballs exist for legacy purposes and
2610 can be suppressed by setting the
2611 <ulink url='&YOCTO_DOCS_REF_URL;#var-MODULE_TARBALL_DEPLOY'><filename>MODULE_TARBALL_DEPLOY</filename></ulink>
2612 variable to "0".
2613 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
2614 directory can contain multiple kernel module tarballs
2615 for the machine.</para></listitem>
2616 <listitem><para><filename><replaceable>bootloaders</replaceable></filename>:
2617 Bootloaders supporting the image, if applicable to the
2618 target machine.
2619 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
2620 directory can contain multiple bootloaders for the
2621 machine.</para></listitem>
2622 <listitem><para><filename><replaceable>symlinks</replaceable></filename>:
2623 The <filename>deploy/images/<replaceable>machine</replaceable></filename>
2624 folder contains
2625 a symbolic link that points to the most recently built file
2626 for each machine.
2627 These links might be useful for external scripts that
2628 need to obtain the latest version of each file.
2629 </para></listitem>
2630 </itemizedlist>
2631 </para>
2632 </section>
2633
2634 <section id='sdk-dev-environment'>
2635 <title>Application Development SDK</title>
2636
2637 <para>
2638 In the
2639 <link linkend='general-yocto-environment-figure'>general Yocto Project Development Environment figure</link>,
2640 the output labeled "Application Development SDK" represents an
2641 SDK.
2642 The SDK generation process differs depending on whether you build
2643 a standard SDK
2644 (e.g. <filename>bitbake -c populate_sdk</filename> <replaceable>imagename</replaceable>)
2645 or an extensible SDK
2646 (e.g. <filename>bitbake -c populate_sdk_ext</filename> <replaceable>imagename</replaceable>).
2647 This section is going to take a closer look at this output:
2648 <imagedata fileref="figures/sdk.png" align="center" width="9in" depth="7.25in" />
2649 </para>
2650
2651 <para>
2652 The specific form of this output is a self-extracting
2653 SDK installer (<filename>*.sh</filename>) that, when run,
2654 installs the SDK, which consists of a cross-development
2655 toolchain, a set of libraries and headers, and an SDK
2656 environment setup script.
2657 Running this installer essentially sets up your
2658 cross-development environment.
2659 You can think of the cross-toolchain as the "host"
2660 part because it runs on the SDK machine.
2661 You can think of the libraries and headers as the "target"
2662 part because they are built for the target hardware.
2663 The environment setup script is added so that you can initialize
2664 the environment before using the tools.
2665 </para>
2666
2667 <note><title>Notes</title>
2668 <itemizedlist>
2669 <listitem><para>
2670 The Yocto Project supports several methods by which you can
2671 set up this cross-development environment.
2672 These methods include downloading pre-built SDK installers
2673 or building and installing your own SDK installer.
2674 </para></listitem>
2675 <listitem><para>
2676 For background information on cross-development toolchains
2677 in the Yocto Project development environment, see the
2678 "<ulink url='&YOCTO_DOCS_REF_URL;#cross-development-toolchain-generation'>Cross-Development Toolchain Generation</ulink>"
2679 section in the Yocto Project Reference Manual.
2680 </para></listitem>
2681 <listitem><para>
2682 For information on setting up a cross-development
2683 environment, see the
2684 <ulink url='&YOCTO_DOCS_SDK_URL;'>Yocto Project Application Development and the Extensible Software Development Kit (eSDK)</ulink>
2685 manual.
2686 </para></listitem>
2687 </itemizedlist>
2688 </note>
2689
2690 <para>
2691 Once built, the SDK installers are written out to the
2692 <filename>deploy/sdk</filename> folder inside the
2693 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>
2694 as shown in the figure at the beginning of this section.
2695 Depending on the type of SDK, several variables exist that help
2696 configure these files.
2697 The following list shows the variables associated with a standard
2698 SDK:
2699 <itemizedlist>
2700 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
2701 Points to the <filename>deploy</filename>
2702 directory.</para></listitem>
2703 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-SDKMACHINE'><filename>SDKMACHINE</filename></ulink>:
2704 Specifies the architecture of the machine
2705 on which the cross-development tools are run to
2706 create packages for the target hardware.
2707 </para></listitem>
2708 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-SDKIMAGE_FEATURES'><filename>SDKIMAGE_FEATURES</filename></ulink>:
2709 Lists the features to include in the "target" part
2710 of the SDK.
2711 </para></listitem>
2712 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_HOST_TASK'><filename>TOOLCHAIN_HOST_TASK</filename></ulink>:
2713 Lists packages that make up the host
2714 part of the SDK (i.e. the part that runs on
2715 the <filename>SDKMACHINE</filename>).
2716 When you use
2717 <filename>bitbake -c populate_sdk <replaceable>imagename</replaceable></filename>
2718 to create the SDK, a set of default packages
2719 apply.
2720 This variable allows you to add more packages.
2721 </para></listitem>
2722 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-TOOLCHAIN_TARGET_TASK'><filename>TOOLCHAIN_TARGET_TASK</filename></ulink>:
2723 Lists packages that make up the target part
2724 of the SDK (i.e. the part built for the
2725 target hardware).
2726 </para></listitem>
2727 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-SDKPATH'><filename>SDKPATH</filename></ulink>:
2728 Defines the default SDK installation path offered by the
2729 installation script.
2730 </para></listitem>
2731 </itemizedlist>
2732 This next list, shows the variables associated with an extensible
2733 SDK:
2734 <itemizedlist>
2735 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-DEPLOY_DIR'><filename>DEPLOY_DIR</filename></ulink>:
2736 Points to the <filename>deploy</filename> directory.
2737 </para></listitem>
2738 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_EXT_TYPE'><filename>SDK_EXT_TYPE</filename></ulink>:
2739 Controls whether or not shared state artifacts are copied
2740 into the extensible SDK.
2741 By default, all required shared state artifacts are copied
2742 into the SDK.
2743 </para></listitem>
2744 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_PKGDATA'><filename>SDK_INCLUDE_PKGDATA</filename></ulink>:
2745 Specifies whether or not packagedata will be included in
2746 the extensible SDK for all recipes in the "world" target.
2747 </para></listitem>
2748 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INCLUDE_TOOLCHAIN'><filename>SDK_INCLUDE_TOOLCHAIN</filename></ulink>:
2749 Specifies whether or not the toolchain will be included
2750 when building the extensible SDK.
2751 </para></listitem>
2752 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_WHITELIST'><filename>SDK_LOCAL_CONF_WHITELIST</filename></ulink>:
2753 A list of variables allowed through from the build system
2754 configuration into the extensible SDK configuration.
2755 </para></listitem>
2756 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_LOCAL_CONF_BLACKLIST'><filename>SDK_LOCAL_CONF_BLACKLIST</filename></ulink>:
2757 A list of variables not allowed through from the build
2758 system configuration into the extensible SDK configuration.
2759 </para></listitem>
2760 <listitem><para><ulink url='&YOCTO_DOCS_REF_URL;#var-SDK_INHERIT_BLACKLIST'><filename>SDK_INHERIT_BLACKLIST</filename></ulink>:
2761 A list of classes to remove from the
2762 <ulink url='&YOCTO_DOCS_REF_URL;#var-INHERIT'><filename>INHERIT</filename></ulink>
2763 value globally within the extensible SDK configuration.
2764 </para></listitem>
2765 </itemizedlist>
2766 </para>
2767 </section>
2768</section>
2769
2770</chapter>
2771<!--
2772vim: expandtab tw=80 ts=4
2773-->