1 files changed, 229 insertions, 0 deletions
diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml
new file mode 100644
index 0000000000..8f83ecb3a1
--- /dev/null
+++ b/documentation/poky-ref-manual/technical-details.xml
@@ -0,0 +1,229 @@
+<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+<chapter id='technical-details'>
+<title>Technical Details</title>
+    <para>
+        This chapter provides technical details for various parts of the Yocto Project. 
+        Currently, topics include Yocto Project components and shared state (sstate) cache.
+    </para>
+<section id='usingpoky-components'>
+    <title>Yocto Project Components</title>
+    <para>
+        The BitBake task executor together with various types of configuration files form the 
+        Yocto Project core.
+        This section overviews the BitBake task executor and the
+        configuration files by describing what they are used for and how they interact.
+    </para>
+    
+    <para>  
+        BitBake handles the parsing and execution of the data files. 
+        The data itself is of various types:
+    <itemizedlist>
+        <listitem><para><emphasis>Recipes:</emphasis>  Provides details about particular 
+            pieces of software</para></listitem>
+        <listitem><para><emphasis>Class Data:</emphasis>  An abstraction of common build 
+            information (e.g. how to build a Linux kernel).</para></listitem>
+        <listitem><para><emphasis>Configuration Data:</emphasis>  Defines machine-specific settings, 
+            policy decisions, etc.
+            Configuration data acts as the glue to bind everything together.</para></listitem>
+    </itemizedlist>
+        For more information on data, see the
+        <ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html#yocto-project-terms'>
+        Yocto Project Terms</ulink> section in 
+        <ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html'>
+        The Yocto Project Development Manual</ulink>.
+    </para>
+    <para> 
+        BitBake knows how to combine multiple data sources together and refers to each data source
+        as a <link linkend='usingpoky-changes-layers'>'layer'</link>.
+    </para>
+    <para>
+        Following are some brief details on these core components.
+        For more detailed information on these components see the 
+        <link linkend='ref-structure'>'Reference: Directory Structure'</link>
+        appendix.
+    </para>
+    <section id='usingpoky-components-bitbake'>
+        <title>BitBake</title>
+        <para>
+            BitBake is the tool at the heart of the Yocto Project and is responsible
+            for parsing the metadata, generating a list of tasks from it,
+            and then executing those tasks. 
+            To see a list of the options BitBake supports, use the following help command:
+            <literallayout class='monospaced'>
+     $ bitbake --help
+            </literallayout>
+        </para>
+        <para>
+            The most common usage for BitBake is <filename>bitbake &lt;packagename&gt;</filename>, where
+            <filename>packagename</filename> is the name of the package you want to build 
+            (referred to as the "target" in this manual). 
+            The target often equates to the first part of a <filename>.bb</filename> filename.
+            So, to run the <filename>matchbox-desktop_1.2.3.bb</filename> file, you
+            might type the following:
+            <literallayout class='monospaced'>
+     $ bitbake matchbox-desktop
+            </literallayout>
+            Several different versions of <filename>matchbox-desktop</filename> might exist.
+            BitBake chooses the one selected by the distribution configuration.
+            You can get more details about how BitBake chooses between different 
+            target versions and providers in the 
+            <link linkend='ref-bitbake-providers'>Preferences and Providers</link> section.
+        </para>
+        <para>
+            BitBake also tries to execute any dependent tasks first.
+            So for example, before building <filename>matchbox-desktop</filename>, BitBake
+            would build a cross compiler and <filename>eglibc</filename> if they had not already 
+            been built.
+            <note>This release of the Yocto Project does not support the <filename>glibc</filename>
+                GNU version of the Unix standard C library.  By default, the Yocto Project builds with
+                <filename>eglibc</filename>.</note>
+        </para>
+        <para>
+            A useful BitBake option to consider is the <filename>-k</filename> or 
+            <filename>--continue</filename> option.  
+            This option instructs BitBake to try and continue processing the job as much 
+            as possible even after encountering an error.  
+            When an error occurs, the target that
+            failed and those that depend on it cannot be remade.  
+            However, when you use this option other dependencies can still be processed.
+        </para>
+    </section>
+    <section id='usingpoky-components-metadata'>
+        <title>Metadata (Recipes)</title>
+        <para>
+            The <filename>.bb</filename> files are usually referred to as "recipes." 
+            In general, a recipe contains information about a single piece of software.
+            The information includes the location from which to download the source patches 
+            (if any are needed), which special configuration options to apply, 
+            how to compile the source files, and how to package the compiled output. 
+        </para>
+        <para>
+            The term "package" can also be used to describe recipes.
+            However, since the same word is used for the packaged output from the Yocto 
+            Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files), 
+            this document avoids using the term "package" to refer to recipes.
+        </para>
+    </section>
+    <section id='usingpoky-components-classes'>
+        <title>Classes</title>
+        <para>
+            Class files (<filename>.bbclass</filename>) contain information that is useful to share
+            between metadata files. 
+            An example is the Autotools class, which contains
+            common settings for any application that Autotools uses.
+            The <link linkend='ref-classes'>Reference: Classes</link> appendix provides details
+            about common classes and how to use them.
+        </para>
+    </section>
+    <section id='usingpoky-components-configuration'>
+        <title>Configuration</title>
+        <para>
+            The configuration files (<filename>.conf</filename>) define various configuration variables
+            that govern the Yocto Project build process. 
+            These files fall into several areas that define machine configuration options, 
+            distribution configuration options, compiler tuning options, general common configuration
+            options and user configuration options (<filename>local.conf</filename>, which is found
+            in the Yocto Project files build directory).
+        </para>
+    </section>
+</section>
+<section id="considering-shared-state-cache">
+    <title>Considering Shared State Cache</title>
+    <para>
+        By design, the Yocto Project builds everything from scratch unless it can determine that
+        a given task's inputs have not changed.  
+        While building from scratch ensures that everything is current, it does also
+        mean that a lot of time could be spent rebuiding things that don't necessarily need built.
+    </para>
+    <para>
+        The Yocto Project build process uses a shared state caching scheme to avoid having to 
+        rebuild software when it is not necessary.  
+        Because the build time for a Yocto image can be significant, it is helpful to try and 
+        determine what really needs built and what can be skipped given a particular project's
+        build process.
+    </para>
+    <para>
+        The scheme that the Yocto Project uses involves checksum generation and comparison for
+        a task's inputs.
+        The scheme also employs an area of memory called the shared state cache that is 
+        pointed to by the <filename>SSTATE_DIR</filename> variable.
+        This area contains task output generated from a previous build.  
+        If a given task's checksum matches the checksum of a previous build for the same  
+        task, the build process uses the state of the cache rather than rerunning that 
+        task.
+    </para>
+    <para>
+        The previous paragraph is a simplistic explanation of how the build process 
+        uses checksums and shared state memory cache to avoide building tasks that 
+        don't need built.
+        If you want a bit more explanation on the topic, 
+        see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
+        State - What does it mean and why should I care?</ulink>" from the Yocto 
+        Project discussion archives.
+    </para>
+    <para>
+        As with all schemes, this one has some drawbacks.
+        It is possible that you could make implicit changes that are not factored into the checksum 
+        calculation, but do affect a task's output. 
+        A good example is perhaps when a tool changes its output.
+        Let's say that the output of <filename>rpmdeps</filename> needed to change.
+        The result of the change should be that all the "package", "package_write_rpm",
+        and "package_deploy-rpm" sstate-cache items would become invalid.
+        But, because this is a change that is external to the code and therefore implicit,
+        the associated sstate-cache items do not become invalidated.
+        In this case, the build process would use the cache items rather than running the 
+        task again. 
+        Obviously, these types of implicit changes can cause problems.
+    </para>
+    <para>
+        To avoid these problems during the build, you need to understand the effects of any
+        change you make.
+        Note that any changes you make directly to a function automatically are factored into 
+        the checksum calculation and thus, will invalidate the associated area of sstate cache.
+        You need to be aware of any implicit changes that are not obvious changes to the 
+        code and could affect the output of a given task. 
+        Once you are aware of such a change, you can take steps to invalidate the cache 
+        and force the task to run. 
+        The step to take is as simple as changing a function's comments in the source code. 
+        For example, to invalidate package sstate files, change the comment statments
+        of <filename>do_package</filename> or one of the functions it calls.
+        The change is purely cosmetic, but it causes the checksum to be recalculated and  
+        forces the task to be run again.
+    </para>
+    <note>
+        For an example of a commit that makes a cosmetic change to invalidate an sstate, 
+        see this
+        <ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
+    </note>
+</section>
+</chapter>
+<!-- 
+vim: expandtab tw=80 ts=4 
+-->

diff --git a/documentation/poky-ref-manual/technical-details.xml b/documentation/poky-ref-manual/technical-details.xml new file mode 100644 index 0000000000..8f83ecb3a1 --- /dev/null +++ b/documentation/poky-ref-manual/technical-details.xml
@@ -0,0 +1,229 @@
	1	<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
	2	"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
	3	<chapter id='technical-details'>
	4	<title>Technical Details</title>
	5
	6	<para>
	7	This chapter provides technical details for various parts of the Yocto Project.
	8	Currently, topics include Yocto Project components and shared state (sstate) cache.
	9	</para>
	10
	11	<section id='usingpoky-components'>
	12	<title>Yocto Project Components</title>
	13
	14	<para>
	15	The BitBake task executor together with various types of configuration files form the
	16	Yocto Project core.
	17	This section overviews the BitBake task executor and the
	18	configuration files by describing what they are used for and how they interact.
	19	</para>
	20
	21	<para>
	22	BitBake handles the parsing and execution of the data files.
	23	The data itself is of various types:
	24	<itemizedlist>
	25	<listitem><para><emphasis>Recipes:</emphasis> Provides details about particular
	26	pieces of software</para></listitem>
	27	<listitem><para><emphasis>Class Data:</emphasis> An abstraction of common build
	28	information (e.g. how to build a Linux kernel).</para></listitem>
	29	<listitem><para><emphasis>Configuration Data:</emphasis> Defines machine-specific settings,
	30	policy decisions, etc.
	31	Configuration data acts as the glue to bind everything together.</para></listitem>
	32	</itemizedlist>
	33	For more information on data, see the
	34	<ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html#yocto-project-terms'>
	35	Yocto Project Terms</ulink> section in
	36	<ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html'>
	37	The Yocto Project Development Manual</ulink>.
	38	</para>
	39
	40	<para>
	41	BitBake knows how to combine multiple data sources together and refers to each data source
	42	as a <link linkend='usingpoky-changes-layers'>'layer'</link>.
	43	</para>
	44
	45	<para>
	46	Following are some brief details on these core components.
	47	For more detailed information on these components see the
	48	<link linkend='ref-structure'>'Reference: Directory Structure'</link>
	49	appendix.
	50	</para>
	51
	52	<section id='usingpoky-components-bitbake'>
	53	<title>BitBake</title>
	54
	55	<para>
	56	BitBake is the tool at the heart of the Yocto Project and is responsible
	57	for parsing the metadata, generating a list of tasks from it,
	58	and then executing those tasks.
	59	To see a list of the options BitBake supports, use the following help command:
	60	<literallayout class='monospaced'>
	61	$ bitbake --help
	62	</literallayout>
	63	</para>
	64
	65	<para>
	66	The most common usage for BitBake is <filename>bitbake <packagename></filename>, where
	67	<filename>packagename</filename> is the name of the package you want to build
	68	(referred to as the "target" in this manual).
	69	The target often equates to the first part of a <filename>.bb</filename> filename.
	70	So, to run the <filename>matchbox-desktop_1.2.3.bb</filename> file, you
	71	might type the following:
	72	<literallayout class='monospaced'>
	73	$ bitbake matchbox-desktop
	74	</literallayout>
	75	Several different versions of <filename>matchbox-desktop</filename> might exist.
	76	BitBake chooses the one selected by the distribution configuration.
	77	You can get more details about how BitBake chooses between different
	78	target versions and providers in the
	79	<link linkend='ref-bitbake-providers'>Preferences and Providers</link> section.
	80	</para>
	81
	82	<para>
	83	BitBake also tries to execute any dependent tasks first.
	84	So for example, before building <filename>matchbox-desktop</filename>, BitBake
	85	would build a cross compiler and <filename>eglibc</filename> if they had not already
	86	been built.
	87	<note>This release of the Yocto Project does not support the <filename>glibc</filename>
	88	GNU version of the Unix standard C library. By default, the Yocto Project builds with
	89	<filename>eglibc</filename>.</note>
	90	</para>
	91
	92	<para>
	93	A useful BitBake option to consider is the <filename>-k</filename> or
	94	<filename>--continue</filename> option.
	95	This option instructs BitBake to try and continue processing the job as much
	96	as possible even after encountering an error.
	97	When an error occurs, the target that
	98	failed and those that depend on it cannot be remade.
	99	However, when you use this option other dependencies can still be processed.
	100	</para>
	101	</section>
	102
	103	<section id='usingpoky-components-metadata'>
	104	<title>Metadata (Recipes)</title>
	105
	106	<para>
	107	The <filename>.bb</filename> files are usually referred to as "recipes."
	108	In general, a recipe contains information about a single piece of software.
	109	The information includes the location from which to download the source patches
	110	(if any are needed), which special configuration options to apply,
	111	how to compile the source files, and how to package the compiled output.
	112	</para>
	113
	114	<para>
	115	The term "package" can also be used to describe recipes.
	116	However, since the same word is used for the packaged output from the Yocto
	117	Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files),
	118	this document avoids using the term "package" to refer to recipes.
	119	</para>
	120	</section>
	121
	122	<section id='usingpoky-components-classes'>
	123	<title>Classes</title>
	124
	125	<para>
	126	Class files (<filename>.bbclass</filename>) contain information that is useful to share
	127	between metadata files.
	128	An example is the Autotools class, which contains
	129	common settings for any application that Autotools uses.
	130	The <link linkend='ref-classes'>Reference: Classes</link> appendix provides details
	131	about common classes and how to use them.
	132	</para>
	133	</section>
	134
	135	<section id='usingpoky-components-configuration'>
	136	<title>Configuration</title>
	137
	138	<para>
	139	The configuration files (<filename>.conf</filename>) define various configuration variables
	140	that govern the Yocto Project build process.
	141	These files fall into several areas that define machine configuration options,
	142	distribution configuration options, compiler tuning options, general common configuration
	143	options and user configuration options (<filename>local.conf</filename>, which is found
	144	in the Yocto Project files build directory).
	145	</para>
	146	</section>
	147	</section>
	148
	149	<section id="considering-shared-state-cache">
	150	<title>Considering Shared State Cache</title>
	151
	152	<para>
	153	By design, the Yocto Project builds everything from scratch unless it can determine that
	154	a given task's inputs have not changed.
	155	While building from scratch ensures that everything is current, it does also
	156	mean that a lot of time could be spent rebuiding things that don't necessarily need built.
	157	</para>
	158
	159	<para>
	160	The Yocto Project build process uses a shared state caching scheme to avoid having to
	161	rebuild software when it is not necessary.
	162	Because the build time for a Yocto image can be significant, it is helpful to try and
	163	determine what really needs built and what can be skipped given a particular project's
	164	build process.
	165	</para>
	166
	167	<para>
	168	The scheme that the Yocto Project uses involves checksum generation and comparison for
	169	a task's inputs.
	170	The scheme also employs an area of memory called the shared state cache that is
	171	pointed to by the <filename>SSTATE_DIR</filename> variable.
	172	This area contains task output generated from a previous build.
	173	If a given task's checksum matches the checksum of a previous build for the same
	174	task, the build process uses the state of the cache rather than rerunning that
	175	task.
	176	</para>
	177
	178	<para>
	179	The previous paragraph is a simplistic explanation of how the build process
	180	uses checksums and shared state memory cache to avoide building tasks that
	181	don't need built.
	182	If you want a bit more explanation on the topic,
	183	see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
	184	State - What does it mean and why should I care?</ulink>" from the Yocto
	185	Project discussion archives.
	186	</para>
	187
	188	<para>
	189	As with all schemes, this one has some drawbacks.
	190	It is possible that you could make implicit changes that are not factored into the checksum
	191	calculation, but do affect a task's output.
	192	A good example is perhaps when a tool changes its output.
	193	Let's say that the output of <filename>rpmdeps</filename> needed to change.
	194	The result of the change should be that all the "package", "package_write_rpm",
	195	and "package_deploy-rpm" sstate-cache items would become invalid.
	196	But, because this is a change that is external to the code and therefore implicit,
	197	the associated sstate-cache items do not become invalidated.
	198	In this case, the build process would use the cache items rather than running the
	199	task again.
	200	Obviously, these types of implicit changes can cause problems.
	201	</para>
	202
	203	<para>
	204	To avoid these problems during the build, you need to understand the effects of any
	205	change you make.
	206	Note that any changes you make directly to a function automatically are factored into
	207	the checksum calculation and thus, will invalidate the associated area of sstate cache.
	208	You need to be aware of any implicit changes that are not obvious changes to the
	209	code and could affect the output of a given task.
	210	Once you are aware of such a change, you can take steps to invalidate the cache
	211	and force the task to run.
	212	The step to take is as simple as changing a function's comments in the source code.
	213	For example, to invalidate package sstate files, change the comment statments
	214	of <filename>do_package</filename> or one of the functions it calls.
	215	The change is purely cosmetic, but it causes the checksum to be recalculated and
	216	forces the task to be run again.
	217	</para>
	218
	219	<note>
	220	For an example of a commit that makes a cosmetic change to invalidate an sstate,
	221	see this
	222	<ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
	223	</note>
	224	</section>
	225
	226	</chapter>
	227	<!--
	228	vim: expandtab tw=80 ts=4
	229	-->