summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
Diffstat
-rw-r--r--documentation/dev-manual/dev-manual-common-tasks.xml6
-rw-r--r--documentation/overview-manual/overview-concepts.xml635
-rw-r--r--documentation/overview-manual/overview-development-environment.xml2
-rw-r--r--documentation/ref-manual/ref-bitbake.xml6
-rw-r--r--documentation/ref-manual/ref-classes.xml4
-rw-r--r--documentation/ref-manual/ref-tasks.xml14
-rw-r--r--documentation/ref-manual/ref-variables.xml4
-rw-r--r--documentation/ref-manual/technical-details.xml603
-rw-r--r--documentation/ref-manual/usingpoky.xml2
9 files changed, 656 insertions, 620 deletions
diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml
index 78825c7f1e..8a0d6a3222 100644
--- a/documentation/dev-manual/dev-manual-common-tasks.xml
+++ b/documentation/dev-manual/dev-manual-common-tasks.xml
@@ -7346,7 +7346,7 @@ Some notes from Cal:
7346 7346
7347 <para> 7347 <para>
7348 Because the OpenEmbedded build system uses 7348 Because the OpenEmbedded build system uses
7349 "<ulink url='&YOCTO_DOCS_REF_URL;#checksums'>signatures</ulink>", 7349 "<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#overview-checksums'>signatures</ulink>",
7350 which are unique to a given build, the build system 7350 which are unique to a given build, the build system
7351 knows when to rebuild packages. 7351 knows when to rebuild packages.
7352 All the inputs into a given task are represented by a 7352 All the inputs into a given task are represented by a
@@ -7450,8 +7450,8 @@ Some notes from Cal:
7450 7450
7451 <para> 7451 <para>
7452 For more information on shared state, see the 7452 For more information on shared state, see the
7453 "<ulink url='&YOCTO_DOCS_REF_URL;#shared-state-cache'>Shared State Cache</ulink>" 7453 "<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>Shared State Cache</ulink>"
7454 section in the Yocto Project Reference Manual. 7454 section in the Yocto Project Overview Manual.
7455 </para> 7455 </para>
7456 </note> 7456 </note>
7457 </section> 7457 </section>
diff --git a/documentation/overview-manual/overview-concepts.xml b/documentation/overview-manual/overview-concepts.xml
index 0a45cd7256..2d704923fa 100644
--- a/documentation/overview-manual/overview-concepts.xml
+++ b/documentation/overview-manual/overview-concepts.xml
@@ -471,6 +471,641 @@
471 </note> 471 </note>
472 </section> 472 </section>
473 473
474
475
476
477 <section id="shared-state-cache">
478 <title>Shared State Cache</title>
479
480 <para>
481 By design, the OpenEmbedded build system builds everything from
482 scratch unless BitBake can determine that parts do not need to be
483 rebuilt.
484 Fundamentally, building from scratch is attractive as it means all
485 parts are built fresh and there is no possibility of stale data
486 causing problems.
487 When developers hit problems, they typically default back to
488 building from scratch so they know the state of things from the
489 start.
490 </para>
491
492 <para>
493 Building an image from scratch is both an advantage and a
494 disadvantage to the process.
495 As mentioned in the previous paragraph, building from scratch
496 ensures that everything is current and starts from a known state.
497 However, building from scratch also takes much longer as it
498 generally means rebuilding things that do not necessarily need
499 to be rebuilt.
500 </para>
501
502 <para>
503 The Yocto Project implements shared state code that supports
504 incremental builds.
505 The implementation of the shared state code answers the following
506 questions that were fundamental roadblocks within the OpenEmbedded
507 incremental build support system:
508 <itemizedlist>
509 <listitem><para>
510 What pieces of the system have changed and what pieces have
511 not changed?
512 </para></listitem>
513 <listitem><para>
514 How are changed pieces of software removed and replaced?
515 </para></listitem>
516 <listitem><para>
517 How are pre-built components that do not need to be rebuilt
518 from scratch used when they are available?
519 </para></listitem>
520 </itemizedlist>
521 </para>
522
523 <para>
524 For the first question, the build system detects changes in the
525 "inputs" to a given task by creating a checksum (or signature) of
526 the task's inputs.
527 If the checksum changes, the system assumes the inputs have changed
528 and the task needs to be rerun.
529 For the second question, the shared state (sstate) code tracks
530 which tasks add which output to the build process.
531 This means the output from a given task can be removed, upgraded
532 or otherwise manipulated.
533 The third question is partly addressed by the solution for the
534 second question assuming the build system can fetch the sstate
535 objects from remote locations and install them if they are deemed
536 to be valid.
537 <note>
538 The OpenEmbedded build system does not maintain
539 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
540 information as part of the shared state packages.
541 Consequently, considerations exist that affect maintaining
542 shared state feeds.
543 For information on how the OpenEmbedded build system
544 works with packages and can track incrementing
545 <filename>PR</filename> information, see the
546 "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
547 section in the Yocto Project Development Tasks Manual.
548 </note>
549 </para>
550
551 <para>
552 The rest of this section goes into detail about the overall
553 incremental build architecture, the checksums (signatures), shared
554 state, and some tips and tricks.
555 </para>
556
557 <section id='overall-architecture'>
558 <title>Overall Architecture</title>
559
560 <para>
561 When determining what parts of the system need to be built,
562 BitBake works on a per-task basis rather than a per-recipe
563 basis.
564 You might wonder why using a per-task basis is preferred over
565 a per-recipe basis.
566 To help explain, consider having the IPK packaging backend
567 enabled and then switching to DEB.
568 In this case, the
569 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
570 and
571 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
572 task outputs are still valid.
573 However, with a per-recipe approach, the build would not
574 include the <filename>.deb</filename> files.
575 Consequently, you would have to invalidate the whole build and
576 rerun it.
577 Rerunning everything is not the best solution.
578 Also, in this case, the core must be "taught" much about
579 specific tasks.
580 This methodology does not scale well and does not allow users
581 to easily add new tasks in layers or as external recipes
582 without touching the packaged-staging core.
583 </para>
584 </section>
585
586 <section id='overview-checksums'>
587 <title>Checksums (Signatures)</title>
588
589 <para>
590 The shared state code uses a checksum, which is a unique
591 signature of a task's inputs, to determine if a task needs to
592 be run again.
593 Because it is a change in a task's inputs that triggers a
594 rerun, the process needs to detect all the inputs to a given
595 task.
596 For shell tasks, this turns out to be fairly easy because
597 the build process generates a "run" shell script for each task
598 and it is possible to create a checksum that gives you a good
599 idea of when the task's data changes.
600 </para>
601
602 <para>
603 To complicate the problem, there are things that should not be
604 included in the checksum.
605 First, there is the actual specific build path of a given
606 task - the
607 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
608 It does not matter if the work directory changes because it
609 should not affect the output for target packages.
610 Also, the build process has the objective of making native
611 or cross packages relocatable.
612 <note>
613 Both native and cross packages run on the build host.
614 However, cross packages generate output for the target
615 architecture.
616 </note>
617 The checksum therefore needs to exclude
618 <filename>WORKDIR</filename>.
619 The simplistic approach for excluding the work directory is to
620 set <filename>WORKDIR</filename> to some fixed value and
621 create the checksum for the "run" script.
622 </para>
623
624 <para>
625 Another problem results from the "run" scripts containing
626 functions that might or might not get called.
627 The incremental build solution contains code that figures out
628 dependencies between shell functions.
629 This code is used to prune the "run" scripts down to the
630 minimum set, thereby alleviating this problem and making the
631 "run" scripts much more readable as a bonus.
632 </para>
633
634 <para>
635 So far we have solutions for shell scripts.
636 What about Python tasks?
637 The same approach applies even though these tasks are more
638 difficult.
639 The process needs to figure out what variables a Python
640 function accesses and what functions it calls.
641 Again, the incremental build solution contains code that first
642 figures out the variable and function dependencies, and then
643 creates a checksum for the data used as the input to the task.
644 </para>
645
646 <para>
647 Like the <filename>WORKDIR</filename> case, situations exist
648 where dependencies should be ignored.
649 For these cases, you can instruct the build process to
650 ignore a dependency by using a line like the following:
651 <literallayout class='monospaced'>
652 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
653 </literallayout>
654 This example ensures that the
655 <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink>
656 variable does not depend on the value of
657 <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>,
658 even if it does reference it.
659 </para>
660
661 <para>
662 Equally, there are cases where we need to add dependencies
663 BitBake is not able to find.
664 You can accomplish this by using a line like the following:
665 <literallayout class='monospaced'>
666 PACKAGE_ARCHS[vardeps] = "MACHINE"
667 </literallayout>
668 This example explicitly adds the <filename>MACHINE</filename>
669 variable as a dependency for
670 <filename>PACKAGE_ARCHS</filename>.
671 </para>
672
673 <para>
674 Consider a case with in-line Python, for example, where
675 BitBake is not able to figure out dependencies.
676 When running in debug mode (i.e. using
677 <filename>-DDD</filename>), BitBake produces output when it
678 discovers something for which it cannot figure out dependencies.
679 The Yocto Project team has currently not managed to cover
680 those dependencies in detail and is aware of the need to fix
681 this situation.
682 </para>
683
684 <para>
685 Thus far, this section has limited discussion to the direct
686 inputs into a task.
687 Information based on direct inputs is referred to as the
688 "basehash" in the code.
689 However, there is still the question of a task's indirect
690 inputs - the things that were already built and present in the
691 <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>.
692 The checksum (or signature) for a particular task needs to add
693 the hashes of all the tasks on which the particular task
694 depends.
695 Choosing which dependencies to add is a policy decision.
696 However, the effect is to generate a master checksum that
697 combines the basehash and the hashes of the task's
698 dependencies.
699 </para>
700
701 <para>
702 At the code level, there are a variety of ways both the
703 basehash and the dependent task hashes can be influenced.
704 Within the BitBake configuration file, we can give BitBake
705 some extra information to help it construct the basehash.
706 The following statement effectively results in a list of
707 global variable dependency excludes - variables never
708 included in any checksum:
709 <literallayout class='monospaced'>
710 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
711 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
712 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
713 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
714 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
715 </literallayout>
716 The previous example excludes
717 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>
718 since that variable is actually constructed as a path within
719 <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>,
720 which is on the whitelist.
721 </para>
722
723 <para>
724 The rules for deciding which hashes of dependent tasks to
725 include through dependency chains are more complex and are
726 generally accomplished with a Python function.
727 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows
728 two examples of this and also illustrates how you can insert
729 your own policy into the system if so desired.
730 This file defines the two basic signature generators
731 <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink>
732 uses: "OEBasic" and "OEBasicHash".
733 By default, there is a dummy "noop" signature handler enabled
734 in BitBake.
735 This means that behavior is unchanged from previous versions.
736 OE-Core uses the "OEBasicHash" signature handler by default
737 through this setting in the <filename>bitbake.conf</filename>
738 file:
739 <literallayout class='monospaced'>
740 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
741 </literallayout>
742 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename>
743 is the same as the "OEBasic" version but adds the task hash to
744 the stamp files.
745 This results in any
746 <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink>
747 change that changes the task hash, automatically
748 causing the task to be run again.
749 This removes the need to bump
750 <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink>
751 values, and changes to Metadata automatically ripple across
752 the build.
753 </para>
754
755 <para>
756 It is also worth noting that the end result of these
757 signature generators is to make some dependency and hash
758 information available to the build.
759 This information includes:
760 <itemizedlist>
761 <listitem><para>
762 <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
763 The base hashes for each task in the recipe.
764 </para></listitem>
765 <listitem><para>
766 <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
767 The base hashes for each dependent task.
768 </para></listitem>
769 <listitem><para>
770 <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
771 The task dependencies for each task.
772 </para></listitem>
773 <listitem><para>
774 <filename>BB_TASKHASH</filename>:
775 The hash of the currently running task.
776 </para></listitem>
777 </itemizedlist>
778 </para>
779 </section>
780
781 <section id='shared-state'>
782 <title>Shared State</title>
783
784 <para>
785 Checksums and dependencies, as discussed in the previous
786 section, solve half the problem of supporting a shared state.
787 The other part of the problem is being able to use checksum
788 information during the build and being able to reuse or rebuild
789 specific components.
790 </para>
791
792 <para>
793 The
794 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
795 class is a relatively generic implementation of how to
796 "capture" a snapshot of a given task.
797 The idea is that the build process does not care about the
798 source of a task's output.
799 Output could be freshly built or it could be downloaded and
800 unpacked from somewhere - the build process does not need to
801 worry about its origin.
802 </para>
803
804 <para>
805 There are two types of output, one is just about creating a
806 directory in
807 <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>.
808 A good example is the output of either
809 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink>
810 or
811 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>.
812 The other type of output occurs when a set of data is merged
813 into a shared directory tree such as the sysroot.
814 </para>
815
816 <para>
817 The Yocto Project team has tried to keep the details of the
818 implementation hidden in <filename>sstate</filename> class.
819 From a user's perspective, adding shared state wrapping to a task
820 is as simple as this
821 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
822 example taken from the
823 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink>
824 class:
825 <literallayout class='monospaced'>
826 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
827 SSTATETASKS += "do_deploy"
828 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
829 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
830
831 python do_deploy_setscene () {
832 sstate_setscene(d)
833 }
834 addtask do_deploy_setscene
835 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
836 </literallayout>
837 The following list explains the previous example:
838 <itemizedlist>
839 <listitem><para>
840 Adding "do_deploy" to <filename>SSTATETASKS</filename>
841 adds some required sstate-related processing, which is
842 implemented in the
843 <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink>
844 class, to before and after the
845 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink>
846 task.
847 </para></listitem>
848 <listitem><para>
849 The
850 <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
851 declares that <filename>do_deploy</filename> places its
852 output in <filename>${DEPLOYDIR}</filename> when run
853 normally (i.e. when not using the sstate cache).
854 This output becomes the input to the shared state cache.
855 </para></listitem>
856 <listitem><para>
857 The
858 <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
859 line causes the contents of the shared state cache to be
860 copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
861 <note>
862 If <filename>do_deploy</filename> is not already in
863 the shared state cache or if its input checksum
864 (signature) has changed from when the output was
865 cached, the task will be run to populate the shared
866 state cache, after which the contents of the shared
867 state cache is copied to
868 <filename>${DEPLOY_DIR_IMAGE}</filename>.
869 If <filename>do_deploy</filename> is in the shared
870 state cache and its signature indicates that the
871 cached output is still valid (i.e. if no
872 relevant task inputs have changed), then the
873 contents of the shared state cache will be copied
874 directly to
875 <filename>${DEPLOY_DIR_IMAGE}</filename> by the
876 <filename>do_deploy_setscene</filename> task
877 instead, skipping the
878 <filename>do_deploy</filename> task.
879 </note>
880 </para></listitem>
881 <listitem><para>
882 The following task definition is glue logic needed to
883 make the previous settings effective:
884 <literallayout class='monospaced'>
885 python do_deploy_setscene () {
886 sstate_setscene(d)
887 }
888 addtask do_deploy_setscene
889 </literallayout>
890 <filename>sstate_setscene()</filename> takes the flags
891 above as input and accelerates the
892 <filename>do_deploy</filename> task through the
893 shared state cache if possible.
894 If the task was accelerated,
895 <filename>sstate_setscene()</filename> returns True.
896 Otherwise, it returns False, and the normal
897 <filename>do_deploy</filename> task runs.
898 For more information, see the
899 "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
900 section in the BitBake User Manual.
901 </para></listitem>
902 <listitem><para>
903 The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
904 line creates <filename>${DEPLOYDIR}</filename> and
905 <filename>${B}</filename> before the
906 <filename>do_deploy</filename> task runs, and also sets
907 the current working directory of
908 <filename>do_deploy</filename> to
909 <filename>${B}</filename>.
910 For more information, see the
911 "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
912 section in the BitBake User Manual.
913 <note>
914 In cases where
915 <filename>sstate-inputdirs</filename> and
916 <filename>sstate-outputdirs</filename> would be the
917 same, you can use
918 <filename>sstate-plaindirs</filename>.
919 For example, to preserve the
920 <filename>${PKGD}</filename> and
921 <filename>${PKGDEST}</filename> output from the
922 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
923 task, use the following:
924 <literallayout class='monospaced'>
925 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
926 </literallayout>
927 </note>
928 </para></listitem>
929 <listitem><para>
930 <filename>sstate-inputdirs</filename> and
931 <filename>sstate-outputdirs</filename> can also be used
932 with multiple directories.
933 For example, the following declares
934 <filename>PKGDESTWORK</filename> and
935 <filename>SHLIBWORK</filename> as shared state
936 input directories, which populates the shared state
937 cache, and <filename>PKGDATA_DIR</filename> and
938 <filename>SHLIBSDIR</filename> as the corresponding
939 shared state output directories:
940 <literallayout class='monospaced'>
941 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
942 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
943 </literallayout>
944 </para></listitem>
945 <listitem><para>
946 These methods also include the ability to take a
947 lockfile when manipulating shared state directory
948 structures, for cases where file additions or removals
949 are sensitive:
950 <literallayout class='monospaced'>
951 do_package[sstate-lockfile] = "${PACKAGELOCK}"
952 </literallayout>
953 </para></listitem>
954 </itemizedlist>
955 </para>
956
957 <para>
958 Behind the scenes, the shared state code works by looking in
959 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>
960 and
961 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink>
962 for shared state files.
963 Here is an example:
964 <literallayout class='monospaced'>
965 SSTATE_MIRRORS ?= "\
966 file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
967 file://.* file:///some/local/dir/sstate/PATH"
968 </literallayout>
969 <note>
970 The shared state directory
971 (<filename>SSTATE_DIR</filename>) is organized into
972 two-character subdirectories, where the subdirectory
973 names are based on the first two characters of the hash.
974 If the shared state directory structure for a mirror has the
975 same structure as <filename>SSTATE_DIR</filename>, you must
976 specify "PATH" as part of the URI to enable the build system
977 to map to the appropriate subdirectory.
978 </note>
979 </para>
980
981 <para>
982 The shared state package validity can be detected just by
983 looking at the filename since the filename contains the task
984 checksum (or signature) as described earlier in this section.
985 If a valid shared state package is found, the build process
986 downloads it and uses it to accelerate the task.
987 </para>
988
989 <para>
990 The build processes use the <filename>*_setscene</filename>
991 tasks for the task acceleration phase.
992 BitBake goes through this phase before the main execution
993 code and tries to accelerate any tasks for which it can find
994 shared state packages.
995 If a shared state package for a task is available, the
996 shared state package is used.
997 This means the task and any tasks on which it is dependent
998 are not executed.
999 </para>
1000
1001 <para>
1002 As a real world example, the aim is when building an IPK-based
1003 image, only the
1004 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink>
1005 tasks would have their shared state packages fetched and
1006 extracted.
1007 Since the sysroot is not used, it would never get extracted.
1008 This is another reason why a task-based approach is preferred
1009 over a recipe-based approach, which would have to install the
1010 output from every task.
1011 </para>
1012 </section>
1013
1014 <section id='tips-and-tricks'>
1015 <title>Tips and Tricks</title>
1016
1017 <para>
1018 The code in the build system that supports incremental builds
1019 is not simple code.
1020 This section presents some tips and tricks that help you work
1021 around issues related to shared state code.
1022 </para>
1023
1024 <section id='overview-debugging'>
1025 <title>Debugging</title>
1026
1027 <para>
1028 Seeing what metadata went into creating the input signature
1029 of a shared state (sstate) task can be a useful debugging
1030 aid.
1031 This information is available in signature information
1032 (<filename>siginfo</filename>) files in
1033 <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>.
1034 For information on how to view and interpret information in
1035 <filename>siginfo</filename> files, see the
1036 "<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>"
1037 section in the Yocto Project Reference Manual.
1038 </para>
1039 </section>
1040
1041 <section id='invalidating-shared-state'>
1042 <title>Invalidating Shared State</title>
1043
1044 <para>
1045 The OpenEmbedded build system uses checksums and shared
1046 state cache to avoid unnecessarily rebuilding tasks.
1047 Collectively, this scheme is known as "shared state code."
1048 </para>
1049
1050 <para>
1051 As with all schemes, this one has some drawbacks.
1052 It is possible that you could make implicit changes to your
1053 code that the checksum calculations do not take into
1054 account.
1055 These implicit changes affect a task's output but do not
1056 trigger the shared state code into rebuilding a recipe.
1057 Consider an example during which a tool changes its output.
1058 Assume that the output of <filename>rpmdeps</filename>
1059 changes.
1060 The result of the change should be that all the
1061 <filename>package</filename> and
1062 <filename>package_write_rpm</filename> shared state cache
1063 items become invalid.
1064 However, because the change to the output is
1065 external to the code and therefore implicit,
1066 the associated shared state cache items do not become
1067 invalidated.
1068 In this case, the build process uses the cached items
1069 rather than running the task again.
1070 Obviously, these types of implicit changes can cause
1071 problems.
1072 </para>
1073
1074 <para>
1075 To avoid these problems during the build, you need to
1076 understand the effects of any changes you make.
1077 Realize that changes you make directly to a function
1078 are automatically factored into the checksum calculation.
1079 Thus, these explicit changes invalidate the associated
1080 area of shared state cache.
1081 However, you need to be aware of any implicit changes that
1082 are not obvious changes to the code and could affect
1083 the output of a given task.
1084 </para>
1085
1086 <para>
1087 When you identify an implicit change, you can easily
1088 take steps to invalidate the cache and force the tasks
1089 to run.
1090 The steps you can take are as simple as changing a
1091 function's comments in the source code.
1092 For example, to invalidate package shared state files,
1093 change the comment statements of
1094 <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>
1095 or the comments of one of the functions it calls.
1096 Even though the change is purely cosmetic, it causes the
1097 checksum to be recalculated and forces the OpenEmbedded
1098 build system to run the task again.
1099 <note>
1100 For an example of a commit that makes a cosmetic
1101 change to invalidate shared state, see this
1102 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
1103 </note>
1104 </para>
1105 </section>
1106 </section>
1107 </section>
1108
474 <section id='x32'> 1109 <section id='x32'>
475 <title>x32 psABI</title> 1110 <title>x32 psABI</title>
476 1111
diff --git a/documentation/overview-manual/overview-development-environment.xml b/documentation/overview-manual/overview-development-environment.xml
index 62f3ccd438..e0d470b179 100644
--- a/documentation/overview-manual/overview-development-environment.xml
+++ b/documentation/overview-manual/overview-development-environment.xml
@@ -2490,7 +2490,7 @@
2490 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink> 2490 <ulink url='&YOCTO_DOCS_REF_URL;#var-STAMP'><filename>STAMP</filename></ulink>
2491 variable, and the end of the name consists of the task's name 2491 variable, and the end of the name consists of the task's name
2492 and current 2492 and current
2493 <ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksum</ulink>. 2493 <link linkend='overview-checksums'>input checksum</link>.
2494 <note> 2494 <note>
2495 This naming scheme assumes that 2495 This naming scheme assumes that
2496 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink> 2496 <ulink url='&YOCTO_DOCS_BB_URL;#var-BB_SIGNATURE_HANDLER'><filename>BB_SIGNATURE_HANDLER</filename></ulink>
diff --git a/documentation/ref-manual/ref-bitbake.xml b/documentation/ref-manual/ref-bitbake.xml
index 17f4c54b9c..7d1dd48128 100644
--- a/documentation/ref-manual/ref-bitbake.xml
+++ b/documentation/ref-manual/ref-bitbake.xml
@@ -348,8 +348,10 @@
348 If doing so results in unnecessary rebuilds of tasks, you can whitelist the 348 If doing so results in unnecessary rebuilds of tasks, you can whitelist the
349 variable so that the shared state code ignores the dependency when it creates 349 variable so that the shared state code ignores the dependency when it creates
350 checksums. 350 checksums.
351 For information on this process, see the <filename>BB_HASHBASE_WHITELIST</filename> 351 For information on this process, see the
352 example in the "<link linkend='checksums'>Checksums (Signatures)</link>" section. 352 <filename>BB_HASHBASE_WHITELIST</filename> example in the
353 "<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#overview-checksums'>Checksums (Signatures)</ulink>"
354 section in the Yocto Project Overview Manual.
353 </note> 355 </note>
354 </section> 356 </section>
355 357
diff --git a/documentation/ref-manual/ref-classes.xml b/documentation/ref-manual/ref-classes.xml
index b11fbea63e..eabb60fe03 100644
--- a/documentation/ref-manual/ref-classes.xml
+++ b/documentation/ref-manual/ref-classes.xml
@@ -3182,8 +3182,8 @@ This check was removed for YP 2.3 release
3182 3182
3183 <para> 3183 <para>
3184 For more information on sstate, see the 3184 For more information on sstate, see the
3185 "<link linkend='shared-state-cache'>Shared State Cache</link>" 3185 "<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>Shared State Cache</ulink>"
3186 section. 3186 section in the Yocto Project Overview Manual.
3187 </para> 3187 </para>
3188</section> 3188</section>
3189 3189
diff --git a/documentation/ref-manual/ref-tasks.xml b/documentation/ref-manual/ref-tasks.xml
index 62a677a430..a90f02b7d8 100644
--- a/documentation/ref-manual/ref-tasks.xml
+++ b/documentation/ref-manual/ref-tasks.xml
@@ -628,8 +628,8 @@
628 628
629 <para> 629 <para>
630 Running this task does not remove the 630 Running this task does not remove the
631 <link linkend='shared-state-cache'>sstate</link>) cache 631 <ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>
632 files. 632 cache files.
633 Consequently, if no changes have been made and the recipe is 633 Consequently, if no changes have been made and the recipe is
634 rebuilt after cleaning, output files are simply restored from the 634 rebuilt after cleaning, output files are simply restored from the
635 sstate cache. 635 sstate cache.
@@ -645,8 +645,9 @@
645 645
646 <para> 646 <para>
647 Removes all output files, shared state 647 Removes all output files, shared state
648 (<link linkend='shared-state-cache'>sstate</link>) cache, and 648 (<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
649 downloaded source files for a target (i.e. the contents of 649 cache, and downloaded source files for a target (i.e. the contents
650 of
650 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>). 651 <link linkend='var-DL_DIR'><filename>DL_DIR</filename></link>).
651 Essentially, the <filename>do_cleanall</filename> task is 652 Essentially, the <filename>do_cleanall</filename> task is
652 identical to the 653 identical to the
@@ -675,13 +676,14 @@
675 676
676 <para> 677 <para>
677 Removes all output files and shared state 678 Removes all output files and shared state
678 (<link linkend='shared-state-cache'>sstate</link>) 679 (<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
679 cache for a target. 680 cache for a target.
680 Essentially, the <filename>do_cleansstate</filename> task is 681 Essentially, the <filename>do_cleansstate</filename> task is
681 identical to the 682 identical to the
682 <link linkend='ref-tasks-clean'><filename>do_clean</filename></link> 683 <link linkend='ref-tasks-clean'><filename>do_clean</filename></link>
683 task with the added removal of shared state 684 task with the added removal of shared state
684 (<link linkend='shared-state-cache'>sstate</link>) cache. 685 (<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
686 cache.
685 </para> 687 </para>
686 688
687 <para> 689 <para>
diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index 48849b1fef..564bb38ac6 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -10654,11 +10654,11 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
10654 <filename>PR</filename> to know when to rebuild a 10654 <filename>PR</filename> to know when to rebuild a
10655 recipe. 10655 recipe.
10656 The build system uses the task 10656 The build system uses the task
10657 <ulink url='&YOCTO_DOCS_BB_URL;#checksums'>input checksums</ulink> 10657 <ulink url='&YOCTO_DOCS_OVERVIEW_URL;#overview-checksums'>input checksums</ulink>
10658 along with the 10658 along with the
10659 <link linkend='structure-build-tmp-stamps'>stamp</link> 10659 <link linkend='structure-build-tmp-stamps'>stamp</link>
10660 and 10660 and
10661 <link linkend='shared-state-cache'>shared state cache</link> 10661 <ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>shared state cache</ulink>
10662 mechanisms. 10662 mechanisms.
10663 </note> 10663 </note>
10664 The <filename>PR</filename> variable primarily becomes 10664 The <filename>PR</filename> variable primarily becomes
diff --git a/documentation/ref-manual/technical-details.xml b/documentation/ref-manual/technical-details.xml
index 3d3def5a65..06a018b8a9 100644
--- a/documentation/ref-manual/technical-details.xml
+++ b/documentation/ref-manual/technical-details.xml
@@ -13,609 +13,6 @@
13 x32, Wayland support, and Licenses. 13 x32, Wayland support, and Licenses.
14 </para> 14 </para>
15 15
16<section id="shared-state-cache">
17 <title>Shared State Cache</title>
18
19 <para>
20 By design, the OpenEmbedded build system builds everything from scratch unless
21 BitBake can determine that parts do not need to be rebuilt.
22 Fundamentally, building from scratch is attractive as it means all parts are
23 built fresh and there is no possibility of stale data causing problems.
24 When developers hit problems, they typically default back to building from scratch
25 so they know the state of things from the start.
26 </para>
27
28 <para>
29 Building an image from scratch is both an advantage and a disadvantage to the process.
30 As mentioned in the previous paragraph, building from scratch ensures that
31 everything is current and starts from a known state.
32 However, building from scratch also takes much longer as it generally means
33 rebuilding things that do not necessarily need to be rebuilt.
34 </para>
35
36 <para>
37 The Yocto Project implements shared state code that supports incremental builds.
38 The implementation of the shared state code answers the following questions that
39 were fundamental roadblocks within the OpenEmbedded incremental build support system:
40 <itemizedlist>
41 <listitem><para>What pieces of the system have changed and what pieces have
42 not changed?</para></listitem>
43 <listitem><para>How are changed pieces of software removed and replaced?</para></listitem>
44 <listitem><para>How are pre-built components that do not need to be rebuilt from scratch
45 used when they are available?</para></listitem>
46 </itemizedlist>
47 </para>
48
49 <para>
50 For the first question, the build system detects changes in the "inputs" to a given task by
51 creating a checksum (or signature) of the task's inputs.
52 If the checksum changes, the system assumes the inputs have changed and the task needs to be
53 rerun.
54 For the second question, the shared state (sstate) code tracks which tasks add which output
55 to the build process.
56 This means the output from a given task can be removed, upgraded or otherwise manipulated.
57 The third question is partly addressed by the solution for the second question
58 assuming the build system can fetch the sstate objects from remote locations and
59 install them if they are deemed to be valid.
60 </para>
61
62 <note>
63 The OpenEmbedded build system does not maintain
64 <link linkend='var-PR'><filename>PR</filename></link> information
65 as part of the shared state packages.
66 Consequently, considerations exist that affect maintaining shared
67 state feeds.
68 For information on how the OpenEmbedded build system
69 works with packages and can
70 track incrementing <filename>PR</filename> information, see the
71 "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>"
72 section in the Yocto Project Development Tasks Manual.
73 </note>
74
75 <para>
76 The rest of this section goes into detail about the overall incremental build
77 architecture, the checksums (signatures), shared state, and some tips and tricks.
78 </para>
79
80 <section id='overall-architecture'>
81 <title>Overall Architecture</title>
82
83 <para>
84 When determining what parts of the system need to be built, BitBake
85 works on a per-task basis rather than a per-recipe basis.
86 You might wonder why using a per-task basis is preferred over a per-recipe basis.
87 To help explain, consider having the IPK packaging backend enabled and then switching to DEB.
88 In this case, the
89 <link linkend='ref-tasks-install'><filename>do_install</filename></link>
90 and
91 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
92 task outputs are still valid.
93 However, with a per-recipe approach, the build would not include the
94 <filename>.deb</filename> files.
95 Consequently, you would have to invalidate the whole build and rerun it.
96 Rerunning everything is not the best solution.
97 Also, in this case, the core must be "taught" much about specific tasks.
98 This methodology does not scale well and does not allow users to easily add new tasks
99 in layers or as external recipes without touching the packaged-staging core.
100 </para>
101 </section>
102
103 <section id='checksums'>
104 <title>Checksums (Signatures)</title>
105
106 <para>
107 The shared state code uses a checksum, which is a unique signature of a task's
108 inputs, to determine if a task needs to be run again.
109 Because it is a change in a task's inputs that triggers a rerun, the process
110 needs to detect all the inputs to a given task.
111 For shell tasks, this turns out to be fairly easy because
112 the build process generates a "run" shell script for each task and
113 it is possible to create a checksum that gives you a good idea of when
114 the task's data changes.
115 </para>
116
117 <para>
118 To complicate the problem, there are things that should not be
119 included in the checksum.
120 First, there is the actual specific build path of a given task -
121 the <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
122 It does not matter if the work directory changes because it should
123 not affect the output for target packages.
124 Also, the build process has the objective of making native
125 or cross packages relocatable.
126 <note>
127 Both native and cross packages run on the build host.
128 However, cross packages generate output for the target
129 architecture.
130 </note>
131 The checksum therefore needs to exclude
132 <filename>WORKDIR</filename>.
133 The simplistic approach for excluding the work directory is to set
134 <filename>WORKDIR</filename> to some fixed value and create the
135 checksum for the "run" script.
136 </para>
137
138 <para>
139 Another problem results from the "run" scripts containing functions that
140 might or might not get called.
141 The incremental build solution contains code that figures out dependencies
142 between shell functions.
143 This code is used to prune the "run" scripts down to the minimum set,
144 thereby alleviating this problem and making the "run" scripts much more
145 readable as a bonus.
146 </para>
147
148 <para>
149 So far we have solutions for shell scripts.
150 What about Python tasks?
151 The same approach applies even though these tasks are more difficult.
152 The process needs to figure out what variables a Python function accesses
153 and what functions it calls.
154 Again, the incremental build solution contains code that first figures out
155 the variable and function dependencies, and then creates a checksum for the data
156 used as the input to the task.
157 </para>
158
159 <para>
160 Like the <filename>WORKDIR</filename> case, situations exist where dependencies
161 should be ignored.
162 For these cases, you can instruct the build process to ignore a dependency
163 by using a line like the following:
164 <literallayout class='monospaced'>
165 PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
166 </literallayout>
167 This example ensures that the
168 <link linkend='var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></link>
169 variable does not
170 depend on the value of
171 <link linkend='var-MACHINE'><filename>MACHINE</filename></link>,
172 even if it does reference it.
173 </para>
174
175 <para>
176 Equally, there are cases where we need to add dependencies BitBake is not able to find.
177 You can accomplish this by using a line like the following:
178 <literallayout class='monospaced'>
179 PACKAGE_ARCHS[vardeps] = "MACHINE"
180 </literallayout>
181 This example explicitly adds the <filename>MACHINE</filename> variable as a
182 dependency for <filename>PACKAGE_ARCHS</filename>.
183 </para>
184
185 <para>
186 Consider a case with in-line Python, for example, where BitBake is not
187 able to figure out dependencies.
188 When running in debug mode (i.e. using <filename>-DDD</filename>), BitBake
189 produces output when it discovers something for which it cannot figure out
190 dependencies.
191 The Yocto Project team has currently not managed to cover those dependencies
192 in detail and is aware of the need to fix this situation.
193 </para>
194
195 <para>
196 Thus far, this section has limited discussion to the direct inputs into a task.
197 Information based on direct inputs is referred to as the "basehash" in the
198 code.
199 However, there is still the question of a task's indirect inputs - the
200 things that were already built and present in the
201 <link linkend='build-directory'>Build Directory</link>.
202 The checksum (or signature) for a particular task needs to add the hashes
203 of all the tasks on which the particular task depends.
204 Choosing which dependencies to add is a policy decision.
205 However, the effect is to generate a master checksum that combines the basehash
206 and the hashes of the task's dependencies.
207 </para>
208
209 <para>
210 At the code level, there are a variety of ways both the basehash and the
211 dependent task hashes can be influenced.
212 Within the BitBake configuration file, we can give BitBake some extra information
213 to help it construct the basehash.
214 The following statement effectively results in a list of global variable
215 dependency excludes - variables never included in any checksum:
216 <literallayout class='monospaced'>
217 BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \
218 SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
219 USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
220 PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
221 CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX"
222 </literallayout>
223 The previous example excludes
224 <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>
225 since that variable is actually constructed as a path within
226 <link linkend='var-TMPDIR'><filename>TMPDIR</filename></link>, which is on
227 the whitelist.
228 </para>
229
230 <para>
231 The rules for deciding which hashes of dependent tasks to include through
232 dependency chains are more complex and are generally accomplished with a
233 Python function.
234 The code in <filename>meta/lib/oe/sstatesig.py</filename> shows two examples
235 of this and also illustrates how you can insert your own policy into the system
236 if so desired.
237 This file defines the two basic signature generators
238 <link linkend='oe-core'>OE-Core</link> uses: "OEBasic" and
239 "OEBasicHash".
240 By default, there is a dummy "noop" signature handler enabled in BitBake.
241 This means that behavior is unchanged from previous versions.
242 OE-Core uses the "OEBasicHash" signature handler by default
243 through this setting in the <filename>bitbake.conf</filename> file:
244 <literallayout class='monospaced'>
245 BB_SIGNATURE_HANDLER ?= "OEBasicHash"
246 </literallayout>
247 The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> is the same as the
248 "OEBasic" version but adds the task hash to the stamp files.
249 This results in any
250 <link linkend='metadata'>Metadata</link>
251 change that changes the task hash, automatically
252 causing the task to be run again.
253 This removes the need to bump <link linkend='var-PR'><filename>PR</filename></link>
254 values, and changes to Metadata automatically ripple across the build.
255 </para>
256
257 <para>
258 It is also worth noting that the end result of these signature generators is to
259 make some dependency and hash information available to the build.
260 This information includes:
261 <itemizedlist>
262 <listitem><para><filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>:
263 The base hashes for each task in the recipe.
264 </para></listitem>
265 <listitem><para><filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
266 The base hashes for each dependent task.
267 </para></listitem>
268 <listitem><para><filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>:
269 The task dependencies for each task.
270 </para></listitem>
271 <listitem><para><filename>BB_TASKHASH</filename>:
272 The hash of the currently running task.
273 </para></listitem>
274 </itemizedlist>
275 </para>
276 </section>
277
278 <section id='shared-state'>
279 <title>Shared State</title>
280
281 <para>
282 Checksums and dependencies, as discussed in the previous section, solve half the
283 problem of supporting a shared state.
284 The other part of the problem is being able to use checksum information during the build
285 and being able to reuse or rebuild specific components.
286 </para>
287
288 <para>
289 The
290 <link linkend='ref-classes-sstate'><filename>sstate</filename></link>
291 class is a relatively generic implementation of how to "capture"
292 a snapshot of a given task.
293 The idea is that the build process does not care about the source of a task's output.
294 Output could be freshly built or it could be downloaded and unpacked from
295 somewhere - the build process does not need to worry about its origin.
296 </para>
297
298 <para>
299 There are two types of output, one is just about creating a directory
300 in <link linkend='var-WORKDIR'><filename>WORKDIR</filename></link>.
301 A good example is the output of either
302 <link linkend='ref-tasks-install'><filename>do_install</filename></link>
303 or
304 <link linkend='ref-tasks-package'><filename>do_package</filename></link>.
305 The other type of output occurs when a set of data is merged into a shared directory
306 tree such as the sysroot.
307 </para>
308
309 <para>
310 The Yocto Project team has tried to keep the details of the
311 implementation hidden in <filename>sstate</filename> class.
312 From a user's perspective, adding shared state wrapping to a task
313 is as simple as this
314 <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link>
315 example taken from the
316 <link linkend='ref-classes-deploy'><filename>deploy</filename></link>
317 class:
318 <literallayout class='monospaced'>
319 DEPLOYDIR = "${WORKDIR}/deploy-${PN}"
320 SSTATETASKS += "do_deploy"
321 do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"
322 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
323
324 python do_deploy_setscene () {
325 sstate_setscene(d)
326 }
327 addtask do_deploy_setscene
328 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
329 </literallayout>
330 The following list explains the previous example:
331 <itemizedlist>
332 <listitem><para>
333 Adding "do_deploy" to <filename>SSTATETASKS</filename>
334 adds some required sstate-related processing, which is
335 implemented in the
336 <link linkend='ref-classes-sstate'><filename>sstate</filename></link>
337 class, to before and after the
338 <link linkend='ref-tasks-deploy'><filename>do_deploy</filename></link>
339 task.
340 </para></listitem>
341 <listitem><para>
342 The
343 <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename>
344 declares that <filename>do_deploy</filename> places its
345 output in <filename>${DEPLOYDIR}</filename> when run
346 normally (i.e. when not using the sstate cache).
347 This output becomes the input to the shared state cache.
348 </para></listitem>
349 <listitem><para>
350 The
351 <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename>
352 line causes the contents of the shared state cache to be
353 copied to <filename>${DEPLOY_DIR_IMAGE}</filename>.
354 <note>
355 If <filename>do_deploy</filename> is not already in
356 the shared state cache or if its input checksum
357 (signature) has changed from when the output was
358 cached, the task will be run to populate the shared
359 state cache, after which the contents of the shared
360 state cache is copied to
361 <filename>${DEPLOY_DIR_IMAGE}</filename>.
362 If <filename>do_deploy</filename> is in the shared
363 state cache and its signature indicates that the
364 cached output is still valid (i.e. if no
365 relevant task inputs have changed), then the contents
366 of the shared state cache will be copied directly to
367 <filename>${DEPLOY_DIR_IMAGE}</filename> by the
368 <filename>do_deploy_setscene</filename> task instead,
369 skipping the <filename>do_deploy</filename> task.
370 </note>
371 </para></listitem>
372 <listitem><para>
373 The following task definition is glue logic needed to make
374 the previous settings effective:
375 <literallayout class='monospaced'>
376 python do_deploy_setscene () {
377 sstate_setscene(d)
378 }
379 addtask do_deploy_setscene
380 </literallayout>
381 <filename>sstate_setscene()</filename> takes the flags
382 above as input and accelerates the
383 <filename>do_deploy</filename> task through the
384 shared state cache if possible.
385 If the task was accelerated,
386 <filename>sstate_setscene()</filename> returns True.
387 Otherwise, it returns False, and the normal
388 <filename>do_deploy</filename> task runs.
389 For more information, see the
390 "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>"
391 section in the BitBake User Manual.
392 </para></listitem>
393 <listitem><para>
394 The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename>
395 line creates <filename>${DEPLOYDIR}</filename> and
396 <filename>${B}</filename> before the
397 <filename>do_deploy</filename> task runs, and also sets
398 the current working directory of
399 <filename>do_deploy</filename> to
400 <filename>${B}</filename>.
401 For more information, see the
402 "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>"
403 section in the BitBake User Manual.
404 <note>
405 In cases where
406 <filename>sstate-inputdirs</filename> and
407 <filename>sstate-outputdirs</filename> would be the
408 same, you can use
409 <filename>sstate-plaindirs</filename>.
410 For example, to preserve the
411 <filename>${PKGD}</filename> and
412 <filename>${PKGDEST}</filename> output from the
413 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
414 task, use the following:
415 <literallayout class='monospaced'>
416 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
417 </literallayout>
418 </note>
419 </para></listitem>
420 <listitem><para>
421 <filename>sstate-inputdirs</filename> and
422 <filename>sstate-outputdirs</filename> can also be used
423 with multiple directories.
424 For example, the following declares
425 <filename>PKGDESTWORK</filename> and
426 <filename>SHLIBWORK</filename> as shared state
427 input directories, which populates the shared state
428 cache, and <filename>PKGDATA_DIR</filename> and
429 <filename>SHLIBSDIR</filename> as the corresponding
430 shared state output directories:
431 <literallayout class='monospaced'>
432 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
433 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
434 </literallayout>
435 </para></listitem>
436 <listitem><para>
437 These methods also include the ability to take a lockfile
438 when manipulating shared state directory structures,
439 for cases where file additions or removals are sensitive:
440 <literallayout class='monospaced'>
441 do_package[sstate-lockfile] = "${PACKAGELOCK}"
442 </literallayout>
443 </para></listitem>
444 </itemizedlist>
445 </para>
446
447<!--
448 <para>
449 In this example, we add some extra flags to the task, a name field ("deploy"), an
450 input directory where the task sends data, and the output
451 directory where the data from the task should eventually be copied.
452 We also add a <filename>_setscene</filename> variant of the task and add the task
453 name to the <filename>SSTATETASKS</filename> list.
454 </para>
455
456 <para>
457 If you have a directory whose contents you need to preserve, you can do this with
458 a line like the following:
459 <literallayout class='monospaced'>
460 do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}"
461 </literallayout>
462 This method, as well as the following example, also works for multiple directories.
463 <literallayout class='monospaced'>
464 do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
465 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
466 do_package[sstate-lockfile] = "${PACKAGELOCK}"
467 </literallayout>
468 These methods also include the ability to take a lockfile when manipulating
469 shared state directory structures since some cases are sensitive to file
470 additions or removals.
471 </para>
472-->
473
474 <para>
475 Behind the scenes, the shared state code works by looking in
476 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> and
477 <link linkend='var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></link>
478 for shared state files.
479 Here is an example:
480 <literallayout class='monospaced'>
481 SSTATE_MIRRORS ?= "\
482 file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
483 file://.* file:///some/local/dir/sstate/PATH"
484 </literallayout>
485 <note>
486 The shared state directory (<filename>SSTATE_DIR</filename>) is
487 organized into two-character subdirectories, where the subdirectory
488 names are based on the first two characters of the hash.
489 If the shared state directory structure for a mirror has the
490 same structure as <filename>SSTATE_DIR</filename>, you must
491 specify "PATH" as part of the URI to enable the build system
492 to map to the appropriate subdirectory.
493 </note>
494 </para>
495
496 <para>
497 The shared state package validity can be detected just by looking at the
498 filename since the filename contains the task checksum (or signature) as
499 described earlier in this section.
500 If a valid shared state package is found, the build process downloads it
501 and uses it to accelerate the task.
502 </para>
503
504 <para>
505 The build processes use the <filename>*_setscene</filename> tasks
506 for the task acceleration phase.
507 BitBake goes through this phase before the main execution code and tries
508 to accelerate any tasks for which it can find shared state packages.
509 If a shared state package for a task is available, the shared state
510 package is used.
511 This means the task and any tasks on which it is dependent are not
512 executed.
513 </para>
514
515 <para>
516 As a real world example, the aim is when building an IPK-based image,
517 only the
518 <link linkend='ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></link>
519 tasks would have their
520 shared state packages fetched and extracted.
521 Since the sysroot is not used, it would never get extracted.
522 This is another reason why a task-based approach is preferred over a
523 recipe-based approach, which would have to install the output from every task.
524 </para>
525 </section>
526
527 <section id='tips-and-tricks'>
528 <title>Tips and Tricks</title>
529
530 <para>
531 The code in the build system that supports incremental builds is not
532 simple code.
533 This section presents some tips and tricks that help you work around
534 issues related to shared state code.
535 </para>
536
537 <section id='debugging'>
538 <title>Debugging</title>
539
540 <para>
541 Seeing what metadata went into creating the input signature
542 of a shared state (sstate) task can be a useful debugging aid.
543 This information is available in signature information
544 (<filename>siginfo</filename>) files in
545 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>.
546 For information on how to view and interpret information in
547 <filename>siginfo</filename> files, see the
548 "<link linkend='usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</link>"
549 section.
550 </para>
551 </section>
552
553 <section id='invalidating-shared-state'>
554 <title>Invalidating Shared State</title>
555
556 <para>
557 The OpenEmbedded build system uses checksums and shared state
558 cache to avoid unnecessarily rebuilding tasks.
559 Collectively, this scheme is known as "shared state code."
560 </para>
561
562 <para>
563 As with all schemes, this one has some drawbacks.
564 It is possible that you could make implicit changes to your
565 code that the checksum calculations do not take into
566 account.
567 These implicit changes affect a task's output but do not trigger
568 the shared state code into rebuilding a recipe.
569 Consider an example during which a tool changes its output.
570 Assume that the output of <filename>rpmdeps</filename> changes.
571 The result of the change should be that all the
572 <filename>package</filename> and
573 <filename>package_write_rpm</filename> shared state cache
574 items become invalid.
575 However, because the change to the output is
576 external to the code and therefore implicit,
577 the associated shared state cache items do not become
578 invalidated.
579 In this case, the build process uses the cached items rather
580 than running the task again.
581 Obviously, these types of implicit changes can cause problems.
582 </para>
583
584 <para>
585 To avoid these problems during the build, you need to
586 understand the effects of any changes you make.
587 Realize that changes you make directly to a function
588 are automatically factored into the checksum calculation.
589 Thus, these explicit changes invalidate the associated area of
590 shared state cache.
591 However, you need to be aware of any implicit changes that
592 are not obvious changes to the code and could affect the output
593 of a given task.
594 </para>
595
596 <para>
597 When you identify an implicit change, you can easily take steps
598 to invalidate the cache and force the tasks to run.
599 The steps you can take are as simple as changing a function's
600 comments in the source code.
601 For example, to invalidate package shared state files, change
602 the comment statements of
603 <link linkend='ref-tasks-package'><filename>do_package</filename></link>
604 or the comments of one of the functions it calls.
605 Even though the change is purely cosmetic, it causes the
606 checksum to be recalculated and forces the OpenEmbedded build
607 system to run the task again.
608 </para>
609
610 <note>
611 For an example of a commit that makes a cosmetic change to
612 invalidate shared state, see this
613 <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
614 </note>
615 </section>
616 </section>
617</section>
618
619<section id='automatically-added-runtime-dependencies'> 16<section id='automatically-added-runtime-dependencies'>
620 <title>Automatically Added Runtime Dependencies</title> 17 <title>Automatically Added Runtime Dependencies</title>
621 18
diff --git a/documentation/ref-manual/usingpoky.xml b/documentation/ref-manual/usingpoky.xml
index f1fd452a06..73b5df88dd 100644
--- a/documentation/ref-manual/usingpoky.xml
+++ b/documentation/ref-manual/usingpoky.xml
@@ -535,7 +535,7 @@
535 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 535 ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1
536 </literallayout> 536 </literallayout>
537 For tasks that are accelerated through the shared state 537 For tasks that are accelerated through the shared state
538 (<link linkend='shared-state-cache'>sstate</link>) 538 (<ulink url='&YOCTO_DOCS_OVERVIEW_URL;#shared-state-cache'>sstate</ulink>)
539 cache, an additional <filename>siginfo</filename> file is 539 cache, an additional <filename>siginfo</filename> file is
540 written into 540 written into
541 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link> 541 <link linkend='var-SSTATE_DIR'><filename>SSTATE_DIR</filename></link>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-08 22:28:54 +0000