diff options
Diffstat (limited to 'documentation/overview-manual/overview-concepts.xml')
-rw-r--r-- | documentation/overview-manual/overview-concepts.xml | 635 |
1 files changed, 635 insertions, 0 deletions
diff --git a/documentation/overview-manual/overview-concepts.xml b/documentation/overview-manual/overview-concepts.xml index 0a45cd7256..2d704923fa 100644 --- a/documentation/overview-manual/overview-concepts.xml +++ b/documentation/overview-manual/overview-concepts.xml | |||
@@ -471,6 +471,641 @@ | |||
471 | </note> | 471 | </note> |
472 | </section> | 472 | </section> |
473 | 473 | ||
474 | |||
475 | |||
476 | |||
477 | <section id="shared-state-cache"> | ||
478 | <title>Shared State Cache</title> | ||
479 | |||
480 | <para> | ||
481 | By design, the OpenEmbedded build system builds everything from | ||
482 | scratch unless BitBake can determine that parts do not need to be | ||
483 | rebuilt. | ||
484 | Fundamentally, building from scratch is attractive as it means all | ||
485 | parts are built fresh and there is no possibility of stale data | ||
486 | causing problems. | ||
487 | When developers hit problems, they typically default back to | ||
488 | building from scratch so they know the state of things from the | ||
489 | start. | ||
490 | </para> | ||
491 | |||
492 | <para> | ||
493 | Building an image from scratch is both an advantage and a | ||
494 | disadvantage to the process. | ||
495 | As mentioned in the previous paragraph, building from scratch | ||
496 | ensures that everything is current and starts from a known state. | ||
497 | However, building from scratch also takes much longer as it | ||
498 | generally means rebuilding things that do not necessarily need | ||
499 | to be rebuilt. | ||
500 | </para> | ||
501 | |||
502 | <para> | ||
503 | The Yocto Project implements shared state code that supports | ||
504 | incremental builds. | ||
505 | The implementation of the shared state code answers the following | ||
506 | questions that were fundamental roadblocks within the OpenEmbedded | ||
507 | incremental build support system: | ||
508 | <itemizedlist> | ||
509 | <listitem><para> | ||
510 | What pieces of the system have changed and what pieces have | ||
511 | not changed? | ||
512 | </para></listitem> | ||
513 | <listitem><para> | ||
514 | How are changed pieces of software removed and replaced? | ||
515 | </para></listitem> | ||
516 | <listitem><para> | ||
517 | How are pre-built components that do not need to be rebuilt | ||
518 | from scratch used when they are available? | ||
519 | </para></listitem> | ||
520 | </itemizedlist> | ||
521 | </para> | ||
522 | |||
523 | <para> | ||
524 | For the first question, the build system detects changes in the | ||
525 | "inputs" to a given task by creating a checksum (or signature) of | ||
526 | the task's inputs. | ||
527 | If the checksum changes, the system assumes the inputs have changed | ||
528 | and the task needs to be rerun. | ||
529 | For the second question, the shared state (sstate) code tracks | ||
530 | which tasks add which output to the build process. | ||
531 | This means the output from a given task can be removed, upgraded | ||
532 | or otherwise manipulated. | ||
533 | The third question is partly addressed by the solution for the | ||
534 | second question assuming the build system can fetch the sstate | ||
535 | objects from remote locations and install them if they are deemed | ||
536 | to be valid. | ||
537 | <note> | ||
538 | The OpenEmbedded build system does not maintain | ||
539 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> | ||
540 | information as part of the shared state packages. | ||
541 | Consequently, considerations exist that affect maintaining | ||
542 | shared state feeds. | ||
543 | For information on how the OpenEmbedded build system | ||
544 | works with packages and can track incrementing | ||
545 | <filename>PR</filename> information, see the | ||
546 | "<ulink url='&YOCTO_DOCS_DEV_URL;#automatically-incrementing-a-binary-package-revision-number'>Automatically Incrementing a Binary Package Revision Number</ulink>" | ||
547 | section in the Yocto Project Development Tasks Manual. | ||
548 | </note> | ||
549 | </para> | ||
550 | |||
551 | <para> | ||
552 | The rest of this section goes into detail about the overall | ||
553 | incremental build architecture, the checksums (signatures), shared | ||
554 | state, and some tips and tricks. | ||
555 | </para> | ||
556 | |||
557 | <section id='overall-architecture'> | ||
558 | <title>Overall Architecture</title> | ||
559 | |||
560 | <para> | ||
561 | When determining what parts of the system need to be built, | ||
562 | BitBake works on a per-task basis rather than a per-recipe | ||
563 | basis. | ||
564 | You might wonder why using a per-task basis is preferred over | ||
565 | a per-recipe basis. | ||
566 | To help explain, consider having the IPK packaging backend | ||
567 | enabled and then switching to DEB. | ||
568 | In this case, the | ||
569 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> | ||
570 | and | ||
571 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> | ||
572 | task outputs are still valid. | ||
573 | However, with a per-recipe approach, the build would not | ||
574 | include the <filename>.deb</filename> files. | ||
575 | Consequently, you would have to invalidate the whole build and | ||
576 | rerun it. | ||
577 | Rerunning everything is not the best solution. | ||
578 | Also, in this case, the core must be "taught" much about | ||
579 | specific tasks. | ||
580 | This methodology does not scale well and does not allow users | ||
581 | to easily add new tasks in layers or as external recipes | ||
582 | without touching the packaged-staging core. | ||
583 | </para> | ||
584 | </section> | ||
585 | |||
586 | <section id='overview-checksums'> | ||
587 | <title>Checksums (Signatures)</title> | ||
588 | |||
589 | <para> | ||
590 | The shared state code uses a checksum, which is a unique | ||
591 | signature of a task's inputs, to determine if a task needs to | ||
592 | be run again. | ||
593 | Because it is a change in a task's inputs that triggers a | ||
594 | rerun, the process needs to detect all the inputs to a given | ||
595 | task. | ||
596 | For shell tasks, this turns out to be fairly easy because | ||
597 | the build process generates a "run" shell script for each task | ||
598 | and it is possible to create a checksum that gives you a good | ||
599 | idea of when the task's data changes. | ||
600 | </para> | ||
601 | |||
602 | <para> | ||
603 | To complicate the problem, there are things that should not be | ||
604 | included in the checksum. | ||
605 | First, there is the actual specific build path of a given | ||
606 | task - the | ||
607 | <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>. | ||
608 | It does not matter if the work directory changes because it | ||
609 | should not affect the output for target packages. | ||
610 | Also, the build process has the objective of making native | ||
611 | or cross packages relocatable. | ||
612 | <note> | ||
613 | Both native and cross packages run on the build host. | ||
614 | However, cross packages generate output for the target | ||
615 | architecture. | ||
616 | </note> | ||
617 | The checksum therefore needs to exclude | ||
618 | <filename>WORKDIR</filename>. | ||
619 | The simplistic approach for excluding the work directory is to | ||
620 | set <filename>WORKDIR</filename> to some fixed value and | ||
621 | create the checksum for the "run" script. | ||
622 | </para> | ||
623 | |||
624 | <para> | ||
625 | Another problem results from the "run" scripts containing | ||
626 | functions that might or might not get called. | ||
627 | The incremental build solution contains code that figures out | ||
628 | dependencies between shell functions. | ||
629 | This code is used to prune the "run" scripts down to the | ||
630 | minimum set, thereby alleviating this problem and making the | ||
631 | "run" scripts much more readable as a bonus. | ||
632 | </para> | ||
633 | |||
634 | <para> | ||
635 | So far we have solutions for shell scripts. | ||
636 | What about Python tasks? | ||
637 | The same approach applies even though these tasks are more | ||
638 | difficult. | ||
639 | The process needs to figure out what variables a Python | ||
640 | function accesses and what functions it calls. | ||
641 | Again, the incremental build solution contains code that first | ||
642 | figures out the variable and function dependencies, and then | ||
643 | creates a checksum for the data used as the input to the task. | ||
644 | </para> | ||
645 | |||
646 | <para> | ||
647 | Like the <filename>WORKDIR</filename> case, situations exist | ||
648 | where dependencies should be ignored. | ||
649 | For these cases, you can instruct the build process to | ||
650 | ignore a dependency by using a line like the following: | ||
651 | <literallayout class='monospaced'> | ||
652 | PACKAGE_ARCHS[vardepsexclude] = "MACHINE" | ||
653 | </literallayout> | ||
654 | This example ensures that the | ||
655 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PACKAGE_ARCHS'><filename>PACKAGE_ARCHS</filename></ulink> | ||
656 | variable does not depend on the value of | ||
657 | <ulink url='&YOCTO_DOCS_REF_URL;#var-MACHINE'><filename>MACHINE</filename></ulink>, | ||
658 | even if it does reference it. | ||
659 | </para> | ||
660 | |||
661 | <para> | ||
662 | Equally, there are cases where we need to add dependencies | ||
663 | BitBake is not able to find. | ||
664 | You can accomplish this by using a line like the following: | ||
665 | <literallayout class='monospaced'> | ||
666 | PACKAGE_ARCHS[vardeps] = "MACHINE" | ||
667 | </literallayout> | ||
668 | This example explicitly adds the <filename>MACHINE</filename> | ||
669 | variable as a dependency for | ||
670 | <filename>PACKAGE_ARCHS</filename>. | ||
671 | </para> | ||
672 | |||
673 | <para> | ||
674 | Consider a case with in-line Python, for example, where | ||
675 | BitBake is not able to figure out dependencies. | ||
676 | When running in debug mode (i.e. using | ||
677 | <filename>-DDD</filename>), BitBake produces output when it | ||
678 | discovers something for which it cannot figure out dependencies. | ||
679 | The Yocto Project team has currently not managed to cover | ||
680 | those dependencies in detail and is aware of the need to fix | ||
681 | this situation. | ||
682 | </para> | ||
683 | |||
684 | <para> | ||
685 | Thus far, this section has limited discussion to the direct | ||
686 | inputs into a task. | ||
687 | Information based on direct inputs is referred to as the | ||
688 | "basehash" in the code. | ||
689 | However, there is still the question of a task's indirect | ||
690 | inputs - the things that were already built and present in the | ||
691 | <ulink url='&YOCTO_DOCS_REF_URL;#build-directory'>Build Directory</ulink>. | ||
692 | The checksum (or signature) for a particular task needs to add | ||
693 | the hashes of all the tasks on which the particular task | ||
694 | depends. | ||
695 | Choosing which dependencies to add is a policy decision. | ||
696 | However, the effect is to generate a master checksum that | ||
697 | combines the basehash and the hashes of the task's | ||
698 | dependencies. | ||
699 | </para> | ||
700 | |||
701 | <para> | ||
702 | At the code level, there are a variety of ways both the | ||
703 | basehash and the dependent task hashes can be influenced. | ||
704 | Within the BitBake configuration file, we can give BitBake | ||
705 | some extra information to help it construct the basehash. | ||
706 | The following statement effectively results in a list of | ||
707 | global variable dependency excludes - variables never | ||
708 | included in any checksum: | ||
709 | <literallayout class='monospaced'> | ||
710 | BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ | ||
711 | SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \ | ||
712 | USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ | ||
713 | PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ | ||
714 | CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_DISABLE LICENSE_PATH SDKPKGSUFFIX" | ||
715 | </literallayout> | ||
716 | The previous example excludes | ||
717 | <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink> | ||
718 | since that variable is actually constructed as a path within | ||
719 | <ulink url='&YOCTO_DOCS_REF_URL;#var-TMPDIR'><filename>TMPDIR</filename></ulink>, | ||
720 | which is on the whitelist. | ||
721 | </para> | ||
722 | |||
723 | <para> | ||
724 | The rules for deciding which hashes of dependent tasks to | ||
725 | include through dependency chains are more complex and are | ||
726 | generally accomplished with a Python function. | ||
727 | The code in <filename>meta/lib/oe/sstatesig.py</filename> shows | ||
728 | two examples of this and also illustrates how you can insert | ||
729 | your own policy into the system if so desired. | ||
730 | This file defines the two basic signature generators | ||
731 | <ulink url='&YOCTO_DOCS_REF_URL;#oe-core'>OE-Core</ulink> | ||
732 | uses: "OEBasic" and "OEBasicHash". | ||
733 | By default, there is a dummy "noop" signature handler enabled | ||
734 | in BitBake. | ||
735 | This means that behavior is unchanged from previous versions. | ||
736 | OE-Core uses the "OEBasicHash" signature handler by default | ||
737 | through this setting in the <filename>bitbake.conf</filename> | ||
738 | file: | ||
739 | <literallayout class='monospaced'> | ||
740 | BB_SIGNATURE_HANDLER ?= "OEBasicHash" | ||
741 | </literallayout> | ||
742 | The "OEBasicHash" <filename>BB_SIGNATURE_HANDLER</filename> | ||
743 | is the same as the "OEBasic" version but adds the task hash to | ||
744 | the stamp files. | ||
745 | This results in any | ||
746 | <ulink url='&YOCTO_DOCS_REF_URL;#metadata'>Metadata</ulink> | ||
747 | change that changes the task hash, automatically | ||
748 | causing the task to be run again. | ||
749 | This removes the need to bump | ||
750 | <ulink url='&YOCTO_DOCS_REF_URL;#var-PR'><filename>PR</filename></ulink> | ||
751 | values, and changes to Metadata automatically ripple across | ||
752 | the build. | ||
753 | </para> | ||
754 | |||
755 | <para> | ||
756 | It is also worth noting that the end result of these | ||
757 | signature generators is to make some dependency and hash | ||
758 | information available to the build. | ||
759 | This information includes: | ||
760 | <itemizedlist> | ||
761 | <listitem><para> | ||
762 | <filename>BB_BASEHASH_task-</filename><replaceable>taskname</replaceable>: | ||
763 | The base hashes for each task in the recipe. | ||
764 | </para></listitem> | ||
765 | <listitem><para> | ||
766 | <filename>BB_BASEHASH_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: | ||
767 | The base hashes for each dependent task. | ||
768 | </para></listitem> | ||
769 | <listitem><para> | ||
770 | <filename>BBHASHDEPS_</filename><replaceable>filename</replaceable><filename>:</filename><replaceable>taskname</replaceable>: | ||
771 | The task dependencies for each task. | ||
772 | </para></listitem> | ||
773 | <listitem><para> | ||
774 | <filename>BB_TASKHASH</filename>: | ||
775 | The hash of the currently running task. | ||
776 | </para></listitem> | ||
777 | </itemizedlist> | ||
778 | </para> | ||
779 | </section> | ||
780 | |||
781 | <section id='shared-state'> | ||
782 | <title>Shared State</title> | ||
783 | |||
784 | <para> | ||
785 | Checksums and dependencies, as discussed in the previous | ||
786 | section, solve half the problem of supporting a shared state. | ||
787 | The other part of the problem is being able to use checksum | ||
788 | information during the build and being able to reuse or rebuild | ||
789 | specific components. | ||
790 | </para> | ||
791 | |||
792 | <para> | ||
793 | The | ||
794 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink> | ||
795 | class is a relatively generic implementation of how to | ||
796 | "capture" a snapshot of a given task. | ||
797 | The idea is that the build process does not care about the | ||
798 | source of a task's output. | ||
799 | Output could be freshly built or it could be downloaded and | ||
800 | unpacked from somewhere - the build process does not need to | ||
801 | worry about its origin. | ||
802 | </para> | ||
803 | |||
804 | <para> | ||
805 | There are two types of output, one is just about creating a | ||
806 | directory in | ||
807 | <ulink url='&YOCTO_DOCS_REF_URL;#var-WORKDIR'><filename>WORKDIR</filename></ulink>. | ||
808 | A good example is the output of either | ||
809 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-install'><filename>do_install</filename></ulink> | ||
810 | or | ||
811 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink>. | ||
812 | The other type of output occurs when a set of data is merged | ||
813 | into a shared directory tree such as the sysroot. | ||
814 | </para> | ||
815 | |||
816 | <para> | ||
817 | The Yocto Project team has tried to keep the details of the | ||
818 | implementation hidden in <filename>sstate</filename> class. | ||
819 | From a user's perspective, adding shared state wrapping to a task | ||
820 | is as simple as this | ||
821 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink> | ||
822 | example taken from the | ||
823 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-deploy'><filename>deploy</filename></ulink> | ||
824 | class: | ||
825 | <literallayout class='monospaced'> | ||
826 | DEPLOYDIR = "${WORKDIR}/deploy-${PN}" | ||
827 | SSTATETASKS += "do_deploy" | ||
828 | do_deploy[sstate-inputdirs] = "${DEPLOYDIR}" | ||
829 | do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" | ||
830 | |||
831 | python do_deploy_setscene () { | ||
832 | sstate_setscene(d) | ||
833 | } | ||
834 | addtask do_deploy_setscene | ||
835 | do_deploy[dirs] = "${DEPLOYDIR} ${B}" | ||
836 | </literallayout> | ||
837 | The following list explains the previous example: | ||
838 | <itemizedlist> | ||
839 | <listitem><para> | ||
840 | Adding "do_deploy" to <filename>SSTATETASKS</filename> | ||
841 | adds some required sstate-related processing, which is | ||
842 | implemented in the | ||
843 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-sstate'><filename>sstate</filename></ulink> | ||
844 | class, to before and after the | ||
845 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-deploy'><filename>do_deploy</filename></ulink> | ||
846 | task. | ||
847 | </para></listitem> | ||
848 | <listitem><para> | ||
849 | The | ||
850 | <filename>do_deploy[sstate-inputdirs] = "${DEPLOYDIR}"</filename> | ||
851 | declares that <filename>do_deploy</filename> places its | ||
852 | output in <filename>${DEPLOYDIR}</filename> when run | ||
853 | normally (i.e. when not using the sstate cache). | ||
854 | This output becomes the input to the shared state cache. | ||
855 | </para></listitem> | ||
856 | <listitem><para> | ||
857 | The | ||
858 | <filename>do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"</filename> | ||
859 | line causes the contents of the shared state cache to be | ||
860 | copied to <filename>${DEPLOY_DIR_IMAGE}</filename>. | ||
861 | <note> | ||
862 | If <filename>do_deploy</filename> is not already in | ||
863 | the shared state cache or if its input checksum | ||
864 | (signature) has changed from when the output was | ||
865 | cached, the task will be run to populate the shared | ||
866 | state cache, after which the contents of the shared | ||
867 | state cache is copied to | ||
868 | <filename>${DEPLOY_DIR_IMAGE}</filename>. | ||
869 | If <filename>do_deploy</filename> is in the shared | ||
870 | state cache and its signature indicates that the | ||
871 | cached output is still valid (i.e. if no | ||
872 | relevant task inputs have changed), then the | ||
873 | contents of the shared state cache will be copied | ||
874 | directly to | ||
875 | <filename>${DEPLOY_DIR_IMAGE}</filename> by the | ||
876 | <filename>do_deploy_setscene</filename> task | ||
877 | instead, skipping the | ||
878 | <filename>do_deploy</filename> task. | ||
879 | </note> | ||
880 | </para></listitem> | ||
881 | <listitem><para> | ||
882 | The following task definition is glue logic needed to | ||
883 | make the previous settings effective: | ||
884 | <literallayout class='monospaced'> | ||
885 | python do_deploy_setscene () { | ||
886 | sstate_setscene(d) | ||
887 | } | ||
888 | addtask do_deploy_setscene | ||
889 | </literallayout> | ||
890 | <filename>sstate_setscene()</filename> takes the flags | ||
891 | above as input and accelerates the | ||
892 | <filename>do_deploy</filename> task through the | ||
893 | shared state cache if possible. | ||
894 | If the task was accelerated, | ||
895 | <filename>sstate_setscene()</filename> returns True. | ||
896 | Otherwise, it returns False, and the normal | ||
897 | <filename>do_deploy</filename> task runs. | ||
898 | For more information, see the | ||
899 | "<ulink url='&YOCTO_DOCS_BB_URL;#setscene'>setscene</ulink>" | ||
900 | section in the BitBake User Manual. | ||
901 | </para></listitem> | ||
902 | <listitem><para> | ||
903 | The <filename>do_deploy[dirs] = "${DEPLOYDIR} ${B}"</filename> | ||
904 | line creates <filename>${DEPLOYDIR}</filename> and | ||
905 | <filename>${B}</filename> before the | ||
906 | <filename>do_deploy</filename> task runs, and also sets | ||
907 | the current working directory of | ||
908 | <filename>do_deploy</filename> to | ||
909 | <filename>${B}</filename>. | ||
910 | For more information, see the | ||
911 | "<ulink url='&YOCTO_DOCS_BB_URL;#variable-flags'>Variable Flags</ulink>" | ||
912 | section in the BitBake User Manual. | ||
913 | <note> | ||
914 | In cases where | ||
915 | <filename>sstate-inputdirs</filename> and | ||
916 | <filename>sstate-outputdirs</filename> would be the | ||
917 | same, you can use | ||
918 | <filename>sstate-plaindirs</filename>. | ||
919 | For example, to preserve the | ||
920 | <filename>${PKGD}</filename> and | ||
921 | <filename>${PKGDEST}</filename> output from the | ||
922 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> | ||
923 | task, use the following: | ||
924 | <literallayout class='monospaced'> | ||
925 | do_package[sstate-plaindirs] = "${PKGD} ${PKGDEST}" | ||
926 | </literallayout> | ||
927 | </note> | ||
928 | </para></listitem> | ||
929 | <listitem><para> | ||
930 | <filename>sstate-inputdirs</filename> and | ||
931 | <filename>sstate-outputdirs</filename> can also be used | ||
932 | with multiple directories. | ||
933 | For example, the following declares | ||
934 | <filename>PKGDESTWORK</filename> and | ||
935 | <filename>SHLIBWORK</filename> as shared state | ||
936 | input directories, which populates the shared state | ||
937 | cache, and <filename>PKGDATA_DIR</filename> and | ||
938 | <filename>SHLIBSDIR</filename> as the corresponding | ||
939 | shared state output directories: | ||
940 | <literallayout class='monospaced'> | ||
941 | do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}" | ||
942 | do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}" | ||
943 | </literallayout> | ||
944 | </para></listitem> | ||
945 | <listitem><para> | ||
946 | These methods also include the ability to take a | ||
947 | lockfile when manipulating shared state directory | ||
948 | structures, for cases where file additions or removals | ||
949 | are sensitive: | ||
950 | <literallayout class='monospaced'> | ||
951 | do_package[sstate-lockfile] = "${PACKAGELOCK}" | ||
952 | </literallayout> | ||
953 | </para></listitem> | ||
954 | </itemizedlist> | ||
955 | </para> | ||
956 | |||
957 | <para> | ||
958 | Behind the scenes, the shared state code works by looking in | ||
959 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink> | ||
960 | and | ||
961 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_MIRRORS'><filename>SSTATE_MIRRORS</filename></ulink> | ||
962 | for shared state files. | ||
963 | Here is an example: | ||
964 | <literallayout class='monospaced'> | ||
965 | SSTATE_MIRRORS ?= "\ | ||
966 | file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \ | ||
967 | file://.* file:///some/local/dir/sstate/PATH" | ||
968 | </literallayout> | ||
969 | <note> | ||
970 | The shared state directory | ||
971 | (<filename>SSTATE_DIR</filename>) is organized into | ||
972 | two-character subdirectories, where the subdirectory | ||
973 | names are based on the first two characters of the hash. | ||
974 | If the shared state directory structure for a mirror has the | ||
975 | same structure as <filename>SSTATE_DIR</filename>, you must | ||
976 | specify "PATH" as part of the URI to enable the build system | ||
977 | to map to the appropriate subdirectory. | ||
978 | </note> | ||
979 | </para> | ||
980 | |||
981 | <para> | ||
982 | The shared state package validity can be detected just by | ||
983 | looking at the filename since the filename contains the task | ||
984 | checksum (or signature) as described earlier in this section. | ||
985 | If a valid shared state package is found, the build process | ||
986 | downloads it and uses it to accelerate the task. | ||
987 | </para> | ||
988 | |||
989 | <para> | ||
990 | The build processes use the <filename>*_setscene</filename> | ||
991 | tasks for the task acceleration phase. | ||
992 | BitBake goes through this phase before the main execution | ||
993 | code and tries to accelerate any tasks for which it can find | ||
994 | shared state packages. | ||
995 | If a shared state package for a task is available, the | ||
996 | shared state package is used. | ||
997 | This means the task and any tasks on which it is dependent | ||
998 | are not executed. | ||
999 | </para> | ||
1000 | |||
1001 | <para> | ||
1002 | As a real world example, the aim is when building an IPK-based | ||
1003 | image, only the | ||
1004 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package_write_ipk'><filename>do_package_write_ipk</filename></ulink> | ||
1005 | tasks would have their shared state packages fetched and | ||
1006 | extracted. | ||
1007 | Since the sysroot is not used, it would never get extracted. | ||
1008 | This is another reason why a task-based approach is preferred | ||
1009 | over a recipe-based approach, which would have to install the | ||
1010 | output from every task. | ||
1011 | </para> | ||
1012 | </section> | ||
1013 | |||
1014 | <section id='tips-and-tricks'> | ||
1015 | <title>Tips and Tricks</title> | ||
1016 | |||
1017 | <para> | ||
1018 | The code in the build system that supports incremental builds | ||
1019 | is not simple code. | ||
1020 | This section presents some tips and tricks that help you work | ||
1021 | around issues related to shared state code. | ||
1022 | </para> | ||
1023 | |||
1024 | <section id='overview-debugging'> | ||
1025 | <title>Debugging</title> | ||
1026 | |||
1027 | <para> | ||
1028 | Seeing what metadata went into creating the input signature | ||
1029 | of a shared state (sstate) task can be a useful debugging | ||
1030 | aid. | ||
1031 | This information is available in signature information | ||
1032 | (<filename>siginfo</filename>) files in | ||
1033 | <ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>. | ||
1034 | For information on how to view and interpret information in | ||
1035 | <filename>siginfo</filename> files, see the | ||
1036 | "<ulink url='&YOCTO_DOCS_REF_URL;#usingpoky-viewing-task-variable-dependencies'>Viewing Task Variable Dependencies</ulink>" | ||
1037 | section in the Yocto Project Reference Manual. | ||
1038 | </para> | ||
1039 | </section> | ||
1040 | |||
1041 | <section id='invalidating-shared-state'> | ||
1042 | <title>Invalidating Shared State</title> | ||
1043 | |||
1044 | <para> | ||
1045 | The OpenEmbedded build system uses checksums and shared | ||
1046 | state cache to avoid unnecessarily rebuilding tasks. | ||
1047 | Collectively, this scheme is known as "shared state code." | ||
1048 | </para> | ||
1049 | |||
1050 | <para> | ||
1051 | As with all schemes, this one has some drawbacks. | ||
1052 | It is possible that you could make implicit changes to your | ||
1053 | code that the checksum calculations do not take into | ||
1054 | account. | ||
1055 | These implicit changes affect a task's output but do not | ||
1056 | trigger the shared state code into rebuilding a recipe. | ||
1057 | Consider an example during which a tool changes its output. | ||
1058 | Assume that the output of <filename>rpmdeps</filename> | ||
1059 | changes. | ||
1060 | The result of the change should be that all the | ||
1061 | <filename>package</filename> and | ||
1062 | <filename>package_write_rpm</filename> shared state cache | ||
1063 | items become invalid. | ||
1064 | However, because the change to the output is | ||
1065 | external to the code and therefore implicit, | ||
1066 | the associated shared state cache items do not become | ||
1067 | invalidated. | ||
1068 | In this case, the build process uses the cached items | ||
1069 | rather than running the task again. | ||
1070 | Obviously, these types of implicit changes can cause | ||
1071 | problems. | ||
1072 | </para> | ||
1073 | |||
1074 | <para> | ||
1075 | To avoid these problems during the build, you need to | ||
1076 | understand the effects of any changes you make. | ||
1077 | Realize that changes you make directly to a function | ||
1078 | are automatically factored into the checksum calculation. | ||
1079 | Thus, these explicit changes invalidate the associated | ||
1080 | area of shared state cache. | ||
1081 | However, you need to be aware of any implicit changes that | ||
1082 | are not obvious changes to the code and could affect | ||
1083 | the output of a given task. | ||
1084 | </para> | ||
1085 | |||
1086 | <para> | ||
1087 | When you identify an implicit change, you can easily | ||
1088 | take steps to invalidate the cache and force the tasks | ||
1089 | to run. | ||
1090 | The steps you can take are as simple as changing a | ||
1091 | function's comments in the source code. | ||
1092 | For example, to invalidate package shared state files, | ||
1093 | change the comment statements of | ||
1094 | <ulink url='&YOCTO_DOCS_REF_URL;#ref-tasks-package'><filename>do_package</filename></ulink> | ||
1095 | or the comments of one of the functions it calls. | ||
1096 | Even though the change is purely cosmetic, it causes the | ||
1097 | checksum to be recalculated and forces the OpenEmbedded | ||
1098 | build system to run the task again. | ||
1099 | <note> | ||
1100 | For an example of a commit that makes a cosmetic | ||
1101 | change to invalidate shared state, see this | ||
1102 | <ulink url='&YOCTO_GIT_URL;/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>. | ||
1103 | </note> | ||
1104 | </para> | ||
1105 | </section> | ||
1106 | </section> | ||
1107 | </section> | ||
1108 | |||
474 | <section id='x32'> | 1109 | <section id='x32'> |
475 | <title>x32 psABI</title> | 1110 | <title>x32 psABI</title> |
476 | 1111 | ||