diff options
| -rw-r--r-- | documentation/dev-manual/hashequivserver.rst | 129 | ||||
| -rw-r--r-- | documentation/dev-manual/index.rst | 1 | ||||
| -rw-r--r-- | documentation/dev-manual/start.rst | 4 | ||||
| -rw-r--r-- | documentation/overview-manual/concepts.rst | 16 |
4 files changed, 137 insertions, 13 deletions
diff --git a/documentation/dev-manual/hashequivserver.rst b/documentation/dev-manual/hashequivserver.rst new file mode 100644 index 0000000000..c866f5001f --- /dev/null +++ b/documentation/dev-manual/hashequivserver.rst | |||
| @@ -0,0 +1,129 @@ | |||
| 1 | .. SPDX-License-Identifier: CC-BY-SA-2.0-UK | ||
| 2 | |||
| 3 | Setting up a Hash Equivalence Server | ||
| 4 | ************************************ | ||
| 5 | |||
| 6 | A :ref:`overview-manual/concepts:Hash Equivalence` server can help reduce build | ||
| 7 | times by using the mechanism described in the :ref:`overview-manual/concepts:Hash Equivalence` | ||
| 8 | section of the Yocto Project Overview and Concepts Manual. | ||
| 9 | |||
| 10 | This document will guide you through the steps required to set up the reference | ||
| 11 | Hash Equivalence server provided by the :oe_git:`bitbake-hashserv | ||
| 12 | </bitbake/tree/bin/bitbake-hashserv>` script in :term:`BitBake`. | ||
| 13 | |||
| 14 | This guide will explain how to setup a local Hash Equivalence server and does | ||
| 15 | not explain how to setup the surrounding infrastructure to secure this server. | ||
| 16 | |||
| 17 | Hash Equivalence Server Setup | ||
| 18 | ============================= | ||
| 19 | |||
| 20 | From this point onwards, the commands displayed below are assumed to be run from | ||
| 21 | the :term:`BitBake` repository, which can be found :oe_git:`here </bitbake>`. | ||
| 22 | |||
| 23 | To start a basic Hash Equivalence server, one could simply run:: | ||
| 24 | |||
| 25 | ./bin/bitbake-hashserv | ||
| 26 | |||
| 27 | This will take all of the default options of the script, which are already | ||
| 28 | sufficient to start a local server. | ||
| 29 | |||
| 30 | Run ``./bin/bitbake-hashserv --help`` to see what options the script can take. | ||
| 31 | Some of the important ones are: | ||
| 32 | |||
| 33 | - ``--database`` controls the location of the hash server database (default: | ||
| 34 | ``./hashserver.db``). | ||
| 35 | |||
| 36 | - ``--bind`` controls the bind address of the server (default: | ||
| 37 | ``unix://./hashserver.sock``). | ||
| 38 | |||
| 39 | You can specify three types of addresses: | ||
| 40 | |||
| 41 | - ``unix://PATH``: will bind to a Unix socket at ``PATH``. | ||
| 42 | |||
| 43 | - ``wss://ADDRESS:PORT``: will bind to a Websocket at ``ADDRESS:PORT``. | ||
| 44 | |||
| 45 | - ``ADDRESS:PORT``: will bind to a raw TCP socket at ``ADDRESS:PORT``. | ||
| 46 | |||
| 47 | - ``--log`` can be used to control the logging level of the server (e.g. | ||
| 48 | ``INFO`` will print information about clients connection to the server). | ||
| 49 | |||
| 50 | - ``--upstream`` can be used to specify an upstream server to pull hashes from. | ||
| 51 | This has no default value, meaning no upstream server is used. | ||
| 52 | |||
| 53 | - ``--db-username`` and ``--db-password`` can be used to control the access to | ||
| 54 | the database. | ||
| 55 | |||
| 56 | - ``--read-only`` will disable hash reports from clients. | ||
| 57 | |||
| 58 | These variables can also be set from the environment from where it is being run. | ||
| 59 | Run ``./bin/bitbake-hashserv --help`` to get the variable names that you can | ||
| 60 | export. | ||
| 61 | |||
| 62 | .. warning:: | ||
| 63 | |||
| 64 | The shared Hash Equivalence server needs to be maintained together with the | ||
| 65 | :ref:`Shared State <overview-manual/concepts:Shared State>` cache. Otherwise, | ||
| 66 | the server could report Shared State hashes that only exist on specific | ||
| 67 | clients. | ||
| 68 | |||
| 69 | We therefore recommend that one Hash Equivalence server be set up to | ||
| 70 | correspond with a given Shared State cache, and to start this server | ||
| 71 | in *read-only mode* (``--read-only`` option), so that it doesn't store | ||
| 72 | equivalences for Shared State caches that are local to clients. | ||
| 73 | |||
| 74 | If there is no pre-existing Shared State Cache, the server should allow | ||
| 75 | hashes to be reported (no ``--read-only`` option) to create the initial | ||
| 76 | Hash Equivalence database. | ||
| 77 | |||
| 78 | Yocto Project Build Setup | ||
| 79 | ========================= | ||
| 80 | |||
| 81 | To use the server started in the previous section, set the following variables | ||
| 82 | in a :term:`configuration file`:: | ||
| 83 | |||
| 84 | BB_HASHSERVE = "<bind address>" | ||
| 85 | BB_SIGNATURE_HANDLER = "OEEquivHash" | ||
| 86 | |||
| 87 | The ``<bind address>`` value should be replaced to point to the server started | ||
| 88 | in the previous section. | ||
| 89 | |||
| 90 | See the documentation of :term:`BB_SIGNATURE_HANDLER` for more details on this | ||
| 91 | variable. | ||
| 92 | |||
| 93 | You can optionally specify an upstream server with :term:`BB_HASHSERVE_UPSTREAM` | ||
| 94 | variable. For example:: | ||
| 95 | |||
| 96 | BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" | ||
| 97 | |||
| 98 | This will make the local server pull hashes from the upstream server. The | ||
| 99 | :term:`BB_HASHSERVE_UPSTREAM` only works when a server is running | ||
| 100 | (:term:`BB_HASHSERVE` is set). | ||
| 101 | |||
| 102 | To output debugging information on what is happening with Hash Equivalence when | ||
| 103 | builds are started, you can configure :term:`BitBake` logging as follows from a | ||
| 104 | :term:`configuration file`:: | ||
| 105 | |||
| 106 | BB_LOGCONFIG = "hashequiv.json" | ||
| 107 | |||
| 108 | With ``hashequiv.json`` containing the following content: | ||
| 109 | |||
| 110 | .. code-block:: json | ||
| 111 | |||
| 112 | { | ||
| 113 | "version": 1, | ||
| 114 | "loggers": { | ||
| 115 | "BitBake.SigGen.HashEquiv": { | ||
| 116 | "level": "VERBOSE", | ||
| 117 | "handlers": ["BitBake.verbconsole"] | ||
| 118 | }, | ||
| 119 | "BitBake.RunQueue.HashEquiv": { | ||
| 120 | "level": "VERBOSE", | ||
| 121 | "handlers": ["BitBake.verbconsole"] | ||
| 122 | } | ||
| 123 | } | ||
| 124 | } | ||
| 125 | |||
| 126 | This will make Hash Equivalence related changes be printed on the console, such | ||
| 127 | as:: | ||
| 128 | |||
| 129 | NOTE: Task <recipe.bb>:do_<task> unihash changed to dc0da29c62a2d78d8d853fbb9c173778fe7d6fa4a68c67494b17afffe8ca1894 | ||
diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst index 4abfa59e9d..ea528855a8 100644 --- a/documentation/dev-manual/index.rst +++ b/documentation/dev-manual/index.rst | |||
| @@ -51,5 +51,6 @@ Yocto Project Development Tasks Manual | |||
| 51 | wayland | 51 | wayland |
| 52 | qemu | 52 | qemu |
| 53 | bblock | 53 | bblock |
| 54 | hashequivserver | ||
| 54 | 55 | ||
| 55 | .. include:: /boilerplate.rst | 56 | .. include:: /boilerplate.rst |
diff --git a/documentation/dev-manual/start.rst b/documentation/dev-manual/start.rst index dd0eebc0a5..77ba16e2f5 100644 --- a/documentation/dev-manual/start.rst +++ b/documentation/dev-manual/start.rst | |||
| @@ -234,6 +234,10 @@ particular working environment and set of practices. | |||
| 234 | Yocto Project Overview and Concepts Manual for more details on the hash | 234 | Yocto Project Overview and Concepts Manual for more details on the hash |
| 235 | equivalence feature. | 235 | equivalence feature. |
| 236 | 236 | ||
| 237 | See the :doc:`/dev-manual/hashequivserver` section of the Yocto Project | ||
| 238 | Development Tasks Manual for details on how to setup a | ||
| 239 | :ref:`overview-manual/concepts:Hash Equivalence` server. | ||
| 240 | |||
| 237 | - Set up an Autobuilder and have it populate the sstate cache and | 241 | - Set up an Autobuilder and have it populate the sstate cache and |
| 238 | source directories. | 242 | source directories. |
| 239 | 243 | ||
diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst index e8569e4f53..397a8b84d0 100644 --- a/documentation/overview-manual/concepts.rst +++ b/documentation/overview-manual/concepts.rst | |||
| @@ -2064,19 +2064,9 @@ on a Hash Equivalence server on a network, by setting:: | |||
| 2064 | 2064 | ||
| 2065 | BB_HASHSERVE = "<HOSTNAME>:<PORT>" | 2065 | BB_HASHSERVE = "<HOSTNAME>:<PORT>" |
| 2066 | 2066 | ||
| 2067 | .. note:: | 2067 | Setting up a Hash Equivalence server is described in the |
| 2068 | 2068 | :doc:`/dev-manual/hashequivserver` section of the Yocto Project Development | |
| 2069 | The shared Hash Equivalence server needs to be maintained together with the | 2069 | Tasks Manual. |
| 2070 | Shared State cache. Otherwise, the server could report Shared State hashes | ||
| 2071 | that only exist on specific clients. | ||
| 2072 | |||
| 2073 | We therefore recommend that one Hash Equivalence server be set up to | ||
| 2074 | correspond with a given Shared State cache, and to start this server | ||
| 2075 | in *read-only mode*, so that it doesn't store equivalences for | ||
| 2076 | Shared State caches that are local to clients. | ||
| 2077 | |||
| 2078 | See the :term:`BB_HASHSERVE` reference for details about starting | ||
| 2079 | a Hash Equivalence server. | ||
| 2080 | 2070 | ||
| 2081 | See the `video <https://www.youtube.com/watch?v=zXEdqGS62Wc>`__ | 2071 | See the `video <https://www.youtube.com/watch?v=zXEdqGS62Wc>`__ |
| 2082 | of Joshua Watt's `Hash Equivalence and Reproducible Builds | 2072 | of Joshua Watt's `Hash Equivalence and Reproducible Builds |
