diff options
Diffstat (limited to 'documentation')
-rw-r--r-- | documentation/dev-manual/hashequivserver.rst | 129 | ||||
-rw-r--r-- | documentation/dev-manual/index.rst | 1 | ||||
-rw-r--r-- | documentation/dev-manual/start.rst | 4 | ||||
-rw-r--r-- | documentation/overview-manual/concepts.rst | 16 |
4 files changed, 137 insertions, 13 deletions
diff --git a/documentation/dev-manual/hashequivserver.rst b/documentation/dev-manual/hashequivserver.rst new file mode 100644 index 0000000000..c866f5001f --- /dev/null +++ b/documentation/dev-manual/hashequivserver.rst | |||
@@ -0,0 +1,129 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-SA-2.0-UK | ||
2 | |||
3 | Setting up a Hash Equivalence Server | ||
4 | ************************************ | ||
5 | |||
6 | A :ref:`overview-manual/concepts:Hash Equivalence` server can help reduce build | ||
7 | times by using the mechanism described in the :ref:`overview-manual/concepts:Hash Equivalence` | ||
8 | section of the Yocto Project Overview and Concepts Manual. | ||
9 | |||
10 | This document will guide you through the steps required to set up the reference | ||
11 | Hash Equivalence server provided by the :oe_git:`bitbake-hashserv | ||
12 | </bitbake/tree/bin/bitbake-hashserv>` script in :term:`BitBake`. | ||
13 | |||
14 | This guide will explain how to setup a local Hash Equivalence server and does | ||
15 | not explain how to setup the surrounding infrastructure to secure this server. | ||
16 | |||
17 | Hash Equivalence Server Setup | ||
18 | ============================= | ||
19 | |||
20 | From this point onwards, the commands displayed below are assumed to be run from | ||
21 | the :term:`BitBake` repository, which can be found :oe_git:`here </bitbake>`. | ||
22 | |||
23 | To start a basic Hash Equivalence server, one could simply run:: | ||
24 | |||
25 | ./bin/bitbake-hashserv | ||
26 | |||
27 | This will take all of the default options of the script, which are already | ||
28 | sufficient to start a local server. | ||
29 | |||
30 | Run ``./bin/bitbake-hashserv --help`` to see what options the script can take. | ||
31 | Some of the important ones are: | ||
32 | |||
33 | - ``--database`` controls the location of the hash server database (default: | ||
34 | ``./hashserver.db``). | ||
35 | |||
36 | - ``--bind`` controls the bind address of the server (default: | ||
37 | ``unix://./hashserver.sock``). | ||
38 | |||
39 | You can specify three types of addresses: | ||
40 | |||
41 | - ``unix://PATH``: will bind to a Unix socket at ``PATH``. | ||
42 | |||
43 | - ``wss://ADDRESS:PORT``: will bind to a Websocket at ``ADDRESS:PORT``. | ||
44 | |||
45 | - ``ADDRESS:PORT``: will bind to a raw TCP socket at ``ADDRESS:PORT``. | ||
46 | |||
47 | - ``--log`` can be used to control the logging level of the server (e.g. | ||
48 | ``INFO`` will print information about clients connection to the server). | ||
49 | |||
50 | - ``--upstream`` can be used to specify an upstream server to pull hashes from. | ||
51 | This has no default value, meaning no upstream server is used. | ||
52 | |||
53 | - ``--db-username`` and ``--db-password`` can be used to control the access to | ||
54 | the database. | ||
55 | |||
56 | - ``--read-only`` will disable hash reports from clients. | ||
57 | |||
58 | These variables can also be set from the environment from where it is being run. | ||
59 | Run ``./bin/bitbake-hashserv --help`` to get the variable names that you can | ||
60 | export. | ||
61 | |||
62 | .. warning:: | ||
63 | |||
64 | The shared Hash Equivalence server needs to be maintained together with the | ||
65 | :ref:`Shared State <overview-manual/concepts:Shared State>` cache. Otherwise, | ||
66 | the server could report Shared State hashes that only exist on specific | ||
67 | clients. | ||
68 | |||
69 | We therefore recommend that one Hash Equivalence server be set up to | ||
70 | correspond with a given Shared State cache, and to start this server | ||
71 | in *read-only mode* (``--read-only`` option), so that it doesn't store | ||
72 | equivalences for Shared State caches that are local to clients. | ||
73 | |||
74 | If there is no pre-existing Shared State Cache, the server should allow | ||
75 | hashes to be reported (no ``--read-only`` option) to create the initial | ||
76 | Hash Equivalence database. | ||
77 | |||
78 | Yocto Project Build Setup | ||
79 | ========================= | ||
80 | |||
81 | To use the server started in the previous section, set the following variables | ||
82 | in a :term:`configuration file`:: | ||
83 | |||
84 | BB_HASHSERVE = "<bind address>" | ||
85 | BB_SIGNATURE_HANDLER = "OEEquivHash" | ||
86 | |||
87 | The ``<bind address>`` value should be replaced to point to the server started | ||
88 | in the previous section. | ||
89 | |||
90 | See the documentation of :term:`BB_SIGNATURE_HANDLER` for more details on this | ||
91 | variable. | ||
92 | |||
93 | You can optionally specify an upstream server with :term:`BB_HASHSERVE_UPSTREAM` | ||
94 | variable. For example:: | ||
95 | |||
96 | BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws" | ||
97 | |||
98 | This will make the local server pull hashes from the upstream server. The | ||
99 | :term:`BB_HASHSERVE_UPSTREAM` only works when a server is running | ||
100 | (:term:`BB_HASHSERVE` is set). | ||
101 | |||
102 | To output debugging information on what is happening with Hash Equivalence when | ||
103 | builds are started, you can configure :term:`BitBake` logging as follows from a | ||
104 | :term:`configuration file`:: | ||
105 | |||
106 | BB_LOGCONFIG = "hashequiv.json" | ||
107 | |||
108 | With ``hashequiv.json`` containing the following content: | ||
109 | |||
110 | .. code-block:: json | ||
111 | |||
112 | { | ||
113 | "version": 1, | ||
114 | "loggers": { | ||
115 | "BitBake.SigGen.HashEquiv": { | ||
116 | "level": "VERBOSE", | ||
117 | "handlers": ["BitBake.verbconsole"] | ||
118 | }, | ||
119 | "BitBake.RunQueue.HashEquiv": { | ||
120 | "level": "VERBOSE", | ||
121 | "handlers": ["BitBake.verbconsole"] | ||
122 | } | ||
123 | } | ||
124 | } | ||
125 | |||
126 | This will make Hash Equivalence related changes be printed on the console, such | ||
127 | as:: | ||
128 | |||
129 | NOTE: Task <recipe.bb>:do_<task> unihash changed to dc0da29c62a2d78d8d853fbb9c173778fe7d6fa4a68c67494b17afffe8ca1894 | ||
diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst index 4abfa59e9d..ea528855a8 100644 --- a/documentation/dev-manual/index.rst +++ b/documentation/dev-manual/index.rst | |||
@@ -51,5 +51,6 @@ Yocto Project Development Tasks Manual | |||
51 | wayland | 51 | wayland |
52 | qemu | 52 | qemu |
53 | bblock | 53 | bblock |
54 | hashequivserver | ||
54 | 55 | ||
55 | .. include:: /boilerplate.rst | 56 | .. include:: /boilerplate.rst |
diff --git a/documentation/dev-manual/start.rst b/documentation/dev-manual/start.rst index dd0eebc0a5..77ba16e2f5 100644 --- a/documentation/dev-manual/start.rst +++ b/documentation/dev-manual/start.rst | |||
@@ -234,6 +234,10 @@ particular working environment and set of practices. | |||
234 | Yocto Project Overview and Concepts Manual for more details on the hash | 234 | Yocto Project Overview and Concepts Manual for more details on the hash |
235 | equivalence feature. | 235 | equivalence feature. |
236 | 236 | ||
237 | See the :doc:`/dev-manual/hashequivserver` section of the Yocto Project | ||
238 | Development Tasks Manual for details on how to setup a | ||
239 | :ref:`overview-manual/concepts:Hash Equivalence` server. | ||
240 | |||
237 | - Set up an Autobuilder and have it populate the sstate cache and | 241 | - Set up an Autobuilder and have it populate the sstate cache and |
238 | source directories. | 242 | source directories. |
239 | 243 | ||
diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst index e8569e4f53..397a8b84d0 100644 --- a/documentation/overview-manual/concepts.rst +++ b/documentation/overview-manual/concepts.rst | |||
@@ -2064,19 +2064,9 @@ on a Hash Equivalence server on a network, by setting:: | |||
2064 | 2064 | ||
2065 | BB_HASHSERVE = "<HOSTNAME>:<PORT>" | 2065 | BB_HASHSERVE = "<HOSTNAME>:<PORT>" |
2066 | 2066 | ||
2067 | .. note:: | 2067 | Setting up a Hash Equivalence server is described in the |
2068 | 2068 | :doc:`/dev-manual/hashequivserver` section of the Yocto Project Development | |
2069 | The shared Hash Equivalence server needs to be maintained together with the | 2069 | Tasks Manual. |
2070 | Shared State cache. Otherwise, the server could report Shared State hashes | ||
2071 | that only exist on specific clients. | ||
2072 | |||
2073 | We therefore recommend that one Hash Equivalence server be set up to | ||
2074 | correspond with a given Shared State cache, and to start this server | ||
2075 | in *read-only mode*, so that it doesn't store equivalences for | ||
2076 | Shared State caches that are local to clients. | ||
2077 | |||
2078 | See the :term:`BB_HASHSERVE` reference for details about starting | ||
2079 | a Hash Equivalence server. | ||
2080 | 2070 | ||
2081 | See the `video <https://www.youtube.com/watch?v=zXEdqGS62Wc>`__ | 2071 | See the `video <https://www.youtube.com/watch?v=zXEdqGS62Wc>`__ |
2082 | of Joshua Watt's `Hash Equivalence and Reproducible Builds | 2072 | of Joshua Watt's `Hash Equivalence and Reproducible Builds |