summaryrefslogtreecommitdiffstats
path: root/documentation
diff options
context:
space:
mode:
Diffstat (limited to 'documentation')
-rw-r--r--documentation/dev-manual/hashequivserver.rst129
-rw-r--r--documentation/dev-manual/index.rst1
-rw-r--r--documentation/dev-manual/start.rst4
-rw-r--r--documentation/overview-manual/concepts.rst16
4 files changed, 137 insertions, 13 deletions
diff --git a/documentation/dev-manual/hashequivserver.rst b/documentation/dev-manual/hashequivserver.rst
new file mode 100644
index 0000000000..c866f5001f
--- /dev/null
+++ b/documentation/dev-manual/hashequivserver.rst
@@ -0,0 +1,129 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Setting up a Hash Equivalence Server
4************************************
5
6A :ref:`overview-manual/concepts:Hash Equivalence` server can help reduce build
7times by using the mechanism described in the :ref:`overview-manual/concepts:Hash Equivalence`
8section of the Yocto Project Overview and Concepts Manual.
9
10This document will guide you through the steps required to set up the reference
11Hash Equivalence server provided by the :oe_git:`bitbake-hashserv
12</bitbake/tree/bin/bitbake-hashserv>` script in :term:`BitBake`.
13
14This guide will explain how to setup a local Hash Equivalence server and does
15not explain how to setup the surrounding infrastructure to secure this server.
16
17Hash Equivalence Server Setup
18=============================
19
20From this point onwards, the commands displayed below are assumed to be run from
21the :term:`BitBake` repository, which can be found :oe_git:`here </bitbake>`.
22
23To start a basic Hash Equivalence server, one could simply run::
24
25 ./bin/bitbake-hashserv
26
27This will take all of the default options of the script, which are already
28sufficient to start a local server.
29
30Run ``./bin/bitbake-hashserv --help`` to see what options the script can take.
31Some of the important ones are:
32
33- ``--database`` controls the location of the hash server database (default:
34 ``./hashserver.db``).
35
36- ``--bind`` controls the bind address of the server (default:
37 ``unix://./hashserver.sock``).
38
39 You can specify three types of addresses:
40
41 - ``unix://PATH``: will bind to a Unix socket at ``PATH``.
42
43 - ``wss://ADDRESS:PORT``: will bind to a Websocket at ``ADDRESS:PORT``.
44
45 - ``ADDRESS:PORT``: will bind to a raw TCP socket at ``ADDRESS:PORT``.
46
47- ``--log`` can be used to control the logging level of the server (e.g.
48 ``INFO`` will print information about clients connection to the server).
49
50- ``--upstream`` can be used to specify an upstream server to pull hashes from.
51 This has no default value, meaning no upstream server is used.
52
53- ``--db-username`` and ``--db-password`` can be used to control the access to
54 the database.
55
56- ``--read-only`` will disable hash reports from clients.
57
58These variables can also be set from the environment from where it is being run.
59Run ``./bin/bitbake-hashserv --help`` to get the variable names that you can
60export.
61
62.. warning::
63
64 The shared Hash Equivalence server needs to be maintained together with the
65 :ref:`Shared State <overview-manual/concepts:Shared State>` cache. Otherwise,
66 the server could report Shared State hashes that only exist on specific
67 clients.
68
69 We therefore recommend that one Hash Equivalence server be set up to
70 correspond with a given Shared State cache, and to start this server
71 in *read-only mode* (``--read-only`` option), so that it doesn't store
72 equivalences for Shared State caches that are local to clients.
73
74 If there is no pre-existing Shared State Cache, the server should allow
75 hashes to be reported (no ``--read-only`` option) to create the initial
76 Hash Equivalence database.
77
78Yocto Project Build Setup
79=========================
80
81To use the server started in the previous section, set the following variables
82in a :term:`configuration file`::
83
84 BB_HASHSERVE = "<bind address>"
85 BB_SIGNATURE_HANDLER = "OEEquivHash"
86
87The ``<bind address>`` value should be replaced to point to the server started
88in the previous section.
89
90See the documentation of :term:`BB_SIGNATURE_HANDLER` for more details on this
91variable.
92
93You can optionally specify an upstream server with :term:`BB_HASHSERVE_UPSTREAM`
94variable. For example::
95
96 BB_HASHSERVE_UPSTREAM = "wss://hashserv.yoctoproject.org/ws"
97
98This will make the local server pull hashes from the upstream server. The
99:term:`BB_HASHSERVE_UPSTREAM` only works when a server is running
100(:term:`BB_HASHSERVE` is set).
101
102To output debugging information on what is happening with Hash Equivalence when
103builds are started, you can configure :term:`BitBake` logging as follows from a
104:term:`configuration file`::
105
106 BB_LOGCONFIG = "hashequiv.json"
107
108With ``hashequiv.json`` containing the following content:
109
110.. code-block:: json
111
112 {
113 "version": 1,
114 "loggers": {
115 "BitBake.SigGen.HashEquiv": {
116 "level": "VERBOSE",
117 "handlers": ["BitBake.verbconsole"]
118 },
119 "BitBake.RunQueue.HashEquiv": {
120 "level": "VERBOSE",
121 "handlers": ["BitBake.verbconsole"]
122 }
123 }
124 }
125
126This will make Hash Equivalence related changes be printed on the console, such
127as::
128
129 NOTE: Task <recipe.bb>:do_<task> unihash changed to dc0da29c62a2d78d8d853fbb9c173778fe7d6fa4a68c67494b17afffe8ca1894
diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
index 4abfa59e9d..ea528855a8 100644
--- a/documentation/dev-manual/index.rst
+++ b/documentation/dev-manual/index.rst
@@ -51,5 +51,6 @@ Yocto Project Development Tasks Manual
51 wayland 51 wayland
52 qemu 52 qemu
53 bblock 53 bblock
54 hashequivserver
54 55
55.. include:: /boilerplate.rst 56.. include:: /boilerplate.rst
diff --git a/documentation/dev-manual/start.rst b/documentation/dev-manual/start.rst
index dd0eebc0a5..77ba16e2f5 100644
--- a/documentation/dev-manual/start.rst
+++ b/documentation/dev-manual/start.rst
@@ -234,6 +234,10 @@ particular working environment and set of practices.
234 Yocto Project Overview and Concepts Manual for more details on the hash 234 Yocto Project Overview and Concepts Manual for more details on the hash
235 equivalence feature. 235 equivalence feature.
236 236
237 See the :doc:`/dev-manual/hashequivserver` section of the Yocto Project
238 Development Tasks Manual for details on how to setup a
239 :ref:`overview-manual/concepts:Hash Equivalence` server.
240
237 - Set up an Autobuilder and have it populate the sstate cache and 241 - Set up an Autobuilder and have it populate the sstate cache and
238 source directories. 242 source directories.
239 243
diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst
index e8569e4f53..397a8b84d0 100644
--- a/documentation/overview-manual/concepts.rst
+++ b/documentation/overview-manual/concepts.rst
@@ -2064,19 +2064,9 @@ on a Hash Equivalence server on a network, by setting::
2064 2064
2065 BB_HASHSERVE = "<HOSTNAME>:<PORT>" 2065 BB_HASHSERVE = "<HOSTNAME>:<PORT>"
2066 2066
2067.. note:: 2067Setting up a Hash Equivalence server is described in the
2068 2068:doc:`/dev-manual/hashequivserver` section of the Yocto Project Development
2069 The shared Hash Equivalence server needs to be maintained together with the 2069Tasks Manual.
2070 Shared State cache. Otherwise, the server could report Shared State hashes
2071 that only exist on specific clients.
2072
2073 We therefore recommend that one Hash Equivalence server be set up to
2074 correspond with a given Shared State cache, and to start this server
2075 in *read-only mode*, so that it doesn't store equivalences for
2076 Shared State caches that are local to clients.
2077
2078 See the :term:`BB_HASHSERVE` reference for details about starting
2079 a Hash Equivalence server.
2080 2070
2081See the `video <https://www.youtube.com/watch?v=zXEdqGS62Wc>`__ 2071See the `video <https://www.youtube.com/watch?v=zXEdqGS62Wc>`__
2082of Joshua Watt's `Hash Equivalence and Reproducible Builds 2072of Joshua Watt's `Hash Equivalence and Reproducible Builds