diff options
Diffstat (limited to 'documentation/dev-manual/limiting-resources.rst')
| -rw-r--r-- | documentation/dev-manual/limiting-resources.rst | 138 |
1 files changed, 0 insertions, 138 deletions
diff --git a/documentation/dev-manual/limiting-resources.rst b/documentation/dev-manual/limiting-resources.rst deleted file mode 100644 index 0a792689c3..0000000000 --- a/documentation/dev-manual/limiting-resources.rst +++ /dev/null | |||
| @@ -1,138 +0,0 @@ | |||
| 1 | .. SPDX-License-Identifier: CC-BY-SA-2.0-UK | ||
| 2 | |||
| 3 | Limiting the Host Resources Usage | ||
| 4 | ********************************* | ||
| 5 | |||
| 6 | While you sometimes need to :doc:`speed up a build | ||
| 7 | </dev-manual/speeding-up-build>`, you may also need to limit the resources used | ||
| 8 | by the :term:`OpenEmbedded Build System`, especially on shared infrastructures | ||
| 9 | where multiple users start heavy-load builds, or when building on low-power | ||
| 10 | machines. | ||
| 11 | |||
| 12 | This document aims at giving the different configuration variables available to | ||
| 13 | limit the resources used by the build system. These variables should be set from | ||
| 14 | a :term:`configuration file` and thus take effect over the entire build environment. | ||
| 15 | For each variable, also see the variable description in the glossary for more | ||
| 16 | details. | ||
| 17 | |||
| 18 | - :term:`BB_NUMBER_THREADS`: | ||
| 19 | |||
| 20 | This sets a hard limit on the number of threads :term:`BitBake` can run at the | ||
| 21 | same time. Lowering this value will set a limit to the number of | ||
| 22 | :term:`BitBake` threads, but will not prevent a single task from starting more | ||
| 23 | compilation threads (see :term:`PARALLEL_MAKE`). | ||
| 24 | |||
| 25 | - :term:`BB_NUMBER_PARSE_THREADS`: | ||
| 26 | |||
| 27 | Like :term:`BB_NUMBER_THREADS`, but this variable sets a limit on the number | ||
| 28 | of threads during the parsing of the environment (before executing tasks). | ||
| 29 | |||
| 30 | - :term:`PARALLEL_MAKE`: | ||
| 31 | |||
| 32 | This variable should be set in the form of ``-jN``, where ``N`` is a positive | ||
| 33 | integer. This integer controls the number of threads used when starting | ||
| 34 | ``make``. Note that this variable is not limited to the usage of ``make``, | ||
| 35 | but extends to the compilation (:ref:`ref-tasks-compile` task) commands | ||
| 36 | defined by the :ref:`ref-classes-meson`, :ref:`ref-classes-cmake` and such | ||
| 37 | classes. | ||
| 38 | |||
| 39 | If you want to have a different limit from the rest of the build for a | ||
| 40 | recipe, it is also possible to achieve with the following line added to your | ||
| 41 | ``local.conf`` :term:`configuration file`:: | ||
| 42 | |||
| 43 | PARALLEL_MAKE:pn-linux-yocto = "-j4" | ||
| 44 | |||
| 45 | The above example will limit the number of threads used by ``make`` for the | ||
| 46 | ``linux-yocto`` recipe to 4. | ||
| 47 | |||
| 48 | - :term:`PARALLEL_MAKEINST`: | ||
| 49 | |||
| 50 | Like :term:`PARALLEL_MAKE`, but this variable controls the number of threads | ||
| 51 | used during the :ref:`ref-tasks-install` task. | ||
| 52 | |||
| 53 | The default value of :term:`PARALLEL_MAKEINST` is the value of | ||
| 54 | :term:`PARALLEL_MAKE`. | ||
| 55 | |||
| 56 | .. note:: | ||
| 57 | |||
| 58 | While most of the variables in this document help to limit the CPU load, it | ||
| 59 | is also possible that the host system runs out of physical RAM when running | ||
| 60 | builds. This can trigger the out-of-memory killer and stop the related | ||
| 61 | processes abruptly. This can create strange looking failures in the output | ||
| 62 | log of the tasks in question. The out-of-memory killer only logs in the | ||
| 63 | kernel dmesg logs, so it is advised to monitor it closely with the ``dmesg`` | ||
| 64 | command when encountering unexpected failures during builds. | ||
| 65 | |||
| 66 | In these situations, lowering the value of :term:`PARALLEL_MAKE` and | ||
| 67 | :term:`BB_NUMBER_THREADS` is recommended. | ||
| 68 | |||
| 69 | - :term:`BB_PRESSURE_MAX_CPU`, :term:`BB_PRESSURE_MAX_IO` and | ||
| 70 | :term:`BB_PRESSURE_MAX_MEMORY`: | ||
| 71 | |||
| 72 | These variables control the limit of pressure (PSI as defined by | ||
| 73 | https://docs.kernel.org/accounting/psi.html) on the system, and will | ||
| 74 | limit the number of :term:`BitBake` threads dynamically depending on the | ||
| 75 | current pressure of the system. This also means that your host must support | ||
| 76 | the PSI kernel feature (otherwise see :term:`BB_LOADFACTOR_MAX` below). | ||
| 77 | |||
| 78 | These variables take a positive integer between 1 (extremely low limit) and | ||
| 79 | 1000000 (value unlikely ever reached). Setting an extremely low value, such | ||
| 80 | as 2, is not desirable as it will result in :term:`BitBake` limiting the | ||
| 81 | number of threads to 1 most of the time. | ||
| 82 | |||
| 83 | To determine a reasonable value to set for your host, follow the steps below: | ||
| 84 | |||
| 85 | #. In a Bash shell, start the following script, which will provide an | ||
| 86 | estimate of the current pressure on your host: | ||
| 87 | |||
| 88 | .. code-block:: bash | ||
| 89 | |||
| 90 | pressure="0" | ||
| 91 | while true; do | ||
| 92 | prev_pressure="$pressure" | ||
| 93 | pressure=$(head -1 /proc/pressure/cpu | cut -d' ' -f5 | cut -d'=' -f2) | ||
| 94 | echo $(( $pressure - $prev_pressure )) | ||
| 95 | sleep 1 | ||
| 96 | done | ||
| 97 | |||
| 98 | .. note:: | ||
| 99 | |||
| 100 | Change ``/proc/pressure/cpu`` to ``/proc/pressure/io`` or | ||
| 101 | ``/proc/pressure/memory`` to change the pressure type to monitor. | ||
| 102 | |||
| 103 | This script can be stopped by pressing Control + C. | ||
| 104 | |||
| 105 | #. Then, start a heavy-load build, for example:: | ||
| 106 | |||
| 107 | bitbake virtual/kernel -c compile -f | ||
| 108 | |||
| 109 | You can stop the build at anytime with Control + C. | ||
| 110 | |||
| 111 | #. Monitor the values printed on the console. These should indicate how the | ||
| 112 | pressure evolves during the build. You can take a value below the maximum | ||
| 113 | printed value as a starting point. | ||
| 114 | |||
| 115 | After setting initial values, :term:`BitBake` will print messages on the | ||
| 116 | console in the following format each time the current pressure exceeds of the | ||
| 117 | limit set by the above variables:: | ||
| 118 | |||
| 119 | Pressure status changed to CPU: True, IO: False, Mem: False (CPU: 1105.9/2.0, IO: 0.0/2.0, Mem: 0.0/2.0) - using 1/64 bitbake threads | ||
| 120 | |||
| 121 | Take a look at the value between parenthesis: ``CPU: 1105.9/2.0, IO: 0.0/2.0, | ||
| 122 | Mem: 0.0/2.0``. They correspond to the current pressure value for the CPU, IO | ||
| 123 | and memory respectively. If :term:`BitBake` prints these messages a lot, it | ||
| 124 | is likely that your pressure limit is too low, and thus can be raised to a | ||
| 125 | higher value. | ||
| 126 | |||
| 127 | - :term:`BB_LOADFACTOR_MAX`: | ||
| 128 | |||
| 129 | This variable will limit the number of threads :term:`BitBake` will start | ||
| 130 | by monitoring the current CPU load of the host system. :term:`BitBake` will | ||
| 131 | print the following when the limit set by :term:`BB_LOADFACTOR_MAX` is | ||
| 132 | reached:: | ||
| 133 | |||
| 134 | Load average limiting set to True as load average: 0.7188262939453125 - using 37/64 bitbake threads | ||
| 135 | |||
| 136 | This variable has no effect when any of :term:`BB_PRESSURE_MAX_CPU`, | ||
| 137 | :term:`BB_PRESSURE_MAX_IO` or :term:`BB_PRESSURE_MAX_MEMORY` is set, as it | ||
| 138 | was designed for systems that do not have pressure information available. | ||
