diff options
Diffstat (limited to 'documentation/dev-manual/debugging.rst')
-rw-r--r-- | documentation/dev-manual/debugging.rst | 1271 |
1 files changed, 1271 insertions, 0 deletions
diff --git a/documentation/dev-manual/debugging.rst b/documentation/dev-manual/debugging.rst new file mode 100644 index 0000000000..92458a0c37 --- /dev/null +++ b/documentation/dev-manual/debugging.rst | |||
@@ -0,0 +1,1271 @@ | |||
1 | .. SPDX-License-Identifier: CC-BY-SA-2.0-UK | ||
2 | |||
3 | Debugging Tools and Techniques | ||
4 | ****************************** | ||
5 | |||
6 | The exact method for debugging build failures depends on the nature of | ||
7 | the problem and on the system's area from which the bug originates. | ||
8 | Standard debugging practices such as comparison against the last known | ||
9 | working version with examination of the changes and the re-application | ||
10 | of steps to identify the one causing the problem are valid for the Yocto | ||
11 | Project just as they are for any other system. Even though it is | ||
12 | impossible to detail every possible potential failure, this section | ||
13 | provides some general tips to aid in debugging given a variety of | ||
14 | situations. | ||
15 | |||
16 | .. note:: | ||
17 | |||
18 | A useful feature for debugging is the error reporting tool. | ||
19 | Configuring the Yocto Project to use this tool causes the | ||
20 | OpenEmbedded build system to produce error reporting commands as part | ||
21 | of the console output. You can enter the commands after the build | ||
22 | completes to log error information into a common database, that can | ||
23 | help you figure out what might be going wrong. For information on how | ||
24 | to enable and use this feature, see the | ||
25 | ":ref:`dev-manual/error-reporting-tool:using the error reporting tool`" | ||
26 | section. | ||
27 | |||
28 | The following list shows the debugging topics in the remainder of this | ||
29 | section: | ||
30 | |||
31 | - ":ref:`dev-manual/debugging:viewing logs from failed tasks`" describes | ||
32 | how to find and view logs from tasks that failed during the build | ||
33 | process. | ||
34 | |||
35 | - ":ref:`dev-manual/debugging:viewing variable values`" describes how to | ||
36 | use the BitBake ``-e`` option to examine variable values after a | ||
37 | recipe has been parsed. | ||
38 | |||
39 | - ":ref:`dev-manual/debugging:viewing package information with \`\`oe-pkgdata-util\`\``" | ||
40 | describes how to use the ``oe-pkgdata-util`` utility to query | ||
41 | :term:`PKGDATA_DIR` and | ||
42 | display package-related information for built packages. | ||
43 | |||
44 | - ":ref:`dev-manual/debugging:viewing dependencies between recipes and tasks`" | ||
45 | describes how to use the BitBake ``-g`` option to display recipe | ||
46 | dependency information used during the build. | ||
47 | |||
48 | - ":ref:`dev-manual/debugging:viewing task variable dependencies`" describes | ||
49 | how to use the ``bitbake-dumpsig`` command in conjunction with key | ||
50 | subdirectories in the :term:`Build Directory` to determine variable | ||
51 | dependencies. | ||
52 | |||
53 | - ":ref:`dev-manual/debugging:running specific tasks`" describes | ||
54 | how to use several BitBake options (e.g. ``-c``, ``-C``, and ``-f``) | ||
55 | to run specific tasks in the build chain. It can be useful to run | ||
56 | tasks "out-of-order" when trying isolate build issues. | ||
57 | |||
58 | - ":ref:`dev-manual/debugging:general BitBake problems`" describes how | ||
59 | to use BitBake's ``-D`` debug output option to reveal more about what | ||
60 | BitBake is doing during the build. | ||
61 | |||
62 | - ":ref:`dev-manual/debugging:building with no dependencies`" | ||
63 | describes how to use the BitBake ``-b`` option to build a recipe | ||
64 | while ignoring dependencies. | ||
65 | |||
66 | - ":ref:`dev-manual/debugging:recipe logging mechanisms`" | ||
67 | describes how to use the many recipe logging functions to produce | ||
68 | debugging output and report errors and warnings. | ||
69 | |||
70 | - ":ref:`dev-manual/debugging:debugging parallel make races`" | ||
71 | describes how to debug situations where the build consists of several | ||
72 | parts that are run simultaneously and when the output or result of | ||
73 | one part is not ready for use with a different part of the build that | ||
74 | depends on that output. | ||
75 | |||
76 | - ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) remotely`" | ||
77 | describes how to use GDB to allow you to examine running programs, which can | ||
78 | help you fix problems. | ||
79 | |||
80 | - ":ref:`dev-manual/debugging:debugging with the gnu project debugger (gdb) on the target`" | ||
81 | describes how to use GDB directly on target hardware for debugging. | ||
82 | |||
83 | - ":ref:`dev-manual/debugging:other debugging tips`" describes | ||
84 | miscellaneous debugging tips that can be useful. | ||
85 | |||
86 | Viewing Logs from Failed Tasks | ||
87 | ============================== | ||
88 | |||
89 | You can find the log for a task in the file | ||
90 | ``${``\ :term:`WORKDIR`\ ``}/temp/log.do_``\ `taskname`. | ||
91 | For example, the log for the | ||
92 | :ref:`ref-tasks-compile` task of the | ||
93 | QEMU minimal image for the x86 machine (``qemux86``) might be in | ||
94 | ``tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp/log.do_compile``. | ||
95 | To see the commands :term:`BitBake` ran | ||
96 | to generate a log, look at the corresponding ``run.do_``\ `taskname` file | ||
97 | in the same directory. | ||
98 | |||
99 | ``log.do_``\ `taskname` and ``run.do_``\ `taskname` are actually symbolic | ||
100 | links to ``log.do_``\ `taskname`\ ``.``\ `pid` and | ||
101 | ``log.run_``\ `taskname`\ ``.``\ `pid`, where `pid` is the PID the task had | ||
102 | when it ran. The symlinks always point to the files corresponding to the | ||
103 | most recent run. | ||
104 | |||
105 | Viewing Variable Values | ||
106 | ======================= | ||
107 | |||
108 | Sometimes you need to know the value of a variable as a result of | ||
109 | BitBake's parsing step. This could be because some unexpected behavior | ||
110 | occurred in your project. Perhaps an attempt to :ref:`modify a variable | ||
111 | <bitbake-user-manual/bitbake-user-manual-metadata:modifying existing | ||
112 | variables>` did not work out as expected. | ||
113 | |||
114 | BitBake's ``-e`` option is used to display variable values after | ||
115 | parsing. The following command displays the variable values after the | ||
116 | configuration files (i.e. ``local.conf``, ``bblayers.conf``, | ||
117 | ``bitbake.conf`` and so forth) have been parsed:: | ||
118 | |||
119 | $ bitbake -e | ||
120 | |||
121 | The following command displays variable values after a specific recipe has | ||
122 | been parsed. The variables include those from the configuration as well:: | ||
123 | |||
124 | $ bitbake -e recipename | ||
125 | |||
126 | .. note:: | ||
127 | |||
128 | Each recipe has its own private set of variables (datastore). | ||
129 | Internally, after parsing the configuration, a copy of the resulting | ||
130 | datastore is made prior to parsing each recipe. This copying implies | ||
131 | that variables set in one recipe will not be visible to other | ||
132 | recipes. | ||
133 | |||
134 | Likewise, each task within a recipe gets a private datastore based on | ||
135 | the recipe datastore, which means that variables set within one task | ||
136 | will not be visible to other tasks. | ||
137 | |||
138 | In the output of ``bitbake -e``, each variable is preceded by a | ||
139 | description of how the variable got its value, including temporary | ||
140 | values that were later overridden. This description also includes | ||
141 | variable flags (varflags) set on the variable. The output can be very | ||
142 | helpful during debugging. | ||
143 | |||
144 | Variables that are exported to the environment are preceded by | ||
145 | ``export`` in the output of ``bitbake -e``. See the following example:: | ||
146 | |||
147 | export CC="i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/ulf/poky/build/tmp/sysroots/qemux86" | ||
148 | |||
149 | In addition to variable values, the output of the ``bitbake -e`` and | ||
150 | ``bitbake -e`` recipe commands includes the following information: | ||
151 | |||
152 | - The output starts with a tree listing all configuration files and | ||
153 | classes included globally, recursively listing the files they include | ||
154 | or inherit in turn. Much of the behavior of the OpenEmbedded build | ||
155 | system (including the behavior of the :ref:`ref-manual/tasks:normal recipe build tasks`) is | ||
156 | implemented in the :ref:`ref-classes-base` class and the | ||
157 | classes it inherits, rather than being built into BitBake itself. | ||
158 | |||
159 | - After the variable values, all functions appear in the output. For | ||
160 | shell functions, variables referenced within the function body are | ||
161 | expanded. If a function has been modified using overrides or using | ||
162 | override-style operators like ``:append`` and ``:prepend``, then the | ||
163 | final assembled function body appears in the output. | ||
164 | |||
165 | Viewing Package Information with ``oe-pkgdata-util`` | ||
166 | ==================================================== | ||
167 | |||
168 | You can use the ``oe-pkgdata-util`` command-line utility to query | ||
169 | :term:`PKGDATA_DIR` and display | ||
170 | various package-related information. When you use the utility, you must | ||
171 | use it to view information on packages that have already been built. | ||
172 | |||
173 | Here are a few of the available ``oe-pkgdata-util`` subcommands. | ||
174 | |||
175 | .. note:: | ||
176 | |||
177 | You can use the standard \* and ? globbing wildcards as part of | ||
178 | package names and paths. | ||
179 | |||
180 | - ``oe-pkgdata-util list-pkgs [pattern]``: Lists all packages | ||
181 | that have been built, optionally limiting the match to packages that | ||
182 | match pattern. | ||
183 | |||
184 | - ``oe-pkgdata-util list-pkg-files package ...``: Lists the | ||
185 | files and directories contained in the given packages. | ||
186 | |||
187 | .. note:: | ||
188 | |||
189 | A different way to view the contents of a package is to look at | ||
190 | the | ||
191 | ``${``\ :term:`WORKDIR`\ ``}/packages-split`` | ||
192 | directory of the recipe that generates the package. This directory | ||
193 | is created by the | ||
194 | :ref:`ref-tasks-package` task | ||
195 | and has one subdirectory for each package the recipe generates, | ||
196 | which contains the files stored in that package. | ||
197 | |||
198 | If you want to inspect the ``${WORKDIR}/packages-split`` | ||
199 | directory, make sure that :ref:`ref-classes-rm-work` is not | ||
200 | enabled when you build the recipe. | ||
201 | |||
202 | - ``oe-pkgdata-util find-path path ...``: Lists the names of | ||
203 | the packages that contain the given paths. For example, the following | ||
204 | tells us that ``/usr/share/man/man1/make.1`` is contained in the | ||
205 | ``make-doc`` package:: | ||
206 | |||
207 | $ oe-pkgdata-util find-path /usr/share/man/man1/make.1 | ||
208 | make-doc: /usr/share/man/man1/make.1 | ||
209 | |||
210 | - ``oe-pkgdata-util lookup-recipe package ...``: Lists the name | ||
211 | of the recipes that produce the given packages. | ||
212 | |||
213 | For more information on the ``oe-pkgdata-util`` command, use the help | ||
214 | facility:: | ||
215 | |||
216 | $ oe-pkgdata-util --help | ||
217 | $ oe-pkgdata-util subcommand --help | ||
218 | |||
219 | Viewing Dependencies Between Recipes and Tasks | ||
220 | ============================================== | ||
221 | |||
222 | Sometimes it can be hard to see why BitBake wants to build other recipes | ||
223 | before the one you have specified. Dependency information can help you | ||
224 | understand why a recipe is built. | ||
225 | |||
226 | To generate dependency information for a recipe, run the following | ||
227 | command:: | ||
228 | |||
229 | $ bitbake -g recipename | ||
230 | |||
231 | This command writes the following files in the current directory: | ||
232 | |||
233 | - ``pn-buildlist``: A list of recipes/targets involved in building | ||
234 | `recipename`. "Involved" here means that at least one task from the | ||
235 | recipe needs to run when building `recipename` from scratch. Targets | ||
236 | that are in | ||
237 | :term:`ASSUME_PROVIDED` | ||
238 | are not listed. | ||
239 | |||
240 | - ``task-depends.dot``: A graph showing dependencies between tasks. | ||
241 | |||
242 | The graphs are in :wikipedia:`DOT <DOT_%28graph_description_language%29>` | ||
243 | format and can be converted to images (e.g. using the ``dot`` tool from | ||
244 | `Graphviz <https://www.graphviz.org/>`__). | ||
245 | |||
246 | .. note:: | ||
247 | |||
248 | - DOT files use a plain text format. The graphs generated using the | ||
249 | ``bitbake -g`` command are often so large as to be difficult to | ||
250 | read without special pruning (e.g. with BitBake's ``-I`` option) | ||
251 | and processing. Despite the form and size of the graphs, the | ||
252 | corresponding ``.dot`` files can still be possible to read and | ||
253 | provide useful information. | ||
254 | |||
255 | As an example, the ``task-depends.dot`` file contains lines such | ||
256 | as the following:: | ||
257 | |||
258 | "libxslt.do_configure" -> "libxml2.do_populate_sysroot" | ||
259 | |||
260 | The above example line reveals that the | ||
261 | :ref:`ref-tasks-configure` | ||
262 | task in ``libxslt`` depends on the | ||
263 | :ref:`ref-tasks-populate_sysroot` | ||
264 | task in ``libxml2``, which is a normal | ||
265 | :term:`DEPENDS` dependency | ||
266 | between the two recipes. | ||
267 | |||
268 | - For an example of how ``.dot`` files can be processed, see the | ||
269 | ``scripts/contrib/graph-tool`` Python script, which finds and | ||
270 | displays paths between graph nodes. | ||
271 | |||
272 | You can use a different method to view dependency information by using | ||
273 | either:: | ||
274 | |||
275 | $ bitbake -g -u taskexp recipename | ||
276 | |||
277 | or:: | ||
278 | |||
279 | $ bitbake -g -u taskexp_ncurses recipename | ||
280 | |||
281 | The ``-u taskdep`` option GUI window from which you can view build-time and | ||
282 | runtime dependencies for the recipes involved in building recipename. The | ||
283 | ``-u taskexp_ncurses`` option uses ncurses instead of GTK to render the UI. | ||
284 | |||
285 | Viewing Task Variable Dependencies | ||
286 | ================================== | ||
287 | |||
288 | As mentioned in the | ||
289 | ":ref:`bitbake-user-manual/bitbake-user-manual-execution:checksums (signatures)`" | ||
290 | section of the BitBake User Manual, BitBake tries to automatically determine | ||
291 | what variables a task depends on so that it can rerun the task if any values of | ||
292 | the variables change. This determination is usually reliable. However, if you | ||
293 | do things like construct variable names at runtime, then you might have to | ||
294 | manually declare dependencies on those variables using ``vardeps`` as described | ||
295 | in the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:variable flags`" | ||
296 | section of the BitBake User Manual. | ||
297 | |||
298 | If you are unsure whether a variable dependency is being picked up | ||
299 | automatically for a given task, you can list the variable dependencies | ||
300 | BitBake has determined by doing the following: | ||
301 | |||
302 | #. Build the recipe containing the task:: | ||
303 | |||
304 | $ bitbake recipename | ||
305 | |||
306 | #. Inside the :term:`STAMPS_DIR` | ||
307 | directory, find the signature data (``sigdata``) file that | ||
308 | corresponds to the task. The ``sigdata`` files contain a pickled | ||
309 | Python database of all the metadata that went into creating the input | ||
310 | checksum for the task. As an example, for the | ||
311 | :ref:`ref-tasks-fetch` task of the | ||
312 | ``db`` recipe, the ``sigdata`` file might be found in the following | ||
313 | location:: | ||
314 | |||
315 | ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 | ||
316 | |||
317 | For tasks that are accelerated through the shared state | ||
318 | (:ref:`sstate <overview-manual/concepts:shared state cache>`) cache, an | ||
319 | additional ``siginfo`` file is written into | ||
320 | :term:`SSTATE_DIR` along with | ||
321 | the cached task output. The ``siginfo`` files contain exactly the | ||
322 | same information as ``sigdata`` files. | ||
323 | |||
324 | #. Run ``bitbake-dumpsig`` on the ``sigdata`` or ``siginfo`` file. Here | ||
325 | is an example:: | ||
326 | |||
327 | $ bitbake-dumpsig ${BUILDDIR}/tmp/stamps/i586-poky-linux/db/6.0.30-r1.do_fetch.sigdata.7c048c18222b16ff0bcee2000ef648b1 | ||
328 | |||
329 | In the output of the above command, you will find a line like the | ||
330 | following, which lists all the (inferred) variable dependencies for | ||
331 | the task. This list also includes indirect dependencies from | ||
332 | variables depending on other variables, recursively:: | ||
333 | |||
334 | Task dependencies: ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[sha256sum]', 'base_do_fetch'] | ||
335 | |||
336 | .. note:: | ||
337 | |||
338 | Functions (e.g. ``base_do_fetch``) also count as variable dependencies. | ||
339 | These functions in turn depend on the variables they reference. | ||
340 | |||
341 | The output of ``bitbake-dumpsig`` also includes the value each | ||
342 | variable had, a list of dependencies for each variable, and | ||
343 | :term:`BB_BASEHASH_IGNORE_VARS` | ||
344 | information. | ||
345 | |||
346 | Debugging signature construction and unexpected task executions | ||
347 | =============================================================== | ||
348 | |||
349 | There is a ``bitbake-diffsigs`` command for comparing two | ||
350 | ``siginfo`` or ``sigdata`` files. This command can be helpful when | ||
351 | trying to figure out what changed between two versions of a task. If you | ||
352 | call ``bitbake-diffsigs`` with just one file, the command behaves like | ||
353 | ``bitbake-dumpsig``. | ||
354 | |||
355 | You can also use BitBake to dump out the signature construction | ||
356 | information without executing tasks by using either of the following | ||
357 | BitBake command-line options:: | ||
358 | |||
359 | ‐‐dump-signatures=SIGNATURE_HANDLER | ||
360 | -S SIGNATURE_HANDLER | ||
361 | |||
362 | |||
363 | .. note:: | ||
364 | |||
365 | Two common values for `SIGNATURE_HANDLER` are "none" and "printdiff", which | ||
366 | dump only the signature or compare the dumped signature with the most recent one, | ||
367 | respectively. "printdiff" will try to establish the most recent | ||
368 | signature match (e.g. in the sstate cache) and then | ||
369 | compare the matched signatures to determine the stamps and delta | ||
370 | where these two stamp trees diverge. This can be used to determine why | ||
371 | tasks need to be re-run in situations where that is not expected. | ||
372 | |||
373 | Using BitBake with either of these options causes BitBake to dump out | ||
374 | ``sigdata`` files in the ``stamps`` directory for every task it would | ||
375 | have executed instead of building the specified target package. | ||
376 | |||
377 | Viewing Metadata Used to Create the Input Signature of a Shared State Task | ||
378 | ========================================================================== | ||
379 | |||
380 | Seeing what metadata went into creating the input signature of a shared | ||
381 | state (sstate) task can be a useful debugging aid. This information is | ||
382 | available in signature information (``siginfo``) files in | ||
383 | :term:`SSTATE_DIR`. For | ||
384 | information on how to view and interpret information in ``siginfo`` | ||
385 | files, see the | ||
386 | ":ref:`dev-manual/debugging:viewing task variable dependencies`" section. | ||
387 | |||
388 | For conceptual information on shared state, see the | ||
389 | ":ref:`overview-manual/concepts:shared state`" | ||
390 | section in the Yocto Project Overview and Concepts Manual. | ||
391 | |||
392 | Invalidating Shared State to Force a Task to Run | ||
393 | ================================================ | ||
394 | |||
395 | The OpenEmbedded build system uses | ||
396 | :ref:`checksums <overview-manual/concepts:checksums (signatures)>` and | ||
397 | :ref:`overview-manual/concepts:shared state` cache to avoid unnecessarily | ||
398 | rebuilding tasks. Collectively, this scheme is known as "shared state | ||
399 | code". | ||
400 | |||
401 | As with all schemes, this one has some drawbacks. It is possible that | ||
402 | you could make implicit changes to your code that the checksum | ||
403 | calculations do not take into account. These implicit changes affect a | ||
404 | task's output but do not trigger the shared state code into rebuilding a | ||
405 | recipe. Consider an example during which a tool changes its output. | ||
406 | Assume that the output of ``rpmdeps`` changes. The result of the change | ||
407 | should be that all the ``package`` and ``package_write_rpm`` shared | ||
408 | state cache items become invalid. However, because the change to the | ||
409 | output is external to the code and therefore implicit, the associated | ||
410 | shared state cache items do not become invalidated. In this case, the | ||
411 | build process uses the cached items rather than running the task again. | ||
412 | Obviously, these types of implicit changes can cause problems. | ||
413 | |||
414 | To avoid these problems during the build, you need to understand the | ||
415 | effects of any changes you make. Realize that changes you make directly | ||
416 | to a function are automatically factored into the checksum calculation. | ||
417 | Thus, these explicit changes invalidate the associated area of shared | ||
418 | state cache. However, you need to be aware of any implicit changes that | ||
419 | are not obvious changes to the code and could affect the output of a | ||
420 | given task. | ||
421 | |||
422 | When you identify an implicit change, you can easily take steps to | ||
423 | invalidate the cache and force the tasks to run. The steps you can take | ||
424 | are as simple as changing a function's comments in the source code. For | ||
425 | example, to invalidate package shared state files, change the comment | ||
426 | statements of | ||
427 | :ref:`ref-tasks-package` or the | ||
428 | comments of one of the functions it calls. Even though the change is | ||
429 | purely cosmetic, it causes the checksum to be recalculated and forces | ||
430 | the build system to run the task again. | ||
431 | |||
432 | .. note:: | ||
433 | |||
434 | For an example of a commit that makes a cosmetic change to invalidate | ||
435 | shared state, see this | ||
436 | :yocto_git:`commit </poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54>`. | ||
437 | |||
438 | Running Specific Tasks | ||
439 | ====================== | ||
440 | |||
441 | Any given recipe consists of a set of tasks. The standard BitBake | ||
442 | behavior in most cases is: :ref:`ref-tasks-fetch`, :ref:`ref-tasks-unpack`, :ref:`ref-tasks-patch`, | ||
443 | :ref:`ref-tasks-configure`, :ref:`ref-tasks-compile`, :ref:`ref-tasks-install`, :ref:`ref-tasks-package`, | ||
444 | :ref:`do_package_write_* <ref-tasks-package_write_deb>`, and :ref:`ref-tasks-build`. The default task is | ||
445 | :ref:`ref-tasks-build` and any tasks on which it depends build first. Some tasks, | ||
446 | such as :ref:`ref-tasks-devshell`, are not part of the default build chain. If you | ||
447 | wish to run a task that is not part of the default build chain, you can | ||
448 | use the ``-c`` option in BitBake. Here is an example:: | ||
449 | |||
450 | $ bitbake matchbox-desktop -c devshell | ||
451 | |||
452 | The ``-c`` option respects task dependencies, which means that all other | ||
453 | tasks (including tasks from other recipes) that the specified task | ||
454 | depends on will be run before the task. Even when you manually specify a | ||
455 | task to run with ``-c``, BitBake will only run the task if it considers | ||
456 | it "out of date". See the | ||
457 | ":ref:`overview-manual/concepts:stamp files and the rerunning of tasks`" | ||
458 | section in the Yocto Project Overview and Concepts Manual for how | ||
459 | BitBake determines whether a task is "out of date". | ||
460 | |||
461 | If you want to force an up-to-date task to be rerun (e.g. because you | ||
462 | made manual modifications to the recipe's | ||
463 | :term:`WORKDIR` that you want to try | ||
464 | out), then you can use the ``-f`` option. | ||
465 | |||
466 | .. note:: | ||
467 | |||
468 | The reason ``-f`` is never required when running the | ||
469 | :ref:`ref-tasks-devshell` task is because the | ||
470 | [\ :ref:`nostamp <bitbake-user-manual/bitbake-user-manual-metadata:variable flags>`\ ] | ||
471 | variable flag is already set for the task. | ||
472 | |||
473 | The following example shows one way you can use the ``-f`` option:: | ||
474 | |||
475 | $ bitbake matchbox-desktop | ||
476 | . | ||
477 | . | ||
478 | make some changes to the source code in the work directory | ||
479 | . | ||
480 | . | ||
481 | $ bitbake matchbox-desktop -c compile -f | ||
482 | $ bitbake matchbox-desktop | ||
483 | |||
484 | This sequence first builds and then recompiles ``matchbox-desktop``. The | ||
485 | last command reruns all tasks (basically the packaging tasks) after the | ||
486 | compile. BitBake recognizes that the :ref:`ref-tasks-compile` task was rerun and | ||
487 | therefore understands that the other tasks also need to be run again. | ||
488 | |||
489 | Another, shorter way to rerun a task and all | ||
490 | :ref:`ref-manual/tasks:normal recipe build tasks` | ||
491 | that depend on it is to use the ``-C`` option. | ||
492 | |||
493 | .. note:: | ||
494 | |||
495 | This option is upper-cased and is separate from the ``-c`` | ||
496 | option, which is lower-cased. | ||
497 | |||
498 | Using this option invalidates the given task and then runs the | ||
499 | :ref:`ref-tasks-build` task, which is | ||
500 | the default task if no task is given, and the tasks on which it depends. | ||
501 | You could replace the final two commands in the previous example with | ||
502 | the following single command:: | ||
503 | |||
504 | $ bitbake matchbox-desktop -C compile | ||
505 | |||
506 | Internally, the ``-f`` and ``-C`` options work by tainting (modifying) | ||
507 | the input checksum of the specified task. This tainting indirectly | ||
508 | causes the task and its dependent tasks to be rerun through the normal | ||
509 | task dependency mechanisms. | ||
510 | |||
511 | .. note:: | ||
512 | |||
513 | BitBake explicitly keeps track of which tasks have been tainted in | ||
514 | this fashion, and will print warnings such as the following for | ||
515 | builds involving such tasks: | ||
516 | |||
517 | .. code-block:: none | ||
518 | |||
519 | WARNING: /home/ulf/poky/meta/recipes-sato/matchbox-desktop/matchbox-desktop_2.1.bb.do_compile is tainted from a forced run | ||
520 | |||
521 | |||
522 | The purpose of the warning is to let you know that the work directory | ||
523 | and build output might not be in the clean state they would be in for | ||
524 | a "normal" build, depending on what actions you took. To get rid of | ||
525 | such warnings, you can remove the work directory and rebuild the | ||
526 | recipe, as follows:: | ||
527 | |||
528 | $ bitbake matchbox-desktop -c clean | ||
529 | $ bitbake matchbox-desktop | ||
530 | |||
531 | |||
532 | You can view a list of tasks in a given package by running the | ||
533 | :ref:`ref-tasks-listtasks` task as follows:: | ||
534 | |||
535 | $ bitbake matchbox-desktop -c listtasks | ||
536 | |||
537 | The results appear as output to the console and are also in | ||
538 | the file ``${WORKDIR}/temp/log.do_listtasks``. | ||
539 | |||
540 | General BitBake Problems | ||
541 | ======================== | ||
542 | |||
543 | You can see debug output from BitBake by using the ``-D`` option. The | ||
544 | debug output gives more information about what BitBake is doing and the | ||
545 | reason behind it. Each ``-D`` option you use increases the logging | ||
546 | level. The most common usage is ``-DDD``. | ||
547 | |||
548 | The output from ``bitbake -DDD -v targetname`` can reveal why BitBake | ||
549 | chose a certain version of a package or why BitBake picked a certain | ||
550 | provider. This command could also help you in a situation where you | ||
551 | think BitBake did something unexpected. | ||
552 | |||
553 | Building with No Dependencies | ||
554 | ============================= | ||
555 | |||
556 | To build a specific recipe (``.bb`` file), you can use the following | ||
557 | command form:: | ||
558 | |||
559 | $ bitbake -b somepath/somerecipe.bb | ||
560 | |||
561 | This command form does | ||
562 | not check for dependencies. Consequently, you should use it only when | ||
563 | you know existing dependencies have been met. | ||
564 | |||
565 | .. note:: | ||
566 | |||
567 | You can also specify fragments of the filename. In this case, BitBake | ||
568 | checks for a unique match. | ||
569 | |||
570 | Recipe Logging Mechanisms | ||
571 | ========================= | ||
572 | |||
573 | The Yocto Project provides several logging functions for producing | ||
574 | debugging output and reporting errors and warnings. For Python | ||
575 | functions, the following logging functions are available. All of these functions | ||
576 | log to ``${T}/log.do_``\ `task`, and can also log to standard output | ||
577 | (stdout) with the right settings: | ||
578 | |||
579 | - ``bb.plain(msg)``: Writes msg as is to the log while also | ||
580 | logging to stdout. | ||
581 | |||
582 | - ``bb.note(msg)``: Writes "NOTE: msg" to the log. Also logs to | ||
583 | stdout if BitBake is called with "-v". | ||
584 | |||
585 | - ``bb.debug(level, msg)``: Writes "DEBUG: msg" to the log. Also logs to | ||
586 | stdout if the log level is greater than or equal to level. See the | ||
587 | ":ref:`bitbake-user-manual/bitbake-user-manual-intro:usage and syntax`" | ||
588 | option in the BitBake User Manual for more information. | ||
589 | |||
590 | - ``bb.warn(msg)``: Writes "WARNING: msg" to the log while also | ||
591 | logging to stdout. | ||
592 | |||
593 | - ``bb.error(msg)``: Writes "ERROR: msg" to the log while also | ||
594 | logging to standard out (stdout). | ||
595 | |||
596 | .. note:: | ||
597 | |||
598 | Calling this function does not cause the task to fail. | ||
599 | |||
600 | - ``bb.fatal(msg)``: This logging function is similar to | ||
601 | ``bb.error(msg)`` but also causes the calling task to fail. | ||
602 | |||
603 | .. note:: | ||
604 | |||
605 | ``bb.fatal()`` raises an exception, which means you do not need to put a | ||
606 | "return" statement after the function. | ||
607 | |||
608 | The same logging functions are also available in shell functions, under | ||
609 | the names ``bbplain``, ``bbnote``, ``bbdebug``, ``bbwarn``, ``bberror``, | ||
610 | and ``bbfatal``. The :ref:`ref-classes-logging` class | ||
611 | implements these functions. See that class in the ``meta/classes`` | ||
612 | folder of the :term:`Source Directory` for information. | ||
613 | |||
614 | Logging With Python | ||
615 | ------------------- | ||
616 | |||
617 | When creating recipes using Python and inserting code that handles build | ||
618 | logs, keep in mind the goal is to have informative logs while keeping | ||
619 | the console as "silent" as possible. Also, if you want status messages | ||
620 | in the log, use the "debug" loglevel. | ||
621 | |||
622 | Here is an example written in Python. The code handles logging for | ||
623 | a function that determines the number of tasks needed to be run. See the | ||
624 | ":ref:`ref-tasks-listtasks`" | ||
625 | section for additional information:: | ||
626 | |||
627 | python do_listtasks() { | ||
628 | bb.debug(2, "Starting to figure out the task list") | ||
629 | if noteworthy_condition: | ||
630 | bb.note("There are 47 tasks to run") | ||
631 | bb.debug(2, "Got to point xyz") | ||
632 | if warning_trigger: | ||
633 | bb.warn("Detected warning_trigger, this might be a problem later.") | ||
634 | if recoverable_error: | ||
635 | bb.error("Hit recoverable_error, you really need to fix this!") | ||
636 | if fatal_error: | ||
637 | bb.fatal("fatal_error detected, unable to print the task list") | ||
638 | bb.plain("The tasks present are abc") | ||
639 | bb.debug(2, "Finished figuring out the tasklist") | ||
640 | } | ||
641 | |||
642 | Logging With Bash | ||
643 | ----------------- | ||
644 | |||
645 | When creating recipes using Bash and inserting code that handles build | ||
646 | logs, you have the same goals --- informative with minimal console output. | ||
647 | The syntax you use for recipes written in Bash is similar to that of | ||
648 | recipes written in Python described in the previous section. | ||
649 | |||
650 | Here is an example written in Bash. The code logs the progress of | ||
651 | the ``do_my_function`` function:: | ||
652 | |||
653 | do_my_function() { | ||
654 | bbdebug 2 "Running do_my_function" | ||
655 | if [ exceptional_condition ]; then | ||
656 | bbnote "Hit exceptional_condition" | ||
657 | fi | ||
658 | bbdebug 2 "Got to point xyz" | ||
659 | if [ warning_trigger ]; then | ||
660 | bbwarn "Detected warning_trigger, this might cause a problem later." | ||
661 | fi | ||
662 | if [ recoverable_error ]; then | ||
663 | bberror "Hit recoverable_error, correcting" | ||
664 | fi | ||
665 | if [ fatal_error ]; then | ||
666 | bbfatal "fatal_error detected" | ||
667 | fi | ||
668 | bbdebug 2 "Completed do_my_function" | ||
669 | } | ||
670 | |||
671 | |||
672 | Debugging Parallel Make Races | ||
673 | ============================= | ||
674 | |||
675 | A parallel ``make`` race occurs when the build consists of several parts | ||
676 | that are run simultaneously and a situation occurs when the output or | ||
677 | result of one part is not ready for use with a different part of the | ||
678 | build that depends on that output. Parallel make races are annoying and | ||
679 | can sometimes be difficult to reproduce and fix. However, there are some simple | ||
680 | tips and tricks that can help you debug and fix them. This section | ||
681 | presents a real-world example of an error encountered on the Yocto | ||
682 | Project autobuilder and the process used to fix it. | ||
683 | |||
684 | .. note:: | ||
685 | |||
686 | If you cannot properly fix a ``make`` race condition, you can work around it | ||
687 | by clearing either the :term:`PARALLEL_MAKE` or :term:`PARALLEL_MAKEINST` | ||
688 | variables. | ||
689 | |||
690 | The Failure | ||
691 | ----------- | ||
692 | |||
693 | For this example, assume that you are building an image that depends on | ||
694 | the "neard" package. And, during the build, BitBake runs into problems | ||
695 | and creates the following output. | ||
696 | |||
697 | .. note:: | ||
698 | |||
699 | This example log file has longer lines artificially broken to make | ||
700 | the listing easier to read. | ||
701 | |||
702 | If you examine the output or the log file, you see the failure during | ||
703 | ``make``: | ||
704 | |||
705 | .. code-block:: none | ||
706 | |||
707 | | DEBUG: SITE files ['endian-little', 'bit-32', 'ix86-common', 'common-linux', 'common-glibc', 'i586-linux', 'common'] | ||
708 | | DEBUG: Executing shell function do_compile | ||
709 | | NOTE: make -j 16 | ||
710 | | make --no-print-directory all-am | ||
711 | | /bin/mkdir -p include/near | ||
712 | | /bin/mkdir -p include/near | ||
713 | | /bin/mkdir -p include/near | ||
714 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
715 | 0.14-r0/neard-0.14/include/types.h include/near/types.h | ||
716 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
717 | 0.14-r0/neard-0.14/include/log.h include/near/log.h | ||
718 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
719 | 0.14-r0/neard-0.14/include/plugin.h include/near/plugin.h | ||
720 | | /bin/mkdir -p include/near | ||
721 | | /bin/mkdir -p include/near | ||
722 | | /bin/mkdir -p include/near | ||
723 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
724 | 0.14-r0/neard-0.14/include/tag.h include/near/tag.h | ||
725 | | /bin/mkdir -p include/near | ||
726 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
727 | 0.14-r0/neard-0.14/include/adapter.h include/near/adapter.h | ||
728 | | /bin/mkdir -p include/near | ||
729 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
730 | 0.14-r0/neard-0.14/include/ndef.h include/near/ndef.h | ||
731 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
732 | 0.14-r0/neard-0.14/include/tlv.h include/near/tlv.h | ||
733 | | /bin/mkdir -p include/near | ||
734 | | /bin/mkdir -p include/near | ||
735 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
736 | 0.14-r0/neard-0.14/include/setting.h include/near/setting.h | ||
737 | | /bin/mkdir -p include/near | ||
738 | | /bin/mkdir -p include/near | ||
739 | | /bin/mkdir -p include/near | ||
740 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
741 | 0.14-r0/neard-0.14/include/device.h include/near/device.h | ||
742 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
743 | 0.14-r0/neard-0.14/include/nfc_copy.h include/near/nfc_copy.h | ||
744 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
745 | 0.14-r0/neard-0.14/include/snep.h include/near/snep.h | ||
746 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
747 | 0.14-r0/neard-0.14/include/version.h include/near/version.h | ||
748 | | ln -s /home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/work/i586-poky-linux/neard/ | ||
749 | 0.14-r0/neard-0.14/include/dbus.h include/near/dbus.h | ||
750 | | ./src/genbuiltin nfctype1 nfctype2 nfctype3 nfctype4 p2p > src/builtin.h | ||
751 | | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/pokybuild/yocto-autobuilder/nightly-x86/ | ||
752 | build/build/tmp/sysroots/qemux86 -DHAVE_CONFIG_H -I. -I./include -I./src -I./gdbus -I/home/pokybuild/ | ||
753 | yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/glib-2.0 | ||
754 | -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/tmp/sysroots/qemux86/usr/ | ||
755 | lib/glib-2.0/include -I/home/pokybuild/yocto-autobuilder/nightly-x86/build/build/ | ||
756 | tmp/sysroots/qemux86/usr/include/dbus-1.0 -I/home/pokybuild/yocto-autobuilder/ | ||
757 | nightly-x86/build/build/tmp/sysroots/qemux86/usr/lib/dbus-1.0/include -I/home/pokybuild/yocto-autobuilder/ | ||
758 | nightly-x86/build/build/tmp/sysroots/qemux86/usr/include/libnl3 | ||
759 | -DNEAR_PLUGIN_BUILTIN -DPLUGINDIR=\""/usr/lib/near/plugins"\" | ||
760 | -DCONFIGDIR=\""/etc/neard\"" -O2 -pipe -g -feliminate-unused-debug-types -c | ||
761 | -o tools/snep-send.o tools/snep-send.c | ||
762 | | In file included from tools/snep-send.c:16:0: | ||
763 | | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory | ||
764 | | #include <near/dbus.h> | ||
765 | | ^ | ||
766 | | compilation terminated. | ||
767 | | make[1]: *** [tools/snep-send.o] Error 1 | ||
768 | | make[1]: *** Waiting for unfinished jobs.... | ||
769 | | make: *** [all] Error 2 | ||
770 | | ERROR: oe_runmake failed | ||
771 | |||
772 | Reproducing the Error | ||
773 | --------------------- | ||
774 | |||
775 | Because race conditions are intermittent, they do not manifest | ||
776 | themselves every time you do the build. In fact, most times the build | ||
777 | will complete without problems even though the potential race condition | ||
778 | exists. Thus, once the error surfaces, you need a way to reproduce it. | ||
779 | |||
780 | In this example, compiling the "neard" package is causing the problem. | ||
781 | So the first thing to do is build "neard" locally. Before you start the | ||
782 | build, set the | ||
783 | :term:`PARALLEL_MAKE` variable | ||
784 | in your ``local.conf`` file to a high number (e.g. "-j 20"). Using a | ||
785 | high value for :term:`PARALLEL_MAKE` increases the chances of the race | ||
786 | condition showing up:: | ||
787 | |||
788 | $ bitbake neard | ||
789 | |||
790 | Once the local build for "neard" completes, start a ``devshell`` build:: | ||
791 | |||
792 | $ bitbake neard -c devshell | ||
793 | |||
794 | For information on how to use a ``devshell``, see the | ||
795 | ":ref:`dev-manual/development-shell:using a development shell`" section. | ||
796 | |||
797 | In the ``devshell``, do the following:: | ||
798 | |||
799 | $ make clean | ||
800 | $ make tools/snep-send.o | ||
801 | |||
802 | The ``devshell`` commands cause the failure to clearly | ||
803 | be visible. In this case, there is a missing dependency for the ``neard`` | ||
804 | Makefile target. Here is some abbreviated, sample output with the | ||
805 | missing dependency clearly visible at the end:: | ||
806 | |||
807 | i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/scott-lenovo/...... | ||
808 | . | ||
809 | . | ||
810 | . | ||
811 | tools/snep-send.c | ||
812 | In file included from tools/snep-send.c:16:0: | ||
813 | tools/../src/near.h:41:23: fatal error: near/dbus.h: No such file or directory | ||
814 | #include <near/dbus.h> | ||
815 | ^ | ||
816 | compilation terminated. | ||
817 | make: *** [tools/snep-send.o] Error 1 | ||
818 | $ | ||
819 | |||
820 | |||
821 | Creating a Patch for the Fix | ||
822 | ---------------------------- | ||
823 | |||
824 | Because there is a missing dependency for the Makefile target, you need | ||
825 | to patch the ``Makefile.am`` file, which is generated from | ||
826 | ``Makefile.in``. You can use Quilt to create the patch:: | ||
827 | |||
828 | $ quilt new parallelmake.patch | ||
829 | Patch patches/parallelmake.patch is now on top | ||
830 | $ quilt add Makefile.am | ||
831 | File Makefile.am added to patch patches/parallelmake.patch | ||
832 | |||
833 | For more information on using Quilt, see the | ||
834 | ":ref:`dev-manual/quilt:using quilt in your workflow`" section. | ||
835 | |||
836 | At this point you need to make the edits to ``Makefile.am`` to add the | ||
837 | missing dependency. For our example, you have to add the following line | ||
838 | to the file:: | ||
839 | |||
840 | tools/snep-send.$(OBJEXT): include/near/dbus.h | ||
841 | |||
842 | Once you have edited the file, use the ``refresh`` command to create the | ||
843 | patch:: | ||
844 | |||
845 | $ quilt refresh | ||
846 | Refreshed patch patches/parallelmake.patch | ||
847 | |||
848 | Once the patch file is created, you need to add it back to the originating | ||
849 | recipe folder. Here is an example assuming a top-level | ||
850 | :term:`Source Directory` named ``poky``:: | ||
851 | |||
852 | $ cp patches/parallelmake.patch poky/meta/recipes-connectivity/neard/neard | ||
853 | |||
854 | The final thing you need to do to implement the fix in the build is to | ||
855 | update the "neard" recipe (i.e. ``neard-0.14.bb``) so that the | ||
856 | :term:`SRC_URI` statement includes | ||
857 | the patch file. The recipe file is in the folder above the patch. Here | ||
858 | is what the edited :term:`SRC_URI` statement would look like:: | ||
859 | |||
860 | SRC_URI = "${KERNELORG_MIRROR}/linux/network/nfc/${BPN}-${PV}.tar.xz \ | ||
861 | file://neard.in \ | ||
862 | file://neard.service.in \ | ||
863 | file://parallelmake.patch \ | ||
864 | " | ||
865 | |||
866 | With the patch complete and moved to the correct folder and the | ||
867 | :term:`SRC_URI` statement updated, you can exit the ``devshell``:: | ||
868 | |||
869 | $ exit | ||
870 | |||
871 | Testing the Build | ||
872 | ----------------- | ||
873 | |||
874 | With everything in place, you can get back to trying the build again | ||
875 | locally:: | ||
876 | |||
877 | $ bitbake neard | ||
878 | |||
879 | This build should succeed. | ||
880 | |||
881 | Now you can open up a ``devshell`` again and repeat the clean and make | ||
882 | operations as follows:: | ||
883 | |||
884 | $ bitbake neard -c devshell | ||
885 | $ make clean | ||
886 | $ make tools/snep-send.o | ||
887 | |||
888 | The build should work without issue. | ||
889 | |||
890 | As with all solved problems, if they originated upstream, you need to | ||
891 | submit the fix for the recipe in OE-Core and upstream so that the | ||
892 | problem is taken care of at its source. See the | ||
893 | ":doc:`../contributor-guide/submit-changes`" section for more information. | ||
894 | |||
895 | Debugging With the GNU Project Debugger (GDB) Remotely | ||
896 | ====================================================== | ||
897 | |||
898 | GDB allows you to examine running programs, which in turn helps you to | ||
899 | understand and fix problems. It also allows you to perform post-mortem | ||
900 | style analysis of program crashes. GDB is available as a package within | ||
901 | the Yocto Project and is installed in SDK images by default. See the | ||
902 | ":ref:`ref-manual/images:Images`" chapter in the Yocto | ||
903 | Project Reference Manual for a description of these images. You can find | ||
904 | information on GDB at https://sourceware.org/gdb/. | ||
905 | |||
906 | .. note:: | ||
907 | |||
908 | For best results, install debug (``-dbg``) packages for the applications you | ||
909 | are going to debug. Doing so makes extra debug symbols available that give | ||
910 | you more meaningful output. | ||
911 | |||
912 | Sometimes, due to memory or disk space constraints, it is not possible | ||
913 | to use GDB directly on the remote target to debug applications. These | ||
914 | constraints arise because GDB needs to load the debugging information | ||
915 | and the binaries of the process being debugged. Additionally, GDB needs | ||
916 | to perform many computations to locate information such as function | ||
917 | names, variable names and values, stack traces and so forth --- even | ||
918 | before starting the debugging process. These extra computations place | ||
919 | more load on the target system and can alter the characteristics of the | ||
920 | program being debugged. | ||
921 | |||
922 | To help get past the previously mentioned constraints, there are two | ||
923 | methods you can use: running a debuginfod server and using gdbserver. | ||
924 | |||
925 | Using the debuginfod server method | ||
926 | ---------------------------------- | ||
927 | |||
928 | ``debuginfod`` from ``elfutils`` is a way to distribute ``debuginfo`` files. | ||
929 | Running a ``debuginfod`` server makes debug symbols readily available, | ||
930 | which means you don't need to download debugging information | ||
931 | and the binaries of the process being debugged. You can just fetch | ||
932 | debug symbols from the server. | ||
933 | |||
934 | To run a ``debuginfod`` server, you need to do the following: | ||
935 | |||
936 | - Ensure that ``debuginfod`` is present in :term:`DISTRO_FEATURES` | ||
937 | (it already is in ``OpenEmbedded-core`` defaults and ``poky`` reference distribution). | ||
938 | If not, set in your distro config file or in ``local.conf``:: | ||
939 | |||
940 | DISTRO_FEATURES:append = " debuginfod" | ||
941 | |||
942 | This distro feature enables the server and client library in ``elfutils``, | ||
943 | and enables ``debuginfod`` support in clients (at the moment, ``gdb`` and ``binutils``). | ||
944 | |||
945 | - Run the following commands to launch the ``debuginfod`` server on the host:: | ||
946 | |||
947 | $ oe-debuginfod | ||
948 | |||
949 | - To use ``debuginfod`` on the target, you need to know the ip:port where | ||
950 | ``debuginfod`` is listening on the host (port defaults to 8002), and export | ||
951 | that into the shell environment, for example in ``qemu``:: | ||
952 | |||
953 | root@qemux86-64:~# export DEBUGINFOD_URLS="http://192.168.7.1:8002/" | ||
954 | |||
955 | - Then debug info fetching should simply work when running the target ``gdb``, | ||
956 | ``readelf`` or ``objdump``, for example:: | ||
957 | |||
958 | root@qemux86-64:~# gdb /bin/cat | ||
959 | ... | ||
960 | Reading symbols from /bin/cat... | ||
961 | Downloading separate debug info for /bin/cat... | ||
962 | Reading symbols from /home/root/.cache/debuginfod_client/923dc4780cfbc545850c616bffa884b6b5eaf322/debuginfo... | ||
963 | |||
964 | - It's also possible to use ``debuginfod-find`` to just query the server:: | ||
965 | |||
966 | root@qemux86-64:~# debuginfod-find debuginfo /bin/ls | ||
967 | /home/root/.cache/debuginfod_client/356edc585f7f82d46f94fcb87a86a3fe2d2e60bd/debuginfo | ||
968 | |||
969 | |||
970 | Using the gdbserver method | ||
971 | -------------------------- | ||
972 | |||
973 | gdbserver, which runs on the remote target and does not load any | ||
974 | debugging information from the debugged process. Instead, a GDB instance | ||
975 | processes the debugging information that is run on a remote computer - | ||
976 | the host GDB. The host GDB then sends control commands to gdbserver to | ||
977 | make it stop or start the debugged program, as well as read or write | ||
978 | memory regions of that debugged program. All the debugging information | ||
979 | loaded and processed as well as all the heavy debugging is done by the | ||
980 | host GDB. Offloading these processes gives the gdbserver running on the | ||
981 | target a chance to remain small and fast. | ||
982 | |||
983 | Because the host GDB is responsible for loading the debugging | ||
984 | information and for doing the necessary processing to make actual | ||
985 | debugging happen, you have to make sure the host can access the | ||
986 | unstripped binaries complete with their debugging information and also | ||
987 | be sure the target is compiled with no optimizations. The host GDB must | ||
988 | also have local access to all the libraries used by the debugged | ||
989 | program. Because gdbserver does not need any local debugging | ||
990 | information, the binaries on the remote target can remain stripped. | ||
991 | However, the binaries must also be compiled without optimization so they | ||
992 | match the host's binaries. | ||
993 | |||
994 | To remain consistent with GDB documentation and terminology, the binary | ||
995 | being debugged on the remote target machine is referred to as the | ||
996 | "inferior" binary. For documentation on GDB see the `GDB | ||
997 | site <https://sourceware.org/gdb/documentation/>`__. | ||
998 | |||
999 | The following steps show you how to debug using the GNU project | ||
1000 | debugger. | ||
1001 | |||
1002 | #. *Configure your build system to construct the companion debug | ||
1003 | filesystem:* | ||
1004 | |||
1005 | In your ``local.conf`` file, set the following:: | ||
1006 | |||
1007 | IMAGE_GEN_DEBUGFS = "1" | ||
1008 | IMAGE_FSTYPES_DEBUGFS = "tar.bz2" | ||
1009 | |||
1010 | These options cause the | ||
1011 | OpenEmbedded build system to generate a special companion filesystem | ||
1012 | fragment, which contains the matching source and debug symbols to | ||
1013 | your deployable filesystem. The build system does this by looking at | ||
1014 | what is in the deployed filesystem, and pulling the corresponding | ||
1015 | ``-dbg`` packages. | ||
1016 | |||
1017 | The companion debug filesystem is not a complete filesystem, but only | ||
1018 | contains the debug fragments. This filesystem must be combined with | ||
1019 | the full filesystem for debugging. Subsequent steps in this procedure | ||
1020 | show how to combine the partial filesystem with the full filesystem. | ||
1021 | |||
1022 | #. *Configure the system to include gdbserver in the target filesystem:* | ||
1023 | |||
1024 | Make the following addition in your ``local.conf`` file:: | ||
1025 | |||
1026 | EXTRA_IMAGE_FEATURES:append = " tools-debug" | ||
1027 | |||
1028 | The change makes | ||
1029 | sure the ``gdbserver`` package is included. | ||
1030 | |||
1031 | #. *Build the environment:* | ||
1032 | |||
1033 | Use the following command to construct the image and the companion | ||
1034 | Debug Filesystem:: | ||
1035 | |||
1036 | $ bitbake image | ||
1037 | |||
1038 | Build the cross GDB component and | ||
1039 | make it available for debugging. Build the SDK that matches the | ||
1040 | image. Building the SDK is best for a production build that can be | ||
1041 | used later for debugging, especially during long term maintenance:: | ||
1042 | |||
1043 | $ bitbake -c populate_sdk image | ||
1044 | |||
1045 | Alternatively, you can build the minimal toolchain components that | ||
1046 | match the target. Doing so creates a smaller than typical SDK and | ||
1047 | only contains a minimal set of components with which to build simple | ||
1048 | test applications, as well as run the debugger:: | ||
1049 | |||
1050 | $ bitbake meta-toolchain | ||
1051 | |||
1052 | A final method is to build Gdb itself within the build system:: | ||
1053 | |||
1054 | $ bitbake gdb-cross-<architecture> | ||
1055 | |||
1056 | Doing so produces a temporary copy of | ||
1057 | ``cross-gdb`` you can use for debugging during development. While | ||
1058 | this is the quickest approach, the two previous methods in this step | ||
1059 | are better when considering long-term maintenance strategies. | ||
1060 | |||
1061 | .. note:: | ||
1062 | |||
1063 | If you run ``bitbake gdb-cross``, the OpenEmbedded build system suggests | ||
1064 | the actual image (e.g. ``gdb-cross-i586``). The suggestion is usually the | ||
1065 | actual name you want to use. | ||
1066 | |||
1067 | #. *Set up the* ``debugfs``\ *:* | ||
1068 | |||
1069 | Run the following commands to set up the ``debugfs``:: | ||
1070 | |||
1071 | $ mkdir debugfs | ||
1072 | $ cd debugfs | ||
1073 | $ tar xvfj build-dir/tmp/deploy/images/machine/image.rootfs.tar.bz2 | ||
1074 | $ tar xvfj build-dir/tmp/deploy/images/machine/image-dbg.rootfs.tar.bz2 | ||
1075 | |||
1076 | #. *Set up GDB:* | ||
1077 | |||
1078 | Install the SDK (if you built one) and then source the correct | ||
1079 | environment file. Sourcing the environment file puts the SDK in your | ||
1080 | ``PATH`` environment variable and sets ``$GDB`` to the SDK's debugger. | ||
1081 | |||
1082 | If you are using the build system, Gdb is located in | ||
1083 | `build-dir`\ ``/tmp/sysroots/``\ `host`\ ``/usr/bin/``\ `architecture`\ ``/``\ `architecture`\ ``-gdb`` | ||
1084 | |||
1085 | #. *Boot the target:* | ||
1086 | |||
1087 | For information on how to run QEMU, see the `QEMU | ||
1088 | Documentation <https://wiki.qemu.org/Documentation/GettingStartedDevelopers>`__. | ||
1089 | |||
1090 | .. note:: | ||
1091 | |||
1092 | Be sure to verify that your host can access the target via TCP. | ||
1093 | |||
1094 | #. *Debug a program:* | ||
1095 | |||
1096 | Debugging a program involves running gdbserver on the target and then | ||
1097 | running Gdb on the host. The example in this step debugs ``gzip``: | ||
1098 | |||
1099 | .. code-block:: shell | ||
1100 | |||
1101 | root@qemux86:~# gdbserver localhost:1234 /bin/gzip —help | ||
1102 | |||
1103 | For | ||
1104 | additional gdbserver options, see the `GDB Server | ||
1105 | Documentation <https://www.gnu.org/software/gdb/documentation/>`__. | ||
1106 | |||
1107 | After running gdbserver on the target, you need to run Gdb on the | ||
1108 | host and configure it and connect to the target. Use these commands:: | ||
1109 | |||
1110 | $ cd directory-holding-the-debugfs-directory | ||
1111 | $ arch-gdb | ||
1112 | (gdb) set sysroot debugfs | ||
1113 | (gdb) set substitute-path /usr/src/debug debugfs/usr/src/debug | ||
1114 | (gdb) target remote IP-of-target:1234 | ||
1115 | |||
1116 | At this | ||
1117 | point, everything should automatically load (i.e. matching binaries, | ||
1118 | symbols and headers). | ||
1119 | |||
1120 | .. note:: | ||
1121 | |||
1122 | The Gdb ``set`` commands in the previous example can be placed into the | ||
1123 | users ``~/.gdbinit`` file. Upon starting, Gdb automatically runs whatever | ||
1124 | commands are in that file. | ||
1125 | |||
1126 | #. *Deploying without a full image rebuild:* | ||
1127 | |||
1128 | In many cases, during development you want a quick method to deploy a | ||
1129 | new binary to the target and debug it, without waiting for a full | ||
1130 | image build. | ||
1131 | |||
1132 | One approach to solving this situation is to just build the component | ||
1133 | you want to debug. Once you have built the component, copy the | ||
1134 | executable directly to both the target and the host ``debugfs``. | ||
1135 | |||
1136 | If the binary is processed through the debug splitting in | ||
1137 | OpenEmbedded, you should also copy the debug items (i.e. ``.debug`` | ||
1138 | contents and corresponding ``/usr/src/debug`` files) from the work | ||
1139 | directory. Here is an example:: | ||
1140 | |||
1141 | $ bitbake bash | ||
1142 | $ bitbake -c devshell bash | ||
1143 | $ cd .. | ||
1144 | $ scp packages-split/bash/bin/bash target:/bin/bash | ||
1145 | $ cp -a packages-split/bash-dbg/\* path/debugfs | ||
1146 | |||
1147 | Debugging with the GNU Project Debugger (GDB) on the Target | ||
1148 | =========================================================== | ||
1149 | |||
1150 | The previous section addressed using GDB remotely for debugging | ||
1151 | purposes, which is the most usual case due to the inherent hardware | ||
1152 | limitations on many embedded devices. However, debugging in the target | ||
1153 | hardware itself is also possible with more powerful devices. This | ||
1154 | section describes what you need to do in order to support using GDB to | ||
1155 | debug on the target hardware. | ||
1156 | |||
1157 | To support this kind of debugging, you need do the following: | ||
1158 | |||
1159 | - Ensure that GDB is on the target. You can do this by making | ||
1160 | the following addition to your ``local.conf`` file:: | ||
1161 | |||
1162 | EXTRA_IMAGE_FEATURES:append = " tools-debug" | ||
1163 | |||
1164 | - Ensure that debug symbols are present. You can do so by adding the | ||
1165 | corresponding ``-dbg`` package to :term:`IMAGE_INSTALL`:: | ||
1166 | |||
1167 | IMAGE_INSTALL:append = " packagename-dbg" | ||
1168 | |||
1169 | Alternatively, you can add the following to ``local.conf`` to include | ||
1170 | all the debug symbols:: | ||
1171 | |||
1172 | EXTRA_IMAGE_FEATURES:append = " dbg-pkgs" | ||
1173 | |||
1174 | .. note:: | ||
1175 | |||
1176 | To improve the debug information accuracy, you can reduce the level | ||
1177 | of optimization used by the compiler. For example, when adding the | ||
1178 | following line to your ``local.conf`` file, you will reduce optimization | ||
1179 | from :term:`FULL_OPTIMIZATION` of "-O2" to :term:`DEBUG_OPTIMIZATION` | ||
1180 | of "-O -fno-omit-frame-pointer":: | ||
1181 | |||
1182 | DEBUG_BUILD = "1" | ||
1183 | |||
1184 | Consider that this will reduce the application's performance and is | ||
1185 | recommended only for debugging purposes. | ||
1186 | |||
1187 | Enabling Minidebuginfo | ||
1188 | ====================== | ||
1189 | |||
1190 | Enabling the :term:`DISTRO_FEATURES` minidebuginfo adds a compressed ELF section ``.gnu_debugdata`` | ||
1191 | to all binary files, containing only function names, and thus increasing the size of the | ||
1192 | binaries only by 5 to 10%. For comparison, full debug symbols can be 10 times as big as | ||
1193 | a stripped binary, and it is thus not always possible to deploy full debug symbols. | ||
1194 | Minidebuginfo data allows, on the one side, to retrieve a call-stack using | ||
1195 | GDB (command backtrace) without deploying full debug symbols to the target. It also | ||
1196 | allows to retrieve a symbolicated call-stack when using ``systemd-coredump`` to manage | ||
1197 | coredumps (commands ``coredumpctl list`` and ``coredumpctl info``). | ||
1198 | |||
1199 | This feature was created by Fedora, see https://fedoraproject.org/wiki/Features/MiniDebugInfo for | ||
1200 | more details. | ||
1201 | |||
1202 | Other Debugging Tips | ||
1203 | ==================== | ||
1204 | |||
1205 | Here are some other tips that you might find useful: | ||
1206 | |||
1207 | - When adding new packages, it is worth watching for undesirable items | ||
1208 | making their way into compiler command lines. For example, you do not | ||
1209 | want references to local system files like ``/usr/lib/`` or | ||
1210 | ``/usr/include/``. | ||
1211 | |||
1212 | - If you want to remove the ``psplash`` boot splashscreen, add | ||
1213 | ``psplash=false`` to the kernel command line. Doing so prevents | ||
1214 | ``psplash`` from loading and thus allows you to see the console. It | ||
1215 | is also possible to switch out of the splashscreen by switching the | ||
1216 | virtual console (e.g. Fn+Left or Fn+Right on a Zaurus). | ||
1217 | |||
1218 | - Removing :term:`TMPDIR` (usually ``tmp/``, within the | ||
1219 | :term:`Build Directory`) can often fix temporary build issues. Removing | ||
1220 | :term:`TMPDIR` is usually a relatively cheap operation, because task output | ||
1221 | will be cached in :term:`SSTATE_DIR` (usually ``sstate-cache/``, which is | ||
1222 | also in the :term:`Build Directory`). | ||
1223 | |||
1224 | .. note:: | ||
1225 | |||
1226 | Removing :term:`TMPDIR` might be a workaround rather than a fix. | ||
1227 | Consequently, trying to determine the underlying cause of an issue before | ||
1228 | removing the directory is a good idea. | ||
1229 | |||
1230 | - Understanding how a feature is used in practice within existing | ||
1231 | recipes can be very helpful. It is recommended that you configure | ||
1232 | some method that allows you to quickly search through files. | ||
1233 | |||
1234 | Using GNU Grep, you can use the following shell function to | ||
1235 | recursively search through common recipe-related files, skipping | ||
1236 | binary files, ``.git`` directories, and the :term:`Build Directory` | ||
1237 | (assuming its name starts with "build"):: | ||
1238 | |||
1239 | g() { | ||
1240 | grep -Ir \ | ||
1241 | --exclude-dir=.git \ | ||
1242 | --exclude-dir='build*' \ | ||
1243 | --include='*.bb*' \ | ||
1244 | --include='*.inc*' \ | ||
1245 | --include='*.conf*' \ | ||
1246 | --include='*.py*' \ | ||
1247 | "$@" | ||
1248 | } | ||
1249 | |||
1250 | Here are some usage examples:: | ||
1251 | |||
1252 | $ g FOO # Search recursively for "FOO" | ||
1253 | $ g -i foo # Search recursively for "foo", ignoring case | ||
1254 | $ g -w FOO # Search recursively for "FOO" as a word, ignoring e.g. "FOOBAR" | ||
1255 | |||
1256 | If figuring | ||
1257 | out how some feature works requires a lot of searching, it might | ||
1258 | indicate that the documentation should be extended or improved. In | ||
1259 | such cases, consider filing a documentation bug using the Yocto | ||
1260 | Project implementation of | ||
1261 | :yocto_bugs:`Bugzilla <>`. For information on | ||
1262 | how to submit a bug against the Yocto Project, see the Yocto Project | ||
1263 | Bugzilla :yocto_wiki:`wiki page </Bugzilla_Configuration_and_Bug_Tracking>` | ||
1264 | and the ":doc:`../contributor-guide/report-defect`" section. | ||
1265 | |||
1266 | .. note:: | ||
1267 | |||
1268 | The manuals might not be the right place to document variables | ||
1269 | that are purely internal and have a limited scope (e.g. internal | ||
1270 | variables used to implement a single ``.bbclass`` file). | ||
1271 | |||