summaryrefslogtreecommitdiffstats
path: root/documentation
diff options
context:
space:
mode:
authorAntonin Godard <antonin.godard@bootlin.com>2025-01-27 10:58:22 +0100
committerRichard Purdie <richard.purdie@linuxfoundation.org>2025-02-07 17:52:48 +0000
commitcfe5da1f75e7b9e43c89cd5f93976f05bb12edfe (patch)
tree0c1cfbf8048a9fb31ac21d597a365624fc2b14e5 /documentation
parent21521c621895019e8fc46b4118f9cbb6abf8b5bb (diff)
downloadpoky-cfe5da1f75e7b9e43c89cd5f93976f05bb12edfe.tar.gz
Move devtool doc from extensible to dev manual
[YOCTO #13417] devtool is not directly tied to the Extensible SDK anymore, and instructions in there can be split and added to the development manual. This patch is a simple bulk move, except for the following: - Intro added to the devtool.rst document (mostly copy-pasted from the devtool quick reference in the ref-manual). - Figures renamed to remove "sdk-" from filenames. - References to sdk-manual/extensible replaced by dev-manual/devtool. Reported-By: Paul Eggleton <bluelightning@bluelightning.org> (From yocto-docs rev: 044d3185b858fce1febcfe3a6834b883f9a598fa) Signed-off-by: Antonin Godard <antonin.godard@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation')
-rw-r--r--documentation/dev-manual/devtool.rst1327
-rw-r--r--documentation/dev-manual/figures/devtool-add-flow.png (renamed from documentation/sdk-manual/figures/sdk-devtool-add-flow.png)bin181699 -> 181699 bytes
-rw-r--r--documentation/dev-manual/figures/devtool-modify-flow.png (renamed from documentation/sdk-manual/figures/sdk-devtool-modify-flow.png)bin171676 -> 171676 bytes
-rw-r--r--documentation/dev-manual/figures/devtool-upgrade-flow.png (renamed from documentation/sdk-manual/figures/sdk-devtool-upgrade-flow.png)bin138917 -> 138917 bytes
-rw-r--r--documentation/dev-manual/index.rst1
-rw-r--r--documentation/dev-manual/new-recipe.rst2
-rw-r--r--documentation/dev-manual/upgrading-recipes.rst2
-rw-r--r--documentation/migration-guides/migration-2.1.rst2
-rw-r--r--documentation/overview-manual/concepts.rst2
-rw-r--r--documentation/ref-manual/devtool-reference.rst2
-rw-r--r--documentation/ref-manual/variables.rst2
-rw-r--r--documentation/sdk-manual/extensible.rst1323
-rw-r--r--documentation/what-i-wish-id-known.rst2
13 files changed, 1338 insertions, 1327 deletions
diff --git a/documentation/dev-manual/devtool.rst b/documentation/dev-manual/devtool.rst
new file mode 100644
index 0000000000..2e3de7a096
--- /dev/null
+++ b/documentation/dev-manual/devtool.rst
@@ -0,0 +1,1327 @@
1.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
2
3Using the ``devtool`` command-line tool
4***************************************
5
6The ``devtool`` command-line tool provides a number of features that
7help you build, test, and package software. This command is available
8alongside the ``bitbake`` command. Additionally, the ``devtool`` command
9is a key part of the :term:`Extensible Software Development Kit (eSDK)`.
10
11Use ``devtool add`` to Add an Application
12=========================================
13
14The ``devtool add`` command generates a new recipe based on existing
15source code. This command takes advantage of the
16:ref:`devtool-the-workspace-layer-structure`
17layer that many ``devtool`` commands use. The command is flexible enough
18to allow you to extract source code into both the workspace or a
19separate local Git repository and to use existing code that does not
20need to be extracted.
21
22Depending on your particular scenario, the arguments and options you use
23with ``devtool add`` form different combinations. The following diagram
24shows common development flows you would use with the ``devtool add``
25command:
26
27.. image:: figures/devtool-add-flow.png
28 :width: 100%
29
30#. *Generating the New Recipe*: The top part of the flow shows three
31 scenarios by which you could use ``devtool add`` to generate a recipe
32 based on existing source code.
33
34 In a shared development environment, it is typical for other
35 developers to be responsible for various areas of source code. As a
36 developer, you are probably interested in using that source code as
37 part of your development within the Yocto Project. All you need is
38 access to the code, a recipe, and a controlled area in which to do
39 your work.
40
41 Within the diagram, three possible scenarios feed into the
42 ``devtool add`` workflow:
43
44 - *Left*: The left scenario in the figure represents a common
45 situation where the source code does not exist locally and needs
46 to be extracted. In this situation, the source code is extracted
47 to the default workspace --- you do not want the files in some
48 specific location outside of the workspace. Thus, everything you
49 need will be located in the workspace::
50
51 $ devtool add recipe fetchuri
52
53 With this command, ``devtool`` extracts the upstream
54 source files into a local Git repository within the ``sources``
55 folder. The command then creates a recipe named recipe and a
56 corresponding append file in the workspace. If you do not provide
57 recipe, the command makes an attempt to determine the recipe name.
58
59 - *Middle*: The middle scenario in the figure also represents a
60 situation where the source code does not exist locally. In this
61 case, the code is again upstream and needs to be extracted to some
62 local area --- this time outside of the default workspace.
63
64 .. note::
65
66 If required, ``devtool`` always creates a Git repository locally
67 during the extraction.
68
69 Furthermore, the first positional argument ``srctree`` in this case
70 identifies where the ``devtool add`` command will locate the
71 extracted code outside of the workspace. You need to specify an
72 empty directory::
73
74 $ devtool add recipe srctree fetchuri
75
76 In summary, the source code is pulled from fetchuri and extracted into the
77 location defined by ``srctree`` as a local Git repository.
78
79 Within workspace, ``devtool`` creates a recipe named recipe along
80 with an associated append file.
81
82 - *Right*: The right scenario in the figure represents a situation
83 where the ``srctree`` has been previously prepared outside of the
84 ``devtool`` workspace.
85
86 The following command provides a new recipe name and identifies
87 the existing source tree location::
88
89 $ devtool add recipe srctree
90
91 The command examines the source code and creates a recipe named
92 recipe for the code and places the recipe into the workspace.
93
94 Because the extracted source code already exists, ``devtool`` does
95 not try to relocate the source code into the workspace --- only the
96 new recipe is placed in the workspace.
97
98 Aside from a recipe folder, the command also creates an associated
99 append folder and places an initial ``*.bbappend`` file within.
100
101#. *Edit the Recipe*: You can use ``devtool edit-recipe`` to open up the
102 editor as defined by the ``$EDITOR`` environment variable and modify
103 the file::
104
105 $ devtool edit-recipe recipe
106
107 From within the editor, you can make modifications to the recipe that
108 take effect when you build it later.
109
110#. *Build the Recipe or Rebuild the Image*: The next step you take
111 depends on what you are going to do with the new code.
112
113 If you need to eventually move the build output to the target
114 hardware, use the following ``devtool`` command::
115
116 $ devtool build recipe
117
118 On the other hand, if you want an image to contain the recipe's
119 packages from the workspace for immediate deployment onto a device
120 (e.g. for testing purposes), you can use the ``devtool build-image``
121 command::
122
123 $ devtool build-image image
124
125#. *Deploy the Build Output*: When you use the ``devtool build`` command
126 to build out your recipe, you probably want to see if the resulting
127 build output works as expected on the target hardware.
128
129 .. note::
130
131 This step assumes you have a previously built image that is
132 already either running in QEMU or is running on actual hardware.
133 Also, it is assumed that for deployment of the image to the
134 target, SSH is installed in the image and, if the image is running
135 on real hardware, you have network access to and from your
136 development machine.
137
138 You can deploy your build output to that target hardware by using the
139 ``devtool deploy-target`` command::
140
141 $ devtool deploy-target recipe target
142
143 The target is a live target machine running as an SSH server.
144
145 You can, of course, also deploy the image you build to actual
146 hardware by using the ``devtool build-image`` command. However,
147 ``devtool`` does not provide a specific command that allows you to
148 deploy the image to actual hardware.
149
150#. *Finish Your Work With the Recipe*: The ``devtool finish`` command
151 creates any patches corresponding to commits in the local Git
152 repository, moves the new recipe to a more permanent layer, and then
153 resets the recipe so that the recipe is built normally rather than
154 from the workspace::
155
156 $ devtool finish recipe layer
157
158 .. note::
159
160 Any changes you want to turn into patches must be committed to the
161 Git repository in the source tree.
162
163 As mentioned, the ``devtool finish`` command moves the final recipe
164 to its permanent layer.
165
166 As a final process of the ``devtool finish`` command, the state of
167 the standard layers and the upstream source is restored so that you
168 can build the recipe from those areas rather than the workspace.
169
170 .. note::
171
172 You can use the ``devtool reset`` command to put things back should you
173 decide you do not want to proceed with your work. If you do use this
174 command, realize that the source tree is preserved.
175
176Use ``devtool modify`` to Modify the Source of an Existing Component
177====================================================================
178
179The ``devtool modify`` command prepares the way to work on existing code
180that already has a local recipe in place that is used to build the
181software. The command is flexible enough to allow you to extract code
182from an upstream source, specify the existing recipe, and keep track of
183and gather any patch files from other developers that are associated
184with the code.
185
186Depending on your particular scenario, the arguments and options you use
187with ``devtool modify`` form different combinations. The following
188diagram shows common development flows for the ``devtool modify``
189command:
190
191.. image:: figures/devtool-modify-flow.png
192 :width: 100%
193
194#. *Preparing to Modify the Code*: The top part of the flow shows three
195 scenarios by which you could use ``devtool modify`` to prepare to
196 work on source files. Each scenario assumes the following:
197
198 - The recipe exists locally in a layer external to the ``devtool``
199 workspace.
200
201 - The source files exist either upstream in an un-extracted state or
202 locally in a previously extracted state.
203
204 The typical situation is where another developer has created a layer
205 for use with the Yocto Project and their recipe already resides in
206 that layer. Furthermore, their source code is readily available
207 either upstream or locally.
208
209 - *Left*: The left scenario in the figure represents a common
210 situation where the source code does not exist locally and it
211 needs to be extracted from an upstream source. In this situation,
212 the source is extracted into the default ``devtool`` workspace
213 location. The recipe, in this scenario, is in its own layer
214 outside the workspace (i.e. ``meta-``\ layername).
215
216 The following command identifies the recipe and, by default,
217 extracts the source files::
218
219 $ devtool modify recipe
220
221 Once ``devtool`` locates the recipe, ``devtool`` uses the recipe's
222 :term:`SRC_URI` statements to locate the source code and any local
223 patch files from other developers.
224
225 With this scenario, there is no ``srctree`` argument. Consequently, the
226 default behavior of the ``devtool modify`` command is to extract
227 the source files pointed to by the :term:`SRC_URI` statements into a
228 local Git structure. Furthermore, the location for the extracted
229 source is the default area within the ``devtool`` workspace. The
230 result is that the command sets up both the source code and an
231 append file within the workspace while the recipe remains in its
232 original location.
233
234 Additionally, if you have any non-patch local files (i.e. files
235 referred to with ``file://`` entries in :term:`SRC_URI` statement
236 excluding ``*.patch/`` or ``*.diff``), these files are copied to
237 an ``oe-local-files`` folder under the newly created source tree.
238 Copying the files here gives you a convenient area from which you
239 can modify the files. Any changes or additions you make to those
240 files are incorporated into the build the next time you build the
241 software just as are other changes you might have made to the
242 source.
243
244 - *Middle*: The middle scenario in the figure represents a situation
245 where the source code also does not exist locally. In this case,
246 the code is again upstream and needs to be extracted to some local
247 area as a Git repository. The recipe, in this scenario, is again
248 local and in its own layer outside the workspace.
249
250 The following command tells ``devtool`` the recipe with which to
251 work and, in this case, identifies a local area for the extracted
252 source files that exists outside of the default ``devtool``
253 workspace::
254
255 $ devtool modify recipe srctree
256
257 .. note::
258
259 You cannot provide a URL for ``srctree`` using the ``devtool`` command.
260
261 As with all extractions, the command uses the recipe's :term:`SRC_URI`
262 statements to locate the source files and any associated patch
263 files. Non-patch files are copied to an ``oe-local-files`` folder
264 under the newly created source tree.
265
266 Once the files are located, the command by default extracts them
267 into ``srctree``.
268
269 Within workspace, ``devtool`` creates an append file for the
270 recipe. The recipe remains in its original location but the source
271 files are extracted to the location you provide with ``srctree``.
272
273 - *Right*: The right scenario in the figure represents a situation
274 where the source tree (``srctree``) already exists locally as a
275 previously extracted Git structure outside of the ``devtool``
276 workspace. In this example, the recipe also exists elsewhere
277 locally in its own layer.
278
279 The following command tells ``devtool`` the recipe with which to
280 work, uses the "-n" option to indicate source does not need to be
281 extracted, and uses ``srctree`` to point to the previously extracted
282 source files::
283
284 $ devtool modify -n recipe srctree
285
286 If an ``oe-local-files`` subdirectory happens to exist and it
287 contains non-patch files, the files are used. However, if the
288 subdirectory does not exist and you run the ``devtool finish``
289 command, any non-patch files that might exist next to the recipe
290 are removed because it appears to ``devtool`` that you have
291 deleted those files.
292
293 Once the ``devtool modify`` command finishes, it creates only an
294 append file for the recipe in the ``devtool`` workspace. The
295 recipe and the source code remain in their original locations.
296
297#. *Edit the Source*: Once you have used the ``devtool modify`` command,
298 you are free to make changes to the source files. You can use any
299 editor you like to make and save your source code modifications.
300
301#. *Build the Recipe or Rebuild the Image*: The next step you take
302 depends on what you are going to do with the new code.
303
304 If you need to eventually move the build output to the target
305 hardware, use the following ``devtool`` command::
306
307 $ devtool build recipe
308
309 On the other hand, if you want an image to contain the recipe's
310 packages from the workspace for immediate deployment onto a device
311 (e.g. for testing purposes), you can use the ``devtool build-image``
312 command::
313
314 $ devtool build-image image
315
316#. *Deploy the Build Output*: When you use the ``devtool build`` command
317 to build out your recipe, you probably want to see if the resulting
318 build output works as expected on target hardware.
319
320 .. note::
321
322 This step assumes you have a previously built image that is
323 already either running in QEMU or running on actual hardware.
324 Also, it is assumed that for deployment of the image to the
325 target, SSH is installed in the image and if the image is running
326 on real hardware that you have network access to and from your
327 development machine.
328
329 You can deploy your build output to that target hardware by using the
330 ``devtool deploy-target`` command::
331
332 $ devtool deploy-target recipe target
333
334 The target is a live target machine running as an SSH server.
335
336 You can, of course, use other methods to deploy the image you built
337 using the ``devtool build-image`` command to actual hardware.
338 ``devtool`` does not provide a specific command to deploy the image
339 to actual hardware.
340
341#. *Finish Your Work With the Recipe*: The ``devtool finish`` command
342 creates any patches corresponding to commits in the local Git
343 repository, updates the recipe to point to them (or creates a
344 ``.bbappend`` file to do so, depending on the specified destination
345 layer), and then resets the recipe so that the recipe is built
346 normally rather than from the workspace::
347
348 $ devtool finish recipe layer
349
350 .. note::
351
352 Any changes you want to turn into patches must be staged and
353 committed within the local Git repository before you use the
354 ``devtool finish`` command.
355
356 Because there is no need to move the recipe, ``devtool finish``
357 either updates the original recipe in the original layer or the
358 command creates a ``.bbappend`` file in a different layer as provided
359 by layer. Any work you did in the ``oe-local-files`` directory is
360 preserved in the original files next to the recipe during the
361 ``devtool finish`` command.
362
363 As a final process of the ``devtool finish`` command, the state of
364 the standard layers and the upstream source is restored so that you
365 can build the recipe from those areas rather than from the workspace.
366
367 .. note::
368
369 You can use the ``devtool reset`` command to put things back should you
370 decide you do not want to proceed with your work. If you do use this
371 command, realize that the source tree is preserved.
372
373``devtool ide-sdk`` configures IDEs and bootstraps SDKs
374=======================================================
375
376The ``devtool ide-sdk`` command can provide an IDE configuration for IDEs when
377working on the source code of one or more recipes.
378Depending on the programming language, and the build system used by the recipe,
379the tools required for cross-development and remote debugging are different.
380For example:
381
382- A C/C++ project usually uses CMake or Meson.
383
384- A Python project uses setuptools or one of its successors.
385
386- A Rust project uses Cargo.
387
388Also, the IDE plugins needed for the integration of a build system with the
389IDE and the corresponding settings are usually specific to these build-systems.
390To hide all these details from the user, ``devtool ide-sdk`` does two things:
391
392- It generates any kind of SDK needed for cross-development and remote
393 debugging of the specified recipes.
394
395- It generates the configuration for the IDE (and the IDE plugins) for using
396 the cross-toolchain and remote debugging tools provided by the SDK directly
397 from the IDE.
398
399For supported build systems the configurations generated by ``devtool ide-sdk``
400combine the advantages of the ``devtool modify`` based workflow
401(see :ref:`using_devtool`) with the advantages of the simple Environment Setup
402script based workflow (see :ref:`running_the_ext_sdk_env`) provided by Yocto's
403SDK or eSDK:
404
405- The source code of the recipe is in the workspace created by
406 ``devtool modify`` or ``devtool add``.
407 Using ``devtool build``, ``devtool build-image``,
408 ``devtool deploy-target`` or ``bitbake`` is possible.
409 Also ``devtool ide-sdk`` can be used to update the SDK and the IDE
410 configuration at any time.
411
412- ``devtool ide-sdk`` aims to support multiple programming languages and
413 multiple IDEs natively. "Natively" means that the IDE is configured to call
414 the build tool (e.g. ``cmake`` or ``meson``) directly. This has several
415 advantages.
416 First of all, it is usually much faster to call for example ``cmake`` than
417 ``devtool build``.
418 It also allows to benefit from the very good integration that IDEs like
419 VSCode offer for tools like CMake or GDB.
420
421 However, supporting many programming languages and multiple
422 IDEs is quite an elaborate and constantly evolving thing. Support for IDEs
423 is therefore implemented as plugins. Plugins can also be provided by
424 optional layers.
425
426So much about the introduction to the default mode of ``devtool sdk-ide`` which
427is called the "modified" mode because it uses the workspace created by
428``devtool modify`` and the per recipe :term:`Sysroots <Sysroot>` of BitBake.
429
430For some recipes and use cases, this default behavior of ``devtool ide-sdk``
431with full ``devtool`` and ``bitbake`` integration might not be suitable.
432To offer full feature parity with the SDK and the eSDK, ``devtool ide-sdk`` has
433a second mode called "shared" mode.
434If ``devtool ide-sdk`` is called with the ``--mode=shared`` option, it
435bootstraps an SDK directly from the BitBake environment, which offers the same
436Environment Setup script as described in :ref:`running_the_ext_sdk_env`.
437In addition to the (e)SDK installer-based setup, the IDE gets configured
438to use the shared :term:`Sysroots <Sysroot>` and the tools from the SDK.
439``devtool ide-sdk --mode=shared`` is basically a wrapper for the setup of the
440extensible SDK as described in :ref:`setting_up_ext_sdk_in_build`.
441
442The use of ``devtool ide-sdk`` is an alternative to using one of the SDK
443installers.
444``devtool ide-sdk`` allows the creation of SDKs that offer all the
445functionality of the SDK and the eSDK installers. Compared to the installers,
446however, the SDK created with ``devtool ide-sdk`` is much more flexible.
447For example, it is very easy to change the :term:`MACHINE` in the
448``local.conf`` file, update the layer meta data and then regenerate the SDK.
449
450Let's take a look at an example of how to use ``devtool ide-sdk`` in each of
451the two modes:
452
453#. *Modified mode*:
454
455 In order to use the ``devtool ide-sdk``, a few settings are needed. As a
456 starting example, the following lines of code can be added to the
457 ``local.conf`` file::
458
459 # Build the companion debug file system
460 IMAGE_GEN_DEBUGFS = "1"
461 # Optimize build time: with devtool ide-sdk the dbg tar is not needed
462 IMAGE_FSTYPES_DEBUGFS = ""
463 # Without copying the binaries into roofs-dbg, GDB does not find all source files.
464 IMAGE_CLASSES += "image-combined-dbg"
465
466 # SSH is mandatory, no password simplifies the usage
467 EXTRA_IMAGE_FEATURES += "\
468 ssh-server-openssh \
469 allow-empty-password \
470 allow-root-login \
471 empty-root-password \
472 "
473
474 # Remote debugging needs gdbserver on the target device
475 IMAGE_INSTALL:append = " gdbserver"
476
477 # Add the recipes which should be modified to the image
478 # Otherwise some dependencies might be missing.
479 IMAGE_INSTALL:append = " my-recipe"
480
481 Assuming the BitBake environment is set up correctly and a workspace has
482 been created for the recipe using ``devtool modify my-recipe`` or probably
483 even better by using ``devtool modify my-recipe --debug-build``, the
484 following command can create the SDK and the configuration for VSCode in
485 the recipe workspace::
486
487 $ devtool ide-sdk my-recipe core-image-minimal --target root@192.168.7.2
488
489 The command requires an image recipe (``core-image-minimal`` for this
490 example) that is used to create the SDK.
491 This firmware image should also be installed on the target device.
492 It is possible to pass multiple package recipes::
493
494 $ devtool ide-sdk my-recipe-1 my-recipe-2 core-image-minimal --target root@192.168.7.2
495
496 ``devtool ide-sdk`` tries to create an IDE configuration for all package
497 recipes.
498
499 What this command does exactly depends on the recipe, more precisely on the
500 build tool used by the recipe. The basic idea is to configure the IDE so
501 that it calls the build tool exactly as ``bitbake`` does.
502
503 For example, a CMake preset is created for a recipe that inherits
504 :ref:`ref-classes-cmake`. In the case of VSCode, CMake presets are supported
505 by the CMake Tools plugin. This is an example of how the build configuration
506 used by ``bitbake`` is exported to an IDE configuration that gives exactly
507 the same build results.
508
509 Support for remote debugging with seamless integration into the IDE is
510 important for a cross-SDK. ``devtool ide-sdk`` automatically generates the
511 necessary helper scripts for deploying the compiled artifacts to the target
512 device as well as the necessary configuration for the debugger and the IDE.
513
514 .. note::
515
516 To ensure that the debug symbols on the build machine match the binaries
517 running on the target device, it is essential that the image built by
518 ``devtool ide-sdk`` is running on the target device.
519
520 The default IDE is VSCode. Some hints about using VSCode:
521
522 - VSCode can be used to work on the BitBake recipes or the application
523 source code.
524 Usually there is one instance of VSCode running in the folder where the
525 BitBake recipes are. This instance has the
526 `Yocto Project BitBake plugin <https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`_
527 running.
528
529 .. warning::
530
531 Some VSCode plugins (Python, BitBake and others) need a reasonable
532 configuration to work as expected. Otherwise, some plugins try to
533 index the build directory of BitBake, which keeps your system quite
534 busy until an out of memory exception stops this nonsense.
535 Other plugins, such as the BitBake plugin, do not behave as expected.
536
537 To work around such issues, the ``oe-init-build-env`` script creates
538 an initial ``.vscode/settings.json`` file if ``code`` can be found
539 and the ``.vscode`` folder does not yet exist.
540 It is best to run ``oe-init-build-env`` once before starting VSCode.
541 An alternative approach is to use a build folder outside the layers,
542 e.g. ``oe-init-build-env ../build``.
543
544 The BitBake plugin also offers to create devtool workspaces and run
545 ``devtool ide-sdk`` with a few mouse clicks.
546 Of course, issuing commands in the terminal works as well.
547
548 - To work on the source code of a recipe another instance of VSCode is
549 started in the recipe's workspace. Example::
550
551 code build/workspace/sources/my-recipe
552
553 This instance of VSCode uses plugins that are useful for the development
554 of the application. ``devtool ide-sdk`` generates the necessary
555 ``extensions.json``, ``settings.json``, ``tasks.json``and ``launch.json``
556 configuration files for all the involved plugins.
557
558 When the source code folder present in the workspace folder is opened in
559 VSCode for the first time, a pop-up message recommends installing the
560 required plugins.
561 After accepting the installation of the plugins, working with the source
562 code or some debugging tasks should work as usual with VSCode.
563
564 Starting the VSCode instances in the recipe workspace folders can also be
565 done by a mouse click on the recipe workspaces in the first VSCode
566 instance.
567
568 - To work with CMake press ``Ctrl + Shift + p``, type ``cmake``. This will
569 show some possible commands like selecting a CMake preset, compiling or
570 running CTest.
571
572 For recipes inheriting :ref:`ref-classes-cmake-qemu` rather than
573 :ref:`ref-classes-cmake`, executing cross-compiled unit tests on the host
574 can be supported transparently with QEMU user-mode.
575
576 - To work with Meson press ``Ctrl + Shift + p``, type ``meson``. This will
577 show some possible commands like compiling or executing the unit tests.
578
579 A note on running cross-compiled unit tests on the host: Meson enables
580 support for QEMU user mode by default. It is expected that the execution
581 of the unit tests from the IDE will work without any additional steps,
582 given that the code is suitable for the execution on the host machine.
583
584 - For the deployment to the target device, just press ``Ctrl + Shift + p``,
585 type ``task``. Select ``install && deploy-target``.
586
587 - For remote debugging, switch to the debugging view by pressing the "play"
588 button with the ``bug icon`` on the left side. This will provide a green
589 play button with a drop-down list where a debug configuration can be
590 selected. After selecting one of the generated configurations, press the
591 "play" button.
592
593 Starting a remote debugging session automatically initiates the
594 deployment to the target device. If this is not desired, the
595 ``"dependsOn": ["install && deploy-target...]`` parameter of the tasks
596 with ``"label": "gdbserver start...`` can be removed from the
597 ``tasks.json`` file.
598
599 VSCode supports GDB with many different setups and configurations for
600 many different use cases. However, most of these setups have some
601 limitations when it comes to cross-development, support only a few target
602 architectures or require a high performance target device. Therefore
603 ``devtool ide-sdk`` supports the classic, generic setup with GDB on the
604 development host and gdbserver on the target device.
605
606 Roughly summarized, this means:
607
608 - The binaries are copied via SSH to the remote target device by a
609 script referred by ``tasks.json``.
610
611 - gdbserver is started on the remote target device via SSH by a script
612 referred by ``tasks.json``.
613
614 Changing the parameters that are passed to the debugging executable
615 requires modifying the generated script. The script is located at
616 ``oe-scripts/gdbserver_*``. Defining the parameters in the ``args``
617 field in the ``launch.json`` file does not work.
618
619 - VSCode connects to gdbserver as documented in
620 `Remote debugging or debugging with a local debugger server
621 <https://code.visualstudio.com/docs/cpp/launch-json-reference#_remote-debugging-or-debugging-with-a-local-debugger-server>`__.
622
623 Additionally ``--ide=none`` is supported. With the ``none`` IDE parameter,
624 some generic configuration files like ``gdbinit`` files and some helper
625 scripts starting gdbserver remotely on the target device as well as the GDB
626 client on the host are generated.
627
628 Here is a usage example for the ``cmake-example`` recipe from the
629 ``meta-selftest`` layer which inherits :ref:`ref-classes-cmake-qemu`:
630
631 .. code-block:: sh
632
633 # Create the SDK
634 devtool modify cmake-example --debug-build
635 devtool ide-sdk cmake-example core-image-minimal -c --ide=none
636
637 # Install the firmware on a target device or start QEMU
638 runqemu
639
640 # From exploring the workspace of cmake-example
641 cd build/workspace/sources/cmake-example
642
643 # Find cmake-native and save the path into a variable
644 # Note: using just cmake instead of $CMAKE_NATIVE would work in many cases
645 CMAKE_NATIVE="$(jq -r '.configurePresets[0] | "\(.cmakeExecutable)"' CMakeUserPresets.json)"
646
647 # List available CMake presets
648 "$CMAKE_NATIVE" --list-presets
649 Available configure presets:
650
651 "cmake-example-cortexa57" - cmake-example: cortexa57
652
653 # Re-compile the already compiled sources
654 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57
655 ninja: no work to do.
656 # Do a clean re-build
657 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target clean
658 [1/1] Cleaning all built files...
659 Cleaning... 8 files.
660 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target all
661 [7/7] Linking CXX executable cmake-example
662
663 # Run the cross-compiled unit tests with QEMU user-mode
664 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target test
665 [0/1] Running tests...
666 Test project .../build/tmp/work/cortexa57-poky-linux/cmake-example/1.0/cmake-example-1.0
667 Start 1: test-cmake-example
668 1/1 Test #1: test-cmake-example ............... Passed 0.03 sec
669
670 100% tests passed, 0 tests failed out of 1
671
672 Total Test time (real) = 0.03 sec
673
674 # Using CTest directly is possible as well
675 CTEST_NATIVE="$(dirname "$CMAKE_NATIVE")/ctest"
676
677 # List available CMake presets
678 "$CTEST_NATIVE" --list-presets
679 Available test presets:
680
681 "cmake-example-cortexa57" - cmake-example: cortexa57
682
683 # Run the cross-compiled unit tests with QEMU user-mode
684 "$CTEST_NATIVE" --preset "cmake-example-cortexa57"
685 Test project ...build/tmp/work/cortexa57-poky-linux/cmake-example/1.0/cmake-example-1.0
686 Start 1: test-cmake-example
687 1/1 Test #1: test-cmake-example ............... Passed 0.03 sec
688
689 100% tests passed, 0 tests failed out of 1
690
691 Total Test time (real) = 0.03 sec
692
693 # Deploying the new build to the target device (default is QEUM at 192.168.7.2)
694 oe-scripts/install_and_deploy_cmake-example-cortexa57
695
696 # Start a remote debugging session with gdbserver on the target and GDB on the host
697 oe-scripts/gdbserver_1234_usr-bin-cmake-example_m
698 oe-scripts/gdb_1234_usr-bin-cmake-example
699 break main
700 run
701 step
702 stepi
703 continue
704 quit
705
706 # Stop gdbserver on the target device
707 oe-scripts/gdbserver_1234_usr-bin-cmake-example_m stop
708
709#. *Shared sysroots mode*
710
711 Creating an SDK with shared :term:`Sysroots <Sysroot>` that contains all the
712 dependencies needed to work with ``my-recipe`` is possible with the following
713 example command::
714
715 $ devtool ide-sdk --mode=shared my-recipe
716
717 For VSCode the cross-toolchain is exposed as a CMake kit. CMake kits are
718 defined in ``~/.local/share/CMakeTools/cmake-tools-kits.json``.
719 The following example shows how the cross-toolchain can be selected in
720 VSCode. First of all we need a folder containing a CMake project.
721 For this example, let's create a CMake project and start VSCode::
722
723 mkdir kit-test
724 echo "project(foo VERSION 1.0)" > kit-test/CMakeLists.txt
725 code kit-test
726
727 If there is a CMake project in the workspace, cross-compilation is
728 supported:
729
730 - Press ``Ctrl + Shift + P``, type ``CMake: Scan for Kits``
731 - Press ``Ctrl + Shift + P``, type ``CMake: Select a Kit``
732
733 Finally most of the features provided by CMake and the IDE should be
734 available.
735
736 Other IDEs than VSCode are supported as well. However,
737 ``devtool ide-sdk --mode=shared --ide=none my-recipe`` is currently
738 just a simple wrapper for the setup of the extensible SDK, as described in
739 :ref:`setting_up_ext_sdk_in_build`.
740
741Use ``devtool upgrade`` to Create a Version of the Recipe that Supports a Newer Version of the Software
742=======================================================================================================
743
744The ``devtool upgrade`` command upgrades an existing recipe to that of a
745more up-to-date version found upstream. Throughout the life of software,
746recipes continually undergo version upgrades by their upstream
747publishers. You can use the ``devtool upgrade`` workflow to make sure
748your recipes you are using for builds are up-to-date with their upstream
749counterparts.
750
751.. note::
752
753 Several methods exist by which you can upgrade recipes ---
754 ``devtool upgrade`` happens to be one. You can read about all the methods by
755 which you can upgrade recipes in the
756 :ref:`dev-manual/upgrading-recipes:upgrading recipes` section of the Yocto
757 Project Development Tasks Manual.
758
759The ``devtool upgrade`` command is flexible enough to allow you to specify
760source code revision and versioning schemes, extract code into or out of the
761``devtool`` :ref:`devtool-the-workspace-layer-structure`, and work with any
762source file forms that the
763:ref:`bitbake-user-manual/bitbake-user-manual-fetching:fetchers` support.
764
765The following diagram shows the common development flow used with the
766``devtool upgrade`` command:
767
768.. image:: figures/devtool-upgrade-flow.png
769 :width: 100%
770
771#. *Initiate the Upgrade*: The top part of the flow shows the typical
772 scenario by which you use the ``devtool upgrade`` command. The
773 following conditions exist:
774
775 - The recipe exists in a local layer external to the ``devtool``
776 workspace.
777
778 - The source files for the new release exist in the same location
779 pointed to by :term:`SRC_URI`
780 in the recipe (e.g. a tarball with the new version number in the
781 name, or as a different revision in the upstream Git repository).
782
783 A common situation is where third-party software has undergone a
784 revision so that it has been upgraded. The recipe you have access to
785 is likely in your own layer. Thus, you need to upgrade the recipe to
786 use the newer version of the software::
787
788 $ devtool upgrade -V version recipe
789
790 By default, the ``devtool upgrade`` command extracts source
791 code into the ``sources`` directory in the
792 :ref:`devtool-the-workspace-layer-structure`.
793 If you want the code extracted to any other location, you need to
794 provide the ``srctree`` positional argument with the command as follows::
795
796 $ devtool upgrade -V version recipe srctree
797
798 .. note::
799
800 In this example, the "-V" option specifies the new version. If you
801 don't use "-V", the command upgrades the recipe to the latest
802 version.
803
804 If the source files pointed to by the :term:`SRC_URI` statement in the
805 recipe are in a Git repository, you must provide the "-S" option and
806 specify a revision for the software.
807
808 Once ``devtool`` locates the recipe, it uses the :term:`SRC_URI` variable
809 to locate the source code and any local patch files from other
810 developers. The result is that the command sets up the source code,
811 the new version of the recipe, and an append file all within the
812 workspace.
813
814 Additionally, if you have any non-patch local files (i.e. files
815 referred to with ``file://`` entries in :term:`SRC_URI` statement
816 excluding ``*.patch/`` or ``*.diff``), these files are copied to an
817 ``oe-local-files`` folder under the newly created source tree.
818 Copying the files here gives you a convenient area from which you can
819 modify the files. Any changes or additions you make to those files
820 are incorporated into the build the next time you build the software
821 just as are other changes you might have made to the source.
822
823#. *Resolve any Conflicts created by the Upgrade*: Conflicts could happen
824 after upgrading the software to a new version. Conflicts occur
825 if your recipe specifies some patch files in :term:`SRC_URI` that
826 conflict with changes made in the new version of the software. For
827 such cases, you need to resolve the conflicts by editing the source
828 and following the normal ``git rebase`` conflict resolution process.
829
830 Before moving onto the next step, be sure to resolve any such
831 conflicts created through use of a newer or different version of the
832 software.
833
834#. *Build the Recipe or Rebuild the Image*: The next step you take
835 depends on what you are going to do with the new code.
836
837 If you need to eventually move the build output to the target
838 hardware, use the following ``devtool`` command::
839
840 $ devtool build recipe
841
842 On the other hand, if you want an image to contain the recipe's
843 packages from the workspace for immediate deployment onto a device
844 (e.g. for testing purposes), you can use the ``devtool build-image``
845 command::
846
847 $ devtool build-image image
848
849#. *Deploy the Build Output*: When you use the ``devtool build`` command
850 or ``bitbake`` to build your recipe, you probably want to see if the
851 resulting build output works as expected on target hardware.
852
853 .. note::
854
855 This step assumes you have a previously built image that is
856 already either running in QEMU or running on actual hardware.
857 Also, it is assumed that for deployment of the image to the
858 target, SSH is installed in the image and if the image is running
859 on real hardware that you have network access to and from your
860 development machine.
861
862 You can deploy your build output to that target hardware by using the
863 ``devtool deploy-target`` command::
864
865 $ devtool deploy-target recipe target
866
867 The target is a live target machine running as an SSH server.
868
869 You can, of course, also deploy the image you build using the
870 ``devtool build-image`` command to actual hardware. However,
871 ``devtool`` does not provide a specific command that allows you to do
872 this.
873
874#. *Finish Your Work With the Recipe*: The ``devtool finish`` command
875 creates any patches corresponding to commits in the local Git
876 repository, moves the new recipe to a more permanent layer, and then
877 resets the recipe so that the recipe is built normally rather than
878 from the workspace.
879
880 Any work you did in the ``oe-local-files`` directory is preserved in
881 the original files next to the recipe during the ``devtool finish``
882 command.
883
884 If you specify a destination layer that is the same as the original
885 source, then the old version of the recipe and associated files are
886 removed prior to adding the new version::
887
888 $ devtool finish recipe layer
889
890 .. note::
891
892 Any changes you want to turn into patches must be committed to the
893 Git repository in the source tree.
894
895 As a final process of the ``devtool finish`` command, the state of
896 the standard layers and the upstream source is restored so that you
897 can build the recipe from those areas rather than the workspace.
898
899 .. note::
900
901 You can use the ``devtool reset`` command to put things back should you
902 decide you do not want to proceed with your work. If you do use this
903 command, realize that the source tree is preserved.
904
905A Closer Look at ``devtool add``
906================================
907
908The ``devtool add`` command automatically creates a recipe based on the
909source tree you provide with the command. Currently, the command has
910support for the following:
911
912- Autotools (``autoconf`` and ``automake``)
913
914- CMake
915
916- Scons
917
918- ``qmake``
919
920- Plain ``Makefile``
921
922- Out-of-tree kernel module
923
924- Binary package (i.e. "-b" option)
925
926- Node.js module
927
928- Python modules that use ``setuptools`` or ``distutils``
929
930Apart from binary packages, the determination of how a source tree
931should be treated is automatic based on the files present within that
932source tree. For example, if a ``CMakeLists.txt`` file is found, then
933the source tree is assumed to be using CMake and is treated accordingly.
934
935.. note::
936
937 In most cases, you need to edit the automatically generated recipe in
938 order to make it build properly. Typically, you would go through
939 several edit and build cycles until the recipe successfully builds.
940 Once the recipe builds, you could use possible further iterations to
941 test the recipe on the target device.
942
943The remainder of this section covers specifics regarding how parts of
944the recipe are generated.
945
946Name and Version
947----------------
948
949If you do not specify a name and version on the command line,
950``devtool add`` uses various metadata within the source tree in an
951attempt to determine the name and version of the software being built.
952Based on what the tool determines, ``devtool`` sets the name of the
953created recipe file accordingly.
954
955If ``devtool`` cannot determine the name and version, the command prints
956an error. For such cases, you must re-run the command and provide the
957name and version, just the name, or just the version as part of the
958command line.
959
960Sometimes the name or version determined from the source tree might be
961incorrect. For such a case, you must reset the recipe::
962
963 $ devtool reset -n recipename
964
965After running the ``devtool reset`` command, you need to
966run ``devtool add`` again and provide the name or the version.
967
968Dependency Detection and Mapping
969--------------------------------
970
971The ``devtool add`` command attempts to detect build-time dependencies and map
972them to other recipes in the system. During this mapping, the command fills in
973the names of those recipes as part of the :term:`DEPENDS` variable within the
974recipe. If a dependency cannot be mapped, ``devtool`` places a comment
975in the recipe indicating such. The inability to map a dependency can
976result from naming not being recognized or because the dependency simply
977is not available. For cases where the dependency is not available, you
978must use the ``devtool add`` command to add an additional recipe that
979satisfies the dependency. Once you add that recipe, you need to update
980the :term:`DEPENDS` variable in the original recipe to include the new
981recipe.
982
983If you need to add runtime dependencies, you can do so by adding the
984following to your recipe::
985
986 RDEPENDS:${PN} += "dependency1 dependency2 ..."
987
988.. note::
989
990 The ``devtool add`` command often cannot distinguish between mandatory and
991 optional dependencies. Consequently, some of the detected dependencies might
992 in fact be optional. When in doubt, consult the documentation or the
993 configure script for the software the recipe is building for further
994 details. In some cases, you might find you can substitute the
995 dependency with an option that disables the associated functionality
996 passed to the configure script.
997
998License Detection
999-----------------
1000
1001The ``devtool add`` command attempts to determine if the software you are
1002adding is able to be distributed under a common, open-source license. If
1003so, the command sets the :term:`LICENSE` value accordingly.
1004You should double-check the value added by the command against the
1005documentation or source files for the software you are building and, if
1006necessary, update that :term:`LICENSE` value.
1007
1008The ``devtool add`` command also sets the :term:`LIC_FILES_CHKSUM`
1009value to point to all files that appear to be license-related. Realize
1010that license statements often appear in comments at the top of source
1011files or within the documentation. In such cases, the command does not
1012recognize those license statements. Consequently, you might need to
1013amend the :term:`LIC_FILES_CHKSUM` variable to point to one or more of those
1014comments if present. Setting :term:`LIC_FILES_CHKSUM` is particularly
1015important for third-party software. The mechanism attempts to ensure
1016correct licensing should you upgrade the recipe to a newer upstream
1017version in future. Any change in licensing is detected and you receive
1018an error prompting you to check the license text again.
1019
1020If the ``devtool add`` command cannot determine licensing information,
1021``devtool`` sets the :term:`LICENSE` value to "CLOSED" and leaves the
1022:term:`LIC_FILES_CHKSUM` value unset. This behavior allows you to continue
1023with development even though the settings are unlikely to be correct in
1024all cases. You should check the documentation or source files for the
1025software you are building to determine the actual license.
1026
1027Adding Makefile-Only Software
1028-----------------------------
1029
1030The use of Make by itself is very common in both proprietary and
1031open-source software. Unfortunately, Makefiles are often not written
1032with cross-compilation in mind. Thus, ``devtool add`` often cannot do
1033very much to ensure that these Makefiles build correctly. It is very
1034common, for example, to explicitly call ``gcc`` instead of using the
1035:term:`CC` variable. Usually, in a
1036cross-compilation environment, ``gcc`` is the compiler for the build
1037host and the cross-compiler is named something similar to
1038``arm-poky-linux-gnueabi-gcc`` and might require arguments (e.g. to
1039point to the associated sysroot for the target machine).
1040
1041When writing a recipe for Makefile-only software, keep the following in
1042mind:
1043
1044- You probably need to patch the Makefile to use variables instead of
1045 hardcoding tools within the toolchain such as ``gcc`` and ``g++``.
1046
1047- The environment in which Make runs is set up with various standard
1048 variables for compilation (e.g. :term:`CC`, :term:`CXX`, and so forth) in a
1049 similar manner to the environment set up by the SDK's environment
1050 setup script. One easy way to see these variables is to run the
1051 ``devtool build`` command on the recipe and then look in
1052 ``oe-logs/run.do_compile``. Towards the top of this file, there is
1053 a list of environment variables that are set. You can take
1054 advantage of these variables within the Makefile.
1055
1056- If the Makefile sets a default for a variable using "=", that default
1057 overrides the value set in the environment, which is usually not
1058 desirable. For this case, you can either patch the Makefile so it
1059 sets the default using the "?=" operator, or you can alternatively
1060 force the value on the ``make`` command line. To force the value on
1061 the command line, add the variable setting to
1062 :term:`EXTRA_OEMAKE` or
1063 :term:`PACKAGECONFIG_CONFARGS`
1064 within the recipe. Here is an example using :term:`EXTRA_OEMAKE`::
1065
1066 EXTRA_OEMAKE += "'CC=${CC}' 'CXX=${CXX}'"
1067
1068 In the above example,
1069 single quotes are used around the variable settings as the values are
1070 likely to contain spaces because required default options are passed
1071 to the compiler.
1072
1073- Hardcoding paths inside Makefiles is often problematic in a
1074 cross-compilation environment. This is particularly true because
1075 those hardcoded paths often point to locations on the build host and
1076 thus will either be read-only or will introduce contamination into
1077 the cross-compilation because they are specific to the build host
1078 rather than the target. Patching the Makefile to use prefix variables
1079 or other path variables is usually the way to handle this situation.
1080
1081- Sometimes a Makefile runs target-specific commands such as
1082 ``ldconfig``. For such cases, you might be able to apply patches that
1083 remove these commands from the Makefile.
1084
1085Adding Native Tools
1086-------------------
1087
1088Often, you need to build additional tools that run on the :term:`Build Host`
1089as opposed to the target. You should indicate this requirement by using one of
1090the following methods when you run ``devtool add``:
1091
1092- Specify the name of the recipe such that it ends with "-native".
1093 Specifying the name like this produces a recipe that only builds for
1094 the build host.
1095
1096- Specify the "--also-native" option with the ``devtool add``
1097 command. Specifying this option creates a recipe file that still
1098 builds for the target but also creates a variant with a "-native"
1099 suffix that builds for the build host.
1100
1101.. note::
1102
1103 If you need to add a tool that is shipped as part of a source tree
1104 that builds code for the target, you can typically accomplish this by
1105 building the native and target parts separately rather than within
1106 the same compilation process. Realize though that with the
1107 "--also-native" option, you can add the tool using just one
1108 recipe file.
1109
1110Adding Node.js Modules
1111----------------------
1112
1113You can use the ``devtool add`` command two different ways to add
1114Node.js modules: through ``npm`` or from a repository or local source.
1115
1116Use the following form to add Node.js modules through ``npm``::
1117
1118 $ devtool add "npm://registry.npmjs.org;name=forever;version=0.15.1"
1119
1120The name and
1121version parameters are mandatory. Lockdown and shrinkwrap files are
1122generated and pointed to by the recipe in order to freeze the version
1123that is fetched for the dependencies according to the first time. This
1124also saves checksums that are verified on future fetches. Together,
1125these behaviors ensure the reproducibility and integrity of the build.
1126
1127.. note::
1128
1129 - You must use quotes around the URL. ``devtool add`` does not
1130 require the quotes, but the shell considers ";" as a splitter
1131 between multiple commands. Thus, without the quotes,
1132 ``devtool add`` does not receive the other parts, which results in
1133 several "command not found" errors.
1134
1135 - In order to support adding Node.js modules, a ``nodejs`` recipe
1136 must be part of your SDK.
1137
1138As mentioned earlier, you can also add Node.js modules directly from a
1139repository or local source tree. To add modules this way, use
1140``devtool add`` in the following form::
1141
1142 $ devtool add https://github.com/diversario/node-ssdp
1143
1144In this example, ``devtool`` fetches the specified Git repository, detects the
1145code as Node.js code, fetches dependencies using ``npm``, and sets
1146:term:`SRC_URI` accordingly.
1147
1148Working With Recipes
1149====================
1150
1151When building a recipe using the ``devtool build`` command, the typical
1152build progresses as follows:
1153
1154#. Fetch the source
1155
1156#. Unpack the source
1157
1158#. Configure the source
1159
1160#. Compile the source
1161
1162#. Install the build output
1163
1164#. Package the installed output
1165
1166For recipes in the workspace, fetching and unpacking is disabled as the
1167source tree has already been prepared and is persistent. Each of these
1168build steps is defined as a function (task), usually with a "do\_" prefix
1169(e.g. :ref:`ref-tasks-fetch`,
1170:ref:`ref-tasks-unpack`, and so
1171forth). These functions are typically shell scripts but can instead be
1172written in Python.
1173
1174If you look at the contents of a recipe, you will see that the recipe
1175does not include complete instructions for building the software.
1176Instead, common functionality is encapsulated in classes inherited with
1177the ``inherit`` directive. This technique leaves the recipe to describe
1178just the things that are specific to the software being built. There is
1179a :ref:`ref-classes-base` class that is implicitly inherited by all recipes
1180and provides the functionality that most recipes typically need.
1181
1182The remainder of this section presents information useful when working
1183with recipes.
1184
1185Finding Logs and Work Files
1186---------------------------
1187
1188After the first run of the ``devtool build`` command, recipes that were
1189previously created using the ``devtool add`` command or whose sources
1190were modified using the ``devtool modify`` command contain symbolic
1191links created within the source tree:
1192
1193- ``oe-logs``: This link points to the directory in which log files and
1194 run scripts for each build step are created.
1195
1196- ``oe-workdir``: This link points to the temporary work area for the
1197 recipe. The following locations under ``oe-workdir`` are particularly
1198 useful:
1199
1200 - ``image/``: Contains all of the files installed during the
1201 :ref:`ref-tasks-install` stage.
1202 Within a recipe, this directory is referred to by the expression
1203 ``${``\ :term:`D`\ ``}``.
1204
1205 - ``sysroot-destdir/``: Contains a subset of files installed within
1206 :ref:`ref-tasks-install` that have been put into the shared sysroot. For
1207 more information, see the
1208 ":ref:`dev-manual/new-recipe:sharing files between recipes`" section.
1209
1210 - ``packages-split/``: Contains subdirectories for each package
1211 produced by the recipe. For more information, see the
1212 ":ref:`dev-manual/devtool:packaging`" section.
1213
1214You can use these links to get more information on what is happening at
1215each build step.
1216
1217Setting Configure Arguments
1218---------------------------
1219
1220If the software your recipe is building uses GNU autoconf, then a fixed
1221set of arguments is passed to it to enable cross-compilation plus any
1222extras specified by :term:`EXTRA_OECONF` or :term:`PACKAGECONFIG_CONFARGS`
1223set within the recipe. If you wish to pass additional options, add them
1224to :term:`EXTRA_OECONF` or :term:`PACKAGECONFIG_CONFARGS`. Other supported build
1225tools have similar variables (e.g. :term:`EXTRA_OECMAKE` for CMake,
1226:term:`EXTRA_OESCONS` for Scons, and so forth). If you need to pass anything on
1227the ``make`` command line, you can use :term:`EXTRA_OEMAKE` or the
1228:term:`PACKAGECONFIG_CONFARGS` variables to do so.
1229
1230You can use the ``devtool configure-help`` command to help you set the
1231arguments listed in the previous paragraph. The command determines the
1232exact options being passed, and shows them to you along with any custom
1233arguments specified through :term:`EXTRA_OECONF` or
1234:term:`PACKAGECONFIG_CONFARGS`. If applicable, the command also shows you
1235the output of the configure script's "--help" option as a
1236reference.
1237
1238Sharing Files Between Recipes
1239-----------------------------
1240
1241Recipes often need to use files provided by other recipes on the
1242:term:`Build Host`. For example,
1243an application linking to a common library needs access to the library
1244itself and its associated headers. The way this access is accomplished
1245within the extensible SDK is through the sysroot. There is one sysroot per
1246"machine" for which the SDK is being built. In practical terms, this
1247means there is a sysroot for the target machine, and a sysroot for
1248the build host.
1249
1250Recipes should never write files directly into the sysroot. Instead,
1251files should be installed into standard locations during the
1252:ref:`ref-tasks-install` task within the ``${``\ :term:`D`\ ``}`` directory. A
1253subset of these files automatically goes into the sysroot. The reason
1254for this limitation is that almost all files that go into the sysroot
1255are cataloged in manifests in order to ensure they can be removed later
1256when a recipe is modified or removed. Thus, the sysroot is able to
1257remain free from stale files.
1258
1259Packaging
1260---------
1261
1262Packaging is not always particularly relevant within the extensible SDK.
1263However, if you examine how build output gets into the final image on
1264the target device, it is important to understand packaging because the
1265contents of the image are expressed in terms of packages and not
1266recipes.
1267
1268During the :ref:`ref-tasks-package` task, files installed during the
1269:ref:`ref-tasks-install` task are split into one main package, which is almost
1270always named the same as the recipe, and into several other packages. This
1271separation exists because not all of those installed files are useful in every
1272image. For example, you probably do not need any of the documentation installed
1273in a production image. Consequently, for each recipe the documentation
1274files are separated into a ``-doc`` package. Recipes that package
1275software containing optional modules or plugins might undergo additional
1276package splitting as well.
1277
1278After building a recipe, you can see where files have gone by looking in
1279the ``oe-workdir/packages-split`` directory, which contains a
1280subdirectory for each package. Apart from some advanced cases, the
1281:term:`PACKAGES` and :term:`FILES` variables controls
1282splitting. The :term:`PACKAGES` variable lists all of the packages to be
1283produced, while the :term:`FILES` variable specifies which files to include
1284in each package by using an override to specify the package. For
1285example, ``FILES:${PN}`` specifies the files to go into the main package
1286(i.e. the main package has the same name as the recipe and
1287``${``\ :term:`PN`\ ``}`` evaluates to the
1288recipe name). The order of the :term:`PACKAGES` value is significant. For
1289each installed file, the first package whose :term:`FILES` value matches the
1290file is the package into which the file goes. Both the :term:`PACKAGES` and
1291:term:`FILES` variables have default values. Consequently, you might find
1292you do not even need to set these variables in your recipe unless the
1293software the recipe is building installs files into non-standard
1294locations.
1295
1296Restoring the Target Device to its Original State
1297=================================================
1298
1299If you use the ``devtool deploy-target`` command to write a recipe's
1300build output to the target, and you are working on an existing component
1301of the system, then you might find yourself in a situation where you
1302need to restore the original files that existed prior to running the
1303``devtool deploy-target`` command. Because the ``devtool deploy-target``
1304command backs up any files it overwrites, you can use the
1305``devtool undeploy-target`` command to restore those files and remove
1306any other files the recipe deployed. Consider the following example::
1307
1308 $ devtool undeploy-target lighttpd root@192.168.7.2
1309
1310If you have deployed
1311multiple applications, you can remove them all using the "-a" option
1312thus restoring the target device to its original state::
1313
1314 $ devtool undeploy-target -a root@192.168.7.2
1315
1316Information about files deployed to
1317the target as well as any backed up files are stored on the target
1318itself. This storage, of course, requires some additional space on the
1319target machine.
1320
1321.. note::
1322
1323 The ``devtool deploy-target`` and ``devtool undeploy-target`` commands do
1324 not currently interact with any package management system on the target
1325 device (e.g. RPM or OPKG). Consequently, you should not intermingle
1326 ``devtool deploy-target`` and package manager operations on the target
1327 device. Doing so could result in a conflicting set of files.
diff --git a/documentation/sdk-manual/figures/sdk-devtool-add-flow.png b/documentation/dev-manual/figures/devtool-add-flow.png
index e7d6173d2d..e7d6173d2d 100644
--- a/documentation/sdk-manual/figures/sdk-devtool-add-flow.png
+++ b/documentation/dev-manual/figures/devtool-add-flow.png
Binary files differ
diff --git a/documentation/sdk-manual/figures/sdk-devtool-modify-flow.png b/documentation/dev-manual/figures/devtool-modify-flow.png
index 18ba8b7e65..18ba8b7e65 100644
--- a/documentation/sdk-manual/figures/sdk-devtool-modify-flow.png
+++ b/documentation/dev-manual/figures/devtool-modify-flow.png
Binary files differ
diff --git a/documentation/sdk-manual/figures/sdk-devtool-upgrade-flow.png b/documentation/dev-manual/figures/devtool-upgrade-flow.png
index 7d4f395e24..7d4f395e24 100644
--- a/documentation/sdk-manual/figures/sdk-devtool-upgrade-flow.png
+++ b/documentation/dev-manual/figures/devtool-upgrade-flow.png
Binary files differ
diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
index 8243c0f4cb..760108f605 100644
--- a/documentation/dev-manual/index.rst
+++ b/documentation/dev-manual/index.rst
@@ -23,6 +23,7 @@ Yocto Project Development Tasks Manual
23 speeding-up-build 23 speeding-up-build
24 libraries 24 libraries
25 prebuilt-libraries 25 prebuilt-libraries
26 devtool
26 x32-psabi 27 x32-psabi
27 gobject-introspection 28 gobject-introspection
28 external-toolchain 29 external-toolchain
diff --git a/documentation/dev-manual/new-recipe.rst b/documentation/dev-manual/new-recipe.rst
index 944b9627f9..4a65d70243 100644
--- a/documentation/dev-manual/new-recipe.rst
+++ b/documentation/dev-manual/new-recipe.rst
@@ -56,7 +56,7 @@ necessary when adding a recipe to build a new piece of software to be
56included in a build. 56included in a build.
57 57
58You can find a complete description of the ``devtool add`` command in 58You can find a complete description of the ``devtool add`` command in
59the ":ref:`sdk-manual/extensible:a closer look at \`\`devtool add\`\``" section 59the ":ref:`dev-manual/devtool:a closer look at \`\`devtool add\`\``" section
60in the Yocto Project Application Development and the Extensible Software 60in the Yocto Project Application Development and the Extensible Software
61Development Kit (eSDK) manual. 61Development Kit (eSDK) manual.
62 62
diff --git a/documentation/dev-manual/upgrading-recipes.rst b/documentation/dev-manual/upgrading-recipes.rst
index 4fac78bdfb..35901a0ea8 100644
--- a/documentation/dev-manual/upgrading-recipes.rst
+++ b/documentation/dev-manual/upgrading-recipes.rst
@@ -203,7 +203,7 @@ As mentioned earlier, an alternative method for upgrading recipes to
203newer versions is to use 203newer versions is to use
204:doc:`devtool upgrade </ref-manual/devtool-reference>`. 204:doc:`devtool upgrade </ref-manual/devtool-reference>`.
205You can read about ``devtool upgrade`` in general in the 205You can read about ``devtool upgrade`` in general in the
206":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`" 206":ref:`dev-manual/devtool:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
207section in the Yocto Project Application Development and the Extensible 207section in the Yocto Project Application Development and the Extensible
208Software Development Kit (eSDK) Manual. 208Software Development Kit (eSDK) Manual.
209 209
diff --git a/documentation/migration-guides/migration-2.1.rst b/documentation/migration-guides/migration-2.1.rst
index 18b05b52cc..4d7aa15af5 100644
--- a/documentation/migration-guides/migration-2.1.rst
+++ b/documentation/migration-guides/migration-2.1.rst
@@ -382,7 +382,7 @@ These additional changes exist:
382 removed at runtime). 382 removed at runtime).
383 383
384- The 384- The
385 :ref:`devtool modify <sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component>` 385 :ref:`devtool modify <dev-manual/devtool:use \`\`devtool modify\`\` to modify the source of an existing component>`
386 command now defaults to extracting the source since that is most 386 command now defaults to extracting the source since that is most
387 commonly expected. The ``-x`` or ``--extract`` options are now no-ops. If 387 commonly expected. The ``-x`` or ``--extract`` options are now no-ops. If
388 you wish to provide your own existing source tree, you will now need 388 you wish to provide your own existing source tree, you will now need
diff --git a/documentation/overview-manual/concepts.rst b/documentation/overview-manual/concepts.rst
index 70766559f8..d1fd6befc8 100644
--- a/documentation/overview-manual/concepts.rst
+++ b/documentation/overview-manual/concepts.rst
@@ -792,7 +792,7 @@ processes patches, see the
792":ref:`dev-manual/new-recipe:patching code`" 792":ref:`dev-manual/new-recipe:patching code`"
793section in the 793section in the
794Yocto Project Development Tasks Manual. You can also see the 794Yocto Project Development Tasks Manual. You can also see the
795":ref:`sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component`" 795":ref:`dev-manual/devtool:use \`\`devtool modify\`\` to modify the source of an existing component`"
796section in the Yocto Project Application Development and the Extensible 796section in the Yocto Project Application Development and the Extensible
797Software Development Kit (SDK) manual and the 797Software Development Kit (SDK) manual and the
798":ref:`kernel-dev/common:using traditional kernel development to patch the kernel`" 798":ref:`kernel-dev/common:using traditional kernel development to patch the kernel`"
diff --git a/documentation/ref-manual/devtool-reference.rst b/documentation/ref-manual/devtool-reference.rst
index a6ea00c210..e173720e1a 100644
--- a/documentation/ref-manual/devtool-reference.rst
+++ b/documentation/ref-manual/devtool-reference.rst
@@ -432,7 +432,7 @@ revision to which you want to upgrade (i.e. the
432forth. 432forth.
433 433
434You can read more on the ``devtool upgrade`` workflow in the 434You can read more on the ``devtool upgrade`` workflow in the
435":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`" 435":ref:`dev-manual/devtool:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
436section in the Yocto Project Application Development and the Extensible 436section in the Yocto Project Application Development and the Extensible
437Software Development Kit (eSDK) manual. You can also see an example of 437Software Development Kit (eSDK) manual. You can also see an example of
438how to use ``devtool upgrade`` in the ":ref:`dev-manual/upgrading-recipes:using \`\`devtool upgrade\`\``" 438how to use ``devtool upgrade`` in the ":ref:`dev-manual/upgrading-recipes:using \`\`devtool upgrade\`\``"
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 47d4e814fd..86023195d3 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -8642,7 +8642,7 @@ system and gives an overview of their function and contents.
8642 8642
8643 For information on how staging for recipe-specific sysroots occurs, 8643 For information on how staging for recipe-specific sysroots occurs,
8644 see the :ref:`ref-tasks-populate_sysroot` 8644 see the :ref:`ref-tasks-populate_sysroot`
8645 task, the ":ref:`sdk-manual/extensible:sharing files between recipes`" 8645 task, the ":ref:`dev-manual/devtool:sharing files between recipes`"
8646 section in the Yocto Project Development Tasks Manual, the 8646 section in the Yocto Project Development Tasks Manual, the
8647 ":ref:`overview-manual/concepts:configuration, compilation, and staging`" 8647 ":ref:`overview-manual/concepts:configuration, compilation, and staging`"
8648 section in the Yocto Project Overview and Concepts Manual, and the 8648 section in the Yocto Project Overview and Concepts Manual, and the
diff --git a/documentation/sdk-manual/extensible.rst b/documentation/sdk-manual/extensible.rst
index 8483951bef..dcb02c4e00 100644
--- a/documentation/sdk-manual/extensible.rst
+++ b/documentation/sdk-manual/extensible.rst
@@ -257,1326 +257,9 @@ the recipe a source tree that is under your control is used in order to
257allow you to make changes to the source as desired. By default, new 257allow you to make changes to the source as desired. By default, new
258recipes and the source go into a "workspace" directory under the SDK. 258recipes and the source go into a "workspace" directory under the SDK.
259 259
260The remainder of this section presents the ``devtool add``, 260To learn how to use ``devtool`` to add, modify, upgrade recipes and more, see
261``devtool modify``, and ``devtool upgrade`` workflows. 261the :ref:`dev-manual/devtool:Using the \`\`devtool\`\` command-line tool`
262 262section of the Yocto Project Development Tasks Manual.
263Use ``devtool add`` to Add an Application
264-----------------------------------------
265
266The ``devtool add`` command generates a new recipe based on existing
267source code. This command takes advantage of the
268:ref:`devtool-the-workspace-layer-structure`
269layer that many ``devtool`` commands use. The command is flexible enough
270to allow you to extract source code into both the workspace or a
271separate local Git repository and to use existing code that does not
272need to be extracted.
273
274Depending on your particular scenario, the arguments and options you use
275with ``devtool add`` form different combinations. The following diagram
276shows common development flows you would use with the ``devtool add``
277command:
278
279.. image:: figures/sdk-devtool-add-flow.png
280 :width: 100%
281
282#. *Generating the New Recipe*: The top part of the flow shows three
283 scenarios by which you could use ``devtool add`` to generate a recipe
284 based on existing source code.
285
286 In a shared development environment, it is typical for other
287 developers to be responsible for various areas of source code. As a
288 developer, you are probably interested in using that source code as
289 part of your development within the Yocto Project. All you need is
290 access to the code, a recipe, and a controlled area in which to do
291 your work.
292
293 Within the diagram, three possible scenarios feed into the
294 ``devtool add`` workflow:
295
296 - *Left*: The left scenario in the figure represents a common
297 situation where the source code does not exist locally and needs
298 to be extracted. In this situation, the source code is extracted
299 to the default workspace --- you do not want the files in some
300 specific location outside of the workspace. Thus, everything you
301 need will be located in the workspace::
302
303 $ devtool add recipe fetchuri
304
305 With this command, ``devtool`` extracts the upstream
306 source files into a local Git repository within the ``sources``
307 folder. The command then creates a recipe named recipe and a
308 corresponding append file in the workspace. If you do not provide
309 recipe, the command makes an attempt to determine the recipe name.
310
311 - *Middle*: The middle scenario in the figure also represents a
312 situation where the source code does not exist locally. In this
313 case, the code is again upstream and needs to be extracted to some
314 local area --- this time outside of the default workspace.
315
316 .. note::
317
318 If required, ``devtool`` always creates a Git repository locally
319 during the extraction.
320
321 Furthermore, the first positional argument ``srctree`` in this case
322 identifies where the ``devtool add`` command will locate the
323 extracted code outside of the workspace. You need to specify an
324 empty directory::
325
326 $ devtool add recipe srctree fetchuri
327
328 In summary, the source code is pulled from fetchuri and extracted into the
329 location defined by ``srctree`` as a local Git repository.
330
331 Within workspace, ``devtool`` creates a recipe named recipe along
332 with an associated append file.
333
334 - *Right*: The right scenario in the figure represents a situation
335 where the ``srctree`` has been previously prepared outside of the
336 ``devtool`` workspace.
337
338 The following command provides a new recipe name and identifies
339 the existing source tree location::
340
341 $ devtool add recipe srctree
342
343 The command examines the source code and creates a recipe named
344 recipe for the code and places the recipe into the workspace.
345
346 Because the extracted source code already exists, ``devtool`` does
347 not try to relocate the source code into the workspace --- only the
348 new recipe is placed in the workspace.
349
350 Aside from a recipe folder, the command also creates an associated
351 append folder and places an initial ``*.bbappend`` file within.
352
353#. *Edit the Recipe*: You can use ``devtool edit-recipe`` to open up the
354 editor as defined by the ``$EDITOR`` environment variable and modify
355 the file::
356
357 $ devtool edit-recipe recipe
358
359 From within the editor, you can make modifications to the recipe that
360 take effect when you build it later.
361
362#. *Build the Recipe or Rebuild the Image*: The next step you take
363 depends on what you are going to do with the new code.
364
365 If you need to eventually move the build output to the target
366 hardware, use the following ``devtool`` command::
367
368 $ devtool build recipe
369
370 On the other hand, if you want an image to contain the recipe's
371 packages from the workspace for immediate deployment onto a device
372 (e.g. for testing purposes), you can use the ``devtool build-image``
373 command::
374
375 $ devtool build-image image
376
377#. *Deploy the Build Output*: When you use the ``devtool build`` command
378 to build out your recipe, you probably want to see if the resulting
379 build output works as expected on the target hardware.
380
381 .. note::
382
383 This step assumes you have a previously built image that is
384 already either running in QEMU or is running on actual hardware.
385 Also, it is assumed that for deployment of the image to the
386 target, SSH is installed in the image and, if the image is running
387 on real hardware, you have network access to and from your
388 development machine.
389
390 You can deploy your build output to that target hardware by using the
391 ``devtool deploy-target`` command::
392
393 $ devtool deploy-target recipe target
394
395 The target is a live target machine running as an SSH server.
396
397 You can, of course, also deploy the image you build to actual
398 hardware by using the ``devtool build-image`` command. However,
399 ``devtool`` does not provide a specific command that allows you to
400 deploy the image to actual hardware.
401
402#. *Finish Your Work With the Recipe*: The ``devtool finish`` command
403 creates any patches corresponding to commits in the local Git
404 repository, moves the new recipe to a more permanent layer, and then
405 resets the recipe so that the recipe is built normally rather than
406 from the workspace::
407
408 $ devtool finish recipe layer
409
410 .. note::
411
412 Any changes you want to turn into patches must be committed to the
413 Git repository in the source tree.
414
415 As mentioned, the ``devtool finish`` command moves the final recipe
416 to its permanent layer.
417
418 As a final process of the ``devtool finish`` command, the state of
419 the standard layers and the upstream source is restored so that you
420 can build the recipe from those areas rather than the workspace.
421
422 .. note::
423
424 You can use the ``devtool reset`` command to put things back should you
425 decide you do not want to proceed with your work. If you do use this
426 command, realize that the source tree is preserved.
427
428Use ``devtool modify`` to Modify the Source of an Existing Component
429--------------------------------------------------------------------
430
431The ``devtool modify`` command prepares the way to work on existing code
432that already has a local recipe in place that is used to build the
433software. The command is flexible enough to allow you to extract code
434from an upstream source, specify the existing recipe, and keep track of
435and gather any patch files from other developers that are associated
436with the code.
437
438Depending on your particular scenario, the arguments and options you use
439with ``devtool modify`` form different combinations. The following
440diagram shows common development flows for the ``devtool modify``
441command:
442
443.. image:: figures/sdk-devtool-modify-flow.png
444 :width: 100%
445
446#. *Preparing to Modify the Code*: The top part of the flow shows three
447 scenarios by which you could use ``devtool modify`` to prepare to
448 work on source files. Each scenario assumes the following:
449
450 - The recipe exists locally in a layer external to the ``devtool``
451 workspace.
452
453 - The source files exist either upstream in an un-extracted state or
454 locally in a previously extracted state.
455
456 The typical situation is where another developer has created a layer
457 for use with the Yocto Project and their recipe already resides in
458 that layer. Furthermore, their source code is readily available
459 either upstream or locally.
460
461 - *Left*: The left scenario in the figure represents a common
462 situation where the source code does not exist locally and it
463 needs to be extracted from an upstream source. In this situation,
464 the source is extracted into the default ``devtool`` workspace
465 location. The recipe, in this scenario, is in its own layer
466 outside the workspace (i.e. ``meta-``\ layername).
467
468 The following command identifies the recipe and, by default,
469 extracts the source files::
470
471 $ devtool modify recipe
472
473 Once ``devtool`` locates the recipe, ``devtool`` uses the recipe's
474 :term:`SRC_URI` statements to locate the source code and any local
475 patch files from other developers.
476
477 With this scenario, there is no ``srctree`` argument. Consequently, the
478 default behavior of the ``devtool modify`` command is to extract
479 the source files pointed to by the :term:`SRC_URI` statements into a
480 local Git structure. Furthermore, the location for the extracted
481 source is the default area within the ``devtool`` workspace. The
482 result is that the command sets up both the source code and an
483 append file within the workspace while the recipe remains in its
484 original location.
485
486 Additionally, if you have any non-patch local files (i.e. files
487 referred to with ``file://`` entries in :term:`SRC_URI` statement
488 excluding ``*.patch/`` or ``*.diff``), these files are copied to
489 an ``oe-local-files`` folder under the newly created source tree.
490 Copying the files here gives you a convenient area from which you
491 can modify the files. Any changes or additions you make to those
492 files are incorporated into the build the next time you build the
493 software just as are other changes you might have made to the
494 source.
495
496 - *Middle*: The middle scenario in the figure represents a situation
497 where the source code also does not exist locally. In this case,
498 the code is again upstream and needs to be extracted to some local
499 area as a Git repository. The recipe, in this scenario, is again
500 local and in its own layer outside the workspace.
501
502 The following command tells ``devtool`` the recipe with which to
503 work and, in this case, identifies a local area for the extracted
504 source files that exists outside of the default ``devtool``
505 workspace::
506
507 $ devtool modify recipe srctree
508
509 .. note::
510
511 You cannot provide a URL for ``srctree`` using the ``devtool`` command.
512
513 As with all extractions, the command uses the recipe's :term:`SRC_URI`
514 statements to locate the source files and any associated patch
515 files. Non-patch files are copied to an ``oe-local-files`` folder
516 under the newly created source tree.
517
518 Once the files are located, the command by default extracts them
519 into ``srctree``.
520
521 Within workspace, ``devtool`` creates an append file for the
522 recipe. The recipe remains in its original location but the source
523 files are extracted to the location you provide with ``srctree``.
524
525 - *Right*: The right scenario in the figure represents a situation
526 where the source tree (``srctree``) already exists locally as a
527 previously extracted Git structure outside of the ``devtool``
528 workspace. In this example, the recipe also exists elsewhere
529 locally in its own layer.
530
531 The following command tells ``devtool`` the recipe with which to
532 work, uses the "-n" option to indicate source does not need to be
533 extracted, and uses ``srctree`` to point to the previously extracted
534 source files::
535
536 $ devtool modify -n recipe srctree
537
538 If an ``oe-local-files`` subdirectory happens to exist and it
539 contains non-patch files, the files are used. However, if the
540 subdirectory does not exist and you run the ``devtool finish``
541 command, any non-patch files that might exist next to the recipe
542 are removed because it appears to ``devtool`` that you have
543 deleted those files.
544
545 Once the ``devtool modify`` command finishes, it creates only an
546 append file for the recipe in the ``devtool`` workspace. The
547 recipe and the source code remain in their original locations.
548
549#. *Edit the Source*: Once you have used the ``devtool modify`` command,
550 you are free to make changes to the source files. You can use any
551 editor you like to make and save your source code modifications.
552
553#. *Build the Recipe or Rebuild the Image*: The next step you take
554 depends on what you are going to do with the new code.
555
556 If you need to eventually move the build output to the target
557 hardware, use the following ``devtool`` command::
558
559 $ devtool build recipe
560
561 On the other hand, if you want an image to contain the recipe's
562 packages from the workspace for immediate deployment onto a device
563 (e.g. for testing purposes), you can use the ``devtool build-image``
564 command::
565
566 $ devtool build-image image
567
568#. *Deploy the Build Output*: When you use the ``devtool build`` command
569 to build out your recipe, you probably want to see if the resulting
570 build output works as expected on target hardware.
571
572 .. note::
573
574 This step assumes you have a previously built image that is
575 already either running in QEMU or running on actual hardware.
576 Also, it is assumed that for deployment of the image to the
577 target, SSH is installed in the image and if the image is running
578 on real hardware that you have network access to and from your
579 development machine.
580
581 You can deploy your build output to that target hardware by using the
582 ``devtool deploy-target`` command::
583
584 $ devtool deploy-target recipe target
585
586 The target is a live target machine running as an SSH server.
587
588 You can, of course, use other methods to deploy the image you built
589 using the ``devtool build-image`` command to actual hardware.
590 ``devtool`` does not provide a specific command to deploy the image
591 to actual hardware.
592
593#. *Finish Your Work With the Recipe*: The ``devtool finish`` command
594 creates any patches corresponding to commits in the local Git
595 repository, updates the recipe to point to them (or creates a
596 ``.bbappend`` file to do so, depending on the specified destination
597 layer), and then resets the recipe so that the recipe is built
598 normally rather than from the workspace::
599
600 $ devtool finish recipe layer
601
602 .. note::
603
604 Any changes you want to turn into patches must be staged and
605 committed within the local Git repository before you use the
606 ``devtool finish`` command.
607
608 Because there is no need to move the recipe, ``devtool finish``
609 either updates the original recipe in the original layer or the
610 command creates a ``.bbappend`` file in a different layer as provided
611 by layer. Any work you did in the ``oe-local-files`` directory is
612 preserved in the original files next to the recipe during the
613 ``devtool finish`` command.
614
615 As a final process of the ``devtool finish`` command, the state of
616 the standard layers and the upstream source is restored so that you
617 can build the recipe from those areas rather than from the workspace.
618
619 .. note::
620
621 You can use the ``devtool reset`` command to put things back should you
622 decide you do not want to proceed with your work. If you do use this
623 command, realize that the source tree is preserved.
624
625``devtool ide-sdk`` configures IDEs and bootstraps SDKs
626-------------------------------------------------------
627
628The ``devtool ide-sdk`` command can provide an IDE configuration for IDEs when
629working on the source code of one or more recipes.
630Depending on the programming language, and the build system used by the recipe,
631the tools required for cross-development and remote debugging are different.
632For example:
633
634- A C/C++ project usually uses CMake or Meson.
635
636- A Python project uses setuptools or one of its successors.
637
638- A Rust project uses Cargo.
639
640Also, the IDE plugins needed for the integration of a build system with the
641IDE and the corresponding settings are usually specific to these build-systems.
642To hide all these details from the user, ``devtool ide-sdk`` does two things:
643
644- It generates any kind of SDK needed for cross-development and remote
645 debugging of the specified recipes.
646
647- It generates the configuration for the IDE (and the IDE plugins) for using
648 the cross-toolchain and remote debugging tools provided by the SDK directly
649 from the IDE.
650
651For supported build systems the configurations generated by ``devtool ide-sdk``
652combine the advantages of the ``devtool modify`` based workflow
653(see :ref:`using_devtool`) with the advantages of the simple Environment Setup
654script based workflow (see :ref:`running_the_ext_sdk_env`) provided by Yocto's
655SDK or eSDK:
656
657- The source code of the recipe is in the workspace created by
658 ``devtool modify`` or ``devtool add``.
659 Using ``devtool build``, ``devtool build-image``,
660 ``devtool deploy-target`` or ``bitbake`` is possible.
661 Also ``devtool ide-sdk`` can be used to update the SDK and the IDE
662 configuration at any time.
663
664- ``devtool ide-sdk`` aims to support multiple programming languages and
665 multiple IDEs natively. "Natively" means that the IDE is configured to call
666 the build tool (e.g. ``cmake`` or ``meson``) directly. This has several
667 advantages.
668 First of all, it is usually much faster to call for example ``cmake`` than
669 ``devtool build``.
670 It also allows to benefit from the very good integration that IDEs like
671 VSCode offer for tools like CMake or GDB.
672
673 However, supporting many programming languages and multiple
674 IDEs is quite an elaborate and constantly evolving thing. Support for IDEs
675 is therefore implemented as plugins. Plugins can also be provided by
676 optional layers.
677
678So much about the introduction to the default mode of ``devtool sdk-ide`` which
679is called the "modified" mode because it uses the workspace created by
680``devtool modify`` and the per recipe :term:`Sysroots <Sysroot>` of BitBake.
681
682For some recipes and use cases, this default behavior of ``devtool ide-sdk``
683with full ``devtool`` and ``bitbake`` integration might not be suitable.
684To offer full feature parity with the SDK and the eSDK, ``devtool ide-sdk`` has
685a second mode called "shared" mode.
686If ``devtool ide-sdk`` is called with the ``--mode=shared`` option, it
687bootstraps an SDK directly from the BitBake environment, which offers the same
688Environment Setup script as described in :ref:`running_the_ext_sdk_env`.
689In addition to the (e)SDK installer-based setup, the IDE gets configured
690to use the shared :term:`Sysroots <Sysroot>` and the tools from the SDK.
691``devtool ide-sdk --mode=shared`` is basically a wrapper for the setup of the
692extensible SDK as described in :ref:`setting_up_ext_sdk_in_build`.
693
694The use of ``devtool ide-sdk`` is an alternative to using one of the SDK
695installers.
696``devtool ide-sdk`` allows the creation of SDKs that offer all the
697functionality of the SDK and the eSDK installers. Compared to the installers,
698however, the SDK created with ``devtool ide-sdk`` is much more flexible.
699For example, it is very easy to change the :term:`MACHINE` in the
700``local.conf`` file, update the layer meta data and then regenerate the SDK.
701
702Let's take a look at an example of how to use ``devtool ide-sdk`` in each of
703the two modes:
704
705#. *Modified mode*:
706
707 In order to use the ``devtool ide-sdk``, a few settings are needed. As a
708 starting example, the following lines of code can be added to the
709 ``local.conf`` file::
710
711 # Build the companion debug file system
712 IMAGE_GEN_DEBUGFS = "1"
713 # Optimize build time: with devtool ide-sdk the dbg tar is not needed
714 IMAGE_FSTYPES_DEBUGFS = ""
715 # Without copying the binaries into roofs-dbg, GDB does not find all source files.
716 IMAGE_CLASSES += "image-combined-dbg"
717
718 # SSH is mandatory, no password simplifies the usage
719 EXTRA_IMAGE_FEATURES += "\
720 ssh-server-openssh \
721 allow-empty-password \
722 allow-root-login \
723 empty-root-password \
724 "
725
726 # Remote debugging needs gdbserver on the target device
727 IMAGE_INSTALL:append = " gdbserver"
728
729 # Add the recipes which should be modified to the image
730 # Otherwise some dependencies might be missing.
731 IMAGE_INSTALL:append = " my-recipe"
732
733 Assuming the BitBake environment is set up correctly and a workspace has
734 been created for the recipe using ``devtool modify my-recipe`` or probably
735 even better by using ``devtool modify my-recipe --debug-build``, the
736 following command can create the SDK and the configuration for VSCode in
737 the recipe workspace::
738
739 $ devtool ide-sdk my-recipe core-image-minimal --target root@192.168.7.2
740
741 The command requires an image recipe (``core-image-minimal`` for this
742 example) that is used to create the SDK.
743 This firmware image should also be installed on the target device.
744 It is possible to pass multiple package recipes::
745
746 $ devtool ide-sdk my-recipe-1 my-recipe-2 core-image-minimal --target root@192.168.7.2
747
748 ``devtool ide-sdk`` tries to create an IDE configuration for all package
749 recipes.
750
751 What this command does exactly depends on the recipe, more precisely on the
752 build tool used by the recipe. The basic idea is to configure the IDE so
753 that it calls the build tool exactly as ``bitbake`` does.
754
755 For example, a CMake preset is created for a recipe that inherits
756 :ref:`ref-classes-cmake`. In the case of VSCode, CMake presets are supported
757 by the CMake Tools plugin. This is an example of how the build configuration
758 used by ``bitbake`` is exported to an IDE configuration that gives exactly
759 the same build results.
760
761 Support for remote debugging with seamless integration into the IDE is
762 important for a cross-SDK. ``devtool ide-sdk`` automatically generates the
763 necessary helper scripts for deploying the compiled artifacts to the target
764 device as well as the necessary configuration for the debugger and the IDE.
765
766 .. note::
767
768 To ensure that the debug symbols on the build machine match the binaries
769 running on the target device, it is essential that the image built by
770 ``devtool ide-sdk`` is running on the target device.
771
772 The default IDE is VSCode. Some hints about using VSCode:
773
774 - VSCode can be used to work on the BitBake recipes or the application
775 source code.
776 Usually there is one instance of VSCode running in the folder where the
777 BitBake recipes are. This instance has the
778 `Yocto Project BitBake plugin <https://marketplace.visualstudio.com/items?itemName=yocto-project.yocto-bitbake>`_
779 running.
780
781 .. warning::
782
783 Some VSCode plugins (Python, BitBake and others) need a reasonable
784 configuration to work as expected. Otherwise, some plugins try to
785 index the build directory of BitBake, which keeps your system quite
786 busy until an out of memory exception stops this nonsense.
787 Other plugins, such as the BitBake plugin, do not behave as expected.
788
789 To work around such issues, the ``oe-init-build-env`` script creates
790 an initial ``.vscode/settings.json`` file if ``code`` can be found
791 and the ``.vscode`` folder does not yet exist.
792 It is best to run ``oe-init-build-env`` once before starting VSCode.
793 An alternative approach is to use a build folder outside the layers,
794 e.g. ``oe-init-build-env ../build``.
795
796 The BitBake plugin also offers to create devtool workspaces and run
797 ``devtool ide-sdk`` with a few mouse clicks.
798 Of course, issuing commands in the terminal works as well.
799
800 - To work on the source code of a recipe another instance of VSCode is
801 started in the recipe's workspace. Example::
802
803 code build/workspace/sources/my-recipe
804
805 This instance of VSCode uses plugins that are useful for the development
806 of the application. ``devtool ide-sdk`` generates the necessary
807 ``extensions.json``, ``settings.json``, ``tasks.json``and ``launch.json``
808 configuration files for all the involved plugins.
809
810 When the source code folder present in the workspace folder is opened in
811 VSCode for the first time, a pop-up message recommends installing the
812 required plugins.
813 After accepting the installation of the plugins, working with the source
814 code or some debugging tasks should work as usual with VSCode.
815
816 Starting the VSCode instances in the recipe workspace folders can also be
817 done by a mouse click on the recipe workspaces in the first VSCode
818 instance.
819
820 - To work with CMake press ``Ctrl + Shift + p``, type ``cmake``. This will
821 show some possible commands like selecting a CMake preset, compiling or
822 running CTest.
823
824 For recipes inheriting :ref:`ref-classes-cmake-qemu` rather than
825 :ref:`ref-classes-cmake`, executing cross-compiled unit tests on the host
826 can be supported transparently with QEMU user-mode.
827
828 - To work with Meson press ``Ctrl + Shift + p``, type ``meson``. This will
829 show some possible commands like compiling or executing the unit tests.
830
831 A note on running cross-compiled unit tests on the host: Meson enables
832 support for QEMU user mode by default. It is expected that the execution
833 of the unit tests from the IDE will work without any additional steps,
834 given that the code is suitable for the execution on the host machine.
835
836 - For the deployment to the target device, just press ``Ctrl + Shift + p``,
837 type ``task``. Select ``install && deploy-target``.
838
839 - For remote debugging, switch to the debugging view by pressing the "play"
840 button with the ``bug icon`` on the left side. This will provide a green
841 play button with a drop-down list where a debug configuration can be
842 selected. After selecting one of the generated configurations, press the
843 "play" button.
844
845 Starting a remote debugging session automatically initiates the
846 deployment to the target device. If this is not desired, the
847 ``"dependsOn": ["install && deploy-target...]`` parameter of the tasks
848 with ``"label": "gdbserver start...`` can be removed from the
849 ``tasks.json`` file.
850
851 VSCode supports GDB with many different setups and configurations for
852 many different use cases. However, most of these setups have some
853 limitations when it comes to cross-development, support only a few target
854 architectures or require a high performance target device. Therefore
855 ``devtool ide-sdk`` supports the classic, generic setup with GDB on the
856 development host and gdbserver on the target device.
857
858 Roughly summarized, this means:
859
860 - The binaries are copied via SSH to the remote target device by a
861 script referred by ``tasks.json``.
862
863 - gdbserver is started on the remote target device via SSH by a script
864 referred by ``tasks.json``.
865
866 Changing the parameters that are passed to the debugging executable
867 requires modifying the generated script. The script is located at
868 ``oe-scripts/gdbserver_*``. Defining the parameters in the ``args``
869 field in the ``launch.json`` file does not work.
870
871 - VSCode connects to gdbserver as documented in
872 `Remote debugging or debugging with a local debugger server
873 <https://code.visualstudio.com/docs/cpp/launch-json-reference#_remote-debugging-or-debugging-with-a-local-debugger-server>`__.
874
875 Additionally ``--ide=none`` is supported. With the ``none`` IDE parameter,
876 some generic configuration files like ``gdbinit`` files and some helper
877 scripts starting gdbserver remotely on the target device as well as the GDB
878 client on the host are generated.
879
880 Here is a usage example for the ``cmake-example`` recipe from the
881 ``meta-selftest`` layer which inherits :ref:`ref-classes-cmake-qemu`:
882
883 .. code-block:: sh
884
885 # Create the SDK
886 devtool modify cmake-example --debug-build
887 devtool ide-sdk cmake-example core-image-minimal -c --ide=none
888
889 # Install the firmware on a target device or start QEMU
890 runqemu
891
892 # From exploring the workspace of cmake-example
893 cd build/workspace/sources/cmake-example
894
895 # Find cmake-native and save the path into a variable
896 # Note: using just cmake instead of $CMAKE_NATIVE would work in many cases
897 CMAKE_NATIVE="$(jq -r '.configurePresets[0] | "\(.cmakeExecutable)"' CMakeUserPresets.json)"
898
899 # List available CMake presets
900 "$CMAKE_NATIVE" --list-presets
901 Available configure presets:
902
903 "cmake-example-cortexa57" - cmake-example: cortexa57
904
905 # Re-compile the already compiled sources
906 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57
907 ninja: no work to do.
908 # Do a clean re-build
909 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target clean
910 [1/1] Cleaning all built files...
911 Cleaning... 8 files.
912 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target all
913 [7/7] Linking CXX executable cmake-example
914
915 # Run the cross-compiled unit tests with QEMU user-mode
916 "$CMAKE_NATIVE" --build --preset cmake-example-cortexa57 --target test
917 [0/1] Running tests...
918 Test project .../build/tmp/work/cortexa57-poky-linux/cmake-example/1.0/cmake-example-1.0
919 Start 1: test-cmake-example
920 1/1 Test #1: test-cmake-example ............... Passed 0.03 sec
921
922 100% tests passed, 0 tests failed out of 1
923
924 Total Test time (real) = 0.03 sec
925
926 # Using CTest directly is possible as well
927 CTEST_NATIVE="$(dirname "$CMAKE_NATIVE")/ctest"
928
929 # List available CMake presets
930 "$CTEST_NATIVE" --list-presets
931 Available test presets:
932
933 "cmake-example-cortexa57" - cmake-example: cortexa57
934
935 # Run the cross-compiled unit tests with QEMU user-mode
936 "$CTEST_NATIVE" --preset "cmake-example-cortexa57"
937 Test project ...build/tmp/work/cortexa57-poky-linux/cmake-example/1.0/cmake-example-1.0
938 Start 1: test-cmake-example
939 1/1 Test #1: test-cmake-example ............... Passed 0.03 sec
940
941 100% tests passed, 0 tests failed out of 1
942
943 Total Test time (real) = 0.03 sec
944
945 # Deploying the new build to the target device (default is QEUM at 192.168.7.2)
946 oe-scripts/install_and_deploy_cmake-example-cortexa57
947
948 # Start a remote debugging session with gdbserver on the target and GDB on the host
949 oe-scripts/gdbserver_1234_usr-bin-cmake-example_m
950 oe-scripts/gdb_1234_usr-bin-cmake-example
951 break main
952 run
953 step
954 stepi
955 continue
956 quit
957
958 # Stop gdbserver on the target device
959 oe-scripts/gdbserver_1234_usr-bin-cmake-example_m stop
960
961#. *Shared sysroots mode*
962
963 Creating an SDK with shared :term:`Sysroots <Sysroot>` that contains all the
964 dependencies needed to work with ``my-recipe`` is possible with the following
965 example command::
966
967 $ devtool ide-sdk --mode=shared my-recipe
968
969 For VSCode the cross-toolchain is exposed as a CMake kit. CMake kits are
970 defined in ``~/.local/share/CMakeTools/cmake-tools-kits.json``.
971 The following example shows how the cross-toolchain can be selected in
972 VSCode. First of all we need a folder containing a CMake project.
973 For this example, let's create a CMake project and start VSCode::
974
975 mkdir kit-test
976 echo "project(foo VERSION 1.0)" > kit-test/CMakeLists.txt
977 code kit-test
978
979 If there is a CMake project in the workspace, cross-compilation is
980 supported:
981
982 - Press ``Ctrl + Shift + P``, type ``CMake: Scan for Kits``
983 - Press ``Ctrl + Shift + P``, type ``CMake: Select a Kit``
984
985 Finally most of the features provided by CMake and the IDE should be
986 available.
987
988 Other IDEs than VSCode are supported as well. However,
989 ``devtool ide-sdk --mode=shared --ide=none my-recipe`` is currently
990 just a simple wrapper for the setup of the extensible SDK, as described in
991 :ref:`setting_up_ext_sdk_in_build`.
992
993Use ``devtool upgrade`` to Create a Version of the Recipe that Supports a Newer Version of the Software
994-------------------------------------------------------------------------------------------------------
995
996The ``devtool upgrade`` command upgrades an existing recipe to that of a
997more up-to-date version found upstream. Throughout the life of software,
998recipes continually undergo version upgrades by their upstream
999publishers. You can use the ``devtool upgrade`` workflow to make sure
1000your recipes you are using for builds are up-to-date with their upstream
1001counterparts.
1002
1003.. note::
1004
1005 Several methods exist by which you can upgrade recipes ---
1006 ``devtool upgrade`` happens to be one. You can read about all the methods by
1007 which you can upgrade recipes in the
1008 :ref:`dev-manual/upgrading-recipes:upgrading recipes` section of the Yocto
1009 Project Development Tasks Manual.
1010
1011The ``devtool upgrade`` command is flexible enough to allow you to specify
1012source code revision and versioning schemes, extract code into or out of the
1013``devtool`` :ref:`devtool-the-workspace-layer-structure`, and work with any
1014source file forms that the
1015:ref:`bitbake-user-manual/bitbake-user-manual-fetching:fetchers` support.
1016
1017The following diagram shows the common development flow used with the
1018``devtool upgrade`` command:
1019
1020.. image:: figures/sdk-devtool-upgrade-flow.png
1021 :width: 100%
1022
1023#. *Initiate the Upgrade*: The top part of the flow shows the typical
1024 scenario by which you use the ``devtool upgrade`` command. The
1025 following conditions exist:
1026
1027 - The recipe exists in a local layer external to the ``devtool``
1028 workspace.
1029
1030 - The source files for the new release exist in the same location
1031 pointed to by :term:`SRC_URI`
1032 in the recipe (e.g. a tarball with the new version number in the
1033 name, or as a different revision in the upstream Git repository).
1034
1035 A common situation is where third-party software has undergone a
1036 revision so that it has been upgraded. The recipe you have access to
1037 is likely in your own layer. Thus, you need to upgrade the recipe to
1038 use the newer version of the software::
1039
1040 $ devtool upgrade -V version recipe
1041
1042 By default, the ``devtool upgrade`` command extracts source
1043 code into the ``sources`` directory in the
1044 :ref:`devtool-the-workspace-layer-structure`.
1045 If you want the code extracted to any other location, you need to
1046 provide the ``srctree`` positional argument with the command as follows::
1047
1048 $ devtool upgrade -V version recipe srctree
1049
1050 .. note::
1051
1052 In this example, the "-V" option specifies the new version. If you
1053 don't use "-V", the command upgrades the recipe to the latest
1054 version.
1055
1056 If the source files pointed to by the :term:`SRC_URI` statement in the
1057 recipe are in a Git repository, you must provide the "-S" option and
1058 specify a revision for the software.
1059
1060 Once ``devtool`` locates the recipe, it uses the :term:`SRC_URI` variable
1061 to locate the source code and any local patch files from other
1062 developers. The result is that the command sets up the source code,
1063 the new version of the recipe, and an append file all within the
1064 workspace.
1065
1066 Additionally, if you have any non-patch local files (i.e. files
1067 referred to with ``file://`` entries in :term:`SRC_URI` statement
1068 excluding ``*.patch/`` or ``*.diff``), these files are copied to an
1069 ``oe-local-files`` folder under the newly created source tree.
1070 Copying the files here gives you a convenient area from which you can
1071 modify the files. Any changes or additions you make to those files
1072 are incorporated into the build the next time you build the software
1073 just as are other changes you might have made to the source.
1074
1075#. *Resolve any Conflicts created by the Upgrade*: Conflicts could happen
1076 after upgrading the software to a new version. Conflicts occur
1077 if your recipe specifies some patch files in :term:`SRC_URI` that
1078 conflict with changes made in the new version of the software. For
1079 such cases, you need to resolve the conflicts by editing the source
1080 and following the normal ``git rebase`` conflict resolution process.
1081
1082 Before moving onto the next step, be sure to resolve any such
1083 conflicts created through use of a newer or different version of the
1084 software.
1085
1086#. *Build the Recipe or Rebuild the Image*: The next step you take
1087 depends on what you are going to do with the new code.
1088
1089 If you need to eventually move the build output to the target
1090 hardware, use the following ``devtool`` command::
1091
1092 $ devtool build recipe
1093
1094 On the other hand, if you want an image to contain the recipe's
1095 packages from the workspace for immediate deployment onto a device
1096 (e.g. for testing purposes), you can use the ``devtool build-image``
1097 command::
1098
1099 $ devtool build-image image
1100
1101#. *Deploy the Build Output*: When you use the ``devtool build`` command
1102 or ``bitbake`` to build your recipe, you probably want to see if the
1103 resulting build output works as expected on target hardware.
1104
1105 .. note::
1106
1107 This step assumes you have a previously built image that is
1108 already either running in QEMU or running on actual hardware.
1109 Also, it is assumed that for deployment of the image to the
1110 target, SSH is installed in the image and if the image is running
1111 on real hardware that you have network access to and from your
1112 development machine.
1113
1114 You can deploy your build output to that target hardware by using the
1115 ``devtool deploy-target`` command::
1116
1117 $ devtool deploy-target recipe target
1118
1119 The target is a live target machine running as an SSH server.
1120
1121 You can, of course, also deploy the image you build using the
1122 ``devtool build-image`` command to actual hardware. However,
1123 ``devtool`` does not provide a specific command that allows you to do
1124 this.
1125
1126#. *Finish Your Work With the Recipe*: The ``devtool finish`` command
1127 creates any patches corresponding to commits in the local Git
1128 repository, moves the new recipe to a more permanent layer, and then
1129 resets the recipe so that the recipe is built normally rather than
1130 from the workspace.
1131
1132 Any work you did in the ``oe-local-files`` directory is preserved in
1133 the original files next to the recipe during the ``devtool finish``
1134 command.
1135
1136 If you specify a destination layer that is the same as the original
1137 source, then the old version of the recipe and associated files are
1138 removed prior to adding the new version::
1139
1140 $ devtool finish recipe layer
1141
1142 .. note::
1143
1144 Any changes you want to turn into patches must be committed to the
1145 Git repository in the source tree.
1146
1147 As a final process of the ``devtool finish`` command, the state of
1148 the standard layers and the upstream source is restored so that you
1149 can build the recipe from those areas rather than the workspace.
1150
1151 .. note::
1152
1153 You can use the ``devtool reset`` command to put things back should you
1154 decide you do not want to proceed with your work. If you do use this
1155 command, realize that the source tree is preserved.
1156
1157A Closer Look at ``devtool add``
1158================================
1159
1160The ``devtool add`` command automatically creates a recipe based on the
1161source tree you provide with the command. Currently, the command has
1162support for the following:
1163
1164- Autotools (``autoconf`` and ``automake``)
1165
1166- CMake
1167
1168- Scons
1169
1170- ``qmake``
1171
1172- Plain ``Makefile``
1173
1174- Out-of-tree kernel module
1175
1176- Binary package (i.e. "-b" option)
1177
1178- Node.js module
1179
1180- Python modules that use ``setuptools`` or ``distutils``
1181
1182Apart from binary packages, the determination of how a source tree
1183should be treated is automatic based on the files present within that
1184source tree. For example, if a ``CMakeLists.txt`` file is found, then
1185the source tree is assumed to be using CMake and is treated accordingly.
1186
1187.. note::
1188
1189 In most cases, you need to edit the automatically generated recipe in
1190 order to make it build properly. Typically, you would go through
1191 several edit and build cycles until the recipe successfully builds.
1192 Once the recipe builds, you could use possible further iterations to
1193 test the recipe on the target device.
1194
1195The remainder of this section covers specifics regarding how parts of
1196the recipe are generated.
1197
1198Name and Version
1199----------------
1200
1201If you do not specify a name and version on the command line,
1202``devtool add`` uses various metadata within the source tree in an
1203attempt to determine the name and version of the software being built.
1204Based on what the tool determines, ``devtool`` sets the name of the
1205created recipe file accordingly.
1206
1207If ``devtool`` cannot determine the name and version, the command prints
1208an error. For such cases, you must re-run the command and provide the
1209name and version, just the name, or just the version as part of the
1210command line.
1211
1212Sometimes the name or version determined from the source tree might be
1213incorrect. For such a case, you must reset the recipe::
1214
1215 $ devtool reset -n recipename
1216
1217After running the ``devtool reset`` command, you need to
1218run ``devtool add`` again and provide the name or the version.
1219
1220Dependency Detection and Mapping
1221--------------------------------
1222
1223The ``devtool add`` command attempts to detect build-time dependencies and map
1224them to other recipes in the system. During this mapping, the command fills in
1225the names of those recipes as part of the :term:`DEPENDS` variable within the
1226recipe. If a dependency cannot be mapped, ``devtool`` places a comment
1227in the recipe indicating such. The inability to map a dependency can
1228result from naming not being recognized or because the dependency simply
1229is not available. For cases where the dependency is not available, you
1230must use the ``devtool add`` command to add an additional recipe that
1231satisfies the dependency. Once you add that recipe, you need to update
1232the :term:`DEPENDS` variable in the original recipe to include the new
1233recipe.
1234
1235If you need to add runtime dependencies, you can do so by adding the
1236following to your recipe::
1237
1238 RDEPENDS:${PN} += "dependency1 dependency2 ..."
1239
1240.. note::
1241
1242 The ``devtool add`` command often cannot distinguish between mandatory and
1243 optional dependencies. Consequently, some of the detected dependencies might
1244 in fact be optional. When in doubt, consult the documentation or the
1245 configure script for the software the recipe is building for further
1246 details. In some cases, you might find you can substitute the
1247 dependency with an option that disables the associated functionality
1248 passed to the configure script.
1249
1250License Detection
1251-----------------
1252
1253The ``devtool add`` command attempts to determine if the software you are
1254adding is able to be distributed under a common, open-source license. If
1255so, the command sets the :term:`LICENSE` value accordingly.
1256You should double-check the value added by the command against the
1257documentation or source files for the software you are building and, if
1258necessary, update that :term:`LICENSE` value.
1259
1260The ``devtool add`` command also sets the :term:`LIC_FILES_CHKSUM`
1261value to point to all files that appear to be license-related. Realize
1262that license statements often appear in comments at the top of source
1263files or within the documentation. In such cases, the command does not
1264recognize those license statements. Consequently, you might need to
1265amend the :term:`LIC_FILES_CHKSUM` variable to point to one or more of those
1266comments if present. Setting :term:`LIC_FILES_CHKSUM` is particularly
1267important for third-party software. The mechanism attempts to ensure
1268correct licensing should you upgrade the recipe to a newer upstream
1269version in future. Any change in licensing is detected and you receive
1270an error prompting you to check the license text again.
1271
1272If the ``devtool add`` command cannot determine licensing information,
1273``devtool`` sets the :term:`LICENSE` value to "CLOSED" and leaves the
1274:term:`LIC_FILES_CHKSUM` value unset. This behavior allows you to continue
1275with development even though the settings are unlikely to be correct in
1276all cases. You should check the documentation or source files for the
1277software you are building to determine the actual license.
1278
1279Adding Makefile-Only Software
1280-----------------------------
1281
1282The use of Make by itself is very common in both proprietary and
1283open-source software. Unfortunately, Makefiles are often not written
1284with cross-compilation in mind. Thus, ``devtool add`` often cannot do
1285very much to ensure that these Makefiles build correctly. It is very
1286common, for example, to explicitly call ``gcc`` instead of using the
1287:term:`CC` variable. Usually, in a
1288cross-compilation environment, ``gcc`` is the compiler for the build
1289host and the cross-compiler is named something similar to
1290``arm-poky-linux-gnueabi-gcc`` and might require arguments (e.g. to
1291point to the associated sysroot for the target machine).
1292
1293When writing a recipe for Makefile-only software, keep the following in
1294mind:
1295
1296- You probably need to patch the Makefile to use variables instead of
1297 hardcoding tools within the toolchain such as ``gcc`` and ``g++``.
1298
1299- The environment in which Make runs is set up with various standard
1300 variables for compilation (e.g. :term:`CC`, :term:`CXX`, and so forth) in a
1301 similar manner to the environment set up by the SDK's environment
1302 setup script. One easy way to see these variables is to run the
1303 ``devtool build`` command on the recipe and then look in
1304 ``oe-logs/run.do_compile``. Towards the top of this file, there is
1305 a list of environment variables that are set. You can take
1306 advantage of these variables within the Makefile.
1307
1308- If the Makefile sets a default for a variable using "=", that default
1309 overrides the value set in the environment, which is usually not
1310 desirable. For this case, you can either patch the Makefile so it
1311 sets the default using the "?=" operator, or you can alternatively
1312 force the value on the ``make`` command line. To force the value on
1313 the command line, add the variable setting to
1314 :term:`EXTRA_OEMAKE` or
1315 :term:`PACKAGECONFIG_CONFARGS`
1316 within the recipe. Here is an example using :term:`EXTRA_OEMAKE`::
1317
1318 EXTRA_OEMAKE += "'CC=${CC}' 'CXX=${CXX}'"
1319
1320 In the above example,
1321 single quotes are used around the variable settings as the values are
1322 likely to contain spaces because required default options are passed
1323 to the compiler.
1324
1325- Hardcoding paths inside Makefiles is often problematic in a
1326 cross-compilation environment. This is particularly true because
1327 those hardcoded paths often point to locations on the build host and
1328 thus will either be read-only or will introduce contamination into
1329 the cross-compilation because they are specific to the build host
1330 rather than the target. Patching the Makefile to use prefix variables
1331 or other path variables is usually the way to handle this situation.
1332
1333- Sometimes a Makefile runs target-specific commands such as
1334 ``ldconfig``. For such cases, you might be able to apply patches that
1335 remove these commands from the Makefile.
1336
1337Adding Native Tools
1338-------------------
1339
1340Often, you need to build additional tools that run on the :term:`Build Host`
1341as opposed to the target. You should indicate this requirement by using one of
1342the following methods when you run ``devtool add``:
1343
1344- Specify the name of the recipe such that it ends with "-native".
1345 Specifying the name like this produces a recipe that only builds for
1346 the build host.
1347
1348- Specify the "--also-native" option with the ``devtool add``
1349 command. Specifying this option creates a recipe file that still
1350 builds for the target but also creates a variant with a "-native"
1351 suffix that builds for the build host.
1352
1353.. note::
1354
1355 If you need to add a tool that is shipped as part of a source tree
1356 that builds code for the target, you can typically accomplish this by
1357 building the native and target parts separately rather than within
1358 the same compilation process. Realize though that with the
1359 "--also-native" option, you can add the tool using just one
1360 recipe file.
1361
1362Adding Node.js Modules
1363----------------------
1364
1365You can use the ``devtool add`` command two different ways to add
1366Node.js modules: through ``npm`` or from a repository or local source.
1367
1368Use the following form to add Node.js modules through ``npm``::
1369
1370 $ devtool add "npm://registry.npmjs.org;name=forever;version=0.15.1"
1371
1372The name and
1373version parameters are mandatory. Lockdown and shrinkwrap files are
1374generated and pointed to by the recipe in order to freeze the version
1375that is fetched for the dependencies according to the first time. This
1376also saves checksums that are verified on future fetches. Together,
1377these behaviors ensure the reproducibility and integrity of the build.
1378
1379.. note::
1380
1381 - You must use quotes around the URL. ``devtool add`` does not
1382 require the quotes, but the shell considers ";" as a splitter
1383 between multiple commands. Thus, without the quotes,
1384 ``devtool add`` does not receive the other parts, which results in
1385 several "command not found" errors.
1386
1387 - In order to support adding Node.js modules, a ``nodejs`` recipe
1388 must be part of your SDK.
1389
1390As mentioned earlier, you can also add Node.js modules directly from a
1391repository or local source tree. To add modules this way, use
1392``devtool add`` in the following form::
1393
1394 $ devtool add https://github.com/diversario/node-ssdp
1395
1396In this example, ``devtool`` fetches the specified Git repository, detects the
1397code as Node.js code, fetches dependencies using ``npm``, and sets
1398:term:`SRC_URI` accordingly.
1399
1400Working With Recipes
1401====================
1402
1403When building a recipe using the ``devtool build`` command, the typical
1404build progresses as follows:
1405
1406#. Fetch the source
1407
1408#. Unpack the source
1409
1410#. Configure the source
1411
1412#. Compile the source
1413
1414#. Install the build output
1415
1416#. Package the installed output
1417
1418For recipes in the workspace, fetching and unpacking is disabled as the
1419source tree has already been prepared and is persistent. Each of these
1420build steps is defined as a function (task), usually with a "do\_" prefix
1421(e.g. :ref:`ref-tasks-fetch`,
1422:ref:`ref-tasks-unpack`, and so
1423forth). These functions are typically shell scripts but can instead be
1424written in Python.
1425
1426If you look at the contents of a recipe, you will see that the recipe
1427does not include complete instructions for building the software.
1428Instead, common functionality is encapsulated in classes inherited with
1429the ``inherit`` directive. This technique leaves the recipe to describe
1430just the things that are specific to the software being built. There is
1431a :ref:`ref-classes-base` class that is implicitly inherited by all recipes
1432and provides the functionality that most recipes typically need.
1433
1434The remainder of this section presents information useful when working
1435with recipes.
1436
1437Finding Logs and Work Files
1438---------------------------
1439
1440After the first run of the ``devtool build`` command, recipes that were
1441previously created using the ``devtool add`` command or whose sources
1442were modified using the ``devtool modify`` command contain symbolic
1443links created within the source tree:
1444
1445- ``oe-logs``: This link points to the directory in which log files and
1446 run scripts for each build step are created.
1447
1448- ``oe-workdir``: This link points to the temporary work area for the
1449 recipe. The following locations under ``oe-workdir`` are particularly
1450 useful:
1451
1452 - ``image/``: Contains all of the files installed during the
1453 :ref:`ref-tasks-install` stage.
1454 Within a recipe, this directory is referred to by the expression
1455 ``${``\ :term:`D`\ ``}``.
1456
1457 - ``sysroot-destdir/``: Contains a subset of files installed within
1458 :ref:`ref-tasks-install` that have been put into the shared sysroot. For
1459 more information, see the
1460 ":ref:`dev-manual/new-recipe:sharing files between recipes`" section.
1461
1462 - ``packages-split/``: Contains subdirectories for each package
1463 produced by the recipe. For more information, see the
1464 ":ref:`sdk-manual/extensible:packaging`" section.
1465
1466You can use these links to get more information on what is happening at
1467each build step.
1468
1469Setting Configure Arguments
1470---------------------------
1471
1472If the software your recipe is building uses GNU autoconf, then a fixed
1473set of arguments is passed to it to enable cross-compilation plus any
1474extras specified by :term:`EXTRA_OECONF` or :term:`PACKAGECONFIG_CONFARGS`
1475set within the recipe. If you wish to pass additional options, add them
1476to :term:`EXTRA_OECONF` or :term:`PACKAGECONFIG_CONFARGS`. Other supported build
1477tools have similar variables (e.g. :term:`EXTRA_OECMAKE` for CMake,
1478:term:`EXTRA_OESCONS` for Scons, and so forth). If you need to pass anything on
1479the ``make`` command line, you can use :term:`EXTRA_OEMAKE` or the
1480:term:`PACKAGECONFIG_CONFARGS` variables to do so.
1481
1482You can use the ``devtool configure-help`` command to help you set the
1483arguments listed in the previous paragraph. The command determines the
1484exact options being passed, and shows them to you along with any custom
1485arguments specified through :term:`EXTRA_OECONF` or
1486:term:`PACKAGECONFIG_CONFARGS`. If applicable, the command also shows you
1487the output of the configure script's "--help" option as a
1488reference.
1489
1490Sharing Files Between Recipes
1491-----------------------------
1492
1493Recipes often need to use files provided by other recipes on the
1494:term:`Build Host`. For example,
1495an application linking to a common library needs access to the library
1496itself and its associated headers. The way this access is accomplished
1497within the extensible SDK is through the sysroot. There is one sysroot per
1498"machine" for which the SDK is being built. In practical terms, this
1499means there is a sysroot for the target machine, and a sysroot for
1500the build host.
1501
1502Recipes should never write files directly into the sysroot. Instead,
1503files should be installed into standard locations during the
1504:ref:`ref-tasks-install` task within the ``${``\ :term:`D`\ ``}`` directory. A
1505subset of these files automatically goes into the sysroot. The reason
1506for this limitation is that almost all files that go into the sysroot
1507are cataloged in manifests in order to ensure they can be removed later
1508when a recipe is modified or removed. Thus, the sysroot is able to
1509remain free from stale files.
1510
1511Packaging
1512---------
1513
1514Packaging is not always particularly relevant within the extensible SDK.
1515However, if you examine how build output gets into the final image on
1516the target device, it is important to understand packaging because the
1517contents of the image are expressed in terms of packages and not
1518recipes.
1519
1520During the :ref:`ref-tasks-package` task, files installed during the
1521:ref:`ref-tasks-install` task are split into one main package, which is almost
1522always named the same as the recipe, and into several other packages. This
1523separation exists because not all of those installed files are useful in every
1524image. For example, you probably do not need any of the documentation installed
1525in a production image. Consequently, for each recipe the documentation
1526files are separated into a ``-doc`` package. Recipes that package
1527software containing optional modules or plugins might undergo additional
1528package splitting as well.
1529
1530After building a recipe, you can see where files have gone by looking in
1531the ``oe-workdir/packages-split`` directory, which contains a
1532subdirectory for each package. Apart from some advanced cases, the
1533:term:`PACKAGES` and :term:`FILES` variables controls
1534splitting. The :term:`PACKAGES` variable lists all of the packages to be
1535produced, while the :term:`FILES` variable specifies which files to include
1536in each package by using an override to specify the package. For
1537example, ``FILES:${PN}`` specifies the files to go into the main package
1538(i.e. the main package has the same name as the recipe and
1539``${``\ :term:`PN`\ ``}`` evaluates to the
1540recipe name). The order of the :term:`PACKAGES` value is significant. For
1541each installed file, the first package whose :term:`FILES` value matches the
1542file is the package into which the file goes. Both the :term:`PACKAGES` and
1543:term:`FILES` variables have default values. Consequently, you might find
1544you do not even need to set these variables in your recipe unless the
1545software the recipe is building installs files into non-standard
1546locations.
1547
1548Restoring the Target Device to its Original State
1549=================================================
1550
1551If you use the ``devtool deploy-target`` command to write a recipe's
1552build output to the target, and you are working on an existing component
1553of the system, then you might find yourself in a situation where you
1554need to restore the original files that existed prior to running the
1555``devtool deploy-target`` command. Because the ``devtool deploy-target``
1556command backs up any files it overwrites, you can use the
1557``devtool undeploy-target`` command to restore those files and remove
1558any other files the recipe deployed. Consider the following example::
1559
1560 $ devtool undeploy-target lighttpd root@192.168.7.2
1561
1562If you have deployed
1563multiple applications, you can remove them all using the "-a" option
1564thus restoring the target device to its original state::
1565
1566 $ devtool undeploy-target -a root@192.168.7.2
1567
1568Information about files deployed to
1569the target as well as any backed up files are stored on the target
1570itself. This storage, of course, requires some additional space on the
1571target machine.
1572
1573.. note::
1574
1575 The ``devtool deploy-target`` and ``devtool undeploy-target`` commands do
1576 not currently interact with any package management system on the target
1577 device (e.g. RPM or OPKG). Consequently, you should not intermingle
1578 ``devtool deploy-target`` and package manager operations on the target
1579 device. Doing so could result in a conflicting set of files.
1580 263
1581Installing Additional Items Into the Extensible SDK 264Installing Additional Items Into the Extensible SDK
1582=================================================== 265===================================================
diff --git a/documentation/what-i-wish-id-known.rst b/documentation/what-i-wish-id-known.rst
index 5bc55804f6..836097910d 100644
--- a/documentation/what-i-wish-id-known.rst
+++ b/documentation/what-i-wish-id-known.rst
@@ -136,7 +136,7 @@ contact us with other suggestions.
136 valuable links: :ref:`dev-manual/development-shell:Using a Development 136 valuable links: :ref:`dev-manual/development-shell:Using a Development
137 Shell` for information on how to build and run a specific task using 137 Shell` for information on how to build and run a specific task using
138 devshell. Also, the :ref:`SDK manual shows how to build out a specific recipe 138 devshell. Also, the :ref:`SDK manual shows how to build out a specific recipe
139 <sdk-manual/extensible:use \`\`devtool modify\`\` to modify the source of an existing component>`. 139 <dev-manual/devtool:use \`\`devtool modify\`\` to modify the source of an existing component>`.
140 140
141#. **An ambiguous definition: Package vs Recipe:** 141#. **An ambiguous definition: Package vs Recipe:**
142 A recipe contains instructions the build system uses to create 142 A recipe contains instructions the build system uses to create