manuals: split dev-manual/common-tasks.rst

A 500 KB source file is always harder to manage,
and can have section title conflicts.

So, the "Common Tasks" document is gone and all
its constituents are moved up one level.
You now have 40 chapters in the Development Tasks Manual.

(From yocto-docs rev: 8a45bc469411410020b8e688c137395fcaf3761b)

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

1 files changed, 400 insertions, 0 deletions

diff --git a/documentation/dev-manual/upgrading-recipes.rst b/documentation/dev-manual/upgrading-recipes.rst
new file mode 100644
index 0000000000..a0856f43da
--- /dev/null
+++ b/documentation/dev-manual/upgrading-recipes.rst
@@ -0,0 +1,400 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+Upgrading Recipes
+*****************
+
+Over time, upstream developers publish new versions for software built
+by layer recipes. It is recommended to keep recipes up-to-date with
+upstream version releases.
+
+While there are several methods to upgrade a recipe, you might
+consider checking on the upgrade status of a recipe first. You can do so
+using the ``devtool check-upgrade-status`` command. See the
+":ref:`devtool-checking-on-the-upgrade-status-of-a-recipe`"
+section in the Yocto Project Reference Manual for more information.
+
+The remainder of this section describes three ways you can upgrade a
+recipe. You can use the Automated Upgrade Helper (AUH) to set up
+automatic version upgrades. Alternatively, you can use
+``devtool upgrade`` to set up semi-automatic version upgrades. Finally,
+you can manually upgrade a recipe by editing the recipe itself.
+
+Using the Auto Upgrade Helper (AUH)
+===================================
+
+The AUH utility works in conjunction with the OpenEmbedded build system
+in order to automatically generate upgrades for recipes based on new
+versions being published upstream. Use AUH when you want to create a
+service that performs the upgrades automatically and optionally sends
+you an email with the results.
+
+AUH allows you to update several recipes with a single use. You can also
+optionally perform build and integration tests using images with the
+results saved to your hard drive and emails of results optionally sent
+to recipe maintainers. Finally, AUH creates Git commits with appropriate
+commit messages in the layer's tree for the changes made to recipes.
+
+.. note::
+
+   In some conditions, you should not use AUH to upgrade recipes
+   and should instead use either ``devtool upgrade`` or upgrade your
+   recipes manually:
+
+   -  When AUH cannot complete the upgrade sequence. This situation
+      usually results because custom patches carried by the recipe
+      cannot be automatically rebased to the new version. In this case,
+      ``devtool upgrade`` allows you to manually resolve conflicts.
+
+   -  When for any reason you want fuller control over the upgrade
+      process. For example, when you want special arrangements for
+      testing.
+
+The following steps describe how to set up the AUH utility:
+
+1. *Be Sure the Development Host is Set Up:* You need to be sure that
+   your development host is set up to use the Yocto Project. For
+   information on how to set up your host, see the
+   ":ref:`dev-manual/start:Preparing the Build Host`" section.
+
+2. *Make Sure Git is Configured:* The AUH utility requires Git to be
+   configured because AUH uses Git to save upgrades. Thus, you must have
+   Git user and email configured. The following command shows your
+   configurations::
+
+      $ git config --list
+
+   If you do not have the user and
+   email configured, you can use the following commands to do so::
+
+      $ git config --global user.name some_name
+      $ git config --global user.email username@domain.com
+
+3. *Clone the AUH Repository:* To use AUH, you must clone the repository
+   onto your development host. The following command uses Git to create
+   a local copy of the repository on your system::
+
+      $ git clone  git://git.yoctoproject.org/auto-upgrade-helper
+      Cloning into 'auto-upgrade-helper'... remote: Counting objects: 768, done.
+      remote: Compressing objects: 100% (300/300), done.
+      remote: Total 768 (delta 499), reused 703 (delta 434)
+      Receiving objects: 100% (768/768), 191.47 KiB | 98.00 KiB/s, done.
+      Resolving deltas: 100% (499/499), done.
+      Checking connectivity... done.
+
+   AUH is not part of the :term:`OpenEmbedded-Core (OE-Core)` or
+   :term:`Poky` repositories.
+
+4. *Create a Dedicated Build Directory:* Run the :ref:`structure-core-script`
+   script to create a fresh :term:`Build Directory` that you use exclusively
+   for running the AUH utility::
+
+      $ cd poky
+      $ source oe-init-build-env your_AUH_build_directory
+
+   Re-using an existing :term:`Build Directory` and its configurations is not
+   recommended as existing settings could cause AUH to fail or behave
+   undesirably.
+
+5. *Make Configurations in Your Local Configuration File:* Several
+   settings are needed in the ``local.conf`` file in the build
+   directory you just created for AUH. Make these following
+   configurations:
+
+   -  If you want to enable :ref:`Build
+      History <dev-manual/build-quality:maintaining build output quality>`,
+      which is optional, you need the following lines in the
+      ``conf/local.conf`` file::
+
+         INHERIT =+ "buildhistory"
+         BUILDHISTORY_COMMIT = "1"
+
+      With this configuration and a successful
+      upgrade, a build history "diff" file appears in the
+      ``upgrade-helper/work/recipe/buildhistory-diff.txt`` file found in
+      your :term:`Build Directory`.
+
+   -  If you want to enable testing through the
+      :ref:`testimage <ref-classes-testimage>`
+      class, which is optional, you need to have the following set in
+      your ``conf/local.conf`` file::
+
+         INHERIT += "testimage"
+
+      .. note::
+
+         If your distro does not enable by default ptest, which Poky
+         does, you need the following in your ``local.conf`` file::
+
+                 DISTRO_FEATURES:append = " ptest"
+
+
+6. *Optionally Start a vncserver:* If you are running in a server
+   without an X11 session, you need to start a vncserver::
+
+      $ vncserver :1
+      $ export DISPLAY=:1
+
+7. *Create and Edit an AUH Configuration File:* You need to have the
+   ``upgrade-helper/upgrade-helper.conf`` configuration file in your
+   :term:`Build Directory`. You can find a sample configuration file in the
+   :yocto_git:`AUH source repository </auto-upgrade-helper/tree/>`.
+
+   Read through the sample file and make configurations as needed. For
+   example, if you enabled build history in your ``local.conf`` as
+   described earlier, you must enable it in ``upgrade-helper.conf``.
+
+   Also, if you are using the default ``maintainers.inc`` file supplied
+   with Poky and located in ``meta-yocto`` and you do not set a
+   "maintainers_whitelist" or "global_maintainer_override" in the
+   ``upgrade-helper.conf`` configuration, and you specify "-e all" on
+   the AUH command-line, the utility automatically sends out emails to
+   all the default maintainers. Please avoid this.
+
+This next set of examples describes how to use the AUH:
+
+-  *Upgrading a Specific Recipe:* To upgrade a specific recipe, use the
+   following form::
+
+      $ upgrade-helper.py recipe_name
+
+   For example, this command upgrades the ``xmodmap`` recipe::
+
+      $ upgrade-helper.py xmodmap
+
+-  *Upgrading a Specific Recipe to a Particular Version:* To upgrade a
+   specific recipe to a particular version, use the following form::
+
+      $ upgrade-helper.py recipe_name -t version
+
+   For example, this command upgrades the ``xmodmap`` recipe to version 1.2.3::
+
+      $ upgrade-helper.py xmodmap -t 1.2.3
+
+-  *Upgrading all Recipes to the Latest Versions and Suppressing Email
+   Notifications:* To upgrade all recipes to their most recent versions
+   and suppress the email notifications, use the following command::
+
+      $ upgrade-helper.py all
+
+-  *Upgrading all Recipes to the Latest Versions and Send Email
+   Notifications:* To upgrade all recipes to their most recent versions
+   and send email messages to maintainers for each attempted recipe as
+   well as a status email, use the following command::
+
+      $ upgrade-helper.py -e all
+
+Once you have run the AUH utility, you can find the results in the AUH
+:term:`Build Directory`::
+
+   ${BUILDDIR}/upgrade-helper/timestamp
+
+The AUH utility
+also creates recipe update commits from successful upgrade attempts in
+the layer tree.
+
+You can easily set up to run the AUH utility on a regular basis by using
+a cron job. See the
+:yocto_git:`weeklyjob.sh </auto-upgrade-helper/tree/weeklyjob.sh>`
+file distributed with the utility for an example.
+
+Using ``devtool upgrade``
+=========================
+
+As mentioned earlier, an alternative method for upgrading recipes to
+newer versions is to use
+:doc:`devtool upgrade </ref-manual/devtool-reference>`.
+You can read about ``devtool upgrade`` in general in the
+":ref:`sdk-manual/extensible:use \`\`devtool upgrade\`\` to create a version of the recipe that supports a newer version of the software`"
+section in the Yocto Project Application Development and the Extensible
+Software Development Kit (eSDK) Manual.
+
+To see all the command-line options available with ``devtool upgrade``,
+use the following help command::
+
+   $ devtool upgrade -h
+
+If you want to find out what version a recipe is currently at upstream
+without any attempt to upgrade your local version of the recipe, you can
+use the following command::
+
+   $ devtool latest-version recipe_name
+
+As mentioned in the previous section describing AUH, ``devtool upgrade``
+works in a less-automated manner than AUH. Specifically,
+``devtool upgrade`` only works on a single recipe that you name on the
+command line, cannot perform build and integration testing using images,
+and does not automatically generate commits for changes in the source
+tree. Despite all these "limitations", ``devtool upgrade`` updates the
+recipe file to the new upstream version and attempts to rebase custom
+patches contained by the recipe as needed.
+
+.. note::
+
+   AUH uses much of ``devtool upgrade`` behind the scenes making AUH somewhat
+   of a "wrapper" application for ``devtool upgrade``.
+
+A typical scenario involves having used Git to clone an upstream
+repository that you use during build operations. Because you have built the
+recipe in the past, the layer is likely added to your
+configuration already. If for some reason, the layer is not added, you
+could add it easily using the
+":ref:`bitbake-layers <bsp-guide/bsp:creating a new bsp layer using the \`\`bitbake-layers\`\` script>`"
+script. For example, suppose you use the ``nano.bb`` recipe from the
+``meta-oe`` layer in the ``meta-openembedded`` repository. For this
+example, assume that the layer has been cloned into following area::
+
+   /home/scottrif/meta-openembedded
+
+The following command from your :term:`Build Directory` adds the layer to
+your build configuration (i.e. ``${BUILDDIR}/conf/bblayers.conf``)::
+
+   $ bitbake-layers add-layer /home/scottrif/meta-openembedded/meta-oe
+   NOTE: Starting bitbake server...
+   Parsing recipes: 100% |##########################################| Time: 0:00:55
+   Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
+   Removing 12 recipes from the x86_64 sysroot: 100% |##############| Time: 0:00:00
+   Removing 1 recipes from the x86_64_i586 sysroot: 100% |##########| Time: 0:00:00
+   Removing 5 recipes from the i586 sysroot: 100% |#################| Time: 0:00:00
+   Removing 5 recipes from the qemux86 sysroot: 100% |##############| Time: 0:00:00
+
+For this example, assume that the ``nano.bb`` recipe that
+is upstream has a 2.9.3 version number. However, the version in the
+local repository is 2.7.4. The following command from your build
+directory automatically upgrades the recipe for you:
+
+.. note::
+
+   Using the ``-V`` option is not necessary. Omitting the version number causes
+   ``devtool upgrade`` to upgrade the recipe to the most recent version.
+
+::
+
+   $ devtool upgrade nano -V 2.9.3
+   NOTE: Starting bitbake server...
+   NOTE: Creating workspace layer in /home/scottrif/poky/build/workspace
+   Parsing recipes: 100% |##########################################| Time: 0:00:46
+   Parsing of 1431 .bb files complete (0 cached, 1431 parsed). 2040 targets, 56 skipped, 0 masked, 0 errors.
+   NOTE: Extracting current version source...
+   NOTE: Resolving any missing task queue dependencies
+          .
+          .
+          .
+   NOTE: Executing SetScene Tasks
+   NOTE: Executing RunQueue Tasks
+   NOTE: Tasks Summary: Attempted 74 tasks of which 72 didn't need to be rerun and all succeeded.
+   Adding changed files: 100% |#####################################| Time: 0:00:00
+   NOTE: Upgraded source extracted to /home/scottrif/poky/build/workspace/sources/nano
+   NOTE: New recipe is /home/scottrif/poky/build/workspace/recipes/nano/nano_2.9.3.bb
+
+Continuing with this example, you can use ``devtool build`` to build the
+newly upgraded recipe::
+
+   $ devtool build nano
+   NOTE: Starting bitbake server...
+   Loading cache: 100% |################################################################################################| Time: 0:00:01
+   Loaded 2040 entries from dependency cache.
+   Parsing recipes: 100% |##############################################################################################| Time: 0:00:00
+   Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
+   NOTE: Resolving any missing task queue dependencies
+          .
+          .
+          .
+   NOTE: Executing SetScene Tasks
+   NOTE: Executing RunQueue Tasks
+   NOTE: nano: compiling from external source tree /home/scottrif/poky/build/workspace/sources/nano
+   NOTE: Tasks Summary: Attempted 520 tasks of which 304 didn't need to be rerun and all succeeded.
+
+Within the ``devtool upgrade`` workflow, you can
+deploy and test your rebuilt software. For this example,
+however, running ``devtool finish`` cleans up the workspace once the
+source in your workspace is clean. This usually means using Git to stage
+and submit commits for the changes generated by the upgrade process.
+
+Once the tree is clean, you can clean things up in this example with the
+following command from the ``${BUILDDIR}/workspace/sources/nano``
+directory::
+
+   $ devtool finish nano meta-oe
+   NOTE: Starting bitbake server...
+   Loading cache: 100% |################################################################################################| Time: 0:00:00
+   Loaded 2040 entries from dependency cache.
+   Parsing recipes: 100% |##############################################################################################| Time: 0:00:01
+   Parsing of 1432 .bb files complete (1431 cached, 1 parsed). 2041 targets, 56 skipped, 0 masked, 0 errors.
+   NOTE: Adding new patch 0001-nano.bb-Stuff-I-changed-when-upgrading-nano.bb.patch
+   NOTE: Updating recipe nano_2.9.3.bb
+   NOTE: Removing file /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano/nano_2.7.4.bb
+   NOTE: Moving recipe file to /home/scottrif/meta-openembedded/meta-oe/recipes-support/nano
+   NOTE: Leaving source tree /home/scottrif/poky/build/workspace/sources/nano as-is; if you no longer need it then please delete it manually
+
+
+Using the ``devtool finish`` command cleans up the workspace and creates a patch
+file based on your commits. The tool puts all patch files back into the
+source directory in a sub-directory named ``nano`` in this case.
+
+Manually Upgrading a Recipe
+===========================
+
+If for some reason you choose not to upgrade recipes using
+:ref:`dev-manual/upgrading-recipes:Using the Auto Upgrade Helper (AUH)` or
+by :ref:`dev-manual/upgrading-recipes:Using \`\`devtool upgrade\`\``,
+you can manually edit the recipe files to upgrade the versions.
+
+.. note::
+
+   Manually updating multiple recipes scales poorly and involves many
+   steps. The recommendation to upgrade recipe versions is through AUH
+   or ``devtool upgrade``, both of which automate some steps and provide
+   guidance for others needed for the manual process.
+
+To manually upgrade recipe versions, follow these general steps:
+
+1. *Change the Version:* Rename the recipe such that the version (i.e.
+   the :term:`PV` part of the recipe name)
+   changes appropriately. If the version is not part of the recipe name,
+   change the value as it is set for :term:`PV` within the recipe itself.
+
+2. *Update* :term:`SRCREV` *if Needed*: If the source code your recipe builds
+   is fetched from Git or some other version control system, update
+   :term:`SRCREV` to point to the
+   commit hash that matches the new version.
+
+3. *Build the Software:* Try to build the recipe using BitBake. Typical
+   build failures include the following:
+
+   -  License statements were updated for the new version. For this
+      case, you need to review any changes to the license and update the
+      values of :term:`LICENSE` and
+      :term:`LIC_FILES_CHKSUM`
+      as needed.
+
+      .. note::
+
+         License changes are often inconsequential. For example, the
+         license text's copyright year might have changed.
+
+   -  Custom patches carried by the older version of the recipe might
+      fail to apply to the new version. For these cases, you need to
+      review the failures. Patches might not be necessary for the new
+      version of the software if the upgraded version has fixed those
+      issues. If a patch is necessary and failing, you need to rebase it
+      into the new version.
+
+4. *Optionally Attempt to Build for Several Architectures:* Once you
+   successfully build the new software for a given architecture, you
+   could test the build for other architectures by changing the
+   :term:`MACHINE` variable and
+   rebuilding the software. This optional step is especially important
+   if the recipe is to be released publicly.
+
+5. *Check the Upstream Change Log or Release Notes:* Checking both these
+   reveals if there are new features that could break
+   backwards-compatibility. If so, you need to take steps to mitigate or
+   eliminate that situation.
+
+6. *Optionally Create a Bootable Image and Test:* If you want, you can
+   test the new software by booting it onto actual hardware.
+
+7. *Create a Commit with the Change in the Layer Repository:* After all
+   builds work and any testing is successful, you can create commits for
+   any changes in the layer holding your upgraded recipe.
+