This chapter helps you understand the Yocto Project as an open source development project. In general, working in an open-source environment is very different than working in a proprietary environment. Additionally, the Yocto Project uses specific tools and constructs as part of its development environment. The chapter specifically addresses open source philosophy, licensing issues, code repositories, the open source distributed version control system Git, and best practices using Yocto Project.

Open source philosophy is characterized by software development directed by peer production and collaboration through a concerned community of developers. Contrast this to the more standard centralized development models used by commercial software companies where a finite set of developers produce a product for sale using a defined set of procedures that ultimately result in an end-product whose architecture and source material are closed to the public. Open source projects conceptually have differing concurrent agendas, approaches, and production. These facets of the development process can come from anyone in the public (community) that has a stake in the software project. The open source environment contains new copyright, licensing, domain, and consumer issues that differ from the more traditional development environment. In an open source environment the end-product, source material, and documentation are all available to the public at no cost. A benchmark example of an open source project is the Linux Kernel, which was initially conceived and created by Finnish computer science student Linus Torvalds in 1991. Conversely, a good example of a non-open source project is the Windows family of operating systems developed by Microsoft Corporation. Wikipedia has a good historical description of the Open Source Philosophy here. You can also find helpful information on how to participate in the Linux Community here.

The Yocto Project team maintains complete source repositories for all Yocto Project files here. This web-based source code browser is organized into categories by function such as IDE Plugins, Matchbox, Poky, Yocto Linux Kernel, and so forth. From the interface, you can click on any particular item in the "Name" column and see the URL at the bottom of the page that you need to set up a Git repository for that particular item. The ability to create Git repositories of the Yocto Project source allows you to make changes, contribute to the history, and ultimately enhance the Yocto Project's tools, Board Support Packages, and so forth. Conversely, if you are a developer that is not interested in contributing back to the Yocto Project you have the ability to simply download and extract release tarballs and use them within the Yocto Project environment. All that is required is a particular release of Yocto Project, a kernel, and your application source code. For any supported release of Yocto Project you can go to the Yocto Project website’s download page and get a tarball of the release. You can also go to this site to download any supported BSP tarballs. Unpacking the tarball gives you a hierarchical directory structure of Yocto Project files that lets you develop using the Yocto Project. Once you are set up through either tarball extraction or creation of Git repositories, you are ready to develop. In summary, here is where you can get the Yocto Project files needed for development: Source Repositories: This area contains IDE Plugins, Matchbox, Poky, Poky Support, Tools, Yocto Linux Kernel, and Yocto Metadata Layers. You can create Git repositories for each of these areas. Index of /downloads: This area contains an index of the Eclipse-plugin, miscellaneous support, poky, pseudo, and all released versions of Yocto Project in the form of images or tarballs. Downloading and extracting these files does not produce a Git repository but rather a snapshot of a particular release or image. [WRITER NOTE: link will be http://downloads.yoctoproject.org.] Yocto Project Download Page This page on the Yocto Project website allows you to download any Yocto Project release or Board Support Package (BSP) in tarball form. The tarballs are similar to those found in the Index of /downloads: area.

Following is a list of terms and definitions users new to the Yocto Project development environment might find helpful. Some terms are universal but are included here just in case: Image - An image is the result produced when BitBake processes a given collection of recipes and related metadata. Images are the binary output that runs on specific hardware and for specific use cases. For a list of the supported image types that the Yocto Project provides, see the Reference: Images appendix in The Yocto Project Reference Manual. Recipe - A set of instructions for building packages. A recipe describes where you get source code and which patches to apply. Recipes describe dependencies for libraries or for other recipes and they also contain configuration and compilation options. Recipes contain the logical unit of execution, the software/images to build, and use the .bb file extension. BitBake - The task executor and scheduler used by Yocto Project to build images. For more information on BitBake, see the BitBake documentation. Package - The packaged output from a baked recipe. A package is generally the compiled binaries produced from the recipe's sources. You ‘bake’ something by running it through BitBake. Layer - A collection of recipes representing the core, a BSP, or an application stack. Metadata - A term used throughout the Yocto Project documentation that refers to the files that BitBake parses when building an image. Metadata includes recipes, classes, and configuration files. Meta-Toolchain - A collection of software development tools and utilities that allow you to develop software for targeted architectures. These toolchains contain cross-compilers, linkers, and debuggers that are specific to an architecure. You can use the Yocto Project to build meta-toolchains in tarball form that when unpacked contain the development tools you need to cross-compile and test your software. The Yocto Project ships with images that contain toolchains for supported architectures as well. Configuration File: Configuration information in the .conf files provides global definitions of variables. The build/conf/local.conf configuration file defines user-defined variables that effect each build. The distro/poky.conf configuration file defines Yocto ‘distro’ configuration variables used only when building with this policy. The machine/beagleboard.conf configuration file defines variables for the Beagleboard and are only used when building for that target (i.e. Texas Instruments ARM Cortex-A8 development board). Configuration files end with a .conf filename extension. Classes - Files that provide for logic encapsulation and inheritance allowing commonly used patterns to be defined once and easily used in multiple recipes. Class files end with the .bbclass filename extension. Append Files - Files that append build information to a recipe file. Information in append files override the information in the similarly-named recipe file. Append files use the .bbappend filename suffix. Tasks - Arbitrary groups of software Recipes. You simply use Tasks to hold recipes that when built usually accomplish a single task. For example, a task could contain the recipes for a company’s proprietary or value-add software. Or the task could contain the recipes that enable graphics. A task is really just another recipe. Because task files are recipes, they end with the .bb filename extension. OE-Core - A core set of metadata originating with OpenEmbedded (OE) that is shared between OE and the Yocto Project. This metadata is found in the meta directory of the Yocto Project files. Upstream - A reference to source code or repositories that are not local to the development system but located in a master area that is controlled by the maintainer of the source code. For example, in order for a developer to work on a particular piece of code they need to first get a copy of it from an "upstream" source.

Because open source projects are open to the public they have different licensing structures in place. License evolution for both Open Source and Free Software has an interesting history. If you are interested in the history you can find basic information here: Open source license history Free software license history In general, the Yocto Project is broadly licensed under the Massachusetts Institute of Technology (MIT) License. MIT licensing permits the reuse of software within proprietary software as long as the license is distributed with that software. MIT is also compatible with the GNU General Public License (GPL). Patches to the Yocto Project follow the upstream licensing scheme. You can find information on the MIT license at here. You can find information on the GNU GPL here. When you build an image using Yocto Project, the build process uses a known list of licenses to ensure compliance. Once the build completes, the list of all licenses found and used during the build are kept in the resulting build directory at <build_directory>/tmp/deploy/images/licenses. If a module requires a license that is not in the base list, the build process generates a warning during the build. These tools make it easier for a developer to be certain of the licenses with which their shipped products must comply. However, it is still up to the developer to resolve potential licensing issues. The base list of licenses used by the build process is a combination of the Software Package Data Exchange (SPDX) list and the Open Source Initiative (OSI) projects. SPDX Group is a working group of the Linux Foundation that maintains a specification for a standard format for communicating the components, licenses, and copyrights associated with a software package. OSI is a corporation dedicated to the Open Source Definition and the effort for reviewing and approving licenses that are OSD-conformant. You can find a list of the combined SPDX and OSI licenses that the Yocto Project uses here. The wiki page discusses the license infrastructure used by the Yocto Project.

The Yocto Project uses Git, which is a free, open source distributed version control system. Git supports distributed development, non-linear development, and can handle large projects. It is best that you know how to work with Git if you are going to use Yocto Project for development. Git has an extensive set of commands that lets you manage changes and perform collaboration over the life of a project. Conveniently though, you can manage with a small set of basic operations and workflows once you understand the basic philosophy behind Git. You do not have to be an expert in Git to be functional. A good place to look for instruction on a minimal set of Git commands is here. If you need to download Git you can do so here. Git works by using branching techniques that track content change (not files) within a project (e.g. a new feature or updated documentation). Creating a tree-like structure based on project divergence allows for excellent historical information over the life of a project. This methodology also allows for an environment in which you can do lots of experimentation on your project as you develop changes or new features. For example, you can create a “branch”, experiment with some feature, and then if you like the feature you incorporate the branch into the tree. If you don’t, you cut the branch off by deleting it. If you don’t know much about Git, we suggest you educate yourself by visiting the links previously mentioned. The following list briefly describes some basic Git operations as a way to get started. As with any set of commands, this list (in most cases) simply shows the base command and omits the many arguments they support. See the Git documentation for complete descriptions and strategies on how to use these commands: git init – Initializes an empty Git repository. You cannot use Git commands unless you have a .git repository. git clone – Creates a clone of a repository. During collaboration this command allows you to create a local repository that is on equal footing with a fellow developer’s repository. git add – Adds updated file contents to the index that Git uses to track changes. All files that have changed must be added before they can be committed. git commit – Creates a “commit” that documents the changes you made. Commits are used for historical purposes, for determining if a maintainer of a project will allow the change, and for ultimately pushing the change from your local Git repository into the project’s upstream (or master) repository. git status – Reports any modified files that possibly need added and committed. git checkout <branch-name> - Changes your working branch. This command is analogous to “cd”. git checkout –b <working-branch> - Creates a working branch on your local machine where you can isolate work. It is a good idea to use local branches when adding specific features or changes. This way if you don’t like what you have done you can easily get rid of the work. git branch – Reports existing branches and tells you which branch in which you are currently working. git branch -D <branch-name> – Deletes an existing branch. You need to be in a branch other than the one you are deleting in order to delete <branch-name>. git pull – Retrieves information from an upstream Git repository and places it in your local Git repository. You use this command to make sure you are synchronized with the upstream repository from which the project’s maintainer uses to pull changes into the master repository. git push – Sends all your local changes you have committed to an upstream Git repository. The maintainer of the project draws from these repositories when adding your changes to the project’s master repository. git merge – Combines or adds changes from one local branch of your repository with another branch. When you create a local Git repository the default branch is named “master”. A typical workflow is to create a temporary branch for isolated work, make and commit your changes, switch to the master branch, merge the changes from the temporary branch into the master branch, and then delete the temporary branch git cherry-pick – Choose and apply specific commits from one branch into another branch. There are times when you might not be able to merge all the changes in one branch with another but need to pick out certain ones. gitk – Provides a GUI view of the branches and changes in your local Git repository. This command is a good way to see where things have diverged in your local repository. git log – Reports a history of your changes to the repository.

This section provides some overview on workflows using Git. In particular, the information covers basic practices that describe roles and actions in a collaborative development environment. Again, if you are familiar with this type of development environment you might want to just skip the section. The Yocto Project files are maintained using Git in a "master" branch whose Git history tracks every change and whose structure provides branches for all diverging functionality. Although there is no need to use Git, This practice is typical for open-source projects. For the Yocto Project a key individual called the "maintainer" is responsible for "master". The "master" branch is the “upstream” repository where the final builds of the project occur. The maintainer is responsible for allowing changes in from other developers and for organizing the underlying branch structure to reflect release strategies and so forth. The project also has contribution repositories known as “contrib” areas. These areas temporarily hold changes to the project that have been submitted or committed by the Yocto Project development team and by community members that contribute to the project. The maintainer determines if the changes are qualified to be moved from the "contrib" areas into the "master" branch of the Git repository. Developers (including contributing community members) create and maintain cloned repositories of the upstream "master" branch. These repositories are local to their development platforms and are used to develop changes. When a developer is satisfied with a particular feature or change they “push” the changes to the appropriate "contrib" repository. Developers are responsible for keeping their local repository up-to-date with "master". They are also responsible for straightening out any conflicts that might arise within files that are being worked on simultaneously by more than one person. All this work is done locally on the developer’s machine before anything is pushed to a "contrib" area and examined at the maintainer’s level. A somewhat formal method exists by which developers commit changes and push them into the "contrib" area and subsequently request that the maintainer include them into "master" This process is called “submitting a patch” or “submitting a change.” To summarize the environment: we have a single point of entry for changes into the project’s "master" branch of the Git repository, which is controlled by the project’s maintainer. And, we have a set of developers who independently develop, test, and submit changes to "contrib" areas for the maintainer to examine. The maintainer then chooses which changes are going to become permanently a part of the project. While each development environment is unique, there are some best practices or methods that help development run smoothly. The following list describes some of these practices. For more detailed information about these strategies see Git Workflows. Make Small Changes - It is best to keep your changes you commit small as compared to bundling many disparate changes into a single commit. This practice not only keeps things manageable but also allows the maintainer to more easily include or refuse changes. It is also good practice to leave the repository in a state that allows you to still successfully build your project. Use Branches Liberally - It is very easy to create, use, and delete local branches in your working Git repository. You can name these branches anything you like. It is helpful to give them names associated with the particular feature or change on which you are working. Once you are done with a feature or change you simply discard the branch. Merge Changes - The git merge command allows you to take the changes from one branch and fold them into another branch. This process is especially helpful when more than a single developer might be working on different parts of the same feature. Merging changes also automatically identifies any collisions or “conflicts” that might happen resulting from the same lines of code being altered by two different developers. Manage Branches - Because branches are easy to use, you should use a system where branches indicate varying levels of code readiness. For example, you can have a “work” branch to develop in, a “test” branch where the code or change is tested, a “stage” branch where changes are ready to be committed, and so forth. As your project develops, you can merge code across the branches to reflect ever-increasing stable states of the development. Use Push and Pull - The push-pull workflow is based on the concept of developers “pushing” local commits to a remote repository, which is usually a contribution repository. It is also based on the developers “pulling” known states of the project down into their local development repositories. This workflow easily allows you to pull changes submitted by other developers from the upstream repository into your work area ensuring that you have the most recent software on which to develop. The Yocto Project has two scripts named create-pull-request and send-pull-request that ship with the release to facilitate this workflow. You can find these scripts in the local Yocto Project files Git repository in scripts. Patch Workflow - This workflow allows you to notify the maintainer through an email that you have a change (or patch) you would like considered for the "master" branch of the Git repository. To send this type of change you format the patch and then send the email using the Git commands git format-patch and git send-email. You can find information on how to submit later in this chapter.

The Yocto Project uses Bugzilla to track bugs. This bug-tracking application works well for group development because it tracks bugs and code changes, can be used to communicate changes and problems with developers, can be used to submit and review patches, and can be used to manage quality assurance. You can find a good overview of Bugzilla here. Sometimes it is helpful to submit, investigate, or track a bug against the Yocto Project itself such as when discovering an issue with some component of the build system that acts contrary to the documentation or expectations. You can find information for Bugzilla configuration and bug tracking procedures specific to the Yocto Project here. The Yocto Project uses its own version of the Bugzilla application. You can find the home page here. You need to use this implementation of Bugzilla when logging a defect against anything released by the Yocto Project team. Here are some things to remember when dealing with bugs against the Yocto Project: The Yocto Project follows a naming bug-naming convention: [YOCTO #<number>], where <number> is the assigned defect ID used in Bugzilla. So, for example, a valid way to refer to a defect when creating a commit comment would be [YOCTO 1011]. This convention becomes important if you are submitting patches against the Yocto Project code itself. See the following section for more information. Defects for Yocto Project fall into one of four classifications: Yocto Projects, Infrastructure, Poky, and Yocto Metadata Layers.

Contributions to the Yocto Project are very welcome. You should send patches to the Yocto Project mailing list to get it in front of the Yocto Project Maintainer. When you send your patch, be sure to include a "signed-off-by:" line in the same style as required by the Linux kernel. Adding this line signifies the developer has agreed to the Developer's Certificate of Origin 1.1 as follows: Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. A Poky contributions tree (poky-contrib, git://git.yoctoproject.org/poky-contrib.git) exists for contributors to stage contributions. If people desire such access, please ask on the mailing list. Usually, the Yocto Project team will grant access to anyone with a proven track record of good patches. In a collaborative environment, it is necessary to have some sort of standard or method through which you submit changes. Otherwise, things would get quite chaotic. When you submit a change or patch to the Yocto Project, you must follow certain procedures. In particular, the headers in patches and the commit messages must follow a certain standard. The general process is the same as described earlier in this chapter. For complete details on how to create proper commit messages and patch headers see [WRITER NOTE: I need the link to Mark's wiki page here that describes the process.] Following are general instructions for both pushing changes upstream and for submitting changes as patches.

The basic flow for pushing a change to an upstream "contrib" Git repository is as follows: Make your changes in your local Git repository. Stage your commit (or change) by using the git add command. Commit the change by using the git commit command and push it to the upstream "contrib" repository. Be sure to provide a commit message that follows the project’s commit standards. Notify the maintainer that you have pushed a change. You can find general Git information on how to push a change upstream here.

If you have a just a few changes you can commit them and then submit them as an email to the maintainer. Here is a general procedure: Make your changes in your local Git repository. Stage your commit (or change) by using the git add command. Commit the change by using the git commit command. Be sure to provide a commit message that follows the project’s commit standards. Format the commit by using the git format-patch command. This step produces a numbered series of files in the current directory – one for each commit. Import the files into your mail client by using the git-send-email command. Send the email by hand to the maintainer. Be aware that there could be protocols and standards that you need to follow for your particular project. You can find general Git information for submitting a patch here.