From e57f8627beda2c2b133b55135ce7aec814993592 Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Tue, 12 Feb 2013 17:03:25 -0600 Subject: dev-manual: Revision of "Team Environment" section. Fixes YOCTO #3274 First draft of a re-written section that describes best practices for scaling the YP over a large development effort. this draft is based on Richard Purdie's text he sent to me. (From yocto-docs rev: e5135f76946997a0a060dd16c0b62308be4f6048) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- documentation/dev-manual/dev-manual-newbie.xml | 394 +++++++++++++++++++------ 1 file changed, 308 insertions(+), 86 deletions(-) diff --git a/documentation/dev-manual/dev-manual-newbie.xml b/documentation/dev-manual/dev-manual-newbie.xml index 7c21379b99..869790d99f 100644 --- a/documentation/dev-manual/dev-manual-newbie.xml +++ b/documentation/dev-manual/dev-manual-newbie.xml @@ -58,99 +58,321 @@ Using the Yocto Project in a Team Environment - It might not be immediately clear how you can use the Yocto Project in a team environment, - or scale it for a large team of developers. - The specifics of any situation determine the best solution. - Granted that the Yocto Project offers immense flexibility regarding this, practices do exist - that experience has shown work well. + It might not be immediately clear how you can use the Yocto + Project in a team environment, or scale it for a large team of + developers. + One of the strengths of the Yocto Project is that it is extremely + flexible. + Thus, you can adapt it to many different use cases and scenarios. + However, these characteristics can cause a struggle if you are trying + to create a working setup that scales across a large team. + To help with these types of situations, this section presents + some of the project's most successful experiences, + practices, solutions, and available technologies that work well. + Keep in mind, the information here is a starting point. + You can build off it and customize it to fit any + particular working environment and set of practices. - - The core component of any development effort with the Yocto Project is often an - automated build and testing framework along with an image generation process. - You can use these core components to check that the metadata can be built, - highlight when commits break the build, and provide up-to-date images that - allow developers to test the end result and use it as a base platform for further - development. - Experience shows that buildbot is a good fit for this role. - What works well is to configure buildbot to make two types of builds: - incremental and full (from scratch). - See "Welcome to the buildbot for the Yocto Project" - for an example implementation that uses buildbot. - +
+ System Configurations - - You can tie an incremental build to a commit hook that triggers the build - each time a commit is made to the metadata. - This practice results in useful acid tests that determine whether a given commit - breaks the build in some serious way. - Associating a build to a commit can catch a lot of simple errors. - Furthermore, the tests are fast so developers can get quick feedback on changes. - + + Systems across a large team should meet the needs of + two types of developers: those working on the direction of the + software stack itself and those developing applications. + Regardless of the type of developer, their workstations must + be both reasonably powerful and run Linux. + - - Full builds build and test everything from the ground up. - These types of builds usually happen at predetermined times like during the - night when the machine load is low. - +
+ Application Development - - Most teams have many pieces of software undergoing active development at any given time. - You can derive large benefits by putting these pieces under the control of a source - control system that is compatible (i.e. Git or Subversion (SVN)) with the OpenEmbedded - build system that the Yocto Project uses. - You can then set the autobuilder to pull the latest revisions of the packages - and test the latest commits by the builds. - This practice quickly highlights issues. - The build system easily supports testing configurations that use both a - stable known good revision and a floating revision. - The build system can also take just the changes from specific source control branches. - This capability allows you to track and test specific changes. - + + For developers who mainly do application level work + on top of an existing software stack, + here are some practices that work best: + + Use a pre-built toolchain that + contains the software stack itself. + Then, develop the application code on top of the + stack. + This method works well for small numbers of relatively + isolated applications. + When possible, use the Yocto Project + plug-in for the Eclipse IDE + and other pieces of Application Development + Technology (ADT). + For more information, see the + "Application + Development Workflow" section as well as the + Yocto Project Application Developer's Guide. + + Keep your cross-development toolchains + updated. + You can do this by provisioning either as new + toolchain downloads or as updates through a package + update mechanism using opkg + to provide updates to an existing toolchain. + The exact mechanics of how and when to do this are a + question for local policy. + Use multiple toolchains installed locally + into different locations to allow development across + versions. + + +
+ +
+ Core System Development + + + For core system development, it is often best to have the + build system itself available on the developer workstations + so developers can run their own builds and directly + rebuild the software stack. + You should keep the core system standard as much as + possible and do your work in layers on top of the core system. + You can share layers amongst the developers of a particular + project and contain the policy configuration that defines + the project. + + + + Aside from the previous best practices, there exists a number + of tips and tricks that can help speed up core development + projects: + + Use a + Shared State Cache + (sstate) among groups of developers who are on a + fast network. + The best way to share sstate is through a + Network File System (NFS) share. + The first user to build a given component for the + first time contributes that object to the sstate, + while subsequent builds from other developers then + reuse the object rather than rebuild it themselves. + + Although it is possible to use other protocols for the + sstate such as HTTP and FTP, you should avoid these. + Using HTTP limits the sstate to read-only and + FTP provides poor performance. + + Have autobuilders contribute to the sstate + pool similarly to how the developer workstations + contribute. + For information, see the + Autobuilders + section. + Build stand-alone tarballs that contain + "missing" system requirements if for some reason + developer workstations do not meet minimum system + requirements such as latest Python versions, + chrpath, or other tools. + You can install and relocate the tarball exactly as you + would the usual cross-development toolchain so that + all developers can meet minimum version requirements + on most distributions. + Use a small number of high performance + systems for testing purposes (e.g. dual six core Xeons + with 24GB RAM and plenty of disk space). + Developers can use these systems for wider, more + extensive testing while they continue to develop + locally using their primary development system. + + + +
+
- - Perhaps the hardest part of setting this up is defining the software project or - the metadata policies that surround the different source control systems. - Of course circumstances will be different in each case. - However, this situation reveals one of the Yocto Project's advantages - - the system itself does not - force any particular policy on users, unlike a lot of build systems. - The system allows the best policies to be chosen for the given circumstances. - +
+ Source Control Manangement (SCM) - - In general, best practices exist that make your work with the Yocto - Project easier in a team environment. - This list presents some of these practices you might consider following. - Of course, you need to understand that you do not have to follow these - practices and your setup can be totally controlled and customized by - your team: - - Use Git - as the source control system. - Maintain your metadata in layers that make sense - for your situation. - See the "Understanding - and Creating Layers" section for more information on - layers. - Separate the project's metadata and code by using - separate Git repositories. - See the "Yocto Project - Source Repositories" section for information on these - repositories. - See the "Getting Set Up" section - for information on how to set up various Yocto Project related - Git repositories. - Set up the directory for the shared state cache - (SSTATE_DIR) - where they make sense. - For example, set up the sstate cache for developers using the - same office and share source directories on the developer's - machines. - Set up an autobuilder and have it populate the - sstate cache and source directories. - - + + Keeping your + Metadata + and any software you are developing under the + control of an SCM system that is compatible + with the OpenEmbedded build system is adviseable. + Of the two SCMs available (Git or Subversion), the + Yocto Project team strongly recommends using + Git. + Git is a distributed system that is easy to backup + (each checkout is a backup in itself), allows you to work + remotely, and then connect back to the infrastructue. + + + + It is relatively easy to set up Git services and create + infrastructure like + http://git.yoctoproject.org, + which is based on server software called + gitolite with cgit + being used to generate the web interface that lets you view the + repositories. + The gitlite software identifies users + using ssh keys and allows branch-based + access controls to repositories that you can control as little + or as much as necessary. + + + + The setup of these services is beyond the scope of this manual. + However, sites such as these exist that describe how to perform + setup: + + Git documentation: + Describes how to install gitolite + on the server. + The gitolite master index: + All topics for gitolite. + + cgit index: + A README file on how to create a + fast web interface for Git. + + +
+ +
+ Autobuilders + + + Autobuilders are often the core of a development project. + It is here that changes from individual developers are brought + together and centrally tested and subsequent decisions about + releases can be made. + Autobuilders also allow for "continuous integration" style + testing of software components and regression identification + and tracking. + + + + See "Welcome to the buildbot for the Yocto Project" + for the Yocto Project's reference implementation that uses + buildbot. + The Yocto Project team has found this implementation + works well in this role. + A public example of this is the Yocto Project + Autobuilders, which we use to test the overall health of the + project. + + + + The features of this system are: + + Highlights when commits break the build + + Populates an sstate cache from which + developers can pull rather than requiring local + builds + Allows commit hook triggers, + which trigger builds when commits are made + + Allows triggering of automated image booting + and testing under the QuickEMUlator (QEMU) + + Supports incremental build testing and from + scratch builds + Shared output that allows developer + testing and historical regression investigation + + Creates output that can be use for releases + + Allows scheduling of builds so that resources + can be used efficiently. + + +
+ +
+ Policies and Change Flow + + + The Yocto Project itself uses a hierarchy structure and a + pull model. + Scripts exist to create and send pull requests + (i.e. create-pull-request and + send-pull-request). + This model is in line with other open source projects where + maintainers are responsible for specific areas of the project + and a single maintainer handles the final top-of-tree merges. + + + + You can also use a more collective push model. + The gitolite software supports both the + push and pull models quite easily. + + + + As with any development environment, it is important + to document the policy used as well as any main project + guidelines so they are understoon by everyone. + It is also a good idea to have well structured + commit messages, which are usually a part of a project's + guidelines. + Good commit messages are essential when looking back in time and + trying to understand why changes were made. + + + + If you discover that changes are needed to the core layer of the + project, it is worth sharing those with the community as soon + as possible. + Chances are if you have discovered the need for changes, someone + someone else in the community needs them also. + sooner than later. + +
+ +
+ Summary + + + This section summarizes thee key recommendations described in the + previous sections: + + Use Git + as the source control system. + Maintain your metadata in layers that make sense + for your situation. + See the "Understanding + and Creating Layers" section for more information on + layers. + Separate the project's metadata and code by using + separate Git repositories. + See the "Yocto Project + Source Repositories" section for information on these + repositories. + See the "Getting Set Up" section + for information on how to set up various Yocto Project related + Git repositories. + Set up the directory for the shared state cache + (SSTATE_DIR) + where they make sense. + For example, set up the sstate cache for developers using the + same office and share source directories on the developer's + machines. + Set up an autobuilder and have it populate the + sstate cache and source directories. + Follow the project commit guidelines for + writing good commit messages. + See the "How to Submit a Change" + section. + Send changes to the core sooner than later + as others likely run into the same issues. + For some guidance on mailing lists to use, see the list in the + "How to Submit a Change" + section. + For a description of the available mailing lists, see + "Mailing Lists" + section in the Yocto Project Reference Manual. + + + +
-- cgit v1.2.3-54-g00ecf