diff options
Diffstat (limited to 'documentation/dev-manual/dev-manual-newbie.xml')
| -rw-r--r-- | documentation/dev-manual/dev-manual-newbie.xml | 394 |
1 files changed, 308 insertions, 86 deletions
diff --git a/documentation/dev-manual/dev-manual-newbie.xml b/documentation/dev-manual/dev-manual-newbie.xml index 7c21379b99..869790d99f 100644 --- a/documentation/dev-manual/dev-manual-newbie.xml +++ b/documentation/dev-manual/dev-manual-newbie.xml | |||
| @@ -58,99 +58,321 @@ | |||
| 58 | <title>Using the Yocto Project in a Team Environment</title> | 58 | <title>Using the Yocto Project in a Team Environment</title> |
| 59 | 59 | ||
| 60 | <para> | 60 | <para> |
| 61 | It might not be immediately clear how you can use the Yocto Project in a team environment, | 61 | It might not be immediately clear how you can use the Yocto |
| 62 | or scale it for a large team of developers. | 62 | Project in a team environment, or scale it for a large team of |
| 63 | The specifics of any situation determine the best solution. | 63 | developers. |
| 64 | Granted that the Yocto Project offers immense flexibility regarding this, practices do exist | 64 | One of the strengths of the Yocto Project is that it is extremely |
| 65 | that experience has shown work well. | 65 | flexible. |
| 66 | Thus, you can adapt it to many different use cases and scenarios. | ||
| 67 | However, these characteristics can cause a struggle if you are trying | ||
| 68 | to create a working setup that scales across a large team. | ||
| 69 | To help with these types of situations, this section presents | ||
| 70 | some of the project's most successful experiences, | ||
| 71 | practices, solutions, and available technologies that work well. | ||
| 72 | Keep in mind, the information here is a starting point. | ||
| 73 | You can build off it and customize it to fit any | ||
| 74 | particular working environment and set of practices. | ||
| 66 | </para> | 75 | </para> |
| 67 | 76 | ||
| 68 | <para> | 77 | <section id='best-practices-system-configurations'> |
| 69 | The core component of any development effort with the Yocto Project is often an | 78 | <title>System Configurations</title> |
| 70 | automated build and testing framework along with an image generation process. | ||
| 71 | You can use these core components to check that the metadata can be built, | ||
| 72 | highlight when commits break the build, and provide up-to-date images that | ||
| 73 | allow developers to test the end result and use it as a base platform for further | ||
| 74 | development. | ||
| 75 | Experience shows that buildbot is a good fit for this role. | ||
| 76 | What works well is to configure buildbot to make two types of builds: | ||
| 77 | incremental and full (from scratch). | ||
| 78 | See "<ulink url='http://autobuilder.yoctoproject.org:8010/'>Welcome to the buildbot for the Yocto Project</ulink>" | ||
| 79 | for an example implementation that uses buildbot. | ||
| 80 | </para> | ||
| 81 | 79 | ||
| 82 | <para> | 80 | <para> |
| 83 | You can tie an incremental build to a commit hook that triggers the build | 81 | Systems across a large team should meet the needs of |
| 84 | each time a commit is made to the metadata. | 82 | two types of developers: those working on the direction of the |
| 85 | This practice results in useful acid tests that determine whether a given commit | 83 | software stack itself and those developing applications. |
| 86 | breaks the build in some serious way. | 84 | Regardless of the type of developer, their workstations must |
| 87 | Associating a build to a commit can catch a lot of simple errors. | 85 | be both reasonably powerful and run Linux. |
| 88 | Furthermore, the tests are fast so developers can get quick feedback on changes. | 86 | </para> |
| 89 | </para> | ||
| 90 | 87 | ||
| 91 | <para> | 88 | <section id='best-practices-application-development'> |
| 92 | Full builds build and test everything from the ground up. | 89 | <title>Application Development</title> |
| 93 | These types of builds usually happen at predetermined times like during the | ||
| 94 | night when the machine load is low. | ||
| 95 | </para> | ||
| 96 | 90 | ||
| 97 | <para> | 91 | <para> |
| 98 | Most teams have many pieces of software undergoing active development at any given time. | 92 | For developers who mainly do application level work |
| 99 | You can derive large benefits by putting these pieces under the control of a source | 93 | on top of an existing software stack, |
| 100 | control system that is compatible (i.e. Git or Subversion (SVN)) with the OpenEmbedded | 94 | here are some practices that work best: |
| 101 | build system that the Yocto Project uses. | 95 | <itemizedlist> |
| 102 | You can then set the autobuilder to pull the latest revisions of the packages | 96 | <listitem><para>Use a pre-built toolchain that |
| 103 | and test the latest commits by the builds. | 97 | contains the software stack itself. |
| 104 | This practice quickly highlights issues. | 98 | Then, develop the application code on top of the |
| 105 | The build system easily supports testing configurations that use both a | 99 | stack. |
| 106 | stable known good revision and a floating revision. | 100 | This method works well for small numbers of relatively |
| 107 | The build system can also take just the changes from specific source control branches. | 101 | isolated applications.</para></listitem> |
| 108 | This capability allows you to track and test specific changes. | 102 | <listitem><para>When possible, use the Yocto Project |
| 109 | </para> | 103 | plug-in for the <trademark class='trade'>Eclipse</trademark> IDE |
| 104 | and other pieces of Application Development | ||
| 105 | Technology (ADT). | ||
| 106 | For more information, see the | ||
| 107 | "<link linkend='application-development-workflow'>Application | ||
| 108 | Development Workflow</link>" section as well as the | ||
| 109 | <ulink url='&YOCTO_DOCS_ADT_URL;'>Yocto Project Application Developer's Guide</ulink>. | ||
| 110 | </para></listitem> | ||
| 111 | <listitem><para>Keep your cross-development toolchains | ||
| 112 | updated. | ||
| 113 | You can do this by provisioning either as new | ||
| 114 | toolchain downloads or as updates through a package | ||
| 115 | update mechanism using <filename>opkg</filename> | ||
| 116 | to provide updates to an existing toolchain. | ||
| 117 | The exact mechanics of how and when to do this are a | ||
| 118 | question for local policy.</para></listitem> | ||
| 119 | <listitem><para>Use multiple toolchains installed locally | ||
| 120 | into different locations to allow development across | ||
| 121 | versions.</para></listitem> | ||
| 122 | </itemizedlist> | ||
| 123 | </para> | ||
| 124 | </section> | ||
| 125 | |||
| 126 | <section id='best-practices-core-system-development'> | ||
| 127 | <title>Core System Development</title> | ||
| 128 | |||
| 129 | <para> | ||
| 130 | For core system development, it is often best to have the | ||
| 131 | build system itself available on the developer workstations | ||
| 132 | so developers can run their own builds and directly | ||
| 133 | rebuild the software stack. | ||
| 134 | You should keep the core system standard as much as | ||
| 135 | possible and do your work in layers on top of the core system. | ||
| 136 | You can share layers amongst the developers of a particular | ||
| 137 | project and contain the policy configuration that defines | ||
| 138 | the project. | ||
| 139 | </para> | ||
| 140 | |||
| 141 | <para> | ||
| 142 | Aside from the previous best practices, there exists a number | ||
| 143 | of tips and tricks that can help speed up core development | ||
| 144 | projects: | ||
| 145 | <itemizedlist> | ||
| 146 | <listitem><para>Use a | ||
| 147 | <ulink url='&YOCTO_DOCS_REF_URL;#shared-state-cache'>Shared State Cache</ulink> | ||
| 148 | (sstate) among groups of developers who are on a | ||
| 149 | fast network. | ||
| 150 | The best way to share sstate is through a | ||
| 151 | Network File System (NFS) share. | ||
| 152 | The first user to build a given component for the | ||
| 153 | first time contributes that object to the sstate, | ||
| 154 | while subsequent builds from other developers then | ||
| 155 | reuse the object rather than rebuild it themselves. | ||
| 156 | </para> | ||
| 157 | <para>Although it is possible to use other protocols for the | ||
| 158 | sstate such as HTTP and FTP, you should avoid these. | ||
| 159 | Using HTTP limits the sstate to read-only and | ||
| 160 | FTP provides poor performance. | ||
| 161 | </para></listitem> | ||
| 162 | <listitem><para>Have autobuilders contribute to the sstate | ||
| 163 | pool similarly to how the developer workstations | ||
| 164 | contribute. | ||
| 165 | For information, see the | ||
| 166 | <link linkend='best-practices-autobuilders'>Autobuilders</link> | ||
| 167 | section.</para></listitem> | ||
| 168 | <listitem><para>Build stand-alone tarballs that contain | ||
| 169 | "missing" system requirements if for some reason | ||
| 170 | developer workstations do not meet minimum system | ||
| 171 | requirements such as latest Python versions, | ||
| 172 | <filename>chrpath</filename>, or other tools. | ||
| 173 | You can install and relocate the tarball exactly as you | ||
| 174 | would the usual cross-development toolchain so that | ||
| 175 | all developers can meet minimum version requirements | ||
| 176 | on most distributions.</para></listitem> | ||
| 177 | <listitem><para>Use a small number of high performance | ||
| 178 | systems for testing purposes (e.g. dual six core Xeons | ||
| 179 | with 24GB RAM and plenty of disk space). | ||
| 180 | Developers can use these systems for wider, more | ||
| 181 | extensive testing while they continue to develop | ||
| 182 | locally using their primary development system. | ||
| 183 | </para></listitem> | ||
| 184 | </itemizedlist> | ||
| 185 | </para> | ||
| 186 | </section> | ||
| 187 | </section> | ||
| 110 | 188 | ||
| 111 | <para> | 189 | <section id='best-practices-source-control-management'> |
| 112 | Perhaps the hardest part of setting this up is defining the software project or | 190 | <title>Source Control Manangement (SCM)</title> |
| 113 | the metadata policies that surround the different source control systems. | ||
| 114 | Of course circumstances will be different in each case. | ||
| 115 | However, this situation reveals one of the Yocto Project's advantages - | ||
| 116 | the system itself does not | ||
| 117 | force any particular policy on users, unlike a lot of build systems. | ||
| 118 | The system allows the best policies to be chosen for the given circumstances. | ||
| 119 | </para> | ||
| 120 | 191 | ||
| 121 | <para> | 192 | <para> |
| 122 | In general, best practices exist that make your work with the Yocto | 193 | Keeping your |
| 123 | Project easier in a team environment. | 194 | <ulink url='&YOCTO_DOCS_DEV_URL;#metadata'>Metadata</ulink> |
| 124 | This list presents some of these practices you might consider following. | 195 | and any software you are developing under the |
| 125 | Of course, you need to understand that you do not have to follow these | 196 | control of an SCM system that is compatible |
| 126 | practices and your setup can be totally controlled and customized by | 197 | with the OpenEmbedded build system is adviseable. |
| 127 | your team: | 198 | Of the two SCMs available (Git or Subversion), the |
| 128 | <itemizedlist> | 199 | Yocto Project team strongly recommends using |
| 129 | <listitem><para>Use <link linkend='git'>Git</link> | 200 | <link linkend='git'>Git</link>. |
| 130 | as the source control system.</para></listitem> | 201 | Git is a distributed system that is easy to backup |
| 131 | <listitem><para>Maintain your metadata in layers that make sense | 202 | (each checkout is a backup in itself), allows you to work |
| 132 | for your situation. | 203 | remotely, and then connect back to the infrastructue. |
| 133 | See the "<link linkend='understanding-and-creating-layers'>Understanding | 204 | </para> |
| 134 | and Creating Layers</link>" section for more information on | 205 | |
| 135 | layers.</para></listitem> | 206 | <para> |
| 136 | <listitem><para>Separate the project's metadata and code by using | 207 | It is relatively easy to set up Git services and create |
| 137 | separate Git repositories. | 208 | infrastructure like |
| 138 | See the "<link linkend='yocto-project-repositories'>Yocto Project | 209 | <ulink url='&YOCTO_GIT_URL;'>http://git.yoctoproject.org</ulink>, |
| 139 | Source Repositories</link>" section for information on these | 210 | which is based on server software called |
| 140 | repositories. | 211 | <filename>gitolite</filename> with <filename>cgit</filename> |
| 141 | See the "<link linkend='getting-setup'>Getting Set Up</link>" section | 212 | being used to generate the web interface that lets you view the |
| 142 | for information on how to set up various Yocto Project related | 213 | repositories. |
| 143 | Git repositories.</para></listitem> | 214 | The <filename>gitlite</filename> software identifies users |
| 144 | <listitem><para>Set up the directory for the shared state cache | 215 | using <filename>ssh</filename> keys and allows branch-based |
| 145 | (<ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>) | 216 | access controls to repositories that you can control as little |
| 146 | where they make sense. | 217 | or as much as necessary. |
| 147 | For example, set up the sstate cache for developers using the | 218 | </para> |
| 148 | same office and share source directories on the developer's | 219 | |
| 149 | machines.</para></listitem> | 220 | <note> |
| 150 | <listitem><para>Set up an autobuilder and have it populate the | 221 | The setup of these services is beyond the scope of this manual. |
| 151 | sstate cache and source directories.</para></listitem> | 222 | However, sites such as these exist that describe how to perform |
| 152 | </itemizedlist> | 223 | setup: |
| 153 | </para> | 224 | <itemizedlist> |
| 225 | <listitem><para><ulink url='http://git-scm.com/book/ch4-8.html'>Git documentation</ulink>: | ||
| 226 | Describes how to install <filename>gitolite</filename> | ||
| 227 | on the server.</para></listitem> | ||
| 228 | <listitem><para><ulink url='http://sitaramc.github.com/gitolite/master-toc.html'>The <filename>gitolite</filename> master index</ulink>: | ||
| 229 | All topics for <filename>gitolite</filename>. | ||
| 230 | </para></listitem> | ||
| 231 | <listitem><para><ulink url='http://hjemli.net/git/cgit/tree/README'><filename>cgit</filename> index</ulink>: | ||
| 232 | A <filename>README</filename> file on how to create a | ||
| 233 | fast web interface for Git.</para></listitem> | ||
| 234 | </itemizedlist> | ||
| 235 | </note> | ||
| 236 | </section> | ||
| 237 | |||
| 238 | <section id='best-practices-autobuilders'> | ||
| 239 | <title>Autobuilders</title> | ||
| 240 | |||
| 241 | <para> | ||
| 242 | Autobuilders are often the core of a development project. | ||
| 243 | It is here that changes from individual developers are brought | ||
| 244 | together and centrally tested and subsequent decisions about | ||
| 245 | releases can be made. | ||
| 246 | Autobuilders also allow for "continuous integration" style | ||
| 247 | testing of software components and regression identification | ||
| 248 | and tracking. | ||
| 249 | </para> | ||
| 250 | |||
| 251 | <para> | ||
| 252 | See "<ulink url='http://autobuilder.yoctoproject.org:8010/'>Welcome to the buildbot for the Yocto Project</ulink>" | ||
| 253 | for the Yocto Project's reference implementation that uses | ||
| 254 | buildbot. | ||
| 255 | The Yocto Project team has found this implementation | ||
| 256 | works well in this role. | ||
| 257 | A public example of this is the Yocto Project | ||
| 258 | Autobuilders, which we use to test the overall health of the | ||
| 259 | project. | ||
| 260 | </para> | ||
| 261 | |||
| 262 | <para> | ||
| 263 | The features of this system are: | ||
| 264 | <itemizedlist> | ||
| 265 | <listitem><para>Highlights when commits break the build | ||
| 266 | </para></listitem> | ||
| 267 | <listitem><para>Populates an sstate cache from which | ||
| 268 | developers can pull rather than requiring local | ||
| 269 | builds</para></listitem> | ||
| 270 | <listitem><para>Allows commit hook triggers, | ||
| 271 | which trigger builds when commits are made | ||
| 272 | </para></listitem> | ||
| 273 | <listitem><para>Allows triggering of automated image booting | ||
| 274 | and testing under the QuickEMUlator (QEMU) | ||
| 275 | </para></listitem> | ||
| 276 | <listitem><para>Supports incremental build testing and from | ||
| 277 | scratch builds</para></listitem> | ||
| 278 | <listitem><para>Shared output that allows developer | ||
| 279 | testing and historical regression investigation | ||
| 280 | </para></listitem> | ||
| 281 | <listitem><para>Creates output that can be use for releases | ||
| 282 | </para></listitem> | ||
| 283 | <listitem><para>Allows scheduling of builds so that resources | ||
| 284 | can be used efficiently.</para></listitem> | ||
| 285 | </itemizedlist> | ||
| 286 | </para> | ||
| 287 | </section> | ||
| 288 | |||
| 289 | <section id='best-practices-policies-and-change-flow'> | ||
| 290 | <title>Policies and Change Flow</title> | ||
| 291 | |||
| 292 | <para> | ||
| 293 | The Yocto Project itself uses a hierarchy structure and a | ||
| 294 | pull model. | ||
| 295 | Scripts exist to create and send pull requests | ||
| 296 | (i.e. <filename>create-pull-request</filename> and | ||
| 297 | <filename>send-pull-request</filename>). | ||
| 298 | This model is in line with other open source projects where | ||
| 299 | maintainers are responsible for specific areas of the project | ||
| 300 | and a single maintainer handles the final top-of-tree merges. | ||
| 301 | </para> | ||
| 302 | |||
| 303 | <note> | ||
| 304 | You can also use a more collective push model. | ||
| 305 | The <filename>gitolite</filename> software supports both the | ||
| 306 | push and pull models quite easily. | ||
| 307 | </note> | ||
| 308 | |||
| 309 | <para> | ||
| 310 | As with any development environment, it is important | ||
| 311 | to document the policy used as well as any main project | ||
| 312 | guidelines so they are understoon by everyone. | ||
| 313 | It is also a good idea to have well structured | ||
| 314 | commit messages, which are usually a part of a project's | ||
| 315 | guidelines. | ||
| 316 | Good commit messages are essential when looking back in time and | ||
| 317 | trying to understand why changes were made. | ||
| 318 | </para> | ||
| 319 | |||
| 320 | <para> | ||
| 321 | If you discover that changes are needed to the core layer of the | ||
| 322 | project, it is worth sharing those with the community as soon | ||
| 323 | as possible. | ||
| 324 | Chances are if you have discovered the need for changes, someone | ||
| 325 | someone else in the community needs them also. | ||
| 326 | sooner than later. | ||
| 327 | </para> | ||
| 328 | </section> | ||
| 329 | |||
| 330 | <section id='best-practices-summary'> | ||
| 331 | <title>Summary</title> | ||
| 332 | |||
| 333 | <para> | ||
| 334 | This section summarizes thee key recommendations described in the | ||
| 335 | previous sections: | ||
| 336 | <itemizedlist> | ||
| 337 | <listitem><para>Use <link linkend='git'>Git</link> | ||
| 338 | as the source control system.</para></listitem> | ||
| 339 | <listitem><para>Maintain your metadata in layers that make sense | ||
| 340 | for your situation. | ||
| 341 | See the "<link linkend='understanding-and-creating-layers'>Understanding | ||
| 342 | and Creating Layers</link>" section for more information on | ||
| 343 | layers.</para></listitem> | ||
| 344 | <listitem><para>Separate the project's metadata and code by using | ||
| 345 | separate Git repositories. | ||
| 346 | See the "<link linkend='yocto-project-repositories'>Yocto Project | ||
| 347 | Source Repositories</link>" section for information on these | ||
| 348 | repositories. | ||
| 349 | See the "<link linkend='getting-setup'>Getting Set Up</link>" section | ||
| 350 | for information on how to set up various Yocto Project related | ||
| 351 | Git repositories.</para></listitem> | ||
| 352 | <listitem><para>Set up the directory for the shared state cache | ||
| 353 | (<ulink url='&YOCTO_DOCS_REF_URL;#var-SSTATE_DIR'><filename>SSTATE_DIR</filename></ulink>) | ||
| 354 | where they make sense. | ||
| 355 | For example, set up the sstate cache for developers using the | ||
| 356 | same office and share source directories on the developer's | ||
| 357 | machines.</para></listitem> | ||
| 358 | <listitem><para>Set up an autobuilder and have it populate the | ||
| 359 | sstate cache and source directories.</para></listitem> | ||
| 360 | <listitem><para>Follow the project commit guidelines for | ||
| 361 | writing good commit messages. | ||
| 362 | See the "<link linkend='how-to-submit-a-change'>How to Submit a Change</link>" | ||
| 363 | section.</para></listitem> | ||
| 364 | <listitem><para>Send changes to the core sooner than later | ||
| 365 | as others likely run into the same issues. | ||
| 366 | For some guidance on mailing lists to use, see the list in the | ||
| 367 | "<link linkend='how-to-submit-a-change'>How to Submit a Change</link>" | ||
| 368 | section. | ||
| 369 | For a description of the available mailing lists, see | ||
| 370 | "<ulink url='&YOCTO_DOCS_REF_URL;#resources-mailinglist'>Mailing Lists</ulink>" | ||
| 371 | section in the Yocto Project Reference Manual. | ||
| 372 | </para></listitem> | ||
| 373 | </itemizedlist> | ||
| 374 | </para> | ||
| 375 | </section> | ||
| 154 | </section> | 376 | </section> |
| 155 | 377 | ||
| 156 | <section id='yocto-project-repositories'> | 378 | <section id='yocto-project-repositories'> |