From 85a0b9b941fbd38d44e4802a27290ad9ec4bb963 Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Wed, 18 Feb 2015 06:24:18 -0800 Subject: toaster-manual: Draft of Manual. Fixes [YOCTO #6901] (From yocto-docs rev: 258b46b0f3e23045b5fb8c97d03dceb69db5ce20) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- .../toaster-manual/toaster-manual-reference.xml | 1069 ++++++++++++++++++++ 1 file changed, 1069 insertions(+) create mode 100644 documentation/toaster-manual/toaster-manual-reference.xml (limited to 'documentation/toaster-manual/toaster-manual-reference.xml') diff --git a/documentation/toaster-manual/toaster-manual-reference.xml b/documentation/toaster-manual/toaster-manual-reference.xml new file mode 100644 index 0000000000..05184fa1eb --- /dev/null +++ b/documentation/toaster-manual/toaster-manual-reference.xml @@ -0,0 +1,1069 @@ + %poky; ] > + + + +Concepts and Reference + + + In order to configure and use Toaster, you should understand some + concepts and have some basic command reference material available. + This final chapter provides conceptual information on layer sources, + releases, and JSON configuration files. + Also provided is a quick look at some useful + manage.py commands. + Information on manage.py commands does exist + across the Web and the information in this manual by no means + attempts to provide a command reference. + However, having relevant command useful for the Toaster context + is helpful. + + +
+ Layer Source + + + In general, a layer source is some source of existing layers. + In particular, we are concerned with layers that you can use + with the Yocto Project and Toaster. + This chapter describes a particular type of layer source called + a "layer index." + + + + A layer index is a web application that contains information + about a set of custom layers. + A good example of an existing layer index is the + OpenEmbedded Metadata Index. + A public instance of this layer index exists at + . + You can find the code for this layer index's web application at + . + + + + When you tie a layer source into Toaster, it can query the layer + source through a + REST + API, store the information about the layers in the Toaster + database, and then show the information to users. + Users are then able to view that information and build layers + from Toaster itself without worrying about cloning or editing + the BitBake layers configuration file + bblayers.conf. + + + + Tying a layer source into Toaster is convenient when you have + many custom layers that need to be built on a regular basis by + a community of developers. + In fact, Toaster comes pre-configured with the OpenEmbedded + Metadata Index. + + You do not have to use a layer source to use Toaster. + Tying into a layer source is optional. + + + +
+ Setting Up and Using a Layer Source + + + To use your own layer source, you need to set up the layer + source and then tie it into Toaster. + This section describes how to tie into a layer index in a manner + similar to the way Toaster ties into the OpenEmbedded Metadata + Index. + + +
+ Understanding Your Layers + + + The obvious first step for using a layer index is to have + several custom layers that developers build and access using + the Yocto Project on a regular basis. + This set of layers needs to exist and you need to be + familiar with where they reside. + You will need that information when you set up the + code for the web application that "hooks" into your set of + layers. + + + + For general information on layers, see the + "BSP Layers" + and + "Using the Yocto Project's BSP Tools" + sections in the Yocto Project Board Support Package (BSP) + Developer's Guide. + +
+ +
+ Configuring Toaster to Hook Into Your Layer Index + + + If you want Toaster to use your layer index, you must host + the web application in a server to which Toaster can + connect. + You also need to give Toaster the information about your + layer index. + In other words, you have to configure Toaster to use your + layer index. + This section describes two methods by which you can + configure and use your layer index. + + + + In the previous section, the code for the OpenEmbedded + Metadata Index (i.e. + ) was + referenced. + You can use this code, which is at + , + as a base to create your own layer index. + + +
+ Use the Administration Interface + + + Access the administration interface through a + browser by entering the URL of your Toaster instance and + adding "/admin" to the end of the + URL. + + + + The administration interface has a "Layer sources" + section that includes an "Add layer source" button. + Click that button and provide the required information. + Make sure you select "layerindex" as the layer source type. + +
+ +
+ Select the <filename>toasterconf.json</filename> File + + + If you do not want to use the Administration + Interface, you can edit the + toasterconf.json + file and reload it to Toaster. + + + + When you set up Toaster in Build Mode, you are prompted to + select a Toaster configuration file. + This configuration file is used to set up the initial + configuration values within the Toaster database + including the layer sources. + Three versions of the configuration file exist: + + + The first version of the file is found in the + conf directory of the + meta-yocto layer + (i.e. + meta-yocto/conf/toasterconf.json). + This version contains the default Yocto Project + configuration for Toaster. + You are prompted to select this file during the + Toaster set up process if you had cloned the + poky repository (i.e. + git://git.yoctoproject.org/poky). + + + The second version of the file is in the + conf directory of the + openembedded-core layer + (i.e. meta/conf/toasterconf.json). + This version contains the default OpenEmbedded + configuration for Toaster. + You are prompted to select this file during the + Toaster set up process if you had cloned the + openembedded-core repository + (i.e. + git://git.openembedded.org/openembedded-core). + + + The third version is a sample configuration + useful for when you want to set up a hosted + service in Build Mode. + You can find this version on the + File:Toasterconf.json.txt.patch + wiki page. + + + +
+ +
+ Edit the Configuration File + + + Edit the version of the + toasterconf.json file you + used to set up your Toaster instance. + In the file, you will find a section for layer sources + such as the following: + + "layersources": [ + { + "name": "Local Yocto Project", + "sourcetype": "local", + "apiurl": "../../", + "branches": ["HEAD", "master", "fido", "dizzy"], + "layers": [ + { + "name": "openembedded-core", + "local_path": "meta", + "vcs_url": "remote:origin", + "dirpath": "meta" + }, + { + "name": "meta-yocto", + "local_path": "meta-yocto", + "vcs_url": "remote:origin", + "dirpath": "meta-yocto" + }, + { + "name": "meta-yocto-bsp", + "local_path": "meta-yocto-bsp", + "vcs_url": "remote:origin", + "dirpath": "meta-yocto-bsp" + } + + ] + }, + { + "name": "OpenEmbedded", + "sourcetype": "layerindex", + "apiurl": "http://layers.openembedded.org/layerindex/api/", + "branches": ["master", "fido", "dizzy"] + }, + { + "name": "Imported layers", + "sourcetype": "imported", + "apiurl": "", + "branches": ["master", "fido", "dizzy", "HEAD"] + + } + ], + + You should add your own layer source to this section by + following the same format used for the "OpenEmbedded" + layer source shown above. + + + + Give your layer source a name, provide the URL of your + layer source API, use the source type "layerindex", and + indicate which branches from your layer source you want + to make available through Toaster. + For example, the OpenEmbedded layer source makes + available only its "master", "fido", and "dizzy" + branches. + + + + The branches must match the branch you + set when configuring your releases. + For example, if you configure one release in Toaster + by setting its branch to "branch-one" and you configure + another release in Toaster by setting its branch to + "branch-two", the branches in your layer source should + be "branch-one" and "branch-two" as well. + Doing so creates a connection between the releases + and the layer information from your layer source. + Thus, when users create a project with a given + release, they will see the appropriate layers from + your layer source. + This connection ensures that only layers that are + compatible with the selected project release can be + selected for building. + + + + Once you have added this information to the + toasterconf.json file, save your + changes. + + + + In a terminal window, navigate to the directory that + contains the Toaster database, which by default is the + root of the Yocto Project + Source Directory. + Once you are located in that directory, run the + "loadconf" command, which takes as + an argument the full path to the + toasterconf.json file you just edited. + For example, if you cloned the + poky repository and you edited the + meta-yocto/conf/toasterconf.json file, + you would type something like the following: + + $ bitbake/lib/toaster/manage.py loadconf /home/scottrif/poky/meta-yocto/conf/toasterconf.json + + After entering this command, you need to update the + information from the layer index to populate the + Toaster database with the information from your layer + index. + To do that, you should run the + "lsupdates" command from the directory + that contains the Toaster database. + Here is an example: + + $ bitbake/lib/toaster/manage.py lsupdates + + If Toaster can reach the API URL, you should see a message + telling you that Toaster is updating the layer source + information. + + + + Once the information has been updated, verify the new layer + information is available by using the Toaster web interface. + To do that, visit the "All compatible layers" page inside a + Toaster project. + The layers from your layer index should be listed there. + +
+
+
+
+ +
+ Releases + + + In the context of Toaster, the term "Release" refers to a set of + layers and a BitBake version the OpenEmbedded build system uses + to build something. + As shipped, Toaster is pre-configured with releases that + correspond to Yocto Project release branches. + However, you can modify, delete, and create new releases + according to your needs. + When you create a Toaster project using the web interface, + you are asked to choose a "Release." + This section provides some background information on releases. + + +
+ Pre-Configured Releases + + + As shipped, Toaster is configured to use a specific set of + releases. + Of course, you can always configure Toaster to use any + release. + For example, you might want your project to build against a + specific commit of any of the "out-of-the-box" releases. + Or, you might want your project to build against different + revisions of OpenEmbedded and BitBake. + + + + As shipped, Toaster is configured to work with the following + releases: + + Yocto Project 1.7 "Dizzy" or OpenEmbedded "Dizzy": + This release causes your Toaster projects to + build against the head of the dizzy branch at + + or + . + + Yocto Project 1.8 "Fido" or OpenEmbedded "Fido": + This release causes your Toaster projects to + build against the head of the fido branch at + + or + . + + Yocto Project "Master" or OpenEmbedded "Master": + This release causes your Toaster Projects to + build against the head of the master branch, which is + where active development takes place, at + + or + . + + Local Yocto Project or Local OpenEmbedded: + This release causes your Toaster Projects to + build against the head of the poky + or openembedded-core clone you + have local to the machine running Toaster. + + + +
+ +
+ What Makes Up a Release? + + + A release consists of the following: + + Name: + The name of the release (name). + This release name never appears in the the Toaster + web interface. + Consequently, a user never sees the release name. + + Description: + The textual description of the release + (description). + This description is what users encounter when creating + projects with the Toaster web interface. + When you configure your release, be sure to use + a description that sufficiently describes and is + understandable. + If Toaster has more than one release configured, the + release descriptions appear listed in a drop down menu + when a user creates a new project. + If Toaster has only one release configured, all + projects created using the web interface take that + release and the drop down menu does not display in the + Toaster web interface. + + BitBake: + The Bitbake version (bitbake) + used to build layers set in the current release. + This version is described by a name, a Git URL, a + branch in the Git URL, and a directory path in the + Git repository. + As an example, consider the following snippet from + a Toaster JSON configuration file. + This BitBake version uses the master branch from the + OpenEmbedded repository: + + "bitbake" : [ + { + "name": "master", + "giturl": "git://git.openembedded.org/bitbake", + "branch": "master", + "dirpath": "" + } + ] + + Here is more detail on each of the items that comprise + the BitBake version: + + Name: + A string + (name) used to refer to + the version of BitBake you are using with + Toaster. + This name is never exposed through Toaster. + + Git URL: + The URL giturl + for the BitBake Git repository cloned + for Toaster projects. + + Branch: + The Git branch, or revision, + (branch) of the BitBake + repository used with Toaster. + + Directory Path: + The sub-directory of the BitBake repository + (dirpath). + If the Git URL includes more than one + repository, you need to set this directory. + If the URL does not include more than a single + repository, you can set + dirpath to a null string + (i.e. ""). + + + + Branch: + The branch for the layer source + (branch) used with the release. + For example, for the OpenEmbedded layer source, the + "master", "fido", and "dizzy" branches are available. + + Default Layers: + The set of default layers + (defaultlayers) automatically + added to the project configuration when a project is + created. + You can download the recommended configuration + for this case from + . + + Layer Source Priorities + A specification of + layer source + priorities (layersourcepriority). + In order for Toaster to work as intended, the + "Imported layers" layer source should have the highest + priority, which means that layers manually imported by + users with the "Import layer" functionality will + always be visible and available for selection. + + Help Text: + Help text (helptext) that explains + what the release does when selected. + This help text appears below the release drop-down + menu when you create a Toaster project. + The help text should assist users in making the correct + decision regarding the release to use for a given + project. + + + + + + To summarize what comprises a release, consider the following + example from a Toaster JSON file. + The configuration names the release "master" and sets it to + use the master branch of the + openembedded-core layer. + The BitBake version used would be defined as shown earlier + in the previous list: + + "releases": [ + { + "name": "master", + "description": "OpenEmbedded master", + "bitbake": "master", + "branch": "master", + "defaultlayers": [ "openembedded-core" ], + "layersourcepriority": { "Imported layers": 99, "Local OpenEmbedded" : 10, "OpenEmbedded" : 0 }, + "helptext": "Toaster will run your builds using the OpenEmbedded master branch, where active development takes place. This is not a stable branch, so your builds might not work as expected." + } + ] + + +
+
+ +
+ JSON Files + + + If you are going to be using the Toaster in Build Mode, it must + be initially configured before use. + Configuration customizes layer source settings and Toaster defaults + for all users and is performed by the person responsible for + Toaster Configuration (i.e the Toaster Administrator). + The Toaster Administrator performs this configuration through the + admin web interface. + + + + To make it easier to initially start Toaster, you can import a + pre-defined configuration file using the + loadconf + command. + + The configuration file is a JSON-formatted text file with + specific fields that Toaster recognizes. + It is not a data dump from the database, so it cannot be + loaded directly in the database. + + + + + By convention, the supplied configuration files are named + toasterconf.json. + The Toaster Administrator can customize the file prior to loading + it into Toaster. + In the Local Toaster Build Mode, the system startup script + actively looks for compatible configuration files and prompts + you to select a file to load if it detects that the database has + not been configured. + + +
+ Configuration File Choices + + + Three versions of the configuration file exist: + + + The + meta-yocto/conf/toasterconf.json + in the conf directory of the + Yocto Project's meta-yocto layer. + This version contains the default Yocto Project + configuration for Toaster. + You are prompted to select this file during the Toaster + set up process if you cloned the + poky repository (i.e. + &YOCTO_GIT_URL;/poky). + + + The meta/conf/toasterconf.json + in the conf directory of the + OpenEmbedded's openembedded-core + layer. + This version contains the default OpenEmbedded + configuration for Toaster. + You are prompted to select this file during the Toaster + set up process if you had cloned the + openembedded-core repository (i.e. + git://git.openembedded.org/openembedded-core). + + + The Toasterconf.json.txt.patch + located on the + File:Toasterconf.json.txt.patch + wiki page. + This version of the file is useful as a sample + configuration for when you want to set up Toaster as a + hosted service in Build Mode. + + + +
+ +
+ File Structure + + + The toasterconf.json file consists of + easily readable areas: configuration, layer sources, BitBake, + default, and releases. + + +
+ Configuration Area + + + This area of the JSON file sets which variables are exposed + to users through the Toaster web interface. + Users can easily edit these variables. + + + + The variables you set here are displayed in the + "Configuration variables" page in Toaster. + Minimally, you should set the + MACHINE + variable, which appears to users as part of the project + page in Toaster. + + + + Here is the default config area: + + "config": { + "MACHINE" : "qemux86", + "DISTRO" : "poky", + "IMAGE_FSTYPES": "ext3 jffs2 tar.bz2", + "IMAGE_INSTALL_append": "", + "PACKAGE_CLASSES": "package_rpm", + "SDKMACHINE" : "x86_64" + }, + + +
+ +
+ Layer Sources Area + + + This area of the JSON file defines the + layer sources + Toaster uses. + Toaster reads layer information from layer sources. + Three types of layer sources exist that Toaster + recognizes: Local, LayerIndex, and Imported. + + + + The Local layer source reads layers from Git clones + available on your local drive. + Using a local layer source enables you to easily test + Toaster. + + If you are setting up a hosted version of Toaster, + it does not make sense to have a local layer source. + + + + + The LayerIndex layer source uses a REST API exposed by + instances of the Layer Index application (e.g the public + ) + to read layer data. + + + + The Imported layer source is reserved for layer data + manually introduced by the user or Toaster Administrator + through the GUI. + This layer source lets users import their own layers + and build them with Toaster. + You should not remove the imported layer source. + + + + Here is the default layersources area: + + "layersources": [ + { + "name": "Local Yocto Project", + "sourcetype": "local", + "apiurl": "../../", + "branches": ["HEAD", "master", "fido", "dizzy"], + "layers": [ + { + "name": "openembedded-core", + "local_path": "meta", + "vcs_url": "remote:origin", + "dirpath": "meta" + }, + { + "name": "meta-yocto", + "local_path": "meta-yocto", + "vcs_url": "remote:origin", + "dirpath": "meta-yocto" + }, + { + "name": "meta-yocto-bsp", + "local_path": "meta-yocto-bsp", + "vcs_url": "remote:origin", + "dirpath": "meta-yocto-bsp" + } + + ] + }, + { + "name": "OpenEmbedded", + "sourcetype": "layerindex", + "apiurl": "http://layers.openembedded.org/layerindex/api/", + "branches": ["master", "fido", "dizzy"] + }, + { + "name": "Imported layers", + "sourcetype": "imported", + "apiurl": "", + "branches": ["master", "fido", "dizzy", "HEAD"] + + } + ], + + +
+ +
+ BitBake Area + + + This area of the JSON file defines the version of + BitBake Toaster uses. + As shipped, Toaster is configured to recognize three + versions of BitBake: master, dizzy, and HEAD. + + + + Here is the default bitbake area: + + "bitbake" : [ + { + "name": "master", + "giturl": "remote:origin", + "branch": "master", + "dirpath": "bitbake" + }, + { + "name": "fido", + "giturl": "remote:origin", + "branch": "fido", + "dirpath": "bitbake" + }, + { + "name": "dizzy", + "giturl": "remote:origin", + "branch": "dizzy", + "dirpath": "bitbake" + }, + { + "name": "HEAD", + "giturl": "remote:origin", + "branch": "HEAD", + "dirpath": "bitbake" + } + ], + + +
+ +
+ Default Area + + + This area of the JSON file establishes a default + release used by Toaster. + As shipped, Toaster uses the "master" release. + + + + Here is the statement in the JSON file that establishes + the default release: + + "defaultrelease": "master", + + +
+ +
+ Releases Area + + + This area of the JSON file defines the versions of the + OpenEmbedded build system Toaster recognizes. + As shipped, Toaster is configured to work with the four + releases described in the + "Pre-Configured Releases" + section. + + + + Here is the default releases area: + + "releases": [ + { + "name": "master", + "description": "Yocto Project master", + "bitbake": "master", + "branch": "master", + "defaultlayers": [ "openembedded-core", "meta-yocto", "meta-yocto-bsp"], + "layersourcepriority": { "Imported layers": 99, "Local Yocto Project" : 10, "OpenEmbedded" : 0 }, + "helptext": "Toaster will run your builds using the tip of the <a href=\"http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/\">Yocto Project master branch</a>, where active development takes place. This is not a stable branch, so your builds might not work as expected." + }, + { + "name": "fido", + "description": "Yocto Project 1.8 Fido", + "bitbake": "fido", + "branch": "fido", + "defaultlayers": [ "openembedded-core", "meta-yocto", "meta-yocto-bsp"], + "layersourcepriority": { "Imported layers": 99, "Local Yocto Project" : 10, "OpenEmbedded" : 0 }, + "helptext": "Toaster will run your builds with the tip of the <a href=\"http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=fido\">Yocto Project 1.8 \"Fido\"</a> branch." + }, + { + "name": "dizzy", + "description": "Yocto Project 1.7 Dizzy", + "bitbake": "dizzy", + "branch": "dizzy", + "defaultlayers": [ "openembedded-core", "meta-yocto", "meta-yocto-bsp"], + "layersourcepriority": { "Imported layers": 99, "Local Yocto Project" : 10, "OpenEmbedded" : 0 }, + "helptext": "Toaster will run your builds with the tip of the <a href=\"http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?h=dizzy\">Yocto Project 1.7 \"Dizzy\"</a> branch." + }, + { + "name": "local", + "description": "Local Yocto Project", + "bitbake": "HEAD", + "branch": "HEAD", + "defaultlayers": [ "openembedded-core", "meta-yocto", "meta-yocto-bsp"], + "layersourcepriority": { "Imported layers": 99, "Local Yocto Project" : 10, "OpenEmbedded" : 0 }, + "helptext": "Toaster will run your builds with the version of the Yocto Project you have cloned or downloaded to your computer." + } + ] + + +
+
+
+ +
+ Useful Commands + + + In addition to the web user interface and the scripts that start + and stop Toaster, command-line commands exist through the + manage.py management script. + You can find general documentation on + manage.py at the + Django + site. + However, several manage.py commands have been + created that are specific to Toaster and are used to control + configuration and back-end tasks. + You can locate these commands in the + Source Directory + (e.g. poky) at + bitbake/lib/manage.py. + This section documents those commands. + + + When using manage.py commands given + a default configuration, you must be sure that your + working directory is set to the + Build Directory. + Using manage.py commands from the + Build Directory allows Toaster to find the + toaster.sqlite file, which is located + in the Build Directory. + + + + For non-default database configurations, it is possible + that you can use manage.py commands + from a directory other than the Build directory. + To do so, the + toastermain/settings.py file must be + configured to point to the correct database backend. + + + + +
+ <filename>buildslist</filename> + + + The buildslist command lists all builds + that Toaster has recorded. + Access the command as follows: + + $ bitbake/lib/toaster/manage.py buildslist + + The command returns a list, which includes numeric + identifications, of the builds that Toaster has recorded in the + current database. + + + + You need to run the buildslist command + first to identify existing builds in the database before + using the + builddelete + command. + For example, assuming default repository and build directory + names, the following two commands put you in the + Build Directory + and then return a list of the current builds, which would + be one in this case: + + $ cd $HOME/poky/build + $ python ../bitbake/lib/toaster/manage.py buildslist + 1: qemux86 poky core-image-minimal + + +
+ +
+ <filename>builddelete</filename> + + + The builddelete command deletes data + associated with a build. + Access the command as follows: + + $ bitbake/lib/toaster/manage.py builddelete build_id + + The command deletes all the build data for the specified + build_id. + This command is useful for removing old and unused data from + the database. + + + + Prior to running the builddelete + command, you need to get the ID associated with builds + by using the + buildslist + command. + +
+ +
+ <filename>perf</filename> + + + The perf command measures Toaster + performance. + Access the command as follows: + + $ bitbake/lib/toaster/manage.py perf + + The command is a sanity check that returns page loading + times in order to identify performance problems. + +
+ +
+ <filename>checksettings</filename> + + + The checksettings command verifies + existing Toaster settings. + Access the command as follows: + + $ bitbake/lib/toaster/manage.py checksettings + + In Managed Mode, Toaster uses settings that are based on the + database to configure the building tasks. + The checksettings command verifies that + the database settings are valid in the sense that they have + the minimal information needed to start a build. + + + + In order for the checksettings command + to work, the database must be correctly set up and not have + existing data. + To be sure the database is ready, you can run the following: + + $ bitbake/lib/toaster/mana​ge.py syncdb + $ bitbake/lib/toaster/mana​ge.py migrate orm + $ bitbake/lib/toaster/mana​ge.py migrate bldcontrol + + After running these commands, you can run the + checksettings command. + +
+ +
+ <filename>loadconf</filename> + + + The loadconf command loads a Toaster + configuration file (JSON file). + You must run this on a new database that does not have any + data. + Running this command on an existing database that has data + results in errors. + Access the command as follows: + + $ bitbake/lib/toaster/manage.py loadconf filepath + + The loadconf command configures a database + based on the supplied toasterconf.json + file. + For information on the toasterconf.json, + see the + "JSON Files" + section. + +
+ +
+ <filename>runbuilds</filename> + + + The runbuilds command launched + scheduled builds. + Access the command as follows: + + $ bitbake/lib/toaster/manage.py runbuilds + + The runbuilds command checks if + scheduled builds exist in the database and then launches them + per schedule. + The command returns after the builds start but before they + complete. + The "toasterui" build recorded updates the database when the + builds complete. + +
+
+ -- cgit v1.2.3-54-g00ecf