summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/ui/buildinfohelper.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: toaster: buildinfohelper Handle regex pathsbrian avery2016-11-031-1/+26
| | | | | | | | | | | | | | We were presuming that all the layer dependency information was of the form "^/path/to/layer" to we were just stripping the leading "^" off of the layer information when we were matching the layer priorities to the toaster database. This patch splits out the priorities layer match which gets a regex from the task/recipe match which is gets a path. (Bitbake rev: 82775c80d169266cc18ca2b2065a05c79dc6fbfc) Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: buildinfohelper: Use correct way to get message from ↵Michael Wood2016-10-061-6/+3
| | | | | | | | | | | | | | | LogMessage Use the correct method to get a message value from the LogMessage object rather than constructing it ourselves which is not recommended. This causes an exception when the msg contains a '%' such as when there are wildcards in file names (something2.%.bbappends) (Bitbake rev: 11b3b6a7087554d14a2812a9ae463dce740b879e) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: check if file existEd Bartosh2016-09-301-0/+3
| | | | | | | | | | | | | | | | | | Buildinfohelper assumes that all files mentioned in manifest exist in deploy/ directory, which is not always the case. Toaster crashes with OSError trying to call os.stat on non-existing file. Checking if file exists before processing it should fix this. [YOCTO #10185] (Bitbake rev: 54565e7ca84d2722a2454e7fa52cda564b28b527) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: buildinfohelper local layer don't construct path using git ↵Michael Wood2016-09-211-3/+5
| | | | | | | | | | | | | | | info When the layer is local source don't try and work out the location of the layer by using the git information (getGitCloneDirectory) [YOCTO #10199] (Bitbake rev: 3dfea5214d4bd006e26630e5024774ecb84ea527) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: unlock BuildEnvirnoment when build is doneEd Bartosh2016-09-211-3/+5
| | | | | | | | | | | | | | | | There is no need to lock build environment before changing build status as this operation is very fast. However, there is a need to unlock it after changing build status. Explicitly unlocked BuildEnvironment after build reaches final status SUCCEEDED, FAILED or CANCELLED. This should allow runbuilds process to pickup next build faster. (Bitbake rev: faa88272d656640c039572c5c8f3e6c56535b6f7) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: notify runbuilds when build status changesEd Bartosh2016-09-211-0/+4
| | | | | | | | | | | | | Called signal_runbuilds API when build is scheduled, cancelled or finished to notify runbuilds process about builds status change. [YOCTO #8918] (Bitbake rev: fe08f0fa4b328908e73695ebbceca87bc86a49f9) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: keep layer name in variable history pathDavid Reyna2016-09-081-1/+2
| | | | | | | | | | | | | | When converting variable history file names to relative paths, keep the layer directory's name so that the user can distinguish between conf files with the same name. [YOCTO #8188] (Bitbake rev: 59561d652af91c2099b735084f0e44275d68e637) Signed-off-by: David Reyna <david.reyna@windriver.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: discover kernel artifacts correctlyElliot Smith2016-09-021-4/+7
| | | | | | | | | | | | | | | | | | | | Because some image_license.manifest files contain multiple FILES lines, and because those lines can sometimes not contain a list of files (i.e. they look like "FILES:\n"), we were resetting the list of kernel artifacts when we hit the second "empty" line. Fix by ignoring any FILES line which doesn't list files, and by appending any files found in a valid FILES line, rather than overwriting the existing list. [YOCTO #10107] (Bitbake rev: 927ec3524625ac731326b3c1c1361c2a4d2bd9e1) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: buildinfohelper Add handling local layers (i.e. non-git) ↵Michael Wood2016-08-111-4/+23
| | | | | | | | | | | | layers Adds handling of the non-git layers to create and update the corresponding layer objects in Toaster. (Bitbake rev: 0a9b5d7d9655dbb09d458fc6e330e932f0f9dab6) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toasterui: ensure that the Build object is always availableElliot Smith2016-08-111-81/+115
| | | | | | | | | | | | | | | | | | | | | | | | | | Many of the methods in toasterui and buildinfohelper rely on the internal state of the buildinfohelper; in particular, they need a Build object to have been created on the buildinfohelper. If the creation of this Build object is tied to an event which may or may not occur, there's no guarantee that it will exist. This then causes assertion errors in those methods. To prevent this from happening, add an _ensure_build() method to buildinfohelper. This ensures that a minimal Build object is always available whenever it is needed, either by retrieving it from the BuildRequest or creating it; it also ensures that the Build object is up to date with whatever data is available on the bitbake server (DISTRO, MACHINE etc.). This method is then called by any other method which relies on a Build object being in the internal state, ensuring that the object is either available, or creating it. (Bitbake rev: 0990b4c73f194ec0be1762e4e48b1a525d8349fb) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: show progress of recipe parsing in recent builds areaElliot Smith2016-08-111-42/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | Modify buildinfohelper and toasterui so that they record the recipe parse progress (from ParseProgress events in bitbake) on the Build object. Note that because the Build object is now created at the point when ParseStarted occurs, it is necessary to set the build name to the empty string initially (hence the migration). The build name can be set when the build properly starts, i.e. at the BuildStarted event. Then use this additional data to determine whether a Build is in a "Parsing" state, and report this in the JSON API. This enables the most recent builds area to show the recipe parse progress. Add additional logic to update the progress bar if the progress for a build object changes. [YOCTO #9631] (Bitbake rev: f33d51d46d70e73e04e325807c1bc4eb68462f7b) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: orm Remove the layerindex specific up_branch fieldsMichael Wood2016-07-261-3/+3
| | | | | | | | | | | | | | | | | | We don't need to keep track of layerindex data in our database. And using branch==release is very confusing in the schema. Instead use the existing Release definition to keep track of which release a layer_version is for. Remove the Branch model and all references to it. Create a migration path to convert from up_branches to their corresponding releases. (Bitbake rev: f8f4cffe6fd371f3a7e63690c68f3fcb5dc1f297) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: add package manifest path to Target objectsElliot Smith2016-07-191-3/+21
| | | | | | | | | | | | | | | | | | | Store the path to the *.rootfs.manifest file for targets which generate images. A link to the package manifest is displayed in the build dashboard for targets which produce image files. Like the license manifest path, if a target would have produced the package manifest (but didn't, because it already existed), that path is copied from the target which did produce the package manifest. (Bitbake rev: 79b8e349a0da2ea6b97ad82daa5837e6dfffe0af) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: only record image files for tasks which make imagesElliot Smith2016-07-191-5/+15
| | | | | | | | | | | | | | | | | | | | | | | | If a target is built which is classified as an "image" target (e.g. "core-image-minimal"), Toaster reads the list of files in the image (from the files-in-image.txt file). However, Toaster continues to do this for builds which don't produce images, if the recipe providing the target is an image recipe. This can result in a list of files in the image being attached to a target which didn't produce an image (e.g. rootfs). When associating files with an image, ensure that only targets with a task which produces an image have "files in the image" associated with them. [YOCTO #9784] (Bitbake rev: 44375d0c2a88e0070b8067c9285b89c54eaf3152) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: improve scan for SDK artifactsElliot Smith2016-07-191-41/+134
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SDK artifacts were previously picked up by toaster.bbclass and notified to buildinfohelper (via toasterui). The artifacts were then added to the Build object, so that it wasn't clear which artifact went with which target; we were also unable to attach SDK artifacts to a Build if they had already been attached to a previous build. Now, toaster.bbclass just notifies the TOOLCHAIN_OUTPUTNAME when a populate_sdk* target completes. The scan is moved to buildinfohelper, where we search the SDK deploy directory for files matching TOOLCHAIN_OUTPUTNAME and attach them to targets (not builds). If an SDK file is not produced by a target, we now look for a similar, previously-run target which did produce artifacts. If there is one, we clone the SDK artifacts from that target onto the current one. This all means that we can show SDK artifacts by target, and should always get artifacts associated with a target, regardless of whether it really build them. This requires an additional model, TargetSDKFile, which tracks the size and path of SDK artifact files with respect to Target objects. [YOCTO #8556] (Bitbake rev: 5e650c611605507e1e0d1588cd5eb6535c2d34fc) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: fix retrieval of targetsElliot Smith2016-07-191-7/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When buildinfohelper records the targets for a build, it looks up any existing targets for a build and creates them if they are not present. This is because in the case of Toaster-triggered builds, the Target objects have already been created (inside triggerBuild()) and don't need to be recreated; but in the case of cli builds, the Target objects have to be created by buildinfohelper. The issue is that the code for retrieving an existing target for a build only looks for Targets with a matching target and build, e.g. Targets for build X with target "core-image-minimal". But it is perfectly legitimate to call bitbake with a command like "bitbake core-image-minimal:do_populate_sdk core-image-minimal:do_populate_sdk_ext". In such a case, the code which looks for matching targets finds two objects, as it doesn't filter by task. Add the task into the filter for the Target so that only one Target object is be returned. Note that a command line like "bitbake recipe:task recipe:task" will still cause an error as bitbake doesn't de-duplicate the command line arguments and will run the recipe:task combination twice. (Bitbake rev: 1c0a689fdaae6469d4afb98583161073d32ea50b) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: attach kernel artifacts to targetsElliot Smith2016-07-191-29/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The bzImage and modules files were previously attached to a build, rather than to the target which produced them. This meant it was not possible to determine which kernel artifact produced by a build came from which target; which in turn made it difficult to associate existing kernel artifact with targets when those targets didn't produce artifacts (e.g. if the same machine + target combination was built again and didn't produce a bzImage or modules file because those files already existed). By associating kernel artifacts with the target (via a new TargetArtifactFile model), we make it possible to find all the artifacts for a given machine + target combination. Then, in cases where a build is completed but its targets don't produce any artifacts, we can find a previous Target object with the same machine + target and copy its artifacts to the targets for a just-completed build. Note that this doesn't cover SDK artifacts yet, which are still retrieved in toaster.bbclass and show up as "Other artifacts", lumped together for the whole build rather than by target. [YOCTO #8556] (Bitbake rev: 9b151416e428c2565a27d89116439f9a8d578e3d) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: do image and artifact scan on BuildCompletedElliot Smith2016-07-191-22/+179
| | | | | | | | | | | | | | | | Move the image and artifact scan code from toaster.bbclass and consolidate its logic with the existing logic in buildinfohelper. Remove handler setup for events which used to be fired from toaster.bbclass but which are now handled directly by buildinfohelper. [YOCTO #8556] (Bitbake rev: f0085cd554604cfff4a3f40a34825fbb6878004f) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: bavery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: ensure task datetimes are timezone-awareElliot Smith2016-07-081-1/+1
| | | | | | | | | | | | | | | | | | | | | When using toaster-eventreplay to run a bitbake event file through toasterui/buildinfohelper, errors occur when the tasks are updated with buildstats info: RuntimeWarning: DateTimeField Task.started received a naive datetime (2016-07-06 09:15:22.070000) while time zone support is active. This is because a method in buildinfohelper returns a naive datetime, but Django is expecting timezone-aware datetimes. Ensure that datetimes used to set the started/ended times on tasks are converted to timezone-aware datetimes. (Bitbake rev: df9f4337bec87024ea6a43138c6080a755eb7fab) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: replace viewkeys() -> keys()Ed Bartosh2016-06-151-1/+1
| | | | | | | | | | | | Python 3 doesn't have dict.viewkeys method, renaming to keys(). [YOCTO #9584] (Bitbake rev: 8ae2fc26fc7c4f42817dece62b9a59fbda27c0b0) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "buildinfohelper: work around unicode exceptions"Ed Bartosh2016-06-021-6/+4
| | | | | | | | | | | This commit causes buildinfohelper to crash when run on python 3 as python 3 doesn't have unicode builtin function. This reverts commit 7a309d964a86f3474eaab7df6d438ed797f935c0. (Bitbake rev: 750ca5c8d5a25fc519b75c56352dec7823c7e240) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Remove DATABASE_URL being passed around as an environment varMichael Wood2016-05-201-10/+10
| | | | | | | | | | | | | | | | | | We don't need to pass the DATABASE_URL around and read it back if we setup the django framework in the correct way. We make the default sqlite database path a full path so that the database isn't being assumed to be in CWD. Also add some more useful comments on the database settings. This is preparation work to migrate the build tests and be able to trigger builds on differently configured databases. (Bitbake rev: 973c740404ca6a09feea250d3433075995067fe0) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: add method to set current build as CANCELLEDElliot Smith2016-05-171-0/+10
| | | | | | | | | | | | | This will be used from toasterui to cancel the current command-line build when a keyboard interrupt is captured. [YOCTO #8515] (Bitbake rev: 1486c770327b53bb5e04baa5f3ea26d8154aed63) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Update logger.warn() -> logger.warning()Richard Purdie2016-05-111-10/+10
| | | | | | | | | | python deprecated logger.warn() in favour of logger.warning(). This is only used in bitbake code so we may as well just translate everything to avoid warnings under python 3. Its safe for python 2.7. (Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: work around unicode exceptionsJoshua Lock2016-04-061-4/+6
| | | | | | | | | | | | | | | | | | | We have been seeing UnicodeDecodeErrors when handling the ImagePkgList MetadataEvent in ORMWrapper's save_target_file_information() if the event includes filenames that include non-ASCII characters. In the short term work around this by converting paths to the unicode type when passing them to Django's ORM. This is a bit of a hack but it's too late in the cycle to do anything more invasive. [YOCTO #9142] (Bitbake rev: f50fff03b3de02e73a3cc2eb9935f7c345dbddc4) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toasterui: update build in internal stateEd Bartosh2016-04-061-0/+1
| | | | | | | | | | | | | | | | | | | | | | buildinfohelper stores current Build object in its internal state. Any changes to Build object will be lost if internal state is not updated as current buildinfohelper code saves Build object from internal state when build is completed. This bug causes incorrect build state when build is cancelled. Updating internal state should fix it. Note, that this commit updates internal state after status of the build is changed to Build.CANCELLED. There are several other places in the code where Build object is updated without updating internal state. They should be carefully analyzed and fixed. (Bitbake rev: d056cf40fc55530cb1736aedfb9a3c355884991e) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: fix KeyErrorEd Bartosh2016-04-061-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | When bitbake doesn't need to build anything it still sends ImagePkgList event with empty 'pkgdata', 'imgdata' and 'filedata' fields. This causes crash in buildinfohelper code as it's assumed that above mentioned fields always have data keyed by build target: ERROR: u'core-image-minimal' Traceback (most recent call last): File "toasterui.py", line 423, in main buildinfohelper.store_target_package_data(event) File "buildinfohelper.py", line 1218, in store_target_package_data imgdata = BuildInfoHelper._get_data_from_event(event)['imgdata'][target.target] KeyError: u'core-image-minimal' Fixed this by using dict.get method with empty dictionary as default return value instead of trying to get value without checking if target key is in the data. (Bitbake rev: c39cc463e6d9594bf2c5ac8bb74e834f6f2cf7c8) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: Add handler for cancelling a buildMichael Wood2016-04-061-1/+18
| | | | | | | | | | | When a build is cancelled the build (action) is complete if it has been caused the request being cancelled then update the build outcome accordingly. (Bitbake rev: d94d12914d351bf560b06d6f4e45c294b04ecaa3) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: fix jethro buildEd Bartosh2016-04-061-8/+10
| | | | | | | | | | | | | | | The keys 'started', 'ended', 'cpu_time_user', 'disk_io_read' and 'disk_io_write' were added to the event recently, so they don't exist in the events generated by bitbake server from older releases. Checking if task_to_update structure has these keys before using them should fix build of older releases. (Bitbake rev: 79611d0ea742263074fbb0bf5f1e39df75fd9f55) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: improve handling of providermapEd Bartosh2016-04-061-2/+1
| | | | | | | | | | | | | DepTreeGenerated event doesn't contain 'providermap' data in jethro. Modified buildinfohelper to handle events without this data. This should make it possible to handle jethro events coming from jethro bitbake server by the latest buildinfohelper. (Bitbake rev: f6dcb1c9967f042beae024146781cb8235a9e1f2) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toasterui: add brbe parameter to buildinfohelperEd Bartosh2016-04-061-2/+2
| | | | | | | | | | | | | | | | | | | In current toaster code BRBE(build request:build environment) value is passed from toaster to buildinfohelper through the 'SetBRBE' event. Passing it through environment variable is easier as it doesn't involve rpc communication between toaster and bitbake server. It also eliminates the need in running bitbake observer process. Added parameter 'brbe' to BuildInfoHelper.__init__ Used environment variable TOASTER_BRBE to set brbe for buildinfohelper object. (Bitbake rev: a0c8e2b309055e5927a8ff729d292ccaa69d0575) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: buildinfohelper Add additional metadata to the built layerMichael Wood2016-04-031-11/+23
| | | | | | | | | | | | | | | | | Add additional metadata to the layer created for build history to be able to identify the layer and recipe later on. Specifically this is the branch and release to which the recipe and layer are associated with enabling differentiation of two recipes which are local release and master and 'master' release. [YOCTO #8528] [YOCTO #8545] (Bitbake rev: 3deebd887bddbbd02fd9829a180aab494b1af7c4) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: orm Add a constant for the CustomImageRecipe's layer nameMichael Wood2016-03-231-2/+2
| | | | | | | | | | | Use a constant to define the name for the toaster custom images layer; this constant is then used to identify this layer in various places. (Bitbake rev: 2540969ec71612af7f9041cadcc401513e9b357b) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: rework task buildstats storage and displayElliot Smith2016-03-091-39/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The data available from buildstats is now more fine grained than previously, so take advantage of that to enrich the data we save against tasks: * Store the CPU usage for user and system separately, and display them separately. * Disk IO is now measured in bytes, not ms. Also store the read/write bytes separately. * Store started and ended times, as well as elapsed_time. This will enable future features such as showing which tasks were running at a particular point in the build. There was also a problem with how we were looking up the Task object, which meant that the buildstats were being added to new tasks which weren't correctly associated with the build. Fix how we look up the Task (only looking for tasks which match the build, and the task and recipe names in the build stats data) so the build stats are associated with the correct task. [YOCTO #8842] (Bitbake rev: efa6f915566b979bdbad233ae195b413cef1b8da) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: unset brbe variable when build finishesElliot Smith2016-02-111-7/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The brbe variable is set on the bitbake server when the Toaster UI starts. This enables Toaster to associate events with the correct build and build environment. However, the brbe variable is also used when a build starts to identify whether a new build needs to be created, or an existing one looked up. This causes a bug for command-line builds which happen after a Toaster-triggered build: because the brbe variable is never unset on the server or the buildinfohelper, the new command-line build events are treated as originating from the previous build. Ensure the brbe variable is reset when the buildinfohelper "closes" a build, so that each build then either sets the brbe variable (Toaster-triggered builds) or leaves it blank (command-line builds). Also modify the localhostbecontroller so that the brbe variable is not set on the server and not looked up from the server. This ensures that it is only set when the triggerBuild() method is called, and that it remains as None for command-line builds. [YOCTO #9021] (Bitbake rev: 4a6a8d0074f62208d843b06344be31ae73d9b745) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: buildinfohelper CustomImagePackage update dependency infoMichael Wood2016-02-101-15/+4
| | | | | | | | | | | Instead of keeping the original dependency information for the pool of CustomImagePackage reset it with each new build. (Bitbake rev: a0b97ffc7a468bad081ce3276c74728bf6830250) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: buildinfohelper Add the concept of CustomImagePackageMichael Wood2016-02-101-16/+89
| | | | | | | | | | | | | | | | | This adds the concept of CustomImagePackage this is similar to the way layers and recipes work in that we have a set of data which is part of the build history and a set of data which is part of the configuration data that toaster uses to guide people in configuring their project. We create a set of built_packages for every build but only create a package for configuration purposes if we don't already have one, so that the CustomImagePackage only ever contains a unique list of packages that are available to be added and removed from a CustomImageRecipe. (Bitbake rev: f81bb65883baa6c0f8a4d48a4de3291a10543992) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: add provides info to the dbEd Bartosh2016-01-111-3/+8
| | | | | | | | | | | | | | | | Added new entries to Provides model and link them to Recipe_Dependency using 'via' field. This data will be used by Toaster UI to show 'Provides:' information for the recipes. [YOCTO #6169] (Bitbake rev: 336ddc8df611d4c8f1c3d3a06d0a85bb544c38bc) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: use providermapEd Bartosh2016-01-111-8/+16
| | | | | | | | | | | | | | Used providermap in store_dependency_information function to find virtual dependencies. This should fix annoying warnings "stpd: KeyError saving recipe dependency" [YOCTO #6169] (Bitbake rev: 85c416ca338c886db6e79651e44727482df9fb07) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: buildinfohelper: Update for buildstats layout changeRichard Purdie2015-12-181-5/+3
| | | | | | | | This updates buildinfo helper for the recent buildstats layout change (Bitbake rev: 30311bbe667e9f22de17fae00ff58da06a7c3e23) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Start Django machinery for database accessElliot Smith2015-12-141-0/+3
| | | | | | | | | | | | | | | | | | buildinfohelper, with the new import paths for our Django models and Django 1.8, was not getting an active connection to the database. In buildinfohelper, call django.setup() explicitly to make sure that the database connection is ready and models can be queried and saved. [YOCTO #8364] (Bitbake rev: 671aaab8cb7c494cd5c7621b45a6f41a203d8bb5) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Create default project with get_or_create* methodElliot Smith2015-12-141-2/+7
| | | | | | | | | | | | | | | | Rather than maintain data as part of the migrations (as was done for the default project previously), create the default (cli builds) project on demand as a by-product of getting it from the database. [YOCTO #8364] (Bitbake rev: 5fd8e90ab9b81d1bd0d301bc1c91228ecbbea74b) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Fix references to app pathsEd Bartosh2015-12-141-8/+8
| | | | | | | | | | | | | | | | | | | | | | The buildinfohelper runs outside of Django, but still needs access to our Django app classes. Previously, the imports referenced the toaster.* app, which worked fine. But in Django 1.8, this causes an error about the same module being loaded multiple times from different paths. Change the paths to our Django modules so they don't cause this error to be thrown. We can do this as we've added our application libraries to sys.path in the buildinfohelper anyway. [YOCTO #8364] (Bitbake rev: 070da64cf32c32b5ffc34d611b463c3a3960b419) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: get rid of complicated heuristicsEd Bartosh2015-12-071-38/+9
| | | | | | | | | | | | | | | | | | Removed buildinfohelper code which was trying to guess layer version of the recipe using build request information. The code caused creation of duplicated recipes as it resulted in layer version from layer index instead of returning build layer version. As a result of this Toaster UI was not showing any information about recipes. Default approach used to find layer version seems to work much better as it finds proper layer version. Now toaster will use it as the only way to find layer version. (Bitbake rev: 101690bda7ad55dc0657483233c90c374713755b) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: buildinfohelper Broaden the toaster created recipe data caseMichael Wood2015-12-071-4/+3
| | | | | | | | | | | | | | | | When build information is the 'original' source of the information we need to return the recipe that was created rather than the copy of the recipe that is taken for keeping build history. We do this already for command line triggered builds, but we also have this case for custom images. We can simply check if the built_recipe exists instead of special casing this. (Bitbake rev: 9a8653bf602b2111dee7ee6a459682a68a695b22) Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: do not create duplicate HelpText objectsEd Bartosh2015-12-071-7/+3
| | | | | | | | | | | | | | | | | buildinfohelper code expects only one HelpText object per build/variable/description. Current code creates more than one such an object, which causes toastergui to crash with this exception: MultipleObjectsReturned: get() returned more than one HelpText -- it returned 2! Used git_or_create API to ensure that only one HelpText object is created. (Bitbake rev: e9b46803eb6f1f4044919abf90c8aeb3536e73ed) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: update brbe and project attributesEd Bartosh2015-12-071-0/+6
| | | | | | | | | | | | Updated attributes of buildinfohelper object as they can be changed for every build. For example brbe is set by runbuilds for every build triggered by Toaster UI. (Bitbake rev: ea3bc8d01704dc64f6cb7b4f5fe66c312a575174) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toasterui: Create per-build logsElliot Smith2015-10-291-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rather than relying on bug 8411, which is conveniently creating separate log files for each of our builds, create our own log file for each build. The log files are created in the same tmp directories that bitbake users, but are timestamped to the millisecond to avoid name collisions. Each log file is opened on a ParseStarted event (for builds triggered by Toaster) or BuildStarted event (for builds on the command line: Toaster doesn't get the ParseStarted event for command-line builds). The log file is closed on the BuildCompleted event, or if the build fails. Because we start logging on ParseStarted for Toaster builds, we're able to capture the "Build Configuration" section which bitbake writes to output. [YOCTO #8373] (Bitbake rev: 7974203cd8bc66dff1fcc55f8723dedefaf72840) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: add get_or_create_targets APIEd Bartosh2015-10-271-15/+21
| | | | | | | | | | | | | | | | | Target objects are created before the build if build is started from UI in build mode. However, in analysis mode Target objects don't exist and need to be created using information from bitbake events. Added new API call get_or_create_targets to retrive existing target objects or create them if they don't exist yet. (Bitbake rev: ef69be31d133696bde54605f5a18da660099734c) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: brian avery <avery.brian@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Record critical errorsElliot Smith2015-10-161-1/+4
| | | | | | | | | | | | | | | | | | | | Critical errors (where a build failed for reasons of misconfiguration, such as a machine being specified which is not in a project's layers) were being ignored (only log records up to ERROR level were being logged to Toaster's db). This meant that the build would fail but would not correctly report why. Add support for CRITICAL error levels to the LogMessage model, include errors at this level in the errors property for a build, and show errors at this level in the build dashboard. [YOCTO #8320] (Bitbake rev: b6eacbca9cacb607de864ab7d093deb296da8226) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-11 20:58:53 +0000