summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/cache.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: implement checksums for local files in SRC_URIPaul Eggleton2012-05-231-3/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | Gathers a list of paths to have checksums calculated at parse time, and processes these when calculating task hashes. Checksums are cached with the file's current mtime. Thus, changing any local file in SRC_URI will now cause the do_fetch taskhash to change, thus forcing a rebuild. This change adds very roughly about an 8% increase in parse time (a few seconds) and maybe a few seconds during runqueue generation, so a fairly moderate performance hit. Note that since paths are resolved at parse time, this will not force a rebuild when files are introduced which would cause that resolved path to be different - for example, where a machine-specific version of a file was added without otherwise changing the recipe. This will need to be handled in a future update. Code to hook this into the signature generator was courtesy of Richard Purdie <richard.purdie@linuxfoundation.org>. Implements [YOCTO #2044]. (Bitbake rev: c993b7c457f8b7776e8a5dff253bfa0724bc2cae) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: refactor out codeparser cache into a separate classPaul Eggleton2012-05-231-2/+114
| | | | | | | | | | We want to be able to reuse most this functionality for the file checksum cache. (Bitbake rev: 0fe3cb1438d297f90dd0fc6b26362ecbff75c76d) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: change for adding progress bar in Hob2.Shane Wang2012-02-231-1/+1
| | | | | | | | | | | | | The changes include: - Clean some events in event.py - Fire essential events for Hob2 to handle with more information. - knotty changes (Bitbake rev: 9ede881620c501574f014e600cea6947ea908ac2) Signed-off-by: Shane Wang <shane.wang@intel.com> Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache: Use configuration's hash value to validate cacheDongxiao Xu2012-02-231-20/+12
| | | | | | | | | | | | | | | Previously we use the file time stamp to judge if a cache is valid. Here this commit introduce a new method, which calculates the total hash value for a certain configuration's key/value paris, and tag it into cache filename, for example, bb_cache.dat.xxxyyyzzz. This mechanism also ensures the cache's correctness if user dynamically setting variables from some frontend GUI, like HOB. (Bitbake rev: 1c1df03a6c4717bfd5faab144c4f8bbfcbae0b57) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add description into the cache and clean up cache duplicationShane Wang2012-02-021-9/+0
| | | | | | | | | | | | | With the addition of new cache domains, the summary, license and section information is no longer requred in the core cache since its only used by the hob UI. This patch removes the duplicated entries. It also adds the DESCRIPTION field to the cache for the benefit of hob2. (Bitbake rev: 33ffb2e99825cb643b148b3462c2d4cf33ff5f58) Signed-off-by: Shane Wang <shane.wang@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Update users of getVar/setVar to use the data store functions directlyRichard Purdie2011-11-271-3/+2
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Add FAKEROOTNOENV variableRichard Purdie2011-11-101-0/+3
| | | | | | | | | | | | | | | | | Add a FAKEROOTNOENV which does the opposite of the FAKEROOTENV variable and is data loaded into the environment for tasks without the fakeroot flag. The intent here is to provide a way to control the environment when we aren't needing fakeroot context which allows us to unload the preload from memory entirely and gain a valuable speedup. I'm not 100% happy with needing this at the bitbake level, particularly with the cache hit but it does give a valuable speedup. (Bitbake rev: 05c29ab0b2ad3c521414cabb6a92bca15c6e919c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bb/cache: rename confusing variableJoshua Lock2011-08-111-3/+3
| | | | | | | | | | | The bNeedUpdate variable doesn't reflect its purpose, and doesn't match coding style (type encoded in variable name, camel case) - rename to cache_ok. (Bitbake rev: cc4158db215493ac270e27f92d977bcbe337f3db) Signed-off-by: Joshua Lock <josh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: show more information for NoProvider errorsPaul Eggleton2011-07-271-1/+2
| | | | | | | | | | | | | | | | | | "Nothing PROVIDES" errors often come up when a recipe has been skipped for some reason, and therefore it is useful to print out that reason information when showing the error so that the user understands why the error has occurred. Given that we already feed the reason information into the skiplist for various situations (COMMERCIAL_LICENSE, COMPATIBLE_MACHINE etc.) this should now output a useful error message for skipped recipes. Fixes [YOCTO #846], [YOCTO #1127] (Bitbake rev: 6765218430e31c165888f26fbc75023c89a6eab2) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/cache: allow class names with arguments to be specifiedPaul Eggleton2011-07-251-2/+3
| | | | | | | | | This ensures we understand e.g. "virtual:multilib:lib64:pn" for multilib. (Bitbake rev: 0b9dee799cfe2229ec62c257dda36bd6c15d34a9) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache: fix remnant broken 'info' reference from recent cache changesChristopher Larson2011-07-211-1/+1
| | | | | | | (Bitbake rev: 807fb0b3bbad5163fdab24d379c2deba8f0b4f13) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache.py: Ensure additional .bbappend files are accounted forRichard Purdie2011-07-071-5/+12
| | | | | | | | | | | | Currently if a user adds a new .bbappend file to the system, the cache still thinks the cached data is valid. This code fixes that to ensure additions and changed in append application order are accounted for. [YOCTO #1091] (Bitbake rev: 54fe91fe96aaae47c40077c5f441c79da71da777) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Add task specific stamp file supportRobert Yang2011-06-281-0/+3
| | | | | | | | | | | | | | | | | | This patch, based on proof of concept code from Richard adds code to bitbake to allow individual tasks to optionally specify their stamp file using the stamp-base flag. This takes the same form as the STAMP variable but can be specified on a per task basis. Code is also added to runqueue to ensure that if two tasks share the same stamp file, only one will be executed at once. A significant usecase for this code is to share source code (${S}) between recipes where separate build directories (${B}) are used. (Bitbake rev: 41bef02bef8379590ba012319aebe05068a8081e) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/cooker: Fix -b option by ensuring the empty cache structure is presentRichard Purdie2011-06-131-1/+0
| | | | | | (Bitbake rev: 1430a36e81737bd92245042710eb9d6ad8b6f1a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache.py: fix bitbake -s commandScott Garman2011-06-091-1/+1
| | | | | | | | | | | | This uses the correct index of self.pn when setting up cachedata's pkg_pn, fixing the output of bitbake -s. This fixes bug [YOCTO #1149]. (Bitbake rev: d000493c09ac5c1dcbab22d3a91296a9cb194ac0) Signed-off-by: Scott Garman <scott.a.garman@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* track skipped packagesPaul Eggleton2011-06-091-3/+6
| | | | | | | | | | | | | Add skiplist to cooker that allows access to the list of packages skipped via SkipPackage (this includes COMPATIBLE_MACHINE, INCOMPATIBLE_LICENSE, etc.) This can be used to enhance error reporting. (From Poky rev: 6c12b7b1099c77b87d4431d55e949cf7c5f52ded) (Bitbake rev: 7d2363f35350be27a33f568c23eb07fcd3d27e53) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache: Implement multiple extra cache fields request supportLiping Ke2011-06-071-58/+119
| | | | | | | | | | | | | | This patch is to support extra cache. If user needs to request extra cache fields besides CoreRecipeInfo fields, just add a new XXXRecipeInfo class definition as Hob Does. Currently supported Extra RecipeInfo name is an array. We can support multiple extra cache fields at the same time besides CoreRecipeInfo which is needed by all clients. (Bitbake rev: 077657e50ad032c0fa876bf54e9802af2686e0fb) Signed-off-by: Liping Ke <liping.ke@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Introduce new param caches_array into Cache impl.Liping Ke2011-06-071-5/+10
| | | | | | | | | | | | | | When using hob ui interface, we need extra cache fields. We will save ui required extra cache fields into a separate cache file. This patch introduce this caches_array parameter. It will be used in the extra cache implementation (following patch). Caches_array at least contains CoreRecipeInfo. If users need extra cache fields support, such as 'hob', caches_array will contain more relevant elements such as HobRecipeInfo. (Bitbake rev: d50389ae692377c957afec7c846fc2ce2c070a09) Signed-off-by: Liping Ke <liping.ke@intel.com>
* cache.py: Refactory Current Cache implementationLiping Ke2011-06-071-193/+165
| | | | | | | | | | | | | | | This patch is for refactorying current cache implementation, the main reason is for introducing extra cache fields requests for image creator as well as other users. The refactory parts include: Move cache data retrieve methods out of Cache Data Fields Definition. Since this retrieve methods will be shared for both CoreRecipeInfo as well as the new introduced extra RecipeInfo in the following patches. (Bitbake rev: f0f53506926a3f79181796dde177f11f0a396b75) Signed-off-by: Liping Ke <liping.ke@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/cache.py: Ensure skipped recipes make it into the cache to avoid ↵Richard Purdie2011-05-051-3/+8
| | | | | | reparsing Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/ast.py: Only run finalise() for the specified variantRichard Purdie2011-05-051-0/+1
| | | | | | | Allows the heavy finalise function to only be run for the case we're interested in when running tasks, saving some processing time. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* universe target: add a new target to collect all recipe targetsSaul Wold2011-05-031-0/+6
| | | | | | | | | This new universe target is not intended to be used for compiling or building everything, it use is for sanity checking and other tasks that need to find all targets. This does not exclude any broken or virtual targets. Signed-off-by: Saul Wold <sgw@linux.intel.com>
* Revert "bitbake/gcc: Enable a shared common source tree"Richard Purdie2011-03-291-4/+0
| | | | | This reverts commit 12b163dbd81cafafec1ebe3c4039c65af60ee261 as it was an unintended change for master
* bitbake/gcc: Enable a shared common source treeRichard Purdie2011-03-281-0/+4
| | | | | | | | | | | | | | | | | | | | | This patch is a quick proof of concept to show how source code could be shared between recipes which use ${B} to have a separate build directory compared to source directory ${S}. Issues: a) gcc uses sed and creates config files against ${S} which means the directory should not be shared. Need to change the way that works. b) Could be extended to cover eglibc except there is a patch applied against nativesdk versions which again makes the source incompatible. c) Need to clean up the layout in work-shared and make a directory level deeper to ensure patch separation. d) clean task does not remove stamps Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/cache/runqueue.py: Move workload for recipe parsing to the child processRichard Purdie2011-02-281-1/+9
| | | | | | | | | | | | | | | Parsing the recipe in the parent before forking off the child worker can mean the parent doesn't hit the idle loop and becomes a bottleneck when lauching many short lived processes. The reason we need this in the parent is to figure out the fakeroot environmental options. To address this, add the fakeroot variables to the cache and move recipe loadData into the child task. For a poky-image-sato build this results in about a 2 minute speedup (1.8%). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/cache: bump cache version after recent changesJoshua Lock2011-02-241-1/+1
| | | | Signed-off-by: Joshua Lock <josh@linux.intel.com>
* bitbake/[cooker|cache]: cache summary, license and group. Add to targets treeJoshua Lock2011-02-241-0/+12
| | | | | | | | | Add summary, license and group metadata to RecipeInfo and the cache. Unfortunately this impacts parse speed but gives us a much richer set of metadata to expose through UI's which can be accessed via the generateTargetsTree command. Signed-off-by: Joshua Lock <josh@linux.intel.com>
* bitbake/cache: store a list of inherited files in the cacheJoshua Lock2011-02-241-0/+5
| | | | Signed-off-by: Joshua Lock <josh@linux.intel.com>
* cache: fix docstring usageChris Larson2011-02-101-7/+2
| | | | | | | | | Use comments instead of docstrings where appropriate in CacheData. (Bitbake rev: 088d516e02bb2b4ce8a50bbaa967e944c46e620b) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fix parameter order for flaglist()Dongxiao Xu2011-01-201-2/+2
| | | | Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
* bitbake: Introduce stamp-extra-info task flag into stamp filenamesDongxiao Xu2011-01-181-1/+10
| | | | | | | | | | | For certain tasks, we need additional information in build stamp file other than the task name and file name. stamp-extra-info is introduced as a task flag which is appended to the stamp file name. [Code simplifcations/tweaks from Richard] Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache: don't expand variables for skipped recipesChris Larson2011-01-101-10/+13
| | | | | | | | | | | Errors can result from these expansions, but for skipped recipes, we shouldn't care about those failures. This fixes the same issue which Richard Purdie fixed in poky, commit 847b717. (Bitbake rev: 96ee6840010c1ae1080e6bf7ff0f4eb2d361e84b) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/cache.py: Expansion of variables in skipped recipes should not be ↵Richard Purdie2011-01-071-1/+10
| | | | | | attempted as they can trigger errors Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cache: defer marking fn as cleanBernhard Reutner-Fischer2011-01-041-4/+1
| | | | | | | | | | | | Only mark fn as clean if it is clean. This saves us from removing (prematurely added) fn from our clean set and saves me a few percent of runtime (and misleading debugging output from remove()). (Bitbake rev: 884365228fcaac07421ac1440d4946693fb628c5) Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: Sync a load of whitespace and other non-functionality changes with ↵Richard Purdie2011-01-041-4/+5
| | | | | | bitbake uptream Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Show the user progress when loading the cacheBob Foerster2011-01-041-0/+17
| | | | | | | | (Bitbake rev: bdd7813d8eecf7b6b636322e748ca6bf69118513) Signed-off-by: Bob Foerster <robert@erafx.com> Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: change to more incremental formatChris Larson2011-01-041-22/+33
| | | | | | | (Bitbake rev: 4fe4ffbef3885887c97eebe021edc3f23feab9ea) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: ensure 'pn' is included in the pkgvarsChris Larson2011-01-041-12/+12
| | | | | | | (Bitbake rev: cefc28a2aa1e5703119dacfd885c8c159a1e47a3) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cooker: stop loading the cache for -bChris Larson2011-01-041-1/+1
| | | | | | | | | | | Previously, the cache was actually being loaded from disk twice whenever using -b or -e -b. This also moves the bb_cache instance into the CookerParser, as it's not needed by the cooker itself at all. (Bitbake rev: dd0ec2f7b18e2a9ab06c499b775670516bd06ac8) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: don't add info to cache if cache is disabledChris Larson2011-01-041-0/+3
| | | | | | | (Bitbake rev: f12bb303f3d86a68d0b3dda1112dd654b9251704) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Simplify cache syncingChris Larson2011-01-041-22/+13
| | | | | | | | | | | Rather than adding nocache items to the cache, then copying the cache and removing them to sync it, don't add them in the first place. Also use 'with' for the cachefile. (Bitbake rev: 343b6f6255ad020c39e30742175a241f0859a5a6) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Implement parallel parsing supportChris Larson2011-01-041-39/+59
| | | | | | | | | | | This utilizes python's multiprocessing module. The default number of threads to be used is the same as the number of available processor cores, however, you can manually set this with the BB_NUMBER_PARSE_THREADS variable. (Bitbake rev: c7b3ec819549e51e438d293969e205883fee725f) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: don't write out the cache unnecessarilyChris Larson2011-01-041-1/+5
| | | | | | | | | | | If the only recipes's we reparsed this run were those flagged as not to be cached, there's no point in re-saving the cache, as those items won't be included anyway. (Bitbake rev: 1e0c4dbcbec886a30b89f8b4bb365c3c927ef609) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: create and use a RecipeInfo classChris Larson2011-01-041-173/+196
| | | | | | | | | | | | | | | | | | This class holds the particular pieces of information about a recipe which are needed for runqueue to do its job. By using it, I think we improve code clarity, reduce method sizes, reduce overuse of primitive types, and prepare for parallel parsing. In addition, this ditches the leaky abstraction whereby bb.cache attempted to hide the difference between cached data and a full recipe parse. This was a remnant from the way things used to be done, and the code using it had to know the difference anyway. If we choose to reimplement caching of the full recipes, we can do it in bb.parse, in a completely transparent way. (Bitbake rev: 992cc252452221f5f23575e50eb67528b2838fdb) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: make loadDataFull a classmethodChris Larson2011-01-041-4/+5
| | | | | | | (Bitbake rev: 16a9d689e61c35cfca94bbecd9772eab3c5072ba) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: Add debug msg for a nonexistant dep fileChris Larson2011-01-041-0/+2
| | | | | | | | | | If a recipe depends on a file, and that file is out of date, we show a message, but if that file was removed, we do not, until now. (Bitbake rev: 67984ba0ac2db79874541bc031f2e3e9ff7a6c32) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: use new style classesChris Larson2011-01-041-6/+3
| | | | | | | (Bitbake rev: e084430446be2544dd1a6b627088f888c37cc7f0) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: pyflakes/pep8/pylint cleanupChris Larson2011-01-041-29/+40
| | | | | | | (Bitbake rev: 06420ff839ffc37de3e42474b8b0e47c4608a985) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: use set for clean, checkedChris Larson2011-01-041-11/+9
| | | | | | | (Bitbake rev: 3a776ef947e23b1a04bdd9e78a3a0c4616249b0e) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* cache: make use of defaultdictChris Larson2011-01-041-34/+10
| | | | | | | (Bitbake rev: 7171019b11ad656d0edb979564941fcf92ab0a02) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-04-30 22:51:07 +0000