summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: runqueue: Optimise task migration code slightlyRichard Purdie2020-01-111-8/+9
| | | | | | | | | | | | | | | Move the calls to difference_update out a code level which improves efficiency significantly. Also further combine the outer loop for efficiency too. These two changes remove a bottleneck from the performance charts. (Bitbake rev: 57df7f191755dd887827d51b125d246c1af3e1b6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e28ec69356f1797de3e4e3fca0fef710bc4564de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix sstate task iteration performanceRichard Purdie2020-01-111-1/+5
| | | | | | | | | | | | | Creating a new sorted list of sstate tasks each iteration through runqueue is extremely ineffecient and was compounded by the recent change from a list to set. Create one sorted list instead of recreating it each time. (Bitbake rev: a80024a4397485f96693c4df108b40f6b1bdf32b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit de18824996841c3f35f54ff5ad12f94f6dc20d88) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Split get_tashhash for performanceRichard Purdie2020-01-112-9/+25
| | | | | | | | | | | | | | | | | There are two operations happening in get_taskhash, the building of the underlying data and the calculation of the hash. Split these into two funtions since the preparation part doesn't need to rerun when unihash changes, only the calculation does. This split allows sigificant performance improvements for hashequiv in builds where many hashes are equivalent and many hashes are changing. (Bitbake rev: c4ce216b4c5a1626764752edd21005750d05a0c2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6a32af2808d748819f4af55c443578c8a63062b3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task mismatch failures from incorrect logicRichard Purdie2020-01-111-1/+1
| | | | | | | | | | | | The "no dependencies" task case was not being correctly considered in this code and seemed to be the cause of occasionaly task hash mismatch errors that were being seen as the dependencies were never accounted for properly. (Bitbake rev: 480939538f5887a70d11c2f7cf74446100e25fab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 608b9f821539de813bfbd9e65950dbc56a274bc2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Rework process_possible_migrations() to improve performanceRichard Purdie2020-01-111-47/+56
| | | | | | | | | | | | | | | | | The looping over multiple changed hashes causes many calls to get_taskhash and get_unihash which are potentially slow and then overwritten. Instead, batch up all the tasks which have changed unihashes and then do one big loop over the changed tasks rather than each in turn. This makes worlds of difference to the performance graphs and should speed up build where many tasks are being rehashed. (Bitbake rev: c9ab598f6f1ea3ae3a0713dc6692b4c4bafbfb50) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c9c68d898985cf0bec6fc95f54c151cc50255cac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils.fileslocked: don't leak files if yield throwsChris Laplante via bitbake-devel2020-01-111-4/+5
| | | | | | | | | | | | | | | Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by externalsrc.bbclass) was leaked, giving a warning like: WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'> exec_func(task, localdata) (Bitbake rev: 81829ab28afae08e02f4a758ec063fc0d90579ea) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6beddf6214e22b4002626761031a9e9d34fb04db) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix performance issue in get_unihashRichard Purdie2020-01-111-3/+3
| | | | | | | | | | | | There is a significant performance issue in get_unihash(). The issue turns out to be the lookups of setscene tasks. We can fix this by using a set() instead of the current list. (Bitbake rev: 5afad266f2ce55db2038c36f2e49a3c80be9bbfc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1e561672d039ebfb8cd0e0654a44dcf48513317c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Batch scenequeue updatesJoshua Watt2020-01-111-4/+7
| | | | | | | | | | | | | | Batch all updates to scenequeue data together in a single invocation instead of checking each task serially. This allows the checks for sstate object to happen in parallel, and also makes sure the log statement only happens once (per set of rehashes). (Bitbake rev: a7426c73a8e9fae468414a2c32a533d9c3729405) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit db033a8f8a276d864bdb2e1eef159ab5794a0658) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Ensure new unihash propagates through the systemRichard Purdie2020-01-111-0/+1
| | | | | | | | | | | | Its possible the new unihash may not exist in sstate. Currently the code would create an sstate object with the old hash however this updates it to create the object with the new unihash. (Bitbake rev: 0aee83e4e31dff7f4354e4eb4cbd35dd592e9f06) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit abcaa1398031fa5338a43859c661e6d4a9ce863d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Avoid taskhash mismatch errors for nostamp tasks when ↵Richard Purdie2020-01-111-4/+8
| | | | | | | | | | | | | | | | | | | | | dependencies rehash An example: NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Started ERROR: Taskhash mismatch b074da4334aff8aa06572e7a8725c941fa6b08de4ce714a65a90c0c0b680abea versus 17375278daed609a7129769b74a1336a37bdef14b534ae85189ccc033a9f2db4 for /home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb:do_check NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Succeeded Is caused by a rehash in a dependency happening somewhere earlier in the build and the taint being reset. Change the code so that nostamp taints are preserved to avoid the issue. (Bitbake rev: c42d00ff293d0538cad1b84c108bf7f5f49d4d84) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 61624a3fc38e8546e01356d5ce7a09f21e7094ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty/uihelper: Switch from pids to tids for Task event managementRichard Purdie2020-01-113-33/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've seen cases where a task can execute with a given pid, complete and a new task can start using the same pid before the UI handler has had time to adapt. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/knotty.py", line 484, in main helper.eventHandler(event) File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/uihelper.py", line 30, in eventHandler del self.running_tasks[event.pid] KeyError: 13490 This means using pids to match up events on the UI side is a bad idea. Change the code to use task ids instead. There is a small amount of fuzzy matching for the progress information since there is no task information there and we don't want the overhead of a task ID in every event, however since pid reuse is unlikely, we can live with a progress bar not quite working properly in a corner case like this. [YOCTO #13667] (Bitbake rev: a109d034cf4fc059fd5a1e1d03246dac65522dd6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e427eafa1bb04008d12100ccc5c862122bba53e0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add extra debugging when locked sigs mismatches occurRichard Purdie2020-01-111-0/+2
| | | | | | | | (Bitbake rev: 6f0b82627edc82601f80f0f096bf96db43afefa8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3aad9978be2a40d4c535a5ae092f374ba2a5f627) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Allow handling of equivalent hashesRichard Purdie2020-01-112-12/+45
| | | | | | | | | | | | | | | Based on the hashserv's new ability to accept hash mappings, update runqueue to use this through a helper function in siggen. This addresses problems with meta-extsdk-toolchain and its dependency on gdb-cross which caused errors when building eSDK. See the previous commit for more details. (Bitbake rev: 222df6d6b832868c6e87334f8acdd74b730a91d6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 39098b4ba2133f4d9229a0aa4fcf4c3e1291286a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue.py: not show warning for deferred multiconfig taskKai Kang2019-11-291-1/+1
| | | | | | | | | | | | | | | | | | | When follow the instructions of multiconfig from Yocto dev manual that set in core-image-sato recipe: do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs" it show too many annoying warnings look like: | WARNING: Deferring mc:x86:virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot | after mc:arm: virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot Treat them as infomations rather than warnings. (Bitbake rev: cfa307aabf710d79c404a8571b4158b864a94727) Signed-off-by: Kai Kang <kai.kang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Fix to match recent task migration fixesRichard Purdie2019-11-271-1/+1
| | | | | | (Bitbake rev: 8569ccb5e9fbdeaaf96b78bd02a263b26de54059) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure failed harddependencies in scenequeue are ↵Richard Purdie2019-11-251-1/+8
| | | | | | | | | | | | | | | | | | accounted for in migrations Setscene hard dependencies were not being correctly handled during task migration. For example, do_package of recipe X might become valid due to hashequiv yet we're still rebuilding pseudo-native, a harddep of do_package. This would mean it would try to execute that setscene task despite pseudo not being present. Fix this by ignoring tasks with failed harddependencies. This does mean stlightly more rebuilds than is optimal but it avoids build crashes. Ultimately the new runqueue model can likely better handle these cases than the older codebase could but that is for another more invasive patch. (Bitbake rev: 9a1072060350dc2e0eee14a5cc5af20c900f8a6d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve sstate rehashing outputRichard Purdie2019-11-251-12/+16
| | | | | | | | | | | | | | Bibake is currently too 'chatty' when hash equivalence is enabled. Fix this by only printing the log output if a rehash happens and it matches an sstate object. Also, pass a summary option to the hash checking function. This was already changed to a mechanism which allows addition of new parameters so this should be backwards and forwards compatible. (Bitbake rev: c5c5d786ca968d0e48002fe8acbcc8a63a954b67) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix hashequiv bug where new hash wasn't referenced correctlyRichard Purdie2019-11-251-0/+1
| | | | | | | | | | | If a hash is reported to the hash server, the stamp written out by the current task didn't account for any new hash the server may have provided. Fix this so the correct stamp is written. This means "bitbake X; bitbake X" no longer rebuilds lots of things when hashequiv is active. (Bitbake rev: 4299afdd290f9d1c5616598f5fe83c195a64b63c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix hash equivalence duplicate tasks runningRichard Purdie2019-11-251-4/+10
| | | | | | | | | | | | | | | | | The key problem is that currently running setscene tasks are not accounted for when processing task migrations. This means can allow two of the same task to execute at the same time with unpredictable effects. This change allows us to stop doing that and refactor the code slightly to make it clearer that these conditions don't arrive even with deferred tasks. (Bitbake rev: 33ffc2128b1a74fa7179a8341db68cddf402536f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/clearcase: Fix warnings from python 3.8Richard Purdie2019-11-251-3/+3
| | | | | | | | | | | | | | | | | | bitbake/lib/bb/fetch2/clearcase.py:148: SyntaxWarning: "is" with a literal. Did you mean "=="? if command is 'mkview': bitbake/lib/bb/fetch2/clearcase.py:155: SyntaxWarning: "is" with a literal. Did you mean "=="? elif command is 'rmview': bitbake/lib/bb/fetch2/clearcase.py:159: SyntaxWarning: "is" with a literal. Did you mean "=="? elif command is 'setcs': Python 3.8 is quite correct and we so mean "==" here, fix it to avoid the warnings. (Bitbake rev: 2cccc14304855cb55f339e465f6ba6ed0c69a7ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: also use mmap for SHA256 and SHA1, for performanceRoss Burton2019-11-251-20/+14
| | | | | | | | | | | | md5_file() uses a mmap() window to improve performance when hashing files, so refactor the code and do the same for SHA1 and SHA256. (Bitbake rev: 94ede642dce8cdbf09f566e3f7e9e260d33fda27) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests: add test for the hashing functionsRoss Burton2019-11-251-0/+26
| | | | | | | | | | | Add a basic test for bb.utils.md5_file() etc. (Bitbake rev: d535e78b14136e74d6e96ff24d3464d62637459d) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Remove a left-over comment about expanded_dataPeter Kjellerstedt2019-11-251-4/+0
| | | | | | | | | | | | This should have been removed together with expanded_data in commit e3694e73 (cooker/command: Drop expanded_data). (Bitbake rev: 987996f01d55bc6433aeb7f43c209eb12f6d796b) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/hg: Fix various runtime issuesVolker Vogelhuber2019-11-251-10/+17
| | | | | | | | | | | | Fix mercurial fetching after breakage from changes to the core fetcher. Fix username and password usage and setting moddir needed by setup_revisions. (Bitbake rev: c61c8356cce4d7307f74147dcf2b2cf103db84a8) Signed-off-by: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: add test for fetching shallow revsChristopher Larson2019-11-251-0/+20
| | | | | | | | | | | [YOCTO #13586] (Bitbake rev: 566a6fe8c217c02f1ba5afc621ae9c3523f35d03) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: fetch shallow revs when neededChristopher Larson2019-11-251-7/+16
| | | | | | | | | | | | | | | | When bitbake determines if a git clone needs updating, it only checks for the needed srcrevs, not the revs listed in BB_GIT_SHALLOW_REVS, which will fail if using shallow and the needed rev was added to the upstream git repo after a previous fetch. Ensure that we also check for shallow revs. [YOCTO #13586] (Bitbake rev: 24e3c7189e7d41bcbb46078a41c3a9daf391202a) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Ensure cached url data is matched to a datastoreRichard Purdie2019-11-131-2/+2
| | | | | | | | | | | | | | | | | | | | There was a weird error in OE-Core where "devtool modify virtual/kernel" was showing basehash mismatch errors. This was due to SRCPV sometimes being: AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28. The latter hash comes from KBRANCH and meant sometimes the correct branch was seen, sometimes it was not. The issue was complicated by the execution using a remote datastore over tinfoil. The problem turns out to be a fetcher caching error. If the datastore changes, the cached url data may not be valid. We therefore ensure we match cached url data against the datastore that generated it, which appears to fix this issue. (Bitbake rev: 1a79651c518abc35b99005c137ab7e82a99c75b0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Update to version 1.44.0Richard Purdie2019-10-091-1/+1
| | | | | | (Bitbake rev: 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump verison 1.43.1 -> 1.43.2uninative-2.7Richard Purdie2019-10-021-1/+1
| | | | | | | | This allows metadata to depend on SignatureGeneratorUniHashMixIn which was recently added. (Bitbake rev: f0f814407fdd2fffa7071c36c011b489bfcd53da) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: improve warnings when adding dependency to packagesDavid Reyna2019-10-021-0/+10
| | | | | | | | | | | | | Some of the objects that bitbake reports to Toaster as dependencies to packages are known objects that are not packages, for example library files and kernel modules. In the Toaster logs, mark these as "Info" instead of "Warning". [YOCTO #13386] (Bitbake rev: 0d66f644d647900e8f5afa526a6d9cee687c41cc) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Fix hashserve shutdown raceRichard Purdie2019-10-021-0/+12
| | | | | | | | | | | | | The hashserve can delete its socket whilst the cleanup us happening leading to backtraces and test failures. Add code to avoid this race condition. [YOCTO #13542] (Bitbake rev: efd7b025cee25d0ee668c09476395d08fcf5ae1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Remove full path from unitaskhashes keysRichard Purdie2019-09-301-2/+2
| | | | | | | | | | The full paths make the cache useless in the sdk. They also bloat the cache size. They're for human debugging benefit only so compromise and reduce this to the filename. (Bitbake rev: 3b275c4083eae1d3781f0862919af9de83932b0f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Small performance optimisationRichard Purdie2019-09-301-0/+2
| | | | | | | | | | | A minor performance optmisation to keep lists smaller when running large builds. We can do this since once a task has been built, we don't need to worry about it. This improves a major bottleneck that shows up on performance profile charts in dryruns. (Bitbake rev: cd6b89230823707c3c9bb9e6883bf5a971916581) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Save unihashes more frequentlyRichard Purdie2019-09-301-0/+1
| | | | | | | | | | There are some runqueue code paths where the unihash cache would not be saved where for example only parsing or an occurred. Save the cache at the end of runqueue generation to ensure entries are cached. (Bitbake rev: 9eee0d36870c11dd303894a6151c33a83bd3a1bc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Avoid writing misleading sigdata filesRichard Purdie2019-09-301-1/+2
| | | | | | | | | | Use the unihash in the output filename of sigdata files else the contents of stamp directories is misleading. Write the unihash into the singature to make it clear what happened. (Bitbake rev: feb01ee54d3706fe93768f332054c7532f7209e4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen/runqueue: Fix signature mismatch issuesRichard Purdie2019-09-302-2/+7
| | | | | | | | | | | | We need to set the setscene tasklist before we call into the taskhash/unihash code else the behaviour is inconsistent. Avoid reporting hashes for non setscene tasks since we'd never query that. (Bitbake rev: 419a7840b8627278db694029c25df00214d01d96) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Change task migration behaviour for rerunning setscene tasksRichard Purdie2019-09-302-4/+5
| | | | | | | | | | | | | | | | | Currently runqueue will rerun setscene tasks multiple times as hashes change. This has caused numerous problems since a setscene task may become "unavailable" for some future signature combination and the code then can't easily "unskip" tasks its already passed into the execution queue. At least for now, only run setscene once and assume they're equivalent at that point. In practise that has been much more stable in testing. Tweak the test to match the change in behaviour. (Bitbake rev: 4205a3ef23834f317642bba155d67cd772176fb6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Ensure setscenetasks list is available to worker contextRichard Purdie2019-09-301-2/+3
| | | | | | | | | | | The setscenetasks list needs to be available in the worker contexts else the signature behaviour there mismatches what the server does. Add the data to get/set_taskdata to ensure this happens. (Bitbake rev: 632980ef90fe126b7ba3d138f4d574ae05914779) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task migration problemsRichard Purdie2019-09-302-133/+55
| | | | | | | | | | | | | | | | | Tasks were not migrating consistently, particularly: * if a task was rehashed which had already run * if a task which was valid became invalid due to a rehash We need to always run the migration code for rehashed tasks and then reprocess them for hash validity. This means rearranging the code. It also means several tests are no longer correct and can't be written correctly to work on all possible workflows so those are removed. (Bitbake rev: 8443989ee41e9b162972935513e437b5c66ea74d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Don't daemonize server processJoshua Watt2019-09-271-1/+0
| | | | | | | | | | | | | | | The hash server process is terminated and waited on with join(), so it should not be a daemon. Daemonizing it cause races with the server cleanup, especially in the selftest because the process may not have terminated and cleanup up its socket before the test cleanup runs and tries to do it. [YOCTO #13542] (Bitbake rev: 7c829675581818f92d57056b57fbd3880829b6bd) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix attribute error when hashserver failsJoshua Watt2019-09-271-2/+2
| | | | | | | | | | | | The HashConnectionError class was moved to the client module and needs to be updated. [YOCTO #13537] (Bitbake rev: 9fb862685e5e5a2aa534bc25cab1e4158d708b40) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Add mc conffiles hashes to cache hashJoshua Watt2019-09-271-1/+5
| | | | | | | | | | | | | | | | | | | | | The variable values that result from parsing multiconfig should be included in the cooker data hash, otherwise changes to these files won't be detected, which will allow the parsing cache to be loaded with the old values for the multiconfigs. This can either manifest as the variable values simply not updating, or getting basehash changed errors when building. This bug was previously undetected because all of the multiconfig base files were a direct file dependency in all parsed recipes. This was fixed in 34137a00f60 ("bitbake: bitbake: cooker: Rename __depends in all multiconfigs"), exposing this bug. [YOCTO #13541] (Bitbake rev: c74481aa15226e1bff9d53e4ee4b702ebfa1ad32) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/data: Test combinations of _append together with overrideJacob Kroon2019-09-271-0/+13
| | | | | | | (Bitbake rev: f31f35e8527c60a95931a4a8311a4cd237770b42) Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: add test case for git-lfs handlingRoss Burton2019-09-271-0/+80
| | | | | | | | | | Add a test case to exercise the detection of git-lfs repositories and the behaviour of the lfs parameter. (Bitbake rev: a7cf4fc72cce357c425084dc2c5f35b5ed1a4b7b) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: refactor check for git-lfs commandRoss Burton2019-09-271-8/+9
| | | | | | | | | | Refactor the git-lfs checking: this means both clearer code in the download() function and allows unit testing to monkeypatch the functionality. (Bitbake rev: 33cf9172ded50a869f7201ba463ab9ecc69b8252) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add ionice option to prunedirRichard Purdie2019-09-191-4/+7
| | | | | | | | | | | | | | | | Autobuilder type infrastructure can benefit from deletion of certain files as background IO due to the way Linux filesystem priority works. We have problems where build directories as part of oe-selftest being delete starves the running tasks of IO to the point builds take much longer to compelte. Having this option of running the deletion at "idle" helps a lot with that. (Bitbake rev: 797354d285f6d624d9adb52bab65823572da0e39) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: validate_hashes(): currentcount should be a numberRobert Yang2019-09-191-3/+3
| | | | | | | | | | | | | | | | According to sstate_checkhashes which is defined in sstate.bbclass, the currentcoun should be a number (0, not None). Fixed: $ bitbake base-files -Sprintdiff > bb.plain("Sstate summary: Wanted %d Found %d Missed %d Current %d (%d%% match, %d%% complete)" % (total, len(found), len(missed), currentcount, match, complete)) TypeError: %d format: a number is required, not NoneType (Bitbake rev: 45cb73e2846eaffe8964a573875f54808e8f3633) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: add git-lfs toggle optionRoss Burton2019-09-191-7/+11
| | | | | | | | | | | | | | | Add a new 'lfs' option to the git fetcher so that we can optionally not fetch git-lfs content, for repositories that contain LFS data that we don't actually need for building. By default lfs is set to 1, so if the repository has LFS content then git-lfs is required. Setting lfs to 0 will mean that git-lfs won't be required to fetch, and some files will be missing. (Bitbake rev: be0b78ccfc5ede98041bc0545a15092494b12b26) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: svn fetcher: allow "svn propget svn:externals" to failMikko Rapeli2019-09-181-1/+1
| | | | | | | | | | | | | | Not all servers and repositories have this property set which results in failures like this when actual svn checkout command succeeded: svn: warning: W200017: Property 'svn:externals' not found on '' svn: E200000: A problem occurred; see other errors for details (Bitbake rev: 238636f033cbf18e5741f0ea0e64db40e84f5838) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rework hash equivalenceJoshua Watt2019-09-184-64/+50
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Reworks the hash equivalence server to address performance issues that were encountered with the REST mechanism used previously, particularly during the heavy request load encountered during signature generation. Notable changes are: 1) The server protocol is no longer HTTP based. Instead, it uses a simpler JSON over a streaming protocol link. This protocol has much lower overhead than HTTP since it eliminates the HTTP headers. 2) The hash equivalence server can either bind to a TCP port, or a Unix domain socket. Unix domain sockets are more efficient for local communication, and so are preferred if the user enables hash equivalence only for the local build. The arguments to the 'bitbake-hashserve' command have been updated accordingly. 3) The value to which BB_HASHSERVE should be set to enable a local hash equivalence server is changed to "auto" instead of "localhost:0". The latter didn't make sense when the local server was using a Unix domain socket. 4) Clients are expected to keep a persistent connection to the server instead of creating a new connection each time a request is made for optimal performance. 5) Most of the client logic has been moved to the hashserve module in bitbake. This makes it easier to share the client code. 6) A new bitbake command has been added called 'bitbake-hashclient'. This command can be used to query a hash equivalence server, including fetching the statistics and running a performance stress test. 7) The table indexes in the SQLite database have been updated to optimize hash lookups. This change is backward compatible, as the database will delete the old indexes first if they exist. 8) The server has been reworked to use python async to maximize performance with persistently connected clients. This requires Python 3.5 or later. (Bitbake rev: 2124eec3a5830afe8e07ffb6f2a0df6a417ac973) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-20 17:31:28 +0000