summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: tests/fetch: add npmsw testsJean-Marie LEMETAYER2020-01-271-0/+251
| | | | | | | | | | | | | | | | | | | This commit adds some tests to validate the npmsw fetcher: - bb.tests.fetch.NPMTest.test_npmsw - bb.tests.fetch.NPMTest.test_npmsw_bad_checksum - bb.tests.fetch.NPMTest.test_npmsw_destsuffix - bb.tests.fetch.NPMTest.test_npmsw_dev - bb.tests.fetch.NPMTest.test_npmsw_mirrors - bb.tests.fetch.NPMTest.test_npmsw_no_network_no_tarball - bb.tests.fetch.NPMTest.test_npmsw_no_network_with_tarball - bb.tests.fetch.NPMTest.test_npmsw_npm_reusability - bb.tests.fetch.NPMTest.test_npmsw_premirrors (Bitbake rev: ba205df20b6a07a4b1125332601c6c54c7b019b5) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add the npmsw fetcherJean-Marie LEMETAYER2020-01-272-0/+257
| | | | | | | | | | | | | | | | | | | | | | | | This commit adds a new npmsw fetcher that fetches every npm dependencies described in a npm shrinkwrap file: https://docs.npmjs.com/files/shrinkwrap.json.html The main package must be fetched separately: SRC_URI = "npm://registry.url;package=foobar;version=1.0.0 \ npmsw://${THISDIR}/npm-shrinkwrap.json" Since a separation has been created between the package and its dependencies, the package can also be fetched with a non npm fetcher without impacting the general behavior: SRC_URI = "git://github.com/foo/bar.git;protocol=https \ npmsw://${THISDIR}/npm-shrinkwrap.json" (Bitbake rev: f5223be54450bf20e0bfbd53b372a7748a44b475) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: add npm testsJean-Marie LEMETAYER2020-01-271-0/+183
| | | | | | | | | | | | | | | | | | | | | | | | | This commit adds some tests to validate the npm fetcher: - bb.tests.fetch.NPMTest.test_npm - bb.tests.fetch.NPMTest.test_npm_bad_checksum - bb.tests.fetch.NPMTest.test_npm_destsuffix_downloadfilename - bb.tests.fetch.NPMTest.test_npm_mirrors - bb.tests.fetch.NPMTest.test_npm_no_network_no_tarball - bb.tests.fetch.NPMTest.test_npm_no_network_with_tarball - bb.tests.fetch.NPMTest.test_npm_package_invalid - bb.tests.fetch.NPMTest.test_npm_package_none - bb.tests.fetch.NPMTest.test_npm_premirrors - bb.tests.fetch.NPMTest.test_npm_registry_alternate - bb.tests.fetch.NPMTest.test_npm_registry_invalid - bb.tests.fetch.NPMTest.test_npm_registry_none - bb.tests.fetch.NPMTest.test_npm_version_invalid - bb.tests.fetch.NPMTest.test_npm_version_latest - bb.tests.fetch.NPMTest.test_npm_version_none (Bitbake rev: b166bd3cc6cc1ca63e885319091f17daaaaa2537) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/npm: refactor the npm fetcherJean-Marie LEMETAYER2020-01-271-271/+267
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit refactors the npm fetcher to improve some points and fix others: - The big change is that the fetcher is only fetching the package source and no more the dependencies. Thus the npm fetcher act as the other fetchers e.g git, wget. The dependencies will be handled later. - The fetcher only resolves the url of the package using 'npm view' and then forwards it to a proxy fetcher. - This commit also fixes a lot of issues with the package names (exotic characters, scoped packages) which were badly handled. - The validation files - lockdown.json and npm-shrinkwrap.json - are no longer used by the fetcher. Instead, the downloaded tarball is verified with the 'integrity' and 'shasum' provided in the 'npm view' of the package [1][2]. 1: https://docs.npmjs.com/files/package-lock.json#integrity 2: https://www.w3.org/TR/SRI (Bitbake rev: 0f451cdc43130d503ada53ed1b4fc5a24943f6ef) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: fix downloadfilename parameterJean-Marie LEMETAYER2020-01-271-3/+4
| | | | | | | | | | | When using a download filename with characters which can be interpreted by the shell ('(', ')', '&', ';', ...) the command fails. Quoting the filename fixes the issue. (Bitbake rev: ed652dce5200161068eccdbfaaaefde33136eb09) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: allow fetchers to forward the done conditionJean-Marie LEMETAYER2020-01-271-10/+20
| | | | | | | | | | | | | This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the done condition to a proxy fetcher. (Bitbake rev: ee3a2545e99e6e99559a72bcda64797ae674ec71) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: allow fetchers to forward the mirrors managementJean-Marie LEMETAYER2020-01-271-4/+10
| | | | | | | | | | | | | This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the "try_mirrors" functions to a proxy fetcher. (Bitbake rev: 462c9a2b368a1720da276310b1d5d0423b7cefea) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: allow fetchers to forward the donestamp managementJean-Marie LEMETAYER2020-01-271-5/+17
| | | | | | | | | | | | | This commit is necessary to introduce proxy fetchers and do not modify the behavior of existing fetchers. This commit allows fetchers to forwards the "verify_donestamp" and "update_stamp" functions to a proxy fetcher. (Bitbake rev: f7612c0704b4252bba5157ce9a94d8888c6d0760) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add more hash functions for checksum verificationJean-Marie LEMETAYER2020-01-271-1/+1
| | | | | | | | | | | | | | | | | | This commit enables the "sha1", "sha384" and "sha512" hash functions in the supported checksum list. This allows to use more SRC_URI checksums functions for a url: SRC_URI[sha1sum] = "..." SRC_URI[sha384sum] = "..." SRC_URI[sha512sum] = "..." The npm fetcher needs this to support subresource integrity: https://www.w3.org/TR/SRI/ (Bitbake rev: cd80a646aa841b71e68282bb8d11194abb5df0e4) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: refactor checksum verificationJean-Marie LEMETAYER2020-01-271-71/+85
| | | | | | | | | | | | This commit refactors the way checksums are verified to be more generic. The support of new hash functions is now limited to the update of the CHECKSUM_LIST variable. (Bitbake rev: debd9eeaf5638755d8956b2d65b904fe02826966) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add is_semver functionJean-Marie LEMETAYER2020-01-271-0/+26
| | | | | | | | | | | | This function checks if a string is a semantic version: https://semver.org/spec/v2.0.0.html The npm fetcher needs this function to validate its version parameter. (Bitbake rev: 61ac4e825fa7afbb76282030586abc9ee4ac215c) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add sha384_file and sha512_file functionsJean-Marie LEMETAYER2020-01-271-0/+14
| | | | | | | | | | The npm fetcher needs these functions to support the subresource integrity: https://www.w3.org/TR/SRI/ (Bitbake rev: 80e2216e2b41cb6170292009064864449bc48bbe) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch: Allow wget upgrade tests to run against a local server3.1_M2Richard Purdie2020-01-2240-20/+7691
| | | | | | | | | | | Currently these tests rely upon multiple uptream webservers which may change or be unavailable. Add local copies of the test data, copy the httpserver from OE-Core (used for testing there) and run these tests against a local server instead. (Bitbake rev: d5a4a352723258b4d499d3a51f340109c4f36f60) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data.py: Immediately get exclusive lock in __setitem__Chris Laplante2020-01-191-0/+3
| | | | | | | | | | To avoid races, SQLTable::__setitem__ needs an exclusive lock for the entire transaction, not just the INSERT/UPDATE part. (Bitbake rev: feb43e7c30f5bfab75d718896c45df621810d06f) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: _revision_key: collapse adjacent slashesChris Laplante2020-01-191-1/+3
| | | | | | | | | | | | | >From a SRCREV caching point of view, there is no reason to treat the following upstreams as different: SRC_URI = "git://github.com/file/file.git" SRC_URI = "git://github.com//file/file.git" (Bitbake rev: 425e21c14955dd38868c6e97637df3bbe0f89fac) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: do not suffix srcrev cache key with PNChris Laplante2020-01-191-2/+1
| | | | | | | | | | | | | | | | | | Prior to this change, two different recipes pulling from the same exact repo could get a different SRCREV during a single parse session. This was originally observed using git. For git at least, it still allows recipes to pull from the same repo, but with different branches or tags, since the form of the srcrev cache key for git is: "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name] Where the 'unresolvedrev' part is the branch or tag name. (Bitbake rev: 6c938e6fd29beebe09b32be839dae008fe6491d2) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/utils: add tests for bb.utils.get_referenced_varsChris Laplante via bitbake-devel2020-01-191-0/+44
| | | | | | | (Bitbake rev: 346e74e5d751aadf7881de70b5ab6670dfc463ce) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils: add get_referenced_varsChris Laplante via bitbake-devel2020-01-191-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | Given a start expression, bb.utils.get_referenced_vars returns the referenced variable names in a quasi-BFS order (variables within the same level are ordered aribitrarily). For example, given an empty data store: bb.utils.get_referenced_vars("${A} ${B} ${d.getVar('C')}", d) returns either ["A", "B", "C"], ["A", "C", "B"], or another permutation. If we then set A = "${F} ${G}", then the same call will return a permutation of [A, B, C] concatenated with a permutation of [F, G]. This method is like a version of d.expandWithRefs().references that gives some insight into the depth of variable references. (Bitbake rev: 076eb5453ca35b8b75b8270efb989d5208095b27) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: amend code to use proper singleton comparisons where possibleFrazer Clews2020-01-1928-65/+65
| | | | | | | | | | | amend the code to handle singleton comparisons properly so it only checks if they only refer to the same object or not, and not bother comparing the values. (Bitbake rev: b809a6812aa15a8a9af97bc382cc4b19571e6bfc) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: remove unused importsFrazer Clews2020-01-1972-202/+36
| | | | | | | | | | removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Use a set for the setscene tasks listRichard Purdie2020-01-131-2/+2
| | | | | | | | | | This should give performance improvements to functions using this list of tasks (sets are used for most of the other code for this reason, not sure why this wasn't a set in the first place). (Bitbake rev: f5daef68703481a3c243dfecc7de404e6ebfdbb6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Be consistent when creating/updating progress barsPeter Kjellerstedt2020-01-131-6/+6
| | | | | | | | | | | | | | | When creating a new progress bar (using BBProgress), a colon was appended to the supplied message. However, when updating the message, no colon was appended. Change this so that the colon is instead part of the widgets that make up the progress bar so that it does not matter when and how the message is updated, it always displays the same. (Bitbake rev: 08f35c04f6e1ce4c4ca5c2bef4cd8a192e12e682) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Hide the footer if a process progress bar is shownPeter Kjellerstedt2020-01-131-1/+4
| | | | | | | | | | | | With the introduction of the hash equivalence server, the progress bar for "Checking sstate mirror object availability" is shown repeatedly while the tasks are being executed. If the footer is not hidden then, it will be moved up one line every time, creating a messy interface. (Bitbake rev: 56b5ec4c2b3e658e73ca6c3a12feeb96df0977fb) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Keep track of watched files using a set instead of a listPeter Kjellerstedt2020-01-101-8/+8
| | | | | | | | | | When there are many watched files, keeping track of them using lists is suboptimal. Using sets improves the performance considerably. (Bitbake rev: 1e96df260e47d160dbd36bfc92c31ef06266f662) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Lower debug level for wold build messagesRichard Purdie2020-01-021-3/+3
| | | | | | | | | These messages spam the logs for no good reason, they were useful for debugging a particular problem long ago but are distracting noise now. Disable them. (Bitbake rev: 1a9247c468cf09da60e5d396ccb81e950841c99e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Test extra cross/native hashserv methodRichard Purdie2020-01-021-6/+21
| | | | | | | | | | | | | | | | Hack the hashserv to allow extra data to be injected into the hashserv method. This allows OE-Core to handle cases where there are multiple sstate objects for the same taskhash, e.g. native/cross objects based upon BUILD_ARCH or the host distro (when uninative isn't used). This has been tested and proven to be very effective. We will likely rework the code to improve how this is handled but for now this improves automated builds until we can get to that refactoring and more invasive changes. (Bitbake rev: 0a09b0fa03d1afc08037964dc63a18ef7cff9c78) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task dependency corner case in sanity testRichard Purdie2020-01-021-1/+4
| | | | | | | | | | | A corner case was identified where tasks with valid stamps from previous builds need to be accounted for in the new sanity test in the migration code. Add a variable to track such completed tasks to ensure the sanity test works correctly. (Bitbake rev: d517b1ef13ca7ab2fb4d761d3bd3b9fb7c591514) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure task dependencies are run correctlyRichard Purdie2020-01-021-0/+6
| | | | | | | | | | | | | | | | | | | | | We've seen a number of mystery failures where task B would run despite task A, its dependency not having run. An example would be do_compile when do_unpack didn't run. This has been tracked down to this code block. In theory it shouldn't trigger however it can and has due to bugs elsewhere. When it does, it causes significant weird failures and possible build corruption. Change the code to abort the build. This avoids any chance of corruption and should ensure the issues get reported, putting an end to the weird build failures. There may be some cases where this triggers and it shouldn't, we'll work through those as they arise and are identified. (Bitbake rev: 7a92b7f58ab187eddfe550bd6fb687240c7b11bb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix equiv hash handling build failuresRichard Purdie2020-01-021-1/+2
| | | | | | | | | | | | | Regardless of whether we remapped the hash on the server or not, we need to have bitbake work as if we did as we need to match how the stamp files look. This change resolves build failures where tasks were rerunning when they shouldn't. (Bitbake rev: 40928f6991436cf687821015324483b205abfcb1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-layers: Keep loglevel in colored loggerOla x Nilsson2019-12-301-1/+3
| | | | | | | (Bitbake rev: 68ce22df49a5f2105d098bfddcc14284043bfa5d) Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Use with while reading pidfileOla x Nilsson2019-12-301-7/+5
| | | | | | | (Bitbake rev: 6fa8a18ea4994031fdd1253fe363c5d8eeeba456) Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on ↵Aníbal Limón2019-12-303-6/+8
| | | | | | | | | | | | | | | directories The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list of directories to exclude when making taskhash, our specific case is using SRC_URI that points local VCS directory. Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn" (Bitbake rev: 923aff060d8aba8456979c35b16d300ba7c13ff9) Signed-off-by: Aníbal Limón <anibal.limon@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/clearcase: Fix tar command working directoryCHerzig@Gauselmann.de2019-12-281-1/+1
| | | | | | | | | | We need to ensure the correct working directory is used for this tar command to succeed. (Bitbake rev: e1a2a0e29a75dead3426ae083387181ad52da91f) Signed-off-by: Christian Herzig <cherzig@gauselmann.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Optimise out debug messages from cookerRichard Purdie2019-12-161-0/+5
| | | | | | | | | | | | | | | | | We have bb.debug(2, xxx) messages in cooker which are useful for debugging but have really bad effects on performance, 640,000 calls on recent profile graphs taking tens of seconds. Rather than commenting out debug which can be useful for debugging, don't create events for debug log messages from cooker which would never be seen. We already stop the messages hitting the IPC but this avoids the overhead of creating the log messages too, which has been shown to be signficiant on the profiles. This allows the code to perform whilst allowing debug messages to be availble when wanted/enabled. (Bitbake rev: f04cd931091fb0508badf3e002d70a6952700495) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Only call into the migrations function if migrations activeRichard Purdie2019-12-161-1/+2
| | | | | | | | | This doesn't save much time but does make the profile counts for the function more accurate which is in itself useful. (Bitbake rev: d446fa89d206fbc6d098215163c968ea5a8cf4a9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise task filteringRichard Purdie2019-12-161-3/+2
| | | | | | | | | We were seeing this running thousands of times with hashequiv, do the filtering where it makes more sense and make it persist. (Bitbake rev: 2cfeb9998a8ad5b1dcda0bb4e192c5e4306dab17) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise out pointless loop iterationRichard Purdie2019-12-161-9/+3
| | | | | | (Bitbake rev: 105d1f0748edde7753a4063e6fdc758ffc8a8a9e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise task migration code slightlyRichard Purdie2019-12-161-8/+9
| | | | | | | | | | | | | Move the calls to difference_update out a code level which improves efficiency significantly. Also further combine the outer loop for efficiency too. These two changes remove a bottleneck from the performance charts. (Bitbake rev: e28ec69356f1797de3e4e3fca0fef710bc4564de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix sstate task iteration performanceRichard Purdie2019-12-161-1/+5
| | | | | | | | | | | Creating a new sorted list of sstate tasks each iteration through runqueue is extremely ineffecient and was compounded by the recent change from a list to set. Create one sorted list instead of recreating it each time. (Bitbake rev: de18824996841c3f35f54ff5ad12f94f6dc20d88) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Split get_tashhash for performanceRichard Purdie2019-12-162-9/+25
| | | | | | | | | | | | | | | There are two operations happening in get_taskhash, the building of the underlying data and the calculation of the hash. Split these into two funtions since the preparation part doesn't need to rerun when unihash changes, only the calculation does. This split allows sigificant performance improvements for hashequiv in builds where many hashes are equivalent and many hashes are changing. (Bitbake rev: 6a32af2808d748819f4af55c443578c8a63062b3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task mismatch failures from incorrect logicRichard Purdie2019-12-161-1/+1
| | | | | | | | | | The "no dependencies" task case was not being correctly considered in this code and seemed to be the cause of occasionaly task hash mismatch errors that were being seen as the dependencies were never accounted for properly. (Bitbake rev: 608b9f821539de813bfbd9e65950dbc56a274bc2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Rework process_possible_migrations() to improve performanceRichard Purdie2019-12-161-47/+56
| | | | | | | | | | | | | | | The looping over multiple changed hashes causes many calls to get_taskhash and get_unihash which are potentially slow and then overwritten. Instead, batch up all the tasks which have changed unihashes and then do one big loop over the changed tasks rather than each in turn. This makes worlds of difference to the performance graphs and should speed up build where many tasks are being rehashed. (Bitbake rev: c9c68d898985cf0bec6fc95f54c151cc50255cac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils.fileslocked: don't leak files if yield throwsChris Laplante via bitbake-devel2019-12-161-4/+5
| | | | | | | | | | | | | Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by externalsrc.bbclass) was leaked, giving a warning like: WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'> exec_func(task, localdata) (Bitbake rev: 6beddf6214e22b4002626761031a9e9d34fb04db) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Bump minimum python version to 3.5Joshua Watt2019-12-161-2/+2
| | | | | | | | | | | | The local hash equivalence server used by bitbake requires python 3.5, so bump up the minimum required version. [YOCTO #13678] (Bitbake rev: 1412dcc077b1bea10b72fc8b525d6258dca46d97) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix performance issue in get_unihashRichard Purdie2019-12-151-3/+3
| | | | | | | | | | There is a significant performance issue in get_unihash(). The issue turns out to be the lookups of setscene tasks. We can fix this by using a set() instead of the current list. (Bitbake rev: 1e561672d039ebfb8cd0e0654a44dcf48513317c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Revert "siggen: Fix hashequiv performance issues"Richard Purdie2019-12-121-13/+14
| | | | | | | | This reverts commit c4b8440f730c33eaf9f818b856ae81b2f1017fec. The logic in this change is flawed and needs to be re-thought. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix hashequiv performance issuesRichard Purdie2019-12-121-14/+13
| | | | | | | | | | | | | | | | | | | We're seeing huge slowdowns on large builds on the autobuilder. A qemux86 world build was running really slowly, a primary feature was lots of rehashing going on due to an sstate change which caused a rebuild when all output should be identical. This was traced to the hundreds of thousands of calls to get_unihash() from get_taskash(). If we simplify the unitaskhashes data structure, we can bypass the function call and access the data directly. In local profile charts, this significanly sped up process_possible_migrations(), both on profiles and visually. Whilst this change doesn't aid readability, it does solve an otherwise huge performance issue. (Bitbake rev: c4b8440f730c33eaf9f818b856ae81b2f1017fec) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Batch scenequeue updatesJoshua Watt2019-12-121-4/+7
| | | | | | | | | | | | Batch all updates to scenequeue data together in a single invocation instead of checking each task serially. This allows the checks for sstate object to happen in parallel, and also makes sure the log statement only happens once (per set of rehashes). (Bitbake rev: db033a8f8a276d864bdb2e1eef159ab5794a0658) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Ensure new unihash propagates through the systemRichard Purdie2019-12-091-0/+1
| | | | | | | | | | Its possible the new unihash may not exist in sstate. Currently the code would create an sstate object with the old hash however this updates it to create the object with the new unihash. (Bitbake rev: abcaa1398031fa5338a43859c661e6d4a9ce863d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Avoid taskhash mismatch errors for nostamp tasks when ↵Richard Purdie2019-12-061-4/+8
| | | | | | | | | | | | | | | | | | | dependencies rehash An example: NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Started ERROR: Taskhash mismatch b074da4334aff8aa06572e7a8725c941fa6b08de4ce714a65a90c0c0b680abea versus 17375278daed609a7129769b74a1336a37bdef14b534ae85189ccc033a9f2db4 for /home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb:do_check NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Succeeded Is caused by a rehash in a dependency happening somewhere earlier in the build and the taint being reset. Change the code so that nostamp taints are preserved to avoid the issue. (Bitbake rev: 61624a3fc38e8546e01356d5ce7a09f21e7094ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-12 05:25:56 +0000