summaryrefslogtreecommitdiffstats
path: root/meta/classes/externalsrc.bbclass
Commit message (Collapse)AuthorAgeFilesLines
* devtool: add support for multiple git url inside a cargo based recipeFrederic Martinsons2023-04-011-3/+1
| | | | | | | | | | | | | | | | Without that, the possible git urls that are in SRC_URI of a recipe are removed from SRC_URI during devtool process and so the cargo_common_do_patch_paths in cargo_common.bbclass cannot patch these packages to fetch them locally. I use a generic type name because I foresee this change will be useful for recipe that used a package manager (cargo but also npm) see https://bugzilla.yoctoproject.org/show_bug.cgi?id=11015 (From OE-Core rev: 474658a3681c343385c359a21c3693401217298d) Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* devtool: fix devtool finish when gitmodules file is emptyThomas Roos2023-01-181-1/+1
| | | | | | | | | | | | | When a .gitmodules file exists but is empty then devtool finish fails. Add an additional check for this. [YOCTO #14999] (From OE-Core rev: b4f0f7c4934bade9e4d4a1086f9d8b29d8e9ad45) Signed-off-by: Thomas Roos <throos@amazon.de> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: fix lookup for .gitmodulesPeter Marko2022-12-081-1/+1
| | | | | | | | | | | | | | | | | | Commit 0533edac277080e1bd130c14df0cbac61ba01a0c broke bitbake parsing when bitbake is executed from directory with existing .gitmodules and the recipe in externalsrc does not have .gitmodules The check needs to search for .gitmodules in sources path, not cwd. iParsing recipes...ERROR: ExpansionError during parsing <path to recipe> ... bb.data_smart.ExpansionError: Failure expanding variable do_compile[file-checksums], expression was ${@srctree_hash_files(d)} which triggered exception CalledProcessError: Command '['git', 'config', '--file', '.gitmodules', '--get-regexp', 'path']' returned non-zero exit status 1. (From OE-Core rev: 66ff3d1f65cd2e7f5319e98fa41f47a59b714c72) Signed-off-by: Peter Marko <peter.marko@siemens.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc.bbclass: Remove a trailing slash from ${B}Peter Kjellerstedt2022-10-261-1/+1
| | | | | | | | | | | | | The trailing slash in ${B} caused -fdebug-prefix-map=${B}=... to not match as intended, resulting in ${TMPDIR} ending up in files in ${PN}-dbg when externalsrc was in use, which in turn triggered buildpath QA warnings. (From OE-Core rev: 9b5031ed5a0d102905fa75acc418246c23df6eef) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc.bbclass: fix git repo detectionMartin Jansa2022-10-261-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * fix issue introduced in: https://git.openembedded.org/openembedded-core/commit/?id=95fbac8dcad6c93f4c9737e9fe13e92ab6befa09 * it added check for s_dir + git-dir (typically '.git') isn't the same as ${TOPDIR} + git-dir, but due to copy-paste issue it was just comparing it with s_dir + git-dir again, resulting in most external repos (where git-dir is '.git') to be processed as regular directory (not taking advantage of git write-tree). * normally this wouldn't be an issue, but for big repo with a lot of files this added a lot of checksums in: d.setVarFlag('do_compile', 'file-checksums', '${@srctree_hash_files(d)}') and I mean *a lot, e.g. in chromium build it was 380227 paths which still wouldn't that bad, but the checksum processing in siggen.py isn't trivial and just looping through all these checksums takes very long time (over 1000sec on fast NVME drive with warm cache) and then https://git.openembedded.org/bitbake/commit/?id=b4975d2ecf615ac4c240808fbc5a3f879a93846b made the processing a bit more complicated and the loop in get_taskhash() function took 6448sec and to make things worse there was no output from bitbake during that time, so even with -DDD it looks like this: DEBUG: virtual/libgles2 resolved to: mesa (langdale/oe-core/meta/recipes-graphics/mesa/mesa_22.2.0.bb) Bitbake still alive (no events for 600s). Active tasks: Bitbake still alive (no events for 1200s). Active tasks: Bitbake still alive (no events for 1800s). Active tasks: Bitbake still alive (no events for 2400s). Active tasks: Bitbake still alive (no events for 3000s). Active tasks: Bitbake still alive (no events for 3600s). Active tasks: Bitbake still alive (no events for 4200s). Active tasks: Bitbake still alive (no events for 4800s). Active tasks: Bitbake still alive (no events for 5400s). Active tasks: Bitbake still alive (no events for 6000s). Active tasks: DEBUG: Starting bitbake-worker without -DDD it will get stuck for almost 2 hours in: "Initialising tasks..." before it finally writes sstate summary like: "Sstate summary: Wanted 3102 Local 0 Mirrors 0 Missed 3102 Current 1483 (0% match, 32% complete)" * fix the copy&paste typo to use git work-tree in most cases, but be aware that this issue still exists for huge local source trees not in git [YOCTO #14942] (From OE-Core rev: 9102e5a94b8146cb1da27afbe41d3db999a914ff) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: move back to classesRoss Burton2022-10-251-0/+272
| | | | | | | | | | | The externalsrc class was moved to classes-recipe as part of oe-core f5c1280, but it can be used in both recipe and global contexts so move it back to classes/. (From OE-Core rev: 7a2edcd4b7cb5a2d829289a11eff62663268fbf3) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes: Update classes to match new bitbake class scope functionalityRichard Purdie2022-08-121-269/+0
| | | | | | | | | Move classes to classes-global or classes-recipe as appropriate to take advantage of new bitbake functionality to check class scope/usage. (From OE-Core rev: f5c128008365e141082c129417eb72d2751e8045) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes: Add SPDX license identifiersRichard Purdie2022-08-121-1/+2
| | | | | | | | | | | | As stated in our top level license files, the license is MIT unless otherwise stated. Add SPDX identifers accordingly. Replace older license statementa with the standardised syntax. Also drop "All Rights Reserved" expression as it isn't used now, doesn't mean anything and is confusing. (From OE-Core rev: 081a391fe09a21265881e39a2a496e4e10b4f80b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc.bbclass: support crate fetcher on externalsrcChanho Park2022-05-201-1/+1
| | | | | | | | | | | To support crate:// fetcher on externalsrc, we need to make pass-through the URIs in SRC_URI. (From OE-Core rev: c55c79507f63a4e758a486537aacfc13009161c7) Signed-off-by: Chanho Park <chanho61.park@samsung.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc/devtool: Fix to work with fixed export funcition flags handlingRichard Purdie2022-04-141-9/+10
| | | | | | | | | | | | | If we fix bitbake to correctly handle the cleandirs and fakeroot flags for tasks handled by EXPORT_FUNCTIONS, we break devtool since it only considers these for top level functions. Add in extra code to intercept the cleandirs flags for commonly used sub functions too. [YOCTO #8621] (From OE-Core rev: 67fad829f37ed0a8e20c599d2b65635158591d06) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* reproducible: Merge code into base.bbclassRichard Purdie2021-10-161-2/+2
| | | | | | | | | | Reproducibility is here to stay and needs to be part of our default workflow. Move the remaining code to base.bbclass so it is always a first class citizen and it is clear people need to be mindful of it. (From OE-Core rev: abb0671d2cebfd7e8df94796404bbe9c7f961058) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Fix a source date epoch race in reproducible buildsRichard Purdie2021-10-041-10/+9
| | | | | | | | | | | | When reproducible builds are enabled and externalsrc is in use, the source date epoch function is added. The conditions on the conditional code removing the unpack task need to match the deltask function, else the source date epoch function can end up running twice and the functions can race with each other causing build failures or corruption. (From OE-Core rev: e82095c02881410035ca23dc12692f074d8ed39b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Work with reproducible_buildMark Hatle2021-09-111-0/+10
| | | | | | | | | | | | | | | | Externalsrc removes do_fetch, do_unpack, and do_patch. The system normally discovers the correct reproducible date as a postfuncs of do_unpack, so this date is never found, so it falls back to the default epoch. Instead we can move the discovery function to a prefuncs on the epoch deploy task. This task will run before do_configure, and since the source is already available can run safely at anytime. (From OE-Core rev: 0b7dd711a54e92ce54abe99f59fc67e683d52dfe) Signed-off-by: Mark Hatle <mark.hatle@xilinx.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Convert to new override syntaxRichard Purdie2021-08-021-5/+5
| | | | | | | | | | | | This is the result of automated script conversion: scripts/contrib/convert-overrides.py <oe-core directory> converting the metadata to use ":" as the override character instead of "_". (From OE-Core rev: 42344347be29f0997cc2f7636d9603b1fe1875ae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Detect code changes in submodulesDouglas Royds2021-04-181-5/+6
| | | | | | | | | | | | | | | | Further to 50ff9afb39, only detect code changes in submodules that are subdirectories of the EXTERNALSRC directory. The (undocumented) git submodule--helper returns a path for each submodule relative to the top of the repo. Don't add submodules that are not within our source subtree. [YOCTO #14333] (From OE-Core rev: 1c18225d3ef94a41fc073ae87c163b68e6d46571) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Revert "externalsrc: Detect code changes in submodules"Douglas Royds2021-04-181-9/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reverts commit 4525310d49d115a37705f04ac5c03d639e5e8f8c. Further to 50ff9afb39, only detect code changes in submodules that are subdirectories of the EXTERNALSRC directory. The (undocumented) git submodule--helper returns a path for each submodule relative to the top of the repo. Don't add submodules that are not within our EXTERNALSRC subtree. If we unpack one git repo inside another, like this: SRC_URI = "git://${GIT_SERVER}/repo1;name=repo1;destsuffix=repo1 \ git://${GIT_SERVER}/repo2;name=repo2;destsuffix=repo1/repo2 \ " Git status reports, for repo1: Untracked files: (use "git add <file>..." to include in what will be committed) repo2/ If we run `devtool modify` on this recipe, do_patch runs with: PATCHTOOL = "git" PATCH_COMMIT_FUNCTIONS = "1" The `patch_task_postfunc` (patch.bbclass, line 82) runs a `git add .` on the top-level repo1, leaving the checkout in an invalid state. The following git warning does not appear in the log: $ git add . warning: adding embedded git repository: repo2 hint: You've added another git repository inside your current repository. hint: Clones of the outer repository will not contain the contents of hint: the embedded repository and will not know how to obtain it. hint: If you meant to add a submodule, use: hint: hint: git submodule add <url> repo2 hint: hint: If you added this path by mistake, you can remove it from the hint: index with: hint: hint: git rm --cached repo2 hint: hint: See "git help submodule" for more information. $ git submodule status fatal: no submodule mapping found in .gitmodules for path 'repo2' No further git submodule commands can be run on the checkout. We could enhance the `patch_task_postfunc` to look for any embedded git checkouts and add them as submodules, but this seems unnecessary complexity for an obscure edge-case. Although the git repo is left in an invalid state with respect to the submodules, it still serves the purpose required by devtool: To take further commits, and generate patch files from them. We are still able to run these commands to examine any submodules, where git submodule--helper reports paths relative to the top of the checkout: $ git ls-files --stage | grep ^160000 160000 5feee12d6e974dd8c0614cf5b593380b046439a5 0 repo2 $ git submodule--helper list 160000 5feee12d6e974dd8c0614cf5b593380b046439a5 0 repo2 When a recipe sets EXTERNALSRC to a subdirectory of the git checkout, we test for the existence of the reported submodule paths within the EXTERNALSRC directory. The latest versions of git submodule--helper accept a path to a subdirectory and correctly report no submodules within that subdirectory. Regrettably, we still support git versions that don't accept a path to a subdirectory. [YOCTO #14333] (From OE-Core rev: 2055718fdd19f925e236d67823017323bbd92a4b) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Detect code changes in submodulesDouglas Royds2021-03-201-7/+9
| | | | | | | | | | | | | Further to 50ff9afb39, only detect code changes in submodules that are subdirectories of the EXTERNALSRC directory. git submodule status returns a path relative to the cwd for each submodule. We don't add submodules that are not within our source subtree. (From OE-Core rev: 4525310d49d115a37705f04ac5c03d639e5e8f8c) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Pass through npmsw URIs in SRC_URIMike Crowe2021-02-261-0/+1
| | | | | | | | | | | NPM shrinkwrap files need to stay in SRC_URI even when using externalsrc so that npm_do_fetch can run to fetch the required dependencies. (From OE-Core rev: e889cbecf9a1731b199e57d79a14e574398586bb) Signed-off-by: Mike Crowe <mac@mcrowe.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Detect code changes in submodulesTomasz Dziendzielski2021-01-281-1/+11
| | | | | | | | | | | | | | | The srctree_hash was calculated only from main source directory ignoring changes in submodules. [YOCTO #13748] Use submodule--helper to determine all submodules, and calculate hash from all git tree objects names combined. (From OE-Core rev: 50ff9afb3990bcf60b4fa1f937506cb84028c32d) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Fix parsing error with devtool non-git sourcesTomasz Dziendzielski2021-01-231-0/+4
| | | | | | | | | | | | | | | | | | If srcdir is under poky directory (e.g. devtool poky/build/workspace/sources) and is not a git repository then ${@srctree_hash_files(d)} will run "git rev-parse --git-dir" and detect poky directory as git-dir and run "'git', 'add', '-A', '.'], cwd=s_dir" trying to add srcdir but build dir is in .gitignore and latest git will fail with "The following paths are ignored by one of your .gitignore files: build" which will end with "ExpansionError during parsing". In this commit I added a check if git_dir is the same as git-dir from TOPDIR (which will detect poky directory) and if yes, then treat srcdir as non-git sources. (From OE-Core rev: 95fbac8dcad6c93f4c9737e9fe13e92ab6befa09) Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: No single-task lock if S != BDouglas Royds2020-10-171-1/+1
| | | | | | | | | | Allow different recipes to build things from the one external source tree in parallel, but only if the build is happening outside the source tree. (From OE-Core rev: 8c6b12683c3160a5b7c62d1fe00a9f848e062df0) Signed-off-by: Douglas Royds <douglas.royds@taitradio.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: always set SRCPVAlexander Kanavin2019-12-281-3/+2
| | | | | | | | | | | | If devupstream is in use by the recipe, then that variant has SRCPV, but the main variant may not, in which case bitbake throws an expansion error. (From OE-Core rev: 740e3f5e1d4a8abb07c7a4e4aedd804d679c7562) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: stop rebuilds of 2+ externalsrc recipes sharing the same git repoQuentin Schulz2019-09-191-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | externalsrc do_configure task watches oe-devtool-tree-sha1 file and its checksum. That file basically contains the result of `git add -A ${EXTERNALSRC} && git write-tree` which is the hash of temporary "commit" of the non committed changes. This file is stored in the .git directory of the git repo of the externalsrc recipe. do_configure then depends on the checksum of oe-devtool-tree-sha1 file. If 2+ recipes with different externalsrc paths but same git repo (e.g. one recipe at /some/path and the other at /some/path/subdir) are parsed, this oe-devtool-tree-sha1 will be overwritten by those recipes at parsing time since .git is shared between those recipes. If there is one non committed git change in /some/path but not in /some/path/subdir, the oe-devtool-tree-sha1 of both recipes will be different. What will happen is that recipe1 will watch over the oe-devtool-tree-sha1 with a specific checksum, fill in file-checksums for do_configure correctly, then recipe2 will watch over the identically named file with different content also fill in the file-checksums varflag. When do_configure of recipe1 will be evaluated for re-execution, oe-devtool-tree-sha1 will be of the value of what is watched over by recipe2, thus triggering a rebuild of recipe1. This behavior is not always reproducible which I'm guessing is due to a small window between recipe1 putting info into oe-devtool-tree-sha1 and calculating the checksum of that file and recipe2 putting its content into oe-devtool-tree-sha1. By appending the name of the recipe to oe-devtool-tree-sha1, we make sure that a recipe won't have its oe-devtool-tree-sha1 overwritten by another recipe sharing the same externalsrc git repo. (From OE-Core rev: 1b727dd7295a7a7fe17800f8038242efbf7fe2b7) Signed-off-by: Quentin Schulz <quentin.schulz@streamunlimited.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc.bbclass: Set BB_DONT_CACHE for non-target recipesOla x Nilsson2018-10-081-2/+2
| | | | | | | | | | | BB_DONT_CACHE was not set for non-virtual recipes where PN != BPN, such as quilt-native. Recipes that do not set BBCLASSEXTEND should always have BB_DONT_CACHE set by externalsrc. (From OE-Core rev: 4eff427a0ee629a1541a420a9591411648569a97) Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc.bbclass: don't configure with --disable-dependency-trackingAndre McCurdy2018-05-291-0/+3
| | | | | | | | | | | | | One of the uses of externalsrc is to enable iterative editing and rebuilding of source files during development. In such situations, disabling Automake dependency tracking can lead to sources not being rebuilt even though files they depend on have been modified. (From OE-Core rev: af2f802d5b59203a887982af83252565b8078085) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/externalsrc: ensure cleandirs code handles non-absolute pathsPaul Eggleton2018-04-051-1/+2
| | | | | | | | | | | | | | | | It's possible that a trailing or extra slash somewhere in the external source path could result in the directory not being removed from cleandirs; it's also possible that a cleandirs entry is somewhere underneath the source tree and that tree should never have parts of it deleted by the build system. Use oe.path.is_path_parent() (which makes paths absolute before checking them) to find out if any path in cleandirs is anywhere underneath the external source path, and drop it if it is. (From OE-Core rev: 87ec0fb470274d980f8553a85f778809c269c5d7) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/externalsrc: handle if cleandirs contains python expressionsPaul Eggleton2018-04-051-1/+3
| | | | | | | | | | | | | | Use the existing oe.recipeutils.split_var_value() function to split the unexpanded value of the cleandirs varflag, in case it contains python expressions - we don't want to split the expression itself as the chunks will not expand properly individually and we can miss something that expands to the source tree (and thus it can get deleted, the avoidance of which is the whole point of this code). (From OE-Core rev: 611cee69d7300a4746edd752c9557af60e7beecc) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: do not call make clean for recipes with CLEANBROKEN = "1" setAndreas Müller2018-03-251-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | ERROR: distrho-ports-0.0.0+git999-r0 do_buildclean: oe_runmake failed ERROR: distrho-ports-0.0.0+git999-r0 do_buildclean: Function failed: do_buildclean ERROR: Logfile of failure stored in: <...>/temp/log.do_buildclean.17285 Log data follows: | DEBUG: Executing shell function do_buildclean | NOTE: make clean | make clean -C libs/drowaudio | make[1]: Entering directory '/home/a.mueller/data/oe-core/workspace/sources/distrho-ports/libs/drowaudio' | make clean -C build-drowaudio | make[2]: Entering directory '/home/a.mueller/data/oe-core/workspace/sources/distrho-ports/libs/drowaudio/build-drowaudio' | make[2]: *** No rule to make target 'clean'. Stop. | make[2]: Leaving directory '/home/a.mueller/data/oe-core/workspace/sources/distrho-ports/libs/drowaudio/build-drowaudio' | make[1]: *** [Makefile:7: clean] Error 2 | make[1]: Leaving directory '/home/a.mueller/data/oe-core/workspace/sources/distrho-ports/libs/drowaudio' | make: *** [Makefile:73: clean] Error 2 | ERROR: oe_runmake failed (From OE-Core rev: 21fa28d8faf33b2717e38886352238dd360ef346) Signed-off-by: Andreas Müller <schnitzeltony@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc.bbclass: Suppress git errorsJoshua Watt2018-02-161-1/+1
| | | | | | | | | | | Suppress any warnings git might generate when searching for a valid git directory, as there are use cases where the directory is expected to not exist and the warning is superfluous (From OE-Core rev: 66011996e1a8b738b31466fccad9973f8b48f71d) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: fix ExpansionError if the source dir does not exist yetLuca Ceresoli2017-11-081-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The externalsrc class code assumes that the source directory (EXTERNALSRC) exists before bitbake is called. Otherwise do_configure will fail obviously since externalsrc does not fetch anything. Commit 3ca6085729d9 ("externalsrc: Handle .git not being a directory") changed this behaviour. Now on a missing EXTERNALSRC directory we get a bb.data_smart.ExpansionError during _parsing_, way before do_configure can be run. This new behaviour creates two problems: * First, there error message is very cryptic (and it's hard to provide a better message since no task is ever run): ERROR: ExpansionError during parsing /<...>/<...>.bb Traceback (most recent call last): bb.data_smart.ExpansionError: Failure expanding variable do_compile[file-checksums], expression was ${@srctree_hash_files(d)} which triggered exception FileNotFoundError: [Errno 2] No such file or directory: '<...>' * Second, this prevents creating a class based on externalsrc that automatically fetches the code in EXTERNALSRC before do_compile runs. Fix both problems by simply calling git with '-C ${EXTERNALSRC}' instead of calling git inside the non-existing directory. This changes from a bb.data_smart.ExpansionError to a subprocess.CalledProcessError, which is in line with what's actually going on: git is telling us it can't find the git dir. Also remove a comment that does not apply anymore. (From OE-Core rev: 8e8251582e977d7c6ab525ac1b493fbbe3af38b2) Signed-off-by: Luca Ceresoli <luca@lucaceresoli.net> Cc: Joshua Watt <jpewhacker@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/externalsrc: delete symlinks at start of do_buildcleanPaul Eggleton2017-09-211-0/+1
| | | | | | | | | | | | | | | | | | | | | | | To help users find the work and log directories (especially within the eSDK) we create symlinks to these from the source tree. However during do_buildclean we call "make clean", and some project Makefiles which delete something like "*/*/lib" which will match files underneath the oe-workdir symlink and fail. do_buildclean is called from do_clean which is in turn called by devtool reset by default, and thus devtool reset is blocked. An example of a recipe where this is visible is the openssl-qat recipe in meta-intel. In order to fix this, delete the symlinks at the start of do_buildclean since we shouldn't need them at that point anyway, and thus make clean won't be able to trip over them. Fixes [YOCTO #11036]. (From OE-Core rev: 927b3b05a7f6b85967f7375b1942605fe8d37d0c) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Handle .git not being a directoryJoshua Watt2017-08-161-3/+11
| | | | | | | | | | | | Use git rev-parse to determine the location of the .git directory, in case it is not an immediate child of EXTERNALSRC (e.g. when using submodules). In the event git can't resolve the .git directory, fall back to the non-git method for hashing. (From OE-Core rev: 95e1341b49f7184d280a03f64f131a4468a06867) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: place copy of git index into /tmp and do not use copyfile2Enrico Scholz2017-07-251-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Using shutil.copy2() to copy .git/index to a temporary file tries to copy SELinux attributes which might fail for confined users in SELinux environments. E.g. our builders are running in docker containers and modification of sources (inclusive updated of .git/index) is done outside. Trying to copy .git/index fails with | $ python3 -c 'import shutil; shutil.copy2("index", "a")' | ... | PermissionError: [Errno 13] Permission denied: 'a' and an AVC like | denied { relabelto } for pid=18043 comm="python3" name="a" dev="dm-29" ino=1067553 scontext=system_u:system_r:container_t:s0:c39,c558 tcontext=unconfined_u:object_r:build_file_t:s0 tclass=file permissive=0 is created. This can not be solved by adapting the SELinux policy because this is a very deep constraint violation: | constrain file { create relabelfrom relabelto } ((u1 == u2 -Fail-) or (t1 == can_change_object_identity -Fail-) ); Constraint DENIED | | Possible cause is the source user (system_u) and target user (unconfined_u) are different. I do not see much sense in using 'shutil.copy2()' here; 'shutil.copyfile()' seems to be a better choice (target file is created in a secure way by tempfile.NamedTemporaryFile()). By placing the tempfile into /tmp we avoid potential problems related to git's 'core.sharedRepository'. As a (positive) side effect, the source tree will not be modified anymore (at least by this part of code) which prevented to mount it read-only from somewhere else. (From OE-Core rev: 3c3c8ecc61dfed68987750d79b5482ab2f6fa02f) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: verify that EXTERNALSRC/EXTERNALSRC_BUILD are absolute pathsRoss Burton2017-06-141-1/+6
| | | | | | | | | If these are set to URLs then the errors produced are not helpful. (From OE-Core rev: 946b6623154e748a0d75ff498802a720aeec27a6) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Use git add -A for compatibility with all git versionsRichard Purdie2017-03-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | I've been debugging a selftest failure on Centos7. The problem turns out to be the elderly git version (1.8.3.1) on those systems. It means that the system doesn't correctly checksum changed files in the source tree, which in turn means do_compile fails to run and this leads to the following selftest failure: ====================================================================== FAIL [141.373s]: test_devtool_buildclean (oeqa.selftest.devtool.DevtoolTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/devtool.py", line 530, in test_devtool_buildclean assertFile(tempdir_mdadm, 'mdadm') File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/devtool.py", line 497, in assertFile self.assertTrue(os.path.exists(f), "%r does not exist" % f) AssertionError: False is not true : '/tmp/devtoolqag88s39z8/mdadm' does not exist The solution is to use -A on the git add commandline which matches the behaviour in git 2.0+ versions and resolves the problem. (From OE-Core rev: 964e8b8cae4b28e21ade12b5effb494e459b1f0f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Hide created symlinks with .git/info/excludeOla x Nilsson2017-01-311-1/+16
| | | | | | | | | | | Add created symlinks to the exclude file. This will both make them less distracting and hide them from the srctree_hash_files function. (From OE-Core rev: c11fcd6fbde4a90913960b223451e0ce9e6b4b64) Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc.bbclass: Add task buildcleanOla x Nilsson2017-01-161-1/+18
| | | | | | | | | | | | | | | | | | The buildclean task should call the package build system clean command, just implemented for Make for now. This is meant for recipes where S == B, but can be useful as a standalone task for other recipes too. When S == B, set it to run before do_clean which will do what most developers expect when calling bitbake -c clean. For S != B, do not add it before clean as it is not needed and may take some time. (From OE-Core rev: cfaad320d9565003e97893efcb14d00d0b8e23bb) Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Add optional srcdir arg to srctree_hash_filesOla x Nilsson2017-01-091-3/+3
| | | | | | | | | | Make it easier to reuse the function for other dirs than EXTERNALSRC. (From OE-Core rev: 9ddcfb51e637acba82089da6430ac77e29f0f1ef) Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: Set STAMPCLEAN to match STAMPOla x Nilsson2017-01-091-0/+1
| | | | | | | | | | | Only the last stamp file should be kept, but unless STAMPCLEAN matches files generated using STAMP old stamp files may linger. This may cause false positives for skipping tasks. (From OE-Core rev: 6d60f60e38e0e00e6753f5b136277f27d6204e63) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: remove True option to getVarFlag callsJoshua Lock2016-12-161-1/+1
| | | | | | | | | | | | | | | getVarFlag() now defaults to expanding by default, thus remove the True option from getVarFlag() calls with a regex search and replace. Search made with the following regex: getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\) (From OE-Core rev: 2dea9e490a98377010b3d4118d054814c317a735) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: remove True option to getVar callsJoshua Lock2016-12-161-19/+19
| | | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/externalsrc: re-run do_configure when configure files changePaul Eggleton2016-10-111-0/+22
| | | | | | | | | | | | | | | | | | If the user modifies files such as CMakeLists.txt in the case of cmake, we want do_configure to re-run so that those changes can take effect. In order to accomplish that, have a variable CONFIGURE_FILES which specifies a list of files that will be put into do_configure's checksum (either full paths, or just filenames which will be searched for in the entire source tree). CONFIGURE_FILES then just needs to be set appropriately depending on what do_configure is doing; for now I've set this for autotools and cmake which are the most common cases. Fixes [YOCTO #7617]. (From OE-Core rev: 923fc20c2862a6d75f949082c9f6532ab7e2d2cd) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc.bbclass: Make reparsing work for BBCLASSEXTENDed recipesPeter Kjellerstedt2016-09-201-3/+17
| | | | | | | | | | | | | | | | | To make sure changes to any source files are detected when externalsrc is used, it sets BB_DONT_CACHE to force the recipe to be reparsed every time. Previously, this was done conditionally based on whether EXTERNALSRC was set. This worked fine for building the base recipe. But if one tried to build, e.g., a native version of it (provided via BBCLASSEXTEND), the recipe would not be reparsed as expected. To solve the above problem, BB_DONT_CACHE is now set for the base recipe if EXTERNALSRC is set for it or any of it derivatives. (From OE-Core rev: 449a0b21255d895e8620383ce76a9d7ea41b5cc6) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/lib: Update to use python3 command pipeline decodingRichard Purdie2016-06-021-1/+1
| | | | | | | | | | | In python3, strings are unicode by default. We need to encode/decode from command pipelines and other places where we interface with the real world using the correct locales. This patch updates various call sites to use the correct encoding/decodings. (From OE-Core rev: bb4685af1bffe17b3aa92a6d21398f38a44ea874) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: avoid race in temporary git index fileMarkus Lehtonen2016-04-061-8/+9
| | | | | | | | | | | | | Use a unique tempfile as the temporary git index file when determining the git hash of the source tree. A fixed filename was obviously causing races (with the git index.lock file) under oe-selftest where multiple bitbake instances were run against the same source tree at the same time. (From OE-Core rev: f81c641022c26a9b89fac769e0f2889eaec5d32f) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: remove nostamp from do_configureMarkus Lehtonen2016-03-281-1/+1
| | | | | | | | | | | | | | | | | | | | | | | Extend the srctree_hash_files() function in externalsrc.bbclass to handle non-Git source trees. If the source tree is not a git repository, srctree_hash_files() now simply adds the whole source tree as a dependency, causing bitbake to basically hash every file in it. Hidden files and directories in the source tree root are ignored by the glob currently used. This has the advantage of automatically ignoring .git directory, for example. During the first bitbake run preparing of the task runqueue may take much longer if the source tree is not a git repository. The reason is that all the files in the source tree are hashed. Subsequent builds are not significantly slower because (most) file hashes are found from the cache. [YOCTO #8853] (From OE-Core rev: a797abe08bfe4144e120c984d1718c2cc3d87c0a) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: do not use do_configure[nostamp] for git srctreesMarkus Lehtonen2016-03-281-2/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Be a bit more intelligent than mindlessly re-compiling every time. Instead of always using 'nostamp' flag for do_compile run a python function to get a list of files to add as 'file-checksums' flag. The intention is to only re-run do_compile if something in the source tree content changes. This python function, srctree_hash_files(), works differently, depending if the source tree is a git repository clone or not. If the source tree is a git repository, the function runs 'git add .' and 'git write tree' to get a hash of the working tree and writes this hash into a file under the .git directory. This file containing the hash is then returned as the file for the task to depend on. Hash is changed if any changes are made in the source tree causing the task to be re-run. A trick is used to parse the recipe every time so that the hash file gets updated. If the source tree is not a git repository behaviour remains the same. In this case srctree_hash_files() currently sets the 'nostamp' flag for do_compile causing it to be re-run every time. This method of tracking changes source tree changes to determine if re-build is needed does not work perofectly, though. Many packages are built under ${S} which effectively changes the source tree causing some unwanted re-compilations. However, if do_compile of the recipe does not produce new/different artefacts on every run (as commonly is and should be the case) the re-compilation loop stops. Thus, you should usually see only one re-compilation (if any) after which the source tree is "stabilized" and no more re-compilations happen. [YOCTO #8853] (From OE-Core rev: a26becdf981b35d7ef8524f9e65c25a74b842f1d) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* externalsrc: use shared stamp directory if B=SMarkus Lehtonen2016-03-201-6/+4
| | | | | | | | | | | | | | | | | | | | | | | We have a common build directory for all target architectures if externalsrc is used and ${B}=${S}. In this case we also need to have a common stamp directory. The reason is that e.g. changing MACHINE will basically "invalidate" the artefacts in ${B} (wrt. to the new MACHINE) but old stamp files with matching hashes from an earlier build might be available in the machine-specific stamps directory and tasks would not be correctly re-run. Previous attempt in f44f12af346888bdeb3ae01a275cb5dd4396b505 to handle this correctly was faulty as it threw some spurious warnings and caused do_configure to be re-run in every build if that would not be necessary. [YOCTO #8950] [YOCTO #9237] (From OE-Core rev: f5031fb37ceef83c85f0dfec773da75c05b2a631) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/externalsrc: fix symlinking if symlink exists pointing to another pathPaul Eggleton2016-03-021-0/+1
| | | | | | | | | | | If the oe-workdir / oe-logs symlink exists and points to a different path then the symlink needs to be removed before calling os.symlink() or it'll fail. (From OE-Core rev: cb8f064e48c24dcb1a15a32cef3399f35e549bdc) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* classes/externalsrc: create symlinks for workdir and logsPaul Eggleton2016-02-111-0/+20
| | | | | | | | | | | | | | Auto-create symlinks in the source directory to the work directory (${WORKDIR}) and logs directory (${T}) so that they are easier for the user to find. This is particularly useful within the extensible SDK where the user is less likely to be familiar enough with the structure of the build system to know where to find these things, but otherwise they are a useful shortcut for anyone. (From OE-Core rev: 58adb3904c18acefd0da319e32f66ebca72eeaac) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-15 16:51:35 +0000