summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/utils.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: utils: Fix lockfile path length issuesRichard Purdie2022-03-241-3/+6
| | | | | | | | | | | | | If the path to bitbake.lock is in a deep directory, bitbake will hang. The reason was that the max file length limiting code (to 255 chars) was including the directory name and it should only act on the filename within the directory. Fix it to just use the base filename. [YOCTO #14766] (Bitbake rev: 89d70e7b71eecfe06592202f326e566c579ba01d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils/ply: Change md5 usages to work on FIPS enabled hostsMark Hatle2022-03-021-1/+1
| | | | | | | | | | | | | | | | | | | | hashlib.md5() is not permitted on a FIPS enabled host system. This is due to md5 not being an approved hash algorithm. Instead use: hashlib.new('MD5', usedforsecurity=False) This is allowed, as it's clear the hash is used for a non-security purpose. Note: utils.py version should never be used to verify file integrity, but instead be used to identify if the file may have changed. sha256 should be used for integrity purposes. (Bitbake rev: af866dd077867cba0129757bfcc689551445e9d7) Signed-off-by: Mark Hatle <mark.hatle@xilinx.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename environment filtering variablesScott Murray2022-02-211-10/+10
| | | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_ENV_WHITELIST -> BB_ENV_PASSTHROUGH BB_ENV_EXTRAWHITE -> BB_ENV_PASSTHROUGH_ADDITIONS (Bitbake rev: fe60627839d4280cf0117ed1afbfccdff1181b6a) (Bitbake rev: 87104b6a167188921da157c7dba45938849fb22a) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Ensure shell function failure in python logging is correctRichard Purdie2022-02-211-1/+1
| | | | | | | | | | | | | If a python function exec_func() calls a shell task, the logging wasn't working correctly in all cases since the exception was turned into a BBHandledException() and the logfile piece was lost which is handled at the top task level. The easiest way to avoid this is to allow the ExecutionError exceptions to be raised to a higher level, we don't need the traceback for them. (Bitbake rev: 7cae11f558f9ff5fd05ef23b789aaef92fb5a327) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix typo "dependency" and "spawning"Zygmunt Krynicki2022-02-201-2/+2
| | | | | | | (Bitbake rev: 8f76117816fa84367753d5eee0264b90e9f29a8c) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix environment decorator logic errorRichard Purdie2022-02-201-1/+1
| | | | | | | | | | | | | | | | | | | File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 328, in checkstatus with bb.utils.environment(**newenv): File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/utils.py", line 1737, in environment del os.environ[var] File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/os.py", line 695, in __delitem__ raise KeyError(key) from None It is possible the wrapped function might change the environment which can lead to KeyError issues like this. The key may no longer be in the original environment so handle that case correctly and avoid the tracebacks. (Bitbake rev: ae4ce4c21998bb2a12a1e1f0b8e0af8d218b87e2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: bitbake-worker: Preserve network non-local uidRobert Yang2022-02-011-0/+16
| | | | | | | | | | | | | | | | The NIS can't work when network is dissable, so preserve network for it, the error is like: do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not. (Bitbake rev: 4eafae7904bae6e5c6bc50356e8a9077f2e207fa) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add disable_network functionRichard Purdie2022-01-121-0/+31
| | | | | | | | | | | | Add a function which uses the unshare glibc call to disable networking in the current process. This doesn't work on older distros/kernels but will on more recent ones so for now we simply ignore the cases we can't execute on. uid/gid can be passed in externally so this can work with pseudo/fakeroot contexts. (Bitbake rev: 9d6341df611a1725090444f6f8eb0244aed08213) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Update to use exec_module() instead of load_module()Richard Purdie2022-01-051-2/+5
| | | | | | | | | This is deprecated in python 3.12 and Fedora 35 is throwing warnings so move to the new functions. (Bitbake rev: 68a18fbcb5959e334cf307d7fa8dc63832edb942) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Handle lockfile filenames that are too long for filesystemsRichard Purdie2021-11-211-1/+5
| | | | | | | | | | | | | The fetcher mirror code can go crazy creating lock filenames which exceed the filesystem limits. When this happens, the code will loop/hang. Handle the filename too long exception correctly but also truncate lockfile lengths to under 256 since the worst case situation is lockfile overlap and lack of parallelism. (Bitbake rev: 63baf3440b16e41ac6601de21ced94a94bdf1509) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add environment updating context managerRoss Burton2021-08-121-0/+16
| | | | | | | | | | bb.utils.environment() is a context manager to alter os.environ inside a specific block, restoring it after the block is closed. (Bitbake rev: 9974848f67581ff7d76cef52a94f505af99b4932) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/tests/toaster: Override conversion fixupsRichard Purdie2021-08-061-1/+1
| | | | | | | | | Fix some references that missed during the overrides syntax migration or were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch. (Bitbake rev: 6184cb07dfa44f5f76f1c423533b4547d80b20ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add bb.utils.rename() helper function and use for renamingDevendra Tewari2021-05-061-3/+15
| | | | | | | | | | | | | | | | | | | | os.rename can fail for example an incremental build in Docker fails with: OSError: [Errno 18] Invalid cross-device link when source and destination are on different overlay filesystems. Rather than trying to fix every call site, add a wrapper in bb.utils for renames. We can then handle cross device failures and fall back to shutil.move. The reason os.rename is still used is because shutil.move is too slow for speed sensitive sections of code. [YOCTO #14301] (Bitbake rev: c5c4e49574ab2a65e06298a0a77bb98b041cf56b) Signed-off-by: Devendra Tewari <devendra.tewari@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-101-3/+3
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add docstrings to functionsMilan Shah2021-01-081-4/+34
| | | | | | | | | | | | | | | | | | A list of functions that now has a docstring. * vercmp_string * explode_dep_versions * prunedir * prune_suffix * to_boolean * contains_any * export_proxies See [YOCTO #9725] for details. (Bitbake rev: b61ba4a18693a9e553d2a93161feb0bcc1c82384) Signed-off-by: Milan Shah <mshah@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add umask changing context managerRoss Burton2020-09-301-0/+11
| | | | | | | | | | Add a umask context manager which can be used to temporarily change the umask in a 'with' block. (Bitbake rev: 6c601e68a27e1c60b04c2a61830d1812cc883e09) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: fix UnboundLocalError when _print_exception raisesChris Laplante2020-09-101-2/+2
| | | | | | | | | | PEP 3110 changed how exceptions work. 'e' is unbound after the 'except' clause. See: https://www.python.org/dev/peps/pep-3110/#semantic-changes (Bitbake rev: b69e97de53eb172ed730993e3b755debaa26f30d) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils.py: get_file_layer(): Improve performanceRobert Yang2020-09-101-3/+9
| | | | | | | | | | | | | | | | | The following code costs a lot of time when there are lot of layers and recipes: for collection in collections: collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or '' My build has more than 100 layers and 3000 recipes, which calls d.getVar() 300K (3000 * 100) times and makes 'bitbake-layers show-recipes' very slow, add a keyword argument to get_file_layer() can fix the problem, it can save about 90% time in my build (6min -> 40s). (Bitbake rev: f08a6601c9bb09622855d62e1cedb92fafd2f71d) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils.py: get_file_layer(): Exit the loop when file is matchedRobert Yang2020-09-101-2/+3
| | | | | | | | | | | | | | | | | | | This can make "$ bitbake-layers show-recipes" save about 60% time (14min -> 6min) in my build (more than 3000 recipes) The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with each recipe, and get_file_layer() compare the file with each item in BBFILES which makes it very time consuming when there are a lot of recipes and items in BBFILES. So use BBFILES_PRIORITIZED and exit when file is matched, it doesn't make sense to go on the loop when file is matched. And use fnmatchcase to replace of fnmatch since the comparison should be case-sensitive. (Bitbake rev: 8d64181d29dc262e066a6114dd51e5f2d04f47de) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: process_profilelog: use context managerChris Laplante2020-09-101-15/+14
| | | | | | | (Bitbake rev: 42172900af06baeee559d33b150d5febdf8e960a) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Drop broken timeout functionRichard Purdie2020-08-261-14/+0
| | | | | | | | | I strongly suspect this function doesn't work with modern python so and its unused now, drop it. (Bitbake rev: a3033cea089c66c8b4614e7ee57c166f4262c590) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils.py: Do not preserve TERM in the environmentJacob Kroon2020-06-161-1/+0
| | | | | | | | | | | The value of TERM is leaking into OE-Core postinst-useradd-${PN} scripts, which in turn can optionally be monitored by buildhistory. Prune the value in order to make the OE-Core buildhistory output more deterministic. (Bitbake rev: 0d5cdd0c0d65f2f81c3af0f3767fee86c4142c3a) Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils.py: Preserve ownership of symlinkDaisuke Yamane2020-02-221-1/+1
| | | | | | | | | | | Yocto Bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13806 Uncomment lchown() to preserve ownership of symlink. (Bitbake rev: 6a0b6dd17c6d842960d448114b252e92c55dea33) Signed-off-by: Daisuke Yamane <daisuke.yamane@cybertrust.co.jp> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add is_semver functionJean-Marie LEMETAYER2020-01-271-0/+26
| | | | | | | | | | | | This function checks if a string is a semantic version: https://semver.org/spec/v2.0.0.html The npm fetcher needs this function to validate its version parameter. (Bitbake rev: 61ac4e825fa7afbb76282030586abc9ee4ac215c) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add sha384_file and sha512_file functionsJean-Marie LEMETAYER2020-01-271-0/+14
| | | | | | | | | | The npm fetcher needs these functions to support the subresource integrity: https://www.w3.org/TR/SRI/ (Bitbake rev: 80e2216e2b41cb6170292009064864449bc48bbe) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils: add get_referenced_varsChris Laplante via bitbake-devel2020-01-191-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | Given a start expression, bb.utils.get_referenced_vars returns the referenced variable names in a quasi-BFS order (variables within the same level are ordered aribitrarily). For example, given an empty data store: bb.utils.get_referenced_vars("${A} ${B} ${d.getVar('C')}", d) returns either ["A", "B", "C"], ["A", "C", "B"], or another permutation. If we then set A = "${F} ${G}", then the same call will return a permutation of [A, B, C] concatenated with a permutation of [F, G]. This method is like a version of d.expandWithRefs().references that gives some insight into the depth of variable references. (Bitbake rev: 076eb5453ca35b8b75b8270efb989d5208095b27) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: remove unused importsFrazer Clews2020-01-191-1/+0
| | | | | | | | | | removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils.fileslocked: don't leak files if yield throwsChris Laplante via bitbake-devel2019-12-161-4/+5
| | | | | | | | | | | | | Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by externalsrc.bbclass) was leaked, giving a warning like: WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'> exec_func(task, localdata) (Bitbake rev: 6beddf6214e22b4002626761031a9e9d34fb04db) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: also use mmap for SHA256 and SHA1, for performanceRoss Burton2019-11-211-20/+14
| | | | | | | | | | md5_file() uses a mmap() window to improve performance when hashing files, so refactor the code and do the same for SHA1 and SHA256. (Bitbake rev: ecf87437ff796e17c3e4f210b5803b0136a9e8a4) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add ionice option to prunedirRichard Purdie2019-09-191-4/+7
| | | | | | | | | | | | | | | | Autobuilder type infrastructure can benefit from deletion of certain files as background IO due to the way Linux filesystem priority works. We have problems where build directories as part of oe-selftest being delete starves the running tasks of IO to the point builds take much longer to compelte. Having this option of running the deletion at "idle" helps a lot with that. (Bitbake rev: 797354d285f6d624d9adb52bab65823572da0e39) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix movefile() exception handling with python3Martin Jansa2019-09-031-1/+1
| | | | | | | | | | | | | | | | | | | | * with python3 this fails with: File: 'bitbake/lib/bb/utils.py', lineno: 799, function: movefile 0795: try: 0796: os.rename(src, destpath) 0797: renamefailed = 0 0798: except Exception as e: *** 0799: if e[0] != errno.EXDEV: 0800: # Some random error. 0801: print("movefile: Failed to move", src, "to", dest, e) 0802: return None 0803: # Invalid cross-device-link 'bind' mounted or actually Cross-Device Exception: TypeError: 'OSError' object is not subscriptable (Bitbake rev: d6e43c443ddbbe467c4380c48d2bc28ae18504a1) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/utils: Drop bb.build.FuncFailedRichard Purdie2019-08-061-1/+1
| | | | | | | | | | | | | Its hard to see what this exception adds in the current codebase. The logfile attribute is effectively ignored, the exception doesn't serve a defined purpose and mostly seems to be worked around. Remove it entirely. If this does cause output problems, we'll figure out better ways to address those. (Bitbake rev: cfeffb602dd5319f071cd6bcf84139ec77f2d170) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Strip old editor directives from file headersRichard Purdie2019-05-041-2/+0
| | | | | | | | | | There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie2019-05-041-12/+0
| | | | | | | | | | With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie2019-05-041-0/+2
| | | | | | | | | | | | | | | | | This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Let mkdirhier fail if existing path is not a folderAndre Rosa2019-04-121-1/+1
| | | | | | | | | | Let mkdirhier fail if existing path is not a folder instead of assuming a directory hierarchy already exists. (Bitbake rev: a8d9b82ccf93dcb74258693f62d88be380b1c0b7) Signed-off-by: Andre Rosa <andre.rosa@lge.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Make prune_suffix prune a suffixAndre Rosa2019-04-091-2/+2
| | | | | | | | | | Make prune_suffix prune a suffix instead of replacing a substring that could happen more than once and not only when it ends with it. (Bitbake rev: 57e765e38c6382a9b36d5ee2a6f3fa96ac905b82) Signed-off-by: Andre Rosa <andre.rosa@lge.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fix version comparison when one of the versions ends in .Alexander Kanavin2019-02-251-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | Previously, this would happen: ====================================================================== ERROR: test_vercmpstring (bb.tests.utils.VerCmpString) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alexander/development/poky/bitbake/lib/bb/tests/utils.py", line 45, in test_vercmpstring result = bb.utils.vercmp_string('1.', '1.1') File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 143, in vercmp_string return vercmp(ta, tb) File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 135, in vercmp r = vercmp_part(va, vb) File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 124, in vercmp_part elif ca < cb: TypeError: '<' not supported between instances of 'NoneType' and 'int' ---------------------------------------------------------------------- (Bitbake rev: fef56d28c3efec4876c379898cbc4d4c65303aee) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add aarch64 support to ioprio_setRichard Purdie2018-12-141-0/+2
| | | | | | | | | With aarch64 hosts coming into use, set the syscall number to avoid ioprio warnings on that platform. (Bitbake rev: 5eaf9e7b26f09f5f106e1c3c6976d517b289450a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Use utils.remove() within prunedir()Richard Purdie2018-12-081-9/+1
| | | | | | | | | | | | We've already tweaked remove() for speed and not to error if it races for deletion. Therefore use this for prunedir() which was starting to show the same bug reports. [YOCTO #13003] (Bitbake rev: 47f359f9b017f8d99d0bb2161ac0dcefcbd915de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: better_compile(): Fix line number when report errorsRobert Yang2018-11-191-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixed: - Add an error line in base.bbclass, e.g.: 15 16 def oe_import(d): 17 import sys 18 Compile error 19 bbpath = d.getVar("BBPATH").split(":") [snip] Note the "Compile error" line, I added it for reporting errors. $ bitbake -p ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 15: The code lines resulting in this error were: 0014: import oe.data 0015: for toimport in oe.data.typed_value("OE_IMPORTS", d): 0016: imported = __import__(toimport) 0017: inject(toimport.split(".", 1)[0], imported) *** 0018: 0019: return "" 0020: SyntaxError: invalid syntax (base.bbclass, line 18) There are 2 problems: - The "line 15" is incorrect, it is a blank line, not the error line. - The "*** 0018" points to incorrect position. These two problems would mislead people a lot sometimes. - Now fix it to: $ bitbake -p ERROR: Error in compiling python function in /buildarea1/lyang1/poky/meta/classes/base.bbclass, line 18: The code lines resulting in this error were: 0001:def oe_import(d): 0002: import sys *** 0003: Compile error 0004: bbpath = d.getVar("BBPATH").split(":") [snip] SyntaxError: invalid syntax (base.bbclass, line 18) Please see comments in the code for more details on how it is fixed. (Bitbake rev: bbb3d87d171da38fd8e9bce011d109fba28a75c0) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Avoid regex value escape warningsRichard Purdie2018-11-141-5/+5
| | | | | | | | | | | | Avoid warnings such as: bitbake/lib/bb/utils.py:72: DeprecationWarning: invalid escape sequence \d numeric_regexp = re.compile('^(\d+)(.*)$') bitbake/lib/bb/utils.py:1165: DeprecationWarning: invalid escape sequence \( var_res[var] = re.compile('^(%s%s)[ \\t]*\([ \\t]*\)[ \\t]*{' % (var[:-2].rstrip(), override_re)) (Bitbake rev: bbf3cbae775383265292a778cd522d4e2f69a3a0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Avoid warnings about deprecated imp moduleRichard Purdie2018-11-141-8/+6
| | | | | | | | | | | The imp module is deprecated, port the code over to use importlib. bitbake/lib/bb/utils.py:30: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp (Bitbake rev: 3c2cb35588e91fbd7b136e5e2c78eeb77e126c84) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: lockfile: Fix infinite loopIoan-Adrian Ratiu2018-09-251-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | A nasty corner case leads to a hang when utils.lockfile is called from oe-core's package-manager:deploy_dir_lock (in turn called from rootfs:_create further up the call stack) with "name" owned by root and the user running bitbake has no write access. Because this code runs under pseudo, the UID and EUID of the bitbake worker process are 0, so the os.access(dirname, os.W_OK) returns True i.e. it thinks the path is writable when in fact it's not writable. Only later when trying to open the file an Exception it thrown because the OS prohibits writing, but the Exception is ignored and the open is retried leading to an infinite loop. So this fix is to not ignore the "Permission Denied" exception. An alternative fix would be to replace the os.access() call with an try: open() except() at the beginning of the function. (Bitbake rev: 0cb64d0f85b41b2fa764baf6ff7ea1b13f95004e) Signed-off-by: Ioan-Adrian Ratiu <adrian.ratiu@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils.py: add break_hardlinks helperRasmus Villemoes2018-08-161-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | bb.utils.copyfile is called in a few places with identical src and dst in order to create an st_nlinks==1 version of the file. That that even works relies on an implementation detail of copyfile (namely, that it creates a temporary file and then does a rename). Moreover, it's a waste of time if the file already has st_nlinks==1. So create a helper that optimizes away the copy in the st_nlinks==1 case. Of course, this helper relies on the same implementation detail, but that's now contained within bb.utils itself. To test that we do at least sometimes hit the no-copy path, I tested locally with if sstat[stat.ST_NLINK] == 1: + bb.note("Woohoo, 2*%d bytes I/O avoided" % sstat[stat.ST_SIZE]) return True (and the obvious places in oe-core patched), and the do_package log files are indeed filled with woohoo notes. (Bitbake rev: 7ae93cf40ab91965147055100432961436bce46c) Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils/md5_file: don't iterate line-by-lineRoss Burton2018-08-151-4/+9
| | | | | | | | | | | | | | | Opening a file in binary mode and iterating it seems like the simple solution but will still break on newlines, which for binary files isn't really useful as the size of the chunks could be huge or tiny. Instead, let's be a bit more clever: we'll be MD5ing lots of files, but we don't want to fill up memory: use mmap() to open the file and read the file in 8k blocks. (Bitbake rev: a0ac8d67f1471a0c611d691b856fede67efb53f6) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add optional callback to edit_bblayers_conf()Paul Eggleton2018-07-301-1/+14
| | | | | | | | | | | | Add a callback that lets you modify or remove items in addition to the current scheme where you can only add or remove. This enables you to for example replace a layer with a temporary copy (which is what we will use this for first in OE's oe-selftest). (Bitbake rev: bfedb4e85a84e817dbe5d8694b8f8fcdd6f2f22a) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: prevent movefile from changing dest dir permissionsMattias Hansson2018-04-231-2/+2
| | | | | | | | | | | | | | | Prevent movefile from falsely setting the source file's owner and permissions on the destination directory instead of the destination file when moving between devices. This bug caused the last file moved into a directory to dictate the directory's owner and permissions. (Bitbake rev: 82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40) Signed-off-by: Mattias Hansson <mattias.hansson@axis.com> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils.py: Add option for explode_dep_versions2 to return unsortedAmanda Brindle2018-03-151-2/+3
| | | | | | | | | | | | Before, explode_dep_versions2 would sort the OrderedDict before returning. This function will still sort the OrderedDict by default, but will now have the option to return the OrderedDict unsorted. This option will allow us to check if the order of the package list has changed. (Bitbake rev: 39d6a30a28f66c599e18beddbd847f40dcff623c) Signed-off-by: Amanda Brindle <amanda.r.brindle@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: lib/bb/utils: fix movefile() copy to dir fallbackMardegan, Alberto2017-09-251-9/+10
| | | | | | | | | | | When the destination is a directory, building the the destination file path is always needed. That's because even if the copy fallback is taken, it's always followed by a rename. (Bitbake rev: 14c17480827ced2e03c1b62dc839696421fc4de8) Signed-off-by: Alberto Mardegan <amardegan@luxoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2025-12-29 08:41:08 +0000