summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/utils.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: utils: better estimate number of available cpusFelix Moessbauer2024-03-221-1/+4
| | | | | | | | | | | | | When running in a cgroup which is limited to a subset of cpus (via cpuset.cpus), cpu_count() should return the number of cpus that can be used instead of the number of cpus the system has. This also aligns the semantics with nproc. (Bitbake rev: a029bfe96c6542f178720c72a772b7ede9898118) Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: remove BB_ENV_PASSTHROUGH from preserved_envvars()Michael Opdenacker2024-03-051-1/+0
| | | | | | | | | | | | | preserved_envvars() is used when the BB_ENV_PASSTHROUGH environment variable is not set. Therefore, its code shouldn't return this variable. (Bitbake rev: 0a33b560233b983456178541603ab96fea22238b) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Reported-by: Robert P. J. Day <rpjday@crashcourse.ca> Tested-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix mkdir with PosixPathRichard Purdie2023-12-171-1/+1
| | | | | | | | | Avoid: Exception: AttributeError: 'PosixPath' object has no attribute 'find' (Bitbake rev: 0b37fe89ba12549109905b6d0e6d07d342162436) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Do not create directories with ${ in the namePavel Zhukov2023-12-081-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In some cases ${ may not be expanded in the WORKDIR (one of the cases is undefined variable) and causes cryptic failures [1]. Guard this by erroring out if directory name contains ${ Fixes: [Yocto #15255] [1] ERROR: x-native-1.0+${SRCPV}-r0 do_deploy_source_date_epoch: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_hardcode_path(d) 0003: File: '/home/mischief/src/poky/meta/classes-global/sstate.bbclass', lineno: 654, function: sstate_hardcode_path 0650: bb.note("Removing hardcoded paths from sstate package: '%s'" % (sstate_hardcode_cmd)) 0651: subprocess.check_output(sstate_hardcode_cmd, shell=True, cwd=sstate_builddir) 0652: 0653: # If the fixmefn is empty, remove it.. *** 0654: if os.stat(fixmefn).st_size == 0: 0655: os.remove(fixmefn) 0656: else: 0657: bb.note("Replacing absolute paths in fixmepath file: '%s'" % (sstate_filelist_relative_cmd)) 0658: subprocess.check_output(sstate_filelist_relative_cmd, shell=True) Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/mischief/src/poky/build/tmp/work/core2-64-poky-linux/x-native/1.0+${SRCPV}-r0/sstate-build-deploy_source_date_epoch/fixmepath' (Bitbake rev: e91c256ec076e70cf8a18e369fe7862e50618c48) Signed-off-by: Pavel Zhukov <pavel@zhukoff.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: utils: remove spaces on empty linesJulien Stephan2023-11-201-8/+8
| | | | | | | | (Bitbake rev: 979ff8b4228e4c6f90985e9a3faa5fc9f3922218) Signed-off-by: Julien Stephan <jstephan@baylibre.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add path_is_descendant()Joshua Watt2023-09-141-0/+23
| | | | | | | | | | | | Adds a utility that checks if one path is an descendant of another. This check uses os.path.samestat() to make it immune to symlinks and bind mounts. (Bitbake rev: c3ae45946886ee2049939dd5a205790657a7de32) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Use internal fetcher function to avoid duplicationRichard Purdie2023-02-261-15/+4
| | | | | | | | | | We currently have two lists of "proxy" or "fetcher" environment exports. Make the one in utils match the one on the fetcher which has a more complete list of variables now. (Bitbake rev: f9165a798a307a6f0fee120d5c3de660d3a44ae8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: include SSL certificate paths in export_proxiesRoss Burton2023-01-261-12/+9
| | | | | | | | | | | | | | | | | | | | | | | | bb.utils.export_proxies() is a poor-man's alternative for the environment setup code in bb/fetch2, but it's used in several places where recipes want to download manually (such as cve-update-db-native). Notably, export_proxies() doesn't pass on the SSL certificate paths from the original environment, so if SSL_CERT_FILE needs to be set (for example, in a buildtools environment) then proxies work but SSL doesn't. In an ideal world export_proxies and the same logic in fetch2 would merge, but until then we can add the SSL_CERT_ variables and duplicate the basic logic: check the datastore first and then the original environment for variables. Also remove the return value as nothing ever checked it. [ YOCTO #15000 ] (Bitbake rev: c19035e8e71c419c5688a86bfc9c946c96f638e8) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Update thread/process locks to use a timeoutRichard Purdie2023-01-051-0/+13
| | | | | | | | | | | | | | | | | | | | | The thread/process locks we use translate to futexes in Linux. If a process dies holding the lock, anything else trying to take the lock will hang indefinitely. An example would be the OOM killer taking out a parser process. To avoid bitbake processes just hanging indefinitely, add a timeout to our lock calls using a context manager. If we can't obtain the lock after waiting 5 minutes, hard exit out using os._exit(1). Use _exit() to avoid locking in any other places trying to write error messages to event handler queues (which also need locks). Whilst a bit harsh, this should mean we stop having lots of long running processes in cases where things are never going to work out and also avoids hanging builds on the autobuilder. (Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bin/utils: Ensure locale en_US.UTF-8 is available on the systemFrank de Brabander2022-12-081-0/+16
| | | | | | | | | | | | | | Get rid of the duplicate code and add extra check that the locale en_US.UTF-8 is available on the system. This new helper method is now located right above the method filter_environment() which sets LC_ALL environment variable to 'en_US.UTF-8'. [YOCTO #10165] (Bitbake rev: a4ce040a6fd540a1cac52f808f909f9fcf8c961c) Signed-off-by: Frank de Brabander <debrabander@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Allow to_boolean to support int valuesRichard Purdie2022-11-271-0/+3
| | | | | | | | | | | | | | | | Some variables may be set as: X = 1 as well the more usual X = "1" so add support to to_boolean to handle this case. (Bitbake rev: ef9c033b011e68bbfedf7ddf118633c14388aaaf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils/ply: Update md5 to better report errors with hashlibMark Hatle2022-10-261-1/+6
| | | | | | | | | | | | | | | | | | | | | | In the case where hashlib is not available, the try would fail and fall through resulting in a backtrace on the usage of the 'sig'. The backtrace itself was confusing and made it difficult to determine what went wrong. Update the import to be in it's own try block with an appropriate message to indicate what went wrong. Note, the current version of ply all of this code has been restructured so this is not applicable upstream. Additionally, some versions of hashlib don't appear to implement the second FIPS related argument. Detect this and support both versions. (Bitbake rev: 484ab42f440070c0369b81f5c69da860fa47a798) Signed-off-by: Mark Hatle <mark.hatle@amd.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add enable_loopback_networking()Mattias Jernberg2022-09-291-1/+41
| | | | | | | | | | | | | It can be used to enable the loopback interface, typically after calling disable_network(). Also correct a typo in a debug message. (Bitbake rev: 0d317209d4234c5f05a9fcdc13c52f502f104018) Signed-off-by: Mattias Jernberg <mattias.jernberg@axis.com> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Pass lock argument in fileslockedJoshua Watt2022-08-311-2/+4
| | | | | | | | | | | | Pass additional arguments in the fileslocked() context manager to the underlying lockfile() function. This allows the context manager to be used for any types of locks (non-blocking, shared, etc.) that the lockfile() function supports. (Bitbake rev: 7a8eb8da8e8495051e174721062da08e06168024) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix insufficent entropy in sigtask file namesJoshua Watt2022-08-091-0/+21
| | | | | | | | | | | | | | | | | | | | | | | | | Signature generation uses mkstemp() to get a file descriptor to a unique file and then write the signature into it. However, the unique file name generation in glibc is based on the system timestamp, which means that with highly parallel builds it is more likely than one might expect expected that a conflict will occur between two different builder nodes. When operating over NFS (such as a shared sstate cache), this can cause race conditions and rare failures (particularly with NFS servers that may not correctly implement O_EXCL). The signature generation code is particularly susceptible to races since a single "sigtask." prefix used for all signatures from all tasks, which makes collision even more likely. To work around this, add an internal implementation of mkstemp() that adds additional truly random entropy to the file name to eliminate conflicts. (Bitbake rev: 97955f3c1c738aa4b4478a6ec10a08094ffc689d) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: movefile: use the logger for printingJose Quaresma2022-07-281-6/+6
| | | | | | | | (Bitbake rev: 274b2d7a2fa0b43b0b542cb5471ff832e692ea93) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: remove: check the path again the expand python globJose Quaresma2022-07-281-2/+2
| | | | | | | | | | | | | When we call the remove with recurse=True we first check if the remove operation is safe in _check_unsafe_delete_path. But the check is been done on the path instaed of the expanded python glog. (Bitbake rev: 7236488b898309ec5f1880936ddae22a28ccf5d3) Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix lockfile path length issuesRichard Purdie2022-03-241-3/+6
| | | | | | | | | | | | | If the path to bitbake.lock is in a deep directory, bitbake will hang. The reason was that the max file length limiting code (to 255 chars) was including the directory name and it should only act on the filename within the directory. Fix it to just use the base filename. [YOCTO #14766] (Bitbake rev: 89d70e7b71eecfe06592202f326e566c579ba01d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils/ply: Change md5 usages to work on FIPS enabled hostsMark Hatle2022-03-021-1/+1
| | | | | | | | | | | | | | | | | | | | hashlib.md5() is not permitted on a FIPS enabled host system. This is due to md5 not being an approved hash algorithm. Instead use: hashlib.new('MD5', usedforsecurity=False) This is allowed, as it's clear the hash is used for a non-security purpose. Note: utils.py version should never be used to verify file integrity, but instead be used to identify if the file may have changed. sha256 should be used for integrity purposes. (Bitbake rev: af866dd077867cba0129757bfcc689551445e9d7) Signed-off-by: Mark Hatle <mark.hatle@xilinx.com> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Rename environment filtering variablesScott Murray2022-02-211-10/+10
| | | | | | | | | | | | | | | | | | In line with the inclusive language migration defined at: https://wiki.yoctoproject.org/wiki/Inclusive_language rename: BB_ENV_WHITELIST -> BB_ENV_PASSTHROUGH BB_ENV_EXTRAWHITE -> BB_ENV_PASSTHROUGH_ADDITIONS (Bitbake rev: fe60627839d4280cf0117ed1afbfccdff1181b6a) (Bitbake rev: 87104b6a167188921da157c7dba45938849fb22a) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Ensure shell function failure in python logging is correctRichard Purdie2022-02-211-1/+1
| | | | | | | | | | | | | If a python function exec_func() calls a shell task, the logging wasn't working correctly in all cases since the exception was turned into a BBHandledException() and the logfile piece was lost which is handled at the top task level. The easiest way to avoid this is to allow the ExecutionError exceptions to be raised to a higher level, we don't need the traceback for them. (Bitbake rev: 7cae11f558f9ff5fd05ef23b789aaef92fb5a327) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix typo "dependency" and "spawning"Zygmunt Krynicki2022-02-201-2/+2
| | | | | | | (Bitbake rev: 8f76117816fa84367753d5eee0264b90e9f29a8c) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix environment decorator logic errorRichard Purdie2022-02-201-1/+1
| | | | | | | | | | | | | | | | | | | File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/fetch2/wget.py", line 328, in checkstatus with bb.utils.environment(**newenv): File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/layers/build/bitbake/lib/bb/utils.py", line 1737, in environment del os.environ[var] File "/home/pokybuild/yocto-worker/qemuarm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/testsdkext/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.10/os.py", line 695, in __delitem__ raise KeyError(key) from None It is possible the wrapped function might change the environment which can lead to KeyError issues like this. The key may no longer be in the original environment so handle that case correctly and avoid the tracebacks. (Bitbake rev: ae4ce4c21998bb2a12a1e1f0b8e0af8d218b87e2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: bitbake-worker: Preserve network non-local uidRobert Yang2022-02-011-0/+16
| | | | | | | | | | | | | | | | The NIS can't work when network is dissable, so preserve network for it, the error is like: do_ypcall: clnt_call: RPC: Unable to send; errno = Network is unreachable Note, enable nscd on the build machine might be a solution, but that isn't reliable since it depends on whether the network function has been cached or not. (Bitbake rev: 4eafae7904bae6e5c6bc50356e8a9077f2e207fa) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add disable_network functionRichard Purdie2022-01-121-0/+31
| | | | | | | | | | | | Add a function which uses the unshare glibc call to disable networking in the current process. This doesn't work on older distros/kernels but will on more recent ones so for now we simply ignore the cases we can't execute on. uid/gid can be passed in externally so this can work with pseudo/fakeroot contexts. (Bitbake rev: 9d6341df611a1725090444f6f8eb0244aed08213) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Update to use exec_module() instead of load_module()Richard Purdie2022-01-051-2/+5
| | | | | | | | | This is deprecated in python 3.12 and Fedora 35 is throwing warnings so move to the new functions. (Bitbake rev: 68a18fbcb5959e334cf307d7fa8dc63832edb942) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Handle lockfile filenames that are too long for filesystemsRichard Purdie2021-11-211-1/+5
| | | | | | | | | | | | | The fetcher mirror code can go crazy creating lock filenames which exceed the filesystem limits. When this happens, the code will loop/hang. Handle the filename too long exception correctly but also truncate lockfile lengths to under 256 since the worst case situation is lockfile overlap and lack of parallelism. (Bitbake rev: 63baf3440b16e41ac6601de21ced94a94bdf1509) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add environment updating context managerRoss Burton2021-08-121-0/+16
| | | | | | | | | | bb.utils.environment() is a context manager to alter os.environ inside a specific block, restoring it after the block is closed. (Bitbake rev: 9974848f67581ff7d76cef52a94f505af99b4932) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/tests/toaster: Override conversion fixupsRichard Purdie2021-08-061-1/+1
| | | | | | | | | Fix some references that missed during the overrides syntax migration or were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch. (Bitbake rev: 6184cb07dfa44f5f76f1c423533b4547d80b20ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add bb.utils.rename() helper function and use for renamingDevendra Tewari2021-05-061-3/+15
| | | | | | | | | | | | | | | | | | | | os.rename can fail for example an incremental build in Docker fails with: OSError: [Errno 18] Invalid cross-device link when source and destination are on different overlay filesystems. Rather than trying to fix every call site, add a wrapper in bb.utils for renames. We can then handle cross device failures and fall back to shutil.move. The reason os.rename is still used is because shutil.move is too slow for speed sensitive sections of code. [YOCTO #14301] (Bitbake rev: c5c4e49574ab2a65e06298a0a77bb98b041cf56b) Signed-off-by: Devendra Tewari <devendra.tewari@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-101-3/+3
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add docstrings to functionsMilan Shah2021-01-081-4/+34
| | | | | | | | | | | | | | | | | | A list of functions that now has a docstring. * vercmp_string * explode_dep_versions * prunedir * prune_suffix * to_boolean * contains_any * export_proxies See [YOCTO #9725] for details. (Bitbake rev: b61ba4a18693a9e553d2a93161feb0bcc1c82384) Signed-off-by: Milan Shah <mshah@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add umask changing context managerRoss Burton2020-09-301-0/+11
| | | | | | | | | | Add a umask context manager which can be used to temporarily change the umask in a 'with' block. (Bitbake rev: 6c601e68a27e1c60b04c2a61830d1812cc883e09) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: fix UnboundLocalError when _print_exception raisesChris Laplante2020-09-101-2/+2
| | | | | | | | | | PEP 3110 changed how exceptions work. 'e' is unbound after the 'except' clause. See: https://www.python.org/dev/peps/pep-3110/#semantic-changes (Bitbake rev: b69e97de53eb172ed730993e3b755debaa26f30d) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils.py: get_file_layer(): Improve performanceRobert Yang2020-09-101-3/+9
| | | | | | | | | | | | | | | | | The following code costs a lot of time when there are lot of layers and recipes: for collection in collections: collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or '' My build has more than 100 layers and 3000 recipes, which calls d.getVar() 300K (3000 * 100) times and makes 'bitbake-layers show-recipes' very slow, add a keyword argument to get_file_layer() can fix the problem, it can save about 90% time in my build (6min -> 40s). (Bitbake rev: f08a6601c9bb09622855d62e1cedb92fafd2f71d) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils.py: get_file_layer(): Exit the loop when file is matchedRobert Yang2020-09-101-2/+3
| | | | | | | | | | | | | | | | | | | This can make "$ bitbake-layers show-recipes" save about 60% time (14min -> 6min) in my build (more than 3000 recipes) The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with each recipe, and get_file_layer() compare the file with each item in BBFILES which makes it very time consuming when there are a lot of recipes and items in BBFILES. So use BBFILES_PRIORITIZED and exit when file is matched, it doesn't make sense to go on the loop when file is matched. And use fnmatchcase to replace of fnmatch since the comparison should be case-sensitive. (Bitbake rev: 8d64181d29dc262e066a6114dd51e5f2d04f47de) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: process_profilelog: use context managerChris Laplante2020-09-101-15/+14
| | | | | | | (Bitbake rev: 42172900af06baeee559d33b150d5febdf8e960a) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Drop broken timeout functionRichard Purdie2020-08-261-14/+0
| | | | | | | | | I strongly suspect this function doesn't work with modern python so and its unused now, drop it. (Bitbake rev: a3033cea089c66c8b4614e7ee57c166f4262c590) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils.py: Do not preserve TERM in the environmentJacob Kroon2020-06-161-1/+0
| | | | | | | | | | | The value of TERM is leaking into OE-Core postinst-useradd-${PN} scripts, which in turn can optionally be monitored by buildhistory. Prune the value in order to make the OE-Core buildhistory output more deterministic. (Bitbake rev: 0d5cdd0c0d65f2f81c3af0f3767fee86c4142c3a) Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils.py: Preserve ownership of symlinkDaisuke Yamane2020-02-221-1/+1
| | | | | | | | | | | Yocto Bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13806 Uncomment lchown() to preserve ownership of symlink. (Bitbake rev: 6a0b6dd17c6d842960d448114b252e92c55dea33) Signed-off-by: Daisuke Yamane <daisuke.yamane@cybertrust.co.jp> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add is_semver functionJean-Marie LEMETAYER2020-01-271-0/+26
| | | | | | | | | | | | This function checks if a string is a semantic version: https://semver.org/spec/v2.0.0.html The npm fetcher needs this function to validate its version parameter. (Bitbake rev: 61ac4e825fa7afbb76282030586abc9ee4ac215c) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: add sha384_file and sha512_file functionsJean-Marie LEMETAYER2020-01-271-0/+14
| | | | | | | | | | The npm fetcher needs these functions to support the subresource integrity: https://www.w3.org/TR/SRI/ (Bitbake rev: 80e2216e2b41cb6170292009064864449bc48bbe) Signed-off-by: Jean-Marie LEMETAYER <jean-marie.lemetayer@savoirfairelinux.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils: add get_referenced_varsChris Laplante via bitbake-devel2020-01-191-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | Given a start expression, bb.utils.get_referenced_vars returns the referenced variable names in a quasi-BFS order (variables within the same level are ordered aribitrarily). For example, given an empty data store: bb.utils.get_referenced_vars("${A} ${B} ${d.getVar('C')}", d) returns either ["A", "B", "C"], ["A", "C", "B"], or another permutation. If we then set A = "${F} ${G}", then the same call will return a permutation of [A, B, C] concatenated with a permutation of [F, G]. This method is like a version of d.expandWithRefs().references that gives some insight into the depth of variable references. (Bitbake rev: 076eb5453ca35b8b75b8270efb989d5208095b27) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: remove unused importsFrazer Clews2020-01-191-1/+0
| | | | | | | | | | removed unused imports which made the code harder to read, and slightly but less efficient (Bitbake rev: 4367692a932ac135c5aa4f9f2a4e4f0150f76697) Signed-off-by: Frazer Clews <frazer.clews@codethink.co.uk> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils.fileslocked: don't leak files if yield throwsChris Laplante via bitbake-devel2019-12-161-4/+5
| | | | | | | | | | | | | Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by externalsrc.bbclass) was leaked, giving a warning like: WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'> exec_func(task, localdata) (Bitbake rev: 6beddf6214e22b4002626761031a9e9d34fb04db) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: also use mmap for SHA256 and SHA1, for performanceRoss Burton2019-11-211-20/+14
| | | | | | | | | | md5_file() uses a mmap() window to improve performance when hashing files, so refactor the code and do the same for SHA1 and SHA256. (Bitbake rev: ecf87437ff796e17c3e4f210b5803b0136a9e8a4) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add ionice option to prunedirRichard Purdie2019-09-191-4/+7
| | | | | | | | | | | | | | | | Autobuilder type infrastructure can benefit from deletion of certain files as background IO due to the way Linux filesystem priority works. We have problems where build directories as part of oe-selftest being delete starves the running tasks of IO to the point builds take much longer to compelte. Having this option of running the deletion at "idle" helps a lot with that. (Bitbake rev: 797354d285f6d624d9adb52bab65823572da0e39) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix movefile() exception handling with python3Martin Jansa2019-09-031-1/+1
| | | | | | | | | | | | | | | | | | | | * with python3 this fails with: File: 'bitbake/lib/bb/utils.py', lineno: 799, function: movefile 0795: try: 0796: os.rename(src, destpath) 0797: renamefailed = 0 0798: except Exception as e: *** 0799: if e[0] != errno.EXDEV: 0800: # Some random error. 0801: print("movefile: Failed to move", src, "to", dest, e) 0802: return None 0803: # Invalid cross-device-link 'bind' mounted or actually Cross-Device Exception: TypeError: 'OSError' object is not subscriptable (Bitbake rev: d6e43c443ddbbe467c4380c48d2bc28ae18504a1) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/utils: Drop bb.build.FuncFailedRichard Purdie2019-08-061-1/+1
| | | | | | | | | | | | | Its hard to see what this exception adds in the current codebase. The logfile attribute is effectively ignored, the exception doesn't serve a defined purpose and mostly seems to be worked around. Remove it entirely. If this does cause output problems, we'll figure out better ways to address those. (Bitbake rev: cfeffb602dd5319f071cd6bcf84139ec77f2d170) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Strip old editor directives from file headersRichard Purdie2019-05-041-2/+0
| | | | | | | | | | There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-21 21:24:58 +0000