summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/utils.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: bitbake: Fix return value checks from subprocess.call()'sMikko Rapeli2017-11-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | Python function subprocess.call() returns the return value of the executed process. If return values are not checked, errors may go unnoticed and bad things can happen. Change all callers of subprocess.call() which do not check for the return value to use subprocess.check_call() which raises CalledProcessError if the subprocess returns with non-zero value. https://docs.python.org/2/library/subprocess.html#using-the-subprocess-module All users of the function were found with: $ git grep "subprocess\.call" | \ egrep -v 'if.*subprocess\.call|=\ +subprocess\.call|return.*subprocess\.call' Tested similar patch on top of yocto jethro. Only compile tested core-image-minimal on poky master branch. (Bitbake rev: aac956a0950c05bf58af1885474cd89bb9c8fbd1) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils: extend which() so it can look for just executablesRoss Burton2017-03-311-3/+12
| | | | | | | | | | | | | Normally bb.utils.which() is used by the unpack code to find a file in a variety of places, but it is useful as a slightly more powerful version of os.which(). Support this by allowing it to only return matches which are executable files, instead of just the first filename that matches. (Bitbake rev: c0b94f02f0cba7a424aaa16cf98c0f7a3f62b889) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: Add filter()Peter Kjellerstedt2017-03-011-0/+24
| | | | | | | | | | | | | | | | | | | | The bb.utils.filter() function can be used to filter a variable containing whitespace separated words based on another set of words. It has been modeled after the bb.utils.contains_any() function. A typical example of how it can be used is to simplify constructs for PACKAGECONFIG that depend on DISTRO_FEATURES: -PACKAGECONFIG ?= "\ - ${@bb.utils.contains('DISTRO_FEATURES', 'acl', 'acl', '', d)} \ - ${@bb.utils.contains('DISTRO_FEATURES', 'selinux', 'selinux', '', d)} \ -" +PACKAGECONFIG ?= "${@bb.utils.filter('DISTRO_FEATURES', 'acl selinux', d)}" (Bitbake rev: 03ae490366d2046f5b5c185fe4ec2adf1b0a902e) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/persist_data/utils: Drop obsolete python2 importsRichard Purdie2017-01-091-17/+5
| | | | | | | | | These imports were from python 2.6 and earlier, 2.4 in some cases. Drop them since we're all python3 now. (Bitbake rev: 7ef12684e8647b006bf46cae695069d4bfece1cf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/build: enable access to logger within tasksPaul Eggleton2016-12-141-0/+11
| | | | | | | | | | | | | | | | | | In certain circumstances it can be useful to get access to BitBake's logger within a task; the main example is in OpenEmbedded's image construction code where we want to be able to check the log file for errors and warnings, but we don't want to see any errors or warnings that were emitted through the logger; so we need a way to exclude those. In order to do this, pass the logger object into the task via a BB_TASK_LOGGER variable, and add a logging handler class to bb.utils that can be added to it in order to keep a list of warnings/errors that have been emitted. (Bitbake rev: f1cd6fab604f14d8686b1d783cbfe012d923ee42) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Avoid traceback errorsRichard Purdie2016-12-081-1/+1
| | | | | | | | | | Avoid errors like: ERROR: Exception handler error: 'NoneType' object has no attribute 'decode' (Bitbake rev: 1aeb45abe56061f044c2347889c191d5256ff21f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: remove True option to getVar callsJoshua Lock2016-11-301-8/+8
| | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: edit_metadata() comment tweaksPaul Eggleton2016-09-081-1/+3
| | | | | | | | | | | | | | No functional changes, just make a couple of minor tweaks to the comments for edit_metadata(): * There are four elements to be returned by the callback function * Add an example return statement for when you don't want to modify the value (Bitbake rev: 99675c19375c96140bc8ae8f9fc3a1945a77cebb) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/utils.py: export_proxies add GIT_PROXY_COMMANDFrancisco Pedraza2016-08-171-1/+2
| | | | | | | | | | | This was added to enable the usage of git through proxies. (Bitbake rev: 449fc52e483a3bf1cec1c5d8cf8c3946ec5292ab) Signed-off-by: Francisco Pedraza <francisco.j.pedraza.gonzalez@intel.com> Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: show subprocess output in stack tracesRoss Burton2016-07-291-0/+6
| | | | | | | | | | If better_exec() throws a subprocess.CalledProcessError then show the output to the user as it likely contains useful information for solving the problem. (Bitbake rev: 8a6424ed871c3cbacd21cae8bc801197f83d67a6) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: utils: add BBSERVER to the list of preserved variablesEd Bartosh2016-07-201-0/+1
| | | | | | | | | | | | | All environment variables that are not in the list returned by preserved_envvars_exported are cleaned by bb.utils.clean_environment. Added BBSERVER to the list as we need to access it in bb/main.py after the call of bb.utils.clean_environment. (Bitbake rev: 15c4ea679f4fe097a9f21cccfc82907b5f39a4e4) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: no need to unsetenv when manipulating os.environRoss Burton2016-07-201-1/+0
| | | | | | | | | | Doing both os.unsetenv(foo) and then del os.environ[foo] is pointless as del will call unsetenv automatically. (Bitbake rev: a4463e2ff3c7d234320176d671719243292f1af0) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils.py: return sorted OrderedDict in explode_dep_versions2Robert Yang2016-07-191-0/+1
| | | | | | | | | | | | | | | | | The OrderedDict's item is sorted by insertion order, there might be a problem when build the same recipe again, for example: - First build of acl: Depends: libattr1 (>= 2.4.47), libc6 (>= 2.24) - Second build of acl: Depends: libc6 (>= 2.24), libattr1 (>= 2.4.47) They are exactly the same depends, but tools like "diff" doesn't think so. Return sorted OrderedDict will fix the problem. (Bitbake rev: a392f19f16ef8202ce3c12afbeb186a02438da17) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: fix set_process_nameRoss Burton2016-07-191-3/+2
| | | | | | | | | | | | With Python 3 create_string_buffer needs a bytes() not a str() but as we were catching all exceptions nobody noticed. [ YOCTO #9910 ] (Bitbake rev: 6576a9a95486c28a01d4211b4a33cc3e2c55a7cc) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart/utils: Add 'd' to the context used for better_eval in ↵Richard Purdie2016-06-131-2/+8
| | | | | | | | | | | | | | | | | | | | | | python expansion If a line like: foo=${@' '.join([d.getVar('D', True) + x for x in (' '.join([d.getVar('FILES_bash-' + p, True) or '' for p in ['lib', 'dev', 'staticdev', 'doc', 'locale', 'ptest']])).split()])} is added to a function like do_install, it fails with Exception name 'd' is not defined. This is due to a change of behaviour in python 3 compared to python 2. Generator expressions, dict comprehensions and set comprehensions are executed in a new scope but list comprehensions in python 2.x are not. In python 3 they all use a new scope. To allow these kinds of expressions to work, the easiest approach is to add 'd' to the global context. To do this, an extra optional parameter is added to better_eval and we use that to add 'd'. (Bitbake rev: 8f74881037bb01013d3d439dc0c269909a198c1c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: fix imports to work for python 3Ed Bartosh2016-06-021-1/+0
| | | | | | | | | | | | | | | | | | Some APIs have been moved to other modules in python 3: getstatusoutput: moved from commands to subproces urlopen: moved from urllib2 to urllib.request urlparse: moved from urlparse to urllib.parse Made the imports work for both python versions by catching ImportError and importing APIs from different modules. [YOCTO #9584] (Bitbake rev: 1abaa1c6a950b327e6468192dd910549643768bb) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Convert to python 3Richard Purdie2016-06-021-10/+10
| | | | | | | | | Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Force bitbake to en_US.UTF-8 locale setting everywhereRichard Purdie2016-05-131-0/+7
| | | | | | | | | | | | | | | | | | | | Under python 3, if we spawn python processes, we need to have a UTF-8 locale, else python's file access methods will use ascii. You can't change that mode once the interpreter is started so we have to ensure a locale is set. Ideally we'd use C.UTF-8 since OE already forces the C locale but not all distros support that and we need to set something. Was tempted to choose en_GB so colour gets spelt correctly :). This is in some ways pretty nasty, forcing it into the environment everywhere however we only have a limited number of ways of making everything work correctly and this beats having to add utf-8 encoding to every file access command. A similar change will be needed to bitbake.conf in OE. (Bitbake rev: 8902c29638411d312e6fc4a197707e5742652e15) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils.py: Fix explode_dep_versions2() determinism issueRichard Purdie2016-05-111-1/+2
| | | | | | | | | | | | | When we pass data into explode_dep_versions2(), we need to result to be able to be processed in a deterministic way so that we end up with consistent hash values. This means we need an ordered structure rather than an unordered one. To do this, return an OrderedDict() rather than a dict(). (Bitbake rev: 0737e003ca549d08a7dfe13452ae982f2e11fecd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Update logger.warn() -> logger.warning()Richard Purdie2016-05-111-5/+5
| | | | | | | | | | python deprecated logger.warn() in favour of logger.warning(). This is only used in bitbake code so we may as well just translate everything to avoid warnings under python 3. Its safe for python 2.7. (Bitbake rev: 676a5f592e8507e81b8f748d58acfea7572f8796) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils: let loaded plugins provide a plugin objectChristopher Larson2016-05-061-2/+4
| | | | | | | | | | This lets us avoid treating the module like an object, so no globals are needed, if one chooses to do so. (Bitbake rev: 71bfd5beb0d0ed88c7c14bbfd5ca1a1b56122bc1) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils: use imp.get_suffixes for load_pluginsChristopher Larson2016-05-061-2/+9
| | | | | | | | | Rather than hardcoding .py, use python's knowledge of its file extensions. (Bitbake rev: 09f838dbaefdaedc01a1f4818ed38280b38db744) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils: add load_plugins from scriptutilsChristopher Larson2016-05-061-0/+21
| | | | | | | | | Imported as of oe-core 184a256. (Bitbake rev: 99db61bf816d9c735032caa762aae8e6a0803402) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: add docstring for contains()Ross Burton2016-04-181-1/+19
| | | | | | (Bitbake rev: e9174723ea6d0dff5f7f3042009761cf42284947) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils.py: Fix a bug in edit_metadata() that could corrupt varsRandy Witt2016-04-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | edit_metadata() would corrupt a variable that was multiline, but had the ending quotes on the same line as the last value. For example: TEST_VAR = " foo \ bar" would become " foo ba" because the code would always delete the last character on the line and then do it again if the line ended in the quote. This however doesn't show up if you have: TEST_VAR = " foo \ bar \ " which is how all the test cases were written. This patch fixes that bug and adds and fixes a test that matched the bugs behavior rather than the expected behavior. (Bitbake rev: 14f05cbdc2ad8d59a94af1c8816567d93c39c88c) Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/utils: Allow python functions to execute with real exception ↵Richard Purdie2016-03-311-1/+3
| | | | | | | | | | | | | | | | | | handling With the code as it stands today it not possible to execute a python function and get "normal" python exception handling behaviour. If a python function raises an exception, it forces a traceback to be printed and the exception becomes a FuncFailed exception. This adds in a parameter 'pythonexception' which allows standard python exceptions to be passed unchanged with no traceback. Ultimately we may want to change to this convention in various places but at least it means we can start to add sane functions now. (Bitbake rev: 85cf22fd0ed26bb7dc7738ef2a10441891f38ae2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils.py: Add sha1_file callBrendan Le Foll2016-02-261-0/+15
| | | | | | | | | | | This is useful as npm-lockdown uses sha1 because npm releases the sha1 of packages and whilst this is undocumented it seems no other algorithm is supported (Bitbake rev: fd5d9011f6dd7029895b64d8a02d33185b9aa8ae) Signed-off-by: Brendan Le Foll <brendan.le.foll@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Drop datastore function inspection during exceptionRichard Purdie2016-02-101-9/+0
| | | | | | | | | | | When we use functions from the data store, they now have correct line number and filename information. This function would attempt to correct line numbers which doesn't need correcting, leading to misleading messages to the user. Therefore remove this code as being obsoleted. (Bitbake rev: 918bec86bc8ee94feb82380ff410d9fdcbe9e720) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb/fetch2: Move export_proxies function from wget to utils.Aníbal Limón2016-02-081-0/+19
| | | | | | | | | | | In order to use in other modules since is a common function when needs to get proxies working. (Bitbake rev: 85c529044381895556d603a3974de22392646a22) Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add expansion parameter to getVarFlagRichard Purdie2016-02-041-1/+1
| | | | | | | | | | | | | | | This sets the scene for removing the default False for expansion from getVarFlag. This would later allow True to become the expand default. On the most part this is an automatic translation with: sed -e 's:\(\.getVarFlag([^,()]*, [^,()]*\)):\1, False):g' -i `grep -ril getVar *` There should be no functional change from this patch. (Bitbake rev: 7c3b99c6a716095af3ffce0b15110e91fb49c913) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add ability to change the process nameRichard Purdie2016-01-301-0/+11
| | | | | | | | | | Being able to tell the bitbake processes apart is useful for debugging. Add a helper function which allows this without making it a hard dependency. Errors are ignored, this is just nice to have. (Bitbake rev: fd7f1a94d196b8a3c445e313d9e699b352b1da97) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Remove double compile from better_compileRichard Purdie2016-01-061-5/+4
| | | | | | | | | | Poking around the ast to correct linenumbers works well for runtime failures but not for parsing ones. We can use blank linefeeds to correct the line numbers instead, with the advantage that we don't need to double compile. (Bitbake rev: 10256ac3e7be7e691176ecc5d55856d88f1fe940) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event/utils/methodpool: Add a cache of compiled code objectsRichard Purdie2015-12-221-1/+6
| | | | | | | | | | | | | | | With the addition of function line number handling, the overhead of the compile functions is no longer negligible. We tend to compile the same pieces of code over and over again so wrapping a cache around this is beneficial and removes the overhead of line numbered functions. Life cycle of a cache using a global like this is in theory problematic although in reality unlikely to be an issue. It can be dealt with if/as/when we deal with the other global caches. (Bitbake rev: 98d7002d1dca4b62042e1589fd5b9b3805d57f7a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Improve traceback from better_exec internal errorsRichard Purdie2015-12-181-2/+4
| | | | | | | | | | If you break the internals of better_exec(), you get a very weird error about tb_next not being a method of None. Fix this by checking we can step back a trace level. (Bitbake rev: 1d710ed484f68fca0789022dde7ba877b9a894f5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast/event/utils: Improve tracebacks to include file and line ↵Richard Purdie2015-12-181-4/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | numbers more correctly Currently bitbake tracebacks can have places where the line numbers are inaccurate and filenames may be missing. These changes start to try and correct this. The only way I could find to correct line numbers was to compile as a python ast, tweak the line numbers then compile to bytecode. I'm open to better ways of doing this if anyone knows of any. This does mean passing a few more parameters into functions, and putting more data into the data store about functions (i.e. their filenames and line numbers) but the improvement in debugging is more than worthwhile). Before: ---------------- ERROR: Execution of event handler 'run_buildstats' failed Traceback (most recent call last): File "run_buildstats(e)", line 43, in run_buildstats(e=<bb.build.TaskStarted object at 0x7f7b7c57a590>) NameError: global name 'notexist' is not defined ERROR: Build of do_patch failed ERROR: Traceback (most recent call last): File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task return _exec_task(fn, task, d, quieterr) File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task event.fire(TaskStarted(task, logfn, flags, localdata), localdata) File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire fire_class_handlers(event, d) File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers execute_handler(name, handler, event, d) File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler ret = handler(event) File "run_buildstats(e)", line 43, in run_buildstats NameError: global name 'notexist' is not defined ---------------- After: ---------------- ERROR: Execution of event handler 'run_buildstats' failed Traceback (most recent call last): File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats(e=<bb.build.TaskStarted object at 0x7efe89284e10>): if isinstance(e, bb.build.TaskStarted): > trigger = notexist pn = d.getVar("PN", True) NameError: global name 'notexist' is not defined ERROR: Build of do_package failed ERROR: Traceback (most recent call last): File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task return _exec_task(fn, task, d, quieterr) File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task event.fire(TaskStarted(task, logfn, flags, localdata), localdata) File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire fire_class_handlers(event, d) File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers execute_handler(name, handler, event, d) File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler ret = handler(event) File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats trigger = notexist NameError: global name 'notexist' is not defined ---------------- (Bitbake rev: 1ff860960919ff6f8097138bc68de85bcb5f88b0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: improve edit_bblayers_conf() handling of ↵Paul Eggleton2015-12-071-14/+64
| | | | | | | | | | | | | | | | | | | | | | bblayers.conf formatting Make the following improvements to edit_bblayers_conf(): * Support ~ in BBLAYERS entries * Handle where BBLAYERS items are added over multiple lines with += instead of one single long item Also add some comments documenting the function arguments and return values as well as a set of bitbake-selftest tests. (This function is used by the bitbake-layers add, remove and layerindex-fetch subcommands, as well as devtool when adding the workspace layer). (Bitbake rev: e9a0858023c7671e30cc8ebb08496304b7f26b31) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: fix error in edit_metadata() when deleting first linePaul Eggleton2015-12-071-1/+1
| | | | | | | | | | | | If you tried to delete the variable on the first line passed to edit_metadata() this failed because the logic for trimming extra blank lines didn't expect the list to be empty at that point - fix that bad assumption. (Bitbake rev: 8bce6fefdc5c046b916588962a2b429c0f648133) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/utils: Add BB_TASK_IONICE_LEVEL supportRichard Purdie2015-10-291-0/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Similarly to BB_TASK_NICE_LEVEL, add BB_TASK_IONICE_LEVEL which allows the ioprio of tasks to be adjusted. This is in response to various qemu runtime timeouts which have been witnessed on the autobuilder, seemingly due to IO starvation (we already use NICE_LEVEL to adjust tasks). This has a fairly urgent need to deal with certain 'random' failures we're seeing on the autobuilders in testing. The format of the data in the variable is BB_TASK_IONICE_LEVEL = "<class>.<prio>". For <class>, 2 is best effort (the default), 1 is real time and 3 is idle. You'd need superuser privileges to use realtime. The <prio> value is a default of 4, and can be set between 0 and 7 with 7 being lowest priority and 0 the highest. The user can set this freely with normal privileges Note that in order for this to take effect, you need the cfq scheduler selected for the backing block device. We could use nice wrapper functions for ioprio from modules like psutil however that would complicate bitbake dependencies. This version has some magic numbers but works on the main 32 and 64 bit x86 build architectures and can easily be extended if ever needed. When we move to python 3.x, we can likely replace this with standard calls. (Bitbake rev: b9471ad147b102c45d65f5ffd9521864df7ff9c1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: only add layer once in edit_bblayers_conf()Markus Lehtonen2015-09-291-0/+1
| | | | | | | | | | | | | | Prevent edit_bblayers_conf() from adding layer(s) multiple times. This happened when BBLAYERS variable was "listed" multiple times in bblayer.conf - i.e. the configuration was split into multiple separate assignments. [YOCTO #8316] (Bitbake rev: 5e423237f9f4ff7e7e03bf066b0142ba4bd82219) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add signal_on_parent_exit() functionRichard Purdie2015-09-091-0/+19
| | | | | | | | | | | | | | | | | | | Add a new bb.utils.signal_on_parent_exit() function so that a process can register to recieve a signal when the parent dies. There is no POSIX standard for this and the implementation is Linux specific. Alternatives would be having an open pipe or polling os.getppid() for changes but this seems more effective and less invasive to most of bitbake's code structure. We need to be able to determine when parents die to ensure child processes stop running in a variety of circumstances to avoid locks being held and ensure clean shutdown. Roughly based on https://gist.github.com/evansd/2346614 (Bitbake rev: 34974f5e30e9b09c016481e4c81c156a5f379784) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix a potential error in movefileBenjamin Esquivel2015-09-041-3/+6
| | | | | | | | | | | bitbake utils' movefile is now prone to malform the destination file with duplicated file name strings. Fixing it to force a file name append iff the dest argument is a dir not a file name (Bitbake rev: 38dd27f7191da002a16c561be3790ce487045b01) Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Specify dest file name in movefile()Benjamin Esquivel2015-09-031-1/+3
| | | | | | | | | | | | When moving a file via the os.rename function, it was missing the destination file name which caused an OSError [YOCTO#8180] (Bitbake rev: b147ba0341d87e077bd2b09ef4355976ecd2d26b) Signed-off-by: Benjamin Esquivel <benjamin.esquivel@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-layers: fix mapping files to layersPaul Eggleton2015-08-191-8/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | bitbake-layers needs to map recipe and class files to the layer they came from within the show-recipes and show-overlayed commands. However, it turns out that mapping a file to the layer it came from is not as trivial as it might seem. To do it properly we need to match the path to an entry in BBFILES then map that to the collection name using BBFILE_PATTERN, then map that to the actual layer using variable history. If it doesn't match any entry in BBFILES, then we can fall back to BBFILE_PATTERN (to handle classes and conf files). This fixes the layer name not showing up properly in the output of the show-recipes and show-overlayed commands for recipes in layers such as meta-intel that have subdirectories in BBFILE_PATTERN. It also fixes the priority not showing up in show-layers for such layers. As part of this I've added a function to VariableHistory which for a space-separated list variable gives you a dict mapping the items added to the files in which they were added. I've also fixed bb.utils.get_file_layer() and reduced some of the duplication by using this function in bitbake-layers. Also fixes the priority not showing up for layers such as meta-intel Fixes [YOCTO #8160]. (Bitbake rev: e852f6cabd7489585477ab567a1afeb2252377ac) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: properly fix bitbake.lock handlingRichard Purdie2015-08-011-4/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If the PR server or indeed any other child process takes some time to exit (which it sometimes does when saving its database), it can end up holding bitbake.lock after the UI exits, which led to errors if you ran bitbake commands successively - we saw this when running the PR server oe-selftest tests in OE-Core. The recent attempt to fix this wasn't quite right and ended up breaking memory resident bitbake. This time we close the lock file when cooker shuts down (inside the UI process) instead of unlocking it, and this is done in the cooker code rather than the actual UI code so it doesn't matter which UI is in use. Additionally we report that we're waiting for the lock to be released, using lsof or fuser if available to list the processes with the lock open. The 'magic' in the locking is due to all spawned subprocesses of bitbake holding an open file descriptor to the bitbake.lock. It is automatically unlocked when all those fds close the file (as all the processes terminate). We close the UI copy of the lock explicitly, then close the server process copy, any remaining open copy is therefore some proess exiting. (The reproducer for the problem is to set PRSERV_HOST = "localhost:0" and add a call to time.sleep(20) after self.server_close() in lib/prserv/serv.py, then run "bitbake -p; bitbake -p" ). Cleanup work done by Paul Eggleton <paul.eggleton@linux.intel.com>. This reverts bitbake commit 69ecd15aece54753154950c55d7af42f85ad8606 and e97a9f1528d77503b5c93e48e3de9933fbb9f3cd. (Bitbake rev: a29780bd43f74b7326fe788dbd65177b86806fcf) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/utils: Improve parsing profilingRichard Purdie2015-05-291-3/+9
| | | | | | | | | | | | | | | Currently the cooker parsing processes each dump an individual profile which is ok, but means absolute numbers of function calls for a given load can be tricky to determine as parsing of recipes may go to different pool threads on different runs. This change collects up the individual thread parsing results and processes them into one profile output. The profile processing function in utils needed tweaks to allow this to work. (Bitbake rev: d3d2541aacd1ea560da0d8b25a3ea3f0563dee70) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: fix and extend edit_metadata_file()Paul Eggleton2015-05-191-58/+180
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix several bugs and add some useful enhancements to make this into a more generic metadata editing function: * Support modifying function values (name must be specified ending with "()") * Support dropping values by returning None as the new value * Split out edit_metadata() function to provide same functionality on a list/iterable * Pass operation to callback and allow function to return them * Pass current output lines to callback so they can be modified * Fix handling of single-quoted values * Handle :=, =+, .=, and =. operators * Support arbitrary indent string * Support indenting by length of assignment (by specifying -1) * Fix typo in variablename - intentspc -> indentspc * Expand function docstring to cover arguments / usage * Add a parameter to enable matching names with overrides applied * Add some bitbake-selftest tests Note that this does change the expected signature of the callback function. The only known caller is in lib/bb/utils.py itself; I doubt anyone else has made extensive use of this function yet. (Bitbake rev: 20059e4d5ab9bf0f32c781ccb208da3c95818018) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: fix several bugs in edit_metadata_file()Paul Eggleton2015-05-191-3/+11
| | | | | | | | | | | | * Fix unchanged assignments being dropped if other lines changed * Fix not passing variable name from single-line assignments to the function * Fix not trimming the trailing quote from values (Bitbake rev: 0b0c82f49cf2de887967d305768cbd95314bb171) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: add function to get layer containing a filePaul Eggleton2015-05-191-0/+16
| | | | | | | | | | | | In certain contexts it can be useful to find the layer that a file (e.g. a recipe) appears in. Implements [YOCTO #7723]. (Bitbake rev: 3bf9c8830c5d5eea5502230d5af84ebd87ad5849) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/utils: add safeguard against recursively deleting things we ↵Paul Eggleton2015-04-191-0/+21
| | | | | | | | | | | | | | shouldn't Add some very basic safeguard against recursively deleting paths such as / and /home in the event of bugs or user mistakes. Addresses [YOCTO #7620]. (Bitbake rev: 56cddeb9e1e4d249f84ccd6ef65db245636e38ea) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: rework LAYERDEPENDS versioning so that it is actually usefulPaul Eggleton2015-02-161-0/+22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've had versioned dependency support in LAYERDEPENDS for quite a long time, but I can say with pretty good certainty that almost nobody has used it up to now because it was too strict - the specified version had to exactly match the version in your configuration or you would get an error; there was no "greater than or equal" option, which is usually what you will want given that LAYERVERSION does get bumped from time to time. However, users mismatching layer branches and then having their builds fail later on with some incomprehensible error is still a pretty common problem. We can't simply use the git branch because not everyone is always on a branch and the branch names don't always match up (and that's not an issue). To provide a practical means to address branch mismatching, I have reworked LAYERDEPENDS version specifications to use the more familiar "dependency (>= version)" syntax as used with package dependencies, support non-integer versions, and clarified the error message a little. If we then take care to bump the version on every breaking change, it is at least possible to have layers depend on these changes when they update to match; we can now even support a major.minor scheme to allow retrospectively adding a version limiter to old branches when a new branch is created and yet still allow the old branch minor version to be bumped if needed. Fixes [YOCTO #5991]. (Bitbake rev: 408be9cdf2b1e32e64ea488d8051a546fb54c144) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-12 07:00:55 +0000