summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* wic/engine: improve error reporting when using rm with wicAnuj Mittal2018-07-261-2/+14
| | | | | | | | | | | | | | | | | When trying to delete something from an ext partition using debugfs, we don't show any error to the user when that operation fails. Change this behavior to show the error generated by debugfs. Also, fallback to use rmdir in case we are trying to delete a directory. However, unlike mdeltree that is used for a FAT partition, there's no easy way to delete a non empty directory. Show an error instead when that happens so user can take appropriate action. (From OE-Core rev: a405383e63c35d7b56108f192ca74755b122a639) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* logrotate: Improve configurability of the installed systemd service filesPeter Kjellerstedt2018-07-261-5/+15
| | | | | | | | | | | | | | | | | | This makes it possible to add extra options to the logrotate application (via ${LOGROTATE_OPTIONS}), and it allows the Persistent option in logrotate.timer to be configured via ${LOGROTATE_SYSTEMD_TIMER_PERSISTENT}. It also changes the sed expressions to allow for ${LOGROTATE_SYSTEMD_TIMER_BASIS} to contain commas without having to prefix them with backslahes, e.g.: LOGROTATE_SYSTEMD_TIMER_BASIS = "*-*-* *:00,30:00" (From OE-Core rev: ca3f2f6b49edee7052ef941228dc477d9060bdd9) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* iputils: upgrade to s20190629Ross Burton2018-07-265-157/+145
| | | | | | | | | | | | | | | | | | | | License-Update: change license checksums to check the License files themselves (BSD3 and GPL2), and the tool-to-license file (LICENSE). Changes to LICENSE will now be detected. Backport a patch to fix the build with musl. Drop upstreamed patch. Update libidn build-dependency to libidn2. Patch in an install target and use that to install binaries, to avoid upstream adding binaries that we don't ship. Also ship all binaries that are installed. (From OE-Core rev: d227122d929330ff71a550689aaedc5d8299d859) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gnutls: rationalise libunistring argumentsRoss Burton2018-07-261-1/+0
| | | | | | | | | | No need to pass --without-libunistring-prefix, and it looks a lot like we're trying to disable it. (From OE-Core rev: 60fef4940de7f0440f1216eb2ea0ea683b3e8fdd) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe: split out machine to ELF data dictionaryRoss Burton2018-07-262-136/+129
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* binutils: Upgrade to 2.31 release branchKhem Raj2018-07-2621-1206/+1318
| | | | | | | | (From OE-Core rev: 7eeec2ad91eca6ba44ea7b761d47082f4ebb04cc) Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* grub: fix build with new binutilsRoss Burton2018-07-262-1/+78
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* base-files: profile: Avoid using "command" to determine if programs existPeter Kjellerstedt2018-07-261-8/+10
| | | | | | | | | | | | | Since the existence of "command" in itself is not guaranteed, using it to determine if other executables exist is moot. Instead just run the executables and let the shell determine if they exist. By piping stderr to /dev/null we avoid unnecessary warnings in case they do not exist. (From OE-Core rev: ba8510deb73e07f17be051fa15a91731ec10382f) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3-pip: clean up obsoleteHongxu Jia2018-07-261-18/+0
| | | | | | | | | | | | | | | | | | | Since unify var-DISTUTILS_INSTALL_ARGS in distutils and setuptools, - The local DISTUTILS_INSTALL_ARGS assignment is obsolete; - The site.py is not be generated any more; - The layout is in a standard pip dir (such as /usr/lib/python3.5/site- packages/pip rather than /lib/python3.5/site-packages/pip-10.0.1- py3.5.egg/pip), the pth file is not required; `#!/usr/bin/env python3' is already used, do not manually sed. [YOCTO #8446] (From OE-Core rev: 5797631dc7a68e10c848ef06dd9eed011bc06e34) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python-setuptools.inc: clean up useless local var-DISTUTILS_INSTALL_ARGSHongxu Jia2018-07-261-3/+0
| | | | | | | | | | | | | | | | | Since unify var-DISTUTILS_INSTALL_ARGS in distutils and setuptools, The local var-DISTUTILS_INSTALL_ARGS broke do_install ... error: option --script-dir not recognized ERROR: python3 setup.py install execution failed. ... [YOCTO #8446] (From OE-Core rev: 1d205a107260d3f60c2af4aa6af0ba983a45c76d) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* distutils/setuptools, distutils3/setuptools3: improve reproducibilityHongxu Jia2018-07-264-15/+10
| | | | | | | | | | | | | | | | | - Unify var-DISTUTILS_INSTALL_ARGS in distutils and setuptools - Supply "--root" directory to the "install" command, and use it as a prefix to strip off the purported filename encoded in bytecode files. (It strips build path prefix from .pyc files) [YOCTO #8446] [YOCTO #12084] (From OE-Core rev: 21f0c5a50e00ac34975d79fd48f08730d8f19b15) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa/sdk/sdkbuildproject: use os.path.joinRoss Burton2018-07-261-2/+1
| | | | | | | | | | Instead of mushing two paths together and hoping that the slashes line up correctly, use os.path.join. (From OE-Core rev: ed0413babaf10ccf729472f603bbe32d84ba1931) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oeqa: rationalise Perl testsRoss Burton2018-07-263-48/+11
| | | | | | | | | | As with the Python test, this can be both better and faster. No need to copy a file, just run a one-liner. (From OE-Core rev: c6eef46747fe58bb2310be4f06d2fa9b67901d72) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* libnss-mdns: Fix search path for avahi socket fileBetacentauri2018-07-261-0/+2
| | | | | | | | | | | Since v0.7 avahi uses /run/avahi-daemon/socket. libnss searches in $(localstatedir)/run/avahi-daemon/. Set localstatedir to / to fix mdns resolving. (From OE-Core rev: 4feb586ba9fa0255d8f83bc1bec11f697dc26a33) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* logging: use warning instead warnChen Qi2018-07-2616-48/+48
| | | | | | | | | | | | | | | | The warn method is deprecated. We should use the documented warning instead. Quoting from the python's official doc: """ Note: There is an obsolete method warn which is functionally identical to warning. As warn is deprecated, please do not use it - use warning instead. """ (From OE-Core rev: f467fd277eb77336097cfc0f5f329bdc8d0f70cb) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bind: update to 9.11.4Armin Kuster2018-07-261-2/+2
| | | | | | | | | | | include: CVE-2018-5738 (From OE-Core rev: fcf45459bc32d833740e3bb237ea0b8f8845f308) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tiff: security fix CVE-2018-7456Joe Slater2018-07-262-0/+179
| | | | | | | | | | NULL pointer use as described at nvd.nist.gov/vuln/detail/CVE-2018-7456. (From OE-Core rev: 122da5cec495fc8ddfd880327e7c3ed0dc70e04f) Signed-off-by: Joe Slater <joe.slater@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* tiff: security fix CVE-2018-8905Joe Slater2018-07-262-0/+62
| | | | | | | | | | Buffer overflow described at nvd.nits.gov/vuln/detail/CVE-2018-8905. (From OE-Core rev: 3f6f2a0619b4e243e6a9e52cee2cdd625ebf6769) Signed-off-by: Joe Slater <joe.slater@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* libx11: remove redundant siteinfo inheritRoss Burton2018-07-261-2/+0
| | | | | | | | | The recipe doesn't use the variables, and autotools inherits this already. (From OE-Core rev: 270fc40c9365a2ad968203ecc0c985f53b574bee) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gnu-config: update to 2018-07-13Ross Burton2018-07-261-3/+3
| | | | | | | | | License checksum updated as URL changed. (From OE-Core rev: 99f97350127ae2760614ad5b829e159d8b2747ee) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* gnu-config: remove obsolete versionRoss Burton2018-07-262-65/+0
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* curl: disable C source code generatorAndre McCurdy2018-07-261-0/+1
| | | | | | | | | | | | | | | | | | The --libcurl command line option causes curl to generate C source code which, when compiled and linked with libcurl, creates a binary which behaves in the same way as curl when run with the other options passed on the curl command line. https://curl.haxx.se/docs/manpage.html#--libcurl It's a development tool and not generally useful on the target, especially if the target doesn't contain a toolchain etc. (From OE-Core rev: a1cda48761f70b5989bb8b9c91d71c0b3a03466e) Signed-off-by: Andre McCurdy <armccurdy@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* distutils/setuptools-native, distutils3/setuptools3-native: do not try to ↵Hongxu Jia2018-07-264-0/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | fetch code during do_compile If a python recipe is using setuptools and the setup_requires argument, where setuptools will use easy_install to fetch the module if it isn't present. The build failed on a machine where a proxy was required, but succeeded on a machine which had direct access to the internet Add var-NO_FETCH_BUILD, and set it in distutils_do_compile which does not allow to fetch code from internet during do_compile. Example result: ... ERROR: Do not try to fetch `pytest-runner1' for building. Please add its native recipe to DEPENDS. Traceback (most recent call last): File "setup.py", line 56, in <module> ... The improvement is flexible for test_requirements argument (used at `setup.py test'), where use easy_install also. [YOCTO #12084] (From OE-Core rev: 57414f4a1c549e6faaa110e6e95ff601b05b7361) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cmake: put cmake.m4 and toolchain file in PNRoss Burton2018-07-261-1/+2
| | | | | | | | | | | | | | | | | Previously cmake-dev held some files which should be in cmake. - cmake.m4 should be in installed in cmake so it can be used out of the box - nativesdk-specific OEToolchainConfig.cmake file used to be in cmake, but the change of default packaging rules move it into cmake-dev. This recipe is the exception and it should be moved back. Add the extra paths to cmake, and clear FILES for cmake-dev to ensure nothing else slips in. (From OE-Core rev: a6ce79b87d3db57033a3d1710cb3292366a0a8f7) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cmake: in SDK use OE env var to set default toolchainCody P Schafer2018-07-263-2/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | Patch the location in cmake where the toolchain file is loaded to use the (new) OE_CMAKE_TOOLCHAIN_FILE variable to select a default toolchain if no toolchain has been specified. The cmake alias is removed. The alternatives: - shell alias fails when cmake is called indirectly (ex: a makefile managing several projects which calls cmake for some of them) because aliases are not inherited - wrapper script that unconditionally adds "-D..." breaks cmake's build tests and many other things as it causes cmake to believe it should be configuring things when it should not be. For example, `cmake -DCMAKE_TOOLCHAIN_FILE=... --build .` does not work (note that this also breaks people directly using `cmake --build .` with the current alias). (From OE-Core rev: da60be3768e7183794d63548166d107dbd0a4973) Signed-off-by: Cody P Schafer <dev@codyps.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* yocto-uninative: Upgrade to version 2.2Richard Purdie2018-07-261-3/+4
| | | | | | | | | | | This version adds nativesdk-libnss-nis to resolve glibc symbol issues We need this to avoid symbol mismatch issues for binaries that use this on newer systems which then won't run on older ones where it isn't present. (From OE-Core rev: 98c7ab9cf32765d604c35dc69bc7bd90e94fc8f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Always place the server logfile in the build directoryuninative-2.2Richard Purdie2018-07-251-2/+3
| | | | | | | | | | Currently the bitbake-cookerdaemon.log is placed into cwd. This seems like a bad idea, we can place it in the build directory alongside the lockfile that represents the server instead. (Bitbake rev: 1620dbc48ffb2a882371cf9174a7b12648befc8a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/bbtests: Add test for bitbake execution outside the build directoryRichard Purdie2018-07-251-1/+11
| | | | | | | | Also fix a related test's error message to match what it does. (From OE-Core rev: 662be652ed39f32c3c7bee6539a9e91af191557e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/bblayers: Place the test layer directory in builddirRichard Purdie2018-07-251-1/+1
| | | | | | | | | | | Placing the layer in meta means the directory is in an unclean state which may influence other tests. Use our build directory instead since we 'own' that. This helps keep oe-selftest parallelisation clean. (From OE-Core rev: f2d32bdf3f27a9ef62fe3abb7fdd2e04ab84b5f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* license_image: Fix raceRichard Purdie2018-07-252-53/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current code pokes into do_deploy manifests from do_image_complete when the do_image_complete task may or may not depend upon the do_deploy tasks in question. Often it gets lucky, sometimes it results in build failures. To fix this, split the functionality to its own task which can have the correct task dependencies. This means the data in BB_TASKDEPDATA is definitive, the other code can be dropped, as can the IMAGE_EXTRATYPES do_populate_lic dependencies from image.bbclass. This fixes bugs which show up as: NOTE: recipe linux-yocto-4.14.48+gitAUTOINC+d64aec9793_97c8063d2d-r0: task do_deploy: Started ERROR: core-image-minimal-1.0-r0 do_image_complete: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:write_deploy_manifest(d) 0003: File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/classes/license.bbclass', lineno: 33, function: write_deploy_manifest 0029: 'w+').write(output) 0030:} 0031: 0032:python write_deploy_manifest() { *** 0033: license_deployed_manifest(d) 0034:} 0035: 0036:python license_create_manifest() { 0037: import oe.packagedata File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/classes/license.bbclass', lineno: 191, function: license_deployed_manifest 0187: # It is necessary to mark this will be used for image manifest 0188: man_dic[dep]["IMAGE_MANIFEST"] = True 0189: man_dic[dep]["PN"] = dep 0190: man_dic[dep]["FILES"] = \ *** 0191: " ".join(get_deployed_files(dep_dic[dep])) 0192: with open(os.path.join(lic_dir, dep, "recipeinfo"), "r") as f: 0193: for line in f.readlines(): 0194: key,val = line.split(": ", 1) 0195: man_dic[dep][key] = val[:-1] File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/classes/license.bbclass', lineno: 289, function: get_deployed_files 0285: """ 0286: 0287: dep_files = [] 0288: excluded_files = [] *** 0289: with open(man_file, "r") as manifest: 0290: all_files = manifest.read() 0291: for f in all_files.splitlines(): 0292: if ((not (os.path.islink(f) or os.path.isdir(f))) and 0293: not os.path.basename(f) in excluded_files): Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build-st-730/tmp/sstate-control/manifest-qemux86_64-linux-yocto.deploy' ERROR: core-image-minimal-1.0-r0 do_image_complete: Function failed: write_deploy_manifest ERROR: Logfile of failure stored in: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build-st-730/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/temp/log.do_image_complete.50537 NOTE: recipe core-image-minimal-1.0-r0: task do_image_complete: Failed ERROR: Task (/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/recipes-core/images/core-image-minimal.bb:do_image_complete) failed with exit code '1' NOTE: recipe linux-yocto-4.14.48+gitAUTOINC+d64aec9793_97c8063d2d-r0: task do_deploy: Succeeded (From OE-Core rev: b54cdaea7844ee3bf0c39eb97cc7c4c17ed5818c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* license: Split image license functions to a separate classRichard Purdie2018-07-252-283/+285
| | | | | | | | | | | This means the image code is only included in image recipes through the IMAGE_CLASSES variable. This sets things up to allow us to fix image deploy dependency problems. (From OE-Core rev: fd44b8b4b2484f2d35c7a0e749e7dc316d601989) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package: Use subprocess to be consistent with the rest of the classRichard Purdie2018-07-251-4/+4
| | | | | | | | | Using 'sub' in one function just confuses things, standardise and fix formatting of the parameters too. (From OE-Core rev: a740c638148ed7236c49eed55ae9a15b94e55b9f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/wic: clean up runCmd error code handlingRichard Purdie2018-07-251-110/+82
| | | | | | | | | | | runCmd handles checking exit 0 internally unless specified otherwise and its error messages are much more useful including the failed command output. Clean up the unneeded 0 exit code checks in the wic tests. (From OE-Core rev: 394f7d4efe80e390e2c9b0be5e6d10954395a82b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* selftest/wic: Improve error message for test_fixed_sizeRichard Purdie2018-07-251-2/+0
| | | | | | | | | | | | | Currently this can fail with a message like 127 != 0 which is unhelpful. If we remove the ignore_status=False, the debugging from runCmd is much more helpful printing status.output. Also remove the now unneeded exit code check. (From OE-Core rev: 1aa7471b11aedc68de5116c461fe73152e3985fd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Revert "relocate_sdk.py: remove hardcoded SDK path"Richard Purdie2018-07-253-10/+14
| | | | | | | | | | | | | This reverts commit 6671a4d980c8bef8f402780a308f6c43a25044aa. This breaks uninative tarball since the call of relocate_sdk.py from uninative.bbclass wasn't updated to account for this change. It isn't clear what value that code could pass in and this isn't simple to fix so revert until a better fix can be found that doesn't break uninative. (From OE-Core rev: b247392b4ced57cfe694656032f6a6723740a9e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* uninative-tarball: Add nativesdk-libnss-nis to resolve glibc symbol issuesRichard Purdie2018-07-241-0/+1
| | | | | | | | | We need this to avoid symbol mismatch issues for binaries that use this on newer systems which then won't run on older ones where it isn't present. (From OE-Core rev: 39c1719a32ed5567e3bf2df5c4f9068d0f5a9400) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* oe-selftest: fix for changes to buildhistory-diffPaul Eggleton2018-07-241-2/+15
| | | | | | | | | | | | | Now we're not reporting the related fields (as of openembedded-core 8658b3677b9f7cb70806061c41570c709086ef05) we shouldn't expect to see PR reported here since it's not monitored by buildhistory-diff. However, with a bit of messing about we can check for the exact output that we should now see as a result of the test changing PR to go backwards. (From OE-Core rev: f9c233908e6ada208d7abfdfe3ff98c6b278f405) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* lib/oe/utils: Drop now unused multiprocess_execRichard Purdie2018-07-241-34/+0
| | | | | | (From OE-Core rev: f3b30def2cd1c9ede7630489c3949a45b6eba6ed) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package/package_manager: multiprocess_exec -> multiprocess_launchRichard Purdie2018-07-243-15/+7
| | | | | | | | | | | | After this replacement, the parent exception handling works so we don't need subprocess wrapping with bb.error in the underlying functions. The underlying contexts also have better module handling so the imports can be cleaned up. (From OE-Core rev: aa8260adf53139d776a2affe6118d28b295c1fab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package: Clean up getstatusoutput usageRichard Purdie2018-07-241-13/+11
| | | | | | | | | | | | | | | | Replace usage of oe.utils.getstatusoutput() with direct subprocess calls. The call is just a wraper to the subprocess call of the same name and requires the caller to handle exceptions themselves. We usually do this badly, failing to show the output or the command or the return code. Its much safer to rely on a call like subprocess.check_output() instead. This also makes it easier to spot and remove cases where shell=True isn't needed in a later cleanup. (From OE-Core rev: 9f058857fb692f1251deb43bcaa7ed0120140093) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstate/bitbake.conf: Use pigz if availableRichard Purdie2018-07-242-3/+10
| | | | | | | | | | | | | | Currently the compression of sstate objects is single threaded. In the case of ltp, this takes around 33s. If we add pigz into the list of non-fatal HOSTTOOLS and then use if it available when building the sstate object, this time drops to around 6s. Since pigz is now widely available this is an optimisation we should utilise. (From OE-Core rev: 2de56aa0792ec93445130d801936a8ea643fad27) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package: Call file to determine elf status in parallelRichard Purdie2018-07-243-43/+64
| | | | | | | | | | This allows the calls to is_elf (which calls file) to happen in parallel allowing a speedup of do_package and do_populate_sysroot for native recipes. (From OE-Core rev: bbe0d3e26484f3f347262d40a8a9d415ce21fb43) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package: Refactor to remove isElf/is_elf function duplicationRichard Purdie2018-07-242-66/+47
| | | | | | | | | | | There are probably further cleanups needed here but this at least removes the major code duplication between these two similar funcitons, keeping the kernel module ".ko" extension check for efficiency to avoid opening and reading file contents in the general case. (From OE-Core rev: 7ad0c0d6ab12bebeac097fc0f5210c876dcfe9be) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package: Allow parallel processing of shlib analysisRichard Purdie2018-07-241-17/+30
| | | | | | | | | | | This function is a bit more invasive to add parallelism to but allows the shlibs analysis to happen in multiple threads. In order to return values correctly/safely the data types needed tweaking to avoid lists and use immutable objects. (From OE-Core rev: b5788fb1f795f2f35d1788d8311e12984ffb2122) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package: Allow parallel processing of debug splittingRichard Purdie2018-07-241-11/+13
| | | | | | | | | | Adjust the code so that the splitting of debug symbols from files happens in parallel. To to this we need to move some path handling code into the main function and pass more parameters in. (From OE-Core rev: 9f0c2ed5d44a16e8268ac521236c4752f930f26a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package: Don't use subshell to execute fileRichard Purdie2018-07-242-12/+4
| | | | | | | | | We don't need any functionality from the shell here, its just extra fork overhead. Therefore remove it and use subprocess directly. (From OE-Core rev: bcc03ea19e103f6aa93bada2f49fcc5cc7bc0790) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package: Drop subshell usage for dwarfsrcfile generation.Richard Purdie2018-07-241-2/+10
| | | | | | | | | | | | The command for running dwarfsrcfiles is simple and does not need a subshell for each execution. By expanding out this function to use check_output() from subprocess and a list of arguments, the shell overhead can be dropped. For recipes with lots of files this gives a significant saving. (From OE-Core rev: 6334129dfbe266602fab70ce445641053a05be6c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package_ipk|deb: Use oe.utils.multiprocess_launchRichard Purdie2018-07-242-100/+2
| | | | | | | | | | | | | | | The current code had broken exception handling due to the use of a "traceback" variable as well as an import. Use the new library code for this instead which reduces code duplication and has fixed/improved exception handling. The chdir code can be dropped since any directory changes are in other processes now so there is no need for it here and the code no longer changes directory. (From OE-Core rev: bcd47389f4b1fc69d2bb4da01933bfa1fdcae092) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* package: Rework PACKAGELOCK based upon sstate for do_packagedataRichard Purdie2018-07-241-14/+9
| | | | | | | | | | | | | | | | | | I think this lock dates from before we had sstate for do_packagedata. Since WORKDIR is recipe specific and we write into WORKDIR, we no longer need any write locks in the do_packagedata code itself, its handled by the sstate task lock for the final copy in at the end. The final write lock can be simply removed. The only time we need read locking is when actually reading data from the shared directory. We can therefore reduce the window the lock is held significantly as well, hence improving the speed of packagedata tasks running in parallel. (From OE-Core rev: f7106cdf2190d9ec59132a1cb2bb431d653cd9c5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* utils: Add multiprocess_launch API and testcaseRichard Purdie2018-07-242-1/+115
| | | | | | | | | | | | | | | | | | | | | The current methods of spawning processes for parallel execution have issues around collection of results or exceptions. Take the code from package_ipk/deb, make it generic, add a results collection mechanism, fix the exception handling and for it into a standard library function. Also add a test case which tests both the success and failure modes of operation to stop this functionality regressiing again. In particular, compared to multiprocess_exec, this fork off the parent approach means we can pass in the datastore and functions work in the same scope as the parent. This removes some of the complexities found trying to scale multiprocess_exec to wider use. (From OE-Core rev: 88f0c214e593a45566df5131bda4c946f5ccc8c2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-30 04:32:02 +0000