summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
...
* libxml2: Fix CVE-2019-19956Lee Chee Yang2020-02-042-0/+41
| | | | | | | | | | | see: https://gitlab.gnome.org/GNOME/libxml2/commit/5a02583c7e683896d84878bd90641d8d9b0d0549 (From OE-Core rev: 9fd363d21df59fff4488763d62548c936b4d14fb) Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Adding memoriam to scottrifJefro2020-01-311-0/+5
| | | | | | | | | | Added a few comment lines in Makefile to commemorate Scott's contributions (From yocto-docs rev: fba30da7b4e55838ce3b19a6a7cc658bc686b947) Signed-off-by: Jefro <jefro@jefro.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Use a set for the setscene tasks listRichard Purdie2020-01-281-2/+2
| | | | | | | | | | | | | This should give performance improvements to functions using this list of tasks (sets are used for most of the other code for this reason, not sure why this wasn't a set in the first place). (Bitbake rev: 1a026d4aca47ed1b0b74a8a818635e7520e8f9c8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f5daef68703481a3c243dfecc7de404e6ebfdbb6) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Be consistent when creating/updating progress barsPeter Kjellerstedt2020-01-281-6/+6
| | | | | | | | | | | | | | | | | | When creating a new progress bar (using BBProgress), a colon was appended to the supplied message. However, when updating the message, no colon was appended. Change this so that the colon is instead part of the widgets that make up the progress bar so that it does not matter when and how the message is updated, it always displays the same. (Bitbake rev: 1115a14c93dc44124c3ab95053142da78032a004) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 08f35c04f6e1ce4c4ca5c2bef4cd8a192e12e682) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Hide the footer if a process progress bar is shownPeter Kjellerstedt2020-01-281-1/+4
| | | | | | | | | | | | | | | With the introduction of the hash equivalence server, the progress bar for "Checking sstate mirror object availability" is shown repeatedly while the tasks are being executed. If the footer is not hidden then, it will be moved up one line every time, creating a messy interface. (Bitbake rev: b7ecb1d0232c12b8388ae516b1dbd025ad782d56) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 56b5ec4c2b3e658e73ca6c3a12feeb96df0977fb) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Keep track of watched files using a set instead of a listPeter Kjellerstedt2020-01-281-8/+8
| | | | | | | | | | | | | When there are many watched files, keeping track of them using lists is suboptimal. Using sets improves the performance considerably. (Bitbake rev: 16799ada1cda8e021cb04c43e469225790525723) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1e96df260e47d160dbd36bfc92c31ef06266f662) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstatesig: Improve debug output if getpwuid() failsRichard Purdie2020-01-281-2/+6
| | | | | | | | | | | | If getpwduid fails, we don't see which file it failed on which is key information to aid debugging. Print this information when exceptions are raised. (From OE-Core rev: 2bba0052597020ea887c84419440df11f9859283) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9d7a5219713af8117eda145052c6d9abdf577d8f) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* scripts/oe-build-perf-report: Avoid buildstats warningRichard Purdie2020-01-281-6/+14
| | | | | | | | | | | | | | | | | | | | When a comparision branch is set we see warnings. If two branches are used, look for buildstats information from both to avoid this kind of warning: INFO: Parsing buildstats from 'refs/notes/buildstats/perf-centos7.yoctoproject.org/master-next/qemux86' WARNING: Buildstats not found for perf-centos7.yoctoproject.org/master/qemux86/56808-g08220028e52992dcba667fc97bf3afe8be1949fb/0 WARNING: Buildstats not found for perf-centos7.yoctoproject.org/master/qemux86/56808-g08220028e52992dcba667fc97bf3afe8be1949fb/1 INFO: Buildstats were missing for some test runs, please run 'git fetch origin refs/notes/buildstats/perf-centos7.yoctoproject.org/master-next/qemux86:refs/notes/buildstats/perf-centos7.yoctoproject.org/master-next/qemux86' and try again (the warning is due to stats being looked for in both branches, choose the other branch and different warnings would be shown) (From OE-Core rev: 232b2f6dabaedff087df15a7e6d431dd9f314b4c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit af58c18fcd0451540dd77ee862956b2b9dee17c4) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sstatesig: Test cross/native hashserv method extensionRichard Purdie2020-01-281-0/+9
| | | | | | | | | | | | | | | | | We can have one taskhash which represents multiple native/cross sstate objects since they're stored by BUILD_ARCH or possibly host distro (or host gcc version). We need to put these into separate namespaces on hashserv since their outhashes will never match and we need deterministic lookups for the different namespaces. Use this extramethod option to handle this. This fixes several problematic failures on the autobuilder. (From OE-Core rev: 766b57a1f9030429b546276a537fbce3a175cc25) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2a76082363d189880613765ad339718e3614049d) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* wic/filemap: If FIGETBSZ iotctl fail, failback to os.statKalle Lampila2020-01-281-2/+4
| | | | | | | | | | | | | | Some file systems don't support fetching the block size (notably the file system Docker uses for containers), so if iotctl() fail, try to use failback via os.stat() to get block size. (From OE-Core rev: 996013b4e5d3cd1c054e87539ac1a8d95581a87f) Signed-off-by: Kalle lampila <kalle.lampila@lempea.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e219f5175177a640dd62833082ea19adc1c13d42) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* multilib.conf: Ensure that RECIPE_SYSROOT is unchanged for nativeMike Crowe2020-01-282-0/+41
| | | | | | | | | | | | | | | | | | | | | | | Ensure that RECIPE_SYSROOT is the same for -native recipes whether multilib.conf is included or not. Without this change task signatures for -native recipes change when switching between MACHINEs that require multilib.conf and those that don't. This fix was one of the ones suggested by Khem Raj in http://lists.openembedded.org/pipermail/openembedded-core/2019-December/290303.html Add test_sstate_multilib_or_not_native_samesigs test case to sstatetests.py to ensure that this stays fixed. (From OE-Core rev: c3e957409bf68e7b310e27781ac91d24cf4452bf) Signed-off-by: Mike Crowe <mac@mcrowe.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit aa05f1ded71366b86eda7fce24d8b5395e85ada2) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* linuxloader: Correct loader for glibc on armhfAlex Kiernan2020-01-281-1/+1
| | | | | | | | | | | | | | | Fix the naming of the loader on armhf to match what glibc expects: #define GLIBC_DYNAMIC_LINKER_SOFT_FLOAT SYSTEMLIBS_DIR "ld-linux.so.3" #define GLIBC_DYNAMIC_LINKER_HARD_FLOAT SYSTEMLIBS_DIR "ld-linux-armhf.so.3" (From OE-Core rev: 70efe880589df1dfe7d71451d62d9d8e4afd7981) Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7ad531093d620f18c04d487d79e9dad9f5a96232) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* license.bbclass: Introduce AVAILABLE_LICENSES that lists all licensesPeter Kjellerstedt2020-01-283-7/+27
| | | | | | | | | | | | | | | | | | | | | | | | | Previously, there was SRC_DISTRIBUTE_LICENSES, an undocumented variable that contained a static list of licenses. It was used by expand_wildcard_licenses() to expand any wildcards used in, e.g., INCOMPATIBLE_LICENSE. However, since this static list of licenses has not been kept up-to-date, many licenses were missing, with the result that if one tried to use any of those licenses with a wildcard, no licenses would be found, effectively ignoring that they should be marked as incompatible. This introduces a new (documented) variable, AVAILABLE_LICENSES, that is automatically updated to contain all licenses found in any directories specified by ${COMMON_LICENSE_DIR} and ${LICENSE_PATH}, and uses it instead of SRC_DISTRIBUTE_LICENSES when expanding wildcards. (From OE-Core rev: 5ed714139f91eb03871e01b68a4370784071234d) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8c9ef587fe499c612a878a1ab42092eb79b334ef) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* licenses.conf: Remove the SRC_DISTRIBUTE_LICENSES variablePeter Kjellerstedt2020-01-281-43/+0
| | | | | | | | | | | | | | The SRC_DISTRIBUTE_LICENSES variable and its static list of licenses has been replaced by AVAILABLE_LICENSES, which automatically contains all available licenses. (From OE-Core rev: c41f1c7f33a8a13ed8e6fd86b494cf460328a9e4) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 64daaf29e2c12c8b587bafdebf9409433187ddf7) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sanity.bbclass: Move sanity_info from conf to cachePeter Kjellerstedt2020-01-282-2/+2
| | | | | | | | | | | | | | | | | Since this file is written during recipe parsing, having it in the ${BUILDDIR}/conf directory, which is covered by an inotify watcher, will trigger a re-parse the next time bitbake is run and the resident bitbake server is enabled. This causes the sanity_info file to be updated again, which triggers a new parse the next time bitbake is run ad infinitum. Moving it to ${BUILDDIR}/cache should avoid this. (From OE-Core rev: f1a609e7176a37e7e5f86f0b786f7da915085f83) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f98103b548aa7dba6b1be6c8e02ef41858a8e85c) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* populate_sdk_ext.bbclass: No longer needed to clean away conf/sanity_infoPeter Kjellerstedt2020-01-281-1/+1
| | | | | | | | | | | | | | | Since the sanity_info file has moved from the conf directory to the cache directory, there is no longer any need to clean it away explicitly in clean_esdk_builddir() since the whole cache directory is already cleaned away anyway. (From OE-Core rev: f0ea8b49a183d9340aaf0c60ee959aa8ec3c041c) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 40c30990e1be72130819c040fe471e2bdc0c6e7d) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* toaster.bbclass: Correct pkgdatadir path in toaster_package_dumpdata()Peter Kjellerstedt2020-01-281-4/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In commit 692b2046 (package: Fix race between do_package and do_packagedata), the path used for do_packagedata[sstate-inputdirs] was changed from "${PKGDESTWORK}" to "${WORKDIR}/pkgdata-pdata-input". This commit adapts the path used for pkgdatadir in toaster_package_dumpdata() accordingly to avoid setscene errors like: ERROR: libgcc-9.2.0-r0 do_packagedata_setscene: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:toaster_package_dumpdata(d) 0003: File: 'meta/classes/toaster.bbclass', lineno: 130, function: toaster_package_dumpdata 0126: lpkgdata = {} 0127: datadir = os.path.join(pkgdatadir, 'runtime') 0128: 0129: # scan and send data for each generated package *** 0130: for datafile in os.listdir(datadir): 0131: if not datafile.endswith('.packaged'): 0132: lpkgdata = _toaster_load_pkgdatafile(datadir, 0133: # Fire an event containing the pkg data 0134: bb.event.fire(bb.event.MetadataEvent( Exception: FileNotFoundError: [Errno 2] No such file or directory: 'tmp/work/mips32r2el-nf-poky-linux/libgcc/9.2.0-r0/pkgdata/runtime' (From OE-Core rev: c1d881448fea0bc48aba0bc0e5b8748442572226) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5a0f6f631b86f7107aa72453b6d23f32ba39f713) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Lower debug level for wold build messagesRichard Purdie2020-01-111-3/+3
| | | | | | | | | | | | These messages spam the logs for no good reason, they were useful for debugging a particular problem long ago but are distracting noise now. Disable them. (Bitbake rev: 46969195492708a705390896c9dea515f158954c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1a9247c468cf09da60e5d396ccb81e950841c99e) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Test extra cross/native hashserv methodRichard Purdie2020-01-111-6/+21
| | | | | | | | | | | | | | | | | | | Hack the hashserv to allow extra data to be injected into the hashserv method. This allows OE-Core to handle cases where there are multiple sstate objects for the same taskhash, e.g. native/cross objects based upon BUILD_ARCH or the host distro (when uninative isn't used). This has been tested and proven to be very effective. We will likely rework the code to improve how this is handled but for now this improves automated builds until we can get to that refactoring and more invasive changes. (Bitbake rev: e21bf572cfe18e805d3f849777189685f7391a67) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0a09b0fa03d1afc08037964dc63a18ef7cff9c78) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task dependency corner case in sanity testRichard Purdie2020-01-111-1/+4
| | | | | | | | | | | | | | A corner case was identified where tasks with valid stamps from previous builds need to be accounted for in the new sanity test in the migration code. Add a variable to track such completed tasks to ensure the sanity test works correctly. (Bitbake rev: 39cc4930676970b2e51b99a778f5d5d9b5259991) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d517b1ef13ca7ab2fb4d761d3bd3b9fb7c591514) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure task dependencies are run correctlyRichard Purdie2020-01-111-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | We've seen a number of mystery failures where task B would run despite task A, its dependency not having run. An example would be do_compile when do_unpack didn't run. This has been tracked down to this code block. In theory it shouldn't trigger however it can and has due to bugs elsewhere. When it does, it causes significant weird failures and possible build corruption. Change the code to abort the build. This avoids any chance of corruption and should ensure the issues get reported, putting an end to the weird build failures. There may be some cases where this triggers and it shouldn't, we'll work through those as they arise and are identified. (Bitbake rev: 182b2ae7bab4a943978154be3a65c018aaf21fdc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7a92b7f58ab187eddfe550bd6fb687240c7b11bb) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix equiv hash handling build failuresRichard Purdie2020-01-111-1/+2
| | | | | | | | | | | | | | | | Regardless of whether we remapped the hash on the server or not, we need to have bitbake work as if we did as we need to match how the stamp files look. This change resolves build failures where tasks were rerunning when they shouldn't. (Bitbake rev: 10058fe590c56ee3b4d2136b6e247c2d29ae47e6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 40928f6991436cf687821015324483b205abfcb1) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Use with while reading pidfileOla x Nilsson2020-01-111-7/+5
| | | | | | | | | | (Bitbake rev: e731f72ca86629eb935bb1c4f7323421a125aace) Signed-off-by: Ola x Nilsson <olani@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6fa8a18ea4994031fdd1253fe363c5d8eeeba456) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Add BB_SIGNATURE_LOCAL_DIRS_EXCLUDE to speed-up taskhash on ↵Aníbal Limón2020-01-113-6/+8
| | | | | | | | | | | | | | | | | | directories The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list of directories to exclude when making taskhash, our specific case is using SRC_URI that points local VCS directory. Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn" (Bitbake rev: 4c7d689341f471efdf8ab7c2a7c6a19d2d370f5c) Signed-off-by: Aníbal Limón <anibal.limon@linaro.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 923aff060d8aba8456979c35b16d300ba7c13ff9) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Optimise out debug messages from cookerRichard Purdie2020-01-111-0/+5
| | | | | | | | | | | | | | | | | | | We have bb.debug(2, xxx) messages in cooker which are useful for debugging but have really bad effects on performance, 640,000 calls on recent profile graphs taking tens of seconds. Rather than commenting out debug which can be useful for debugging, don't create events for debug log messages from cooker which would never be seen. We already stop the messages hitting the IPC but this avoids the overhead of creating the log messages too, which has been shown to be signficiant on the profiles. This allows the code to perform whilst allowing debug messages to be availble when wanted/enabled. (Bitbake rev: f224201b5003b89367d688b1f08a062754fce13a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f04cd931091fb0508badf3e002d70a6952700495) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Only call into the migrations function if migrations activeRichard Purdie2020-01-111-1/+2
| | | | | | | | | | | This doesn't save much time but does make the profile counts for the function more accurate which is in itself useful. (Bitbake rev: 53a3cba93401c902d1d214cafe0bc036e1b101e5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit d446fa89d206fbc6d098215163c968ea5a8cf4a9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise task filteringRichard Purdie2020-01-111-3/+2
| | | | | | | | | | | We were seeing this running thousands of times with hashequiv, do the filtering where it makes more sense and make it persist. (Bitbake rev: 7fa3ede953971f761d8cafafa37d1bcb65ad6a77) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2cfeb9998a8ad5b1dcda0bb4e192c5e4306dab17) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise out pointless loop iterationRichard Purdie2020-01-111-9/+3
| | | | | | | | (Bitbake rev: 14babbf0a32637db640e0494b0331397dcb0f715) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 105d1f0748edde7753a4063e6fdc758ffc8a8a9e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Optimise task migration code slightlyRichard Purdie2020-01-111-8/+9
| | | | | | | | | | | | | | | Move the calls to difference_update out a code level which improves efficiency significantly. Also further combine the outer loop for efficiency too. These two changes remove a bottleneck from the performance charts. (Bitbake rev: 57df7f191755dd887827d51b125d246c1af3e1b6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e28ec69356f1797de3e4e3fca0fef710bc4564de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix sstate task iteration performanceRichard Purdie2020-01-111-1/+5
| | | | | | | | | | | | | Creating a new sorted list of sstate tasks each iteration through runqueue is extremely ineffecient and was compounded by the recent change from a list to set. Create one sorted list instead of recreating it each time. (Bitbake rev: a80024a4397485f96693c4df108b40f6b1bdf32b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit de18824996841c3f35f54ff5ad12f94f6dc20d88) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Split get_tashhash for performanceRichard Purdie2020-01-112-9/+25
| | | | | | | | | | | | | | | | | There are two operations happening in get_taskhash, the building of the underlying data and the calculation of the hash. Split these into two funtions since the preparation part doesn't need to rerun when unihash changes, only the calculation does. This split allows sigificant performance improvements for hashequiv in builds where many hashes are equivalent and many hashes are changing. (Bitbake rev: c4ce216b4c5a1626764752edd21005750d05a0c2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6a32af2808d748819f4af55c443578c8a63062b3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix task mismatch failures from incorrect logicRichard Purdie2020-01-111-1/+1
| | | | | | | | | | | | The "no dependencies" task case was not being correctly considered in this code and seemed to be the cause of occasionaly task hash mismatch errors that were being seen as the dependencies were never accounted for properly. (Bitbake rev: 480939538f5887a70d11c2f7cf74446100e25fab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 608b9f821539de813bfbd9e65950dbc56a274bc2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Rework process_possible_migrations() to improve performanceRichard Purdie2020-01-111-47/+56
| | | | | | | | | | | | | | | | | The looping over multiple changed hashes causes many calls to get_taskhash and get_unihash which are potentially slow and then overwritten. Instead, batch up all the tasks which have changed unihashes and then do one big loop over the changed tasks rather than each in turn. This makes worlds of difference to the performance graphs and should speed up build where many tasks are being rehashed. (Bitbake rev: c9ab598f6f1ea3ae3a0713dc6692b4c4bafbfb50) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c9c68d898985cf0bec6fc95f54c151cc50255cac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bb.utils.fileslocked: don't leak files if yield throwsChris Laplante via bitbake-devel2020-01-111-4/+5
| | | | | | | | | | | | | | | Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by externalsrc.bbclass) was leaked, giving a warning like: WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'> exec_func(task, localdata) (Bitbake rev: 81829ab28afae08e02f4a758ec063fc0d90579ea) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 6beddf6214e22b4002626761031a9e9d34fb04db) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix performance issue in get_unihashRichard Purdie2020-01-111-3/+3
| | | | | | | | | | | | There is a significant performance issue in get_unihash(). The issue turns out to be the lookups of setscene tasks. We can fix this by using a set() instead of the current list. (Bitbake rev: 5afad266f2ce55db2038c36f2e49a3c80be9bbfc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1e561672d039ebfb8cd0e0654a44dcf48513317c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Batch scenequeue updatesJoshua Watt2020-01-111-4/+7
| | | | | | | | | | | | | | Batch all updates to scenequeue data together in a single invocation instead of checking each task serially. This allows the checks for sstate object to happen in parallel, and also makes sure the log statement only happens once (per set of rehashes). (Bitbake rev: a7426c73a8e9fae468414a2c32a533d9c3729405) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit db033a8f8a276d864bdb2e1eef159ab5794a0658) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Ensure new unihash propagates through the systemRichard Purdie2020-01-111-0/+1
| | | | | | | | | | | | Its possible the new unihash may not exist in sstate. Currently the code would create an sstate object with the old hash however this updates it to create the object with the new unihash. (Bitbake rev: 0aee83e4e31dff7f4354e4eb4cbd35dd592e9f06) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit abcaa1398031fa5338a43859c661e6d4a9ce863d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Avoid taskhash mismatch errors for nostamp tasks when ↵Richard Purdie2020-01-111-4/+8
| | | | | | | | | | | | | | | | | | | | | dependencies rehash An example: NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Started ERROR: Taskhash mismatch b074da4334aff8aa06572e7a8725c941fa6b08de4ce714a65a90c0c0b680abea versus 17375278daed609a7129769b74a1336a37bdef14b534ae85189ccc033a9f2db4 for /home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb:do_check NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Succeeded Is caused by a rehash in a dependency happening somewhere earlier in the build and the taint being reset. Change the code so that nostamp taints are preserved to avoid the issue. (Bitbake rev: c42d00ff293d0538cad1b84c108bf7f5f49d4d84) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 61624a3fc38e8546e01356d5ce7a09f21e7094ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty/uihelper: Switch from pids to tids for Task event managementRichard Purdie2020-01-113-33/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've seen cases where a task can execute with a given pid, complete and a new task can start using the same pid before the UI handler has had time to adapt. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/knotty.py", line 484, in main helper.eventHandler(event) File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/uihelper.py", line 30, in eventHandler del self.running_tasks[event.pid] KeyError: 13490 This means using pids to match up events on the UI side is a bad idea. Change the code to use task ids instead. There is a small amount of fuzzy matching for the progress information since there is no task information there and we don't want the overhead of a task ID in every event, however since pid reuse is unlikely, we can live with a progress bar not quite working properly in a corner case like this. [YOCTO #13667] (Bitbake rev: a109d034cf4fc059fd5a1e1d03246dac65522dd6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e427eafa1bb04008d12100ccc5c862122bba53e0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Add extra debugging when locked sigs mismatches occurRichard Purdie2020-01-111-0/+2
| | | | | | | | (Bitbake rev: 6f0b82627edc82601f80f0f096bf96db43afefa8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3aad9978be2a40d4c535a5ae092f374ba2a5f627) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/siggen: Allow handling of equivalent hashesRichard Purdie2020-01-112-12/+45
| | | | | | | | | | | | | | | Based on the hashserv's new ability to accept hash mappings, update runqueue to use this through a helper function in siggen. This addresses problems with meta-extsdk-toolchain and its dependency on gdb-cross which caused errors when building eSDK. See the previous commit for more details. (Bitbake rev: 222df6d6b832868c6e87334f8acdd74b730a91d6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 39098b4ba2133f4d9229a0aa4fcf4c3e1291286a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: hashserv: Add support for equivalent hash reportingRichard Purdie2020-01-112-0/+44
| | | | | | | | | | | | | | | | | | | | | | | | | The reason for this should be recorded in the commit logs. Imagine you have a target recipe (e.g. meta-extsdk-toolchain) which depends on gdb-cross. sstate in OE-Core allows gdb-cross to have the same hash regardless of whether its built on x86 or arm. The outhash will be different. We need hashequiv to be able to adapt to the prescence of sstate artefacts for meta-extsdk-toolchain and allow the hashes to re-intersect, rather than trying to force a rebuild of meta-extsdk-toolchain. By this point in the build, it would have already been installed from sstate so the build needs to adapt. Equivalent hashes should be reported to the server as a taskhash that needs to map to an specific unihash. This patch adds API to the hashserv client/server to allow this. [Thanks to Joshua Watt for help with this patch] (Bitbake rev: 0d154434ed8e3e88ad440a8dd21a164e72ba4ac5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 674692fd46a7691a1de59ace6af0556cc5dd6a71) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* kernel: Make symbol link to vmlinux.64 in boot directoryVasyl Gomonovych2020-01-111-0/+3
| | | | | | | | | | | | | Some mips 64 bit platforms use vmlinux.64 image name Make a symbol link to vmlinux.64 in arch/mips/boot/ (From OE-Core rev: f9739e0f58f5d8a5dc01fdb0efaa778af4edd671) Signed-off-by: Vasyl Gomonovych <gomonovych@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 70626b52e5e61c80018e9f1f85bc169d0434b986) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* libsdl2: fix failure appeared in bitbake libsdl2-nativeYanfei Xu2020-01-111-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When you bitbake libsdl2-native and your host had installed fcitx or ibus, it may cause you bitbake failed in do_compile. Disable fcitx and ibus check with EXTRA_OECONF in bb file. -------------snip--------------------- CC build/SDL_sysfilesystem.lo CC build/SDL_syshaptic.lo CC build/SDL_systimer.lo /poky/build/tmp/work/x86_64-linux/libsdl2-native/2.0.10-r0/SDL2-2.0.10/src/core/linux/SDL_fcitx.c:52:5: error: unknown type name ‘SDL_DBusContext’ SDL_DBusContext *dbus; ^~~~~~~~~~~~~~~ /poky/build/tmp/work/x86_64-linux/libsdl2-native/2.0.10-r0/SDL2-2.0.10/src/core/linux/SDL_fcitx.c:121:8: error: unknown type name ‘DBusHandlerResult’ static DBusHandlerResult ^~~~~~~~~~~~~~~~~ /poky/build/tmp/work/x86_64-linux/libsdl2-native/2.0.10-r0/SDL2-2.0.10/src/core/linux/SDL_fcitx.c:122:20: error: unknown type name ‘DBusConnection’; ------------snip---------------------- (From OE-Core rev: f48ab4268746a2780ee0fdc84e5c39afe593ade7) Signed-off-by: Yanfei Xu <yanfei.xu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit eb286172e413e9d6449dc13ccec1b78c04481eb5) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* pseudo: Make realpath() remove trailing slashesRobert Yang2020-01-112-0/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Linux system's realpath() remove trailing slashes, but pseudo's doesn't, need make them identical. E.g., the following code (rel.c) prints '/tmp' with system's realpath, but pseudo's realpath prints '/tmp/': #include <stdio.h> #include <limits.h> #include <stdlib.h> int main() { char out[PATH_MAX]; printf("%s\n", realpath("/tmp/", out)); return 0; } $ bitbake base-passwd -cdevshell # For pseudo env $ gcc rel.c $ ./a.out /tmp/ (but should be /tmp) This patch fixes the problem. (From OE-Core rev: c251e753486dae9b460df315a6d19d6c81098ecb) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 319bbf66e03377adf2db7efa93ef578e3460eb38) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Update to apply libgcc fix to libpython, glibc onlyRichard Purdie2020-01-111-1/+1
| | | | | | | | | | | Update to account for review feedback on list. (From OE-Core rev: 9a2748db44c4382bbba81a5a9b96c998f0fab983) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ec788594f3f6a47687c6eb321437f2d2b58b1518) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: RDEPEND on libgccJoshua Watt2020-01-111-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | =help> List-Subscribe: <http://lists.openembedded.org/mailman/listinfo/openembedded-core>, <mailto:openembedded-core-request@lists.openembedded.org?subject=subscribe> Content-Type: text/plain; charset="us-ascii" Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org X-Virus-Scanned: clamav-milter 0.101.4 at dan X-Virus-Status: Clean X-Evolution-Source: 1525863794.3857.16@hex Content-Transfer-Encoding: 8bit Python uses features of glibc that require it to dynamically load (i.e. dlopen()) libgcc_s at runtime. However, since this isn't a link time dependency, it doesn't get picked up automatically by bitbake so manually add it to RDEPENDS. There is an outstanding bug in Python to make it explicitly link against libgcc at link time which would remove the need for this. See: https://bugs.python.org/issue37395 (From OE-Core rev: e6c4017727008ac2f665e843d59d53b584f3f1b3) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit df107f3a149b1e88d9f869e7ff87950ccf5aaee0) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* python3: Upgrade 3.7.5 -> 3.7.6Adrian Bunk2020-01-111-2/+2
| | | | | | | | | | | (From OE-Core rev: b8926f3898fbf6828b908d741ab3b450adb85643) Signed-off-by: Adrian Bunk <bunk@stusta.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit aee9beb12226abf7a195b8ee801ea488920b2fdb) [Bug fix only update] Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* systemd: Upgrade 243 -> 243.2Adrian Bunk2020-01-114-3/+1
| | | | | | | | | | | Bugfix-only changes on the 243 stable branch. (From OE-Core rev: fd1aa49d8a25767953e8b1beb91001d5e115f781) Signed-off-by: Adrian Bunk <bunk@stusta.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* podfix: only alter normal filesRoss Burton2020-01-111-0/+3
| | | | | | | | | | (From OE-Core rev: dcbaf2a7a444f07ab516833f4ed4a2322828bddd) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 23d38b2ad6f7a39e5c1ffd092322942474935c33) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-07 06:18:22 +0000