summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: fetch/git: Handle github dropping git:// supportRichard Purdie2021-11-031-0/+4
| | | | | | | | | github is dropping support for git protocol in Git urls. Add code to remap this to https in a way that could be used in older bitbake versions. (Bitbake rev: 0fe1a9e2d2e33f80d807cefc7a23df4a5f760c74) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: clarify the command-no-found error messageRoss Burton2021-10-251-1/+1
| | | | | | | | | | | | | | | | If runfetchcmd() fails with bb.process.NotFoundError, the message output is simply "Fetch command" which doesn't really explain what the problem is. Add "not found" to clarify what happened. (Bitbake rev: 7f3fd1f47f7972756b9b39b6c5ab25084fb4212c) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Make exception printing clearerRichard Purdie2021-10-251-1/+1
| | | | | | | | | | | | | | | | | Shows: ERROR: SystemExit(1) instead of: ERROR: 1 (Bitbake rev: 16bb44e00fcad8daefd3b79c5564d9c8a0d03a6c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit df89e37c33e4398a5f8ece9a8b973be3fe2ff361) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Handle pseudo shutdown in Ctrl+C caseRichard Purdie2021-10-251-4/+6
| | | | | | | | | | | | | | If the build is interrupted, handle the shutdown of pseudo even in this case to avoid data corruption inside docker containers. [YOCTO #14555] (Bitbake rev: 10d61b276d4ffc65640a862c7e5b5852a5716813) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a2a04c6fe94bc56efcff299c669a151746e35916) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Ensure python stdout/stderr is logged correctlyRichard Purdie2021-10-251-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently we see things like: Log data follows: | DEBUG: Executing python function do_pythontest_exit | DEBUG: Python function do_pythontest_exit finished | ERROR: 1 | This is python stdout Whilst after the change we see things like: Log data follows: | DEBUG: Executing python function do_pythontest_exit | This is python stdout | DEBUG: Python function do_pythontest_exit finished | ERROR: 1 since the output is now correctly mixed with the log messages. In some cases the logging tests indicate the output is being lost entirely which is bad for debugging and makes things rather confusing. (Bitbake rev: 3fbc46735ee3679ef1b7466810e4345d78ed05ea) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8966b43761500e0505333d8c9a3f0f2c3dbe7559) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Allow shutdown/database flush of pseudo server at ↵Richard Purdie2021-10-251-1/+9
| | | | | | | | | | | | | | | | | | | | | task exit We have a problem where pseudo server processes exist after bitbake exits and hold the pseudo database in memory. In a docker container, the processes will be killed as the container is destroyed with no warning and no opportunity to write the data to disk. This leads to permissions/inode corruptions and data loss. Send a shutdown message to pseudo which in new versions of pseudo will flush the database, thereby fixing some of the issues people using docker containers see. (Bitbake rev: 7752c2c00245e91aeb17e22de484535190b18e5d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a07a971b40acd3eee12e203d2cfa3e49f56109f6) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: npmsw: Avoid race condition with multiple npm fetchersCaner Altinbasak2021-10-251-0/+4
| | | | | | | | | | | | | | | If multiple npmsw fetchers are trying to download the same npm file, one of them can try to download the file while other is calling verify. npmsw methods gets called without holding the lock, which causes race conditions in fetching and verification etc. Lock the lockfile before calling proxy fetcher methods. (Bitbake rev: 591a05d6126f29b501b9fa284c0618de8c903c69) Signed-off-by: Caner Altinbasak <cal@brightsign.biz> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit fa39e6689d0f0fff772e1c81682698f4b1587b8a) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/runqueue: Ensure hashserv exits before deleting filesRichard Purdie2021-10-251-1/+1
| | | | | | | | | | | | | | We've seen races where the socket may be gone but the server is still writing out it's database. Handle that case too to avoid cleanup tracebacks. [YOCTO #14440] (Bitbake rev: ca25bc22c87b340cf5c430bd4a025586e0e9e30b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b9e4fb843cb9d3a4d4404af093a781fab5520465) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Use os.rename instead of mvRichard Purdie2021-10-251-4/+4
| | | | | | | | | | | | os.rename will overwrite the destination file if present so we can use this instead of the process call overhead. (Bitbake rev: 7be6d18cd74291371f5327dcab2412f508c70189) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit b3cccaa6a896c41d8c9be5eebc327f726542d16b) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Avoid races over mirror tarball creationRichard Purdie2021-10-251-2/+19
| | | | | | | | | | | | | | | | | There is a potential race over the mirror tarballs where a partial git repo could be extracted causing fetcher failures if the tarball is being rewritten whilst another build accesses it. Create the mirror tarball atomically to avoid this. [YOCTO #14441] (Bitbake rev: 7c8f344b81b8f8936214f87f695e24dc4e546659) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3250bc950c56bd7dd2114df26e5a8e13b04ceac8) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: test/fetch: Update urls to match upstream branch name changesRichard Purdie2021-10-191-3/+3
| | | | | | (Bitbake rev: bcd4285116ea4990f10d53698e0a81ae1e7ce24c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Handle SystemExit in python tasks correctlyRichard Purdie2021-10-021-1/+1
| | | | | | | | | | | | | | | | If a python task fails with sys.exit(), we currently see no TaskFailed event. The high level code does detect the exit code and fail the task but it can leave the UI inconsistent with log output. Fix this be intercepting SystemExit explicitly. This makes python task failures consistent with shell task failures. (Bitbake rev: fdc2d3df35ad1282c4ad85d76519e395b023a563) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 9eee9fd4f2f96789ad2b037e74d561bdc1426856) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Avoid duplicating logs in verbose modeRichard Purdie2021-10-021-0/+4
| | | | | | | | | | | | With "bitbake -v", for task failures you'd see the log output twice. Avoid this by using the existing "did we print info" switch. (Bitbake rev: 69c622b744d987b48f7dd805ead973ad90479e4e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit e2c1afda4cb8023ed4ffeb5dc5bee4f0055659a8) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Don't include logs in error message if piping themRichard Purdie2021-10-021-0/+3
| | | | | | | | | | | | | If the caller is piping the logs, they likely don't want them in the error exception as well. This removes duplicate output from the build output allowing the UI level controls on whether to show logs to work correctly. (Bitbake rev: cf864cd84172f605b0e1777c3defc000fa3a7379) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit fc58ad84a9deb2620ad90611684dad65dafedb11) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/knotty: Improve UI handling of setscene task countingRichard Purdie2021-09-243-5/+20
| | | | | | | | | | | | | | | | | | | The recent fixes to merge setscene and normal task accounting in runqueue fixed some display issues but broke the task numbering of setscene tasks. Add new accounting methods to the stats structure specifically designed for setscene. This accounts for the fact that setscene tasks can rerun multiple times in the build. Then use the new data in the UI to correctly display the numbers the user wants to see to understand progress. (Bitbake rev: 7d938703d9321cde5a32e4dff005f07e8821b704) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ed7e2da88bf4b7bfc7ebfc12b9bd6c0fb7d8c1aa) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Show a readable error for invalid multiconfig nameRichard Purdie2021-09-241-0/+2
| | | | | | | | | | | | | | | | | If a multiconfig starts with a digit, users would see pages of errors as we use the multiconfig as a python function name prefix and python functions cannot start with a digit. We could avoid doing that but it is easier just to ask users to name multiconfigs not starting with digits. This tweak ensures the user sees an easier to understand error. (Bitbake rev: fb2c154c8a96fd30d46e34c7ae715d1b964ced35) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f9cddaeef35b2ea0dadf717101ed896f6b857abd) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Improve error handlingRichard Purdie2021-09-241-3/+5
| | | | | | | | | | | | | | | If bitbake-worker fails, return an error code showing that. Also make the thread cleanup code explict in a finally clause as it would otherwise hang. [YOCTO #14393] (Bitbake rev: 0637703046d3f3f7d3bfdd1e5ecb524440897c35) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7e0af70fb53fb13f824ca954b8cc1dffee730233) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/process: Fix typos in exiting messageMartin Jansa2021-09-243-3/+3
| | | | | | | | | | (Bitbake rev: 284ca139b3a2ce61cef91b3076fd8bb544461c16) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1ff1ea3880d293b14ce0fc65e3bc4c938d587a2f) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Clean up task stats handlingRichard Purdie2021-09-242-16/+8
| | | | | | | | | | | | | | | | | | | | | | | | When we parallelised normal and setscene tasks, the task stats handling was left separate pending further thought. We had to remove handling of the setscene tasks from the UI in order to maintain consistent task numbering. Currently, "0 of 0" tasks can be shown as setscene tasks execute until the first normal task runs. The only use left for sq_stats is in the active task numbers which we can use the length of sq_ive for instead. We can therefore drop it and return stats in all cases. This removes the "0 of 0" task problem since the stats in all normal and setscene tasks matches. [YOCTO #14479] (Bitbake rev: fac319e2d5463be2a82335b9cb348c7893a1e65f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit eae6e947e37e18cded053814bd2a268b44fb25cd) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Improve error display for handled exceptionsRichard Purdie2021-09-241-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We don't need tracebacks for BBHandledException. Reduces confusing output like: ERROR: /meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio ERROR: ExpansionError during parsing /meta/recipes-core/images/core-image-tiny-initramfs.bb Traceback (most recent call last): File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_qa', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_cpio', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1038, in follow_chain(task='do_image_complete', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in chain: > bb.fatal("Circular task dependencies as %s depends itself via the chain %s?!" % (task, " -> ".join(chain))) chain.append(task) File "/bitbake/lib/bb/__init__.py", line 165, in fatal: mainlogger.critical(''.join(args), extra=kwargs) > raise BBHandledException() to the real error: ERROR: /media/build1/poky/meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio (Bitbake rev: bdaab86ae7c2c16e5e1a0afa3e3f68130e9d57ca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 551d4c0576a0a0c3406000029df9238b312f2263) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Improve missing core layer error messageRichard Purdie2021-09-241-0/+2
| | | | | | | | | | | | | | If the core layer is missing from bblayers.conf, the message the user sees is hard to understand. Improve it. [YOCTO #14340] (Bitbake rev: b460667572994066921c1971f73c58c081463bfc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 5815a7258ebb8a989e0c6f5798853559d9413f02) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Show error for no BBLAYERS in bblayers.confRichard Purdie2021-09-241-0/+3
| | | | | | | | | | | | | | If there is no BBLAYERS set in bblayers.conf show a more helpful error and exit. [YOCTO #14340] (Bitbake rev: ba79d4711c8010bb7f28784e18a0a42b98284d48) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 97183e10faf9862b5d9489d6e2c27ac77c3b697d) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch2: Use our own git server for dtc test repoRichard Purdie2021-09-221-1/+1
| | | | | | (Bitbake rev: 19c5605b72e25e06f5ea57bcc1672650e7978a7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Catch and error upon circular task referencesRichard Purdie2021-09-161-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | If there are circular task references, error on them rather than show a recursion error. A simple reproducer is: """ do_packageswu () { : } addtask do_packageswu after do_image_complete before do_image_qa """ into image_types.bbclass. There is code in runqueue to detect these but we never get that far with the current codebase. [YOCTO #13140] (Bitbake rev: 4d83d9103d9225a2ac42c0f3dfada2f5cf140b68) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 339d4d6be515a71311b81fb9e99742af0d8a5130) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Improve multiconfig deferred task issuesRichard Purdie2021-09-161-26/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The previous patches have exposed new issues with this code path, the issues being around what should happen when the hash of a task changes and the task is or is not on the deferred task list. Rather than rebuilding the deferred task list during each rehash event, build it once at the start of a build. This avoids the problem of tasks being added back after they have run and also avoids problems of always ensuring the same task is deferred. It also allows the 'outrightfail' codepath to be handled separately as the conditions are subtly differnt. One significant win for the new approch is the build is not continually printing out lists of deferred tasks, that list remains fairly static from the start of the build. Logic is added in to ensure a rehashed task with a hash matching other deferred tasks is deferred along with them as a small optimization. An interesting test case for this code was reported by Mark Hatle with four multiconfigs, each the same apart from TMPDIR and running a build of: bitbake buildtools-tarball mc:{one,two,three,four}:core-image-minimal which is interesting in that the build of buildtools partially overlaps core-image-minimal and the build has a rehash event for qemuwrapper-cross even without any external hash equivalence server or preexisting data. (Bitbake rev: 1775bcb95f61af21712bc1ba5098a49df8f21128) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit bb424e0a6d274d398f434f7df63951da9ce305b3) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Allow upstream for local hash equivalence serverJoshua Watt2021-09-161-1/+6
| | | | | | | | | | | | | | | | | The hash equivalence server has had the option to support a read-only upstream server for some time now when launched as a standalone program, but there was no way to set the upstream when using a locally started server. Add a new variable called BB_HASHSERVE_UPSTREAM that can be used to specify an upstream server when a local hash equivalence server is used (e.g. BB_HASHSERVE is "auto") (Bitbake rev: c225638a1bef06aa7b3c07e37102d550afc0107e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 250fa17f1391ff1ee01ab9b51d2a4f9aa35c1d1e) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix issues with multiconfig deferred task deadlock messagesRichard Purdie2021-09-161-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | In multiconfig builds with large numbers of identical tasks, builds were deadlocking after recent runqueue changes upon rebuilds where there was heavy sstate usage (i.e. on second builds after a first completed). The issue was that deferred tasks were being left indefinitely on the deferred list. The deadlock handler was then "breaking" things by failing tasks that had already succeeded, leading to the task being on both covered and not covered lists, giving a further error. The fix is to clean up the deferred task list when each setscene task completes. I'd previously been hoping to avoid iterating that list but it appears unavoidable. [YOCTO #14342] (Bitbake rev: 27d5b55bb33c319ca595c192c910eac91a5d9428) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit ae24a0f2d2d8b4b5ec10efabd0e9362e560832ea) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Avoid deadlock avoidance task graph corruptionRichard Purdie2021-09-161-1/+2
| | | | | | | | | | | | | If the deferred task deadlock avoidance code triggers, it could mark an executed task as failed which leads to "covered and not covered" error messages. Improve the logic so if the deadlock code is triggered, it doesn't cause the errors. (Bitbake rev: 72f51767da47af12d2c56bd3ee87cb0018b16705) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 51bdd6cb3bd9e2c02e261fb578bb945b86b82c75) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server: Fix early parsing errors preventing zombie bitbakeJoshua Watt2021-09-091-1/+1
| | | | | | | | | | | | | | | | | | | | If the client process never sends cooker data, the server timeout will be 0.0, not None. This will prevent the server from exiting, as it is waiting for a new client. In particular, the client will disconnect with a bad "INHERIT" line, such as: INHERIT += "this-class-does-not-exist" Instead of checking explicitly for None, check for a false value, which means either 0.0 or None. (Bitbake rev: c7b506d11df78cfc4610db6578745eaa6220b13a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 13e2855bff6a6ead6dbd33c5be4b988aafcd4afa) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Allow colon in variable expansion regexRichard Purdie2021-07-302-3/+3
| | | | | | | | | Now that ":" is a valid character in variable key names, it needs to be allowed by the variable expansion code too, to match. (Bitbake rev: 9b2d96b27f550da0fa68ba9ea96be98eb3a832a6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart/parse: Allow ':' characters in variable/function namesRichard Purdie2021-07-304-2/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It is becomming increasingly clear we need to find a way to show what is/is not an override in our syntax. We need to do this in a way which is clear to users, readable and in a way we can transition to. The most effective way I've found to this is to use the ":" charater to directly replace "_" where an override is being specified. This includes "append", "prepend" and "remove" which are effectively special override directives. This patch simply adds the character to the parser so bitbake accepts the value but maps it back to "_" internally so there is no behaviour change. This change is simple enough it could potentially be backported to older version of bitbake meaning layers using the new syntax/markup could work with older releases. Even if other no other changes are accepted at this time and we don't backport, it does set us on a path where at some point in future we could require a more explict syntax. I've tested this patch by converting oe-core/meta-yocto to the new syntax for overrides (9000+ changes) and then seeing that builds continue to work with this patch. (Bitbake rev: f8d1bc200460fc1cb5cbf7a7d1986fd86424b22d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: add check for empty SRC_URI hash stringScott Weaver2021-07-061-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | No error was being reported when the hash string was set to empty. For example: SRC_URI[md5sum] = "" On a related note (not a bug): Because whitespace in the string will result in a checksum mismatch, the error message was updated to make it a little clearer why the error was thrown. For example: SRC_URI[md5sum] = " " or SRC_URI[md5sum] = " 209f8326f5137d8817a6276d9577a2f1" Now creates a message like this: File: '/home/scott/yocto-cache/downloads/rsync-3.2.3.tar.gz' has md5 checksum '209f8326f5137d8817a6276d9577a2f1' when ' 209f8326f5137d8817a6276d9577a2f1' was expected [YOCTO #14232] (Bitbake rev: 731fb52eb03338c0bdb2a2256c22c64c22bcbace) Signed-off-by: Scott Weaver <weaverjs@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a13510d0028e234ea2f4744b0d0c38558395c70f) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/svn: Fix parsing revision of SVN repos with redirectsHarald Brinkmann2021-07-061-1/+1
| | | | | | | | | | | | | svn was printing a message when encountering HTTP redirects. This confused the revision parser. (Bitbake rev: 40b4e2dcd24d3ed01d35d705928133aaa1c4a93b) Signed-off-by: Harald Brinkmann <Harald.Brinkmann@detectomat.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit a944a335f8f4c4fe5df55f3d7d8e757bd2835146) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Handle error in heartbeat funciton in OOM caseRichard Purdie2021-05-201-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've seen cases where an OOM error causes bitbake server to hang: 9171 02:21:09.127810 Command Completed Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/bin/bitbake-server", line 51, in <module> bb.server.process.execServer(lockfd, readypipeinfd, lockname, sockname, timeout, xmlrpcinterface) File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 550, in execServer server.run() File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 108, in run ret = self.main() File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 242, in main ready = self.idle_commands(.1, fds) File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/server/process.py", line 370, in idle_commands bb.event.fire(heartbeat, self.cooker.data) File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 216, in fire fire_class_handlers(event, d) File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 123, in fire_class_handlers execute_handler(name, handler, event, d) File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 93, in execute_handler ret = handler(event) File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 182, in defaultrun_buildstats write_host_data(os.path.join(bsdir, "host_stats"), e, d, "interval") File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 160, in write_host_data output = subprocess.check_output(c.split(), stderr=subprocess.STDOUT, timeout=limit).decode('utf-8') File "/usr/lib/python3.6/subprocess.py", line 356, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 423, in run with Popen(*popenargs, **kwargs) as process: File "/usr/lib/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory We need to wrap the calls in the same high level wrapper as idle function calls and trigger an exit upon an unhandled exception. (Bitbake rev: 0c83e98b0b4129e43e05129074fff60b22a9f2eb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: when checking latest versions, consider all numerical ↵Alexander Kanavin2021-05-201-1/+1
| | | | | | | | | | | | | | directories Previously the regex was maching x.y, but wasn't matching x, which is a problem e.g. here: https://download.gnome.org/sources/epiphany/ (the new gnome version scheme adds 40-series at the end). (Bitbake rev: 8eddd1808387025a22d8ad1b009c2894d19bf79b) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Handle deferred task rehashing in multiconfig buildsRichard Purdie2021-05-061-4/+10
| | | | | | | | | | | | | | | If the hash of a task changes and that hash is a deferred task (e.g. a multiconfig build), we need to ensure that the hash change propagates through to all the tasks else the build will run multiple copies of the task, sometimes with oddly differing results as the outhashes of native tasks built in differing locations can confuse things. (Bitbake rev: b67476d4758915db7a5d9f58bc903ae7501a1774) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 2db571324f755edc4981deecbcfdf0aaa5a97627) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix multiconfig deferred task sstate validity caching issueRichard Purdie2021-05-061-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | We were testing the validity of deferred tasks setscene status "up front" which is very unlikely to succeed and leads to cache invalidation issues. With the change to rebuild the deferred task list, this status becomes out of sync. The result was tasks being executed when they should not have been leading to extra work for the build unnecessarily. Instead, don't process validity status for deferred tasks and assume their data will become available. If it doesn't, this will now result in a build error as the setscene task will fail and the main task will run instead. In theory we could try and track the state changes in the deferred list and re-test validity then but I'm not sure it is worth the effort when the other code path and errors in setscene tasks will give a pretty good idea of what is happening anyway. (Bitbake rev: e70cba8d5861d79ed0da9e760e618af8b759c8a9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit edcafac13b3b241b6687419e59018d21811507a1) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: remove write protected files tooMikko Rapeli2021-04-271-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For some reason several git-annex files in Debian 10 buster are read-only and removing them with "rm -rf" fails. Fixes test failures like: $ bitbake-selftest ... rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied rm: cannot remove '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource/annex/objects/f87/4d5/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/SHA256E-s0--e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855': Permission denied EE..................................ssss.sssssssssssssss.sssss....................................................................................................... ====================================================================== ERROR: test_shallow_annex (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1773, in test_shallow_annex fetcher, ud = self.fetch_shallow(uri) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1541, in fetch_shallow bb.utils.remove(ud.clonedir, recurse=True) File "/home/builder/src/base/poky/bitbake/lib/bb/utils.py", line 700, in remove subprocess.check_call(cmd + ['rm', '-rf'] + glob.glob(path)) File "/usr/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['rm', '-rf', '/tmp/tmpwmfn4w64/download/git2/tmp.tmpwmfn4w64.gitsource']' returned non-zero exit status 1. Also, one "chmod" call was failing since the .git/annex subdirectory doesn't exist so just chmod the whole temporary directory which should cover any directory name differences between different git-annex versions. Fixes tests failing after chmod call: Running 'export PSEUDO_DISABLED=1; unset _PYTHON_SYSCONFIGDATA_NAME; chmod u+w -R /tmp/tmpwmfn4w64/git//.git/annex' in /tmp/tmpwmfn4w64/git/ (Bitbake rev: 14c5f0735947307b9d69c57f7334fefaea7311b3) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7729ef2983c72867e99fad82d671069ba5cb32b2) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch/gitsm: Fix crash when using git LFS and submodulesNiels Avonds2021-04-271-1/+1
| | | | | | | | | | | | | | | | | | | Gitsm fetcher crashes when cloning a repository that contains LFS files. This happens because the unpack method is called during download, but the submodules have not been downloaded yet at this point. This issue was introduced in this commit: 977b7268bf4fd425cb86d4a57500350c9b829162 [YOCTO #14283] (Bitbake rev: 88d1d2b65a70081389a1c8f9b590a013a1cb4452) Signed-off-by: Niels Avonds <niels@codebits.be> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 26caedc4d2e9b5a0f1d57f9291754a7f6c5e437e) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-server: ensure server timeout is a floatRoss Burton2021-04-272-2/+2
| | | | | | | | | | | | | | | | | | | | | bitbake-server is spawned by process.py and passes the arguments it is given to ProcessServer. There's some type confusion here: bitbake-server is called with a string representation of the timeout, which may be None. If the timeout is not set, pass 0 instead of None. Inside bitbake-server a ProcessServer is created which expects the timeout to be a float not a string, so always float() the value. [ YOCTO #14350 ] (Bitbake rev: f2cfb9f6710808ea37aecb6c34c62f92191e1d4b) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c93ae1f861208f6d39fd15c84fbcd0e2b54331f5) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: fix test execution without .gitconfigMikko Rapeli2021-04-271-1/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A CI user validating changes does not have any git push rights or even a .gitconfig file so fix tests so that they run by setting the user.name and user.email for the repo before committing changes. Fixes errors like: ERROR: test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist (bb.tests.fetch.GitShallowTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 2055, in test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist self.add_empty_file('a') File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1562, in add_empty_file self.git(['commit', '-m', msg, path], cwd) File "/home/builder/src/base/poky/bitbake/lib/bb/tests/fetch.py", line 1553, in git return bb.process.run(cmd, cwd=cwd)[0] File "/home/builder/src/base/poky/bitbake/lib/bb/process.py", line 184, in run raise ExecutionError(cmd, pipe.returncode, stdout, stderr) bb.process.ExecutionError: Execution of 'git commit -m a a' failed with exit code 128: *** Please tell me who you are. Run git config --global user.email "you@example.com" git config --global user.name "Your Name" to set your account's default identity. Omit --global to set the identity only in this repository. (Bitbake rev: 1e1d1187e602aa1ef50c23551eec07f1a0cd81ef) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 57c0811f1ee19b6619f4840a39e01e3cb98c34c4) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix deferred task issuesRichard Purdie2021-04-271-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In a multiconfig situation there are circumstances where firstly, tasks are deferred when they shouldn't be, then later, tasks can end up as both covered and not covered. This patch fixes two related issues. Firstly, the stamp validity checking is done up front in the build and not reevaulated. When rebuilding the deferred task list after scenequeue hash change updates, we need therefore need to check if a task was in notcovered *or* covered when deciding to defer it. This avoids strange logs like: NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene) NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch where tasks have run but are then deferred. Since we're recalculating the whole list, we also need to clear it before iterating to rebuild it. By ensuring covered tasks aren't added to the deferred queue, the covered + notcovered issue should also be avoided. in the task deadlock forcing code. [YOCTO #14342] (Bitbake rev: 1ec855731800cf8e2bae2b1e7241640e0bad8aae) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 3c8717fb9ee1114dd80fc1ad22ee6c9e312bdac7) Signed-off-by: Anuj Mittal <anuj.mittal@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: add REQUIRED_VERSION and adjust ↵Paul Eggleton2021-04-151-2/+16
| | | | | | | | | | | | PREFERRED_VERSION entry Add REQUIRED_VERSION, add a reference to it in PREFERRED_VERSION and adjust the opening statement to read slightly better. (Bitbake rev: b32e6c8d4ea2f83fe77021207e9db883fec82d97) Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: document no support for using passwords in git ↵Paul Eggleton2021-04-151-0/+9
| | | | | | | | | | | | URLs This is based on the comment added in revision aded964eed4ce5a725ed1ab477efabc86b1aa481. (Bitbake rev: 082683da089115d8b6f71f221cabb41ac401f733) Signed-off-by: Paul Eggleton <paul.eggleton@microsoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Update version to 1.50.0 stable release seriesRichard Purdie2021-04-092-2/+2
| | | | | | (Bitbake rev: e70b925ba98fd4fedf3940d141a4210c953087ca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Further fixes for confused setscene tasksRichard Purdie2021-04-061-13/+26
| | | | | | | | | | | | | | | There is further evidence of tasks ending up being "covered" and "notcovered" which shouldn't happen and is bad. The code that caused this problem last time appears to have issues where stamps for tasks already exist. Split out the setscene stamp checking code to a separate function and use this when checking "hard dependencies" (like pseudo-native) so that if the stamps exist and it will be "covered", it is not put on the notcovered list. (Bitbake rev: a1848a481e36b729c8e4130c394b1d462d4b488a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: Update links to documentationMichael Opdenacker2021-03-232-2/+2
| | | | | | | (Bitbake rev: ed8e1fd4cf9d5ac8a8203638add99d686b4b3521) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/event: Add an event for notifying of stale setscene tasksRichard Purdie2021-03-232-0/+25
| | | | | | | | | | | Use the new functionality in build.py to identify stale setscene tasks and send an event to the metadata listing them. The metadata then has the option of performing cleanup operations if it thinks that appropriate. (Bitbake rev: ef8c980a3ae92c168b7ca16a4d19cd38a9574761) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Add find_stale_stamps functionRichard Purdie2021-03-231-0/+17
| | | | | | | | | | Add a new function which compares the stamp filename we want (including taskhash) with what is in the stamp directory (using the clean mask). This tells us which stamp files are stale and are due to be rerun. (Bitbake rev: b126a53882d202e4df0f9661303355c9fe9ec80e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: fix syntax error in layer.conf exampleRobert P. J. Day2021-03-221-1/+2
| | | | | | | | | | | While this example really needs to be rewritten to not define multiple patterns in the same layer.conf, as long as it's there, it might as well be syntactically correct. (Bitbake rev: 1c2d2875099a3ff9149710d42c679ab31b00f68b) Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-09 14:17:00 +0000