summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
...
* bitbake: build: Fix log flushing raceRichard Purdie2021-09-231-38/+38
| | | | | | | | | | | | | | There is a race between sending the TaskFailed event which could trigger the UI to look at the logs and flushing and closing the log files. Reverse the order of the finally clause and the exception handling to ensure we've handled the logfiles before sending the task events. This should fix a race seen in bblogging.BitBakeLogging.test_python_exit_logging (Bitbake rev: 032190aac31604d37740d8aecf6e74a5448de358) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Make exception printing clearerRichard Purdie2021-09-231-1/+1
| | | | | | | | | | | | | | Shows: ERROR: SystemExit(1) instead of: ERROR: 1 (Bitbake rev: df89e37c33e4398a5f8ece9a8b973be3fe2ff361) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Ensure python stdout/stderr is logged correctlyRichard Purdie2021-09-231-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | Currently we see things like: Log data follows: | DEBUG: Executing python function do_pythontest_exit | DEBUG: Python function do_pythontest_exit finished | ERROR: 1 | This is python stdout Whilst after the change we see things like: Log data follows: | DEBUG: Executing python function do_pythontest_exit | This is python stdout | DEBUG: Python function do_pythontest_exit finished | ERROR: 1 since the output is now correctly mixed with the log messages. In some cases the logging tests indicate the output is being lost entirely which is bad for debugging and makes things rather confusing. (Bitbake rev: 8966b43761500e0505333d8c9a3f0f2c3dbe7559) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch2: Use our own git server for dtc test repoRichard Purdie2021-09-221-1/+1
| | | | | | (Bitbake rev: 36f6dce1b21a7d9f39a73f081395c71045960318) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/svn: Allow peg-revision functionality to be disabledRichard Purdie2021-09-211-2/+8
| | | | | | | | | | | | Sometimes the peg revision functionality we use in the svn fetcher is not the correct option. Add a parameter nopegrevision which can be added to disable this behaviour. [YOCTO #6258] (Bitbake rev: ea26682b706f655a8e418f56bebe742e2d130a1d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Don't add None to ExpansionError varlistRichard Purdie2021-09-211-2/+3
| | | | | | | | | If a "None" value gets into the varlist, it doesn't display properly. Ensure we don't add one to have the exception display properly. (Bitbake rev: ee26e258888114143e66330c256b5bfe7d071c53) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch2: Fix quoting warningRichard Purdie2021-09-211-6/+6
| | | | | | | | | | | | Fix: lib/bb/tests/fetch.py:1288: DeprecationWarning: invalid escape sequence for several lines of the fetch tests. (Bitbake rev: 9d84fd557a3fcbae2cdd70b24e69325ad737a01e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Use os.rename instead of mvRichard Purdie2021-09-211-4/+4
| | | | | | | | | os.rename will overwrite the destination file if present so we can use this instead of the process call overhead. (Bitbake rev: b3cccaa6a896c41d8c9be5eebc327f726542d16b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Show a readable error for invalid multiconfig nameRichard Purdie2021-09-191-0/+2
| | | | | | | | | | | | | | If a multiconfig starts with a digit, users would see pages of errors as we use the multiconfig as a python function name prefix and python functions cannot start with a digit. We could avoid doing that but it is easier just to ask users to name multiconfigs not starting with digits. This tweak ensures the user sees an easier to understand error. (Bitbake rev: f9cddaeef35b2ea0dadf717101ed896f6b857abd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: enable python warnings at the first opportunityAlexander Kanavin2021-09-1912-0/+21
| | | | | | | | | | | We really do want to see those, as they tend to turn into hard errors eventually, as what happened with collections vs collections.abc in python 3.10. (Bitbake rev: bc43fbb86361a21dc2d5deb910810c5a77fdabe8) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/s3: allow to switch profile from environment variableValentin Danaila2021-09-171-0/+1
| | | | | | | | | | Make usage of Bitbake's s3 fetcher more flexible with different AWS profiles and switch between profiles as export of the AWS_PROFILE environment variable (Bitbake rev: 0f35dc4dfc829fe9c51c239d15567f86c5c14e58) Signed-off-by: Valentin Danaila <drlv85@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: correct deprecation warning in process.pyAlexander Kanavin2021-09-171-1/+1
| | | | | | | (Bitbake rev: aff52fe21a0b27f6302555c1e52a864550eb46ce) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: adjust parser error check for python 3.10 compatibilityAlexander Kanavin2021-09-171-1/+1
| | | | | | | | | | The change was introduced in https://github.com/python/cpython/commit/a698d52c3975c80b45b139b2f08402ec514dce75 (Bitbake rev: 8d3c6cbbe6ee734495713ae3b99c609527842506) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: do not import imp in layerindexlibAlexander Kanavin2021-09-171-1/+0
| | | | | | | | | The module is deprecated and unused. (Bitbake rev: 76c9030d6c91cd776a1aa732fb92b7cd4539b503) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fix regexp deprecation warningsAlexander Kanavin2021-09-171-3/+3
| | | | | | | | | | See here for details: https://docs.python.org/3/library/re.html (Bitbake rev: 660e6ad4abb77c6f3c1d48bd64777dd76c05d7e2) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: correct the collections vs collections.abc deprecationAlexander Kanavin2021-09-173-4/+6
| | | | | | | | | This becomes a hard error in python 3.10. (Bitbake rev: ae219e1f7460077f4492b31ac91cef4cf9b17277) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: drop old rules for python warningsAlexander Kanavin2021-09-171-7/+0
| | | | | | | | | | These no longer even work, and it's much better to just see all warnings and fix them as they happen. (Bitbake rev: 62d96ea2afbb0cfdb0d1f932dc96643e7bcd7f26) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Avoid races over mirror tarball creationRichard Purdie2021-09-171-2/+19
| | | | | | | | | | | | | | There is a potential race over the mirror tarballs where a partial git repo could be extracted causing fetcher failures if the tarball is being rewritten whilst another build accesses it. Create the mirror tarball atomically to avoid this. [YOCTO #14441] (Bitbake rev: 3250bc950c56bd7dd2114df26e5a8e13b04ceac8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue/knotty: Improve UI handling of setscene task countingRichard Purdie2021-09-173-5/+20
| | | | | | | | | | | | | | | | The recent fixes to merge setscene and normal task accounting in runqueue fixed some display issues but broke the task numbering of setscene tasks. Add new accounting methods to the stats structure specifically designed for setscene. This accounts for the fact that setscene tasks can rerun multiple times in the build. Then use the new data in the UI to correctly display the numbers the user wants to see to understand progress. (Bitbake rev: ed7e2da88bf4b7bfc7ebfc12b9bd6c0fb7d8c1aa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Improve error handlingRichard Purdie2021-09-161-3/+5
| | | | | | | | | | | | If bitbake-worker fails, return an error code showing that. Also make the thread cleanup code explict in a finally clause as it would otherwise hang. [YOCTO #14393] (Bitbake rev: 7e0af70fb53fb13f824ca954b8cc1dffee730233) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Enable ftpsDaniel Ammann2021-09-161-1/+1
| | | | | | | | | | The fetcher would fail with: Could not find a fetcher which supports the URL: ftps://... (Bitbake rev: 9e56710c7203b1ec6cbefa758c81b69b697fe1a4) Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake.conf: fix vars_from_file() callMartin Jansa2021-09-143-5/+5
| | | | | | | | | | The reference to this function was dropped from BBHandler.py in: https://git.openembedded.org/bitbake/commit/?id=aaa5292ef96ea27f505bc5c5a4b1eb4f497ed061 (Bitbake rev: b4fe8507079b6464287549f3eac1f1fc8d0d9be2) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Allow upstream for local hash equivalence serverJoshua Watt2021-09-111-1/+6
| | | | | | | | | | | | | | The hash equivalence server has had the option to support a read-only upstream server for some time now when launched as a standalone program, but there was no way to set the upstream when using a locally started server. Add a new variable called BB_HASHSERVE_UPSTREAM that can be used to specify an upstream server when a local hash equivalence server is used (e.g. BB_HASHSERVE is "auto") (Bitbake rev: 250fa17f1391ff1ee01ab9b51d2a4f9aa35c1d1e) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Clean up task stats handlingRichard Purdie2021-09-112-16/+8
| | | | | | | | | | | | | | | | | | | | | When we parallelised normal and setscene tasks, the task stats handling was left separate pending further thought. We had to remove handling of the setscene tasks from the UI in order to maintain consistent task numbering. Currently, "0 of 0" tasks can be shown as setscene tasks execute until the first normal task runs. The only use left for sq_stats is in the active task numbers which we can use the length of sq_ive for instead. We can therefore drop it and return stats in all cases. This removes the "0 of 0" task problem since the stats in all normal and setscene tasks matches. [YOCTO #14479] (Bitbake rev: eae6e947e37e18cded053814bd2a268b44fb25cd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Show error for no BBLAYERS in bblayers.confRichard Purdie2021-09-111-0/+3
| | | | | | | | | | | If there is no BBLAYERS set in bblayers.conf show a more helpful error and exit. [YOCTO #14340] (Bitbake rev: 97183e10faf9862b5d9489d6e2c27ac77c3b697d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Improve missing core layer error messageRichard Purdie2021-09-111-0/+2
| | | | | | | | | | | If the core layer is missing from bblayers.conf, the message the user sees is hard to understand. Improve it. [YOCTO #14340] (Bitbake rev: 5815a7258ebb8a989e0c6f5798853559d9413f02) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Add recursion guardRichard Purdie2021-09-111-0/+7
| | | | | | | | | | | Users sometimes put ${S} references in ${SRC_URI} without realising this can be problematic. Improve the error messages if they accidentally do. [YOCTO #11593] (Bitbake rev: 89e0b19ec0b245a6cd414088904c91808e8814ab) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Improve error display for handled exceptionsRichard Purdie2021-09-111-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We don't need tracebacks for BBHandledException. Reduces confusing output like: ERROR: /meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio ERROR: ExpansionError during parsing /meta/recipes-core/images/core-image-tiny-initramfs.bb Traceback (most recent call last): File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_qa', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_cpio', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in deps: > follow_chain(othertask, endtask, chain) chain.pop() File "/bitbake/lib/bb/build.py", line 1038, in follow_chain(task='do_image_complete', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']): if task in chain: > bb.fatal("Circular task dependencies as %s depends itself via the chain %s?!" % (task, " -> ".join(chain))) chain.append(task) File "/bitbake/lib/bb/__init__.py", line 165, in fatal: mainlogger.critical(''.join(args), extra=kwargs) > raise BBHandledException() to the real error: ERROR: /media/build1/poky/meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio (Bitbake rev: 551d4c0576a0a0c3406000029df9238b312f2263) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Catch and error upon circular task referencesRichard Purdie2021-09-111-0/+2
| | | | | | | | | | | | | | | | | | | | | | If there are circular task references, error on them rather than show a recursion error. A simple reproducer is: """ do_packageswu () { : } addtask do_packageswu after do_image_complete before do_image_qa """ into image_types.bbclass. There is code in runqueue to detect these but we never get that far with the current codebase. [YOCTO #13140] (Bitbake rev: 339d4d6be515a71311b81fb9e99742af0d8a5130) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Make ExpansionErrors more readableRichard Purdie2021-09-111-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This adds context to ExpansionError messages which show the variable chain for which expansion is being attempted. This should allow users to debug the issues more easily than the current message (the first line alone below). Example output from a SRC_URI which references ${S}: bb.data_smart.ExpansionError: Failure expanding variable PV, expression was 0.1+git${SRCPV} which triggered exception RecursionError: maximum recursion depth exceeded while calling a Python object The variable dependency chain for the failure is: PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> WORKDIR -> S -> SRC_URI -> SRCPV -> PV -> BP -> FILESPATH which is more useful that no output. We could truncate at repetition but I suspect this makes this clearer as it stands so there is little value in complicating the code. (Bitbake rev: 699634bec47964fa7ab18689dc23db6f0bc22fb3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Avoid duplicating logs in verbose modeRichard Purdie2021-09-111-0/+4
| | | | | | | | | With "bitbake -v", for task failures you'd see the log output twice. Avoid this by using the existing "did we print info" switch. (Bitbake rev: e2c1afda4cb8023ed4ffeb5dc5bee4f0055659a8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Don't include logs in error message if piping themRichard Purdie2021-09-111-0/+3
| | | | | | | | | | If the caller is piping the logs, they likely don't want them in the error exception as well. This removes duplicate output from the build output allowing the UI level controls on whether to show logs to work correctly. (Bitbake rev: fc58ad84a9deb2620ad90611684dad65dafedb11) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Handle SystemExit in python tasks correctlyRichard Purdie2021-09-111-1/+1
| | | | | | | | | | | | | If a python task fails with sys.exit(), we currently see no TaskFailed event. The high level code does detect the exit code and fail the task but it can leave the UI inconsistent with log output. Fix this be intercepting SystemExit explicitly. This makes python task failures consistent with shell task failures. (Bitbake rev: 9eee9fd4f2f96789ad2b037e74d561bdc1426856) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build: Match markup to real function nameRichard Purdie2021-09-111-2/+2
| | | | | | | | | The point of the injected text is to identify where the function comes from. Using the correct function name would therefore be better. (Bitbake rev: 30c6ff8551c235254ab90663ab88f66bb0c71edb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: parse_py: Drop deprecated function referenceRichard Purdie2021-09-101-3/+0
| | | | | | | | This is now used from bb.parse everywhere so drop this long deprecated reference. (Bitbake rev: aaa5292ef96ea27f505bc5c5a4b1eb4f497ed061) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Drop deprecated/unused functionRichard Purdie2021-09-101-50/+0
| | | | | | | | | | This class has long since been deprecated and is unused, drop it. I'd love to get rid of the rest of persist_data but it is still used by the fetcher, sadly. (Bitbake rev: 1c574aae8c8ec427f27ab0d68bac9e7483016f18) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual-fetching: S should be set to WORKDIR/git ↵Quentin Schulz2021-09-101-0/+7
| | | | | | | | | | | | | for git fetcher Document that S has to be set to "${WORKDIR}/git" in order for the recipe to work if the git fetcher is fetching the main source of code for the recipe. (Bitbake rev: c394f34437686c2f57a27f0c06d7aeb3268bfd41) Signed-off-by: Quentin Schulz <quentin.schulz@theobroma-systems.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix issues with multiconfig deferred task deadlock messagesRichard Purdie2021-09-081-0/+6
| | | | | | | | | | | | | | | | | | | | | In multiconfig builds with large numbers of identical tasks, builds were deadlocking after recent runqueue changes upon rebuilds where there was heavy sstate usage (i.e. on second builds after a first completed). The issue was that deferred tasks were being left indefinitely on the deferred list. The deadlock handler was then "breaking" things by failing tasks that had already succeeded, leading to the task being on both covered and not covered lists, giving a further error. The fix is to clean up the deferred task list when each setscene task completes. I'd previously been hoping to avoid iterating that list but it appears unavoidable. [YOCTO #14342] (Bitbake rev: ae24a0f2d2d8b4b5ec10efabd0e9362e560832ea) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Avoid deadlock avoidance task graph corruptionRichard Purdie2021-09-081-1/+2
| | | | | | | | | | If the deferred task deadlock avoidance code triggers, it could mark an executed task as failed which leads to "covered and not covered" error messages. Improve the logic so if the deadlock code is triggered, it doesn't cause the errors. (Bitbake rev: 51bdd6cb3bd9e2c02e261fb578bb945b86b82c75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: add and fix npm testsScott Weaver2021-09-071-2/+24
| | | | | | | | | | | | | | | | | | | | This adds one new test that verifies the use of a specific filename when defined in PREMIRRORS. bb.tests.fetch.NPMTest: - test_npm_premirrors_with_specified_filename While testing bz#13039, it was found that test_npm_registry_alternate fails with ENOTFOUND. This was corrected by using npmjs's public mirror. The change to fetch2 for bz#13039 highlighted an issue with the test_npm_premirrors test where the created file:// mirror was using the downloadfilename rather than the tarball that is defined by the npm url. (Bitbake rev: 5ba191a0407af9e652e3b86302dce3e952d6b587) Signed-off-by: Scott Weaver <weaverjs@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: tests/fetch: add downloadfilename testsScott Weaver2021-09-071-0/+21
| | | | | | | | | | | | | | | This adds three new tests which evaluate different use cases of the downloadfilename property. bb.tests.fetch.FetcherNetworkTest: - test_fetch_specify_downloadfilename - test_fetch_premirror_specify_downloadfilename_regex_uri - test_fetch_premirror_specify_downloadfilename_specific_uri (Bitbake rev: 61db3e96530d650e098436fd086f0182d32998f7) Signed-off-by: Scott Weaver <weaverjs@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fetch2: fix premirror URI when downloadfilename definedScott Weaver2021-09-071-1/+1
| | | | | | | | | | | | | When downloadfilename is defined in a recipe's SRC_URI and PREMIRRORS is also defined using the same URI, the downloadfilename is appended to the mirror URI and it should not be. [YOCTO #13039] (Bitbake rev: 8a3ff9f3eaf19d4258eb070c5dc230dface269b2) Signed-off-by: Scott Weaver <weaverjs@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: doc: bitbake-user-manual-execution: remove mention to long-gone ↵Quentin Schulz2021-09-061-3/+0
| | | | | | | | | | | | | | | BBHASHDEPS variable BBHASHDEPS was removed from bitbake in April 2014, with the following commit: 92526eadd09d "bitbake-worker: Drop BBHASH variables" so let's remove mentions from the docs. (Bitbake rev: 7816a8de70adc3806a3739384cf08b281b4ee401) Signed-off-by: Quentin Schulz <foss@0leil.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: replace "file name" by "filename"Michael Opdenacker2021-09-062-3/+3
| | | | | | | | | | | | There are many more instances of "filename" or "filenames" than of "file name" or "file names". The winner takes it all! (Bitbake rev: dcd115176f63256f10db0b24b563683c4bdf8f96) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: make localhost workMingli Yu2021-09-021-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After [1] introduced, the PR server doesn't work in docker when use below setting like before. PRSERV_HOST = "localhost:0" And it's because the localhost is resolved to an ipv6 address ::1 as the below bitbake-prserv shows. bitbake$ bitbake-prserv --start --host=localhost --port=42005 bitbake$ cat prserv.log Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile)) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon func() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address So add the extra logic to make the localhost resolved as expected to make the PR service work especially in docker. [1] 6a2b23e2 prserv: Replace XML RPC with modern asyncrpc implementation (Bitbake rev: 0a11696e0898c3c5108e6d7c5ad28da50e00ea66) Signed-off-by: Mingli Yu <mingli.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/process: Fix typos in exiting messageMartin Jansa2021-09-013-3/+3
| | | | | | | (Bitbake rev: 1ff1ea3880d293b14ce0fc65e3bc4c938d587a2f) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: handle PRSERV_HOST = "127.0.0.1:0" the same as "localhost:0"Martin Jansa2021-09-012-7/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * When using PRSERV_HOST = "localhost:0" inside Docker container (tested with ubuntu 20.04 and 21.04) the self.loop.run_until_complete never completed, so self.address wasn't ever assigned few lines bellow and then self.port = int(self.prserv.address.rsplit(':', 1)[1]) in lib/prserv/serv.py caused a bit ugly exception: bitbake@599696cd20aa:~/nodistro/honister$ bitbake -k pkgconfig-native Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake", line 35, in <module> sys.exit(bitbake_main(BitBakeConfigParameters(sys.argv), File "/OE/nodistro/honister/bitbake/lib/bb/main.py", line 385, in bitbake_main return ui_module.main(server_connection.connection, server_connection.events, File "/OE/nodistro/honister/bitbake/lib/bb/ui/knotty.py", line 397, in main params.updateToServer(server, os.environ.copy()) File "/OE/nodistro/honister/bitbake/lib/bb/cookerdata.py", line 75, in updateToServer raise Exception("Unable to update the server configuration with local parameters: %s" % error) Exception: Unable to update the server configuration with local parameters: Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 90, in runCommand result = command_method(self, commandline) File "/OE/nodistro/honister/bitbake/lib/bb/command.py", line 286, in updateConfig command.cooker.updateConfigOpts(options, environment, cmdline) File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 491, in updateConfigOpts self.reset() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 1717, in reset self.handlePRServ() File "/OE/nodistro/honister/bitbake/lib/bb/cooker.py", line 383, in handlePRServ self.prhost = prserv.serv.auto_start(self.data) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 318, in auto_start singleton.start() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 133, in start self.port = int(self.prserv.address.rsplit(':', 1)[1]) AttributeError: 'NoneType' object has no attribute 'rsplit' * the issue was caused by "localhost" being resolved as IPv6 address ::1 and then asyncio failing to bind it, the same is reproducible with hashserv, but hashserve at least shows nice error message: bitbake$ bitbake-hashserv -l DEBUG -b localhost:0 Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 59, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-hashserv", line 53, in main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address * or by bitbake-prserv in prserv.log: bitbake$ bitbake-prserv --start --host=localhost --port=42005 bitbake$ cat prserv.log Traceback (most recent call last): File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 55, in <module> ret = main() File "/OE/nodistro/honister/bitbake/bin/bitbake-prserv", line 46, in main ret=prserv.serv.start_daemon(options.dbfile, options.host, options.port,os.path.abspath(options.logfile), options.read_only) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 226, in start_daemon run_as_daemon(daemon_main, pidfile, os.path.abspath(logfile)) File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 202, in run_as_daemon func() File "/OE/nodistro/honister/bitbake/lib/prserv/serv.py", line 224, in daemon_main server.serve_forever() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 233, in serve_forever self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 42005, 0, 0): cannot assign requested address * while 127.0.0.1 works fine: bitbake$ bitbake-prserv --start --host=127.0.0.1 --port=42005 bitbake$ cat prserv.log DEBUG: Listening on ('127.0.0.1', 42005) 2021-08-26 22:28:05,828 Listening on ('127.0.0.1', 42005) DEBUG: Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' 2021-08-26 22:28:05,829 Opening PRServ database 'file:/OE/nodistro/honister/prserv.sqlite3' NOTE: Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 2021-08-26 22:28:05,831 Started PRServer with DBfile: /OE/nodistro/honister/prserv.sqlite3, Address: 127.0.0.1:42005, PID: 39 but 127.0.0.1:0 wasn't handled as "autostart" like localhost:0 is update is_local_special to allow that * /etc/hosts file generated by docker contails localhost for both IPv4 and IPv6: $ grep localhost /etc/hosts 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback even when ipv6 is disabled in dockerd as reported in: https://github.com/docker/for-linux/issues/250 * add a check for self.prserv.address to provide better error message: ERROR: Unable to start PR Server, exitting when something bad happens, but in this case you still need to read bitbake-cookerdaemon.log to see the actuall error, in this case: 90 22:30:39.008441 --- Starting bitbake server pid 90 at 2021-08-26 22:30:39.008419 --- 90 22:30:39.023734 Started bitbake server pid 90 90 22:30:39.024286 Entering server connection loop 90 22:30:39.024753 Accepting [<socket.socket fd=6, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=bitbake.sock>] ([]) 90 22:30:39.026314 Processing Client 90 22:30:39.026456 Connecting Client 90 22:30:39.027509 Running command ['setFeatures', [2]] 90 22:30:39.027757 Command Completed 90 22:30:39.028711 Running command ['updateConfig', {'abort': False, 'force': False, 'invalidate_stamp': None, 'dry_run': False, 'dump_signatures': [], 'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile': [], 'server_timeout': None, 'nosetscene': False, 'setsceneonly': False, 'skipsetscene': False, 'runall': None, 'runonly': None, 'writeeventlog': None, 'build_verbose_shell': False, 'build_verbose_stdout': False, 'default_loglevel': 20, 'debug_domains': {}}, {'DISTRO': '', 'PWD': '/OE/nodistro/honister', 'HOME': '/OE', 'MACHINE': 'qemux86', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO TCMODE TCLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS PARALLEL_MAKE GIT_PROXY_COMMAND GIT_PROXY_IGNORE SOCKS5_PASSWD SOCKS5_USER WEBOS_DISTRO_BUILD_ID PSEUDO_DISABLED PSEUDO_BUILD', 'PATH': '/OE/nodistro/honister/oe-core/scripts:/OE/nodistro/honister/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'LC_ALL': 'en_US.UTF-8', 'MACHINES': 'qemux86', 'HOSTNAME': '6a439759e3c6', 'TOPDIR': '/OE/nodistro/honister', 'LANG': 'en_US.UTF-8', 'TERM': 'xterm', 'SHLVL': '1', 'BITBAKE_HOME': '/OE', 'BUILDDIR': '/OE/nodistro/honister/BUILD', 'OLDPWD': '/OE/nodistro/honister/bitbake', '_': '/OE/nodistro/honister/bitbake/bin/bitbake'}, ['/OE/nodistro/honister/bitbake/bin/bitbake', '-k', 'zlib-native']] Process Process-1: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 255, in run self.start() File "/OE/nodistro/honister/bitbake/lib/bb/asyncrpc/serv.py", line 144, in start_tcp self.server = self.loop.run_until_complete(server_coro) File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete return future.result() File "/usr/lib/python3.8/asyncio/streams.py", line 94, in start_server return await loop.create_server(factory, host, port, **kwds) File "/usr/lib/python3.8/asyncio/base_events.py", line 1463, in create_server raise OSError(err.errno, 'error while attempting ' OSError: [Errno 99] error while attempting to bind on address ('::1', 0, 0, 0): cannot assign requested address 90 22:30:39.530037 Command Completed 90 22:30:39.530913 Processing Client 90 22:30:39.531023 Disconnecting Client 90 22:30:39.531638 No timeout, exiting. 90 22:30:39.632137 Exiting 90 22:30:39.637562 Original lockfile contents: ['90\n'] 90 22:30:39.638107 Exiting as we could obtain the lock (Bitbake rev: c2cdda0e5fc720c60d3b8537fc702cb118981bd2) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: providers: Use new override syntax when handling pn- "override"Peter Kjellerstedt2021-08-271-1/+1
| | | | | | | | | | Make versionVariableMatch() support pn-foo overrides using the new override syntax. (Bitbake rev: 653df4bc413d595d5611d5fa678e7f2e30bb7431) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: README: Fix typoRichard Purdie2021-08-261-1/+1
| | | | | | | | Thanks Peter for spotting. (Bitbake rev: 8c7a54ea7fdc721278380895d01868b96b330b90) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: README: Add note about test suite and new testsRichard Purdie2021-08-231-0/+8
| | | | | | | | Document that bitbake-selftest exists and that we appreciate test cases. (Bitbake rev: 52896ca1fabd22cce01b75cc6fe3412b1ec09b5b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-06-25 04:04:56 +0000