summaryrefslogtreecommitdiffstats
path: root/bitbake/lib
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: cooker: Clean up inotify idle handlerRichard Purdie2023-01-062-8/+3
| | | | | | | | | We no longer need to abstract the inotify callback handler, remove the abstraction and simplify/clean up the code. (Bitbake rev: af4ccab8acc49e91bf7647f209d69f4858618466) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Rework the parsing results submissionRichard Purdie2023-01-061-11/+15
| | | | | | | | | | | | | The loop submitting the parser results was not efficient on many core systems as may of the parsers could block for a long time waiting to send the results. While a result was pending, the code wouldn't parse further jobs. Change the code to parse further jobs even if the results can't be submitted and lower the result submission timeout to keep the parsers busy. (Bitbake rev: 9ece4a2e406509bd89dbce8dac80a02e600b0ecf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb: Update thread/process locks to use a timeoutRichard Purdie2023-01-055-31/+36
| | | | | | | | | | | | | | | | | | | | | The thread/process locks we use translate to futexes in Linux. If a process dies holding the lock, anything else trying to take the lock will hang indefinitely. An example would be the OOM killer taking out a parser process. To avoid bitbake processes just hanging indefinitely, add a timeout to our lock calls using a context manager. If we can't obtain the lock after waiting 5 minutes, hard exit out using os._exit(1). Use _exit() to avoid locking in any other places trying to write error messages to event handler queues (which also need locks). Whilst a bit harsh, this should mean we stop having lots of long running processes in cases where things are never going to work out and also avoids hanging builds on the autobuilder. (Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: Ensure that failure cases call finishAsyncComandRichard Purdie2023-01-021-2/+2
| | | | | | | | | | | | | | | | | There are a couple of failure cases that runCommands needs to better handle, ensuring finishAsyncCommand is called. This doesn't matter if the server is about to shut down but for memory resident bitbake and with threading enabled, correctness is more important this could could in theory lead to problems with the recent code changes. By using the idleFinish class, it ensures the current async command is terminated correctly and the various state pieces around commands finishing are called. This also makes the code more uniform matching the other exception handling code. (Bitbake rev: 367a83ed46c7fbcdd06579b6cc3a6e48a89ca7fb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Fix previous commit to use a string, not a generatorRichard Purdie2023-01-021-1/+1
| | | | | | | | | | The previous commit was injecting a generator object instead of a string, which happened to fix the issue but we should fix it properly! Thanks to Christopher Larson for spotting. (Bitbake rev: 36589f13c5babb3b2af82666c8a96338e6ea4eb9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Fix cache/reparsing issueRichard Purdie2023-01-011-2/+2
| | | | | | | | | | When setting the LAYERSERIES_COMPAT and LAYERSERIES_CORENAMES variables, we need to be deterministic. The random ordering from the sets was causing unexpected reparses. (Bitbake rev: 8c405c97430ac830837e25438e8795f6f7abbdaa) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Ping the server/cooker periodicallyRichard Purdie2022-12-311-1/+11
| | | | | | | | | | We're seeing failures where the UI hangs if the server disappears. Ping the cooker/server if we've not had any events in the last minute so we can check if it is still alive. (Bitbake rev: 6567ad6181f9e39812097f0154647e4b38238fdd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Run idle commands in a separate idle threadRichard Purdie2022-12-314-50/+97
| | | | | | | | | | | | | | | | | | | | | | | | | When bitbake is off running heavier "idle" commands, it doesn't service it's command socket which means stopping/interrupting it is hard. It also means we can't "ping" from the UI to know if it is still alive. For those reasons, split idle command execution into it's own thread. The commands are generally already self containted so this is easier than expected. We do have to be careful to only handle inotify poll() from a single thread at a time. It also means we always have to use a thread lock when sending events since both the idle thread and the command thread may generate log messages (and hence events). The patch depends on previous fixes to the builtins locking in event.py and the heartbeat enable/disable changes as well as other locking additions. We use a condition to signal from the idle thread when other sections of code can continue, thanks to Joshua Watt for the review and tweaks squashed into this patch. We do have some sync points where we need to ensure any currently executing commands have finished before we can start a new async command for example. (Bitbake rev: 67dd9a5e84811df8869a82da6a37a41ee8fe94e2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Add locking around idle functions accessesRichard Purdie2022-12-311-5/+14
| | | | | | | | | In preparation for adding splitting bitbakes work into two threads, add locking around the idle functions list accesses. (Bitbake rev: a9c63ce8932898b595fb7776cf5467d3c0afe4f7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Always use threadlockRichard Purdie2022-12-312-55/+35
| | | | | | | | | | | | | | | | With the move to a server idle thread, we always need threading. The existing accessor functions could end up turning this off! I was going to hold the lock whilst changing it, check if the value was already set, cache the result and also fix the event code to always release the lock with a try/finally. Instead, disable the existing functions and use a with: block to handle the lock, keeping things much simpler. (Bitbake rev: 645c9d3b50e55f69b222cc338373cdfd91d524ce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Improve idle loop exit codeRichard Purdie2022-12-313-17/+20
| | | | | | | | | | | | | | When idle handlers want to exit, returning "False" isn't very clear and also causes challenges with the ordering of the removing the idle handler and marking that no async command is running. Use a specific class to signal the exit condition allowing clearer code and allowing the async command to be cleared after the handler has been removed, reducing any opportunity for races. (Bitbake rev: 102e8d0d4c5c0dd8c7ba09ad26589deec77e4308) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure commands clean up any parser processesRichard Purdie2022-12-311-0/+3
| | | | | | | | | When finishing a command, we need to ensure any parsing processes that may have been started are cleaned up before we reset the cooker state. (Bitbake rev: 6569ab64bea35de21acc89053ba76e2828163f3f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: Tweak finishAsyncCommand ordering to avoid racesRichard Purdie2022-12-311-1/+1
| | | | | | (Bitbake rev: 89435442946767cfe58eedde363802add8f1ab29) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Improve exception and idle function loggingRichard Purdie2022-12-311-0/+4
| | | | | | | | | | | | | | | Currently if the idle functions loop suffers a traceback, it is silently dropped and there is no log message to say what happened. This change at least means the traceback is in the cooker log, making some debugging possible. Add some logging to show when handlers are added/removed to allow a better idea of what the server code is doing from the server log file. (Bitbake rev: 9cf3102dc36513124fe5ead2f1e448b51833b6ac) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker/cookerdata: Rework the way the datastores are resetRichard Purdie2022-12-312-18/+22
| | | | | | | | | | | | | | | | | | | | | | | | As far as I could tell, the current code could result in some strange situations where some data was set back to the original data store copy but the multiconfig data was not restored. There are also some changes made to the datastore which did not persist. The data store was also being reset at every client reset, which seems a little excessive if we can reset it to the original condition properly. Move the __depends -> __base_depends rename into databuilder along with the __bbclasstype change so these are saved in the original data. Tweak the databuilder code to be clearer and easier to follow on which copies are where, then save copies of all the mc datastores. Finally, drop the cache invalidation upon reset for the base config as we shouldn't need that now (oe-selftest -r tinfoil works with memory resident bitbake which was the original reproducer requiring that change). (Bitbake rev: 5f8b9b9c35b4ec0c8c539bf54ca85f068f4a5094) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: Add enable/disable heartbeat codeRichard Purdie2022-12-313-2/+15
| | | | | | | | | | | Currently heartbeat events are always generated by the server whilst it is active. Change this so they only appear when builds are running, which is when most code would expect to be executed. This removes a number of races around changes in the datastore which can happen outside of builds. (Bitbake rev: 8c36c90afc392980d999a981a924dc7d22e2766e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: knotty: Avoid looping with tracebacksRichard Purdie2022-12-311-3/+1
| | | | | | | | | | If there are events queued and there is an exception in the main loop of the UI code, it will print tracebacks on the console indefinitely. Avoid that by improving the loop exit conditions. (Bitbake rev: 2d0940b920a22b244f3ba6849c7cd019578386b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Drop reciever side counting for SiggenRecipeInfoRichard Purdie2022-12-301-7/+8
| | | | | | | | | | Joshua Watt pointed out maintaining the counting on both sides of the connection isn't needed. Remove the receiver side counting and simplify the code, also allowing errors if the counts do go out of sync. (Bitbake rev: aeacfd391903fe68ae600470fc2bbad0502d401e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tinfoil: Ensure CommandExit is handledRichard Purdie2022-12-291-1/+1
| | | | | | | | | | By inspection, tinfoil handles two of the three command exit cases but one is missing. Add the CommandExit case in case this is the cause of one of our recipetool/devtool hangs. Regardless, the fix is necessary. (Bitbake rev: eadddd94835b6b6a8517dfed3d29e6dbb2d35988) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Ensure cache is cleared for partial resetsRichard Purdie2022-12-291-0/+3
| | | | | | | | | | We're still seeing occasional SiggenRecipeInfo coherency issues, add some further reset points into the parsing code to ensure the cache is cleared before reparsing. (Bitbake rev: 26ed783caf11dc9ebf53d3790681eb44c0c360f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: event: builtins fix for 'd' deletionRichard Purdie2022-12-294-26/+17
| | | | | | | | | | | | | | | | | | | I've been seeing event handlers where 'd' seems to disappear half way through event handler execution. This is problematic when multiple threads are active since this code assumes single threading. The easiest fix is to change the handler function calls to contain d as a parameter as we do elsewhere for other functions. This will break any non-text handlers but I was only able to spot one of those in runqueue. It will also break handlers than call functions that assume 'd' is in the global namespace but those failures should be obvious and we can fix those to pass d around. This solution avoids manipulating builtins which was always a horrible thing to do anyway and solves the issue without needing locking, thankfully. (Bitbake rev: 1e12f0a4b592dacd006d370ec29cd71d2a44312e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache/siggen: Fix cache issues with signature handlingRichard Purdie2022-12-292-1/+3
| | | | | | | | | | | | | There is a bug in the current cache code where the restored data structures were "inverted" leading to some very weird issues, not sure how anything worked like that, this patch fixes it. Also fix some issues with multiconfig cache ordering problems by resetting the stream counters when appropriate. (Bitbake rev: cd06beb948eff5eaf2d474f5b127d51a61b0b2ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Add bitbake.sock race handlingRichard Purdie2022-12-211-1/+11
| | | | | | | | | | | | | | | | We've seen cases where the bitbake.sock file appears to disappear but the server continues to hold bitbake.lock. The most likely explaination is that some previous build directory was moved out the way, a server there kept running, eventually exited and removed the sock file from the wrong directory. To guard against this, save the inode information for the sock file and check it before deleting the file. The new code isn't entirely race free but should guard against what is a rare but annoying potential issue. (Bitbake rev: b02ebbffdae27e564450446bf84c4e98d094ee4a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Minor code improvementRichard Purdie2022-12-211-3/+3
| | | | | | | | | Tweak the code to remove duplication and only set if the attribute isn't already there to avoid overwriting. (Bitbake rev: 513e6c4e9233e0d0bc31e1169077fdbf9aaf4ec3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Allow compression of the data in SiggenRecipeInfoRichard Purdie2022-12-212-0/+76
| | | | | | | | | | | | | | | | | | | | | | | | | The data in SiggenRecipeInfo is large and has a lot of duplication. The size causes a few problems, impacting: - bitbake's overall memory usage - the amount of data sent over IPC between parsing processes and the server - the size of the cache files on disk - the size of "sigdata" hash information files on disk The data consists of strings (some large) or frozenset lists of variables. To reduce the impact we can: a) deplicate the data b) pass references to the object on the second usage (e.g. over IPC or saving into pickle). This patch does this for SiggenRecipeInfo mostly behind the scenes but we do need a couple of reset points so that streamed data is written correctly on the second usage. (Bitbake rev: 9a2b13af483c20763d6559a823310954884f6ab1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: command: Add ping commandRichard Purdie2022-12-211-0/+6
| | | | | | | | Add a simple ping command so the UI can check the server is still there. (Bitbake rev: fd3359de0b9f18fac187a629df203be0b2c87545) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Clarify the meaning of namespaceMarek Vasut2022-12-211-1/+2
| | | | | | | | | | | | | | Namespace in this context means a branch, a tag, etc., clarify it in the description. Also, fix a typo "a any", replace with plain "any". This patch is based of feedback on new applied patch d32e5b0e ("fetch2/git: Prevent git fetcher from fetching gitlab repository metadata") (Bitbake rev: b4999425c812b25cb359d5163d11e3c1b030dc28) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Convert to argparseJoshua Watt2022-12-171-171/+175
| | | | | | | | | | | | | | | | Converts the main bitbake program to use argparse instead of the deprecated optparse. The resulting Namespace returned by argparse should be equivalent to the one returned from optparse; the only major difference is that the positional "target" arguments are consumed by argparse and returned as the "targets" property instead of an additional argument from parse_args(). (Bitbake rev: bb2ea00274a594b7cc87a7cb0b165e9b28f6f3f4) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: log odd unlink events with bitbake.sockFrank de Brabander2022-12-171-2/+3
| | | | | | | | | | | | Log when the socket file already exists and is removed before recreating a new socket. Log when unlinking the socket file failed. (Bitbake rev: cfd7c9899f988bab6d9fe7bbfbdb60603fb5ed34) Signed-off-by: Frank de Brabander <debrabander@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser/data: Add vardepsexclude support to module dependency codeRichard Purdie2022-12-172-0/+7
| | | | | | | | | | | We need to be able to exclude dependencies from the python module dependency code. Add support for the vardepexclude flag for these. It only works from the configuration namespace rather than per recipe for efficiency. (Bitbake rev: 1aa672b01037fda4ca82f2c7e394783287c09ecd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: ast/data/codeparser: Add dependencies from python module functionsRichard Purdie2022-12-174-16/+68
| | | | | | | | | | | | | | | | | | | | Moving code into python modules is a very effective way to reduce parsing time and overhead in recipes. The downside has always been that any dependency information on which variables those functions access is lost and the hashes can therefore become less reliable. This patch adds parsing of the imported module functions and that dependency information is them injected back into the hash dependency information. Intermodule function references are resolved to the full function call names in our module namespace to ensure interfunction dependencies are correctly handled too. (Bitbake rev: 605c478ce14cdc3c02d6ef6d57146a76d436a83c) (Bitbake rev: 91441e157e495b02db44e19e836afad366ee8924) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Clarify which fn is meantRichard Purdie2022-12-171-38/+39
| | | | | | | | | | | | | | | "fn" can mean different things in bitbake, we added support for class extensions and then mutlticonfigs by extending it. In siggen, it generally means that mc is prefixed to it and that it is a virtual filename. Replace "fn" with "mcfn" in the code to make this clearer as if I'm getting confused, everyone else likely is as well. "mcfn" is sometimes referred to as taskfn as well but mcfn is probably the easiest to understand as the taskname isn't included. (Bitbake rev: e1c1139ab90f8da1b5036db11d943daefbe87859) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/siggen: Rework stamps functionsRichard Purdie2022-12-174-124/+109
| | | | | | | | | | | | | | | The current method of passing either a task's datastore, or dataCaches and a filename into the stamp functions is rather horrible. Due to the different contexts, fixing this is hard but we do control the bitbake side of the API usage so we can migrate those to use other functions and then only support a datastore in the public bb.build API which is only called from task context in OE-Core. (Bitbake rev: c79ecec580e4c2a141ae483ec0f6448f70593dcf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Add dummy dataCaches from task context/datastoreRichard Purdie2022-12-172-37/+42
| | | | | | | | | | | | | | | One of the challenges in maintaining the code is that it sometimes uses a datacaches structure and sometimes a datastore. Rather than continue the current dual API madness, have the worker contexts create a dummy datacaches structure with the entries we need. Whilst this does need to be kept in sync with the real structure, that doesn't change and this allows the code to be simplified. With this new approach, we can unify the stamps dependency code again. (Bitbake rev: c6d325fc9b53e9d588ab273ee3c2a99a70fba42c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: siggen/runqueue: Switch to using RECIPE_SIGGEN_INFO ↵Richard Purdie2022-12-173-33/+37
| | | | | | | | | | | | | | | | | | | feature for signature dumping Now that we have cache support for the taskdep/gendep/lookupcache data, we can switch to use that cooker feature and skip the secondary reparse to write the sig files. This does make the initial parse longer but means the secondary one isn't needed. At present parsing with the larger cache isn't optimal but we have plans in place which will make this faster than the current reparse code being removed here. (Bitbake rev: 5951b5b56449855bc2a30146af65eb287a35fcef) (Bitbake rev: 1252e5bce51ae912ecff9dcc354a371786ff2c72) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Directly store datacaches referenceRichard Purdie2022-12-172-0/+5
| | | | | | | | | | | It is becomming clear the siggen needs access to our cache data but we can't always obtain it in the contexts we need to. Add it directly, meaning over time we should be able to simplify the APIs and stop convoluting new ones! (Bitbake rev: 6b213590ed0e77683cf7fbce6bbe9605ddecf3d3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main/process: Add extra sockname debuggingRichard Purdie2022-12-132-4/+6
| | | | | | | | | | | | | | We're struggling to understand how bitbake.sock can sometimes disappear in live builds when we can't see where it could have been deleted. This causes connection failures to the server and failed builds. Add some extra debugging around the server log and client retry log messages to give more information for the next time this issue occurs. (Bitbake rev: 376a516dc8c96727fd042ada65f803013601ee2d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Bump minimum python version requirement to 3.8Richard Purdie2022-12-121-2/+2
| | | | | | | | | | | Most of our older distros with python older than 3.8 already need buildtools tarballs apart from Ubuntu 18.04. 3.8 allows us to fix a few things, tidy code in places and is widely available or can be obtained with buildtools. Therefore update our minimum version to 3.8. (Bitbake rev: 744310f360d2288ac2ef07745abc86852126b5b9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: build/siggen/runqueue: Drop do_setscene referencesRichard Purdie2022-12-113-5/+5
| | | | | | | | | | do_setscene was from a different era before our modern setscene per task code. It hasn't been used for years so remove some old obsolete references to it. (Bitbake rev: ef72282298f7c4db74383c23bb0251dd06d3c6d3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Drop non-multiconfig aware siggen supportRichard Purdie2022-12-113-55/+4
| | | | | | | | | All siggens in common use should now support multiconfig, drop the compatibility code. (Bitbake rev: b36545b4df6d935ed312ff407d4e0474c3ed8d1a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget.py: correctly match versioned directoriesAlexander Kanavin2022-12-115-2/+84
| | | | | | | | | | | | | | | | | | | | | | | | | | | When obtaining latest upstream versions, the code needs to check if the existing tarball is in a versioned directory (e.g. component-name/x.y/component-name-x.y.z.tar.gz) and if it is, it needs to first obtain the list of all such versioned directories and then check all of them by going one step up in the directory hierarchy. Existing code was returning a correct match when the component name did not have numbers, e.g. a check on 'source/epiphany/43/' would return 43, but was stopping too soon when the component name itself had numbers ('source/libxml2/2.10/' would return libxml2). This change ensures the last match is taken instead of the first. Also, adjust the fetcher tests to check that versioned directories are correctly traversed in this case (e.g. the step to go one level up is taken and a new tarball is discovered in a different versioned directory). (Bitbake rev: b6601be22c6d776327acdcd1fa931400f41ac786) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/git: Prevent git fetcher from fetching gitlab repository ↵Marek Vasut2022-12-091-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | metadata The bitbake git fetcher currently fetches 'refs/*:refs/*', i.e. every single object in the remote repository. This works poorly with gitlab and github, which use the remote git repository to track its metadata like merge requests, CI pipelines and such. Specifically, gitlab generates refs/merge-requests/*, refs/pipelines/* and refs/keep-around/* and they all contain massive amount of data that are useless for the bitbake build purposes. The amount of useless data can in fact be so massive (e.g. with FDO mesa.git repository) that some proxies may outright terminate the 'git fetch' connection, and make it appear as if bitbake got stuck on 'git fetch' with no output. To avoid fetching all these useless metadata, tweak the git fetcher such that it only fetches refs/heads/* and refs/tags/* . Avoid using negative refspecs as those are only available in new git versions. Per feedback on the ML, Gerrit may push commits outsides of branches or tags during CI runs, which currently works with the 'nobranch=1' fetcher parameter. To retain this functionality, keep fetching everything in case the 'nobranch=1' is present. This still avoids fetching massive amount of data in the common case, since 'nobranch=1' is rare. Update 'nobranch' documentation. Reviewed-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> (Bitbake rev: d32e5b0ec2ab85ffad7e56ac5b3160860b732556) Signed-off-by: Marek Vasut <marex@denx.de> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: handle username/password in uriKasper Revsbech2022-12-091-1/+2
| | | | | | | | | | | | | | | | | In checkstatus() opener.open() is used to check if an artifact is available. The check fails if the uri contains username password in the format: "username:password@hostname..". Moreover, the checkstatus function already uses the username from the "ud" object to craft a header, is username and password is provided. This fix ensure the uri in the Requests object used does not contain username as password. (Bitbake rev: 88350002d45e0aa85ecd5356da2c8d71e450641e) Signed-off-by: Kasper Revsbech <kasper.revsbech.ext@siemensgamesa.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main/server: Add lockfile debugging upon server retryRichard Purdie2022-12-092-25/+37
| | | | | | | | | | | | We keep seeing server issues where the lockfile is present but we can't connect to it. Reuse the lockfile debugging code from the server to dump better information to the console from the client side when we run into this issue. Whilst not pretty, this might give us a chance of being able to debug the problems further. (Bitbake rev: 22685460b5ecb1aeb4ff3436088ecdacb43044d7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main: Add timestamp to server retry messagesRichard Purdie2022-12-091-3/+7
| | | | | | | | | | | We have timestamps in the server logs but we don't know which UI messages correlate to them. Add some timestamps to the messages which doesn't make them pretty but which might make it possible to debug problems. (Bitbake rev: a1a86f8c311cb1fc4f5562f1c77f82aa95141eee) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cookerdata: Ensure layers use LAYERSERIES_COMPAT fairlyRichard Purdie2022-12-081-2/+27
| | | | | | | | | | | | | | Some layers think they're going to be 'clever' and copy the values from another layer, e.g. using ${LAYERSERIES_COMPAT_core}. The whole point of this mechanism is to make it clear which releases a layer supports and show when a layer master branch is bitrotting and is unmaintained. Therefore add some code to avoid people doing this. I wish we didn't have to but... (Bitbake rev: 6709aedccbb2e7ddbb1b2e7e4893481a7b536436) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: parse: Add support for addpylib conf file directive and ↵Richard Purdie2022-12-085-8/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | BB_GLOBAL_PYMODULES For many years OE-Core has injected it's own python modules into the python namespace using an immediate expansion of a variable in base.bbclass. It also added all entries from BBPATH to sys.path. We really need this to become a first class citizen of the langauge, this new addpylib directive allows that. Usage is of the form: addpylib <directory> <namespace> The namespace is imported and if there is an attribute BBIMPORT, that list of names is iterated and imported too. This mirrors what OE-Core has done for a long time with one difference in implmentation, sys.path is only appended to. This means later imported namespaces can't overwrite an earlier one and can't overwrite the main python module space. In practice we've never done that and it isn't something we should encourage or support. The new directive is only applied for .conf files and not other filetypes as it only makes sense in that context. It is also only allowed in the "base configuration" context of cookerdata since adding it at the recipe level wouldn't work properly due to the way it changes the global namespace. At the same time, move the list of modules to place in the global namespace into a BB_GLOBAL_PYMODULES variable. It is intended that only the core layer should touch this and it is meant to be a very small list, usually os and sys. BB_GLOBAL_PYMODULES is expected to be set before the first addpylib directive. Layers adding a lib directory will now need to use this directive as BBPATH is not going to be added automatically by OE-Core in future. The directives are immediate operations so it does make modules available sooner than the current OE-Core approach. The new code appends to sys.path rather than prepends as core did, as overwriting python standard library modules would be a bad idea and naturally encouraging people to collaborate around our own core modules is desireable. (Bitbake rev: afb8478d3853f6edf3669b93588314627d617d6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bin/utils: Ensure locale en_US.UTF-8 is available on the systemFrank de Brabander2022-12-081-0/+16
| | | | | | | | | | | | | | Get rid of the duplicate code and add extra check that the locale en_US.UTF-8 is available on the system. This new helper method is now located right above the method filter_environment() which sets LC_ALL environment variable to 'en_US.UTF-8'. [YOCTO #10165] (Bitbake rev: a4ce040a6fd540a1cac52f808f909f9fcf8c961c) Signed-off-by: Frank de Brabander <debrabander@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Start sync thread a little earlierJoshua Watt2022-12-081-6/+7
| | | | | | | | | | | Starts the sync thread slightly earlier to give it some extra time to dump out the caches while the main process tears down the parsing processes (Bitbake rev: 105f2897b0618713b036fc0f7a6e0f3e78d1707a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Use event to terminate parser threadsJoshua Watt2022-12-081-12/+3
| | | | | | | | | | Uses an event to terminate the parser threads instead of a queue. This is not only simpler, but saves about 500ms on shutdown time (Bitbake rev: 2aed34e1d4bf24bba6263f168ff31b55b5fbe982) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-07-02 02:12:00 +0000