summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: tests/fetch: Resolve fetch error in bitbake-selftestArmin Kuster2019-09-091-2/+2
| | | | | | | | | | | | | | | | | | | | | | | FAIL: test_wget_latest_versionstring (bb.tests.fetch.FetchLatestVersionTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest/build/bitbake/lib/bb/tests/fetch.py", line 1229, in test_wget_latest_versionstring self.assertTrue(verstring, msg="Could not find upstream version for %s" % k[0]) AssertionError: '' is not true : Could not find upstream version for db [YOCTO #13496] The Oracle UPSTREAM_CHECK_URI used changed and does not work with logic in wget. Update UPSTREAM_CHECK_URI and UPSTREAM_CHECK_REGEX to match the ones used in the recipe. Also change the version being checked. (Bitbake rev: 8a58c3c64240c6ab14858d18e6b89febdb315311) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: show warning when renaming the archive with bad checksum failedMartin Jansa2019-09-061-1/+2
| | | | | | | | | | | | | | | | | | | | | * noticed on read-only sshfs premirror * it was showing the warning about renaming the file: WARNING: laser-geometry-1.6.4-r0 do_fetch: Renaming /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 and then failed because of movefile() issue with python3 (fixed in previous commit): ERROR: laser-geometry-1.6.4-r0 do_fetch: Error executing a python function in exec_python_func() autogenerated: with movefile() fixed, it let do_fetch continue and re-fetch locally with the right checksum, but still the renamed file didn't exist, because of movefile failure - add another warning when the movefile fails - for whatever reason - unfortunately movefile prints error messages with just print() so the real error is hidden only in log.do_fetch in this case: movefile: Failed to move /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz to /jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441 [Errno 30] Read-only file system: '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz' -> '/jenkins/mjansa/sshfs/webos-ose-thud/downloads/laser_geometry-1.6.4.tar.gz_bad-checksum_1ee7479b8c5914b4ffae996945121441' (Bitbake rev: d36438759344caa447d9a0bf30749a0aa31d1fba) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Fix movefile() exception handling with python3Martin Jansa2019-09-061-1/+1
| | | | | | | | | | | | | | | | | | | | * with python3 this fails with: File: 'bitbake/lib/bb/utils.py', lineno: 799, function: movefile 0795: try: 0796: os.rename(src, destpath) 0797: renamefailed = 0 0798: except Exception as e: *** 0799: if e[0] != errno.EXDEV: 0800: # Some random error. 0801: print("movefile: Failed to move", src, "to", dest, e) 0802: return None 0803: # Invalid cross-device-link 'bind' mounted or actually Cross-Device Exception: TypeError: 'OSError' object is not subscriptable (Bitbake rev: 9f92322fa8d6f1a68c0c3f4984afdf65126b51dc) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Add need_update method to determine when we are going to a ↵Mark Hatle2019-06-182-0/+39
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | new SRCREV If the system had previously fetched a source repository for use by gitsm, and then the SRCREV was updated and the new commit already existed, the system would not re-evaluate the submodules and update them accordingly. The cause of this issue was that need_update was being used, unmodified, from the base git fetcher. It did not have any knowledge, nor did it care if we were moving commits and needed to re-evaluate what was happening due to this switch. To fix the issue, during the download process we add all processed (by gitsm) srcrevs to the git config file, as bitbake.srcrev. This allows us to use a new need_update function that not only checks if the git commit is present, but if we have previously processed this commit to ensure all of the submodule components are also present. This approach is used, instead of iterating over the submodules in need_update to avoid a potential race condition that has affected us in the past. The need_update is called only with the parent locking. Any time we need to dive into the submodules, we need to lock, and unlock them, at each stage. This opens the possibility of errors in either the code, or unintended race conditions with rm_work. This issue was discovered by William A. Kennington III <wak@google.com>. The included test case was also written by him, and included unmodified. (Bitbake rev: 4ce92f43eeac6a4bfd06e8567fa6891614b5b3b0) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: Fix a bug where the wrong path was used for the submodule initMark Hatle2019-06-182-5/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Because we are trying to avoid network activity and use our own fetcher, the system emulates the behavior of 'git submodule init'. git submodule init uses the .gitmodules file, where typically the module name and path are the same. However, in this case the module name and path (in the tree) were different. i.e.: [submodule "edgelet/hsm-sys/azure-iot-hsm-c/deps/azure-c-shared-utility"] path = edgelet/hsm-sys/azure-iot-hsm-c/deps/c-shared url = https://github.com/Azure/azure-c-shared-utility.git Previously the code assumed the 'path' was both the checkout location under .git/modules, as well as the path to extract the components. This proved to be incorrect as the .git/modules path needs to match the submodule 'name'. This causes the components that were fetched to be initialized in the wrong location, which later caused the 'git submodule update' process to skip not properly initialized modules. A test case was added for this specific case to ensure a regression does not appear in the future. (Bitbake rev: ffd7ed530a17d22df576d986ac78428a6979e79c) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: fix version comparison when one of the versions ends in .Alexander Kanavin2019-06-152-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | Previously, this would happen: ====================================================================== ERROR: test_vercmpstring (bb.tests.utils.VerCmpString) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/alexander/development/poky/bitbake/lib/bb/tests/utils.py", line 45, in test_vercmpstring result = bb.utils.vercmp_string('1.', '1.1') File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 143, in vercmp_string return vercmp(ta, tb) File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 135, in vercmp r = vercmp_part(va, vb) File "/home/alexander/development/poky/bitbake/lib/bb/utils.py", line 124, in vercmp_part elif ca < cb: TypeError: '<' not supported between instances of 'NoneType' and 'int' ---------------------------------------------------------------------- (Bitbake rev: bd953d56d007a8bfa5ecb6e753da4abfb035f9f2) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: cooker: Rename __depends in all multiconfigsJoshua Watt2019-06-111-2/+3
| | | | | | | | | | | | | | | | | | | The renaming of the __depends variable to __base_depends and file watches needs to occurs for all multiconfigs, not just the base config. Failing to do this for all multiconfigs will result in a huge increase in the size of the parsing cache (about 5x for a single mulitconfig) because all multiconfig caches will still depend on the base config files. This will also seen a similar jump in the amount of time required to load the parsing cache from memory, both because the cache is larger and because of explosion of additional existence checks that must be done for the base files. [YOCTO #13359] (Bitbake rev: da5d1560d4ad1c735f6166a5d9ce94f36c94186b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: Fix Thud Bitbake release metadataDavid Reyna2019-06-102-2/+2
| | | | | | | | | | | Fix the Bitbake version ID for Thud (1.40). [YOCTO #13356] (Bitbake rev: f8a23ad4f8f0336470a5d5b074b7a2d02ec94631) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm: The fetcher did not process some recursive submodules properly.Mark Hatle2019-03-242-1/+23
| | | | | | | | | | | | | | | | | | Move the submodule processing outside of the if statement to avoid any optimizations that may happen. Update the test cases to include the additional case, and split the other test cases into individual tests to make it easier to figure out what the failure may be. (Bitbake rev: 7c1eb51d1e8a4c5f39bf9dddf05fb0b3598da72b) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 0ec98c01ae50f95c9c74acf53013ac59e0e72b08) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsmy.py: Fix unpack of submodules of submodulesMark Hatle2019-03-242-3/+32
| | | | | | | | | | | | | | | | | | If the submodule is in a subdirectory, it needs to have that structure preserved. This means the unpack path needs to be in the 'dirname' of the final path -- since the unpack directory name is specified in the URI. Additional specific test cases were added to ensure this is working properly based on two recent error reports. (Bitbake rev: acca06d060e49b2441562b4dc94416af9ab8187e) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 8c8ecec2a722bc2885e2648d41ac8df07bdf660d) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Fix relative URLsMark Hatle2019-03-242-7/+8
| | | | | | | | | | | | | | | Prior code happened to work with relative URLs, when the code was recently restructured it caused all relative urls to no longer work. Restore the prior code flow for relative support and better comment why that code is there. (Bitbake rev: 39afa6ead1d72813b2a294f065d759bad08fb53d) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 14bb383af0ca98e0e04ec217e537b6a899f3658e) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Refactor the functions and simplify the classMark Hatle2019-03-242-146/+98
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The update_submodules and unpack_submodules functions were nearly indentical, so we made a common function where the different behavior could be passed in by the download and unpack users. The new function is process_submodules. Moved the parse_gitmodules function under the new process_submodules, since there are no external callers. Refactor the file relative path processing to the URL translation code. We also add a warning to the translation if a relative ssh URL has been detected. Since this can cause a problem. In the case of a relative URL that does not work after being translated, it should be possible to use the MIRROR functions to manual translate the generated relative URL into one that works properly. Remove 'git config' processing on download contents. It turns out this is not necessary since all of the later components work using the git fetcher. Limit the 'git submodule update' call to only when unpacking a non-bare repository. Submodules are always loaded as bare, so this prevents intermediate unpacks from being attempted. Finally, the test cases were updated and the new commit ids in the test repository were updates as well. (Bitbake rev: 8b02eb8e5d8e3a09e19ed96a4ccdf5f755e97a74) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 610dbee5634677f5055e2b36a3043cd197fb8c51) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Rework the shallow fetcher and test caseMark Hatle2019-03-242-63/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A custom shallow submodule is no longer necessary, as the regular git fetcher is used and shallow handling works with the same code. The only general difference between the regular change is simply declaring a clone as shallow, when appropriate. This also removes a potential race condition in copying repositories vs cloning them. The gitsm shallow fetcher test was revised to verify that the submodule is shallow cloned along with the primary repository. The first step of this change was to be sure to clean the gitsubmodule download directory, as was previously done with the may gitsource directory. Additional test components were added to verify commit counts, and an obsolete (and likely incorrect) test for the .git/modules directory to be empty was also removed. (Bitbake rev: 85dc1c65b661f9712ae98587d4d0d868146c8cff) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f9cc4684dcf4281acc557cda8cb35602354ac3d6) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: revise unpackMark Hatle2019-03-241-38/+78
| | | | | | | | | | | | | | | | | | | | | Greatly simply the unpack rule by copying the general functionality of update_submodules as unpack_submodules. This will recursively construct a set of urls and unpack them using the standard system behaviors. The overall code may be slightly bigger, but this ensures that all of the standard locks are inplace, ensuring the code doesn't change out from under the unpack function. (This could have happened before due to using 'cp' instead of further unpacks on submodules. This may still happen in shallow clones.) (Bitbake rev: 02ce6783bd3cfc117c77ca0380a87b0e1c60d8db) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 7d7ee630f1c65e7dd234f945edf5e3b3bcb0fc30) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Optimize code and attempt to resolve locking issueMark Hatle2019-03-241-14/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | It was reported that a race condition on a shared download directory could occur with the gitsm fetcher, the result happened with a call to git config that occured within the update_submodules. Since the fetch is locked by the upper level, it was probably the prior need_update(...) function causing this because of some old code. The gitsm class inherits the git class. The need_update was overridding the version in gitsm, so that it forceably checked the submodules. It's clear we can optimize the code by only updating if the primary repository needs updating. Since we don't care if the submodule repository has changed because if the primary hasn't, references to the submodule won't change. (Bitbake rev: 4660933f83e528766d71eab662cc79dcf17b4be7) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 346338667edca1f58ace769ad417548da2b8d981) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/fetch.py: Add alternative gitsm test caseMark Hatle2019-03-241-1/+5
| | | | | | | | | | | | | | | In order to test the ssh processing in gitsm, we add an alternative testcase that can be downloaded from git.yoctoproject.org. However, this test case requries (read) access, via ssh, to git.yoctoproject.org. (Bitbake rev: 54802438f2f2d73aba3e7fb8d6cce45aa7cffe77) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit c8554cdc1287620fe8e8960561e614567879a010) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Add support for alternative URL formats from submodule filesMark Hatle2019-03-241-2/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | The following appear to be the git supported formats: proto://user:pass@host/path (URI format) user@host:path (SSH format) /path or ./path or ../path (local file format) We adjust the parsing to find out if we have a URI format or not. When we are NOT in URI format, we do our best to determine SSH or file format by looking for a ':' in the overall string. If we find a ':' we assume SSH format and adjust accordingly. Note, in SSH format we simply replace the ':' with a '/' when constructing the URL. However, if the original path was ":/...", we don't want '//' so we deal with this corner case as well. (Bitbake rev: a21b2598531d52123933a0ac1c4deeecfdd1697d) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit dcac05e7dc6b0c5f8e63d36ad105b7eab41f0016) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: gitsm.py: Fix when a submodule is defined, but not initializedMark Hatle2019-03-241-4/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | It is possible for a submodule to be defined in the .gitmodules file, but never initialized in the repository itself. This shows itself when searching for the defined module hash you will get back a empty value. Similarly we need to identify and skip defined but not initialized submodules during the unpack stages as well. Thanks to raphael.lisicki@siemens.com for their help is figuring out how to resolve this issue. Additionally a problem was found where, while unlikely, it may be possible for the wrong revision to have been searched using ls-tree. This has been resolved in the update_submodules function by keeping the correct revision along with the submodule path. (Bitbake rev: e9c965bd816c8e4e5cc8e45db8e0dab597ce5a6f) Signed-off-by: Mark Hatle <mark.hatle@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 49e1ff920143dcd4e7f73933d7ab8a84b8ffa1a3) Signed-off-by: Scott Murray <scott.murray@konsulko.com> Signed-off-by: Armin Kuster <akuster@mvista.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Tweak multiconfig dependency resolutionRichard Purdie2019-02-251-29/+24
| | | | | | | | | | | | | | | | | | | There were a couple of problems with the multiconfig dependency resolution: - the "if mc" condition triggering this code wasn't correct, it needs to be "if more than one multiconfig" configured - after adding providers we need to call add_unresolved again and rebuild mcdeps within the "while new" loop By fixing these issues we allow various other combinations of multiconfig builds to work which previously didn't. [YOCTO #13090] [YOCTO #13130] (Bitbake rev: b59cb2bc63940b9ebd8288de7ca4b1d9e96e026c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Fix multiconfig corner caseRichard Purdie2019-02-251-0/+2
| | | | | | | | | | | | | There was already a fix to ignore some multiconfig dependencies but its 'opposite' case wasn't covered. Cover that combination to so as to avoid tracebacks in multiconfig builds. [YOCTO #13090] [YOCTO #13130] (Bitbake rev: 3d4e557a539a8ef4a3b7a1e36fc99b118c01cbf4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Filter out multiconfig dependencies from BB_TASKDEPDATARichard Purdie2019-02-251-0/+19
| | | | | | | | | | | | | | | | | The consumers of BB_TASKDEPDATA in OE metadata can't cope with multiconfig dependencies. The choice is either to start adding code to each of them to filter out multiconfig dependencies, or do this at source. After consideration we've decided to do this at source as doing otherwise is code duplication and error prone and in any case we've looked at, they don't make sense. [YOCTO #13090] [YOCTO #13130] (Bitbake rev: 7f157ea8ecf9ba259bb7e226cfd5f2870b7853a3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Fix dependency loop analysis 'hangs'Richard Purdie2019-02-251-3/+9
| | | | | | | | | | | | | Currently the mechanism for breaking out of the dependnecy loop analysis code is broken and doesn't work leading to bitbake appearing to hang. Add in a custom exception for this purpose and fix the code to exit as intended, fixing the hang and making the dependency loop code usable again. (Bitbake rev: e39dbd72ef44eebae32f9fe3b75a1bf789605558) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: Fixed section head typoScott Rifenbark2019-02-121-1/+1
| | | | | | | | | Unseting -> Unsetting (Bitbake rev: 238a936f60cbffb73f7b370b867e45f91925951c) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Add missing exception raiseRichard Purdie2019-02-061-0/+1
| | | | | | | | | | | The intent of the code was to catch one kind of error, it was actually swallowing all exceptions and looping indefinitely. Fix it to work as intended. This explains some mystery hangs we've been seeing. (Bitbake rev: d89358c7b8aa69f12b8c384c4fdb493782633494) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: toaster: correctly register the thud branchDavid Reyna2019-02-062-5/+5
| | | | | | | | | | | | There are typos in the fixture files that need to be fixed to correctly check out the 'thud' branch. [YOCTO #13064] (Bitbake rev: 03a2bc00a070794452aa1df0fd75a338bdccc47d) Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Rewrite multiple connection handlingRichard Purdie2019-01-281-22/+34
| | | | | | | | | | | | | | | | | | | | | | | | If the bitbake server recieved multiple connections, it currently closes ones it can't handle (while its dealing with another). This is rather antisocial behaviour which causes clients to quickly run through their retries and abort. Instead, queue any other connections until the current one is closed. This way the client can decide when it wants to stop waiting for the server. If the client is gone by the time we handle it, we handle that gracefully. This also fixes a number of bugs in the connection handling where connections which did drop early were badly handled causing tracebacks in the logs. Also, handle queue incomming connections in a loop to ensure that the main client handling doesn't starve that piece of the system. This code was stress tested by running 50 connection attempts in parallel at once, ensuring the code correctly handled them. (Bitbake rev: 220193dc38c4e78cb7cf36132a3a5b499a35bc8a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Handle EWOULDBLOCK in socket connectRichard Purdie2019-01-281-1/+8
| | | | | | | | | | | | | Now that we set a timeout for the socket, it can return EWOULDBLOCK if a signal or other event happens to wake up even if we don't timeout. If this happens, retry the connection, else we simply see it quickly loop through the retries and abort the connection in a very short interval. (Bitbake rev: c2000651a200530ba08161207ade5eea8bbeec43) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process.py: Set socket timeout to 10 secondsRichard Purdie2019-01-281-1/+1
| | | | | | | | | The current value of 2 seconds has shown to be short in wider testing. (Bitbake rev: 469cc520593ba52775a373faad03072b7af05dba) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Ensure socket has a timeout setRichard Purdie2019-01-281-0/+2
| | | | | | | | | | | We're seeing hangs in oe-selftest where server startup and shutdown are racing. The assumption was a connect would timeout however no timeout is set which can leave processes hanging. Set a short timeout for the connection to avoid this. (Bitbake rev: 833d95f538c007c27c6eb8d8f2f97094dc2b1a41) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib/bb/server: Avoid UnboundLocalError tracebackRichard Purdie2019-01-281-3/+3
| | | | | | | | | | | | | Traceback (most recent call last): File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake server_connection = bb.server.process.connectProcessServer(sockname, featureset) File "/home/pokybuild/yocto-worker/nightly-oe-selftest/build/bitbake/lib/bb/server/process.py", line 490, in connectProcessServer if command_chan_recv: UnboundLocalError: local variable 'command_chan_recv' referenced before assignment (Bitbake rev: 257ed88590883f46beec0164749e45733e67954e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Handle short readsRichard Purdie2019-01-281-3/+3
| | | | | | | | | Its possible the read may return a smaller number of characters. Remove the possibility by using a single character to signal the server is ready. (Bitbake rev: 7fb2d6bed06439e59a81dd91798d886ee0a72e99) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add some timing debug messages to the server startupRichard Purdie2019-01-281-0/+19
| | | | | | | | | We're seeing slow startup in bitbake, add some timeing debug messages so the logs are more useful for debugging when its slow. (Bitbake rev: 92f61dfe409da2fdd7c609cf1125878aad273d04) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Increase server startup time delayRichard Purdie2019-01-281-1/+4
| | | | | | | | | | | | | | | | | On loaded production systems we've seen bitbake server take over 40s to start up. Increase the timeout to 90s which tries to avoid failures in selftests. The delays come from setting up the inotify watches (31s) so can't really be avoided. After 5s delay we now warn the user we're waiting for 90s so the interactive exeperience shouldn't be much changed and its very unlikely the user would see that anyway. (Bitbake rev: 492a5c1b32bee1f5d7978954ee0ebdf4fccdf56f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix indirect multiconfig dependenciesAlejandro Enedino Hernandez Samaniego2019-01-281-20/+31
| | | | | | | | | | | | | | | | | | When an indirect multiconfig dependency exists, such as: A depends on B, B has a multiconfig dependency to C,and our build target is A, the multiconfig dependency to C is not processed on time, hence no providers are added for it, causing an exception in the runqueue because the dependency does exist in it. Call add_unresolved() for all available multiconfigs before processing providers for multiconfig dependencies, detecting mcdepends on time so providers for them can be added correctly. (Bitbake rev: 25b585b981cc7e4ed48b0f7c89a075486fa1eb2b) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: Support recursive deps with signature filesPeter Kjellerstedt2019-01-081-1/+3
| | | | | | | | | | | Follow dependent hash changes recursively also when specifying two signature files explicitly. Previously this was only done when using the --task option. (Bitbake rev: a5f5ec9e09e2b2891cade97d0568284fc064cb26) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: Merge with bitbake-dumpsigPeter Kjellerstedt2019-01-082-152/+82
| | | | | | | | | | | | | | | | | The functionalities of bitbake-diffsigs and bitbake-dumpsig are so similar that they can be merged into one. Add an option --dump to make bitbake-diffsigs dump the last signature data instead of comparing it. Keep bitbake-dumpsig as a symbolic link to bitbake-diffsigs. When it is called as bitbake-dumpsig, it behaves as if --dump was specified. Also make -D the short option for --debug again (the way it used to be, and still was for bitbake-dumpsig), so that -d can be used as the short option for --dump. (Bitbake rev: e7130f8bd86843c0b780b2ecabd297cd35ddcbe3) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Show last 60 lines of the log if the server didn't ↵Richard Purdie2019-01-081-0/+6
| | | | | | | | | | | | start We're seeing issues where the server doesn't start with no logs as to why. Allow the server to print the last 60 log lines just in case this shows us something useful about what is failing. (Bitbake rev: 1351978585b76262cb104f3d609d79c184ee5d2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Show the last 60 log lines, not the last 10Richard Purdie2019-01-081-2/+2
| | | | | | | | | 10 log lines may not capture any full traceback, increase the number of lines to 60 which covers most tracebacks. (Bitbake rev: 2626ff964c0a5726037e539cfd07027aded0b7a9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: print a message when no logfileRobert Yang2019-01-081-0/+4
| | | | | | | | | | | [YOCTO #12898] There might be no bitbake-cookerdaemon.log, print a message for debugging. (Bitbake rev: 9c3c965b9d90617e8aa2cf439019534a35a7bcb1) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Make lockfile handling clearerRichard Purdie2019-01-081-4/+6
| | | | | | | | | This simplifies the code and makes it easier to read but has the same functionality. (Bitbake rev: b829d1b5eb486786cae088f6927530433a7e08e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch: don't use shell=True when listing ar filesRoss Burton2019-01-081-1/+1
| | | | | | | (Bitbake rev: ff2e5d435046886791551587a0988f21ddfe6e2a) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: layerindex: don't use shell=True when cloningRoss Burton2019-01-081-1/+1
| | | | | | | (Bitbake rev: b94d4728b35350b812838cdbc7afada776ad5122) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Adapt colors used by bitbake-diffsigs to support light themesPeter Kjellerstedt2019-01-021-4/+4
| | | | | | | | | | | | | | | | | The colors specified for use with bitbake-diffsigs were adapted for a dark theme, e.g., by setting the background color to black, which made it look very bad when used with a light theme. To make it look good both with a dark or a light theme, it is better to drop the background color. It is also better to leave out the color altogether for the title and just use bold. Finally, dropping bold for the red and green texts indicating removed/added values better matches other colorized diff implementations as, e.g., git diff. (Bitbake rev: 35769412de6387f55e5efa4ebc59a8d1e3c8870f) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: Added "usehead" parameter.Yong, Jonathan2018-12-151-0/+8
| | | | | | | (Bitbake rev: a9d361e39f110f6deecd1f35c425e0f0affcf828) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main: Don't use print() directly, use loggerRichard Purdie2018-12-061-1/+1
| | | | | | | | | | | | | | | | | | | Avoid failures like: 2018-12-04 03:30:56,033 - oe-selftest - INFO - ====================================================================== 2018-12-04 03:30:56,033 - oe-selftest - INFO - FAIL: pkgdata.OePkgdataUtilTests.test_find_path (subunit.RemotedTestCase) 2018-12-04 03:30:56,033 - oe-selftest - INFO - ---------------------------------------------------------------------- 2018-12-04 03:30:56,033 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): [...] AssertionError: 'Previous bitbake instance shutting down?,[77 chars]xist' != 'ERROR: Unable to find any package produci[14 chars]xist' - Previous bitbake instance shutting down?, waiting to retry... ERROR: Unable to find any package producing path /not/exist We need to use the logger so output is correctly handled in such cases. (Bitbake rev: 1711a17dfe3980af96d713da6926e8aaa4545453) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Flush key output to logsRichard Purdie2018-12-061-0/+4
| | | | | | | | | | Small tweak to ensure these items are printed into the log since there is other logging code which looks for the header and this makes it clearer the server did start but is slow somewhere in startup. (Bitbake rev: 9b15f265643af8bd85af87164ee5cab0bf58ccbb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/svn: Fix SVN repository concurrent update raceUlf Magnusson2018-12-061-28/+36
| | | | | | | | | | | | | | | | | | | | | | | | | The ${DL_DIR}/svn directory is used by BitBake to keep checked-out SVN repositories from which tarballs are generated. These repositories were protected from concurrent update with a lock on the tarballs. However, the tarballs are specific to the SRCREV and module checked out (many tarballs can come from the same repository), meaning a repository could be modified concurrently if two recipes checked out two different SRCREVs or modules from it in parallel. This caused errors like the following: ERROR: Fetcher failure: Fetch command failed with exit code 1, output: svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for details) svn: E155004: Working copy '/home/foo/downloads/svn/repo/trunk' locked. svn: E155004: '/home/foo/downloads/svn/repo/trunk' is already locked. Fix it by adding a per-repository lock that's independent of the module and SRCREV. (Bitbake rev: cddd27fb66bf82465cf8eb5012cb8e3c088625a5) Signed-off-by: Ulf Magnusson <Ulf.Magnusson@bmw.de> Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure disk monitor is started when no setscene tasks are runRichard Purdie2018-12-061-5/+6
| | | | | | | | | | | Currently if there are no setscene tasks, the disk monitor isn't started. Move the startup code to somewhere to ensure it always is started. This issue would partially explain occasional selftest failures. (Bitbake rev: 7e76237c930d354ee7eb37d247d71b4f4fbf7993) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Fix ConnectionRefusedError tracebacksRichard Purdie2018-12-061-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | Improve connetion refused error handling: NOTE: Retrying server connection... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection... (Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/server/process.py", line 471, in connectProcessServer sock.connect(os.path.basename(sockname)) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake server_connection = bb.server.process.connectProcessServer(sockname, featureset) File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/server/process.py", line 502, in connectProcessServer os.close(i) TypeError: an integer is required (got type NoneType) ) WARNING: /home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/main.py:481: ResourceWarning: unclosed <socket.socket fd=14, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0> logger.info("Retrying server connection... (%s)" % traceback.format_exc()) (Bitbake rev: c36803dba4fce8e6b16e437d702116a918ff718b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Avoid warning about incorrect character escaping in regexRichard Purdie2018-12-061-1/+1
| | | | | | | | | | | Fixes: lib/bb/fetch2/__init__.py:259: DeprecationWarning: invalid escape sequence \w re.compile("^\w+:(?!//)").match(uri): (Bitbake rev: 9c619f1eb1583582fafda6acabaf08a5761215ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-16 22:11:59 +0000