summaryrefslogtreecommitdiffstats
path: root/bitbake
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: data/siggen: Switch md5 -> sha256Richard Purdie2019-01-083-5/+5
| | | | | | | | | | Similarly to the codeparser change, change to sha256 hashes due to worries over collisions. The main impact of this change is slightly slower parsing time as well as longer sstate file names. (Bitbake rev: 66f1b766997d53b4375fdd25719b1175f3828903) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: codeparser: Switch to sha256 from md5Richard Purdie2019-01-081-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've reports of hash collision with codeparser. Looking at the way collision problems occur with md5 and the way our function templating works, I can believe we may run into issues. This patch therefore switches to sha256. Performance wise, parse time could appear to rise by 4s in 374s Before: 384329 in 2.966s (md5) After: 349743 in 2.340s (sha256) 34723 in 1.245s (md5) since we still have md5 used elsewhere in the code, something we should look at next (using sha256 everywhere is around 5.3s in total) Unfortunately this does nearly double the size of the codeparser cache file due to the hash size change. (Bitbake rev: 4bed7a97198176503fe8c72d8dd7c36b28fc9dd1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Split recipes to parse amongst threads ahead of timeRichard Purdie2019-01-081-11/+6
| | | | | | | | | | | | | | | | We have two choices, split the recipes amongst the parsing threads in blocks ahead of time, or have a queue which parsers pull from when idle. The optimum approach depends on how similar the pieces are. For the single recipe reparse case, there is currently a significant wait for the feeder thread to start (around 0.25s in a 2s command). Its possible splitting into blocks in advance may be unluckly for some other workloads but experimentally it seems to work better overall for me at least. (Bitbake rev: ae79868861568d673a70472e85a4bde9e2d84a8f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Remove Feeder() since its no longer needed with moderm ↵Richard Purdie2019-01-081-37/+5
| | | | | | | | | | | | | multiprocessing There used to be many bugs in multiprocessing and we implemented our own feeder process to avoid them. Now that we have python 3.x, these are fixed and just using the standard Queue mechanism appears to work fine. We can therefore drop the unneeded code and simplify. (Bitbake rev: b2d39fc37fcf3c81a562ec1ef4f8b4c1a493fc57) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests.data: Add underscore+numeric in overrides datastore testRichard Purdie2019-01-081-0/+9
| | | | | | | | | | Add a test for x86_64 in overrides where is was being incorrectly handled. There was a previous fix (3a3be518536acc868c7eeb3c1111ad1b321480b7) but this ensures we don't regress. (Bitbake rev: 08c314eac231ac9292c8c95f1c5a6fc2023fe749) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: hashserv: Add hash equivalence reference serverJoshua Watt2019-01-084-0/+362
| | | | | | | | | | | | | | Implements a reference implementation of the hash equivalence server. This server has minimal dependencies (and no dependencies outside of the standard Python library), and implements the minimum required to be a conforming hash equivalence server. [YOCTO #13030] (Bitbake rev: 1bb2ad0b44b94ee04870bf3f7dac4e663bed6e4d) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: persist_data: Fix Locking Protocol ErrorJoshua Watt2019-01-081-1/+1
| | | | | | | | | | | | | | | Under heavy load with process delays, sqlite can issues a "locking protocol" error (SQLITE_PROTOCOL). Unfortunately, it is impossible to distinguish between actual locking protocol errors and this race condition, so they best that can be done is to retry the operation when the error is detected. [YOCTO #13108] (Bitbake rev: 93cd15644f9d12b38abea276fee7b5bade0276df) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Pass unique hash to hash validateJoshua Watt2019-01-031-11/+30
| | | | | | | | | | | | | If the unique hash is being used to track task dependencies, the hash validation function needs to know about it in order to properly validate the hash. [YOCTO #13030] (Bitbake rev: 9a529bb2658a4046dafbf32e1eb503d84e64e947) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Pass unique hash to taskJoshua Watt2019-01-032-7/+10
| | | | | | | | | | | The unique hash is now passed to the task in the BB_UNIHASH variable [YOCTO #13030] (Bitbake rev: aab80b099f6f259e4b57cba2c26dd385d07c5947) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Track task unique hashJoshua Watt2019-01-031-8/+17
| | | | | | | | | | | Requests the task unique hash from siggen and tracks it [YOCTO #13030] (Bitbake rev: 1ecc47f0831b35c8c92b37a81cef4e43ff9f67b2) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Split out task unique hashJoshua Watt2019-01-031-2/+5
| | | | | | | | | | | | | | Abstracts the function to get the unique hash for a task. This hash is used as in place of the taskhash for the purpose of determine how other tasks depend on this one. Unless overridden, the taskhash is the same as the unique hash, preserving the original behavior. [YOCTO #13030] (Bitbake rev: d0065b34cea81fcadff14b0383779e9559d94508) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: fix indirect multiconfig dependenciesAlejandro Enedino Hernandez Samaniego2019-01-031-20/+31
| | | | | | | | | | | | | | | | | | When an indirect multiconfig dependency exists, such as: A depends on B, B has a multiconfig dependency to C,and our build target is A, the multiconfig dependency to C is not processed on time, hence no providers are added for it, causing an exception in the runqueue because the dependency does exist in it. Call add_unresolved() for all available multiconfigs before processing providers for multiconfig dependencies, detecting mcdepends on time so providers for them can be added correctly. (Bitbake rev: 8a6bc7584ad61b4de98af92a86066602006262f9) Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: Support recursive deps with signature filesPeter Kjellerstedt2019-01-031-1/+3
| | | | | | | | | | | Follow dependent hash changes recursively also when specifying two signature files explicitly. Previously this was only done when using the --task option. (Bitbake rev: 353f0f3c77c3cdd75c1be2a565234a5e53dba3ef) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-diffsigs: Merge with bitbake-dumpsigPeter Kjellerstedt2019-01-032-152/+82
| | | | | | | | | | | | | | | | | The functionalities of bitbake-diffsigs and bitbake-dumpsig are so similar that they can be merged into one. Add an option --dump to make bitbake-diffsigs dump the last signature data instead of comparing it. Keep bitbake-dumpsig as a symbolic link to bitbake-diffsigs. When it is called as bitbake-dumpsig, it behaves as if --dump was specified. Also make -D the short option for --debug again (the way it used to be, and still was for bitbake-dumpsig), so that -d can be used as the short option for --dump. (Bitbake rev: 3635b829e4eb940ada2b52bfb5b5e5be93a3b0aa) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: Created unique tags for glossary variables.Scott Rifenbark2018-12-276-328/+328
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes [YOCTO #12399] The bug was to get the BitBake User Manual into the YP Mega-manual. All the changes here create unique tags used with variables in the BitBake Manual. Prior to the fix, tags were identical between like variables in the YP reference manual and the BitBake User Manual. The reason for this is because when I created the BitBake manual's glossary, it was a cut-and-paste operation to get the bulk of the work started. At the time, the BitBake User Manual was not a part of the Mega-manual. Once we decided to include the BitBake User Manual as part of the Mega-Manual, building the mega-manual produced warnings for all these duplicate links. To fix, I have updated the variable tags in the BitBake User Manual to use the following form: 'var-bb-<variable_name>' The tags used in the YP ref-manual follow this form (original): 'var-<variable_name>' (Bitbake rev: fb6de2057aae3fbdf37f007d2e47794b332020e1) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Rewrite multiple connection handlingRichard Purdie2018-12-261-22/+34
| | | | | | | | | | | | | | | | | | | | | | | | If the bitbake server recieved multiple connections, it currently closes ones it can't handle (while its dealing with another). This is rather antisocial behaviour which causes clients to quickly run through their retries and abort. Instead, queue any other connections until the current one is closed. This way the client can decide when it wants to stop waiting for the server. If the client is gone by the time we handle it, we handle that gracefully. This also fixes a number of bugs in the connection handling where connections which did drop early were badly handled causing tracebacks in the logs. Also, handle queue incomming connections in a loop to ensure that the main client handling doesn't starve that piece of the system. This code was stress tested by running 50 connection attempts in parallel at once, ensuring the code correctly handled them. (Bitbake rev: 02845a561b38658ac3edf5cc9c34625ed860d34f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Handle EWOULDBLOCK in socket connectRichard Purdie2018-12-261-1/+8
| | | | | | | | | | | | | Now that we set a timeout for the socket, it can return EWOULDBLOCK if a signal or other event happens to wake up even if we don't timeout. If this happens, retry the connection, else we simply see it quickly loop through the retries and abort the connection in a very short interval. (Bitbake rev: d5b0a9a302ac0eafa4f797ac15ea77db87e82b3c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main: When retrying the connection, show the attempt numberRichard Purdie2018-12-261-2/+3
| | | | | | | | | The current bitbake output makes it hard to know which retry is being attempted. Add this information to the output to make it clearer. (Bitbake rev: 0774e6e03d27adb7aca6fa9c47ab6ad426c937de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process.py: Set socket timeout to 10 secondsRichard Purdie2018-12-261-1/+1
| | | | | | | | | The current value of 2 seconds has shown to be short in wider testing. (Bitbake rev: 8a1f2fcf35d61d83bbafa8fa3ae215fd5f51728b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: tests/persist_data: Add testsJoshua Watt2018-12-262-0/+143
| | | | | | | | | | | Adds a test suite for testing the persistent data cache [YOCTO #13030] (Bitbake rev: 96a4155049e834af17069d981cc2215e50d18c1a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Ensure socket has a timeout setRichard Purdie2018-12-151-0/+2
| | | | | | | | | | | We're seeing hangs in oe-selftest where server startup and shutdown are racing. The assumption was a connect would timeout however no timeout is set which can leave processes hanging. Set a short timeout for the connection to avoid this. (Bitbake rev: f02114cb70e8f6f1d32e19c02b758fe0aadecd19) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Add aarch64 support to ioprio_setRichard Purdie2018-12-141-0/+2
| | | | | | | | | With aarch64 hosts coming into use, set the syscall number to avoid ioprio warnings on that platform. (Bitbake rev: 5eaf9e7b26f09f5f106e1c3c6976d517b289450a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Allow numeric characters in overridesRichard Purdie2018-12-141-1/+2
| | | | | | | | | | | We're seeing problems due to the way x86-64 is handled (or not handled) as an override. Relax the containts on overrides from being lowercase to being lowercase or numeric. This fixes problem where MACHINE=qemux86 would work but MACHINE=qemux86-64 would fail the same tests. (Bitbake rev: 3a3be518536acc868c7eeb3c1111ad1b321480b7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: data_smart: Add missing regexp markupRichard Purdie2018-12-141-2/+2
| | | | | | | | Fix some further python3 warnings about unescaped regexs. (Bitbake rev: 8667605d016e82add95638fcb15c2bbc1b489ecc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-user-manual: Added "usehead" parameter.Yong, Jonathan2018-12-101-0/+8
| | | | | | | (Bitbake rev: 91e308992d676ea541b891261e21b840606fc1c4) Signed-off-by: Scott Rifenbark <srifenbark@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: pysh: Improve error handling for shell codeRobert Yang2018-12-092-9/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The p_error() is used for printing errors when parse shell code, but it can't the EOF error correctly - Add the following lines to quilt.inc do_configure_prepend () { find ${s} -name "*.in" -exec sed -i -e "1s,^#\!.*@perl@ -w$,#\! @perl@\nuse warnings;," {} \; if [ hello ]; then } - Before the patch: $ rm -fr cache/ tmp/cache/; bitbake -p [snip] WARNING: /path/to/quilt/quilt-native_0.65.bb: Error during finalise of /path/to/quilt/quilt-native_0.65.bb [snip] bb.pysh.sherrors.ShellSyntaxError: None followed by: We can see that this isn't easy to debug, let p_error() check wheter it is EOF and print appropriate errors can improve the error message. And don't let codeparser.py except pyshlex.NeedMore (in fact, it never worked since p_error() only raise ShellSyntaxError), but make it print the last 5 lines which might be useful for debuging. - After the patch $ rm -fr cache/ tmp/cache/; bitbake -p [snip] ERROR: /path/to/quilt/quilt_0.65.bb: Error during parse shell code, the last 5 lines are: find /path/to/quilt/0.65-r0/quilt-0.65 -name "*.in" -exec sed -i -e "1s,^#\!.*@PERL@ -w$,#\! @PERL@\nuse warnings;," {} \; if [ hello ]; then autotools_do_configure sed -e 's,^COMPAT_SYMLINKS.*:=.*,COMPAT_SYMLINKS :=,' -i /path/to/quilt/0.65-r0/quilt-0.65/Makefile [snip] File "/path/to/bb/pysh/pyshyacc.py", line 649, in p_error(p=None): w('Unexpected EOF') > raise sherrors.ShellSyntaxError(''.join(msg)) bb.pysh.sherrors.ShellSyntaxError: Unexpected EOF (Bitbake rev: 44790597951638e32eb1672de2e40bd5a603326b) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: pysh/pyshlex.py: Set is not importableRobert Yang2018-12-091-5/+0
| | | | | | | | | | | | | | >>> from Set import Set Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named 'Set' So remove related code. (Bitbake rev: 118e9d8894e64d102543562314b318676006e710) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: pysh/sherrors.py: Remove unused classesRobert Yang2018-12-091-26/+0
| | | | | | | | | | The only two that we need are ShellSyntaxError and ShellError, others are not used, so remove them. (Bitbake rev: abee2ff5ee71bf8c3221cd9d3cd4243ebb4ca273) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: pysh: Remove unused modulesRobert Yang2018-12-095-2437/+0
| | | | | | | | | | | | We don't use these modules in bitbake, so remove them. And pysh's author declared that pysh is no longer maintained: http://pysh.sourceforge.net/ (Bitbake rev: 9cc4f12d2aef366a593c7977d3818a3002b1d8bf) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: persist_data: Retry database setupJoshua Watt2018-12-081-35/+48
| | | | | | | | | | | | | The configuration of the sqlite database can timeout due to locking under heavy load and should be subject to the same retry logic as the other statements. [YOCTO #13069] (Bitbake rev: 5a2a95b0396e39662968690b3065d2f88167a71c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: utils: Use utils.remove() within prunedir()Richard Purdie2018-12-081-9/+1
| | | | | | | | | | | | We've already tweaked remove() for speed and not to error if it races for deletion. Therefore use this for prunedir() which was starting to show the same bug reports. [YOCTO #13003] (Bitbake rev: 47f359f9b017f8d99d0bb2161ac0dcefcbd915de) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2: Fix os.errno referencesRichard Purdie2018-12-081-2/+2
| | | | | | | | | | os.errno used to happen to work but is invalid. Correct to use errno. [YOCTO #13068] (Bitbake rev: 03aae96a8859409b1ce7e3d5c1197371fa96bd14) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/wget: Fix typo in exception nameRichard Purdie2018-12-081-1/+1
| | | | | | | | | This could clearly never have worked since the python3 migration but as its in an error path, it doesn't get tested/used much. (Bitbake rev: 704f27dc28d50a6dc02c8b64274ee4ecb3058c4a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cooker: Add some timing debug messages to the server startupRichard Purdie2018-12-081-0/+19
| | | | | | | | | We're seeing slow startup in bitbake, add some timeing debug messages so the logs are more useful for debugging when its slow. (Bitbake rev: 8d1fc115b8a176009f1f3a8ce840b422e7e0b45e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Increase server startup time delayRichard Purdie2018-12-081-1/+4
| | | | | | | | | | | | | | | | | On loaded production systems we've seen bitbake server take over 40s to start up. Increase the timeout to 90s which tries to avoid failures in selftests. The delays come from setting up the inotify watches (31s) so can't really be avoided. After 5s delay we now warn the user we're waiting for 90s so the interactive exeperience shouldn't be much changed and its very unlikely the user would see that anyway. (Bitbake rev: 7a4e105093c3080c1087cd06abc2883852a23e28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: cache: Cleanup deprecated function usageRichard Purdie2018-12-071-1/+1
| | | | | | | | Remove a deprecated warning and stop using our own deprecated API! (Bitbake rev: 83ece2c6f4b000e906fec9148f25bd1dff66cfb0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: siggen: Split out stampfile hash fetchJoshua Watt2018-12-071-4/+9
| | | | | | | | | | | | The mechanism used to get the hash for a stamp file is split out so that it can be overridden by derived classes [YOCTO #13030] (Bitbake rev: ce241534d19b2f1c51dbdb3b92419676d234e464) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake-worker: Pass taskhash as runtask parameterJoshua Watt2018-12-072-13/+10
| | | | | | | | | | | | | | | Pass the task hash as a parameter to the 'runtask' message instead of passing the entire dictionary of hashes when the worker is setup. This is possible less efficient, but prevents the worker taskhashes from being out of sync with the runqueue in the event that the taskhashes in the runqueue change. [YOCTO #13030] (Bitbake rev: 1e86d8c1bec7ea5d016a5ad2097f999362e29033) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Disable enable_shared_cacheJoshua Watt2018-12-071-6/+0
| | | | | | | | | | | | | Turns off the shared cache. It isn't a significant factor in performance (now that WAL is enabled), and is a really bad idea to have enabled in processes that fork() (as bitbake it prone to do). [YOCTO #13030] (Bitbake rev: 71b427bf01374973a971c10cb64024c8ef2a11eb) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Enable Write Ahead LogJoshua Watt2018-12-071-0/+5
| | | | | | | | | | | | | | | | Enabling the write ahead log improves database reliability, speeds up writes (since they mostly happen sequentially), and speeds up readers (since they are no longer blocked by most write operations). The persistent database is very read heavy, so the auto-checkpoint size is reduced from the default (usually 1000) to 100 so that reads remain fast. [YOCTO #13030] (Bitbake rev: 79100fa67539f9654af9bf6d3e6842eb5c12e989) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Add key constraintsJoshua Watt2018-12-071-1/+1
| | | | | | | | | | | | | Constructs the "key" column in the persistent database as a non-NULL primary key. This significantly speeds up lookup operations in large databases. [YOCTO #13030] (Bitbake rev: f5ba7775cfcb90401522d977cc66fe0f5aeb7a66) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Fix leaking cursors causing deadlockJoshua Watt2018-12-071-53/+135
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The original implementation of persistent data executed all SQL statements via sqlite3.Connection.execute(). Behind the scenes, this function created a sqlite3 Cursor object, executed the statement, then returned the cursor. However, the implementation did not account for this and failed to close the cursor object when it was done. The cursor would eventually be closed when the garbage collector got around to destroying it. However, sqlite has a limit on the number of cursors that can exist at any given time, and once this limit is reached it will block a query to wait for a cursor to be destroyed. Under heavy database queries, this can result in Python deadlocking with itself, since the SQL query will block waiting for a free cursor, but Python can no longer run garbage collection (as it is blocked) to free one. This restructures the SQLTable class to use two decorators to aid in performing actions correctly. The first decorator (@retry) wraps a member function in the retry logic that automatically restarts the function in the event that the database is locked. The second decorator (@transaction) wraps the function so that it occurs in a database transaction, which will automatically COMMIT the changes on success and ROLLBACK on failure. This function additionally creates an explicit cursor, passes it to the wrapped function, and cleans it up when the function is finished. Note that it is still possible to leak cursors when iterating. This is much less frequent, but can still be mitigated by wrapping the iteration in a `with` statement: with db.iteritems() as it: for (k, v) in it: ... As a side effect, since most statements are wrapped in a transaction, setting the isolation_level when the connection is created is no longer necessary. [YOCTO #13030] (Bitbake rev: e8b9d3f534ef404780be23b601d5a4bb9cec928a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: main: Don't use print() directly, use loggerRichard Purdie2018-12-051-1/+1
| | | | | | | | | | | | | | | | | | | Avoid failures like: 2018-12-04 03:30:56,033 - oe-selftest - INFO - ====================================================================== 2018-12-04 03:30:56,033 - oe-selftest - INFO - FAIL: pkgdata.OePkgdataUtilTests.test_find_path (subunit.RemotedTestCase) 2018-12-04 03:30:56,033 - oe-selftest - INFO - ---------------------------------------------------------------------- 2018-12-04 03:30:56,033 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): [...] AssertionError: 'Previous bitbake instance shutting down?,[77 chars]xist' != 'ERROR: Unable to find any package produci[14 chars]xist' - Previous bitbake instance shutting down?, waiting to retry... ERROR: Unable to find any package producing path /not/exist We need to use the logger so output is correctly handled in such cases. (Bitbake rev: b5686fc07d06e1f0a55161b5ebcec6552d74ceff) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: process: Flush key output to logsRichard Purdie2018-12-051-0/+4
| | | | | | | | | | Small tweak to ensure these items are printed into the log since there is other logging code which looks for the header and this makes it clearer the server did start but is slow somewhere in startup. (Bitbake rev: 7c0b84cac892744225fa0462f918ea9a79b356cb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Handle short readsRichard Purdie2018-12-051-3/+3
| | | | | | | | | Its possible the read may return a smaller number of characters. Remove the possibility by using a single character to signal the server is ready. (Bitbake rev: 767c9596d129d21ddf5d2e00c55f0a0525c641dc) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Make lockfile handling clearerRichard Purdie2018-12-051-4/+6
| | | | | | | | | This simplifies the code and makes it easier to read but has the same functionality. (Bitbake rev: a0b9cfaf2d03fd047a79d32e668001718d02c4bf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: BBHandler: Check tab indentation for python codeRobert Yang2018-12-052-2/+11
| | | | | | | | | | | | | | | | | | The previous check was in data.py which only can check code like "python funcname()" in the dependency chain, but there are 3 kinds of python functions: - python() - def py_funcname() - python funcname() Add the checking to BBHandler to check and warn for all of them. The warning looks like: WARNING: /path/to/recipes-core/busybox/busybox_1.29.2.bb: python should use 4 spaces indentation, but found tabs in busybox.inc, line 75 (Bitbake rev: 0cdc5b81fc1f5e5281a525a657e420ebc3bb9e90) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: fetch2/svn: Fix SVN repository concurrent update raceUlf Magnusson2018-12-011-28/+36
| | | | | | | | | | | | | | | | | | | | | | | | | The ${DL_DIR}/svn directory is used by BitBake to keep checked-out SVN repositories from which tarballs are generated. These repositories were protected from concurrent update with a lock on the tarballs. However, the tarballs are specific to the SRCREV and module checked out (many tarballs can come from the same repository), meaning a repository could be modified concurrently if two recipes checked out two different SRCREVs or modules from it in parallel. This caused errors like the following: ERROR: Fetcher failure: Fetch command failed with exit code 1, output: svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for details) svn: E155004: Working copy '/home/foo/downloads/svn/repo/trunk' locked. svn: E155004: '/home/foo/downloads/svn/repo/trunk' is already locked. Fix it by adding a per-repository lock that's independent of the module and SRCREV. (Bitbake rev: 3f1f183a17bf3580da8a4ffd6dab30b62c2654a8) Signed-off-by: Ulf Magnusson <Ulf.Magnusson@bmw.de> Signed-off-by: Michael Ho <Michael.Ho@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: runqueue: Ensure disk monitor is started when no setscene tasks are runRichard Purdie2018-12-011-5/+6
| | | | | | | | | | | Currently if there are no setscene tasks, the disk monitor isn't started. Move the startup code to somewhere to ensure it always is started. This issue would partially explain occasional selftest failures. (Bitbake rev: 5ba83ee25c1c9cba349edb68a22476b1d5fca6ce) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/process: Fix ConnectionRefusedError tracebacksRichard Purdie2018-12-011-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | Improve connetion refused error handling: NOTE: Retrying server connection... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection... (Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/server/process.py", line 471, in connectProcessServer sock.connect(os.path.basename(sockname)) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/main.py", line 464, in setup_bitbake server_connection = bb.server.process.connectProcessServer(sockname, featureset) File "/home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/server/process.py", line 502, in connectProcessServer os.close(i) TypeError: an integer is required (got type NoneType) ) WARNING: /home/pokybuild/yocto-worker/oe-selftest-opensuse/build/bitbake/lib/bb/main.py:481: ResourceWarning: unclosed <socket.socket fd=14, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0> logger.info("Retrying server connection... (%s)" % traceback.format_exc()) (Bitbake rev: 6998fd816ceb0034c852a8fb994901fdf1975cfd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-15 15:55:46 +0000