summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/persist_data.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: persist_data: Handle sqlite error when cachefile path is too longYoann Congal2023-01-131-1/+20
| | | | | | | | | | | | | | | | | | | Sqlite can't open the cachefile when its path is too long (>= 505 char by my tests). We do have a path length check in sanity.bbclass but this code is called before sanity checks. Treat the error in the exception instead of checking before hand in case sqlite eventually fixes this. Fixes [YOCTO #12374] (Bitbake rev: bf681d173263cd42ffc143655f61abe0573fd83c) Signed-off-by: Yoann Congal <yoann.congal@smile.fr> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Use a valid exception for missing implementationRichard Purdie2022-04-211-1/+1
| | | | | | (Bitbake rev: 7254eb6b3e8ef504ef2274541dcc55f1d42238c6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Fix typo "committed"Zygmunt Krynicki2022-02-201-1/+1
| | | | | | | (Bitbake rev: f9bea2b29ad8121ca82df82cdbc71f5b9d15d515) Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: correct the collections vs collections.abc deprecationAlexander Kanavin2021-09-171-2/+3
| | | | | | | | | This becomes a hard error in python 3.10. (Bitbake rev: ae219e1f7460077f4492b31ac91cef4cf9b17277) Signed-off-by: Alexander Kanavin <alex@linutronix.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Drop deprecated/unused functionRichard Purdie2021-09-101-50/+0
| | | | | | | | | | This class has long since been deprecated and is unused, drop it. I'd love to get rid of the rest of persist_data but it is still used by the fetcher, sadly. (Bitbake rev: 1c574aae8c8ec427f27ab0d68bac9e7483016f18) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: logging: Make bitbake logger compatible with python loggerJoshua Watt2021-02-101-1/+1
| | | | | | | | | | | | | | | | | The bitbake logger overrode the definition of the debug() logging call to include a debug level, but this causes problems with code that may be using standard python logging, since the extra argument is interpreted differently. Instead, change the bitbake loggers debug() call to match the python logger call and add a debug2() and debug3() API to replace calls that were logging to a different debug level. [RP: Small fix to ensure bb.debug calls bbdebug()] (Bitbake rev: f68682a79d83e6399eb403f30a1f113516575f51) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: compat.py: remove file since it no longer actually implements anythingChris Laplante2020-08-261-4/+4
| | | | | | | | | | Now that compat.py just imports Python standard library stuff, get rid of the layer of indirection. (Bitbake rev: e2be6defbb9fcf25f9df04c3b452d0dba48dfd03) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data.py: Immediately get exclusive lock in __setitem__Chris Laplante2020-01-191-0/+3
| | | | | | | | | | To avoid races, SQLTable::__setitem__ needs an exclusive lock for the entire transaction, not just the INSERT/UPDATE part. (Bitbake rev: feb43e7c30f5bfab75d718896c45df621810d06f) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie2019-05-041-12/+0
| | | | | | | | | | With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie2019-05-041-0/+2
| | | | | | | | | | | | | | | | | This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: persist_data: Fix Locking Protocol ErrorJoshua Watt2019-01-081-1/+1
| | | | | | | | | | | | | | | Under heavy load with process delays, sqlite can issues a "locking protocol" error (SQLITE_PROTOCOL). Unfortunately, it is impossible to distinguish between actual locking protocol errors and this race condition, so they best that can be done is to retry the operation when the error is detected. [YOCTO #13108] (Bitbake rev: 93cd15644f9d12b38abea276fee7b5bade0276df) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: persist_data: Retry database setupJoshua Watt2018-12-081-35/+48
| | | | | | | | | | | | | The configuration of the sqlite database can timeout due to locking under heavy load and should be subject to the same retry logic as the other statements. [YOCTO #13069] (Bitbake rev: 5a2a95b0396e39662968690b3065d2f88167a71c) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Disable enable_shared_cacheJoshua Watt2018-12-071-6/+0
| | | | | | | | | | | | | Turns off the shared cache. It isn't a significant factor in performance (now that WAL is enabled), and is a really bad idea to have enabled in processes that fork() (as bitbake it prone to do). [YOCTO #13030] (Bitbake rev: 71b427bf01374973a971c10cb64024c8ef2a11eb) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Enable Write Ahead LogJoshua Watt2018-12-071-0/+5
| | | | | | | | | | | | | | | | Enabling the write ahead log improves database reliability, speeds up writes (since they mostly happen sequentially), and speeds up readers (since they are no longer blocked by most write operations). The persistent database is very read heavy, so the auto-checkpoint size is reduced from the default (usually 1000) to 100 so that reads remain fast. [YOCTO #13030] (Bitbake rev: 79100fa67539f9654af9bf6d3e6842eb5c12e989) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Add key constraintsJoshua Watt2018-12-071-1/+1
| | | | | | | | | | | | | Constructs the "key" column in the persistent database as a non-NULL primary key. This significantly speeds up lookup operations in large databases. [YOCTO #13030] (Bitbake rev: f5ba7775cfcb90401522d977cc66fe0f5aeb7a66) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Fix leaking cursors causing deadlockJoshua Watt2018-12-071-53/+135
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The original implementation of persistent data executed all SQL statements via sqlite3.Connection.execute(). Behind the scenes, this function created a sqlite3 Cursor object, executed the statement, then returned the cursor. However, the implementation did not account for this and failed to close the cursor object when it was done. The cursor would eventually be closed when the garbage collector got around to destroying it. However, sqlite has a limit on the number of cursors that can exist at any given time, and once this limit is reached it will block a query to wait for a cursor to be destroyed. Under heavy database queries, this can result in Python deadlocking with itself, since the SQL query will block waiting for a free cursor, but Python can no longer run garbage collection (as it is blocked) to free one. This restructures the SQLTable class to use two decorators to aid in performing actions correctly. The first decorator (@retry) wraps a member function in the retry logic that automatically restarts the function in the event that the database is locked. The second decorator (@transaction) wraps the function so that it occurs in a database transaction, which will automatically COMMIT the changes on success and ROLLBACK on failure. This function additionally creates an explicit cursor, passes it to the wrapped function, and cleans it up when the function is finished. Note that it is still possible to leak cursors when iterating. This is much less frequent, but can still be mitigated by wrapping the iteration in a `with` statement: with db.iteritems() as it: for (k, v) in it: ... As a side effect, since most statements are wrapped in a transaction, setting the isolation_level when the connection is created is no longer necessary. [YOCTO #13030] (Bitbake rev: e8b9d3f534ef404780be23b601d5a4bb9cec928a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/persist_data/utils: Drop obsolete python2 importsRichard Purdie2017-01-091-5/+1
| | | | | | | | | These imports were from python 2.6 and earlier, 2.4 in some cases. Drop them since we're all python3 now. (Bitbake rev: 7ef12684e8647b006bf46cae695069d4bfece1cf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: remove True option to getVar callsJoshua Lock2016-11-301-2/+2
| | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Fix py3 update stack overflowRichard Purdie2016-08-111-2/+2
| | | | | | | | | | Revision d0f904d407f57998419bd9c305ce53e5eaa36b24 accidentally broke items() and values() and made them cause stack overflows. Undo that breakage. (Bitbake rev: 88c5beca705efa7df4a96fb2aaf3f13c336ac328) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Convert to python 3Richard Purdie2016-06-021-5/+5
| | | | | | | | | Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Return str instead of unicode for sqlite3 text queriesKlauer, Daniel2016-04-061-0/+1
| | | | | | | | | | | | | | | | | | Python 2's sqlite3 module defaults to returning Unicode strings for SQL text queries, which could trickle down to other parts of bitbake code and cause unexpected Unicode conversions. Using byte strings avoids this issue. For example, the git fetcher's AUTOREV support caches HEAD SHA1's using bb.persist_data, so sometimes the git command strings passed to fetch2's runfetchcmd() were unicode, potentially causing UnicodeDecodeErrors when it appended the values of environment variables containing non-ASCII chars. [YOCTO #9382] (Bitbake rev: 09623a0811c613a47a01ae465b822d8156faca30) Signed-off-by: Daniel Klauer <daniel.klauer@gin.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: Avoid fsync() callsRichard Purdie2014-05-031-1/+3
| | | | | | | | | | | | | If the power were to fail, it doesn't matter to us much if the data makes it to disk or not, we'd have other problems. However an fsync() call on a multi build autobuilder is painful so lets avoid them. This is particularly true in this case if a timeout causes a reconnect during a build. (Bitbake rev: ec28256ac2a30f047585e8f61200d764bc295ded) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: persist_data: add get_by_pattern method to APIConstantin Musca2013-01-221-0/+5
| | | | | | | | | | - one can use get_by_pattern to get a list of values associated with keys that match the specified pattern (Bitbake rev: 6ee1f58698e2d782c54ce5aec271bcec26107eac) Signed-off-by: Constantin Musca <constantinx.musca@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: survive if sqlite3 couldn't enable the shared cacheChristopher Larson2012-02-221-1/+4
| | | | | | | (Bitbake rev: e870fd57bb84a17a871a5b5de3bf5202fe9b135f) Signed-off-by: Christopher Larson <kergoth@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/persist_data: Reconnect when DB is lockedLianhao Lu2012-01-191-5/+7
| | | | | | | | | | | [YOCTO #1761] Reconnect to the backend Sqlite DB in 'database is locked' exception so the timeout can be leveraged in each time retry. (Bitbake rev: b310382764367b573c84f33d847c6eb821266f9e) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Update users of getVar/setVar to use the data store functions directlyRichard Purdie2011-11-271-3/+3
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/persist_data: Sync file with upstream bitbakeRichard Purdie2011-05-061-12/+24
| | | | Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: implement comparison, same as dictChris Larson2011-05-061-31/+17
| | | | | | | (Bitbake rev: 1190406c526c7bb7cf415867be83e0403812a7dd) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: don't allow non-string keys/valuesChris Larson2011-05-061-0/+5
| | | | | | | (Bitbake rev: 28958cd55e592853c68f5f2ba79381d1b8dcfb8f) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: add has_keyChris Larson2011-05-061-0/+3
| | | | | | | (Bitbake rev: 7942833ca0685cf4f3b243dde6203499ef97420c) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: raise KeyError on missing elementsChris Larson2011-05-061-0/+3
| | | | | | | (Bitbake rev: a4f62433845c29f98c6a9746d5d2847bf9506ea5) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Initial work on getting bitbake working under pypyChris Larson2011-05-061-0/+2
| | | | | | | | | | | - use os.chmod, not os.fchmod, as the latter is missing under pypy - rearrange our imports a bit - don't die if sqlite3 is missing shared cache support (Bitbake rev: f229824dc9c453adf6067500e2bf6761536e4f2f) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: kill unreachable break lineChris Larson2011-02-101-1/+0
| | | | | | | (Bitbake rev: 7486b38603f2766adaf976a9f95e9276c83abe31) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: loop on database lock for table creationChris Larson2011-02-101-2/+2
| | | | | | | (Bitbake rev: d93fcbd64ab5d806288424170f55323b4297e7d6) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* persist_data: resurrect the lock wait for selectsChris Larson2011-01-051-7/+7
| | | | | | | | | Think this got inadvertantly dropped when switching to the new API. (Bitbake rev: 628c5159d1151b89f2b7210c8819489e8dc9a84d) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Rework the persist_data APIChris Larson2011-01-051-72/+137
| | | | | | | | | | | | | | | | | | Rather than having to run .addDomain() and then .getValue(domain, key), .setValue(domain, key), etc, now it just works as mappings. As an example: setValue(domain, key) -> persist[domain][key] = value It also arranges things so we can construct objects of this type using any arbitrary filename/path for the sqlite3 database, rather than being so tightly bound to the metadata. (Bitbake rev: d9e8b8af308ae871efdc8ef0782be30af8c1f894) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Revert "persist_data: cache connection and use cursor"Chris Larson2011-01-051-8/+3
| | | | | | | | | | | | | Caching the database connection can cause serious issues if it results in multiple processes (e.g. multiple tasks) simultaneously using the same connection. This reverts commit 8a6876752b90efd81d92f0947bfc9527d8260969. (Bitbake rev: 60b9b18eafad5ac46c7cf1048d749d673c2ee0ad) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* persist_data: handle locked db for SELECTChris Larson2011-01-041-3/+1
| | | | | | | | | | Parallel processes interacting with the persist_data db can quite easily explode without this. (Bitbake rev: b3d5432cff0ff28f4c8a5bcf10efa3e383b4fd4d) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Switch bitbake internals to use logging directly rather than bb.msgChris Larson2011-01-041-2/+6
| | | | | | | | | | | | | | We use a custom Logger subclass for our loggers This logger provides: - 'debug' method which accepts a debug level - 'plain' method which bypasses log formatting - 'verbose' method which is more detail than info, but less than debug (Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake/persist_data: Attempt to fix locking issuesRichard Purdie2010-08-201-8/+17
| | | | | | | | | It appears the timeout sometimes has no effect and we see database access failures. Combat this by wrapping the execute function in all cases and retrying manually ourselves. Thanks to Kevin Tian for help debugging this. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* persist_data: cache connection and use cursorBernhard Reutner-Fischer2010-07-021-8/+13
| | | | | | | | | | Store database connection to persistent database in fetcher. (Bitbake rev: 8a6876752b90efd81d92f0947bfc9527d8260969) Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Apply some 2to3 transforms that don't cause issues in 2.6Chris Larson2010-07-021-2/+2
| | | | | | | (Bitbake rev: d39ab776e7ceaefc8361150151cf0892dcb70d9c) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Switch some references to moved functionsChris Larson2010-07-021-1/+2
| | | | | | | (Bitbake rev: ddea2978cb969cf1381cfee5c055de0c9e56a4e3) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Formatting cleanupsChris Larson2010-07-021-8/+5
| | | | | | | (Bitbake rev: 2caf134b43a44dad30af4fbe33033b3c58deee57) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: Remove persist_data domain renaming code as it appears to expose ↵Richard Purdie2009-07-301-10/+0
| | | | | | sqlite bugs and data consistency issues Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: Make sure tables exist when renamingRichard Purdie2009-07-301-0/+1
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: Add renameDomain and getKeyValues functions to the persistent data codeRichard Purdie2009-07-291-0/+20
| | | | Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* bitbake: Sync with 1.8 branchRichard Purdie2008-05-131-2/+2
| | | | git-svn-id: https://svn.o-hand.com/repos/poky/trunk@4463 311d38ba-8fff-0310-9ca6-ca027cbcb966
* bitbake: Update to 1.8.10 releaseRichard Purdie2008-01-201-1/+5
| | | | git-svn-id: https://svn.o-hand.com/repos/poky/trunk@3555 311d38ba-8fff-0310-9ca6-ca027cbcb966
* bitbake persist_data: Fix for python 2.4Richard Purdie2007-08-041-2/+2
| | | | git-svn-id: https://svn.o-hand.com/repos/poky/trunk@2366 311d38ba-8fff-0310-9ca6-ca027cbcb966
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-10 03:32:17 +0000