summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/prserv/db.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: prserv: Add dump_db()Haris Okanovic2016-02-261-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Returns a script (string) that reconstructs the state of the entire database at the time this function is called. The script language is defined by the backing database engine, which is a function of server configuration. Returns None if the database engine does not support dumping to script or if some other error is encountered in processing. The SQLite3 implementation in db.py calls iterdump() [1] to generate a script. iterdump() is the library equivalent of the `sqlite3 .dump` shell command, and the scripts are compatible. Execute the script in an empty SQLite3 database using the sqlite3 utility to restore a backup of prserv. Use case: Backup a live PR server database in a non-racy way, such that one could snapshot the entire database after a set of bitbake builds all using a shared server. I.e. All changes made prior to the start of a dump_db() operation should be committed and captured in the script. Subsequent changes made during the backup process are not guaranteed to be captured. Testing: ~7MB database backs up in ~1s while PR server is under load from 32 thread bitbake builds on two separate machines. [1] https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.iterdump (Bitbake rev: 004003daf6bd0f0233ce5c2d95f1d7d64ab91bb3) Signed-off-by: Haris Okanovic <haris.okanovic@ni.com> Reviewed-by: Ken Sharp <ken.sharp@ni.com> Reviewed-by: Bill Pittman <bill.pittman@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Close the DB connection out of class destructorLeonardo Sandoval2015-09-281-1/+1
| | | | | | | | | | | | | | | | | When launching the PR server daemon, the PRData __del__ function was being called (no reason found yet) where the DB connection closed, thus following PR updates were not getting into the DB. This patch closes the connection explicitly, not relaying on the __del__ function execution. Closing the connection in turn causes all WAL file transactions to be moved into the database (checkpoint), thus effectively updating the database. [YOCTO #8215] (Bitbake rev: c1b4754f69003df1a83fafc1c80a9ef74400b6dd) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/db: Document history modesRichard Purdie2015-07-121-0/+10
| | | | | | | | | I keep having to dig into the archives to remember this information. Add it as a comment to the file instead. (Bitbake rev: 21dce82056887d8d28edde61b1b82f78bdf7613c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Use WAL modeRichard Purdie2014-11-061-0/+1
| | | | | | | | | | | | | | Ideally, we want the PR service to have minimal influence from queued disk IO. sqlite tends to be paranoid about data loss and locks/fsync calls. There is a "WAL mode" which changes the journalling mechanism and would appear much better suited to our use case. This patch therefore switches the database to use WAL mode. With this change, write overhead appears significantly reduced. (Bitbake rev: 0cdd48261daeb17efc528b5de0ac81c8836e8565) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: don't wait until exit to syncBen Shelton2014-10-281-0/+14
| | | | | | | | | | | | | | | | | In the commit 'prserv: Ensure data is committed', the PR server moved to only committing transactions to the database when the PR server is stopped. This improves performance, but it means that if the machine running the PR server loses power unexpectedly or if the PR server process gets SIGKILL, the uncommitted package revision data is lost. To fix this issue, sync the database periodically, once per 30 seconds by default, if it has been marked as dirty. To be safe, continue to sync the database at exit regardless of its status. (Bitbake rev: 424df64f2e9679043f0ce2b4f7dfc59c3d404304) Signed-off-by: Ben Shelton <ben.shelton@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/db: Avoid fsync() callsRichard Purdie2014-05-031-0/+1
| | | | | | | | | | If the power were to fail, it doesn't matter to us much if the data makes it to disk or not, we'd have other problems. However an fsync() call on a multi build autobuilder is painful so lets avoid them. (Bitbake rev: 4eb2dc8048e2722d64d589f453df1ce6262c71b8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Ensure data is committedRichard Purdie2013-09-081-0/+4
| | | | | | | | | | | | In exclusive mode, we need to complete the transaction for writes to make it to the database. Therefore add sync calls to ensure this happens. Autocommit mode is significantly (100 times) slower so caching the data is of significant benefit. (Bitbake rev: 4e55f7821786a59c2cd7dbd8bfa2a22f5f196e99) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Don't use BEGIN/COMMITRichard Purdie2013-09-011-13/+3
| | | | | | | | | | Since we don't support using multiple servers on the same database file, don't use the BEGIN/COMMIT syntax and allow writes to the database to work ~100 times faster with no transaction locking. (Bitbake rev: 42144a54979658f93fbbb43f7e271c1fff4d88ff) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Take an excluside lock on the databaseRichard Purdie2013-09-011-1/+1
| | | | | | | | | We only support one server using the database at a time so take an exclusive lock and avoid later lock overhead. (Bitbake rev: e3e39be6f2d063858c92971ce8ccd89c95d4f26d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Fix looping upon database locked issuesRichard Purdie2013-09-011-3/+4
| | | | | | | | | | | | | If the database is locked we will get an immediate error indicating so, there is no retry timeout. The looping code is therefore useless, the loop count is near instantly exceeded. Using a time based retry means we can wait a sensible time, then gracefully exit. (Bitbake rev: 9f9e6d87007ea87e62495705464f4232c996a165) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Allow 'table is locked' matching for retry loopRichard Purdie2013-08-301-1/+1
| | | | | | | | | Try and avoid errors like "ERROR: database table is locked: PRMAIN_nohist" by retrying if we see the string "is locked". (Bitbake rev: 1a175b51f80d13f747b653d29e9c0d2201b5109c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/db: Threading fixesRichard Purdie2013-08-291-2/+2
| | | | | | | | | | | | | | | | | Enabling threading for the PRServer causes a number of issues. Firstly is the obtuse error: sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type which is due to the class not being derived from object. See: http://docs.python.org/2/library/sqlite3.html#registering-an-adapter-callable Secondly, we want to enable multithreadded access to the database so we do this when we open it. This opens the way up to multithreading the PR server. (Bitbake rev: 5709efc2ff1e36529bd28f49cd093ccfa7abff7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Automatically start local PR service.Lianhao Lu2012-01-111-5/+7
| | | | | | | | | | | | | | | | | [YOCTO #1126] A local PR service will be started and stopped automatically along with the bitbake invocation/ternimation. This local PR service will be started only and if only when the PRSERV_HOST is set to 'localhost' and PRSERV_PORT is set to '0'. When started, the sqlite3 database is stored at "${PERSISTEN_DIR}/prserv.sqlite3" or "${CACHE}/prserv.sqlite3". (Bitbake rev: 9d8f45407c67ed0d3c4f820cf646de3c385067c7) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/PRservice: Added no_hist mode and export/import.Lianhao Lu2012-01-111-32/+177
| | | | | | | | | | | | | | | | | | | | | | | | | [YOCTO #1556] 1. Added the package_arch into the index to the DB table. Because the change in PACKAGE_ARCH will results in different checksum, and it is better to have seperate PR value domains for differnt PACKAGE_ARCH of the same pakcage. 2. Changed the PR service to operate in no history mode. In this mode, the for a given query tuple (version, pkgarch, checksum), the returned value will be the largest among all the values of the same (version, pkgarch). This means the PR value returned can NOT be decremented. 3. Added export function. For each (version, pkgarch) tuple, only the record with the maximum value will be exported. 4. Added import function. The record will only be imported if the imported value is larger than the value stored in the DB with the same (version, pkgarch, checksum) tuple. (Bitbake rev: 379567ee879dcdc09a51f7f1212bde1076147a6f) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Add PR service deamon to bitbakeLianhao Lu2011-05-271-0/+100
Added the initial implementation of the server side PR service. (Bitbake rev: 4d0e79e5591ff58ce35c7fb96f6e9217ddc27466) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>