summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/prserv/db.py
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: prserv: Use WAL modeRichard Purdie2014-11-061-0/+1
| | | | | | | | | | | | | | Ideally, we want the PR service to have minimal influence from queued disk IO. sqlite tends to be paranoid about data loss and locks/fsync calls. There is a "WAL mode" which changes the journalling mechanism and would appear much better suited to our use case. This patch therefore switches the database to use WAL mode. With this change, write overhead appears significantly reduced. (Bitbake rev: 0cdd48261daeb17efc528b5de0ac81c8836e8565) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: don't wait until exit to syncBen Shelton2014-10-281-0/+14
| | | | | | | | | | | | | | | | | In the commit 'prserv: Ensure data is committed', the PR server moved to only committing transactions to the database when the PR server is stopped. This improves performance, but it means that if the machine running the PR server loses power unexpectedly or if the PR server process gets SIGKILL, the uncommitted package revision data is lost. To fix this issue, sync the database periodically, once per 30 seconds by default, if it has been marked as dirty. To be safe, continue to sync the database at exit regardless of its status. (Bitbake rev: 424df64f2e9679043f0ce2b4f7dfc59c3d404304) Signed-off-by: Ben Shelton <ben.shelton@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/db: Avoid fsync() callsRichard Purdie2014-05-031-0/+1
| | | | | | | | | | If the power were to fail, it doesn't matter to us much if the data makes it to disk or not, we'd have other problems. However an fsync() call on a multi build autobuilder is painful so lets avoid them. (Bitbake rev: 4eb2dc8048e2722d64d589f453df1ce6262c71b8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Ensure data is committedRichard Purdie2013-09-081-0/+4
| | | | | | | | | | | | In exclusive mode, we need to complete the transaction for writes to make it to the database. Therefore add sync calls to ensure this happens. Autocommit mode is significantly (100 times) slower so caching the data is of significant benefit. (Bitbake rev: 4e55f7821786a59c2cd7dbd8bfa2a22f5f196e99) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Don't use BEGIN/COMMITRichard Purdie2013-09-011-13/+3
| | | | | | | | | | Since we don't support using multiple servers on the same database file, don't use the BEGIN/COMMIT syntax and allow writes to the database to work ~100 times faster with no transaction locking. (Bitbake rev: 42144a54979658f93fbbb43f7e271c1fff4d88ff) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Take an excluside lock on the databaseRichard Purdie2013-09-011-1/+1
| | | | | | | | | We only support one server using the database at a time so take an exclusive lock and avoid later lock overhead. (Bitbake rev: e3e39be6f2d063858c92971ce8ccd89c95d4f26d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Fix looping upon database locked issuesRichard Purdie2013-09-011-3/+4
| | | | | | | | | | | | | If the database is locked we will get an immediate error indicating so, there is no retry timeout. The looping code is therefore useless, the loop count is near instantly exceeded. Using a time based retry means we can wait a sensible time, then gracefully exit. (Bitbake rev: 9f9e6d87007ea87e62495705464f4232c996a165) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Allow 'table is locked' matching for retry loopRichard Purdie2013-08-301-1/+1
| | | | | | | | | Try and avoid errors like "ERROR: database table is locked: PRMAIN_nohist" by retrying if we see the string "is locked". (Bitbake rev: 1a175b51f80d13f747b653d29e9c0d2201b5109c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/db: Threading fixesRichard Purdie2013-08-291-2/+2
| | | | | | | | | | | | | | | | | Enabling threading for the PRServer causes a number of issues. Firstly is the obtuse error: sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type which is due to the class not being derived from object. See: http://docs.python.org/2/library/sqlite3.html#registering-an-adapter-callable Secondly, we want to enable multithreadded access to the database so we do this when we open it. This opens the way up to multithreading the PR server. (Bitbake rev: 5709efc2ff1e36529bd28f49cd093ccfa7abff7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Automatically start local PR service.Lianhao Lu2012-01-111-5/+7
| | | | | | | | | | | | | | | | | [YOCTO #1126] A local PR service will be started and stopped automatically along with the bitbake invocation/ternimation. This local PR service will be started only and if only when the PRSERV_HOST is set to 'localhost' and PRSERV_PORT is set to '0'. When started, the sqlite3 database is stored at "${PERSISTEN_DIR}/prserv.sqlite3" or "${CACHE}/prserv.sqlite3". (Bitbake rev: 9d8f45407c67ed0d3c4f820cf646de3c385067c7) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/PRservice: Added no_hist mode and export/import.Lianhao Lu2012-01-111-32/+177
| | | | | | | | | | | | | | | | | | | | | | | | | [YOCTO #1556] 1. Added the package_arch into the index to the DB table. Because the change in PACKAGE_ARCH will results in different checksum, and it is better to have seperate PR value domains for differnt PACKAGE_ARCH of the same pakcage. 2. Changed the PR service to operate in no history mode. In this mode, the for a given query tuple (version, pkgarch, checksum), the returned value will be the largest among all the values of the same (version, pkgarch). This means the PR value returned can NOT be decremented. 3. Added export function. For each (version, pkgarch) tuple, only the record with the maximum value will be exported. 4. Added import function. The record will only be imported if the imported value is larger than the value stored in the DB with the same (version, pkgarch, checksum) tuple. (Bitbake rev: 379567ee879dcdc09a51f7f1212bde1076147a6f) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Add PR service deamon to bitbakeLianhao Lu2011-05-271-0/+100
Added the initial implementation of the server side PR service. (Bitbake rev: 4d0e79e5591ff58ce35c7fb96f6e9217ddc27466) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>