linux/poky.git/bitbake/lib/prserv/db.py, branch dizzy-12.0.1

bitbake: prserv: Use WAL mode

2014-12-31T10:10:20+00:00

Ideally, we want the PR service to have minimal influence from queued disk IO. sqlite tends to be paranoid about data loss and locks/fsync calls. There is a "WAL mode" which changes the journalling mechanism and would appear much better suited to our use case. This patch therefore switches the database to use WAL mode. With this change, write overhead appears significantly reduced. (Bitbake rev: 90b05e79764b684b20ce8454e89f05763b02ac97) Signed-off-by: Richard Purdie

bitbake: prserv: don't wait until exit to sync

2014-12-31T10:10:20+00:00

In the commit 'prserv: Ensure data is committed', the PR server moved to only committing transactions to the database when the PR server is stopped. This improves performance, but it means that if the machine running the PR server loses power unexpectedly or if the PR server process gets SIGKILL, the uncommitted package revision data is lost. To fix this issue, sync the database periodically, once per 30 seconds by default, if it has been marked as dirty. To be safe, continue to sync the database at exit regardless of its status. (Bitbake rev: 973ac2cc63323ca9c3e916effa4765747db3564c) Signed-off-by: Ben Shelton Signed-off-by: Richard Purdie

bitbake: prserv/db: Avoid fsync() calls

2014-05-03T10:24:48+00:00

If the power were to fail, it doesn't matter to us much if the data makes it to disk or not, we'd have other problems. However an fsync() call on a multi build autobuilder is painful so lets avoid them. (Bitbake rev: 4eb2dc8048e2722d64d589f453df1ce6262c71b8) Signed-off-by: Richard Purdie

bitbake: prserv: Ensure data is committed

2013-09-08T09:51:56+00:00

In exclusive mode, we need to complete the transaction for writes to make it to the database. Therefore add sync calls to ensure this happens. Autocommit mode is significantly (100 times) slower so caching the data is of significant benefit. (Bitbake rev: 4e55f7821786a59c2cd7dbd8bfa2a22f5f196e99) Signed-off-by: Richard Purdie

bitbake: serv/db: Don't use BEGIN/COMMIT

2013-09-01T14:51:11+00:00

Since we don't support using multiple servers on the same database file, don't use the BEGIN/COMMIT syntax and allow writes to the database to work ~100 times faster with no transaction locking. (Bitbake rev: 42144a54979658f93fbbb43f7e271c1fff4d88ff) Signed-off-by: Richard Purdie

bitbake: serv/db: Take an excluside lock on the database

2013-09-01T14:51:11+00:00

We only support one server using the database at a time so take an exclusive lock and avoid later lock overhead. (Bitbake rev: e3e39be6f2d063858c92971ce8ccd89c95d4f26d) Signed-off-by: Richard Purdie

bitbake: serv/db: Fix looping upon database locked issues

2013-09-01T14:51:11+00:00

If the database is locked we will get an immediate error indicating so, there is no retry timeout. The looping code is therefore useless, the loop count is near instantly exceeded. Using a time based retry means we can wait a sensible time, then gracefully exit. (Bitbake rev: 9f9e6d87007ea87e62495705464f4232c996a165) Signed-off-by: Richard Purdie

bitbake: prserv: Allow 'table is locked' matching for retry loop

2013-08-30T16:53:26+00:00

Try and avoid errors like "ERROR: database table is locked: PRMAIN_nohist" by retrying if we see the string "is locked". (Bitbake rev: 1a175b51f80d13f747b653d29e9c0d2201b5109c) Signed-off-by: Richard Purdie

bitbake: prserv/db: Threading fixes

2013-08-28T23:13:22+00:00

Enabling threading for the PRServer causes a number of issues. Firstly is the obtuse error: sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type which is due to the class not being derived from object. See: http://docs.python.org/2/library/sqlite3.html#registering-an-adapter-callable Secondly, we want to enable multithreadded access to the database so we do this when we open it. This opens the way up to multithreading the PR server. (Bitbake rev: 5709efc2ff1e36529bd28f49cd093ccfa7abff7f) Signed-off-by: Richard Purdie

bitbake: Automatically start local PR service.

2012-01-11T10:36:20+00:00

[YOCTO #1126] A local PR service will be started and stopped automatically along with the bitbake invocation/ternimation. This local PR service will be started only and if only when the PRSERV_HOST is set to 'localhost' and PRSERV_PORT is set to '0'. When started, the sqlite3 database is stored at "${PERSISTEN_DIR}/prserv.sqlite3" or "${CACHE}/prserv.sqlite3". (Bitbake rev: 9d8f45407c67ed0d3c4f820cf646de3c385067c7) Signed-off-by: Lianhao Lu Signed-off-by: Richard Purdie