linux/poky.git/bitbake/lib/prserv, branch krogoth-15.0.2

bitbake: prserv: Add dump_db()

2016-02-26T17:20:31+00:00

Returns a script (string) that reconstructs the state of the entire database at the time this function is called. The script language is defined by the backing database engine, which is a function of server configuration. Returns None if the database engine does not support dumping to script or if some other error is encountered in processing. The SQLite3 implementation in db.py calls iterdump() [1] to generate a script. iterdump() is the library equivalent of the `sqlite3 .dump` shell command, and the scripts are compatible. Execute the script in an empty SQLite3 database using the sqlite3 utility to restore a backup of prserv. Use case: Backup a live PR server database in a non-racy way, such that one could snapshot the entire database after a set of bitbake builds all using a shared server. I.e. All changes made prior to the start of a dump_db() operation should be committed and captured in the script. Subsequent changes made during the backup process are not guaranteed to be captured. Testing: ~7MB database backs up in ~1s while PR server is under load from 32 thread bitbake builds on two separate machines. [1] https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.iterdump (Bitbake rev: 004003daf6bd0f0233ce5c2d95f1d7d64ab91bb3) Signed-off-by: Haris Okanovic Reviewed-by: Ken Sharp Reviewed-by: Bill Pittman Signed-off-by: Richard Purdie

bitbake: bitbake: prserv: do not clear umask when daemonizing

2016-02-04T23:47:48+00:00

Clearing the umask when daemonizing is not the correct thing to do, as it will create files writable by anyone by default. For instance the pid file was being created with mode 777. This could also potentially affect the sqlite database. Better let the calling process decide on the umask. [YOCTO #9036] (Bitbake rev: ff6d3f53a4504eae7ec4c190b9f7595b09aed017) Signed-off-by: Diego Santa Cruz Signed-off-by: Richard Purdie

bitbake: bitbake: prserv: SIGTERM handling hung process

2016-02-04T23:47:48+00:00

The current SIGTERM handler hungs the process instead of making it exit. The problem seems to be that the handler thread is not signaled to quit, so it stays there doing its work, as it is not a daemon thread. Setting the quit variable fixes this. While at it, to not use the SystemExit exception to terminate upon SIGTERM but instead left the quit flag do its job. This way the PID file is properly removed. [YOCTO #9035] (Bitbake rev: 655ec800d54da581229f12efb6f0baf54975fed4) Signed-off-by: Diego Santa Cruz Signed-off-by: Richard Purdie

bitbake: bitbake: prserv: -wal and -shm sqlite lost when daemonizing

2016-02-04T23:47:48+00:00

When daemonizing the PR service the -wal and -shm sqlite files were being deleted, although they should never be. While the daemonized process keeps the file descriptors open and thus a clean exit does not loose any data, a power outage would loose all data in the WAL. Removing these files also breaks sqlite collaboration between processes and furthermore prevents taking proper backups without stopping the PR service. The reason this happens is that the DB connection is opened in the initial process, before forking for daemonization. When the DB connection is closed by the exiting parent processes it can delete the -wal and -shm files if it considers itself to be the last connection to the DB. This is easily fixed by opening the DB connection after all forking. [YOCTO #9034] (Bitbake rev: bc867c81e3894da5bdd2e45fa695bb5f5f1bb58b) Signed-off-by: Diego Santa Cruz Signed-off-by: Richard Purdie

bitbake: bitbake: Set process names to be meaninful

2016-01-30T11:43:56+00:00

This means that when you view the process tree, the processes have meaningful names, aiding debugging: $ pstree -p 30021 bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593) ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634) └─{ProcessEQueue}(115591) $ pstree -p 30021 bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335) ├─PRServ(117332)───{PRServ Handler}(117333) ├─Parser-1:2(117336) └─{ProcessEQueue}(117331) Applies to parse threads, PR Server, cooker, the workers and execution threads, working within the 16 character limit as best we can. Needed to tweak the bitbake-worker magic values to tell the workers apart. (Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5) Signed-off-by: Richard Purdie

bitbake: prserv/serv.py: Better messaging when starting/stopping the server with port=0

2015-09-28T11:00:33+00:00

When starting the server using port=0, the server actually starts with a different port, so print a message with this new value. When stopping the server with port=0, advise the user which ports the server is listening to, so next time it tries to close it, user can pick up the correct one. [YOCTO #8560] (Bitbake rev: 851e53a216682fc9133f51c05a24527cfc677741) Signed-off-by: Leonardo Sandoval Signed-off-by: Richard Purdie

bitbake: prserv/serv: Close the DB connection out of class destructor

2015-09-28T11:00:32+00:00

When launching the PR server daemon, the PRData __del__ function was being called (no reason found yet) where the DB connection closed, thus following PR updates were not getting into the DB. This patch closes the connection explicitly, not relaying on the __del__ function execution. Closing the connection in turn causes all WAL file transactions to be moved into the database (checkpoint), thus effectively updating the database. [YOCTO #8215] (Bitbake rev: c1b4754f69003df1a83fafc1c80a9ef74400b6dd) Signed-off-by: Leonardo Sandoval Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: prserv/serv: Start/Stop daemon using ip instead of host

2015-09-24T16:55:05+00:00

In cases where hostname is given instead of an IP (i.e. localhost instead of 127.0.0.1) when stopping the server with bitbake-prserv --stop, the server shows a misleading message indicating that the daemon was not found, where it is actually stopped. This patch converts host to IP values before starting/stopping the daemon, so it will always work on IP, not on hostnames, avoiding problems like the latter. [YOCTO #8258] (Bitbake rev: bd6398e967c234e89d773f509512ebf460fa76ff) Signed-off-by: Leonardo Sandoval Signed-off-by: Ross Burton Signed-off-by: Richard Purdie

bitbake: prserv: SIGTERM and deamonization fixes

2015-09-09T13:27:56+00:00

Worryingly, if you SIGKILL the bitbake cooker, an autostarted PR server will remain behind. It turns out there are a few things we should do: * The PR service doesn't need to daemonize when started from cooker, it just complicated the process lifecycle. Add a fork() method to handle this and use the non-daemon mode for the singleton. * Reset the sigterm and sigint handlers. Bitbake cooker installs its own which we inherit meaning PR server was ignoring SIGTERM. Installing our own handlers which include a sync makes most sense here. Since we're in the code, make it sync the database on SIGINT. * Use the new bb.utils.signal_on_parent_exit() call so that we get a SIGTERM when the parent (usually cooker) exits and we can shutdown too. Alternatives would be having an open pipe or polling os.getppid() for changes but this seems more effective. (Bitbake rev: 05d31fa1f56bd3d3d363a16a421d9ba7541d4293) Signed-off-by: Richard Purdie

bitbake: prserv/db: Document history modes

2015-07-12T21:50:40+00:00

I keep having to dig into the archives to remember this information. Add it as a comment to the file instead. (Bitbake rev: 21dce82056887d8d28edde61b1b82f78bdf7613c) Signed-off-by: Richard Purdie