linux/poky.git/bitbake/lib/prserv, branch pyro-enea

bitbake: prserv/persist_data/utils: Drop obsolete python2 imports

2017-01-09T13:39:13+00:00

These imports were from python 2.6 and earlier, 2.4 in some cases. Drop them since we're all python3 now. (Bitbake rev: 7ef12684e8647b006bf46cae695069d4bfece1cf) Signed-off-by: Richard Purdie

bitbake: prserv/serv: Tweak stdout manipulation to be stream safe

2017-01-05T13:54:07+00:00

We've been seeing oe-selftest failures under puzzling circumstances. It turns out if you run oe-selftest on a machine with xmlrunner installed and have the recent tinfoil2 changes, the launching of PR server crashes leading to selftest hanging if using an autoloaded PR server. The reason is that xmlrunner uses an io.StringIO object as stdout/stderr instead of the usual io.TextIOWrapper and StringIO lacks a fileno() method. We have to deal with both cases and in the python way, we try and then seek forgivness if we see an AttributeError or UnSupportedOperation exception. Unfortunately we have to deal with both cases as we may be performing a traditiional double fork() from the commandline, or a larger python program. [YOCTO #10866] (Bitbake rev: 26243f04e3af652291d13e85c084057104fe155b) Signed-off-by: Richard Purdie

bitbake: bitbake: remove True option to getVar calls

2016-11-30T15:48:09+00:00

getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?$( ?[^,()]*), True$ (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock Signed-off-by: Richard Purdie

bitbake: bitbake: Convert to python 3

2016-06-02T07:24:02+00:00

Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie

bitbake: daemonize/prserv/tests/fetch: Convert file() -> open()

2016-05-13T12:41:32+00:00

Use python3 compatible functions. (Bitbake rev: e6a0296ba29c3fbc8417d1df7a01d50562668a41) Signed-off-by: Richard Purdie

bitbake: prserv: Add dump_db()

2016-02-26T17:20:31+00:00

Returns a script (string) that reconstructs the state of the entire database at the time this function is called. The script language is defined by the backing database engine, which is a function of server configuration. Returns None if the database engine does not support dumping to script or if some other error is encountered in processing. The SQLite3 implementation in db.py calls iterdump() [1] to generate a script. iterdump() is the library equivalent of the `sqlite3 .dump` shell command, and the scripts are compatible. Execute the script in an empty SQLite3 database using the sqlite3 utility to restore a backup of prserv. Use case: Backup a live PR server database in a non-racy way, such that one could snapshot the entire database after a set of bitbake builds all using a shared server. I.e. All changes made prior to the start of a dump_db() operation should be committed and captured in the script. Subsequent changes made during the backup process are not guaranteed to be captured. Testing: ~7MB database backs up in ~1s while PR server is under load from 32 thread bitbake builds on two separate machines. [1] https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.iterdump (Bitbake rev: 004003daf6bd0f0233ce5c2d95f1d7d64ab91bb3) Signed-off-by: Haris Okanovic Reviewed-by: Ken Sharp Reviewed-by: Bill Pittman Signed-off-by: Richard Purdie

bitbake: bitbake: prserv: do not clear umask when daemonizing

2016-02-04T23:47:48+00:00

Clearing the umask when daemonizing is not the correct thing to do, as it will create files writable by anyone by default. For instance the pid file was being created with mode 777. This could also potentially affect the sqlite database. Better let the calling process decide on the umask. [YOCTO #9036] (Bitbake rev: ff6d3f53a4504eae7ec4c190b9f7595b09aed017) Signed-off-by: Diego Santa Cruz Signed-off-by: Richard Purdie

bitbake: bitbake: prserv: SIGTERM handling hung process

2016-02-04T23:47:48+00:00

The current SIGTERM handler hungs the process instead of making it exit. The problem seems to be that the handler thread is not signaled to quit, so it stays there doing its work, as it is not a daemon thread. Setting the quit variable fixes this. While at it, to not use the SystemExit exception to terminate upon SIGTERM but instead left the quit flag do its job. This way the PID file is properly removed. [YOCTO #9035] (Bitbake rev: 655ec800d54da581229f12efb6f0baf54975fed4) Signed-off-by: Diego Santa Cruz Signed-off-by: Richard Purdie

bitbake: bitbake: prserv: -wal and -shm sqlite lost when daemonizing

2016-02-04T23:47:48+00:00

When daemonizing the PR service the -wal and -shm sqlite files were being deleted, although they should never be. While the daemonized process keeps the file descriptors open and thus a clean exit does not loose any data, a power outage would loose all data in the WAL. Removing these files also breaks sqlite collaboration between processes and furthermore prevents taking proper backups without stopping the PR service. The reason this happens is that the DB connection is opened in the initial process, before forking for daemonization. When the DB connection is closed by the exiting parent processes it can delete the -wal and -shm files if it considers itself to be the last connection to the DB. This is easily fixed by opening the DB connection after all forking. [YOCTO #9034] (Bitbake rev: bc867c81e3894da5bdd2e45fa695bb5f5f1bb58b) Signed-off-by: Diego Santa Cruz Signed-off-by: Richard Purdie

bitbake: bitbake: Set process names to be meaninful

2016-01-30T11:43:56+00:00

This means that when you view the process tree, the processes have meaningful names, aiding debugging: $ pstree -p 30021 bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593) ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634) └─{ProcessEQueue}(115591) $ pstree -p 30021 bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335) ├─PRServ(117332)───{PRServ Handler}(117333) ├─Parser-1:2(117336) └─{ProcessEQueue}(117331) Applies to parse threads, PR Server, cooker, the workers and execution threads, working within the 16 character limit as best we can. Needed to tweak the bitbake-worker magic values to tell the workers apart. (Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5) Signed-off-by: Richard Purdie