summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/prserv
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: prserv/serv: Gracefully handle the PR server exiting quicklyRichard Purdie2017-08-311-1/+5
| | | | | | | | | If the server exits quickly its PID may no longer exist. Handle this gracefully. (Bitbake rev: c1b00a9265fa4146b8db8b7d03a51bf2bfcf9f51) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Rename self.quit -> self.quitflagRichard Purdie2017-08-311-6/+6
| | | | | | | | | self has a quit function and a variable. Separate this to two different things as the current setup is prone to breakage. (Bitbake rev: ba7e3c73d8f4d2bd1d7434b97c326e7ab935231a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Send sentinel to stop handler threadRichard Purdie2017-08-311-1/+4
| | | | | | | | | Shutdown from SIGTERM currently has to wait for the handler thread to timeout. Add a sentinel value which triggers it to loop and allows for a quick exit. (Bitbake rev: a7591ef34ce70ff1d7aa9362d7473e6f16fbd10f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Shut down any existing server before restartingRichard Purdie2017-08-311-0/+3
| | | | | | | | | This allows for cleaner code in cooker as any existing server is dealt with before a new one is started. (Bitbake rev: b8616931bc0e523a3a3bb23b4f623f8b6e71d690) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/cooker: Drop unused paramRichard Purdie2017-08-311-1/+1
| | | | | | | | Drop pointless unused function parameter. (Bitbake rev: 8104b33656de0b619943bd7a9884eb650ccafbf4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Improve process exit handlingRichard Purdie2017-07-241-2/+17
| | | | | | | | | | | | | | | | | | | The server shutdown is currenlty laggy and race prone. This patch: * adds a waitpid so that no zombie server is left around if its not running in daemon mode. * adds a quit "sentinal" using a pipe so that we're not sitting in a socket poll() until timeout in order just to quit. * use a select() call to poll the socket and the pipe for a quit signal. The net result of this change is that the prserv exits with the cooker server and it does so immediately and doesn't wait for the select/poll calls to timeout. This makes bitbake a lot more responsive for startup/shutdown and doesn't cause UI timeout errors as often when prserv is used. (Bitbake rev: 0b5a837477d18442caf97dd0fa14a806c54f5842) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server: Rework the server API so process and xmlrpc servers coexistRichard Purdie2017-07-211-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This changes the way bitbake server works quite radically. Now, the server is always a process based server with the option of starting an XMLRPC listener on a specific inferface/port. Behind the scenes this is done with a "bitbake.sock" file alongside the bitbake.lock file. If we can obtain the lock, we know we need to start a server. The server always listens on the socket and UIs can then connect to this. UIs connect by sending a set of three file descriptors over the domain socket, one for sending commands, one for receiving command results and the other for receiving events. These changes meant we can throw away all the horrid server abstraction code, the plugable transport option to bitbake and the code becomes much more readable and debuggable. It also likely removes a ton of ways you could hang the UI/cooker in weird ways due to all the race conditions that existed with previous processes. Changes: * The foreground option for bitbake-server was dropped. Just tail the log if you really want this, the codepaths were complicated enough without adding one for this. * BBSERVER="autodetect" was dropped. The server will autostart and autoconnect in process mode. You have to specify an xmlrpc server address since that can't be autodetected. I can't see a use case for autodetect now. * The transport/servetype option to bitbake was dropped. * A BB_SERVER_TIMEOUT variable is added which allows the server to stay resident for a period of time after the last client disconnects before unloading. This is used if the -T/--idle-timeout option is not passed to bitbake. This change is invasive and may well introduce new issues however I believe the codebase is in a much better position for further development and debugging. (Bitbake rev: 72a3dbe13a23588e24c0baca6d58c35cdeba3f63) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/persist_data/utils: Drop obsolete python2 importsRichard Purdie2017-01-091-6/+1
| | | | | | | | | These imports were from python 2.6 and earlier, 2.4 in some cases. Drop them since we're all python3 now. (Bitbake rev: 7ef12684e8647b006bf46cae695069d4bfece1cf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Tweak stdout manipulation to be stream safeRichard Purdie2017-01-051-4/+17
| | | | | | | | | | | | | | | | | | | | | We've been seeing oe-selftest failures under puzzling circumstances. It turns out if you run oe-selftest on a machine with xmlrunner installed and have the recent tinfoil2 changes, the launching of PR server crashes leading to selftest hanging if using an autoloaded PR server. The reason is that xmlrunner uses an io.StringIO object as stdout/stderr instead of the usual io.TextIOWrapper and StringIO lacks a fileno() method. We have to deal with both cases and in the python way, we try and then seek forgivness if we see an AttributeError or UnSupportedOperation exception. Unfortunately we have to deal with both cases as we may be performing a traditiional double fork() from the commandline, or a larger python program. [YOCTO #10866] (Bitbake rev: 26243f04e3af652291d13e85c084057104fe155b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: remove True option to getVar callsJoshua Lock2016-11-301-2/+2
| | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Convert to python 3Richard Purdie2016-06-022-8/+8
| | | | | | | | | Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: daemonize/prserv/tests/fetch: Convert file() -> open()Richard Purdie2016-05-131-5/+5
| | | | | | | | Use python3 compatible functions. (Bitbake rev: e6a0296ba29c3fbc8417d1df7a01d50562668a41) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Add dump_db()Haris Okanovic2016-02-262-0/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Returns a script (string) that reconstructs the state of the entire database at the time this function is called. The script language is defined by the backing database engine, which is a function of server configuration. Returns None if the database engine does not support dumping to script or if some other error is encountered in processing. The SQLite3 implementation in db.py calls iterdump() [1] to generate a script. iterdump() is the library equivalent of the `sqlite3 .dump` shell command, and the scripts are compatible. Execute the script in an empty SQLite3 database using the sqlite3 utility to restore a backup of prserv. Use case: Backup a live PR server database in a non-racy way, such that one could snapshot the entire database after a set of bitbake builds all using a shared server. I.e. All changes made prior to the start of a dump_db() operation should be committed and captured in the script. Subsequent changes made during the backup process are not guaranteed to be captured. Testing: ~7MB database backs up in ~1s while PR server is under load from 32 thread bitbake builds on two separate machines. [1] https://docs.python.org/2/library/sqlite3.html#sqlite3.Connection.iterdump (Bitbake rev: 004003daf6bd0f0233ce5c2d95f1d7d64ab91bb3) Signed-off-by: Haris Okanovic <haris.okanovic@ni.com> Reviewed-by: Ken Sharp <ken.sharp@ni.com> Reviewed-by: Bill Pittman <bill.pittman@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: prserv: do not clear umask when daemonizingDiego Santa Cruz2016-02-041-1/+0
| | | | | | | | | | | | | | | Clearing the umask when daemonizing is not the correct thing to do, as it will create files writable by anyone by default. For instance the pid file was being created with mode 777. This could also potentially affect the sqlite database. Better let the calling process decide on the umask. [YOCTO #9036] (Bitbake rev: ff6d3f53a4504eae7ec4c190b9f7595b09aed017) Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: prserv: SIGTERM handling hung processDiego Santa Cruz2016-02-041-1/+1
| | | | | | | | | | | | | | | | | | The current SIGTERM handler hungs the process instead of making it exit. The problem seems to be that the handler thread is not signaled to quit, so it stays there doing its work, as it is not a daemon thread. Setting the quit variable fixes this. While at it, to not use the SystemExit exception to terminate upon SIGTERM but instead left the quit flag do its job. This way the PID file is properly removed. [YOCTO #9035] (Bitbake rev: 655ec800d54da581229f12efb6f0baf54975fed4) Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: prserv: -wal and -shm sqlite lost when daemonizingDiego Santa Cruz2016-02-041-5/+8
| | | | | | | | | | | | | | | | | | | | | | | | When daemonizing the PR service the -wal and -shm sqlite files were being deleted, although they should never be. While the daemonized process keeps the file descriptors open and thus a clean exit does not loose any data, a power outage would loose all data in the WAL. Removing these files also breaks sqlite collaboration between processes and furthermore prevents taking proper backups without stopping the PR service. The reason this happens is that the DB connection is opened in the initial process, before forking for daemonization. When the DB connection is closed by the exiting parent processes it can delete the -wal and -shm files if it considers itself to be the last connection to the DB. This is easily fixed by opening the DB connection after all forking. [YOCTO #9034] (Bitbake rev: bc867c81e3894da5bdd2e45fa695bb5f5f1bb58b) Signed-off-by: Diego Santa Cruz <Diego.SantaCruz@spinetix.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Set process names to be meaninfulRichard Purdie2016-01-301-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | This means that when you view the process tree, the processes have meaningful names, aiding debugging: $ pstree -p 30021 bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593) ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634) └─{ProcessEQueue}(115591) $ pstree -p 30021 bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335) ├─PRServ(117332)───{PRServ Handler}(117333) ├─Parser-1:2(117336) └─{ProcessEQueue}(117331) Applies to parse threads, PR Server, cooker, the workers and execution threads, working within the 16 character limit as best we can. Needed to tweak the bitbake-worker magic values to tell the workers apart. (Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv.py: Better messaging when starting/stopping the server ↵Leonardo Sandoval2015-09-281-2/+21
| | | | | | | | | | | | | | | | with port=0 When starting the server using port=0, the server actually starts with a different port, so print a message with this new value. When stopping the server with port=0, advise the user which ports the server is listening to, so next time it tries to close it, user can pick up the correct one. [YOCTO #8560] (Bitbake rev: 851e53a216682fc9133f51c05a24527cfc677741) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Close the DB connection out of class destructorLeonardo Sandoval2015-09-282-2/+2
| | | | | | | | | | | | | | | | | When launching the PR server daemon, the PRData __del__ function was being called (no reason found yet) where the DB connection closed, thus following PR updates were not getting into the DB. This patch closes the connection explicitly, not relaying on the __del__ function execution. Closing the connection in turn causes all WAL file transactions to be moved into the database (checkpoint), thus effectively updating the database. [YOCTO #8215] (Bitbake rev: c1b4754f69003df1a83fafc1c80a9ef74400b6dd) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Start/Stop daemon using ip instead of hostLeonardo Sandoval2015-09-241-5/+8
| | | | | | | | | | | | | | | | | In cases where hostname is given instead of an IP (i.e. localhost instead of 127.0.0.1) when stopping the server with bitbake-prserv --stop, the server shows a misleading message indicating that the daemon was not found, where it is actually stopped. This patch converts host to IP values before starting/stopping the daemon, so it will always work on IP, not on hostnames, avoiding problems like the latter. [YOCTO #8258] (Bitbake rev: bd6398e967c234e89d773f509512ebf460fa76ff) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: SIGTERM and deamonization fixesRichard Purdie2015-09-091-3/+31
| | | | | | | | | | | | | | | | | | | | | | | Worryingly, if you SIGKILL the bitbake cooker, an autostarted PR server will remain behind. It turns out there are a few things we should do: * The PR service doesn't need to daemonize when started from cooker, it just complicated the process lifecycle. Add a fork() method to handle this and use the non-daemon mode for the singleton. * Reset the sigterm and sigint handlers. Bitbake cooker installs its own which we inherit meaning PR server was ignoring SIGTERM. Installing our own handlers which include a sync makes most sense here. Since we're in the code, make it sync the database on SIGINT. * Use the new bb.utils.signal_on_parent_exit() call so that we get a SIGTERM when the parent (usually cooker) exits and we can shutdown too. Alternatives would be having an open pipe or polling os.getppid() for changes but this seems more effective. (Bitbake rev: 05d31fa1f56bd3d3d363a16a421d9ba7541d4293) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/db: Document history modesRichard Purdie2015-07-121-0/+10
| | | | | | | | | I keep having to dig into the archives to remember this information. Add it as a comment to the file instead. (Bitbake rev: 21dce82056887d8d28edde61b1b82f78bdf7613c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: serv.py: remove unused and duplicate importsMaxin B. John2015-05-081-2/+1
| | | | | | | | | Remove unused xmlrpclib, atexit and duplicated threading module imports (Bitbake rev: 3e12f4e8e3ec66d1df772a64be04b90ec72462ae) Signed-off-by: Maxin B. John <maxin.john@enea.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Improve exit handlingRichard Purdie2015-01-231-5/+9
| | | | | | | | | | | | | | | | | | Currently, I'm not sure how the prserver managed to shut down cleanly. These issues may explain some of the hangs people have reported. This change: * Ensures the connection acceptance thread monitors self.quit * We wait for the thread to exit before exitting * We sync the database when the thread exits * We do what the comment mentions, timeout after 30s and sync the database if needed. Previously, there was no timeout (the 0.5 applies to sockets, not the Queue object) (Bitbake rev: 0926492295d485813d8a4f6b77c7b152e4c5b4c4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Use WAL modeRichard Purdie2014-11-061-0/+1
| | | | | | | | | | | | | | Ideally, we want the PR service to have minimal influence from queued disk IO. sqlite tends to be paranoid about data loss and locks/fsync calls. There is a "WAL mode" which changes the journalling mechanism and would appear much better suited to our use case. This patch therefore switches the database to use WAL mode. With this change, write overhead appears significantly reduced. (Bitbake rev: 0cdd48261daeb17efc528b5de0ac81c8836e8565) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Ensure sync happens in the correct threadRichard Purdie2014-11-041-7/+8
| | | | | | | | | | | | | The sync/commit calls are happening in the submission thread which can race against the handler. The handler may start new transactions which then causes the submission thread to error with "cannot start a transaction within a transaction". The fix is to move the calls to the correct thread. (Bitbake rev: 92e128a0e331e563cfe48827e95939041c16c88e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: don't wait until exit to syncBen Shelton2014-10-282-0/+21
| | | | | | | | | | | | | | | | | In the commit 'prserv: Ensure data is committed', the PR server moved to only committing transactions to the database when the PR server is stopped. This improves performance, but it means that if the machine running the PR server loses power unexpectedly or if the PR server process gets SIGKILL, the uncommitted package revision data is lost. To fix this issue, sync the database periodically, once per 30 seconds by default, if it has been marked as dirty. To be safe, continue to sync the database at exit regardless of its status. (Bitbake rev: 424df64f2e9679043f0ce2b4f7dfc59c3d404304) Signed-off-by: Ben Shelton <ben.shelton@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Improve error message when prserver cannot bind to ↵Konrad Scherer2014-09-291-2/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | supplied host address If localhost resolves to a remote address (due to a misconfigured network), starting the pr server will fail without useful information. To reproduce, add '<bogus ip> localhost' to /etc/hosts and run 'bitbake -p'. The error message will be: ERROR: Timeout while attempting to communicate with bitbake server ERROR: Could not connect to server False: Running 'bitbake-prserv --host=localhost --port=0 --start' will fail with: error: [Errno 99] Cannot assign requested address Since these errors does not show the IP address of the attempted socket binding, this results in a lot of wasted time looking at firewall rules, etc. This patch results in the following error message if the socket binding fails: PR Server unable to bind to <bogus ip>:0 (Bitbake rev: fae5914030bcf4c061c22fc61034c40c87b7121a) Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: fix no newline at end of fileRobert Yang2014-07-041-1/+1
| | | | | | | | | | | Add a '\n' to the last line of the file to fix: No newline at end of file (Bitbake rev: 54f1359ed2e9d47980cd221b7b43ef56543fe06d) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/db: Avoid fsync() callsRichard Purdie2014-05-031-0/+1
| | | | | | | | | | If the power were to fail, it doesn't matter to us much if the data makes it to disk or not, we'd have other problems. However an fsync() call on a multi build autobuilder is painful so lets avoid them. (Bitbake rev: 4eb2dc8048e2722d64d589f453df1ce6262c71b8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Fix exit race issuesRichard Purdie2014-03-301-7/+6
| | | | | | | | | | | | | | | | | We shouldn't immediately remove the pid file when stopping the server, if we do, this causes a traceback within the server itself which can then hang. Fix this by removing the stale pid file as the last thing we do. Also: * don't printing a new "waiting" line every 0.5 seconds. * make the loop more granular since the user can 'feel' the 0.5 seconds [YOCTO #5984] (Bitbake rev: 81f41a806aeddcc38992163557672e296bcbc967) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv.py: Give pr-server up to 5 seconds to commit dataKonrad Scherer2013-11-181-2/+21
| | | | | | | | | | | | | The default value of 0.5 seconds before sending the pr-server a SIGTERM is not enough to guarantee that sqlite has committed all the pr data to the database. By polling the pid to see if it is still running, this allows the pr-server process to shutdown cleanly and finish the final pr data commit. (Bitbake rev: 22eec978e70794923c85689928c6be0cfe71cdcd) Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Ensure data is committedRichard Purdie2013-09-082-1/+6
| | | | | | | | | | | | In exclusive mode, we need to complete the transaction for writes to make it to the database. Therefore add sync calls to ensure this happens. Autocommit mode is significantly (100 times) slower so caching the data is of significant benefit. (Bitbake rev: 4e55f7821786a59c2cd7dbd8bfa2a22f5f196e99) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Settle on two threads for optimal performanceRichard Purdie2013-09-011-14/+34
| | | | | | | | | | | | Using the threading mixin class resulted in large amounts of memory being used by the PR server for no good reason. Using a receiver thread and a thread to do the actual database operations on a single connection gives the same performance with a much saner memory overhead so switch to this. (Bitbake rev: e08455d5f3b8e96765942b9c3b9767c30650557d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Don't use BEGIN/COMMITRichard Purdie2013-09-011-13/+3
| | | | | | | | | | Since we don't support using multiple servers on the same database file, don't use the BEGIN/COMMIT syntax and allow writes to the database to work ~100 times faster with no transaction locking. (Bitbake rev: 42144a54979658f93fbbb43f7e271c1fff4d88ff) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Take an excluside lock on the databaseRichard Purdie2013-09-011-1/+1
| | | | | | | | | We only support one server using the database at a time so take an exclusive lock and avoid later lock overhead. (Bitbake rev: e3e39be6f2d063858c92971ce8ccd89c95d4f26d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Fix looping upon database locked issuesRichard Purdie2013-09-011-3/+4
| | | | | | | | | | | | | If the database is locked we will get an immediate error indicating so, there is no retry timeout. The looping code is therefore useless, the loop count is near instantly exceeded. Using a time based retry means we can wait a sensible time, then gracefully exit. (Bitbake rev: 9f9e6d87007ea87e62495705464f4232c996a165) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Allow 'table is locked' matching for retry loopRichard Purdie2013-08-301-1/+1
| | | | | | | | | Try and avoid errors like "ERROR: database table is locked: PRMAIN_nohist" by retrying if we see the string "is locked". (Bitbake rev: 1a175b51f80d13f747b653d29e9c0d2201b5109c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Multithread the serverRichard Purdie2013-08-291-9/+15
| | | | | | | | | | | | | | | This makes the PR server multithreaded and able to handle multiple connections at once which means its no longer a build bottle neck when serving one connection at a time. I've experimented and database connection for each thread seems to cause the least issues, pushing the contention for sqllite to handle itself. This means moving the db/table connection code into the actual function methods. It doesn't abstract well as a function since we need the db object around for the lifetime of the function as well as the table else we lose the connection. (Bitbake rev: bf9be2029b2bded5f532bdda4c38ae3dff5d1cf6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/db: Threading fixesRichard Purdie2013-08-292-4/+4
| | | | | | | | | | | | | | | | | Enabling threading for the PRServer causes a number of issues. Firstly is the obtuse error: sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type which is due to the class not being derived from object. See: http://docs.python.org/2/library/sqlite3.html#registering-an-adapter-callable Secondly, we want to enable multithreadded access to the database so we do this when we open it. This opens the way up to multithreading the PR server. (Bitbake rev: 5709efc2ff1e36529bd28f49cd093ccfa7abff7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv.py: Fix regression from 972bc43e6d5bJason Wessel2013-08-281-1/+1
| | | | | | | | | | | | | | | commit 972bc43e6d5b1207b944b3baa8f9805adb35dda7 (serv.py: Fix hang when spawned dynamically with bitbake) introduced a regression, because the wrong patch was submitted. The syntax was incorrect in the original patch. The logger iterator must be used with a call to getLogger(). [YOCTO #5059] (Bitbake rev: 85fed8acc3af3e15bf119db2f51c486a9de3646b) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv.py: Fix hang when spawned dynamically with bitbakeJason Wessel2013-08-281-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The PRServer has the possibility to hang indefinitely blocking on a semaphore processing a xmlrpc request to send an event back to the main bitbake instance. This was observed during a "bitbake -e" on a heavily loaded machine and the main bitbake instance and cooker exited before the PRServer emitted its first log. The stack trace is provided below as to show what happens every time a logger.info() is executed in the PRServer. Not only does it write to the stream handler but it also tries to send the event to the main event processor. self._notempty.acquire() self.queue.put(event) _ui_handlers[h].event.send(event) fire_ui_handlers(event, d) fire(record, None) self.emit(record) hdlr.handle(record) self.callHandlers(record) self.handle(record) self._log(INFO, msg, args, **kwargs) (self.dbfile, self.host, self.port, str(os.getpid()))) self.work_forever() pid = self.daemonize() self.prserv.start() singleton.start() self.prhost = prserv.serv.auto_start(self.data) cooker.pre_serve() bb.cooker.server_main(self.cooker, self.main) self.run() code = process_obj._bootstrap() self._popen = Popen(self) self.serverImpl.start() server.detach() server = start_server(servermodule, configParams, configuration) ret = main() It was never intended for the PRServer to send its logs anywhere but its own log file. The event processing is an artifact of how the PRServer was forked and it inherits the event log handlers. The simple fix is to clean up and purge all the log handlers after the fork() but before doing any of the typical PRServer work or logging. (Bitbake rev: 972bc43e6d5b1207b944b3baa8f9805adb35dda7) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/xmlrpc/prserv: Add sane timeout to default xmlrpc serverRichard Purdie2013-08-261-3/+0
| | | | | | | | | | | | | | | | | The standard python socket connect has long timouts which make sense for remote connections but not local things like the PR Service. This adds a timeout parameter to the common xmlrpc server creation function and sets it to a more reasonable 5 seconds. Making the PR server instantly exit is a good way to test the effect of this on bitbake. We can remove the bodged timeout in the PRServer terminate function which has the side effect of affecting global scope. (Bitbake rev: 8c01cff94787abbb64fbdf0c16cd63f8f97a7e03) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Fix pid file removalRichard Purdie2013-08-231-1/+1
| | | | | | | | | | Mark Hatle spotted there were pid files being left around. This patch fixes things so the removal function is called correctly, the code contained a typo. (Bitbake rev: c696a16c8200c31c52750037eeafe07e065b6517) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Adapt autostart to bitbake-workerRichard Purdie2013-06-141-2/+9
| | | | | | | | | | With the change to bitbake-worker we need to ensure the workers know how to contact the PR service, the magic 0 port and singleton is no longer enough. (Bitbake rev: c761751e259bb8e940552a28794b45887b5a72d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Unbreak after bb.server changesRichard Purdie2013-05-301-1/+1
| | | | | | (Bitbake rev: e2cc22fb8b2e97b068b6037540c746ecb1856de6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Drop StandardError usageRichard Purdie2013-05-091-1/+1
| | | | | | | | StandardError doesn't exist in python 3, use Exception instead. (Bitbake rev: 4a40046036493f0cdf0f66487ad5ce083461a5c2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/cooker: Handle PRService errors cleanlyRichard Purdie2013-05-031-3/+6
| | | | | | | | | | | | Current if the PR Service fails to start, bitbake carries on regardless or hangs with no error message. This adds an exception and then handles it correctly so the UIs correctly handle the error and exit cleanly. [YOCTO #4010] (Bitbake rev: 949c01228a977c3b92bfc0802f6c71b40d8e05b3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv.py: Fix logging in daemon modeRichard Purdie2013-02-061-4/+16
| | | | | | | | | In deamon mode we need to ensure the logging module is sending log data to the log file. These changes ensure this happens correctly. (Bitbake rev: bb53b47710ca4579e20284668cb354f734c3d502) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Always use separate process for PR ServiceRichard Purdie2013-02-061-22/+7
| | | | | | | | | | | | | | | Using the threading module interacts badly with multiprocessing used elsewhere in bitbake under certain machine loads. This was leading to bitbake hanging on Ctrl+C when the PR Server was being used. This patch converts it to always use the daemonize code which then means the threading code isn't required. [YOCTO #3742] (Bitbake rev: 2d0bbd9398ab839bd2d1e29e50b25d52efb1ce2a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-10 16:40:19 +0000