summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/prserv
Commit message (Collapse)AuthorAgeFilesLines
* bitbake: prserv/db: Document history modesRichard Purdie2015-07-121-0/+10
| | | | | | | | | I keep having to dig into the archives to remember this information. Add it as a comment to the file instead. (Bitbake rev: 21dce82056887d8d28edde61b1b82f78bdf7613c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: serv.py: remove unused and duplicate importsMaxin B. John2015-05-081-2/+1
| | | | | | | | | Remove unused xmlrpclib, atexit and duplicated threading module imports (Bitbake rev: 3e12f4e8e3ec66d1df772a64be04b90ec72462ae) Signed-off-by: Maxin B. John <maxin.john@enea.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Improve exit handlingRichard Purdie2015-01-231-5/+9
| | | | | | | | | | | | | | | | | | Currently, I'm not sure how the prserver managed to shut down cleanly. These issues may explain some of the hangs people have reported. This change: * Ensures the connection acceptance thread monitors self.quit * We wait for the thread to exit before exitting * We sync the database when the thread exits * We do what the comment mentions, timeout after 30s and sync the database if needed. Previously, there was no timeout (the 0.5 applies to sockets, not the Queue object) (Bitbake rev: 0926492295d485813d8a4f6b77c7b152e4c5b4c4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Use WAL modeRichard Purdie2014-11-061-0/+1
| | | | | | | | | | | | | | Ideally, we want the PR service to have minimal influence from queued disk IO. sqlite tends to be paranoid about data loss and locks/fsync calls. There is a "WAL mode" which changes the journalling mechanism and would appear much better suited to our use case. This patch therefore switches the database to use WAL mode. With this change, write overhead appears significantly reduced. (Bitbake rev: 0cdd48261daeb17efc528b5de0ac81c8836e8565) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Ensure sync happens in the correct threadRichard Purdie2014-11-041-7/+8
| | | | | | | | | | | | | The sync/commit calls are happening in the submission thread which can race against the handler. The handler may start new transactions which then causes the submission thread to error with "cannot start a transaction within a transaction". The fix is to move the calls to the correct thread. (Bitbake rev: 92e128a0e331e563cfe48827e95939041c16c88e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: don't wait until exit to syncBen Shelton2014-10-282-0/+21
| | | | | | | | | | | | | | | | | In the commit 'prserv: Ensure data is committed', the PR server moved to only committing transactions to the database when the PR server is stopped. This improves performance, but it means that if the machine running the PR server loses power unexpectedly or if the PR server process gets SIGKILL, the uncommitted package revision data is lost. To fix this issue, sync the database periodically, once per 30 seconds by default, if it has been marked as dirty. To be safe, continue to sync the database at exit regardless of its status. (Bitbake rev: 424df64f2e9679043f0ce2b4f7dfc59c3d404304) Signed-off-by: Ben Shelton <ben.shelton@ni.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Improve error message when prserver cannot bind to ↵Konrad Scherer2014-09-291-2/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | supplied host address If localhost resolves to a remote address (due to a misconfigured network), starting the pr server will fail without useful information. To reproduce, add '<bogus ip> localhost' to /etc/hosts and run 'bitbake -p'. The error message will be: ERROR: Timeout while attempting to communicate with bitbake server ERROR: Could not connect to server False: Running 'bitbake-prserv --host=localhost --port=0 --start' will fail with: error: [Errno 99] Cannot assign requested address Since these errors does not show the IP address of the attempted socket binding, this results in a lot of wasted time looking at firewall rules, etc. This patch results in the following error message if the socket binding fails: PR Server unable to bind to <bogus ip>:0 (Bitbake rev: fae5914030bcf4c061c22fc61034c40c87b7121a) Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: lib: fix no newline at end of fileRobert Yang2014-07-041-1/+1
| | | | | | | | | | | Add a '\n' to the last line of the file to fix: No newline at end of file (Bitbake rev: 54f1359ed2e9d47980cd221b7b43ef56543fe06d) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/db: Avoid fsync() callsRichard Purdie2014-05-031-0/+1
| | | | | | | | | | If the power were to fail, it doesn't matter to us much if the data makes it to disk or not, we'd have other problems. However an fsync() call on a multi build autobuilder is painful so lets avoid them. (Bitbake rev: 4eb2dc8048e2722d64d589f453df1ce6262c71b8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Fix exit race issuesRichard Purdie2014-03-301-7/+6
| | | | | | | | | | | | | | | | | We shouldn't immediately remove the pid file when stopping the server, if we do, this causes a traceback within the server itself which can then hang. Fix this by removing the stale pid file as the last thing we do. Also: * don't printing a new "waiting" line every 0.5 seconds. * make the loop more granular since the user can 'feel' the 0.5 seconds [YOCTO #5984] (Bitbake rev: 81f41a806aeddcc38992163557672e296bcbc967) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv.py: Give pr-server up to 5 seconds to commit dataKonrad Scherer2013-11-181-2/+21
| | | | | | | | | | | | | The default value of 0.5 seconds before sending the pr-server a SIGTERM is not enough to guarantee that sqlite has committed all the pr data to the database. By polling the pid to see if it is still running, this allows the pr-server process to shutdown cleanly and finish the final pr data commit. (Bitbake rev: 22eec978e70794923c85689928c6be0cfe71cdcd) Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Ensure data is committedRichard Purdie2013-09-082-1/+6
| | | | | | | | | | | | In exclusive mode, we need to complete the transaction for writes to make it to the database. Therefore add sync calls to ensure this happens. Autocommit mode is significantly (100 times) slower so caching the data is of significant benefit. (Bitbake rev: 4e55f7821786a59c2cd7dbd8bfa2a22f5f196e99) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Settle on two threads for optimal performanceRichard Purdie2013-09-011-14/+34
| | | | | | | | | | | | Using the threading mixin class resulted in large amounts of memory being used by the PR server for no good reason. Using a receiver thread and a thread to do the actual database operations on a single connection gives the same performance with a much saner memory overhead so switch to this. (Bitbake rev: e08455d5f3b8e96765942b9c3b9767c30650557d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Don't use BEGIN/COMMITRichard Purdie2013-09-011-13/+3
| | | | | | | | | | Since we don't support using multiple servers on the same database file, don't use the BEGIN/COMMIT syntax and allow writes to the database to work ~100 times faster with no transaction locking. (Bitbake rev: 42144a54979658f93fbbb43f7e271c1fff4d88ff) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Take an excluside lock on the databaseRichard Purdie2013-09-011-1/+1
| | | | | | | | | We only support one server using the database at a time so take an exclusive lock and avoid later lock overhead. (Bitbake rev: e3e39be6f2d063858c92971ce8ccd89c95d4f26d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv/db: Fix looping upon database locked issuesRichard Purdie2013-09-011-3/+4
| | | | | | | | | | | | | If the database is locked we will get an immediate error indicating so, there is no retry timeout. The looping code is therefore useless, the loop count is near instantly exceeded. Using a time based retry means we can wait a sensible time, then gracefully exit. (Bitbake rev: 9f9e6d87007ea87e62495705464f4232c996a165) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Allow 'table is locked' matching for retry loopRichard Purdie2013-08-301-1/+1
| | | | | | | | | Try and avoid errors like "ERROR: database table is locked: PRMAIN_nohist" by retrying if we see the string "is locked". (Bitbake rev: 1a175b51f80d13f747b653d29e9c0d2201b5109c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Multithread the serverRichard Purdie2013-08-291-9/+15
| | | | | | | | | | | | | | | This makes the PR server multithreaded and able to handle multiple connections at once which means its no longer a build bottle neck when serving one connection at a time. I've experimented and database connection for each thread seems to cause the least issues, pushing the contention for sqllite to handle itself. This means moving the db/table connection code into the actual function methods. It doesn't abstract well as a function since we need the db object around for the lifetime of the function as well as the table else we lose the connection. (Bitbake rev: bf9be2029b2bded5f532bdda4c38ae3dff5d1cf6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/db: Threading fixesRichard Purdie2013-08-292-4/+4
| | | | | | | | | | | | | | | | | Enabling threading for the PRServer causes a number of issues. Firstly is the obtuse error: sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type which is due to the class not being derived from object. See: http://docs.python.org/2/library/sqlite3.html#registering-an-adapter-callable Secondly, we want to enable multithreadded access to the database so we do this when we open it. This opens the way up to multithreading the PR server. (Bitbake rev: 5709efc2ff1e36529bd28f49cd093ccfa7abff7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv.py: Fix regression from 972bc43e6d5bJason Wessel2013-08-281-1/+1
| | | | | | | | | | | | | | | commit 972bc43e6d5b1207b944b3baa8f9805adb35dda7 (serv.py: Fix hang when spawned dynamically with bitbake) introduced a regression, because the wrong patch was submitted. The syntax was incorrect in the original patch. The logger iterator must be used with a call to getLogger(). [YOCTO #5059] (Bitbake rev: 85fed8acc3af3e15bf119db2f51c486a9de3646b) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: serv.py: Fix hang when spawned dynamically with bitbakeJason Wessel2013-08-281-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The PRServer has the possibility to hang indefinitely blocking on a semaphore processing a xmlrpc request to send an event back to the main bitbake instance. This was observed during a "bitbake -e" on a heavily loaded machine and the main bitbake instance and cooker exited before the PRServer emitted its first log. The stack trace is provided below as to show what happens every time a logger.info() is executed in the PRServer. Not only does it write to the stream handler but it also tries to send the event to the main event processor. self._notempty.acquire() self.queue.put(event) _ui_handlers[h].event.send(event) fire_ui_handlers(event, d) fire(record, None) self.emit(record) hdlr.handle(record) self.callHandlers(record) self.handle(record) self._log(INFO, msg, args, **kwargs) (self.dbfile, self.host, self.port, str(os.getpid()))) self.work_forever() pid = self.daemonize() self.prserv.start() singleton.start() self.prhost = prserv.serv.auto_start(self.data) cooker.pre_serve() bb.cooker.server_main(self.cooker, self.main) self.run() code = process_obj._bootstrap() self._popen = Popen(self) self.serverImpl.start() server.detach() server = start_server(servermodule, configParams, configuration) ret = main() It was never intended for the PRServer to send its logs anywhere but its own log file. The event processing is an artifact of how the PRServer was forked and it inherits the event log handlers. The simple fix is to clean up and purge all the log handlers after the fork() but before doing any of the typical PRServer work or logging. (Bitbake rev: 972bc43e6d5b1207b944b3baa8f9805adb35dda7) Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: server/xmlrpc/prserv: Add sane timeout to default xmlrpc serverRichard Purdie2013-08-261-3/+0
| | | | | | | | | | | | | | | | | The standard python socket connect has long timouts which make sense for remote connections but not local things like the PR Service. This adds a timeout parameter to the common xmlrpc server creation function and sets it to a more reasonable 5 seconds. Making the PR server instantly exit is a good way to test the effect of this on bitbake. We can remove the bodged timeout in the PRServer terminate function which has the side effect of affecting global scope. (Bitbake rev: 8c01cff94787abbb64fbdf0c16cd63f8f97a7e03) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv: Fix pid file removalRichard Purdie2013-08-231-1/+1
| | | | | | | | | | Mark Hatle spotted there were pid files being left around. This patch fixes things so the removal function is called correctly, the code contained a typo. (Bitbake rev: c696a16c8200c31c52750037eeafe07e065b6517) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Adapt autostart to bitbake-workerRichard Purdie2013-06-141-2/+9
| | | | | | | | | | With the change to bitbake-worker we need to ensure the workers know how to contact the PR service, the magic 0 port and singleton is no longer enough. (Bitbake rev: c761751e259bb8e940552a28794b45887b5a72d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Unbreak after bb.server changesRichard Purdie2013-05-301-1/+1
| | | | | | (Bitbake rev: e2cc22fb8b2e97b068b6037540c746ecb1856de6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: Drop StandardError usageRichard Purdie2013-05-091-1/+1
| | | | | | | | StandardError doesn't exist in python 3, use Exception instead. (Bitbake rev: 4a40046036493f0cdf0f66487ad5ce083461a5c2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/cooker: Handle PRService errors cleanlyRichard Purdie2013-05-031-3/+6
| | | | | | | | | | | | Current if the PR Service fails to start, bitbake carries on regardless or hangs with no error message. This adds an exception and then handles it correctly so the UIs correctly handle the error and exit cleanly. [YOCTO #4010] (Bitbake rev: 949c01228a977c3b92bfc0802f6c71b40d8e05b3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv/serv.py: Fix logging in daemon modeRichard Purdie2013-02-061-4/+16
| | | | | | | | | In deamon mode we need to ensure the logging module is sending log data to the log file. These changes ensure this happens correctly. (Bitbake rev: bb53b47710ca4579e20284668cb354f734c3d502) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: bitbake: Always use separate process for PR ServiceRichard Purdie2013-02-061-22/+7
| | | | | | | | | | | | | | | Using the threading module interacts badly with multiprocessing used elsewhere in bitbake under certain machine loads. This was leading to bitbake hanging on Ctrl+C when the PR Server was being used. This patch converts it to always use the daemonize code which then means the threading code isn't required. [YOCTO #3742] (Bitbake rev: 2d0bbd9398ab839bd2d1e29e50b25d52efb1ce2a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: prserv: use only PRSERV_HOSTConstantin Musca2013-01-251-4/+11
| | | | | | | | | | | | | - remove PRSERV_PORT variable - use 'hostname:port' as PRSERV_HOST format - use 'localhost:0' for enabling the local PRServer [YOCTO #3744] (Bitbake rev: ad62eed9e1f0867a406e9fbfa44916a0f1ad1282) Signed-off-by: Constantin Musca <constantinx.musca@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* prserv: Do not ping PRService if not requiredLianhao Lu2012-02-011-1/+1
| | | | | | | | | | [YOCTO #1942] Ping the PRService only if PRSERV_HOST and PRSERV_PORT are set. (Bitbake rev: 20f24de0bdafac21f5d8a58701f977efa7041288) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* prserv: fix import of sqlite3Martin Jansa2012-01-201-1/+6
| | | | | | | | | | | | | | | | | | * this is used in all other bitbake parts where sqlite3 is used, don't know why it wasn't used here, but it fails e.g. on Gentoo Traceback (most recent call last): File "bin/bitbake", line 39, in <module> from bb import cooker File "lib/bb/cooker.py", line 39, in <module> import prserv.serv File "lib/prserv/serv.py", line 4, in <module> import xmlrpclib,sqlite3 ImportError: No module named sqlite3 (Bitbake rev: 9a57ec705cf5c932d8c2a35852db7a4627c57937) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake: Automatically start local PR service.Lianhao Lu2012-01-112-40/+116
| | | | | | | | | | | | | | | | | [YOCTO #1126] A local PR service will be started and stopped automatically along with the bitbake invocation/ternimation. This local PR service will be started only and if only when the PRSERV_HOST is set to 'localhost' and PRSERV_PORT is set to '0'. When started, the sqlite3 database is stored at "${PERSISTEN_DIR}/prserv.sqlite3" or "${CACHE}/prserv.sqlite3". (Bitbake rev: 9d8f45407c67ed0d3c4f820cf646de3c385067c7) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* bitbake/PRservice: Added no_hist mode and export/import.Lianhao Lu2012-01-113-70/+250
| | | | | | | | | | | | | | | | | | | | | | | | | [YOCTO #1556] 1. Added the package_arch into the index to the DB table. Because the change in PACKAGE_ARCH will results in different checksum, and it is better to have seperate PR value domains for differnt PACKAGE_ARCH of the same pakcage. 2. Changed the PR service to operate in no history mode. In this mode, the for a given query tuple (version, pkgarch, checksum), the returned value will be the largest among all the values of the same (version, pkgarch). This means the PR value returned can NOT be decremented. 3. Added export function. For each (version, pkgarch) tuple, only the record with the maximum value will be exported. 4. Added import function. The record will only be imported if the imported value is larger than the value stored in the DB with the same (version, pkgarch, checksum) tuple. (Bitbake rev: 379567ee879dcdc09a51f7f1212bde1076147a6f) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* make exception handling syntax consistentScott Garman2011-06-151-3/+3
| | | | | | | | | | Update exception handling syntax to use the modern style: except ExcType as localvar (Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d) Signed-off-by: Scott Garman <scott.a.garman@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Add PR service deamon to bitbakeLianhao Lu2011-05-273-0/+309
Added the initial implementation of the server side PR service. (Bitbake rev: 4d0e79e5591ff58ce35c7fb96f6e9217ddc27466) Signed-off-by: Lianhao Lu <lianhao.lu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-11-16 08:45:06 +0000