summaryrefslogtreecommitdiffstats
path: root/meta/lib/buildstats.py
Commit message (Collapse)AuthorAgeFilesLines
* lib: Add copyright statements to files without oneRichard Purdie2022-08-121-0/+2
| | | | | | | | | Where there isn't a copyright statement, add one to make it explicit. Also add license identifiers as MIT if there isn't one. (From OE-Core rev: bb731d1f3d2a1d50ec0aed864dbca54cf795b040) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* buildstats.py: close /proc/pressure/cpu file descriptorAryaman Gupta2022-06-301-2/+2
| | | | | | | | | | | | Use python 'with' symantics to ensure that the /proc/pressure/cpu file descriptor used in SystemStats init is closed. Previously, this would lead to a single file descriptor being leaked. For example: ResourceWarning: unclosed file <_io.BufferedReader name='/proc/pressure/cpu'> (From OE-Core rev: 643653160cd77d346cdc9b9ec25c7212c7dfe176) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* buildstats.bbclass: correct sampling of system statsAryaman Gupta2022-06-291-4/+19
| | | | | | | | | | | | | | | | | | | | | | | | The last time of sampling would be updated within the SystemStats class but not re-recorded into the datastore, leading to multiple samples being collected in the same second in the sample function of buildstats.py. Fix this to collect and store only one sample per second within a certain tolerance to deal with variation in the arrival time. This fix elimates the spikiness of sampled data, in cases where the difference between the current and the last sample is taken. Previously, since many samples per second were recorded, certain types of data would result in a very small elapsed time and hence a small numerical difference. For example, the CPU usage from /proc/stat is a running total of usage and taking the difference between data collected 0.1 seconds apart would result in usage appearing lower than it actually was. (From OE-Core rev: 0e2df45ab066bb4ad2c4f8622ee9c1a8ecdea9cb) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy MacLeod <randy.macleod@windriver.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* buildstats.py: enable collection of /proc/pressure dataAryaman Gupta2022-06-291-10/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Linux pressure monitoring system helps determine when system resources are being overutilized by measuring how contended the CPU, IO and memory are. This information can be found under /proc/pressure/ which contains 3 files - cpu, memory and io. In each of the files, the format is as follows: some avg10=70.24 avg60=68.52 avg300=69.91 total=3559632828 full avg10=57.59 avg60=58.06 avg300=60.38 total=3300487258 The "some" state of a given resource represents when one or more tasks are delayed on that resource whereas the "full" state represents when all the tasks are delayed. Currently, we only collect data from the "some" state but the "full" data can simply be appended to the log files if neccessary. The "avg10", "avg60" and "avg300" fields represent the average percentage of time runnable tasks were delayed in the last 10, 60 or 300 seconds respectively. The "total" field represents the total time, in microseconds, that some runnable task was delayed on a resource. More information can be found at: https://www.kernel.org/doc/html/latest/accounting/psi.html and in the source code under kernel/sched/psi.c This commit adds functionality to collect and log the "some" CPU, memory and IO pressure. The "avg10", "avg60" and "avg300" fields are logged without change. In place of the "total" field, the difference between the current "total" and the previous sample's "total" is logged, allowing the measurement of pressure in between each polling interval, as was done for /proc/stat data. The log files are stored in: <build_name>/tmp/buildstats/<build_time>/reduced_proc_pressure/{cpu,io,memory}.log mirroring the directory structure of /proc/pressure. If the /proc/pressure directory does not exist or the resource files can't be read/opened, the reduced_proc_pressure directory is not created. (From OE-Core rev: 061931520b8baa7f3a03bf466aa9ec8bf995bc14) Signed-off-by: Aryaman Gupta <aryaman.gupta@windriver.com> Signed-off-by: Randy MacLeod <randy.macleod@windriver.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* libc_package/buildstats: Fix python regex quoting warningsRichard Purdie2021-09-261-2/+2
| | | | | | (From OE-Core rev: 6a8077317ce12e13018ec4472f728dd24880bda9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta/lib+scripts: Convert to SPDX license headersRichard Purdie2019-05-091-0/+3
| | | | | | | | | | | | | | | | | | | | | | | This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. (From OE-Core rev: f8c9c511b5f1b7dbd45b77f345cb6c048ae6763e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* meta: remove True option to getVar callsJoshua Lock2016-12-161-2/+2
| | | | | | | | | | | | | getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* buildstats.py: skip collecting unavailable /proc dataPatrick Ohly2016-12-131-7/+12
| | | | | | | | | | | | | | | | | | | | | | Some virtualized environments like Linux-VServer do not have the entries under /proc that the new system usage sampling expected, leading to an exception when trying to open the files. Now the presence of these files is checked once before enabling the corresponding data collection. When a file is missing, the corresponding log file is not written either and pybootchart will not draw the chart that normally displays the data. Errors while reading or writing of data samples is intentionally still a fatal error, because that points towards a bigger problem that should not be ignored. Reported-by: Andreas Oberritter <obi@opendreambox.org> (From OE-Core rev: daeee2d6731014c33f0d1f8a3846830c099932b4) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* buildstats: reduce amount of data stored for system utilizationPatrick Ohly2016-12-071-10/+104
| | | | | | | | | | | | | | | | Pre-processing /proc data during the build considerably reduces the amount of data written to disk: 176KB instead of 4.7MB for a 20 minuted build. Parsing also becomes faster. The disk monitor log added another 16KB in that example build. The overall buildstat was 20MB, so the overhead for monitoring system utilization is small enough that it can be enabled by default. (From OE-Core rev: b17812385cd55e81066d3ceda92dffdc6e5564da) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* buildstats: record disk space usagePatrick Ohly2016-12-071-5/+17
| | | | | | | | | | | | | Hooks into the new monitordisk.py event and records the used space for each volume. That is probably the only relevant value when it comes to visualizing the build and recording more would only increase disk usage. (From OE-Core rev: 21a5b569370f47cc02291e1d8b76fe43faa04ea6) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* buildstats: add system state samplingPatrick Ohly2016-12-071-0/+47
/proc/[diskstats|meminfo|stat] get sampled and written to the same proc_<filename>.log files as during normal bootchat logging. This will allow rendering the CPU, disk and memory usage charts. Right now sampling happens once a second, triggered by the heartbeat event.That produces quite a bit of data for long builds, which will be addressed in a separate commit by storing the data in a more compact form. (From OE-Core rev: 6f4e8180b5b4857eaf6caf410fd3a4a41ed85930) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>