summaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMichael Opdenacker <michael.opdenacker@bootlin.com>2024-03-14 13:28:00 +0100
committerSteve Sakoman <steve@sakoman.com>2024-04-05 07:24:00 -0700
commit2b4a64396e423160ac65ae4c017b42e37e94e49d (patch)
treebef1d7557c8333cb1fa176412ece097f7c8c0fc5
parent28cd826b579931b69bb2525a9a93fe478911a331 (diff)
downloadpoky-2b4a64396e423160ac65ae4c017b42e37e94e49d.tar.gz
profile-manual: usage.rst: further style improvements
According to errors reported by "make stylecheck" (From yocto-docs rev: b3aaf4523190f7528d49c29a9aea234bb1647eae) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Steve Sakoman <steve@sakoman.com>
-rw-r--r--documentation/profile-manual/usage.rst335
-rw-r--r--documentation/styles/config/vocabularies/OpenSource/accept.txt20
-rw-r--r--documentation/styles/config/vocabularies/Yocto/accept.txt5
3 files changed, 187 insertions, 173 deletions
diff --git a/documentation/profile-manual/usage.rst b/documentation/profile-manual/usage.rst
index 28dcb969a0..542bd918b9 100644
--- a/documentation/profile-manual/usage.rst
+++ b/documentation/profile-manual/usage.rst
@@ -10,7 +10,7 @@ Basic Usage (with examples) for each of the Yocto Tracing Tools
10This chapter presents basic usage examples for each of the tracing 10This chapter presents basic usage examples for each of the tracing
11tools. 11tools.
12 12
13Perf 13perf
14==== 14====
15 15
16The perf tool is the profiling and tracing tool that comes bundled 16The perf tool is the profiling and tracing tool that comes bundled
@@ -26,12 +26,12 @@ of what's going on.
26 26
27In many ways, perf aims to be a superset of all the tracing and 27In many ways, perf aims to be a superset of all the tracing and
28profiling tools available in Linux today, including all the other tools 28profiling tools available in Linux today, including all the other tools
29covered in this HOWTO. The past couple of years have seen perf subsume a 29covered in this How-to. The past couple of years have seen perf subsume a
30lot of the functionality of those other tools and, at the same time, 30lot of the functionality of those other tools and, at the same time,
31those other tools have removed large portions of their previous 31those other tools have removed large portions of their previous
32functionality and replaced it with calls to the equivalent functionality 32functionality and replaced it with calls to the equivalent functionality
33now implemented by the perf subsystem. Extrapolation suggests that at 33now implemented by the perf subsystem. Extrapolation suggests that at
34some point those other tools will simply become completely redundant and 34some point those other tools will become completely redundant and
35go away; until then, we'll cover those other tools in these pages and in 35go away; until then, we'll cover those other tools in these pages and in
36many cases show how the same things can be accomplished in perf and the 36many cases show how the same things can be accomplished in perf and the
37other tools when it seems useful to do so. 37other tools when it seems useful to do so.
@@ -41,7 +41,7 @@ want to apply the tool; full documentation can be found either within
41the tool itself or in the manual pages at 41the tool itself or in the manual pages at
42`perf(1) <https://linux.die.net/man/1/perf>`__. 42`perf(1) <https://linux.die.net/man/1/perf>`__.
43 43
44Perf Setup 44perf Setup
45---------- 45----------
46 46
47For this section, we'll assume you've already performed the basic setup 47For this section, we'll assume you've already performed the basic setup
@@ -54,14 +54,14 @@ image built with the following in your ``local.conf`` file::
54 54
55perf runs on the target system for the most part. You can archive 55perf runs on the target system for the most part. You can archive
56profile data and copy it to the host for analysis, but for the rest of 56profile data and copy it to the host for analysis, but for the rest of
57this document we assume you've ssh'ed to the host and will be running 57this document we assume you're connected to the host through SSH and will be
58the perf commands on the target. 58running the perf commands on the target.
59 59
60Basic Perf Usage 60Basic perf Usage
61---------------- 61----------------
62 62
63The perf tool is pretty much self-documenting. To remind yourself of the 63The perf tool is pretty much self-documenting. To remind yourself of the
64available commands, simply type ``perf``, which will show you basic usage 64available commands, just type ``perf``, which will show you basic usage
65along with the available perf subcommands:: 65along with the available perf subcommands::
66 66
67 root@crownbay:~# perf 67 root@crownbay:~# perf
@@ -101,7 +101,7 @@ As a simple test case, we'll profile the ``wget`` of a fairly large file,
101which is a minimally interesting case because it has both file and 101which is a minimally interesting case because it has both file and
102network I/O aspects, and at least in the case of standard Yocto images, 102network I/O aspects, and at least in the case of standard Yocto images,
103it's implemented as part of BusyBox, so the methods we use to analyze it 103it's implemented as part of BusyBox, so the methods we use to analyze it
104can be used in a very similar way to the whole host of supported BusyBox 104can be used in a similar way to the whole host of supported BusyBox
105applets in Yocto:: 105applets in Yocto::
106 106
107 root@crownbay:~# rm linux-2.6.19.2.tar.bz2; \ 107 root@crownbay:~# rm linux-2.6.19.2.tar.bz2; \
@@ -164,17 +164,17 @@ hits and misses::
164 164
165 44.831023415 seconds time elapsed 165 44.831023415 seconds time elapsed
166 166
167So ``perf stat`` gives us a nice easy 167As you can see, ``perf stat`` gives us a nice easy
168way to get a quick overview of what might be happening for a set of 168way to get a quick overview of what might be happening for a set of
169events, but normally we'd need a little more detail in order to 169events, but normally we'd need a little more detail in order to
170understand what's going on in a way that we can act on in a useful way. 170understand what's going on in a way that we can act on in a useful way.
171 171
172To dive down into a next level of detail, we can use ``perf record`` / 172To dive down into a next level of detail, we can use ``perf record`` /
173``perf report`` which will collect profiling data and present it to use using an 173``perf report`` which will collect profiling data and present it to use using an
174interactive text-based UI (or simply as text if we specify ``--stdio`` to 174interactive text-based UI (or just as text if we specify ``--stdio`` to
175``perf report``). 175``perf report``).
176 176
177As our first attempt at profiling this workload, we'll simply run ``perf 177As our first attempt at profiling this workload, we'll just run ``perf
178record``, handing it the workload we want to profile (everything after 178record``, handing it the workload we want to profile (everything after
179``perf record`` and any perf options we hand it --- here none, will be 179``perf record`` and any perf options we hand it --- here none, will be
180executed in a new shell). perf collects samples until the process exits 180executed in a new shell). perf collects samples until the process exits
@@ -189,7 +189,7 @@ directory::
189 [ perf record: Captured and wrote 0.176 MB perf.data (~7700 samples) ] 189 [ perf record: Captured and wrote 0.176 MB perf.data (~7700 samples) ]
190 190
191To see the results in a 191To see the results in a
192"text-based UI" (tui), simply run ``perf report``, which will read the 192"text-based UI" (tui), just run ``perf report``, which will read the
193perf.data file in the current working directory and display the results 193perf.data file in the current working directory and display the results
194in an interactive UI:: 194in an interactive UI::
195 195
@@ -203,10 +203,10 @@ The above screenshot displays a "flat" profile, one entry for each
203profiling run, ordered from the most popular to the least (perf has 203profiling run, ordered from the most popular to the least (perf has
204options to sort in various orders and keys as well as display entries 204options to sort in various orders and keys as well as display entries
205only above a certain threshold and so on --- see the perf documentation 205only above a certain threshold and so on --- see the perf documentation
206for details). Note that this includes both userspace functions (entries 206for details). Note that this includes both user space functions (entries
207containing a ``[.]``) and kernel functions accounted to the process (entries 207containing a ``[.]``) and kernel functions accounted to the process (entries
208containing a ``[k]``). perf has command-line modifiers that can be used to 208containing a ``[k]``). perf has command-line modifiers that can be used to
209restrict the profiling to kernel or userspace, among others. 209restrict the profiling to kernel or user space, among others.
210 210
211Notice also that the above report shows an entry for ``busybox``, which is 211Notice also that the above report shows an entry for ``busybox``, which is
212the executable that implements ``wget`` in Yocto, but that instead of a 212the executable that implements ``wget`` in Yocto, but that instead of a
@@ -217,7 +217,7 @@ Before we do that, however, let's try running a different profile, one
217which shows something a little more interesting. The only difference 217which shows something a little more interesting. The only difference
218between the new profile and the previous one is that we'll add the ``-g`` 218between the new profile and the previous one is that we'll add the ``-g``
219option, which will record not just the address of a sampled function, 219option, which will record not just the address of a sampled function,
220but the entire callchain to the sampled function as well:: 220but the entire call chain to the sampled function as well::
221 221
222 root@crownbay:~# perf record -g wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2 222 root@crownbay:~# perf record -g wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2
223 Connecting to downloads.yoctoproject.org (140.211.169.59:80) 223 Connecting to downloads.yoctoproject.org (140.211.169.59:80)
@@ -231,26 +231,26 @@ but the entire callchain to the sampled function as well::
231.. image:: figures/perf-wget-g-copy-to-user-expanded-stripped.png 231.. image:: figures/perf-wget-g-copy-to-user-expanded-stripped.png
232 :align: center 232 :align: center
233 233
234Using the callgraph view, we can actually see not only which functions 234Using the call graph view, we can actually see not only which functions
235took the most time, but we can also see a summary of how those functions 235took the most time, but we can also see a summary of how those functions
236were called and learn something about how the program interacts with the 236were called and learn something about how the program interacts with the
237kernel in the process. 237kernel in the process.
238 238
239Notice that each entry in the above screenshot now contains a ``+`` on the 239Notice that each entry in the above screenshot now contains a ``+`` on the
240left-hand side. This means that we can expand the entry and drill down 240left side. This means that we can expand the entry and drill down
241into the callchains that feed into that entry. Pressing ``Enter`` on any 241into the call chains that feed into that entry. Pressing ``Enter`` on any
242one of them will expand the callchain (you can also press ``E`` to expand 242one of them will expand the call chain (you can also press ``E`` to expand
243them all at the same time or ``C`` to collapse them all). 243them all at the same time or ``C`` to collapse them all).
244 244
245In the screenshot above, we've toggled the ``__copy_to_user_ll()`` entry 245In the screenshot above, we've toggled the ``__copy_to_user_ll()`` entry
246and several subnodes all the way down. This lets us see which callchains 246and several subnodes all the way down. This lets us see which call chains
247contributed to the profiled ``__copy_to_user_ll()`` function which 247contributed to the profiled ``__copy_to_user_ll()`` function which
248contributed 1.77% to the total profile. 248contributed 1.77% to the total profile.
249 249
250As a bit of background explanation for these callchains, think about 250As a bit of background explanation for these call chains, think about
251what happens at a high level when you run wget to get a file out on the 251what happens at a high level when you run ``wget`` to get a file out on the
252network. Basically what happens is that the data comes into the kernel 252network. Basically what happens is that the data comes into the kernel
253via the network connection (socket) and is passed to the userspace 253via the network connection (socket) and is passed to the user space
254program ``wget`` (which is actually a part of BusyBox, but that's not 254program ``wget`` (which is actually a part of BusyBox, but that's not
255important for now), which takes the buffers the kernel passes to it and 255important for now), which takes the buffers the kernel passes to it and
256writes it to a disk file to save it. 256writes it to a disk file to save it.
@@ -260,15 +260,15 @@ is the part where the kernel passes the data it has read from the socket
260down to wget i.e. a ``copy-to-user``. 260down to wget i.e. a ``copy-to-user``.
261 261
262Notice also that here there's also a case where the hex value is 262Notice also that here there's also a case where the hex value is
263displayed in the callstack, here in the expanded ``sys_clock_gettime()`` 263displayed in the call stack, here in the expanded ``sys_clock_gettime()``
264function. Later we'll see it resolve to a userspace function call in 264function. Later we'll see it resolve to a user space function call in
265busybox. 265BusyBox.
266 266
267.. image:: figures/perf-wget-g-copy-from-user-expanded-stripped.png 267.. image:: figures/perf-wget-g-copy-from-user-expanded-stripped.png
268 :align: center 268 :align: center
269 269
270The above screenshot shows the other half of the journey for the data --- 270The above screenshot shows the other half of the journey for the data ---
271from the ``wget`` program's userspace buffers to disk. To get the buffers to 271from the ``wget`` program's user space buffers to disk. To get the buffers to
272disk, the wget program issues a ``write(2)``, which does a ``copy-from-user`` to 272disk, the wget program issues a ``write(2)``, which does a ``copy-from-user`` to
273the kernel, which then takes care via some circuitous path (probably 273the kernel, which then takes care via some circuitous path (probably
274also present somewhere in the profile data), to get it safely to disk. 274also present somewhere in the profile data), to get it safely to disk.
@@ -278,8 +278,8 @@ of how to extract useful information out of it, let's get back to the
278task at hand and see if we can get some basic idea about where the time 278task at hand and see if we can get some basic idea about where the time
279is spent in the program we're profiling, wget. Remember that wget is 279is spent in the program we're profiling, wget. Remember that wget is
280actually implemented as an applet in BusyBox, so while the process name 280actually implemented as an applet in BusyBox, so while the process name
281is ``wget``, the executable we're actually interested in is BusyBox. So 281is ``wget``, the executable we're actually interested in is ``busybox``.
282let's expand the first entry containing BusyBox: 282Therefore, let's expand the first entry containing BusyBox:
283 283
284.. image:: figures/perf-wget-busybox-expanded-stripped.png 284.. image:: figures/perf-wget-busybox-expanded-stripped.png
285 :align: center 285 :align: center
@@ -289,7 +289,7 @@ hex value instead of a symbol as with most of the kernel entries.
289Expanding the BusyBox entry doesn't make it any better. 289Expanding the BusyBox entry doesn't make it any better.
290 290
291The problem is that perf can't find the symbol information for the 291The problem is that perf can't find the symbol information for the
292busybox binary, which is actually stripped out by the Yocto build 292``busybox`` binary, which is actually stripped out by the Yocto build
293system. 293system.
294 294
295One way around that is to put the following in your ``local.conf`` file 295One way around that is to put the following in your ``local.conf`` file
@@ -299,20 +299,20 @@ when you build the image::
299 299
300However, we already have an image with the binaries stripped, so 300However, we already have an image with the binaries stripped, so
301what can we do to get perf to resolve the symbols? Basically we need to 301what can we do to get perf to resolve the symbols? Basically we need to
302install the debuginfo for the BusyBox package. 302install the debugging information for the BusyBox package.
303 303
304To generate the debug info for the packages in the image, we can add 304To generate the debug info for the packages in the image, we can add
305``dbg-pkgs`` to :term:`EXTRA_IMAGE_FEATURES` in ``local.conf``. For example:: 305``dbg-pkgs`` to :term:`EXTRA_IMAGE_FEATURES` in ``local.conf``. For example::
306 306
307 EXTRA_IMAGE_FEATURES = "debug-tweaks tools-profile dbg-pkgs" 307 EXTRA_IMAGE_FEATURES = "debug-tweaks tools-profile dbg-pkgs"
308 308
309Additionally, in order to generate the type of debuginfo that perf 309Additionally, in order to generate the type of debugging information that perf
310understands, we also need to set :term:`PACKAGE_DEBUG_SPLIT_STYLE` 310understands, we also need to set :term:`PACKAGE_DEBUG_SPLIT_STYLE`
311in the ``local.conf`` file:: 311in the ``local.conf`` file::
312 312
313 PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory' 313 PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory'
314 314
315Once we've done that, we can install the debuginfo for BusyBox. The 315Once we've done that, we can install the debugging information for BusyBox. The
316debug packages once built can be found in ``build/tmp/deploy/rpm/*`` 316debug packages once built can be found in ``build/tmp/deploy/rpm/*``
317on the host system. Find the ``busybox-dbg-...rpm`` file and copy it 317on the host system. Find the ``busybox-dbg-...rpm`` file and copy it
318to the target. For example:: 318to the target. For example::
@@ -320,11 +320,11 @@ to the target. For example::
320 [trz@empanada core2]$ scp /home/trz/yocto/crownbay-tracing-dbg/build/tmp/deploy/rpm/core2_32/busybox-dbg-1.20.2-r2.core2_32.rpm root@192.168.1.31: 320 [trz@empanada core2]$ scp /home/trz/yocto/crownbay-tracing-dbg/build/tmp/deploy/rpm/core2_32/busybox-dbg-1.20.2-r2.core2_32.rpm root@192.168.1.31:
321 busybox-dbg-1.20.2-r2.core2_32.rpm 100% 1826KB 1.8MB/s 00:01 321 busybox-dbg-1.20.2-r2.core2_32.rpm 100% 1826KB 1.8MB/s 00:01
322 322
323Now install the debug rpm on the target:: 323Now install the debug RPM on the target::
324 324
325 root@crownbay:~# rpm -i busybox-dbg-1.20.2-r2.core2_32.rpm 325 root@crownbay:~# rpm -i busybox-dbg-1.20.2-r2.core2_32.rpm
326 326
327Now that the debuginfo is installed, we see that the BusyBox entries now display 327Now that the debugging information is installed, we see that the BusyBox entries now display
328their functions symbolically: 328their functions symbolically:
329 329
330.. image:: figures/perf-wget-busybox-debuginfo.png 330.. image:: figures/perf-wget-busybox-debuginfo.png
@@ -344,7 +344,7 @@ expanded all the nodes using the ``E`` key):
344.. image:: figures/perf-wget-busybox-dso-zoom.png 344.. image:: figures/perf-wget-busybox-dso-zoom.png
345 :align: center 345 :align: center
346 346
347Finally, we can see that now that the BusyBox debuginfo is installed, 347Finally, we can see that now that the BusyBox debugging information is installed,
348the previously unresolved symbol in the ``sys_clock_gettime()`` entry 348the previously unresolved symbol in the ``sys_clock_gettime()`` entry
349mentioned previously is now resolved, and shows that the 349mentioned previously is now resolved, and shows that the
350``sys_clock_gettime`` system call that was the source of 6.75% of the 350``sys_clock_gettime`` system call that was the source of 6.75% of the
@@ -376,8 +376,8 @@ counter, something other than the default ``cycles``.
376The tracing and profiling infrastructure in Linux has become unified in 376The tracing and profiling infrastructure in Linux has become unified in
377a way that allows us to use the same tool with a completely different 377a way that allows us to use the same tool with a completely different
378set of counters, not just the standard hardware counters that 378set of counters, not just the standard hardware counters that
379traditional tools have had to restrict themselves to (of course the 379traditional tools have had to restrict themselves to (the
380traditional tools can also make use of the expanded possibilities now 380traditional tools can now actually make use of the expanded possibilities now
381available to them, and in some cases have, as mentioned previously). 381available to them, and in some cases have, as mentioned previously).
382 382
383We can get a list of the available events that can be used to profile a 383We can get a list of the available events that can be used to profile a
@@ -517,14 +517,14 @@ workload via ``perf list``::
517.. admonition:: Tying it Together 517.. admonition:: Tying it Together
518 518
519 These are exactly the same set of events defined by the trace event 519 These are exactly the same set of events defined by the trace event
520 subsystem and exposed by ftrace / tracecmd / kernelshark as files in 520 subsystem and exposed by ftrace / trace-cmd / KernelShark as files in
521 ``/sys/kernel/debug/tracing/events``, by SystemTap as 521 ``/sys/kernel/debug/tracing/events``, by SystemTap as
522 kernel.trace("tracepoint_name") and (partially) accessed by LTTng. 522 kernel.trace("tracepoint_name") and (partially) accessed by LTTng.
523 523
524Only a subset of these would be of interest to us when looking at this 524Only a subset of these would be of interest to us when looking at this
525workload, so let's choose the most likely subsystems (identified by the 525workload, so let's choose the most likely subsystems (identified by the
526string before the colon in the Tracepoint events) and do a ``perf stat`` 526string before the colon in the ``Tracepoint`` events) and do a ``perf stat``
527run using only those wildcarded subsystems:: 527run using only those subsystem wildcards::
528 528
529 root@crownbay:~# perf stat -e skb:* -e net:* -e napi:* -e sched:* -e workqueue:* -e irq:* -e syscalls:* wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2 529 root@crownbay:~# perf stat -e skb:* -e net:* -e napi:* -e sched:* -e workqueue:* -e irq:* -e syscalls:* wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2
530 Performance counter stats for 'wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2': 530 Performance counter stats for 'wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2':
@@ -614,8 +614,8 @@ accounts for the function name actually displayed in the profile:
614 } 614 }
615 615
616A couple of the more interesting 616A couple of the more interesting
617callchains are expanded and displayed above, basically some network 617call chains are expanded and displayed above, basically some network
618receive paths that presumably end up waking up wget (busybox) when 618receive paths that presumably end up waking up wget (BusyBox) when
619network data is ready. 619network data is ready.
620 620
621Note that because tracepoints are normally used for tracing, the default 621Note that because tracepoints are normally used for tracing, the default
@@ -635,7 +635,7 @@ high-level view of what's going on with a workload or across the system.
635It is however by definition an approximation, as suggested by the most 635It is however by definition an approximation, as suggested by the most
636prominent word associated with it, ``sampling``. On the one hand, it 636prominent word associated with it, ``sampling``. On the one hand, it
637allows a representative picture of what's going on in the system to be 637allows a representative picture of what's going on in the system to be
638cheaply taken, but on the other hand, that cheapness limits its utility 638cheaply taken, but alternatively, that cheapness limits its utility
639when that data suggests a need to "dive down" more deeply to discover 639when that data suggests a need to "dive down" more deeply to discover
640what's really going on. In such cases, the only way to see what's really 640what's really going on. In such cases, the only way to see what's really
641going on is to be able to look at (or summarize more intelligently) the 641going on is to be able to look at (or summarize more intelligently) the
@@ -700,7 +700,7 @@ an infinite variety of ways.
700Another way to look at it is that there are only so many ways that the 700Another way to look at it is that there are only so many ways that the
701'primitive' counters can be used on their own to generate interesting 701'primitive' counters can be used on their own to generate interesting
702output; to get anything more complicated than simple counts requires 702output; to get anything more complicated than simple counts requires
703some amount of additional logic, which is typically very specific to the 703some amount of additional logic, which is typically specific to the
704problem at hand. For example, if we wanted to make use of a 'counter' 704problem at hand. For example, if we wanted to make use of a 'counter'
705that maps to the value of the time difference between when a process was 705that maps to the value of the time difference between when a process was
706scheduled to run on a processor and the time it actually ran, we 706scheduled to run on a processor and the time it actually ran, we
@@ -711,12 +711,12 @@ standard profiling tools how much data every process on the system reads
711and writes, along with how many of those reads and writes fail 711and writes, along with how many of those reads and writes fail
712completely. If we have sufficient trace data, however, we could with the 712completely. If we have sufficient trace data, however, we could with the
713right tools easily extract and present that information, but we'd need 713right tools easily extract and present that information, but we'd need
714something other than pre-canned profiling tools to do that. 714something other than ready-made profiling tools to do that.
715 715
716Luckily, there is a general-purpose way to handle such needs, called 716Luckily, there is a general-purpose way to handle such needs, called
717"programming languages". Making programming languages easily available 717"programming languages". Making programming languages easily available
718to apply to such problems given the specific format of data is called a 718to apply to such problems given the specific format of data is called a
719'programming language binding' for that data and language. Perf supports 719'programming language binding' for that data and language. perf supports
720two programming language bindings, one for Python and one for Perl. 720two programming language bindings, one for Python and one for Perl.
721 721
722.. admonition:: Tying it Together 722.. admonition:: Tying it Together
@@ -726,7 +726,7 @@ two programming language bindings, one for Python and one for Perl.
726 DProbes dpcc compiler, an ANSI C compiler which targeted a low-level 726 DProbes dpcc compiler, an ANSI C compiler which targeted a low-level
727 assembly language running on an in-kernel interpreter on the target 727 assembly language running on an in-kernel interpreter on the target
728 system. This is exactly analogous to what Sun's DTrace did, except 728 system. This is exactly analogous to what Sun's DTrace did, except
729 that DTrace invented its own language for the purpose. Systemtap, 729 that DTrace invented its own language for the purpose. SystemTap,
730 heavily inspired by DTrace, also created its own one-off language, 730 heavily inspired by DTrace, also created its own one-off language,
731 but rather than running the product on an in-kernel interpreter, 731 but rather than running the product on an in-kernel interpreter,
732 created an elaborate compiler-based machinery to translate its 732 created an elaborate compiler-based machinery to translate its
@@ -739,8 +739,8 @@ entry / exit events we recorded::
739 root@crownbay:~# perf script -g python 739 root@crownbay:~# perf script -g python
740 generated Python script: perf-script.py 740 generated Python script: perf-script.py
741 741
742The skeleton script simply creates a Python function for each event type in the 742The skeleton script just creates a Python function for each event type in the
743``perf.data`` file. The body of each function simply prints the event name along 743``perf.data`` file. The body of each function just prints the event name along
744with its parameters. For example: 744with its parameters. For example:
745 745
746.. code-block:: python 746.. code-block:: python
@@ -783,7 +783,7 @@ We can run that script directly to print all of the events contained in the
783 syscalls__sys_exit_read 1 11624.859944032 1262 wget nr=3, ret=1024 783 syscalls__sys_exit_read 1 11624.859944032 1262 wget nr=3, ret=1024
784 784
785That in itself isn't very useful; after all, we can accomplish pretty much the 785That in itself isn't very useful; after all, we can accomplish pretty much the
786same thing by simply running ``perf script`` without arguments in the same 786same thing by just running ``perf script`` without arguments in the same
787directory as the ``perf.data`` file. 787directory as the ``perf.data`` file.
788 788
789We can however replace the print statements in the generated function 789We can however replace the print statements in the generated function
@@ -805,7 +805,7 @@ event. For example:
805 805
806Each event handler function in the generated code 806Each event handler function in the generated code
807is modified to do this. For convenience, we define a common function 807is modified to do this. For convenience, we define a common function
808called ``inc_counts()`` that each handler calls; ``inc_counts()`` simply tallies 808called ``inc_counts()`` that each handler calls; ``inc_counts()`` just tallies
809a count for each event using the ``counts`` hash, which is a specialized 809a count for each event using the ``counts`` hash, which is a specialized
810hash function that does Perl-like autovivification, a capability that's 810hash function that does Perl-like autovivification, a capability that's
811extremely useful for kinds of multi-level aggregation commonly used in 811extremely useful for kinds of multi-level aggregation commonly used in
@@ -865,7 +865,7 @@ System-Wide Tracing and Profiling
865~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 865~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
866 866
867The examples so far have focused on tracing a particular program or 867The examples so far have focused on tracing a particular program or
868workload --- in other words, every profiling run has specified the program 868workload --- that is, every profiling run has specified the program
869to profile in the command-line e.g. ``perf record wget ...``. 869to profile in the command-line e.g. ``perf record wget ...``.
870 870
871It's also possible, and more interesting in many cases, to run a 871It's also possible, and more interesting in many cases, to run a
@@ -894,13 +894,13 @@ other processes running on the system as well:
894.. image:: figures/perf-systemwide.png 894.. image:: figures/perf-systemwide.png
895 :align: center 895 :align: center
896 896
897In the snapshot above, we can see callchains that originate in libc, and 897In the snapshot above, we can see call chains that originate in ``libc``, and
898a callchain from Xorg that demonstrates that we're using a proprietary X 898a call chain from ``Xorg`` that demonstrates that we're using a proprietary X
899driver in userspace (notice the presence of ``PVR`` and some other 899driver in user space (notice the presence of ``PVR`` and some other
900unresolvable symbols in the expanded Xorg callchain). 900unresolvable symbols in the expanded ``Xorg`` call chain).
901 901
902Note also that we have both kernel and userspace entries in the above 902Note also that we have both kernel and user space entries in the above
903snapshot. We can also tell perf to focus on userspace but providing a 903snapshot. We can also tell perf to focus on user space but providing a
904modifier, in this case ``u``, to the ``cycles`` hardware counter when we 904modifier, in this case ``u``, to the ``cycles`` hardware counter when we
905record a profile:: 905record a profile::
906 906
@@ -911,7 +911,7 @@ record a profile::
911.. image:: figures/perf-report-cycles-u.png 911.. image:: figures/perf-report-cycles-u.png
912 :align: center 912 :align: center
913 913
914Notice in the screenshot above, we see only userspace entries (``[.]``) 914Notice in the screenshot above, we see only user space entries (``[.]``)
915 915
916Finally, we can press ``Enter`` on a leaf node and select the ``Zoom into 916Finally, we can press ``Enter`` on a leaf node and select the ``Zoom into
917DSO`` menu item to show only entries associated with a specific DSO. In 917DSO`` menu item to show only entries associated with a specific DSO. In
@@ -946,7 +946,7 @@ We can look at the raw output using ``perf script`` with no arguments::
946Filtering 946Filtering
947^^^^^^^^^ 947^^^^^^^^^
948 948
949Notice that there are a lot of events that don't really have anything to 949Notice that there are many events that don't really have anything to
950do with what we're interested in, namely events that schedule ``perf`` 950do with what we're interested in, namely events that schedule ``perf``
951itself in and out or that wake perf up. We can get rid of those by using 951itself in and out or that wake perf up. We can get rid of those by using
952the ``--filter`` option --- for each event we specify using ``-e``, we can add a 952the ``--filter`` option --- for each event we specify using ``-e``, we can add a
@@ -985,7 +985,7 @@ purpose of demonstrating how to use filters, it's close enough.
985.. admonition:: Tying it Together 985.. admonition:: Tying it Together
986 986
987 These are exactly the same set of event filters defined by the trace 987 These are exactly the same set of event filters defined by the trace
988 event subsystem. See the ftrace / tracecmd / kernelshark section for more 988 event subsystem. See the ftrace / trace-cmd / KernelShark section for more
989 discussion about these event filters. 989 discussion about these event filters.
990 990
991.. admonition:: Tying it Together 991.. admonition:: Tying it Together
@@ -995,14 +995,14 @@ purpose of demonstrating how to use filters, it's close enough.
995 indispensable part of the perf design as it relates to tracing. 995 indispensable part of the perf design as it relates to tracing.
996 kernel-based event filters provide a mechanism to precisely throttle 996 kernel-based event filters provide a mechanism to precisely throttle
997 the event stream that appears in user space, where it makes sense to 997 the event stream that appears in user space, where it makes sense to
998 provide bindings to real programming languages for postprocessing the 998 provide bindings to real programming languages for post-processing the
999 event stream. This architecture allows for the intelligent and 999 event stream. This architecture allows for the intelligent and
1000 flexible partitioning of processing between the kernel and user 1000 flexible partitioning of processing between the kernel and user
1001 space. Contrast this with other tools such as SystemTap, which does 1001 space. Contrast this with other tools such as SystemTap, which does
1002 all of its processing in the kernel and as such requires a special 1002 all of its processing in the kernel and as such requires a special
1003 project-defined language in order to accommodate that design, or 1003 project-defined language in order to accommodate that design, or
1004 LTTng, where everything is sent to userspace and as such requires a 1004 LTTng, where everything is sent to user space and as such requires a
1005 super-efficient kernel-to-userspace transport mechanism in order to 1005 super-efficient kernel-to-user space transport mechanism in order to
1006 function properly. While perf certainly can benefit from for instance 1006 function properly. While perf certainly can benefit from for instance
1007 advances in the design of the transport, it doesn't fundamentally 1007 advances in the design of the transport, it doesn't fundamentally
1008 depend on them. Basically, if you find that your perf tracing 1008 depend on them. Basically, if you find that your perf tracing
@@ -1014,7 +1014,7 @@ Using Dynamic Tracepoints
1014 1014
1015perf isn't restricted to the fixed set of static tracepoints listed by 1015perf isn't restricted to the fixed set of static tracepoints listed by
1016``perf list``. Users can also add their own "dynamic" tracepoints anywhere 1016``perf list``. Users can also add their own "dynamic" tracepoints anywhere
1017in the kernel. For instance, suppose we want to define our own 1017in the kernel. For example, suppose we want to define our own
1018tracepoint on ``do_fork()``. We can do that using the ``perf probe`` perf 1018tracepoint on ``do_fork()``. We can do that using the ``perf probe`` perf
1019subcommand:: 1019subcommand::
1020 1020
@@ -1069,7 +1069,7 @@ up after 30 seconds)::
1069 [ perf record: Woken up 1 times to write data ] 1069 [ perf record: Woken up 1 times to write data ]
1070 [ perf record: Captured and wrote 0.087 MB perf.data (~3812 samples) ] 1070 [ perf record: Captured and wrote 0.087 MB perf.data (~3812 samples) ]
1071 1071
1072Using ``perf script`` we can see each do_fork event that fired:: 1072Using ``perf script`` we can see each ``do_fork`` event that fired::
1073 1073
1074 root@crownbay:~# perf script 1074 root@crownbay:~# perf script
1075 1075
@@ -1111,7 +1111,7 @@ Using ``perf script`` we can see each do_fork event that fired::
1111 gaku 1312 [000] 34237.202388: do_fork: (c1028460) 1111 gaku 1312 [000] 34237.202388: do_fork: (c1028460)
1112 1112
1113And using ``perf report`` on the same file, we can see the 1113And using ``perf report`` on the same file, we can see the
1114callgraphs from starting a few programs during those 30 seconds: 1114call graphs from starting a few programs during those 30 seconds:
1115 1115
1116.. image:: figures/perf-probe-do_fork-profile.png 1116.. image:: figures/perf-probe-do_fork-profile.png
1117 :align: center 1117 :align: center
@@ -1125,11 +1125,11 @@ callgraphs from starting a few programs during those 30 seconds:
1125 1125
1126.. admonition:: Tying it Together 1126.. admonition:: Tying it Together
1127 1127
1128 Dynamic tracepoints are implemented under the covers by kprobes and 1128 Dynamic tracepoints are implemented under the covers by Kprobes and
1129 uprobes. kprobes and uprobes are also used by and in fact are the 1129 Uprobes. Kprobes and Uprobes are also used by and in fact are the
1130 main focus of SystemTap. 1130 main focus of SystemTap.
1131 1131
1132Perf Documentation 1132perf Documentation
1133------------------ 1133------------------
1134 1134
1135Online versions of the manual pages for the commands discussed in this 1135Online versions of the manual pages for the commands discussed in this
@@ -1153,7 +1153,7 @@ section can be found here:
1153 1153
1154- The top-level `perf(1) manual page <https://linux.die.net/man/1/perf>`__. 1154- The top-level `perf(1) manual page <https://linux.die.net/man/1/perf>`__.
1155 1155
1156Normally, you should be able to invoke the manual pages via perf itself 1156Normally, you should be able to open the manual pages via perf itself
1157e.g. ``perf help`` or ``perf help record``. 1157e.g. ``perf help`` or ``perf help record``.
1158 1158
1159To have the perf manual pages installed on your target, modify your 1159To have the perf manual pages installed on your target, modify your
@@ -1168,14 +1168,14 @@ of examples, can also be found in the ``perf`` directory of the kernel tree::
1168 tools/perf/Documentation 1168 tools/perf/Documentation
1169 1169
1170There's also a nice perf tutorial on the perf 1170There's also a nice perf tutorial on the perf
1171wiki that goes into more detail than we do here in certain areas: `Perf 1171wiki that goes into more detail than we do here in certain areas: `perf
1172Tutorial <https://perf.wiki.kernel.org/index.php/Tutorial>`__ 1172Tutorial <https://perf.wiki.kernel.org/index.php/Tutorial>`__
1173 1173
1174ftrace 1174ftrace
1175====== 1175======
1176 1176
1177"ftrace" literally refers to the "ftrace function tracer" but in reality 1177"ftrace" literally refers to the "ftrace function tracer" but in reality
1178this encompasses a number of related tracers along with the 1178this encompasses several related tracers along with the
1179infrastructure that they all make use of. 1179infrastructure that they all make use of.
1180 1180
1181ftrace Setup 1181ftrace Setup
@@ -1184,11 +1184,11 @@ ftrace Setup
1184For this section, we'll assume you've already performed the basic setup 1184For this section, we'll assume you've already performed the basic setup
1185outlined in the ":ref:`profile-manual/intro:General Setup`" section. 1185outlined in the ":ref:`profile-manual/intro:General Setup`" section.
1186 1186
1187ftrace, trace-cmd, and kernelshark run on the target system, and are 1187ftrace, trace-cmd, and KernelShark run on the target system, and are
1188ready to go out-of-the-box --- no additional setup is necessary. For the 1188ready to go out-of-the-box --- no additional setup is necessary. For the
1189rest of this section we assume you've ssh'ed to the host and will be 1189rest of this section we assume you're connected to the host through SSH and
1190running ftrace on the target. kernelshark is a GUI application and if 1190will be running ftrace on the target. KernelShark is a GUI application and if
1191you use the ``-X`` option to ssh you can have the kernelshark GUI run on 1191you use the ``-X`` option to ``ssh`` you can have the KernelShark GUI run on
1192the target but display remotely on the host if you want. 1192the target but display remotely on the host if you want.
1193 1193
1194Basic ftrace usage 1194Basic ftrace usage
@@ -1196,8 +1196,8 @@ Basic ftrace usage
1196 1196
1197"ftrace" essentially refers to everything included in the ``/tracing`` 1197"ftrace" essentially refers to everything included in the ``/tracing``
1198directory of the mounted debugfs filesystem (Yocto follows the standard 1198directory of the mounted debugfs filesystem (Yocto follows the standard
1199convention and mounts it at ``/sys/kernel/debug``). Here's a listing of all 1199convention and mounts it at ``/sys/kernel/debug``). All the files found in
1200the files found in ``/sys/kernel/debug/tracing`` on a Yocto system:: 1200``/sys/kernel/debug/tracing`` on a Yocto system are::
1201 1201
1202 root@sugarbay:/sys/kernel/debug/tracing# ls 1202 root@sugarbay:/sys/kernel/debug/tracing# ls
1203 README kprobe_events trace 1203 README kprobe_events trace
@@ -1222,7 +1222,7 @@ the ftrace documentation.
1222 1222
1223We'll start by looking at some of the available built-in tracers. 1223We'll start by looking at some of the available built-in tracers.
1224 1224
1225cat'ing the ``available_tracers`` file lists the set of available tracers:: 1225The ``available_tracers`` file lists the set of available tracers::
1226 1226
1227 root@sugarbay:/sys/kernel/debug/tracing# cat available_tracers 1227 root@sugarbay:/sys/kernel/debug/tracing# cat available_tracers
1228 blk function_graph function nop 1228 blk function_graph function nop
@@ -1232,11 +1232,11 @@ The ``current_tracer`` file contains the tracer currently in effect::
1232 root@sugarbay:/sys/kernel/debug/tracing# cat current_tracer 1232 root@sugarbay:/sys/kernel/debug/tracing# cat current_tracer
1233 nop 1233 nop
1234 1234
1235The above listing of current_tracer shows that the 1235The above listing of ``current_tracer`` shows that the
1236``nop`` tracer is in effect, which is just another way of saying that 1236``nop`` tracer is in effect, which is just another way of saying that
1237there's actually no tracer currently in effect. 1237there's actually no tracer currently in effect.
1238 1238
1239echo'ing one of the available_tracers into ``current_tracer`` makes the 1239Writing one of the available tracers into ``current_tracer`` makes the
1240specified tracer the current tracer:: 1240specified tracer the current tracer::
1241 1241
1242 root@sugarbay:/sys/kernel/debug/tracing# echo function > current_tracer 1242 root@sugarbay:/sys/kernel/debug/tracing# echo function > current_tracer
@@ -1292,7 +1292,7 @@ tracer::
1292 . 1292 .
1293 1293
1294Each line in the trace above shows what was happening in the kernel on a given 1294Each line in the trace above shows what was happening in the kernel on a given
1295cpu, to the level of detail of function calls. Each entry shows the function 1295CPU, to the level of detail of function calls. Each entry shows the function
1296called, followed by its caller (after the arrow). 1296called, followed by its caller (after the arrow).
1297 1297
1298The function tracer gives you an extremely detailed idea of what the 1298The function tracer gives you an extremely detailed idea of what the
@@ -1306,7 +1306,7 @@ great way to learn about how the kernel code works in a dynamic sense.
1306 1306
1307It is a little more difficult to follow the call chains than it needs to 1307It is a little more difficult to follow the call chains than it needs to
1308be --- luckily there's a variant of the function tracer that displays the 1308be --- luckily there's a variant of the function tracer that displays the
1309callchains explicitly, called the ``function_graph`` tracer:: 1309call chains explicitly, called the ``function_graph`` tracer::
1310 1310
1311 root@sugarbay:/sys/kernel/debug/tracing# echo function_graph > current_tracer 1311 root@sugarbay:/sys/kernel/debug/tracing# echo function_graph > current_tracer
1312 root@sugarbay:/sys/kernel/debug/tracing# cat trace | less 1312 root@sugarbay:/sys/kernel/debug/tracing# cat trace | less
@@ -1425,7 +1425,7 @@ As you can see, the ``function_graph`` display is much easier
1425to follow. Also note that in addition to the function calls and 1425to follow. Also note that in addition to the function calls and
1426associated braces, other events such as scheduler events are displayed 1426associated braces, other events such as scheduler events are displayed
1427in context. In fact, you can freely include any tracepoint available in 1427in context. In fact, you can freely include any tracepoint available in
1428the trace events subsystem described in the next section by simply 1428the trace events subsystem described in the next section by just
1429enabling those events, and they'll appear in context in the function 1429enabling those events, and they'll appear in context in the function
1430graph display. Quite a powerful tool for understanding kernel dynamics. 1430graph display. Quite a powerful tool for understanding kernel dynamics.
1431 1431
@@ -1528,7 +1528,7 @@ The ``format`` file for the
1528tracepoint describes the event in memory, which is used by the various 1528tracepoint describes the event in memory, which is used by the various
1529tracing tools that now make use of these tracepoint to parse the event 1529tracing tools that now make use of these tracepoint to parse the event
1530and make sense of it, along with a ``print fmt`` field that allows tools 1530and make sense of it, along with a ``print fmt`` field that allows tools
1531like ftrace to display the event as text. Here's what the format of the 1531like ftrace to display the event as text. The format of the
1532``kmalloc`` event looks like:: 1532``kmalloc`` event looks like::
1533 1533
1534 root@sugarbay:/sys/kernel/debug/tracing/events/kmem/kmalloc# cat format 1534 root@sugarbay:/sys/kernel/debug/tracing/events/kmem/kmalloc# cat format
@@ -1581,7 +1581,7 @@ events in the output buffer::
1581 root@sugarbay:/sys/kernel/debug/tracing# echo 1 > tracing_on 1581 root@sugarbay:/sys/kernel/debug/tracing# echo 1 > tracing_on
1582 1582
1583Now, if we look at the ``trace`` file, we see nothing 1583Now, if we look at the ``trace`` file, we see nothing
1584but the kmalloc events we just turned on:: 1584but the ``kmalloc`` events we just turned on::
1585 1585
1586 root@sugarbay:/sys/kernel/debug/tracing# cat trace | less 1586 root@sugarbay:/sys/kernel/debug/tracing# cat trace | less
1587 # tracer: nop 1587 # tracer: nop
@@ -1636,8 +1636,8 @@ using the ``enable`` file in the subsystem directory) and get an
1636arbitrarily fine-grained idea of what's going on in the system by 1636arbitrarily fine-grained idea of what's going on in the system by
1637enabling as many of the appropriate tracepoints as applicable. 1637enabling as many of the appropriate tracepoints as applicable.
1638 1638
1639A number of the tools described in this HOWTO do just that, including 1639Several tools described in this How-to do just that, including
1640``trace-cmd`` and kernelshark in the next section. 1640``trace-cmd`` and KernelShark in the next section.
1641 1641
1642.. admonition:: Tying it Together 1642.. admonition:: Tying it Together
1643 1643
@@ -1653,7 +1653,7 @@ A number of the tools described in this HOWTO do just that, including
1653 ``/sys/kernel/debug/tracing`` will be removed and replaced with 1653 ``/sys/kernel/debug/tracing`` will be removed and replaced with
1654 equivalent tracers based on the "trace events" subsystem. 1654 equivalent tracers based on the "trace events" subsystem.
1655 1655
1656trace-cmd / kernelshark 1656trace-cmd / KernelShark
1657----------------------- 1657-----------------------
1658 1658
1659trace-cmd is essentially an extensive command-line "wrapper" interface 1659trace-cmd is essentially an extensive command-line "wrapper" interface
@@ -1662,31 +1662,30 @@ that hides the details of all the individual files in
1662events within the ``/sys/kernel/debug/tracing/events/`` subdirectory and to 1662events within the ``/sys/kernel/debug/tracing/events/`` subdirectory and to
1663collect traces and avoid having to deal with those details directly. 1663collect traces and avoid having to deal with those details directly.
1664 1664
1665As yet another layer on top of that, kernelshark provides a GUI that 1665As yet another layer on top of that, KernelShark provides a GUI that
1666allows users to start and stop traces and specify sets of events using 1666allows users to start and stop traces and specify sets of events using
1667an intuitive interface, and view the output as both trace events and as 1667an intuitive interface, and view the output as both trace events and as
1668a per-CPU graphical display. It directly uses trace-cmd as the 1668a per-CPU graphical display. It directly uses trace-cmd as the
1669plumbing that accomplishes all that underneath the covers (and actually 1669plumbing that accomplishes all that underneath the covers (and actually
1670displays the trace-cmd command it uses, as we'll see). 1670displays the trace-cmd command it uses, as we'll see).
1671 1671
1672To start a trace using kernelshark, first start kernelshark:: 1672To start a trace using KernelShark, first start this tool::
1673 1673
1674 root@sugarbay:~# kernelshark 1674 root@sugarbay:~# kernelshark
1675 1675
1676Then bring up the ``Capture`` dialog by 1676Then open up the ``Capture`` dialog by choosing from the KernelShark menu::
1677choosing from the kernelshark menu::
1678 1677
1679 Capture | Record 1678 Capture | Record
1680 1679
1681That will display the following dialog, which allows you to choose one or more 1680That will display the following dialog, which allows you to choose one or more
1682events (or even one or more complete subsystems) to trace: 1681events (or even entire subsystems) to trace:
1683 1682
1684.. image:: figures/kernelshark-choose-events.png 1683.. image:: figures/kernelshark-choose-events.png
1685 :align: center 1684 :align: center
1686 1685
1687Note that these are exactly the same sets of events described in the 1686Note that these are exactly the same sets of events described in the
1688previous trace events subsystem section, and in fact is where trace-cmd 1687previous trace events subsystem section, and in fact is where trace-cmd
1689gets them for kernelshark. 1688gets them for KernelShark.
1690 1689
1691In the above screenshot, we've decided to explore the graphics subsystem 1690In the above screenshot, we've decided to explore the graphics subsystem
1692a bit and so have chosen to trace all the tracepoints contained within 1691a bit and so have chosen to trace all the tracepoints contained within
@@ -1699,12 +1698,12 @@ will turn into the 'Stop' button after the trace has started):
1699.. image:: figures/kernelshark-output-display.png 1698.. image:: figures/kernelshark-output-display.png
1700 :align: center 1699 :align: center
1701 1700
1702Notice that the right-hand pane shows the exact trace-cmd command-line 1701Notice that the right pane shows the exact trace-cmd command-line
1703that's used to run the trace, along with the results of the trace-cmd 1702that's used to run the trace, along with the results of the trace-cmd
1704run. 1703run.
1705 1704
1706Once the ``Stop`` button is pressed, the graphical view magically fills up 1705Once the ``Stop`` button is pressed, the graphical view magically fills up
1707with a colorful per-cpu display of the trace data, along with the 1706with a colorful per-CPU display of the trace data, along with the
1708detailed event listing below that: 1707detailed event listing below that:
1709 1708
1710.. image:: figures/kernelshark-i915-display.png 1709.. image:: figures/kernelshark-i915-display.png
@@ -1717,7 +1716,7 @@ events``:
1717 :align: center 1716 :align: center
1718 1717
1719The tool is pretty self-explanatory, but for more detailed information 1718The tool is pretty self-explanatory, but for more detailed information
1720on navigating through the data, see the `kernelshark 1719on navigating through the data, see the `KernelShark
1721website <https://kernelshark.org/Documentation.html>`__. 1720website <https://kernelshark.org/Documentation.html>`__.
1722 1721
1723ftrace Documentation 1722ftrace Documentation
@@ -1733,41 +1732,41 @@ Documentation directory::
1733 1732
1734 Documentation/trace/events.txt 1733 Documentation/trace/events.txt
1735 1734
1736There is a nice series of articles on using ftrace and trace-cmd at LWN: 1735A nice series of articles on using ftrace and trace-cmd are available at LWN:
1737 1736
1738- `Debugging the kernel using Ftrace - part 1737- `Debugging the kernel using ftrace - part
1739 1 <https://lwn.net/Articles/365835/>`__ 1738 1 <https://lwn.net/Articles/365835/>`__
1740 1739
1741- `Debugging the kernel using Ftrace - part 1740- `Debugging the kernel using ftrace - part
1742 2 <https://lwn.net/Articles/366796/>`__ 1741 2 <https://lwn.net/Articles/366796/>`__
1743 1742
1744- `Secrets of the Ftrace function 1743- `Secrets of the ftrace function
1745 tracer <https://lwn.net/Articles/370423/>`__ 1744 tracer <https://lwn.net/Articles/370423/>`__
1746 1745
1747- `trace-cmd: A front-end for 1746- `trace-cmd: A front-end for
1748 Ftrace <https://lwn.net/Articles/410200/>`__ 1747 ftrace <https://lwn.net/Articles/410200/>`__
1749 1748
1750See also `KernelShark's documentation <https://kernelshark.org/Documentation.html>`__ 1749See also `KernelShark's documentation <https://kernelshark.org/Documentation.html>`__
1751for further usage details. 1750for further usage details.
1752 1751
1753An amusing yet useful README (a tracing mini-HOWTO) can be found in 1752An amusing yet useful README (a tracing mini-How-to) can be found in
1754``/sys/kernel/debug/tracing/README``. 1753``/sys/kernel/debug/tracing/README``.
1755 1754
1756systemtap 1755SystemTap
1757========= 1756=========
1758 1757
1759SystemTap is a system-wide script-based tracing and profiling tool. 1758SystemTap is a system-wide script-based tracing and profiling tool.
1760 1759
1761SystemTap scripts are C-like programs that are executed in the kernel to 1760SystemTap scripts are C-like programs that are executed in the kernel to
1762gather / print / aggregate data extracted from the context they end up being 1761gather / print / aggregate data extracted from the context they end up being
1763invoked under. 1762called under.
1764 1763
1765For example, this probe from the `SystemTap 1764For example, this probe from the `SystemTap
1766tutorial <https://sourceware.org/systemtap/tutorial/>`__ simply prints a 1765tutorial <https://sourceware.org/systemtap/tutorial/>`__ just prints a
1767line every time any process on the system runs ``open()`` on a file. For each line, 1766line every time any process on the system runs ``open()`` on a file. For each line,
1768it prints the executable name of the program that opened the file, along 1767it prints the executable name of the program that opened the file, along
1769with its PID, and the name of the file it opened (or tried to open), 1768with its PID, and the name of the file it opened (or tried to open), which it
1770which it extracts from the open syscall's argstr. 1769extracts from the argument string (``argstr``) of the ``open`` system call.
1771 1770
1772.. code-block:: none 1771.. code-block:: none
1773 1772
@@ -1782,13 +1781,13 @@ which it extracts from the open syscall's argstr.
1782 } 1781 }
1783 1782
1784Normally, to execute this 1783Normally, to execute this
1785probe, you'd simply install systemtap on the system you want to probe, 1784probe, you'd just install SystemTap on the system you want to probe,
1786and directly run the probe on that system e.g. assuming the name of the 1785and directly run the probe on that system e.g. assuming the name of the
1787file containing the above text is trace_open.stp:: 1786file containing the above text is ``trace_open.stp``::
1788 1787
1789 # stap trace_open.stp 1788 # stap trace_open.stp
1790 1789
1791What systemtap does under the covers to run this probe is 1) parse and 1790What SystemTap does under the covers to run this probe is 1) parse and
1792convert the probe to an equivalent "C" form, 2) compile the "C" form 1791convert the probe to an equivalent "C" form, 2) compile the "C" form
1793into a kernel module, 3) insert the module into the kernel, which arms 1792into a kernel module, 3) insert the module into the kernel, which arms
1794it, and 4) collect the data generated by the probe and display it to the 1793it, and 4) collect the data generated by the probe and display it to the
@@ -1801,25 +1800,25 @@ kernel build system unfortunately isn't typically part of the image
1801running on the target. It is normally available on the "host" system 1800running on the target. It is normally available on the "host" system
1802that produced the target image however; in such cases, steps 1 and 2 are 1801that produced the target image however; in such cases, steps 1 and 2 are
1803executed on the host system, and steps 3 and 4 are executed on the 1802executed on the host system, and steps 3 and 4 are executed on the
1804target system, using only the systemtap "runtime". 1803target system, using only the SystemTap "runtime".
1805 1804
1806The systemtap support in Yocto assumes that only steps 3 and 4 are run 1805The SystemTap support in Yocto assumes that only steps 3 and 4 are run
1807on the target; it is possible to do everything on the target, but this 1806on the target; it is possible to do everything on the target, but this
1808section assumes only the typical embedded use-case. 1807section assumes only the typical embedded use-case.
1809 1808
1810So basically what you need to do in order to run a systemtap script on 1809Therefore, what you need to do in order to run a SystemTap script on
1811the target is to 1) on the host system, compile the probe into a kernel 1810the target is to 1) on the host system, compile the probe into a kernel
1812module that makes sense to the target, 2) copy the module onto the 1811module that makes sense to the target, 2) copy the module onto the
1813target system and 3) insert the module into the target kernel, which 1812target system and 3) insert the module into the target kernel, which
1814arms it, and 4) collect the data generated by the probe and display it 1813arms it, and 4) collect the data generated by the probe and display it
1815to the user. 1814to the user.
1816 1815
1817systemtap Setup 1816SystemTap Setup
1818--------------- 1817---------------
1819 1818
1820Those are a lot of steps and a lot of details, but fortunately Yocto 1819Those are many steps and details, but fortunately Yocto
1821includes a script called ``crosstap`` that will take care of those 1820includes a script called ``crosstap`` that will take care of those
1822details, allowing you to simply execute a systemtap script on the remote 1821details, allowing you to just execute a SystemTap script on the remote
1823target, with arguments if necessary. 1822target, with arguments if necessary.
1824 1823
1825In order to do this from a remote host, however, you need to have access 1824In order to do this from a remote host, however, you need to have access
@@ -1832,7 +1831,7 @@ having done a build::
1832 Error: No target kernel build found. 1831 Error: No target kernel build found.
1833 Did you forget to create a local build of your image? 1832 Did you forget to create a local build of your image?
1834 1833
1835'crosstap' requires a local sdk build of the target system 1834'crosstap' requires a local SDK build of the target system
1836(or a build that includes 'tools-profile') in order to build 1835(or a build that includes 'tools-profile') in order to build
1837kernel modules that can probe the target system. 1836kernel modules that can probe the target system.
1838 1837
@@ -1848,11 +1847,11 @@ Practically speaking, that means you need to do the following:
1848 the BSP README and/or the widely available basic documentation 1847 the BSP README and/or the widely available basic documentation
1849 that discusses how to build images). 1848 that discusses how to build images).
1850 1849
1851- Build an -sdk version of the image e.g.:: 1850- Build an ``-sdk`` version of the image e.g.::
1852 1851
1853 $ bitbake core-image-sato-sdk 1852 $ bitbake core-image-sato-sdk
1854 1853
1855- Or build a non-sdk image but include the profiling tools 1854- Or build a non-SDK image but include the profiling tools
1856 (edit ``local.conf`` and add ``tools-profile`` to the end of 1855 (edit ``local.conf`` and add ``tools-profile`` to the end of
1857 :term:``EXTRA_IMAGE_FEATURES`` variable):: 1856 :term:``EXTRA_IMAGE_FEATURES`` variable)::
1858 1857
@@ -1868,15 +1867,14 @@ Practically speaking, that means you need to do the following:
1868 1867
1869.. note:: 1868.. note::
1870 1869
1871 SystemTap, which uses ``crosstap``, assumes you can establish an ssh 1870 SystemTap, which uses ``crosstap``, assumes you can establish an SSH
1872 connection to the remote target. Please refer to the crosstap wiki 1871 connection to the remote target. Please refer to the crosstap wiki
1873 page for details on verifying ssh connections. Also, the ability to ssh 1872 page for details on verifying SSH connections. Also, the ability to SSH
1874 into the target system is not enabled by default in ``*-minimal`` images. 1873 into the target system is not enabled by default in ``*-minimal`` images.
1875 1874
1876So essentially what you need to 1875Therefore, what you need to do is build an SDK image or image with
1877do is build an SDK image or image with ``tools-profile`` as detailed in 1876``tools-profile`` as detailed in the ":ref:`profile-manual/intro:General Setup`"
1878the ":ref:`profile-manual/intro:General Setup`" section of this 1877section of this manual, and boot the resulting target image.
1879manual, and boot the resulting target image.
1880 1878
1881.. note:: 1879.. note::
1882 1880
@@ -1884,12 +1882,12 @@ manual, and boot the resulting target image.
1884 to have the :term:`MACHINE` you're connecting to selected in ``local.conf``, and 1882 to have the :term:`MACHINE` you're connecting to selected in ``local.conf``, and
1885 the kernel in that machine's :term:`Build Directory` must match the kernel on 1883 the kernel in that machine's :term:`Build Directory` must match the kernel on
1886 the booted system exactly, or you'll get the above ``crosstap`` message 1884 the booted system exactly, or you'll get the above ``crosstap`` message
1887 when you try to invoke a script. 1885 when you try to call a script.
1888 1886
1889Running a Script on a Target 1887Running a Script on a Target
1890---------------------------- 1888----------------------------
1891 1889
1892Once you've done that, you should be able to run a systemtap script on 1890Once you've done that, you should be able to run a SystemTap script on
1893the target:: 1891the target::
1894 1892
1895 $ cd /path/to/yocto 1893 $ cd /path/to/yocto
@@ -1918,7 +1916,7 @@ If you get an error connecting to the target e.g.::
1918 $ crosstap root@192.168.7.2 trace_open.stp 1916 $ crosstap root@192.168.7.2 trace_open.stp
1919 error establishing ssh connection on remote 'root@192.168.7.2' 1917 error establishing ssh connection on remote 'root@192.168.7.2'
1920 1918
1921Try ssh'ing to the target and see what happens:: 1919Try connecting to the target through SSH and see what happens::
1922 1920
1923 $ ssh root@192.168.7.2 1921 $ ssh root@192.168.7.2
1924 1922
@@ -1936,7 +1934,7 @@ no password):
1936 matchbox-termin(1036) open ("/tmp/vte3FS2LW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) 1934 matchbox-termin(1036) open ("/tmp/vte3FS2LW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600)
1937 matchbox-termin(1036) open ("/tmp/vteJMC7LW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) 1935 matchbox-termin(1036) open ("/tmp/vteJMC7LW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600)
1938 1936
1939systemtap Documentation 1937SystemTap Documentation
1940----------------------- 1938-----------------------
1941 1939
1942The SystemTap language reference can be found here: `SystemTap Language 1940The SystemTap language reference can be found here: `SystemTap Language
@@ -1949,7 +1947,7 @@ page <https://sourceware.org/systemtap/documentation.html>`__
1949Sysprof 1947Sysprof
1950======= 1948=======
1951 1949
1952Sysprof is a very easy to use system-wide profiler that consists of a 1950Sysprof is an easy to use system-wide profiler that consists of a
1953single window with three panes and a few buttons which allow you to 1951single window with three panes and a few buttons which allow you to
1954start, stop, and view the profile from one place. 1952start, stop, and view the profile from one place.
1955 1953
@@ -1959,16 +1957,16 @@ Sysprof Setup
1959For this section, we'll assume you've already performed the basic setup 1957For this section, we'll assume you've already performed the basic setup
1960outlined in the ":ref:`profile-manual/intro:General Setup`" section. 1958outlined in the ":ref:`profile-manual/intro:General Setup`" section.
1961 1959
1962Sysprof is a GUI-based application that runs on the target system. For 1960Sysprof is a GUI-based application that runs on the target system. For the rest
1963the rest of this document we assume you've ssh'ed to the host and will 1961of this document we assume you're connected to the host through SSH and will be
1964be running Sysprof on the target (you can use the ``-X`` option to ssh and 1962running Sysprof on the target (you can use the ``-X`` option to ``ssh`` and
1965have the Sysprof GUI run on the target but display remotely on the host 1963have the Sysprof GUI run on the target but display remotely on the host
1966if you want). 1964if you want).
1967 1965
1968Basic Sysprof Usage 1966Basic Sysprof Usage
1969------------------- 1967-------------------
1970 1968
1971To start profiling the system, you simply press the ``Start`` button. To 1969To start profiling the system, you just press the ``Start`` button. To
1972stop profiling and to start viewing the profile data in one easy step, 1970stop profiling and to start viewing the profile data in one easy step,
1973press the ``Profile`` button. 1971press the ``Profile`` button.
1974 1972
@@ -1981,11 +1979,11 @@ with profiling data:
1981The left pane shows a list of functions and processes. Selecting one of 1979The left pane shows a list of functions and processes. Selecting one of
1982those expands that function in the right pane, showing all its callees. 1980those expands that function in the right pane, showing all its callees.
1983Note that this caller-oriented display is essentially the inverse of 1981Note that this caller-oriented display is essentially the inverse of
1984perf's default callee-oriented callchain display. 1982perf's default callee-oriented call chain display.
1985 1983
1986In the screenshot above, we're focusing on ``__copy_to_user_ll()`` and 1984In the screenshot above, we're focusing on ``__copy_to_user_ll()`` and
1987looking up the callchain we can see that one of the callers of 1985looking up the call chain we can see that one of the callers of
1988``__copy_to_user_ll`` is ``sys_read()`` and the complete callpath between them. 1986``__copy_to_user_ll`` is ``sys_read()`` and the complete call path between them.
1989Notice that this is essentially a portion of the same information we saw 1987Notice that this is essentially a portion of the same information we saw
1990in the perf display shown in the perf section of this page. 1988in the perf display shown in the perf section of this page.
1991 1989
@@ -1993,7 +1991,7 @@ in the perf display shown in the perf section of this page.
1993 :align: center 1991 :align: center
1994 1992
1995Similarly, the above is a snapshot of the Sysprof display of a 1993Similarly, the above is a snapshot of the Sysprof display of a
1996``copy-from-user`` callchain. 1994``copy-from-user`` call chain.
1997 1995
1998Finally, looking at the third Sysprof pane in the lower left, we can see 1996Finally, looking at the third Sysprof pane in the lower left, we can see
1999a list of all the callers of a particular function selected in the top 1997a list of all the callers of a particular function selected in the top
@@ -2008,18 +2006,17 @@ to the selected function, and so on.
2008 2006
2009.. admonition:: Tying it Together 2007.. admonition:: Tying it Together
2010 2008
2011 If you like sysprof's ``caller-oriented`` display, you may be able to 2009 If you like Sysprof's ``caller-oriented`` display, you may be able to
2012 approximate it in other tools as well. For example, ``perf report`` has 2010 approximate it in other tools as well. For example, ``perf report`` has
2013 the ``-g`` (``--call-graph``) option that you can experiment with; one of the 2011 the ``-g`` (``--call-graph``) option that you can experiment with; one of the
2014 options is ``caller`` for an inverted caller-based callgraph display. 2012 options is ``caller`` for an inverted caller-based call graph display.
2015 2013
2016Sysprof Documentation 2014Sysprof Documentation
2017--------------------- 2015---------------------
2018 2016
2019There doesn't seem to be any documentation for Sysprof, but maybe that's 2017There doesn't seem to be any documentation for Sysprof, but maybe that's
2020because it's pretty self-explanatory. The Sysprof website, however, is 2018because it's pretty self-explanatory. The Sysprof website, however, is here:
2021here: `Sysprof, System-wide Performance Profiler for 2019`Sysprof, System-wide Performance Profiler for Linux <http://sysprof.com/>`__
2022Linux <http://sysprof.com/>`__
2023 2020
2024LTTng (Linux Trace Toolkit, next generation) 2021LTTng (Linux Trace Toolkit, next generation)
2025============================================ 2022============================================
@@ -2029,7 +2026,7 @@ LTTng Setup
2029 2026
2030For this section, we'll assume you've already performed the basic setup 2027For this section, we'll assume you've already performed the basic setup
2031outlined in the ":ref:`profile-manual/intro:General Setup`" section. 2028outlined in the ":ref:`profile-manual/intro:General Setup`" section.
2032LTTng is run on the target system by ssh'ing to it. 2029LTTng is run on the target system by connecting to it through SSH.
2033 2030
2034Collecting and Viewing Traces 2031Collecting and Viewing Traces
2035----------------------------- 2032-----------------------------
@@ -2042,7 +2039,7 @@ tracing.
2042Collecting and viewing a trace on the target (inside a shell) 2039Collecting and viewing a trace on the target (inside a shell)
2043~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2040~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2044 2041
2045First, from the host, ssh to the target:: 2042First, from the host, connect to the target through SSH::
2046 2043
2047 $ ssh -l root 192.168.1.47 2044 $ ssh -l root 192.168.1.47
2048 The authenticity of host '192.168.1.47 (192.168.1.47)' can't be established. 2045 The authenticity of host '192.168.1.47 (192.168.1.47)' can't be established.
@@ -2134,11 +2131,11 @@ supplying your own name to ``lttng create``)::
2134 drwxr-xr-x 5 root root 1024 Oct 15 23:57 .. 2131 drwxr-xr-x 5 root root 1024 Oct 15 23:57 ..
2135 drwxrwx--- 3 root root 1024 Oct 15 23:21 auto-20121015-232120 2132 drwxrwx--- 3 root root 1024 Oct 15 23:21 auto-20121015-232120
2136 2133
2137Collecting and viewing a userspace trace on the target (inside a shell) 2134Collecting and viewing a user space trace on the target (inside a shell)
2138~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2135~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2139 2136
2140For LTTng userspace tracing, you need to have a properly instrumented 2137For LTTng user space tracing, you need to have a properly instrumented
2141userspace program. For this example, we'll use the ``hello`` test program 2138user space program. For this example, we'll use the ``hello`` test program
2142generated by the ``lttng-ust`` build. 2139generated by the ``lttng-ust`` build.
2143 2140
2144The ``hello`` test program isn't installed on the root filesystem by the ``lttng-ust`` 2141The ``hello`` test program isn't installed on the root filesystem by the ``lttng-ust``
@@ -2154,7 +2151,7 @@ Copy that over to the target machine::
2154You now have the instrumented LTTng "hello world" test program on the 2151You now have the instrumented LTTng "hello world" test program on the
2155target, ready to test. 2152target, ready to test.
2156 2153
2157First, from the host, ssh to the target:: 2154First, from the host, connect to the target through SSH::
2158 2155
2159 $ ssh -l root 192.168.1.47 2156 $ ssh -l root 192.168.1.47
2160 The authenticity of host '192.168.1.47 (192.168.1.47)' can't be established. 2157 The authenticity of host '192.168.1.47 (192.168.1.47)' can't be established.
@@ -2169,7 +2166,7 @@ Once on the target, use these steps to create a trace::
2169 Session auto-20190303-021943 created. 2166 Session auto-20190303-021943 created.
2170 Traces will be written in /home/root/lttng-traces/auto-20190303-021943 2167 Traces will be written in /home/root/lttng-traces/auto-20190303-021943
2171 2168
2172Enable the events you want to trace (in this case all userspace events):: 2169Enable the events you want to trace (in this case all user space events)::
2173 2170
2174 root@crownbay:~# lttng enable-event --userspace --all 2171 root@crownbay:~# lttng enable-event --userspace --all
2175 All UST events are enabled in channel channel0 2172 All UST events are enabled in channel channel0
@@ -2241,13 +2238,13 @@ the entire blktrace and blkparse pipeline on the target, or you can run
2241blktrace in 'listen' mode on the target and have blktrace and blkparse 2238blktrace in 'listen' mode on the target and have blktrace and blkparse
2242collect and analyze the data on the host (see the 2239collect and analyze the data on the host (see the
2243":ref:`profile-manual/usage:Using blktrace Remotely`" section 2240":ref:`profile-manual/usage:Using blktrace Remotely`" section
2244below). For the rest of this section we assume you've ssh'ed to the host and 2241below). For the rest of this section we assume you've to the host through SSH
2245will be running blkrace on the target. 2242and will be running blktrace on the target.
2246 2243
2247Basic blktrace Usage 2244Basic blktrace Usage
2248-------------------- 2245--------------------
2249 2246
2250To record a trace, simply run the ``blktrace`` command, giving it the name 2247To record a trace, just run the ``blktrace`` command, giving it the name
2251of the block device you want to trace activity on:: 2248of the block device you want to trace activity on::
2252 2249
2253 root@crownbay:~# blktrace /dev/sdc 2250 root@crownbay:~# blktrace /dev/sdc
@@ -2258,10 +2255,10 @@ In another shell, execute a workload you want to trace::
2258 Connecting to downloads.yoctoproject.org (140.211.169.59:80) 2255 Connecting to downloads.yoctoproject.org (140.211.169.59:80)
2259 linux-2.6.19.2.tar.b 100% \|*******************************\| 41727k 0:00:00 ETA 2256 linux-2.6.19.2.tar.b 100% \|*******************************\| 41727k 0:00:00 ETA
2260 2257
2261Press Ctrl-C in the blktrace shell to stop the trace. It 2258Press ``Ctrl-C`` in the blktrace shell to stop the trace. It
2262will display how many events were logged, along with the per-cpu file 2259will display how many events were logged, along with the per-cpu file
2263sizes (blktrace records traces in per-cpu kernel buffers and simply 2260sizes (blktrace records traces in per-cpu kernel buffers and just
2264dumps them to userspace for blkparse to merge and sort later):: 2261dumps them to user space for blkparse to merge and sort later)::
2265 2262
2266 ^C=== sdc === 2263 ^C=== sdc ===
2267 CPU 0: 7082 events, 332 KiB data 2264 CPU 0: 7082 events, 332 KiB data
@@ -2277,7 +2274,7 @@ with the device name as the first part of the filename::
2277 -rw-r--r-- 1 root root 339938 Oct 27 22:40 sdc.blktrace.0 2274 -rw-r--r-- 1 root root 339938 Oct 27 22:40 sdc.blktrace.0
2278 -rw-r--r-- 1 root root 75753 Oct 27 22:40 sdc.blktrace.1 2275 -rw-r--r-- 1 root root 75753 Oct 27 22:40 sdc.blktrace.1
2279 2276
2280To view the trace events, simply invoke ``blkparse`` in the directory 2277To view the trace events, just call ``blkparse`` in the directory
2281containing the trace files, giving it the device name that forms the 2278containing the trace files, giving it the device name that forms the
2282first part of the filenames:: 2279first part of the filenames::
2283 2280
@@ -2376,8 +2373,8 @@ Live Mode
2376~~~~~~~~~ 2373~~~~~~~~~
2377 2374
2378blktrace and blkparse are designed from the ground up to be able to 2375blktrace and blkparse are designed from the ground up to be able to
2379operate together in a "pipe mode" where the stdout of blktrace can be 2376operate together in a "pipe mode" where the standard output of blktrace can be
2380fed directly into the stdin of blkparse:: 2377fed directly into the standard input of blkparse::
2381 2378
2382 root@crownbay:~# blktrace /dev/sdc -o - | blkparse -i - 2379 root@crownbay:~# blktrace /dev/sdc -o - | blkparse -i -
2383 2380
@@ -2446,7 +2443,7 @@ just ended::
2446 Total: 11800 events (dropped 0), 554 KiB data 2443 Total: 11800 events (dropped 0), 554 KiB data
2447 2444
2448The blktrace instance on the host will 2445The blktrace instance on the host will
2449save the target output inside a hostname-timestamp directory:: 2446save the target output inside a ``<hostname>-<timestamp>`` directory::
2450 2447
2451 $ ls -al 2448 $ ls -al
2452 drwxr-xr-x 10 root root 1024 Oct 28 02:40 . 2449 drwxr-xr-x 10 root root 1024 Oct 28 02:40 .
@@ -2518,7 +2515,7 @@ Tracing Block I/O via 'ftrace'
2518It's also possible to trace block I/O using only 2515It's also possible to trace block I/O using only
2519:ref:`profile-manual/usage:The 'trace events' Subsystem`, which 2516:ref:`profile-manual/usage:The 'trace events' Subsystem`, which
2520can be useful for casual tracing if you don't want to bother dealing with the 2517can be useful for casual tracing if you don't want to bother dealing with the
2521userspace tools. 2518user space tools.
2522 2519
2523To enable tracing for a given device, use ``/sys/block/xxx/trace/enable``, 2520To enable tracing for a given device, use ``/sys/block/xxx/trace/enable``,
2524where ``xxx`` is the device name. This for example enables tracing for 2521where ``xxx`` is the device name. This for example enables tracing for
@@ -2576,7 +2573,7 @@ section can be found here:
2576- https://linux.die.net/man/8/btrace 2573- https://linux.die.net/man/8/btrace
2577 2574
2578The above manual pages, along with manuals for the other blktrace utilities 2575The above manual pages, along with manuals for the other blktrace utilities
2579(btt, blkiomon, etc) can be found in the ``/doc`` directory of the blktrace 2576(``btt``, ``blkiomon``, etc) can be found in the ``/doc`` directory of the blktrace
2580tools git repo:: 2577tools git repository::
2581 2578
2582 $ git clone git://git.kernel.dk/blktrace.git 2579 $ git clone git://git.kernel.dk/blktrace.git
diff --git a/documentation/styles/config/vocabularies/OpenSource/accept.txt b/documentation/styles/config/vocabularies/OpenSource/accept.txt
index 98e76ae1f5..e378fbf79b 100644
--- a/documentation/styles/config/vocabularies/OpenSource/accept.txt
+++ b/documentation/styles/config/vocabularies/OpenSource/accept.txt
@@ -1,4 +1,20 @@
1autovivification
2blkparse
3blktrace
4callee
5debugfs
1ftrace 6ftrace
2toolchain 7KernelShark
3systemd 8Kprobe
4LTTng 9LTTng
10perf
11profiler
12subcommand
13subnode
14superset
15Sysprof
16systemd
17toolchain
18tracepoint
19Uprobe
20wget
diff --git a/documentation/styles/config/vocabularies/Yocto/accept.txt b/documentation/styles/config/vocabularies/Yocto/accept.txt
index b725414014..ca622ba412 100644
--- a/documentation/styles/config/vocabularies/Yocto/accept.txt
+++ b/documentation/styles/config/vocabularies/Yocto/accept.txt
@@ -1,4 +1,5 @@
1Yocto
2BSP
3BitBake 1BitBake
2BSP
3crosstap
4OpenEmbedded 4OpenEmbedded
5Yocto