summaryrefslogtreecommitdiffstats
path: root/documentation
diff options
context:
space:
mode:
authorMichael Opdenacker <michael.opdenacker@bootlin.com>2024-03-14 13:28:00 +0100
committerRichard Purdie <richard.purdie@linuxfoundation.org>2024-03-19 15:22:27 +0000
commitd255faa35e5b4128d8891d8e375049a1aa8d1f6e (patch)
tree69bf8eac02940c84225152120a9a1755669851a6 /documentation
parentf6584a38ae0f103876564448428ec88adff97c0c (diff)
downloadpoky-d255faa35e5b4128d8891d8e375049a1aa8d1f6e.tar.gz
profile-manual: usage.rst: further style improvements
According to errors reported by "make stylecheck" (From yocto-docs rev: fc808b2dca4321209bfa6206d9e43aa1a5a0c5b8) Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Diffstat (limited to 'documentation')
-rw-r--r--documentation/profile-manual/usage.rst335
-rw-r--r--documentation/styles/config/vocabularies/OpenSource/accept.txt20
-rw-r--r--documentation/styles/config/vocabularies/Yocto/accept.txt5
3 files changed, 187 insertions, 173 deletions
diff --git a/documentation/profile-manual/usage.rst b/documentation/profile-manual/usage.rst
index 0d368792b5..2f82137538 100644
--- a/documentation/profile-manual/usage.rst
+++ b/documentation/profile-manual/usage.rst
@@ -10,7 +10,7 @@ Basic Usage (with examples) for each of the Yocto Tracing Tools
10This chapter presents basic usage examples for each of the tracing 10This chapter presents basic usage examples for each of the tracing
11tools. 11tools.
12 12
13Perf 13perf
14==== 14====
15 15
16The perf tool is the profiling and tracing tool that comes bundled 16The perf tool is the profiling and tracing tool that comes bundled
@@ -26,12 +26,12 @@ of what's going on.
26 26
27In many ways, perf aims to be a superset of all the tracing and 27In many ways, perf aims to be a superset of all the tracing and
28profiling tools available in Linux today, including all the other tools 28profiling tools available in Linux today, including all the other tools
29covered in this HOWTO. The past couple of years have seen perf subsume a 29covered in this How-to. The past couple of years have seen perf subsume a
30lot of the functionality of those other tools and, at the same time, 30lot of the functionality of those other tools and, at the same time,
31those other tools have removed large portions of their previous 31those other tools have removed large portions of their previous
32functionality and replaced it with calls to the equivalent functionality 32functionality and replaced it with calls to the equivalent functionality
33now implemented by the perf subsystem. Extrapolation suggests that at 33now implemented by the perf subsystem. Extrapolation suggests that at
34some point those other tools will simply become completely redundant and 34some point those other tools will become completely redundant and
35go away; until then, we'll cover those other tools in these pages and in 35go away; until then, we'll cover those other tools in these pages and in
36many cases show how the same things can be accomplished in perf and the 36many cases show how the same things can be accomplished in perf and the
37other tools when it seems useful to do so. 37other tools when it seems useful to do so.
@@ -41,7 +41,7 @@ want to apply the tool; full documentation can be found either within
41the tool itself or in the manual pages at 41the tool itself or in the manual pages at
42`perf(1) <https://linux.die.net/man/1/perf>`__. 42`perf(1) <https://linux.die.net/man/1/perf>`__.
43 43
44Perf Setup 44perf Setup
45---------- 45----------
46 46
47For this section, we'll assume you've already performed the basic setup 47For this section, we'll assume you've already performed the basic setup
@@ -54,14 +54,14 @@ image built with the following in your ``local.conf`` file::
54 54
55perf runs on the target system for the most part. You can archive 55perf runs on the target system for the most part. You can archive
56profile data and copy it to the host for analysis, but for the rest of 56profile data and copy it to the host for analysis, but for the rest of
57this document we assume you've ssh'ed to the host and will be running 57this document we assume you're connected to the host through SSH and will be
58the perf commands on the target. 58running the perf commands on the target.
59 59
60Basic Perf Usage 60Basic perf Usage
61---------------- 61----------------
62 62
63The perf tool is pretty much self-documenting. To remind yourself of the 63The perf tool is pretty much self-documenting. To remind yourself of the
64available commands, simply type ``perf``, which will show you basic usage 64available commands, just type ``perf``, which will show you basic usage
65along with the available perf subcommands:: 65along with the available perf subcommands::
66 66
67 root@crownbay:~# perf 67 root@crownbay:~# perf
@@ -101,7 +101,7 @@ As a simple test case, we'll profile the ``wget`` of a fairly large file,
101which is a minimally interesting case because it has both file and 101which is a minimally interesting case because it has both file and
102network I/O aspects, and at least in the case of standard Yocto images, 102network I/O aspects, and at least in the case of standard Yocto images,
103it's implemented as part of BusyBox, so the methods we use to analyze it 103it's implemented as part of BusyBox, so the methods we use to analyze it
104can be used in a very similar way to the whole host of supported BusyBox 104can be used in a similar way to the whole host of supported BusyBox
105applets in Yocto:: 105applets in Yocto::
106 106
107 root@crownbay:~# rm linux-2.6.19.2.tar.bz2; \ 107 root@crownbay:~# rm linux-2.6.19.2.tar.bz2; \
@@ -164,17 +164,17 @@ hits and misses::
164 164
165 44.831023415 seconds time elapsed 165 44.831023415 seconds time elapsed
166 166
167So ``perf stat`` gives us a nice easy 167As you can see, ``perf stat`` gives us a nice easy
168way to get a quick overview of what might be happening for a set of 168way to get a quick overview of what might be happening for a set of
169events, but normally we'd need a little more detail in order to 169events, but normally we'd need a little more detail in order to
170understand what's going on in a way that we can act on in a useful way. 170understand what's going on in a way that we can act on in a useful way.
171 171
172To dive down into a next level of detail, we can use ``perf record`` / 172To dive down into a next level of detail, we can use ``perf record`` /
173``perf report`` which will collect profiling data and present it to use using an 173``perf report`` which will collect profiling data and present it to use using an
174interactive text-based UI (or simply as text if we specify ``--stdio`` to 174interactive text-based UI (or just as text if we specify ``--stdio`` to
175``perf report``). 175``perf report``).
176 176
177As our first attempt at profiling this workload, we'll simply run ``perf 177As our first attempt at profiling this workload, we'll just run ``perf
178record``, handing it the workload we want to profile (everything after 178record``, handing it the workload we want to profile (everything after
179``perf record`` and any perf options we hand it --- here none, will be 179``perf record`` and any perf options we hand it --- here none, will be
180executed in a new shell). perf collects samples until the process exits 180executed in a new shell). perf collects samples until the process exits
@@ -189,7 +189,7 @@ directory::
189 [ perf record: Captured and wrote 0.176 MB perf.data (~7700 samples) ] 189 [ perf record: Captured and wrote 0.176 MB perf.data (~7700 samples) ]
190 190
191To see the results in a 191To see the results in a
192"text-based UI" (tui), simply run ``perf report``, which will read the 192"text-based UI" (tui), just run ``perf report``, which will read the
193perf.data file in the current working directory and display the results 193perf.data file in the current working directory and display the results
194in an interactive UI:: 194in an interactive UI::
195 195
@@ -204,10 +204,10 @@ The above screenshot displays a "flat" profile, one entry for each
204profiling run, ordered from the most popular to the least (perf has 204profiling run, ordered from the most popular to the least (perf has
205options to sort in various orders and keys as well as display entries 205options to sort in various orders and keys as well as display entries
206only above a certain threshold and so on --- see the perf documentation 206only above a certain threshold and so on --- see the perf documentation
207for details). Note that this includes both userspace functions (entries 207for details). Note that this includes both user space functions (entries
208containing a ``[.]``) and kernel functions accounted to the process (entries 208containing a ``[.]``) and kernel functions accounted to the process (entries
209containing a ``[k]``). perf has command-line modifiers that can be used to 209containing a ``[k]``). perf has command-line modifiers that can be used to
210restrict the profiling to kernel or userspace, among others. 210restrict the profiling to kernel or user space, among others.
211 211
212Notice also that the above report shows an entry for ``busybox``, which is 212Notice also that the above report shows an entry for ``busybox``, which is
213the executable that implements ``wget`` in Yocto, but that instead of a 213the executable that implements ``wget`` in Yocto, but that instead of a
@@ -218,7 +218,7 @@ Before we do that, however, let's try running a different profile, one
218which shows something a little more interesting. The only difference 218which shows something a little more interesting. The only difference
219between the new profile and the previous one is that we'll add the ``-g`` 219between the new profile and the previous one is that we'll add the ``-g``
220option, which will record not just the address of a sampled function, 220option, which will record not just the address of a sampled function,
221but the entire callchain to the sampled function as well:: 221but the entire call chain to the sampled function as well::
222 222
223 root@crownbay:~# perf record -g wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2 223 root@crownbay:~# perf record -g wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2
224 Connecting to downloads.yoctoproject.org (140.211.169.59:80) 224 Connecting to downloads.yoctoproject.org (140.211.169.59:80)
@@ -233,26 +233,26 @@ but the entire callchain to the sampled function as well::
233 :align: center 233 :align: center
234 :width: 70% 234 :width: 70%
235 235
236Using the callgraph view, we can actually see not only which functions 236Using the call graph view, we can actually see not only which functions
237took the most time, but we can also see a summary of how those functions 237took the most time, but we can also see a summary of how those functions
238were called and learn something about how the program interacts with the 238were called and learn something about how the program interacts with the
239kernel in the process. 239kernel in the process.
240 240
241Notice that each entry in the above screenshot now contains a ``+`` on the 241Notice that each entry in the above screenshot now contains a ``+`` on the
242left-hand side. This means that we can expand the entry and drill down 242left side. This means that we can expand the entry and drill down
243into the callchains that feed into that entry. Pressing ``Enter`` on any 243into the call chains that feed into that entry. Pressing ``Enter`` on any
244one of them will expand the callchain (you can also press ``E`` to expand 244one of them will expand the call chain (you can also press ``E`` to expand
245them all at the same time or ``C`` to collapse them all). 245them all at the same time or ``C`` to collapse them all).
246 246
247In the screenshot above, we've toggled the ``__copy_to_user_ll()`` entry 247In the screenshot above, we've toggled the ``__copy_to_user_ll()`` entry
248and several subnodes all the way down. This lets us see which callchains 248and several subnodes all the way down. This lets us see which call chains
249contributed to the profiled ``__copy_to_user_ll()`` function which 249contributed to the profiled ``__copy_to_user_ll()`` function which
250contributed 1.77% to the total profile. 250contributed 1.77% to the total profile.
251 251
252As a bit of background explanation for these callchains, think about 252As a bit of background explanation for these call chains, think about
253what happens at a high level when you run wget to get a file out on the 253what happens at a high level when you run ``wget`` to get a file out on the
254network. Basically what happens is that the data comes into the kernel 254network. Basically what happens is that the data comes into the kernel
255via the network connection (socket) and is passed to the userspace 255via the network connection (socket) and is passed to the user space
256program ``wget`` (which is actually a part of BusyBox, but that's not 256program ``wget`` (which is actually a part of BusyBox, but that's not
257important for now), which takes the buffers the kernel passes to it and 257important for now), which takes the buffers the kernel passes to it and
258writes it to a disk file to save it. 258writes it to a disk file to save it.
@@ -262,16 +262,16 @@ is the part where the kernel passes the data it has read from the socket
262down to wget i.e. a ``copy-to-user``. 262down to wget i.e. a ``copy-to-user``.
263 263
264Notice also that here there's also a case where the hex value is 264Notice also that here there's also a case where the hex value is
265displayed in the callstack, here in the expanded ``sys_clock_gettime()`` 265displayed in the call stack, here in the expanded ``sys_clock_gettime()``
266function. Later we'll see it resolve to a userspace function call in 266function. Later we'll see it resolve to a user space function call in
267busybox. 267BusyBox.
268 268
269.. image:: figures/perf-wget-g-copy-from-user-expanded-stripped.png 269.. image:: figures/perf-wget-g-copy-from-user-expanded-stripped.png
270 :align: center 270 :align: center
271 :width: 70% 271 :width: 70%
272 272
273The above screenshot shows the other half of the journey for the data --- 273The above screenshot shows the other half of the journey for the data ---
274from the ``wget`` program's userspace buffers to disk. To get the buffers to 274from the ``wget`` program's user space buffers to disk. To get the buffers to
275disk, the wget program issues a ``write(2)``, which does a ``copy-from-user`` to 275disk, the wget program issues a ``write(2)``, which does a ``copy-from-user`` to
276the kernel, which then takes care via some circuitous path (probably 276the kernel, which then takes care via some circuitous path (probably
277also present somewhere in the profile data), to get it safely to disk. 277also present somewhere in the profile data), to get it safely to disk.
@@ -281,8 +281,8 @@ of how to extract useful information out of it, let's get back to the
281task at hand and see if we can get some basic idea about where the time 281task at hand and see if we can get some basic idea about where the time
282is spent in the program we're profiling, wget. Remember that wget is 282is spent in the program we're profiling, wget. Remember that wget is
283actually implemented as an applet in BusyBox, so while the process name 283actually implemented as an applet in BusyBox, so while the process name
284is ``wget``, the executable we're actually interested in is BusyBox. So 284is ``wget``, the executable we're actually interested in is ``busybox``.
285let's expand the first entry containing BusyBox: 285Therefore, let's expand the first entry containing BusyBox:
286 286
287.. image:: figures/perf-wget-busybox-expanded-stripped.png 287.. image:: figures/perf-wget-busybox-expanded-stripped.png
288 :align: center 288 :align: center
@@ -293,7 +293,7 @@ hex value instead of a symbol as with most of the kernel entries.
293Expanding the BusyBox entry doesn't make it any better. 293Expanding the BusyBox entry doesn't make it any better.
294 294
295The problem is that perf can't find the symbol information for the 295The problem is that perf can't find the symbol information for the
296busybox binary, which is actually stripped out by the Yocto build 296``busybox`` binary, which is actually stripped out by the Yocto build
297system. 297system.
298 298
299One way around that is to put the following in your ``local.conf`` file 299One way around that is to put the following in your ``local.conf`` file
@@ -303,20 +303,20 @@ when you build the image::
303 303
304However, we already have an image with the binaries stripped, so 304However, we already have an image with the binaries stripped, so
305what can we do to get perf to resolve the symbols? Basically we need to 305what can we do to get perf to resolve the symbols? Basically we need to
306install the debuginfo for the BusyBox package. 306install the debugging information for the BusyBox package.
307 307
308To generate the debug info for the packages in the image, we can add 308To generate the debug info for the packages in the image, we can add
309``dbg-pkgs`` to :term:`EXTRA_IMAGE_FEATURES` in ``local.conf``. For example:: 309``dbg-pkgs`` to :term:`EXTRA_IMAGE_FEATURES` in ``local.conf``. For example::
310 310
311 EXTRA_IMAGE_FEATURES = "debug-tweaks tools-profile dbg-pkgs" 311 EXTRA_IMAGE_FEATURES = "debug-tweaks tools-profile dbg-pkgs"
312 312
313Additionally, in order to generate the type of debuginfo that perf 313Additionally, in order to generate the type of debugging information that perf
314understands, we also need to set :term:`PACKAGE_DEBUG_SPLIT_STYLE` 314understands, we also need to set :term:`PACKAGE_DEBUG_SPLIT_STYLE`
315in the ``local.conf`` file:: 315in the ``local.conf`` file::
316 316
317 PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory' 317 PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory'
318 318
319Once we've done that, we can install the debuginfo for BusyBox. The 319Once we've done that, we can install the debugging information for BusyBox. The
320debug packages once built can be found in ``build/tmp/deploy/rpm/*`` 320debug packages once built can be found in ``build/tmp/deploy/rpm/*``
321on the host system. Find the ``busybox-dbg-...rpm`` file and copy it 321on the host system. Find the ``busybox-dbg-...rpm`` file and copy it
322to the target. For example:: 322to the target. For example::
@@ -324,11 +324,11 @@ to the target. For example::
324 [trz@empanada core2]$ scp /home/trz/yocto/crownbay-tracing-dbg/build/tmp/deploy/rpm/core2_32/busybox-dbg-1.20.2-r2.core2_32.rpm root@192.168.1.31: 324 [trz@empanada core2]$ scp /home/trz/yocto/crownbay-tracing-dbg/build/tmp/deploy/rpm/core2_32/busybox-dbg-1.20.2-r2.core2_32.rpm root@192.168.1.31:
325 busybox-dbg-1.20.2-r2.core2_32.rpm 100% 1826KB 1.8MB/s 00:01 325 busybox-dbg-1.20.2-r2.core2_32.rpm 100% 1826KB 1.8MB/s 00:01
326 326
327Now install the debug rpm on the target:: 327Now install the debug RPM on the target::
328 328
329 root@crownbay:~# rpm -i busybox-dbg-1.20.2-r2.core2_32.rpm 329 root@crownbay:~# rpm -i busybox-dbg-1.20.2-r2.core2_32.rpm
330 330
331Now that the debuginfo is installed, we see that the BusyBox entries now display 331Now that the debugging information is installed, we see that the BusyBox entries now display
332their functions symbolically: 332their functions symbolically:
333 333
334.. image:: figures/perf-wget-busybox-debuginfo.png 334.. image:: figures/perf-wget-busybox-debuginfo.png
@@ -351,7 +351,7 @@ expanded all the nodes using the ``E`` key):
351 :align: center 351 :align: center
352 :width: 70% 352 :width: 70%
353 353
354Finally, we can see that now that the BusyBox debuginfo is installed, 354Finally, we can see that now that the BusyBox debugging information is installed,
355the previously unresolved symbol in the ``sys_clock_gettime()`` entry 355the previously unresolved symbol in the ``sys_clock_gettime()`` entry
356mentioned previously is now resolved, and shows that the 356mentioned previously is now resolved, and shows that the
357``sys_clock_gettime`` system call that was the source of 6.75% of the 357``sys_clock_gettime`` system call that was the source of 6.75% of the
@@ -386,8 +386,8 @@ counter, something other than the default ``cycles``.
386The tracing and profiling infrastructure in Linux has become unified in 386The tracing and profiling infrastructure in Linux has become unified in
387a way that allows us to use the same tool with a completely different 387a way that allows us to use the same tool with a completely different
388set of counters, not just the standard hardware counters that 388set of counters, not just the standard hardware counters that
389traditional tools have had to restrict themselves to (of course the 389traditional tools have had to restrict themselves to (the
390traditional tools can also make use of the expanded possibilities now 390traditional tools can now actually make use of the expanded possibilities now
391available to them, and in some cases have, as mentioned previously). 391available to them, and in some cases have, as mentioned previously).
392 392
393We can get a list of the available events that can be used to profile a 393We can get a list of the available events that can be used to profile a
@@ -527,14 +527,14 @@ workload via ``perf list``::
527.. admonition:: Tying it Together 527.. admonition:: Tying it Together
528 528
529 These are exactly the same set of events defined by the trace event 529 These are exactly the same set of events defined by the trace event
530 subsystem and exposed by ftrace / tracecmd / kernelshark as files in 530 subsystem and exposed by ftrace / trace-cmd / KernelShark as files in
531 ``/sys/kernel/debug/tracing/events``, by SystemTap as 531 ``/sys/kernel/debug/tracing/events``, by SystemTap as
532 kernel.trace("tracepoint_name") and (partially) accessed by LTTng. 532 kernel.trace("tracepoint_name") and (partially) accessed by LTTng.
533 533
534Only a subset of these would be of interest to us when looking at this 534Only a subset of these would be of interest to us when looking at this
535workload, so let's choose the most likely subsystems (identified by the 535workload, so let's choose the most likely subsystems (identified by the
536string before the colon in the Tracepoint events) and do a ``perf stat`` 536string before the colon in the ``Tracepoint`` events) and do a ``perf stat``
537run using only those wildcarded subsystems:: 537run using only those subsystem wildcards::
538 538
539 root@crownbay:~# perf stat -e skb:* -e net:* -e napi:* -e sched:* -e workqueue:* -e irq:* -e syscalls:* wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2 539 root@crownbay:~# perf stat -e skb:* -e net:* -e napi:* -e sched:* -e workqueue:* -e irq:* -e syscalls:* wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2
540 Performance counter stats for 'wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2': 540 Performance counter stats for 'wget &YOCTO_DL_URL;/mirror/sources/linux-2.6.19.2.tar.bz2':
@@ -625,8 +625,8 @@ accounts for the function name actually displayed in the profile:
625 } 625 }
626 626
627A couple of the more interesting 627A couple of the more interesting
628callchains are expanded and displayed above, basically some network 628call chains are expanded and displayed above, basically some network
629receive paths that presumably end up waking up wget (busybox) when 629receive paths that presumably end up waking up wget (BusyBox) when
630network data is ready. 630network data is ready.
631 631
632Note that because tracepoints are normally used for tracing, the default 632Note that because tracepoints are normally used for tracing, the default
@@ -646,7 +646,7 @@ high-level view of what's going on with a workload or across the system.
646It is however by definition an approximation, as suggested by the most 646It is however by definition an approximation, as suggested by the most
647prominent word associated with it, ``sampling``. On the one hand, it 647prominent word associated with it, ``sampling``. On the one hand, it
648allows a representative picture of what's going on in the system to be 648allows a representative picture of what's going on in the system to be
649cheaply taken, but on the other hand, that cheapness limits its utility 649cheaply taken, but alternatively, that cheapness limits its utility
650when that data suggests a need to "dive down" more deeply to discover 650when that data suggests a need to "dive down" more deeply to discover
651what's really going on. In such cases, the only way to see what's really 651what's really going on. In such cases, the only way to see what's really
652going on is to be able to look at (or summarize more intelligently) the 652going on is to be able to look at (or summarize more intelligently) the
@@ -711,7 +711,7 @@ an infinite variety of ways.
711Another way to look at it is that there are only so many ways that the 711Another way to look at it is that there are only so many ways that the
712'primitive' counters can be used on their own to generate interesting 712'primitive' counters can be used on their own to generate interesting
713output; to get anything more complicated than simple counts requires 713output; to get anything more complicated than simple counts requires
714some amount of additional logic, which is typically very specific to the 714some amount of additional logic, which is typically specific to the
715problem at hand. For example, if we wanted to make use of a 'counter' 715problem at hand. For example, if we wanted to make use of a 'counter'
716that maps to the value of the time difference between when a process was 716that maps to the value of the time difference between when a process was
717scheduled to run on a processor and the time it actually ran, we 717scheduled to run on a processor and the time it actually ran, we
@@ -722,12 +722,12 @@ standard profiling tools how much data every process on the system reads
722and writes, along with how many of those reads and writes fail 722and writes, along with how many of those reads and writes fail
723completely. If we have sufficient trace data, however, we could with the 723completely. If we have sufficient trace data, however, we could with the
724right tools easily extract and present that information, but we'd need 724right tools easily extract and present that information, but we'd need
725something other than pre-canned profiling tools to do that. 725something other than ready-made profiling tools to do that.
726 726
727Luckily, there is a general-purpose way to handle such needs, called 727Luckily, there is a general-purpose way to handle such needs, called
728"programming languages". Making programming languages easily available 728"programming languages". Making programming languages easily available
729to apply to such problems given the specific format of data is called a 729to apply to such problems given the specific format of data is called a
730'programming language binding' for that data and language. Perf supports 730'programming language binding' for that data and language. perf supports
731two programming language bindings, one for Python and one for Perl. 731two programming language bindings, one for Python and one for Perl.
732 732
733.. admonition:: Tying it Together 733.. admonition:: Tying it Together
@@ -737,7 +737,7 @@ two programming language bindings, one for Python and one for Perl.
737 DProbes dpcc compiler, an ANSI C compiler which targeted a low-level 737 DProbes dpcc compiler, an ANSI C compiler which targeted a low-level
738 assembly language running on an in-kernel interpreter on the target 738 assembly language running on an in-kernel interpreter on the target
739 system. This is exactly analogous to what Sun's DTrace did, except 739 system. This is exactly analogous to what Sun's DTrace did, except
740 that DTrace invented its own language for the purpose. Systemtap, 740 that DTrace invented its own language for the purpose. SystemTap,
741 heavily inspired by DTrace, also created its own one-off language, 741 heavily inspired by DTrace, also created its own one-off language,
742 but rather than running the product on an in-kernel interpreter, 742 but rather than running the product on an in-kernel interpreter,
743 created an elaborate compiler-based machinery to translate its 743 created an elaborate compiler-based machinery to translate its
@@ -750,8 +750,8 @@ entry / exit events we recorded::
750 root@crownbay:~# perf script -g python 750 root@crownbay:~# perf script -g python
751 generated Python script: perf-script.py 751 generated Python script: perf-script.py
752 752
753The skeleton script simply creates a Python function for each event type in the 753The skeleton script just creates a Python function for each event type in the
754``perf.data`` file. The body of each function simply prints the event name along 754``perf.data`` file. The body of each function just prints the event name along
755with its parameters. For example: 755with its parameters. For example:
756 756
757.. code-block:: python 757.. code-block:: python
@@ -794,7 +794,7 @@ We can run that script directly to print all of the events contained in the
794 syscalls__sys_exit_read 1 11624.859944032 1262 wget nr=3, ret=1024 794 syscalls__sys_exit_read 1 11624.859944032 1262 wget nr=3, ret=1024
795 795
796That in itself isn't very useful; after all, we can accomplish pretty much the 796That in itself isn't very useful; after all, we can accomplish pretty much the
797same thing by simply running ``perf script`` without arguments in the same 797same thing by just running ``perf script`` without arguments in the same
798directory as the ``perf.data`` file. 798directory as the ``perf.data`` file.
799 799
800We can however replace the print statements in the generated function 800We can however replace the print statements in the generated function
@@ -816,7 +816,7 @@ event. For example:
816 816
817Each event handler function in the generated code 817Each event handler function in the generated code
818is modified to do this. For convenience, we define a common function 818is modified to do this. For convenience, we define a common function
819called ``inc_counts()`` that each handler calls; ``inc_counts()`` simply tallies 819called ``inc_counts()`` that each handler calls; ``inc_counts()`` just tallies
820a count for each event using the ``counts`` hash, which is a specialized 820a count for each event using the ``counts`` hash, which is a specialized
821hash function that does Perl-like autovivification, a capability that's 821hash function that does Perl-like autovivification, a capability that's
822extremely useful for kinds of multi-level aggregation commonly used in 822extremely useful for kinds of multi-level aggregation commonly used in
@@ -876,7 +876,7 @@ System-Wide Tracing and Profiling
876~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 876~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
877 877
878The examples so far have focused on tracing a particular program or 878The examples so far have focused on tracing a particular program or
879workload --- in other words, every profiling run has specified the program 879workload --- that is, every profiling run has specified the program
880to profile in the command-line e.g. ``perf record wget ...``. 880to profile in the command-line e.g. ``perf record wget ...``.
881 881
882It's also possible, and more interesting in many cases, to run a 882It's also possible, and more interesting in many cases, to run a
@@ -906,13 +906,13 @@ other processes running on the system as well:
906 :align: center 906 :align: center
907 :width: 70% 907 :width: 70%
908 908
909In the snapshot above, we can see callchains that originate in libc, and 909In the snapshot above, we can see call chains that originate in ``libc``, and
910a callchain from Xorg that demonstrates that we're using a proprietary X 910a call chain from ``Xorg`` that demonstrates that we're using a proprietary X
911driver in userspace (notice the presence of ``PVR`` and some other 911driver in user space (notice the presence of ``PVR`` and some other
912unresolvable symbols in the expanded Xorg callchain). 912unresolvable symbols in the expanded ``Xorg`` call chain).
913 913
914Note also that we have both kernel and userspace entries in the above 914Note also that we have both kernel and user space entries in the above
915snapshot. We can also tell perf to focus on userspace but providing a 915snapshot. We can also tell perf to focus on user space but providing a
916modifier, in this case ``u``, to the ``cycles`` hardware counter when we 916modifier, in this case ``u``, to the ``cycles`` hardware counter when we
917record a profile:: 917record a profile::
918 918
@@ -924,7 +924,7 @@ record a profile::
924 :align: center 924 :align: center
925 :width: 70% 925 :width: 70%
926 926
927Notice in the screenshot above, we see only userspace entries (``[.]``) 927Notice in the screenshot above, we see only user space entries (``[.]``)
928 928
929Finally, we can press ``Enter`` on a leaf node and select the ``Zoom into 929Finally, we can press ``Enter`` on a leaf node and select the ``Zoom into
930DSO`` menu item to show only entries associated with a specific DSO. In 930DSO`` menu item to show only entries associated with a specific DSO. In
@@ -960,7 +960,7 @@ We can look at the raw output using ``perf script`` with no arguments::
960Filtering 960Filtering
961^^^^^^^^^ 961^^^^^^^^^
962 962
963Notice that there are a lot of events that don't really have anything to 963Notice that there are many events that don't really have anything to
964do with what we're interested in, namely events that schedule ``perf`` 964do with what we're interested in, namely events that schedule ``perf``
965itself in and out or that wake perf up. We can get rid of those by using 965itself in and out or that wake perf up. We can get rid of those by using
966the ``--filter`` option --- for each event we specify using ``-e``, we can add a 966the ``--filter`` option --- for each event we specify using ``-e``, we can add a
@@ -999,7 +999,7 @@ purpose of demonstrating how to use filters, it's close enough.
999.. admonition:: Tying it Together 999.. admonition:: Tying it Together
1000 1000
1001 These are exactly the same set of event filters defined by the trace 1001 These are exactly the same set of event filters defined by the trace
1002 event subsystem. See the ftrace / tracecmd / kernelshark section for more 1002 event subsystem. See the ftrace / trace-cmd / KernelShark section for more
1003 discussion about these event filters. 1003 discussion about these event filters.
1004 1004
1005.. admonition:: Tying it Together 1005.. admonition:: Tying it Together
@@ -1009,14 +1009,14 @@ purpose of demonstrating how to use filters, it's close enough.
1009 indispensable part of the perf design as it relates to tracing. 1009 indispensable part of the perf design as it relates to tracing.
1010 kernel-based event filters provide a mechanism to precisely throttle 1010 kernel-based event filters provide a mechanism to precisely throttle
1011 the event stream that appears in user space, where it makes sense to 1011 the event stream that appears in user space, where it makes sense to
1012 provide bindings to real programming languages for postprocessing the 1012 provide bindings to real programming languages for post-processing the
1013 event stream. This architecture allows for the intelligent and 1013 event stream. This architecture allows for the intelligent and
1014 flexible partitioning of processing between the kernel and user 1014 flexible partitioning of processing between the kernel and user
1015 space. Contrast this with other tools such as SystemTap, which does 1015 space. Contrast this with other tools such as SystemTap, which does
1016 all of its processing in the kernel and as such requires a special 1016 all of its processing in the kernel and as such requires a special
1017 project-defined language in order to accommodate that design, or 1017 project-defined language in order to accommodate that design, or
1018 LTTng, where everything is sent to userspace and as such requires a 1018 LTTng, where everything is sent to user space and as such requires a
1019 super-efficient kernel-to-userspace transport mechanism in order to 1019 super-efficient kernel-to-user space transport mechanism in order to
1020 function properly. While perf certainly can benefit from for instance 1020 function properly. While perf certainly can benefit from for instance
1021 advances in the design of the transport, it doesn't fundamentally 1021 advances in the design of the transport, it doesn't fundamentally
1022 depend on them. Basically, if you find that your perf tracing 1022 depend on them. Basically, if you find that your perf tracing
@@ -1028,7 +1028,7 @@ Using Dynamic Tracepoints
1028 1028
1029perf isn't restricted to the fixed set of static tracepoints listed by 1029perf isn't restricted to the fixed set of static tracepoints listed by
1030``perf list``. Users can also add their own "dynamic" tracepoints anywhere 1030``perf list``. Users can also add their own "dynamic" tracepoints anywhere
1031in the kernel. For instance, suppose we want to define our own 1031in the kernel. For example, suppose we want to define our own
1032tracepoint on ``do_fork()``. We can do that using the ``perf probe`` perf 1032tracepoint on ``do_fork()``. We can do that using the ``perf probe`` perf
1033subcommand:: 1033subcommand::
1034 1034
@@ -1083,7 +1083,7 @@ up after 30 seconds)::
1083 [ perf record: Woken up 1 times to write data ] 1083 [ perf record: Woken up 1 times to write data ]
1084 [ perf record: Captured and wrote 0.087 MB perf.data (~3812 samples) ] 1084 [ perf record: Captured and wrote 0.087 MB perf.data (~3812 samples) ]
1085 1085
1086Using ``perf script`` we can see each do_fork event that fired:: 1086Using ``perf script`` we can see each ``do_fork`` event that fired::
1087 1087
1088 root@crownbay:~# perf script 1088 root@crownbay:~# perf script
1089 1089
@@ -1125,7 +1125,7 @@ Using ``perf script`` we can see each do_fork event that fired::
1125 gaku 1312 [000] 34237.202388: do_fork: (c1028460) 1125 gaku 1312 [000] 34237.202388: do_fork: (c1028460)
1126 1126
1127And using ``perf report`` on the same file, we can see the 1127And using ``perf report`` on the same file, we can see the
1128callgraphs from starting a few programs during those 30 seconds: 1128call graphs from starting a few programs during those 30 seconds:
1129 1129
1130.. image:: figures/perf-probe-do_fork-profile.png 1130.. image:: figures/perf-probe-do_fork-profile.png
1131 :align: center 1131 :align: center
@@ -1140,11 +1140,11 @@ callgraphs from starting a few programs during those 30 seconds:
1140 1140
1141.. admonition:: Tying it Together 1141.. admonition:: Tying it Together
1142 1142
1143 Dynamic tracepoints are implemented under the covers by kprobes and 1143 Dynamic tracepoints are implemented under the covers by Kprobes and
1144 uprobes. kprobes and uprobes are also used by and in fact are the 1144 Uprobes. Kprobes and Uprobes are also used by and in fact are the
1145 main focus of SystemTap. 1145 main focus of SystemTap.
1146 1146
1147Perf Documentation 1147perf Documentation
1148------------------ 1148------------------
1149 1149
1150Online versions of the manual pages for the commands discussed in this 1150Online versions of the manual pages for the commands discussed in this
@@ -1168,7 +1168,7 @@ section can be found here:
1168 1168
1169- The top-level `perf(1) manual page <https://linux.die.net/man/1/perf>`__. 1169- The top-level `perf(1) manual page <https://linux.die.net/man/1/perf>`__.
1170 1170
1171Normally, you should be able to invoke the manual pages via perf itself 1171Normally, you should be able to open the manual pages via perf itself
1172e.g. ``perf help`` or ``perf help record``. 1172e.g. ``perf help`` or ``perf help record``.
1173 1173
1174To have the perf manual pages installed on your target, modify your 1174To have the perf manual pages installed on your target, modify your
@@ -1183,14 +1183,14 @@ of examples, can also be found in the ``perf`` directory of the kernel tree::
1183 tools/perf/Documentation 1183 tools/perf/Documentation
1184 1184
1185There's also a nice perf tutorial on the perf 1185There's also a nice perf tutorial on the perf
1186wiki that goes into more detail than we do here in certain areas: `Perf 1186wiki that goes into more detail than we do here in certain areas: `perf
1187Tutorial <https://perf.wiki.kernel.org/index.php/Tutorial>`__ 1187Tutorial <https://perf.wiki.kernel.org/index.php/Tutorial>`__
1188 1188
1189ftrace 1189ftrace
1190====== 1190======
1191 1191
1192"ftrace" literally refers to the "ftrace function tracer" but in reality 1192"ftrace" literally refers to the "ftrace function tracer" but in reality
1193this encompasses a number of related tracers along with the 1193this encompasses several related tracers along with the
1194infrastructure that they all make use of. 1194infrastructure that they all make use of.
1195 1195
1196ftrace Setup 1196ftrace Setup
@@ -1199,11 +1199,11 @@ ftrace Setup
1199For this section, we'll assume you've already performed the basic setup 1199For this section, we'll assume you've already performed the basic setup
1200outlined in the ":ref:`profile-manual/intro:General Setup`" section. 1200outlined in the ":ref:`profile-manual/intro:General Setup`" section.
1201 1201
1202ftrace, trace-cmd, and kernelshark run on the target system, and are 1202ftrace, trace-cmd, and KernelShark run on the target system, and are
1203ready to go out-of-the-box --- no additional setup is necessary. For the 1203ready to go out-of-the-box --- no additional setup is necessary. For the
1204rest of this section we assume you've ssh'ed to the host and will be 1204rest of this section we assume you're connected to the host through SSH and
1205running ftrace on the target. kernelshark is a GUI application and if 1205will be running ftrace on the target. KernelShark is a GUI application and if
1206you use the ``-X`` option to ssh you can have the kernelshark GUI run on 1206you use the ``-X`` option to ``ssh`` you can have the KernelShark GUI run on
1207the target but display remotely on the host if you want. 1207the target but display remotely on the host if you want.
1208 1208
1209Basic ftrace usage 1209Basic ftrace usage
@@ -1211,8 +1211,8 @@ Basic ftrace usage
1211 1211
1212"ftrace" essentially refers to everything included in the ``/tracing`` 1212"ftrace" essentially refers to everything included in the ``/tracing``
1213directory of the mounted debugfs filesystem (Yocto follows the standard 1213directory of the mounted debugfs filesystem (Yocto follows the standard
1214convention and mounts it at ``/sys/kernel/debug``). Here's a listing of all 1214convention and mounts it at ``/sys/kernel/debug``). All the files found in
1215the files found in ``/sys/kernel/debug/tracing`` on a Yocto system:: 1215``/sys/kernel/debug/tracing`` on a Yocto system are::
1216 1216
1217 root@sugarbay:/sys/kernel/debug/tracing# ls 1217 root@sugarbay:/sys/kernel/debug/tracing# ls
1218 README kprobe_events trace 1218 README kprobe_events trace
@@ -1237,7 +1237,7 @@ the ftrace documentation.
1237 1237
1238We'll start by looking at some of the available built-in tracers. 1238We'll start by looking at some of the available built-in tracers.
1239 1239
1240cat'ing the ``available_tracers`` file lists the set of available tracers:: 1240The ``available_tracers`` file lists the set of available tracers::
1241 1241
1242 root@sugarbay:/sys/kernel/debug/tracing# cat available_tracers 1242 root@sugarbay:/sys/kernel/debug/tracing# cat available_tracers
1243 blk function_graph function nop 1243 blk function_graph function nop
@@ -1247,11 +1247,11 @@ The ``current_tracer`` file contains the tracer currently in effect::
1247 root@sugarbay:/sys/kernel/debug/tracing# cat current_tracer 1247 root@sugarbay:/sys/kernel/debug/tracing# cat current_tracer
1248 nop 1248 nop
1249 1249
1250The above listing of current_tracer shows that the 1250The above listing of ``current_tracer`` shows that the
1251``nop`` tracer is in effect, which is just another way of saying that 1251``nop`` tracer is in effect, which is just another way of saying that
1252there's actually no tracer currently in effect. 1252there's actually no tracer currently in effect.
1253 1253
1254echo'ing one of the available_tracers into ``current_tracer`` makes the 1254Writing one of the available tracers into ``current_tracer`` makes the
1255specified tracer the current tracer:: 1255specified tracer the current tracer::
1256 1256
1257 root@sugarbay:/sys/kernel/debug/tracing# echo function > current_tracer 1257 root@sugarbay:/sys/kernel/debug/tracing# echo function > current_tracer
@@ -1307,7 +1307,7 @@ tracer::
1307 . 1307 .
1308 1308
1309Each line in the trace above shows what was happening in the kernel on a given 1309Each line in the trace above shows what was happening in the kernel on a given
1310cpu, to the level of detail of function calls. Each entry shows the function 1310CPU, to the level of detail of function calls. Each entry shows the function
1311called, followed by its caller (after the arrow). 1311called, followed by its caller (after the arrow).
1312 1312
1313The function tracer gives you an extremely detailed idea of what the 1313The function tracer gives you an extremely detailed idea of what the
@@ -1321,7 +1321,7 @@ great way to learn about how the kernel code works in a dynamic sense.
1321 1321
1322It is a little more difficult to follow the call chains than it needs to 1322It is a little more difficult to follow the call chains than it needs to
1323be --- luckily there's a variant of the function tracer that displays the 1323be --- luckily there's a variant of the function tracer that displays the
1324callchains explicitly, called the ``function_graph`` tracer:: 1324call chains explicitly, called the ``function_graph`` tracer::
1325 1325
1326 root@sugarbay:/sys/kernel/debug/tracing# echo function_graph > current_tracer 1326 root@sugarbay:/sys/kernel/debug/tracing# echo function_graph > current_tracer
1327 root@sugarbay:/sys/kernel/debug/tracing# cat trace | less 1327 root@sugarbay:/sys/kernel/debug/tracing# cat trace | less
@@ -1440,7 +1440,7 @@ As you can see, the ``function_graph`` display is much easier
1440to follow. Also note that in addition to the function calls and 1440to follow. Also note that in addition to the function calls and
1441associated braces, other events such as scheduler events are displayed 1441associated braces, other events such as scheduler events are displayed
1442in context. In fact, you can freely include any tracepoint available in 1442in context. In fact, you can freely include any tracepoint available in
1443the trace events subsystem described in the next section by simply 1443the trace events subsystem described in the next section by just
1444enabling those events, and they'll appear in context in the function 1444enabling those events, and they'll appear in context in the function
1445graph display. Quite a powerful tool for understanding kernel dynamics. 1445graph display. Quite a powerful tool for understanding kernel dynamics.
1446 1446
@@ -1543,7 +1543,7 @@ The ``format`` file for the
1543tracepoint describes the event in memory, which is used by the various 1543tracepoint describes the event in memory, which is used by the various
1544tracing tools that now make use of these tracepoint to parse the event 1544tracing tools that now make use of these tracepoint to parse the event
1545and make sense of it, along with a ``print fmt`` field that allows tools 1545and make sense of it, along with a ``print fmt`` field that allows tools
1546like ftrace to display the event as text. Here's what the format of the 1546like ftrace to display the event as text. The format of the
1547``kmalloc`` event looks like:: 1547``kmalloc`` event looks like::
1548 1548
1549 root@sugarbay:/sys/kernel/debug/tracing/events/kmem/kmalloc# cat format 1549 root@sugarbay:/sys/kernel/debug/tracing/events/kmem/kmalloc# cat format
@@ -1596,7 +1596,7 @@ events in the output buffer::
1596 root@sugarbay:/sys/kernel/debug/tracing# echo 1 > tracing_on 1596 root@sugarbay:/sys/kernel/debug/tracing# echo 1 > tracing_on
1597 1597
1598Now, if we look at the ``trace`` file, we see nothing 1598Now, if we look at the ``trace`` file, we see nothing
1599but the kmalloc events we just turned on:: 1599but the ``kmalloc`` events we just turned on::
1600 1600
1601 root@sugarbay:/sys/kernel/debug/tracing# cat trace | less 1601 root@sugarbay:/sys/kernel/debug/tracing# cat trace | less
1602 # tracer: nop 1602 # tracer: nop
@@ -1651,8 +1651,8 @@ using the ``enable`` file in the subsystem directory) and get an
1651arbitrarily fine-grained idea of what's going on in the system by 1651arbitrarily fine-grained idea of what's going on in the system by
1652enabling as many of the appropriate tracepoints as applicable. 1652enabling as many of the appropriate tracepoints as applicable.
1653 1653
1654A number of the tools described in this HOWTO do just that, including 1654Several tools described in this How-to do just that, including
1655``trace-cmd`` and kernelshark in the next section. 1655``trace-cmd`` and KernelShark in the next section.
1656 1656
1657.. admonition:: Tying it Together 1657.. admonition:: Tying it Together
1658 1658
@@ -1668,7 +1668,7 @@ A number of the tools described in this HOWTO do just that, including
1668 ``/sys/kernel/debug/tracing`` will be removed and replaced with 1668 ``/sys/kernel/debug/tracing`` will be removed and replaced with
1669 equivalent tracers based on the "trace events" subsystem. 1669 equivalent tracers based on the "trace events" subsystem.
1670 1670
1671trace-cmd / kernelshark 1671trace-cmd / KernelShark
1672----------------------- 1672-----------------------
1673 1673
1674trace-cmd is essentially an extensive command-line "wrapper" interface 1674trace-cmd is essentially an extensive command-line "wrapper" interface
@@ -1677,24 +1677,23 @@ that hides the details of all the individual files in
1677events within the ``/sys/kernel/debug/tracing/events/`` subdirectory and to 1677events within the ``/sys/kernel/debug/tracing/events/`` subdirectory and to
1678collect traces and avoid having to deal with those details directly. 1678collect traces and avoid having to deal with those details directly.
1679 1679
1680As yet another layer on top of that, kernelshark provides a GUI that 1680As yet another layer on top of that, KernelShark provides a GUI that
1681allows users to start and stop traces and specify sets of events using 1681allows users to start and stop traces and specify sets of events using
1682an intuitive interface, and view the output as both trace events and as 1682an intuitive interface, and view the output as both trace events and as
1683a per-CPU graphical display. It directly uses trace-cmd as the 1683a per-CPU graphical display. It directly uses trace-cmd as the
1684plumbing that accomplishes all that underneath the covers (and actually 1684plumbing that accomplishes all that underneath the covers (and actually
1685displays the trace-cmd command it uses, as we'll see). 1685displays the trace-cmd command it uses, as we'll see).
1686 1686
1687To start a trace using kernelshark, first start kernelshark:: 1687To start a trace using KernelShark, first start this tool::
1688 1688
1689 root@sugarbay:~# kernelshark 1689 root@sugarbay:~# kernelshark
1690 1690
1691Then bring up the ``Capture`` dialog by 1691Then open up the ``Capture`` dialog by choosing from the KernelShark menu::
1692choosing from the kernelshark menu::
1693 1692
1694 Capture | Record 1693 Capture | Record
1695 1694
1696That will display the following dialog, which allows you to choose one or more 1695That will display the following dialog, which allows you to choose one or more
1697events (or even one or more complete subsystems) to trace: 1696events (or even entire subsystems) to trace:
1698 1697
1699.. image:: figures/kernelshark-choose-events.png 1698.. image:: figures/kernelshark-choose-events.png
1700 :align: center 1699 :align: center
@@ -1702,7 +1701,7 @@ events (or even one or more complete subsystems) to trace:
1702 1701
1703Note that these are exactly the same sets of events described in the 1702Note that these are exactly the same sets of events described in the
1704previous trace events subsystem section, and in fact is where trace-cmd 1703previous trace events subsystem section, and in fact is where trace-cmd
1705gets them for kernelshark. 1704gets them for KernelShark.
1706 1705
1707In the above screenshot, we've decided to explore the graphics subsystem 1706In the above screenshot, we've decided to explore the graphics subsystem
1708a bit and so have chosen to trace all the tracepoints contained within 1707a bit and so have chosen to trace all the tracepoints contained within
@@ -1716,12 +1715,12 @@ will turn into the 'Stop' button after the trace has started):
1716 :align: center 1715 :align: center
1717 :width: 70% 1716 :width: 70%
1718 1717
1719Notice that the right-hand pane shows the exact trace-cmd command-line 1718Notice that the right pane shows the exact trace-cmd command-line
1720that's used to run the trace, along with the results of the trace-cmd 1719that's used to run the trace, along with the results of the trace-cmd
1721run. 1720run.
1722 1721
1723Once the ``Stop`` button is pressed, the graphical view magically fills up 1722Once the ``Stop`` button is pressed, the graphical view magically fills up
1724with a colorful per-cpu display of the trace data, along with the 1723with a colorful per-CPU display of the trace data, along with the
1725detailed event listing below that: 1724detailed event listing below that:
1726 1725
1727.. image:: figures/kernelshark-i915-display.png 1726.. image:: figures/kernelshark-i915-display.png
@@ -1736,7 +1735,7 @@ events``:
1736 :width: 70% 1735 :width: 70%
1737 1736
1738The tool is pretty self-explanatory, but for more detailed information 1737The tool is pretty self-explanatory, but for more detailed information
1739on navigating through the data, see the `kernelshark 1738on navigating through the data, see the `KernelShark
1740website <https://kernelshark.org/Documentation.html>`__. 1739website <https://kernelshark.org/Documentation.html>`__.
1741 1740
1742ftrace Documentation 1741ftrace Documentation
@@ -1752,41 +1751,41 @@ Documentation directory::
1752 1751
1753 Documentation/trace/events.txt 1752 Documentation/trace/events.txt
1754 1753
1755There is a nice series of articles on using ftrace and trace-cmd at LWN: 1754A nice series of articles on using ftrace and trace-cmd are available at LWN:
1756 1755
1757- `Debugging the kernel using Ftrace - part 1756- `Debugging the kernel using ftrace - part
1758 1 <https://lwn.net/Articles/365835/>`__ 1757 1 <https://lwn.net/Articles/365835/>`__
1759 1758
1760- `Debugging the kernel using Ftrace - part 1759- `Debugging the kernel using ftrace - part
1761 2 <https://lwn.net/Articles/366796/>`__ 1760 2 <https://lwn.net/Articles/366796/>`__
1762 1761
1763- `Secrets of the Ftrace function 1762- `Secrets of the ftrace function
1764 tracer <https://lwn.net/Articles/370423/>`__ 1763 tracer <https://lwn.net/Articles/370423/>`__
1765 1764
1766- `trace-cmd: A front-end for 1765- `trace-cmd: A front-end for
1767 Ftrace <https://lwn.net/Articles/410200/>`__ 1766 ftrace <https://lwn.net/Articles/410200/>`__
1768 1767
1769See also `KernelShark's documentation <https://kernelshark.org/Documentation.html>`__ 1768See also `KernelShark's documentation <https://kernelshark.org/Documentation.html>`__
1770for further usage details. 1769for further usage details.
1771 1770
1772An amusing yet useful README (a tracing mini-HOWTO) can be found in 1771An amusing yet useful README (a tracing mini-How-to) can be found in
1773``/sys/kernel/debug/tracing/README``. 1772``/sys/kernel/debug/tracing/README``.
1774 1773
1775systemtap 1774SystemTap
1776========= 1775=========
1777 1776
1778SystemTap is a system-wide script-based tracing and profiling tool. 1777SystemTap is a system-wide script-based tracing and profiling tool.
1779 1778
1780SystemTap scripts are C-like programs that are executed in the kernel to 1779SystemTap scripts are C-like programs that are executed in the kernel to
1781gather / print / aggregate data extracted from the context they end up being 1780gather / print / aggregate data extracted from the context they end up being
1782invoked under. 1781called under.
1783 1782
1784For example, this probe from the `SystemTap 1783For example, this probe from the `SystemTap
1785tutorial <https://sourceware.org/systemtap/tutorial/>`__ simply prints a 1784tutorial <https://sourceware.org/systemtap/tutorial/>`__ just prints a
1786line every time any process on the system runs ``open()`` on a file. For each line, 1785line every time any process on the system runs ``open()`` on a file. For each line,
1787it prints the executable name of the program that opened the file, along 1786it prints the executable name of the program that opened the file, along
1788with its PID, and the name of the file it opened (or tried to open), 1787with its PID, and the name of the file it opened (or tried to open), which it
1789which it extracts from the open syscall's argstr. 1788extracts from the argument string (``argstr``) of the ``open`` system call.
1790 1789
1791.. code-block:: none 1790.. code-block:: none
1792 1791
@@ -1801,13 +1800,13 @@ which it extracts from the open syscall's argstr.
1801 } 1800 }
1802 1801
1803Normally, to execute this 1802Normally, to execute this
1804probe, you'd simply install systemtap on the system you want to probe, 1803probe, you'd just install SystemTap on the system you want to probe,
1805and directly run the probe on that system e.g. assuming the name of the 1804and directly run the probe on that system e.g. assuming the name of the
1806file containing the above text is trace_open.stp:: 1805file containing the above text is ``trace_open.stp``::
1807 1806
1808 # stap trace_open.stp 1807 # stap trace_open.stp
1809 1808
1810What systemtap does under the covers to run this probe is 1) parse and 1809What SystemTap does under the covers to run this probe is 1) parse and
1811convert the probe to an equivalent "C" form, 2) compile the "C" form 1810convert the probe to an equivalent "C" form, 2) compile the "C" form
1812into a kernel module, 3) insert the module into the kernel, which arms 1811into a kernel module, 3) insert the module into the kernel, which arms
1813it, and 4) collect the data generated by the probe and display it to the 1812it, and 4) collect the data generated by the probe and display it to the
@@ -1820,25 +1819,25 @@ kernel build system unfortunately isn't typically part of the image
1820running on the target. It is normally available on the "host" system 1819running on the target. It is normally available on the "host" system
1821that produced the target image however; in such cases, steps 1 and 2 are 1820that produced the target image however; in such cases, steps 1 and 2 are
1822executed on the host system, and steps 3 and 4 are executed on the 1821executed on the host system, and steps 3 and 4 are executed on the
1823target system, using only the systemtap "runtime". 1822target system, using only the SystemTap "runtime".
1824 1823
1825The systemtap support in Yocto assumes that only steps 3 and 4 are run 1824The SystemTap support in Yocto assumes that only steps 3 and 4 are run
1826on the target; it is possible to do everything on the target, but this 1825on the target; it is possible to do everything on the target, but this
1827section assumes only the typical embedded use-case. 1826section assumes only the typical embedded use-case.
1828 1827
1829So basically what you need to do in order to run a systemtap script on 1828Therefore, what you need to do in order to run a SystemTap script on
1830the target is to 1) on the host system, compile the probe into a kernel 1829the target is to 1) on the host system, compile the probe into a kernel
1831module that makes sense to the target, 2) copy the module onto the 1830module that makes sense to the target, 2) copy the module onto the
1832target system and 3) insert the module into the target kernel, which 1831target system and 3) insert the module into the target kernel, which
1833arms it, and 4) collect the data generated by the probe and display it 1832arms it, and 4) collect the data generated by the probe and display it
1834to the user. 1833to the user.
1835 1834
1836systemtap Setup 1835SystemTap Setup
1837--------------- 1836---------------
1838 1837
1839Those are a lot of steps and a lot of details, but fortunately Yocto 1838Those are many steps and details, but fortunately Yocto
1840includes a script called ``crosstap`` that will take care of those 1839includes a script called ``crosstap`` that will take care of those
1841details, allowing you to simply execute a systemtap script on the remote 1840details, allowing you to just execute a SystemTap script on the remote
1842target, with arguments if necessary. 1841target, with arguments if necessary.
1843 1842
1844In order to do this from a remote host, however, you need to have access 1843In order to do this from a remote host, however, you need to have access
@@ -1851,7 +1850,7 @@ having done a build::
1851 Error: No target kernel build found. 1850 Error: No target kernel build found.
1852 Did you forget to create a local build of your image? 1851 Did you forget to create a local build of your image?
1853 1852
1854'crosstap' requires a local sdk build of the target system 1853'crosstap' requires a local SDK build of the target system
1855(or a build that includes 'tools-profile') in order to build 1854(or a build that includes 'tools-profile') in order to build
1856kernel modules that can probe the target system. 1855kernel modules that can probe the target system.
1857 1856
@@ -1867,11 +1866,11 @@ Practically speaking, that means you need to do the following:
1867 the BSP README and/or the widely available basic documentation 1866 the BSP README and/or the widely available basic documentation
1868 that discusses how to build images). 1867 that discusses how to build images).
1869 1868
1870- Build an -sdk version of the image e.g.:: 1869- Build an ``-sdk`` version of the image e.g.::
1871 1870
1872 $ bitbake core-image-sato-sdk 1871 $ bitbake core-image-sato-sdk
1873 1872
1874- Or build a non-sdk image but include the profiling tools 1873- Or build a non-SDK image but include the profiling tools
1875 (edit ``local.conf`` and add ``tools-profile`` to the end of 1874 (edit ``local.conf`` and add ``tools-profile`` to the end of
1876 :term:``EXTRA_IMAGE_FEATURES`` variable):: 1875 :term:``EXTRA_IMAGE_FEATURES`` variable)::
1877 1876
@@ -1887,15 +1886,14 @@ Practically speaking, that means you need to do the following:
1887 1886
1888.. note:: 1887.. note::
1889 1888
1890 SystemTap, which uses ``crosstap``, assumes you can establish an ssh 1889 SystemTap, which uses ``crosstap``, assumes you can establish an SSH
1891 connection to the remote target. Please refer to the crosstap wiki 1890 connection to the remote target. Please refer to the crosstap wiki
1892 page for details on verifying ssh connections. Also, the ability to ssh 1891 page for details on verifying SSH connections. Also, the ability to SSH
1893 into the target system is not enabled by default in ``*-minimal`` images. 1892 into the target system is not enabled by default in ``*-minimal`` images.
1894 1893
1895So essentially what you need to 1894Therefore, what you need to do is build an SDK image or image with
1896do is build an SDK image or image with ``tools-profile`` as detailed in 1895``tools-profile`` as detailed in the ":ref:`profile-manual/intro:General Setup`"
1897the ":ref:`profile-manual/intro:General Setup`" section of this 1896section of this manual, and boot the resulting target image.
1898manual, and boot the resulting target image.
1899 1897
1900.. note:: 1898.. note::
1901 1899
@@ -1903,12 +1901,12 @@ manual, and boot the resulting target image.
1903 to have the :term:`MACHINE` you're connecting to selected in ``local.conf``, and 1901 to have the :term:`MACHINE` you're connecting to selected in ``local.conf``, and
1904 the kernel in that machine's :term:`Build Directory` must match the kernel on 1902 the kernel in that machine's :term:`Build Directory` must match the kernel on
1905 the booted system exactly, or you'll get the above ``crosstap`` message 1903 the booted system exactly, or you'll get the above ``crosstap`` message
1906 when you try to invoke a script. 1904 when you try to call a script.
1907 1905
1908Running a Script on a Target 1906Running a Script on a Target
1909---------------------------- 1907----------------------------
1910 1908
1911Once you've done that, you should be able to run a systemtap script on 1909Once you've done that, you should be able to run a SystemTap script on
1912the target:: 1910the target::
1913 1911
1914 $ cd /path/to/yocto 1912 $ cd /path/to/yocto
@@ -1937,7 +1935,7 @@ If you get an error connecting to the target e.g.::
1937 $ crosstap root@192.168.7.2 trace_open.stp 1935 $ crosstap root@192.168.7.2 trace_open.stp
1938 error establishing ssh connection on remote 'root@192.168.7.2' 1936 error establishing ssh connection on remote 'root@192.168.7.2'
1939 1937
1940Try ssh'ing to the target and see what happens:: 1938Try connecting to the target through SSH and see what happens::
1941 1939
1942 $ ssh root@192.168.7.2 1940 $ ssh root@192.168.7.2
1943 1941
@@ -1955,7 +1953,7 @@ no password):
1955 matchbox-termin(1036) open ("/tmp/vte3FS2LW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) 1953 matchbox-termin(1036) open ("/tmp/vte3FS2LW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600)
1956 matchbox-termin(1036) open ("/tmp/vteJMC7LW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) 1954 matchbox-termin(1036) open ("/tmp/vteJMC7LW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600)
1957 1955
1958systemtap Documentation 1956SystemTap Documentation
1959----------------------- 1957-----------------------
1960 1958
1961The SystemTap language reference can be found here: `SystemTap Language 1959The SystemTap language reference can be found here: `SystemTap Language
@@ -1968,7 +1966,7 @@ page <https://sourceware.org/systemtap/documentation.html>`__
1968Sysprof 1966Sysprof
1969======= 1967=======
1970 1968
1971Sysprof is a very easy to use system-wide profiler that consists of a 1969Sysprof is an easy to use system-wide profiler that consists of a
1972single window with three panes and a few buttons which allow you to 1970single window with three panes and a few buttons which allow you to
1973start, stop, and view the profile from one place. 1971start, stop, and view the profile from one place.
1974 1972
@@ -1978,16 +1976,16 @@ Sysprof Setup
1978For this section, we'll assume you've already performed the basic setup 1976For this section, we'll assume you've already performed the basic setup
1979outlined in the ":ref:`profile-manual/intro:General Setup`" section. 1977outlined in the ":ref:`profile-manual/intro:General Setup`" section.
1980 1978
1981Sysprof is a GUI-based application that runs on the target system. For 1979Sysprof is a GUI-based application that runs on the target system. For the rest
1982the rest of this document we assume you've ssh'ed to the host and will 1980of this document we assume you're connected to the host through SSH and will be
1983be running Sysprof on the target (you can use the ``-X`` option to ssh and 1981running Sysprof on the target (you can use the ``-X`` option to ``ssh`` and
1984have the Sysprof GUI run on the target but display remotely on the host 1982have the Sysprof GUI run on the target but display remotely on the host
1985if you want). 1983if you want).
1986 1984
1987Basic Sysprof Usage 1985Basic Sysprof Usage
1988------------------- 1986-------------------
1989 1987
1990To start profiling the system, you simply press the ``Start`` button. To 1988To start profiling the system, you just press the ``Start`` button. To
1991stop profiling and to start viewing the profile data in one easy step, 1989stop profiling and to start viewing the profile data in one easy step,
1992press the ``Profile`` button. 1990press the ``Profile`` button.
1993 1991
@@ -2001,11 +1999,11 @@ with profiling data:
2001The left pane shows a list of functions and processes. Selecting one of 1999The left pane shows a list of functions and processes. Selecting one of
2002those expands that function in the right pane, showing all its callees. 2000those expands that function in the right pane, showing all its callees.
2003Note that this caller-oriented display is essentially the inverse of 2001Note that this caller-oriented display is essentially the inverse of
2004perf's default callee-oriented callchain display. 2002perf's default callee-oriented call chain display.
2005 2003
2006In the screenshot above, we're focusing on ``__copy_to_user_ll()`` and 2004In the screenshot above, we're focusing on ``__copy_to_user_ll()`` and
2007looking up the callchain we can see that one of the callers of 2005looking up the call chain we can see that one of the callers of
2008``__copy_to_user_ll`` is ``sys_read()`` and the complete callpath between them. 2006``__copy_to_user_ll`` is ``sys_read()`` and the complete call path between them.
2009Notice that this is essentially a portion of the same information we saw 2007Notice that this is essentially a portion of the same information we saw
2010in the perf display shown in the perf section of this page. 2008in the perf display shown in the perf section of this page.
2011 2009
@@ -2014,7 +2012,7 @@ in the perf display shown in the perf section of this page.
2014 :width: 70% 2012 :width: 70%
2015 2013
2016Similarly, the above is a snapshot of the Sysprof display of a 2014Similarly, the above is a snapshot of the Sysprof display of a
2017``copy-from-user`` callchain. 2015``copy-from-user`` call chain.
2018 2016
2019Finally, looking at the third Sysprof pane in the lower left, we can see 2017Finally, looking at the third Sysprof pane in the lower left, we can see
2020a list of all the callers of a particular function selected in the top 2018a list of all the callers of a particular function selected in the top
@@ -2030,18 +2028,17 @@ to the selected function, and so on.
2030 2028
2031.. admonition:: Tying it Together 2029.. admonition:: Tying it Together
2032 2030
2033 If you like sysprof's ``caller-oriented`` display, you may be able to 2031 If you like Sysprof's ``caller-oriented`` display, you may be able to
2034 approximate it in other tools as well. For example, ``perf report`` has 2032 approximate it in other tools as well. For example, ``perf report`` has
2035 the ``-g`` (``--call-graph``) option that you can experiment with; one of the 2033 the ``-g`` (``--call-graph``) option that you can experiment with; one of the
2036 options is ``caller`` for an inverted caller-based callgraph display. 2034 options is ``caller`` for an inverted caller-based call graph display.
2037 2035
2038Sysprof Documentation 2036Sysprof Documentation
2039--------------------- 2037---------------------
2040 2038
2041There doesn't seem to be any documentation for Sysprof, but maybe that's 2039There doesn't seem to be any documentation for Sysprof, but maybe that's
2042because it's pretty self-explanatory. The Sysprof website, however, is 2040because it's pretty self-explanatory. The Sysprof website, however, is here:
2043here: `Sysprof, System-wide Performance Profiler for 2041`Sysprof, System-wide Performance Profiler for Linux <http://sysprof.com/>`__
2044Linux <http://sysprof.com/>`__
2045 2042
2046LTTng (Linux Trace Toolkit, next generation) 2043LTTng (Linux Trace Toolkit, next generation)
2047============================================ 2044============================================
@@ -2051,7 +2048,7 @@ LTTng Setup
2051 2048
2052For this section, we'll assume you've already performed the basic setup 2049For this section, we'll assume you've already performed the basic setup
2053outlined in the ":ref:`profile-manual/intro:General Setup`" section. 2050outlined in the ":ref:`profile-manual/intro:General Setup`" section.
2054LTTng is run on the target system by ssh'ing to it. 2051LTTng is run on the target system by connecting to it through SSH.
2055 2052
2056Collecting and Viewing Traces 2053Collecting and Viewing Traces
2057----------------------------- 2054-----------------------------
@@ -2064,7 +2061,7 @@ tracing.
2064Collecting and viewing a trace on the target (inside a shell) 2061Collecting and viewing a trace on the target (inside a shell)
2065~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2062~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2066 2063
2067First, from the host, ssh to the target:: 2064First, from the host, connect to the target through SSH::
2068 2065
2069 $ ssh -l root 192.168.1.47 2066 $ ssh -l root 192.168.1.47
2070 The authenticity of host '192.168.1.47 (192.168.1.47)' can't be established. 2067 The authenticity of host '192.168.1.47 (192.168.1.47)' can't be established.
@@ -2156,11 +2153,11 @@ supplying your own name to ``lttng create``)::
2156 drwxr-xr-x 5 root root 1024 Oct 15 23:57 .. 2153 drwxr-xr-x 5 root root 1024 Oct 15 23:57 ..
2157 drwxrwx--- 3 root root 1024 Oct 15 23:21 auto-20121015-232120 2154 drwxrwx--- 3 root root 1024 Oct 15 23:21 auto-20121015-232120
2158 2155
2159Collecting and viewing a userspace trace on the target (inside a shell) 2156Collecting and viewing a user space trace on the target (inside a shell)
2160~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2157~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2161 2158
2162For LTTng userspace tracing, you need to have a properly instrumented 2159For LTTng user space tracing, you need to have a properly instrumented
2163userspace program. For this example, we'll use the ``hello`` test program 2160user space program. For this example, we'll use the ``hello`` test program
2164generated by the ``lttng-ust`` build. 2161generated by the ``lttng-ust`` build.
2165 2162
2166The ``hello`` test program isn't installed on the root filesystem by the ``lttng-ust`` 2163The ``hello`` test program isn't installed on the root filesystem by the ``lttng-ust``
@@ -2176,7 +2173,7 @@ Copy that over to the target machine::
2176You now have the instrumented LTTng "hello world" test program on the 2173You now have the instrumented LTTng "hello world" test program on the
2177target, ready to test. 2174target, ready to test.
2178 2175
2179First, from the host, ssh to the target:: 2176First, from the host, connect to the target through SSH::
2180 2177
2181 $ ssh -l root 192.168.1.47 2178 $ ssh -l root 192.168.1.47
2182 The authenticity of host '192.168.1.47 (192.168.1.47)' can't be established. 2179 The authenticity of host '192.168.1.47 (192.168.1.47)' can't be established.
@@ -2191,7 +2188,7 @@ Once on the target, use these steps to create a trace::
2191 Session auto-20190303-021943 created. 2188 Session auto-20190303-021943 created.
2192 Traces will be written in /home/root/lttng-traces/auto-20190303-021943 2189 Traces will be written in /home/root/lttng-traces/auto-20190303-021943
2193 2190
2194Enable the events you want to trace (in this case all userspace events):: 2191Enable the events you want to trace (in this case all user space events)::
2195 2192
2196 root@crownbay:~# lttng enable-event --userspace --all 2193 root@crownbay:~# lttng enable-event --userspace --all
2197 All UST events are enabled in channel channel0 2194 All UST events are enabled in channel channel0
@@ -2263,13 +2260,13 @@ the entire blktrace and blkparse pipeline on the target, or you can run
2263blktrace in 'listen' mode on the target and have blktrace and blkparse 2260blktrace in 'listen' mode on the target and have blktrace and blkparse
2264collect and analyze the data on the host (see the 2261collect and analyze the data on the host (see the
2265":ref:`profile-manual/usage:Using blktrace Remotely`" section 2262":ref:`profile-manual/usage:Using blktrace Remotely`" section
2266below). For the rest of this section we assume you've ssh'ed to the host and 2263below). For the rest of this section we assume you've to the host through SSH
2267will be running blkrace on the target. 2264and will be running blktrace on the target.
2268 2265
2269Basic blktrace Usage 2266Basic blktrace Usage
2270-------------------- 2267--------------------
2271 2268
2272To record a trace, simply run the ``blktrace`` command, giving it the name 2269To record a trace, just run the ``blktrace`` command, giving it the name
2273of the block device you want to trace activity on:: 2270of the block device you want to trace activity on::
2274 2271
2275 root@crownbay:~# blktrace /dev/sdc 2272 root@crownbay:~# blktrace /dev/sdc
@@ -2280,10 +2277,10 @@ In another shell, execute a workload you want to trace::
2280 Connecting to downloads.yoctoproject.org (140.211.169.59:80) 2277 Connecting to downloads.yoctoproject.org (140.211.169.59:80)
2281 linux-2.6.19.2.tar.b 100% \|*******************************\| 41727k 0:00:00 ETA 2278 linux-2.6.19.2.tar.b 100% \|*******************************\| 41727k 0:00:00 ETA
2282 2279
2283Press Ctrl-C in the blktrace shell to stop the trace. It 2280Press ``Ctrl-C`` in the blktrace shell to stop the trace. It
2284will display how many events were logged, along with the per-cpu file 2281will display how many events were logged, along with the per-cpu file
2285sizes (blktrace records traces in per-cpu kernel buffers and simply 2282sizes (blktrace records traces in per-cpu kernel buffers and just
2286dumps them to userspace for blkparse to merge and sort later):: 2283dumps them to user space for blkparse to merge and sort later)::
2287 2284
2288 ^C=== sdc === 2285 ^C=== sdc ===
2289 CPU 0: 7082 events, 332 KiB data 2286 CPU 0: 7082 events, 332 KiB data
@@ -2299,7 +2296,7 @@ with the device name as the first part of the filename::
2299 -rw-r--r-- 1 root root 339938 Oct 27 22:40 sdc.blktrace.0 2296 -rw-r--r-- 1 root root 339938 Oct 27 22:40 sdc.blktrace.0
2300 -rw-r--r-- 1 root root 75753 Oct 27 22:40 sdc.blktrace.1 2297 -rw-r--r-- 1 root root 75753 Oct 27 22:40 sdc.blktrace.1
2301 2298
2302To view the trace events, simply invoke ``blkparse`` in the directory 2299To view the trace events, just call ``blkparse`` in the directory
2303containing the trace files, giving it the device name that forms the 2300containing the trace files, giving it the device name that forms the
2304first part of the filenames:: 2301first part of the filenames::
2305 2302
@@ -2398,8 +2395,8 @@ Live Mode
2398~~~~~~~~~ 2395~~~~~~~~~
2399 2396
2400blktrace and blkparse are designed from the ground up to be able to 2397blktrace and blkparse are designed from the ground up to be able to
2401operate together in a "pipe mode" where the stdout of blktrace can be 2398operate together in a "pipe mode" where the standard output of blktrace can be
2402fed directly into the stdin of blkparse:: 2399fed directly into the standard input of blkparse::
2403 2400
2404 root@crownbay:~# blktrace /dev/sdc -o - | blkparse -i - 2401 root@crownbay:~# blktrace /dev/sdc -o - | blkparse -i -
2405 2402
@@ -2468,7 +2465,7 @@ just ended::
2468 Total: 11800 events (dropped 0), 554 KiB data 2465 Total: 11800 events (dropped 0), 554 KiB data
2469 2466
2470The blktrace instance on the host will 2467The blktrace instance on the host will
2471save the target output inside a hostname-timestamp directory:: 2468save the target output inside a ``<hostname>-<timestamp>`` directory::
2472 2469
2473 $ ls -al 2470 $ ls -al
2474 drwxr-xr-x 10 root root 1024 Oct 28 02:40 . 2471 drwxr-xr-x 10 root root 1024 Oct 28 02:40 .
@@ -2540,7 +2537,7 @@ Tracing Block I/O via 'ftrace'
2540It's also possible to trace block I/O using only 2537It's also possible to trace block I/O using only
2541:ref:`profile-manual/usage:The 'trace events' Subsystem`, which 2538:ref:`profile-manual/usage:The 'trace events' Subsystem`, which
2542can be useful for casual tracing if you don't want to bother dealing with the 2539can be useful for casual tracing if you don't want to bother dealing with the
2543userspace tools. 2540user space tools.
2544 2541
2545To enable tracing for a given device, use ``/sys/block/xxx/trace/enable``, 2542To enable tracing for a given device, use ``/sys/block/xxx/trace/enable``,
2546where ``xxx`` is the device name. This for example enables tracing for 2543where ``xxx`` is the device name. This for example enables tracing for
@@ -2598,7 +2595,7 @@ section can be found here:
2598- https://linux.die.net/man/8/btrace 2595- https://linux.die.net/man/8/btrace
2599 2596
2600The above manual pages, along with manuals for the other blktrace utilities 2597The above manual pages, along with manuals for the other blktrace utilities
2601(btt, blkiomon, etc) can be found in the ``/doc`` directory of the blktrace 2598(``btt``, ``blkiomon``, etc) can be found in the ``/doc`` directory of the blktrace
2602tools git repo:: 2599tools git repository::
2603 2600
2604 $ git clone git://git.kernel.dk/blktrace.git 2601 $ git clone git://git.kernel.dk/blktrace.git
diff --git a/documentation/styles/config/vocabularies/OpenSource/accept.txt b/documentation/styles/config/vocabularies/OpenSource/accept.txt
index 98e76ae1f5..e378fbf79b 100644
--- a/documentation/styles/config/vocabularies/OpenSource/accept.txt
+++ b/documentation/styles/config/vocabularies/OpenSource/accept.txt
@@ -1,4 +1,20 @@
1autovivification
2blkparse
3blktrace
4callee
5debugfs
1ftrace 6ftrace
2toolchain 7KernelShark
3systemd 8Kprobe
4LTTng 9LTTng
10perf
11profiler
12subcommand
13subnode
14superset
15Sysprof
16systemd
17toolchain
18tracepoint
19Uprobe
20wget
diff --git a/documentation/styles/config/vocabularies/Yocto/accept.txt b/documentation/styles/config/vocabularies/Yocto/accept.txt
index b725414014..ca622ba412 100644
--- a/documentation/styles/config/vocabularies/Yocto/accept.txt
+++ b/documentation/styles/config/vocabularies/Yocto/accept.txt
@@ -1,4 +1,5 @@
1Yocto
2BSP
3BitBake 1BitBake
2BSP
3crosstap
4OpenEmbedded 4OpenEmbedded
5Yocto