docs: add workload-tracing document to admin-guide
Shuah Khan
reviewerPATCH v2 1/1 docs: add workload-tracing document to admin-guide
Add a new section to the admin-guide with information of interest to application developers and system integrators doing analysis of the Linux kernel for safety critical applications.
This section will contain documents supporting analysis of kernel interactions with applications, and key kernel subsystems expectations.
Add a new workload-tracing document to this new section.
---
Changes since v1:
- Addressed review comments on v1 on long lines, rst syntax, license
Documentation/admin-guide/index.rst | 11 + .../admin-guide/workload-tracing.rst | 591 ++++++++++++++++++ 2 files changed, 602 insertions(+) create mode 100644 Documentation/admin-guide/workload-tracing.rst
Documentation/admin-guide/index.rst +11 -0
@@ -56,6 +56,17 @@ ABI will be found here. sysfs-rules +This is the beginning of a section with information of interest to +application developers and system integrators doing analysis of the +Linux kernel for safety critical applications. Documents supporting +analysis of kernel interactions with applications, and key kernel +subsystems expectations will be found here. + +.. toctree:: + :maxdepth: 1 + + workload-tracing + The rest of this manual consists of various unordered guides on how to configure specific aspects of kernel behavior to your liking.
Documentation/admin-guide/workload-tracing.rst new file +591 -0
591 lines not on this page · raw message
Signature · 1 line
2.34.1
Documentation/admin-guide/workload-tracing.rst line 52
Quoted 2 lines · Shuah Khan
Documentation/admin-guide/workload-tracing.rst new file +2
-0+strace tool can be used to trace system calls made by a process and signals +it receives. a process to the perf, stress-ng, paxtest workloads. System
"... to trace system calls made by a process (which can be perf, stress-ng, paxtest workloads, or neither)."
Quoted 1 line
+Before we can get started we will have to get our system ready. We assume
"Before we can get started the system must be prepared first."
Quoted 1 line
+Browsing kernel sources ::
"To browse kernel sources, you will need:"
Documentation/admin-guide/workload-tracing.rst line 95
Quoted 4 lines
Documentation/admin-guide/workload-tracing.rst new file +4
-0+Workload overview +================= + +We used strace to trace perf bench, stress-ng and paxtest workloads to
"In this section, we use strace to trace ..."
Documentation/admin-guide/workload-tracing.rst line 98
Quoted 3 lines
Documentation/admin-guide/workload-tracing.rst new file +3
-0+show how to analyze a workload and identify Linux subsystems used by +these workloads. We hope this process can be applied to trace workload(s). +We will go over the workloads first.
"This process can also be applied to trace other workloads."
Quoted 5 lines
Documentation/admin-guide/workload-tracing.rst new file +4
-0+ +perf bench (all) workload: +-------------------------- +
Documentation/admin-guide/workload-tracing.rst line 125
Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +8
-0+ +Stress-ng netdev stressor workload: +----------------------------------- + + +paxtest kiddie workload: +------------------------ +
Drop trailing colon from these subsection texts.
Quoted 1 line
+We used the “-c” option to gather fine grained information.
``` The basic usage is strace <command>. To run verbose tracing, specify -v to strace. To generate the report, specify -c option. ```
Documentation/admin-guide/workload-tracing.rst line 178
Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +10
-0+Now let’s look at `cscope <https://cscope.sourceforge.net/>`_, a command +line tool for browsing C, C++ or Java code-bases. We can use it to find +all the references to a symbol, global definitions, functions called by a +function, functions calling a function, text strings, regular expression +patterns, files including a file. + +We can use cscope to find which system call belongs to which subsystem. +This way we can find the kernel subsystems used by a process when it is +executed. To use it navigate to the source code directory. Here we are +analyzing the kernel source tree.
I guess "you" is better fit for the wording above?
Documentation/admin-guide/workload-tracing.rst line 215
Quoted 2 lines
Documentation/admin-guide/workload-tracing.rst new file +2
-0+We used perf stat and perf bench options. For a detailed information on the +perf tool, run perf -h.
"In this section, we highlight stat and bench options. For help on other options, see perf -h."
Quoted 1 line
+**perf bench all** command runs the following benchmarks: ::
Either don't markup that command or inline it (``perf bench all``). Also, use bullet list for benchmarking list below.
Quoted 5 lines
Documentation/admin-guide/workload-tracing.rst new file +5
-0+ + sched/messaging + sched/pipe + syscall/basic + mem/memcpy
Documentation/admin-guide/workload-tracing.rst line 257
Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +9
-0+ mem/memset + +The netdev stressor starts N workers that exercise various netdevice ioctl +commands across all the available network devices. The following ioctls are +exercised: :: + + SIOCGIFCONF, SIOCGIFINDEX, SIOCGIFNAME, SIOCGIFFLAGS + SIOCGIFADDR, SIOCGIFNETMASK, SIOCGIFMETRIC, SIOCGIFMTU + SIOCGIFHWADDR, SIOCGIFMAP, SIOCGIFTXQLEN
Again, use bullet list instead.
Documentation/admin-guide/workload-tracing.rst line 261
Quoted 4 lines
Documentation/admin-guide/workload-tracing.rst new file +4
-0+ +The following command runs the stressor: :: + + stress-ng --netdev 1 -t 60 --metrics command.
Simply end the sentence with double colon to make the code block above.
Documentation/admin-guide/workload-tracing.rst line 269
Quoted 8 lines
Documentation/admin-guide/workload-tracing.rst new file +8
-0+ +We can use the perf record command to record the events and information +associated with a process. This command records the profiling data in the +perf.data file in the same directory. + +Using the following commands you can record the events associated with the +netdev stressor, view the generated report perf.data and annotate the to +see the statistics of each instruction of the program. ::
"... view the generated report and statistics::"
Quoted 14 lines
Documentation/admin-guide/workload-tracing.rst new file +14
-0+ + perf record stress-ng --netdev 1 -t 60 --metrics command. + perf report + perf annotate + +What is paxtest and how do we use it? +===================================== + +paxtest is a program that tests buffer overflows in the kernel. It tests +kernel enforcements over memory usage. Generally, execution in some memory +segments makes buffer overflows possible. It runs a set of programs that +attempt to subvert memory usage. It is used as a regression test suite for +PaX, and will be useful to test other memory protection patches for the +kernel.
Documentation/admin-guide/workload-tracing.rst line 293
Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +10
-0+ +paxtest provides kiddie and blackhat modes. paxtest kiddie mode runs in +normal mode, whereas blackhat mode tries to get around the protection of +the kernel testing for vulnerabilities. We focus on the kiddie mode here +and combine "paxtest kiddie" run with "perf record" to collect CPU stack +traces for the paxtest kiddie run to see which function is calling other +functions in the performance profile. Then the "dwarf" (DWARF's CFI - Call +Frame Information) mode can be used to unwind the stack. The resulting +report can be viewed in call-graph format as follows: :: +
"... The resulting reported can be viewed in dwarf call-graph format by::"
Documentation/admin-guide/workload-tracing.rst line 300
Quoted 7 lines
Documentation/admin-guide/workload-tracing.rst new file +7
-0+ perf record --call-graph dwarf paxtest kiddie + perf report --stdio + +Tracing workloads +================= + +Now that we understand the workloads, let's start tracing them.
What are the workloads? I don't see any definition here.
Documentation/admin-guide/workload-tracing.rst line 309
Quoted 9 lines
Documentation/admin-guide/workload-tracing.rst new file +9
-0+ +Tracing perf bench all workload +------------------------------- + +Run the following command to trace perf bench all workload: :: + + strace -c perf bench all + +**System Calls made by the workload:**
What about third level heading (with tilde underline)?
Documentation/admin-guide/workload-tracing.rst line 312
Quoted 3 lines
Documentation/admin-guide/workload-tracing.rst new file +3
-0+ +The following table shows you the system calls, number of times the system +call was invoked, and the Linux subsystem they fall under.
"Below table is the list of syscalls, with number of times each is invoked, and the corresponding subsystem:"
Documentation/admin-guide/workload-tracing.rst line 312
Quoted 4 lines
Documentation/admin-guide/workload-tracing.rst new file +4
-0+**System Calls made by the workload:** + +The following table shows you the system calls, number of times the system +call was invoked, and the Linux subsystem they fall under.
Same here.
Documentation/admin-guide/workload-tracing.rst line 312
Quoted 4 lines
Documentation/admin-guide/workload-tracing.rst new file +4
-0+**System Calls made by the workload:** + +The following table shows you the system calls, number of times the system +call was invoked, and the Linux subsystem they fall under.
Again, same here.
But I guess the syscall tables are from the example workload (YMMV)?
Thanks.
Signature · 1 line
An old man doll... just what I always wanted! - Clara
Shuah Khan
reviewerQuoted 13 lines · Bagas Sanjaya
Documentation/admin-guide/workload-tracing.rst line 52
Quoted 2 lines · Shuah Khan
Documentation/admin-guide/workload-tracing.rst new file +2
-0+strace tool can be used to trace system calls made by a process and signals +it receives. a process to the perf, stress-ng, paxtest workloads. System"... to trace system calls made by a process (which can be perf, stress-ng, paxtest workloads, or neither)."
Quoted 1 line
+Before we can get started we will have to get our system ready. We assume
"Before we can get started the system must be prepared first."
Quoted 1 line
+Browsing kernel sources ::
Quoted 9 lines
"To browse kernel sources, you will need:"
Documentation/admin-guide/workload-tracing.rst line 95
Quoted 4 lines
Documentation/admin-guide/workload-tracing.rst new file +4
-0+Workload overview +================= + +We used strace to trace perf bench, stress-ng and paxtest workloads to"In this section, we use strace to trace ..."
Thank you for the review. I will address them in v3.
thanks,
-- Shuah
Randy Dunlap
reviewerHi--
Quoted 31 lines · Shuah Khan
Add a new section to the admin-guide with information of interest to application developers and system integrators doing analysis of the Linux kernel for safety critical applications.
This section will contain documents supporting analysis of kernel interactions with applications, and key kernel subsystems expectations.
Add a new workload-tracing document to this new section.
Signed-off-by Shefali Sharma <[email protected]>Signed-off-by Shuah Khan <[email protected]>---
Changes since v1:
- Addressed review comments on v1 on long lines, rst syntax, licenseDocumentation/admin-guide/index.rst | 11 + .../admin-guide/workload-tracing.rst | 591 ++++++++++++++++++ 2 files changed, 602 insertions(+) create mode 100644 Documentation/admin-guide/workload-tracing.rstDocumentation/admin-guide/workload-tracing.rst new file +2
-0@@ -0,0 +1,591 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0) +
Documentation/admin-guide/workload-tracing.rst line 201
Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +10
-0+What is perf and how do we use it? +==================================== + +Perf is an analysis tool based on Linux 2.6+ systems, which abstracts the +CPU hardware difference in performance measurement in Linux, and provides +a simple command line interface. Perf is based on the perf_events interface +exported by the kernel. It is very useful for profiling the system and +finding performance bottlenecks in an application. + +If you haven't already checkout the Linux mainline repository, you can do
checked out
Documentation/admin-guide/workload-tracing.rst line 211
Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +10
-0+so and then build kernel and perf tool: :: + + git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux + cd linux + make -j3 all + cd tools/perf + make + +Note: The perf command can be built without building the kernel in the +repo and can be run on older kernels. However matching the kernel and
preferably "repository"
Documentation/admin-guide/workload-tracing.rst line 216
Quoted 5 lines
Documentation/admin-guide/workload-tracing.rst new file +5
-0+perf revisions gives more accurate information on the subsystem usage. + +We used perf stat and perf bench options. For a detailed information on the +perf tool, run perf -h. +
Fix table line below:
Quoted 65 lines
Documentation/admin-guide/workload-tracing.rst new file +65
-0+Tracing paxtest kiddie workload +------------------------------- + +Run the following command to trace paxtest kiddie workload: :: + + strace -c paxtest kiddie + +**System Calls made by the workload:** + +The following table shows you the system calls, number of times the system +call was invoked, and the Linux subsystem they fall under. + ++-------------------+-----------+-----------------+----------------------+ +| System Call | # calls | Linux Subsystem | System Call (API) | ++===================+===========+=================+======================+ +| read | 3 | Filesystem | sys_read() | ++-------------------+-----------+-----------------+----------------------+ +| write | 11 | Filesystem | sys_write() | ++-------------------+-----------+-----------------+----------------------+ +| close | 41 | Filesystem | sys_close() | ++-------------------+-----------+-----------------+----------------------+ +| stat | 24 | Filesystem | sys_stat() | ++-------------------+-----------+-----------------+----------------------+ +| fstat | 2 | Filesystem | sys_fstat() | ++-------------------+-----------+-----------------+----------------------+ +| pread64 | 6 | Filesystem | sys_pread64() | ++-------------------+-----------+-----------------+----------------------+ +| access | 1 | Filesystem | sys_access() | ++-------------------+-----------+-----------------+----------------------+ +| pipe | 1 | Filesystem | sys_pipe() | ++-------------------+-----------+-----------------+----------------------+ +| dup2 | 24 | Filesystem | sys_dup2() | ++-------------------+-----------+-----------------+----------------------+ +| execve | 1 | Filesystem | sys_execve() | ++-------------------+-----------+-----------------+----------------------+ +| fcntl | 26 | Filesystem | sys_fcntl() | ++-------------------+-----------+-----------------+----------------------+ +| openat | 14 | Filesystem | sys_openat() | ++-------------------+-----------+-----------------+----------------------+ +| rt_sigaction | 7 | Signal | sys_rt_sigaction() | ++-------------------+-----------+-----------------+----------------------+ +| rt_sigreturn | 38 | Signal | sys_rt_sigreturn() | ++-------------------+-----------+-----------------+----------------------+ +| clone | 38 | Process Mgmt. | sys_clone() | ++-------------------+-----------+-----------------+----------------------+ +| wait4 | 44 | Time | sys_wait4() | ++-------------------+-----------+-----------------+----------------------+ +| mmap | 7 | Memory Mgmt. | sys_mmap() | ++-------------------+-----------+-----------------+----------------------+ +| mprotect | 3 | Memory Mgmt. | sys_mprotect() | ++-------------------+-----------+-----------------+----------------------+ +| munmap | 1 | Memory Mgmt. | sys_munmap() | ++-------------------+-----------+-----------------+----------------------+ +| brk | 3 | Memory Mgmt. | sys_brk() | ++-------------------+-----------+-----------------+----------------------+ +| getpid | 1 | Process Mgmt. | sys_getpid() | ++-------------------+-----------+-----------------+----------------------+ +| getuid | 1 | Process Mgmt. | sys_getuid() | ++-------------------+-----------+-----------------+----------------------+ +| getgid | 1 | Process Mgmt. | sys_getgid() | ++-------------------+-----------+-----------------+----------------------+ +| geteuid | 2 | Process Mgmt. | sys_geteuid() | ++-------------------+-----------+-----------------+----------------------+ +| getegid | 1 | Process Mgmt. | sys_getegid() | ++-------------------+-----------+-----------------+----------------------+
Documentation/admin-guide/workload-tracing.rst line 580
Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +10
-0+| getppid | 1 | Process Mgmt. | sys_getppid() | ++-------------------+-----------+-----------------+----------------------+ +| arch_prctl | 2 | Process Mgmt. | sys_arch_prctl() | ++-------------------+-----------+-----------------+----------------------+ + +Conclusion +========== + +This document is intended to be used as a guide on how to gather fine +grained information on the resources in use by workloads using strace.
fine-grained
Thanks.
Signature · 1 line
~Randy
Shuah Khan
reviewerQuoted 24 lines · Randy Dunlap
Hi--
Quoted 20 lines · Shuah Khan
Add a new section to the admin-guide with information of interest to application developers and system integrators doing analysis of the Linux kernel for safety critical applications.
This section will contain documents supporting analysis of kernel interactions with applications, and key kernel subsystems expectations.
Add a new workload-tracing document to this new section.
Signed-off-by Shefali Sharma <[email protected]>Signed-off-by Shuah Khan <[email protected]>---
Changes since v1:
- Addressed review comments on v1 on long lines, rst syntax, licenseDocumentation/admin-guide/index.rst | 11 + .../admin-guide/workload-tracing.rst | 591 ++++++++++++++++++ 2 files changed, 602 insertions(+)create mode 100644
Documentation/admin-guide/workload-tracing.rst
Thank you for the review. I will send v3 to address them.
thanks,
-- Shuah
Lo! One small quick question.
Quoted 3 lines · Shuah Khan
Add a new section to the admin-guide with information of interest to application developers and system integrators doing analysis of the Linux kernel for safety critical applications.
That intro made my mind go "neither 'application developers' or 'system integrators' are admins, so why is this added to the 'admin guide' then?"
That made me look up the into for the admin guide, which is
``` The Linux kernel user’s and administrator’s guide The following is a collection of user-oriented documents that have been added to the kernel over time. There is, as yet, little overall order or organization here — this material was not written to be a single, coherent document! With luck things will improve quickly over time. ```
Adding another section that doesn't seem to a be a good fit feels a bit like "making things worse than better". So wouldn't this maybe be a good opportunity to create a totally new top-level section for application developers and system integrators, even if it's tiny for now? The audience is likely big enough to justify that -- and the topics are likely different enough, too (of course sometimes there will be overlap, as always). Any maybe it will help to bring other stuff over from admin guide that is more targeted for application developers and system integrators, which will help to get more structure into the admin-guide.
Anyway, I have no strong feeling about this, just wanted to get this of my chest.
Ciao, Thorsten
Shuah Khan
reviewerQuoted 10 lines · Thorsten Leemhuis
Lo! One small quick question.
Quoted 3 lines · Shuah Khan
Add a new section to the admin-guide with information of interest to application developers and system integrators doing analysis of the Linux kernel for safety critical applications.
That intro made my mind go "neither 'application developers' or 'system integrators' are admins, so why is this added to the 'admin guide' then?"
I see what you are saying.
Quoted 10 lines
That made me look up the into for the admin guide, which is
``` The Linux kernel user’s and administrator’s guide The following is a collection of user-oriented documents that have been added to the kernel over time. There is, as yet, little overall order or organization here — this material was not written to be a single, coherent document! With luck things will improve quickly over time. ```
Quoted 11 lines
Adding another section that doesn't seem to a be a good fit feels a bit like "making things worse than better". So wouldn't this maybe be a good opportunity to create a totally new top-level section for application developers and system integrators, even if it's tiny for now? The audience is likely big enough to justify that -- and the topics are likely different enough, too (of course sometimes there will be overlap, as always). Any maybe it will help to bring other stuff over from admin guide that is more targeted for application developers and system integrators, which will help to get more structure into the admin-guide.
I like the idea of creating a new top-level section for application developers and system integrators. I foresee this section growing and also we can look at other documents that are a good fit under this category and move them over.
Jon, Thoughts on this.
thanks,
-- Shuah
Jonathan Corbet
maintainerQuoted 13 lines · Shuah Khan
Quoted 11 lines · Thorsten Leemhuis
Adding another section that doesn't seem to a be a good fit feels a bit like "making things worse than better". So wouldn't this maybe be a good opportunity to create a totally new top-level section for application developers and system integrators, even if it's tiny for now? The audience is likely big enough to justify that -- and the topics are likely different enough, too (of course sometimes there will be overlap, as always). Any maybe it will help to bring other stuff over from admin guide that is more targeted for application developers and system integrators, which will help to get more structure into the admin-guide.
Quoted 5 lines
I like the idea of creating a new top-level section for application developers and system integrators. I foresee this section growing and also we can look at other documents that are a good fit under this category and move them over.
Jon, Thoughts on this.
The fit with the admin guide caught my attention as well, but I didn't immediately have a better place to suggest. I am somewhat resistant to creating another top-level directory; I really want to have *fewer* of them. I certainly don't want to create one for a single document. My inclination would be to leave it where it is for now; we can always revisit this if it turns out we have a lot of material that justifies a new book.
Thanks,
jon
Shuah Khan
reviewerQuoted 21 lines · Jonathan Corbet
Quoted 13 lines · Shuah Khan
Quoted 11 lines · Thorsten Leemhuis
Adding another section that doesn't seem to a be a good fit feels a bit like "making things worse than better". So wouldn't this maybe be a good opportunity to create a totally new top-level section for application developers and system integrators, even if it's tiny for now? The audience is likely big enough to justify that -- and the topics are likely different enough, too (of course sometimes there will be overlap, as always). Any maybe it will help to bring other stuff over from admin guide that is more targeted for application developers and system integrators, which will help to get more structure into the admin-guide.
Quoted 5 lines
I like the idea of creating a new top-level section for application developers and system integrators. I foresee this section growing and also we can look at other documents that are a good fit under this category and move them over.
Jon, Thoughts on this.
Quoted 8 lines
The fit with the admin guide caught my attention as well, but I didn't immediately have a better place to suggest. I am somewhat resistant to creating another top-level directory; I really want to have *fewer* of them. I certainly don't want to create one for a single document. My inclination would be to leave it where it is for now; we can always revisit this if it turns out we have a lot of material that justifies a new book.
Makes perfect sense. I didn't want to suggest a new top level for the same reasons. For now I will send v3 keeping it in the same location.
I will take a pass at the existing documents under admin-guide and in other places to see if others could be grouped under a top-level as suggested by Thorsten
thanks,
-- Shuah