Skip to contentSkip to conversation

docs: add workload-tracing document to admin-guide

PATCH Superseded Shuah Khan 5 participants 9 messages Jan 27, 2023 – Jan 31, 2023 Documentation

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Shuah Khan

author reviewer
#0 lore raw

PATCH v2 1/1 docs: add workload-tracing document to admin-guide

Add a new section to the admin-guide with information of interest to application developers and system integrators doing analysis of the Linux kernel for safety critical applications.

This section will contain documents supporting analysis of kernel interactions with applications, and key kernel subsystems expectations.

Add a new workload-tracing document to this new section.

---
Changes since v1:
- Addressed review comments on v1 on long lines, rst syntax, license

 Documentation/admin-guide/index.rst           |  11 +
 .../admin-guide/workload-tracing.rst          | 591 ++++++++++++++++++
 2 files changed, 602 insertions(+)
 create mode 100644 Documentation/admin-guide/workload-tracing.rst
Documentation/admin-guide/index.rst +11 -0
index 5bfafcbb9562..f475554382e2 100644
@@ -56,6 +56,17 @@ ABI will be found here.
 
    sysfs-rules
 
+This is the beginning of a section with information of interest to
+application developers and system integrators doing analysis of the
+Linux kernel for safety critical applications. Documents supporting
+analysis of kernel interactions with applications, and key kernel
+subsystems expectations will be found here.
+
+.. toctree::
+   :maxdepth: 1
+
+   workload-tracing
+
 The rest of this manual consists of various unordered guides on how to
 configure specific aspects of kernel behavior to your liking.
 
Documentation/admin-guide/workload-tracing.rst new file +591 -0

591 lines not on this page · raw message

Signature · 1 line
2.34.1
#1 lore raw

Documentation/admin-guide/workload-tracing.rst line 52

Quoted 2 lines · Shuah Khan
Documentation/admin-guide/workload-tracing.rst new file +2 -0
+strace tool can be used to trace system calls made by a process and signals
+it receives. a process to the perf, stress-ng, paxtest workloads. System

"... to trace system calls made by a process (which can be perf, stress-ng, paxtest workloads, or neither)."

Quoted 1 line

+Before we can get started we will have to get our system ready. We assume

"Before we can get started the system must be prepared first."

Quoted 1 line

+Browsing kernel sources ::

"To browse kernel sources, you will need:"

Documentation/admin-guide/workload-tracing.rst line 95

Quoted 4 lines
Documentation/admin-guide/workload-tracing.rst new file +4 -0
+Workload overview
+=================
+
+We used strace to trace perf bench, stress-ng and paxtest workloads to

"In this section, we use strace to trace ..."

Documentation/admin-guide/workload-tracing.rst line 98

Quoted 3 lines
Documentation/admin-guide/workload-tracing.rst new file +3 -0
+show how to analyze a workload and identify Linux subsystems used by
+these workloads. We hope this process can be applied to trace workload(s).
+We will go over the workloads first.

"This process can also be applied to trace other workloads."

Quoted 5 lines
Documentation/admin-guide/workload-tracing.rst new file +4 -0
+
+perf bench (all) workload:
+--------------------------
+
<snipped>

Documentation/admin-guide/workload-tracing.rst line 125

Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +8 -0
+
+Stress-ng netdev stressor workload:
+-----------------------------------
+
<snipped>
+
+paxtest kiddie workload:
+------------------------
+
<snipped>

Drop trailing colon from these subsection texts.

Quoted 1 line

+We used the “-c” option to gather fine grained information.

```
The basic usage is strace <command>. To run verbose tracing, specify -v
to strace. To generate the report, specify -c option.
```

Documentation/admin-guide/workload-tracing.rst line 178

Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +10 -0
+Now let’s look at `cscope <https://cscope.sourceforge.net/>`_, a command
+line tool for browsing C, C++ or Java code-bases. We can use it to find
+all the references to a symbol, global definitions, functions called by a
+function, functions calling a function, text strings, regular expression
+patterns, files including a file.
+
+We can use cscope to find which system call belongs to which subsystem.
+This way we can find the kernel subsystems used by a process when it is
+executed. To use it navigate to the source code directory. Here we are
+analyzing the kernel source tree.

I guess "you" is better fit for the wording above?

Documentation/admin-guide/workload-tracing.rst line 215

Quoted 2 lines
Documentation/admin-guide/workload-tracing.rst new file +2 -0
+We used perf stat and perf bench options. For a detailed information on the
+perf tool, run perf -h.

"In this section, we highlight stat and bench options. For help on other options, see perf -h."

Quoted 1 line

+**perf bench all** command runs the following benchmarks: ::

Either don't markup that command or inline it (``perf bench all``). Also, use bullet list for benchmarking list below.

Quoted 5 lines
Documentation/admin-guide/workload-tracing.rst new file +5 -0
+
+  sched/messaging
+  sched/pipe
+  syscall/basic
+  mem/memcpy

Documentation/admin-guide/workload-tracing.rst line 257

Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +9 -0
+  mem/memset
+
 
+The netdev stressor starts N workers that exercise various netdevice ioctl
+commands across all the available network devices. The following ioctls are
+exercised: ::
+
+  SIOCGIFCONF, SIOCGIFINDEX, SIOCGIFNAME, SIOCGIFFLAGS
+  SIOCGIFADDR, SIOCGIFNETMASK, SIOCGIFMETRIC, SIOCGIFMTU
+  SIOCGIFHWADDR, SIOCGIFMAP, SIOCGIFTXQLEN

Again, use bullet list instead.

Documentation/admin-guide/workload-tracing.rst line 261

Quoted 4 lines
Documentation/admin-guide/workload-tracing.rst new file +4 -0
+
+The following command runs the stressor: ::
+
+  stress-ng --netdev 1 -t 60 --metrics command.

Simply end the sentence with double colon to make the code block above.

Documentation/admin-guide/workload-tracing.rst line 269

Quoted 8 lines
Documentation/admin-guide/workload-tracing.rst new file +8 -0
+
+We can use the perf record command to record the events and information
+associated with a process. This command records the profiling data in the
+perf.data file in the same directory.
+
+Using the following commands you can record the events associated with the
+netdev stressor, view the generated report perf.data and annotate the to
+see the statistics of each instruction of the program. ::

"... view the generated report and statistics::"

Quoted 14 lines
Documentation/admin-guide/workload-tracing.rst new file +14 -0
+
+  perf record stress-ng --netdev 1 -t 60 --metrics command.
+  perf report
+  perf annotate
+
+What is paxtest and how do we use it?
+=====================================
+
+paxtest is a program that tests buffer overflows in the kernel. It tests
+kernel enforcements over memory usage. Generally, execution in some memory
+segments makes buffer overflows possible. It runs a set of programs that
+attempt to subvert memory usage. It is used as a regression test suite for
+PaX, and will be useful to test other memory protection patches for the
+kernel.

Documentation/admin-guide/workload-tracing.rst line 293

Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +10 -0
+
+paxtest provides kiddie and blackhat modes. paxtest kiddie mode runs in
+normal mode, whereas blackhat mode tries to get around the protection of
+the kernel testing for vulnerabilities. We focus on the kiddie mode here
+and combine "paxtest kiddie" run with "perf record" to collect CPU stack
+traces for the paxtest kiddie run to see which function is calling other
+functions in the performance profile. Then the "dwarf" (DWARF's CFI - Call
+Frame Information) mode can be used to unwind the stack. The resulting
+report can be viewed in call-graph format as follows: ::
+

"... The resulting reported can be viewed in dwarf call-graph format by::"

Documentation/admin-guide/workload-tracing.rst line 300

Quoted 7 lines
Documentation/admin-guide/workload-tracing.rst new file +7 -0
+  perf record --call-graph dwarf paxtest kiddie
+  perf report --stdio
+
+Tracing workloads
+=================
+
+Now that we understand the workloads, let's start tracing them.

What are the workloads? I don't see any definition here.

Documentation/admin-guide/workload-tracing.rst line 309

Quoted 9 lines
Documentation/admin-guide/workload-tracing.rst new file +9 -0
+
+Tracing perf bench all workload
+-------------------------------
+
+Run the following command to trace perf bench all workload: ::
+
+ strace -c perf bench all
+
+**System Calls made by the workload:**

What about third level heading (with tilde underline)?

Documentation/admin-guide/workload-tracing.rst line 312

Quoted 3 lines
Documentation/admin-guide/workload-tracing.rst new file +3 -0
+
+The following table shows you the system calls, number of times the system
+call was invoked, and the Linux subsystem they fall under.

"Below table is the list of syscalls, with number of times each is invoked, and the corresponding subsystem:"

Documentation/admin-guide/workload-tracing.rst line 312

Quoted 4 lines
Documentation/admin-guide/workload-tracing.rst new file +4 -0
+**System Calls made by the workload:**
+
+The following table shows you the system calls, number of times the system
+call was invoked, and the Linux subsystem they fall under.

Same here.

Documentation/admin-guide/workload-tracing.rst line 312

Quoted 4 lines
Documentation/admin-guide/workload-tracing.rst new file +4 -0
+**System Calls made by the workload:**
+
+The following table shows you the system calls, number of times the system
+call was invoked, and the Linux subsystem they fall under.

Again, same here.

But I guess the syscall tables are from the example workload (YMMV)?

Thanks.

Signature · 1 line
An old man doll... just what I always wanted! - Clara
author reviewer
#2 lore raw
Quoted 13 lines · Bagas Sanjaya

Documentation/admin-guide/workload-tracing.rst line 52

Quoted 2 lines · Shuah Khan
Documentation/admin-guide/workload-tracing.rst new file +2 -0
+strace tool can be used to trace system calls made by a process and signals
+it receives. a process to the perf, stress-ng, paxtest workloads. System

"... to trace system calls made by a process (which can be perf, stress-ng, paxtest workloads, or neither)."

Quoted 1 line

+Before we can get started we will have to get our system ready. We assume

"Before we can get started the system must be prepared first."

Quoted 1 line

+Browsing kernel sources ::

Quoted 9 lines

"To browse kernel sources, you will need:"

Documentation/admin-guide/workload-tracing.rst line 95

Quoted 4 lines
Documentation/admin-guide/workload-tracing.rst new file +4 -0
+Workload overview
+=================
+
+We used strace to trace perf bench, stress-ng and paxtest workloads to

"In this section, we use strace to trace ..."

Thank you for the review. I will address them in v3.

thanks,
-- Shuah

reviewer
#3 lore raw
In reply to Shuah Khan

Hi--

Quoted 31 lines · Shuah Khan

Add a new section to the admin-guide with information of interest to application developers and system integrators doing analysis of the Linux kernel for safety critical applications.

This section will contain documents supporting analysis of kernel interactions with applications, and key kernel subsystems expectations.

Add a new workload-tracing document to this new section.

---
Changes since v1:
- Addressed review comments on v1 on long lines, rst syntax, license

 Documentation/admin-guide/index.rst           |  11 +
 .../admin-guide/workload-tracing.rst          | 591 ++++++++++++++++++
 2 files changed, 602 insertions(+)
 create mode 100644 Documentation/admin-guide/workload-tracing.rst
Documentation/admin-guide/workload-tracing.rst new file +2 -0
new file mode 100644
index 000000000000..43753f3ea915
@@ -0,0 +1,591 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
+
 
 

Documentation/admin-guide/workload-tracing.rst line 201

Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +10 -0
+What is perf and how do we use it?
+====================================
+
+Perf is an analysis tool based on Linux 2.6+ systems, which abstracts the
+CPU hardware difference in performance measurement in Linux, and provides
+a simple command line interface. Perf is based on the perf_events interface
+exported by the kernel. It is very useful for profiling the system and
+finding performance bottlenecks in an application.
+
+If you haven't already checkout the Linux mainline repository, you can do

checked out

Documentation/admin-guide/workload-tracing.rst line 211

Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +10 -0
+so and then build kernel and perf tool: ::
+
+  git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git linux
+  cd linux
+  make -j3 all
+  cd tools/perf
+  make
+
+Note: The perf command can be built without building the kernel in the
+repo and can be run on older kernels. However matching the kernel and

preferably "repository"

Documentation/admin-guide/workload-tracing.rst line 216

Quoted 5 lines
Documentation/admin-guide/workload-tracing.rst new file +5 -0
+perf revisions gives more accurate information on the subsystem usage.
+
+We used perf stat and perf bench options. For a detailed information on the
+perf tool, run perf -h.
+

Fix table line below:

Quoted 65 lines
Documentation/admin-guide/workload-tracing.rst new file +65 -0
+Tracing paxtest kiddie workload
+-------------------------------
+
+Run the following command to trace paxtest kiddie workload: ::
+
+ strace -c paxtest kiddie
+
+**System Calls made by the workload:**
+
+The following table shows you the system calls, number of times the system
+call was invoked, and the Linux subsystem they fall under.
+
++-------------------+-----------+-----------------+----------------------+
+| System Call       | # calls   | Linux Subsystem | System Call (API)    |
++===================+===========+=================+======================+
+| read              | 3         | Filesystem      | sys_read()           |
++-------------------+-----------+-----------------+----------------------+
+| write             | 11        | Filesystem      | sys_write()          |
++-------------------+-----------+-----------------+----------------------+
+| close             | 41        | Filesystem      | sys_close()          |
++-------------------+-----------+-----------------+----------------------+
+| stat              | 24        | Filesystem      | sys_stat()           |
++-------------------+-----------+-----------------+----------------------+
+| fstat             | 2         | Filesystem      | sys_fstat()          |
++-------------------+-----------+-----------------+----------------------+
+| pread64           | 6         | Filesystem      | sys_pread64()        |
++-------------------+-----------+-----------------+----------------------+
+| access	    | 1         | Filesystem      | sys_access()         |
++-------------------+-----------+-----------------+----------------------+
+| pipe              | 1         | Filesystem      | sys_pipe()           |
++-------------------+-----------+-----------------+----------------------+
+| dup2              | 24        | Filesystem      | sys_dup2()           |
++-------------------+-----------+-----------------+----------------------+
+| execve            | 1         | Filesystem      | sys_execve()         |
++-------------------+-----------+-----------------+----------------------+
+| fcntl             | 26        | Filesystem      | sys_fcntl()          |
++-------------------+-----------+-----------------+----------------------+
+| openat            | 14        | Filesystem      | sys_openat()         |
++-------------------+-----------+-----------------+----------------------+
+| rt_sigaction      | 7         | Signal          | sys_rt_sigaction()   |
++-------------------+-----------+-----------------+----------------------+
+| rt_sigreturn      | 38        | Signal          | sys_rt_sigreturn()   |
++-------------------+-----------+-----------------+----------------------+
+| clone             | 38        | Process Mgmt.   | sys_clone()          |
++-------------------+-----------+-----------------+----------------------+
+| wait4             | 44        | Time            | sys_wait4()          |
++-------------------+-----------+-----------------+----------------------+
+| mmap              | 7         | Memory Mgmt.    | sys_mmap()           |
++-------------------+-----------+-----------------+----------------------+
+| mprotect          | 3         | Memory Mgmt.    | sys_mprotect()       |
++-------------------+-----------+-----------------+----------------------+
+| munmap            | 1         | Memory Mgmt.    | sys_munmap()         |
++-------------------+-----------+-----------------+----------------------+
+| brk               | 3         | Memory Mgmt.    | sys_brk()            |
++-------------------+-----------+-----------------+----------------------+
+| getpid            | 1         | Process Mgmt.   | sys_getpid()         |
++-------------------+-----------+-----------------+----------------------+
+| getuid            | 1         | Process Mgmt.   | sys_getuid()         |
++-------------------+-----------+-----------------+----------------------+
+| getgid            | 1         | Process Mgmt.   | sys_getgid()         |
++-------------------+-----------+-----------------+----------------------+
+| geteuid           | 2         | Process Mgmt.   | sys_geteuid()        |
++-------------------+-----------+-----------------+----------------------+
+| getegid           | 1         | Process Mgmt.   | sys_getegid()        |
++-------------------+-----------+-----------------+----------------------+

Documentation/admin-guide/workload-tracing.rst line 580

Quoted 10 lines
Documentation/admin-guide/workload-tracing.rst new file +10 -0
+| getppid           | 1         | Process Mgmt.   | sys_getppid()        |
++-------------------+-----------+-----------------+----------------------+
+| arch_prctl        | 2         | Process Mgmt.   | sys_arch_prctl()     |
++-------------------+-----------+-----------------+----------------------+
+
+Conclusion
+==========
+
+This document is intended to be used as a guide on how to gather fine
+grained information on the resources in use by workloads using strace.

fine-grained

Thanks.

Signature · 1 line
~Randy
author reviewer
#4 lore raw
Quoted 24 lines · Randy Dunlap

Hi--

Quoted 20 lines · Shuah Khan

Add a new section to the admin-guide with information of interest to application developers and system integrators doing analysis of the Linux kernel for safety critical applications.

This section will contain documents supporting analysis of kernel interactions with applications, and key kernel subsystems expectations.

Add a new workload-tracing document to this new section.

---
Changes since v1:
- Addressed review comments on v1 on long lines, rst syntax, license

  Documentation/admin-guide/index.rst           |  11 +
  .../admin-guide/workload-tracing.rst          | 591 ++++++++++++++++++
  2 files changed, 602 insertions(+)

create mode 100644 Documentation/admin-guide/workload-tracing.rst

Thank you for the review. I will send v3 to address them.

thanks,
-- Shuah

#5 lore raw
In reply to Shuah Khan

Lo! One small quick question.

Quoted 3 lines · Shuah Khan

Add a new section to the admin-guide with information of interest to application developers and system integrators doing analysis of the Linux kernel for safety critical applications.

That intro made my mind go "neither 'application developers' or 'system integrators' are admins, so why is this added to the 'admin guide' then?"

That made me look up the into for the admin guide, which is

```
The Linux kernel user’s and administrator’s guide

The following is a collection of user-oriented documents that have been
added to the kernel over time. There is, as yet, little overall order or
organization here — this material was not written to be a single,
coherent document! With luck things will improve quickly over time.
```

Adding another section that doesn't seem to a be a good fit feels a bit like "making things worse than better". So wouldn't this maybe be a good opportunity to create a totally new top-level section for application developers and system integrators, even if it's tiny for now? The audience is likely big enough to justify that -- and the topics are likely different enough, too (of course sometimes there will be overlap, as always). Any maybe it will help to bring other stuff over from admin guide that is more targeted for application developers and system integrators, which will help to get more structure into the admin-guide.

Anyway, I have no strong feeling about this, just wanted to get this of my chest.

Ciao, Thorsten

author reviewer
#6 lore raw
Quoted 10 lines · Thorsten Leemhuis

Lo! One small quick question.

Quoted 3 lines · Shuah Khan

Add a new section to the admin-guide with information of interest to application developers and system integrators doing analysis of the Linux kernel for safety critical applications.

That intro made my mind go "neither 'application developers' or 'system integrators' are admins, so why is this added to the 'admin guide' then?"

I see what you are saying.

Quoted 10 lines

That made me look up the into for the admin guide, which is

```
The Linux kernel user’s and administrator’s guide

The following is a collection of user-oriented documents that have been
added to the kernel over time. There is, as yet, little overall order or
organization here — this material was not written to be a single,
coherent document! With luck things will improve quickly over time.
```
Quoted 11 lines

Adding another section that doesn't seem to a be a good fit feels a bit like "making things worse than better". So wouldn't this maybe be a good opportunity to create a totally new top-level section for application developers and system integrators, even if it's tiny for now? The audience is likely big enough to justify that -- and the topics are likely different enough, too (of course sometimes there will be overlap, as always). Any maybe it will help to bring other stuff over from admin guide that is more targeted for application developers and system integrators, which will help to get more structure into the admin-guide.

I like the idea of creating a new top-level section for application developers and system integrators. I foresee this section growing and also we can look at other documents that are a good fit under this category and move them over.

Jon, Thoughts on this.

thanks,
-- Shuah

maintainer
#7 lore raw
Quoted 13 lines · Shuah Khan
Quoted 11 lines · Thorsten Leemhuis

Adding another section that doesn't seem to a be a good fit feels a bit like "making things worse than better". So wouldn't this maybe be a good opportunity to create a totally new top-level section for application developers and system integrators, even if it's tiny for now? The audience is likely big enough to justify that -- and the topics are likely different enough, too (of course sometimes there will be overlap, as always). Any maybe it will help to bring other stuff over from admin guide that is more targeted for application developers and system integrators, which will help to get more structure into the admin-guide.

Quoted 5 lines

I like the idea of creating a new top-level section for application developers and system integrators. I foresee this section growing and also we can look at other documents that are a good fit under this category and move them over.

Jon, Thoughts on this.

The fit with the admin guide caught my attention as well, but I didn't immediately have a better place to suggest. I am somewhat resistant to creating another top-level directory; I really want to have *fewer* of them. I certainly don't want to create one for a single document. My inclination would be to leave it where it is for now; we can always revisit this if it turns out we have a lot of material that justifies a new book.

Thanks,

jon

author reviewer
#8 lore raw
Quoted 21 lines · Jonathan Corbet
Quoted 13 lines · Shuah Khan
Quoted 11 lines · Thorsten Leemhuis

Adding another section that doesn't seem to a be a good fit feels a bit like "making things worse than better". So wouldn't this maybe be a good opportunity to create a totally new top-level section for application developers and system integrators, even if it's tiny for now? The audience is likely big enough to justify that -- and the topics are likely different enough, too (of course sometimes there will be overlap, as always). Any maybe it will help to bring other stuff over from admin guide that is more targeted for application developers and system integrators, which will help to get more structure into the admin-guide.

Quoted 5 lines

I like the idea of creating a new top-level section for application developers and system integrators. I foresee this section growing and also we can look at other documents that are a good fit under this category and move them over.

Jon, Thoughts on this.

Quoted 8 lines

The fit with the admin guide caught my attention as well, but I didn't immediately have a better place to suggest. I am somewhat resistant to creating another top-level directory; I really want to have *fewer* of them. I certainly don't want to create one for a single document. My inclination would be to leave it where it is for now; we can always revisit this if it turns out we have a lot of material that justifies a new book.

Makes perfect sense. I didn't want to suggest a new top level for the same reasons. For now I will send v3 keeping it in the same location.

I will take a pass at the existing documents under admin-guide and in other places to see if others could be grouped under a top-level as suggested by Thorsten

thanks,
-- Shuah