C and Rust support for perf script
Stefan Ene
cover C and Rust support for perf script
From: Stefan Ene <[email protected]>
============================COVER=LETTER============================
This proposal is addressing the usability and performance of the available scripting languages for custom perf event data processing inside of the perf toolset, specifically with the perf script command.
With the perf-script custom event processing functionality for C and Rust, we noticed between 2x to 5x speed improvement with our new methods compared to the existent Python and Perl scripting methods.
To explain the proposed method, you begin with the C or Rust script we’ve templatized, then just add changes for custom event processing in the preferred language using this provided template, compile the respective script into a dynamic library, then give the resulting .so file as a parameter to the newly implemented perf script option.
List of functionality additions:
1/ Added new "--new_script" option inside the perf script command that takes in as parameter .so files. Code addition in tools/perf/builtin-script.c
2/ The functional code for the newly implemented option was added to the tools/perf/util/new_script.c and tools/perf/util/new_script.h files.
3/ Folder added at tools/perf/new_script_templates, containing C and Rust script templates for the new option, along with compilation instructions.
4/ Finally, a short bash script for updating the perf tool set within the kernel code was provided in the base-level file update_perf_tools.sh.
Common Questions
* How can I use the new toolset?
The new implementations for the perf script have a detailed usage guide inside of the tools/perf/new_script_templates/README file, along with some script templates for C and Rust!
* Why a new option instead of expanding dlfilter?
The new option gave us the flexibility to make use of the fast dlfilter dynamic library approach, as opposed to implementing another interpreting methodology. This allows for scalability, with great potential to other languages supporting dynamic library calls from the base C code.
* Why use C and Rust instead?
As of kernel version 6.11, the perf tool has a large overhead for data processing using Python and Perl, given the languages having to use their respective perf built-in interpreters. Furthermore, while Python is widley used in the development comunity, as of 20204, Perl is only used by 2.5 of developers worldwide, while C and Rust are more common, with 20.3% and 12.6% usage, respectively (Source: statista.com).
* What are the actual performance improvements?
As last tested, the C and Rust approach are anywhere between 2 to 5 times faster than the existent Python and Perl scripting methods, with Rust being the fastest all across!
Acknowledgements:
This code was completed as part of an Intel summer internship project, under the mentoring of Vinicius Gomes, Intel Linux Kernel Team.
=========================END=COVER=LETTER===========================
Stefan Ene (3):
add the new perf script option (--new_script) and related changes added the C sample script
added the Rust sample script
tools/perf/builtin-script.c | 22 +- tools/perf/new_script_templates/README | 65 ++++ tools/perf/new_script_templates/lib.rs | 108 +++++++ tools/perf/new_script_templates/script.c | 113 +++++++ tools/perf/util/Build | 1 + tools/perf/util/new_script.c | 376 +++++++++++++++++++++++ tools/perf/util/new_script.h | 54 ++++ tools/perf/util/new_script_rs_lib.h | 35 +++ 8 files changed, 773 insertions(+), 1 deletion(-) create mode 100644 tools/perf/new_script_templates/README create mode 100644 tools/perf/new_script_templates/lib.rs create mode 100644 tools/perf/new_script_templates/script.c create mode 100644 tools/perf/util/new_script.c create mode 100644 tools/perf/util/new_script.h create mode 100644 tools/perf/util/new_script_rs_lib.h
Signature · 1 line
2.46.0
Stefan Ene
RFC 1/3 C and Rust support for perf script
From: Stefan Ene <[email protected]>
[PATCH 1/3] add the new perf script option (--new_script) and related
changes
--- tools/perf/builtin-script.c | 22 +- tools/perf/util/Build | 1 + tools/perf/util/new_script.c | 376 +++++++++++++++++++++++++++++++++++ tools/perf/util/new_script.h | 54 +++++ 4 files changed, 452 insertions(+), 1 deletion(-) create mode 100644 tools/perf/util/new_script.c create mode 100644 tools/perf/util/new_script.h
tools/perf/builtin-script.c +21 -1
@@ -63,6 +63,7 @@#include "util/util.h" #include "util/cgroup.h" #include "perf.h" +#include "util/new_script.h" #include <linux/ctype.h> #ifdef HAVE_LIBTRACEEVENT
@@ -88,6 +89,7 @@ static struct perf_stat_config stat_config;static int max_blocks;static bool native_arch;static struct dlfilter *dlfilter; +static struct new_script *new_struct;static int dlargc;static char **dlargv;
@@ -3898,6 +3900,7 @@ int cmd_script(int argc, const char **argv)struct utsname uts;char *script_path =NULL ;const char *dlfilter_file =NULL ; +const char *new_script_file =NULL ;const char **__argv;int i, j, err =0 ;struct perf_script script = {
@@ -3954,6 +3957,7 @@ int cmd_script(int argc, const char **argv)OPT_STRING ('g' ,"gen-script" , &generate_script_lang,"lang" ,"generate perf-script.xx script in specified language" ),OPT_STRING (0 ,"dlfilter" , &dlfilter_file,"file" ,"filter .so file name" ), +OPT_STRING (0 ,"new_script" , &new_script_file,"file" ,"specify .so file name" ),OPT_CALLBACK (0 ,"dlarg" ,NULL ,"argument" ,"filter argument" , add_dlarg), OPT_STRING('i', "input", &input_name, "file", "input file name"),
@@ -4355,6 +4359,21 @@ int cmd_script(int argc, const char **argv) goto out_delete; }#endif + +if (new_script_file) { + new_struct =new_script__new (new_script_file); + +if (!new_struct) { +perror ("cannot create new_script objects\n" ); + goto out_delete; + } +if (new_script__start (new_struct, &data) <0 ) { +perror ("cannot start new_script object\n" ); + goto out_delete; + } + goto out_delete; + } +if (generate_script_lang) {struct stat perf_stat;int input;
@@ -4458,7 +4477,8 @@ int cmd_script(int argc, const char **argv)if (script_started)cleanup_scripting ();dlfilter__cleanup (dlfilter); +new_script__cleanup (new_struct);free_dlarg (); out:return err; -} +}
tools/perf/util/Build +1 -0
@@ -140,6 +140,7 @@ perf-util-y += parse-sublevel-options.o perf-util-y += term.o perf-util-y += help-unknown-cmd.o perf-util-y += dlfilter.o +perf-util-y += new_script.o perf-util-y += mem-events.o perf-util-y += mem-info.o perf-util-y += vsprintf.o
tools/perf/util/new_script.c new file +376 -0
377 lines not on this page · raw message
tools/perf/util/new_script.h new file +54 -0
55 lines not on this page · raw message
Signature · 1 line
2.46.0
Stefan Ene
RFC 2/3 C and Rust support for perf script
In reply to Stefan EneFrom: Stefan Ene <[email protected]>
[PATCH 2/3] added the C sample script
--- tools/perf/new_script_templates/README | 16 ++++ tools/perf/new_script_templates/script.c | 107 +++++++++++++++++++++++ 2 files changed, 123 insertions(+) create mode 100644 tools/perf/new_script_templates/README create mode 100644 tools/perf/new_script_templates/script.c
tools/perf/new_script_templates/README new file +16 -0
@@ -0,0 +1,16 @@ +Linux kernel additions for C and Rust script support inside the perf-script tool set. + +Steps to use new feature: + + First, use the provided update_perf_tools.sh script to make sure your perf toolset is up to date with the latest implementation: + $ bash update_perf_tools.sh + + a) For C scripts: + 1. Use the default C template test.c to write your own custom perf event processing + + 2. Compile the C script into a dynamic library using the following two commands: + $ gcc -c -I ~/include -fpic new_script_templates/script.c + $ gcc -shared -o test.so new_script_templates/script.o + + 3. Call the new perf script option to use the newly created .so file using the command: + $ sudo perf script --new_script new_script_templates/script.so
tools/perf/new_script_templates/script.c new file +107 -0
108 lines not on this page · raw message
Signature · 1 line
2.46.0
Stefan Ene
RFC 3/3 C and Rust support for perf script
In reply to Stefan EneFrom: Stefan Ene <[email protected]>
[PATCH 3/3] added the Rust sample script
--- tools/perf/new_script_templates/README | 61 +++++++++++-- tools/perf/new_script_templates/lib.rs | 108 +++++++++++++++++++++++ tools/perf/new_script_templates/script.c | 6 ++ tools/perf/util/new_script_rs_lib.h | 35 ++++++++ 4 files changed, 204 insertions(+), 6 deletions(-) create mode 100644 tools/perf/new_script_templates/lib.rs create mode 100644 tools/perf/util/new_script_rs_lib.h
tools/perf/new_script_templates/README +55 -6
@@ -2,15 +2,64 @@ Linux kernel additions for C and Rust script support inside the perf-script tool Steps to use new feature: - First, use the provided update_perf_tools.sh script to make sure your perf toolset is up to date with the latest implementation: - $ bash update_perf_tools.sh + First, use run the following lines to make sure your perf toolset is up to date with the latest implementation: + $ cd ./tools/perf/ + $ make + $ sudo cp perf /usr/bin + $ perf --version + Now, you can make use of our new features! + a) For C scripts: - 1. Use the default C template test.c to write your own custom perf event processing + 1. Use the default C template in tools/perf/new_script_templates/script.c to write your own custom perf event processing 2. Compile the C script into a dynamic library using the following two commands: - $ gcc -c -I ~/include -fpic new_script_templates/script.c - $ gcc -shared -o test.so new_script_templates/script.o + $ gcc -c -I ~/include -fpic script.c + $ gcc -shared -o script.so script.o 3. Call the new perf script option to use the newly created .so file using the command: - $ sudo perf script --new_script new_script_templates/script.so + $ sudo perf script --new_script script.so + + + b) For Rust scripts: + 1. Create a new Rust project using Cargo: + $ cargo new rslib_script --lib + $ cd rslib_script + + 2. In the Cargo.toml file, specify the crate type as a dynamic library as follows: + [lib] + crate-type = ["cdylib"] + + 3. Use the default Rust template in tools/perf/new_script_templates/lib.rs to write your own custom perf event processing, and store it into src/lib.rs inside of the Rust project folder. + + 4. Compile the Rust project using Cargo: + $ cargo build --release + + 5. Call the new perf script option to use the newly created /target/release/librslib.so within the Rust project using the followig command: + $ sudo perf script --new_script rslib/target/release/librslib.so + +Enjoy using the new scripting languages for an added bonus of usability and performance over the existent Python and Perl options in the upstream kernel code. + + +<!-- ============================= --> + +This section contains some initial benchmark results on the Intel Lab machine using the perf script tools on a 4026869-byte perf.data file. + +Process ----- Time + +> Perf tool vanilla: +perf script raw: 1.425s +perf script names: 0.915s +perf script Python: 1.334s +perf script Perl: 2.210s + + +> Pre-optimizations: +new_script raw C: 0.284s +new_script raw Rust: 0.607s + +> Post-optimizations: +new_script raw C batch=1MB: 0.306s +new_script raw C batch=2MB: 0.301s +new_script raw C batch=4MB: 0.296s +new_script raw Rust batch=1MB: 0.262s
tools/perf/new_script_templates/lib.rs new file +108 -0
109 lines not on this page · raw message
tools/perf/new_script_templates/script.c +6 -0
@@ -1,3 +1,9 @@ +// Stefan Ene's code, under Intel - Linux Kernel Team +/* +* code for new perf script custom C script template +* +*/ +#include <stdlib.h> #include <stdio.h>
tools/perf/util/new_script_rs_lib.h new file +35 -0
@@ -0,0 +1,35 @@ +// Stefan Ene's code, under Intel - Linux Kernel Team +/* +* new_script_lib_rs.h: header file for new_script Rust FFI +* +*/ + +#ifndef NEW_SCRIPT_LIB_RS_H +#define NEW_SCRIPT_LIB_RS_H + +#ifdef __cplusplus +extern "C" { +#endif + +#include <stdint.h> + +// Match the struct definitions with the Rust code +struct NewScript { +char *file; +void *handle; + +void (*begin)(void *data,void *ctx); +void (*end)(void *data,void *ctx); + +int (*process_file_header)(void *data,void *ctx); +int (*process_event_header)(void *data,void *ctx); +int (*process_event_raw_data)(void *data,const int size,void *ctx); +}; + +void initialize_new_script (struct NewScript * s); + +#ifdef __cplusplus +} +#endif + +#endif // NEW_SCRIPT_LIB_RS_H
Signature · 1 line
2.46.0
Namhyung Kim
maintainerHello,
Quoted 44 lines · StefanEne
From: Stefan Ene <[email protected]>
============================COVER=LETTER============================
This proposal is addressing the usability and performance of the available scripting languages for custom perf event data processing inside of the perf toolset, specifically with the perf script command.
With the perf-script custom event processing functionality for C and Rust, we noticed between 2x to 5x speed improvement with our new methods compared to the existent Python and Perl scripting methods.
To explain the proposed method, you begin with the C or Rust script we’ve templatized, then just add changes for custom event processing in the preferred language using this provided template, compile the respective script into a dynamic library, then give the resulting .so file as a parameter to the newly implemented perf script option.
List of functionality additions:
1/ Added new "--new_script" option inside the perf script command that takes in as parameter .so files. Code addition in
tools/perf/builtin-script.c2/ The functional code for the newly implemented option was added to the
tools/perf/util/new_script.candtools/perf/util/new_script.hfiles.3/ Folder added at
tools/perf/new_script_templates, containing C and Rust script templates for the new option, along with compilation instructions.4/ Finally, a short bash script for updating the perf tool set within the kernel code was provided in the base-level file
update_perf_tools.sh.Common Questions
* How can I use the new toolset?
The new implementations for the perf script have a detailed usage guide inside of the
tools/perf/new_script_templates/READMEfile, along with some script templates for C and Rust!* Why a new option instead of expanding dlfilter?
The new option gave us the flexibility to make use of the fast dlfilter dynamic library approach, as opposed to implementing another interpreting methodology. This allows for scalability, with great potential to other languages supporting dynamic library calls from the base C code.
* Why use C and Rust instead?
As of kernel version 6.11, the perf tool has a large overhead for data processing using Python and Perl, given the languages having to use their respective perf built-in interpreters. Furthermore, while Python is widley used in the development comunity, as of 20204, Perl is only used by 2.5 of developers worldwide, while C and Rust are more common, with 20.3% and 12.6% usage, respectively (Source: statista.com).
* What are the actual performance improvements?
As last tested, the C and Rust approach are anywhere between 2 to 5 times faster than the existent Python and Perl scripting methods, with Rust being the fastest all across!
Acknowledgements:
This code was completed as part of an Intel summer internship project, under the mentoring of Vinicius Gomes, Intel Linux Kernel Team.
Quoted 7 lines
=========================END=COVER=LETTER===========================
Stefan Ene (3):
add the new perf script option (--new_script) and related changes added the C sample script
added the Rust sample script
Thanks for sharing your work. But I think there are many more work to support scripting of native languages properly. For example, you need to resolve symbols and callchains before passing them to user. And we might think about the safety as it can modify the internal state or something.
Thanks,
Namhyung
Quoted 20 lines
tools/perf/builtin-script.c | 22 +- tools/perf/new_script_templates/README | 65 ++++ tools/perf/new_script_templates/lib.rs | 108 +++++++ tools/perf/new_script_templates/script.c | 113 +++++++ tools/perf/util/Build | 1 + tools/perf/util/new_script.c | 376 +++++++++++++++++++++++ tools/perf/util/new_script.h | 54 ++++ tools/perf/util/new_script_rs_lib.h | 35 +++ 8 files changed, 773 insertions(+), 1 deletion(-) create mode 100644 tools/perf/new_script_templates/README create mode 100644 tools/perf/new_script_templates/lib.rs create mode 100644 tools/perf/new_script_templates/script.c create mode 100644 tools/perf/util/new_script.c create mode 100644 tools/perf/util/new_script.h create mode 100644 tools/perf/util/new_script_rs_lib.hSignature · 1 line
2.46.0