Skip to contentSkip to conversation

C and Rust support for perf script

RFC In review Stefan Ene 2 participants 5 messages Sep 19, 2024 – Sep 26, 2024 perf events

lore
Share & tools
lore

Stefan Ene

#0 lore raw

cover C and Rust support for perf script

From: Stefan Ene <[email protected]>

============================COVER=LETTER============================

This proposal is addressing the usability and performance of the available scripting languages for custom perf event data processing inside of the perf toolset, specifically with the perf script command.

With the perf-script custom event processing functionality for C and Rust, we noticed between 2x to 5x speed improvement with our new methods compared to the existent Python and Perl scripting methods.

To explain the proposed method, you begin with the C or Rust script we’ve templatized, then just add changes for custom event processing in the preferred language using this provided template, compile the respective script into a dynamic library, then give the resulting .so file as a parameter to the newly implemented perf script option.

List of functionality additions:

1/ Added new "--new_script" option inside the perf script command that takes in as parameter .so files. Code addition in tools/perf/builtin-script.c

2/ The functional code for the newly implemented option was added to the tools/perf/util/new_script.c and tools/perf/util/new_script.h files.

3/ Folder added at tools/perf/new_script_templates, containing C and Rust script templates for the new option, along with compilation instructions.

4/ Finally, a short bash script for updating the perf tool set within the kernel code was provided in the base-level file update_perf_tools.sh.

Common Questions

* How can I use the new toolset?

The new implementations for the perf script have a detailed usage guide inside of the tools/perf/new_script_templates/README file, along with some script templates for C and Rust!

* Why a new option instead of expanding dlfilter?

The new option gave us the flexibility to make use of the fast dlfilter dynamic library approach, as opposed to implementing another interpreting methodology. This allows for scalability, with great potential to other languages supporting dynamic library calls from the base C code.

* Why use C and Rust instead?

As of kernel version 6.11, the perf tool has a large overhead for data processing using Python and Perl, given the languages having to use their respective perf built-in interpreters. Furthermore, while Python is widley used in the development comunity, as of 20204, Perl is only used by 2.5 of developers worldwide, while C and Rust are more common, with 20.3% and 12.6% usage, respectively (Source: statista.com).

* What are the actual performance improvements?

As last tested, the C and Rust approach are anywhere between 2 to 5 times faster than the existent Python and Perl scripting methods, with Rust being the fastest all across!

Acknowledgements:

This code was completed as part of an Intel summer internship project, under the mentoring of Vinicius Gomes, Intel Linux Kernel Team.

=========================END=COVER=LETTER===========================

Stefan Ene (3):
add the new perf script option (--new_script) and related changes added the C sample script
added the Rust sample script

 tools/perf/builtin-script.c              |  22 +-
 tools/perf/new_script_templates/README   |  65 ++++
 tools/perf/new_script_templates/lib.rs   | 108 +++++++
 tools/perf/new_script_templates/script.c | 113 +++++++
 tools/perf/util/Build                    |   1 +
 tools/perf/util/new_script.c             | 376 +++++++++++++++++++++++
 tools/perf/util/new_script.h             |  54 ++++
 tools/perf/util/new_script_rs_lib.h      |  35 +++
 8 files changed, 773 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/new_script_templates/README
 create mode 100644 tools/perf/new_script_templates/lib.rs
 create mode 100644 tools/perf/new_script_templates/script.c
 create mode 100644 tools/perf/util/new_script.c
 create mode 100644 tools/perf/util/new_script.h
 create mode 100644 tools/perf/util/new_script_rs_lib.h
Signature · 1 line
2.46.0
#1 lore raw

RFC 1/3 C and Rust support for perf script

From: Stefan Ene <[email protected]>

[PATCH 1/3] add the new perf script option (--new_script) and related
changes

---
 tools/perf/builtin-script.c  |  22 +-
 tools/perf/util/Build        |   1 +
 tools/perf/util/new_script.c | 376 +++++++++++++++++++++++++++++++++++
 tools/perf/util/new_script.h |  54 +++++
 4 files changed, 452 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/util/new_script.c
 create mode 100644 tools/perf/util/new_script.h
tools/perf/builtin-script.c +21 -1
index c16224b1fef3..e91a1e2481bb 100644
@@ -63,6 +63,7 @@
 #include "util/util.h"
 #include "util/cgroup.h"
 #include "perf.h"
+#include "util/new_script.h"
 
 #include <linux/ctype.h>
 #ifdef HAVE_LIBTRACEEVENT
@@ -88,6 +89,7 @@ static struct perf_stat_config	stat_config;
 static int			max_blocks;
 static bool			native_arch;
 static struct dlfilter		*dlfilter;
+static struct new_script	*new_struct;
 static int			dlargc;
 static char			**dlargv;
 
@@ -3898,6 +3900,7 @@ int cmd_script(int argc, const char **argv)
 	struct utsname uts;
 	char *script_path = NULL;
 	const char *dlfilter_file = NULL;
+	const char *new_script_file = NULL;
 	const char **__argv;
 	int i, j, err = 0;
 	struct perf_script script = {
@@ -3954,6 +3957,7 @@ int cmd_script(int argc, const char **argv)
 	OPT_STRING('g', "gen-script", &generate_script_lang, "lang",
 		   "generate perf-script.xx script in specified language"),
 	OPT_STRING(0, "dlfilter", &dlfilter_file, "file", "filter .so file name"),
+	OPT_STRING(0, "new_script", &new_script_file, "file", "specify .so file name"),
 	OPT_CALLBACK(0, "dlarg", NULL, "argument", "filter argument",
 		     add_dlarg),
 	OPT_STRING('i', "input", &input_name, "file", "input file name"),
@@ -4355,6 +4359,21 @@ int cmd_script(int argc, const char **argv)
 		goto out_delete;
 	}
 #endif
+
+	if (new_script_file) {
+		new_struct = new_script__new(new_script_file);
+
+		if (!new_struct) {
+			perror("cannot create new_script objects\n");
+			goto out_delete;
+		}
+		if (new_script__start(new_struct, &data) < 0) {
+			perror("cannot start new_script object\n");
+			goto out_delete;  
+		}
+		goto out_delete;
+	}
+
 	if (generate_script_lang) {
 		struct stat perf_stat;
 		int input;
@@ -4458,7 +4477,8 @@ int cmd_script(int argc, const char **argv)
 	if (script_started)
 		cleanup_scripting();
 	dlfilter__cleanup(dlfilter);
+	new_script__cleanup(new_struct);
 	free_dlarg();
 out:
 	return err;
-}
+}
\ No newline at end of file
tools/perf/util/Build +1 -0
index 0f18fe81ef0b..386cdd0fe13c 100644
@@ -140,6 +140,7 @@ perf-util-y += parse-sublevel-options.o
 perf-util-y += term.o
 perf-util-y += help-unknown-cmd.o
 perf-util-y += dlfilter.o
+perf-util-y += new_script.o
 perf-util-y += mem-events.o
 perf-util-y += mem-info.o
 perf-util-y += vsprintf.o
tools/perf/util/new_script.c new file +376 -0

377 lines not on this page · raw message

tools/perf/util/new_script.h new file +54 -0

55 lines not on this page · raw message

Signature · 1 line
2.46.0
#2 lore raw

RFC 2/3 C and Rust support for perf script

In reply to Stefan Ene

From: Stefan Ene <[email protected]>

[PATCH 2/3] added the C sample script

---
 tools/perf/new_script_templates/README   |  16 ++++
 tools/perf/new_script_templates/script.c | 107 +++++++++++++++++++++++
 2 files changed, 123 insertions(+)
 create mode 100644 tools/perf/new_script_templates/README
 create mode 100644 tools/perf/new_script_templates/script.c
tools/perf/new_script_templates/README new file +16 -0
new file mode 100644
index 000000000000..c2a55a65d444
@@ -0,0 +1,16 @@
+Linux kernel additions for C and Rust script support inside the perf-script tool set.
+
+Steps to use new feature:
+
+    First, use the provided update_perf_tools.sh script to make sure your perf toolset is up to date with the latest implementation:
+        $ bash update_perf_tools.sh
+
+    a) For C scripts:
+        1. Use the default C template test.c to write your own custom perf event processing
+
+        2. Compile the C script into a dynamic library using the following two commands:
+            $ gcc -c -I ~/include -fpic new_script_templates/script.c
+            $ gcc -shared -o test.so new_script_templates/script.o
+
+        3. Call the new perf script option to use the newly created .so file using the command:
+            $ sudo perf script --new_script new_script_templates/script.so
\ No newline at end of file
tools/perf/new_script_templates/script.c new file +107 -0

108 lines not on this page · raw message

Signature · 1 line
2.46.0
#3 lore raw

RFC 3/3 C and Rust support for perf script

In reply to Stefan Ene

From: Stefan Ene <[email protected]>

[PATCH 3/3] added the Rust sample script

---
 tools/perf/new_script_templates/README   |  61 +++++++++++--
 tools/perf/new_script_templates/lib.rs   | 108 +++++++++++++++++++++++
 tools/perf/new_script_templates/script.c |   6 ++
 tools/perf/util/new_script_rs_lib.h      |  35 ++++++++
 4 files changed, 204 insertions(+), 6 deletions(-)
 create mode 100644 tools/perf/new_script_templates/lib.rs
 create mode 100644 tools/perf/util/new_script_rs_lib.h
tools/perf/new_script_templates/README +55 -6
index c2a55a65d444..f16a9759d37e 100644
@@ -2,15 +2,64 @@ Linux kernel additions for C and Rust script support inside the perf-script tool
 
 Steps to use new feature:
 
-    First, use the provided update_perf_tools.sh script to make sure your perf toolset is up to date with the latest implementation:
-        $ bash update_perf_tools.sh
+    First, use run the following lines to make sure your perf toolset is up to date with the latest implementation:
+        $ cd ./tools/perf/
+        $ make
+        $ sudo cp perf /usr/bin
+        $ perf --version
+    Now, you can make use of our new features!
+
 
     a) For C scripts:
-        1. Use the default C template test.c to write your own custom perf event processing
+        1. Use the default C template in tools/perf/new_script_templates/script.c to write your own custom perf event processing
 
         2. Compile the C script into a dynamic library using the following two commands:
-            $ gcc -c -I ~/include -fpic new_script_templates/script.c
-            $ gcc -shared -o test.so new_script_templates/script.o
+            $ gcc -c -I ~/include -fpic script.c
+            $ gcc -shared -o script.so script.o
 
         3. Call the new perf script option to use the newly created .so file using the command:
-            $ sudo perf script --new_script new_script_templates/script.so
\ No newline at end of file
+            $ sudo perf script --new_script script.so
+
+
+     b) For Rust scripts:
+          1. Create a new Rust project using Cargo:
+               $ cargo new rslib_script --lib
+               $ cd rslib_script
+          
+          2. In the Cargo.toml file, specify the crate type as a dynamic library as follows:
+               [lib]
+               crate-type = ["cdylib"]
+          
+          3. Use the default Rust template in tools/perf/new_script_templates/lib.rs to write your own custom perf event processing, and store it into src/lib.rs inside of the Rust project folder.
+
+          4. Compile the Rust project using Cargo:
+               $ cargo build --release
+          
+          5. Call the new perf script option to use the newly created /target/release/librslib.so within the Rust project using the followig command:
+               $  sudo perf script --new_script rslib/target/release/librslib.so
+
+Enjoy using the new scripting languages for an added bonus of usability and performance over the existent Python and Perl options in the upstream kernel code. 
+
+
+<!-- ============================= -->
+
+This section contains some initial benchmark results on the Intel Lab machine using the perf script tools on a 4026869-byte perf.data file.
+
+Process ----- Time
+
+> Perf tool vanilla:
+perf script raw: 1.425s
+perf script names: 0.915s
+perf script Python: 1.334s
+perf script Perl: 2.210s
+
+
+> Pre-optimizations:
+new_script raw C: 0.284s
+new_script raw Rust: 0.607s
+
+> Post-optimizations:
+new_script raw C batch=1MB: 0.306s 
+new_script raw C batch=2MB: 0.301s 
+new_script raw C batch=4MB: 0.296s 
+new_script raw Rust batch=1MB: 0.262s
\ No newline at end of file
tools/perf/new_script_templates/lib.rs new file +108 -0

109 lines not on this page · raw message

tools/perf/new_script_templates/script.c +6 -0
index e4942ba689db..787f67424d5e 100644
@@ -1,3 +1,9 @@
+// Stefan Ene's code, under Intel - Linux Kernel Team
+/*
+ * code for new perf script custom C script template
+ *
+ */
+
 #include <stdlib.h>
 #include <stdio.h>
 
tools/perf/util/new_script_rs_lib.h new file +35 -0
new file mode 100644
index 000000000000..4a8d6f71b122
@@ -0,0 +1,35 @@
+// Stefan Ene's code, under Intel - Linux Kernel Team
+/*
+ * new_script_lib_rs.h: header file for new_script Rust FFI
+ *
+ */
+
+#ifndef NEW_SCRIPT_LIB_RS_H
+#define NEW_SCRIPT_LIB_RS_H
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <stdint.h>
+
+// Match the struct definitions with the Rust code
+struct NewScript {
+    char		*file;
+    void        *handle;
+
+    void (*begin)(void *data, void *ctx);
+	void (*end)(void *data, void *ctx);
+
+    int (*process_file_header)(void *data, void *ctx);
+    int (*process_event_header)(void *data, void *ctx);
+    int (*process_event_raw_data)(void *data, const int size, void *ctx);
+};
+
+void initialize_new_script(struct NewScript* s);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif // NEW_SCRIPT_LIB_RS_H
\ No newline at end of file
Signature · 1 line
2.46.0
maintainer
#4 lore raw
In reply to Stefan Ene

Hello,

Quoted 44 lines · StefanEne

From: Stefan Ene <[email protected]>

============================COVER=LETTER============================

This proposal is addressing the usability and performance of the available scripting languages for custom perf event data processing inside of the perf toolset, specifically with the perf script command.

With the perf-script custom event processing functionality for C and Rust, we noticed between 2x to 5x speed improvement with our new methods compared to the existent Python and Perl scripting methods.

To explain the proposed method, you begin with the C or Rust script we’ve templatized, then just add changes for custom event processing in the preferred language using this provided template, compile the respective script into a dynamic library, then give the resulting .so file as a parameter to the newly implemented perf script option.

List of functionality additions:

1/ Added new "--new_script" option inside the perf script command that takes in as parameter .so files. Code addition in tools/perf/builtin-script.c

2/ The functional code for the newly implemented option was added to the tools/perf/util/new_script.c and tools/perf/util/new_script.h files.

3/ Folder added at tools/perf/new_script_templates, containing C and Rust script templates for the new option, along with compilation instructions.

4/ Finally, a short bash script for updating the perf tool set within the kernel code was provided in the base-level file update_perf_tools.sh.

Common Questions

* How can I use the new toolset?

The new implementations for the perf script have a detailed usage guide inside of the tools/perf/new_script_templates/README file, along with some script templates for C and Rust!

* Why a new option instead of expanding dlfilter?

The new option gave us the flexibility to make use of the fast dlfilter dynamic library approach, as opposed to implementing another interpreting methodology. This allows for scalability, with great potential to other languages supporting dynamic library calls from the base C code.

* Why use C and Rust instead?

As of kernel version 6.11, the perf tool has a large overhead for data processing using Python and Perl, given the languages having to use their respective perf built-in interpreters. Furthermore, while Python is widley used in the development comunity, as of 20204, Perl is only used by 2.5 of developers worldwide, while C and Rust are more common, with 20.3% and 12.6% usage, respectively (Source: statista.com).

* What are the actual performance improvements?

As last tested, the C and Rust approach are anywhere between 2 to 5 times faster than the existent Python and Perl scripting methods, with Rust being the fastest all across!

Acknowledgements:

This code was completed as part of an Intel summer internship project, under the mentoring of Vinicius Gomes, Intel Linux Kernel Team.

Quoted 7 lines

=========================END=COVER=LETTER===========================

Stefan Ene (3):
add the new perf script option (--new_script) and related changes added the C sample script
added the Rust sample script

Thanks for sharing your work. But I think there are many more work to support scripting of native languages properly. For example, you need to resolve symbols and callchains before passing them to user. And we might think about the safety as it can modify the internal state or something.

Thanks,
Namhyung

Quoted 20 lines
 tools/perf/builtin-script.c              |  22 +-
 tools/perf/new_script_templates/README   |  65 ++++
 tools/perf/new_script_templates/lib.rs   | 108 +++++++
 tools/perf/new_script_templates/script.c | 113 +++++++
 tools/perf/util/Build                    |   1 +
 tools/perf/util/new_script.c             | 376 +++++++++++++++++++++++
 tools/perf/util/new_script.h             |  54 ++++
 tools/perf/util/new_script_rs_lib.h      |  35 +++
 8 files changed, 773 insertions(+), 1 deletion(-)
 create mode 100644 tools/perf/new_script_templates/README
 create mode 100644 tools/perf/new_script_templates/lib.rs
 create mode 100644 tools/perf/new_script_templates/script.c
 create mode 100644 tools/perf/util/new_script.c
 create mode 100644 tools/perf/util/new_script.h
 create mode 100644 tools/perf/util/new_script_rs_lib.h
Signature · 1 line
2.46.0