Skip to contentSkip to conversation

pgo: Fix allocate_node() v2

PATCH Applied Jarmo Tiitto 5 participants 7 messages Jun 4, 2021 – Jun 5, 2021

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 pgo: Fix allocate_node() v2

When clang instrumentation eventually calls allocate_node() the struct llvm_prf_data *p argument tells us from what section we should reserve the vnode: It either points into vmlinux's core __llvm_prf_data section or some loaded module's __llvm_prf_data section.

But since we don't have access to corresponding __llvm_prf_vnds section(s) for any module, the function should return just NULL and ignore any profiling attempts from modules for now.

---
Based on Kees and others feedback here is v3 patch that clarifies why the current checks in allocate_node() are flawed. I did fair amount of KGDB time on it.

The commit is based on kees/for-next/clang/features tree, hopefully this is ok. Should I have based it on linux-next instead?

I grep -R'd where the memory_contains() can be found and it is only found in #include <asm-generic/sections.h>

I cross my fingers and await if this is my first accepted patch. :-)

---
 kernel/pgo/instrument.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)
kernel/pgo/instrument.c +14 -9
index 0e07ee1b17d9..c69b4f7ebaad 100644
@@ -18,6 +18,7 @@
 
 #define pr_fmt(fmt)	"pgo: " fmt
 
+#include <asm-generic/sections.h>
 #include <linux/bitops.h>
 #include <linux/kernel.h>
 #include <linux/export.h>
@@ -55,17 +56,21 @@ void prf_unlock(unsigned long flags)
 static struct llvm_prf_value_node *allocate_node(struct llvm_prf_data *p,
 						 u32 index, u64 value)
 {
-	if (&__llvm_prf_vnds_start[current_node + 1] >= __llvm_prf_vnds_end)
-		return NULL; /* Out of nodes */
-
-	current_node++;
-
-	/* Make sure the node is entirely within the section */
-	if (&__llvm_prf_vnds_start[current_node] >= __llvm_prf_vnds_end ||
-	    &__llvm_prf_vnds_start[current_node + 1] > __llvm_prf_vnds_end)
+	const int max_vnds = prf_vnds_count();
+
+	/*
+	 * Check that p is within vmlinux __llvm_prf_data section.
+	 * If not, don't allocate since we can't handle modules yet.
+	 */
+	if (!memory_contains(__llvm_prf_data_start,
+		__llvm_prf_data_end, p, sizeof(*p)))
 		return NULL;
 
-	return &__llvm_prf_vnds_start[current_node];
+	if (WARN_ON_ONCE(current_node >= max_vnds))
+		return NULL; /* Out of nodes */
+
+	/* reserve vnode for vmlinux */
+	return &__llvm_prf_vnds_start[current_node++];
 }
 
 /*

base-commit: 5d0cda65918279ada060417c5fecb7e86ccb3def

Signature · 1 line
2.31.1
Reviewed-by on v3 1/1
#1 lore raw
Quoted 6 lines · Jarmo Tiitto

When clang instrumentation eventually calls allocate_node() the struct llvm_prf_data *p argument tells us from what section we should reserve the vnode: It either points into vmlinux's core __llvm_prf_data section or some loaded module's __llvm_prf_data section.

Quoted 6 lines

But since we don't have access to corresponding __llvm_prf_vnds section(s) for any module, the function should return just NULL and ignore any profiling attempts from modules for now.

Quoted 63 lines

---
Based on Kees and others feedback here is v3 patch that clarifies why the current checks in allocate_node() are flawed. I did fair amount of KGDB time on it.

The commit is based on kees/for-next/clang/features tree, hopefully this is ok. Should I have based it on linux-next instead?

I grep -R'd where the memory_contains() can be found and it is only found in #include <asm-generic/sections.h>

I cross my fingers and await if this is my first accepted patch. :-)

---
  kernel/pgo/instrument.c | 23 ++++++++++++++---------
  1 file changed, 14 insertions(+), 9 deletions(-)
kernel/pgo/instrument.c +14 -9
index 0e07ee1b17d9..c69b4f7ebaad 100644
@@ -18,6 +18,7 @@
 
  #define pr_fmt(fmt)	"pgo: " fmt
 
+#include <asm-generic/sections.h>
  #include <linux/bitops.h>
  #include <linux/kernel.h>
  #include <linux/export.h>
@@ -55,17 +56,21 @@ void prf_unlock(unsigned long flags)
  static struct llvm_prf_value_node *allocate_node(struct llvm_prf_data *p,
  						 u32 index, u64 value)
  {
-	if (&__llvm_prf_vnds_start[current_node + 1] >= __llvm_prf_vnds_end)
-		return NULL; /* Out of nodes */
-
-	current_node++;
-
-	/* Make sure the node is entirely within the section */
-	if (&__llvm_prf_vnds_start[current_node] >= __llvm_prf_vnds_end ||
-	    &__llvm_prf_vnds_start[current_node + 1] > __llvm_prf_vnds_end)
+	const int max_vnds = prf_vnds_count();
+
+	/*
+	 * Check that p is within vmlinux __llvm_prf_data section.
+	 * If not, don't allocate since we can't handle modules yet.
+	 */
+	if (!memory_contains(__llvm_prf_data_start,
+		__llvm_prf_data_end, p, sizeof(*p)))
  		return NULL;
 
-	return &__llvm_prf_vnds_start[current_node];
+	if (WARN_ON_ONCE(current_node >= max_vnds))
+		return NULL; /* Out of nodes */
+
+	/* reserve vnode for vmlinux */
+	return &__llvm_prf_vnds_start[current_node++];
  }
 
  /*

base-commit: 5d0cda65918279ada060417c5fecb7e86ccb3def

#2 lore raw
In reply to Jarmo Tiitto
Quoted 35 lines · Jarmo Tiitto

When clang instrumentation eventually calls allocate_node() the struct llvm_prf_data *p argument tells us from what section we should reserve the vnode: It either points into vmlinux's core __llvm_prf_data section or some loaded module's __llvm_prf_data section.

But since we don't have access to corresponding __llvm_prf_vnds section(s) for any module, the function should return just NULL and ignore any profiling attempts from modules for now.

---
Based on Kees and others feedback here is v3 patch that clarifies why the current checks in allocate_node() are flawed. I did fair amount of KGDB time on it.

The commit is based on kees/for-next/clang/features tree, hopefully this is ok. Should I have based it on linux-next instead?

I grep -R'd where the memory_contains() can be found and it is only found in #include <asm-generic/sections.h>

I cross my fingers and await if this is my first accepted patch. :-)

---
 kernel/pgo/instrument.c | 23 ++++++++++++++---------
 1 file changed, 14 insertions(+), 9 deletions(-)
kernel/pgo/instrument.c +0 -0
index 0e07ee1b17d9..c69b4f7ebaad 100644

kernel/pgo/instrument.c line 21

Quoted 4 lines
kernel/pgo/instrument.c +1 -0
@@ -18,6 +18,7 @@
 
 #define pr_fmt(fmt)    "pgo: " fmt
 
+#include <asm-generic/sections.h>

^ What about Nathan's feedback on this being just <asm/sections.h>?

Quoted 39 lines
kernel/pgo/instrument.c +13 -9
 #include <linux/bitops.h>
 #include <linux/kernel.h>
 #include <linux/export.h>
@@ -55,17 +56,21 @@ void prf_unlock(unsigned long flags)
 static struct llvm_prf_value_node *allocate_node(struct llvm_prf_data *p,
                                                 u32 index, u64 value)
 {
-       if (&__llvm_prf_vnds_start[current_node + 1] >= __llvm_prf_vnds_end)
-               return NULL; /* Out of nodes */
-
-       current_node++;
-
-       /* Make sure the node is entirely within the section */
-       if (&__llvm_prf_vnds_start[current_node] >= __llvm_prf_vnds_end ||
-           &__llvm_prf_vnds_start[current_node + 1] > __llvm_prf_vnds_end)
+       const int max_vnds = prf_vnds_count();
+
+       /*
+        * Check that p is within vmlinux __llvm_prf_data section.
+        * If not, don't allocate since we can't handle modules yet.
+        */
+       if (!memory_contains(__llvm_prf_data_start,
+               __llvm_prf_data_end, p, sizeof(*p)))
                return NULL;
 
-       return &__llvm_prf_vnds_start[current_node];
+       if (WARN_ON_ONCE(current_node >= max_vnds))
+               return NULL; /* Out of nodes */
+
+       /* reserve vnode for vmlinux */
+       return &__llvm_prf_vnds_start[current_node++];
 }
 
 /*

base-commit: 5d0cda65918279ada060417c5fecb7e86ccb3def --
2.31.1

Signature · 2 lines
Thanks,
~Nick Desaulniers
Applied
#3 lore raw
In reply to Jarmo Tiitto
Quoted 10 lines · Jarmo Tiitto

When clang instrumentation eventually calls allocate_node() the struct llvm_prf_data *p argument tells us from what section we should reserve the vnode: It either points into vmlinux's core __llvm_prf_data section or some loaded module's __llvm_prf_data section.

But since we don't have access to corresponding __llvm_prf_vnds section(s) for any module, the function should return just NULL and ignore any profiling attempts from modules for now.

Thanks for working on this! I tweaked the commit title, reflowed the commit log to 80 columns, and adjusted asm-generic to asm.

Applied to for-next/clang/features, thanks!

[1/1] pgo: Limit allocate_node() to vmlinux sections

      https://git.kernel.org/kees/c/46773f32ddf1
Signature · 1 line
Kees Cook
#4 lore raw
In reply to Jarmo Tiitto
Quoted 17 lines · Jarmo Tiitto

When clang instrumentation eventually calls allocate_node() the struct llvm_prf_data *p argument tells us from what section we should reserve the vnode: It either points into vmlinux's core __llvm_prf_data section or some loaded module's __llvm_prf_data section.

But since we don't have access to corresponding __llvm_prf_vnds section(s) for any module, the function should return just NULL and ignore any profiling attempts from modules for now.

---
Based on Kees and others feedback here is v3 patch that clarifies why the current checks in allocate_node() are flawed. I did fair amount of KGDB time on it.

Quoted 6 lines

The commit is based on kees/for-next/clang/features tree, hopefully this is ok. Should I have based it on linux-next instead?

I grep -R'd where the memory_contains() can be found and it is only found in #include <asm-generic/sections.h>

That's true, but the way to use "asm-generic" is to include the top-level "asm" file, so that architectures can override things as needed.

Quoted 1 line

I cross my fingers and await if this is my first accepted patch. :-)

I tweaked it a bit and applied it (see the separate email).

Thank you!

-Kees

Signature · 1 line
Kees Cook
#5 lore raw
Quoted 9 lines

Kees Cook wrote perjantaina 4. kesäkuuta 2021 21.06.37 EEST:

Quoted 3 lines

I grep -R'd where the memory_contains() can be found and it is only found in #include <asm-generic/sections.h>

That's true, but the way to use "asm-generic" is to include the top-level "asm" file, so that architectures can override things as needed.

Thanks, I didn't know that.

Quoted 11 lines
Quoted 1 line

I cross my fingers and await if this is my first accepted patch. :-)

I tweaked it a bit and applied it (see the separate email).

Thank you!

-Kees

Signature · 1 line
Kees Cook

Whoa!
Thanks, I'm glad it worked out. :-)

Btw. I have almost forgotten that I once wrote code (that I didn't send) for the GCC gcov subsystem that also enabled -fprofile-generate/use for the kernel.
However the Clang PGO looks much more approachable and easier to hack on since the profile data format is simpler.

So starting to work on this felt just natural to me. :-)

-Jarmo

#6 lore raw
Quoted 32 lines · Jarmo Tiitto
Quoted 9 lines

Kees Cook wrote perjantaina 4. kesäkuuta 2021 21.06.37 EEST:

Quoted 3 lines

I grep -R'd where the memory_contains() can be found and it is only found in #include <asm-generic/sections.h>

That's true, but the way to use "asm-generic" is to include the top-level "asm" file, so that architectures can override things as needed.

Thanks, I didn't know that.

Quoted 11 lines
Quoted 1 line

I cross my fingers and await if this is my first accepted patch. :-)

I tweaked it a bit and applied it (see the separate email).

Thank you!

-Kees

--
Kees Cook

Whoa!
Thanks, I'm glad it worked out. :-)

Btw. I have almost forgotten that I once wrote code (that I didn't send) for the GCC gcov subsystem that also enabled -fprofile-generate/use for the kernel.
However the Clang PGO looks much more approachable and easier to hack on since the profile data format is simpler.

Quoted 3 lines

So starting to work on this felt just natural to me. :-)

-Jarmo

Right, __llvm_prf_vnodes reserves space for static allocation. There is no relocation referencing __llvm_prf_vnodes so there is no straightforward way using the space for the kernel.

In userspace shared objects, the scheme works by linking libclang_rt.profile-*.a into every shared object. The __start___llvm_prf_vnodes/__stop___llvm_prf_vnodes symbols are delieberately hidden in compiler-rt InstrProfilingPlatformLinux.c.

The GCC gcov format is definitely simpler than the LLVM format:)