Skip to contentSkip to conversation

mm: kfence: print the age time for alloacted objectes to trace memleak

PATCH In review Qiwu Chen 2 participants 8 messages Aug 3, 2024 – Aug 5, 2024

Subsystems (2)
lore
Share & tools
lore

Qiwu Chen

author
#0 lore raw

PATCH 1/1 mm: kfence: print the age time for alloacted objectes to trace memleak

From: "qiwu.chen" <[email protected]>

For a convienince of tracing slab object leak, print the age time for alloacted objectes in kfence_print_stack().

---
 mm/kfence/report.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
mm/kfence/report.c +12 -2
index c509aed326ce..44c3f82b25a8 100644
@@ -16,6 +16,7 @@
 #include <linux/sprintf.h>
 #include <linux/stacktrace.h>
 #include <linux/string.h>
+#include <linux/sched/clock.h>
 #include <trace/events/error_report.h>
 
 #include <asm/kfence.h>
@@ -110,9 +111,18 @@ static void kfence_print_stack(struct seq_file *seq, const struct kfence_metadat
 	unsigned long rem_nsec = do_div(ts_sec, NSEC_PER_SEC);
 
 	/* Timestamp matches printk timestamp format. */
-	seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus:\n",
+	if (meta->state == KFENCE_OBJECT_ALLOCATED) {
+		u64 interval_nsec = local_clock() - meta->alloc_track.ts_nsec;
+		unsigned long rem_interval_nsec = do_div(interval_nsec, NSEC_PER_SEC);
+
+		seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus (age: %lu.%06lus):\n",
 		       show_alloc ? "allocated" : "freed", track->pid,
-		       track->cpu, (unsigned long)ts_sec, rem_nsec / 1000);
+		       track->cpu, (unsigned long)ts_sec, rem_nsec / 1000,
+			   (unsigned long)interval_nsec, rem_interval_nsec / 1000);
+	} else
+		seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus:\n",
+				   show_alloc ? "allocated" : "freed", track->pid,
+				   track->cpu, (unsigned long)ts_sec, rem_nsec / 1000);
 
 	if (track->num_stack_entries) {
 		/* Skip allocation/free internals stack. */
Signature · 1 line
2.25.1
maintainer
#1 lore raw
Quoted 4 lines · Qiwu Chen

From: "qiwu.chen" <[email protected]>

For a convienince of tracing slab object leak, print the age time for

typo: convenience

What do you mean by "object leak"?

From what I see the additional info is only printed on out-of-bounds access.

Or do you mean when you inspect /sys/kernel/debug/kfence/objects? If so, that information would be useful in the commit message.

However, to detect leaks there are better tools than KFENCE. Have you tried KMEMLEAK? KFENCE is really not a good choice to manually look for old objects, which themselves are sampled, to find leaks. Have you been able to successfully debug a leak this way?

Quoted 1 line

alloacted objectes in kfence_print_stack().

typo: allocated objects

Quoted 18 lines
---
 mm/kfence/report.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
mm/kfence/report.c +1 -0
index c509aed326ce..44c3f82b25a8 100644
@@ -16,6 +16,7 @@
 #include <linux/sprintf.h>
 #include <linux/stacktrace.h>
 #include <linux/string.h>
+#include <linux/sched/clock.h>
 #include <trace/events/error_report.h>
 
 #include <asm/kfence.h>

mm/kfence/report.c static void kfence_print_stack(struct seq_file *seq, const struct kfence_metadat · line 114

Quoted 5 lines
mm/kfence/report.c +1 -1
@@ -110,9 +111,18 @@ static void kfence_print_stack(struct seq_file *seq, const struct kfence_metadat
        unsigned long rem_nsec = do_div(ts_sec, NSEC_PER_SEC);
 
        /* Timestamp matches printk timestamp format. */
-       seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus:\n",
+       if (meta->state == KFENCE_OBJECT_ALLOCATED) {

In principle, the additonal info is convenient, but I'd like to generalize if possible.

mm/kfence/report.c static void kfence_print_stack(struct seq_file *seq, const struct kfence_metadat · line 118

Quoted 4 lines
mm/kfence/report.c +4 -0
+               u64 interval_nsec = local_clock() - meta->alloc_track.ts_nsec;
+               unsigned long rem_interval_nsec = do_div(interval_nsec, NSEC_PER_SEC);
+
+               seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus (age: %lu.%06lus):\n",

I've found myself trying to figure out the elapsed time since the allocation or free, based on the current timestamp.

So something that would be more helpful is if you just change the printed line for all alloc and free stack infos to say something like:

seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus
(%lu.%06lus ago):\n",

So rather than saying this info is the "age", we just say the elapsed time. That generalizes this bit of info, and it'll be available for both alloc and free stacks.

Does that work for you?

mm/kfence/report.c static void kfence_print_stack(struct seq_file *seq, const struct kfence_metadat · line 122

Quoted 5 lines
mm/kfence/report.c +3 -1
                       show_alloc ? "allocated" : "freed", track->pid,
-                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000);
+                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000,
+                          (unsigned long)interval_nsec, rem_interval_nsec / 1000);
+       } else

Add braces {} even though it's a single statement - it spans several lines and the above is also {}-enclosed, so it looks balanced.

But if you follow my suggestion, you won't have the else branch anymore.

Quoted 6 lines
mm/kfence/report.c +3 -0
+               seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus:\n",
+                                  show_alloc ? "allocated" : "freed", track->pid,
+                                  track->cpu, (unsigned long)ts_sec, rem_nsec / 1000);
 
        if (track->num_stack_entries) {
                /* Skip allocation/free internals stack. */
author
#2 lore raw
Quoted 5 lines · Marco Elver

typo: convenience

What do you mean by "object leak"?

It means an allocated object of slab memory which is considered orphan, perhaps it's more clear to say "For a convenience of tracing memory leaks by kfence", what do you think?

Quoted 5 lines

From what I see the additional info is only printed on out-of-bounds access.

Or do you mean when you inspect /sys/kernel/debug/kfence/objects? If so, that information would be useful in the commit message.

The extra elapsed time of current allocated object would be useful to figure out memory leaks when inspect /sys/kernel/debug/kfence/objects.

Quoted 5 lines

However, to detect leaks there are better tools than KFENCE. Have you tried KMEMLEAK? KFENCE is really not a good choice to manually look for old objects, which themselves are sampled, to find leaks. Have you been able to successfully debug a leak this way?

The kmemleak tool has limitations and drawbacks which cannot be used in productive environment directly. KFENCE is a good choice to find leaks in productive environment.

Quoted 4 lines
Quoted 1 line

alloacted objectes in kfence_print_stack().

typo: allocated objects

Thank's for your comment.

Quoted 15 lines

In principle, the additonal info is convenient, but I'd like to generalize if possible.

mm/kfence/report.c static void kfence_print_stack(struct seq_file *seq, const struct kfence_metadat · line 118

Quoted 4 lines
mm/kfence/report.c +4 -0
+               u64 interval_nsec = local_clock() - meta->alloc_track.ts_nsec;
+               unsigned long rem_interval_nsec = do_div(interval_nsec, NSEC_PER_SEC);
+
+               seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus (age: %lu.%06lus):\n",

I've found myself trying to figure out the elapsed time since the allocation or free, based on the current timestamp.

So something that would be more helpful is if you just change the printed line for all alloc and free stack infos to say something like:

Quoted 9 lines

seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus
(%lu.%06lus ago):\n",

So rather than saying this info is the "age", we just say the elapsed time. That generalizes this bit of info, and it'll be available for both alloc and free stacks.

Does that work for you?

It does not work for me actually, since it's unintuitive to figure out memory leaks by the elapsed time of allocated stacks when inspect /sys/kernel/debug/kfence/objects. It's unnecessary to print the elapsed time of allocated stacks for the objects in KFENCE_OBJECT_FREED state. For the elapsed time of free stacks, it seems no available scenarion currently. BTW, The change from "age" to "ago" is okay to me!

Quoted 11 lines

mm/kfence/report.c static void kfence_print_stack(struct seq_file *seq, const struct kfence_metadat · line 122

Quoted 5 lines
                       show_alloc ? "allocated" : "freed", track->pid,
mm/kfence/report.c +3 -1
-                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000);
+                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000,
+                          (unsigned long)interval_nsec, rem_interval_nsec / 1000);
+       } else

Add braces {} even though it's a single statement - it spans several lines and the above is also {}-enclosed, so it looks balanced.

But if you follow my suggestion, you won't have the else branch anymore.

I'm not opposed to convert a single statement, but it will have an effort to find leaks.
This change will be helpful to find leaks easier by grep "ago" keyword when inspect /sys/kernel/debug/kfence/objects.

Thanks
Qiwu

maintainer
#3 lore raw

On Sun, 4 Aug 2024 at 05:46, chenqiwu <[email protected]> wrote:
[...]

Quoted 8 lines
Quoted 8 lines

I've found myself trying to figure out the elapsed time since the allocation or free, based on the current timestamp.

So something that would be more helpful is if you just change the printed line for all alloc and free stack infos to say something like:

    seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus (%lu.%06lus ago):\n",
Quoted 7 lines
Quoted 6 lines

So rather than saying this info is the "age", we just say the elapsed time. That generalizes this bit of info, and it'll be available for both alloc and free stacks.

Does that work for you?

It does not work for me actually, since it's unintuitive to figure out memory leaks

The number printed is the same. It's just the change of "age" to "ago".

Quoted 4 lines

by the elapsed time of allocated stacks when inspect /sys/kernel/debug/kfence/objects. It's unnecessary to print the elapsed time of allocated stacks for the objects in KFENCE_OBJECT_FREED state. For the elapsed time of free stacks, it seems no available scenarion currently. BTW, The change from "age" to "ago" is okay to me!

Well, what I'm saying, having this info also for FREED objects on the free stack can be useful in some debugging scenarios when you get a use-after-free, and you want to know the elapsed time since the free happened. I have done this calculation manually before, which is why I suggested it. Maybe it's not useful for you for finding leaks, but that's just one usecase.

Thanks,
-- Marco

author
#4 lore raw
Quoted 8 lines · Marco Elver

Well, what I'm saying, having this info also for FREED objects on the free stack can be useful in some debugging scenarios when you get a use-after-free, and you want to know the elapsed time since the free happened. I have done this calculation manually before, which is why I suggested it. Maybe it's not useful for you for finding leaks, but that's just one usecase.

Agreed with your concern scenarios.
How about the following change with additonal object state info?

+ u64 interval_nsec = local_clock() - meta->alloc_track.ts_nsec;
+ unsigned long rem_interval_nsec = do_div(interval_nsec, NSEC_PER_SEC);

        /* Timestamp matches printk timestamp format. */
-       seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus:\n",
+       seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus (%lu.%06lus ago) for %s object:\n",
                       show_alloc ? "allocated" : "freed", track->pid,
-                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000);
+                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000,
+                      (unsigned long)interval_nsec, rem_interval_nsec / 1000,
+                      meta->state == KFENCE_OBJECT_ALLOCATED? "allocated" : "freed");

In this way, we can find leaks by grep "allocated object" and inspect the elapsed time of use-after-free by grep "freed object".

Thanks
Qiwu

maintainer
#5 lore raw
Quoted 25 lines · chenqiwu
Quoted 8 lines · Marco Elver

Well, what I'm saying, having this info also for FREED objects on the free stack can be useful in some debugging scenarios when you get a use-after-free, and you want to know the elapsed time since the free happened. I have done this calculation manually before, which is why I suggested it. Maybe it's not useful for you for finding leaks, but that's just one usecase.

Agreed with your concern scenarios.
How about the following change with additonal object state info?

mm/kfence/report.c +6 -2
+       u64 interval_nsec = local_clock() - meta->alloc_track.ts_nsec;
+       unsigned long rem_interval_nsec = do_div(interval_nsec, NSEC_PER_SEC);
 
        /* Timestamp matches printk timestamp format. */
-       seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus:\n",
+       seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus (%lu.%06lus ago) for %s object:\n",
                       show_alloc ? "allocated" : "freed", track->pid,
-                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000);
+                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000,
+                      (unsigned long)interval_nsec, rem_interval_nsec / 1000,
+                      meta->state == KFENCE_OBJECT_ALLOCATED? "allocated" : "freed");
Quoted 2 lines

In this way, we can find leaks by grep "allocated object" and inspect the elapsed time of use-after-free by grep "freed object".

The "allocated/freed" info is superfluous, as freed objects will have a free stack.

Consider a slightly better script vs. just using grep. /sys/kernel/debug/kfence/objects is of secondary concern and was added primarily as a debugging aid for KFENCE developers. We never thought it could be used to look for leaks, but good you found another use for it. ;-)
The priority is to keep regular error reports generated by KFENCE readable. Adding this "allocated/freed" info just makes the line longer and is not useful.

I'm happy with the "(%lu.%06lus ago)" part alone.

author
#6 lore raw
Quoted 5 lines · Marco Elver

The "allocated/freed" info is superfluous, as freed objects will have a free stack.

Consider a slightly better script vs. just using grep.

Well, I think using grep is eaiser than a script to find leaks by a large number of alloc tracks.

Quoted 8 lines

/sys/kernel/debug/kfence/objects is of secondary concern and was added primarily as a debugging aid for KFENCE developers. We never thought it could be used to look for leaks, but good you found another use for it. ;-)
The priority is to keep regular error reports generated by KFENCE readable. Adding this "allocated/freed" info just makes the line longer and is not useful.

How about print meta->state directly to get the object state for its alloc/free track?
- seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus:\n",
+ seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus (%lu.%06lus ago) state %d:\n",

                       show_alloc ? "allocated" : "freed", track->pid,
-                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000);
+                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000,
+                      (unsigned long)interval_nsec, rem_interval_nsec / 1000,
+                      meta->state);
Quoted 1 line

I'm happy with the "(%lu.%06lus ago)" part alone.

If it's still a not good idea, I will follow your suggestion and resend it as v2.

Thanks
Qiwu

maintainer
#7 lore raw
Quoted 9 lines · chenqiwu
Quoted 5 lines · Marco Elver

The "allocated/freed" info is superfluous, as freed objects will have a free stack.

Consider a slightly better script vs. just using grep.

Well, I think using grep is eaiser than a script to find leaks by a large number of alloc tracks.

Sure. But a slightly more complex script is a better trade-off vs. impacting _all_ KFENCE users world-wide with slightly less readable error reports.

Quoted 18 lines
Quoted 8 lines

/sys/kernel/debug/kfence/objects is of secondary concern and was added primarily as a debugging aid for KFENCE developers. We never thought it could be used to look for leaks, but good you found another use for it. ;-)
The priority is to keep regular error reports generated by KFENCE readable. Adding this "allocated/freed" info just makes the line longer and is not useful.

How about print meta->state directly to get the object state for its alloc/free track?

(quoted diff) +4 -2
-       seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus:\n",
+       seq_con_printf(seq, "%s by task %d on cpu %d at %lu.%06lus (%lu.%06lus ago) state %d:\n",
                       show_alloc ? "allocated" : "freed", track->pid,
-                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000);
+                      track->cpu, (unsigned long)ts_sec, rem_nsec / 1000,
+                      (unsigned long)interval_nsec, rem_interval_nsec / 1000,
+                      meta->state);
Quoted 1 line

I'm happy with the "(%lu.%06lus ago)" part alone.

Quoted 2 lines

If it's still a not good idea, I will follow your suggestion and resend it as v2.

No, that's just making it more ugly for no reason. It's replicating the state info (just like before) for alloc and free stacks and generally does not add anything useful.

See, we are writing code that is deployed on millions of machines, and KFENCE error reports do appear in the wild occasionally. We have to optimize for the common case.

Your change might be useful for you, which is a relatively unique usecase. The common use case of KFENCE is to detect memory-safety errors, and good error reports are a major feature of KFENCE. All information is already present in the reports (and /sys/kernel/debug/kfence/objects).

I argue that you are able to write a slightly more complex script that simply looks for the free stack right after the allocation stack to determine if an object is live or freed. Maybe doing it in bash won't work so nicely, but a small Python script can easily do that job. Once you have that Python script you might even do further processing, sort things by age, size, etc. etc., and then print whole stack traces. Just grep can't do that. So if you want something useful, you'd have to give up on grep sooner or later.