Skip to contentSkip to conversation

tracing: fix memcpy size when copying stack entries

PATCH In review Sven Schnelle 3 participants 11 messages Jun 12, 2023 – Jul 12, 2023 Tracing

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tracing: fix memcpy size when copying stack entries

Noticed the following warning during boot:

[    2.316341] Testing tracer wakeup:
[    2.383512] ------------[ cut here ]------------
[    2.383517] memcpy: detected field-spanning write (size 104) of single field "&entry->caller" at kernel/trace/trace.c:3167 (size 64)

The reason seems to be that the maximum number of entries is calculated from the size of the fstack->calls array which is 128. But later the same size is used to memcpy() the entries to entry->callers, which has only room for eight elements. Therefore use the minimum of both arrays as limit.

---
 kernel/trace/trace.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace.c +1 -1
index 64a4dde073ef..988d664c13ec 100644
@@ -3146,7 +3146,7 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
 	barrier();
 
 	fstack = this_cpu_ptr(ftrace_stacks.stacks) + stackidx;
-	size = ARRAY_SIZE(fstack->calls);
+	size = min(ARRAY_SIZE(entry->caller), ARRAY_SIZE(fstack->calls));
 
 	if (regs) {
 		nr_entries = stack_trace_save_regs(regs, fstack->calls,
Signature · 1 line
2.39.2
maintainer
#1 lore raw
Quoted 21 lines · On Mon, 12 Jun 2023 18:07:48 +0200 Sven Schnelle <[email protected]> wrote:

Noticed the following warning during boot:

[    2.316341] Testing tracer wakeup:
[    2.383512] ------------[ cut here ]------------
[    2.383517] memcpy: detected field-spanning write (size 104) of single field "&entry->caller" at kernel/trace/trace.c:3167 (size 64)

The reason seems to be that the maximum number of entries is calculated from the size of the fstack->calls array which is 128. But later the same size is used to memcpy() the entries to entry->callers, which has only room for eight elements. Therefore use the minimum of both arrays as limit.

---
 kernel/trace/trace.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace.c +0 -0
index 64a4dde073ef..988d664c13ec 100644

kernel/trace/trace.c static void __ftrace_trace_stack(struct trace_buffer *buffer, · line 3149

Quoted 5 lines
kernel/trace/trace.c +1 -1
@@ -3146,7 +3146,7 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
 	barrier();
 
 	fstack = this_cpu_ptr(ftrace_stacks.stacks) + stackidx;
-	size = ARRAY_SIZE(fstack->calls);
+	size = min(ARRAY_SIZE(entry->caller), ARRAY_SIZE(fstack->calls));

No, this is not how it works, and this breaks the stack tracing code.

kernel/trace/trace.c static void __ftrace_trace_stack(struct trace_buffer *buffer, · line 3152

Quoted 3 lines
kernel/trace/trace.c +0 -0
 	if (regs) {
 		nr_entries = stack_trace_save_regs(regs, fstack->calls,

I guess we need to add some type of annotation to make the memcpy() checking happy.

Let me explain what is happening. By default the stack trace has a minimum of 8 entries (defined by struct stack_entry, which is used to show to user space the default size - for backward compatibility).

Let's take a look at the code in more detail:

/* What is the size of the temp buffer to use to find the stack? */

	size = ARRAY_SIZE(fstack->calls);

	if (regs) {
/* Fills in the stack into the temp buffer */
		nr_entries = stack_trace_save_regs(regs, fstack->calls,
						   size, skip);
	} else {
/* Also fills in the stack into the temp buffer */
		nr_entries = stack_trace_save(fstack->calls, size, skip);
	}

/* Calculate the size from the number of entries stored in the temp buffer */

	size = nr_entries * sizeof(unsigned long);

/* Now reserve space on the ring buffer */
event = __trace_buffer_lock_reserve(buffer, TRACE_STACK,

/*
* Notice how it calculates the size! It subtracts the sizeof
* entry->caller and then adds size again!
*/

				    (sizeof(*entry) - sizeof(entry->caller)) + size,
				    trace_ctx);
	if (!event)
		goto out;

/* Point entry to the ring buffer data */

	entry = ring_buffer_event_data(event);

/* Now copy the stack to the location for the data on the ftrace ring buffer */

	memcpy(&entry->caller, fstack->calls, size);
	entry->size = nr_entries;

The old way use to just record the 8 entries, but that was not very useful in real world analysis. Your patch takes that away. Might as well just record directly into the ring buffer again like it use to.

Yes the above may be special, but your patch breaks it.

NAK on the patch, but I'm willing to update this to make your tooling handle this special case.

-- Steve

#2 lore raw
Quoted 36 lines · Steven Rostedt
Quoted 21 lines · On Mon, 12 Jun 2023 18:07:48 +0200 Sven Schnelle <[email protected]> wrote:

Noticed the following warning during boot:

[    2.316341] Testing tracer wakeup:
[    2.383512] ------------[ cut here ]------------
[    2.383517] memcpy: detected field-spanning write (size 104) of single field "&entry->caller" at kernel/trace/trace.c:3167 (size 64)

The reason seems to be that the maximum number of entries is calculated from the size of the fstack->calls array which is 128. But later the same size is used to memcpy() the entries to entry->callers, which has only room for eight elements. Therefore use the minimum of both arrays as limit.

---
 kernel/trace/trace.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace.c +0 -0
index 64a4dde073ef..988d664c13ec 100644

kernel/trace/trace.c static void __ftrace_trace_stack(struct trace_buffer *buffer, · line 3149

Quoted 5 lines
kernel/trace/trace.c +1 -1
@@ -3146,7 +3146,7 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
 	barrier();
 
 	fstack = this_cpu_ptr(ftrace_stacks.stacks) + stackidx;
-	size = ARRAY_SIZE(fstack->calls);
+	size = min(ARRAY_SIZE(entry->caller), ARRAY_SIZE(fstack->calls));

No, this is not how it works, and this breaks the stack tracing code. [..]
The old way use to just record the 8 entries, but that was not very useful in real world analysis. Your patch takes that away. Might as well just record directly into the ring buffer again like it use to.

Quoted 1 line

Yes the above may be special, but your patch breaks it.

Indeed, i'm feeling a bit stupid for sending that patch, should have used my brain during reading the source. Thanks for the explanation.

maintainer
#3 lore raw
Quoted 4 lines · On Tue, 13 Jun 2023 07:19:14 +0200 Sven Schnelle <[email protected]> wrote:
Quoted 1 line

Yes the above may be special, but your patch breaks it.

Indeed, i'm feeling a bit stupid for sending that patch, should have used my brain during reading the source. Thanks for the explanation.

Does this quiet the fortifier?

-- Steve

kernel/trace/trace.c +13 -1
index 64a4dde073ef..1bac7df1f4b6 100644
@@ -3118,6 +3118,7 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
 	struct ftrace_stack *fstack;
 	struct stack_entry *entry;
 	int stackidx;
+	void *stack;
 
 	/*
 	 * Add one, for this function and the call to save_stack_trace()
@@ -3163,7 +3164,18 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
 		goto out;
 	entry = ring_buffer_event_data(event);
 
-	memcpy(&entry->caller, fstack->calls, size);
+	/*
+	 * For backward compatibility reasons, the entry->caller is an
+	 * array of 8 slots to store the stack. This is also exported
+	 * to user space. The amount allocated on the ring buffer actually
+	 * holds enough for the stack specified by nr_entries. This will
+	 * go into the location of entry->caller. Due to string fortifiers
+	 * checking the size of the destination of memcpy() it triggers
+	 * when it detects that size is greater than 8. To hide this from
+	 * the fortifiers, use a different pointer "stack".
+	 */
+	stack = (void *)&entry->caller;
+	memcpy(stack, fstack->calls, size);
 	entry->size = nr_entries;
 
 	if (!call_filter_check_discard(call, entry, buffer, event))
#4 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 4 lines · On Tue, 13 Jun 2023 07:19:14 +0200 Sven Schnelle <[email protected]> wrote:
Quoted 1 line

Yes the above may be special, but your patch breaks it.

Indeed, i'm feeling a bit stupid for sending that patch, should have used my brain during reading the source. Thanks for the explanation.

Does this quiet the fortifier?
[..]

No, still getting the same warning:

[    2.302776] memcpy: detected field-spanning write (size 104) of single field "stack" at kernel/trace/trace.c:3178 (size 64)
#5 lore raw

From: Sven Schnelle

Quoted 12 lines

Sent: 14 June 2023 11:41

Quoted 8 lines · Steven Rostedt
Quoted 4 lines · On Tue, 13 Jun 2023 07:19:14 +0200 Sven Schnelle <[email protected]> wrote:
Quoted 1 line

Yes the above may be special, but your patch breaks it.

Indeed, i'm feeling a bit stupid for sending that patch, should have used my brain during reading the source. Thanks for the explanation.

Quoted 7 lines
Quoted 2 lines

Does this quiet the fortifier?
[..]

No, still getting the same warning:

[    2.302776] memcpy: detected field-spanning write (size 104) of single field "stack" at

kernel/trace/trace.c:3178 (size 64)

What about:

	(memcpy)(......)

Or maybe:

	(__builtin_memcpy)(....)


	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#6 lore raw
In reply to Sven Schnelle

Hi Steven,

Quoted 10 lines · Sven Schnelle
Quoted 8 lines · Steven Rostedt
Quoted 4 lines · On Tue, 13 Jun 2023 07:19:14 +0200 Sven Schnelle <[email protected]> wrote:
Quoted 1 line

Yes the above may be special, but your patch breaks it.

Indeed, i'm feeling a bit stupid for sending that patch, should have used my brain during reading the source. Thanks for the explanation.

Quoted 6 lines
Quoted 2 lines

Does this quiet the fortifier?
[..]

No, still getting the same warning:

[    2.302776] memcpy: detected field-spanning write (size 104) of single field "stack" at kernel/trace/trace.c:3178 (size 64)

BTW, i'm seeing the same error on x86 with current master when CONFIG_FORTIFY_SOURCE=y and CONFIG_SCHED_TRACER=y:

[    3.089395] Testing tracer wakeup: 
[    3.205602] ------------[ cut here ]------------
[    3.205958] memcpy: detected field-spanning write (size 112) of single field "&entry->caller" at kernel/trace/trace.c:3173 (size 64)
[    3.205958] WARNING: CPU: 1 PID: 0 at kernel/trace/trace.c:3173 __ftrace_trace_stack+0x1d1/0x1e0
[    3.205958] Modules linked in:
[    3.205958] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.5.0-rc1-00012-g77341f6d2110-dirty #50
[    3.205958] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014
[    3.205958] RIP: 0010:__ftrace_trace_stack+0x1d1/0x1e0
[    3.205958] Code: ff ff ff b9 40 00 00 00 4c 89 f6 48 c7 c2 d8 d3 9a 82 48 c7 c7 e8 82 99 82 48 89 44 24 08 c6 05 9d 8c 30 02 01 e8 0f 88 ed ff <0f> 0b 48 8b 44 24 08 e9 f4 fe ff ff 0f 1f 00 90 90 90 90 90 90 90
[    3.205958] RSP: 0000:ffffc90000100ee0 EFLAGS: 00010086
[    3.205958] RAX: 0000000000000000 RBX: ffff8881003db034 RCX: c0000000ffffdfff
[    3.205958] RDX: 0000000000000000 RSI: 00000000ffffdfff RDI: 0000000000000001
[    3.205958] RBP: ffff8881003db03c R08: 0000000000000000 R09: ffffc90000100d88
[    3.205958] R10: 0000000000000003 R11: ffffffff83343008 R12: ffff88810007a100
[    3.205958] R13: 000000000000000e R14: 0000000000000070 R15: 0000000000000070
[    3.205958] FS:  0000000000000000(0000) GS:ffff88817bc40000(0000) knlGS:0000000000000000
[    3.205958] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    3.205958] CR2: 0000000000000000 CR3: 000000000322e000 CR4: 00000000000006e0
[    3.205958] Call Trace:
[    3.205958]  <IRQ>
[    3.205958]  ? __ftrace_trace_stack+0x1d1/0x1e0
[    3.205958]  ? __warn+0x81/0x130
[    3.205958]  ? __ftrace_trace_stack+0x1d1/0x1e0
[    3.205958]  ? report_bug+0x171/0x1a0
[    3.205958]  ? handle_bug+0x3a/0x70
[    3.205958]  ? exc_invalid_op+0x17/0x70
[    3.205958]  ? asm_exc_invalid_op+0x1a/0x20
[    3.205958]  ? __ftrace_trace_stack+0x1d1/0x1e0
[    3.205958]  probe_wakeup+0x28e/0x340
[    3.205958]  ttwu_do_activate.isra.0+0x132/0x190
[    3.205958]  sched_ttwu_pending+0x97/0x110
[    3.205958]  __flush_smp_call_function_queue+0x131/0x400
[    3.205958]  __sysvec_call_function_single+0x2d/0xd0
[    3.205958]  sysvec_call_function_single+0x65/0x80
[    3.205958]  </IRQ>
[    3.205958]  <TASK>
[    3.205958]  asm_sysvec_call_function_single+0x1a/0x20
[    3.205958] RIP: 0010:default_idle+0xf/0x20
[    3.205958] Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 43 5f 31 00 fb f4 <fa> c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90
maintainer
#7 lore raw
Quoted 6 lines · On Wed, 12 Jul 2023 16:06:27 +0200 Sven Schnelle <[email protected]> wrote:
Quoted 3 lines

No, still getting the same warning:

[    2.302776] memcpy: detected field-spanning write (size 104) of single field "stack" at kernel/trace/trace.c:3178 (size 64)  

BTW, i'm seeing the same error on x86 with current master when CONFIG_FORTIFY_SOURCE=y and CONFIG_SCHED_TRACER=y:

As I don't know how the fortifier works, nor what exactly it is checking, do you have any idea on how to quiet it?

This is a false positive, as I described before.

-- Steve

maintainer
#8 lore raw
Quoted 7 lines · On Wed, 12 Jul 2023 10:14:34 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 4 lines · On Wed, 12 Jul 2023 16:06:27 +0200 Sven Schnelle <[email protected]> wrote:
Quoted 3 lines

No, still getting the same warning:

[    2.302776] memcpy: detected field-spanning write (size 104) of single field "stack" at kernel/trace/trace.c:3178 (size 64)    
Quoted 7 lines
Quoted 2 lines

BTW, i'm seeing the same error on x86 with current master when CONFIG_FORTIFY_SOURCE=y and CONFIG_SCHED_TRACER=y:

As I don't know how the fortifier works, nor what exactly it is checking, do you have any idea on how to quiet it?

This is a false positive, as I described before.

Hmm, maybe this would work?

kernel/trace/trace.c +19 -2
index 4529e264cb86..20122eeccf97 100644
@@ -3118,6 +3118,7 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
 	struct ftrace_stack *fstack;
 	struct stack_entry *entry;
 	int stackidx;
+	void *ptr;
 
 	/*
 	 * Add one, for this function and the call to save_stack_trace()
@@ -3161,9 +3162,25 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
 				    trace_ctx);
 	if (!event)
 		goto out;
-	entry = ring_buffer_event_data(event);
+	ptr = ring_buffer_event_data(event);
+	entry = ptr;
+
+	/*
+	 * For backward compatibility reasons, the entry->caller is an
+	 * array of 8 slots to store the stack. This is also exported
+	 * to user space. The amount allocated on the ring buffer actually
+	 * holds enough for the stack specified by nr_entries. This will
+	 * go into the location of entry->caller. Due to string fortifiers
+	 * checking the size of the destination of memcpy() it triggers
+	 * when it detects that size is greater than 8. To hide this from
+	 * the fortifiers, we use "ptr" and pointer arithmetic to assign caller.
+	 *
+	 * The below is really just:
+	 *   memcpy(&entry->caller, fstack->calls, size);
+	 */
+	ptr += offsetof(typeof(*entry), caller);
+	memcpy(ptr, fstack->calls, size);
 
-	memcpy(&entry->caller, fstack->calls, size);
 	entry->size = nr_entries;
 
 	if (!call_filter_check_discard(call, entry, buffer, event))

-- Steve

#9 lore raw

Hi Steven,

Quoted 40 lines · Steven Rostedt
Quoted 4 lines

As I don't know how the fortifier works, nor what exactly it is checking, do you have any idea on how to quiet it?

This is a false positive, as I described before.

Hmm, maybe this would work?

kernel/trace/trace.c +14 -1
index 4529e264cb86..20122eeccf97 100644
@@ -3118,6 +3118,7 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
 	struct ftrace_stack *fstack;
 	struct stack_entry *entry;
 	int stackidx;
+	void *ptr;
 
 	/*
 	 * Add one, for this function and the call to save_stack_trace()
@@ -3161,9 +3162,25 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
 				    trace_ctx);
 	if (!event)
 		goto out;
-	entry = ring_buffer_event_data(event);
+	ptr = ring_buffer_event_data(event);
+	entry = ptr;
+
+	/*
+	 * For backward compatibility reasons, the entry->caller is an
+	 * array of 8 slots to store the stack. This is also exported
+	 * to user space. The amount allocated on the ring buffer actually
+	 * holds enough for the stack specified by nr_entries. This will
+	 * go into the location of entry->caller. Due to string fortifiers
+	 * checking the size of the destination of memcpy() it triggers
+	 * when it detects that size is greater than 8. To hide this from
+	 * the fortifiers, we use "ptr" and pointer arithmetic to assign caller.
+	 *

kernel/trace/trace.c static void __ftrace_trace_stack(struct trace_buffer *buffer, · line 3186

Quoted 10 lines
kernel/trace/trace.c +5 -1
+	 * The below is really just:
+	 *   memcpy(&entry->caller, fstack->calls, size);
+	 */
+	ptr += offsetof(typeof(*entry), caller);
+	memcpy(ptr, fstack->calls, size);
 
-	memcpy(&entry->caller, fstack->calls, size);
 	entry->size = nr_entries;
 
 	if (!call_filter_check_discard(call, entry, buffer, event))

I just sent about the same thing without the nice comment. So yes, this works. :-)

#10 lore raw
In reply to Steven Rostedt

Hi Steven,

Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Wed, 12 Jul 2023 16:06:27 +0200 Sven Schnelle <[email protected]> wrote:
Quoted 3 lines

No, still getting the same warning:

[    2.302776] memcpy: detected field-spanning write (size 104) of single field "stack" at kernel/trace/trace.c:3178 (size 64)  
Quoted 7 lines
Quoted 2 lines

BTW, i'm seeing the same error on x86 with current master when CONFIG_FORTIFY_SOURCE=y and CONFIG_SCHED_TRACER=y:

As I don't know how the fortifier works, nor what exactly it is checking, do you have any idea on how to quiet it?

This is a false positive, as I described before.

The "problem" is that struct stack_entry is

struct stack_entry {
       int size;
       unsigned long caller[8];
};

So, as you explained, the ringbuffer code allocates some space after the struct for additional entries:

struct stack_entry 1;
<additional space for 1>
struct stack_entry 2;
<additional space for 2>
...

But the struct member that is passed to memcpy still has the type information 'caller is an array with 8 members of 8 bytes', so memcpy fortify complains. I'm not sure whether we can blame the compiler or the fortify code here.

One (ugly and whitespace damaged) workaround is:

kernel/trace/trace.c +2 -1
index 35b11f5a9519..31acd8a6b97e 100644
@@ -3170,7 +3170,8 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
                goto out;
        entry = ring_buffer_event_data(event);
 
-       memcpy(&entry->caller, fstack->calls, size);
+       void *p = entry + offsetof(struct stack_entry, caller);
+       memcpy(p, fstack->calls, size);
        entry->size = nr_entries;
 
        if (!call_filter_check_discard(call, entry, buffer, event))

So with that offsetof calculation the compiler doesn't know about the 8 entries * 8 bytes limitation. Adding Kees to the thread, maybe he knows some way.