Skip to contentSkip to conversation

arm64: Fix task tracing

PATCH Superseded Christopher Covington 3 participants 14 messages Apr 3, 2013 – Apr 15, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 arm64: Fix task tracing

For accurate accounting call contextidr_thread_switch before a task is scheduled, rather than after.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +1 -1
index 0337cdb..c2cc249 100644
@@ -311,11 +311,11 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	fpsimd_thread_switch(next);
 	tls_thread_switch(next);
 	hw_breakpoint_thread_switch(next);
+	contextidr_thread_switch(next);
 
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
-	contextidr_thread_switch(next);
 	return last;
 }
 
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by the Linux Foundation.
#1 lore raw

Hi Christopher,

Quoted 14 lines · Christopher Covington

For accurate accounting call contextidr_thread_switch before a task is scheduled, rather than after.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +0 -0
index 0337cdb..c2cc249 100644
@@ -311,11 +311,11 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	fpsimd_thread_switch(next);

arch/arm64/kernel/process.c struct task_struct *__switch_to(struct task_struct *prev, · line 320

Quoted 10 lines
arch/arm64/kernel/process.c +1 -1
 	tls_thread_switch(next);
 	hw_breakpoint_thread_switch(next);
+	contextidr_thread_switch(next);
 
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
-	contextidr_thread_switch(next);
 	return last;
 }

Catalin and I wondered about this and decided to go with the current approach in case a debugger, in response to the contextidr write, decided to go off and mine information about the *new* task using the sp.

If we update the register before we've switched the registers, there's a sizeable window where the debugger will get confused.

Do you have a real use-case that has triggered this patch?

Cheers,

Will

#2 lore raw

Hi Will,

Quoted 24 lines · Will Deacon

Hi Christopher,

Quoted 21 lines · Christopher Covington

For accurate accounting call contextidr_thread_switch before a task is scheduled, rather than after.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +1 -0
index 0337cdb..c2cc249 100644
@@ -311,11 +311,11 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	fpsimd_thread_switch(next);
 	tls_thread_switch(next);
 	hw_breakpoint_thread_switch(next);
+	contextidr_thread_switch(next);
 
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
Quoted 7 lines

arch/arm64/kernel/process.c struct task_struct *__switch_to(struct task_struct *prev, · line 320

Quoted 3 lines
arch/arm64/kernel/process.c +0 -1
-	contextidr_thread_switch(next);
 	return last;
 }

Catalin and I wondered about this and decided to go with the current approach in case a debugger, in response to the contextidr write, decided to go off and mine information about the *new* task using the sp.

The problem with the existing implementation is that it doesn't seem to compensate for how cpu_switch_to changes the stack pointer. Consider the following sequence.

cpu_switch_to(prev=A, next=B)
cpu_switch_to(prev=B, next=C)
cpu_switch_to(prev=C, next=A)

After the third call, using A's stack, next will be B, and its thread ID will be written to CONTEXTIDR. An easy way to see this in a simulator is to just instrument the code with some printk's.

Thanks,
Christopher

Signature · 3 lines
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.
#3 lore raw
Quoted 37 lines · Christopher Covington
Quoted 24 lines · Will Deacon

Hi Christopher,

Quoted 21 lines · Christopher Covington

For accurate accounting call contextidr_thread_switch before a task is scheduled, rather than after.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +1 -0
index 0337cdb..c2cc249 100644
@@ -311,11 +311,11 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	fpsimd_thread_switch(next);
 	tls_thread_switch(next);
 	hw_breakpoint_thread_switch(next);
+	contextidr_thread_switch(next);
 
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
Quoted 7 lines

arch/arm64/kernel/process.c struct task_struct *__switch_to(struct task_struct *prev, · line 320

Quoted 3 lines
arch/arm64/kernel/process.c +0 -1
-	contextidr_thread_switch(next);
 	return last;
 }

Catalin and I wondered about this and decided to go with the current approach in case a debugger, in response to the contextidr write, decided to go off and mine information about the *new* task using the sp.

The problem with the existing implementation is that it doesn't seem to compensate for how cpu_switch_to changes the stack pointer. Consider the following sequence.

Quoted 7 lines

cpu_switch_to(prev=A, next=B)
cpu_switch_to(prev=B, next=C)
cpu_switch_to(prev=C, next=A)

After the third call, using A's stack, next will be B, and its thread ID will be written to CONTEXTIDR. An easy way to see this in a simulator is to just instrument the code with some printk's.

Yes, but moving the call still isn't the right thing to do if we can avoid it. How about making that contextidr_thread_switch take prev instead of next?

Will

#4 lore raw

PATCH v2 1/1 arm64: Fix task tracing

For accurate accounting pass contextidr_thread_switch the prev task pointer, since cpu_switch_to has at that point changed the the stack pointer.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +1 -1
index 0337cdb..a49b25a 100644
@@ -315,7 +315,7 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
-	contextidr_thread_switch(next);
+	contextidr_thread_switch(prev);
 	return last;
 }
 
Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by the Linux Foundation.
#5 lore raw
Quoted 5 lines · Christopher Covington

For accurate accounting pass contextidr_thread_switch the prev task pointer, since cpu_switch_to has at that point changed the the stack pointer.

Thanks Christopher -- I assume that using prev did resolve your issues?

Will

#6 lore raw

Hi Will,

Quoted 8 lines · Will Deacon
Quoted 5 lines · Christopher Covington

For accurate accounting pass contextidr_thread_switch the prev task pointer, since cpu_switch_to has at that point changed the the stack pointer.

Thanks Christopher -- I assume that using prev did resolve your issues?

Yes indeed. We're now able to see in simulation that if a userspace process uses 100% CPU, its thread ID, rather than what was usually some random kthread, gets written out most of the time. I donno if that meets your criteria for a "real" use case, but I hope it's at least sufficient testing of the code for now.

Thanks,
Christopher

Signature · 3 lines
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.
maintainer
#7 lore raw
In reply to Christopher Covington
Quoted 14 lines · Christopher Covington

For accurate accounting pass contextidr_thread_switch the prev task pointer, since cpu_switch_to has at that point changed the the stack pointer.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +0 -0
index 0337cdb..a49b25a 100644

arch/arm64/kernel/process.c struct task_struct *__switch_to(struct task_struct *prev, · line 318

Quoted 5 lines
arch/arm64/kernel/process.c +1 -1
@@ -315,7 +315,7 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
-	contextidr_thread_switch(next);
+	contextidr_thread_switch(prev);

The original code was indeed wrong but using prev isn't any better. For a newly created thread, prev is probably 0 (if it's in a register, cpu_context has been zeroed by copy_thread()) or some random stack value.

So we either use current or move the call before cpu_switch_to() (I would go for the former).

Signature · 1 line
Catalin
#8 lore raw
Quoted 18 lines · Catalin Marinas
Quoted 17 lines · Christopher Covington

For accurate accounting pass contextidr_thread_switch the prev task pointer, since cpu_switch_to has at that point changed the the stack pointer.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +0 -0
index 0337cdb..a49b25a 100644
@@ -315,7 +315,7 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
Quoted 7 lines

arch/arm64/kernel/process.c struct task_struct *__switch_to(struct task_struct *prev, · line 318

Quoted 2 lines
arch/arm64/kernel/process.c +1 -1
-	contextidr_thread_switch(next);
+	contextidr_thread_switch(prev);

The original code was indeed wrong but using prev isn't any better. For a newly created thread, prev is probably 0 (if it's in a register, cpu_context has been zeroed by copy_thread()) or some random stack value.

Really? If prev is NULL in context_switch(...), the scheduler will implode, and I can't see where else switch_to is called from.

Which code path are you thinking of?

Will

maintainer
#9 lore raw
Quoted 27 lines · Will Deacon
Quoted 18 lines · Catalin Marinas
Quoted 17 lines · Christopher Covington

For accurate accounting pass contextidr_thread_switch the prev task pointer, since cpu_switch_to has at that point changed the the stack pointer.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +0 -0
index 0337cdb..a49b25a 100644
@@ -315,7 +315,7 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
Quoted 7 lines

arch/arm64/kernel/process.c struct task_struct *__switch_to(struct task_struct *prev, · line 318

Quoted 2 lines
arch/arm64/kernel/process.c +1 -1
-	contextidr_thread_switch(next);
+	contextidr_thread_switch(prev);

The original code was indeed wrong but using prev isn't any better. For a newly created thread, prev is probably 0 (if it's in a register, cpu_context has been zeroed by copy_thread()) or some random stack value.

Quoted 4 lines

Really? If prev is NULL in context_switch(...), the scheduler will implode, and I can't see where else switch_to is called from.

Which code path are you thinking of?

copy_thread() zeros cpu_context which is used by cpu_switch_to() to load the next saved registers. The switch_to() function sets prev to last as returned by __switch_to(), so this is valid but in __switch_to() we don't have a valid prev (nor next) after cpu_switch_to() for newly created threads.

Signature · 1 line
Catalin
maintainer
#10 lore raw
Quoted 31 lines · Catalin Marinas
Quoted 30 lines · Will Deacon
Quoted 18 lines · Catalin Marinas
Quoted 17 lines · Christopher Covington

For accurate accounting pass contextidr_thread_switch the prev task pointer, since cpu_switch_to has at that point changed the the stack pointer.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +0 -0
index 0337cdb..a49b25a 100644
@@ -315,7 +315,7 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
Quoted 7 lines

arch/arm64/kernel/process.c struct task_struct *__switch_to(struct task_struct *prev, · line 318

Quoted 2 lines
arch/arm64/kernel/process.c +1 -1
-	contextidr_thread_switch(next);
+	contextidr_thread_switch(prev);

The original code was indeed wrong but using prev isn't any better. For a newly created thread, prev is probably 0 (if it's in a register, cpu_context has been zeroed by copy_thread()) or some random stack value.

Really? If prev is NULL in context_switch(...), the scheduler will implode, and I can't see where else switch_to is called from.

Quoted 7 lines
Quoted 1 line

Which code path are you thinking of?

copy_thread() zeros cpu_context which is used by cpu_switch_to() to load the next saved registers. The switch_to() function sets prev to last as returned by __switch_to(), so this is valid but in __switch_to() we don't have a valid prev (nor next) after cpu_switch_to() for newly created threads.

Correction - newly created threads return to ret_from_fork rather than __switch_to(), which means that we miss the first contextidr_thread_switch() call for a new thread. I would vote for Christopher's original patch moving the call before cpu_switch_to(). The alternative is to define finish_arch_switch() and add the call there. If you are primarily tracing user space, it doesn't really matter whether the stack was switched or not when we set the contextidr. For kernel tracking, it could be a problem as we have the next task with the old stack. But the same could be said about the prev task with the new stack.

Signature · 1 line
Catalin
#11 lore raw

Hi Catalin,

Quoted 21 lines · Catalin Marinas
Quoted 20 lines · Catalin Marinas
Quoted 19 lines · Will Deacon
Quoted 18 lines · Catalin Marinas
Quoted 17 lines · Christopher Covington

For accurate accounting pass contextidr_thread_switch the prev task pointer, since cpu_switch_to has at that point changed the the stack pointer.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +0 -0
index 0337cdb..a49b25a 100644
@@ -315,7 +315,7 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

arch/arm64/kernel/process.c struct task_struct *__switch_to(struct task_struct *prev, · line 318

Quoted 2 lines
arch/arm64/kernel/process.c +1 -1
-	contextidr_thread_switch(next);
+	contextidr_thread_switch(prev);

The original code was indeed wrong but using prev isn't any better. For a newly created thread, prev is probably 0 (if it's in a register, cpu_context has been zeroed by copy_thread()) or some random stack value.

<nit>
I have to I disagree with the statement that using prev isn't _any_ better. Even if there are unhandled cases, from my observations, using prev is _measurably_ better. On the other hand, I agree that 100% accuracy is essential. </nit>

Quoted 11 lines
Quoted 10 lines
Quoted 4 lines

Really? If prev is NULL in context_switch(...), the scheduler will implode, and I can't see where else switch_to is called from.

Which code path are you thinking of?

copy_thread() zeros cpu_context which is used by cpu_switch_to() to load the next saved registers. The switch_to() function sets prev to last as returned by __switch_to(), so this is valid but in __switch_to() we don't have a valid prev (nor next) after cpu_switch_to() for newly created threads.

Quoted 10 lines

Correction - newly created threads return to ret_from_fork rather than __switch_to(), which means that we miss the first contextidr_thread_switch() call for a new thread. I would vote for Christopher's original patch moving the call before cpu_switch_to(). The alternative is to define finish_arch_switch() and add the call there. If you are primarily tracing user space, it doesn't really matter whether the stack was switched or not when we set the contextidr. For kernel tracking, it could be a problem as we have the next task with the old stack. But the same could be said about the prev task with the new stack.

I'm fine with using either of my previous patches (or are there still cases where the second one is suspected to be wrong?) or rolling a new one using finish_arch_switch(). Let me know if you all would prefer for me to start on the latter.

Christopher

Signature · 3 lines
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.
maintainer
#12 lore raw
Quoted 30 lines · Christopher Covington
Quoted 21 lines · Catalin Marinas
Quoted 20 lines · Catalin Marinas
Quoted 19 lines · Will Deacon
Quoted 18 lines · Catalin Marinas
Quoted 17 lines · Christopher Covington

For accurate accounting pass contextidr_thread_switch the prev task pointer, since cpu_switch_to has at that point changed the the stack pointer.

---
 arch/arm64/kernel/process.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/process.c +0 -0
index 0337cdb..a49b25a 100644
@@ -315,7 +315,7 @@ struct task_struct *__switch_to(struct task_struct *prev,
 	/* the actual thread switch */
 	last = cpu_switch_to(prev, next);
 
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

arch/arm64/kernel/process.c struct task_struct *__switch_to(struct task_struct *prev, · line 318

Quoted 2 lines
arch/arm64/kernel/process.c +1 -1
-	contextidr_thread_switch(next);
+	contextidr_thread_switch(prev);

The original code was indeed wrong but using prev isn't any better. For a newly created thread, prev is probably 0 (if it's in a register, cpu_context has been zeroed by copy_thread()) or some random stack value.

Quoted 5 lines

<nit>
I have to I disagree with the statement that using prev isn't _any_ better. Even if there are unhandled cases, from my observations, using prev is _measurably_ better. On the other hand, I agree that 100% accuracy is essential. </nit>

It is indeed better but we still miss the task creation (we only start tracing once the task is scheduled out and scheduled back in.

Quoted 22 lines
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines

Really? If prev is NULL in context_switch(...), the scheduler will implode, and I can't see where else switch_to is called from.

Which code path are you thinking of?

copy_thread() zeros cpu_context which is used by cpu_switch_to() to load the next saved registers. The switch_to() function sets prev to last as returned by __switch_to(), so this is valid but in __switch_to() we don't have a valid prev (nor next) after cpu_switch_to() for newly created threads.

Quoted 10 lines

Correction - newly created threads return to ret_from_fork rather than __switch_to(), which means that we miss the first contextidr_thread_switch() call for a new thread. I would vote for Christopher's original patch moving the call before cpu_switch_to(). The alternative is to define finish_arch_switch() and add the call there. If you are primarily tracing user space, it doesn't really matter whether the stack was switched or not when we set the contextidr. For kernel tracking, it could be a problem as we have the next task with the old stack. But the same could be said about the prev task with the new stack.

Quoted 4 lines

I'm fine with using either of my previous patches (or are there still cases where the second one is suspected to be wrong?) or rolling a new one using finish_arch_switch(). Let me know if you all would prefer for me to start on the latter.

The second patch is not wrong but insufficient since it doesn't cover ret_from_fork. Will has a point that debuggers may use the contextidr event to look into the state of the tread which would still have the old stack with your first patch. But at least it is consistent with the arch/arm implementation which uses notifiers.

So I would go with your first patch until we hear otherwise from the debuggers people, in which case we would probably need to fix both arch/arm and arch/arm64.

Signature · 1 line
Catalin
#13 lore raw
In reply to Catalin Marinas
Quoted 13 lines · Catalin Marinas
Quoted 11 lines · Catalin Marinas
Quoted 4 lines · Will Deacon

Really? If prev is NULL in context_switch(...), the scheduler will implode, and I can't see where else switch_to is called from.

Which code path are you thinking of?

copy_thread() zeros cpu_context which is used by cpu_switch_to() to load the next saved registers. The switch_to() function sets prev to last as returned by __switch_to(), so this is valid but in __switch_to() we don't have a valid prev (nor next) after cpu_switch_to() for newly created threads.

Quoted 10 lines

Correction - newly created threads return to ret_from_fork rather than __switch_to(), which means that we miss the first contextidr_thread_switch() call for a new thread. I would vote for Christopher's original patch moving the call before cpu_switch_to(). The alternative is to define finish_arch_switch() and add the call there. If you are primarily tracing user space, it doesn't really matter whether the stack was switched or not when we set the contextidr. For kernel tracking, it could be a problem as we have the next task with the old stack. But the same could be said about the prev task with the new stack.

The sp defines the current task, which is what the debugger will be interested in and will likely try to correlate with the PID reported in the contextidr, so I still maintain that it's important for these to be in sync.

Will