arm64: avoid race condition issue in dump_backtrace

When we dump the backtrace of some specific task, there is a potential race condition due to the task may be running on other cores if SMP enabled. That is because for current implementation, if the task is not the current task, we will get the registers used for unwind from cpu_context saved in thread_info, which is the snapshot before context switch, but if the task is running on other cores, the registers and the content of stack are changed.
This may cause that we get the wrong backtrace or incomplete backtrace or even crash the kernel.
To avoid this case, do not dump the backtrace of the tasks which are running on other cores.
This patch cannot solve the issue completely but can shrink the window of race condition.

Signed-off-by Ji Zhang <[email protected]>

---
 arch/arm64/kernel/traps.c | 3 +++
 1 file changed, 3 insertions(+)

arch/arm64/kernel/traps.c +0 -0

index eb2d151..95749364 100644

Hi Ji Zhang,

Quoted 24 lines · Ji Zhang

When we dump the backtrace of some specific task, there is a potential race condition due to the task may be running on other cores if SMP enabled. That is because for current implementation, if the task is not the current task, we will get the registers used for unwind from cpu_context saved in thread_info, which is the snapshot before context switch, but if the task is running on other cores, the registers and the content of stack are changed.
This may cause that we get the wrong backtrace or incomplete backtrace or even crash the kernel.
To avoid this case, do not dump the backtrace of the tasks which are running on other cores.
This patch cannot solve the issue completely but can shrink the window of race condition.

Signed-off-by Ji Zhang <[email protected]>

---
 arch/arm64/kernel/traps.c | 3 +++
 1 file changed, 3 insertions(+)

arch/arm64/kernel/traps.c +0 -0

index eb2d151..95749364 100644

arch/arm64/kernel/traps.c void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk) · line 116

Quoted 4 lines

arch/arm64/kernel/traps.c +1 -0

@@ -113,6 +113,9 @@ void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
 	if (tsk == current) {
 		frame.fp = (unsigned long)__builtin_frame_address(0);
 		frame.pc = (unsigned long)dump_backtrace;
+	else if (tsk->state == TASK_RUNNING) {

Missing closing brace. Does this build?

arch/arm64/kernel/traps.c void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk) · line 121

Quoted 5 lines

arch/arm64/kernel/traps.c +2 -0

+		pr_notice("Do not dump other running tasks\n");
+		return;
 	} else {
 		/*
 		 * task blocked in __switch_to

baruch

When we dump the backtrace of some specific task, there is a potential race condition due to the task may be running on other cores if SMP enabled. That is because for current implementation, if the task is not the current task, we will get the registers used for unwind from cpu_context saved in thread_info, which is the snapshot before context switch, but if the task is running on other cores, the registers and the content of stack are changed.
This may cause that we get the wrong backtrace or incomplete backtrace or even crash the kernel.

Quoted 9 lines · Ji Zhang

When we dump the backtrace of some specific task, there is a potential race condition due to the task may be running on other cores if SMP enabled. That is because for current implementation, if the task is not the current task, we will get the registers used for unwind from cpu_context saved in thread_info, which is the snapshot before context switch, but if the task is running on other cores, the registers and the content of stack are changed.
This may cause that we get the wrong backtrace or incomplete backtrace or even crash the kernel.

When do we call dump_backtrace() on a running task that is not current?

AFAICT, we don't do that in the arm64-specific callers of dump_backtrace(), and this would have to be some caller of show_stack() in generic code.

We pin the task's stack via try_get_task_stack(), so this cannot be unmapped while we walk it. In unwind_frame() we check that the frame record falls entirely within the task's stack. So AFAICT, we cannot crash the kernel here, though the backtrace may be misleading (and we could potentially get stuck in an infinite loop).

arch/arm64/kernel/traps.c void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk) · line 118

Quoted 13 lines

To avoid this case, do not dump the backtrace of the tasks which are running on other cores.
This patch cannot solve the issue completely but can shrink the window of race condition.

arch/arm64/kernel/traps.c +3 -0

@@ -113,6 +113,9 @@ void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
 	if (tsk == current) {
 		frame.fp = (unsigned long)__builtin_frame_address(0);
 		frame.pc = (unsigned long)dump_backtrace;
+	else if (tsk->state == TASK_RUNNING) {
+		pr_notice("Do not dump other running tasks\n");
+		return;

As you note, if we can race with the task being scheduled, this doesn't help.

Can we rule this out at a higher level?
Thanks,
Mark.

Quoted 11 lines · Mark Rutland

Quoted 9 lines · Ji Zhang

When we dump the backtrace of some specific task, there is a potential race condition due to the task may be running on other cores if SMP enabled. That is because for current implementation, if the task is not the current task, we will get the registers used for unwind from cpu_context saved in thread_info, which is the snapshot before context switch, but if the task is running on other cores, the registers and the content of stack are changed.
This may cause that we get the wrong backtrace or incomplete backtrace or even crash the kernel.

Quoted 4 lines

When do we call dump_backtrace() on a running task that is not current?

AFAICT, we don't do that in the arm64-specific callers of dump_backtrace(), and this would have to be some caller of show_stack() in generic code.

Yes, show_stack() can make caller specify a task and dump its backtrace. For example, SysRq-T (echo t > /proc/sysrq-trigger) will use this to dump the backtrace of specific tasks.

Quoted 5 lines

We pin the task's stack via try_get_task_stack(), so this cannot be unmapped while we walk it. In unwind_frame() we check that the frame record falls entirely within the task's stack. So AFAICT, we cannot crash the kernel here, though the backtrace may be misleading (and we could potentially get stuck in an infinite loop).

You are right, I have checked the code and it seems that the check for fp in unwind_frame() is strong enough to handle the case which stack being changed due to task running. And as you mentioned, if unfortunately fp is point to the address of itself, the unwind will be an infinite loop, but it is a very small probability event, so we can ignore this, is that right?

Quoted 15 lines

arch/arm64/kernel/traps.c void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk) · line 118

Quoted 12 lines

To avoid this case, do not dump the backtrace of the tasks which are running on other cores.
This patch cannot solve the issue completely but can shrink the window of race condition.

arch/arm64/kernel/traps.c +3 -0

@@ -113,6 +113,9 @@ void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
 	if (tsk == current) {
 		frame.fp = (unsigned long)__builtin_frame_address(0);
 		frame.pc = (unsigned long)dump_backtrace;
+	else if (tsk->state == TASK_RUNNING) {
+		pr_notice("Do not dump other running tasks\n");
+		return;

As you note, if we can race with the task being scheduled, this doesn't help.

Quoted 3 lines

Can we rule this out at a higher level?
Thanks,
Mark.

Actually, according to my previous understanding, the low level function should be transparent to callers and should provide the right result and handle some unexpected cases, which means that if the result may be misleading, we should drop it. That is why I bypass all TASK_RUNNING tasks. I am not sure if this understanding is reasonable for this case.

And as you mentioned that rule this out at a higher level, is there any suggestions, handle this in the caller of show_stack()?

Quoted 12 lines · Ji.Zhang

Quoted 11 lines · Mark Rutland

Quoted 9 lines · Ji Zhang

When we dump the backtrace of some specific task, there is a potential race condition due to the task may be running on other cores if SMP enabled. That is because for current implementation, if the task is not the current task, we will get the registers used for unwind from cpu_context saved in thread_info, which is the snapshot before context switch, but if the task is running on other cores, the registers and the content of stack are changed.
This may cause that we get the wrong backtrace or incomplete backtrace or even crash the kernel.

Quoted 7 lines

Quoted 4 lines

When do we call dump_backtrace() on a running task that is not current?

AFAICT, we don't do that in the arm64-specific callers of dump_backtrace(), and this would have to be some caller of show_stack() in generic code.

Yes, show_stack() can make caller specify a task and dump its backtrace. For example, SysRq-T (echo t > /proc/sysrq-trigger) will use this to dump the backtrace of specific tasks.

Ok. I see that this eventually calls show_state_filter(0), where we call sched_show_task() for every task.

Quoted 11 lines

Quoted 5 lines

We pin the task's stack via try_get_task_stack(), so this cannot be unmapped while we walk it. In unwind_frame() we check that the frame record falls entirely within the task's stack. So AFAICT, we cannot crash the kernel here, though the backtrace may be misleading (and we could potentially get stuck in an infinite loop).

You are right, I have checked the code and it seems that the check for fp in unwind_frame() is strong enough to handle the case which stack being changed due to task running. And as you mentioned, if unfortunately fp is point to the address of itself, the unwind will be an infinite loop, but it is a very small probability event, so we can ignore this, is that right?

I think that it would be preferable to try to avoid the inifinite loop case. We could hit that by accident if we're tracing a live task.

It's a little tricky to ensure that we don't loop, since we can have traces that span several stacks, e.g. overflow -> irq -> task, so we need to know where the last frame was, and we need to defnie a strict order for stack nesting.

Quoted 18 lines

Quoted 18 lines

Review void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk) · line 118

Quoted 12 lines

To avoid this case, do not dump the backtrace of the tasks which are running on other cores.
This patch cannot solve the issue completely but can shrink the window of race condition.

(quoted diff) +3 -0

@@ -113,6 +113,9 @@ void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk)
 	if (tsk == current) {
 		frame.fp = (unsigned long)__builtin_frame_address(0);
 		frame.pc = (unsigned long)dump_backtrace;
+	else if (tsk->state == TASK_RUNNING) {
+		pr_notice("Do not dump other running tasks\n");
+		return;

As you note, if we can race with the task being scheduled, this doesn't help.

Can we rule this out at a higher level?
Thanks,
Mark.

Quoted 5 lines

Actually, according to my previous understanding, the low level function should be transparent to callers and should provide the right result and handle some unexpected cases, which means that if the result may be misleading, we should drop it. That is why I bypass all TASK_RUNNING tasks. I am not sure if this understanding is reasonable for this case.

Given that this can change under our feet, I think this only provides a false sense of security and complicates the code.

Quoted 2 lines

And as you mentioned that rule this out at a higher level, is there any suggestions, handle this in the caller of show_stack()?

Unfortunately, it doesn't look like we can do this in general given cases like sysrq-t.

Thanks,
Mark.

Quoted 7 lines · Mark Rutland

I think that it would be preferable to try to avoid the inifinite loop case. We could hit that by accident if we're tracing a live task.

It's a little tricky to ensure that we don't loop, since we can have traces that span several stacks, e.g. overflow -> irq -> task, so we need to know where the last frame was, and we need to defnie a strict order for stack nesting.

Can we consider this through an easier way? According to AArch64 PCS, stack should be full-descending, which means we can add validation on fp by comparing the fp and previous fp, if they are equal means there is an exactly loop, while if current fp is smaller than previous means the uwnind is rollback, which is also unexpected. The only concern is how to handle the unwind from one stack span to another (eg. overflow->irq, or irq->task, etc)
Below diff is a proposal that we check if stack spans, and if yes, a tricky is used to bypass the fp check.

arch/arm64/kernel/traps.c +8 -0

index eb2d151..760ea59 100644
@@ -101,6 +101,7 @@ void dump_backtrace(struct pt_regs *regs, struct
 task_struct *tsk)
 {
        struct stackframe frame;
        int skip;
+       unsigned long fp = 0x0;
 
        pr_debug("%s(regs = %p tsk = %p)\n", __func__, regs, tsk);

@@ -127,6 +128,20 @@ void dump_backtrace(struct pt_regs *regs, struct
 task_struct *tsk)
        skip = !!regs;
        printk("Call trace:\n");
        do {
+               unsigned long stack;
+               if (fp) {
+                       if (in_entry_text(frame.pc)) {
+                               stack = frame.fp - offsetof(struct
 pt_regs, stackframe);
+
+                               if (on_accessible_stack(tsk, stack))
+                                       fp = frame.fp + 0x8; //tricky to

bypass the fp check

On Wed, Mar 28, 2018 at 05:33:32PM +0800, Ji.Zhang wrote:

I'm very much not keen on this.

I think that if we're going to do this, the only sane way to do it is to have unwind_frame() verify the current fp against the previous one, and verify that we have some strict nesting of stacks. Generally, that means we can go:

overflow -> irq -> task

... though I'm not sure what to do about the SDEI stack vs the overflow stack.

Quoted 9 lines · Mark Rutland

On Wed, Mar 28, 2018 at 05:33:32PM +0800, Ji.Zhang wrote:

I'm very much not keen on this.

I think that if we're going to do this, the only sane way to do it is to have unwind_frame() verify the current fp against the previous one, and verify that we have some strict nesting of stacks. Generally, that means we can go:

Quoted 4 lines

overflow -> irq -> task

... though I'm not sure what to do about the SDEI stack vs the overflow stack.

Actually I have had the fp check in unwind_frame(), but since I use the in_entry_text() to determine if stack spans, and I did not want to include traps.h in stacktrace.c, so I move the check out to dump_backtrace.
Anyway, It seems that the key point is how should we verify that there are some nesting of stacks. Since in unwind_frame() we already have the previous fp and current fp, could we assume that if these two fps are NOT belong to the same stack, there should be stack spans (no matter task->irq, or irq->overflow, etc), and we can do the tricky to bypass the fp check.The sample of the prosal just like:

arch/arm64/include/asm/stacktrace.h +0 -0

index 902f9ed..fc2bf4d 100644
@@ -92,4 +92,18 @@ static inline bool on_accessible_stack(struct

task_struct *tsk, unsigned long sp

        return false;

}

+static inline bool on_same_stack(struct task_struct *tsk, unsigned long sp1, unsigned sp2)

arch/arm64/include/asm/stacktrace.h +13 -0

+{
+       if (on_task_stack(tsk, sp1) && on_task_stack(tsk, sp2))
+               return true;
+       if (on_irq_stack(sp1) && on_irq_stack(sp2))
+               return true;
+       if (on_overflow_stack(sp1) && on_overflow_stack(sp2))
+               return true;
+       if (on_sdei_stack(sp1) && on_sdei_stack(sp2))
+               return true;
+
+       return false;
+}
+
 #endif /* __ASM_STACKTRACE_H */

arch/arm64/kernel/stacktrace.c +7 -0

index d5718a0..4907a67 100644
@@ -56,6 +56,13 @@ int notrace unwind_frame(struct task_struct *tsk,
 struct stackframe *frame)
        frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp));
        frame->pc = READ_ONCE_NOCHECK(*(unsigned long *)(fp + 8));
 
+       if (!on_same_stack(fp, frame->fp))
+               fp = frame->fp + 0x8;
+       if (fp <= frame->fp) {
+               pr_notice("fp invalid, stop unwind\n");
+               return -EINVAL;
+       }
+
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
        if (tsk->ret_stack &&

                        (frame->pc == (unsigned long)return_to_handler))

{

Could this work?

Best Regards,
Ji

Quoted 14 lines · Ji.Zhang

Quoted 9 lines · Mark Rutland

On Wed, Mar 28, 2018 at 05:33:32PM +0800, Ji.Zhang wrote:

I'm very much not keen on this.

I think that if we're going to do this, the only sane way to do it is to have unwind_frame() verify the current fp against the previous one, and verify that we have some strict nesting of stacks. Generally, that means we can go:

Quoted 4 lines

overflow -> irq -> task

... though I'm not sure what to do about the SDEI stack vs the overflow stack.

Quoted 10 lines

Actually I have had the fp check in unwind_frame(), but since I use the in_entry_text() to determine if stack spans, and I did not want to include traps.h in stacktrace.c, so I move the check out to dump_backtrace.
Anyway, It seems that the key point is how should we verify that there are some nesting of stacks. Since in unwind_frame() we already have the previous fp and current fp, could we assume that if these two fps are NOT belong to the same stack, there should be stack spans (no matter task->irq, or irq->overflow, etc), and we can do the tricky to bypass the fp check.The sample of the prosal just like:

Unfortuantely, this still allows for loops, like task -> irq -> task, so I think if we're going to try to fix this, we must define a nesting order and check against that.

Thanks,
Mark.

Quoted 53 lines

arch/arm64/include/asm/stacktrace.h +0 -0

index 902f9ed..fc2bf4d 100644
@@ -92,4 +92,18 @@ static inline bool on_accessible_stack(struct

task_struct *tsk, unsigned long sp

        return false;

}

+static inline bool on_same_stack(struct task_struct *tsk, unsigned long sp1, unsigned sp2)

arch/arm64/include/asm/stacktrace.h +13 -0

+{
+       if (on_task_stack(tsk, sp1) && on_task_stack(tsk, sp2))
+               return true;
+       if (on_irq_stack(sp1) && on_irq_stack(sp2))
+               return true;
+       if (on_overflow_stack(sp1) && on_overflow_stack(sp2))
+               return true;
+       if (on_sdei_stack(sp1) && on_sdei_stack(sp2))
+               return true;
+
+       return false;
+}
+
 #endif /* __ASM_STACKTRACE_H */

arch/arm64/kernel/stacktrace.c +7 -0

index d5718a0..4907a67 100644
@@ -56,6 +56,13 @@ int notrace unwind_frame(struct task_struct *tsk,
 struct stackframe *frame)
        frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp));
        frame->pc = READ_ONCE_NOCHECK(*(unsigned long *)(fp + 8));
 
+       if (!on_same_stack(fp, frame->fp))
+               fp = frame->fp + 0x8;
+       if (fp <= frame->fp) {
+               pr_notice("fp invalid, stop unwind\n");
+               return -EINVAL;
+       }
+
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
        if (tsk->ret_stack &&

                        (frame->pc == (unsigned long)return_to_handler))

{

Could this work?

Best Regards,
Ji

Yes, I see where the loop is, I have missed that the loop may cross different stacks.
Define a nesting order and check against is a good idea, and it can resolve the issue exactly, but as you mentioned before, we have no idea how to handle with overflow and sdei stack, and the nesting order is strongly related with the scenario of the stack, which means if someday we add another stack, we should consider the relationship of the new stack with other stacks. From the perspective of your experts, is that suitable for doing this in unwind?

Or could we just find some way easier but not so accurate, eg. Proposal 1:
When we do unwind and detect that the stack spans, record the last fp of previous stack and next time if we get into the same stack, compare it with that last fp, the new fp should still smaller than last fp, or there should be potential loop.
For example, when we unwind from irq to task, we record the last fp in irq stack such as last_irq_fp, and if it unwind task stack back to irq stack, no matter if it is the same irq stack with previous, just let it go and compare the new irq fp with last_irq_fp, although the process may be wrong since from task stack it could not unwind to irq stack, but the whole process will eventually stop.

Proposal 2:
So far we have four types of stack: task, irq, overflow and sdei, could we just assume that the MAX number of stack spanning is just 3 times?(task->irq->overflow->sdei or task->irq->sdei->overflow), if yes, we can just check the number of stack spanning when we detect the stack spans.

Quoted 10 lines · Ji.Zhang

Yes, I see where the loop is, I have missed that the loop may cross different stacks.
Define a nesting order and check against is a good idea, and it can resolve the issue exactly, but as you mentioned before, we have no idea how to handle with overflow and sdei stack, and the nesting order is strongly related with the scenario of the stack, which means if someday we add another stack, we should consider the relationship of the new stack with other stacks. From the perspective of your experts, is that suitable for doing this in unwind?

Quoted 12 lines

Or could we just find some way easier but not so accurate, eg. Proposal 1:
When we do unwind and detect that the stack spans, record the last fp of previous stack and next time if we get into the same stack, compare it with that last fp, the new fp should still smaller than last fp, or there should be potential loop.
For example, when we unwind from irq to task, we record the last fp in irq stack such as last_irq_fp, and if it unwind task stack back to irq stack, no matter if it is the same irq stack with previous, just let it go and compare the new irq fp with last_irq_fp, although the process may be wrong since from task stack it could not unwind to irq stack, but the whole process will eventually stop.

I agree that saving the last fp per-stack could work.

Quoted 6 lines

Proposal 2:
So far we have four types of stack: task, irq, overflow and sdei, could we just assume that the MAX number of stack spanning is just 3 times?(task->irq->overflow->sdei or task->irq->sdei->overflow), if yes, we can just check the number of stack spanning when we detect the stack spans.

I also agree that counting the number of stack transitions will prevent an inifinite loop, even if less accurately than proposal 1.

I don't have a strong preference either way.

Thanks,
Mark.

Quoted 33 lines · Mark Rutland

Quoted 10 lines · Ji.Zhang

Yes, I see where the loop is, I have missed that the loop may cross different stacks.
Define a nesting order and check against is a good idea, and it can resolve the issue exactly, but as you mentioned before, we have no idea how to handle with overflow and sdei stack, and the nesting order is strongly related with the scenario of the stack, which means if someday we add another stack, we should consider the relationship of the new stack with other stacks. From the perspective of your experts, is that suitable for doing this in unwind?

Quoted 12 lines

Or could we just find some way easier but not so accurate, eg. Proposal 1:
When we do unwind and detect that the stack spans, record the last fp of previous stack and next time if we get into the same stack, compare it with that last fp, the new fp should still smaller than last fp, or there should be potential loop.
For example, when we unwind from irq to task, we record the last fp in irq stack such as last_irq_fp, and if it unwind task stack back to irq stack, no matter if it is the same irq stack with previous, just let it go and compare the new irq fp with last_irq_fp, although the process may be wrong since from task stack it could not unwind to irq stack, but the whole process will eventually stop.

I agree that saving the last fp per-stack could work.

Quoted 6 lines

Proposal 2:
So far we have four types of stack: task, irq, overflow and sdei, could we just assume that the MAX number of stack spanning is just 3 times?(task->irq->overflow->sdei or task->irq->sdei->overflow), if yes, we can just check the number of stack spanning when we detect the stack spans.

Quoted 4 lines

I also agree that counting the number of stack transitions will prevent an inifinite loop, even if less accurately than proposal 1.

I don't have a strong preference either way.

Thank you for your comment.
Compared with proposal 1 and 2, I decide to use proposal2 since proposal1 seems a little complicated and it is not as easy as proposal2 when new stack is added.
The sample is as below:

arch/arm64/include/asm/stacktrace.h +0 -0

index 902f9ed..72d1f34 100644
@@ -92,4 +92,22 @@ static inline bool on_accessible_stack(struct

task_struct *tsk, unsigned long sp

        return false;

}

+#define MAX_STACK_SPAN 3

+DECLARE_PER_CPU(int, num_stack_span);

+DEFINE_PER_CPU(int, num_stack_span);

arch/arm64/include/asm/stacktrace.h +4 -0

+
 /*
  * AArch64 PCS assigns the frame pointer to x29.
  *

@@ -56,6 +58,20 @@ int notrace unwind_frame(struct task_struct *tsk,
 struct stackframe *frame)
        frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp));
        frame->pc = READ_ONCE_NOCHECK(*(unsigned long *)(fp + 8));
 
+       if (!on_same_stack(tsk, fp, frame->fp)) {
+               int num = (int)__this_cpu_read(num_stack_span);
+

Quoted 50 lines · Ji.Zhang

Quoted 33 lines · Mark Rutland

Quoted 10 lines · Ji.Zhang

Yes, I see where the loop is, I have missed that the loop may cross different stacks.
Define a nesting order and check against is a good idea, and it can resolve the issue exactly, but as you mentioned before, we have no idea how to handle with overflow and sdei stack, and the nesting order is strongly related with the scenario of the stack, which means if someday we add another stack, we should consider the relationship of the new stack with other stacks. From the perspective of your experts, is that suitable for doing this in unwind?

Quoted 12 lines

Or could we just find some way easier but not so accurate, eg. Proposal 1:
When we do unwind and detect that the stack spans, record the last fp of previous stack and next time if we get into the same stack, compare it with that last fp, the new fp should still smaller than last fp, or there should be potential loop.
For example, when we unwind from irq to task, we record the last fp in irq stack such as last_irq_fp, and if it unwind task stack back to irq stack, no matter if it is the same irq stack with previous, just let it go and compare the new irq fp with last_irq_fp, although the process may be wrong since from task stack it could not unwind to irq stack, but the whole process will eventually stop.

I agree that saving the last fp per-stack could work.

Quoted 6 lines

Proposal 2:
So far we have four types of stack: task, irq, overflow and sdei, could we just assume that the MAX number of stack spanning is just 3 times?(task->irq->overflow->sdei or task->irq->sdei->overflow), if yes, we can just check the number of stack spanning when we detect the stack spans.

Quoted 4 lines

I also agree that counting the number of stack transitions will prevent an inifinite loop, even if less accurately than proposal 1.

I don't have a strong preference either way.

Thank you for your comment.
Compared with proposal 1 and 2, I decide to use proposal2 since proposal1 seems a little complicated and it is not as easy as proposal2 when new stack is added.
The sample is as below:

arch/arm64/include/asm/stacktrace.h +0 -0

index 902f9ed..72d1f34 100644
@@ -92,4 +92,22 @@ static inline bool on_accessible_stack(struct

task_struct *tsk, unsigned long sp

Quoted 4 lines

        return false;

}

+#define MAX_STACK_SPAN 3

Depending on configuration we can have:

* task
* irq
* overflow (optional with VMAP_STACK)
* sdei (optional with ARM_SDE_INTERFACE && VMAP_STACK)

So 3 isn't always correct.

Also, could we please call this something like MAX_NR_STACKS?

Quoted 1 line

+DECLARE_PER_CPU(int, num_stack_span);

I'm pretty sure we can call unwind_frame() in a preemptible context, so this isn't safe.

Put this counter into the struct stackframe, and call it something like nr_stacks;

[...]

Quoted 1 line

+DEFINE_PER_CPU(int, num_stack_span);

As above, this can go.

Quoted 12 lines

arch/arm64/include/asm/stacktrace.h +4 -0

+
 /*
  * AArch64 PCS assigns the frame pointer to x29.
  *

@@ -56,6 +58,20 @@ int notrace unwind_frame(struct task_struct *tsk,
 struct stackframe *frame)
        frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp));
        frame->pc = READ_ONCE_NOCHECK(*(unsigned long *)(fp + 8));
 
+       if (!on_same_stack(tsk, fp, frame->fp)) {
+               int num = (int)__this_cpu_read(num_stack_span);
+

arch/arm64/include/asm/stacktrace.h int notrace unwind_frame(struct task_struct *tsk, · line 74

Quoted 10 lines

arch/arm64/include/asm/stacktrace.h +10 -0

+               if (num >= MAX_STACK_SPAN)
+                       return -EINVAL;
+               num++;
+               __this_cpu_write(num_stack_span, num);
+               fp = frame->fp + 0x8;
+       }
+       if (fp <= frame->fp) {
+               pr_notice("fp invalid, stop unwind\n");
+               return -EINVAL;
+       }

I think this can be simplified to something like:

	bool same_stack;

	same_stack = on_same_stack(tsk, fp, frame->fp);

	if (fp <= frame->fp && same_stack)
		return -EINVAL;
	if (!same_stack && ++frame->nr_stacks > MAX_NR_STACKS)
		return -EINVAL;

... assuming we add nr_stacks to struct stackframe.

Quoted 115 lines · Mark Rutland

Quoted 50 lines · Ji.Zhang

Quoted 33 lines · Mark Rutland

Quoted 10 lines · Ji.Zhang

Yes, I see where the loop is, I have missed that the loop may cross different stacks.
Define a nesting order and check against is a good idea, and it can resolve the issue exactly, but as you mentioned before, we have no idea how to handle with overflow and sdei stack, and the nesting order is strongly related with the scenario of the stack, which means if someday we add another stack, we should consider the relationship of the new stack with other stacks. From the perspective of your experts, is that suitable for doing this in unwind?

Quoted 12 lines

Or could we just find some way easier but not so accurate, eg. Proposal 1:
When we do unwind and detect that the stack spans, record the last fp of previous stack and next time if we get into the same stack, compare it with that last fp, the new fp should still smaller than last fp, or there should be potential loop.
For example, when we unwind from irq to task, we record the last fp in irq stack such as last_irq_fp, and if it unwind task stack back to irq stack, no matter if it is the same irq stack with previous, just let it go and compare the new irq fp with last_irq_fp, although the process may be wrong since from task stack it could not unwind to irq stack, but the whole process will eventually stop.

I agree that saving the last fp per-stack could work.

Quoted 6 lines

Proposal 2:
So far we have four types of stack: task, irq, overflow and sdei, could we just assume that the MAX number of stack spanning is just 3 times?(task->irq->overflow->sdei or task->irq->sdei->overflow), if yes, we can just check the number of stack spanning when we detect the stack spans.

Quoted 4 lines

I also agree that counting the number of stack transitions will prevent an inifinite loop, even if less accurately than proposal 1.

I don't have a strong preference either way.

Thank you for your comment.
Compared with proposal 1 and 2, I decide to use proposal2 since proposal1 seems a little complicated and it is not as easy as proposal2 when new stack is added.
The sample is as below:

arch/arm64/include/asm/stacktrace.h +0 -0

index 902f9ed..72d1f34 100644
@@ -92,4 +92,22 @@ static inline bool on_accessible_stack(struct

task_struct *tsk, unsigned long sp

Quoted 4 lines

        return false;

}

+#define MAX_STACK_SPAN 3

Depending on configuration we can have:

* task
* irq
* overflow (optional with VMAP_STACK)
* sdei (optional with ARM_SDE_INTERFACE && VMAP_STACK)

So 3 isn't always correct.

Also, could we please call this something like MAX_NR_STACKS?

Quoted 1 line

+DECLARE_PER_CPU(int, num_stack_span);

I'm pretty sure we can call unwind_frame() in a preemptible context, so this isn't safe.

Put this counter into the struct stackframe, and call it something like nr_stacks;

[...]

Quoted 1 line

+DEFINE_PER_CPU(int, num_stack_span);

As above, this can go.

Quoted 12 lines

arch/arm64/include/asm/stacktrace.h +4 -0

+
 /*
  * AArch64 PCS assigns the frame pointer to x29.
  *

@@ -56,6 +58,20 @@ int notrace unwind_frame(struct task_struct *tsk,
 struct stackframe *frame)
        frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp));
        frame->pc = READ_ONCE_NOCHECK(*(unsigned long *)(fp + 8));
 
+       if (!on_same_stack(tsk, fp, frame->fp)) {
+               int num = (int)__this_cpu_read(num_stack_span);
+

arch/arm64/include/asm/stacktrace.h int notrace unwind_frame(struct task_struct *tsk, · line 74

Quoted 10 lines

arch/arm64/include/asm/stacktrace.h +10 -0

+               if (num >= MAX_STACK_SPAN)
+                       return -EINVAL;
+               num++;
+               __this_cpu_write(num_stack_span, num);
+               fp = frame->fp + 0x8;
+       }
+       if (fp <= frame->fp) {
+               pr_notice("fp invalid, stop unwind\n");
+               return -EINVAL;
+       }

I think this can be simplified to something like:

	bool same_stack;

	same_stack = on_same_stack(tsk, fp, frame->fp);

	if (fp <= frame->fp && same_stack)
		return -EINVAL;
	if (!same_stack && ++frame->nr_stacks > MAX_NR_STACKS)
		return -EINVAL;

Quoted 1 line

... assuming we add nr_stacks to struct stackframe.

Thank you very much for your advice, they are very valuable. According to your suggestion, the modified code as follows. I did a little change that define MAX_NR_STACKS as the number of stacks, instead of the number of stack spans.

arch/arm64/include/asm/stacktrace.h +9 -0

index 902f9ed..f235b86 100644
@@ -24,9 +24,18 @@
 #include <asm/ptrace.h>
 #include <asm/sdei.h>
 
+#ifndef CONFIG_VMAP_STACK
+#define MAX_NR_STACKS  2
+#elif !defined(CONFIG_ARM_SDE_INTERFACE)
+#define MAX_NR_STACKS  3
+#else
+#define MAX_NR_STACKS  4
+#endif
+
 struct stackframe {
        unsigned long fp;
        unsigned long pc;
+       int nr_stacks;
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
        int graph;
 #endif

@@ -92,4 +101,20 @@ static inline bool on_accessible_stack(struct

task_struct *tsk, unsigned long sp

        return false;

}

arch/arm64/include/asm/stacktrace.h +16 -0

+
+static inline bool on_same_stack(struct task_struct *tsk,
+                               unsigned long sp1, unsigned long sp2)
+{
+       if (on_task_stack(tsk, sp1) && on_task_stack(tsk, sp2))
+               return true;
+       if (on_irq_stack(sp1) && on_irq_stack(sp2))
+               return true;
+       if (on_overflow_stack(sp1) && on_overflow_stack(sp2))
+               return true;
+       if (on_sdei_stack(sp1) && on_sdei_stack(sp2))
+               return true;
+
+       return false;
+}
+
 #endif /* __ASM_STACKTRACE_H */

arch/arm64/kernel/stacktrace.c +2 -0

index d5718a0..a09e247 100644
@@ -27,6 +27,7 @@
 #include <asm/stack_pointer.h>
 #include <asm/stacktrace.h>
 
+
 /*
  * AArch64 PCS assigns the frame pointer to x29.
  *

@@ -43,6 +44,7 @@
 int notrace unwind_frame(struct task_struct *tsk, struct stackframe
 *frame)
 {
        unsigned long fp = frame->fp;
+       bool same_stack;
 
        if (fp & 0xf)

                return -EINVAL;

arch/arm64/kernel/stacktrace.c +7 -0

@@ -56,6 +58,13 @@ int notrace unwind_frame(struct task_struct *tsk,
 struct stackframe *frame)
        frame->fp = READ_ONCE_NOCHECK(*(unsigned long *)(fp));
        frame->pc = READ_ONCE_NOCHECK(*(unsigned long *)(fp + 8));
 
+       same_stack = on_same_stack(tsk, fp, frame->fp);
+
+       if (fp <= frame->fp && same_stack)
+               return -EINVAL;
+       if (!same_stack && ++frame->nr_stacks > MAX_NR_STACKS)
+               return -EINVAL;
+
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
        if (tsk->ret_stack &&

                        (frame->pc == (unsigned long)return_to_handler))

{

arch/arm64/kernel/traps.c +1 -0

index eb2d151..3b1c472 100644
@@ -120,6 +120,7 @@ void dump_backtrace(struct pt_regs *regs, struct
 task_struct *tsk)
                frame.fp = thread_saved_fp(tsk);
                frame.pc = thread_saved_pc(tsk);
        }
+       frame.nr_stacks = 1;
 #ifdef CONFIG_FUNCTION_GRAPH_TRACER
        frame.graph = tsk->curr_ret_stack;

#endif