fs/proc: Crash observed in next_tgid (fs/proc/base.c)
Subsystems (2)
Hi Kees Cook/Luis,
We are observing one kernel crash in next_tgid function through getdents64 path. Call stack is as shown below:
-000|has_group_leader_pid(inline)
-000|next_tgid(
| [X20] ns = 0xFFFFFF87CABB1AC0, | [locdesc] iter = ( | [locdesc] tgid = 424, | [locdesc] task = ?)) | [X21] p = 0xFFFFFFD0FFFFF948 | [X21] task = 0xFFFFFFD0FFFFF948 -001|proc_pid_readdir( | [X20] file = 0xFFFFFFD1AC60FC40, | [X19] ctx = 0xFFFFFF8027363E40) | [X21] ns = 0xFFFFFF87CABB1AC0
-002|proc_root_readdir(
| [X20] file = 0xFFFFFFD1AC60FC40,
| [X19] ctx = 0xFFFFFF8027363E40)
-003|iterate_dir(
| [X19] file = 0xFFFFFFD1AC60FC40, | [X22] ctx = 0xFFFFFF8027363E40) | [X23] inode = 0xFFFFFFD1F20246D0 -004|SYSC_getdents64(inline) -004|sys_getdents64( | ?, | ?, | [X19] count = 4200) | [X19] count = 4200 | [X20] f = ([X20] file = 0xAC60FC43AC60FC40, [X20] flags = 1207898624) | [X0] error = -1720
-005|el0_svc_naked(asm)
-->|exception
-006|NUX:0x78C5AD7D38(asm)
---|end of frame
From this call stack,task: 0xFFFFFFD0FFFFF948, seems to be invalid. As(from ramdumps) it doesn't have any valid fields. And while trying to access the fields of this task struct in has_group_leader_pid, abort is happening.
From the dumps, its not clear why the task struct is coming to be some invalid (Possibly task has already exited). This issue is observed during normal monkey testing for long hours.
Could you please provide some pointers which could help in debugging this issue further.
Thanks,
Jitendra
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
Quoted 44 lines · Jitendra Sharma
Hi Kees Cook/Luis,
We are observing one kernel crash in
next_tgidfunction through getdents64 path. Call stack is as shown below:-000|has_group_leader_pid(inline)
-000|next_tgid(| [X20] ns = 0xFFFFFF87CABB1AC0, | [locdesc] iter = ( | [locdesc] tgid = 424, | [locdesc] task = ?)) | [X21] p = 0xFFFFFFD0FFFFF948 | [X21] task = 0xFFFFFFD0FFFFF948 -001|proc_pid_readdir( | [X20] file = 0xFFFFFFD1AC60FC40, | [X19] ctx = 0xFFFFFF8027363E40) | [X21] ns = 0xFFFFFF87CABB1AC0-002|proc_root_readdir(
| [X20] file =0xFFFFFFD1AC60FC40,
| [X19] ctx =0xFFFFFF8027363E40)
-003|iterate_dir(| [X19] file = 0xFFFFFFD1AC60FC40, | [X22] ctx = 0xFFFFFF8027363E40) | [X23] inode = 0xFFFFFFD1F20246D0 -004|SYSC_getdents64(inline) -004|sys_getdents64( | ?, | ?, | [X19] count = 4200) | [X19] count = 4200 | [X20] f = ([X20] file = 0xAC60FC43AC60FC40, [X20] flags = 1207898624) | [X0] error = -1720-005|el0_svc_naked(asm)
-->|exception
-006|NUX:0x78C5AD7D38(asm)
---|end of frameFrom this call stack,task:
0xFFFFFFD0FFFFF948, seems to be invalid.
As(from ramdumps) it doesn't have any valid fields. And while trying to access the fields of this task struct inhas_group_leader_pid, abort is happening.
Quoted 6 lines
From the dumps, its not clear why the task struct is coming to be some
invalid (Possibly task has already exited). This issue is observed during normal monkey testing for long hours.Could you please provide some pointers which could help in debugging this issue further.
Do you have any hints on how to reproduce this? I assume something is missing proper locking or RCU handling, but I don't see anything obvious in the surrounding code yet...
Signature · 1 line
Kees Cook
Quoted 3 lines · Kees Cook
Do you have any hints on how to reproduce this? I assume something is missing proper locking or RCU handling,
or we simply have an unbalanced put_task_struct() anywhere else ...
Quoted 2 lines
but I don't see anything
obvious in the surrounding code yet...
I too do not see anything wrong in proc_pid_readdir() paths
Oleg.
Quoted 2 lines · Jitendra Sharma
We are observing one kernel crash in
next_tgidfunction through getdents64 path. Call stack is as shown below:
It might help if you specify the exact kernel version you are discussing, as in which tag or commit hash you are running.
Also, what are you doing to trigger the issue?
Regards.
Thanks Marc and Kees for replying.
Answer to your queries:
Kernel version: 4.14.83
Test case: Not some specific test case. Issue reproduced while doing monkey testing for very long hours.
Thanks,
Jitendra
Quoted 12 lines · Marc Gonzalez
Quoted 2 lines · Jitendra Sharma
We are observing one kernel crash in
next_tgidfunction through getdents64 path. Call stack is as shown below:It might help if you specify the exact kernel version you are discussing, as in which tag or commit hash you are running.
Also, what are you doing to trigger the issue?
Regards.
Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
NB: it is preferable to avoid top-posting here.
Quoted 1 line · Jitendra Sharma
Kernel version: 4.14.83
NB2: 4.14.83 is obsolete, as it stands 2255 patches behind the tip of linux-4.14.y (though only 4 patches in fs/proc).
NB3: 4.14 is 108499(!!) patches behind v5.1-rc5 (latest release). It makes sense to test on a recent kernel, in case someone has solved the issue along the way (then the bisecting fun can begin).
Quoted 2 lines
Test case: Not some specific test case. Issue reproduced while doing monkey testing for very long hours.
I'm not sure what "monkey testing" entails...
Any relation to the infinite monkey theorem?
https://en.wikipedia.org/wiki/Infinite_monkey_theorem
Regards.