Skip to contentSkip to conversation

fs/proc: Crash observed in next_tgid (fs/proc/base.c)

DISCUSSION Jitendra Sharma 4 participants 6 messages Apr 15, 2019 – Apr 18, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi Kees Cook/Luis,

We are observing one kernel crash in next_tgid function through getdents64 path. Call stack is as shown below:

-000|has_group_leader_pid(inline)
-000|next_tgid(

| [X20] ns = 0xFFFFFF87CABB1AC0,
| [locdesc] iter = (
| [locdesc] tgid = 424,
| [locdesc] task = ?))
| [X21] p = 0xFFFFFFD0FFFFF948
| [X21] task = 0xFFFFFFD0FFFFF948
-001|proc_pid_readdir(
| [X20] file = 0xFFFFFFD1AC60FC40,
| [X19] ctx = 0xFFFFFF8027363E40)
| [X21] ns = 0xFFFFFF87CABB1AC0

-002|proc_root_readdir(
| [X20] file = 0xFFFFFFD1AC60FC40,
| [X19] ctx = 0xFFFFFF8027363E40)
-003|iterate_dir(

| [X19] file = 0xFFFFFFD1AC60FC40,
| [X22] ctx = 0xFFFFFF8027363E40)
| [X23] inode = 0xFFFFFFD1F20246D0
-004|SYSC_getdents64(inline)
-004|sys_getdents64(
| ?,
| ?,
| [X19] count = 4200)
| [X19] count = 4200
| [X20] f = ([X20] file = 0xAC60FC43AC60FC40, [X20] flags = 1207898624)
| [X0] error = -1720

-005|el0_svc_naked(asm)
-->|exception
-006|NUX:0x78C5AD7D38(asm)
---|end of frame

From this call stack,task: 0xFFFFFFD0FFFFF948, seems to be invalid. As(from ramdumps) it doesn't have any valid fields. And while trying to access the fields of this task struct in has_group_leader_pid, abort is happening.

From the dumps, its not clear why the task struct is coming to be some invalid (Possibly task has already exited). This issue is observed during normal monkey testing for long hours.

Could you please provide some pointers which could help in debugging this issue further.

Thanks,

Jitendra

Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
#1 lore raw
Quoted 44 lines · Jitendra Sharma

Hi Kees Cook/Luis,

We are observing one kernel crash in next_tgid function through getdents64 path. Call stack is as shown below:

-000|has_group_leader_pid(inline)
-000|next_tgid(

| [X20] ns = 0xFFFFFF87CABB1AC0,
| [locdesc] iter = (
| [locdesc] tgid = 424,
| [locdesc] task = ?))
| [X21] p = 0xFFFFFFD0FFFFF948
| [X21] task = 0xFFFFFFD0FFFFF948
-001|proc_pid_readdir(
| [X20] file = 0xFFFFFFD1AC60FC40,
| [X19] ctx = 0xFFFFFF8027363E40)
| [X21] ns = 0xFFFFFF87CABB1AC0

-002|proc_root_readdir(
| [X20] file = 0xFFFFFFD1AC60FC40,
| [X19] ctx = 0xFFFFFF8027363E40)
-003|iterate_dir(

| [X19] file = 0xFFFFFFD1AC60FC40,
| [X22] ctx = 0xFFFFFF8027363E40)
| [X23] inode = 0xFFFFFFD1F20246D0
-004|SYSC_getdents64(inline)
-004|sys_getdents64(
| ?,
| ?,
| [X19] count = 4200)
| [X19] count = 4200
| [X20] f = ([X20] file = 0xAC60FC43AC60FC40, [X20] flags = 1207898624)
| [X0] error = -1720

-005|el0_svc_naked(asm)
-->|exception
-006|NUX:0x78C5AD7D38(asm)
---|end of frame

From this call stack,task: 0xFFFFFFD0FFFFF948, seems to be invalid.
As(from ramdumps) it doesn't have any valid fields. And while trying to access the fields of this task struct in has_group_leader_pid, abort is happening.

Quoted 6 lines

From the dumps, its not clear why the task struct is coming to be some
invalid (Possibly task has already exited). This issue is observed during normal monkey testing for long hours.

Could you please provide some pointers which could help in debugging this issue further.

Do you have any hints on how to reproduce this? I assume something is missing proper locking or RCU handling, but I don't see anything obvious in the surrounding code yet...

Signature · 1 line
Kees Cook
#2 lore raw
Quoted 3 lines · Kees Cook

Do you have any hints on how to reproduce this? I assume something is missing proper locking or RCU handling,

or we simply have an unbalanced put_task_struct() anywhere else ...

Quoted 2 lines

but I don't see anything
obvious in the surrounding code yet...

I too do not see anything wrong in proc_pid_readdir() paths

Oleg.

#3 lore raw
In reply to Jitendra Sharma
Quoted 2 lines · Jitendra Sharma

We are observing one kernel crash in next_tgid function through getdents64 path. Call stack is as shown below:

It might help if you specify the exact kernel version you are discussing, as in which tag or commit hash you are running.

Also, what are you doing to trigger the issue?

Regards.

#4 lore raw

Thanks Marc and Kees for replying.
Answer to your queries:
Kernel version: 4.14.83
Test case: Not some specific test case. Issue reproduced while doing monkey testing for very long hours.

Thanks,
Jitendra

Quoted 12 lines · Marc Gonzalez
Quoted 2 lines · Jitendra Sharma

We are observing one kernel crash in next_tgid function through getdents64 path. Call stack is as shown below:

It might help if you specify the exact kernel version you are discussing, as in which tag or commit hash you are running.

Also, what are you doing to trigger the issue?

Regards.

Signature · 2 lines
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a member
of Code Aurora Forum, hosted by The Linux Foundation
#5 lore raw

NB: it is preferable to avoid top-posting here.

Quoted 1 line · Jitendra Sharma

Kernel version: 4.14.83

NB2: 4.14.83 is obsolete, as it stands 2255 patches behind the tip of linux-4.14.y (though only 4 patches in fs/proc).

NB3: 4.14 is 108499(!!) patches behind v5.1-rc5 (latest release). It makes sense to test on a recent kernel, in case someone has solved the issue along the way (then the bisecting fun can begin).

Quoted 2 lines

Test case: Not some specific test case. Issue reproduced while doing monkey testing for very long hours.

I'm not sure what "monkey testing" entails...

Any relation to the infinite monkey theorem?
https://en.wikipedia.org/wiki/Infinite_monkey_theorem

Regards.