Free after use in fw_pm_notify()->kill_requests_without_uevent() due pending_fw_head
Hi All,
Device has crashed due to memory access after free while pending_fw_head list accessed. Kernel 4.4 stable version is used to reproduce this use after free.
------------------------------------------------------------------------------------------
[ 9031.178428] Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b6b6b [ 9031.178508] pgd = ffffffc0de9d2000 [ 9031.185888] [6b6b6b6b6b6b6b6b] *pgd=0000000000000000, *pud=0000000000000000 [ 9031.253045] ------------[ cut here ]------------ [ 9031.253100] Kernel BUG at ffffff800864c0a0 [verbose debug info unavailable] [ 9031.256860] Internal error: Oops - BUG: 96000004 [#1] PREEMPT SMP [ 9031.263539] Modules linked in: [ 9031.272708] CPU: 6 PID: 1373 Comm: system_server Tainted: G W L 4.4.16+ #1 [ 9031.280648] task: ffffffc0d1a1d700 ti: ffffffc0d1a2c000 task.ti: ffffffc0d1a2c000 [ 9031.287776] PC is at fw_pm_notify+0x84/0x19c [ 9031.295215] LR is at fw_pm_notify+0x60/0x19c [ 9031.511559] [] fw_pm_notify+0x84/0x19c [ 9031.519355] [] notifier_call_chain+0x58/0x8c [ 9031.524739] [] __blocking_notifier_call_chain+0x54/0x70 [ 9031.530387] [] blocking_notifier_call_chain+0x38/0x44 [ 9031.537243] [] pm_notifier_call_chain+0x28/0x48 [ 9031.543662] [] pm_suspend+0x278/0x674 [ 9031.549906] [] state_store+0x58/0x90 [ 9031.554942] [] kobj_attr_store+0x18/0x28 [ 9031.560154] [] sysfs_kf_write+0x5c/0x68 [ 9031.565620] [] kernfs_fop_write+0x114/0x16c [ 9031.571092] [] __vfs_write+0x48/0xf0 [ 9031.576816] [] vfs_write+0xb8/0x150 [ 9031.581848] [] SyS_write+0x58/0x94 [ 9031.586973] [] el0_svc_naked+0x24/0x28
-----------------------------------------------------------------------------------------------
Kernel panic is observed during device suspend/resume path in the kill_requests_without_uevent() called from fw_pm_notify().
when pending_list of a firmware_buf is accessed 0x6b(free pattern) pattern observed. Based on this firmware_buf is freed even if firmware_buf is part of pending_fw_head list.
{code snippet}
1147 static void kill_requests_without_uevent(void) 1148{
1149 struct firmware_buf *buf;
1150 struct firmware_buf *next;
1151
1152 mutex_lock(&fw_lock);
1153 list_for_each_entry_safe(buf, next, &pending_fw_head, pending_list) {
1154 if (!buf->need_uevent)
1155 __fw_load_abort(buf);
1156 }
1157 mutex_unlock(&fw_lock);
1158}
1159#endif
{code}
-Thanks, Prasad
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, Linux Foundation Collaborative Project
Greg KH
maintainerQuoted 37 lines · Sodagudi Prasad
Hi All,
Device has crashed due to memory access after free while
pending_fw_headlist accessed. Kernel 4.4 stable version is used to reproduce this use after free.
------------------------------------------------------------------------------------------[ 9031.178428] Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b6b6b [ 9031.178508] pgd = ffffffc0de9d2000 [ 9031.185888] [6b6b6b6b6b6b6b6b] *pgd=0000000000000000, *pud=0000000000000000 [ 9031.253045] ------------[ cut here ]------------ [ 9031.253100] Kernel BUG at ffffff800864c0a0 [verbose debug info unavailable] [ 9031.256860] Internal error: Oops - BUG: 96000004 [#1] PREEMPT SMP [ 9031.263539] Modules linked in: [ 9031.272708] CPU: 6 PID: 1373 Comm: system_server Tainted: G W L 4.4.16+ #1 [ 9031.280648] task: ffffffc0d1a1d700 ti: ffffffc0d1a2c000 task.ti: ffffffc0d1a2c000 [ 9031.287776] PC is at fw_pm_notify+0x84/0x19c [ 9031.295215] LR is at fw_pm_notify+0x60/0x19c [ 9031.511559] [] fw_pm_notify+0x84/0x19c [ 9031.519355] [] notifier_call_chain+0x58/0x8c [ 9031.524739] [] __blocking_notifier_call_chain+0x54/0x70 [ 9031.530387] [] blocking_notifier_call_chain+0x38/0x44 [ 9031.537243] [] pm_notifier_call_chain+0x28/0x48 [ 9031.543662] [] pm_suspend+0x278/0x674 [ 9031.549906] [] state_store+0x58/0x90 [ 9031.554942] [] kobj_attr_store+0x18/0x28 [ 9031.560154] [] sysfs_kf_write+0x5c/0x68 [ 9031.565620] [] kernfs_fop_write+0x114/0x16c [ 9031.571092] [] __vfs_write+0x48/0xf0 [ 9031.576816] [] vfs_write+0xb8/0x150 [ 9031.581848] [] SyS_write+0x58/0x94 [ 9031.586973] [] el0_svc_naked+0x24/0x28
Quoted 8 lines
-----------------------------------------------------------------------------------------------
Kernel panic is observed during device suspend/resume path in the
kill_requests_without_uevent()called fromfw_pm_notify(). whenpending_listof afirmware_bufis accessed 0x6b(free pattern) pattern observed. Based on thisfirmware_bufis freed even iffirmware_bufis part ofpending_fw_headlist.
What are you doing in userspace to trigger this problem? What kernel driver is this happening with?
And 4.4.16 is pretty old, can you try 4.9?
thanks,
greg k-h
Quoted 47 lines · Greg KH
Quoted 37 lines · Sodagudi Prasad
Hi All,
Device has crashed due to memory access after free while
pending_fw_headlist accessed. Kernel 4.4 stable version is used to reproduce this use after free.
------------------------------------------------------------------------------------------[ 9031.178428] Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b6b6b [ 9031.178508] pgd = ffffffc0de9d2000 [ 9031.185888] [6b6b6b6b6b6b6b6b] *pgd=0000000000000000, *pud=0000000000000000 [ 9031.253045] ------------[ cut here ]------------ [ 9031.253100] Kernel BUG at ffffff800864c0a0 [verbose debug info unavailable] [ 9031.256860] Internal error: Oops - BUG: 96000004 [#1] PREEMPT SMP [ 9031.263539] Modules linked in: [ 9031.272708] CPU: 6 PID: 1373 Comm: system_server Tainted: G W L 4.4.16+ #1 [ 9031.280648] task: ffffffc0d1a1d700 ti: ffffffc0d1a2c000 task.ti: ffffffc0d1a2c000 [ 9031.287776] PC is at fw_pm_notify+0x84/0x19c [ 9031.295215] LR is at fw_pm_notify+0x60/0x19c [ 9031.511559] [] fw_pm_notify+0x84/0x19c [ 9031.519355] [] notifier_call_chain+0x58/0x8c [ 9031.524739] [] __blocking_notifier_call_chain+0x54/0x70 [ 9031.530387] [] blocking_notifier_call_chain+0x38/0x44 [ 9031.537243] [] pm_notifier_call_chain+0x28/0x48 [ 9031.543662] [] pm_suspend+0x278/0x674 [ 9031.549906] [] state_store+0x58/0x90 [ 9031.554942] [] kobj_attr_store+0x18/0x28 [ 9031.560154] [] sysfs_kf_write+0x5c/0x68 [ 9031.565620] [] kernfs_fop_write+0x114/0x16c [ 9031.571092] [] __vfs_write+0x48/0xf0 [ 9031.576816] [] vfs_write+0xb8/0x150 [ 9031.581848] [] SyS_write+0x58/0x94 [ 9031.586973] [] el0_svc_naked+0x24/0x28Quoted 8 lines
-----------------------------------------------------------------------------------------------
Kernel panic is observed during device suspend/resume path in the
kill_requests_without_uevent()called fromfw_pm_notify(). whenpending_listof afirmware_bufis accessed 0x6b(free pattern) pattern observed. Based on thisfirmware_bufis freed even iffirmware_bufis part ofpending_fw_headlist.
Quoted 2 lines
What are you doing in userspace to trigger this problem? What kernel driver is this happening with?
Device continuous suspend and resume is happening here. I think, echo mem > /sys/power/state issued here. It is not clear what driver involved here, because after firmware_buf is freed all memory gets filled with 0x6b pattern.
Quoted 2 lines
And 4.4.16 is pretty old, can you try 4.9?
We don't have system which runs on new kernels. Looking for possible reasons/path, how firmware_buf can get freed when that in pending_fw_head list.
Quoted 4 lines
thanks,
greg k-h
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, Linux Foundation Collaborative Project
Quoted 54 lines · Sodagudi Prasad
Quoted 53 lines · Greg KH
Quoted 51 lines · Sodagudi Prasad
Hi All,
Device has crashed due to memory access after free while
pending_fw_head
list accessed. Kernel 4.4 stable version is used to reproduce this use after
free.
------------------------------------------------------------------------------------------[ 9031.178428] Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b6b6b [ 9031.178508] pgd = ffffffc0de9d2000 [ 9031.185888] [6b6b6b6b6b6b6b6b] *pgd=0000000000000000, *pud=0000000000000000 [ 9031.253045] ------------[ cut here ]------------ [ 9031.253100] Kernel BUG at ffffff800864c0a0 [verbose debug info unavailable] [ 9031.256860] Internal error: Oops - BUG: 96000004 [#1] PREEMPT SMP [ 9031.263539] Modules linked in: [ 9031.272708] CPU: 6 PID: 1373 Comm: system_server Tainted: G W L 4.4.16+ #1 [ 9031.280648] task: ffffffc0d1a1d700 ti: ffffffc0d1a2c000 task.ti: ffffffc0d1a2c000 [ 9031.287776] PC is at fw_pm_notify+0x84/0x19c [ 9031.295215] LR is at fw_pm_notify+0x60/0x19c [ 9031.511559] [] fw_pm_notify+0x84/0x19c [ 9031.519355] [] notifier_call_chain+0x58/0x8c [ 9031.524739] [] __blocking_notifier_call_chain+0x54/0x70 [ 9031.530387] [] blocking_notifier_call_chain+0x38/0x44 [ 9031.537243] [] pm_notifier_call_chain+0x28/0x48 [ 9031.543662] [] pm_suspend+0x278/0x674 [ 9031.549906] [] state_store+0x58/0x90 [ 9031.554942] [] kobj_attr_store+0x18/0x28 [ 9031.560154] [] sysfs_kf_write+0x5c/0x68 [ 9031.565620] [] kernfs_fop_write+0x114/0x16c [ 9031.571092] [] __vfs_write+0x48/0xf0 [ 9031.576816] [] vfs_write+0xb8/0x150 [ 9031.581848] [] SyS_write+0x58/0x94 [ 9031.586973] [] el0_svc_naked+0x24/0x28-----------------------------------------------------------------------------------------------
Kernel panic is observed during device suspend/resume path in the
kill_requests_without_uevent()called fromfw_pm_notify(). whenpending_listof afirmware_bufis accessed 0x6b(free pattern) pattern
observed. Based on thisfirmware_bufis freed even iffirmware_bufis part
ofpending_fw_headlist.
Quoted 6 lines
Quoted 2 lines
What are you doing in userspace to trigger this problem? What kernel driver is this happening with?
Device continuous suspend and resume is happening here. I think, echo mem >
/sys/power/stateissued here.
It is not clear what driver involved here, because afterfirmware_bufis freed all memory gets filled with0x6bpattern.
It is important to find what API call was used too.
The only thing I can think of is what follows:
drivers/base/firmware_class.c +1 -1
@@ -1005,7 +1005,6 @@ static int _request_firmware_load(struct firmware_priv *fw_priv,mutex_lock (&fw_lock);list_add (&buf->pending_list, &pending_fw_head); -mutex_unlock (&fw_lock);if (opt_flags &FW_OPT_UEVENT ) { buf->need_uevent = true;
@@ -1015,6 +1014,7 @@ static int _request_firmware_load(struct firmware_priv *fw_priv, }else { timeout =MAX_JIFFY_OFFSET ; } +mutex_unlock (&fw_lock); retval =fw_state_wait_timeout (&buf->fw_st, timeout);if (retval <0 ) {
The reason is that kill_requests_without_uevent() checks for the buf->need_uevent but we modify the buf->need_uevent = true; above after we unlock. However, I've tried creating a test case against this and ran a loop against tools/testing/selftests/firmware/fw_fallback.sh first will all test and then with only load_fw_custom test and a timeout of 1 second while I loop suspend/resume and I cannot reproduce. I tried to aggregate the situation by sprinkling ssleep(10) after the lock (without the above patch) and suspending after but no dice. This should give you an idea of what I mean by crafting a hack to try to force the situation if the above really is the fix.
So -- try the above patch, if that does not fix it, then please try to see if you can create a test case or get crafty and try to force the situation around areas you suspect might cause it, use qemu to try to force this issue and use the test driver.
Then on a loop run the test and also do: systemctl suspend0
If using qemu you can force resume using:
echo system_wakeup | socat - /dev/pts/7,raw,echo=0,crnl
where /dev/pts/7 here is my control monitor interface.
Also add to your /etc/default/grub (I know, you are probalby on Android right... so whatever you can use to modify the kernel command line) if dynamic debug is enabled and use:
dyndbg=\"file drivers/base/dd.c +p; file drivers/base/firmware_class.c +p;\"
This should hopefully hep narrow down that driver is doing what here.
Quoted 4 lines
Quoted 1 line
And 4.4.16 is pretty old, can you try 4.9?
We don't have system which runs on new kernels. Looking for possible reasons/path, how
firmware_bufcan get freed when that inpending_fw_headlist.
The above is the only thing I can think of. The best is for you to try to reproduce using the test driver and building a careful test case that tries to explode things. The test driver is lib/test_firmware.c and the script is tools/testing/selftests/firmware/fw_fallback.sh.
Please note I did test using CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y.
For those on the CC list: please note I was busy and I will review your patches or questions hopefully this week. Also note that due to historic issues which are not well tested like the fallback interface -- I am now asking folks to draw up test cases for their new code. You can however wait until I review your code.
Luis
Quoted 55 lines · Luis R. Rodriguez
Quoted 54 lines · Sodagudi Prasad
Quoted 53 lines · Greg KH
Quoted 51 lines · Sodagudi Prasad
Hi All,
Device has crashed due to memory access after free while
pending_fw_head
list accessed. Kernel 4.4 stable version is used to reproduce this use after
free.
------------------------------------------------------------------------------------------[ 9031.178428] Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b6b6b [ 9031.178508] pgd = ffffffc0de9d2000 [ 9031.185888] [6b6b6b6b6b6b6b6b] *pgd=0000000000000000, *pud=0000000000000000 [ 9031.253045] ------------[ cut here ]------------ [ 9031.253100] Kernel BUG at ffffff800864c0a0 [verbose debug info unavailable] [ 9031.256860] Internal error: Oops - BUG: 96000004 [#1] PREEMPT SMP [ 9031.263539] Modules linked in: [ 9031.272708] CPU: 6 PID: 1373 Comm: system_server Tainted: G W L 4.4.16+ #1 [ 9031.280648] task: ffffffc0d1a1d700 ti: ffffffc0d1a2c000 task.ti: ffffffc0d1a2c000 [ 9031.287776] PC is at fw_pm_notify+0x84/0x19c [ 9031.295215] LR is at fw_pm_notify+0x60/0x19c [ 9031.511559] [] fw_pm_notify+0x84/0x19c [ 9031.519355] [] notifier_call_chain+0x58/0x8c [ 9031.524739] [] __blocking_notifier_call_chain+0x54/0x70 [ 9031.530387] [] blocking_notifier_call_chain+0x38/0x44 [ 9031.537243] [] pm_notifier_call_chain+0x28/0x48 [ 9031.543662] [] pm_suspend+0x278/0x674 [ 9031.549906] [] state_store+0x58/0x90 [ 9031.554942] [] kobj_attr_store+0x18/0x28 [ 9031.560154] [] sysfs_kf_write+0x5c/0x68 [ 9031.565620] [] kernfs_fop_write+0x114/0x16c [ 9031.571092] [] __vfs_write+0x48/0xf0 [ 9031.576816] [] vfs_write+0xb8/0x150 [ 9031.581848] [] SyS_write+0x58/0x94 [ 9031.586973] [] el0_svc_naked+0x24/0x28-----------------------------------------------------------------------------------------------
Kernel panic is observed during device suspend/resume path in the
kill_requests_without_uevent()called fromfw_pm_notify(). whenpending_listof afirmware_bufis accessed 0x6b(free pattern) pattern
observed. Based on thisfirmware_bufis freed even iffirmware_bufis part
ofpending_fw_headlist.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
What are you doing in userspace to trigger this problem? What kernel driver is this happening with?
Device continuous suspend and resume is happening here. I think, echo mem >
/sys/power/stateissued here.
It is not clear what driver involved here, because afterfirmware_bufis freed all memory gets filled with0x6bpattern.It is important to find what API call was used too.
Did you find out ?
Quoted 33 lines
The only thing I can think of is what follows:
drivers/base/firmware_class.c +1
-1@@ -1005,7 +1005,6 @@ static int _request_firmware_load(struct firmware_priv *fw_priv,mutex_lock (&fw_lock);list_add (&buf->pending_list, &pending_fw_head); -mutex_unlock (&fw_lock);if (opt_flags &FW_OPT_UEVENT ) { buf->need_uevent = true;@@ -1015,6 +1014,7 @@ static int _request_firmware_load(struct firmware_priv *fw_priv, }else { timeout =MAX_JIFFY_OFFSET ; } +mutex_unlock (&fw_lock); retval =fw_state_wait_timeout (&buf->fw_st, timeout);if (retval <0 ) {The reason is that
kill_requests_without_uevent()checks for thebuf->need_ueventbut we modify thebuf->need_uevent= true; above after we unlock. However, I've tried creating a test case against this and ran a loop againsttools/testing/selftests/firmware/fw_fallback.shfirst will all test and then with onlyload_fw_customtest and a timeout of 1 second while I loop suspend/resume and I cannot reproduce. I tried to aggregate the situation by sprinkling ssleep(10) after the lock (without the above patch) and suspending after but no dice. This should give you an idea of what I mean by crafting a hack to try to force the situation if the above really is the fix.
Quoted 1 line
So -- try the above patch,
Did you test it?
Quoted 4 lines
if that does not fix it, then please try to see if
you can create a test case or get crafty and try to force the situation around areas you suspect might cause it, use qemu to try to force this issue and use the test driver.
Did you ?
Quoted 8 lines
Then on a loop run the test and also do: systemctl suspend
If using qemu you can force resume using:
echo
system_wakeup| socat - /dev/pts/7,raw,echo=0,crnlwhere
/dev/pts/7here is my control monitor interface.
Quoted 7 lines
Also add to your
/etc/default/grub(I know, you are probalby on Android right... so whatever you can use to modify the kernel command line) if dynamic debug is enabled and use:dyndbg=\"file
drivers/base/dd.c+p; filedrivers/base/firmware_class.c+p;\"This should hopefully hep narrow down that driver is doing what here.
Did you ?
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
And 4.4.16 is pretty old, can you try 4.9?
We don't have system which runs on new kernels. Looking for possible reasons/path, how
firmware_bufcan get freed when that inpending_fw_headlist.The above is the only thing I can think of. The best is for you to try to reproduce using the test driver and building a careful test case that tries to explode things. The test driver is
lib/test_firmware.cand the script istools/testing/selftests/firmware/fw_fallback.sh.
Please note that working upstream is important, the lost touch of doing so will keep you in the stone age. You need active developers both testing, writing test cases. Without any of this you are working with rather old code and without proper feedback it will be hard to fix any issue.
Luis
Quoted 9 lines
Please note I did test using
CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y.For those on the CC list: please note I was busy and I will review your patches or questions hopefully this week. Also note that due to historic issues which are not well tested like the fallback interface -- I am now asking folks to draw up test cases for their new code. You can however wait until I review your code.
Luis