use-after-free access in bt_iter()
Hi Jens, Bart,
This is with regards to use-after-free access in bt_iter().
i saw this got discussed and reported on many separate threads but could see more discussions and conversations over the solution was made on [1] as pointed in [2].
[1] https://lore.kernel.org/linux-block/[email protected]/
[2] https://lkml.org/lkml/2019/2/14/942
A similar issue was reported again on 5.4 kernel during internal stability testing.
<2> Unable to handle kernel paging request at virtual address ffffff8107929600 <2> Mem abort info:
<2> ESR = 0x96000007 <2> EC = 0x25: DABT (current EL), IL = 32 bits <2> SET = 0, FnV = 0 <2> EA = 0, S1PTW = 0
<2> Data abort info:
<2> ISV = 0, ISS = 0x00000007
<2> CM = 0, WnR = 0
<2> swapper pgtable: 4k pages, 39-bit VAs, pgdp=00000000a2603000
<2> [ffffff8107929600] pgd=00000001bf909003, pud=00000001bf909003, pmd=00000001bf8cc003, pte=0068000187929f12 <2> Internal error: Oops: 96000007 [#1] PREEMPT SMP
<2> Skip md ftrace buffer dump for: 0x1609e0
<2> CPU: 0 PID: 220 Comm: kworker/0:1H Tainted: G S W O 5.4.61-qgki-debug-g85faaf6 #2
<2> Workqueue: kblockd blk_mq_timeout_work
<2> pstate: 20c00005 (nzCv daif +PAN +UAO)
<2> pc : bt_for_each+0x114/0x1a4
<2> lr : bt_for_each+0xe0/0x1a4
<2> sp : ffffffc017f7bc60
<2> x29: ffffffc017f7bc80 x28: 0000000000000001
<2> x27: 0000000000000008 x26: 0000000000000001
<2> x25: 0000000000000001 x24: 0000000000000008
<2> x23: ffffff8107bcd800 x22: ffffff810872bd10
<2> x21: ffffffd764e6ea50 x20: 0000000000000008
<2> x19: 0000000000000000 x18: ffffffc017f51030
<2> x17: 0000000005f5e100 x16: 0000000000000000
<2> x15: ffffffffff84bf5c x14: 0000000000000598
<2> x13: 0000000000000008 x12: 00000000212d4a53
<2> x11: 00000000000000ff x10: 0000000000000000
<2> x9 : ffffff810872cd00 x8 : 0000000000000009
<2> x7 : 0000000000000000 x6 : ffffffd763890758
<2> x5 : 0000000000000000 x4 : 0000000000000000
<2> x3 : ffffffc017f7bd20 x2 : 0000000000000001
<2> x1 : ffffff8107929600 x0 : 0000000000000001
<2> Call trace:
<2> bt_for_each+0x114/0x1a4 <2> blk_mq_queue_tag_busy_iter+0xd8/0x1a4 <2> blk_mq_timeout_work+0xd4/0x1c0 <2> process_one_work+0x280/0x460 <2> worker_thread+0x27c/0x4dc <2> kthread+0x160/0x170 <2> ret_from_fork+0x10/0x18
Is this issue got fixed on any latest kernel ? if so, can you please help point the patch ? If not got fixed, can we have a final solution ? i can even help in testing the solution.
Thanks and Regards,
Pradeep
Quoted 4 lines · On 2/4/21 3:46 AM, [email protected] wrote:
Is this issue got fixed on any latest kernel ? if so, can you please help point the patch ?
If not got fixed, can we have a final solution ? i can even help in testing the solution.
Hi John,
Some time ago you replied the following to an email from me with a suggestion for a fix: "Please let me consider it a bit more." Are you still working on a fix?
See also
https://lore.kernel.org/linux-block/[email protected]/
Thanks,
Bart.
Quoted 7 lines · Bart Van Assche
Quoted 4 lines · On 2/4/21 3:46 AM,[email protected] wrote:
Is this issue got fixed on any latest kernel ? if so, can you please help point the patch ?
If not got fixed, can we have a final solution ? i can even help in testing the solution.Hi John,
Hi Bart,
Quoted 3 lines
Some time ago you replied the following to an email from me with a suggestion for a fix: "Please let me consider it a bit more." Are you still working on a fix?
Unfortunately I have not had a chance, sorry. But I can look again.
So I have only seen KASAN use-after-free's myself, but never an actual oops. IIRC, someone did report an oops.
@Pradeep, do you have a reliable re-creator? I noticed the timeout handler stackframe in your mail, so I guess not. However, as an experiment, could you test:
https://lore.kernel.org/linux-block/[email protected]/
This should fix the common issue. But no final solution to issues discussed from patch 2/2, which is more exotic.
BTW, is this the same Pradeep who reported:
https://lore.kernel.org/linux-block/[email protected]/
I did cc [email protected] on earlier version of my series, but it bounced.
Quoted 3 lines
See also
https://lore.kernel.org/linux-block/[email protected]/
Thanks,
John
Quoted 20 lines · John Garry
Quoted 7 lines · Bart Van Assche
Quoted 4 lines · On 2/4/21 3:46 AM,[email protected] wrote:
Is this issue got fixed on any latest kernel ? if so, can you please help point the patch ?
If not got fixed, can we have a final solution ? i can even help in testing the solution.Hi John,
Hi Bart,
Quoted 3 lines
Some time ago you replied the following to an email from me with a suggestion for a fix: "Please let me consider it a bit more." Are you still working on a fix?
Unfortunately I have not had a chance, sorry. But I can look again.
So I have only seen KASAN use-after-free's myself, but never an actual oops. IIRC, someone did report an oops.
Quoted 8 lines
@Pradeep, do you have a reliable re-creator? I noticed the timeout handler stackframe in your mail, so I guess not. However, as an experiment, could you test:
https://lore.kernel.org/linux-block/[email protected]/This should fix the common issue. But no final solution to issues discussed from patch 2/2, which is more exotic.
If still no progress, I'd suggest to consider the patches I posted:
https://lore.kernel.org/linux-block/[email protected]/#r
The idea is quite simple at least, :-)
Signature · 1 line
Ming
Quoted 11 lines · John Garry
Quoted 7 lines · Bart Van Assche
Quoted 4 lines · On 2/4/21 3:46 AM,[email protected] wrote:
Is this issue got fixed on any latest kernel ? if so, can you please help point the patch ?
If not got fixed, can we have a final solution ? i can even help in testing the solution.Hi John,
Hi Bart,
Quoted 9 lines
Quoted 3 lines
Some time ago you replied the following to an email from me with a suggestion for a fix: "Please let me consider it a bit more." Are you still working on a fix?
Unfortunately I have not had a chance, sorry. But I can look again.
So I have only seen KASAN use-after-free's myself, but never an actual oops. IIRC, someone did report an oops.
Hi John,
Quoted 5 lines
@Pradeep, do you have a reliable re-creator? I noticed the timeout handler stackframe in your mail, so I guess not. However, as an experiment, could you test:
https://lore.kernel.org/linux-block/[email protected]/
Yes, i don't have a reliable re-creator. The oops was noticed as a part of stability testing and was not an intentional try. This was noticed couple of times.
Please share the steps (if any) to easy hit or to exercise this path more frequently. Meanwhile, i will go with the usual stability procedure. i will update the results here later.
Quoted 8 lines
This should fix the common issue. But no final solution to issues discussed from patch 2/2, which is more exotic.
BTW, is this the same Pradeep who reported:
https://lore.kernel.org/linux-block/[email protected]/I did cc [email protected] on earlier version of my series, but it bounced.
Yes, it's the same Pradeep. Unfortunately my old email "[email protected]" got expired and couldn't able to restore. Hence the bounced emails. Now this got resolved with a new email "[email protected]" which I'm now currently replying.
Quoted 6 lines
Quoted 3 lines
See also
https://lore.kernel.org/linux-block/[email protected]/Thanks,
John
Thanks and Regards,
Pradeep
- bouncing [email protected]
Quoted 18 lines
Quoted 10 lines
Quoted 4 lines
Some time ago you replied the following to an email from me with a suggestion for a fix: "Please let me consider it a bit more." Are you still working on a fix?
Unfortunately I have not had a chance, sorry. But I can look again.
So I have only seen KASAN use-after-free's myself, but never an actual oops. IIRC, someone did report an oops.
Hi John,
Quoted 6 lines
@Pradeep, do you have a reliable re-creator? I noticed the timeout handler stackframe in your mail, so I guess not. However, as an experiment, could you test:
https://lore.kernel.org/linux-block/[email protected]/
Quoted 5 lines
Yes, i don't have a reliable re-creator. The oops was noticed as a part of stability testing and was not an intentional try. This was noticed couple of times.
Please share the steps (if any) to easy hit or to exercise this path more frequently. Meanwhile, i will go with the usual stability procedure. i will update the results here later.
Do you have a full kernel log for your crash?
So there are different flavors of this issue, and you reported a crash from blk_mq_queue_tag_busy_iter().
If you check:
https://lore.kernel.org/linux-block/[email protected]/
You can see how I artificially trigger an issue in blk_mq_queue_tag_busy_iter().
Quoted 6 lines
Quoted 6 lines
This should fix the common issue. But no final solution to issues discussed from patch 2/2, which is more exotic.
BTW, is this the same Pradeep who reported:
https://lore.kernel.org/linux-block/[email protected]/
Thanks,
John
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 19 lines · On 2021-02-05 21:51, [email protected] wrote:
Quoted 18 lines · John Garry
- bouncing [email protected]
Quoted 16 lines
Quoted 10 lines
Quoted 4 lines
Some time ago you replied the following to an email from me with a suggestion for a fix: "Please let me consider it a bit more." Are you still working on a fix?
Unfortunately I have not had a chance, sorry. But I can look again.
So I have only seen KASAN use-after-free's myself, but never an actual oops. IIRC, someone did report an oops.
Hi John,
Quoted 4 lines
@Pradeep, do you have a reliable re-creator? I noticed the timeout handler stackframe in your mail, so I guess not. However, as an experiment, could you test:
https://lore.kernel.org/linux-block/[email protected]/
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Yes, i don't have a reliable re-creator. The oops was noticed as a part of stability testing and was not an intentional try. This was noticed couple of times.
Please share the steps (if any) to easy hit or to exercise this path more frequently. Meanwhile, i will go with the usual stability procedure. i will update the results here later.
Hi John,
we ran the stability with the above patch
(https://lore.kernel.org/linux-block/[email protected]/)
with switching the io-schedulers in b/w for ~88hrs on 2 devices, we didn't notice any crash/issue.
Quoted 23 lines
Quoted 1 line
Do you have a full kernel log for your crash?
Yes. Attaching the full kernel dmesg log.
Quoted 8 lines
So there are different flavors of this issue, and you reported a crash from
blk_mq_queue_tag_busy_iter().If you check:
https://lore.kernel.org/linux-block/[email protected]/You can see how I artificially trigger an issue in
blk_mq_queue_tag_busy_iter().Sure, i will go through the steps on the recreation part. Thanks.
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
This should fix the common issue. But no final solution to issues discussed from patch 2/2, which is more exotic.
BTW, is this the same Pradeep who reported:
https://lore.kernel.org/linux-block/[email protected]/Thanks,
JohnThanks and Regards,
Pradeep
Quoted 9 lines · On 19/02/2021 06:22, [email protected] wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Hi John,
we ran the stability with the above patch
(https://lore.kernel.org/linux-block/[email protected]/)with switching the io-schedulers in b/w for ~88hrs on 2 devices, we didn't notice any crash/issue.
Oh, good. I assume that this is same test which you were reporting crashes on previously.
So we still have the issue of changing IO scheduler and the tagset iters holding old references to tags. I tried Bart's idea to deal with the request queue tagset iter, and it seems to work, but not sure if acceptable. And also I might have a solution for blk_mq_tagset_busy_iter().
However I am not sure if these are seen in real-life, and whether the patch you tested is good enough for now.
Let me look at this again and I will report back.
Thanks very much,
John
Quoted 1 line
Quoted 1 line
Quoted 1 line
Do you have a full kernel log for your crash?