Race fixes in sdhci
cover Race fixes in sdhci
I've had this pair of patches sitting in my tree for a while now (I believe they were previously posted) providing stability improvements in sdhci on my systems. Having looked through the code I believe but have not confirmed that the issue is that the timeout is racing with an actual completion of a pending task - both paths will trigger the tasklet, and if you trigger a tasklet while it is running this causes it to be rescheduled. The result will be that the tasklet gets run a second time with no work pending for it.
I'm not convinced that these are the best fixes (it feels like we should instead be closing the races down) but I don't really have time to come up with something better myself right now so I'm pushing them out as-is for comment.
Ben Dooks (1):
MMC: SDHCI: Check mrq->cmd in sdhci_tasklet_finish
Dimitris Papastamos (1):
MMC: SDHCI: Check mrq != NULL in sdhci_tasklet_finish
drivers/mmc/host/sdhci.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
Quoted 14 lines · Mark Brown
I've had this pair of patches sitting in my tree for a while now (I believe they were previously posted) providing stability improvements in sdhci on my systems. Having looked through the code I believe but have not confirmed that the issue is that the timeout is racing with an actual completion of a pending task - both paths will trigger the tasklet, and if you trigger a tasklet while it is running this causes it to be rescheduled. The result will be that the tasklet gets run a second time with no work pending for it.
I'm not convinced that these are the best fixes (it feels like we should instead be closing the races down) but I don't really have time to come up with something better myself right now so I'm pushing them out as-is for comment.
Quoted 8 lines
Ben Dooks (1):
MMC: SDHCI: Checkmrq->cmdinsdhci_tasklet_finishDimitris Papastamos (1):
MMC: SDHCI: Check mrq != NULL insdhci_tasklet_finishdrivers/mmc/host/sdhci.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
So the timeout interrupt occurs after even though the command succeeds? Am I interpreting that correctly?
A
Hi,
Quoted 11 lines · Andrei Warkentin
Quoted 9 lines · Mark Brown
I've had this pair of patches sitting in my tree for a while now (I believe they were previously posted) providing stability improvements in sdhci on my systems. Having looked through the code I believe but have not confirmed that the issue is that the timeout is racing with an actual completion of a pending task - both paths will trigger the tasklet, and if you trigger a tasklet while it is running this causes it to be rescheduled. The result will be that the tasklet gets run a second time with no work pending for it.
Quoted 7 lines
Quoted 4 lines
I'm not convinced that these are the best fixes (it feels like we should instead be closing the races down) but I don't really have time to come up with something better myself right now so I'm pushing them out as-is for comment.
So the timeout interrupt occurs after even though the command succeeds? Am I interpreting that correctly?
No, I think Mark's saying there's a race of:
* the successful completion interrupt fires, and
* the host timer fires to signify timeout due to *lack* of an interrupt (via sdhci_timeout_timer()). i.e., the completion interrupt fires very close to the timeout period.
Both cases call tasklet_schedule(&host->finish_tasklet), and if you manage to schedule a tasklet while it's already running, it runs again after it completes -- but during the first run we set host->mrq->cmd to NULL, so then it oopses on the second run.
We could consider taking Mark's first patch and also adding to the top:
/*
* If we get scheduled twice concurrently, this tasklet will
* be run again afterwards but without any active request.
*/
if (!host->mrq)
return;.. and pushing to .39 with a stable@ tag.
- Chris.
Signature · 2 lines
Chris Ball <[email protected]> <http://printf.net/> One Laptop Per Child
Quoted 4 lines · Chris Ball
No, I think Mark's saying there's a race of:
* the successful completion interrupt fires, and
Quoted 8 lines
* the host timer fires to signify timeout due to *lack* of an interrupt (via
sdhci_timeout_timer()). i.e., the completion interrupt fires very close to the timeout period.Both cases call
tasklet_schedule(&host->finish_tasklet), and if you manage to schedule a tasklet while it's already running, it runs again after it completes -- but during the first run we sethost->mrq->cmdto NULL, so then it oopses on the second run.
Yes, exactly - that's what I saw through code inspection. I haven't actually seen this directly and analysed enough to understand a particular race, I just looked at the fixes that we are carrying in our tree and looked at the code sufficiently to determine that the fixes addressed an issue I could identify.
Quoted 1 line
We could consider taking Mark's first patch and also adding to the top:
Ben's patch.
Quoted 8 lines
/* * If we get scheduled twice concurrently, this tasklet will * be run again afterwards but without any active request. */ if (!host->mrq) return;.. and pushing to .39 with a stable@ tag.
Yes, that's a broader way of writing Dimitris' patch. The issue isn't concurrent *schedules* - they will just fall out into a single queue for the task - it's that the tasklet can be rescheduled while it's running.
Hi Mark,
Quoted 5 lines · Mark Brown
Ben Dooks (1):
MMC: SDHCI: Check mrq->cmd in sdhci_tasklet_finishDimitris Papastamos (1):
MMC: SDHCI: Check mrq != NULL in sdhci_tasklet_finish
Thanks. I've merged Ben's patch for .39, and also:
From: Chris Ball <[email protected]>
Subject: [PATCH] mmc: sdhci: Check mrq != NULL in sdhci_tasklet_finish
It seems that under certain circumstances the sdhci_tasklet_finish() call can be entered with mrq set to NULL, causing the system to crash with a NULL pointer de-reference.
Seen on S3C6410 system. Based on a patch by Dimitris Papastamos.
--- drivers/mmc/host/sdhci.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)
drivers/mmc/host/sdhci.c +7 -0
@@ -1808,6 +1808,13 @@ static void sdhci_tasklet_finish(unsigned long param) host = (struct sdhci_host *)param; +/* +* If this tasklet gets rescheduled while running, it will +* be run again afterwards but without any active request. +*/ +if (!host->mrq) +return ; +spin_lock_irqsave (&host->lock, flags);del_timer (&host->timer);
Signature · 2 lines
Chris Ball <[email protected]> <http://printf.net/> One Laptop Per Child