Skip to contentSkip to conversation

Race fixes in sdhci

PATCH In review Mark Brown 3 participants 5 messages Apr 27, 2011

Subsystems (2)
lore
Share & tools
lore

Mark Brown

author
#0 lore raw

cover Race fixes in sdhci

I've had this pair of patches sitting in my tree for a while now (I believe they were previously posted) providing stability improvements in sdhci on my systems. Having looked through the code I believe but have not confirmed that the issue is that the timeout is racing with an actual completion of a pending task - both paths will trigger the tasklet, and if you trigger a tasklet while it is running this causes it to be rescheduled. The result will be that the tasklet gets run a second time with no work pending for it.

I'm not convinced that these are the best fixes (it feels like we should instead be closing the races down) but I don't really have time to come up with something better myself right now so I'm pushing them out as-is for comment.

Ben Dooks (1):

      MMC: SDHCI: Check mrq->cmd in sdhci_tasklet_finish

Dimitris Papastamos (1):

      MMC: SDHCI: Check mrq != NULL in sdhci_tasklet_finish
 drivers/mmc/host/sdhci.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
#1 lore raw
Quoted 14 lines · Mark Brown

I've had this pair of patches sitting in my tree for a while now (I believe they were previously posted) providing stability improvements in sdhci on my systems.  Having looked through the code I believe but have not confirmed that the issue is that the timeout is racing with an actual completion of a pending task - both paths will trigger the tasklet, and if you trigger a tasklet while it is running this causes it to be rescheduled.  The result will be that the tasklet gets run a second time with no work pending for it.

I'm not convinced that these are the best fixes (it feels like we should instead be closing the races down) but I don't really have time to come up with something better myself right now so I'm pushing them out as-is for comment.

Quoted 8 lines

Ben Dooks (1):
     MMC: SDHCI: Check mrq->cmd in sdhci_tasklet_finish

Dimitris Papastamos (1):
     MMC: SDHCI: Check mrq != NULL in sdhci_tasklet_finish

 drivers/mmc/host/sdhci.c |    2 +-

 1 files changed, 1 insertions(+), 1 deletions(-)

So the timeout interrupt occurs after even though the command succeeds? Am I interpreting that correctly?

A

#2 lore raw

Hi,

Quoted 11 lines · Andrei Warkentin
Quoted 9 lines · Mark Brown

I've had this pair of patches sitting in my tree for a while now (I believe they were previously posted) providing stability improvements in sdhci on my systems. Having looked through the code I believe but have not confirmed that the issue is that the timeout is racing with an actual completion of a pending task - both paths will trigger the tasklet, and if you trigger a tasklet while it is running this causes it to be rescheduled. The result will be that the tasklet gets run a second time with no work pending for it.

Quoted 7 lines
Quoted 4 lines

I'm not convinced that these are the best fixes (it feels like we should instead be closing the races down) but I don't really have time to come up with something better myself right now so I'm pushing them out as-is for comment.

So the timeout interrupt occurs after even though the command succeeds? Am I interpreting that correctly?

No, I think Mark's saying there's a race of:

* the successful completion interrupt fires, and

* the host timer fires to signify timeout due to *lack* of an interrupt (via sdhci_timeout_timer()). i.e., the completion interrupt fires very close to the timeout period.

Both cases call tasklet_schedule(&host->finish_tasklet), and if you manage to schedule a tasklet while it's already running, it runs again after it completes -- but during the first run we set host->mrq->cmd to NULL, so then it oopses on the second run.

We could consider taking Mark's first patch and also adding to the top:

        /*
         * If we get scheduled twice concurrently, this tasklet will
         * be run again afterwards but without any active request.
         */
        if (!host->mrq)
                return;

.. and pushing to .39 with a stable@ tag.

- Chris.

Signature · 2 lines
Chris Ball   <[email protected]>   <http://printf.net/>
One Laptop Per Child
author
#3 lore raw
Quoted 4 lines · Chris Ball

No, I think Mark's saying there's a race of:

* the successful completion interrupt fires, and

Quoted 8 lines

* the host timer fires to signify timeout due to *lack* of an interrupt (via sdhci_timeout_timer()). i.e., the completion interrupt fires very close to the timeout period.

Both cases call tasklet_schedule(&host->finish_tasklet), and if you manage to schedule a tasklet while it's already running, it runs again after it completes -- but during the first run we set host->mrq->cmd to NULL, so then it oopses on the second run.

Yes, exactly - that's what I saw through code inspection. I haven't actually seen this directly and analysed enough to understand a particular race, I just looked at the fixes that we are carrying in our tree and looked at the code sufficiently to determine that the fixes addressed an issue I could identify.

Quoted 1 line

We could consider taking Mark's first patch and also adding to the top:

Ben's patch.

Quoted 8 lines
        /*
         * If we get scheduled twice concurrently, this tasklet will
         * be run again afterwards but without any active request.
         */
        if (!host->mrq)
                return;

.. and pushing to .39 with a stable@ tag.

Yes, that's a broader way of writing Dimitris' patch. The issue isn't concurrent *schedules* - they will just fall out into a single queue for the task - it's that the tasklet can be rescheduled while it's running.

#4 lore raw
In reply to Mark Brown

Hi Mark,

Quoted 5 lines · Mark Brown

Ben Dooks (1):

      MMC: SDHCI: Check mrq->cmd in sdhci_tasklet_finish

Dimitris Papastamos (1):

      MMC: SDHCI: Check mrq != NULL in sdhci_tasklet_finish

Thanks. I've merged Ben's patch for .39, and also:

From: Chris Ball <[email protected]>
Subject: [PATCH] mmc: sdhci: Check mrq != NULL in sdhci_tasklet_finish

It seems that under certain circumstances the sdhci_tasklet_finish() call can be entered with mrq set to NULL, causing the system to crash with a NULL pointer de-reference.

Seen on S3C6410 system. Based on a patch by Dimitris Papastamos.

---
 drivers/mmc/host/sdhci.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
drivers/mmc/host/sdhci.c +7 -0
index e4084a3..f197c67 100644
@@ -1808,6 +1808,13 @@ static void sdhci_tasklet_finish(unsigned long param)
 
 	host = (struct sdhci_host*)param;
 
+        /*
+         * If this tasklet gets rescheduled while running, it will
+         * be run again afterwards but without any active request.
+         */
+	if (!host->mrq)
+		return;
+
 	spin_lock_irqsave(&host->lock, flags);
 
 	del_timer(&host->timer);
Signature · 2 lines
Chris Ball   <[email protected]>   <http://printf.net/>
One Laptop Per Child