SDHC card affected by preemption model in 2.6.35

HW: Beagleboard rev. C2 and C4
Processor: OMAP3
Kernel: 2.6.35-rc2
Driver: mmci-omap-hs

I am faced with an SDHC card problem on a beagleboard.  Some cards cannot be initialized on startup while others work perfectly.  I tracked the issue down to one single kernel config: PREEMPT_VOLUNTARY.

When going from PREEMPT_VOLUNTARY to PREEMPT_NONE the problem goes away.

When booting, a failing card (PREEMPT_VOLUNTARY) will output the following:

[ 2.283355] mmc0: error -110 whilst initialising SD card

I have also seen transfer errors such as this one:

[ 105.343780] mmcblk0: error -110 transferring data, sector 798431, nr

26, card status 0xc00

When working properly (PREEMPT_NONE), you get:
[   27.026519] mmc0: new high speed SDHC card at address 0007 [   27.075775] mmcblk0: mmc0:0007 SD08G 7.49 GiB

We seem to have a little timing problem - has anyone seen the same issue ?  Can driver "mmci-omap-hs" actually work under PREEMPT_VOLUNTARY ?

Thanks, Mathieu.

Quoted 20 lines · Mathieu Poirier

HW: Beagleboard rev. C2 and C4
Processor: OMAP3
Kernel: 2.6.35-rc2
Driver: mmci-omap-hs

I am faced with an SDHC card problem on a beagleboard. Some cards cannot be initialized on startup while others work perfectly. I tracked the issue down to one single kernel config: PREEMPT_VOLUNTARY.

When going from PREEMPT_VOLUNTARY to PREEMPT_NONE the problem goes away.

When booting, a failing card (PREEMPT_VOLUNTARY) will output the following:

[ 2.283355] mmc0: error -110 whilst initialising SD card

I have also seen transfer errors such as this one:

[ 105.343780] mmcblk0: error -110 transferring data, sector 798431, nr

26, card status 0xc00

When working properly (PREEMPT_NONE), you get:

Quoted 9 lines

[   27.026519] mmc0: new high speed SDHC card at address 0007
[   27.075775] mmcblk0: mmc0:0007 SD08G 7.49 GiB

We seem to have a little timing problem - has anyone seen the same issue ? Can driver "mmci-omap-hs" actually work under PREEMPT_VOLUNTARY ?

Thanks, Mathieu.

I will check this out - not tested with PREEMPT_VOLUNTARY so far. If it's not too much trouble, can you provide a log with CONFIG_MMC_DEBUG ? Also, some details about the failing card would be helpful.

Thanks,
Venkat.

Quoted 2 lines

Quoted 2 lines

When going from PREEMPT_VOLUNTARY to PREEMPT_NONE

the problem goes away.

Quoted 2 lines

Quoted 1 line

Quoted 1 line

When going from PREEMPT_VOLUNTARY to PREEMPT_NONE

the problem goes away.

Have you found whether this is a problem
with the OMAP HSMMC driver
versus the MMC/SD framework code

Quoted 31 lines · Venkatraman S

Quoted 23 lines · Mathieu Poirier

HW: Beagleboard rev. C2 and C4
Processor: OMAP3
Kernel: 2.6.35-rc2
Driver: mmci-omap-hs

I am faced with an SDHC card problem on a beagleboard.  Some cards cannot be initialized on startup while others work perfectly.  I tracked the issue down to one single kernel config: PREEMPT_VOLUNTARY.

When going from PREEMPT_VOLUNTARY to PREEMPT_NONE the problem goes away.

When booting, a failing card (PREEMPT_VOLUNTARY) will output the following:

[ 2.283355] mmc0: error -110 whilst initialising SD card

I have also seen transfer errors such as this one:

[ 105.343780] mmcblk0: error -110 transferring data, sector 798431, nr

26, card status 0xc00

When working properly (PREEMPT_NONE), you get:
[   27.026519] mmc0: new high speed SDHC card at address 0007 [   27.075775] mmcblk0: mmc0:0007 SD08G 7.49 GiB

Quoted 6 lines

We seem to have a little timing problem - has anyone seen the same issue ?  Can driver "mmci-omap-hs" actually work under PREEMPT_VOLUNTARY ?

Thanks, Mathieu.

Quoted 6 lines

I will check this out - not tested with PREEMPT_VOLUNTARY so far. If it's not too much trouble, can you provide a log with CONFIG_MMC_DEBUG ? Also, some details about the failing card would be helpful.

Thanks,
Venkat.

Venkat,

Unfortunately enabling CONFIG_MMC_DEBUG doesn't yield more information - the error message is the same and no additional output shows on the console.

As for the cards, had failures with 3 different manufacturer:
- Patriot Memory, MicroSD card, 8GB, class 4, SDHC.
- Kinston, 4GB, class 6, SDHC.
- Sandisk, 4GB, Class 2, SDHC.

I am more than willing to test kernels for you if need be.

Thanks, Mathieu.

Quoted 45 lines · Mathieu Poirier

Quoted 31 lines · Venkatraman S

Quoted 20 lines · Mathieu Poirier

HW: Beagleboard rev. C2 and C4
Processor: OMAP3
Kernel: 2.6.35-rc2
Driver: mmci-omap-hs

I am faced with an SDHC card problem on a beagleboard. Some cards cannot be initialized on startup while others work perfectly. I tracked the issue down to one single kernel config: PREEMPT_VOLUNTARY.

When going from PREEMPT_VOLUNTARY to PREEMPT_NONE the problem goes away.

When booting, a failing card (PREEMPT_VOLUNTARY) will output the following:

[ 2.283355] mmc0: error -110 whilst initialising SD card

I have also seen transfer errors such as this one:

[ 105.343780] mmcblk0: error -110 transferring data, sector 798431, nr

26, card status 0xc00

When working properly (PREEMPT_NONE), you get:

Quoted 9 lines

[   27.026519] mmc0: new high speed SDHC card at address 0007
[   27.075775] mmcblk0: mmc0:0007 SD08G 7.49 GiB

We seem to have a little timing problem - has anyone seen the same issue ? Can driver "mmci-omap-hs" actually work under PREEMPT_VOLUNTARY ?

Thanks, Mathieu.

Quoted 6 lines

I will check this out - not tested with PREEMPT_VOLUNTARY so far. If it's not too much trouble, can you provide a log with CONFIG_MMC_DEBUG ? Also, some details about the failing card would be helpful.

Thanks,
Venkat.

Venkat,

Unfortunately enabling CONFIG_MMC_DEBUG doesn't yield more information - the error message is the same and no additional output shows on the console.

Quoted 9 lines

As for the cards, had failures with 3 different manufacturer:
- Patriot Memory, MicroSD card, 8GB, class 4, SDHC.
- Kinston, 4GB, class 6, SDHC.
- Sandisk, 4GB, Class 2, SDHC.

I am more than willing to test kernels for you if need be.

Thanks, Mathieu.

For MMC/SD logs to be sent to the console, you need to
a) "echo 8 > /proc/sys/kernel/printk" in the shell and
b) insert the card

If you are booting from the card itself, then this won't work and DEBUG_LL has to be enabled (in addition to CONFIG_MMC_DEBUG)

Apologies - I should have explained this initially.

Regards,
Venkat.

Quoted 60 lines · Venkatraman S

Quoted 45 lines · Mathieu Poirier

Quoted 31 lines · Venkatraman S

Quoted 23 lines · Mathieu Poirier

HW: Beagleboard rev. C2 and C4
Processor: OMAP3
Kernel: 2.6.35-rc2
Driver: mmci-omap-hs

I am faced with an SDHC card problem on a beagleboard.  Some cards cannot be initialized on startup while others work perfectly.  I tracked the issue down to one single kernel config: PREEMPT_VOLUNTARY.

When going from PREEMPT_VOLUNTARY to PREEMPT_NONE the problem goes away.

When booting, a failing card (PREEMPT_VOLUNTARY) will output the following:

[ 2.283355] mmc0: error -110 whilst initialising SD card

I have also seen transfer errors such as this one:

[ 105.343780] mmcblk0: error -110 transferring data, sector 798431, nr

26, card status 0xc00

When working properly (PREEMPT_NONE), you get:
[   27.026519] mmc0: new high speed SDHC card at address 0007 [   27.075775] mmcblk0: mmc0:0007 SD08G 7.49 GiB

Quoted 6 lines

We seem to have a little timing problem - has anyone seen the same issue ?  Can driver "mmci-omap-hs" actually work under PREEMPT_VOLUNTARY ?

Thanks, Mathieu.

Quoted 6 lines

I will check this out - not tested with PREEMPT_VOLUNTARY so far. If it's not too much trouble, can you provide a log with CONFIG_MMC_DEBUG ? Also, some details about the failing card would be helpful.

Thanks,
Venkat.

Venkat,

Unfortunately enabling CONFIG_MMC_DEBUG doesn't yield more information - the error message is the same and no additional output shows on the console.

Quoted 9 lines

As for the cards, had failures with 3 different manufacturer:
- Patriot Memory, MicroSD card, 8GB, class 4, SDHC.
- Kinston, 4GB, class 6, SDHC.
- Sandisk, 4GB, Class 2, SDHC.

I am more than willing to test kernels for you if need be.

Thanks, Mathieu.

For MMC/SD logs to be sent to the console, you need to
a) "echo 8 > /proc/sys/kernel/printk" in the shell and
b) insert the card

Quoted 7 lines

If you are booting from the card itself, then this won't work and DEBUG_LL has to be enabled (in addition to CONFIG_MMC_DEBUG)

Apologies - I should have explained this initially.

Regards,
Venkat.

Venkat,

I am indeed booting from the card itself, making things more difficult. DEBUG_LL has been configured since the very beginning and still not much to look at on the console.  To see something I had to pass loglevel=8 on the kernel command line.  At that point there is tones of stuff coming out and the card is initialized properly, which points to a timing issue.

Since I couldn't reproduce the failure when debug messages are enabled, I turned them back off and started to instrument the code on the hunt for the failure.

I have cornered the source of the problem in "drivers/mmc/core/sd_ops.c", function "mmc_sd_switch".  When the kernel is configured with PREEMPT_NONE, the value of "data.error" is set to "0" after "mmc_wait_for_req" returns.  When PREEMPT_VOLUNTARY is configured, "data.error" gets set to "-110" upon "mmc_wait_for_req" returning, which prevent the remaining of the configuration to take place.

Quoted 82 lines · Mathieu Poirier

Quoted 60 lines · Venkatraman S

Quoted 45 lines · Mathieu Poirier

Quoted 31 lines · Venkatraman S

Quoted 20 lines · Mathieu Poirier

HW: Beagleboard rev. C2 and C4
Processor: OMAP3
Kernel: 2.6.35-rc2
Driver: mmci-omap-hs

I am faced with an SDHC card problem on a beagleboard. Some cards cannot be initialized on startup while others work perfectly. I tracked the issue down to one single kernel config: PREEMPT_VOLUNTARY.

When going from PREEMPT_VOLUNTARY to PREEMPT_NONE the problem goes away.

When booting, a failing card (PREEMPT_VOLUNTARY) will output the following:

[ 2.283355] mmc0: error -110 whilst initialising SD card

I have also seen transfer errors such as this one:

[ 105.343780] mmcblk0: error -110 transferring data, sector 798431, nr

26, card status 0xc00

When working properly (PREEMPT_NONE), you get:

Quoted 9 lines

[   27.026519] mmc0: new high speed SDHC card at address 0007
[   27.075775] mmcblk0: mmc0:0007 SD08G 7.49 GiB

We seem to have a little timing problem - has anyone seen the same issue ? Can driver "mmci-omap-hs" actually work under PREEMPT_VOLUNTARY ?

Thanks, Mathieu.

Quoted 6 lines

I will check this out - not tested with PREEMPT_VOLUNTARY so far. If it's not too much trouble, can you provide a log with CONFIG_MMC_DEBUG ? Also, some details about the failing card would be helpful.

Thanks,
Venkat.

Venkat,

Unfortunately enabling CONFIG_MMC_DEBUG doesn't yield more information - the error message is the same and no additional output shows on the console.

Quoted 9 lines

As for the cards, had failures with 3 different manufacturer:
- Patriot Memory, MicroSD card, 8GB, class 4, SDHC.
- Kinston, 4GB, class 6, SDHC.
- Sandisk, 4GB, Class 2, SDHC.

I am more than willing to test kernels for you if need be.

Thanks, Mathieu.

For MMC/SD logs to be sent to the console, you need to
a) "echo 8 > /proc/sys/kernel/printk" in the shell and
b) insert the card

Quoted 7 lines

If you are booting from the card itself, then this won't work and DEBUG_LL has to be enabled (in addition to CONFIG_MMC_DEBUG)

Apologies - I should have explained this initially.

Regards,
Venkat.

Venkat,

I am indeed booting from the card itself, making things more difficult. DEBUG_LL has been configured since the very beginning and still not much to look at on the console. To see something I had to pass loglevel=8 on the kernel command line. At that point there is tones of stuff coming out and the card is initialized properly, which points to a timing issue.

Since I couldn't reproduce the failure when debug messages are enabled, I turned them back off and started to instrument the code on the hunt for the failure.

Quoted 7 lines

I have cornered the source of the problem in "drivers/mmc/core/sd_ops.c", function "mmc_sd_switch". When the kernel is configured with PREEMPT_NONE, the value of "data.error" is set to "0" after "mmc_wait_for_req" returns. When PREEMPT_VOLUNTARY is configured, "data.error" gets set to "-110" upon "mmc_wait_for_req" returning, which prevent the remaining of the configuration to take place.

-110 = -ETIMEDOUT.
The error is set from omap_hsmmc.c line 1029, when the controller returns a DATA_TIMEOUT error.
This can happen for any mmc_wait_for_req() call, not just in mmc_sd_switch().

A simple workaround is to increase the timeout value, as below. Could you please try with this patch ?
Yes, it's a debugging hack and not the right solution.
[I have so far not reproduced the problem with my setup. Still trying..]

----

drivers/mmc/host/omap_hsmmc.c +1 -1

index b032828..9ca399e 100644
@@ -1413,7 +1413,7 @@ omap_hsmmc_prepare_data(struct omap_hsmmc_host
 *host, struct mmc_request *req)
 
        OMAP_HSMMC_WRITE(host->base, BLK, (req->data->blksz)
                                        | (req->data->blocks << 16));
-       set_data_timeout(host, req->data->timeout_ns, req->data->timeout_clks);
+       set_data_timeout(host, 100000000U, 0);
 
        if (host->use_dma) {

                ret = omap_hsmmc_start_dma_transfer(host, req);

----

This provides a good 100ms window for delays due to scheduling variations.
I have to still find out which section is sensitive to it. Regards,
Venkat.
--

Quoted 11 lines · Thomas Weber

Hello Mathieu, hello Venkat,

I hope it is not too old and you remember this:

Quoted 6 lines

  This provides a good 100ms window for delays due to scheduling variations. I have to still find out which section is sensitive to it. Regards,
Venkat.
--

Quoted 7 lines

We have a custom omap3 board and use kernel 2.6.37-rc8.  We have the problem that we sometimes cannot mount our rootfs from sd-card. This happens only when CONFIG_PREEMPT=y. When using CONFIG_PREEMPT_NONE or enabling CONFIG_MMC_DEBUG everything works fine.

The "100 ms patch" doesn't work for us. Do you found other solution(s)?