Skip to contentSkip to conversation

MMC/SD Root filesystem suspend/resume problems

DISCUSSION Richard Purdie 3 participants 10 messages Jul 19, 2007 – Jul 25, 2007

lore
Share & tools
lore
#0 lore raw

Hi Pierre,

Lots of Linux handhelds use MMC/SD devices as the root file system. This has worked quite reliably for many kernel versions. In 2.6.22, it seems that if you suspend such a system then resume it, the device locks up. Trying to execute anything on the filesystem results in a "Permission Denied" message. I did see a message from the MMC subsystem saying it had redetected the card. There are also messages on the console like "MMC: killing requests for dead queue" each time you suspend/resume.

I'm away from my serial cables at the moment but I may be able to provide more debug when I have them over the weekend. Have you any ideas on why this is breaking?

For reference, I've reproduced the problem with both the PXA host driver and different driver not merged into mainline (ASIC3).

Cheers,

Richard

#1 lore raw
Quoted 8 lines · Richard Purdie

Lots of Linux handhelds use MMC/SD devices as the root file system. This has worked quite reliably for many kernel versions. In 2.6.22, it seems that if you suspend such a system then resume it, the device locks up. Trying to execute anything on the filesystem results in a "Permission Denied" message. I did see a message from the MMC subsystem saying it had redetected the card. There are also messages on the console like "MMC: killing requests for dead queue" each time you suspend/resume.

Quoted 6 lines

I'm away from my serial cables at the moment but I may be able to provide more debug when I have them over the weekend. Have you any ideas on why this is breaking?

For reference, I've reproduced the problem with both the PXA host driver and different driver not merged into mainline (ASIC3).

Just to follow up, if I boot with a rootfs from elsewhere and mount the mmc card then suspend/resume, it corrupts the data on the card too so it looks like some general suspend/resume problem on mounted filesystems.

Richard

#2 lore raw
In reply to Richard Purdie
Quoted 11 lines · On Thu, 19 Jul 2007 16:53:39 +0100 Richard Purdie <[email protected]> wrote:

Hi Pierre,

Lots of Linux handhelds use MMC/SD devices as the root file system. This has worked quite reliably for many kernel versions. In 2.6.22, it seems that if you suspend such a system then resume it, the device locks up. Trying to execute anything on the filesystem results in a "Permission Denied" message. I did see a message from the MMC subsystem saying it had redetected the card. There are also messages on the console like "MMC: killing requests for dead queue" each time you suspend/resume.

The card is removed when you suspend and readded when you resume. That's the only safe thing we can do until we get suspend support in the filesystems.

If you really want to shoot yourself in the foot, there is a Kconfig option that keeps the card around across the suspend.

Rgds

Signature · 5 lines
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org
#3 lore raw
Quoted 11 lines · Pierre Ossman
Quoted 8 lines · On Thu, 19 Jul 2007 16:53:39 +0100 Richard Purdie <[email protected]> wrote:

Lots of Linux handhelds use MMC/SD devices as the root file system. This has worked quite reliably for many kernel versions. In 2.6.22, it seems that if you suspend such a system then resume it, the device locks up. Trying to execute anything on the filesystem results in a "Permission Denied" message. I did see a message from the MMC subsystem saying it had redetected the card. There are also messages on the console like "MMC: killing requests for dead queue" each time you suspend/resume.

Quoted 6 lines

The card is removed when you suspend and readded when you resume. That's the only safe thing we can do until we get suspend support in the filesystems.

If you really want to shoot yourself in the foot, there is a Kconfig option that keeps the card around across the suspend.

I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing was "fixed". I think having this option is a bad idea (in its current form) as it doesn't actually stop filesystem corruption.

With the option disabled, if a filesystem is mounted when you suspend my tests show the filesystem is corrupted. At least if the option is enabled, the filesystem is only corrupted if you remove the card whilst suspended which is more preferable.

I guess the solution would be to abort the suspend if mounted systems were detected and the option was disabled? Alternatively the option could be "auto" enabled only for mounted systems maybe with a printk warning?

Of course the best solution would be to have filesystems support suspend/resume requests since other subsystems like pcmcia also suffer this problem and would benefit from this but I accept that teaching filesystems this is more difficult.

Regards,

Richard

#4 lore raw
Quoted 10 lines · On Sun, 22 Jul 2007 14:18:33 +0100 Richard Purdie <[email protected]> wrote:

I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing was "fixed". I think having this option is a bad idea (in its current form) as it doesn't actually stop filesystem corruption.

With the option disabled, if a filesystem is mounted when you suspend my tests show the filesystem is corrupted. At least if the option is enabled, the filesystem is only corrupted if you remove the card whilst suspended which is more preferable.

I disagree. With this option you get silent corruption, without you get noisy corruption. And I would always prefer the latter, even if it increases the risk of it happening.

Quoted 5 lines

I guess the solution would be to abort the suspend if mounted systems were detected and the option was disabled? Alternatively the option could be "auto" enabled only for mounted systems maybe with a printk warning?

This is a general problem for all removable/hotpluggable storage. So sticking it in the MMC block device would be the wrong layer IMO.

Until the filesystems can be made to store something sane on disk before the suspend, I'd say this is best handled in user space. Let the user space tools refuse to initiate the suspend as long as any removable devices are mounted.

Quoted 5 lines

Of course the best solution would be to have filesystems support suspend/resume requests since other subsystems like pcmcia also suffer this problem and would benefit from this but I accept that teaching filesystems this is more difficult.

Doesn't mean we shouldn't do it. And if we keep papering over the problems, you reduce the motivation of fixing this properly.

Rgds

Signature · 5 lines
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org
#5 lore raw
Quoted 6 lines · Pierre Ossman
Quoted 4 lines · On Sun, 22 Jul 2007 14:18:33 +0100 Richard Purdie <[email protected]> wrote:

I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing was "fixed". I think having this option is a bad idea (in its current form) as it doesn't actually stop filesystem corruption.

Quoted 8 lines
Quoted 4 lines

With the option disabled, if a filesystem is mounted when you suspend my tests show the filesystem is corrupted. At least if the option is enabled, the filesystem is only corrupted if you remove the card whilst suspended which is more preferable.

I disagree. With this option you get silent corruption, without you get noisy corruption. And I would always prefer the latter, even if it increases the risk of it happening.

Corruption is corruption and it shouldn't happen if we can avoid it. It happens with complete certainty in one case and only happens in the other if the user does something which is a fairly obvious bad idea (which is documented as such).

Quoted 7 lines
Quoted 4 lines

I guess the solution would be to abort the suspend if mounted systems were detected and the option was disabled? Alternatively the option could be "auto" enabled only for mounted systems maybe with a printk warning?

This is a general problem for all removable/hotpluggable storage. So sticking it in the MMC block device would be the wrong layer IMO.

It is however if the MMC layer is going to add Kconfig options which corrupt things, it can add things to start fixing things too. If those things can be adapted into more generic code paths, so much the better.

Quoted 4 lines

Until the filesystems can be made to store something sane on disk before the suspend, I'd say this is best handled in user space. Let the user space tools refuse to initiate the suspend as long as any removable devices are mounted.

Given I can suspend the device with "echo mem > /sys/power/state", that implies we need to fix echo? ;-)

Quoted 7 lines
Quoted 5 lines

Of course the best solution would be to have filesystems support suspend/resume requests since other subsystems like pcmcia also suffer this problem and would benefit from this but I accept that teaching filesystems this is more difficult.

Doesn't mean we shouldn't do it.

Agreed.

Quoted 2 lines

And if we keep papering over the problems, you reduce the motivation of fixing this properly.

Maybe although I don't like existing functionality being broken even if its less than ideal.

Regards.

Richard

#6 lore raw
Quoted 6 lines · On Sun, 22 Jul 2007 15:28:00 +0100 Richard Purdie <[email protected]> wrote:

Corruption is corruption and it shouldn't happen if we can avoid it. It happens with complete certainty in one case and only happens in the other if the user does something which is a fairly obvious bad idea (which is documented as such).

The corruption will only occur if the filesystem is dirty. Granted, the mount will be dead and useless, but I wouldn't call that corruption.

Anyway, this behaviour was selected after seeing the long discussion about how USB should handle the same problem. It was decided that it was best to play it safe and remove any devices that couldn't be determined to have remained in the slot. We also have the USB_PERSIST option these days, which does the same thing as MMC_UNSAFE_RESUME.

Quoted 4 lines

Given I can suspend the device with "echo mem > /sys/power/state", that implies we need to fix echo? ;-)

Or that direct usage of /sys/power/state is only for those who know what they are doing (and have umounted their filesystems beforehand).

Quoted 7 lines
Quoted 3 lines

And if we keep papering over the problems, you reduce the motivation of fixing this properly.

Maybe although I don't like existing functionality being broken even if its less than ideal.

I am of the opinion that it was more broken before I touched it. Silent corruption is never acceptable in my book. But if it is in yours, just enable MMC_UNSAFE_RESUME and you'll have the old behaviour.

Rgds

Signature · 5 lines
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org
#7 lore raw
In reply to Richard Purdie

Hi!

Quoted 16 lines
Quoted 9 lines
Quoted 8 lines

Lots of Linux handhelds use MMC/SD devices as the root file system. This has worked quite reliably for many kernel versions. In 2.6.22, it seems that if you suspend such a system then resume it, the device locks up. Trying to execute anything on the filesystem results in a "Permission Denied" message. I did see a message from the MMC subsystem saying it had redetected the card. There are also messages on the console like "MMC: killing requests for dead queue" each time you suspend/resume.

Quoted 6 lines

The card is removed when you suspend and readded when you resume. That's the only safe thing we can do until we get suspend support in the filesystems.

If you really want to shoot yourself in the foot, there is a Kconfig option that keeps the card around across the suspend.

Quoted 8 lines

I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing was "fixed". I think having this option is a bad idea (in its current form) as it doesn't actually stop filesystem corruption.

With the option disabled, if a filesystem is mounted when you suspend my tests show the filesystem is corrupted. At least if the option is enabled, the filesystem is only corrupted if you remove the card whilst suspended which is more preferable.

Are we talking _corruption_ here, or are we talking 'the kind of corruption recoverable by fsck that happens on powerfail'?

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#8 lore raw
Quoted 11 lines · Pavel Machek
Quoted 8 lines

I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing was "fixed". I think having this option is a bad idea (in its current form) as it doesn't actually stop filesystem corruption.

With the option disabled, if a filesystem is mounted when you suspend my tests show the filesystem is corrupted. At least if the option is enabled, the filesystem is only corrupted if you remove the card whilst suspended which is more preferable.

Are we talking _corruption_ here, or are we talking 'the kind of corruption recoverable by fsck that happens on powerfail'?

There was more damage to the system than just a dirty bit set. Yes, fsck could fix it but I don't think it should happen in the first place...

Richard

#9 lore raw
Quoted 10 lines · Richard Purdie
Quoted 9 lines · Pavel Machek
Quoted 8 lines

I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing was "fixed". I think having this option is a bad idea (in its current form) as it doesn't actually stop filesystem corruption.

With the option disabled, if a filesystem is mounted when you suspend my tests show the filesystem is corrupted. At least if the option is enabled, the filesystem is only corrupted if you remove the card whilst suspended which is more preferable.

Quoted 5 lines
Quoted 2 lines

Are we talking _corruption_ here, or are we talking 'the kind of corruption recoverable by fsck that happens on powerfail'?

There was more damage to the system than just a dirty bit set. Yes, fsck could fix it but I don't think it should happen in the first place...

Well, that's "ok", that happens on sudden powerdowns, too.

(Well, but we do sync() during suspend, so it is a bit strange). Do you have fsck logs perhaps?
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html