Skip to contentSkip to conversation

revert yenta free_irq on suspend

DISCUSSION [email protected] 9 participants 21 messages Jul 31, 2005 – Aug 1, 2005

lore
Share & tools
lore
#0 lore raw

----- Original Message -----
From: Linus Torvalds <[email protected]>
Date: Sunday, July 31, 2005 11:53 am
Subject: Re: revert yenta free_irq on suspend

Quoted 23 lines · Pavel Machek
Quoted 2 lines

Well, on some machines interrupts can change during suspend (or

so I

Quoted 1 line

was told). I did not like the ACPI change at one point, but it

is very

Quoted 1 line

wrong to revert PCMCIA fix without also fixing ACPI interpretter.

We _are_ going to fix the ACPI interpreter.

As to irq's changing during suspend - I'll believe that when I see it, not
when some chicken little runs around worrying about it. I doubt anybody
has ever seen it, and I'm 100% sure that we have serious breakage right
now on machines where it definitely doesn't happen.

Quoted 1 line

And it indeed seems that ACPI interpretter is hard to fix in the

right> way.

Quoted 7 lines

We'll revert to the behaviour that it has traditionally had, and start
working forwards in a more careful manner. Where we don't break working
setups.

Linus

Hi Linus,

In general, I think that calling free_irq is the right behavior. Although irqs changing after suspend is rare, there are also some more serious issues. This has been discussed in the past, and a summary is as follows:

1.) race conditions:
Consider the case where several PCI devices are sharing a single PCI interrupt. One device is suspended, but doesn't call free_irq. Now another properly behaving device on the same interrupt line generates an interrupt. Let's say the suspended device had its handler registered first, and therefore is called first. The handler will attempt to access registers on the physically-off device to determine if it generated the interrupt. The PCI bus will issue a master abort. The driver's interrupt handler will likely interpret the read incorrectly.

Every interrupt handler could be modified to check if the device is available, but it would be cleaner and more efficient to unregister the interrupt. Either way, every driver has to be changed to support PM correctly.

2.) runtime power management:
We don't want to leave stale interrupt handlers registered when only suspending a specific device. As we move toward supporting runtime power management it will be important to ensure every driver calls free_irq() in its suspend() (or whatever we're using at that point) routine. This avoids interrupt handler bugs and extra interrupt overhead.

Also I'd like to point out that this patch broke APM suspend-to-ram, not ACPI S3. IMO, it may not be possible to support both APM and ACPI on every system, as their specs are not intended to be compatible. Progress toward proper suspend-to-ram support will, in many cases, be a small setback for APM. This really can't be avoided.

There are, however, some things we can do to mitigate the breakage toward APM. Specifically, we should indicate the type of suspend state, including if it's an ACPI or APM state, for each driver's ->suspend() routine. This will give drivers the opportunity to act differently for APM when necessary. I'm currenlty working on this issue.

APM is useful for legacy hardware and systems with blacklisted ACPI support. I don't think we should attempt to support APM on any system with working ACPI suspend/resume.

Thanks,
Adam

#1 lore raw

Hi!

Quoted 5 lines

Also I'd like to point out that this patch broke APM suspend-to-ram, not ACPI S3. IMO, it may not be possible to support both APM and ACPI on every system, as their specs are not intended to be compatible. Progress toward proper suspend-to-ram support will, in many cases, be a small setback for APM. This really can't be avoided.

Actually, for APM, OS theoretically does *not* need to FREEZE the devices (or do anything else). "Doing nothing" should be easy...
Pavel

Signature · 1 line
if you have sharp zaurus hardware you don't need... you know my address
#3 lore raw
In reply to [email protected]
Quoted 2 lines · On Sun, 31 Jul 2005 [email protected] wrote:

In general, I think that calling free_irq is the right behavior.

I DO NOT CARE!

It breaks hundreds of drivers. End of discussion.

You can do the free_irq() and request_irq() changes _without_ breaking hundreds of drivers by just doing one driver at a time.

And if ACPI then restores the irq controller state, the drivers that _don't_ do this will _also_ continue to work.

Let me re-iterate: the ACPI changes provably BROKE REAL PEOPLES SETUPS.

For absolutely _zero_ gain. Drivers that want to free and re-aquire an interrupt can do so _regardless_ of whether ACPI restores irq routings automatically or not.

And that's my argument. We don't do stupid things that break peoples existing setups in ways that nobody can debug.

Linus

#4 lore raw

Hi!

Quoted 12 lines
Quoted 1 line

In general, I think that calling free_irq is the right behavior.

I DO NOT CARE!

It breaks hundreds of drivers. End of discussion.

You can do the free_irq() and request_irq() changes _without_ breaking hundreds of drivers by just doing one driver at a time.

And if ACPI then restores the irq controller state, the drivers that _don't_ do this will _also_ continue to work.

Quoted 8 lines

Let me re-iterate: the ACPI changes provably BROKE REAL PEOPLES SETUPS.

For absolutely _zero_ gain. Drivers that want to free and re-aquire an interrupt can do so _regardless_ of whether ACPI restores irq routings automatically or not.

And that's my argument. We don't do stupid things that break peoples existing setups in ways that nobody can debug.

Ok, so we'll keep adding those free_irq/request_irq pairs, and re-introduce that ACPI change when we are ready? It would be helpfull to keep the "right thing" in -mm, so there's real motivation to add free_irq/request_irq.
Pavel

Signature · 1 line
if you have sharp zaurus hardware you don't need... you know my address
#15 lore raw
In reply to Linus Torvalds
Quoted 10 lines
Quoted 2 lines

In general, I think that calling free_irq is the right behavior.

I DO NOT CARE!

It breaks hundreds of drivers. End of discussion.

You can do the free_irq() and request_irq() changes _without_ breaking hundreds of drivers by just doing one driver at a time.

So are driver writers supposed to start accepting patches to free/request irqs? in a sense of making it all go forward so at some point we can switch over to doing the correct thing? but my patch for yenta breaks setups for some strange reason..... (maybe just APM ones..)

If so we should put the patch to break links into -mm and then start fixing up drivers...

Dave.

#5 lore raw
In reply to Pavel Machek
Quoted 2 lines · Pavel Machek

Ok, so we'll keep adding those free_irq/request_irq pairs

I would suggest doing so only if you have a case where it can matter.

Quoted 1 line

and re-introduce that ACPI change when we are ready?

Why do it _ever_? There is _zero_ upside to doing it, I don't see why you want to.

Just make ACPI restore the dang thing. It's the right thing to do.

Linus

#6 lore raw

Hi!

Quoted 8 lines
Quoted 1 line

Ok, so we'll keep adding those free_irq/request_irq pairs

I would suggest doing so only if you have a case where it can matter.

Quoted 1 line

and re-introduce that ACPI change when we are ready?

Why do it _ever_? There is _zero_ upside to doing it, I don't see why you want to.

Being able to turn off your soundcard at runtime when you are not using it was one of examples...

Quoted 1 line

Just make ACPI restore the dang thing. It's the right thing to do.

Requires interpretter running with interrupts disabled => ugly :-(.

Pavel

Signature · 1 line
if you have sharp zaurus hardware you don't need... you know my address
#7 lore raw
Quoted 6 lines · Pavel Machek
Quoted 3 lines

Why do it _ever_? There is _zero_ upside to doing it, I don't see why you want to.

Being able to turn off your soundcard at runtime when you are not using it was one of examples...

I meant the "ACPI restores irq controller state" thing.

Just leave it in. There's never any downside. If all the drivers end up doing free_irq/request_irq(), restoring the irq controller state still won't have any negative effect, and it solves the case where you have drivers that don't do it.

Quoted 3 lines
Quoted 1 line

Just make ACPI restore the dang thing. It's the right thing to do.

Requires interpretter running with interrupts disabled => ugly :-(.

I don't see that. What's ugly? I agree that ACPI is ugly, but I do _not_ agree that it's ugly to restore irq controller state with interrupts disabled. It MakesSense(tm) to do so.

The fact that ACPI was designed by a group of monkeys high on LSD, and is some of the worst designs in the industry obviously makes running it at _any_ point pretty damn ugly. And the fact that MB vendors don't test it with anything else than Windows (and sometimes you wonder whether they do even that) doesn't help. So hell yes, it's ugly, and nasty. But interrupts disabled has nothing to do with any of it.

Besides, there's no real reason why you'd even have to do it with interrupts disabled. I personally think that it makes _sense_ to try to restore the irq controller state with irq's off, but as I made clear earlier in this flame-fest, there's no real reason why you couldn't just run with interrupts on.

If an interrupt is screaming due to lack of initialization and gets turned off, just make sure it gets re-enabled when it is being initialized.

Linus

#8 lore raw
Quoted 4 lines

If an interrupt is screaming due to lack of initialization and gets turned off, just make sure it gets re-enabled when it is being initialized.

That still doesn't handle the case where a device has an interrupt handler on a shared IRQ and another device on the chain interrupts it after it has suspended its device,

we need to either fix *for all drivers* (otherwise people sharing IRQs will have breakages that people not sharing them won't see ... )

a) add request/free irq sets
b) add code to the interrupt handlers to make sure we aren't in a powerdown state...

I don't really mind which is the recommended one I'd just prefer we do it the same way everwhere... so I still believe the yenta_irq patch is correct if we are doing a, or if not we need to do b....

Dave.

#9 lore raw
Quoted 4 lines · Dave Airlie

That still doesn't handle the case where a device has an interrupt handler on a shared IRQ and another device on the chain interrupts it after it has suspended its device,

I don't know why people bother arguing about this. Face the facts: we have to do things incrementally. Any design that breaks unmodified drivers is by _definition_ broken. You can't fix all drivers, and anybody who starts their arguments with "we should just fix drivers" is living in la-la-land.

Anyway.

My original PM suggestion handled that case perfectly well. The rule was to make "go to sleep" be a two-phase operation:

- phase 1: save state, and possibly return errors
- phase 2: turn off

where the second stage was done with interrupts disabled and atomically.

And the first phase was done without actually changing the state of the device at all (so that if some device said "I can't do that, Dave", there is no need to go back and wake anything up at all), and we could allocate memory freely, because the disk was still working etc etc.

For some totally inexplicable reason, the PM people never liked this, and ended up doing a single "power off" setup. Which was always known to be broken, and caused tons of problems, like the fact that save-to-disk had to wake up a device that had already been shut off etc.

So the fact that the PM layer was "simplified" to a single-phase setup causes a lot of problems, but hey, stupid is as stupid does. I've given up worrying about what crazy things the PM list comes up with, and instead I now have a hard rule: a patch that breaks some machine gets reverted.

Is that too hard to understand?

I'll repeat it again:

A patch that breaks some machine gets reverted.

and maybe somebody on the PM list will one day understand what it means to have slow and steady progress instead of dicking around with the idea of the week.

Linus

#10 lore raw
Quoted 12 lines · Dave Airlie
Quoted 4 lines

That still doesn't handle the case where a device has an interrupt handler on a shared IRQ and another device on the chain interrupts it after it has suspended its device,

I don't know why people bother arguing about this. Face the facts: we have to do things incrementally. Any design that breaks unmodified drivers is by _definition_ broken. You can't fix all drivers, and anybody who starts their arguments with "we should just fix drivers" is living in la-la-land.

I'm not arguing at all I agree with you, but I think you are missing the point I'm trying to make,

You said earlier we only should fix drivers that need fixing, but they all need fixing, I'm trying to see which way they should be fixed, either the PM people say we need request/free irq pairs or say we need to put support code in the interrupt handlers,

I fail to see how I can fix this very well incrementally, on my hardware I have a yenta, my patch fixes it to work, on Hughs hardware he has his yenta sharing IRQs with another driver which doesn't do the request/free and it breaks, it isn't the yenta drivers fault the other driver causes the issue, so therefore in order to apply the yenta fix I need to go around and fix all the other drivers that might share an interrupt with it before I can get patch accepted so that I don't break someones machine? this is irrelevant to whether the ACPI link change is in or not, adding the request/free pairs to one driver can show up issues in other drivers that share that IRQ..

This has nothing to do with the issues with highlevel PM interfaces for shutting down hardware, this is do with fixing the drivers in the kernel currently and what the correct way to do it is without breaking someone elses hardware....

Dave.

#11 lore raw
Quoted 3 lines · Dave Airlie

You said earlier we only should fix drivers that need fixing, but they all need fixing

I think you're still talking from a theoretical standpoing, while all my arguments are practical.

In _practice_, I hope that

(a) we don't see that very much (ie the people for whom things work already are a strong argument that this is less of a problem in practice than people try to make it appear)

(b) drivers, _especially_ on notebooks, are already able to handle an incoming interrupt with the device in D3 state and returning 0xff for all reads.

In particular, this is exactly the same thing that you get on a surprise device removal too.

iow it really _really_ shouldn't matter that a shared interrupt comes in after (or before) a device has gone to sleep. Because a driver that can't handle that schenario is buggy for totally unrelated reasons, and doing a "free_irq()/request_irq()" pair at suspend time is _not_ the solution!

Quoted 3 lines

I'm trying to see which way they should be fixed, either the PM people say we need request/free irq pairs or say we need to put support code in the interrupt handlers,

And I say that's not true. See (b) above. As far as I can tell, the "interrupt when in D3" really looks _exactly_ the same as "interrupt when device was just removed by the user", and nobody will hopefully argue that free_irq/request_irq can protect against forced removal?

Quoted 4 lines

This has nothing to do with the issues with highlevel PM interfaces for shutting down hardware, this is do with fixing the drivers in the kernel currently and what the correct way to do it is without breaking someone elses hardware....

... and I don't think this has _anything_ to do with free_irq/request_irq, and everything to do with the fact that we can try to make at least the common drivers "hardened" for unexpected interrupts coming in when the hw might not be ready for them.

Linus

#16 lore raw
In reply to [email protected]

Hi,

Quoted 4 lines

In general, I think that calling free_irq is the right behavior. Although irqs changing after suspend is rare, there are also some more serious issues. This has been discussed in the past, and a summary is as follows:

irqs actually isn't changed after suspend currently, it's a considering for future usage like hotplug.
Calling free_irq actually isn't a complete ACPI issue, but ACPI requires it to solve nasty 'sleep in atomic' warning. You will find such break with swsusp without ACPI. Could we revert the ACPI change in Linus's tree but keep it in -mm tree? So we get a chance to fix drivers.

Thanks,
Shaohua

#17 lore raw
Quoted 11 lines · Shaohua Li

Hi,

Quoted 4 lines

In general, I think that calling free_irq is the right behavior. Although irqs changing after suspend is rare, there are also some more serious issues. This has been discussed in the past, and a summary is as follows:

irqs actually isn't changed after suspend currently, it's a considering for future usage like hotplug.
Calling free_irq actually isn't a complete ACPI issue, but ACPI requires it to solve nasty 'sleep in atomic' warning.

Is that the only problem? If so, then surely we can make free_irq() run happily with interrupts disabled: unlink the IRQ handler synchronously, defer the /proc teardown or something like that.

Quoted 3 lines

You will find such break
with swsusp without ACPI. Could we revert the ACPI change in Linus's tree but keep it in -mm tree? So we get a chance to fix drivers.

That depends on the amount of brokenness involved: if it's significant then I'll get a ton of bug reports concerning something which we already know is broken and we'll drive away our long-suffering testers.

#18 lore raw
Quoted 8 lines · Andrew Morton
Quoted 7 lines · Shaohua Li

Hi,

Quoted 4 lines

In general, I think that calling free_irq is the right behavior. Although irqs changing after suspend is rare, there are also some more serious issues. This has been discussed in the past, and a summary is as follows:

Quoted 8 lines
Quoted 4 lines

irqs actually isn't changed after suspend currently, it's a considering for future usage like hotplug.
Calling free_irq actually isn't a complete ACPI issue, but ACPI requires it to solve nasty 'sleep in atomic' warning.

Is that the only problem? If so, then surely we can make free_irq() run happily with interrupts disabled: unlink the IRQ handler synchronously, defer the /proc teardown or something like that.

The problem is we are going to use ACPI interpreter with interrupt disabled. The interpreter itself might use kmalloc, semaphore, iomap, msleep and etc, depends on BIOS. Fixing interpreter is hard. Originally we think to introduce a new system state for suspend/resume to avoid warning, but later we found drivers calling pci_disable_irq/free_irq is better and safer not just for the issue at hand. We might reconsider previous option (a new system state) if free_irq is rejected.

Thanks,
Shaohua

#13 lore raw
In reply to Linus Torvalds
Quoted 1 line

slow and steady progress

The oscillations are indeed discouraging. For S3 sleep/wake on my TP 600X:

2.6.11.4 : works well (the console was hosed with jittering text, but
X restores fine), which hugely improved using my laptop.
2.6.12.3 : ditto

But:

2.6.13-rc3      : goes to sleep but hangs when waking up
      -rc3-mm2  : same
      -rc3-mm3  : same
      -rc4-git3 : same

With those 2.6.13 variants, once it has hung, the power switch will turn it off, but to turn it on I first have to unplug the power and remove both batteries.

On the good side, swsusp seems to be improving with later kernel versions. No luck with 2.6.11.4; reasonable luck with 2.6.12.3; and good results with 2.6.13*.

-Sanjoy

#12 lore raw
In reply to Linus Torvalds

Hi!

Quoted 12 lines
Quoted 2 lines

You said earlier we only should fix drivers that need fixing, but they all need fixing

I think you're still talking from a theoretical standpoing, while all my arguments are practical.

In _practice_, I hope that

(a) we don't see that very much (ie the people for whom things work already are a strong argument that this is less of a problem in practice than people try to make it appear)

Quoted 6 lines

(b) drivers, _especially_ on notebooks, are already able to handle an incoming interrupt with the device in D3 state and returning 0xff for all reads.

In particular, this is exactly the same thing that you get on a surprise device removal too.

Not all devices that we want power-managed are hot-pluggable, even on notebooks. If I need to go all over the tree fixing drivers, I'd much rather add free_irq/request_irq than mess with interrupt routines on hardware I don't have...
Pavel

Signature · 1 line
if you have sharp zaurus hardware you don't need... you know my address
#19 lore raw
In reply to Andrew Morton

Hi!

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

In general, I think that calling free_irq is the right behavior. Although irqs changing after suspend is rare, there are also some more serious issues. This has been discussed in the past, and a summary is as follows:

Quoted 8 lines
Quoted 4 lines

irqs actually isn't changed after suspend currently, it's a considering for future usage like hotplug.
Calling free_irq actually isn't a complete ACPI issue, but ACPI requires it to solve nasty 'sleep in atomic' warning.

Is that the only problem? If so, then surely we can make free_irq() run happily with interrupts disabled: unlink the IRQ handler synchronously, defer the /proc teardown or something like that.

No, the problem is that

a) restoring interrupt links needs interrupts enabled [or rewriting half of ACPI interpretter]

b) to solve a) [and to solve other stuff, too], we need free_irq/request_irq all over the tree.

Quoted 7 lines
Quoted 3 lines

You will find such break
with swsusp without ACPI. Could we revert the ACPI change in Linus's tree but keep it in -mm tree? So we get a chance to fix drivers.

That depends on the amount of brokenness involved: if it's significant then I'll get a ton of bug reports concerning something which we already know is broken and we'll drive away our long-suffering testers.

The amount of brokenness is not that bad, and it fixes some machines, too.
Pavel

Signature · 1 line
if you have sharp zaurus hardware you don't need... you know my address
#14 lore raw
In reply to Linus Torvalds

Hi!

Quoted 13 lines
Quoted 5 lines
Quoted 2 lines

Why do it _ever_? There is _zero_ upside to doing it, I don't see why you want to.

Being able to turn off your soundcard at runtime when you are not using it was one of examples...

I meant the "ACPI restores irq controller state" thing.

Just leave it in. There's never any downside. If all the drivers end up doing free_irq/request_irq(), restoring the irq controller state still won't have any negative effect, and it solves the case where you have drivers that don't do it.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Just make ACPI restore the dang thing. It's the right thing to do.

Requires interpretter running with interrupts disabled => ugly :-(.

I don't see that. What's ugly? I agree that ACPI is ugly, but I do _not_ agree that it's ugly to restore irq controller state with interrupts disabled. It MakesSense(tm) to do so.

ACPI people claim it would mean rewritting lot of code in interpretter. And they are probably right: restoring irq states is done via running ACPI AML code, and it may ask you to grab semaphore; which is something you can't do with interrupts disabled.

So you'd have to teach ACPI interpretter completely different "irq disabled" mode, and even then it is not clear what to do if AML code asks you for some "sleeping operation".

[Hey, it should be Len doing this debate, not me. I hope I'll never have to touch ACPI interpretter.]
Pavel

Signature · 1 line
if you have sharp zaurus hardware you don't need... you know my address
#2 lore raw
In reply to Pavel Machek
Quoted 10 lines · Pavel Machek

Hi!

Quoted 5 lines

Also I'd like to point out that this patch broke APM suspend-to-ram, not ACPI S3. IMO, it may not be possible to support both APM and ACPI on every system, as their specs are not intended to be compatible. Progress toward proper suspend-to-ram support will, in many cases, be a small setback for APM. This really can't be avoided.

Actually, for APM, OS theoretically does *not* need to FREEZE the devices (or do anything else). "Doing nothing" should be easy...

Bullshit. See what happens if you try to APM suspend while a IDE DMA transfer is in progress ...

Ben.

#20 lore raw
In reply to Andrew Morton

Hi,

Quoted 17 lines · Andrew Morton
Quoted 11 lines · Shaohua Li

Hi,

Quoted 4 lines

In general, I think that calling free_irq is the right behavior. Although irqs changing after suspend is rare, there are also some more serious issues. This has been discussed in the past, and a summary is as follows:

irqs actually isn't changed after suspend currently, it's a considering for future usage like hotplug.
Calling free_irq actually isn't a complete ACPI issue, but ACPI requires it to solve nasty 'sleep in atomic' warning.

Is that the only problem? If so, then surely we can make free_irq() run happily with interrupts disabled: unlink the IRQ handler synchronously, defer the /proc teardown or something like that.

Quoted 7 lines
Quoted 3 lines

You will find such break
with swsusp without ACPI. Could we revert the ACPI change in Linus's tree but keep it in -mm tree? So we get a chance to fix drivers.

That depends on the amount of brokenness involved: if it's significant then I'll get a ton of bug reports concerning something which we already know is broken and we'll drive away our long-suffering testers.

Well, IMO there's no such danger. ;-) The relevent ACPI change has been in -mm since 2.6.12-rc1-mm1 and _nobody_ except for me has reported _any_ problems with it. :-)

OTOH, if the change is kept in -mm, we hopefully will be able to identify the drivers that need to be converted first, on the basis of the bug reports.

And the testers need not be driven away if the issues they report are fixed in a timely manner, which is quite simple in this particuar case, because we know what's potentially broken, we know how to fix it, and the fix is quite not that complicated.

Greets,
Rafael

Signature · 3 lines
- Would you tell me, please, which way I ought to go from here?
- That depends a good deal on where you want to get to.
		-- Lewis Carroll "Alice's Adventures in Wonderland"